matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: migrate Matrix + Headscale to edge2 (CT 106/107)

Browse source 
- Matrix -> edge2 CT 106 (5 containers, 3 Postgres DBs, Signal bridge state preserved)
- Headscale -> edge2 CT 107 (noise_private.key carried; tailnet data-plane survived cutover)
- runbook: tailnet-control-plane pattern + self-verifying auto-rollback tmux cutover
- both sources stopped + intact on Contabo as cold rollback

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-19 01:31:34 +00:00
commit 74a0aaae43
4 changed files with 55 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

7
vault/runbooks/lxc-service-migration.md
View file

@ -10,11 +10,11 @@ related:
- [[headscale-onboard-node]]
- [[caddy]]
- [[expose-service-contabo]]
updated: 2026-06-18
updated: 2026-06-19
---
# LXC Service Migration — Contabo → edge2
> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), and **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), **Matrix stack → edge2 CT 106** (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and **Headscale → edge2 CT 107** (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
---
@ -443,6 +443,7 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
| G14 | **Reboot race — Docker binding to the tailnet IP can start before Tailscale is online, failing the bind and leaving the service unreachable after a reboot.** Fix: create a systemd unit on the CT that runs `docker compose up` and has `After=tailscale-online.target` + `Requires=tailscale-online.target` (or equivalent `tailscale status --wait` pre-check). Alternatively, `restart: unless-stopped` in the compose file will cause Docker to self-heal via restarts, but the service will be unreachable for the first ~10–30 s after reboot. Verify reboot survival explicitly (Phase 8). Proven required for Authentik (CT 105, 2026-06-18). |
| G15 | **Do NOT change the dnsmasq split-DNS entry during cutover.** The dnsmasq entry for `<service>.echo6.co` points at the Caddy/TLS host (100.64.0.1 = Contabo), NOT the backend. Only the Caddy upstream changes. Repointing dnsmasq to the backend tailnet IP would break internal HTTPS (no cert, no TLS termination). The Caddy host is always the internal DNS target; the backend IP only appears in the Caddy `reverse_proxy` directive. |
| G16 | **SECRET_KEY and session-signing material must travel byte-for-byte for keystone/session-bearing [[services]]** (e.g. Authentik `AUTHENTIK_SECRET_KEY`, Vaultwarden `rsa_key.pem`). Carrying them verbatim means existing browser sessions survive the cutover — users drop straight in with no forced re-login. If the key is regenerated on the target, all active sessions are invalidated immediately. Confirm from startup logs that no new key was generated. |
| G17 | **Tailnet control-plane migration pattern (proven: Headscale → edge2 CT 107, 2026-06-19).** The data-plane (all 39 registered nodes' WireGuard tunnels) survives a brief control-plane outage — existing node connections stay up; only new key exchanges and re-auths are blocked during the gap. Key material: `noise_private.key` MUST travel byte-for-byte (this is the server's WireGuard identity; losing it forces every node to re-auth). Out-of-band re-entry path: Contabo public IP `5.189.158.149` and edge2 public IP `184.174.35.153` are reachable even when the tailnet is broken — always confirm these work before cutover. **Cutover MUST run inside a `tmux` session on a host reachable via public IP** (not only via tailnet), implementing a self-verifying auto-rollback: if the target fails its health gate within a deadline, the script automatically restores the source Caddy block and restarts the source — so a dropped SSH cannot leave the backbone broken mid-cutover. Drop the public direct-bind (`5.189.158.149:8084`) only after Caddy-proxy reachability is confirmed. |
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |
---
@ -460,4 +461,4 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
---
*Last updated: 2026-06-18 — Added Authentik pilot (PostgreSQL keystone, multi-block Caddy cutover across 2 site blocks, reboot tailscale-before-docker race); Phase 7 multi-block example; Gotchas G14 (reboot race), G15 (dnsmasq must not be repointed), G16 (SECRET_KEY must travel); previously added LiveSync/CouchDB pilot (Variant D, G11-G13)*
*Last updated: 2026-06-19 — Added Matrix pilot (multi-DB Postgres + stateful Signal bridge, CT 106) and Headscale pilot (tailnet control plane, noise_private.key must travel, CT 107); Gotcha G17 (tailnet-control-plane migration pattern: data-plane survives outage, out-of-band re-entry via public IPs, self-verifying auto-rollback in tmux); previously added Authentik pilot (G14–G16)*