echo6-docs/vault/runbooks/lxc-service-migration.md

# LXC Service Migration — Contabo → edge2

> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), and **Authentik (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block Caddy cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

## Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

**Architecture after migration:**

```
Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)
```

**edge2 access (always):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.

---

## Phases

### Phase 0 — Recon & baseline `[G]`

- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `<service>.echo6.co` Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
- Read the live compose file + `.env` + data directory listing **from the running host**. Never use `.ref/archive` docs — they may be stale.
- On edge2 (`ssh edge2`):
  - Confirm the next free CT ID: `sudo pvesh get /cluster/nextid`
  - Confirm chosen IP is free: check `sudo pct list` and the `10.10.10.x` allocation table in `ip-allocation.md`
  - Check for needed template: `sudo pvesm list local | grep <os>`. If absent: `sudo pveam download local <template-volid>`
  - Confirm storage free space: `sudo pvesm status`
- **Rollback:** n/a (read-only except optional template download).

**Service-specific (example: Vaultwarden):** greppeed for `:3012`/`notifications/hub` sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

### Phase 0a — Pre-migration gate `[S]`

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

**Service-specific (example: Vaultwarden):** Verified ≥1 account has a non-empty `password_hash` (local login works without SSO). Recorded `ADMIN_TOKEN` to credentials file.

---

### Phase 1 — Provision the LXC `[G]`

Pick right-sized resources for the service (not a copy of another CT's sizing).

```bash
ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"
```

Append TUN device lines to the CT config (required for Tailscale):

```bash
ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"
```

Verify: internet access via NAT, DNS resolution.

**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`

**Service-specific (example: Vaultwarden):** CT 102, hostname `vaultwarden`, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

### Phase 2 — Docker baseline `[G]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"
```

(`nesting=1` + `keyctl=1` from Phase 1 make Docker-in-LXC work.)

**Rollback:** destroy CT.

---

### Phase 3 — Tailscale onboard `[G]`

Create a preauth key on the Headscale host:

```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```

Install Tailscale + DNS-bootstrap drop-in inside the CT:

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"
```

Capture the tailnet IP: `sudo pct exec <CTID> -- tailscale ip -4` → call it `SVC_TS_IP`.

**Hard pre-cutover reachability gate** (run now, should return `000`; must return `200` after Phase 6):

```bash
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
```

**Rollback:** destroy CT + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`

**Service-specific (example: Vaultwarden):** tailnet IP became 100.64.0.33, Headscale node id 45. Health path: `/alive`.

---

### Phase 4 — Stage compose + config `[G]`

Copy compose file + config **byte-for-byte from the live running host** into the CT. Never copy from `.ref/archive` docs.

```bash
# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files
```

**The ONE compose edit `[G3]`:** change the `ports` bind address from `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>` (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

**Rollback:** destroy CT.

**Service-specific (example: Vaultwarden):** `.env` carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed `127.0.0.1:8086:80` → `100.64.0.33:8086:80`. Confirmed no `:3012` line.

---

### Phase 5 — Migrate data `[S]`

This phase is entirely service-specific. Common patterns:

| Data type | Method |
|-----------|--------|
| Stateless (no persistent data) | Skip this phase |
| SQLite (WAL mode) | Stop source → `sqlite3 .backup` → verify integrity → transfer |
| PostgreSQL | Stop app only (keep DB up) → `pg_dump -Fc` → transfer → restore on target |
| Multi-volume (file data + keys) | `tar` the data volume → transfer → extract into target volume |
| File data only | `rsync` or `tar` |

**Variant A — Zero-loss for single-container SQLite:**
```bash
# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"
```

Do NOT copy live `-wal`/`-shm` files alongside the snapshot.

**Variant B — PostgreSQL dump/restore (proven with Forgejo):**

Stop the application container only — keep the DB container running so `pg_dump` can connect cleanly:

```bash
# Stop app only; DB stays up
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop <app-container-name>'

# Dump to custom format (compressed, schema + data)
ssh root@100.64.0.1 'docker exec <db-container-name> pg_dump -U <db-user> -d <db-name> -Fc -f /tmp/<db-name>.dump'
ssh root@100.64.0.1 'docker cp <db-container-name>:/tmp/<db-name>.dump /tmp/<db-name>.dump'

# Transfer to cortex, then push into CT
scp root@100.64.0.1:/tmp/<db-name>.dump /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<db-name>.dump /tmp/<db-name>.dump"

# On target: start DB container only, wait for it to be healthy, then restore
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d <db-container-name>'"
# Wait ~10s for postgres to initialise
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker cp /tmp/<db-name>.dump <db-container-name>:/tmp/<db-name>.dump'"
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> pg_restore -U <db-user> -d <db-name> --no-owner /tmp/<db-name>.dump'"
```

**Integrity gate — compare key table row counts source vs target:**

```bash
# Source (still stopped app, DB still up)
ssh root@100.64.0.1 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c "SELECT COUNT(*) FROM <key-table>;"'

# Target
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c \"SELECT COUNT(*) FROM <key-table>;\"'"
```

Row counts must match exactly before proceeding. Do not proceed to Phase 6 if they differ.

**Variant C — Multi-volume tar (file data + SSH host keys, proven with Forgejo):**

When a service has a data volume containing binary or structured data (e.g., git repos, SSH host keys, app config) alongside a database, tar the entire named Docker volume and extract it into a freshly created target volume:

```bash
# On source: tar the named volume via a throwaway container
ssh root@100.64.0.1 'docker run --rm -v <volume-name>:/source -v /tmp:/target alpine \
  tar -czf /target/<volume-name>.tar.gz -C /source .'

# Transfer
scp root@100.64.0.1:/tmp/<volume-name>.tar.gz /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<volume-name>.tar.gz /tmp/<volume-name>.tar.gz"

# On target: create volume + extract
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  docker volume create <volume-name>
  docker run --rm -v <volume-name>:/target -v /tmp:/source alpine \
    tar -xzf /source/<volume-name>.tar.gz -C /target
'"
```

SSH host keys MUST travel in the data volume (see G11) so clients see no key-change warning.

**Variant D — CouchDB named volumes + bind-mounted config files (proven with LiveSync):**

CouchDB uses named Docker volumes for data (`couchdb-data`, `provisioner-data`) and bind-mounted config files (`couchdb/local.ini`, `couchdb/local.d/` including `jwt-keys.ini`). Use the same cold named-volume tar as Variant C for each volume, AND copy the compose directory (with all bind-mount source paths) intact:

```bash
# Stop source entirely (CouchDB does not support online volume export safely)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Tar each named volume via throwaway container (same as Variant C)
ssh root@100.64.0.1 'docker run --rm -v couchdb-data:/source -v /tmp:/target alpine tar -czf /target/couchdb-data.tar.gz -C /source .'
ssh root@100.64.0.1 'docker run --rm -v provisioner-data:/source -v /tmp:/target alpine tar -czf /target/provisioner-data.tar.gz -C /source .'

# Copy the whole compose directory (includes bind-mounted config: local.ini, local.d/*, jwt-keys.ini)
ssh root@100.64.0.1 'tar -czf /tmp/<service>-compose-dir.tar.gz -C /opt <service>'

# Transfer + extract in CT
# (transfer tarballs to cortex, push to CT via edge2 pct push, then extract)
```

**Critical:** bind-mounted config files (e.g. `jwt-keys.ini` in `couchdb/local.d/`) MUST travel as part of the compose-dir copy alongside the named volumes. If `jwt-keys.ini` is missing, per-user JWT auth breaks on startup.

**Rollback:** destroy CT + `ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'`

**Service-specific (example: Vaultwarden):** Also transferred `rsa_key.pem` byte-for-byte (JWT signing key — see G9). Did NOT transfer `db.sqlite3-wal`, `db.sqlite3-shm`, `tmp/`.

**Service-specific (example: Forgejo):** Used Variant B (PostgreSQL) + Variant C (forgejo-data volume with git repos + SSH host keys). Stopped only the `forgejo` app container; kept `forgejo-db` running for dump. Integrity gate: `SELECT COUNT(*) FROM repository` == 9 source and target. SSH host keys in forgejo-data volume transferred intact — no client warning on reconnect.

**Service-specific (example: LiveSync/CouchDB):** Used Variant D. Stopped source entirely; tarred `couchdb-data` + `provisioner-data` named volumes; copied compose dir (incl. `couchdb/local.d/jwt-keys.ini`). Data: ~16 MB. Integrity gate: verified document count in `cc-db` + `userdb-matt` matched source via `curl http://localhost:5984/<db>` against both hosts after cutover.

---

### Phase 6 — Start + health gate `[G/S]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"
```

All must pass before cutover:

1. **Data integrity check** (if applicable, e.g. `sqlite3 db.sqlite3 "PRAGMA integrity_check"` == `ok`)
2. **Service-specific startup log checks** (no error markers)
3. **`/alive` == 200 over tailnet from Contabo** (the HARD composed-path gate):
   ```bash
   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   ```
   Must return `200` — do not proceed to Phase 7 if it doesn't.

**Rollback:** destroy CT + restart source on Contabo.

**Service-specific (example: Vaultwarden):** Also checked that container logs did NOT contain `generating new RSA key` (confirms `rsa_key.pem` loaded — G9). Checked `SELECT count(*) FROM users` == 1.

---

### Phase 7 — Cutover `[G]`

On Contabo, take a backup first:

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'
```

**The ONE edit:** replace only the upstream token in the `<service>.echo6.co` block — `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>`. No scheme prefix (bare `host:port` defaults to HTTP). Preserve all other directives (e.g. `header_up X-Real-IP {remote_host}`).

Validate, then restart (G10 — `caddy reload` fails under `admin off`):

```bash
ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'
```

**Blast radius (G10):** `systemctl restart caddy` briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

**Rollback (~1–2 s, public DNS never moved):**

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'
```

**Service-specific (example: Vaultwarden):** Final block:
```caddyfile
vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}
```

**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x` → `100.64.0.35:598x` in one edit. A third upstream in the same block — the Authentik outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.

**Multi-block cutover example (Authentik, 2026-06-18):** Authentik appeared in 4 places across 2 site blocks — `auth.echo6.co` (outpost path matcher + catch-all both pointing to `127.0.0.1:9000`) and `notes.echo6.co` (outpost path matcher + `forward_auth` directive both pointing to `127.0.0.1:9000`). All 4 occurrences were updated to `100.64.0.36:9000` in one edit. Grep the entire Caddyfile for the service's port before cutting over — do not assume a service lives in only one block. See also G15 (dnsmasq must NOT be repointed) and G16 (SECRET_KEY must travel).

---

#### Phase 7a — Non-Caddy public TCP port (iptables DNAT) `[S]` *(Forgejo SSH variant)*

When a service exposes a raw TCP port that is NOT handled by Caddy (e.g., Forgejo git SSH on port 2222), the one-token Caddy edit in Phase 7 is insufficient. Use a reboot-persistent **iptables DNAT systemd unit** on the Contabo host to forward the port to the target tailnet IP.

**Do NOT apply by rebooting the Contabo host** — it runs production services. Create the unit directly and start it without a reboot.

Template unit (`/etc/systemd/system/forgejo-ssh-dnat.service` — parameterize PORT and TARGET_TS_IP):

```ini
[Unit]
Description=Forgejo SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
```

Deploy and enable:

```bash
# Write unit to Contabo
ssh root@100.64.0.1 'cat > /etc/systemd/system/<service>-ssh-dnat.service' << 'EOF'
[Unit]
Description=<Service> SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
EOF

ssh root@100.64.0.1 'systemctl daemon-reload && systemctl enable --now <service>-ssh-dnat.service'
```

Verify DNAT is live (before restarting caddy):

```bash
ssh -p <PORT> git@5.189.158.149  # Should land on target service, not source
```

**Rollback:**

```bash
ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /etc/systemd/system/<service>-ssh-dnat.service && systemctl daemon-reload'
```

**Service-specific (example: Forgejo):** PORT=2222, TARGET_TS_IP=100.64.0.34. Unit: `forgejo-ssh-dnat.service`. Verified: `ssh -p 2222 git@forge.echo6.co` arrived at edge2 CT 103 with preserved host keys (no client warning).

---

### Phase 8 — End-to-end verification `[G/S]`

- `<service>.echo6.co` DNS still resolves to `5.189.158.149` (unchanged)
- `curl -I https://<service>.echo6.co` → 200, valid cert
- Functional login + core workflow test
- **Reboot survival:** `ssh edge2 'sudo pct reboot <CTID>'` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), Tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 `[G8]`
- **Rollback:** Phase 7 revert.

**Service-specific (example: Vaultwarden):** Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

### Phase 9 — Decommission (deferred) `[G]`

- Source stays **installed + stopped** as cold rollback during soak period
- Before eventual deletion: `tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>`
- Update `.ref` docs (services.md, environment.md, ip-allocation.md)
- Retire or banner any stale archive docs
- Promote migration plan to this runbook (done)

---

## Gotchas

| ID | Issue |
|----|-------|
| G1 | Stale docs may show sub-routes (e.g. `:3012`/`notifications/hub`) that don't exist in the live Caddyfile. Always read from the live host in Phase 0. |
| G2 | edge2 `vmbr0` has **no DHCP** and no `local-lvm` → must use static IP + `local` (dir) storage. |
| G3 | Source binds `127.0.0.1:<PORT>:<CONTAINER_PORT>`; LXC must bind `<SVC_TS_IP>:<PORT>:<CONTAINER_PORT>`. Keep the container-side port unchanged. |
| G4 | SQLite in WAL mode has a live `.wal` file — never copy `db.sqlite3` + `-wal` naively. Use `sqlite3 .backup` (produces one self-contained file). Stop the source first for zero-loss. |
| G5 | Preserve all existing Caddy directives on the backend block (e.g. `header_up X-Real-IP {remote_host}`) — edit only the upstream token. |
| G6 | SSO redirect URIs derive from the unchanged `DOMAIN`/DNS — **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path. |
| G7 | Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port. |
| G8 | Reboot survival requires all three: `--onboot 1`, `restart: unless-stopped`, and the Tailscale DNS-bootstrap drop-in. |
| G9 | JWT signing keys (e.g. `rsa_key.pem` for Vaultwarden) **must travel byte-for-byte**. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages. |
| G10 | Contabo Caddy runs with `admin off` → `caddy reload` and `systemctl reload caddy` FAIL. Always use `caddy validate` then `systemctl restart caddy`. Restart drops all Contabo-fronted sites for ~1–2 s. |
| G11 | SSH host keys must travel inside the data volume (Variant C tar). If they are missing or regenerated on the target, every git client gets a host-key-changed warning and must manually clear `~/.ssh/known_hosts`. Transfer the full data volume; verify on target with `docker exec <app> cat /data/ssh/forgejo.rsa.pub` or equivalent before cutover. |
| G12 | DB row-count integrity gate is mandatory for PostgreSQL migrations. Never skip it — a silent pg_restore failure (wrong role, encoding mismatch) leaves the DB empty or partially populated while `pg_restore` exits 0. Compare at least one key business table. |
| G13 | iptables DNAT must be made reboot-persistent via a systemd `oneshot`/`RemainAfterExit` unit (Phase 7a). Do NOT rely on iptables-persistent packages or manual rules — they require package installs (forbidden) or do not survive all reboot paths. Do NOT apply by rebooting the Contabo host (production). Create the unit file, `daemon-reload`, `enable --now`. |
| G14 | **Reboot race — Docker binding to the tailnet IP can start before Tailscale is online, failing the bind and leaving the service unreachable after a reboot.** Fix: create a systemd unit on the CT that runs `docker compose up` and has `After=tailscale-online.target` + `Requires=tailscale-online.target` (or equivalent `tailscale status --wait` pre-check). Alternatively, `restart: unless-stopped` in the compose file will cause Docker to self-heal via restarts, but the service will be unreachable for the first ~10–30 s after reboot. Verify reboot survival explicitly (Phase 8). Proven required for Authentik (CT 105, 2026-06-18). |
| G15 | **Do NOT change the dnsmasq split-DNS entry during cutover.** The dnsmasq entry for `<service>.echo6.co` points at the Caddy/TLS host (100.64.0.1 = Contabo), NOT the backend. Only the Caddy upstream changes. Repointing dnsmasq to the backend tailnet IP would break internal HTTPS (no cert, no TLS termination). The Caddy host is always the internal DNS target; the backend IP only appears in the Caddy `reverse_proxy` directive. |
| G16 | **SECRET_KEY and session-signing material must travel byte-for-byte for keystone/session-bearing services** (e.g. Authentik `AUTHENTIK_SECRET_KEY`, Vaultwarden `rsa_key.pem`). Carrying them verbatim means existing browser sessions survive the cutover — users drop straight in with no forced re-login. If the key is regenerated on the target, all active sessions are invalidated immediately. Confirm from startup logs that no new key was generated. |
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |

---

## Template Summary

**Generic phases (identical for every service):** 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 7a (if service exposes non-Caddy TCP port: iptables DNAT systemd unit on Contabo) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

**Service-specific phases:** 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

**Access pins:**
- Contabo: `ssh root@100.64.0.1`
- edge2: `ssh admin@184.174.35.153` (alias `edge2`) + `sudo` for pct/pvesm/pveam
- `root@100.64.0.26` is refused on edge2

---

*Last updated: 2026-06-18 — Added Authentik pilot (PostgreSQL keystone, multi-block Caddy cutover across 2 site blocks, reboot tailscale-before-docker race); Phase 7 multi-block example; Gotchas G14 (reboot race), G15 (dnsmasq must not be repointed), G16 (SECRET_KEY must travel); previously added LiveSync/CouchDB pilot (Variant D, G11-G13)*