echo6-docs/runbooks/lxc-service-migration.md

# LXC Service Migration — Contabo → edge2

> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16) and **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16). This runbook generalizes both patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

## Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

**Architecture after migration:**

```
Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)
```

**edge2 access (always):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.

---

## Phases

### Phase 0 — Recon & baseline `[G]`

- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `<service>.echo6.co` Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
- Read the live compose file + `.env` + data directory listing **from the running host**. Never use `.ref/archive` docs — they may be stale.
- On edge2 (`ssh edge2`):
  - Confirm the next free CT ID: `sudo pvesh get /cluster/nextid`
  - Confirm chosen IP is free: check `sudo pct list` and the `10.10.10.x` allocation table in `ip-allocation.md`
  - Check for needed template: `sudo pvesm list local | grep <os>`. If absent: `sudo pveam download local <template-volid>`
  - Confirm storage free space: `sudo pvesm status`
- **Rollback:** n/a (read-only except optional template download).

**Service-specific (example: Vaultwarden):** greppeed for `:3012`/`notifications/hub` sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

### Phase 0a — Pre-migration gate `[S]`

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

**Service-specific (example: Vaultwarden):** Verified ≥1 account has a non-empty `password_hash` (local login works without SSO). Recorded `ADMIN_TOKEN` to credentials file.

---

### Phase 1 — Provision the LXC `[G]`

Pick right-sized resources for the service (not a copy of another CT's sizing).

```bash
ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"
```

Append TUN device lines to the CT config (required for Tailscale):

```bash
ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"
```

Verify: internet access via NAT, DNS resolution.

**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`

**Service-specific (example: Vaultwarden):** CT 102, hostname `vaultwarden`, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

### Phase 2 — Docker baseline `[G]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"
```

(`nesting=1` + `keyctl=1` from Phase 1 make Docker-in-LXC work.)

**Rollback:** destroy CT.

---

### Phase 3 — Tailscale onboard `[G]`

Create a preauth key on the Headscale host:

```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```

Install Tailscale + DNS-bootstrap drop-in inside the CT:

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"
```

Capture the tailnet IP: `sudo pct exec <CTID> -- tailscale ip -4` → call it `SVC_TS_IP`.

**Hard pre-cutover reachability gate** (run now, should return `000`; must return `200` after Phase 6):

```bash
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
```

**Rollback:** destroy CT + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`

**Service-specific (example: Vaultwarden):** tailnet IP became 100.64.0.33, Headscale node id 45. Health path: `/alive`.

---

### Phase 4 — Stage compose + config `[G]`

Copy compose file + config **byte-for-byte from the live running host** into the CT. Never copy from `.ref/archive` docs.

```bash
# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files
```

**The ONE compose edit `[G3]`:** change the `ports` bind address from `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>` (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

**Rollback:** destroy CT.

**Service-specific (example: Vaultwarden):** `.env` carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed `127.0.0.1:8086:80` → `100.64.0.33:8086:80`. Confirmed no `:3012` line.

---

### Phase 5 — Migrate data `[S]`

This phase is entirely service-specific. Common patterns:

| Data type | Method |
|-----------|--------|
| Stateless (no persistent data) | Skip this phase |
| SQLite (WAL mode) | Stop source → `sqlite3 .backup` → verify integrity → transfer |
| PostgreSQL | Stop app only (keep DB up) → `pg_dump -Fc` → transfer → restore on target |
| Multi-volume (file data + keys) | `tar` the data volume → transfer → extract into target volume |
| File data only | `rsync` or `tar` |

**Variant A — Zero-loss for single-container SQLite:**
```bash
# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"
```

Do NOT copy live `-wal`/`-shm` files alongside the snapshot.

**Variant B — PostgreSQL dump/restore (proven with Forgejo):**

Stop the application container only — keep the DB container running so `pg_dump` can connect cleanly:

```bash
# Stop app only; DB stays up
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop <app-container-name>'

# Dump to custom format (compressed, schema + data)
ssh root@100.64.0.1 'docker exec <db-container-name> pg_dump -U <db-user> -d <db-name> -Fc -f /tmp/<db-name>.dump'
ssh root@100.64.0.1 'docker cp <db-container-name>:/tmp/<db-name>.dump /tmp/<db-name>.dump'

# Transfer to cortex, then push into CT
scp root@100.64.0.1:/tmp/<db-name>.dump /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<db-name>.dump /tmp/<db-name>.dump"

# On target: start DB container only, wait for it to be healthy, then restore
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d <db-container-name>'"
# Wait ~10s for postgres to initialise
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker cp /tmp/<db-name>.dump <db-container-name>:/tmp/<db-name>.dump'"
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> pg_restore -U <db-user> -d <db-name> --no-owner /tmp/<db-name>.dump'"
```

**Integrity gate — compare key table row counts source vs target:**

```bash
# Source (still stopped app, DB still up)
ssh root@100.64.0.1 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c "SELECT COUNT(*) FROM <key-table>;"'

# Target
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c \"SELECT COUNT(*) FROM <key-table>;\"'"
```

Row counts must match exactly before proceeding. Do not proceed to Phase 6 if they differ.

**Variant C — Multi-volume tar (file data + SSH host keys, proven with Forgejo):**

When a service has a data volume containing binary or structured data (e.g., git repos, SSH host keys, app config) alongside a database, tar the entire named Docker volume and extract it into a freshly created target volume:

```bash
# On source: tar the named volume via a throwaway container
ssh root@100.64.0.1 'docker run --rm -v <volume-name>:/source -v /tmp:/target alpine \
  tar -czf /target/<volume-name>.tar.gz -C /source .'

# Transfer
scp root@100.64.0.1:/tmp/<volume-name>.tar.gz /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<volume-name>.tar.gz /tmp/<volume-name>.tar.gz"

# On target: create volume + extract
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  docker volume create <volume-name>
  docker run --rm -v <volume-name>:/target -v /tmp:/source alpine \
    tar -xzf /source/<volume-name>.tar.gz -C /target
'"
```

SSH host keys MUST travel in the data volume (see G11) so clients see no key-change warning.

**Rollback:** destroy CT + `ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'`

**Service-specific (example: Vaultwarden):** Also transferred `rsa_key.pem` byte-for-byte (JWT signing key — see G9). Did NOT transfer `db.sqlite3-wal`, `db.sqlite3-shm`, `tmp/`.

**Service-specific (example: Forgejo):** Used Variant B (PostgreSQL) + Variant C (forgejo-data volume with git repos + SSH host keys). Stopped only the `forgejo` app container; kept `forgejo-db` running for dump. Integrity gate: `SELECT COUNT(*) FROM repository` == 9 source and target. SSH host keys in forgejo-data volume transferred intact — no client warning on reconnect.

---

### Phase 6 — Start + health gate `[G/S]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"
```

All must pass before cutover:

1. **Data integrity check** (if applicable, e.g. `sqlite3 db.sqlite3 "PRAGMA integrity_check"` == `ok`)
2. **Service-specific startup log checks** (no error markers)
3. **`/alive` == 200 over tailnet from Contabo** (the HARD composed-path gate):
   ```bash
   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   ```
   Must return `200` — do not proceed to Phase 7 if it doesn't.

**Rollback:** destroy CT + restart source on Contabo.

**Service-specific (example: Vaultwarden):** Also checked that container logs did NOT contain `generating new RSA key` (confirms `rsa_key.pem` loaded — G9). Checked `SELECT count(*) FROM users` == 1.

---

### Phase 7 — Cutover `[G]`

On Contabo, take a backup first:

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'
```

**The ONE edit:** replace only the upstream token in the `<service>.echo6.co` block — `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>`. No scheme prefix (bare `host:port` defaults to HTTP). Preserve all other directives (e.g. `header_up X-Real-IP {remote_host}`).

Validate, then restart (G10 — `caddy reload` fails under `admin off`):

```bash
ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'
```

**Blast radius (G10):** `systemctl restart caddy` briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

**Rollback (~1–2 s, public DNS never moved):**

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'
```

**Service-specific (example: Vaultwarden):** Final block:
```caddyfile
vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}
```

---

#### Phase 7a — Non-Caddy public TCP port (iptables DNAT) `[S]` *(Forgejo SSH variant)*

When a service exposes a raw TCP port that is NOT handled by Caddy (e.g., Forgejo git SSH on port 2222), the one-token Caddy edit in Phase 7 is insufficient. Use a reboot-persistent **iptables DNAT systemd unit** on the Contabo host to forward the port to the target tailnet IP.

**Do NOT apply by rebooting the Contabo host** — it runs production services. Create the unit directly and start it without a reboot.

Template unit (`/etc/systemd/system/forgejo-ssh-dnat.service` — parameterize PORT and TARGET_TS_IP):

```ini
[Unit]
Description=Forgejo SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
```

Deploy and enable:

```bash
# Write unit to Contabo
ssh root@100.64.0.1 'cat > /etc/systemd/system/<service>-ssh-dnat.service' << 'EOF'
[Unit]
Description=<Service> SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
EOF

ssh root@100.64.0.1 'systemctl daemon-reload && systemctl enable --now <service>-ssh-dnat.service'
```

Verify DNAT is live (before restarting caddy):

```bash
ssh -p <PORT> git@5.189.158.149  # Should land on target service, not source
```

**Rollback:**

```bash
ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /etc/systemd/system/<service>-ssh-dnat.service && systemctl daemon-reload'
```

**Service-specific (example: Forgejo):** PORT=2222, TARGET_TS_IP=100.64.0.34. Unit: `forgejo-ssh-dnat.service`. Verified: `ssh -p 2222 git@forge.echo6.co` arrived at edge2 CT 103 with preserved host keys (no client warning).

---

### Phase 8 — End-to-end verification `[G/S]`

- `<service>.echo6.co` DNS still resolves to `5.189.158.149` (unchanged)
- `curl -I https://<service>.echo6.co` → 200, valid cert
- Functional login + core workflow test
- **Reboot survival:** `ssh edge2 'sudo pct reboot <CTID>'` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), Tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 `[G8]`
- **Rollback:** Phase 7 revert.

**Service-specific (example: Vaultwarden):** Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

### Phase 9 — Decommission (deferred) `[G]`

- Source stays **installed + stopped** as cold rollback during soak period
- Before eventual deletion: `tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>`
- Update `.ref` docs (services.md, environment.md, ip-allocation.md)
- Retire or banner any stale archive docs
- Promote migration plan to this runbook (done)

---

## Gotchas

| ID | Issue |
|----|-------|
| G1 | Stale docs may show sub-routes (e.g. `:3012`/`notifications/hub`) that don't exist in the live Caddyfile. Always read from the live host in Phase 0. |
| G2 | edge2 `vmbr0` has **no DHCP** and no `local-lvm` → must use static IP + `local` (dir) storage. |
| G3 | Source binds `127.0.0.1:<PORT>:<CONTAINER_PORT>`; LXC must bind `<SVC_TS_IP>:<PORT>:<CONTAINER_PORT>`. Keep the container-side port unchanged. |
| G4 | SQLite in WAL mode has a live `.wal` file — never copy `db.sqlite3` + `-wal` naively. Use `sqlite3 .backup` (produces one self-contained file). Stop the source first for zero-loss. |
| G5 | Preserve all existing Caddy directives on the backend block (e.g. `header_up X-Real-IP {remote_host}`) — edit only the upstream token. |
| G6 | SSO redirect URIs derive from the unchanged `DOMAIN`/DNS — **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path. |
| G7 | Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port. |
| G8 | Reboot survival requires all three: `--onboot 1`, `restart: unless-stopped`, and the Tailscale DNS-bootstrap drop-in. |
| G9 | JWT signing keys (e.g. `rsa_key.pem` for Vaultwarden) **must travel byte-for-byte**. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages. |
| G10 | Contabo Caddy runs with `admin off` → `caddy reload` and `systemctl reload caddy` FAIL. Always use `caddy validate` then `systemctl restart caddy`. Restart drops all Contabo-fronted sites for ~1–2 s. |
| G11 | SSH host keys must travel inside the data volume (Variant C tar). If they are missing or regenerated on the target, every git client gets a host-key-changed warning and must manually clear `~/.ssh/known_hosts`. Transfer the full data volume; verify on target with `docker exec <app> cat /data/ssh/forgejo.rsa.pub` or equivalent before cutover. |
| G12 | DB row-count integrity gate is mandatory for PostgreSQL migrations. Never skip it — a silent pg_restore failure (wrong role, encoding mismatch) leaves the DB empty or partially populated while `pg_restore` exits 0. Compare at least one key business table. |
| G13 | iptables DNAT must be made reboot-persistent via a systemd `oneshot`/`RemainAfterExit` unit (Phase 7a). Do NOT rely on iptables-persistent packages or manual rules — they require package installs (forbidden) or do not survive all reboot paths. Do NOT apply by rebooting the Contabo host (production). Create the unit file, `daemon-reload`, `enable --now`. |
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |

---

## Template Summary

**Generic phases (identical for every service):** 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 7a (if service exposes non-Caddy TCP port: iptables DNAT systemd unit on Contabo) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

**Service-specific phases:** 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

**Access pins:**
- Contabo: `ssh root@100.64.0.1`
- edge2: `ssh admin@184.174.35.153` (alias `edge2`) + `sudo` for pct/pvesm/pveam
- `root@100.64.0.26` is refused on edge2

---

*Last updated: 2026-06-17 — Added Forgejo pilot (PostgreSQL + multi-volume + iptables DNAT); updated Phase 5 Variants B/C, Phase 7a, Gotchas G11-G13, Template Summary*