echo6-docs/runbooks/lxc-service-migration.md

LXC Service Migration — Contabo → edge2

Proven pilots: Vaultwarden → edge2 CT 102 (SQLite, 2026-06-16) and Forgejo → edge2 CT 103 (PostgreSQL + non-Caddy SSH port, 2026-06-16). This runbook generalizes both patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

Architecture after migration:

Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)

edge2 access (always): ssh admin@184.174.35.153 (alias edge2, key ~/.ssh/contabo2_ed25519), then sudo for every pct/pvesm/pveam command. root@100.64.0.26 is refused — do not use it.

---

Phases

Phase 0 — Recon & baseline [G]

• On Contabo (ssh root@100.64.0.1): capture the verbatim <service>.echo6.co Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
• Read the live compose file + .env + data directory listing from the running host. Never use .ref/archive docs — they may be stale.
• On edge2 (ssh edge2):
  • Confirm the next free CT ID: sudo pvesh get /cluster/nextid
  • Confirm chosen IP is free: check sudo pct list and the 10.10.10.x allocation table in ip-allocation.md
  • Check for needed template: sudo pvesm list local | grep <os>. If absent: sudo pveam download local <template-volid>
  • Confirm storage free space: sudo pvesm status
• Rollback: n/a (read-only except optional template download).

Service-specific (example: Vaultwarden): greppeed for :3012/notifications/hub sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

Phase 0a — Pre-migration gate [S]

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

Service-specific (example: Vaultwarden): Verified ≥1 account has a non-empty password_hash (local login works without SSO). Recorded ADMIN_TOKEN to credentials file.

---

Phase 1 — Provision the LXC [G]

Pick right-sized resources for the service (not a copy of another CT's sizing).

ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"

Append TUN device lines to the CT config (required for Tailscale):

ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"

Verify: internet access via NAT, DNS resolution.

Rollback: ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'

Service-specific (example: Vaultwarden): CT 102, hostname vaultwarden, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

Phase 2 — Docker baseline [G]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"

(nesting=1 + keyctl=1 from Phase 1 make Docker-in-LXC work.)

Rollback: destroy CT.

---

Phase 3 — Tailscale onboard [G]

Create a preauth key on the Headscale host:

ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'

Install Tailscale + DNS-bootstrap drop-in inside the CT:

ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"

Capture the tailnet IP: sudo pct exec <CTID> -- tailscale ip -4 → call it SVC_TS_IP.

Hard pre-cutover reachability gate (run now, should return 000; must return 200 after Phase 6):

ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"

Rollback: destroy CT + ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'

Service-specific (example: Vaultwarden): tailnet IP became 100.64.0.33, Headscale node id 45. Health path: /alive.

---

Phase 4 — Stage compose + config [G]

Copy compose file + config byte-for-byte from the live running host into the CT. Never copy from .ref/archive docs.

# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files

The ONE compose edit [G3]: change the ports bind address from 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT> (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

Rollback: destroy CT.

Service-specific (example: Vaultwarden): .env carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed 127.0.0.1:8086:80 → 100.64.0.33:8086:80. Confirmed no :3012 line.

---

Phase 5 — Migrate data [S]

This phase is entirely service-specific. Common patterns:

Data type	Method
Stateless (no persistent data)	Skip this phase
SQLite (WAL mode)	Stop source → sqlite3 .backup → verify integrity → transfer
PostgreSQL	Stop app only (keep DB up) → pg_dump -Fc → transfer → restore on target
Multi-volume (file data + keys)	tar the data volume → transfer → extract into target volume
File data only	rsync or tar

Variant A — Zero-loss for single-container SQLite:

# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"

Do NOT copy live -wal/-shm files alongside the snapshot.

Variant B — PostgreSQL dump/restore (proven with Forgejo):

Stop the application container only — keep the DB container running so pg_dump can connect cleanly:

# Stop app only; DB stays up
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop <app-container-name>'

# Dump to custom format (compressed, schema + data)
ssh root@100.64.0.1 'docker exec <db-container-name> pg_dump -U <db-user> -d <db-name> -Fc -f /tmp/<db-name>.dump'
ssh root@100.64.0.1 'docker cp <db-container-name>:/tmp/<db-name>.dump /tmp/<db-name>.dump'

# Transfer to cortex, then push into CT
scp root@100.64.0.1:/tmp/<db-name>.dump /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<db-name>.dump /tmp/<db-name>.dump"

# On target: start DB container only, wait for it to be healthy, then restore
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d <db-container-name>'"
# Wait ~10s for postgres to initialise
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker cp /tmp/<db-name>.dump <db-container-name>:/tmp/<db-name>.dump'"
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> pg_restore -U <db-user> -d <db-name> --no-owner /tmp/<db-name>.dump'"

Integrity gate — compare key table row counts source vs target:

# Source (still stopped app, DB still up)
ssh root@100.64.0.1 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c "SELECT COUNT(*) FROM <key-table>;"'

# Target
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c \"SELECT COUNT(*) FROM <key-table>;\"'"

Row counts must match exactly before proceeding. Do not proceed to Phase 6 if they differ.

Variant C — Multi-volume tar (file data + SSH host keys, proven with Forgejo):

When a service has a data volume containing binary or structured data (e.g., git repos, SSH host keys, app config) alongside a database, tar the entire named Docker volume and extract it into a freshly created target volume:

# On source: tar the named volume via a throwaway container
ssh root@100.64.0.1 'docker run --rm -v <volume-name>:/source -v /tmp:/target alpine \
  tar -czf /target/<volume-name>.tar.gz -C /source .'

# Transfer
scp root@100.64.0.1:/tmp/<volume-name>.tar.gz /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<volume-name>.tar.gz /tmp/<volume-name>.tar.gz"

# On target: create volume + extract
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  docker volume create <volume-name>
  docker run --rm -v <volume-name>:/target -v /tmp:/source alpine \
    tar -xzf /source/<volume-name>.tar.gz -C /target
'"

SSH host keys MUST travel in the data volume (see G11) so clients see no key-change warning.

Rollback: destroy CT + ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Also transferred rsa_key.pem byte-for-byte (JWT signing key — see G9). Did NOT transfer db.sqlite3-wal, db.sqlite3-shm, tmp/.

Service-specific (example: Forgejo): Used Variant B (PostgreSQL) + Variant C (forgejo-data volume with git repos + SSH host keys). Stopped only the forgejo app container; kept forgejo-db running for dump. Integrity gate: SELECT COUNT(*) FROM repository == 9 source and target. SSH host keys in forgejo-data volume transferred intact — no client warning on reconnect.

---

Phase 6 — Start + health gate [G/S]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"

All must pass before cutover:

1. Data integrity check (if applicable, e.g. sqlite3 db.sqlite3 "PRAGMA integrity_check" == ok)
2. Service-specific startup log checks (no error markers)
3. /alive == 200 over tailnet from Contabo (the HARD composed-path gate):

   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   

   Must return 200 — do not proceed to Phase 7 if it doesn't.

Rollback: destroy CT + restart source on Contabo.

Service-specific (example: Vaultwarden): Also checked that container logs did NOT contain generating new RSA key (confirms rsa_key.pem loaded — G9). Checked SELECT count(*) FROM users == 1.

---

Phase 7 — Cutover [G]

On Contabo, take a backup first:

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'

The ONE edit: replace only the upstream token in the <service>.echo6.co block — 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT>. No scheme prefix (bare host:port defaults to HTTP). Preserve all other directives (e.g. header_up X-Real-IP {remote_host}).

Validate, then restart (G10 — caddy reload fails under admin off):

ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'

Blast radius (G10): systemctl restart caddy briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

Rollback (~1–2 s, public DNS never moved):

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Final block:

vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}

---

Phase 7a — Non-Caddy public TCP port (iptables DNAT) [S] (Forgejo SSH variant)

When a service exposes a raw TCP port that is NOT handled by Caddy (e.g., Forgejo git SSH on port 2222), the one-token Caddy edit in Phase 7 is insufficient. Use a reboot-persistent iptables DNAT systemd unit on the Contabo host to forward the port to the target tailnet IP.

Do NOT apply by rebooting the Contabo host — it runs production services. Create the unit directly and start it without a reboot.

Template unit (/etc/systemd/system/forgejo-ssh-dnat.service — parameterize PORT and TARGET_TS_IP):

[Unit]
Description=Forgejo SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target

Deploy and enable:

# Write unit to Contabo
ssh root@100.64.0.1 'cat > /etc/systemd/system/<service>-ssh-dnat.service' << 'EOF'
[Unit]
Description=<Service> SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
EOF

ssh root@100.64.0.1 'systemctl daemon-reload && systemctl enable --now <service>-ssh-dnat.service'

Verify DNAT is live (before restarting caddy):

ssh -p <PORT> git@5.189.158.149  # Should land on target service, not source

Rollback:

ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /etc/systemd/system/<service>-ssh-dnat.service && systemctl daemon-reload'

Service-specific (example: Forgejo): PORT=2222, TARGET_TS_IP=100.64.0.34. Unit: forgejo-ssh-dnat.service. Verified: ssh -p 2222 git@forge.echo6.co arrived at edge2 CT 103 with preserved host keys (no client warning).

---

Phase 8 — End-to-end verification [G/S]

• <service>.echo6.co DNS still resolves to 5.189.158.149 (unchanged)
• curl -I https://<service>.echo6.co → 200, valid cert
• Functional login + core workflow test
• Reboot survival: ssh edge2 'sudo pct reboot <CTID>' → CT comes back, Docker + container auto-start (--onboot 1 + restart: unless-stopped), Tailscale reconnects (DNS-bootstrap drop-in), /alive 200 [G8]
• Rollback: Phase 7 revert.

Service-specific (example: Vaultwarden): Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

Phase 9 — Decommission (deferred) [G]

• Source stays installed + stopped as cold rollback during soak period
• Before eventual deletion: tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>
• Update .ref docs (services.md, environment.md, ip-allocation.md)
• Retire or banner any stale archive docs
• Promote migration plan to this runbook (done)

---

Gotchas

ID	Issue
G1	Stale docs may show sub-routes (e.g. :3012/notifications/hub) that don't exist in the live Caddyfile. Always read from the live host in Phase 0.
G2	edge2 vmbr0 has no DHCP and no local-lvm → must use static IP + local (dir) storage.
G3	Source binds 127.0.0.1:<PORT>:<CONTAINER_PORT>; LXC must bind <SVC_TS_IP>:<PORT>:<CONTAINER_PORT>. Keep the container-side port unchanged.
G4	SQLite in WAL mode has a live .wal file — never copy db.sqlite3 + -wal naively. Use sqlite3 .backup (produces one self-contained file). Stop the source first for zero-loss.
G5	Preserve all existing Caddy directives on the backend block (e.g. header_up X-Real-IP {remote_host}) — edit only the upstream token.
G6	SSO redirect URIs derive from the unchanged DOMAIN/DNS — do not touch Authentik. Verify the round-trip lands logged-in; don't assert a specific redirect path.
G7	Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port.
G8	Reboot survival requires all three: --onboot 1, restart: unless-stopped, and the Tailscale DNS-bootstrap drop-in.
G9	JWT signing keys (e.g. rsa_key.pem for Vaultwarden) must travel byte-for-byte. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages.
G10	Contabo Caddy runs with admin off → caddy reload and systemctl reload caddy FAIL. Always use caddy validate then systemctl restart caddy. Restart drops all Contabo-fronted sites for ~1–2 s.
G11	SSH host keys must travel inside the data volume (Variant C tar). If they are missing or regenerated on the target, every git client gets a host-key-changed warning and must manually clear ~/.ssh/known_hosts. Transfer the full data volume; verify on target with docker exec <app> cat /data/ssh/forgejo.rsa.pub or equivalent before cutover.
G12	DB row-count integrity gate is mandatory for PostgreSQL migrations. Never skip it — a silent pg_restore failure (wrong role, encoding mismatch) leaves the DB empty or partially populated while pg_restore exits 0. Compare at least one key business table.
G13	iptables DNAT must be made reboot-persistent via a systemd oneshot/RemainAfterExit unit (Phase 7a). Do NOT rely on iptables-persistent packages or manual rules — they require package installs (forbidden) or do not survive all reboot paths. Do NOT apply by rebooting the Contabo host (production). Create the unit file, daemon-reload, enable --now.
N	The composed Contabo-Caddy → edge2-LXC tailnet path is unexercised for each new service. Keep the Phase 6 HTTP /alive 200 gate as a HARD pre-cutover requirement (use curl, not ICMP).

---

Template Summary

Generic phases (identical for every service): 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 7a (if service exposes non-Caddy TCP port: iptables DNAT systemd unit on Contabo) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

Service-specific phases: 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

Access pins:

• Contabo: ssh root@100.64.0.1
• edge2: ssh admin@184.174.35.153 (alias edge2) + sudo for pct/pvesm/pveam
• root@100.64.0.26 is refused on edge2

---

Last updated: 2026-06-17 — Added Forgejo pilot (PostgreSQL + multi-volume + iptables DNAT); updated Phase 5 Variants B/C, Phase 7a, Gotchas G11-G13, Template Summary