echo6-docs/runbooks/lxc-service-migration.md

LXC Service Migration — Contabo → edge2

Proven pilot: Vaultwarden → edge2 CT 102 (2026-06-16). This runbook generalizes that pattern into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

Architecture after migration:

Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)

edge2 access (always): ssh admin@184.174.35.153 (alias edge2, key ~/.ssh/contabo2_ed25519), then sudo for every pct/pvesm/pveam command. root@100.64.0.26 is refused — do not use it.

---

Phases

Phase 0 — Recon & baseline [G]

• On Contabo (ssh root@100.64.0.1): capture the verbatim <service>.echo6.co Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
• Read the live compose file + .env + data directory listing from the running host. Never use .ref/archive docs — they may be stale.
• On edge2 (ssh edge2):
  • Confirm the next free CT ID: sudo pvesh get /cluster/nextid
  • Confirm chosen IP is free: check sudo pct list and the 10.10.10.x allocation table in ip-allocation.md
  • Check for needed template: sudo pvesm list local | grep <os>. If absent: sudo pveam download local <template-volid>
  • Confirm storage free space: sudo pvesm status
• Rollback: n/a (read-only except optional template download).

Service-specific (example: Vaultwarden): greppeed for :3012/notifications/hub sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

Phase 0a — Pre-migration gate [S]

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

Service-specific (example: Vaultwarden): Verified ≥1 account has a non-empty password_hash (local login works without SSO). Recorded ADMIN_TOKEN to credentials file.

---

Phase 1 — Provision the LXC [G]

Pick right-sized resources for the service (not a copy of another CT's sizing).

ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"

Append TUN device lines to the CT config (required for Tailscale):

ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"

Verify: internet access via NAT, DNS resolution.

Rollback: ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'

Service-specific (example: Vaultwarden): CT 102, hostname vaultwarden, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

Phase 2 — Docker baseline [G]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"

(nesting=1 + keyctl=1 from Phase 1 make Docker-in-LXC work.)

Rollback: destroy CT.

---

Phase 3 — Tailscale onboard [G]

Create a preauth key on the Headscale host:

ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'

Install Tailscale + DNS-bootstrap drop-in inside the CT:

ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"

Capture the tailnet IP: sudo pct exec <CTID> -- tailscale ip -4 → call it SVC_TS_IP.

Hard pre-cutover reachability gate (run now, should return 000; must return 200 after Phase 6):

ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"

Rollback: destroy CT + ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'

Service-specific (example: Vaultwarden): tailnet IP became 100.64.0.33, Headscale node id 45. Health path: /alive.

---

Phase 4 — Stage compose + config [G]

Copy compose file + config byte-for-byte from the live running host into the CT. Never copy from .ref/archive docs.

# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files

The ONE compose edit [G3]: change the ports bind address from 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT> (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

Rollback: destroy CT.

Service-specific (example: Vaultwarden): .env carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed 127.0.0.1:8086:80 → 100.64.0.33:8086:80. Confirmed no :3012 line.

---

Phase 5 — Migrate data [S]

This phase is entirely service-specific. Common patterns:

Data type	Method
Stateless (no persistent data)	Skip this phase
SQLite (WAL mode)	Stop source → sqlite3 .backup → verify integrity → transfer
PostgreSQL	pg_dump → transfer → psql restore
File data	rsync or tar

Zero-loss for single-user SQLite:

# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"

Do NOT copy live -wal/-shm files alongside the snapshot.

Rollback: destroy CT + ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Also transferred rsa_key.pem byte-for-byte (JWT signing key — see G9). Did NOT transfer db.sqlite3-wal, db.sqlite3-shm, tmp/.

---

Phase 6 — Start + health gate [G/S]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"

All must pass before cutover:

1. Data integrity check (if applicable, e.g. sqlite3 db.sqlite3 "PRAGMA integrity_check" == ok)
2. Service-specific startup log checks (no error markers)
3. /alive == 200 over tailnet from Contabo (the HARD composed-path gate):

   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   

   Must return 200 — do not proceed to Phase 7 if it doesn't.

Rollback: destroy CT + restart source on Contabo.

Service-specific (example: Vaultwarden): Also checked that container logs did NOT contain generating new RSA key (confirms rsa_key.pem loaded — G9). Checked SELECT count(*) FROM users == 1.

---

Phase 7 — Cutover [G]

On Contabo, take a backup first:

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'

The ONE edit: replace only the upstream token in the <service>.echo6.co block — 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT>. No scheme prefix (bare host:port defaults to HTTP). Preserve all other directives (e.g. header_up X-Real-IP {remote_host}).

Validate, then restart (G10 — caddy reload fails under admin off):

ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'

Blast radius (G10): systemctl restart caddy briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

Rollback (~1–2 s, public DNS never moved):

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Final block:

vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}

---

Phase 8 — End-to-end verification [G/S]

• <service>.echo6.co DNS still resolves to 5.189.158.149 (unchanged)
• curl -I https://<service>.echo6.co → 200, valid cert
• Functional login + core workflow test
• Reboot survival: ssh edge2 'sudo pct reboot <CTID>' → CT comes back, Docker + container auto-start (--onboot 1 + restart: unless-stopped), Tailscale reconnects (DNS-bootstrap drop-in), /alive 200 [G8]
• Rollback: Phase 7 revert.

Service-specific (example: Vaultwarden): Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

Phase 9 — Decommission (deferred) [G]

• Source stays installed + stopped as cold rollback during soak period
• Before eventual deletion: tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>
• Update .ref docs (services.md, environment.md, ip-allocation.md)
• Retire or banner any stale archive docs
• Promote migration plan to this runbook (done)

---

Gotchas

ID	Issue
G1	Stale docs may show sub-routes (e.g. :3012/notifications/hub) that don't exist in the live Caddyfile. Always read from the live host in Phase 0.
G2	edge2 vmbr0 has no DHCP and no local-lvm → must use static IP + local (dir) storage.
G3	Source binds 127.0.0.1:<PORT>:<CONTAINER_PORT>; LXC must bind <SVC_TS_IP>:<PORT>:<CONTAINER_PORT>. Keep the container-side port unchanged.
G4	SQLite in WAL mode has a live .wal file — never copy db.sqlite3 + -wal naively. Use sqlite3 .backup (produces one self-contained file). Stop the source first for zero-loss.
G5	Preserve all existing Caddy directives on the backend block (e.g. header_up X-Real-IP {remote_host}) — edit only the upstream token.
G6	SSO redirect URIs derive from the unchanged DOMAIN/DNS — do not touch Authentik. Verify the round-trip lands logged-in; don't assert a specific redirect path.
G7	Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port.
G8	Reboot survival requires all three: --onboot 1, restart: unless-stopped, and the Tailscale DNS-bootstrap drop-in.
G9	JWT signing keys (e.g. rsa_key.pem for Vaultwarden) must travel byte-for-byte. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages.
G10	Contabo Caddy runs with admin off → caddy reload and systemctl reload caddy FAIL. Always use caddy validate then systemctl restart caddy. Restart drops all Contabo-fronted sites for ~1–2 s.
N	The composed Contabo-Caddy → edge2-LXC tailnet path is unexercised for each new service. Keep the Phase 6 HTTP /alive 200 gate as a HARD pre-cutover requirement (use curl, not ICMP).

---

Template Summary

Generic phases (identical for every service): 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

Service-specific phases: 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

Access pins:

• Contabo: ssh root@100.64.0.1
• edge2: ssh admin@184.174.35.153 (alias edge2) + sudo for pct/pvesm/pveam
• root@100.64.0.26 is refused on edge2

---

Last updated: 2026-06-17