echo6-docs/vault/runbooks/lxc-service-migration.md

title	type	tags	aliases	related	updated
LXC Service Migration — Contabo → edge2	runbook	proxmox		expose-service-edge2 caddy expose-service-contabo services matrix_host	2026-07-13

LXC Service Migration — Contabo → edge2

Note on the source host references below: Contabo (100.64.0.1) was the migration source host during the 2026-06 service evacuation and was decommissioned/rebuilt as edge1 (mail-only) on 2026-06-19 — it no longer exists at that tailnet address. The ssh root@100.64.0.1 commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use ssh edge2 + sudo pct exec — never ssh root@<edge2-IP> (root SSH is refused on edge2). See edge2-access-reference. The migration pattern itself (phases, gates, rollback structure) remains valid regardless of which host is the source.

Proven pilots: Vaultwarden → edge2 CT 102 (SQLite, 2026-06-16), Forgejo → edge2 CT 103 (PostgreSQL + non-Caddy SSH port, 2026-06-16), LiveSync (CouchDB) → edge2 CT 104 (cold named-volume tar + bind-mounted config, 2026-06-16), authentik (PostgreSQL keystone) → edge2 CT 105 (SECRET_KEY-must-travel, multi-block caddy cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), Matrix stack → edge2 CT 106 (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and Headscale → edge2 CT 107 (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo caddy frontend unchanged (public dns never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

Architecture after migration:

Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)

edge2 access (always): ssh admin@184.174.35.153 (alias edge2, key ~/.ssh/contabo2_ed25519), then sudo for every pct/pvesm/pveam command. root@100.64.0.26 is refused — do not use it.

---

Phases

Phase 0 — Recon & baseline [G]

• On Contabo (ssh root@100.64.0.1): capture the verbatim <service>.echo6.co Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
• Read the live compose file + .env + data directory listing from the running host. Never use .ref/archive docs — they may be stale.
• On edge2 (ssh edge2):
  • Confirm the next free CT ID: sudo pvesh get /cluster/nextid
  • Confirm chosen IP is free: check sudo pct list and the 10.10.10.x allocation table in ip-allocation.md
  • Check for needed template: sudo pvesm list local | grep <os>. If absent: sudo pveam download local <template-volid>
  • Confirm storage free space: sudo pvesm status
• Rollback: n/a (read-only except optional template download).

Service-specific (example: Vaultwarden): greppeed for :3012/notifications/hub sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

Phase 0a — Pre-migration gate [S]

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

Service-specific (example: Vaultwarden): Verified ≥1 account has a non-empty password_hash (local login works without SSO). Recorded ADMIN_TOKEN to credentials file.

---

Phase 1 — Provision the LXC [G]

Pick right-sized resources for the service (not a copy of another CT's sizing).

ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"

Append TUN device lines to the CT config (required for Tailscale):

ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"

Verify: internet access via NAT, dns resolution.

Rollback: ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'

Service-specific (example: Vaultwarden): CT 102, hostname vaultwarden, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

Phase 2 — Docker baseline [G]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"

(nesting=1 + keyctl=1 from Phase 1 make Docker-in-LXC work.)

Rollback: destroy CT.

---

Phase 3 — Tailscale onboard [G]

Create a preauth key on the Headscale host:

ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'

Install Tailscale + DNS-bootstrap drop-in inside the CT:

ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"

Capture the tailnet IP: sudo pct exec <CTID> -- tailscale ip -4 → call it SVC_TS_IP.

Hard pre-cutover reachability gate (run now, should return 000; must return 200 after Phase 6):

ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"

Rollback: destroy CT + ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'

Service-specific (example: Vaultwarden): tailnet IP became 100.64.0.33, Headscale node id 45. Health path: /alive.

---

Phase 4 — Stage compose + config [G]

Copy compose file + config byte-for-byte from the live running host into the CT. Never copy from .ref/archive docs.

# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files

The ONE compose edit [G3]: change the ports bind address from 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT> (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

Rollback: destroy CT.

Service-specific (example: Vaultwarden): .env carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed 127.0.0.1:8086:80 → 100.64.0.33:8086:80. Confirmed no :3012 line.

---

Phase 5 — Migrate data [S]

This phase is entirely service-specific. Common patterns:

Data type	Method
Stateless (no persistent data)	Skip this phase
SQLite (WAL mode)	Stop source → sqlite3 .backup → verify integrity → transfer
PostgreSQL	Stop app only (keep DB up) → pg_dump -Fc → transfer → restore on target
Multi-volume (file data + keys)	tar the data volume → transfer → extract into target volume
File data only	rsync or tar

Variant A — Zero-loss for single-container SQLite:

# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"

Do NOT copy live -wal/-shm files alongside the snapshot.

Variant B — PostgreSQL dump/restore (proven with Forgejo):

Stop the application container only — keep the DB container running so pg_dump can connect cleanly:

# Stop app only; DB stays up
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop <app-container-name>'

# Dump to custom format (compressed, schema + data)
ssh root@100.64.0.1 'docker exec <db-container-name> pg_dump -U <db-user> -d <db-name> -Fc -f /tmp/<db-name>.dump'
ssh root@100.64.0.1 'docker cp <db-container-name>:/tmp/<db-name>.dump /tmp/<db-name>.dump'

# Transfer to cortex, then push into CT
scp root@100.64.0.1:/tmp/<db-name>.dump /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<db-name>.dump /tmp/<db-name>.dump"

# On target: start DB container only, wait for it to be healthy, then restore
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d <db-container-name>'"
# Wait ~10s for postgres to initialise
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker cp /tmp/<db-name>.dump <db-container-name>:/tmp/<db-name>.dump'"
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> pg_restore -U <db-user> -d <db-name> --no-owner /tmp/<db-name>.dump'"

Integrity gate — compare key table row counts source vs target:

# Source (still stopped app, DB still up)
ssh root@100.64.0.1 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c "SELECT COUNT(*) FROM <key-table>;"'

# Target
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c \"SELECT COUNT(*) FROM <key-table>;\"'"

Row counts must match exactly before proceeding. Do not proceed to Phase 6 if they differ.

Variant C — Multi-volume tar (file data + SSH host keys, proven with Forgejo):

When a service has a data volume containing binary or structured data (e.g., git repos, SSH host keys, app config) alongside a database, tar the entire named Docker volume and extract it into a freshly created target volume:

# On source: tar the named volume via a throwaway container
ssh root@100.64.0.1 'docker run --rm -v <volume-name>:/source -v /tmp:/target alpine \
  tar -czf /target/<volume-name>.tar.gz -C /source .'

# Transfer
scp root@100.64.0.1:/tmp/<volume-name>.tar.gz /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<volume-name>.tar.gz /tmp/<volume-name>.tar.gz"

# On target: create volume + extract
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  docker volume create <volume-name>
  docker run --rm -v <volume-name>:/target -v /tmp:/source alpine \
    tar -xzf /source/<volume-name>.tar.gz -C /target
'"

SSH host keys MUST travel in the data volume (see G11) so clients see no key-change warning.

Variant D — CouchDB named volumes + bind-mounted config files (proven with LiveSync):

CouchDB uses named Docker volumes for data (couchdb-data, provisioner-data) and bind-mounted config files (couchdb/local.ini, couchdb/local.d/ including jwt-keys.ini). Use the same cold named-volume tar as Variant C for each volume, AND copy the compose directory (with all bind-mount source paths) intact:

# Stop source entirely (CouchDB does not support online volume export safely)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Tar each named volume via throwaway container (same as Variant C)
ssh root@100.64.0.1 'docker run --rm -v couchdb-data:/source -v /tmp:/target alpine tar -czf /target/couchdb-data.tar.gz -C /source .'
ssh root@100.64.0.1 'docker run --rm -v provisioner-data:/source -v /tmp:/target alpine tar -czf /target/provisioner-data.tar.gz -C /source .'

# Copy the whole compose directory (includes bind-mounted config: local.ini, local.d/*, jwt-keys.ini)
ssh root@100.64.0.1 'tar -czf /tmp/<service>-compose-dir.tar.gz -C /opt <service>'

# Transfer + extract in CT
# (transfer tarballs to cortex, push to CT via edge2 pct push, then extract)

Critical: bind-mounted config files (e.g. jwt-keys.ini in couchdb/local.d/) MUST travel as part of the compose-dir copy alongside the named volumes. If jwt-keys.ini is missing, per-user JWT auth breaks on startup.

Rollback: destroy CT + ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Also transferred rsa_key.pem byte-for-byte (JWT signing key — see G9). Did NOT transfer db.sqlite3-wal, db.sqlite3-shm, tmp/.

Service-specific (example: Forgejo): Used Variant B (PostgreSQL) + Variant C (forgejo-data volume with git repos + SSH host keys). Stopped only the forgejo app container; kept forgejo-db running for dump. Integrity gate: SELECT COUNT(*) FROM repository == 9 source and target. SSH host keys in forgejo-data volume transferred intact — no client warning on reconnect.

Service-specific (example: LiveSync/CouchDB): Used Variant D. Stopped source entirely; tarred couchdb-data + provisioner-data named volumes; copied compose dir (incl. couchdb/local.d/jwt-keys.ini). Data: ~16 MB. Integrity gate: verified document count in cc-db + userdb-matt matched source via curl http://localhost:5984/<db> against both hosts after cutover.

---

Phase 6 — Start + health gate [G/S]

ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"

All must pass before cutover:

1. Data integrity check (if applicable, e.g. sqlite3 db.sqlite3 "PRAGMA integrity_check" == ok)
2. Service-specific startup log checks (no error markers)
3. /alive == 200 over tailnet from Contabo (the HARD composed-path gate):

   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   

   Must return 200 — do not proceed to Phase 7 if it doesn't.

Rollback: destroy CT + restart source on Contabo.

Service-specific (example: Vaultwarden): Also checked that container logs did NOT contain generating new RSA key (confirms rsa_key.pem loaded — G9). Checked SELECT count(*) FROM users == 1.

---

Phase 7 — Cutover [G]

On Contabo, take a backup first:

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'

The ONE edit: replace only the upstream token in the <service>.echo6.co block — 127.0.0.1:<PORT> → <SVC_TS_IP>:<PORT>. No scheme prefix (bare host:port defaults to HTTP). Preserve all other directives (e.g. header_up X-Real-IP {remote_host}).

Validate, then restart (G10 — caddy reload fails under admin off):

ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'

Blast radius (G10): systemctl restart caddy briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

Rollback (~1–2 s, public DNS never moved):

ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'

Service-specific (example: Vaultwarden): Final block:

vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}

Multi-token cutover example (LiveSync, 2026-06-16): LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from 127.0.0.1:598x → 100.64.0.35:598x in one edit. A third upstream in the same block — the authentik outpost at 127.0.0.1:9000 (used for forward_auth on /_provision) — was left untouched because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.

Multi-block cutover example (Authentik, 2026-06-18): Authentik appeared in 4 places across 2 site blocks — auth.echo6.co (outpost path matcher + catch-all both pointing to 127.0.0.1:9000) and notes.echo6.co (outpost path matcher + forward_auth directive both pointing to 127.0.0.1:9000). All 4 occurrences were updated to 100.64.0.36:9000 in one edit. Grep the entire Caddyfile for the service's port before cutting over — do not assume a service lives in only one block. See also G15 (dnsmasq must NOT be repointed) and G16 (SECRET_KEY must travel).

---

Phase 7a — Non-Caddy public TCP port (iptables DNAT) [S] (Forgejo SSH variant)

When a service exposes a raw TCP port that is NOT handled by Caddy (e.g., Forgejo git SSH on port 2222), the one-token Caddy edit in Phase 7 is insufficient. Use a reboot-persistent iptables DNAT systemd unit on the Contabo host to forward the port to the target tailnet IP.

Do NOT apply by rebooting the Contabo host — it runs production services. Create the unit directly and start it without a reboot.

Template unit (/etc/systemd/system/forgejo-ssh-dnat.service — parameterize PORT and TARGET_TS_IP):

[Unit]
Description=Forgejo SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target

Deploy and enable:

# Write unit to Contabo
ssh root@100.64.0.1 'cat > /etc/systemd/system/<service>-ssh-dnat.service' << 'EOF'
[Unit]
Description=<Service> SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
EOF

ssh root@100.64.0.1 'systemctl daemon-reload && systemctl enable --now <service>-ssh-dnat.service'

Verify DNAT is live (before restarting caddy):

ssh -p <PORT> git@5.189.158.149  # Should land on target service, not source

Rollback:

ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /etc/systemd/system/<service>-ssh-dnat.service && systemctl daemon-reload'

Service-specific (example: Forgejo): PORT=2222, TARGET_TS_IP=100.64.0.34. Unit: forgejo-ssh-dnat.service. Verified: ssh -p 2222 git@forge.echo6.co arrived at edge2 CT 103 with preserved host keys (no client warning).

---

Phase 8 — End-to-end verification [G/S]

• <service>.echo6.co DNS still resolves to 5.189.158.149 (unchanged)
• curl -I https://<service>.echo6.co → 200, valid cert
• Functional login + core workflow test
• Reboot survival: ssh edge2 'sudo pct reboot <CTID>' → CT comes back, Docker + container auto-start (--onboot 1 + restart: unless-stopped), Tailscale reconnects (DNS-bootstrap drop-in), /alive 200 [G8]
• Rollback: Phase 7 revert.

Service-specific (example: Vaultwarden): Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

Phase 9 — Decommission (deferred) [G]

• Source stays installed + stopped as cold rollback during soak period
• Before eventual deletion: tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>
• Update .ref docs (services.md, environment.md, ip-allocation.md)
• Retire or banner any stale archive docs
• Promote migration plan to this runbook (done)

---

---

Mail / Host-in-Place Rebuild (edge1, 2026-06-19)

This section documents the Mailcow migration pattern, which cannot use the standard Contabo→edge2 LXC runbook above. It serves as both a record of what was done and a set of hard-won lessons for future mail-server rebuilds.

Why Mailcow cannot use the standard cutover

Mailcow cannot be "moved" to a different host IP via the one-token Caddy approach:

• Raw mail ports are hardlocked to the IP. SMTP (25/465/587) and IMAP/POP3 (110/143/993/995/4190) require a real public IP — not a Caddy reverse proxy. Deliverability (PTR, SPF, DMARC) is hardlocked to 5.189.158.149. Moving the IP would require re-establishing that reputation.
• Approach: OS rebuild the same host, keep the same IP. Re-install the Contabo VPS (now edge1) with Debian 12 + Proxmox 8.4.19 in place. Mailcow runs in a privileged LXC (CT 101) on the 10.10.10.0/24 internal bridge. The host Caddy terminates HTTPS for mail/autodiscover/autoconfig, and a mailcow-dnat.service unit (systemd oneshot) does source-IP-preserving DNAT for all raw mail ports to the LXC.

Rebuild bootstrap pattern

The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:

1. Pre-stage on edge2: Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated services were already on edge2; no outage for those).
2. DNS during rebuild: echo6.co resolves via public GoDaddy DNS. During the rebuild the mail.* records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. Headscale pre-auth key: Generate before the rebuild starts (headscale preauthkeys create). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. Tailscale bootstrap DNS: The tailnet uses vpn.echo6.co for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.

Mailcow-specific lessons

ID	Lesson
M1	mail_crypt: capture the raw key BEFORE backup/restore. The dovecot mail_crypt private key (ecprivkey.pem) lives in the crypt volume at data/conf/dovecot/. A mailcow backup/restore can generate a FRESH key — all old mail becomes undecryptable. Always extract the raw ecprivkey.pem before decommissioning the source and inject it into the crypt volume on the target.
M2	Keep mail_crypt customization in extra.conf, not dovecot.conf. dovecot.conf is overwritten by mailcow updates. Custom mail_crypt config (extra key loading) belongs in data/conf/dovecot/extra.conf which is update-safe.
M3	Load BOTH keys (primary + secondary) after a restore-with-new-key. A single undecryptable message in a folder poisons SOGo's bulk header fetch for that folder — the entire folder appears blank. By loading both the original key (ecprivkey.pem) AND the fresh key generated by the restore (ecprivkey.pem.fresh-20260619) via mail_crypt_global_private_key_path (and mail_crypt_global_private_key_path2), all old and new mail decrypts correctly.
M4	Integrity gate MUST test decryption — not just counts. DKIM, mailbox counts, and alias counts all passed while all message content was key-locked (mail_crypt was using the wrong key). The integrity gate must open a mailbox, fetch a message, and confirm the body renders (non-empty).
M5	SOGo after restore: re-run bootstrap-sogo.sh if auth sources are lost. After a fresh Mailcow install + backup restore, SOGo may lose its IMAP auth sources. Run docker exec -it sogo-mailcow /bootstrap-sogo.sh to re-apply them.
M6	SOGo caches stale state in sogo_cache_folder. After fixing the mail_crypt key issue, SOGo still shows blank folders until its cache is cleared. Run: docker exec -it sogo-mailcow sogo-tool expire-cache-folder '*' (or truncate the sogo_cache_folder table in the database).
M7	Mailcow behind Caddy on a DIFFERENT host: send header_up Host {host}. Mailcow's nginx uses the Host header to build redirects. Without header_up Host {host} in the Caddy block, nginx leaks its internal address (e.g. 10.10.10.2:8453) in HTTP redirects, breaking web UI navigation.
M8	Front-door placement: co-locate Caddy with its backend CTs. edge2 serves as front door for its own CTs (vault/forge/notes/auth/matrix/element/vpn) over the local tailnet. edge1 serves as front door for its own CT (mailcow). Avoid cross-host Caddy→backend paths when co-location is an option — it eliminates one network hop and removes tailnet dependency for the mail-critical path.
M9	iptables DNAT for mail ports requires MASQUERADE or source-preserving DNAT. For mail servers, the real sender IP must reach the MTA (Postfix checks SPF/HELO). Use --to-destination with conntrack (nf_conntrack module) so the source IP is preserved end-to-end through the DNAT. Verify with Received: headers in delivered mail.
M10	Forge git-SSH DNAT on edge2 is now a permanent systemd unit (forgejo-ssh-dnat.service, enabled, reboot-persistent). No longer a workaround — it is the production path.

edge1 host-level systemd units (reboot-persistent)

Unit	Purpose
mailcow-dnat.service	oneshot: iptables DNAT for ports 25/465/587/110/143/993/995/4190 → 10.10.10.2
staticroute.service	oneshot: ip route add 10.10.10.0/24 via <bridge> (LXC subnet route)
tailscale	Tailscale on host; node contabo, IP 100.64.0.40
caddy	host Caddy: mail/autodiscover/autoconfig.echo6.co → 10.10.10.2:8453
unattended-upgrades	automatic security patching

Netfilter modules (edge1)

Required for DNAT to work across bridge to LXC. Persisted via /etc/modules-load.d/netfilter.conf:

nf_conntrack
nf_nat
xt_conntrack
xt_MASQUERADE
nf_nat_ipv4

---

Gotchas

ID	Issue
G1	Stale docs may show sub-routes (e.g. :3012/notifications/hub) that don't exist in the live Caddyfile. Always read from the live host in Phase 0.
G2	edge2 vmbr0 has no DHCP and no local-lvm → must use static IP + local (dir) storage.
G3	Source binds 127.0.0.1:<PORT>:<CONTAINER_PORT>; LXC must bind <SVC_TS_IP>:<PORT>:<CONTAINER_PORT>. Keep the container-side port unchanged.
G4	SQLite in WAL mode has a live .wal file — never copy db.sqlite3 + -wal naively. Use sqlite3 .backup (produces one self-contained file). Stop the source first for zero-loss.
G5	Preserve all existing Caddy directives on the backend block (e.g. header_up X-Real-IP {remote_host}) — edit only the upstream token.
G6	SSO redirect URIs derive from the unchanged DOMAIN/DNS — do not touch Authentik. Verify the round-trip lands logged-in; don't assert a specific redirect path.
G7	Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port.
G8	Reboot survival requires all three: --onboot 1, restart: unless-stopped, and the Tailscale DNS-bootstrap drop-in.
G9	JWT signing keys (e.g. rsa_key.pem for Vaultwarden) must travel byte-for-byte. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages.
G10	Contabo Caddy runs with admin off → caddy reload and systemctl reload caddy FAIL. Always use caddy validate then systemctl restart caddy. Restart drops all Contabo-fronted sites for ~1–2 s.
G11	SSH host keys must travel inside the data volume (Variant C tar). If they are missing or regenerated on the target, every git client gets a host-key-changed warning and must manually clear ~/.ssh/known_hosts. Transfer the full data volume; verify on target with docker exec <app> cat /data/ssh/forgejo.rsa.pub or equivalent before cutover.
G12	DB row-count integrity gate is mandatory for PostgreSQL migrations. Never skip it — a silent pg_restore failure (wrong role, encoding mismatch) leaves the DB empty or partially populated while pg_restore exits 0. Compare at least one key business table.
G13	iptables DNAT must be made reboot-persistent via a systemd oneshot/RemainAfterExit unit (Phase 7a). Do NOT rely on iptables-persistent packages or manual rules — they require package installs (forbidden) or do not survive all reboot paths. Do NOT apply by rebooting the Contabo host (production). Create the unit file, daemon-reload, enable --now.
G14	Reboot race — Docker binding to the tailnet IP can start before Tailscale is online, failing the bind and leaving the service unreachable after a reboot. Fix: create a systemd unit on the CT that runs docker compose up and has After=tailscale-online.target + Requires=tailscale-online.target (or equivalent tailscale status --wait pre-check). Alternatively, restart: unless-stopped in the compose file will cause Docker to self-heal via restarts, but the service will be unreachable for the first ~10–30 s after reboot. Verify reboot survival explicitly (Phase 8). Proven required for Authentik (CT 105, 2026-06-18).
G15	Do NOT change the dnsmasq split-DNS entry during cutover. The dnsmasq entry for <service>.echo6.co points at the Caddy/TLS host (100.64.0.1 = Contabo), NOT the backend. Only the Caddy upstream changes. Repointing dnsmasq to the backend tailnet IP would break internal HTTPS (no cert, no TLS termination). The Caddy host is always the internal DNS target; the backend IP only appears in the Caddy reverse_proxy directive.
G16	SECRET_KEY and session-signing material must travel byte-for-byte for keystone/session-bearing services (e.g. Authentik AUTHENTIK_SECRET_KEY, Vaultwarden rsa_key.pem). Carrying them verbatim means existing browser sessions survive the cutover — users drop straight in with no forced re-login. If the key is regenerated on the target, all active sessions are invalidated immediately. Confirm from startup logs that no new key was generated.
G17	Tailnet control-plane migration pattern (proven: Headscale → edge2 CT 107, 2026-06-19). The data-plane (all 39 registered nodes' WireGuard tunnels) survives a brief control-plane outage — existing node connections stay up; only new key exchanges and re-auths are blocked during the gap. Key material: noise_private.key MUST travel byte-for-byte (this is the server's WireGuard identity; losing it forces every node to re-auth). Out-of-band re-entry path: Contabo public IP 5.189.158.149 and edge2 public IP 184.174.35.153 are reachable even when the tailnet is broken — always confirm these work before cutover. Cutover MUST run inside a tmux session on a host reachable via public IP (not only via tailnet), implementing a self-verifying auto-rollback: if the target fails its health gate within a deadline, the script automatically restores the source Caddy block and restarts the source — so a dropped SSH cannot leave the backbone broken mid-cutover. Drop the public direct-bind (5.189.158.149:8084) only after Caddy-proxy reachability is confirmed.
N	The composed Contabo-Caddy → edge2-LXC tailnet path is unexercised for each new service. Keep the Phase 6 HTTP /alive 200 gate as a HARD pre-cutover requirement (use curl, not ICMP).

---

Template Summary

Generic phases (identical for every service): 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 7a (if service exposes non-Caddy TCP port: iptables DNAT systemd unit on Contabo) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

Service-specific phases: 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

Access pins:

• Contabo: ssh root@100.64.0.1
• edge2: ssh admin@184.174.35.153 (alias edge2) + sudo for pct/pvesm/pveam
• root@100.64.0.26 is refused on edge2

---

Last updated: 2026-06-19 — Added "Mail / Host-in-Place Rebuild (edge1)" section with lessons M1–M10 (mail_crypt key capture, extra.conf, dual-key load, decryption integrity gate, SOGo bootstrap + cache, Caddy Host header, front-door co-location, DNAT source-preserve, forgejo-ssh-dnat permanent); previously added Matrix pilot (CT 106), Headscale pilot (CT 107, G17), Authentik pilot (G14–G16)