echo6-docs/vault/runbooks/lxc-service-migration.md

---
title: LXC Service Migration — Contabo → edge2
type: runbook
tags:
  - proxmox
aliases: []
related:
  - [[expose-service-edge2]]
  - [[caddy]]
  - [[expose-service-contabo]]
  - [[services]]
  - [[matrix_host]]
updated: 2026-07-13
---
# LXC Service Migration — Contabo → edge2

> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on [[2026-06-19]] — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.

> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), **Matrix stack → edge2 CT 106** (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and **Headscale → edge2 CT 107** (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

## Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo [[caddy]] frontend unchanged (public [[dns]] never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

**Architecture after migration:**

```
Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)
```

**edge2 access (always):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.

---

## Phases

### Phase 0 — Recon & baseline `[G]`

- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `<service>.echo6.co` Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
- Read the live compose file + `.env` + data directory listing **from the running host**. Never use `.ref/archive` docs — they may be stale.
- On edge2 (`ssh edge2`):
  - Confirm the next free CT ID: `sudo pvesh get /cluster/nextid`
  - Confirm chosen IP is free: check `sudo pct list` and the `10.10.10.x` allocation table in `ip-allocation.md`
  - Check for needed template: `sudo pvesm list local | grep <os>`. If absent: `sudo pveam download local <template-volid>`
  - Confirm storage free space: `sudo pvesm status`
- **Rollback:** n/a (read-only except optional template download).

**Service-specific (example: Vaultwarden):** greppeed for `:3012`/`notifications/hub` sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

### Phase 0a — Pre-migration gate `[S]`

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

**Service-specific (example: Vaultwarden):** Verified ≥1 account has a non-empty `password_hash` (local login works without SSO). Recorded `ADMIN_TOKEN` to credentials file.

---

### Phase 1 — Provision the LXC `[G]`

Pick right-sized resources for the service (not a copy of another CT's sizing).

```bash
ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"
```

Append TUN device lines to the CT config (required for Tailscale):

```bash
ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"
```

Verify: internet access via NAT, [[dns]] resolution.

**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`

**Service-specific (example: Vaultwarden):** CT 102, hostname `vaultwarden`, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

### Phase 2 — Docker baseline `[G]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"
```

(`nesting=1` + `keyctl=1` from Phase 1 make Docker-in-LXC work.)

**Rollback:** destroy CT.

---

### Phase 3 — Tailscale onboard `[G]`

Create a preauth key on the Headscale host:

```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```

Install Tailscale + DNS-bootstrap drop-in inside the CT:

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"
```

Capture the tailnet IP: `sudo pct exec <CTID> -- tailscale ip -4` → call it `SVC_TS_IP`.

**Hard pre-cutover reachability gate** (run now, should return `000`; must return `200` after Phase 6):

```bash
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
```

**Rollback:** destroy CT + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`

**Service-specific (example: Vaultwarden):** tailnet IP became 100.64.0.33, Headscale node id 45. Health path: `/alive`.

---

### Phase 4 — Stage compose + config `[G]`

Copy compose file + config **byte-for-byte from the live running host** into the CT. Never copy from `.ref/archive` docs.

```bash
# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files
```

**The ONE compose edit `[G3]`:** change the `ports` bind address from `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>` (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

**Rollback:** destroy CT.

**Service-specific (example: Vaultwarden):** `.env` carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed `127.0.0.1:8086:80` → `100.64.0.33:8086:80`. Confirmed no `:3012` line.

---

### Phase 5 — Migrate data `[S]`

This phase is entirely service-specific. Common patterns:

| Data type | Method |
|-----------|--------|
| Stateless (no persistent data) | Skip this phase |
| SQLite (WAL mode) | Stop source → `sqlite3 .backup` → verify integrity → transfer |
| PostgreSQL | Stop app only (keep DB up) → `pg_dump -Fc` → transfer → restore on target |
| Multi-volume (file data + keys) | `tar` the data volume → transfer → extract into target volume |
| File data only | `rsync` or `tar` |

**Variant A — Zero-loss for single-container SQLite:**
```bash
# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"
```

Do NOT copy live `-wal`/`-shm` files alongside the snapshot.

**Variant B — PostgreSQL dump/restore (proven with Forgejo):**

Stop the application container only — keep the DB container running so `pg_dump` can connect cleanly:

```bash
# Stop app only; DB stays up
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop <app-container-name>'

# Dump to custom format (compressed, schema + data)
ssh root@100.64.0.1 'docker exec <db-container-name> pg_dump -U <db-user> -d <db-name> -Fc -f /tmp/<db-name>.dump'
ssh root@100.64.0.1 'docker cp <db-container-name>:/tmp/<db-name>.dump /tmp/<db-name>.dump'

# Transfer to cortex, then push into CT
scp root@100.64.0.1:/tmp/<db-name>.dump /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<db-name>.dump /tmp/<db-name>.dump"

# On target: start DB container only, wait for it to be healthy, then restore
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d <db-container-name>'"
# Wait ~10s for postgres to initialise
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker cp /tmp/<db-name>.dump <db-container-name>:/tmp/<db-name>.dump'"
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> pg_restore -U <db-user> -d <db-name> --no-owner /tmp/<db-name>.dump'"
```

**Integrity gate — compare key table row counts source vs target:**

```bash
# Source (still stopped app, DB still up)
ssh root@100.64.0.1 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c "SELECT COUNT(*) FROM <key-table>;"'

# Target
ssh edge2 "sudo pct exec <CTID> -- bash -c 'docker exec <db-container-name> psql -U <db-user> -d <db-name> -c \"SELECT COUNT(*) FROM <key-table>;\"'"
```

Row counts must match exactly before proceeding. Do not proceed to Phase 6 if they differ.

**Variant C — Multi-volume tar (file data + SSH host keys, proven with Forgejo):**

When a service has a data volume containing binary or structured data (e.g., git repos, SSH host keys, app config) alongside a database, tar the entire named Docker volume and extract it into a freshly created target volume:

```bash
# On source: tar the named volume via a throwaway container
ssh root@100.64.0.1 'docker run --rm -v <volume-name>:/source -v /tmp:/target alpine \
  tar -czf /target/<volume-name>.tar.gz -C /source .'

# Transfer
scp root@100.64.0.1:/tmp/<volume-name>.tar.gz /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/<volume-name>.tar.gz /tmp/<volume-name>.tar.gz"

# On target: create volume + extract
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  docker volume create <volume-name>
  docker run --rm -v <volume-name>:/target -v /tmp:/source alpine \
    tar -xzf /source/<volume-name>.tar.gz -C /target
'"
```

SSH host keys MUST travel in the data volume (see G11) so clients see no key-change warning.

**Variant D — CouchDB named volumes + bind-mounted config files (proven with LiveSync):**

CouchDB uses named Docker volumes for data (`couchdb-data`, `provisioner-data`) and bind-mounted config files (`couchdb/local.ini`, `couchdb/local.d/` including `jwt-keys.ini`). Use the same cold named-volume tar as Variant C for each volume, AND copy the compose directory (with all bind-mount source paths) intact:

```bash
# Stop source entirely (CouchDB does not support online volume export safely)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Tar each named volume via throwaway container (same as Variant C)
ssh root@100.64.0.1 'docker run --rm -v couchdb-data:/source -v /tmp:/target alpine tar -czf /target/couchdb-data.tar.gz -C /source .'
ssh root@100.64.0.1 'docker run --rm -v provisioner-data:/source -v /tmp:/target alpine tar -czf /target/provisioner-data.tar.gz -C /source .'

# Copy the whole compose directory (includes bind-mounted config: local.ini, local.d/*, jwt-keys.ini)
ssh root@100.64.0.1 'tar -czf /tmp/<service>-compose-dir.tar.gz -C /opt <service>'

# Transfer + extract in CT
# (transfer tarballs to cortex, push to CT via edge2 pct push, then extract)
```

**Critical:** bind-mounted config files (e.g. `jwt-keys.ini` in `couchdb/local.d/`) MUST travel as part of the compose-dir copy alongside the named volumes. If `jwt-keys.ini` is missing, per-user JWT auth breaks on startup.

**Rollback:** destroy CT + `ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'`

**Service-specific (example: Vaultwarden):** Also transferred `rsa_key.pem` byte-for-byte (JWT signing key — see G9). Did NOT transfer `db.sqlite3-wal`, `db.sqlite3-shm`, `tmp/`.

**Service-specific (example: Forgejo):** Used Variant B (PostgreSQL) + Variant C (forgejo-data volume with git repos + SSH host keys). Stopped only the `forgejo` app container; kept `forgejo-db` running for dump. Integrity gate: `SELECT COUNT(*) FROM repository` == 9 source and target. SSH host keys in forgejo-data volume transferred intact — no client warning on reconnect.

**Service-specific (example: LiveSync/CouchDB):** Used Variant D. Stopped source entirely; tarred `couchdb-data` + `provisioner-data` named volumes; copied compose dir (incl. `couchdb/local.d/jwt-keys.ini`). Data: ~16 MB. Integrity gate: verified document count in `cc-db` + `userdb-matt` matched source via `curl http://localhost:5984/<db>` against both hosts after cutover.

---

### Phase 6 — Start + health gate `[G/S]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"
```

All must pass before cutover:

1. **Data integrity check** (if applicable, e.g. `sqlite3 db.sqlite3 "PRAGMA integrity_check"` == `ok`)
2. **Service-specific startup log checks** (no error markers)
3. **`/alive` == 200 over tailnet from Contabo** (the HARD composed-path gate):
   ```bash
   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   ```
   Must return `200` — do not proceed to Phase 7 if it doesn't.

**Rollback:** destroy CT + restart source on Contabo.

**Service-specific (example: Vaultwarden):** Also checked that container logs did NOT contain `generating new RSA key` (confirms `rsa_key.pem` loaded — G9). Checked `SELECT count(*) FROM users` == 1.

---

### Phase 7 — Cutover `[G]`

On Contabo, take a backup first:

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'
```

**The ONE edit:** replace only the upstream token in the `<service>.echo6.co` block — `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>`. No scheme prefix (bare `host:port` defaults to HTTP). Preserve all other directives (e.g. `header_up X-Real-IP {remote_host}`).

Validate, then restart (G10 — `caddy reload` fails under `admin off`):

```bash
ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'
```

**Blast radius (G10):** `systemctl restart caddy` briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

**Rollback (~1–2 s, public DNS never moved):**

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'
```

**Service-specific (example: Vaultwarden):** Final block:
```caddyfile
vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}
```

**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x` → `100.64.0.35:598x` in one edit. A third upstream in the same block — the [[authentik]] outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.

**Multi-block cutover example (Authentik, 2026-06-18):** Authentik appeared in 4 places across 2 site blocks — `auth.echo6.co` (outpost path matcher + catch-all both pointing to `127.0.0.1:9000`) and `notes.echo6.co` (outpost path matcher + `forward_auth` directive both pointing to `127.0.0.1:9000`). All 4 occurrences were updated to `100.64.0.36:9000` in one edit. Grep the entire Caddyfile for the service's port before cutting over — do not assume a service lives in only one block. See also G15 (dnsmasq must NOT be repointed) and G16 (SECRET_KEY must travel).

---

#### Phase 7a — Non-Caddy public TCP port (iptables DNAT) `[S]` *(Forgejo SSH variant)*

When a service exposes a raw TCP port that is NOT handled by Caddy (e.g., Forgejo git SSH on port 2222), the one-token Caddy edit in Phase 7 is insufficient. Use a reboot-persistent **iptables DNAT systemd unit** on the Contabo host to forward the port to the target tailnet IP.

**Do NOT apply by rebooting the Contabo host** — it runs production services. Create the unit directly and start it without a reboot.

Template unit (`/etc/systemd/system/forgejo-ssh-dnat.service` — parameterize PORT and TARGET_TS_IP):

```ini
[Unit]
Description=Forgejo SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
```

Deploy and enable:

```bash
# Write unit to Contabo
ssh root@100.64.0.1 'cat > /etc/systemd/system/<service>-ssh-dnat.service' << 'EOF'
[Unit]
Description=<Service> SSH DNAT — forward :<PORT> to edge2 LXC
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iptables -t nat -A PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStart=/sbin/iptables -t nat -A OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>
ExecStop=/sbin/iptables -t nat -D OUTPUT -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_TS_IP>:<PORT>

[Install]
WantedBy=multi-user.target
EOF

ssh root@100.64.0.1 'systemctl daemon-reload && systemctl enable --now <service>-ssh-dnat.service'
```

Verify DNAT is live (before restarting caddy):

```bash
ssh -p <PORT> git@5.189.158.149  # Should land on target service, not source
```

**Rollback:**

```bash
ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /etc/systemd/system/<service>-ssh-dnat.service && systemctl daemon-reload'
```

**Service-specific (example: Forgejo):** PORT=2222, TARGET_TS_IP=100.64.0.34. Unit: `forgejo-ssh-dnat.service`. Verified: `ssh -p 2222 git@forge.echo6.co` arrived at edge2 CT 103 with preserved host keys (no client warning).

---

### Phase 8 — End-to-end verification `[G/S]`

- `<service>.echo6.co` DNS still resolves to `5.189.158.149` (unchanged)
- `curl -I https://<service>.echo6.co` → 200, valid cert
- Functional login + core workflow test
- **Reboot survival:** `ssh edge2 'sudo pct reboot <CTID>'` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), Tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 `[G8]`
- **Rollback:** Phase 7 revert.

**Service-specific (example: Vaultwarden):** Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

### Phase 9 — Decommission (deferred) `[G]`

- Source stays **installed + stopped** as cold rollback during soak period
- Before eventual deletion: `tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>`
- Update `.ref` docs (services.md, environment.md, ip-allocation.md)
- Retire or banner any stale archive docs
- Promote migration plan to this runbook (done)

---

---

## Mail / Host-in-Place Rebuild (edge1, 2026-06-19)

> This section documents the Mailcow migration pattern, which cannot use the standard Contabo→edge2 LXC runbook above. It serves as both a record of what was done and a set of hard-won lessons for future mail-server rebuilds.

### Why Mailcow cannot use the standard cutover

Mailcow cannot be "moved" to a different host IP via the one-token Caddy approach:

- **Raw mail ports are hardlocked to the IP.** SMTP (25/465/587) and IMAP/POP3 (110/143/993/995/4190) require a real public IP — not a Caddy reverse proxy. Deliverability (PTR, SPF, DMARC) is hardlocked to `5.189.158.149`. Moving the IP would require re-establishing that reputation.
- **Approach: OS rebuild the same host, keep the same IP.** Re-install the Contabo VPS (now edge1) with Debian 12 + Proxmox 8.4.19 in place. Mailcow runs in a **privileged LXC (CT 101)** on the `10.10.10.0/24` internal bridge. The host Caddy terminates HTTPS for mail/autodiscover/autoconfig, and a `mailcow-dnat.service` unit (systemd oneshot) does source-IP-preserving DNAT for all raw mail ports to the LXC.

### Rebuild bootstrap pattern

The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:

1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated [[services]] were already on edge2; no outage for those).
2. **DNS during rebuild:** echo6.co resolves via public GoDaddy DNS. During the rebuild the `mail.*` records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. **Headscale pre-auth key:** Generate before the rebuild starts (`headscale preauthkeys create`). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. **Tailscale bootstrap DNS:** The tailnet uses `vpn.echo6.co` for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.

### Mailcow-specific lessons

| ID | Lesson |
|----|--------|
| M1 | **mail_crypt: capture the raw key BEFORE backup/restore.** The dovecot mail_crypt private key (`ecprivkey.pem`) lives in the crypt volume at `data/conf/dovecot/`. A mailcow backup/restore can generate a FRESH key — all old mail becomes undecryptable. Always extract the raw `ecprivkey.pem` before decommissioning the source and inject it into the crypt volume on the target. |
| M2 | **Keep mail_crypt customization in `extra.conf`, not `dovecot.conf`.** `dovecot.conf` is overwritten by mailcow updates. Custom mail_crypt config (extra key loading) belongs in `data/conf/dovecot/extra.conf` which is update-safe. |
| M3 | **Load BOTH keys (primary + secondary) after a restore-with-new-key.** A single undecryptable message in a folder poisons SOGo's bulk header fetch for that folder — the entire folder appears blank. By loading both the original key (`ecprivkey.pem`) AND the fresh key generated by the restore (`ecprivkey.pem.fresh-20260619`) via `mail_crypt_global_private_key_path` (and `mail_crypt_global_private_key_path2`), all old and new mail decrypts correctly. |
| M4 | **Integrity gate MUST test decryption — not just counts.** DKIM, mailbox counts, and alias counts all passed while all message content was key-locked (mail_crypt was using the wrong key). The integrity gate must open a mailbox, fetch a message, and confirm the body renders (non-empty). |
| M5 | **SOGo after restore: re-run `bootstrap-sogo.sh` if auth sources are lost.** After a fresh Mailcow install + backup restore, SOGo may lose its IMAP auth sources. Run `docker exec -it sogo-mailcow /bootstrap-sogo.sh` to re-apply them. |
| M6 | **SOGo caches stale state in `sogo_cache_folder`.** After fixing the mail_crypt key issue, SOGo still shows blank folders until its cache is cleared. Run: `docker exec -it sogo-mailcow sogo-tool expire-cache-folder '*'` (or truncate the `sogo_cache_folder` table in the database). |
| M7 | **Mailcow behind Caddy on a DIFFERENT host: send `header_up Host {host}`.** Mailcow's nginx uses the `Host` header to build redirects. Without `header_up Host {host}` in the Caddy block, nginx leaks its internal address (e.g. `10.10.10.2:8453`) in HTTP redirects, breaking web UI navigation. |
| M8 | **Front-door placement: co-locate Caddy with its backend CTs.** edge2 serves as front door for its own CTs (vault/forge/notes/auth/matrix/element/vpn) over the local tailnet. edge1 serves as front door for its own CT (mailcow). Avoid cross-host Caddy→backend paths when co-location is an option — it eliminates one network hop and removes tailnet dependency for the mail-critical path. |
| M9 | **iptables DNAT for mail ports requires `MASQUERADE` or source-preserving DNAT.** For mail servers, the real sender IP must reach the MTA (Postfix checks SPF/HELO). Use `--to-destination` with conntrack (`nf_conntrack` module) so the source IP is preserved end-to-end through the DNAT. Verify with `Received:` headers in delivered mail. |
| M10 | **Forge git-SSH DNAT on edge2 is now a permanent systemd unit** (`forgejo-ssh-dnat.service`, enabled, reboot-persistent). No longer a workaround — it is the production path. |

### edge1 host-level systemd units (reboot-persistent)

| Unit | Purpose |
|------|---------|
| `mailcow-dnat.service` | oneshot: iptables DNAT for ports 25/465/587/110/143/993/995/4190 → 10.10.10.2 |
| `staticroute.service` | oneshot: `ip route add 10.10.10.0/24 via <bridge>` (LXC subnet route) |
| `tailscale` | Tailscale on host; node `contabo`, IP 100.64.0.40 |
| `caddy` | host Caddy: mail/autodiscover/autoconfig.echo6.co → `10.10.10.2:8453` |
| `unattended-upgrades` | automatic security patching |

### Netfilter modules (edge1)

Required for DNAT to work across bridge to LXC. Persisted via `/etc/modules-load.d/netfilter.conf`:

```
nf_conntrack
nf_nat
xt_conntrack
xt_MASQUERADE
nf_nat_ipv4
```

---

## Gotchas

| ID | Issue |
|----|-------|
| G1 | Stale docs may show sub-routes (e.g. `:3012`/`notifications/hub`) that don't exist in the live Caddyfile. Always read from the live host in Phase 0. |
| G2 | edge2 `vmbr0` has **no DHCP** and no `local-lvm` → must use static IP + `local` (dir) storage. |
| G3 | Source binds `127.0.0.1:<PORT>:<CONTAINER_PORT>`; LXC must bind `<SVC_TS_IP>:<PORT>:<CONTAINER_PORT>`. Keep the container-side port unchanged. |
| G4 | SQLite in WAL mode has a live `.wal` file — never copy `db.sqlite3` + `-wal` naively. Use `sqlite3 .backup` (produces one self-contained file). Stop the source first for zero-loss. |
| G5 | Preserve all existing Caddy directives on the backend block (e.g. `header_up X-Real-IP {remote_host}`) — edit only the upstream token. |
| G6 | SSO redirect URIs derive from the unchanged `DOMAIN`/DNS — **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path. |
| G7 | Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port. |
| G8 | Reboot survival requires all three: `--onboot 1`, `restart: unless-stopped`, and the Tailscale DNS-bootstrap drop-in. |
| G9 | JWT signing keys (e.g. `rsa_key.pem` for Vaultwarden) **must travel byte-for-byte**. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages. |
| G10 | Contabo Caddy runs with `admin off` → `caddy reload` and `systemctl reload caddy` FAIL. Always use `caddy validate` then `systemctl restart caddy`. Restart drops all Contabo-fronted sites for ~1–2 s. |
| G11 | SSH host keys must travel inside the data volume (Variant C tar). If they are missing or regenerated on the target, every git client gets a host-key-changed warning and must manually clear `~/.ssh/known_hosts`. Transfer the full data volume; verify on target with `docker exec <app> cat /data/ssh/forgejo.rsa.pub` or equivalent before cutover. |
| G12 | DB row-count integrity gate is mandatory for PostgreSQL migrations. Never skip it — a silent pg_restore failure (wrong role, encoding mismatch) leaves the DB empty or partially populated while `pg_restore` exits 0. Compare at least one key business table. |
| G13 | iptables DNAT must be made reboot-persistent via a systemd `oneshot`/`RemainAfterExit` unit (Phase 7a). Do NOT rely on iptables-persistent packages or manual rules — they require package installs (forbidden) or do not survive all reboot paths. Do NOT apply by rebooting the Contabo host (production). Create the unit file, `daemon-reload`, `enable --now`. |
| G14 | **Reboot race — Docker binding to the tailnet IP can start before Tailscale is online, failing the bind and leaving the service unreachable after a reboot.** Fix: create a systemd unit on the CT that runs `docker compose up` and has `After=tailscale-online.target` + `Requires=tailscale-online.target` (or equivalent `tailscale status --wait` pre-check). Alternatively, `restart: unless-stopped` in the compose file will cause Docker to self-heal via restarts, but the service will be unreachable for the first ~10–30 s after reboot. Verify reboot survival explicitly (Phase 8). Proven required for Authentik (CT 105, 2026-06-18). |
| G15 | **Do NOT change the dnsmasq split-DNS entry during cutover.** The dnsmasq entry for `<service>.echo6.co` points at the Caddy/TLS host (100.64.0.1 = Contabo), NOT the backend. Only the Caddy upstream changes. Repointing dnsmasq to the backend tailnet IP would break internal HTTPS (no cert, no TLS termination). The Caddy host is always the internal DNS target; the backend IP only appears in the Caddy `reverse_proxy` directive. |
| G16 | **SECRET_KEY and session-signing material must travel byte-for-byte for keystone/session-bearing [[services]]** (e.g. Authentik `AUTHENTIK_SECRET_KEY`, Vaultwarden `rsa_key.pem`). Carrying them verbatim means existing browser sessions survive the cutover — users drop straight in with no forced re-login. If the key is regenerated on the target, all active sessions are invalidated immediately. Confirm from startup logs that no new key was generated. |
| G17 | **Tailnet control-plane migration pattern (proven: Headscale → edge2 CT 107, 2026-06-19).** The data-plane (all 39 registered nodes' WireGuard tunnels) survives a brief control-plane outage — existing node connections stay up; only new key exchanges and re-auths are blocked during the gap. Key material: `noise_private.key` MUST travel byte-for-byte (this is the server's WireGuard identity; losing it forces every node to re-auth). Out-of-band re-entry path: Contabo public IP `5.189.158.149` and edge2 public IP `184.174.35.153` are reachable even when the tailnet is broken — always confirm these work before cutover. **Cutover MUST run inside a `tmux` session on a host reachable via public IP** (not only via tailnet), implementing a self-verifying auto-rollback: if the target fails its health gate within a deadline, the script automatically restores the source Caddy block and restarts the source — so a dropped SSH cannot leave the backbone broken mid-cutover. Drop the public direct-bind (`5.189.158.149:8084`) only after Caddy-proxy reachability is confirmed. |
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |

---

## Template Summary

**Generic phases (identical for every service):** 0 ([[recon]]) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 7a (if service exposes non-Caddy TCP port: iptables DNAT systemd unit on Contabo) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

**Service-specific phases:** 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

**Access pins:**
- Contabo: `ssh root@100.64.0.1`
- edge2: `ssh admin@184.174.35.153` (alias `edge2`) + `sudo` for pct/pvesm/pveam
- `root@100.64.0.26` is refused on edge2

---

*Last updated: 2026-06-19 — Added "Mail / Host-in-Place Rebuild (edge1)" section with lessons M1–M10 (mail_crypt key capture, extra.conf, dual-key load, decryption integrity gate, SOGo bootstrap + cache, Caddy Host header, front-door co-location, DNAT source-preserve, forgejo-ssh-dnat permanent); previously added Matrix pilot (CT 106), Headscale pilot (CT 107, G17), Authentik pilot (G14–G16)*