matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Contabo rebuilt as edge1 (mail-only); edge2 = front door for all other services

Browse source 
- edge1 (5.189.158.149): Debian 12 + Proxmox 8.4.19, mail-only; Mailcow in CT 101 (10.10.10.2); mail-port DNAT; mail_crypt key in extra.conf; tailnet node contabo now 100.64.0.40
- edge2 (184.174.35.153): permanent front door for vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh/intermountainmesh; forgejo-ssh-dnat.service is permanent systemd unit on edge2
- runbook: mail/host-in-place rebuild (M1-M10) + mail_crypt capture + dual-key load + decryption-gate + SOGo bootstrap/cache + reverse-proxy-Host + front-door co-location lessons
- credentials: edge1 SSH access note; Mailcow CT 101 location; mail_crypt key paths
- DNS split documented: mail/* -> edge1 (5.189.158.149); all other services -> edge2 (184.174.35.153)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-19 15:24:59 +00:00
commit 5236f3ba4c
5 changed files with 169 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

63
vault/runbooks/lxc-service-migration.md
View file

@ -423,6 +423,67 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
---
---
## Mail / Host-in-Place Rebuild (edge1, 2026-06-19)
> This section documents the Mailcow migration pattern, which cannot use the standard Contabo→edge2 LXC runbook above. It serves as both a record of what was done and a set of hard-won lessons for future mail-server rebuilds.
### Why Mailcow cannot use the standard cutover
Mailcow cannot be "moved" to a different host IP via the one-token Caddy approach:
- **Raw mail ports are hardlocked to the IP.** SMTP (25/465/587) and IMAP/POP3 (110/143/993/995/4190) require a real public IP — not a Caddy reverse proxy. Deliverability (PTR, SPF, DMARC) is hardlocked to `5.189.158.149`. Moving the IP would require re-establishing that reputation.
- **Approach: OS rebuild the same host, keep the same IP.** Re-install the Contabo VPS (now edge1) with Debian 12 + Proxmox 8.4.19 in place. Mailcow runs in a **privileged LXC (CT 101)** on the `10.10.10.0/24` internal bridge. The host Caddy terminates HTTPS for mail/autodiscover/autoconfig, and a `mailcow-dnat.service` unit (systemd oneshot) does source-IP-preserving DNAT for all raw mail ports to the LXC.
### Rebuild bootstrap pattern
The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated services were already on edge2; no outage for those).
2. **DNS during rebuild:** echo6.co resolves via public GoDaddy DNS. During the rebuild the `mail.*` records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. **Headscale pre-auth key:** Generate before the rebuild starts (`headscale preauthkeys create`). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. **Tailscale bootstrap DNS:** The tailnet uses `vpn.echo6.co` for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.
### Mailcow-specific lessons
| ID | Lesson |
|----|--------|
| M1 | **mail_crypt: capture the raw key BEFORE backup/restore.** The dovecot mail_crypt private key (`ecprivkey.pem`) lives in the crypt volume at `data/conf/dovecot/`. A mailcow backup/restore can generate a FRESH key — all old mail becomes undecryptable. Always extract the raw `ecprivkey.pem` before decommissioning the source and inject it into the crypt volume on the target. |
| M2 | **Keep mail_crypt customization in `extra.conf`, not `dovecot.conf`.** `dovecot.conf` is overwritten by mailcow updates. Custom mail_crypt config (extra key loading) belongs in `data/conf/dovecot/extra.conf` which is update-safe. |
| M3 | **Load BOTH keys (primary + secondary) after a restore-with-new-key.** A single undecryptable message in a folder poisons SOGo's bulk header fetch for that folder — the entire folder appears blank. By loading both the original key (`ecprivkey.pem`) AND the fresh key generated by the restore (`ecprivkey.pem.fresh-20260619`) via `mail_crypt_global_private_key_path` (and `mail_crypt_global_private_key_path2`), all old and new mail decrypts correctly. |
| M4 | **Integrity gate MUST test decryption — not just counts.** DKIM, mailbox counts, and alias counts all passed while all message content was key-locked (mail_crypt was using the wrong key). The integrity gate must open a mailbox, fetch a message, and confirm the body renders (non-empty). |
| M5 | **SOGo after restore: re-run `bootstrap-sogo.sh` if auth sources are lost.** After a fresh Mailcow install + backup restore, SOGo may lose its IMAP auth sources. Run `docker exec -it sogo-mailcow /bootstrap-sogo.sh` to re-apply them. |
| M6 | **SOGo caches stale state in `sogo_cache_folder`.** After fixing the mail_crypt key issue, SOGo still shows blank folders until its cache is cleared. Run: `docker exec -it sogo-mailcow sogo-tool expire-cache-folder '*'` (or truncate the `sogo_cache_folder` table in the database). |
| M7 | **Mailcow behind Caddy on a DIFFERENT host: send `header_up Host {host}`.** Mailcow's nginx uses the `Host` header to build redirects. Without `header_up Host {host}` in the Caddy block, nginx leaks its internal address (e.g. `10.10.10.2:8453`) in HTTP redirects, breaking web UI navigation. |
| M8 | **Front-door placement: co-locate Caddy with its backend CTs.** edge2 serves as front door for its own CTs (vault/forge/notes/auth/matrix/element/vpn) over the local tailnet. edge1 serves as front door for its own CT (mailcow). Avoid cross-host Caddy→backend paths when co-location is an option — it eliminates one network hop and removes tailnet dependency for the mail-critical path. |
| M9 | **iptables DNAT for mail ports requires `MASQUERADE` or source-preserving DNAT.** For mail servers, the real sender IP must reach the MTA (Postfix checks SPF/HELO). Use `--to-destination` with conntrack (`nf_conntrack` module) so the source IP is preserved end-to-end through the DNAT. Verify with `Received:` headers in delivered mail. |
| M10 | **Forge git-SSH DNAT on edge2 is now a permanent systemd unit** (`forgejo-ssh-dnat.service`, enabled, reboot-persistent). No longer a workaround — it is the production path. |
### edge1 host-level systemd units (reboot-persistent)
| Unit | Purpose |
|------|---------|
| `mailcow-dnat.service` | oneshot: iptables DNAT for ports 25/465/587/110/143/993/995/4190 → 10.10.10.2 |
| `staticroute.service` | oneshot: `ip route add 10.10.10.0/24 via <bridge>` (LXC subnet route) |
| `tailscale` | Tailscale on host; node `contabo`, IP 100.64.0.40 |
| `caddy` | host Caddy: mail/autodiscover/autoconfig.echo6.co → `10.10.10.2:8453` |
| `unattended-upgrades` | automatic security patching |
### Netfilter modules (edge1)
Required for DNAT to work across bridge to LXC. Persisted via `/etc/modules-load.d/netfilter.conf`:
```
nf_conntrack
nf_nat
xt_conntrack
xt_MASQUERADE
nf_nat_ipv4
```
---
## Gotchas
| ID | Issue |
@ -461,4 +522,4 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
---
*Last updated: 2026-06-19 — Added Matrix pilot (multi-DB Postgres + stateful Signal bridge, CT 106) and Headscale pilot (tailnet control plane, noise_private.key must travel, CT 107); Gotcha G17 (tailnet-control-plane migration pattern: data-plane survives outage, out-of-band re-entry via public IPs, self-verifying auto-rollback in tmux); previously added Authentik pilot (G14–G16)*
*Last updated: 2026-06-19 — Added "Mail / Host-in-Place Rebuild (edge1)" section with lessons M1–M10 (mail_crypt key capture, extra.conf, dual-key load, decryption integrity gate, SOGo bootstrap + cache, Caddy Host header, front-door co-location, DNAT source-preserve, forgejo-ssh-dnat permanent); previously added Matrix pilot (CT 106), Headscale pilot (CT 107, G17), Authentik pilot (G14–G16)*