matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Contabo rebuilt as edge1 (mail-only); edge2 = front door for all other services

Browse source 
- edge1 (5.189.158.149): Debian 12 + Proxmox 8.4.19, mail-only; Mailcow in CT 101 (10.10.10.2); mail-port DNAT; mail_crypt key in extra.conf; tailnet node contabo now 100.64.0.40
- edge2 (184.174.35.153): permanent front door for vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh/intermountainmesh; forgejo-ssh-dnat.service is permanent systemd unit on edge2
- runbook: mail/host-in-place rebuild (M1-M10) + mail_crypt capture + dual-key load + decryption-gate + SOGo bootstrap/cache + reverse-proxy-Host + front-door co-location lessons
- credentials: edge1 SSH access note; Mailcow CT 101 location; mail_crypt key paths
- DNS split documented: mail/* -> edge1 (5.189.158.149); all other services -> edge2 (184.174.35.153)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-19 15:24:59 +00:00
commit 5236f3ba4c
5 changed files with 169 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

29
credentials
View file

@ -81,12 +81,29 @@ ECHO6_PORTAL_OIDC_ISSUER=https://auth.echo6.co/application/o/echo6-portal/
# Note: NordVPN token was entered interactively, re-login if needed
NORDVPN_LOCATION="CT 110 (media)"
# Contabo Mailcow VPS SSH Access
# Server: mail (contabo-mailcow-ypesiosi)
# Tailscale IP: 100.64.0.6
# Public IP: 5.189.158.149
CONTABO_MAILCOW_SSH_USER=root
CONTABO_MAILCOW_SSH_PASS=7redditGold
# edge1 SSH Access (rebuilt Contabo VPS, mail-only, 2026-06-19)
# Server: edge1 — Debian 12 + Proxmox 8.4.19, mail-only role
# Public IP: 5.189.158.149 | Tailscale: 100.64.0.40 (node: contabo)
# SSH key: ~/.ssh/contabo2_ed25519 (same key as before rebuild)
# Box password (unrotated): 7redditGold
# ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149
EDGE1_SSH_USER=root
EDGE1_SSH_KEY=~/.ssh/contabo2_ed25519
EDGE1_SSH_PASS=7redditGold
EDGE1_PUBLIC_IP=5.189.158.149
EDGE1_TAILSCALE_IP=100.64.0.40
# Mailcow (mail.echo6.co) — now on edge1 CT 101 (10.10.10.2), updated commit 52a41b4d (SOGo 5.12.8)
# Deployed: 2026-06-19 rebuild (privileged LXC on edge1 Proxmox)
# Access: via host Caddy (mail/autodiscover/autoconfig.echo6.co) + host DNAT (ports 25/465/587/110/143/993/995/4190 → 10.10.10.2)
# Admin UI: https://mail.echo6.co (same as before)
# mail_crypt keys (dovecot crypt volume: data/conf/dovecot/):
# Primary key: ecprivkey.pem (original; must be preserved across rebuilds)
# Secondary key: ecprivkey.pem.fresh-20260619 (generated by restore; loaded alongside primary)
# Config: data/conf/dovecot/extra.conf (update-safe, not dovecot.conf)
# Both keys loaded — required to decrypt old mail encrypted with primary and new mail with secondary
MAILCOW_ADMIN_USER=admin
MAILCOW_ADMIN_PASS=7redditGold
# RECON Pipeline Email Alerts (recon@echo6.co)
# Deployed: 2026-02-01 on Media-Server for pipeline monitoring

16
vault/docs/hardware/environment.md
View file

@ -99,10 +99,10 @@ Five nodes running Proxmox VE:
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
| matt-desktop | 192.168.1.254 | 100.64.0.10 | Personal workstation (Windows, your PC) |
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, [[authentik]], Headscale, Forge, Matrix |
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB |
| **edge1** (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, **mail-only** — Mailcow in CT 101; host Caddy + mailcow-dnat.service; rebuilt 2026-06-19 |
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB — **permanent front door** for vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh/intermountainmesh |
*Last updated: 2026-06-19 — Added edge2 CT 106 (matrix, 10.10.10.24, 100.64.0.37, migrated 2026-06-18) and CT 107 (headscale, 10.10.10.25, 100.64.0.38, migrated 2026-06-19); previously added CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101, central (utility CT 104)*
*Last updated: 2026-06-19 — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other services; Headscale node `contabo` moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
## LXC Containers
@ -121,6 +121,7 @@ Five nodes running Proxmox VE:
| [[central]] | utility (CT 104) | 192.168.1.104 | 100.64.0.12 | Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see [[central]] |
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.23 | PeerTube video streaming |
| mcc | media (CT 111) | 192.168.1.111 | — | pymc console web app (Caddy + Postfix, /api+/auth+/ws → aida-nebra :8000) |
| mailcow | edge1 (CT 101) | 10.10.10.2 | — | Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + Caddy) |
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | WordPress for intermountainmesh.com |
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
@ -130,7 +131,7 @@ Five nodes running Proxmox VE:
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
> **Note (2026-06-18):** edge2 CT placements above (CT 102–105) are sourced from migration git log. edge2 was not directly SSH-reachable during the 2026-06-18 fleet audit — placements pending live confirmation. CT 106 (matrix) and CT 107 (headscale) confirmed migrated 2026-06-18/19.
> **Note (2026-06-19):** edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (`forgejo-ssh-dnat.service`) is a permanent systemd unit on edge2 host.
## IP Allocation Scheme
@ -151,7 +152,7 @@ Current registered nodes (26 total):
| Node | Tailscale IP | Type |
|------|-------------|------|
| contabo | 100.64.0.1 | VPS |
| contabo | 100.64.0.40 | VPS (edge1, mail-only, rebuilt 2026-06-19; formerly 100.64.0.1) |
| immich | 100.64.0.2 | LXC |
| media | 100.64.0.3 | Proxmox |
| cloud | 100.64.0.4 | Proxmox |
@ -212,7 +213,7 @@ ssh zvx@<ip-address>
ssh zvx@192.168.1.244 # TOC (Proxmox host)
ssh zvx@192.168.1.150 # cortex VM
ssh zvx@192.168.1.241 # utility Proxmox
ssh root@100.64.0.1 # Contabo (via Tailscale)
ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149 # edge1 (mail-only; formerly Contabo)
ssh zvx@cortex # cortex via Tailscale hostname
ssh edge2 # edge2 via SSH config alias (admin@100.64.0.26)
```
@ -250,4 +251,5 @@ matt-desktop is accessible via local IP (192.168.1.254) or Tailscale (100.64.0.1
| Purpose | IP |
|---------|-----|
| Home external (public services) | 199.6.36.163 |
| Contabo VPS | 5.189.158.149 |
| edge1 (mail-only, rebuilt Contabo VPS) | 5.189.158.149 |
| edge2 (front door: vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh/intermountainmesh) | 184.174.35.153 |

19
vault/docs/hardware/ip-allocation.md
View file

@ -61,20 +61,29 @@ updated: 2026-06-19
| .182 | immich (CT 120) | cloud | Immich photos |
| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO |
### edge2 LXC Containers (10.10.10.x, vmbr0)
### edge1 LXC Containers (10.10.10.x, vmbr0) — mail-only host (5.189.158.149, rebuilt 2026-06-19)
edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate 10.10.10.0/24 subnet on vmbr0 (gw 10.10.10.1). No DHCP — all IPs are static. Storage: `local` (dir, no local-lvm).
edge1 (rebuilt Contabo VPS, 5.189.158.149 / Tailscale 100.64.0.40) runs Debian 12 + Proxmox 8.4.19, mail-only role. Same 10.10.10.0/24 subnet pattern on vmbr0 (gw 10.10.10.1); internal IP via `staticroute.service`.
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.2 | mailcow | CT 101 | — | Mailcow email server (privileged LXC; reached via host DNAT ports 25/465/587/110/143/993/995/4190 and host Caddy for mail/autodiscover/autoconfig.echo6.co → :8453) |
### edge2 LXC Containers (10.10.10.x, vmbr0) — permanent front door (184.174.35.153)
edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate 10.10.10.0/24 subnet on vmbr0 (gw 10.10.10.1). No DHCP — all IPs are static. Storage: `local` (dir, no local-lvm). **Front door for:** vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh/intermountainmesh.
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.10 | pdm | CT 100 | 100.64.0.28 | Proxmox Datacenter Manager |
| 10.10.10.11 | wordpress | CT 101 | 100.64.0.31 | WordPress for intermountainmesh.com |
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16); git SSH → edge2 forgejo-ssh-dnat.service |
| 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| 10.10.10.23 | [[authentik]] | CT 105 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
| 10.10.10.24 | matrix | CT 106 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
| 10.10.10.25 | headscale | CT 107 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
| 10.10.10.26 | mailcow-staging | CT 108 | — | Stopped Mailcow staging replica (fallback; prune after soak) |
### VMs (.150-.199)
| IP | VM | Host | Purpose |
@ -116,7 +125,7 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
| Tailscale IP | Device | Local IP |
|--------------|--------|----------|
| 100.64.0.5 | echo6-toc | 192.168.1.244 |
| 100.64.0.6 | contabo-mailcow | 5.189.158.149 (external) |
| 100.64.0.40 | contabo / edge1 (rebuilt 2026-06-19, mail-only) | 5.189.158.149 (external) |
| 100.64.0.18 | aida-nebra | 192.168.1.253 |
| 100.64.0.19 | utility | 192.168.1.241 |
| 100.64.0.20 | data | 192.168.1.240 |
@ -142,4 +151,4 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
---
*Last updated: 2026-06-19 — Added edge2 CT 106 (matrix) at 10.10.10.24 / 100.64.0.37 and CT 107 (headscale) at 10.10.10.25 / 100.64.0.38; previously added CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
*Last updated: 2026-06-19 — Added edge1 section (rebuilt Contabo VPS, 5.189.158.149 / 100.64.0.40, mail-only, Mailcow CT 101 at 10.10.10.2); added edge2 CT 108 (mailcow-staging, stopped); Tailscale table: contabo-mailcow entry replaced with edge1 at 100.64.0.40; previously added edge2 CTs 102–107*

110
vault/docs/services/services.md
View file

@ -14,6 +14,8 @@ updated: 2026-06-19
---
# Current Services Inventory
> **DNS split (2026-06-19):** `mail/autodiscover/autoconfig.echo6.co` → **edge1** (5.189.158.149, mail-only rebuilt Contabo VPS). `vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co` → **edge2** (184.174.35.153, permanent front door for all other services). Tailnet split-DNS is NOT used for echo6.co; echo6.co resolves via public GoDaddy DNS.
## Active Services
| Service | Location | IP:Port | Access | Notes |
@ -31,14 +33,13 @@ updated: 2026-06-19
| [[central]] | utility (CT 104) | 192.168.1.104:8000 / 100.64.0.12 | central.echo6.mesh (mesh) | Data-hub spine — ~25 adapters → NATS/JetStream → TimescaleDB; serves traffic tiles to navi — see [[central]] |
| NATS/JetStream (central) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | Central backend message bus (NATS :4222 client, :8222 monitoring) |
| TimescaleDB/PostGIS (central) | utility (CT 104) | 192.168.1.104:5432 | Internal | Central backend time-series + geospatial database (PostgreSQL 16 + TimescaleDB + PostGIS) |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by Contabo Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via Contabo DNAT) | https://forge.echo6.co | Git server — fronted by Contabo Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on Contabo (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — **migrated from Contabo 2026-06-19** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) — **migrated from Contabo 2026-06-19** |
| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by Contabo Caddy (reverse_proxy 100.64.0.33:8086) |
| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by edge2 host Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via edge2 DNAT) | https://forge.echo6.co | Git server — fronted by edge2 host Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on edge2 (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Mailcow | **edge1 CT 101** (10.10.10.2) | 5.189.158.149 | https://mail.echo6.co | Email server (privileged LXC on rebuilt Contabo VPS, updated commit 52a41b4d / SOGo 5.12.8) — **rebuilt in-place 2026-06-19** |
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by edge2 host Caddy (reverse_proxy 100.64.0.33:8086) |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ (Syncthing on Contabo decommissioned 2026-06-19 with edge1 rebuild) |
| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
| Immich | cloud (CT 120) | 192.168.1.182:2283 | https://immich.echo6.co | Photo management (Docker, NFS storage on pi-nas) |
| Nextcloud | cloud (CT 121) | 192.168.1.183:11000 | https://nextcloud.echo6.co | Cloud storage (AIO Docker, NFS on pi-nas, SSO) |
@ -71,12 +72,10 @@ updated: 2026-06-19
| Matrix [[synapse]] | edge2 (CT 106) | 100.64.0.37:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) — **migrated from Contabo 2026-06-18** |
| Element Web | edge2 (CT 106) | 100.64.0.37:8088 | https://element.echo6.co | Matrix web client (Docker) — **migrated from Contabo 2026-06-18** |
| [[mautrix_signal]] | edge2 (CT 106) | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) — **migrated from Contabo 2026-06-18** |
| LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by Contabo Caddy (reverse_proxy 100.64.0.35:5984 / :5985); **migrated from Contabo 2026-06-16** |
| LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by edge2 host Caddy; **migrated from Contabo 2026-06-16** |
| OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] |
| Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
| mautrix-signal | Contabo | 29328 (internal) | Internal (matrix-net) | Signal bridge — @signalbot:echo6.co, E2BE, MSC4190, auto-portals |
| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) — **migrated from Contabo 2026-06-18** |
| Termix | Contabo | 0.0.0.0:8083 | Internal (no Caddy block) | Terminal sharing tool (Docker, ghcr.io/lukegus/termix:latest) |
| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via edge2 Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) — **migrated from Contabo 2026-06-18** |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | Internal | Signal/Matrix room archive bot (systemd) — see archivist.md for details |
| pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
| recon-sparse | cortex (VM 150) | 192.168.1.150:8091 | Internal | RECON sparse embedding service (systemd, bge-m3 model, port 8091) |
@ -301,18 +300,17 @@ updated: 2026-06-19
- Tailscale registered on IdahoMesh Headscale (vpn.idahomesh.com) under malice user
> **Note (2026-06-18):** edge2 CT placements (CT 102–05) are sourced from the migration git log. edge2 was not directly SSH-reachable during the 2026-06-18 fleet audit — placements pending live confirmation.
### edge2 - CT 103 (10.10.10.21 / Tailscale: 100.64.0.34, node 46 `forgejo`)
- Forgejo git server (https://forge.echo6.co — **migrated from Contabo 2026-06-16**)
- Headscale node id 46, name `forgejo`, user `echo6`
- Compose path: `/opt/forgejo/docker-compose.yml`
- Containers: `forgejo` (codeberg.org/forgejo/forgejo:14) + `forgejo-db` (postgres:16-alpine)
- Volumes: `forgejo-data` (git repos, app.ini, SSH host keys) + `forgejo-db`
- HTTP: binds to `100.64.0.34:3001`; Contabo Caddy proxies here over tailnet (`reverse_proxy 100.64.0.34:3001`)
- SSH: `git@forge.echo6.co:2222` → Contabo iptables DNAT (systemd unit `forgejo-ssh-dnat.service`, enabled, reboot-persistent) → `100.64.0.34:2222` → container port 22; SSH host keys preserved (no client warning)
- HTTP: binds to `100.64.0.34:3001`; edge2 host Caddy proxies here over tailnet (`reverse_proxy 100.64.0.34:3001`)
- SSH: `git@forge.echo6.co:2222` → edge2 iptables DNAT (systemd unit `forgejo-ssh-dnat.service`, enabled, reboot-persistent) → `100.64.0.34:2222` → container port 22; SSH host keys preserved (no client warning)
- Login via Authentik OIDC (auth.echo6.co) — unchanged
- DB: PostgreSQL 16 (forgejo_db container); 9 repos, 1 user verified at migration
- Source (Contabo `/opt/forgejo`) STOPPED but intact as cold rollback; rollback = `systemctl disable --now forgejo-ssh-dnat` + restore `/etc/caddy/Caddyfile.bak-preforge` + `systemctl restart caddy` + `docker compose up -d` on Contabo
- Source (Contabo `/opt/forgejo`) was wiped with edge1 OS rebuild 2026-06-19
### edge2 - CT 104 (10.10.10.22 / Tailscale: 100.64.0.35, node `livesync`)
- LiveSync Obsidian sync service (https://notes.echo6.co — **migrated from Contabo 2026-06-16**)
@ -322,10 +320,10 @@ updated: 2026-06-19
- Named volumes: `couchdb-data`, `provisioner-data`
- Bind mounts: `couchdb/local.ini`, `couchdb/local.d/` (incl. `jwt-keys.ini`)
- CouchDB binds to `100.64.0.35:5984`; provisioner binds to `100.64.0.35:5985`
- Contabo Caddy proxies `notes.echo6.co` → `reverse_proxy 100.64.0.35:5984` (CouchDB) + `reverse_proxy 100.64.0.35:5985` (provisioner); Authentik forward_auth on `/_provision`; Obsidian CORS preserved; Authentik outpost stays `127.0.0.1:9000`
- edge2 host Caddy proxies `notes.echo6.co` → `reverse_proxy 100.64.0.35:5984` (CouchDB) + `reverse_proxy 100.64.0.35:5985` (provisioner); Authentik forward_auth on `/_provision`; Obsidian CORS preserved
- Auth: per-user JWT (ES512) via provisioner; databases: `cc-db`, `userdb-matt`
- Data: ~16 MB in couchdb-data
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-prelivesync` exists
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs on `local`; unprivileged; onboot; Docker
### edge2 - CT 105 (10.10.10.23 / Tailscale: 100.64.0.36, node 48 `authentik`)
@ -334,12 +332,12 @@ updated: 2026-06-19
- Compose path: `/opt/authentik/docker-compose.yml`
- Containers: `authentik-server` + `authentik-worker` (ghcr.io/goauthentik/server:2025.12.4) + `authentik-postgres` (postgres:16); NO Redis
- Worker runs as user:root and has docker.sock bind-mount (manages embedded outposts)
- Binds to `100.64.0.36:9000`; Contabo Caddy proxies here over tailnet for both `auth.echo6.co` (catch-all + outpost path matcher) and `notes.echo6.co` outpost/forward_auth references
- Binds to `100.64.0.36:9000`; edge2 host Caddy proxies here over tailnet for both `auth.echo6.co` (catch-all + outpost path matcher) and `notes.echo6.co` outpost/forward_auth references
- `AUTHENTIK_SECRET_KEY` carried byte-for-byte (sessions stayed valid across cutover — users dropped straight in)
- Bind-mounts (data/media, branding, certs, custom-templates) migrated intact
- Email dep: mail.echo6.co (unchanged)
- DB: PostgreSQL 16 (`authentik-postgres` container); ~705 MB (~18 MB pg_dump)
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists on Contabo
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- Reboot-survival fix: systemd unit on CT 105 gates `docker compose up` on `tailscale-online` (Docker was racing Tailscale on boot, failing the bind to the tailnet IP)
- **Resources:** 2 cores / 4096 MB RAM / 512 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
@ -349,14 +347,14 @@ updated: 2026-06-19
- Compose path: `/opt/matrix/matrix-stack.service` (systemd service with tailscale-before-docker reboot guard)
- Containers: `synapse` (:8008), `matrix-mas` (:8085), `element` (:8088), `matrix-postgres`, `mautrix-signal` (29328 internal) — 5 containers total
- Databases: PostgreSQL DBs `synapse`, `mas`, `mautrix_signal` on `matrix-postgres` container — 3 DBs total
- Binds to `100.64.0.37`; Contabo Caddy proxies:
- Binds to `100.64.0.37`; edge2 host Caddy proxies:
- `matrix.echo6.co` → Synapse (`100.64.0.37:8008`) + MAS sub-routes
- `element.echo6.co` → Element (`100.64.0.37:8088`)
- MAS (`100.64.0.37:8085`) handles login/logout/refresh/auth_metadata
- Federation delegation from apex `echo6.co` (`.well-known`) unchanged
- MAS→Authentik OIDC (`auth.echo6.co`) unchanged
- Signal bridge: `@signalbot:echo6.co`, E2BE + MSC4190, double puppeting; bridge state preserved in `mautrix_signal` DB
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-matrix` exists
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 2 cores / 4096 MB RAM / 1024 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
### edge2 - CT 107 (10.10.10.25 / Tailscale: 100.64.0.38, node `headscale`)
@ -364,44 +362,58 @@ updated: 2026-06-19
- Headscale hostname `headscale`, tailnet IP 100.64.0.38
- Compose path: `/opt/headscale/` (managed via `headscale-stack.service` systemd reboot guard)
- Headscale 0.28.0 on `:8084`; Headplane on `:3100`
- Contabo Caddy proxies `vpn.echo6.co` → `100.64.0.38:8084` (headscale) / `vpn.echo6.co/admin` → `100.64.0.38:3100` (headplane)
- edge2 host Caddy proxies `vpn.echo6.co` → `100.64.0.38:8084` (headscale) / `vpn.echo6.co/admin` → `100.64.0.38:3100` (headplane)
- `noise_private.key` carried byte-for-byte (server identity preserved); 39 nodes
- Public direct bind `5.189.158.149:8084` DROPPED — all clients use `vpn.echo6.co` via Caddy
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-preheadscale` exists
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 1 core / 512 MB RAM / 256 MB swap / 4 GB rootfs on `local`; unprivileged; onboot
### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
- Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
- Headscale node id 45, name `vaultwarden`, user `echo6`
- Compose path: `/opt/vaultwarden/docker-compose.yml`; data: `./data/`
- Binds to tailnet IP `100.64.0.33:8086:80`; Contabo Caddy proxies here over tailnet
- Binds to tailnet IP `100.64.0.33:8086:80`; edge2 host Caddy proxies here over tailnet
- SSO via Authentik (`SSO_ONLY=false` — local email+master-password login also works)
- Source (Contabo `/opt/vaultwarden`) STOPPED but intact as cold rollback; rollback = restore `/etc/caddy/Caddyfile.bak-prevault` + `systemctl restart caddy` + `docker compose up -d` on Contabo
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Migrated from Contabo to edge2 CT 102 on 2026-06-16**
### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
- ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (Caddy now proxies auth.echo6.co + notes.echo6.co outpost/forward_auth → 100.64.0.36:9000; source STOPPED at `/opt/authentik`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists)
- Forge (Git) — **migrated to edge2 CT 103 on 2026-06-16** (Caddy now proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service)
- ~~Headscale~~ — **migrated to edge2 CT 107 on 2026-06-19** (Caddy now proxies vpn.echo6.co → 100.64.0.38:8084 / headplane → 100.64.0.38:3100; noise_private.key carried byte-for-byte; direct public bind 5.189.158.149:8084 DROPPED; source STOPPED at `/opt/headscale`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-preheadscale` exists)
- Mailcow (email)
- Vaultwarden — **migrated to edge2 CT 102 on 2026-06-16** (Caddy now proxies to 100.64.0.33:8086)
- Syncthing (syncs with cortex)
- ~~WATCHTOWER~~ — **decommissioned 2026-06-16** (was `/opt/watchtower`; archived to forge.echo6.co/matt/archive-watchtower; `wt.echo6.co` Caddy block + GoDaddy A record removed)
- ~~Matrix Synapse~~ — **migrated to edge2 CT 106 on 2026-06-18** (Caddy now proxies matrix.echo6.co → 100.64.0.37:8008 / element.echo6.co → 100.64.0.37:8088 / MAS → 100.64.0.37:8085; source STOPPED at `/opt/matrix`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-matrix` exists)
- ~~Element Web~~ — **migrated to edge2 CT 106 on 2026-06-18** (see Matrix entry above)
- ~~mautrix-signal bridge~~ — **migrated to edge2 CT 106 on 2026-06-18** (see Matrix entry above; Signal bridge state preserved in mautrix_signal DB)
- ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16** (source `/opt/livesync` STOPPED, intact as cold rollback; Caddy now proxies `notes.echo6.co` → `100.64.0.35:5984/5985`; `/etc/caddy/Caddyfile.bak-prelivesync` exists)
- ~~TAK Server~~ — **decommissioned 2026-06-16** (was `/opt/tak-server-deploy`; archived to forge.echo6.co/matt/archive-tak-server; `tak.echo6.co` Caddy block + GoDaddy A record removed)
- ~~SIGIL~~ — **decommissioned 2026-06-16** (was `/opt/sigil`; archived to forge.echo6.co/matt/archive-tak-server)
- Termix (port 8083, internal, Docker)
- Container: `termix` on `termix_default` network
- Image: `ghcr.io/lukegus/termix:latest`
- Port: 8080→8083 (bound to 0.0.0.0, NOT 127.0.0.1)
- Volume: `termix_termix-data` → `/app/data`
- No Caddy block — direct access only on port 8083
- Compose: `/opt/termix/` (inferred from Docker volume naming)
- ~~echo6-agent~~ — **decommissioned 2026-06-16** (was systemd `echo6-agent.service` at `/opt/echo6-agent/`; archived to forge.echo6.co/matt/archive-echo6-agent; unit disabled)
- ~~nexus-hub~~ + ~~nexus-agent~~ — **decommissioned 2026-06-16** (were systemd units at `/root/nexus-hub` and `/root/nexus-agent`; archived to forge.echo6.co/matt/archive-nexus-hub + archive-nexus-agent; units disabled)
### edge1 (5.189.158.149 / Tailscale: 100.64.0.40) — MAIL-ONLY host (rebuilt Contabo VPS, 2026-06-19)
> **Rebuilt in-place 2026-06-19.** Former Contabo VPS (tailnet node previously `contabo` / 100.64.0.1) was OS-reinstalled as Debian 12 + Proxmox 8.4.19. Renamed edge1. All evacuated services now live on edge2. Tailnet identity re-registered as `contabo` at **100.64.0.40**.
- **Role: MAIL ONLY.** All non-mail services (vault/forge/notes/auth/matrix/element/vpn/proxmox) are fronted by **edge2** (184.174.35.153).
- **Host Caddy** serves `mail.echo6.co`, `autodiscover.echo6.co`, `autoconfig.echo6.co` → Mailcow CT 101 at `10.10.10.2:8453`; includes `header_up Host {host}` to prevent nginx leaking internal address in redirects.
- **mailcow-dnat.service** — source-IP-preserving DNAT for raw mail ports 25/465/587/110/143/993/995/4190 → 10.10.10.2 (systemd oneshot, reboot-persistent).
- **staticroute.service** — systemd oneshot for 10.10.10.0/24 route (reboot-persistent).
- Netfilter modules (`nf_conntrack`, `nf_nat`, etc.) persisted via `/etc/modules-load.d/`.
- Tailscale on host (node `contabo`, 100.64.0.40).
- unattended-upgrades enabled.
- SSH: `ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149`
#### edge1 — CT 101 (10.10.10.2) — Mailcow
- **Mailcow** email server (privileged LXC, updated to commit `52a41b4d`, SOGo 5.12.8)
- Accessible at `mail.echo6.co` / `5.189.158.149` (via host Caddy + DNAT)
- mail_crypt secondary key in `data/conf/dovecot/extra.conf` (`ecprivkey.pem.fresh-20260619` as secondary; both primary + secondary loaded for decryption of old mail)
- DNS: `mail/autodiscover/autoconfig.echo6.co` → 5.189.158.149
#### Previously on Contabo (now evacuated or decommissioned)
- ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (edge2 Caddy now proxies auth.echo6.co → 100.64.0.36:9000)
- ~~Forge (Git)~~ — **migrated to edge2 CT 103 on 2026-06-16** (edge2 Caddy proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service on edge2)
- ~~Headscale~~ — **migrated to edge2 CT 107 on 2026-06-19** (edge2 Caddy proxies vpn.echo6.co → 100.64.0.38:8084; noise_private.key carried byte-for-byte; direct public bind 5.189.158.149:8084 DROPPED)
- ~~Vaultwarden~~ — **migrated to edge2 CT 102 on 2026-06-16** (edge2 Caddy proxies to 100.64.0.33:8086)
- ~~Syncthing~~ — **decommissioned 2026-06-19** with edge1 OS rebuild (Syncthing state removed; Forge is now the durable backup via autocommit cron)
- ~~WATCHTOWER~~ — **decommissioned 2026-06-16**
- ~~Matrix Synapse~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~Element Web~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~mautrix-signal bridge~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16**
- ~~TAK Server~~ — **decommissioned 2026-06-16** (archived to forge.echo6.co/matt/archive-tak-server)
- ~~SIGIL~~ — **decommissioned 2026-06-16**
- ~~echo6-agent~~ — **decommissioned 2026-06-16**
- ~~nexus-hub~~ + ~~nexus-agent~~ — **decommissioned 2026-06-16**
- ~~Termix~~ — **wiped with edge1 rebuild 2026-06-19**
## Decommissioned Services

63
vault/runbooks/lxc-service-migration.md
View file

@ -423,6 +423,67 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
---
---
## Mail / Host-in-Place Rebuild (edge1, 2026-06-19)
> This section documents the Mailcow migration pattern, which cannot use the standard Contabo→edge2 LXC runbook above. It serves as both a record of what was done and a set of hard-won lessons for future mail-server rebuilds.
### Why Mailcow cannot use the standard cutover
Mailcow cannot be "moved" to a different host IP via the one-token Caddy approach:
- **Raw mail ports are hardlocked to the IP.** SMTP (25/465/587) and IMAP/POP3 (110/143/993/995/4190) require a real public IP — not a Caddy reverse proxy. Deliverability (PTR, SPF, DMARC) is hardlocked to `5.189.158.149`. Moving the IP would require re-establishing that reputation.
- **Approach: OS rebuild the same host, keep the same IP.** Re-install the Contabo VPS (now edge1) with Debian 12 + Proxmox 8.4.19 in place. Mailcow runs in a **privileged LXC (CT 101)** on the `10.10.10.0/24` internal bridge. The host Caddy terminates HTTPS for mail/autodiscover/autoconfig, and a `mailcow-dnat.service` unit (systemd oneshot) does source-IP-preserving DNAT for all raw mail ports to the LXC.
### Rebuild bootstrap pattern
The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated services were already on edge2; no outage for those).
2. **DNS during rebuild:** echo6.co resolves via public GoDaddy DNS. During the rebuild the `mail.*` records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. **Headscale pre-auth key:** Generate before the rebuild starts (`headscale preauthkeys create`). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. **Tailscale bootstrap DNS:** The tailnet uses `vpn.echo6.co` for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.
### Mailcow-specific lessons
| ID | Lesson |
|----|--------|
| M1 | **mail_crypt: capture the raw key BEFORE backup/restore.** The dovecot mail_crypt private key (`ecprivkey.pem`) lives in the crypt volume at `data/conf/dovecot/`. A mailcow backup/restore can generate a FRESH key — all old mail becomes undecryptable. Always extract the raw `ecprivkey.pem` before decommissioning the source and inject it into the crypt volume on the target. |
| M2 | **Keep mail_crypt customization in `extra.conf`, not `dovecot.conf`.** `dovecot.conf` is overwritten by mailcow updates. Custom mail_crypt config (extra key loading) belongs in `data/conf/dovecot/extra.conf` which is update-safe. |
| M3 | **Load BOTH keys (primary + secondary) after a restore-with-new-key.** A single undecryptable message in a folder poisons SOGo's bulk header fetch for that folder — the entire folder appears blank. By loading both the original key (`ecprivkey.pem`) AND the fresh key generated by the restore (`ecprivkey.pem.fresh-20260619`) via `mail_crypt_global_private_key_path` (and `mail_crypt_global_private_key_path2`), all old and new mail decrypts correctly. |
| M4 | **Integrity gate MUST test decryption — not just counts.** DKIM, mailbox counts, and alias counts all passed while all message content was key-locked (mail_crypt was using the wrong key). The integrity gate must open a mailbox, fetch a message, and confirm the body renders (non-empty). |
| M5 | **SOGo after restore: re-run `bootstrap-sogo.sh` if auth sources are lost.** After a fresh Mailcow install + backup restore, SOGo may lose its IMAP auth sources. Run `docker exec -it sogo-mailcow /bootstrap-sogo.sh` to re-apply them. |
| M6 | **SOGo caches stale state in `sogo_cache_folder`.** After fixing the mail_crypt key issue, SOGo still shows blank folders until its cache is cleared. Run: `docker exec -it sogo-mailcow sogo-tool expire-cache-folder '*'` (or truncate the `sogo_cache_folder` table in the database). |
| M7 | **Mailcow behind Caddy on a DIFFERENT host: send `header_up Host {host}`.** Mailcow's nginx uses the `Host` header to build redirects. Without `header_up Host {host}` in the Caddy block, nginx leaks its internal address (e.g. `10.10.10.2:8453`) in HTTP redirects, breaking web UI navigation. |
| M8 | **Front-door placement: co-locate Caddy with its backend CTs.** edge2 serves as front door for its own CTs (vault/forge/notes/auth/matrix/element/vpn) over the local tailnet. edge1 serves as front door for its own CT (mailcow). Avoid cross-host Caddy→backend paths when co-location is an option — it eliminates one network hop and removes tailnet dependency for the mail-critical path. |
| M9 | **iptables DNAT for mail ports requires `MASQUERADE` or source-preserving DNAT.** For mail servers, the real sender IP must reach the MTA (Postfix checks SPF/HELO). Use `--to-destination` with conntrack (`nf_conntrack` module) so the source IP is preserved end-to-end through the DNAT. Verify with `Received:` headers in delivered mail. |
| M10 | **Forge git-SSH DNAT on edge2 is now a permanent systemd unit** (`forgejo-ssh-dnat.service`, enabled, reboot-persistent). No longer a workaround — it is the production path. |
### edge1 host-level systemd units (reboot-persistent)
| Unit | Purpose |
|------|---------|
| `mailcow-dnat.service` | oneshot: iptables DNAT for ports 25/465/587/110/143/993/995/4190 → 10.10.10.2 |
| `staticroute.service` | oneshot: `ip route add 10.10.10.0/24 via <bridge>` (LXC subnet route) |
| `tailscale` | Tailscale on host; node `contabo`, IP 100.64.0.40 |
| `caddy` | host Caddy: mail/autodiscover/autoconfig.echo6.co → `10.10.10.2:8453` |
| `unattended-upgrades` | automatic security patching |
### Netfilter modules (edge1)
Required for DNAT to work across bridge to LXC. Persisted via `/etc/modules-load.d/netfilter.conf`:
```
nf_conntrack
nf_nat
xt_conntrack
xt_MASQUERADE
nf_nat_ipv4
```
---
## Gotchas
| ID | Issue |
@ -461,4 +522,4 @@ ssh root@100.64.0.1 'systemctl disable --now <service>-ssh-dnat.service && rm /e
---
*Last updated: 2026-06-19 — Added Matrix pilot (multi-DB Postgres + stateful Signal bridge, CT 106) and Headscale pilot (tailnet control plane, noise_private.key must travel, CT 107); Gotcha G17 (tailnet-control-plane migration pattern: data-plane survives outage, out-of-band re-entry via public IPs, self-verifying auto-rollback in tmux); previously added Authentik pilot (G14–G16)*
*Last updated: 2026-06-19 — Added "Mail / Host-in-Place Rebuild (edge1)" section with lessons M1–M10 (mail_crypt key capture, extra.conf, dual-key load, decryption integrity gate, SOGo bootstrap + cache, Caddy Host header, front-door co-location, DNAT source-preserve, forgejo-ssh-dnat permanent); previously added Matrix pilot (CT 106), Headscale pilot (CT 107, G17), Authentik pilot (G14–G16)*