docs: migrate Matrix + Headscale to edge2 (CT 106/107)

- Matrix -> edge2 CT 106 (5 containers, 3 Postgres DBs, Signal bridge state preserved) - Headscale -> edge2 CT 107 (noise_private.key carried; tailnet data-plane survived cutover) - runbook: tailnet-control-plane pattern + self-verifying auto-rollback tmux cutover - both sources stopped + intact on Contabo as cold rollback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 01:31:34 +00:00 · 2026-06-19 01:31:34 +00:00 · 74a0aaae43
commit 74a0aaae43
parent 75df23b89e
4 changed files with 55 additions and 45 deletions
--- a/vault/docs/hardware/environment.md
+++ b/vault/docs/hardware/environment.md
@ -10,7 +10,7 @@ related:
  - [[caddy]]
  - [[ct-runbook]]
  - [[proxmox-create-ubuntu-vm]]
-updated: 2026-06-18
+updated: 2026-06-19
 ---
 # Echo6 Environment Reference

@ -102,7 +102,7 @@ Five nodes running Proxmox VE:
 | Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, [[authentik]], Headscale, Forge, Matrix |
 | edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB |

-*Last updated: 2026-06-18 — Added central (utility CT 104, 192.168.1.104, 100.64.0.12); also added edge2 CT 105 (authentik, 10.10.10.23, 100.64.0.36, node 48, migrated 2026-06-18); previously added edge2 CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101*
+*Last updated: 2026-06-19 — Added edge2 CT 106 (matrix, 10.10.10.24, 100.64.0.37, migrated 2026-06-18) and CT 107 (headscale, 10.10.10.25, 100.64.0.38, migrated 2026-06-19); previously added CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101, central (utility CT 104)*

 ## LXC Containers

@ -127,8 +127,10 @@ Five nodes running Proxmox VE:
 | forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
 | livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
 | authentik | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
+| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
+| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |

-> **Note (2026-06-18):** edge2 CT placements above (CT 102–105) are sourced from migration git log. edge2 was not directly SSH-reachable during the 2026-06-18 fleet audit — placements pending live confirmation.
+> **Note (2026-06-18):** edge2 CT placements above (CT 102–105) are sourced from migration git log. edge2 was not directly SSH-reachable during the 2026-06-18 fleet audit — placements pending live confirmation. CT 106 (matrix) and CT 107 (headscale) confirmed migrated 2026-06-18/19.

 ## IP Allocation Scheme

@ -181,6 +183,8 @@ Current registered nodes (26 total):
 | forgejo | 100.64.0.34 | LXC (edge2 CT 103) — node id 46 |
 | livesync | 100.64.0.35 | LXC (edge2 CT 104) — migrated 2026-06-16 |
 | authentik | 100.64.0.36 | LXC (edge2 CT 105) — node id 48, migrated 2026-06-18 |
+| matrix | 100.64.0.37 | LXC (edge2 CT 106) — migrated 2026-06-18 |
+| headscale | 100.64.0.38 | LXC (edge2 CT 107) — migrated 2026-06-19 |

 ## IdahoMesh Headscale Node List

--- a/vault/docs/hardware/ip-allocation.md
+++ b/vault/docs/hardware/ip-allocation.md
@ -10,7 +10,7 @@ related:
  - [[caddy]]
  - [[headscale-onboard-node]]
  - [[meshtastic-headscale-runbook]]
-updated: 2026-06-18
+updated: 2026-06-19
 ---
 # Echo6 Network IP Allocation

@ -73,6 +73,8 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
 | 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
 | 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
 | 10.10.10.23 | [[authentik]] | CT 105 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
+| 10.10.10.24 | matrix | CT 106 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
+| 10.10.10.25 | headscale | CT 107 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |

 ### VMs (.150-.199)
 | IP | VM | Host | Purpose |
@ -126,6 +128,8 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
 | 100.64.0.34 | forgejo (CT 103 on edge2) — node id 46 | 10.10.10.21 (vmbr0) |
 | 100.64.0.35 | livesync (CT 104 on edge2) — hostname `livesync` | 10.10.10.22 (vmbr0) |
 | 100.64.0.36 | authentik (CT 105 on edge2) — node id 48 | 10.10.10.23 (vmbr0) |
+| 100.64.0.37 | matrix (CT 106 on edge2) — hostname `matrix` | 10.10.10.24 (vmbr0) |
+| 100.64.0.38 | headscale (CT 107 on edge2) — hostname `headscale` | 10.10.10.25 (vmbr0) |

 ---

@ -138,4 +142,4 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate

 ---

-*Last updated: 2026-06-18 — Added edge2 CT 105 (authentik) at 10.10.10.23 / 100.64.0.36 (node id 48); previously added CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
+*Last updated: 2026-06-19 — Added edge2 CT 106 (matrix) at 10.10.10.24 / 100.64.0.37 and CT 107 (headscale) at 10.10.10.25 / 100.64.0.38; previously added CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
--- a/vault/docs/services/services.md
+++ b/vault/docs/services/services.md
@ -10,7 +10,7 @@ related:
  - [[glossary]]
  - [[meshtastic-headscale-runbook]]
  - [[lxc-service-migration]]
-updated: 2026-06-18
+updated: 2026-06-19
 ---
 # Current Services Inventory

@ -33,8 +33,8 @@ updated: 2026-06-18
 | TimescaleDB/PostGIS (central) | utility (CT 104) | 192.168.1.104:5432 | Internal | Central backend time-series + geospatial database (PostgreSQL 16 + TimescaleDB + PostGIS) |
 | [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by Contabo Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
 | Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via Contabo DNAT) | https://forge.echo6.co | Git server — fronted by Contabo Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on Contabo (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
-| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
-| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
+| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — **migrated from Contabo 2026-06-19** |
+| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) — **migrated from Contabo 2026-06-19** |
 | Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
 | Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by Contabo Caddy (reverse_proxy 100.64.0.33:8086) |
 | Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
@ -68,14 +68,14 @@ updated: 2026-06-18
 | PostgreSQL/PostGIS | data (VM 1130) | 192.168.1.130:5432 | Internal | navi geo backend (padus, overture DBs) — see [[navi]] |
 | Files | data (VM 1130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, Authentik forward auth) |
 | Samba | data | 192.168.1.240:445 | Internal | SMB file sharing — `//data/library` → /mnt/data/library (guest access) |
-| Matrix [[synapse]] | Contabo | 127.0.0.1:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) |
-| Element Web | Contabo | 127.0.0.1:8088 | https://element.echo6.co | Matrix web client (Docker) |
-| [[mautrix_signal]] | Contabo | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) |
+| Matrix [[synapse]] | edge2 (CT 106) | 100.64.0.37:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) — **migrated from Contabo 2026-06-18** |
+| Element Web | edge2 (CT 106) | 100.64.0.37:8088 | https://element.echo6.co | Matrix web client (Docker) — **migrated from Contabo 2026-06-18** |
+| [[mautrix_signal]] | edge2 (CT 106) | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) — **migrated from Contabo 2026-06-18** |
 | LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by Contabo Caddy (reverse_proxy 100.64.0.35:5984 / :5985); **migrated from Contabo 2026-06-16** |
 | OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] |
 | Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
 | mautrix-signal | Contabo | 29328 (internal) | Internal (matrix-net) | Signal bridge — @signalbot:echo6.co, E2BE, MSC4190, auto-portals |
-| Matrix MAS | Contabo | 127.0.0.1:8085 | Internal (via Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) |
+| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) — **migrated from Contabo 2026-06-18** |
 | Termix | Contabo | 0.0.0.0:8083 | Internal (no Caddy block) | Terminal sharing tool (Docker, ghcr.io/lukegus/termix:latest) |
 | [[archivist]] | utility (CT 118) | 192.168.1.118 | Internal | Signal/Matrix room archive bot (systemd) — see archivist.md for details |
 | pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
@ -343,6 +343,33 @@ updated: 2026-06-18
  - Reboot-survival fix: systemd unit on CT 105 gates `docker compose up` on `tailscale-online` (Docker was racing Tailscale on boot, failing the bind to the tailnet IP)
  - **Resources:** 2 cores / 4096 MB RAM / 512 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker

+### edge2 - CT 106 (10.10.10.24 / Tailscale: 100.64.0.37, node `matrix`)
+- Matrix stack (https://matrix.echo6.co, https://element.echo6.co — **migrated from Contabo 2026-06-18**)
+  - Headscale hostname `matrix`, tailnet IP 100.64.0.37
+  - Compose path: `/opt/matrix/matrix-stack.service` (systemd service with tailscale-before-docker reboot guard)
+  - Containers: `synapse` (:8008), `matrix-mas` (:8085), `element` (:8088), `matrix-postgres`, `mautrix-signal` (29328 internal) — 5 containers total
+  - Databases: PostgreSQL DBs `synapse`, `mas`, `mautrix_signal` on `matrix-postgres` container — 3 DBs total
+  - Binds to `100.64.0.37`; Contabo Caddy proxies:
+    - `matrix.echo6.co` → Synapse (`100.64.0.37:8008`) + MAS sub-routes
+    - `element.echo6.co` → Element (`100.64.0.37:8088`)
+    - MAS (`100.64.0.37:8085`) handles login/logout/refresh/auth_metadata
+  - Federation delegation from apex `echo6.co` (`.well-known`) unchanged
+  - MAS→Authentik OIDC (`auth.echo6.co`) unchanged
+  - Signal bridge: `@signalbot:echo6.co`, E2BE + MSC4190, double puppeting; bridge state preserved in `mautrix_signal` DB
+  - Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-matrix` exists
+  - **Resources:** 2 cores / 4096 MB RAM / 1024 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
+
+### edge2 - CT 107 (10.10.10.25 / Tailscale: 100.64.0.38, node `headscale`)
+- Headscale + Headplane (https://vpn.echo6.co — **migrated from Contabo 2026-06-19**)
+  - Headscale hostname `headscale`, tailnet IP 100.64.0.38
+  - Compose path: `/opt/headscale/` (managed via `headscale-stack.service` systemd reboot guard)
+  - Headscale 0.28.0 on `:8084`; Headplane on `:3100`
+  - Contabo Caddy proxies `vpn.echo6.co` → `100.64.0.38:8084` (headscale) / `vpn.echo6.co/admin` → `100.64.0.38:3100` (headplane)
+  - `noise_private.key` carried byte-for-byte (server identity preserved); 39 nodes
+  - Public direct bind `5.189.158.149:8084` DROPPED — all clients use `vpn.echo6.co` via Caddy
+  - Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-preheadscale` exists
+  - **Resources:** 1 core / 512 MB RAM / 256 MB swap / 4 GB rootfs on `local`; unprivileged; onboot
+
 ### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
 - Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
  - Headscale node id 45, name `vaultwarden`, user `echo6`
@ -355,33 +382,17 @@ updated: 2026-06-18
 ### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
 - ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (Caddy now proxies auth.echo6.co + notes.echo6.co outpost/forward_auth → 100.64.0.36:9000; source STOPPED at `/opt/authentik`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists)
 - Forge (Git) — **migrated to edge2 CT 103 on 2026-06-16** (Caddy now proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service)
- Headscale (mesh VPN)
+- ~~Headscale~~ — **migrated to edge2 CT 107 on 2026-06-19** (Caddy now proxies vpn.echo6.co → 100.64.0.38:8084 / headplane → 100.64.0.38:3100; noise_private.key carried byte-for-byte; direct public bind 5.189.158.149:8084 DROPPED; source STOPPED at `/opt/headscale`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-preheadscale` exists)
 - Mailcow (email)
 - Vaultwarden — **migrated to edge2 CT 102 on 2026-06-16** (Caddy now proxies to 100.64.0.33:8086)
 - Syncthing (syncs with cortex)
 - ~~WATCHTOWER~~ — **decommissioned 2026-06-16** (was `/opt/watchtower`; archived to forge.echo6.co/matt/archive-watchtower; `wt.echo6.co` Caddy block + GoDaddy A record removed)
- Matrix Synapse homeserver (port 8008, https://matrix.echo6.co, Docker, SSO via Authentik)
- Element Web client (port 8088, https://element.echo6.co, Docker)
- mautrix-signal bridge (port 29328 internal, Docker, E2BE with MSC4190)
-  - Image: `dock.mau.dev/mautrix/signal:v0.2603.0`
-  - Container: `mautrix-signal` on `matrix-net`
-  - Config: `/opt/matrix/mautrix-signal/config.yaml`
-  - Registration: `/opt/matrix/synapse/registration.yaml` + `/opt/matrix/synapse/doublepuppet.yaml`
-  - Database: `mautrix_signal` on `matrix-postgres` (role: `mautrix_signal`, minimal grants)
-  - Bot user: `@signalbot:echo6.co`, device: `UPX4KKLZVY`
-  - Permissions: `@matt:echo6.co` = admin, `echo6.co` = user
-  - Double puppeting: appservice-based (doublepuppet.yaml as_token)
-  - Encryption: E2BE enabled (allow+default+require), MSC4190, self-signed cross-signing keys
- Compose path: `/opt/matrix/docker-compose.yml`
- Backup: daily at 3AM, 14-day retention (synapse + mas + mautrix_signal databases)
+- ~~Matrix Synapse~~ — **migrated to edge2 CT 106 on 2026-06-18** (Caddy now proxies matrix.echo6.co → 100.64.0.37:8008 / element.echo6.co → 100.64.0.37:8088 / MAS → 100.64.0.37:8085; source STOPPED at `/opt/matrix`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-matrix` exists)
+- ~~Element Web~~ — **migrated to edge2 CT 106 on 2026-06-18** (see Matrix entry above)
+- ~~mautrix-signal bridge~~ — **migrated to edge2 CT 106 on 2026-06-18** (see Matrix entry above; Signal bridge state preserved in mautrix_signal DB)
 - ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16** (source `/opt/livesync` STOPPED, intact as cold rollback; Caddy now proxies `notes.echo6.co` → `100.64.0.35:5984/5985`; `/etc/caddy/Caddyfile.bak-prelivesync` exists)
 - ~~TAK Server~~ — **decommissioned 2026-06-16** (was `/opt/tak-server-deploy`; archived to forge.echo6.co/matt/archive-tak-server; `tak.echo6.co` Caddy block + GoDaddy A record removed)
 - ~~SIGIL~~ — **decommissioned 2026-06-16** (was `/opt/sigil`; archived to forge.echo6.co/matt/archive-tak-server)
- Matrix Authentication Service (MAS) (port 8085, internal, Docker)
-  - Container: `matrix-mas` on `matrix-net`
-  - Handles login/logout/refresh/auth_metadata for Synapse
-  - Caddy routes: `/_matrix/client/*/login`, `/_matrix/client/*/logout`, `/_matrix/client/*/refresh`, `/_matrix/client/*/auth_metadata` → MAS (8085); `/_matrix/*` and `/_synapse/*` → Synapse (8008); default → MAS (8085)
-  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
 - Termix (port 8083, internal, Docker)
  - Container: `termix` on `termix_default` network
  - Image: `ghcr.io/lukegus/termix:latest`
@ -391,16 +402,6 @@ updated: 2026-06-18
  - Compose: `/opt/termix/` (inferred from Docker volume naming)
 - ~~echo6-agent~~ — **decommissioned 2026-06-16** (was systemd `echo6-agent.service` at `/opt/echo6-agent/`; archived to forge.echo6.co/matt/archive-echo6-agent; unit disabled)
 - ~~nexus-hub~~ + ~~nexus-agent~~ — **decommissioned 2026-06-16** (were systemd units at `/root/nexus-hub` and `/root/nexus-agent`; archived to forge.echo6.co/matt/archive-nexus-hub + archive-nexus-agent; units disabled)
- mautrix-signal bridge (mautrix-signal container, port 29328 internal)
-  - Image: dock.mau.dev/mautrix/signal:v0.2603.0
-  - Config: `/opt/matrix/mautrix-signal/config.yaml`
-  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
-  - DB: mautrix_signal on matrix-postgres
-  - Bot: @signalbot:echo6.co, management room !fDjIRTMjxILVQoAcEN:echo6.co
-  - E2BE enabled (MSC4190), double puppeting via doublepuppet.yaml
-  - Signal account: +12083080811 (@matt:echo6.co)
-  - Portals auto-create on incoming messages (no autocreate toggle available)
-  - Ref: `/home/zvx/projects/.ref/docs/matrix/mautrix_signal.md`

 ## Decommissioned Services