docs: migrate Authentik (SSO keystone) to edge2 CT 105

- Authentik -> edge2 CT 105 (Postgres pg_dump/restore; SECRET_KEY carried verbatim; zero-downtime until ~2s cutover)
- Multi-block Caddy cutover: auth.echo6.co + notes.echo6.co outpost/forward_auth -> 100.64.0.36:9000
- runbook: add reboot tailscale-before-docker gotcha; clarify dnsmasq must NOT be repointed (points at Caddy host)
- source left stopped + intact on Contabo as cold rollback

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-18 05:49:07 +00:00
commit 44f0257376
140 changed files with 4013 additions and 24 deletions

View file

@ -0,0 +1,228 @@
# Echo6 Environment Reference
## Proxmox Cluster (echo6-cluster)
Five nodes running Proxmox VE:
| Node | Local IP | Tailscale | Hardware | RAM | Purpose |
| ------- | ------------- | ----------- | ----------------------------------------------- | -------------- | -------------------------------- |
| data | 192.168.1.240 | 100.64.0.6 | AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD | 32GB DDR4-3200 | Database services |
| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility services, monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | Intel i7-12700T, 512GB NVMe | 32GB DDR4-3200 | Cloud storage, personal services |
| media | 192.168.1.243 | 100.64.0.3 | Intel i7-14700T, 2x 512GB NVMe | 32GB DDR5-5600 | Media server, *arr stack |
| toc | 192.168.1.244 | 100.64.0.13 | Workstation (i9-10900X) | 64GB DDR4 | GPU compute, AI/ML workloads |
### Node Storage Details
| Node | Primary Disk | Secondary Disk |
|------|-------------|----------------|
| data | Samsung SSD 980 1TB (NVMe) | SanDisk SDSSDH3 1TB (SATA SSD) |
| utility | WD PC SN740 512GB (NVMe) | — |
| cloud | SK Hynix HFS512GEJ9X164N 512GB (NVMe) | — |
| media | 2x Intel SSDPEKNU512GZH 512GB (NVMe) | — |
| toc | 512GB NVMe | — |
### Network Notes
- **media NIC:** Original Intel e1000e NIC crashes under sustained NFS load — replaced with USB Realtek RTL8153 GbE adapter on vmbr0
- **Tailscale DNS bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback DNS exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot
### TOC Node Details
- **Hardware:** Intel i9-10900X (20 threads), 64GB RAM (4×8GB + 2×16GB DDR4), 512GB NVMe, RTX A4000
- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
- **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
## Virtual Machines
| VM | Host | VMID | Local IP | Tailscale | Purpose |
|----|------|------|----------|-----------|---------|
| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — Open WebUI, Ollama, Qdrant, TEI, Claude Code |
| recon-vm | data | 1130 | 192.168.1.130 | 100.64.0.24 | RECON knowledge extraction pipeline, Files, Kiwix |
| arr | media | 105 | 192.168.1.160 | 100.64.0.18 | ARR media automation stack (Jellyfin, Sonarr, Radarr, etc.) |
### cortex VM Details
- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
- **Resources:** 16 threads, 32GB RAM, 300GB disk
- **Swap:** 32GB swapfile (`/swapfile`), `vm.swappiness=10` (prefer RAM, swap under pressure only). Provisioned 2026-04-23 to support bursty jobs (gdal_contour, tippecanoe) alongside Docker ML containers.
- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
- **Docker containers:** open-webui (8080), ollama (11434 w/ GPU), qdrant (6333), tei (8090)
- **User:** zvx (sudo, SSH keys from cluster)
- **Claude Code:** installed
### recon-vm Details
- **OS:** Ubuntu 24.04.4 LTS (cloud-init), kernel 6.8.0-110-generic
- **Resources:** 4 cores, 16GB RAM, 100GB disk
- **Software:** Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
- **Systemd services:** recon (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **NFS mounts:** pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
- **User:** zvx (sudo, SSH key auth)
- **Migrated from:** CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
- **Note:** Old CT 130 (LXC) has since been destroyed (verified 2026-06-15) — only VM 1130 remains.
### arr VM Details
- **OS:** Ubuntu 24.04 (cloud-init)
- **Resources:** 4 cores, 8GB RAM, 30GB disk on local-lvm
- **Software:** Docker 29.2.1, Tailscale, NFS client, sshpass, qemu-guest-agent
- **User:** zvx (sudo, SSH key from cortex)
- **NFS:** pi-nas:/export/arr → /mnt/arr (22TB, movies/tv/downloads)
- **Docker containers:** jellyfin (8096), jellyseer (5055), sonarr (8989), radarr (7878), prowlarr (9696), sabnzbd (8080), lidarr (8686/6595), navidrome (4533)
- **Docker network:** arr-net (bridge)
- **Config dirs:** /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd,lidarr}
## Key Servers
| Server | Local IP | Tailscale | Purpose |
|--------|----------|-----------|---------|
| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. MeshAI (CT 108) connects here via TCP |
| mt-isr | 192.168.1.141 | 100.100.0.5 (IdahoMesh) | Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI) |
| mt-burleybutte | 192.168.1.185 | — | Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN) |
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
| matt-desktop | 192.168.1.111 | 100.64.0.10 | Personal workstation (Windows, your PC) |
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge, Matrix |
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB |
*Last updated: 2026-06-18 — Added edge2 CT 105 (authentik, 10.10.10.23, 100.64.0.36, node 48, migrated 2026-06-18); previously added CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101*
## LXC Containers
| Container | Host | Local IP | Tailscale | Purpose |
|-----------|------|----------|-----------|---------|
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel) |
| caddy | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage (SearXNG, echo6.co) |
| immich | cloud (CT 120) | 192.168.1.182 | 100.64.0.2 | Immich photo management |
| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| meshai | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | MeshAI - LLM-powered Meshtastic assistant |
| archivist | utility (CT 118) | 192.168.1.118 | — | Archivist knowledge pipeline |
| argus | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | ARGUS - OSINT intelligence gathering platform |
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.23 | PeerTube video streaming |
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | WordPress for intermountainmesh.com |
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| authentik | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
## IP Allocation Scheme
| Range | Purpose |
|-------|---------|
| .1-.10 | Network infrastructure |
| .11-.99 | DHCP clients |
| .100-.149 | LXC containers |
| .150-.199 | VMs |
| .240-.250 | Proxmox hosts + bare metal |
| .251-.254 | Meshtastic nodes |
Full details: [[ip-allocation]]
## Headscale Node List
Current registered nodes (26 total):
| Node | Tailscale IP | Type |
|------|-------------|------|
| contabo | 100.64.0.1 | VPS |
| immich | 100.64.0.2 | LXC |
| media | 100.64.0.3 | Proxmox |
| cloud | 100.64.0.4 | Proxmox |
| utility | 100.64.0.5 | Proxmox |
| data | 100.64.0.6 | Proxmox |
| meshmonitor | 100.64.0.7 | LXC |
| caddy | 100.64.0.8 | LXC |
| aida-nebra | 100.64.0.9 | Pi |
| matt-desktop | 100.64.0.10 | Desktop |
| nextcloud | 100.64.0.11 | LXC |
| toc | 100.64.0.13 | Proxmox |
| cortex | 100.64.0.14 | VM |
| searxng | 100.64.0.15 | LXC |
| iphone-eud | 100.64.0.16 | Mobile |
| arr | 100.64.0.18 | VM |
| pi-nas | 100.64.0.21 | Pi |
| mesh-bridge | 100.64.0.22 | LXC |
| peertube | 100.64.0.23 | LXC |
| recon | 100.64.0.24 | VM |
| argus | 100.64.0.25 | LXC |
| edge2 | 100.64.0.26 | Proxmox/Contabo VPS |
| meshmonitor-dev | 100.64.0.27 | LXC |
| gl-a1300 | 100.64.0.29 | Router |
| bluefin | 100.64.0.30 | Desktop |
| wordpress | 100.64.0.31 | LXC |
| meshai | 100.64.0.32 | LXC |
| vaultwarden | 100.64.0.33 | LXC (edge2 CT 102) |
| forgejo | 100.64.0.34 | LXC (edge2 CT 103) — node id 46 |
| livesync | 100.64.0.35 | LXC (edge2 CT 104) — migrated 2026-06-16 |
| authentik | 100.64.0.36 | LXC (edge2 CT 105) — node id 48, migrated 2026-06-18 |
## IdahoMesh Headscale Node List
Separate Headscale instance on CT 106 (192.168.1.106), prefix 100.100.0.0/16.
Reachable from echo6 tailnet via mesh-bridge (CT 107).
| Node | Tailscale IP | User | Type |
|------|-------------|------|------|
| mesh-bridge | 100.100.0.3 | malice | LXC (bridge) |
| burley-butte | 100.100.0.1 | nebra | Pi (offline) |
| mt-isr | 100.100.0.5 | nebra | Pi Zero 2 W |
## SSH Access
**Standard user:** `zvx`
**Credentials:** Source from `/home/zvx/projects/.ref/credentials`
Most servers use SSH key auth. Exceptions noted below.
```bash
# SSH to any server (key auth)
ssh zvx@<ip-address>
# Examples
ssh zvx@192.168.1.244 # TOC (Proxmox host)
ssh zvx@192.168.1.150 # cortex VM
ssh zvx@192.168.1.241 # utility Proxmox
ssh root@100.64.0.1 # Contabo (via Tailscale)
ssh zvx@cortex # cortex via Tailscale hostname
ssh edge2 # edge2 via SSH config alias (admin@100.64.0.26)
```
### edge2 SSH access
edge2 is hardened: `PasswordAuthentication no`, root login disabled. SSH user is `admin` with passwordless sudo. The SSH config alias `edge2` maps to `admin@100.64.0.26`.
Authorized keys on edge2 (`/home/admin/.ssh/authorized_keys`):
- `echo6-contabo2-184.174.35.153` — the original provisioning key (cortex `~/.ssh/contabo2_ed25519`)
- `cortex` — cortex default `id_ed25519`
- `matt-desktop-wsl2` — WSL2 default key
- `cookie-sync` — Windows native SSH key
To access CTs on edge2: `ssh edge2` then `sudo pct exec <vmid> -- bash`.
### Password-auth hosts
These require password authentication (no SSH keys installed):
| Host | User | Password | Access |
|------|------|----------|--------|
| aida-nebra | zvx | 7redditGold | `sshpass -p '7redditGold' ssh zvx@aida-nebra` |
| mt-isr | isr | UfPsfwyMIUIKb1 | `sshpass -p 'UfPsfwyMIUIKb1' ssh isr@192.168.1.141` |
| mt-burleybutte | bb | (see credentials) | `sshpass -p '<pw>' ssh bb@192.168.1.185` |
| matt-desktop | administrator | Qw1290opzx | `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no administrator@192.168.1.111` |
| toc | root | 7redditGold | `sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@100.64.0.13` |
Use the Tailscale hostname (`aida-nebra`) or local IP (`192.168.1.253`) — both work for aida-nebra.
mt-isr is on IdahoMesh tailnet (100.100.0.5) — reachable from echo6 via bridge.
matt-desktop is accessible via local IP (192.168.1.111) or Tailscale (100.64.0.10) — requires explicit password auth flags.
## Key External IPs
| Purpose | IP |
|---------|-----|
| Home external (public services) | 199.6.36.163 |
| Contabo VPS | 5.189.158.149 |

View file

@ -0,0 +1,127 @@
# Echo6 Network IP Allocation
## IP Ranges
| Range | Purpose |
|-------|---------|
| .1-.10 | Network Infrastructure (router, switches, APs) |
| .11-.99 | DHCP (phones, laptops, guests) |
| .100-.149 | LXC Containers |
| .150-.199 | VMs |
| .200-.224 | Security Cameras |
| .225-.239 | IoT/Embedded Devices |
| .240-.250 | Proxmox Hosts + Bare Metal Servers |
| .251-.254 | Meshtastic Nodes |
---
## Current Allocations
### Network Infrastructure (.1-.10)
| IP | Device | Notes |
|----|--------|-------|
| .1 | Router/Gateway | |
| .2-.10 | (available) | Switches, APs |
### LXC Containers (.100-.149)
| IP | Container | Host | Purpose |
|----|-----------|------|---------|
| .100 | meshmonitor (CT 100) | utility | MeshMonitor web UI |
| .101 | caddy (CT 101) | utility | Home reverse proxy |
| .102 | searxng (CT 102) | utility | Echo6 Search (SearXNG) |
| .103 | argus (CT 103) | utility | ARGUS OSINT platform |
| .104 | meshing-around (CT 104) | utility | Mesh bot + WebGUI |
| .106 | meshtastic-hs (CT 106) | utility | IdahoMesh Headscale |
| .107 | mesh-bridge (CT 107) | utility | Dual-tailscaled bridge |
| .109 | mmud (CT 109) | utility | MMUD game daemon + web dashboard |
| .111 | mmud-embr (CT 111) | utility | MMUD SIM: EMBR (game server) |
| .112 | mmud-dcrg (CT 112) | utility | MMUD SIM: DCRG (broadcast) |
| .113 | mmud-grst (CT 113) | utility | MMUD SIM: GRST (Grist barkeep) |
| .114 | mmud-mrn (CT 114) | utility | MMUD SIM: MRN (Maren healer) |
| .115 | mmud-trvl (CT 115) | utility | MMUD SIM: TRVL (Torval merchant) |
| .116 | mmud-wspr (CT 116) | utility | MMUD SIM: WSPR (Whisper sage) |
| .118 | archivist (CT 118) | utility | Signal/Matrix archive bot |
| .130 | recon (VM 1130) | data | RECON pipeline (migrated from CT 130) |
| .144 | meshai (CT 108) | utility | MeshAI assistant |
| .170 | peertube (CT 110) | media | PeerTube streaming |
| .182 | immich (CT 120) | cloud | Immich photos |
| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO |
### edge2 LXC Containers (10.10.10.x, vmbr0)
edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate 10.10.10.0/24 subnet on vmbr0 (gw 10.10.10.1). No DHCP — all IPs are static. Storage: `local` (dir, no local-lvm).
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.10 | pdm | CT 100 | 100.64.0.28 | Proxmox Datacenter Manager |
| 10.10.10.11 | wordpress | CT 101 | 100.64.0.31 | WordPress for intermountainmesh.com |
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| 10.10.10.23 | authentik | CT 105 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
### VMs (.150-.199)
| IP | VM | Host | Purpose |
|----|-----|------|---------|
| .150-.199 | (available) | | |
### Security Cameras (.200-.224)
| IP | Camera | Location |
|----|--------|----------|
| .200-.224 | (available) | |
### IoT/Embedded (.225-.239)
| IP | Device | Notes |
|----|--------|-------|
| .225-.238 | (available) | |
| .239 | (reserved) | Former TOC IP |
### Proxmox Hosts + Bare Metal (.240-.250)
| IP | Hostname | Role | Tailscale |
|----|----------|------|-----------|
| .240 | data | Proxmox - DB node | 100.64.0.20 |
| .241 | utility | Proxmox - Utility services | 100.64.0.19 |
| .242 | cloud | Proxmox - Cloud storage | 100.64.0.22 |
| .243 | media | Proxmox - Media server | 100.64.0.21 |
| .244 | echo6-toc | LLM/AI server | 100.64.0.5 |
| .245-.250 | (available) | | |
### Meshtastic Nodes (.251-.254)
| IP | Node | Notes |
|----|------|-------|
| .251-.252 | (available) | |
| .253 | AIDA-NEBRA | Raspberry Pi running meshtasticd |
| .254 | (available) | |
---
## Tailscale IPs (100.64.0.x)
| Tailscale IP | Device | Local IP |
|--------------|--------|----------|
| 100.64.0.5 | echo6-toc | 192.168.1.244 |
| 100.64.0.6 | contabo-mailcow | 5.189.158.149 (external) |
| 100.64.0.18 | aida-nebra | 192.168.1.253 |
| 100.64.0.19 | utility | 192.168.1.241 |
| 100.64.0.20 | data | 192.168.1.240 |
| 100.64.0.21 | media | 192.168.1.243 |
| 100.64.0.22 | cloud | 192.168.1.242 |
| 100.64.0.26 | edge2 | 184.174.35.153 (external) |
| 100.64.0.28 | pdm (CT 100 on edge2) | 10.10.10.10 (vmbr0) |
| 100.64.0.33 | vaultwarden (CT 102 on edge2) — node id 45 | 10.10.10.20 (vmbr0) |
| 100.64.0.34 | forgejo (CT 103 on edge2) — node id 46 | 10.10.10.21 (vmbr0) |
| 100.64.0.35 | livesync (CT 104 on edge2) — hostname `livesync` | 10.10.10.22 (vmbr0) |
| 100.64.0.36 | authentik (CT 105 on edge2) — node id 48 | 10.10.10.23 (vmbr0) |
---
## Notes
- DHCP range starts at .11 to leave room for network infra
- LXC containers get .100-.149 for easy identification
- Meshtastic nodes at .251-.254 (max 4 hardware nodes)
- AIDA-NEBRA at .253 is technically a Meshtastic node (runs meshtasticd)
---
*Last updated: 2026-06-18 — Added edge2 CT 105 (authentik) at 10.10.10.23 / 100.64.0.36 (node id 48); previously added CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*