Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 06:02:16 +00:00 · 2026-04-13 06:02:16 +00:00 · e9231ac24a
commit e9231ac24a
parent 89834796ff
93 changed files with 51223 additions and 254 deletions
--- a/docs/hardware/environment.md
+++ b/docs/hardware/environment.md
@ -4,17 +4,32 @@

 Five nodes running Proxmox VE:

-| Node | Local IP | Tailscale | Hardware | Purpose |
-|------|----------|-----------|----------|---------|
-| data | 192.168.1.240 | 100.64.0.6 | Mini PC | Database services |
-| utility | 192.168.1.241 | 100.64.0.5 | Mini PC | Utility services, monitoring |
-| cloud | 192.168.1.242 | 100.64.0.4 | Mini PC | Cloud storage, personal services |
-| media | 192.168.1.243 | 100.64.0.3 | Mini PC | Media server, *arr stack |
-| toc | 192.168.1.244 | 100.64.0.13 | Workstation | GPU compute, AI/ML workloads |
+| Node | Local IP | Tailscale | Hardware | RAM | Purpose |
+|------|----------|-----------|----------|-----|---------|
+| data | 192.168.1.240 | 100.64.0.6 | AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD | 32GB DDR4-3200 | Database services |
+| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility services, monitoring |
+| cloud | 192.168.1.242 | 100.64.0.4 | Intel i7-12700T, 512GB NVMe | 32GB DDR4-3200 | Cloud storage, personal services |
+| media | 192.168.1.243 | 100.64.0.3 | Intel i7-14700T, 2x 512GB NVMe | 32GB DDR5-5600 | Media server, *arr stack |
+| toc | 192.168.1.244 | 100.64.0.13 | Workstation (i9-10900X) | 64GB DDR4 | GPU compute, AI/ML workloads |
+
+### Node Storage Details
+
+| Node | Primary Disk | Secondary Disk |
+|------|-------------|----------------|
+| data | Samsung SSD 980 1TB (NVMe) | SanDisk SDSSDH3 1TB (SATA SSD) |
+| utility | WD PC SN740 512GB (NVMe) | — |
+| cloud | SK Hynix HFS512GEJ9X164N 512GB (NVMe) | — |
+| media | 2x Intel SSDPEKNU512GZH 512GB (NVMe) | — |
+| toc | 512GB NVMe | — |
+
+### Network Notes
+
+- **media NIC:** Original Intel e1000e NIC crashes under sustained NFS load — replaced with USB Realtek RTL8153 GbE adapter on vmbr0
+- **Tailscale DNS bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback DNS exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot

 ### TOC Node Details

- **Hardware:** Intel i9-10900X (20 threads), 48GB RAM, 512GB NVMe, RTX A4000
+- **Hardware:** Intel i9-10900X (20 threads), 64GB RAM (4×8GB + 2×16GB DDR4), 512GB NVMe, RTX A4000
 - **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
 - **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
 - **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
@ -23,7 +38,8 @@ Five nodes running Proxmox VE:

 | VM | Host | VMID | Local IP | Tailscale | Purpose |
 |----|------|------|----------|-----------|---------|
-| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — LLMs, ARGUS, Aurora, model training |
+| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — Open WebUI, Ollama, Qdrant, TEI, Claude Code |
+| arr | media | 105 | 192.168.1.160 | 100.64.0.18 | ARR media automation stack (Jellyfin, Sonarr, Radarr, etc.) |

 ### cortex VM Details

@ -31,24 +47,48 @@ Five nodes running Proxmox VE:
 - **Resources:** 16 threads, 32GB RAM, 300GB disk
 - **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
 - **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
+- **Docker containers:** open-webui (8080), ollama (11434 w/ GPU), qdrant (6333), tei (8090)
 - **User:** zvx (sudo, SSH keys from cluster)
- **Claude Code:** v2.1.34 installed
+- **Claude Code:** installed
+
+### arr VM Details
+
+- **OS:** Ubuntu 24.04 (cloud-init)
+- **Resources:** 4 cores, 8GB RAM, 30GB disk on local-lvm
+- **Software:** Docker 29.2.1, Tailscale, NFS client, sshpass, qemu-guest-agent
+- **User:** zvx (sudo, SSH key from cortex)
+- **NFS:** pi-nas:/export/arr → /mnt/arr (22TB, movies/tv/downloads)
+- **Docker containers:** jellyfin (8096), jellyseer (5055), sonarr (8989), radarr (7878), prowlarr (9696), sabnzbd (8080), lidarr (8686/6595), navidrome (4533)
+- **Docker network:** arr-net (bridge)
+- **Config dirs:** /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd,lidarr}

 ## Key Servers

 | Server | Local IP | Tailscale | Purpose |
 |--------|----------|-----------|---------|
-| aida-nebra | 192.168.1.253 | 100.64.0.9 | Meshtastic node (meshtasticd on Pi) |
-| matt-desktop | — | 100.64.0.10 | Personal workstation |
-| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge |
+| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. MeshAI (CT 108) connects here via TCP |
+| mt-isr | 192.168.1.141 | 100.100.0.5 (IdahoMesh) | Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI) |
+| mt-burleybutte | 192.168.1.185 | — | Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN) |
+| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
+| matt-desktop | 192.168.1.111 | 100.64.0.10 | Personal workstation (Windows, your PC) |
+| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge, Matrix |

 ## LXC Containers

 | Container | Host | Local IP | Tailscale | Purpose |
 |-----------|------|----------|-----------|---------|
-| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring |
+| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel) |
 | caddy | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
-| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | SearXNG metasearch engine |
+| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage (SearXNG, echo6.co) |
+| advbbs | utility (CT 103) | 192.168.1.103 | 100.64.0.31 | Meshtastic sim node (ADVBBS) |
+| immich | cloud (CT 120) | 192.168.1.182 | 100.64.0.2 | Immich photo management |
+| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
+| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
+| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
+| meshai | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | MeshAI - LLM-powered Meshtastic assistant |
+| archivist | utility (CT 118) | 192.168.1.118 | — | Archivist knowledge pipeline |
+| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.23 | PeerTube video streaming |
+| recon | data (CT 130) | 192.168.1.130 | 100.64.0.24 | RECON knowledge extraction pipeline |

 ## IP Allocation Scheme

@ -65,11 +105,12 @@ Full details: `/home/zvx/projects/utility/ip-allocation.md`

 ## Headscale Node List

-Current registered nodes (12 total):
+Current registered nodes (25 total):

 | Node | Tailscale IP | Type |
 |------|-------------|------|
 | contabo | 100.64.0.1 | VPS |
+| immich | 100.64.0.2 | LXC |
 | media | 100.64.0.3 | Proxmox |
 | cloud | 100.64.0.4 | Proxmox |
 | utility | 100.64.0.5 | Proxmox |
@ -78,17 +119,42 @@ Current registered nodes (12 total):
 | caddy | 100.64.0.8 | LXC |
 | aida-nebra | 100.64.0.9 | Pi |
 | matt-desktop | 100.64.0.10 | Desktop |
+| nextcloud | 100.64.0.11 | LXC |
 | toc | 100.64.0.13 | Proxmox |
 | cortex | 100.64.0.14 | VM |
 | searxng | 100.64.0.15 | LXC |
+| iphone-eud | 100.64.0.16 | Mobile |
+| arr | 100.64.0.18 | VM |
+| pi-nas | 100.64.0.21 | Pi |
+| mesh-bridge | 100.64.0.22 | LXC |
+| peertube | 100.64.0.23 | LXC |
+| recon | 100.64.0.24 | LXC |
+| meshmonitor-dev | 100.64.0.27 | LXC |
+| gl-a1300 | 100.64.0.29 | Router |
+| bluefin | 100.64.0.30 | Desktop |
+| advbbs | 100.64.0.31 | LXC |
+| meshai | 100.64.0.32 | LXC |
+
+## IdahoMesh Headscale Node List
+
+Separate Headscale instance on CT 106 (192.168.1.106), prefix 100.100.0.0/16.
+Reachable from echo6 tailnet via mesh-bridge (CT 107).
+
+| Node | Tailscale IP | User | Type |
+|------|-------------|------|------|
+| mesh-bridge | 100.100.0.3 | malice | LXC (bridge) |
+| burley-butte | 100.100.0.1 | nebra | Pi (offline) |
+| mt-isr | 100.100.0.5 | nebra | Pi Zero 2 W |

 ## SSH Access

 **Standard user:** `zvx`
 **Credentials:** Source from `/home/zvx/projects/.ref/credentials`

+Most servers use SSH key auth. Exceptions noted below.
+
 ```bash
-# SSH to any server
+# SSH to any server (key auth)
 ssh zvx@<ip-address>

 # Examples
@ -99,6 +165,22 @@ ssh root@100.64.0.1    # Contabo (via Tailscale)
 ssh zvx@cortex         # cortex via Tailscale hostname
 ```

+### Password-auth hosts
+
+These require password authentication (no SSH keys installed):
+
+| Host | User | Password | Access |
+|------|------|----------|--------|
+| aida-nebra | zvx | 7redditGold | `sshpass -p '7redditGold' ssh zvx@aida-nebra` |
+| mt-isr | isr | UfPsfwyMIUIKb1 | `sshpass -p 'UfPsfwyMIUIKb1' ssh isr@192.168.1.141` |
+| mt-burleybutte | bb | (see credentials) | `sshpass -p '<pw>' ssh bb@192.168.1.185` |
+| matt-desktop | administrator | Qw1290opzx | `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no administrator@192.168.1.111` |
+| toc | root | 7redditGold | `sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@100.64.0.13` |
+
+Use the Tailscale hostname (`aida-nebra`) or local IP (`192.168.1.253`) — both work for aida-nebra.
+mt-isr is on IdahoMesh tailnet (100.100.0.5) — reachable from echo6 via bridge.
+matt-desktop is accessible via local IP (192.168.1.111) or Tailscale (100.64.0.10) — requires explicit password auth flags.
+
 ## Key External IPs

 | Purpose | IP |
--- a/docs/services/services.md
+++ b/docs/services/services.md
@ -4,11 +4,16 @@

 | Service | Location | IP:Port | Access | Notes |
 |---------|----------|---------|--------|-------|
-| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring |
+| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel AutoAnnounce/AutoResponder) |
 | Utility Caddy | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
-| SearXNG | utility (CT 102) | 192.168.1.102:8080 | https://search.echo6.co | Metasearch engine (Docker) |
-| meshtasticd | aida-nebra | 192.168.1.253:4403 | Internal | Software Meshtastic node |
-| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider |
+| Echo6 Search (SearXNG) | utility (CT 102) | 192.168.1.102:8080 | https://echo6.co | Branded search homepage (Docker, custom theme) |
+| meshtasticd (AIDA-N2) | aida-nebra | 192.168.1.253:4403 | Internal | AIDA-N2(RPT,LLM) node !27780c47, Nebra 2W hat (ZebraHat), CLIENT_BASE role, fw 2.7.19. MeshAI (CT 108) connects via TCP localhost:4403 |
+| Meshtastic CLI | mt-isr | 192.168.1.141 | Internal | Station G2 WiFi bridge + TCP management |
+| meshtasticd | mt-burleybutte | 192.168.1.185:4403 | Internal | Software Meshtastic node (Nebra 2W hat) |
+| IdahoMesh Headscale | utility (CT 106) | 192.168.1.106:8080 | https://vpn.idahomesh.com | Meshtastic mesh VPN coordination |
+| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
+| MeshAI | utility (CT 108) | 192.168.1.144:4403 | Internal | LLM-powered Meshtastic assistant (Docker, Gemini Flash, Google grounding) |
+| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) |
 | Forge | Contabo | 5.189.158.149 | https://forge.echo6.co | Git server |
 | Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
 | Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
@ -17,6 +22,36 @@
 | Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
 | Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
 | Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
+| Immich | cloud (CT 120) | 192.168.1.182:2283 | https://immich.echo6.co | Photo management (Docker, NFS storage on pi-nas) |
+| Nextcloud | cloud (CT 121) | 192.168.1.183:11000 | https://nextcloud.echo6.co | Cloud storage (AIO Docker, NFS on pi-nas, SSO) |
+| Jellyfin | media (VM 105) | 192.168.1.160:8096 | https://jellyfin.echo6.co | Media server (Docker, NFS on pi-nas, SSO) |
+| Jellyseer | media (VM 105) | 192.168.1.160:5055 | https://requests.echo6.co | Media request management (Docker, SSO) |
+| Sonarr | media (VM 105) | 192.168.1.160:8989 | Internal | TV automation (Docker) |
+| Radarr | media (VM 105) | 192.168.1.160:7878 | Internal | Movie automation (Docker) |
+| Prowlarr | media (VM 105) | 192.168.1.160:9696 | Internal | Indexer manager (Docker) |
+| SABnzbd | media (VM 105) | 192.168.1.160:8080 | Internal | Usenet download client (Docker) |
+| PeerTube | media (CT 110) | 192.168.1.170:9000 | https://stream.echo6.co | Video streaming (native, NFS on pi-nas, SSO) |
+| WATCHTOWER | Contabo | 127.0.0.1:8099 | https://wt.echo6.co | Ops dashboard (Docker, Authentik forward auth) |
+| Open WebUI | cortex (VM 150) | 192.168.1.150:8080 | https://ai.echo6.co | AI chat interface (Docker, Ollama backend, SSO) |
+| Qdrant | cortex (VM 150) | 192.168.1.150:6333 | Internal | Vector database (Docker, RECON knowledge store) |
+| TEI | cortex (VM 150) | 192.168.1.150:8090 | Internal | Text embeddings (Docker, bge-m3 1024-dim) |
+| RECON | data (CT 130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
+| Files | data (CT 130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, Authentik forward auth) |
+| Matrix Synapse | Contabo | 127.0.0.1:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) |
+| Element Web | Contabo | 127.0.0.1:8088 | https://element.echo6.co | Matrix web client (Docker) |
+| mautrix-signal | Contabo | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) |
+| LiveSync | Contabo | 127.0.0.1:5984 | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) |
+| TAK Server | Contabo | 100.64.0.1:8446/8443/8089 | https://tak.echo6.co | TAK Server (Docker, Authentik forward auth on admin portal) |
+| SIGIL | Contabo | 100.64.0.1:8990 | https://tak.echo6.co/sigil | TAK management console (Docker, Authentik forward auth) |
+| Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
+| Echo6 Contabo Agent | Contabo | N/A (Matrix bot) | #contabo:echo6.co in echo6-ops space | Claude Code bridge — @contabo:echo6.co, session continuity, E2EE (systemd) |
+| mautrix-signal | Contabo | 29328 (internal) | Internal (matrix-net) | Signal bridge — @signalbot:echo6.co, E2BE, MSC4190, auto-portals |
+| Matrix MAS | Contabo | 127.0.0.1:8085 | Internal (via Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) |
+| Termix | Contabo | 0.0.0.0:8083 | Internal (no Caddy block) | Terminal sharing tool (Docker, ghcr.io/lukegus/termix:latest) |
+| Archivist | utility (CT 118) | 192.168.1.118 | Internal | Archivist knowledge pipeline — see archivist.ref for details |
+| pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
+| recon-sparse | cortex (VM 150) | 192.168.1.150:8091 | Internal | RECON sparse embedding service (systemd, bge-m3 model, port 8091) |
+| Samba | cortex (VM 150) | 192.168.1.150:445 | Internal | SMB file sharing — `//cortex/projects` → /home/zvx/projects (guest access) |

 ## Services by Server

@ -29,29 +64,243 @@
 - GPU compute VM (RTX A4000)
 - Claude Code host
 - Syncthing (syncs with Contabo)
- **Planned:** Ollama, Open-WebUI, LiteLLM, ARGUS, Aurora
+- Open WebUI (port 8080, https://ai.echo6.co, Docker, SSO via Authentik, Echo6 theme)
+  - Compose path: `/opt/open-webui/docker-compose.yml`
+  - Echo6 theme: togglable via "E6" button (bottom-right), persisted in localStorage
+  - Theme files bind-mounted from `/home/zvx/echo6-theme/` into container
+  - DEFAULT_USER_ROLE=user (new signups auto-activated, not pending)
+- Ollama (port 11434, internal, Docker with GPU)
+- Qdrant (port 6333, internal, Docker — vector DB for RECON)
+- TEI (port 8090, internal, Docker — bge-m3 embeddings for RECON)
+- PeerTube remote runner (peertube-runner service, Whisper auto-captioning via smart GPU/CPU wrapper, concurrency=2, MemoryMax=20G)
+- pt-transcoder (systemd: pt-transcoder.service, PeerTube H.265 NVENC transcoder)
+  - Script: `/opt/bulk-import/transcoder.py`
+  - MemoryMax=12G, Restart=always, RestartSec=60
+  - Depends on: nvidia-persistenced.service
+- recon-sparse (systemd: recon-sparse.service, RECON sparse embedding service)
+  - Script: `/opt/recon-sparse/sparse_embed_service.py --port 8091`
+  - Model: BAAI/bge-m3 (HuggingFace cache)
+  - Restart=on-failure, RestartSec=10
+- Samba (smbd/nmbd, system packages)
+  - Share: `//cortex/projects` → `/home/zvx/projects` (browseable, read-write, guest OK, force user/group zvx)
+  - Workgroup: WORKGROUP, standalone server
+- Echo6 Cortex Agent (systemd: echo6-agent.service, matrix-nio bot, @cortex:echo6.co)
+  - Install path: `/opt/echo6-agent/`
+  - Matrix space: echo6-ops, room: #cortex:echo6.co (E2EE, private)
+  - Session continuity via `claude -p --resume`, persistent per-room sessions
+  - `!new` resets conversation session
+  - Allowed users: @matt:echo6.co
+  - MAS user ID: 01KKX88ARGK0BTA1JMB2QVAW4C

 ### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7)
- MeshMonitor (port 8080)
+- MeshMonitor (port 8080, https://mesh.echo6.co)
+- Image: `meshmonitor:multichannel-new` (local build from zvx-echo6/meshmonitor fork, branch `feature/multi-channel-automation`)
+- Fork of Yeraze/meshmonitor with multi-channel AutoAnnounce and AutoResponder support (PR #2078 open upstream)

 ### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8)
 - Utility Caddy (reverse proxy for VPN-only services)

 ### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
- SearXNG metasearch engine (port 8080)
- Redis/Valkey cache
+- Echo6 Search — branded SearXNG homepage (port 8080, https://echo6.co)
+- Custom cyberpunk theme: JetBrains Mono font, cyan/yellow palette, dark backgrounds
+- Homepage: centered Echo6 logo + pill search bar (Google-style, viewport-locked no-scroll)
+- Results page: full-width two-column grid (results + sidebar), stretched search header
+- Top nav bar: `.//photos`, `.//mail`, waffle app launcher (11 services), login avatar
+- All nav links use Authentik launch URLs for seamless SSO pass-through
+- search.echo6.co permanently redirects to echo6.co (301)
+- Redis/Valkey cache (valkey container)
 - Compose path: `/opt/searxng/docker-compose.yml`
+- Theme files: `/opt/searxng/custom/` (bind-mounted into container)
+  - `templates/simple/base.html` — custom template (nav, CSS, waffle menu, footer)
+  - `templates/simple/index.html` — custom homepage (Echo6 logo replaces SearXNG title)
+  - `img/echo6-logo.png` — Echo6 logo (replaces SearXNG logo)
+  - `img/favicon.png` — Echo6 favicon
+- Config: `/opt/searxng/searxng-config/settings.yml` (instance_name: "Echo6", dark theme, center_alignment: false)
+- SearXNG version: 2026.2.6 (Docker image: searxng/searxng:latest)
+
+### utility - CT 103 (192.168.1.103 / Tailscale: 100.64.0.31)
+- Meshtastic sim node (advbbs)
+
+### utility - CT 108 (192.168.1.144 / Tailscale: 100.64.0.32)
+- MeshAI — LLM-powered Meshtastic mesh assistant (Docker)
+- Bot name: AIDA, node ID !27780c47, channel 8 whitelist
+- Image: ghcr.io/zvx-echo6/meshai:latest (GitHub Actions multi-arch build)
+- Backend: Gemini 2.5 Flash with Google Search grounding
+- Connects to meshtasticd **on aida-nebra** (192.168.1.253:4403) — the AIDA-N2 node !27780c47
+- Config TUI on port 7682 (`meshai --config`)
+- Commands: !help, !ping, !status, !weather, !reset, !clear
+- 7-day rolling conversation memory (SQLite), full history sent to LLM
+- Response: 175 char chunks × 3 messages max
+- Compose path: `/home/zvx/meshai/docker-compose.yml`
+
+### utility - CT 118 (192.168.1.118)
+- Archivist knowledge pipeline
+- 1 core, 1GB RAM, 8GB disk
+- Not registered in Headscale (no Tailscale)
+- See `/home/zvx/projects/.ref/archivist.ref` for implementation details
+
+### cloud - CT 120 (192.168.1.182 / Tailscale: 100.64.0.2)
+- Immich photo management (https://immich.echo6.co)
+- Port 2283
+- NFS storage from pi-nas (/mnt/immich)
+- Compose path: `/opt/immich/docker-compose.yml`
+
+### cloud - CT 121 (192.168.1.183 / Tailscale: 100.64.0.11)
+- Nextcloud AIO (https://nextcloud.echo6.co)
+- Apache port 11000, AIO management on 8080
+- NFS storage from pi-nas (/mnt/nextcloud)
+- SSO via Authentik OIDC
+
+### media - VM 105 (192.168.1.160 / Tailscale: 100.64.0.18)
+- ARR media automation stack (Docker)
+- Jellyfin media server (port 8096, https://jellyfin.echo6.co)
+- Jellyseer request management (port 5055, https://requests.echo6.co)
+- Sonarr TV automation (port 8989, internal)
+- Radarr movie automation (port 7878, internal)
+- Prowlarr indexer manager (port 9696, internal)
+- SABnzbd Usenet downloader (port 8080, internal)
+- NFS storage from pi-nas (/mnt/arr)
+- Config dirs: /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
+
+### media - CT 110 (192.168.1.170 / Tailscale: 100.64.0.23)
+- PeerTube video streaming (https://stream.echo6.co)
+- Native install (Node.js 22, PostgreSQL 16, Redis, nginx)
+- Port 9000 (PeerTube), proxied via nginx on port 80
+- NFS storage from pi-nas (/var/www/peertube/storage, /export/peertube)
+- SSO via Authentik OIDC (peertube-plugin-auth-openid-connect)
+- Privileged container (NFS bind-mount)
+- Auto-transcription enabled (remote runners on cortex, Whisper medium model)
+- **PeerTube Bulk Import Pipeline:**
+  - pt-downloader.service — YouTube channel downloader (yt-dlp, sliding window, cookie auth)
+  - pt-importer.service — Uploads downloaded videos to PeerTube via resumable upload API
+  - NordVPN (nordvpnd.service) — IP rotation for downloads
+  - Config: `/opt/bulk-import/config/` (channel-map.json, cookies.txt, downloader-state.json)
+  - Logs: `/opt/bulk-import/logs/`
+  - Pipeline dirs: `/var/www/peertube/storage/pipeline/{staging,completed,transcoded,failed}`
+
+### data - CT 130 (192.168.1.130 / Tailscale: 100.64.0.24)
+- RECON knowledge extraction pipeline
+- systemd service: `recon.service`
+- Dashboard + API on port 8420 (internal)
+- nginx file server on port 8888 (files.echo6.co)
+- Install: `/opt/recon/` (Python 3, Flask, venv)
+- NFS mount: pi-nas:/export/library → /mnt/library (PDF source)
+- Pipeline: Extract (PyPDF2→pdftotext→Tesseract→Gemini Vision) → Enrich (Gemini) → Embed (TEI/Qdrant)
+- DB: SQLite (status), Qdrant on cortex:6333 (vectors)
+- Backups: rsync to Contabo every 6hrs (concepts, text, DB, config)
+- Config: `/opt/recon/config.yaml`, keys in `/opt/recon/.env`
+- Docs: `/opt/recon/PROJECT-BIBLE.md`
+
+### utility - CT 106 (192.168.1.106)
+- IdahoMesh Headscale (https://vpn.idahomesh.com)
+- Container name: meshtastic-hs
+- Manages meshtastic mesh VPN (separate from echo6 Headscale on Contabo)
+- Users: malice, sidpatchy, nebra
+
+### utility - CT 107 (192.168.1.107)
+- mesh-bridge — dual tailscaled instance
+- Bridges echo6 (100.64.0.0/10) ↔ idahomesh (100.100.0.0/16) networks
+- NAT masquerade + subnet route advertisement
+- Echo6 clients need `--accept-routes` to reach idahomesh devices
+- iptables FORWARD rules must be BEFORE `ts-forward` jump (Tailscale drops cross-tailnet packets otherwise)
+- Echo6 socket: `/run/tailscale/tailscaled.sock` (port 41641)
+- IdahoMesh socket: `/var/run/tailscale-meshtastic/tailscaled.sock` (port 41642, tun=tailscale1)
+- Rules persisted: `/etc/iptables/rules.v4` via `iptables-restore.service`
+
+### pi-nas (192.168.1.245 / Tailscale: 100.64.0.21)
+- OpenMediaVault NAS (https://nas.echo6.co)
+- Port 80 (HTTP)
+- Internet Archive CLI (`ia` v5.7.2) installed for archive.org uploads

 ### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9)
- meshtasticd (software Meshtastic node)
+- **AIDA-N2(RPT,LLM)** — meshtasticd node `!27780c47` (short name: AIDA)
+- Hardware: Nebra 2W SX1262 hat (ZebraHat config in `/etc/meshtasticd/config.d/`)
+- Port: 4403 (default), firmware 2.7.19 (PORTDUINO/native)
+- Role: CLIENT_BASE, position: 42.574, -114.607 (manual)
+- MAC source: eth0 (derived MAC `00:bd:27:78:0c:47`)
+- MeshAI bot (CT 108) connects to this node via TCP `localhost:4403` (Docker network)
+- Service: `meshtasticd.service` (single instance, runs as user meshtastic)
+- Config: `/etc/meshtasticd/config.yaml` + `/etc/meshtasticd/config.d/ZebraHat_2W.yaml`
+- User: zvx, password auth (`sshpass -p '7redditGold' ssh zvx@aida-nebra`)
+
+### mt-isr (192.168.1.141 / IdahoMesh: 100.100.0.5)
+- Raspberry Pi Zero 2 W, Debian 13 (trixie), Waveshare ETH/USB HUB HAT
+- No meshtasticd (G2 managed via WiFi TCP, not local daemon)
+- Meshtastic Python CLI v2.7.7 in venv (`/home/isr/meshtastic-cli/`)
+- Tailscale on IdahoMesh tailnet (vpn.idahomesh.com, nebra user)
+- WiFi hotspot: ISR-MESH (192.168.4.0/24, PMF disabled for ESP32 compatibility)
+- Station G2 radio connected via WiFi at 192.168.4.241, managed via TCP
+- G2 config: Freq51 (ch0, psk=1A==) + MediumFast (ch1), MEDIUM_FAST preset, ch=51, txPower=11
+- G2 gold config backup: `isr@192.168.1.141:~/backups/g2-gold-config.yaml`
+- DNS bootstrap drop-in for tailscaled (reboot-safe)
+- User: isr, password auth (see credentials)
+
+### mt-burleybutte (192.168.1.185)
+- meshtasticd (software Meshtastic node, Nebra 2W hat)
+- Raspberry Pi OS, user bb
+- Static MAC: A7:A1:30:79:BB:BB
+- Tailscale registered on IdahoMesh Headscale (vpn.idahomesh.com) under malice user

 ### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
- Authentik (SSO)
+- Authentik (SSO, Echo6 branded — custom CSS, dark theme, logo, favicon, flow titles)
 - Forge (Git)
 - Headscale (mesh VPN)
 - Mailcow (email)
 - Vaultwarden (passwords)
 - Syncthing (syncs with cortex)
+- WATCHTOWER (ops dashboard, port 8099, https://wt.echo6.co)
+- Matrix Synapse homeserver (port 8008, https://matrix.echo6.co, Docker, SSO via Authentik)
+- Element Web client (port 8088, https://element.echo6.co, Docker)
+- mautrix-signal bridge (port 29328 internal, Docker, E2BE with MSC4190)
+  - Image: `dock.mau.dev/mautrix/signal:v0.2603.0`
+  - Container: `mautrix-signal` on `matrix-net`
+  - Config: `/opt/matrix/mautrix-signal/config.yaml`
+  - Registration: `/opt/matrix/synapse/registration.yaml` + `/opt/matrix/synapse/doublepuppet.yaml`
+  - Database: `mautrix_signal` on `matrix-postgres` (role: `mautrix_signal`, minimal grants)
+  - Bot user: `@signalbot:echo6.co`, device: `UPX4KKLZVY`
+  - Permissions: `@matt:echo6.co` = admin, `echo6.co` = user
+  - Double puppeting: appservice-based (doublepuppet.yaml as_token)
+  - Encryption: E2BE enabled (allow+default+require), MSC4190, self-signed cross-signing keys
+- Compose path: `/opt/matrix/docker-compose.yml`
+- Backup: daily at 3AM, 14-day retention (synapse + mas + mautrix_signal databases)
+- LiveSync CouchDB (port 5984, https://notes.echo6.co, Docker, JWT auth)
+- LiveSync Provisioner (port 5985, https://notes.echo6.co/_provision/, Docker, Authentik forward auth)
+- Compose path: `/opt/livesync/docker-compose.yml`
+- Per-user ES512 key pairs, encrypted setup URIs, per-user CouchDB databases
+- TAK Server (port 8446 web admin, 8443 mutual TLS API, 8089 TLS for EUDs)
+  - https://tak.echo6.co (Authentik forward auth on admin portal)
+  - Compose path: `/opt/tak-server-deploy/docker-compose.yml`
+  - Certs: `/opt/tak-server-deploy/tak/certs/files/`
+  - Container names: `tak-server-deploy-tak-1`, `tak-server-deploy-db-1`
+- SIGIL console (port 8990, https://tak.echo6.co/sigil, Authentik forward auth)
+  - Compose path: `/opt/sigil/docker-compose.yml`
+- Matrix Authentication Service (MAS) (port 8085, internal, Docker)
+  - Container: `matrix-mas` on `matrix-net`
+  - Handles login/logout/refresh/auth_metadata for Synapse
+  - Caddy routes: `/_matrix/client/*/login`, `/_matrix/client/*/logout`, `/_matrix/client/*/refresh`, `/_matrix/client/*/auth_metadata` → MAS (8085); `/_matrix/*` and `/_synapse/*` → Synapse (8008); default → MAS (8085)
+  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
+- Termix (port 8083, internal, Docker)
+  - Container: `termix` on `termix_default` network
+  - Image: `ghcr.io/lukegus/termix:latest`
+  - Port: 8080→8083 (bound to 0.0.0.0, NOT 127.0.0.1)
+  - Volume: `termix_termix-data` → `/app/data`
+  - No Caddy block — direct access only on port 8083
+  - Compose: `/opt/termix/` (inferred from Docker volume naming)
+- Echo6 Contabo Agent (systemd: echo6-agent.service, matrix-nio bot, @contabo:echo6.co)
+  - Install path: `/opt/echo6-agent/`
+  - Claude Code bridge with session continuity + E2EE
+  - Watches #contabo:echo6.co in echo6-ops space
+  - CLAUDE_CWD=/root, runs as root
+- mautrix-signal bridge (mautrix-signal container, port 29328 internal)
+  - Image: dock.mau.dev/mautrix/signal:v0.2603.0
+  - Config: `/opt/matrix/mautrix-signal/config.yaml`
+  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
+  - DB: mautrix_signal on matrix-postgres
+  - Bot: @signalbot:echo6.co, management room !fDjIRTMjxILVQoAcEN:echo6.co
+  - E2BE enabled (MSC4190), double puppeting via doublepuppet.yaml
+  - Signal account: +12083080811 (@matt:echo6.co)
+  - Portals auto-create on incoming messages (no autocreate toggle available)
+  - Ref: `/home/zvx/projects/.ref/mautrix_signal.ref`

 ## Adding New Services

@ -66,3 +315,32 @@ When deploying a new service, update this file with:

 - **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP
 - **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy
+
+### Lidarr on Steroids (lidarr.echo6.co)
+- **Container:** lidarr (youegraillot/lidarr-on-steroids:latest)
+- **Host:** media VM 105 (192.168.1.160)
+- **Ports:** 8686 (Lidarr), 6595 (Deemix)
+- **Network:** arr-net
+- **Config:** /opt/arr/lidarr/config (Lidarr), /opt/arr/lidarr/config_deemix (Deemix)
+- **Compose:** /opt/arr/docker-compose.yml
+- **Music root:** /mnt/arr/music (NFS from pi-nas)
+- **Downloads:** /mnt/arr/downloads (shared with SABnzbd)
+- **API key:** 78f026ec93a94d8eb3177816b74a57b7
+- **Caddy:** lidarr.echo6.co -> 100.64.0.18:8686 (Authentik forward auth)
+- **Prowlarr:** fullSync configured
+- **SABnzbd:** configured (music category)
+- **Deemix:** port 6595, NOT exposed via Caddy (Tailscale-only access)
+- **PUID/PGID:** 1000/1000, TZ: America/Boise
+
+### Navidrome (navidrome.echo6.co)
+- **Container:** navidrome (deluan/navidrome:latest)
+- **Host:** media VM 105 (192.168.1.160)
+- **Port:** 4533
+- **Network:** arr-net
+- **Data volume:** arr_navidrome-data (named Docker volume)
+- **Music volume:** /mnt/arr/music (read-only, shared with Lidarr)
+- **Compose:** /opt/arr/docker-compose.yml
+- **Caddy:** navidrome.echo6.co -> 100.64.0.18:4533 (Authentik forward auth)
+- **User:** 1000:1000
+- **Scan schedule:** every 1 hour
+- **Admin setup:** First login at https://navidrome.echo6.co creates admin account
--- a/docs/services/usenet.md
+++ b/docs/services/usenet.md
@ -0,0 +1,53 @@
+# Usenet Configuration
+
+> **Credentials:** All passwords and API keys stored in `/home/zvx/projects/.ref/credentials`
+
+## Infrastructure
+
+| Component | Location | IP | Port | Access |
+|-----------|----------|-----|------|--------|
+| SABnzbd | arr VM (media VM 105) | 192.168.1.160 | 8080 | Internal only (LAN/Tailscale) |
+
+## Usenet Providers
+
+### Giganews
+
+| Field | Value |
+|-------|-------|
+| Host | news.giganews.com |
+| Port | 563 |
+| SSL | Yes |
+| Username | gn1369344 |
+| Password | (see .ref/credentials) |
+| Connections | 100 |
+
+### EasyUsenet
+
+| Field | Value |
+|-------|-------|
+| Host | reader.easyusenet.nl |
+| Port | 563 |
+| SSL | Yes |
+| Username | esy_6926b8b819b97 |
+| Password | (see .ref/credentials) |
+| Connections | 100 |
+| Speed | 2000 Mbit/s |
+| Retention | 3700+ days |
+| Alt Ports (SSL) | 443 |
+| Alt Ports (non-SSL) | 119, 80 |
+
+## Usenet Indexers
+
+### NZBgeek
+
+| Field | Value |
+|-------|-------|
+| URL | https://api.nzbgeek.info |
+| API Key | (see .ref/credentials) |
+
+### NinjaCentral
+
+| Field | Value |
+|-------|-------|
+| URL | https://ninjacentral.co.za |
+| API Key | (see .ref/credentials) |
--- a/docs/software/authentik.md
+++ b/docs/software/authentik.md
@ -2,7 +2,7 @@

 ## Location

- **Server:** Contabo (5.189.158.149 / 100.64.0.6)
+- **Server:** Contabo (5.189.158.149 / 100.64.0.1)
 - **URL:** https://auth.echo6.co
 - **Internal Port:** 9000

@ -10,6 +10,12 @@

 API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN`

+```bash
+# Test API access
+curl -s "https://auth.echo6.co/api/v3/core/applications/" \
+  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" | python3 -m json.tool
+```
+
 ## Flow UUIDs

 Required for OAuth2 provider creation:
@ -21,28 +27,88 @@ Required for OAuth2 provider creation:
 | Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` |
 | Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` |

+## Signing Key
+
+| Key | UUID |
+|-----|------|
+| authentik Self-signed Certificate | `09f508f0-6b8e-4031-8563-0d3cebf86868` |
+
+## Property Mappings (Standard OIDC Set)
+
+Include these three in every provider:
+
+| Mapping | UUID |
+|---------|------|
+| OpenID Connect scope: openid | `c6426dad-0d85-4daa-89fe-cb850ad4bfd7` |
+| Echo6 OAuth: email (verified=true) | `02c22323-da89-457a-bc12-7f4dd6a3d8ab` |
+| OpenID Connect scope: profile | `113ab791-fa04-4e8c-b103-09d706bc21b4` |
+
+**Note:** The default email scope (`096b0d6f`) was changed in 2025.10 to return `email_verified: false`. Our custom scope (`02c22323`) overrides this to always return `true`. All 14 OAuth2 providers were migrated to use the custom scope on 2026-02-16.
+
+## Current OAuth2 Providers
+
+| PK | Name | Client ID | Application | Redirect URI |
+|----|------|-----------|-------------|--------------|
+| 1 | Mailcow | mailcow | Mailcow | `https://mail.echo6.co/sso/oidc` |
+| 2 | Forgejo | forgejo | Forgejo | `https://forge.echo6.co/user/oauth2/Authentik/callback` |
+| 3 | Vaultwarden | vaultwarden | Vaultwarden | `https://vault.echo6.co/identity/connect/callback` |
+| 4 | Proxmox | proxmox | Proxmox VE | `https://proxmox.echo6.co` (regex) |
+| 5 | Headscale | headscale | Headscale VPN | `https://vpn.echo6.co/oidc/callback` |
+| 6 | Headplane | headplane | Headplane | `https://vpn.echo6.co/admin/oidc/callback` |
+| 8 | Nextcloud | nextcloud | Nextcloud | `https://nextcloud.echo6.co/apps/oidc_login/oidc` |
+| 9 | Immich | immich | Immich | `https://immich.echo6.co/auth/login`, `/api/oauth/mobile-redirect` |
+| 10 | jellyfin | jellyfin | Jellyfin | `https://jellyfin.echo6.co/sso/OID/redirect/Authentik` |
+| 11 | jellyseer | jellyseer | Jellyseer | `https://requests.echo6.co/(login\|api/v1/auth/oidc-callback).*` (regex) |
+| 12 | PeerTube | peertube | PeerTube | `https://stream.echo6.co/plugins/auth-openid-connect/...` |
+| 13 | WATCHTOWER | watchtower | WATCHTOWER | Forward auth (proxy provider) |
+| 14 | Open WebUI | open-webui | Open WebUI | `https://ai.echo6.co/oauth/oidc/callback` |
+| 15 | Matrix | 93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T | Matrix | `https://matrix.echo6.co/_synapse/client/oidc/callback` |
+| 16 | Files Forward Auth | — | Files | Forward auth (proxy provider) |
+| 17 | LiveSync Provisioner | ZBoLdYmxlSUyMqgekswIPS4YuaeBn5uCr8GtWm5H | LiveSync | Forward auth (proxy provider) |
+
+## Groups
+
+| Name | PK | Superuser | Members | Used By |
+|------|----|-----------|---------|----- ---|
+| authentik Admins | `9944e153-f860-4443-81d1-ae544f611806` | Yes | akadmin, matt | All apps (admin access) |
+| media-users | `0820b2b8-6c54-4c20-9a0a-872820e6d9ea` | No | jodie, matt | Jellyfin, Jellyseer, PeerTube |
+| ai-users | `0631b273-cfd8-4ed1-afa6-e262d0dc5a69` | No | matt | Open WebUI |
+| cloud-users | `db3cbf5d-8057-4e33-8e8d-95bfdb35fbac` | No | — | Immich, Nextcloud |
+| communication-users | `31bce176-cd86-4aea-8db3-a57e03d5c2d1` | No | — | Mailcow, Matrix |
+| productivity-users | `698d80c7-7c29-43cd-b5d4-9eb24c85a6cc` | No | — | — |
+| security-users | `f345a043-c2a4-4906-a43b-9860eae86ee1` | No | — | — |
+| proxmox_admins | `d85a868d-7d1e-4585-92a8-b8bb86771b53` | No | akadmin, matt | Proxmox VE |
+| proxmox_users | `cf26703a-a824-47dd-9550-30b848a8ce5f` | No | — | Proxmox VE |
+| authentik Read-only | `ce03664b-46f3-43b7-9967-5f65d591fdb6` | No | — | RBAC read-only role |
+| livesync-users | `8e575a86-326e-4df8-8828-8379d8ab861f` | No | matt | LiveSync |
+
+## Application Access Pattern
+
+Every application uses `policy_engine_mode: "any"` with two group bindings:
+
+1. **authentik Admins** — gives admin/superuser access to all apps
+2. **Service-specific group** — controls which regular users can access the app
+
+Users must be in at least one bound group to access the application.
+
 ## Create New API Token

 ```bash
-ssh root@100.64.0.6 'docker exec authentik-server ak shell -c "
-from authentik.core.models import Token, User
-user = User.objects.get(username=\"akadmin\")
-token, created = Token.objects.get_or_create(
-    identifier=\"token-name\",
-    user=user,
-    defaults={\"intent\": \"api\", \"expiring\": False}
-)
-print(token.key)
-"'
+ssh root@100.64.0.1 'docker exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.core.models import Token, TokenIntents, User
+user = User.objects.get(username="akadmin")
+Token.objects.filter(identifier="token-name").delete()
+t = Token(identifier="token-name", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
+t.save()
+print(t.key)
+PYEOF
 ```

-## Quick OAuth2 Provider Creation
+## Full OAuth2 App Setup (Provider + Application + Group + Bindings)
+
+### Step 1: Create the OAuth2 provider

 ```bash
-# Source credentials
-source /home/zvx/projects/.ref/credentials
-
-# Create provider
 curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  -H "Content-Type: application/json" \
@ -50,13 +116,25 @@ curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
    "name": "AppName",
    "authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd",
    "invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a",
+    "property_mappings": [
+      "c6426dad-0d85-4daa-89fe-cb850ad4bfd7",
+      "02c22323-da89-457a-bc12-7f4dd6a3d8ab",
+      "113ab791-fa04-4e8c-b103-09d706bc21b4"
+    ],
    "client_type": "confidential",
    "client_id": "appname",
+    "client_secret": "<generate-a-long-random-secret>",
    "redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}],
-    "sub_mode": "user_username"
+    "sub_mode": "user_username",
+    "signing_key": "09f508f0-6b8e-4031-8563-0d3cebf86868",
+    "include_claims_in_id_token": true
  }'
+# Note the "pk" from the response — needed for Step 2
+```

-# Create application (use pk from provider response)
+### Step 2: Create the application
+
+```bash
 curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  -H "Content-Type: application/json" \
@ -64,14 +142,232 @@ curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
    "name": "AppName",
    "slug": "appname",
    "provider": PROVIDER_PK,
-    "meta_launch_url": "https://app.echo6.co"
+    "meta_launch_url": "https://app.echo6.co",
+    "policy_engine_mode": "any"
  }'
+# Note the "pk" from the response — needed for Step 4
+```
+
+### Step 3: Create the user group
+
+```bash
+curl -s -X POST "https://auth.echo6.co/api/v3/core/groups/" \
+  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "appname-users",
+    "is_superuser": false,
+    "users": [7]
+  }'
+# user 7 = matt. Note the "pk" from the response.
+```
+
+### Step 4: Bind groups to the application
+
+```bash
+# Bind authentik Admins
+curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
+  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "target": "APPLICATION_PK",
+    "group": "9944e153-f860-4443-81d1-ae544f611806",
+    "order": 0, "enabled": true, "negate": false, "timeout": 30
+  }'
+
+# Bind service-specific group
+curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
+  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "target": "APPLICATION_PK",
+    "group": "GROUP_PK",
+    "order": 0, "enabled": true, "negate": false, "timeout": 30
+  }'
+```
+
+### Step 5: Verify
+
+```bash
+# OIDC discovery should return endpoints
+curl -s "https://auth.echo6.co/application/o/appname/.well-known/openid-configuration" | python3 -m json.tool
+
+# Check bindings
+curl -s "https://auth.echo6.co/api/v3/policies/bindings/?target=APPLICATION_PK" \
+  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN"
 ```

 ## Common Redirect URI Patterns

 | Application Type | Redirect URI Pattern |
 |------------------|---------------------|
-| Web app | `https://app.echo6.co/callback` |
-| Web app (oauth) | `https://app.echo6.co/oauth/callback` |
+| Web app (generic) | `https://app.echo6.co/callback` |
+| Web app (oauth path) | `https://app.echo6.co/oauth/callback` |
+| Open WebUI (OIDC) | `https://app.echo6.co/oauth/oidc/callback` |
+| Forgejo | `https://app.echo6.co/user/oauth2/Authentik/callback` |
+| Jellyfin (SSO plugin) | `https://app.echo6.co/sso/OID/redirect/Authentik` |
 | Caddy forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` |
+
+## Users
+
+| PK | Username | Name | Email |
+|----|----------|------|-------|
+| 6 | akadmin | authentik Default Admin | root@example.com |
+| 7 | matt | Matt Johnson | matt@echo6.co |
+| 9 | jodie | Jodie | johnsonsinidaho@gmail.com |
+
+## Email Invitation System
+
+### SMTP Configuration
+
+Authentik sends email via Mailcow:
+
+| Setting | Value |
+|---------|-------|
+| SMTP Host | mail.echo6.co |
+| SMTP Port | 587 (STARTTLS) |
+| Username | no-reply@echo6.co |
+| From | no-reply@echo6.co |
+
+**Important:** The no-reply@echo6.co mailbox MUST have `authsource=mailcow` in the Mailcow database (not `generic-oidc`). If it gets reset to `generic-oidc`, SMTP auth will fail because Dovecot tries to authenticate via Authentik SSO instead of the local password. Fix with:
+```sql
+docker exec mailcowdockerized-mysql-mailcow-1 mysql -u mailcow -p<DBPASS> mailcow \
+  -e "UPDATE mailbox SET authsource='mailcow' WHERE username='no-reply@echo6.co'"
+```
+
+### Enrollment Flow
+
+| Component | PK |
+|-----------|-----|
+| Flow (invitation-enrollment) | `184a9e20-f5c5-4b44-8775-266a568439c0` |
+| Invitation stage (enrollment-invitation) | `994252a6-a659-4304-b9aa-a6591857a53b` |
+| Prompt stage (enrollment-credentials) | `a5e2a2f1-95c9-4627-a3dd-e0915ff64cda` |
+| User write stage (enrollment-user-write) | `05f565e0-9d64-4ecc-b738-8ffc697ad829` |
+| User login stage (enrollment-user-login) | `f23958ea-47bd-4fb2-a657-b6f85fc6331a` |
+
+Stage order: Invitation (10) → Prompt (20) → User Write (30) → User Login (40)
+
+Flow settings: `authentication=require_unauthenticated`, `continue_flow_without_invitation=false`
+
+User write creates users under `users/enrolled` path.
+
+### Email Automation
+
+Uses the community pattern from [authentik/discussions/13305](https://github.com/goauthentik/authentik/discussions/13305):
+
+| Component | PK |
+|-----------|-----|
+| Expression policy (invitation-email-sender) | `770188de-2b60-4e87-820a-fa64d419ed89` |
+| Notification rule (invitation-email-trigger) | `a145a8c9-b3eb-440d-8b77-27139c346a17` |
+
+When an invitation is created with an `email` field in custom attributes, the expression policy triggers `ak_send_email()` to send the enrollment link to the invitee automatically.
+
+### How to Use
+
+**Invite via email** (Admin UI → Directory → Invitations → Create):
+1. Name it, select **Invitation Enrollment** flow, toggle **Single use** on, set expiry
+2. Custom attributes:
+   ```yaml
+   name: Jane Smith
+   email: jane@example.com
+   ```
+3. Click Create — email is sent automatically
+
+**Invite via link** (no email):
+1. Same as above but leave custom attributes empty (or omit `email` field)
+2. Click Create → expand the row → copy the invitation link
+3. Send the link manually
+
+---
+
+## Branding & Theming
+
+Echo6 cyberpunk branding applied to Authentik 2025.12.4 via System → Brands.
+
+### Brand Settings
+
+| Setting | Value |
+|---------|-------|
+| Brand title | `echo6` |
+| Theme | `dark` (forced, not automatic) |
+| Logo | `/media/custom/echo6-logo.png` (uploaded via Customization → Files) |
+| Favicon | `/media/custom/echo6-favicon.png` (uploaded via Customization → Files) |
+| Custom CSS | Echo6 Authentik CSS (~200 rules, applied via Brand → Custom CSS field) |
+
+### Flow Titles
+
+| Flow | Title |
+|------|-------|
+| Authentication | `echo6 // login` |
+| Invalidation | `echo6 // logout` |
+| Recovery | `echo6 // recovery` |
+| User Settings | `echo6 // settings` |
+
+### Custom CSS Highlights
+
+- **Font:** JetBrains Mono globally (with `:not()` exclusions for FontAwesome/PatternFly icon fonts)
+- **Colors:** Cyan `#28C0E8` primary accent, Yellow `#F0D848` secondary, dark backgrounds `#0a0e17`/`#111827`/`#1a2332`
+- **Login card:** Dark background, cyan-glow focus on inputs, branded submit button
+- **Admin sidebar:** Dark with cyan hover/active states
+- **User dashboard:** 3-column grid layout, dark cards with cyan border on hover
+- **Application icons:** Custom SVG icons uploaded for all 15 services
+
+### CSS Storage
+
+The custom CSS is stored in the Brand model's `branding_custom_css` field. To update:
+
+```bash
+# Copy CSS to Contabo
+scp /path/to/echo6-authentik.css root@100.64.0.1:/opt/authentik/branding/custom.css
+
+# Load into Brand model via ak shell
+ssh root@100.64.0.1 'docker exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.brands.models import Brand
+b = Brand.objects.get(domain="auth.echo6.co")
+b.branding_custom_css = open("/media/custom/custom.css").read()
+b.save()
+PYEOF
+
+# Restart to apply
+ssh root@100.64.0.1 'cd /opt/authentik && docker compose restart server worker'
+```
+
+### SSO Launch URL Pattern
+
+All authenticated service links use Authentik's application launch URL:
+
+```
+https://auth.echo6.co/application/launch/<app-slug>/
+```
+
+This provides seamless SSO: authenticated users pass through to the app, unauthenticated users get the login page then redirect to the app. Used by the SearXNG waffle menu and nav bar.
+
+| App Slug | Service | Launch URL |
+|----------|---------|-----------|
+| open-webui | Aurora (AI) | `https://auth.echo6.co/application/launch/open-webui/` |
+| peertube | Stream | `https://auth.echo6.co/application/launch/peertube/` |
+| files | Files | `https://auth.echo6.co/application/launch/files/` |
+| watchtower | Watchtower | `https://auth.echo6.co/application/launch/watchtower/` |
+| immich | Photos | `https://auth.echo6.co/application/launch/immich/` |
+| mailcow | Mail | `https://auth.echo6.co/application/launch/mailcow/` |
+| nextcloud | Cloud | `https://auth.echo6.co/application/launch/nextcloud/` |
+| jellyfin | Jellyfin | `https://auth.echo6.co/application/launch/jellyfin/` |
+| jellyseer | Requests | `https://auth.echo6.co/application/launch/jellyseer/` |
+
+### Brand Color Reference
+
+| Color | Hex | Usage |
+|-------|-----|-------|
+| Cyan | `#28C0E8` | Primary accent, links, focus states |
+| Cyan Light | `#5DD4F5` | Hover states |
+| Yellow | `#F0D848` | Secondary accent, badges |
+| BG Primary | `#0a0e17` | Page backgrounds |
+| BG Secondary | `#111827` | Cards, inputs |
+| BG Tertiary | `#1a2332` | Elevated surfaces |
+| Border | `#1e3a5f` | All borders |
+| Text Primary | `#e0e6ed` | Main text |
+| Text Muted | `#7a8ca0` | Secondary text |
+
+---
+
+*Last updated: 2026-02-18 — Added LiveSync proxy provider (PK 17) + livesync-users group, Files Forward Auth (PK 16). Authentik 2025.12.4*
--- a/docs/software/caddy.md
+++ b/docs/software/caddy.md
@ -4,19 +4,26 @@

 **Config:** `/etc/caddy/Caddyfile` on Contabo (ssh root@100.64.0.1)

+**Global options:** `email admin@echo6.co`, `admin off` (no live reload — must `systemctl restart caddy`)
+
 ### Current Site Blocks

 | Domain | Backend | Service |
 |--------|---------|---------|
 | auth.echo6.co | 127.0.0.1:9000 | Authentik SSO |
 | forge.echo6.co | 127.0.0.1:3001 | Forgejo Git |
-| mail.echo6.co | https://127.0.0.1:8443 | Mailcow (tls_insecure_skip_verify) |
+| mail.echo6.co | https://127.0.0.1:8453 | Mailcow (tls_insecure_skip_verify, r/w timeout 3600s) |
 | vpn.echo6.co | 127.0.0.1:8084 | Headscale |
 | vpn.echo6.co/admin* | 127.0.0.1:3100 | Headplane |
 | autodiscover.echo6.co | https://127.0.0.1:8443 | Mailcow autodiscover |
 | autoconfig.echo6.co | https://127.0.0.1:8443 | Mailcow autoconfig |
 | vault.echo6.co | 127.0.0.1:8086 | Vaultwarden |
 | proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) |
+| wt.echo6.co | 127.0.0.1:8099 (Authentik forward auth) | WATCHTOWER ops dashboard |
+| matrix.echo6.co | 127.0.0.1:8008 + 127.0.0.1:8085 | Matrix Synapse + MAS (login/logout/refresh/auth_metadata → MAS:8085, _matrix/* → Synapse:8008, default → MAS:8085) |
+| element.echo6.co | 127.0.0.1:8088 | Element Web client |
+| notes.echo6.co | 127.0.0.1:5984 + 127.0.0.1:5985 | LiveSync (CouchDB + provisioner, forward auth on /_provision*, CORS for Obsidian) |
+| tak.echo6.co | https://100.64.0.1:8446 + 100.64.0.1:8990 | TAK Server admin (8446, Authentik forward auth) + SIGIL console (/sigil, 8990) |

 ### Commands

@ -41,8 +48,21 @@ journalctl -u caddy -f

 | Domain | Backend | Pattern | Service |
 |--------|---------|---------|---------|
-| mesh.echo6.co | 100.64.0.7:8080 | Tailscale | MeshMonitor |
-| search.echo6.co | 100.64.0.15:8080 | Tailscale | SearXNG |
+| mesh.echo6.co | 192.168.1.100:8080 | Local IP | MeshMonitor (Authentik forward auth) |
+| echo6.co | 100.64.0.15:8080 | Tailscale | Echo6 Search (SearXNG) + Matrix well-known |
+| search.echo6.co | — | — | 301 redirect to echo6.co |
+| nas.echo6.co | 100.64.0.21:80 | Tailscale | OpenMediaVault (pi-nas) |
+| immich.echo6.co | 192.168.1.182:2283 | Local IP | Immich (has 2FA) |
+| nextcloud.echo6.co | 192.168.1.183:11000 | Local IP | Nextcloud AIO (SSO via Authentik) |
+| jellyfin.echo6.co | 100.64.0.18:8096 | Tailscale | Jellyfin media server (SSO via Authentik) |
+| requests.echo6.co | 100.64.0.18:5055 | Tailscale | Jellyseer request management (SSO via Authentik) |
+| stream.echo6.co | 192.168.1.170:80 | Local IP | PeerTube video streaming (SSO via Authentik) |
+| ai.echo6.co | 100.64.0.14:8080 | Tailscale | Open WebUI (SSO via Authentik) |
+| files.echo6.co | 100.64.0.24:8888 | Tailscale | RECON PDF library (Authentik forward auth) |
+| recon.echo6.co | 100.64.0.24:8420 | Tailscale | RECON dashboard + API |
+| lidarr.echo6.co | 100.64.0.18:8686 | Tailscale | Lidarr music automation (Authentik forward auth) |
+| navidrome.echo6.co | 100.64.0.18:4533 | Tailscale | Navidrome music server (Authentik forward auth, /rest/* exempt for Subsonic API) |
+| vpn.idahomesh.com | 192.168.1.106:8080 | Local IP | IdahoMesh Headscale VPN coordination |

 ### Commands

@ -68,10 +88,21 @@ ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
 | mail.echo6.co | 100.64.0.1 | Mailcow |
 | vpn.echo6.co | 100.64.0.1 | Headscale |
 | vault.echo6.co | 100.64.0.1 | Vaultwarden |
-| docs.echo6.co | 100.64.0.1 | Wiki.js |
 | proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) |
-| stream.echo6.co | *TBD* | PeerTube - needs host verification |
-| notes.echo6.co | *TBD* | Obsidian LiveSync - needs host verification |
+| stream.echo6.co | 100.64.0.8 | PeerTube (via utility Caddy) |
+| notes.echo6.co | 100.64.0.1 | LiveSync CouchDB + provisioner (via Contabo Caddy) |
+| tak.echo6.co | 100.64.0.1 | TAK Server + SIGIL (via Contabo Caddy) |
+| jellyfin.echo6.co | 100.64.0.8 | Jellyfin (via utility Caddy) |
+| requests.echo6.co | 100.64.0.8 | Jellyseer (via utility Caddy) |
+| wt.echo6.co | 100.64.0.1 | WATCHTOWER ops dashboard |
+| ai.echo6.co | 100.64.0.8 | Open WebUI (via utility Caddy) |
+| matrix.echo6.co | 100.64.0.1 | Matrix Synapse (via Contabo Caddy) |
+| element.echo6.co | 100.64.0.1 | Element Web (via Contabo Caddy) |
+| echo6.co | 100.64.0.8 | Echo6 Search homepage (via utility Caddy) |
+| files.echo6.co | 100.64.0.8 | RECON PDF library (via utility Caddy) |
+| recon.echo6.co | 100.64.0.8 | RECON dashboard (via utility Caddy) |
+| lidarr.echo6.co | 100.64.0.8 | Lidarr music automation (via utility Caddy) |
+| navidrome.echo6.co | 100.64.0.8 | Navidrome music server (via utility Caddy) |

 ### Commands

@ -95,19 +126,31 @@ dig +short forge.echo6.co @100.64.0.1   # Test
 | mail | Mailcow Email |
 | vpn | Headscale VPN |
 | vault | Vaultwarden |
+| wt | WATCHTOWER ops dashboard |
+| matrix | Matrix Synapse |
+| element | Element Web |
+| notes | LiveSync (CouchDB + provisioner) |
+| proxmox | Proxmox VE (via Tailscale to data node) |
+| tak | TAK Server + SIGIL |

 ### Home Services → 199.6.36.163

 | Subdomain | Service |
 |-----------|---------|
-| @ | Main site |
+| @ | Echo6 Search homepage (SearXNG) |
 | ai | Open WebUI |
-| docs | Wiki.js |
 | stream | PeerTube |
-| notes | Obsidian LiveSync |
 | jellyfin | Jellyfin |
 | mesh | MeshMonitor |
-| search | SearXNG |
+| nas | OpenMediaVault (pi-nas) |
+| search | SearXNG (redirects to echo6.co) |
+| immich | Immich |
+| nextcloud | Nextcloud |
+| requests | Jellyseer |
+| files | RECON PDF library |
+| recon | RECON dashboard |
+| lidarr | Lidarr music automation |
+| navidrome | Navidrome music server |

 ### Email Records

@ -151,12 +194,21 @@ oidc:
 |---------|---------------|--------------|---------------|
 | Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
 | Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
+| Forgejo SSH | 22 | 0.0.0.0:2222 | Direct (not proxied) |
 | Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
 | Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
 | Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
 | Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
 | Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
+| WATCHTOWER | 8084 | host network :8099 | wt.echo6.co |
+| Matrix Synapse | 8008 | 127.0.0.1:8008 | matrix.echo6.co (/_matrix/*, /_synapse/*) |
+| Matrix MAS | 8080 | 127.0.0.1:8085 | matrix.echo6.co (login/logout/refresh/auth_metadata, default) |
+| Element Web | 80 | 127.0.0.1:8088 | element.echo6.co |
+| LiveSync CouchDB | 5984 | 127.0.0.1:5984 | notes.echo6.co |
+| LiveSync Provisioner | 8080 | 127.0.0.1:5985 | notes.echo6.co/_provision/* |
+| TAK Server Admin | 8446 | https://100.64.0.1:8446 (Tailscale) | tak.echo6.co |
+| SIGIL Console | 8990 | 100.64.0.1:8990 | tak.echo6.co/sigil |

 ---

-*Last updated: 2026-02-06 — Added SearXNG (search.echo6.co) on utility CT 102*
+*Last updated: 2026-04-13 — Audit sync: added MAS routing on matrix.echo6.co, lidarr/navidrome/vpn.idahomesh.com to utility Caddy, proxmox/tak to GoDaddy, removed ghost docs.echo6.co entries, added dnsmasq lidarr/navidrome*
--- a/docs/software/recon.md
+++ b/docs/software/recon.md
@ -0,0 +1,131 @@
+# RECON — Knowledge Extraction Pipeline
+
+## Overview
+
+RECON extracts knowledge from PDFs and web content into a searchable vector database. PDFs are scanned from an NFS-mounted library, text is extracted (with Gemini Vision fallback for scanned docs), concepts are enriched via Gemini, and embeddings are stored in Qdrant. Aurora (Open WebUI) queries the knowledge base via RAG filter.
+
+## Location
+
+- **Host:** recon LXC (CT 130 on data node, 192.168.1.240)
+- **IP:** 192.168.1.130 / 100.64.0.24 (Tailscale)
+- **Install:** `/opt/recon/`
+- **User:** zvx
+- **Service:** `recon.service` (systemd, Type=simple, Restart=on-failure)
+- **Dashboard:** https://recon.echo6.co (internal: http://100.64.0.24:8420)
+- **Health:** https://recon.echo6.co/api/health
+
+## Stack
+
+| Component | Technology | Location |
+|-----------|-----------|----------|
+| Pipeline + CLI | Python 3.12, argparse | /opt/recon/recon.py |
+| Dashboard + API | Flask | /opt/recon/lib/api.py (port 8420) |
+| Status DB | SQLite (WAL mode) | /opt/recon/data/recon.db |
+| Vector DB | Qdrant | cortex:6333 (Docker) |
+| Embeddings | TEI (bge-m3, 1024-dim) | cortex:8090 (Docker) |
+| Enrichment | Gemini 2.0 Flash | Google API (4 keys) |
+| Vision OCR | Gemini 2.0 Flash | Google API (shared keys) |
+| Text extraction | PyPDF2, poppler-utils, Tesseract | Local |
+| PDF source | NFS | pi-nas:/export/library → /mnt/library |
+| File server | nginx | localhost:8888 → files.echo6.co |
+
+## Pipeline Stages
+
+All stages run concurrently as daemon threads in the service:
+
+1. **Scanner** (hourly) — walks /mnt/library, catalogues new PDFs, queues them
+2. **Extract** (4 workers) — PyPDF2 → pdftotext → Tesseract → Gemini Vision per page
+3. **Enrich** (16 workers, 4 API keys) — Gemini extracts structured concepts from text windows
+4. **Embed** (4 workers) — TEI generates vectors, upserted to Qdrant
+
+## Extraction Chain
+
+Per page, in order. Each method only runs if the previous returned <50 chars:
+
+1. **PyPDF2** — fast, free, works on text-based PDFs
+2. **pdftotext** (poppler) — handles some PDFs PyPDF2 misses
+3. **Tesseract OCR** — renders page to image, runs local OCR
+4. **Gemini Vision** — renders page to PNG, sends to Gemini 2.0 Flash vision API
+
+Method tracking saved in `data/text/{hash}/meta.json` as `ocr_methods` dict.
+
+## Scale (as of Feb 2026)
+
+- ~10,162 documents in pipeline
+- ~95,000+ vectors in Qdrant (HNSW index, <10ms search latency)
+- Collection: `recon_knowledge`
+- ~13,239 PDFs catalogued from NFS library
+
+## Resilience
+
+- **Enricher**: Exponential backoff (5s→80s) for transient errors (429, 500, 503). Window-level failure isolation — partial enrichment beats zero.
+- **Extractor**: Per-page timeout (30s), per-document timeout (1800s). Partial extractions saved.
+- **Embedder**: Skip-on-failure per concept, batch processing.
+- **Service**: Restart=on-failure, RestartSec=30, MemoryMax=3G.
+
+## Configuration
+
+**Config file:** `/opt/recon/config.yaml`
+
+Key sections:
+- `processing.extract_workers` (4), `enrich_workers` (16), `embed_workers` (4)
+- `processing.extract_timeout` (1800s), `page_timeout` (30s)
+- `processing.enrich_max_retries` (5), `enrich_base_delay` (5.0)
+- `gemini.model` (gemini-2.0-flash), `gemini.response_mime_type` (application/json)
+- `service.scan_interval` (3600), `stage_poll_interval` (30)
+
+**API keys:** `/opt/recon/.env` — GEMINI_KEY_1 through GEMINI_KEY_4
+
+## API Endpoints
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/` | GET | Dashboard HTML |
+| `/api/knowledge-stats` | GET | Full pipeline stats, per-source breakdown |
+| `/api/health` | GET | Health check (Qdrant, TEI, NFS, Gemini, pipeline) |
+| `/api/search` | GET | Vector search (`?q=query&limit=5`) |
+| `/api/upload` | POST | Upload PDF (multipart: file + category) |
+| `/api/upload/<hash>/status` | GET | Upload status tracking |
+| `/api/upload/categories` | GET | Available upload categories |
+| `/api/ingest` | POST | Ingest intel JSON data |
+| `/api/peertube/channels` | GET | List all channels from channel-map.json with video counts from PeerTube DB |
+| `/api/peertube/channels/stats` | GET | Channel count, total videos, downloader status |
+| `/api/peertube/channels/add` | POST | Add channel: resolve YT URL, create PeerTube channel, update JSON |
+| `/api/peertube/channels/<name>` | DELETE | Remove channel from JSON and optionally from PeerTube |
+
+## Backups
+
+- **Destination:** `root@100.64.0.1:/opt/backups/recon/`
+- **Full sync:** every 6 hours (concepts, text, DB, config)
+- **DB snapshot:** every 2 hours
+- **Recovery:** restore from Contabo → `recon rebuild` (reconstructs Qdrant from concept JSONs)
+- **Critical data:** `data/concepts/` — Gemini extraction work, costs money to regenerate
+
+## Key Files
+
+```
+/opt/recon/
+├── recon.py              # CLI entry point + service command
+├── config.yaml           # Configuration
+├── .env                  # Gemini API keys
+├── PROJECT-BIBLE.md      # Full documentation
+├── lib/
+│   ├── api.py            # Flask dashboard + API
+│   ├── extractor.py      # PDF → text (4-method chain)
+│   ├── enricher.py       # Text → concepts (Gemini)
+│   ├── embedder.py       # Concepts → vectors (TEI/Qdrant)
+│   ├── status.py         # SQLite DB (WAL, thread-safe)
+│   └── utils.py          # Config, hashing, logging
+├── scripts/
+│   ├── backup.sh         # Backup to Contabo
+│   ├── validate.py       # Pipeline consistency checker
+│   └── rebuild_qdrant.py # Nuclear Qdrant rebuild
+└── data/
+    ├── recon.db           # SQLite status DB
+    ├── concepts/{hash}/   # Enriched concept JSONs
+    └── text/{hash}/       # Extracted page text
+```
+
+---
+
+*Last updated: 2026-02-16 — Initial creation*
--- a/docs/software/searxng.md
+++ b/docs/software/searxng.md
@ -0,0 +1,130 @@
+# SearXNG — Echo6 Search Homepage
+
+## Overview
+
+SearXNG is deployed as the branded Echo6 search homepage at `echo6.co`. The default simple theme is overridden with custom templates (base.html, index.html) and static assets via Docker bind mounts. Features a cyberpunk aesthetic with JetBrains Mono font, cyan/yellow palette, top navigation bar with service links, and a waffle app launcher menu.
+
+## Location
+
+- **Host:** searxng LXC (CT 102 on utility node, 192.168.1.241)
+- **IP:** 192.168.1.102 / 100.64.0.15 (Tailscale)
+- **Install:** `/opt/searxng/` (Docker Compose)
+- **User:** zvx
+- **URL:** https://echo6.co
+- **Redirect:** https://search.echo6.co → https://echo6.co (301 permanent)
+
+## Stack
+
+| Component | Technology | Location |
+|-----------|-----------|----------|
+| Search engine | SearXNG (Docker, v2026.2.6) | searxng container |
+| Cache | Valkey (Redis-compatible) | valkey container |
+| Reverse proxy | Utility Caddy (CT 101) | 192.168.1.101 |
+| SSL certs | acme.sh (Let's Encrypt) | /etc/caddy/certs/ on CT 101 |
+
+## Theme Customization
+
+Custom Echo6 theme applied via Docker bind mounts that override SearXNG's simple theme:
+
+### Overridden Files
+
+| Override | Container Path | Purpose |
+|----------|---------------|---------|
+| `custom/templates/simple/base.html` | `/usr/local/searxng/searx/templates/simple/base.html` | Nav bar, CSS, JS, footer, waffle menu |
+| `custom/templates/simple/index.html` | `/usr/local/searxng/searx/templates/simple/index.html` | Homepage (Echo6 logo + search bar) |
+| `custom/img/echo6-logo.png` | `/usr/local/searxng/searx/static/themes/simple/img/echo6-logo.png` | Echo6 logo |
+| `custom/img/favicon.png` | `/usr/local/searxng/searx/static/themes/simple/img/favicon.png` | Echo6 favicon |
+
+### Brand Palette
+
+| Variable | Value | Usage |
+|----------|-------|-------|
+| `--bg-primary` | `#0a0e17` | Page background |
+| `--bg-secondary` | `#111827` | Cards, nav, waffle |
+| `--bg-tertiary` | `#1a2332` | Hover states |
+| `--accent-cyan` | `#28C0E8` | Links, focus glow, active states |
+| `--accent-yellow` | `#F0D848` | Logo accents, highlights |
+| `--text-primary` | `#e0e6ed` | Main text |
+| `--text-secondary` | `#7a8ca0` | Muted text |
+| `--border` | `#1e3a5f` | Borders |
+| `--border-focus` | `#28C0E8` | Focus ring |
+| Font | JetBrains Mono | All text |
+
+### Layout
+
+- **Homepage:** Centered Echo6 logo + pill-shaped search bar (viewport-locked, no scroll)
+- **Results page:** Two-column grid (`1fr 25rem`) — results left, sidebar right
+- **Nav bar:** Left: `.//files`, `.//stream` — Right: `.//photos`, `.//mail`, waffle menu, login avatar
+- **Waffle menu:** 3x3 grid of service tiles (Aurora, Stream, Files, Watchtower, Photos, Mail, Cloud, Admin, Search) with inline SVG icons
+- **All nav links:** Use Authentik SSO launch URLs (`https://auth.echo6.co/application/launch/<slug>/`)
+
+## Configuration
+
+**Config file:** `/opt/searxng/searxng-config/settings.yml` (mounted to `/etc/searxng/settings.yml`)
+
+Key settings:
+- `general.instance_name`: "Echo6"
+- `ui.default_theme`: simple
+- `ui.theme_args.simple_style`: dark
+- `ui.center_alignment`: false (enables full-width results layout)
+- `server.base_url`: "https://echo6.co/"
+
+**Environment:** Set in docker-compose.yml:
+- `SEARXNG_BASE_URL=https://echo6.co/`
+- `INSTANCE_NAME=Echo6`
+
+## Key Files
+
+```
+/opt/searxng/
+├── docker-compose.yml        # Compose config (searxng + valkey)
+├── searxng-config/
+│   └── settings.yml          # SearXNG configuration
+└── custom/
+    ├── templates/simple/
+    │   ├── base.html          # Custom base template (nav, CSS, JS, footer, waffle)
+    │   └── index.html         # Custom homepage (Echo6 logo + search bar)
+    └── img/
+        ├── echo6-logo.png     # Echo6 logo
+        └── favicon.png        # Echo6 favicon
+```
+
+## Commands
+
+```bash
+# Access CT 102 via utility Proxmox host
+ssh root@192.168.1.241 'pct exec 102 -- <command>'
+
+# Restart SearXNG
+ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose restart searxng'
+
+# View logs
+ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose logs -f searxng'
+
+# Full recreate (after compose changes)
+ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose down && docker compose up -d'
+
+# Push template file via pct
+scp /tmp/file.html root@192.168.1.241:/tmp/
+ssh root@192.168.1.241 'pct push 102 /tmp/file.html /opt/searxng/custom/templates/simple/file.html'
+
+# Test locally
+curl -s http://192.168.1.102:8080 | head -30
+```
+
+## Caddy Configuration
+
+**Utility Caddy (CT 101):**
+- `echo6.co` → `100.64.0.15:8080` + Matrix `.well-known` handlers
+- `search.echo6.co` → 301 redirect to `https://echo6.co`
+
+**dnsmasq (Contabo):**
+- `echo6.co` → `100.64.0.8` (utility Caddy)
+
+**GoDaddy DNS:**
+- `@` (echo6.co) → `199.6.36.163` (home)
+- `search` → `199.6.36.163` (home)
+
+---
+
+*Last updated: 2026-02-17 — Initial creation after Echo6 homepage deployment*