docs: migrate Authentik (SSO keystone) to edge2 CT 105

- Authentik -> edge2 CT 105 (Postgres pg_dump/restore; SECRET_KEY carried verbatim; zero-downtime until ~2s cutover) - Multi-block Caddy cutover: auth.echo6.co + notes.echo6.co outpost/forward_auth -> 100.64.0.36:9000 - runbook: add reboot tailscale-before-docker gotcha; clarify dnsmasq must NOT be repointed (points at Caddy host) - source left stopped + intact on Contabo as cold rollback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 05:49:07 +00:00 · 2026-06-18 05:49:07 +00:00 · 44f0257376
commit 44f0257376
parent 30f70793f8
140 changed files with 4013 additions and 24 deletions
--- a/vault/docs/services/ots-setup.md
+++ b/vault/docs/services/ots-setup.md
@ -0,0 +1,571 @@
+# OpenTAKServer Setup Documentation
+
+## Installation Date
+April 17, 2026
+
+## Overview
+OpenTAKServer installation on Echo6 infrastructure for TAK (Team Awareness Kit) server capabilities.
+
+---
+
+## Container Specifications
+
+**VMID:** 109
+**Hostname:** opentakserver
+**Host:** utility Proxmox (192.168.1.241)
+**IP Address:** 192.168.1.109/24
+**Gateway:** 192.168.1.1
+**OS:** Ubuntu 24.04 LTS
+**Resources:** 2 cores, 4GB RAM, 8GB disk
+**Storage:** local-lvm
+**Features:** nesting=1
+**Auto-start:** enabled
+
+---
+
+## Access Information
+
+### External Access
+- **URL:** https://ots.k7zvx.com
+- **Admin Username:** administrator
+- **Admin Password:** password (**CHANGE THIS IMMEDIATELY**)
+
+### SSH Access
+```bash
+# Via Proxmox host
+ssh root@192.168.1.241 'pct exec 109 -- bash'
+
+# As zvx user
+ssh root@192.168.1.241 'pct exec 109 -- su - zvx'
+
+# Direct container entry
+pct enter 109
+```
+
+**Container User:**
+- Username: zvx
+- Password: 7redditGold
+- Sudo: NOPASSWD enabled
+
+---
+
+## Network Configuration
+
+### DNS
+- **Domain:** ots.k7zvx.com
+- **DNS Provider:** GoDaddy
+- **Record Type:** A
+- **Points to:** 199.6.36.163 (home external IP)
+- **TTL:** 3600 seconds
+
+### SSL Certificate
+- **Provider:** Let's Encrypt
+- **Method:** acme.sh with GoDaddy DNS validation
+- **Location:** /etc/caddy/certs/ots.k7zvx.com.* (on CT 101)
+- **Auto-renewal:** Configured via acme.sh
+
+### Reverse Proxy (Caddy on CT 101)
+```
+ots.k7zvx.com {
+    tls /etc/caddy/certs/ots.k7zvx.com.fullchain.crt /etc/caddy/certs/ots.k7zvx.com.key
+    reverse_proxy https://192.168.1.109:443 {
+        transport http {
+            tls_insecure_skip_verify
+        }
+    }
+}
+```
+
+### Port Forwarding
+- **Router:** Ubiquiti firewall (192.168.1.28)
+- **External ports:** 80/443 → 192.168.1.101 (Caddy CT)
+- **Internal proxy:** Caddy → 192.168.1.109:443 (OpenTAKServer)
+
+---
+
+## OpenTAKServer Configuration
+
+### Installation Details
+- **Installation Method:** Official installer script
+- **Install User:** zvx
+- **Install Path:** /home/zvx/ots/
+- **Virtual Environment:** /home/zvx/.opentakserver_venv/
+- **Version:** 1.7.10
+
+### Services
+```bash
+# Main services
+systemctl status opentakserver
+systemctl status eud_handler
+systemctl status eud_handler_ssl
+systemctl status cot_parser
+systemctl status nginx
+
+# Service logs
+tail -f /home/zvx/ots/logs/opentakserver.log
+tail -f /home/zvx/ots/logs/eud_handler_tcp.log
+tail -f /home/zvx/ots/logs/eud_handler_ssl.log
+tail -f /home/zvx/ots/logs/cot_parser.log
+```
+
+### Listening Ports
+- **80:** nginx (HTTP, redirects to HTTPS)
+- **443:** nginx (HTTPS, web UI)
+- **8080:** nginx (HTTP Marti API)
+- **8088:** eud_handler (TCP CoT)
+- **8089:** eud_handler (TLS CoT)
+- **8443:** nginx (HTTPS Marti API)
+- **8446:** nginx (Admin API)
+- **1936:** nginx
+- **8322:** nginx
+- **8883:** nginx
+
+### Database Configuration
+- **Type:** PostgreSQL
+- **Database name:** ots
+- **Username:** ots
+- **Password:** eCt8HwnBFlZJ6K1arTn4
+- **Connection string:** postgresql+psycopg2://ots:eCt8HwnBFlZJ6K1arTn4@127.0.0.1/ots
+- **Service:** postgresql (systemd)
+
+### Configuration File
+- **Location:** /home/zvx/ots/config.yml
+- **Backup:** /home/zvx/ots/config.yml.bak (if needed)
+
+---
+
+## Meshtastic MQTT Gateway
+
+### Overview
+OpenTAKServer's native Meshtastic integration allows remote gateway radios to connect via MQTT over TLS. This enables:
+- Meshtastic mesh positions to appear as TAK contacts on the map
+- Text messages from mesh to flow into TAK chat
+- Remote gateway deployment over Starlink, cell hotspots, or any internet connection
+
+**Setup Date:** April 17, 2026
+**Status:** ✅ Fully Operational
+
+### Architecture
+```
+Meshtastic Radio (GPS)
+  └─→ MQTT TLS (port 8883) via ots.k7zvx.com
+      └─→ Nginx Stream Proxy (TLS termination)
+          └─→ RabbitMQ MQTT (localhost:1883)
+              └─→ OpenTAKServer (HTTP auth backend)
+                  └─→ PostgreSQL (user database)
+                      └─→ TAK Clients (ATAK/iTAK/WinTAK)
+```
+
+### Server Configuration
+
+#### RabbitMQ MQTT Settings
+**Location:** `/etc/rabbitmq/rabbitmq.conf`
+
+```conf
+# MQTT listener bound to loopback only (security)
+mqtt.listeners.tcp.default = 127.0.0.1:1883
+
+# Authentication via OTS HTTP backend
+auth_backends.1 = internal
+auth_backends.2 = http
+auth_http.http_method   = post
+auth_http.user_path     = http://127.0.0.1:8081/api/rabbitmq/auth
+auth_http.vhost_path    = http://127.0.0.1:8081/api/rabbitmq/vhost
+auth_http.resource_path = http://127.0.0.1:8081/api/rabbitmq/resource
+auth_http.topic_path    = http://127.0.0.1:8081/api/rabbitmq/topic
+
+# AMQP also restricted to loopback
+listeners.tcp.default = 127.0.0.1:5672
+```
+
+**Security:** Ports 1883 (MQTT) and 5672 (AMQP) are bound to localhost only. External access is via nginx TLS proxy on port 8883.
+
+#### Nginx Stream Configuration
+**Location:** `/etc/nginx/streams-available/rabbitmq`
+
+```nginx
+server {
+    listen     8883 ssl;
+    proxy_pass 127.0.0.1:1883;
+
+    ssl_certificate /home/zvx/ots/ca/certs/opentakserver/opentakserver.pem;
+    ssl_certificate_key /home/zvx/ots/ca/certs/opentakserver/opentakserver.nopass.key;
+}
+```
+
+**Note:** Currently uses self-signed OTS CA certificate. Meshtastic radios must disable TLS certificate verification ("allow insecure").
+
+#### OTS Meshtastic Configuration
+**Location:** `/home/zvx/ots/config.yml`
+
+```yaml
+OTS_ENABLE_MESHTASTIC: true
+OTS_MESHTASTIC_GROUP: __ANON__
+OTS_MESHTASTIC_TOPIC: opentakserver
+OTS_MESHTASTIC_PUBLISH_INTERVAL: 30
+OTS_MESHTASTIC_NODEINFO_INTERVAL: 3
+OTS_MESHTASTIC_DOWNLINK_CHANNELS: []
+```
+
+- **Group `__ANON__`:** Broadcasts Meshtastic positions to all connected TAK clients
+- **Topic `opentakserver`:** MQTT topic root for all Meshtastic messages
+- **Publish interval:** OTS publishes updates to TAK clients every 30 seconds
+
+### Client Configuration (Meshtastic Radio)
+
+#### Connection Settings
+- **Server/Address:** `ots.k7zvx.com`
+- **Port:** Auto (uses 8883 when TLS enabled)
+- **Enable TLS/SSL:** ✅ **REQUIRED**
+- **TLS Certificate Verification:** ❌ **DISABLED** ("Allow Insecure") *— server uses self-signed cert*
+
+#### Authentication
+- **Username:** `[unique_username]` — created in OTS web UI (Settings → Users)
+- **Password:** `[user_password]` — set in OTS web UI
+
+#### MQTT Settings
+- **Topic:** `opentakserver`
+- **Encryption Enabled:** ❌ **DISABLED** — **CRITICAL:** OTS cannot decode PSK-encrypted MQTT payloads. The LoRa RF hop between radios is still PSK-encrypted regardless of this setting. This toggle only controls whether the radio re-encrypts before publishing to MQTT. TLS on port 8883 provides wire security instead.
+- **JSON Output:** ❌ **DISABLED** — OTS expects protobuf format, not JSON. Leave this off.
+- **Proxy to Client:** ❌ **DISABLED** — required per OTS docs; enabling this causes presence/LWT issues with gateway operation
+- **Uplink Enabled:** ✅ **REQUIRED** — configured per-channel when scanning QR code from OTS web UI
+
+#### Position Settings
+- **Map Publish:** ❌ **DISABLED** — prevents public map sharing (map.meshtastic.org)
+- **Position Enabled:** ✅ **REQUIRED**
+- **Position Precision:** Any — full precision is sent to private server regardless of this setting
+
+**Important:** "Map Publish" only affects the public Meshtastic map. Position data still flows to custom MQTT servers (ots.k7zvx.com) with full precision when disabled.
+
+### User Setup Process
+
+1. **Create OTS user account:**
+   ```bash
+   # Via OTS Web UI
+   https://ots.k7zvx.com → Settings → Users → Create User
+
+   # Set username (e.g., "matt", "jeepnjonny")
+   # Set password
+   ```
+
+2. **Configure Meshtastic radio** with connection settings above
+
+3. **Test connection:**
+   - Send a text message from the radio
+   - Message should appear in iTAK/ATAK within seconds (will appear in triplicate due to mesh retry logic)
+
+4. **Verify position data:**
+   - Position updates send based on mesh broadcast interval (typically 15-30 minutes)
+   - Node should appear as contact on TAK map with accurate location
+   - Updates continue at configured interval
+
+### Monitoring & Verification
+
+#### Check MQTT Connection
+```bash
+# View RabbitMQ logs for connection activity
+sudo tail -f /var/log/rabbitmq/rabbit@opentakserver.log | grep MQTT
+
+# List active MQTT connections
+sudo rabbitmqctl list_connections protocol user peer_host peer_port
+
+# Monitor MQTT traffic (subscribe to all topics)
+mosquitto_sub -h 127.0.0.1 -p 1883 -u [username] -P [password] -t "#" -v
+```
+
+#### Check OTS Processing
+```bash
+# Monitor Meshtastic controller logs
+sudo tail -f /home/zvx/ots/logs/opentakserver.log | grep -i mesh
+
+# Check for position data
+sudo grep "latitude_i" /home/zvx/ots/logs/opentakserver.log | tail -20
+
+# View Meshtastic EUDs in database
+sudo -u postgres psql -d ots -c "SELECT callsign, uid, meshtastic_id, last_event_time FROM euds WHERE meshtastic_id IS NOT NULL;"
+```
+
+### Troubleshooting
+
+#### Connection Refused
+**Symptom:** Radio shows "connection refused" error
+
+**Causes & Fixes:**
+1. **TLS not enabled** — Radio connecting to port 1883 (blocked externally). Enable TLS on radio to use port 8883.
+2. **Wrong server** — Verify `ots.k7zvx.com` (not `mqtt.k7zvx.com`)
+3. **Firewall** — Verify port 8883 is accessible: `telnet ots.k7zvx.com 8883`
+
+#### Authentication Failed (CONNACK code 4)
+**Symptom:** RabbitMQ logs show "access refused for user"
+
+**Causes & Fixes:**
+1. **User doesn't exist** — Create user in OTS web UI first
+2. **Wrong password** — Verify credentials match OTS user settings
+3. **Check logs:**
+   ```bash
+   sudo grep "MQTT connection failed" /var/log/rabbitmq/rabbit@opentakserver.log
+   ```
+
+#### TLS Certificate Verification Failed
+**Symptom:** Connection fails with SSL/certificate errors
+
+**Fix:** Disable TLS certificate verification on radio ("Allow Insecure" or "Skip Verify")
+
+**Reason:** OTS uses self-signed certificate. Option to install Let's Encrypt cert exists but not currently implemented.
+
+#### Radio Connected But No Data
+**Symptom:** MQTT connection shows as active but no messages/position in OTS
+
+**Causes & Fixes:**
+1. **Uplink disabled** — Ensure uplink is enabled on the channel (set via QR code from OTS Meshtastic web UI page)
+2. **No GPS lock** — Verify radio has satellite fix and coordinates displayed
+3. **Wrong topic** — Ensure radio is publishing to `opentakserver` topic
+4. **Test with text message** — Send message from radio to verify MQTT publishing works
+
+#### Node Appears But No Position on Map
+**Symptom:** Contact exists in iTAK but no location shown
+
+**Causes & Fixes:**
+1. **No GPS lock** — Radio must have satellite fix to send position
+2. **Position broadcasts disabled** — Enable "Position Enabled" on radio
+3. **Waiting for interval** — Position updates send less frequently than text messages (typically 15-30 min)
+4. **Check logs:**
+   ```bash
+   sudo grep "latitude_i\|position" /home/zvx/ots/logs/opentakserver.log | tail -20
+   ```
+
+#### Unicode Encoding Errors
+**Symptom:** OTS logs show `UnicodeEncodeError` related to emojis
+
+**Fix:** Remove emojis from Meshtastic node name (long name and short name). PostgreSQL database cannot handle emojis with current encoding.
+
+**Example error:**
+```
+UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3f4' in position 0
+```
+
+### Known Behaviors
+
+#### Triplicate Messages
+**Behavior:** Each text message appears 3 times in iTAK
+
+**Reason:** Meshtastic retry logic sends messages 3 times for reliability over mesh network. OTS forwards all copies.
+
+**Impact:** Normal behavior, ensures messages get through despite poor mesh coverage. Not configurable without modifying OTS source.
+
+#### Position Update Frequency
+**Mesh vs MQTT:** Meshtastic has separate intervals:
+- **Mesh broadcasts:** Position sent to local mesh (15-30 min typical)
+- **MQTT publishing:** Position sent to MQTT server (can be same or different)
+
+Radio may show position locally more often than it publishes to MQTT server. Configure "MQTT Position Interval" separately from mesh broadcast interval.
+
+#### Map Publish vs Custom MQTT
+**Map Publish:** Controls publishing to public Meshtastic map (map.meshtastic.org)
+- Has 1-hour minimum interval due to privacy concerns
+- Shows degraded precision on public map
+
+**Custom MQTT (OTS):** Unaffected by map publish settings
+- Receives full precision position data
+- Updates at configured mesh/MQTT interval
+- Completely private — only visible to your TAK clients
+
+Disabling "Map Publish" does NOT affect position data sent to custom MQTT servers like OTS.
+
+### Active Gateways
+
+| Node ID | Callsign | Owner | Channel | Status |
+|---------|----------|-------|---------|--------|
+| !f861851e | Meshtastic 851e | matt | LFtrackers | Active |
+
+### Port Summary
+
+| Port | Service | Bind Address | External Access | Purpose |
+|------|---------|--------------|-----------------|---------|
+| 1883 | RabbitMQ MQTT | 127.0.0.1 | ❌ No (loopback only) | MQTT plaintext (internal) |
+| 5672 | RabbitMQ AMQP | 127.0.0.1 | ❌ No (loopback only) | Internal OTS ↔ RabbitMQ |
+| 8883 | Nginx Stream | 0.0.0.0 | ✅ Yes (via ots.k7zvx.com) | MQTT TLS (Meshtastic radios) |
+
+---
+
+## Critical Fixes Applied
+
+### 1. Ubuntu Mirror Issues
+**Problem:** Canonical's GeoDNS is broken for Filer Telephone ISP (Idaho). All DNS servers (local, Google, Cloudflare) returned only UK mirror IPs (91.189.x.x range) which had severe packet loss and timeouts.
+
+**Root Cause:**
+- Your ISP has poor peering with Canonical's UK infrastructure
+- GeoDNS should return US mirrors but only returned UK for this region
+- 30-50% packet loss between ISP and Canonical's network
+
+**Fix Applied:**
+```bash
+# Changed /etc/apt/sources.list to use Oregon mirror
+sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
+
+# Results:
+# - Before: Timeouts (10+ seconds)
+# - After: 11 MB/s download speed
+```
+
+**Testing:**
+- archive.ubuntu.com: TIMEOUT
+- mirrors.kernel.org: 0.16s response time
+- mirror.arizona.edu: 0.31s response time
+
+### 2. PostgreSQL Driver Compatibility
+**Problem:** OpenTAKServer installer uses psycopg3 by default, which has a compatibility bug with SQLAlchemy's PostgreSQL version detection. This causes the service to crash-loop with:
+```
+TypeError: cannot use a string pattern on a bytes-like object
+```
+
+**Fix Applied:**
+```bash
+# Install psycopg2-binary (stable driver)
+pip install psycopg2-binary
+
+# Update connection string in /home/zvx/ots/config.yml
+# Changed: postgresql+psycopg://
+# To:      postgresql+psycopg2://
+```
+
+**Impact:** Without this fix, the backend API doesn't run and login fails silently.
+
+---
+
+## Troubleshooting
+
+### Service Won't Start
+```bash
+# Check service status
+systemctl status opentakserver
+
+# Check logs for errors
+tail -50 /home/zvx/ots/logs/opentakserver.log
+
+# Common issues:
+# 1. Database connection errors → check PostgreSQL is running
+# 2. Port conflicts → check nginx/other services
+# 3. psycopg errors → ensure psycopg2-binary is installed
+```
+
+### Login Fails
+```bash
+# Verify backend API is running
+systemctl status opentakserver
+
+# Check if service is crash-looping
+journalctl -u opentakserver -f
+
+# Test local API access
+curl -I http://localhost:8080
+curl -I https://localhost:443
+```
+
+### Slow Package Updates
+```bash
+# Verify using fast mirror
+grep "URIs:" /etc/apt/sources.list
+
+# Should show: http://mirrors.kernel.org/ubuntu
+# If not, run:
+sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
+apt update
+```
+
+### External Access Issues
+```bash
+# Test from outside network
+curl -I https://ots.k7zvx.com
+
+# Check DNS resolution
+dig +short ots.k7zvx.com @8.8.8.8
+# Should return: 199.6.36.163
+
+# Check Caddy is proxying correctly (on CT 101)
+systemctl status caddy
+journalctl -u caddy -f
+```
+
+---
+
+## Maintenance
+
+### Update OpenTAKServer
+```bash
+# Stop services
+systemctl stop opentakserver eud_handler eud_handler_ssl cot_parser
+
+# Activate virtualenv
+su - zvx
+source ~/.opentakserver_venv/bin/activate
+
+# Update via pip (if available)
+pip install --upgrade opentakserver
+
+# Or follow official upgrade docs:
+# https://docs.opentakserver.io/
+
+# Restart services
+sudo systemctl start opentakserver eud_handler eud_handler_ssl cot_parser
+```
+
+### Backup
+```bash
+# Database backup
+sudo -u postgres pg_dump ots > /tmp/ots_backup_$(date +%Y%m%d).sql
+
+# Configuration backup
+cp /home/zvx/ots/config.yml /home/zvx/ots/config.yml.backup
+
+# Certificate backup (from CT 101)
+tar -czf /tmp/ots_certs_$(date +%Y%m%d).tar.gz /etc/caddy/certs/ots.k7zvx.com.*
+```
+
+### Change Admin Password
+1. Log in to https://ots.k7zvx.com with administrator/password
+2. Navigate to Settings → Users
+3. Select administrator user
+4. Change password
+5. Save changes
+
+---
+
+## Next Steps
+
+1. ✅ Change default administrator password
+2. ⏳ Configure TAK client connections
+3. ⏳ Set up user certificates for client authentication
+4. ⏳ Configure data packages and plugins
+5. ⏳ Set up monitoring/alerting for the service
+6. ⏳ Document backup/restore procedures
+7. ⏳ Test failover scenarios
+
+---
+
+## Reference Links
+
+- **OpenTAKServer Documentation:** https://docs.opentakserver.io/
+- **OpenTAKServer GitHub:** https://github.com/brian7704/OpenTAKServer
+- **TAK.gov Resources:** https://tak.gov/
+- **Echo6 Infrastructure Docs:** /home/zvx/projects/.ref/docs/
+
+---
+
+## Notes
+
+### Why Not Docker?
+Previous attempts to install OpenTAKServer via Docker encountered persistent authentication issues. The native installation method worked successfully.
+
+### ISP-Specific Issue
+The Ubuntu mirror issue is specific to Filer Telephone Company's network peering. This may affect other services in the future. Consider:
+- Using mirrors.kernel.org for all Ubuntu-based containers
+- Setting up a local apt-cacher-ng if this becomes widespread
+- Monitoring ISP routing changes
+
+### Tailscale Not Involved
+Initial suspicion was that Tailscale exit nodes or DNS were causing the mirror issues. Testing confirmed this was false - the issue persists even with local DNS and no Tailscale routing.
+
+---
+
+*Last updated: 2026-04-17*
--- a/vault/docs/services/services.md
+++ b/vault/docs/services/services.md
@ -0,0 +1,411 @@
+# Current Services Inventory
+
+## Active Services
+
+| Service | Location | IP:Port | Access | Notes |
+|---------|----------|---------|--------|-------|
+| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel AutoAnnounce/AutoResponder) |
+| Utility Caddy | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
+| Echo6 Search (SearXNG) | utility (CT 102) | 192.168.1.102:8080 | https://echo6.co | Branded search homepage (Docker, custom theme) |
+| meshtasticd (AIDA-N2) | aida-nebra | 192.168.1.253:4403 | Internal | AIDA-N2(RPT,LLM) node !27780c47, Nebra 2W hat (ZebraHat), CLIENT_BASE role, fw 2.7.19. MeshAI (CT 108) connects via TCP localhost:4403 |
+| Meshtastic CLI | mt-isr | 192.168.1.141 | Internal | Station G2 WiFi bridge + TCP management |
+| meshtasticd | mt-burleybutte | 192.168.1.185:4403 | Internal | Software Meshtastic node (Nebra 2W hat) |
+| IdahoMesh Headscale | utility (CT 106) | 192.168.1.106:8080 | https://vpn.idahomesh.com | Meshtastic mesh VPN coordination |
+| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
+| MeshAI | utility (CT 108) | 192.168.1.144:4403 | Internal | LLM-powered Meshtastic assistant (Docker, Gemini Flash, Google grounding) |
+| ARGUS | utility (CT 103) | 192.168.1.103 | Internal | OSINT intelligence gathering platform (Docker, SearXNG + local LLM analysis) |
+| Authentik | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by Contabo Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
+| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via Contabo DNAT) | https://forge.echo6.co | Git server — fronted by Contabo Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on Contabo (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
+| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
+| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
+| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
+| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by Contabo Caddy (reverse_proxy 100.64.0.33:8086) |
+| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
+| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
+| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
+| Immich | cloud (CT 120) | 192.168.1.182:2283 | https://immich.echo6.co | Photo management (Docker, NFS storage on pi-nas) |
+| Nextcloud | cloud (CT 121) | 192.168.1.183:11000 | https://nextcloud.echo6.co | Cloud storage (AIO Docker, NFS on pi-nas, SSO) |
+| Jellyfin | media (VM 105) | 192.168.1.160:8096 | https://jellyfin.echo6.co | Media server (Docker, NFS on pi-nas, SSO) |
+| Jellyseer | media (VM 105) | 192.168.1.160:5055 | https://requests.echo6.co | Media request management (Docker, SSO) |
+| Sonarr | media (VM 105) | 192.168.1.160:8989 | Internal | TV automation (Docker) |
+| Radarr | media (VM 105) | 192.168.1.160:7878 | Internal | Movie automation (Docker) |
+| Prowlarr | media (VM 105) | 192.168.1.160:9696 | Internal | Indexer manager (Docker) |
+| SABnzbd | media (VM 105) | 192.168.1.160:8080 | Internal | Usenet download client (Docker) |
+| PeerTube | media (CT 110) | 192.168.1.170:9000 | https://stream.echo6.co | Video streaming (native, NFS on pi-nas, SSO) |
+| WATCHTOWER | **Decommissioned (2026-06-16)** | — | ~~wt.echo6.co~~ | Was Docker on Contabo `/opt/watchtower`; stopped & archived to forge.echo6.co/matt/archive-watchtower |
+| Open WebUI | cortex (VM 150) | 192.168.1.150:8080 | https://ai.echo6.co | AI chat interface (Docker, Ollama backend, SSO) |
+| Qdrant | cortex (VM 150) | 192.168.1.150:6333 | Internal | Vector database (Docker, RECON knowledge store) |
+| TEI | cortex (VM 150) | 192.168.1.150:8090 | Internal | Text embeddings (Docker, bge-m3 1024-dim) |
+| RECON | data (VM 1130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
+| Files | data (VM 1130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, Authentik forward auth) |
+| Samba | data | 192.168.1.240:445 | Internal | SMB file sharing — `//data/library` → /mnt/data/library (guest access) |
+| Matrix Synapse | Contabo | 127.0.0.1:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) |
+| Element Web | Contabo | 127.0.0.1:8088 | https://element.echo6.co | Matrix web client (Docker) |
+| mautrix-signal | Contabo | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) |
+| LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by Contabo Caddy (reverse_proxy 100.64.0.35:5984 / :5985); **migrated from Contabo 2026-06-16** |
+| TAK Server | **Decommissioned (2026-06-16)** | — | ~~tak.echo6.co~~ | Was Docker on Contabo `/opt/tak-server-deploy`; stopped & archived to forge.echo6.co/matt/archive-tak-server |
+| SIGIL | **Decommissioned (2026-06-16)** | — | ~~tak.echo6.co/sigil~~ | Was Docker on Contabo `/opt/sigil`; stopped & archived to forge.echo6.co/matt/archive-tak-server |
+| OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] |
+| Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
+| Echo6 Contabo Agent | Contabo | N/A (Matrix bot) | #contabo:echo6.co in echo6-ops space | Claude Code bridge — @contabo:echo6.co, session continuity, E2EE (systemd) |
+| mautrix-signal | Contabo | 29328 (internal) | Internal (matrix-net) | Signal bridge — @signalbot:echo6.co, E2BE, MSC4190, auto-portals |
+| Matrix MAS | Contabo | 127.0.0.1:8085 | Internal (via Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) |
+| Termix | Contabo | 0.0.0.0:8083 | Internal (no Caddy block) | Terminal sharing tool (Docker, ghcr.io/lukegus/termix:latest) |
+| Archivist | utility (CT 118) | 192.168.1.118 | Internal | Signal/Matrix room archive bot (systemd) — see archivist.md for details |
+| pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
+| recon-sparse | cortex (VM 150) | 192.168.1.150:8091 | Internal | RECON sparse embedding service (systemd, bge-m3 model, port 8091) |
+| Samba | cortex (VM 150) | 192.168.1.150:445 | Internal | SMB file sharing — `//cortex/projects` → /home/zvx/projects (guest access) |
+
+## Services by Server
+
+### toc - Proxmox Host (192.168.1.244 / Tailscale: 100.64.0.13)
+- Proxmox VE node (echo6-cluster)
+- GPU passthrough host for cortex VM
+- No direct services — workloads run on cortex VM
+
+### cortex - VM 150 on toc (192.168.1.150 / Tailscale: 100.64.0.14)
+- GPU compute VM (RTX A4000)
+- Claude Code host
+- Syncthing (syncs with Contabo)
+- Open WebUI (port 8080, https://ai.echo6.co, Docker, SSO via Authentik, Echo6 theme)
+  - Compose path: `/opt/open-webui/docker-compose.yml`
+  - Echo6 theme: togglable via "E6" button (bottom-right), persisted in localStorage
+  - Theme files bind-mounted from `/home/zvx/echo6-theme/` into container
+  - DEFAULT_USER_ROLE=user (new signups auto-activated, not pending)
+- Ollama (port 11434, internal, Docker with GPU)
+- Qdrant (port 6333, internal, Docker — vector DB for RECON)
+- TEI (port 8090, internal, Docker — bge-m3 embeddings for RECON)
+- PeerTube remote runner (peertube-runner service, Whisper auto-captioning via smart GPU/CPU wrapper, concurrency=2, MemoryMax=20G)
+- pt-transcoder (systemd: pt-transcoder.service, PeerTube H.265 NVENC transcoder)
+  - Script: `/opt/bulk-import/transcoder.py`
+  - MemoryMax=12G, Restart=always, RestartSec=60
+  - Depends on: nvidia-persistenced.service
+- recon-sparse (systemd: recon-sparse.service, RECON sparse embedding service)
+  - Script: `/opt/recon-sparse/sparse_embed_service.py --port 8091`
+  - Model: BAAI/bge-m3 (HuggingFace cache)
+  - Restart=on-failure, RestartSec=10
+- Samba (smbd/nmbd, system packages)
+  - Share: `//cortex/projects` → `/home/zvx/projects` (browseable, read-write, guest OK, force user/group zvx)
+  - Workgroup: WORKGROUP, standalone server
+- Echo6 Cortex Agent (systemd: echo6-agent.service, matrix-nio bot, @cortex:echo6.co)
+  - Install path: `/opt/echo6-agent/`
+  - Matrix space: echo6-ops, room: #cortex:echo6.co (E2EE, private)
+  - Session continuity via `claude -p --resume`, persistent per-room sessions
+  - `!new` resets conversation session
+  - Allowed users: @matt:echo6.co
+  - MAS user ID: 01KKX88ARGK0BTA1JMB2QVAW4C
+
+### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7)
+- MeshMonitor (port 8080, https://mesh.echo6.co)
+- Image: `meshmonitor:multichannel-new` (local build from zvx-echo6/meshmonitor fork, branch `feature/multi-channel-automation`)
+- Fork of Yeraze/meshmonitor with multi-channel AutoAnnounce and AutoResponder support (PR #2078 open upstream)
+
+### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8)
+- Utility Caddy (reverse proxy for VPN-only services)
+
+### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
+- Echo6 Search — branded SearXNG homepage (port 8080, https://echo6.co)
+- Custom cyberpunk theme: JetBrains Mono font, cyan/yellow palette, dark backgrounds
+- Homepage: centered Echo6 logo + pill search bar (Google-style, viewport-locked no-scroll)
+- Results page: full-width two-column grid (results + sidebar), stretched search header
+- Top nav bar: `.//photos`, `.//mail`, waffle app launcher (11 services), login avatar
+- All nav links use Authentik launch URLs for seamless SSO pass-through
+- search.echo6.co permanently redirects to echo6.co (301)
+- Redis/Valkey cache (valkey container)
+- Compose path: `/opt/searxng/docker-compose.yml`
+- Theme files: `/opt/searxng/custom/` (bind-mounted into container)
+  - `templates/simple/base.html` — custom template (nav, CSS, waffle menu, footer)
+  - `templates/simple/index.html` — custom homepage (Echo6 logo replaces SearXNG title)
+  - `img/echo6-logo.png` — Echo6 logo (replaces SearXNG logo)
+  - `img/favicon.png` — Echo6 favicon
+- Config: `/opt/searxng/searxng-config/settings.yml` (instance_name: "Echo6", dark theme, center_alignment: false)
+- SearXNG version: 2026.2.6 (Docker image: searxng/searxng:latest)
+
+### utility - CT 108 (192.168.1.144 / Tailscale: 100.64.0.32)
+- MeshAI — LLM-powered Meshtastic mesh assistant (Docker)
+- Bot name: AIDA, node ID !27780c47, channel 8 whitelist
+- Image: ghcr.io/zvx-echo6/meshai:latest (GitHub Actions multi-arch build)
+- Backend: Gemini 2.5 Flash with Google Search grounding
+- Connects to meshtasticd **on aida-nebra** (192.168.1.253:4403) — the AIDA-N2 node !27780c47
+- Config TUI on port 7682 (`meshai --config`)
+- Commands: !help, !ping, !status, !weather, !reset, !clear
+- 7-day rolling conversation memory (SQLite), full history sent to LLM
+- Response: 175 char chunks × 3 messages max
+- Compose path: `/home/zvx/meshai/docker-compose.yml`
+
+### utility - CT 118 (192.168.1.118)
+- Signal/Matrix room archive bot (archivist.service via systemd)
+- 1 core, 1GB RAM, 8GB disk
+- Not registered in Headscale (no Tailscale)
+- Source: forge.echo6.co/matt/matrix-archivist (private)
+- See `/home/zvx/projects/.ref/docs/matrix/archivist.md` for implementation details
+
+### cloud - CT 120 (192.168.1.182 / Tailscale: 100.64.0.2)
+- Immich photo management (https://immich.echo6.co)
+- Port 2283
+- NFS storage from pi-nas (/mnt/immich)
+- Compose path: `/opt/immich/docker-compose.yml`
+
+### cloud - CT 121 (192.168.1.183 / Tailscale: 100.64.0.11)
+- Nextcloud AIO (https://nextcloud.echo6.co)
+- Apache port 11000, AIO management on 8080
+- NFS storage from pi-nas (/mnt/nextcloud)
+- SSO via Authentik OIDC
+
+### media - VM 105 (192.168.1.160 / Tailscale: 100.64.0.18)
+- ARR media automation stack (Docker)
+- Jellyfin media server (port 8096, https://jellyfin.echo6.co)
+- Jellyseer request management (port 5055, https://requests.echo6.co)
+- Sonarr TV automation (port 8989, internal)
+- Radarr movie automation (port 7878, internal)
+- Prowlarr indexer manager (port 9696, internal)
+- SABnzbd Usenet downloader (port 8080, internal)
+- NFS storage from pi-nas (/mnt/arr)
+- Config dirs: /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
+
+### media - CT 110 (192.168.1.170 / Tailscale: 100.64.0.23)
+- PeerTube video streaming (https://stream.echo6.co)
+- Native install (Node.js 22, PostgreSQL 16, Redis, nginx)
+- Port 9000 (PeerTube), proxied via nginx on port 80
+- NFS storage from pi-nas (/var/www/peertube/storage, /export/peertube)
+- SSO via Authentik OIDC (peertube-plugin-auth-openid-connect)
+- Privileged container (NFS bind-mount)
+- Auto-transcription enabled (remote runners on cortex, Whisper medium model)
+- **PeerTube Bulk Import Pipeline:**
+  - pt-downloader.service — YouTube channel downloader (yt-dlp, sliding window, cookie auth)
+  - pt-importer.service — Uploads downloaded videos to PeerTube via resumable upload API
+  - NordVPN (nordvpnd.service) — IP rotation for downloads
+  - Config: `/opt/bulk-import/config/` (channel-map.json, cookies.txt, downloader-state.json)
+  - Logs: `/opt/bulk-import/logs/`
+  - Pipeline dirs: `/var/www/peertube/storage/pipeline/{staging,completed,transcoded,failed}`
+
+### data - Proxmox Host (192.168.1.240 / Tailscale: 100.64.0.6)
+- Proxmox VE node (echo6-cluster)
+- Samba (smbd/nmbd, system packages)
+  - Share: `//data/library` → `/mnt/data/library` (browseable, read-write, guest OK, force user/group root)
+  - Workgroup: WORKGROUP, standalone server
+
+### data - VM 1130 "recon-vm" (192.168.1.130 / Tailscale: 100.64.0.24)
+- **Migrated from CT 130 (LXC) on 2026-04-19**
+- OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-110-generic
+- Resources: 4 cores, 16GB RAM, 100GB disk
+- Software: Docker 29.4.0, Python 3.12.3 (venv), Tailscale, nginx, sqlite3
+- RECON knowledge extraction pipeline
+- systemd services: `recon.service`, `recon-watchdog.service`, `kiwix.service`
+- Dashboard + API on port 8420 (https://recon.echo6.co)
+- nginx file server on port 8888 (https://files.echo6.co, Authentik forward auth)
+- Kiwix-serve on port 8430 (ZIM library, 10 sources)
+- Install: `/opt/recon/` (Python 3, Flask, venv)
+- NFS mounts: pi-nas:/export/library → /mnt/library (PDF source), /mnt/nav, /mnt/kiwix
+- Pipeline: Extract (PyPDF2→pdftotext→Tesseract→Gemini Vision) → Enrich (Gemini) → Embed (TEI/Qdrant)
+- DB: SQLite (status), Qdrant on cortex:6333 (vectors)
+- Backups: rsync to Contabo every 6hrs (concepts, text, DB, config), DB snapshot every 2hrs
+- Config: `/opt/recon/config.yaml`, keys in `/opt/recon/.env`
+- Docs: `/opt/recon/PROJECT-BIBLE.md`
+- User: zvx (sudo, SSH key auth)
+
+### utility - CT 106 (192.168.1.106)
+- IdahoMesh Headscale (https://vpn.idahomesh.com)
+- Container name: meshtastic-hs
+- Manages meshtastic mesh VPN (separate from echo6 Headscale on Contabo)
+- Users: malice, sidpatchy, nebra
+
+### utility - CT 107 (192.168.1.107)
+- mesh-bridge — dual tailscaled instance
+- Bridges echo6 (100.64.0.0/10) ↔ idahomesh (100.100.0.0/16) networks
+- NAT masquerade + subnet route advertisement
+- Echo6 clients need `--accept-routes` to reach idahomesh devices
+- iptables FORWARD rules must be BEFORE `ts-forward` jump (Tailscale drops cross-tailnet packets otherwise)
+- Echo6 socket: `/run/tailscale/tailscaled.sock` (port 41641)
+- IdahoMesh socket: `/var/run/tailscale-meshtastic/tailscaled.sock` (port 41642, tun=tailscale1)
+- Rules persisted: `/etc/iptables/rules.v4` via `iptables-restore.service`
+
+### pi-nas (192.168.1.245 / Tailscale: 100.64.0.21)
+- OpenMediaVault NAS (https://nas.echo6.co)
+- Port 80 (HTTP)
+- Internet Archive CLI (`ia` v5.7.2) installed for archive.org uploads
+
+### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9)
+- **AIDA-N2(RPT,LLM)** — meshtasticd node `!27780c47` (short name: AIDA)
+- Hardware: Nebra 2W SX1262 hat (ZebraHat config in `/etc/meshtasticd/config.d/`)
+- Port: 4403 (default), firmware 2.7.19 (PORTDUINO/native)
+- Role: CLIENT_BASE, position: 42.574, -114.607 (manual)
+- MAC source: eth0 (derived MAC `00:bd:27:78:0c:47`)
+- MeshAI bot (CT 108) connects to this node via TCP `localhost:4403` (Docker network)
+- Service: `meshtasticd.service` (single instance, runs as user meshtastic)
+- Config: `/etc/meshtasticd/config.yaml` + `/etc/meshtasticd/config.d/ZebraHat_2W.yaml`
+- User: zvx, password auth (`sshpass -p '7redditGold' ssh zvx@aida-nebra`)
+
+### mt-isr (192.168.1.141 / IdahoMesh: 100.100.0.5)
+- Raspberry Pi Zero 2 W, Debian 13 (trixie), Waveshare ETH/USB HUB HAT
+- No meshtasticd (G2 managed via WiFi TCP, not local daemon)
+- Meshtastic Python CLI v2.7.7 in venv (`/home/isr/meshtastic-cli/`)
+- Tailscale on IdahoMesh tailnet (vpn.idahomesh.com, nebra user)
+- WiFi hotspot: ISR-MESH (192.168.4.0/24, PMF disabled for ESP32 compatibility)
+- Station G2 radio connected via WiFi at 192.168.4.241, managed via TCP
+- G2 config: Freq51 (ch0, psk=1A==) + MediumFast (ch1), MEDIUM_FAST preset, ch=51, txPower=11
+- G2 gold config backup: `isr@192.168.1.141:~/backups/g2-gold-config.yaml`
+- DNS bootstrap drop-in for tailscaled (reboot-safe)
+- User: isr, password auth (see credentials)
+
+### mt-burleybutte (192.168.1.185)
+- meshtasticd (software Meshtastic node, Nebra 2W hat)
+- Raspberry Pi OS, user bb
+- Static MAC: A7:A1:30:79:BB:BB
+- Tailscale registered on IdahoMesh Headscale (vpn.idahomesh.com) under malice user
+
+### edge2 - CT 103 (10.10.10.21 / Tailscale: 100.64.0.34, node 46 `forgejo`)
+- Forgejo git server (https://forge.echo6.co — **migrated from Contabo 2026-06-16**)
+  - Headscale node id 46, name `forgejo`, user `echo6`
+  - Compose path: `/opt/forgejo/docker-compose.yml`
+  - Containers: `forgejo` (codeberg.org/forgejo/forgejo:14) + `forgejo-db` (postgres:16-alpine)
+  - Volumes: `forgejo-data` (git repos, app.ini, SSH host keys) + `forgejo-db`
+  - HTTP: binds to `100.64.0.34:3001`; Contabo Caddy proxies here over tailnet (`reverse_proxy 100.64.0.34:3001`)
+  - SSH: `git@forge.echo6.co:2222` → Contabo iptables DNAT (systemd unit `forgejo-ssh-dnat.service`, enabled, reboot-persistent) → `100.64.0.34:2222` → container port 22; SSH host keys preserved (no client warning)
+  - Login via Authentik OIDC (auth.echo6.co) — unchanged
+  - DB: PostgreSQL 16 (forgejo_db container); 9 repos, 1 user verified at migration
+  - Source (Contabo `/opt/forgejo`) STOPPED but intact as cold rollback; rollback = `systemctl disable --now forgejo-ssh-dnat` + restore `/etc/caddy/Caddyfile.bak-preforge` + `systemctl restart caddy` + `docker compose up -d` on Contabo
+
+### edge2 - CT 104 (10.10.10.22 / Tailscale: 100.64.0.35, node `livesync`)
+- LiveSync Obsidian sync service (https://notes.echo6.co — **migrated from Contabo 2026-06-16**)
+  - Headscale hostname `livesync`, tailnet IP 100.64.0.35
+  - Compose path: `/opt/livesync/docker-compose.yml`
+  - Containers: `livesync-couchdb` (couchdb:3.4) + `livesync-provisioner` (custom image)
+  - Named volumes: `couchdb-data`, `provisioner-data`
+  - Bind mounts: `couchdb/local.ini`, `couchdb/local.d/` (incl. `jwt-keys.ini`)
+  - CouchDB binds to `100.64.0.35:5984`; provisioner binds to `100.64.0.35:5985`
+  - Contabo Caddy proxies `notes.echo6.co` → `reverse_proxy 100.64.0.35:5984` (CouchDB) + `reverse_proxy 100.64.0.35:5985` (provisioner); Authentik forward_auth on `/_provision`; Obsidian CORS preserved; Authentik outpost stays `127.0.0.1:9000`
+  - Auth: per-user JWT (ES512) via provisioner; databases: `cc-db`, `userdb-matt`
+  - Data: ~16 MB in couchdb-data
+  - Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-prelivesync` exists
+  - **Resources:** 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs on `local`; unprivileged; onboot; Docker
+
+### edge2 - CT 105 (10.10.10.23 / Tailscale: 100.64.0.36, node 48 `authentik`)
+- Authentik SSO platform (https://auth.echo6.co — **migrated from Contabo 2026-06-18**)
+  - Headscale node id 48, hostname `authentik`, tailnet IP 100.64.0.36
+  - Compose path: `/opt/authentik/docker-compose.yml`
+  - Containers: `authentik-server` + `authentik-worker` (ghcr.io/goauthentik/server:2025.12.4) + `authentik-postgres` (postgres:16); NO Redis
+  - Worker runs as user:root and has docker.sock bind-mount (manages embedded outposts)
+  - Binds to `100.64.0.36:9000`; Contabo Caddy proxies here over tailnet for both `auth.echo6.co` (catch-all + outpost path matcher) and `notes.echo6.co` outpost/forward_auth references
+  - `AUTHENTIK_SECRET_KEY` carried byte-for-byte (sessions stayed valid across cutover — users dropped straight in)
+  - Bind-mounts (data/media, branding, certs, custom-templates) migrated intact
+  - Email dep: mail.echo6.co (unchanged)
+  - DB: PostgreSQL 16 (`authentik-postgres` container); ~705 MB (~18 MB pg_dump)
+  - Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists on Contabo
+  - Reboot-survival fix: systemd unit on CT 105 gates `docker compose up` on `tailscale-online` (Docker was racing Tailscale on boot, failing the bind to the tailnet IP)
+  - **Resources:** 2 cores / 4096 MB RAM / 512 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
+
+### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
+- Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
+  - Headscale node id 45, name `vaultwarden`, user `echo6`
+  - Compose path: `/opt/vaultwarden/docker-compose.yml`; data: `./data/`
+  - Binds to tailnet IP `100.64.0.33:8086:80`; Contabo Caddy proxies here over tailnet
+  - SSO via Authentik (`SSO_ONLY=false` — local email+master-password login also works)
+  - Source (Contabo `/opt/vaultwarden`) STOPPED but intact as cold rollback; rollback = restore `/etc/caddy/Caddyfile.bak-prevault` + `systemctl restart caddy` + `docker compose up -d` on Contabo
+  - **Migrated from Contabo to edge2 CT 102 on 2026-06-16**
+
+### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
+- ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (Caddy now proxies auth.echo6.co + notes.echo6.co outpost/forward_auth → 100.64.0.36:9000; source STOPPED at `/opt/authentik`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists)
+- Forge (Git) — **migrated to edge2 CT 103 on 2026-06-16** (Caddy now proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service)
+- Headscale (mesh VPN)
+- Mailcow (email)
+- Vaultwarden — **migrated to edge2 CT 102 on 2026-06-16** (Caddy now proxies to 100.64.0.33:8086)
+- Syncthing (syncs with cortex)
+- ~~WATCHTOWER~~ — **decommissioned 2026-06-16** (was `/opt/watchtower`; archived to forge.echo6.co/matt/archive-watchtower; `wt.echo6.co` Caddy block + GoDaddy A record removed)
+- Matrix Synapse homeserver (port 8008, https://matrix.echo6.co, Docker, SSO via Authentik)
+- Element Web client (port 8088, https://element.echo6.co, Docker)
+- mautrix-signal bridge (port 29328 internal, Docker, E2BE with MSC4190)
+  - Image: `dock.mau.dev/mautrix/signal:v0.2603.0`
+  - Container: `mautrix-signal` on `matrix-net`
+  - Config: `/opt/matrix/mautrix-signal/config.yaml`
+  - Registration: `/opt/matrix/synapse/registration.yaml` + `/opt/matrix/synapse/doublepuppet.yaml`
+  - Database: `mautrix_signal` on `matrix-postgres` (role: `mautrix_signal`, minimal grants)
+  - Bot user: `@signalbot:echo6.co`, device: `UPX4KKLZVY`
+  - Permissions: `@matt:echo6.co` = admin, `echo6.co` = user
+  - Double puppeting: appservice-based (doublepuppet.yaml as_token)
+  - Encryption: E2BE enabled (allow+default+require), MSC4190, self-signed cross-signing keys
+- Compose path: `/opt/matrix/docker-compose.yml`
+- Backup: daily at 3AM, 14-day retention (synapse + mas + mautrix_signal databases)
+- ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16** (source `/opt/livesync` STOPPED, intact as cold rollback; Caddy now proxies `notes.echo6.co` → `100.64.0.35:5984/5985`; `/etc/caddy/Caddyfile.bak-prelivesync` exists)
+- ~~TAK Server~~ — **decommissioned 2026-06-16** (was `/opt/tak-server-deploy`; archived to forge.echo6.co/matt/archive-tak-server; `tak.echo6.co` Caddy block + GoDaddy A record removed)
+- ~~SIGIL~~ — **decommissioned 2026-06-16** (was `/opt/sigil`; archived to forge.echo6.co/matt/archive-tak-server)
+- Matrix Authentication Service (MAS) (port 8085, internal, Docker)
+  - Container: `matrix-mas` on `matrix-net`
+  - Handles login/logout/refresh/auth_metadata for Synapse
+  - Caddy routes: `/_matrix/client/*/login`, `/_matrix/client/*/logout`, `/_matrix/client/*/refresh`, `/_matrix/client/*/auth_metadata` → MAS (8085); `/_matrix/*` and `/_synapse/*` → Synapse (8008); default → MAS (8085)
+  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
+- Termix (port 8083, internal, Docker)
+  - Container: `termix` on `termix_default` network
+  - Image: `ghcr.io/lukegus/termix:latest`
+  - Port: 8080→8083 (bound to 0.0.0.0, NOT 127.0.0.1)
+  - Volume: `termix_termix-data` → `/app/data`
+  - No Caddy block — direct access only on port 8083
+  - Compose: `/opt/termix/` (inferred from Docker volume naming)
+- ~~echo6-agent~~ — **decommissioned 2026-06-16** (was systemd `echo6-agent.service` at `/opt/echo6-agent/`; archived to forge.echo6.co/matt/archive-echo6-agent; unit disabled)
+- ~~nexus-hub~~ + ~~nexus-agent~~ — **decommissioned 2026-06-16** (were systemd units at `/root/nexus-hub` and `/root/nexus-agent`; archived to forge.echo6.co/matt/archive-nexus-hub + archive-nexus-agent; units disabled)
+- mautrix-signal bridge (mautrix-signal container, port 29328 internal)
+  - Image: dock.mau.dev/mautrix/signal:v0.2603.0
+  - Config: `/opt/matrix/mautrix-signal/config.yaml`
+  - Compose: `/opt/matrix/docker-compose.yml` (shared with Synapse stack)
+  - DB: mautrix_signal on matrix-postgres
+  - Bot: @signalbot:echo6.co, management room !fDjIRTMjxILVQoAcEN:echo6.co
+  - E2BE enabled (MSC4190), double puppeting via doublepuppet.yaml
+  - Signal account: +12083080811 (@matt:echo6.co)
+  - Portals auto-create on incoming messages (no autocreate toggle available)
+  - Ref: `/home/zvx/projects/.ref/docs/matrix/mautrix_signal.md`
+
+## Decommissioned Services
+
+Services stopped, archived, and removed from Caddy/DNS as of their decommission date. On-disk dirs on Contabo retained until edge1 rebuild wipes them.
+
+| Service | Decommissioned | Archive Repo | Notes |
+|---------|---------------|-------------|-------|
+| TAK Server | 2026-06-16 | forge.echo6.co/matt/archive-tak-server | Was Docker `/opt/tak-server-deploy` on Contabo; `tak.echo6.co` Caddy block + GoDaddy A record removed |
+| SIGIL | 2026-06-16 | forge.echo6.co/matt/archive-tak-server | Was Docker `/opt/sigil` on Contabo; served at `tak.echo6.co/sigil` |
+| WATCHTOWER | 2026-06-16 | forge.echo6.co/matt/archive-watchtower | Was Docker `/opt/watchtower` on Contabo; `wt.echo6.co` Caddy block + GoDaddy A record removed |
+| echo6-agent | 2026-06-16 | forge.echo6.co/matt/archive-echo6-agent | Was systemd unit at `/opt/echo6-agent/` on Contabo; unit disabled |
+| nexus-hub | 2026-06-16 | forge.echo6.co/matt/archive-nexus-hub | Was systemd unit at `/root/nexus-hub` on Contabo; unit disabled |
+| nexus-agent | 2026-06-16 | forge.echo6.co/matt/archive-nexus-agent | Was systemd unit at `/root/nexus-agent` on Contabo; unit disabled |
+
+## Adding New Services
+
+When deploying a new service, update this file with:
+1. Service name
+2. Host location (server + container if applicable)
+3. IP:Port
+4. Access method (internal only vs public URL)
+5. Brief description
+
+## Naming Conventions
+
+- **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP
+- **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy
+
+### Lidarr on Steroids (lidarr.echo6.co)
+- **Container:** lidarr (youegraillot/lidarr-on-steroids:latest)
+- **Host:** media VM 105 (192.168.1.160)
+- **Ports:** 8686 (Lidarr), 6595 (Deemix)
+- **Network:** arr-net
+- **Config:** /opt/arr/lidarr/config (Lidarr), /opt/arr/lidarr/config_deemix (Deemix)
+- **Compose:** /opt/arr/docker-compose.yml
+- **Music root:** /mnt/arr/music (NFS from pi-nas)
+- **Downloads:** /mnt/arr/downloads (shared with SABnzbd)
+- **API key:** 78f026ec93a94d8eb3177816b74a57b7
+- **Caddy:** lidarr.echo6.co -> 100.64.0.18:8686 (Authentik forward auth)
+- **Prowlarr:** fullSync configured
+- **SABnzbd:** configured (music category)
+- **Deemix:** port 6595, NOT exposed via Caddy (Tailscale-only access)
+- **PUID/PGID:** 1000/1000, TZ: America/Boise
+
+### Navidrome (navidrome.echo6.co)
+- **Container:** navidrome (deluan/navidrome:latest)
+- **Host:** media VM 105 (192.168.1.160)
+- **Port:** 4533
+- **Network:** arr-net
+- **Data volume:** arr_navidrome-data (named Docker volume)
+- **Music volume:** /mnt/arr/music (read-only, shared with Lidarr)
+- **Compose:** /opt/arr/docker-compose.yml
+- **Caddy:** navidrome.echo6.co -> 100.64.0.18:4533 (Authentik forward auth)
+- **User:** 1000:1000
+- **Scan schedule:** every 1 hour
+- **Admin setup:** First login at https://navidrome.echo6.co creates admin account
--- a/vault/docs/services/usenet.md
+++ b/vault/docs/services/usenet.md
@ -0,0 +1,53 @@
+# Usenet Configuration
+
+> **Credentials:** All passwords and API keys stored in `/home/zvx/projects/.ref/credentials`
+
+## Infrastructure
+
+| Component | Location | IP | Port | Access |
+|-----------|----------|-----|------|--------|
+| SABnzbd | arr VM (media VM 105) | 192.168.1.160 | 8080 | Internal only (LAN/Tailscale) |
+
+## Usenet Providers
+
+### Giganews
+
+| Field | Value |
+|-------|-------|
+| Host | news.giganews.com |
+| Port | 563 |
+| SSL | Yes |
+| Username | gn1369344 |
+| Password | (see .ref/credentials) |
+| Connections | 100 |
+
+### EasyUsenet
+
+| Field | Value |
+|-------|-------|
+| Host | reader.easyusenet.nl |
+| Port | 563 |
+| SSL | Yes |
+| Username | esy_6926b8b819b97 |
+| Password | (see .ref/credentials) |
+| Connections | 100 |
+| Speed | 2000 Mbit/s |
+| Retention | 3700+ days |
+| Alt Ports (SSL) | 443 |
+| Alt Ports (non-SSL) | 119, 80 |
+
+## Usenet Indexers
+
+### NZBgeek
+
+| Field | Value |
+|-------|-------|
+| URL | https://api.nzbgeek.info |
+| API Key | (see .ref/credentials) |
+
+### NinjaCentral
+
+| Field | Value |
+|-------|-------|
+| URL | https://ninjacentral.co.za |
+| API Key | (see .ref/credentials) |