matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-07-13T12:00:23+00:00

Browse source 
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.trash/2026-06-19.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/matrix/archivist.md vault/docs/matrix/matrix_host.md vault/docs/matrix/mautrix_signal.md vault/docs/matrix/synapse.md vault/docs/matrix/synapse_retention_discovery.md vault/docs/navi/cc-rules.md vault/docs/navi/deployment.md vault/docs/navi/themes.md vault/docs/services/ots-setup.md vault/docs/services/services.md vault/docs/services/usenet.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/central.md vault/docs/software/dns.md vault/docs/software/geo-tools.md vault/docs/software/navi.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/notes/ia-download-queue.md vault/projects/advbbs-project.md vault/projects/argus.md vault/projects/deploy-livesync.md vault/projects/fleet-patch-audit.md vault/projects/fleet-platform-baseline.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai-config-hot-apply.md vault/projects/meshai-region-routing-plan.md vault/projects/meshai.md vault/projects/meshcore-transport.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/projects/nominatim-v5-reimport.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/central-deploy-cutover.md vault/runbooks/ct-runbook.md vault/runbooks/edge2-access-reference.md vault/runbooks/expose-service-contabo.md vault/runbooks/expose-service-edge2.md vault/runbooks/expose-service-home.md vault/runbooks/fleet-magicdns-resolved-migration.md vault/runbooks/headless-browser-page-verification.md vault/runbooks/headscale-oidc-boot-order.md vault/runbooks/headscale-onboard-node.md vault/runbooks/ia-cli-reference.md vault/runbooks/ia-download-mirror.md vault/runbooks/idahomesh-bridge-setup.md vault/runbooks/idahomesh-vpn-device-setup.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshai-prod-compose-override.md vault/runbooks/meshmonitor-password-reset.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/nordvpn-lxc.md vault/runbooks/peertube-remote-runner.md vault/runbooks/pg-backup.md vault/runbooks/pi-nas-omv-runbook.md vault/runbooks/pipeline-patterns.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/proxmox-onboard-node.md vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md vault/runbooks/toc-cortex-pve9.2-update.md vault/session-resume/SESSION-HANDOFF-meshai-test.md
This commit is contained in:
echo6-autocommit 2026-07-13 12:00:23 +00:00
commit ef8b1e0bd9
79 changed files with 448 additions and 360 deletions
Download patch file Download diff file Expand all files Collapse all files

2
engine/.embcache.json
View file

File diff suppressed because one or more lines are too long

2
engine/changelog.md
View file

@ -161,3 +161,5 @@
## 2026-07-10T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-11T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-13T09:00:01Z — sweep run

2
engine/lint-report.md
View file

@ -1,6 +1,6 @@
# Vault Lint Report
Generated: 2026-07-13T00:00:34Z | Docs scanned: 105 | Elapsed: 0.0s
Generated: 2026-07-13T09:00:01Z | Docs scanned: 105 | Elapsed: 0.0s
## Summary

8
vault/.trash/2026-06-19.md
View file

@ -0,0 +1,8 @@
---
title: 2026 06 19
type: reference
tags:
- mesh
aliases: []
updated: 2026-07-13
---

30
vault/docs/hardware/environment.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ip-allocation]]
- [[headscale-onboard-node]]
- [[caddy]]
- [[ct-runbook]]
- [[fleet-platform-baseline]]
- [[proxmox-create-ubuntu-vm]]
updated: 2026-06-19
- [[headscale-onboard-node]]
- [[ct-runbook]]
updated: 2026-07-13
---
# Echo6 Environment Reference
@ -21,7 +21,7 @@ Five nodes running Proxmox VE:
| Node | Local IP | Tailscale | Hardware | RAM | Purpose |
| ------- | ------------- | ----------- | ----------------------------------------------- | -------------- | -------------------------------- |
| data | 192.168.1.240 | 100.64.0.6 | AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD | 32GB DDR4-3200 | Database [[services]] |
| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility services, monitoring |
| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility [[services]], monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | Intel i7-12700T, 512GB NVMe | 32GB DDR4-3200 | Cloud storage, personal services |
| media | 192.168.1.243 | 100.64.0.3 | Intel i7-14700T, 2x 512GB NVMe | 32GB DDR5-5600 | Media server, *arr stack |
| toc | 192.168.1.244 | 100.64.0.13 | Workstation (i9-10900X) | 64GB DDR4 | GPU compute, AI/ML workloads |
@ -38,7 +38,7 @@ Five nodes running Proxmox VE:
### Network Notes
- **media NIC:** Original Intel e1000e NIC crashes under sustained NFS load — replaced with USB Realtek RTL8153 GbE adapter on vmbr0
- **Tailscale [[dns]] bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback DNS exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot
- **Tailscale [[dns]] bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback [[dns]] exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot
### TOC Node Details
@ -72,7 +72,7 @@ Five nodes running Proxmox VE:
- **Resources:** 4 cores, 24GB RAM, 180GB disk
- **Software:** Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
- **Platforms:** [[recon]] (knowledge extraction pipeline, :8420) and [[navi]] (offline navigation, navi.echo6.co, :8440) with geo backends (Valhalla :8002, Nominatim :8010, Photon :2322, PostgreSQL/PostGIS :5432)
- **Systemd services:** recon (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **Systemd services:** [[recon]] (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **NFS mounts:** pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
- **User:** zvx (sudo, SSH key auth)
- **Migrated from:** CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
@ -93,17 +93,17 @@ Five nodes running Proxmox VE:
| Server | Local IP | Tailscale | Purpose |
|--------|----------|-----------|---------|
| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. MeshAI (CT 108) connects here via TCP |
| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. [[meshai]] (CT 108) connects here via TCP |
| mt-isr | 192.168.1.141 | 100.100.0.5 (IdahoMesh) | Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI) |
| mt-burleybutte | 192.168.1.185 | — | Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN) |
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
| matt-desktop | 192.168.1.254 | 100.64.0.10 | Personal workstation (Windows, your PC) |
| ha | 192.168.1.151 | 100.64.0.16 | Home Assistant (VM 151 on cloud, Docker, home automation) |
| **edge1** (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, **mail-only** — Mailcow in CT 101; host Caddy + mailcow-dnat.service; rebuilt 2026-06-19 |
| **edge1** (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, **mail-only** — Mailcow in CT 101; host [[caddy]] + mailcow-dnat.service; rebuilt [[2026-06-19]] |
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB — **permanent front door** for vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh/intermountainmesh |
*Last updated: 2026-06-19 — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other services; Headscale node `contabo` moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
*Last updated: 2026-06-19 — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other services; Headscale node `contabo` moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 ([[authentik]]), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
## LXC Containers
@ -111,14 +111,14 @@ Five nodes running Proxmox VE:
|-----------|------|----------|-----------|---------|
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel) |
| [[caddy]] | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
| [[searxng]] | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage (SearXNG, echo6.co) |
| [[searxng]] | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage ([[searxng]], echo6.co) |
| immich | cloud (CT 120) | 192.168.1.182 | 100.64.0.2 | Immich photo management |
| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| meshai | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | MeshAI - LLM-powered Meshtastic assistant |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | — | Archivist knowledge pipeline |
| [[argus]] | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | ARGUS - OSINT intelligence gathering platform |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | — | [[archivist]] knowledge pipeline |
| [[argus]] | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | [[argus]] - OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104 | 100.64.0.12 | Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see [[central]] |
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.23 | PeerTube video streaming |
| mcc | media (CT 111) | 192.168.1.111 | — | pymc console web app (Caddy + Postfix, /api+/auth+/ws → aida-nebra :8000) |
@ -129,7 +129,7 @@ Five nodes running Proxmox VE:
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| authentik | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
> **Note (2026-06-19):** edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (`forgejo-ssh-dnat.service`) is a permanent systemd unit on edge2 host.
@ -174,7 +174,7 @@ Current registered nodes (25 total):
| peertube | 100.64.0.23 | LXC |
| recon | 100.64.0.24 | VM |
| argus | 100.64.0.25 | LXC |
| central | 100.64.0.12 | LXC (utility CT 104 — central.echo6.mesh) |
| [[central]] | 100.64.0.12 | LXC (utility CT 104 — central.echo6.mesh) |
| edge2 | 100.64.0.26 | Proxmox/Contabo VPS |
| gl-a1300 | 100.64.0.29 | Router |
| bluefin | 100.64.0.30 | Desktop |

22
vault/docs/hardware/ip-allocation.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[services]]
- [[environment]]
- [[caddy]]
- [[environment]]
- [[glossary]]
- [[headscale-onboard-node]]
- [[meshtastic-headscale-runbook]]
updated: 2026-06-19
updated: 2026-07-13
---
# Echo6 Network IP Allocation
@ -42,8 +42,8 @@ updated: 2026-06-19
|----|-----------|------|---------|
| .100 | meshmonitor (CT 100) | utility | MeshMonitor web UI |
| .101 | [[caddy]] (CT 101) | utility | Home reverse proxy |
| .102 | [[searxng]] (CT 102) | utility | Echo6 Search (SearXNG) |
| .103 | [[argus]] (CT 103) | utility | ARGUS OSINT platform |
| .102 | [[searxng]] (CT 102) | utility | Echo6 Search ([[searxng]]) |
| .103 | [[argus]] (CT 103) | utility | [[argus]] OSINT platform |
| .104 | meshing-around (CT 104) | utility | Mesh bot + WebGUI |
| .106 | meshtastic-hs (CT 106) | utility | IdahoMesh Headscale |
| .107 | mesh-bridge (CT 107) | utility | Dual-tailscaled bridge |
@ -55,8 +55,8 @@ updated: 2026-06-19
| .115 | mmud-trvl (CT 115) | utility | MMUD SIM: TRVL (Torval merchant) |
| .116 | mmud-wspr (CT 116) | utility | MMUD SIM: WSPR (Whisper sage) |
| .118 | [[archivist]] (CT 118) | utility | Signal/Matrix archive bot |
| .130 | [[recon]] (VM 1130) | data | RECON pipeline (migrated from CT 130) |
| .144 | meshai (CT 108) | utility | MeshAI assistant |
| .130 | [[recon]] (VM 1130) | data | [[recon]] pipeline (migrated from CT 130) |
| .144 | [[meshai]] (CT 108) | utility | MeshAI assistant |
| .170 | peertube (CT 110) | media | PeerTube streaming |
| .182 | immich (CT 120) | cloud | Immich photos |
| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO |
@ -67,7 +67,7 @@ edge1 (rebuilt Contabo VPS, 5.189.158.149 / Tailscale 100.64.0.40) runs Debian 1
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.2 | mailcow | CT 101 | — | Mailcow email server (privileged LXC; reached via host DNAT ports 25/465/587/110/143/993/995/4190 and host Caddy for mail/autodiscover/autoconfig.echo6.co → :8453) |
| 10.10.10.2 | mailcow | CT 101 | — | Mailcow email server (privileged LXC; reached via host DNAT ports 25/465/587/110/143/993/995/4190 and host [[caddy]] for mail/autodiscover/autoconfig.echo6.co → :8453) |
### edge2 LXC Containers (10.10.10.x, vmbr0) — permanent front door (184.174.35.153)
@ -80,9 +80,9 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16); git SSH → edge2 forgejo-ssh-dnat.service |
| 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| 10.10.10.23 | [[authentik]] | CT 105 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
| 10.10.10.24 | matrix | CT 106 | 100.64.0.37 | Matrix stack (Synapse + MAS + Element + mautrix-signal; migrated from Contabo 2026-06-18) |
| 10.10.10.25 | headscale | CT 107 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
| 10.10.10.23 | [[authentik]] | CT 105 | 100.64.0.36 | [[authentik]] SSO platform (migrated from Contabo 2026-06-18) |
| 10.10.10.24 | matrix | CT 106 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
| 10.10.10.25 | headscale | CT 107 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo [[2026-06-19]]) |
| 10.10.10.26 | mailcow-staging | CT 108 | — | Stopped Mailcow staging replica (fallback; prune after soak) |
### VMs (.150-.199)

8
vault/docs/matrix/archivist.md
View file

@ -5,12 +5,12 @@ tags:
- matrix
aliases: []
related:
- [[caddy]]
- [[mautrix_signal]]
- [[synapse]]
- [[matrix-synapse-deployment]]
- [[services]]
- [[caddy]]
- [[recon-operations]]
updated: 2026-06-18
updated: 2026-07-13
---
# Signal Archive Bot — Deployment Reference
# Created: 2026-04-12 (Phase 3)
@ -167,7 +167,7 @@ pct exec 118 -- bash
- Shared-secret registration (`/_synapse/admin/v1/register`) returns 404 under MAS — endpoint disabled
- Must use `mas-cli manage register-user` or `manage set-password` for existing users
- MAS creates user in both MAS DB and [[synapse]] DB
- Orphaned Synapse `profiles` row caused provisioning failure — fixed by DELETE
- Orphaned [[synapse]] `profiles` row caused provisioning failure — fixed by DELETE
- Each `client.login()` creates a NEW MAS compat session with random device ID — use `restore_login()` with stable compat token instead
- matrix-nio v0.25.2 does NOT implement `bootstrap_cross_signing()` — manual implementation required via python-olm PkSigning + raw HTTP API

8
vault/docs/matrix/matrix_host.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[matrix-synapse-deployment]]
- [[synapse]]
- [[expose-service-contabo]]
- [[ct-runbook]]
- [[lxc-service-migration]]
updated: 2026-07-11
- [[caddy]]
- [[ip-allocation]]
- [[services]]
updated: 2026-07-13
---
# Matrix Host Reference — edge2 CT 106

6
vault/docs/matrix/mautrix_signal.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[synapse]]
- [[matrix-synapse-deployment]]
- [[synapse_retention_discovery]]
- [[matrix-synapse-deployment]]
- [[archivist]]
- [[advbbs-project]]
updated: 2026-06-18
updated: 2026-07-13
---
# mautrix-signal Bridge Reference
# Deployed: 2026-04-09 (Phase 3)
@ -29,7 +29,7 @@ updated: 2026-06-18
- **DB name:** mautrix_signal
- **DB role:** mautrix_signal (NOSUPERUSER, NOCREATEDB, NOCREATEROLE)
- **Host:** matrix-postgres:5432 (same container as Synapse/MAS)
- **Host:** matrix-postgres:5432 (same container as [[synapse]]/MAS)
- **Collation:** C/C (matches Synapse)
## Signal Account

4
vault/docs/matrix/synapse.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[matrix-synapse-deployment]]
- [[mautrix_signal]]
- [[matrix_host]]
- [[synapse_retention_discovery]]
- [[archivist]]
- [[caddy]]
updated: 2026-07-11
updated: 2026-07-13
---
# Synapse Deployment Reference
# Generated: 2026-04-09 (Phase 1)

8
vault/docs/matrix/synapse_retention_discovery.md
View file

@ -5,12 +5,12 @@ tags:
- matrix
aliases: []
related:
- [[mautrix_signal]]
- [[synapse]]
- [[mautrix_signal]]
- [[matrix-synapse-deployment]]
- [[SESSION-HANDOFF-meshai-test]]
- [[caddy]]
updated: 2026-06-18
- [[pymc-repeater-kiss-tnc-reenumeration]]
updated: 2026-07-13
---
# Synapse Retention Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 1)
@ -46,7 +46,7 @@ Two settings added to homeserver.yaml:
| Setting | Previous Value | Current Value | Source |
|---------|---------------|---------------|--------|
| redaction_retention_period | NOT SET (default 7d) | null (disabled) | synapse/config/server.py |
| redaction_retention_period | NOT SET (default 7d) | null (disabled) | [[synapse]]/config/server.py |
| msc2815_enabled | NOT SET (default false) | true | synapse/config/experimental.py |
| forgotten_room_retention_period | NOT SET | NOT SET (unchanged) | synapse/config/server.py |
| media_retention.local_media_lifetime | NOT SET | NOT SET (unchanged) | synapse/config/repository.py |

8
vault/docs/navi/cc-rules.md
View file

@ -7,16 +7,16 @@ aliases: []
related:
- [[deployment]]
- [[CLAUDE-baseline]]
- [[environment]]
- [[navi]]
- [[themes]]
- [[caddy]]
updated: 2026-06-18
- [[environment]]
updated: 2026-07-13
---
# Navi: Claude Code Rules
## Repository & SSH
**All Navi SSH goes to:** `recon-vm` (VM 1130, 192.168.1.130)
**All [[navi]] SSH goes to:** `recon-vm` (VM 1130, 192.168.1.130)
**Never SSH to cortex for Navi work.** Previous attempt to deploy from cortex wrecked production by deploying from a stale clone.

6
vault/docs/navi/deployment.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[cc-rules]]
- [[environment]]
- [[navi]]
- [[recon-operations]]
- [[ct-runbook]]
- [[central-deploy-cutover]]
- [[themes]]
updated: 2026-06-18
updated: 2026-07-13
---
# Navi Deployment

6
vault/docs/navi/themes.md
View file

@ -5,19 +5,19 @@ tags:
- auth
aliases: []
related:
- [[navi]]
- [[cc-rules]]
- [[deployment]]
- [[searxng]]
- [[echo6-landing-page-data-export]]
- [[pipeline-patterns]]
updated: 2026-06-18
updated: 2026-07-13
---
# Navi Theme System
## Architecture
**Registry:** `src/themes/registry.js`
Central source for theme metadata, overlay config, UI CSS vars, and satellite adjustments.
[[central]] source for theme metadata, overlay config, UI CSS vars, and satellite adjustments.
## Critical: namedTheme Import

12
vault/docs/services/ots-setup.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ct-runbook]]
- [[caddy]]
- [[ip-allocation]]
- [[environment]]
- [[synapse]]
updated: 2026-06-18
- [[ip-allocation]]
- [[expose-service-home]]
- [[proxmox-onboard-node]]
updated: 2026-07-13
---
# OpenTAKServer Setup Documentation
@ -74,7 +74,7 @@ pct enter 109
### SSL Certificate
- **Provider:** Let's Encrypt
- **Method:** acme.sh with GoDaddy DNS validation
- **Method:** acme.sh with GoDaddy [[dns]] validation
- **Location:** /etc/[[caddy]]/certs/ots.k7zvx.com.* (on CT 101)
- **Auto-renewal:** Configured via acme.sh
@ -92,7 +92,7 @@ ots.k7zvx.com {
### Port Forwarding
- **Router:** Ubiquiti firewall (192.168.1.28)
- **External ports:** 80/443 → 192.168.1.101 (Caddy CT)
- **External ports:** 80/443 → 192.168.1.101 ([[caddy]] CT)
- **Internal proxy:** Caddy → 192.168.1.109:443 (OpenTAKServer)
---

30
vault/docs/services/services.md
View file

@ -5,16 +5,16 @@ tags:
- media
aliases: []
related:
- [[ip-allocation]]
- [[caddy]]
- [[glossary]]
- [[meshtastic-headscale-runbook]]
- [[ip-allocation]]
- [[lxc-service-migration]]
updated: 2026-07-11
- [[meshtastic-headscale-runbook]]
- [[expose-service-edge2]]
updated: 2026-07-13
---
# Current Services Inventory
> **DNS split (2026-06-19):** `mail/autodiscover/autoconfig.echo6.co` → **edge1** (5.189.158.149, mail-only rebuilt Contabo VPS). `vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co` → **edge2** (184.174.35.153, permanent front door for all other services). Tailnet split-DNS is NOT used for echo6.co; echo6.co resolves via public GoDaddy DNS.
> **[[dns]] split ([[2026-06-19]]):** `mail/autodiscover/autoconfig.echo6.co` → **edge1** (5.189.158.149, mail-only rebuilt Contabo VPS). `vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co` → **edge2** (184.174.35.153, permanent front door for all other services). Tailnet split-DNS is NOT used for echo6.co; echo6.co resolves via public GoDaddy DNS.
## Active Services
@ -23,20 +23,20 @@ updated: 2026-07-11
| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 / :4404 | https://mesh.echo6.co | Meshtastic mesh monitoring (upstream ghcr.io/yeraze/meshmonitor:latest, multi-channel AutoAnnounce/AutoResponder) |
| Utility [[caddy]] | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
| Echo6 Search ([[searxng]]) | utility (CT 102) | 192.168.1.102:8080 | https://echo6.co | Branded search homepage (Docker, custom theme) |
| meshtasticd (AIDA-N2) | aida-nebra | 192.168.1.253:4403 | Internal | AIDA-N2(RPT,LLM) node !27780c47, Nebra 2W hat (ZebraHat), CLIENT_BASE role, fw 2.7.19. MeshAI (CT 108) connects via TCP localhost:4403 |
| meshtasticd (AIDA-N2) | aida-nebra | 192.168.1.253:4403 | Internal | AIDA-N2(RPT,LLM) node !27780c47, Nebra 2W hat (ZebraHat), CLIENT_BASE role, fw 2.7.19. [[meshai]] (CT 108) connects via TCP localhost:4403 |
| Meshtastic CLI | mt-isr | 192.168.1.141 | Internal | Station G2 WiFi bridge + TCP management |
| meshtasticd | mt-burleybutte | 192.168.1.185:4403 | Internal | Software Meshtastic node (Nebra 2W hat) |
| IdahoMesh Headscale | utility (CT 106) | 192.168.1.106:8080 | https://vpn.idahomesh.com | Meshtastic mesh VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| MeshAI | utility (CT 108) | 192.168.1.144:4403 / :8080 | Internal | LLM-powered Meshtastic assistant (Docker, work-meshai local build, gemini-3.1-flash-lite, Google grounding) |
| [[argus]] | utility (CT 103) | 192.168.1.103:8080 | Internal | Python app on :8080 — OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104:8000 / 100.64.0.12 | central.echo6.mesh (mesh) | Data-hub spine — ~25 adapters → NATS/JetStream → TimescaleDB; serves traffic tiles to navi — see [[central]] |
| NATS/JetStream (central) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | Central backend message bus (NATS :4222 client, :8222 monitoring) |
| [[central]] | utility (CT 104) | 192.168.1.104:8000 / 100.64.0.12 | central.echo6.mesh (mesh) | Data-hub spine — ~25 adapters → NATS/JetStream → TimescaleDB; serves traffic tiles to [[navi]] — see [[central]] |
| NATS/JetStream ([[central]]) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | Central backend message bus (NATS :4222 client, :8222 monitoring) |
| TimescaleDB/PostGIS (central) | utility (CT 104) | 192.168.1.104:5432 | Internal | Central backend time-series + geospatial database (PostgreSQL 16 + TimescaleDB + PostGIS) |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by edge2 host Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via edge2 DNAT) | https://forge.echo6.co | Git server — fronted by edge2 host Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on edge2 (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via [[authentik]]) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Mailcow | **edge1 CT 101** (10.10.10.2) | 5.189.158.149 | https://mail.echo6.co | Email server (privileged LXC on rebuilt Contabo VPS, updated commit 52a41b4d / SOGo 5.12.8) — **rebuilt in-place 2026-06-19** |
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by edge2 host Caddy (reverse_proxy 100.64.0.33:8086) |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ (Syncthing on Contabo decommissioned 2026-06-19 with edge1 rebuild) |
@ -53,7 +53,7 @@ updated: 2026-07-11
| Open WebUI | cortex (VM 150) | 192.168.1.150:8080 | https://ai.echo6.co | AI chat interface (Docker, Ollama backend, SSO) |
| Qdrant | cortex (VM 150) | 192.168.1.150:6333 | Internal | Vector database (Docker, [[recon]] knowledge store) |
| TEI | cortex (VM 150) | 192.168.1.150:8090 | Internal | Text embeddings (Docker, bge-m3 1024-dim) |
| RECON | data (VM 1130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
| [[recon]] | data (VM 1130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
| navi-config | data (VM 1130) | 192.168.1.130:8422 | Internal | RECON navi node config API |
| navi-contacts | data (VM 1130) | 192.168.1.130:8423 | Internal | RECON navi contact enrichment API |
| navi-landclass | data (VM 1130) | 192.168.1.130:8424 | Internal | RECON navi land classification API |
@ -75,7 +75,7 @@ updated: 2026-07-11
| LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by edge2 host Caddy; **migrated from Contabo 2026-06-16** |
| OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] |
| Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via edge2 Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for Synapse) — **migrated from Contabo 2026-06-18** |
| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via edge2 Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for [[synapse]]) — **migrated from Contabo 2026-06-18** |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | Internal | Signal/Matrix room archive bot (systemd) — see archivist.md for details |
| pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
| recon-sparse | cortex (VM 150) | 192.168.1.150:8091 | Internal | RECON sparse embedding service (systemd, bge-m3 model, port 8091) |
@ -132,7 +132,7 @@ updated: 2026-07-11
- Utility Caddy (reverse proxy for VPN-only services)
### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
- Echo6 Search — branded SearXNG homepage (port 8080, https://echo6.co)
- Echo6 Search — branded [[searxng]] homepage (port 8080, https://echo6.co)
- Custom cyberpunk theme: JetBrains Mono font, cyan/yellow palette, dark backgrounds
- Homepage: centered Echo6 logo + pill search bar (Google-style, viewport-locked no-scroll)
- Results page: full-width two-column grid (results + sidebar), stretched search header
@ -201,7 +201,7 @@ updated: 2026-07-11
- Sonarr TV automation (port 8989, internal)
- Radarr movie automation (port 7878, internal)
- Prowlarr indexer manager (port 9696, internal)
- SABnzbd Usenet downloader (port 8080, internal)
- SABnzbd [[usenet]] downloader (port 8080, internal)
- NFS storage from pi-nas (/mnt/arr)
- Config dirs: /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
@ -414,7 +414,7 @@ updated: 2026-07-11
- ~~WATCHTOWER~~ — **decommissioned 2026-06-16**
- ~~Matrix Synapse~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~Element Web~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~mautrix-signal bridge~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~[[mautrix_signal]] bridge~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16**
- ~~TAK Server~~ — **decommissioned 2026-06-16** (archived to forge.echo6.co/matt/archive-tak-server)
- ~~SIGIL~~ — **decommissioned 2026-06-16**

8
vault/docs/services/usenet.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[dns]]
- [[recon-operations]]
- [[proxmox-onboard-node]]
- [[caddy]]
- [[glossary]]
updated: 2026-06-18
- [[proxmox-onboard-node]]
- [[environment]]
- [[recon-service-integration]]
updated: 2026-07-13
---
# Usenet Configuration

6
vault/docs/software/authentik.md
View file

@ -8,9 +8,9 @@ related:
- [[authentik-oidc-application]]
- [[mailcow-create-mailbox]]
- [[caddy]]
- [[echo6-landing-page-data-export]]
- [[authentik-access-groups]]
updated: 2026-07-11
- [[edge2-access-reference]]
- [[headscale-oidc-boot-order]]
updated: 2026-07-13
---
# Authentik SSO Configuration

20
vault/docs/software/caddy.md
View file

@ -2,15 +2,15 @@
title: "Caddy & DNS Reference"
type: reference
tags:
- dns
- mesh
aliases: []
related:
- [[services]]
- [[ip-allocation]]
- [[headscale-onboard-node]]
- [[expose-service-home]]
- [[lxc-service-migration]]
updated: 2026-07-11
- [[expose-service-edge2]]
- [[authentik]]
updated: 2026-07-13
---
# Caddy & DNS Reference
@ -64,7 +64,7 @@ journalctl -u caddy -f
| Domain | Backend | Pattern | Service |
|--------|---------|---------|---------|
| mesh.echo6.co | 192.168.1.100:8080 | Local IP | meshmonitor (Authentik forward auth) |
| mesh.echo6.co | 192.168.1.100:8080 | Local IP | meshmonitor ([[authentik]] forward auth) |
| echo6.co | 100.64.0.15:8080 | Tailscale | Echo6 Search ([[searxng]]) + Matrix well-known |
| search.echo6.co | — | — | 301 redirect to echo6.co |
| nas.echo6.co | 100.64.0.21:80 | Tailscale | OpenMediaVault (pi-nas) |
@ -92,7 +92,7 @@ ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
## dnsmasq (Tailscale Split DNS) — HISTORICAL / OBSOLETE
> **Not in use.** Tailnet split-DNS for echo6.co was retired: echo6.co now resolves via public GoDaddy DNS, not internal dnsmasq (see [[services]]). This section documents the OLD setup that ran on the original Contabo VPS (100.64.0.1) before its 2026-06-19 rebuild into edge1 (mail-only). That host and its 100.64.0.1 tailnet identity are dead — do not repoint these records to edge1 or edge2; kept below for historical reference only.
> **Not in use.** Tailnet split-DNS for echo6.co was retired: echo6.co now resolves via public GoDaddy [[dns]], not internal dnsmasq (see [[services]]). This section documents the OLD setup that ran on the original Contabo VPS (100.64.0.1) before its [[2026-06-19]] rebuild into edge1 (mail-only). That host and its 100.64.0.1 tailnet identity are dead — do not repoint these records to edge1 or edge2; kept below for historical reference only.
**Config:** `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo (dead host, pre-2026-06-19)
**Listens on:** 100.64.0.1:53 (dead)
@ -114,11 +114,11 @@ ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
| requests.echo6.co | 100.64.0.8 | Jellyseer (via utility Caddy) |
| wt.echo6.co | 100.64.0.1 | WATCHTOWER ops dashboard |
| ai.echo6.co | 100.64.0.8 | Open WebUI (via utility Caddy) |
| matrix.echo6.co | 100.64.0.1 | Matrix Synapse (via Contabo Caddy) |
| matrix.echo6.co | 100.64.0.1 | Matrix [[synapse]] (via Contabo Caddy) |
| element.echo6.co | 100.64.0.1 | Element Web (via Contabo Caddy) |
| echo6.co | 100.64.0.8 | Echo6 Search homepage (via utility Caddy) |
| files.echo6.co | 100.64.0.8 | [[recon]] PDF library (via utility Caddy) |
| recon.echo6.co | 100.64.0.8 | RECON dashboard (via utility Caddy) |
| recon.echo6.co | 100.64.0.8 | [[recon]] dashboard (via utility Caddy) |
| lidarr.echo6.co | 100.64.0.8 | Lidarr music automation (via utility Caddy) |
| navidrome.echo6.co | 100.64.0.8 | Navidrome music server (via utility Caddy) |
@ -172,7 +172,7 @@ dig +short forge.echo6.co @100.64.0.1 # Test
| jellyfin | Jellyfin |
| mesh | MeshMonitor |
| nas | OpenMediaVault (pi-nas) |
| search | SearXNG (redirects to echo6.co) |
| search | [[searxng]] (redirects to echo6.co) |
| immich | Immich |
| nextcloud | Nextcloud |
| requests | Jellyseer |
@ -245,4 +245,4 @@ oidc:
---
*Last updated: 2026-07-11 — Flip off Contabo completed: "Contabo Caddy" section → "edge2 Caddy" (front door for auth/forge/vpn/vault/matrix/element/notes/proxmox, CTs verified against [[ip-allocation]]/[[services]]); Mailcow + autodiscover/autoconfig moved to edge1 (separate mail-only host, not on edge2); WATCHTOWER + TAK/SIGIL marked decommissioned (dead 100.64.0.1 backends removed); Headscale config location + Port Map updated to edge2; `ssh root@100.64.0.1` → `ssh edge2`. dnsmasq split-DNS section marked HISTORICAL/OBSOLETE (echo6.co split-DNS retired, ran on the dead pre-2026-06-19 Contabo host, not repointed to edge1/edge2 per [[services]]); GoDaddy DNS Records section corrected — edge2 services (auth/forge/vpn/vault/matrix/element/notes/proxmox) → 184.174.35.153, mail → edge1 5.189.158.149 (unchanged public IP), wt/tak marked as removed records. Prior: 2026-04-13 — Audit sync: added MAS routing on matrix.echo6.co, lidarr/navidrome/vpn.idahomesh.com to utility Caddy, proxmox/tak to GoDaddy, removed ghost docs.echo6.co entries, added dnsmasq lidarr/navidrome*
*Last updated: 2026-07-11 — Flip off Contabo completed: "Contabo Caddy" section → "edge2 Caddy" (front door for auth/forge/vpn/vault/matrix/element/notes/proxmox, CTs verified against [[ip-allocation]]/[[services]]); Mailcow + autodiscover/autoconfig moved to edge1 (separate mail-only host, not on edge2); WATCHTOWER + TAK/SIGIL marked decommissioned (dead 100.64.0.1 backends removed); Headscale config location + Port Map updated to edge2; `ssh root@100.64.0.1` → `ssh edge2`. dnsmasq split-DNS section marked HISTORICAL/OBSOLETE (echo6.co split-DNS retired, ran on the dead pre-2026-06-19 Contabo host, not repointed to edge1/edge2 per [[services]]); GoDaddy DNS Records section corrected — edge2 [[services]] (auth/forge/vpn/vault/matrix/element/notes/proxmox) → 184.174.35.153, mail → edge1 5.189.158.149 (unchanged public IP), wt/tak marked as removed records. Prior: 2026-04-13 — Audit sync: added MAS routing on matrix.echo6.co, lidarr/navidrome/vpn.idahomesh.com to utility Caddy, proxmox/tak to GoDaddy, removed ghost docs.echo6.co entries, added dnsmasq lidarr/navidrome*

17
vault/docs/software/central.md
View file

@ -1,15 +1,22 @@
---
title: central — Data-Hub Spine
type: reference
tags: [recon]
related: ["[[navi]]", "[[services]]", "[[environment]]"]
updated: 2026-06-27
tags:
- mesh
aliases: []
related:
- [[navi]]
- [[central-deploy-cutover]]
- [[services]]
- [[fleet-platform-baseline]]
- [[caddy]]
updated: 2026-07-13
---
# central — Data-Hub Spine
## Overview
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for navi traffic tiles and related situational-awareness feeds.
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for [[navi]] traffic tiles and related situational-awareness feeds.
- **URL (internal):** http://central.echo6.mesh:8000 (mesh-only, no public exposure)
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
@ -32,7 +39,7 @@ The data flow is: upstream APIs → adapters (central-supervisor) → NATS/JetSt
## Systemd Services
All three units are **enabled and active**; deployment survives reboot. Deps: `nats-server`, `postgresql@16-main`.
All three units are **enabled and active**; [[deployment]] survives reboot. Deps: `nats-server`, `postgresql@16-main`.
| Unit | Role |
|------|------|

10
vault/docs/software/dns.md
View file

@ -5,12 +5,12 @@ tags:
- dns
aliases: []
related:
- [[usenet]]
- [[caddy]]
- [[services]]
- [[expose-service-contabo]]
- [[headscale-onboard-node]]
- [[authentik-oidc-application]]
updated: 2026-07-11
- [[usenet]]
- [[expose-service-edge2]]
updated: 2026-07-13
---
# GoDaddy DNS Management
@ -29,7 +29,7 @@ Stored in `/home/zvx/projects/.ref/credentials` as:
| Purpose | IP |
|---------|-----|
| External (home [[services]]) | `199.6.36.163` |
| edge1 (mail/autodiscover/autoconfig only — mail-only host, rebuilt 2026-06-19) | `5.189.158.149` |
| edge1 (mail/autodiscover/autoconfig only — mail-only host, rebuilt [[2026-06-19]]) | `5.189.158.149` |
| edge2 (front door: auth/forge/vpn/vault/matrix/element/notes/proxmox) | `184.174.35.153` |
## Managed Domains

8
vault/docs/software/geo-tools.md
View file

@ -5,12 +5,12 @@ tags:
- storage
aliases: []
related:
- [[cc-rules]]
- [[navi]]
- [[ct-runbook]]
- [[cc-rules]]
- [[environment]]
- [[meshtasticd-sim-nodes-runbook]]
- [[recon-operations]]
updated: 2026-06-18
- [[toc-cortex-pve9.2-update]]
updated: 2026-07-13
---
# Geo Processing Tools — Cortex

17
vault/docs/software/navi.md
View file

@ -1,9 +1,16 @@
---
title: navi — Offline Navigation Platform
type: reference
tags: [recon]
related: ["[[recon]]", "[[services]]", "[[environment]]"]
updated: 2026-06-23
tags:
- recon
aliases: []
related:
- [[deployment]]
- [[central]]
- [[themes]]
- [[nominatim-v5-reimport]]
- [[cc-rules]]
updated: 2026-07-13
---
# navi — Offline Navigation Platform
@ -11,7 +18,7 @@ updated: 2026-06-23
navi is an offline-capable navigation web app served from **recon-vm** (VM 1130, 192.168.1.130). It provides geocoding, routing, land classification, fleet admin, and DEM-backed elevation data — all from self-hosted geo backends. Frontend is a Vite SPA; backend is a suite of 8 Python microservices behind nginx.
- **URL:** https://navi.echo6.co (fronted by utility Caddy + Authentik)
- **URL:** https://navi.echo6.co (fronted by utility [[caddy]] + [[authentik]])
- **Host:** recon-vm (data node, VM 1130)
- **Repos:** `github.com/zvx-echo6/navi` (canonical), Forge mirror `matt/navi`
- **Layout:** monorepo — `backend/` (Python) + `frontend/` (Vite)
@ -41,7 +48,7 @@ All 8 are gunicorn processes, bound to `127.0.0.1`, working directory `navi-mono
| # | Service | Port | Status | Purpose |
|---|---------|------|--------|---------|
| 1 | navi-traffic | :8421 | **DISABLED** | Traffic — now proxied externally to `central.echo6.mesh:8000` |
| 2 | navi-config | :8422 | Active | Deployment profile API |
| 2 | navi-config | :8422 | Active | [[deployment]] profile API |
| 3 | navi-contacts | :8423 | Active | Contacts + address book |
| 4 | navi-landclass | :8424 | Active | PAD-US land classification (Postgres `padus` DB) |
| 5 | navi-places | :8425 | Active | OSM place detail/enrichment (Postgres `overture` DB) |

10
vault/docs/software/recon.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[recon-operations]]
- [[recon-service-integration]]
- [[ia-download-queue]]
- [[services]]
- [[usenet]]
- [[caddy]]
updated: 2026-07-11
- [[navi]]
- [[ia-download-mirror]]
updated: 2026-07-13
---
# RECON — Knowledge Extraction Pipeline
@ -145,4 +145,4 @@ Key sections:
---
*Last updated: 2026-07-11 — Gemini model refs updated to gemini-3.1-flash-lite (retired gemini-2.5-flash-lite); backup destination corrected Contabo (100.64.0.1, dead) → edge1 (100.64.0.40). Prior: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current deployment) — verified against live 2026-06-18*
*Last updated: 2026-07-11 — Gemini model refs updated to gemini-3.1-flash-lite (retired gemini-2.5-flash-lite); backup destination corrected Contabo (100.64.0.1, dead) → edge1 (100.64.0.40). Prior: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current [[deployment]]) — verified against live 2026-06-18*

12
vault/docs/software/searxng.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[echo6-landing-page-data-export]]
- [[caddy]]
- [[environment]]
- [[ip-allocation]]
- [[headscale-onboard-node]]
- [[services]]
updated: 2026-07-11
- [[caddy]]
- [[ots-setup]]
updated: 2026-07-13
---
# SearXNG — Echo6 Search Homepage
@ -34,7 +34,7 @@ SearXNG is deployed as the branded Echo6 search homepage at `echo6.co`. The defa
| Search engine | SearXNG (Docker, v2026.2.6) | searxng container |
| Cache | Valkey (Redis-compatible) | valkey container |
| Reverse proxy | Utility [[caddy]] (CT 101) | 192.168.1.101 |
| SSL certs | acme.sh (Let's Encrypt) | /etc/caddy/certs/ on CT 101 |
| SSL certs | acme.sh (Let's Encrypt) | /etc/[[caddy]]/certs/ on CT 101 |
## Theme Customization
@ -132,7 +132,7 @@ curl -s http://192.168.1.102:8080 | head -30
- `echo6.co` → `100.64.0.15:8080` + Matrix `.well-known` handlers
- `search.echo6.co` → 301 redirect to `https://echo6.co`
**dnsmasq (historical — this ran on Contabo, decommissioned 2026-06-19):**
**dnsmasq (historical — this ran on Contabo, decommissioned [[2026-06-19]]):**
- `echo6.co` → `100.64.0.8` (utility Caddy)
- Per [[services]]: tailnet split-DNS is NOT used for `echo6.co` — it resolves via public GoDaddy DNS. This dnsmasq entry describes the pre-migration setup and should not be assumed current.

32
vault/glossary.md
View file

@ -5,12 +5,12 @@ tags:
- proxmox
aliases: []
related:
- [[services]]
- [[ip-allocation]]
- [[services]]
- [[caddy]]
- [[authentik]]
- [[usenet]]
updated: 2026-07-11
- [[edge2-access-reference]]
- [[proxmox-onboard-node]]
updated: 2026-07-13
---
# Glossary & Vocabulary
@ -38,14 +38,14 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
- **[[authentik]]** — aliases: 100.64.0.36
- **bluefin** — aliases: 100.64.0.30
- **cloud** (Cloud) — aliases: 192.168.1.242, 100.64.0.4
- **contabo** (edge1, mail-only — rebuilt in-place 2026-06-19, tailnet identity re-registered as `contabo`) — aliases: 100.64.0.40, 5.189.158.149
- **contabo** (edge1, mail-only — rebuilt in-place [[2026-06-19]], tailnet identity re-registered as `contabo`) — aliases: 100.64.0.40, 5.189.158.149
- **data** (Data) — aliases: 192.168.1.240, 100.64.0.6
- **edge2** — aliases: 184.174.35.153, 100.64.0.26
- **forgejo** — aliases: 100.64.0.34
- **iphone-eud** — aliases: 100.64.0.16
- **media** (Media) — aliases: 192.168.1.243, 100.64.0.3
- **mesh-bridge** — aliases: 100.100.0.3, 100.64.0.22
- **meshai** — aliases: 100.64.0.32
- **[[meshai]]** — aliases: 100.64.0.32
- **meshmonitor** — aliases: 100.64.0.7
- **nextcloud** — aliases: 100.64.0.11
- **peertube** — aliases: 100.64.0.23
@ -91,7 +91,7 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
- **echo6-agent** — aliases: echo6-agent — on: 2026-06-16
- **echo6-contabo-agent** (Echo6 Contabo Agent) — aliases: Echo6 Contabo Agent — on: contabo _(decommissioned/historical — Contabo-local agent; host rebuilt as edge1, mail-only, 2026-06-19)_
- **echo6-cortex-agent** (Echo6 Cortex Agent) — aliases: Echo6 Cortex Agent — on: cortex
- **echo6-search-searxng** (Echo6 Search (SearXNG)) — aliases: Echo6 Search (SearXNG) — on: utility
- **echo6-search-searxng** (Echo6 Search ([[searxng]])) — aliases: Echo6 Search (SearXNG) — on: utility
- **element-web** (Element Web) — aliases: Element Web, element — on: edge2
- **files** (Files) — aliases: Files, files — on: data
- **forge-forgejo** (Forge (Forgejo)) — aliases: Forge (Forgejo), forge — on: edge2
@ -123,8 +123,8 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
- **matrix-element** — aliases: matrix-element — on: edge2 _(live)_
- **matrix-mas** (Matrix MAS) — aliases: Matrix MAS — on: edge2
- **matrix-postgres** — aliases: matrix-postgres — on: contabo _(live)_
- **matrix-synapse** (Matrix [[synapse]]) — aliases: Matrix Synapse, matrix — on: edge2
- **[[mautrix_signal]]** — aliases: mautrix-signal — on: edge2
- **matrix-synapse** (Matrix [[synapse]]) — aliases: Matrix [[synapse]], matrix — on: edge2
- **[[mautrix_signal]]** — aliases: [[mautrix_signal]] — on: edge2
- **meshtastic-cli** (Meshtastic CLI) — aliases: Meshtastic CLI — on: mt-isr
- **meshtasticd** — aliases: meshtasticd — on: mt-burleybutte
- **meshtasticd-aida-n2** (meshtasticd (AIDA-N2)) — aliases: meshtasticd (AIDA-N2) — on: aida-nebra
@ -149,15 +149,15 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
- **tak-server** (TAK Server) — aliases: TAK Server — on: 2026-06-16
- **tei** (TEI) — aliases: TEI — on: cortex
- **termix** (Termix) — aliases: Termix — on: contabo _(decommissioned — wiped with edge1 rebuild 2026-06-19, not migrated to edge2)_
- **utility-caddy** (Utility Caddy) — aliases: Utility Caddy — on: utility
- **utility-caddy** (Utility [[caddy]]) — aliases: Utility Caddy — on: utility
- **watchtower** (WATCHTOWER) — aliases: WATCHTOWER — on: 2026-06-16
### Projects
- **[[advbbs-project]]** — aliases: advbbs-project
- **argus** — aliases: argus
- **[[deploy-livesync]]** — aliases: deploy-livesync
- **[[matrix-synapse-deployment]]** — aliases: matrix-synapse-deployment
- **[[meshtastic-headscale-runbook]]** — aliases: meshtastic-headscale-runbook
- **[[mmud-project]]** — aliases: mmud-project
- **[[advbbs-project]]** — aliases: [[advbbs-project]]
- **[[argus]]** — aliases: argus
- **[[deploy-livesync]]** — aliases: [[deploy-livesync]]
- **[[matrix-synapse-deployment]]** — aliases: [[matrix-synapse-deployment]]
- **[[meshtastic-headscale-runbook]]** — aliases: [[meshtastic-headscale-runbook]]
- **[[mmud-project]]** — aliases: [[mmud-project]]

16
vault/notes/echo6-landing-page-data-export.md
View file

@ -8,9 +8,9 @@ related:
- [[searxng]]
- [[authentik]]
- [[ip-allocation]]
- [[environment]]
- [[caddy]]
- [[CLAUDE-baseline]]
updated: 2026-07-11
updated: 2026-07-13
---
# Echo6 Landing Page — Data Export
## Echo6 Platform Reference — Infrastructure, Services & Brand Identity
@ -52,7 +52,7 @@ updated: 2026-07-11
|---------|-----|-------------|------|
| Echo6 Search (Homepage) | https://echo6.co | [[searxng]] search — branded cyberpunk homepage, Google-style layout | Public ([[searxng]]) |
| Aurora (AI Assistant) | https://ai.echo6.co | RAG-augmented LLM chat — locally-hosted, queries a 95K+ vector knowledge base | [[authentik]] OIDC |
| PeerTube (Video) | https://stream.echo6.co | Self-hosted video platform — 99 curated YouTube channels mirrored, GPU-transcoded | Authentik OIDC |
| PeerTube (Video) | https://stream.echo6.co | Self-hosted video platform — 99 curated YouTube channels mirrored, GPU-transcoded | [[authentik]] OIDC |
| File Server | https://files.echo6.co | PDF/document library — ~13,239 documents (military doctrine, survival, comms, trades) | Public |
| Photos (Immich) | https://immich.echo6.co | Self-hosted photo management | Authentik OIDC |
| Mail (Mailcow) | https://mail.echo6.co | Email — Mailcow webmail | Authentik OIDC |
@ -67,7 +67,7 @@ updated: 2026-07-11
## 3. Echo6 Homepage — SearXNG Custom Theme
### Overview
The Echo6 homepage at `echo6.co` is a customized SearXNG instance (not a standalone static page). The default SearXNG UI is reskinned via CSS overlay, template overrides, and settings.yml changes to match the Echo6 cyberpunk brand.
The Echo6 homepage at `echo6.co` is a customized [[searxng]] instance (not a standalone static page). The default SearXNG UI is reskinned via CSS overlay, template overrides, and settings.yml changes to match the Echo6 cyberpunk brand.
### Implementation Approach
- **CSS overlay** (`echo6-custom.css`) — all color, font, layout, and component overrides
@ -408,7 +408,7 @@ Downloader (CT 110, yt-dlp + VPN rotation)
- VPN rotation on rate limit (NordVPN, 6 countries: US, CA, UK, DE, NL, SE)
- Pre-encode probe gate skips already-efficient codecs (H.265/AV1/VP9) and low-bitrate H.264
- Automatic dedup via download archive
- All three stages run as systemd services
- All three stages run as systemd [[services]]
---
@ -418,7 +418,7 @@ Downloader (CT 110, yt-dlp + VPN rotation)
|-------|-----------|
| Mesh VPN | Tailscale (self-hosted Headscale) |
| Reverse proxy | [[caddy]] (CT 101 on utility) — auto TLS |
| [[dns]] | GoDaddy (external), dnsmasq split DNS (internal) |
| [[dns]] | GoDaddy (external), dnsmasq split [[dns]] (internal) |
| Authentication | Authentik OIDC SSO across all services |
| SSO Launch URLs | `https://auth.echo6.co/application/launch/<slug>/` for seamless pass-through |
| Backup transport | rsync over SSH (ed25519 keys) |
@ -445,11 +445,11 @@ files.echo6.co → Document/PDF download server
| Layer | Technologies |
|-------|-------------|
| Virtualization | Proxmox (5 nodes) |
| Networking | Tailscale/Headscale, Caddy, nginx, dnsmasq |
| Networking | Tailscale/Headscale, [[caddy]], nginx, dnsmasq |
| GPU compute | NVIDIA RTX A4000 (CUDA, NVENC, Tensor) |
| AI/ML | gemini-3.1-flash-lite, Ollama, TEI (bge-m3), JOSIEFIED Qwen3 8B |
| Vector DB | Qdrant (HNSW index, cosine similarity) |
| Databases | SQLite (RECON), PostgreSQL (PeerTube) |
| Databases | SQLite ([[recon]]), PostgreSQL (PeerTube) |
| Video | PeerTube v8, yt-dlp, ffmpeg/NVENC, Whisper |
| Search | SearXNG (custom Echo6 theme) |
| Auth | Authentik (OIDC, custom Echo6 theme) |

4
vault/notes/ia-download-queue.md
View file

@ -9,8 +9,8 @@ related:
- [[ia-cli-reference]]
- [[recon]]
- [[usenet]]
- [[glossary]]
updated: 2026-06-18
- [[recon-operations]]
updated: 2026-07-13
---
# Internet Archive Download Queue

6
vault/projects/advbbs-project.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[meshtastic-sidecar-node]]
- [[meshtasticd-sim-nodes-runbook]]
- [[mautrix_signal]]
- [[mmud-project]]
- [[meshtastic-headscale-runbook]]
- [[services]]
updated: 2026-06-18
- [[mautrix_signal]]
updated: 2026-07-13
---
# advBBS — Claude Code Project Context

8
vault/projects/argus.md
View file

@ -9,8 +9,8 @@ related:
- [[ip-allocation]]
- [[caddy]]
- [[ct-runbook]]
- [[ots-setup]]
updated: 2026-06-18
- [[services]]
updated: 2026-07-13
---
# ARGUS - OSINT Intelligence Platform
@ -71,7 +71,7 @@ ARGUS (Automated Reconnaissance & Gathering for Unified Situational-awareness) i
**Registration:** `tailscale up --login-server=https://vpn.echo6.co --authkey=<key> --ssh --accept-routes`
**[[dns]] Bootstrap Fix:**
Systemd drop-in at `/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf` ensures fallback [[dns]] (1.1.1.1, 8.8.8.8) exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot.
Systemd drop-in at `/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf` ensures fallback [[dns]] (1.1.1.1, 8.8.8.8) exists before tailscaled starts, preventing chicken-and-egg [[dns]] resolution failures on reboot.
```bash
[Service]
@ -321,4 +321,4 @@ pct status 103 --verbose
---
**Provisioned by:** Claude Code
**Container ready for:** ARGUS application deployment
**Container ready for:** ARGUS application [[deployment]]

10
vault/projects/deploy-livesync.md
View file

@ -8,13 +8,13 @@ related:
- [[authentik-oidc-application]]
- [[authentik]]
- [[authentik-access-groups]]
- [[expose-service-home]]
- [[authentik-upgrade]]
updated: 2026-06-18
- [[headscale-oidc-boot-order]]
updated: 2026-07-13
---
# Deploying CouchDB with JWT auth for Obsidian LiveSync via Authentik
**LiveSync has native client-side JWT support that eliminates the need for a browser-based OIDC flow.** The plugin generates and signs JWTs internally using a stored private key, sending `Authorization: Bearer` headers directly to CouchDB. This fundamentally changes the architecture: instead of proxying OIDC tokens, you provision per-user key pairs, configure CouchDB with the public keys, and distribute setup URIs containing the private keys. [[authentik]] serves as the identity backbone for a provisioning service — not as a runtime token issuer. No one has publicly documented a complete LiveSync + SSO [[deployment]], making this guide a synthesis of the Kishieel Keycloak series, CouchDB JWT internals, Authentik's claim customization, and the LiveSync plugin's JWT implementation.
**LiveSync has native client-side JWT support that eliminates the need for a browser-based OIDC flow.** The plugin generates and signs JWTs internally using a stored private key, sending `Authorization: Bearer` headers directly to CouchDB. This fundamentally changes the architecture: instead of proxying OIDC tokens, you provision per-user key pairs, configure CouchDB with the public keys, and distribute setup URIs containing the private keys. [[authentik]] serves as the identity backbone for a provisioning service — not as a runtime token issuer. No one has publicly documented a complete LiveSync + SSO [[deployment]], making this guide a synthesis of the Kishieel Keycloak series, CouchDB JWT internals, [[authentik]]'s claim customization, and the LiveSync plugin's JWT implementation.
---
@ -107,7 +107,7 @@ This iterates all user groups, extracts the `couchdb_role` attribute where it ex
## LiveSync's native JWT: how the plugin signs its own tokens
The Obsidian LiveSync plugin has **built-in JWT generation** that changes the deployment model fundamentally. Instead of obtaining tokens from an IdP at runtime, the plugin stores a private key and signs short-lived JWTs client-side. The relevant plugin settings are:
The Obsidian LiveSync plugin has **built-in JWT generation** that changes the [[deployment]] model fundamentally. Instead of obtaining tokens from an IdP at runtime, the plugin stores a private key and signs short-lived JWTs client-side. The relevant plugin settings are:
| Setting | Type | Default | Purpose |
|---------|------|---------|---------|
@ -236,7 +236,7 @@ notes.echo6.co {
Given the constraints — LiveSync can't do OIDC flows, but it can sign JWTs client-side — the architecture has three components:
**1. CouchDB container** at `notes.echo6.co` behind Caddy, configured with JWT auth handler, CORS, and per-user databases with `_security` documents.
**1. CouchDB container** at `notes.echo6.co` behind [[caddy]], configured with JWT auth handler, CORS, and per-user databases with `_security` documents.
**2. A provisioning service** (a small web app hosted on `forge.echo6.co` or as a Docker container) that:
- Is protected by Authentik forward auth (browser-based OIDC login)

39
vault/projects/fleet-patch-audit.md
View file

@ -3,9 +3,14 @@ title: Fleet Patch Audit — 2026-06-19
type: project
tags:
- proxmox
- ai
related: []
updated: 2026-06-22
aliases: []
related:
- [[fleet-platform-baseline]]
- [[lxc-service-migration]]
- [[caddy]]
- [[services]]
- [[ip-allocation]]
updated: 2026-07-13
status: complete
---
@ -13,7 +18,7 @@ status: complete
Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this is a planning document to build the patch plan from.**
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No services route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149).
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No [[services]] route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149).
---
@ -22,7 +27,7 @@ Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this i
**Phases 1–3 are fully done. The entire fleet is on the current platform.**
- **Phase 1** (guest/VM security apt) — COMPLETE 2026-06-20. 26 guests patched, ~600+ security packages cleared, zero data loss.
- **Phase 2** (app/container updates) — COMPLETE 2026-06-21. All app upgrades done: Authentik 2025.12.4→2026.5.3 (sequential), Forgejo 14→15, Headscale 0.28→0.29.1 (both instances), Immich 2.5.6→2.7.5, Nextcloud AIO→NC 33.0.5, media stack (Jellyfin/SABnzbd/arr), cortex AI stack (Ollama/TEI/Qdrant/Open-WebUI), and the low-urgency batch.
- **Phase 2** (app/container updates) — COMPLETE 2026-06-21. All app upgrades done: [[authentik]] 2025.12.4→2026.5.3 (sequential), Forgejo 14→15, Headscale 0.28→0.29.1 (both instances), Immich 2.5.6→2.7.5, Nextcloud AIO→NC 33.0.5, media stack (Jellyfin/SABnzbd/arr), cortex AI stack (Ollama/TEI/Qdrant/Open-WebUI), and the low-urgency batch.
- **Phase 3** (platform/reboot windows) — COMPLETE 2026-06-22. All 5 PVE nodes on 9.2.3/kernel 7.0.12-1-pve (including toc+cortex); pi-nas on OMV 8.4/kernel 6.18; cortex NVIDIA driver 580.167.08 + DKMS + nvidia-container-toolkit 1.19.1; GPU passthrough (vfio) survived the 7.0 kernel; cluster 5/5 quorate.
**Intentionally deferred / out of scope (not failures):**
@ -42,12 +47,12 @@ These containers have the highest raw security-update counts and have not been p
| Host | Guest | Upgradable / Security | Notes |
|------|-------|-----------------------|-------|
| utility | CT119 mesh-territory | 179 / 91 sec | Never patched |
| utility | CT108 meshai | 109 / 79 | — |
| utility | CT108 [[meshai]] | 109 / 79 | — |
| cloud | CT120 immich guest-OS | 191 / 101 | — |
| cloud | CT121 nextcloud guest-OS | 98 / 75 | — |
| media | CT110 peertube | 81 / 37 | — |
| utility | CT109 opentakserver | 34 / 31 | — |
| utility | CT104 central | 50 / 38 | Includes PostgreSQL 16.13 → 16.14 |
| utility | CT104 [[central]] | 50 / 38 | Includes PostgreSQL 16.13 → 16.14 |
### Tier 2 — App / Container Updates
@ -58,7 +63,7 @@ Updates where the application or its Docker images have drifted from current ups
| edge2 | CT105 | authentik 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
| edge2 | CT107 | headscale 0.28.0 → 0.29.1 | Also a 2nd headscale on utility CT106 |
| edge2 | CT103 | forgejo 14.0.5 → 15.0.3 | **14.x EOL 2026-04-30** — migrate branch, not just patch |
| edge2 | CT106 | Synapse 1.155.0 / Element / MAS | Image drift + pending OS apt security updates |
| edge2 | CT106 | [[synapse]] 1.155.0 / Element / MAS | Image drift + pending OS apt security updates |
| edge2 | CT104 | livesync couchdb:3.4 | Docker image drift |
| edge2 | CT108 | ~~mailcow (18 containers)~~ | ✅ **Decommissioned 2026-06-20** — superseded by edge1; no longer an update target |
| cloud | CT120 | immich — server/ml/valkey:9/postgres(14-vectorchord) | 4 images drifted |
@ -98,10 +103,10 @@ Issues noted that are not package/image updates but warrant attention.
| Host / Guest | Flag | Detail |
|---|---|---|
| data | Disk 92% full | ~73 GB / 938 GB free; address before patching |
| utility CT118 archivist | rpcbind on 0.0.0.0:111 | No Tailscale client or firewall on this CT; exposed port |
| utility CT118 [[archivist]] | rpcbind on 0.0.0.0:111 | No Tailscale client or firewall on this CT; exposed port |
| media VM105 jellyseerr | Non-stable image | Running preview-OIDC tag, not a stable release |
| data VM1130 nominatim | Stale image (14 months) | nominatim:4.5, pinned; confirm intentional |
| edge2 CT106 matrix / CT107 headscale | No Tailscale client | Ingress via Caddy; verify internal routing before patching |
| edge2 CT106 matrix / CT107 headscale | No Tailscale client | Ingress via [[caddy]]; verify internal routing before patching |
---
@ -138,15 +143,15 @@ Running application version vs latest stable upstream, per app — the "is every
### Current / already past the fix (no action)
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/themes (edge2 CT101) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/[[themes]] (edge2 CT101) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
### Lower urgency
Mumble 1.5.517→1.5.901 · Caddy 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · searxng (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · mautrix-signal v0.2603.0.
Mumble 1.5.517→1.5.901 · Caddy 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · [[searxng]] (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · [[mautrix_signal]] v0.2603.0.
### Internal echo6 apps (no upstream to track)
central-*, meshai, archivist, meshwars, recon / recon-watchdog, navi-* — running; version = current git head.
central-*, meshai, archivist, meshwars, [[recon]] / recon-watchdog, navi-* — running; version = current git head.
---
@ -240,7 +245,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
### Incidental fixes made during Phase 1
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN DNS protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and DNS remained healthy after boot.
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN [[dns]] protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and DNS remained healthy after boot.
- **(b) CT111 mcc — DNS hijacked to unreachable MagicDNS.** Tailscale `accept-dns` was redirecting DNS to a MagicDNS address that was not reachable from this CT. Disabled `tailscale accept-dns`, set `1.1.1.1` / `8.8.8.8` persistently.
- **(c) PostgreSQL on central CT104 moved 16.13→16.14** as part of the security-pocket apt pass.
- **(d) Fleet-wide stale `/etc/hosts` fix** — see Critical Finding 2.
@ -268,7 +273,7 @@ Known minor leftover (cosmetic, non-blocking): CT107's OWN tailscale client node
### 🔴 Critical Finding 2 — fleet-wide stale /etc/hosts broke coordinator connectivity
7 fleet nodes — data, cloud, media, utility hosts, plus caddy CT101, cobalt CT112, and peertube CT110 — had a stale `5.189.158.149 vpn.echo6.co` line in `/etc/hosts` left over from before the 2026-06-19 headscale migration to edge2. This pinned `vpn.echo6.co` to edge1 (now mail-only), so tailscaled hit Mailcow's TLS cert and could never reach the real coordinator — affected nodes showed OFFLINE in headscale while coasting on persistent WireGuard tunnels (still SSH-reachable, masking the problem).
7 fleet nodes — data, cloud, media, utility hosts, plus caddy CT101, cobalt CT112, and peertube CT110 — had a stale `5.189.158.149 vpn.echo6.co` line in `/etc/hosts` left over from before the [[2026-06-19]] headscale migration to edge2. This pinned `vpn.echo6.co` to edge1 (now mail-only), so tailscaled hit Mailcow's TLS cert and could never reach the real coordinator — affected nodes showed OFFLINE in headscale while coasting on persistent WireGuard tunnels (still SSH-reachable, masking the problem).
**Fixed 2026-06-20:** removed the stale line and ran `tailscale up` on all 7; all confirmed ONLINE in the coordinator. `/etc/hosts.bak-20260620` backups left on each host.
@ -331,7 +336,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
### Incidental fixes and side-work during Phase 2
- **CT107 boot-survival fix** (applied earlier in the effort, during Phase 1 resolution) — rebound headscale/headplane ports to 10.10.10.25, dropped `tailscale-online.target` dependency, disabled `only_start_if_oidc_is_available` gate, repointed edge2 Caddy; proven by reboot self-heal in ~45 s. Also corrected CT107's own tailscale node ControlURL to `vpn.echo6.co` so it self-registers cleanly.
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 argus running hot (transcription + docker-compose build churn). Matt migrated argus to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 [[argus]] running hot (transcription + docker-compose build churn). Matt migrated argus to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
- **Nextcloud:** granted `matt@echo6.co` the NC admin role. (user_oidc has no group-claim sync, so this is durable across SSO logins.)
- **Radarr:** set up a `\\192.168.1.160\manual` SMB drop folder on the same NFS export as the library (atomic-move imports) for manual movie filing.
- **Snapshot hygiene:** all rollback snapshots cleaned up after validation — Phase 1 `presec-*`, Phase 2 `prewave2-*`, OTS `pre-ots-*` snapshots all removed.
@ -362,7 +367,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
### Separate deferred projects
- Nominatim v5 re-import — see [[nominatim-v5-reimport]]
- [[Nominatim v5 Re-import]] — see [[nominatim-v5-reimport]]
---

18
vault/projects/fleet-platform-baseline.md
View file

@ -1,18 +1,22 @@
---
title: Fleet Platform Baseline — post-patch 2026-06-22
type: reference
type: project
tags:
- proxmox
- ai
aliases: []
related:
- projects/fleet-patch-audit
updated: 2026-06-22
- [[fleet-patch-audit]]
- [[environment]]
- [[toc-cortex-pve9.2-update]]
- [[central]]
- [[ip-allocation]]
updated: 2026-07-13
status: current
---
# Fleet Platform Baseline — post-patch 2026-06-22
Point-in-time platform state after the 2026-06-19/22 patch campaign. Full campaign record and accepted caveats in [[fleet-patch-audit]].
Point-in-time platform state after the [[2026-06-19]]/22 patch campaign. Full campaign record and accepted caveats in [[fleet-patch-audit]].
---
@ -45,7 +49,7 @@ Ubuntu 24.04 (security-patched). PostgreSQL 16, Valhalla, Nominatim 4.5 (v5 defe
| App | Host | Version | Notes |
|---|---|---|---|
| Authentik | edge2 CT105 | 2026.5.3 | Fleet SSO |
| [[authentik]] | edge2 CT105 | 2026.5.3 | Fleet SSO |
| Forgejo | edge2 CT103 | 15.0.3 | — |
| Headscale | edge2 CT107 | 0.29.1 | Fleet tailnet coordinator at `vpn.echo6.co`; boot-survival fixed (ports bound to 10.10.10.25, not tailscale IP) |
| Headscale (IdahoMesh) | utility CT106 | 0.29.1 | Separate IdahoMesh mesh at `vpn.idahomesh.com` |
@ -53,7 +57,7 @@ Ubuntu 24.04 (security-patched). PostgreSQL 16, Valhalla, Nominatim 4.5 (v5 defe
| Immich | cloud CT120 | 2.7.5 | Photos on pi-nas NFS |
| PeerTube | media CT110 | 8.2.1 | — |
| OpenTAKServer | utility CT109 | 1.7.12 | RabbitMQ stays 3.12 by decision; MediaMTX 1.19.1; Mumble 1.5.517 |
| Matrix/Synapse | edge2 CT106 | 1.155.0 | — |
| Matrix/[[synapse]] | edge2 CT106 | 1.155.0 | — |
| Vaultwarden | edge2 CT102 | 1.36.0 | — |
| PDM | edge2 CT100 | 1.1.4 | — |
| CouchDB/LiveSync | edge2 CT104 | 3.5.2 | — |

14
vault/projects/matrix-synapse-deployment.md
View file

@ -7,15 +7,15 @@ aliases: []
related:
- [[synapse]]
- [[matrix_host]]
- [[mautrix_signal]]
- [[caddy]]
- [[mautrix_signal]]
- [[lxc-service-migration]]
updated: 2026-07-11
updated: 2026-07-13
---
# Matrix Synapse Deployment
**Status:** Deployed 2026-02-15, migrated to Contabo 2026-02-15. Migrated to edge2 CT 106 2026-06-19.
**Target (historical, at time of deployment):** Contabo VPS (5.189.158.149 / 100.64.0.1)
**Target (historical, at time of [[deployment]]):** Contabo VPS (5.189.158.149 / 100.64.0.1)
**URLs:** https://matrix.echo6.co ([[synapse]]), https://element.echo6.co (Element Web)
**Server Name:** echo6.co (federated identity: @user:echo6.co)
@ -27,9 +27,9 @@ updated: 2026-07-11
|-----------|--------|
| Host (historical, at deployment time) | Contabo VPS (5.189.158.149 / 100.64.0.1) |
| Host (current) | edge2 CT 106 (100.64.0.37) — migrated 2026-06-18 |
| Docker [[services]] | Synapse (127.0.0.1:8008), Element Web (127.0.0.1:8088), PostgreSQL 16 |
| Docker [[services]] | [[synapse]] (127.0.0.1:8008), Element Web (127.0.0.1:8088), PostgreSQL 16 |
| Reverse proxy (historical) | Contabo [[caddy]] (auto ACME certs) |
| Reverse proxy (current) | edge2 host Caddy |
| Reverse proxy (current) | edge2 host [[caddy]] |
| SSO | [[authentik]] OIDC → communication-users group |
| Federation | Well-known delegation on echo6.co base domain (served by utility Caddy) |
| Compose path | `/opt/matrix/docker-compose.yml` |
@ -264,7 +264,7 @@ matrix.echo6.co {
### element.echo6.co
- Backend: `192.168.1.108:8080` (local IP)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS
- Issue cert, install cert, add Caddy site block, add GoDaddy [[dns]]
```caddyfile
element.echo6.co {
@ -413,7 +413,7 @@ Must return `true`.
1. Open https://element.echo6.co
2. Click SSO login
3. Should redirect to auth.echo6.co → authenticate → redirect back to Element
4. Verify user identity matches Authentik profile
4. Verify user identity matches [[authentik]] profile
---

8
vault/projects/meshai-config-hot-apply.md
View file

@ -3,10 +3,14 @@ title: meshai Config Hot-Apply — Kill the Restart-Required GUI Friction
type: project
tags:
- mesh
aliases: []
related:
- [[meshai]]
- [[meshai-region-routing-plan]]
updated: 2026-07-07
- [[meshai-prod-compose-override]]
- [[meshai]]
- [[SESSION-HANDOFF-meshai-test]]
- [[meshcore-transport]]
updated: 2026-07-13
status: proposed
---

11
vault/projects/meshai-region-routing-plan.md
View file

@ -3,11 +3,14 @@ title: meshai Region × Family Routing — Implementation Plan
type: project
tags:
- mesh
aliases: []
related:
- [[meshai]]
- [[meshai-config-hot-apply]]
- [[meshai-prod-compose-override]]
- [[meshcore-transport]]
- [[meshai-fire-alerting-and-persistence]]
updated: 2026-07-07
- [[meshai]]
- [[SESSION-HANDOFF-meshai-test]]
updated: 2026-07-13
status: proposed
---
@ -15,7 +18,7 @@ status: proposed
> Produced by a local multi-agent ultraplan (3 planners → 3 judges → synthesize → 3 red-team critics → finalize), verified against prod code at commit ceb95fb.
>
> **Scope principle (Matt, 2026-07-07): build the PLUMBING, not automation.** meshai exposes the mechanism — region-tagged events + a compact `region_routes` matrix the dispatcher honors + a plain editor to set it. **Matt does all configuration by hand:** defines the region boxes, provisions the radio channels, fills each family×region→channel cell himself. meshai auto-creates nothing — no channels, no regions, no routes, no generated rule objects. "No sprawl" comes from the config being ONE compact object he edits, not from automation.
> **Scope principle (Matt, 2026-07-07): build the PLUMBING, not automation.** [[meshai]] exposes the mechanism — region-tagged events + a compact `region_routes` matrix the dispatcher honors + a plain editor to set it. **Matt does all configuration by hand:** defines the region boxes, provisions the radio channels, fills each family×region→channel cell himself. meshai auto-creates nothing — no channels, no regions, no routes, no generated rule objects. "No sprawl" comes from the config being ONE compact object he edits, not from automation.
## 1. Verdict
**GO, phased.** Achievable, mostly config/GUI once the keystone code gap is closed: **nothing currently writes `event.region` / `event.regions`** (`notifications/events.py:55-56` define them, the dispatcher *reads* them, zero writers exist). Close that and the existing region-scope machinery comes alive.

13
vault/projects/meshai.md
View file

@ -3,15 +3,16 @@ title: meshai
type: project
tags:
- mesh
- ai
aliases:
- meshai
- MeshAI
related:
- [[services]]
- [[meshcore-transport]]
- [[central]]
updated: 2026-07-11
- [[meshai-region-routing-plan]]
- [[meshai-prod-compose-override]]
- [[SESSION-HANDOFF-meshai-test]]
- [[meshai-config-hot-apply]]
updated: 2026-07-13
---
# meshai
@ -35,7 +36,7 @@ The LLM backend is gemini-3.1-flash-lite with Google Search grounding (multi-bac
- **Liveness:** container healthcheck is a PID-file liveness probe (`/tmp/meshai.pid`), not an HTTP endpoint.
- **Mesh link:** connects over **IP to MeshMonitor's virtual-node (vnode) service**, which fronts the actual Meshtastic radio (a meshtasticd node — ultimately AIDA-N2 / channel 8 on **aida-nebra**). Everything is over the network; there is no radio physically attached to the meshai host. meshai *can* also connect directly to a meshtasticd over TCP, but the deployed configuration routes over IP/TCP to MeshMonitor's virtual node.
- **Source:** GitHub `origin` = `zvx-echo6/meshai`. Deploy = git pull + `docker compose build && up -d` (survives reboot).
- **Distinct from** the [[central]] service (utility CT 104) — meshai is a *consumer* of Central's feed, not Central itself.
- **Distinct from** the [[central]] service (utility CT 104) — meshai is a *consumer* of [[central]]'s feed, not Central itself.
## Architecture
@ -58,6 +59,6 @@ The pipeline is feeds → events → notifications → mesh, plus a separate inb
## Active / planned work
- [[meshcore-transport]] — proposed dual Meshtastic + MeshCore transport (send/receive on both meshes simultaneously; uniform message sizing to the smaller radio budget). Design agreed; open on hardware, the `meshcore` dependency, and branch timing.
- [[meshcore-transport]] — proposed dual Meshtastic + [[meshcore-transport]] (send/receive on both meshes simultaneously; uniform message sizing to the smaller radio budget). Design agreed; open on hardware, the `meshcore` dependency, and branch timing.
- NWS severity normalization — CAP-severity pre-filter removed; NWS breadth is now governed solely by the dashboard Weather toggle threshold (warnings broadcast immediately).
- Fire-path correctness — WFIGS/FIRMS fire correlation and drain/pacer spam controls on the fire feed.

13
vault/projects/meshcore-transport.md
View file

@ -9,14 +9,17 @@ aliases:
- MeshCore transport for meshai
related:
- [[meshai]]
- [[services]]
updated: 2026-07-02
- [[meshai-region-routing-plan]]
- [[meshai-prod-compose-override]]
- [[SESSION-HANDOFF-meshai-test]]
- [[meshai-config-hot-apply]]
updated: 2026-07-13
status: proposed
---
# MeshCore transport for meshai (dual Meshtastic + MeshCore)
Design for adding MeshCore as a second mesh transport to the meshai LLM mesh assistant ([[meshai]]), running alongside Meshtastic.
Design for adding MeshCore as a second mesh transport to the [[meshai]] LLM mesh assistant ([[meshai]]), running alongside Meshtastic.
## Status
@ -41,7 +44,7 @@ Let meshai speak both **Meshtastic** and **MeshCore**, config-selectable as eith
## Companion vs client (how we connect to MeshCore)
MeshCore firmware is role-specific (flashed, not runtime): **Companion**, **Repeater**, **Room Server**. The **Companion** node is the one a computer attaches to and drives — analogous to Meshtastic's phone+node model. meshai attaches to a Companion-firmware radio via the official `meshcore` Python lib (asyncio). **TCP to a Companion node is the natural fit** here — either native-TCP MeshCore firmware or a `ser2net`/serial-to-IP bridge — matching meshai's all-over-IP deployment (the Meshtastic side already runs over IP to MeshMonitor's vnode). A USB-on-host radio is not how this deployment works. Avoid BLE on Linux. Broadcast to a channel with `send_chan_msg(index, text)`; channel 0 = "Public" (well-known PSK) = the broadcast primitive. Do NOT attach to a Repeater or Room Server for messaging.
MeshCore firmware is role-specific (flashed, not runtime): **Companion**, **Repeater**, **Room Server**. The **Companion** node is the one a computer attaches to and drives — analogous to Meshtastic's phone+node model. meshai attaches to a Companion-firmware radio via the official `meshcore` Python lib (asyncio). **TCP to a Companion node is the natural fit** here — either native-TCP MeshCore firmware or a `ser2net`/serial-to-IP bridge — matching meshai's all-over-IP [[deployment]] (the Meshtastic side already runs over IP to MeshMonitor's vnode). A USB-on-host radio is not how this deployment works. Avoid BLE on Linux. Broadcast to a channel with `send_chan_msg(index, text)`; channel 0 = "Public" (well-known PSK) = the broadcast primitive. Do NOT attach to a Repeater or Room Server for messaging.
## Connecting to MeshCore via pyMC (companion TCP frame server)
@ -54,7 +57,7 @@ MeshCore firmware is role-specific (flashed, not runtime): **Companion**, **Repe
- **Meshtastic (today):** rich, global, passive picture via MeshView + MeshMonitor (fed by Meshtastic's MQTT firehose).
- **MeshCore:** no MQTT firehose and no passive "every node ever heard" nodeDB — by design (privacy/routing model). Awareness is LOCAL/contact-scoped and mostly PULL-based.
- **Convenient fit:** MeshMonitor (the same tool meshai already uses for Meshtastic) supports MeshCore as a first-class source since v4.5+, over USB/TCP, with a REST API (`/api/nodes`). So MeshCore awareness reuses the existing MeshMonitor pattern. MeshMonitor is already central to meshai's mesh connectivity — meshai's Meshtastic link itself runs over IP through MeshMonitor's vnode — which makes reusing MeshMonitor for MeshCore awareness an especially natural fit.
- **Convenient fit:** MeshMonitor (the same tool meshai already uses for Meshtastic) supports MeshCore as a first-class source since v4.5+, over USB/TCP, with a REST API (`/api/nodes`). So MeshCore awareness reuses the existing MeshMonitor pattern. MeshMonitor is already [[central]] to meshai's mesh connectivity — meshai's Meshtastic link itself runs over IP through MeshMonitor's vnode — which makes reusing MeshMonitor for MeshCore awareness an especially natural fit.
- **What we CAN give a MeshCore user's LLM query:** contact roster (name, node type, last-advert, position if shared, known path/hops via `get_contacts()`), per-message SNR/RSSI, own device telemetry, on-demand telemetry from other nodes (`req_telemetry`/`req_status`, Cayenne LPP), trace/path discovery. Repeater stats (uptime/airtime/neighbors) only if the operator enabled guest access.
- **What's missing vs Meshtastic:** no global/passive view, no firehose, advert SNR/position not inline, companion has no neighbor table (only Repeaters do).
- **Verdict:** less than Meshtastic's effortless global view, but a real local picture — plan MeshCore awareness around active polling of a curated contact/repeater set, not passive ingestion.

6
vault/projects/meshtastic-headscale-runbook.md
View file

@ -9,8 +9,8 @@ related:
- [[idahomesh-vpn-device-setup]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-07-11
- [[services]]
updated: 2026-07-13
---
# IdahoMesh Tailnet Runbook
@ -644,7 +644,7 @@ Any tailscale client older than v1.80.0 will be rejected by 0.29. Verify all reg
- **Date:** 2026-06-21
- **From:** v0.28.0 → **To:** v0.29.1
- **Deployment:** native systemd binary at `/usr/local/bin/headscale`
- **[[deployment]]:** native systemd binary at `/usr/local/bin/headscale`
- **Config changes made:**
- Removed `randomize_client_port: false`
- Replaced `ephemeral_node_inactivity_timeout: 30m` with `node.ephemeral.inactivity_timeout: 30m`

4
vault/projects/mmud-project.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[advbbs-project]]
- [[meshtasticd-sim-nodes-runbook]]
- [[meshai]]
- [[ip-allocation]]
- [[services]]
- [[meshtastic-headscale-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# MMUD — Mesh Multi-User Dungeon

19
vault/projects/nominatim-v5-reimport.md
View file

@ -1,15 +1,22 @@
---
title: "Nominatim v5 Re-import"
title: Nominatim v5 Re-import
type: project
tags: [recon, storage]
related: []
tags:
- recon
aliases: []
related:
- [[navi]]
- [[fleet-platform-baseline]]
- [[fleet-patch-audit]]
- [[recon-operations]]
- [[themes]]
updated: 2026-07-13
status: complete
updated: 2026-06-23
---
# Nominatim v5 Re-import
Spun off from the [[fleet-patch-audit]] (2026-06-19). Completed 2026-06-23 as a standalone maintenance window.
Spun off from the [[fleet-patch-audit]] ([[2026-06-19]]). Completed 2026-06-23 as a standalone maintenance window.
## Deployed 2026-06-23
@ -45,7 +52,7 @@ The prior v4.5 container, its ~26 GB DB, and the `mediagis/nominatim:4.5` image
### Coverage clarification
The v4.5 deployment was also western-11 only — this was not always clearly documented. The v5 upgrade was a like-for-like data freshness + engine refresh, not a coverage expansion.
The v4.5 [[deployment]] was also western-11 only — this was not always clearly documented. The v5 upgrade was a like-for-like data freshness + engine refresh, not a coverage expansion.
### Photon — not coupled this upgrade

2
vault/runbooks/add-peertube-channel.md
View file

@ -10,7 +10,7 @@ related:
- [[recon-service-integration]]
- [[proxmox-onboard-node]]
- [[ct-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Add PeerTube Channel

8
vault/runbooks/authentik-access-groups.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[authentik-oidc-application]]
- [[authentik]]
- [[authentik-create-invitation]]
- [[authentik]]
- [[authentik-upgrade]]
- [[deploy-livesync]]
- [[proxmox-onboard-node]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik Access Groups
@ -24,7 +24,7 @@ Manage group-based application access via the [[authentik]] API. No web UI inter
## How It Works
By default, any authenticated Authentik user can access any application. Adding a **policy binding** that ties a **group** to an **application** restricts that app to group members only (plus superusers).
By default, any authenticated [[authentik]] user can access any application. Adding a **policy binding** that ties a **group** to an **application** restricts that app to group members only (plus superusers).
- One binding per group-application pair
- An app can have multiple group bindings (policy_engine_mode=`any` means membership in ANY bound group grants access)

6
vault/runbooks/authentik-create-invitation.md
View file

@ -5,12 +5,12 @@ tags:
- auth
aliases: []
related:
- [[authentik-oidc-application]]
- [[authentik-access-groups]]
- [[authentik]]
- [[authentik-oidc-application]]
- [[authentik-upgrade]]
- [[mailcow-create-mailbox]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik: Create Invitation
@ -26,7 +26,7 @@ Any time a new user needs to be invited to Echo6 services. Invitations create a
## Prerequisites
- Authentik admin access at https://auth.echo6.co
- [[authentik]] admin access at https://auth.echo6.co
- For email mode: SMTP must be configured and working (no-reply@echo6.co via Mailcow)
---

8
vault/runbooks/authentik-oidc-application.md
View file

@ -8,9 +8,9 @@ related:
- [[authentik]]
- [[authentik-access-groups]]
- [[authentik-upgrade]]
- [[mailcow-create-mailbox]]
- [[expose-service-home]]
updated: 2026-07-11
- [[headscale-oidc-boot-order]]
- [[authentik-create-invitation]]
updated: 2026-07-13
---
# Add Authentik OIDC to an Application
@ -18,7 +18,7 @@ Fully automated via [[authentik]] API. No web UI interaction required.
**Prerequisite:** [[dns]] must already exist for the service (run expose-service-edge2.md or expose-service-home.md first).
**Authentik instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
**[[authentik]] instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
---

8
vault/runbooks/authentik-upgrade.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[authentik-oidc-application]]
- [[lxc-service-migration]]
- [[authentik]]
- [[headscale-oidc-boot-order]]
- [[lxc-service-migration]]
- [[authentik-create-invitation]]
- [[ct-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik: Major Version Upgrade
@ -22,7 +22,7 @@ Upgrade [[authentik]] between major versions on edge2 CT 105. Covers backup, upg
## When to Use This
Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
Any time [[authentik]] is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
---

15
vault/runbooks/central-deploy-cutover.md
View file

@ -1,9 +1,16 @@
---
title: "central — Deploy & Cutover Runbook"
type: runbook
tags: [recon]
related: ["[[central]]"]
updated: 2026-06-28
tags:
- mesh
aliases: []
related:
- [[central]]
- [[recon-operations]]
- [[lxc-service-migration]]
- [[deployment]]
- [[syncthing-add-node]]
updated: 2026-07-13
---
# central — Deploy & Cutover Runbook
@ -34,7 +41,7 @@ Use this runbook whenever you need to deploy a new release or roll back.
## Pre-flight (ALWAYS run before any deploy)
**1. Confirm services are healthy now:**
**1. Confirm [[services]] are healthy now:**
```bash
systemctl is-active central-supervisor central-archive central-gui

8
vault/runbooks/ct-runbook.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[meshtasticd-sim-nodes-runbook]]
- [[proxmox-create-ubuntu-vm]]
- [[ots-setup]]
updated: 2026-06-18
- [[headscale-onboard-node]]
- [[nordvpn-lxc]]
- [[meshtasticd-sim-nodes-runbook]]
updated: 2026-07-13
---
# Proxmox CT/LXC Provisioning Runbook

6
vault/runbooks/edge2-access-reference.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[expose-service-edge2]]
- [[proxmox-onboard-node]]
- [[vaultwarden-plan]]
- [[lxc-service-migration]]
- [[authentik]]
- [[headscale-onboard-node]]
updated: 2026-06-18
updated: 2026-07-13
---
# edge2 Access Reference
@ -113,4 +113,4 @@ ssh edge2 'sudo pct stop <CTID>'
**Root cause:** PVE `root@pam` password and the system root SSH password are managed separately. On edge2, the system root password was set by cloud-init at provisioning and may differ. Additionally, root SSH login is disabled entirely.
**Prevention:** Document both auth paths (SSH user + PVE API) separately in credentials and environment docs.
**Prevention:** Document both auth paths (SSH user + PVE API) separately in credentials and [[environment]] docs.

6
vault/runbooks/expose-service-contabo.md
View file

@ -8,13 +8,13 @@ related:
- [[expose-service-edge2]]
- [[expose-service-home]]
- [[lxc-service-migration]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-07-11
- [[services]]
updated: 2026-07-13
---
# Expose Service on Contabo
> SUPERSEDED — Contabo was decommissioned 2026-06-19. Use [[expose-service-edge2]] (services) or [[expose-service-contabo]]→edge1 for mail. This doc is kept for history only.
> SUPERSEDED — Contabo was decommissioned 2026-06-19. Use [[expose-service-edge2]] ([[services]]) or [[expose-service-contabo]]→edge1 for mail. This doc is kept for history only.
## Prerequisites
- Service running in Docker on Contabo

10
vault/runbooks/expose-service-edge2.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[lxc-service-migration]]
- [[edge2-access-reference]]
- [[expose-service-contabo]]
- [[expose-service-home]]
- [[vaultwarden-plan]]
updated: 2026-06-18
- [[edge2-access-reference]]
- [[caddy]]
updated: 2026-07-13
---
# Expose Service on edge2 (Contabo Cloud VPS)
@ -67,7 +67,7 @@ sudo pct push <CTID> /path/on/host /path/in/ct
### 3. Add Caddy site block on edge2 host
Caddy runs on the edge2 host and terminates TLS.
[[caddy]] runs on the edge2 host and terminates TLS.
**For Cloudflare-proxied domains** (orange cloud / Full SSL mode):
```bash
@ -153,7 +153,7 @@ curl -I https://<domain>/
| edge2 Tailscale | 100.64.0.26 |
| Internal bridge | vmbr0, 10.10.10.0/24 |
| Gateway | 10.10.10.1 (edge2 host) |
| DNS in CTs | 1.1.1.1 |
| [[dns]] in CTs | 1.1.1.1 |
| CT IP range | 10.10.10.10+ (10=pdm, 11=wordpress) |
## CT Creation via PVE API (alternative)

8
vault/runbooks/expose-service-home.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[expose-service-edge2]]
- [[expose-service-contabo]]
- [[caddy]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[ct-runbook]]
updated: 2026-06-18
updated: 2026-07-13
---
# Expose Service on Home Network
@ -25,8 +25,8 @@ updated: 2026-06-18
| Has OIDC? | Proxy to | Why |
|-----------|----------|-----|
| YES | Local IP (192.168.1.x:port) | Authentik SSO protects access |
| NO | Tailscale IP (100.64.0.x:port) | Only Caddy can reach backend |
| YES | Local IP (192.168.1.x:port) | [[authentik]] SSO protects access |
| NO | Tailscale IP (100.64.0.x:port) | Only [[caddy]] can reach backend |
If no OIDC, service MUST have Tailscale installed and registered with Headscale first.

29
vault/runbooks/fleet-magicdns-resolved-migration.md
View file

@ -1,16 +1,23 @@
---
title: Fleet MagicDNS / systemd-resolved Migration
type: runbook
tags: [dns, vpn]
related: []
updated: 2026-06-22
tags:
- dns
aliases: []
related:
- [[headscale-onboard-node]]
- [[caddy]]
- [[meshtastic-headscale-runbook]]
- [[headscale-oidc-boot-order]]
- [[lxc-service-migration]]
updated: 2026-07-13
---
# Fleet MagicDNS / systemd-resolved Migration
## Context / why
Fleet guests on Tailscale had a fragile DNS setup. On LXC CTs without systemd-resolved, Tailscale owns `/etc/resolv.conf` and points **all** queries (public included) at the MagicDNS proxy `100.100.100.100`. If tailscaled loses its link, all DNS dies — including the lookup of the coordinator `vpn.echo6.co` needed to reconnect. That circular dependency is a hard brick.
Fleet guests on Tailscale had a fragile [[dns]] setup. On LXC CTs without systemd-resolved, Tailscale owns `/etc/resolv.conf` and points **all** queries (public included) at the MagicDNS proxy `100.100.100.100`. If tailscaled loses its link, all DNS dies — including the lookup of the coordinator `vpn.echo6.co` needed to reconnect. That circular dependency is a hard brick.
This triggered during the June 2026 patch campaign: a CT went completely offline when tailscaled dropped and couldn't resolve its way back.
@ -176,7 +183,7 @@ systemctl daemon-reload
### Step 4 — Guard #2: PVE resolv.conf overwrite protection
PVE rewrites `/etc/resolv.conf` from the host at `pct start` — and may mark it immutable with `chattr +i`. This service restores the stub symlink before any container services run.
PVE rewrites `/etc/resolv.conf` from the host at `pct start` — and may mark it immutable with `chattr +i`. This service restores the stub symlink before any container [[services]] run.
```bash
pct exec <ID> -- bash -lc "
@ -431,16 +438,16 @@ Across all 19 guests the recipe held with no failures. The "two tailscaled resta
| CT | Host | Bucket | Docker | Notes |
|---|---|---|---|---|
| CT108 meshai | utility | R | yes | — |
| CT102 searxng | utility | I | yes | — |
| CT108 [[meshai]] | utility | R | yes | — |
| CT102 [[searxng]] | utility | I | yes | — |
| CT112 cobalt | utility | R | yes | No containers deployed; daemon.json DNS pin applied |
| CT100 meshmonitor | utility | I | yes | meshai (CT108) stayed healthy throughout; MagicDNS canary validated via media.echo6.mesh |
| CT104 central | utility | I | no | — |
| CT101 caddy | utility | I | no | — |
| CT104 [[central]] | utility | I | no | — |
| CT101 [[caddy]] | utility | I | no | — |
| CT107 mesh-bridge | utility | I | no | Dual-tailnet bridge (echo6 tailscale0 + IdahoMesh tailscale1); both daemons migrated |
| CT110 peertube | media | I | no | NordVPN allowlist applied |
| CT111 mcc | media | I | no | Caddy on :80; healthy before+after |
| CT103 argus | cloud | R | yes | daemon.json merged (had runtime keys, no dns pin); argus-app stack (grafana/postgres) healthy; pre-existing: argus-app compose lacks restart:always — needs manual `docker compose up -d` after reboot (not a DNS issue) |
| CT103 [[argus]] | cloud | R | yes | daemon.json merged (had runtime keys, no dns pin); argus-app stack (grafana/postgres) healthy; pre-existing: argus-app compose lacks restart:always — needs manual `docker compose up -d` after reboot (not a DNS issue) |
| CT120 immich | cloud | R | yes | daemon.json pre-pinned; machine_learning unhealthy pre-existing (self-cleared after reboot) |
| CT121 nextcloud | cloud | R | yes | daemon.json created (missing); 12-container AIO stack (apache/app/db/redis/collabora/…) all healthy before+after; CorpDNS null (expected for this build) — validated via resolvectl tailscale0 echo6.mesh ~.; reboot-persistent |
| CT101 wordpress | edge2 | I | yes | — |
@ -448,7 +455,7 @@ Across all 19 guests the recipe held with no failures. The "two tailscaled resta
| CT102 vaultwarden | edge2 | I | no | — |
| CT103 forgejo | edge2 | I | no | — |
| CT104 livesync | edge2 | I | no | — |
| CT105 authentik | edge2 | I | yes | — |
| CT105 [[authentik]] | edge2 | I | yes | — |
| CT106 matrix | edge2 | I | no | — |
### Already compliant (gold-standard VMs)

19
vault/runbooks/headless-browser-page-verification.md
View file

@ -1,16 +1,23 @@
---
title: "Headless Browser — Visual Page Verification"
title: Headless Browser — Visual Page Verification
type: runbook
tags: [tooling]
related: ["[[central]]", "[[central-deploy-cutover]]"]
updated: 2026-06-29
tags:
- tooling
aliases: []
related:
- [[peertube-remote-runner]]
- [[central-deploy-cutover]]
- [[toc-cortex-pve9.2-update]]
- [[syncthing-add-node]]
- [[headscale-oidc-boot-order]]
updated: 2026-07-13
---
# Headless Browser — Visual Page Verification
Drive a real headless browser (Playwright + Chromium) from **cortex** to log in and **screenshot a deployed web page**, then view the screenshot. Use this to *actually look* at a page after a deploy — it catches render/layout/styling bugs that `curl` status codes and unit tests sail right past (a route can return `200`/`302` and still look broken).
Works for **any** web UI reachable from cortex (central, navi, Authentik, PeerTube, Mailcow, etc.) — just change the base URL, path, and credentials.
Works for **any** web UI reachable from cortex ([[central]], [[navi]], [[authentik]], PeerTube, Mailcow, etc.) — just change the base URL, path, and credentials.
> Reusing this in a prompt: tell Claude *"follow the headless-browser-page-verification runbook to screenshot `<service> <path>`"* and point it at the creds. Everything needed is self-contained below.
@ -18,7 +25,7 @@ Works for **any** web UI reachable from cortex (central, navi, Authentik, PeerTu
## Prerequisites
- Runs on **cortex** (where Claude Code executes). The target must be reachable from cortex — most fleet UIs are on the mesh (e.g. `http://100.64.0.12:8000`) or via a Caddy host.
- Runs on **cortex** (where Claude Code executes). The target must be reachable from cortex — most fleet UIs are on the mesh (e.g. `http://100.64.0.12:8000`) or via a [[caddy]] host.
- `python3` available.
- Credentials for auth-gated pages come from **`.ref/credentials`** (e.g. `CENTRAL_OPERATOR_USER`/`CENTRAL_OPERATOR_PASS`). Public/auth-exempt paths (`/login`, `/health`) need none.
- **Installing Playwright + Chromium is a package install** — get Matt's OK first per host policy (he authorized it 2026-06-29). No `apt`/system-dep install is needed on cortex; the browser is a self-contained download to `~/.cache/ms-playwright`.

12
vault/runbooks/headscale-oidc-boot-order.md
View file

@ -2,17 +2,19 @@
title: Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)
type: runbook
tags:
- vpn
- auth
aliases: []
related:
- [[headscale-onboard-node]]
- [[authentik-oidc-application]]
- [[edge2-access-reference]]
updated: 2026-06-30
- [[headscale-onboard-node]]
- [[authentik]]
- [[authentik-upgrade]]
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)
Headscale wires up its OIDC provider **once, at process startup**, by fetching Authentik's discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs **OIDC-disabled until it is restarted**. This is a boot-ordering hazard: the fleet Headscale (edge2 **CT107**) and Authentik (edge2 **CT105**, `auth.echo6.co`) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.
Headscale wires up its OIDC provider **once, at process startup**, by fetching [[authentik]]'s discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs **OIDC-disabled until it is restarted**. This is a boot-ordering hazard: the fleet Headscale (edge2 **CT107**) and Authentik (edge2 **CT105**, `auth.echo6.co`) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.
## Symptom

6
vault/runbooks/headscale-onboard-node.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[proxmox-onboard-node]]
- [[ct-runbook]]
- [[caddy]]
- [[meshtastic-headscale-runbook]]
- [[lxc-service-migration]]
updated: 2026-06-30
- [[fleet-magicdns-resolved-migration]]
- [[idahomesh-vpn-device-setup]]
updated: 2026-07-13
---
# Headscale / Tailscale — Onboard a New Node

2
vault/runbooks/ia-cli-reference.md
View file

@ -10,7 +10,7 @@ related:
- [[idahomesh-vpn-device-setup]]
- [[archivist]]
- [[usenet]]
updated: 2026-06-18
updated: 2026-07-13
---
# Internet Archive CLI Reference

4
vault/runbooks/ia-download-mirror.md
View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[ia-cli-reference]]
- [[ia-download-queue]]
- [[recon]]
- [[pipeline-patterns]]
- [[syncthing-add-node]]
- [[idahomesh-vpn-device-setup]]
updated: 2026-06-18
updated: 2026-07-13
---
# Download & Mirror from Internet Archive

6
vault/runbooks/idahomesh-bridge-setup.md
View file

@ -5,12 +5,12 @@ tags:
- mesh
aliases: []
related:
- [[meshtastic-headscale-runbook]]
- [[idahomesh-vpn-device-setup]]
- [[meshtastic-headscale-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-06-18
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# IdahoMesh Bridge Setup

4
vault/runbooks/idahomesh-vpn-device-setup.md
View file

@ -9,8 +9,8 @@ related:
- [[meshtastic-headscale-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-06-18
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# IdahoMesh VPN — Device Setup

17
vault/runbooks/lxc-service-migration.md
View file

@ -6,16 +6,15 @@ tags:
aliases: []
related:
- [[expose-service-edge2]]
- [[vaultwarden-plan]]
- [[headscale-onboard-node]]
- [[caddy]]
- [[expose-service-contabo]]
- [[edge2-access-reference]]
updated: 2026-07-11
- [[services]]
- [[matrix_host]]
updated: 2026-07-13
---
# LXC Service Migration — Contabo → edge2
> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on 2026-06-19 — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.
> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on [[2026-06-19]] — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.
> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), **Matrix stack → edge2 CT 106** (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and **Headscale → edge2 CT 107** (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
@ -23,7 +22,7 @@ updated: 2026-07-11
## Overview
Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public [[dns]] never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.
Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo [[caddy]] frontend unchanged (public [[dns]] never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.
**Architecture after migration:**
@ -79,7 +78,7 @@ ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /de
ssh edge2 "sudo pct start <CTID>"
```
Verify: internet access via NAT, DNS resolution.
Verify: internet access via NAT, [[dns]] resolution.
**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`
@ -333,7 +332,7 @@ vault.echo6.co {
}
```
**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x` → `100.64.0.35:598x` in one edit. A third upstream in the same block — the Authentik outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.
**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x` → `100.64.0.35:598x` in one edit. A third upstream in the same block — the [[authentik]] outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.
**Multi-block cutover example (Authentik, 2026-06-18):** Authentik appeared in 4 places across 2 site blocks — `auth.echo6.co` (outpost path matcher + catch-all both pointing to `127.0.0.1:9000`) and `notes.echo6.co` (outpost path matcher + `forward_auth` directive both pointing to `127.0.0.1:9000`). All 4 occurrences were updated to `100.64.0.36:9000` in one edit. Grep the entire Caddyfile for the service's port before cutting over — do not assume a service lives in only one block. See also G15 (dnsmasq must NOT be repointed) and G16 (SECRET_KEY must travel).
@ -443,7 +442,7 @@ Mailcow cannot be "moved" to a different host IP via the one-token Caddy approac
The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated services were already on edge2; no outage for those).
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated [[services]] were already on edge2; no outage for those).
2. **DNS during rebuild:** echo6.co resolves via public GoDaddy DNS. During the rebuild the `mail.*` records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. **Headscale pre-auth key:** Generate before the rebuild starts (`headscale preauthkeys create`). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. **Tailscale bootstrap DNS:** The tailnet uses `vpn.echo6.co` for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.

8
vault/runbooks/mailcow-create-mailbox.md
View file

@ -9,8 +9,8 @@ related:
- [[authentik-oidc-application]]
- [[caddy]]
- [[authentik-create-invitation]]
- [[proxmox-onboard-node]]
updated: 2026-07-11
- [[expose-service-edge2]]
updated: 2026-07-13
---
# Mailcow: Create Mailbox
@ -112,7 +112,7 @@ Regular user accounts can leave all access flags at their defaults (all enabled)
### The Problem
Mailcow domains configured with OIDC authentication (like `echo6.co` with Authentik SSO) set `authsource=generic-oidc` on **every new mailbox by default**. This tells Dovecot to authenticate the account through the OIDC provider instead of the local password hash.
Mailcow domains configured with OIDC authentication (like `echo6.co` with [[authentik]] SSO) set `authsource=generic-oidc` on **every new mailbox by default**. This tells Dovecot to authenticate the account through the OIDC provider instead of the local password hash.
For service accounts that log in via SMTP with a username and password, this means:
@ -265,7 +265,7 @@ cd /opt/mailcow-dockerized && docker compose restart netfilter-mailcow
|---------|---------|-----------|---------|
| no-reply@echo6.co | Authentik | mailcow | SSO invitation emails, notifications |
| cipher@echo6.co | CIPHER | generic-oidc | Daily intelligence briefs |
| recon@echo6.co | RECON | generic-oidc | Pipeline notifications |
| [[recon]]@echo6.co | RECON | generic-oidc | Pipeline notifications |
| fulcrum@echo6.co | Fulcrum | generic-oidc | Hub notifications |
**Note:** cipher, recon, and fulcrum currently use `generic-oidc`. If any of these need to send mail via SMTP (not through the SSO web UI), their authsource must be changed to `mailcow` per Step 2.

10
vault/runbooks/meshai-prod-compose-override.md
View file

@ -3,10 +3,14 @@ title: meshai prod compose override (cutover state + MeshCore radio)
type: runbook
tags:
- mesh
aliases: []
related:
- [[meshai]]
- [[meshai-region-routing-plan]]
- [[meshcore-transport]]
updated: 2026-07-07
- [[meshai]]
- [[meshai-config-hot-apply]]
- [[SESSION-HANDOFF-meshai-test]]
updated: 2026-07-13
---
# meshai prod compose override
@ -49,7 +53,7 @@ services:
the utility host into the container at a stable udev symlink path. Without it
the container cannot see the MeshCore hardware (see [[meshcore-transport]]).
Both are deployment state (cutover progress + host-specific device path), which
Both are [[deployment]] state (cutover progress + host-specific device path), which
is why they live in an override rather than the committed public compose file.
## Restore

4
vault/runbooks/meshmonitor-password-reset.md
View file

@ -9,8 +9,8 @@ related:
- [[recon-service-integration]]
- [[ct-runbook]]
- [[headscale-onboard-node]]
- [[recon-operations]]
updated: 2026-06-18
- [[pg-backup]]
updated: 2026-07-13
---
# MeshMonitor Admin Password Reset

4
vault/runbooks/meshtastic-sidecar-node.md
View file

@ -9,8 +9,8 @@ related:
- [[meshtastic-headscale-runbook]]
- [[idahomesh-bridge-setup]]
- [[headscale-onboard-node]]
- [[advbbs-project]]
updated: 2026-07-11
- [[meshtasticd-sim-nodes-runbook]]
updated: 2026-07-13
---
# Meshtastic Sidecar Node — Modular Deployment Runbook

6
vault/runbooks/meshtasticd-sim-nodes-runbook.md
View file

@ -8,9 +8,9 @@ related:
- [[ct-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[meshtastic-headscale-runbook]]
- [[proxmox-onboard-node]]
- [[ip-allocation]]
updated: 2026-07-11
updated: 2026-07-13
---
# Meshtasticd SIM Node Runbook — LXC Deployment
@ -415,7 +415,7 @@ pct exec <CTID> -- systemctl status meshtasticd # Check without entering
## Container Inventory Template
Track your deployment:
Track your [[deployment]]:
| CTID | Hostname | MAC Address | Service | Port | Notes |
|------|-------------|---------------------|------------|------|-----------------|

8
vault/runbooks/nordvpn-lxc.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ct-runbook]]
- [[meshtasticd-sim-nodes-runbook]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[peertube-remote-runner]]
updated: 2026-06-18
- [[meshtasticd-sim-nodes-runbook]]
- [[headscale-onboard-node]]
updated: 2026-07-13
---
# NordVPN / WireGuard in LXC
@ -151,7 +151,7 @@ Endpoint = <server-ip>:51820
PersistentKeepalive = 25
```
**Critical for LXC:** If the container runs services that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
**Critical for LXC:** If the container runs [[services]] that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
```ini
# Route everything EXCEPT local network through VPN

6
vault/runbooks/peertube-remote-runner.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[add-peertube-channel]]
- [[headless-browser-page-verification]]
- [[nordvpn-lxc]]
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
updated: 2026-06-18
- [[recon-service-integration]]
updated: 2026-07-13
---
# PeerTube Remote Runner — GPU Transcoding

8
vault/runbooks/pg-backup.md
View file

@ -5,12 +5,12 @@ tags:
- storage
aliases: []
related:
- [[ct-runbook]]
- [[recon-operations]]
- [[meshmonitor-password-reset]]
- [[matrix-synapse-deployment]]
- [[ct-runbook]]
- [[synapse]]
updated: 2026-06-18
- [[meshmonitor-password-reset]]
- [[recon-operations]]
updated: 2026-07-13
---
# PostgreSQL Backup (Docker)

8
vault/runbooks/pi-nas-omv-runbook.md
View file

@ -8,9 +8,9 @@ related:
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[proxmox-create-ubuntu-vm]]
- [[headscale-onboard-node]]
- [[environment]]
updated: 2026-06-18
- [[toc-cortex-pve9.2-update]]
- [[nordvpn-lxc]]
updated: 2026-07-13
---
# Pi 5 NAS — OMV Provisioning Runbook
@ -126,7 +126,7 @@ Each drive is used individually — no RAID array.
### Enable SMB (Windows Shares)
1. **[[services]] → SMB/CIFS → Settings** — toggle **Enabled**, click **Save**
2. **Services → SMB/CIFS → Shares** — click **Create** for each shared folder you want accessible from Windows:
2. **[[services]] → SMB/CIFS → Shares** — click **Create** for each shared folder you want accessible from Windows:
- Select the shared folder
- **Public:** No
- **Browseable:** Yes

9
vault/runbooks/pipeline-patterns.md
View file

@ -1,15 +1,16 @@
---
title: "Pipeline & Wrapper Patterns"
type: runbook
tags: [tooling]
tags:
- tooling
aliases: []
related:
- [[meshtastic-sidecar-node]]
- [[meshtastic-headscale-runbook]]
- [[headscale-onboard-node]]
- [[idahomesh-vpn-device-setup]]
- [[syncthing-add-node]]
updated: 2026-06-18
- [[idahomesh-bridge-setup]]
- [[headscale-onboard-node]]
updated: 2026-07-13
---
# Pipeline & Wrapper Patterns

4
vault/runbooks/proxmox-create-ubuntu-vm.md
View file

@ -5,12 +5,12 @@ tags:
- proxmox
aliases: []
related:
- [[proxmox-onboard-node]]
- [[ct-runbook]]
- [[environment]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[pi-nas-omv-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Proxmox — Create Ubuntu VM (Cloud-Init)

6
vault/runbooks/proxmox-onboard-node.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ct-runbook]]
- [[headscale-onboard-node]]
- [[proxmox-create-ubuntu-vm]]
- [[edge2-access-reference]]
- [[headscale-onboard-node]]
- [[expose-service-home]]
updated: 2026-06-18
- [[edge2-access-reference]]
updated: 2026-07-13
---
# Runbook: Onboard a Proxmox Node

9
vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md
View file

@ -3,9 +3,14 @@ title: pymc-repeater KISS TNC Re-enumeration Fix
type: runbook
tags:
- mesh
aliases: []
related:
- [[meshtastic-sidecar-node]]
updated: 2026-07-01
- [[recon-operations]]
- [[SESSION-HANDOFF-meshai-test]]
- [[meshtasticd-sim-nodes-runbook]]
- [[meshtastic-headscale-runbook]]
- [[syncthing-add-node]]
updated: 2026-07-13
---
# pymc-repeater KISS TNC Re-enumeration Fix

8
vault/runbooks/recon-operations.md
View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[recon]]
- [[deployment]]
- [[caddy]]
- [[recon-service-integration]]
- [[services]]
updated: 2026-07-11
- [[central-deploy-cutover]]
- [[deployment]]
- [[pymc-repeater-kiss-tnc-reenumeration]]
updated: 2026-07-13
---
# RECON Operations Runbook

8
vault/runbooks/recon-service-integration.md
View file

@ -5,12 +5,12 @@ tags:
- recon
aliases: []
related:
- [[proxmox-onboard-node]]
- [[recon-operations]]
- [[headscale-onboard-node]]
- [[proxmox-onboard-node]]
- [[lxc-service-migration]]
- [[caddy]]
updated: 2026-07-11
- [[headscale-onboard-node]]
- [[expose-service-home]]
updated: 2026-07-13
---
# RECON Dashboard Service Integration

8
vault/runbooks/syncthing-add-node.md
View file

@ -7,14 +7,14 @@ aliases: []
related:
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[central-deploy-cutover]]
- [[meshtasticd-sim-nodes-runbook]]
- [[idahomesh-vpn-device-setup]]
- [[headscale-onboard-node]]
updated: 2026-07-11
- [[matrix-synapse-deployment]]
updated: 2026-07-13
---
# Syncthing: Add a New Node to the Project Sync Cluster
> **Syncthing on Contabo was decommissioned 2026-06-19** with the edge1 rebuild (state removed; Forge is now the durable backup via the `echo6-docs-autocommit` cron). The `contabo` row below and its device ID are historical — do not treat it as a live cluster member. Any new-node onboarding should reassess whether this cluster still has a live counterpart before assuming `contabo` is reachable.
> **Syncthing on Contabo was decommissioned [[2026-06-19]]** with the edge1 rebuild (state removed; Forge is now the durable backup via the `echo6-docs-autocommit` cron). The `contabo` row below and its device ID are historical — do not treat it as a live cluster member. Any new-node onboarding should reassess whether this cluster still has a live counterpart before assuming `contabo` is reachable.
## Overview

11
vault/runbooks/toc-cortex-pve9.2-update.md
View file

@ -3,9 +3,14 @@ title: toc + cortex PVE 9.2 / GPU update — run from matt-desktop
type: runbook
tags:
- proxmox
- ai
related: []
updated: 2026-06-22
aliases: []
related:
- [[fleet-platform-baseline]]
- [[environment]]
- [[ct-runbook]]
- [[fleet-patch-audit]]
- [[pi-nas-omv-runbook]]
updated: 2026-07-13
status: active
---

12
vault/session-resume/SESSION-HANDOFF-meshai-test.md
View file

@ -5,12 +5,12 @@ tags:
- mesh
aliases: []
related:
- [[meshai-prod-compose-override]]
- [[meshtastic-sidecar-node]]
- [[meshtastic-headscale-runbook]]
- [[services]]
- [[synapse_retention_discovery]]
- [[caddy]]
updated: 2026-06-18
- [[meshai]]
- [[meshcore-transport]]
- [[meshai-config-hot-apply]]
updated: 2026-07-13
status: open
created: 2026-06-17
origin: matt-desktop (WSL)
@ -24,7 +24,7 @@ resume-on: cortex
## What we were doing
Matt unplugged/replugged the network cable on **aida-nebra** (AIDA-N2 Meshtastic
node) to reboot it. The link dropped for ~12s then recovered. Matt then realized
this is an accidental **resilience test for MeshAI** — he wants to see "how it
this is an accidental **resilience test for [[meshai]]** — he wants to see "how it
dumps": how MeshAI handled losing and regaining its radio TCP connection
(clean reconnect vs. errors/stack traces vs. crash+restart).