matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-07-18T12:00:17+00:00

Browse source 
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/services/services.md vault/glossary.md vault/projects/fleet-patch-audit.md
This commit is contained in:
echo6-autocommit 2026-07-18 12:00:17 +00:00
commit 222a2f198e
8 changed files with 49 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

2
engine/.embcache.json
View file

File diff suppressed because one or more lines are too long

6
engine/changelog.md
View file

@ -179,3 +179,9 @@
- processed: 8 | written: 8 | flagged: 0 | errors: 0
## 2026-07-17T09:00:01Z — sweep deferred (competing GPU process: 3755501 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/8458a2cd-0d04-4bc6-a3c4-37247ca48b37 --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/2WgcrDazx5XSWLUTsBdV9L --model medium --device cuda --compute_type float16 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-07-18T09:00:01Z — sweep run
- end: 2026-07-18T09:00:16Z
- mode: incremental
- docs selected: 5
- processed: 5 | written: 5 | flagged: 0 | errors: 0

2
engine/lint-report.md
View file

@ -1,6 +1,6 @@
# Vault Lint Report
Generated: 2026-07-18T00:00:20Z | Docs scanned: 107 | Elapsed: 0.0s
Generated: 2026-07-18T09:00:01Z | Docs scanned: 107 | Elapsed: 0.0s
## Summary

14
vault/docs/hardware/environment.md
View file

@ -8,9 +8,9 @@ related:
- [[fleet-platform-baseline]]
- [[ip-allocation]]
- [[headscale-onboard-node]]
- [[proxmox-create-ubuntu-vm]]
- [[toc-cortex-pve9.2-update]]
updated: 2026-07-17
- [[proxmox-create-ubuntu-vm]]
updated: 2026-07-18
---
# Echo6 Environment Reference
@ -84,7 +84,7 @@ Five nodes running Proxmox VE:
- **Resources:** 4 cores, 24GB RAM, 180GB disk
- **Software:** Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
- **Platforms:** [[recon]] (knowledge extraction pipeline, :8420) and [[navi]] (offline navigation, navi.echo6.co, :8440) with geo backends (Valhalla :8002, Nominatim :8010, Photon :2322, PostgreSQL/PostGIS :5432)
- **Systemd services:** [[recon]] (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **Systemd [[services]]:** [[recon]] (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **NFS mounts:** pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
- **User:** zvx (sudo, SSH key auth)
- **Migrated from:** CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
@ -128,21 +128,21 @@ Five nodes running Proxmox VE:
| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| [[meshai]] | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | MeshAI - LLM-powered Meshtastic assistant |
| [[meshai]] | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | [[meshai]] - LLM-powered Meshtastic assistant |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | — | [[archivist]] knowledge pipeline |
| [[argus]] | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | [[argus]] - OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104 | 100.64.0.12 | Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see [[central]] |
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.17 | PeerTube video streaming — Tailscale identity is `peertube-4hve9pdr` (collision-suffixed) since ~2026-06-22; `.23` is stale/gone |
| mcc | media (CT 111) | 192.168.1.111 | 100.64.0.19 | pymc console web app ([[caddy]] + Postfix, /api+/auth+/ws → aida-nebra :8000) |
| mailcow | edge1 (CT 101) | 10.10.10.2 | — | Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + Caddy) |
| mailcow | edge1 (CT 101) | 10.10.10.2 | — | Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + [[caddy]]) |
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | Grav CMS 2.0.11 for idahomesh.com (migrated from WordPress 2026-07-17; hostname `wordpress` unchanged; MariaDB purged — flat-file, no DB) |
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| [[authentik]] | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
| [[authentik]] | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | [[authentik]] SSO platform (migrated from Contabo 2026-06-18) |
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo [[2026-06-19]]) |
> **Note (2026-06-19):** edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (`forgejo-ssh-dnat.service`) is a permanent systemd unit on edge2 host.

4
vault/docs/hardware/ip-allocation.md
View file

@ -10,7 +10,7 @@ related:
- [[environment]]
- [[glossary]]
- [[headscale-onboard-node]]
updated: 2026-07-17
updated: 2026-07-18
---
# Echo6 Network IP Allocation
@ -152,4 +152,4 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
---
*Last updated: 2026-06-19 — Added edge1 section (rebuilt Contabo VPS, 5.189.158.149 / 100.64.0.40, mail-only, Mailcow CT 101 at 10.10.10.2); added edge2 CT 108 (mailcow-staging, stopped); Tailscale table: contabo-mailcow entry replaced with edge1 at 100.64.0.40; previously added edge2 CTs 102–107*
*Last updated: [[2026-06-19]] — Added edge1 section (rebuilt Contabo VPS, 5.189.158.149 / 100.64.0.40, mail-only, Mailcow CT 101 at 10.10.10.2); added edge2 CT 108 (mailcow-staging, stopped); Tailscale table: contabo-mailcow entry replaced with edge1 at 100.64.0.40; previously added edge2 CTs 102–107*

24
vault/docs/services/services.md
View file

@ -2,15 +2,15 @@
title: Current Services Inventory
type: reference
tags:
- media
- mesh
aliases: []
related:
- [[caddy]]
- [[ip-allocation]]
- [[lxc-service-migration]]
- [[meshtastic-headscale-runbook]]
- [[expose-service-edge2]]
updated: 2026-07-17
- [[meshtastic-headscale-runbook]]
updated: 2026-07-18
---
# Current Services Inventory
@ -28,14 +28,14 @@ updated: 2026-07-17
| meshtasticd | mt-burleybutte | 192.168.1.185:4403 | Internal | Software Meshtastic node (Nebra 2W hat) |
| IdahoMesh Headscale | utility (CT 106) | 192.168.1.106:8080 | https://vpn.idahomesh.com | Meshtastic mesh VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| MeshAI | utility (CT 108) | 192.168.1.144:4403 / :8080 | Internal | LLM-powered Meshtastic assistant (Docker, work-meshai local build, gemini-3.1-flash-lite, Google grounding) |
| [[meshai]] | utility (CT 108) | 192.168.1.144:4403 / :8080 | Internal | LLM-powered Meshtastic assistant (Docker, work-meshai local build, gemini-3.1-flash-lite, Google grounding) |
| [[argus]] | utility (CT 103) | 192.168.1.103:8080 | Internal | Python app on :8080 — OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104:8000 / 100.64.0.12 | central.echo6.mesh (mesh) | Data-hub spine — ~25 adapters → NATS/JetStream → TimescaleDB; serves traffic tiles to [[navi]] — see [[central]] |
| NATS/JetStream ([[central]]) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | Central backend message bus (NATS :4222 client, :8222 monitoring) |
| NATS/JetStream ([[central]]) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | [[central]] backend message bus (NATS :4222 client, :8222 monitoring) |
| TimescaleDB/PostGIS (central) | utility (CT 104) | 192.168.1.104:5432 | Internal | Central backend time-series + geospatial database (PostgreSQL 16 + TimescaleDB + PostGIS) |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via edge2 DNAT) | https://forge.echo6.co | Git server — fronted by edge2 host Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on edge2 (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via edge2 DNAT) | https://forge.echo6.co | Git server — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on edge2 (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — fronted by edge2 host Caddy — **migrated from Contabo [[2026-06-19]]** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via [[authentik]]) — fronted by edge2 host Caddy — **migrated from Contabo 2026-06-19** |
| Mailcow | **edge1 CT 101** (10.10.10.2) | 5.189.158.149 | https://mail.echo6.co | Email server (privileged LXC on rebuilt Contabo VPS, updated commit 52a41b4d / SOGo 5.12.8) — **rebuilt in-place 2026-06-19** |
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by edge2 host Caddy (reverse_proxy 100.64.0.33:8086) |
@ -55,7 +55,7 @@ updated: 2026-07-17
| Qdrant | cortex (VM 150) | 192.168.1.150:6333 | Internal | Vector database (Docker, [[recon]] knowledge store) |
| TEI | cortex (VM 150) | 192.168.1.150:8090 | Internal | Text embeddings (Docker, bge-m3 1024-dim) |
| [[recon]] | data (VM 1130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
| navi-config | data (VM 1130) | 192.168.1.130:8422 | Internal | RECON navi node config API |
| navi-config | data (VM 1130) | 192.168.1.130:8422 | Internal | [[recon]] [[navi]] node config API |
| navi-contacts | data (VM 1130) | 192.168.1.130:8423 | Internal | RECON navi contact enrichment API |
| navi-landclass | data (VM 1130) | 192.168.1.130:8424 | Internal | RECON navi land classification API |
| navi-places | data (VM 1130) | 192.168.1.130:8425 | Internal | RECON navi OSM place detail/enrichment |
@ -68,7 +68,7 @@ updated: 2026-07-17
| Nominatim | data (VM 1130) | 192.168.1.130:8010 | Internal | navi geo backend (geocoder) — see [[navi]] |
| Photon | data (VM 1130) | 192.168.1.130:2322 | Internal | navi geo backend (geocoder + Elasticsearch :9201) — see [[navi]] |
| PostgreSQL/PostGIS | data (VM 1130) | 192.168.1.130:5432 | Internal | navi geo backend (padus, overture DBs) — see [[navi]] |
| Files | data (VM 1130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, Authentik forward auth) |
| Files | data (VM 1130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, [[authentik]] forward auth) |
| Samba | data | 192.168.1.240:445 | Internal | SMB file sharing — `//data/library` → /mnt/data/library (guest access) |
| Matrix [[synapse]] | edge2 (CT 106) | 100.64.0.37:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) — **migrated from Contabo 2026-06-18** |
| Element Web | edge2 (CT 106) | 100.64.0.37:8088 | https://element.echo6.co | Matrix web client (Docker) — **migrated from Contabo 2026-06-18** |
@ -144,7 +144,7 @@ updated: 2026-07-17
- Compose path: `/opt/searxng/docker-compose.yml`
- Theme files: `/opt/searxng/custom/` (bind-mounted into container)
- `templates/simple/base.html` — custom template (nav, CSS, waffle menu, footer)
- `templates/simple/index.html` — custom homepage (Echo6 logo replaces SearXNG title)
- `templates/simple/index.html` — custom homepage (Echo6 logo replaces [[searxng]] title)
- `img/echo6-logo.png` — Echo6 logo (replaces SearXNG logo)
- `img/favicon.png` — Echo6 favicon
- Config: `/opt/searxng/searxng-config/settings.yml` (instance_name: "Echo6", dark theme, center_alignment: false)
@ -356,7 +356,7 @@ updated: 2026-07-17
- Containers: `synapse` (:8008), `matrix-mas` (:8085), `element` (:8088), `matrix-postgres`, `mautrix-signal` (29328 internal) — 5 containers total
- Databases: PostgreSQL DBs `synapse`, `mas`, `mautrix_signal` on `matrix-postgres` container — 3 DBs total
- Binds to `100.64.0.37`; edge2 host Caddy proxies:
- `matrix.echo6.co` → Synapse (`100.64.0.37:8008`) + MAS sub-routes
- `matrix.echo6.co` → [[synapse]] (`100.64.0.37:8008`) + MAS sub-routes
- `element.echo6.co` → Element (`100.64.0.37:8088`)
- MAS (`100.64.0.37:8085`) handles login/logout/refresh/auth_metadata
- Federation delegation from apex `echo6.co` (`.well-known`) unchanged
@ -403,7 +403,7 @@ updated: 2026-07-17
- **Mailcow** email server (privileged LXC, updated to commit `52a41b4d`, SOGo 5.12.8)
- Accessible at `mail.echo6.co` / `5.189.158.149` (via host Caddy + DNAT)
- mail_crypt secondary key in `data/conf/dovecot/extra.conf` (`ecprivkey.pem.fresh-20260619` as secondary; both primary + secondary loaded for decryption of old mail)
- DNS: `mail/autodiscover/autoconfig.echo6.co` → 5.189.158.149
- [[dns]]: `mail/autodiscover/autoconfig.echo6.co` → 5.189.158.149
#### Previously on Contabo (now evacuated or decommissioned)

12
vault/glossary.md
View file

@ -8,9 +8,9 @@ related:
- [[ip-allocation]]
- [[services]]
- [[caddy]]
- [[edge2-access-reference]]
- [[proxmox-onboard-node]]
updated: 2026-07-17
- [[edge2-access-reference]]
updated: 2026-07-18
---
# Glossary & Vocabulary
@ -89,9 +89,9 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
### Services
- **echo6-agent** — aliases: echo6-agent — on: 2026-06-16
- **echo6-contabo-agent** (Echo6 Contabo Agent) — aliases: Echo6 Contabo Agent — on: contabo _(decommissioned/historical — Contabo-local agent; host rebuilt as edge1, mail-only, 2026-06-19)_
- **echo6-contabo-agent** (Echo6 Contabo Agent) — aliases: Echo6 Contabo Agent — on: contabo _(decommissioned/historical — Contabo-local agent; host rebuilt as edge1, mail-only, [[2026-06-19]])_
- **echo6-cortex-agent** (Echo6 Cortex Agent) — aliases: Echo6 Cortex Agent — on: cortex
- **echo6-search-searxng** (Echo6 Search ([[searxng]])) — aliases: Echo6 Search (SearXNG) — on: utility
- **echo6-search-searxng** (Echo6 Search ([[searxng]])) — aliases: Echo6 Search ([[searxng]]) — on: utility
- **element-web** (Element Web) — aliases: Element Web, element — on: edge2
- **files** (Files) — aliases: Files, files — on: data
- **forge-forgejo** (Forge (Forgejo)) — aliases: Forge (Forgejo), forge — on: edge2
@ -149,13 +149,13 @@ mesh · matrix · [[recon]] · media · auth · [[dns]] · vpn · storage · pro
- **tak-server** (TAK Server) — aliases: TAK Server — on: 2026-06-16
- **tei** (TEI) — aliases: TEI — on: cortex
- **termix** (Termix) — aliases: Termix — on: contabo _(decommissioned — wiped with edge1 rebuild 2026-06-19, not migrated to edge2)_
- **utility-caddy** (Utility [[caddy]]) — aliases: Utility Caddy — on: utility
- **utility-caddy** (Utility [[caddy]]) — aliases: Utility [[caddy]] — on: utility
- **watchtower** (WATCHTOWER) — aliases: WATCHTOWER — on: 2026-06-16
### Projects
- **[[advbbs-project]]** — aliases: [[advbbs-project]]
- **[[argus]]** — aliases: argus
- **[[argus]]** — aliases: [[argus]]
- **[[deploy-livesync]]** — aliases: [[deploy-livesync]]
- **[[matrix-synapse-deployment]]** — aliases: [[matrix-synapse-deployment]]
- **[[meshtastic-headscale-runbook]]** — aliases: [[meshtastic-headscale-runbook]]

28
vault/projects/fleet-patch-audit.md
View file

@ -8,9 +8,9 @@ related:
- [[fleet-platform-baseline]]
- [[lxc-service-migration]]
- [[caddy]]
- [[services]]
- [[ip-allocation]]
updated: 2026-07-17
- [[services]]
updated: 2026-07-18
status: complete
---
@ -18,7 +18,7 @@ status: complete
Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this is a planning document to build the patch plan from.**
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No [[services]] route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149). **CT101 update (2026-07-17):** migrated from WordPress/MariaDB to **Grav CMS 2.0.11** (flat-file, no database — MariaDB purged from the container); serves idahomesh.com (not intermountainmesh.com — that domain has always been parked at a third party and never reached this container); hostname `wordpress` unchanged. See [[environment]] / [[services]] for current state. All WordPress/MariaDB references below reflect the pre-migration state as audited on 2026-06-19 and are left as historical record.
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No [[services]] route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149). **CT101 update (2026-07-17):** migrated from WordPress/MariaDB to **Grav CMS 2.0.11** (flat-file, no database — MariaDB purged from the container); serves idahomesh.com (not intermountainmesh.com — that domain has always been parked at a third party and never reached this container); hostname `wordpress` unchanged. See [[environment]] / [[services]] for current state. All WordPress/MariaDB references below reflect the pre-migration state as audited on [[2026-06-19]] and are left as historical record.
---
@ -60,7 +60,7 @@ Updates where the application or its Docker images have drifted from current ups
| Scope | Guest | Item | Notes |
|-------|-------|------|-------|
| edge2 | CT105 | authentik 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
| edge2 | CT105 | [[authentik]] 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
| edge2 | CT107 | headscale 0.28.0 → 0.29.1 | Also a 2nd headscale on utility CT106 |
| edge2 | CT103 | forgejo 14.0.5 → 15.0.3 | **14.x EOL 2026-04-30** — migrate branch, not just patch |
| edge2 | CT106 | [[synapse]] 1.155.0 / Element / MAS | Image drift + pending OS apt security updates |
@ -143,15 +143,15 @@ Running application version vs latest stable upstream, per app — the "is every
### Current / already past the fix (no action)
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/[[themes]] (edge2 CT101 — **superseded 2026-07-17, see topology note above**) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/[[themes]] (edge2 CT101 — **superseded 2026-07-17, see topology note above**) · [[synapse]] 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
### Lower urgency
Mumble 1.5.517→1.5.901 · Caddy 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · [[searxng]] (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · [[mautrix_signal]] v0.2603.0.
Mumble 1.5.517→1.5.901 · [[caddy]] 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · [[searxng]] (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · [[mautrix_signal]] v0.2603.0.
### Internal echo6 apps (no upstream to track)
central-*, meshai, archivist, meshwars, [[recon]] / recon-watchdog, navi-* — running; version = current git head.
central-*, [[meshai]], [[archivist]], meshwars, [[recon]] / recon-watchdog, navi-* — running; version = current git head.
---
@ -237,7 +237,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
| wordpress CT101 | dump taken on CT101 | — |
| peertube CT110 | DB dump | — |
| opentakserver CT109 | PG dump + LVM snapshot | — |
| central CT104 | PG dump + snapshot | — |
| [[central]] CT104 | PG dump + snapshot | — |
| immich CT120 | `/root/immich-predates-20260620.sql` | — |
| headscale CT107 | `/root/headscale-db-20260620.sqlite` | — |
@ -245,7 +245,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
### Incidental fixes made during Phase 1
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN [[dns]] protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and DNS remained healthy after boot.
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN [[dns]] protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and [[dns]] remained healthy after boot.
- **(b) CT111 mcc — DNS hijacked to unreachable MagicDNS.** Tailscale `accept-dns` was redirecting DNS to a MagicDNS address that was not reachable from this CT. Disabled `tailscale accept-dns`, set `1.1.1.1` / `8.8.8.8` persistently.
- **(c) PostgreSQL on central CT104 moved 16.13→16.14** as part of the security-pocket apt pass.
- **(d) Fleet-wide stale `/etc/hosts` fix** — see Critical Finding 2.
@ -327,7 +327,7 @@ EOL-branch migration; v15 schema migrations applied cleanly; web 200, API report
**OpenTAKServer (utility CT109) 1.7.10 → 1.7.12**
Clean; all 9 TAK services healthy.
Clean; all 9 TAK [[services]] healthy.
### Key decision — RabbitMQ LEFT on 3.12.1 (EOL)
@ -336,7 +336,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
### Incidental fixes and side-work during Phase 2
- **CT107 boot-survival fix** (applied earlier in the effort, during Phase 1 resolution) — rebound headscale/headplane ports to 10.10.10.25, dropped `tailscale-online.target` dependency, disabled `only_start_if_oidc_is_available` gate, repointed edge2 Caddy; proven by reboot self-heal in ~45 s. Also corrected CT107's own tailscale node ControlURL to `vpn.echo6.co` so it self-registers cleanly.
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 [[argus]] running hot (transcription + docker-compose build churn). Matt migrated argus to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 [[argus]] running hot (transcription + docker-compose build churn). Matt migrated [[argus]] to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
- **Nextcloud:** granted `matt@echo6.co` the NC admin role. (user_oidc has no group-claim sync, so this is durable across SSO logins.)
- **Radarr:** set up a `\\192.168.1.160\manual` SMB drop folder on the same NFS export as the library (atomic-move imports) for manual movie filing.
- **Snapshot hygiene:** all rollback snapshots cleaned up after validation — Phase 1 `presec-*`, Phase 2 `prewave2-*`, OTS `pre-ots-*` snapshots all removed.
@ -346,7 +346,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
**Low-urgency batch — DONE:**
- Caddy (utility CT101): 2.10.2 → 2.11.4
- CouchDB (edge2 CT104 livesync): 3.4 → 3.5.2 — all 5 DBs intact
- valkey-8 sidecar (utility CT102 searxng): bumped to latest 8.x
- valkey-8 sidecar (utility CT102 [[searxng]]): bumped to latest 8.x
- NATS (utility CT104 central): 2.14.0 → 2.14.2 — all 12 JetStream streams intact
- MediaMTX (CT109): 1.13.0 → 1.19.1
- Mumble (CT109): already latest in Ubuntu repo (1.5.517 — no upstream action possible without going off-distro)
@ -414,7 +414,7 @@ One at a time; cluster stayed 5/5 quorate throughout.
- **Guest OS = security-only.** LXC containers and VMs received security-pocket apt updates only (the intentional Phase 1 scope). Non-security package drift (e.g. Docker CE versions, miscellaneous libs) was deliberately not swept with a full `apt full-upgrade`. The PVE hosts, pi-nas, and cortex did receive full upgrades. Operator accepted this state. A full guest `apt full-upgrade` ("Phase 1.5") remains an option if ever wanted.
- **Apps capped by external factors (decisions, not failures):** RabbitMQ 3.12 left by decision — OTS depends on it and 4.x would break it; Lidarr v2 — the lidarr-on-steroids image maintainer has not shipped v3, would require an image swap; Jellyseerr on `preview-OIDC` — kept because stable 3.3.0 lacks OIDC/SSO support; Mumble 1.5.517 — newest version in the Ubuntu 24.04 repo, upstream 1.5.901 is not available without going off-distro.
- **Deferred project:** Nominatim v5 re-import — spun off to [[nominatim-v5-reimport]].
- **Deferred project:** [[Nominatim v5 Re-import]] — spun off to [[nominatim-v5-reimport]].
- **Optional cosmetic follow-ups (non-urgent, non-blocking):** navidrome.echo6.co expired unmanaged cert; MediaMTX deprecated config param names (`protocols`/`encryption`); `rpi-eeprom` held on pi-nas; vestigial utility exit-node route (0.0.0.0/0).
- **Kernel summary:** all 5 PVE hosts on kernel 7.0.12-1-pve; LXC containers share the host kernel (7.0); VM guests (recon-vm, arr VM105) have their own Ubuntu kernels (security-patched during Phase 1, not necessarily absolute-latest upstream); pi-nas on 6.18.34+rpt-rpi-2712; cortex kernel updated as part of the toc+cortex window.
@ -752,7 +752,7 @@ Complete point-in-time state of every node, guest, and container service.
| CT103 | forgejo | 14.0.5 (forgejo:14 + postgres:16-alpine drifted) |
| CT104 | livesync | couchdb:3.4 drifted + local provisioner |
| CT105 | authentik | 2025.12.4 (server/worker/postgres) → upgrade to 2026.2.0 |
| CT106 | matrix | Synapse 1.155.0 / Element / MAS + mautrix-signal + postgres; OS apt security updates pending; no Tailscale client |
| CT106 | matrix | Synapse 1.155.0 / Element / MAS + [[mautrix_signal]] + postgres; OS apt security updates pending; no Tailscale client |
| CT107 | headscale | 0.28.0 → 0.29.0 + headplane; no Tailscale client |
| ~~CT108~~ | ~~mailcow~~ | **Decommissioned 2026-06-20** — destroyed (`pct destroy 108 --purge`); superseded by edge1, backup on pi-nas |