auto: docs sync 2026-07-18T12:00:17+00:00
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/services/services.md vault/glossary.md vault/projects/fleet-patch-audit.md
This commit is contained in:
echo6-autocommit
parent
21c4fdbac3
commit
222a2f198e
8 changed files with 49 additions and 43 deletions
|
|
@ -8,9 +8,9 @@ related:
|
|||
- [[fleet-platform-baseline]]
|
||||
- [[lxc-service-migration]]
|
||||
- [[caddy]]
|
||||
- [[services]]
|
||||
- [[ip-allocation]]
|
||||
updated: 2026-07-17
|
||||
- [[services]]
|
||||
updated: 2026-07-18
|
||||
status: complete
|
||||
---
|
||||
|
||||
|
|
@ -18,7 +18,7 @@ status: complete
|
|||
|
||||
Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this is a planning document to build the patch plan from.**
|
||||
|
||||
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No [[services]] route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149). **CT101 update (2026-07-17):** migrated from WordPress/MariaDB to **Grav CMS 2.0.11** (flat-file, no database — MariaDB purged from the container); serves idahomesh.com (not intermountainmesh.com — that domain has always been parked at a third party and never reached this container); hostname `wordpress` unchanged. See [[environment]] / [[services]] for current state. All WordPress/MariaDB references below reflect the pre-migration state as audited on 2026-06-19 and are left as historical record.
|
||||
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No [[services]] route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149). **CT101 update (2026-07-17):** migrated from WordPress/MariaDB to **Grav CMS 2.0.11** (flat-file, no database — MariaDB purged from the container); serves idahomesh.com (not intermountainmesh.com — that domain has always been parked at a third party and never reached this container); hostname `wordpress` unchanged. See [[environment]] / [[services]] for current state. All WordPress/MariaDB references below reflect the pre-migration state as audited on [[2026-06-19]] and are left as historical record.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -60,7 +60,7 @@ Updates where the application or its Docker images have drifted from current ups
|
|||
|
||||
| Scope | Guest | Item | Notes |
|
||||
|-------|-------|------|-------|
|
||||
| edge2 | CT105 | authentik 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
|
||||
| edge2 | CT105 | [[authentik]] 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
|
||||
| edge2 | CT107 | headscale 0.28.0 → 0.29.1 | Also a 2nd headscale on utility CT106 |
|
||||
| edge2 | CT103 | forgejo 14.0.5 → 15.0.3 | **14.x EOL 2026-04-30** — migrate branch, not just patch |
|
||||
| edge2 | CT106 | [[synapse]] 1.155.0 / Element / MAS | Image drift + pending OS apt security updates |
|
||||
|
|
@ -143,15 +143,15 @@ Running application version vs latest stable upstream, per app — the "is every
|
|||
|
||||
### Current / already past the fix (no action)
|
||||
|
||||
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/[[themes]] (edge2 CT101 — **superseded 2026-07-17, see topology note above**) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
|
||||
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/[[themes]] (edge2 CT101 — **superseded 2026-07-17, see topology note above**) · [[synapse]] 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
|
||||
|
||||
### Lower urgency
|
||||
|
||||
Mumble 1.5.517→1.5.901 · Caddy 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · [[searxng]] (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · [[mautrix_signal]] v0.2603.0.
|
||||
Mumble 1.5.517→1.5.901 · [[caddy]] 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · [[searxng]] (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · [[mautrix_signal]] v0.2603.0.
|
||||
|
||||
### Internal echo6 apps (no upstream to track)
|
||||
|
||||
central-*, meshai, archivist, meshwars, [[recon]] / recon-watchdog, navi-* — running; version = current git head.
|
||||
central-*, [[meshai]], [[archivist]], meshwars, [[recon]] / recon-watchdog, navi-* — running; version = current git head.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -237,7 +237,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
|
|||
| wordpress CT101 | dump taken on CT101 | — |
|
||||
| peertube CT110 | DB dump | — |
|
||||
| opentakserver CT109 | PG dump + LVM snapshot | — |
|
||||
| central CT104 | PG dump + snapshot | — |
|
||||
| [[central]] CT104 | PG dump + snapshot | — |
|
||||
| immich CT120 | `/root/immich-predates-20260620.sql` | — |
|
||||
| headscale CT107 | `/root/headscale-db-20260620.sqlite` | — |
|
||||
|
||||
|
|
@ -245,7 +245,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
|
|||
|
||||
### Incidental fixes made during Phase 1
|
||||
|
||||
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN [[dns]] protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and DNS remained healthy after boot.
|
||||
- **(a) CT110 peertube — immutable `/etc/resolv.conf` blocked reboot.** The file had `chattr +i` set (intentional NordVPN [[dns]] protection). Cleared the immutable flag to allow the reboot, then verified the flag was restored and [[dns]] remained healthy after boot.
|
||||
- **(b) CT111 mcc — DNS hijacked to unreachable MagicDNS.** Tailscale `accept-dns` was redirecting DNS to a MagicDNS address that was not reachable from this CT. Disabled `tailscale accept-dns`, set `1.1.1.1` / `8.8.8.8` persistently.
|
||||
- **(c) PostgreSQL on central CT104 moved 16.13→16.14** as part of the security-pocket apt pass.
|
||||
- **(d) Fleet-wide stale `/etc/hosts` fix** — see Critical Finding 2.
|
||||
|
|
@ -327,7 +327,7 @@ EOL-branch migration; v15 schema migrations applied cleanly; web 200, API report
|
|||
|
||||
**OpenTAKServer (utility CT109) 1.7.10 → 1.7.12**
|
||||
|
||||
Clean; all 9 TAK services healthy.
|
||||
Clean; all 9 TAK [[services]] healthy.
|
||||
|
||||
### Key decision — RabbitMQ LEFT on 3.12.1 (EOL)
|
||||
|
||||
|
|
@ -336,7 +336,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
|
|||
### Incidental fixes and side-work during Phase 2
|
||||
|
||||
- **CT107 boot-survival fix** (applied earlier in the effort, during Phase 1 resolution) — rebound headscale/headplane ports to 10.10.10.25, dropped `tailscale-online.target` dependency, disabled `only_start_if_oidc_is_available` gate, repointed edge2 Caddy; proven by reboot self-heal in ~45 s. Also corrected CT107's own tailscale node ControlURL to `vpn.echo6.co` so it self-registers cleanly.
|
||||
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 [[argus]] running hot (transcription + docker-compose build churn). Matt migrated argus to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
|
||||
- **Utility node incident (resolved):** a batch delete of 9 LVM-thin snapshots triggered an SSD TRIM/discard storm that spiked I/O and load transiently; compounded by CT103 [[argus]] running hot (transcription + docker-compose build churn). Matt migrated [[argus]] to the cloud node, resolving the issue; utility load returned to normal. **LESSON: delete thin-pool snapshots one at a time — not in a batch — to avoid the discard storm.**
|
||||
- **Nextcloud:** granted `matt@echo6.co` the NC admin role. (user_oidc has no group-claim sync, so this is durable across SSO logins.)
|
||||
- **Radarr:** set up a `\\192.168.1.160\manual` SMB drop folder on the same NFS export as the library (atomic-move imports) for manual movie filing.
|
||||
- **Snapshot hygiene:** all rollback snapshots cleaned up after validation — Phase 1 `presec-*`, Phase 2 `prewave2-*`, OTS `pre-ots-*` snapshots all removed.
|
||||
|
|
@ -346,7 +346,7 @@ Empirically confirmed during the OTS update: updating OTS to 1.7.12 does **not**
|
|||
**Low-urgency batch — DONE:**
|
||||
- Caddy (utility CT101): 2.10.2 → 2.11.4
|
||||
- CouchDB (edge2 CT104 livesync): 3.4 → 3.5.2 — all 5 DBs intact
|
||||
- valkey-8 sidecar (utility CT102 searxng): bumped to latest 8.x
|
||||
- valkey-8 sidecar (utility CT102 [[searxng]]): bumped to latest 8.x
|
||||
- NATS (utility CT104 central): 2.14.0 → 2.14.2 — all 12 JetStream streams intact
|
||||
- MediaMTX (CT109): 1.13.0 → 1.19.1
|
||||
- Mumble (CT109): already latest in Ubuntu repo (1.5.517 — no upstream action possible without going off-distro)
|
||||
|
|
@ -414,7 +414,7 @@ One at a time; cluster stayed 5/5 quorate throughout.
|
|||
|
||||
- **Guest OS = security-only.** LXC containers and VMs received security-pocket apt updates only (the intentional Phase 1 scope). Non-security package drift (e.g. Docker CE versions, miscellaneous libs) was deliberately not swept with a full `apt full-upgrade`. The PVE hosts, pi-nas, and cortex did receive full upgrades. Operator accepted this state. A full guest `apt full-upgrade` ("Phase 1.5") remains an option if ever wanted.
|
||||
- **Apps capped by external factors (decisions, not failures):** RabbitMQ 3.12 left by decision — OTS depends on it and 4.x would break it; Lidarr v2 — the lidarr-on-steroids image maintainer has not shipped v3, would require an image swap; Jellyseerr on `preview-OIDC` — kept because stable 3.3.0 lacks OIDC/SSO support; Mumble 1.5.517 — newest version in the Ubuntu 24.04 repo, upstream 1.5.901 is not available without going off-distro.
|
||||
- **Deferred project:** Nominatim v5 re-import — spun off to [[nominatim-v5-reimport]].
|
||||
- **Deferred project:** [[Nominatim v5 Re-import]] — spun off to [[nominatim-v5-reimport]].
|
||||
- **Optional cosmetic follow-ups (non-urgent, non-blocking):** navidrome.echo6.co expired unmanaged cert; MediaMTX deprecated config param names (`protocols`/`encryption`); `rpi-eeprom` held on pi-nas; vestigial utility exit-node route (0.0.0.0/0).
|
||||
- **Kernel summary:** all 5 PVE hosts on kernel 7.0.12-1-pve; LXC containers share the host kernel (7.0); VM guests (recon-vm, arr VM105) have their own Ubuntu kernels (security-patched during Phase 1, not necessarily absolute-latest upstream); pi-nas on 6.18.34+rpt-rpi-2712; cortex kernel updated as part of the toc+cortex window.
|
||||
|
||||
|
|
@ -752,7 +752,7 @@ Complete point-in-time state of every node, guest, and container service.
|
|||
| CT103 | forgejo | 14.0.5 (forgejo:14 + postgres:16-alpine drifted) |
|
||||
| CT104 | livesync | couchdb:3.4 drifted + local provisioner |
|
||||
| CT105 | authentik | 2025.12.4 (server/worker/postgres) → upgrade to 2026.2.0 |
|
||||
| CT106 | matrix | Synapse 1.155.0 / Element / MAS + mautrix-signal + postgres; OS apt security updates pending; no Tailscale client |
|
||||
| CT106 | matrix | Synapse 1.155.0 / Element / MAS + [[mautrix_signal]] + postgres; OS apt security updates pending; no Tailscale client |
|
||||
| CT107 | headscale | 0.28.0 → 0.29.0 + headplane; no Tailscale client |
|
||||
| ~~CT108~~ | ~~mailcow~~ | **Decommissioned 2026-06-20** — destroyed (`pct destroy 108 --purge`); superseded by edge1, backup on pi-nas |
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue