echo6-docs/vault/projects/fleet-patch-audit.md

---
title: Fleet Patch Audit — 2026-06-19
type: project
tags:
  - proxmox
  - ai
related: []
updated: 2026-06-19
status: active
---

# Fleet Patch Audit — 2026-06-19

Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this is a planning document to build the patch plan from.**

**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). The `CLAUDE.md` cheat-sheet still lists the old Contabo layout and is **stale** — refreshing it is a follow-up task (see Open Decisions).

---

## Prioritized Backlog

### Tier 1 — Security-Urgent Guest OS

These containers have the highest raw security-update counts and have not been patched recently (or never). Address before any platform work.

| Host | Guest | Upgradable / Security | Notes |
|------|-------|-----------------------|-------|
| utility | CT119 mesh-territory | 179 / 91 sec | Never patched |
| utility | CT108 meshai | 109 / 79 | — |
| cloud | CT120 immich guest-OS | 191 / 101 | — |
| cloud | CT121 nextcloud guest-OS | 98 / 75 | — |
| media | CT110 peertube | 81 / 37 | — |
| utility | CT109 opentakserver | 34 / 31 | — |
| utility | CT104 central | 50 / 38 | Includes PostgreSQL 16.13 → 16.14 |

### Tier 2 — App / Container Updates

Updates where the application or its Docker images have drifted from current upstream, ordered roughly by operational risk.

| Scope | Guest | Item | Notes |
|-------|-------|------|-------|
| edge2 | CT105 | authentik 2025.12.4 → 2026.5.3 | **#1 security item** — 7 CVEs + 5 GHSAs in gap; sequential upgrade (min: 2025.12.6) |
| edge2 | CT107 | headscale 0.28.0 → 0.29.1 | Also a 2nd headscale on utility CT106 |
| edge2 | CT103 | forgejo 14.0.5 → 15.0.3 | **14.x EOL 2026-04-30** — migrate branch, not just patch |
| edge2 | CT106 | Synapse 1.155.0 / Element / MAS | Image drift + pending OS apt security updates |
| edge2 | CT104 | livesync couchdb:3.4 | Docker image drift |
| edge2 | CT108 | mailcow (18 containers) | Upgrade via `update.sh` only |
| cloud | CT120 | immich — server/ml/valkey:9/postgres(14-vectorchord) | 4 images drifted |
| cloud | CT121 | nextcloud AIO — mastercontainer + NC app 32.0.4 | Mastercontainer behind; 12-container stack |
| cortex | VM150 | ollama / tei(1.7) / qdrant / open-webui / obsidian | 5 AI containers drifted |
| media | VM105 | arr stack — 8 containers | jellyfin/sonarr/radarr/prowlarr/sabnzbd/lidarr/navidrome/jellyseerr all :latest |

### Tier 3 — Platform / Reboot Windows

Reboot-bearing updates. Coordinate maintenance windows carefully. cortex and toc are PROTECTED hosts.

| Scope | Item | Detail |
|-------|------|--------|
| data, utility, cloud, media, toc (PVE 9 nodes) | PVE 9.1.1 → 9.2 | Reboot required |
| data, utility, cloud, media, toc | QEMU 10 → 11 | Reboot required |
| data, utility, cloud, media, toc | LXC 6 → 7 | Reboot required |
| data, utility, cloud, media, toc | Kernel 6.17.2 → 6.17.13 | Reboot required |
| edge2 | PVE 8.4.19 | Already fully patched — no action needed |
| cortex VM150 (PROTECTED) | NVIDIA driver 580.159 → 580.167 + DKMS | Reboot required |
| cortex VM150 (PROTECTED) | nvidia-container-toolkit 1.18 → 1.19 | — |
| pi-nas | Kernel 6.12 → 6.18 | Reboot required |
| pi-nas | OMV 8.1 → 8.4 | — |

### Cross-Cutting (All / Most Hosts)

| Item | Detail |
|------|--------|
| Tailscale | 1.94 → 1.98 nearly everywhere |
| Docker CE | → 29.6 wherever Docker is installed |

---

## Non-Update Flags

Issues noted that are not package/image updates but warrant attention.

| Host / Guest | Flag | Detail |
|---|---|---|
| data | Disk 92% full | ~73 GB / 938 GB free; address before patching |
| utility CT118 archivist | rpcbind on 0.0.0.0:111 | No Tailscale client or firewall on this CT; exposed port |
| media VM105 jellyseerr | Non-stable image | Running preview-OIDC tag, not a stable release |
| data VM1130 nominatim | Stale image (14 months) | nominatim:4.5, pinned; confirm intentional |
| edge2 CT106 matrix / CT107 headscale | No Tailscale client | Ingress via Caddy; verify internal routing before patching |

---

## Application Currency (2026-06-19)

Running application version vs latest stable upstream, per app — the "is everything up to date?" view (distinct from OS-package/image-drift above). Read-only. **Caveat:** version deltas were measured directly from the running apps and are reliable; the specific CVE/advisory IDs below were surfaced by the audit agents and are **post-knowledge-cutoff — verify against primary advisories before acting on them.** GitHub's anonymous API rate-limited a few "latest" lookups (noted inline).

### Security-relevant — prioritize

| App | Where | Running → Latest | Behind | Security note (verify) |
|-----|-------|------------------|--------|------------------------|
| **Authentik** | edge2 CT105 | 2025.12.4 → **2026.5.3** | ~6 mo / 5 majors | **#1** — claimed 7 CVEs + 5 GHSAs across two May-2026 waves. Min-disruption: 2025.12.6 (same branch, backported fixes); full: 2026.5.3. SSO — upgrade sequentially. |
| **Immich** | cloud CT120 | 2.5.6 → **2.7.5** | 9 rel | shared-link ACL bypass (2.6.0) + stored XSS panorama viewer (2.7.0) |
| **Valkey** | cloud CT120 | 9.0.2 → **9.1.0** | 3 patch | 6 CVEs, "SECURITY" urgency (use-after-free, DoS, RESP injection) |
| **RabbitMQ** | utility CT109 | 3.12.1 → **4.2.8** | EOL major | 3.x abandoned upstream; 4 advisories 2026-06-18, no 3.x backports |
| **Nominatim** | data/recon-vm | 4.5.0 → **5.3.2** | ~19 mo, major | v4→v5 data-model break; needs image swap + full re-import (pair w/ Photon 1.2.0) |
| **Ollama** | cortex | 0.16.1 → **0.30.10** | 14 minor | history of SSRF / path-traversal CVEs |
| **Nextcloud** | cloud CT121 | 32.0.4 → **32.0.11**; AIO ~v12.5 → **13.2.1** | 7 patch / ~8 AIO | resource-exhaustion fix (32.0.10); AIO image ~5 mo stale |
| **Forgejo** | edge2 CT103 | 14.0.5 → **15.0.3** | EOL branch | 14.x EOL 2026-04-30; no further backports — migrate to 15.0.3 (LTS) |
| **SABnzbd** | media VM105 | 4.5.5 → **5.0.4** | major | 5.x security hardening over 4.x |
| **Jellyfin** | media VM105 | 10.11.6 → **10.11.11** | 5 patch | 10.11.x patch cycle has carried CVE fixes |
| **Jellyseerr** | media VM105 | **preview-OIDC dev** → 3.3.0 | unreleased | running unpinned dev code — no stable security posture |
| **PeerTube** | media CT110 | 8.0.2 → **8.2.1** | 2 patch | 8.x patch releases include security fixes |
| **Open-WebUI** | cortex | 0.8.1 → **0.9.6** | ~1.5 minor | touches auth/session surface |
| **MediaMTX** | utility CT109 | 1.13.0 → **1.19.1** | 6 minor | RTSP/WebRTC streaming attack surface |
| **Headscale** ×2 | utility CT106 + edge2 CT107 | 0.28.0 → **0.29.1** | 1 minor | upgrade-guide-required; two separate instances; no security flag noted |

### Operational / host-level (not a simple app bump)

| Item | Where | Finding |
|------|-------|---------|
| **Mailcow** | edge2 CT108 | **STOPPED** — very likely superseded by the new **edge1 mail-only** node. Confirm and decommission rather than update. |
| **Host kernel** | **edge2 host** | Agent flagged DirtyFrag (CVE-2026-43284/-43500) + copy.fail (CVE-2026-31431, claimed CISA KEV) as host-kernel LPE. **Tension:** the host audit showed edge2 fully patched (0 upgradable) on its repo — **verify** whether these need a kernel newer than the no-subscription repo provides. |

### Current / already past the fix (no action)

Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/themes (edge2 CT101) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).

### Lower urgency

Mumble 1.5.517→1.5.901 · Caddy 2.10.2/2.11.3→2.11.4 · Qdrant 1.16.3→1.18.2 · TEI 1.7.4→1.9.3 · Valhalla 3.6.3→3.7.0 · Photon 1.1.0→1.2.0 · kiwix 3.7.0→3.8.2 · CouchDB 3.4.3→3.5.2 (livesync) · Navidrome 0.60.3→0.62.0 · Sonarr/Radarr/Prowlarr/Lidarr 1–2 versions · NATS 2.14.0→2.14.2 · PostgreSQL 16.12/16.13→16.14 · meshmonitor (~1 mo, exact ver undeterminable) · searxng (rolling, ~4.5 mo) + valkey-8 sidecar 8.1.5→8.1.8 · mautrix-signal v0.2603.0.

### Internal echo6 apps (no upstream to track)

central-*, meshai, archivist, meshwars, recon / recon-watchdog, navi-* — running; version = current git head.

---

## Proposed Patch Approach (NOT executed — for planning)

Lowest-risk changes first; everything reboot-bearing deferred to scheduled windows.

**Principles**
- **Security-pocket apt only** in the first pass (openssl, openssh, gnutls, krb5, libc6, samba, nghttp2, etc.). No PVE 9.2 / QEMU / LXC / kernel, no Docker image pulls, no OMV/NVIDIA, no reboots.
- **Protected hosts (cortex, toc) never go in a bulk pass** — handled individually in their own window. Note: toc hosts cortex (VM150), so a toc reboot drops cortex — the two must be coordinated together.
- **edge1 (mail)** excluded while it's mid-rebuild.
- `needrestart` will bounce affected daemons after glibc/openssl upgrades — seconds of blip per guest, no data risk.

**Phased plan**

| Phase | Scope | Reboot? | Notes |
|------|-------|---------|-------|
| **1 — Guest/VM security apt** | utility CT100,101,102,103,104,106,107,108,109,112,118,119 · cloud CT120,121 · media VM105,CT110,CT111 · data VM1130 · edge2 CT100–108 | No | Lowest blast radius. Worst-first: CT119, CT108, immich/nextcloud guest-OS, peertube. |
| **2 — Hypervisor host OS security** | data, utility, cloud, media host OSes (**not toc**) | No | One node at a time; restarts hypervisor-side daemons (smbd etc.). edge2 host already patched. |
| **3 — App / container updates (Tier 2)** | per-app, native updater each | Per-app | See "special handling" below — not a generic `docker pull`. |
| **4 — Reboot windows (Tier 3)** | PVE 9.2 + QEMU 11 + LXC 7 + kernel on the 5 PVE-9 nodes; pi-nas kernel + OMV; cortex NVIDIA/DKMS | **Yes** | Schedule deliberately; toc+cortex coordinated. |
| **Cross-cutting** | Tailscale 1.94→1.98 fleet-wide | No | Can ride along Phase 1/2. |

**Special handling — do NOT bulk-patch these; use the native updater**
- **mailcow** (edge2 CT108) → `./update.sh` (upgrades all 18 components in lockstep)
- **nextcloud AIO** (cloud CT121) → update mastercontainer, then in-UI update button (port 8080)
- **immich** (cloud CT120) → `docker compose pull && up -d` from its compose dir
- **authentik** (edge2 CT105) → sequential version upgrades with migrations; 2025.12.4 → 2026.2.0 cannot skip releases
- **pi-nas OMV** → OMV's own update path, not raw apt
- **cortex / toc** → manual, protected, own window

---

## Open Decisions (for tomorrow's plan)

1. **Phase 1 scope** — all non-protected guests at once, or staged worst-first?
2. **Phase 2** — patch the four hypervisor host OSes now (no reboot), or fold into the Phase 4 reboot window?
3. **Reboot-window scheduling** — order of the 5 PVE-9 nodes; **toc + cortex must be done together** (toc reboot drops cortex). edge2 needs none.
4. **Tier-2 app upgrades** — which to take on: authentik major (2025→2026, migration-heavy), Matrix/Synapse, Forgejo, headscale 0.29, mailcow. Each is its own task.
5. **data disk at 92%** — remediate before/independently of patching (operational risk regardless).
6. **CT118 archivist** rpcbind on `0.0.0.0:111` with no Tailscale/firewall — treat as a separate exposure fix.
7. **Refresh the stale `CLAUDE.md` cheat-sheet** to the edge1/edge2 topology — separate doc task.
8. **Daemon-restart tolerance** — confirm brief blips are acceptable for the stateful services (central PG16/NATS, opentakserver, peertube) during Phase 1/2.
9. **edge2 CT108 mailcow is stopped** — confirm it's superseded by the edge1 mail node and decommission it, vs. it being an unintended outage.
10. **edge2 host-kernel CVEs (DirtyFrag / copy.fail)** — verify whether the flagged in-the-wild LPEs actually apply given the host shows fully patched; if real, this is a host-kernel + reboot action on edge2 (which otherwise needs none).
11. **App-currency CVE IDs are post-cutoff** — verify the specific advisories (Authentik waves, Valkey, Immich, RabbitMQ) against primary sources before using them to justify urgency.

---

## Execution Runbook (Meticulous)

Step-by-step plan to bring every application and package current. Per-app target versions live in **Application Currency** above; this section is the *how/when/order*. Execution model: scoped Sonnet task per target → Opus verifies output → report to Matt; **approval gate before every load-bearing or reboot step.**

### Standing guardrails (apply to EVERY step)

- **Protected hosts — cortex & toc — never in a bulk pass.** They get a dedicated, manual window. Note the coupling: **toc reboot drops cortex (VM150), which is the Claude Code host** — expect to lose the control session during that window; drive it from elsewhere or accept the outage.
- **One target at a time.** Verify health before moving to the next.
- **Snapshot/backup before each mutating step:** `pct snapshot` / `qm snapshot` (or `vzdump`) the guest first; DB dump before any DB-bearing upgrade; run the app's own backup where it has one (Nextcloud AIO, mailcow).
- **No host reboots outside the explicit Phase 3 windows.** Security upgrades that set `reboot-required` are applied but activation deferred to Phase 3.
- **Rollback = restore the snapshot / redeploy the previous image tag.** Record before/after version in the tracking table.
- Every result must survive a reboot (standing infra policy).

### Load-bearing dependencies (drive the ordering)

- **Authentik (edge2 CT105) = SSO.** Upgrading it briefly breaks login to everything behind it. Confirm **break-glass admin** access first; do in a low-traffic window; verify dependent-service logins after.
- **Caddy (utility CT101) = home-services ingress.** A restart blips all home services — fold its update into a deliberate moment, not mid-day.
- **Headscale ×2 (edge2 CT107 main + utility CT106 mesh) = tailnet coordination.** Existing tunnels usually persist a restart, but edge2 is the front door and you may be managing *over* Tailscale — back up the DB and verify nodes stay joined.
- **Nominatim 4→5 is NOT a routine update** — it's a re-import project (see Phase 2C).

### Phase 0 — Pre-flight (no app changes yet)

1. **Free space on `data`** (92% full) so snapshots/image pulls have room.
2. **Resolve edge2 CT108 mailcow** — confirm it's superseded by edge1; if so, back up then stop/destroy → removes it from scope. If it's an *unintended* outage, that's a separate incident.
3. **Verify break-glass:** local admin creds for Authentik-protected apps; PVE console access independent of the services being changed.
4. **Confirm snapshot headroom** on each PVE node's storage.
5. **Resolve the edge2 host-kernel CVE question** (DirtyFrag/copy.fail) → decides whether edge2 needs a Phase 3 reboot after all.
6. **Freeze the baseline** — this doc's versions are the pre-change record.

### Phase 1 — OS security packages (no reboot)

Guests first (low blast radius), then hypervisor hosts. Per target: `apt-get update` → snapshot → apply **security-pocket** upgrades → `needrestart` to bounce affected daemons → verify service health and that the security-upgradable count hits 0.

- **Worst-first guests:** utility CT119 (91 sec), CT108 (79), cloud CT120/CT121 guest-OS (101/75), media CT110 (37), utility CT109 (31), CT104 (38, incl. PG 16.13→16.14).
- **Remaining guests:** all other utility/cloud/media/edge2 CTs + data VM1130 (recon-vm).
- **Hosts (one at a time):** data, utility, cloud, media. **edge2 host already patched. toc excluded** (Phase 3, with cortex).
- Kernel/libc security that flags `reboot-required` → applied, reboot deferred to Phase 3.

### Phase 2 — Application updates

**2A — Self-managed updaters** (snapshot → run native updater → verify):
- **OpenTAKServer (utility CT109)** — update per OTS upgrade procedure; **this is the agreed fix for the EOL RabbitMQ** plus MediaMTX/Mumble currency. Snapshot first; verify TAK clients reconnect.
- **Immich (cloud CT120)** — `docker compose pull && up -d` in its dir (also clears Valkey 9.0.2→9.1.0). Verify web, mobile sync, ML.
- **Nextcloud AIO (cloud CT121)** — run AIO backup → update mastercontainer → trigger child update via AIO UI (:8080). Verify `occ status`, apps.

**2B — Versioned apps** (snapshot → bump/upgrade → verify):
- **Authentik (edge2 CT105)** — **sequential, no skipping:** 2025.12.4 → **2025.12.6** (backported security, lowest risk) → 2026.2.x → **2026.5.3**, running DB migrations at each hop. SSO window + break-glass. Verify dependent logins after each hop.
- **Forgejo (edge2 CT103)** — **branch migration 14.x → 15.0.3**, not a patch: read 15.0 breaking changes, back up repos + DB, bump tag, verify.
- **Headscale ×2 (edge2 CT107, utility CT106)** — follow the 0.28→0.29 upgrade guide; back up DB; **one at a time**; verify nodes stay connected.
- **Media stack (media VM105)** — `docker compose pull && up -d` for jellyfin / sonarr / radarr / prowlarr / navidrome; **read release notes for the majors** — SABnzbd 4→5 and Lidarr 2→3; **move jellyseerr off the `preview-OIDC` dev tag to stable 3.3.0.** Snapshot VM first; verify each UI.
- **PeerTube (media CT110)** — native install: back up DB → run PeerTube's upgrade script 8.0.2→8.2.1 → verify.
- **cortex AI stack (PROTECTED — manual, own window):** ollama 0.16→0.30 (14 versions — check model compat), open-webui 0.8→0.9, qdrant, tei. Pull images, verify.
- **Lower-urgency batch:** caddy, couchdb (livesync), valhalla, photon, kiwix, navidrome, NATS, valkey-8 sidecar — snapshot + update as convenient.
- Vaultwarden, PDM, WordPress, Synapse/Element/MAS, obsidian, recon-vm Postgres → **already current, skip.**

**2C — Separate project (not routine):**
- **Nominatim 4.5 → 5.3 (recon-vm)** — requires the `mediagis/nominatim:5.x` image + **full OSM re-import**, coordinated with **Photon 1.1→1.2** (1.2 reads the v5 data format). Schedule as its own effort with disk/time budget.

### Phase 3 — Platform & reboot windows (coordinated, approval-gated)

- **PVE-9 nodes** (data, utility, cloud, media, toc): PVE 9.1.1→9.2, QEMU 10→11, LXC 6→7, kernel 6.17.2→6.17.13. Per node: snapshot/backup guests → `apt full-upgrade` the proxmox stack → reboot → verify all guests return. **One node at a time.** Validate on a low-stakes node (data or media) before utility/cloud.
- **toc + cortex = the special window, done LAST and together:** toc gets PVE 9.2 + reboot; cortex gets NVIDIA 580.159→580.167 + DKMS + container-toolkit 1.18→1.19 + apt + reboot. **The Claude Code host goes down here** — run this step from a different control point.
- **pi-nas** (standalone window): kernel 6.12→6.18 + OMV 8.1→8.4 (via OMV's update path) → reboot.
- **edge2**: reboot only if Phase 0 step 5 confirmed it needs a newer kernel; otherwise none.

### Phase 4 — Cross-cutting (fold into earlier phases)

- **Tailscale 1.94→1.98** fleet-wide — rides along Phase 1.
- **Docker CE →29.6** wherever installed — rides along Phase 1/2 under the same snapshots.

### Tracking checklist (tick per target)

### Enumerated Target Checklist

| Phase | Target | Action | Snapshot | Update method | Verify | Rollback | Depends-on |
|---|---|---|---|---|---|---|---|
| 0 | Break-glass / OOB access matrix (ALL nodes) | Verify non-Tailscale, non-Caddy path to every node + each lockout-critical CT (headscale, Authentik CT105, Caddy, Forgejo CT103); confirm PVE/Proxmox web auth is PAM/local not Authentik | None (read-only) | Log into Contabo/PVE console; `pct enter` 105/106/107; confirm LAN `192.168.1.x` SSH; locate break-glass creds in `.ref/credentials`; TEST Authentik akadmin + each app local-admin fallback | Every node + CT shell reachable WITHOUT Authentik/headscale/Caddy; akadmin login screenshotted | N/A | FIRST Phase 0 step; gates everything mutating |
| 0 | Per-node free space + snapshot backend | `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2; record backend (LVM-thin/ZFS/dir-qcow2) + failure mode; size vs largest guests | None | Read-only; set numeric rule: never snapshot below ~20% free, require ≥2× write-delta | Free GB recorded per pool; cloud node measured; vzdump target located | N/A | Precedes all snapshot-bearing steps |
| 0 | `data` host — disk remediation (92% full, ~73 GB) | Free space sized to largest planned snapshot (VM1130) | None (cleanup; confirm deletions are cache/backup not live) | Prune stale vzdump/snapshots/ISO; `docker image prune` on recon-vm (keep `nominatim:4.5`) | Usage <~85% on snapshot-backing storage; test snapshot succeeds then removed | Restore from Forge/Syncthing/backup if a needed file removed | Free-space capture; BLOCKS data host step + canary use |
| 0 | Off-host restorable backups (stateful guests) | Verify last-good vzdump/PBS off the node for central, OTS, forgejo, matrix, nextcloud, edge2 livesync CouchDB | N/A | Read-only verification + on-demand `pg_dump`/app backup copied off-host | ≥1 backup off the changing node, restore-testable | N/A | Precedes all mutating stateful steps |
| 0 | Baseline HEALTH capture (all targets) | Record up/down, container states, endpoint 200s, `apt upgradable`/security counts; note oddities | None | `pct/qm status`, `docker ps`, curl probes, `apt list --upgradable` | Baseline recorded; oddities logged (jellyseerr dev tag, CT108 stopped, nominatim pin, CT118 rpcbind) | N/A | Precedes Phase 1 |
| 0 | Decision gates | Close Open Dec #8 (needrestart tolerance); verify post-cutoff CVE claims vs primary advisories w/ owner+URL; resolve edge2 kernel question; reconcile CT106-vs-CT107 headscale location; define+announce Boise windows | None | Read-only / sign-off | Each decision recorded before Phase 1/2/3 can start | N/A | Gates Phase 1/2/3 |
| 0 | edge2 CT108 mailcow (STOPPED) | Confirm superseded by edge1, back up, decommission | `pct snapshot 108 predecommission` + vzdump + mailcow native backup | Confirm edge1 mail live + no MX at edge2; `pct stop`/`pct destroy 108` ON APPROVAL | CT108 gone/archived; edge1 mail in+out works; vzdump restorable | `pct restore 108` + start; re-point MX | edge1 confirmed; approval (Open Dec #9) |
| 0 | edge2 host kernel CVE question | Decide if DirtyFrag/copy.fail require a reboot | None (record `pveversion -v`) | Compare installed proxmox-kernel vs verified-advisory fixed versions; escalate if no-sub repo lacks fix (no repo changes w/o approval) | Decision (reboot yes/no) recorded | N/A | Gates edge2 Phase 3 row |
| 1 | Procedure canary — utility CT112 cobalt (idle) or CT102 searxng | Prove apt→snapshot→security upgrade→needrestart→verify on low-stakes guest | `pct snapshot` pre-phase1 | `apt-get update`; security pocket only; `needrestart -r l` then deliberate | Security-upgradable=0; service healthy; needrestart clear | `pct rollback` | Phase 0; runs BEFORE worst-first guests |
| 1 | utility CT119 mesh-territory (179/91, never patched) | Security-pocket apt | `pct snapshot 119` | `pct exec`; security pocket; `needrestart -r a` | sec count=0; meshwars up | rollback snapshot | After canary |
| 1 | utility CT108 meshai (109/79) | Security-pocket apt | `pct snapshot 108` | `pct exec`; security pocket; needrestart | sec=0; meshai up | rollback snapshot | After canary |
| 1 | utility CT104 central (PG16 16.13→16.14 + NATS/JetStream) | Security apt incl. PG minor; STAGED, controlled bounce | `pct snapshot 104` + `pg_dumpall` | `NEEDRESTART_MODE=l`; stage pkgs; quiesce JetStream publishers; bounce PG then NATS deliberately | sec=0; `select version()`=16.14; JetStream/MQTT consumers reconnected | rollback snapshot; restore dump | Phase 0 #8; stateful carve-out |
| 1 | utility CT109 opentakserver (OS layer, 31 sec) | Security apt; STAGED, do NOT touch RabbitMQ/MediaMTX/Mumble here | `pct snapshot 109` | `NEEDRESTART_MODE=l`; stage; deliberate bounce | sec=0; TAK clients reconnect | rollback snapshot | Precedes CT109 Phase 2 app work |
| 1 | media CT110 peertube (81/37, native PG16+redis) | Security apt; STAGED controlled bounce | `pct snapshot 110` (PG rides in CT snap) | `NEEDRESTART_MODE=l`; stop import/transcode jobs; snapshot redis; deliberate bounce | sec=0; PeerTube plays; PG16/redis/nginx up; nordvpn up | rollback snapshot | Phase 0 #8 |
| 1 | data VM1130 recon-vm (34/10, PG16+navi-backend) | Security apt; STAGED controlled bounce | `qm snapshot 1130 pre-phase1-apt` | `NEEDRESTART_MODE=l`; deliberate PG + navi-backend bounce | sec=0; psql overture/padus; navi-backend/recon.py/photon/kiwix up | `qm rollback` | data disk remediation done first |
| 1 | cloud CT120 immich (191/101) | Guest security apt; Tailscale 1.94→1.98 + Docker CE→29.6 ride along | `pct snapshot 120 pre-phase1-os` | `pct exec`; security pocket; docker bounce acceptable | sec=0; immich stack Up; web reachable; TS=1.98 | `pct rollback` | Before cloud host step + CT120 app upgrade |
| 1 | cloud CT121 nextcloud AIO (98/75) | Guest security apt; TS+Docker ride along | `pct snapshot 121 pre-phase1-os` | `pct exec`; security pocket | sec=0; 12 AIO containers Up; web reachable | `pct rollback` | Before cloud host step + AIO app upgrade |
| 1 | media VM105 arr (43/9) | Guest security apt; TS+Docker ride along | `qm snapshot 105 pre-os-sec` | `apt`; security pocket; needrestart | sec=0; 8 containers Up; Samba + all UIs load | `qm rollback` | Before VM105 Phase 2 apps |
| 1 | media CT111 mcc (51/13, local caddy+postfix) | Guest security apt | `pct snapshot 111 pre-os-sec` | `pct exec`; security pocket | sec=0; caddy+postfix active | `pct rollback` | Independent (no Phase 2 app) |
| 1 | utility CT100/101/102/103/106/107/112/118 (OS layer) | Security-pocket apt per CT; lockout-critical (Caddy CT101, headscale CT106) carved to LIST mode + LAST; CT118 leave rpcbind alone | `pct snapshot` each + headscale DB backup for CT106 | `pct exec`; security pocket; `needrestart -r l` for control-plane CTs | sec=0 each; ingress spot-check; `headscale nodes list` joined | `pct rollback` | Phase 0; control-plane CTs last as singletons |
| 1 | edge2 CT100-105 (OS layer, via `pct exec` from edge2 host) | Security-pocket apt per CT; DB dumps for CT101 WP MariaDB, CT104 CouchDB, CT105/CT106 PG | `pct snapshot` each pre-phase1 + DB dumps | From `root@184.174.35.153` `pct exec`; security pocket; defer reboot | sec=0; each app reachable via Caddy front door | `pct rollback`; restore DB dump | Phase 0; edge2 host fully patched (host = no-op) |
| 1 | edge2 CT106 matrix / CT107 headscale (NO Tailscale client) | Security apt driven via `pct exec` from edge2 HOST only; singletons | `pct snapshot` + Synapse PG dump (106) + headscale DB backup (107) | From edge2 host shell; `needrestart -r l`; do NOT restart headscaled/synapse unless required | sec=0; matrix federation+login; `headscale nodes list` all joined; Caddy ingress works | `pct rollback`; restore DB | OOB path confirmed first; NOT in edge2 batch |
| 1 | cortex VM150 (PROTECTED, 90 upgradable) | Security apt; manual in toc+cortex window; TS+Docker ride along | `qm snapshot 150 pre-sec-apt` (from toc) | `apt --only-upgrade` security pkgs; `needrestart -r a`; defer reboot | sec=0; 5 AI containers Up; `nvidia-smi` ok | `qm rollback` | Phase 0; protected — not bulk |
| 1 | pi-nas host OS (subset of 130) | Debian security pocket; HOLD kernel 6.12→6.18 + OMV pkgs | Off-box `config.xml` + `dpkg --get-selections` baseline | `apt-mark hold linux-image-* openmediavault*`; `-t trixie-security upgrade`; unhold; needrestart | sec=0; shares mount; OMV UI loads; no reboot now | Reinstall prior pkg from cache; restore config.xml | Phase 0; precedes pi-nas kernel reboot |
| 1 | Cross-cutting Tailscale 1.94→1.98 (all TS-client nodes) | Bump client; rides along Phase 1; EXCLUDE edge2 CT106/CT107 | Covered by per-target Phase 1 snapshot | `apt install --only-upgrade tailscale`; daemon restart | `tailscale version`=1.98; node still joined | Reinstall 1.94; snapshot rollback | Folds into Phase 1; LAN/console fallback confirmed |
| 1 | Cross-cutting Docker CE→29.6 (Docker hosts) | Bump engine; rides along Phase 1/2 | Covered by per-target snapshot | `apt install --only-upgrade docker-ce docker-ce-cli containerd.io`; daemon restart | `docker version`=29.6; containers return | Downgrade pkg; snapshot rollback | Folds into Phase 1; before compose pulls |
| 2 | edge2 CT105 Authentik 2025.12.4→2025.12.6 (hop 1) | SSO hop #1 (FIRST load-bearing app); pre-flight: dedup group names, `/media`→`/data/media` if local storage | `pct snapshot 105` + `pg_dump` (per-hop) | Edit tag; `compose pull && up -d`; run migrations; NO skipping | UI=12.6; migrations clean; dependent login works | `compose down`; restore tag+dump; or `pct rollback` | Phase 0 break-glass proven; window |
| 2 | edge2 CT105 Authentik 2025.12.6→2026.2.x (hop 2) | SSO hop #2 cross-major | `pct snapshot` + fresh `pg_dump` | tag→2026.2.x; pull/up; migrations | UI=2026.2.x; logins re-verified | restore prior tag+dump | after hop 1 |
| 2 | edge2 CT105 Authentik 2026.2.x→2026.5.3 (hop 3, final) | SSO hop #3; verify CVE/GHSA IDs; insert any mandatory intermediate | `pct snapshot` + fresh `pg_dump` | tag→2026.5.3; pull/up; final migrations | UI=2026.5.3; full sweep of dependent logins; close window | restore prior tag+dump | after hop 2; verify dependents BEFORE upgrading them |
| 2 | utility CT101 + media CT111 Caddy (ingress, EARLY) | Caddy bump (before home-app verification); map Caddy↔Authentik path; document direct Authentik admin URL | `pct snapshot` + Caddyfile/certs backup | `caddy validate`; restart in low-traffic window | home services reachable; certs valid | rollback snapshot/Caddyfile | After CT101 Phase 1; separate window from Authentik |
| 2 | edge2 CT103 Forgejo 14.0.5→15.0.3 (branch migration) | Major migration; pre: off-edge2 `git clone --mirror` echo6-docs + pause autocommit cron + `forgejo doctor check --all --fix` | `pct snapshot 103` + repos volume + `pg_dump` + app.ini | `doctor check`→repair→backup→bump tag 14→15; pull/up; migrations | reports 15.0.3; clone/push works; CI/webhooks ok | restore 14.0.5 image + volume + dump | After CT103 Phase 1; NOT same window as Authentik |
| 2 | utility CT109 OTS app | OTS updater (app+webUI+schema) — does NOT migrate RabbitMQ | `pct snapshot 109` + PG dump | Run OTS upgrade script | OTS web/API up; CoT works; clients reconnect | rollback snapshot + dump | After CT109 Phase 1 OS |
| 2 | utility CT109 RabbitMQ 3.12.1→4.2.8 (SEPARATE) | Decoupled EOL major: Erlang≥26→3.13.x→`enable_feature_flag all`→4.x | `pct snapshot` + `rabbitmqctl export_definitions` | staged hops; confirm flags enabled; no classic-mirroring config | 4.2.x; queues intact; TAK clients reconnect | restore snapshot + definitions | GATES MediaMTX/Mumble accept; after OTS app |
| 2 | utility CT106 + edge2 CT107 Headscale 0.28→0.29 | Lower-stakes instance FIRST (rehearsal), front-door LAST; disable key-expiry; OOB-driven | `pct snapshot` + headscale DB dump off-tailnet | stop→backup→replace binary→config migrate→start; headplane to match | version=0.29.1; `nodes list` all joined; tunnels intact | restore snapshot+DB via console | one-at-a-time; non-tailnet path; not same window as Caddy/Authentik/Forgejo |
| 2 | cloud CT120 Immich 2.5.6→2.7.5 (+Valkey 9.0.2→9.1.0, vectorchord PG) | Stack upgrade; prune images first; do NOT interrupt first-boot migration; disable bg regen | `pct snapshot 120 pre-immich-2.7.5` + `pg_dump` | `compose pull && up -d`; let migrations run; no skip across breaking migration | web+login; jobs process; valkey 9.1.0 ping; version 2.7.5 | restore tag (only with dump) or `pct rollback` | After CT120 Phase 1; free space confirmed |
| 2 | cloud CT121 Nextcloud AIO (master ~12.5→13.2.1, NC 32.0.4→32.0.11) | AIO orchestrated update; borg backup as rollback artifact | AIO borg backup + `pct snapshot 121` | AIO backup→stop→update mastercontainer→UI :8080 "Update containers" | 12 containers green; `occ status` 32.0.11; integrity clean | restore from AIO borg; else `pct rollback` | After CT121 Phase 1; AIO self-gates intermediate versions |
| 2 | media CT110 PeerTube 8.0.2→8.2.1 (native) | Native upgrade script; quiesce jobs | `pct snapshot 110` + `pg_dump peertube_prod` | back up DB → run PeerTube `upgrade.sh`/version steps → restart | reports 8.2.1; video plays; PG16/redis/nginx healthy | `pct rollback` + restore dump | After CT110 Phase 1 |
| 2 | media VM105 jellyfin 10.11.6→10.11.11 | Patch bump | `qm snapshot 105` (pre-app batch) | `compose pull jellyfin && up -d` (pin tag) | UI=10.11.11; plays/transcodes | redeploy prior digest; `qm rollback` | After VM105 Phase 1 |
| 2 | media VM105 SABnzbd 4.5.5→5.0.4 (MAJOR) | Major; pause queue; audit post-proc scripts | `qm snapshot 105 pre-sabnzbd` + `sabnzbd.ini` | pin 5.0.4; pull/up; migrate on start | UI=5.0.4; queue intact; test NZB; arr integrations ok | redeploy 4.5.5 + restore ini (queue repair); `qm rollback` | After VM105 Phase 1; before Lidarr |
| 2 | media VM105 Lidarr 2.x→3.x (MAJOR) | Major branch migration | `qm snapshot 105 pre-lidarr` + `lidarr.db` | pin v3; pull/up; DB migration on start | UI=v3; library+indexers+download client ok | restore db (migration one-way); `qm rollback` | After SABnzbd (verify download-client link) |
| 2 | media VM105 Sonarr/Radarr/Prowlarr (1-2 ver) | Routine bumps | pre-app VM snapshot + config DBs | `compose pull && up -d` | UIs load; prowlarr sync; test grab | redeploy prior tags + DBs | After SABnzbd/Lidarr |
| 2 | media VM105 Navidrome 0.60.3→0.62.0 | Routine bump | pre-app snapshot + DB | `compose pull navidrome && up -d` | UI=0.62.0; library scans; track streams | redeploy 0.60.3 + DB | Independent of arr chain |
| 2 | media VM105 jellyseerr preview-OIDC→stable 3.3.0 | DECISION GATE: schema downgrade + correct artifact (jellyseerr vs Seerr) + OIDC parity | `qm snapshot 105 pre-jellyseerr` + config dir | test stable vs DB COPY first; swap tag only if compatible | UI=3.3.0; requests intact; OIDC/login works | restore preview tag + config; `qm rollback` | After jellyfin/arr; STOP+report if OIDC parity unclear |
| 2 | edge2 CT104 livesync CouchDB 3.4→3.5.2 | Lower-urgency image bump | `pct snapshot 104` + CouchDB volume backup | bump tag; `compose pull && up -d` | reports 3.5.2; `_up` healthy; LiveSync replicates test edit | restore 3.4 + volume; `pct rollback` | After CT104 Phase 1; triple-sync verify |
| 2 | utility CT104 NATS 2.14.0→2.14.2 | Lower-urgency patch | `pct snapshot 104` (JetStream dir in snap) | replace binary; restart | 2.14.2; JetStream/MQTT intact | `pct rollback` | After CT104 Phase 1 (PG already done); stateful blip |
| 2 | utility CT100 meshmonitor / CT102 searxng+valkey-8 (8.1.5→8.1.8) | Lower-urgency image refresh | `pct snapshot` + record digests | `compose pull && up -d` | containers Up; function ok | redeploy prior digest; `pct rollback` | After respective Phase 1 |
| 2 | edge2 CT100 PDM / CT101 WP core / CT102 Vaultwarden / CT106 Synapse | Verify-current / no-op (already at/past fix); WP plugin/theme = separate WP-CLI follow-up | None (Phase 1 snaps) | optional digest re-pull only | versions confirmed; logged as current | redeploy prior digest | Closes as no-op |
| 2 | cortex Ollama 0.16.1→0.30.10 (PROTECTED) | Image bump; no downgrade after new pull | `qm snapshot 150 pre-ollama` + models volume | `compose pull ollama && up -d`; test inference | `/api/version`=0.30.10; models intact; GPU used; vault-tagger works | re-pin 0.16.1; restore snapshot if model format broke | After cortex Phase 1 + Docker + toolkit |
| 2 | cortex Open-WebUI 0.8.1→0.9.6 | Image bump (auth surface) | `qm snapshot 150 pre-openwebui` | bump tag; pull/up | UI loads; login works; reaches Ollama | re-pin 0.8.1; snapshot | After Ollama |
| 2 | cortex Qdrant 1.16.3→1.18.2 / TEI 1.7.4→1.9.3 | Lower-urgency image bumps | `qm snapshot 150` per app | bump tag; pull/up | health endpoints ok; collections/embeddings intact; docs engine works | re-pin prior; snapshot | After cortex Docker+toolkit; verify engines |
| 2 | cortex obsidian-remote v1.12.7 | Verify-current / no-op | None | none | container up; UI reachable | N/A | — |
| 2 | utility CT118 archivist rpcbind 0.0.0.0:111 | Remediate exposure (network change — APPROVAL) | `pct snapshot 118` | per approved option: disable rpcbind / bind localhost+TS / firewall | port 111 not on 0.0.0.0; app functions; external scan closed | re-enable / `pct rollback` | After CT118 Phase 1; approval-gated (Open Dec #6) |
| 2 | edge2 CT108 mailcow `./update.sh` — CONDITIONAL | Only if RETAINED; else SKIP | `pct snapshot 108` + mailcow backup | `./update.sh` (native lockstep) | 18 containers up; mail in/out | `./update.sh restore`; `pct rollback` | GATED on Phase 0 decision (likely skipped) |
| 3 | Per-node corosync/HA pre-flight (data/utility/cloud/media/toc) | Before EACH reboot | None | `pvecm status` (Quorate:Yes, 5 votes); `ha-manager status` | quorate + expected votes=5; HA implications known | abort if not quorate | Gates each Phase 3 node reboot |
| 3 | Canary node (lowest blast + headroom; media, NOT data until disk freed) | Full 9.2/QEMU11/LXC7/kernel reboot to prove the path | vzdump all guests to EXTERNAL target + record `pveversion -v` | `apt update && apt dist-upgrade` → reboot; NO live-migration in mixed window | `pveversion`=9.2/QEMU11/LXC7/kernel 6.17.13; all guests `onboot` return; quorate | boot prior kernel (GRUB); restore guests from vzdump | Phase 1+2 done; canary before high-stakes nodes |
| 3 | media host PVE 9.1.1→9.2 (+QEMU11/LXC7/kernel) | Platform + reboot | vzdump VM105/CT110/CT111 + snapshots | `dist-upgrade` → reboot | guests return; arr/peertube/caddy healthy; quorate | GRUB prior kernel; vzdump restore | After media Phase 1+2; one node at a time |
| 3 | cloud host PVE 9.2 platform + reboot | Platform + reboot | vzdump CT120/CT121 + record versions | `dist-upgrade` → reboot | immich+nextcloud return healthy; quorate | GRUB prior kernel; vzdump restore | After cloud Phase 1+2; after canary proven |
| 3 | utility host PVE 9.2 platform + reboot | Platform + reboot (known full home-ingress + mesh + central outage) | vzdump all 12 guests + versions | `dist-upgrade` → reboot | all 12 return; Caddy ingress + headscale + central back; quorate | GRUB prior kernel; vzdump restore | LAST of early nodes; low-traffic window |
| 3 | data host PVE 9.2 platform + reboot | Platform + reboot | vzdump VM1130 + versions | `dist-upgrade` → reboot | VM1130 returns; NFS/Samba serve; quorate | GRUB prior kernel; vzdump restore | After disk remediation; one at a time |
| 3 | Hypervisor host OS-security (folded in) | Apply held host libc/openssl WITH the platform full-upgrade per node | covered by per-node vzdump | included in `dist-upgrade` (not applied back in Phase 1) | sec=0 host-side post-reboot; daemons up | per-node rollback | Fold-in decision (Open Dec #2) |
| 3 | Post-reboot app re-verification (per node) | Re-run Phase-2 health checks for majored guests on each rebooted node | None | health probes | CT120/CT121/CT110/VM105/CT109 healthy on new substrate | re-snapshot/redeploy degraded component | After each node reboot |
| 3 | cortex nvidia-container-toolkit 1.18→1.19 | Toolkit bump (pair w/ driver) | `qm snapshot 150 pre-nvtoolkit` | `apt --only-upgrade nvidia-container-toolkit`; `nvidia-ctk runtime configure`; restart docker | `--version`=1.19; `docker run --gpus all nvidia-smi` ok | downgrade 1.18; snapshot | toc+cortex window |
| 3 | cortex NVIDIA driver 580.159→580.167 + DKMS (reboot) | Driver + DKMS as discrete reversible step BEFORE toc reboot | `qm snapshot 150 pre-nvidia` + vzdump | `apt --only-upgrade nvidia-driver-580 nvidia-dkms-580`; DKMS rebuild; verify BEFORE toc reboot | `nvidia-smi`=580.167; `dkms status` installed; containers see GPU | restore snapshot; reinstall 580.159 + DKMS | Keep old driver pkg; after toolkit |
| 3 | toc host PVE 9.2 + QEMU11 + LXC7 + kernel (reboot — LAST, drops cortex) | Platform + reboot; only guest is VM150 | vzdump/snapshot VM150 + record `pveversion` | snapshot VM150 → `dist-upgrade` toc → reboot → VM150 returns → cortex driver reboot | `pveversion`=9.2; toc rejoins (5/5 votes); VM150 + GPU healthy | restore VM150 from vzdump; GRUB prior kernel | LAST + alone; alternate control point TESTED; all 4 other nodes quorate |
| 3 | cortex final stack health verification | Post-window full AI-stack + engine check | keep pre-window snaps until verified | verification only | driver 580.167; all 5 containers; ollama/tei/qdrant/open-webui; vault-tagger+TEI engines ok; TS online; Docker 29.6 | restore degraded component; worst case VM150 vzdump | LAST action of node group |
| 3 | pi-nas OMV 8.1→8.4 (reboot #1, separate window) | OMV native update path (NOT raw apt) → reboot | off-box `config.xml` export + plugin/layout record | OMV UI Update Mgmt or `omv-upgrade`; reboot | OMV=8.4; shares/SMB/NFS/RAID/mergerfs healthy; client mounts | restore config.xml + `dpkg --set-selections` 8.1 + `omv-salt deploy` | Physical access; not mid-sync; after Phase 1 |
| 3 | pi-nas kernel 6.12→6.18 (reboot #2, separate window) | Kernel bump → reboot; activates deferred security | back up `/boot`+`/boot/firmware`; keep 6.12 installed; fresh config.xml | `apt install linux-image-arm64` (unhold); update bootloader; reboot ON APPROVAL | `uname -r`=6.18; LAN+tailnet up; shares mount; Docker up | boot retained 6.12; restore `/boot`; on-site SD reflash | After OMV reboot verified; physical access |
| 3 | edge2 host PVE 8.4 reboot — CONDITIONAL | Only if Phase 0 confirmed kernel fix needed; own no-failover window AFTER CT105/CT107 verified | vzdump all guests + `pveversion` | scoped kernel/security upgrade → reboot in approved window | `uname -r`=fixed; all CTs return; Caddy-fronted services reachable | GRUB prior kernel; vzdump restore | GATED on Phase 0; announce SSO+tailnet downtime; non-tailnet path |
| — | Nominatim 4.5→5.3 + Photon 1.1→1.2 (SEPARATE PROJECT) | Full re-import into separate DB/instance; keep 4.5 until validated; Photon AFTER nominatim validated | `qm snapshot 1130` + retain 4.5 image/data | import to new DB; size flat-nodes vs free space (NOT on data); cut over after verify | 5.x geocode correct; Photon 1.2 reads v5 | re-pin nominatim:4.5 + old data | Wholly separate from patch campaign; after disk remediation |

---

## Plan Hardening (from review)

The review surfaced corrections across six dimensions. They are merged and de-duplicated below, critical/high first, grouped by phase, with hosts/CTs named. Several issues recur because the same architectural fact (one corosync cluster; control plane runs over the tailnet; `data` is 92% full; SSO + ingress + git + tailnet are mutually entangled) drives multiple failure modes — the consolidated corrections address the root cause once.

### Cluster & control-plane facts that the whole plan must respect
- **The five PVE nodes are ONE corosync cluster (`echo6-cluster`), not standalone hypervisors.** Confirmed in `vault/docs/hardware/environment.md` (line 17) and `vault/runbooks/proxmox-onboard-node.md`. Quorum = 3 of 5. The plan never mentions corosync/quorum. **Every Phase 3 node reboot must be gated on `pvecm status` → `Quorate: Yes`, expected votes = 5; reboot ONE node at a time; wait for full rejoin (5/5) before the next.** Losing quorum makes `/etc/pve` read-only fleet-wide and freezes all VM/CT start/stop/config on every surviving node. Also run `ha-manager status` first — if any guest is HA-managed, a reboot triggers fencing/auto-migration, not a clean local stop/start.
- **toc is the 5th cluster vote AND the GPU host for cortex (VM150) AND cortex is the documented cluster "Management host" + the Claude Code control host.** Rebooting toc removes a vote and kills the box you'd diagnose from. **toc+cortex must be the final cluster action and must never overlap any other node reboot** (toc down + a second node down = bare 3/5; one corosync flap loses quorum).
- **Reconcile the headscale location contradiction BEFORE touching any of it.** The audit places headscale at utility CT106 *and* edge2 CT107 (and calls CT106 both "meshtastic-hs" and "headscale control plane"); the vault docs (`headscale-onboard-node.md`) describe headscale as a Docker container on Contabo (`100.64.0.1`) reached over the VPN. You cannot protect a control plane you have mislocated. Pin the real instance(s) and their reachability first; this gates every headscale step below.

### Phase 0 — preflight (promote these to hard gates; some must move to first)
- **(MOVE TO FIRST) Verify a non-Tailscale, non-Caddy break-glass path to every node** — Contabo/Proxmox console for edge2, LAN `192.168.1.x` for the home nodes, PVE noVNC into each CT — and confirm PVE/Proxmox web auth is PAM/local, NOT behind Authentik. Losing the management path mid-run is the single highest-consequence failure on this fleet. This must pass before anything mutating.
- **Measure, don't hand-wave, snapshot headroom.** Run `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2. Record the storage backend per node (LVM-thin/ZFS/dir-qcow2) and its over-provision failure mode. Set a hard numeric rule: never start a snapshot below ~20% free; require ≥2× the largest guest's expected write-delta. `data` at 92% (~73 GB free) cannot safely snapshot VM1130 recon-vm (nominatim/overture/padus), and **cloud node free space (CT120 immich, CT121 nextcloud AIO) is not measured at all** — add it. Identify where vzdump lands; if it's `data`, freeing `data` is a fleet-wide rollback prerequisite, not a data-node nicety.
- **`data` disk remediation is a blocking gate, sized from measurement.** Identify consumers (old vzdump, snapshots, ISO/template store, recon-vm dangling Docker layers — do NOT remove the pinned `nominatim:4.5` image), confirm nothing deleted is the only copy of a backup or live NAS data, and size cleanup to the largest planned snapshot on the node. Must complete before `data`'s own host step and before it is used as a canary.
- **Verify off-host restorable backups exist** (distinct from per-step snapshots) for every stateful guest: central PG/NATS, opentakserver PG + RabbitMQ definitions, forgejo PG + repos, matrix Synapse PG, nextcloud AIO (borg), edge2 livesync CouchDB. At least one copy must live off the node being changed.
- **Capture a pre-change baseline HEALTH snapshot** (per-target up/down, `docker ps`/`pct`/`qm status`, key endpoint 200 checks, and `apt list --upgradable`/security counts) so "verify after" has a comparison and known oddities are recorded: jellyseerr on the preview-OIDC dev tag, mailcow CT108 already STOPPED, pinned-stale nominatim, CT118 archivist rpcbind on `0.0.0.0:111`.
- **Close Open Decision #8 as a gate:** explicit sign-off on needrestart-driven daemon bounces for central (CT104 PG16/NATS/JetStream), opentakserver (CT109), peertube (CT110), matrix Synapse, recon-vm VM1130 PG16 + navi-backend, and the auth/tailnet daemons — or set those guests `NEEDRESTART_MODE=l` (list-only) so the operator controls the bounce.
- **Verify the load-bearing post-cutoff CVE claims against primary advisories before they drive ordering** (Authentik May-2026 waves justifying the mandatory 2025.12.6 hop; Valkey 9.1.0; Immich 2.6/2.7; RabbitMQ 4.x; edge2 DirtyFrag CVE-2026-43284/-43500 and copy.fail CVE-2026-31431). Assign an owner + primary-source URL per claim. **Resolve the edge2 kernel question fully here** (pin `uname -r`/`proxmox-boot-tool kernel list`, check repo kernel availability) so edge2 is either firmly scheduled for its own no-failover reboot window or firmly excluded — not left conditional inside Phase 3.
- **Decommission decision for mailcow CT108** (STOPPED, superseded by edge1): vzdump + mailcow native backup, confirm edge1 is handling mail and no MX points at edge2, then `pct destroy` only on approval. Removes CT108 from all later scope.
- **Define and announce maintenance windows in America/Boise**, naming user-facing blips: Authentik SSO (the upgrade "briefly breaks login to everything"), all home services behind Caddy, matrix CT106, media VM105, immich/nextcloud/peertube.

### Phase 1 — guest OS security sweep (ordering + needrestart policy)
- **Prove the procedure on a low-stakes guest first** (utility CT112 cobalt idle, or CT102 searxng) before the never-patched worst-first targets (CT119 "never patched", 179 upgradable / 91 sec; CT108 meshai). Keep worst-first for security urgency only AFTER the procedure is proven; snapshot each never-patched guest immediately before its first-ever security upgrade.
- **Carve the stateful guests out of the bulk needrestart pass** (CT104 central, CT109 opentakserver, CT110 peertube, VM1130 recon-vm): set `NEEDRESTART_MODE=l`, stage the packages, take a logical DB dump (`pg_dump`/JetStream snapshot) with the app quiesced, then bounce each service deliberately in a controlled order. A JetStream restart mid-write loses messages on an at-most-once pipeline; a RabbitMQ/redis bounce mid-job is not a clean "blip."
- **Carve the lockout-critical daemons out too** (Authentik CT105, the headscale instance(s), Caddy, and sshd on any host you're connected over). Run `needrestart -r l` to see what *would* restart and bounce deliberately, so a libc/openssl bump can't auto-bounce the auth/tailnet/SSH daemons out from under the operator mid-pass.
- **Order control-plane guests LAST as singletons** — the home-ingress Caddy and the headscale instance(s) — each with DB backup + node-stays-joined verification, never inside a batch. Before touching headscale, confirm the out-of-band path (LAN SSH / PVE console via `pct exec`) to every still-queued target.
- **edge2 CT106 matrix / CT107 headscale have NO Tailscale client (Caddy-only ingress).** Drive their security-apt step via `pct exec` from the edge2 HOST (`root@184.174.35.153`), never over service ingress, so a needrestart bounce of a network/resolver daemon can't strand the only path in. Treat them as their own singleton steps, not part of the edge2 batch.
- **Gate `data`'s host step on the disk remediation being verified DONE** (its snapshot guardrail needs the space).

### Phase 1 hosts vs Phase 3 (sequencing)
- **Fold the four cluster hosts' OS-security apt into their own Phase 3 reboot window** rather than applying host libc/openssl in Phase 1 and living mixed-state (new libc / old kernel, bounced smbd) across the entire Phase 2 app campaign. Guests still get Phase-1 security immediately; each host transitions in one clean window. (Open Decision #2 — this is the sequencing-correct answer.)

### Phase 2 — app upgrades (sequence so a failure is isolatable, and never co-schedule recovery paths)
- **Hard rule: {Headscale, Authentik, Caddy, Forgejo} are NEVER in the same maintenance window** so at least one of {tailnet, SSO, ingress, git} is always a known-good recovery path.
- **Authentik first** among load-bearing apps (it fronts Forgejo, Vaultwarden, WordPress, Nextcloud, Matrix). Prove break-glass with evidence first: log in with the local akadmin in a private session; confirm each protected app has a working local-admin fallback (Forgejo `admin user`, Nextcloud `occ`, PDM/Vaultwarden local). The chain `2025.12.4 → 2025.12.6 → 2026.2.x → 2026.5.3` runs irreversible Django migrations at each hop — **`pg_dump` per hop (labelled), one hop at a time, verified-login gate between each, abort+restore on first migration error.** Add the 2025.12 hard pre-flight: audit/rename duplicate group names (migration "fails loudly" on dupes), and if local `/media` storage is used, stop and `mv ./media ./data/media` + rewrite the compose volume before starting the new version. Verify all dependent SSO logins BEFORE upgrading the dependents.
- **Caddy early** (utility ingress + media CT111), before home-service app verification, so backend sign-offs run through the new ingress; pair CT101/CT111 in one deliberate window. Map the Caddy↔Authentik path first and document a direct (Caddy-bypassing) Authentik admin URL as break-glass.
- **OpenTAKServer CT109 — decouple RabbitMQ from the OTS bump.** The OTS updater only upgrades OTS + webUI + schema; it does NOT migrate RabbitMQ, Erlang, or feature flags (verified against docs.opentakserver.io/installation/upgrading.html), so the most security-urgent EOL item (RabbitMQ 3.12.1, no 3.x backports) would silently go unfixed. Write RabbitMQ as its own sub-task: snapshot + `export_definitions` → Erlang ≥26 → 3.12 → 3.13.x → `rabbitmqctl enable_feature_flag all` (confirm all enabled) → only then 4.x; confirm no classic-queue-mirroring config remains. A naive 3.12→4.x jump refuses to boot. Gate the OTS step on RabbitMQ 4.x up + TAK clients reconnecting before accepting the bundled MediaMTX/Mumble bumps.
- **Forgejo CT103 14→15** is a branch migration on an EOL branch (14.x EOL 2026-04-30) with forward-only FK migrations. Run `forgejo doctor check --all [--fix]` to repair stopwatch/tracked_time inconsistencies BEFORE the backup+bump. Pre-step: `git clone --mirror` echo6-docs (and other critical repos) to an OFF-edge2 location and pause the `echo6-docs-autocommit` cron so the recovery docs survive an edge2/Forgejo failure and the cron doesn't loop-error into a half-migrated instance.
- **Headscale: upgrade the lower-stakes instance FIRST as a rehearsal, the front-door LAST**, each driven from a path that does NOT depend on the instance being upgraded (PVE console / direct edge2-host → CT internal IP). Before the front-door hop: disable/extend node key-expiry on all nodes so persistent tunnels don't drop on re-handshake mid-migration; `pct snapshot` + dump the headscale DB off-tailnet; keep a second already-authenticated SSH session open. Rollback restores the snapshot via console, not via tailnet. Verify headplane compatibility with 0.29.
- **Immich CT120 / Nextcloud AIO CT121 — method is correct, but size disk and protect the DB.** Prune old images + `docker image prune` before the snapshot; do NOT interrupt Immich's first-boot 2.5→2.7 DB migration; AIO self-gates intermediate mastercontainer versions (a stale AIO may need two cycles) — use AIO's borg backup as the rollback artifact (mastercontainer downgrades unsupported). The rollback unit for every DB-bearing app is a **quiesced logical dump + old binary together** — "redeploy previous image tag" is NOT valid after forward-only migrations, and a `pct snapshot` of a hot, separate-volume DB can restore torn. Disable Immich background regeneration until free space is confirmed.
- **jellyseerr is a data-compatibility decision, not a retag.** preview-OIDC (dev) → stable 3.3.0 is a schema DOWNGRADE; TypeORM stable may refuse to start against a dev-migrated DB. Confirm the correct destination artifact (jellyseerr stable vs the unified "Seerr" successor) and, if OIDC is in active use, confirm 3.3.0 parity — else STOP and report. Test the stable image against a COPY of the DB first.
- **cortex ollama 0.16→0.30:** reframe — old local blobs read fine; the real trap is no clean downgrade after pulling any new model. Snapshot the models volume (the only rollback), upgrade, verify existing models `ollama run` without re-pull. Verify the vault-tagger engine (localhost:11434) and TEI `related:` engine (localhost:8090) still work after.
- **SABnzbd 4.5→5.0:** pause/empty the queue, audit custom post-proc scripts (scripts now run on failed jobs), note `empty_postproc` removed and that downgrade needs a queue repair.
- **Nominatim 4→5 + Photon 1.1→1.2 is a separate project, not part of the patch campaign.** Re-import into a SEPARATE DB/instance; keep the `nominatim:4.5` image + data until 5.x is validated; size transient import disk (flat-nodes tens of GB) against real free space on a non-starved node — this cannot live on the 92%-full `data` pool; sequence Photon 1.2 AFTER nominatim 5.x is validated, never concurrently.

### Phase 3 — platform & reboot windows
- **Per-node order: `apt update && apt dist-upgrade` (full-upgrade so proxmox-ve/qemu/lxc metapackages pull) → reboot → verify `pveversion` + all guests return.** Hard rule for the mixed-version window: **NO live migration and disable HA migration across the QEMU 10/11 (PVE 9.1/9.2) boundary** — running VMs keep QEMU 10 machine type until cold-started, and migration across the boundary fails. Optionally bump VM machine types + cold-restart after all 5 are on 9.2 (noted follow-up).
- **Canary by blast radius + snapshot headroom, not label.** `data` (92% full, hosts recon-vm overture/padus) must NOT be canary until its disk is remediated, and its near-full pool can fail the mandatory pre-reboot snapshot. Lowest-stakes is **media** (single arr VM + peertube, non-auth/non-DB-critical) — but note media carries freshly-majored apps; the reconciled order is: canary the genuinely lowest-blast node with headroom → then **utility LAST among the early nodes** (it carries Caddy ingress + the mesh/central stack — a deliberately-scheduled full home-ingress + mesh-coordination outage) → cloud (immich/nextcloud) → data → **toc+cortex last and alone**, all four other nodes confirmed quorate.
- **Pre-window: confirm every guest has `onboot=1`** so they actually return; take vzdump of guests to an EXTERNAL target (not the local pool, never `data`) so a guest that won't start under LXC 7 can be restored elsewhere. Read PVE 9.2 + LXC 7 release notes for unprivileged-CT/cgroupv2/apparmor breaking changes before the first node.
- **Add a post-Phase-3 re-verification gate:** after each node boots into 9.2/LXC7, re-run the Phase-2 health check for every guest that received a major app upgrade on that node (CT120 immich, CT121 nextcloud, CT110 peertube, VM105 arr, CT109 OTS) — the substrate changed under them.
- **toc+cortex window:** name and TEST a concrete alternate control point (a non-cortex host with keys + tooling to the fleet) BEFORE the window; stabilize the tailnet well before it (never in the same period as a headscale upgrade). Split cortex: snapshot VM150 from toc → NVIDIA 580.159→580.167 + DKMS + container-toolkit 1.19 as a discrete reversible step, confirm `dkms status` built and `nvidia-smi` BEFORE rebooting toc → reboot toc → confirm toc rejoins (5/5 votes) and cortex + GPU passthrough return. Keep the old driver package for reinstall.
- **edge2 is PVE 8.4, architecturally separate, single SPOF for ingress (no failover).** Do NOT fold it into the cluster windows. If the Phase 0 kernel verification says it needs a reboot, give it a dedicated window AFTER Authentik CT105 + headscale CT107 are upgraded and verified, announce SSO+tailnet downtime, drive from a non-tailnet path, and confirm all CTs auto-start.
- **pi-nas: split into two reboots, not one.** OMV 8.1→8.4 via OMV's own update path (NOT raw `apt full-upgrade`) → reboot → verify shares/SMB/NFS/omv-salt; THEN kernel 6.12→6.18 → reboot → verify boot + disk remount. Confirm physical/serial console access (no remote KVM on an RPi), back up `/boot`+`/boot/firmware` and keep the old kernel installed as fallback, export `config.xml` off-box, and confirm it's not mid-sync as a Syncthing/backup target before taking it down. Independent of the cluster windows.

---

## Full Inventory

Complete point-in-time state of every node, guest, and container service.

### data (PVE 9.1.1)

- Host: 126 upgradable / 51 security; no Docker installed; roles: NAS, NFS, Samba
- **VM1130 recon-vm** (Ubuntu 24.04) — 34 upgradable / 10 security
  - PostgreSQL 16.14 (DBs: overture, padus), photon, kiwix, recon.py, 7x navi-backend, nginx, Apache, Samba
  - Docker: valhalla:latest, nominatim:4.5 (stale 14 months), zimit:latest (not running)

### utility (PVE 9.1.1)

- Host: 148 upgradable / 38 security; PVE 9.2 platform update pending; 12 LXC guests

| CT | Name | Upgradable / Sec | Services |
|----|------|-----------------|----------|
| CT100 | meshmonitor | 62 / 15 | ghcr.io/yeraze/meshmonitor:latest |
| CT101 | caddy (home ingress) | 59 / 18 | — |
| CT102 | searxng | 54 / 15 | searxng/searxng:latest + valkey/valkey:8-alpine |
| CT103 | argus | 29 / 26 | RF capture / transcribe / viewer |
| CT104 | central | 50 / 38 | PostgreSQL 16 + NATS/MQTT/JetStream |
| CT106 | meshtastic-hs | 54 / 17 | headscale control plane |
| CT107 | mesh-bridge | 53 / 15 | dual tailscaled |
| CT108 | meshai | 109 / 79 | work-meshai local build |
| CT109 | opentakserver | 34 / 31 | nginx / PG16 / rabbitmq / mumble / mediamtx / CoT |
| CT112 | cobalt | 35 / 29 | build/CI (idle) |
| CT118 | archivist | 62 / 17 | archivist + rpcbind (FLAG: port 111 on 0.0.0.0) |
| CT119 | mesh-territory | 179 / 91 | meshwars:latest (never patched) |

### cloud (PVE 9.1.1)

- Host: 113 upgradable / 38 security; 2 LXC guests

| CT | Name | Upgradable / Sec | Services |
|----|------|-----------------|----------|
| CT120 | immich | 191 / 101 | immich_server, immich_machine_learning, valkey/valkey:9, immich postgres (14-vectorchord) — all drifted |
| CT121 | nextcloud AIO | 98 / 75 | 12 containers: mastercontainer + apache + nextcloud + postgresql + redis + collabora + clamav + imaginary + fulltextsearch + notify-push + whiteboard + docker-socket-proxy; NC 32.0.4; mastercontainer behind |

### media (PVE 9.1.1)

- Host: 107 upgradable / 36 security; 3 guests

| Guest | Name | Upgradable / Sec | Services |
|-------|------|-----------------|----------|
| VM105 | arr | 43 / 9 (Ubuntu 24.04) | jellyfin / sonarr / radarr / prowlarr / sabnzbd / lidarr / navidrome / jellyseerr (preview-OIDC) all :latest + Samba |
| CT110 | peertube | 81 / 37 | v8.0.2; nginx / PG16 / redis / peertube / pt-downloader + importer + monitor / nordvpn |
| CT111 | mcc | 51 / 13 | caddy + postfix |

### toc (PVE 9.1.1) — PROTECTED

- Host: 189 upgradable / 43 security; PVE 9.2 platform update pending; reboot required
- Hosts only VM150 cortex — coordinate any toc work with cortex maintenance window

### cortex (VM150, Ubuntu 24.04) — PROTECTED GPU / Claude Code host

- 90 apt upgradable
- NVIDIA driver 580.159 → 580.167 + DKMS (reboot required)
- nvidia-container-toolkit 1.18 → 1.19
- Docker: ollama / tei 1.7 / qdrant / open-webui / obsidian — all drifted

### pi-nas (Debian 13, arm64, RPi + OMV)

- 130 apt upgradable; kernel 6.12 → 6.18 (reboot required); OMV 8.1 → 8.4
- Docker engine installed; 0 containers running

### edge2 (PVE 8.4.19) — Host Fully Patched

- 9 LXC guests

| CT | Name | Services / Status |
|----|------|-------------------|
| CT100 | pdm | PDM 1.1.4, current (native) |
| CT101 | wordpress | Apache 2.4.67 / PHP 8.4 / MariaDB 11.8.6 / WP core 7.0 — plugin/theme status needs WP-CLI |
| CT102 | vaultwarden | vaultwarden/server:latest — drift indeterminate |
| CT103 | forgejo | 14.0.5 (forgejo:14 + postgres:16-alpine drifted) |
| CT104 | livesync | couchdb:3.4 drifted + local provisioner |
| CT105 | authentik | 2025.12.4 (server/worker/postgres) → upgrade to 2026.2.0 |
| CT106 | matrix | Synapse 1.155.0 / Element / MAS + mautrix-signal + postgres; OS apt security updates pending; no Tailscale client |
| CT107 | headscale | 0.28.0 → 0.29.0 + headplane; no Tailscale client |
| CT108 | mailcow | 18 containers (postfix/nginx/dovecot/rspamd/clamd/sogo/php-fpm/mariadb/redis/memcached/unbound/acme/olefy/watchdog/netfilter/ofelia/dockerapi/tlspol) — upgrade via `update.sh` only |

---

## Coverage Notes

- **Complete:** all five PVE-9 nodes (data, utility, cloud, media, toc), edge2, cortex, pi-nas, and all discovered guests/containers.
- **Excluded:** edge1 (the rebuilt ex-Contabo, mail-only) — mid-rebuild/maintenance at time of audit; re-audit once it settles.
- **Incomplete:** WordPress (CT101) plugin and theme status — requires WP-CLI; not assessed.
- **Method:** read-only throughout — SSH / `pct exec`, `apt list --upgradable`, `docker manifest inspect` for same-tag drift. Floating/pinned-tag caveats noted inline where drift could not be confirmed.