Read-only audit snapshot as of 2026-06-19. **Nothing has been applied — this is a planning document to build the patch plan from.**
**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). The `CLAUDE.md` cheat-sheet still lists the old Contabo layout and is **stale** — refreshing it is a follow-up task (see Open Decisions).
---
## Prioritized Backlog
### Tier 1 — Security-Urgent Guest OS
These containers have the highest raw security-update counts and have not been patched recently (or never). Address before any platform work.
Running application version vs latest stable upstream, per app — the "is everything up to date?" view (distinct from OS-package/image-drift above). Read-only. **Caveat:** version deltas were measured directly from the running apps and are reliable; the specific CVE/advisory IDs below were surfaced by the audit agents and are **post-knowledge-cutoff — verify against primary advisories before acting on them.** GitHub's anonymous API rate-limited a few "latest" lookups (noted inline).
| **Headscale**×2 | utility CT106 + edge2 CT107 | 0.28.0 → **0.29.1** | 1 minor | upgrade-guide-required; two separate instances; no security flag noted |
### Operational / host-level (not a simple app bump)
| Item | Where | Finding |
|------|-------|---------|
| **Mailcow** | edge2 CT108 | **STOPPED** — very likely superseded by the new **edge1 mail-only** node. Confirm and decommission rather than update. |
| **Host kernel** | **edge2 host** | Agent flagged DirtyFrag (CVE-2026-43284/-43500) + copy.fail (CVE-2026-31431, claimed CISA KEV) as host-kernel LPE. **Tension:** the host audit showed edge2 fully patched (0 upgradable) on its repo — **verify** whether these need a kernel newer than the no-subscription repo provides. |
### Current / already past the fix (no action)
Vaultwarden 1.36.0 (edge2 CT102 — has the SSO-takeover/org-access CVE fixes) · PDM 1.1.4 (edge2 CT100 — past the RCE PSA) · WordPress 7.0 core + all plugins/themes (edge2 CT101) · Synapse 1.155.0 / Element / MAS (edge2 CT106 — current, only minor `:latest` digest drift) · obsidian-remote v1.12.7 (cortex) · PostgreSQL 16.14 (recon-vm).
## Proposed Patch Approach (NOT executed — for planning)
Lowest-risk changes first; everything reboot-bearing deferred to scheduled windows.
**Principles**
- **Security-pocket apt only** in the first pass (openssl, openssh, gnutls, krb5, libc6, samba, nghttp2, etc.). No PVE 9.2 / QEMU / LXC / kernel, no Docker image pulls, no OMV/NVIDIA, no reboots.
- **Protected hosts (cortex, toc) never go in a bulk pass** — handled individually in their own window. Note: toc hosts cortex (VM150), so a toc reboot drops cortex — the two must be coordinated together.
- **edge1 (mail)** excluded while it's mid-rebuild.
-`needrestart` will bounce affected daemons after glibc/openssl upgrades — seconds of blip per guest, no data risk.
- **immich** (cloud CT120) → `docker compose pull && up -d` from its compose dir
- **authentik** (edge2 CT105) → sequential version upgrades with migrations; 2025.12.4 → 2026.2.0 cannot skip releases
- **pi-nas OMV** → OMV's own update path, not raw apt
- **cortex / toc** → manual, protected, own window
---
## Open Decisions (for tomorrow's plan)
1.**Phase 1 scope** — all non-protected guests at once, or staged worst-first?
2.**Phase 2** — patch the four hypervisor host OSes now (no reboot), or fold into the Phase 4 reboot window?
3.**Reboot-window scheduling** — order of the 5 PVE-9 nodes; **toc + cortex must be done together** (toc reboot drops cortex). edge2 needs none.
4.**Tier-2 app upgrades** — which to take on: authentik major (2025→2026, migration-heavy), Matrix/Synapse, Forgejo, headscale 0.29, mailcow. Each is its own task.
5.**data disk at 92%** — remediate before/independently of patching (operational risk regardless).
6.**CT118 archivist** rpcbind on `0.0.0.0:111` with no Tailscale/firewall — treat as a separate exposure fix.
7.**Refresh the stale `CLAUDE.md` cheat-sheet** to the edge1/edge2 topology — separate doc task.
8.**Daemon-restart tolerance** — confirm brief blips are acceptable for the stateful services (central PG16/NATS, opentakserver, peertube) during Phase 1/2.
9.**edge2 CT108 mailcow is stopped** — confirm it's superseded by the edge1 mail node and decommission it, vs. it being an unintended outage.
10.**edge2 host-kernel CVEs (DirtyFrag / copy.fail)** — verify whether the flagged in-the-wild LPEs actually apply given the host shows fully patched; if real, this is a host-kernel + reboot action on edge2 (which otherwise needs none).
11.**App-currency CVE IDs are post-cutoff** — verify the specific advisories (Authentik waves, Valkey, Immich, RabbitMQ) against primary sources before using them to justify urgency.
---
## Execution Runbook (Meticulous)
Step-by-step plan to bring every application and package current. Per-app target versions live in **Application Currency** above; this section is the *how/when/order*. Execution model: scoped Sonnet task per target → Opus verifies output → report to Matt; **approval gate before every load-bearing or reboot step.**
### Standing guardrails (apply to EVERY step)
- **Protected hosts — cortex & toc — never in a bulk pass.** They get a dedicated, manual window. Note the coupling: **toc reboot drops cortex (VM150), which is the Claude Code host** — expect to lose the control session during that window; drive it from elsewhere or accept the outage.
- **One target at a time.** Verify health before moving to the next.
- **Snapshot/backup before each mutating step:** `pct snapshot` / `qm snapshot` (or `vzdump`) the guest first; DB dump before any DB-bearing upgrade; run the app's own backup where it has one (Nextcloud AIO, mailcow).
- **No host reboots outside the explicit Phase 3 windows.** Security upgrades that set `reboot-required` are applied but activation deferred to Phase 3.
- **Rollback = restore the snapshot / redeploy the previous image tag.** Record before/after version in the tracking table.
- Every result must survive a reboot (standing infra policy).
### Load-bearing dependencies (drive the ordering)
- **Authentik (edge2 CT105) = SSO.** Upgrading it briefly breaks login to everything behind it. Confirm **break-glass admin** access first; do in a low-traffic window; verify dependent-service logins after.
- **Caddy (utility CT101) = home-services ingress.** A restart blips all home services — fold its update into a deliberate moment, not mid-day.
- **Headscale ×2 (edge2 CT107 main + utility CT106 mesh) = tailnet coordination.** Existing tunnels usually persist a restart, but edge2 is the front door and you may be managing *over* Tailscale — back up the DB and verify nodes stay joined.
- **Nominatim 4→5 is NOT a routine update** — it's a re-import project (see Phase 2C).
### Phase 0 — Pre-flight (no app changes yet)
1.**Free space on `data`** (92% full) so snapshots/image pulls have room.
2.**Resolve edge2 CT108 mailcow** — confirm it's superseded by edge1; if so, back up then stop/destroy → removes it from scope. If it's an *unintended* outage, that's a separate incident.
3.**Verify break-glass:** local admin creds for Authentik-protected apps; PVE console access independent of the services being changed.
4.**Confirm snapshot headroom** on each PVE node's storage.
5.**Resolve the edge2 host-kernel CVE question** (DirtyFrag/copy.fail) → decides whether edge2 needs a Phase 3 reboot after all.
6.**Freeze the baseline** — this doc's versions are the pre-change record.
### Phase 1 — OS security packages (no reboot)
Guests first (low blast radius), then hypervisor hosts. Per target: `apt-get update` → snapshot → apply **security-pocket** upgrades → `needrestart` to bounce affected daemons → verify service health and that the security-upgradable count hits 0.
- **OpenTAKServer (utility CT109)** — update per OTS upgrade procedure; **this is the agreed fix for the EOL RabbitMQ** plus MediaMTX/Mumble currency. Snapshot first; verify TAK clients reconnect.
- **Immich (cloud CT120)** — `docker compose pull && up -d` in its dir (also clears Valkey 9.0.2→9.1.0). Verify web, mobile sync, ML.
- **Authentik (edge2 CT105)** — **sequential, no skipping:** 2025.12.4 → **2025.12.6** (backported security, lowest risk) → 2026.2.x → **2026.5.3**, running DB migrations at each hop. SSO window + break-glass. Verify dependent logins after each hop.
- **Forgejo (edge2 CT103)** — **branch migration 14.x → 15.0.3**, not a patch: read 15.0 breaking changes, back up repos + DB, bump tag, verify.
- **Headscale ×2 (edge2 CT107, utility CT106)** — follow the 0.28→0.29 upgrade guide; back up DB; **one at a time**; verify nodes stay connected.
- **Media stack (media VM105)** — `docker compose pull && up -d` for jellyfin / sonarr / radarr / prowlarr / navidrome; **read release notes for the majors** — SABnzbd 4→5 and Lidarr 2→3; **move jellyseerr off the `preview-OIDC` dev tag to stable 3.3.0.** Snapshot VM first; verify each UI.
- **PeerTube (media CT110)** — native install: back up DB → run PeerTube's upgrade script 8.0.2→8.2.1 → verify.
- **cortex AI stack (PROTECTED — manual, own window):** ollama 0.16→0.30 (14 versions — check model compat), open-webui 0.8→0.9, qdrant, tei. Pull images, verify.
- **Nominatim 4.5 → 5.3 (recon-vm)** — requires the `mediagis/nominatim:5.x` image + **full OSM re-import**, coordinated with **Photon 1.1→1.2** (1.2 reads the v5 data format). Schedule as its own effort with disk/time budget.
### Phase 3 — Platform & reboot windows (coordinated, approval-gated)
- **PVE-9 nodes** (data, utility, cloud, media, toc): PVE 9.1.1→9.2, QEMU 10→11, LXC 6→7, kernel 6.17.2→6.17.13. Per node: snapshot/backup guests → `apt full-upgrade` the proxmox stack → reboot → verify all guests return. **One node at a time.** Validate on a low-stakes node (data or media) before utility/cloud.
- **toc + cortex = the special window, done LAST and together:** toc gets PVE 9.2 + reboot; cortex gets NVIDIA 580.159→580.167 + DKMS + container-toolkit 1.18→1.19 + apt + reboot. **The Claude Code host goes down here** — run this step from a different control point.
| 0 | Break-glass / OOB access matrix (ALL nodes) | Verify non-Tailscale, non-Caddy path to every node + each lockout-critical CT (headscale, Authentik CT105, Caddy, Forgejo CT103); confirm PVE/Proxmox web auth is PAM/local not Authentik | None (read-only) | Log into Contabo/PVE console; `pct enter` 105/106/107; confirm LAN `192.168.1.x` SSH; locate break-glass creds in `.ref/credentials`; TEST Authentik akadmin + each app local-admin fallback | Every node + CT shell reachable WITHOUT Authentik/headscale/Caddy; akadmin login screenshotted | N/A | FIRST Phase 0 step; gates everything mutating |
| 0 | Per-node free space + snapshot backend | `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2; record backend (LVM-thin/ZFS/dir-qcow2) + failure mode; size vs largest guests | None | Read-only; set numeric rule: never snapshot below ~20% free, require ≥2× write-delta | Free GB recorded per pool; cloud node measured; vzdump target located | N/A | Precedes all snapshot-bearing steps |
| 0 | `data` host — disk remediation (92% full, ~73 GB) | Free space sized to largest planned snapshot (VM1130) | None (cleanup; confirm deletions are cache/backup not live) | Prune stale vzdump/snapshots/ISO; `docker image prune` on recon-vm (keep `nominatim:4.5`) | Usage <~85% on snapshot-backing storage; test snapshot succeeds then removed | Restore from Forge/Syncthing/backup if a needed file removed | Free-space capture; BLOCKS data host step + canary use |
| 0 | Off-host restorable backups (stateful guests) | Verify last-good vzdump/PBS off the node for central, OTS, forgejo, matrix, nextcloud, edge2 livesync CouchDB | N/A | Read-only verification + on-demand `pg_dump`/app backup copied off-host | ≥1 backup off the changing node, restore-testable | N/A | Precedes all mutating stateful steps |
| 2 | media VM105 SABnzbd 4.5.5→5.0.4 (MAJOR) | Major; pause queue; audit post-proc scripts | `qm snapshot 105 pre-sabnzbd` + `sabnzbd.ini` | pin 5.0.4; pull/up; migrate on start | UI=5.0.4; queue intact; test NZB; arr integrations ok | redeploy 4.5.5 + restore ini (queue repair); `qm rollback` | After VM105 Phase 1; before Lidarr |
| 2 | media VM105 Lidarr 2.x→3.x (MAJOR) | Major branch migration | `qm snapshot 105 pre-lidarr` + `lidarr.db` | pin v3; pull/up; DB migration on start | UI=v3; library+indexers+download client ok | restore db (migration one-way); `qm rollback` | After SABnzbd (verify download-client link) |
| 2 | media VM105 Sonarr/Radarr/Prowlarr (1-2 ver) | Routine bumps | pre-app VM snapshot + config DBs | `compose pull && up -d` | UIs load; prowlarr sync; test grab | redeploy prior tags + DBs | After SABnzbd/Lidarr |
| 2 | media VM105 Navidrome 0.60.3→0.62.0 | Routine bump | pre-app snapshot + DB | `compose pull navidrome && up -d` | UI=0.62.0; library scans; track streams | redeploy 0.60.3 + DB | Independent of arr chain |
| 2 | media VM105 jellyseerr preview-OIDC→stable 3.3.0 | DECISION GATE: schema downgrade + correct artifact (jellyseerr vs Seerr) + OIDC parity | `qm snapshot 105 pre-jellyseerr` + config dir | test stable vs DB COPY first; swap tag only if compatible | UI=3.3.0; requests intact; OIDC/login works | restore preview tag + config; `qm rollback` | After jellyfin/arr; STOP+report if OIDC parity unclear |
| 3 | Per-node corosync/HA pre-flight (data/utility/cloud/media/toc) | Before EACH reboot | None | `pvecm status` (Quorate:Yes, 5 votes); `ha-manager status` | quorate + expected votes=5; HA implications known | abort if not quorate | Gates each Phase 3 node reboot |
| 3 | Canary node (lowest blast + headroom; media, NOT data until disk freed) | Full 9.2/QEMU11/LXC7/kernel reboot to prove the path | vzdump all guests to EXTERNAL target + record `pveversion -v` | `apt update && apt dist-upgrade` → reboot; NO live-migration in mixed window | `pveversion`=9.2/QEMU11/LXC7/kernel 6.17.13; all guests `onboot` return; quorate | boot prior kernel (GRUB); restore guests from vzdump | Phase 1+2 done; canary before high-stakes nodes |
| 3 | media host PVE 9.1.1→9.2 (+QEMU11/LXC7/kernel) | Platform + reboot | vzdump VM105/CT110/CT111 + snapshots | `dist-upgrade` → reboot | guests return; arr/peertube/caddy healthy; quorate | GRUB prior kernel; vzdump restore | After media Phase 1+2; one node at a time |
| 3 | utility host PVE 9.2 platform + reboot | Platform + reboot (known full home-ingress + mesh + central outage) | vzdump all 12 guests + versions | `dist-upgrade` → reboot | all 12 return; Caddy ingress + headscale + central back; quorate | GRUB prior kernel; vzdump restore | LAST of early nodes; low-traffic window |
| 3 | data host PVE 9.2 platform + reboot | Platform + reboot | vzdump VM1130 + versions | `dist-upgrade` → reboot | VM1130 returns; NFS/Samba serve; quorate | GRUB prior kernel; vzdump restore | After disk remediation; one at a time |
| 3 | Hypervisor host OS-security (folded in) | Apply held host libc/openssl WITH the platform full-upgrade per node | covered by per-node vzdump | included in `dist-upgrade` (not applied back in Phase 1) | sec=0 host-side post-reboot; daemons up | per-node rollback | Fold-in decision (Open Dec #2) |
| 3 | Post-reboot app re-verification (per node) | Re-run Phase-2 health checks for majored guests on each rebooted node | None | health probes | CT120/CT121/CT110/VM105/CT109 healthy on new substrate | re-snapshot/redeploy degraded component | After each node reboot |
| 3 | edge2 host PVE 8.4 reboot — CONDITIONAL | Only if Phase 0 confirmed kernel fix needed; own no-failover window AFTER CT105/CT107 verified | vzdump all guests + `pveversion` | scoped kernel/security upgrade → reboot in approved window | `uname -r`=fixed; all CTs return; Caddy-fronted services reachable | GRUB prior kernel; vzdump restore | GATED on Phase 0; announce SSO+tailnet downtime; non-tailnet path |
| — | Nominatim 4.5→5.3 + Photon 1.1→1.2 (SEPARATE PROJECT) | Full re-import into separate DB/instance; keep 4.5 until validated; Photon AFTER nominatim validated | `qm snapshot 1130` + retain 4.5 image/data | import to new DB; size flat-nodes vs free space (NOT on data); cut over after verify | 5.x geocode correct; Photon 1.2 reads v5 | re-pin nominatim:4.5 + old data | Wholly separate from patch campaign; after disk remediation |
---
## Plan Hardening (from review)
The review surfaced corrections across six dimensions. They are merged and de-duplicated below, critical/high first, grouped by phase, with hosts/CTs named. Several issues recur because the same architectural fact (one corosync cluster; control plane runs over the tailnet; `data` is 92% full; SSO + ingress + git + tailnet are mutually entangled) drives multiple failure modes — the consolidated corrections address the root cause once.
### Cluster & control-plane facts that the whole plan must respect
- **The five PVE nodes are ONE corosync cluster (`echo6-cluster`), not standalone hypervisors.** Confirmed in `vault/docs/hardware/environment.md` (line 17) and `vault/runbooks/proxmox-onboard-node.md`. Quorum = 3 of 5. The plan never mentions corosync/quorum. **Every Phase 3 node reboot must be gated on `pvecm status` → `Quorate: Yes`, expected votes = 5; reboot ONE node at a time; wait for full rejoin (5/5) before the next.** Losing quorum makes `/etc/pve` read-only fleet-wide and freezes all VM/CT start/stop/config on every surviving node. Also run `ha-manager status` first — if any guest is HA-managed, a reboot triggers fencing/auto-migration, not a clean local stop/start.
- **toc is the 5th cluster vote AND the GPU host for cortex (VM150) AND cortex is the documented cluster "Management host" + the Claude Code control host.** Rebooting toc removes a vote and kills the box you'd diagnose from. **toc+cortex must be the final cluster action and must never overlap any other node reboot** (toc down + a second node down = bare 3/5; one corosync flap loses quorum).
- **Reconcile the headscale location contradiction BEFORE touching any of it.** The audit places headscale at utility CT106 *and* edge2 CT107 (and calls CT106 both "meshtastic-hs" and "headscale control plane"); the vault docs (`headscale-onboard-node.md`) describe headscale as a Docker container on Contabo (`100.64.0.1`) reached over the VPN. You cannot protect a control plane you have mislocated. Pin the real instance(s) and their reachability first; this gates every headscale step below.
### Phase 0 — preflight (promote these to hard gates; some must move to first)
- **(MOVE TO FIRST) Verify a non-Tailscale, non-Caddy break-glass path to every node** — Contabo/Proxmox console for edge2, LAN `192.168.1.x` for the home nodes, PVE noVNC into each CT — and confirm PVE/Proxmox web auth is PAM/local, NOT behind Authentik. Losing the management path mid-run is the single highest-consequence failure on this fleet. This must pass before anything mutating.
- **Measure, don't hand-wave, snapshot headroom.** Run `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2. Record the storage backend per node (LVM-thin/ZFS/dir-qcow2) and its over-provision failure mode. Set a hard numeric rule: never start a snapshot below ~20% free; require ≥2× the largest guest's expected write-delta. `data` at 92% (~73 GB free) cannot safely snapshot VM1130 recon-vm (nominatim/overture/padus), and **cloud node free space (CT120 immich, CT121 nextcloud AIO) is not measured at all** — add it. Identify where vzdump lands; if it's `data`, freeing `data` is a fleet-wide rollback prerequisite, not a data-node nicety.
- **`data` disk remediation is a blocking gate, sized from measurement.** Identify consumers (old vzdump, snapshots, ISO/template store, recon-vm dangling Docker layers — do NOT remove the pinned `nominatim:4.5` image), confirm nothing deleted is the only copy of a backup or live NAS data, and size cleanup to the largest planned snapshot on the node. Must complete before `data`'s own host step and before it is used as a canary.
- **Verify off-host restorable backups exist** (distinct from per-step snapshots) for every stateful guest: central PG/NATS, opentakserver PG + RabbitMQ definitions, forgejo PG + repos, matrix Synapse PG, nextcloud AIO (borg), edge2 livesync CouchDB. At least one copy must live off the node being changed.
- **Capture a pre-change baseline HEALTH snapshot** (per-target up/down, `docker ps`/`pct`/`qm status`, key endpoint 200 checks, and `apt list --upgradable`/security counts) so "verify after" has a comparison and known oddities are recorded: jellyseerr on the preview-OIDC dev tag, mailcow CT108 already STOPPED, pinned-stale nominatim, CT118 archivist rpcbind on `0.0.0.0:111`.
- **Close Open Decision #8 as a gate:** explicit sign-off on needrestart-driven daemon bounces for central (CT104 PG16/NATS/JetStream), opentakserver (CT109), peertube (CT110), matrix Synapse, recon-vm VM1130 PG16 + navi-backend, and the auth/tailnet daemons — or set those guests `NEEDRESTART_MODE=l` (list-only) so the operator controls the bounce.
- **Verify the load-bearing post-cutoff CVE claims against primary advisories before they drive ordering** (Authentik May-2026 waves justifying the mandatory 2025.12.6 hop; Valkey 9.1.0; Immich 2.6/2.7; RabbitMQ 4.x; edge2 DirtyFrag CVE-2026-43284/-43500 and copy.fail CVE-2026-31431). Assign an owner + primary-source URL per claim. **Resolve the edge2 kernel question fully here** (pin `uname -r`/`proxmox-boot-tool kernel list`, check repo kernel availability) so edge2 is either firmly scheduled for its own no-failover reboot window or firmly excluded — not left conditional inside Phase 3.
- **Decommission decision for mailcow CT108** (STOPPED, superseded by edge1): vzdump + mailcow native backup, confirm edge1 is handling mail and no MX points at edge2, then `pct destroy` only on approval. Removes CT108 from all later scope.
- **Define and announce maintenance windows in America/Boise**, naming user-facing blips: Authentik SSO (the upgrade "briefly breaks login to everything"), all home services behind Caddy, matrix CT106, media VM105, immich/nextcloud/peertube.
- **Prove the procedure on a low-stakes guest first** (utility CT112 cobalt idle, or CT102 searxng) before the never-patched worst-first targets (CT119 "never patched", 179 upgradable / 91 sec; CT108 meshai). Keep worst-first for security urgency only AFTER the procedure is proven; snapshot each never-patched guest immediately before its first-ever security upgrade.
- **Carve the stateful guests out of the bulk needrestart pass** (CT104 central, CT109 opentakserver, CT110 peertube, VM1130 recon-vm): set `NEEDRESTART_MODE=l`, stage the packages, take a logical DB dump (`pg_dump`/JetStream snapshot) with the app quiesced, then bounce each service deliberately in a controlled order. A JetStream restart mid-write loses messages on an at-most-once pipeline; a RabbitMQ/redis bounce mid-job is not a clean "blip."
- **Carve the lockout-critical daemons out too** (Authentik CT105, the headscale instance(s), Caddy, and sshd on any host you're connected over). Run `needrestart -r l` to see what *would* restart and bounce deliberately, so a libc/openssl bump can't auto-bounce the auth/tailnet/SSH daemons out from under the operator mid-pass.
- **Order control-plane guests LAST as singletons** — the home-ingress Caddy and the headscale instance(s) — each with DB backup + node-stays-joined verification, never inside a batch. Before touching headscale, confirm the out-of-band path (LAN SSH / PVE console via `pct exec`) to every still-queued target.
- **edge2 CT106 matrix / CT107 headscale have NO Tailscale client (Caddy-only ingress).** Drive their security-apt step via `pct exec` from the edge2 HOST (`root@184.174.35.153`), never over service ingress, so a needrestart bounce of a network/resolver daemon can't strand the only path in. Treat them as their own singleton steps, not part of the edge2 batch.
- **Gate `data`'s host step on the disk remediation being verified DONE** (its snapshot guardrail needs the space).
### Phase 1 hosts vs Phase 3 (sequencing)
- **Fold the four cluster hosts' OS-security apt into their own Phase 3 reboot window** rather than applying host libc/openssl in Phase 1 and living mixed-state (new libc / old kernel, bounced smbd) across the entire Phase 2 app campaign. Guests still get Phase-1 security immediately; each host transitions in one clean window. (Open Decision #2 — this is the sequencing-correct answer.)
### Phase 2 — app upgrades (sequence so a failure is isolatable, and never co-schedule recovery paths)
- **Hard rule: {Headscale, Authentik, Caddy, Forgejo} are NEVER in the same maintenance window** so at least one of {tailnet, SSO, ingress, git} is always a known-good recovery path.
- **Authentik first** among load-bearing apps (it fronts Forgejo, Vaultwarden, WordPress, Nextcloud, Matrix). Prove break-glass with evidence first: log in with the local akadmin in a private session; confirm each protected app has a working local-admin fallback (Forgejo `admin user`, Nextcloud `occ`, PDM/Vaultwarden local). The chain `2025.12.4 → 2025.12.6 → 2026.2.x → 2026.5.3` runs irreversible Django migrations at each hop — **`pg_dump` per hop (labelled), one hop at a time, verified-login gate between each, abort+restore on first migration error.** Add the 2025.12 hard pre-flight: audit/rename duplicate group names (migration "fails loudly" on dupes), and if local `/media` storage is used, stop and `mv ./media ./data/media` + rewrite the compose volume before starting the new version. Verify all dependent SSO logins BEFORE upgrading the dependents.
- **Caddy early** (utility ingress + media CT111), before home-service app verification, so backend sign-offs run through the new ingress; pair CT101/CT111 in one deliberate window. Map the Caddy↔Authentik path first and document a direct (Caddy-bypassing) Authentik admin URL as break-glass.
- **OpenTAKServer CT109 — decouple RabbitMQ from the OTS bump.** The OTS updater only upgrades OTS + webUI + schema; it does NOT migrate RabbitMQ, Erlang, or feature flags (verified against docs.opentakserver.io/installation/upgrading.html), so the most security-urgent EOL item (RabbitMQ 3.12.1, no 3.x backports) would silently go unfixed. Write RabbitMQ as its own sub-task: snapshot + `export_definitions` → Erlang ≥26 → 3.12 → 3.13.x → `rabbitmqctl enable_feature_flag all` (confirm all enabled) → only then 4.x; confirm no classic-queue-mirroring config remains. A naive 3.12→4.x jump refuses to boot. Gate the OTS step on RabbitMQ 4.x up + TAK clients reconnecting before accepting the bundled MediaMTX/Mumble bumps.
- **Forgejo CT103 14→15** is a branch migration on an EOL branch (14.x EOL 2026-04-30) with forward-only FK migrations. Run `forgejo doctor check --all [--fix]` to repair stopwatch/tracked_time inconsistencies BEFORE the backup+bump. Pre-step: `git clone --mirror` echo6-docs (and other critical repos) to an OFF-edge2 location and pause the `echo6-docs-autocommit` cron so the recovery docs survive an edge2/Forgejo failure and the cron doesn't loop-error into a half-migrated instance.
- **Headscale: upgrade the lower-stakes instance FIRST as a rehearsal, the front-door LAST**, each driven from a path that does NOT depend on the instance being upgraded (PVE console / direct edge2-host → CT internal IP). Before the front-door hop: disable/extend node key-expiry on all nodes so persistent tunnels don't drop on re-handshake mid-migration; `pct snapshot` + dump the headscale DB off-tailnet; keep a second already-authenticated SSH session open. Rollback restores the snapshot via console, not via tailnet. Verify headplane compatibility with 0.29.
- **Immich CT120 / Nextcloud AIO CT121 — method is correct, but size disk and protect the DB.** Prune old images + `docker image prune` before the snapshot; do NOT interrupt Immich's first-boot 2.5→2.7 DB migration; AIO self-gates intermediate mastercontainer versions (a stale AIO may need two cycles) — use AIO's borg backup as the rollback artifact (mastercontainer downgrades unsupported). The rollback unit for every DB-bearing app is a **quiesced logical dump + old binary together** — "redeploy previous image tag" is NOT valid after forward-only migrations, and a `pct snapshot` of a hot, separate-volume DB can restore torn. Disable Immich background regeneration until free space is confirmed.
- **jellyseerr is a data-compatibility decision, not a retag.** preview-OIDC (dev) → stable 3.3.0 is a schema DOWNGRADE; TypeORM stable may refuse to start against a dev-migrated DB. Confirm the correct destination artifact (jellyseerr stable vs the unified "Seerr" successor) and, if OIDC is in active use, confirm 3.3.0 parity — else STOP and report. Test the stable image against a COPY of the DB first.
- **cortex ollama 0.16→0.30:** reframe — old local blobs read fine; the real trap is no clean downgrade after pulling any new model. Snapshot the models volume (the only rollback), upgrade, verify existing models `ollama run` without re-pull. Verify the vault-tagger engine (localhost:11434) and TEI `related:` engine (localhost:8090) still work after.
- **SABnzbd 4.5→5.0:** pause/empty the queue, audit custom post-proc scripts (scripts now run on failed jobs), note `empty_postproc` removed and that downgrade needs a queue repair.
- **Nominatim 4→5 + Photon 1.1→1.2 is a separate project, not part of the patch campaign.** Re-import into a SEPARATE DB/instance; keep the `nominatim:4.5` image + data until 5.x is validated; size transient import disk (flat-nodes tens of GB) against real free space on a non-starved node — this cannot live on the 92%-full `data` pool; sequence Photon 1.2 AFTER nominatim 5.x is validated, never concurrently.
### Phase 3 — platform & reboot windows
- **Per-node order: `apt update && apt dist-upgrade` (full-upgrade so proxmox-ve/qemu/lxc metapackages pull) → reboot → verify `pveversion` + all guests return.** Hard rule for the mixed-version window: **NO live migration and disable HA migration across the QEMU 10/11 (PVE 9.1/9.2) boundary** — running VMs keep QEMU 10 machine type until cold-started, and migration across the boundary fails. Optionally bump VM machine types + cold-restart after all 5 are on 9.2 (noted follow-up).
- **Canary by blast radius + snapshot headroom, not label.** `data` (92% full, hosts recon-vm overture/padus) must NOT be canary until its disk is remediated, and its near-full pool can fail the mandatory pre-reboot snapshot. Lowest-stakes is **media** (single arr VM + peertube, non-auth/non-DB-critical) — but note media carries freshly-majored apps; the reconciled order is: canary the genuinely lowest-blast node with headroom → then **utility LAST among the early nodes** (it carries Caddy ingress + the mesh/central stack — a deliberately-scheduled full home-ingress + mesh-coordination outage) → cloud (immich/nextcloud) → data → **toc+cortex last and alone**, all four other nodes confirmed quorate.
- **Pre-window: confirm every guest has `onboot=1`** so they actually return; take vzdump of guests to an EXTERNAL target (not the local pool, never `data`) so a guest that won't start under LXC 7 can be restored elsewhere. Read PVE 9.2 + LXC 7 release notes for unprivileged-CT/cgroupv2/apparmor breaking changes before the first node.
- **Add a post-Phase-3 re-verification gate:** after each node boots into 9.2/LXC7, re-run the Phase-2 health check for every guest that received a major app upgrade on that node (CT120 immich, CT121 nextcloud, CT110 peertube, VM105 arr, CT109 OTS) — the substrate changed under them.
- **toc+cortex window:** name and TEST a concrete alternate control point (a non-cortex host with keys + tooling to the fleet) BEFORE the window; stabilize the tailnet well before it (never in the same period as a headscale upgrade). Split cortex: snapshot VM150 from toc → NVIDIA 580.159→580.167 + DKMS + container-toolkit 1.19 as a discrete reversible step, confirm `dkms status` built and `nvidia-smi` BEFORE rebooting toc → reboot toc → confirm toc rejoins (5/5 votes) and cortex + GPU passthrough return. Keep the old driver package for reinstall.
- **edge2 is PVE 8.4, architecturally separate, single SPOF for ingress (no failover).** Do NOT fold it into the cluster windows. If the Phase 0 kernel verification says it needs a reboot, give it a dedicated window AFTER Authentik CT105 + headscale CT107 are upgraded and verified, announce SSO+tailnet downtime, drive from a non-tailnet path, and confirm all CTs auto-start.
- **pi-nas: split into two reboots, not one.** OMV 8.1→8.4 via OMV's own update path (NOT raw `apt full-upgrade`) → reboot → verify shares/SMB/NFS/omv-salt; THEN kernel 6.12→6.18 → reboot → verify boot + disk remount. Confirm physical/serial console access (no remote KVM on an RPi), back up `/boot`+`/boot/firmware` and keep the old kernel installed as fallback, export `config.xml` off-box, and confirm it's not mid-sync as a Syncthing/backup target before taking it down. Independent of the cluster windows.
| CT108 | mailcow | 18 containers (postfix/nginx/dovecot/rspamd/clamd/sogo/php-fpm/mariadb/redis/memcached/unbound/acme/olefy/watchdog/netfilter/ofelia/dockerapi/tlspol) — upgrade via `update.sh` only |
---
## Coverage Notes
- **Complete:** all five PVE-9 nodes (data, utility, cloud, media, toc), edge2, cortex, pi-nas, and all discovered guests/containers.
- **Excluded:** edge1 (the rebuilt ex-Contabo, mail-only) — mid-rebuild/maintenance at time of audit; re-audit once it settles.
- **Incomplete:** WordPress (CT101) plugin and theme status — requires WP-CLI; not assessed.
- **Method:** read-only throughout — SSH / `pct exec`, `apt list --upgradable`, `docker manifest inspect` for same-tag drift. Floating/pinned-tag caveats noted inline where drift could not be confirmed.
