**Topology note:** the old Contabo VPS has been rebuilt as **edge1 (mail-only)**; **edge2 is now the front door for everything else**. edge1 is excluded from this audit (mid-rebuild/maintenance). **Headscale:** edge2 CT107 is the main fleet tailnet (34 nodes, `vpn.echo6.co`, self-hosted Headscale 0.28.0); utility CT106 is a separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes, low-risk). No services route through old-Contabo. **Mailcow CT108:** destroyed 2026-06-20 (`pct destroy 108 --purge`); backup preserved durably on pi-nas (`…/contabo-prewipe-2026-06/mailcow/`, sha256-verified); live mail on edge1 (MX/A for mail.echo6.co → 5.189.158.149).
Running application version vs latest stable upstream, per app — the "is everything up to date?" view (distinct from OS-package/image-drift above). Read-only. **Caveat:** version deltas were measured directly from the running apps and are reliable; the specific CVE/advisory IDs below were surfaced by the audit agents and are **post-knowledge-cutoff — verify against primary advisories before acting on them.** GitHub's anonymous API rate-limited a few "latest" lookups (noted inline).
| **Headscale**×2 | utility CT106 + edge2 CT107 | 0.28.0 → **0.29.1** | 1 minor | upgrade-guide-required; two separate instances; no security flag noted |
### Operational / host-level (not a simple app bump)
## Proposed Patch Approach (NOT executed — for planning)
Lowest-risk changes first; everything reboot-bearing deferred to scheduled windows.
**Principles**
- **Security-pocket apt only** in the first pass (openssl, openssh, gnutls, krb5, libc6, samba, nghttp2, etc.). No PVE 9.2 / QEMU / LXC / kernel, no Docker image pulls, no OMV/NVIDIA, no reboots.
- **Protected hosts (cortex, toc) never go in a bulk pass** — handled individually in their own window. Note: toc hosts cortex (VM150), so a toc reboot drops cortex — the two must be coordinated together.
- **edge1 (mail)** excluded while it's mid-rebuild.
-`needrestart` will bounce affected daemons after glibc/openssl upgrades — seconds of blip per guest, no data risk.
1.**RabbitMQ 3.12→4.x** — **ACCEPTED** as a decoupled sub-task (Erlang ≥26 → 3.13.x → enable feature-flags → 4.x); the OpenTAKServer bump does NOT cover it. In scope for this campaign.
2.**Break-glass** — **VALIDATED**: all 7 LAN SSH paths work independent of the tailnet; every PVE node (incl. edge2) has local `pam`/`pve` realms; Authentik `akadmin`, Forgejo `matt`, Nextcloud `admin` local logins confirmed. ⚠️ **One gap → PDM (edge2 CT100)** has only `openid:authentik` in `domains.cfg`; confirm `root@pam` login at `https://100.64.0.28:8443` (or add a `pam:` stanza) **before** the Authentik upgrade.
4.**Daemon-restart tolerance** — **DOWNTIME ACCEPTED**; needrestart blips OK on the stateful guests (still take logical DB dumps for safety, but no holding restarts for availability).
5.**edge2 host kernel (DirtyFrag/copy.fail)** — **NO ACTION / NO REBOOT**: running `6.8.12-30-pve`, the newest its repos offer; no update available, no reboot-required flag. Not actionable without a repo/branch change. edge2 needs no Phase 3 reboot.
6.**CVE verification** — **NOT GATING**: being behind on versions is sufficient justification; post-cutoff CVE IDs are not chased or relied on for ordering.
7.**Maintenance windows** — **NO CONSTRAINT** (single user); disruptive steps may run anytime, no scheduling/announcement needed.
8.**Nominatim 4→5** — **DEFERRED / OUT OF SCOPE**: spun off to [[nominatim-v5-reimport]] as its own project.
- **PDM break-glass** — ✅ **DONE**: local `admin@pam` (Administrator) created on PDM CT100, login verified via API, Authentik realm untouched; cred in `credentials`, config backup at CT100 `/root/access.bak-2026-06-20`.
- **CT118 archivist rpcbind** — ✅ **DONE**: nftables rule restricts port 111 to source `192.168.1.240` (NFS server) only; NFS mount healthy, ruleset persisted. Now 100% LAN-internal.
- **data disk (92%)** — ⏏️ **DE-SCOPED**: not a gate. OS package updates are roll-back-able (reinstall prior version); the migration-heavy apps that need real rollback (Authentik, Forgejo, Nextcloud, PeerTube) live on cloud/edge2, not `data`. Optional cleanup only (~6 GB obviously-safe: stale ISO, zimit temp) if ever wanted.
**Rollback model (corrected):** OS packages → reinstall the prior version (no VM snapshot needed). App DB-migration upgrades (Authentik/Forgejo/Nextcloud/PeerTube) → restore a quiesced DB dump (cheap), since reinstalling the old binary won't unwind a migrated schema.
Step-by-step plan to bring every application and package current. Per-app target versions live in **Application Currency** above; this section is the *how/when/order*. Execution model: scoped Sonnet task per target → Opus verifies output → report to Matt; **approval gate before every load-bearing or reboot step.**
### Standing guardrails (apply to EVERY step)
- **Protected hosts — cortex & toc — never in a bulk pass.** They get a dedicated, manual window. Note the coupling: **toc reboot drops cortex (VM150), which is the Claude Code host** — expect to lose the control session during that window; drive it from elsewhere or accept the outage.
- **One target at a time.** Verify health before moving to the next.
- **Snapshot/backup before each mutating step:** `pct snapshot` / `qm snapshot` (or `vzdump`) the guest first; DB dump before any DB-bearing upgrade; run the app's own backup where it has one (Nextcloud AIO, mailcow).
- **No host reboots outside the explicit Phase 3 windows.** Security upgrades that set `reboot-required` are applied but activation deferred to Phase 3. Host OS-security apt for the four cluster nodes is NOT applied in Phase 1 — it rides the Phase 3 full-upgrade window.
- **Rollback for DB-bearing apps = quiesced logical dump + old binary together.** For authentik, forgejo, nextcloud, peertube, and immich the rollback unit is a `pg_dump` / AIO borg / peertube dump taken with the app stopped and the old binary still in place. "Redeploy previous image tag" is NOT valid after forward-only migrations — Django/TypeORM migrations do not reverse, and a `pct snapshot` of a hot separate-volume DB may restore torn. Record before/after version in the tracking table.
- Every result must survive a reboot (standing infra policy).
### Load-bearing dependencies (drive the ordering)
- **Authentik (edge2 CT105) = SSO.** Upgrading it briefly breaks login to everything behind it. Confirm **break-glass admin** access first; do in a low-traffic window; verify dependent-service logins after.
- **Caddy (utility CT101) = home-services ingress.** A restart blips all home services — fold its update into a deliberate moment, not mid-day.
- **Headscale ×2:** edge2 CT107 = main fleet tailnet (34 nodes, `vpn.echo6.co`, ControlURL `https://vpn.echo6.co` → 184.174.35.153) — **HIGH lockout risk**. utility CT106 = IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes) — low risk. Upgrade CT106 first as rehearsal. Edge2 CT107 upgrade MUST be driven from an out-of-band path (edge2 host console / direct SSH to 184.174.35.153, NOT over `vpn.echo6.co`); extend node key-expiry first; dump DB off-tailnet before touching it.
**STEP 1 (FIRST — gates everything mutating): Verify break-glass and out-of-band access.** Confirm a non-Tailscale, non-Caddy path to every node: Proxmox/Contabo console for edge2, LAN `192.168.1.x` SSH for home nodes, PVE noVNC/`pct enter` for each CT. Confirm PVE/Proxmox web auth is PAM/local, NOT behind Authentik. Prove it: log into Contabo/PVE console; `pct enter` 105/106/107; confirm LAN SSH; locate break-glass creds in `.ref/credentials`. Verify Authentik akadmin (private browser session) + each protected app has a local-admin fallback (Forgejo `admin user`, Nextcloud `occ`, PDM/Vaultwarden local). Every node + CT shell must be reachable WITHOUT Authentik/headscale/Caddy. Losing the management path mid-run is the single highest-consequence failure mode.
**STEP 2: Measure snapshot headroom (per node).** Run `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2. Record the storage backend per node (LVM-thin/ZFS/dir-qcow2) and its over-provision failure mode. Hard numeric rule: never start a snapshot below ~20% free; require ≥2× the largest guest's expected write-delta. Cloud node CT120/CT121 free space must be measured now — not assumed. Identify where vzdump lands; if it's `data`, freeing `data` is a fleet-wide rollback prerequisite.
**STEP 3: Free space on `data`** (92% full, ~73 GB free) — sized to the largest planned snapshot (VM1130 recon-vm / nominatim+overture+padus). Prune stale vzdump/snapshots/ISOs; `docker image prune` on recon-vm (do NOT remove the pinned `nominatim:4.5` image). Confirm nothing deleted is the only copy of a backup or live NAS data. Verify: usage <~85% on snapshot-backing storage; test snapshot succeeds then remove. **Blocks data host Phase 3 and canary use.**
**STEP 4: Verify off-host restorable backups** (distinct from per-step snapshots) for every stateful guest: central PG/NATS (CT104), opentakserver PG + RabbitMQ definitions (CT109), forgejo PG + repos (CT103), matrix Synapse PG (CT106), nextcloud AIO borg (CT121), edge2 livesync CouchDB (CT104). At least one copy must live off the node being changed.
**STEP 5: Capture pre-change baseline HEALTH snapshot.** Record per-target up/down, container states (`docker ps`/`pct`/`qm status`), key endpoint 200 checks, and `apt list --upgradable`/security counts. Note known oddities: jellyseerr on preview-OIDC dev tag, mailcow CT108 already STOPPED, pinned-stale nominatim, CT118 archivist rpcbind on `0.0.0.0:111`.
**STEP 6: Decision gates.** Close genuinely-open decisions before Phase 1/2/3 can start: daemon-restart tolerance sign-off (Open Dec #7); verify post-cutoff CVE claims against primary advisories with owner+URL per claim; resolve edge2 host-kernel CVE question fully (pin `uname -r`, check repo kernel availability, decide reboot yes/no — not left conditional); define and announce maintenance windows in America/Boise naming user-facing blips.
### Phase 1 — Guest OS security packages (no reboot; hosts folded into Phase 3)
Guests only in Phase 1 (low blast radius). **The four cluster hosts' OS-security apt rides their Phase 3 full-upgrade window** — do NOT patch data/utility/cloud/media host OS in Phase 1 (avoids a mixed-state libc/openssl across the entire Phase 2 app campaign; both phases get cleaned up in one reboot anyway). edge2 host is already fully patched — no action. toc excluded (Phase 3, with cortex).
Per guest target: `apt-get update` → snapshot → apply **security-pocket** upgrades → needrestart policy (see below) → verify service health and security-upgradable count hits 0. Kernel/libc security that flags `reboot-required` → applied, reboot activation deferred to Phase 3.
**needrestart policy:**
- **Stateful guests — `NEEDRESTART_MODE=l` (list-only), then deliberate bounce:** CT104 central (PG16/NATS/JetStream), CT109 opentakserver, CT110 peertube, VM1130 recon-vm. Stage the packages; take a logical DB dump with the app quiesced; bounce each service deliberately in controlled order. A JetStream restart mid-write loses messages on at-most-once pipelines; a RabbitMQ/redis bounce mid-job is not a clean "blip."
- **Lockout-critical daemons — `NEEDRESTART_MODE=l` + deliberate controlled bounce:** Authentik CT105, headscale instance(s), Caddy CT101/CT111, sshd on any host you're connected through. Run `needrestart -r l` to see what would restart; bounce deliberately so a libc/openssl bump can't auto-bounce auth/tailnet/SSH daemons out from under the operator.
- **All other guests:** `needrestart -r a` (automatic) is acceptable — seconds of blip, no data risk.
**Ordering:**
1.**Canary first** (utility CT112 cobalt idle, or CT102 searxng) — prove the full procedure on a low-stakes guest before touching never-patched targets.
3.**Remaining guests:** all other utility/cloud/media/edge2 CTs + data VM1130 (recon-vm). Gate data VM1130 step on data disk remediation verified DONE.
4.**edge2 CT106 matrix / CT107 headscale (NO Tailscale client):** drive via `pct exec` from the edge2 HOST (`root@184.174.35.153`) ONLY — never over service ingress. Treat as own singleton steps, not part of edge2 batch. `needrestart -r l`; do NOT restart headscaled/synapse unless required.
5.**Control-plane singletons LAST:** home-ingress Caddy CT101 and headscale instance(s) — each with DB backup + node-stays-joined verification, never inside a batch.
6.**cortex VM150 (PROTECTED):** security apt manually in the toc+cortex window (Phase 3 coordination); not in bulk.
7.**Cross-cutting:** Tailscale 1.94→1.98 and Docker CE→29.6 ride along Phase 1 under the same snapshots; EXCLUDE edge2 CT106/CT107 from Tailscale bump (no TS client).
8.**pi-nas:** Debian security pocket with `linux-image-*` and `openmediavault*` packages held; kernel upgrade rides Phase 3.
**Hard isolation rule:** {Headscale, Authentik, Caddy, Forgejo} are NEVER in the same maintenance window — at least one of {tailnet, SSO, ingress, git} must always be a known-good recovery path.
**2A — Load-bearing SSO and control-plane apps (in this order):**
- **Pre-flight before any hop:** (1) Prove break-glass: log in as akadmin in a private browser session; confirm every protected app has a working local-admin fallback (Forgejo `admin user`, Nextcloud `occ`, PDM/Vaultwarden local, PVE PAM). (2) Audit/rename duplicate group names — the Django migration fails loudly on dupes. (3) If local `/media` storage is used, stop the service, `mv ./media ./data/media`, rewrite compose volume, before starting any new version.
- **Hop sequence (no skipping):** 2025.12.4 → **2025.12.6** (backported security) → **2026.2.x** → **2026.5.3**. One hop at a time: `pct snapshot` + `pg_dump` (labelled per hop) → pull new tag → `compose up -d` → migrations run → verified-login gate (all dependent SSO logins tested) → abort+restore on FIRST migration error. Verify all dependent SSO logins BEFORE upgrading any of the dependents.
- Rollback: `compose down` → restore prior tag + dump; or `pct rollback` for the full CT.
- **Caddy (utility CT101 + media CT111) — early in Phase 2, before home-service app verification.** Pair CT101/CT111 in one deliberate low-traffic window (separate from Authentik, Forgejo, and headscale windows). Validate `caddy validate`; spot-check Caddy↔Authentik path; document a direct Caddy-bypassing Authentik admin URL as break-glass. Verify all home services reachable through the new ingress before proceeding with other app upgrades.
- **RabbitMQ 3.12.1→4.2.8 is a SEPARATE sub-task (decoupled, after OTS app step).** The OTS updater does NOT migrate RabbitMQ, Erlang, or feature flags — the EOL RabbitMQ 3.12 would silently remain. Decoupled upgrade: `pct snapshot` + `rabbitmqctl export_definitions` → upgrade Erlang to ≥26 → 3.12→3.13.x → `rabbitmqctl enable_feature_flag all` (confirm all enabled, no classic-queue-mirroring config remains) → only then 4.x. A naive 3.12→4.x jump refuses to boot. Gate "OTS done" on RabbitMQ 4.x up + TAK clients reconnecting before accepting the bundled MediaMTX/Mumble bumps.
- **Immich (cloud CT120)** — prune old images + `docker image prune` before snapshot; disable Immich background regeneration until free space is confirmed; `docker compose pull && up -d`. Do NOT interrupt first-boot 2.5→2.7 DB migration. Also clears Valkey 9.0.2→9.1.0. Rollback = restore `pct snapshot` + `pg_dump` taken with the app stopped (not a hot-DB snapshot alone). Verify web, mobile sync, ML.
- **Nextcloud AIO (cloud CT121)** — AIO borg backup → stop → update mastercontainer → trigger child update via AIO UI (:8080). A stale AIO may need two cycles. Rollback unit = AIO borg backup (mastercontainer downgrade is not supported — borg is the only path back). Verify `occ status` 32.0.11; 12 containers green; integrity clean.
- Pre-steps: `git clone --mirror` echo6-docs (and other critical repos) to an OFF-edge2 location; pause the `echo6-docs-autocommit` cron so recovery docs survive an edge2/Forgejo failure; run `forgejo doctor check --all [--fix]` to repair stopwatch/tracked_time inconsistencies BEFORE backup+bump. Read 15.0 breaking changes + forward-only FK migration notes.
- CT107 (main fleet, 34 nodes, HIGH lockout risk — edge2 front door): extend/disable node key-expiry on all 34 nodes BEFORE starting; `pct snapshot 107` + dump headscale DB off-tailnet (NOT via vpn.echo6.co); keep a second authenticated SSH session open to 184.174.35.153. Drive entirely from edge2 host console / direct SSH to `root@184.174.35.153` — NOT over the tailnet being upgraded. Verify boot chain: edge2 host → CT107 autostart → headscale up → `vpn.echo6.co` resolves → Caddy proxies. Rollback: restore snapshot via console (not via tailnet).
- NOT in the same window as Caddy, Authentik, or Forgejo.
-`qm snapshot 105` before batch. SABnzbd 4.5→5.0: pause/empty queue; audit custom post-proc scripts (`empty_postproc` removed in 5.x, scripts now run on failed jobs); pin 5.0.4 tag; pull/up; verify queue intact. Lidarr 2.x→3.x: pin v3; pull/up; DB migration on start; verify library+indexers+download client. Sonarr/Radarr/Prowlarr (1-2 ver): routine bumps after SABnzbd/Lidarr. Jellyfin 10.11.6→10.11.11: pin tag; pull/up. Navidrome: routine bump.
- **Jellyseerr preview-OIDC→stable 3.3.0: DECISION GATE before retag.** preview-OIDC → stable 3.3.0 is a schema DOWNGRADE; TypeORM stable may refuse to start against a dev-migrated DB. Confirm the correct destination artifact (jellyseerr stable vs "Seerr" successor) and, if OIDC is in active use, confirm 3.3.0 parity. Test stable image against a COPY of the DB first. If parity unclear → STOP and report.
- Verify each UI; snapshot before, per-app snapshots for majors.
- **cortex AI stack (PROTECTED — manual, own window):** ollama 0.16.1→0.30.10 (snapshot models volume — the only rollback after pulling new models; upgrade; verify existing `ollama run` without re-pull; verify vault-tagger at localhost:11434 and TEI at localhost:8090 still work); open-webui 0.8.1→0.9.6; qdrant 1.16→1.18; tei 1.7→1.9. Pull images, verify.
- **Nominatim 4.5→5.3 + Photon 1.1→1.2 (recon-vm)** — requires full OSM re-import into a SEPARATE DB/instance; keep `nominatim:4.5` image + data until 5.x validated; transient import disk (flat-nodes tens of GB) cannot live on the 92%-full `data` pool; Photon 1.2 AFTER nominatim 5.x validated, never concurrently. Schedule as its own effort.
**Corosync cluster rules (apply to EVERY node reboot in this section):**
- The five PVE nodes (data, utility, cloud, media, toc) are ONE `echo6-cluster`, quorum = 3 of 5. Losing quorum makes `/etc/pve` read-only fleet-wide and freezes all VM/CT start/stop/config on every surviving node.
- Before EACH node reboot: `pvecm status` must show `Quorate: Yes` and `Expected votes: 5`. Also run `ha-manager status` — if any guest is HA-managed, the reboot triggers fencing/auto-migration rather than a clean local stop/start.
- Reboot ONE node at a time. Wait for full 5/5 rejoin before the next node.
- Disable HA migration and NO live migration during the mixed-version window (QEMU 10/11 boundary — running VMs keep QEMU 10 machine type until cold-started; cross-boundary migration fails).
**Per-node procedure:** `apt update && apt full-upgrade` (so proxmox-ve/qemu/lxc metapackages pull) → this also clears the held host OS-security packages → reboot → verify `pveversion` + all guests return with `onboot=1`. Confirm `pvecm status` shows 5/5 before proceeding.
**Pre-window for each node:** vzdump all guests to an EXTERNAL target (not the local pool, never `data`) so a guest that won't start under LXC 7 can be restored elsewhere. Read PVE 9.2 + LXC 7 release notes for unprivileged-CT/cgroupv2/apparmor breaking changes before the first node.
**Canary node order (lowest blast-radius with headroom first):** media (single arr VM + peertube, non-auth/non-DB-critical) → cloud (immich/nextcloud, after media proven) → data (after disk remediation verified; its near-full pool can fail the mandatory pre-reboot snapshot — do NOT use as canary until remediated) → **utility LAST of the four** (carries Caddy ingress + mesh/central stack — deliberately-scheduled full home-ingress + mesh-coordination outage, announce in advance) → **toc+cortex LAST AND ALONE** (see below).
**Post-reboot re-verification gate (per node):** after each node boots into 9.2/LXC7, re-run the Phase-2 health checks for every guest that received a major app upgrade on that node (CT120 immich, CT121 nextcloud, CT110 peertube, VM105 arr, CT109 OTS). The substrate changed under them.
**toc + cortex — DONE LAST AND ALONE, after all four other nodes are quorate:**
- toc is the 5th cluster vote AND the GPU host for cortex (VM150) AND the management/Claude Code host. Rebooting toc removes a vote and kills the box you'd diagnose from. It must NEVER overlap any other node reboot (toc down + one other = 3/5 → one corosync flap loses quorum).
- Name and TEST a concrete alternate control point (a non-cortex host with keys + tooling) BEFORE this window. Stabilize the tailnet well before it — never in the same period as a headscale upgrade.
2.`apt --only-upgrade nvidia-container-toolkit`; `nvidia-ctk runtime configure`; restart docker → verify `--version`=1.19 + `docker run --gpus all nvidia-smi`
3.`apt --only-upgrade nvidia-driver-580 nvidia-dkms-580`; DKMS rebuild → verify `dkms status` shows installed + `nvidia-smi`=580.167 BEFORE rebooting toc. Keep the old driver package installed as a reinstall fallback.
4. cortex Phase 1 security apt (if not already done in its own window)
- Then reboot toc → verify toc rejoins (`pvecm status` 5/5) → verify VM150 autostart + cortex GPU passthrough return → final AI-stack health check (driver 580.167; all 5 containers; ollama/tei/qdrant/open-webui; vault-tagger+TEI engines; TS online; Docker 29.6).
- **The Claude Code host goes down here** — run this step from the alternate control point.
**pi-nas — standalone window, TWO reboots:**
- Confirm physical/serial console access (no remote KVM on an RPi); export `config.xml` off-box; confirm pi-nas is not mid-sync as a Syncthing/backup target before each reboot. Back up `/boot` + `/boot/firmware`; keep the old kernel installed as fallback.
- Reboot 1: OMV 8.1→8.4 via OMV's own update path (NOT raw `apt full-upgrade` — use OMV UI Update Mgmt or `omv-upgrade`) → reboot → verify shares/SMB/NFS/omv-salt healthy.
- Reboot only if Phase 0 resolved the DirtyFrag/copy.fail CVE question as "yes, needs a newer kernel." If reboot required: give it a dedicated window AFTER Authentik CT105 + headscale CT107 are upgraded and verified; announce SSO+tailnet downtime; drive from a non-tailnet path; confirm all CTs auto-start. edge2 is a single SPOF for ingress — no failover. Do NOT fold into cluster windows.
| 0 | Break-glass / OOB access matrix (ALL nodes) | Verify non-Tailscale, non-Caddy path to every node + each lockout-critical CT (headscale, Authentik CT105, Caddy, Forgejo CT103); confirm PVE/Proxmox web auth is PAM/local not Authentik | None (read-only) | Log into Contabo/PVE console; `pct enter` 105/106/107; confirm LAN `192.168.1.x` SSH; locate break-glass creds in `.ref/credentials`; TEST Authentik akadmin + each app local-admin fallback | Every node + CT shell reachable WITHOUT Authentik/headscale/Caddy; akadmin login screenshotted | N/A | FIRST Phase 0 step; gates everything mutating |
| 0 | Per-node free space + snapshot backend | `pvesm status` + `df -h` on data/utility/cloud/media/toc/edge2; record backend (LVM-thin/ZFS/dir-qcow2) + failure mode; size vs largest guests | None | Read-only; set numeric rule: never snapshot below ~20% free, require ≥2× write-delta | Free GB recorded per pool; cloud node measured; vzdump target located | N/A | Precedes all snapshot-bearing steps |
| 0 | `data` host — disk remediation (92% full, ~73 GB) | Free space sized to largest planned snapshot (VM1130) | None (cleanup; confirm deletions are cache/backup not live) | Prune stale vzdump/snapshots/ISO; `docker image prune` on recon-vm (keep `nominatim:4.5`) | Usage <~85% on snapshot-backing storage; test snapshot succeeds then removed | Restore from Forge/Syncthing/backup if a needed file removed | Free-space capture; BLOCKS data host step + canary use |
| 0 | Off-host restorable backups (stateful guests) | Verify last-good vzdump/PBS off the node for central, OTS, forgejo, matrix, nextcloud, edge2 livesync CouchDB | N/A | Read-only verification + on-demand `pg_dump`/app backup copied off-host | ≥1 backup off the changing node, restore-testable | N/A | Precedes all mutating stateful steps |
| 2 | media VM105 SABnzbd 4.5.5→5.0.4 (MAJOR) | Major; pause queue; audit post-proc scripts | `qm snapshot 105 pre-sabnzbd` + `sabnzbd.ini` | pin 5.0.4; pull/up; migrate on start | UI=5.0.4; queue intact; test NZB; arr integrations ok | redeploy 4.5.5 + restore ini (queue repair); `qm rollback` | After VM105 Phase 1; before Lidarr |
| 2 | media VM105 Lidarr 2.x→3.x (MAJOR) | Major branch migration | `qm snapshot 105 pre-lidarr` + `lidarr.db` | pin v3; pull/up; DB migration on start | UI=v3; library+indexers+download client ok | restore db (migration one-way); `qm rollback` | After SABnzbd (verify download-client link) |
| 2 | media VM105 Sonarr/Radarr/Prowlarr (1-2 ver) | Routine bumps | pre-app VM snapshot + config DBs | `compose pull && up -d` | UIs load; prowlarr sync; test grab | redeploy prior tags + DBs | After SABnzbd/Lidarr |
| 2 | media VM105 Navidrome 0.60.3→0.62.0 | Routine bump | pre-app snapshot + DB | `compose pull navidrome && up -d` | UI=0.62.0; library scans; track streams | redeploy 0.60.3 + DB | Independent of arr chain |
| 2 | media VM105 jellyseerr preview-OIDC→stable 3.3.0 | DECISION GATE: schema downgrade + correct artifact (jellyseerr vs Seerr) + OIDC parity | `qm snapshot 105 pre-jellyseerr` + config dir | test stable vs DB COPY first; swap tag only if compatible | UI=3.3.0; requests intact; OIDC/login works | restore preview tag + config; `qm rollback` | After jellyfin/arr; STOP+report if OIDC parity unclear |
| 3 | Per-node corosync/HA pre-flight (data/utility/cloud/media/toc) | Before EACH reboot | None | `pvecm status` (Quorate:Yes, 5 votes); `ha-manager status` | quorate + expected votes=5; HA implications known | abort if not quorate | Gates each Phase 3 node reboot |
| 3 | Canary node (lowest blast + headroom; media, NOT data until disk freed) | Full 9.2/QEMU11/LXC7/kernel reboot to prove the path | vzdump all guests to EXTERNAL target + record `pveversion -v` | `apt update && apt dist-upgrade` → reboot; NO live-migration in mixed window | `pveversion`=9.2/QEMU11/LXC7/kernel 6.17.13; all guests `onboot` return; quorate | boot prior kernel (GRUB); restore guests from vzdump | Phase 1+2 done; canary before high-stakes nodes |
| 3 | media host PVE 9.1.1→9.2 (+QEMU11/LXC7/kernel) | Platform + reboot | vzdump VM105/CT110/CT111 + snapshots | `dist-upgrade` → reboot | guests return; arr/peertube/caddy healthy; quorate | GRUB prior kernel; vzdump restore | After media Phase 1+2; one node at a time |
| 3 | utility host PVE 9.2 platform + reboot | Platform + reboot (known full home-ingress + mesh + central outage) | vzdump all 12 guests + versions | `dist-upgrade` → reboot | all 12 return; Caddy ingress + headscale + central back; quorate | GRUB prior kernel; vzdump restore | LAST of early nodes; low-traffic window |
| 3 | data host PVE 9.2 platform + reboot | Platform + reboot | vzdump VM1130 + versions | `dist-upgrade` → reboot | VM1130 returns; NFS/Samba serve; quorate | GRUB prior kernel; vzdump restore | After disk remediation; one at a time |
| 3 | Hypervisor host OS-security (folded in) | Apply held host libc/openssl WITH the platform full-upgrade per node | covered by per-node vzdump | included in `dist-upgrade` (not applied back in Phase 1) | sec=0 host-side post-reboot; daemons up | per-node rollback | Fold-in decision (Open Dec #2) |
| 3 | Post-reboot app re-verification (per node) | Re-run Phase-2 health checks for majored guests on each rebooted node | None | health probes | CT120/CT121/CT110/VM105/CT109 healthy on new substrate | re-snapshot/redeploy degraded component | After each node reboot |
| 3 | edge2 host PVE 8.4 reboot — CONDITIONAL | Only if Phase 0 confirmed kernel fix needed; own no-failover window AFTER CT105/CT107 verified | vzdump all guests + `pveversion` | scoped kernel/security upgrade → reboot in approved window | `uname -r`=fixed; all CTs return; Caddy-fronted services reachable | GRUB prior kernel; vzdump restore | GATED on Phase 0; announce SSO+tailnet downtime; non-tailnet path |
| — | Nominatim 4.5→5.3 + Photon 1.1→1.2 (SEPARATE PROJECT) | Full re-import into separate DB/instance; keep 4.5 until validated; Photon AFTER nominatim validated | `qm snapshot 1130` + retain 4.5 image/data | import to new DB; size flat-nodes vs free space (NOT on data); cut over after verify | 5.x geocode correct; Photon 1.2 reads v5 | re-pin nominatim:4.5 + old data | Wholly separate from patch campaign; after disk remediation |
- **One corosync cluster.** data, utility, cloud, media, toc are `echo6-cluster` — quorum 3 of 5. Every Phase 3 reboot is gated on `pvecm status` (Quorate:Yes, expected votes=5); ONE node at a time; confirm 5/5 rejoin before next.
- **toc coupling.** toc is the 5th cluster vote + GPU host for cortex VM150 + the management/Claude Code host. toc+cortex are the FINAL Phase 3 action, alone — never overlap another node reboot.
- **Headscale topology (resolved).** Main fleet tailnet (34 nodes, ControlURL `vpn.echo6.co`) = edge2 CT107 (Headscale 0.28.0) — HIGH lockout risk; upgrade LAST, drive from out-of-band. Separate IdahoMesh sub-tailnet (`vpn.idahomesh.com`, 3 nodes) = utility CT106 — low risk; upgrade FIRST as rehearsal. No routing through old-Contabo.
### Top risks (critical first)
1.**Headscale edge2 CT107 self-lockout** (CRITICAL) — upgrading the main tailnet over the tailnet itself loses the control path to 34 nodes. Mitigation: out-of-band only (edge2 host console / direct SSH 184.174.35.153), key-expiry extended, DB dumped off-tailnet, boot chain verified.
2.**Corosync quorum loss** (CRITICAL) — two nodes down simultaneously = quorum lost, `/etc/pve` read-only fleet-wide. Mitigation: one node at a time, `pvecm status` gated before each, HA disabled during window.
3.**Authentik SSO migration chain** (CRITICAL) — 4-hop irreversible Django migration sequence; a failed hop with no per-hop pg_dump can strand the entire SSO surface. Mitigation: pg_dump per hop, verified-login gate, break-glass proven before starting.
4.**data node snapshot-space exhaustion** (CRITICAL) — 92% full; a snapshot failure mid-upgrade leaves the guest in a half-upgraded, unrollbackable state. Mitigation: disk remediated to <85% before any snapshot on that node; do not use as Phase 3 canary until resolved.
5.**RabbitMQ EOL/won't-boot** (HIGH) — 3.12→4.x direct jump refuses to start; 3.x has no security backports. Mitigation: staged hops (3.12→3.13.x→feature-flags→4.x), decoupled from OTS updater.
6.**Forward-only DB migrations / false-rollback assumption** (HIGH) — "redeploy previous image tag" is NOT a valid rollback for authentik, forgejo, nextcloud, peertube, immich after a migration runs. Rollback = quiesced logical dump + old binary together.
7.**toc reboot drops cortex + management host** (HIGH) — cortex is the Claude Code host; losing it during diagnosis is double-jeopardy. Mitigation: alternate control point tested before window; cortex NVIDIA/DKMS verified before toc reboot; toc+cortex done last and alone.
8.**pi-nas non-boot after combined kernel+OMV change** (HIGH) — RPi has no remote KVM; a bad combined update requires on-site SD reflash. Mitigation: two separate reboots (OMV first, kernel second); retain 6.12; `/boot`+`/boot/firmware` backed up; physical access confirmed before each.
- **Complete:** all five PVE-9 nodes (data, utility, cloud, media, toc), edge2, cortex, pi-nas, and all discovered guests/containers.
- **Excluded:** edge1 (the rebuilt ex-Contabo, mail-only) — mid-rebuild/maintenance at time of audit; re-audit once it settles.
- **Incomplete:** WordPress (CT101) plugin and theme status — requires WP-CLI; not assessed.
- **Method:** read-only throughout — SSH / `pct exec`, `apt list --upgradable`, `docker manifest inspect` for same-tag drift. Floating/pinned-tag caveats noted inline where drift could not be confirmed.
