matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-06-21T00:00:05+00:00

Browse source 
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/projects/fleet-patch-audit.md
This commit is contained in:
echo6-autocommit 2026-06-21 00:00:05 +00:00
commit 5bcbe073c1
3 changed files with 22 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

2
engine/lint-report.md
View file

@ -1,6 +1,6 @@
# Vault Lint Report
Generated: 2026-06-20T12:00:06Z | Docs scanned: 92 | Elapsed: 0.0s
Generated: 2026-06-20T18:00:05Z | Docs scanned: 92 | Elapsed: 0.0s
## Summary

8
vault/.obsidian/workspace.json vendored
View file

@ -199,6 +199,10 @@
},
"active": "8d53cdb6c257e685",
"lastOpenFiles": [
"projects/fleet-patch-audit.md.tmp.1493418.23b2caf55e12",
"projects/fleet-patch-audit.md.tmp.1493418.f30558958079",
"projects/fleet-patch-audit.md.tmp.1493418.5fe5d2d39856",
"projects/fleet-patch-audit.md.tmp.1493418.6c2a45ea0ecd",
"projects/fleet-patch-audit.md.tmp.1493418.9a000c8ac5aa",
"projects/fleet-patch-audit.md.tmp.1493418.c361bd10be1b",
"projects/fleet-patch-audit.md.tmp.1493418.83a57813be85",
@ -206,10 +210,6 @@
"projects/fleet-patch-audit.md.tmp.1493418.d191aac142fc",
"projects/fleet-patch-audit.md.tmp.1493418.6777b1d5dff6",
"projects/fleet-patch-audit.md.tmp.1493418.731843387669",
"projects/fleet-patch-audit.md.tmp.1493418.b5aeb852fc62",
"projects/fleet-patch-audit.md.tmp.1493418.425c00fcc97b",
"projects/fleet-patch-audit.md.tmp.1493418.61ad7f49d904",
"projects/fleet-patch-audit.md.tmp.1493418.f6bb16e19902",
"projects/nominatim-v5-reimport.md",
"2026-06-19.md",
"Untitled.canvas",

23
vault/projects/fleet-patch-audit.md
View file

@ -229,7 +229,7 @@ Lowest-risk changes first; everything reboot-bearing deferred to scheduled windo
- **(c) PostgreSQL on central CT104 moved 16.13→16.14** as part of the security-pocket apt pass.
- **(d) Fleet-wide stale `/etc/hosts` fix** — see Critical Finding 2.
### 🔴 Critical Finding 1 — CT107 headscale does NOT survive a reboot
### ✅ Critical Finding 1 — CT107 headscale reboot-survival — FIXED 2026-06-20
On `pct reboot 107`, the fleet headscale coordinator crash-looped and required approximately 15 minutes of manual recovery. Two root causes:
@ -238,7 +238,17 @@ On `pct reboot 107`, the fleet headscale coordinator crash-looped and required a
Recovery required: `compose down` → `docker restart` → manually add `100.64.0.38/32` to tailscale0 → `compose up` → re-auth CT107's own tailscale node with a fresh preauthkey → temporarily DNAT `vpn.echo6.co` through CT107's internal IP to bootstrap, then revert Caddy config and clean up iptables.
**IMPACT: This is now the #1 Phase-3 blocker.** edge2 must NOT be rebooted (and CT107 must not be restarted) until a permanent boot-survival fix exists — for example, a systemd oneshot that recreates the bridge and seeds the tailscale0 IP before headscale starts, or removing headscale's bind-to-tailnet-IP dependency. A naive edge2 reboot takes the whole fleet coordinator down.
**Resolution (2026-06-20):** Root cause was a triple boot deadlock: (a) headscale/headplane Docker ports were bound to the tailscale IP `100.64.0.38` (which only exists after tailscaled connects to headscale — circular); (b) the `headscale-stack.service` systemd unit waited on `tailscale-online.target` (also needs headscale up); (c) `only_start_if_oidc_is_available: true` blocked startup on Authentik reachability.
Fix applied (all originals backed up `.bak-20260620`):
1. Rebound both ports `100.64.0.38:8084/3100` → `10.10.10.25:8084/3100` (CT's static internal IP, always up at boot) in `/opt/headscale/docker-compose.yml`.
2. `headscale-stack.service` → `After=docker.service` only (dropped `tailscale-online.target` dependency).
3. `only_start_if_oidc_is_available: false` in `/opt/headscale/config.yaml`.
4. Repointed edge2 `/etc/caddy/Caddyfile` `vpn.echo6.co` upstream (and `/admin*` headplane) to `10.10.10.25`.
**PROVEN by a controlled `pct reboot 107`: headscale self-healed in ~45 seconds with ZERO manual intervention** — `/health` 200, RestartCount 0, clean logs, 32 fleet nodes online.
Known minor leftover (cosmetic, non-blocking): CT107's OWN tailscale client node (100.64.0.38) stays in NoState because its tailscaled ControlURL points at its own IP (`http://100.64.0.38:8084`) — a self-bootstrap chicken-and-egg. The headscale SERVICE is fully healthy and the fleet is coordinated; CT107 is managed via the edge2 host OOB path, so its own tailnet membership is optional. Candidate follow-up: point CT107 tailscaled ControlURL at `https://vpn.echo6.co` (now always reachable via Caddy→10.10.10.25) so it self-registers cleanly.
### 🔴 Critical Finding 2 — fleet-wide stale /etc/hosts broke coordinator connectivity
@ -363,7 +373,7 @@ Vaultwarden, PDM, WordPress, Synapse/Element/MAS, obsidian, recon-vm Postgres
### Phase 3 — Platform & reboot windows (coordinated, approval-gated)
> **🔴 BLOCKED — edge2 reboot is NOT permitted until the CT107 headscale boot-survival fix is in place (see Critical Finding 1 in the Phase 1 Execution Log).** A naive edge2 reboot crash-loops the fleet coordinator and requires ~15 min manual recovery. Do not schedule or start any edge2 reboot until a permanent fix exists (systemd oneshot to recreate the bridge / seed tailscale0 IP, or equivalent).
> ~~🔴 BLOCKED — edge2 reboot was NOT permitted until the CT107 headscale boot-survival fix was in place.~~ **✅ CLEARED 2026-06-20 — the CT107 headscale boot-survival fix is applied and proven** (see Critical Finding 1 in the Phase 1 Execution Log). The triple boot deadlock (tailscale-IP port bind + tailscale-online.target wait + OIDC gate) has been resolved; headscale now self-heals in ~45 seconds after a CT107 or edge2 reboot with zero manual intervention. The Phase-3 edge2 reboot blocker is lifted. The remaining gates below (corosync quorum + Finding-2 headscale-ONLINE check per node) still apply.
> **🔴 GATE for every Phase-3 host reboot (Critical Finding 2):** Before AND after rebooting any node, confirm that node is ONLINE in the headscale coordinator (`headscale nodes list`). Stale `/etc/hosts` entries previously masked coordinator disconnects behind coasting WireGuard tunnels — the stale entries have been removed fleet-wide, but verify ONLINE status at each reboot step to catch any regression. This gate is IN ADDITION TO the corosync-quorum gate below.
@ -398,8 +408,9 @@ Vaultwarden, PDM, WordPress, Synapse/Element/MAS, obsidian, recon-vm Postgres
- Reboot 2 (AFTER reboot 1 verified): kernel 6.12→6.18 — `apt install linux-image-arm64` (unhold); update bootloader; reboot ON APPROVAL → verify `uname -r`=6.18; LAN+tailnet up; shares mount; Docker up. Rollback: boot retained 6.12; restore `/boot`; on-site SD reflash.
- Independent of cluster windows.
**edge2 — CONDITIONAL:**
- Reboot only if Phase 0 resolved the DirtyFrag/copy.fail CVE question as "yes, needs a newer kernel." If reboot required: give it a dedicated window AFTER Authentik CT105 + headscale CT107 are upgraded and verified; announce SSO+tailnet downtime; drive from a non-tailnet path; confirm all CTs auto-start. edge2 is a single SPOF for ingress — no failover. Do NOT fold into cluster windows.
**edge2 — CONDITIONAL (CT107 boot-survival blocker CLEARED 2026-06-20):**
- ~~Previously also blocked on the CT107 headscale boot-survival issue~~ — that blocker is resolved; headscale survives a CT107 or edge2 reboot and self-heals in ~45 seconds (see Critical Finding 1, fixed 2026-06-20).
- Reboot only if Phase 0 resolved the DirtyFrag/copy.fail CVE question as "yes, needs a newer kernel" (Open Decision #5 recorded "no action / no reboot" — edge2 host already fully patched). If a reboot is ever required: give it a dedicated window AFTER Authentik CT105 + headscale CT107 are upgraded and verified; announce SSO+tailnet downtime; drive from a non-tailnet path; confirm all CTs auto-start. edge2 is a single SPOF for ingress — no failover. Do NOT fold into cluster windows.
### Phase 4 — Cross-cutting (fold into earlier phases)
@ -479,7 +490,7 @@ Vaultwarden, PDM, WordPress, Synapse/Element/MAS, obsidian, recon-vm Postgres
| 3 | cortex final stack health verification | Post-window full AI-stack + engine check | keep pre-window snaps until verified | verification only | driver 580.167; all 5 containers; ollama/tei/qdrant/open-webui; vault-tagger+TEI engines ok; TS online; Docker 29.6 | restore degraded component; worst case VM150 vzdump | LAST action of node group |
| 3 | pi-nas OMV 8.1→8.4 (reboot #1, separate window) | OMV native update path (NOT raw apt) → reboot | off-box `config.xml` export + plugin/layout record | OMV UI Update Mgmt or `omv-upgrade`; reboot | OMV=8.4; shares/SMB/NFS/RAID/mergerfs healthy; client mounts | restore config.xml + `dpkg --set-selections` 8.1 + `omv-salt deploy` | Physical access; not mid-sync; after Phase 1 |
| 3 | pi-nas kernel 6.12→6.18 (reboot #2, separate window) | Kernel bump → reboot; activates deferred security | back up `/boot`+`/boot/firmware`; keep 6.12 installed; fresh config.xml | `apt install linux-image-arm64` (unhold); update bootloader; reboot ON APPROVAL | `uname -r`=6.18; LAN+tailnet up; shares mount; Docker up | boot retained 6.12; restore `/boot`; on-site SD reflash | After OMV reboot verified; physical access |
| 3 | edge2 host PVE 8.4 reboot — CONDITIONAL | Only if Phase 0 confirmed kernel fix needed; own no-failover window AFTER CT105/CT107 verified | vzdump all guests + `pveversion` | scoped kernel/security upgrade → reboot in approved window | `uname -r`=fixed; all CTs return; Caddy-fronted services reachable | GRUB prior kernel; vzdump restore | GATED on Phase 0; announce SSO+tailnet downtime; non-tailnet path |
| 3 | edge2 host PVE 8.4 reboot — CONDITIONAL (CT107 boot-survival blocker CLEARED 2026-06-20) | ~~Previously blocked on CT107 headscale boot-survival~~ — cleared (self-heals in ~45s). Now gated only on Phase 0 kernel-CVE decision (Open Dec #5 = no reboot needed — host fully patched). If ever required: own no-failover window AFTER CT105/CT107 verified | vzdump all guests + `pveversion` | scoped kernel/security upgrade → reboot in approved window | `uname -r`=fixed; all CTs return; Caddy-fronted services reachable | GRUB prior kernel; vzdump restore | GATED on Phase 0; announce SSO+tailnet downtime; non-tailnet path |
| — | Nominatim 4.5→5.3 + Photon 1.1→1.2 (SEPARATE PROJECT) | Full re-import into separate DB/instance; keep 4.5 until validated; Photon AFTER nominatim validated | `qm snapshot 1130` + retain 4.5 image/data | import to new DB; size flat-nodes vs free space (NOT on data); cut over after verify | 5.x geocode correct; Photon 1.2 reads v5 | re-pin nominatim:4.5 + old data | Wholly separate from patch campaign; after disk remediation |
---