From 72a8cd4f4501ef9f18dc813916a181ee5a7c8e2d Mon Sep 17 00:00:00 2001 From: echo6-autocommit Date: Thu, 9 Jul 2026 00:00:07 +0000 Subject: [PATCH] auto: docs sync 2026-07-09T00:00:07+00:00 Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/projects/meshai-native-fire-severity-audit-cc-handoff.md --- engine/lint-report.md | 2 +- vault/.obsidian/workspace.json | 2 +- ...i-native-fire-severity-audit-cc-handoff.md | 260 ++++++++++++++++++ 3 files changed, 262 insertions(+), 2 deletions(-) create mode 100644 vault/projects/meshai-native-fire-severity-audit-cc-handoff.md diff --git a/engine/lint-report.md b/engine/lint-report.md index bc09b2e..9d353e7 100644 --- a/engine/lint-report.md +++ b/engine/lint-report.md @@ -1,6 +1,6 @@ # Vault Lint Report -Generated: 2026-07-08T12:00:08Z | Docs scanned: 104 | Elapsed: 0.0s +Generated: 2026-07-08T18:00:11Z | Docs scanned: 104 | Elapsed: 0.0s ## Summary diff --git a/vault/.obsidian/workspace.json b/vault/.obsidian/workspace.json index 2b3ae91..90c43c3 100644 --- a/vault/.obsidian/workspace.json +++ b/vault/.obsidian/workspace.json @@ -199,6 +199,7 @@ }, "active": "17bd4a6166f789d0", "lastOpenFiles": [ + "projects/meshai-native-fire-severity-audit-cc-handoff.md", "projects/meshai-config-hot-apply.md", "projects/meshai-config-hot-apply.md.tmp.5281.28f86510539f", "projects/meshai-region-routing-plan.md.tmp.5281.292f05969651", @@ -235,7 +236,6 @@ "projects/mmud-project.md", "concepts/lxc-container.md", "concepts/osint.md", - "concepts/split-dns.md", "assets/echo6yellow_logo_422x422_square.png", "assets/echo6yellow_logo_422x81.png", "assets/echo6_logo.png", diff --git a/vault/projects/meshai-native-fire-severity-audit-cc-handoff.md b/vault/projects/meshai-native-fire-severity-audit-cc-handoff.md new file mode 100644 index 0000000..3e1d095 --- /dev/null +++ b/vault/projects/meshai-native-fire-severity-audit-cc-handoff.md @@ -0,0 +1,260 @@ +# MeshAI Audit Handoff — Native Fire Severity Regression + +**For:** Claude Code (CC), operating from cortex against the MeshAI LXC (192.168.1.144) +**Repo:** zvx-echo6/meshai (deployed via docker compose on the LXC) +**Date of audit:** 2026-07-08, against HEAD `aef9877` (#95) +**Scope:** One minimal backend fix in `work/meshai/env/store.py`, plus a mandatory DB pre-seed. Nothing else. No drive-by refactors. + +--- + +## 0. Standing operational rules (non-negotiable) + +1. **SSH prereq gate.** Before ANY work: verify SSH from cortex to the MeshAI LXC (192.168.1.144) using the configured method in `~/.ssh/config`. If auth fails, STOP and report. Never assume SSH works. +2. **Never use `docker compose run`.** Ever. Use `docker exec` against the running container, or host tooling. +3. **Verbatim diffs only.** Apply the diff in §10 exactly. If the anchor lines don't match, STOP and report — do not improvise. +4. **Plain-English explanation before every commit.** Matt's understanding is the gate. Present the summary, wait for his confirmation, then commit. +5. **Every status report back to Matt ends with `sudo docker ps`.** +6. **Rebuild/deploy is ONLY:** `sudo docker compose build meshai && sudo docker compose up -d`. +7. **Mesh-spam stop gate** — see §1. Binding. + +--- + +## 1. ⚠️ MESH-SPAM STOP GATE — BINDING + +Deploying the fix in this document **will release a burst of fire broadcasts** unless the pre-seed in §9 is completed and verified first. + +Mechanics: every fire currently blocked by this bug retries its "New" broadcast **on every poll** and dies silently at the dispatcher severity floor. Because delivery never succeeds, the decider's commit never runs, so its `fires` row keeps `last_broadcast_at = NULL`. The first poll after the fixed code starts will pass **every one of those fires at once** — and there is no pacer on the native path (`FirePacer` is wired to the Central consumer only, `main.py:128–136`; its own docstring says it serves the Central reconnect/drain path). + +**Rule:** Phase −1 (§7) and Phase 0 (§8) are read-only and may run immediately. **Do not execute anything in §9 onward until Matt replies verbatim: `I know and I accept`.** When you reach that point, report the burst size from Phase 0 step D and ask for the ack explicitly. Even with the pre-seed done, note to Matt that legitimate post-fix fire traffic will begin (that's the point), and multiple genuinely-new fires arriving in one poll batch will emit together. + +--- + +## 2. Symptom + +Since the central→native swap (Jul 4–6), the mesh carries only WZDx, NWS weather alerts, satpass, and band conditions. Fire alerts — which should be active in southern Idaho in July — are silent. (Avalanche silence is seasonal and expected; it is NOT part of this problem.) + +## 3. Root cause — plain English + +The central→native swap dropped the decider's severity promotion, and native fire events now die at the dispatcher's `min_severity` floor — silently, with no log line and no drop counter. + +Evidence chain (all verified by reading the code at HEAD): + +1. **Central path (old behavior):** `central/wfigs_handler.py:238, 348` stamped `data["_severity_override"] = "priority"` on every fire New/Update. `central/consumer.py:649–654` promoted it onto the Event: `severity=sev_override or map_severity(...)`. Every fire reached the dispatcher as **priority**. +2. **Native adapter severity:** `env/fires.py` (~lines 139–148) assigns severity by anchor proximity — `"priority"` only within 25 km of a region anchor, **`"routine"` otherwise**. `to_event` (line 333+) passes it through unchanged. +3. **The decider still stamps the override:** `notifications/gating/fire.py::_broadcast_patch` sets `"_severity_override": "priority"` on every New and Update GateResult. The intent is unchanged from Central. +4. **The promotion is lost:** `env/store.py::_emit_event` (line 678+) builds the Event from `adapter.to_event()` FIRST, then merges the decider patch into **`event.data` only** (`event.data.update(gate.data_patch)`, ~line 731). Nothing anywhere in the native delivery path promotes `_severity_override` onto `event.severity` — verified by grep across dispatcher, composer, and events modules: zero consumers. +5. **The floor:** `notifications/pipeline/dispatcher.py` Section 2 (~line 558): `if event_rank < SEVERITY_RANK[tog.min_severity]: return` — a bare return. **No log, no counter** (the four persisted counters cover cold-start/stale/cooldown/dedup, not the severity floor). The matrix path (Section 1.5) has an equivalent per-cell floor with an "authoritative no-send" return. +6. **The default floor is "priority":** `config.py:641` (`NotificationToggle.min_severity: str = "priority"`) and again at the factory default, `config.py:681`. + +Net: routine-severity fire → silent drop, every poll, forever. Fire is forced through the decider regardless of cutover state (`NATIVE_ALWAYS_DECIDE`, `notifications/cutover.py:44`), so every native fire New/Update is affected. + +**Timeline:** WFIGS→decider migration `8bc9b14` (Jul 4, 22:51), received-delta gate `0c76a80` (Jul 5), native WFIGS routed through decider `8d61b16` (Jul 6, 15:35). Fire silence began at the native flip — matches the operator's report exactly. + +## 4. Why the surviving feeds survive + +They handle severity in-adapter, so they never depended on the lost promotion: +- `env/satpass.py:250` — reads `severity=data.get("_severity_override", "routine")` itself. **This is the correct pattern the fix replicates at the store level.** +- `env/nws.py:39` — derives severity in-adapter via `_map_nws_severity` from CAP severity; warnings clear the floor. +- `env/wzdx.py:365` — full closures are `"priority"`. +- Band conditions — scheduled broadcaster (`BandConditionsScheduler`) that calls the dispatcher directly. + +Of the six deciders that emit `_severity_override` (fire, nws, quake, swpc, avalanche, firms), only **firms** (`env/firms.py:456`) and **satpass** consume it in-adapter. + +## 5. Ruled out during the audit + +- Avalanche: out of season; expected silence. Not a bug. +- Received-delta gate (`_delta_emit`, commit `0c76a80`): fires deliberately bypass it (`store.py::_ingest`, `nifc` branch) — the decider is the fire gate. Not the drop point. +- FirePacer black hole: pacer is Central-consumer-only; native fires go straight to the bus. Not a factor. +- Toggle mapping: `wildfire_declared`/`wildfire_incident` both map to the `fire` toggle (`notifications/categories.py:~380–390`). Not a factor. +- Coverage filter: fail-open for non-weather categories; fires carry centroids. Not the primary drop. +- ToggleFilter/inhibitor/grouper: all log their drops; fire family is presumed enabled (verify in Phase 0). + +## 6. Secondary findings — explicitly OUT OF SCOPE for this change (backlog) + +1. **Restart-window ignition swallowing.** `_fires_seeded` is per-process (`store.py::_ingest_fires`); every restart's first poll silent-seeds any fire that first appeared while the container was down — it is never announced as New. Deliberate anti-backlog design, but during dense deploy periods in fire season it eats real ignitions. Needs a design decision (e.g., persisted seen-baseline), not a hotfix. +2. **Zero-acre New fires deferred.** `env/fires.py::to_event` returns None when `not acres`; fresh WFIGS ignitions often report 0 acres initially. Self-heals when acreage posts (row exists with NULL `last_broadcast_at` → decider fires "New"), but delays first announcements. +3. **Same bug class armed for quake/swpc/avalanche/nws.** Their deciders emit `_severity_override` with no native consumer; currently masked behind `MESHAI_CUTOVER_CATEGORIES` shadow gating. The §10 fix resolves this at the shared choke point for all cut-over categories too. +4. **No native fire pacer.** Multiple legit new fires in one poll batch will emit back-to-back. Consider extending FirePacer to the native emit path later. + +--- + +## 7. Phase −1 — SSH prereq gate (mandatory first step) + +From cortex: + +```bash +ssh @192.168.1.144 'echo SSH_OK && hostname' +``` + +If this fails for any reason: STOP. Report to Matt. Do not proceed. + +Then locate the deployment on the LXC (do not guess paths): + +```bash +sudo docker ps --format '{{.Names}}\t{{.Image}}' | grep -i meshai +find /opt /root /home -maxdepth 3 -name docker-compose.yml 2>/dev/null | xargs grep -l meshai 2>/dev/null +``` + +Record the compose project directory and the container name (assumed `meshai` below — substitute if different). Confirm `git status` in the repo checkout is clean before any edit; if dirty, STOP and report. + +## 8. Phase 0 — read-only diagnostics (safe to run now) + +Run all of these and report results to Matt before anything else. Determine the DB path first: + +```bash +sudo docker exec meshai ls /app/data +``` + +**A. Are fire events reaching the bus and dying downstream?** (Diagnosis predicts: repeatedly nonzero.) + +```bash +sudo docker logs meshai --since 24h 2>&1 | grep -c "Emitted nifc" +sudo docker logs meshai --since 24h 2>&1 | grep "Emitted nifc" | tail -5 +``` + +**B. Live fire family config — the floor:** + +```bash +sudo docker exec meshai grep -B2 -A8 -i "fire" /app/config/config.yaml | grep -i -B1 -A4 "min_severity\|enabled\|region" +``` + +**C. Cutover state (determines blast radius of the bug class):** + +```bash +sudo docker exec meshai printenv MESHAI_CUTOVER_CATEGORIES +``` + +**D. Burst size if fixed today** — this number goes in the stop-gate report to Matt: + +```bash +sudo docker exec meshai sqlite3 /app/data/ \ + "SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;" +sudo docker exec meshai sqlite3 /app/data/ \ + "SELECT irwin_id, incident_name, current_acres, current_contained_pct, last_broadcast_at + FROM fires ORDER BY last_event_at DESC LIMIT 20;" +``` + +**E. Dispatcher drop counters** (severity-floor drops are NOT counted — expected to look quiet; that's part of the finding): + +```bash +sudo docker exec meshai sqlite3 /app/data/ "SELECT * FROM dispatcher_state;" +``` + +**Decision matrix:** +- A nonzero AND B shows fire `min_severity: priority` (or matrix fire cells with priority floors) → **diagnosis confirmed.** Proceed to the stop gate. +- A is ZERO → fires aren't reaching the bus at all; the diagnosis does NOT hold as-is. STOP. Report findings; do not apply the fix. (Check adapter enablement, NIFC fetch errors: `sudo docker logs meshai --since 24h 2>&1 | grep -i "nifc\|wfigs" | tail -20`.) +- A nonzero but B shows fire floor already `routine` → drop is elsewhere (matrix cells or region scope). STOP. Report B plus the matrix config; do not apply the fix blind. + +End the Phase 0 report with `sudo docker ps`. + +--- + +## ⛔ EVERYTHING BELOW REQUIRES MATT'S VERBATIM ACK: `I know and I accept` + +Report the Phase 0 results including the §8-D burst count, restate the risk (§1), and wait. + +## 9. Phase 1 — pre-seed the fires table (mandatory BEFORE the fixed code ever starts) + +Run against the **still-running old container** (old code keeps suppressing at the floor, so this changes nothing live; after pre-seed the decider simply returns cooldown for those rows): + +```bash +sudo docker exec meshai sqlite3 /app/data/ " +UPDATE fires + SET last_broadcast_at = strftime('%s','now'), + first_broadcast_at = COALESCE(first_broadcast_at, strftime('%s','now')), + last_broadcast_acres = current_acres, + last_broadcast_contained = current_contained_pct + WHERE last_broadcast_at IS NULL;" +``` + +Verify — MUST return 0 before proceeding: + +```bash +sudo docker exec meshai sqlite3 /app/data/ \ + "SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;" +``` + +If `sqlite3` is not present in the image: stop the container, run the same SQL with host `sqlite3` against the bind-mounted data directory (check the volume mapping in docker-compose.yml), verify count 0, and only then continue. Do NOT start the new image before the verify shows 0. + +Consequence to state to Matt: currently-active fires will not get a retroactive "New" announcement; they will broadcast on their next real growth/containment change. Only genuinely new post-deploy ignitions announce as New. + +## 10. Phase 2 — the fix (verbatim diff) + +Target: `work/meshai/env/store.py`, inside `_emit_event` (function starts line 678). Anchor by grepping `event.data.update(gate.data_patch)` — it occurs once. Verify both anchor lines exist verbatim, then apply exactly: + +```diff +--- a/work/meshai/env/store.py ++++ b/work/meshai/env/store.py +@@ def _emit_event(self, adapter, raw_evt: dict): + # Apply data_patch into event.data + event.data.update(gate.data_patch) ++ # Promote decider overrides onto the Event itself, mirroring ++ # the Central path (central/consumer.py:649-653). The gating ++ # deciders stamp _severity_override (fire: "priority" on every ++ # New/Update) and, for fire New/tombstone, a category ++ # override. Merging them into event.data alone leaves ++ # event.severity at the adapter's value ("routine" for fires ++ # >=25 km from an anchor), which silently fails the ++ # toggle/matrix min_severity floors (default "priority", ++ # config.py:641). Native and Central must share identical ++ # broadcast decisions. ++ _sev_override = gate.data_patch.get("_severity_override") ++ if _sev_override: ++ event.severity = _sev_override ++ _cat_override = gate.data_patch.get("category") ++ if _cat_override: ++ event.category = _cat_override + if gate.commit is not None: + event.data["_on_broadcast_committed"] = gate.commit +``` + +Notes for review: +- `Event` is a plain mutable `@dataclass` (`notifications/events.py:38`); attribute assignment is valid. +- Category promotion is intentional and safe: `wildfire_declared` maps to the `fire` toggle (`categories.py:~387`); formatter branch selection already falls back to `data["category"]`; cooldown keys become category-scoped, matching Central behavior (New vs Update cooldowns separate). +- This is the shared choke point: any category later added to `MESHAI_CUTOVER_CATEGORIES` (quake, swpc, avalanche, nws) gets the correct promotion automatically. + +**Tests:** if a pytest environment exists on the LXC checkout, run the targeted subsets and compare against the known 10-failure baseline (per commit `0c76a80`, 1697 passed): + +```bash +python3 -m pytest work/tests -k "fire or store or dispatch or gating" -q +``` + +If no local pytest env exists, note that in the report and proceed — the change is 16 lines with fully-traced semantics. Do NOT use `docker compose run` to run tests. + +## 11. Phase 3 — commit and deploy + +1. Present Matt a plain-English summary of the change (what/why in 3–5 sentences, drawn from §3). Wait for his confirmation. His understanding is the gate. +2. Commit with a message in the repo's convention, e.g.: + `fix(native): promote decider _severity_override/category onto Event — restores Central-parity fire severity (silent min_severity floor drop)` +3. Deploy exactly: + +```bash +sudo docker compose build meshai && sudo docker compose up -d +``` + +## 12. Phase 4 — post-deploy verification + +Watch the first 2–3 NIFC poll cycles (read the poll interval from the wfigs/nifc adapter config during Phase 0): + +```bash +sudo docker logs -f meshai 2>&1 | grep -Ei "Emitted nifc|matrix: dispatched|dispatched|SUPPRESSED|cold-start" +``` + +Expected: NO burst at startup (pre-seed verified 0); cold-start seed lines for any brand-new first-sight fires; subsequent genuine New/growth events showing `matrix: dispatched` or toggle-path delivery lines. Confirm on-air on the mesh when the first legit event lands. + +Then confirm the counters are stable and report: + +```bash +sudo docker exec meshai sqlite3 /app/data/ "SELECT * FROM dispatcher_state;" +sudo docker ps +``` + +Every report back to Matt ends with `sudo docker ps`. + +## 13. Backlog (do NOT do now — list for future sessions) + +1. Persist the fire seen-baseline across restarts to close the restart-window ignition gap (§6.1). +2. Decide policy for zero-acre first sightings (§6.2). +3. Extend pacing to the native emit path for multi-fire poll batches (§6.4). +4. Before cutting over quake/swpc/avalanche/nws via `MESHAI_CUTOVER_CATEGORIES`, re-verify override promotion end-to-end (covered by this fix, but confirm with shadow logs). +5. Add a drop counter + log line for the dispatcher severity-floor return (it is currently the only fully silent drop in the pipeline).