# MeshAI Audit Handoff — Native Fire Severity Regression **For:** Claude Code (CC), operating from cortex against the MeshAI LXC (192.168.1.144) **Repo:** zvx-echo6/meshai (deployed via docker compose on the LXC) **Date of audit:** 2026-07-08, against HEAD `aef9877` (#95) **Scope:** One minimal backend fix in `work/meshai/env/store.py`, plus a mandatory DB pre-seed. Nothing else. No drive-by refactors. --- ## 0. Standing operational rules (non-negotiable) 1. **SSH prereq gate.** Before ANY work: verify SSH from cortex to the MeshAI LXC (192.168.1.144) using the configured method in `~/.ssh/config`. If auth fails, STOP and report. Never assume SSH works. 2. **Never use `docker compose run`.** Ever. Use `docker exec` against the running container, or host tooling. 3. **Verbatim diffs only.** Apply the diff in §10 exactly. If the anchor lines don't match, STOP and report — do not improvise. 4. **Plain-English explanation before every commit.** Matt's understanding is the gate. Present the summary, wait for his confirmation, then commit. 5. **Every status report back to Matt ends with `sudo docker ps`.** 6. **Rebuild/deploy is ONLY:** `sudo docker compose build meshai && sudo docker compose up -d`. 7. **Mesh-spam stop gate** — see §1. Binding. --- ## 1. ⚠️ MESH-SPAM STOP GATE — BINDING Deploying the fix in this document **will release a burst of fire broadcasts** unless the pre-seed in §9 is completed and verified first. Mechanics: every fire currently blocked by this bug retries its "New" broadcast **on every poll** and dies silently at the dispatcher severity floor. Because delivery never succeeds, the decider's commit never runs, so its `fires` row keeps `last_broadcast_at = NULL`. The first poll after the fixed code starts will pass **every one of those fires at once** — and there is no pacer on the native path (`FirePacer` is wired to the Central consumer only, `main.py:128–136`; its own docstring says it serves the Central reconnect/drain path). **Rule:** Phase −1 (§7) and Phase 0 (§8) are read-only and may run immediately. **Do not execute anything in §9 onward until Matt replies verbatim: `I know and I accept`.** When you reach that point, report the burst size from Phase 0 step D and ask for the ack explicitly. Even with the pre-seed done, note to Matt that legitimate post-fix fire traffic will begin (that's the point), and multiple genuinely-new fires arriving in one poll batch will emit together. --- ## 2. Symptom Since the central→native swap (Jul 4–6), the mesh carries only WZDx, NWS weather alerts, satpass, and band conditions. Fire alerts — which should be active in southern Idaho in July — are silent. (Avalanche silence is seasonal and expected; it is NOT part of this problem.) ## 3. Root cause — plain English The central→native swap dropped the decider's severity promotion, and native fire events now die at the dispatcher's `min_severity` floor — silently, with no log line and no drop counter. Evidence chain (all verified by reading the code at HEAD): 1. **Central path (old behavior):** `central/wfigs_handler.py:238, 348` stamped `data["_severity_override"] = "priority"` on every fire New/Update. `central/consumer.py:649–654` promoted it onto the Event: `severity=sev_override or map_severity(...)`. Every fire reached the dispatcher as **priority**. 2. **Native adapter severity:** `env/fires.py` (~lines 139–148) assigns severity by anchor proximity — `"priority"` only within 25 km of a region anchor, **`"routine"` otherwise**. `to_event` (line 333+) passes it through unchanged. 3. **The decider still stamps the override:** `notifications/gating/fire.py::_broadcast_patch` sets `"_severity_override": "priority"` on every New and Update GateResult. The intent is unchanged from Central. 4. **The promotion is lost:** `env/store.py::_emit_event` (line 678+) builds the Event from `adapter.to_event()` FIRST, then merges the decider patch into **`event.data` only** (`event.data.update(gate.data_patch)`, ~line 731). Nothing anywhere in the native delivery path promotes `_severity_override` onto `event.severity` — verified by grep across dispatcher, composer, and events modules: zero consumers. 5. **The floor:** `notifications/pipeline/dispatcher.py` Section 2 (~line 558): `if event_rank < SEVERITY_RANK[tog.min_severity]: return` — a bare return. **No log, no counter** (the four persisted counters cover cold-start/stale/cooldown/dedup, not the severity floor). The matrix path (Section 1.5) has an equivalent per-cell floor with an "authoritative no-send" return. 6. **The default floor is "priority":** `config.py:641` (`NotificationToggle.min_severity: str = "priority"`) and again at the factory default, `config.py:681`. Net: routine-severity fire → silent drop, every poll, forever. Fire is forced through the decider regardless of cutover state (`NATIVE_ALWAYS_DECIDE`, `notifications/cutover.py:44`), so every native fire New/Update is affected. **Timeline:** WFIGS→decider migration `8bc9b14` (Jul 4, 22:51), received-delta gate `0c76a80` (Jul 5), native WFIGS routed through decider `8d61b16` (Jul 6, 15:35). Fire silence began at the native flip — matches the operator's report exactly. ## 4. Why the surviving feeds survive They handle severity in-adapter, so they never depended on the lost promotion: - `env/satpass.py:250` — reads `severity=data.get("_severity_override", "routine")` itself. **This is the correct pattern the fix replicates at the store level.** - `env/nws.py:39` — derives severity in-adapter via `_map_nws_severity` from CAP severity; warnings clear the floor. - `env/wzdx.py:365` — full closures are `"priority"`. - Band conditions — scheduled broadcaster (`BandConditionsScheduler`) that calls the dispatcher directly. Of the six deciders that emit `_severity_override` (fire, nws, quake, swpc, avalanche, firms), only **firms** (`env/firms.py:456`) and **satpass** consume it in-adapter. ## 5. Ruled out during the audit - Avalanche: out of season; expected silence. Not a bug. - Received-delta gate (`_delta_emit`, commit `0c76a80`): fires deliberately bypass it (`store.py::_ingest`, `nifc` branch) — the decider is the fire gate. Not the drop point. - FirePacer black hole: pacer is Central-consumer-only; native fires go straight to the bus. Not a factor. - Toggle mapping: `wildfire_declared`/`wildfire_incident` both map to the `fire` toggle (`notifications/categories.py:~380–390`). Not a factor. - Coverage filter: fail-open for non-weather categories; fires carry centroids. Not the primary drop. - ToggleFilter/inhibitor/grouper: all log their drops; fire family is presumed enabled (verify in Phase 0). ## 6. Secondary findings — explicitly OUT OF SCOPE for this change (backlog) 1. **Restart-window ignition swallowing.** `_fires_seeded` is per-process (`store.py::_ingest_fires`); every restart's first poll silent-seeds any fire that first appeared while the container was down — it is never announced as New. Deliberate anti-backlog design, but during dense deploy periods in fire season it eats real ignitions. Needs a design decision (e.g., persisted seen-baseline), not a hotfix. 2. **Zero-acre New fires deferred.** `env/fires.py::to_event` returns None when `not acres`; fresh WFIGS ignitions often report 0 acres initially. Self-heals when acreage posts (row exists with NULL `last_broadcast_at` → decider fires "New"), but delays first announcements. 3. **Same bug class armed for quake/swpc/avalanche/nws.** Their deciders emit `_severity_override` with no native consumer; currently masked behind `MESHAI_CUTOVER_CATEGORIES` shadow gating. The §10 fix resolves this at the shared choke point for all cut-over categories too. 4. **No native fire pacer.** Multiple legit new fires in one poll batch will emit back-to-back. Consider extending FirePacer to the native emit path later. --- ## 7. Phase −1 — SSH prereq gate (mandatory first step) From cortex: ```bash ssh @192.168.1.144 'echo SSH_OK && hostname' ``` If this fails for any reason: STOP. Report to Matt. Do not proceed. Then locate the deployment on the LXC (do not guess paths): ```bash sudo docker ps --format '{{.Names}}\t{{.Image}}' | grep -i meshai find /opt /root /home -maxdepth 3 -name docker-compose.yml 2>/dev/null | xargs grep -l meshai 2>/dev/null ``` Record the compose project directory and the container name (assumed `meshai` below — substitute if different). Confirm `git status` in the repo checkout is clean before any edit; if dirty, STOP and report. ## 8. Phase 0 — read-only diagnostics (safe to run now) Run all of these and report results to Matt before anything else. Determine the DB path first: ```bash sudo docker exec meshai ls /app/data ``` **A. Are fire events reaching the bus and dying downstream?** (Diagnosis predicts: repeatedly nonzero.) ```bash sudo docker logs meshai --since 24h 2>&1 | grep -c "Emitted nifc" sudo docker logs meshai --since 24h 2>&1 | grep "Emitted nifc" | tail -5 ``` **B. Live fire family config — the floor:** ```bash sudo docker exec meshai grep -B2 -A8 -i "fire" /app/config/config.yaml | grep -i -B1 -A4 "min_severity\|enabled\|region" ``` **C. Cutover state (determines blast radius of the bug class):** ```bash sudo docker exec meshai printenv MESHAI_CUTOVER_CATEGORIES ``` **D. Burst size if fixed today** — this number goes in the stop-gate report to Matt: ```bash sudo docker exec meshai sqlite3 /app/data/ \ "SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;" sudo docker exec meshai sqlite3 /app/data/ \ "SELECT irwin_id, incident_name, current_acres, current_contained_pct, last_broadcast_at FROM fires ORDER BY last_event_at DESC LIMIT 20;" ``` **E. Dispatcher drop counters** (severity-floor drops are NOT counted — expected to look quiet; that's part of the finding): ```bash sudo docker exec meshai sqlite3 /app/data/ "SELECT * FROM dispatcher_state;" ``` **Decision matrix:** - A nonzero AND B shows fire `min_severity: priority` (or matrix fire cells with priority floors) → **diagnosis confirmed.** Proceed to the stop gate. - A is ZERO → fires aren't reaching the bus at all; the diagnosis does NOT hold as-is. STOP. Report findings; do not apply the fix. (Check adapter enablement, NIFC fetch errors: `sudo docker logs meshai --since 24h 2>&1 | grep -i "nifc\|wfigs" | tail -20`.) - A nonzero but B shows fire floor already `routine` → drop is elsewhere (matrix cells or region scope). STOP. Report B plus the matrix config; do not apply the fix blind. End the Phase 0 report with `sudo docker ps`. --- ## ⛔ EVERYTHING BELOW REQUIRES MATT'S VERBATIM ACK: `I know and I accept` Report the Phase 0 results including the §8-D burst count, restate the risk (§1), and wait. ## 9. Phase 1 — pre-seed the fires table (mandatory BEFORE the fixed code ever starts) Run against the **still-running old container** (old code keeps suppressing at the floor, so this changes nothing live; after pre-seed the decider simply returns cooldown for those rows): ```bash sudo docker exec meshai sqlite3 /app/data/ " UPDATE fires SET last_broadcast_at = strftime('%s','now'), first_broadcast_at = COALESCE(first_broadcast_at, strftime('%s','now')), last_broadcast_acres = current_acres, last_broadcast_contained = current_contained_pct WHERE last_broadcast_at IS NULL;" ``` Verify — MUST return 0 before proceeding: ```bash sudo docker exec meshai sqlite3 /app/data/ \ "SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;" ``` If `sqlite3` is not present in the image: stop the container, run the same SQL with host `sqlite3` against the bind-mounted data directory (check the volume mapping in docker-compose.yml), verify count 0, and only then continue. Do NOT start the new image before the verify shows 0. Consequence to state to Matt: currently-active fires will not get a retroactive "New" announcement; they will broadcast on their next real growth/containment change. Only genuinely new post-deploy ignitions announce as New. ## 10. Phase 2 — the fix (verbatim diff) Target: `work/meshai/env/store.py`, inside `_emit_event` (function starts line 678). Anchor by grepping `event.data.update(gate.data_patch)` — it occurs once. Verify both anchor lines exist verbatim, then apply exactly: ```diff --- a/work/meshai/env/store.py +++ b/work/meshai/env/store.py @@ def _emit_event(self, adapter, raw_evt: dict): # Apply data_patch into event.data event.data.update(gate.data_patch) + # Promote decider overrides onto the Event itself, mirroring + # the Central path (central/consumer.py:649-653). The gating + # deciders stamp _severity_override (fire: "priority" on every + # New/Update) and, for fire New/tombstone, a category + # override. Merging them into event.data alone leaves + # event.severity at the adapter's value ("routine" for fires + # >=25 km from an anchor), which silently fails the + # toggle/matrix min_severity floors (default "priority", + # config.py:641). Native and Central must share identical + # broadcast decisions. + _sev_override = gate.data_patch.get("_severity_override") + if _sev_override: + event.severity = _sev_override + _cat_override = gate.data_patch.get("category") + if _cat_override: + event.category = _cat_override if gate.commit is not None: event.data["_on_broadcast_committed"] = gate.commit ``` Notes for review: - `Event` is a plain mutable `@dataclass` (`notifications/events.py:38`); attribute assignment is valid. - Category promotion is intentional and safe: `wildfire_declared` maps to the `fire` toggle (`categories.py:~387`); formatter branch selection already falls back to `data["category"]`; cooldown keys become category-scoped, matching Central behavior (New vs Update cooldowns separate). - This is the shared choke point: any category later added to `MESHAI_CUTOVER_CATEGORIES` (quake, swpc, avalanche, nws) gets the correct promotion automatically. **Tests:** if a pytest environment exists on the LXC checkout, run the targeted subsets and compare against the known 10-failure baseline (per commit `0c76a80`, 1697 passed): ```bash python3 -m pytest work/tests -k "fire or store or dispatch or gating" -q ``` If no local pytest env exists, note that in the report and proceed — the change is 16 lines with fully-traced semantics. Do NOT use `docker compose run` to run tests. ## 11. Phase 3 — commit and deploy 1. Present Matt a plain-English summary of the change (what/why in 3–5 sentences, drawn from §3). Wait for his confirmation. His understanding is the gate. 2. Commit with a message in the repo's convention, e.g.: `fix(native): promote decider _severity_override/category onto Event — restores Central-parity fire severity (silent min_severity floor drop)` 3. Deploy exactly: ```bash sudo docker compose build meshai && sudo docker compose up -d ``` ## 12. Phase 4 — post-deploy verification Watch the first 2–3 NIFC poll cycles (read the poll interval from the wfigs/nifc adapter config during Phase 0): ```bash sudo docker logs -f meshai 2>&1 | grep -Ei "Emitted nifc|matrix: dispatched|dispatched|SUPPRESSED|cold-start" ``` Expected: NO burst at startup (pre-seed verified 0); cold-start seed lines for any brand-new first-sight fires; subsequent genuine New/growth events showing `matrix: dispatched` or toggle-path delivery lines. Confirm on-air on the mesh when the first legit event lands. Then confirm the counters are stable and report: ```bash sudo docker exec meshai sqlite3 /app/data/ "SELECT * FROM dispatcher_state;" sudo docker ps ``` Every report back to Matt ends with `sudo docker ps`. ## 13. Backlog (do NOT do now — list for future sessions) 1. Persist the fire seen-baseline across restarts to close the restart-window ignition gap (§6.1). 2. Decide policy for zero-acre first sightings (§6.2). 3. Extend pacing to the native emit path for multi-fire poll batches (§6.4). 4. Before cutting over quake/swpc/avalanche/nws via `MESHAI_CUTOVER_CATEGORIES`, re-verify override promotion end-to-end (covered by this fix, but confirm with shadow logs). 5. Add a drop counter + log line for the dispatcher severity-floor return (it is currently the only fully silent drop in the pipeline).