Three independent bugs kept firms_handler's growth/spotting/halt/cluster
fusion decisions from reaching a correct mesh Event:
- #117: consumer._normalize() computed `category` from the raw Central
category BEFORE the per-adapter handler ran and never re-read
data["category"] afterward, so every firms_handler category stamp was a
silent no-op. Now re-read post-dispatch, validated against the known
category registry (unrecognized overrides are logged and ignored).
- #118: consumer.py only ever honors data["_severity_override"], but
firms_handler's halt/spotting/cluster sites stamped the plain
data["severity"] key instead (only growth used the right key). Switched
all three sites to `_severity_override` for one consistent contract.
This is severity plumbing only -- it does not change which events fire.
- #119: FirePacer's gate only matched source in ("fires","wfigs") at
severity=="priority", so FIRMS fusion broadcasts (source="firms",
growth/spotting at "immediate") never reached the pacer. Broadened the
gate to cover "firms" + {"priority","immediate"}, and gave FirePacer
head-of-line insertion so an "immediate" event is never stuck behind
already-queued "priority" events. Still unbounded/never-drops.
Cluster detection is left exactly as main ships it: live, always on, no
toggle (PR #73's curated new-fire cluster broadcasts with cold-start
silent-seeding). Only its severity-override key changes, under #118.
Updated existing tests that asserted the old (buggy) data["severity"]
contract, and added tests/test_firms_fusion_event_contract.py covering
all three fixes end-to-end through consumer._normalize()/_handle() and
FirePacer directly.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(firms): curated new-fire cluster broadcasts (no per-pixel, no cold-start dump)
Enable the built _maybe_emit_cluster path (was dead-coded) so FIRMS broadcasts
curated hotspot clusters as possible new fires — clustered, deduped via
cluster_broadcast_at, attributed against known WFIGS fires first (so MORA's
hotspots don't false-cluster). Give FIRMS a default Idaho bbox so it fetches
when coverage is off (coverage bbox still overrides). First-fetch silent-seed
prevents a cold-start dump of the day's existing hotspots. Raw pixels stay
store-only. Coverage geometry gate filters cluster broadcasts to the region.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(firms): first-fetch silent-seed suppresses fusion wires too (no cold-start)
Extend the FIRMS cold-start seed to suppress growth/spotting/halt fusion
broadcasts on the first fetch, not just clusters — enabling FIRMS must emit
zero broadcasts on the initial hotspot sweep. Persistence, attribution, and
dedup baselines still run during seed; only later new activity broadcasts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two live backlog-broadcast leaks traced to the in-memory first-poll seed:
(1) incremental-fetch adapters (wzdx: registry tick [0 events] then feeds
tick [many]) got marked _seeded on the EMPTY first tick, so the real batch
next tick all looked "new" and broadcast; (2) in-memory seed lost on restart.
Fix — durable baseline + guard:
- _seed_from_persistent() at store init: pre-load already-received item keys
from the persistent hazard tables into self._seen, so nothing ever received
can re-broadcast (immune to fetch staging + restart). Only sources whose
native emit key PROVABLY equals a persistent key are durably seeded:
wzdx (traffic_events.external_id) + usgs_quake (quake_events.event_id).
Resilient (per-table try/except; missing table -> skip).
- _seen_key() now namespaces by evt["source"] (matches persistent tables),
via shared _key_ext/_key_eid helpers used by both seed and live emit so
they can't drift.
- non-empty-seed guard: _ingest marks only sources that carried >=1 event
this poll as _seeded -> an empty first tick can never seed-then-leak. This
is the root-cause fix; covers all adapters (roads511/traffic fetch
atomically per tick, so the guard fully protects them).
- storage untouched (self._events populated for every event); Central
path/deciders untouched.
Live-DB verified: seed pre-loads 784 wzdx + 8 quake keys -> a live wzdx poll
of 784 known zones broadcasts 0. +6 tests (incremental staging, restart,
persistent-preseed, fresh-DB fallback); suite at 10-failure baseline.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the native path's "scan accumulated state + suppress what we've
already broadcast" model with "broadcast only what newly arrived from the
API this poll." Storage is unchanged (self._events + firms_pixels etc. are
populated for EVERY received item, so the LLM/get_active backlog is intact);
only the BROADCAST decision changes.
- env/store.py: per-adapter in-memory seen-set (_seen) + _seeded. First
data-bearing poll for an adapter seeds keys and emits NOTHING (that batch
is pre-existing backlog); later polls emit only keys not seen before.
Restart => empty sets => next poll re-seeds silently. Structurally
impossible to broadcast backlog on cold start / restart / re-enable.
Key = external_id -> event_id -> content hash, namespaced per adapter.
self._events[key]=evt still runs unconditionally (storage preserved).
- Fixes the ~175 (roads511) / ~782 (wzdx) cold-start bursts AND the latent
quake/nws version (they only looked safe because Central pre-populated
their broadcast tables).
- env/satpass.py: broadcast on AOS IMMINENCE (now < aos <= now+lead,
broadcast_lead_seconds default 3600), future-only; window_hours still
governs prediction depth. Strict norad_ids post-filter + fixed
_parse_norad_ids char-iteration bug (cause of GOES/METEOR leak).
- Central path + broadcast-state tables untouched (native-only gate).
13 new tests; full suite at 10-failure baseline (1697 passed).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the last standalone gap. Extract ingest_hotspot_pixel(pixel, *, now)
from firms_handler so the FIRMS attribution/fusion engine (firms_pixels ->
_attribute_or_cluster -> fire_pixels/fire_passes/centroid -> growth/spotting/
halt) is source-agnostic. Both the Central NATS path and native env/firms.py
drive one identical engine.
- shared _ingest_pixel_core(conn, ...) called by both ingest_hotspot_pixel
and handle_firms; Central path byte-identical (its tests pass unchanged)
- env/firms.py _fetch() feeds each fetched pixel into ingest_hotspot_pixel;
DB-level dedup makes re-fetched pixels no-ops (no double count); to_event()
returns None for raw hotspots, precomposed Event for fusion outputs
(wildfire_growth/spotting/halted via the Phase-3c formatters/gating)
- raw hotspots / new_ignition / cluster NEVER broadcast (cluster stays dead)
FLIP NOTE: keep wildfire_growth/spotting/halted OUT of cutover — native
emits precomposed with gating done inside the engine; cutover would re-run
the _kind-keyed decider on data lacking _kind and suppress.
10 new tests; Central firms/fire-tracker suites unchanged; full suite
10 failed/1682 passed (baseline 10, +10 new).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
