matt/meshai
1
0
Fork 0
mirror of https://github.com/zvx-echo6/meshai.git synced 2026-08-26 17:31:34 +00:00
Code Issues Projects Releases Packages Wiki Activity

chore(central-ripout 2d-ii): remove dead fire scraps (NOT the 'rewrite' — there wasn't one) (#166)

Browse source 
* chore(fire-ripout 2dii-a): remove dead wildfire_growth cutover path + handle_firms

wildfire_growth events are always fully precomposed (env/firms.py sets
_meshai_precomposed=True + title=<wire from _render()>) and the category is
not in cutover.NATIVE_ALWAYS_DECIDE, so the registered fire formatter could
never be reached live via compose_mesh_message for this category -- the
precomposed-title bypass always won first. Removes the dead
is_cutover("wildfire_growth") NEW-PATH branch in
fire_fusion._handle_pass_boundary (kept the live legacy leg that calls
_render unconditionally) and the now-unreachable wildfire_growth formatter
registration in notifications/formatters/__init__.py.

Also removes handle_firms, the dead Central NATS-envelope entrypoint (zero
live production callers -- Central's consumer that drove it is gone), plus
its envelope-specific filtering helpers (_confidence_passes, _in_bbox,
_coerce_severity, _log_event, FIRMS_CONFIDENCE_FLOOR/FRP_FLOOR/BBOX_OPTIONAL)
that had no live callers left. The shared, LIVE fusion core
(_ingest_pixel_core, attribution, clustering, growth/spotting/halt) and
_parse_acq_epoch are unaffected -- still the engine behind the native
ingest_hotspot_pixel entrypoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(fire-ripout 2dii-b): remove dead handle_wfigs entrypoint

handle_wfigs (the dead Central NATS-envelope entrypoint) had zero live
production callers -- Central's consumer that drove it is gone. The LIVE
WFIGS path is env/fires.py (native adapter) -> env/store.py::_emit_event,
forced onto gating.fire.decide + the shared fire formatter via
cutover.NATIVE_ALWAYS_DECIDE, independent of this dead handler.

Removes handle_wfigs and its private-only helpers (_coerce_severity,
_log_event, _log_event_returning_id) that had no callers left. _render
remains -- it is called directly by env.fire_fusion._handle_pass_boundary on
the FIRMS wildfire_growth path, and is used as a byte-identity oracle by
tests against the shared, live fire formatter. _build_canonical,
_attach_commit_handles, _fire_too_old_to_announce, _now, _cleanup_stale_fires
and WFIGS_BROADCAST_COOLDOWN_S are kept -- each has its own direct test
coverage independent of handle_wfigs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(fire-ripout 2dii-c): rewire fire tests off the deleted dead entrypoints

handle_firms and handle_wfigs (both removed in the prior two commits) were
used throughout this test suite purely as convenient DRIVERS for live logic
(pixel attribution/clustering/growth/spotting/halt fusion; the WFIGS
New/Update/cooldown/closed decider). Per-file disposition:

Rewired to drive the LIVE native entrypoint directly (no behavior change --
same underlying _ingest_pixel_core / gating.fire.decide engines):
  - test_fire_tracker_phase1/2/3.py: handle_firms -> ingest_hotspot_pixel
  - test_firms_refactor.py: handle_firms driver helpers -> ingest_hotspot_pixel;
    dropped the dead wildfire_growth cutover test + the fully-redundant
    TestNotCutoverLegacyVerbatim class (already covered by
    test_firms_native_fusion.py's TestIngestGrowth/Spotting/Halt against the
    live entrypoint); wildfire_growth formatter registration test now
    asserts None (matches source change).
  - test_wfigs_handler.py: handle_wfigs -> _render (the live WFIGS renderer)
    called directly, for the anchor-priority + missing-acres cases that
    exercise shared/live code (_location_anchor).
  - test_fire_refactor.py: handle_wfigs -> gating.fire.decide() driven
    directly, with state written the same unconditional shape the live
    native path uses; TestGateSequenceParity trimmed to focus on the
    tombstone/closed lifecycle step (not covered by
    test_fire_native_growth.py's native-adapter New/Update/cooldown coverage).
  - test_tombstone_broadcast.py, test_fire_age_gate.py: handle_wfigs ->
    gating.fire.decide() driven directly; same assertions, off the dead path.

Deleted as pure dead-entrypoint contract testing with no live equivalent:
  - test_firms_native_fusion.py::TestCentralPathUnchanged (2 tests) --
    existed only to guard handle_firms's own envelope parsing.
  - test_firms_handler.py: confidence/FRP/bbox filtering, missing-coords,
    non-firms-adapter guard, event_log accounting (all handle_firms-specific;
    the native adapter filters upstream in a different code path). Kept +
    rewired: the shared _ingest_pixel_core dedup behavior and
    _parse_acq_epoch's int/short acq_time parsing (both genuinely live/shared).
  - test_wfigs_handler.py: envelope field-extraction (acres-fallback chain,
    IA-placeholder-as-name), tombstone/perimeter subject -> event_log,
    New/Update/cooldown decision + audit-row wiring (all redundant with
    tests/test_fire_native_growth.py's coverage of gating.fire.decide()
    through the real native adapter).
  - test_tombstone_broadcast.py::test_commit_callback_flips_handled --
    asserted handle_wfigs's own event_log-row flip on commit, a Central-only
    concept the native path never used.
  - test_tail_followups.py::test_wfigs_tombstone_stamps_column -- asserted
    handle_wfigs's own inline tombstoned_at UPDATE; no live producer emits
    _kind=wfigs_tombstone today, so nothing in the live system stamps it.

Both live renderers stay covered: fire_format (wildfire_declared/incident,
via test_fire_refactor.py + test_fire_native_growth.py, untouched) and
_render (wildfire_growth via FIRMS, via test_fire_tracker_phase2.py +
test_firms_native_fusion.py's TestIngestGrowth, both driving the live
ingest_hotspot_pixel entrypoint).

Collection: 2010 -> 1974 tests (net -36: dead-only tests deleted, live
behavior rewired 1:1 or consolidated onto already-existing native-path
coverage). Full suite: 1974 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
malice 2026-07-17 22:11:00 -06:00 • committed by GitHub
commit 67176d66d5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 629 additions and 2026 deletions
Download patch file Download diff file Expand all files Collapse all files

347
work/tests/test_firms_handler.py
View file

@ -1,17 +1,27 @@
"""Tests for v0.6-1 FIRMS handler (storage-only).
"""Tests for the shared FIRMS pixel-ingest core (`_ingest_pixel_core`) +
`_parse_acq_epoch` -- the parts of the retired v0.6-1 FIRMS handler that are
still LIVE, shared by the native `env/firms.py` adapter via
`ingest_hotspot_pixel`.
The handler never returns a wire string (storage-only contract). All
assertions check side effects on `firms_pixels` + `event_log`. Per the
audit doc finding #2 the handler must close the v0.5.13 silent-drop on
`central.fire.hotspot.>` envelopes.
chore/ripout-2dii: `handle_firms` (the dead Central NATS-envelope entrypoint
this file used to test -- confidence/FRP/bbox filtering, envelope field
extraction, missing-coords/missing-acq-time drops, non-firms-adapter guard,
event_log accounting) has been REMOVED from `meshai.env.fire_fusion` -- zero
live production callers, and no live equivalent (the native adapter's own
confidence/bbox filtering happens upstream in `env/firms.py`'s CSV fetch, a
different code path entirely; see `tests/test_adapter_firms.py`). Those tests
were deleted with it.
Envelope shape sourced from firms-investigation.md sampling (250 envelopes
2026-05-28..06-04, all VIIRS).
What remains: the dedup behavior of `_ingest_pixel_core` (INSERT OR IGNORE on
a meters-quantized dedup key -- genuinely shared/live, exercised by BOTH the
native adapter and formerly by handle_firms) and `_parse_acq_epoch`'s
int/short/zero-padded acq_time parsing (also shared/live -- env/firms.py
imports and calls it directly). Both rewritten to drive
`ingest_hotspot_pixel` directly instead of the dead handler.
"""
import pytest
from meshai.env import fire_fusion as firms_handler
from meshai.env.fire_fusion import handle_firms
from meshai.env.fire_fusion import ingest_hotspot_pixel, _parse_acq_epoch
from meshai.persistence import close_thread_connection, init_db
from meshai.persistence import db as persistence_db
@ -31,319 +41,88 @@ def mem_db(monkeypatch, tmp_path):
persistence_db._initialised.discard(db_path)
def _firms_env(*,
lat=42.19664, lon=-113.70981, frp=135.93,
confidence="high", satellite="N",
acq_date="2026-05-28", acq_time="1949",
bright_ti4=367.0, daynight="D",
envelope_id="firms_001",
category="fire.hotspot.viirs",
severity=3,
region="unknown",
extras=None):
"""Build a FIRMS CloudEvents envelope matching the live wire shape.
Default fixture is SAMPLE B from firms-investigation.md (Cache Peak
high-confidence 135 MW fire). Override individual fields for filter
coverage.
"""
payload = {
"id": envelope_id,
"latitude": lat,
"longitude": lon,
"frp": frp,
"confidence": confidence,
"satellite": satellite,
"instrument": "VIIRS",
"acq_date": acq_date,
"acq_time": acq_time,
"bright_ti4": bright_ti4,
"daynight": daynight,
"version": "2.0NRT",
"_enriched": {"geocoder": {"landclass": "Cache Peak Roadless Area",
"elevation_m": 2151.9}},
}
if extras: payload.update(extras)
return {
"subject": f"central.fire.hotspot.viirs_snpp.{confidence}.{region}",
"id": envelope_id,
"data": {
"id": envelope_id,
"adapter": "firms",
"category": category,
"severity": severity,
"time": "2026-05-28T19:49:00Z",
"geo": {"centroid": [lon, lat]},
"data": payload,
},
}
def _pixel(*, lat=42.19664, lon=-113.70981, frp=135.93, confidence="high",
satellite="N", acq_date="2026-05-28", acq_time="1949",
brightness=367.0):
acq_epoch = _parse_acq_epoch(acq_date, acq_time)
return {"lat": lat, "lon": lon, "frp": frp, "confidence": confidence,
"brightness": brightness, "satellite": satellite,
"acq_epoch": acq_epoch}
def _row_count(mem_db, table):
return mem_db.execute(f"SELECT COUNT(*) AS n FROM {table}").fetchone()["n"]
def _last_event_log(mem_db):
r = mem_db.execute(
"SELECT * FROM event_log ORDER BY id DESC LIMIT 1"
).fetchone()
return dict(r) if r else None
# ============================================================================
# Happy path: high-confidence pixel stored
# _parse_acq_epoch -- acq_time format quirks (shared with env/firms.py)
# ============================================================================
def test_high_confidence_pixel_persisted(mem_db):
env = _firms_env()
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
def test_acq_time_int_accepted():
"""FIRMS sometimes publishes acq_time as int 2013 rather than '2013'."""
assert _parse_acq_epoch("2026-05-28", 1949) is not None
assert _parse_acq_epoch("2026-05-28", 1949) == _parse_acq_epoch("2026-05-28", "1949")
assert out is None, "storage-only handler must never return a wire string"
assert _row_count(mem_db, "firms_pixels") == 1
row = mem_db.execute("SELECT * FROM firms_pixels").fetchone()
assert row["lat"] == pytest.approx(42.19664)
assert row["lon"] == pytest.approx(-113.70981)
assert row["frp"] == pytest.approx(135.93)
assert row["confidence"] == "high"
assert row["satellite"] == "N"
assert row["brightness"] == pytest.approx(367.0)
assert row["irwin_id"] is None # unattached; v0.6 fire-tracker fills later
# acq_time should be the parsed UTC epoch of 2026-05-28 19:49Z.
def test_short_acq_time_zero_padded():
"""acq_time '49' (early-morning pass) must zero-pad to '0049'."""
epoch = _parse_acq_epoch("2026-05-28", "49")
assert epoch is not None
import datetime as _dt
expected = int(_dt.datetime(2026, 5, 28, 19, 49,
tzinfo=_dt.timezone.utc).timestamp())
assert row["acq_time"] == expected
# event_log: handled=1, table_name set, table_pk = inserted rowid.
log = _last_event_log(mem_db)
assert log["source"] == "firms"
assert log["handled"] == 1
assert log["table_name"] == "firms_pixels"
assert log["table_pk"] == str(row["id"])
dt = _dt.datetime.fromtimestamp(epoch, tz=_dt.timezone.utc)
assert (dt.hour, dt.minute) == (0, 49)
def test_nominal_confidence_pixel_persisted_under_default_floor(mem_db):
"""Default FIRMS_CONFIDENCE_FLOOR='low' must accept nominal/high/low."""
env = _firms_env(confidence="nominal", envelope_id="firms_002")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
def test_low_confidence_pixel_persisted_under_default_floor(mem_db):
env = _firms_env(confidence="low", envelope_id="firms_003")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
def test_missing_acq_time_returns_none():
assert _parse_acq_epoch(None, None) is None
assert _parse_acq_epoch("2026-05-28", None) is None
# ============================================================================
# Confidence floor when bumped up
# ============================================================================
def test_low_confidence_dropped_when_floor_is_nominal(mem_db, monkeypatch):
monkeypatch.setattr(firms_handler, "FIRMS_CONFIDENCE_FLOOR", "nominal")
env = _firms_env(confidence="low", envelope_id="firms_lo")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["handled"] == 0
assert log["category"].endswith("|below_confidence_floor")
def test_unknown_confidence_value_dropped(mem_db):
env = _firms_env(confidence="bogus", envelope_id="firms_bog")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["category"].endswith("|below_confidence_floor")
# ============================================================================
# FRP floor
# ============================================================================
def test_low_frp_dropped_when_floor_set(mem_db, monkeypatch):
monkeypatch.setattr(firms_handler, "FIRMS_FRP_FLOOR", 5.0)
env = _firms_env(frp=1.4, confidence="high", envelope_id="firms_frp_low")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["category"].endswith("|below_frp_floor")
def test_frp_at_floor_stored(mem_db, monkeypatch):
monkeypatch.setattr(firms_handler, "FIRMS_FRP_FLOOR", 5.0)
env = _firms_env(frp=5.0, confidence="high", envelope_id="firms_frp_at")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
def test_missing_frp_dropped_when_floor_set(mem_db, monkeypatch):
monkeypatch.setattr(firms_handler, "FIRMS_FRP_FLOOR", 5.0)
env = _firms_env(confidence="high", envelope_id="firms_no_frp",
extras={"frp": None})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
def test_missing_frp_stored_when_floor_zero(mem_db):
"""Default FIRMS_FRP_FLOOR=0: missing FRP still stores (null in column)."""
env = _firms_env(confidence="high", envelope_id="firms_no_frp_2",
extras={"frp": None})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
row = mem_db.execute("SELECT * FROM firms_pixels").fetchone()
assert row["frp"] is None
# ============================================================================
# Bbox filter (default None = pass-through; explicit bbox tested both sides)
# ============================================================================
def test_bbox_none_passes_through(mem_db):
env = _firms_env(lat=51.0, lon=10.0, # Germany
envelope_id="firms_eu")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
def test_bbox_drops_outside(mem_db, monkeypatch):
# Idaho-ish bbox.
monkeypatch.setattr(firms_handler, "FIRMS_BBOX_OPTIONAL",
(42.0, -117.5, 49.0, -111.0))
env = _firms_env(lat=51.0, lon=10.0, envelope_id="firms_out")
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["category"].endswith("|outside_bbox")
def test_bbox_keeps_inside(mem_db, monkeypatch):
monkeypatch.setattr(firms_handler, "FIRMS_BBOX_OPTIONAL",
(42.0, -117.5, 49.0, -111.0))
env = _firms_env(envelope_id="firms_in") # Cache Peak is inside
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
# ============================================================================
# Dedup: same satellite pixel observation arriving twice = no-op
# Dedup: same satellite pixel observation arriving twice = no-op.
# `_ingest_pixel_core`'s meters-quantized dedup_key + INSERT OR IGNORE is the
# LIVE, source-agnostic core shared by ingest_hotspot_pixel.
# ============================================================================
def test_dedup_same_pixel_idempotent(mem_db):
env = _firms_env(envelope_id="dup_001")
handle_firms(env, env["subject"], data={}, now=1_780_660_000)
handle_firms(env, env["subject"], data={}, now=1_780_660_001)
p = _pixel()
ingest_hotspot_pixel(p, now=1_780_660_000)
ingest_hotspot_pixel(p, now=1_780_660_001)
assert _row_count(mem_db, "firms_pixels") == 1, "OR IGNORE collapses dup"
# Two event_log rows; second is dedup_hit.
rows = mem_db.execute(
"SELECT category, table_name, handled FROM event_log "
"WHERE source='firms' ORDER BY id"
).fetchall()
assert len(rows) == 2
assert rows[0]["table_name"] == "firms_pixels"
assert rows[1]["table_name"] is None
assert rows[1]["category"].endswith("|dedup_hit")
def test_dedup_collapses_lat_lon_float_noise(mem_db):
"""Same coord with sub-1m float noise must hit the same dedup key.
5-decimal rounding => differences in the 6th+ decimal are absorbed."""
e1 = _firms_env(lat=42.196641234567, lon=-113.709810000001,
envelope_id="dup_a")
e2 = _firms_env(lat=42.196641111111, lon=-113.709810999999,
envelope_id="dup_b")
handle_firms(e1, e1["subject"], data={}, now=1_780_660_000)
handle_firms(e2, e2["subject"], data={}, now=1_780_660_001)
Meters-based quantization absorbs differences well under 1 pixel."""
p1 = _pixel(lat=42.196641234567, lon=-113.709810000001)
p2 = _pixel(lat=42.196641111111, lon=-113.709810999999)
ingest_hotspot_pixel(p1, now=1_780_660_000)
ingest_hotspot_pixel(p2, now=1_780_660_001)
assert _row_count(mem_db, "firms_pixels") == 1
def test_dedup_different_satellite_stored_separately(mem_db):
"""Same coord + acq_time but different satellite is 2 distinct observations."""
e1 = _firms_env(satellite="N", envelope_id="sat_n")
e2 = _firms_env(satellite="N20", envelope_id="sat_n20")
handle_firms(e1, e1["subject"], data={}, now=1_780_660_000)
handle_firms(e2, e2["subject"], data={}, now=1_780_660_001)
ingest_hotspot_pixel(_pixel(satellite="N"), now=1_780_660_000)
ingest_hotspot_pixel(_pixel(satellite="N20"), now=1_780_660_001)
assert _row_count(mem_db, "firms_pixels") == 2
def test_dedup_different_acq_time_stored_separately(mem_db):
"""Same pixel observed on two passes 12h apart -> 2 rows."""
e1 = _firms_env(acq_time="0700", envelope_id="t1")
e2 = _firms_env(acq_time="1900", envelope_id="t2")
handle_firms(e1, e1["subject"], data={}, now=1_780_660_000)
handle_firms(e2, e2["subject"], data={}, now=1_780_660_001)
ingest_hotspot_pixel(_pixel(acq_time="0700"), now=1_780_660_000)
ingest_hotspot_pixel(_pixel(acq_time="1900"), now=1_780_660_001)
assert _row_count(mem_db, "firms_pixels") == 2
# ============================================================================
# Bad / missing inputs
# ============================================================================
def test_missing_coords_dropped(mem_db):
env = _firms_env(envelope_id="no_coords",
extras={"latitude": None, "longitude": None})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["category"].endswith("|missing_coords")
def test_missing_acq_time_dropped(mem_db):
env = _firms_env(envelope_id="no_acq",
extras={"acq_date": None, "acq_time": None})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
log = _last_event_log(mem_db)
assert log["category"].endswith("|missing_acq_time")
def test_non_firms_adapter_passes_through(mem_db):
"""Defense in depth: handler must early-return on non-firms envelopes."""
env = _firms_env(envelope_id="wrong_adapter")
env["data"]["adapter"] = "nws"
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 0
assert _row_count(mem_db, "event_log") == 0
def test_acq_time_int_accepted(mem_db):
"""FIRMS sometimes publishes acq_time as int 2013 rather than '2013'."""
env = _firms_env(envelope_id="int_acq",
extras={"acq_time": 1949})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
def test_missing_frp_stored(mem_db):
"""A pixel with no FRP still stores (null in column) -- ingest_hotspot_pixel
never filters on FRP (that was handle_firms-specific, now gone)."""
p = _pixel()
p["frp"] = None
ingest_hotspot_pixel(p, now=1_780_660_000)
assert _row_count(mem_db, "firms_pixels") == 1
def test_short_acq_time_zero_padded(mem_db):
"""acq_time '49' (early-morning pass) must zero-pad to '0049'."""
env = _firms_env(envelope_id="short_acq",
extras={"acq_time": "49"})
out = handle_firms(env, env["subject"], data={}, now=1_780_660_000)
assert out is None
assert _row_count(mem_db, "firms_pixels") == 1
row = mem_db.execute("SELECT * FROM firms_pixels").fetchone()
assert row["frp"] is None