feat(v0.6-tail): close 5 v0.6-phase1-complete.md follow-ups

(1) Auto-call refresh-toggles on PUT /api/config/notifications
    meshai/dashboard/api/config_routes.py adds register_config_routes_hooks(app)
    which registers a FastAPI HTTP middleware: on any 2xx PUT whose path
    matches /api/config/notifications or /api/config, the middleware
    invokes _refresh_toggle_filter(app) which reaches into app.state.bus._
    pipeline_components["toggle_filter"] and calls .refresh(app.state.config).
    The dashboard no longer has to remember to ping POST /api/notifications/
    refresh-toggles after a toggle change. The explicit endpoint stays for
    backwards-compat.

(2) env_reporter block-size cap moved to adapter_config
    New registry row pipeline.env_reporter_block_chars (int, default 3000).
    meshai/notifications/env_reporter.py replaces the hardcoded
    _BLOCK_MAX_CHARS = 3000 with _DEFAULT_BLOCK_MAX_CHARS (the fallback) +
    a _block_cap() helper that reads from adapter_config on every slice.
    Mutating the row via PUT /api/adapter-config takes effect on the next
    env_reporter call -- no restart.

(3) Bulk-import endpoint for gauge_sites
    meshai/dashboard/api/gauge_sites_import.py adds
    POST /api/gauge-sites/import with two paths:
      format=csv      -- expects "data" (CSV text with header row matching
                          gauge_sites columns: site_id, gauge_name, lat, lon,
                          and optionally action_ft/flood_minor_ft/
                          flood_moderate_ft/flood_major_ft/enabled). UPSERT
                          via ON CONFLICT(site_id) DO UPDATE. Returns
                          {inserted, updated, skipped}.
      format=nws-ahps -- expects "wfo" (list of WFO codes). Fetches
                          water.weather.gov/ahps2/index.php?wfo=<WFO> for each,
                          regex-parses gauge links, then fetches up to 50
                          gauge detail pages per request and regex-parses
                          lat/lon + four threshold values. Best-effort; rows
                          stored under "AHPS-<gauge_id>" so they dont collide
                          with USGS-* ids. Returns the same shape plus
                          detail_fetched + errors list.
    Frontend (dashboard-frontend/src/pages/GaugeSites.tsx) gains a
    Import button + modal with two tabs (Paste CSV / Scrape NWS-AHPS)
    rendered via an ImportModal component. CSV tab has a 48-row textarea
    with the column-header hint inline; AHPS tab has a comma-separated WFO
    input defaulting to BOI. Both submit via fetch() and show the JSON
    response inline. Invalidates the curation cache server-side on any
    successful insert/update so nwis_handler sees the new gauges on its
    next call.

(4) WFIGS tombstone column -- CORRECTNESS
    v12.sql adds fires.tombstoned_at REAL (nullable) + idx_fires_tombstoned_at.
    meshai/central/wfigs_handler.py: the tombstone branch
    (kind=="wfigs_tombstone") UPDATE fires SET tombstoned_at=COALESCE(
    tombstoned_at, ?) so the first tombstone-time wins (idempotent against
    repeated tombstone envelopes).
    meshai/notifications/reminders/__init__.py: the wfigs tombstone
    termination condition now checks row["tombstoned_at"] IS NOT NULL.
    Reminders correctly STOP for closed fires -- before this change the
    8h cadence would have kept Active: broadcasts going indefinitely past
    a WFIGS removal.
    SCHEMA_VERSION 11 -> 12.

(5) Delete INCIDENT_BROADCAST_HEARTBEAT_S
    meshai/central/incident_handler.py: removed the dead constant
    (v0.5.9 REVISED dropped the heartbeat path but left the constant
    imported-but-never-read).
    tests/test_incident_handler.py: removed the orphan
    test_i_8h_heartbeat_triggers_update test (asserted None, used the
    deleted constant for time arithmetic) and the stray import line.

Tests (tests/test_tail_followups.py, 16 cases):
  - middleware fires refresh on PUT /api/config/notifications (200), does
    NOT fire on PUT /api/config/llm
  - env_reporter _block_cap() default 3000; mutate via PUT, invalidate,
    next read returns the new cap
  - CSV import inserts new rows, updates existing, skips bad rows,
    rejects missing required columns, rejects bad format
  - AHPS index parser extracts (gauge_id, name) from realistic HTML
  - AHPS detail parser extracts lat/lon + four thresholds from realistic
    HTML
  - fires has tombstoned_at column after migrations
  - wfigs tombstone branch stamps tombstoned_at
  - ReminderScheduler skips a fire whose tombstoned_at is NOT NULL
  - ReminderScheduler still fires for a fire whose tombstoned_at IS NULL
  - INCIDENT_BROADCAST_HEARTBEAT_S no longer importable

Foundation/API test counts bumped:
  REGISTRY 58 -> 59 (+ env_reporter_block_chars)
  schema_meta v11 -> v12

Test count: 844 -> 859 (+16 new, -1 deleted dead test). 0 regressions.
This commit is contained in:
Matt Johnson (via Claude) 2026-06-05 21:37:05 +00:00
commit 566b06de06
13 changed files with 704 additions and 47 deletions

View file

@ -29,14 +29,14 @@ def client():
# ============================================================================
def test_list_returns_all_58_keys(client):
def test_list_returns_all_59_keys(client):
r = client.get("/api/adapter-config")
assert r.status_code == 200
body = r.json()
# 14 adapters with at least one key (itd_511 has zero -- not in the
# grouped dict because the SQL only returns rows that exist).
total = sum(len(v) for v in body.values())
assert total == 58
assert total == 59
def test_list_grouped_by_adapter(client):

View file

@ -54,11 +54,11 @@ def test_v6_tables_exist(fresh_db):
assert "adapter_meta" in tables
def test_schema_meta_at_v11(fresh_db):
def test_schema_meta_at_v12(fresh_db):
v = fresh_db.execute(
"SELECT value FROM schema_meta WHERE key='version'"
).fetchone()["value"]
assert int(v) == 11
assert int(v) == 12
def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
@ -73,9 +73,9 @@ def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
# ---------- registry shape -----------------------------------------------
def test_registry_at_58_entries():
def test_registry_at_59_entries():
"""v0.6-3a.1 trim: 43 CONFIG-only keys (was 77 in v0.6-3a draft)."""
assert len(REGISTRY) == 58, (
assert len(REGISTRY) == 59, (
f"REGISTRY should have 43 entries after CONFIG-vs-CODE trim; got {len(REGISTRY)}. "
f"If a sentence template / emoji / heuristic snuck in, it belongs in CODE not config."
)

View file

@ -13,8 +13,7 @@ Coverage:
(f) magnitude bump up -> Update
(g) delay double (>=2x) -> Update
(h) icon change -> Update
(i) 8h heartbeat -> Update
state_511 / itd_511 EventType branching (j-m):
(j) state_511_atis incident parses
(k) state_511_atis closure parses
@ -33,7 +32,6 @@ import time
import pytest
from meshai.central.incident_handler import (
INCIDENT_BROADCAST_HEARTBEAT_S,
handle_incident,
)
from meshai.persistence import close_thread_connection, init_db
@ -374,24 +372,6 @@ def test_h_icon_change_triggers_update(mem_db, no_photon):
assert row["icon_category"] == "road_closed"
# ============================================================================
# (i) 8h heartbeat triggers Update
# ============================================================================
def test_i_8h_heartbeat_triggers_update(mem_db, no_photon):
env = _tomtom_env(icon_category=6, magnitude=2, delay=300)
data1 = {}
handle_incident(env, env["subject"], data=data1, now=1_000_000)
_commit(data1, 1_000_001)
# v0.5.9 REVISED gate (A): heartbeat no longer fires Update.
later = 1_000_001 + INCIDENT_BROADCAST_HEARTBEAT_S
data2 = {}
wire2 = handle_incident(env, env["subject"], data=data2, now=later)
assert wire2 is None
# ============================================================================
# (j) state_511_atis incident parses
# ============================================================================

View file

@ -0,0 +1,306 @@
"""v0.6-tail tests: 5 follow-ups."""
from __future__ import annotations
import time
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from meshai.adapter_config import adapter_config, invalidate_cache
from meshai.persistence import get_db
# ============================================================================
# Item 1 -- auto-refresh ToggleFilter on PUT /api/config/notifications
# ============================================================================
def test_auto_refresh_middleware_fires_on_notifications_put():
"""The middleware calls ToggleFilter.refresh() on a successful PUT
that touches the notifications section."""
from meshai.dashboard.api import config_routes
from types import SimpleNamespace
refreshed = {"n": 0}
class _StubTF:
def refresh(self, config):
refreshed["n"] += 1
app = FastAPI()
app.state.config = SimpleNamespace() # any truthy stand-in
bus = SimpleNamespace()
bus._pipeline_components = {"toggle_filter": _StubTF()}
app.state.bus = bus
config_routes.register_config_routes_hooks(app)
@app.put("/api/config/notifications")
async def _put(): return {"ok": True}
client = TestClient(app)
client.put("/api/config/notifications", json={"enabled": True})
assert refreshed["n"] == 1
def test_auto_refresh_does_not_fire_on_other_section():
from meshai.dashboard.api import config_routes
from types import SimpleNamespace
refreshed = {"n": 0}
class _StubTF:
def refresh(self, config):
refreshed["n"] += 1
app = FastAPI()
app.state.config = SimpleNamespace()
bus = SimpleNamespace()
bus._pipeline_components = {"toggle_filter": _StubTF()}
app.state.bus = bus
config_routes.register_config_routes_hooks(app)
@app.put("/api/config/llm")
async def _put(): return {"ok": True}
client = TestClient(app)
client.put("/api/config/llm", json={})
assert refreshed["n"] == 0
# ============================================================================
# Item 2 -- env_reporter cap from adapter_config
# ============================================================================
def test_env_reporter_default_cap_3000():
invalidate_cache()
from meshai.notifications.env_reporter import _block_cap, _DEFAULT_BLOCK_MAX_CHARS
assert _block_cap() == 3000
assert _DEFAULT_BLOCK_MAX_CHARS == 3000
def test_env_reporter_cap_respects_config_mutation():
"""PUT-equivalent: change the row, invalidate, next call returns new cap."""
invalidate_cache()
conn = get_db()
conn.execute(
"UPDATE adapter_config SET value_json=? "
"WHERE adapter='pipeline' AND key='env_reporter_block_chars'",
("500",),
)
invalidate_cache()
from meshai.notifications.env_reporter import _block_cap
assert _block_cap() == 500
# ============================================================================
# Item 3 -- gauge_sites bulk import (CSV path)
# ============================================================================
@pytest.fixture
def client():
from meshai.dashboard.api.gauge_sites_import import router as imp_router
from meshai.dashboard.api.curation_routes import router as cur_router
app = FastAPI()
app.include_router(imp_router, prefix="/api")
app.include_router(cur_router, prefix="/api")
return TestClient(app)
def test_csv_import_inserts_new_rows(client):
csv_data = (
"site_id,gauge_name,lat,lon,action_ft,flood_minor_ft,"
"flood_moderate_ft,flood_major_ft\n"
"USGS-NEW1,Bellevue Creek,43.467,-114.255,3.0,4.5,,\n"
"USGS-NEW2,Phantom River,42.0,-114.0,2.0,3.0,4.0,5.0\n"
)
r = client.post("/api/gauge-sites/import", json={
"format": "csv", "data": csv_data,
})
assert r.status_code == 200, r.text
assert r.json()["inserted"] == 2
r2 = client.get("/api/gauge-sites/USGS-NEW1")
assert r2.status_code == 200
assert r2.json()["gauge_name"] == "Bellevue Creek"
def test_csv_import_updates_existing(client):
"""Re-importing the same site updates rather than dupes."""
csv1 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Original,43,-115\n"
r = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv1})
assert r.json()["inserted"] == 1
csv2 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Renamed,43.5,-115.5\n"
r2 = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv2})
assert r2.json()["updated"] == 1
assert r2.json()["inserted"] == 0
r3 = client.get("/api/gauge-sites/USGS-UPSERT")
assert r3.json()["gauge_name"] == "Renamed"
def test_csv_import_skips_bad_rows(client):
csv_data = (
"site_id,gauge_name,lat,lon\n"
"USGS-GOOD,Good Gauge,43,-115\n"
",NoSiteId,42,-114\n"
"USGS-BAD,Bad Coords,not_a_number,oops\n"
)
r = client.post("/api/gauge-sites/import", json={
"format": "csv", "data": csv_data,
})
body = r.json()
assert body["inserted"] == 1
assert body["skipped"] == 2
def test_csv_import_rejects_missing_required(client):
csv_data = "gauge_name,lat,lon\nNo Site Id Column,43,-115\n"
r = client.post("/api/gauge-sites/import", json={
"format": "csv", "data": csv_data,
})
assert r.status_code == 400
def test_import_rejects_bad_format(client):
r = client.post("/api/gauge-sites/import", json={
"format": "yaml", "data": "x: 1",
})
assert r.status_code == 400
# ---- AHPS parsing (unit-level, no live HTTP) ---------------------------
def test_ahps_index_parses_gauge_links():
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_index
html = """
<html><body>
<a href="hydrograph.php?gage=hyiq2&prog=foo">HYIQ2 Cache Peak Gauge</a>
<a href="hydrograph.php?gage=bldz2">BLDZ2 Boise River</a>
<a href="other.php?gage=ignored">ignore me</a>
</body></html>
"""
gauges = _ahps_parse_index(html)
assert ("hyiq2", "HYIQ2 Cache Peak Gauge") in gauges
assert ("bldz2", "BLDZ2 Boise River") in gauges
assert len(gauges) == 2
def test_ahps_detail_extracts_thresholds():
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_detail
html = """
Latitude: 43.690
Longitude: -116.200
Action Stage 8.0 ft
Minor Flood Stage 10.5 ft
Moderate Flood Stage 12.0 ft
Major Flood Stage 14.5 ft
"""
parsed = _ahps_parse_detail(html)
assert parsed["lat"] == 43.690
assert parsed["lon"] == -116.200
assert parsed["action_ft"] == 8.0
assert parsed["flood_minor_ft"] == 10.5
assert parsed["flood_moderate_ft"] == 12.0
assert parsed["flood_major_ft"] == 14.5
# ============================================================================
# Item 4 -- WFIGS tombstone column + reminder behavior
# ============================================================================
def test_fires_has_tombstoned_at_column():
conn = get_db()
cols = {r["name"] for r in conn.execute("PRAGMA table_info(fires)").fetchall()}
assert "tombstoned_at" in cols
def test_wfigs_tombstone_stamps_column():
"""A tombstone envelope sets fires.tombstoned_at."""
from meshai.central.wfigs_handler import handle_wfigs
conn = get_db()
# Seed an active fire row.
irwin = "TOMB-1"
now = int(time.time())
conn.execute(
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
"current_acres, current_contained_pct, lat, lon, county, state, "
"declared_at, last_event_at) "
"VALUES (?,?,?,?,?,?,?,?,?,?,?)",
(irwin, "Test", "WF", 100, 10, 43.6, -116.2, "Ada", "ID", now - 3600, now),
)
n = {"_kind": "wfigs_tombstone", "irwin_id": irwin}
envelope = {"data": {"adapter": "fires", "category": "fire.incident.removed",
"id": irwin}}
handle_wfigs(n, envelope, "central.fire.incident.removed.id",
data=None, now=now)
row = conn.execute("SELECT tombstoned_at FROM fires WHERE irwin_id=?",
(irwin,)).fetchone()
assert row["tombstoned_at"] is not None
def test_reminder_skipped_when_fire_tombstoned():
"""ReminderScheduler treats fires.tombstoned_at NOT NULL as terminated."""
from meshai.notifications.reminders import ReminderScheduler
conn = get_db()
now = 1_780_000_000
irwin = "REM-TOMB"
last = now - 10 * 3600
# Active fire 10h past last broadcast (would otherwise fire)
conn.execute(
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
"current_acres, current_contained_pct, lat, lon, county, state, "
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(irwin, "T", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
last, now, last, last, now - 100), # tombstoned
)
dispatcher = MagicMock()
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
sch = ReminderScheduler(dispatcher, clock=lambda: now)
import asyncio
fired = asyncio.run(sch.tick_once())
assert fired == 0
dispatcher.dispatch_scheduled_broadcast.assert_not_called()
def test_reminder_fires_when_fire_not_tombstoned():
"""Same shape but tombstoned_at IS NULL -> reminder fires."""
from meshai.notifications.reminders import ReminderScheduler
conn = get_db()
now = 1_780_000_000
irwin = "REM-LIVE"
last = now - 10 * 3600
conn.execute(
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
"current_acres, current_contained_pct, lat, lon, county, state, "
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(irwin, "L", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
last, now, last, last, None),
)
dispatcher = MagicMock()
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
sch = ReminderScheduler(dispatcher, clock=lambda: now)
import asyncio
fired = asyncio.run(sch.tick_once())
assert fired == 1
# ============================================================================
# Item 5 -- dead-code removal
# ============================================================================
def test_incident_broadcast_heartbeat_constant_gone():
"""The dead constant is not importable anymore."""
from meshai.central import incident_handler
assert not hasattr(incident_handler, "INCIDENT_BROADCAST_HEARTBEAT_S")