mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-10 17:04:45 +02:00
feat(v0.6-tail): close 5 v0.6-phase1-complete.md follow-ups
(1) Auto-call refresh-toggles on PUT /api/config/notifications
meshai/dashboard/api/config_routes.py adds register_config_routes_hooks(app)
which registers a FastAPI HTTP middleware: on any 2xx PUT whose path
matches /api/config/notifications or /api/config, the middleware
invokes _refresh_toggle_filter(app) which reaches into app.state.bus._
pipeline_components["toggle_filter"] and calls .refresh(app.state.config).
The dashboard no longer has to remember to ping POST /api/notifications/
refresh-toggles after a toggle change. The explicit endpoint stays for
backwards-compat.
(2) env_reporter block-size cap moved to adapter_config
New registry row pipeline.env_reporter_block_chars (int, default 3000).
meshai/notifications/env_reporter.py replaces the hardcoded
_BLOCK_MAX_CHARS = 3000 with _DEFAULT_BLOCK_MAX_CHARS (the fallback) +
a _block_cap() helper that reads from adapter_config on every slice.
Mutating the row via PUT /api/adapter-config takes effect on the next
env_reporter call -- no restart.
(3) Bulk-import endpoint for gauge_sites
meshai/dashboard/api/gauge_sites_import.py adds
POST /api/gauge-sites/import with two paths:
format=csv -- expects "data" (CSV text with header row matching
gauge_sites columns: site_id, gauge_name, lat, lon,
and optionally action_ft/flood_minor_ft/
flood_moderate_ft/flood_major_ft/enabled). UPSERT
via ON CONFLICT(site_id) DO UPDATE. Returns
{inserted, updated, skipped}.
format=nws-ahps -- expects "wfo" (list of WFO codes). Fetches
water.weather.gov/ahps2/index.php?wfo=<WFO> for each,
regex-parses gauge links, then fetches up to 50
gauge detail pages per request and regex-parses
lat/lon + four threshold values. Best-effort; rows
stored under "AHPS-<gauge_id>" so they dont collide
with USGS-* ids. Returns the same shape plus
detail_fetched + errors list.
Frontend (dashboard-frontend/src/pages/GaugeSites.tsx) gains a
Import button + modal with two tabs (Paste CSV / Scrape NWS-AHPS)
rendered via an ImportModal component. CSV tab has a 48-row textarea
with the column-header hint inline; AHPS tab has a comma-separated WFO
input defaulting to BOI. Both submit via fetch() and show the JSON
response inline. Invalidates the curation cache server-side on any
successful insert/update so nwis_handler sees the new gauges on its
next call.
(4) WFIGS tombstone column -- CORRECTNESS
v12.sql adds fires.tombstoned_at REAL (nullable) + idx_fires_tombstoned_at.
meshai/central/wfigs_handler.py: the tombstone branch
(kind=="wfigs_tombstone") UPDATE fires SET tombstoned_at=COALESCE(
tombstoned_at, ?) so the first tombstone-time wins (idempotent against
repeated tombstone envelopes).
meshai/notifications/reminders/__init__.py: the wfigs tombstone
termination condition now checks row["tombstoned_at"] IS NOT NULL.
Reminders correctly STOP for closed fires -- before this change the
8h cadence would have kept Active: broadcasts going indefinitely past
a WFIGS removal.
SCHEMA_VERSION 11 -> 12.
(5) Delete INCIDENT_BROADCAST_HEARTBEAT_S
meshai/central/incident_handler.py: removed the dead constant
(v0.5.9 REVISED dropped the heartbeat path but left the constant
imported-but-never-read).
tests/test_incident_handler.py: removed the orphan
test_i_8h_heartbeat_triggers_update test (asserted None, used the
deleted constant for time arithmetic) and the stray import line.
Tests (tests/test_tail_followups.py, 16 cases):
- middleware fires refresh on PUT /api/config/notifications (200), does
NOT fire on PUT /api/config/llm
- env_reporter _block_cap() default 3000; mutate via PUT, invalidate,
next read returns the new cap
- CSV import inserts new rows, updates existing, skips bad rows,
rejects missing required columns, rejects bad format
- AHPS index parser extracts (gauge_id, name) from realistic HTML
- AHPS detail parser extracts lat/lon + four thresholds from realistic
HTML
- fires has tombstoned_at column after migrations
- wfigs tombstone branch stamps tombstoned_at
- ReminderScheduler skips a fire whose tombstoned_at is NOT NULL
- ReminderScheduler still fires for a fire whose tombstoned_at IS NULL
- INCIDENT_BROADCAST_HEARTBEAT_S no longer importable
Foundation/API test counts bumped:
REGISTRY 58 -> 59 (+ env_reporter_block_chars)
schema_meta v11 -> v12
Test count: 844 -> 859 (+16 new, -1 deleted dead test). 0 regressions.
This commit is contained in:
parent
3a410d5087
commit
566b06de06
13 changed files with 704 additions and 47 deletions
|
|
@ -29,14 +29,14 @@ def client():
|
|||
# ============================================================================
|
||||
|
||||
|
||||
def test_list_returns_all_58_keys(client):
|
||||
def test_list_returns_all_59_keys(client):
|
||||
r = client.get("/api/adapter-config")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
# 14 adapters with at least one key (itd_511 has zero -- not in the
|
||||
# grouped dict because the SQL only returns rows that exist).
|
||||
total = sum(len(v) for v in body.values())
|
||||
assert total == 58
|
||||
assert total == 59
|
||||
|
||||
|
||||
def test_list_grouped_by_adapter(client):
|
||||
|
|
|
|||
|
|
@ -54,11 +54,11 @@ def test_v6_tables_exist(fresh_db):
|
|||
assert "adapter_meta" in tables
|
||||
|
||||
|
||||
def test_schema_meta_at_v11(fresh_db):
|
||||
def test_schema_meta_at_v12(fresh_db):
|
||||
v = fresh_db.execute(
|
||||
"SELECT value FROM schema_meta WHERE key='version'"
|
||||
).fetchone()["value"]
|
||||
assert int(v) == 11
|
||||
assert int(v) == 12
|
||||
|
||||
|
||||
def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
|
||||
|
|
@ -73,9 +73,9 @@ def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
|
|||
# ---------- registry shape -----------------------------------------------
|
||||
|
||||
|
||||
def test_registry_at_58_entries():
|
||||
def test_registry_at_59_entries():
|
||||
"""v0.6-3a.1 trim: 43 CONFIG-only keys (was 77 in v0.6-3a draft)."""
|
||||
assert len(REGISTRY) == 58, (
|
||||
assert len(REGISTRY) == 59, (
|
||||
f"REGISTRY should have 43 entries after CONFIG-vs-CODE trim; got {len(REGISTRY)}. "
|
||||
f"If a sentence template / emoji / heuristic snuck in, it belongs in CODE not config."
|
||||
)
|
||||
|
|
|
|||
|
|
@ -13,8 +13,7 @@ Coverage:
|
|||
(f) magnitude bump up -> Update
|
||||
(g) delay double (>=2x) -> Update
|
||||
(h) icon change -> Update
|
||||
(i) 8h heartbeat -> Update
|
||||
|
||||
|
||||
state_511 / itd_511 EventType branching (j-m):
|
||||
(j) state_511_atis incident parses
|
||||
(k) state_511_atis closure parses
|
||||
|
|
@ -33,7 +32,6 @@ import time
|
|||
import pytest
|
||||
|
||||
from meshai.central.incident_handler import (
|
||||
INCIDENT_BROADCAST_HEARTBEAT_S,
|
||||
handle_incident,
|
||||
)
|
||||
from meshai.persistence import close_thread_connection, init_db
|
||||
|
|
@ -374,24 +372,6 @@ def test_h_icon_change_triggers_update(mem_db, no_photon):
|
|||
assert row["icon_category"] == "road_closed"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# (i) 8h heartbeat triggers Update
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def test_i_8h_heartbeat_triggers_update(mem_db, no_photon):
|
||||
env = _tomtom_env(icon_category=6, magnitude=2, delay=300)
|
||||
data1 = {}
|
||||
handle_incident(env, env["subject"], data=data1, now=1_000_000)
|
||||
_commit(data1, 1_000_001)
|
||||
|
||||
# v0.5.9 REVISED gate (A): heartbeat no longer fires Update.
|
||||
later = 1_000_001 + INCIDENT_BROADCAST_HEARTBEAT_S
|
||||
data2 = {}
|
||||
wire2 = handle_incident(env, env["subject"], data=data2, now=later)
|
||||
assert wire2 is None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# (j) state_511_atis incident parses
|
||||
# ============================================================================
|
||||
|
|
|
|||
306
tests/test_tail_followups.py
Normal file
306
tests/test_tail_followups.py
Normal file
|
|
@ -0,0 +1,306 @@
|
|||
"""v0.6-tail tests: 5 follow-ups."""
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from meshai.adapter_config import adapter_config, invalidate_cache
|
||||
from meshai.persistence import get_db
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Item 1 -- auto-refresh ToggleFilter on PUT /api/config/notifications
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def test_auto_refresh_middleware_fires_on_notifications_put():
|
||||
"""The middleware calls ToggleFilter.refresh() on a successful PUT
|
||||
that touches the notifications section."""
|
||||
from meshai.dashboard.api import config_routes
|
||||
from types import SimpleNamespace
|
||||
|
||||
refreshed = {"n": 0}
|
||||
|
||||
class _StubTF:
|
||||
def refresh(self, config):
|
||||
refreshed["n"] += 1
|
||||
|
||||
app = FastAPI()
|
||||
app.state.config = SimpleNamespace() # any truthy stand-in
|
||||
bus = SimpleNamespace()
|
||||
bus._pipeline_components = {"toggle_filter": _StubTF()}
|
||||
app.state.bus = bus
|
||||
|
||||
config_routes.register_config_routes_hooks(app)
|
||||
|
||||
@app.put("/api/config/notifications")
|
||||
async def _put(): return {"ok": True}
|
||||
|
||||
client = TestClient(app)
|
||||
client.put("/api/config/notifications", json={"enabled": True})
|
||||
assert refreshed["n"] == 1
|
||||
|
||||
|
||||
def test_auto_refresh_does_not_fire_on_other_section():
|
||||
from meshai.dashboard.api import config_routes
|
||||
from types import SimpleNamespace
|
||||
|
||||
refreshed = {"n": 0}
|
||||
|
||||
class _StubTF:
|
||||
def refresh(self, config):
|
||||
refreshed["n"] += 1
|
||||
|
||||
app = FastAPI()
|
||||
app.state.config = SimpleNamespace()
|
||||
bus = SimpleNamespace()
|
||||
bus._pipeline_components = {"toggle_filter": _StubTF()}
|
||||
app.state.bus = bus
|
||||
|
||||
config_routes.register_config_routes_hooks(app)
|
||||
|
||||
@app.put("/api/config/llm")
|
||||
async def _put(): return {"ok": True}
|
||||
|
||||
client = TestClient(app)
|
||||
client.put("/api/config/llm", json={})
|
||||
assert refreshed["n"] == 0
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Item 2 -- env_reporter cap from adapter_config
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def test_env_reporter_default_cap_3000():
|
||||
invalidate_cache()
|
||||
from meshai.notifications.env_reporter import _block_cap, _DEFAULT_BLOCK_MAX_CHARS
|
||||
assert _block_cap() == 3000
|
||||
assert _DEFAULT_BLOCK_MAX_CHARS == 3000
|
||||
|
||||
|
||||
def test_env_reporter_cap_respects_config_mutation():
|
||||
"""PUT-equivalent: change the row, invalidate, next call returns new cap."""
|
||||
invalidate_cache()
|
||||
conn = get_db()
|
||||
conn.execute(
|
||||
"UPDATE adapter_config SET value_json=? "
|
||||
"WHERE adapter='pipeline' AND key='env_reporter_block_chars'",
|
||||
("500",),
|
||||
)
|
||||
invalidate_cache()
|
||||
from meshai.notifications.env_reporter import _block_cap
|
||||
assert _block_cap() == 500
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Item 3 -- gauge_sites bulk import (CSV path)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
from meshai.dashboard.api.gauge_sites_import import router as imp_router
|
||||
from meshai.dashboard.api.curation_routes import router as cur_router
|
||||
app = FastAPI()
|
||||
app.include_router(imp_router, prefix="/api")
|
||||
app.include_router(cur_router, prefix="/api")
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_csv_import_inserts_new_rows(client):
|
||||
csv_data = (
|
||||
"site_id,gauge_name,lat,lon,action_ft,flood_minor_ft,"
|
||||
"flood_moderate_ft,flood_major_ft\n"
|
||||
"USGS-NEW1,Bellevue Creek,43.467,-114.255,3.0,4.5,,\n"
|
||||
"USGS-NEW2,Phantom River,42.0,-114.0,2.0,3.0,4.0,5.0\n"
|
||||
)
|
||||
r = client.post("/api/gauge-sites/import", json={
|
||||
"format": "csv", "data": csv_data,
|
||||
})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["inserted"] == 2
|
||||
|
||||
r2 = client.get("/api/gauge-sites/USGS-NEW1")
|
||||
assert r2.status_code == 200
|
||||
assert r2.json()["gauge_name"] == "Bellevue Creek"
|
||||
|
||||
|
||||
def test_csv_import_updates_existing(client):
|
||||
"""Re-importing the same site updates rather than dupes."""
|
||||
csv1 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Original,43,-115\n"
|
||||
r = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv1})
|
||||
assert r.json()["inserted"] == 1
|
||||
|
||||
csv2 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Renamed,43.5,-115.5\n"
|
||||
r2 = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv2})
|
||||
assert r2.json()["updated"] == 1
|
||||
assert r2.json()["inserted"] == 0
|
||||
|
||||
r3 = client.get("/api/gauge-sites/USGS-UPSERT")
|
||||
assert r3.json()["gauge_name"] == "Renamed"
|
||||
|
||||
|
||||
def test_csv_import_skips_bad_rows(client):
|
||||
csv_data = (
|
||||
"site_id,gauge_name,lat,lon\n"
|
||||
"USGS-GOOD,Good Gauge,43,-115\n"
|
||||
",NoSiteId,42,-114\n"
|
||||
"USGS-BAD,Bad Coords,not_a_number,oops\n"
|
||||
)
|
||||
r = client.post("/api/gauge-sites/import", json={
|
||||
"format": "csv", "data": csv_data,
|
||||
})
|
||||
body = r.json()
|
||||
assert body["inserted"] == 1
|
||||
assert body["skipped"] == 2
|
||||
|
||||
|
||||
def test_csv_import_rejects_missing_required(client):
|
||||
csv_data = "gauge_name,lat,lon\nNo Site Id Column,43,-115\n"
|
||||
r = client.post("/api/gauge-sites/import", json={
|
||||
"format": "csv", "data": csv_data,
|
||||
})
|
||||
assert r.status_code == 400
|
||||
|
||||
|
||||
def test_import_rejects_bad_format(client):
|
||||
r = client.post("/api/gauge-sites/import", json={
|
||||
"format": "yaml", "data": "x: 1",
|
||||
})
|
||||
assert r.status_code == 400
|
||||
|
||||
|
||||
# ---- AHPS parsing (unit-level, no live HTTP) ---------------------------
|
||||
|
||||
|
||||
def test_ahps_index_parses_gauge_links():
|
||||
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_index
|
||||
html = """
|
||||
<html><body>
|
||||
<a href="hydrograph.php?gage=hyiq2&prog=foo">HYIQ2 Cache Peak Gauge</a>
|
||||
<a href="hydrograph.php?gage=bldz2">BLDZ2 Boise River</a>
|
||||
<a href="other.php?gage=ignored">ignore me</a>
|
||||
</body></html>
|
||||
"""
|
||||
gauges = _ahps_parse_index(html)
|
||||
assert ("hyiq2", "HYIQ2 Cache Peak Gauge") in gauges
|
||||
assert ("bldz2", "BLDZ2 Boise River") in gauges
|
||||
assert len(gauges) == 2
|
||||
|
||||
|
||||
def test_ahps_detail_extracts_thresholds():
|
||||
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_detail
|
||||
html = """
|
||||
Latitude: 43.690
|
||||
Longitude: -116.200
|
||||
Action Stage 8.0 ft
|
||||
Minor Flood Stage 10.5 ft
|
||||
Moderate Flood Stage 12.0 ft
|
||||
Major Flood Stage 14.5 ft
|
||||
"""
|
||||
parsed = _ahps_parse_detail(html)
|
||||
assert parsed["lat"] == 43.690
|
||||
assert parsed["lon"] == -116.200
|
||||
assert parsed["action_ft"] == 8.0
|
||||
assert parsed["flood_minor_ft"] == 10.5
|
||||
assert parsed["flood_moderate_ft"] == 12.0
|
||||
assert parsed["flood_major_ft"] == 14.5
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Item 4 -- WFIGS tombstone column + reminder behavior
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def test_fires_has_tombstoned_at_column():
|
||||
conn = get_db()
|
||||
cols = {r["name"] for r in conn.execute("PRAGMA table_info(fires)").fetchall()}
|
||||
assert "tombstoned_at" in cols
|
||||
|
||||
|
||||
def test_wfigs_tombstone_stamps_column():
|
||||
"""A tombstone envelope sets fires.tombstoned_at."""
|
||||
from meshai.central.wfigs_handler import handle_wfigs
|
||||
conn = get_db()
|
||||
# Seed an active fire row.
|
||||
irwin = "TOMB-1"
|
||||
now = int(time.time())
|
||||
conn.execute(
|
||||
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||
"declared_at, last_event_at) "
|
||||
"VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(irwin, "Test", "WF", 100, 10, 43.6, -116.2, "Ada", "ID", now - 3600, now),
|
||||
)
|
||||
n = {"_kind": "wfigs_tombstone", "irwin_id": irwin}
|
||||
envelope = {"data": {"adapter": "fires", "category": "fire.incident.removed",
|
||||
"id": irwin}}
|
||||
handle_wfigs(n, envelope, "central.fire.incident.removed.id",
|
||||
data=None, now=now)
|
||||
row = conn.execute("SELECT tombstoned_at FROM fires WHERE irwin_id=?",
|
||||
(irwin,)).fetchone()
|
||||
assert row["tombstoned_at"] is not None
|
||||
|
||||
|
||||
def test_reminder_skipped_when_fire_tombstoned():
|
||||
"""ReminderScheduler treats fires.tombstoned_at NOT NULL as terminated."""
|
||||
from meshai.notifications.reminders import ReminderScheduler
|
||||
conn = get_db()
|
||||
now = 1_780_000_000
|
||||
irwin = "REM-TOMB"
|
||||
last = now - 10 * 3600
|
||||
# Active fire 10h past last broadcast (would otherwise fire)
|
||||
conn.execute(
|
||||
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
|
||||
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(irwin, "T", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
|
||||
last, now, last, last, now - 100), # tombstoned
|
||||
)
|
||||
dispatcher = MagicMock()
|
||||
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
|
||||
sch = ReminderScheduler(dispatcher, clock=lambda: now)
|
||||
import asyncio
|
||||
fired = asyncio.run(sch.tick_once())
|
||||
assert fired == 0
|
||||
dispatcher.dispatch_scheduled_broadcast.assert_not_called()
|
||||
|
||||
|
||||
def test_reminder_fires_when_fire_not_tombstoned():
|
||||
"""Same shape but tombstoned_at IS NULL -> reminder fires."""
|
||||
from meshai.notifications.reminders import ReminderScheduler
|
||||
conn = get_db()
|
||||
now = 1_780_000_000
|
||||
irwin = "REM-LIVE"
|
||||
last = now - 10 * 3600
|
||||
conn.execute(
|
||||
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
|
||||
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(irwin, "L", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
|
||||
last, now, last, last, None),
|
||||
)
|
||||
dispatcher = MagicMock()
|
||||
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
|
||||
sch = ReminderScheduler(dispatcher, clock=lambda: now)
|
||||
import asyncio
|
||||
fired = asyncio.run(sch.tick_once())
|
||||
assert fired == 1
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Item 5 -- dead-code removal
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def test_incident_broadcast_heartbeat_constant_gone():
|
||||
"""The dead constant is not importable anymore."""
|
||||
from meshai.central import incident_handler
|
||||
assert not hasattr(incident_handler, "INCIDENT_BROADCAST_HEARTBEAT_S")
|
||||
Loading…
Add table
Add a link
Reference in a new issue