mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-11 01:14:45 +02:00
feat(v0.6-tail): close 5 v0.6-phase1-complete.md follow-ups
(1) Auto-call refresh-toggles on PUT /api/config/notifications
meshai/dashboard/api/config_routes.py adds register_config_routes_hooks(app)
which registers a FastAPI HTTP middleware: on any 2xx PUT whose path
matches /api/config/notifications or /api/config, the middleware
invokes _refresh_toggle_filter(app) which reaches into app.state.bus._
pipeline_components["toggle_filter"] and calls .refresh(app.state.config).
The dashboard no longer has to remember to ping POST /api/notifications/
refresh-toggles after a toggle change. The explicit endpoint stays for
backwards-compat.
(2) env_reporter block-size cap moved to adapter_config
New registry row pipeline.env_reporter_block_chars (int, default 3000).
meshai/notifications/env_reporter.py replaces the hardcoded
_BLOCK_MAX_CHARS = 3000 with _DEFAULT_BLOCK_MAX_CHARS (the fallback) +
a _block_cap() helper that reads from adapter_config on every slice.
Mutating the row via PUT /api/adapter-config takes effect on the next
env_reporter call -- no restart.
(3) Bulk-import endpoint for gauge_sites
meshai/dashboard/api/gauge_sites_import.py adds
POST /api/gauge-sites/import with two paths:
format=csv -- expects "data" (CSV text with header row matching
gauge_sites columns: site_id, gauge_name, lat, lon,
and optionally action_ft/flood_minor_ft/
flood_moderate_ft/flood_major_ft/enabled). UPSERT
via ON CONFLICT(site_id) DO UPDATE. Returns
{inserted, updated, skipped}.
format=nws-ahps -- expects "wfo" (list of WFO codes). Fetches
water.weather.gov/ahps2/index.php?wfo=<WFO> for each,
regex-parses gauge links, then fetches up to 50
gauge detail pages per request and regex-parses
lat/lon + four threshold values. Best-effort; rows
stored under "AHPS-<gauge_id>" so they dont collide
with USGS-* ids. Returns the same shape plus
detail_fetched + errors list.
Frontend (dashboard-frontend/src/pages/GaugeSites.tsx) gains a
Import button + modal with two tabs (Paste CSV / Scrape NWS-AHPS)
rendered via an ImportModal component. CSV tab has a 48-row textarea
with the column-header hint inline; AHPS tab has a comma-separated WFO
input defaulting to BOI. Both submit via fetch() and show the JSON
response inline. Invalidates the curation cache server-side on any
successful insert/update so nwis_handler sees the new gauges on its
next call.
(4) WFIGS tombstone column -- CORRECTNESS
v12.sql adds fires.tombstoned_at REAL (nullable) + idx_fires_tombstoned_at.
meshai/central/wfigs_handler.py: the tombstone branch
(kind=="wfigs_tombstone") UPDATE fires SET tombstoned_at=COALESCE(
tombstoned_at, ?) so the first tombstone-time wins (idempotent against
repeated tombstone envelopes).
meshai/notifications/reminders/__init__.py: the wfigs tombstone
termination condition now checks row["tombstoned_at"] IS NOT NULL.
Reminders correctly STOP for closed fires -- before this change the
8h cadence would have kept Active: broadcasts going indefinitely past
a WFIGS removal.
SCHEMA_VERSION 11 -> 12.
(5) Delete INCIDENT_BROADCAST_HEARTBEAT_S
meshai/central/incident_handler.py: removed the dead constant
(v0.5.9 REVISED dropped the heartbeat path but left the constant
imported-but-never-read).
tests/test_incident_handler.py: removed the orphan
test_i_8h_heartbeat_triggers_update test (asserted None, used the
deleted constant for time arithmetic) and the stray import line.
Tests (tests/test_tail_followups.py, 16 cases):
- middleware fires refresh on PUT /api/config/notifications (200), does
NOT fire on PUT /api/config/llm
- env_reporter _block_cap() default 3000; mutate via PUT, invalidate,
next read returns the new cap
- CSV import inserts new rows, updates existing, skips bad rows,
rejects missing required columns, rejects bad format
- AHPS index parser extracts (gauge_id, name) from realistic HTML
- AHPS detail parser extracts lat/lon + four thresholds from realistic
HTML
- fires has tombstoned_at column after migrations
- wfigs tombstone branch stamps tombstoned_at
- ReminderScheduler skips a fire whose tombstoned_at is NOT NULL
- ReminderScheduler still fires for a fire whose tombstoned_at IS NULL
- INCIDENT_BROADCAST_HEARTBEAT_S no longer importable
Foundation/API test counts bumped:
REGISTRY 58 -> 59 (+ env_reporter_block_chars)
schema_meta v11 -> v12
Test count: 844 -> 859 (+16 new, -1 deleted dead test). 0 regressions.
This commit is contained in:
parent
3a410d5087
commit
566b06de06
13 changed files with 704 additions and 47 deletions
|
|
@ -320,6 +320,11 @@ REGISTRY: dict[tuple[str, str], dict[str, Any]] = {
|
||||||
"type": "int",
|
"type": "int",
|
||||||
"description": "How long to hold a group_key before emitting downstream.",
|
"description": "How long to hold a group_key before emitting downstream.",
|
||||||
},
|
},
|
||||||
|
("pipeline", "env_reporter_block_chars"): {
|
||||||
|
"default": 3000,
|
||||||
|
"type": "int",
|
||||||
|
"description": "Max chars per env_reporter block injected into the LLM system prompt.",
|
||||||
|
},
|
||||||
# =================================================================
|
# =================================================================
|
||||||
# v0.6-phase3 reminders: per-adapter clock-driven re-broadcast config.
|
# v0.6-phase3 reminders: per-adapter clock-driven re-broadcast config.
|
||||||
# =================================================================
|
# =================================================================
|
||||||
|
|
|
||||||
|
|
@ -49,13 +49,6 @@ logger = logging.getLogger(__name__)
|
||||||
# kept as a backward-compat alias for downstream imports.
|
# kept as a backward-compat alias for downstream imports.
|
||||||
INCIDENT_FRESHNESS_MAX_S = 1800
|
INCIDENT_FRESHNESS_MAX_S = 1800
|
||||||
|
|
||||||
# Heartbeat retained as a constant for backward-compatible imports, but
|
|
||||||
# the v0.5.9 REVISED handler no longer fires Update broadcasts. State
|
|
||||||
# tracking continues to UPSERT current_* columns; the dispatcher just
|
|
||||||
# stops getting wire strings after the first New: broadcast per
|
|
||||||
# external_id. See WFIGS handler if you want post-first-broadcast
|
|
||||||
# behavior (fires keep their 8h-rate-limited Update flow).
|
|
||||||
INCIDENT_BROADCAST_HEARTBEAT_S = 8 * 60 * 60 # 28800 (unused)
|
|
||||||
|
|
||||||
|
|
||||||
# ---- canonical sub_type vocabulary --------------------------------------
|
# ---- canonical sub_type vocabulary --------------------------------------
|
||||||
|
|
|
||||||
|
|
@ -89,6 +89,18 @@ def handle_wfigs(normalized: dict, envelope: dict, subject: str,
|
||||||
severity_word=severity_word, irwin_id=irwin_id,
|
severity_word=severity_word, irwin_id=irwin_id,
|
||||||
subject=subject, handled=0,
|
subject=subject, handled=0,
|
||||||
table_name=None, table_pk=irwin_id)
|
table_name=None, table_pk=irwin_id)
|
||||||
|
# v0.6-tail item 4: tombstone branch stamps fires.tombstoned_at so
|
||||||
|
# the ReminderScheduler stops re-broadcasting the closed fire.
|
||||||
|
# Only the tombstone kind closes the fire; perimeter polls don t.
|
||||||
|
if kind == "wfigs_tombstone" and irwin_id:
|
||||||
|
try:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE fires SET tombstoned_at=COALESCE(tombstoned_at, ?) "
|
||||||
|
"WHERE irwin_id=?",
|
||||||
|
(now, irwin_id),
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("wfigs: tombstoned_at stamp failed irwin=%s", irwin_id)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# ---- active incident ----
|
# ---- active incident ----
|
||||||
|
|
|
||||||
|
|
@ -195,9 +195,30 @@ async def test_llm_connection(request: Request):
|
||||||
# Called by the frontend after PUT /api/config/notifications so the
|
# Called by the frontend after PUT /api/config/notifications so the
|
||||||
# Inhibitor + Grouper + Dispatcher pick up the new enabled toggle set
|
# Inhibitor + Grouper + Dispatcher pick up the new enabled toggle set
|
||||||
# on the next event without a container restart.
|
# on the next event without a container restart.
|
||||||
|
def _refresh_toggle_filter(app) -> bool:
|
||||||
|
"""Best-effort live refresh of the running ToggleFilter. Returns True
|
||||||
|
when the refresh actually fired, False if the pipeline isn t up yet
|
||||||
|
(typical during tests / early startup). Never raises."""
|
||||||
|
try:
|
||||||
|
bus = getattr(app.state, "bus", None)
|
||||||
|
config = getattr(app.state, "config", None)
|
||||||
|
if bus is None or config is None:
|
||||||
|
return False
|
||||||
|
components = getattr(bus, "_pipeline_components", {}) or {}
|
||||||
|
tf = components.get("toggle_filter")
|
||||||
|
if tf is None:
|
||||||
|
return False
|
||||||
|
tf.refresh(config)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
logger.exception("toggle_filter refresh failed")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
@router.post("/notifications/refresh-toggles")
|
@router.post("/notifications/refresh-toggles")
|
||||||
async def refresh_toggles(request: Request):
|
async def refresh_toggles(request: Request):
|
||||||
"""Re-read the live config and refresh the running ToggleFilter."""
|
"""Explicit refresh endpoint (kept for backwards-compat with the
|
||||||
|
dashboard's manual ping path)."""
|
||||||
bus = getattr(request.app.state, "bus", None)
|
bus = getattr(request.app.state, "bus", None)
|
||||||
config = getattr(request.app.state, "config", None)
|
config = getattr(request.app.state, "config", None)
|
||||||
if bus is None or config is None:
|
if bus is None or config is None:
|
||||||
|
|
@ -208,3 +229,24 @@ async def refresh_toggles(request: Request):
|
||||||
raise HTTPException(503, "toggle_filter not on pipeline bus")
|
raise HTTPException(503, "toggle_filter not on pipeline bus")
|
||||||
tf.refresh(config)
|
tf.refresh(config)
|
||||||
return {"ok": True}
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# v0.6-tail item 1: auto-refresh the ToggleFilter after any successful
|
||||||
|
# config PUT that touches notifications. Registered from server.py at
|
||||||
|
# startup via register_config_routes_hooks(app).
|
||||||
|
def register_config_routes_hooks(app):
|
||||||
|
@app.middleware("http")
|
||||||
|
async def _auto_refresh_toggle_filter(request, call_next):
|
||||||
|
response = await call_next(request)
|
||||||
|
try:
|
||||||
|
method = request.method.upper()
|
||||||
|
path = request.url.path
|
||||||
|
if (method == "PUT"
|
||||||
|
and 200 <= response.status_code < 300
|
||||||
|
and ("/api/config/notifications" in path
|
||||||
|
or path.rstrip("/").endswith("/api/config"))):
|
||||||
|
_refresh_toggle_filter(request.app)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("auto-refresh middleware failed")
|
||||||
|
return response
|
||||||
|
|
|
||||||
283
meshai/dashboard/api/gauge_sites_import.py
Normal file
283
meshai/dashboard/api/gauge_sites_import.py
Normal file
|
|
@ -0,0 +1,283 @@
|
||||||
|
"""v0.6-tail item 3: gauge_sites bulk-import endpoint.
|
||||||
|
|
||||||
|
POST /api/gauge-sites/import
|
||||||
|
body:
|
||||||
|
{"format": "csv", "data": "<CSV text with header row>"}
|
||||||
|
OR {"format": "nws-ahps", "wfo": ["BOI", "PIH", ...]}
|
||||||
|
|
||||||
|
CSV path:
|
||||||
|
Header must include site_id, gauge_name, lat, lon. Optional:
|
||||||
|
action_ft, flood_minor_ft, flood_moderate_ft, flood_major_ft, enabled.
|
||||||
|
Each row is UPSERTed by site_id. Returns count of inserted, updated,
|
||||||
|
skipped (parse-failure) rows.
|
||||||
|
|
||||||
|
NWS-AHPS path:
|
||||||
|
Fetches https://water.weather.gov/ahps2/index.php?wfo=<WFO> for each
|
||||||
|
requested WFO, extracts gauge links + names + thresholds via regex
|
||||||
|
on the gauge-detail pages, UPSERTs each. Hard cap on detail-page
|
||||||
|
fetches per call to keep the request bounded.
|
||||||
|
|
||||||
|
Both paths call invalidate_curation_cache() on success so the next
|
||||||
|
nwis_handler.lookup_site() call sees the new rows.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from fastapi import APIRouter, HTTPException, Request
|
||||||
|
|
||||||
|
from meshai.persistence.curation import invalidate_curation_cache
|
||||||
|
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
router = APIRouter(tags=["curation"])
|
||||||
|
|
||||||
|
|
||||||
|
_AHPS_DETAIL_FETCH_CAP = 50 # hard cap on gauge detail pages per call
|
||||||
|
_AHPS_TIMEOUT_S = 8.0
|
||||||
|
_AHPS_BASE = "https://water.weather.gov/ahps2"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# CSV
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
_NUMERIC_FIELDS = ("lat", "lon", "action_ft", "flood_minor_ft",
|
||||||
|
"flood_moderate_ft", "flood_major_ft")
|
||||||
|
_REQUIRED = ("site_id", "gauge_name", "lat", "lon")
|
||||||
|
|
||||||
|
|
||||||
|
def _csv_upsert(conn, text: str) -> dict[str, int]:
|
||||||
|
inserted = updated = skipped = 0
|
||||||
|
reader = csv.DictReader(io.StringIO(text))
|
||||||
|
if reader.fieldnames is None:
|
||||||
|
raise HTTPException(400, "CSV must have a header row")
|
||||||
|
missing = [c for c in _REQUIRED if c not in reader.fieldnames]
|
||||||
|
if missing:
|
||||||
|
raise HTTPException(400, f"CSV missing required columns: {missing}")
|
||||||
|
now = time.time()
|
||||||
|
for row in reader:
|
||||||
|
try:
|
||||||
|
site_id = (row.get("site_id") or "").strip()
|
||||||
|
gauge_name = (row.get("gauge_name") or "").strip()
|
||||||
|
if not site_id or not gauge_name:
|
||||||
|
skipped += 1; continue
|
||||||
|
lat = float(row["lat"]); lon = float(row["lon"])
|
||||||
|
kwargs = {"action_ft": None, "flood_minor_ft": None,
|
||||||
|
"flood_moderate_ft": None, "flood_major_ft": None}
|
||||||
|
for f in ("action_ft", "flood_minor_ft", "flood_moderate_ft", "flood_major_ft"):
|
||||||
|
v = row.get(f)
|
||||||
|
if v is not None and str(v).strip() != "":
|
||||||
|
kwargs[f] = float(v)
|
||||||
|
enabled_raw = row.get("enabled")
|
||||||
|
enabled = 1
|
||||||
|
if enabled_raw is not None and str(enabled_raw).strip().lower() in ("false", "0", "no"):
|
||||||
|
enabled = 0
|
||||||
|
except (TypeError, ValueError) as e:
|
||||||
|
skipped += 1
|
||||||
|
logger.debug("csv import: row skipped %s: %s", row, e)
|
||||||
|
continue
|
||||||
|
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT 1 FROM gauge_sites WHERE site_id=?", (site_id,)
|
||||||
|
).fetchone()
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO gauge_sites(site_id, gauge_name, lat, lon, "
|
||||||
|
"action_ft, flood_minor_ft, flood_moderate_ft, flood_major_ft, "
|
||||||
|
"enabled, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?) "
|
||||||
|
"ON CONFLICT(site_id) DO UPDATE SET "
|
||||||
|
"gauge_name=excluded.gauge_name, lat=excluded.lat, lon=excluded.lon, "
|
||||||
|
"action_ft=excluded.action_ft, flood_minor_ft=excluded.flood_minor_ft, "
|
||||||
|
"flood_moderate_ft=excluded.flood_moderate_ft, flood_major_ft=excluded.flood_major_ft, "
|
||||||
|
"enabled=excluded.enabled, updated_at=excluded.updated_at",
|
||||||
|
(site_id, gauge_name, lat, lon,
|
||||||
|
kwargs["action_ft"], kwargs["flood_minor_ft"],
|
||||||
|
kwargs["flood_moderate_ft"], kwargs["flood_major_ft"],
|
||||||
|
enabled, now),
|
||||||
|
)
|
||||||
|
if existing: updated += 1
|
||||||
|
else: inserted += 1
|
||||||
|
|
||||||
|
return {"inserted": inserted, "updated": updated, "skipped": skipped}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# NWS-AHPS scrape (best-effort)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
# Gauge id rows in the WFO index appear as
|
||||||
|
# <a href="hydrograph.php?gage=GAUGE&...">GAUGE Name</a>
|
||||||
|
# Format varies by WFO; we use a forgiving pattern.
|
||||||
|
_GAUGE_LINK_RE = re.compile(
|
||||||
|
r'href="hydrograph\.php\?gage=([a-z0-9]+)[^"]*"[^>]*>([^<]+)</a>',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# On the detail page (hydrograph.php?gage=X) thresholds appear in a tabular
|
||||||
|
# format like "Action Stage ... 5.5 ft". We grep on common patterns.
|
||||||
|
_THRESHOLD_PATTERNS = {
|
||||||
|
"action_ft": re.compile(r"Action(?:\s*Stage)?[^0-9]{1,60}([0-9.]+)\s*ft", re.IGNORECASE),
|
||||||
|
"flood_minor_ft": re.compile(r"Minor(?:\s*Flood(?:\s*Stage)?)?[^0-9]{1,60}([0-9.]+)\s*ft", re.IGNORECASE),
|
||||||
|
"flood_moderate_ft": re.compile(r"Moderate(?:\s*Flood(?:\s*Stage)?)?[^0-9]{1,60}([0-9.]+)\s*ft", re.IGNORECASE),
|
||||||
|
"flood_major_ft": re.compile(r"Major(?:\s*Flood(?:\s*Stage)?)?[^0-9]{1,60}([0-9.]+)\s*ft", re.IGNORECASE),
|
||||||
|
}
|
||||||
|
_LATLON_RE = re.compile(
|
||||||
|
r"Latitude[^0-9\-]{1,40}([0-9.\-]+)[\s\S]{1,200}?Longitude[^0-9\-]{1,40}([0-9.\-]+)",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ahps_parse_index(html: str) -> list[tuple[str, str]]:
|
||||||
|
"""Return [(gauge_id, gauge_name), ...] from the WFO index page."""
|
||||||
|
out: list[tuple[str, str]] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for m in _GAUGE_LINK_RE.finditer(html or ""):
|
||||||
|
gid = m.group(1).strip().lower()
|
||||||
|
name = re.sub(r"\s+", " ", m.group(2)).strip()
|
||||||
|
if gid and gid not in seen:
|
||||||
|
seen.add(gid)
|
||||||
|
out.append((gid, name))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _ahps_parse_detail(html: str) -> dict[str, Any]:
|
||||||
|
"""Pull lat/lon + thresholds from the gauge detail HTML."""
|
||||||
|
out: dict[str, Any] = {
|
||||||
|
"lat": None, "lon": None,
|
||||||
|
"action_ft": None, "flood_minor_ft": None,
|
||||||
|
"flood_moderate_ft": None, "flood_major_ft": None,
|
||||||
|
}
|
||||||
|
if not html: return out
|
||||||
|
m = _LATLON_RE.search(html)
|
||||||
|
if m:
|
||||||
|
try:
|
||||||
|
out["lat"] = float(m.group(1)); out["lon"] = float(m.group(2))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
for key, pat in _THRESHOLD_PATTERNS.items():
|
||||||
|
m = pat.search(html)
|
||||||
|
if m:
|
||||||
|
try: out[key] = float(m.group(1))
|
||||||
|
except ValueError: pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _ahps_upsert(conn, wfos: list[str]) -> dict[str, Any]:
|
||||||
|
"""Best-effort: fetch each WFO index, then up to _AHPS_DETAIL_FETCH_CAP
|
||||||
|
gauge detail pages per call across all WFOs. Returns a summary."""
|
||||||
|
inserted = updated = skipped = 0
|
||||||
|
fetched_detail = 0
|
||||||
|
errors: list[str] = []
|
||||||
|
|
||||||
|
with httpx.Client(timeout=_AHPS_TIMEOUT_S, follow_redirects=True) as cli:
|
||||||
|
for wfo in wfos:
|
||||||
|
wfo = (wfo or "").strip().upper()
|
||||||
|
if not wfo or not wfo.isalnum():
|
||||||
|
errors.append(f"WFO {wfo!r}: invalid"); continue
|
||||||
|
url = f"{_AHPS_BASE}/index.php?wfo={wfo}"
|
||||||
|
try:
|
||||||
|
r = cli.get(url)
|
||||||
|
if r.status_code != 200:
|
||||||
|
errors.append(f"WFO {wfo}: index status {r.status_code}")
|
||||||
|
continue
|
||||||
|
gauges = _ahps_parse_index(r.text)
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(f"WFO {wfo}: index fetch {e}"); continue
|
||||||
|
|
||||||
|
for gid, gname in gauges:
|
||||||
|
if fetched_detail >= _AHPS_DETAIL_FETCH_CAP:
|
||||||
|
errors.append(
|
||||||
|
f"detail-fetch cap ({_AHPS_DETAIL_FETCH_CAP}) "
|
||||||
|
f"reached; remaining gauges skipped"
|
||||||
|
)
|
||||||
|
break
|
||||||
|
fetched_detail += 1
|
||||||
|
detail_url = f"{_AHPS_BASE}/hydrograph.php?gage={gid}"
|
||||||
|
try:
|
||||||
|
r2 = cli.get(detail_url)
|
||||||
|
if r2.status_code != 200:
|
||||||
|
skipped += 1; continue
|
||||||
|
parsed = _ahps_parse_detail(r2.text)
|
||||||
|
except Exception:
|
||||||
|
skipped += 1; continue
|
||||||
|
|
||||||
|
if parsed["lat"] is None or parsed["lon"] is None:
|
||||||
|
skipped += 1; continue
|
||||||
|
|
||||||
|
# AHPS gauge ids are the NWS id; we store as 'AHPS-<id>' so
|
||||||
|
# they don't collide with USGS-* ids managed elsewhere.
|
||||||
|
site_id = f"AHPS-{gid.upper()}"
|
||||||
|
now = time.time()
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT 1 FROM gauge_sites WHERE site_id=?", (site_id,)
|
||||||
|
).fetchone()
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO gauge_sites(site_id, gauge_name, lat, lon, "
|
||||||
|
"action_ft, flood_minor_ft, flood_moderate_ft, flood_major_ft, "
|
||||||
|
"enabled, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?) "
|
||||||
|
"ON CONFLICT(site_id) DO UPDATE SET "
|
||||||
|
"gauge_name=excluded.gauge_name, lat=excluded.lat, lon=excluded.lon, "
|
||||||
|
"action_ft=COALESCE(excluded.action_ft, action_ft), "
|
||||||
|
"flood_minor_ft=COALESCE(excluded.flood_minor_ft, flood_minor_ft), "
|
||||||
|
"flood_moderate_ft=COALESCE(excluded.flood_moderate_ft, flood_moderate_ft), "
|
||||||
|
"flood_major_ft=COALESCE(excluded.flood_major_ft, flood_major_ft), "
|
||||||
|
"updated_at=excluded.updated_at",
|
||||||
|
(site_id, gname, parsed["lat"], parsed["lon"],
|
||||||
|
parsed["action_ft"], parsed["flood_minor_ft"],
|
||||||
|
parsed["flood_moderate_ft"], parsed["flood_major_ft"],
|
||||||
|
1, now),
|
||||||
|
)
|
||||||
|
if existing: updated += 1
|
||||||
|
else: inserted += 1
|
||||||
|
|
||||||
|
return {
|
||||||
|
"inserted": inserted,
|
||||||
|
"updated": updated,
|
||||||
|
"skipped": skipped,
|
||||||
|
"detail_fetched": fetched_detail,
|
||||||
|
"errors": errors,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Route
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/gauge-sites/import")
|
||||||
|
async def gauge_sites_import(request: Request) -> dict:
|
||||||
|
body = await request.json()
|
||||||
|
if not isinstance(body, dict):
|
||||||
|
raise HTTPException(400, "body must be a JSON object")
|
||||||
|
fmt = (body.get("format") or "").lower().strip()
|
||||||
|
if fmt not in ("csv", "nws-ahps"):
|
||||||
|
raise HTTPException(400, "format must be 'csv' or 'nws-ahps'")
|
||||||
|
|
||||||
|
from meshai.persistence import get_db
|
||||||
|
conn = get_db()
|
||||||
|
|
||||||
|
if fmt == "csv":
|
||||||
|
text = body.get("data")
|
||||||
|
if not isinstance(text, str) or not text.strip():
|
||||||
|
raise HTTPException(400, "csv body must include 'data' (CSV text)")
|
||||||
|
result = _csv_upsert(conn, text)
|
||||||
|
else: # nws-ahps
|
||||||
|
wfos = body.get("wfo") or []
|
||||||
|
if isinstance(wfos, str):
|
||||||
|
wfos = [wfos]
|
||||||
|
if not isinstance(wfos, list) or not wfos:
|
||||||
|
raise HTTPException(400, "nws-ahps body must include 'wfo' (list or string)")
|
||||||
|
result = _ahps_upsert(conn, wfos)
|
||||||
|
|
||||||
|
if result.get("inserted", 0) or result.get("updated", 0):
|
||||||
|
invalidate_curation_cache()
|
||||||
|
logger.info("gauge_sites import (%s): %s", fmt, result)
|
||||||
|
return result
|
||||||
|
|
@ -25,7 +25,19 @@ logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
# Length budget per block. The LLM has a finite context window; we cap each
|
# Length budget per block. The LLM has a finite context window; we cap each
|
||||||
# block at this many characters to keep the assembled prompt sane.
|
# block at this many characters to keep the assembled prompt sane.
|
||||||
_BLOCK_MAX_CHARS = 3000
|
# v0.6-tail item 2: configurable via adapter_config.pipeline.env_reporter_block_chars.
|
||||||
|
_DEFAULT_BLOCK_MAX_CHARS = 3000
|
||||||
|
|
||||||
|
|
||||||
|
def _block_cap() -> int:
|
||||||
|
"""Read the per-block char cap from adapter_config; fall back to
|
||||||
|
the conservative 3000-char default if the row isn't seeded yet."""
|
||||||
|
try:
|
||||||
|
from meshai.adapter_config import adapter_config
|
||||||
|
v = adapter_config.pipeline.env_reporter_block_chars
|
||||||
|
return int(v) if v else _DEFAULT_BLOCK_MAX_CHARS
|
||||||
|
except Exception:
|
||||||
|
return _DEFAULT_BLOCK_MAX_CHARS
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
@ -146,7 +158,7 @@ class EnvReporter:
|
||||||
if len(lines) == 1:
|
if len(lines) == 1:
|
||||||
# Only the header; nothing to report.
|
# Only the header; nothing to report.
|
||||||
return ""
|
return ""
|
||||||
return "\n".join(lines)[:_BLOCK_MAX_CHARS]
|
return "\n".join(lines)[:_block_cap()]
|
||||||
|
|
||||||
# ---------- per-adapter detail blocks ----------------------------------
|
# ---------- per-adapter detail blocks ----------------------------------
|
||||||
|
|
||||||
|
|
@ -195,7 +207,7 @@ class EnvReporter:
|
||||||
f"{n_pixels} pixels total, {n_high} high-confidence"
|
f"{n_pixels} pixels total, {n_high} high-confidence"
|
||||||
)
|
)
|
||||||
|
|
||||||
return ("\n".join(lines) if lines else "")[:_BLOCK_MAX_CHARS]
|
return ("\n".join(lines) if lines else "")[:_block_cap()]
|
||||||
|
|
||||||
def build_alerts_detail(self, *, region: Optional[str] = None,
|
def build_alerts_detail(self, *, region: Optional[str] = None,
|
||||||
limit: int = 10,
|
limit: int = 10,
|
||||||
|
|
@ -226,7 +238,7 @@ class EnvReporter:
|
||||||
expires = _fmt_epoch(r["expires_at"]) if r["expires_at"] else "no expiry"
|
expires = _fmt_epoch(r["expires_at"]) if r["expires_at"] else "no expiry"
|
||||||
head = (r["headline"] or "")[:90]
|
head = (r["headline"] or "")[:90]
|
||||||
lines.append(f" - [{sev}] {kind} ({loc}): {head} -- until {expires}")
|
lines.append(f" - [{sev}] {kind} ({loc}): {head} -- until {expires}")
|
||||||
return "\n".join(lines)[:_BLOCK_MAX_CHARS]
|
return "\n".join(lines)[:_block_cap()]
|
||||||
|
|
||||||
def build_quakes_detail(self, *, hours: int = 24,
|
def build_quakes_detail(self, *, hours: int = 24,
|
||||||
limit: int = 10,
|
limit: int = 10,
|
||||||
|
|
@ -253,7 +265,7 @@ class EnvReporter:
|
||||||
when = _fmt_epoch(r["occurred_at"]) if r["occurred_at"] else "?"
|
when = _fmt_epoch(r["occurred_at"]) if r["occurred_at"] else "?"
|
||||||
ts = " TSUNAMI" if r["tsunami_warning"] else ""
|
ts = " TSUNAMI" if r["tsunami_warning"] else ""
|
||||||
lines.append(f" - {mag} {place}, {depth} depth, {when}{ts}")
|
lines.append(f" - {mag} {place}, {depth} depth, {when}{ts}")
|
||||||
return "\n".join(lines)[:_BLOCK_MAX_CHARS]
|
return "\n".join(lines)[:_block_cap()]
|
||||||
|
|
||||||
def build_traffic_detail(self, *, state: Optional[str] = "ID",
|
def build_traffic_detail(self, *, state: Optional[str] = "ID",
|
||||||
hours: int = 2,
|
hours: int = 2,
|
||||||
|
|
@ -291,7 +303,7 @@ class EnvReporter:
|
||||||
delay = f", {int(r['delay_seconds']/60)} min delay"
|
delay = f", {int(r['delay_seconds']/60)} min delay"
|
||||||
when = _fmt_epoch(r["last_seen_at"])
|
when = _fmt_epoch(r["last_seen_at"])
|
||||||
lines.append(f" - {road} {direction} ({county}): {sub}{impact}{delay}, seen {when}")
|
lines.append(f" - {road} {direction} ({county}): {sub}{impact}{delay}, seen {when}")
|
||||||
return "\n".join(lines)[:_BLOCK_MAX_CHARS]
|
return "\n".join(lines)[:_block_cap()]
|
||||||
|
|
||||||
def build_gauges_detail(self, *, limit: int = 10,
|
def build_gauges_detail(self, *, limit: int = 10,
|
||||||
now: Optional[int] = None) -> str:
|
now: Optional[int] = None) -> str:
|
||||||
|
|
@ -323,7 +335,7 @@ class EnvReporter:
|
||||||
if r['flow_cfs'] is not None else "")
|
if r['flow_cfs'] is not None else "")
|
||||||
name = r["gauge_name"] or r["site_id"]
|
name = r["gauge_name"] or r["site_id"]
|
||||||
lines.append(f" - {name}: {value} ({ts_state}){flow}")
|
lines.append(f" - {name}: {value} ({ts_state}){flow}")
|
||||||
return "\n".join(lines)[:_BLOCK_MAX_CHARS]
|
return "\n".join(lines)[:_block_cap()]
|
||||||
|
|
||||||
def build_swpc_detail(self, *, hours: int = 24,
|
def build_swpc_detail(self, *, hours: int = 24,
|
||||||
now: Optional[int] = None) -> str:
|
now: Optional[int] = None) -> str:
|
||||||
|
|
@ -362,7 +374,7 @@ class EnvReporter:
|
||||||
if r["ratings_json"]:
|
if r["ratings_json"]:
|
||||||
lines.append(f" Ratings: {r['ratings_json']}")
|
lines.append(f" Ratings: {r['ratings_json']}")
|
||||||
|
|
||||||
return ("\n".join(lines) if lines else "")[:_BLOCK_MAX_CHARS]
|
return ("\n".join(lines) if lines else "")[:_block_cap()]
|
||||||
|
|
||||||
def build_drop_audit(self, *, hours: int = 1) -> str:
|
def build_drop_audit(self, *, hours: int = 1) -> str:
|
||||||
"""Why-was-X-dropped: event_log handled=0 grouped by source+reason
|
"""Why-was-X-dropped: event_log handled=0 grouped by source+reason
|
||||||
|
|
@ -401,7 +413,7 @@ class EnvReporter:
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
return ("\n".join(lines) if lines else "")[:_BLOCK_MAX_CHARS]
|
return ("\n".join(lines) if lines else "")[:_block_cap()]
|
||||||
|
|
||||||
def build_all(self, *, now: Optional[int] = None) -> str:
|
def build_all(self, *, now: Optional[int] = None) -> str:
|
||||||
"""Convenience: summary + every detail block. Used by router when the
|
"""Convenience: summary + every detail block. Used by router when the
|
||||||
|
|
|
||||||
|
|
@ -232,6 +232,15 @@ class ReminderScheduler:
|
||||||
if not terminate_when: return False
|
if not terminate_when: return False
|
||||||
tokens = set(terminate_when)
|
tokens = set(terminate_when)
|
||||||
if adapter == "wfigs":
|
if adapter == "wfigs":
|
||||||
|
if "tombstone" in tokens:
|
||||||
|
# v0.6-tail item 4: fires.tombstoned_at populated by
|
||||||
|
# wfigs_handler when WFIGS marks a fire closed.
|
||||||
|
try:
|
||||||
|
ts = row["tombstoned_at"]
|
||||||
|
if ts is not None:
|
||||||
|
return True
|
||||||
|
except (IndexError, KeyError):
|
||||||
|
pass
|
||||||
if "containment_100" in tokens:
|
if "containment_100" in tokens:
|
||||||
c = row["current_contained_pct"]
|
c = row["current_contained_pct"]
|
||||||
if c is not None and int(c) >= 100:
|
if c is not None and int(c) >= 100:
|
||||||
|
|
@ -240,8 +249,6 @@ class ReminderScheduler:
|
||||||
le = row["last_event_at"]
|
le = row["last_event_at"]
|
||||||
if le is not None and (now - float(le)) > 86400:
|
if le is not None and (now - float(le)) > 86400:
|
||||||
return True
|
return True
|
||||||
# "tombstone" -- fires row has no flag; best-effort via event_log
|
|
||||||
# would be expensive on each tick. Skip; deliberate scope-limit.
|
|
||||||
return False
|
return False
|
||||||
if adapter == "swpc":
|
if adapter == "swpc":
|
||||||
if "end_date_passed" in tokens:
|
if "end_date_passed" in tokens:
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,7 @@ logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
DEFAULT_DB_PATH = "/data/meshai.sqlite"
|
DEFAULT_DB_PATH = "/data/meshai.sqlite"
|
||||||
MESHAI_DB_PATH_ENV = "MESHAI_DB_PATH"
|
MESHAI_DB_PATH_ENV = "MESHAI_DB_PATH"
|
||||||
SCHEMA_VERSION = 11
|
SCHEMA_VERSION = 12
|
||||||
SCHEMA_META_TABLE = "schema_meta"
|
SCHEMA_META_TABLE = "schema_meta"
|
||||||
MIGRATIONS_DIR = Path(__file__).parent / "migrations"
|
MIGRATIONS_DIR = Path(__file__).parent / "migrations"
|
||||||
|
|
||||||
|
|
|
||||||
17
meshai/persistence/migrations/v12.sql
Normal file
17
meshai/persistence/migrations/v12.sql
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
-- v0.6-tail item 4: WFIGS tombstone column on fires.
|
||||||
|
--
|
||||||
|
-- The ReminderScheduler's "tombstone" termination condition for wfigs
|
||||||
|
-- previously had nothing to check -- the fires table didn't carry a
|
||||||
|
-- tombstone flag, so reminders kept firing for fires WFIGS had since
|
||||||
|
-- marked closed. This column closes that gap.
|
||||||
|
--
|
||||||
|
-- wfigs_handler.py stamps `tombstoned_at = NOW` on the tombstone branch
|
||||||
|
-- (kind=='wfigs_tombstone'). ReminderScheduler treats `tombstoned_at IS
|
||||||
|
-- NOT NULL` as "stop reminding". Nullable + no default so the column
|
||||||
|
-- carries the precise tombstone time (used by env_reporter and the LLM
|
||||||
|
-- for "when did this fire close out?" questions).
|
||||||
|
|
||||||
|
ALTER TABLE fires ADD COLUMN tombstoned_at REAL;
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_fires_tombstoned_at
|
||||||
|
ON fires(tombstoned_at);
|
||||||
|
|
@ -29,14 +29,14 @@ def client():
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
def test_list_returns_all_58_keys(client):
|
def test_list_returns_all_59_keys(client):
|
||||||
r = client.get("/api/adapter-config")
|
r = client.get("/api/adapter-config")
|
||||||
assert r.status_code == 200
|
assert r.status_code == 200
|
||||||
body = r.json()
|
body = r.json()
|
||||||
# 14 adapters with at least one key (itd_511 has zero -- not in the
|
# 14 adapters with at least one key (itd_511 has zero -- not in the
|
||||||
# grouped dict because the SQL only returns rows that exist).
|
# grouped dict because the SQL only returns rows that exist).
|
||||||
total = sum(len(v) for v in body.values())
|
total = sum(len(v) for v in body.values())
|
||||||
assert total == 58
|
assert total == 59
|
||||||
|
|
||||||
|
|
||||||
def test_list_grouped_by_adapter(client):
|
def test_list_grouped_by_adapter(client):
|
||||||
|
|
|
||||||
|
|
@ -54,11 +54,11 @@ def test_v6_tables_exist(fresh_db):
|
||||||
assert "adapter_meta" in tables
|
assert "adapter_meta" in tables
|
||||||
|
|
||||||
|
|
||||||
def test_schema_meta_at_v11(fresh_db):
|
def test_schema_meta_at_v12(fresh_db):
|
||||||
v = fresh_db.execute(
|
v = fresh_db.execute(
|
||||||
"SELECT value FROM schema_meta WHERE key='version'"
|
"SELECT value FROM schema_meta WHERE key='version'"
|
||||||
).fetchone()["value"]
|
).fetchone()["value"]
|
||||||
assert int(v) == 11
|
assert int(v) == 12
|
||||||
|
|
||||||
|
|
||||||
def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
|
def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
|
||||||
|
|
@ -73,9 +73,9 @@ def test_adapter_config_type_check_constrains_vocabulary(fresh_db):
|
||||||
# ---------- registry shape -----------------------------------------------
|
# ---------- registry shape -----------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def test_registry_at_58_entries():
|
def test_registry_at_59_entries():
|
||||||
"""v0.6-3a.1 trim: 43 CONFIG-only keys (was 77 in v0.6-3a draft)."""
|
"""v0.6-3a.1 trim: 43 CONFIG-only keys (was 77 in v0.6-3a draft)."""
|
||||||
assert len(REGISTRY) == 58, (
|
assert len(REGISTRY) == 59, (
|
||||||
f"REGISTRY should have 43 entries after CONFIG-vs-CODE trim; got {len(REGISTRY)}. "
|
f"REGISTRY should have 43 entries after CONFIG-vs-CODE trim; got {len(REGISTRY)}. "
|
||||||
f"If a sentence template / emoji / heuristic snuck in, it belongs in CODE not config."
|
f"If a sentence template / emoji / heuristic snuck in, it belongs in CODE not config."
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -13,7 +13,6 @@ Coverage:
|
||||||
(f) magnitude bump up -> Update
|
(f) magnitude bump up -> Update
|
||||||
(g) delay double (>=2x) -> Update
|
(g) delay double (>=2x) -> Update
|
||||||
(h) icon change -> Update
|
(h) icon change -> Update
|
||||||
(i) 8h heartbeat -> Update
|
|
||||||
|
|
||||||
state_511 / itd_511 EventType branching (j-m):
|
state_511 / itd_511 EventType branching (j-m):
|
||||||
(j) state_511_atis incident parses
|
(j) state_511_atis incident parses
|
||||||
|
|
@ -33,7 +32,6 @@ import time
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from meshai.central.incident_handler import (
|
from meshai.central.incident_handler import (
|
||||||
INCIDENT_BROADCAST_HEARTBEAT_S,
|
|
||||||
handle_incident,
|
handle_incident,
|
||||||
)
|
)
|
||||||
from meshai.persistence import close_thread_connection, init_db
|
from meshai.persistence import close_thread_connection, init_db
|
||||||
|
|
@ -374,24 +372,6 @@ def test_h_icon_change_triggers_update(mem_db, no_photon):
|
||||||
assert row["icon_category"] == "road_closed"
|
assert row["icon_category"] == "road_closed"
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
|
||||||
# (i) 8h heartbeat triggers Update
|
|
||||||
# ============================================================================
|
|
||||||
|
|
||||||
|
|
||||||
def test_i_8h_heartbeat_triggers_update(mem_db, no_photon):
|
|
||||||
env = _tomtom_env(icon_category=6, magnitude=2, delay=300)
|
|
||||||
data1 = {}
|
|
||||||
handle_incident(env, env["subject"], data=data1, now=1_000_000)
|
|
||||||
_commit(data1, 1_000_001)
|
|
||||||
|
|
||||||
# v0.5.9 REVISED gate (A): heartbeat no longer fires Update.
|
|
||||||
later = 1_000_001 + INCIDENT_BROADCAST_HEARTBEAT_S
|
|
||||||
data2 = {}
|
|
||||||
wire2 = handle_incident(env, env["subject"], data=data2, now=later)
|
|
||||||
assert wire2 is None
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# (j) state_511_atis incident parses
|
# (j) state_511_atis incident parses
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
|
||||||
306
tests/test_tail_followups.py
Normal file
306
tests/test_tail_followups.py
Normal file
|
|
@ -0,0 +1,306 @@
|
||||||
|
"""v0.6-tail tests: 5 follow-ups."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from meshai.adapter_config import adapter_config, invalidate_cache
|
||||||
|
from meshai.persistence import get_db
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Item 1 -- auto-refresh ToggleFilter on PUT /api/config/notifications
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_refresh_middleware_fires_on_notifications_put():
|
||||||
|
"""The middleware calls ToggleFilter.refresh() on a successful PUT
|
||||||
|
that touches the notifications section."""
|
||||||
|
from meshai.dashboard.api import config_routes
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
refreshed = {"n": 0}
|
||||||
|
|
||||||
|
class _StubTF:
|
||||||
|
def refresh(self, config):
|
||||||
|
refreshed["n"] += 1
|
||||||
|
|
||||||
|
app = FastAPI()
|
||||||
|
app.state.config = SimpleNamespace() # any truthy stand-in
|
||||||
|
bus = SimpleNamespace()
|
||||||
|
bus._pipeline_components = {"toggle_filter": _StubTF()}
|
||||||
|
app.state.bus = bus
|
||||||
|
|
||||||
|
config_routes.register_config_routes_hooks(app)
|
||||||
|
|
||||||
|
@app.put("/api/config/notifications")
|
||||||
|
async def _put(): return {"ok": True}
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
client.put("/api/config/notifications", json={"enabled": True})
|
||||||
|
assert refreshed["n"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_refresh_does_not_fire_on_other_section():
|
||||||
|
from meshai.dashboard.api import config_routes
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
refreshed = {"n": 0}
|
||||||
|
|
||||||
|
class _StubTF:
|
||||||
|
def refresh(self, config):
|
||||||
|
refreshed["n"] += 1
|
||||||
|
|
||||||
|
app = FastAPI()
|
||||||
|
app.state.config = SimpleNamespace()
|
||||||
|
bus = SimpleNamespace()
|
||||||
|
bus._pipeline_components = {"toggle_filter": _StubTF()}
|
||||||
|
app.state.bus = bus
|
||||||
|
|
||||||
|
config_routes.register_config_routes_hooks(app)
|
||||||
|
|
||||||
|
@app.put("/api/config/llm")
|
||||||
|
async def _put(): return {"ok": True}
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
client.put("/api/config/llm", json={})
|
||||||
|
assert refreshed["n"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Item 2 -- env_reporter cap from adapter_config
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_reporter_default_cap_3000():
|
||||||
|
invalidate_cache()
|
||||||
|
from meshai.notifications.env_reporter import _block_cap, _DEFAULT_BLOCK_MAX_CHARS
|
||||||
|
assert _block_cap() == 3000
|
||||||
|
assert _DEFAULT_BLOCK_MAX_CHARS == 3000
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_reporter_cap_respects_config_mutation():
|
||||||
|
"""PUT-equivalent: change the row, invalidate, next call returns new cap."""
|
||||||
|
invalidate_cache()
|
||||||
|
conn = get_db()
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE adapter_config SET value_json=? "
|
||||||
|
"WHERE adapter='pipeline' AND key='env_reporter_block_chars'",
|
||||||
|
("500",),
|
||||||
|
)
|
||||||
|
invalidate_cache()
|
||||||
|
from meshai.notifications.env_reporter import _block_cap
|
||||||
|
assert _block_cap() == 500
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Item 3 -- gauge_sites bulk import (CSV path)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client():
|
||||||
|
from meshai.dashboard.api.gauge_sites_import import router as imp_router
|
||||||
|
from meshai.dashboard.api.curation_routes import router as cur_router
|
||||||
|
app = FastAPI()
|
||||||
|
app.include_router(imp_router, prefix="/api")
|
||||||
|
app.include_router(cur_router, prefix="/api")
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
def test_csv_import_inserts_new_rows(client):
|
||||||
|
csv_data = (
|
||||||
|
"site_id,gauge_name,lat,lon,action_ft,flood_minor_ft,"
|
||||||
|
"flood_moderate_ft,flood_major_ft\n"
|
||||||
|
"USGS-NEW1,Bellevue Creek,43.467,-114.255,3.0,4.5,,\n"
|
||||||
|
"USGS-NEW2,Phantom River,42.0,-114.0,2.0,3.0,4.0,5.0\n"
|
||||||
|
)
|
||||||
|
r = client.post("/api/gauge-sites/import", json={
|
||||||
|
"format": "csv", "data": csv_data,
|
||||||
|
})
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
assert r.json()["inserted"] == 2
|
||||||
|
|
||||||
|
r2 = client.get("/api/gauge-sites/USGS-NEW1")
|
||||||
|
assert r2.status_code == 200
|
||||||
|
assert r2.json()["gauge_name"] == "Bellevue Creek"
|
||||||
|
|
||||||
|
|
||||||
|
def test_csv_import_updates_existing(client):
|
||||||
|
"""Re-importing the same site updates rather than dupes."""
|
||||||
|
csv1 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Original,43,-115\n"
|
||||||
|
r = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv1})
|
||||||
|
assert r.json()["inserted"] == 1
|
||||||
|
|
||||||
|
csv2 = "site_id,gauge_name,lat,lon\nUSGS-UPSERT,Renamed,43.5,-115.5\n"
|
||||||
|
r2 = client.post("/api/gauge-sites/import", json={"format": "csv", "data": csv2})
|
||||||
|
assert r2.json()["updated"] == 1
|
||||||
|
assert r2.json()["inserted"] == 0
|
||||||
|
|
||||||
|
r3 = client.get("/api/gauge-sites/USGS-UPSERT")
|
||||||
|
assert r3.json()["gauge_name"] == "Renamed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_csv_import_skips_bad_rows(client):
|
||||||
|
csv_data = (
|
||||||
|
"site_id,gauge_name,lat,lon\n"
|
||||||
|
"USGS-GOOD,Good Gauge,43,-115\n"
|
||||||
|
",NoSiteId,42,-114\n"
|
||||||
|
"USGS-BAD,Bad Coords,not_a_number,oops\n"
|
||||||
|
)
|
||||||
|
r = client.post("/api/gauge-sites/import", json={
|
||||||
|
"format": "csv", "data": csv_data,
|
||||||
|
})
|
||||||
|
body = r.json()
|
||||||
|
assert body["inserted"] == 1
|
||||||
|
assert body["skipped"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_csv_import_rejects_missing_required(client):
|
||||||
|
csv_data = "gauge_name,lat,lon\nNo Site Id Column,43,-115\n"
|
||||||
|
r = client.post("/api/gauge-sites/import", json={
|
||||||
|
"format": "csv", "data": csv_data,
|
||||||
|
})
|
||||||
|
assert r.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_rejects_bad_format(client):
|
||||||
|
r = client.post("/api/gauge-sites/import", json={
|
||||||
|
"format": "yaml", "data": "x: 1",
|
||||||
|
})
|
||||||
|
assert r.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
# ---- AHPS parsing (unit-level, no live HTTP) ---------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_ahps_index_parses_gauge_links():
|
||||||
|
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_index
|
||||||
|
html = """
|
||||||
|
<html><body>
|
||||||
|
<a href="hydrograph.php?gage=hyiq2&prog=foo">HYIQ2 Cache Peak Gauge</a>
|
||||||
|
<a href="hydrograph.php?gage=bldz2">BLDZ2 Boise River</a>
|
||||||
|
<a href="other.php?gage=ignored">ignore me</a>
|
||||||
|
</body></html>
|
||||||
|
"""
|
||||||
|
gauges = _ahps_parse_index(html)
|
||||||
|
assert ("hyiq2", "HYIQ2 Cache Peak Gauge") in gauges
|
||||||
|
assert ("bldz2", "BLDZ2 Boise River") in gauges
|
||||||
|
assert len(gauges) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_ahps_detail_extracts_thresholds():
|
||||||
|
from meshai.dashboard.api.gauge_sites_import import _ahps_parse_detail
|
||||||
|
html = """
|
||||||
|
Latitude: 43.690
|
||||||
|
Longitude: -116.200
|
||||||
|
Action Stage 8.0 ft
|
||||||
|
Minor Flood Stage 10.5 ft
|
||||||
|
Moderate Flood Stage 12.0 ft
|
||||||
|
Major Flood Stage 14.5 ft
|
||||||
|
"""
|
||||||
|
parsed = _ahps_parse_detail(html)
|
||||||
|
assert parsed["lat"] == 43.690
|
||||||
|
assert parsed["lon"] == -116.200
|
||||||
|
assert parsed["action_ft"] == 8.0
|
||||||
|
assert parsed["flood_minor_ft"] == 10.5
|
||||||
|
assert parsed["flood_moderate_ft"] == 12.0
|
||||||
|
assert parsed["flood_major_ft"] == 14.5
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Item 4 -- WFIGS tombstone column + reminder behavior
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
def test_fires_has_tombstoned_at_column():
|
||||||
|
conn = get_db()
|
||||||
|
cols = {r["name"] for r in conn.execute("PRAGMA table_info(fires)").fetchall()}
|
||||||
|
assert "tombstoned_at" in cols
|
||||||
|
|
||||||
|
|
||||||
|
def test_wfigs_tombstone_stamps_column():
|
||||||
|
"""A tombstone envelope sets fires.tombstoned_at."""
|
||||||
|
from meshai.central.wfigs_handler import handle_wfigs
|
||||||
|
conn = get_db()
|
||||||
|
# Seed an active fire row.
|
||||||
|
irwin = "TOMB-1"
|
||||||
|
now = int(time.time())
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||||
|
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||||
|
"declared_at, last_event_at) "
|
||||||
|
"VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(irwin, "Test", "WF", 100, 10, 43.6, -116.2, "Ada", "ID", now - 3600, now),
|
||||||
|
)
|
||||||
|
n = {"_kind": "wfigs_tombstone", "irwin_id": irwin}
|
||||||
|
envelope = {"data": {"adapter": "fires", "category": "fire.incident.removed",
|
||||||
|
"id": irwin}}
|
||||||
|
handle_wfigs(n, envelope, "central.fire.incident.removed.id",
|
||||||
|
data=None, now=now)
|
||||||
|
row = conn.execute("SELECT tombstoned_at FROM fires WHERE irwin_id=?",
|
||||||
|
(irwin,)).fetchone()
|
||||||
|
assert row["tombstoned_at"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_reminder_skipped_when_fire_tombstoned():
|
||||||
|
"""ReminderScheduler treats fires.tombstoned_at NOT NULL as terminated."""
|
||||||
|
from meshai.notifications.reminders import ReminderScheduler
|
||||||
|
conn = get_db()
|
||||||
|
now = 1_780_000_000
|
||||||
|
irwin = "REM-TOMB"
|
||||||
|
last = now - 10 * 3600
|
||||||
|
# Active fire 10h past last broadcast (would otherwise fire)
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||||
|
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||||
|
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
|
||||||
|
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(irwin, "T", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
|
||||||
|
last, now, last, last, now - 100), # tombstoned
|
||||||
|
)
|
||||||
|
dispatcher = MagicMock()
|
||||||
|
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
|
||||||
|
sch = ReminderScheduler(dispatcher, clock=lambda: now)
|
||||||
|
import asyncio
|
||||||
|
fired = asyncio.run(sch.tick_once())
|
||||||
|
assert fired == 0
|
||||||
|
dispatcher.dispatch_scheduled_broadcast.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
def test_reminder_fires_when_fire_not_tombstoned():
|
||||||
|
"""Same shape but tombstoned_at IS NULL -> reminder fires."""
|
||||||
|
from meshai.notifications.reminders import ReminderScheduler
|
||||||
|
conn = get_db()
|
||||||
|
now = 1_780_000_000
|
||||||
|
irwin = "REM-LIVE"
|
||||||
|
last = now - 10 * 3600
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO fires(irwin_id, incident_name, incident_type, "
|
||||||
|
"current_acres, current_contained_pct, lat, lon, county, state, "
|
||||||
|
"declared_at, last_event_at, first_broadcast_at, last_broadcast_at, "
|
||||||
|
"tombstoned_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(irwin, "L", "WF", 100, 10, 43.6, -116.2, "Ada", "ID",
|
||||||
|
last, now, last, last, None),
|
||||||
|
)
|
||||||
|
dispatcher = MagicMock()
|
||||||
|
dispatcher.dispatch_scheduled_broadcast = AsyncMock(return_value=True)
|
||||||
|
sch = ReminderScheduler(dispatcher, clock=lambda: now)
|
||||||
|
import asyncio
|
||||||
|
fired = asyncio.run(sch.tick_once())
|
||||||
|
assert fired == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Item 5 -- dead-code removal
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
def test_incident_broadcast_heartbeat_constant_gone():
|
||||||
|
"""The dead constant is not importable anymore."""
|
||||||
|
from meshai.central import incident_handler
|
||||||
|
assert not hasattr(incident_handler, "INCIDENT_BROADCAST_HEARTBEAT_S")
|
||||||
Loading…
Add table
Add a link
Reference in a new issue