mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-11 01:14:45 +02:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 6149900917 |
feat(incident): config-driven TomTom min_magnitude filter
Add tomtom_incidents.min_magnitude setting (default 4 = severe) to adapter_config registry. Replace the hardcoded magnitude==0 drop check with a config-driven floor that silently drops any TomTom event below the configured threshold. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
|||
| 87cce0048d |
fix(dispatcher): disable staleness filter for fire events
Fire events are always relevant regardless of age (a wildfire burning for 9 hours is not stale — it's ongoing). The staleness filter was designed for incidents with time_validity semantics, not persistent fire state. - defaults.py: add wfigs.freshness_seconds = 0 (disabled) - dispatcher.py: for fire toggle family, read from adapter_config instead of toggle; skip staleness check when freshness_s == 0 Fixes Blue Ridge fire being dropped after LAST_PER_SUBJECT replay. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
|||
| f69a05dd6d |
feat(v0.7-fire-tracker-4): fix LLM DM path + daily fire digest + ?status queries
Phase 4 of FIRMS+WFIGS fusion. Foundation: every direct LLM DM
mentioning a fire/weather/quake/avalanche/flood/etc. keyword was
failing silently in prod with UnboundLocalError because router.py
referenced scope_type before assigning it. With that path restored,
two new features land: a twice-daily fire-digest scheduled broadcast
(LLM-rendered) and a ?status <fire_name> on-demand mesh-DM intent.
BUG-FIX ROOT CAUSE (Job Zero):
router.py:745 ("if should_inject_mesh and scope_type == 'env'") read
`scope_type` -- a local variable bound only at line 761 inside an
unrelated `if self.source_manager and self.mesh_reporter` block.
Python's lexical scoping made scope_type a local of the whole
generate_llm_response function, so reading it before the assignment
raised UnboundLocalError on every env-keyword DM. The exception
propagated to main.py's outer except, no response went out, bot
appeared dead on fire/weather/quake/avalanche/flood queries.
Evidence (synthetic in-process trace against the live container's
config + GoogleBackend):
"are there any fires near me?" -> UnboundLocalError (pre-fix)
-> real LLM answer (post-fix)
"Yes, there are a few active
fires reported in the region.
Salmon River: 4,200 acres, 78%
contained. Cache Peak: 1,847
acres, 23% contained. ..."
"what's the weather?" -> UnboundLocalError (pre-fix)
-> "I do not have current weather
information. I can tell you
about active fires, stream gauge
levels, space weather, or band
conditions if you'd like." (post-fix)
"hi there" -> normal LLM answer in both cases
Fix: hoist `scope_type, scope_value = self._detect_mesh_scope(query)`
to right after `should_inject_mesh` is computed; remove the
now-duplicate detection inside the source_manager block.
Secondary mitigation: tightened the "do not invent commands" prompt
with an explicit "if no list appears above, you have NO commands"
clause. The prior prompt told the LLM "answer based on the command
list provided below" without always providing one, so the LLM
hallucinated plausible-sounding !commands (the "use ! commands"
canned-looking response Matt was seeing on non-env queries).
PHASE 4 FEATURES:
1. Fire-digest scheduler (meshai/notifications/scheduled/fire_digest.py).
Modeled after BandConditionsScheduler. Runs in the pipeline's
start_pipeline coroutine alongside band_conditions + reminders.
On each slot (default 06:00 + 18:00 America/Boise):
- Queries active fires (tombstoned_at IS NULL) + last 24h passes.
- Builds a prompt asking for a single mesh-wire summary <= 200
chars.
- Calls the LLM (Google/Anthropic/OpenAI per config).
- Falls back to a terse "Fires today (N): Cache Peak 1847 ac;
Twin Peaks 320 ac; +N more" line when the LLM is unavailable.
- Dispatches via dispatcher.dispatch_scheduled_broadcast (same
path band_conditions uses).
Idempotency: v16.sql adds fire_digest_broadcasts(slot_epoch PK,
sent_at, summary, source). INSERT OR IGNORE pattern blocks the same
slot firing twice (matters when container restarts mid-day).
2. ?status <fire_name> on-demand intent (router.py).
Before falling through to the LLM, route() now checks for a leading
"?status" / "status:" sigil or natural-language triggers like
"how is X fire?". On match:
- _lookup_fire_fuzzy walks fires by exact -> startswith ->
contains -> word-overlap (skipping a trailing " fire" word so
"cache peak fire" matches "Cache Peak"). Active fires rank
above tombstoned ones.
- _build_fire_status_context composes a small context block
(name, acres, containment, county/state, last 3 passes with
drift).
- The query is REWRITTEN into an LLM prompt with that context
inlined; the rest of the normal LLM path (chunking, history,
summary persistence) runs unchanged.
Live verification: "?status Cache Peak" -> "The Cache Peak fire is
1,847 acres and 23% contained. It's located in Probe / ID.";
"?status Salmon" -> word-overlap matches "Salmon River" ->
"The Salmon River fire is 4,200 acres and 78% contained, located
in Probe / ID."
3. adapter_config rows (GUI-editable per CONFIG-vs-CODE rule):
fires.digest_enabled = true (master toggle)
fires.digest_schedule = ["06:00", "18:00"]
fires.digest_timezone = "America/Boise"
fires.digest_max_chars = 200
Schema (v16.sql):
- fire_digest_broadcasts(slot_epoch INTEGER PK, sent_at, summary,
source) with source in {'llm', 'fallback_terse', 'skipped_no_fires'}.
- Index on sent_at for ops queries.
Tests (tests/test_fire_tracker_phase4.py, 10 cases all green):
- Regression guard: scope_type appears as an assignment BEFORE the
env_reporter check (prevents the UnboundLocalError from coming back).
- adapter_config seeds all 4 digest keys with expected defaults.
- render_digest returns ('', 'no_fires') when no active fires.
- render_digest falls back to terse line when LLM is None; wire fits cap.
- render_digest with a stub LLM returns ('<llm text>', 'llm').
- _lookup_fire_fuzzy: exact, "X fire" trim, word-overlap, no-match.
- _maybe_rewrite_status_query: builds context-bearing prompt; returns
None on non-status queries.
Combined suite: 60 passed in 3.81s across phase1+phase2+phase3+phase4
+or-arch+include-roundtrip.
Live verification on CT108 after rebuild:
- v16 migration applied (schema_meta=16, no Traceback in 3 min).
- FireDigestScheduler started: enabled=True schedule=['06:00','18:00']
tz=America/Boise.
- LLM DM probe (real Gemini) returns real answers on env queries
(Bug A fixed end-to-end).
- ?status Cache Peak + ?status Salmon return fire-specific summaries.
- render_digest with real LLM returns source=llm + non-empty wire.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 31e543ca04 |
feat(v0.7-fire-tracker-3): spotting detection -- pixels beyond perimeter trigger immediate broadcast
Phase 3 of FIRMS+WFIGS fusion. v15.sql adds perimeter_geojson to fire_passes + last_spotting_broadcast_at to fires. FIRMS handler computes convex hull of each pass on pass-boundary close; attributed pixels >= 1.5 mi (configurable) from previous-pass perimeter emit wildfire_spotting broadcast. Cooldown 1h between spotting broadcasts per fire so rapid embers do not spam. wildfire_spotting category at immediate severity -- spotting is the highest-actionable fire signal (spread beyond perimeter). All thresholds GUI-editable. Phase 4 (LLM summaries + on-demand queries) deferred. Schema (v15.sql): - fire_passes gains perimeter_geojson TEXT (nullable; populated by _close_prev_perimeter at boundary). GeoJSON Polygon, single outer ring in (lon, lat) order per RFC 7946, closed (first == last). - fires gains last_spotting_broadcast_at REAL (per-fire cooldown latch). Index (irwin_id, last_spotting_broadcast_at) for the cooldown probe. adapter_config (defaults.py REGISTRY): - fires.spotting_distance_threshold_mi = 1.5 (float). Matches design doc Phase 3 spec; design doc open question #6 lists this as TBD pending real spotting observation data. - fires.spotting_cooldown_seconds = 3600 (int, 1h). Suppresses rapid-ember spam from a single satellite pass. ALERT_CATEGORIES (notifications/categories.py): - wildfire_spotting: immediate / fire. Highest fire severity -- spotting represents fire spread BEYOND the existing perimeter, the most actionable detection signal. FIRMS handler (central/firms_handler.py): - _handle_pass_boundary now closes the prior pass's perimeter (convex hull of fire_pixels via Andrew's monotone chain) on the first boundary; subsequent in-pass pixels reuse the stored hull. - _check_spotting runs for every attributed pixel: looks up the most recent CLOSED pass (perimeter_geojson NOT NULL AND pass_id != current), point-in-polygon test, vertex-distance approximation per design doc Q (sparse pixels make edge projection overkill at VIIRS 375 m resolution), per-fire cooldown gate. - Priority order: spotting (immediate) > growth (priority) > cluster (priority) > halt (routine). Spotting preempts growth at the same pixel because immediate > priority. - Helpers: _convex_hull (Andrew's monotone chain), _hull_to_geojson (RFC 7946 Polygon), _point_in_polygon (ray casting), _close_prev_perimeter, _check_spotting, _prev_has_perimeter. Wire string: - wildfire_spotting: "🔥 Possible spotting <dist:.1f> mi <dir> of <incident_name> perimeter" -- direction is 8-way bearing from the previous pass's centroid to the spotting pixel. Tests (tests/test_fire_tracker_phase3.py, 11 cases all green): - Pass close stamps perimeter_geojson as a closed Polygon (6 hex vertices -> 7-entry closed ring). - Pixel 2 mi NE of perimeter fires spotting with distance in the 1.0..2.5 mi band (vertex-distance approximation) and direction NE. - Pixel inside perimeter -> NO spotting wire. - Second spotting candidate within 1h cooldown -> suppressed. - Past-cooldown spotting fires again. - Convex hull / point-in-polygon / GeoJSON round-trip helper tests. - adapter_config seed for both new fires.* keys. - wildfire_spotting category registered with immediate severity. - 49 tests green across phase1/phase2/phase3/or-arch/include-roundtrip. Live verification on CT108 after rebuild: - v15 migration applied (schema_meta=15, no Traceback in 3 min). - Container healthy. Synthetic 25-pixel probe (PROBE-V07P3-*, cleaned up after): - Pass A: 20 pixels in a ~0.3 mi circle. Perimeter stored on boundary. - Pass B: 5 pixels at distances 0.5/1.0/2.0/5.0/7.0 mi from center. Observed wires: "🔥 Possible spotting 1.7 mi NE of Probe Spotting Fire perimeter" "🔥 Possible spotting 4.7 mi NW of Probe Spotting Fire perimeter" (Plus a Phase 2 growth wire on the first pass B pixel -- documented side effect: single-pixel pass B centroid shows 0.5 mi drift from pass A.) - 7.0 mi E pixel: outside 5 mi spread, no broadcast (cluster check found no co-located unattributed pixels). Cleanup confirmed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| f5c566c6c0 |
feat(v0.7-fire-tracker-2): movement analysis -- growth + halt detection
Phase 2 of FIRMS+WFIGS fusion. v14.sql adds fire_passes table for per-satellite-pass centroid tracking + drift computation. FIRMS handler now detects pass boundaries (satellite + time bucket), computes pass centroid (median of pass pixels), Haversine drift from previous pass, bearing to 8-way direction, mi/h speed. Drift >= 0.5 mi (configurable) emits wildfire_growth broadcast with wire including movement vector and nearest-town context. Halt detection: fire with no new pixels for >=12h (configurable) emits wildfire_halted broadcast (routine). Two new ALERT_CATEGORIES: wildfire_growth (priority), wildfire_halted (routine). All thresholds GUI-editable via adapter_config.fires.*. Phase 3 (spotting) and Phase 4 (LLM summaries) deferred to subsequent commits. Schema (v14.sql): - fire_passes table (irwin_id FK CASCADE, pass_id, pass_centroid_lat/lon, pixel_count, total_frp, pass_started_at, pass_ended_at, drift_mi_from_prev, drift_direction, drift_mi_per_hour). PRIMARY KEY (irwin_id, pass_id) so the UPSERT path is cheap; secondary index on (irwin_id, pass_ended_at) for the prev-pass lookup + halt counter. - fires gains last_pass_id, last_pass_at, halt_broadcast_at columns. halt_broadcast_at is latched per halt event; the detector filter (halt_broadcast_at IS NULL OR halt_broadcast_at < last_pass_at) reopens eligibility automatically when an idle fire receives a new attributed pixel that advances last_pass_at. adapter_config (defaults.py REGISTRY): - fires.growth_drift_threshold_mi = 0.5 (float). Per-pass centroid drift at or above this fires wildfire_growth. 0.5 mi matches the design doc Phase 2 spec and is roughly 2x the VIIRS 375m pixel size (i.e., detectable as more than centroid jitter). - fires.halt_passes_threshold = 2 (int). Documented intent; the operational rule uses halt_minimum_seconds below as the time gate because per-satellite pass-count enforcement would require modeling the global VIIRS schedule per satellite. The 12h gate subsumes it (4 passes/day in Idaho). - fires.halt_minimum_seconds = 43200 (int, 12h). ALERT_CATEGORIES (notifications/categories.py): - wildfire_growth: priority/fire. FIRMS handler tags data["category"] + data["severity"] on the pass-boundary path when drift >= threshold. - wildfire_halted: routine/fire. Halt detector tags data["category"] + data["severity"] when a fire transitions to idle for >=12h. FIRMS handler (central/firms_handler.py): - The Phase 1 attribution branch now passes through _handle_pass_boundary(): UPSERT fire_passes row for the current (irwin_id, pass_id) with median centroid + pixel count + total FRP + min/max acq_time; lookup the prior pass; compute drift mi + 8-way direction + mi/h speed and write them into the current pass row (only the FIRST boundary fills these; subsequent in-pass pixels COALESCE keep them stable). Update fires cursor (last_pass_id, last_pass_at) and current_centroid_lat/lon to the latest pass centroid -- this overrides Phase 1's 24h all-pixels median for fires that have pass data. - Growth wire emitted ONLY at the boundary (last_pass_id != current, prev pass exists, drift >= threshold). Subsequent in-pass pixels stay silent because pass_id == last_pass_id. - _maybe_emit_halt runs as a final fallback when neither growth nor cluster has fired. SELECT one fire matching the halt criteria, stamp halt_broadcast_at, return the wire. The fallback ordering is growth > cluster > halt so a busy fire's growth broadcast doesn't starve a quiet fire's halt. - New helpers: _bearing() (great-circle initial bearing, deg CW from N), _direction_8() (compass 8-way mapping with +/-22.5 deg sectors). Wire strings: - wildfire_growth: `🔥 <incident_name> moving <dir> <speed:.1f> mi/h ~<dist_to_nearest_town:.1f> mi from <nearest_town>`. nearest_town via meshai.central_normalizer.nearest_town (same Photon-backed cache that wfigs_handler uses); failure falls back to bare "moving <dir> <speed> mi/h". - wildfire_halted: `🔥 <incident_name> no growth in <hours>h`. Tests (tests/test_fire_tracker_phase2.py, 10 cases all green): - 2-pass attribution with pass2 1.0 mi N of pass1 -> drift=1.0, direction='N', mi/h computed, growth wire returned, data tagged. - Drift below threshold (0.3 mi) -> NO growth broadcast; pass row still records the (sub-threshold) drift for ops visibility. - Halt detector: last_pass_at 14h ago -> fires once, halt_broadcast_at stamped. - Re-run halt detector with halt latched -> NO second broadcast. - Halt re-eligibility: halt_broadcast_at < last_pass_at -> eligible again (a resurrected then re-idled fire). - Bearing + direction round-trip across all 8 cardinals. - Direction sector boundary (22.5/67.5 deg) correctness. - adapter_config seed for 3 new fires.* keys. - Two new ALERT_CATEGORIES registered. - 5-pixel single-pass aggregate (pixel_count, total_frp sum, median centroid, started/ended_at min/max). Phase 1 test fix: - tests/test_fire_tracker_phase1.py::test_centroid_recomputes_as_median_across_passes retimed to 12:00/12:10/12:20 so all 3 pixels land in one N20 bucket. Phase 2 makes current_centroid_* the per-pass median (latest pass overrides Phase 1's 24h median); the same-pass shape preserves the original median-computation intent. 39 total tests green across phase1/phase2/or-arch/include-roundtrip. Live verification on CT108 after rebuild: - v14 migration applied (schema_meta version=14, no Traceback in 3 min). - adapter_config.fires.growth_drift_threshold_mi = 0.5 - adapter_config.fires.halt_passes_threshold = 2 - adapter_config.fires.halt_minimum_seconds = 43200 - Container healthy. Synthetic 100-pixel probe inside prod container (PROBE-V07P2-*, cleaned up after): - Pass A (50 pixels @ 12:00-12:25, N20 bucket 329768): centroid (44.30000, -115.50000), pixel_count=50, total_frp=975.0, drift=NULL (first pass). - Pass B (50 pixels @ 18:00-18:25, N20 bucket 329772, centered 1.2 mi NE of A): centroid (44.31230, -115.48282), pixel_count=50, total_frp=975.0, drift_mi_from_prev=1.1703 (~design target 1.2 mi with -0.03 mi rounding), drift_direction="NE", drift_mi_per_hour=0.209 (1.17 mi over 5.5h between pass ends). - Growth wire: "🔥 Probe Movement Fire moving NE 0.2 mi/h, ~13.0 mi from Long Creek Summit Home" (Photon nearest-town anchor populated successfully). - Exactly ONE growth broadcast (first pixel of pass B); 99 other pixels stayed silent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| dd8e687aca |
feat(v0.7-fire-tracker-1): registry correlation + 2 new categories
Phase 1 of the FIRMS+WFIGS fusion design doc. v13.sql adds fire_pixels table for per-fire pixel history + spread_radius_mi/current_centroid_*/ last_hotspot_at on fires. FIRMS handler now attributes incoming pixels to fires via point-in-circle within configurable radius (default 5 mi), updating per-fire centroid as median of recent pixels. Unattributed pixels go through a cluster detector: 3+ pixels within 1 mi within 60 min triggers a single unattributed_hotspot_cluster broadcast (Possible new fire). Two new ALERT_CATEGORIES: wildfire_declared (priority, WFIGS first-sight) and unattributed_hotspot_cluster (priority, FIRMS cluster). All thresholds GUI-editable via adapter_config.fires.* and adapter_config.firms.*. Phases 2-4 (movement analysis, spotting, LLM summaries) deferred to subsequent commits. Schema (v13.sql): - fire_pixels table (irwin_id FK CASCADE, acq_time, lat/lon, frp, satellite, pass_id, attributed_at). Indexed on (irwin_id, acq_time) for centroid queries + on (acq_time) for Phase 2. - fires gains spread_radius_mi (nullable; NULL => use global default), current_centroid_lat/lon (median of last 24h pixels, distinct from the WFIGS-declared anchor lat/lon), last_hotspot_at (Phase 2 halt detector). - firms_pixels gains attributed_at + cluster_broadcast_at + compound index on (attributed_at, cluster_broadcast_at, acq_time) for the cluster query. adapter_config (defaults.py REGISTRY + ADAPTER_META): - fires.spread_radius_mi_default = 5.0 (float) - firms.cluster_min_pixels = 3 (int) - firms.cluster_max_radius_mi = 1.0 (float) - firms.cluster_time_window_minutes = 60 (int) - ADAPTER_META["fires"] meta block (display_name + description). ALERT_CATEGORIES (notifications/categories.py): - wildfire_declared: priority/fire. WFIGS handler tags data["category"] on cases (i)+(ii) [INSERT or row-exists-but-never-broadcast]; case (iii) Update keeps the existing wildfire_incident category. - unattributed_hotspot_cluster: priority/fire. FIRMS handler tags data["category"] + data["severity"] when emitting the cluster wire. FIRMS handler (central/firms_handler.py): - Unchanged storage path: filter, INSERT OR IGNORE into firms_pixels. - New _attribute_or_cluster() runs on every newly-stored pixel (dedup hits skip -- the original insert had its shot already). - Attribution: bbox prefilter on fires.tombstoned_at IS NULL, then exact Haversine to fires(current_centroid_lat ?? lat, current_centroid_lon ?? lon) inside spread_radius_mi (per-fire ?? global default). Multi-match resolves to nearest (design doc Q2). On match: INSERT fire_pixels, UPDATE firms_pixels.attributed_at, recompute centroid as median of last 24h pixels for this fire. - Cluster: on attribution miss, query firms_pixels WHERE attributed_at IS NULL AND cluster_broadcast_at IS NULL AND acq_time > NOW-window. If count >= cluster_min_pixels, fire the cluster wire and stamp cluster_broadcast_at on every member so a 4th arrival cannot re-fire. WFIGS handler (central/wfigs_handler.py): the existing prefix=New branches (i)+(ii) now set data["category"]="wildfire_declared". Existing _render() unchanged. Wire strings: - wildfire_declared: re-uses _render(prefix="New") -- emoji + name + type + anchor + acres + containment + coords. - unattributed_hotspot_cluster: _render_cluster_wire() emits "Possible new fire: <N> hotspots within <r> mi @ <lat>,<lon> (combined <total_frp> MW)". Tests (tests/test_fire_tracker_phase1.py, 10 cases all green): - Pixel within radius -> attribution + centroid + last_hotspot_at. - Centroid recomputes as median across multiple passes. - Pixel outside radius -> NO attribution + stays unattributed. - 3 unattributed within 1 mi within 60 min -> cluster broadcast fires exactly once, all 3 stamped cluster_broadcast_at. - 4th pixel in the same footprint -> NO second broadcast (existing 3 are stamped so SQL filter excludes them). - 5th-7th pixels 2h later -> form a NEW cluster (window prune fires). - WFIGS first-sight tags data["category"]="wildfire_declared". - WFIGS Update branch does NOT retag wildfire_declared. - New adapter_config rows seeded on init_db. - ALERT_CATEGORIES contains both new entries with correct toggle/severity. Live verification on CT108 after rebuild: - v13 migration applied (schema_meta version=13, no Traceback). - adapter_config.fires.spread_radius_mi_default = 5.0 - adapter_config.firms.cluster_min_pixels = 3 - adapter_config.firms.cluster_max_radius_mi = 1.0 - adapter_config.firms.cluster_time_window_minutes = 60 - fires gains 4 new columns; firms_pixels gains 2 new columns; fire_pixels table created. - Container healthy, FIRMS pixels continue arriving (126 pre-deploy). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 566b06de06 |
feat(v0.6-tail): close 5 v0.6-phase1-complete.md follow-ups
(1) Auto-call refresh-toggles on PUT /api/config/notifications
meshai/dashboard/api/config_routes.py adds register_config_routes_hooks(app)
which registers a FastAPI HTTP middleware: on any 2xx PUT whose path
matches /api/config/notifications or /api/config, the middleware
invokes _refresh_toggle_filter(app) which reaches into app.state.bus._
pipeline_components["toggle_filter"] and calls .refresh(app.state.config).
The dashboard no longer has to remember to ping POST /api/notifications/
refresh-toggles after a toggle change. The explicit endpoint stays for
backwards-compat.
(2) env_reporter block-size cap moved to adapter_config
New registry row pipeline.env_reporter_block_chars (int, default 3000).
meshai/notifications/env_reporter.py replaces the hardcoded
_BLOCK_MAX_CHARS = 3000 with _DEFAULT_BLOCK_MAX_CHARS (the fallback) +
a _block_cap() helper that reads from adapter_config on every slice.
Mutating the row via PUT /api/adapter-config takes effect on the next
env_reporter call -- no restart.
(3) Bulk-import endpoint for gauge_sites
meshai/dashboard/api/gauge_sites_import.py adds
POST /api/gauge-sites/import with two paths:
format=csv -- expects "data" (CSV text with header row matching
gauge_sites columns: site_id, gauge_name, lat, lon,
and optionally action_ft/flood_minor_ft/
flood_moderate_ft/flood_major_ft/enabled). UPSERT
via ON CONFLICT(site_id) DO UPDATE. Returns
{inserted, updated, skipped}.
format=nws-ahps -- expects "wfo" (list of WFO codes). Fetches
water.weather.gov/ahps2/index.php?wfo=<WFO> for each,
regex-parses gauge links, then fetches up to 50
gauge detail pages per request and regex-parses
lat/lon + four threshold values. Best-effort; rows
stored under "AHPS-<gauge_id>" so they dont collide
with USGS-* ids. Returns the same shape plus
detail_fetched + errors list.
Frontend (dashboard-frontend/src/pages/GaugeSites.tsx) gains a
Import button + modal with two tabs (Paste CSV / Scrape NWS-AHPS)
rendered via an ImportModal component. CSV tab has a 48-row textarea
with the column-header hint inline; AHPS tab has a comma-separated WFO
input defaulting to BOI. Both submit via fetch() and show the JSON
response inline. Invalidates the curation cache server-side on any
successful insert/update so nwis_handler sees the new gauges on its
next call.
(4) WFIGS tombstone column -- CORRECTNESS
v12.sql adds fires.tombstoned_at REAL (nullable) + idx_fires_tombstoned_at.
meshai/central/wfigs_handler.py: the tombstone branch
(kind=="wfigs_tombstone") UPDATE fires SET tombstoned_at=COALESCE(
tombstoned_at, ?) so the first tombstone-time wins (idempotent against
repeated tombstone envelopes).
meshai/notifications/reminders/__init__.py: the wfigs tombstone
termination condition now checks row["tombstoned_at"] IS NOT NULL.
Reminders correctly STOP for closed fires -- before this change the
8h cadence would have kept Active: broadcasts going indefinitely past
a WFIGS removal.
SCHEMA_VERSION 11 -> 12.
(5) Delete INCIDENT_BROADCAST_HEARTBEAT_S
meshai/central/incident_handler.py: removed the dead constant
(v0.5.9 REVISED dropped the heartbeat path but left the constant
imported-but-never-read).
tests/test_incident_handler.py: removed the orphan
test_i_8h_heartbeat_triggers_update test (asserted None, used the
deleted constant for time arithmetic) and the stray import line.
Tests (tests/test_tail_followups.py, 16 cases):
- middleware fires refresh on PUT /api/config/notifications (200), does
NOT fire on PUT /api/config/llm
- env_reporter _block_cap() default 3000; mutate via PUT, invalidate,
next read returns the new cap
- CSV import inserts new rows, updates existing, skips bad rows,
rejects missing required columns, rejects bad format
- AHPS index parser extracts (gauge_id, name) from realistic HTML
- AHPS detail parser extracts lat/lon + four thresholds from realistic
HTML
- fires has tombstoned_at column after migrations
- wfigs tombstone branch stamps tombstoned_at
- ReminderScheduler skips a fire whose tombstoned_at is NOT NULL
- ReminderScheduler still fires for a fire whose tombstoned_at IS NULL
- INCIDENT_BROADCAST_HEARTBEAT_S no longer importable
Foundation/API test counts bumped:
REGISTRY 58 -> 59 (+ env_reporter_block_chars)
schema_meta v11 -> v12
Test count: 844 -> 859 (+16 new, -1 deleted dead test). 0 regressions.
|
|||
| 3a410d5087 |
feat(v0.6-phase3): reminder system + schema split + NWS dedup relaxation
Third broadcast type Active: clock-driven re-broadcasts of still-live
events at human-scale cadences. WFIGS fires 8h, itd_511 work zones daily
8 AM Mountain, SWPC G-storms 8h. NWS is NOT a clock reminder -- instead
the per-CAP-id dedup is relaxed to allow re-broadcast if >3h since last.
Schema split first_broadcast_at + last_broadcast_at on all reminder-
eligible tables. Wire prefix logic: New (first sight), Update (WFIGS
material change), Active (clock reminder). All cadences, channels, day-
of-week patterns, timezones, and termination conditions GUI-editable
from day one via the existing adapter_config editor. Termination:
tombstone OR containment_100 OR end_date_passed (no max-count). Quiet
hours not respected -- ripped out in Phase 2.
Schema (v11.sql):
- ALTER TABLE fires|nws_alerts|traffic_events|quake_events|swpc_events|
gauge_readings ADD COLUMN first_broadcast_at REAL
- Backfill: UPDATE ... SET first_broadcast_at = last_broadcast_at
WHERE last_broadcast_at IS NOT NULL
- ALTER TABLE adapter_meta ADD COLUMN reminder_enabled INTEGER NOT NULL
DEFAULT 0
- UPDATE adapter_meta SET reminder_enabled=1 WHERE adapter IN
('wfigs', 'swpc') -- itd_511_work_zone is a new meta row seeded
with reminder_enabled=1
- SCHEMA_VERSION 10 -> 11
Handler commit-callbacks (wfigs/nws/quake/swpc/incident):
- UPDATE ... SET last_broadcast_at=?, first_broadcast_at=COALESCE(
first_broadcast_at, ?) -- first_broadcast_at stamped once, never
overwritten
NWS handler (meshai/central/nws_handler.py):
- _render() gains a prefix kwarg
- After-first-broadcast branch: when (now - last_broadcast_at) >=
adapter_config.nws.duplicate_allowed_after_seconds (default 10800
= 3h), allow the re-broadcast with prefix=Active. Under the
window, suppress as before. The commit callback continues to
update last_broadcast_at.
ReminderScheduler (meshai/notifications/reminders/__init__.py):
- Async loop, ticks every 60s
- Each tick: SELECT adapter FROM adapter_meta WHERE reminder_enabled=1
- Per adapter, load reminders_<adapter> config from adapter_config
(cadence_kind, cadence_value, channels, terminate_when, dow_mask,
timezone)
- Interval cadence: rows where last_broadcast_at <= now - cadence_value
- Clock cadence: localizes now to configured tz, finds slots that
just passed in the last tick window, gated by dow_mask
- Termination conditions checked per adapter:
wfigs.containment_100 -> current_contained_pct >= 100
wfigs.last_event_age_24h -> last_event_at older than 24h
swpc.end_date_passed -> payload_json end_time in past
itd_511_work_zone.end_date_passed -> traffic_events.end_at in past
- Active: prefix on every emitted wire; dispatcher.dispatch_scheduled_
broadcast() honors cold-start grace, bypasses toggle path
- On success, last_broadcast_at = now; first_broadcast_at preserved
Launched from notifications/pipeline/__init__.py:start_pipeline()
alongside BandConditionsScheduler.
adapter_config registry (+15 new keys, 43 -> 58):
- reminders_wfigs.cadence_kind/cadence_value/channels/terminate_when
- reminders_swpc.cadence_kind/cadence_value/channels/terminate_when
- reminders_itd_511_work_zone.cadence_kind/cadence_value/channels/
dow_mask/timezone/terminate_when
- nws.duplicate_allowed_after_seconds
adapter_meta (+4 rows, 15 -> 19):
- reminders_wfigs, reminders_swpc, reminders_itd_511_work_zone
(pseudo-adapters carrying the reminder config)
- itd_511_work_zone (reminder target row; reminder_enabled=1)
- reminder_enabled flag added to wfigs/swpc (existing rows updated by
v11.sql) and to itd_511_work_zone seed.
Tests (tests/test_reminders.py, 10 cases):
- wfigs reminder fires past 8h cadence, stamps last_broadcast_at,
preserves first_broadcast_at
- reminder skipped within cadence
- reminder skipped when containment_100, last_event_age_24h
- swpc reminder fires (interval)
- work_zone clock reminder fires at 08:00 Mountain on enabled DOW
- work_zone reminder skipped when end_date_passed
- work_zone reminder skipped outside slot window
- reminder_enabled=0 suppresses all reminders for that adapter
tests/test_nws_dedup_relaxation.py (5 cases):
- First sighting renders without Active: prefix
- Re-broadcast within 3h suppressed
- Re-broadcast after 3h allowed with Active: prefix
- adapter_config.nws.duplicate_allowed_after_seconds override takes
effect (1h window verified)
- First sighting stamps first_broadcast_at=committed_at,
last_broadcast_at=committed_at; 4h later broadcast stamps
last_broadcast_at only, first_broadcast_at preserved
Test count: 829 -> 844 (+15 new, 0 regressions). Foundation tests
updated for new counts (REGISTRY=58, ADAPTER_META=19, schema=v11).
|
|||
| 68dcbc74d0 |
feat(v0.6-3a.1): trim adapter_config registry to CONFIG-only per Matt config-vs-code rule + log-on-delete safety net for orphan cleanup
Drops 35 of the v0.6-3a-draft 77 keys + adds 1 net-new key
(firms.dedup_distance_m) for a final count of 43. The trim rules:
CONFIG (lives in adapter_config, surfaces in the GUI):
where we send (channels), how often (cadences/schedules),
thresholds (magnitude floors, severity gates, distance radius,
cooldown durations, freshness windows), curation data (which
sites/states/codes), toggles (enabled, include_in_llm_context,
drop_zero_magnitude).
CODE (stays in handlers, never reaches the GUI):
sentence templates, emoji choices, mapping/translation functions
(TomTom icon_map, ITD sub_type_map, Central adapter_map and
category_map), rendering logic (anchor priority order,
expires-bucket formatting, threshold-state labels), heuristic
logic (band_conditions Kp/SFI -> Good/Fair/Poor function).
Per-adapter outcome (kept | killed):
wfigs 4 | 4 (cooldown_seconds, anchor_max_mi, two re-broadcast toggles)
nws 3 | 4 (broadcast_severities, tombstone_msgtypes, warning_suffix_promotes)
usgs_quake 6 | 3 (centroid, radius, PAGER list, 3 mag floors)
swpc 3 | 7 (three storm-tier floors)
usgs_nwis 2 | 4 (parameter_codes, broadcast_on_recede)
incident 2 | 0 (freshness_seconds, broadcast_on_update)
tomtom_incidents 2 | 1 (drop_zero_magnitude, drop_non_present)
state_511_atis 1 | 0 (skipped_states)
itd_511 0 | 3 (all sub_type maps/emoji/phrase = CODE)
central 1 | 2 (severity_thresholds)
dispatcher 4 | 0 (LRU cap, prune params, retention days)
band_conditions 3 | 6 (SWPC freshness + HamQSL endpoint config)
geocoder 6 | 1 (Photon endpoint + town-OSM curation + cache cap)
firms 4 | 1* (confidence_floor, frp_floor, bbox, dedup_distance_m)
pipeline 2 | 0 (inhibitor TTL, grouper window)
* firms: dedup_lat_lon_decimals is replaced by dedup_distance_m=5 per
Matt s call (user-facing unit is meters, not decimal places; the
handler will internally translate to quantization step in v0.6-3b).
adapter_meta stays at 15 rows -- itd_511 keeps its include_in_llm_context
toggle even with zero config keys.
Live-DB cleanup:
meshai/adapter_config/__init__.py:prune_orphans(conn) DELETEs every
adapter_config row whose (adapter, key) is no longer in REGISTRY. Each
delete is INFO-logged with the prefix "adapter_config orphan removed:"
so docker logs carry the paper trail. Called from init_db() after
seed_defaults; idempotent (zero deletes on every subsequent boot).
Cache is invalidated when any orphan is removed.
adapter_meta is NOT pruned -- meta rows are cheap and useful even for
adapters that ended up with zero config keys.
Tests (34 cases, replaces v0.6-3a 24-case set):
- Registry count is 43; ADAPTER_META is 15
- Seed lands every REGISTRY + ADAPTER_META row; idempotent; never
overwrites user edits
- prune_orphans removes a synthetic legacy row, logs at INFO with the
exact prefix, leaves known keys untouched, leaves adapter_meta
untouched, invalidates the accessor cache
- Accessor returns correctly-typed values incl new
firms.dedup_distance_m
- Guard tests: no key in REGISTRY contains "emoji", ends with "_map",
or contains "template" / "prefix" (catches CODE leaking back in)
Test count: 721 -> 731 (+10 net: +5 prune cases, +1 firms.dedup_distance_m,
+3 CODE-guard cases, +1 registry-count assertion).
Refs Matt s locked CONFIG-vs-CODE rule.
|
|||
| cb3c5aec7e |
feat(v0.6-3a): adapter_config foundation -- migration + defaults registry + typed accessor
Closes the foundation slice of audit doc Section A (Rule 17). Lands two
SQLite tables, the seed routine that populates them from a Python
defaults registry, and a typed accessor that handler code will read in
v0.6-3b. No handler changes in this commit -- ZERO behavior risk, every
existing test still passes (721 / 69 skipped / 0 failed).
v6.sql tables:
- adapter_config(adapter, key, value_json, default_json, type, description,
updated_at) PRIMARY KEY(adapter, key) -- JSON-encoded
values flow through a single column uniformly. CHECK
constraint on `type` closes the vocab (int/float/str/
bool/json).
- adapter_meta(adapter PK, display_name, include_in_llm_context,
description, updated_at) -- per-adapter metadata + the
user-scopable LLM-context toggle (Matt refinement #5).
meshai/adapter_config/ package:
- defaults.py: REGISTRY dict mapping (adapter, key) -> {default, type,
description}. Covers audit doc sections A.1-A.12: wfigs, nws,
usgs_quake, swpc, usgs_nwis, incident family (tomtom_incidents,
state_511_atis, itd_511, shared "incident"), central consumer,
dispatcher, band_conditions, geocoder, firms, pipeline (Inhibitor +
Grouper). ~85 keys total. ADAPTER_META covers 15 adapters with
display_name + include_in_llm_context defaulting to True. Per Matt
refinement #3, every default matches the current handler constant
EXACTLY -- first deploy behavior is unchanged.
- _accessor.py: AdapterConfig class with `adapter_config.<adapter>.<key>`
syntax. Read pipeline: in-memory cache hit -> SQL -> registry
fallback (with WARNING) -> AttributeError. Process-wide cache; PUT
via v0.6-3c REST API calls invalidate_cache() to drop the cache.
GIL-atomic dict reads on the fast path (handlers call this hot).
- __init__.py: seed_defaults(conn) -- INSERT OR IGNOREs one row per
registry entry. Idempotent, never overwrites user edits.
Wiring:
- meshai/persistence/db.py: SCHEMA_VERSION 5 -> 6, and init_db() now
calls seed_defaults() after migrations apply.
- meshai/main.py: _init_components() now calls init_db() FIRST (per
commit #1 lessons-learned: a startup-time migration is required
when handlers will rely on the new schema; lazy-on-first-handler
is fine for v4/v5 but not for v6 where handler reads start in
v0.6-3b).
- tests/conftest.py: autouse fixture now calls init_db() + clears
the accessor cache around each test, so every test gets the v6
seed AND a clean cache without per-test boilerplate.
Tests (tests/test_adapter_config_foundation.py, 24 cases):
- v6 tables exist + schema_meta at 6 + type-vocabulary CHECK enforced
- seed populates every REGISTRY + ADAPTER_META row, value_json ==
default_json on first seed, type matches
- seed is idempotent + does not overwrite user edits
- accessor returns correctly typed values for int/float/str/bool/
json list/json dict/json None
- cache hit: second read does not touch the DB (patched _load_from_db
raises, accessor still succeeds)
- invalidate_cache forces a re-read; mutated DB value wins
- registry fallback path triggers when a row is missing (with WARNING)
- unknown key raises AttributeError
- setattr blocked (writes go via the REST API in 3c)
- every default JSON round-trips cleanly; every type is in vocabulary
- ADAPTER_META covers every adapter in REGISTRY
Test count: 697 -> 721 (+24 new, 0 regressions).
v0.6-3b will wire handlers one at a time (wfigs, nws, quake, swpc, nwis,
incident, central, dispatcher, band_conditions, geocoder, firms). Per
the audit lock, defaults match exactly so each wiring step is a pure
refactor -- bisect-safe.
v0.6-3c lands the /api/adapter-config CRUD + the AdapterConfig.tsx
dashboard editor + cache invalidation on PUT.
Refs audit doc v0.6-phase1-audit.md Section A + finding #4.
|
