meshai followup spec on top of v0.10.10. Three additive changes:
1. Tombstone emission. When a zone that previously passed the severity
gate stops passing (drops below danger_level 3, flips off_season=true,
or disappears from the upstream feed entirely), the adapter yields a
retraction Event so subscribers can clear it from their displays.
Mirrors the wfigs_incidents fall-off pattern:
- per-adapter sqlite table 'avalanche_org_observed' keyed by
(center_id, zone_name); tracks last_published_at + state.
- poll() diff: previously_published - currently_published = removed set.
- tombstone Event per removed zone: category
'avy.advisory.removed.<center_id_lower>', severity=0 (matches wfigs
tombstone convention), empty Geo, id '<cid>_<slug>:removed:<iso>'
(unique per emission so repeat retraction cycles aren't deduped).
- subject_for routes 'avy.advisory.removed.*' to
'central.avy.advisory.removed.us.<state>'.
- reason field on the tombstone is one of:
'off_season' -- upstream still has the zone, off_season=true
'below_threshold' -- upstream still has the zone, danger_level<3
'fallen_off_feed' -- zone absent from upstream entirely (center
reorganised etc.; meshai's renderer is
expected to treat this the same as
below_threshold for retraction rendering)
- State updates (upsert + delete) happen AFTER yields, matching the
wfigs at-least-once convention: a supervisor crash mid-publish
re-emits on the next poll rather than silently swallowing.
2. geo.bbox. Added shapely_shape(geometry).bounds as (W, S, E, N) to the
Geo construction in _build_event_record. Defensive try/except so a
malformed geometry doesn't crash; falls back to bbox=None.
3. Slug format change: underscores -> hyphens
('banner_summit' -> 'banner-summit'). One-line regex change. Safe to
ship because off-season published_ids count for avalanche_org was 0
at the time of this PR -- no live event.id values to invalidate. event.id
shape stays '<center_id>_<slug>' (underscore between center and slug
remains; only the slug itself changes).
Tests (13 new in tests/test_avalanche_org.py, 51 total):
- Slug parametrize: all 8 cases flipped to hyphens.
- Live-publish test: asserts bbox present and ev.id uses hyphenated slug.
- _removal_reason classification: 5 parametrized cases covering all three
reasons + None input.
- State-transition tests covering every cell of the matrix:
- Considerable -> Low: tombstone(below_threshold)
- Considerable -> off_season: tombstone(off_season)
- Considerable -> absent: tombstone(fallen_off_feed)
- Considerable -> Considerable: no tombstone (live publish only)
- Low -> off_season: no tombstone (never published, nothing to retract)
- Load-bearing test_no_duplicate_tombstone_across_consecutive_polls:
Considerable -> Low (emit tombstone + explicit DB query confirming
observed row deleted) -> still Low (no second tombstone) -> recovers
to Considerable (normal live publish). Guards against the most likely
bug class meshai is exposed to if the diff logic mishandles state.
- subject_for routing: tombstone -> 'central.avy.advisory.removed.us.id';
live publish still -> 'central.avy.advisory.us.id'.
- Tombstone-id uniqueness: each emission gets a fresh :removed:<iso>
suffix so JetStream's per-stream dedup doesn't swallow repeats.
Wiring NOT changed:
- streams.py / supervisor.py STREAM_CATEGORY_DOMAINS: tombstones still
route to CENTRAL_AVY (category prefix 'avy.advisory.removed.*' starts
with 'avy', covered by the existing ('avy',) family domain).
- gui partials, doc updates: no new adapter, no new domain -- no
consistency-test updates needed.
Diff size: +365 / -13 = +352 net. Slightly over the 300-line target;
220 of those lines are the 13 new tombstone tests (state-transition
correctness coverage). Adapter logic itself is +145 -- which is what
the new feature requires (sqlite table + 3 helpers + reason classifier
+ diff phase in poll). No defensive scaffolding beyond what wfigs
already establishes.
Full sweep: 1085 passed (+13 from this PR), ruff clean on both files.
Deploy plan: code-only change, no migration, no new stream/adapter
config rows. Squash-merge -> tag v0.10.11 -> pull on central -> restart
central-supervisor. NO archive restart (extending an existing stream,
not adding a new one). NO published_ids flush. Post-deploy verify the
poll still completes (events_yielded=0, events_omitted=6,
tombstones_emitted=0 during off-season since no zones were previously
published).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
meshai-requested adapter for avalanche.org's per-center map layers (SNFAC
Sawtooth + PAC Payette by default; operator-extensible to any avalanche.org
forecast center). Pure passthrough + severity gate, no cross-source fusion,
fits Central's adapter pattern cleanly.
Adapter surface:
- Polls https://api.avalanche.org/v2/public/products/map-layer/{center_id}
per configured center; default cadence 1800s (30 min).
- Severity gate: only danger_level >= 3 publishes. danger_level 0/1/2
(None/Low/Moderate), -1 ('no rating'), and off_season=true all omitted at
adapter level. Idaho summer = all 4 SNFAC + 2 PAC zones yield 0 events;
that's correct behavior, verified by the negative-case test against the
frozen 2026-06-08 SNFAC fixture.
- Severity mapping (corrected from meshai's inverted spec): danger_level
3 (Considerable) → severity 2, 4 (High) → 3, 5 (Extreme) → 4. Matches
Central's 4-most-severe convention (nws.SEVERITY_MAP).
- Subject: central.avy.advisory.us.{state_lower} — one per state; v0.10.8's
category-discriminated Nats-Msg-Id keeps multiple zones in the same state
from colliding in JetStream dedup.
- Stream: CENTRAL_AVY (central.avy.>); 7-day / 1 GiB retention defaults.
- Event.data fields per meshai spec: center_id, zone_name, danger_level,
danger_name, travel_advice (truncated to 200 chars), state, valid_date,
end_date, off_season=false, latitude/longitude (polygon centroid via
shapely), plus geo.geometry passes through as the upstream Polygon.
Tests (38 in test_avalanche_org.py):
- Pure helpers: _slug (8 cases), _parse_iso (6 cases), _centroid (2 cases).
- Severity gate: 3 publish cases (danger 3/4/5 → severity 2/3/4),
4 omit cases (danger -1/0/1/2), off_season=true omit, missing state omit,
unparseable geom omit, travel_advice truncation, subject derivation.
- Real-fixture negative case: 4-zone SNFAC fixture all omitted off-season.
- Real-fixture positive case: same fixture with synthetic winter overrides
publishes all 4 with valid centroids on actual Idaho polygons.
- End-to-end poll() with mixed severities and the new wiring (streams
registry + supervisor family map).
- Defensive: empty center_ids list yields nothing without crashing.
Wiring + plumbing:
- src/central/streams.py: StreamEntry('CENTRAL_AVY', 'central.avy.>')
- src/central/supervisor.py: STREAM_CATEGORY_DOMAINS['CENTRAL_AVY']=('avy',)
- sql/migrations/035: seed config.streams row (mirror of 019/CENTRAL_SPACE,
idempotent ON CONFLICT DO NOTHING). Note: migrations don't auto-run on
supervisor restart -- see deferred ops list (schema_migrations cleanup
blocks central-migrate from running anything cleanly).
- src/central/gui/templates/_event_rows/avalanche_org.html (8 lines)
- src/central/gui/templates/_event_summaries/avalanche_org.html (2 lines)
Both required by the existing per-adapter template consistency tests.
Doc updates (required by existing doc-vs-registry tests):
- docs/PRODUCER-INTEGRATION.md §6.1: added 'avy' to top-level-domain list.
- docs/PRODUCER-INTEGRATION.md §8: added StreamEntry('CENTRAL_AVY',...) line
to the verbatim snippet.
- docs/CONSUMER-INTEGRATION.md §3 stream layout table: added CENTRAL_AVY row.
- docs/CONSUMER-INTEGRATION.md §6: new '### avalanche_org' subsection with
source, subject convention, dedup key, severity gate, Event.data field
table, and off-season behavior note.
- tests/test_events_feed_frontend.py: added avalanche_org to _SAMPLE_INNER
and _EXPECTED_SUBJECT (the events-JSON subject-derivation coverage tests).
Budget note: this PR is well over the ~400-line target -- the new-adapter
surface picked up downstream consistency tests (doc validators + frontend
sample coverage + template partials) I didn't anticipate at probe time.
Most of the overrun is the SNFAC fixture (1,135 lines pretty-printed JSON,
non-code) and the adapter + tests pair. Stripping the fixture and the
required doc/template edits would leave ~620 lines of code; the fixture
itself is a frozen snapshot, not a maintenance burden.
Full sweep: 1072 passed, 0 failures (+41 from this PR), ruff clean on
all new files. One PRE-EXISTING ruff violation in supervisor.py (unused
poll_start variable at line 388) surfaces when we touch supervisor.py;
confirmed not introduced by this PR via git stash check.
Deploy plan (NEW STREAM — archive restart required per
[[feedback_new_stream_needs_archive_restart]]):
1. Squash-merge -> tag v0.10.10 -> push.
2. On central: pull main -> systemctl restart central-supervisor -> ALSO
systemctl restart central-archive (new event-bearing stream; archive
enumerates consumers at startup and doesn't hot-reload).
3. Migration 035 deferred to morning per the schema_migrations cleanup
task -- the stream creation itself doesn't depend on it (supervisor
creates JetStream streams from the STREAMS registry at startup; the
config.streams row is for operator-tunable retention only).
4. Verify: nats stream info CENTRAL_AVY (created), poll log shows
yielded=0 / omitted=N (off-season), no positive publishes during
summer (correct).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.10.2's migration 030 set monitor_north=44.5 as the schema default.
Idaho extends to 49.0N -- the original default cut off the entire northern
panhandle (Coeur d'Alene, Lewiston, Sandpoint, Moscow, McCall). Tonight's
investigation caught the impact at scale: supervisor's publish-time bbox
filter was dropping ~56 itd_511 events per 60-second poll (~5,376/day),
the entire north half of Idaho's roadwork, closures, and incidents. NWS
Idaho UGC-zone alerts in the panhandle were similarly dropped (closes
the v0.10.7 PR #95 followup-ticket-(b)).
Three changes:
1. sql/migrations/034: ALTER COLUMN SET DEFAULT 49.0 for fresh installs,
plus an idempotent UPDATE ... WHERE monitor_north = 44.5 that bumps
existing deployments still at the dev value. The WHERE 44.5 guard
preserves any operator who deliberately narrowed -- those rows are
untouched.
2. src/central/gui/routes.py: _DEFAULT_MONITOR["north"] 44.5 -> 49.0
for the Python-level fallback when no system row exists.
3. tests/test_monitoring_area.py: new TestDefaultMonitoringAreaCoversIdaho
class with 9 Idaho sentinel cities spread panhandle-to-south,
west-to-east. Three tests: (a) every sentinel is in-bounds against
the current default; (b) belt-and-suspenders corner-value assertions;
(c) anti-regression test using the OLD v0.10.2 narrow bbox documenting
that it would reject the panhandle.
Phase 1 (immediate, no-PR): the production row was UPDATEd from 44.5 to
49.0 directly via psql before this PR. Drop counter verified to stop
incrementing for itd_511 within one refresh cycle; first post-fix
panhandle publish landed at seq 80741 (Coeur d'Alene closure at
lat=47.7N) at 07:35:33 UTC. published_ids count went from 145 to 201
per poll, no longer dropping the 56-event panhandle batch. This PR
ensures the new code's default is consistent with the live config row
and that future fresh installs don't reintroduce the narrow bound.
Existing IDAHO test fixtures (test_archive_bbox_filter, test_supervisor_
publish_filter, test_fire_fused, test_monitoring_area) still use the
narrow 44.5 value; those are testing bbox behavior with arbitrary inputs
not the production default, so leaving them alone keeps this PR focused
on the production default fix. A separate cosmetic cleanup could update
the IDAHO fixtures if desired.
Full sweep: 1031 passed (+3 from this PR), ruff clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
JetStream's 2-min per-stream duplicate window keys on Nats-Msg-Id alone --
the subject is NOT part of the dedup key. So when wfigs_incidents and
wfigs_perimeters published envelopes for the same Summit Creek IRWIN
{6B1C6EB1-30F7-4613-9C58-4801DC8FD822} within seconds of each other on
2026-06-07, the second publish was silently dropped: supervisor logged
'Published event' for both, but CENTRAL_FIRE's last_seq advanced by 25
instead of 26.
Root cause: cloudevents_wire.wrap_event returned event.id verbatim as
msg_id, and both wfigs adapters use the bare IRWIN GUID as event.id.
Fix: synthesize msg_id as f'{event.id}:{event.category}' in wrap_event.
Categories already differ ('fire.incident.X' vs 'fire.perimeter.X') so
this is a natural and load-bearing discriminator. envelope['id'] stays
the bare event.id per CloudEvents spec, so subscribers that key off the
payload id field are unaffected. Adapter-side event.id construction
unchanged.
Backward compatibility: this is a one-time msg_id-shape change. The
JetStream 2-min dedup window is per-(event-id, category) starting now
instead of per-event-id. For events still in the dedup window at the
moment of the deploy under the OLD shape, their first post-deploy
publish under the NEW shape is a fresh msg_id and lands normally -- no
collision risk. Quake and other single-class adapters get a no-op shape
change (their category is constant per adapter, so dedup still catches
genuine same-id duplicates).
Tests:
- test_required_fields_present updated for the new assertion.
- New test_msgid_disambiguates_incident_vs_perimeter_same_guid regression
guard with the real Summit Creek IRWIN -- assert the two msg_ids differ
and match the expected shape exactly.
- New parametrize test_msgid_shape_is_id_colon_category over 5
categories covering multi-class (fire.incident, fire.perimeter) and
single-class (quake, wx.alert, hydro) adapters.
Full sweep: 1028 passed, 0 failures (+6 from this PR), ruff clean.
Deploy: NOT an output-shape change to envelope payload, no published_ids
flush needed. Squash-merge -> tag v0.10.8 -> pull on central -> restart
central-supervisor. Verify by forcing a Summit Creek republish and
confirming both .incident.id.cassia AND .perimeter.id.cassia land
post-deploy with the same IRWIN.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SAME format (FCC standard) is PSSCCC: P=area-type indicator, SS=state FIPS,
CCC=county FIPS. Pre-v0.10.7 _build_regions read code[:2] (PS) as the state,
so the leading P=0 collapsed every SAME code to state 01 = Alabama. The
dumped CAP envelope from CENTRAL_WX#968 showed Bannock County Idaho alerts
flagged as US-AL-FIPS016005 + primary_region=US-AL-FIPS016005, which then
routed to central.wx.alert.us.al.county.fips016005 instead of
central.wx.alert.us.id.county.fips16005.
Fix:
- Slice code[1:3] for the state FIPS, code[1:] for the 5-digit ANSI county
FIPS (SSCCC -- standard interoperable form). Drops the P padding from
emitted region/subject; P stays preserved verbatim in data.geocode.SAME
for any power user that needs it.
- Length guard tightened: ==6 + isdigit + isinstance str (was >= 2). Now
malformed entries (too short, too long, non-digit, None) are silently
skipped with no crash.
- Deleted dead _extract_states_from_codes (defined but never called; same
bug, removed rather than fixed).
Tests:
- New TestSameStateParse parametrized over 4 distinct-state cases per spec:
016005 -> US-ID-FIPS16005 (Bannock area), 001005 -> US-AL-FIPS01005
(Autauga area), 056005 -> US-WY-FIPS56005 (Carbon area), 049005 ->
US-UT-FIPS49005 (Cache UT).
- Area-subset (P>=1) and unknown-state-FIPS coverage.
- Malformed-input parametrize: empty, too short (2 forms), too long (7
digits), non-digit char, all-alpha, None -- each silently skipped.
- Existing SAMPLE_FEATURE_* fixtures updated from constructed-to-match-bug
values (160001/410051/060037/530033) to proper 0SSCCC format
(016001/041051/006037/053033); existing TestBuildRegions assertions
updated to expect 5-digit ANSI form.
Followup ticket (NOT v0.10.7 scope, recorded in PR body):
(a) Null-geometry alerts with valid Idaho UGC zones are silently dropped
by _geometry_intersects_region (line 297-298): needs UGC-fallback or
geometry-or-UGC check. NWS issues many Special Weather Statements
without GeoJSON polygons but with UGC IDZ* zones that should pass.
(b) Configured monitoring bbox north=44.5 only covers the southern third
of Idaho; Idaho extends to 49.0N, so Coeur d'Alene / Lewiston / etc.
are out of scope. Verify whether the narrow bbox was an intentional
dev limit or accidental.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
itd_511's free-text Comment field carries a milepost in roughly a third of
the live samples ('milepost 32.5', 'MP 80 to MP 81', etc.). meshai's roads
integration needs that as a structured field; wzdx and tomtom_incidents
already speak in structured mile-post / from-to so itd_511 is the only
adapter that needs the regex extraction layer.
Design (per Step-0 review):
- Shared module src/central/enrichment/mile_marker.py exporting
extract(text) -> {value, source, confidence} | None. Pure regex, no I/O,
re-usable by future per-state-DOT adapters (Wyoming, Montana, ...).
- itd_511 calls extract on the Comment in _build_event_record; result lands
under the established _enriched namespace (NOT a new _enrichment one),
keyed 'mile_marker'. Same convention the supervisor's geocoder uses, same
merge semantics apply_enrichment already supports. Absent when no match
(no null placeholder) so subscribers can tell 'not mentioned' from
'extraction found nothing'.
- Confidence tiers: 'high' (single unambiguous MP/milepost/MM match),
'medium' (multiple matches like 'MP 80 to MP 81' -- first wins), 'low'
(bare 'mile N' only; consumers can ignore).
Tests:
- tests/test_enrichment_mile_marker.py: 36 cases parametrized over the 15
real ITD comments I pulled from CENTRAL_TRAFFIC, including the critical
red-herring classes the regex must reject (phone numbers, project key
numbers, state-highway numbers, date/time numbers). Crafted samples
cover M.P. / MM / milemarker / bare-mile patterns not in live ITD data
but required by spec for future DOT adapters.
- tests/test_itd_511.py: 2 integration tests confirming the bundle is
attached on a milepost-bearing Comment and absent otherwise.
Pure enrichment, no schema-breaking changes; meshai's renderer picks it up
additively.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v0.10.5 ephemeral pull-consumer used DeliverPolicy.BY_START_TIME with
no upper bound, so every republish satisfied the same time filter and
the consumer fetched its own output back -- an unbounded loop bounded
only by the per-stream cap. Operator-triggered 5-minute resend on
2026-06-07 ran the loop long enough to time out central-gui's POST and
the host went down with it.
Fix: snapshot each event-bearing stream's last_seq up front via a new
_snapshot_last_seqs() helper, pass it to _iter_window as max_stream_seq,
and exit the generator the first time msg.metadata.sequence.stream
exceeds the snapshot. Pull-consumer delivery is stream-seq ascending so
one boundary check suffices.
Also drop _MAX_MSGS_PER_STREAM 50_000 -> 5_000 and add a WARNING log
when the cap is hit -- a legitimate operator window should never reach
it, and silent truncation hid the v0.10.5 loop until the host fell over.
Two regression tests cover the new behavior: one stages pre/post-snapshot
batches and asserts the post-snapshot batch is never yielded; one
overwhelms the cap and asserts the warning fires.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First official-state-DOT-API pattern landing. Two adapters in one PR:
- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
to LineString via the polyline lib (bookend LineString or Point fallback);
ITD Severity string mapped None->1 / Minor->2 / Major->3 with
IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
Restrictions / DetourPolyline pass through unmodified. Advisories ship
as structural pass-through under data.advisory since the upstream
/alerts endpoint currently returns []; per-record try/except keeps a
surprise shape from sinking the cycle when ITD posts its first one.
- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
camera per UTC day; image URL passes straight through to <img src>.
Region uniform US-ID with data.source_jurisdiction preserving the raw
upstream Source field for the ~1.2% cross-DOT border-region mirrors
(UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).
Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.
Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.
Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.
Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FIRMS hotspots rendered as tiny dots on /events because the adapter shipped
`bbox=(lon, lat, lon, lat)` and no `geometry`. `_build_geom_sql` collapsed
that into a 5-vertex zero-area `ST_Polygon`. Fix: construct a real rectangular
GeoJSON Polygon from `(latitude, longitude, scan, track)` per record and ship
it via `Geo.geometry`, mirroring the v0.9.3 pattern used by `tomtom_flow`.
- `_pixel_polygon()` builds a CCW 4-corner ring with the closing vertex
duplicated. Cross-track (longitude) extent scales by `cos(lat)`. Latitude
is pole-clamped to ±89° defensively.
- `_row_to_event` drops the degenerate bbox and sets `geo.geometry`;
centroid stays alongside for consumers that read centroid only.
- `poll()` emits a single per-cycle warn if any record fell back to
centroid-only Geo (missing scan/track).
- Forward-only at the geometry layer: same dedup keys, new payload shape
for new records only. Existing PostGIS rows retain their degenerate
polygons until republished or aged out (48h dedup window).
- Side-quest: removed a pre-existing dead RegionConfig import in
tests/test_firms.py since the imports block was touched.
Tests: 5 new `_pixel_polygon` unit tests + 2 archive round-trip regression
guards mirroring `test_archive_prefers_geo_geometry`. Full suite 930 passed
/ 1 skipped under both `central` and `zvx` users.
CAVEAT — axis mapping convention: the `_pixel_polygon` docstring describes
`scan` as the along-track (latitude) extent and `track` as the cross-track
(longitude) extent. This matches the v0.9.21 prompt and is internally
consistent. NASA FIRMS convention may have these inverted — `scan` is often
documented as the cross-track scan width and `track` as the along-track
extent. If deploy-time visual inspection shows polygons rotated 90° from
expected, swap the assignments inside `_pixel_polygon`:
half_scan_deg → multiply by cos(lat) (treat as longitude offset)
half_track_deg → divide by 111.0 only (treat as latitude offset)
Clickability is the first-order goal and is unaffected — the polygon will
still have non-zero area in both axes either way; only the aspect ratio
flips. Tracked as a fast-follow if visual inspection requires it.
PR #84.
* v0.9.20: regional subject routing on quake / fire / hydro / disaster adapters
Adds regional subject tokens to four adapters that previously published
without location-based routing:
- usgs_quake: central.quake.event.<tier>.us.<state> (US) or .<country> (intl)
- firms: central.fire.hotspot.<sat>.<conf>.us.<state> or .<country>
- nwis: central.hydro.<param>.<agency>.<site>.us.<state> (always US)
- eonet: central.disaster.eonet.<cat>.<country> (replaces hardcoded .global)
The us.<state> pattern (two tokens for US events) matches the NWS precedent
and resolves the ISO-2 collision between Idaho (id) and Indonesia.
New shared helper module: src/central/adapters/_subject_helpers.py
- US_STATE_NAME_TO_CODE: 50 states + DC + territories
- subject_for_country(): normalized country token
- subject_for_region(): returns us.<state>, <country>, or unknown
gdacs.py refactored to import subject_for_country from shared module.
Fixes: meshai v0.4 Phase C.3 bug (M4.1 Nevada quake routed globally)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* v0.9.20: stale docstring nit on renamed test
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
NWSAdapter.sweep_old_ids ran an unscoped global DELETE FROM published_ids
(no adapter = ? filter), so each per-adapter sweep cycle purged EVERY
adapter dedup rows older than 8 days -- capping dedup memory at ~8d for the
8 adapters whose window exceeds it (eonet/gdacs/nwis 30d; swpc_kindex/
protons/alerts + wfigs_incidents/perimeters 14d). Events that went silent
past 8d then re-listed were re-published as downstream duplicates.
Extract the 3 dedup methods onto the SourceAdapter base (finishing v0.9.19;
is_published/mark_published were byte-identical) and preserve the 8-day
window via dedup_sweep_days = 8, so the inherited sweep scopes to adapter=?.
Effect: each adapter retains dedup to its configured window -> fewer
downstream duplicates. Retention-window scoping, NOT a dedup-forward-only
change (no event-shape change, no migration/backfill).
Adds a regression guard asserting a foreign adapter row survives an nws
sweep. Suite 900->901/1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrations 025-029 (wzdx + traffic-family adapters/streams) were applied
out-of-band via direct psql during their deploys and never recorded in
schema_migrations. Result: a fresh restore would replay them, and an audit of
the tracking table understated what was actually live. All five are pure
additive INSERT ... ON CONFLICT DO NOTHING (verified idempotent by inspection),
so they were back-filled directly into schema_migrations (5 rows, applied_at =
now() = the reconciliation event; the original out-of-band apply dates are
unrecoverable and are documented as such rather than guessed).
Adds `central-migrate --check` to catch this class of drift going forward:
- find_drift(): pure function comparing schema_migrations rows vs *.sql files,
returning (untracked, orphan). Unit-tested, no DB dependency.
- check_drift(): CI-assertion form -- exit 1 if any file is untracked, else 0.
- log_drift(): WARNs per drifted entry, called on every migrate run too.
No migration 031 for the v0.9.17 wzdx state default: that default lives in
adapter code (_read_states falls back to _DEFAULT_STATES), so the live row's
explicit 7-state array is behaviorally identical to a fresh-install null.
Materializing it into SQL would freeze a snapshot of _DEFAULT_STATES and create
a new drift vector. Deferred to a future "show effective defaults" UI PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Narrow which WZDx registry feeds are fetched at poll time, so out-of-state
feeds are never requested (no bandwidth/quota cost), rather than fetching
nationally and dropping out-of-bounds events at the archive INSERT (v0.9.12).
- states is now list[StateCode] (Literal of 56 canonical 2-letter codes),
default ["ID","WA","OR","NV","UT","WY","MT"]. Semantic flip: empty/unset/null
now means the Idaho-region default, NOT "every eligible feed". Poll nationally
by selecting all states in the GUI.
- list[Literal] renders as the existing checkboxes multi-select widget (firms.py
precedent); pydantic rejects malformed codes at save time.
- Add informational quota_estimate (cap=None: never warns/blocks) so the edit
page shows the fetch-volume reduction.
- adapters_edit.html: standalone quota panel for flat-config adapters, guarded
by not has_model_list so model_list adapters don't double-render; reuses the
identical quota.* keys + CSS so the v0.9.15 client recompute works for them too.
- 7 new tests (default/null fallback, _discover filtering, malformed rejection,
quota math, offline Jinja panel render).
Event data shape is unchanged (poll-behavior-only), so no published_ids reset
is needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mechanical lint fixes only — ruff 31 -> 0 across the two files.
- E402 module-import-not-at-top: 15 -> 0 (relocated module-level
`logger = logging.getLogger(...)` from mid-import to below the import
block in routes.py; imports now contiguous at top of file)
- F811 redefined-while-unused: 10 -> 0 (removed 10 redundant function-local
`from central.gui.csrf import reuse_or_generate_pre_auth_csrf` re-imports;
the module-level import at line 26 is now load-bearing)
- F401 unused-import: 4 -> 0 (routes.py: fastapi.Depends, and
central.gui.csrf.reuse_or_generate_pre_auth_csrf resolved by the F811 fix;
test_wfigs.py: sqlite3, central.config_models.RegionConfig)
- E702 multiple-statements-on-one-line-semicolon: 2 -> 0 (split the two
semicolon-joined statements in _fused_bbox, indentation preserved)
Deliberate function-local wizard imports (circular-import workaround) left
untouched. pytest: 890 passed / 1 skipped, unchanged across 3 runs.
no behavior change; ruff mechanical fixes only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror TomTomIncidentsAdapter.quota_estimate in client-side JS so the
quota panel updates live as the operator edits a per-row cadence or the
adapter cadence, and on Add/Delete bbox -- instead of only on Save.
- tomtom_incidents.py: expose seconds_per_month + default_cadence_s in
the quota dict (no JS magic numbers).
- model_list.html: data-quota-* attrs + .quota-detail/.quota-msg spans;
self-contained DOMContentLoaded-guarded IIFE applying the same
max(adapter_cadence, bbox_cadence||default) floor and 80%/100%
warn/block thresholds. No-op when no quota panel present.
- tests: structural asserts for data-attrs + span hooks (JS exec needs
a browser; live behaviour eyeballed manually).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs each live WFIGS perimeter with its nearby/contemporaneous FIRMS hotspots
into a single "fire" on the /events map. FIRMS hotspots carry no IrwinID, so the
link is spatial+temporal: a hotspot is confirmed (part of a known fire) when it
lies within 1km of a perimeter AND within 72h of it; hotspots matching no
perimeter render amber as "unconfirmed" -- a possible new fire detected by
satellite before an official perimeter exists (early-warning signal).
- routes.py: read-only /events/fire-fused.json (PostGIS ST_DWithin geography join)
- events_list.html: "Fuse fire layers" toggle (default on); centroid fire glyph
that expands to polygon + hotspot dots on click; amber unconfirmed hotspots
- central.css: --fire-confirmed / --fire-unconfirmed vars (retune without code)
- 11 tests (shaping, bbox parse, R/T + bbox param wiring); spatial correctness
verified on prod (Summit Creek: perimeter + 90 hotspots; 191 unconfirmed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hourly supervisor sweep deletes archived events older than their stream's
config.streams.max_age_s (the /streams 1/7/14/30/365d buttons -- the per-stream
source-of-truth). Explicit STREAM_CATEGORY_DOMAINS map (category domains do not
equal stream subject domains for the traffic family); fail-safe skip on
missing/<=0 max_age_s; unmapped-domain warning so a new adapter cannot silently
dodge retention.
TimescaleDB hypertable gotcha: events.ctid is chunk-local, NOT globally unique --
DELETE batching keys on the composite PK (id, time). See the inline NOTE in
config_store.delete_events_older_than and the PR body for the incident write-up.
- config_store: delete_events_older_than (batched on (id,time)) + unmapped_event_domains
- supervisor: STREAM_CATEGORY_DOMAINS, _sweep_events_retention, _events_retention_loop
- streams_list.html: max-age also governs archived-events retention
- 9 tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a single archive-level geographic filter at the events INSERT path: events
whose geometry falls entirely outside a system-configured monitoring area are
dropped before archival. Null-geom events (SWPC trio, .removed tombstones) are
always kept. Uses a pure shapely intersects() predicate so border-straddlers
are kept (matches ST_Intersects), and is fail-open on unparseable geometry.
- config.system gains monitor_{north,south,east,west} (migration 030, Idaho default)
- archive refreshes the bbox every 60s (no restart needed to change it); adds a
per-adapter dropped-count counter, debug log per drop, cumulative INFO rollup
- new GUI editor at /monitoring-area (Leaflet draggable rectangle, N/S/E/W inputs)
- no adapter code changes; well-behaved adapters keep upstream API filtering
- 12 tests covering all five verdicts, drop-and-count, border/point-on-edge, fail-open
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /events feed was dominated by *.removed tombstone events (audit records
for features dropped from upstream feeds), burying geometry-bearing events
like fire perimeters (wfigs_perimeters: 54 real perimeters vs 1015 tombstones).
The GUI now default-hides any event whose category ends in .removed, with a
"Show removed" checkbox to restore them; URL state is preserved (HX-Push-Url)
so a shared link shows what the sharer saw. events.json is unchanged (still
returns tombstones) so API consumers are unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a passive Leaflet map to the generic model_list editor that renders
every bbox row as a labeled translucent rectangle, auto-detected via the
min/max lon+lat sub-fields (TomTom incidents). Read-only: no drag/draw,
precise tuning stays in the per-row coord inputs. Rectangles redraw live
on input/add/remove; viewport fits only on initial render so typing never
jumps the map. Non-bbox model_lists (StateConfig, TileCoord) are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes a shared form_descriptors 500 (NotImplementedError: unsupported list
type) that broke the Edit page for ALL FOUR adapters whose settings carry a
list[<BaseModel>] field: tomtom_incidents, tomtom_flow, state_511_atis,
state_511_atis_cameras.
- form_descriptors: list[BaseModel] -> generic "model_list" widget with
recursive per-column sub_field descriptors.
- New _partials/model_list.html: vanilla-JS repeatable-row editor
(add/remove/renumber), driven entirely by sub_fields (no adapter-name
branching). Single-region edit pages render byte-identically.
- TomTom: BBox/Settings Pydantic validators (10,000 km^2 cap, coord ranges,
min<max, cadence_s>=60, unique names) as the single source of truth
(enforced at supervisor load AND GUI POST). Duck-typed quota_estimate hook
+ read-only quota panel; POST hard-blocks estimates over the 2,500/mo free
tier (422). TOMTOM_FREE_TIER_CALLS_PER_MONTH is a tunable for paid tiers.
- routes: model_list form parse, row-aware ValidationError messages, 422 for
model_list failures (single-region region errors still re-render at 200).
- tests: 11 new (real-Jinja render across 3 adapters + byte-identical nws
no-regression guard, POST persist + oversized/degenerate/duplicate/cadence/
quota 422 matrix, quota estimate). Full suite 848 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plumb the upstream GeoJSON Polygon/MultiPolygon into Geo.geometry so the
archive renders the real fire-perimeter shape (ST_GeomFromGeoJSON) instead
of the bbox 4-corner fallback. Adapter-local change; the v0.9.3 Geo.geometry
framework already carries it end-to-end. Graceful null + dict-or-string
coercion via _as_geometry; fall-off removal events stay NULL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Castle Rock caps each DataTables page at 100 rows regardless of `length`, so the
single-request fetcher silently dropped rows on any layer over 100 (confirmed
live: Construction recordsFiltered=114, returned 100 -> 14 rows invisible).
Backports the v0.9.6 cameras pagination loop into _fetch_details.
- _LIST_PAGE_LENGTH 1000 -> 100; new _list_body(start) builder; new @retry
_fetch_page(base_url, layer, start).
- _fetch_details loops start+=100 until recordsFiltered collected or an empty
page, with a _MAX_PAGES=50 ceiling that warns+breaks. Mid-pagination failure
returns rows-so-far (retried next poll).
- Incidents (1) / Closures (29) are under 100 today but pagination applies
uniformly; future-proof.
central-supervisor restart only (no stream, migration, template, or dep change).
Full suite: 833 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).
supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.
- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
cams). The 100-row page cap also means v0.9.2 state_511_atis silently
truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
-- always shows today's cameras, no per-poll flooding, no retention
coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).
Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.
- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
falls back to the adapter's default_cadence_s. Existing settings without the
field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
(one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).
central-supervisor restart only. No gui/archive restart, no migration, no new dep.
Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.
central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.
- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
state_511_atis; cross-source overlap is by design = additive coverage, distinct
dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
key redacted from logs; poll skips cleanly without a key.
Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phased PR2 of v0.9.3. Adds the on-demand tile passthrough so Navi can call
Central instead of navi-traffic, and so tiles outside the polled coverage set
still get persisted. central-gui restart only (new route); no supervisor, no
archive, no migration, no new stream/dep.
GET /api/traffic/flow/<z>/<x>/<y>.{png,pbf} -- exactly matches navi-traffic's
route for a drop-in frontend flip (s/navi-traffic/central:8000/). No auth
(tailnet-trusted, matches navi-traffic's existing posture; exempted in
middleware). Cache-Control max-age=60 (TomTom's advertised tile TTL).
- .png -> passthrough + cache only (raster unparseable for storage).
- .pbf -> passthrough + decode via the shared tomtom_flow_parse.decode_flow_tile
and publish segments to CENTRAL_TRAFFIC_FLOW via the GUI's NATS connection
(wrap_event, supervisor's exact envelope shape). Same minute-bucketed dedup id
as the polling adapter, so adapter+passthrough fetches of one tile in the same
minute collapse at the archive's (id, time) upsert -- intentional collision.
- Publish is best-effort: wrapped in try/except + guards get_js() None, so NATS
state never blocks serving the tile (HTTP > storage).
- API key via ConfigStore(get_pool()).get_api_key("tomtom"); 503 if unset;
redacted from any logged upstream error; 502 on upstream failure (no publish).
Single-flight (shared in-flight upstream request per tile) queued for v0.9.5.
navi-frontend flip + navi-traffic deprecation live in the navi repo (Matt's call).
Full suite: 787 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.
Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.
- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
transitive that uv re-lock would otherwise prune).
Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.
Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.
Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.
Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.
Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.
Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui
restart (adapter base change + template change). No migration, no new stream.
(a) Work-zone subject + detail no longer leak vehicle direction "unknown"
(common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both
wzdx partials. Was "Work zone on MORELAND ST unknown".
(b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline
copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days
class attr parameterizes the retention window (NWIS=30, default=14). Inline
copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a
future cleanup. Net dedup code down ~52 lines.
Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens the v0.8.x data-quality cleanup arc. Production code; central-gui AND
central-supervisor restart (adapter contract + enrichment behavior change).
NWIS events rendered as a bare "Water reading: 111 ft3/s" with an empty Location
column -- an operator couldn't tell where the gauge is or whether 111 ft3/s is
drought-low, normal, or near-flood. Coordinates were present but the reverse
geocoder returns null city/state/county for rural gauge points, and USGS site +
percentile data was never fetched. v0.8.0 fetches both.
Approach B (adapter-owned, per the proposal decision): the NWIS adapter -- which
already owns the USGS APIs -- fetches site metadata and daily stats itself and
writes two provenance bundles under event.data["_enriched"]:
- usgs_site {name, lat, lon, state, county} from the OGC monitoring-locations
item-by-id (the API family the adapter already speaks; JSON, no RDB parser).
- usgs_stats {value, percentile, class_label, severity_band, p10..p90, record_max,
count, period} from the legacy RDB daily-statistics service (the OGC API has no
stats endpoint). USGS percentiles are % of days at-or-below, so higher = higher
flow; classified to the WaterWatch bands -> severity 0-4 (record=4, much
above/below=3, above/below=2, normal=1; None reserved for "no stats", distinct
from a normal-flow gauge). Severity is set on the event, so it drives the v0.7.1
severity chip-picker filter + v0.7.2 map-marker opacity.
- new nwis_enrich.py: pure parse/classify/percentile/band helpers + a sqlite
SiteStatsCache (site TTL 365d, stats TTL 90d -- one fetch per site+param serves
every reading for the window, so a warm cache makes zero USGS calls). USGS down
-> cached-if-present else all-null bundle; the event still publishes.
Framework: the single agreed generic change -- supervisor apply_enrichment now
MERGES into _enriched instead of overwriting, so the still-global geocoder phase
doesn't clobber the adapter's bundles. No other adapter writes _enriched, so this
is inert for them.
GUI: _event_summaries/nwis.html -> "<site> -- <value> <units> (<band>, <Nth>
percentile)", with graceful fallback to "<site> -- <value>" then the bare
"Water reading:". _event_rows/nwis.html detail gains site/normalcy/typical/location
rows. _events_rows.html Location column falls back generically to any
_enriched.<source> carrying state/county when the geocoder is null (works for
future enrichers). events.json contract unchanged (additions under _enriched only).
conftest isolate_enrichment_cache also redirects NWIS_CACHE_DB_PATH off the prod
path (unprivileged-user test isolation). Adds tests/test_nwis_enrichment.py (28
tests: parse, band edges incl P0/P9/P10/P75/P90/record, percentile interpolation,
cache hit/miss/expire, adapter enrich + graceful-null + cache-hit-no-refetch,
summary rendering per band).
Full suite: 710 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 of the v0.7.x GUI rework arc. Production code; central-gui restart only.
- Adapter legend: collapsed by default ("{n} adapters · Show legend ▾"). Expands
to domain-grouped chips (same grouping as the v0.7.1 chip-picker) with uniform
ellipsis-truncated names + full-name title tooltips. Clicking a legend chip
toggles that adapter's filter (reuses the chip-picker's hidden CSV via
syncField), so the legend doubles as a filter affordance.
- Row stability: time cell is single-line MM-DD HH:MM UTC (year dropped from the
cell; full ISO in the cell tooltip + a new Time row in the expanded detail).
Adapter cell is a chip (color swatch + short name; display_name is the
tooltip). table-layout:fixed + per-column widths + fixed 37px row height with
nowrap/ellipsis cells -> no per-row wrap variation.
- Real paginator: _fetch_events offset-mode returns the exact page slice plus the
grand total via count(*) OVER() in one roundtrip. Previous/Next + windowed page
numbers (1 ... 4 5 [6] 7 8 ... 47) + "showing X-Y of N" + a 25/50/100/250
per-page selector. URL state persists offset + limit. events.json keeps cursor
pagination (back-compat): offset param presence selects offset-mode, its
absence keeps the cursor path -- cleanly separable by endpoint.
Adds TestEventsPagination (12 tests: offset/limit parse incl. max 250,
offset-vs-cursor query shape, _build_pagination windowing). Updates the time
format + adapter-cell + pagination-mode assertions in the existing frontend
tests to the new contract.
Full suite: 674 passed, 1 skipped (central and unprivileged zvx). count(*) OVER()
is ~7.5ms at current volume; vanilla JS + HTMX; CSS functional-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3 of the v0.7.x GUI rework arc. Makes the /events Leaflet map readable and
intentional. Production code; central-gui restart only (no adapter change).
- Fit-to-results default: the map now fits the actual event distribution on
load (previously disabled -> fixed global zoom-4). Empty result set falls
back to the CONUS setView (no crash). Re-fits after each HTMX swap, but only
when the map-filter toggle is OFF (when ON the viewport drives the bbox, so
re-fitting would fight/loop the filter).
- leaflet.markercluster (1.5.3, via CDN): point markers cluster into numbered
badges (disableClusteringAtZoom=9, showCoverageOnHover=false,
spiderfyOnMaxZoom=true). markercluster supports point markers only, so
polygons/lines render in a separate un-clustered featureGroup; fit unions both.
- Map-filter toggle ("Filter table by map view"), default OFF. When off the
table shows all filter-matching events regardless of map zoom; the backend
ignores region_* unless map_filter is set (guards bookmarked URLs too). URL
carries map_filter=1 only when on (hidden input disabled otherwise).
- Per-event_type marker shape (derived event_type = first category segment):
circle = quake/hydro/space (points), square = fire (areas),
triangle = wx (NWS alerts/warnings), star = disaster (GDACS/EONET).
Rendered as divIcon + CSS clip-path; point markers switched from circleMarker
to L.marker(divIcon) (also required for markercluster compatibility).
- Per-severity opacity: critical(4)=1.0, high(3)=0.85, moderate(2)=0.7,
low(1)=0.5, unknown(0/NULL)=0.4. Needed adding severity to the _fetch_events
SELECT + event dict (row.get for mock-tolerance) + a data-severity row attr.
Adds 4 tests (map_filter gating on/off, bbox reaches query only when on,
severity in SELECT); updates test_events_bbox_guard for the new toggle contract.
Full suite: 662 passed, 1 skipped (central and unprivileged zvx). Vanilla JS +
HTMX + Leaflet/markercluster; CSS functional-only (polish deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Biggest PR of the v0.7.x GUI rework arc. Replaces the single-select /events
filter row with a multi-select, URL-addressable filtering surface.
- Search: full-width box, debounced 300ms, server-side ILIKE over the inner
adapter payload (covers the derived subject + location); parameterized with
LIKE wildcards escaped (ESCAPE '\'). Injection-safe.
- Adapter / Category / Event Type / Severity: multi-select chip-pickers (shared
_chip_picker.html macro). Adapter is grouped by domain with color swatches and
an in-panel search. Backend uses `= ANY(...)`. URL state is comma-separated.
- Event Type is derived as split_part(category,'.',1) (no event_type column yet;
a stand-in until the v0.8 canonical schema). Severity maps labels to the
numeric scale (4=critical..1=low, 0/NULL=unknown).
- Time: preset dropdown (15m/1h/6h/24h/7d/active/all) + custom from/to range,
encoded in a single `time` token. GUI defaults to last_24h; events.json keeps
its single-value adapter/since/until contract (no default).
- Active pills: server-rendered from parsed state, updated out-of-band on each
HTMX swap; each x clears that filter and re-submits.
- URL state persistence: every filter in the query string; /events/rows sets
HX-Push-Url to the /events?... full-page URL so bookmarking/back-forward work.
Filter options are rendered server-side at page load (DISTINCT category +
split_part, registry adapters, severity enum) -- no new AJAX endpoints.
Vanilla JS + HTMX (no framework added). CSS is functional-only; visual polish
is deferred to a later pass per the rework plan.
Adds TestEventsFiltering (24 tests: multi-value parse, ILIKE injection safety,
time-preset resolution with injected clock, severity/NULL handling, active-pill
descriptors, URL round-trip). Updates four TestEventsFeedFrontend assertions to
the new filter_state/adapters contract.
Full suite: 658 passed, 1 skipped (central and unprivileged zvx). No adapter
base class change -> central-gui restart only (no supervisor restart).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events_json SELECT read payload->>'subject', but the CloudEvents
envelope has no top-level subject, so every JSON consumer saw
subject: null. The /events GUI already derives readable subjects via
per-adapter templates/_event_summaries/{adapter}.html (PR L-c).
This makes the JSON path produce the same plain-text subjects with no
duplicated logic: _derive_subject(event) renders the same partial the
table uses (falling back to _default.html) and html.unescapes the
autoescaped output so JSON consumers get plain text (e.g. ">=1 MeV"
rather than the escaped ">=1 MeV"). _fetch_events now sets subject
from it and drops the always-null SQL expression. The GUI Subject cell
is unchanged.
Adds TestEventsJsonSubject (parameterized over discover_adapters(), no
hardcoded list): non-null subject per adapter, equality with the rendered
partial, pinned human text for the deterministic adapters, swpc_alerts
truncation, and null fallbacks. Updates one TestEventRowDataAttributes
assertion that pinned the old SQL pass-through contract.
One route change plus tests; central-gui restart required.
Full suite: 629 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleans up unused imports and dead locals flagged by ruff in the test
files PR #50 (M-b) touched. Tests-only; no production code, no service
restart.
- test_supervisor_hotreload.py: drop unused AsyncMock/patch imports,
dead expected_wait/expected_next_poll locals, and two dead
state = AdapterState(...) blocks plus their now-orphaned local imports
- test_supervisor_integration.py: drop unused asyncio/patch/pytest_asyncio
imports and AdapterState from two function-local imports
ruff tests/ 92 -> 82 (the 4 named files now 0; all other files unchanged).
Full suite: 590 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
