Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.
- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
falls back to the adapter's default_cadence_s. Existing settings without the
field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
(one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).
central-supervisor restart only. No gui/archive restart, no migration, no new dep.
Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.
central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.
- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
state_511_atis; cross-source overlap is by design = additive coverage, distinct
dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
key redacted from logs; poll skips cleanly without a key.
Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phased PR2 of v0.9.3. Adds the on-demand tile passthrough so Navi can call
Central instead of navi-traffic, and so tiles outside the polled coverage set
still get persisted. central-gui restart only (new route); no supervisor, no
archive, no migration, no new stream/dep.
GET /api/traffic/flow/<z>/<x>/<y>.{png,pbf} -- exactly matches navi-traffic's
route for a drop-in frontend flip (s/navi-traffic/central:8000/). No auth
(tailnet-trusted, matches navi-traffic's existing posture; exempted in
middleware). Cache-Control max-age=60 (TomTom's advertised tile TTL).
- .png -> passthrough + cache only (raster unparseable for storage).
- .pbf -> passthrough + decode via the shared tomtom_flow_parse.decode_flow_tile
and publish segments to CENTRAL_TRAFFIC_FLOW via the GUI's NATS connection
(wrap_event, supervisor's exact envelope shape). Same minute-bucketed dedup id
as the polling adapter, so adapter+passthrough fetches of one tile in the same
minute collapse at the archive's (id, time) upsert -- intentional collision.
- Publish is best-effort: wrapped in try/except + guards get_js() None, so NATS
state never blocks serving the tile (HTTP > storage).
- API key via ConfigStore(get_pool()).get_api_key("tomtom"); 503 if unset;
redacted from any logged upstream error; 502 on upstream failure (no publish).
Single-flight (shared in-flight upstream request per tile) queued for v0.9.5.
navi-frontend flip + navi-traffic deprecation live in the navi repo (Matt's call).
Full suite: 787 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.
Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.
- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
transitive that uv re-lock would otherwise prune).
Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.
Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.
Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.
Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.
Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.
Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui
restart (adapter base change + template change). No migration, no new stream.
(a) Work-zone subject + detail no longer leak vehicle direction "unknown"
(common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both
wzdx partials. Was "Work zone on MORELAND ST unknown".
(b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline
copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days
class attr parameterizes the retention window (NWIS=30, default=14). Inline
copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a
future cleanup. Net dedup code down ~52 lines.
Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens the v0.8.x data-quality cleanup arc. Production code; central-gui AND
central-supervisor restart (adapter contract + enrichment behavior change).
NWIS events rendered as a bare "Water reading: 111 ft3/s" with an empty Location
column -- an operator couldn't tell where the gauge is or whether 111 ft3/s is
drought-low, normal, or near-flood. Coordinates were present but the reverse
geocoder returns null city/state/county for rural gauge points, and USGS site +
percentile data was never fetched. v0.8.0 fetches both.
Approach B (adapter-owned, per the proposal decision): the NWIS adapter -- which
already owns the USGS APIs -- fetches site metadata and daily stats itself and
writes two provenance bundles under event.data["_enriched"]:
- usgs_site {name, lat, lon, state, county} from the OGC monitoring-locations
item-by-id (the API family the adapter already speaks; JSON, no RDB parser).
- usgs_stats {value, percentile, class_label, severity_band, p10..p90, record_max,
count, period} from the legacy RDB daily-statistics service (the OGC API has no
stats endpoint). USGS percentiles are % of days at-or-below, so higher = higher
flow; classified to the WaterWatch bands -> severity 0-4 (record=4, much
above/below=3, above/below=2, normal=1; None reserved for "no stats", distinct
from a normal-flow gauge). Severity is set on the event, so it drives the v0.7.1
severity chip-picker filter + v0.7.2 map-marker opacity.
- new nwis_enrich.py: pure parse/classify/percentile/band helpers + a sqlite
SiteStatsCache (site TTL 365d, stats TTL 90d -- one fetch per site+param serves
every reading for the window, so a warm cache makes zero USGS calls). USGS down
-> cached-if-present else all-null bundle; the event still publishes.
Framework: the single agreed generic change -- supervisor apply_enrichment now
MERGES into _enriched instead of overwriting, so the still-global geocoder phase
doesn't clobber the adapter's bundles. No other adapter writes _enriched, so this
is inert for them.
GUI: _event_summaries/nwis.html -> "<site> -- <value> <units> (<band>, <Nth>
percentile)", with graceful fallback to "<site> -- <value>" then the bare
"Water reading:". _event_rows/nwis.html detail gains site/normalcy/typical/location
rows. _events_rows.html Location column falls back generically to any
_enriched.<source> carrying state/county when the geocoder is null (works for
future enrichers). events.json contract unchanged (additions under _enriched only).
conftest isolate_enrichment_cache also redirects NWIS_CACHE_DB_PATH off the prod
path (unprivileged-user test isolation). Adds tests/test_nwis_enrichment.py (28
tests: parse, band edges incl P0/P9/P10/P75/P90/record, percentile interpolation,
cache hit/miss/expire, adapter enrich + graceful-null + cache-hit-no-refetch,
summary rendering per band).
Full suite: 710 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #6 of the v0.7.x GUI rework arc, and its close-out. Production code;
central-gui restart only (templates + a static stylesheet; supervisor untouched).
The v0.7.0-v0.7.4 arc shipped the FEATURES the gui_preview.html mock shows
(chip-pickers, active pills, markercluster + shape/opacity markers, collapsed
grouped legend, real paginator, telemetry tab) but on top of the old CSS
framework. v0.7.5 closes the visual gap: the framework is removed entirely and
replaced by a hand-authored design system that IS the preview (neutral greys, a
single blue accent, 6px radii, 14px type, bordered cards/tables).
- New static/css/central.css: design tokens + element base (forms, buttons,
tables, cards, nav) + every events-feed component, authored so the existing
template class names + element IDs the page JS/HTMX depends on (.chip-picker*,
.events-table, .event-row, .legend-chip, .evt-marker, #fit-to-results,
#map-filter-toggle, ...) render in the preview's language unchanged. Mounted at
/static (already wired in __init__.py); linked from base.html + base_wizard.html.
- base.html / base_wizard.html: framework <link> removed; nav restructured to the
preview shell (.nav / .brand / .nav-links / .logout-btn); flash messages moved
off framework color vars to .flash-* classes.
- events_list.html: 190-line inline <style> deleted (moved to central.css); map
wrapped in .map-container with a floating .map-toolbar (Fit / map-filter toggle),
legend below. HTMX wiring (hx-get/hx-target/hx-push-url), the v0.7.1 chip-picker
JS, the v0.7.2 marker shapes/opacity, and the v0.7.4 base_path split are
untouched.
- Every other page (dashboard, adapters, api-keys, enrichment, streams, login,
change-password, and the full setup wizard) de-framework'd: framework class
names (outline/secondary/contrast/grid/container) renamed to neutral
.btn-*/.cols; no framework default rendering; same tokens/typography/forms/tables.
- /events.json + the v0.7.1 registry-index adapter palette + v0.7.2 severity
opacity + v0.7.4 stable cross-tab chip colors are all unchanged -- this is a
visual shell swap, not a behavior or data change.
grep -rn pico over templates + static returns zero. All GUI routes return 200
(authed) / 302 (unauthed); no 5xx. Full suite: 682 passed, 1 skipped (central and
unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 of the v0.7.x GUI rework arc. Production code; central-gui restart only.
- Adapter legend: collapsed by default ("{n} adapters · Show legend ▾"). Expands
to domain-grouped chips (same grouping as the v0.7.1 chip-picker) with uniform
ellipsis-truncated names + full-name title tooltips. Clicking a legend chip
toggles that adapter's filter (reuses the chip-picker's hidden CSV via
syncField), so the legend doubles as a filter affordance.
- Row stability: time cell is single-line MM-DD HH:MM UTC (year dropped from the
cell; full ISO in the cell tooltip + a new Time row in the expanded detail).
Adapter cell is a chip (color swatch + short name; display_name is the
tooltip). table-layout:fixed + per-column widths + fixed 37px row height with
nowrap/ellipsis cells -> no per-row wrap variation.
- Real paginator: _fetch_events offset-mode returns the exact page slice plus the
grand total via count(*) OVER() in one roundtrip. Previous/Next + windowed page
numbers (1 ... 4 5 [6] 7 8 ... 47) + "showing X-Y of N" + a 25/50/100/250
per-page selector. URL state persists offset + limit. events.json keeps cursor
pagination (back-compat): offset param presence selects offset-mode, its
absence keeps the cursor path -- cleanly separable by endpoint.
Adds TestEventsPagination (12 tests: offset/limit parse incl. max 250,
offset-vs-cursor query shape, _build_pagination windowing). Updates the time
format + adapter-cell + pagination-mode assertions in the existing frontend
tests to the new contract.
Full suite: 674 passed, 1 skipped (central and unprivileged zvx). count(*) OVER()
is ~7.5ms at current volume; vanilla JS + HTMX; CSS functional-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3 of the v0.7.x GUI rework arc. Makes the /events Leaflet map readable and
intentional. Production code; central-gui restart only (no adapter change).
- Fit-to-results default: the map now fits the actual event distribution on
load (previously disabled -> fixed global zoom-4). Empty result set falls
back to the CONUS setView (no crash). Re-fits after each HTMX swap, but only
when the map-filter toggle is OFF (when ON the viewport drives the bbox, so
re-fitting would fight/loop the filter).
- leaflet.markercluster (1.5.3, via CDN): point markers cluster into numbered
badges (disableClusteringAtZoom=9, showCoverageOnHover=false,
spiderfyOnMaxZoom=true). markercluster supports point markers only, so
polygons/lines render in a separate un-clustered featureGroup; fit unions both.
- Map-filter toggle ("Filter table by map view"), default OFF. When off the
table shows all filter-matching events regardless of map zoom; the backend
ignores region_* unless map_filter is set (guards bookmarked URLs too). URL
carries map_filter=1 only when on (hidden input disabled otherwise).
- Per-event_type marker shape (derived event_type = first category segment):
circle = quake/hydro/space (points), square = fire (areas),
triangle = wx (NWS alerts/warnings), star = disaster (GDACS/EONET).
Rendered as divIcon + CSS clip-path; point markers switched from circleMarker
to L.marker(divIcon) (also required for markercluster compatibility).
- Per-severity opacity: critical(4)=1.0, high(3)=0.85, moderate(2)=0.7,
low(1)=0.5, unknown(0/NULL)=0.4. Needed adding severity to the _fetch_events
SELECT + event dict (row.get for mock-tolerance) + a data-severity row attr.
Adds 4 tests (map_filter gating on/off, bbox reaches query only when on,
severity in SELECT); updates test_events_bbox_guard for the new toggle contract.
Full suite: 662 passed, 1 skipped (central and unprivileged zvx). Vanilla JS +
HTMX + Leaflet/markercluster; CSS functional-only (polish deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Biggest PR of the v0.7.x GUI rework arc. Replaces the single-select /events
filter row with a multi-select, URL-addressable filtering surface.
- Search: full-width box, debounced 300ms, server-side ILIKE over the inner
adapter payload (covers the derived subject + location); parameterized with
LIKE wildcards escaped (ESCAPE '\'). Injection-safe.
- Adapter / Category / Event Type / Severity: multi-select chip-pickers (shared
_chip_picker.html macro). Adapter is grouped by domain with color swatches and
an in-panel search. Backend uses `= ANY(...)`. URL state is comma-separated.
- Event Type is derived as split_part(category,'.',1) (no event_type column yet;
a stand-in until the v0.8 canonical schema). Severity maps labels to the
numeric scale (4=critical..1=low, 0/NULL=unknown).
- Time: preset dropdown (15m/1h/6h/24h/7d/active/all) + custom from/to range,
encoded in a single `time` token. GUI defaults to last_24h; events.json keeps
its single-value adapter/since/until contract (no default).
- Active pills: server-rendered from parsed state, updated out-of-band on each
HTMX swap; each x clears that filter and re-submits.
- URL state persistence: every filter in the query string; /events/rows sets
HX-Push-Url to the /events?... full-page URL so bookmarking/back-forward work.
Filter options are rendered server-side at page load (DISTINCT category +
split_part, registry adapters, severity enum) -- no new AJAX endpoints.
Vanilla JS + HTMX (no framework added). CSS is functional-only; visual polish
is deferred to a later pass per the rework plan.
Adds TestEventsFiltering (24 tests: multi-value parse, ILIKE injection safety,
time-preset resolution with injected clock, severity/NULL handling, active-pill
descriptors, URL round-trip). Updates four TestEventsFeedFrontend assertions to
the new filter_state/adapters contract.
Full suite: 658 passed, 1 skipped (central and unprivileged zvx). No adapter
base class change -> central-gui restart only (no supervisor restart).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events_json SELECT read payload->>'subject', but the CloudEvents
envelope has no top-level subject, so every JSON consumer saw
subject: null. The /events GUI already derives readable subjects via
per-adapter templates/_event_summaries/{adapter}.html (PR L-c).
This makes the JSON path produce the same plain-text subjects with no
duplicated logic: _derive_subject(event) renders the same partial the
table uses (falling back to _default.html) and html.unescapes the
autoescaped output so JSON consumers get plain text (e.g. ">=1 MeV"
rather than the escaped ">=1 MeV"). _fetch_events now sets subject
from it and drops the always-null SQL expression. The GUI Subject cell
is unchanged.
Adds TestEventsJsonSubject (parameterized over discover_adapters(), no
hardcoded list): non-null subject per adapter, equality with the rendered
partial, pinned human text for the deterministic adapters, swpc_alerts
truncation, and null fallbacks. Updates one TestEventRowDataAttributes
assertion that pinned the old SQL pass-through contract.
One route change plus tests; central-gui restart required.
Full suite: 629 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleans up unused imports and dead locals flagged by ruff in the test
files PR #50 (M-b) touched. Tests-only; no production code, no service
restart.
- test_supervisor_hotreload.py: drop unused AsyncMock/patch imports,
dead expected_wait/expected_next_poll locals, and two dead
state = AdapterState(...) blocks plus their now-orphaned local imports
- test_supervisor_integration.py: drop unused asyncio/patch/pytest_asyncio
imports and AdapterState from two function-local imports
ruff tests/ 92 -> 82 (the 4 named files now 0; all other files unchanged).
Full suite: 590 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coordless events such as removal tombstones with null lat/lon, from adapters that declare enrichment_locations, previously fell off the loop without writing _enriched and carried no geocoder bundle at all, violating the every-event-carries-_enriched design rule. Add a post-loop fallback that resolves the null location to an all-null bundle per enricher. Adapters with no enrichment_locations remain skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First half of the split PR L (events-tab + map deferred to L-b). Only FIRMS
declared enrichment_locations, so apply_enrichment silently bypassed every
other adapter. This declares it for all 12.
Pre-read finding (resolved per PM): apply_enrichment is a FLAT lookup
(event.data.get(lat_path)); FIRMS/usgs_quake already carry top-level
latitude/longitude in event.data, but the other point adapters kept coords
only in Geo.centroid where the flat path can't reach them. Per PM (option b),
the 5 centroid-only adapters now also write top-level latitude/longitude into
event.data, mirroring their existing Geo.centroid (lon, lat) — 2-3 lines each,
no framework refactor. Geo retained for existing rendering uses.
Declarations (verbatim):
firms [("latitude","longitude")] (unchanged)
usgs_quake [("latitude","longitude")] (already top-level in data)
nwis [("latitude","longitude")] + centroid mirror
eonet [("latitude","longitude")] + centroid mirror
gdacs [("latitude","longitude")] + centroid mirror
wfigs_incidents [("latitude","longitude")] + centroid mirror (inline data)
inciweb [("latitude","longitude")] + centroid mirror (inline data)
wfigs_perimeters [] # polygons, no point
nws [] # forecast zones/counties, no point
swpc_alerts [] # space weather, no coordinate
swpc_kindex [] # space weather, no coordinate
swpc_protons [] # space weather, no coordinate
Centroid mirror is `latitude = centroid[1]; longitude = centroid[0]` (centroid
is GeoJSON (lon, lat)); guarded on centroid presence so coordinate-less events
get no lat/lon keys (apply_enrichment then skips them).
map_render_kind concept dropped — the existing /events map is already
geometry-kind-agnostic (renders any row's data-geometry via L.geoJSON), so it
was unnecessary. Events-tab enhancements are PR L-b.
Tests (test_enrichment_locations_coverage.py, 6, all registry-derived):
- every adapter explicitly declares enrichment_locations in its own class body
- declarations are valid list[(str,str)]
- point adapters all use the canonical ("latitude","longitude") paths
- >=5 point adapters are non-empty (regression guard)
- synthetic-event builders prove the keys resolve: usgs_quake._feature_to_event
and nwis._build_event (the two adapters with isolated builders; the four
inline-build adapters are covered by the post-merge live smoke).
Verification: full pytest 552 passed, 1 skipped (was 546; +6). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src empty.
Follow-ups (NOT here): consumer-doc per-adapter _enriched.geocoder notes for
the newly-enriched adapters belong in L-b's doc pass; live end-to-end smoke
runs post-merge (USGS quake + one other) per the acceptance bar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during the 2026-05-20 NaviBackend activation: toggling
config.enrichment.backend_class to NoOpBackend while backend_settings still
held {"base_url": ...} crashed _rebuild_enrichers with
`TypeError: NoOpBackend() takes no arguments`, BEFORE invalidate() ran. Fixed
by mirroring the SourceAdapter.settings_schema pattern: each backend declares a
Pydantic settings_schema; validation happens at write-time (GUI POST) and
read-time (supervisor). A mismatch is now a clean ValidationError, never a
constructor TypeError.
Backends — each gets a `<Name>BackendSettings(BaseModel, extra="forbid")` +
`settings_schema` class attr, mirroring __init__ defaults EXACTLY (note:
timeout_s stays 10.0 — the brief's "5.0" was a transcription slip; preserve the
production default):
NoOpBackend -> NoOpBackendSettings (no fields)
NaviBackend -> NaviBackendSettings (base_url, timeout_s, headers, warmup)
PhotonBackend -> PhotonBackendSettings (base_url, timeout_s, headers)
NominatimBackend-> NominatimBackendSettings (base_url, user_agent, rate_limit_per_sec, timeout_s)
GeocoderBackend Protocol (in geocoder.py, where the base actually lives — not
base.py, which only has Enricher) gains `settings_schema: type[BaseModel]`.
supervisor:
- build_enrichers validates backend_cls.settings_schema.model_validate(
backend_settings) before instantiating, and constructs from the validated
.model_dump(). ValidationError (not TypeError) on mismatch.
- _rebuild_enrichers builds into locals and commits to instance state only on
success — a ValidationError leaves the previously-active enrichers/config/
cache untouched.
- _handle_enrichment_change wraps the rebuild in try/except ValidationError:
logs and returns, keeping the previous backend running (supervisor stays up;
operator fixes the row; next NOTIFY applies cleanly). No cache invalidation
on a failed change.
GUI /enrichment:
- GET skips the outer EnrichmentConfig.backend_settings field and renders a
separate <fieldset> from describe_fields(backend_cls.settings_schema, ...)
for the row's current backend_class. Backend fields namespaced bs_<name>.
- POST reassembles bs_<name> inputs into a backend_settings dict, validates it
against the SUBMITTED backend_class's schema (so errors attach to the right
fields when an operator is mid-switch), then validates the outer
EnrichmentConfig. DB row written only if both pass; otherwise re-renders with
field-level errors against the submitted backend.
- backend_class stays a plain text field (no <select>, no client-side reshape).
form_descriptors: generic `float -> "number"` widget (2 lines, mirrors K.5's
`dict -> "json"`), needed because backend schemas have float fields
(timeout_s, rate_limit_per_sec). Benefits any float field codebase-wide.
DB schema unchanged: backend_settings stays JSONB; validation moved to
use-site. _BACKEND_REGISTRY / _ENRICHER_REGISTRY unchanged beyond schema lookup.
Tests (test_backend_settings_schema.py, 11): schemas exist + extra='forbid';
Navi schema preserves defaults (timeout_s == 10.0); NoOp has zero fields;
build_enrichers raises ValidationError-not-TypeError for the exact 2026-05-20
case; supervisor keeps previous backend on a bad NOTIFY (the incident
scenario); valid NoOp-with-empty-settings applies + invalidates; GUI POST
rejects bad backend_settings without writing + re-renders against submitted
backend; GUI POST writes on valid settings. test_enrichment_config_plumbing
updated for the new context shape (outer_fields/backend_fields).
Verification: full pytest 546 passed, 1 skipped (was 535; +11). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src both
empty.
Does NOT touch PR L scope (events tab, remaining adapter enrichment_locations),
the DB schema, or the registries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
