The v0.10.5 ephemeral pull-consumer used DeliverPolicy.BY_START_TIME with
no upper bound, so every republish satisfied the same time filter and
the consumer fetched its own output back -- an unbounded loop bounded
only by the per-stream cap. Operator-triggered 5-minute resend on
2026-06-07 ran the loop long enough to time out central-gui's POST and
the host went down with it.
Fix: snapshot each event-bearing stream's last_seq up front via a new
_snapshot_last_seqs() helper, pass it to _iter_window as max_stream_seq,
and exit the generator the first time msg.metadata.sequence.stream
exceeds the snapshot. Pull-consumer delivery is stream-seq ascending so
one boundary check suffices.
Also drop _MAX_MSGS_PER_STREAM 50_000 -> 5_000 and add a WARNING log
when the cap is hit -- a legitimate operator window should never reach
it, and silent truncation hid the v0.10.5 loop until the host fell over.
Two regression tests cover the new behavior: one stages pre/post-snapshot
batches and asserts the post-snapshot batch is never yielded; one
overwhelms the cap and asserts the warning fires.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First official-state-DOT-API pattern landing. Two adapters in one PR:
- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
to LineString via the polyline lib (bookend LineString or Point fallback);
ITD Severity string mapped None->1 / Minor->2 / Major->3 with
IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
Restrictions / DetourPolyline pass through unmodified. Advisories ship
as structural pass-through under data.advisory since the upstream
/alerts endpoint currently returns []; per-record try/except keeps a
surprise shape from sinking the cycle when ITD posts its first one.
- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
camera per UTC day; image URL passes straight through to <img src>.
Region uniform US-ID with data.source_jurisdiction preserving the raw
upstream Source field for the ~1.2% cross-DOT border-region mirrors
(UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).
Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.
Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.
Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.
Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FIRMS hotspots rendered as tiny dots on /events because the adapter shipped
`bbox=(lon, lat, lon, lat)` and no `geometry`. `_build_geom_sql` collapsed
that into a 5-vertex zero-area `ST_Polygon`. Fix: construct a real rectangular
GeoJSON Polygon from `(latitude, longitude, scan, track)` per record and ship
it via `Geo.geometry`, mirroring the v0.9.3 pattern used by `tomtom_flow`.
- `_pixel_polygon()` builds a CCW 4-corner ring with the closing vertex
duplicated. Cross-track (longitude) extent scales by `cos(lat)`. Latitude
is pole-clamped to ±89° defensively.
- `_row_to_event` drops the degenerate bbox and sets `geo.geometry`;
centroid stays alongside for consumers that read centroid only.
- `poll()` emits a single per-cycle warn if any record fell back to
centroid-only Geo (missing scan/track).
- Forward-only at the geometry layer: same dedup keys, new payload shape
for new records only. Existing PostGIS rows retain their degenerate
polygons until republished or aged out (48h dedup window).
- Side-quest: removed a pre-existing dead RegionConfig import in
tests/test_firms.py since the imports block was touched.
Tests: 5 new `_pixel_polygon` unit tests + 2 archive round-trip regression
guards mirroring `test_archive_prefers_geo_geometry`. Full suite 930 passed
/ 1 skipped under both `central` and `zvx` users.
CAVEAT — axis mapping convention: the `_pixel_polygon` docstring describes
`scan` as the along-track (latitude) extent and `track` as the cross-track
(longitude) extent. This matches the v0.9.21 prompt and is internally
consistent. NASA FIRMS convention may have these inverted — `scan` is often
documented as the cross-track scan width and `track` as the along-track
extent. If deploy-time visual inspection shows polygons rotated 90° from
expected, swap the assignments inside `_pixel_polygon`:
half_scan_deg → multiply by cos(lat) (treat as longitude offset)
half_track_deg → divide by 111.0 only (treat as latitude offset)
Clickability is the first-order goal and is unaffected — the polygon will
still have non-zero area in both axes either way; only the aspect ratio
flips. Tracked as a fast-follow if visual inspection requires it.
PR #84.
* v0.9.20: regional subject routing on quake / fire / hydro / disaster adapters
Adds regional subject tokens to four adapters that previously published
without location-based routing:
- usgs_quake: central.quake.event.<tier>.us.<state> (US) or .<country> (intl)
- firms: central.fire.hotspot.<sat>.<conf>.us.<state> or .<country>
- nwis: central.hydro.<param>.<agency>.<site>.us.<state> (always US)
- eonet: central.disaster.eonet.<cat>.<country> (replaces hardcoded .global)
The us.<state> pattern (two tokens for US events) matches the NWS precedent
and resolves the ISO-2 collision between Idaho (id) and Indonesia.
New shared helper module: src/central/adapters/_subject_helpers.py
- US_STATE_NAME_TO_CODE: 50 states + DC + territories
- subject_for_country(): normalized country token
- subject_for_region(): returns us.<state>, <country>, or unknown
gdacs.py refactored to import subject_for_country from shared module.
Fixes: meshai v0.4 Phase C.3 bug (M4.1 Nevada quake routed globally)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* v0.9.20: stale docstring nit on renamed test
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
NWSAdapter.sweep_old_ids ran an unscoped global DELETE FROM published_ids
(no adapter = ? filter), so each per-adapter sweep cycle purged EVERY
adapter dedup rows older than 8 days -- capping dedup memory at ~8d for the
8 adapters whose window exceeds it (eonet/gdacs/nwis 30d; swpc_kindex/
protons/alerts + wfigs_incidents/perimeters 14d). Events that went silent
past 8d then re-listed were re-published as downstream duplicates.
Extract the 3 dedup methods onto the SourceAdapter base (finishing v0.9.19;
is_published/mark_published were byte-identical) and preserve the 8-day
window via dedup_sweep_days = 8, so the inherited sweep scopes to adapter=?.
Effect: each adapter retains dedup to its configured window -> fewer
downstream duplicates. Retention-window scoping, NOT a dedup-forward-only
change (no event-shape change, no migration/backfill).
Adds a regression guard asserting a foreign adapter row survives an nws
sweep. Suite 900->901/1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrations 025-029 (wzdx + traffic-family adapters/streams) were applied
out-of-band via direct psql during their deploys and never recorded in
schema_migrations. Result: a fresh restore would replay them, and an audit of
the tracking table understated what was actually live. All five are pure
additive INSERT ... ON CONFLICT DO NOTHING (verified idempotent by inspection),
so they were back-filled directly into schema_migrations (5 rows, applied_at =
now() = the reconciliation event; the original out-of-band apply dates are
unrecoverable and are documented as such rather than guessed).
Adds `central-migrate --check` to catch this class of drift going forward:
- find_drift(): pure function comparing schema_migrations rows vs *.sql files,
returning (untracked, orphan). Unit-tested, no DB dependency.
- check_drift(): CI-assertion form -- exit 1 if any file is untracked, else 0.
- log_drift(): WARNs per drifted entry, called on every migrate run too.
No migration 031 for the v0.9.17 wzdx state default: that default lives in
adapter code (_read_states falls back to _DEFAULT_STATES), so the live row's
explicit 7-state array is behaviorally identical to a fresh-install null.
Materializing it into SQL would freeze a snapshot of _DEFAULT_STATES and create
a new drift vector. Deferred to a future "show effective defaults" UI PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Narrow which WZDx registry feeds are fetched at poll time, so out-of-state
feeds are never requested (no bandwidth/quota cost), rather than fetching
nationally and dropping out-of-bounds events at the archive INSERT (v0.9.12).
- states is now list[StateCode] (Literal of 56 canonical 2-letter codes),
default ["ID","WA","OR","NV","UT","WY","MT"]. Semantic flip: empty/unset/null
now means the Idaho-region default, NOT "every eligible feed". Poll nationally
by selecting all states in the GUI.
- list[Literal] renders as the existing checkboxes multi-select widget (firms.py
precedent); pydantic rejects malformed codes at save time.
- Add informational quota_estimate (cap=None: never warns/blocks) so the edit
page shows the fetch-volume reduction.
- adapters_edit.html: standalone quota panel for flat-config adapters, guarded
by not has_model_list so model_list adapters don't double-render; reuses the
identical quota.* keys + CSS so the v0.9.15 client recompute works for them too.
- 7 new tests (default/null fallback, _discover filtering, malformed rejection,
quota math, offline Jinja panel render).
Event data shape is unchanged (poll-behavior-only), so no published_ids reset
is needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mechanical lint fixes only — ruff 31 -> 0 across the two files.
- E402 module-import-not-at-top: 15 -> 0 (relocated module-level
`logger = logging.getLogger(...)` from mid-import to below the import
block in routes.py; imports now contiguous at top of file)
- F811 redefined-while-unused: 10 -> 0 (removed 10 redundant function-local
`from central.gui.csrf import reuse_or_generate_pre_auth_csrf` re-imports;
the module-level import at line 26 is now load-bearing)
- F401 unused-import: 4 -> 0 (routes.py: fastapi.Depends, and
central.gui.csrf.reuse_or_generate_pre_auth_csrf resolved by the F811 fix;
test_wfigs.py: sqlite3, central.config_models.RegionConfig)
- E702 multiple-statements-on-one-line-semicolon: 2 -> 0 (split the two
semicolon-joined statements in _fused_bbox, indentation preserved)
Deliberate function-local wizard imports (circular-import workaround) left
untouched. pytest: 890 passed / 1 skipped, unchanged across 3 runs.
no behavior change; ruff mechanical fixes only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror TomTomIncidentsAdapter.quota_estimate in client-side JS so the
quota panel updates live as the operator edits a per-row cadence or the
adapter cadence, and on Add/Delete bbox -- instead of only on Save.
- tomtom_incidents.py: expose seconds_per_month + default_cadence_s in
the quota dict (no JS magic numbers).
- model_list.html: data-quota-* attrs + .quota-detail/.quota-msg spans;
self-contained DOMContentLoaded-guarded IIFE applying the same
max(adapter_cadence, bbox_cadence||default) floor and 80%/100%
warn/block thresholds. No-op when no quota panel present.
- tests: structural asserts for data-attrs + span hooks (JS exec needs
a browser; live behaviour eyeballed manually).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs each live WFIGS perimeter with its nearby/contemporaneous FIRMS hotspots
into a single "fire" on the /events map. FIRMS hotspots carry no IrwinID, so the
link is spatial+temporal: a hotspot is confirmed (part of a known fire) when it
lies within 1km of a perimeter AND within 72h of it; hotspots matching no
perimeter render amber as "unconfirmed" -- a possible new fire detected by
satellite before an official perimeter exists (early-warning signal).
- routes.py: read-only /events/fire-fused.json (PostGIS ST_DWithin geography join)
- events_list.html: "Fuse fire layers" toggle (default on); centroid fire glyph
that expands to polygon + hotspot dots on click; amber unconfirmed hotspots
- central.css: --fire-confirmed / --fire-unconfirmed vars (retune without code)
- 11 tests (shaping, bbox parse, R/T + bbox param wiring); spatial correctness
verified on prod (Summit Creek: perimeter + 90 hotspots; 191 unconfirmed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hourly supervisor sweep deletes archived events older than their stream's
config.streams.max_age_s (the /streams 1/7/14/30/365d buttons -- the per-stream
source-of-truth). Explicit STREAM_CATEGORY_DOMAINS map (category domains do not
equal stream subject domains for the traffic family); fail-safe skip on
missing/<=0 max_age_s; unmapped-domain warning so a new adapter cannot silently
dodge retention.
TimescaleDB hypertable gotcha: events.ctid is chunk-local, NOT globally unique --
DELETE batching keys on the composite PK (id, time). See the inline NOTE in
config_store.delete_events_older_than and the PR body for the incident write-up.
- config_store: delete_events_older_than (batched on (id,time)) + unmapped_event_domains
- supervisor: STREAM_CATEGORY_DOMAINS, _sweep_events_retention, _events_retention_loop
- streams_list.html: max-age also governs archived-events retention
- 9 tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a single archive-level geographic filter at the events INSERT path: events
whose geometry falls entirely outside a system-configured monitoring area are
dropped before archival. Null-geom events (SWPC trio, .removed tombstones) are
always kept. Uses a pure shapely intersects() predicate so border-straddlers
are kept (matches ST_Intersects), and is fail-open on unparseable geometry.
- config.system gains monitor_{north,south,east,west} (migration 030, Idaho default)
- archive refreshes the bbox every 60s (no restart needed to change it); adds a
per-adapter dropped-count counter, debug log per drop, cumulative INFO rollup
- new GUI editor at /monitoring-area (Leaflet draggable rectangle, N/S/E/W inputs)
- no adapter code changes; well-behaved adapters keep upstream API filtering
- 12 tests covering all five verdicts, drop-and-count, border/point-on-edge, fail-open
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /events feed was dominated by *.removed tombstone events (audit records
for features dropped from upstream feeds), burying geometry-bearing events
like fire perimeters (wfigs_perimeters: 54 real perimeters vs 1015 tombstones).
The GUI now default-hides any event whose category ends in .removed, with a
"Show removed" checkbox to restore them; URL state is preserved (HX-Push-Url)
so a shared link shows what the sharer saw. events.json is unchanged (still
returns tombstones) so API consumers are unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a passive Leaflet map to the generic model_list editor that renders
every bbox row as a labeled translucent rectangle, auto-detected via the
min/max lon+lat sub-fields (TomTom incidents). Read-only: no drag/draw,
precise tuning stays in the per-row coord inputs. Rectangles redraw live
on input/add/remove; viewport fits only on initial render so typing never
jumps the map. Non-bbox model_lists (StateConfig, TileCoord) are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes a shared form_descriptors 500 (NotImplementedError: unsupported list
type) that broke the Edit page for ALL FOUR adapters whose settings carry a
list[<BaseModel>] field: tomtom_incidents, tomtom_flow, state_511_atis,
state_511_atis_cameras.
- form_descriptors: list[BaseModel] -> generic "model_list" widget with
recursive per-column sub_field descriptors.
- New _partials/model_list.html: vanilla-JS repeatable-row editor
(add/remove/renumber), driven entirely by sub_fields (no adapter-name
branching). Single-region edit pages render byte-identically.
- TomTom: BBox/Settings Pydantic validators (10,000 km^2 cap, coord ranges,
min<max, cadence_s>=60, unique names) as the single source of truth
(enforced at supervisor load AND GUI POST). Duck-typed quota_estimate hook
+ read-only quota panel; POST hard-blocks estimates over the 2,500/mo free
tier (422). TOMTOM_FREE_TIER_CALLS_PER_MONTH is a tunable for paid tiers.
- routes: model_list form parse, row-aware ValidationError messages, 422 for
model_list failures (single-region region errors still re-render at 200).
- tests: 11 new (real-Jinja render across 3 adapters + byte-identical nws
no-regression guard, POST persist + oversized/degenerate/duplicate/cadence/
quota 422 matrix, quota estimate). Full suite 848 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plumb the upstream GeoJSON Polygon/MultiPolygon into Geo.geometry so the
archive renders the real fire-perimeter shape (ST_GeomFromGeoJSON) instead
of the bbox 4-corner fallback. Adapter-local change; the v0.9.3 Geo.geometry
framework already carries it end-to-end. Graceful null + dict-or-string
coercion via _as_geometry; fall-off removal events stay NULL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Castle Rock caps each DataTables page at 100 rows regardless of `length`, so the
single-request fetcher silently dropped rows on any layer over 100 (confirmed
live: Construction recordsFiltered=114, returned 100 -> 14 rows invisible).
Backports the v0.9.6 cameras pagination loop into _fetch_details.
- _LIST_PAGE_LENGTH 1000 -> 100; new _list_body(start) builder; new @retry
_fetch_page(base_url, layer, start).
- _fetch_details loops start+=100 until recordsFiltered collected or an empty
page, with a _MAX_PAGES=50 ceiling that warns+breaks. Mid-pagination failure
returns rows-so-far (retried next poll).
- Incidents (1) / Closures (29) are under 100 today but pagination applies
uniformly; future-proof.
central-supervisor restart only (no stream, migration, template, or dep change).
Full suite: 833 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).
supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.
- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
cams). The 100-row page cap also means v0.9.2 state_511_atis silently
truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
-- always shows today's cameras, no per-poll flooding, no retention
coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).
Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.
- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
falls back to the adapter's default_cadence_s. Existing settings without the
field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
(one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).
central-supervisor restart only. No gui/archive restart, no migration, no new dep.
Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.
central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.
- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
state_511_atis; cross-source overlap is by design = additive coverage, distinct
dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
key redacted from logs; poll skips cleanly without a key.
Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phased PR2 of v0.9.3. Adds the on-demand tile passthrough so Navi can call
Central instead of navi-traffic, and so tiles outside the polled coverage set
still get persisted. central-gui restart only (new route); no supervisor, no
archive, no migration, no new stream/dep.
GET /api/traffic/flow/<z>/<x>/<y>.{png,pbf} -- exactly matches navi-traffic's
route for a drop-in frontend flip (s/navi-traffic/central:8000/). No auth
(tailnet-trusted, matches navi-traffic's existing posture; exempted in
middleware). Cache-Control max-age=60 (TomTom's advertised tile TTL).
- .png -> passthrough + cache only (raster unparseable for storage).
- .pbf -> passthrough + decode via the shared tomtom_flow_parse.decode_flow_tile
and publish segments to CENTRAL_TRAFFIC_FLOW via the GUI's NATS connection
(wrap_event, supervisor's exact envelope shape). Same minute-bucketed dedup id
as the polling adapter, so adapter+passthrough fetches of one tile in the same
minute collapse at the archive's (id, time) upsert -- intentional collision.
- Publish is best-effort: wrapped in try/except + guards get_js() None, so NATS
state never blocks serving the tile (HTTP > storage).
- API key via ConfigStore(get_pool()).get_api_key("tomtom"); 503 if unset;
redacted from any logged upstream error; 502 on upstream failure (no publish).
Single-flight (shared in-flight upstream request per tile) queued for v0.9.5.
navi-frontend flip + navi-traffic deprecation live in the navi repo (Matt's call).
Full suite: 787 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.
Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.
- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
transitive that uv re-lock would otherwise prune).
Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.
Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.
Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.
Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.
Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.
Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui
restart (adapter base change + template change). No migration, no new stream.
(a) Work-zone subject + detail no longer leak vehicle direction "unknown"
(common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both
wzdx partials. Was "Work zone on MORELAND ST unknown".
(b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline
copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days
class attr parameterizes the retention window (NWIS=30, default=14). Inline
copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a
future cleanup. Net dedup code down ~52 lines.
Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens the v0.8.x data-quality cleanup arc. Production code; central-gui AND
central-supervisor restart (adapter contract + enrichment behavior change).
NWIS events rendered as a bare "Water reading: 111 ft3/s" with an empty Location
column -- an operator couldn't tell where the gauge is or whether 111 ft3/s is
drought-low, normal, or near-flood. Coordinates were present but the reverse
geocoder returns null city/state/county for rural gauge points, and USGS site +
percentile data was never fetched. v0.8.0 fetches both.
Approach B (adapter-owned, per the proposal decision): the NWIS adapter -- which
already owns the USGS APIs -- fetches site metadata and daily stats itself and
writes two provenance bundles under event.data["_enriched"]:
- usgs_site {name, lat, lon, state, county} from the OGC monitoring-locations
item-by-id (the API family the adapter already speaks; JSON, no RDB parser).
- usgs_stats {value, percentile, class_label, severity_band, p10..p90, record_max,
count, period} from the legacy RDB daily-statistics service (the OGC API has no
stats endpoint). USGS percentiles are % of days at-or-below, so higher = higher
flow; classified to the WaterWatch bands -> severity 0-4 (record=4, much
above/below=3, above/below=2, normal=1; None reserved for "no stats", distinct
from a normal-flow gauge). Severity is set on the event, so it drives the v0.7.1
severity chip-picker filter + v0.7.2 map-marker opacity.
- new nwis_enrich.py: pure parse/classify/percentile/band helpers + a sqlite
SiteStatsCache (site TTL 365d, stats TTL 90d -- one fetch per site+param serves
every reading for the window, so a warm cache makes zero USGS calls). USGS down
-> cached-if-present else all-null bundle; the event still publishes.
Framework: the single agreed generic change -- supervisor apply_enrichment now
MERGES into _enriched instead of overwriting, so the still-global geocoder phase
doesn't clobber the adapter's bundles. No other adapter writes _enriched, so this
is inert for them.
GUI: _event_summaries/nwis.html -> "<site> -- <value> <units> (<band>, <Nth>
percentile)", with graceful fallback to "<site> -- <value>" then the bare
"Water reading:". _event_rows/nwis.html detail gains site/normalcy/typical/location
rows. _events_rows.html Location column falls back generically to any
_enriched.<source> carrying state/county when the geocoder is null (works for
future enrichers). events.json contract unchanged (additions under _enriched only).
conftest isolate_enrichment_cache also redirects NWIS_CACHE_DB_PATH off the prod
path (unprivileged-user test isolation). Adds tests/test_nwis_enrichment.py (28
tests: parse, band edges incl P0/P9/P10/P75/P90/record, percentile interpolation,
cache hit/miss/expire, adapter enrich + graceful-null + cache-hit-no-refetch,
summary rendering per band).
Full suite: 710 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 of the v0.7.x GUI rework arc. Production code; central-gui restart only.
- Adapter legend: collapsed by default ("{n} adapters · Show legend ▾"). Expands
to domain-grouped chips (same grouping as the v0.7.1 chip-picker) with uniform
ellipsis-truncated names + full-name title tooltips. Clicking a legend chip
toggles that adapter's filter (reuses the chip-picker's hidden CSV via
syncField), so the legend doubles as a filter affordance.
- Row stability: time cell is single-line MM-DD HH:MM UTC (year dropped from the
cell; full ISO in the cell tooltip + a new Time row in the expanded detail).
Adapter cell is a chip (color swatch + short name; display_name is the
tooltip). table-layout:fixed + per-column widths + fixed 37px row height with
nowrap/ellipsis cells -> no per-row wrap variation.
- Real paginator: _fetch_events offset-mode returns the exact page slice plus the
grand total via count(*) OVER() in one roundtrip. Previous/Next + windowed page
numbers (1 ... 4 5 [6] 7 8 ... 47) + "showing X-Y of N" + a 25/50/100/250
per-page selector. URL state persists offset + limit. events.json keeps cursor
pagination (back-compat): offset param presence selects offset-mode, its
absence keeps the cursor path -- cleanly separable by endpoint.
Adds TestEventsPagination (12 tests: offset/limit parse incl. max 250,
offset-vs-cursor query shape, _build_pagination windowing). Updates the time
format + adapter-cell + pagination-mode assertions in the existing frontend
tests to the new contract.
Full suite: 674 passed, 1 skipped (central and unprivileged zvx). count(*) OVER()
is ~7.5ms at current volume; vanilla JS + HTMX; CSS functional-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3 of the v0.7.x GUI rework arc. Makes the /events Leaflet map readable and
intentional. Production code; central-gui restart only (no adapter change).
- Fit-to-results default: the map now fits the actual event distribution on
load (previously disabled -> fixed global zoom-4). Empty result set falls
back to the CONUS setView (no crash). Re-fits after each HTMX swap, but only
when the map-filter toggle is OFF (when ON the viewport drives the bbox, so
re-fitting would fight/loop the filter).
- leaflet.markercluster (1.5.3, via CDN): point markers cluster into numbered
badges (disableClusteringAtZoom=9, showCoverageOnHover=false,
spiderfyOnMaxZoom=true). markercluster supports point markers only, so
polygons/lines render in a separate un-clustered featureGroup; fit unions both.
- Map-filter toggle ("Filter table by map view"), default OFF. When off the
table shows all filter-matching events regardless of map zoom; the backend
ignores region_* unless map_filter is set (guards bookmarked URLs too). URL
carries map_filter=1 only when on (hidden input disabled otherwise).
- Per-event_type marker shape (derived event_type = first category segment):
circle = quake/hydro/space (points), square = fire (areas),
triangle = wx (NWS alerts/warnings), star = disaster (GDACS/EONET).
Rendered as divIcon + CSS clip-path; point markers switched from circleMarker
to L.marker(divIcon) (also required for markercluster compatibility).
- Per-severity opacity: critical(4)=1.0, high(3)=0.85, moderate(2)=0.7,
low(1)=0.5, unknown(0/NULL)=0.4. Needed adding severity to the _fetch_events
SELECT + event dict (row.get for mock-tolerance) + a data-severity row attr.
Adds 4 tests (map_filter gating on/off, bbox reaches query only when on,
severity in SELECT); updates test_events_bbox_guard for the new toggle contract.
Full suite: 662 passed, 1 skipped (central and unprivileged zvx). Vanilla JS +
HTMX + Leaflet/markercluster; CSS functional-only (polish deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Biggest PR of the v0.7.x GUI rework arc. Replaces the single-select /events
filter row with a multi-select, URL-addressable filtering surface.
- Search: full-width box, debounced 300ms, server-side ILIKE over the inner
adapter payload (covers the derived subject + location); parameterized with
LIKE wildcards escaped (ESCAPE '\'). Injection-safe.
- Adapter / Category / Event Type / Severity: multi-select chip-pickers (shared
_chip_picker.html macro). Adapter is grouped by domain with color swatches and
an in-panel search. Backend uses `= ANY(...)`. URL state is comma-separated.
- Event Type is derived as split_part(category,'.',1) (no event_type column yet;
a stand-in until the v0.8 canonical schema). Severity maps labels to the
numeric scale (4=critical..1=low, 0/NULL=unknown).
- Time: preset dropdown (15m/1h/6h/24h/7d/active/all) + custom from/to range,
encoded in a single `time` token. GUI defaults to last_24h; events.json keeps
its single-value adapter/since/until contract (no default).
- Active pills: server-rendered from parsed state, updated out-of-band on each
HTMX swap; each x clears that filter and re-submits.
- URL state persistence: every filter in the query string; /events/rows sets
HX-Push-Url to the /events?... full-page URL so bookmarking/back-forward work.
Filter options are rendered server-side at page load (DISTINCT category +
split_part, registry adapters, severity enum) -- no new AJAX endpoints.
Vanilla JS + HTMX (no framework added). CSS is functional-only; visual polish
is deferred to a later pass per the rework plan.
Adds TestEventsFiltering (24 tests: multi-value parse, ILIKE injection safety,
time-preset resolution with injected clock, severity/NULL handling, active-pill
descriptors, URL round-trip). Updates four TestEventsFeedFrontend assertions to
the new filter_state/adapters contract.
Full suite: 658 passed, 1 skipped (central and unprivileged zvx). No adapter
base class change -> central-gui restart only (no supervisor restart).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events_json SELECT read payload->>'subject', but the CloudEvents
envelope has no top-level subject, so every JSON consumer saw
subject: null. The /events GUI already derives readable subjects via
per-adapter templates/_event_summaries/{adapter}.html (PR L-c).
This makes the JSON path produce the same plain-text subjects with no
duplicated logic: _derive_subject(event) renders the same partial the
table uses (falling back to _default.html) and html.unescapes the
autoescaped output so JSON consumers get plain text (e.g. ">=1 MeV"
rather than the escaped ">=1 MeV"). _fetch_events now sets subject
from it and drops the always-null SQL expression. The GUI Subject cell
is unchanged.
Adds TestEventsJsonSubject (parameterized over discover_adapters(), no
hardcoded list): non-null subject per adapter, equality with the rendered
partial, pinned human text for the deterministic adapters, swpc_alerts
truncation, and null fallbacks. Updates one TestEventRowDataAttributes
assertion that pinned the old SQL pass-through contract.
One route change plus tests; central-gui restart required.
Full suite: 629 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleans up unused imports and dead locals flagged by ruff in the test
files PR #50 (M-b) touched. Tests-only; no production code, no service
restart.
- test_supervisor_hotreload.py: drop unused AsyncMock/patch imports,
dead expected_wait/expected_next_poll locals, and two dead
state = AdapterState(...) blocks plus their now-orphaned local imports
- test_supervisor_integration.py: drop unused asyncio/patch/pytest_asyncio
imports and AdapterState from two function-local imports
ruff tests/ 92 -> 82 (the 4 named files now 0; all other files unchanged).
Full suite: 590 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coordless events such as removal tombstones with null lat/lon, from adapters that declare enrichment_locations, previously fell off the loop without writing _enriched and carried no geocoder bundle at all, violating the every-event-carries-_enriched design rule. Add a post-loop fallback that resolves the null location to an all-null bundle per enricher. Adapters with no enrichment_locations remain skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First half of the split PR L (events-tab + map deferred to L-b). Only FIRMS
declared enrichment_locations, so apply_enrichment silently bypassed every
other adapter. This declares it for all 12.
Pre-read finding (resolved per PM): apply_enrichment is a FLAT lookup
(event.data.get(lat_path)); FIRMS/usgs_quake already carry top-level
latitude/longitude in event.data, but the other point adapters kept coords
only in Geo.centroid where the flat path can't reach them. Per PM (option b),
the 5 centroid-only adapters now also write top-level latitude/longitude into
event.data, mirroring their existing Geo.centroid (lon, lat) — 2-3 lines each,
no framework refactor. Geo retained for existing rendering uses.
Declarations (verbatim):
firms [("latitude","longitude")] (unchanged)
usgs_quake [("latitude","longitude")] (already top-level in data)
nwis [("latitude","longitude")] + centroid mirror
eonet [("latitude","longitude")] + centroid mirror
gdacs [("latitude","longitude")] + centroid mirror
wfigs_incidents [("latitude","longitude")] + centroid mirror (inline data)
inciweb [("latitude","longitude")] + centroid mirror (inline data)
wfigs_perimeters [] # polygons, no point
nws [] # forecast zones/counties, no point
swpc_alerts [] # space weather, no coordinate
swpc_kindex [] # space weather, no coordinate
swpc_protons [] # space weather, no coordinate
Centroid mirror is `latitude = centroid[1]; longitude = centroid[0]` (centroid
is GeoJSON (lon, lat)); guarded on centroid presence so coordinate-less events
get no lat/lon keys (apply_enrichment then skips them).
map_render_kind concept dropped — the existing /events map is already
geometry-kind-agnostic (renders any row's data-geometry via L.geoJSON), so it
was unnecessary. Events-tab enhancements are PR L-b.
Tests (test_enrichment_locations_coverage.py, 6, all registry-derived):
- every adapter explicitly declares enrichment_locations in its own class body
- declarations are valid list[(str,str)]
- point adapters all use the canonical ("latitude","longitude") paths
- >=5 point adapters are non-empty (regression guard)
- synthetic-event builders prove the keys resolve: usgs_quake._feature_to_event
and nwis._build_event (the two adapters with isolated builders; the four
inline-build adapters are covered by the post-merge live smoke).
Verification: full pytest 552 passed, 1 skipped (was 546; +6). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src empty.
Follow-ups (NOT here): consumer-doc per-adapter _enriched.geocoder notes for
the newly-enriched adapters belong in L-b's doc pass; live end-to-end smoke
runs post-merge (USGS quake + one other) per the acceptance bar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during the 2026-05-20 NaviBackend activation: toggling
config.enrichment.backend_class to NoOpBackend while backend_settings still
held {"base_url": ...} crashed _rebuild_enrichers with
`TypeError: NoOpBackend() takes no arguments`, BEFORE invalidate() ran. Fixed
by mirroring the SourceAdapter.settings_schema pattern: each backend declares a
Pydantic settings_schema; validation happens at write-time (GUI POST) and
read-time (supervisor). A mismatch is now a clean ValidationError, never a
constructor TypeError.
Backends — each gets a `<Name>BackendSettings(BaseModel, extra="forbid")` +
`settings_schema` class attr, mirroring __init__ defaults EXACTLY (note:
timeout_s stays 10.0 — the brief's "5.0" was a transcription slip; preserve the
production default):
NoOpBackend -> NoOpBackendSettings (no fields)
NaviBackend -> NaviBackendSettings (base_url, timeout_s, headers, warmup)
PhotonBackend -> PhotonBackendSettings (base_url, timeout_s, headers)
NominatimBackend-> NominatimBackendSettings (base_url, user_agent, rate_limit_per_sec, timeout_s)
GeocoderBackend Protocol (in geocoder.py, where the base actually lives — not
base.py, which only has Enricher) gains `settings_schema: type[BaseModel]`.
supervisor:
- build_enrichers validates backend_cls.settings_schema.model_validate(
backend_settings) before instantiating, and constructs from the validated
.model_dump(). ValidationError (not TypeError) on mismatch.
- _rebuild_enrichers builds into locals and commits to instance state only on
success — a ValidationError leaves the previously-active enrichers/config/
cache untouched.
- _handle_enrichment_change wraps the rebuild in try/except ValidationError:
logs and returns, keeping the previous backend running (supervisor stays up;
operator fixes the row; next NOTIFY applies cleanly). No cache invalidation
on a failed change.
GUI /enrichment:
- GET skips the outer EnrichmentConfig.backend_settings field and renders a
separate <fieldset> from describe_fields(backend_cls.settings_schema, ...)
for the row's current backend_class. Backend fields namespaced bs_<name>.
- POST reassembles bs_<name> inputs into a backend_settings dict, validates it
against the SUBMITTED backend_class's schema (so errors attach to the right
fields when an operator is mid-switch), then validates the outer
EnrichmentConfig. DB row written only if both pass; otherwise re-renders with
field-level errors against the submitted backend.
- backend_class stays a plain text field (no <select>, no client-side reshape).
form_descriptors: generic `float -> "number"` widget (2 lines, mirrors K.5's
`dict -> "json"`), needed because backend schemas have float fields
(timeout_s, rate_limit_per_sec). Benefits any float field codebase-wide.
DB schema unchanged: backend_settings stays JSONB; validation moved to
use-site. _BACKEND_REGISTRY / _ENRICHER_REGISTRY unchanged beyond schema lookup.
Tests (test_backend_settings_schema.py, 11): schemas exist + extra='forbid';
Navi schema preserves defaults (timeout_s == 10.0); NoOp has zero fields;
build_enrichers raises ValidationError-not-TypeError for the exact 2026-05-20
case; supervisor keeps previous backend on a bad NOTIFY (the incident
scenario); valid NoOp-with-empty-settings applies + invalidates; GUI POST
rejects bad backend_settings without writing + re-renders against submitted
backend; GUI POST writes on valid settings. test_enrichment_config_plumbing
updated for the new context shape (outer_fields/backend_fields).
Verification: full pytest 546 passed, 1 skipped (was 535; +11). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src both
empty.
Does NOT touch PR L scope (events tab, remaining adapter enrichment_locations),
the DB schema, or the registries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of three PRs for v0.5.0 (J: framework; K: real geocoder backends +
doc revisions; L: operator events tab + per-adapter render + events-map fix).
Design pivot: the Phase 2 "no enrichment, upstream verbatim" reading of
Matt's principle is reframed — consumers can't do follow-up lookups, they
only see what's on the wire, so whatever Central doesn't enrich is
effectively missing downstream. Enrichment is now expected. The producer-doc
§2/§10.1 rewrite lands in PR K; this PR builds the framework PR K documents.
New package src/central/enrichment/:
- base.py Enricher Protocol (name + async enrich(location) -> dict).
- geocoder.py GeocoderEnricher + GeocoderBackend Protocol + the locked
GEOCODER_FIELDS set (name, city, county, state, country,
postal_code, timezone, landclass, elevation_m) + all_null_bundle().
- cache.py EnrichmentCache — stdlib sqlite3 off the event loop via
asyncio.to_thread (no async-sqlite dep). Keyed on
(enricher_name, lat_4dp, lon_4dp); per-enricher TTL (24h
default); fresh connection per op (sqlite3 isn't thread-safe
to share). Cache even all-null; never cache backend failures.
- backends/no_op.py NoOpBackend — all-null bundle, the PR J default.
Provenance: enrichment results land under event.data["_enriched"][<name>];
everything else in data stays upstream verbatim.
Wiring:
- adapter.py enrichment_locations: list[tuple[str,str]] = [] class attr.
Empty (default) = publish as-is, no enrichment.
- config_models.py EnrichmentConfig (enricher_class, backend_class,
backend_settings, cache_ttl_s). Read once at startup.
- supervisor.py build_enrichers() + apply_enrichment(); enrichment runs
after dedup, before wrap_event, in the poll loop. Class-name
registries for enricher/backend resolution (PR K extends).
- firms.py enrichment_locations = [("latitude","longitude")] — pilot.
Enrichment config is read once at supervisor startup; hot-reload is out of
scope for PR J (noted in EnrichmentConfig + build_enrichers docstrings).
Tests (16 new):
- test_enrichment_framework.py (9): parent-dir/table init, cache miss->hit,
TTL expiry, 4dp rounding, nearby-coord collapse, concurrent-set single-row,
backend-failure all-null-not-cached (retries), success cached (one backend
call), all-null cached.
- test_geocoder_enricher.py (5): NoOp all-null, field-set == GEOCODER_FIELDS,
null-coords short-circuit (no backend call), name=="geocoder", sequential
same-coords single backend call.
- test_firms.py (+2): enrichment_locations declared + paths resolve to floats
in a real event (structural, not literal); event through supervisor
apply_enrichment emerges with data._enriched.geocoder == all-null bundle.
Verification: full pytest 495 passed (was 479; +16). grep for
subject_for_event/_ADAPTER_REGISTRY clean. Module imports cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
