Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui
restart (adapter base change + template change). No migration, no new stream.
(a) Work-zone subject + detail no longer leak vehicle direction "unknown"
(common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both
wzdx partials. Was "Work zone on MORELAND ST unknown".
(b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline
copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days
class attr parameterizes the retention window (NWIS=30, default=14). Inline
copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a
future cleanup. Net dedup code down ~52 lines.
Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens the v0.8.x data-quality cleanup arc. Production code; central-gui AND
central-supervisor restart (adapter contract + enrichment behavior change).
NWIS events rendered as a bare "Water reading: 111 ft3/s" with an empty Location
column -- an operator couldn't tell where the gauge is or whether 111 ft3/s is
drought-low, normal, or near-flood. Coordinates were present but the reverse
geocoder returns null city/state/county for rural gauge points, and USGS site +
percentile data was never fetched. v0.8.0 fetches both.
Approach B (adapter-owned, per the proposal decision): the NWIS adapter -- which
already owns the USGS APIs -- fetches site metadata and daily stats itself and
writes two provenance bundles under event.data["_enriched"]:
- usgs_site {name, lat, lon, state, county} from the OGC monitoring-locations
item-by-id (the API family the adapter already speaks; JSON, no RDB parser).
- usgs_stats {value, percentile, class_label, severity_band, p10..p90, record_max,
count, period} from the legacy RDB daily-statistics service (the OGC API has no
stats endpoint). USGS percentiles are % of days at-or-below, so higher = higher
flow; classified to the WaterWatch bands -> severity 0-4 (record=4, much
above/below=3, above/below=2, normal=1; None reserved for "no stats", distinct
from a normal-flow gauge). Severity is set on the event, so it drives the v0.7.1
severity chip-picker filter + v0.7.2 map-marker opacity.
- new nwis_enrich.py: pure parse/classify/percentile/band helpers + a sqlite
SiteStatsCache (site TTL 365d, stats TTL 90d -- one fetch per site+param serves
every reading for the window, so a warm cache makes zero USGS calls). USGS down
-> cached-if-present else all-null bundle; the event still publishes.
Framework: the single agreed generic change -- supervisor apply_enrichment now
MERGES into _enriched instead of overwriting, so the still-global geocoder phase
doesn't clobber the adapter's bundles. No other adapter writes _enriched, so this
is inert for them.
GUI: _event_summaries/nwis.html -> "<site> -- <value> <units> (<band>, <Nth>
percentile)", with graceful fallback to "<site> -- <value>" then the bare
"Water reading:". _event_rows/nwis.html detail gains site/normalcy/typical/location
rows. _events_rows.html Location column falls back generically to any
_enriched.<source> carrying state/county when the geocoder is null (works for
future enrichers). events.json contract unchanged (additions under _enriched only).
conftest isolate_enrichment_cache also redirects NWIS_CACHE_DB_PATH off the prod
path (unprivileged-user test isolation). Adds tests/test_nwis_enrichment.py (28
tests: parse, band edges incl P0/P9/P10/P75/P90/record, percentile interpolation,
cache hit/miss/expire, adapter enrich + graceful-null + cache-hit-no-refetch,
summary rendering per band).
Full suite: 710 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #6 of the v0.7.x GUI rework arc, and its close-out. Production code;
central-gui restart only (templates + a static stylesheet; supervisor untouched).
The v0.7.0-v0.7.4 arc shipped the FEATURES the gui_preview.html mock shows
(chip-pickers, active pills, markercluster + shape/opacity markers, collapsed
grouped legend, real paginator, telemetry tab) but on top of the old CSS
framework. v0.7.5 closes the visual gap: the framework is removed entirely and
replaced by a hand-authored design system that IS the preview (neutral greys, a
single blue accent, 6px radii, 14px type, bordered cards/tables).
- New static/css/central.css: design tokens + element base (forms, buttons,
tables, cards, nav) + every events-feed component, authored so the existing
template class names + element IDs the page JS/HTMX depends on (.chip-picker*,
.events-table, .event-row, .legend-chip, .evt-marker, #fit-to-results,
#map-filter-toggle, ...) render in the preview's language unchanged. Mounted at
/static (already wired in __init__.py); linked from base.html + base_wizard.html.
- base.html / base_wizard.html: framework <link> removed; nav restructured to the
preview shell (.nav / .brand / .nav-links / .logout-btn); flash messages moved
off framework color vars to .flash-* classes.
- events_list.html: 190-line inline <style> deleted (moved to central.css); map
wrapped in .map-container with a floating .map-toolbar (Fit / map-filter toggle),
legend below. HTMX wiring (hx-get/hx-target/hx-push-url), the v0.7.1 chip-picker
JS, the v0.7.2 marker shapes/opacity, and the v0.7.4 base_path split are
untouched.
- Every other page (dashboard, adapters, api-keys, enrichment, streams, login,
change-password, and the full setup wizard) de-framework'd: framework class
names (outline/secondary/contrast/grid/container) renamed to neutral
.btn-*/.cols; no framework default rendering; same tokens/typography/forms/tables.
- /events.json + the v0.7.1 registry-index adapter palette + v0.7.2 severity
opacity + v0.7.4 stable cross-tab chip colors are all unchanged -- this is a
visual shell swap, not a behavior or data change.
grep -rn pico over templates + static returns zero. All GUI routes return 200
(authed) / 302 (unauthed); no 5xx. Full suite: 682 passed, 1 skipped (central and
unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 of the v0.7.x GUI rework arc. Production code; central-gui restart only.
- Adapter legend: collapsed by default ("{n} adapters · Show legend ▾"). Expands
to domain-grouped chips (same grouping as the v0.7.1 chip-picker) with uniform
ellipsis-truncated names + full-name title tooltips. Clicking a legend chip
toggles that adapter's filter (reuses the chip-picker's hidden CSV via
syncField), so the legend doubles as a filter affordance.
- Row stability: time cell is single-line MM-DD HH:MM UTC (year dropped from the
cell; full ISO in the cell tooltip + a new Time row in the expanded detail).
Adapter cell is a chip (color swatch + short name; display_name is the
tooltip). table-layout:fixed + per-column widths + fixed 37px row height with
nowrap/ellipsis cells -> no per-row wrap variation.
- Real paginator: _fetch_events offset-mode returns the exact page slice plus the
grand total via count(*) OVER() in one roundtrip. Previous/Next + windowed page
numbers (1 ... 4 5 [6] 7 8 ... 47) + "showing X-Y of N" + a 25/50/100/250
per-page selector. URL state persists offset + limit. events.json keeps cursor
pagination (back-compat): offset param presence selects offset-mode, its
absence keeps the cursor path -- cleanly separable by endpoint.
Adds TestEventsPagination (12 tests: offset/limit parse incl. max 250,
offset-vs-cursor query shape, _build_pagination windowing). Updates the time
format + adapter-cell + pagination-mode assertions in the existing frontend
tests to the new contract.
Full suite: 674 passed, 1 skipped (central and unprivileged zvx). count(*) OVER()
is ~7.5ms at current volume; vanilla JS + HTMX; CSS functional-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3 of the v0.7.x GUI rework arc. Makes the /events Leaflet map readable and
intentional. Production code; central-gui restart only (no adapter change).
- Fit-to-results default: the map now fits the actual event distribution on
load (previously disabled -> fixed global zoom-4). Empty result set falls
back to the CONUS setView (no crash). Re-fits after each HTMX swap, but only
when the map-filter toggle is OFF (when ON the viewport drives the bbox, so
re-fitting would fight/loop the filter).
- leaflet.markercluster (1.5.3, via CDN): point markers cluster into numbered
badges (disableClusteringAtZoom=9, showCoverageOnHover=false,
spiderfyOnMaxZoom=true). markercluster supports point markers only, so
polygons/lines render in a separate un-clustered featureGroup; fit unions both.
- Map-filter toggle ("Filter table by map view"), default OFF. When off the
table shows all filter-matching events regardless of map zoom; the backend
ignores region_* unless map_filter is set (guards bookmarked URLs too). URL
carries map_filter=1 only when on (hidden input disabled otherwise).
- Per-event_type marker shape (derived event_type = first category segment):
circle = quake/hydro/space (points), square = fire (areas),
triangle = wx (NWS alerts/warnings), star = disaster (GDACS/EONET).
Rendered as divIcon + CSS clip-path; point markers switched from circleMarker
to L.marker(divIcon) (also required for markercluster compatibility).
- Per-severity opacity: critical(4)=1.0, high(3)=0.85, moderate(2)=0.7,
low(1)=0.5, unknown(0/NULL)=0.4. Needed adding severity to the _fetch_events
SELECT + event dict (row.get for mock-tolerance) + a data-severity row attr.
Adds 4 tests (map_filter gating on/off, bbox reaches query only when on,
severity in SELECT); updates test_events_bbox_guard for the new toggle contract.
Full suite: 662 passed, 1 skipped (central and unprivileged zvx). Vanilla JS +
HTMX + Leaflet/markercluster; CSS functional-only (polish deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Biggest PR of the v0.7.x GUI rework arc. Replaces the single-select /events
filter row with a multi-select, URL-addressable filtering surface.
- Search: full-width box, debounced 300ms, server-side ILIKE over the inner
adapter payload (covers the derived subject + location); parameterized with
LIKE wildcards escaped (ESCAPE '\'). Injection-safe.
- Adapter / Category / Event Type / Severity: multi-select chip-pickers (shared
_chip_picker.html macro). Adapter is grouped by domain with color swatches and
an in-panel search. Backend uses `= ANY(...)`. URL state is comma-separated.
- Event Type is derived as split_part(category,'.',1) (no event_type column yet;
a stand-in until the v0.8 canonical schema). Severity maps labels to the
numeric scale (4=critical..1=low, 0/NULL=unknown).
- Time: preset dropdown (15m/1h/6h/24h/7d/active/all) + custom from/to range,
encoded in a single `time` token. GUI defaults to last_24h; events.json keeps
its single-value adapter/since/until contract (no default).
- Active pills: server-rendered from parsed state, updated out-of-band on each
HTMX swap; each x clears that filter and re-submits.
- URL state persistence: every filter in the query string; /events/rows sets
HX-Push-Url to the /events?... full-page URL so bookmarking/back-forward work.
Filter options are rendered server-side at page load (DISTINCT category +
split_part, registry adapters, severity enum) -- no new AJAX endpoints.
Vanilla JS + HTMX (no framework added). CSS is functional-only; visual polish
is deferred to a later pass per the rework plan.
Adds TestEventsFiltering (24 tests: multi-value parse, ILIKE injection safety,
time-preset resolution with injected clock, severity/NULL handling, active-pill
descriptors, URL round-trip). Updates four TestEventsFeedFrontend assertions to
the new filter_state/adapters contract.
Full suite: 658 passed, 1 skipped (central and unprivileged zvx). No adapter
base class change -> central-gui restart only (no supervisor restart).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events_json SELECT read payload->>'subject', but the CloudEvents
envelope has no top-level subject, so every JSON consumer saw
subject: null. The /events GUI already derives readable subjects via
per-adapter templates/_event_summaries/{adapter}.html (PR L-c).
This makes the JSON path produce the same plain-text subjects with no
duplicated logic: _derive_subject(event) renders the same partial the
table uses (falling back to _default.html) and html.unescapes the
autoescaped output so JSON consumers get plain text (e.g. ">=1 MeV"
rather than the escaped ">=1 MeV"). _fetch_events now sets subject
from it and drops the always-null SQL expression. The GUI Subject cell
is unchanged.
Adds TestEventsJsonSubject (parameterized over discover_adapters(), no
hardcoded list): non-null subject per adapter, equality with the rendered
partial, pinned human text for the deterministic adapters, swpc_alerts
truncation, and null fallbacks. Updates one TestEventRowDataAttributes
assertion that pinned the old SQL pass-through contract.
One route change plus tests; central-gui restart required.
Full suite: 629 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleans up unused imports and dead locals flagged by ruff in the test
files PR #50 (M-b) touched. Tests-only; no production code, no service
restart.
- test_supervisor_hotreload.py: drop unused AsyncMock/patch imports,
dead expected_wait/expected_next_poll locals, and two dead
state = AdapterState(...) blocks plus their now-orphaned local imports
- test_supervisor_integration.py: drop unused asyncio/patch/pytest_asyncio
imports and AdapterState from two function-local imports
ruff tests/ 92 -> 82 (the 4 named files now 0; all other files unchanged).
Full suite: 590 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coordless events such as removal tombstones with null lat/lon, from adapters that declare enrichment_locations, previously fell off the loop without writing _enriched and carried no geocoder bundle at all, violating the every-event-carries-_enriched design rule. Add a post-loop fallback that resolves the null location to an all-null bundle per enricher. Adapters with no enrichment_locations remain skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First half of the split PR L (events-tab + map deferred to L-b). Only FIRMS
declared enrichment_locations, so apply_enrichment silently bypassed every
other adapter. This declares it for all 12.
Pre-read finding (resolved per PM): apply_enrichment is a FLAT lookup
(event.data.get(lat_path)); FIRMS/usgs_quake already carry top-level
latitude/longitude in event.data, but the other point adapters kept coords
only in Geo.centroid where the flat path can't reach them. Per PM (option b),
the 5 centroid-only adapters now also write top-level latitude/longitude into
event.data, mirroring their existing Geo.centroid (lon, lat) — 2-3 lines each,
no framework refactor. Geo retained for existing rendering uses.
Declarations (verbatim):
firms [("latitude","longitude")] (unchanged)
usgs_quake [("latitude","longitude")] (already top-level in data)
nwis [("latitude","longitude")] + centroid mirror
eonet [("latitude","longitude")] + centroid mirror
gdacs [("latitude","longitude")] + centroid mirror
wfigs_incidents [("latitude","longitude")] + centroid mirror (inline data)
inciweb [("latitude","longitude")] + centroid mirror (inline data)
wfigs_perimeters [] # polygons, no point
nws [] # forecast zones/counties, no point
swpc_alerts [] # space weather, no coordinate
swpc_kindex [] # space weather, no coordinate
swpc_protons [] # space weather, no coordinate
Centroid mirror is `latitude = centroid[1]; longitude = centroid[0]` (centroid
is GeoJSON (lon, lat)); guarded on centroid presence so coordinate-less events
get no lat/lon keys (apply_enrichment then skips them).
map_render_kind concept dropped — the existing /events map is already
geometry-kind-agnostic (renders any row's data-geometry via L.geoJSON), so it
was unnecessary. Events-tab enhancements are PR L-b.
Tests (test_enrichment_locations_coverage.py, 6, all registry-derived):
- every adapter explicitly declares enrichment_locations in its own class body
- declarations are valid list[(str,str)]
- point adapters all use the canonical ("latitude","longitude") paths
- >=5 point adapters are non-empty (regression guard)
- synthetic-event builders prove the keys resolve: usgs_quake._feature_to_event
and nwis._build_event (the two adapters with isolated builders; the four
inline-build adapters are covered by the post-merge live smoke).
Verification: full pytest 552 passed, 1 skipped (was 546; +6). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src empty.
Follow-ups (NOT here): consumer-doc per-adapter _enriched.geocoder notes for
the newly-enriched adapters belong in L-b's doc pass; live end-to-end smoke
runs post-merge (USGS quake + one other) per the acceptance bar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during the 2026-05-20 NaviBackend activation: toggling
config.enrichment.backend_class to NoOpBackend while backend_settings still
held {"base_url": ...} crashed _rebuild_enrichers with
`TypeError: NoOpBackend() takes no arguments`, BEFORE invalidate() ran. Fixed
by mirroring the SourceAdapter.settings_schema pattern: each backend declares a
Pydantic settings_schema; validation happens at write-time (GUI POST) and
read-time (supervisor). A mismatch is now a clean ValidationError, never a
constructor TypeError.
Backends — each gets a `<Name>BackendSettings(BaseModel, extra="forbid")` +
`settings_schema` class attr, mirroring __init__ defaults EXACTLY (note:
timeout_s stays 10.0 — the brief's "5.0" was a transcription slip; preserve the
production default):
NoOpBackend -> NoOpBackendSettings (no fields)
NaviBackend -> NaviBackendSettings (base_url, timeout_s, headers, warmup)
PhotonBackend -> PhotonBackendSettings (base_url, timeout_s, headers)
NominatimBackend-> NominatimBackendSettings (base_url, user_agent, rate_limit_per_sec, timeout_s)
GeocoderBackend Protocol (in geocoder.py, where the base actually lives — not
base.py, which only has Enricher) gains `settings_schema: type[BaseModel]`.
supervisor:
- build_enrichers validates backend_cls.settings_schema.model_validate(
backend_settings) before instantiating, and constructs from the validated
.model_dump(). ValidationError (not TypeError) on mismatch.
- _rebuild_enrichers builds into locals and commits to instance state only on
success — a ValidationError leaves the previously-active enrichers/config/
cache untouched.
- _handle_enrichment_change wraps the rebuild in try/except ValidationError:
logs and returns, keeping the previous backend running (supervisor stays up;
operator fixes the row; next NOTIFY applies cleanly). No cache invalidation
on a failed change.
GUI /enrichment:
- GET skips the outer EnrichmentConfig.backend_settings field and renders a
separate <fieldset> from describe_fields(backend_cls.settings_schema, ...)
for the row's current backend_class. Backend fields namespaced bs_<name>.
- POST reassembles bs_<name> inputs into a backend_settings dict, validates it
against the SUBMITTED backend_class's schema (so errors attach to the right
fields when an operator is mid-switch), then validates the outer
EnrichmentConfig. DB row written only if both pass; otherwise re-renders with
field-level errors against the submitted backend.
- backend_class stays a plain text field (no <select>, no client-side reshape).
form_descriptors: generic `float -> "number"` widget (2 lines, mirrors K.5's
`dict -> "json"`), needed because backend schemas have float fields
(timeout_s, rate_limit_per_sec). Benefits any float field codebase-wide.
DB schema unchanged: backend_settings stays JSONB; validation moved to
use-site. _BACKEND_REGISTRY / _ENRICHER_REGISTRY unchanged beyond schema lookup.
Tests (test_backend_settings_schema.py, 11): schemas exist + extra='forbid';
Navi schema preserves defaults (timeout_s == 10.0); NoOp has zero fields;
build_enrichers raises ValidationError-not-TypeError for the exact 2026-05-20
case; supervisor keeps previous backend on a bad NOTIFY (the incident
scenario); valid NoOp-with-empty-settings applies + invalidates; GUI POST
rejects bad backend_settings without writing + re-renders against submitted
backend; GUI POST writes on valid settings. test_enrichment_config_plumbing
updated for the new context shape (outer_fields/backend_fields).
Verification: full pytest 546 passed, 1 skipped (was 535; +11). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src both
empty.
Does NOT touch PR L scope (events tab, remaining adapter enrichment_locations),
the DB schema, or the registries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of three PRs for v0.5.0 (J: framework; K: real geocoder backends +
doc revisions; L: operator events tab + per-adapter render + events-map fix).
Design pivot: the Phase 2 "no enrichment, upstream verbatim" reading of
Matt's principle is reframed — consumers can't do follow-up lookups, they
only see what's on the wire, so whatever Central doesn't enrich is
effectively missing downstream. Enrichment is now expected. The producer-doc
§2/§10.1 rewrite lands in PR K; this PR builds the framework PR K documents.
New package src/central/enrichment/:
- base.py Enricher Protocol (name + async enrich(location) -> dict).
- geocoder.py GeocoderEnricher + GeocoderBackend Protocol + the locked
GEOCODER_FIELDS set (name, city, county, state, country,
postal_code, timezone, landclass, elevation_m) + all_null_bundle().
- cache.py EnrichmentCache — stdlib sqlite3 off the event loop via
asyncio.to_thread (no async-sqlite dep). Keyed on
(enricher_name, lat_4dp, lon_4dp); per-enricher TTL (24h
default); fresh connection per op (sqlite3 isn't thread-safe
to share). Cache even all-null; never cache backend failures.
- backends/no_op.py NoOpBackend — all-null bundle, the PR J default.
Provenance: enrichment results land under event.data["_enriched"][<name>];
everything else in data stays upstream verbatim.
Wiring:
- adapter.py enrichment_locations: list[tuple[str,str]] = [] class attr.
Empty (default) = publish as-is, no enrichment.
- config_models.py EnrichmentConfig (enricher_class, backend_class,
backend_settings, cache_ttl_s). Read once at startup.
- supervisor.py build_enrichers() + apply_enrichment(); enrichment runs
after dedup, before wrap_event, in the poll loop. Class-name
registries for enricher/backend resolution (PR K extends).
- firms.py enrichment_locations = [("latitude","longitude")] — pilot.
Enrichment config is read once at supervisor startup; hot-reload is out of
scope for PR J (noted in EnrichmentConfig + build_enrichers docstrings).
Tests (16 new):
- test_enrichment_framework.py (9): parent-dir/table init, cache miss->hit,
TTL expiry, 4dp rounding, nearby-coord collapse, concurrent-set single-row,
backend-failure all-null-not-cached (retries), success cached (one backend
call), all-null cached.
- test_geocoder_enricher.py (5): NoOp all-null, field-set == GEOCODER_FIELDS,
null-coords short-circuit (no backend call), name=="geocoder", sequential
same-coords single backend call.
- test_firms.py (+2): enrichment_locations declared + paths resolve to floats
in a real event (structural, not literal); event through supervisor
apply_enrichment emerges with data._enriched.geocoder == all-null bundle.
Verification: full pytest 495 passed (was 479; +16). grep for
subject_for_event/_ADAPTER_REGISTRY clean. Module imports cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /adapters list view's "⚠️ API Key Missing" chip, the /adapters/{name}
edit form's disabled enable-checkbox, the POST error re-render path, AND
the supervisor's adapter-start precondition all compared the hardcoded
SourceAdapter class attribute `requires_api_key` against `config.api_keys`,
ignoring the per-row `settings[api_key_field]` alias the operator
actually selected via the form.
FIRMS' class attr is `requires_api_key = "firms"`; the api_keys_new.html
placeholder text steers operators toward aliases like `firms_production`
instead, and the FIRMSSettings.api_key_alias field is exactly the
overridable slot that the form writes. The four predicates ignored that
slot, so a working key under any non-default alias was treated as
missing — chip on, checkbox disabled, supervisor refusing to start with
`last_error = "missing api key: firms"`.
Audit: FIRMS is the only adapter today with `requires_api_key != None`.
Every other adapter is unaffected by either the route or supervisor
predicate.
Helper module:
- src/central/api_key_resolver.py exposes:
resolve_api_key_alias(adapter_cls, settings) -> str | None
Pure sync function. Returns the alias to consult, or None when no
key is required. Supervisor uses this directly + its own
get_api_key.
adapter_has_resolved_api_key(conn, adapter_cls, settings) -> (bool, alias)
Async wrapper that runs the SELECT 1 against config.api_keys.
The three GUI routes use this.
Resolution: settings[api_key_field] when set to a non-empty str,
otherwise the class-attr default.
Four call sites swapped:
- routes.py:adapters_list (/adapters list — warning chip)
- routes.py:adapters_edit_form (/adapters/{name} edit GET — disabled checkbox)
- routes.py:adapters_edit_submit (POST error re-render)
- supervisor.py:_start_adapter (adapter-start precondition)
Side-effect tests/test_adapters.py fix:
- TestAdaptersJsonbRegression::test_adapters_edit_fetches_api_keys_into_context
used `AsyncMock()` (no return_value) for mock_conn.__aexit__. AsyncMock
without a return_value yields a MagicMock — which is truthy, and the
async context manager protocol reads truthy from __aexit__ as
"exception suppressed." That silently swallowed any error inside
`async with` blocks. The route refactor moved an assignment inside the
one async with at site 2, so a swallowed mock error left the variable
unbound. Fixed: `AsyncMock(return_value=None)` + a comment so the next
person doesn't re-introduce the bug. fetchval mock added because the
resolver now issues it (the swallowed exception previously hid the
missing mock).
Verification:
- pytest: 479 passed (was 469; +10 new resolver tests).
- grep -rn "adapter_cls.requires_api_key" /opt/central/src returns only
the new helper (2 lines, same file).
- Resolver against live FIRMS settings: resolved_alias='firms_production',
has_key=True, api_key_missing=False -> NO warning chip, checkbox
CLICKABLE.
- Supervisor on live CT104: FIRMS flipped enabled=true via DB UPDATE;
supervisor started the adapter with `api_key_present: true,
api_key_alias: 'firms_production'`; last_error cleared from "missing
api key: firms" -> NULL; two satellite polls completed (VIIRS_SNPP_NRT
477 features, VIIRS_NOAA20_NRT 400 features); 869 new events published
to JetStream.
NOTE: This commit's verification flipped FIRMS to enabled=true in the
running config — the adapter is now actively polling. Pause via the UI
if that's not intended for now; the bug fix itself does not require
FIRMS to be enabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
Fixup 1 — Contract section appended to SourceAdapter.preview_for_settings's
docstring. Override authors read adapter.py, not routes.py, so the contract
(pure function of settings; open your own short-lived aiohttp session; None
vs [] semantics) belongs on the base method, not on the GUI stub class.
Fixup 2 — _adapter_preview.html distinguishes [] from None. Previously the
elif test was truthiness (`elif preview_rows`) which collapsed both into
"render nothing". Now uses `elif preview_rows is not none` and special-cases
the empty-list case inside: legend "Preview (0 rows)" with no table; None
still renders nothing at all. Lets adapters signal "query ran, matched zero"
distinctly from "preview not meaningful".
Tests +1:
- test_partial_renders_empty_list — [] yields "Preview (0 rows)" legend,
no table, no headers. Distinct from the existing None case.
Acceptance:
- 27/27 targeted (preview_hook +1 new, nwis, stream_registry).
- 458/458 full suite.
- (b) framework GUI dir still has zero adapter-name branches.
Adds an optional async hook on SourceAdapter so any adapter can surface a
settings-driven preview on its /adapters/<name> edit page. The framework
renders the result generically as a table — no adapter-name branches in
GUI templates or route code.
Framework changes:
- src/central/adapter.py: new async preview_for_settings(self, settings)
on the base class, default returns None. Adapters opt in by overriding;
non-overriding adapters render unchanged.
- src/central/gui/routes.py: GET /adapters/{name} instantiates the adapter
with a no-op _PreviewConfigStore stub and a /dev/null cursor path (GUI
has no live ConfigStore), constructs settings_obj via the schema, and
calls preview_for_settings inside a try/except. Result lands in template
context as preview_rows / preview_error.
- src/central/gui/templates/_adapter_preview.html: new partial. Generic
table with columns derived from the first dict's keys; error banner
mirrors the existing last_error article style.
- src/central/gui/templates/adapters_edit.html: one-line include between
the Region fieldset and Save/Cancel.
NWIS opt-in:
- New NWIS_MONITORING_LOCATIONS_URL constant and _PREVIEW_LIMIT cap of 50.
- preview_for_settings returns None when region is None, otherwise one-shot
fetches monitoring-locations within the bbox via a fresh aiohttp session.
Must work even when adapter is not started -- the GUI process never calls
startup(). Returns list[dict] with the contract column order: site_id,
name, site_type, state. Errors propagate so the framework can render the
operator-visible banner.
- HTTP call factored into _fetch_preview_text so tests mock cleanly.
Tests (7 new):
- tests/test_preview_hook.py: default returns None; partial renders list
with correct headers/rows/count; partial renders error banner; partial
renders empty when both context values are None.
- tests/test_nwis.py adds TestNWISPreview: returns None without region,
returns rows with correct column order, propagates HTTP errors.
Verification:
- 457/457 full suite green (was 450; +7 new tests).
- Live /adapters/nwis preview returns 50 rows with the contract keys
against the current production Iowa bbox.
- /adapters/eonet preview_for_settings returns None via base default --
proves framework is duck-typed, no NWIS-specific code in framework.
