Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens the v0.8.x data-quality cleanup arc. Production code; central-gui AND
central-supervisor restart (adapter contract + enrichment behavior change).
NWIS events rendered as a bare "Water reading: 111 ft3/s" with an empty Location
column -- an operator couldn't tell where the gauge is or whether 111 ft3/s is
drought-low, normal, or near-flood. Coordinates were present but the reverse
geocoder returns null city/state/county for rural gauge points, and USGS site +
percentile data was never fetched. v0.8.0 fetches both.
Approach B (adapter-owned, per the proposal decision): the NWIS adapter -- which
already owns the USGS APIs -- fetches site metadata and daily stats itself and
writes two provenance bundles under event.data["_enriched"]:
- usgs_site {name, lat, lon, state, county} from the OGC monitoring-locations
item-by-id (the API family the adapter already speaks; JSON, no RDB parser).
- usgs_stats {value, percentile, class_label, severity_band, p10..p90, record_max,
count, period} from the legacy RDB daily-statistics service (the OGC API has no
stats endpoint). USGS percentiles are % of days at-or-below, so higher = higher
flow; classified to the WaterWatch bands -> severity 0-4 (record=4, much
above/below=3, above/below=2, normal=1; None reserved for "no stats", distinct
from a normal-flow gauge). Severity is set on the event, so it drives the v0.7.1
severity chip-picker filter + v0.7.2 map-marker opacity.
- new nwis_enrich.py: pure parse/classify/percentile/band helpers + a sqlite
SiteStatsCache (site TTL 365d, stats TTL 90d -- one fetch per site+param serves
every reading for the window, so a warm cache makes zero USGS calls). USGS down
-> cached-if-present else all-null bundle; the event still publishes.
Framework: the single agreed generic change -- supervisor apply_enrichment now
MERGES into _enriched instead of overwriting, so the still-global geocoder phase
doesn't clobber the adapter's bundles. No other adapter writes _enriched, so this
is inert for them.
GUI: _event_summaries/nwis.html -> "<site> -- <value> <units> (<band>, <Nth>
percentile)", with graceful fallback to "<site> -- <value>" then the bare
"Water reading:". _event_rows/nwis.html detail gains site/normalcy/typical/location
rows. _events_rows.html Location column falls back generically to any
_enriched.<source> carrying state/county when the geocoder is null (works for
future enrichers). events.json contract unchanged (additions under _enriched only).
conftest isolate_enrichment_cache also redirects NWIS_CACHE_DB_PATH off the prod
path (unprivileged-user test isolation). Adds tests/test_nwis_enrichment.py (28
tests: parse, band edges incl P0/P9/P10/P75/P90/record, percentile interpolation,
cache hit/miss/expire, adapter enrich + graceful-null + cache-hit-no-refetch,
summary rendering per band).
Full suite: 710 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 of the v0.7.x GUI rework arc. Production code; central-gui restart only.
- Adapter legend: collapsed by default ("{n} adapters · Show legend ▾"). Expands
to domain-grouped chips (same grouping as the v0.7.1 chip-picker) with uniform
ellipsis-truncated names + full-name title tooltips. Clicking a legend chip
toggles that adapter's filter (reuses the chip-picker's hidden CSV via
syncField), so the legend doubles as a filter affordance.
- Row stability: time cell is single-line MM-DD HH:MM UTC (year dropped from the
cell; full ISO in the cell tooltip + a new Time row in the expanded detail).
Adapter cell is a chip (color swatch + short name; display_name is the
tooltip). table-layout:fixed + per-column widths + fixed 37px row height with
nowrap/ellipsis cells -> no per-row wrap variation.
- Real paginator: _fetch_events offset-mode returns the exact page slice plus the
grand total via count(*) OVER() in one roundtrip. Previous/Next + windowed page
numbers (1 ... 4 5 [6] 7 8 ... 47) + "showing X-Y of N" + a 25/50/100/250
per-page selector. URL state persists offset + limit. events.json keeps cursor
pagination (back-compat): offset param presence selects offset-mode, its
absence keeps the cursor path -- cleanly separable by endpoint.
Adds TestEventsPagination (12 tests: offset/limit parse incl. max 250,
offset-vs-cursor query shape, _build_pagination windowing). Updates the time
format + adapter-cell + pagination-mode assertions in the existing frontend
tests to the new contract.
Full suite: 674 passed, 1 skipped (central and unprivileged zvx). count(*) OVER()
is ~7.5ms at current volume; vanilla JS + HTMX; CSS functional-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3 of the v0.7.x GUI rework arc. Makes the /events Leaflet map readable and
intentional. Production code; central-gui restart only (no adapter change).
- Fit-to-results default: the map now fits the actual event distribution on
load (previously disabled -> fixed global zoom-4). Empty result set falls
back to the CONUS setView (no crash). Re-fits after each HTMX swap, but only
when the map-filter toggle is OFF (when ON the viewport drives the bbox, so
re-fitting would fight/loop the filter).
- leaflet.markercluster (1.5.3, via CDN): point markers cluster into numbered
badges (disableClusteringAtZoom=9, showCoverageOnHover=false,
spiderfyOnMaxZoom=true). markercluster supports point markers only, so
polygons/lines render in a separate un-clustered featureGroup; fit unions both.
- Map-filter toggle ("Filter table by map view"), default OFF. When off the
table shows all filter-matching events regardless of map zoom; the backend
ignores region_* unless map_filter is set (guards bookmarked URLs too). URL
carries map_filter=1 only when on (hidden input disabled otherwise).
- Per-event_type marker shape (derived event_type = first category segment):
circle = quake/hydro/space (points), square = fire (areas),
triangle = wx (NWS alerts/warnings), star = disaster (GDACS/EONET).
Rendered as divIcon + CSS clip-path; point markers switched from circleMarker
to L.marker(divIcon) (also required for markercluster compatibility).
- Per-severity opacity: critical(4)=1.0, high(3)=0.85, moderate(2)=0.7,
low(1)=0.5, unknown(0/NULL)=0.4. Needed adding severity to the _fetch_events
SELECT + event dict (row.get for mock-tolerance) + a data-severity row attr.
Adds 4 tests (map_filter gating on/off, bbox reaches query only when on,
severity in SELECT); updates test_events_bbox_guard for the new toggle contract.
Full suite: 662 passed, 1 skipped (central and unprivileged zvx). Vanilla JS +
HTMX + Leaflet/markercluster; CSS functional-only (polish deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Biggest PR of the v0.7.x GUI rework arc. Replaces the single-select /events
filter row with a multi-select, URL-addressable filtering surface.
- Search: full-width box, debounced 300ms, server-side ILIKE over the inner
adapter payload (covers the derived subject + location); parameterized with
LIKE wildcards escaped (ESCAPE '\'). Injection-safe.
- Adapter / Category / Event Type / Severity: multi-select chip-pickers (shared
_chip_picker.html macro). Adapter is grouped by domain with color swatches and
an in-panel search. Backend uses `= ANY(...)`. URL state is comma-separated.
- Event Type is derived as split_part(category,'.',1) (no event_type column yet;
a stand-in until the v0.8 canonical schema). Severity maps labels to the
numeric scale (4=critical..1=low, 0/NULL=unknown).
- Time: preset dropdown (15m/1h/6h/24h/7d/active/all) + custom from/to range,
encoded in a single `time` token. GUI defaults to last_24h; events.json keeps
its single-value adapter/since/until contract (no default).
- Active pills: server-rendered from parsed state, updated out-of-band on each
HTMX swap; each x clears that filter and re-submits.
- URL state persistence: every filter in the query string; /events/rows sets
HX-Push-Url to the /events?... full-page URL so bookmarking/back-forward work.
Filter options are rendered server-side at page load (DISTINCT category +
split_part, registry adapters, severity enum) -- no new AJAX endpoints.
Vanilla JS + HTMX (no framework added). CSS is functional-only; visual polish
is deferred to a later pass per the rework plan.
Adds TestEventsFiltering (24 tests: multi-value parse, ILIKE injection safety,
time-preset resolution with injected clock, severity/NULL handling, active-pill
descriptors, URL round-trip). Updates four TestEventsFeedFrontend assertions to
the new filter_state/adapters contract.
Full suite: 658 passed, 1 skipped (central and unprivileged zvx). No adapter
base class change -> central-gui restart only (no supervisor restart).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The events_json SELECT read payload->>'subject', but the CloudEvents
envelope has no top-level subject, so every JSON consumer saw
subject: null. The /events GUI already derives readable subjects via
per-adapter templates/_event_summaries/{adapter}.html (PR L-c).
This makes the JSON path produce the same plain-text subjects with no
duplicated logic: _derive_subject(event) renders the same partial the
table uses (falling back to _default.html) and html.unescapes the
autoescaped output so JSON consumers get plain text (e.g. ">=1 MeV"
rather than the escaped ">=1 MeV"). _fetch_events now sets subject
from it and drops the always-null SQL expression. The GUI Subject cell
is unchanged.
Adds TestEventsJsonSubject (parameterized over discover_adapters(), no
hardcoded list): non-null subject per adapter, equality with the rendered
partial, pinned human text for the deterministic adapters, swpc_alerts
truncation, and null fallbacks. Updates one TestEventRowDataAttributes
assertion that pinned the old SQL pass-through contract.
One route change plus tests; central-gui restart required.
Full suite: 629 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleans up unused imports and dead locals flagged by ruff in the test
files PR #50 (M-b) touched. Tests-only; no production code, no service
restart.
- test_supervisor_hotreload.py: drop unused AsyncMock/patch imports,
dead expected_wait/expected_next_poll locals, and two dead
state = AdapterState(...) blocks plus their now-orphaned local imports
- test_supervisor_integration.py: drop unused asyncio/patch/pytest_asyncio
imports and AdapterState from two function-local imports
ruff tests/ 92 -> 82 (the 4 named files now 0; all other files unchanged).
Full suite: 590 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coordless events such as removal tombstones with null lat/lon, from adapters that declare enrichment_locations, previously fell off the loop without writing _enriched and carried no geocoder bundle at all, violating the every-event-carries-_enriched design rule. Add a post-loop fallback that resolves the null location to an all-null bundle per enricher. Adapters with no enrichment_locations remain skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First half of the split PR L (events-tab + map deferred to L-b). Only FIRMS
declared enrichment_locations, so apply_enrichment silently bypassed every
other adapter. This declares it for all 12.
Pre-read finding (resolved per PM): apply_enrichment is a FLAT lookup
(event.data.get(lat_path)); FIRMS/usgs_quake already carry top-level
latitude/longitude in event.data, but the other point adapters kept coords
only in Geo.centroid where the flat path can't reach them. Per PM (option b),
the 5 centroid-only adapters now also write top-level latitude/longitude into
event.data, mirroring their existing Geo.centroid (lon, lat) — 2-3 lines each,
no framework refactor. Geo retained for existing rendering uses.
Declarations (verbatim):
firms [("latitude","longitude")] (unchanged)
usgs_quake [("latitude","longitude")] (already top-level in data)
nwis [("latitude","longitude")] + centroid mirror
eonet [("latitude","longitude")] + centroid mirror
gdacs [("latitude","longitude")] + centroid mirror
wfigs_incidents [("latitude","longitude")] + centroid mirror (inline data)
inciweb [("latitude","longitude")] + centroid mirror (inline data)
wfigs_perimeters [] # polygons, no point
nws [] # forecast zones/counties, no point
swpc_alerts [] # space weather, no coordinate
swpc_kindex [] # space weather, no coordinate
swpc_protons [] # space weather, no coordinate
Centroid mirror is `latitude = centroid[1]; longitude = centroid[0]` (centroid
is GeoJSON (lon, lat)); guarded on centroid presence so coordinate-less events
get no lat/lon keys (apply_enrichment then skips them).
map_render_kind concept dropped — the existing /events map is already
geometry-kind-agnostic (renders any row's data-geometry via L.geoJSON), so it
was unnecessary. Events-tab enhancements are PR L-b.
Tests (test_enrichment_locations_coverage.py, 6, all registry-derived):
- every adapter explicitly declares enrichment_locations in its own class body
- declarations are valid list[(str,str)]
- point adapters all use the canonical ("latitude","longitude") paths
- >=5 point adapters are non-empty (regression guard)
- synthetic-event builders prove the keys resolve: usgs_quake._feature_to_event
and nwis._build_event (the two adapters with isolated builders; the four
inline-build adapters are covered by the post-merge live smoke).
Verification: full pytest 552 passed, 1 skipped (was 546; +6). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src empty.
Follow-ups (NOT here): consumer-doc per-adapter _enriched.geocoder notes for
the newly-enriched adapters belong in L-b's doc pass; live end-to-end smoke
runs post-merge (USGS quake + one other) per the acceptance bar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during the 2026-05-20 NaviBackend activation: toggling
config.enrichment.backend_class to NoOpBackend while backend_settings still
held {"base_url": ...} crashed _rebuild_enrichers with
`TypeError: NoOpBackend() takes no arguments`, BEFORE invalidate() ran. Fixed
by mirroring the SourceAdapter.settings_schema pattern: each backend declares a
Pydantic settings_schema; validation happens at write-time (GUI POST) and
read-time (supervisor). A mismatch is now a clean ValidationError, never a
constructor TypeError.
Backends — each gets a `<Name>BackendSettings(BaseModel, extra="forbid")` +
`settings_schema` class attr, mirroring __init__ defaults EXACTLY (note:
timeout_s stays 10.0 — the brief's "5.0" was a transcription slip; preserve the
production default):
NoOpBackend -> NoOpBackendSettings (no fields)
NaviBackend -> NaviBackendSettings (base_url, timeout_s, headers, warmup)
PhotonBackend -> PhotonBackendSettings (base_url, timeout_s, headers)
NominatimBackend-> NominatimBackendSettings (base_url, user_agent, rate_limit_per_sec, timeout_s)
GeocoderBackend Protocol (in geocoder.py, where the base actually lives — not
base.py, which only has Enricher) gains `settings_schema: type[BaseModel]`.
supervisor:
- build_enrichers validates backend_cls.settings_schema.model_validate(
backend_settings) before instantiating, and constructs from the validated
.model_dump(). ValidationError (not TypeError) on mismatch.
- _rebuild_enrichers builds into locals and commits to instance state only on
success — a ValidationError leaves the previously-active enrichers/config/
cache untouched.
- _handle_enrichment_change wraps the rebuild in try/except ValidationError:
logs and returns, keeping the previous backend running (supervisor stays up;
operator fixes the row; next NOTIFY applies cleanly). No cache invalidation
on a failed change.
GUI /enrichment:
- GET skips the outer EnrichmentConfig.backend_settings field and renders a
separate <fieldset> from describe_fields(backend_cls.settings_schema, ...)
for the row's current backend_class. Backend fields namespaced bs_<name>.
- POST reassembles bs_<name> inputs into a backend_settings dict, validates it
against the SUBMITTED backend_class's schema (so errors attach to the right
fields when an operator is mid-switch), then validates the outer
EnrichmentConfig. DB row written only if both pass; otherwise re-renders with
field-level errors against the submitted backend.
- backend_class stays a plain text field (no <select>, no client-side reshape).
form_descriptors: generic `float -> "number"` widget (2 lines, mirrors K.5's
`dict -> "json"`), needed because backend schemas have float fields
(timeout_s, rate_limit_per_sec). Benefits any float field codebase-wide.
DB schema unchanged: backend_settings stays JSONB; validation moved to
use-site. _BACKEND_REGISTRY / _ENRICHER_REGISTRY unchanged beyond schema lookup.
Tests (test_backend_settings_schema.py, 11): schemas exist + extra='forbid';
Navi schema preserves defaults (timeout_s == 10.0); NoOp has zero fields;
build_enrichers raises ValidationError-not-TypeError for the exact 2026-05-20
case; supervisor keeps previous backend on a bad NOTIFY (the incident
scenario); valid NoOp-with-empty-settings applies + invalidates; GUI POST
rejects bad backend_settings without writing + re-renders against submitted
backend; GUI POST writes on valid settings. test_enrichment_config_plumbing
updated for the new context shape (outer_fields/backend_fields).
Verification: full pytest 546 passed, 1 skipped (was 535; +11). grep
subject_for_event/_ADAPTER_REGISTRY and grep 100.64.0./192.168.1. in src both
empty.
Does NOT touch PR L scope (events tab, remaining adapter enrichment_locations),
the DB schema, or the registries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of three PRs for v0.5.0 (J: framework; K: real geocoder backends +
doc revisions; L: operator events tab + per-adapter render + events-map fix).
Design pivot: the Phase 2 "no enrichment, upstream verbatim" reading of
Matt's principle is reframed — consumers can't do follow-up lookups, they
only see what's on the wire, so whatever Central doesn't enrich is
effectively missing downstream. Enrichment is now expected. The producer-doc
§2/§10.1 rewrite lands in PR K; this PR builds the framework PR K documents.
New package src/central/enrichment/:
- base.py Enricher Protocol (name + async enrich(location) -> dict).
- geocoder.py GeocoderEnricher + GeocoderBackend Protocol + the locked
GEOCODER_FIELDS set (name, city, county, state, country,
postal_code, timezone, landclass, elevation_m) + all_null_bundle().
- cache.py EnrichmentCache — stdlib sqlite3 off the event loop via
asyncio.to_thread (no async-sqlite dep). Keyed on
(enricher_name, lat_4dp, lon_4dp); per-enricher TTL (24h
default); fresh connection per op (sqlite3 isn't thread-safe
to share). Cache even all-null; never cache backend failures.
- backends/no_op.py NoOpBackend — all-null bundle, the PR J default.
Provenance: enrichment results land under event.data["_enriched"][<name>];
everything else in data stays upstream verbatim.
Wiring:
- adapter.py enrichment_locations: list[tuple[str,str]] = [] class attr.
Empty (default) = publish as-is, no enrichment.
- config_models.py EnrichmentConfig (enricher_class, backend_class,
backend_settings, cache_ttl_s). Read once at startup.
- supervisor.py build_enrichers() + apply_enrichment(); enrichment runs
after dedup, before wrap_event, in the poll loop. Class-name
registries for enricher/backend resolution (PR K extends).
- firms.py enrichment_locations = [("latitude","longitude")] — pilot.
Enrichment config is read once at supervisor startup; hot-reload is out of
scope for PR J (noted in EnrichmentConfig + build_enrichers docstrings).
Tests (16 new):
- test_enrichment_framework.py (9): parent-dir/table init, cache miss->hit,
TTL expiry, 4dp rounding, nearby-coord collapse, concurrent-set single-row,
backend-failure all-null-not-cached (retries), success cached (one backend
call), all-null cached.
- test_geocoder_enricher.py (5): NoOp all-null, field-set == GEOCODER_FIELDS,
null-coords short-circuit (no backend call), name=="geocoder", sequential
same-coords single backend call.
- test_firms.py (+2): enrichment_locations declared + paths resolve to floats
in a real event (structural, not literal); event through supervisor
apply_enrichment emerges with data._enriched.geocoder == all-null bundle.
Verification: full pytest 495 passed (was 479; +16). grep for
subject_for_event/_ADAPTER_REGISTRY clean. Module imports cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /adapters list view's "⚠️ API Key Missing" chip, the /adapters/{name}
edit form's disabled enable-checkbox, the POST error re-render path, AND
the supervisor's adapter-start precondition all compared the hardcoded
SourceAdapter class attribute `requires_api_key` against `config.api_keys`,
ignoring the per-row `settings[api_key_field]` alias the operator
actually selected via the form.
FIRMS' class attr is `requires_api_key = "firms"`; the api_keys_new.html
placeholder text steers operators toward aliases like `firms_production`
instead, and the FIRMSSettings.api_key_alias field is exactly the
overridable slot that the form writes. The four predicates ignored that
slot, so a working key under any non-default alias was treated as
missing — chip on, checkbox disabled, supervisor refusing to start with
`last_error = "missing api key: firms"`.
Audit: FIRMS is the only adapter today with `requires_api_key != None`.
Every other adapter is unaffected by either the route or supervisor
predicate.
Helper module:
- src/central/api_key_resolver.py exposes:
resolve_api_key_alias(adapter_cls, settings) -> str | None
Pure sync function. Returns the alias to consult, or None when no
key is required. Supervisor uses this directly + its own
get_api_key.
adapter_has_resolved_api_key(conn, adapter_cls, settings) -> (bool, alias)
Async wrapper that runs the SELECT 1 against config.api_keys.
The three GUI routes use this.
Resolution: settings[api_key_field] when set to a non-empty str,
otherwise the class-attr default.
Four call sites swapped:
- routes.py:adapters_list (/adapters list — warning chip)
- routes.py:adapters_edit_form (/adapters/{name} edit GET — disabled checkbox)
- routes.py:adapters_edit_submit (POST error re-render)
- supervisor.py:_start_adapter (adapter-start precondition)
Side-effect tests/test_adapters.py fix:
- TestAdaptersJsonbRegression::test_adapters_edit_fetches_api_keys_into_context
used `AsyncMock()` (no return_value) for mock_conn.__aexit__. AsyncMock
without a return_value yields a MagicMock — which is truthy, and the
async context manager protocol reads truthy from __aexit__ as
"exception suppressed." That silently swallowed any error inside
`async with` blocks. The route refactor moved an assignment inside the
one async with at site 2, so a swallowed mock error left the variable
unbound. Fixed: `AsyncMock(return_value=None)` + a comment so the next
person doesn't re-introduce the bug. fetchval mock added because the
resolver now issues it (the swallowed exception previously hid the
missing mock).
Verification:
- pytest: 479 passed (was 469; +10 new resolver tests).
- grep -rn "adapter_cls.requires_api_key" /opt/central/src returns only
the new helper (2 lines, same file).
- Resolver against live FIRMS settings: resolved_alias='firms_production',
has_key=True, api_key_missing=False -> NO warning chip, checkbox
CLICKABLE.
- Supervisor on live CT104: FIRMS flipped enabled=true via DB UPDATE;
supervisor started the adapter with `api_key_present: true,
api_key_alias: 'firms_production'`; last_error cleared from "missing
api key: firms" -> NULL; two satellite polls completed (VIIRS_SNPP_NRT
477 features, VIIRS_NOAA20_NRT 400 features); 869 new events published
to JetStream.
NOTE: This commit's verification flipped FIRMS to enabled=true in the
running config — the adapter is now actively polling. Pause via the UI
if that's not intended for now; the bug fix itself does not require
FIRMS to be enabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
Fixup 1 — Contract section appended to SourceAdapter.preview_for_settings's
docstring. Override authors read adapter.py, not routes.py, so the contract
(pure function of settings; open your own short-lived aiohttp session; None
vs [] semantics) belongs on the base method, not on the GUI stub class.
Fixup 2 — _adapter_preview.html distinguishes [] from None. Previously the
elif test was truthiness (`elif preview_rows`) which collapsed both into
"render nothing". Now uses `elif preview_rows is not none` and special-cases
the empty-list case inside: legend "Preview (0 rows)" with no table; None
still renders nothing at all. Lets adapters signal "query ran, matched zero"
distinctly from "preview not meaningful".
Tests +1:
- test_partial_renders_empty_list — [] yields "Preview (0 rows)" legend,
no table, no headers. Distinct from the existing None case.
Acceptance:
- 27/27 targeted (preview_hook +1 new, nwis, stream_registry).
- 458/458 full suite.
- (b) framework GUI dir still has zero adapter-name branches.
Adds an optional async hook on SourceAdapter so any adapter can surface a
settings-driven preview on its /adapters/<name> edit page. The framework
renders the result generically as a table — no adapter-name branches in
GUI templates or route code.
Framework changes:
- src/central/adapter.py: new async preview_for_settings(self, settings)
on the base class, default returns None. Adapters opt in by overriding;
non-overriding adapters render unchanged.
- src/central/gui/routes.py: GET /adapters/{name} instantiates the adapter
with a no-op _PreviewConfigStore stub and a /dev/null cursor path (GUI
has no live ConfigStore), constructs settings_obj via the schema, and
calls preview_for_settings inside a try/except. Result lands in template
context as preview_rows / preview_error.
- src/central/gui/templates/_adapter_preview.html: new partial. Generic
table with columns derived from the first dict's keys; error banner
mirrors the existing last_error article style.
- src/central/gui/templates/adapters_edit.html: one-line include between
the Region fieldset and Save/Cancel.
NWIS opt-in:
- New NWIS_MONITORING_LOCATIONS_URL constant and _PREVIEW_LIMIT cap of 50.
- preview_for_settings returns None when region is None, otherwise one-shot
fetches monitoring-locations within the bbox via a fresh aiohttp session.
Must work even when adapter is not started -- the GUI process never calls
startup(). Returns list[dict] with the contract column order: site_id,
name, site_type, state. Errors propagate so the framework can render the
operator-visible banner.
- HTTP call factored into _fetch_preview_text so tests mock cleanly.
Tests (7 new):
- tests/test_preview_hook.py: default returns None; partial renders list
with correct headers/rows/count; partial renders error banner; partial
renders empty when both context values are None.
- tests/test_nwis.py adds TestNWISPreview: returns None without region,
returns rows with correct column order, propagates HTTP errors.
Verification:
- 457/457 full suite green (was 450; +7 new tests).
- Live /adapters/nwis preview returns 50 rows with the contract keys
against the current production Iowa bbox.
- /adapters/eonet preview_for_settings returns None via base default --
proves framework is duck-typed, no NWIS-specific code in framework.
NASA WaterData OGC API v0 (latest-continuous collection) — polls configured
parameter codes within an operator-set bbox and publishes on the new
CENTRAL_HYDRO stream.
- Subject: central.hydro.<parameter_code>.<agency>.<bare_site_no>
(e.g. central.hydro.00060.usgs.05420500). The agency/site decomposition
lives in a single _subject_tokens_for_id helper.
- Default parameter codes: 00060 (discharge), 00065 (gage height),
00010 (water temperature). Operator-tunable; single SoT in
_DEFAULT_PARAMETER_CODES — no parallel literals.
- Composite dedup: nwis:<monitoring_location_id>:<param>:<time_iso>.
Prefix kept in dedup key for cross-agency uniqueness.
- Pagination: follows OGC 'rel=next' link until absent (cursor-based).
- Region bbox is REQUIRED in practice; adapter logs WARN at startup if
region is None (does not refuse to start).
- New stream CENTRAL_HYDRO added to streams.py registry (one line).
Retention mirrors CENTRAL_DISASTER (7 days, 1 GiB).
- No removal pattern in v1 — sites are static; missing data is the signal.
Upstream divergences from the original spec brief, caught by pre-build curl:
- Collection is 'latest-continuous', not 'instantaneous-values'.
- Site filter param is 'monitoring_location_id' (singular), not
'monitoring_locations_id' (plural).
- Site identifier requires agency prefix in queries (USGS-NNNNN).
- feature.id is a per-record UUID, not stable; dedup uses joint key.
Ships disabled; operator enables via GUI after setting a bbox.
Adds the NASA Earth Observatory Natural Event Tracker (EONET v3) adapter,
publishing on the existing CENTRAL_DISASTER stream under
central.disaster.eonet.<category>.global subjects.
- One Central event per EONET event id; geo = most-recent geometry point.
- Composite dedup key (eonet:<id>:<latest_geometry_date_iso>) — timeline
advance re-publishes, idle re-poll suppresses.
- category_allowlist defaults to all 13 upstream categories; operator opts
OUT per-category if GDACS overlap (wildfires/floods/severeStorms/volcanoes)
produces unwanted dupes on gdacs.* subjects.
- camelCase upstream IDs (seaLakeIce, dustHaze, etc.) mapped to
lower_snake_case subject components by a single _subject_category helper.
- Country resolves to literal 'global' (no reverse-geocode in v1).
- Fall-off: missing-from-feed event emits central.disaster.eonet.<cat>.removed.global,
subtype before 'removed' per §8 canonical pattern.
Adapter ships disabled; operator enables via GUI.
Eliminates the duplication that has been hand-bumped through PRs B, C, D, E.
Adding a stream is now one StreamEntry in src/central/streams.py + one
migration row in config.streams. supervisor STREAM_SUBJECTS / archive
STREAMS / gui DASHBOARD_STREAMS all derive at import time. No drift
possible because there is one source.
Pure refactor; no behavior change. Runtime verified: derived structures
are byte-equivalent to the previous literal definitions.
src/central/streams.py (new):
@dataclass(frozen=True)
class StreamEntry:
name: str
subject_filter: str
event_bearing: bool = True # archive consumes from this stream
dashboard: bool = True # GUI dashboard surfaces this stream
STREAMS: list[StreamEntry] = [
StreamEntry("CENTRAL_WX", "central.wx.>"),
StreamEntry("CENTRAL_FIRE", "central.fire.>"),
StreamEntry("CENTRAL_QUAKE", "central.quake.>"),
StreamEntry("CENTRAL_SPACE", "central.space.>"),
StreamEntry("CENTRAL_DISASTER", "central.disaster.>"),
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
]
Consumers derive:
supervisor.STREAM_SUBJECTS = {s.name: [s.subject_filter] for s in STREAMS}
(includes META: supervisor must create every stream in JetStream)
archive.STREAMS = [(s.name, s.subject_filter) for s in STREAMS if s.event_bearing]
(excludes META: status messages, not events)
gui.DASHBOARD_STREAMS = [s.name for s in STREAMS if s.dashboard]
To resolve the name collision between the registry STREAMS and the
existing archive.STREAMS public symbol, archive.py imports the registry
under an alias: from central.streams import STREAMS as STREAM_REGISTRY.
The archives STREAMS surface (the tuple-list) is unchanged for callers.
Same alias used in supervisor.py and gui/routes.py for symmetry.
Migration files unchanged. config.streams keeps seeding retention/bytes --
operator-tunable ops state, separate SoT from the structural mapping.
Tests:
Dropped from test_archive_multi_stream.py (7, all tautological vs. registry):
test_streams_list_has_five_entries (magic-number count)
test_streams_contains_central_wx / fire / quake / space / disaster
test_streams_excludes_central_meta
Dropped from test_dashboard.py:
`assert len(streams) == 6` line inside test_single_stream_failure_doesnt_crash_card
(the test itself stays; only the magic-number assertion is removed)
Added in test_stream_registry.py (8 invariant tests):
test_stream_names_unique
test_subject_filters_unique
test_subject_filter_central_prefix_wildcard
test_meta_is_only_non_event_bearing
test_supervisor_stream_subjects_includes_meta
test_supervisor_stream_subjects_includes_all
test_archive_streams_excludes_non_event_bearing
test_dashboard_streams_matches_dashboard_flag
The new tests assert properties (uniqueness, format, derivation correctness),
not literals. Future stream additions need zero new test code -- every
invariant automatically covers them.
Note: test file named tests/test_stream_registry.py (not test_streams.py)
to avoid colliding with the pre-existing tests/test_streams.py, which
covers the GUI streams-management page.
Full suite: 427 passed (was 426 on main: -7 dropped + 8 added).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per handoff §9 the removed-event convention is
central.<domain>.<subtype>.removed.<geo> -- WFIGS uses
central.fire.incident.removed.<state>. GDACS tombstones were emitting
central.disaster.removed.<country> with the eventtype only in the
category (disaster.removed.wf), which would silently miss type-filtered
subscribers (e.g. central.disaster.wf.> would not see WF removals).
Fix:
- poll() iscurrent=false branch and missing-from-feed loop both set
category=f"disaster.{eventtype.lower()}.removed" (eventtype before
the .removed token, matching the live-event subject hierarchy).
- subject_for() detects parts[-1] == "removed" and emits
central.disaster.<eventtype>.removed.<country>.
Tests updated:
test_fall_off_iscurrent_false now asserts category disaster.wf.removed
and subject central.disaster.wf.removed.greece.
test_fall_off_missing_from_feed adds the category assertion.
Both tombstone-collection filters flip from startswith("disaster.removed")
to endswith(".removed") for general-shape coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the GDACS (Global Disaster Alert and Coordination System) adapter
against the self-describing framework. Polls https://www.gdacs.org/xml/rss.xml
every 600s, parses the RSS items, and publishes to a new CENTRAL_DISASTER
JetStream stream on central.disaster.<eventtype_lower>.<country_lower>.
Locked decisions:
- Keep: WF, DR, FL, VO, TC. Drop: EQ (USGS canonical on central.quake.>),
plus any future-unknown eventtype.
- Filter via settings_schema event_types: list[str] so operators can
re-allow without a code change.
- Dedup by RSS guid (format <eventtype><eventid>, stable across reissue).
- Severity from gdacs:alertlevel (Green=1, Orange=2, Red=3, default 0).
- Fall-off uses GDACS gdacs:iscurrent=false as explicit tombstone signal,
with a fallback for items that vanish entirely from the feed. Tombstones
publish on disaster.removed.<eventtype>.<country>.
- Geo: centroid from geo:Point, bbox from gdacs:bbox (reordered to Geo
(minLon, minLat, maxLon, maxLat)), primary_region from gdacs:iso3.
CENTRAL_DISASTER stream: 7d retention, 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE / CENTRAL_SPACE. Migrations 020 (adapter row,
enabled=false, default event_types in settings) and 021 (stream seed).
STREAM_SUBJECTS, archive STREAMS, GUI DASHBOARD_STREAMS each pick up
the new stream.
Tests: 14 new in tests/test_gdacs.py using frozen RSS fixtures with WF/DR/EQ/XX
items (covering normalization, EQ drop, unknown drop, settings override,
guid dedup, iscurrent=false tombstone, missing-from-feed tombstone,
helper boundaries). Stream-count assertions bumped 4->5 and 5->6 for
the new stream (anti-pattern noted; queued as a follow-up PR E.5).
+1 membership test test_streams_contains_central_disaster.
Full suite: 426 passed.
End-to-end on CT104: 48 events published on first poll (44 disaster.wf +
4 disaster.fl), zero EQ events, all subjects under central.disaster.>
with lowercase-hyphenated country suffixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent adapters sharing src/central/adapters/swpc_common.py,
mirroring the WFIGS two-adapter pattern. Each adapter has its own row in
config.adapters (ships disabled), its own cadence, and its own dedup
state, so operators can independently enable/disable and so a broken
upstream endpoint does not silently mask a healthy one.
Subjects:
swpc_alerts -> central.space.alert.<product_id_lower>
swpc_kindex -> central.space.kindex
swpc_protons -> central.space.proton_flux
Dedup keys:
alerts: product_id + issue_datetime
kindex: time_tag
protons: time_tag + energy
Severity: G-scale on product_id for K0[5-9][AW] alerts (G1-G5 -> 1-4),
G-scale on Kp for kindex, 0 for protons (raw flux carried in event.data).
No geo on any SWPC events (centroid=None, regions=[], primary_region=None).
No fall-off detection for alerts -- a single 115-row sample cannot confirm
whether alerts disappear from the upstream JSON when expired; deferred to
a later pass after 24h of observation.
CENTRAL_SPACE stream seeded with 7-day retention / 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE. STREAM_SUBJECTS, archive STREAMS, and
DASHBOARD_STREAMS each pick up the new stream.
Tests: 16 new cases in tests/test_swpc.py using real-shape frozen JSON
fixtures (alerts product_ids EF3A/K05A/K07A; kindex Kp boundaries; protons
composite dedup). Two existing tests updated for the new stream count
(test_archive_multi_stream.test_streams_list_has_three_entries renamed to
_has_four_entries; test_dashboard expects 5 streams not 4); added a
test_streams_contains_central_space companion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug fixes:
1. Wire is_published/mark_published/bump_last_seen into poll() loop
- Skip already-published items, bump TTL to prevent sweep
- Mark published after yield to track new items
2. Add conditional fetch support (If-Modified-Since, If-None-Match)
- Store Last-Modified/ETag from responses
- Send conditional headers on subsequent requests
- Handle 304 Not Modified gracefully (return empty list)
3. Document state parsing rationale in docstring
- Description has structured State: field vs unreliable title prefixes
Tests added:
- test_dedup_in_poll_loop: verify second poll yields 0 for same items
- test_conditional_304_yields_zero: verify 304 returns empty list
- test_conditional_headers_sent_after_first_poll: verify headers sent
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
InciWeb adapter for RSS-based wildfire narrative updates:
- Parse DMS coordinates from description text
- Extract state name and map to 2-letter code
- Strip HTML tags and decode entities
- Bbox filtering for regional focus
- Dedup via published_ids table (14-day sweep)
- Category: fire.narrative.inciweb
- Subject: central.fire.narrative.inciweb.<state>
Includes migration 017 and 15 unit tests.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
WFIGS returns ISO 3166-2 state codes (US-MT) and 2-letter incident
type codes (WF, RX). Normalize at parse boundary:
- normalize_state: strips US- prefix (US-MT -> MT)
- normalize_incident_type: maps codes to names (WF -> wildfire)
Fixes:
- category was fire.incident.wf, now fire.incident.wildfire
- region was US-US-MT-GLACIER, now US-MT-GLACIER
Both raw and normalized values stored in event.data.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two new adapters for wildfire data from NIFC WFIGS:
- wfigs_incidents: Active fire incident locations
- wfigs_perimeters: Active fire perimeter polygons
Features:
- IRWIN GUID dedup via is_published/mark_published
- Fall-off detection with removal events when fires exit current
- Bbox post-filtering with shapely polygon intersection
- Severity mapping from DailyAcres (0-4 scale)
- Subject hierarchy: central.fire.<layer>.<state>.<county>
Ships disabled by default; operators enable via GUI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add set_adapter_last_error method to ConfigStore for setting/clearing
adapter error states
- Add API key precondition check in supervisor._start_adapter that:
- Checks if adapter has requires_api_key attribute
- Looks up the key via config_store.get_api_key
- Sets last_error and returns early if key is missing
- Clears last_error when adapter successfully starts
- Update adapters_list handler to compute api_key_missing flag
for each adapter and pass to template
- Update adapters_edit_form handler to compute api_key_missing
and requires_api_key_alias for template context
- Update adapters_list.html to show warning badge when api_key_missing
- Update adapters_edit.html to show warning article and disable
Enable checkbox when api_key_missing
- Add tests for new functionality
- Fix test mocks to include requires_api_key and last_error fields
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change 5: Move contact_email validation to Pydantic schema
- NWSSettings now uses Field(pattern=...) for email validation
- Pydantic pattern validation catches invalid emails
- No special handler branch needed in routes.py
Change 6: Generic api_key_field mechanism
- Add api_key_field attribute to SourceAdapter base class
- FIRMSAdapter sets api_key_field="api_key_alias"
- GET handlers swap widget to "api_key_select" when field matches
- POST handlers validate against state.api_keys generically
- Templates use new api_key_select widget branch
- adapters_edit handlers now fetch and pass api_keys to context
Tests added:
- test_invalid_contact_email_via_pydantic_pattern
- test_invalid_api_key_alias_generic
- test_api_key_field_none_no_check
- test_adapters_edit_fetches_api_keys_into_context
Zero field.name hardcoded branches remain in routes.py or templates.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove dead _get_valid_satellites/_get_valid_feeds calls from error render
- Replace hardcoded adapter list with dynamic wizard_adapters discovery
- Use RegionConfig model validation instead of hand-rolled bounds check
- Add Pydantic settings validation after field parsing to catch Literal violations
- Add TestSetupAdaptersErrorRerender with cadence and region error tests
Fixes error path gaps that would cause NameError on form re-render.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Literal type support to form_descriptors.py
- Literal fields map to select widget
- list[Literal] fields map to checkboxes widget
- Options list extracted from Literal type args
- Update FIRMS adapter: satellites is now list[Literal[...]]
- Update USGS adapter: feed is now Literal[...]
- Refactor wizard to use wizard_order for adapter filtering
- Replace hardcoded adapter lists with dynamic discovery
- Remove _get_valid_satellites() and _get_valid_feeds() helpers
- Generic field parsing using describe_fields() pattern
- Update templates for generic widget rendering
- Add select/checkboxes widgets to adapters_edit.html
- Update tests for new widget types
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Make migration 015 idempotent with IF NOT EXISTS
2. Remove hardcoded cadence range from routes.py and template:
- Added ge=10 constraint to AdapterConfig.cadence_s field
- Removed manual 60-3600 check from routes.py POST handler
- Validate cadence using AdapterConfig field metadata
- Removed min/max attributes from template input
3. Move discover_adapters to its own module:
- Created src/central/adapter_discovery.py
- Updated supervisor.py to import from adapter_discovery
- Updated routes.py to import from adapter_discovery
- GUI no longer transitively imports nats or stream_manager
4. Remove dead code branch in form_descriptors.py:
- Removed unreachable RegionConfig check (already handled earlier)
- Improved error message for unsupported nested types
5. Updated test_adapters.py:
- Changed invalid cadence test from 30 to 5 (below ge=10)
- Updated assertion to check for "10" in error message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement Central 2-A2: generic adapter edit form feature.
- Add form_descriptors.py with describe_fields() and FieldDescriptor
- Maps Pydantic types to HTML widgets (text, number, checkbox, csv, region)
- Handles Optional types by recursively resolving inner type
- Uses PydanticUndefined handling for proper default values
- Update routes.py GET/POST handlers:
- Use cached _adapter_classes() for adapter class lookup
- Generate field descriptors from adapter settings_schema
- Parse form values based on widget type in POST handler
- Validate settings via Pydantic ValidationError
- Update adapters_edit.html template:
- Render form dynamically from field descriptors
- Support all widget types (text, number, checkbox, csv, region)
- Use adapter.display_name and adapter.description from class
- Delete per-adapter templates:
- adapters_edit_nws.html
- adapters_edit_firms.html
- adapters_edit_usgs_quake.html
- Add tests/test_form_descriptors.py with comprehensive coverage
- Update tests/test_adapters.py to include last_error in mock rows
- Update tests/test_region_picker.py to include last_error in mock rows
Adding a new adapter no longer requires GUI template work.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace settings_schema classmethod with Pydantic model class attribute
- Add display_name, description, requires_api_key, wizard_order, default_cadence_s
- Remove stream_name from adapters (JetStream routes by subject filter)
- Define NWSSettings, FIRMSSettings, USGSQuakeSettings Pydantic models
- Make discover_adapters() public with error handling
- Move adapter registry to Supervisor instance (self._adapters)
- Add subject_for tests for all 6 quake magnitude tiers
- Fix test_supervisor_integration to use injected mock adapters
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stream_name, subject_for(), and settings_schema() to SourceAdapter ABC
- Implement all three methods in NWSAdapter, FIRMSAdapter, USGSQuakeAdapter
- Replace manual _ADAPTER_REGISTRY with pkgutil.iter_modules auto-discovery
- Remove subject_for_event from models.py (each adapter owns its subject logic)
- Update supervisor to use adapter.subject_for(event) instead of helper
- Fix quake events going to wrong stream (was publishing to CENTRAL_WX)
- Update test files to use adapter methods
This fixes the quake stream bug where events were published to
central.wx.alert.us.unknown instead of central.quake.event.<tier>.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: events feed UX iteration - colors, popups, viewport filter
A. Color-code polygons by adapter (NWS amber, FIRMS red, USGS violet)
B. Click popup on polygons showing time + adapter + category + subject
C. Map viewport drives spatial filter - pan/zoom updates table via HTMX
D. Add legend showing adapter color mapping
E. Remove draw-bbox control, region inputs now hidden (auto-managed)
Template changes:
- _events_rows.html: add data-adapter, data-category, data-time, data-subject
- events_list.html: ADAPTER_COLORS mapping, bindPopup, moveend handler
Test: verify template renders adapter/category/subject for JS consumption
* fix: remove isoformat() call on already-formatted time string
* feat: full events feed UX iteration
A. Color-code polygons by adapter with legend
B. Click popup on polygons with "View details" link
C. Viewport-driven spatial filter - pan/zoom updates table via HTMX
Map never auto-fits after initial load (user controls viewport)
D. Expandable row details showing full event data payload
Changes:
- _events_rows.html: add data-event-id, expand button, detail row
- events_list.html: eventLayerGroup pattern, buildPopup, rebindEventLayers
Fit to results button, expand/collapse handlers, CSS.escape for IDs
* fix: add programmaticMove flag to prevent viewport refresh loop
Suppress moveend handler during fitBounds/setView calls to prevent
feedback loop: fitBounds -> moveend -> applyViewportFilter -> HTMX
swap -> repeat.
* fix: map never auto-fits - user controls viewport
- Disable initial fitToAllLayers on page load
- Remove fitBounds/setView from row click handler
- Map only moves when user pans/zooms
- Table filters based on visible viewport
* fix: map shows all events always, only table filters
Map polygons are drawn once on load and never cleared/redrawn.
HTMX swap only updates the table, not the map layers.
User viewport is fully preserved.
* fix: use htmx.trigger instead of dispatchEvent for HTMX swap
dispatchEvent(submit) was triggering native form submission (full page
reload). htmx.trigger() properly triggers HTMX swap.
Also re-enable initial rebindEventLayers so polygons load on first render.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
* feat(gui): add events feed frontend with map and filters
GET /events: Full page with filter form, table, and Leaflet map
GET /events/rows: HTMX fragment for table updates
Features:
- Filterable by adapter, category, time range, region bbox
- Cursor-based pagination with Next button
- Leaflet map showing event geometries
- Click/hover row highlights geometry on map
- Draw rectangle on map to filter by region
- Validation errors shown as banner, not 400
- Events link added to nav between Adapters and Streams
Refactored events query into shared helper for JSON and HTML routes.
Tests: 14 new tests covering filters, fragments, geometry handling.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(events): use shared helpers for /events.json, fix tests
- Refactor /events.json to use _parse_events_params and _fetch_events
helpers, removing ~200 lines of duplicate query logic
- Delete smoke test (test_events_unauthenticated_redirects) that had
no assertions
- Add TestCrossEndpointParity: verify /events.json and /events return
identical results with same params, test category filter and cursor
pagination on both endpoints
- Add TestErrorSemantics: verify /events.json returns 400 on bad params
while /events returns 200 with error banner (intentional API vs HTML
divergence)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add real assertions to since/until and region filter tests
Replace trivial status_code==200 assertions with checks that verify
the filter values were actually parsed and passed to the template.
These tests now fail if the handler ignores the filter parameters.
* fix: remove double-escaping from data-geometry attribute
tojson already produces HTML-attribute-safe JSON. The extra |e filter
was double-escaping, causing JSON.parse to fail in the browser JS.
Switch to single-quoted attribute to avoid conflicts with JSON double
quotes.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(wizard): implement deferred-commit pattern for setup wizard
Replace the current "POST each step -> DB write -> redirect" architecture
with "collect values across steps in a signed cookie, commit everything
in one transaction at Finish."
Key changes:
- Add wizard.py: WizardState dataclass and cookie helpers
- csrf.py: Add reuse_or_generate_pre_auth_csrf helper
- routes.py: All wizard handlers now use cookie state, no DB writes until finish
- middleware.py: Cookie-based wizard step routing instead of DB queries
- setup_operator.html: Remove "Operator Already Configured" branch
Benefits:
- Back navigation works: can return to any step and edit values
- Atomic commit: all DB writes happen in single transaction at finish
- No orphaned state: failed wizard leaves no DB artifacts
- Simpler auth: pre-auth CSRF for all 5 steps (no session until finish)
Tests updated for new behavior. 287 tests passing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(templates): correct SRI hashes for leaflet.draw assets
The integrity hashes for leaflet.draw.css and leaflet.draw.js were
incorrect, causing browsers to silently block these resources. This
broke the Leaflet.draw toolbar and map rendering for FIRMS/USGS
adapter region pickers.
Updated both setup_adapters.html and adapters_edit.html with the
correct sha512 hashes computed from the actual CDN files.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): return 204 for browser-noise paths to prevent CSRF races
Browser requests for /favicon.ico, /apple-touch-icon.png, etc. were
triggering parallel GET requests that could race with form loads,
causing CSRF token rotation issues.
Added BROWSER_NOISE_PATHS constant and early 204 response in both
SetupGateMiddleware and SessionMiddleware to short-circuit these
requests before any cookie/token handling occurs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
GET /events.json with cursor-based pagination and filtering:
- Filter by adapter, category, since/until, region bbox
- Cursor pagination via (time DESC, id DESC) ordering
- Returns events with GeoJSON geometry parsed as objects
- Validation returns 400 with clear error messages
Migration 014 adds composite index for efficient pagination.
Tests: 17 new tests covering filters, pagination, validation.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): implement first-run setup wizard (1b-8)
Add a 5-step setup wizard that replaces the single-step /setup:
1. Create Operator - create initial operator account
2. System Settings - configure map tile URL and attribution
3. API Keys - optionally add API keys for adapters
4. Configure Adapters - enable/disable adapters with region picker
5. Finish Setup - review and complete setup
Key changes:
- Update middleware to handle wizard URL structure and step routing
- Add wizard routes for each step with proper auth checks
- Create new templates using base_wizard.html for consistent styling
- Add audit events for system.update and setup.complete
- Update tests for new middleware behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle CSRF errors on wizard paths
Update csrf_exception_handler to re-render wizard forms with error
message instead of redirecting to /login when CSRF validation fails.
- /setup/operator: re-render with error
- /setup/system: re-render with current system values + error
- /setup/keys: re-render with current keys list + error
- /setup/adapters: re-render with current adapter config + error
- /setup/finish: re-render with summary data + error
- /setup: redirect to /setup (middleware routes to appropriate step)
Add error display to setup_keys.html and setup_finish.html templates.
Add 7 new CSRF handler tests for wizard paths.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): region picker render + click-to-draw
Bug A: Maps render blank on /setup/adapters for FIRMS and USGS
because Leaflet computed zero dimensions before container layout
settled. Fix: add setTimeout invalidateSize() after map creation.
Bug B: No click-to-draw functionality - only drag corners. Fix:
add L.Control.Draw for rectangle drawing with CREATED event handler
to replace existing rectangle.
Both fixes applied to:
- setup_adapters.html (wizard inline JS)
- _region_picker.html (standalone edit page)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle revisiting /setup/operator after operator created
When an operator already exists, /setup/operator now shows a
confirmation page instead of the create form. This prevents:
- Unique constraint violations on duplicate username
- Silent creation of duplicate operators
GET /setup/operator: queries config.operators; if any exist,
renders confirmation state with existing_operator context.
POST /setup/operator: checks operator count before INSERT; if
non-zero, renders confirmation state without inserting.
Template updated with conditional to show "Operator Already
Configured" message when existing_operator is set.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(csrf): replace fastapi-csrf-protect with session-bound CSRF
Fixes CSRF race condition where every GET rotated the CSRF token,
causing POST failures when users had multiple tabs or slow connections.
Changes:
- Remove fastapi-csrf-protect dependency
- Add session-bound CSRF tokens stored in config.sessions table
- Add pre-auth CSRF for unauthenticated routes (/login, /setup/operator)
- Add csrf.py module for pre-auth token generation/validation
- Update routes to use new CSRF token handling
- Add migration 013 to add csrf_token column to sessions
The session-bound approach ensures CSRF tokens remain stable for the
duration of a session, eliminating the race condition.
Note: Route tests (test_wizard.py, test_adapters.py, etc.) need
refactoring to mock get_settings() instead of CsrfProtect dependency.
Core auth/CSRF handler tests pass (74 tests).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(csrf): update test suite for session-bound CSRF tokens
- Add CSRF fixtures to conftest.py for pre-auth and session CSRF
- Update test_wizard.py: use bypass_pre_auth_csrf and patch_route_settings
- Update test_adapters.py: set request.state.csrf_token and form mock data
- Update test_api_keys.py: add CSRF token to form data for POST routes
- Update test_streams.py: change return_value to side_effect for CSRF support
- Update test_region_picker.py: add CSRF token handling
- Update test_config_store.py: set CENTRAL_CSRF_SECRET env var in fixture
All 285 tests now pass with session-bound CSRF validation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Matt Johnson <mj@k7zvx.com>
