Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of three PRs for v0.5.0 (J: framework; K: real geocoder backends +
doc revisions; L: operator events tab + per-adapter render + events-map fix).
Design pivot: the Phase 2 "no enrichment, upstream verbatim" reading of
Matt's principle is reframed — consumers can't do follow-up lookups, they
only see what's on the wire, so whatever Central doesn't enrich is
effectively missing downstream. Enrichment is now expected. The producer-doc
§2/§10.1 rewrite lands in PR K; this PR builds the framework PR K documents.
New package src/central/enrichment/:
- base.py Enricher Protocol (name + async enrich(location) -> dict).
- geocoder.py GeocoderEnricher + GeocoderBackend Protocol + the locked
GEOCODER_FIELDS set (name, city, county, state, country,
postal_code, timezone, landclass, elevation_m) + all_null_bundle().
- cache.py EnrichmentCache — stdlib sqlite3 off the event loop via
asyncio.to_thread (no async-sqlite dep). Keyed on
(enricher_name, lat_4dp, lon_4dp); per-enricher TTL (24h
default); fresh connection per op (sqlite3 isn't thread-safe
to share). Cache even all-null; never cache backend failures.
- backends/no_op.py NoOpBackend — all-null bundle, the PR J default.
Provenance: enrichment results land under event.data["_enriched"][<name>];
everything else in data stays upstream verbatim.
Wiring:
- adapter.py enrichment_locations: list[tuple[str,str]] = [] class attr.
Empty (default) = publish as-is, no enrichment.
- config_models.py EnrichmentConfig (enricher_class, backend_class,
backend_settings, cache_ttl_s). Read once at startup.
- supervisor.py build_enrichers() + apply_enrichment(); enrichment runs
after dedup, before wrap_event, in the poll loop. Class-name
registries for enricher/backend resolution (PR K extends).
- firms.py enrichment_locations = [("latitude","longitude")] — pilot.
Enrichment config is read once at supervisor startup; hot-reload is out of
scope for PR J (noted in EnrichmentConfig + build_enrichers docstrings).
Tests (16 new):
- test_enrichment_framework.py (9): parent-dir/table init, cache miss->hit,
TTL expiry, 4dp rounding, nearby-coord collapse, concurrent-set single-row,
backend-failure all-null-not-cached (retries), success cached (one backend
call), all-null cached.
- test_geocoder_enricher.py (5): NoOp all-null, field-set == GEOCODER_FIELDS,
null-coords short-circuit (no backend call), name=="geocoder", sequential
same-coords single backend call.
- test_firms.py (+2): enrichment_locations declared + paths resolve to floats
in a real event (structural, not literal); event through supervisor
apply_enrichment emerges with data._enriched.geocoder == all-null bundle.
Verification: full pytest 495 passed (was 479; +16). grep for
subject_for_event/_ADAPTER_REGISTRY clean. Module imports cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /adapters list view's "⚠️ API Key Missing" chip, the /adapters/{name}
edit form's disabled enable-checkbox, the POST error re-render path, AND
the supervisor's adapter-start precondition all compared the hardcoded
SourceAdapter class attribute `requires_api_key` against `config.api_keys`,
ignoring the per-row `settings[api_key_field]` alias the operator
actually selected via the form.
FIRMS' class attr is `requires_api_key = "firms"`; the api_keys_new.html
placeholder text steers operators toward aliases like `firms_production`
instead, and the FIRMSSettings.api_key_alias field is exactly the
overridable slot that the form writes. The four predicates ignored that
slot, so a working key under any non-default alias was treated as
missing — chip on, checkbox disabled, supervisor refusing to start with
`last_error = "missing api key: firms"`.
Audit: FIRMS is the only adapter today with `requires_api_key != None`.
Every other adapter is unaffected by either the route or supervisor
predicate.
Helper module:
- src/central/api_key_resolver.py exposes:
resolve_api_key_alias(adapter_cls, settings) -> str | None
Pure sync function. Returns the alias to consult, or None when no
key is required. Supervisor uses this directly + its own
get_api_key.
adapter_has_resolved_api_key(conn, adapter_cls, settings) -> (bool, alias)
Async wrapper that runs the SELECT 1 against config.api_keys.
The three GUI routes use this.
Resolution: settings[api_key_field] when set to a non-empty str,
otherwise the class-attr default.
Four call sites swapped:
- routes.py:adapters_list (/adapters list — warning chip)
- routes.py:adapters_edit_form (/adapters/{name} edit GET — disabled checkbox)
- routes.py:adapters_edit_submit (POST error re-render)
- supervisor.py:_start_adapter (adapter-start precondition)
Side-effect tests/test_adapters.py fix:
- TestAdaptersJsonbRegression::test_adapters_edit_fetches_api_keys_into_context
used `AsyncMock()` (no return_value) for mock_conn.__aexit__. AsyncMock
without a return_value yields a MagicMock — which is truthy, and the
async context manager protocol reads truthy from __aexit__ as
"exception suppressed." That silently swallowed any error inside
`async with` blocks. The route refactor moved an assignment inside the
one async with at site 2, so a swallowed mock error left the variable
unbound. Fixed: `AsyncMock(return_value=None)` + a comment so the next
person doesn't re-introduce the bug. fetchval mock added because the
resolver now issues it (the swallowed exception previously hid the
missing mock).
Verification:
- pytest: 479 passed (was 469; +10 new resolver tests).
- grep -rn "adapter_cls.requires_api_key" /opt/central/src returns only
the new helper (2 lines, same file).
- Resolver against live FIRMS settings: resolved_alias='firms_production',
has_key=True, api_key_missing=False -> NO warning chip, checkbox
CLICKABLE.
- Supervisor on live CT104: FIRMS flipped enabled=true via DB UPDATE;
supervisor started the adapter with `api_key_present: true,
api_key_alias: 'firms_production'`; last_error cleared from "missing
api key: firms" -> NULL; two satellite polls completed (VIIRS_SNPP_NRT
477 features, VIIRS_NOAA20_NRT 400 features); 869 new events published
to JetStream.
NOTE: This commit's verification flipped FIRMS to enabled=true in the
running config — the adapter is now actively polling. Pause via the UI
if that's not intended for now; the bug fix itself does not require
FIRMS to be enabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
Fixup 1 — Contract section appended to SourceAdapter.preview_for_settings's
docstring. Override authors read adapter.py, not routes.py, so the contract
(pure function of settings; open your own short-lived aiohttp session; None
vs [] semantics) belongs on the base method, not on the GUI stub class.
Fixup 2 — _adapter_preview.html distinguishes [] from None. Previously the
elif test was truthiness (`elif preview_rows`) which collapsed both into
"render nothing". Now uses `elif preview_rows is not none` and special-cases
the empty-list case inside: legend "Preview (0 rows)" with no table; None
still renders nothing at all. Lets adapters signal "query ran, matched zero"
distinctly from "preview not meaningful".
Tests +1:
- test_partial_renders_empty_list — [] yields "Preview (0 rows)" legend,
no table, no headers. Distinct from the existing None case.
Acceptance:
- 27/27 targeted (preview_hook +1 new, nwis, stream_registry).
- 458/458 full suite.
- (b) framework GUI dir still has zero adapter-name branches.
Adds an optional async hook on SourceAdapter so any adapter can surface a
settings-driven preview on its /adapters/<name> edit page. The framework
renders the result generically as a table — no adapter-name branches in
GUI templates or route code.
Framework changes:
- src/central/adapter.py: new async preview_for_settings(self, settings)
on the base class, default returns None. Adapters opt in by overriding;
non-overriding adapters render unchanged.
- src/central/gui/routes.py: GET /adapters/{name} instantiates the adapter
with a no-op _PreviewConfigStore stub and a /dev/null cursor path (GUI
has no live ConfigStore), constructs settings_obj via the schema, and
calls preview_for_settings inside a try/except. Result lands in template
context as preview_rows / preview_error.
- src/central/gui/templates/_adapter_preview.html: new partial. Generic
table with columns derived from the first dict's keys; error banner
mirrors the existing last_error article style.
- src/central/gui/templates/adapters_edit.html: one-line include between
the Region fieldset and Save/Cancel.
NWIS opt-in:
- New NWIS_MONITORING_LOCATIONS_URL constant and _PREVIEW_LIMIT cap of 50.
- preview_for_settings returns None when region is None, otherwise one-shot
fetches monitoring-locations within the bbox via a fresh aiohttp session.
Must work even when adapter is not started -- the GUI process never calls
startup(). Returns list[dict] with the contract column order: site_id,
name, site_type, state. Errors propagate so the framework can render the
operator-visible banner.
- HTTP call factored into _fetch_preview_text so tests mock cleanly.
Tests (7 new):
- tests/test_preview_hook.py: default returns None; partial renders list
with correct headers/rows/count; partial renders error banner; partial
renders empty when both context values are None.
- tests/test_nwis.py adds TestNWISPreview: returns None without region,
returns rows with correct column order, propagates HTTP errors.
Verification:
- 457/457 full suite green (was 450; +7 new tests).
- Live /adapters/nwis preview returns 50 rows with the contract keys
against the current production Iowa bbox.
- /adapters/eonet preview_for_settings returns None via base default --
proves framework is duck-typed, no NWIS-specific code in framework.
NASA WaterData OGC API v0 (latest-continuous collection) — polls configured
parameter codes within an operator-set bbox and publishes on the new
CENTRAL_HYDRO stream.
- Subject: central.hydro.<parameter_code>.<agency>.<bare_site_no>
(e.g. central.hydro.00060.usgs.05420500). The agency/site decomposition
lives in a single _subject_tokens_for_id helper.
- Default parameter codes: 00060 (discharge), 00065 (gage height),
00010 (water temperature). Operator-tunable; single SoT in
_DEFAULT_PARAMETER_CODES — no parallel literals.
- Composite dedup: nwis:<monitoring_location_id>:<param>:<time_iso>.
Prefix kept in dedup key for cross-agency uniqueness.
- Pagination: follows OGC 'rel=next' link until absent (cursor-based).
- Region bbox is REQUIRED in practice; adapter logs WARN at startup if
region is None (does not refuse to start).
- New stream CENTRAL_HYDRO added to streams.py registry (one line).
Retention mirrors CENTRAL_DISASTER (7 days, 1 GiB).
- No removal pattern in v1 — sites are static; missing data is the signal.
Upstream divergences from the original spec brief, caught by pre-build curl:
- Collection is 'latest-continuous', not 'instantaneous-values'.
- Site filter param is 'monitoring_location_id' (singular), not
'monitoring_locations_id' (plural).
- Site identifier requires agency prefix in queries (USGS-NNNNN).
- feature.id is a per-record UUID, not stable; dedup uses joint key.
Ships disabled; operator enables via GUI after setting a bbox.
Adds the NASA Earth Observatory Natural Event Tracker (EONET v3) adapter,
publishing on the existing CENTRAL_DISASTER stream under
central.disaster.eonet.<category>.global subjects.
- One Central event per EONET event id; geo = most-recent geometry point.
- Composite dedup key (eonet:<id>:<latest_geometry_date_iso>) — timeline
advance re-publishes, idle re-poll suppresses.
- category_allowlist defaults to all 13 upstream categories; operator opts
OUT per-category if GDACS overlap (wildfires/floods/severeStorms/volcanoes)
produces unwanted dupes on gdacs.* subjects.
- camelCase upstream IDs (seaLakeIce, dustHaze, etc.) mapped to
lower_snake_case subject components by a single _subject_category helper.
- Country resolves to literal 'global' (no reverse-geocode in v1).
- Fall-off: missing-from-feed event emits central.disaster.eonet.<cat>.removed.global,
subtype before 'removed' per §8 canonical pattern.
Adapter ships disabled; operator enables via GUI.
Eliminates the duplication that has been hand-bumped through PRs B, C, D, E.
Adding a stream is now one StreamEntry in src/central/streams.py + one
migration row in config.streams. supervisor STREAM_SUBJECTS / archive
STREAMS / gui DASHBOARD_STREAMS all derive at import time. No drift
possible because there is one source.
Pure refactor; no behavior change. Runtime verified: derived structures
are byte-equivalent to the previous literal definitions.
src/central/streams.py (new):
@dataclass(frozen=True)
class StreamEntry:
name: str
subject_filter: str
event_bearing: bool = True # archive consumes from this stream
dashboard: bool = True # GUI dashboard surfaces this stream
STREAMS: list[StreamEntry] = [
StreamEntry("CENTRAL_WX", "central.wx.>"),
StreamEntry("CENTRAL_FIRE", "central.fire.>"),
StreamEntry("CENTRAL_QUAKE", "central.quake.>"),
StreamEntry("CENTRAL_SPACE", "central.space.>"),
StreamEntry("CENTRAL_DISASTER", "central.disaster.>"),
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
]
Consumers derive:
supervisor.STREAM_SUBJECTS = {s.name: [s.subject_filter] for s in STREAMS}
(includes META: supervisor must create every stream in JetStream)
archive.STREAMS = [(s.name, s.subject_filter) for s in STREAMS if s.event_bearing]
(excludes META: status messages, not events)
gui.DASHBOARD_STREAMS = [s.name for s in STREAMS if s.dashboard]
To resolve the name collision between the registry STREAMS and the
existing archive.STREAMS public symbol, archive.py imports the registry
under an alias: from central.streams import STREAMS as STREAM_REGISTRY.
The archives STREAMS surface (the tuple-list) is unchanged for callers.
Same alias used in supervisor.py and gui/routes.py for symmetry.
Migration files unchanged. config.streams keeps seeding retention/bytes --
operator-tunable ops state, separate SoT from the structural mapping.
Tests:
Dropped from test_archive_multi_stream.py (7, all tautological vs. registry):
test_streams_list_has_five_entries (magic-number count)
test_streams_contains_central_wx / fire / quake / space / disaster
test_streams_excludes_central_meta
Dropped from test_dashboard.py:
`assert len(streams) == 6` line inside test_single_stream_failure_doesnt_crash_card
(the test itself stays; only the magic-number assertion is removed)
Added in test_stream_registry.py (8 invariant tests):
test_stream_names_unique
test_subject_filters_unique
test_subject_filter_central_prefix_wildcard
test_meta_is_only_non_event_bearing
test_supervisor_stream_subjects_includes_meta
test_supervisor_stream_subjects_includes_all
test_archive_streams_excludes_non_event_bearing
test_dashboard_streams_matches_dashboard_flag
The new tests assert properties (uniqueness, format, derivation correctness),
not literals. Future stream additions need zero new test code -- every
invariant automatically covers them.
Note: test file named tests/test_stream_registry.py (not test_streams.py)
to avoid colliding with the pre-existing tests/test_streams.py, which
covers the GUI streams-management page.
Full suite: 427 passed (was 426 on main: -7 dropped + 8 added).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per handoff §9 the removed-event convention is
central.<domain>.<subtype>.removed.<geo> -- WFIGS uses
central.fire.incident.removed.<state>. GDACS tombstones were emitting
central.disaster.removed.<country> with the eventtype only in the
category (disaster.removed.wf), which would silently miss type-filtered
subscribers (e.g. central.disaster.wf.> would not see WF removals).
Fix:
- poll() iscurrent=false branch and missing-from-feed loop both set
category=f"disaster.{eventtype.lower()}.removed" (eventtype before
the .removed token, matching the live-event subject hierarchy).
- subject_for() detects parts[-1] == "removed" and emits
central.disaster.<eventtype>.removed.<country>.
Tests updated:
test_fall_off_iscurrent_false now asserts category disaster.wf.removed
and subject central.disaster.wf.removed.greece.
test_fall_off_missing_from_feed adds the category assertion.
Both tombstone-collection filters flip from startswith("disaster.removed")
to endswith(".removed") for general-shape coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the GDACS (Global Disaster Alert and Coordination System) adapter
against the self-describing framework. Polls https://www.gdacs.org/xml/rss.xml
every 600s, parses the RSS items, and publishes to a new CENTRAL_DISASTER
JetStream stream on central.disaster.<eventtype_lower>.<country_lower>.
Locked decisions:
- Keep: WF, DR, FL, VO, TC. Drop: EQ (USGS canonical on central.quake.>),
plus any future-unknown eventtype.
- Filter via settings_schema event_types: list[str] so operators can
re-allow without a code change.
- Dedup by RSS guid (format <eventtype><eventid>, stable across reissue).
- Severity from gdacs:alertlevel (Green=1, Orange=2, Red=3, default 0).
- Fall-off uses GDACS gdacs:iscurrent=false as explicit tombstone signal,
with a fallback for items that vanish entirely from the feed. Tombstones
publish on disaster.removed.<eventtype>.<country>.
- Geo: centroid from geo:Point, bbox from gdacs:bbox (reordered to Geo
(minLon, minLat, maxLon, maxLat)), primary_region from gdacs:iso3.
CENTRAL_DISASTER stream: 7d retention, 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE / CENTRAL_SPACE. Migrations 020 (adapter row,
enabled=false, default event_types in settings) and 021 (stream seed).
STREAM_SUBJECTS, archive STREAMS, GUI DASHBOARD_STREAMS each pick up
the new stream.
Tests: 14 new in tests/test_gdacs.py using frozen RSS fixtures with WF/DR/EQ/XX
items (covering normalization, EQ drop, unknown drop, settings override,
guid dedup, iscurrent=false tombstone, missing-from-feed tombstone,
helper boundaries). Stream-count assertions bumped 4->5 and 5->6 for
the new stream (anti-pattern noted; queued as a follow-up PR E.5).
+1 membership test test_streams_contains_central_disaster.
Full suite: 426 passed.
End-to-end on CT104: 48 events published on first poll (44 disaster.wf +
4 disaster.fl), zero EQ events, all subjects under central.disaster.>
with lowercase-hyphenated country suffixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent adapters sharing src/central/adapters/swpc_common.py,
mirroring the WFIGS two-adapter pattern. Each adapter has its own row in
config.adapters (ships disabled), its own cadence, and its own dedup
state, so operators can independently enable/disable and so a broken
upstream endpoint does not silently mask a healthy one.
Subjects:
swpc_alerts -> central.space.alert.<product_id_lower>
swpc_kindex -> central.space.kindex
swpc_protons -> central.space.proton_flux
Dedup keys:
alerts: product_id + issue_datetime
kindex: time_tag
protons: time_tag + energy
Severity: G-scale on product_id for K0[5-9][AW] alerts (G1-G5 -> 1-4),
G-scale on Kp for kindex, 0 for protons (raw flux carried in event.data).
No geo on any SWPC events (centroid=None, regions=[], primary_region=None).
No fall-off detection for alerts -- a single 115-row sample cannot confirm
whether alerts disappear from the upstream JSON when expired; deferred to
a later pass after 24h of observation.
CENTRAL_SPACE stream seeded with 7-day retention / 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE. STREAM_SUBJECTS, archive STREAMS, and
DASHBOARD_STREAMS each pick up the new stream.
Tests: 16 new cases in tests/test_swpc.py using real-shape frozen JSON
fixtures (alerts product_ids EF3A/K05A/K07A; kindex Kp boundaries; protons
composite dedup). Two existing tests updated for the new stream count
(test_archive_multi_stream.test_streams_list_has_three_entries renamed to
_has_four_entries; test_dashboard expects 5 streams not 4); added a
test_streams_contains_central_space companion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug fixes:
1. Wire is_published/mark_published/bump_last_seen into poll() loop
- Skip already-published items, bump TTL to prevent sweep
- Mark published after yield to track new items
2. Add conditional fetch support (If-Modified-Since, If-None-Match)
- Store Last-Modified/ETag from responses
- Send conditional headers on subsequent requests
- Handle 304 Not Modified gracefully (return empty list)
3. Document state parsing rationale in docstring
- Description has structured State: field vs unreliable title prefixes
Tests added:
- test_dedup_in_poll_loop: verify second poll yields 0 for same items
- test_conditional_304_yields_zero: verify 304 returns empty list
- test_conditional_headers_sent_after_first_poll: verify headers sent
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
InciWeb adapter for RSS-based wildfire narrative updates:
- Parse DMS coordinates from description text
- Extract state name and map to 2-letter code
- Strip HTML tags and decode entities
- Bbox filtering for regional focus
- Dedup via published_ids table (14-day sweep)
- Category: fire.narrative.inciweb
- Subject: central.fire.narrative.inciweb.<state>
Includes migration 017 and 15 unit tests.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
WFIGS returns ISO 3166-2 state codes (US-MT) and 2-letter incident
type codes (WF, RX). Normalize at parse boundary:
- normalize_state: strips US- prefix (US-MT -> MT)
- normalize_incident_type: maps codes to names (WF -> wildfire)
Fixes:
- category was fire.incident.wf, now fire.incident.wildfire
- region was US-US-MT-GLACIER, now US-MT-GLACIER
Both raw and normalized values stored in event.data.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two new adapters for wildfire data from NIFC WFIGS:
- wfigs_incidents: Active fire incident locations
- wfigs_perimeters: Active fire perimeter polygons
Features:
- IRWIN GUID dedup via is_published/mark_published
- Fall-off detection with removal events when fires exit current
- Bbox post-filtering with shapely polygon intersection
- Severity mapping from DailyAcres (0-4 scale)
- Subject hierarchy: central.fire.<layer>.<state>.<county>
Ships disabled by default; operators enable via GUI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add set_adapter_last_error method to ConfigStore for setting/clearing
adapter error states
- Add API key precondition check in supervisor._start_adapter that:
- Checks if adapter has requires_api_key attribute
- Looks up the key via config_store.get_api_key
- Sets last_error and returns early if key is missing
- Clears last_error when adapter successfully starts
- Update adapters_list handler to compute api_key_missing flag
for each adapter and pass to template
- Update adapters_edit_form handler to compute api_key_missing
and requires_api_key_alias for template context
- Update adapters_list.html to show warning badge when api_key_missing
- Update adapters_edit.html to show warning article and disable
Enable checkbox when api_key_missing
- Add tests for new functionality
- Fix test mocks to include requires_api_key and last_error fields
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change 5: Move contact_email validation to Pydantic schema
- NWSSettings now uses Field(pattern=...) for email validation
- Pydantic pattern validation catches invalid emails
- No special handler branch needed in routes.py
Change 6: Generic api_key_field mechanism
- Add api_key_field attribute to SourceAdapter base class
- FIRMSAdapter sets api_key_field="api_key_alias"
- GET handlers swap widget to "api_key_select" when field matches
- POST handlers validate against state.api_keys generically
- Templates use new api_key_select widget branch
- adapters_edit handlers now fetch and pass api_keys to context
Tests added:
- test_invalid_contact_email_via_pydantic_pattern
- test_invalid_api_key_alias_generic
- test_api_key_field_none_no_check
- test_adapters_edit_fetches_api_keys_into_context
Zero field.name hardcoded branches remain in routes.py or templates.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove dead _get_valid_satellites/_get_valid_feeds calls from error render
- Replace hardcoded adapter list with dynamic wizard_adapters discovery
- Use RegionConfig model validation instead of hand-rolled bounds check
- Add Pydantic settings validation after field parsing to catch Literal violations
- Add TestSetupAdaptersErrorRerender with cadence and region error tests
Fixes error path gaps that would cause NameError on form re-render.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Literal type support to form_descriptors.py
- Literal fields map to select widget
- list[Literal] fields map to checkboxes widget
- Options list extracted from Literal type args
- Update FIRMS adapter: satellites is now list[Literal[...]]
- Update USGS adapter: feed is now Literal[...]
- Refactor wizard to use wizard_order for adapter filtering
- Replace hardcoded adapter lists with dynamic discovery
- Remove _get_valid_satellites() and _get_valid_feeds() helpers
- Generic field parsing using describe_fields() pattern
- Update templates for generic widget rendering
- Add select/checkboxes widgets to adapters_edit.html
- Update tests for new widget types
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat(gui): generic adapter edit form
- Add form_descriptors.py with describe_fields() for Pydantic-to-HTML mapping
- Update routes.py with generic GET/POST handlers using field descriptors
- Delete per-adapter templates (nws, firms, usgs_quake)
- Adding new adapters no longer requires GUI template work
db: add last_error column to adapters table
- Migration 015 with IF NOT EXISTS for idempotency
refactor(gui): clean up flagged issues
- Move discover_adapters to adapter_discovery.py (GUI no longer imports nats)
- Use dynamic cadence validation via AdapterConfig field constraint (ge=10)
- Remove dead code in form_descriptors.py
refactor(wizard): use dynamic cadence validation
- Wizard POST handler uses same dynamic pattern as edit form
Update wizard POST handler to use the same dynamic cadence validation
pattern as the adapter edit form:
- Use AdapterConfig.model_fields["cadence_s"].metadata[0].ge for min bound
- Remove hardcoded 60-3600 range check
- Remove min/max attributes from setup_adapters.html template
No tests in test_wizard.py referenced the old cadence range.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Make migration 015 idempotent with IF NOT EXISTS
2. Remove hardcoded cadence range from routes.py and template:
- Added ge=10 constraint to AdapterConfig.cadence_s field
- Removed manual 60-3600 check from routes.py POST handler
- Validate cadence using AdapterConfig field metadata
- Removed min/max attributes from template input
3. Move discover_adapters to its own module:
- Created src/central/adapter_discovery.py
- Updated supervisor.py to import from adapter_discovery
- Updated routes.py to import from adapter_discovery
- GUI no longer transitively imports nats or stream_manager
4. Remove dead code branch in form_descriptors.py:
- Removed unreachable RegionConfig check (already handled earlier)
- Improved error message for unsupported nested types
5. Updated test_adapters.py:
- Changed invalid cadence test from 30 to 5 (below ge=10)
- Updated assertion to check for "10" in error message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migration 015: Adds last_error TEXT column to config.adapters.
Populated by supervisor when an adapter fails to start or apply config.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement Central 2-A2: generic adapter edit form feature.
- Add form_descriptors.py with describe_fields() and FieldDescriptor
- Maps Pydantic types to HTML widgets (text, number, checkbox, csv, region)
- Handles Optional types by recursively resolving inner type
- Uses PydanticUndefined handling for proper default values
- Update routes.py GET/POST handlers:
- Use cached _adapter_classes() for adapter class lookup
- Generate field descriptors from adapter settings_schema
- Parse form values based on widget type in POST handler
- Validate settings via Pydantic ValidationError
- Update adapters_edit.html template:
- Render form dynamically from field descriptors
- Support all widget types (text, number, checkbox, csv, region)
- Use adapter.display_name and adapter.description from class
- Delete per-adapter templates:
- adapters_edit_nws.html
- adapters_edit_firms.html
- adapters_edit_usgs_quake.html
- Add tests/test_form_descriptors.py with comprehensive coverage
- Update tests/test_adapters.py to include last_error in mock rows
- Update tests/test_region_picker.py to include last_error in mock rows
Adding a new adapter no longer requires GUI template work.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add display_name, description, settings_schema (Pydantic), requires_api_key, wizard_order, default_cadence_s to SourceAdapter ABC
- Implement in NWSAdapter, FIRMSAdapter, USGSQuakeAdapter
- Auto-discovery via pkgutil.iter_modules
- Fix quake stream bug (events now route to CENTRAL_QUAKE)
- 308 tests pass, live verified on CT104
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace settings_schema classmethod with Pydantic model class attribute
- Add display_name, description, requires_api_key, wizard_order, default_cadence_s
- Remove stream_name from adapters (JetStream routes by subject filter)
- Define NWSSettings, FIRMSSettings, USGSQuakeSettings Pydantic models
- Make discover_adapters() public with error handling
- Move adapter registry to Supervisor instance (self._adapters)
- Add subject_for tests for all 6 quake magnitude tiers
- Fix test_supervisor_integration to use injected mock adapters
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stream_name, subject_for(), and settings_schema() to SourceAdapter ABC
- Implement all three methods in NWSAdapter, FIRMSAdapter, USGSQuakeAdapter
- Replace manual _ADAPTER_REGISTRY with pkgutil.iter_modules auto-discovery
- Remove subject_for_event from models.py (each adapter owns its subject logic)
- Update supervisor to use adapter.subject_for(event) instead of helper
- Fix quake events going to wrong stream (was publishing to CENTRAL_WX)
- Update test files to use adapter methods
This fixes the quake stream bug where events were published to
central.wx.alert.us.unknown instead of central.quake.event.<tier>.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Pyproject was stuck at 0.1.0 since Phase 0; bumping to match
the v0.3.0 release tag landing alongside this.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
* feat: events feed UX iteration - colors, popups, viewport filter
A. Color-code polygons by adapter (NWS amber, FIRMS red, USGS violet)
B. Click popup on polygons showing time + adapter + category + subject
C. Map viewport drives spatial filter - pan/zoom updates table via HTMX
D. Add legend showing adapter color mapping
E. Remove draw-bbox control, region inputs now hidden (auto-managed)
Template changes:
- _events_rows.html: add data-adapter, data-category, data-time, data-subject
- events_list.html: ADAPTER_COLORS mapping, bindPopup, moveend handler
Test: verify template renders adapter/category/subject for JS consumption
* fix: remove isoformat() call on already-formatted time string
* feat: full events feed UX iteration
A. Color-code polygons by adapter with legend
B. Click popup on polygons with "View details" link
C. Viewport-driven spatial filter - pan/zoom updates table via HTMX
Map never auto-fits after initial load (user controls viewport)
D. Expandable row details showing full event data payload
Changes:
- _events_rows.html: add data-event-id, expand button, detail row
- events_list.html: eventLayerGroup pattern, buildPopup, rebindEventLayers
Fit to results button, expand/collapse handlers, CSS.escape for IDs
* fix: add programmaticMove flag to prevent viewport refresh loop
Suppress moveend handler during fitBounds/setView calls to prevent
feedback loop: fitBounds -> moveend -> applyViewportFilter -> HTMX
swap -> repeat.
* fix: map never auto-fits - user controls viewport
- Disable initial fitToAllLayers on page load
- Remove fitBounds/setView from row click handler
- Map only moves when user pans/zooms
- Table filters based on visible viewport
* fix: map shows all events always, only table filters
Map polygons are drawn once on load and never cleared/redrawn.
HTMX swap only updates the table, not the map layers.
User viewport is fully preserved.
* fix: use htmx.trigger instead of dispatchEvent for HTMX swap
dispatchEvent(submit) was triggering native form submission (full page
reload). htmx.trigger() properly triggers HTMX swap.
Also re-enable initial rebindEventLayers so polygons load on first render.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
* feat(gui): add events feed frontend with map and filters
GET /events: Full page with filter form, table, and Leaflet map
GET /events/rows: HTMX fragment for table updates
Features:
- Filterable by adapter, category, time range, region bbox
- Cursor-based pagination with Next button
- Leaflet map showing event geometries
- Click/hover row highlights geometry on map
- Draw rectangle on map to filter by region
- Validation errors shown as banner, not 400
- Events link added to nav between Adapters and Streams
Refactored events query into shared helper for JSON and HTML routes.
Tests: 14 new tests covering filters, fragments, geometry handling.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(events): use shared helpers for /events.json, fix tests
- Refactor /events.json to use _parse_events_params and _fetch_events
helpers, removing ~200 lines of duplicate query logic
- Delete smoke test (test_events_unauthenticated_redirects) that had
no assertions
- Add TestCrossEndpointParity: verify /events.json and /events return
identical results with same params, test category filter and cursor
pagination on both endpoints
- Add TestErrorSemantics: verify /events.json returns 400 on bad params
while /events returns 200 with error banner (intentional API vs HTML
divergence)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add real assertions to since/until and region filter tests
Replace trivial status_code==200 assertions with checks that verify
the filter values were actually parsed and passed to the template.
These tests now fail if the handler ignores the filter parameters.
* fix: remove double-escaping from data-geometry attribute
tojson already produces HTML-attribute-safe JSON. The extra |e filter
was double-escaping, causing JSON.parse to fail in the browser JS.
Switch to single-quoted attribute to avoid conflicts with JSON double
quotes.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(wizard): implement deferred-commit pattern for setup wizard
Replace the current "POST each step -> DB write -> redirect" architecture
with "collect values across steps in a signed cookie, commit everything
in one transaction at Finish."
Key changes:
- Add wizard.py: WizardState dataclass and cookie helpers
- csrf.py: Add reuse_or_generate_pre_auth_csrf helper
- routes.py: All wizard handlers now use cookie state, no DB writes until finish
- middleware.py: Cookie-based wizard step routing instead of DB queries
- setup_operator.html: Remove "Operator Already Configured" branch
Benefits:
- Back navigation works: can return to any step and edit values
- Atomic commit: all DB writes happen in single transaction at finish
- No orphaned state: failed wizard leaves no DB artifacts
- Simpler auth: pre-auth CSRF for all 5 steps (no session until finish)
Tests updated for new behavior. 287 tests passing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(templates): correct SRI hashes for leaflet.draw assets
The integrity hashes for leaflet.draw.css and leaflet.draw.js were
incorrect, causing browsers to silently block these resources. This
broke the Leaflet.draw toolbar and map rendering for FIRMS/USGS
adapter region pickers.
Updated both setup_adapters.html and adapters_edit.html with the
correct sha512 hashes computed from the actual CDN files.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): return 204 for browser-noise paths to prevent CSRF races
Browser requests for /favicon.ico, /apple-touch-icon.png, etc. were
triggering parallel GET requests that could race with form loads,
causing CSRF token rotation issues.
Added BROWSER_NOISE_PATHS constant and early 204 response in both
SetupGateMiddleware and SessionMiddleware to short-circuit these
requests before any cookie/token handling occurs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
