Fixup 1 — Contract section appended to SourceAdapter.preview_for_settings's
docstring. Override authors read adapter.py, not routes.py, so the contract
(pure function of settings; open your own short-lived aiohttp session; None
vs [] semantics) belongs on the base method, not on the GUI stub class.
Fixup 2 — _adapter_preview.html distinguishes [] from None. Previously the
elif test was truthiness (`elif preview_rows`) which collapsed both into
"render nothing". Now uses `elif preview_rows is not none` and special-cases
the empty-list case inside: legend "Preview (0 rows)" with no table; None
still renders nothing at all. Lets adapters signal "query ran, matched zero"
distinctly from "preview not meaningful".
Tests +1:
- test_partial_renders_empty_list — [] yields "Preview (0 rows)" legend,
no table, no headers. Distinct from the existing None case.
Acceptance:
- 27/27 targeted (preview_hook +1 new, nwis, stream_registry).
- 458/458 full suite.
- (b) framework GUI dir still has zero adapter-name branches.
Adds an optional async hook on SourceAdapter so any adapter can surface a
settings-driven preview on its /adapters/<name> edit page. The framework
renders the result generically as a table — no adapter-name branches in
GUI templates or route code.
Framework changes:
- src/central/adapter.py: new async preview_for_settings(self, settings)
on the base class, default returns None. Adapters opt in by overriding;
non-overriding adapters render unchanged.
- src/central/gui/routes.py: GET /adapters/{name} instantiates the adapter
with a no-op _PreviewConfigStore stub and a /dev/null cursor path (GUI
has no live ConfigStore), constructs settings_obj via the schema, and
calls preview_for_settings inside a try/except. Result lands in template
context as preview_rows / preview_error.
- src/central/gui/templates/_adapter_preview.html: new partial. Generic
table with columns derived from the first dict's keys; error banner
mirrors the existing last_error article style.
- src/central/gui/templates/adapters_edit.html: one-line include between
the Region fieldset and Save/Cancel.
NWIS opt-in:
- New NWIS_MONITORING_LOCATIONS_URL constant and _PREVIEW_LIMIT cap of 50.
- preview_for_settings returns None when region is None, otherwise one-shot
fetches monitoring-locations within the bbox via a fresh aiohttp session.
Must work even when adapter is not started -- the GUI process never calls
startup(). Returns list[dict] with the contract column order: site_id,
name, site_type, state. Errors propagate so the framework can render the
operator-visible banner.
- HTTP call factored into _fetch_preview_text so tests mock cleanly.
Tests (7 new):
- tests/test_preview_hook.py: default returns None; partial renders list
with correct headers/rows/count; partial renders error banner; partial
renders empty when both context values are None.
- tests/test_nwis.py adds TestNWISPreview: returns None without region,
returns rows with correct column order, propagates HTTP errors.
Verification:
- 457/457 full suite green (was 450; +7 new tests).
- Live /adapters/nwis preview returns 50 rows with the contract keys
against the current production Iowa bbox.
- /adapters/eonet preview_for_settings returns None via base default --
proves framework is duck-typed, no NWIS-specific code in framework.
NASA WaterData OGC API v0 (latest-continuous collection) — polls configured
parameter codes within an operator-set bbox and publishes on the new
CENTRAL_HYDRO stream.
- Subject: central.hydro.<parameter_code>.<agency>.<bare_site_no>
(e.g. central.hydro.00060.usgs.05420500). The agency/site decomposition
lives in a single _subject_tokens_for_id helper.
- Default parameter codes: 00060 (discharge), 00065 (gage height),
00010 (water temperature). Operator-tunable; single SoT in
_DEFAULT_PARAMETER_CODES — no parallel literals.
- Composite dedup: nwis:<monitoring_location_id>:<param>:<time_iso>.
Prefix kept in dedup key for cross-agency uniqueness.
- Pagination: follows OGC 'rel=next' link until absent (cursor-based).
- Region bbox is REQUIRED in practice; adapter logs WARN at startup if
region is None (does not refuse to start).
- New stream CENTRAL_HYDRO added to streams.py registry (one line).
Retention mirrors CENTRAL_DISASTER (7 days, 1 GiB).
- No removal pattern in v1 — sites are static; missing data is the signal.
Upstream divergences from the original spec brief, caught by pre-build curl:
- Collection is 'latest-continuous', not 'instantaneous-values'.
- Site filter param is 'monitoring_location_id' (singular), not
'monitoring_locations_id' (plural).
- Site identifier requires agency prefix in queries (USGS-NNNNN).
- feature.id is a per-record UUID, not stable; dedup uses joint key.
Ships disabled; operator enables via GUI after setting a bbox.
Adds the NASA Earth Observatory Natural Event Tracker (EONET v3) adapter,
publishing on the existing CENTRAL_DISASTER stream under
central.disaster.eonet.<category>.global subjects.
- One Central event per EONET event id; geo = most-recent geometry point.
- Composite dedup key (eonet:<id>:<latest_geometry_date_iso>) — timeline
advance re-publishes, idle re-poll suppresses.
- category_allowlist defaults to all 13 upstream categories; operator opts
OUT per-category if GDACS overlap (wildfires/floods/severeStorms/volcanoes)
produces unwanted dupes on gdacs.* subjects.
- camelCase upstream IDs (seaLakeIce, dustHaze, etc.) mapped to
lower_snake_case subject components by a single _subject_category helper.
- Country resolves to literal 'global' (no reverse-geocode in v1).
- Fall-off: missing-from-feed event emits central.disaster.eonet.<cat>.removed.global,
subtype before 'removed' per §8 canonical pattern.
Adapter ships disabled; operator enables via GUI.
Eliminates the duplication that has been hand-bumped through PRs B, C, D, E.
Adding a stream is now one StreamEntry in src/central/streams.py + one
migration row in config.streams. supervisor STREAM_SUBJECTS / archive
STREAMS / gui DASHBOARD_STREAMS all derive at import time. No drift
possible because there is one source.
Pure refactor; no behavior change. Runtime verified: derived structures
are byte-equivalent to the previous literal definitions.
src/central/streams.py (new):
@dataclass(frozen=True)
class StreamEntry:
name: str
subject_filter: str
event_bearing: bool = True # archive consumes from this stream
dashboard: bool = True # GUI dashboard surfaces this stream
STREAMS: list[StreamEntry] = [
StreamEntry("CENTRAL_WX", "central.wx.>"),
StreamEntry("CENTRAL_FIRE", "central.fire.>"),
StreamEntry("CENTRAL_QUAKE", "central.quake.>"),
StreamEntry("CENTRAL_SPACE", "central.space.>"),
StreamEntry("CENTRAL_DISASTER", "central.disaster.>"),
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
]
Consumers derive:
supervisor.STREAM_SUBJECTS = {s.name: [s.subject_filter] for s in STREAMS}
(includes META: supervisor must create every stream in JetStream)
archive.STREAMS = [(s.name, s.subject_filter) for s in STREAMS if s.event_bearing]
(excludes META: status messages, not events)
gui.DASHBOARD_STREAMS = [s.name for s in STREAMS if s.dashboard]
To resolve the name collision between the registry STREAMS and the
existing archive.STREAMS public symbol, archive.py imports the registry
under an alias: from central.streams import STREAMS as STREAM_REGISTRY.
The archives STREAMS surface (the tuple-list) is unchanged for callers.
Same alias used in supervisor.py and gui/routes.py for symmetry.
Migration files unchanged. config.streams keeps seeding retention/bytes --
operator-tunable ops state, separate SoT from the structural mapping.
Tests:
Dropped from test_archive_multi_stream.py (7, all tautological vs. registry):
test_streams_list_has_five_entries (magic-number count)
test_streams_contains_central_wx / fire / quake / space / disaster
test_streams_excludes_central_meta
Dropped from test_dashboard.py:
`assert len(streams) == 6` line inside test_single_stream_failure_doesnt_crash_card
(the test itself stays; only the magic-number assertion is removed)
Added in test_stream_registry.py (8 invariant tests):
test_stream_names_unique
test_subject_filters_unique
test_subject_filter_central_prefix_wildcard
test_meta_is_only_non_event_bearing
test_supervisor_stream_subjects_includes_meta
test_supervisor_stream_subjects_includes_all
test_archive_streams_excludes_non_event_bearing
test_dashboard_streams_matches_dashboard_flag
The new tests assert properties (uniqueness, format, derivation correctness),
not literals. Future stream additions need zero new test code -- every
invariant automatically covers them.
Note: test file named tests/test_stream_registry.py (not test_streams.py)
to avoid colliding with the pre-existing tests/test_streams.py, which
covers the GUI streams-management page.
Full suite: 427 passed (was 426 on main: -7 dropped + 8 added).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per handoff §9 the removed-event convention is
central.<domain>.<subtype>.removed.<geo> -- WFIGS uses
central.fire.incident.removed.<state>. GDACS tombstones were emitting
central.disaster.removed.<country> with the eventtype only in the
category (disaster.removed.wf), which would silently miss type-filtered
subscribers (e.g. central.disaster.wf.> would not see WF removals).
Fix:
- poll() iscurrent=false branch and missing-from-feed loop both set
category=f"disaster.{eventtype.lower()}.removed" (eventtype before
the .removed token, matching the live-event subject hierarchy).
- subject_for() detects parts[-1] == "removed" and emits
central.disaster.<eventtype>.removed.<country>.
Tests updated:
test_fall_off_iscurrent_false now asserts category disaster.wf.removed
and subject central.disaster.wf.removed.greece.
test_fall_off_missing_from_feed adds the category assertion.
Both tombstone-collection filters flip from startswith("disaster.removed")
to endswith(".removed") for general-shape coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the GDACS (Global Disaster Alert and Coordination System) adapter
against the self-describing framework. Polls https://www.gdacs.org/xml/rss.xml
every 600s, parses the RSS items, and publishes to a new CENTRAL_DISASTER
JetStream stream on central.disaster.<eventtype_lower>.<country_lower>.
Locked decisions:
- Keep: WF, DR, FL, VO, TC. Drop: EQ (USGS canonical on central.quake.>),
plus any future-unknown eventtype.
- Filter via settings_schema event_types: list[str] so operators can
re-allow without a code change.
- Dedup by RSS guid (format <eventtype><eventid>, stable across reissue).
- Severity from gdacs:alertlevel (Green=1, Orange=2, Red=3, default 0).
- Fall-off uses GDACS gdacs:iscurrent=false as explicit tombstone signal,
with a fallback for items that vanish entirely from the feed. Tombstones
publish on disaster.removed.<eventtype>.<country>.
- Geo: centroid from geo:Point, bbox from gdacs:bbox (reordered to Geo
(minLon, minLat, maxLon, maxLat)), primary_region from gdacs:iso3.
CENTRAL_DISASTER stream: 7d retention, 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE / CENTRAL_SPACE. Migrations 020 (adapter row,
enabled=false, default event_types in settings) and 021 (stream seed).
STREAM_SUBJECTS, archive STREAMS, GUI DASHBOARD_STREAMS each pick up
the new stream.
Tests: 14 new in tests/test_gdacs.py using frozen RSS fixtures with WF/DR/EQ/XX
items (covering normalization, EQ drop, unknown drop, settings override,
guid dedup, iscurrent=false tombstone, missing-from-feed tombstone,
helper boundaries). Stream-count assertions bumped 4->5 and 5->6 for
the new stream (anti-pattern noted; queued as a follow-up PR E.5).
+1 membership test test_streams_contains_central_disaster.
Full suite: 426 passed.
End-to-end on CT104: 48 events published on first poll (44 disaster.wf +
4 disaster.fl), zero EQ events, all subjects under central.disaster.>
with lowercase-hyphenated country suffixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent adapters sharing src/central/adapters/swpc_common.py,
mirroring the WFIGS two-adapter pattern. Each adapter has its own row in
config.adapters (ships disabled), its own cadence, and its own dedup
state, so operators can independently enable/disable and so a broken
upstream endpoint does not silently mask a healthy one.
Subjects:
swpc_alerts -> central.space.alert.<product_id_lower>
swpc_kindex -> central.space.kindex
swpc_protons -> central.space.proton_flux
Dedup keys:
alerts: product_id + issue_datetime
kindex: time_tag
protons: time_tag + energy
Severity: G-scale on product_id for K0[5-9][AW] alerts (G1-G5 -> 1-4),
G-scale on Kp for kindex, 0 for protons (raw flux carried in event.data).
No geo on any SWPC events (centroid=None, regions=[], primary_region=None).
No fall-off detection for alerts -- a single 115-row sample cannot confirm
whether alerts disappear from the upstream JSON when expired; deferred to
a later pass after 24h of observation.
CENTRAL_SPACE stream seeded with 7-day retention / 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE. STREAM_SUBJECTS, archive STREAMS, and
DASHBOARD_STREAMS each pick up the new stream.
Tests: 16 new cases in tests/test_swpc.py using real-shape frozen JSON
fixtures (alerts product_ids EF3A/K05A/K07A; kindex Kp boundaries; protons
composite dedup). Two existing tests updated for the new stream count
(test_archive_multi_stream.test_streams_list_has_three_entries renamed to
_has_four_entries; test_dashboard expects 5 streams not 4); added a
test_streams_contains_central_space companion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug fixes:
1. Wire is_published/mark_published/bump_last_seen into poll() loop
- Skip already-published items, bump TTL to prevent sweep
- Mark published after yield to track new items
2. Add conditional fetch support (If-Modified-Since, If-None-Match)
- Store Last-Modified/ETag from responses
- Send conditional headers on subsequent requests
- Handle 304 Not Modified gracefully (return empty list)
3. Document state parsing rationale in docstring
- Description has structured State: field vs unreliable title prefixes
Tests added:
- test_dedup_in_poll_loop: verify second poll yields 0 for same items
- test_conditional_304_yields_zero: verify 304 returns empty list
- test_conditional_headers_sent_after_first_poll: verify headers sent
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
InciWeb adapter for RSS-based wildfire narrative updates:
- Parse DMS coordinates from description text
- Extract state name and map to 2-letter code
- Strip HTML tags and decode entities
- Bbox filtering for regional focus
- Dedup via published_ids table (14-day sweep)
- Category: fire.narrative.inciweb
- Subject: central.fire.narrative.inciweb.<state>
Includes migration 017 and 15 unit tests.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
WFIGS returns ISO 3166-2 state codes (US-MT) and 2-letter incident
type codes (WF, RX). Normalize at parse boundary:
- normalize_state: strips US- prefix (US-MT -> MT)
- normalize_incident_type: maps codes to names (WF -> wildfire)
Fixes:
- category was fire.incident.wf, now fire.incident.wildfire
- region was US-US-MT-GLACIER, now US-MT-GLACIER
Both raw and normalized values stored in event.data.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two new adapters for wildfire data from NIFC WFIGS:
- wfigs_incidents: Active fire incident locations
- wfigs_perimeters: Active fire perimeter polygons
Features:
- IRWIN GUID dedup via is_published/mark_published
- Fall-off detection with removal events when fires exit current
- Bbox post-filtering with shapely polygon intersection
- Severity mapping from DailyAcres (0-4 scale)
- Subject hierarchy: central.fire.<layer>.<state>.<county>
Ships disabled by default; operators enable via GUI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add set_adapter_last_error method to ConfigStore for setting/clearing
adapter error states
- Add API key precondition check in supervisor._start_adapter that:
- Checks if adapter has requires_api_key attribute
- Looks up the key via config_store.get_api_key
- Sets last_error and returns early if key is missing
- Clears last_error when adapter successfully starts
- Update adapters_list handler to compute api_key_missing flag
for each adapter and pass to template
- Update adapters_edit_form handler to compute api_key_missing
and requires_api_key_alias for template context
- Update adapters_list.html to show warning badge when api_key_missing
- Update adapters_edit.html to show warning article and disable
Enable checkbox when api_key_missing
- Add tests for new functionality
- Fix test mocks to include requires_api_key and last_error fields
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change 5: Move contact_email validation to Pydantic schema
- NWSSettings now uses Field(pattern=...) for email validation
- Pydantic pattern validation catches invalid emails
- No special handler branch needed in routes.py
Change 6: Generic api_key_field mechanism
- Add api_key_field attribute to SourceAdapter base class
- FIRMSAdapter sets api_key_field="api_key_alias"
- GET handlers swap widget to "api_key_select" when field matches
- POST handlers validate against state.api_keys generically
- Templates use new api_key_select widget branch
- adapters_edit handlers now fetch and pass api_keys to context
Tests added:
- test_invalid_contact_email_via_pydantic_pattern
- test_invalid_api_key_alias_generic
- test_api_key_field_none_no_check
- test_adapters_edit_fetches_api_keys_into_context
Zero field.name hardcoded branches remain in routes.py or templates.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove dead _get_valid_satellites/_get_valid_feeds calls from error render
- Replace hardcoded adapter list with dynamic wizard_adapters discovery
- Use RegionConfig model validation instead of hand-rolled bounds check
- Add Pydantic settings validation after field parsing to catch Literal violations
- Add TestSetupAdaptersErrorRerender with cadence and region error tests
Fixes error path gaps that would cause NameError on form re-render.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Literal type support to form_descriptors.py
- Literal fields map to select widget
- list[Literal] fields map to checkboxes widget
- Options list extracted from Literal type args
- Update FIRMS adapter: satellites is now list[Literal[...]]
- Update USGS adapter: feed is now Literal[...]
- Refactor wizard to use wizard_order for adapter filtering
- Replace hardcoded adapter lists with dynamic discovery
- Remove _get_valid_satellites() and _get_valid_feeds() helpers
- Generic field parsing using describe_fields() pattern
- Update templates for generic widget rendering
- Add select/checkboxes widgets to adapters_edit.html
- Update tests for new widget types
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat(gui): generic adapter edit form
- Add form_descriptors.py with describe_fields() for Pydantic-to-HTML mapping
- Update routes.py with generic GET/POST handlers using field descriptors
- Delete per-adapter templates (nws, firms, usgs_quake)
- Adding new adapters no longer requires GUI template work
db: add last_error column to adapters table
- Migration 015 with IF NOT EXISTS for idempotency
refactor(gui): clean up flagged issues
- Move discover_adapters to adapter_discovery.py (GUI no longer imports nats)
- Use dynamic cadence validation via AdapterConfig field constraint (ge=10)
- Remove dead code in form_descriptors.py
refactor(wizard): use dynamic cadence validation
- Wizard POST handler uses same dynamic pattern as edit form
Update wizard POST handler to use the same dynamic cadence validation
pattern as the adapter edit form:
- Use AdapterConfig.model_fields["cadence_s"].metadata[0].ge for min bound
- Remove hardcoded 60-3600 range check
- Remove min/max attributes from setup_adapters.html template
No tests in test_wizard.py referenced the old cadence range.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Make migration 015 idempotent with IF NOT EXISTS
2. Remove hardcoded cadence range from routes.py and template:
- Added ge=10 constraint to AdapterConfig.cadence_s field
- Removed manual 60-3600 check from routes.py POST handler
- Validate cadence using AdapterConfig field metadata
- Removed min/max attributes from template input
3. Move discover_adapters to its own module:
- Created src/central/adapter_discovery.py
- Updated supervisor.py to import from adapter_discovery
- Updated routes.py to import from adapter_discovery
- GUI no longer transitively imports nats or stream_manager
4. Remove dead code branch in form_descriptors.py:
- Removed unreachable RegionConfig check (already handled earlier)
- Improved error message for unsupported nested types
5. Updated test_adapters.py:
- Changed invalid cadence test from 30 to 5 (below ge=10)
- Updated assertion to check for "10" in error message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migration 015: Adds last_error TEXT column to config.adapters.
Populated by supervisor when an adapter fails to start or apply config.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement Central 2-A2: generic adapter edit form feature.
- Add form_descriptors.py with describe_fields() and FieldDescriptor
- Maps Pydantic types to HTML widgets (text, number, checkbox, csv, region)
- Handles Optional types by recursively resolving inner type
- Uses PydanticUndefined handling for proper default values
- Update routes.py GET/POST handlers:
- Use cached _adapter_classes() for adapter class lookup
- Generate field descriptors from adapter settings_schema
- Parse form values based on widget type in POST handler
- Validate settings via Pydantic ValidationError
- Update adapters_edit.html template:
- Render form dynamically from field descriptors
- Support all widget types (text, number, checkbox, csv, region)
- Use adapter.display_name and adapter.description from class
- Delete per-adapter templates:
- adapters_edit_nws.html
- adapters_edit_firms.html
- adapters_edit_usgs_quake.html
- Add tests/test_form_descriptors.py with comprehensive coverage
- Update tests/test_adapters.py to include last_error in mock rows
- Update tests/test_region_picker.py to include last_error in mock rows
Adding a new adapter no longer requires GUI template work.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add display_name, description, settings_schema (Pydantic), requires_api_key, wizard_order, default_cadence_s to SourceAdapter ABC
- Implement in NWSAdapter, FIRMSAdapter, USGSQuakeAdapter
- Auto-discovery via pkgutil.iter_modules
- Fix quake stream bug (events now route to CENTRAL_QUAKE)
- 308 tests pass, live verified on CT104
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace settings_schema classmethod with Pydantic model class attribute
- Add display_name, description, requires_api_key, wizard_order, default_cadence_s
- Remove stream_name from adapters (JetStream routes by subject filter)
- Define NWSSettings, FIRMSSettings, USGSQuakeSettings Pydantic models
- Make discover_adapters() public with error handling
- Move adapter registry to Supervisor instance (self._adapters)
- Add subject_for tests for all 6 quake magnitude tiers
- Fix test_supervisor_integration to use injected mock adapters
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stream_name, subject_for(), and settings_schema() to SourceAdapter ABC
- Implement all three methods in NWSAdapter, FIRMSAdapter, USGSQuakeAdapter
- Replace manual _ADAPTER_REGISTRY with pkgutil.iter_modules auto-discovery
- Remove subject_for_event from models.py (each adapter owns its subject logic)
- Update supervisor to use adapter.subject_for(event) instead of helper
- Fix quake events going to wrong stream (was publishing to CENTRAL_WX)
- Update test files to use adapter methods
This fixes the quake stream bug where events were published to
central.wx.alert.us.unknown instead of central.quake.event.<tier>.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Pyproject was stuck at 0.1.0 since Phase 0; bumping to match
the v0.3.0 release tag landing alongside this.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
* feat: events feed UX iteration - colors, popups, viewport filter
A. Color-code polygons by adapter (NWS amber, FIRMS red, USGS violet)
B. Click popup on polygons showing time + adapter + category + subject
C. Map viewport drives spatial filter - pan/zoom updates table via HTMX
D. Add legend showing adapter color mapping
E. Remove draw-bbox control, region inputs now hidden (auto-managed)
Template changes:
- _events_rows.html: add data-adapter, data-category, data-time, data-subject
- events_list.html: ADAPTER_COLORS mapping, bindPopup, moveend handler
Test: verify template renders adapter/category/subject for JS consumption
* fix: remove isoformat() call on already-formatted time string
* feat: full events feed UX iteration
A. Color-code polygons by adapter with legend
B. Click popup on polygons with "View details" link
C. Viewport-driven spatial filter - pan/zoom updates table via HTMX
Map never auto-fits after initial load (user controls viewport)
D. Expandable row details showing full event data payload
Changes:
- _events_rows.html: add data-event-id, expand button, detail row
- events_list.html: eventLayerGroup pattern, buildPopup, rebindEventLayers
Fit to results button, expand/collapse handlers, CSS.escape for IDs
* fix: add programmaticMove flag to prevent viewport refresh loop
Suppress moveend handler during fitBounds/setView calls to prevent
feedback loop: fitBounds -> moveend -> applyViewportFilter -> HTMX
swap -> repeat.
* fix: map never auto-fits - user controls viewport
- Disable initial fitToAllLayers on page load
- Remove fitBounds/setView from row click handler
- Map only moves when user pans/zooms
- Table filters based on visible viewport
* fix: map shows all events always, only table filters
Map polygons are drawn once on load and never cleared/redrawn.
HTMX swap only updates the table, not the map layers.
User viewport is fully preserved.
* fix: use htmx.trigger instead of dispatchEvent for HTMX swap
dispatchEvent(submit) was triggering native form submission (full page
reload). htmx.trigger() properly triggers HTMX swap.
Also re-enable initial rebindEventLayers so polygons load on first render.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
* feat(gui): add events feed frontend with map and filters
GET /events: Full page with filter form, table, and Leaflet map
GET /events/rows: HTMX fragment for table updates
Features:
- Filterable by adapter, category, time range, region bbox
- Cursor-based pagination with Next button
- Leaflet map showing event geometries
- Click/hover row highlights geometry on map
- Draw rectangle on map to filter by region
- Validation errors shown as banner, not 400
- Events link added to nav between Adapters and Streams
Refactored events query into shared helper for JSON and HTML routes.
Tests: 14 new tests covering filters, fragments, geometry handling.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(events): use shared helpers for /events.json, fix tests
- Refactor /events.json to use _parse_events_params and _fetch_events
helpers, removing ~200 lines of duplicate query logic
- Delete smoke test (test_events_unauthenticated_redirects) that had
no assertions
- Add TestCrossEndpointParity: verify /events.json and /events return
identical results with same params, test category filter and cursor
pagination on both endpoints
- Add TestErrorSemantics: verify /events.json returns 400 on bad params
while /events returns 200 with error banner (intentional API vs HTML
divergence)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add real assertions to since/until and region filter tests
Replace trivial status_code==200 assertions with checks that verify
the filter values were actually parsed and passed to the template.
These tests now fail if the handler ignores the filter parameters.
* fix: remove double-escaping from data-geometry attribute
tojson already produces HTML-attribute-safe JSON. The extra |e filter
was double-escaping, causing JSON.parse to fail in the browser JS.
Switch to single-quoted attribute to avoid conflicts with JSON double
quotes.
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(wizard): implement deferred-commit pattern for setup wizard
Replace the current "POST each step -> DB write -> redirect" architecture
with "collect values across steps in a signed cookie, commit everything
in one transaction at Finish."
Key changes:
- Add wizard.py: WizardState dataclass and cookie helpers
- csrf.py: Add reuse_or_generate_pre_auth_csrf helper
- routes.py: All wizard handlers now use cookie state, no DB writes until finish
- middleware.py: Cookie-based wizard step routing instead of DB queries
- setup_operator.html: Remove "Operator Already Configured" branch
Benefits:
- Back navigation works: can return to any step and edit values
- Atomic commit: all DB writes happen in single transaction at finish
- No orphaned state: failed wizard leaves no DB artifacts
- Simpler auth: pre-auth CSRF for all 5 steps (no session until finish)
Tests updated for new behavior. 287 tests passing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(templates): correct SRI hashes for leaflet.draw assets
The integrity hashes for leaflet.draw.css and leaflet.draw.js were
incorrect, causing browsers to silently block these resources. This
broke the Leaflet.draw toolbar and map rendering for FIRMS/USGS
adapter region pickers.
Updated both setup_adapters.html and adapters_edit.html with the
correct sha512 hashes computed from the actual CDN files.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): return 204 for browser-noise paths to prevent CSRF races
Browser requests for /favicon.ico, /apple-touch-icon.png, etc. were
triggering parallel GET requests that could race with form loads,
causing CSRF token rotation issues.
Added BROWSER_NOISE_PATHS constant and early 204 response in both
SetupGateMiddleware and SessionMiddleware to short-circuit these
requests before any cookie/token handling occurs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
GET /events.json with cursor-based pagination and filtering:
- Filter by adapter, category, since/until, region bbox
- Cursor pagination via (time DESC, id DESC) ordering
- Returns events with GeoJSON geometry parsed as objects
- Validation returns 400 with clear error messages
Migration 014 adds composite index for efficient pagination.
Tests: 17 new tests covering filters, pagination, validation.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): implement first-run setup wizard (1b-8)
Add a 5-step setup wizard that replaces the single-step /setup:
1. Create Operator - create initial operator account
2. System Settings - configure map tile URL and attribution
3. API Keys - optionally add API keys for adapters
4. Configure Adapters - enable/disable adapters with region picker
5. Finish Setup - review and complete setup
Key changes:
- Update middleware to handle wizard URL structure and step routing
- Add wizard routes for each step with proper auth checks
- Create new templates using base_wizard.html for consistent styling
- Add audit events for system.update and setup.complete
- Update tests for new middleware behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle CSRF errors on wizard paths
Update csrf_exception_handler to re-render wizard forms with error
message instead of redirecting to /login when CSRF validation fails.
- /setup/operator: re-render with error
- /setup/system: re-render with current system values + error
- /setup/keys: re-render with current keys list + error
- /setup/adapters: re-render with current adapter config + error
- /setup/finish: re-render with summary data + error
- /setup: redirect to /setup (middleware routes to appropriate step)
Add error display to setup_keys.html and setup_finish.html templates.
Add 7 new CSRF handler tests for wizard paths.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): region picker render + click-to-draw
Bug A: Maps render blank on /setup/adapters for FIRMS and USGS
because Leaflet computed zero dimensions before container layout
settled. Fix: add setTimeout invalidateSize() after map creation.
Bug B: No click-to-draw functionality - only drag corners. Fix:
add L.Control.Draw for rectangle drawing with CREATED event handler
to replace existing rectangle.
Both fixes applied to:
- setup_adapters.html (wizard inline JS)
- _region_picker.html (standalone edit page)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle revisiting /setup/operator after operator created
When an operator already exists, /setup/operator now shows a
confirmation page instead of the create form. This prevents:
- Unique constraint violations on duplicate username
- Silent creation of duplicate operators
GET /setup/operator: queries config.operators; if any exist,
renders confirmation state with existing_operator context.
POST /setup/operator: checks operator count before INSERT; if
non-zero, renders confirmation state without inserting.
Template updated with conditional to show "Operator Already
Configured" message when existing_operator is set.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(csrf): replace fastapi-csrf-protect with session-bound CSRF
Fixes CSRF race condition where every GET rotated the CSRF token,
causing POST failures when users had multiple tabs or slow connections.
Changes:
- Remove fastapi-csrf-protect dependency
- Add session-bound CSRF tokens stored in config.sessions table
- Add pre-auth CSRF for unauthenticated routes (/login, /setup/operator)
- Add csrf.py module for pre-auth token generation/validation
- Update routes to use new CSRF token handling
- Add migration 013 to add csrf_token column to sessions
The session-bound approach ensures CSRF tokens remain stable for the
duration of a session, eliminating the race condition.
Note: Route tests (test_wizard.py, test_adapters.py, etc.) need
refactoring to mock get_settings() instead of CsrfProtect dependency.
Core auth/CSRF handler tests pass (74 tests).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(csrf): update test suite for session-bound CSRF tokens
- Add CSRF fixtures to conftest.py for pre-auth and session CSRF
- Update test_wizard.py: use bypass_pre_auth_csrf and patch_route_settings
- Update test_adapters.py: set request.state.csrf_token and form mock data
- Update test_api_keys.py: add CSRF token to form data for POST routes
- Update test_streams.py: change return_value to side_effect for CSRF support
- Update test_region_picker.py: add CSRF token handling
- Update test_config_store.py: set CENTRAL_CSRF_SECRET env var in fixture
All 285 tests now pass with session-bound CSRF validation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Implement CRUD-lite for config.api_keys with:
- List view showing all keys with usage info (which adapters reference each)
- Add form with alias validation (letters, numbers, underscores only)
- Rotate form to replace encrypted value
- Delete with protection against removing keys still referenced by adapters
Security:
- Plaintext keys never displayed back to user
- Values encrypted via crypto.encrypt() before storage
- Audit logs contain only metadata, never plaintext or encrypted values
Routes:
- GET /api-keys - list all keys
- GET /api-keys/new - add form
- POST /api-keys - create key
- GET /api-keys/{alias} - edit/rotate/delete form
- POST /api-keys/{alias} - rotate key
- POST /api-keys/{alias}/delete - delete key
Tests: 11 new tests covering list, create, rotate, delete, and audit
verification.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* test(bootstrap): isolate env vars in test_reads_from_env_file
The test was failing on CT104 because live CENTRAL_DB_DSN
environment variable overrode the test .env file content.
Fix: use monkeypatch.delenv to clear all CENTRAL_* env vars
before creating the Settings object, ensuring the test env
file is the only source of configuration values.
Also add CENTRAL_CSRF_SECRET to test env file since it's
now a required field.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(models): remove stale test_custom_prefix test
The test called subject_for_event(event, prefix="myapp.events")
but the prefix parameter was removed from the API.
The prefix functionality was intentionally removed - subjects
now always use the "central." prefix hardcoded in the function.
Delete the test rather than re-add the parameter.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(nws): update fixtures for new adapter signature and region filtering
NWSAdapter.__init__ signature changed from (config, cursor_db_path)
to (config, config_store, cursor_db_path) with config now being
AdapterConfig with a settings dict instead of NWSAdapterConfig.
Also adapts tests to region-based bbox filtering:
- TestStateFilter now uses region bbox to accept PNW, reject CA
- Add geometry to SAMPLE_FEATURE_OR so it passes region filter
- Other test fixtures use region=None to skip filtering
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): add streams view (1b-6)
Add streams list and edit routes with live JetStream data:
- GET /streams: list all streams with live size/messages
- POST /streams/{name}: update max_age_s with validation
Features:
- Live data from JetStream (bytes, messages, timestamps)
- Graceful degradation when NATS unavailable
- Preset chip buttons (1d, 7d, 14d, 30d, 365d)
- Custom days input with Save button
- Current selection highlighted
- Managed by supervisor badge
- Audit logging with before/after max_age_s
Files:
- src/central/gui/audit.py: add STREAM_UPDATE constant
- src/central/gui/routes.py: add streams_list and streams_update handlers
- src/central/gui/templates/base.html: add Streams nav link
- src/central/gui/templates/streams_list.html: new template
- tests/test_streams.py: 9 tests covering all requirements
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): use get_msg().time for stream timestamps, fix badge layout
- nats-py StreamState doesn't expose first_ts/last_ts
- Fetch timestamps via js.get_msg(stream, seq=N).time instead
- Handle edge cases: empty streams, single-message streams, get_msg failures
- Fix badge overlap using flex layout instead of float:right
- Change label from "Max bytes (config)" to "Max bytes (current)"
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Update test fixtures to match current Supervisor and adapter signatures:
- Add mock_config_store fixture
- Pass config_store to Supervisor constructor
- Update MockNWSAdapter to accept (config, config_store, cursor_db_path)
- Add apply_config method to MockNWSAdapter
The supervisor code correctly preserves last_completed_poll across
enable/disable cycles. Tests were failing due to outdated constructor
signatures, not a bug in the rate-limiting logic.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): add Leaflet region picker to adapter edit (1b-5)
- Add _region_picker.html template with Leaflet map and editable rectangle
- Add Leaflet 1.9.4 and Leaflet.draw 1.0.4 CDN deps to adapters_edit.html
- Update GET /adapters/{name} to fetch map_tile_url from config.system
- Update POST /adapters/{name} to validate and save region coordinates
- Validation: -90 <= south < north <= 90, -180 <= west < east <= 180
- Region changes flow through to audit log via existing settings capture
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(tests): update adapter tests for region picker mocks
Add region coordinates to form data mocks and system settings rows
to fetchrow.side_effect for tests that re-render on validation errors.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Fix A - /dashboard/polls:
- Use get_last_msg instead of pull_subscribe (no durable consumers)
- Fix subject filter: central.meta.adapter.{name}.status
- Parse correct fields: ts and ok from status message
- Handle NotFoundError gracefully when no status exists
Fix B - CSRF exception handler:
- Add global CsrfProtectError handler in __init__.py
- Return friendly "session expired" message instead of 500
- Re-render forms with error or redirect to /login
- Update templates to display error messages
Tests:
- Add get_last_msg mocking tests for polls
- Add regression test verifying no pull_subscribe
- Add CSRF handler tests
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Now that routes.py no longer calls json.loads() on settings, the test
mocks must return dicts directly (as asyncpg does with jsonb).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
