Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NASA WaterData OGC API v0 (latest-continuous collection) — polls configured
parameter codes within an operator-set bbox and publishes on the new
CENTRAL_HYDRO stream.
- Subject: central.hydro.<parameter_code>.<agency>.<bare_site_no>
(e.g. central.hydro.00060.usgs.05420500). The agency/site decomposition
lives in a single _subject_tokens_for_id helper.
- Default parameter codes: 00060 (discharge), 00065 (gage height),
00010 (water temperature). Operator-tunable; single SoT in
_DEFAULT_PARAMETER_CODES — no parallel literals.
- Composite dedup: nwis:<monitoring_location_id>:<param>:<time_iso>.
Prefix kept in dedup key for cross-agency uniqueness.
- Pagination: follows OGC 'rel=next' link until absent (cursor-based).
- Region bbox is REQUIRED in practice; adapter logs WARN at startup if
region is None (does not refuse to start).
- New stream CENTRAL_HYDRO added to streams.py registry (one line).
Retention mirrors CENTRAL_DISASTER (7 days, 1 GiB).
- No removal pattern in v1 — sites are static; missing data is the signal.
Upstream divergences from the original spec brief, caught by pre-build curl:
- Collection is 'latest-continuous', not 'instantaneous-values'.
- Site filter param is 'monitoring_location_id' (singular), not
'monitoring_locations_id' (plural).
- Site identifier requires agency prefix in queries (USGS-NNNNN).
- feature.id is a per-record UUID, not stable; dedup uses joint key.
Ships disabled; operator enables via GUI after setting a bbox.
Adds the NASA Earth Observatory Natural Event Tracker (EONET v3) adapter,
publishing on the existing CENTRAL_DISASTER stream under
central.disaster.eonet.<category>.global subjects.
- One Central event per EONET event id; geo = most-recent geometry point.
- Composite dedup key (eonet:<id>:<latest_geometry_date_iso>) — timeline
advance re-publishes, idle re-poll suppresses.
- category_allowlist defaults to all 13 upstream categories; operator opts
OUT per-category if GDACS overlap (wildfires/floods/severeStorms/volcanoes)
produces unwanted dupes on gdacs.* subjects.
- camelCase upstream IDs (seaLakeIce, dustHaze, etc.) mapped to
lower_snake_case subject components by a single _subject_category helper.
- Country resolves to literal 'global' (no reverse-geocode in v1).
- Fall-off: missing-from-feed event emits central.disaster.eonet.<cat>.removed.global,
subtype before 'removed' per §8 canonical pattern.
Adapter ships disabled; operator enables via GUI.
Adds the GDACS (Global Disaster Alert and Coordination System) adapter
against the self-describing framework. Polls https://www.gdacs.org/xml/rss.xml
every 600s, parses the RSS items, and publishes to a new CENTRAL_DISASTER
JetStream stream on central.disaster.<eventtype_lower>.<country_lower>.
Locked decisions:
- Keep: WF, DR, FL, VO, TC. Drop: EQ (USGS canonical on central.quake.>),
plus any future-unknown eventtype.
- Filter via settings_schema event_types: list[str] so operators can
re-allow without a code change.
- Dedup by RSS guid (format <eventtype><eventid>, stable across reissue).
- Severity from gdacs:alertlevel (Green=1, Orange=2, Red=3, default 0).
- Fall-off uses GDACS gdacs:iscurrent=false as explicit tombstone signal,
with a fallback for items that vanish entirely from the feed. Tombstones
publish on disaster.removed.<eventtype>.<country>.
- Geo: centroid from geo:Point, bbox from gdacs:bbox (reordered to Geo
(minLon, minLat, maxLon, maxLat)), primary_region from gdacs:iso3.
CENTRAL_DISASTER stream: 7d retention, 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE / CENTRAL_SPACE. Migrations 020 (adapter row,
enabled=false, default event_types in settings) and 021 (stream seed).
STREAM_SUBJECTS, archive STREAMS, GUI DASHBOARD_STREAMS each pick up
the new stream.
Tests: 14 new in tests/test_gdacs.py using frozen RSS fixtures with WF/DR/EQ/XX
items (covering normalization, EQ drop, unknown drop, settings override,
guid dedup, iscurrent=false tombstone, missing-from-feed tombstone,
helper boundaries). Stream-count assertions bumped 4->5 and 5->6 for
the new stream (anti-pattern noted; queued as a follow-up PR E.5).
+1 membership test test_streams_contains_central_disaster.
Full suite: 426 passed.
End-to-end on CT104: 48 events published on first poll (44 disaster.wf +
4 disaster.fl), zero EQ events, all subjects under central.disaster.>
with lowercase-hyphenated country suffixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent adapters sharing src/central/adapters/swpc_common.py,
mirroring the WFIGS two-adapter pattern. Each adapter has its own row in
config.adapters (ships disabled), its own cadence, and its own dedup
state, so operators can independently enable/disable and so a broken
upstream endpoint does not silently mask a healthy one.
Subjects:
swpc_alerts -> central.space.alert.<product_id_lower>
swpc_kindex -> central.space.kindex
swpc_protons -> central.space.proton_flux
Dedup keys:
alerts: product_id + issue_datetime
kindex: time_tag
protons: time_tag + energy
Severity: G-scale on product_id for K0[5-9][AW] alerts (G1-G5 -> 1-4),
G-scale on Kp for kindex, 0 for protons (raw flux carried in event.data).
No geo on any SWPC events (centroid=None, regions=[], primary_region=None).
No fall-off detection for alerts -- a single 115-row sample cannot confirm
whether alerts disappear from the upstream JSON when expired; deferred to
a later pass after 24h of observation.
CENTRAL_SPACE stream seeded with 7-day retention / 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE. STREAM_SUBJECTS, archive STREAMS, and
DASHBOARD_STREAMS each pick up the new stream.
Tests: 16 new cases in tests/test_swpc.py using real-shape frozen JSON
fixtures (alerts product_ids EF3A/K05A/K07A; kindex Kp boundaries; protons
composite dedup). Two existing tests updated for the new stream count
(test_archive_multi_stream.test_streams_list_has_three_entries renamed to
_has_four_entries; test_dashboard expects 5 streams not 4); added a
test_streams_contains_central_space companion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
InciWeb adapter for RSS-based wildfire narrative updates:
- Parse DMS coordinates from description text
- Extract state name and map to 2-letter code
- Strip HTML tags and decode entities
- Bbox filtering for regional focus
- Dedup via published_ids table (14-day sweep)
- Category: fire.narrative.inciweb
- Subject: central.fire.narrative.inciweb.<state>
Includes migration 017 and 15 unit tests.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two new adapters for wildfire data from NIFC WFIGS:
- wfigs_incidents: Active fire incident locations
- wfigs_perimeters: Active fire perimeter polygons
Features:
- IRWIN GUID dedup via is_published/mark_published
- Fall-off detection with removal events when fires exit current
- Bbox post-filtering with shapely polygon intersection
- Severity mapping from DailyAcres (0-4 scale)
- Subject hierarchy: central.fire.<layer>.<state>.<county>
Ships disabled by default; operators enable via GUI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Make migration 015 idempotent with IF NOT EXISTS
2. Remove hardcoded cadence range from routes.py and template:
- Added ge=10 constraint to AdapterConfig.cadence_s field
- Removed manual 60-3600 check from routes.py POST handler
- Validate cadence using AdapterConfig field metadata
- Removed min/max attributes from template input
3. Move discover_adapters to its own module:
- Created src/central/adapter_discovery.py
- Updated supervisor.py to import from adapter_discovery
- Updated routes.py to import from adapter_discovery
- GUI no longer transitively imports nats or stream_manager
4. Remove dead code branch in form_descriptors.py:
- Removed unreachable RegionConfig check (already handled earlier)
- Improved error message for unsupported nested types
5. Updated test_adapters.py:
- Changed invalid cadence test from 30 to 5 (below ge=10)
- Updated assertion to check for "10" in error message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migration 015: Adds last_error TEXT column to config.adapters.
Populated by supervisor when an adapter fails to start or apply config.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
GET /events.json with cursor-based pagination and filtering:
- Filter by adapter, category, since/until, region bbox
- Cursor pagination via (time DESC, id DESC) ordering
- Returns events with GeoJSON geometry parsed as objects
- Validation returns 400 with clear error messages
Migration 014 adds composite index for efficient pagination.
Tests: 17 new tests covering filters, pagination, validation.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): implement first-run setup wizard (1b-8)
Add a 5-step setup wizard that replaces the single-step /setup:
1. Create Operator - create initial operator account
2. System Settings - configure map tile URL and attribution
3. API Keys - optionally add API keys for adapters
4. Configure Adapters - enable/disable adapters with region picker
5. Finish Setup - review and complete setup
Key changes:
- Update middleware to handle wizard URL structure and step routing
- Add wizard routes for each step with proper auth checks
- Create new templates using base_wizard.html for consistent styling
- Add audit events for system.update and setup.complete
- Update tests for new middleware behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle CSRF errors on wizard paths
Update csrf_exception_handler to re-render wizard forms with error
message instead of redirecting to /login when CSRF validation fails.
- /setup/operator: re-render with error
- /setup/system: re-render with current system values + error
- /setup/keys: re-render with current keys list + error
- /setup/adapters: re-render with current adapter config + error
- /setup/finish: re-render with summary data + error
- /setup: redirect to /setup (middleware routes to appropriate step)
Add error display to setup_keys.html and setup_finish.html templates.
Add 7 new CSRF handler tests for wizard paths.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): region picker render + click-to-draw
Bug A: Maps render blank on /setup/adapters for FIRMS and USGS
because Leaflet computed zero dimensions before container layout
settled. Fix: add setTimeout invalidateSize() after map creation.
Bug B: No click-to-draw functionality - only drag corners. Fix:
add L.Control.Draw for rectangle drawing with CREATED event handler
to replace existing rectangle.
Both fixes applied to:
- setup_adapters.html (wizard inline JS)
- _region_picker.html (standalone edit page)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle revisiting /setup/operator after operator created
When an operator already exists, /setup/operator now shows a
confirmation page instead of the create form. This prevents:
- Unique constraint violations on duplicate username
- Silent creation of duplicate operators
GET /setup/operator: queries config.operators; if any exist,
renders confirmation state with existing_operator context.
POST /setup/operator: checks operator count before INSERT; if
non-zero, renders confirmation state without inserting.
Template updated with conditional to show "Operator Already
Configured" message when existing_operator is set.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(csrf): replace fastapi-csrf-protect with session-bound CSRF
Fixes CSRF race condition where every GET rotated the CSRF token,
causing POST failures when users had multiple tabs or slow connections.
Changes:
- Remove fastapi-csrf-protect dependency
- Add session-bound CSRF tokens stored in config.sessions table
- Add pre-auth CSRF for unauthenticated routes (/login, /setup/operator)
- Add csrf.py module for pre-auth token generation/validation
- Update routes to use new CSRF token handling
- Add migration 013 to add csrf_token column to sessions
The session-bound approach ensures CSRF tokens remain stable for the
duration of a session, eliminating the race condition.
Note: Route tests (test_wizard.py, test_adapters.py, etc.) need
refactoring to mock get_settings() instead of CsrfProtect dependency.
Core auth/CSRF handler tests pass (74 tests).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(csrf): update test suite for session-bound CSRF tokens
- Add CSRF fixtures to conftest.py for pre-auth and session CSRF
- Update test_wizard.py: use bypass_pre_auth_csrf and patch_route_settings
- Update test_adapters.py: set request.state.csrf_token and form mock data
- Update test_api_keys.py: add CSRF token to form data for POST routes
- Update test_streams.py: change return_value to side_effect for CSRF support
- Update test_region_picker.py: add CSRF token handling
- Update test_config_store.py: set CENTRAL_CSRF_SECRET env var in fixture
All 285 tests now pass with session-bound CSRF validation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Replaces module-path-based source column (e.g. "central/adapters/nws")
with stable adapter identifier (e.g. "nws") that foreign-keys to
config.adapters.name.
Migration 011:
- ADD COLUMN adapter TEXT
- Backfill via REPLACE(source, 'central/adapters/', '')
- SET NOT NULL + FK RESTRICT
- CREATE INDEX (adapter, received DESC) for dashboard queries
- DROP COLUMN source
Code changes:
- Event model: source field renamed to adapter
- All adapters: use adapter="name" instead of source="central/adapters/name"
- Archive: write adapter column instead of source
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove states array from NWS settings
- Add region bbox covering ID/OR/WA/MT/WY/UT/NV
- Bbox: north=49.5, south=31.0, east=-102.0, west=-124.5
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- config.streams table for JetStream retention config
- Column-filtered NOTIFY: only fires on max_age_s changes
- Prevents self-loop when supervisor updates max_bytes
- Seeds CENTRAL_WX (7d/10GB) and CENTRAL_META (1d/100MB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds auto-update trigger for updated_at column on adapters table
and partial index for efficient enabled adapter queries.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
