Adds the GDACS (Global Disaster Alert and Coordination System) adapter
against the self-describing framework. Polls https://www.gdacs.org/xml/rss.xml
every 600s, parses the RSS items, and publishes to a new CENTRAL_DISASTER
JetStream stream on central.disaster.<eventtype_lower>.<country_lower>.
Locked decisions:
- Keep: WF, DR, FL, VO, TC. Drop: EQ (USGS canonical on central.quake.>),
plus any future-unknown eventtype.
- Filter via settings_schema event_types: list[str] so operators can
re-allow without a code change.
- Dedup by RSS guid (format <eventtype><eventid>, stable across reissue).
- Severity from gdacs:alertlevel (Green=1, Orange=2, Red=3, default 0).
- Fall-off uses GDACS gdacs:iscurrent=false as explicit tombstone signal,
with a fallback for items that vanish entirely from the feed. Tombstones
publish on disaster.removed.<eventtype>.<country>.
- Geo: centroid from geo:Point, bbox from gdacs:bbox (reordered to Geo
(minLon, minLat, maxLon, maxLat)), primary_region from gdacs:iso3.
CENTRAL_DISASTER stream: 7d retention, 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE / CENTRAL_SPACE. Migrations 020 (adapter row,
enabled=false, default event_types in settings) and 021 (stream seed).
STREAM_SUBJECTS, archive STREAMS, GUI DASHBOARD_STREAMS each pick up
the new stream.
Tests: 14 new in tests/test_gdacs.py using frozen RSS fixtures with WF/DR/EQ/XX
items (covering normalization, EQ drop, unknown drop, settings override,
guid dedup, iscurrent=false tombstone, missing-from-feed tombstone,
helper boundaries). Stream-count assertions bumped 4->5 and 5->6 for
the new stream (anti-pattern noted; queued as a follow-up PR E.5).
+1 membership test test_streams_contains_central_disaster.
Full suite: 426 passed.
End-to-end on CT104: 48 events published on first poll (44 disaster.wf +
4 disaster.fl), zero EQ events, all subjects under central.disaster.>
with lowercase-hyphenated country suffixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent adapters sharing src/central/adapters/swpc_common.py,
mirroring the WFIGS two-adapter pattern. Each adapter has its own row in
config.adapters (ships disabled), its own cadence, and its own dedup
state, so operators can independently enable/disable and so a broken
upstream endpoint does not silently mask a healthy one.
Subjects:
swpc_alerts -> central.space.alert.<product_id_lower>
swpc_kindex -> central.space.kindex
swpc_protons -> central.space.proton_flux
Dedup keys:
alerts: product_id + issue_datetime
kindex: time_tag
protons: time_tag + energy
Severity: G-scale on product_id for K0[5-9][AW] alerts (G1-G5 -> 1-4),
G-scale on Kp for kindex, 0 for protons (raw flux carried in event.data).
No geo on any SWPC events (centroid=None, regions=[], primary_region=None).
No fall-off detection for alerts -- a single 115-row sample cannot confirm
whether alerts disappear from the upstream JSON when expired; deferred to
a later pass after 24h of observation.
CENTRAL_SPACE stream seeded with 7-day retention / 1 GiB max_bytes, mirroring
CENTRAL_FIRE / CENTRAL_QUAKE. STREAM_SUBJECTS, archive STREAMS, and
DASHBOARD_STREAMS each pick up the new stream.
Tests: 16 new cases in tests/test_swpc.py using real-shape frozen JSON
fixtures (alerts product_ids EF3A/K05A/K07A; kindex Kp boundaries; protons
composite dedup). Two existing tests updated for the new stream count
(test_archive_multi_stream.test_streams_list_has_three_entries renamed to
_has_four_entries; test_dashboard expects 5 streams not 4); added a
test_streams_contains_central_space companion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
InciWeb adapter for RSS-based wildfire narrative updates:
- Parse DMS coordinates from description text
- Extract state name and map to 2-letter code
- Strip HTML tags and decode entities
- Bbox filtering for regional focus
- Dedup via published_ids table (14-day sweep)
- Category: fire.narrative.inciweb
- Subject: central.fire.narrative.inciweb.<state>
Includes migration 017 and 15 unit tests.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two new adapters for wildfire data from NIFC WFIGS:
- wfigs_incidents: Active fire incident locations
- wfigs_perimeters: Active fire perimeter polygons
Features:
- IRWIN GUID dedup via is_published/mark_published
- Fall-off detection with removal events when fires exit current
- Bbox post-filtering with shapely polygon intersection
- Severity mapping from DailyAcres (0-4 scale)
- Subject hierarchy: central.fire.<layer>.<state>.<county>
Ships disabled by default; operators enable via GUI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Make migration 015 idempotent with IF NOT EXISTS
2. Remove hardcoded cadence range from routes.py and template:
- Added ge=10 constraint to AdapterConfig.cadence_s field
- Removed manual 60-3600 check from routes.py POST handler
- Validate cadence using AdapterConfig field metadata
- Removed min/max attributes from template input
3. Move discover_adapters to its own module:
- Created src/central/adapter_discovery.py
- Updated supervisor.py to import from adapter_discovery
- Updated routes.py to import from adapter_discovery
- GUI no longer transitively imports nats or stream_manager
4. Remove dead code branch in form_descriptors.py:
- Removed unreachable RegionConfig check (already handled earlier)
- Improved error message for unsupported nested types
5. Updated test_adapters.py:
- Changed invalid cadence test from 30 to 5 (below ge=10)
- Updated assertion to check for "10" in error message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migration 015: Adds last_error TEXT column to config.adapters.
Populated by supervisor when an adapter fails to start or apply config.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
GET /events.json with cursor-based pagination and filtering:
- Filter by adapter, category, since/until, region bbox
- Cursor pagination via (time DESC, id DESC) ordering
- Returns events with GeoJSON geometry parsed as objects
- Validation returns 400 with clear error messages
Migration 014 adds composite index for efficient pagination.
Tests: 17 new tests covering filters, pagination, validation.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat(gui): implement first-run setup wizard (1b-8)
Add a 5-step setup wizard that replaces the single-step /setup:
1. Create Operator - create initial operator account
2. System Settings - configure map tile URL and attribution
3. API Keys - optionally add API keys for adapters
4. Configure Adapters - enable/disable adapters with region picker
5. Finish Setup - review and complete setup
Key changes:
- Update middleware to handle wizard URL structure and step routing
- Add wizard routes for each step with proper auth checks
- Create new templates using base_wizard.html for consistent styling
- Add audit events for system.update and setup.complete
- Update tests for new middleware behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle CSRF errors on wizard paths
Update csrf_exception_handler to re-render wizard forms with error
message instead of redirecting to /login when CSRF validation fails.
- /setup/operator: re-render with error
- /setup/system: re-render with current system values + error
- /setup/keys: re-render with current keys list + error
- /setup/adapters: re-render with current adapter config + error
- /setup/finish: re-render with summary data + error
- /setup: redirect to /setup (middleware routes to appropriate step)
Add error display to setup_keys.html and setup_finish.html templates.
Add 7 new CSRF handler tests for wizard paths.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): region picker render + click-to-draw
Bug A: Maps render blank on /setup/adapters for FIRMS and USGS
because Leaflet computed zero dimensions before container layout
settled. Fix: add setTimeout invalidateSize() after map creation.
Bug B: No click-to-draw functionality - only drag corners. Fix:
add L.Control.Draw for rectangle drawing with CREATED event handler
to replace existing rectangle.
Both fixes applied to:
- setup_adapters.html (wizard inline JS)
- _region_picker.html (standalone edit page)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(gui): handle revisiting /setup/operator after operator created
When an operator already exists, /setup/operator now shows a
confirmation page instead of the create form. This prevents:
- Unique constraint violations on duplicate username
- Silent creation of duplicate operators
GET /setup/operator: queries config.operators; if any exist,
renders confirmation state with existing_operator context.
POST /setup/operator: checks operator count before INSERT; if
non-zero, renders confirmation state without inserting.
Template updated with conditional to show "Operator Already
Configured" message when existing_operator is set.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(csrf): replace fastapi-csrf-protect with session-bound CSRF
Fixes CSRF race condition where every GET rotated the CSRF token,
causing POST failures when users had multiple tabs or slow connections.
Changes:
- Remove fastapi-csrf-protect dependency
- Add session-bound CSRF tokens stored in config.sessions table
- Add pre-auth CSRF for unauthenticated routes (/login, /setup/operator)
- Add csrf.py module for pre-auth token generation/validation
- Update routes to use new CSRF token handling
- Add migration 013 to add csrf_token column to sessions
The session-bound approach ensures CSRF tokens remain stable for the
duration of a session, eliminating the race condition.
Note: Route tests (test_wizard.py, test_adapters.py, etc.) need
refactoring to mock get_settings() instead of CsrfProtect dependency.
Core auth/CSRF handler tests pass (74 tests).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(csrf): update test suite for session-bound CSRF tokens
- Add CSRF fixtures to conftest.py for pre-auth and session CSRF
- Update test_wizard.py: use bypass_pre_auth_csrf and patch_route_settings
- Update test_adapters.py: set request.state.csrf_token and form mock data
- Update test_api_keys.py: add CSRF token to form data for POST routes
- Update test_streams.py: change return_value to side_effect for CSRF support
- Update test_region_picker.py: add CSRF token handling
- Update test_config_store.py: set CENTRAL_CSRF_SECRET env var in fixture
All 285 tests now pass with session-bound CSRF validation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Matt Johnson <mj@k7zvx.com>
- Replace pytest.skip stubs with actual DB tests against central_test
- Test backfill for all three adapters (nws, firms, usgs_quake)
- Test FK RESTRICT, NOT NULL, and FK validation constraints
- Test schema changes (source dropped, adapter exists with constraints)
- Delete stale sql/schema.sql (migrations are sole source of truth)
- Update docs/migrations.md with schema.sql removal note
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replaces module-path-based source column (e.g. "central/adapters/nws")
with stable adapter identifier (e.g. "nws") that foreign-keys to
config.adapters.name.
Migration 011:
- ADD COLUMN adapter TEXT
- Backfill via REPLACE(source, 'central/adapters/', '')
- SET NOT NULL + FK RESTRICT
- CREATE INDEX (adapter, received DESC) for dashboard queries
- DROP COLUMN source
Code changes:
- Event model: source field renamed to adapter
- All adapters: use adapter="name" instead of source="central/adapters/name"
- Archive: write adapter column instead of source
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove states array from NWS settings
- Add region bbox covering ID/OR/WA/MT/WY/UT/NV
- Bbox: north=49.5, south=31.0, east=-102.0, west=-124.5
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- config.streams table for JetStream retention config
- Column-filtered NOTIFY: only fires on max_age_s changes
- Prevents self-loop when supervisor updates max_bytes
- Seeds CENTRAL_WX (7d/10GB) and CENTRAL_META (1d/100MB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds auto-update trigger for updated_at column on adapters table
and partial index for efficient enabled adapter queries.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
