Commit graph

146 commits

Author SHA1 Message Date
Ubuntu
9396e5dbe8 fix(gui): dashboard polls card + CSRF exception handler
Fix A - /dashboard/polls:
- Use get_last_msg instead of pull_subscribe (no durable consumers)
- Fix subject filter: central.meta.adapter.{name}.status
- Parse correct fields: ts and ok from status message
- Handle NotFoundError gracefully when no status exists

Fix B - CSRF exception handler:
- Add global CsrfProtectError handler in __init__.py
- Return friendly "session expired" message instead of 500
- Re-render forms with error or redirect to /login
- Update templates to display error messages

Tests:
- Add get_last_msg mocking tests for polls
- Add regression test verifying no pull_subscribe
- Add CSRF handler tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 22:34:13 +00:00
4368c83613
Merge pull request #17 from zvx-echo6/feature/1b-4-adapters-list-edit
feat(gui): adapters list and edit UI (1b-4)
2026-05-17 16:04:54 -06:00
Ubuntu
5be002cb03 test(adapters): fix mock settings to use dicts instead of JSON strings
Now that routes.py no longer calls json.loads() on settings, the test
mocks must return dicts directly (as asyncpg does with jsonb).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 21:35:51 +00:00
Ubuntu
0f127399b3 fix(gui): remove JSONB double-encoding in adapter updates
The GUI pool has init=_setup_json_codec registered, which makes asyncpg
auto-serialize Python dicts to JSONB. Calling json.dumps() on a dict
before passing it to asyncpg double-encodes - the value gets stored as
a JSON-encoded string rather than a JSON object.

Changes:
- Remove json.dumps() from UPDATE statement in adapters_edit_submit
- Remove defensive isinstance(settings, str) checks that masked the bug
- Add regression tests to verify settings is passed as dict, not string

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 21:33:48 +00:00
Matt Johnson
dec8ce8545 feat(gui): add adapters list and edit UI (1b-4)
- Add GET /adapters route for listing all adapters
- Add GET /adapters/{name} for edit form with per-adapter fields
- Add POST /adapters/{name} for validation, update, and audit
- Add ADAPTER_UPDATE audit constant
- Add Adapters nav link to base.html
- Server-side validation for cadence (60-3600), email format,
  api_key_alias existence, satellites, and feed values
- Region displayed read-only with 1b-5 placeholder
- Hot reload via existing NOTIFY trigger (no new mechanism)
- Add comprehensive tests (9 tests)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 21:19:40 +00:00
dbe7f8f868
Merge pull request #16 from zvx-echo6/feature/1b-3-dashboard
feat(gui): read-only dashboard (1b-3)
2026-05-17 14:37:21 -06:00
Matt Johnson
736b637d31 feat(gui): add read-only dashboard with HTMX polling
- Add NATS connection module (nats.py) for JetStream access
- Add three dashboard cards: events (24h), stream sizes, poll times
- Replace placeholder index with HTMX-polling dashboard
- Graceful degradation when NATS unavailable (200 with error, not 500)
- Per-stream/adapter failure isolation
- Add comprehensive dashboard tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 20:09:05 +00:00
c31de2499d
Merge pull request #15 from zvx-echo6/feature/1b-3b-archive-multi-stream
fix(archive): subscribe to all event streams
2026-05-17 13:49:15 -06:00
Matt Johnson
6b5f6709e4 fix(archive): subscribe to all event streams
- One durable consumer per event-bearing stream (CENTRAL_WX,
  CENTRAL_FIRE, CENTRAL_QUAKE) for independent ack tracking
- max_deliver=5 prevents poison-message infinite loops
- Orphaned 'archive' consumer on CENTRAL_WX cleaned up on startup
- Consumer naming: archive-{stream_name_lower}

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 19:29:38 +00:00
a1e16547a0
Merge pull request #14 from zvx-echo6/feature/postgis-test-db
docs: add test database setup, restore geom to test fixture
2026-05-17 12:37:33 -06:00
Matt Johnson
83b1e45fa8 docs: add test database setup, restore geom to test fixture
- Add docs/test-database.md with one-time setup, DSN convention, reset
  instructions, and explanation of why PostGIS is not in migrations
- Update docs/migrations.md with "Extensions are not in migrations"
  section explaining superuser requirement
- Restore geom GEOMETRY(Geometry, 4326) column to test fixture now that
  central_test has PostGIS installed
- Add CREATE EXTENSION IF NOT EXISTS postgis to test fixture for
  self-bootstrap (central_test is superuser)
- Add Testing section to README.md pointing to docs/test-database.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 18:26:48 +00:00
71c73b4eb1
Merge pull request #13 from zvx-echo6/feature/1b-3a-events-adapter-column
feat(schema): add adapter column to events, drop source
2026-05-17 12:11:53 -06:00
Matt Johnson
a25b4af4e8 fix(tests): remove geom column from test fixture (no PostGIS in test DB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 17:40:24 +00:00
Matt Johnson
98e9d95810 fix(tests): replace stub tests with real DB migration tests
- Replace pytest.skip stubs with actual DB tests against central_test
- Test backfill for all three adapters (nws, firms, usgs_quake)
- Test FK RESTRICT, NOT NULL, and FK validation constraints
- Test schema changes (source dropped, adapter exists with constraints)
- Delete stale sql/schema.sql (migrations are sole source of truth)
- Update docs/migrations.md with schema.sql removal note

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 17:39:38 +00:00
Ubuntu
8601a19f60 feat(schema): add adapter column to events, drop source
Replaces module-path-based source column (e.g. "central/adapters/nws")
with stable adapter identifier (e.g. "nws") that foreign-keys to
config.adapters.name.

Migration 011:
- ADD COLUMN adapter TEXT
- Backfill via REPLACE(source, 'central/adapters/', '')
- SET NOT NULL + FK RESTRICT
- CREATE INDEX (adapter, received DESC) for dashboard queries
- DROP COLUMN source

Code changes:
- Event model: source field renamed to adapter
- All adapters: use adapter="name" instead of source="central/adapters/name"
- Archive: write adapter column instead of source

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 16:09:59 +00:00
4c9ca176a9
Merge pull request #12 from zvx-echo6/feature/1b-2-auth-and-setup
feat(gui): auth core + first-run gate + operator creation (1b-2)
2026-05-17 01:12:32 -06:00
Ubuntu
e469c3833b fix(gui): pass raw CSRF token to form templates
The library's validate_csrf expects the raw token in the form and
the signed token in the cookie. Previously we were putting the signed
token in both places, which caused signature mismatch errors.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 07:05:25 +00:00
Ubuntu
17dd653bd8 fix(gui): use fastapi-csrf-protect native body-token validation
The library supports form-data tokens via token_location="body" and
token_key config options, which we missed in the initial integration.
Removed hand-rolled _validate_csrf_form helper in favor of the
library's validate_csrf method.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 07:00:57 +00:00
Matt Johnson
c529708c75 fix(gui): add form-based CSRF validation and fix index context
- Add _validate_csrf_form helper for form-based CSRF token validation
  (compares form csrf_token with fastapi-csrf-token cookie)
- Fix index route to pass operator and csrf_token to template context

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 06:28:16 +00:00
Matt Johnson
b1ba2d1863 fix(tests): update tests for lazy app loading and 302 redirect
- test_gui_scaffold.py: use standalone router instead of importing app
  to avoid triggering settings load during test collection
- test_setup_gate.py: expect 302 (not 307) for setup gate redirect

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 06:14:25 +00:00
Matt Johnson
1fefc0f491 fix(gui): revert port to 8000, use 302 for setup gate redirect
- Revert uvicorn port from 8088 to 8000 (1b-1 pinned value)
- Change SetupGateMiddleware redirect from 307 to 302 for consistency
  with all other redirects in the codebase

Port 8000 confirmed free on CT104. Earlier change to 8088 was
incorrect — 8080 is held by NATS WebSocket, not 8000.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 06:13:13 +00:00
Matt Johnson
f059f982bc feat(gui): add auth core, setup gate, and first-run operator creation
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 05:30:49 +00:00
afde118d35
Merge pull request #11 from zvx-echo6/feature/1b-1-gui-scaffold
feat(gui): Phase 1b-1 GUI scaffold
2026-05-16 22:39:12 -06:00
Matt Johnson
614312db36 feat(gui): add FastAPI + Jinja2 + HTMX scaffold
- FastAPI app with Jinja2 templates and Pico CSS + HTMX from CDN
- Routes: GET / (placeholder page), GET /health (JSON healthcheck)
- systemd unit (no Install section - manual start only)
- TestClient tests for both endpoints

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-17 04:32:39 +00:00
69ef8b3e93
Merge pull request #10 from zvx-echo6/docs/migration-policy
docs: migration idempotency and runner policy
2026-05-16 22:08:48 -06:00
Matt Johnson
315f5cdab6 docs: migration idempotency and runner policy 2026-05-17 04:08:20 +00:00
158504f335
Merge pull request #9 from zvx-echo6/chore/close-out-1a v0.2.0
chore: Phase 1a close-out
2026-05-16 16:26:34 -06:00
Matt Johnson
9bdd4bc73e docs: changelog for v0.2.0 2026-05-16 22:26:12 +00:00
Matt Johnson
374a8c067f chore: normalize line endings to LF 2026-05-16 22:26:12 +00:00
43088d7fbb
Merge pull request #8 from zvx-echo6/feature/1a-7-usgs-quake
feat: Phase 1a-7 USGS earthquake adapter
2026-05-16 16:24:16 -06:00
Matt Johnson
e0df0bb4aa docs: add USGS quake GUI planning notes
- Feed selection (all_hour vs all_day)
- Magnitude tier color coding for GUI display

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:51 +00:00
Matt Johnson
39d5226661 feat(supervisor): wire USGS quake adapter
- Add USGSQuakeAdapter to _ADAPTER_REGISTRY
- Add CENTRAL_QUAKE stream to STREAM_SUBJECTS

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:46 +00:00
Matt Johnson
668027b442 feat(models): add quake event subject routing
Update subject_for_event to handle quake.* category events.
Subject format: central.quake.event.<magnitude_tier>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:41 +00:00
Matt Johnson
aacf06499b feat(adapters): add USGS earthquake adapter
USGS Earthquake Hazards Program adapter:
- Polls GeoJSON feed (all_hour default, configurable)
- Magnitude tier classification (minor/light/moderate/strong/major/great)
- Deduplication via USGS stable event ID
- Region filter via shapely point-in-bbox
- Skips events with null magnitude (quarry blasts, etc.)

Includes comprehensive unit tests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:36 +00:00
Matt Johnson
be307b000c feat(schema): add USGS quake adapter and CENTRAL_QUAKE stream
Migration 006 seeds:
- config.adapters row for usgs_quake (60s cadence, PNW bbox)
- config.streams row for CENTRAL_QUAKE (7d retention)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:28 +00:00
2fd5bc01c0
Merge pull request #7 from zvx-echo6/feature/1a-6-firms-adapter
feat: FIRMS fire hotspot adapter (Phase 1a-6)
2026-05-16 14:25:55 -06:00
Matt Johnson
cbe9e50383 refactor(supervisor): use adapter registry pattern
- Add _ADAPTER_REGISTRY dict for adapter class lookup
- Unify adapter __init__ signatures (all take config, config_store, cursor_db_path)
- NWSAdapter now accepts config_store param (unused, for signature uniformity)
- Adding new adapters requires only one dict entry, no supervisor changes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:21:34 +00:00
Matt Johnson
95853200b2 fix(firms): use public sweep_old_ids method
Match NWS adapter pattern for supervisor compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:11:12 +00:00
Matt Johnson
22c50d3176 fix(firms): use public is_published/mark_published methods
Match NWS adapter pattern for supervisor compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:03:34 +00:00
Matt Johnson
47359a8144 docs: add FIRMS GUI planning notes
- MAP_KEY management (alias display, rotation)
- Satellite selection (toggle SNPP/NOAA20/NOAA21)
- SNPP end-of-life notice (~Oct 2026)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:46 +00:00
Matt Johnson
5dbaf1dd5c feat(supervisor): wire FIRMS adapter
- Add FIRMSAdapter import and factory case
- Add CENTRAL_FIRE stream to STREAM_SUBJECTS

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:41 +00:00
Matt Johnson
a007418e0a feat(models): add fire event subject routing
Update subject_for_event to handle fire.* category events:
- Fire events: central.<category> (e.g., central.fire.hotspot.viirs_snpp.high)
- Weather events: existing geo-based subject logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:37 +00:00
Matt Johnson
0097163edf feat(adapters): add FIRMS fire hotspot adapter
NASA FIRMS adapter for VIIRS satellite fire detections:
- Polls VIIRS_SNPP_NRT and VIIRS_NOAA20_NRT satellites
- Deduplication via stable ID (satellite📅time:lat:lon)
- Hot-reload support for region, satellites, and API key
- Confidence mapping: l/n/h -> low/nominal/high
- Severity: high=3, nominal=2, low=1

Includes comprehensive unit tests for:
- CSV parsing and event generation
- Deduplication logic
- URL building and config application

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:31 +00:00
Matt Johnson
b42589c69c feat(schema): add FIRMS adapter and CENTRAL_FIRE stream
Migration 005 seeds:
- config.adapters row for firms (300s cadence, PNW bbox)
- config.streams row for CENTRAL_FIRE (7d retention)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:20 +00:00
025ccc6e62
Merge pull request #6 from zvx-echo6/feature/1a-5-stream-retention
Phase 1a-5: Stream retention + adapter config refactor + unified region
2026-05-16 13:10:11 -06:00
Matt Johnson
f7a55c3cc4 docs: add Phase 1a-5 verification report
Documents test results for:
- Gate 5: max_bytes self-loop prevention (PASS)
- Gate 6: bbox hot-reload (PASS)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:08:00 +00:00
Matt Johnson
a157f39fe0 fix(nws): replace centroid filter with polygon intersection
- Add shapely dependency for geometry intersection
- Replace _point_in_region with _geometry_intersects_region
- Uses Shapely shape() and box() for proper GeoJSON handling
- Avoids false negatives on large alert polygons

Also adds antimeridian-crossing bbox rejection to RegionConfig validator.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:05:05 +00:00
Matt Johnson
f9426caa27 feat: add stream management infrastructure
- config_store: add stream CRUD methods
- stream_manager: ensure_stream, apply_retention, recompute_max_bytes
- Auto-clamp max_bytes to [1GB floor, 30% ceiling]
- Parse server max_file_store from nats-server.conf

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:50:12 +00:00
Matt Johnson
da8942a457 schema: migrate NWS settings from states to region bbox
- Remove states array from NWS settings
- Add region bbox covering ID/OR/WA/MT/WY/UT/NV
- Bbox: north=49.5, south=31.0, east=-102.0, west=-124.5

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:50:05 +00:00
Matt Johnson
71a43d3c98 schema: add config.streams table with column-filtered notify
- config.streams table for JetStream retention config
- Column-filtered NOTIFY: only fires on max_age_s changes
- Prevents self-loop when supervisor updates max_bytes
- Seeds CENTRAL_WX (7d/10GB) and CENTRAL_META (1d/100MB)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:49:59 +00:00