Commit graph

35 commits

Author SHA1 Message Date
Matt Johnson
aacf06499b feat(adapters): add USGS earthquake adapter
USGS Earthquake Hazards Program adapter:
- Polls GeoJSON feed (all_hour default, configurable)
- Magnitude tier classification (minor/light/moderate/strong/major/great)
- Deduplication via USGS stable event ID
- Region filter via shapely point-in-bbox
- Skips events with null magnitude (quarry blasts, etc.)

Includes comprehensive unit tests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:51:36 +00:00
Matt Johnson
cbe9e50383 refactor(supervisor): use adapter registry pattern
- Add _ADAPTER_REGISTRY dict for adapter class lookup
- Unify adapter __init__ signatures (all take config, config_store, cursor_db_path)
- NWSAdapter now accepts config_store param (unused, for signature uniformity)
- Adding new adapters requires only one dict entry, no supervisor changes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:21:34 +00:00
Matt Johnson
95853200b2 fix(firms): use public sweep_old_ids method
Match NWS adapter pattern for supervisor compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:11:12 +00:00
Matt Johnson
22c50d3176 fix(firms): use public is_published/mark_published methods
Match NWS adapter pattern for supervisor compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 20:03:34 +00:00
Matt Johnson
5dbaf1dd5c feat(supervisor): wire FIRMS adapter
- Add FIRMSAdapter import and factory case
- Add CENTRAL_FIRE stream to STREAM_SUBJECTS

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:41 +00:00
Matt Johnson
a007418e0a feat(models): add fire event subject routing
Update subject_for_event to handle fire.* category events:
- Fire events: central.<category> (e.g., central.fire.hotspot.viirs_snpp.high)
- Weather events: existing geo-based subject logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:37 +00:00
Matt Johnson
0097163edf feat(adapters): add FIRMS fire hotspot adapter
NASA FIRMS adapter for VIIRS satellite fire detections:
- Polls VIIRS_SNPP_NRT and VIIRS_NOAA20_NRT satellites
- Deduplication via stable ID (satellite📅time:lat:lon)
- Hot-reload support for region, satellites, and API key
- Confidence mapping: l/n/h -> low/nominal/high
- Severity: high=3, nominal=2, low=1

Includes comprehensive unit tests for:
- CSV parsing and event generation
- Deduplication logic
- URL building and config application

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:58:31 +00:00
Matt Johnson
a157f39fe0 fix(nws): replace centroid filter with polygon intersection
- Add shapely dependency for geometry intersection
- Replace _point_in_region with _geometry_intersects_region
- Uses Shapely shape() and box() for proper GeoJSON handling
- Avoids false negatives on large alert polygons

Also adds antimeridian-crossing bbox rejection to RegionConfig validator.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 19:05:05 +00:00
Matt Johnson
f9426caa27 feat: add stream management infrastructure
- config_store: add stream CRUD methods
- stream_manager: ensure_stream, apply_retention, recompute_max_bytes
- Auto-clamp max_bytes to [1GB floor, 30% ceiling]
- Parse server max_file_store from nats-server.conf

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:50:12 +00:00
Matt Johnson
ab7126ec8d refactor(supervisor): remove adapter-specific branches, add stream wiring
- Replace if name == nws with generic apply_config call
- Add _create_adapter factory method
- Add stream management: ensure_stream, retention recompute loop
- Handle streams config changes via NOTIFY

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:49:53 +00:00
Matt Johnson
dfcc0c3a5c refactor(nws): migrate from states to bbox region filtering
- Add RegionConfig pydantic model with validators
- NWSAdapter now uses bbox for client-side alert filtering
- Implement apply_config for hot-reload of region changes
- Remove states-based filtering logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:49:46 +00:00
Matt Johnson
1ea56b67fd refactor(adapter): add abstract apply_config method
SourceAdapter now requires apply_config() for hot-reload support.
Each adapter implements its own config extraction from settings.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 18:49:40 +00:00
Matt Johnson
0eba319071 refactor(supervisor): remove adapter.cadence_s update
The NWSAdapter no longer has a cadence_s attribute since the
internal limiter was removed. The supervisor's rate limiting
via state.config.cadence_s and last_completed_poll is sufficient.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 17:18:55 +00:00
Matt Johnson
9d4ba97537 refactor(nws): remove internal AsyncLimiter rate limiting
The NWSAdapter had an internal AsyncLimiter that duplicated the
supervisor's rate-limit guarantee. When cadence changed, only
state.adapter.cadence_s was updated, not the internal limiter,
causing the cadence-decrease bug.

Since the supervisor already enforces rate limiting via
last_completed_poll + cadence_s scheduling, the adapter-level
limiter was redundant and caused the 30-second blocking observed
in diagnostic logs.

Removes:
- aiolimiter import
- self.cadence_s attribute (unused elsewhere)
- self._limiter creation
- async with self._limiter context in _fetch_alerts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 17:17:11 +00:00
Matt Johnson
4215744a30 fix: move cancel_event signal outside lock for immediate delivery
The cancel_event.set() call was inside the async lock context in
_on_config_change, causing delayed signal delivery to the sleeping
loop. This manifested as cadence decreases not applying without a
restart - the loop would sleep its full original timeout before
seeing the new cadence.

Fix: _reschedule_adapter now returns the AdapterState to signal,
and _on_config_change signals AFTER releasing the lock. This ensures
immediate event delivery per asyncio semantics.

The lock protects state consistency during config fetches and updates.
The cancel_event is a one-way notification that does not need lock
protection - it simply wakes the sleeping coroutine.

See docs/BUG-CADENCE-DECREASE.md for full investigation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 05:59:45 +00:00
Ubuntu
c6f4f3b081 refactor: supervisor always uses DbConfigSource
Remove conditional config source loading, simplify async_main.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 03:42:38 +00:00
Ubuntu
a1e81bae8a refactor: remove config_source flag from bootstrap settings
Database config is now the only option, no need for feature flag.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 03:42:33 +00:00
Ubuntu
4376588baf refactor: remove TomlConfigSource, keep only DbConfigSource
TOML config is now retired. Database is the sole configuration source.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 03:42:28 +00:00
Matt Johnson
41439c52b3 refactor: rename DEFAULT_CLOUDEVENTS_CONFIG to CLOUDEVENTS_CONFIG
These are protocol-level constants, not defaults that get overridden.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 03:08:09 +00:00
Matt Johnson
73beb90b25 chore: remove unused ABC import from config_source.py
ConfigSource uses Protocol, not ABC. Removed unused import.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 02:38:15 +00:00
Matt Johnson
b16151abf1 refactor: move CloudEvents config to code constants
CloudEvents envelope format is protocol-level (not operator config).
When using DB config source without TOML, wrap_event() now uses
DEFAULT_CLOUDEVENTS_CONFIG from cloudevents_constants.py.

Changes:
- Add cloudevents_constants.py with DEFAULT_CLOUDEVENTS_CONFIG
- Update wrap_event() to accept Config, CloudEventsConfig, or None
- Simplify supervisor: always use wrap_event (has defaults)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 02:38:11 +00:00
Matt Johnson
c39e3174b8 fix: preserve last_completed_poll across adapter disable/enable
Previously, _stop_adapter() used pop() to remove adapter state,
which lost last_completed_poll. On re-enable, a fresh state was
created, causing immediate poll and violating rate-limit guarantee.

Changes:
- Add is_running property to AdapterState
- _stop_adapter: preserve state, just cancel task
- _start_adapter: reuse existing stopped state if present
- Add _remove_adapter for full cleanup when adapter is deleted
- _on_config_change: call _remove_adapter for deleted adapters

Integration tests verify:
- Test A: gap > cadence -> immediate poll (correct)
- Test B: gap < cadence -> wait until last_poll + cadence (was broken)
- Test C: delete + re-add -> fresh state (correct)

Tests-fail-before-fix verified: Test A/B failed on unfixed code
with "State was removed on stop!", pass with fix.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 02:37:57 +00:00
Ubuntu
daa7852cc0 refactor(archive): use bootstrap_config for connection strings
Archive now reads NATS URL and Postgres DSN from bootstrap_config
instead of TOML file. This is sufficient for archive since it only
needs connection strings, not adapter configuration.

No ConfigSource wiring needed - archive just consumes from JetStream.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 01:55:39 +00:00
Ubuntu
29fa49c5c2 feat(supervisor): add hot-reload support with rate-limit guarantee
Refactors supervisor to use ConfigSource abstraction:
- AdapterState tracks last_completed_poll for rate limiting
- Hot-reload via NOTIFY: cadence/enable/disable changes take effect
- Rate-limit guarantee: next poll at last_poll + new_cadence, not now
- Logs config source at startup (toml or db)
- Logs reschedule decisions with next poll timestamp

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 01:55:33 +00:00
Ubuntu
ee593abc54 feat(config): add ConfigSource abstraction and CENTRAL_CONFIG_SOURCE flag
- ConfigSource protocol with list_enabled_adapters, get_adapter, watch_for_changes
- TomlConfigSource: loads from TOML file, watch_for_changes is no-op
- DbConfigSource: wraps ConfigStore with LISTEN/NOTIFY support
- CENTRAL_CONFIG_SOURCE bootstrap flag: toml (default) or db
- CENTRAL_CONFIG_TOML_PATH for specifying TOML file location

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 01:55:25 +00:00
Ubuntu
b183a621bb feat(config_store): add listener reconnect with exponential backoff
Listener now automatically reconnects on connection loss with
exponential backoff (1s-30s). Cancellation propagates cleanly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-16 01:36:35 +00:00
Ubuntu
3e392cad81 feat(config): add CLI smoke command and dependencies
Add central-cli with config-store-check command that:
- Connects via bootstrap config
- Lists adapters from config store
- Verifies crypto round-trip

Updates pyproject.toml with new dependencies:
- pydantic-settings>=2.7.0
- cryptography>=44.0.0

New entry points:
- central-migrate
- central-cli

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 23:08:03 +00:00
Ubuntu
8c5349c880 feat(config): add database-backed config store
Add ConfigStore class providing async access to config schema:

- get_adapter/list_adapters/upsert_adapter for adapter config
- pause_adapter/unpause_adapter for runtime control
- set_api_key/get_api_key with encryption via crypto.py
- listen_for_changes using Postgres LISTEN/NOTIFY

Includes Pydantic models (AdapterConfig, ApiKeyInfo) and tests
using real Postgres test database.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 23:07:56 +00:00
Ubuntu
a9b7dcab62 feat(config): add migration framework and config schema
Add simple SQL migration runner tracking applied migrations in
schema_migrations table. First migration creates:

- config schema
- config.adapters table (name, enabled, cadence_s, settings JSONB)
- config.api_keys table (alias, encrypted_value BYTEA)
- NOTIFY triggers for real-time config change detection
- Seeds NWS adapter row from current TOML config

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 23:07:49 +00:00
Ubuntu
fab452aa02 feat(config): add AES-256-GCM crypto primitives
Add encrypt/decrypt functions using AES-256-GCM for secret storage.
Master key loaded from file path specified in bootstrap config.

Features:
- 32-byte key from base64-encoded file
- 12-byte random nonce per encryption
- AEAD authentication (detects tampering)
- Key caching with clear_key_cache() for rotation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 23:07:41 +00:00
Ubuntu
e126569a4d feat(config): add bootstrap config from environment
Add pydantic-settings based Settings class for loading configuration
from environment variables or .env file. Provides early-stage config
before database-backed config store is available.

Includes:
- CENTRAL_DB_DSN, CENTRAL_NATS_URL, CENTRAL_MASTER_KEY_PATH, CENTRAL_LOG_LEVEL
- Cached loader with get_settings()
- Tests for env vars, .env file, validation, caching

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 23:07:33 +00:00
Matt Johnson
4a7f1a76c7 refactor: rename CloudEvents extension attributes hub* → central*
Rename extension attributes for consistency with project naming:
- hubschemaversion → centralschemaversion
- hubcategory → centralcategory
- hubseverity → centralseverity

Non-breaking change - no consumers depend on these names yet.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 21:47:50 +00:00
Matt Johnson
31be17430d runtime: NWS adapter, supervisor, archive consumer, systemd units
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 21:29:08 +00:00
Matt Johnson
714971fe99 foundation: models, adapter ABC, config, CE wire, schema
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-15 21:08:56 +00:00
Matt Johnson
36ebbcb250 scaffold: initial repository structure 2026-05-15 19:16:24 +00:00