mirror of
https://github.com/zvx-echo6/central.git
synced 2026-06-10 20:04:43 +02:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
621148ac46 |
v0.11.0: new celestrak_tle adapter + CENTRAL_SAT satellite-tracking stream (#100) | |||
|
e92b51c518 |
v0.10.10: new avalanche_org adapter — backcountry avalanche advisories (#98)
meshai-requested adapter for avalanche.org's per-center map layers (SNFAC Sawtooth + PAC Payette by default; operator-extensible to any avalanche.org forecast center). Pure passthrough + severity gate, no cross-source fusion, fits Central's adapter pattern cleanly. Adapter surface: - Polls https://api.avalanche.org/v2/public/products/map-layer/{center_id} per configured center; default cadence 1800s (30 min). - Severity gate: only danger_level >= 3 publishes. danger_level 0/1/2 (None/Low/Moderate), -1 ('no rating'), and off_season=true all omitted at adapter level. Idaho summer = all 4 SNFAC + 2 PAC zones yield 0 events; that's correct behavior, verified by the negative-case test against the frozen 2026-06-08 SNFAC fixture. - Severity mapping (corrected from meshai's inverted spec): danger_level 3 (Considerable) → severity 2, 4 (High) → 3, 5 (Extreme) → 4. Matches Central's 4-most-severe convention (nws.SEVERITY_MAP). - Subject: central.avy.advisory.us.{state_lower} — one per state; v0.10.8's category-discriminated Nats-Msg-Id keeps multiple zones in the same state from colliding in JetStream dedup. - Stream: CENTRAL_AVY (central.avy.>); 7-day / 1 GiB retention defaults. - Event.data fields per meshai spec: center_id, zone_name, danger_level, danger_name, travel_advice (truncated to 200 chars), state, valid_date, end_date, off_season=false, latitude/longitude (polygon centroid via shapely), plus geo.geometry passes through as the upstream Polygon. Tests (38 in test_avalanche_org.py): - Pure helpers: _slug (8 cases), _parse_iso (6 cases), _centroid (2 cases). - Severity gate: 3 publish cases (danger 3/4/5 → severity 2/3/4), 4 omit cases (danger -1/0/1/2), off_season=true omit, missing state omit, unparseable geom omit, travel_advice truncation, subject derivation. - Real-fixture negative case: 4-zone SNFAC fixture all omitted off-season. - Real-fixture positive case: same fixture with synthetic winter overrides publishes all 4 with valid centroids on actual Idaho polygons. - End-to-end poll() with mixed severities and the new wiring (streams registry + supervisor family map). - Defensive: empty center_ids list yields nothing without crashing. Wiring + plumbing: - src/central/streams.py: StreamEntry('CENTRAL_AVY', 'central.avy.>') - src/central/supervisor.py: STREAM_CATEGORY_DOMAINS['CENTRAL_AVY']=('avy',) - sql/migrations/035: seed config.streams row (mirror of 019/CENTRAL_SPACE, idempotent ON CONFLICT DO NOTHING). Note: migrations don't auto-run on supervisor restart -- see deferred ops list (schema_migrations cleanup blocks central-migrate from running anything cleanly). - src/central/gui/templates/_event_rows/avalanche_org.html (8 lines) - src/central/gui/templates/_event_summaries/avalanche_org.html (2 lines) Both required by the existing per-adapter template consistency tests. Doc updates (required by existing doc-vs-registry tests): - docs/PRODUCER-INTEGRATION.md §6.1: added 'avy' to top-level-domain list. - docs/PRODUCER-INTEGRATION.md §8: added StreamEntry('CENTRAL_AVY',...) line to the verbatim snippet. - docs/CONSUMER-INTEGRATION.md §3 stream layout table: added CENTRAL_AVY row. - docs/CONSUMER-INTEGRATION.md §6: new '### avalanche_org' subsection with source, subject convention, dedup key, severity gate, Event.data field table, and off-season behavior note. - tests/test_events_feed_frontend.py: added avalanche_org to _SAMPLE_INNER and _EXPECTED_SUBJECT (the events-JSON subject-derivation coverage tests). Budget note: this PR is well over the ~400-line target -- the new-adapter surface picked up downstream consistency tests (doc validators + frontend sample coverage + template partials) I didn't anticipate at probe time. Most of the overrun is the SNFAC fixture (1,135 lines pretty-printed JSON, non-code) and the adapter + tests pair. Stripping the fixture and the required doc/template edits would leave ~620 lines of code; the fixture itself is a frozen snapshot, not a maintenance burden. Full sweep: 1072 passed, 0 failures (+41 from this PR), ruff clean on all new files. One PRE-EXISTING ruff violation in supervisor.py (unused poll_start variable at line 388) surfaces when we touch supervisor.py; confirmed not introduced by this PR via git stash check. Deploy plan (NEW STREAM — archive restart required per [[feedback_new_stream_needs_archive_restart]]): 1. Squash-merge -> tag v0.10.10 -> push. 2. On central: pull main -> systemctl restart central-supervisor -> ALSO systemctl restart central-archive (new event-bearing stream; archive enumerates consumers at startup and doesn't hot-reload). 3. Migration 035 deferred to morning per the schema_migrations cleanup task -- the stream creation itself doesn't depend on it (supervisor creates JetStream streams from the STREAMS registry at startup; the config.streams row is for operator-tunable retention only). 4. Verify: nats stream info CENTRAL_AVY (created), poll log shows yielded=0 / omitted=N (off-season), no positive publishes during summer (correct). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
Matt Johnson
|
02bc692bda |
feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6)
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).
supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.
- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
cams). The 100-row page cap also means v0.9.2 state_511_atis silently
truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
-- always shows today's cameras, no per-poll flooding, no retention
coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).
Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Johnson
|
b8033444ec |
feat(tomtom_flow): TomTom Orbis vector flow-tile telemetry adapter + CENTRAL_TRAFFIC_FLOW (v0.9.3)
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.
Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.
- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
transitive that uv re-lock would otherwise prune).
Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Johnson
|
9cd2183cc3 |
fix(wzdx): drop 'unknown' direction from subject + extract dedup mixin (v0.9.1)
Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui restart (adapter base change + template change). No migration, no new stream. (a) Work-zone subject + detail no longer leak vehicle direction "unknown" (common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both wzdx partials. Was "Work zone on MORELAND ST unknown". (b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days class attr parameterizes the retention window (NWIS=30, default=14). Inline copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a future cleanup. Net dedup code down ~52 lines. Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Johnson
|
7eab5fc1b1 |
feat(wzdx): WZDx adapter + CENTRAL_TRAFFIC family bootstrap (v0.9.0)
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Johnson
|
8d193d3266 |
feat(telemetry-separation): dedicated /telemetry tab split from /events by adapter data_class (v0.7.4)
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only (supervisor untouched -- data_class is read only by central-gui per request). - SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"], default "event"). NWIS opts in as "telemetry" (continuous high-volume water gauges); every other adapter stays "event". The /events vs /telemetry split is thus registry-derived from class attrs -- no hardcoded adapter-name lists. - routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists to a class (colors stay keyed to the FULL registry, so an adapter keeps one color across tabs). `_fetch_events` accepts `class_adapters` and adds an `adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class, base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`, `/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers. - Templates parameterized with a `base_path` context var (form action, hx-get, hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows` paginator macro takes `base`. Same templates serve both tabs; nav gains a Telemetry link. - /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the subject + pagination contract is intact (TestEventsJsonSubject still passes). Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1 telemetry, class-scoped filter options, color stability, and the `adapter = ANY(...)` SQL shape incl. the no-class events.json path). Updates the events frontend tests for the base_path-parameterized templates. Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Johnson
|
47e7b4f267 |
feat(gui-bugs): fix eonet dashboard exception + out-of-range map bbox
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed live; production code, central-gui + central-supervisor restart required. Bug 1 (eonet exception leaking to /dashboard): The supervisor calls adapter.bump_last_seen on every dedup hit, but only 4 of 12 adapters defined it and the base class did not. Adapters that re-emit already-published events (eonet re-lists open natural events each poll) raised AttributeError; the supervisor published it as the adapter's status error, which /dashboard rendered as literal text in the Last Poll cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on getattr(self, "_db", None)); remove the 4 now-redundant identical overrides. Fixes all 8 affected adapters, not just eonet. Documents the method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard). Bug 2 (map bbox out of valid range): applyViewportFilter serialized raw Leaflet getEast()/getWest(), which exceed [-180,180] when panned past the dateline at low zoom (e.g. region_east=411.3281, region_west=-608.2031), and _parse_events_params passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox entirely. Fix (backend, defense in depth): _parse_events_params treats an out-of-range or inverted envelope as "no bbox" rather than erroring or querying a bogus envelope. Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the planning walkthrough were investigated and refuted (FIRMS rows are distinct fire pixels, not satellite dupes -- dropping satellite collapses 0 rows; the expand button is present and functional on main), so they are not part of this PR. Tests: registry-derived guard that every adapter resolves bump_last_seen + base-method behavior test; 3 bbox-guard unit tests on _parse_events_params. Full suite: 634 passed, 1 skipped (central and unprivileged zvx). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Johnson
|
98b050b2af |
feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real backends + documents the reframed design principle in-tree; L is the events tab + map fix, then tag). Backends (all satisfy GeocoderBackend; never raise, all-null on any failure): - NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone + landclass + elevation in one call). Near-passthrough: response already matches the canonical 9-field shape. Best-effort warmup ping (Boise) on construction when a loop is running; config `headers` slot for a future Authorization: Bearer (config-only, no code change). Default base_url http://192.168.1.130:8440. - PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only). Maps features[0].properties; postal_code <- postcode; timezone/landclass/ elevation_m null (Navi-composed-endpoint extras). - NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only). Configurable rate limit (default 1/sec; 0 disables for self-hosted) + required User-Agent. Maps the address block; landclass/elevation_m/timezone null. Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig backend_class name). Docs — design pivot now in-tree: - PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts. Central is the consumer's only data plane (consumers can't do follow-up lookups), so enrich deliberately and centrally, namespaced under _enriched, failing to null. "No enrichment" is gone. - PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it OUTSIDE the framework (inline in poll(), bypassing cache + _enriched namespacing + the never-raise safety net). - PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher / GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL + cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance, per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the landclass antimeridian known wrinkle. - CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9 fields), per-region coverage (US-full, non-US timezone+elevation), and the antimeridian landclass caveat. Tests: - test_navi/photon/nominatim_backend.py — happy-path field mapping, null handling, extra-key drop, network/timeout/non-200/malformed -> all-null (never raises), Nominatim rate-limit (disabled + spacing) + User-Agent. Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the 192.168.1.130 endpoint isn't reachable from CT104's segment). - test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists, §13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS (derived from code, not hardcoded). Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend + 4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY clean. All three backends import + resolve via the registry. Flagged for later (NOT done here): adapters besides FIRMS that should declare enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L scope alongside the events tab. See PR description. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Johnson
|
6afe80ded3 |
docs(2-I): producer integration spec — docs/PRODUCER-INTEGRATION.md
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
