matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-06-10 20:04:43 +02:00
Code Issues Projects Releases Packages Wiki Activity
256 commits 8 branches 67 tags 4.7 MiB
Commit graph

14 commits

Author SHA1 Message Date
malice
c49f2db95f
 		v0.12.0: sat_positions adapter (live global satellite positions) + sat_common refactor 
## Architectural framing

The v0.11.1 `satpass_predict` adapter is **observer-anchored**: "when does satellite X pass over fixed observer Y, and what's the elevation/azimuth at that observer's site?" It answers a fixed-QTH question and emits one event per (observer, satellite, AOS) tuple.

The new `sat_positions` adapter is the **global** counterpart: "where is satellite X right now?" No observer. One event per tracked NORAD ID per poll, on subject `central.sat.position.<norad_id>`. Consumers (meshAI, GUI map widgets, anything that wants a live world map) subscribe to `central.sat.position.>` and plot.

They complement each other; neither replaces the other.

Direct quote from Matt's use-case: *"location of the sats... map of where the sats are then we have meshai or whatever other service calling central's data grab it and do whatever work it needed."* This adapter is that.

## sat_common extraction rationale

The four pure SGP4 / coordinate helpers (`EARTH_RADIUS_KM`, `gmst_rad`, `eci_to_ecef`, `subsatellite_point`) were private symbols inside `satpass_predict.py`. `sat_positions` needs the same three helpers. Three options were considered:

1. **Cross-import** from `satpass_predict.py` — creates an adapter-to-adapter dependency, ugly.
2. **Extract to `sat_common.py`** — matches the existing `wfigs_common.py` / `swpc_common.py` precedent. Both adapters become siblings of a shared helper module. ✓ chosen.
3. **Duplicate** — math drift over time.

Symbol names dropped their leading underscore on extraction (public-API convention matching `swpc_common.parse_swpc_timestamp` / `wfigs_common.severity_from_acres`). Existing internal call sites in `satpass_predict.py` were updated via mechanical `replace_all`. Observer-specific helpers (`_observer_ecef`, `_topocentric_az_el`, `_visibility_footprint`, `_severity_from_elev`, `_build_pass_geometry`, `_next_passes`) stay in `satpass_predict.py` per YAGNI — they're not used by `sat_positions` today.

Existing `tests/test_satpass_predict.py` was updated mechanically to import the helpers from `sat_common` via aliases (preserves the underscore-prefixed local names in the tests so the rest of the test body needs no change). All 44 satpass_predict tests pass unchanged.

## CENTRAL_SAT stream cap bump

`config.streams.max_bytes` for `CENTRAL_SAT` goes from **1 GiB → 5 GiB** in migration 039. Sizing math:

- celestrak_tle: ~190 sats × 1 envelope/day = ~190 events/day = ~1.4k events/week. Fit in 1 GiB easily.
- sat_positions: ~190 sats × 1440 ticks/day (60s cadence) = **~273.6k events/day = ~1.9M events/week**. At ~1 KB per envelope including the CloudEvents wrapper, that's **~1.9 GiB/week**.
- Plus existing TLE + pass envelopes already on the stream → ~3 GiB headroom needed.
- 5 GiB = 5368709120 bytes = operator-tunable margin without over-provisioning.

`STREAM_CATEGORY_DOMAINS["CENTRAL_SAT"]` extends from `("tle", "pass")` to `("tle", "pass", "position")` so the supervisor's retention sweep covers position events too.

## Subject + dedup

| Field | Value |
|---|---|
| Subject | `central.sat.position.<norad_id>` — one subject per satellite, globally |
| Dedup id | `<norad_id>:<position_iso>` where `position_iso` is the propagation timestamp truncated to whole seconds (defensive collapse if cadence is ever tightened) |
| Severity | 1 (informational telemetry, no alerting) |
| data_class | `telemetry` — surfaces on `/telemetry`, not `/events` |
| Cadence | 60s default; operator-tunable via standard `cadence_s` field |

## Settings shape

```json
{"track_only_norad_ids": [], "max_tle_age_days": 14}
```

- Empty `track_only_norad_ids` = track every NORAD ID with a fresh TLE in the events table (derive-from-celestrak_tle, default behavior).
- Non-empty list pins to those NORAD IDs only (operator override — "I only care about the ISS and these 12 Starlink sats").
- `max_tle_age_days` bounds TLE freshness; LEO drag means TLEs go stale in days, GEO is good for months. Parameterized into the SQL query as a timedelta interval so operator-tightened windows (e.g. 3d) apply without code change.

## Event.data fields

`norad_id`, `satellite_name`, `lon_deg`, `lat_deg`, `alt_km`, `velocity_kmps`, `heading_deg`, `tle_epoch`.

- `lon_deg`/`lat_deg`/`alt_km`: sub-satellite point via SGP4 → ECI → ECEF rotation → spherical-earth lon/lat/alt.
- `velocity_kmps`: magnitude of the SGP4 ECI velocity vector. ECI vs ECEF difference is ~6% for LEO (earth rotation 0.46 km/s vs 7.7 km/s orbital speed); fine for consumer "the sat is moving at X km/s" text.
- `heading_deg`: great-circle initial bearing from the sub-sat point at `t` to the sub-sat point at `t+1s` (finite-difference; simpler than rotating velocity through GMST + the earth-rotation cross term).

## Diff size — flag for review

**+894 / -63 = +831 net** across 14 files. Spec budget was ≤700 lines. **Over by ~131 net** (or ~194 gross).

Breakdown:
- `sat_positions.py`: 286 lines (under the ≤350 adapter line cap ✓)
- `sat_common.py`: 65 lines (the extraction)
- Migration 039: 58 lines (heavy on inline comments documenting the size math; could trim ~25 lines if you want)
- satpass_predict.py: net -1 line (refactor; lost 4 helper defs and one constant comment, gained 5-line import block)
- Templates: 14 lines (event_rows + event_summaries partials)
- Wiring: 4 lines (supervisor + ADAPTER_GROUPS)
- Docs (CONSUMER-INTEGRATION.md): 40 lines (required by `tests/test_consumer_doc.py::test_every_adapter_has_a_subsection`)
- **Tests: 426 lines.** This is the bulk of the overage.

The tests are all spec-mandated (sub-sat math, velocity range, heading range, build_event, subject_for, empty-TLE, track_only gate, stale-TLE skip, sat_common helpers, regression-guard on the moved helpers via test_satpass_predict.py preservation). I could shrink `test_sat_positions.py` by consolidating the 11 spec-mandated tests into fewer parameterized cases, but each test pins one behavior the spec called out by name. Flagging for your call: keep as-is, or do you want a tighter parameterized version?

## Test plan

- [x] `pytest tests/test_sat_common.py tests/test_sat_positions.py` — **28 new tests, all pass**.
- [x] `pytest tests/test_satpass_predict.py` — **44/44 pass** (regression guard: existing tests work after the sat_common extraction).
- [x] `pytest tests/test_events_feed_frontend.py` — **119/119 pass** (JSON-feed coverage extended to include sat_positions sample event + expected subject string).
- [x] `pytest tests/test_telemetry_separation.py` — **9/9 pass** (`_TELEMETRY` pin extended to include `sat_positions`).
- [x] `pytest tests/test_consumer_doc.py` — **6/6 pass** (CONSUMER-INTEGRATION.md `### sat_positions` subsection added).
- [x] `pytest tests/test_producer_doc.py` — **10/10 pass** (no PRODUCER-INTEGRATION update needed; CENTRAL_SAT stream is pre-existing).
- [x] Full sweep `pytest tests/` (excluding postgres-dep files): **1209 passed, 1 skipped, 0 failures**.
- [x] Ruff: clean on all new code. 3 pre-existing F841 unused-variable warnings (supervisor.py:390 `poll_start`, test_events_feed_frontend.py:425 / :466 `result`) confirmed via `git blame` to be from commits May 2026 — not introduced.

## Deploy plan

1. Squash-merge → tag v0.12.0 at merge SHA → push tag.
2. `ssh central`, `git pull` on `/opt/central`. **No `uv sync`** (no new dep).
3. **`central-migrate`** to apply migration 039 (seeds `config.adapters` row + bumps `config.streams.max_bytes` for CENTRAL_SAT).
4. `sudo systemctl restart central-supervisor` (picks up STREAM_CATEGORY_DOMAINS extension + new adapter discovery).
5. `sudo systemctl restart central-gui` (picks up new partials + ADAPTER_GROUPS change).
6. **No** `central-archive` restart (CENTRAL_SAT stream already exists; no new stream).
7. Verify: `nats stream info CENTRAL_SAT` shows max_bytes=5368709120; supervisor journal shows sat_positions discovered.
8. Smoke-test: enable celestrak_tle first if not already, wait for one poll, then enable sat_positions via GUI. Within 60s expect one `central.sat.position.<norad_id>` event per tracked sat on the stream.

## Halt acknowledgment

Per spec acceptance bar #6: **squash-merge NOT authorized**. Branch + PR open. Halting for line-by-line review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-09 15:23:32 -06:00
malice
86e8b6b56a
 		v0.11.1: satpass_predict adapter (server-side pass alerts for fixed observers) (#101) 2026-06-09 01:16:43 -06:00
malice
621148ac46
 		v0.11.0: new celestrak_tle adapter + CENTRAL_SAT satellite-tracking stream (#100) 2026-06-09 00:54:19 -06:00
malice
e92b51c518
 		v0.10.10: new avalanche_org adapter — backcountry avalanche advisories (#98) 
meshai-requested adapter for avalanche.org's per-center map layers (SNFAC
Sawtooth + PAC Payette by default; operator-extensible to any avalanche.org
forecast center). Pure passthrough + severity gate, no cross-source fusion,
fits Central's adapter pattern cleanly.

Adapter surface:
- Polls https://api.avalanche.org/v2/public/products/map-layer/{center_id}
  per configured center; default cadence 1800s (30 min).
- Severity gate: only danger_level >= 3 publishes. danger_level 0/1/2
  (None/Low/Moderate), -1 ('no rating'), and off_season=true all omitted at
  adapter level. Idaho summer = all 4 SNFAC + 2 PAC zones yield 0 events;
  that's correct behavior, verified by the negative-case test against the
  frozen 2026-06-08 SNFAC fixture.
- Severity mapping (corrected from meshai's inverted spec): danger_level
  3 (Considerable) → severity 2, 4 (High) → 3, 5 (Extreme) → 4. Matches
  Central's 4-most-severe convention (nws.SEVERITY_MAP).
- Subject: central.avy.advisory.us.{state_lower} — one per state; v0.10.8's
  category-discriminated Nats-Msg-Id keeps multiple zones in the same state
  from colliding in JetStream dedup.
- Stream: CENTRAL_AVY (central.avy.>); 7-day / 1 GiB retention defaults.
- Event.data fields per meshai spec: center_id, zone_name, danger_level,
  danger_name, travel_advice (truncated to 200 chars), state, valid_date,
  end_date, off_season=false, latitude/longitude (polygon centroid via
  shapely), plus geo.geometry passes through as the upstream Polygon.

Tests (38 in test_avalanche_org.py):
- Pure helpers: _slug (8 cases), _parse_iso (6 cases), _centroid (2 cases).
- Severity gate: 3 publish cases (danger 3/4/5 → severity 2/3/4),
  4 omit cases (danger -1/0/1/2), off_season=true omit, missing state omit,
  unparseable geom omit, travel_advice truncation, subject derivation.
- Real-fixture negative case: 4-zone SNFAC fixture all omitted off-season.
- Real-fixture positive case: same fixture with synthetic winter overrides
  publishes all 4 with valid centroids on actual Idaho polygons.
- End-to-end poll() with mixed severities and the new wiring (streams
  registry + supervisor family map).
- Defensive: empty center_ids list yields nothing without crashing.

Wiring + plumbing:
- src/central/streams.py: StreamEntry('CENTRAL_AVY', 'central.avy.>')
- src/central/supervisor.py: STREAM_CATEGORY_DOMAINS['CENTRAL_AVY']=('avy',)
- sql/migrations/035: seed config.streams row (mirror of 019/CENTRAL_SPACE,
  idempotent ON CONFLICT DO NOTHING). Note: migrations don't auto-run on
  supervisor restart -- see deferred ops list (schema_migrations cleanup
  blocks central-migrate from running anything cleanly).
- src/central/gui/templates/_event_rows/avalanche_org.html (8 lines)
- src/central/gui/templates/_event_summaries/avalanche_org.html (2 lines)
  Both required by the existing per-adapter template consistency tests.

Doc updates (required by existing doc-vs-registry tests):
- docs/PRODUCER-INTEGRATION.md §6.1: added 'avy' to top-level-domain list.
- docs/PRODUCER-INTEGRATION.md §8: added StreamEntry('CENTRAL_AVY',...) line
  to the verbatim snippet.
- docs/CONSUMER-INTEGRATION.md §3 stream layout table: added CENTRAL_AVY row.
- docs/CONSUMER-INTEGRATION.md §6: new '### avalanche_org' subsection with
  source, subject convention, dedup key, severity gate, Event.data field
  table, and off-season behavior note.
- tests/test_events_feed_frontend.py: added avalanche_org to _SAMPLE_INNER
  and _EXPECTED_SUBJECT (the events-JSON subject-derivation coverage tests).

Budget note: this PR is well over the ~400-line target -- the new-adapter
surface picked up downstream consistency tests (doc validators + frontend
sample coverage + template partials) I didn't anticipate at probe time.
Most of the overrun is the SNFAC fixture (1,135 lines pretty-printed JSON,
non-code) and the adapter + tests pair. Stripping the fixture and the
required doc/template edits would leave ~620 lines of code; the fixture
itself is a frozen snapshot, not a maintenance burden.

Full sweep: 1072 passed, 0 failures (+41 from this PR), ruff clean on
all new files. One PRE-EXISTING ruff violation in supervisor.py (unused
poll_start variable at line 388) surfaces when we touch supervisor.py;
confirmed not introduced by this PR via git stash check.

Deploy plan (NEW STREAM — archive restart required per
[[feedback_new_stream_needs_archive_restart]]):
1. Squash-merge -> tag v0.10.10 -> push.
2. On central: pull main -> systemctl restart central-supervisor -> ALSO
   systemctl restart central-archive (new event-bearing stream; archive
   enumerates consumers at startup and doesn't hot-reload).
3. Migration 035 deferred to morning per the schema_migrations cleanup
   task -- the stream creation itself doesn't depend on it (supervisor
   creates JetStream streams from the STREAMS registry at startup; the
   config.streams row is for operator-tunable retention only).
4. Verify: nats stream info CENTRAL_AVY (created), poll log shows
   yielded=0 / omitted=N (off-season), no positive publishes during
   summer (correct).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-08 21:57:56 -06:00
malice
0dd83a340e
 		v0.10.3: rip out state_511_atis adapter (superseded by itd_511 v0.10.0; Castle Rock legacy shape EOL per sister-site discovery) (#88) 
Closes #88

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-06 14:44:00 -06:00
malice
1d5548c24c
 		v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85) 
First official-state-DOT-API pattern landing. Two adapters in one PR:

- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
  https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
  to LineString via the polyline lib (bookend LineString or Point fallback);
  ITD Severity string mapped None->1 / Minor->2 / Major->3 with
  IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
  Restrictions / DetourPolyline pass through unmodified. Advisories ship
  as structural pass-through under data.advisory since the upstream
  /alerts endpoint currently returns []; per-record try/except keeps a
  surprise shape from sinking the cycle when ITD posts its first one.

- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
  camera per UTC day; image URL passes straight through to <img src>.
  Region uniform US-ID with data.source_jurisdiction preserving the raw
  upstream Source field for the ~1.2% cross-DOT border-region mirrors
  (UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).

Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.

Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.

Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.

Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-03 22:36:26 -06:00
Matt Johnson
02bc692bda feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6) 
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).

supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.

- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
  100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
  cams). The 100-row page cap also means v0.9.2 state_511_atis silently
  truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
  camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
  -- always shows today's cameras, no per-poll flooding, no retention
  coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
  source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
  shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).

Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 01:33:21 +00:00
Matt Johnson
7fdf47f2f0 feat(tomtom_incidents): per-bbox cadence (v0.9.5.1) 
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.

- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
  falls back to the adapter's default_cadence_s. Existing settings without the
  field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
  fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
  (one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
  due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
  of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).

central-supervisor restart only. No gui/archive restart, no migration, no new dep.

Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 01:01:55 +00:00
Matt Johnson
42d5faa80c feat(tomtom_incidents): TomTom real-time traffic incidents adapter (v0.9.5) 
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.

central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.

- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
  is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
  bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
  rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
  state_511_atis; cross-source overlap is by design = additive coverage, distinct
  dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
  the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
  verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
  key redacted from logs; poll skips cleanly without a key.

Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 00:25:27 +00:00
Matt Johnson
b8033444ec feat(tomtom_flow): TomTom Orbis vector flow-tile telemetry adapter + CENTRAL_TRAFFIC_FLOW (v0.9.3) 
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.

Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.

- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
  with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
  inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
  the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
  adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
  so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
  explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
  transitive that uv re-lock would otherwise prune).

Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 23:25:44 +00:00
Matt Johnson
30e25bf475 feat(state_511_atis): Castle Rock 511 adapter — Idaho incidents/closures/road work (v0.9.2) 
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.

Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.

Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.

Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.

Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.

Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 22:01:11 +00:00
Matt Johnson
7eab5fc1b1 feat(wzdx): WZDx adapter + CENTRAL_TRAFFIC family bootstrap (v0.9.0) 
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.

First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.

Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 20:35:08 +00:00
Matt Johnson
98b050b2af feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment 
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).

Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
  + landclass + elevation in one call). Near-passthrough: response already
  matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
  construction when a loop is running; config `headers` slot for a future
  Authorization: Bearer (config-only, no code change). Default base_url
  http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
  Maps features[0].properties; postal_code <- postcode; timezone/landclass/
  elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
  Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
  required User-Agent. Maps the address block; landclass/elevation_m/timezone
  null.

Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).

Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
  Central is the consumer's only data plane (consumers can't do follow-up
  lookups), so enrich deliberately and centrally, namespaced under _enriched,
  failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
  OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
  namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
  GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
  cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
  per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
  landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
  fields), per-region coverage (US-full, non-US timezone+elevation), and the
  antimeridian landclass caveat.

Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
  handling, extra-key drop, network/timeout/non-200/malformed -> all-null
  (never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
  Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
  192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
  §13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
  (derived from code, not hardcoded).

Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.

Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:10:44 +00:00
malice
d92074b134
 		docs(2-H): consumer integration spec — docs/CONSUMER-INTEGRATION.md (#38) 
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".

Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.

Sections (11 total):
  1. Quick start (5-line nats-py subscribe-and-print)
  2. Connection details (URL / auth / JetStream context / stream discovery)
  3. Stream layout (7 streams, derived from streams.py registry)
  4. Subject namespace registry (Mermaid tree + full pattern table)
  5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
     -- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
  6. Per-adapter reference (12 subsections, locked template)
  7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
  8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
  9. Dedup implementation guide (single-token vs composite-key adapters)
  10. Writing a new consumer checklist
  11. Troubleshooting

Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.

Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.

The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.

New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
  - The "Stream layout" table matches central.streams.STREAMS exactly
    (stream names + subject filters).
  - The (name, subject_filter) pairs match the registry as pairs (catches
    swapped subject filters on existing streams).
  - Every adapter discovered via central.adapter_discovery.discover_adapters()
    has a per-adapter subsection -- and vice versa.
  - The subsection count equals the registry size (catches duplicates).

Verification:
  - 463/463 full suite green (was 458; +5 new consumer_doc tests).
  - Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
    12 JSON blocks (all parse).
  - All 12 adapters covered.
  - No regressions elsewhere.

Acceptance bars (a)-(e) verbatim:
  (a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
  (b) all 12 adapters have per-adapter subsections
  (c) 5/5 consumer-doc tests pass
  (d) 463/463 full suite
  (e) doc length 1878 lines

markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).

Co-authored-by: zvx <zvx@central>
 2026-05-19 14:33:51 -06:00