matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-06-10 11:54:37 +02:00
Code Issues Projects Releases Packages Wiki Activity
257 commits 8 branches 67 tags 4.7 MiB
Commit graph

15 commits

Author SHA1 Message Date
malice
8e388dabd5
 		v0.12.1: n2yo_visualpasses adapter (server-side visible-pass alerts) 
## Architectural placement — complement, not replacement

| | satpass_predict (v0.11.1) | **n2yo_visualpasses (this PR)** |
|---|---|---|
| Computes from | Raw TLEs via local SGP4 | n2yo's pre-computed visualpasses endpoint |
| Magnitude data? | ✗ (SGP4 alone can't compute illumination) | ✓ (server-side sun-geometry) |
| Sun illumination filter? | ✗ | ✓ (n2yo returns sunlit passes only) |
| Cost per (observer, sat) pair | Local compute, free | One n2yo API transaction |
| Failure mode | TLE drift over time | Quota exhaustion, vendor outage |

Both adapters serve the same operator question ("when is sat X overhead at site Y?") but with different data sources. Matt's stated use case is to have **both** running so a vendor outage or quota burn on n2yo doesn't blind him to passes that satpass_predict can still propagate locally.

## Subject collision is intentional

Both adapters emit on `central.sat.pass.us.<state_lower>.<observer_slug>`. A consumer subscribing to e.g. `central.sat.pass.us.id.boise` receives events from **both** adapters. Disambiguation lives in `data.category`:

- `pass.satpass_predict` → local SGP4
- `pass.n2yo_visualpasses` → n2yo API

The v0.10.8 category-discriminated `Nats-Msg-Id` keeps both adapters' JetStream dedup windows separate even when they emit for the same (observer, satellite, AOS) tuple (which they will, by design, for sunlit passes).

This is documented explicitly in the new `### n2yo_visualpasses` subsection of `docs/CONSUMER-INTEGRATION.md` so future consumer integrators don't get surprised.

## Quota math

Default settings ship a curated **6 observers × 6 sats** configuration:

- **Observers** (ID + UT): Filer (primary), Boise, Idaho Falls, Ogden, Salt Lake City, Provo
- **Satellites** (curated for amateur observation): ISS (25544), NOAA-15 (25338), NOAA-18 (28654), NOAA-19 (33591), SO-50 (27607), AO-91 (43017)

At 1h cadence: **6 × 6 × 24 = 864 transactions/day**, comfortably under n2yo's free-tier **1000/day cap** with ~13% headroom for retries or expansion. Operator can extend either dimension if they upgrade quota.

## API key plumbing (tomtom_flow pattern)

Exact mirror of the v0.9.3 tomtom_flow precedent — confirmed during recon to be the established pattern:

```python
requires_api_key = "n2yo"          # class attr, GUI surfaces "requires X" warning
api_key_field = "api_key_alias"    # class attr, GUI renders api_key_select dropdown
# Settings field:
api_key_alias: str = "n2yo"
```

Cached `_api_key` populated via `ConfigStore.get_api_key(alias)` in `startup()` and `apply_config()`. Missing-key path: log INFO, return immediately (zero events, no exception). The live key is scrubbed from log strings via a `_redact()` helper before they hit journald.

**`python -m set_api_key` does not exist** — that was a speculative invocation in the spec. The actual flow is GUI-based: Matt adds the `n2yo` alias via the `/api-keys` page, then enables the adapter via `/adapters/n2yo_visualpasses/edit`.

## Diff size — flag for review

**+848 / −1 = +847 net** across 8 files. Spec budget was ≤600 lines. **Over by ~247** (~41%, similar shape to v0.12.0's overage).

| File | Lines | Notes |
|---|---|---|
| `src/central/adapters/n2yo_visualpasses.py` | 330 | **Under** the ≤350 adapter cap ✓ |
| `tests/test_n2yo_visualpasses.py` | 411 | The bulk of the overage |
| `sql/migrations/040_add_n2yo_visualpasses_adapter.sql` | 45 | Heavy comment block; could trim ~15 lines |
| `docs/CONSUMER-INTEGRATION.md` | 40 | Required by `test_consumer_doc` |
| Partials (event_rows + event_summaries) | 13 | |
| `tests/test_events_feed_frontend.py` | 8 | _SAMPLE_INNER + _EXPECTED_SUBJECT |
| `src/central/gui/routes.py` | 1 | ADAPTER_GROUPS extension |

**Test breakdown** (31 tests in 8 classes):
- 9 severity-bucketing tests — spec called out 4 boundaries (-3.1, -2.9, -0.5, 2.5); the extra 5 pin inclusive-vs-exclusive at -3.0, -1.0, 2.0 boundaries + the ranges in between. Useful regression guards but not strictly spec-required.
- 4 settings-default tests — pin the curated 6×6 set + quota math.
- 4 adapter-class-attrs tests — pin requires_api_key/api_key_field/data_class/default_cadence_s wiring.
- 3 subject_for tests — happy path + UT-state lowercasing + unknown fallback.
- 1 _pass_to_event shape test.
- 7 poll-loop tests — missing key, empty observers, empty norad_ids, happy path, empty passes array, fetch-failure-doesn't-kill-poll, multi-obs-multi-sat 6×6 aggregate.
- 1 HTTP-layer test — 401 → None (the one test that goes through the real session.get mock).
- 2 static-isolation tests — acceptance bar #2 (no hardcoded keys) and #4 (no absolute paths).

I can trim the test file to ~250 lines by dropping the non-strictly-spec-mandated tests (settings defaults, class attrs, extra severity boundaries, extra subject_for variants). **Flag for your call:** keep the comprehensive suite, or trim to spec minimum?

## Test plan

- [x] `pytest tests/test_n2yo_visualpasses.py` — **31/31 pass** (all offline, zero n2yo API hits).
- [x] `pytest tests/test_events_feed_frontend.py` — **122/122 pass** (fixture coverage extended).
- [x] `pytest tests/test_consumer_doc.py` — **6/6 pass** (new `### n2yo_visualpasses` subsection accepted).
- [x] Full sweep `pytest tests/` (excluding postgres-dep files) — **1243 passed, 1 skipped, 0 failures**.
- [x] Ruff: **clean on new files** (`n2yo_visualpasses.py`, `test_n2yo_visualpasses.py`). The pre-existing F841 warnings in routes.py / test_events_feed_frontend.py / supervisor.py are unchanged from v0.11.3-pre.
- [x] **No hardcoded API key in diff** — `git diff main..HEAD | grep -iE 'apiKey=[A-Z0-9]{6,}|api_key.*=.*"[A-Z0-9]{6,}'` returns empty.
- [x] **No absolute paths in test code** — `TestStaticIsolation` enforces this at runtime.

## Deploy plan

1. Squash-merge PR #N → tag v0.12.1 at merge SHA → push tag.
2. `ssh central`, `git pull` on `/opt/central`. **No `uv sync`** (aiohttp already in venv from earlier adapters).
3. **Matt adds the n2yo API key via GUI `/api-keys` page** (Add → alias `n2yo` → paste key). Do this **before** enabling the adapter — missing-key path is graceful but the adapter logs INFO and skips polling until the key lands.
4. Apply migration 040 manually via psql (per option C established pattern):
   `sudo -u postgres psql central -f /opt/central/sql/migrations/040_add_n2yo_visualpasses_adapter.sql`
   **Do NOT** run `central-migrate` — orphan migrations 032-039 stay deferred for the morning queue.
5. `sudo systemctl restart central-supervisor` (picks up the new adapter via discovery) + `sudo systemctl restart central-gui` (picks up new partials + ADAPTER_GROUPS change).
6. **No** `central-archive` restart (CENTRAL_SAT pre-existed; only the adapter row is new).
7. Verify: `config.adapters` has `n2yo_visualpasses` row with `enabled=false`; `config.api_keys` has alias `n2yo`; supervisor log shows the adapter discovered but not polling (matches `enabled=false`).
8. Matt enables via `/adapters/n2yo_visualpasses/edit` when ready. First poll happens within 1h; events surface at `/events` filtered by adapter=n2yo_visualpasses.

## Halt acknowledgment

Per spec acceptance bar #6: **squash-merge NOT authorized**. Branch + PR open. Halting for line-by-line review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-09 16:00:55 -06:00
malice
c49f2db95f
 		v0.12.0: sat_positions adapter (live global satellite positions) + sat_common refactor 
## Architectural framing

The v0.11.1 `satpass_predict` adapter is **observer-anchored**: "when does satellite X pass over fixed observer Y, and what's the elevation/azimuth at that observer's site?" It answers a fixed-QTH question and emits one event per (observer, satellite, AOS) tuple.

The new `sat_positions` adapter is the **global** counterpart: "where is satellite X right now?" No observer. One event per tracked NORAD ID per poll, on subject `central.sat.position.<norad_id>`. Consumers (meshAI, GUI map widgets, anything that wants a live world map) subscribe to `central.sat.position.>` and plot.

They complement each other; neither replaces the other.

Direct quote from Matt's use-case: *"location of the sats... map of where the sats are then we have meshai or whatever other service calling central's data grab it and do whatever work it needed."* This adapter is that.

## sat_common extraction rationale

The four pure SGP4 / coordinate helpers (`EARTH_RADIUS_KM`, `gmst_rad`, `eci_to_ecef`, `subsatellite_point`) were private symbols inside `satpass_predict.py`. `sat_positions` needs the same three helpers. Three options were considered:

1. **Cross-import** from `satpass_predict.py` — creates an adapter-to-adapter dependency, ugly.
2. **Extract to `sat_common.py`** — matches the existing `wfigs_common.py` / `swpc_common.py` precedent. Both adapters become siblings of a shared helper module. ✓ chosen.
3. **Duplicate** — math drift over time.

Symbol names dropped their leading underscore on extraction (public-API convention matching `swpc_common.parse_swpc_timestamp` / `wfigs_common.severity_from_acres`). Existing internal call sites in `satpass_predict.py` were updated via mechanical `replace_all`. Observer-specific helpers (`_observer_ecef`, `_topocentric_az_el`, `_visibility_footprint`, `_severity_from_elev`, `_build_pass_geometry`, `_next_passes`) stay in `satpass_predict.py` per YAGNI — they're not used by `sat_positions` today.

Existing `tests/test_satpass_predict.py` was updated mechanically to import the helpers from `sat_common` via aliases (preserves the underscore-prefixed local names in the tests so the rest of the test body needs no change). All 44 satpass_predict tests pass unchanged.

## CENTRAL_SAT stream cap bump

`config.streams.max_bytes` for `CENTRAL_SAT` goes from **1 GiB → 5 GiB** in migration 039. Sizing math:

- celestrak_tle: ~190 sats × 1 envelope/day = ~190 events/day = ~1.4k events/week. Fit in 1 GiB easily.
- sat_positions: ~190 sats × 1440 ticks/day (60s cadence) = **~273.6k events/day = ~1.9M events/week**. At ~1 KB per envelope including the CloudEvents wrapper, that's **~1.9 GiB/week**.
- Plus existing TLE + pass envelopes already on the stream → ~3 GiB headroom needed.
- 5 GiB = 5368709120 bytes = operator-tunable margin without over-provisioning.

`STREAM_CATEGORY_DOMAINS["CENTRAL_SAT"]` extends from `("tle", "pass")` to `("tle", "pass", "position")` so the supervisor's retention sweep covers position events too.

## Subject + dedup

| Field | Value |
|---|---|
| Subject | `central.sat.position.<norad_id>` — one subject per satellite, globally |
| Dedup id | `<norad_id>:<position_iso>` where `position_iso` is the propagation timestamp truncated to whole seconds (defensive collapse if cadence is ever tightened) |
| Severity | 1 (informational telemetry, no alerting) |
| data_class | `telemetry` — surfaces on `/telemetry`, not `/events` |
| Cadence | 60s default; operator-tunable via standard `cadence_s` field |

## Settings shape

```json
{"track_only_norad_ids": [], "max_tle_age_days": 14}
```

- Empty `track_only_norad_ids` = track every NORAD ID with a fresh TLE in the events table (derive-from-celestrak_tle, default behavior).
- Non-empty list pins to those NORAD IDs only (operator override — "I only care about the ISS and these 12 Starlink sats").
- `max_tle_age_days` bounds TLE freshness; LEO drag means TLEs go stale in days, GEO is good for months. Parameterized into the SQL query as a timedelta interval so operator-tightened windows (e.g. 3d) apply without code change.

## Event.data fields

`norad_id`, `satellite_name`, `lon_deg`, `lat_deg`, `alt_km`, `velocity_kmps`, `heading_deg`, `tle_epoch`.

- `lon_deg`/`lat_deg`/`alt_km`: sub-satellite point via SGP4 → ECI → ECEF rotation → spherical-earth lon/lat/alt.
- `velocity_kmps`: magnitude of the SGP4 ECI velocity vector. ECI vs ECEF difference is ~6% for LEO (earth rotation 0.46 km/s vs 7.7 km/s orbital speed); fine for consumer "the sat is moving at X km/s" text.
- `heading_deg`: great-circle initial bearing from the sub-sat point at `t` to the sub-sat point at `t+1s` (finite-difference; simpler than rotating velocity through GMST + the earth-rotation cross term).

## Diff size — flag for review

**+894 / -63 = +831 net** across 14 files. Spec budget was ≤700 lines. **Over by ~131 net** (or ~194 gross).

Breakdown:
- `sat_positions.py`: 286 lines (under the ≤350 adapter line cap ✓)
- `sat_common.py`: 65 lines (the extraction)
- Migration 039: 58 lines (heavy on inline comments documenting the size math; could trim ~25 lines if you want)
- satpass_predict.py: net -1 line (refactor; lost 4 helper defs and one constant comment, gained 5-line import block)
- Templates: 14 lines (event_rows + event_summaries partials)
- Wiring: 4 lines (supervisor + ADAPTER_GROUPS)
- Docs (CONSUMER-INTEGRATION.md): 40 lines (required by `tests/test_consumer_doc.py::test_every_adapter_has_a_subsection`)
- **Tests: 426 lines.** This is the bulk of the overage.

The tests are all spec-mandated (sub-sat math, velocity range, heading range, build_event, subject_for, empty-TLE, track_only gate, stale-TLE skip, sat_common helpers, regression-guard on the moved helpers via test_satpass_predict.py preservation). I could shrink `test_sat_positions.py` by consolidating the 11 spec-mandated tests into fewer parameterized cases, but each test pins one behavior the spec called out by name. Flagging for your call: keep as-is, or do you want a tighter parameterized version?

## Test plan

- [x] `pytest tests/test_sat_common.py tests/test_sat_positions.py` — **28 new tests, all pass**.
- [x] `pytest tests/test_satpass_predict.py` — **44/44 pass** (regression guard: existing tests work after the sat_common extraction).
- [x] `pytest tests/test_events_feed_frontend.py` — **119/119 pass** (JSON-feed coverage extended to include sat_positions sample event + expected subject string).
- [x] `pytest tests/test_telemetry_separation.py` — **9/9 pass** (`_TELEMETRY` pin extended to include `sat_positions`).
- [x] `pytest tests/test_consumer_doc.py` — **6/6 pass** (CONSUMER-INTEGRATION.md `### sat_positions` subsection added).
- [x] `pytest tests/test_producer_doc.py` — **10/10 pass** (no PRODUCER-INTEGRATION update needed; CENTRAL_SAT stream is pre-existing).
- [x] Full sweep `pytest tests/` (excluding postgres-dep files): **1209 passed, 1 skipped, 0 failures**.
- [x] Ruff: clean on all new code. 3 pre-existing F841 unused-variable warnings (supervisor.py:390 `poll_start`, test_events_feed_frontend.py:425 / :466 `result`) confirmed via `git blame` to be from commits May 2026 — not introduced.

## Deploy plan

1. Squash-merge → tag v0.12.0 at merge SHA → push tag.
2. `ssh central`, `git pull` on `/opt/central`. **No `uv sync`** (no new dep).
3. **`central-migrate`** to apply migration 039 (seeds `config.adapters` row + bumps `config.streams.max_bytes` for CENTRAL_SAT).
4. `sudo systemctl restart central-supervisor` (picks up STREAM_CATEGORY_DOMAINS extension + new adapter discovery).
5. `sudo systemctl restart central-gui` (picks up new partials + ADAPTER_GROUPS change).
6. **No** `central-archive` restart (CENTRAL_SAT stream already exists; no new stream).
7. Verify: `nats stream info CENTRAL_SAT` shows max_bytes=5368709120; supervisor journal shows sat_positions discovered.
8. Smoke-test: enable celestrak_tle first if not already, wait for one poll, then enable sat_positions via GUI. Within 60s expect one `central.sat.position.<norad_id>` event per tracked sat on the stream.

## Halt acknowledgment

Per spec acceptance bar #6: **squash-merge NOT authorized**. Branch + PR open. Halting for line-by-line review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-09 15:23:32 -06:00
malice
86e8b6b56a
 		v0.11.1: satpass_predict adapter (server-side pass alerts for fixed observers) (#101) 2026-06-09 01:16:43 -06:00
malice
621148ac46
 		v0.11.0: new celestrak_tle adapter + CENTRAL_SAT satellite-tracking stream (#100) 2026-06-09 00:54:19 -06:00
malice
e92b51c518
 		v0.10.10: new avalanche_org adapter — backcountry avalanche advisories (#98) 
meshai-requested adapter for avalanche.org's per-center map layers (SNFAC
Sawtooth + PAC Payette by default; operator-extensible to any avalanche.org
forecast center). Pure passthrough + severity gate, no cross-source fusion,
fits Central's adapter pattern cleanly.

Adapter surface:
- Polls https://api.avalanche.org/v2/public/products/map-layer/{center_id}
  per configured center; default cadence 1800s (30 min).
- Severity gate: only danger_level >= 3 publishes. danger_level 0/1/2
  (None/Low/Moderate), -1 ('no rating'), and off_season=true all omitted at
  adapter level. Idaho summer = all 4 SNFAC + 2 PAC zones yield 0 events;
  that's correct behavior, verified by the negative-case test against the
  frozen 2026-06-08 SNFAC fixture.
- Severity mapping (corrected from meshai's inverted spec): danger_level
  3 (Considerable) → severity 2, 4 (High) → 3, 5 (Extreme) → 4. Matches
  Central's 4-most-severe convention (nws.SEVERITY_MAP).
- Subject: central.avy.advisory.us.{state_lower} — one per state; v0.10.8's
  category-discriminated Nats-Msg-Id keeps multiple zones in the same state
  from colliding in JetStream dedup.
- Stream: CENTRAL_AVY (central.avy.>); 7-day / 1 GiB retention defaults.
- Event.data fields per meshai spec: center_id, zone_name, danger_level,
  danger_name, travel_advice (truncated to 200 chars), state, valid_date,
  end_date, off_season=false, latitude/longitude (polygon centroid via
  shapely), plus geo.geometry passes through as the upstream Polygon.

Tests (38 in test_avalanche_org.py):
- Pure helpers: _slug (8 cases), _parse_iso (6 cases), _centroid (2 cases).
- Severity gate: 3 publish cases (danger 3/4/5 → severity 2/3/4),
  4 omit cases (danger -1/0/1/2), off_season=true omit, missing state omit,
  unparseable geom omit, travel_advice truncation, subject derivation.
- Real-fixture negative case: 4-zone SNFAC fixture all omitted off-season.
- Real-fixture positive case: same fixture with synthetic winter overrides
  publishes all 4 with valid centroids on actual Idaho polygons.
- End-to-end poll() with mixed severities and the new wiring (streams
  registry + supervisor family map).
- Defensive: empty center_ids list yields nothing without crashing.

Wiring + plumbing:
- src/central/streams.py: StreamEntry('CENTRAL_AVY', 'central.avy.>')
- src/central/supervisor.py: STREAM_CATEGORY_DOMAINS['CENTRAL_AVY']=('avy',)
- sql/migrations/035: seed config.streams row (mirror of 019/CENTRAL_SPACE,
  idempotent ON CONFLICT DO NOTHING). Note: migrations don't auto-run on
  supervisor restart -- see deferred ops list (schema_migrations cleanup
  blocks central-migrate from running anything cleanly).
- src/central/gui/templates/_event_rows/avalanche_org.html (8 lines)
- src/central/gui/templates/_event_summaries/avalanche_org.html (2 lines)
  Both required by the existing per-adapter template consistency tests.

Doc updates (required by existing doc-vs-registry tests):
- docs/PRODUCER-INTEGRATION.md §6.1: added 'avy' to top-level-domain list.
- docs/PRODUCER-INTEGRATION.md §8: added StreamEntry('CENTRAL_AVY',...) line
  to the verbatim snippet.
- docs/CONSUMER-INTEGRATION.md §3 stream layout table: added CENTRAL_AVY row.
- docs/CONSUMER-INTEGRATION.md §6: new '### avalanche_org' subsection with
  source, subject convention, dedup key, severity gate, Event.data field
  table, and off-season behavior note.
- tests/test_events_feed_frontend.py: added avalanche_org to _SAMPLE_INNER
  and _EXPECTED_SUBJECT (the events-JSON subject-derivation coverage tests).

Budget note: this PR is well over the ~400-line target -- the new-adapter
surface picked up downstream consistency tests (doc validators + frontend
sample coverage + template partials) I didn't anticipate at probe time.
Most of the overrun is the SNFAC fixture (1,135 lines pretty-printed JSON,
non-code) and the adapter + tests pair. Stripping the fixture and the
required doc/template edits would leave ~620 lines of code; the fixture
itself is a frozen snapshot, not a maintenance burden.

Full sweep: 1072 passed, 0 failures (+41 from this PR), ruff clean on
all new files. One PRE-EXISTING ruff violation in supervisor.py (unused
poll_start variable at line 388) surfaces when we touch supervisor.py;
confirmed not introduced by this PR via git stash check.

Deploy plan (NEW STREAM — archive restart required per
[[feedback_new_stream_needs_archive_restart]]):
1. Squash-merge -> tag v0.10.10 -> push.
2. On central: pull main -> systemctl restart central-supervisor -> ALSO
   systemctl restart central-archive (new event-bearing stream; archive
   enumerates consumers at startup and doesn't hot-reload).
3. Migration 035 deferred to morning per the schema_migrations cleanup
   task -- the stream creation itself doesn't depend on it (supervisor
   creates JetStream streams from the STREAMS registry at startup; the
   config.streams row is for operator-tunable retention only).
4. Verify: nats stream info CENTRAL_AVY (created), poll log shows
   yielded=0 / omitted=N (off-season), no positive publishes during
   summer (correct).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-08 21:57:56 -06:00
malice
0dd83a340e
 		v0.10.3: rip out state_511_atis adapter (superseded by itd_511 v0.10.0; Castle Rock legacy shape EOL per sister-site discovery) (#88) 
Closes #88

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-06 14:44:00 -06:00
malice
1d5548c24c
 		v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85) 
First official-state-DOT-API pattern landing. Two adapters in one PR:

- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
  https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
  to LineString via the polyline lib (bookend LineString or Point fallback);
  ITD Severity string mapped None->1 / Minor->2 / Major->3 with
  IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
  Restrictions / DetourPolyline pass through unmodified. Advisories ship
  as structural pass-through under data.advisory since the upstream
  /alerts endpoint currently returns []; per-record try/except keeps a
  surprise shape from sinking the cycle when ITD posts its first one.

- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
  camera per UTC day; image URL passes straight through to <img src>.
  Region uniform US-ID with data.source_jurisdiction preserving the raw
  upstream Source field for the ~1.2% cross-DOT border-region mirrors
  (UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).

Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.

Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.

Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.

Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-03 22:36:26 -06:00
Matt Johnson
02bc692bda feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6) 
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).

supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.

- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
  100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
  cams). The 100-row page cap also means v0.9.2 state_511_atis silently
  truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
  camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
  -- always shows today's cameras, no per-poll flooding, no retention
  coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
  source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
  shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).

Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 01:33:21 +00:00
Matt Johnson
7fdf47f2f0 feat(tomtom_incidents): per-bbox cadence (v0.9.5.1) 
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.

- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
  falls back to the adapter's default_cadence_s. Existing settings without the
  field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
  fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
  (one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
  due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
  of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).

central-supervisor restart only. No gui/archive restart, no migration, no new dep.

Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 01:01:55 +00:00
Matt Johnson
42d5faa80c feat(tomtom_incidents): TomTom real-time traffic incidents adapter (v0.9.5) 
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.

central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.

- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
  is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
  bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
  rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
  state_511_atis; cross-source overlap is by design = additive coverage, distinct
  dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
  the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
  verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
  key redacted from logs; poll skips cleanly without a key.

Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 00:25:27 +00:00
Matt Johnson
b8033444ec feat(tomtom_flow): TomTom Orbis vector flow-tile telemetry adapter + CENTRAL_TRAFFIC_FLOW (v0.9.3) 
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.

Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.

- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
  with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
  inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
  the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
  adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
  so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
  explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
  transitive that uv re-lock would otherwise prune).

Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 23:25:44 +00:00
Matt Johnson
30e25bf475 feat(state_511_atis): Castle Rock 511 adapter — Idaho incidents/closures/road work (v0.9.2) 
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.

Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.

Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.

Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.

Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.

Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 22:01:11 +00:00
Matt Johnson
7eab5fc1b1 feat(wzdx): WZDx adapter + CENTRAL_TRAFFIC family bootstrap (v0.9.0) 
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.

First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.

Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 20:35:08 +00:00
Matt Johnson
98b050b2af feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment 
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).

Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
  + landclass + elevation in one call). Near-passthrough: response already
  matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
  construction when a loop is running; config `headers` slot for a future
  Authorization: Bearer (config-only, no code change). Default base_url
  http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
  Maps features[0].properties; postal_code <- postcode; timezone/landclass/
  elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
  Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
  required User-Agent. Maps the address block; landclass/elevation_m/timezone
  null.

Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).

Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
  Central is the consumer's only data plane (consumers can't do follow-up
  lookups), so enrich deliberately and centrally, namespaced under _enriched,
  failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
  OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
  namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
  GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
  cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
  per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
  landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
  fields), per-region coverage (US-full, non-US timezone+elevation), and the
  antimeridian landclass caveat.

Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
  handling, extra-key drop, network/timeout/non-200/malformed -> all-null
  (never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
  Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
  192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
  §13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
  (derived from code, not hardcoded).

Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.

Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:10:44 +00:00
malice
d92074b134
 		docs(2-H): consumer integration spec — docs/CONSUMER-INTEGRATION.md (#38) 
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".

Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.

Sections (11 total):
  1. Quick start (5-line nats-py subscribe-and-print)
  2. Connection details (URL / auth / JetStream context / stream discovery)
  3. Stream layout (7 streams, derived from streams.py registry)
  4. Subject namespace registry (Mermaid tree + full pattern table)
  5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
     -- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
  6. Per-adapter reference (12 subsections, locked template)
  7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
  8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
  9. Dedup implementation guide (single-token vs composite-key adapters)
  10. Writing a new consumer checklist
  11. Troubleshooting

Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.

Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.

The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.

New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
  - The "Stream layout" table matches central.streams.STREAMS exactly
    (stream names + subject filters).
  - The (name, subject_filter) pairs match the registry as pairs (catches
    swapped subject filters on existing streams).
  - Every adapter discovered via central.adapter_discovery.discover_adapters()
    has a per-adapter subsection -- and vice versa.
  - The subsection count equals the registry size (catches duplicates).

Verification:
  - 463/463 full suite green (was 458; +5 new consumer_doc tests).
  - Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
    12 JSON blocks (all parse).
  - All 12 adapters covered.
  - No regressions elsewhere.

Acceptance bars (a)-(e) verbatim:
  (a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
  (b) all 12 adapters have per-adapter subsections
  (c) 5/5 consumer-doc tests pass
  (d) 463/463 full suite
  (e) doc length 1878 lines

markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).

Co-authored-by: zvx <zvx@central>
 2026-05-19 14:33:51 -06:00