## Architectural placement — complement, not replacement
| | satpass_predict (v0.11.1) | **n2yo_visualpasses (this PR)** |
|---|---|---|
| Computes from | Raw TLEs via local SGP4 | n2yo's pre-computed visualpasses endpoint |
| Magnitude data? | ✗ (SGP4 alone can't compute illumination) | ✓ (server-side sun-geometry) |
| Sun illumination filter? | ✗ | ✓ (n2yo returns sunlit passes only) |
| Cost per (observer, sat) pair | Local compute, free | One n2yo API transaction |
| Failure mode | TLE drift over time | Quota exhaustion, vendor outage |
Both adapters serve the same operator question ("when is sat X overhead at site Y?") but with different data sources. Matt's stated use case is to have **both** running so a vendor outage or quota burn on n2yo doesn't blind him to passes that satpass_predict can still propagate locally.
## Subject collision is intentional
Both adapters emit on `central.sat.pass.us.<state_lower>.<observer_slug>`. A consumer subscribing to e.g. `central.sat.pass.us.id.boise` receives events from **both** adapters. Disambiguation lives in `data.category`:
- `pass.satpass_predict` → local SGP4
- `pass.n2yo_visualpasses` → n2yo API
The v0.10.8 category-discriminated `Nats-Msg-Id` keeps both adapters' JetStream dedup windows separate even when they emit for the same (observer, satellite, AOS) tuple (which they will, by design, for sunlit passes).
This is documented explicitly in the new `### n2yo_visualpasses` subsection of `docs/CONSUMER-INTEGRATION.md` so future consumer integrators don't get surprised.
## Quota math
Default settings ship a curated **6 observers × 6 sats** configuration:
- **Observers** (ID + UT): Filer (primary), Boise, Idaho Falls, Ogden, Salt Lake City, Provo
- **Satellites** (curated for amateur observation): ISS (25544), NOAA-15 (25338), NOAA-18 (28654), NOAA-19 (33591), SO-50 (27607), AO-91 (43017)
At 1h cadence: **6 × 6 × 24 = 864 transactions/day**, comfortably under n2yo's free-tier **1000/day cap** with ~13% headroom for retries or expansion. Operator can extend either dimension if they upgrade quota.
## API key plumbing (tomtom_flow pattern)
Exact mirror of the v0.9.3 tomtom_flow precedent — confirmed during recon to be the established pattern:
```python
requires_api_key = "n2yo" # class attr, GUI surfaces "requires X" warning
api_key_field = "api_key_alias" # class attr, GUI renders api_key_select dropdown
# Settings field:
api_key_alias: str = "n2yo"
```
Cached `_api_key` populated via `ConfigStore.get_api_key(alias)` in `startup()` and `apply_config()`. Missing-key path: log INFO, return immediately (zero events, no exception). The live key is scrubbed from log strings via a `_redact()` helper before they hit journald.
**`python -m set_api_key` does not exist** — that was a speculative invocation in the spec. The actual flow is GUI-based: Matt adds the `n2yo` alias via the `/api-keys` page, then enables the adapter via `/adapters/n2yo_visualpasses/edit`.
## Diff size — flag for review
**+848 / −1 = +847 net** across 8 files. Spec budget was ≤600 lines. **Over by ~247** (~41%, similar shape to v0.12.0's overage).
| File | Lines | Notes |
|---|---|---|
| `src/central/adapters/n2yo_visualpasses.py` | 330 | **Under** the ≤350 adapter cap ✓ |
| `tests/test_n2yo_visualpasses.py` | 411 | The bulk of the overage |
| `sql/migrations/040_add_n2yo_visualpasses_adapter.sql` | 45 | Heavy comment block; could trim ~15 lines |
| `docs/CONSUMER-INTEGRATION.md` | 40 | Required by `test_consumer_doc` |
| Partials (event_rows + event_summaries) | 13 | |
| `tests/test_events_feed_frontend.py` | 8 | _SAMPLE_INNER + _EXPECTED_SUBJECT |
| `src/central/gui/routes.py` | 1 | ADAPTER_GROUPS extension |
**Test breakdown** (31 tests in 8 classes):
- 9 severity-bucketing tests — spec called out 4 boundaries (-3.1, -2.9, -0.5, 2.5); the extra 5 pin inclusive-vs-exclusive at -3.0, -1.0, 2.0 boundaries + the ranges in between. Useful regression guards but not strictly spec-required.
- 4 settings-default tests — pin the curated 6×6 set + quota math.
- 4 adapter-class-attrs tests — pin requires_api_key/api_key_field/data_class/default_cadence_s wiring.
- 3 subject_for tests — happy path + UT-state lowercasing + unknown fallback.
- 1 _pass_to_event shape test.
- 7 poll-loop tests — missing key, empty observers, empty norad_ids, happy path, empty passes array, fetch-failure-doesn't-kill-poll, multi-obs-multi-sat 6×6 aggregate.
- 1 HTTP-layer test — 401 → None (the one test that goes through the real session.get mock).
- 2 static-isolation tests — acceptance bar #2 (no hardcoded keys) and #4 (no absolute paths).
I can trim the test file to ~250 lines by dropping the non-strictly-spec-mandated tests (settings defaults, class attrs, extra severity boundaries, extra subject_for variants). **Flag for your call:** keep the comprehensive suite, or trim to spec minimum?
## Test plan
- [x] `pytest tests/test_n2yo_visualpasses.py` — **31/31 pass** (all offline, zero n2yo API hits).
- [x] `pytest tests/test_events_feed_frontend.py` — **122/122 pass** (fixture coverage extended).
- [x] `pytest tests/test_consumer_doc.py` — **6/6 pass** (new `### n2yo_visualpasses` subsection accepted).
- [x] Full sweep `pytest tests/` (excluding postgres-dep files) — **1243 passed, 1 skipped, 0 failures**.
- [x] Ruff: **clean on new files** (`n2yo_visualpasses.py`, `test_n2yo_visualpasses.py`). The pre-existing F841 warnings in routes.py / test_events_feed_frontend.py / supervisor.py are unchanged from v0.11.3-pre.
- [x] **No hardcoded API key in diff** — `git diff main..HEAD | grep -iE 'apiKey=[A-Z0-9]{6,}|api_key.*=.*"[A-Z0-9]{6,}'` returns empty.
- [x] **No absolute paths in test code** — `TestStaticIsolation` enforces this at runtime.
## Deploy plan
1. Squash-merge PR #N → tag v0.12.1 at merge SHA → push tag.
2. `ssh central`, `git pull` on `/opt/central`. **No `uv sync`** (aiohttp already in venv from earlier adapters).
3. **Matt adds the n2yo API key via GUI `/api-keys` page** (Add → alias `n2yo` → paste key). Do this **before** enabling the adapter — missing-key path is graceful but the adapter logs INFO and skips polling until the key lands.
4. Apply migration 040 manually via psql (per option C established pattern):
`sudo -u postgres psql central -f /opt/central/sql/migrations/040_add_n2yo_visualpasses_adapter.sql`
**Do NOT** run `central-migrate` — orphan migrations 032-039 stay deferred for the morning queue.
5. `sudo systemctl restart central-supervisor` (picks up the new adapter via discovery) + `sudo systemctl restart central-gui` (picks up new partials + ADAPTER_GROUPS change).
6. **No** `central-archive` restart (CENTRAL_SAT pre-existed; only the adapter row is new).
7. Verify: `config.adapters` has `n2yo_visualpasses` row with `enabled=false`; `config.api_keys` has alias `n2yo`; supervisor log shows the adapter discovered but not polling (matches `enabled=false`).
8. Matt enables via `/adapters/n2yo_visualpasses/edit` when ready. First poll happens within 1h; events surface at `/events` filtered by adapter=n2yo_visualpasses.
## Halt acknowledgment
Per spec acceptance bar #6: **squash-merge NOT authorized**. Branch + PR open. Halting for line-by-line review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Architectural framing
The v0.11.1 `satpass_predict` adapter is **observer-anchored**: "when does satellite X pass over fixed observer Y, and what's the elevation/azimuth at that observer's site?" It answers a fixed-QTH question and emits one event per (observer, satellite, AOS) tuple.
The new `sat_positions` adapter is the **global** counterpart: "where is satellite X right now?" No observer. One event per tracked NORAD ID per poll, on subject `central.sat.position.<norad_id>`. Consumers (meshAI, GUI map widgets, anything that wants a live world map) subscribe to `central.sat.position.>` and plot.
They complement each other; neither replaces the other.
Direct quote from Matt's use-case: *"location of the sats... map of where the sats are then we have meshai or whatever other service calling central's data grab it and do whatever work it needed."* This adapter is that.
## sat_common extraction rationale
The four pure SGP4 / coordinate helpers (`EARTH_RADIUS_KM`, `gmst_rad`, `eci_to_ecef`, `subsatellite_point`) were private symbols inside `satpass_predict.py`. `sat_positions` needs the same three helpers. Three options were considered:
1. **Cross-import** from `satpass_predict.py` — creates an adapter-to-adapter dependency, ugly.
2. **Extract to `sat_common.py`** — matches the existing `wfigs_common.py` / `swpc_common.py` precedent. Both adapters become siblings of a shared helper module. ✓ chosen.
3. **Duplicate** — math drift over time.
Symbol names dropped their leading underscore on extraction (public-API convention matching `swpc_common.parse_swpc_timestamp` / `wfigs_common.severity_from_acres`). Existing internal call sites in `satpass_predict.py` were updated via mechanical `replace_all`. Observer-specific helpers (`_observer_ecef`, `_topocentric_az_el`, `_visibility_footprint`, `_severity_from_elev`, `_build_pass_geometry`, `_next_passes`) stay in `satpass_predict.py` per YAGNI — they're not used by `sat_positions` today.
Existing `tests/test_satpass_predict.py` was updated mechanically to import the helpers from `sat_common` via aliases (preserves the underscore-prefixed local names in the tests so the rest of the test body needs no change). All 44 satpass_predict tests pass unchanged.
## CENTRAL_SAT stream cap bump
`config.streams.max_bytes` for `CENTRAL_SAT` goes from **1 GiB → 5 GiB** in migration 039. Sizing math:
- celestrak_tle: ~190 sats × 1 envelope/day = ~190 events/day = ~1.4k events/week. Fit in 1 GiB easily.
- sat_positions: ~190 sats × 1440 ticks/day (60s cadence) = **~273.6k events/day = ~1.9M events/week**. At ~1 KB per envelope including the CloudEvents wrapper, that's **~1.9 GiB/week**.
- Plus existing TLE + pass envelopes already on the stream → ~3 GiB headroom needed.
- 5 GiB = 5368709120 bytes = operator-tunable margin without over-provisioning.
`STREAM_CATEGORY_DOMAINS["CENTRAL_SAT"]` extends from `("tle", "pass")` to `("tle", "pass", "position")` so the supervisor's retention sweep covers position events too.
## Subject + dedup
| Field | Value |
|---|---|
| Subject | `central.sat.position.<norad_id>` — one subject per satellite, globally |
| Dedup id | `<norad_id>:<position_iso>` where `position_iso` is the propagation timestamp truncated to whole seconds (defensive collapse if cadence is ever tightened) |
| Severity | 1 (informational telemetry, no alerting) |
| data_class | `telemetry` — surfaces on `/telemetry`, not `/events` |
| Cadence | 60s default; operator-tunable via standard `cadence_s` field |
## Settings shape
```json
{"track_only_norad_ids": [], "max_tle_age_days": 14}
```
- Empty `track_only_norad_ids` = track every NORAD ID with a fresh TLE in the events table (derive-from-celestrak_tle, default behavior).
- Non-empty list pins to those NORAD IDs only (operator override — "I only care about the ISS and these 12 Starlink sats").
- `max_tle_age_days` bounds TLE freshness; LEO drag means TLEs go stale in days, GEO is good for months. Parameterized into the SQL query as a timedelta interval so operator-tightened windows (e.g. 3d) apply without code change.
## Event.data fields
`norad_id`, `satellite_name`, `lon_deg`, `lat_deg`, `alt_km`, `velocity_kmps`, `heading_deg`, `tle_epoch`.
- `lon_deg`/`lat_deg`/`alt_km`: sub-satellite point via SGP4 → ECI → ECEF rotation → spherical-earth lon/lat/alt.
- `velocity_kmps`: magnitude of the SGP4 ECI velocity vector. ECI vs ECEF difference is ~6% for LEO (earth rotation 0.46 km/s vs 7.7 km/s orbital speed); fine for consumer "the sat is moving at X km/s" text.
- `heading_deg`: great-circle initial bearing from the sub-sat point at `t` to the sub-sat point at `t+1s` (finite-difference; simpler than rotating velocity through GMST + the earth-rotation cross term).
## Diff size — flag for review
**+894 / -63 = +831 net** across 14 files. Spec budget was ≤700 lines. **Over by ~131 net** (or ~194 gross).
Breakdown:
- `sat_positions.py`: 286 lines (under the ≤350 adapter line cap ✓)
- `sat_common.py`: 65 lines (the extraction)
- Migration 039: 58 lines (heavy on inline comments documenting the size math; could trim ~25 lines if you want)
- satpass_predict.py: net -1 line (refactor; lost 4 helper defs and one constant comment, gained 5-line import block)
- Templates: 14 lines (event_rows + event_summaries partials)
- Wiring: 4 lines (supervisor + ADAPTER_GROUPS)
- Docs (CONSUMER-INTEGRATION.md): 40 lines (required by `tests/test_consumer_doc.py::test_every_adapter_has_a_subsection`)
- **Tests: 426 lines.** This is the bulk of the overage.
The tests are all spec-mandated (sub-sat math, velocity range, heading range, build_event, subject_for, empty-TLE, track_only gate, stale-TLE skip, sat_common helpers, regression-guard on the moved helpers via test_satpass_predict.py preservation). I could shrink `test_sat_positions.py` by consolidating the 11 spec-mandated tests into fewer parameterized cases, but each test pins one behavior the spec called out by name. Flagging for your call: keep as-is, or do you want a tighter parameterized version?
## Test plan
- [x] `pytest tests/test_sat_common.py tests/test_sat_positions.py` — **28 new tests, all pass**.
- [x] `pytest tests/test_satpass_predict.py` — **44/44 pass** (regression guard: existing tests work after the sat_common extraction).
- [x] `pytest tests/test_events_feed_frontend.py` — **119/119 pass** (JSON-feed coverage extended to include sat_positions sample event + expected subject string).
- [x] `pytest tests/test_telemetry_separation.py` — **9/9 pass** (`_TELEMETRY` pin extended to include `sat_positions`).
- [x] `pytest tests/test_consumer_doc.py` — **6/6 pass** (CONSUMER-INTEGRATION.md `### sat_positions` subsection added).
- [x] `pytest tests/test_producer_doc.py` — **10/10 pass** (no PRODUCER-INTEGRATION update needed; CENTRAL_SAT stream is pre-existing).
- [x] Full sweep `pytest tests/` (excluding postgres-dep files): **1209 passed, 1 skipped, 0 failures**.
- [x] Ruff: clean on all new code. 3 pre-existing F841 unused-variable warnings (supervisor.py:390 `poll_start`, test_events_feed_frontend.py:425 / :466 `result`) confirmed via `git blame` to be from commits May 2026 — not introduced.
## Deploy plan
1. Squash-merge → tag v0.12.0 at merge SHA → push tag.
2. `ssh central`, `git pull` on `/opt/central`. **No `uv sync`** (no new dep).
3. **`central-migrate`** to apply migration 039 (seeds `config.adapters` row + bumps `config.streams.max_bytes` for CENTRAL_SAT).
4. `sudo systemctl restart central-supervisor` (picks up STREAM_CATEGORY_DOMAINS extension + new adapter discovery).
5. `sudo systemctl restart central-gui` (picks up new partials + ADAPTER_GROUPS change).
6. **No** `central-archive` restart (CENTRAL_SAT stream already exists; no new stream).
7. Verify: `nats stream info CENTRAL_SAT` shows max_bytes=5368709120; supervisor journal shows sat_positions discovered.
8. Smoke-test: enable celestrak_tle first if not already, wait for one poll, then enable sat_positions via GUI. Within 60s expect one `central.sat.position.<norad_id>` event per tracked sat on the stream.
## Halt acknowledgment
Per spec acceptance bar #6: **squash-merge NOT authorized**. Branch + PR open. Halting for line-by-line review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
meshai-requested adapter for avalanche.org's per-center map layers (SNFAC
Sawtooth + PAC Payette by default; operator-extensible to any avalanche.org
forecast center). Pure passthrough + severity gate, no cross-source fusion,
fits Central's adapter pattern cleanly.
Adapter surface:
- Polls https://api.avalanche.org/v2/public/products/map-layer/{center_id}
per configured center; default cadence 1800s (30 min).
- Severity gate: only danger_level >= 3 publishes. danger_level 0/1/2
(None/Low/Moderate), -1 ('no rating'), and off_season=true all omitted at
adapter level. Idaho summer = all 4 SNFAC + 2 PAC zones yield 0 events;
that's correct behavior, verified by the negative-case test against the
frozen 2026-06-08 SNFAC fixture.
- Severity mapping (corrected from meshai's inverted spec): danger_level
3 (Considerable) → severity 2, 4 (High) → 3, 5 (Extreme) → 4. Matches
Central's 4-most-severe convention (nws.SEVERITY_MAP).
- Subject: central.avy.advisory.us.{state_lower} — one per state; v0.10.8's
category-discriminated Nats-Msg-Id keeps multiple zones in the same state
from colliding in JetStream dedup.
- Stream: CENTRAL_AVY (central.avy.>); 7-day / 1 GiB retention defaults.
- Event.data fields per meshai spec: center_id, zone_name, danger_level,
danger_name, travel_advice (truncated to 200 chars), state, valid_date,
end_date, off_season=false, latitude/longitude (polygon centroid via
shapely), plus geo.geometry passes through as the upstream Polygon.
Tests (38 in test_avalanche_org.py):
- Pure helpers: _slug (8 cases), _parse_iso (6 cases), _centroid (2 cases).
- Severity gate: 3 publish cases (danger 3/4/5 → severity 2/3/4),
4 omit cases (danger -1/0/1/2), off_season=true omit, missing state omit,
unparseable geom omit, travel_advice truncation, subject derivation.
- Real-fixture negative case: 4-zone SNFAC fixture all omitted off-season.
- Real-fixture positive case: same fixture with synthetic winter overrides
publishes all 4 with valid centroids on actual Idaho polygons.
- End-to-end poll() with mixed severities and the new wiring (streams
registry + supervisor family map).
- Defensive: empty center_ids list yields nothing without crashing.
Wiring + plumbing:
- src/central/streams.py: StreamEntry('CENTRAL_AVY', 'central.avy.>')
- src/central/supervisor.py: STREAM_CATEGORY_DOMAINS['CENTRAL_AVY']=('avy',)
- sql/migrations/035: seed config.streams row (mirror of 019/CENTRAL_SPACE,
idempotent ON CONFLICT DO NOTHING). Note: migrations don't auto-run on
supervisor restart -- see deferred ops list (schema_migrations cleanup
blocks central-migrate from running anything cleanly).
- src/central/gui/templates/_event_rows/avalanche_org.html (8 lines)
- src/central/gui/templates/_event_summaries/avalanche_org.html (2 lines)
Both required by the existing per-adapter template consistency tests.
Doc updates (required by existing doc-vs-registry tests):
- docs/PRODUCER-INTEGRATION.md §6.1: added 'avy' to top-level-domain list.
- docs/PRODUCER-INTEGRATION.md §8: added StreamEntry('CENTRAL_AVY',...) line
to the verbatim snippet.
- docs/CONSUMER-INTEGRATION.md §3 stream layout table: added CENTRAL_AVY row.
- docs/CONSUMER-INTEGRATION.md §6: new '### avalanche_org' subsection with
source, subject convention, dedup key, severity gate, Event.data field
table, and off-season behavior note.
- tests/test_events_feed_frontend.py: added avalanche_org to _SAMPLE_INNER
and _EXPECTED_SUBJECT (the events-JSON subject-derivation coverage tests).
Budget note: this PR is well over the ~400-line target -- the new-adapter
surface picked up downstream consistency tests (doc validators + frontend
sample coverage + template partials) I didn't anticipate at probe time.
Most of the overrun is the SNFAC fixture (1,135 lines pretty-printed JSON,
non-code) and the adapter + tests pair. Stripping the fixture and the
required doc/template edits would leave ~620 lines of code; the fixture
itself is a frozen snapshot, not a maintenance burden.
Full sweep: 1072 passed, 0 failures (+41 from this PR), ruff clean on
all new files. One PRE-EXISTING ruff violation in supervisor.py (unused
poll_start variable at line 388) surfaces when we touch supervisor.py;
confirmed not introduced by this PR via git stash check.
Deploy plan (NEW STREAM — archive restart required per
[[feedback_new_stream_needs_archive_restart]]):
1. Squash-merge -> tag v0.10.10 -> push.
2. On central: pull main -> systemctl restart central-supervisor -> ALSO
systemctl restart central-archive (new event-bearing stream; archive
enumerates consumers at startup and doesn't hot-reload).
3. Migration 035 deferred to morning per the schema_migrations cleanup
task -- the stream creation itself doesn't depend on it (supervisor
creates JetStream streams from the STREAMS registry at startup; the
config.streams row is for operator-tunable retention only).
4. Verify: nats stream info CENTRAL_AVY (created), poll log shows
yielded=0 / omitted=N (off-season), no positive publishes during
summer (correct).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First official-state-DOT-API pattern landing. Two adapters in one PR:
- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
to LineString via the polyline lib (bookend LineString or Point fallback);
ITD Severity string mapped None->1 / Minor->2 / Major->3 with
IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
Restrictions / DetourPolyline pass through unmodified. Advisories ship
as structural pass-through under data.advisory since the upstream
/alerts endpoint currently returns []; per-record try/except keeps a
surprise shape from sinking the cycle when ITD posts its first one.
- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
camera per UTC day; image URL passes straight through to <img src>.
Region uniform US-ID with data.source_jurisdiction preserving the raw
upstream Source field for the ~1.2% cross-DOT border-region mirrors
(UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).
Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.
Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.
Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.
Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).
supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.
- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
cams). The 100-row page cap also means v0.9.2 state_511_atis silently
truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
-- always shows today's cameras, no per-poll flooding, no retention
coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).
Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.
- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
falls back to the adapter's default_cadence_s. Existing settings without the
field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
(one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).
central-supervisor restart only. No gui/archive restart, no migration, no new dep.
Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.
central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.
- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
state_511_atis; cross-source overlap is by design = additive coverage, distinct
dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
key redacted from logs; poll skips cleanly without a key.
Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third CENTRAL_TRAFFIC-family member, first telemetry traffic source. Polls a
configured tile coverage set (Idaho metros, z=10), fetches Orbis vector flow
tiles, decodes per-segment relative_speed + road geometry, emits one telemetry
Event per road segment per poll to the new CENTRAL_TRAFFIC_FLOW stream. Renders
as colored polylines (green free-flow -> red jam) on the /telemetry map.
Production code; supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_flow.> -> archive must resubscribe). Ships disabled; needs a
"tomtom" api key in config.api_keys before enable.
- Subject central.traffic_flow.{z}.{x}.{y} (token traffic_flow, non-overlapping
with central.traffic.>). category="flow.tomtom_flow" -> GUI event_type "flow".
- Severity from relative_speed: >=0.75=1, 0.5-0.75=2, 0.25-0.5=3, <0.25=4.
- Cadence 300s; 7-day retention (high-volume telemetry). Dedup minute-bucketed,
inherited from the v0.9.1 SourceAdapter mixin.
- Shared tomtom_flow_parse module (decode + slippy-tile georeference) reused by
the v0.9.4 on-demand passthrough endpoint.
- Generic framework change (Option A, ~3 lines, inert for the other 14
adapters): Geo.geometry optional field + archive _build_geom_sql prefers it,
so segments persist their real LineString to the PostGIS geom column.
- Idaho-only (Orbis tier confirmed live). Cameras + Navi passthrough are follow-ups.
- deps: mapbox-vector-tile (vector PBF decode); itsdangerous promoted to an
explicit dependency (gui/csrf.py + gui/wizard.py imported it as an undeclared
transitive that uv re-lock would otherwise prune).
Full suite: 780 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second CENTRAL_TRAFFIC adapter. Production code; central-supervisor + central-gui
restart (new adapter class + ADAPTER_GROUPS). No new stream -> no archive restart;
migration 026 adds the adapter row only. Ships disabled.
Two-endpoint join per layer: GET /map/mapIcons/<Layer> (markers: itemId + coords)
joined on id with POST /List/GetData/<Layer> (DataTables detail: roadwayName,
description, county, severity). The marker feed has coords but no text; the List
feed has text but no coords.
Layers -> event_types (wzdx category/subject precedent): Incidents->incident,
Closures->closure, Construction (type "Roadwork")->work_zone. category is
"<event_type>.state_511_atis"; subject central.traffic.<event_type>.<state>.
Severity 3 if isFullClosure else 1. Cadence 300s. Dedup inherited from the
v0.9.1 SourceAdapter mixin. enrichment_locations canonical (latitude,longitude)
from the marker join; county/state come upstream.
Templatized per state via settings {"states":[{code,base_url}]} but ships
Idaho-only: cross-state spot-checks refuted the shared-URL hypothesis (Oregon
TripCheck is HTML, Wyoming wyoroad 404 -- neither is Castle Rock). Add states as
settings rows once each host is verified.
Also fixes a latent test bug: test_consumer_doc per-adapter heading regex was
[a-z_]+ (no digits); state_511_atis is the first adapter name with digits, so
widened to [a-z0-9_]+.
Full suite: 759 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two v0.9.0 fast-follows. Production code; central-supervisor + central-gui
restart (adapter base change + template change). No migration, no new stream.
(a) Work-zone subject + detail no longer leak vehicle direction "unknown"
(common in AZ mcdot etc.) -- gated on direction not in (None, "unknown") in both
wzdx partials. Was "Work zone on MORELAND ST unknown".
(b) is_published/mark_published/sweep_old_ids extracted from per-adapter inline
copies onto the SourceAdapter base (beside bump_last_seen); a dedup_sweep_days
class attr parameterizes the retention window (NWIS=30, default=14). Inline
copies deleted from inciweb/nwis/wzdx; the other 10 adapters keep theirs as a
future cleanup. Net dedup code down ~52 lines.
Full suite: 744 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opens Phase 4 transportation aggregation (Design B, Central-direct). New
registry-driven wzdx adapter polls the FHWA WZDx Feed Registry, fetches each
eligible v4.x GeoJSON feed concurrently, and emits work_zone events into the new
CENTRAL_TRAFFIC stream. Production code; central-supervisor AND central-gui
restart (new adapter class + stream + ADAPTER_GROUPS). Ships disabled.
First adapter to use the category/subject split: category="work_zone.wzdx" (GUI
event_type "work_zone" via split_part) while the NATS subject is
central.traffic.work_zone.{state}. Subject state from the registry row, geocoder
state as fallback. Severity from vehicle_impact (all-lanes-closed=3,
some-lanes-closed=2, all-lanes-open=1, unknown/missing=1). Feed filter
geojson + active + needapikey=false + version 4.x (21 of 39 feeds). 600s cadence.
Dedup composite <data_source_id>:<feature_id> in the shared cursors.db; stateless
discovery (no conftest isolation entry). enrichment_locations uses the canonical
("latitude","longitude") paths.
Full suite: 739 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).
- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
default "event"). NWIS opts in as "telemetry" (continuous high-volume water
gauges); every other adapter stays "event". The /events vs /telemetry split is
thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
to a class (colors stay keyed to the FULL registry, so an adapter keeps one
color across tabs). `_fetch_events` accepts `class_adapters` and adds an
`adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
`/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
paginator macro takes `base`. Same templates serve both tabs; nav gains a
Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
subject + pagination contract is intact (TestEventsJsonSubject still passes).
Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.
Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.
Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).
Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.
Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.
Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
- Add docs/test-database.md with one-time setup, DSN convention, reset
instructions, and explanation of why PostGIS is not in migrations
- Update docs/migrations.md with "Extensions are not in migrations"
section explaining superuser requirement
- Restore geom GEOMETRY(Geometry, 4326) column to test fixture now that
central_test has PostGIS installed
- Add CREATE EXTENSION IF NOT EXISTS postgis to test fixture for
self-bootstrap (central_test is superuser)
- Add Testing section to README.md pointing to docs/test-database.md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace pytest.skip stubs with actual DB tests against central_test
- Test backfill for all three adapters (nws, firms, usgs_quake)
- Test FK RESTRICT, NOT NULL, and FK validation constraints
- Test schema changes (source dropped, adapter exists with constraints)
- Delete stale sql/schema.sql (migrations are sole source of truth)
- Update docs/migrations.md with schema.sql removal note
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Stream retention GUI design
- Region picker for bbox selection
- API key management requirements
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents production verification of the AsyncLimiter removal fix:
- Decrease 60-30s: poll at Tlast+30s (not 60s)
- Increase 30-60s: poll at Tlast+60s
- Decrease 60-15s: immediate poll (deadline passed)
- All subsequent intervals use new cadence
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
environment.md:
- Documents CT104 as the active development location
- Lists SSH access, repository paths, and service commands
- Notes that cortex clone is parked, matt-desktop has no clones
BUG-CADENCE-DECREASE.md:
- Full investigation of the cadence-decrease hot-reload bug
- Root cause analysis: cancel_event.set() inside lock context
- Proposed fix (Option A - structural)
- Test gap identification
- Production verification steps
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase B operational cutover verification:
- Config source cutover from TOML to DB confirmed
- Hot-reload cadence test passed (rate-limit guarantee)
- Enable/disable cycle test passed (preserved_last_poll)
- 10-minute soak with zero errors
- Data integrity verified (all alerts in DB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
