Commit graph

247 commits

Author SHA1 Message Date
305ce5458a
v0.10.7: fix NWS SAME state-FIPS parse + 5-digit ANSI county form (#95) v0.10.7
SAME format (FCC standard) is PSSCCC: P=area-type indicator, SS=state FIPS,
CCC=county FIPS. Pre-v0.10.7 _build_regions read code[:2] (PS) as the state,
so the leading P=0 collapsed every SAME code to state 01 = Alabama. The
dumped CAP envelope from CENTRAL_WX#968 showed Bannock County Idaho alerts
flagged as US-AL-FIPS016005 + primary_region=US-AL-FIPS016005, which then
routed to central.wx.alert.us.al.county.fips016005 instead of
central.wx.alert.us.id.county.fips16005.

Fix:
- Slice code[1:3] for the state FIPS, code[1:] for the 5-digit ANSI county
  FIPS (SSCCC -- standard interoperable form). Drops the P padding from
  emitted region/subject; P stays preserved verbatim in data.geocode.SAME
  for any power user that needs it.
- Length guard tightened: ==6 + isdigit + isinstance str (was >= 2). Now
  malformed entries (too short, too long, non-digit, None) are silently
  skipped with no crash.
- Deleted dead _extract_states_from_codes (defined but never called; same
  bug, removed rather than fixed).

Tests:
- New TestSameStateParse parametrized over 4 distinct-state cases per spec:
  016005 -> US-ID-FIPS16005 (Bannock area), 001005 -> US-AL-FIPS01005
  (Autauga area), 056005 -> US-WY-FIPS56005 (Carbon area), 049005 ->
  US-UT-FIPS49005 (Cache UT).
- Area-subset (P>=1) and unknown-state-FIPS coverage.
- Malformed-input parametrize: empty, too short (2 forms), too long (7
  digits), non-digit char, all-alpha, None -- each silently skipped.
- Existing SAMPLE_FEATURE_* fixtures updated from constructed-to-match-bug
  values (160001/410051/060037/530033) to proper 0SSCCC format
  (016001/041051/006037/053033); existing TestBuildRegions assertions
  updated to expect 5-digit ANSI form.

Followup ticket (NOT v0.10.7 scope, recorded in PR body):
(a) Null-geometry alerts with valid Idaho UGC zones are silently dropped
    by _geometry_intersects_region (line 297-298): needs UGC-fallback or
    geometry-or-UGC check. NWS issues many Special Weather Statements
    without GeoJSON polygons but with UGC IDZ* zones that should pass.
(b) Configured monitoring bbox north=44.5 only covers the southern third
    of Idaho; Idaho extends to 49.0N, so Coeur d'Alene / Lewiston / etc.
    are out of scope. Verify whether the narrow bbox was an intentional
    dev limit or accidental.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-08 00:30:13 -06:00
e807750a72
v0.10.6: extract mile_marker from itd_511 comment field as _enriched.mile_marker (#94) v0.10.6
itd_511's free-text Comment field carries a milepost in roughly a third of
the live samples ('milepost 32.5', 'MP 80 to MP 81', etc.). meshai's roads
integration needs that as a structured field; wzdx and tomtom_incidents
already speak in structured mile-post / from-to so itd_511 is the only
adapter that needs the regex extraction layer.

Design (per Step-0 review):
- Shared module src/central/enrichment/mile_marker.py exporting
  extract(text) -> {value, source, confidence} | None. Pure regex, no I/O,
  re-usable by future per-state-DOT adapters (Wyoming, Montana, ...).
- itd_511 calls extract on the Comment in _build_event_record; result lands
  under the established _enriched namespace (NOT a new _enrichment one),
  keyed 'mile_marker'. Same convention the supervisor's geocoder uses, same
  merge semantics apply_enrichment already supports. Absent when no match
  (no null placeholder) so subscribers can tell 'not mentioned' from
  'extraction found nothing'.
- Confidence tiers: 'high' (single unambiguous MP/milepost/MM match),
  'medium' (multiple matches like 'MP 80 to MP 81' -- first wins), 'low'
  (bare 'mile N' only; consumers can ignore).

Tests:
- tests/test_enrichment_mile_marker.py: 36 cases parametrized over the 15
  real ITD comments I pulled from CENTRAL_TRAFFIC, including the critical
  red-herring classes the regex must reject (phone numbers, project key
  numbers, state-highway numbers, date/time numbers). Crafted samples
  cover M.P. / MM / milemarker / bare-mile patterns not in live ITD data
  but required by spec for future DOT adapters.
- tests/test_itd_511.py: 2 integration tests confirming the bundle is
  attached on a milepost-bearing Comment and absent otherwise.

Pure enrichment, no schema-breaking changes; meshai's renderer picks it up
additively.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-07 21:38:04 -06:00
b17d8bcd54
v0.10.5.2: fix BY_START_TIME feedback loop in Re-send (snapshot last_seq boundary) (#93) v0.10.5.2
The v0.10.5 ephemeral pull-consumer used DeliverPolicy.BY_START_TIME with
no upper bound, so every republish satisfied the same time filter and
the consumer fetched its own output back -- an unbounded loop bounded
only by the per-stream cap. Operator-triggered 5-minute resend on
2026-06-07 ran the loop long enough to time out central-gui's POST and
the host went down with it.

Fix: snapshot each event-bearing stream's last_seq up front via a new
_snapshot_last_seqs() helper, pass it to _iter_window as max_stream_seq,
and exit the generator the first time msg.metadata.sequence.stream
exceeds the snapshot. Pull-consumer delivery is stream-seq ascending so
one boundary check suffices.

Also drop _MAX_MSGS_PER_STREAM 50_000 -> 5_000 and add a WARNING log
when the cap is hit -- a legitimate operator window should never reach
it, and silent truncation hid the v0.10.5 loop until the host fell over.

Two regression tests cover the new behavior: one stages pre/post-snapshot
batches and asserts the post-snapshot batch is never yielded; one
overwhelms the cap and asserts the warning fires.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 22:36:04 -06:00
b490a4eec9
v0.10.5.1: fix inactive_threshold unit (seconds, not nanoseconds) — silent verification failure (#92) v0.10.5.1
Closes #92

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 19:52:17 -06:00
93f403a656
v0.10.5: dashboard Re-send recent events button with time-window selector (operator-controlled republish across all streams) (#91) v0.10.5
Closes #91

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 19:45:43 -06:00
7fa4f36e46
v0.10.3.1: soft-disable state_511_atis* adapters instead of DELETE (FK blocked v0.10.3 migration) (#90) v0.10.3.1
Closes #90

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 18:39:33 -06:00
2232718509
v0.10.4: switch wfigs_incidents to non-Current endpoint w/ WF active-only filter (resurrects IMT-managed fires like Blue Ridge) (#89) v0.10.4
Closes #89

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 18:10:16 -06:00
0dd83a340e
v0.10.3: rip out state_511_atis adapter (superseded by itd_511 v0.10.0; Castle Rock legacy shape EOL per sister-site discovery) (#88) v0.10.3
Closes #88

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 14:44:00 -06:00
557230c7a7
v0.10.2.1: drop broken incremental where-clause in wfigs adapters (use where=1=1) (#87) v0.10.2.1
Closes #87

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 01:59:45 -06:00
1bebf2570b
v0.10.2: monitoring-area bbox enforced at supervisor publish (was archive-only) (#PR_NUMBER_PLACEHOLDER) v0.10.2
Closes #86

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 20:34:10 -06:00
1d5548c24c
v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85) v0.10.0
First official-state-DOT-API pattern landing. Two adapters in one PR:

- itd_511 (event-class): polls Events (60s) + Advisories (300s) from
  https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline
  to LineString via the polyline lib (bookend LineString or Point fallback);
  ITD Severity string mapped None->1 / Minor->2 / Major->3 with
  IsFullClosure=true forcing 3 regardless; RecurrenceSchedules /
  Restrictions / DetourPolyline pass through unmodified. Advisories ship
  as structural pass-through under data.advisory since the upstream
  /alerts endpoint currently returns []; per-record try/except keeps a
  surprise shape from sinking the cycle when ITD posts its first one.

- itd_511_cameras (telemetry-class): polls Cameras (600s). One event per
  camera per UTC day; image URL passes straight through to <img src>.
  Region uniform US-ID with data.source_jurisdiction preserving the raw
  upstream Source field for the ~1.2% cross-DOT border-region mirrors
  (UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County).

Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id
and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis
keeps its bare-state subject; consumers stay on central.traffic.>
wildcards during the A/B comparison window.

Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx +
connection / timeout retry; 4xx other than 429 skip-with-warn (don't
burn quota on permanent errors); 429 honors Retry-After once then
retries. API key (alias 'idaho_511') travels in the ?key= query string,
so every error log path runs through self._redact() to scrub the URL.

Both adapters ship disabled; operator enables via GUI after registering
the API key with 'python -m set_api_key idaho_511'. Reuses existing
CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive
restart needed.

Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line
cap, authorized as a one-time exception for the first
official-state-DOT-API pattern landing. Two adapters + their tests +
real-API fixtures naturally exceed the v0.9.x adapter-cap budget.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-03 22:36:26 -06:00
1f7bccaac6
v0.9.21: FIRMS satellite pixel polygons via Geo.geometry (#84) v0.9.21
FIRMS hotspots rendered as tiny dots on /events because the adapter shipped
`bbox=(lon, lat, lon, lat)` and no `geometry`. `_build_geom_sql` collapsed
that into a 5-vertex zero-area `ST_Polygon`. Fix: construct a real rectangular
GeoJSON Polygon from `(latitude, longitude, scan, track)` per record and ship
it via `Geo.geometry`, mirroring the v0.9.3 pattern used by `tomtom_flow`.

- `_pixel_polygon()` builds a CCW 4-corner ring with the closing vertex
  duplicated. Cross-track (longitude) extent scales by `cos(lat)`. Latitude
  is pole-clamped to ±89° defensively.
- `_row_to_event` drops the degenerate bbox and sets `geo.geometry`;
  centroid stays alongside for consumers that read centroid only.
- `poll()` emits a single per-cycle warn if any record fell back to
  centroid-only Geo (missing scan/track).
- Forward-only at the geometry layer: same dedup keys, new payload shape
  for new records only. Existing PostGIS rows retain their degenerate
  polygons until republished or aged out (48h dedup window).
- Side-quest: removed a pre-existing dead RegionConfig import in
  tests/test_firms.py since the imports block was touched.

Tests: 5 new `_pixel_polygon` unit tests + 2 archive round-trip regression
guards mirroring `test_archive_prefers_geo_geometry`. Full suite 930 passed
/ 1 skipped under both `central` and `zvx` users.

CAVEAT — axis mapping convention: the `_pixel_polygon` docstring describes
`scan` as the along-track (latitude) extent and `track` as the cross-track
(longitude) extent. This matches the v0.9.21 prompt and is internally
consistent. NASA FIRMS convention may have these inverted — `scan` is often
documented as the cross-track scan width and `track` as the along-track
extent. If deploy-time visual inspection shows polygons rotated 90° from
expected, swap the assignments inside `_pixel_polygon`:

    half_scan_deg → multiply by cos(lat) (treat as longitude offset)
    half_track_deg → divide by 111.0 only (treat as latitude offset)

Clickability is the first-order goal and is unaffected — the polygon will
still have non-zero area in both axes either way; only the aspect ratio
flips. Tracked as a fast-follow if visual inspection requires it.

PR #84.
2026-06-03 20:51:31 -06:00
6ea7bd70f1
v0.9.20: regional subject routing on quake / fire / hydro / disaster adapters v0.9.20
* v0.9.20: regional subject routing on quake / fire / hydro / disaster adapters

Adds regional subject tokens to four adapters that previously published
without location-based routing:

- usgs_quake: central.quake.event.<tier>.us.<state> (US) or .<country> (intl)
- firms: central.fire.hotspot.<sat>.<conf>.us.<state> or .<country>
- nwis: central.hydro.<param>.<agency>.<site>.us.<state> (always US)
- eonet: central.disaster.eonet.<cat>.<country> (replaces hardcoded .global)

The us.<state> pattern (two tokens for US events) matches the NWS precedent
and resolves the ISO-2 collision between Idaho (id) and Indonesia.

New shared helper module: src/central/adapters/_subject_helpers.py
- US_STATE_NAME_TO_CODE: 50 states + DC + territories
- subject_for_country(): normalized country token
- subject_for_region(): returns us.<state>, <country>, or unknown

gdacs.py refactored to import subject_for_country from shared module.

Fixes: meshai v0.4 Phase C.3 bug (M4.1 Nevada quake routed globally)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* v0.9.20: stale docstring nit on renamed test

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-27 23:50:30 -06:00
f09f749052
Merge pull request #82 from zvx-echo6/v0_9_19_1_nws_sweep_scope v0.9.19.1
v0.9.19.1 - scope nws sweep_old_ids to its own adapter
2026-05-27 01:23:38 -06:00
Matt Johnson
5c6d77381b fix(nws): scope sweep_old_ids to its own adapter (v0.9.19.1)
NWSAdapter.sweep_old_ids ran an unscoped global DELETE FROM published_ids
(no adapter = ? filter), so each per-adapter sweep cycle purged EVERY
adapter dedup rows older than 8 days -- capping dedup memory at ~8d for the
8 adapters whose window exceeds it (eonet/gdacs/nwis 30d; swpc_kindex/
protons/alerts + wfigs_incidents/perimeters 14d). Events that went silent
past 8d then re-listed were re-published as downstream duplicates.

Extract the 3 dedup methods onto the SourceAdapter base (finishing v0.9.19;
is_published/mark_published were byte-identical) and preserve the 8-day
window via dedup_sweep_days = 8, so the inherited sweep scopes to adapter=?.
Effect: each adapter retains dedup to its configured window -> fewer
downstream duplicates. Retention-window scoping, NOT a dedup-forward-only
change (no event-shape change, no migration/backfill).

Adds a regression guard asserting a foreign adapter row survives an nws
sweep. Suite 900->901/1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 07:23:05 +00:00
195a544c36
Merge pull request #81 from zvx-echo6/v0_9_19_dedup_extraction v0.9.19
v0.9.19 - extract 9 adapters inline dedup onto SourceAdapter base
2026-05-27 01:13:26 -06:00
Matt Johnson
417c7abad6 refactor(adapters): extract inline dedup onto SourceAdapter base (v0.9.19)
Finishes the v0.9.1 dedup-mixin extraction (9cd2183) for the remaining 9
adapters. is_published/mark_published were byte-identical to the base;
sweep_old_ids windows preserved via dedup_sweep_days class attr where != 14
(usgs_quake=7, eonet=30, gdacs=30, firms=2 == old 48h). Pure behavior-
preserving refactor; suite holds 900/1.

nws left fully inline (unscoped global sweep) with a TODO(v0.9.19.1) marker
-- its fix carries a real behavior change, shipped separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 07:12:54 +00:00
7b13ea1886
Merge pull request #80 from zvx-echo6/v0_9_18_1_test_gui_ruff_cleanup v0.9.18.1
v0.9.18.1 — ruff cleanup: tests/test_gui_adapter_edit.py (3 violations -> 0)
2026-05-27 00:58:49 -06:00
Matt Johnson
0f59fe63ef tests: ruff cleanup in test_gui_adapter_edit.py (3 violations -> 0)
- F401: drop unused TOMTOM_FREE_TIER_CALLS_PER_MONTH import
- E702 x2: split semicolon-joined statements (lines 142, 202)

Mechanical only; zero behavior change. Suite holds at 900/1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 06:58:05 +00:00
3b75f33eaf
Merge pull request #79 from zvx-echo6/v0_9_18_migrate_reconcile v0.9.18
v0.9.18: reconcile schema_migrations drift + add --check drift detection
2026-05-27 00:47:21 -06:00
Matt Johnson
ce66ff9361 v0.9.18: reconcile schema_migrations drift + add --check drift detection
Migrations 025-029 (wzdx + traffic-family adapters/streams) were applied
out-of-band via direct psql during their deploys and never recorded in
schema_migrations. Result: a fresh restore would replay them, and an audit of
the tracking table understated what was actually live. All five are pure
additive INSERT ... ON CONFLICT DO NOTHING (verified idempotent by inspection),
so they were back-filled directly into schema_migrations (5 rows, applied_at =
now() = the reconciliation event; the original out-of-band apply dates are
unrecoverable and are documented as such rather than guessed).

Adds `central-migrate --check` to catch this class of drift going forward:
  - find_drift(): pure function comparing schema_migrations rows vs *.sql files,
    returning (untracked, orphan). Unit-tested, no DB dependency.
  - check_drift(): CI-assertion form -- exit 1 if any file is untracked, else 0.
  - log_drift(): WARNs per drifted entry, called on every migrate run too.

No migration 031 for the v0.9.17 wzdx state default: that default lives in
adapter code (_read_states falls back to _DEFAULT_STATES), so the live row's
explicit 7-state array is behaviorally identical to a fresh-install null.
Materializing it into SQL would freeze a snapshot of _DEFAULT_STATES and create
a new drift vector. Deferred to a future "show effective defaults" UI PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 06:40:38 +00:00
d41c418276
Merge pull request #78 from zvx-echo6/v0_9_17_wzdx_state_filter v0.9.17
WZDx: poll-time state allowlist with Idaho-region default (v0.9.17)
2026-05-27 00:00:51 -06:00
Matt Johnson
80460c83a8 WZDx: poll-time state allowlist with Idaho-region default (v0.9.17)
Narrow which WZDx registry feeds are fetched at poll time, so out-of-state
feeds are never requested (no bandwidth/quota cost), rather than fetching
nationally and dropping out-of-bounds events at the archive INSERT (v0.9.12).

- states is now list[StateCode] (Literal of 56 canonical 2-letter codes),
  default ["ID","WA","OR","NV","UT","WY","MT"]. Semantic flip: empty/unset/null
  now means the Idaho-region default, NOT "every eligible feed". Poll nationally
  by selecting all states in the GUI.
- list[Literal] renders as the existing checkboxes multi-select widget (firms.py
  precedent); pydantic rejects malformed codes at save time.
- Add informational quota_estimate (cap=None: never warns/blocks) so the edit
  page shows the fetch-volume reduction.
- adapters_edit.html: standalone quota panel for flat-config adapters, guarded
  by not has_model_list so model_list adapters don't double-render; reuses the
  identical quota.* keys + CSS so the v0.9.15 client recompute works for them too.
- 7 new tests (default/null fallback, _discover filtering, malformed rejection,
  quota math, offline Jinja panel render).

Event data shape is unchanged (poll-behavior-only), so no published_ids reset
is needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 05:57:57 +00:00
4fa7e031e3
Merge pull request #77 from zvx-echo6/v0_9_16_ruff_cleanup v0.9.16
v0.9.16: ruff baseline cleanup (routes.py, test_wfigs.py)
2026-05-26 22:57:18 -06:00
Matt Johnson
c379e3688e v0.9.16: ruff baseline cleanup (routes.py, test_wfigs.py)
Mechanical lint fixes only — ruff 31 -> 0 across the two files.

- E402 module-import-not-at-top: 15 -> 0 (relocated module-level
  `logger = logging.getLogger(...)` from mid-import to below the import
  block in routes.py; imports now contiguous at top of file)
- F811 redefined-while-unused: 10 -> 0 (removed 10 redundant function-local
  `from central.gui.csrf import reuse_or_generate_pre_auth_csrf` re-imports;
  the module-level import at line 26 is now load-bearing)
- F401 unused-import: 4 -> 0 (routes.py: fastapi.Depends, and
  central.gui.csrf.reuse_or_generate_pre_auth_csrf resolved by the F811 fix;
  test_wfigs.py: sqlite3, central.config_models.RegionConfig)
- E702 multiple-statements-on-one-line-semicolon: 2 -> 0 (split the two
  semicolon-joined statements in _fused_bbox, indentation preserved)

Deliberate function-local wizard imports (circular-import workaround) left
untouched. pytest: 890 passed / 1 skipped, unchanged across 3 runs.

no behavior change; ruff mechanical fixes only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 04:51:58 +00:00
eb97ffb24c
Merge pull request #76 from zvx-echo6/v0_9_15_live_quota v0.9.15
v0.9.15: live JS quota recompute on multi-bbox editor
2026-05-26 22:30:07 -06:00
Matt Johnson
f52e545ddf v0.9.15: live JS quota recompute on multi-bbox editor
Mirror TomTomIncidentsAdapter.quota_estimate in client-side JS so the
quota panel updates live as the operator edits a per-row cadence or the
adapter cadence, and on Add/Delete bbox -- instead of only on Save.

- tomtom_incidents.py: expose seconds_per_month + default_cadence_s in
  the quota dict (no JS magic numbers).
- model_list.html: data-quota-* attrs + .quota-detail/.quota-msg spans;
  self-contained DOMContentLoaded-guarded IIFE applying the same
  max(adapter_cadence, bbox_cadence||default) floor and 80%/100%
  warn/block thresholds. No-op when no quota panel present.
- tests: structural asserts for data-attrs + span hooks (JS exec needs
  a browser; live behaviour eyeballed manually).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 04:29:32 +00:00
91f8478f80
Merge pull request #75 from zvx-echo6/v0_9_14_fused_fire v0.9.14
v0.9.14: fused FIRMS+WFIGS fire view
2026-05-26 21:50:11 -06:00
Matt Johnson
d5367ff55e v0.9.14: fused FIRMS+WFIGS fire view
Pairs each live WFIGS perimeter with its nearby/contemporaneous FIRMS hotspots
into a single "fire" on the /events map. FIRMS hotspots carry no IrwinID, so the
link is spatial+temporal: a hotspot is confirmed (part of a known fire) when it
lies within 1km of a perimeter AND within 72h of it; hotspots matching no
perimeter render amber as "unconfirmed" -- a possible new fire detected by
satellite before an official perimeter exists (early-warning signal).

- routes.py: read-only /events/fire-fused.json (PostGIS ST_DWithin geography join)
- events_list.html: "Fuse fire layers" toggle (default on); centroid fire glyph
  that expands to polygon + hotspot dots on click; amber unconfirmed hotspots
- central.css: --fire-confirmed / --fire-unconfirmed vars (retune without code)
- 11 tests (shaping, bbox parse, R/T + bbox param wiring); spatial correctness
  verified on prod (Summit Creek: perimeter + 90 hotspots; 191 unconfirmed)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 03:49:30 +00:00
05b89df3a6
Merge pull request #74 from zvx-echo6/v0_9_13_events_retention v0.9.13
v0.9.13: per-stream archived-events retention sweep
2026-05-26 20:36:44 -06:00
Matt Johnson
9f67b4c1f2 v0.9.13: per-stream archived-events retention sweep
Hourly supervisor sweep deletes archived events older than their stream's
config.streams.max_age_s (the /streams 1/7/14/30/365d buttons -- the per-stream
source-of-truth). Explicit STREAM_CATEGORY_DOMAINS map (category domains do not
equal stream subject domains for the traffic family); fail-safe skip on
missing/<=0 max_age_s; unmapped-domain warning so a new adapter cannot silently
dodge retention.

TimescaleDB hypertable gotcha: events.ctid is chunk-local, NOT globally unique --
DELETE batching keys on the composite PK (id, time). See the inline NOTE in
config_store.delete_events_older_than and the PR body for the incident write-up.

- config_store: delete_events_older_than (batched on (id,time)) + unmapped_event_domains
- supervisor: STREAM_CATEGORY_DOMAINS, _sweep_events_retention, _events_retention_loop
- streams_list.html: max-age also governs archived-events retention
- 9 tests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 02:31:11 +00:00
3ff6f1ebc0
Merge pull request #73 from zvx-echo6/v0_9_12_archive_bbox_filter v0.9.12
v0.9.12: archive-level monitoring-area bbox filter
2026-05-26 17:40:59 -06:00
Matt Johnson
384d6118a6 v0.9.12: archive-level monitoring-area bbox filter
Add a single archive-level geographic filter at the events INSERT path: events
whose geometry falls entirely outside a system-configured monitoring area are
dropped before archival. Null-geom events (SWPC trio, .removed tombstones) are
always kept. Uses a pure shapely intersects() predicate so border-straddlers
are kept (matches ST_Intersects), and is fail-open on unparseable geometry.

- config.system gains monitor_{north,south,east,west} (migration 030, Idaho default)
- archive refreshes the bbox every 60s (no restart needed to change it); adds a
  per-adapter dropped-count counter, debug log per drop, cumulative INFO rollup
- new GUI editor at /monitoring-area (Leaflet draggable rectangle, N/S/E/W inputs)
- no adapter code changes; well-behaved adapters keep upstream API filtering
- 12 tests covering all five verdicts, drop-and-count, border/point-on-edge, fail-open

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 23:40:17 +00:00
c772d117d0
Merge pull request #72 from zvx-echo6/v0.9.11-hide-tombstones v0.9.11
Hide tombstones from default events view + show-removed toggle (v0.9.11)
2026-05-26 16:15:05 -06:00
Matt Johnson
85d0e8f1cc Hide tombstones from default events view + show-removed toggle (v0.9.11)
The /events feed was dominated by *.removed tombstone events (audit records
for features dropped from upstream feeds), burying geometry-bearing events
like fire perimeters (wfigs_perimeters: 54 real perimeters vs 1015 tombstones).
The GUI now default-hides any event whose category ends in .removed, with a
"Show removed" checkbox to restore them; URL state is preserved (HX-Push-Url)
so a shared link shows what the sharer saw. events.json is unchanged (still
returns tombstones) so API consumers are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 22:14:38 +00:00
92a1b3f2c6
Merge pull request #71 from zvx-echo6/v0.9.10-bbox-map v0.9.10
Read-only Leaflet map preview for multi-bbox editor (v0.9.10)
2026-05-26 12:43:54 -06:00
Matt Johnson
c5ed52db4b Read-only Leaflet map preview for multi-bbox editor (v0.9.10)
Adds a passive Leaflet map to the generic model_list editor that renders
every bbox row as a labeled translucent rectangle, auto-detected via the
min/max lon+lat sub-fields (TomTom incidents). Read-only: no drag/draw,
precise tuning stays in the per-row coord inputs. Rectangles redraw live
on input/add/remove; viewport fits only on initial render so typing never
jumps the map. Non-bbox model_lists (StateConfig, TileCoord) are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:43:08 +00:00
9ff8a33415
Merge PR #70: generic model_list adapter editor + TomTom bbox validation & quota (v0.9.9) v0.9.9
fix(gui): generic model_list editor — un-breaks 4 list-of-model adapter Edit pages + TomTom bbox validation & quota (v0.9.9)
2026-05-26 00:14:29 -06:00
Matt Johnson
7a5092c77f fix(gui): generic model_list editor for list-of-model adapters + TomTom bbox validation & quota (v0.9.9)
Fixes a shared form_descriptors 500 (NotImplementedError: unsupported list
type) that broke the Edit page for ALL FOUR adapters whose settings carry a
list[<BaseModel>] field: tomtom_incidents, tomtom_flow, state_511_atis,
state_511_atis_cameras.

- form_descriptors: list[BaseModel] -> generic "model_list" widget with
  recursive per-column sub_field descriptors.
- New _partials/model_list.html: vanilla-JS repeatable-row editor
  (add/remove/renumber), driven entirely by sub_fields (no adapter-name
  branching). Single-region edit pages render byte-identically.
- TomTom: BBox/Settings Pydantic validators (10,000 km^2 cap, coord ranges,
  min<max, cadence_s>=60, unique names) as the single source of truth
  (enforced at supervisor load AND GUI POST). Duck-typed quota_estimate hook
  + read-only quota panel; POST hard-blocks estimates over the 2,500/mo free
  tier (422). TOMTOM_FREE_TIER_CALLS_PER_MONTH is a tunable for paid tiers.
- routes: model_list form parse, row-aware ValidationError messages, 422 for
  model_list failures (single-region region errors still re-render at 200).
- tests: 11 new (real-Jinja render across 3 adapters + byte-identical nws
  no-regression guard, POST persist + oversized/degenerate/duplicate/cadence/
  quota 422 matrix, quota estimate). Full suite 848 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 05:57:34 +00:00
00a450b22f
Merge PR #69: wfigs_perimeters true polygons via Geo.geometry (v0.9.8) v0.9.8
feat(wfigs_perimeters): emit true polygons via Geo.geometry (v0.9.8)
2026-05-25 20:59:56 -06:00
Matt Johnson
382f744c12 feat(wfigs_perimeters): emit true polygons via Geo.geometry (v0.9.8)
Plumb the upstream GeoJSON Polygon/MultiPolygon into Geo.geometry so the
archive renders the real fire-perimeter shape (ST_GeomFromGeoJSON) instead
of the bbox 4-corner fallback. Adapter-local change; the v0.9.3 Geo.geometry
framework already carries it end-to-end. Graceful null + dict-or-string
coercion via _as_geometry; fall-off removal events stay NULL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 02:58:54 +00:00
8c78df7834
Merge PR #68: paginate state_511_atis /List/GetData (v0.9.7) v0.9.7
fix(state_511_atis): paginate /List/GetData to fix 100-row truncation (v0.9.7)
2026-05-25 20:46:20 -06:00
Matt Johnson
52b9ae0bbd fix(state_511_atis): paginate /List/GetData to fix 100-row truncation (v0.9.7)
Castle Rock caps each DataTables page at 100 rows regardless of `length`, so the
single-request fetcher silently dropped rows on any layer over 100 (confirmed
live: Construction recordsFiltered=114, returned 100 -> 14 rows invisible).
Backports the v0.9.6 cameras pagination loop into _fetch_details.

- _LIST_PAGE_LENGTH 1000 -> 100; new _list_body(start) builder; new @retry
  _fetch_page(base_url, layer, start).
- _fetch_details loops start+=100 until recordsFiltered collected or an empty
  page, with a _MAX_PAGES=50 ceiling that warns+breaks. Mid-pagination failure
  returns rows-so-far (retried next poll).
- Incidents (1) / Closures (29) are under 100 today but pagination applies
  uniformly; future-proof.

central-supervisor restart only (no stream, migration, template, or dep change).

Full suite: 833 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 02:40:58 +00:00
cc8847bec7
Merge pull request #67 from zvx-echo6/feat/state-511-cameras v0.9.6
feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6)
2026-05-25 19:34:35 -06:00
Matt Johnson
02bc692bda feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6)
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).

supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.

- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
  100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
  cams). The 100-row page cap also means v0.9.2 state_511_atis silently
  truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
  camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
  -- always shows today's cameras, no per-poll flooding, no retention
  coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
  source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
  shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).

Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 01:33:21 +00:00
d241bfea26
Merge pull request #66 from zvx-echo6/feat/tomtom-incidents-per-bbox-cadence v0.9.5.1
feat(tomtom_incidents): per-bbox cadence (v0.9.5.1)
2026-05-25 19:03:12 -06:00
Matt Johnson
7fdf47f2f0 feat(tomtom_incidents): per-bbox cadence (v0.9.5.1)
Lets each incident bbox poll at its own interval so busy metros refresh more
often than quiet corridors. Backward-compatible, code-only patch.

- Optional BBox.cadence_s (int | None = None) -> per-bbox poll interval; None
  falls back to the adapter's default_cadence_s. Existing settings without the
  field keep their current behavior.
- In-memory _last_polled {bbox_name: datetime}, per process. _bbox_due() gates
  fetches; poll() fetches only due bboxes. First poll after (re)start fetches all
  (one-shot catch-up; storage dedup on <state>:tomtom:<id> collapses overlap).
- _last_polled is recorded ONLY after a successful fetch -- a failed bbox stays
  due and retries next cycle (regression-guarded).
- Supervisor wakes the adapter at the adapter-level cadence; set that to the GCD
  of the per-bbox cadences for exact intervals (extra wakeups cost zero API calls).

central-supervisor restart only. No gui/archive restart, no migration, no new dep.

Full suite: 815 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 01:01:55 +00:00
0cb8f5a96b
Merge pull request #65 from zvx-echo6/feat/tomtom-incidents v0.9.5
feat(tomtom_incidents): TomTom real-time traffic incidents adapter (v0.9.5)
2026-05-25 18:26:45 -06:00
Matt Johnson
42d5faa80c feat(tomtom_incidents): TomTom real-time traffic incidents adapter (v0.9.5)
Fourth CENTRAL_TRAFFIC event adapter. Complements wzdx (federal work zones) and
state_511_atis (state-DOT reports) with TomTom commercial vehicle-telematics
coverage. Polls the Orbis incidentDetails endpoint per metro bbox, emits one
event per incident to central.traffic.incident.<state>. Ships disabled.

central-supervisor + central-gui restart only -- adapter row on the EXISTING
CENTRAL_TRAFFIC stream, so NO archive restart and no new stream/dependency.
Reuses the existing "tomtom" api key.

- Bbox limit refutation: incidentDetails rejects bbox > 10,000 km^2, so coverage
  is per-metro bboxes (Treasure Valley / Boise, 8,601 km^2), NOT statewide. One
  bbox @ 1800s = 1,440 calls/mo = 58% of the 2,500/mo free-tier cap. Expansion
  rows must respect N*(43200/cadence_min) <= 2500.
- category="incident.tomtom_incidents" -> GUI event_type "incident" (shared with
  state_511_atis; cross-source overlap is by design = additive coverage, distinct
  dedup ids + categories, no Central-side cross-source dedup).
- Severity from magnitudeOfDelay (0->1,1->1,2->2,3->3,4->4; 4=closure). Never None.
- geo.geometry carries TomTom's Point/LineString directly (already lon/lat GeoJSON;
  the v0.9.3 framework renders the affected road as a polyline). No decode needed.
- Dedup id <state_code>:tomtom:<tomtom_id> (upstream id stable across polls,
  verified 154/154 over 60s). Inherits the v0.9.1 dedup mixin.
- aiohttp params= URL-encodes the fields{} GraphQL braces (no curl-glob issue);
  key redacted from logs; poll skips cleanly without a key.

Full suite: 809 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 00:25:27 +00:00
a70b53572e
Merge pull request #64 from zvx-echo6/feat/tomtom-flow-passthrough v0.9.4
feat(tomtom_flow): Navi passthrough endpoint /api/traffic/flow (v0.9.4)
2026-05-25 18:05:22 -06:00