mirror of
https://github.com/zvx-echo6/central.git
synced 2026-06-10 11:54:37 +02:00
v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85)
First official-state-DOT-API pattern landing. Two adapters in one PR: - itd_511 (event-class): polls Events (60s) + Advisories (300s) from https://511.idaho.gov/api/v2/get/{event,alerts}. Decodes EncodedPolyline to LineString via the polyline lib (bookend LineString or Point fallback); ITD Severity string mapped None->1 / Minor->2 / Major->3 with IsFullClosure=true forcing 3 regardless; RecurrenceSchedules / Restrictions / DetourPolyline pass through unmodified. Advisories ship as structural pass-through under data.advisory since the upstream /alerts endpoint currently returns []; per-record try/except keeps a surprise shape from sinking the cycle when ITD posts its first one. - itd_511_cameras (telemetry-class): polls Cameras (600s). One event per camera per UTC day; image URL passes straight through to <img src>. Region uniform US-ID with data.source_jurisdiction preserving the raw upstream Source field for the ~1.2% cross-DOT border-region mirrors (UDOT / ODOT / WYDOT / WSDOT / NDot / MTD / DriveBC / Lemhi County). Subject convention (v0.9.20 forward): central.traffic.<event_type>.us.id and central.traffic_cameras.us.id.<camera_id>. Castle Rock state_511_atis keeps its bare-state subject; consumers stay on central.traffic.> wildcards during the A/B comparison window. Retry predicate tightened from the Castle Rock / TomTom precedent: 5xx + connection / timeout retry; 4xx other than 429 skip-with-warn (don't burn quota on permanent errors); 429 honors Retry-After once then retries. API key (alias 'idaho_511') travels in the ?key= query string, so every error log path runs through self._redact() to scrub the URL. Both adapters ship disabled; operator enables via GUI after registering the API key with 'python -m set_api_key idaho_511'. Reuses existing CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams -- no archive restart needed. Scope-cap exception: this PR is ~1.5k lines vs. the standard 500-line cap, authorized as a one-time exception for the first official-state-DOT-API pattern landing. Two adapters + their tests + real-API fixtures naturally exceed the v0.9.x adapter-cap budget. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
GitHub
parent
1f7bccaac6
commit
1d5548c24c
18 changed files with 1783 additions and 2 deletions
|
|
@ -1557,6 +1557,105 @@ conditions.
|
||||||
`lastUpdated` field is camera-config time), so the drawer shows no "captured at".
|
`lastUpdated` field is camera-config time), so the drawer shows no "captured at".
|
||||||
- **Removal semantics:** none; offline cameras serve an empty image but stay listed.
|
- **Removal semantics:** none; offline cameras serve an empty image but stay listed.
|
||||||
|
|
||||||
|
### itd_511 — Idaho 511 official DOT API (events + advisories)
|
||||||
|
|
||||||
|
Idaho Transportation Department's official 511 REST API. Polls roadwork,
|
||||||
|
closures, incidents, special events, and advisories statewide. First
|
||||||
|
official-state-DOT-API adapter (v0.10.0); runs in parallel with state_511_atis
|
||||||
|
(Castle Rock) post-deploy for A/B comparison. Idaho-only; subject suffix is
|
||||||
|
uniformly `us.id`.
|
||||||
|
|
||||||
|
- **Stream:** `CENTRAL_TRAFFIC` (event class). **event_type:** one of
|
||||||
|
`work_zone`, `closure`, `incident`, `special_event`, `advisory` (from
|
||||||
|
`category = "<event_type>.itd_511"`).
|
||||||
|
- **Subject pattern:** `central.traffic.<event_type>.us.id` (v0.9.20 forward
|
||||||
|
convention -- `us.id` is the ISO-3166-2 region suffix; subscribe to all ITD
|
||||||
|
events via `central.traffic.>.us.id` or per-type via the leading wildcard).
|
||||||
|
- **Source:** `GET /api/v2/get/event?key=<KEY>` (events, **cadence 60s**) +
|
||||||
|
`GET /api/v2/get/alerts?key=<KEY>` (advisories, **cadence 300s** -- every
|
||||||
|
5th poll). API key required (alias `idaho_511`). Documented limit 10 calls
|
||||||
|
/ 60s; combined load ~1.3 calls/min.
|
||||||
|
- **EventType mapping:** `roadwork`->`work_zone`, `closures`->`closure`,
|
||||||
|
`accidentsAndIncidents`->`incident`, `specialEvents`->`special_event`.
|
||||||
|
Advisories always emit `event_type=advisory`.
|
||||||
|
- **Dedup key shape:** `idaho_511:event:<SourceId>` (events) /
|
||||||
|
`idaho_511:advisory:<id>` (advisories). SourceId is the upstream-allocated
|
||||||
|
stable id; the ITD-internal `ID` is used as fallback.
|
||||||
|
- **Geometry:** decoded EncodedPolyline (Google polyline format) -> LineString;
|
||||||
|
falls back to bookend LineString (Latitude,Longitude + Latitude2,Longitude2)
|
||||||
|
-> single Point. Shipped via `geo.geometry` so PostGIS renders the affected
|
||||||
|
segment as a polyline on the map.
|
||||||
|
- **Severity:** ITD `Severity` (string) mapped `None`->1, `Minor`->2, `Major`->3.
|
||||||
|
`IsFullClosure=true` forces severity 3 regardless (orthogonal upstream
|
||||||
|
signals in the live data -- 15 of 152 None-severity events were full
|
||||||
|
closures at landing).
|
||||||
|
- **Advisories:** structural pass-through under `data.advisory`. The upstream
|
||||||
|
`/alerts` endpoint returned `[]` at adapter landing; the parser probes a few
|
||||||
|
likely id / timestamp / coord fields best-effort and stores the entire
|
||||||
|
record so v0.10.x can refine field mapping once a real advisory lands.
|
||||||
|
- **Event.data fields (events):**
|
||||||
|
|
||||||
|
| key | type | nullable | description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `event_type_short` | str | no | One of work_zone / closure / incident / special_event / advisory |
|
||||||
|
| `event_sub_type` | str | yes | Rich vocabulary, e.g. `bridgeConstruction`, `nightTimeConstructionWork` |
|
||||||
|
| `roadway_name` | str | yes | e.g. `I-84`, `SH-16` |
|
||||||
|
| `direction` | str | yes | `East` / `Both` / `Unknown` (Unknown suppressed in L-c text) |
|
||||||
|
| `description` / `comment` | str | yes | Operator text |
|
||||||
|
| `lanes_affected` | str | yes | e.g. `2 Left Lanes Blocked` |
|
||||||
|
| `is_full_closure` | bool | no | Drives the severity-3 override |
|
||||||
|
| `itd_severity` | str | yes | Raw ITD value (`Major` / `Minor` / `None`) |
|
||||||
|
| `cause` | str | yes | Usually mirrors EventType (`roadwork`, `Incident`, `specialEvents`) |
|
||||||
|
| `organization` | str | yes | Uniformly `ERS` at landing |
|
||||||
|
| `recurrence_text` | str | yes | HTML schedule (consumer should `striptags`) |
|
||||||
|
| `recurrence_schedules` | list | yes | Structured `[{StartDate, EndDate, Times, DaysOfWeek}]` |
|
||||||
|
| `restrictions` | dict | yes | `{Width, Height, Length, Weight, Speed}` (often all null) |
|
||||||
|
| `detour_polyline` / `detour_instructions` | str | yes | Detour geometry + text |
|
||||||
|
| `encoded_polyline` | str | yes | Raw EncodedPolyline (also decoded into geo.geometry) |
|
||||||
|
| `id_internal` / `source_id` | int / str | no | ITD-internal id + upstream-stable SourceId |
|
||||||
|
| `reported_epoch` / `last_updated_epoch` / `start_epoch` / `planned_end_epoch` | int | yes | Unix epoch (UTC); `Event.time` uses LastUpdated -> Reported -> StartDate priority |
|
||||||
|
| `latitude` / `longitude` | float | yes | Primary point (enrichment input) |
|
||||||
|
|
||||||
|
### itd_511_cameras — Idaho 511 official DOT API cameras (telemetry)
|
||||||
|
|
||||||
|
Idaho Transportation Department's traffic camera directory. One telemetry
|
||||||
|
event per camera per UTC day; the `/telemetry` detail drawer renders the live
|
||||||
|
image inline (`<img>` fetched direct from the source -- Central stores the URL,
|
||||||
|
never the image bytes). Sibling adapter of itd_511 (shared API key alias
|
||||||
|
`idaho_511`).
|
||||||
|
|
||||||
|
- **Stream:** `CENTRAL_TRAFFIC_CAMERAS` (telemetry; `/telemetry` tab).
|
||||||
|
- **Subject pattern:** `central.traffic_cameras.us.id.<camera_id>` -- subscribe
|
||||||
|
to one camera or `central.traffic_cameras.us.id.>` for all ITD cameras.
|
||||||
|
- **GUI event_type:** `camera` (from `category = "camera.itd_511_cameras"`).
|
||||||
|
- **Source:** `GET /api/v2/get/cameras?key=<KEY>`. 664 cameras at landing;
|
||||||
|
**cadence 600s**. ITD aggregates ~1.2% border-region mirrors from
|
||||||
|
neighbouring DOTs (UDOT, ODOT, WYDOT, WSDOT, NDot, MTD, DriveBC, Lemhi
|
||||||
|
County). All cameras are tagged region `US-ID`; `data.source_jurisdiction`
|
||||||
|
preserves the raw upstream `Source` so consumers can re-bucket the cross-DOT
|
||||||
|
mirrors if needed.
|
||||||
|
- **Dedup key shape:** `idaho_511:cam:<camera_id>:<YYYY-MM-DD>` -- one event
|
||||||
|
per camera per UTC day. The table always shows today's cameras; no per-poll
|
||||||
|
flooding.
|
||||||
|
- **Image URL:** `https://511.idaho.gov/map/Cctv/<view_id>` -- publicly
|
||||||
|
reachable, no auth. Format may be jpeg / gif / png (mixed per camera);
|
||||||
|
`<img>` handles all.
|
||||||
|
- **Event.data fields:**
|
||||||
|
|
||||||
|
| key | type | nullable | description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `camera_id` | int | no | Stable upstream id |
|
||||||
|
| `roadway` | str | yes | e.g. `I-84`, `I-15` |
|
||||||
|
| `direction` | str | yes | `North` / `Unknown` (Unknown suppressed in row partial) |
|
||||||
|
| `location` | str | yes | Humanized (e.g. `I-15 UT/ID State Line UT`) |
|
||||||
|
| `source` / `source_jurisdiction` | str | yes | ITDNET / ACHD / RWIS / Idaho511 / UDOT / ODOT / WYDOT / ... |
|
||||||
|
| `source_id_upstream` | str | yes | Source-specific id (e.g. UDOT `10.C1`) |
|
||||||
|
| `image_url` | str | yes | First Views[].Url (live image; jpeg/gif/png) |
|
||||||
|
| `additional_views` | list[str] | no | URLs of Views[1:] when a camera has multiple angles |
|
||||||
|
| `view_count` / `view_descriptions` | int / list[str] | no / no | Total views + per-view labels |
|
||||||
|
| `sort_order` | int | yes | Upstream display order hint |
|
||||||
|
| `latitude` / `longitude` | float | yes | Camera location (enrichment input) |
|
||||||
|
|
||||||
### tomtom_incidents — TomTom real-time traffic incidents (commercial coverage)
|
### tomtom_incidents — TomTom real-time traffic incidents (commercial coverage)
|
||||||
|
|
||||||
Real-time incidents (closures, jams, hazards, road work, accidents) from TomTom
|
Real-time incidents (closures, jams, hazards, road work, accidents) from TomTom
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,7 @@ dependencies = [
|
||||||
"fastapi>=0.115.0",
|
"fastapi>=0.115.0",
|
||||||
"jinja2>=3.1.6",
|
"jinja2>=3.1.6",
|
||||||
"nats-py>=2.14.0",
|
"nats-py>=2.14.0",
|
||||||
|
"polyline>=2.0,<3", # itd_511: Google polyline decode (EncodedPolyline → LineString)
|
||||||
"pydantic>=2,<3",
|
"pydantic>=2,<3",
|
||||||
"pydantic-settings>=2.7.0",
|
"pydantic-settings>=2.7.0",
|
||||||
"python-multipart>=0.0.20",
|
"python-multipart>=0.0.20",
|
||||||
|
|
|
||||||
28
sql/migrations/031_add_itd_511_adapters.sql
Normal file
28
sql/migrations/031_add_itd_511_adapters.sql
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
-- Migration: 031_add_itd_511_adapters
|
||||||
|
-- Adds the itd_511 (event-class) and itd_511_cameras (telemetry-class) adapters
|
||||||
|
-- onto the EXISTING CENTRAL_TRAFFIC and CENTRAL_TRAFFIC_CAMERAS streams. NO
|
||||||
|
-- new streams -> no central-archive restart needed
|
||||||
|
-- (feedback_new_stream_needs_archive_restart). Both ship disabled; operator
|
||||||
|
-- enables via GUI after registering the 'idaho_511' API key
|
||||||
|
-- (python -m set_api_key, alias 'idaho_511').
|
||||||
|
-- itd_511 polls 60s (events) + 300s (advisories sub-poll); ~1.3 calls/min
|
||||||
|
-- combined, well under ITD's documented 10/60s budget.
|
||||||
|
-- Additive-only: idempotent via ON CONFLICT DO NOTHING.
|
||||||
|
|
||||||
|
INSERT INTO config.adapters (name, enabled, cadence_s, settings)
|
||||||
|
VALUES (
|
||||||
|
'itd_511',
|
||||||
|
false,
|
||||||
|
60,
|
||||||
|
'{"api_key_alias": "idaho_511"}'::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (name) DO NOTHING;
|
||||||
|
|
||||||
|
INSERT INTO config.adapters (name, enabled, cadence_s, settings)
|
||||||
|
VALUES (
|
||||||
|
'itd_511_cameras',
|
||||||
|
false,
|
||||||
|
600,
|
||||||
|
'{"api_key_alias": "idaho_511"}'::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (name) DO NOTHING;
|
||||||
440
src/central/adapters/itd_511.py
Normal file
440
src/central/adapters/itd_511.py
Normal file
|
|
@ -0,0 +1,440 @@
|
||||||
|
"""ITD 511 traffic adapter — Idaho DOT official REST API (events + advisories).
|
||||||
|
|
||||||
|
Polls /api/v2/get/event (60s) and /api/v2/get/alerts (300s = every 5th poll)
|
||||||
|
from https://511.idaho.gov per v0.10.0. Sibling adapter ``itd_511_cameras``
|
||||||
|
(data_class="telemetry") handles /get/cameras. Both ship to existing streams
|
||||||
|
(CENTRAL_TRAFFIC / CENTRAL_TRAFFIC_CAMERAS) — no archive restart needed.
|
||||||
|
|
||||||
|
EncodedPolyline (Google polyline format) decodes to a LineString via the
|
||||||
|
``polyline`` PyPI dep; falls back to (Latitude, LatitudeSecondary) bookend
|
||||||
|
LineString or a single Point when geometry is sparse. RecurrenceSchedules,
|
||||||
|
Restrictions, and DetourPolyline pass through unmodified for downstream
|
||||||
|
consumers (truckers, route planners).
|
||||||
|
|
||||||
|
ITD Severity is a string ("Major"/"Minor"/"None") not 0-3 as the v0.10.0 spec
|
||||||
|
initially assumed; mapping is None->1, Minor->2, Major->3 with IsFullClosure
|
||||||
|
=true forcing 3 regardless (Severity and full-closure are orthogonal upstream
|
||||||
|
signals — 15 of 152 "None"-severity events are full closures in the live data).
|
||||||
|
|
||||||
|
Subject convention (v0.9.20 forward, locked v0.10.0 architectural call A):
|
||||||
|
central.traffic.<event_type>.us.id
|
||||||
|
|
||||||
|
where event_type ∈ {incident, closure, work_zone, special_event, advisory}.
|
||||||
|
|
||||||
|
Dedup id: ``idaho_511:event:<SourceId>`` (SourceId is upstream-allocated, more
|
||||||
|
stable across ITD-side restamps than the internal ID; falls back to ID if
|
||||||
|
SourceId is null, never happened in the v0.10.0 step-0 snapshot).
|
||||||
|
|
||||||
|
Retry predicate tightened from the Castle Rock/TomTom precedent (architectural
|
||||||
|
call B): 5xx + connection/timeout retry; 4xx other than 429 skip the poll with
|
||||||
|
a warn (don't burn quota on permanent errors like a bad key); 429 honors
|
||||||
|
Retry-After once then retries.
|
||||||
|
|
||||||
|
API key (alias 'idaho_511') travels in the ?key=<KEY> query string — aiohttp's
|
||||||
|
default error messages leak the URL, so every error log path runs through
|
||||||
|
``self._redact()``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import sqlite3
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
import polyline as polyline_lib
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from tenacity import (
|
||||||
|
after_nothing,
|
||||||
|
retry,
|
||||||
|
retry_if_exception_type,
|
||||||
|
stop_after_attempt,
|
||||||
|
wait_exponential_jitter,
|
||||||
|
)
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
from central.config_store import ConfigStore
|
||||||
|
from central.models import Event, Geo
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_BASE_URL = "https://511.idaho.gov/api/v2/get"
|
||||||
|
_FETCH_TIMEOUT_S = 30
|
||||||
|
_MAX_RETRY_AFTER_S = 60 # cap on Retry-After honor; longer => skip the cycle
|
||||||
|
_ADVISORY_EVERY_N_POLLS = 5 # 60s * 5 = 300s effective cadence
|
||||||
|
|
||||||
|
_DEDUP_DDL = (
|
||||||
|
"CREATE TABLE IF NOT EXISTS published_ids ("
|
||||||
|
"adapter TEXT NOT NULL, event_id TEXT NOT NULL, "
|
||||||
|
"first_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"last_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"PRIMARY KEY (adapter, event_id))"
|
||||||
|
)
|
||||||
|
|
||||||
|
# ITD EventType -> Central event_type prefix (category = "<prefix>.itd_511",
|
||||||
|
# subject = "central.traffic.<prefix>.us.id"). Advisories use "advisory".
|
||||||
|
EVENT_TYPE_MAP = {
|
||||||
|
"roadwork": "work_zone",
|
||||||
|
"closures": "closure",
|
||||||
|
"accidentsAndIncidents": "incident",
|
||||||
|
"specialEvents": "special_event",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ITD Severity (string) -> Central 1-4. IsFullClosure=true forces 3 regardless
|
||||||
|
# (orthogonal upstream signals; 15/152 "None"-severity events are full closures).
|
||||||
|
_SEVERITY_MAP = {"None": 1, "Minor": 2, "Major": 3}
|
||||||
|
|
||||||
|
|
||||||
|
class _Transient(Exception):
|
||||||
|
"""Internal: retryable. 5xx or 429 (with optional Retry-After honor).
|
||||||
|
|
||||||
|
``wait_s`` carries an explicit wait override the retry strategy reads,
|
||||||
|
so a documented Retry-After value drives the next attempt's delay
|
||||||
|
directly. Fixes the BUG C double-wait: the previous shape did an inline
|
||||||
|
``await asyncio.sleep(wait_s)`` AND then let tenacity's exponential
|
||||||
|
jitter wait again on top, blocking ~120s+ per cycle for a single 429
|
||||||
|
with Retry-After=60.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, msg: str, *, wait_s: int | None = None) -> None:
|
||||||
|
super().__init__(msg)
|
||||||
|
self.wait_s = wait_s
|
||||||
|
|
||||||
|
|
||||||
|
_EXP_WAIT = wait_exponential_jitter(initial=1, max=30)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_strategy(retry_state: Any) -> float:
|
||||||
|
"""Tenacity wait callable. Honors ``_Transient.wait_s`` when set (429
|
||||||
|
Retry-After path); else falls through to exponential jitter. Shared
|
||||||
|
across itd_511 and itd_511_cameras via direct import."""
|
||||||
|
if retry_state.outcome is not None:
|
||||||
|
exc = retry_state.outcome.exception()
|
||||||
|
if isinstance(exc, _Transient) and exc.wait_s is not None:
|
||||||
|
return float(exc.wait_s)
|
||||||
|
return float(_EXP_WAIT(retry_state))
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_epoch(value: Any) -> datetime | None:
|
||||||
|
"""Unix epoch seconds -> UTC datetime; None on missing/non-numeric/out-of-range."""
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return datetime.fromtimestamp(int(value), tz=timezone.utc)
|
||||||
|
except (ValueError, OSError, TypeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_or_none(value: Any) -> Any:
|
||||||
|
"""Strip a string (handles EventSubType trailing-space artifact); '' -> None.
|
||||||
|
Non-strings pass through unchanged so the call site can stay terse."""
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return value
|
||||||
|
stripped = value.strip()
|
||||||
|
return stripped or None
|
||||||
|
|
||||||
|
|
||||||
|
def _itd_severity(itd_sev: str | None, is_full_closure: bool) -> int:
|
||||||
|
"""ITD Severity + IsFullClosure -> Central 1-4 (None->1, Minor->2, Major->3,
|
||||||
|
full-closure overrides to 3 regardless)."""
|
||||||
|
if is_full_closure:
|
||||||
|
return 3
|
||||||
|
return _SEVERITY_MAP.get(itd_sev or "", 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_polyline(encoded: str | None) -> list[tuple[float, float]]:
|
||||||
|
"""Decode a Google polyline -> list of (lat, lon). [] on null/empty/invalid."""
|
||||||
|
if not encoded:
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
return polyline_lib.decode(encoded)
|
||||||
|
except Exception: # the lib raises a grab-bag of unicode/index errors on bad input
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _build_geometry(
|
||||||
|
lat: float | None, lon: float | None,
|
||||||
|
lat2: float | None, lon2: float | None,
|
||||||
|
encoded: str | None,
|
||||||
|
) -> tuple[dict[str, Any] | None, tuple[float, float] | None]:
|
||||||
|
"""Pick the best geometry available -> (GeoJSON geometry, centroid lon/lat).
|
||||||
|
|
||||||
|
Priority: decoded EncodedPolyline LineString -> bookend LineString
|
||||||
|
(primary + secondary lat/lon) -> single Point. Centroid is the first vertex.
|
||||||
|
"""
|
||||||
|
decoded = _decode_polyline(encoded)
|
||||||
|
if len(decoded) >= 2:
|
||||||
|
coords = [(lon_, lat_) for lat_, lon_ in decoded]
|
||||||
|
return {"type": "LineString", "coordinates": coords}, coords[0]
|
||||||
|
if lat is not None and lon is not None:
|
||||||
|
if lat2 is not None and lon2 is not None:
|
||||||
|
return ({"type": "LineString", "coordinates": [(lon, lat), (lon2, lat2)]},
|
||||||
|
(lon, lat))
|
||||||
|
return {"type": "Point", "coordinates": (lon, lat)}, (lon, lat)
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
class Itd511Settings(BaseModel):
|
||||||
|
"""api_key_alias: config.api_keys entry (default 'idaho_511', shared with
|
||||||
|
itd_511_cameras). ITD is Idaho-only — no bbox/state list; region tagged
|
||||||
|
uniformly US-ID."""
|
||||||
|
|
||||||
|
api_key_alias: str = "idaho_511"
|
||||||
|
|
||||||
|
|
||||||
|
class Itd511Adapter(SourceAdapter):
|
||||||
|
"""Idaho 511 official API — events + advisories."""
|
||||||
|
|
||||||
|
name = "itd_511"
|
||||||
|
display_name = "Idaho 511 (Official API)"
|
||||||
|
description = (
|
||||||
|
"Idaho Transportation Department's official 511 REST API. Polls roadwork, "
|
||||||
|
"closures, incidents, special events, and advisories statewide. Geometry "
|
||||||
|
"via decoded EncodedPolyline; ships to CENTRAL_TRAFFIC."
|
||||||
|
)
|
||||||
|
settings_schema = Itd511Settings
|
||||||
|
requires_api_key = "idaho_511"
|
||||||
|
api_key_field = "api_key_alias"
|
||||||
|
wizard_order = None # Ships disabled
|
||||||
|
default_cadence_s = 60
|
||||||
|
data_class = "event"
|
||||||
|
enrichment_locations = [("latitude", "longitude")]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
config: AdapterConfig,
|
||||||
|
config_store: ConfigStore,
|
||||||
|
cursor_db_path: Path,
|
||||||
|
) -> None:
|
||||||
|
self._config_store = config_store
|
||||||
|
self._cursor_db_path = cursor_db_path
|
||||||
|
self._session: aiohttp.ClientSession | None = None
|
||||||
|
self._db: sqlite3.Connection | None = None
|
||||||
|
self._api_key_alias: str = config.settings.get("api_key_alias", "idaho_511")
|
||||||
|
self._api_key: str | None = None
|
||||||
|
self._adv_counter = 0 # advisories polled on counter == 0
|
||||||
|
|
||||||
|
def _redact(self, text: str) -> str:
|
||||||
|
return text.replace(self._api_key, "<KEY>") if self._api_key else text
|
||||||
|
|
||||||
|
async def startup(self) -> None:
|
||||||
|
self._session = aiohttp.ClientSession(
|
||||||
|
timeout=aiohttp.ClientTimeout(total=_FETCH_TIMEOUT_S),
|
||||||
|
headers={"User-Agent": "Central/0.10 (+itd_511)"},
|
||||||
|
)
|
||||||
|
self._db = sqlite3.connect(self._cursor_db_path)
|
||||||
|
self._db.execute(_DEDUP_DDL)
|
||||||
|
self._db.execute("CREATE INDEX IF NOT EXISTS published_ids_last_seen ON published_ids (last_seen)")
|
||||||
|
self._db.commit()
|
||||||
|
self._api_key = await self._config_store.get_api_key(self._api_key_alias)
|
||||||
|
logger.info("itd_511 adapter started",
|
||||||
|
extra={"alias": self._api_key_alias, "api_key_present": bool(self._api_key)})
|
||||||
|
|
||||||
|
async def shutdown(self) -> None:
|
||||||
|
if self._session:
|
||||||
|
await self._session.close()
|
||||||
|
self._session = None
|
||||||
|
if self._db:
|
||||||
|
self._db.close()
|
||||||
|
self._db = None
|
||||||
|
|
||||||
|
async def apply_config(self, new_config: AdapterConfig) -> None:
|
||||||
|
self._api_key_alias = new_config.settings.get("api_key_alias", "idaho_511")
|
||||||
|
self._api_key = await self._config_store.get_api_key(self._api_key_alias)
|
||||||
|
logger.info("itd_511 config updated",
|
||||||
|
extra={"alias": self._api_key_alias, "api_key_present": bool(self._api_key)})
|
||||||
|
|
||||||
|
@retry(
|
||||||
|
stop=stop_after_attempt(3),
|
||||||
|
wait=_wait_strategy,
|
||||||
|
retry=retry_if_exception_type((_Transient, aiohttp.ClientConnectionError,
|
||||||
|
asyncio.TimeoutError, TimeoutError)),
|
||||||
|
reraise=True, # propagate _Transient/ClientError verbatim, not RetryError
|
||||||
|
before_sleep=None, # explicit per BUG D5 audit: tenacity must not log between attempts
|
||||||
|
after=after_nothing, # explicit per BUG D5 audit: tenacity must not log after attempts
|
||||||
|
)
|
||||||
|
async def _fetch(self, endpoint: str) -> list[dict[str, Any]] | None:
|
||||||
|
"""GET /api/v2/get/<endpoint>?key=<KEY>. Returns the list, or None on a
|
||||||
|
permanent 4xx (skip cycle, don't retry). Raises _Transient on 5xx/429
|
||||||
|
for tenacity to retry; the 429 path attaches the Retry-After value via
|
||||||
|
``_Transient(wait_s=...)`` so ``_wait_strategy`` honors it directly
|
||||||
|
(no in-handler sleep — fixes BUG C double-wait). Key is redacted in
|
||||||
|
every error log."""
|
||||||
|
if self._session is None:
|
||||||
|
raise RuntimeError("itd_511 session not started")
|
||||||
|
async with self._session.get(
|
||||||
|
f"{_BASE_URL}/{endpoint}", params={"key": self._api_key},
|
||||||
|
) as resp:
|
||||||
|
if resp.status == 200:
|
||||||
|
return await resp.json(content_type=None)
|
||||||
|
body_preview = self._redact((await resp.text())[:200])
|
||||||
|
if resp.status == 429:
|
||||||
|
ra = resp.headers.get("Retry-After", "")
|
||||||
|
wait_s = min(int(ra) if ra.isdigit() else 5, _MAX_RETRY_AFTER_S)
|
||||||
|
logger.warning("itd_511 rate-limited",
|
||||||
|
extra={"endpoint": endpoint, "retry_after": wait_s, "body": body_preview})
|
||||||
|
raise _Transient(f"429 retry-after={wait_s}", wait_s=wait_s)
|
||||||
|
if 500 <= resp.status < 600:
|
||||||
|
logger.warning("itd_511 upstream 5xx",
|
||||||
|
extra={"endpoint": endpoint, "status": resp.status, "body": body_preview})
|
||||||
|
raise _Transient(f"{resp.status} server error")
|
||||||
|
logger.warning("itd_511 permanent error; skipping endpoint",
|
||||||
|
extra={"endpoint": endpoint, "status": resp.status, "body": body_preview})
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _build_event_record(self, rec: dict[str, Any]) -> Event | None:
|
||||||
|
"""Build an Event from one /get/event row. Returns None without a stable id."""
|
||||||
|
source_id = rec.get("SourceId") or rec.get("ID")
|
||||||
|
if source_id is None:
|
||||||
|
return None
|
||||||
|
et = EVENT_TYPE_MAP.get(rec.get("EventType") or "", "incident")
|
||||||
|
lat, lon = rec.get("Latitude"), rec.get("Longitude")
|
||||||
|
geom, centroid = _build_geometry(
|
||||||
|
lat, lon, rec.get("LatitudeSecondary"), rec.get("LongitudeSecondary"),
|
||||||
|
rec.get("EncodedPolyline"),
|
||||||
|
)
|
||||||
|
return Event(
|
||||||
|
id=f"idaho_511:event:{source_id}",
|
||||||
|
adapter=self.name,
|
||||||
|
category=f"{et}.itd_511",
|
||||||
|
time=(_parse_epoch(rec.get("LastUpdated"))
|
||||||
|
or _parse_epoch(rec.get("Reported"))
|
||||||
|
or _parse_epoch(rec.get("StartDate"))
|
||||||
|
or datetime.now(timezone.utc)),
|
||||||
|
expires=_parse_epoch(rec.get("PlannedEndDate")),
|
||||||
|
severity=_itd_severity(rec.get("Severity"), bool(rec.get("IsFullClosure"))),
|
||||||
|
geo=Geo(
|
||||||
|
centroid=centroid, geometry=geom,
|
||||||
|
regions=["US-ID"], primary_region="US-ID",
|
||||||
|
),
|
||||||
|
data={
|
||||||
|
"event_type_short": et,
|
||||||
|
"event_sub_type": _strip_or_none(rec.get("EventSubType")),
|
||||||
|
"roadway_name": _strip_or_none(rec.get("RoadwayName")),
|
||||||
|
"direction": _strip_or_none(rec.get("DirectionOfTravel")),
|
||||||
|
"description": _strip_or_none(rec.get("Description")),
|
||||||
|
"lanes_affected": _strip_or_none(rec.get("LanesAffected")),
|
||||||
|
"is_full_closure": bool(rec.get("IsFullClosure")),
|
||||||
|
"itd_severity": rec.get("Severity"),
|
||||||
|
"comment": _strip_or_none(rec.get("Comment")),
|
||||||
|
"cause": _strip_or_none(rec.get("Cause")),
|
||||||
|
"organization": rec.get("Organization"),
|
||||||
|
"recurrence_text": _strip_or_none(rec.get("Recurrence")),
|
||||||
|
"recurrence_schedules": rec.get("RecurrenceSchedules") or [],
|
||||||
|
"restrictions": rec.get("Restrictions") or {},
|
||||||
|
"detour_polyline": rec.get("DetourPolyline") or None,
|
||||||
|
"detour_instructions": _strip_or_none(rec.get("DetourInstructions")),
|
||||||
|
"encoded_polyline": rec.get("EncodedPolyline"),
|
||||||
|
"id_internal": rec.get("ID"),
|
||||||
|
"source_id": rec.get("SourceId"),
|
||||||
|
"reported_epoch": rec.get("Reported"),
|
||||||
|
"last_updated_epoch": rec.get("LastUpdated"),
|
||||||
|
"start_epoch": rec.get("StartDate"),
|
||||||
|
"planned_end_epoch": rec.get("PlannedEndDate"),
|
||||||
|
"latitude": lat,
|
||||||
|
"longitude": lon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _build_advisory_record(self, rec: dict[str, Any]) -> Event | None:
|
||||||
|
"""Build an Event from one /get/alerts row. ITD's currently-empty
|
||||||
|
response means we can't field-map exactly; we structural-pass-through
|
||||||
|
the whole record under ``data.advisory`` and probe a few likely id /
|
||||||
|
timestamp / coord fields best-effort. Per-record try/except in poll()
|
||||||
|
keeps a surprise shape from sinking the cycle."""
|
||||||
|
source_id = (rec.get("SourceId") or rec.get("ID")
|
||||||
|
or rec.get("AlertId") or rec.get("Id"))
|
||||||
|
if source_id is None:
|
||||||
|
return None
|
||||||
|
lat, lon = rec.get("Latitude"), rec.get("Longitude")
|
||||||
|
geom, centroid = _build_geometry(
|
||||||
|
lat, lon, rec.get("LatitudeSecondary"), rec.get("LongitudeSecondary"),
|
||||||
|
rec.get("EncodedPolyline"),
|
||||||
|
)
|
||||||
|
return Event(
|
||||||
|
id=f"idaho_511:advisory:{source_id}",
|
||||||
|
adapter=self.name,
|
||||||
|
category="advisory.itd_511",
|
||||||
|
time=(_parse_epoch(rec.get("LastUpdated"))
|
||||||
|
or _parse_epoch(rec.get("Reported"))
|
||||||
|
or _parse_epoch(rec.get("StartDate"))
|
||||||
|
or datetime.now(timezone.utc)),
|
||||||
|
expires=_parse_epoch(rec.get("PlannedEndDate") or rec.get("EndDate")),
|
||||||
|
severity=_itd_severity(rec.get("Severity"), False),
|
||||||
|
geo=Geo(
|
||||||
|
centroid=centroid, geometry=geom,
|
||||||
|
regions=["US-ID"], primary_region="US-ID",
|
||||||
|
),
|
||||||
|
data={
|
||||||
|
"event_type_short": "advisory",
|
||||||
|
"description": _strip_or_none(rec.get("Description") or rec.get("Message")),
|
||||||
|
"roadway_name": _strip_or_none(rec.get("RoadwayName")),
|
||||||
|
"advisory": rec, # full structural pass-through
|
||||||
|
"source_id": rec.get("SourceId"),
|
||||||
|
"latitude": lat,
|
||||||
|
"longitude": lon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
async def poll(self) -> AsyncIterator[Event]:
|
||||||
|
if not self._session:
|
||||||
|
raise RuntimeError("Session not initialized")
|
||||||
|
if not self._api_key:
|
||||||
|
logger.warning("itd_511: no API key for alias; skipping poll",
|
||||||
|
extra={"alias": self._api_key_alias})
|
||||||
|
return
|
||||||
|
|
||||||
|
do_advisories = self._adv_counter == 0
|
||||||
|
self._adv_counter = (self._adv_counter + 1) % _ADVISORY_EVERY_N_POLLS
|
||||||
|
|
||||||
|
endpoints = ["event"] + (["alerts"] if do_advisories else [])
|
||||||
|
results = await asyncio.gather(
|
||||||
|
*[self._fetch(ep) for ep in endpoints], return_exceptions=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
events_raw: list[dict[str, Any]] | None = None
|
||||||
|
adv_raw: list[dict[str, Any]] | None = None
|
||||||
|
for ep, result in zip(endpoints, results):
|
||||||
|
if isinstance(result, BaseException):
|
||||||
|
logger.warning("itd_511 fetch failed",
|
||||||
|
extra={"endpoint": ep, "error": self._redact(str(result))})
|
||||||
|
continue
|
||||||
|
if ep == "event":
|
||||||
|
events_raw = result
|
||||||
|
else:
|
||||||
|
adv_raw = result
|
||||||
|
|
||||||
|
yielded = 0
|
||||||
|
for rec in (events_raw or []):
|
||||||
|
try:
|
||||||
|
ev = self._build_event_record(rec)
|
||||||
|
except Exception: # one bad record never sinks the poll
|
||||||
|
logger.exception("itd_511 event parse failed",
|
||||||
|
extra={"id": rec.get("ID"), "source_id": rec.get("SourceId")})
|
||||||
|
continue
|
||||||
|
if ev is not None:
|
||||||
|
yield ev
|
||||||
|
yielded += 1
|
||||||
|
for rec in (adv_raw or []):
|
||||||
|
try:
|
||||||
|
ev = self._build_advisory_record(rec)
|
||||||
|
except Exception: # advisory shape unverified — be defensive
|
||||||
|
logger.exception("itd_511 advisory parse failed",
|
||||||
|
extra={"id": rec.get("Id") or rec.get("ID")})
|
||||||
|
continue
|
||||||
|
if ev is not None:
|
||||||
|
yield ev
|
||||||
|
yielded += 1
|
||||||
|
|
||||||
|
self.sweep_old_ids()
|
||||||
|
logger.info("itd_511 poll completed",
|
||||||
|
extra={"events_yielded": yielded,
|
||||||
|
"events_raw": len(events_raw or []),
|
||||||
|
"advisories_raw": len(adv_raw or []) if do_advisories else None})
|
||||||
|
|
||||||
|
def subject_for(self, event: Event) -> str:
|
||||||
|
et = event.category.split(".", 1)[0]
|
||||||
|
return f"central.traffic.{et}.us.id"
|
||||||
249
src/central/adapters/itd_511_cameras.py
Normal file
249
src/central/adapters/itd_511_cameras.py
Normal file
|
|
@ -0,0 +1,249 @@
|
||||||
|
"""ITD 511 cameras adapter — telemetry.
|
||||||
|
|
||||||
|
Polls /api/v2/get/cameras (664 cameras in the v0.10.0 step-0 snapshot) and
|
||||||
|
emits one telemetry Event per camera per UTC day. ITD aggregates border-region
|
||||||
|
mirrors from neighbouring DOTs (UDOT, ODOT, WYDOT, WSDOT, NDot, MTD, DriveBC,
|
||||||
|
Lemhi County) plus its own ITDNET/ACHD/RWIS/Idaho511 feeds; ~1.2% of cameras
|
||||||
|
sit outside Idaho but ride the same feed.
|
||||||
|
|
||||||
|
Per the v0.10.0 finding-4 architectural call: region tagged uniformly US-ID;
|
||||||
|
``data.source_jurisdiction`` carries the raw upstream ``Source`` field so
|
||||||
|
border-region cameras remain distinguishable for downstream consumers (lat/lon
|
||||||
|
bounds-checking deferred to a later release).
|
||||||
|
|
||||||
|
Subject convention (v0.9.20 forward, locked v0.10.0 architectural call A):
|
||||||
|
central.traffic_cameras.us.id.<camera_id>
|
||||||
|
|
||||||
|
Dedup id: ``idaho_511:cam:<Camera_Id>:<YYYY-MM-DD>`` — one telemetry event per
|
||||||
|
camera per UTC day, matching the state_511_atis_cameras precedent. Image URLs
|
||||||
|
ship straight through; the partial renders <img src=...> from Views[0].Url
|
||||||
|
(jpeg/gif/png all confirmed publicly reachable).
|
||||||
|
|
||||||
|
Retry predicate matches itd_511's (architectural call B): no retry on 4xx
|
||||||
|
except 429 with Retry-After. Shares the ``_Transient`` sentinel and
|
||||||
|
``_MAX_RETRY_AFTER_S`` cap from the sibling module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import sqlite3
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from tenacity import (
|
||||||
|
after_nothing,
|
||||||
|
retry,
|
||||||
|
retry_if_exception_type,
|
||||||
|
stop_after_attempt,
|
||||||
|
)
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.adapters.itd_511 import _MAX_RETRY_AFTER_S, _Transient, _wait_strategy
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
from central.config_store import ConfigStore
|
||||||
|
from central.models import Event, Geo
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_BASE_URL = "https://511.idaho.gov/api/v2/get"
|
||||||
|
_FETCH_TIMEOUT_S = 30
|
||||||
|
|
||||||
|
_DEDUP_DDL = (
|
||||||
|
"CREATE TABLE IF NOT EXISTS published_ids ("
|
||||||
|
"adapter TEXT NOT NULL, event_id TEXT NOT NULL, "
|
||||||
|
"first_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"last_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"PRIMARY KEY (adapter, event_id))"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Source values that are native ITD-internal feeds (not cross-DOT border-region
|
||||||
|
# mirrors). Drives the row-partial "(cross-DOT mirror)" annotation via the
|
||||||
|
# data.is_native_source boolean computed in _build_event — per
|
||||||
|
# feedback_no_hardcoding, the partial must not carry the allow-list itself.
|
||||||
|
NATIVE_SOURCES = frozenset({"ITDNET", "Idaho511", "ACHD", "RWIS"})
|
||||||
|
|
||||||
|
|
||||||
|
class Itd511CamerasSettings(BaseModel):
|
||||||
|
"""api_key_alias: config.api_keys entry (default 'idaho_511', shared with
|
||||||
|
the itd_511 event adapter)."""
|
||||||
|
|
||||||
|
api_key_alias: str = "idaho_511"
|
||||||
|
|
||||||
|
|
||||||
|
class Itd511CamerasAdapter(SourceAdapter):
|
||||||
|
"""Idaho 511 official cameras directory adapter (telemetry)."""
|
||||||
|
|
||||||
|
name = "itd_511_cameras"
|
||||||
|
display_name = "Idaho 511 Cameras (Official API)"
|
||||||
|
description = (
|
||||||
|
"Idaho Transportation Department's traffic camera directory (664 cameras "
|
||||||
|
"statewide plus border-region cross-DOT mirrors). One telemetry event "
|
||||||
|
"per camera per UTC day; the detail drawer streams the live image "
|
||||||
|
"direct from the source."
|
||||||
|
)
|
||||||
|
settings_schema = Itd511CamerasSettings
|
||||||
|
requires_api_key = "idaho_511"
|
||||||
|
api_key_field = "api_key_alias"
|
||||||
|
wizard_order = None # Ships disabled
|
||||||
|
default_cadence_s = 600
|
||||||
|
data_class = "telemetry"
|
||||||
|
enrichment_locations = [("latitude", "longitude")]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
config: AdapterConfig,
|
||||||
|
config_store: ConfigStore,
|
||||||
|
cursor_db_path: Path,
|
||||||
|
) -> None:
|
||||||
|
self._config_store = config_store
|
||||||
|
self._cursor_db_path = cursor_db_path
|
||||||
|
self._session: aiohttp.ClientSession | None = None
|
||||||
|
self._db: sqlite3.Connection | None = None
|
||||||
|
self._api_key_alias: str = config.settings.get("api_key_alias", "idaho_511")
|
||||||
|
self._api_key: str | None = None
|
||||||
|
|
||||||
|
def _redact(self, text: str) -> str:
|
||||||
|
return text.replace(self._api_key, "<KEY>") if self._api_key else text
|
||||||
|
|
||||||
|
async def startup(self) -> None:
|
||||||
|
self._session = aiohttp.ClientSession(
|
||||||
|
timeout=aiohttp.ClientTimeout(total=_FETCH_TIMEOUT_S),
|
||||||
|
headers={"User-Agent": "Central/0.10 (+itd_511_cameras)"},
|
||||||
|
)
|
||||||
|
self._db = sqlite3.connect(self._cursor_db_path)
|
||||||
|
self._db.execute(_DEDUP_DDL)
|
||||||
|
self._db.execute("CREATE INDEX IF NOT EXISTS published_ids_last_seen ON published_ids (last_seen)")
|
||||||
|
self._db.commit()
|
||||||
|
self._api_key = await self._config_store.get_api_key(self._api_key_alias)
|
||||||
|
logger.info("itd_511_cameras adapter started",
|
||||||
|
extra={"alias": self._api_key_alias, "api_key_present": bool(self._api_key)})
|
||||||
|
|
||||||
|
async def shutdown(self) -> None:
|
||||||
|
if self._session:
|
||||||
|
await self._session.close()
|
||||||
|
self._session = None
|
||||||
|
if self._db:
|
||||||
|
self._db.close()
|
||||||
|
self._db = None
|
||||||
|
|
||||||
|
async def apply_config(self, new_config: AdapterConfig) -> None:
|
||||||
|
self._api_key_alias = new_config.settings.get("api_key_alias", "idaho_511")
|
||||||
|
self._api_key = await self._config_store.get_api_key(self._api_key_alias)
|
||||||
|
logger.info("itd_511_cameras config updated",
|
||||||
|
extra={"alias": self._api_key_alias, "api_key_present": bool(self._api_key)})
|
||||||
|
|
||||||
|
@retry(
|
||||||
|
stop=stop_after_attempt(3),
|
||||||
|
wait=_wait_strategy,
|
||||||
|
retry=retry_if_exception_type((_Transient, aiohttp.ClientConnectionError,
|
||||||
|
asyncio.TimeoutError, TimeoutError)),
|
||||||
|
reraise=True, # propagate _Transient/ClientError verbatim, not RetryError
|
||||||
|
before_sleep=None, # explicit per BUG D5 audit: tenacity must not log between attempts
|
||||||
|
after=after_nothing, # explicit per BUG D5 audit: tenacity must not log after attempts
|
||||||
|
)
|
||||||
|
async def _fetch_cameras(self) -> list[dict[str, Any]] | None:
|
||||||
|
"""GET /api/v2/get/cameras?key=<KEY>. Same retry policy as itd_511
|
||||||
|
(the _wait_strategy + _Transient(wait_s=...) shape avoids the BUG C
|
||||||
|
double-wait on Retry-After)."""
|
||||||
|
if self._session is None:
|
||||||
|
raise RuntimeError("itd_511_cameras session not started")
|
||||||
|
async with self._session.get(
|
||||||
|
f"{_BASE_URL}/cameras", params={"key": self._api_key},
|
||||||
|
) as resp:
|
||||||
|
if resp.status == 200:
|
||||||
|
return await resp.json(content_type=None)
|
||||||
|
body_preview = self._redact((await resp.text())[:200])
|
||||||
|
if resp.status == 429:
|
||||||
|
ra = resp.headers.get("Retry-After", "")
|
||||||
|
wait_s = min(int(ra) if ra.isdigit() else 5, _MAX_RETRY_AFTER_S)
|
||||||
|
logger.warning("itd_511_cameras rate-limited",
|
||||||
|
extra={"retry_after": wait_s, "body": body_preview})
|
||||||
|
raise _Transient(f"429 retry-after={wait_s}", wait_s=wait_s)
|
||||||
|
if 500 <= resp.status < 600:
|
||||||
|
logger.warning("itd_511_cameras upstream 5xx",
|
||||||
|
extra={"status": resp.status, "body": body_preview})
|
||||||
|
raise _Transient(f"{resp.status} server error")
|
||||||
|
logger.warning("itd_511_cameras permanent error; skipping poll",
|
||||||
|
extra={"status": resp.status, "body": body_preview})
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _build_event(self, cam: dict[str, Any]) -> Event | None:
|
||||||
|
cam_id = cam.get("Id")
|
||||||
|
if cam_id is None:
|
||||||
|
return None
|
||||||
|
views = cam.get("Views") or []
|
||||||
|
image_url = views[0].get("Url") if views else None
|
||||||
|
lat, lon = cam.get("Latitude"), cam.get("Longitude")
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
day = now.strftime("%Y-%m-%d")
|
||||||
|
source = cam.get("Source")
|
||||||
|
return Event(
|
||||||
|
id=f"idaho_511:cam:{cam_id}:{day}",
|
||||||
|
adapter=self.name,
|
||||||
|
category="camera.itd_511_cameras",
|
||||||
|
time=now,
|
||||||
|
severity=1, # telemetry; cameras carry no severity signal
|
||||||
|
geo=Geo(
|
||||||
|
centroid=((lon, lat) if lat is not None and lon is not None else None),
|
||||||
|
regions=["US-ID"],
|
||||||
|
primary_region="US-ID",
|
||||||
|
),
|
||||||
|
data={
|
||||||
|
"camera_id": cam_id,
|
||||||
|
"roadway": cam.get("Roadway"),
|
||||||
|
"direction": cam.get("Direction"),
|
||||||
|
"location": cam.get("Location"),
|
||||||
|
"source": source,
|
||||||
|
"source_jurisdiction": source, # alias per v0.10.0 finding 4
|
||||||
|
# Drives the row-partial "(cross-DOT mirror)" annotation
|
||||||
|
# without hardcoding the allow-list at the template layer
|
||||||
|
# (BUG D2 / [[feedback_no_hardcoding]]).
|
||||||
|
"is_native_source": source in NATIVE_SOURCES,
|
||||||
|
"source_id_upstream": cam.get("SourceId"),
|
||||||
|
"image_url": image_url,
|
||||||
|
"additional_views": [v.get("Url") for v in views[1:]],
|
||||||
|
"view_count": len(views),
|
||||||
|
"view_descriptions": [v.get("Description") for v in views],
|
||||||
|
"sort_order": cam.get("SortOrder"),
|
||||||
|
"latitude": lat,
|
||||||
|
"longitude": lon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
async def poll(self) -> AsyncIterator[Event]:
|
||||||
|
if not self._session:
|
||||||
|
raise RuntimeError("Session not initialized")
|
||||||
|
if not self._api_key:
|
||||||
|
logger.warning("itd_511_cameras: no API key for alias; skipping poll",
|
||||||
|
extra={"alias": self._api_key_alias})
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
cams = await self._fetch_cameras()
|
||||||
|
except (_Transient, aiohttp.ClientError, TimeoutError) as exc:
|
||||||
|
# BUG E fix: _Transient must be caught alongside ClientError —
|
||||||
|
# else tenacity's reraise of a persistent 5xx/429 (after exhausted
|
||||||
|
# retries) crashes the whole poll cycle.
|
||||||
|
logger.warning("itd_511_cameras fetch failed",
|
||||||
|
extra={"error": self._redact(str(exc))})
|
||||||
|
return
|
||||||
|
yielded = 0
|
||||||
|
for cam in (cams or []):
|
||||||
|
try:
|
||||||
|
ev = self._build_event(cam)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("itd_511_cameras parse failed",
|
||||||
|
extra={"id": cam.get("Id")})
|
||||||
|
continue
|
||||||
|
if ev is not None:
|
||||||
|
yield ev
|
||||||
|
yielded += 1
|
||||||
|
self.sweep_old_ids()
|
||||||
|
logger.info("itd_511_cameras poll completed",
|
||||||
|
extra={"cameras_yielded": yielded, "cameras_raw": len(cams or [])})
|
||||||
|
|
||||||
|
def subject_for(self, event: Event) -> str:
|
||||||
|
return f"central.traffic_cameras.us.id.{event.data.get('camera_id')}"
|
||||||
|
|
@ -2851,7 +2851,7 @@ ADAPTER_GROUPS = {
|
||||||
"Space": ["swpc_alerts", "swpc_kindex", "swpc_protons"],
|
"Space": ["swpc_alerts", "swpc_kindex", "swpc_protons"],
|
||||||
"Geophysical": ["usgs_quake", "nwis"],
|
"Geophysical": ["usgs_quake", "nwis"],
|
||||||
"Earth Observation": ["eonet"],
|
"Earth Observation": ["eonet"],
|
||||||
"Transportation": ["wzdx", "state_511_atis", "tomtom_flow", "tomtom_incidents", "state_511_atis_cameras"],
|
"Transportation": ["wzdx", "state_511_atis", "tomtom_flow", "tomtom_incidents", "state_511_atis_cameras", "itd_511", "itd_511_cameras"],
|
||||||
}
|
}
|
||||||
# Same palette the map legend uses, indexed by sorted-adapter position.
|
# Same palette the map legend uses, indexed by sorted-adapter position.
|
||||||
EVENTS_PALETTE = [
|
EVENTS_PALETTE = [
|
||||||
|
|
|
||||||
16
src/central/gui/templates/_event_rows/itd_511.html
Normal file
16
src/central/gui/templates/_event_rows/itd_511.html
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
{# ITD 511 detail rows. Fields from payload->data->data; every block guarded.
|
||||||
|
Direction "Unknown"/"None" suppressed (wzdx lesson). Recurrence HTML
|
||||||
|
stripped via striptags so the source's <b>/<br/> markup doesn't leak. #}
|
||||||
|
{%- set d = (event.data.get('data') or {}).get('data') or {} -%}
|
||||||
|
{% if d.get('roadway_name') %}<dt>Road</dt><dd>{{ d.roadway_name }}{% if d.get('direction') and d.direction not in ['Unknown', 'None'] %} ({{ d.direction }}){% endif %}</dd>{% endif %}
|
||||||
|
{% if d.get('event_sub_type') %}<dt>Type</dt><dd>{{ d.event_sub_type }}</dd>{% endif %}
|
||||||
|
{% if d.get('lanes_affected') and d.lanes_affected != 'No Data' %}<dt>Lanes</dt><dd>{{ d.lanes_affected }}</dd>{% endif %}
|
||||||
|
{% if d.get('is_full_closure') %}<dt>Full closure</dt><dd>Yes</dd>{% endif %}
|
||||||
|
{% if d.get('itd_severity') and d.itd_severity != 'None' %}<dt>ITD severity</dt><dd>{{ d.itd_severity }}</dd>{% endif %}
|
||||||
|
{% if d.get('description') %}<dt>Description</dt><dd>{{ d.description | truncate(220) }}</dd>{% endif %}
|
||||||
|
{% if d.get('comment') %}<dt>Comment</dt><dd>{{ d.comment | truncate(220) }}</dd>{% endif %}
|
||||||
|
{% if d.get('cause') and d.cause != d.get('event_type_short') %}<dt>Cause</dt><dd>{{ d.cause }}</dd>{% endif %}
|
||||||
|
{% if d.get('detour_instructions') %}<dt>Detour</dt><dd>{{ d.detour_instructions | truncate(220) }}</dd>{% endif %}
|
||||||
|
{% if d.get('recurrence_text') %}<dt>Schedule</dt><dd>{{ d.recurrence_text | striptags | truncate(120) }}</dd>{% endif %}
|
||||||
|
{%- set r = d.get('restrictions') or {} -%}
|
||||||
|
{% if r.get('Width') or r.get('Height') or r.get('Length') or r.get('Weight') or r.get('Speed') %}<dt>Restrictions</dt><dd>{% if r.Width %}W:{{ r.Width }} {% endif %}{% if r.Height %}H:{{ r.Height }} {% endif %}{% if r.Length %}L:{{ r.Length }} {% endif %}{% if r.Weight %}Wt:{{ r.Weight }} {% endif %}{% if r.Speed %}Spd:{{ r.Speed }}{% endif %}</dd>{% endif %}
|
||||||
10
src/central/gui/templates/_event_rows/itd_511_cameras.html
Normal file
10
src/central/gui/templates/_event_rows/itd_511_cameras.html
Normal file
|
|
@ -0,0 +1,10 @@
|
||||||
|
{# ITD 511 camera detail. Live <img> fetched direct from the source (publicly
|
||||||
|
reachable, no auth). Format may be jpeg/gif/png — <img> handles all.
|
||||||
|
source_jurisdiction surfaced for cross-DOT border-region cameras (UDOT/
|
||||||
|
ODOT/etc.) per v0.10.0 finding 4. Fields from payload->data->data. #}
|
||||||
|
{%- set d = (event.data.get('data') or {}).get('data') or {} -%}
|
||||||
|
{% if d.get('image_url') %}<dt>View</dt><dd><img src="{{ d.image_url }}" alt="Camera view" style="max-width:480px;max-height:360px;display:block;" loading="lazy"></dd>{% endif %}
|
||||||
|
{% if d.get('roadway') %}<dt>Road</dt><dd>{{ d.roadway }}{% if d.get('direction') and d.direction not in ['Unknown', 'None'] %} ({{ d.direction }}){% endif %}</dd>{% endif %}
|
||||||
|
{% if d.get('location') %}<dt>Location</dt><dd>{{ d.location }}</dd>{% endif %}
|
||||||
|
{% if d.get('source_jurisdiction') and not d.get('is_native_source') %}<dt>Source</dt><dd>{{ d.source_jurisdiction }} (cross-DOT mirror)</dd>{% endif %}
|
||||||
|
{% if d.get('view_count') and d.view_count > 1 %}<dt>Views</dt><dd>{{ d.view_count }} camera angles</dd>{% endif %}
|
||||||
9
src/central/gui/templates/_event_summaries/itd_511.html
Normal file
9
src/central/gui/templates/_event_summaries/itd_511.html
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
{# ITD 511 one-line subject. event_type_short ∈ {work_zone, closure, incident,
|
||||||
|
special_event, advisory}. Fields from payload->data->data. Drops
|
||||||
|
"Unknown"/"None" direction tokens per the v0.9.1 wzdx lesson. #}
|
||||||
|
{%- set d = (event.data.get('data') or {}).get('data') or {} -%}
|
||||||
|
{%- set labels = {'work_zone': 'Road work', 'closure': 'Closure', 'incident': 'Incident', 'special_event': 'Special event', 'advisory': 'Advisory'} -%}
|
||||||
|
{%- set lead = labels.get(d.get('event_type_short'), 'Traffic event') -%}
|
||||||
|
{%- set dir = d.get('direction') -%}
|
||||||
|
{%- set show_dir = dir and dir not in ['Unknown', 'None'] -%}
|
||||||
|
{{ lead }}{% if d.get('roadway_name') %} on {{ d.roadway_name }}{% if show_dir %} {{ dir }}{% endif %}{% endif %}{% if d.get('is_full_closure') %} (full closure){% endif %}
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
{# ITD 511 camera one-line subject. Location field is already humanized (e.g.
|
||||||
|
"I-15 UT/ID State Line UT"), so it stands alone. Falls back to roadway then
|
||||||
|
camera_id. Fields from payload->data->data. #}
|
||||||
|
{%- set d = (event.data.get('data') or {}).get('data') or {} -%}
|
||||||
|
Camera: {{ d.get('location') or d.get('roadway') or ('#' ~ d.get('camera_id')) }}
|
||||||
1
tests/fixtures/itd_511_alerts_sample.json
vendored
Normal file
1
tests/fixtures/itd_511_alerts_sample.json
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
[]
|
||||||
90
tests/fixtures/itd_511_cameras_sample.json
vendored
Normal file
90
tests/fixtures/itd_511_cameras_sample.json
vendored
Normal file
|
|
@ -0,0 +1,90 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"Id": 3,
|
||||||
|
"Source": "ITDNET",
|
||||||
|
"SourceId": "1000",
|
||||||
|
"Roadway": "SH-55 Eagle",
|
||||||
|
"Direction": "Southbound",
|
||||||
|
"Latitude": 43.619167,
|
||||||
|
"Longitude": -116.35478,
|
||||||
|
"Location": "SH-55 Eagle Fairview",
|
||||||
|
"SortOrder": 0,
|
||||||
|
"Views": [
|
||||||
|
{
|
||||||
|
"Id": 1039,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/1039",
|
||||||
|
"Status": "Disabled",
|
||||||
|
"Description": "D3 SH-55 37.9 Eagle Fairview 526"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": 436,
|
||||||
|
"Source": "ACHD",
|
||||||
|
"SourceId": "4001",
|
||||||
|
"Roadway": "Local Boise",
|
||||||
|
"Direction": "Unknown",
|
||||||
|
"Latitude": 43.60304,
|
||||||
|
"Longitude": -116.18841,
|
||||||
|
"Location": "Park Parkcenter Front Clearwater",
|
||||||
|
"SortOrder": 0,
|
||||||
|
"Views": [
|
||||||
|
{
|
||||||
|
"Id": 631,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/631",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Description": ""
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": 1,
|
||||||
|
"Source": "UDOT",
|
||||||
|
"SourceId": "10.C1",
|
||||||
|
"Roadway": "I-15",
|
||||||
|
"Direction": "Unknown",
|
||||||
|
"Latitude": 42.0011,
|
||||||
|
"Longitude": -112.198,
|
||||||
|
"Location": "I-15 UT/ID State Line UT",
|
||||||
|
"SortOrder": 1,
|
||||||
|
"Views": [
|
||||||
|
{
|
||||||
|
"Id": 1,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/1",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Description": "N/A"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": 2,
|
||||||
|
"Source": "RWIS",
|
||||||
|
"SourceId": "100.C1",
|
||||||
|
"Roadway": "SH-75",
|
||||||
|
"Direction": "Unknown",
|
||||||
|
"Latitude": 43.5946,
|
||||||
|
"Longitude": -114.345,
|
||||||
|
"Location": "SH-75 Wood River",
|
||||||
|
"SortOrder": 325,
|
||||||
|
"Views": [
|
||||||
|
{
|
||||||
|
"Id": 2,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/2",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Description": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": 3,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/3",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Description": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Id": 4,
|
||||||
|
"Url": "https://511.idaho.gov/map/Cctv/4",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Description": ""
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
214
tests/fixtures/itd_511_event_sample.json
vendored
Normal file
214
tests/fixtures/itd_511_event_sample.json
vendored
Normal file
|
|
@ -0,0 +1,214 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"ID": 23,
|
||||||
|
"SourceId": "4277",
|
||||||
|
"Organization": "ERS",
|
||||||
|
"RoadwayName": "SH-81",
|
||||||
|
"DirectionOfTravel": "Unknown",
|
||||||
|
"Description": " Work on the shoulder on SH-81 near Poverty Gulch. 7/8/2024 9:29 PM Mon: 12:00 PM - 5:00 PM, Tue, Wed, Thu, Fri, Sat, Sun: Active all day Activities: use caution, warning.",
|
||||||
|
"Reported": 1720495740,
|
||||||
|
"LastUpdated": 1749675731,
|
||||||
|
"StartDate": 1720495740,
|
||||||
|
"PlannedEndDate": null,
|
||||||
|
"LanesAffected": "No Data",
|
||||||
|
"Latitude": 42.5168038430856,
|
||||||
|
"Longitude": -113.711287649613,
|
||||||
|
"LatitudeSecondary": null,
|
||||||
|
"LongitudeSecondary": null,
|
||||||
|
"EventType": "roadwork",
|
||||||
|
"EventSubType": "workOnTheShoulder",
|
||||||
|
"IsFullClosure": false,
|
||||||
|
"Severity": "None",
|
||||||
|
"Comment": null,
|
||||||
|
"EncodedPolyline": null,
|
||||||
|
"Restrictions": {
|
||||||
|
"Width": null,
|
||||||
|
"Height": null,
|
||||||
|
"Length": null,
|
||||||
|
"Weight": null,
|
||||||
|
"Speed": null
|
||||||
|
},
|
||||||
|
"DetourPolyline": "",
|
||||||
|
"DetourInstructions": "",
|
||||||
|
"Recurrence": "<b>Mon:</b><br/>12:00 PM - 5:00 PM<br/><br/><b>Tue, Wed, Thu, Fri, Sat, Sun:</b><br/>Active all day<br/><br/>",
|
||||||
|
"RecurrenceSchedules": [
|
||||||
|
{
|
||||||
|
"StartDate": "7/8/2024 9:29:00 PM-06:00:00",
|
||||||
|
"EndDate": null,
|
||||||
|
"Times": [
|
||||||
|
{
|
||||||
|
"StartTime": "12:00:00-06:00:00",
|
||||||
|
"EndTime": "16:59:59-06:00:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"DaysOfWeek": [
|
||||||
|
"Monday"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"StartDate": "7/8/2024 9:29:00 PM-06:00:00",
|
||||||
|
"EndDate": null,
|
||||||
|
"Times": [
|
||||||
|
{
|
||||||
|
"StartTime": "00:00:00-06:00:00",
|
||||||
|
"EndTime": "23:59:59-06:00:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"DaysOfWeek": [
|
||||||
|
"Tuesday",
|
||||||
|
"Wednesday",
|
||||||
|
"Thursday",
|
||||||
|
"Friday",
|
||||||
|
"Saturday",
|
||||||
|
"Sunday"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Cause": "roadwork"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"ID": 17,
|
||||||
|
"SourceId": "469",
|
||||||
|
"Organization": "ERS",
|
||||||
|
"RoadwayName": "N McDermott Rd",
|
||||||
|
"DirectionOfTravel": "Both",
|
||||||
|
"Description": " Long term road construction on N McDermott Rd Both Directions from Five Mile Creek to US-20. All lanes closed. 1/30/2023 2:24 PM Mon, Tue, Wed, Thu, Fri, Sat, Sun: Active all day",
|
||||||
|
"Reported": 1675113840,
|
||||||
|
"LastUpdated": 1718758080,
|
||||||
|
"StartDate": 1675113840,
|
||||||
|
"PlannedEndDate": null,
|
||||||
|
"LanesAffected": "All lanes closed",
|
||||||
|
"Latitude": 43.6485700000001,
|
||||||
|
"Longitude": -116.47349,
|
||||||
|
"LatitudeSecondary": 43.6630000000001,
|
||||||
|
"LongitudeSecondary": -116.47421,
|
||||||
|
"EventType": "closures",
|
||||||
|
"EventSubType": "longTermRoadConstruction",
|
||||||
|
"IsFullClosure": true,
|
||||||
|
"Severity": "None",
|
||||||
|
"Comment": "Open to local traffic only.",
|
||||||
|
"EncodedPolyline": "qbliGhv{eUsk@MeTEsBAsAGiAQw@Qi@MoAYmA]aA]}@a@[?[JSNKNENCTEzCGnAIX",
|
||||||
|
"Restrictions": {
|
||||||
|
"Width": null,
|
||||||
|
"Height": null,
|
||||||
|
"Length": null,
|
||||||
|
"Weight": null,
|
||||||
|
"Speed": null
|
||||||
|
},
|
||||||
|
"DetourPolyline": "",
|
||||||
|
"DetourInstructions": "",
|
||||||
|
"Recurrence": "<b>Mon, Tue, Wed, Thu, Fri, Sat, Sun:</b><br/>Active all day<br/><br/>",
|
||||||
|
"RecurrenceSchedules": [
|
||||||
|
{
|
||||||
|
"StartDate": "1/30/2023 2:24:00 PM-06:00:00",
|
||||||
|
"EndDate": null,
|
||||||
|
"Times": [
|
||||||
|
{
|
||||||
|
"StartTime": "00:00:00-06:00:00",
|
||||||
|
"EndTime": "23:59:59-06:00:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"DaysOfWeek": [
|
||||||
|
"Monday",
|
||||||
|
"Tuesday",
|
||||||
|
"Wednesday",
|
||||||
|
"Thursday",
|
||||||
|
"Friday",
|
||||||
|
"Saturday",
|
||||||
|
"Sunday"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Cause": "roadwork"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"ID": 34585,
|
||||||
|
"SourceId": "11146",
|
||||||
|
"Organization": "ERS",
|
||||||
|
"RoadwayName": "I-84",
|
||||||
|
"DirectionOfTravel": "East",
|
||||||
|
"Description": " Left Lane Blocked on I-84 Eastbound near N Franklin Blvd. 2 Left Lanes Blocked. Activities: Expect Delays, Look Out for Flagger, Use Caution.",
|
||||||
|
"Reported": 1780543740,
|
||||||
|
"LastUpdated": 1780543820,
|
||||||
|
"StartDate": 1780543740,
|
||||||
|
"PlannedEndDate": null,
|
||||||
|
"LanesAffected": "2 Left Lanes Blocked",
|
||||||
|
"Latitude": 43.5980897425697,
|
||||||
|
"Longitude": -116.542818286917,
|
||||||
|
"LatitudeSecondary": null,
|
||||||
|
"LongitudeSecondary": null,
|
||||||
|
"EventType": "accidentsAndIncidents",
|
||||||
|
"EventSubType": "leftLaneBlocked",
|
||||||
|
"IsFullClosure": false,
|
||||||
|
"Severity": "None",
|
||||||
|
"Comment": "The two left lanes of eastbound I-84 will be blocked near milepost 36.5 tonight for road maintenance. Keep Right.",
|
||||||
|
"EncodedPolyline": null,
|
||||||
|
"Restrictions": {
|
||||||
|
"Width": null,
|
||||||
|
"Height": null,
|
||||||
|
"Length": null,
|
||||||
|
"Weight": null,
|
||||||
|
"Speed": null
|
||||||
|
},
|
||||||
|
"DetourPolyline": "",
|
||||||
|
"DetourInstructions": "",
|
||||||
|
"Recurrence": "",
|
||||||
|
"RecurrenceSchedules": "",
|
||||||
|
"Cause": "Incident"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"ID": 33663,
|
||||||
|
"SourceId": "11000",
|
||||||
|
"Organization": "ERS",
|
||||||
|
"RoadwayName": "SH-16",
|
||||||
|
"DirectionOfTravel": "Both",
|
||||||
|
"Description": " Special event on SH-16 Both Directions at W Chaparral Rd. 6/8/2026 8:00 AM to 6/15/2026 5:00 PM Mon, Tue, Wed, Thu, Fri, Sat, Sun: Active all day Activities: Use Caution.",
|
||||||
|
"Reported": 1780927200,
|
||||||
|
"LastUpdated": 1779805341,
|
||||||
|
"StartDate": 1780927200,
|
||||||
|
"PlannedEndDate": 1781564400,
|
||||||
|
"LanesAffected": "No Data",
|
||||||
|
"Latitude": 43.780078841046,
|
||||||
|
"Longitude": -116.473209123902,
|
||||||
|
"LatitudeSecondary": null,
|
||||||
|
"LongitudeSecondary": null,
|
||||||
|
"EventType": "specialEvents",
|
||||||
|
"EventSubType": "specialEvent",
|
||||||
|
"IsFullClosure": false,
|
||||||
|
"Severity": "None",
|
||||||
|
"Comment": "Rodeo Traffic entering/leaving roadway ",
|
||||||
|
"EncodedPolyline": null,
|
||||||
|
"Restrictions": {
|
||||||
|
"Width": null,
|
||||||
|
"Height": null,
|
||||||
|
"Length": null,
|
||||||
|
"Weight": null,
|
||||||
|
"Speed": null
|
||||||
|
},
|
||||||
|
"DetourPolyline": "",
|
||||||
|
"DetourInstructions": "",
|
||||||
|
"Recurrence": "<b>Mon, Tue, Wed, Thu, Fri, Sat, Sun:</b><br/>Active all day<br/><br/>",
|
||||||
|
"RecurrenceSchedules": [
|
||||||
|
{
|
||||||
|
"StartDate": "6/8/2026 8:00:00 AM-06:00:00",
|
||||||
|
"EndDate": "6/15/2026 5:00:00 PM-06:00:00",
|
||||||
|
"Times": [
|
||||||
|
{
|
||||||
|
"StartTime": "00:00:00-06:00:00",
|
||||||
|
"EndTime": "23:59:59-06:00:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"DaysOfWeek": [
|
||||||
|
"Monday",
|
||||||
|
"Tuesday",
|
||||||
|
"Wednesday",
|
||||||
|
"Thursday",
|
||||||
|
"Friday",
|
||||||
|
"Saturday",
|
||||||
|
"Sunday"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Cause": "specialEvents"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
@ -1147,6 +1147,8 @@ _SAMPLE_INNER = {
|
||||||
"tomtom_flow": {"road_category": "primary", "relative_speed": 0.11},
|
"tomtom_flow": {"road_category": "primary", "relative_speed": 0.11},
|
||||||
"tomtom_incidents": {"description": "Roadworks", "from": "Early Road", "to": "Slade Road"},
|
"tomtom_incidents": {"description": "Roadworks", "from": "Early Road", "to": "Slade Road"},
|
||||||
"state_511_atis_cameras": {"location_description": "I-84 Mountain Home", "camera_id": 42},
|
"state_511_atis_cameras": {"location_description": "I-84 Mountain Home", "camera_id": 42},
|
||||||
|
"itd_511": {"event_type_short": "work_zone", "roadway_name": "I-84"},
|
||||||
|
"itd_511_cameras": {"location": "I-84 Mountain Home", "camera_id": 42},
|
||||||
}
|
}
|
||||||
|
|
||||||
# Exact expected subjects for the deterministic adapters. swpc_alerts is omitted
|
# Exact expected subjects for the deterministic adapters. swpc_alerts is omitted
|
||||||
|
|
@ -1169,6 +1171,8 @@ _EXPECTED_SUBJECT = {
|
||||||
"tomtom_flow": "Traffic flow (primary) — 11% of free-flow",
|
"tomtom_flow": "Traffic flow (primary) — 11% of free-flow",
|
||||||
"tomtom_incidents": "Roadworks on Early Road → Slade Road",
|
"tomtom_incidents": "Roadworks on Early Road → Slade Road",
|
||||||
"state_511_atis_cameras": "Camera: I-84 Mountain Home",
|
"state_511_atis_cameras": "Camera: I-84 Mountain Home",
|
||||||
|
"itd_511": "Road work on I-84",
|
||||||
|
"itd_511_cameras": "Camera: I-84 Mountain Home",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
398
tests/test_itd_511.py
Normal file
398
tests/test_itd_511.py
Normal file
|
|
@ -0,0 +1,398 @@
|
||||||
|
"""Tests for the itd_511 adapter (v0.10.0).
|
||||||
|
|
||||||
|
Fixtures are real captures from https://511.idaho.gov/api/v2/get/event,alerts
|
||||||
|
trimmed to one record per EventType plus an empty advisories baseline:
|
||||||
|
tests/fixtures/itd_511_event_sample.json
|
||||||
|
tests/fixtures/itd_511_alerts_sample.json
|
||||||
|
|
||||||
|
No conftest entry: dedup uses the supervisor-injected cursors.db (inherited
|
||||||
|
mixin); polling is stateless.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.adapters.itd_511 import (
|
||||||
|
EVENT_TYPE_MAP,
|
||||||
|
Itd511Adapter,
|
||||||
|
_build_geometry,
|
||||||
|
_decode_polyline,
|
||||||
|
_itd_severity,
|
||||||
|
_parse_epoch,
|
||||||
|
_strip_or_none,
|
||||||
|
_Transient,
|
||||||
|
_wait_strategy,
|
||||||
|
)
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
from central.models import Event, Geo
|
||||||
|
|
||||||
|
FIX = Path(__file__).parent / "fixtures"
|
||||||
|
EVENT = json.loads((FIX / "itd_511_event_sample.json").read_text())
|
||||||
|
ALERTS = json.loads((FIX / "itd_511_alerts_sample.json").read_text())
|
||||||
|
BY_TYPE = {r["EventType"]: r for r in EVENT}
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg():
|
||||||
|
return AdapterConfig(
|
||||||
|
name="itd_511", enabled=True, cadence_s=60,
|
||||||
|
settings={"api_key_alias": "idaho_511"},
|
||||||
|
updated_at=datetime.now(timezone.utc),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def adapter(tmp_path):
|
||||||
|
cs = MagicMock()
|
||||||
|
cs.get_api_key = AsyncMock(return_value="testkey-32chars-deadbeefdeadbeef")
|
||||||
|
return Itd511Adapter(_cfg(), cs, tmp_path / "cursors.db")
|
||||||
|
|
||||||
|
|
||||||
|
def test_event_type_map_is_complete():
|
||||||
|
assert EVENT_TYPE_MAP == {
|
||||||
|
"roadwork": "work_zone", "closures": "closure",
|
||||||
|
"accidentsAndIncidents": "incident", "specialEvents": "special_event",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_epoch():
|
||||||
|
assert _parse_epoch(1675113840) == datetime(2023, 1, 30, 21, 24, tzinfo=timezone.utc)
|
||||||
|
assert _parse_epoch(None) is None
|
||||||
|
assert _parse_epoch("not-an-int") is None
|
||||||
|
assert _parse_epoch("1675113840") == datetime(2023, 1, 30, 21, 24, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("sev,fc,expected", [
|
||||||
|
("None", False, 1), ("Minor", False, 2), ("Major", False, 3),
|
||||||
|
("None", True, 3), ("Minor", True, 3), ("Major", True, 3), # full-closure forces 3
|
||||||
|
(None, False, 1), ("Bogus", False, 1),
|
||||||
|
])
|
||||||
|
def test_severity_mapping(sev, fc, expected):
|
||||||
|
assert _itd_severity(sev, fc) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_strip_or_none_handles_eventsubtype_trailing_space():
|
||||||
|
assert _strip_or_none("pavementMarkingOperations ") == "pavementMarkingOperations"
|
||||||
|
assert _strip_or_none("") is None
|
||||||
|
assert _strip_or_none(" ") is None
|
||||||
|
assert _strip_or_none(None) is None
|
||||||
|
assert _strip_or_none(42) == 42 # non-string passthrough
|
||||||
|
|
||||||
|
|
||||||
|
def test_decode_polyline_roundtrip():
|
||||||
|
import polyline as polyline_lib
|
||||||
|
enc = polyline_lib.encode([(43.6, -116.5), (43.7, -116.4)])
|
||||||
|
assert _decode_polyline(enc) == [(43.6, -116.5), (43.7, -116.4)]
|
||||||
|
assert _decode_polyline(None) == []
|
||||||
|
assert _decode_polyline("") == []
|
||||||
|
# malformed string => library raises => caught => []
|
||||||
|
assert _decode_polyline("\x00\x00\x00not-a-polyline") == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_geometry_polyline_wins():
|
||||||
|
import polyline as polyline_lib
|
||||||
|
enc = polyline_lib.encode([(43.6, -116.5), (43.7, -116.4)])
|
||||||
|
geom, centroid = _build_geometry(40.0, -100.0, None, None, enc)
|
||||||
|
assert geom["type"] == "LineString"
|
||||||
|
assert len(geom["coordinates"]) == 2
|
||||||
|
assert centroid == geom["coordinates"][0] # first vertex (lon, lat) order
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_geometry_bookend_linestring():
|
||||||
|
geom, centroid = _build_geometry(43.6, -116.5, 43.7, -116.4, None)
|
||||||
|
assert geom == {"type": "LineString",
|
||||||
|
"coordinates": [(-116.5, 43.6), (-116.4, 43.7)]}
|
||||||
|
assert centroid == (-116.5, 43.6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_geometry_point_only():
|
||||||
|
geom, centroid = _build_geometry(43.6, -116.5, None, None, None)
|
||||||
|
assert geom == {"type": "Point", "coordinates": (-116.5, 43.6)}
|
||||||
|
assert centroid == (-116.5, 43.6)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_geometry_missing_all():
|
||||||
|
assert _build_geometry(None, None, None, None, None) == (None, None)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("etype,short", [
|
||||||
|
("roadwork", "work_zone"), ("closures", "closure"),
|
||||||
|
("accidentsAndIncidents", "incident"), ("specialEvents", "special_event"),
|
||||||
|
])
|
||||||
|
def test_build_event_category_and_dedup_id(adapter, etype, short):
|
||||||
|
rec = BY_TYPE[etype]
|
||||||
|
e = adapter._build_event_record(rec)
|
||||||
|
assert e.category == f"{short}.itd_511"
|
||||||
|
assert e.id == f"idaho_511:event:{rec['SourceId']}"
|
||||||
|
assert e.adapter == "itd_511"
|
||||||
|
assert e.geo.primary_region == "US-ID"
|
||||||
|
assert e.geo.regions == ["US-ID"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_closure_has_linestring_geometry(adapter):
|
||||||
|
e = adapter._build_event_record(BY_TYPE["closures"])
|
||||||
|
assert e.geo.geometry is not None
|
||||||
|
assert e.geo.geometry["type"] == "LineString"
|
||||||
|
assert len(e.geo.geometry["coordinates"]) >= 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_full_closure_forces_severity_3(adapter):
|
||||||
|
e = adapter._build_event_record(BY_TYPE["closures"])
|
||||||
|
assert e.data["is_full_closure"] is True
|
||||||
|
assert e.severity == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_unknown_event_type_falls_back_to_incident(adapter):
|
||||||
|
rec = {**BY_TYPE["roadwork"], "EventType": "WhoKnows", "SourceId": "X1"}
|
||||||
|
e = adapter._build_event_record(rec)
|
||||||
|
assert e.category == "incident.itd_511"
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_dedup_id_falls_back_to_id_when_sourceid_missing(adapter):
|
||||||
|
rec = {**BY_TYPE["roadwork"], "SourceId": None, "ID": 99999}
|
||||||
|
e = adapter._build_event_record(rec)
|
||||||
|
assert e.id == "idaho_511:event:99999"
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_returns_none_without_any_id(adapter):
|
||||||
|
rec = {**BY_TYPE["roadwork"], "SourceId": None, "ID": None}
|
||||||
|
assert adapter._build_event_record(rec) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_strips_trailing_space_on_event_sub_type(adapter):
|
||||||
|
rec = {**BY_TYPE["roadwork"], "EventSubType": "pavementMarkingOperations "}
|
||||||
|
e = adapter._build_event_record(rec)
|
||||||
|
assert e.data["event_sub_type"] == "pavementMarkingOperations"
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_captures_cause_and_organization(adapter):
|
||||||
|
e = adapter._build_event_record(BY_TYPE["roadwork"])
|
||||||
|
assert e.data["cause"] == BY_TYPE["roadwork"]["Cause"]
|
||||||
|
assert e.data["organization"] == BY_TYPE["roadwork"]["Organization"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_passes_through_recurrence_and_restrictions(adapter):
|
||||||
|
e = adapter._build_event_record(BY_TYPE["closures"])
|
||||||
|
assert e.data["recurrence_schedules"] == BY_TYPE["closures"]["RecurrenceSchedules"]
|
||||||
|
assert e.data["restrictions"] == BY_TYPE["closures"]["Restrictions"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("short,expected_subject", [
|
||||||
|
("work_zone", "central.traffic.work_zone.us.id"),
|
||||||
|
("closure", "central.traffic.closure.us.id"),
|
||||||
|
("incident", "central.traffic.incident.us.id"),
|
||||||
|
("special_event", "central.traffic.special_event.us.id"),
|
||||||
|
("advisory", "central.traffic.advisory.us.id"),
|
||||||
|
])
|
||||||
|
def test_subject_for(adapter, short, expected_subject):
|
||||||
|
e = Event(id="x", adapter="itd_511", category=f"{short}.itd_511",
|
||||||
|
time=datetime.now(timezone.utc), severity=1, geo=Geo(), data={})
|
||||||
|
assert adapter.subject_for(e) == expected_subject
|
||||||
|
|
||||||
|
|
||||||
|
def test_advisory_structural_passthrough(adapter):
|
||||||
|
# Synthesize an advisory (ITD currently returns []); per-record try/except
|
||||||
|
# in poll() means downstream surprises won't sink the cycle.
|
||||||
|
rec = {"SourceId": "ADV-1", "Description": "Snow event in central Idaho",
|
||||||
|
"Latitude": 44.0, "Longitude": -114.5, "Reported": 1780500000}
|
||||||
|
e = adapter._build_advisory_record(rec)
|
||||||
|
assert e is not None
|
||||||
|
assert e.category == "advisory.itd_511"
|
||||||
|
assert e.id == "idaho_511:advisory:ADV-1"
|
||||||
|
assert e.data["advisory"] == rec # full pass-through, schema-free
|
||||||
|
assert e.data["latitude"] == 44.0
|
||||||
|
assert e.data["event_type_short"] == "advisory"
|
||||||
|
|
||||||
|
|
||||||
|
def test_advisory_returns_none_without_any_id(adapter):
|
||||||
|
assert adapter._build_advisory_record({"Description": "no id"}) is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_yields_events_from_both_endpoints(adapter):
|
||||||
|
await adapter.startup()
|
||||||
|
adapter._fetch = AsyncMock(side_effect=lambda ep: {"event": EVENT, "alerts": ALERTS}[ep])
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
# alerts fixture is [] so events == EVENT count
|
||||||
|
assert len(events) == len(EVENT)
|
||||||
|
assert all(e.adapter == "itd_511" for e in events)
|
||||||
|
assert {e.category for e in events} == {
|
||||||
|
"work_zone.itd_511", "closure.itd_511",
|
||||||
|
"incident.itd_511", "special_event.itd_511",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_advisory_cadence_throttles_alerts_endpoint(adapter):
|
||||||
|
"""Advisories poll on the 0th, 5th, 10th... event-poll (5x throttle)."""
|
||||||
|
await adapter.startup()
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
async def fake_fetch(ep):
|
||||||
|
calls.append(ep)
|
||||||
|
return EVENT if ep == "event" else ALERTS
|
||||||
|
|
||||||
|
adapter._fetch = fake_fetch
|
||||||
|
for _ in range(6):
|
||||||
|
[_ async for _ in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
# 6 polls: events every time, alerts on poll 0 and poll 5
|
||||||
|
assert calls.count("event") == 6
|
||||||
|
assert calls.count("alerts") == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_skips_cleanly_without_api_key(tmp_path):
|
||||||
|
cs = MagicMock()
|
||||||
|
cs.get_api_key = AsyncMock(return_value=None)
|
||||||
|
adapter = Itd511Adapter(_cfg(), cs, tmp_path / "cursors.db")
|
||||||
|
await adapter.startup()
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
assert events == [] # no fetch, clean skip per tomtom_flow precedent
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_key_never_leaks_in_error_path(adapter, caplog):
|
||||||
|
"""The key travels in ?key=, so aiohttp's default error messages include
|
||||||
|
the full URL; every error-log path must run through self._redact().
|
||||||
|
Regression guard via caplog inspection. NOTE: ``caplog.text`` only contains
|
||||||
|
the message field — structured ``extra={}`` kwargs land as attributes on
|
||||||
|
the LogRecord, so we inspect both surfaces."""
|
||||||
|
await adapter.startup()
|
||||||
|
key_value = adapter._api_key # the testkey set up in the adapter fixture
|
||||||
|
assert key_value and len(key_value) > 16
|
||||||
|
|
||||||
|
async def boom(endpoint):
|
||||||
|
raise aiohttp.ClientConnectionError(
|
||||||
|
f"Cannot connect to host 511.idaho.gov ssl:default [key={key_value}]"
|
||||||
|
)
|
||||||
|
|
||||||
|
adapter._fetch = boom
|
||||||
|
with caplog.at_level(logging.WARNING, logger="central.adapters.itd_511"):
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
assert events == []
|
||||||
|
surfaces = [r.getMessage() for r in caplog.records]
|
||||||
|
surfaces.extend(str(getattr(r, "error", "")) for r in caplog.records)
|
||||||
|
joined = " ".join(surfaces)
|
||||||
|
assert key_value not in joined, f"key leaked to log: {joined!r}"
|
||||||
|
assert "<KEY>" in joined # redaction marker proves _redact() actually fired
|
||||||
|
|
||||||
|
|
||||||
|
def test_inherits_dedup_mixin_from_source_adapter():
|
||||||
|
for m in ("is_published", "mark_published", "sweep_old_ids"):
|
||||||
|
assert m not in Itd511Adapter.__dict__, f"redefines {m}"
|
||||||
|
assert getattr(Itd511Adapter, m) is getattr(SourceAdapter, m)
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_partial_renders_per_event_type():
|
||||||
|
from central.gui.routes import _derive_subject
|
||||||
|
cases = [
|
||||||
|
({"event_type_short": "work_zone", "roadway_name": "I-84"},
|
||||||
|
"Road work on I-84"),
|
||||||
|
({"event_type_short": "closure", "roadway_name": "N McDermott Rd",
|
||||||
|
"direction": "Both", "is_full_closure": True},
|
||||||
|
"Closure on N McDermott Rd Both (full closure)"),
|
||||||
|
({"event_type_short": "incident", "roadway_name": "I-84",
|
||||||
|
"direction": "East"},
|
||||||
|
"Incident on I-84 East"),
|
||||||
|
({"event_type_short": "advisory"}, "Advisory"),
|
||||||
|
# Drops "Unknown" direction per wzdx lesson
|
||||||
|
({"event_type_short": "work_zone", "roadway_name": "I-84",
|
||||||
|
"direction": "Unknown"},
|
||||||
|
"Road work on I-84"),
|
||||||
|
]
|
||||||
|
for inner, expected in cases:
|
||||||
|
row = {"adapter": "itd_511", "data": {"data": {"data": inner}}}
|
||||||
|
assert _derive_subject(row) == expected, f"mismatch for {inner!r}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_class_attributes_match_spec():
|
||||||
|
assert Itd511Adapter.name == "itd_511"
|
||||||
|
assert Itd511Adapter.data_class == "event"
|
||||||
|
assert Itd511Adapter.requires_api_key == "idaho_511"
|
||||||
|
assert Itd511Adapter.api_key_field == "api_key_alias"
|
||||||
|
assert Itd511Adapter.default_cadence_s == 60
|
||||||
|
assert Itd511Adapter.wizard_order is None
|
||||||
|
assert Itd511Adapter.enrichment_locations == [("latitude", "longitude")]
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG C: 429 Retry-After must drive the wait directly; no double-sleep --
|
||||||
|
|
||||||
|
def test_transient_carries_wait_s():
|
||||||
|
t = _Transient("429 retry-after=42", wait_s=42)
|
||||||
|
assert t.wait_s == 42
|
||||||
|
assert str(t) == "429 retry-after=42"
|
||||||
|
assert _Transient("5xx").wait_s is None # default omits
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_strategy_honors_transient_wait_s():
|
||||||
|
"""BUG C regression: a 429 Retry-After must drive the wait directly via
|
||||||
|
_Transient.wait_s; tenacity must NOT also wait its exponential jitter on
|
||||||
|
top (the previous shape did both, blocking ~120s+ per cycle)."""
|
||||||
|
retry_state = MagicMock()
|
||||||
|
outcome = MagicMock()
|
||||||
|
outcome.exception.return_value = _Transient("429", wait_s=42)
|
||||||
|
retry_state.outcome = outcome
|
||||||
|
assert _wait_strategy(retry_state) == 42.0
|
||||||
|
outcome.exception.return_value = _Transient("429", wait_s=60)
|
||||||
|
assert _wait_strategy(retry_state) == 60.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_strategy_falls_back_for_transient_without_wait_s():
|
||||||
|
"""5xx _Transient (no Retry-After) falls through to exponential jitter."""
|
||||||
|
retry_state = MagicMock()
|
||||||
|
outcome = MagicMock()
|
||||||
|
outcome.exception.return_value = _Transient("503 server error") # wait_s None
|
||||||
|
retry_state.outcome = outcome
|
||||||
|
retry_state.attempt_number = 1
|
||||||
|
retry_state.idle_for = 0
|
||||||
|
wait = _wait_strategy(retry_state)
|
||||||
|
assert isinstance(wait, float) and wait >= 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_strategy_falls_back_for_non_transient():
|
||||||
|
"""Network errors (no wait_s) get exponential jitter."""
|
||||||
|
retry_state = MagicMock()
|
||||||
|
outcome = MagicMock()
|
||||||
|
outcome.exception.return_value = aiohttp.ClientConnectionError("net error")
|
||||||
|
retry_state.outcome = outcome
|
||||||
|
retry_state.attempt_number = 1
|
||||||
|
retry_state.idle_for = 0
|
||||||
|
wait = _wait_strategy(retry_state)
|
||||||
|
assert isinstance(wait, float) and wait >= 0
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG D3: assert→if-raise (asserts strip under python -O) -----------------
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_session_unset_raises_runtime_not_assert(adapter):
|
||||||
|
"""D3 regression: asserts strip under ``python -O``, so the session-not-
|
||||||
|
started precondition must be enforced with an explicit if-raise."""
|
||||||
|
assert adapter._session is None # precondition: not yet started
|
||||||
|
with pytest.raises(RuntimeError, match="session not started"):
|
||||||
|
await adapter._fetch("event")
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG D5: tenacity has no default logging hooks (audit guard) -------------
|
||||||
|
|
||||||
|
def test_tenacity_decorator_has_explicit_no_log_hooks():
|
||||||
|
"""D5 audit: tenacity's defaults (before_sleep=None, after=after_nothing)
|
||||||
|
have no logging — so the URL-with-key can't leak via the retry path. We
|
||||||
|
pin them explicitly on @retry; if a future tenacity upgrade changes the
|
||||||
|
defaults, this test fails loudly. Also confirms reraise=True so we get
|
||||||
|
_Transient/ClientError verbatim instead of RetryError."""
|
||||||
|
from tenacity import after_nothing, before_nothing
|
||||||
|
retrying = Itd511Adapter._fetch.retry
|
||||||
|
assert retrying.before_sleep is None
|
||||||
|
assert retrying.after is after_nothing
|
||||||
|
assert retrying.before is before_nothing
|
||||||
|
assert retrying.reraise is True
|
||||||
206
tests/test_itd_511_cameras.py
Normal file
206
tests/test_itd_511_cameras.py
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
"""Tests for the itd_511_cameras adapter (v0.10.0).
|
||||||
|
|
||||||
|
Fixture covers 4 cameras: ITDNET, ACHD, UDOT (cross-border per v0.10.0
|
||||||
|
finding 4), and an RWIS multi-view (to exercise the additional_views capture):
|
||||||
|
tests/fixtures/itd_511_cameras_sample.json
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.adapters.itd_511 import _Transient
|
||||||
|
from central.adapters.itd_511_cameras import NATIVE_SOURCES, Itd511CamerasAdapter
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
|
||||||
|
FIX = Path(__file__).parent / "fixtures"
|
||||||
|
CAMS = json.loads((FIX / "itd_511_cameras_sample.json").read_text())
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg():
|
||||||
|
return AdapterConfig(
|
||||||
|
name="itd_511_cameras", enabled=True, cadence_s=600,
|
||||||
|
settings={"api_key_alias": "idaho_511"},
|
||||||
|
updated_at=datetime.now(timezone.utc),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def adapter(tmp_path):
|
||||||
|
cs = MagicMock()
|
||||||
|
cs.get_api_key = AsyncMock(return_value="testkey-32chars-deadbeefdeadbeef")
|
||||||
|
return Itd511CamerasAdapter(_cfg(), cs, tmp_path / "cursors.db")
|
||||||
|
|
||||||
|
|
||||||
|
def test_class_attributes_match_spec():
|
||||||
|
assert Itd511CamerasAdapter.name == "itd_511_cameras"
|
||||||
|
assert Itd511CamerasAdapter.data_class == "telemetry"
|
||||||
|
assert Itd511CamerasAdapter.requires_api_key == "idaho_511"
|
||||||
|
assert Itd511CamerasAdapter.default_cadence_s == 600
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_category_and_subject(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0])
|
||||||
|
assert e.category == "camera.itd_511_cameras"
|
||||||
|
assert e.adapter == "itd_511_cameras"
|
||||||
|
assert adapter.subject_for(e) == f"central.traffic_cameras.us.id.{CAMS[0]['Id']}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_id_per_utc_day(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0])
|
||||||
|
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||||
|
assert e.id == f"idaho_511:cam:{CAMS[0]['Id']}:{today}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_url_passthrough(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0])
|
||||||
|
assert e.data["image_url"] == CAMS[0]["Views"][0]["Url"]
|
||||||
|
assert e.data["image_url"].startswith("https://511.idaho.gov/map/Cctv/")
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_jurisdiction_preserves_border_cameras(adapter):
|
||||||
|
"""Per v0.10.0 finding 4: ITD aggregates ~1.2% cross-DOT mirrors (UDOT,
|
||||||
|
ODOT, WYDOT, ...). Region stays US-ID; source_jurisdiction preserves the
|
||||||
|
raw upstream Source value for downstream re-bucketing."""
|
||||||
|
udot = next((c for c in CAMS if c["Source"] == "UDOT"), None)
|
||||||
|
assert udot is not None, "fixture must include a UDOT cross-border camera"
|
||||||
|
e = adapter._build_event(udot)
|
||||||
|
assert e.data["source_jurisdiction"] == "UDOT"
|
||||||
|
assert e.data["source"] == "UDOT"
|
||||||
|
assert e.geo.primary_region == "US-ID" # uniform Idaho tagging per locked decision
|
||||||
|
assert e.geo.regions == ["US-ID"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_multiple_views_captured(adapter):
|
||||||
|
multi = next((c for c in CAMS if len(c.get("Views") or []) > 1), None)
|
||||||
|
assert multi is not None, "fixture must include a multi-view camera"
|
||||||
|
e = adapter._build_event(multi)
|
||||||
|
assert e.data["view_count"] == len(multi["Views"])
|
||||||
|
assert e.data["additional_views"] == [v["Url"] for v in multi["Views"][1:]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_view_has_empty_additional_views(adapter):
|
||||||
|
single = next((c for c in CAMS if len(c.get("Views") or []) == 1), None)
|
||||||
|
assert single is not None, "fixture must include a single-view camera"
|
||||||
|
e = adapter._build_event(single)
|
||||||
|
assert e.data["additional_views"] == []
|
||||||
|
assert e.data["view_count"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_returns_none_without_id(adapter):
|
||||||
|
assert adapter._build_event({"Source": "ITDNET"}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_severity_is_1_for_telemetry(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0])
|
||||||
|
assert e.severity == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_yields_one_event_per_camera(adapter):
|
||||||
|
await adapter.startup()
|
||||||
|
adapter._fetch_cameras = AsyncMock(return_value=CAMS)
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
assert len(events) == len(CAMS)
|
||||||
|
assert all(e.adapter == "itd_511_cameras" for e in events)
|
||||||
|
assert all(e.category == "camera.itd_511_cameras" for e in events)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_skips_cleanly_without_api_key(tmp_path):
|
||||||
|
cs = MagicMock()
|
||||||
|
cs.get_api_key = AsyncMock(return_value=None)
|
||||||
|
adapter = Itd511CamerasAdapter(_cfg(), cs, tmp_path / "cursors.db")
|
||||||
|
await adapter.startup()
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
assert events == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_partial_renders():
|
||||||
|
from central.gui.routes import _derive_subject
|
||||||
|
inner = {"camera_id": 1, "location": "I-15 UT/ID State Line UT", "roadway": "I-15"}
|
||||||
|
row = {"adapter": "itd_511_cameras", "data": {"data": {"data": inner}}}
|
||||||
|
assert _derive_subject(row) == "Camera: I-15 UT/ID State Line UT"
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_partial_falls_back_to_id_when_location_missing():
|
||||||
|
from central.gui.routes import _derive_subject
|
||||||
|
inner = {"camera_id": 42}
|
||||||
|
row = {"adapter": "itd_511_cameras", "data": {"data": {"data": inner}}}
|
||||||
|
assert _derive_subject(row) == "Camera: #42"
|
||||||
|
|
||||||
|
|
||||||
|
def test_inherits_dedup_mixin_from_source_adapter():
|
||||||
|
for m in ("is_published", "mark_published", "sweep_old_ids"):
|
||||||
|
assert m not in Itd511CamerasAdapter.__dict__, f"redefines {m}"
|
||||||
|
assert getattr(Itd511CamerasAdapter, m) is getattr(SourceAdapter, m)
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG E: poll() must catch _Transient (tenacity reraise after retries) ---
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_catches_transient_after_exhausted_retries(adapter):
|
||||||
|
"""BUG E regression: cameras.poll() except tuple must include _Transient
|
||||||
|
so tenacity's reraise of a persistent 5xx or 429 (after exhausted
|
||||||
|
retries) does NOT crash the whole poll cycle."""
|
||||||
|
await adapter.startup()
|
||||||
|
|
||||||
|
async def boom_transient():
|
||||||
|
raise _Transient("503 persistent")
|
||||||
|
|
||||||
|
adapter._fetch_cameras = boom_transient
|
||||||
|
events = [e async for e in adapter.poll()]
|
||||||
|
await adapter.shutdown()
|
||||||
|
assert events == [] # poll exited cleanly, didn't raise
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG D2: NATIVE_SOURCES allow-list lives at the adapter, not the partial -
|
||||||
|
|
||||||
|
def test_native_sources_module_constant():
|
||||||
|
assert NATIVE_SOURCES == frozenset({"ITDNET", "Idaho511", "ACHD", "RWIS"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_native_source_flag_set_per_camera(adapter):
|
||||||
|
"""Border-region UDOT camera is non-native; ITDNET is native."""
|
||||||
|
udot = next(c for c in CAMS if c["Source"] == "UDOT")
|
||||||
|
itdnet = next(c for c in CAMS if c["Source"] == "ITDNET")
|
||||||
|
assert adapter._build_event(udot).data["is_native_source"] is False
|
||||||
|
assert adapter._build_event(itdnet).data["is_native_source"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_partial_does_not_hardcode_source_list(adapter):
|
||||||
|
"""D2 regression: the cross-DOT-mirror annotation is driven by
|
||||||
|
data.is_native_source — the partial must NOT carry the source allow-list
|
||||||
|
itself ([[feedback_no_hardcoding]])."""
|
||||||
|
from central.gui.routes import _get_templates
|
||||||
|
tmpl = _get_templates().env.get_template("_event_rows/itd_511_cameras.html")
|
||||||
|
udot_evt = adapter._build_event(next(c for c in CAMS if c["Source"] == "UDOT"))
|
||||||
|
itdnet_evt = adapter._build_event(next(c for c in CAMS if c["Source"] == "ITDNET"))
|
||||||
|
udot_html = tmpl.render(event={"data": {"data": {"data": udot_evt.data}}})
|
||||||
|
itdnet_html = tmpl.render(event={"data": {"data": {"data": itdnet_evt.data}}})
|
||||||
|
assert "(cross-DOT mirror)" in udot_html
|
||||||
|
assert "(cross-DOT mirror)" not in itdnet_html
|
||||||
|
# Audit: the partial source on disk must not contain the allow-list.
|
||||||
|
partial = Path(__file__).resolve().parents[1] / (
|
||||||
|
"src/central/gui/templates/_event_rows/itd_511_cameras.html"
|
||||||
|
)
|
||||||
|
text = partial.read_text()
|
||||||
|
for name in NATIVE_SOURCES:
|
||||||
|
assert name not in text, f"partial hardcodes {name}"
|
||||||
|
|
||||||
|
|
||||||
|
# --- BUG D3: assert→if-raise on the cameras sibling --------------------------
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_session_unset_raises_runtime(adapter):
|
||||||
|
"""D3 regression: asserts strip under python -O; the session-not-started
|
||||||
|
precondition must hold even with optimizations."""
|
||||||
|
assert adapter._session is None
|
||||||
|
with pytest.raises(RuntimeError, match="session not started"):
|
||||||
|
await adapter._fetch_cameras()
|
||||||
|
|
@ -11,7 +11,7 @@ from central.adapter_discovery import discover_adapters
|
||||||
from central.gui import routes
|
from central.gui import routes
|
||||||
|
|
||||||
# Adapters with data_class="telemetry" (the pinned split; grow as telemetry adapters land).
|
# Adapters with data_class="telemetry" (the pinned split; grow as telemetry adapters land).
|
||||||
_TELEMETRY = ["nwis", "state_511_atis_cameras", "tomtom_flow"]
|
_TELEMETRY = ["itd_511_cameras", "nwis", "state_511_atis_cameras", "tomtom_flow"]
|
||||||
|
|
||||||
|
|
||||||
# --- data_class defaults / registry split -----------------------------------
|
# --- data_class defaults / registry split -----------------------------------
|
||||||
|
|
|
||||||
11
uv.lock
generated
11
uv.lock
generated
|
|
@ -185,6 +185,7 @@ dependencies = [
|
||||||
{ name = "jinja2" },
|
{ name = "jinja2" },
|
||||||
{ name = "mapbox-vector-tile" },
|
{ name = "mapbox-vector-tile" },
|
||||||
{ name = "nats-py" },
|
{ name = "nats-py" },
|
||||||
|
{ name = "polyline" },
|
||||||
{ name = "pydantic" },
|
{ name = "pydantic" },
|
||||||
{ name = "pydantic-settings" },
|
{ name = "pydantic-settings" },
|
||||||
{ name = "python-multipart" },
|
{ name = "python-multipart" },
|
||||||
|
|
@ -214,6 +215,7 @@ requires-dist = [
|
||||||
{ name = "jinja2", specifier = ">=3.1.6" },
|
{ name = "jinja2", specifier = ">=3.1.6" },
|
||||||
{ name = "mapbox-vector-tile", specifier = ">=2.0" },
|
{ name = "mapbox-vector-tile", specifier = ">=2.0" },
|
||||||
{ name = "nats-py", specifier = ">=2.14.0" },
|
{ name = "nats-py", specifier = ">=2.14.0" },
|
||||||
|
{ name = "polyline", specifier = ">=2.0,<3" },
|
||||||
{ name = "pydantic", specifier = ">=2,<3" },
|
{ name = "pydantic", specifier = ">=2,<3" },
|
||||||
{ name = "pydantic-settings", specifier = ">=2.7.0" },
|
{ name = "pydantic-settings", specifier = ">=2.7.0" },
|
||||||
{ name = "python-multipart", specifier = ">=0.0.20" },
|
{ name = "python-multipart", specifier = ">=0.0.20" },
|
||||||
|
|
@ -648,6 +650,15 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
|
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "polyline"
|
||||||
|
version = "2.0.4"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/2c/c1/1a6ee4f9f02a55b2a9241bbadd342970160f7e42423307f21ee8f5530d4e/polyline-2.0.4.tar.gz", hash = "sha256:f05ade694522bf1720febebe1672f820f43a13c6a1664751e7769d47e8ca9b1b", size = 8261, upload-time = "2025-12-02T17:55:22.735Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/34/a8/4ebd3cb31d380e018efb1c8bf92664b196a41aba19506015b682af2587b9/polyline-2.0.4-py3-none-any.whl", hash = "sha256:a4e0c15b8ecb32915559f8cf210f1f8c2f5cc53d3cd32c91d7c1668d6e936e10", size = 7167, upload-time = "2025-12-02T17:55:20.323Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "propcache"
|
name = "propcache"
|
||||||
version = "0.5.2"
|
version = "0.5.2"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue