mirror of
https://github.com/zvx-echo6/central.git
synced 2026-06-10 11:54:37 +02:00
feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6)
New CENTRAL_TRAFFIC_CAMERAS stream + state_511_atis_cameras adapter. Telemetry
half of Castle Rock (events shipped in v0.9.2). Each Idaho camera -> one
telemetry event on /telemetry; detail drawer renders <img> direct from the
source (no blob storage / proxy in Central -- URL only).
supervisor + gui + ARCHIVE restart (NEW event-bearing stream
central.traffic_cameras.>). Ships disabled; public-unauth (no api key). Idaho only.
- Full camera list via POST /List/GetData/Cameras (DataTables), PAGINATED at
100/page (Idaho ~455 = 5 pages). GetUserCameras was a red herring (4 default
cams). The 100-row page cap also means v0.9.2 state_511_atis silently
truncates its 114-row Construction layer -> separate v0.9.7 fix.
- Subject central.traffic_cameras.{state}.{camera_id}; category
camera.state_511_atis_cameras -> GUI event_type "camera". data_class=telemetry.
- Per-UTC-day dedup {state}:cam:{id}:{YYYY-MM-DD}: one event per camera per day
-- always shows today's cameras, no per-poll flooding, no retention
coordination. Inherits the v0.9.1 dedup mixin.
- All sources included (Idaho511/ITDNET/RWIS/UDOT/ODOT/WYDOT/MTD border cameras);
source surfaced in data + the drawer for provenance. WKT POINT (lon lat) -> geo.
- No upstream image-capture timestamp (lastUpdated is config-edit time); drawer
shows no false "Captured" line. Cadence 600s. Severity 1 (telemetry).
Full suite: 829 passed, 1 skipped (central and unprivileged zvx, 3x each).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
d241bfea26
commit
02bc692bda
12 changed files with 461 additions and 3 deletions
|
|
@ -133,6 +133,7 @@ Central's archive.
|
||||||
| `CENTRAL_HYDRO` | `central.hydro.>` | 7 | 1 GiB | ✓ | ✓ |
|
| `CENTRAL_HYDRO` | `central.hydro.>` | 7 | 1 GiB | ✓ | ✓ |
|
||||||
| `CENTRAL_TRAFFIC` | `central.traffic.>` | 7 | 1 GiB | ✓ | ✓ |
|
| `CENTRAL_TRAFFIC` | `central.traffic.>` | 7 | 1 GiB | ✓ | ✓ |
|
||||||
| `CENTRAL_TRAFFIC_FLOW` | `central.traffic_flow.>` | 7 | 1 GiB | ✓ | ✓ |
|
| `CENTRAL_TRAFFIC_FLOW` | `central.traffic_flow.>` | 7 | 1 GiB | ✓ | ✓ |
|
||||||
|
| `CENTRAL_TRAFFIC_CAMERAS` | `central.traffic_cameras.>` | 7 | 1 GiB | ✓ | ✓ |
|
||||||
| `CENTRAL_META` | `central.meta.>` | 1 | 1 GiB | — | ✓ |
|
| `CENTRAL_META` | `central.meta.>` | 1 | 1 GiB | — | ✓ |
|
||||||
|
|
||||||
Retention and storage caps are migration-seeded defaults visible in `config.streams`;
|
Retention and storage caps are migration-seeded defaults visible in `config.streams`;
|
||||||
|
|
@ -1518,6 +1519,44 @@ road name, description, county, severity). Verified for Idaho only.
|
||||||
- **Removal semantics:** none in v1. Events age out of the upstream feed; the
|
- **Removal semantics:** none in v1. Events age out of the upstream feed; the
|
||||||
14-day dedup sweep expires stale ids.
|
14-day dedup sweep expires stale ids.
|
||||||
|
|
||||||
|
### state_511_atis_cameras — State 511 traffic cameras (Castle Rock ATIS, telemetry)
|
||||||
|
|
||||||
|
State-DOT 511 traffic cameras (Idaho). One telemetry event per camera per UTC day;
|
||||||
|
the `/telemetry` detail drawer renders the live image inline (`<img>` fetched
|
||||||
|
direct from the source -- Central stores the URL, never the image bytes). Pairs
|
||||||
|
with state_511_atis incidents: see an incident, click a nearby camera, see road
|
||||||
|
conditions.
|
||||||
|
|
||||||
|
- **Stream:** `CENTRAL_TRAFFIC_CAMERAS` (telemetry; `/telemetry` tab).
|
||||||
|
- **Subject pattern:** `central.traffic_cameras.<state>.<camera_id>` -- subscribe to
|
||||||
|
one camera or `central.traffic_cameras.id.>` for all Idaho.
|
||||||
|
- **GUI event_type:** `camera` (from `category = "camera.state_511_atis_cameras"`).
|
||||||
|
- **Source:** full state list via `POST /List/GetData/Cameras` (DataTables,
|
||||||
|
**paginated** at 100/page; Idaho ~455). Public-unauth. **Cadence 600s.**
|
||||||
|
- **Dedup key shape:** `<state_code>:cam:<camera_id>:<YYYY-MM-DD>` -- one event per
|
||||||
|
camera per UTC day. The table always shows today's cameras; no per-poll flooding
|
||||||
|
and no dedup-window/retention coordination needed.
|
||||||
|
- **Event.data fields:**
|
||||||
|
|
||||||
|
| key | type | nullable | description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `camera_id` | int | no | Stable upstream id |
|
||||||
|
| `roadway_name` | str | yes | e.g. `I-84` |
|
||||||
|
| `location_description` | str | yes | e.g. `I-84 Mountain Home` (embeds the roadway) |
|
||||||
|
| `direction` | str | yes | `North` / `Unknown` / ... |
|
||||||
|
| `source` | str | yes | Owning agency: `Idaho511`, `ITDNET`, `RWIS`, `UDOT`, `ODOT`, ... (border cameras Idaho 511 surfaces) |
|
||||||
|
| `image_url` | str | yes | Full live image URL (`<base_url>/map/Cctv/<id>`); browser fetches direct |
|
||||||
|
| `image_count` | int | no | Number of camera angles (1-4) |
|
||||||
|
| `record_updated` | str | yes | Camera config edit time (NOT image-capture time) |
|
||||||
|
| `state_code` | str | no | Routing |
|
||||||
|
| `latitude` / `longitude` | float | yes | From the WKT `POINT (lon lat)` |
|
||||||
|
|
||||||
|
- **Severity:** always `1` (cameras have no severity signal; telemetry styling).
|
||||||
|
- **Decipherable as-is:** yes -- location + source are user-ready; the live image
|
||||||
|
is one click away. **No image-capture timestamp** is available upstream (the
|
||||||
|
`lastUpdated` field is camera-config time), so the drawer shows no "captured at".
|
||||||
|
- **Removal semantics:** none; offline cameras serve an empty image but stay listed.
|
||||||
|
|
||||||
### tomtom_incidents — TomTom real-time traffic incidents (commercial coverage)
|
### tomtom_incidents — TomTom real-time traffic incidents (commercial coverage)
|
||||||
|
|
||||||
Real-time incidents (closures, jams, hazards, road work, accidents) from TomTom
|
Real-time incidents (closures, jams, hazards, road work, accidents) from TomTom
|
||||||
|
|
|
||||||
|
|
@ -362,7 +362,7 @@ central.<domain>.<subtype>[.<dimensions>...]
|
||||||
```
|
```
|
||||||
|
|
||||||
- `<domain>` is one of `wx`, `fire`, `quake`, `space`, `disaster`, `hydro`,
|
- `<domain>` is one of `wx`, `fire`, `quake`, `space`, `disaster`, `hydro`,
|
||||||
`traffic`, `traffic_flow`, `meta` (the current set — see [§8](#8-the-streamentry-registry) for adding
|
`traffic`, `traffic_flow`, `traffic_cameras`, `meta` (the current set — see [§8](#8-the-streamentry-registry) for adding
|
||||||
one). Operators MUST be able to subscribe to all of one domain with
|
one). Operators MUST be able to subscribe to all of one domain with
|
||||||
`central.<domain>.>`.
|
`central.<domain>.>`.
|
||||||
- `<subtype>` is adapter-driven and identifies the event category within the
|
- `<subtype>` is adapter-driven and identifies the event category within the
|
||||||
|
|
@ -552,6 +552,7 @@ STREAMS: list[StreamEntry] = [
|
||||||
StreamEntry("CENTRAL_HYDRO", "central.hydro.>"),
|
StreamEntry("CENTRAL_HYDRO", "central.hydro.>"),
|
||||||
StreamEntry("CENTRAL_TRAFFIC", "central.traffic.>"),
|
StreamEntry("CENTRAL_TRAFFIC", "central.traffic.>"),
|
||||||
StreamEntry("CENTRAL_TRAFFIC_FLOW", "central.traffic_flow.>"),
|
StreamEntry("CENTRAL_TRAFFIC_FLOW", "central.traffic_flow.>"),
|
||||||
|
StreamEntry("CENTRAL_TRAFFIC_CAMERAS", "central.traffic_cameras.>"),
|
||||||
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
|
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
|
||||||
]
|
]
|
||||||
```
|
```
|
||||||
|
|
|
||||||
19
sql/migrations/029_add_state_511_atis_cameras_adapter.sql
Normal file
19
sql/migrations/029_add_state_511_atis_cameras_adapter.sql
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
-- Migration: 029_add_state_511_atis_cameras_adapter
|
||||||
|
-- Adds the CENTRAL_TRAFFIC_CAMERAS JetStream stream (telemetry; central.traffic_cameras.>)
|
||||||
|
-- AND the state_511_atis_cameras adapter row. NEW event-bearing stream ->
|
||||||
|
-- central-archive restart required at deploy (feedback_new_stream_needs_archive_restart).
|
||||||
|
-- 7-day retention. Ships disabled; public-unauth (no api key). Idaho only.
|
||||||
|
-- Additive-only: idempotent via ON CONFLICT DO NOTHING.
|
||||||
|
|
||||||
|
INSERT INTO config.streams (name, max_age_s, max_bytes)
|
||||||
|
VALUES ('CENTRAL_TRAFFIC_CAMERAS', 604800, 1073741824)
|
||||||
|
ON CONFLICT (name) DO NOTHING;
|
||||||
|
|
||||||
|
INSERT INTO config.adapters (name, enabled, cadence_s, settings)
|
||||||
|
VALUES (
|
||||||
|
'state_511_atis_cameras',
|
||||||
|
false,
|
||||||
|
600,
|
||||||
|
'{"states": [{"code": "ID", "base_url": "https://511.idaho.gov"}]}'::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (name) DO NOTHING;
|
||||||
249
src/central/adapters/state_511_atis_cameras.py
Normal file
249
src/central/adapters/state_511_atis_cameras.py
Normal file
|
|
@ -0,0 +1,249 @@
|
||||||
|
"""State 511 (Castle Rock ATIS) cameras adapter — telemetry.
|
||||||
|
|
||||||
|
Polls the full state camera directory via POST /List/GetData/Cameras (paginated,
|
||||||
|
100/page), emitting one telemetry Event per camera to CENTRAL_TRAFFIC_CAMERAS
|
||||||
|
(subject central.traffic_cameras.{state}.{camera_id}). data_class="telemetry" ->
|
||||||
|
the /telemetry tab. The detail drawer renders <img> straight from the upstream
|
||||||
|
image URL -- no blob storage or proxy in Central. Idaho-only; templatized per
|
||||||
|
state via settings, same shape as state_511_atis.
|
||||||
|
|
||||||
|
Dedup is per-UTC-day ({state}:cam:{id}:{YYYY-MM-DD}): one event per camera per day
|
||||||
|
-- the table always shows today's cameras, with no dedup-window/retention
|
||||||
|
coordination and no per-poll flooding. Inherits the v0.9.1 dedup mixin.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from tenacity import (
|
||||||
|
retry,
|
||||||
|
retry_if_exception_type,
|
||||||
|
stop_after_attempt,
|
||||||
|
wait_exponential_jitter,
|
||||||
|
)
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
from central.config_store import ConfigStore
|
||||||
|
from central.models import Event, Geo
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_PAGE_LENGTH = 100 # Castle Rock caps the DataTables page at 100 rows
|
||||||
|
_MAX_PAGES = 20 # defensive ceiling (~2,000 cameras)
|
||||||
|
_XHR = {"X-Requested-With": "XMLHttpRequest"}
|
||||||
|
_FETCH_CONCURRENCY = 4
|
||||||
|
_FETCH_TIMEOUT_S = 30
|
||||||
|
_WKT_POINT = re.compile(r"POINT \(([-0-9.]+) ([-0-9.]+)\)")
|
||||||
|
|
||||||
|
_DEDUP_DDL = (
|
||||||
|
"CREATE TABLE IF NOT EXISTS published_ids ("
|
||||||
|
"adapter TEXT NOT NULL, event_id TEXT NOT NULL, "
|
||||||
|
"first_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"last_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
|
||||||
|
"PRIMARY KEY (adapter, event_id))"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_wkt(wkt: str | None) -> tuple[float | None, float | None]:
|
||||||
|
"""'POINT (lon lat)' -> (lat, lon); (None, None) on failure."""
|
||||||
|
if not wkt:
|
||||||
|
return (None, None)
|
||||||
|
m = _WKT_POINT.match(wkt)
|
||||||
|
if not m:
|
||||||
|
return (None, None)
|
||||||
|
try:
|
||||||
|
return (float(m.group(2)), float(m.group(1)))
|
||||||
|
except ValueError:
|
||||||
|
return (None, None)
|
||||||
|
|
||||||
|
|
||||||
|
class StateConfig(BaseModel):
|
||||||
|
code: str
|
||||||
|
base_url: str
|
||||||
|
|
||||||
|
|
||||||
|
class State511CamerasSettings(BaseModel):
|
||||||
|
"""states: Castle Rock 511 deployments to poll for cameras."""
|
||||||
|
|
||||||
|
states: list[StateConfig] = []
|
||||||
|
|
||||||
|
|
||||||
|
class State511ATISCamerasAdapter(SourceAdapter):
|
||||||
|
"""Castle Rock ATIS 511 camera directory adapter (telemetry)."""
|
||||||
|
|
||||||
|
name = "state_511_atis_cameras"
|
||||||
|
display_name = "State 511 Cameras (Castle Rock ATIS)"
|
||||||
|
description = (
|
||||||
|
"State DOT 511 traffic cameras from the Castle Rock ATIS platform. One "
|
||||||
|
"telemetry event per camera (per UTC day); the detail drawer shows the live "
|
||||||
|
"image direct from the source. Verified for Idaho."
|
||||||
|
)
|
||||||
|
settings_schema = State511CamerasSettings
|
||||||
|
requires_api_key = None
|
||||||
|
api_key_field = None
|
||||||
|
wizard_order = None # Ships disabled
|
||||||
|
default_cadence_s = 600
|
||||||
|
data_class = "telemetry"
|
||||||
|
enrichment_locations = [("latitude", "longitude")]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
config: AdapterConfig,
|
||||||
|
config_store: ConfigStore,
|
||||||
|
cursor_db_path: Path,
|
||||||
|
) -> None:
|
||||||
|
self._config_store = config_store
|
||||||
|
self._cursor_db_path = cursor_db_path
|
||||||
|
self._session: aiohttp.ClientSession | None = None
|
||||||
|
self._db: sqlite3.Connection | None = None
|
||||||
|
self._states: list[StateConfig] = self._read_states(config)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _read_states(config: AdapterConfig) -> list[StateConfig]:
|
||||||
|
return [StateConfig(**s) for s in (config.settings.get("states") or [])]
|
||||||
|
|
||||||
|
async def startup(self) -> None:
|
||||||
|
self._session = aiohttp.ClientSession(
|
||||||
|
timeout=aiohttp.ClientTimeout(total=_FETCH_TIMEOUT_S),
|
||||||
|
headers={"User-Agent": "Central/0.9 (+state_511_atis_cameras)"},
|
||||||
|
)
|
||||||
|
self._db = sqlite3.connect(self._cursor_db_path)
|
||||||
|
self._db.execute(_DEDUP_DDL)
|
||||||
|
self._db.execute("CREATE INDEX IF NOT EXISTS published_ids_last_seen ON published_ids (last_seen)")
|
||||||
|
self._db.commit()
|
||||||
|
logger.info("state_511_atis_cameras adapter started",
|
||||||
|
extra={"states": [s.code for s in self._states]})
|
||||||
|
|
||||||
|
async def shutdown(self) -> None:
|
||||||
|
if self._session:
|
||||||
|
await self._session.close()
|
||||||
|
self._session = None
|
||||||
|
if self._db:
|
||||||
|
self._db.close()
|
||||||
|
self._db = None
|
||||||
|
|
||||||
|
async def apply_config(self, new_config: AdapterConfig) -> None:
|
||||||
|
self._states = self._read_states(new_config)
|
||||||
|
logger.info("state_511_atis_cameras config updated",
|
||||||
|
extra={"states": [s.code for s in self._states]})
|
||||||
|
|
||||||
|
@retry(
|
||||||
|
stop=stop_after_attempt(3),
|
||||||
|
wait=wait_exponential_jitter(initial=1, max=30),
|
||||||
|
retry=retry_if_exception_type((aiohttp.ClientError, TimeoutError)),
|
||||||
|
)
|
||||||
|
async def _fetch_page(self, base_url: str, start: int) -> dict[str, Any]:
|
||||||
|
assert self._session is not None
|
||||||
|
body = {
|
||||||
|
"draw": "1", "start": str(start), "length": str(_PAGE_LENGTH),
|
||||||
|
"columns[0][data]": "0", "order[0][column]": "0",
|
||||||
|
"order[0][dir]": "asc", "search[value]": "",
|
||||||
|
}
|
||||||
|
async with self._session.post(
|
||||||
|
f"{base_url}/List/GetData/Cameras", data=body, headers=_XHR
|
||||||
|
) as resp:
|
||||||
|
resp.raise_for_status()
|
||||||
|
return await resp.json(content_type=None)
|
||||||
|
|
||||||
|
async def _fetch_all(self, base_url: str) -> list[dict[str, Any]]:
|
||||||
|
"""Paginate the camera directory (100/page) -> all rows. [] on failure."""
|
||||||
|
try:
|
||||||
|
first = await self._fetch_page(base_url, 0)
|
||||||
|
except (aiohttp.ClientError, TimeoutError, asyncio.TimeoutError, ValueError) as exc:
|
||||||
|
logger.warning("state_511_atis_cameras page fetch failed",
|
||||||
|
extra={"base_url": base_url, "start": 0, "error": str(exc)})
|
||||||
|
return []
|
||||||
|
total = first.get("recordsTotal") or 0
|
||||||
|
rows = list(first.get("data") or [])
|
||||||
|
start = _PAGE_LENGTH
|
||||||
|
pages = 1
|
||||||
|
while start < total and pages < _MAX_PAGES:
|
||||||
|
try:
|
||||||
|
page = await self._fetch_page(base_url, start)
|
||||||
|
except (aiohttp.ClientError, TimeoutError, asyncio.TimeoutError, ValueError) as exc:
|
||||||
|
logger.warning("state_511_atis_cameras page fetch failed",
|
||||||
|
extra={"base_url": base_url, "start": start, "error": str(exc)})
|
||||||
|
break
|
||||||
|
rows.extend(page.get("data") or [])
|
||||||
|
start += _PAGE_LENGTH
|
||||||
|
pages += 1
|
||||||
|
return rows
|
||||||
|
|
||||||
|
def _build_event(self, cam: dict[str, Any], state: StateConfig) -> Event | None:
|
||||||
|
cam_id = cam.get("id")
|
||||||
|
if cam_id is None:
|
||||||
|
return None
|
||||||
|
lat, lon = _parse_wkt((cam.get("latLng") or {}).get("geography", {}).get("wellKnownText"))
|
||||||
|
images = cam.get("images") or []
|
||||||
|
image_url = None
|
||||||
|
if images and images[0].get("imageUrl"):
|
||||||
|
image_url = state.base_url + images[0]["imageUrl"]
|
||||||
|
day = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||||
|
return Event(
|
||||||
|
id=f"{state.code}:cam:{cam_id}:{day}",
|
||||||
|
adapter=self.name,
|
||||||
|
category="camera.state_511_atis_cameras",
|
||||||
|
time=datetime.now(timezone.utc),
|
||||||
|
severity=1, # telemetry; cameras have no severity signal
|
||||||
|
geo=Geo(
|
||||||
|
centroid=(lon, lat) if lat is not None and lon is not None else None,
|
||||||
|
regions=[f"US-{state.code}"],
|
||||||
|
primary_region=f"US-{state.code}",
|
||||||
|
),
|
||||||
|
data={
|
||||||
|
"camera_id": cam_id,
|
||||||
|
"roadway_name": cam.get("roadway"),
|
||||||
|
"location_description": cam.get("location"),
|
||||||
|
"direction": cam.get("direction"),
|
||||||
|
"source": cam.get("source"),
|
||||||
|
"image_url": image_url,
|
||||||
|
"image_count": len(images),
|
||||||
|
"record_updated": cam.get("lastUpdated"), # camera config edit time (not capture time)
|
||||||
|
"state_code": state.code,
|
||||||
|
"latitude": lat,
|
||||||
|
"longitude": lon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
async def poll(self) -> AsyncIterator[Event]:
|
||||||
|
if not self._session:
|
||||||
|
raise RuntimeError("Session not initialized")
|
||||||
|
sem = asyncio.Semaphore(_FETCH_CONCURRENCY)
|
||||||
|
|
||||||
|
async def _one(state: StateConfig) -> list[Event]:
|
||||||
|
async with sem:
|
||||||
|
cams = await self._fetch_all(state.base_url)
|
||||||
|
out: list[Event] = []
|
||||||
|
for cam in cams:
|
||||||
|
try:
|
||||||
|
ev = self._build_event(cam, state)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("state_511_atis_cameras parse failed", extra={"state": state.code})
|
||||||
|
continue
|
||||||
|
if ev is not None:
|
||||||
|
out.append(ev)
|
||||||
|
return out
|
||||||
|
|
||||||
|
results = await asyncio.gather(*[_one(s) for s in self._states])
|
||||||
|
yielded = 0
|
||||||
|
for evs in results:
|
||||||
|
for ev in evs:
|
||||||
|
yield ev
|
||||||
|
yielded += 1
|
||||||
|
|
||||||
|
self.sweep_old_ids()
|
||||||
|
logger.info("state_511_atis_cameras poll completed", extra={"events_yielded": yielded})
|
||||||
|
|
||||||
|
def subject_for(self, event: Event) -> str:
|
||||||
|
d = event.data
|
||||||
|
code = (d.get("state_code") or "").lower() or "unknown"
|
||||||
|
return f"central.traffic_cameras.{code}.{d.get('camera_id')}"
|
||||||
|
|
@ -2658,7 +2658,7 @@ ADAPTER_GROUPS = {
|
||||||
"Space": ["swpc_alerts", "swpc_kindex", "swpc_protons"],
|
"Space": ["swpc_alerts", "swpc_kindex", "swpc_protons"],
|
||||||
"Geophysical": ["usgs_quake", "nwis"],
|
"Geophysical": ["usgs_quake", "nwis"],
|
||||||
"Earth Observation": ["eonet"],
|
"Earth Observation": ["eonet"],
|
||||||
"Transportation": ["wzdx", "state_511_atis", "tomtom_flow", "tomtom_incidents"],
|
"Transportation": ["wzdx", "state_511_atis", "tomtom_flow", "tomtom_incidents", "state_511_atis_cameras"],
|
||||||
}
|
}
|
||||||
# Same palette the map legend uses, indexed by sorted-adapter position.
|
# Same palette the map legend uses, indexed by sorted-adapter position.
|
||||||
EVENTS_PALETTE = [
|
EVENTS_PALETTE = [
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
{# State 511 camera detail. Live <img> fetched direct from the source (no proxy /
|
||||||
|
blob storage in Central). Fields from payload->data->data. #}
|
||||||
|
{% set d = (event.data.get('data') or {}).get('data') or {} %}
|
||||||
|
{% if d.get('image_url') %}<dt>View</dt><dd><img src="{{ d.image_url }}" alt="Camera view" style="max-width:480px;max-height:360px;display:block;" loading="lazy"></dd>{% endif %}
|
||||||
|
{% if d.get('roadway_name') %}<dt>Road</dt><dd>{{ d.roadway_name }}{% if d.get('direction') and d.direction not in ['Unknown', 'None'] %} ({{ d.direction }}){% endif %}</dd>{% endif %}
|
||||||
|
{% if d.get('location_description') %}<dt>Location</dt><dd>{{ d.location_description }}</dd>{% endif %}
|
||||||
|
{% if d.get('source') %}<dt>Source</dt><dd>{{ d.source }}</dd>{% endif %}
|
||||||
|
{% if d.get('image_count') and d.image_count > 1 %}<dt>Views</dt><dd>{{ d.image_count }} camera angles</dd>{% endif %}
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
{# State 511 camera one-line subject. The `location` field already embeds the
|
||||||
|
roadway (e.g. "I-84 Mountain Home"), so it stands alone. Fields from payload->data->data. #}
|
||||||
|
{% set d = (event.data.get('data') or {}).get('data') or {} %}
|
||||||
|
Camera: {{ d.get('location_description') or d.get('roadway_name') or ('#' ~ d.get('camera_id')) }}
|
||||||
|
|
@ -31,5 +31,6 @@ STREAMS: list[StreamEntry] = [
|
||||||
StreamEntry("CENTRAL_HYDRO", "central.hydro.>"),
|
StreamEntry("CENTRAL_HYDRO", "central.hydro.>"),
|
||||||
StreamEntry("CENTRAL_TRAFFIC", "central.traffic.>"),
|
StreamEntry("CENTRAL_TRAFFIC", "central.traffic.>"),
|
||||||
StreamEntry("CENTRAL_TRAFFIC_FLOW", "central.traffic_flow.>"),
|
StreamEntry("CENTRAL_TRAFFIC_FLOW", "central.traffic_flow.>"),
|
||||||
|
StreamEntry("CENTRAL_TRAFFIC_CAMERAS", "central.traffic_cameras.>"),
|
||||||
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
|
StreamEntry("CENTRAL_META", "central.meta.>", event_bearing=False),
|
||||||
]
|
]
|
||||||
|
|
|
||||||
1
tests/fixtures/state_511_atis_cameras_sample.json
vendored
Normal file
1
tests/fixtures/state_511_atis_cameras_sample.json
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
{"recordsTotal":2,"recordsFiltered":2,"data":[{"DT_RowId":"1","tooltipUrl":"/tooltip/Cameras/1?lang=%7Blang%7D&noCss=true","agencyLogoEnabled":false,"visible":true,"isDefault":false,"images":[{"id":1,"cameraSiteId":1,"sortOrder":1,"description":"N/A","imageUrl":"/map/Cctv/1","imageType":0,"isVideoAuthRequired":false,"videoDisabled":false,"disabled":false,"blocked":false,"language":"en"}],"id":1,"sourceId":"10.C1","source":"UDOT","type":"UDOT","areaId":null,"area":null,"sortOrder":1,"roadway":"I-15","direction":"Unknown","location":"I-15 UT/ID State Line UT","latLng":{"geography":{"coordinateSystemId":4326,"wellKnownText":"POINT (-112.198 42.0011)"}},"linkId1":"112984999F","linkId2":"114769197T","created":"2023-01-23T13:51:47.3533333+00:00","lastUpdated":"2023-01-23T13:51:47.3533333+00:00","lastEditedBy":"zeeshawn.ahmad@gmail.com","defaultCameraSite":false,"nickname":null,"language":"en","jsonData":{},"jsonDataSerialized":null,"region":null,"state":null,"county":null,"city":null,"dotDistrict":null},{"DT_RowId":"2","tooltipUrl":"/tooltip/Cameras/2?lang=%7Blang%7D&noCss=true","agencyLogoEnabled":false,"visible":true,"isDefault":false,"images":[{"id":2,"cameraSiteId":2,"sortOrder":325,"imageUrl":"/map/Cctv/2","imageType":0,"refreshRateMs":300000,"isVideoAuthRequired":false,"videoDisabled":false,"disabled":false,"blocked":false,"language":"en"},{"id":3,"cameraSiteId":2,"sortOrder":325,"imageUrl":"/map/Cctv/3","imageType":0,"refreshRateMs":300000,"isVideoAuthRequired":false,"videoDisabled":false,"disabled":false,"blocked":false,"language":"en"},{"id":4,"cameraSiteId":2,"sortOrder":325,"imageUrl":"/map/Cctv/4","imageType":0,"refreshRateMs":300000,"isVideoAuthRequired":false,"videoDisabled":false,"disabled":false,"blocked":false,"language":"en"}],"id":2,"sourceId":"100.C1","source":"RWIS","type":"RWIS","areaId":null,"area":null,"sortOrder":325,"roadway":"SH-75","direction":"Unknown","location":"SH-75 Wood River","latLng":{"geography":{"coordinateSystemId":4326,"wellKnownText":"POINT (-114.345 43.5946)"}},"linkId1":"41105278T","linkId2":"851613715T","created":"2024-08-12T07:57:34.1833333+00:00","lastUpdated":"2024-08-12T07:57:34.1833333+00:00","lastEditedBy":"barton.phelps@itd.idaho.gov","defaultCameraSite":false,"nickname":null,"language":"en","jsonData":{"rwisType":"Normal","status":"Normal"},"jsonDataSerialized":"{\"RwisType\":\"Normal\",\"Status\":\"Normal\"}","region":null,"state":null,"county":null,"city":null,"dotDistrict":null}]}
|
||||||
|
|
@ -1146,6 +1146,7 @@ _SAMPLE_INNER = {
|
||||||
"state_511_atis": {"layer": "Incidents", "roadway_name": "US-95", "location_description": "Ponderosa Mobile Home Park"},
|
"state_511_atis": {"layer": "Incidents", "roadway_name": "US-95", "location_description": "Ponderosa Mobile Home Park"},
|
||||||
"tomtom_flow": {"road_category": "primary", "relative_speed": 0.11},
|
"tomtom_flow": {"road_category": "primary", "relative_speed": 0.11},
|
||||||
"tomtom_incidents": {"description": "Roadworks", "from": "Early Road", "to": "Slade Road"},
|
"tomtom_incidents": {"description": "Roadworks", "from": "Early Road", "to": "Slade Road"},
|
||||||
|
"state_511_atis_cameras": {"location_description": "I-84 Mountain Home", "camera_id": 42},
|
||||||
}
|
}
|
||||||
|
|
||||||
# Exact expected subjects for the deterministic adapters. swpc_alerts is omitted
|
# Exact expected subjects for the deterministic adapters. swpc_alerts is omitted
|
||||||
|
|
@ -1167,6 +1168,7 @@ _EXPECTED_SUBJECT = {
|
||||||
"wzdx": "Work zone on I-80 eastbound",
|
"wzdx": "Work zone on I-80 eastbound",
|
||||||
"tomtom_flow": "Traffic flow (primary) — 11% of free-flow",
|
"tomtom_flow": "Traffic flow (primary) — 11% of free-flow",
|
||||||
"tomtom_incidents": "Roadworks on Early Road → Slade Road",
|
"tomtom_incidents": "Roadworks on Early Road → Slade Road",
|
||||||
|
"state_511_atis_cameras": "Camera: I-84 Mountain Home",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
134
tests/test_state_511_atis_cameras.py
Normal file
134
tests/test_state_511_atis_cameras.py
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
"""Tests for the state_511_atis_cameras adapter (v0.9.6).
|
||||||
|
|
||||||
|
Fixture is a real /List/GetData/Cameras capture (2 cameras: one single-image
|
||||||
|
UDOT border camera, one multi-image RWIS camera):
|
||||||
|
tests/fixtures/state_511_atis_cameras_sample.json
|
||||||
|
|
||||||
|
No conftest entry: dedup uses the supervisor-injected cursors.db (inherited
|
||||||
|
mixin); polling is stateless.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from central.adapter import SourceAdapter
|
||||||
|
from central.adapters.state_511_atis_cameras import (
|
||||||
|
State511ATISCamerasAdapter,
|
||||||
|
StateConfig,
|
||||||
|
_parse_wkt,
|
||||||
|
)
|
||||||
|
from central.config_models import AdapterConfig
|
||||||
|
|
||||||
|
FIX = json.loads((Path(__file__).parent / "fixtures" / "state_511_atis_cameras_sample.json").read_text())
|
||||||
|
CAMS = FIX["data"]
|
||||||
|
ID = StateConfig(code="ID", base_url="https://511.idaho.gov")
|
||||||
|
TODAY = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg():
|
||||||
|
return AdapterConfig(
|
||||||
|
name="state_511_atis_cameras", enabled=True, cadence_s=600,
|
||||||
|
settings={"states": [ID.model_dump()]}, updated_at=datetime.now(timezone.utc),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def adapter(tmp_path):
|
||||||
|
return State511ATISCamerasAdapter(_cfg(), MagicMock(), tmp_path / "cursors.db")
|
||||||
|
|
||||||
|
|
||||||
|
def test_wkt_parse():
|
||||||
|
assert _parse_wkt("POINT (-112.198 42.0011)") == (42.0011, -112.198) # (lat, lon)
|
||||||
|
assert _parse_wkt(None) == (None, None)
|
||||||
|
assert _parse_wkt("nonsense") == (None, None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_key_shape(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0], ID)
|
||||||
|
assert e.id == f"ID:cam:{CAMS[0]['id']}:{TODAY}" # per-UTC-day bucketing
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_with_image_url(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0], ID)
|
||||||
|
assert e.category == "camera.state_511_atis_cameras"
|
||||||
|
assert e.severity == 1
|
||||||
|
assert e.data["image_url"] == "https://511.idaho.gov" + CAMS[0]["images"][0]["imageUrl"]
|
||||||
|
assert e.data["source"] == CAMS[0]["source"] # provenance surfaced
|
||||||
|
assert e.data["roadway_name"] == CAMS[0]["roadway"]
|
||||||
|
assert e.data["latitude"] is not None and e.data["longitude"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_event_multi_image(adapter):
|
||||||
|
e = adapter._build_event(CAMS[1], ID)
|
||||||
|
assert e.data["image_count"] == len(CAMS[1]["images"])
|
||||||
|
assert e.data["image_count"] >= 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_image_url_handled_gracefully(adapter):
|
||||||
|
cam = {"id": 999, "roadway": "US-95", "location": "US-95 Somewhere", "source": "ITDNET",
|
||||||
|
"direction": "Unknown", "images": [],
|
||||||
|
"latLng": {"geography": {"wellKnownText": "POINT (-116.5 46.4)"}}}
|
||||||
|
e = adapter._build_event(cam, ID)
|
||||||
|
assert e is not None
|
||||||
|
assert e.data["image_url"] is None and e.data["image_count"] == 0
|
||||||
|
assert e.data["location_description"] == "US-95 Somewhere"
|
||||||
|
|
||||||
|
|
||||||
|
def test_subject_for_state_id(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0], ID)
|
||||||
|
assert adapter.subject_for(e) == f"central.traffic_cameras.id.{CAMS[0]['id']}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_subject_for_unknown_state(adapter):
|
||||||
|
e = adapter._build_event(CAMS[0], StateConfig(code="", base_url="https://x"))
|
||||||
|
assert adapter.subject_for(e) == f"central.traffic_cameras.unknown.{CAMS[0]['id']}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_partial_renders():
|
||||||
|
from central.gui.routes import _derive_subject
|
||||||
|
inner = {"location_description": "I-84 Mountain Home", "camera_id": 42}
|
||||||
|
row = {"adapter": "state_511_atis_cameras", "data": {"data": {"data": inner}}}
|
||||||
|
assert _derive_subject(row) == "Camera: I-84 Mountain Home"
|
||||||
|
|
||||||
|
|
||||||
|
def test_rows_partial_includes_img_tag():
|
||||||
|
from central.gui.routes import _get_templates
|
||||||
|
inner = {"image_url": "https://511.idaho.gov/map/Cctv/1", "roadway_name": "I-84",
|
||||||
|
"location_description": "I-84 Mountain Home", "source": "ITDNET", "image_count": 1}
|
||||||
|
row = {"data": {"data": {"data": inner}}}
|
||||||
|
html = _get_templates().env.get_template("_event_rows/state_511_atis_cameras.html").render(event=row)
|
||||||
|
assert "<img" in html
|
||||||
|
assert "https://511.idaho.gov/map/Cctv/1" in html
|
||||||
|
assert "ITDNET" in html # source provenance rendered
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_poll_paginates(tmp_path):
|
||||||
|
a = State511ATISCamerasAdapter(_cfg(), MagicMock(), tmp_path / "cursors.db")
|
||||||
|
await a.startup()
|
||||||
|
|
||||||
|
def _cam(i):
|
||||||
|
return {"id": i, "roadway": "I-84", "location": f"loc {i}", "direction": "N",
|
||||||
|
"source": "ITDNET", "images": [{"imageUrl": f"/map/Cctv/{i}"}],
|
||||||
|
"latLng": {"geography": {"wellKnownText": "POINT (-116.2 43.6)"}}}
|
||||||
|
|
||||||
|
pages = {0: {"recordsTotal": 150, "data": [_cam(i) for i in range(100)]},
|
||||||
|
100: {"recordsTotal": 150, "data": [_cam(i) for i in range(100, 150)]}}
|
||||||
|
|
||||||
|
async def fake_page(base_url, start):
|
||||||
|
return pages[start]
|
||||||
|
|
||||||
|
a._fetch_page = fake_page
|
||||||
|
events = [e async for e in a.poll()]
|
||||||
|
await a.shutdown()
|
||||||
|
assert len(events) == 150 # all 150 fetched across 2 pages, NOT truncated at the 100-row cap
|
||||||
|
|
||||||
|
|
||||||
|
def test_inherits_dedup_mixin():
|
||||||
|
for m in ("is_published", "mark_published", "sweep_old_ids"):
|
||||||
|
assert m not in State511ATISCamerasAdapter.__dict__, f"redefines {m}"
|
||||||
|
assert getattr(State511ATISCamerasAdapter, m) is getattr(SourceAdapter, m)
|
||||||
|
|
@ -11,7 +11,7 @@ from central.adapter_discovery import discover_adapters
|
||||||
from central.gui import routes
|
from central.gui import routes
|
||||||
|
|
||||||
# Adapters with data_class="telemetry" (the pinned split; grow as telemetry adapters land).
|
# Adapters with data_class="telemetry" (the pinned split; grow as telemetry adapters land).
|
||||||
_TELEMETRY = ["nwis", "tomtom_flow"]
|
_TELEMETRY = ["nwis", "state_511_atis_cameras", "tomtom_flow"]
|
||||||
|
|
||||||
|
|
||||||
# --- data_class defaults / registry split -----------------------------------
|
# --- data_class defaults / registry split -----------------------------------
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue