Compare commits

...

5 commits

Author SHA1 Message Date
3c8da28d80
chore: bump version to 0.15.0 (#124)
Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 11:33:12 -06:00
dfcdda5ecf
feat: wire GenericHttpAdapter into geocoder enrichment (v0.15.0 PR2.5) (#123)
Set enrichment_locations = [("latitude", "longitude")] and surface
extracted coords as top-level data keys so apply_enrichment can geocode
generic_http events; non-Point geometries and coordless items degrade
gracefully to region unknown.

Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 11:30:35 -06:00
7688d268a3
feat: GUI create/delete for adapter instances (v0.15.0 PR3) (#122)
Completes the CRUD loop for operator-creatable adapters.  Operators can
now create generic_http instances (GET+POST /adapters/new) and delete
operator instances (POST /adapters/{name}/delete) without editing Python.

Key changes:
- SourceAdapter.operator_creatable class attr (default False); set True on
  GenericHttpAdapter so it appears in the kind select
- ADAPTER_CREATE / ADAPTER_DELETE audit constants
- _parse_adapter_settings() shared helper extracted from adapters_edit_submit
  (single place for widget parsing, region handling, Pydantic validation,
  quota blocking) — edit_submit refactored to call it
- GET/POST /adapters/new: kind select → name/cadence/enabled → settings fields;
  INSERT with kind supplied explicitly; 302 → edit page on success
- POST /adapters/{name}/delete: primary guard (name in adapter_classes → 403);
  second guard (kind not operator_creatable → 403); DELETE + audit; 302 → list
- adapters_list.html: "New adapter" button; per-row Delete form (operator
  instances only; built-ins have no button); deletable flag passed from route
- adapters_new.html: new template reusing same field-rendering blocks as
  adapters_edit.html; enabled checkbox unchecked by default
- 46 new tests (all pass; mock-DB pattern mirrors test_gui_adapter_edit.py)

Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-30 13:03:14 -06:00
3effd677a5
feat: add GenericHttpAdapter (kind=generic_http, v0.15.0 PR2) (#121)
One config-driven adapter class that operators can instantiate many times
via distinct config.adapters rows (unique name, kind="generic_http").
Ships class + Pydantic settings schema + unit tests only; no GUI create
route, no migration, no seeded row (PR3). Class is dormant until an
operator row exists.

Key design decisions:
- self.name = config.name in __init__ scopes dedup to each instance
- domain validated against STREAMS registry at import time
- subject_for derives region from event.data["_enriched"]["geocoder"]
  identically to usgs_quake (no static subject_region setting)
- _dig() helper supports dotted paths into nested dicts/lists

Also adds: GUI partials (_event_summaries, _event_rows), CONSUMER-
INTEGRATION.md §6 subsection, and test_events_feed_frontend.py sample
entry — all required by the existing adapter-coverage consistency tests.

Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-30 11:13:06 -06:00
13722d07dd
refactor: decouple adapter class-identity (kind) from instance-identity (name) (#120)
Adds a `kind` TEXT column to `config.adapters` (migration 043) so one adapter
class can later back many operator-created instances.  Every built-in adapter
row is back-filled with `kind = name`, keeping all 23 adapters working
unchanged.

- Migration 043: ADD COLUMN IF NOT EXISTS kind, back-fill, SET NOT NULL.
- AdapterConfig gains `kind: str | None` with a model_validator that falls
  back to `name` when the column is NULL/absent (pre-043 rows).
- config_store.get_adapter / list_adapters: add `kind` to SELECT.
- supervisor._create_adapter AND _start_adapter (api-key precondition):
  resolve the class by config.kind (class key) not config.name (instance key);
  runtime state keying stays on config.name.
- routes.py GET/POST /adapters/{name} and adapters_list: add `kind` to SELECT,
  resolve the adapter class by row["kind"].
- adapter_discovery.py: comment clarifying .name is the kind (class identity).
- Out of scope, left as-is (built-ins only, name==kind): setup wizard paths in
  routes.py (~750/~891/~1109) and gui/__init__.py.
- Tests: 17 new pure-unit tests (migration SQL shape, AdapterConfig back-compat,
  _create_adapter + _start_adapter kind dispatch). Existing test mock rows
  updated to include the `kind` field.

Co-authored-by: Ubuntu <zvx@cortex.echo6.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-30 00:53:34 -06:00
24 changed files with 2949 additions and 164 deletions

View file

@ -2023,6 +2023,55 @@ at parameter `00060`, gage height (ft) at `00065`, water temperature (°C) at
\
---
### generic_http — Generic HTTP Source (v0.15.0)
- **Source:** Any public REST / GeoJSON endpoint; configured per-instance
via `config.adapters.settings.url`. v1 supports public (unauthenticated)
feeds only; auth is a planned follow-up.
- **Stream:** Determined by the `domain` setting, which must match an
existing Central stream domain (e.g. `wx`, `fire`, `quake`). Subject
filter: `central.<domain>.>`.
- **Subject:** `central.<domain>.<region>``<region>` is derived from
`event.data["_enriched"]["geocoder"]` by the Photon enrichment pipeline
(identical convention to `usgs_quake`). Returns `unknown` when enrichment
has not yet run or found no geo signal.
- **Category:** `<domain>.<category_suffix>` (default suffix: `alert`, so
e.g. `wx.alert`).
- **Dedup key:** the value resolved by `id_path` within each source item
(required field; coerced to string).
- **Geo:** GeoJSON geometry from `geometry_path` (default `"geometry"`)
when it resolves to a dict; `geo.centroid` is also set for Point
geometries. Falls back to `lat_path` / `lon_path` for non-GeoJSON
responses.
- **Event.data fields:** operator-configured. `title_path``data["title"]`
when set; additional fields via `field_mappings` (`source_path → dest_key`
pairs). No guaranteed fixed schema — varies per operator configuration.
- **Settings summary:**
| Setting | Required | Default | Notes |
|---|---|---|---|
| `url` | yes | — | Polled endpoint (GET, no auth in v1) |
| `domain` | yes | — | Must be an existing stream domain |
| `id_path` | yes | — | Dotted path to stable unique item id |
| `items_path` | no | `features` | Dotted path to the items array |
| `format` | no | `geojson` | `geojson` or `json` (informational) |
| `time_path` | no | `null` | ISO-8601 timestamp path; uses poll time if absent |
| `title_path` | no | `null` | Path → `data["title"]` |
| `geometry_path` | no | `geometry` | GeoJSON geometry path |
| `lat_path` / `lon_path` | no | `null` | Numeric lat/lon paths (non-GeoJSON fallback) |
| `severity_path` | no | `null` | Int severity (04) path |
| `category_suffix` | no | `alert` | Appended to domain for Event.category |
| `field_mappings` | no | `[]` | List of `{source_path, dest_key}` |
- **Cadence:** 300s (5 min) default.
- **Multiple instances:** one `generic_http` class supports many
`config.adapters` rows (each with a distinct instance `name`); dedup is
scoped per-instance. The class is dormant until an operator row is
created — no built-in seeded instance.
\
---
## 7. Fall-off / removal semantics
Central adapters fall into three buckets for handling upstream events that

View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "central"
version = "0.14.9"
version = "0.15.0"
requires-python = ">=3.12,<3.13"
description = "Data hub spine — adapters, bus, archive."
readme = "README.md"

View file

@ -0,0 +1,25 @@
-- Migration 043: decouple adapter class-identity (kind) from instance-identity (name)
--
-- Until now config.adapters.name served two roles:
-- 1. Instance primary key — the unique name used in runtime state, logs, dedup
-- tables, NATS subjects, etc.
-- 2. Registry key — the key used to look up the adapter class in the
-- supervisor's in-memory registry (discover_adapters returns {class.name: cls}).
--
-- These roles must be split so that one adapter class can later back many
-- operator-created instances (e.g. a generic HTTP adapter). The new `kind`
-- column holds the class identity (registry key), while `name` remains the
-- unique instance identifier. All built-in adapters have name == kind, so
-- every existing row is back-filled with kind = name and behaviour is unchanged.
--
-- Idempotent: ADD COLUMN IF NOT EXISTS is safe on re-run.
-- No DEFAULT is set — future INSERT paths must always supply kind explicitly.
ALTER TABLE config.adapters ADD COLUMN IF NOT EXISTS kind TEXT;
-- Back-fill: existing rows get kind = name (class identity == instance identity
-- for all 23 built-in adapters shipped before v0.15.0).
UPDATE config.adapters SET kind = name WHERE kind IS NULL;
-- Enforce non-null going forward.
ALTER TABLE config.adapters ALTER COLUMN kind SET NOT NULL;

View file

@ -66,6 +66,12 @@ class SourceAdapter(ABC):
set in ``central.archive`` -- the two MUST stay in sync (enforced by
``tests/test_bypass_bbox_consistency.py``)."""
operator_creatable: bool = False
"""True for adapters that operators may instantiate many times from the
GUI (one config.adapters row per instance, each with a unique name and
kind=<class_name>). False (default) for singleton built-ins where the
name equals the kind and the row is seeded by migrations."""
@abstractmethod
async def poll(self) -> AsyncIterator[Event]:
"""

View file

@ -30,5 +30,7 @@ def discover_adapters() -> dict[str, type[SourceAdapter]]:
and attr is not SourceAdapter
and hasattr(attr, "name")
):
# attr.name is the *kind* (class identity); AdapterConfig.name
# is the instance identity. For all built-ins kind == name.
registry[attr.name] = attr
return registry

View file

@ -0,0 +1,459 @@
"""Generic HTTP / GeoJSON adapter — operator-instantiable, no Python required.
One class (kind="generic_http") supports many config.adapters rows. Each
row carries a distinct ``name`` (instance identity) and a ``settings`` dict
that drives a separate poll loop. PR3 ships the GUI create route; this PR
ships only the class.
v1: public feeds only; auth is a follow-up.
"""
import logging
import sqlite3
from collections.abc import AsyncIterator
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal
import aiohttp
from pydantic import BaseModel, field_validator
from tenacity import (
retry,
retry_if_exception_type,
stop_after_attempt,
wait_exponential_jitter,
)
from central.adapter import SourceAdapter
from central.adapters._subject_helpers import subject_for_region
from central.config_models import AdapterConfig
from central.config_store import ConfigStore
from central.models import Event, Geo
from central.streams import STREAMS
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Derive the valid domain set from the stream registry at import time.
# Subject filters are always "central.<domain>.>" so we split on "." and
# take index 1.
# ---------------------------------------------------------------------------
_VALID_DOMAINS: frozenset[str] = frozenset(
s.subject_filter.split(".")[1] for s in STREAMS
)
_DEDUP_DDL = (
"CREATE TABLE IF NOT EXISTS published_ids ("
"adapter TEXT NOT NULL, event_id TEXT NOT NULL, "
"first_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
"last_seen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, "
"PRIMARY KEY (adapter, event_id))"
)
_DEDUP_IDX = (
"CREATE INDEX IF NOT EXISTS published_ids_last_seen "
"ON published_ids (last_seen)"
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _dig(obj: Any, path: str) -> Any:
"""Walk a nested dict/list by a dotted path string.
Each segment is tried as a dict key first; if the current node is a
list/tuple the segment is coerced to an integer index. Returns None on
any miss (missing key, out-of-range index, wrong node type).
Examples::
_dig({"a": {"b": 1}}, "a.b") # 1
_dig({"a": [10, 20]}, "a.1") # 20
_dig({"a": 1}, "a.b") # None
_dig(None, "anything") # None
"""
parts = path.split(".")
cur = obj
for part in parts:
if cur is None:
return None
if isinstance(cur, dict):
cur = cur.get(part)
elif isinstance(cur, (list, tuple)):
try:
cur = cur[int(part)]
except (ValueError, IndexError):
return None
else:
return None
return cur
# ---------------------------------------------------------------------------
# Settings schema
# ---------------------------------------------------------------------------
class FieldMapping(BaseModel):
"""Map one path inside a source item to a key in Event.data.
Mirrors TomTomFlowSettings / TileCoord so the GUI model_list widget
renders it identically.
"""
source_path: str # dotted path into the source item
dest_key: str # key written into Event.data
class GenericHttpSettings(BaseModel):
"""Settings schema for GenericHttpAdapter instances."""
url: str
"""Endpoint to poll (GET, no auth in v1)."""
format: Literal["geojson", "json"] = "geojson"
"""Response format hint (currently informational; both paths use the same
JSON fetch; the geometry_path / lat_path controls shape extraction)."""
items_path: str = "features"
"""Dotted path to the array of items inside the response object."""
domain: str
"""Event domain; must match an existing NATS stream (e.g. ``wx``, ``fire``,
``quake``). Determines which JetStream stream the event lands in."""
@field_validator("domain")
@classmethod
def _validate_domain(cls, v: str) -> str:
if v not in _VALID_DOMAINS:
valid = ", ".join(sorted(_VALID_DOMAINS))
raise ValueError(
f"unknown domain '{v}'; must be one of: {valid}"
)
return v
category_suffix: str = "alert"
"""Appended to domain to form Event.category: ``{domain}.{category_suffix}``."""
id_path: str
"""Dotted path to a stable, unique identifier within each item (required
for deduplication)."""
time_path: str | None = None
"""Dotted path to an ISO-8601 timestamp string. When None, poll time
(UTC now) is used."""
title_path: str | None = None
"""Dotted path to a human-readable title → written to ``data['title']``."""
geometry_path: str = "geometry"
"""(GeoJSON) dotted path to a GeoJSON geometry object within each item.
Ignored when the resolved value is not a dict."""
lat_path: str | None = None
"""(JSON) dotted path to a numeric latitude. Used only when
geometry_path does not resolve to a geometry dict."""
lon_path: str | None = None
"""(JSON) dotted path to a numeric longitude. See lat_path."""
severity_path: str | None = None
"""Dotted path to an integer severity (0-4). When None, severity is
omitted from the event."""
field_mappings: list[FieldMapping] = []
"""Extra source-path → dest-key pairs written into Event.data."""
# ---------------------------------------------------------------------------
# Adapter class
# ---------------------------------------------------------------------------
class GenericHttpAdapter(SourceAdapter):
"""Config-driven REST/GeoJSON source adapter.
A single class that operators can instantiate many times via distinct
``config.adapters`` rows (each row has a unique ``name`` / instance
identity and ``kind="generic_http"``). No Python required to add a
new source.
"""
# Class-level identity (kind) — used by discover_adapters() as the
# registry key. Each *instance* overrides self.name = config.name in
# __init__ so dedup and logs are scoped to the instance.
name = "generic_http"
display_name = "Generic HTTP Source"
description = (
"Config-driven adapter that polls any public REST or GeoJSON endpoint "
"and maps response fields to Central events. Create one row per "
"source; no Python required."
)
settings_schema = GenericHttpSettings
default_cadence_s = 300
data_class = "event"
wizard_order = None # not in the setup wizard; created via operator GUI
# Generic instances surface lat/lon into event.data so the supervisor's
# geocoder enrichment can reach them. Coordless instances degrade to
# region unknown, which is fine — apply_enrichment handles it gracefully.
enrichment_locations = [("latitude", "longitude")]
bypass_bbox_filter = False
operator_creatable = True # GUI allows operators to create multiple instances
def __init__(
self,
config: AdapterConfig,
config_store: ConfigStore, # unused; accepted for signature uniformity
cursor_db_path: Path,
) -> None:
# Override class-level name with the *instance* name so that the
# inherited dedup helpers (is_published / mark_published / sweep_old_ids)
# scope their SQL queries to this instance, not the generic_http class.
self.name = config.name
self._cursor_db_path = cursor_db_path
self._session: aiohttp.ClientSession | None = None
self._db: sqlite3.Connection | None = None
self._settings = GenericHttpSettings(**config.settings)
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
async def startup(self) -> None:
"""Initialize HTTP session and dedup tracker."""
self._session = aiohttp.ClientSession(
headers={"User-Agent": "Central/1.0 (generic_http adapter)"},
timeout=aiohttp.ClientTimeout(total=30),
)
self._db = sqlite3.connect(str(self._cursor_db_path))
self._db.execute(_DEDUP_DDL)
self._db.execute(_DEDUP_IDX)
self._db.commit()
self.sweep_old_ids()
logger.info(
"generic_http adapter started",
extra={"adapter": self.name, "url": self._settings.url},
)
async def shutdown(self) -> None:
"""Close HTTP session and database."""
if self._session:
await self._session.close()
self._session = None
if self._db:
self._db.close()
self._db = None
logger.info("generic_http adapter shut down", extra={"adapter": self.name})
# ------------------------------------------------------------------
# Configuration hot-reload
# ------------------------------------------------------------------
async def apply_config(self, new_config: AdapterConfig) -> None:
"""Re-parse settings in place (no reconstruction needed)."""
self._settings = GenericHttpSettings(**new_config.settings)
logger.info(
"generic_http config applied",
extra={"adapter": self.name, "url": self._settings.url},
)
# ------------------------------------------------------------------
# Fetch
# ------------------------------------------------------------------
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential_jitter(initial=1, max=15),
retry=retry_if_exception_type((aiohttp.ClientError,)),
reraise=True,
)
async def _fetch(self, url: str) -> dict[str, Any]:
"""GET ``url`` and return parsed JSON. Retried up to 3 times."""
if not self._session:
raise RuntimeError("Session not initialized")
async with self._session.get(url) as resp:
resp.raise_for_status()
return await resp.json(content_type=None)
# ------------------------------------------------------------------
# Poll
# ------------------------------------------------------------------
async def poll(self) -> AsyncIterator[Event]:
"""Fetch the configured URL and yield new Events."""
self.sweep_old_ids()
s = self._settings
try:
raw = await self._fetch(s.url)
except Exception as exc:
logger.error(
"generic_http fetch failed",
extra={"adapter": self.name, "url": s.url, "error": str(exc)},
)
raise
items = _dig(raw, s.items_path)
if not isinstance(items, list):
logger.warning(
"generic_http items_path did not resolve to a list",
extra={
"adapter": self.name,
"items_path": s.items_path,
"got": type(items).__name__,
},
)
return
new_count = 0
for item in items:
event = self._item_to_event(item)
if event is None:
continue
if self.is_published(event.id):
continue
yield event
self.mark_published(event.id)
new_count += 1
logger.info(
"generic_http poll completed",
extra={"adapter": self.name, "count": new_count},
)
# ------------------------------------------------------------------
# Item → Event
# ------------------------------------------------------------------
def _item_to_event(self, item: Any) -> Event | None:
"""Convert a single source item to an Event, or None to skip."""
s = self._settings
# --- id (required) ---
raw_id = _dig(item, s.id_path)
if raw_id is None:
logger.warning(
"generic_http item missing id",
extra={"adapter": self.name, "id_path": s.id_path},
)
return None
event_id = str(raw_id)
# --- category ---
category = f"{s.domain}.{s.category_suffix}"
# --- time ---
if s.time_path:
raw_time = _dig(item, s.time_path)
if raw_time is not None:
try:
# Python 3.11+ fromisoformat handles "Z"; older needs replace.
event_time = datetime.fromisoformat(
str(raw_time).replace("Z", "+00:00")
)
if event_time.tzinfo is None:
event_time = event_time.replace(tzinfo=timezone.utc)
except (ValueError, TypeError):
logger.warning(
"generic_http could not parse time; using now",
extra={
"adapter": self.name,
"time_path": s.time_path,
"raw": raw_time,
},
)
event_time = datetime.now(timezone.utc)
else:
event_time = datetime.now(timezone.utc)
else:
event_time = datetime.now(timezone.utc)
# --- geo ---
geo_kwargs: dict[str, Any] = {}
lat_val: float | None = None
lon_val: float | None = None
# Try GeoJSON geometry first.
if s.geometry_path:
geom = _dig(item, s.geometry_path)
if isinstance(geom, dict):
geo_kwargs["geometry"] = geom
# Also set centroid when geometry is a simple Point.
# Non-Point geometries (LineString, Polygon) have no single
# representative point; lat_val/lon_val stay None → region unknown.
if geom.get("type") == "Point":
coords = geom.get("coordinates") or []
if len(coords) >= 2:
try:
lon_val = float(coords[0])
lat_val = float(coords[1])
geo_kwargs["centroid"] = (lon_val, lat_val)
except (TypeError, ValueError):
pass
# Fall back to explicit lat/lon paths.
if "geometry" not in geo_kwargs and s.lat_path and s.lon_path:
raw_lat = _dig(item, s.lat_path)
raw_lon = _dig(item, s.lon_path)
if raw_lat is not None and raw_lon is not None:
try:
lat_val = float(raw_lat)
lon_val = float(raw_lon)
geo_kwargs["centroid"] = (lon_val, lat_val)
except (TypeError, ValueError):
pass
geo = Geo(**geo_kwargs)
# --- data ---
data: dict[str, Any] = {}
if s.title_path:
title = _dig(item, s.title_path)
if title is not None:
data["title"] = title
# Surface coordinates into data so the geocoder enrichment can reach them.
# Coordless instances degrade to region unknown — no special-casing needed.
# Written BEFORE field_mappings so an explicit operator mapping to
# latitude/longitude (rare) takes precedence.
if lat_val is not None and lon_val is not None:
data["latitude"] = lat_val
data["longitude"] = lon_val
for fm in s.field_mappings:
data[fm.dest_key] = _dig(item, fm.source_path)
# --- severity ---
severity: int | None = None
if s.severity_path:
raw_sev = _dig(item, s.severity_path)
if raw_sev is not None:
try:
severity = int(raw_sev)
except (TypeError, ValueError):
pass
return Event(
id=event_id,
adapter=self.name,
category=category,
time=event_time,
geo=geo,
data=data,
severity=severity,
)
# ------------------------------------------------------------------
# Subject routing
# ------------------------------------------------------------------
def subject_for(self, event: Event) -> str:
"""Return the NATS subject for a generic-http event.
Pattern: ``central.<domain>.<region>``
Region is derived from ``event.data["_enriched"]["geocoder"]``
identically to usgs_quake the Photon enrichment pipeline populates
that key at publish time. Returns "unknown" when enrichment hasn't
run yet or found no geo signal.
"""
region = subject_for_region(event.data)
return f"central.{self._settings.domain}.{region}"

View file

@ -30,7 +30,15 @@ class RegionConfig(BaseModel):
class AdapterConfig(BaseModel):
"""Configuration for a single adapter."""
name: str = Field(description="Unique adapter identifier")
name: str = Field(description="Unique adapter identifier (instance identity)")
kind: str | None = Field(
default=None,
description=(
"Adapter class identifier (registry key / class identity). "
"Matches SourceAdapter.name on the class. Defaults to `name` "
"for back-compat when the DB column is absent or NULL (pre-043)."
),
)
enabled: bool = Field(default=True, description="Whether adapter is active")
cadence_s: int = Field(ge=10, description="Poll interval in seconds")
settings: dict[str, Any] = Field(
@ -41,6 +49,13 @@ class AdapterConfig(BaseModel):
)
updated_at: datetime = Field(description="Last configuration update time")
@model_validator(mode="after")
def _backfill_kind(self) -> "AdapterConfig":
"""Default kind to name when the DB column is NULL or absent (pre-043)."""
if self.kind is None:
self.kind = self.name
return self
@property
def is_paused(self) -> bool:
"""Check if adapter is currently paused."""

View file

@ -95,7 +95,7 @@ class ConfigStore:
async with self._pool.acquire() as conn:
row = await conn.fetchrow(
"""
SELECT name, enabled, cadence_s, settings, paused_at, updated_at
SELECT name, kind, enabled, cadence_s, settings, paused_at, updated_at
FROM config.adapters
WHERE name = $1
""",
@ -110,7 +110,7 @@ class ConfigStore:
async with self._pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT name, enabled, cadence_s, settings, paused_at, updated_at
SELECT name, kind, enabled, cadence_s, settings, paused_at, updated_at
FROM config.adapters
ORDER BY name
"""

View file

@ -9,7 +9,9 @@ AUTH_LOGIN_FAILED = "auth.login_failed"
AUTH_LOGOUT = "auth.logout"
AUTH_PASSWORD_CHANGE = "auth.password_change"
OPERATOR_CREATE = "operator.create"
ADAPTER_CREATE = "adapter.create"
ADAPTER_UPDATE = "adapter.update"
ADAPTER_DELETE = "adapter.delete"
STREAM_UPDATE = "stream.update"
API_KEY_CREATE = "api_key.create"
API_KEY_ROTATE = "api_key.rotate"

View file

@ -36,6 +36,8 @@ from central.gui.auth import (
verify_password,
)
from central.gui.audit import (
ADAPTER_CREATE,
ADAPTER_DELETE,
ADAPTER_UPDATE,
API_KEY_CREATE,
API_KEY_DELETE,
@ -100,6 +102,10 @@ ALIAS_REGEX = re.compile(r"^[a-zA-Z0-9_]+$")
# Email validation regex (simple but effective)
EMAIL_REGEX = re.compile(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")
# Adapter instance-name regex: must start with a lowercase letter, followed by
# 163 lowercase letters, digits, or underscores (total 264 chars).
ADAPTER_NAME_REGEX = re.compile(r"^[a-z][a-z0-9_]{1,63}$")
def _get_templates():
"""Get templates instance (deferred import to avoid circular)."""
@ -1437,6 +1443,164 @@ async def change_password_submit(
# =============================================================================
def _parse_adapter_settings(
form,
adapter_cls,
current_settings: dict,
cadence_s: int,
) -> tuple[dict, dict[str, str]]:
"""Parse and validate adapter settings from a form submission.
Shared by ``adapters_edit_submit`` and ``adapters_create_submit`` so that
field parsing, region handling, Pydantic validation, and quota-blocking live
in exactly one place.
Args:
form: Starlette ``FormData`` from ``await request.form()``.
adapter_cls: The resolved ``SourceAdapter`` subclass, or ``None``.
current_settings: Existing settings dict (``{}`` for a new adapter).
Used by ``describe_fields`` to populate ``field.current_value``.
cadence_s: Validated cadence for quota estimation.
Returns:
``(new_settings, errors)`` on success ``errors`` is empty and
``new_settings`` is the Pydantic-validated dict; on failure
``new_settings`` is ``{}`` and ``errors`` maps field names to messages.
"""
errors: dict[str, str] = {}
if not (adapter_cls and hasattr(adapter_cls, "settings_schema")):
# No schema — preserve existing settings unchanged.
return dict(current_settings), errors
schema = adapter_cls.settings_schema
fields = describe_fields(schema, current_settings)
parsed_values: dict = {}
for field in fields:
raw = form.get(field.name, "")
if field.widget == "text":
parsed_values[field.name] = raw.strip() if raw else None
elif field.widget == "number":
try:
parsed_values[field.name] = int(raw) if raw else None
except ValueError:
errors[field.name] = f"{field.label} must be a number"
elif field.widget == "checkbox":
parsed_values[field.name] = field.name in form
elif field.widget == "csv":
if raw.strip():
parsed_values[field.name] = [v.strip() for v in raw.split(",") if v.strip()]
else:
parsed_values[field.name] = []
elif field.widget == "csv_int":
parsed_ints: list[int] = []
if raw.strip():
for tok in raw.split(","):
tok = tok.strip()
if not tok:
continue
try:
parsed_ints.append(int(tok))
except ValueError:
logger.warning(
"csv_int: dropped non-numeric token",
extra={"field": field.name, "token": tok},
)
parsed_values[field.name] = parsed_ints
elif field.widget == "select":
value = raw.strip() if raw else None
if value and field.options and value not in field.options:
errors[field.name] = f"Invalid {field.label.lower()}"
else:
parsed_values[field.name] = value
elif field.widget == "checkboxes":
values = form.getlist(field.name)
if field.options:
invalid = [v for v in values if v not in field.options]
if invalid:
errors[field.name] = f"Invalid values: {', '.join(invalid)}"
else:
parsed_values[field.name] = values
else:
parsed_values[field.name] = values
elif field.widget == "api_key_select":
value = raw.strip() if raw else None
parsed_values[field.name] = value
elif field.widget == "model_list":
rows = _parse_model_list(form, field)
parsed_values[field.name] = rows
elif field.widget == "region":
pass # handled in the region block below
# Region fields (common to adapters that expose a bounding-box region).
region_north_str = form.get("region_north", "").strip()
region_south_str = form.get("region_south", "").strip()
region_east_str = form.get("region_east", "").strip()
region_west_str = form.get("region_west", "").strip()
has_region = any([region_north_str, region_south_str, region_east_str, region_west_str])
if has_region:
try:
region_north = float(region_north_str)
region_south = float(region_south_str)
region_east = float(region_east_str)
region_west = float(region_west_str)
if not (-90 <= region_south < region_north <= 90):
errors["region"] = (
"Invalid latitude: south must be less than north, "
"both between -90 and 90"
)
elif not (-180 <= region_west < region_east <= 180):
errors["region"] = (
"Invalid longitude: west must be less than east, "
"both between -180 and 180"
)
else:
parsed_values["region"] = {
"north": region_north,
"south": region_south,
"east": region_east,
"west": region_west,
}
except ValueError:
errors["region"] = "Region coordinates must be valid numbers"
else:
parsed_values["region"] = None
if errors:
return {}, errors
# Pydantic validation + quota check.
try:
validated_data = {k: v for k, v in parsed_values.items() if v is not None}
validated = schema(**validated_data)
new_settings = validated.model_dump(mode="json")
q = adapter_cls.quota_estimate(validated, cadence_s)
if q and q.get("blocked"):
ml = next((f.name for f in fields if f.widget == "model_list"), "quota")
errors[ml] = (
f"Estimated {q['calls_per_month']:,} calls/month exceeds the "
f"{q['cap']:,}/month free-tier cap — raise cadence or remove rows."
)
return {}, errors
except ValidationError as e:
ml_name = next((f.name for f in fields if f.widget == "model_list"), None)
for err in e.errors():
loc = err["loc"]
key = str(loc[0]) if loc else (ml_name or "unknown")
if len(loc) >= 2 and isinstance(loc[1], int):
errors[key] = f"Row {loc[1] + 1}: {err['msg']}"
else:
errors[key] = err["msg"]
return {}, errors
return new_settings, errors
@router.get("/adapters", response_class=HTMLResponse)
async def adapters_list(
request: Request,
@ -1451,7 +1615,7 @@ async def adapters_list(
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT name, enabled, cadence_s, settings, paused_at, updated_at, last_error
SELECT name, kind, enabled, cadence_s, settings, paused_at, updated_at, last_error
FROM config.adapters
ORDER BY name
"""
@ -1460,7 +1624,8 @@ async def adapters_list(
adapters = []
for row in rows:
settings = row["settings"] or {}
adapter_cls = adapter_classes.get(row["name"])
# Resolve the class by kind (class identity), not instance name.
adapter_cls = adapter_classes.get(row["kind"])
# Check if required API key is missing — resolve via the per-row
# settings[api_key_field] (operator-selected alias), falling back
@ -1470,6 +1635,10 @@ async def adapters_list(
)
api_key_missing = not has_key
# Operator instances have a name that is NOT a registered kind key.
# Built-ins always have name == kind which IS in the registry.
deletable = row["name"] not in adapter_classes
adapters.append({
"name": row["name"],
"display_name": getattr(adapter_cls, "display_name", row["name"]) if adapter_cls else row["name"],
@ -1481,6 +1650,7 @@ async def adapters_list(
"last_error": row["last_error"],
"api_key_missing": api_key_missing,
"requires_api_key_alias": requires_api_key_alias,
"deletable": deletable,
})
csrf_token = request.state.csrf_token
@ -1529,6 +1699,202 @@ def _parse_model_list(form, field) -> list[dict]:
return out
@router.get("/adapters/new", response_class=HTMLResponse)
async def adapters_create_form(request: Request) -> Response:
"""Render the create-adapter form.
Lists only adapter kinds where ``operator_creatable is True`` so operators
can instantiate them freely without touching Python code.
"""
templates = _get_templates()
pool = get_pool()
operator = request.state.operator
csrf_token = request.state.csrf_token
adapter_classes = _adapter_classes()
creatable_kinds = {
kind: cls
for kind, cls in adapter_classes.items()
if getattr(cls, "operator_creatable", False)
}
if not creatable_kinds:
return Response(status_code=404, content="No operator-creatable adapter kinds are registered.")
# NOTE: single creatable kind today; multi-kind HTMX field-swap is a future enhancement.
first_kind, first_cls = next(iter(creatable_kinds.items()))
fields = []
if hasattr(first_cls, "settings_schema"):
fields = describe_fields(first_cls.settings_schema, {})
if first_cls.api_key_field is not None:
for f in fields:
if f.name == first_cls.api_key_field:
f.widget = "api_key_select"
async with pool.acquire() as conn:
api_key_rows = await conn.fetch("SELECT alias FROM config.api_keys ORDER BY alias")
api_keys = [{"alias": r["alias"]} for r in api_key_rows]
return templates.TemplateResponse(
request=request,
name="adapters_new.html",
context={
"operator": operator,
"csrf_token": csrf_token,
"creatable_kinds": [
{"kind": kind, "display_name": getattr(cls, "display_name", kind)}
for kind, cls in creatable_kinds.items()
],
"selected_kind": first_kind,
"default_cadence_s": first_cls.default_cadence_s,
"fields": fields,
"api_keys": api_keys,
"errors": None,
"form_data": None,
},
)
@router.post("/adapters/new")
async def adapters_create_submit(request: Request) -> Response:
"""Process the create-adapter form (first INSERT in the codebase)."""
templates = _get_templates()
pool = get_pool()
operator = request.state.operator
form = await request.form()
form_csrf = form.get("csrf_token", "")
if not form_csrf or form_csrf != request.state.csrf_token:
raise CsrfValidationError("Invalid CSRF token")
adapter_classes = _adapter_classes()
creatable_kinds = {
kind: cls
for kind, cls in adapter_classes.items()
if getattr(cls, "operator_creatable", False)
}
kind = (form.get("kind") or "").strip()
name = (form.get("name") or "").strip()
enabled = "enabled" in form
cadence_s_str = form.get("cadence_s", "")
errors: dict[str, str] = {}
form_data: dict[str, Any] = {
"kind": kind,
"name": name,
"enabled": enabled,
"cadence_s": cadence_s_str,
}
# Validate kind — must be operator-creatable.
kind_cls = creatable_kinds.get(kind)
if kind not in creatable_kinds:
errors["kind"] = f"'{kind}' is not a valid operator-creatable adapter kind."
# Validate instance name.
if "kind" not in errors:
if not ADAPTER_NAME_REGEX.match(name):
errors["name"] = (
"Name must start with a lowercase letter followed by 163 "
"lowercase letters, digits, or underscores."
)
elif name in adapter_classes:
errors["name"] = (
f"'{name}' is a reserved kind name and cannot be used as an "
"instance name."
)
# Validate cadence_s.
cadence_s = 0
try:
cadence_s = int(cadence_s_str)
if cadence_s < 10:
errors["cadence_s"] = "Input should be greater than or equal to 10"
except ValueError:
errors["cadence_s"] = "Cadence must be a valid integer"
# Check for duplicate name in DB (only when name passed format + kind checks).
if "name" not in errors and "kind" not in errors:
async with pool.acquire() as conn:
existing = await conn.fetchval(
"SELECT 1 FROM config.adapters WHERE name = $1", name
)
if existing:
return Response(
status_code=409,
content=f"An adapter named '{name}' already exists.",
)
# Parse + validate settings via the shared helper.
new_settings: dict = {}
if not errors and kind_cls:
new_settings, settings_errors = _parse_adapter_settings(
form, kind_cls, {}, cadence_s
)
errors.update(settings_errors)
# Re-render on error.
if errors:
fields = []
if kind_cls and hasattr(kind_cls, "settings_schema"):
fields = describe_fields(kind_cls.settings_schema, {})
if kind_cls.api_key_field is not None:
for f in fields:
if f.name == kind_cls.api_key_field:
f.widget = "api_key_select"
# Populate form_data for settings fields so inputs restore values.
for field in fields:
form_data.setdefault(field.name, form.get(field.name, ""))
async with pool.acquire() as conn:
api_key_rows = await conn.fetch("SELECT alias FROM config.api_keys ORDER BY alias")
api_keys = [{"alias": r["alias"]} for r in api_key_rows]
selected_kind = kind if kind in creatable_kinds else (next(iter(creatable_kinds)) if creatable_kinds else "")
return templates.TemplateResponse(
request=request,
name="adapters_new.html",
context={
"operator": operator,
"csrf_token": request.state.csrf_token,
"creatable_kinds": [
{"kind": k, "display_name": getattr(c, "display_name", k)}
for k, c in creatable_kinds.items()
],
"selected_kind": selected_kind,
"default_cadence_s": getattr(kind_cls, "default_cadence_s", 300) if kind_cls else 300,
"fields": fields,
"api_keys": api_keys,
"errors": errors,
"form_data": form_data,
},
status_code=422,
)
# INSERT INTO config.adapters — kind supplied explicitly (migration 043 has no DEFAULT).
async with pool.acquire() as conn:
await conn.execute(
"""
INSERT INTO config.adapters (name, kind, enabled, cadence_s, settings, updated_at)
VALUES ($1, $2, $3, $4, $5, now())
""",
name,
kind,
enabled,
cadence_s,
new_settings,
)
await write_audit(
conn,
ADAPTER_CREATE,
operator_id=operator.id,
target=name,
after={"kind": kind, "enabled": enabled, "cadence_s": cadence_s, "settings": new_settings},
)
return RedirectResponse(url=f"/adapters/{name}", status_code=302)
@router.get("/adapters/{name}", response_class=HTMLResponse)
async def adapters_edit_form(
request: Request,
@ -1540,14 +1906,10 @@ async def adapters_edit_form(
pool = get_pool()
operator = request.state.operator
# Look up the adapter class
adapter_classes = _adapter_classes()
adapter_cls = adapter_classes.get(name)
async with pool.acquire() as conn:
row = await conn.fetchrow(
"""
SELECT name, enabled, cadence_s, settings, paused_at, updated_at, last_error
SELECT name, kind, enabled, cadence_s, settings, paused_at, updated_at, last_error
FROM config.adapters
WHERE name = $1
""",
@ -1564,6 +1926,10 @@ async def adapters_edit_form(
tile_url = sys_row["map_tile_url"] if sys_row else "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"
tile_attribution = sys_row["map_attribution"] if sys_row else "&copy; OpenStreetMap contributors"
# Look up the adapter class by kind (class identity), not instance name.
adapter_classes = _adapter_classes()
adapter_cls = adapter_classes.get(row["kind"])
settings = row["settings"] or {}
# Build adapter dict with class metadata
@ -1609,6 +1975,7 @@ async def adapters_edit_form(
settings_obj = adapter_cls.settings_schema(**settings)
preview_cfg = AdapterConfig(
name=row["name"],
kind=row["kind"],
enabled=row["enabled"],
cadence_s=row["cadence_s"],
settings=settings,
@ -1672,10 +2039,6 @@ async def adapters_edit_submit(
if not form_csrf or form_csrf != request.state.csrf_token:
raise CsrfValidationError("Invalid CSRF token")
# Look up the adapter class
adapter_classes = _adapter_classes()
adapter_cls = adapter_classes.get(name)
# Parse common form fields
enabled = "enabled" in form
cadence_s_str = form.get("cadence_s", "")
@ -1701,7 +2064,7 @@ async def adapters_edit_submit(
# Get current adapter state
row = await conn.fetchrow(
"""
SELECT name, enabled, cadence_s, settings, paused_at, updated_at, last_error
SELECT name, kind, enabled, cadence_s, settings, paused_at, updated_at, last_error
FROM config.adapters
WHERE name = $1
""",
@ -1711,146 +2074,35 @@ async def adapters_edit_submit(
if row is None:
return Response(status_code=404, content="Adapter not found")
# Look up the adapter class by kind (class identity), not instance name.
adapter_classes = _adapter_classes()
adapter_cls = adapter_classes.get(row["kind"])
current_settings = row["settings"] or {}
# Parse and validate settings via Pydantic if we have the adapter class
new_settings = {}
# Collect raw form values into form_data for error re-renders.
if adapter_cls and hasattr(adapter_cls, "settings_schema"):
schema = adapter_cls.settings_schema
fields = describe_fields(schema, current_settings)
# Parse form values based on widget type
parsed_values = {}
for field in fields:
raw = form.get(field.name, "")
form_data[field.name] = raw
if field.widget == "text":
parsed_values[field.name] = raw.strip() if raw else None
elif field.widget == "number":
try:
parsed_values[field.name] = int(raw) if raw else None
except ValueError:
errors[field.name] = f"{field.label} must be a number"
elif field.widget == "checkbox":
parsed_values[field.name] = field.name in form
elif field.widget == "csv":
if raw.strip():
parsed_values[field.name] = [v.strip() for v in raw.split(",") if v.strip()]
for _f in describe_fields(adapter_cls.settings_schema, current_settings):
if _f.widget == "checkboxes":
form_data[_f.name] = form.getlist(_f.name)
elif _f.widget == "model_list":
form_data[_f.name] = _parse_model_list(form, _f)
else:
parsed_values[field.name] = []
elif field.widget == "csv_int":
# v0.11.3: parallel to "csv" but coerces each token through
# int(), dropping non-numeric entries with a warning.
parsed_ints: list[int] = []
if raw.strip():
for tok in raw.split(","):
tok = tok.strip()
if not tok:
continue
try:
parsed_ints.append(int(tok))
except ValueError:
logger.warning(
"csv_int: dropped non-numeric token",
extra={"field": field.name, "token": tok},
)
parsed_values[field.name] = parsed_ints
elif field.widget == "select":
value = raw.strip() if raw else None
if value and field.options and value not in field.options:
errors[field.name] = f"Invalid {field.label.lower()}"
else:
parsed_values[field.name] = value
elif field.widget == "checkboxes":
# Use getlist for checkbox groups
values = form.getlist(field.name)
form_data[field.name] = values # Override raw value
if field.options:
invalid = [v for v in values if v not in field.options]
if invalid:
errors[field.name] = f"Invalid values: {', '.join(invalid)}"
else:
parsed_values[field.name] = values
else:
parsed_values[field.name] = values
elif field.widget == "api_key_select":
# API key select - validate against existing keys
value = raw.strip() if raw else None
parsed_values[field.name] = value
elif field.widget == "model_list":
rows = _parse_model_list(form, field)
form_data[field.name] = rows
parsed_values[field.name] = rows
elif field.widget == "region":
# Region handled separately below
pass
form_data[_f.name] = form.get(_f.name, "")
form_data["region_north"] = form.get("region_north", "").strip()
form_data["region_south"] = form.get("region_south", "").strip()
form_data["region_east"] = form.get("region_east", "").strip()
form_data["region_west"] = form.get("region_west", "").strip()
# Handle region fields (common pattern)
region_north_str = form.get("region_north", "").strip()
region_south_str = form.get("region_south", "").strip()
region_east_str = form.get("region_east", "").strip()
region_west_str = form.get("region_west", "").strip()
form_data["region_north"] = region_north_str
form_data["region_south"] = region_south_str
form_data["region_east"] = region_east_str
form_data["region_west"] = region_west_str
# Check if any region field has a value
has_region = any([region_north_str, region_south_str, region_east_str, region_west_str])
if has_region:
try:
region_north = float(region_north_str)
region_south = float(region_south_str)
region_east = float(region_east_str)
region_west = float(region_west_str)
if not (-90 <= region_south < region_north <= 90):
errors["region"] = "Invalid latitude: south must be less than north, both between -90 and 90"
elif not (-180 <= region_west < region_east <= 180):
errors["region"] = "Invalid longitude: west must be less than east, both between -180 and 180"
else:
parsed_values["region"] = {
"north": region_north,
"south": region_south,
"east": region_east,
"west": region_west,
}
except ValueError:
errors["region"] = "Region coordinates must be valid numbers"
else:
parsed_values["region"] = None
# Only validate with Pydantic if no parse errors
# Parse + validate settings via the shared helper.
# Mirror the old behavior: skip Pydantic validation when upstream
# checks (e.g. cadence) already failed, same as the old
# "if not errors: try: validated = schema(...)" guard.
if not errors:
try:
# Filter out None values for optional fields without defaults
validated_data = {k: v for k, v in parsed_values.items() if v is not None}
validated = schema(**validated_data)
new_settings = validated.model_dump(mode="json")
# Hard-block a save that would blow the provider free tier.
q = adapter_cls.quota_estimate(validated, cadence_s)
if q and q.get("blocked"):
ml = next((f.name for f in fields if f.widget == "model_list"), "quota")
errors[ml] = (
f"Estimated {q['calls_per_month']:,} calls/month exceeds the "
f"{q['cap']:,}/month free-tier cap — raise cadence or remove rows."
new_settings, settings_errors = _parse_adapter_settings(
form, adapter_cls, current_settings, cadence_s
)
except ValidationError as e:
ml_name = next((f.name for f in fields if f.widget == "model_list"), None)
for err in e.errors():
loc = err["loc"]
key = str(loc[0]) if loc else (ml_name or "unknown")
if len(loc) >= 2 and isinstance(loc[1], int):
errors[key] = f"Row {loc[1] + 1}: {err['msg']}"
else:
errors[key] = err["msg"]
else:
# No schema - just preserve existing settings
new_settings = dict(current_settings)
errors.update(settings_errors)
# If there are errors, re-render the form
if errors:
@ -1966,6 +2218,71 @@ async def adapters_edit_submit(
return RedirectResponse(url="/adapters", status_code=302)
@router.post("/adapters/{name}/delete")
async def adapters_delete(request: Request, name: str) -> Response:
"""Delete an operator-created adapter instance.
Safety rule: a row is deletable iff its ``name`` is NOT a key in the
adapter class registry. Built-in adapters always have ``name == kind``
which IS a registry key; operator instances have a unique name that is NOT.
NOTE: orphaned ``published_ids`` rows in cursors.db (a separate SQLite
database) are left to age out via ``dedup_sweep_days``; they are not
cleaned here because the two stores are decoupled by design.
"""
pool = get_pool()
operator = request.state.operator
form = await request.form()
form_csrf = form.get("csrf_token", "")
if not form_csrf or form_csrf != request.state.csrf_token:
raise CsrfValidationError("Invalid CSRF token")
adapter_classes = _adapter_classes()
async with pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT name, kind FROM config.adapters WHERE name = $1", name
)
if row is None:
return Response(status_code=404, content=f"Adapter '{name}' not found.")
# Primary guard: built-ins have name == kind (a registered class key).
if name in adapter_classes:
return Response(
status_code=403,
content=(
f"'{name}' is a built-in adapter and cannot be deleted; "
"disable it instead."
),
)
# Second guard: the row's kind must be operator_creatable.
kind_cls = adapter_classes.get(row["kind"])
if kind_cls is not None and not getattr(kind_cls, "operator_creatable", False):
return Response(
status_code=403,
content=(
f"Adapter kind '{row['kind']}' is not operator-creatable; "
"cannot delete."
),
)
await conn.execute(
"DELETE FROM config.adapters WHERE name = $1", name
)
await write_audit(
conn,
ADAPTER_DELETE,
operator_id=operator.id,
target=name,
before={"kind": row["kind"], "name": name},
)
return RedirectResponse(url="/adapters", status_code=302)
# =============================================================================
# Streams routes
# =============================================================================

View file

@ -0,0 +1,3 @@
{# Generic HTTP source. Fields from payload->data->data. #}
{% set d = (event.data.get('data') or {}).get('data') or {} %}
{% if d.get('title') is not none %}<dt>Title</dt><dd>{{ d.title }}</dd>{% endif %}

View file

@ -0,0 +1,2 @@
{% set d = (event.data.get('data') or {}).get('data') or {} %}
{%- if d.get('title') %}{{ d.title }}{% endif -%}

View file

@ -4,6 +4,7 @@
{% block content %}
<h1>Adapters</h1>
<p><a href="/adapters/new" role="button">+ New adapter</a></p>
<table>
<thead>
<tr>
@ -12,6 +13,7 @@
<th>Cadence</th>
<th>Last Updated</th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
@ -27,6 +29,15 @@
<td>{{ adapter.cadence_s }}s</td>
<td>{{ adapter.updated_at.strftime('%Y-%m-%d %H:%M') if adapter.updated_at else '—' }}</td>
<td><a href="/adapters/{{ adapter.name }}">Edit</a></td>
<td>
{% if adapter.deletable %}
<form method="post" action="/adapters/{{ adapter.name }}/delete" style="display:inline;"
onsubmit="return confirm('Delete adapter &quot;{{ adapter.name }}&quot;? This cannot be undone.')">
<input type="hidden" name="csrf_token" value="{{ csrf_token }}">
<button type="submit" class="btn-danger">Delete</button>
</form>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>

View file

@ -0,0 +1,192 @@
{% extends "base.html" %}
{% block title %}Central — New Adapter{% endblock %}
{% block content %}
<h1>New Adapter</h1>
<p class="muted">Create a new adapter instance from an operator-creatable kind.</p>
<form method="post" action="/adapters/new">
<input type="hidden" name="csrf_token" value="{{ csrf_token }}">
<fieldset>
<legend>Adapter Kind</legend>
{# NOTE: single creatable kind today; multi-kind HTMX field-swap is a future enhancement. #}
<label for="kind">Kind</label>
<select id="kind" name="kind">
{% for ck in creatable_kinds %}
<option value="{{ ck.kind }}"
{% if ck.kind == (form_data.kind if form_data else selected_kind) %}selected{% endif %}>
{{ ck.display_name }} ({{ ck.kind }})
</option>
{% endfor %}
</select>
{% if errors and errors.kind %}
<small class="field-error">{{ errors.kind }}</small>
{% endif %}
</fieldset>
<fieldset>
<legend>Instance Identity</legend>
<label for="name">Instance Name</label>
<input type="text" id="name" name="name"
value="{{ form_data.name if form_data else '' }}"
placeholder="e.g. my_source_v2"
required>
<small>Lowercase letters, digits, and underscores; starts with a letter; 264 characters.
Must not match a built-in kind name.</small>
{% if errors and errors.name %}
<small class="field-error">{{ errors.name }}</small>
{% endif %}
</fieldset>
<fieldset>
<legend>Core Settings</legend>
<label>
<input type="checkbox" name="enabled"
{% if form_data and form_data.enabled %}checked{% endif %}>
Enabled <small>(leave unchecked — ships disabled; enable after verifying settings)</small>
</label>
<label for="cadence_s">Cadence (seconds)</label>
<input type="number" id="cadence_s" name="cadence_s"
value="{{ form_data.cadence_s if form_data else default_cadence_s }}"
min="10"
required>
{% if errors and errors.cadence_s %}
<small class="field-error">{{ errors.cadence_s }}</small>
{% endif %}
</fieldset>
{% if fields %}
<fieldset>
<legend>Adapter Settings</legend>
{% for field in fields %}
{% if field.widget == "region" %}
{# Region is rendered in a separate fieldset below #}
{% elif field.widget == "text" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<input type="text" id="{{ field.name }}" name="{{ field.name }}"
value="{{ form_data[field.name] if form_data and field.name in form_data else field.current_value or '' }}"
{% if field.required %}required{% endif %}>
{% if field.description %}
<small>{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "number" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<input type="number" id="{{ field.name }}" name="{{ field.name }}"
value="{{ form_data[field.name] if form_data and field.name in form_data else field.current_value or '' }}"
{% if field.required %}required{% endif %}>
{% if field.description %}
<small>{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "checkbox" %}
<label>
<input type="checkbox" name="{{ field.name }}"
{% if form_data and field.name in form_data %}
{% if form_data[field.name] %}checked{% endif %}
{% elif field.current_value %}checked{% endif %}>
{{ field.label }}
</label>
{% if field.description %}
<small>{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "csv" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<input type="text" id="{{ field.name }}" name="{{ field.name }}"
value="{{ form_data[field.name] if form_data and field.name in form_data else (field.current_value | join(',') if field.current_value else '') }}"
{% if field.required %}required{% endif %}>
<small>Comma-separated values{% if field.description %} — {{ field.description }}{% endif %}</small>
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "csv_int" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<input type="text" id="{{ field.name }}" name="{{ field.name }}"
value="{{ form_data[field.name] if form_data and field.name in form_data else (field.current_value | join(',') if field.current_value else '') }}"
{% if field.required %}required{% endif %}>
<small>Comma-separated integers{% if field.description %} — {{ field.description }}{% endif %}</small>
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "select" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<select id="{{ field.name }}" name="{{ field.name }}">
{% for opt in field.options %}
<option value="{{ opt }}"
{% if (form_data[field.name] if form_data and field.name in form_data else field.current_value) == opt %}selected{% endif %}>
{{ opt }}
</option>
{% endfor %}
</select>
{% if field.description %}
<small>{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "checkboxes" %}
<label>{{ field.label }}</label>
{% set current_values = form_data.getlist(field.name) if form_data and form_data.getlist else (field.current_value or []) %}
{% for opt in field.options %}
<label style="display: inline-block; margin-right: 1rem;">
<input type="checkbox" name="{{ field.name }}" value="{{ opt }}"
{% if opt in current_values %}checked{% endif %}>
{{ opt }}
</label>
{% endfor %}
{% if field.description %}
<small style="display: block;">{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "api_key_select" %}
<label for="{{ field.name }}">{{ field.label }}</label>
<select id="{{ field.name }}" name="{{ field.name }}">
<option value="">(none)</option>
{% for key in api_keys %}
<option value="{{ key.alias }}"
{% if (form_data[field.name] if form_data and field.name in form_data else field.current_value) == key.alias %}selected{% endif %}>
{{ key.alias }}
</option>
{% endfor %}
</select>
{% if field.description %}
<small>{{ field.description }}</small>
{% endif %}
{% if errors and errors[field.name] %}
<small class="field-error">{{ errors[field.name] }}</small>
{% endif %}
{% elif field.widget == "model_list" %}
{% include "_partials/model_list.html" %}
{% endif %}
{% endfor %}
</fieldset>
{% endif %}
<button type="submit">Create Adapter</button>
<a href="/adapters" role="button" class="btn-outline">Cancel</a>
</form>
{% endblock %}

View file

@ -355,10 +355,18 @@ class Supervisor:
)
def _create_adapter(self, config: AdapterConfig) -> SourceAdapter:
"""Create an adapter instance based on config name."""
cls = self._adapters.get(config.name)
"""Create an adapter instance based on config kind (class identity).
config.kind names the adapter *class* in the registry (discover_adapters
keys by SourceAdapter.name on the class). config.name is the unique
*instance* identifier and is unchanged here runtime state, logs, and
dedup tables continue to key on config.name.
"""
cls = self._adapters.get(config.kind)
if cls is None:
raise ValueError(f"Unknown adapter type: {config.name}")
raise ValueError(
f"Unknown adapter kind: {config.kind!r} (instance: {config.name!r})"
)
return cls(
config=config,
config_store=self._config_store,
@ -524,8 +532,9 @@ class Supervisor:
# API key precondition — resolve via per-row settings[api_key_field]
# (operator-selected alias), falling back to the class-attribute
# default when settings hasn't been set. Returns None when no key
# is required.
adapter_cls = self._adapters.get(config.name)
# is required. Resolve the class by config.kind (class identity), not
# config.name (instance identity) — mirrors _create_adapter.
adapter_cls = self._adapters.get(config.kind)
alias = resolve_api_key_alias(adapter_cls, config.settings)
if alias is not None:
key_value = await self._config_store.get_api_key(alias)

View file

@ -0,0 +1,230 @@
"""Tests for AdapterConfig.kind and supervisor._create_adapter kind-based dispatch.
All tests are pure-unit (no DB, no NATS) so they run anywhere.
"""
import os
import tempfile
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock
import pytest
# Provide required env vars before importing central modules.
os.environ.setdefault("CENTRAL_DB_DSN", "postgresql://test:test@localhost/test")
os.environ.setdefault("CENTRAL_CSRF_SECRET", "testsecret12345678901234567890ab")
os.environ.setdefault("CENTRAL_NATS_URL", "nats://localhost:4222")
from central.config_models import AdapterConfig
_NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
def _cfg(**kwargs) -> AdapterConfig:
"""Build a minimal AdapterConfig with sensible defaults."""
defaults = {
"name": "usgs_quake",
"cadence_s": 120,
"updated_at": _NOW,
}
defaults.update(kwargs)
return AdapterConfig(**defaults)
# ---------------------------------------------------------------------------
# AdapterConfig.kind back-compat
# ---------------------------------------------------------------------------
class TestAdapterConfigKind:
def test_kind_defaults_to_name_when_absent(self):
"""When kind is not supplied, it falls back to name (pre-043 rows)."""
cfg = _cfg(name="usgs_quake")
assert cfg.kind == "usgs_quake"
def test_kind_defaults_to_name_when_none(self):
"""When kind is explicitly None (NULL from DB), it falls back to name."""
cfg = _cfg(name="nws", kind=None)
assert cfg.kind == "nws"
def test_kind_preserved_when_set(self):
"""When kind is explicitly supplied, it is kept as-is."""
cfg = _cfg(name="my_quake_instance", kind="usgs_quake")
assert cfg.kind == "usgs_quake"
assert cfg.name == "my_quake_instance"
def test_name_unchanged_when_kind_differs(self):
"""Instance identity (name) is never overwritten by the kind back-fill."""
cfg = _cfg(name="my_custom_quake", kind="usgs_quake")
assert cfg.name == "my_custom_quake"
def test_kind_equals_name_for_builtin_pattern(self):
"""Built-in adapters: name == kind both before and after the migration."""
cfg = _cfg(name="nws", kind="nws")
assert cfg.kind == cfg.name == "nws"
# ---------------------------------------------------------------------------
# supervisor._create_adapter resolves class by kind
# ---------------------------------------------------------------------------
class TestCreateAdapterResolvesbyKind:
"""Unit-test _create_adapter without a live Supervisor (no NATS, no DB)."""
def _make_supervisor_with_registry(self, registry: dict):
"""Return a minimal Supervisor-like object with _adapters = registry."""
# Import lazily to avoid triggering the ENRICHMENT_CACHE_DB_PATH side
# effect before conftest patches it (conftest autouse fixture handles it
# in the full test run; here we patch manually).
from unittest.mock import patch
import central.supervisor as sup_mod
with tempfile.TemporaryDirectory() as td:
with patch.object(sup_mod, "ENRICHMENT_CACHE_DB_PATH", Path(td) / "ec.db"):
with patch.object(sup_mod, "CURSOR_DB_PATH", Path(td) / "cursor.db"):
supervisor = MagicMock()
supervisor._adapters = registry
supervisor._config_store = MagicMock()
# Bind the real _create_adapter method to our mock object.
supervisor._create_adapter = (
sup_mod.Supervisor._create_adapter.__get__(supervisor)
)
return supervisor
def _make_mock_adapter_cls(self, class_name: str):
"""Return a mock adapter class whose constructor returns a mock instance."""
instance = MagicMock()
instance.name = class_name # class-level .name attribute
cls = MagicMock()
cls.return_value = instance
return cls
def test_resolves_class_by_kind_when_kind_differs_from_name(self):
"""_create_adapter uses config.kind (class key), not config.name."""
usgs_cls = self._make_mock_adapter_cls("usgs_quake")
registry = {"usgs_quake": usgs_cls}
sup = self._make_supervisor_with_registry(registry)
cfg = _cfg(name="my_quake_instance", kind="usgs_quake")
result = sup._create_adapter(cfg)
# Class was looked up by kind and instantiated.
usgs_cls.assert_called_once()
assert result is usgs_cls.return_value
def test_builtin_pattern_kind_equals_name_still_works(self):
"""Regression: existing adapters with name == kind construct correctly."""
usgs_cls = self._make_mock_adapter_cls("usgs_quake")
registry = {"usgs_quake": usgs_cls}
sup = self._make_supervisor_with_registry(registry)
cfg = _cfg(name="usgs_quake", kind="usgs_quake")
result = sup._create_adapter(cfg)
usgs_cls.assert_called_once()
assert result is usgs_cls.return_value
def test_raises_on_unknown_kind(self):
"""_create_adapter raises ValueError mentioning both kind and instance name."""
registry = {}
sup = self._make_supervisor_with_registry(registry)
cfg = _cfg(name="my_instance", kind="nonexistent_adapter")
with pytest.raises(ValueError) as exc_info:
sup._create_adapter(cfg)
msg = str(exc_info.value)
assert "nonexistent_adapter" in msg
assert "my_instance" in msg
def test_instance_identity_passed_through_to_constructor(self):
"""The adapter constructor receives the full config (name = instance id)."""
some_cls = self._make_mock_adapter_cls("some_adapter")
registry = {"some_adapter": some_cls}
sup = self._make_supervisor_with_registry(registry)
cfg = _cfg(name="prod_instance_1", kind="some_adapter")
sup._create_adapter(cfg)
# Check the config passed to constructor has the instance name.
call_kwargs = some_cls.call_args
passed_config = call_kwargs[1].get("config") or call_kwargs[0][0]
assert passed_config.name == "prod_instance_1"
class TestStartAdapterResolvesByKind:
"""The api-key precondition in _start_adapter also resolves the class by kind.
_start_adapter is callable directly with mocks (no live DB), mirroring
tests/test_requires_api_key.py::TestSupervisorApiKeyPrecondition. A generic
instance where name != kind must still find its class via the registry's
kind key otherwise the lookup returns None, resolve_api_key_alias treats
it as "no key required", and the adapter silently skips its api-key check
and starts. Resolving by kind keeps the precondition enforced.
"""
@pytest.mark.asyncio
async def test_precondition_refuses_when_key_missing_for_instance_named_differently(
self, tmp_path: Path
):
from central.supervisor import Supervisor
from central.adapters.firms import FIRMSAdapter
mock_config_store = MagicMock()
mock_config_store.get_api_key = AsyncMock(return_value=None) # key missing
mock_config_store.set_adapter_last_error = AsyncMock()
mock_nats = MagicMock()
mock_nats.publish = AsyncMock()
supervisor = Supervisor.__new__(Supervisor)
supervisor._config_store = mock_config_store
# Registry is keyed by class identity (kind == FIRMSAdapter.name == "firms").
supervisor._adapters = {"firms": FIRMSAdapter}
supervisor._adapter_states = {}
supervisor._nats = mock_nats
supervisor._cursor_db_path = tmp_path / "cursors.db"
supervisor._log = MagicMock()
# Instance name differs from kind — the generic-instance case.
config = _cfg(
name="my_firms_instance",
kind="firms",
enabled=True,
cadence_s=300,
settings={"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"]},
)
await supervisor._start_adapter(config)
# The class WAS resolved by kind: requires_api_key fired, key was found
# missing, and the adapter refused to start. Had the lookup used
# config.name ("my_firms_instance", absent from the registry), the
# precondition would have been skipped and the adapter would have started.
mock_config_store.get_api_key.assert_called_once_with("firms")
mock_config_store.set_adapter_last_error.assert_called_once()
err_args = mock_config_store.set_adapter_last_error.call_args[0]
assert err_args[0] == "my_firms_instance" # error keyed by instance id
assert "missing api key" in err_args[1].lower()
# Did not start.
assert "my_firms_instance" not in supervisor._adapter_states
mock_nats.publish.assert_not_called()
def test_start_path_lookup_expression_uses_kind(self):
"""Belt-and-suspenders: the start-path lookup keys on config.kind.
Direct assertion of the resolution expression (supervisor._adapters.get(
config.kind)) independent of the heavier _start_adapter flow above.
"""
from central.adapters.firms import FIRMSAdapter
registry = {"firms": FIRMSAdapter}
config = _cfg(name="my_firms_instance", kind="firms")
# Resolving by kind finds the class; resolving by name would miss.
assert registry.get(config.kind) is FIRMSAdapter
assert registry.get(config.name) is None

View file

@ -42,9 +42,9 @@ class TestAdaptersListAuthenticated:
mock_conn = AsyncMock()
mock_conn.fetch.return_value = [
{"name": "firms", "enabled": True, "cadence_s": 300, "settings": {"api_key_alias": "firms"}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "nws", "enabled": True, "cadence_s": 60, "settings": {"contact_email": "test@test.com"}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "usgs_quake", "enabled": True, "cadence_s": 120, "settings": {"feed": "all_hour"}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "firms", "kind": "firms", "enabled": True, "cadence_s": 300, "settings": {"api_key_alias": "firms"}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "nws", "kind": "nws", "enabled": True, "cadence_s": 60, "settings": {"contact_email": "test@test.com"}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "usgs_quake", "kind": "usgs_quake", "enabled": True, "cadence_s": 120, "settings": {"feed": "all_hour"}, "paused_at": None, "updated_at": None, "last_error": None},
]
mock_pool = MagicMock()
@ -97,6 +97,7 @@ class TestAdaptersEditForm:
mock_conn.fetchrow.side_effect = [
{
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "test@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}},
@ -178,6 +179,7 @@ class TestAdaptersEditSubmit:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "old@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}},
@ -227,6 +229,7 @@ class TestAdaptersEditSubmit:
mock_conn.fetchrow.side_effect = [
{
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "test@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}},
@ -286,6 +289,7 @@ class TestAdaptersAudit:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "old@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}},
@ -350,6 +354,7 @@ class TestAdaptersJsonbRegression:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "old@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}}, # dict, as asyncpg returns
@ -402,6 +407,7 @@ class TestAdaptersJsonbRegression:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "nws",
"kind": "nws",
"enabled": True,
"cadence_s": 60,
"settings": {"contact_email": "old@example.com", "region": {"north": 49, "south": 24, "east": -66, "west": -125}}, # dict
@ -443,7 +449,7 @@ class TestAdaptersJsonbRegression:
mock_conn = MagicMock()
mock_conn.fetchrow = AsyncMock(side_effect=[
# Adapter row
{"name": "firms", "enabled": True, "cadence_s": 300, "settings": {},
{"name": "firms", "kind": "firms", "enabled": True, "cadence_s": 300, "settings": {},
"paused_at": None, "updated_at": None, "last_error": None},
# System row
{"map_tile_url": "https://tile.example.com", "map_attribution": "Test"},

View file

@ -1189,6 +1189,7 @@ _SAMPLE_INNER = {
"current_lat_deg": -17.1487,
"current_alt_km": 417.4,
},
"generic_http": {"title": "Generic source alert"},
}
# Exact expected subjects for the deterministic adapters. swpc_alerts is omitted

816
tests/test_generic_http.py Normal file
View file

@ -0,0 +1,816 @@
"""Unit tests for GenericHttpAdapter.
Pure unit tests: no database mocking required (we use a real sqlite tmp file
for dedup tests, same as test_usgs_quake.py), and HTTP is patched.
"""
import tempfile
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from pydantic import ValidationError
from central.adapters.generic_http import (
FieldMapping,
GenericHttpAdapter,
GenericHttpSettings,
_dig,
)
from central.config_models import AdapterConfig
from central.models import Event, Geo
from central.streams import STREAMS
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _valid_domain() -> str:
"""Return a known valid domain (first event-bearing stream)."""
for s in STREAMS:
domain = s.subject_filter.split(".")[1]
if domain != "meta":
return domain
raise RuntimeError("No non-meta stream found in STREAMS")
def make_config(
name: str = "test_generic",
domain: str | None = None,
extra_settings: dict | None = None,
) -> AdapterConfig:
domain = domain or _valid_domain()
settings: dict = {
"url": "https://example.com/feed.json",
"domain": domain,
"id_path": "id",
"items_path": "features",
"geometry_path": "geometry",
}
if extra_settings:
settings.update(extra_settings)
return AdapterConfig(
name=name,
kind="generic_http",
enabled=True,
cadence_s=300,
settings=settings,
updated_at=datetime.now(timezone.utc),
)
@pytest.fixture
def temp_db_path():
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f:
yield Path(f.name)
@pytest.fixture
def mock_config_store():
return MagicMock()
# ---------------------------------------------------------------------------
# _dig
# ---------------------------------------------------------------------------
class TestDig:
def test_simple_key(self):
assert _dig({"a": 1}, "a") == 1
def test_nested_keys(self):
assert _dig({"a": {"b": {"c": 42}}}, "a.b.c") == 42
def test_list_index(self):
assert _dig({"a": [10, 20, 30]}, "a.1") == 20
def test_list_index_zero(self):
assert _dig({"items": [{"x": 99}]}, "items.0.x") == 99
def test_missing_key_returns_none(self):
assert _dig({"a": 1}, "b") is None
def test_missing_nested_returns_none(self):
assert _dig({"a": {"b": 1}}, "a.c") is None
def test_out_of_range_index_returns_none(self):
assert _dig({"a": [1, 2]}, "a.5") is None
def test_none_root_returns_none(self):
assert _dig(None, "a.b") is None
def test_non_dict_mid_path_returns_none(self):
assert _dig({"a": 42}, "a.b") is None
def test_empty_list(self):
assert _dig([], "0") is None
# ---------------------------------------------------------------------------
# Domain validation
# ---------------------------------------------------------------------------
class TestDomainValidation:
def test_unknown_domain_raises(self):
with pytest.raises(ValidationError) as exc_info:
GenericHttpSettings(
url="https://example.com/feed",
domain="totally_unknown_domain_xyz",
id_path="id",
)
err_str = str(exc_info.value)
assert "unknown domain" in err_str
# Should list valid options in the error message
assert "wx" in err_str or "fire" in err_str
def test_known_domain_ok(self):
s = GenericHttpSettings(
url="https://example.com/feed",
domain=_valid_domain(),
id_path="id",
)
assert s.domain == _valid_domain()
def test_wx_domain_valid(self):
s = GenericHttpSettings(
url="https://example.com/feed",
domain="wx",
id_path="id",
)
assert s.domain == "wx"
def test_fire_domain_valid(self):
s = GenericHttpSettings(
url="https://example.com/feed",
domain="fire",
id_path="id",
)
assert s.domain == "fire"
# ---------------------------------------------------------------------------
# Category composition
# ---------------------------------------------------------------------------
class TestCategoryComposition:
@pytest.mark.asyncio
async def test_category_default_suffix(self, temp_db_path, mock_config_store):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {
"features": [
{
"id": "item-1",
"geometry": {"type": "Point", "coordinates": [-116.0, 43.0]},
}
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mock_fetch:
mock_fetch.return_value = geojson
events = [e async for e in adapter.poll()]
assert len(events) == 1
assert events[0].category == "wx.alert"
await adapter.shutdown()
@pytest.mark.asyncio
async def test_category_custom_suffix(self, temp_db_path, mock_config_store):
config = make_config(domain="wx", extra_settings={"category_suffix": "warning"})
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {
"features": [
{
"id": "item-wx-warn",
"geometry": {"type": "Point", "coordinates": [-116.0, 43.0]},
}
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
events = [e async for e in adapter.poll()]
assert events[0].category == "wx.warning"
await adapter.shutdown()
# ---------------------------------------------------------------------------
# GeoJSON path — geometry + centroid
# ---------------------------------------------------------------------------
GEOJSON_FIXTURE = {
"type": "FeatureCollection",
"features": [
{
"id": "feat-001",
"type": "Feature",
"properties": {
"title": "Test Alert",
"severity": 2,
"updated": "2025-01-15T12:00:00Z",
},
"geometry": {
"type": "Point",
"coordinates": [-116.2, 43.7],
},
},
{
"id": "feat-002",
"type": "Feature",
"properties": {
"title": "Polygon Alert",
"severity": 3,
"updated": "2025-01-15T13:00:00Z",
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-116, 43], [-115, 43], [-115, 44], [-116, 44], [-116, 43]]],
},
},
],
}
class TestGeoJsonPath:
@pytest.mark.asyncio
async def test_point_geometry_and_centroid(self, temp_db_path, mock_config_store):
config = make_config(
domain="wx",
extra_settings={
"items_path": "features",
"id_path": "id",
"geometry_path": "geometry",
"title_path": "properties.title",
"severity_path": "properties.severity",
"field_mappings": [
{"source_path": "properties.title", "dest_key": "title"},
],
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = GEOJSON_FIXTURE
events = [e async for e in adapter.poll()]
assert len(events) == 2
point_event = next(e for e in events if e.id == "feat-001")
# geo.geometry set
assert point_event.geo.geometry == {
"type": "Point",
"coordinates": [-116.2, 43.7],
}
# centroid extracted from Point
assert point_event.geo.centroid == (-116.2, 43.7)
await adapter.shutdown()
@pytest.mark.asyncio
async def test_polygon_geometry_no_centroid(self, temp_db_path, mock_config_store):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = GEOJSON_FIXTURE
events = [e async for e in adapter.poll()]
poly_event = next(e for e in events if e.id == "feat-002")
# geometry is set
assert poly_event.geo.geometry is not None
assert poly_event.geo.geometry["type"] == "Polygon"
# centroid is NOT set (non-Point geometry)
assert poly_event.geo.centroid is None
await adapter.shutdown()
@pytest.mark.asyncio
async def test_field_mappings_populate_data(self, temp_db_path, mock_config_store):
config = make_config(
domain="wx",
extra_settings={
"id_path": "id",
"title_path": "properties.title",
"field_mappings": [
{"source_path": "properties.severity", "dest_key": "level"},
{"source_path": "properties.updated", "dest_key": "updated_at"},
],
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = GEOJSON_FIXTURE
events = [e async for e in adapter.poll()]
e = next(ev for ev in events if ev.id == "feat-001")
assert e.data["title"] == "Test Alert"
assert e.data["level"] == 2
assert e.data["updated_at"] == "2025-01-15T12:00:00Z"
await adapter.shutdown()
# ---------------------------------------------------------------------------
# JSON (non-GeoJSON) path — lat_path / lon_path
# ---------------------------------------------------------------------------
JSON_FIXTURE = {
"alerts": [
{"uid": "a1", "name": "Alert One", "lat": 43.5, "lon": -116.1, "sev": 1},
{"uid": "a2", "name": "Alert Two", "lat": 44.0, "lon": -115.5, "sev": 3},
]
}
class TestJsonLatLonPath:
@pytest.mark.asyncio
async def test_lat_lon_centroid(self, temp_db_path, mock_config_store):
config = make_config(
domain="fire",
extra_settings={
"items_path": "alerts",
"id_path": "uid",
"geometry_path": "", # disable geometry_path extraction
"lat_path": "lat",
"lon_path": "lon",
"title_path": "name",
"severity_path": "sev",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = JSON_FIXTURE
events = [e async for e in adapter.poll()]
assert len(events) == 2
e1 = next(e for e in events if e.id == "a1")
# centroid is (lon, lat) per GeoJSON convention
assert e1.geo.centroid == (-116.1, 43.5)
assert e1.data["title"] == "Alert One"
assert e1.severity == 1
await adapter.shutdown()
# ---------------------------------------------------------------------------
# Dedup
# ---------------------------------------------------------------------------
class TestDedup:
@pytest.mark.asyncio
async def test_second_poll_yields_nothing(self, temp_db_path, mock_config_store):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {
"features": [
{"id": "dedup-1", "geometry": None},
{"id": "dedup-2", "geometry": None},
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
first = [e async for e in adapter.poll()]
second = [e async for e in adapter.poll()]
assert len(first) == 2
assert len(second) == 0
await adapter.shutdown()
@pytest.mark.asyncio
async def test_new_item_in_second_poll_is_yielded(
self, temp_db_path, mock_config_store
):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
first_batch = {"features": [{"id": "old-1", "geometry": None}]}
second_batch = {
"features": [
{"id": "old-1", "geometry": None},
{"id": "new-2", "geometry": None},
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = first_batch
[e async for e in adapter.poll()] # consume first
mf.return_value = second_batch
second = [e async for e in adapter.poll()]
assert len(second) == 1
assert second[0].id == "new-2"
await adapter.shutdown()
@pytest.mark.asyncio
async def test_instance_name_used_for_dedup(
self, temp_db_path, mock_config_store
):
"""Two adapter instances share the same db but scope by instance name."""
config_a = make_config(name="instance_a", domain="wx")
config_b = make_config(name="instance_b", domain="fire")
adapter_a = GenericHttpAdapter(config_a, mock_config_store, temp_db_path)
adapter_b = GenericHttpAdapter(config_b, mock_config_store, temp_db_path)
await adapter_a.startup()
await adapter_b.startup()
# Publish "shared-id" under instance_a
adapter_a.mark_published("shared-id")
# same id is NOT published under instance_b
assert not adapter_b.is_published("shared-id")
# and IS published under instance_a
assert adapter_a.is_published("shared-id")
await adapter_a.shutdown()
await adapter_b.shutdown()
# ---------------------------------------------------------------------------
# subject_for
# ---------------------------------------------------------------------------
class TestSubjectFor:
@pytest.mark.asyncio
async def test_subject_no_enrichment(self, temp_db_path, mock_config_store):
"""Without enrichment data, subject should end in 'unknown'."""
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
event = Event(
id="subj-1",
adapter="test_generic",
category="wx.alert",
time=datetime.now(timezone.utc),
geo=Geo(centroid=(-116.0, 43.0)),
data={},
)
subject = adapter.subject_for(event)
assert subject == "central.wx.unknown"
await adapter.shutdown()
@pytest.mark.asyncio
async def test_subject_with_us_enrichment(self, temp_db_path, mock_config_store):
"""With US geocoder enrichment, subject should be central.<domain>.us.<state>."""
config = make_config(domain="fire")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
event = Event(
id="subj-2",
adapter="test_generic",
category="fire.alert",
time=datetime.now(timezone.utc),
geo=Geo(centroid=(-116.0, 43.0)),
data={
"_enriched": {
"geocoder": {
"country": "United States",
"state": "Idaho",
}
}
},
)
subject = adapter.subject_for(event)
assert subject == "central.fire.us.id"
await adapter.shutdown()
@pytest.mark.asyncio
async def test_subject_format_domain_region(self, temp_db_path, mock_config_store):
"""Subject always matches central.<domain>.<region> — NOT central.<category>.<region>."""
config = make_config(domain="quake", extra_settings={"category_suffix": "event.minor"})
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
event = Event(
id="subj-3",
adapter="test_generic",
category="quake.event.minor",
time=datetime.now(timezone.utc),
geo=Geo(),
data={},
)
subject = adapter.subject_for(event)
# Should be central.quake.unknown, NOT central.quake.event.minor.unknown
assert subject == "central.quake.unknown"
assert subject.startswith("central.quake.")
await adapter.shutdown()
# ---------------------------------------------------------------------------
# time_path parsing
# ---------------------------------------------------------------------------
class TestTimePath:
@pytest.mark.asyncio
async def test_time_path_none_uses_now(self, temp_db_path, mock_config_store):
before = datetime.now(timezone.utc)
config = make_config(domain="wx", extra_settings={"time_path": None})
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {"features": [{"id": "t1", "geometry": None}]}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
events = [e async for e in adapter.poll()]
after = datetime.now(timezone.utc)
assert len(events) == 1
assert before <= events[0].time <= after
await adapter.shutdown()
@pytest.mark.asyncio
async def test_time_path_parsed_iso(self, temp_db_path, mock_config_store):
config = make_config(
domain="wx",
extra_settings={
"time_path": "properties.updated",
"id_path": "id",
"items_path": "features",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {
"features": [
{
"id": "t2",
"geometry": None,
"properties": {"updated": "2025-06-15T08:30:00Z"},
}
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
events = [e async for e in adapter.poll()]
assert len(events) == 1
assert events[0].time == datetime(2025, 6, 15, 8, 30, 0, tzinfo=timezone.utc)
await adapter.shutdown()
@pytest.mark.asyncio
async def test_time_path_missing_value_uses_now(
self, temp_db_path, mock_config_store
):
"""When time_path is set but the value is absent, fall back to now."""
before = datetime.now(timezone.utc)
config = make_config(
domain="wx",
extra_settings={"time_path": "properties.ts"},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {"features": [{"id": "t3", "geometry": None, "properties": {}}]}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
events = [e async for e in adapter.poll()]
after = datetime.now(timezone.utc)
assert len(events) == 1
assert before <= events[0].time <= after
await adapter.shutdown()
# ---------------------------------------------------------------------------
# apply_config
# ---------------------------------------------------------------------------
class TestApplyConfig:
@pytest.mark.asyncio
async def test_apply_config_updates_url(self, temp_db_path, mock_config_store):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
assert adapter._settings.url == "https://example.com/feed.json"
new_config = make_config(
domain="wx", extra_settings={"url": "https://other.example.com/data.json"}
)
await adapter.apply_config(new_config)
assert adapter._settings.url == "https://other.example.com/data.json"
await adapter.shutdown()
@pytest.mark.asyncio
async def test_apply_config_updates_domain(self, temp_db_path, mock_config_store):
config = make_config(domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
new_config = make_config(domain="fire")
await adapter.apply_config(new_config)
assert adapter._settings.domain == "fire"
await adapter.shutdown()
# ---------------------------------------------------------------------------
# Instance name scoping
# ---------------------------------------------------------------------------
class TestInstanceName:
def test_instance_name_is_config_name(self, temp_db_path, mock_config_store):
"""adapter.name must be the instance name, not the class name."""
config = make_config(name="my_noaa_alerts", domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
assert adapter.name == "my_noaa_alerts"
assert GenericHttpAdapter.name == "generic_http"
@pytest.mark.asyncio
async def test_event_adapter_field_is_instance_name(
self, temp_db_path, mock_config_store
):
config = make_config(name="my_source", domain="wx")
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
geojson = {"features": [{"id": "inst-1", "geometry": None}]}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = geojson
events = [e async for e in adapter.poll()]
assert len(events) == 1
assert events[0].adapter == "my_source"
await adapter.shutdown()
# ---------------------------------------------------------------------------
# Geocoder enrichment wiring
# ---------------------------------------------------------------------------
class TestEnrichmentLocations:
def test_enrichment_locations_declared(self):
assert GenericHttpAdapter.enrichment_locations == [("latitude", "longitude")]
@pytest.mark.asyncio
async def test_geojson_point_writes_latlon_to_data(
self, temp_db_path, mock_config_store
):
"""GeoJSON Point item: data["latitude"]/["longitude"] == geometry coords,
and geo.centroid == (lon, lat)."""
config = make_config(
domain="wx",
extra_settings={
"id_path": "id",
"geometry_path": "geometry",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
fixture = {
"features": [
{
"id": "enrich-point-1",
"geometry": {"type": "Point", "coordinates": [-116.2, 43.7]},
}
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = fixture
events = [e async for e in adapter.poll()]
assert len(events) == 1
e = events[0]
# lat = coords[1], lon = coords[0]
assert e.data["latitude"] == 43.7
assert e.data["longitude"] == -116.2
assert e.geo.centroid == (-116.2, 43.7)
await adapter.shutdown()
@pytest.mark.asyncio
async def test_lat_lon_path_writes_latlon_to_data(
self, temp_db_path, mock_config_store
):
"""lat_path/lon_path item: data["latitude"]/["longitude"] populated."""
config = make_config(
domain="fire",
extra_settings={
"items_path": "alerts",
"id_path": "uid",
"geometry_path": "",
"lat_path": "lat",
"lon_path": "lon",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
fixture = {
"alerts": [
{"uid": "enrich-latlon-1", "lat": 43.5, "lon": -116.1},
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = fixture
events = [e async for e in adapter.poll()]
assert len(events) == 1
e = events[0]
assert e.data["latitude"] == 43.5
assert e.data["longitude"] == -116.1
await adapter.shutdown()
@pytest.mark.asyncio
async def test_non_point_geometry_no_latlon_in_data(
self, temp_db_path, mock_config_store
):
"""Non-Point geometry (LineString/Polygon) has no representative point;
latitude/longitude must NOT appear in data degrades to region unknown."""
config = make_config(
domain="wx",
extra_settings={
"id_path": "id",
"geometry_path": "geometry",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
fixture = {
"features": [
{
"id": "enrich-linestring-1",
"geometry": {
"type": "LineString",
"coordinates": [[-116.0, 43.0], [-115.0, 44.0]],
},
}
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = fixture
events = [e async for e in adapter.poll()]
assert len(events) == 1
e = events[0]
assert "latitude" not in e.data
assert "longitude" not in e.data
await adapter.shutdown()
@pytest.mark.asyncio
async def test_coordless_item_no_latlon_in_data(
self, temp_db_path, mock_config_store
):
"""Item with no geometry and no lat/lon paths: latitude/longitude absent
from data will degrade to region unknown."""
config = make_config(
domain="wx",
extra_settings={
"id_path": "id",
"geometry_path": "geometry",
},
)
adapter = GenericHttpAdapter(config, mock_config_store, temp_db_path)
await adapter.startup()
fixture = {
"features": [
{"id": "enrich-coordless-1", "geometry": None},
]
}
with patch.object(adapter, "_fetch", new_callable=AsyncMock) as mf:
mf.return_value = fixture
events = [e async for e in adapter.poll()]
assert len(events) == 1
e = events[0]
assert "latitude" not in e.data
assert "longitude" not in e.data
await adapter.shutdown()

View file

@ -0,0 +1,585 @@
"""v0.15.0 PR3 — GUI create + delete for adapter instances.
Test strategy
* Pure-unit (always run, no DB):
- ADAPTER_NAME_REGEX validation
- Deletability rule (name in registry not deletable)
* Mock-DB (always run; mirrors test_gui_adapter_edit.py pattern):
- GET /adapters/new renders correctly
- POST create: valid INSERT, audit, 302
- POST create: duplicate name 409
- POST create: bad name format 422
- POST create: non-creatable kind 422
- POST create: invalid settings (missing required field) 422
- POST delete: operator instance DELETE, audit, 302
- POST delete: built-in adapter 403, no DELETE
DB-backed INSERT/DELETE tests (test_db_* below) use the central_test
Postgres fixture. They will raise ConnectionRefusedError when the test DB
is absent the same behaviour as other DB-backed tests in this suite (e.g.
test_config_store.py, test_supervisor_hotreload.py).
"""
import re
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from starlette.datastructures import FormData
from starlette.requests import Request
from central.gui import templates as gui_templates
from central.gui.routes import (
ADAPTER_NAME_REGEX,
adapters_create_form,
adapters_create_submit,
adapters_delete,
adapters_list,
)
from central.adapters.generic_http import GenericHttpAdapter
# ---------------------------------------------------------------------------
# Helpers shared across test classes
# ---------------------------------------------------------------------------
def _make_request(method="GET", form_pairs=None, csrf="x"):
"""Build a mock Request with CSRF + optional form data."""
req = MagicMock()
req.state.operator = SimpleNamespace(id=1, username="admin")
req.state.csrf_token = csrf
if form_pairs is not None:
pairs = [("csrf_token", csrf)] + list(form_pairs)
req.form = AsyncMock(return_value=FormData(pairs))
else:
req.form = AsyncMock(return_value=FormData([("csrf_token", csrf)]))
return req
def _make_pool(fetchrow_returns=None, fetchval_returns=None, fetch_returns=None):
"""Build a mock asyncpg pool.
Values are set unconditionally so that None (e.g. "row not found") is
returned faithfully instead of the default truthy AsyncMock sentinel.
Pass a list for fetchrow_returns to use side_effect for sequential calls.
"""
conn = AsyncMock()
if isinstance(fetchrow_returns, list):
conn.fetchrow.side_effect = fetchrow_returns
else:
conn.fetchrow.return_value = fetchrow_returns # None = not found
conn.fetchval.return_value = fetchval_returns # None = not found
conn.fetch.return_value = fetch_returns if fetch_returns is not None else []
pool = MagicMock()
pool.acquire.return_value.__aenter__ = AsyncMock(return_value=conn)
pool.acquire.return_value.__aexit__ = AsyncMock(return_value=None)
return pool, conn
# ---------------------------------------------------------------------------
# UNIT: name-regex validation
# ---------------------------------------------------------------------------
class TestAdapterNameRegex:
"""Pure-unit — no I/O, always run."""
VALID = [
"my_source",
"mysource2",
"aa", # minimum length (2 chars)
"a" + "b" * 63, # maximum length (64 chars)
"a1_b2_c3",
]
INVALID = [
"", # empty
"a", # too short (only 1 char)
"A_source", # uppercase
"1source", # starts with digit
"_source", # starts with underscore
"my-source", # hyphen not allowed
"my source", # space not allowed
"a" + "b" * 64, # 65 chars — too long
]
@pytest.mark.parametrize("name", VALID)
def test_valid(self, name):
assert ADAPTER_NAME_REGEX.match(name), f"Expected {name!r} to match"
@pytest.mark.parametrize("name", INVALID)
def test_invalid(self, name):
assert not ADAPTER_NAME_REGEX.match(name), f"Expected {name!r} not to match"
# ---------------------------------------------------------------------------
# UNIT: deletability rule
# ---------------------------------------------------------------------------
class TestDeletabilityRule:
"""Pure-unit — the rule is: name NOT IN adapter_classes → deletable.
Built-ins have name == kind (the class's .name attribute) which IS a key in
the adapter class registry. Operator instances have a unique name that is
NOT a registry key.
"""
def test_builtin_not_deletable(self):
from central.adapter_discovery import discover_adapters
classes = discover_adapters()
# Every registered kind key should be considered a built-in.
for kind in classes:
assert kind in classes, "sanity"
# The deletability check: name in adapter_classes → NOT deletable
assert kind in classes # confirms the guard fires
def test_operator_instance_is_deletable(self):
from central.adapter_discovery import discover_adapters
classes = discover_adapters()
operator_name = "my_custom_source_42"
assert operator_name not in classes, (
"Test assumes operator_name is not a registered kind; "
"update the name if a new kind was added with this identifier."
)
def test_generic_http_kind_is_not_deletable_by_name(self):
"""The KIND 'generic_http' itself should not be deletable (it's a built-in key)."""
from central.adapter_discovery import discover_adapters
classes = discover_adapters()
assert "generic_http" in classes
def test_generic_http_instance_is_deletable(self):
"""An operator instance named 'my_feed' (not a kind key) should be deletable."""
from central.adapter_discovery import discover_adapters
classes = discover_adapters()
assert "my_feed" not in classes
# ---------------------------------------------------------------------------
# UNIT: GenericHttpAdapter.operator_creatable
# ---------------------------------------------------------------------------
def test_generic_http_is_operator_creatable():
assert GenericHttpAdapter.operator_creatable is True
def test_base_class_default_not_creatable():
from central.adapter import SourceAdapter
assert SourceAdapter.operator_creatable is False
# ---------------------------------------------------------------------------
# Mock-DB: GET /adapters/new
# ---------------------------------------------------------------------------
class TestGetAdaptersNew:
@pytest.mark.asyncio
async def test_renders_200_with_generic_http_in_kind_select(self):
pool, conn = _make_pool(fetch_returns=[])
tmpl = MagicMock()
tmpl.TemplateResponse.return_value = MagicMock(status_code=200)
req = _make_request()
with patch("central.gui.routes._get_templates", return_value=tmpl), \
patch("central.gui.routes.get_pool", return_value=pool):
await adapters_create_form(req)
ctx = tmpl.TemplateResponse.call_args.kwargs["context"]
kind_names = [ck["kind"] for ck in ctx["creatable_kinds"]]
assert "generic_http" in kind_names
@pytest.mark.asyncio
async def test_fields_present_for_generic_http(self):
pool, conn = _make_pool(fetch_returns=[])
tmpl = MagicMock()
tmpl.TemplateResponse.return_value = MagicMock(status_code=200)
req = _make_request()
with patch("central.gui.routes._get_templates", return_value=tmpl), \
patch("central.gui.routes.get_pool", return_value=pool):
await adapters_create_form(req)
ctx = tmpl.TemplateResponse.call_args.kwargs["context"]
field_names = [f.name for f in ctx["fields"]]
# GenericHttpSettings requires url, domain, id_path at minimum
assert "url" in field_names
assert "domain" in field_names
assert "id_path" in field_names
@pytest.mark.asyncio
async def test_template_renders_without_errors(self):
"""Smoke test: the template itself renders without crashing."""
pool, conn = _make_pool(fetch_returns=[])
tmpl = MagicMock()
tmpl.TemplateResponse.return_value = MagicMock(status_code=200)
req = _make_request()
with patch("central.gui.routes._get_templates", return_value=tmpl), \
patch("central.gui.routes.get_pool", return_value=pool):
resp = await adapters_create_form(req)
# Template was called — no exception raised
assert tmpl.TemplateResponse.called
# ---------------------------------------------------------------------------
# Mock-DB: POST /adapters/new — happy path
# ---------------------------------------------------------------------------
def _valid_generic_http_pairs(name="my_feed"):
"""Minimal valid form pairs for a generic_http instance."""
return [
("kind", "generic_http"),
("name", name),
("cadence_s", "300"),
# enabled intentionally absent → ships disabled
("url", "https://example.com/feed.geojson"),
("domain", "fire"),
("id_path", "properties.id"),
]
class TestPostAdaptersNewHappyPath:
@pytest.mark.asyncio
async def test_valid_creates_and_redirects(self):
pool, conn = _make_pool(
fetchval_returns=None, # name does not exist yet
fetch_returns=[], # no api keys
)
inserted: list = []
async def cap_execute(q, *args):
if "INSERT INTO config.adapters" in q:
inserted.append(args)
conn.execute.side_effect = cap_execute
req = _make_request(form_pairs=_valid_generic_http_pairs())
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_create_submit(req)
assert resp.status_code == 302
assert "/adapters/my_feed" in resp.headers["location"]
assert len(inserted) == 1
# args: name, kind, enabled, cadence_s, settings
_name, _kind, _enabled, _cadence, _settings = inserted[0]
assert _name == "my_feed"
assert _kind == "generic_http"
assert _enabled is False # no 'enabled' in form → ships disabled
assert _cadence == 300
assert _settings["url"] == "https://example.com/feed.geojson"
@pytest.mark.asyncio
async def test_enabled_flag_set_when_checked(self):
pool, conn = _make_pool(fetchval_returns=None, fetch_returns=[])
inserted: list = []
async def cap(q, *args):
if "INSERT" in q:
inserted.append(args)
conn.execute.side_effect = cap
pairs = _valid_generic_http_pairs() + [("enabled", "on")]
req = _make_request(form_pairs=pairs)
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_create_submit(req)
assert resp.status_code == 302
_name, _kind, _enabled, *_ = inserted[0]
assert _enabled is True
@pytest.mark.asyncio
async def test_audit_record_written_on_create(self):
pool, conn = _make_pool(fetchval_returns=None, fetch_returns=[])
conn.execute.return_value = None
audited: list = []
async def cap_audit(conn_, action, **kw):
audited.append((action, kw))
req = _make_request(form_pairs=_valid_generic_http_pairs())
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", side_effect=cap_audit):
await adapters_create_submit(req)
assert len(audited) == 1
action, kw = audited[0]
assert action == "adapter.create"
assert kw["target"] == "my_feed"
# ---------------------------------------------------------------------------
# Mock-DB: POST /adapters/new — validation errors
# ---------------------------------------------------------------------------
async def _post_new(pairs, fetchval=None, fetch_returns=None):
"""Helper: POST /adapters/new and return (response, template_call_args).
fetchval=None means "adapter name does not exist" (duplicate check passes).
Pass fetchval=1 to simulate a duplicate.
"""
pool, conn = _make_pool(
fetchval_returns=fetchval, # None = not found; passed unconditionally
fetch_returns=fetch_returns or [],
)
tmpl = MagicMock()
tmpl.TemplateResponse.return_value = MagicMock()
req = _make_request(form_pairs=pairs)
with patch("central.gui.routes._get_templates", return_value=tmpl), \
patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_create_submit(req)
return resp, tmpl.TemplateResponse.call_args
class TestPostAdaptersNewValidationErrors:
@pytest.mark.asyncio
async def test_duplicate_name_returns_409(self):
pool, conn = _make_pool(fetchval_returns=1, fetch_returns=[])
req = _make_request(form_pairs=_valid_generic_http_pairs())
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_create_submit(req)
assert resp.status_code == 409
@pytest.mark.asyncio
async def test_bad_name_format_returns_422(self):
pairs = _valid_generic_http_pairs(name="Bad-Name!")
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
assert "name" in ca.kwargs["context"]["errors"]
@pytest.mark.asyncio
async def test_name_starts_with_digit_returns_422(self):
pairs = _valid_generic_http_pairs(name="1invalid")
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
@pytest.mark.asyncio
async def test_name_too_short_returns_422(self):
pairs = _valid_generic_http_pairs(name="a") # only 1 char
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
@pytest.mark.asyncio
async def test_name_shadows_kind_returns_422(self):
"""Cannot use a registered kind name as an instance name."""
pairs = _valid_generic_http_pairs(name="generic_http")
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
assert "name" in ca.kwargs["context"]["errors"]
@pytest.mark.asyncio
async def test_non_creatable_kind_returns_422(self):
# usgs_quake is a real built-in kind that is NOT operator_creatable
pairs = [
("kind", "usgs_quake"),
("name", "my_quake"),
("cadence_s", "300"),
("url", "https://example.com"),
("domain", "quake"),
("id_path", "id"),
]
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
assert "kind" in ca.kwargs["context"]["errors"]
@pytest.mark.asyncio
async def test_invalid_settings_missing_required_field_returns_422(self):
# Omit required 'url' field from generic_http settings
pairs = [
("kind", "generic_http"),
("name", "my_feed"),
("cadence_s", "300"),
# no 'url', no 'domain', no 'id_path'
]
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
@pytest.mark.asyncio
async def test_cadence_below_10_returns_422(self):
pairs = _valid_generic_http_pairs()
# replace cadence_s
pairs = [(k, "5") if k == "cadence_s" else (k, v) for k, v in pairs]
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
assert "cadence_s" in ca.kwargs["context"]["errors"]
@pytest.mark.asyncio
async def test_invalid_domain_returns_422(self):
pairs = _valid_generic_http_pairs()
pairs = [(k, "notadomain") if k == "domain" else (k, v) for k, v in pairs]
resp, ca = await _post_new(pairs)
assert ca.kwargs["status_code"] == 422
# ---------------------------------------------------------------------------
# Mock-DB: POST /adapters/{name}/delete
# ---------------------------------------------------------------------------
class TestPostAdaptersDelete:
@pytest.mark.asyncio
async def test_operator_instance_is_deleted(self):
"""Deleting an operator instance removes the row and audits."""
pool, conn = _make_pool(
fetchrow_returns={"name": "my_feed", "kind": "generic_http"},
)
deleted: list = []
async def cap(q, *args):
if "DELETE FROM config.adapters" in q:
deleted.append(args)
conn.execute.side_effect = cap
req = _make_request()
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_delete(req, "my_feed")
assert resp.status_code == 302
assert resp.headers["location"] == "/adapters"
assert len(deleted) == 1
assert deleted[0][0] == "my_feed"
@pytest.mark.asyncio
async def test_builtin_adapter_returns_403(self):
"""Attempting to delete a built-in adapter (name in registry) → 403."""
pool, conn = _make_pool(
fetchrow_returns={"name": "usgs_quake", "kind": "usgs_quake"},
)
req = _make_request()
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_delete(req, "usgs_quake")
assert resp.status_code == 403
assert "built-in" in resp.body.decode()
# DELETE must NOT have been called
for call in conn.execute.call_args_list:
assert "DELETE" not in str(call)
@pytest.mark.asyncio
async def test_missing_adapter_returns_404(self):
pool, conn = _make_pool(fetchrow_returns=None)
req = _make_request()
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_delete(req, "nonexistent")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_delete_audit_record_written(self):
pool, conn = _make_pool(
fetchrow_returns={"name": "my_feed", "kind": "generic_http"},
)
conn.execute.return_value = None
audited: list = []
async def cap_audit(conn_, action, **kw):
audited.append((action, kw))
req = _make_request()
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", side_effect=cap_audit):
await adapters_delete(req, "my_feed")
assert len(audited) == 1
action, kw = audited[0]
assert action == "adapter.delete"
assert kw["target"] == "my_feed"
@pytest.mark.asyncio
async def test_generic_http_kind_itself_is_protected(self):
"""The class entry 'generic_http' IS in the registry → 403."""
pool, conn = _make_pool(
fetchrow_returns={"name": "generic_http", "kind": "generic_http"},
)
req = _make_request()
with patch("central.gui.routes.get_pool", return_value=pool), \
patch("central.gui.routes.write_audit", new=AsyncMock()):
resp = await adapters_delete(req, "generic_http")
assert resp.status_code == 403
# ---------------------------------------------------------------------------
# Template smoke test: adapters_new.html renders without crashing
# ---------------------------------------------------------------------------
class TestAdaptersNewTemplate:
def _render(self, ctx):
req = Request({
"type": "http", "method": "GET", "path": "/",
"headers": [], "query_string": b"",
})
return gui_templates.TemplateResponse(
request=req, name="adapters_new.html", context=ctx
).body.decode()
def _ctx(self, errors=None, form_data=None):
from central.gui.form_descriptors import describe_fields
from central.adapters.generic_http import GenericHttpSettings
fields = describe_fields(GenericHttpSettings, {})
return {
"operator": SimpleNamespace(username="admin"),
"csrf_token": "x",
"creatable_kinds": [{"kind": "generic_http", "display_name": "Generic HTTP Source"}],
"selected_kind": "generic_http",
"default_cadence_s": 300,
"fields": fields,
"api_keys": [],
"errors": errors,
"form_data": form_data,
}
def test_renders_kind_select(self):
out = self._render(self._ctx())
assert "generic_http" in out
assert 'name="kind"' in out
def test_renders_name_input(self):
out = self._render(self._ctx())
assert 'name="name"' in out
def test_renders_cadence_input_with_default(self):
out = self._render(self._ctx())
assert 'name="cadence_s"' in out
assert "300" in out
def test_renders_url_field_for_generic_http(self):
out = self._render(self._ctx())
assert 'name="url"' in out
def test_enabled_unchecked_by_default(self):
out = self._render(self._ctx())
# The enabled checkbox must not be checked in default render
# (spec: ships disabled)
assert 'name="enabled"' in out
# Extract the enabled checkbox line and confirm no 'checked' attribute
for line in out.splitlines():
if 'name="enabled"' in line:
assert "checked" not in line, f"enabled checkbox should be unchecked by default: {line}"
break
def test_error_messages_displayed(self):
errors = {"name": "Name is invalid", "url": "URL is required"}
out = self._render(self._ctx(errors=errors))
assert "Name is invalid" in out
assert "URL is required" in out
def test_form_data_restores_values(self):
form_data = {"kind": "generic_http", "name": "restored_name",
"cadence_s": "600", "url": "https://example.com/data.json",
"domain": "fire", "id_path": "id", "enabled": False}
out = self._render(self._ctx(form_data=form_data))
assert "restored_name" in out
assert "https://example.com/data.json" in out

View file

@ -116,7 +116,7 @@ def _post_req(pairs, cadence_s="1800"):
def _pool(settings=_SETTINGS):
conn = AsyncMock()
conn.fetchrow.side_effect = [
{"name": "tomtom_incidents", "enabled": True, "cadence_s": 1800, "settings": settings,
{"name": "tomtom_incidents", "kind": "tomtom_incidents", "enabled": True, "cadence_s": 1800, "settings": settings,
"paused_at": None, "updated_at": None, "last_error": None},
{"map_tile_url": None, "map_attribution": None},
]

View file

@ -0,0 +1,48 @@
"""v0.15.0 migration 043: add kind column to config.adapters.
No live Postgres required asserts the migration SQL is structurally correct:
adds the column idempotently, back-fills kind = name for pre-existing rows, and
enforces NOT NULL going forward.
"""
from pathlib import Path
_SQL = Path("sql/migrations/043_add_adapters_kind_column.sql").read_text()
_NORM = " ".join(_SQL.split()) # whitespace-insensitive matching
def test_uses_add_column_if_not_exists():
"""Must be idempotent — safe to re-run after partial application."""
assert "ADD COLUMN IF NOT EXISTS kind TEXT" in _NORM
def test_backfills_kind_equals_name_for_null_rows():
"""Existing rows must get kind = name so built-in adapters keep working."""
assert "UPDATE config.adapters SET kind = name WHERE kind IS NULL" in _NORM
def test_enforces_not_null_after_backfill():
"""kind must be NOT NULL once every row is back-filled."""
assert "ALTER COLUMN kind SET NOT NULL" in _NORM
def test_does_not_add_default_clause():
"""No DEFAULT — future INSERT paths must always supply kind explicitly."""
# The ADD COLUMN line should not contain DEFAULT
add_line = next(
(line for line in _SQL.splitlines() if "ADD COLUMN" in line), ""
)
assert "DEFAULT" not in add_line.upper()
def test_does_not_drop_any_columns():
"""Pure additive migration — must not drop anything."""
upper = _NORM.upper()
assert "DROP COLUMN" not in upper
assert "DROP TABLE" not in upper
def test_explains_name_vs_kind_split_in_comment():
"""Header comment must explain the instance-vs-class identity split."""
assert "instance" in _SQL.lower()
assert "class" in _SQL.lower()

View file

@ -27,6 +27,7 @@ class TestRegionPickerInTemplate:
mock_conn.fetchrow.side_effect = [
{ # Adapter row
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {
@ -94,6 +95,7 @@ class TestRegionValidation:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"], "region": {"north": 49.5, "south": 31.0, "east": -102.0, "west": -124.5}},
@ -153,6 +155,7 @@ class TestRegionValidation:
mock_conn.fetchrow.side_effect = [
{
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"], "region": {"north": 49.5, "south": 31.0, "east": -102.0, "west": -124.5}},
@ -209,6 +212,7 @@ class TestRegionValidation:
mock_conn.fetchrow.side_effect = [
{
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"], "region": {"north": 49.5, "south": 31.0, "east": -102.0, "west": -124.5}},
@ -265,6 +269,7 @@ class TestRegionValidation:
mock_conn.fetchrow.side_effect = [
{
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"], "region": {"north": 49.5, "south": 31.0, "east": -102.0, "west": -124.5}},
@ -323,6 +328,7 @@ class TestRegionAuditLog:
mock_conn = AsyncMock()
mock_conn.fetchrow.return_value = {
"name": "firms",
"kind": "firms",
"enabled": True,
"cadence_s": 300,
"settings": {

View file

@ -74,7 +74,7 @@ class TestRoutesApiKeyMissing:
mock_pool = MagicMock()
mock_conn = MagicMock()
mock_conn.fetch = AsyncMock(return_value=[
{"name": "firms", "enabled": False, "cadence_s": 300, "settings": {}, "paused_at": None, "updated_at": None, "last_error": None},
{"name": "firms", "kind": "firms", "enabled": False, "cadence_s": 300, "settings": {}, "paused_at": None, "updated_at": None, "last_error": None},
])
mock_conn.fetchval = AsyncMock(return_value=None) # No API key exists
mock_conn.__aenter__ = AsyncMock(return_value=mock_conn)
@ -307,6 +307,7 @@ class TestAdaptersEditSubmitErrorRerender:
# First call: adapter row
{
"name": "firms",
"kind": "firms",
"enabled": False,
"cadence_s": 300,
"settings": {"api_key_alias": "firms", "satellites": ["VIIRS_SNPP_NRT"]},