feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 16:10:44 +00:00
|
|
|
"""Tests for NaviBackend (composed Navi /api/reverse endpoint).
|
|
|
|
|
|
|
|
|
|
HTTP is exercised via patching the backend's `_fetch` (the codebase has no
|
|
|
|
|
aioresponses/respx dep); URL construction is asserted on the pure `_url`
|
|
|
|
|
helper. An env-gated integration smoke against the live Navi endpoint is
|
|
|
|
|
skipped by default.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import os
|
|
|
|
|
from unittest.mock import AsyncMock
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from central.enrichment.backends.navi import NaviBackend
|
|
|
|
|
from central.enrichment.geocoder import GEOCODER_FIELDS, all_null_bundle
|
|
|
|
|
|
|
|
|
|
# Full Navi response — already canonical shape.
|
|
|
|
|
_NAVI_OK = {
|
|
|
|
|
"name": "Where you are",
|
|
|
|
|
"city": "Boise",
|
|
|
|
|
"county": "Ada",
|
|
|
|
|
"state": "Idaho",
|
|
|
|
|
"country": "United States",
|
|
|
|
|
"postal_code": "83702",
|
|
|
|
|
"timezone": "America/Boise",
|
|
|
|
|
"landclass": "Public — National Forest",
|
|
|
|
|
"elevation_m": 824,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _backend() -> NaviBackend:
|
|
|
|
|
# warmup=False so construction issues no background task in tests.
|
|
|
|
|
return NaviBackend(base_url="http://navi.test:8440", warmup=False)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_url_construction():
|
|
|
|
|
b = _backend()
|
|
|
|
|
assert b._url(43.615, -116.2023) == "http://navi.test:8440/api/reverse/43.615/-116.2023"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_base_url_trailing_slash_stripped():
|
|
|
|
|
b = NaviBackend(base_url="http://navi.test:8440/", warmup=False)
|
|
|
|
|
assert b._url(1.0, 2.0) == "http://navi.test:8440/api/reverse/1.0/2.0"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_happy_path_passthrough():
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(return_value=dict(_NAVI_OK))
|
|
|
|
|
result = await b.reverse(43.615, -116.2023)
|
|
|
|
|
assert result == _NAVI_OK
|
|
|
|
|
assert set(result.keys()) == set(GEOCODER_FIELDS)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_partial_nulls_preserved():
|
|
|
|
|
"""Navi 200-with-nulls (non-US: timezone + elevation, rest null)."""
|
|
|
|
|
partial = {**all_null_bundle(), "timezone": "Europe/Paris", "elevation_m": 35}
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(return_value=partial)
|
|
|
|
|
result = await b.reverse(48.85, 2.35)
|
|
|
|
|
assert result["timezone"] == "Europe/Paris"
|
|
|
|
|
assert result["elevation_m"] == 35
|
|
|
|
|
assert result["city"] is None
|
|
|
|
|
assert set(result.keys()) == set(GEOCODER_FIELDS)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_extra_keys_dropped():
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(return_value={**_NAVI_OK, "debug_internal": "leak"})
|
|
|
|
|
result = await b.reverse(1.0, 2.0)
|
|
|
|
|
assert "debug_internal" not in result
|
|
|
|
|
assert set(result.keys()) == set(GEOCODER_FIELDS)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_network_error_returns_all_null_never_raises():
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(side_effect=ConnectionError("boom"))
|
|
|
|
|
result = await b.reverse(1.0, 2.0)
|
|
|
|
|
assert result == all_null_bundle()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_timeout_returns_all_null():
|
|
|
|
|
import asyncio
|
|
|
|
|
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(side_effect=asyncio.TimeoutError())
|
|
|
|
|
assert await b.reverse(1.0, 2.0) == all_null_bundle()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_malformed_response_returns_all_null():
|
|
|
|
|
b = _backend()
|
|
|
|
|
b._fetch = AsyncMock(side_effect=ValueError("not json"))
|
|
|
|
|
assert await b.reverse(1.0, 2.0) == all_null_bundle()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_headers_passed_through_config():
|
|
|
|
|
b = NaviBackend(base_url="http://navi.test", headers={"Authorization": "Bearer x"}, warmup=False)
|
|
|
|
|
assert b._headers == {"Authorization": "Bearer x"}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
@pytest.mark.skipif(
|
|
|
|
|
os.environ.get("NAVI_INTEGRATION_TEST") != "1",
|
|
|
|
|
reason="set NAVI_INTEGRATION_TEST=1 to hit the live Navi endpoint",
|
|
|
|
|
)
|
|
|
|
|
async def test_live_navi_boise():
|
feat(3-K.5): operator-settable EnrichmentConfig (config plumbing)
Bridge PR for v0.5.0. PR J wired the supervisor with a hardcoded
EnrichmentConfig() default; PR K added real backends to the registry but
left no operator path to select one. K.5 closes that gap by mirroring the
config.adapters storage + LISTEN/NOTIFY hot-reload pattern.
config.enrichment (migration 024): single-row table (id BOOLEAN PK CHECK
(id = true), mirroring config.system). Columns enricher_class, backend_class,
backend_settings JSONB, cache_ttl_s, updated_at. Reuses the existing
config.set_updated_at + config.notify_config_change triggers (the NOTIFY
function's ELSE branch emits 'enrichment:' for this keyless single-row table).
Seeds framework DEFAULTS ONLY — GeocoderEnricher + NoOpBackend, empty
backend_settings, 24h TTL. NO URLs/IPs/auth in the seed; a fresh deploy runs
NoOp out of the box. Idempotent (CREATE IF NOT EXISTS / DROP TRIGGER IF
EXISTS / INSERT ON CONFLICT DO NOTHING).
Supervisor:
- Reads config.enrichment at startup (start() -> config_source
.get_enrichment_config()), overriding the constructor default.
- Hot-reloads via _on_config_change(table == "enrichment"): re-reads the row,
rebuilds the enricher set, and invalidates the enrichment cache when the
enricher/backend/settings changed (a new backend must not keep serving the
old backend's cached bundles until TTL). TTL-only changes retain the cache.
- build_enrichers now takes an explicit EnrichmentCache (the supervisor owns
it so it can invalidate); cache no longer built inside build_enrichers.
ConfigStore / ConfigSource: get_enrichment_config() (falls back to defaults if
the row is somehow absent) + upsert_enrichment_config(). Mirrors the adapter
accessors.
cache.py: EnrichmentCache.invalidate(enricher_name=None) — DELETE all or
enricher-scoped; returns rows deleted.
GUI /enrichment: GET renders the EnrichmentConfig form via the generic
describe_fields machinery (no enrichment-specific Jinja); POST validates via
Pydantic, writes config.enrichment, and lets the NOTIFY trigger propagate the
hot-reload. New enrichment.html + a nav link. backend_settings (a dict field)
needed a generic "json" widget in describe_fields + the template — usable by
any dict-typed settings field, not enrichment-specific.
Necessary deviation (surfaced): PR K shipped a deployment-specific default
DEFAULT_BASE_URL = "http://192.168.1.130:8440" in navi.py. Bar (b) forbids
deployer IPs in src, and operator-settable base_url is exactly K.5's purpose,
so the default is changed to http://localhost:8440 (matching Photon/Nominatim
defaults). The live integration smoke (tests/, env-gated, skipped) now reads
the endpoint from NAVI_BASE_URL — no IP anywhere in src.
Tests (test_enrichment_config_plumbing.py, 10): ConfigStore read / default
fallback / upsert-passes-dict; cache invalidate all + scoped; supervisor builds
NaviBackend from config; hot-reload rebuilds + invalidates on backend change;
no-invalidate on TTL-only change; describe_fields json widget; /enrichment GET
render. test_firms updated for the build_enrichers signature change.
Hot-reload mechanism mirrored: Postgres LISTEN/NOTIFY on channel
'config_changed' (payload 'table:key'), same path adapters/streams use; the
supervisor's existing _on_config_change dispatch gains an "enrichment" branch.
Verification: full pytest 535 passed, 1 skipped (was 525; +10). Migration
applied cleanly on the live prod schema; SELECT * FROM config.enrichment
returns the NoOp default row. grep subject_for_event/_ADAPTER_REGISTRY and
grep 100.64.0./192.168.1. in src both empty.
Does NOT activate NaviBackend (ships NoOp default; operator action) and does
NOT declare enrichment_locations on other adapters (PR L scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:52:22 +00:00
|
|
|
"""Integration smoke against the real endpoint (default skipped).
|
|
|
|
|
|
|
|
|
|
The endpoint host is supplied via NAVI_BASE_URL so no deployment-specific
|
|
|
|
|
address lives in source; defaults to localhost when unset.
|
|
|
|
|
"""
|
|
|
|
|
base_url = os.environ.get("NAVI_BASE_URL", "http://localhost:8440")
|
|
|
|
|
b = NaviBackend(base_url=base_url, warmup=False)
|
feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 16:10:44 +00:00
|
|
|
result = await b.reverse(43.6150, -116.2023)
|
|
|
|
|
assert result["name"] == "Where you are"
|
|
|
|
|
assert result["city"] == "Boise"
|
|
|
|
|
assert result["state"] == "Idaho"
|
|
|
|
|
assert result["elevation_m"] is not None
|
|
|
|
|
assert abs(float(result["elevation_m"]) - 824) < 50
|