Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).
Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
+ landclass + elevation in one call). Near-passthrough: response already
matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
construction when a loop is running; config `headers` slot for a future
Authorization: Bearer (config-only, no code change). Default base_url
http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
Maps features[0].properties; postal_code <- postcode; timezone/landclass/
elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
required User-Agent. Maps the address block; landclass/elevation_m/timezone
null.
Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).
Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
Central is the consumer's only data plane (consumers can't do follow-up
lookups), so enrich deliberately and centrally, namespaced under _enriched,
failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
fields), per-region coverage (US-full, non-US timezone+elevation), and the
antimeridian landclass caveat.
Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
handling, extra-key drop, network/timeout/non-200/malformed -> all-null
(never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
§13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
(derived from code, not hardcoded).
Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.
Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
- Add docs/test-database.md with one-time setup, DSN convention, reset
instructions, and explanation of why PostGIS is not in migrations
- Update docs/migrations.md with "Extensions are not in migrations"
section explaining superuser requirement
- Restore geom GEOMETRY(Geometry, 4326) column to test fixture now that
central_test has PostGIS installed
- Add CREATE EXTENSION IF NOT EXISTS postgis to test fixture for
self-bootstrap (central_test is superuser)
- Add Testing section to README.md pointing to docs/test-database.md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace pytest.skip stubs with actual DB tests against central_test
- Test backfill for all three adapters (nws, firms, usgs_quake)
- Test FK RESTRICT, NOT NULL, and FK validation constraints
- Test schema changes (source dropped, adapter exists with constraints)
- Delete stale sql/schema.sql (migrations are sole source of truth)
- Update docs/migrations.md with schema.sql removal note
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Stream retention GUI design
- Region picker for bbox selection
- API key management requirements
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents production verification of the AsyncLimiter removal fix:
- Decrease 60-30s: poll at Tlast+30s (not 60s)
- Increase 30-60s: poll at Tlast+60s
- Decrease 60-15s: immediate poll (deadline passed)
- All subsequent intervals use new cadence
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
environment.md:
- Documents CT104 as the active development location
- Lists SSH access, repository paths, and service commands
- Notes that cortex clone is parked, matt-desktop has no clones
BUG-CADENCE-DECREASE.md:
- Full investigation of the cadence-decrease hot-reload bug
- Root cause analysis: cancel_event.set() inside lock context
- Proposed fix (Option A - structural)
- Test gap identification
- Production verification steps
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase B operational cutover verification:
- Config source cutover from TOML to DB confirmed
- Hot-reload cadence test passed (rate-limit guarantee)
- Enable/disable cycle test passed (preserved_last_poll)
- 10-minute soak with zero errors
- Data integrity verified (all alerts in DB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
