mirror of
https://github.com/zvx-echo6/central.git
synced 2026-05-21 18:14:44 +02:00
malice
d92074b134Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
|
||
|---|---|---|
| .. | ||
| fixtures | ||
| __init__.py | ||
| conftest.py | ||
| README.md | ||
| test_adapters.py | ||
| test_api_keys.py | ||
| test_archive_multi_stream.py | ||
| test_audit.py | ||
| test_auth.py | ||
| test_bootstrap_config.py | ||
| test_config_source.py | ||
| test_config_store.py | ||
| test_consumer_doc.py | ||
| test_crypto.py | ||
| test_csrf_handler.py | ||
| test_csrf_race_condition.py | ||
| test_dashboard.py | ||
| test_eonet.py | ||
| test_events_adapter_column.py | ||
| test_events_feed.py | ||
| test_events_feed_frontend.py | ||
| test_firms.py | ||
| test_form_descriptors.py | ||
| test_gdacs.py | ||
| test_gui_scaffold.py | ||
| test_inciweb.py | ||
| test_models.py | ||
| test_nwis.py | ||
| test_nws_normalization.py | ||
| test_preview_hook.py | ||
| test_region_picker.py | ||
| test_requires_api_key.py | ||
| test_session_auth.py | ||
| test_setup_gate.py | ||
| test_stream_registry.py | ||
| test_streams.py | ||
| test_supervisor_hotreload.py | ||
| test_supervisor_integration.py | ||
| test_swpc.py | ||
| test_usgs_quake.py | ||
| test_wfigs.py | ||
| test_wizard.py | ||
Central Tests
Test Database
Some tests (notably test_config_store.py) require a real PostgreSQL database.
By default, tests connect to:
postgresql://central_test:testpass@localhost/central_test
If your test database uses different credentials, set the CENTRAL_TEST_DB_DSN
environment variable:
export CENTRAL_TEST_DB_DSN="postgresql://myuser:mypass@localhost/mydb"
uv run pytest tests/test_config_store.py