The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.
Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.
12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.
Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
- bidirectional == between SourceAdapter API and §4 method coverage
- preview_for_settings contract verbatim against live docstring
- top-level domain enumeration vs central.streams.STREAMS prefixes
- §8 STREAMS snippet vs central.streams.STREAMS
- anti-patterns adapter-name examples vs discover_adapters()
No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.
Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).
Acceptance bars:
(a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
(b) bidirectional override-method coverage asserted in test
(c) tests/test_producer_doc.py → 6/6 pass
(d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
(e) doc length: 823 lines (within 500–1200 envelope)
(f) code fences balanced; JSON/Python blocks parse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the consumer contract for Central's NATS event streams. Primary reader:
a Claude Code instance building MeshAI's ingestion layer. The doc IS the spec --
no "see source for details".
Opens with Matt's framing: "Central takes it all and gives it all. It's up to
the pipe to do with it what it will." Central is a faithful firehose --
adapters preserve every upstream field with no enrichment / formatting /
opinionated translation. The CloudEvents envelope adds routing + dedup support;
everything else is upstream-shaped. Where the doc lists upstream lookup
endpoints for ID-only fields, that is consumer-side convenience -- explicitly
NOT a recommendation that Central enrich.
Sections (11 total):
1. Quick start (5-line nats-py subscribe-and-print)
2. Connection details (URL / auth / JetStream context / stream discovery)
3. Stream layout (7 streams, derived from streams.py registry)
4. Subject namespace registry (Mermaid tree + full pattern table)
5. Wire format (5a CloudEvents envelope; 5b inner Event payload)
-- explicit callout that geo.centroid is [lon, lat] GeoJSON, NOT [lat, lon]
6. Per-adapter reference (12 subsections, locked template)
7. Fall-off / removal semantics (explicit subjects vs absence-as-signal)
8. Consumer patterns (durable vs ephemeral, ack/nack/term, worked example)
9. Dedup implementation guide (single-token vs composite-key adapters)
10. Writing a new consumer checklist
11. Troubleshooting
Doc length: 1878 lines (target was 600-1000 originally; revised to 1200-1800
once full-fidelity JSON examples + inciweb 3x narratives + wfigs_perimeters
polygon were folded in). Completeness wins per the design principle.
Every JSON example is verbatim from CT104. 11 examples sourced from
/tmp/nwis-build/evidence.txt (dumped via psql jsonb_pretty); the wfigs_perimeters
example is a freshly pulled smallest-active-polygon record so the doc captures
the live polygon shape without flooding the page with thousands of coordinate
pairs.
The doc is assembled by /tmp/nwis-build/build_doc.py which splices live JSON
blocks into a markdown template. The build script is local-only (not committed)
because the doc itself is the artifact; future updates regenerate by re-pulling
live evidence and re-running the assembler.
New test: tests/test_consumer_doc.py (5 tests). Parses the doc and asserts:
- The "Stream layout" table matches central.streams.STREAMS exactly
(stream names + subject filters).
- The (name, subject_filter) pairs match the registry as pairs (catches
swapped subject filters on existing streams).
- Every adapter discovered via central.adapter_discovery.discover_adapters()
has a per-adapter subsection -- and vice versa.
- The subsection count equals the registry size (catches duplicates).
Verification:
- 463/463 full suite green (was 458; +5 new consumer_doc tests).
- Doc structure: 1 H1, 12 H2, 33 H3, 12 per-adapter sections, 1 mermaid block,
12 JSON blocks (all parse).
- All 12 adapters covered.
- No regressions elsewhere.
Acceptance bars (a)-(e) verbatim:
(a) grep "subject_for_event|_ADAPTER_REGISTRY" -> empty
(b) all 12 adapters have per-adapter subsections
(c) 5/5 consumer-doc tests pass
(d) 463/463 full suite
(e) doc length 1878 lines
markdownlint was not available on CT104; substituted an inline Python sanity
check confirming code-fence balance, JSON-block validity, and structural
integrity (12 H2 / 33 H3 / 1 mermaid).
Co-authored-by: zvx <zvx@central>
- Add docs/test-database.md with one-time setup, DSN convention, reset
instructions, and explanation of why PostGIS is not in migrations
- Update docs/migrations.md with "Extensions are not in migrations"
section explaining superuser requirement
- Restore geom GEOMETRY(Geometry, 4326) column to test fixture now that
central_test has PostGIS installed
- Add CREATE EXTENSION IF NOT EXISTS postgis to test fixture for
self-bootstrap (central_test is superuser)
- Add Testing section to README.md pointing to docs/test-database.md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace pytest.skip stubs with actual DB tests against central_test
- Test backfill for all three adapters (nws, firms, usgs_quake)
- Test FK RESTRICT, NOT NULL, and FK validation constraints
- Test schema changes (source dropped, adapter exists with constraints)
- Delete stale sql/schema.sql (migrations are sole source of truth)
- Update docs/migrations.md with schema.sql removal note
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add migrations 007-010 for system config, operators, sessions, audit_log
- Implement argon2id password hashing via argon2-cffi
- Implement session-based authentication with database-stored tokens
- Add SetupGateMiddleware to redirect to /setup until first operator created
- Add SessionMiddleware to load session from cookie and attach operator
- Create /setup, /login, /logout, /change-password routes with CSRF protection
- Add periodic session cleanup task (hourly)
- Add audit logging for auth events
- Update systemd unit with EnvironmentFile for /etc/central/central.env
- Add comprehensive tests for auth, middleware, and audit modules
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Stream retention GUI design
- Region picker for bbox selection
- API key management requirements
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents production verification of the AsyncLimiter removal fix:
- Decrease 60-30s: poll at Tlast+30s (not 60s)
- Increase 30-60s: poll at Tlast+60s
- Decrease 60-15s: immediate poll (deadline passed)
- All subsequent intervals use new cadence
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
environment.md:
- Documents CT104 as the active development location
- Lists SSH access, repository paths, and service commands
- Notes that cortex clone is parked, matt-desktop has no clones
BUG-CADENCE-DECREASE.md:
- Full investigation of the cadence-decrease hot-reload bug
- Root cause analysis: cancel_event.set() inside lock context
- Proposed fix (Option A - structural)
- Test gap identification
- Production verification steps
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase B operational cutover verification:
- Config source cutover from TOML to DB confirmed
- Hot-reload cadence test passed (rate-limit guarantee)
- Enable/disable cycle test passed (preserved_last_poll)
- 10-minute soak with zero errors
- Data integrity verified (all alerts in DB)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
