matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-05-21 18:14:44 +02:00
Code Issues Projects Releases Packages Wiki Activity
Central - data hub spine. Adapters -> NATS/JetStream -> archive.
146 commits 2 branches 11 tags 1.7 MiB
  • Python 90.5%
  • HTML 9.1%
  • PLpgSQL 0.4%
Find a file
Matt Johnson 6afe80ded3 docs(2-I): producer integration spec — docs/PRODUCER-INTEGRATION.md 
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.

Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.

12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.

Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
  - bidirectional == between SourceAdapter API and §4 method coverage
  - preview_for_settings contract verbatim against live docstring
  - top-level domain enumeration vs central.streams.STREAMS prefixes
  - §8 STREAMS snippet vs central.streams.STREAMS
  - anti-patterns adapter-name examples vs discover_adapters()

No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.

Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).

Acceptance bars:
  (a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
  (b) bidirectional override-method coverage asserted in test
  (c) tests/test_producer_doc.py → 6/6 pass
  (d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
  (e) doc length: 823 lines (within 500–1200 envelope)
  (f) code fences balanced; JSON/Python blocks parse

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 21:17:48 +00:00
docs docs(2-I): producer integration spec — docs/PRODUCER-INTEGRATION.md 2026-05-19 21:17:48 +00:00
etc-templates scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
scripts scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
sql feat(2-G): USGS NWIS adapter (OGC API) + CENTRAL_HYDRO stream 2026-05-19 16:50:21 +00:00
src/central fix(2-G.5): preview_for_settings contract in adapter docstring + distinguish [] from None 2026-05-19 17:55:39 +00:00
systemd feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
tests docs(2-I): producer integration spec — docs/PRODUCER-INTEGRATION.md 2026-05-19 21:17:48 +00:00
.gitattributes chore: normalize line endings to LF 2026-05-16 22:26:12 +00:00
.gitignore feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
.python-version foundation: models, adapter ABC, config, CE wire, schema 2026-05-15 21:08:56 +00:00
CHANGELOG.md docs: add v0.3.0 changelog entry and network bindings reference (#29) 2026-05-18 14:26:09 -06:00
LICENSE scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
pyproject.toml release: bump version to 0.3.0 (#30) 2026-05-18 14:29:28 -06:00
README.md docs: add test database setup, restore geom to test fixture 2026-05-17 18:26:48 +00:00
uv.lock feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00

README.md

Central

Central is the data hub spine for the infrastructure. Adapters normalize upstream sources into a canonical event shape, publish CloudEvents to NATS/JetStream, and archive to TimescaleDB for historical query. Single-LXC deployment.

Status

Phase 0 — scaffold. Not yet operational.

Architecture

  • Python 3.12 (uv-managed)
  • NATS + JetStream for live event bus
  • TimescaleDB + PostGIS for archive and geospatial query
  • One supervisor process managing adapter lifecycle
  • One archive consumer process persisting events to TimescaleDB
  • Both processes systemd-managed

Testing

See docs/test-database.md for test database setup.

License

MIT. See LICENSE.