matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-06-10 11:54:37 +02:00
Code Issues Projects Releases Packages Wiki Activity
Central - data hub spine. Adapters -> NATS/JetStream -> archive.
245 commits 8 branches 67 tags 4.7 MiB
  • Python 89.1%
  • HTML 9.3%
  • CSS 1.3%
  • PLpgSQL 0.3%
Find a file
malice b17d8bcd54
 v0.10.5.2: fix BY_START_TIME feedback loop in Re-send (snapshot last_seq boundary) (#93) 
The v0.10.5 ephemeral pull-consumer used DeliverPolicy.BY_START_TIME with
no upper bound, so every republish satisfied the same time filter and
the consumer fetched its own output back -- an unbounded loop bounded
only by the per-stream cap. Operator-triggered 5-minute resend on
2026-06-07 ran the loop long enough to time out central-gui's POST and
the host went down with it.

Fix: snapshot each event-bearing stream's last_seq up front via a new
_snapshot_last_seqs() helper, pass it to _iter_window as max_stream_seq,
and exit the generator the first time msg.metadata.sequence.stream
exceeds the snapshot. Pull-consumer delivery is stream-seq ascending so
one boundary check suffices.

Also drop _MAX_MSGS_PER_STREAM 50_000 -> 5_000 and add a WARNING log
when the cap is hit -- a legitimate operator window should never reach
it, and silent truncation hid the v0.10.5 loop until the host fell over.

Two regression tests cover the new behavior: one stages pre/post-snapshot
batches and asserts the post-snapshot batch is never yielded; one
overwhelms the cap and asserts the warning fires.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 22:36:04 -06:00
docs v0.10.3: rip out state_511_atis adapter (superseded by itd_511 v0.10.0; Castle Rock legacy shape EOL per sister-site discovery) (#88) 2026-06-06 14:44:00 -06:00
etc-templates scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
scripts scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
sql v0.10.3.1: soft-disable state_511_atis* adapters instead of DELETE (FK blocked v0.10.3 migration) (#90) 2026-06-06 18:39:33 -06:00
src/central v0.10.5.2: fix BY_START_TIME feedback loop in Re-send (snapshot last_seq boundary) (#93) 2026-06-06 22:36:04 -06:00
systemd feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
tests v0.10.5.2: fix BY_START_TIME feedback loop in Re-send (snapshot last_seq boundary) (#93) 2026-06-06 22:36:04 -06:00
.gitattributes chore: normalize line endings to LF 2026-05-16 22:26:12 +00:00
.gitignore feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
.python-version foundation: models, adapter ABC, config, CE wire, schema 2026-05-15 21:08:56 +00:00
CHANGELOG.md docs: add v0.3.0 changelog entry and network bindings reference (#29) 2026-05-18 14:26:09 -06:00
LICENSE scaffold: initial repository structure 2026-05-15 19:16:24 +00:00
pyproject.toml v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85) 2026-06-03 22:36:26 -06:00
README.md docs: add test database setup, restore geom to test fixture 2026-05-17 18:26:48 +00:00
uv.lock v0.10.0: ITD 511 official API adapter (events + advisories + cameras) (#85) 2026-06-03 22:36:26 -06:00

README.md

Central

Central is the data hub spine for the infrastructure. Adapters normalize upstream sources into a canonical event shape, publish CloudEvents to NATS/JetStream, and archive to TimescaleDB for historical query. Single-LXC deployment.

Status

Phase 0 — scaffold. Not yet operational.

Architecture

  • Python 3.12 (uv-managed)
  • NATS + JetStream for live event bus
  • TimescaleDB + PostGIS for archive and geospatial query
  • One supervisor process managing adapter lifecycle
  • One archive consumer process persisting events to TimescaleDB
  • Both processes systemd-managed

Testing

See docs/test-database.md for test database setup.

License

MIT. See LICENSE.