matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-06-10 11:54:37 +02:00
Code Issues Projects Releases Packages Wiki Activity
Central - data hub spine. Adapters -> NATS/JetStream -> archive.
233 commits 8 branches 67 tags 4.7 MiB
  • Python 89.1%
  • HTML 9.3%
  • CSS 1.3%
  • PLpgSQL 0.3%
Find a file
Matt Johnson 5c6d77381b fix(nws): scope sweep_old_ids to its own adapter (v0.9.19.1) 
NWSAdapter.sweep_old_ids ran an unscoped global DELETE FROM published_ids
(no adapter = ? filter), so each per-adapter sweep cycle purged EVERY
adapter dedup rows older than 8 days -- capping dedup memory at ~8d for the
8 adapters whose window exceeds it (eonet/gdacs/nwis 30d; swpc_kindex/
protons/alerts + wfigs_incidents/perimeters 14d). Events that went silent
past 8d then re-listed were re-published as downstream duplicates.

Extract the 3 dedup methods onto the SourceAdapter base (finishing v0.9.19;
is_published/mark_published were byte-identical) and preserve the 8-day
window via dedup_sweep_days = 8, so the inherited sweep scopes to adapter=?.
Effect: each adapter retains dedup to its configured window -> fewer
downstream duplicates. Retention-window scoping, NOT a dedup-forward-only
change (no event-shape change, no migration/backfill).

Adds a regression guard asserting a foreign adapter row survives an nws
sweep. Suite 900->901/1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 07:23:05 +00:00
docs feat(state_511_atis_cameras): Castle Rock 511 traffic cameras telemetry (v0.9.6) 2026-05-26 01:33:21 +00:00
etc-templates
scripts
sql v0.9.12: archive-level monitoring-area bbox filter 2026-05-26 23:40:17 +00:00
src/central fix(nws): scope sweep_old_ids to its own adapter (v0.9.19.1) 2026-05-27 07:23:05 +00:00
systemd feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
tests fix(nws): scope sweep_old_ids to its own adapter (v0.9.19.1) 2026-05-27 07:23:05 +00:00
.gitattributes
.gitignore feat(gui): add auth core, setup gate, and first-run operator creation 2026-05-17 05:30:49 +00:00
.python-version
CHANGELOG.md docs: add v0.3.0 changelog entry and network bindings reference (#29) 2026-05-18 14:26:09 -06:00
LICENSE
pyproject.toml feat(tomtom_flow): TomTom Orbis vector flow-tile telemetry adapter + CENTRAL_TRAFFIC_FLOW (v0.9.3) 2026-05-25 23:25:44 +00:00
README.md docs: add test database setup, restore geom to test fixture 2026-05-17 18:26:48 +00:00
uv.lock feat(tomtom_flow): TomTom Orbis vector flow-tile telemetry adapter + CENTRAL_TRAFFIC_FLOW (v0.9.3) 2026-05-25 23:25:44 +00:00

README.md

Central

Central is the data hub spine for the infrastructure. Adapters normalize upstream sources into a canonical event shape, publish CloudEvents to NATS/JetStream, and archive to TimescaleDB for historical query. Single-LXC deployment.

Status

Phase 0 — scaffold. Not yet operational.

Architecture

  • Python 3.12 (uv-managed)
  • NATS + JetStream for live event bus
  • TimescaleDB + PostGIS for archive and geospatial query
  • One supervisor process managing adapter lifecycle
  • One archive consumer process persisting events to TimescaleDB
  • Both processes systemd-managed

Testing

See docs/test-database.md for test database setup.

License

MIT. See LICENSE.