mirror of
https://github.com/zvx-echo6/central.git
synced 2026-06-10 11:54:37 +02:00
Central - data hub spine. Adapters -> NATS/JetStream -> archive.
- Python 89.1%
- HTML 9.3%
- CSS 1.3%
- PLpgSQL 0.3%
malice
81a2c30fdbJetStream's 2-min per-stream duplicate window keys on Nats-Msg-Id alone --
the subject is NOT part of the dedup key. So when wfigs_incidents and
wfigs_perimeters published envelopes for the same Summit Creek IRWIN
{6B1C6EB1-30F7-4613-9C58-4801DC8FD822} within seconds of each other on
2026-06-07, the second publish was silently dropped: supervisor logged
'Published event' for both, but CENTRAL_FIRE's last_seq advanced by 25
instead of 26.
Root cause: cloudevents_wire.wrap_event returned event.id verbatim as
msg_id, and both wfigs adapters use the bare IRWIN GUID as event.id.
Fix: synthesize msg_id as f'{event.id}:{event.category}' in wrap_event.
Categories already differ ('fire.incident.X' vs 'fire.perimeter.X') so
this is a natural and load-bearing discriminator. envelope['id'] stays
the bare event.id per CloudEvents spec, so subscribers that key off the
payload id field are unaffected. Adapter-side event.id construction
unchanged.
Backward compatibility: this is a one-time msg_id-shape change. The
JetStream 2-min dedup window is per-(event-id, category) starting now
instead of per-event-id. For events still in the dedup window at the
moment of the deploy under the OLD shape, their first post-deploy
publish under the NEW shape is a fresh msg_id and lands normally -- no
collision risk. Quake and other single-class adapters get a no-op shape
change (their category is constant per adapter, so dedup still catches
genuine same-id duplicates).
Tests:
- test_required_fields_present updated for the new assertion.
- New test_msgid_disambiguates_incident_vs_perimeter_same_guid regression
guard with the real Summit Creek IRWIN -- assert the two msg_ids differ
and match the expected shape exactly.
- New parametrize test_msgid_shape_is_id_colon_category over 5
categories covering multi-class (fire.incident, fire.perimeter) and
single-class (quake, wx.alert, hydro) adapters.
Full sweep: 1028 passed, 0 failures (+6 from this PR), ruff clean.
Deploy: NOT an output-shape change to envelope payload, no published_ids
flush needed. Squash-merge -> tag v0.10.8 -> pull on central -> restart
central-supervisor. Verify by forcing a Summit Creek republish and
confirming both .incident.id.cassia AND .perimeter.id.cassia land
post-deploy with the same IRWIN.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| docs | ||
| etc-templates | ||
| scripts | ||
| sql | ||
| src/central | ||
| systemd | ||
| tests | ||
| .gitattributes | ||
| .gitignore | ||
| .python-version | ||
| CHANGELOG.md | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
Central
Central is the data hub spine for the infrastructure. Adapters normalize upstream sources into a canonical event shape, publish CloudEvents to NATS/JetStream, and archive to TimescaleDB for historical query. Single-LXC deployment.
Status
Phase 0 — scaffold. Not yet operational.
Architecture
- Python 3.12 (uv-managed)
- NATS + JetStream for live event bus
- TimescaleDB + PostGIS for archive and geospatial query
- One supervisor process managing adapter lifecycle
- One archive consumer process persisting events to TimescaleDB
- Both processes systemd-managed
Testing
See docs/test-database.md for test database setup.
License
MIT. See LICENSE.