matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-05-21 18:14:44 +02:00
Code Issues Projects Releases Packages Wiki Activity
Central - data hub spine. Adapters -> NATS/JetStream -> archive.
140 commits 2 branches 11 tags 1.7 MiB
  • Python 90.5%
  • HTML 9.1%
  • PLpgSQL 0.4%
Find a file
zvx 5d64a8f70d feat(2-G): USGS NWIS adapter (OGC API) + CENTRAL_HYDRO stream 
NASA WaterData OGC API v0 (latest-continuous collection) — polls configured
parameter codes within an operator-set bbox and publishes on the new
CENTRAL_HYDRO stream.

- Subject: central.hydro.<parameter_code>.<agency>.<bare_site_no>
  (e.g. central.hydro.00060.usgs.05420500). The agency/site decomposition
  lives in a single _subject_tokens_for_id helper.
- Default parameter codes: 00060 (discharge), 00065 (gage height),
  00010 (water temperature). Operator-tunable; single SoT in
  _DEFAULT_PARAMETER_CODES — no parallel literals.
- Composite dedup: nwis:<monitoring_location_id>:<param>:<time_iso>.
  Prefix kept in dedup key for cross-agency uniqueness.
- Pagination: follows OGC 'rel=next' link until absent (cursor-based).
- Region bbox is REQUIRED in practice; adapter logs WARN at startup if
  region is None (does not refuse to start).
- New stream CENTRAL_HYDRO added to streams.py registry (one line).
  Retention mirrors CENTRAL_DISASTER (7 days, 1 GiB).
- No removal pattern in v1 — sites are static; missing data is the signal.

Upstream divergences from the original spec brief, caught by pre-build curl:
- Collection is 'latest-continuous', not 'instantaneous-values'.
- Site filter param is 'monitoring_location_id' (singular), not
  'monitoring_locations_id' (plural).
- Site identifier requires agency prefix in queries (USGS-NNNNN).
- feature.id is a per-record UUID, not stable; dedup uses joint key.

Ships disabled; operator enables via GUI after setting a bbox.
2026-05-19 16:50:21 +00:00
docs
etc-templates
scripts
sql feat(2-G): USGS NWIS adapter (OGC API) + CENTRAL_HYDRO stream 2026-05-19 16:50:21 +00:00
src/central feat(2-G): USGS NWIS adapter (OGC API) + CENTRAL_HYDRO stream 2026-05-19 16:50:21 +00:00
systemd
tests feat(2-G): USGS NWIS adapter (OGC API) + CENTRAL_HYDRO stream 2026-05-19 16:50:21 +00:00
.gitattributes
.gitignore
.python-version
CHANGELOG.md
LICENSE
pyproject.toml release: bump version to 0.3.0 (#30) 2026-05-18 14:29:28 -06:00
README.md
uv.lock

README.md

Central

Central is the data hub spine for the infrastructure. Adapters normalize upstream sources into a canonical event shape, publish CloudEvents to NATS/JetStream, and archive to TimescaleDB for historical query. Single-LXC deployment.

Status

Phase 0 — scaffold. Not yet operational.

Architecture

  • Python 3.12 (uv-managed)
  • NATS + JetStream for live event bus
  • TimescaleDB + PostGIS for archive and geospatial query
  • One supervisor process managing adapter lifecycle
  • One archive consumer process persisting events to TimescaleDB
  • Both processes systemd-managed

Testing

See docs/test-database.md for test database setup.

License

MIT. See LICENSE.