77b4715384
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/add-peertube-channel.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md vault/runbooks/peertube-remote-runner.md
116 lines
6.9 KiB
Markdown
116 lines
6.9 KiB
Markdown
---
|
||
title: central — Data-Hub Spine
|
||
type: reference
|
||
tags:
|
||
- mesh
|
||
aliases: []
|
||
related:
|
||
- [[central-deploy-cutover]]
|
||
- [[services]]
|
||
- [[caddy]]
|
||
- [[fleet-platform-baseline]]
|
||
- [[fleet-patch-audit]]
|
||
updated: 2026-07-16
|
||
---
|
||
# central — Data-Hub Spine
|
||
|
||
> **RETIRED + DROPPED (2026-07-15).** Central has been replaced by [[conduit]]. Its app [[services]] were stopped and disabled 2026-07-14 (zero live consumers remained); on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed), and the shared Postgres instance was cleaned back to plain (TimescaleDB removed). Central is recoverable only from the pi-nas archive. **Everything below this point is historical** — it describes how central worked while it was live, and is kept for reference only.
|
||
|
||
## Overview
|
||
|
||
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for [[navi]] traffic tiles and related situational-awareness feeds.
|
||
|
||
- **URL (internal):** http://central.echo6.mesh:8000 (mesh-only, no public exposure)
|
||
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
|
||
- **Repo:** github.com/zvx-echo6/central (public, Python, deployed at detached HEAD, tag v0.15.0)
|
||
- **Deploy path:** /opt/central (venv: /opt/central/.venv, env: /etc/central/central.env, system user: central)
|
||
|
||
## Host
|
||
|
||
| Attribute | Value |
|
||
|-----------|-------|
|
||
| Container | utility CT 104 |
|
||
| Local IP | 192.168.1.104 |
|
||
| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh |
|
||
| Resources | 4 cores / 12 GB RAM / 100 GB disk |
|
||
| OS | Ubuntu LXC (unprivileged) |
|
||
|
||
## Architecture
|
||
|
||
The data flow is: upstream APIs → adapters (central-supervisor) → NATS/JetStream :4222 → central-archive (TimescaleDB/PostGIS :5432). The central-gui (FastAPI + HTMX, :8000) exposes the API consumed by [[navi]] via recon-vm's nginx (port 8440, `^~ /api/traffic/` → `central.echo6.mesh:8000`).
|
||
|
||
## Systemd Services
|
||
|
||
All three units are **enabled and active**; [[deployment]] survives reboot. Deps: `nats-server`, `postgresql@16-main`.
|
||
|
||
| Unit | Role |
|
||
|------|------|
|
||
| central-supervisor | Adapter scheduler + CloudEvents publisher |
|
||
| central-archive | JetStream → TimescaleDB consumer |
|
||
| central-gui | FastAPI + HTMX web app + API (the :8000 listener) |
|
||
|
||
## Ports
|
||
|
||
| Port | Protocol | Purpose |
|
||
|------|----------|---------|
|
||
| :8000 | HTTP | GUI / API (bound 0.0.0.0) |
|
||
| :4222 | TCP | NATS client connections |
|
||
| :8222 | HTTP | NATS monitoring |
|
||
| :5432 | TCP | PostgreSQL 16 + TimescaleDB/PostGIS |
|
||
|
||
## Adapters (23 configured, 22 enabled)
|
||
|
||
| Domain | Sources |
|
||
|--------|---------|
|
||
| Traffic | ITD 511, WZDX, TomTom flow/incidents, 511 cameras |
|
||
| Wildfire | WFIGS incidents/perimeters, InciWeb, FIRMS |
|
||
| Weather / Space-weather | NWS, SWPC k-index/protons/alerts |
|
||
| Hydro | NWIS |
|
||
| Earthquakes | USGS |
|
||
| Avalanche | avalanche.org |
|
||
| Disasters | GDACS, EONET |
|
||
| Satellite | CelesTrak TLE, sat positions/orbits, satpass_predict (n2yo_visualpasses disabled 2026-06-27) |
|
||
|
||
## GUI / API Surface
|
||
|
||
Authenticated app with login/sessions/CSRF, first-run setup wizard, operator management, adapter configuration (incl. **no-code creation of generic REST/GeoJSON sources**, v0.15.0), stream viewer, **JetStream consumer management** (`/consumers` — view + delete consumers; central's own `archive-*` durable consumers are protected/non-deletable; CSRF + audit-logged), enrichment pipeline, monitoring-area management, API key management, audit log, and manual resend.
|
||
|
||
**Auth-exempt tile endpoints** (used by [[navi]], verified HTTP 200):
|
||
|
||
| Endpoint | Format |
|
||
|----------|--------|
|
||
| /api/traffic/flow/{z}/{x}/{y}.png | PNG tile (raster, image/png) |
|
||
| /api/traffic/flow/{z}/{x}/{y}.pbf | PBF tile (vector, application/x-protobuf) |
|
||
|
||
`/health` is also auth-exempt (returns `{"status":"ok"}`). All other `/api/` endpoints redirect to `/login`.
|
||
|
||
## Consumer — navi Integration
|
||
|
||
navi-traffic (navi's in-VM :8421 extraction service) was **retired on 2026-05-26** and cut over to central. recon-vm's nginx (`/etc/nginx/sites-available/navi.echo6.co`, port 8440) proxied `^~ /api/traffic/` → `central.echo6.mesh:8000` with a 120s tile cache. navi-traffic:8421 is confirmed dead and disabled.
|
||
|
||
**As of 2026-07-14, navi's `/api/traffic/` tiles were repointed to [[conduit]]:** recon-vm's nginx now rewrites `^~ /api/traffic/` → `/up/tomtom_flow_tiles/...` and proxies to `central.echo6.mesh:8010` ([[conduit]], co-resident on the same CT 104 host). Central's own tile endpoint (`:8000/api/traffic/...`) still exists and still works, but is no longer on navi's hot path — it remains the rollback target if the Conduit cutover needs to be reverted. See [[conduit-operations]] for the cutover and rollback procedure.
|
||
|
||
## Dependencies
|
||
|
||
| Component | Location |
|
||
|-----------|----------|
|
||
| NATS / JetStream | Local (central CT 104) |
|
||
| PostgreSQL 16 + TimescaleDB + PostGIS | Local (central CT 104) |
|
||
| Upstream APIs | ~20 external sources (see Adapters table) |
|
||
|
||
## Data Plane
|
||
|
||
Single TimescaleDB hypertable `public.events` (~40 GB, ~3.42 M rows, 331 chunks, compression not enabled). JetStream ~5.9 GB / 20 GB across 12 streams.
|
||
|
||
## Deploy / State Notes (as of 2026-07-13)
|
||
|
||
- **Deployed HEAD:** v0.15.0 (detached HEAD — the box tracks tags; sha `3c8da28`, `pyproject.toml` 0.15.0). All 3 units active, `/health` 200, 23 adapters configured / 22 enabled (only `n2yo_visualpasses` disabled).
|
||
- **v0.15.0 (deployed 2026-07-02, PRs #120–124):** the GUI can now add public REST/GeoJSON data sources **no-code** — a generic adapter kind created and configured entirely through adapter management, no repo change. None enabled in this [[deployment]] yet; all 23 current adapters remain bespoke per-source kinds (each `config.adapters.kind` is unique to its source).
|
||
- **Deploy mechanism:** manual, tag-based (`git checkout <tag>` → `uv sync` → `central-migrate` → restart). One-command `scripts/deploy.sh` ships in-repo; full procedure in the [[central-deploy-cutover]] runbook. No CI/CD; no automated DB backup (pre-flight `pg_dump` is the only migration safety net; migrations are forward-only).
|
||
- **`central-migrate` env gotcha:** manual calls need `cd /opt/central && set -a && . /etc/central/central.env && set +a && …` (the units supply this via WorkingDirectory + EnvironmentFile).
|
||
- **EONET:** now **global** as of v0.14.6 (2026-06-28) — `bypass_bbox_filter=True` + the adapter `region` key removed. NOTE: its `events`-table row count is a misleading health signal (dedup + partition expiry); use the CENTRAL_DISASTER stream as the EONET flow signal. `avalanche_org` zero-events is expected (off-season gate).
|
||
- **Supervisor CPU** ~17%→~1.4% and **cursors.db** 1.3 GB→496 MB after the v0.14.6 dedup/WAL fix + a VACUUM.
|
||
|
||
---
|
||
|
||
*Last updated: 2026-07-13 — v0.15.0 deployed & verified (GUI no-code generic REST/GeoJSON adapters, PRs #120–124); 3.42 M events / ~40 GB, all units active.*
|