auto: docs sync 2026-07-16T00:00:19+00:00
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md
This commit is contained in:
echo6-autocommit
parent
0a2440593b
commit
8ba425146b
6 changed files with 103 additions and 65 deletions
|
|
@ -9,18 +9,20 @@ related:
|
|||
- [[central]]
|
||||
- [[navi]]
|
||||
- [[caddy]]
|
||||
updated: 2026-07-14
|
||||
- [[meshai]]
|
||||
updated: 2026-07-15
|
||||
---
|
||||
# Conduit — Raw-API Broker
|
||||
|
||||
## Overview
|
||||
|
||||
Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch an upstream API **once**, store the raw response **as-is** (byte-for-byte, no normalization), and serve it to **many** internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
|
||||
Conduit is the fleet's **single egress point** for every external API feed. It fetches an upstream once, stores the raw response **as-is** (byte-for-byte, no normalization), and fans it out to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
|
||||
|
||||
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010` as of 2026-07-14 for [[navi]] to reach it)
|
||||
- **Host:** utility CT 104 (unprivileged Ubuntu LXC), co-resident with [[central]]
|
||||
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010`)
|
||||
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
|
||||
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
|
||||
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
|
||||
- **Consumers:** [[navi]]'s traffic tiles (`tomtom_flow_tiles`) and all 13 of [[meshai]]'s hazard adapters (migrated 2026-07-15 — see [[conduit-operations]])
|
||||
|
||||
## Host
|
||||
|
||||
|
|
@ -33,58 +35,78 @@ Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch
|
|||
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
|
||||
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
|
||||
| Master key | `/etc/conduit/master.key` |
|
||||
| Systemd unit | `conduit.service` (enabled, single-worker, `Wants=postgresql@16-main`) |
|
||||
| Systemd unit | `conduit.service` (enabled, **single uvicorn worker only** — never add `--workers`; single-flight + quota lock are per-process) |
|
||||
| Bind | `0.0.0.0:8010` |
|
||||
|
||||
## Architecture — the "source" model
|
||||
|
||||
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction.
|
||||
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. The `sources` table holds ~21 rows.
|
||||
|
||||
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) or **push** (an app POSTs a raw payload in — planned, not yet exercised).
|
||||
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL. This is a **cache, not a historical archive** — unlike central's TimescaleDB event archive.
|
||||
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) is live. **push** (an app POSTs a raw payload in) is planned, not yet exercised.
|
||||
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL — a cache, not by itself a historical archive (see Retention below for the opt-in exception).
|
||||
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
|
||||
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
|
||||
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).
|
||||
|
||||
**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.
|
||||
|
||||
### Per-source columns
|
||||
|
||||
`name`, `url_template` (`{path}`/`{key}` substitution; an inbound query string appends rather than colliding), `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, plus:
|
||||
|
||||
- **static `headers`** (jsonb) — for UA-sensitive feeds: NWS needs a contact User-Agent, Idaho Power WAFs non-browser UAs, avalanche.org sends a UA.
|
||||
- **quota caps** — `max_calls_per_day` / `max_calls_per_minute` / `max_calls_per_month`.
|
||||
- **retention** — `retain`, `retention_days`, `poll_interval_seconds`, `poll_path`.
|
||||
|
||||
## Capabilities
|
||||
|
||||
- **Single-flight coalescing** — concurrent misses for the same `(source, request_key)` → one upstream call.
|
||||
- **Quota guard** (`quota.py`) — per-source day/minute/month caps enforced against a durable `upstream_calls` log, atomic per-source `asyncio.Lock` (single-process). Protects free-tier upstreams (e.g. TomTom's free plan).
|
||||
- **Serve-stale** — on quota-block, upstream 429, or 5xx/transport failure, Conduit serves the last-known-good cached copy instead of failing the caller (`X-Conduit-Stale: 1`).
|
||||
- **Faithful 4xx passthrough** (PR #16, 2026-07-15) — a genuine upstream 4xx *except* 429 (e.g. TomTom flow's `400 "Point too far from nearest existing segment"`) is returned to the caller with its real status + body, uncached, not wrapped as a 502. Only 429/5xx/transport failures fall back to serve-stale-or-502. This unblocked the last meshai adapter (traffic) to migrate.
|
||||
- **Hot-reload of sources** — a source add/edit/delete made through the GUI calls `Broker.set_sources()` and takes effect with no restart. A direct out-of-band SQL change to `sources` still needs `systemctl restart conduit` — the GUI path is the live one.
|
||||
- **Retention engine** (`poller.py` + `store/history.py`) — opt-in per source (`retain=true`). A background `Poller` fetches retained sources on their `poll_interval_seconds` and appends changed raw payloads to `payload_history` (append-on-change, sha256-dedup). Idle by construction when nothing is retained. Read back via `GET /history/{source}` and `/history/{source}/{id}/body` (unauthenticated, mesh-internal). Purpose: accumulate raw feeds for later forecast/trend models.
|
||||
- **Management GUI** (`gui/`) — built in meshai's visual language. argon2 operator auth + two-tier CSRF + schema-reflection forms harvested from central. Pages: sources CRUD (incl. headers/quota/retention fields), API keys, a Cmd-K command palette. Admin operator provisioned. `/up` and `/history` stay unauthenticated (mesh-internal, tiles-trust model); the GUI is the authenticated surface.
|
||||
- **Keystore** (`keystore.py`, `crypto.py`) — AES-256-GCM `api_keys` by alias, under Conduit's own master key. Holds `tomtom`, `roads511`, and `firms` keys (harvested from central / provisioned) — meshai no longer holds any of these itself.
|
||||
|
||||
## Modules (`src/conduit/`)
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`), ported from central's `crypto.py` |
|
||||
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`) |
|
||||
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
|
||||
| `config.py` | App configuration loading |
|
||||
| `migrate.py` | Forward-only SQL migration runner (console script `conduit-migrate`) |
|
||||
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff) |
|
||||
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch + quota + serve-stale; the `get(source_name, request_key)` payoff path |
|
||||
| `quota.py` | Per-source day/minute/month call-quota enforcement against `upstream_calls` |
|
||||
| `poller.py` | Background retention poller for `retain=true` sources |
|
||||
| `admin_cli.py` | `conduit-admin` command-line administration |
|
||||
| `app.py` | FastAPI app wiring all routes |
|
||||
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
|
||||
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only; does not cross-coalesce across workers) |
|
||||
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff), consolidating central's per-adapter fetch idiom |
|
||||
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template` with `{path}`/`{key}` substitution) |
|
||||
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch; the `get(source_name, request_key)` payoff path |
|
||||
| `app.py` | FastAPI app: `GET /health`, `GET /up/{source}/{path}` |
|
||||
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only) |
|
||||
| `store/history.py` | `payload_history` accessor for the retention engine |
|
||||
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template`, headers, quota, retention fields) |
|
||||
| `gui/{auth,csrf,deps,routes}` + templates/static | Management GUI: operator auth, CSRF, forms, command palette |
|
||||
|
||||
**Endpoints:** `GET /health`, `GET /up/{source}/{path}`, `GET /history/{source}`, `GET /history/{source}/{id}/body`, plus the GUI routes.
|
||||
|
||||
**Migrations:** `sql/migrations/001`–`007` (schema-only, no seeds), run by `conduit-migrate`, tracked in `schema_migrations`.
|
||||
|
||||
## Repo / deploy model
|
||||
|
||||
Pull-based deploy, mirroring central: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table — simplified port of central's migration runner.
|
||||
|
||||
## Sources
|
||||
|
||||
Sources are DB-backed (the `sources` table, not a hardcoded dict). `app.py` loads all enabled sources once at startup and hands them to the `Broker` — **there is no hot-reload yet**; a change to the `sources` table requires a `systemctl restart conduit` to take effect.
|
||||
|
||||
Live source: `tomtom_flow_tiles` (TomTom Orbis flow tiles), real TomTom key provisioned into Conduit's keystore.
|
||||
Pull-based deploy: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table.
|
||||
|
||||
## Relationship to [[central]]
|
||||
|
||||
Conduit was born by harvesting central's proven, decoupled pieces:
|
||||
**Central is retired and dropped (2026-07-15) — Conduit replaced it.** Central's app services were stopped and disabled 2026-07-14 (zero live consumers remained), then on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed); the shared Postgres instance was cleaned back to plain (TimescaleDB removed from `shared_preload_libraries`). Central is recoverable only from the pi-nas archive. Conduit's own `conduit` DB shares that same Postgres instance, which was never stopped.
|
||||
|
||||
- The AES-256-GCM encrypted key store (lifted ~as-is).
|
||||
- GUI auth/CSRF + schema-reflection "add a source" form patterns (the Conduit GUI itself is still TODO).
|
||||
- The aiohttp+tenacity fetch idiom.
|
||||
Conduit was born by harvesting central's proven, decoupled pieces — the AES-256-GCM encrypted key store, the GUI auth/CSRF + schema-reflection form patterns, and the aiohttp+tenacity fetch idiom — while deliberately shedding central's NATS/JetStream, CloudEvents normalization, and enrichment pipeline.
|
||||
|
||||
It deliberately does **not** carry central's NATS/JetStream, CloudEvents normalization, TimescaleDB event archive, or enrichment pipeline — that complexity is exactly what Conduit sheds. Central is not being retired; it's kept, and partially superseded over time on a per-consumer basis.
|
||||
## Current state (as of 2026-07-15)
|
||||
|
||||
**[[navi]]'s traffic tiles were repointed from central to Conduit on 2026-07-14** — see [[conduit-operations]] for the cutover and rollback procedure.
|
||||
|
||||
## Current state (as of 2026-07-14)
|
||||
|
||||
- PRs #1–#5 merged to `main`. #5 added the DB-backed source registry (`sources` table, seeded with `tomtom_flow_tiles`).
|
||||
- First vertical live and durable: navi's traffic tiles are served by `conduit.service`, byte-identical to central's, confirmed via the real nginx path and `payloads` table rows.
|
||||
- Known next steps: a source-management **GUI** (DB registry is SQL-only so far) and **source hot-reload** (currently requires a service restart after any `sources` table change).
|
||||
- Single egress point for the whole fleet: [[navi]]'s traffic tiles plus all 13 [[meshai]] hazard adapters (NWS, SWPC ×4, ducting, WFIGS/fires ×2, FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx, satpass) — see [[conduit-operations]] for the migration record.
|
||||
- `sources` table holds ~21 rows.
|
||||
- Quota guard, serve-stale, faithful-4xx-passthrough, hot-reload, the retention engine, and the management GUI are all live (not planned).
|
||||
- Keystore holds `tomtom`, `roads511`, `firms` — meshai no longer holds any of these keys directly.
|
||||
- Central is gone; it is no longer a rollback target for anything.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue