77b4715384
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/add-peertube-channel.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md vault/runbooks/peertube-remote-runner.md
112 lines
9.3 KiB
Markdown
112 lines
9.3 KiB
Markdown
---
|
||
title: Conduit — Raw-API Broker
|
||
type: reference
|
||
tags:
|
||
- mesh
|
||
aliases: []
|
||
related:
|
||
- [[conduit-operations]]
|
||
- [[recon-operations]]
|
||
- [[recon]]
|
||
- [[central]]
|
||
- [[fleet-patch-audit]]
|
||
updated: 2026-07-16
|
||
---
|
||
# Conduit — Raw-API Broker
|
||
|
||
## Overview
|
||
|
||
Conduit is the fleet's **single egress point** for every external API feed. It fetches an upstream once, stores the raw response **as-is** (byte-for-byte, no normalization), and fans it out to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
|
||
|
||
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010`)
|
||
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
|
||
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
|
||
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
|
||
- **Consumers:** [[navi]]'s traffic tiles (`tomtom_flow_tiles`) and all 13 of [[meshai]]'s hazard adapters (migrated 2026-07-15 — see [[conduit-operations]])
|
||
|
||
## Host
|
||
|
||
| Attribute | Value |
|
||
|-----------|-------|
|
||
| Container | utility CT 104 |
|
||
| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh:8010 |
|
||
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
|
||
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
|
||
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
|
||
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
|
||
| Master key | `/etc/conduit/master.key` |
|
||
| Systemd unit | `conduit.service` (enabled, **single uvicorn worker only** — never add `--workers`; single-flight + quota lock are per-process) |
|
||
| Bind | `0.0.0.0:8010` |
|
||
|
||
## Architecture — the "source" model
|
||
|
||
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. The `sources` table holds ~21 rows.
|
||
|
||
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) is live. **push** (an app POSTs a raw payload in) is planned, not yet exercised.
|
||
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL — a cache, not by itself a historical archive (see Retention below for the opt-in exception).
|
||
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
|
||
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
|
||
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).
|
||
|
||
**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.
|
||
|
||
### Per-source columns
|
||
|
||
`name`, `url_template` (`{path}`/`{key}` substitution; an inbound query string appends rather than colliding), `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, plus:
|
||
|
||
- **static `headers`** (jsonb) — for UA-sensitive feeds: NWS needs a contact User-Agent, Idaho Power WAFs non-browser UAs, avalanche.org sends a UA.
|
||
- **quota caps** — `max_calls_per_day` / `max_calls_per_minute` / `max_calls_per_month`.
|
||
- **retention** — `retain`, `retention_days`, `poll_interval_seconds`, `poll_path`.
|
||
|
||
## Capabilities
|
||
|
||
- **Single-flight coalescing** — concurrent misses for the same `(source, request_key)` → one upstream call.
|
||
- **Quota guard** (`quota.py`) — per-source day/minute/month caps enforced against a durable `upstream_calls` log, atomic per-source `asyncio.Lock` (single-process). Protects free-tier upstreams (e.g. TomTom's free plan).
|
||
- **Serve-stale** — on quota-block, upstream 429, or 5xx/transport failure, Conduit serves the last-known-good cached copy instead of failing the caller (`X-Conduit-Stale: 1`).
|
||
- **Faithful 4xx passthrough** (PR #16, 2026-07-15) — a genuine upstream 4xx *except* 429 (e.g. TomTom flow's `400 "Point too far from nearest existing segment"`) is returned to the caller with its real status + body, uncached, not wrapped as a 502. Only 429/5xx/transport failures fall back to serve-stale-or-502. This unblocked the last [[meshai]] adapter (traffic) to migrate.
|
||
- **Hot-reload of sources** — a source add/edit/delete made through the GUI calls `Broker.set_sources()` and takes effect with no restart. A direct out-of-band SQL change to `sources` still needs `systemctl restart conduit` — the GUI path is the live one.
|
||
- **Retention engine** (`poller.py` + `store/history.py`) — opt-in per source (`retain=true`). A background `Poller` fetches retained sources on their `poll_interval_seconds` and appends changed raw payloads to `payload_history` (append-on-change, sha256-dedup). Idle by construction when nothing is retained. Read back via `GET /history/{source}` and `/history/{source}/{id}/body` (unauthenticated, mesh-internal). Purpose: accumulate raw feeds for later forecast/trend models.
|
||
- **Management GUI** (`gui/`) — built in meshai's visual language. argon2 operator auth + two-tier CSRF + schema-reflection forms harvested from central. Pages: sources CRUD (incl. headers/quota/retention fields), API keys, a Cmd-K command palette. Admin operator provisioned. `/up` and `/history` stay unauthenticated (mesh-internal, tiles-trust model); the GUI is the authenticated surface.
|
||
- **Keystore** (`keystore.py`, `crypto.py`) — AES-256-GCM `api_keys` by alias, under Conduit's own master key. Holds `tomtom`, `roads511`, and `firms` keys (harvested from [[central]] / provisioned) — meshai no longer holds any of these itself.
|
||
|
||
## Modules (`src/conduit/`)
|
||
|
||
| Module | Role |
|
||
|--------|------|
|
||
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`) |
|
||
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
|
||
| `config.py` | App configuration loading |
|
||
| `migrate.py` | Forward-only SQL migration runner (console script `conduit-migrate`) |
|
||
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff) |
|
||
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch + quota + serve-stale; the `get(source_name, request_key)` payoff path |
|
||
| `quota.py` | Per-source day/minute/month call-quota enforcement against `upstream_calls` |
|
||
| `poller.py` | Background retention poller for `retain=true` sources |
|
||
| `admin_cli.py` | `conduit-admin` command-line administration |
|
||
| `app.py` | FastAPI app wiring all routes |
|
||
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
|
||
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only) |
|
||
| `store/history.py` | `payload_history` accessor for the retention engine |
|
||
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template`, headers, quota, retention fields) |
|
||
| `gui/{auth,csrf,deps,routes}` + templates/static | Management GUI: operator auth, CSRF, forms, command palette |
|
||
|
||
**Endpoints:** `GET /health`, `GET /up/{source}/{path}`, `GET /history/{source}`, `GET /history/{source}/{id}/body`, plus the GUI routes.
|
||
|
||
**Migrations:** `sql/migrations/001`–`007` (schema-only, no seeds), run by `conduit-migrate`, tracked in `schema_migrations`.
|
||
|
||
## Repo / deploy model
|
||
|
||
Pull-based deploy: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table.
|
||
|
||
## Relationship to [[central]]
|
||
|
||
**Central is retired and dropped (2026-07-15) — Conduit replaced it.** Central's app [[services]] were stopped and disabled 2026-07-14 (zero live consumers remained), then on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed); the shared Postgres instance was cleaned back to plain (TimescaleDB removed from `shared_preload_libraries`). Central is recoverable only from the pi-nas archive. Conduit's own `conduit` DB shares that same Postgres instance, which was never stopped.
|
||
|
||
Conduit was born by harvesting central's proven, decoupled pieces — the AES-256-GCM encrypted key store, the GUI auth/CSRF + schema-reflection form patterns, and the aiohttp+tenacity fetch idiom — while deliberately shedding central's NATS/JetStream, CloudEvents normalization, and enrichment pipeline.
|
||
|
||
## Current state (as of 2026-07-15)
|
||
|
||
- Single egress point for the whole fleet: [[navi]]'s traffic tiles plus all 13 [[meshai]] hazard adapters (NWS, SWPC ×4, ducting, WFIGS/fires ×2, FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx, satpass) — see [[conduit-operations]] for the migration record.
|
||
- `sources` table holds ~21 rows.
|
||
- Quota guard, serve-stale, faithful-4xx-passthrough, hot-reload, the retention engine, and the management GUI are all live (not planned).
|
||
- Keystore holds `tomtom`, `roads511`, `firms` — meshai no longer holds any of these keys directly.
|
||
- Central is gone; it is no longer a rollback target for anything.
|