echo6-docs/vault/docs/software/conduit.md

title	type	tags	aliases	related	updated
Conduit — Raw-API Broker	reference	mesh		conduit-operations central navi caddy meshai	2026-07-15

Conduit — Raw-API Broker

Overview

Conduit is the fleet's single egress point for every external API feed. It fetches an upstream once, stores the raw response as-is (byte-for-byte, no normalization), and fans it out to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."

• URL (internal): http://central.echo6.mesh:8010 (mesh; bound 0.0.0.0:8010)
• Host: utility CT 104 (unprivileged Ubuntu LXC)
• Repo: github.com/zvx-echo6/conduit (private, Python/FastAPI)
• Deploy path: /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
• Consumers: navi's traffic tiles (tomtom_flow_tiles) and all 13 of meshai's hazard adapters (migrated 2026-07-15 — see conduit-operations)

Host

Attribute	Value
Container	utility CT 104
Tailscale / mesh	100.64.0.12 → central.echo6.mesh:8010
Deploy dir	/opt/conduit (owned conduit:conduit)
Virtualenv	/opt/conduit/.venv (uv-managed, editable install)
Database	own conduit DB + role on the shared Postgres 16 (plain, no extensions)
Env file	/etc/conduit/conduit.env (CONDUIT_DB_DSN, CONDUIT_MASTER_KEY_PATH)
Master key	/etc/conduit/master.key
Systemd unit	conduit.service (enabled, single uvicorn worker only — never add --workers; single-flight + quota lock are per-process)
Bind	0.0.0.0:8010

Architecture — the "source" model

Everything Conduit brokers is a source — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. The sources table holds ~21 rows.

• Ingest: pull (Conduit fetches the upstream on cadence/on-demand) is live. push (an app POSTs a raw payload in) is planned, not yet exercised.
• Store: the latest raw payload per (source, request_key), with a per-source freshness/TTL — a cache, not by itself a historical archive (see Retention below for the opt-in exception).
• Serve: a uniform GET /up/{source}/{path} returns the raw bytes byte-for-byte, no transformation.
• Single-flight: concurrent cache misses for the same (source, request_key) coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
• Topologies — all fall out of the one model: internet → conduit → app, app → conduit → app, and chained (internet → conduit → A → conduit → B, where A registers its own output as a new source).

Key property: upstream calls scale with unique resources × refresh rate, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.

Per-source columns

name, url_template ({path}/{key} substitution; an inbound query string appends rather than colliding), api_key_alias, ttl_seconds, header_auth, enabled, plus:

• static headers (jsonb) — for UA-sensitive feeds: NWS needs a contact User-Agent, Idaho Power WAFs non-browser UAs, avalanche.org sends a UA.
• quota caps — max_calls_per_day / max_calls_per_minute / max_calls_per_month.
• retention — retain, retention_days, poll_interval_seconds, poll_path.

Capabilities

• Single-flight coalescing — concurrent misses for the same (source, request_key) → one upstream call.
• Quota guard (quota.py) — per-source day/minute/month caps enforced against a durable upstream_calls log, atomic per-source asyncio.Lock (single-process). Protects free-tier upstreams (e.g. TomTom's free plan).
• Serve-stale — on quota-block, upstream 429, or 5xx/transport failure, Conduit serves the last-known-good cached copy instead of failing the caller (X-Conduit-Stale: 1).
• Faithful 4xx passthrough (PR #16, 2026-07-15) — a genuine upstream 4xx except 429 (e.g. TomTom flow's 400 "Point too far from nearest existing segment") is returned to the caller with its real status + body, uncached, not wrapped as a 502. Only 429/5xx/transport failures fall back to serve-stale-or-502. This unblocked the last meshai adapter (traffic) to migrate.
• Hot-reload of sources — a source add/edit/delete made through the GUI calls Broker.set_sources() and takes effect with no restart. A direct out-of-band SQL change to sources still needs systemctl restart conduit — the GUI path is the live one.
• Retention engine (poller.py + store/history.py) — opt-in per source (retain=true). A background Poller fetches retained sources on their poll_interval_seconds and appends changed raw payloads to payload_history (append-on-change, sha256-dedup). Idle by construction when nothing is retained. Read back via GET /history/{source} and /history/{source}/{id}/body (unauthenticated, mesh-internal). Purpose: accumulate raw feeds for later forecast/trend models.
• Management GUI (gui/) — built in meshai's visual language. argon2 operator auth + two-tier CSRF + schema-reflection forms harvested from central. Pages: sources CRUD (incl. headers/quota/retention fields), API keys, a Cmd-K command palette. Admin operator provisioned. /up and /history stay unauthenticated (mesh-internal, tiles-trust model); the GUI is the authenticated surface.
• Keystore (keystore.py, crypto.py) — AES-256-GCM api_keys by alias, under Conduit's own master key. Holds tomtom, roads511, and firms keys (harvested from central / provisioned) — meshai no longer holds any of these itself.

Modules (src/conduit/)

Module	Role
crypto.py	AES-256-GCM encrypted secret storage (master key from CONDUIT_MASTER_KEY_PATH)
keystore.py	KeyStore — async accessor for the api_keys table (encrypted API keys by alias)
config.py	App configuration loading
migrate.py	Forward-only SQL migration runner (console script conduit-migrate)
fetch.py	Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff)
broker.py	Broker — orchestrates store + keystore + single-flight + fetch + quota + serve-stale; the get(source_name, request_key) payoff path
quota.py	Per-source day/minute/month call-quota enforcement against upstream_calls
poller.py	Background retention poller for retain=true sources
admin_cli.py	conduit-admin command-line administration
app.py	FastAPI app wiring all routes
store/payloads.py	PayloadStore / StoredPayload — latest-by-(source, request_key) raw-byte cache with TTL-derived freshness
store/singleflight.py	SingleFlight — in-process coalescer for concurrent misses on the same key (per-process only)
store/history.py	payload_history accessor for the retention engine
sources/__init__.py	SourceRegistry (DB-backed sources table accessor) + PullSource (per-source url_template, headers, quota, retention fields)
gui/{auth,csrf,deps,routes} + templates/static	Management GUI: operator auth, CSRF, forms, command palette

Endpoints: GET /health, GET /up/{source}/{path}, GET /history/{source}, GET /history/{source}/{id}/body, plus the GUI routes.

Migrations: sql/migrations/001–007 (schema-only, no seeds), run by conduit-migrate, tracked in schema_migrations.

Repo / deploy model

Pull-based deploy: authored/pushed from a cortex clone, CT 104 pulls via a read-only deploy key (ct104-conduit-deploy) — CT 104 cannot push. conduit-migrate runs forward-only SQL migrations (sql/migrations/*.sql), tracked in a schema_migrations table.

Relationship to central

Central is retired and dropped (2026-07-15) — Conduit replaced it. Central's app services were stopped and disabled 2026-07-14 (zero live consumers remained), then on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (DROP DATABASE central, ~41 GB reclaimed); the shared Postgres instance was cleaned back to plain (TimescaleDB removed from shared_preload_libraries). Central is recoverable only from the pi-nas archive. Conduit's own conduit DB shares that same Postgres instance, which was never stopped.

Conduit was born by harvesting central's proven, decoupled pieces — the AES-256-GCM encrypted key store, the GUI auth/CSRF + schema-reflection form patterns, and the aiohttp+tenacity fetch idiom — while deliberately shedding central's NATS/JetStream, CloudEvents normalization, and enrichment pipeline.

Current state (as of 2026-07-15)

• Single egress point for the whole fleet: navi's traffic tiles plus all 13 meshai hazard adapters (NWS, SWPC ×4, ducting, WFIGS/fires ×2, FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx, satpass) — see conduit-operations for the migration record.
• sources table holds ~21 rows.
• Quota guard, serve-stale, faithful-4xx-passthrough, hot-reload, the retention engine, and the management GUI are all live (not planned).
• Keystore holds tomtom, roads511, firms — meshai no longer holds any of these keys directly.
• Central is gone; it is no longer a rollback target for anything.