echo6-docs/vault/docs/software/conduit.md

title	type	tags	aliases	related	updated
Conduit — Raw-API Broker	reference	mesh		conduit-operations central navi caddy	2026-07-14

Conduit — Raw-API Broker

Overview

Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch an upstream API once, store the raw response as-is (byte-for-byte, no normalization), and serve it to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."

• URL (internal): http://central.echo6.mesh:8010 (mesh; bound 0.0.0.0:8010 as of 2026-07-14 for navi to reach it)
• Host: utility CT 104 (unprivileged Ubuntu LXC), co-resident with central
• Repo: github.com/zvx-echo6/conduit (private, Python/FastAPI)
• Deploy path: /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)

Host

Attribute	Value
Container	utility CT 104
Tailscale / mesh	100.64.0.12 → central.echo6.mesh:8010
Deploy dir	/opt/conduit (owned conduit:conduit)
Virtualenv	/opt/conduit/.venv (uv-managed, editable install)
Database	own conduit DB + role on the shared Postgres 16 (plain, no extensions)
Env file	/etc/conduit/conduit.env (CONDUIT_DB_DSN, CONDUIT_MASTER_KEY_PATH)
Master key	/etc/conduit/master.key
Systemd unit	conduit.service (enabled, single-worker, Wants=postgresql@16-main)
Bind	0.0.0.0:8010

Architecture — the "source" model

Everything Conduit brokers is a source — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction.

• Ingest: pull (Conduit fetches the upstream on cadence/on-demand) or push (an app POSTs a raw payload in — planned, not yet exercised).
• Store: the latest raw payload per (source, request_key), with a per-source freshness/TTL. This is a cache, not a historical archive — unlike central's TimescaleDB event archive.
• Serve: a uniform GET /up/{source}/{path} returns the raw bytes byte-for-byte, no transformation.
• Single-flight: concurrent cache misses for the same (source, request_key) coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
• Topologies — all fall out of the one model: internet → conduit → app, app → conduit → app, and chained (internet → conduit → A → conduit → B, where A registers its own output as a new source).

Key property: upstream calls scale with unique resources × refresh rate, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.

Modules (src/conduit/)

Module	Role
crypto.py	AES-256-GCM encrypted secret storage (master key from CONDUIT_MASTER_KEY_PATH), ported from central's crypto.py
keystore.py	KeyStore — async accessor for the api_keys table (encrypted API keys by alias)
store/payloads.py	PayloadStore / StoredPayload — latest-by-(source, request_key) raw-byte cache with TTL-derived freshness
store/singleflight.py	SingleFlight — in-process coalescer for concurrent misses on the same key (per-process only; does not cross-coalesce across workers)
fetch.py	Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff), consolidating central's per-adapter fetch idiom
sources/__init__.py	SourceRegistry (DB-backed sources table accessor) + PullSource (per-source url_template with {path}/{key} substitution)
broker.py	Broker — orchestrates store + keystore + single-flight + fetch; the get(source_name, request_key) payoff path
app.py	FastAPI app: GET /health, GET /up/{source}/{path}

Repo / deploy model

Pull-based deploy, mirroring central: authored/pushed from a cortex clone, CT 104 pulls via a read-only deploy key (ct104-conduit-deploy) — CT 104 cannot push. conduit-migrate runs forward-only SQL migrations (sql/migrations/*.sql), tracked in a schema_migrations table — simplified port of central's migration runner.

Sources

Sources are DB-backed (the sources table, not a hardcoded dict). app.py loads all enabled sources once at startup and hands them to the Broker — there is no hot-reload yet; a change to the sources table requires a systemctl restart conduit to take effect.

Live source: tomtom_flow_tiles (TomTom Orbis flow tiles), real TomTom key provisioned into Conduit's keystore.

Relationship to central

Conduit was born by harvesting central's proven, decoupled pieces:

• The AES-256-GCM encrypted key store (lifted ~as-is).
• GUI auth/CSRF + schema-reflection "add a source" form patterns (the Conduit GUI itself is still TODO).
• The aiohttp+tenacity fetch idiom.

It deliberately does not carry central's NATS/JetStream, CloudEvents normalization, TimescaleDB event archive, or enrichment pipeline — that complexity is exactly what Conduit sheds. Central is not being retired; it's kept, and partially superseded over time on a per-consumer basis.

navi's traffic tiles were repointed from central to Conduit on 2026-07-14 — see conduit-operations for the cutover and rollback procedure.

Current state (as of 2026-07-14)

• PRs #1–#5 merged to main. #5 added the DB-backed source registry (sources table, seeded with tomtom_flow_tiles).
• First vertical live and durable: navi's traffic tiles are served by conduit.service, byte-identical to central's, confirmed via the real nginx path and payloads table rows.
• Known next steps: a source-management GUI (DB registry is SQL-only so far) and source hot-reload (currently requires a service restart after any sources table change).