echo6-docs/vault/docs/software/conduit.md

---
title: Conduit — Raw-API Broker
type: reference
tags:
  - mesh
aliases: []
related:
  - [[conduit-operations]]
  - [[central]]
  - [[navi]]
  - [[caddy]]
updated: 2026-07-14
---
# Conduit — Raw-API Broker

## Overview

Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch an upstream API **once**, store the raw response **as-is** (byte-for-byte, no normalization), and serve it to **many** internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."

- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010` as of 2026-07-14 for [[navi]] to reach it)
- **Host:** utility CT 104 (unprivileged Ubuntu LXC), co-resident with [[central]]
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)

## Host

| Attribute | Value |
|-----------|-------|
| Container | utility CT 104 |
| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh:8010 |
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
| Master key | `/etc/conduit/master.key` |
| Systemd unit | `conduit.service` (enabled, single-worker, `Wants=postgresql@16-main`) |
| Bind | `0.0.0.0:8010` |

## Architecture — the "source" model

Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction.

- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) or **push** (an app POSTs a raw payload in — planned, not yet exercised).
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL. This is a **cache, not a historical archive** — unlike central's TimescaleDB event archive.
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).

**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.

## Modules (`src/conduit/`)

| Module | Role |
|--------|------|
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`), ported from central's `crypto.py` |
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only; does not cross-coalesce across workers) |
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff), consolidating central's per-adapter fetch idiom |
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template` with `{path}`/`{key}` substitution) |
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch; the `get(source_name, request_key)` payoff path |
| `app.py` | FastAPI app: `GET /health`, `GET /up/{source}/{path}` |

## Repo / deploy model

Pull-based deploy, mirroring central: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table — simplified port of central's migration runner.

## Sources

Sources are DB-backed (the `sources` table, not a hardcoded dict). `app.py` loads all enabled sources once at startup and hands them to the `Broker` — **there is no hot-reload yet**; a change to the `sources` table requires a `systemctl restart conduit` to take effect.

Live source: `tomtom_flow_tiles` (TomTom Orbis flow tiles), real TomTom key provisioned into Conduit's keystore.

## Relationship to [[central]]

Conduit was born by harvesting central's proven, decoupled pieces:

- The AES-256-GCM encrypted key store (lifted ~as-is).
- GUI auth/CSRF + schema-reflection "add a source" form patterns (the Conduit GUI itself is still TODO).
- The aiohttp+tenacity fetch idiom.

It deliberately does **not** carry central's NATS/JetStream, CloudEvents normalization, TimescaleDB event archive, or enrichment pipeline — that complexity is exactly what Conduit sheds. Central is not being retired; it's kept, and partially superseded over time on a per-consumer basis.

**[[navi]]'s traffic tiles were repointed from central to Conduit on 2026-07-14** — see [[conduit-operations]] for the cutover and rollback procedure.

## Current state (as of 2026-07-14)

- PRs #1–#5 merged to `main`. #5 added the DB-backed source registry (`sources` table, seeded with `tomtom_flow_tiles`).
- First vertical live and durable: navi's traffic tiles are served by `conduit.service`, byte-identical to central's, confirmed via the real nginx path and `payloads` table rows.
- Known next steps: a source-management **GUI** (DB registry is SQL-only so far) and **source hot-reload** (currently requires a service restart after any `sources` table change).