echo6-docs/vault/runbooks/conduit-operations.md

title	type	tags	aliases	related	updated
Conduit — Operations Runbook	runbook	mesh		conduit central navi	2026-07-15

Conduit — Operations Runbook

When to use

Deploying a change, adding a pull source, provisioning an API key, or working the navi traffic-tile cutover/rollback for conduit. See conduit for architecture and current state.

Key facts

Item	Value
Host / SSH	utility CT 104 — ssh zvx@100.64.0.12
Deploy dir	/opt/conduit (owned conduit:conduit)
Virtualenv	/opt/conduit/.venv (uv-managed, editable install)
Systemd unit	conduit.service (single-worker — do NOT add --workers, single-flight is per-process)
DB / DSN env	/etc/conduit/conduit.env → CONDUIT_DB_DSN (own conduit DB on the shared Postgres 16)
Master key	/etc/conduit/master.key (CONDUIT_MASTER_KEY_PATH)
Deploy-key model	Pull-only: CT 104 pulls main via read-only deploy key ct104-conduit-deploy; cannot push (mirrors central)
Migration runner	conduit-migrate (forward-only SQL in sql/migrations/*.sql, tracked in schema_migrations)
Bind	0.0.0.0:8010 (widened from loopback for the navi cutover, 2026-07-14)

---

Deploy a change

Run on CT 104 as zvx.

# 1. Pull latest main
sudo -u conduit git -C /opt/conduit fetch origin
sudo -u conduit git -C /opt/conduit checkout main
sudo -u conduit git -C /opt/conduit pull

# 2. Install (editable, picks up code + dep changes)
sudo -u conduit bash -c 'cd /opt/conduit && uv pip install -e ".[dev]"'

# 3. Preview then apply migrations
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate --check'
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate'

# 4. Restart
sudo systemctl restart conduit

# 5. Verify
systemctl is-active conduit
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health   # expect 200
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf   # expect 200, a live tile

---

Fresh deploy from scratch

scripts/provision.sh (repo root) is the canonical from-scratch installer — it stands up one Conduit instance from nothing: Postgres role + database (plain Postgres, no timescaledb/postgis), generated secrets (AES-256 master key + session secret), a uv-managed venv with an editable install, schema migrations, an optional GUI operator, and (optionally) a systemd unit. It takes every name/path/port as an env var or flag — no Matt-specific values are hardcoded, so it's the same script for the live deploy and for a disposable test instance.

Example (adjust for a real deploy — this example matches the throwaway shape used to prove the script works):

DB_NAME=conduit_fresh DB_ROLE=conduit_fresh \
INSTALL_DIR=/tmp/conduit-fresh ENV_FILE=/tmp/conduit-fresh/env \
MASTER_KEY_PATH=/tmp/conduit-fresh/master.key BIND_PORT=8011 \
SYSTEMD=off ADMIN_USER=admin SERVICE_NAME=conduit-fresh \
/tmp/conduit-fresh/scripts/provision.sh

For a real systemd-managed deploy, set SYSTEMD=on plus SERVICE_NAME/SERVICE_USER/SERVICE_GROUP/BIND_HOST/BIND_PORT to taste; the script renders, installs, enables, and starts the unit. Run scripts/provision.sh --help for the full parameter list.

Migrations are schema-only. As of the feat/provisioning schema-only-migrations change, 003_sources.sql creates the sources table but seeds no rows — a fresh install's sources table is empty after conduit-migrate. Sources (and the API keys they reference) are provisioned after the schema exists, via the GUI or conduit-admin/SQL — see "Add a pull source" and "Provision an API key" below. (The live conduit DB's tomtom_flow_tiles row predates this change and is untouched — 003 already applied there and won't re-run.)

---

Add a pull source

Sources are rows in the sources table — there is no hot-reload, app.py loads enabled sources once at startup. Any INSERT/UPDATE/DELETE requires a systemctl restart conduit to take effect.

Columns: name, url_template, api_key_alias, ttl_seconds, header_auth, enabled.

url_template contract: a Python str.format() template filled with two named substitutions —

• {path} — the upstream path only (no query string), taken from the inbound request after the source name and before the first ?.
• {key} — the decrypted API key for api_key_alias, if set.

If the inbound request itself carries a query string, it's appended to the templated URL (& if the template already has its own ?, else ?) rather than substituted in — so caller-supplied params compose instead of colliding into a double ?. If header_auth is true, the key is injected as a header instead of into the URL.

Worked example — tomtom_flow_tiles:

sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
INSERT INTO sources (name, url_template, api_key_alias, ttl_seconds, header_auth)
VALUES (
  '"'"'tomtom_flow_tiles'"'"',
  '"'"'https://api.tomtom.com/maps/orbis/traffic/tile/{path}?key={key}&apiVersion=1'"'"',
  '"'"'tomtom'"'"',
  60,
  false
)
ON CONFLICT (name) DO NOTHING;
"'

sudo systemctl restart conduit

---

Provision an API key

API keys live in Conduit's own AES-256-GCM encrypted keystore (api_keys table), addressed by the alias a source's api_key_alias references — separate from central's keystore, encrypted under Conduit's own master key (/etc/conduit/master.key).

The pattern used for tomtom: the plaintext key was decrypted from central's config.api_keys and re-encrypted under Conduit's master key, in-process, never logged in plaintext. Use the same approach for any key that already lives in central; for a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs).

---

navi traffic-tile cutover (2026-07-14)

navi's /api/traffic/ tiles were repointed from central to conduit. Exact change:

On CT 104 — widen Conduit's bind:

conduit.service's ExecStart uvicorn flag was changed from loopback to mesh-reachable — --host 127.0.0.1 → --host 0.0.0.0 (the --port 8010 is unchanged). Unit backed up first (see Rollback below), then:

# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
#   --host 127.0.0.1  →  --host 0.0.0.0
sudo systemctl daemon-reload
sudo systemctl restart conduit

On recon-vm — rewrite the nginx proxy:

/etc/nginx/sites-available/navi.echo6.co, location ^~ /api/traffic/ block changed to:

location ^~ /api/traffic/ {
    rewrite ^/api/traffic/(.*)$ /up/tomtom_flow_tiles/$1 break;
    proxy_pass http://central.echo6.mesh:8010;
    # (existing tile-cache directives unchanged)
}

Then:

sudo nginx -t
sudo systemctl reload nginx

Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's payloads table.

Rollback

Both pre-cutover backups exist — restore them to revert:

# CT 104 — revert the unit
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
sudo systemctl daemon-reload
sudo systemctl restart conduit

# recon-vm — revert nginx (back to :8000, no rewrite)
sudo cp /etc/nginx/sites-available/navi.echo6.co.bak-precutover /etc/nginx/sites-available/navi.echo6.co
sudo nginx -t
sudo systemctl reload nginx

central must stay running — it's the rollback target.

---

Gotchas

• Single-worker only. Single-flight coalescing is per-process; adding uvicorn --workers breaks the "one upstream call" guarantee. Do not add workers.
• /up rejects HEAD (405) — GET only.
• /up is unauthenticated, like central's auth-exempt tile endpoints. Fine for tiles; revisit before adding sensitive sources.
• No source hot-reload — any sources table change needs systemctl restart conduit.
• central must stay running — it's the rollback target for the navi cutover, not a decommissioned dependency.