echo6-docs/vault/runbooks/conduit-operations.md

title	type	tags	aliases	related	updated
Conduit — Operations Runbook	runbook	mesh		central-deploy-cutover conduit recon-operations deployment ct-runbook	2026-07-16

Conduit — Operations Runbook

When to use

Deploying a change, adding a pull source, provisioning an API key, or reviewing the navi/meshai migration history for conduit. See conduit for architecture and current state.

Key facts

Item	Value
Host / SSH	utility CT 104 — ssh zvx@100.64.0.12
Deploy dir	/opt/conduit (owned conduit:conduit)
Virtualenv	/opt/conduit/.venv (uv-managed, editable install)
Systemd unit	conduit.service (single-worker — do NOT add --workers, single-flight + quota lock are per-process)
DB / DSN env	/etc/conduit/conduit.env → CONDUIT_DB_DSN (own conduit DB on the shared Postgres 16)
Master key	/etc/conduit/master.key (CONDUIT_MASTER_KEY_PATH)
Deploy-key model	Pull-only: CT 104 pulls main via read-only deploy key ct104-conduit-deploy; cannot push
Migration runner	conduit-migrate (forward-only SQL in sql/migrations/001–007, tracked in schema_migrations)
Bind	0.0.0.0:8010
Test DB	conduit_test (separate from prod conduit; a conftest guard hard-aborts if tests ever resolve to prod)

---

Deploy a change

Run on CT 104 as zvx.

# 1. Pull latest main
sudo -u conduit git -C /opt/conduit fetch origin
sudo -u conduit git -C /opt/conduit checkout main
sudo -u conduit git -C /opt/conduit pull

# 2. Install (editable, picks up code + dep changes)
sudo -u conduit bash -c 'cd /opt/conduit && uv pip install -e ".[dev]"'

# 3. Preview then apply migrations
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate --check'
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate'

# 4. Restart
sudo systemctl restart conduit

# 5. Verify
systemctl is-active conduit
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health   # expect 200
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf   # expect 200, a live tile

Note: code and schema changes still need this deploy + restart cycle. Only source-table changes made through the GUI hot-reload without a restart (see "Add a pull source" below).

---

Fresh deploy from scratch

scripts/provision.sh (repo root) is the canonical from-scratch installer — it stands up one conduit instance from nothing: Postgres role + database (plain Postgres, no timescaledb/postgis), generated secrets (AES-256 master key + session secret), a uv-managed venv with an editable install, schema migrations, an optional GUI operator, and (optionally) a systemd unit. It takes every name/path/port as an env var or flag — no Matt-specific values are hardcoded, so it's the same script for the live deploy and for a disposable test instance.

Example (adjust for a real deploy — this example matches the throwaway shape used to prove the script works):

DB_NAME=conduit_fresh DB_ROLE=conduit_fresh \
INSTALL_DIR=/tmp/conduit-fresh ENV_FILE=/tmp/conduit-fresh/env \
MASTER_KEY_PATH=/tmp/conduit-fresh/master.key BIND_PORT=8011 \
SYSTEMD=off ADMIN_USER=admin SERVICE_NAME=conduit-fresh \
/tmp/conduit-fresh/scripts/provision.sh

For a real systemd-managed deploy, set SYSTEMD=on plus SERVICE_NAME/SERVICE_USER/SERVICE_GROUP/BIND_HOST/BIND_PORT to taste; the script renders, installs, enables, and starts the unit. Run scripts/provision.sh --help for the full parameter list.

Migrations are schema-only. 003_sources.sql creates the sources table but seeds no rows — a fresh install's sources table is empty after conduit-migrate. Sources (and the API keys they reference) are provisioned after the schema exists, via the GUI or conduit-admin/SQL — see "Add a pull source" and "Provision an API key" below.

The test suite provisions its own conduit_test database (via provision.sh) and never touches prod — a conftest guard hard-aborts if tests ever resolve to the prod conduit DSN.

---

Add a pull source

Sources are rows in the sources table. The GUI hot-reloads: a source add/edit/delete made through the management GUI calls Broker.set_sources() and takes effect immediately, no restart. A direct out-of-band SQL INSERT/UPDATE/DELETE against sources still requires a systemctl restart conduit to take effect — prefer the GUI.

Columns: name, url_template, api_key_alias, ttl_seconds, header_auth, enabled, static headers (jsonb, for UA-sensitive upstreams), quota caps (max_calls_per_day/_minute/_month), and retention fields (retain, retention_days, poll_interval_seconds, poll_path).

url_template contract: a Python str.format() template filled with two named substitutions —

• {path} — the upstream path only (no query string), taken from the inbound request after the source name and before the first ?.
• {key} — the decrypted API key for api_key_alias, if set.

If the inbound request itself carries a query string, it's appended to the templated URL (& if the template already has its own ?, else ?) rather than substituted in — so caller-supplied params compose instead of colliding into a double ?. If header_auth is true, the key is injected as a header instead of into the URL.

Worked example — tomtom_flow_tiles (via SQL; the GUI form is equivalent and hot-reloads):

sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
INSERT INTO sources (name, url_template, api_key_alias, ttl_seconds, header_auth)
VALUES (
  '"'"'tomtom_flow_tiles'"'"',
  '"'"'https://api.tomtom.com/maps/orbis/traffic/tile/{path}?key={key}&apiVersion=1'"'"',
  '"'"'tomtom'"'"',
  60,
  false
)
ON CONFLICT (name) DO NOTHING;
"'

sudo systemctl restart conduit   # only required for this out-of-band SQL path

For UA-sensitive upstreams (NWS contact header, Idaho Power WAF, avalanche.org UA), set the static headers jsonb column instead of/alongside header_auth. For quota-capped upstreams (e.g. TomTom's free plan), set max_calls_per_day/_minute/_month. For a source that should accumulate history, set retain=true plus retention_days and poll_interval_seconds (and poll_path if the retained resource differs from the primary path) — the background Poller picks it up without any additional step.

---

Provision an API key

API keys live in Conduit's own AES-256-GCM encrypted keystore (api_keys table), addressed by the alias a source's api_key_alias references. Encrypted under Conduit's own master key (/etc/conduit/master.key) — this store is separate from central's (central no longer exists to have one).

Keys currently held: tomtom, roads511, firms — all harvested from central before its retirement or provisioned directly. For a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs), or use the GUI's API Keys page.

---

meshai migration (2026-07-15) — completed record

All 13 of meshai's native hazard adapters were repointed from their direct upstream to Conduit's /up/{source}/…: NWS, SWPC (×4 endpoints), ducting, WFIGS/fires (2 feeds), FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx (pinned 511.idaho.gov/api/wzdx), and satpass (its Celestrak TLE call).

meshai kept all of its own transform/fusion/mesh-routing logic — only the fetch step moved to Conduit. Keys for tomtom, roads511, and firms now live only in Conduit's keystore; meshai no longer holds them. Every repoint was a zero-restart hot-reload on meshai's side (apply_config()), coordinated via meshai's config API. traffic (tomtom_traffic) was migrated last, unblocked by the faithful-4xx-passthrough fix (PR #16) — before that fix, TomTom's legitimate 400 "Point too far from nearest existing segment" responses were being swallowed/mis-wrapped as 502 by Conduit.

---

navi traffic-tile cutover (2026-07-14) — historical record

navi's /api/traffic/ tiles were repointed from central to conduit. Exact change:

On CT 104 — widen Conduit's bind:

conduit.service's ExecStart uvicorn flag was changed from loopback to mesh-reachable — --host 127.0.0.1 → --host 0.0.0.0 (the --port 8010 is unchanged). Unit backed up first, then:

# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
#   --host 127.0.0.1  →  --host 0.0.0.0
sudo systemctl daemon-reload
sudo systemctl restart conduit

On recon-vm — rewrite the nginx proxy:

/etc/nginx/sites-available/navi.echo6.co, location ^~ /api/traffic/ block changed to:

location ^~ /api/traffic/ {
    rewrite ^/api/traffic/(.*)$ /up/tomtom_flow_tiles/$1 break;
    proxy_pass http://central.echo6.mesh:8010;
    # (existing tile-cache directives unchanged)
}

Then:

sudo nginx -t
sudo systemctl reload nginx

Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's payloads table.

Rollback target has changed

At the time of this cutover, central was the rollback target. Central no longer exists (retired + DB dropped 2026-07-15). If the navi tile path ever needs to roll back off Conduit, the rollback target is now direct TomTom (bypassing both Conduit and the defunct central), not central — a new direct-TomTom nginx config would need to be written; the old pre-cutover nginx backup pointing at central is no longer a valid rollback target.

# CT 104 — revert the unit if narrowing the bind back to loopback is ever needed
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
sudo systemctl daemon-reload
sudo systemctl restart conduit

The recon-vm nginx .bak-precutover backup (proxying to central) is now historical only — do not restore it, since central is gone.

---

Gotchas

• Single-worker only. Single-flight coalescing and the quota lock are per-process; adding uvicorn --workers breaks both guarantees. Do not add workers.
• /up rejects HEAD (405) — GET only.
• /up and /history are unauthenticated, mesh-internal, tiles-trust model. The GUI is the authenticated surface (operator auth + CSRF).
• Source hot-reload is GUI-only. A source change through the GUI takes effect immediately; a direct SQL change to sources still needs systemctl restart conduit.
• Central is gone. It is not a rollback target for anything anymore — see "Rollback target has changed" above. Its data is recoverable only from the pi-nas archive dump.
• Retention is opt-in and idle by default. The Poller only does work for sources with retain=true; an empty/near-empty payload_history is expected unless retention has been turned on for specific sources.