auto: docs sync 2026-07-14T18:00:17+00:00
Files changed: credentials engine/lint-report.md vault/.obsidian/workspace.json vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/conduit-operations.md
This commit is contained in:
echo6-autocommit
parent
3efdd560d5
commit
e9271ee987
6 changed files with 276 additions and 10 deletions
167
vault/runbooks/conduit-operations.md
Normal file
167
vault/runbooks/conduit-operations.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
---
|
||||
title: "Conduit — Operations Runbook"
|
||||
type: runbook
|
||||
tags:
|
||||
- mesh
|
||||
aliases: []
|
||||
related:
|
||||
- [[conduit]]
|
||||
- [[central]]
|
||||
- [[navi]]
|
||||
updated: 2026-07-14
|
||||
---
|
||||
|
||||
# Conduit — Operations Runbook
|
||||
|
||||
## When to use
|
||||
|
||||
Deploying a change, adding a pull source, provisioning an API key, or working the navi traffic-tile cutover/rollback for [[conduit]]. See [[conduit]] for architecture and current state.
|
||||
|
||||
## Key facts
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Host / SSH | utility CT 104 — `ssh zvx@100.64.0.12` |
|
||||
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
|
||||
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
|
||||
| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight is per-process) |
|
||||
| DB / DSN env | `/etc/conduit/conduit.env` → `CONDUIT_DB_DSN` (own `conduit` DB on the shared Postgres 16) |
|
||||
| Master key | `/etc/conduit/master.key` (`CONDUIT_MASTER_KEY_PATH`) |
|
||||
| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push (mirrors central) |
|
||||
| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/*.sql`, tracked in `schema_migrations`) |
|
||||
| Bind | `0.0.0.0:8010` (widened from loopback for the navi cutover, 2026-07-14) |
|
||||
|
||||
---
|
||||
|
||||
## Deploy a change
|
||||
|
||||
Run on CT 104 as `zvx`.
|
||||
|
||||
```bash
|
||||
# 1. Pull latest main
|
||||
sudo -u conduit git -C /opt/conduit fetch origin
|
||||
sudo -u conduit git -C /opt/conduit checkout main
|
||||
sudo -u conduit git -C /opt/conduit pull
|
||||
|
||||
# 2. Install (editable, picks up code + dep changes)
|
||||
sudo -u conduit bash -c 'cd /opt/conduit && uv pip install -e ".[dev]"'
|
||||
|
||||
# 3. Preview then apply migrations
|
||||
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate --check'
|
||||
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate'
|
||||
|
||||
# 4. Restart
|
||||
sudo systemctl restart conduit
|
||||
|
||||
# 5. Verify
|
||||
systemctl is-active conduit
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health # expect 200
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf # expect 200, a live tile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Add a pull source
|
||||
|
||||
Sources are rows in the `sources` table — there is **no hot-reload**, `app.py` loads enabled sources once at startup. Any INSERT/UPDATE/DELETE requires a `systemctl restart conduit` to take effect.
|
||||
|
||||
Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`.
|
||||
|
||||
`url_template` contract: a Python `str.format()` template filled with two named substitutions —
|
||||
|
||||
- `{path}` — the upstream path only (no query string), taken from the inbound request after the source name and before the first `?`.
|
||||
- `{key}` — the decrypted API key for `api_key_alias`, if set.
|
||||
|
||||
If the inbound request itself carries a query string, it's appended to the templated URL (`&` if the template already has its own `?`, else `?`) rather than substituted in — so caller-supplied params compose instead of colliding into a double `?`. If `header_auth` is true, the key is injected as a header instead of into the URL.
|
||||
|
||||
**Worked example — `tomtom_flow_tiles`:**
|
||||
|
||||
```bash
|
||||
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
|
||||
INSERT INTO sources (name, url_template, api_key_alias, ttl_seconds, header_auth)
|
||||
VALUES (
|
||||
'"'"'tomtom_flow_tiles'"'"',
|
||||
'"'"'https://api.tomtom.com/maps/orbis/traffic/tile/{path}?key={key}&apiVersion=1'"'"',
|
||||
'"'"'tomtom'"'"',
|
||||
60,
|
||||
false
|
||||
)
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
"'
|
||||
|
||||
sudo systemctl restart conduit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provision an API key
|
||||
|
||||
API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references — separate from central's keystore, encrypted under Conduit's own master key (`/etc/conduit/master.key`).
|
||||
|
||||
The pattern used for `tomtom`: the plaintext key was decrypted from central's `config.api_keys` and re-encrypted under Conduit's master key, in-process, never logged in plaintext. Use the same approach for any key that already lives in central; for a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs).
|
||||
|
||||
---
|
||||
|
||||
## navi traffic-tile cutover (2026-07-14)
|
||||
|
||||
navi's `/api/traffic/` tiles were repointed from [[central]] to [[conduit]]. Exact change:
|
||||
|
||||
**On CT 104 — widen Conduit's bind:**
|
||||
|
||||
`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first (see Rollback below), then:
|
||||
|
||||
```bash
|
||||
# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
|
||||
# --host 127.0.0.1 → --host 0.0.0.0
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart conduit
|
||||
```
|
||||
|
||||
**On recon-vm — rewrite the nginx proxy:**
|
||||
|
||||
`/etc/nginx/sites-available/navi.echo6.co`, `location ^~ /api/traffic/` block changed to:
|
||||
|
||||
```nginx
|
||||
location ^~ /api/traffic/ {
|
||||
rewrite ^/api/traffic/(.*)$ /up/tomtom_flow_tiles/$1 break;
|
||||
proxy_pass http://central.echo6.mesh:8010;
|
||||
# (existing tile-cache directives unchanged)
|
||||
}
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's `payloads` table.
|
||||
|
||||
### Rollback
|
||||
|
||||
Both pre-cutover backups exist — restore them to revert:
|
||||
|
||||
```bash
|
||||
# CT 104 — revert the unit
|
||||
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart conduit
|
||||
|
||||
# recon-vm — revert nginx (back to :8000, no rewrite)
|
||||
sudo cp /etc/nginx/sites-available/navi.echo6.co.bak-precutover /etc/nginx/sites-available/navi.echo6.co
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
[[central]] must stay running — it's the rollback target.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Single-worker only.** Single-flight coalescing is per-process; adding uvicorn `--workers` breaks the "one upstream call" guarantee. Do not add workers.
|
||||
- **`/up` rejects HEAD** (405) — GET only.
|
||||
- **`/up` is unauthenticated**, like central's auth-exempt tile endpoints. Fine for tiles; revisit before adding sensitive sources.
|
||||
- **No source hot-reload** — any `sources` table change needs `systemctl restart conduit`.
|
||||
- **central must stay running** — it's the rollback target for the navi cutover, not a decommissioned dependency.
|
||||
Loading…
Add table
Add a link
Reference in a new issue