auto: docs sync 2026-07-16T00:00:19+00:00
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md
This commit is contained in:
echo6-autocommit
parent
0a2440593b
commit
8ba425146b
6 changed files with 103 additions and 65 deletions
|
|
@ -8,6 +8,7 @@ related:
|
|||
- [[conduit]]
|
||||
- [[central]]
|
||||
- [[navi]]
|
||||
- [[meshai]]
|
||||
updated: 2026-07-15
|
||||
---
|
||||
|
||||
|
|
@ -15,7 +16,7 @@ updated: 2026-07-15
|
|||
|
||||
## When to use
|
||||
|
||||
Deploying a change, adding a pull source, provisioning an API key, or working the navi traffic-tile cutover/rollback for [[conduit]]. See [[conduit]] for architecture and current state.
|
||||
Deploying a change, adding a pull source, provisioning an API key, or reviewing the navi/meshai migration history for [[conduit]]. See [[conduit]] for architecture and current state.
|
||||
|
||||
## Key facts
|
||||
|
||||
|
|
@ -24,12 +25,13 @@ Deploying a change, adding a pull source, provisioning an API key, or working th
|
|||
| Host / SSH | utility CT 104 — `ssh zvx@100.64.0.12` |
|
||||
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
|
||||
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
|
||||
| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight is per-process) |
|
||||
| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight + quota lock are per-process) |
|
||||
| DB / DSN env | `/etc/conduit/conduit.env` → `CONDUIT_DB_DSN` (own `conduit` DB on the shared Postgres 16) |
|
||||
| Master key | `/etc/conduit/master.key` (`CONDUIT_MASTER_KEY_PATH`) |
|
||||
| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push (mirrors central) |
|
||||
| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/*.sql`, tracked in `schema_migrations`) |
|
||||
| Bind | `0.0.0.0:8010` (widened from loopback for the navi cutover, 2026-07-14) |
|
||||
| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push |
|
||||
| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/001`–`007`, tracked in `schema_migrations`) |
|
||||
| Bind | `0.0.0.0:8010` |
|
||||
| Test DB | `conduit_test` (separate from prod `conduit`; a conftest guard hard-aborts if tests ever resolve to prod) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -59,6 +61,8 @@ curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health # expec
|
|||
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf # expect 200, a live tile
|
||||
```
|
||||
|
||||
Note: **code and schema changes still need this deploy + restart cycle.** Only *source-table* changes made through the GUI hot-reload without a restart (see "Add a pull source" below).
|
||||
|
||||
---
|
||||
|
||||
## Fresh deploy from scratch
|
||||
|
|
@ -77,15 +81,17 @@ SYSTEMD=off ADMIN_USER=admin SERVICE_NAME=conduit-fresh \
|
|||
|
||||
For a real systemd-managed deploy, set `SYSTEMD=on` plus `SERVICE_NAME`/`SERVICE_USER`/`SERVICE_GROUP`/`BIND_HOST`/`BIND_PORT` to taste; the script renders, installs, enables, and starts the unit. Run `scripts/provision.sh --help` for the full parameter list.
|
||||
|
||||
**Migrations are schema-only.** As of the `feat/provisioning` schema-only-migrations change, `003_sources.sql` creates the `sources` table but seeds no rows — a fresh install's `sources` table is empty after `conduit-migrate`. Sources (and the API keys they reference) are provisioned **after** the schema exists, via the GUI or `conduit-admin`/SQL — see "Add a pull source" and "Provision an API key" below. (The live `conduit` DB's `tomtom_flow_tiles` row predates this change and is untouched — 003 already applied there and won't re-run.)
|
||||
**Migrations are schema-only.** `003_sources.sql` creates the `sources` table but seeds no rows — a fresh install's `sources` table is empty after `conduit-migrate`. Sources (and the API keys they reference) are provisioned **after** the schema exists, via the GUI or `conduit-admin`/SQL — see "Add a pull source" and "Provision an API key" below.
|
||||
|
||||
The test suite provisions its own `conduit_test` database (via `provision.sh`) and never touches prod — a conftest guard hard-aborts if tests ever resolve to the prod `conduit` DSN.
|
||||
|
||||
---
|
||||
|
||||
## Add a pull source
|
||||
|
||||
Sources are rows in the `sources` table — there is **no hot-reload**, `app.py` loads enabled sources once at startup. Any INSERT/UPDATE/DELETE requires a `systemctl restart conduit` to take effect.
|
||||
Sources are rows in the `sources` table. **The GUI hot-reloads:** a source add/edit/delete made through the management GUI calls `Broker.set_sources()` and takes effect immediately, no restart. A direct out-of-band SQL INSERT/UPDATE/DELETE against `sources` still requires a `systemctl restart conduit` to take effect — prefer the GUI.
|
||||
|
||||
Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`.
|
||||
Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, static `headers` (jsonb, for UA-sensitive upstreams), quota caps (`max_calls_per_day`/`_minute`/`_month`), and retention fields (`retain`, `retention_days`, `poll_interval_seconds`, `poll_path`).
|
||||
|
||||
`url_template` contract: a Python `str.format()` template filled with two named substitutions —
|
||||
|
||||
|
|
@ -94,7 +100,7 @@ Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`,
|
|||
|
||||
If the inbound request itself carries a query string, it's appended to the templated URL (`&` if the template already has its own `?`, else `?`) rather than substituted in — so caller-supplied params compose instead of colliding into a double `?`. If `header_auth` is true, the key is injected as a header instead of into the URL.
|
||||
|
||||
**Worked example — `tomtom_flow_tiles`:**
|
||||
**Worked example — `tomtom_flow_tiles` (via SQL; the GUI form is equivalent and hot-reloads):**
|
||||
|
||||
```bash
|
||||
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
|
||||
|
|
@ -109,26 +115,36 @@ VALUES (
|
|||
ON CONFLICT (name) DO NOTHING;
|
||||
"'
|
||||
|
||||
sudo systemctl restart conduit
|
||||
sudo systemctl restart conduit # only required for this out-of-band SQL path
|
||||
```
|
||||
|
||||
For UA-sensitive upstreams (NWS contact header, Idaho Power WAF, avalanche.org UA), set the static `headers` jsonb column instead of/alongside `header_auth`. For quota-capped upstreams (e.g. TomTom's free plan), set `max_calls_per_day`/`_minute`/`_month`. For a source that should accumulate history, set `retain=true` plus `retention_days` and `poll_interval_seconds` (and `poll_path` if the retained resource differs from the primary path) — the background `Poller` picks it up without any additional step.
|
||||
|
||||
---
|
||||
|
||||
## Provision an API key
|
||||
|
||||
API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references — separate from central's keystore, encrypted under Conduit's own master key (`/etc/conduit/master.key`).
|
||||
API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references. Encrypted under Conduit's own master key (`/etc/conduit/master.key`) — this store is separate from central's (central no longer exists to have one).
|
||||
|
||||
The pattern used for `tomtom`: the plaintext key was decrypted from central's `config.api_keys` and re-encrypted under Conduit's master key, in-process, never logged in plaintext. Use the same approach for any key that already lives in central; for a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs).
|
||||
Keys currently held: `tomtom`, `roads511`, `firms` — all harvested from central before its retirement or provisioned directly. For a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs), or use the GUI's API Keys page.
|
||||
|
||||
---
|
||||
|
||||
## navi traffic-tile cutover (2026-07-14)
|
||||
## meshai migration (2026-07-15) — completed record
|
||||
|
||||
All 13 of [[meshai]]'s native hazard adapters were repointed from their direct upstream to Conduit's `/up/{source}/…`: NWS, SWPC (×4 endpoints), ducting, WFIGS/fires (2 feeds), FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx (pinned `511.idaho.gov/api/wzdx`), and satpass (its Celestrak TLE call).
|
||||
|
||||
meshai kept all of its own transform/fusion/mesh-routing logic — only the fetch step moved to Conduit. Keys for `tomtom`, `roads511`, and `firms` now live only in Conduit's keystore; meshai no longer holds them. Every repoint was a zero-restart hot-reload on meshai's side (`apply_config()`), coordinated via meshai's config API. `traffic` (tomtom_traffic) was migrated last, unblocked by the faithful-4xx-passthrough fix (PR #16) — before that fix, TomTom's legitimate `400 "Point too far from nearest existing segment"` responses were being swallowed/mis-wrapped as 502 by Conduit.
|
||||
|
||||
---
|
||||
|
||||
## navi traffic-tile cutover (2026-07-14) — historical record
|
||||
|
||||
navi's `/api/traffic/` tiles were repointed from [[central]] to [[conduit]]. Exact change:
|
||||
|
||||
**On CT 104 — widen Conduit's bind:**
|
||||
|
||||
`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first (see Rollback below), then:
|
||||
`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first, then:
|
||||
|
||||
```bash
|
||||
# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
|
||||
|
|
@ -158,30 +174,26 @@ sudo systemctl reload nginx
|
|||
|
||||
Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's `payloads` table.
|
||||
|
||||
### Rollback
|
||||
### Rollback target has changed
|
||||
|
||||
Both pre-cutover backups exist — restore them to revert:
|
||||
At the time of this cutover, [[central]] was the rollback target. **Central no longer exists** (retired + DB dropped 2026-07-15). If the navi tile path ever needs to roll back off Conduit, the rollback target is now **direct TomTom** (bypassing both Conduit and the defunct central), not central — a new direct-TomTom nginx config would need to be written; the old pre-cutover nginx backup pointing at central is no longer a valid rollback target.
|
||||
|
||||
```bash
|
||||
# CT 104 — revert the unit
|
||||
# CT 104 — revert the unit if narrowing the bind back to loopback is ever needed
|
||||
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart conduit
|
||||
|
||||
# recon-vm — revert nginx (back to :8000, no rewrite)
|
||||
sudo cp /etc/nginx/sites-available/navi.echo6.co.bak-precutover /etc/nginx/sites-available/navi.echo6.co
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
[[central]] must stay running — it's the rollback target.
|
||||
The recon-vm nginx `.bak-precutover` backup (proxying to central) is now historical only — do not restore it, since central is gone.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Single-worker only.** Single-flight coalescing is per-process; adding uvicorn `--workers` breaks the "one upstream call" guarantee. Do not add workers.
|
||||
- **Single-worker only.** Single-flight coalescing and the quota lock are per-process; adding uvicorn `--workers` breaks both guarantees. Do not add workers.
|
||||
- **`/up` rejects HEAD** (405) — GET only.
|
||||
- **`/up` is unauthenticated**, like central's auth-exempt tile endpoints. Fine for tiles; revisit before adding sensitive sources.
|
||||
- **No source hot-reload** — any `sources` table change needs `systemctl restart conduit`.
|
||||
- **central must stay running** — it's the rollback target for the navi cutover, not a decommissioned dependency.
|
||||
- **`/up` and `/history` are unauthenticated**, mesh-internal, tiles-trust model. The GUI is the authenticated surface (operator auth + CSRF).
|
||||
- **Source hot-reload is GUI-only.** A source change through the GUI takes effect immediately; a direct SQL change to `sources` still needs `systemctl restart conduit`.
|
||||
- **Central is gone.** It is not a rollback target for anything anymore — see "Rollback target has changed" above. Its data is recoverable only from the pi-nas archive dump.
|
||||
- **Retention is opt-in and idle by default.** The `Poller` only does work for sources with `retain=true`; an empty/near-empty `payload_history` is expected unless retention has been turned on for specific sources.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue