auto: docs sync 2026-07-16T00:00:19+00:00
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md
This commit is contained in:
parent
0a2440593b
commit
8ba425146b
6 changed files with 103 additions and 65 deletions
|
|
@ -1,6 +1,6 @@
|
||||||
# Vault Lint Report
|
# Vault Lint Report
|
||||||
|
|
||||||
Generated: 2026-07-15T12:00:17Z | Docs scanned: 107 | Elapsed: 0.0s
|
Generated: 2026-07-15T18:00:15Z | Docs scanned: 107 | Elapsed: 0.0s
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
|
|
|
||||||
6
vault/.obsidian/workspace.json
vendored
6
vault/.obsidian/workspace.json
vendored
|
|
@ -199,6 +199,9 @@
|
||||||
},
|
},
|
||||||
"active": "17bd4a6166f789d0",
|
"active": "17bd4a6166f789d0",
|
||||||
"lastOpenFiles": [
|
"lastOpenFiles": [
|
||||||
|
"runbooks/central-deploy-cutover.md.tmp.1002651.75f56e212940",
|
||||||
|
"docs/software/central.md.tmp.1002651.11f9fdeacea8",
|
||||||
|
"runbooks/conduit-operations.md.tmp.1002651.8bca4793e607",
|
||||||
"runbooks/add-peertube-channel.md.tmp.767109.d14b6e68779c",
|
"runbooks/add-peertube-channel.md.tmp.767109.d14b6e68779c",
|
||||||
"runbooks/add-peertube-channel.md.tmp.767109.366b5b6ebfe2",
|
"runbooks/add-peertube-channel.md.tmp.767109.366b5b6ebfe2",
|
||||||
"runbooks/peertube-remote-runner.md.tmp.767109.9cf753daf1b2",
|
"runbooks/peertube-remote-runner.md.tmp.767109.9cf753daf1b2",
|
||||||
|
|
@ -207,9 +210,6 @@
|
||||||
"docs/hardware/environment.md.tmp.767109.4e9ed4494a50",
|
"docs/hardware/environment.md.tmp.767109.4e9ed4494a50",
|
||||||
"docs/hardware/environment.md.tmp.767109.3462012af99e",
|
"docs/hardware/environment.md.tmp.767109.3462012af99e",
|
||||||
"docs/hardware/environment.md.tmp.767109.55addc2a6c2b",
|
"docs/hardware/environment.md.tmp.767109.55addc2a6c2b",
|
||||||
"docs/hardware/environment.md.tmp.767109.5759a0dcf2a1",
|
|
||||||
"runbooks/conduit-operations.md.tmp.1002651.4b545d0c7965",
|
|
||||||
"runbooks/conduit-operations.md.tmp.1002651.015f0192a793",
|
|
||||||
"runbooks/conduit-operations.md",
|
"runbooks/conduit-operations.md",
|
||||||
"docs/software/conduit.md",
|
"docs/software/conduit.md",
|
||||||
"archive/projects/vaultwarden-plan.md",
|
"archive/projects/vaultwarden-plan.md",
|
||||||
|
|
|
||||||
|
|
@ -11,10 +11,12 @@ related:
|
||||||
- [[fleet-platform-baseline]]
|
- [[fleet-platform-baseline]]
|
||||||
- [[caddy]]
|
- [[caddy]]
|
||||||
- [[conduit]]
|
- [[conduit]]
|
||||||
updated: 2026-07-14
|
updated: 2026-07-15
|
||||||
---
|
---
|
||||||
# central — Data-Hub Spine
|
# central — Data-Hub Spine
|
||||||
|
|
||||||
|
> **RETIRED + DROPPED (2026-07-15).** Central has been replaced by [[conduit]]. Its app services were stopped and disabled 2026-07-14 (zero live consumers remained); on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed), and the shared Postgres instance was cleaned back to plain (TimescaleDB removed). Central is recoverable only from the pi-nas archive. **Everything below this point is historical** — it describes how central worked while it was live, and is kept for reference only.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for [[navi]] traffic tiles and related situational-awareness feeds.
|
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for [[navi]] traffic tiles and related situational-awareness feeds.
|
||||||
|
|
|
||||||
|
|
@ -9,18 +9,20 @@ related:
|
||||||
- [[central]]
|
- [[central]]
|
||||||
- [[navi]]
|
- [[navi]]
|
||||||
- [[caddy]]
|
- [[caddy]]
|
||||||
updated: 2026-07-14
|
- [[meshai]]
|
||||||
|
updated: 2026-07-15
|
||||||
---
|
---
|
||||||
# Conduit — Raw-API Broker
|
# Conduit — Raw-API Broker
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch an upstream API **once**, store the raw response **as-is** (byte-for-byte, no normalization), and serve it to **many** internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
|
Conduit is the fleet's **single egress point** for every external API feed. It fetches an upstream once, stores the raw response **as-is** (byte-for-byte, no normalization), and fans it out to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
|
||||||
|
|
||||||
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010` as of 2026-07-14 for [[navi]] to reach it)
|
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010`)
|
||||||
- **Host:** utility CT 104 (unprivileged Ubuntu LXC), co-resident with [[central]]
|
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
|
||||||
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
|
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
|
||||||
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
|
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
|
||||||
|
- **Consumers:** [[navi]]'s traffic tiles (`tomtom_flow_tiles`) and all 13 of [[meshai]]'s hazard adapters (migrated 2026-07-15 — see [[conduit-operations]])
|
||||||
|
|
||||||
## Host
|
## Host
|
||||||
|
|
||||||
|
|
@ -33,58 +35,78 @@ Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch
|
||||||
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
|
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
|
||||||
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
|
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
|
||||||
| Master key | `/etc/conduit/master.key` |
|
| Master key | `/etc/conduit/master.key` |
|
||||||
| Systemd unit | `conduit.service` (enabled, single-worker, `Wants=postgresql@16-main`) |
|
| Systemd unit | `conduit.service` (enabled, **single uvicorn worker only** — never add `--workers`; single-flight + quota lock are per-process) |
|
||||||
| Bind | `0.0.0.0:8010` |
|
| Bind | `0.0.0.0:8010` |
|
||||||
|
|
||||||
## Architecture — the "source" model
|
## Architecture — the "source" model
|
||||||
|
|
||||||
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction.
|
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. The `sources` table holds ~21 rows.
|
||||||
|
|
||||||
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) or **push** (an app POSTs a raw payload in — planned, not yet exercised).
|
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) is live. **push** (an app POSTs a raw payload in) is planned, not yet exercised.
|
||||||
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL. This is a **cache, not a historical archive** — unlike central's TimescaleDB event archive.
|
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL — a cache, not by itself a historical archive (see Retention below for the opt-in exception).
|
||||||
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
|
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
|
||||||
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
|
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
|
||||||
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).
|
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).
|
||||||
|
|
||||||
**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.
|
**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.
|
||||||
|
|
||||||
|
### Per-source columns
|
||||||
|
|
||||||
|
`name`, `url_template` (`{path}`/`{key}` substitution; an inbound query string appends rather than colliding), `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, plus:
|
||||||
|
|
||||||
|
- **static `headers`** (jsonb) — for UA-sensitive feeds: NWS needs a contact User-Agent, Idaho Power WAFs non-browser UAs, avalanche.org sends a UA.
|
||||||
|
- **quota caps** — `max_calls_per_day` / `max_calls_per_minute` / `max_calls_per_month`.
|
||||||
|
- **retention** — `retain`, `retention_days`, `poll_interval_seconds`, `poll_path`.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
- **Single-flight coalescing** — concurrent misses for the same `(source, request_key)` → one upstream call.
|
||||||
|
- **Quota guard** (`quota.py`) — per-source day/minute/month caps enforced against a durable `upstream_calls` log, atomic per-source `asyncio.Lock` (single-process). Protects free-tier upstreams (e.g. TomTom's free plan).
|
||||||
|
- **Serve-stale** — on quota-block, upstream 429, or 5xx/transport failure, Conduit serves the last-known-good cached copy instead of failing the caller (`X-Conduit-Stale: 1`).
|
||||||
|
- **Faithful 4xx passthrough** (PR #16, 2026-07-15) — a genuine upstream 4xx *except* 429 (e.g. TomTom flow's `400 "Point too far from nearest existing segment"`) is returned to the caller with its real status + body, uncached, not wrapped as a 502. Only 429/5xx/transport failures fall back to serve-stale-or-502. This unblocked the last meshai adapter (traffic) to migrate.
|
||||||
|
- **Hot-reload of sources** — a source add/edit/delete made through the GUI calls `Broker.set_sources()` and takes effect with no restart. A direct out-of-band SQL change to `sources` still needs `systemctl restart conduit` — the GUI path is the live one.
|
||||||
|
- **Retention engine** (`poller.py` + `store/history.py`) — opt-in per source (`retain=true`). A background `Poller` fetches retained sources on their `poll_interval_seconds` and appends changed raw payloads to `payload_history` (append-on-change, sha256-dedup). Idle by construction when nothing is retained. Read back via `GET /history/{source}` and `/history/{source}/{id}/body` (unauthenticated, mesh-internal). Purpose: accumulate raw feeds for later forecast/trend models.
|
||||||
|
- **Management GUI** (`gui/`) — built in meshai's visual language. argon2 operator auth + two-tier CSRF + schema-reflection forms harvested from central. Pages: sources CRUD (incl. headers/quota/retention fields), API keys, a Cmd-K command palette. Admin operator provisioned. `/up` and `/history` stay unauthenticated (mesh-internal, tiles-trust model); the GUI is the authenticated surface.
|
||||||
|
- **Keystore** (`keystore.py`, `crypto.py`) — AES-256-GCM `api_keys` by alias, under Conduit's own master key. Holds `tomtom`, `roads511`, and `firms` keys (harvested from central / provisioned) — meshai no longer holds any of these itself.
|
||||||
|
|
||||||
## Modules (`src/conduit/`)
|
## Modules (`src/conduit/`)
|
||||||
|
|
||||||
| Module | Role |
|
| Module | Role |
|
||||||
|--------|------|
|
|--------|------|
|
||||||
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`), ported from central's `crypto.py` |
|
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`) |
|
||||||
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
|
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
|
||||||
|
| `config.py` | App configuration loading |
|
||||||
|
| `migrate.py` | Forward-only SQL migration runner (console script `conduit-migrate`) |
|
||||||
|
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff) |
|
||||||
|
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch + quota + serve-stale; the `get(source_name, request_key)` payoff path |
|
||||||
|
| `quota.py` | Per-source day/minute/month call-quota enforcement against `upstream_calls` |
|
||||||
|
| `poller.py` | Background retention poller for `retain=true` sources |
|
||||||
|
| `admin_cli.py` | `conduit-admin` command-line administration |
|
||||||
|
| `app.py` | FastAPI app wiring all routes |
|
||||||
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
|
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
|
||||||
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only; does not cross-coalesce across workers) |
|
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only) |
|
||||||
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff), consolidating central's per-adapter fetch idiom |
|
| `store/history.py` | `payload_history` accessor for the retention engine |
|
||||||
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template` with `{path}`/`{key}` substitution) |
|
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template`, headers, quota, retention fields) |
|
||||||
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch; the `get(source_name, request_key)` payoff path |
|
| `gui/{auth,csrf,deps,routes}` + templates/static | Management GUI: operator auth, CSRF, forms, command palette |
|
||||||
| `app.py` | FastAPI app: `GET /health`, `GET /up/{source}/{path}` |
|
|
||||||
|
**Endpoints:** `GET /health`, `GET /up/{source}/{path}`, `GET /history/{source}`, `GET /history/{source}/{id}/body`, plus the GUI routes.
|
||||||
|
|
||||||
|
**Migrations:** `sql/migrations/001`–`007` (schema-only, no seeds), run by `conduit-migrate`, tracked in `schema_migrations`.
|
||||||
|
|
||||||
## Repo / deploy model
|
## Repo / deploy model
|
||||||
|
|
||||||
Pull-based deploy, mirroring central: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table — simplified port of central's migration runner.
|
Pull-based deploy: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table.
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
Sources are DB-backed (the `sources` table, not a hardcoded dict). `app.py` loads all enabled sources once at startup and hands them to the `Broker` — **there is no hot-reload yet**; a change to the `sources` table requires a `systemctl restart conduit` to take effect.
|
|
||||||
|
|
||||||
Live source: `tomtom_flow_tiles` (TomTom Orbis flow tiles), real TomTom key provisioned into Conduit's keystore.
|
|
||||||
|
|
||||||
## Relationship to [[central]]
|
## Relationship to [[central]]
|
||||||
|
|
||||||
Conduit was born by harvesting central's proven, decoupled pieces:
|
**Central is retired and dropped (2026-07-15) — Conduit replaced it.** Central's app services were stopped and disabled 2026-07-14 (zero live consumers remained), then on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed); the shared Postgres instance was cleaned back to plain (TimescaleDB removed from `shared_preload_libraries`). Central is recoverable only from the pi-nas archive. Conduit's own `conduit` DB shares that same Postgres instance, which was never stopped.
|
||||||
|
|
||||||
- The AES-256-GCM encrypted key store (lifted ~as-is).
|
Conduit was born by harvesting central's proven, decoupled pieces — the AES-256-GCM encrypted key store, the GUI auth/CSRF + schema-reflection form patterns, and the aiohttp+tenacity fetch idiom — while deliberately shedding central's NATS/JetStream, CloudEvents normalization, and enrichment pipeline.
|
||||||
- GUI auth/CSRF + schema-reflection "add a source" form patterns (the Conduit GUI itself is still TODO).
|
|
||||||
- The aiohttp+tenacity fetch idiom.
|
|
||||||
|
|
||||||
It deliberately does **not** carry central's NATS/JetStream, CloudEvents normalization, TimescaleDB event archive, or enrichment pipeline — that complexity is exactly what Conduit sheds. Central is not being retired; it's kept, and partially superseded over time on a per-consumer basis.
|
## Current state (as of 2026-07-15)
|
||||||
|
|
||||||
**[[navi]]'s traffic tiles were repointed from central to Conduit on 2026-07-14** — see [[conduit-operations]] for the cutover and rollback procedure.
|
- Single egress point for the whole fleet: [[navi]]'s traffic tiles plus all 13 [[meshai]] hazard adapters (NWS, SWPC ×4, ducting, WFIGS/fires ×2, FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx, satpass) — see [[conduit-operations]] for the migration record.
|
||||||
|
- `sources` table holds ~21 rows.
|
||||||
## Current state (as of 2026-07-14)
|
- Quota guard, serve-stale, faithful-4xx-passthrough, hot-reload, the retention engine, and the management GUI are all live (not planned).
|
||||||
|
- Keystore holds `tomtom`, `roads511`, `firms` — meshai no longer holds any of these keys directly.
|
||||||
- PRs #1–#5 merged to `main`. #5 added the DB-backed source registry (`sources` table, seeded with `tomtom_flow_tiles`).
|
- Central is gone; it is no longer a rollback target for anything.
|
||||||
- First vertical live and durable: navi's traffic tiles are served by `conduit.service`, byte-identical to central's, confirmed via the real nginx path and `payloads` table rows.
|
|
||||||
- Known next steps: a source-management **GUI** (DB registry is SQL-only so far) and **source hot-reload** (currently requires a service restart after any `sources` table change).
|
|
||||||
|
|
|
||||||
|
|
@ -10,11 +10,13 @@ related:
|
||||||
- [[lxc-service-migration]]
|
- [[lxc-service-migration]]
|
||||||
- [[deployment]]
|
- [[deployment]]
|
||||||
- [[syncthing-add-node]]
|
- [[syncthing-add-node]]
|
||||||
updated: 2026-07-13
|
updated: 2026-07-15
|
||||||
---
|
---
|
||||||
|
|
||||||
# central — Deploy & Cutover Runbook
|
# central — Deploy & Cutover Runbook
|
||||||
|
|
||||||
|
> **HISTORICAL** — central was retired + dropped 2026-07-15 (see [[central]] / [[conduit]]); this runbook is kept for reference only.
|
||||||
|
|
||||||
## Overview / when to use
|
## Overview / when to use
|
||||||
|
|
||||||
`central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state.
|
`central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state.
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,7 @@ related:
|
||||||
- [[conduit]]
|
- [[conduit]]
|
||||||
- [[central]]
|
- [[central]]
|
||||||
- [[navi]]
|
- [[navi]]
|
||||||
|
- [[meshai]]
|
||||||
updated: 2026-07-15
|
updated: 2026-07-15
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -15,7 +16,7 @@ updated: 2026-07-15
|
||||||
|
|
||||||
## When to use
|
## When to use
|
||||||
|
|
||||||
Deploying a change, adding a pull source, provisioning an API key, or working the navi traffic-tile cutover/rollback for [[conduit]]. See [[conduit]] for architecture and current state.
|
Deploying a change, adding a pull source, provisioning an API key, or reviewing the navi/meshai migration history for [[conduit]]. See [[conduit]] for architecture and current state.
|
||||||
|
|
||||||
## Key facts
|
## Key facts
|
||||||
|
|
||||||
|
|
@ -24,12 +25,13 @@ Deploying a change, adding a pull source, provisioning an API key, or working th
|
||||||
| Host / SSH | utility CT 104 — `ssh zvx@100.64.0.12` |
|
| Host / SSH | utility CT 104 — `ssh zvx@100.64.0.12` |
|
||||||
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
|
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
|
||||||
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
|
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
|
||||||
| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight is per-process) |
|
| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight + quota lock are per-process) |
|
||||||
| DB / DSN env | `/etc/conduit/conduit.env` → `CONDUIT_DB_DSN` (own `conduit` DB on the shared Postgres 16) |
|
| DB / DSN env | `/etc/conduit/conduit.env` → `CONDUIT_DB_DSN` (own `conduit` DB on the shared Postgres 16) |
|
||||||
| Master key | `/etc/conduit/master.key` (`CONDUIT_MASTER_KEY_PATH`) |
|
| Master key | `/etc/conduit/master.key` (`CONDUIT_MASTER_KEY_PATH`) |
|
||||||
| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push (mirrors central) |
|
| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push |
|
||||||
| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/*.sql`, tracked in `schema_migrations`) |
|
| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/001`–`007`, tracked in `schema_migrations`) |
|
||||||
| Bind | `0.0.0.0:8010` (widened from loopback for the navi cutover, 2026-07-14) |
|
| Bind | `0.0.0.0:8010` |
|
||||||
|
| Test DB | `conduit_test` (separate from prod `conduit`; a conftest guard hard-aborts if tests ever resolve to prod) |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -59,6 +61,8 @@ curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health # expec
|
||||||
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf # expect 200, a live tile
|
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf # expect 200, a live tile
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Note: **code and schema changes still need this deploy + restart cycle.** Only *source-table* changes made through the GUI hot-reload without a restart (see "Add a pull source" below).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Fresh deploy from scratch
|
## Fresh deploy from scratch
|
||||||
|
|
@ -77,15 +81,17 @@ SYSTEMD=off ADMIN_USER=admin SERVICE_NAME=conduit-fresh \
|
||||||
|
|
||||||
For a real systemd-managed deploy, set `SYSTEMD=on` plus `SERVICE_NAME`/`SERVICE_USER`/`SERVICE_GROUP`/`BIND_HOST`/`BIND_PORT` to taste; the script renders, installs, enables, and starts the unit. Run `scripts/provision.sh --help` for the full parameter list.
|
For a real systemd-managed deploy, set `SYSTEMD=on` plus `SERVICE_NAME`/`SERVICE_USER`/`SERVICE_GROUP`/`BIND_HOST`/`BIND_PORT` to taste; the script renders, installs, enables, and starts the unit. Run `scripts/provision.sh --help` for the full parameter list.
|
||||||
|
|
||||||
**Migrations are schema-only.** As of the `feat/provisioning` schema-only-migrations change, `003_sources.sql` creates the `sources` table but seeds no rows — a fresh install's `sources` table is empty after `conduit-migrate`. Sources (and the API keys they reference) are provisioned **after** the schema exists, via the GUI or `conduit-admin`/SQL — see "Add a pull source" and "Provision an API key" below. (The live `conduit` DB's `tomtom_flow_tiles` row predates this change and is untouched — 003 already applied there and won't re-run.)
|
**Migrations are schema-only.** `003_sources.sql` creates the `sources` table but seeds no rows — a fresh install's `sources` table is empty after `conduit-migrate`. Sources (and the API keys they reference) are provisioned **after** the schema exists, via the GUI or `conduit-admin`/SQL — see "Add a pull source" and "Provision an API key" below.
|
||||||
|
|
||||||
|
The test suite provisions its own `conduit_test` database (via `provision.sh`) and never touches prod — a conftest guard hard-aborts if tests ever resolve to the prod `conduit` DSN.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Add a pull source
|
## Add a pull source
|
||||||
|
|
||||||
Sources are rows in the `sources` table — there is **no hot-reload**, `app.py` loads enabled sources once at startup. Any INSERT/UPDATE/DELETE requires a `systemctl restart conduit` to take effect.
|
Sources are rows in the `sources` table. **The GUI hot-reloads:** a source add/edit/delete made through the management GUI calls `Broker.set_sources()` and takes effect immediately, no restart. A direct out-of-band SQL INSERT/UPDATE/DELETE against `sources` still requires a `systemctl restart conduit` to take effect — prefer the GUI.
|
||||||
|
|
||||||
Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`.
|
Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, static `headers` (jsonb, for UA-sensitive upstreams), quota caps (`max_calls_per_day`/`_minute`/`_month`), and retention fields (`retain`, `retention_days`, `poll_interval_seconds`, `poll_path`).
|
||||||
|
|
||||||
`url_template` contract: a Python `str.format()` template filled with two named substitutions —
|
`url_template` contract: a Python `str.format()` template filled with two named substitutions —
|
||||||
|
|
||||||
|
|
@ -94,7 +100,7 @@ Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`,
|
||||||
|
|
||||||
If the inbound request itself carries a query string, it's appended to the templated URL (`&` if the template already has its own `?`, else `?`) rather than substituted in — so caller-supplied params compose instead of colliding into a double `?`. If `header_auth` is true, the key is injected as a header instead of into the URL.
|
If the inbound request itself carries a query string, it's appended to the templated URL (`&` if the template already has its own `?`, else `?`) rather than substituted in — so caller-supplied params compose instead of colliding into a double `?`. If `header_auth` is true, the key is injected as a header instead of into the URL.
|
||||||
|
|
||||||
**Worked example — `tomtom_flow_tiles`:**
|
**Worked example — `tomtom_flow_tiles` (via SQL; the GUI form is equivalent and hot-reloads):**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
|
sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c "
|
||||||
|
|
@ -109,26 +115,36 @@ VALUES (
|
||||||
ON CONFLICT (name) DO NOTHING;
|
ON CONFLICT (name) DO NOTHING;
|
||||||
"'
|
"'
|
||||||
|
|
||||||
sudo systemctl restart conduit
|
sudo systemctl restart conduit # only required for this out-of-band SQL path
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For UA-sensitive upstreams (NWS contact header, Idaho Power WAF, avalanche.org UA), set the static `headers` jsonb column instead of/alongside `header_auth`. For quota-capped upstreams (e.g. TomTom's free plan), set `max_calls_per_day`/`_minute`/`_month`. For a source that should accumulate history, set `retain=true` plus `retention_days` and `poll_interval_seconds` (and `poll_path` if the retained resource differs from the primary path) — the background `Poller` picks it up without any additional step.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Provision an API key
|
## Provision an API key
|
||||||
|
|
||||||
API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references — separate from central's keystore, encrypted under Conduit's own master key (`/etc/conduit/master.key`).
|
API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references. Encrypted under Conduit's own master key (`/etc/conduit/master.key`) — this store is separate from central's (central no longer exists to have one).
|
||||||
|
|
||||||
The pattern used for `tomtom`: the plaintext key was decrypted from central's `config.api_keys` and re-encrypted under Conduit's master key, in-process, never logged in plaintext. Use the same approach for any key that already lives in central; for a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs).
|
Keys currently held: `tomtom`, `roads511`, `firms` — all harvested from central before its retirement or provisioned directly. For a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs), or use the GUI's API Keys page.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## navi traffic-tile cutover (2026-07-14)
|
## meshai migration (2026-07-15) — completed record
|
||||||
|
|
||||||
|
All 13 of [[meshai]]'s native hazard adapters were repointed from their direct upstream to Conduit's `/up/{source}/…`: NWS, SWPC (×4 endpoints), ducting, WFIGS/fires (2 feeds), FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx (pinned `511.idaho.gov/api/wzdx`), and satpass (its Celestrak TLE call).
|
||||||
|
|
||||||
|
meshai kept all of its own transform/fusion/mesh-routing logic — only the fetch step moved to Conduit. Keys for `tomtom`, `roads511`, and `firms` now live only in Conduit's keystore; meshai no longer holds them. Every repoint was a zero-restart hot-reload on meshai's side (`apply_config()`), coordinated via meshai's config API. `traffic` (tomtom_traffic) was migrated last, unblocked by the faithful-4xx-passthrough fix (PR #16) — before that fix, TomTom's legitimate `400 "Point too far from nearest existing segment"` responses were being swallowed/mis-wrapped as 502 by Conduit.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## navi traffic-tile cutover (2026-07-14) — historical record
|
||||||
|
|
||||||
navi's `/api/traffic/` tiles were repointed from [[central]] to [[conduit]]. Exact change:
|
navi's `/api/traffic/` tiles were repointed from [[central]] to [[conduit]]. Exact change:
|
||||||
|
|
||||||
**On CT 104 — widen Conduit's bind:**
|
**On CT 104 — widen Conduit's bind:**
|
||||||
|
|
||||||
`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first (see Rollback below), then:
|
`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first, then:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
|
# edit the ExecStart --host token in /etc/systemd/system/conduit.service:
|
||||||
|
|
@ -158,30 +174,26 @@ sudo systemctl reload nginx
|
||||||
|
|
||||||
Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's `payloads` table.
|
Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's `payloads` table.
|
||||||
|
|
||||||
### Rollback
|
### Rollback target has changed
|
||||||
|
|
||||||
Both pre-cutover backups exist — restore them to revert:
|
At the time of this cutover, [[central]] was the rollback target. **Central no longer exists** (retired + DB dropped 2026-07-15). If the navi tile path ever needs to roll back off Conduit, the rollback target is now **direct TomTom** (bypassing both Conduit and the defunct central), not central — a new direct-TomTom nginx config would need to be written; the old pre-cutover nginx backup pointing at central is no longer a valid rollback target.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# CT 104 — revert the unit
|
# CT 104 — revert the unit if narrowing the bind back to loopback is ever needed
|
||||||
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
|
sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service
|
||||||
sudo systemctl daemon-reload
|
sudo systemctl daemon-reload
|
||||||
sudo systemctl restart conduit
|
sudo systemctl restart conduit
|
||||||
|
|
||||||
# recon-vm — revert nginx (back to :8000, no rewrite)
|
|
||||||
sudo cp /etc/nginx/sites-available/navi.echo6.co.bak-precutover /etc/nginx/sites-available/navi.echo6.co
|
|
||||||
sudo nginx -t
|
|
||||||
sudo systemctl reload nginx
|
|
||||||
```
|
```
|
||||||
|
|
||||||
[[central]] must stay running — it's the rollback target.
|
The recon-vm nginx `.bak-precutover` backup (proxying to central) is now historical only — do not restore it, since central is gone.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Gotchas
|
## Gotchas
|
||||||
|
|
||||||
- **Single-worker only.** Single-flight coalescing is per-process; adding uvicorn `--workers` breaks the "one upstream call" guarantee. Do not add workers.
|
- **Single-worker only.** Single-flight coalescing and the quota lock are per-process; adding uvicorn `--workers` breaks both guarantees. Do not add workers.
|
||||||
- **`/up` rejects HEAD** (405) — GET only.
|
- **`/up` rejects HEAD** (405) — GET only.
|
||||||
- **`/up` is unauthenticated**, like central's auth-exempt tile endpoints. Fine for tiles; revisit before adding sensitive sources.
|
- **`/up` and `/history` are unauthenticated**, mesh-internal, tiles-trust model. The GUI is the authenticated surface (operator auth + CSRF).
|
||||||
- **No source hot-reload** — any `sources` table change needs `systemctl restart conduit`.
|
- **Source hot-reload is GUI-only.** A source change through the GUI takes effect immediately; a direct SQL change to `sources` still needs `systemctl restart conduit`.
|
||||||
- **central must stay running** — it's the rollback target for the navi cutover, not a decommissioned dependency.
|
- **Central is gone.** It is not a rollback target for anything anymore — see "Rollback target has changed" above. Its data is recoverable only from the pi-nas archive dump.
|
||||||
|
- **Retention is opt-in and idle by default.** The `Poller` only does work for sources with `retain=true`; an empty/near-empty `payload_history` is expected unless retention has been turned on for specific sources.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue