diff --git a/credentials b/credentials index ef9e506..c959cb2 100755 --- a/credentials +++ b/credentials @@ -388,6 +388,12 @@ PDM_URL="https://pdm.echo6.co" PDM_BREAKGLASS_USER="admin@pam" PDM_BREAKGLASS_PASS="7redditGold" +# Conduit GUI admin login (http://100.64.0.12:8010/login) — CT 104 +# Created 2026-07-14 via conduit-admin create-operator (GUI slice 1) +CONDUIT_GUI_URL="http://100.64.0.12:8010/login" +CONDUIT_GUI_USER="admin" +CONDUIT_GUI_PASS="GQNvu72dPmPsXt8vpi0wy_L2b8RxiTgy" + # ARGUS web UI (https://argus.echo6.co) — utility CT 101 Caddy, HTTP basic auth # Backend: ARGUS app CT 103 tailnet 100.64.0.25:8090. Added 2026-07-03. # Only the bcrypt hash lives in the Caddyfile; this is the plaintext. diff --git a/engine/lint-report.md b/engine/lint-report.md index eab08c6..6fba89d 100644 --- a/engine/lint-report.md +++ b/engine/lint-report.md @@ -1,6 +1,6 @@ # Vault Lint Report -Generated: 2026-07-14T09:00:01Z | Docs scanned: 105 | Elapsed: 0.0s +Generated: 2026-07-14T12:00:21Z | Docs scanned: 105 | Elapsed: 0.0s ## Summary diff --git a/vault/.obsidian/workspace.json b/vault/.obsidian/workspace.json index 65390ce..a34de6a 100644 --- a/vault/.obsidian/workspace.json +++ b/vault/.obsidian/workspace.json @@ -199,17 +199,19 @@ }, "active": "17bd4a6166f789d0", "lastOpenFiles": [ + "runbooks/conduit-operations.md.tmp.2217597.60ef73d82d1c", + "runbooks/conduit-operations.md.tmp.2217597.cf9b78291c98", + "docs/software/central.md.tmp.2217597.90c56d780840", + "docs/software/central.md.tmp.2217597.5af41a8bdfd0", + "runbooks/conduit-operations.md", + "runbooks/conduit-operations.md.tmp.2217597.e825be725a27", + "docs/software/conduit.md", + "docs/software/conduit.md.tmp.2217597.5f47d4e96ae8", "docs/software/central.md.tmp.2217597.5f057dc86d0a", "docs/software/central.md.tmp.2217597.f65fc65a678c", "docs/software/central.md.tmp.2217597.51f154bfbc88", "docs/software/central.md.tmp.2217597.5bf096d10d72", "docs/software/central.md.tmp.2217597.ac900704af53", - "docs/software/central.md.tmp.2217597.24f28a928ea9", - "runbooks/lxc-service-migration.md.tmp.2217597.ef0b7b5fefc5", - "runbooks/lxc-service-migration.md.tmp.2217597.ff9206e1ecc9", - "runbooks/authentik-create-invitation.md.tmp.2217597.48bf7c4dd5b1", - "runbooks/authentik-create-invitation.md.tmp.2217597.e42e1524f89f", - "runbooks/authentik-create-invitation.md.tmp.2217597.28c10044e078", "archive/projects/vaultwarden-plan.md", "archive/projects/meshai-native-fire-severity-audit-cc-handoff.md", "projects/meshai-native-fire-severity-audit-cc-handoff.md", @@ -234,8 +236,6 @@ "docs/services/services.md", "docs/software/central.md", "docs/software/navi.md", - "archive/projects/mmud/mmud-phase6-prompt.md", - "archive/projects/last-ember-project.md", "assets/echo6yellow_logo_422x422_square.png", "assets/echo6yellow_logo_422x81.png", "assets/echo6_logo.png", diff --git a/vault/docs/software/central.md b/vault/docs/software/central.md index f9e5b8f..96aa567 100644 --- a/vault/docs/software/central.md +++ b/vault/docs/software/central.md @@ -10,6 +10,7 @@ related: - [[navi]] - [[fleet-platform-baseline]] - [[caddy]] + - [[conduit]] updated: 2026-07-14 --- # central — Data-Hub Spine @@ -84,7 +85,9 @@ Authenticated app with login/sessions/CSRF, first-run setup wizard, operator man ## Consumer — navi Integration -navi-traffic (navi's in-VM :8421 extraction service) was **retired on 2026-05-26** and cut over to central. recon-vm's nginx (`/etc/nginx/sites-available/navi.echo6.co`, port 8440) now proxies `^~ /api/traffic/` → `central.echo6.mesh:8000` with a 120s tile cache. navi-traffic:8421 is confirmed dead and disabled. Tile endpoints verified returning HTTP 200. +navi-traffic (navi's in-VM :8421 extraction service) was **retired on 2026-05-26** and cut over to central. recon-vm's nginx (`/etc/nginx/sites-available/navi.echo6.co`, port 8440) proxied `^~ /api/traffic/` → `central.echo6.mesh:8000` with a 120s tile cache. navi-traffic:8421 is confirmed dead and disabled. + +**As of 2026-07-14, navi's `/api/traffic/` tiles were repointed to [[conduit]]:** recon-vm's nginx now rewrites `^~ /api/traffic/` → `/up/tomtom_flow_tiles/...` and proxies to `central.echo6.mesh:8010` (Conduit, co-resident on the same CT 104 host). Central's own tile endpoint (`:8000/api/traffic/...`) still exists and still works, but is no longer on navi's hot path — it remains the rollback target if the Conduit cutover needs to be reverted. See [[conduit-operations]] for the cutover and rollback procedure. ## Dependencies diff --git a/vault/docs/software/conduit.md b/vault/docs/software/conduit.md new file mode 100644 index 0000000..425f4e9 --- /dev/null +++ b/vault/docs/software/conduit.md @@ -0,0 +1,90 @@ +--- +title: Conduit — Raw-API Broker +type: reference +tags: + - mesh +aliases: [] +related: + - [[conduit-operations]] + - [[central]] + - [[navi]] + - [[caddy]] +updated: 2026-07-14 +--- +# Conduit — Raw-API Broker + +## Overview + +Conduit is a bidirectional raw-API broker with a caching/fan-out purpose: fetch an upstream API **once**, store the raw response **as-is** (byte-for-byte, no normalization), and serve it to **many** internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times." + +- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010` as of 2026-07-14 for [[navi]] to reach it) +- **Host:** utility CT 104 (unprivileged Ubuntu LXC), co-resident with [[central]] +- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI) +- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit) + +## Host + +| Attribute | Value | +|-----------|-------| +| Container | utility CT 104 | +| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh:8010 | +| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) | +| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) | +| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) | +| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) | +| Master key | `/etc/conduit/master.key` | +| Systemd unit | `conduit.service` (enabled, single-worker, `Wants=postgresql@16-main`) | +| Bind | `0.0.0.0:8010` | + +## Architecture — the "source" model + +Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. + +- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) or **push** (an app POSTs a raw payload in — planned, not yet exercised). +- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL. This is a **cache, not a historical archive** — unlike central's TimescaleDB event archive. +- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation. +- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back. +- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source). + +**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls. + +## Modules (`src/conduit/`) + +| Module | Role | +|--------|------| +| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`), ported from central's `crypto.py` | +| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) | +| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness | +| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only; does not cross-coalesce across workers) | +| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff), consolidating central's per-adapter fetch idiom | +| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template` with `{path}`/`{key}` substitution) | +| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch; the `get(source_name, request_key)` payoff path | +| `app.py` | FastAPI app: `GET /health`, `GET /up/{source}/{path}` | + +## Repo / deploy model + +Pull-based deploy, mirroring central: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table — simplified port of central's migration runner. + +## Sources + +Sources are DB-backed (the `sources` table, not a hardcoded dict). `app.py` loads all enabled sources once at startup and hands them to the `Broker` — **there is no hot-reload yet**; a change to the `sources` table requires a `systemctl restart conduit` to take effect. + +Live source: `tomtom_flow_tiles` (TomTom Orbis flow tiles), real TomTom key provisioned into Conduit's keystore. + +## Relationship to [[central]] + +Conduit was born by harvesting central's proven, decoupled pieces: + +- The AES-256-GCM encrypted key store (lifted ~as-is). +- GUI auth/CSRF + schema-reflection "add a source" form patterns (the Conduit GUI itself is still TODO). +- The aiohttp+tenacity fetch idiom. + +It deliberately does **not** carry central's NATS/JetStream, CloudEvents normalization, TimescaleDB event archive, or enrichment pipeline — that complexity is exactly what Conduit sheds. Central is not being retired; it's kept, and partially superseded over time on a per-consumer basis. + +**[[navi]]'s traffic tiles were repointed from central to Conduit on 2026-07-14** — see [[conduit-operations]] for the cutover and rollback procedure. + +## Current state (as of 2026-07-14) + +- PRs #1–#5 merged to `main`. #5 added the DB-backed source registry (`sources` table, seeded with `tomtom_flow_tiles`). +- First vertical live and durable: navi's traffic tiles are served by `conduit.service`, byte-identical to central's, confirmed via the real nginx path and `payloads` table rows. +- Known next steps: a source-management **GUI** (DB registry is SQL-only so far) and **source hot-reload** (currently requires a service restart after any `sources` table change). diff --git a/vault/runbooks/conduit-operations.md b/vault/runbooks/conduit-operations.md new file mode 100644 index 0000000..20a9d38 --- /dev/null +++ b/vault/runbooks/conduit-operations.md @@ -0,0 +1,167 @@ +--- +title: "Conduit — Operations Runbook" +type: runbook +tags: + - mesh +aliases: [] +related: + - [[conduit]] + - [[central]] + - [[navi]] +updated: 2026-07-14 +--- + +# Conduit — Operations Runbook + +## When to use + +Deploying a change, adding a pull source, provisioning an API key, or working the navi traffic-tile cutover/rollback for [[conduit]]. See [[conduit]] for architecture and current state. + +## Key facts + +| Item | Value | +|---|---| +| Host / SSH | utility CT 104 — `ssh zvx@100.64.0.12` | +| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) | +| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) | +| Systemd unit | `conduit.service` (single-worker — do NOT add `--workers`, single-flight is per-process) | +| DB / DSN env | `/etc/conduit/conduit.env` → `CONDUIT_DB_DSN` (own `conduit` DB on the shared Postgres 16) | +| Master key | `/etc/conduit/master.key` (`CONDUIT_MASTER_KEY_PATH`) | +| Deploy-key model | Pull-only: CT 104 pulls `main` via read-only deploy key `ct104-conduit-deploy`; cannot push (mirrors central) | +| Migration runner | `conduit-migrate` (forward-only SQL in `sql/migrations/*.sql`, tracked in `schema_migrations`) | +| Bind | `0.0.0.0:8010` (widened from loopback for the navi cutover, 2026-07-14) | + +--- + +## Deploy a change + +Run on CT 104 as `zvx`. + +```bash +# 1. Pull latest main +sudo -u conduit git -C /opt/conduit fetch origin +sudo -u conduit git -C /opt/conduit checkout main +sudo -u conduit git -C /opt/conduit pull + +# 2. Install (editable, picks up code + dep changes) +sudo -u conduit bash -c 'cd /opt/conduit && uv pip install -e ".[dev]"' + +# 3. Preview then apply migrations +sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate --check' +sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && /opt/conduit/.venv/bin/conduit-migrate' + +# 4. Restart +sudo systemctl restart conduit + +# 5. Verify +systemctl is-active conduit +curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/health # expect 200 +curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8010/up/tomtom_flow_tiles/flow/8/48/95.pbf # expect 200, a live tile +``` + +--- + +## Add a pull source + +Sources are rows in the `sources` table — there is **no hot-reload**, `app.py` loads enabled sources once at startup. Any INSERT/UPDATE/DELETE requires a `systemctl restart conduit` to take effect. + +Columns: `name`, `url_template`, `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`. + +`url_template` contract: a Python `str.format()` template filled with two named substitutions — + +- `{path}` — the upstream path only (no query string), taken from the inbound request after the source name and before the first `?`. +- `{key}` — the decrypted API key for `api_key_alias`, if set. + +If the inbound request itself carries a query string, it's appended to the templated URL (`&` if the template already has its own `?`, else `?`) rather than substituted in — so caller-supplied params compose instead of colliding into a double `?`. If `header_auth` is true, the key is injected as a header instead of into the URL. + +**Worked example — `tomtom_flow_tiles`:** + +```bash +sudo -u conduit bash -c 'cd /opt/conduit && set -a && . /etc/conduit/conduit.env && set +a && psql "$CONDUIT_DB_DSN" -c " +INSERT INTO sources (name, url_template, api_key_alias, ttl_seconds, header_auth) +VALUES ( + '"'"'tomtom_flow_tiles'"'"', + '"'"'https://api.tomtom.com/maps/orbis/traffic/tile/{path}?key={key}&apiVersion=1'"'"', + '"'"'tomtom'"'"', + 60, + false +) +ON CONFLICT (name) DO NOTHING; +"' + +sudo systemctl restart conduit +``` + +--- + +## Provision an API key + +API keys live in Conduit's own AES-256-GCM encrypted keystore (`api_keys` table), addressed by the `alias` a source's `api_key_alias` references — separate from central's keystore, encrypted under Conduit's own master key (`/etc/conduit/master.key`). + +The pattern used for `tomtom`: the plaintext key was decrypted from central's `config.api_keys` and re-encrypted under Conduit's master key, in-process, never logged in plaintext. Use the same approach for any key that already lives in central; for a brand-new key, insert it directly through the keystore's encrypt path (no plaintext in shell history or logs). + +--- + +## navi traffic-tile cutover (2026-07-14) + +navi's `/api/traffic/` tiles were repointed from [[central]] to [[conduit]]. Exact change: + +**On CT 104 — widen Conduit's bind:** + +`conduit.service`'s `ExecStart` uvicorn flag was changed from loopback to mesh-reachable — `--host 127.0.0.1` → `--host 0.0.0.0` (the `--port 8010` is unchanged). Unit backed up first (see Rollback below), then: + +```bash +# edit the ExecStart --host token in /etc/systemd/system/conduit.service: +# --host 127.0.0.1 → --host 0.0.0.0 +sudo systemctl daemon-reload +sudo systemctl restart conduit +``` + +**On recon-vm — rewrite the nginx proxy:** + +`/etc/nginx/sites-available/navi.echo6.co`, `location ^~ /api/traffic/` block changed to: + +```nginx +location ^~ /api/traffic/ { + rewrite ^/api/traffic/(.*)$ /up/tomtom_flow_tiles/$1 break; + proxy_pass http://central.echo6.mesh:8010; + # (existing tile-cache directives unchanged) +} +``` + +Then: + +```bash +sudo nginx -t +sudo systemctl reload nginx +``` + +Verified via the real nginx path (curl through the vhost) plus fresh rows in Conduit's `payloads` table. + +### Rollback + +Both pre-cutover backups exist — restore them to revert: + +```bash +# CT 104 — revert the unit +sudo cp /etc/systemd/system/conduit.service.bak-precutover /etc/systemd/system/conduit.service +sudo systemctl daemon-reload +sudo systemctl restart conduit + +# recon-vm — revert nginx (back to :8000, no rewrite) +sudo cp /etc/nginx/sites-available/navi.echo6.co.bak-precutover /etc/nginx/sites-available/navi.echo6.co +sudo nginx -t +sudo systemctl reload nginx +``` + +[[central]] must stay running — it's the rollback target. + +--- + +## Gotchas + +- **Single-worker only.** Single-flight coalescing is per-process; adding uvicorn `--workers` breaks the "one upstream call" guarantee. Do not add workers. +- **`/up` rejects HEAD** (405) — GET only. +- **`/up` is unauthenticated**, like central's auth-exempt tile endpoints. Fine for tiles; revisit before adding sensitive sources. +- **No source hot-reload** — any `sources` table change needs `systemctl restart conduit`. +- **central must stay running** — it's the rollback target for the navi cutover, not a decommissioned dependency.