--- title: "central — Deploy & Cutover Runbook" type: runbook tags: [recon] related: ["[[central]]"] updated: 2026-06-28 --- # central — Deploy & Cutover Runbook ## Overview / when to use `central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state. Use this runbook whenever you need to deploy a new release or roll back. --- ## Key facts | Item | Value | |---|---| | Deploy dir | `/opt/central` (owned `central:central`) | | Virtualenv | `/opt/central/.venv` (uv-managed, editable install of `src/central`) | | Lockfile | `/opt/central/uv.lock` | | Env file | `/etc/central/central.env` (`CENTRAL_DB_DSN`, NATS URL; readable only by `central`) | | Migration runner | `central.migrate` / console script `central-migrate`; tracks `public.schema_migrations`; reads `sql/migrations/*.sql` | | Migration policy | Forward-only, idempotent, **no down-scripts** | | Systemd units | `central-supervisor`, `central-archive`, `central-gui` (all `User=central`, in `/etc/systemd/system`) | | Start order | `nats-server` + `postgresql@16-main` must be up first; `central-archive` Requires both; `central-supervisor` Requires nats; `central-gui` has no hard deps | | Deploy model | Tracks **tags** — `git status` showing "detached HEAD" is normal, not a problem | --- ## Pre-flight (ALWAYS run before any deploy) **1. Confirm services are healthy now:** ```bash systemctl is-active central-supervisor central-archive central-gui ``` **2. Confirm migration state is clean:** ```bash sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check' ``` Exits 1 on drift. Resolve before proceeding. > **`central-migrate` env gotcha:** a bare `sudo -u central central-migrate` reads `.env` from the caller's cwd and the DB DSN is missing — it fails with `PermissionError: '.env'`. The systemd units supply it via `WorkingDirectory=/opt/central` + `EnvironmentFile=/etc/central/central.env`; manual calls must reproduce that with `cd /opt/central && set -a && . /etc/central/central.env && set +a && …` (the `central` user can read the env file). This wrapper is applied to every `central-migrate` invocation below. **3. Manual DB backup — REQUIRED before migrations (no automated backup exists):** ```bash sudo -u central pg_dump -Fc central > /tmp/central-pre-.pgdump ``` > Rollback of migrations depends ENTIRELY on this dump — there are no down-scripts. --- ## A. General deploy procedure > **Scripted path (preferred):** once `v0.14.6`+ is deployed, the repo ships `scripts/deploy.sh`, which automates everything below (pre-flight `pg_dump` to `/var/backups/central`, drift gate, fetch/checkout, `uv sync`, migrate, restart, verify, ERR-trap rollback guidance). Run: > ```bash > sudo /opt/central/scripts/deploy.sh v # add -y to skip the confirm prompt > ``` > **Bootstrap caveat:** the script ships *inside* the repo, so the `v0.14.6` cutover that first introduces it must be done with the manual steps below; every deploy after that can use the script. The manual procedure remains the source of truth and the fallback. Run on CT 104 as `zvx`. Substitute `` throughout. ```bash # 1. Fetch tags sudo -u central git -C /opt/central fetch origin # 2. Checkout the target tag (detached HEAD — normal for this box) sudo -u central git -C /opt/central checkout v # 3. Sync venv from uv.lock (handles dep changes; editable install picks up code automatically) sudo -u central bash -c 'cd /opt/central && uv sync' # 4. Preview then apply migrations (cd+env wrapper — see the env gotcha note above) sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --dry-run' sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate' # 5. Restart (nats/postgres assumed already up) sudo systemctl restart central-supervisor central-archive central-gui # 6. Verify systemctl is-active central-supervisor central-archive central-gui sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check' # should be clean journalctl -u central-supervisor --since "2 min ago" --no-pager | tail -30 curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/health # expect 200 ``` > Because the install is editable, pure source-only changes are live after restart even without `uv sync`. Always run `uv sync` anyway to be safe. --- ## B. Pending cutover — 3 open PRs + EONET go-global (worked example) **Prerequisite:** Deploys are tag-based. You must merge and tag on Forge **before** deploying. ### Step 1 — Merge the PRs | PR | Contents | Notes | |---|---|---| | #112 | Migration 036, version bump to 0.14.6, README | Only PR touching `pyproject.toml` — no collisions | | #113 | EONET `bypass_bbox_filter` flag + archive mirror | Code-only prerequisite for go-global | | #114 | Supervisor CPU: sat dedup + WAL `cursors.db` | Independent CPU fix | | #115 | `scripts/deploy.sh` + `scripts/README.md` | Deploy automation; no runtime effect | Review and merge all four into main. ### Step 2 — Cut and push the tag ```bash git tag v0.14.6 git push origin v0.14.6 ``` ### Step 3 — Deploy on CT 104 Run **Section A** with `NEW_TAG=0.14.6`. > **Migration 036 note:** This migration is already applied on this box. `central-migrate` will skip it (idempotent). That is expected, not an error. ### Step 4 — EONET go-global DB cutover The code flag from PR #113 is necessary but **not sufficient**. After the code deploy, remove the Idaho `region` key from `config.adapters` to stop dropping global EONET events: ```bash # Read current setting first sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';" # Remove ONLY the region key (JSONB minus operator) sudo -u central psql -d central -c "UPDATE config.adapters SET settings = settings - 'region' WHERE name='eonet';" # Read back to confirm region is gone sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';" # Ensure supervisor picks up the config change sudo systemctl restart central-supervisor ``` > A config hot-reload mechanism may exist, but a supervisor restart is the guaranteed path. --- ## Post-deploy verification (this cutover specifically) **EONET flowing globally:** ```bash sudo -u central psql -d central -c "SELECT count(*), max(time) FROM events WHERE source='eonet';" ``` Expect a rising count and a `max(time)` within ~30 min (EONET's ingestion cadence). **CENTRAL_DISASTER stream gaining messages:** ```bash curl -sS http://localhost:8222/jsz?streams=1 | grep -A3 CENTRAL_DISASTER ``` Or use `nats stream info CENTRAL_DISASTER`. **Supervisor CPU dropped:** After ~30–60 min, `systemctl status central-supervisor` CPU time should grow far slower than before (~17% baseline → low single digits). **cursors.db shrank:** ```bash ls -lh /var/lib/central/cursors.db ``` Expect ~600–800 MB smaller than the prior ~1.3 GB once sat dedup sweeps complete (may take a full sweep cycle). **[[navi]] tiles unaffected:** ```bash curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:8000/api/traffic/flow/8/48/95.png ``` Expect `200 image/png`. --- ## Rollback ### Code rollback ```bash sudo -u central git -C /opt/central checkout v0.14.5 sudo -u central bash -c 'cd /opt/central && uv sync' sudo systemctl restart central-supervisor central-archive central-gui ``` ### DB rollback (schema changes) Forward-only migrations have **no down-scripts**. To undo schema changes, restore the pre-flight dump: ```bash sudo -u central pg_restore -d central --clean /tmp/central-pre-.pgdump ``` > Destructive. Understand exactly what you're reverting before running. ### EONET region key only (no redeploy needed) If you only need to revert go-global without rolling back code: ```bash sudo -u central psql -d central -c "UPDATE config.adapters SET settings = jsonb_set(settings, '{region}', '{\"east\":-111.0,\"west\":-117.5,\"north\":44.5,\"south\":41.8}') WHERE name='eonet';" sudo systemctl restart central-supervisor ``` --- ## Notes / gotchas - No deploy automation, no CI/CD, no auto-pull — every deploy is manual. - No automated DB backups on this box. The pre-flight `pg_dump` is the only safety net for migrations. - Migrations are forward-only and idempotent; re-running `central-migrate` is safe. - Detached HEAD after checkout is normal — the box tracks tags, not a branch. - `uv sync` is always safe to re-run; it is a no-op if the lockfile hasn't changed.