echo6-docs/vault/runbooks/central-deploy-cutover.md

title	type	tags	aliases	related	updated
central — Deploy & Cutover Runbook	runbook	mesh		central recon-operations lxc-service-migration deployment syncthing-add-node	2026-07-13

central — Deploy & Cutover Runbook

Overview / when to use

central runs at /opt/central on utility CT 104 — ssh zvx@100.64.0.12 (mesh; preferred) or 192.168.1.104 (LAN). Runs as user central, three systemd units. Deploys are manual, tag-based, detached-HEAD checkouts — there is no deploy script and no CI/CD on this box. zvx has passwordless sudo. See central for architecture and current state.

Use this runbook whenever you need to deploy a new release or roll back.

---

Key facts

Item	Value
Deploy dir	/opt/central (owned central:central)
Virtualenv	/opt/central/.venv (uv-managed, editable install of src/central)
Lockfile	/opt/central/uv.lock
Env file	/etc/central/central.env (CENTRAL_DB_DSN, NATS URL; readable only by central)
Migration runner	central.migrate / console script central-migrate; tracks public.schema_migrations; reads sql/migrations/*.sql
Migration policy	Forward-only, idempotent, no down-scripts
Systemd units	central-supervisor, central-archive, central-gui (all User=central, in /etc/systemd/system)
Start order	nats-server + postgresql@16-main must be up first; central-archive Requires both; central-supervisor Requires nats; central-gui has no hard deps
Deploy model	Tracks tags — git status showing "detached HEAD" is normal, not a problem

---

Pre-flight (ALWAYS run before any deploy)

1. Confirm services are healthy now:

systemctl is-active central-supervisor central-archive central-gui

2. Confirm migration state is clean:

sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check'

Exits 1 on drift. Resolve before proceeding.

central-migrate env gotcha: a bare sudo -u central central-migrate reads .env from the caller's cwd and the DB DSN is missing — it fails with PermissionError: '.env'. The systemd units supply it via WorkingDirectory=/opt/central + EnvironmentFile=/etc/central/central.env; manual calls must reproduce that with cd /opt/central && set -a && . /etc/central/central.env && set +a && … (the central user can read the env file). This wrapper is applied to every central-migrate invocation below.

3. Manual DB backup — REQUIRED before migrations (no automated backup exists):

sudo -u central pg_dump -Fc central > /tmp/central-pre-<tag>.pgdump

Rollback of migrations depends ENTIRELY on this dump — there are no down-scripts.

---

A. General deploy procedure

Scripted path (preferred): once v0.14.6+ is deployed, the repo ships scripts/deploy.sh, which automates everything below (pre-flight pg_dump to /var/backups/central, drift gate, fetch/checkout, uv sync, migrate, restart, verify, ERR-trap rollback guidance). Run:

sudo /opt/central/scripts/deploy.sh v<NEW_TAG>      # add -y to skip the confirm prompt

Bootstrap caveat: the script ships inside the repo, so the v0.14.6 cutover that first introduces it must be done with the manual steps below; every deploy after that can use the script. The manual procedure remains the source of truth and the fallback.

Run on CT 104 as zvx. Substitute <NEW_TAG> throughout.

# 1. Fetch tags
sudo -u central git -C /opt/central fetch origin

# 2. Checkout the target tag (detached HEAD — normal for this box)
sudo -u central git -C /opt/central checkout v<NEW_TAG>

# 3. Sync venv from uv.lock (handles dep changes; editable install picks up code automatically)
sudo -u central bash -c 'cd /opt/central && uv sync'

# 4. Preview then apply migrations (cd+env wrapper — see the env gotcha note above)
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --dry-run'
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate'

# 5. Restart (nats/postgres assumed already up)
sudo systemctl restart central-supervisor central-archive central-gui

# 6. Verify
systemctl is-active central-supervisor central-archive central-gui
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check'   # should be clean
journalctl -u central-supervisor --since "2 min ago" --no-pager | tail -30
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/health   # expect 200

Because the install is editable, pure source-only changes are live after restart even without uv sync. Always run uv sync anyway to be safe.

---

B. Pending cutover — 3 open PRs + EONET go-global (worked example)

Prerequisite: Deploys are tag-based. You must merge and tag on Forge before deploying.

Step 1 — Merge the PRs

PR	Contents	Notes
#112	Migration 036, version bump to 0.14.6, README	Only PR touching pyproject.toml — no collisions
#113	EONET bypass_bbox_filter flag + archive mirror	Code-only prerequisite for go-global
#114	Supervisor CPU: sat dedup + WAL cursors.db	Independent CPU fix
#115	scripts/deploy.sh + scripts/README.md	Deploy automation; no runtime effect

Review and merge all four into main.

Step 2 — Cut and push the tag

git tag v0.14.6 <merge-sha>
git push origin v0.14.6

Step 3 — Deploy on CT 104

Run Section A with NEW_TAG=0.14.6.

Migration 036 note: This migration is already applied on this box. central-migrate will skip it (idempotent). That is expected, not an error.

Step 4 — EONET go-global DB cutover

The code flag from PR #113 is necessary but not sufficient. After the code deploy, remove the Idaho region key from config.adapters to stop dropping global EONET events:

# Read current setting first
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"

# Remove ONLY the region key (JSONB minus operator)
sudo -u central psql -d central -c "UPDATE config.adapters SET settings = settings - 'region' WHERE name='eonet';"

# Read back to confirm region is gone
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"

# Ensure supervisor picks up the config change
sudo systemctl restart central-supervisor

A config hot-reload mechanism may exist, but a supervisor restart is the guaranteed path.

---

Post-deploy verification (this cutover specifically)

EONET flowing globally:

sudo -u central psql -d central -c "SELECT count(*), max(time) FROM events WHERE source='eonet';"

Expect a rising count and a max(time) within ~30 min (EONET's ingestion cadence).

CENTRAL_DISASTER stream gaining messages:

curl -sS http://localhost:8222/jsz?streams=1 | grep -A3 CENTRAL_DISASTER

Or use nats stream info CENTRAL_DISASTER.

Supervisor CPU dropped:

After ~30–60 min, systemctl status central-supervisor CPU time should grow far slower than before (~17% baseline → low single digits).

cursors.db shrank:

ls -lh /var/lib/central/cursors.db

Expect ~600–800 MB smaller than the prior ~1.3 GB once sat dedup sweeps complete (may take a full sweep cycle).

navi tiles unaffected:

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:8000/api/traffic/flow/8/48/95.png

Expect 200 image/png.

---

Rollback

Code rollback

sudo -u central git -C /opt/central checkout v0.14.5
sudo -u central bash -c 'cd /opt/central && uv sync'
sudo systemctl restart central-supervisor central-archive central-gui

DB rollback (schema changes)

Forward-only migrations have no down-scripts. To undo schema changes, restore the pre-flight dump:

sudo -u central pg_restore -d central --clean /tmp/central-pre-<tag>.pgdump

Destructive. Understand exactly what you're reverting before running.

EONET region key only (no redeploy needed)

If you only need to revert go-global without rolling back code:

sudo -u central psql -d central -c "UPDATE config.adapters SET settings = jsonb_set(settings, '{region}', '{\"east\":-111.0,\"west\":-117.5,\"north\":44.5,\"south\":41.8}') WHERE name='eonet';"
sudo systemctl restart central-supervisor

---

Notes / gotchas

• No deploy automation, no CI/CD, no auto-pull — every deploy is manual.
• No automated DB backups on this box. The pre-flight pg_dump is the only safety net for migrations.
• Migrations are forward-only and idempotent; re-running central-migrate is safe.
• Detached HEAD after checkout is normal — the box tracks tags, not a branch.
• uv sync is always safe to re-run; it is a no-op if the lockfile hasn't changed.