echo6-docs/vault/runbooks/central-deploy-cutover.md

---
title: "central — Deploy & Cutover Runbook"
type: runbook
tags: [recon]
related: ["[[central]]"]
updated: 2026-06-28
---

# central — Deploy & Cutover Runbook

## Overview / when to use

`central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state.

Use this runbook whenever you need to deploy a new release or roll back.

---

## Key facts

| Item | Value |
|---|---|
| Deploy dir | `/opt/central` (owned `central:central`) |
| Virtualenv | `/opt/central/.venv` (uv-managed, editable install of `src/central`) |
| Lockfile | `/opt/central/uv.lock` |
| Env file | `/etc/central/central.env` (`CENTRAL_DB_DSN`, NATS URL; readable only by `central`) |
| Migration runner | `central.migrate` / console script `central-migrate`; tracks `public.schema_migrations`; reads `sql/migrations/*.sql` |
| Migration policy | Forward-only, idempotent, **no down-scripts** |
| Systemd units | `central-supervisor`, `central-archive`, `central-gui` (all `User=central`, in `/etc/systemd/system`) |
| Start order | `nats-server` + `postgresql@16-main` must be up first; `central-archive` Requires both; `central-supervisor` Requires nats; `central-gui` has no hard deps |
| Deploy model | Tracks **tags** — `git status` showing "detached HEAD" is normal, not a problem |

---

## Pre-flight (ALWAYS run before any deploy)

**1. Confirm services are healthy now:**

```bash
systemctl is-active central-supervisor central-archive central-gui
```

**2. Confirm migration state is clean:**

```bash
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check'
```

Exits 1 on drift. Resolve before proceeding.

> **`central-migrate` env gotcha:** a bare `sudo -u central central-migrate` reads `.env` from the caller's cwd and the DB DSN is missing — it fails with `PermissionError: '.env'`. The systemd units supply it via `WorkingDirectory=/opt/central` + `EnvironmentFile=/etc/central/central.env`; manual calls must reproduce that with `cd /opt/central && set -a && . /etc/central/central.env && set +a && …` (the `central` user can read the env file). This wrapper is applied to every `central-migrate` invocation below.

**3. Manual DB backup — REQUIRED before migrations (no automated backup exists):**

```bash
sudo -u central pg_dump -Fc central > /tmp/central-pre-<tag>.pgdump
```

> Rollback of migrations depends ENTIRELY on this dump — there are no down-scripts.

---

## A. General deploy procedure

> **Scripted path (preferred):** once `v0.14.6`+ is deployed, the repo ships `scripts/deploy.sh`, which automates everything below (pre-flight `pg_dump` to `/var/backups/central`, drift gate, fetch/checkout, `uv sync`, migrate, restart, verify, ERR-trap rollback guidance). Run:
> ```bash
> sudo /opt/central/scripts/deploy.sh v<NEW_TAG>      # add -y to skip the confirm prompt
> ```
> **Bootstrap caveat:** the script ships *inside* the repo, so the `v0.14.6` cutover that first introduces it must be done with the manual steps below; every deploy after that can use the script. The manual procedure remains the source of truth and the fallback.

Run on CT 104 as `zvx`. Substitute `<NEW_TAG>` throughout.

```bash
# 1. Fetch tags
sudo -u central git -C /opt/central fetch origin

# 2. Checkout the target tag (detached HEAD — normal for this box)
sudo -u central git -C /opt/central checkout v<NEW_TAG>

# 3. Sync venv from uv.lock (handles dep changes; editable install picks up code automatically)
sudo -u central bash -c 'cd /opt/central && uv sync'

# 4. Preview then apply migrations (cd+env wrapper — see the env gotcha note above)
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --dry-run'
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate'

# 5. Restart (nats/postgres assumed already up)
sudo systemctl restart central-supervisor central-archive central-gui

# 6. Verify
systemctl is-active central-supervisor central-archive central-gui
sudo -u central bash -c 'cd /opt/central && set -a && . /etc/central/central.env && set +a && /opt/central/.venv/bin/central-migrate --check'   # should be clean
journalctl -u central-supervisor --since "2 min ago" --no-pager | tail -30
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/health   # expect 200
```

> Because the install is editable, pure source-only changes are live after restart even without `uv sync`. Always run `uv sync` anyway to be safe.

---

## B. Pending cutover — 3 open PRs + EONET go-global (worked example)

**Prerequisite:** Deploys are tag-based. You must merge and tag on Forge **before** deploying.

### Step 1 — Merge the PRs

| PR | Contents | Notes |
|---|---|---|
| #112 | Migration 036, version bump to 0.14.6, README | Only PR touching `pyproject.toml` — no collisions |
| #113 | EONET `bypass_bbox_filter` flag + archive mirror | Code-only prerequisite for go-global |
| #114 | Supervisor CPU: sat dedup + WAL `cursors.db` | Independent CPU fix |
| #115 | `scripts/deploy.sh` + `scripts/README.md` | Deploy automation; no runtime effect |

Review and merge all four into main.

### Step 2 — Cut and push the tag

```bash
git tag v0.14.6 <merge-sha>
git push origin v0.14.6
```

### Step 3 — Deploy on CT 104

Run **Section A** with `NEW_TAG=0.14.6`.

> **Migration 036 note:** This migration is already applied on this box. `central-migrate` will skip it (idempotent). That is expected, not an error.

### Step 4 — EONET go-global DB cutover

The code flag from PR #113 is necessary but **not sufficient**. After the code deploy, remove the Idaho `region` key from `config.adapters` to stop dropping global EONET events:

```bash
# Read current setting first
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"

# Remove ONLY the region key (JSONB minus operator)
sudo -u central psql -d central -c "UPDATE config.adapters SET settings = settings - 'region' WHERE name='eonet';"

# Read back to confirm region is gone
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"

# Ensure supervisor picks up the config change
sudo systemctl restart central-supervisor
```

> A config hot-reload mechanism may exist, but a supervisor restart is the guaranteed path.

---

## Post-deploy verification (this cutover specifically)

**EONET flowing globally:**

```bash
sudo -u central psql -d central -c "SELECT count(*), max(time) FROM events WHERE source='eonet';"
```

Expect a rising count and a `max(time)` within ~30 min (EONET's ingestion cadence).

**CENTRAL_DISASTER stream gaining messages:**

```bash
curl -sS http://localhost:8222/jsz?streams=1 | grep -A3 CENTRAL_DISASTER
```

Or use `nats stream info CENTRAL_DISASTER`.

**Supervisor CPU dropped:**

After ~30–60 min, `systemctl status central-supervisor` CPU time should grow far slower than before (~17% baseline → low single digits).

**cursors.db shrank:**

```bash
ls -lh /var/lib/central/cursors.db
```

Expect ~600–800 MB smaller than the prior ~1.3 GB once sat dedup sweeps complete (may take a full sweep cycle).

**[[navi]] tiles unaffected:**

```bash
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:8000/api/traffic/flow/8/48/95.png
```

Expect `200 image/png`.

---

## Rollback

### Code rollback

```bash
sudo -u central git -C /opt/central checkout v0.14.5
sudo -u central bash -c 'cd /opt/central && uv sync'
sudo systemctl restart central-supervisor central-archive central-gui
```

### DB rollback (schema changes)

Forward-only migrations have **no down-scripts**. To undo schema changes, restore the pre-flight dump:

```bash
sudo -u central pg_restore -d central --clean /tmp/central-pre-<tag>.pgdump
```

> Destructive. Understand exactly what you're reverting before running.

### EONET region key only (no redeploy needed)

If you only need to revert go-global without rolling back code:

```bash
sudo -u central psql -d central -c "UPDATE config.adapters SET settings = jsonb_set(settings, '{region}', '{\"east\":-111.0,\"west\":-117.5,\"north\":44.5,\"south\":41.8}') WHERE name='eonet';"
sudo systemctl restart central-supervisor
```

---

## Notes / gotchas

- No deploy automation, no CI/CD, no auto-pull — every deploy is manual.
- No automated DB backups on this box. The pre-flight `pg_dump` is the only safety net for migrations.
- Migrations are forward-only and idempotent; re-running `central-migrate` is safe.
- Detached HEAD after checkout is normal — the box tracks tags, not a branch.
- `uv sync` is always safe to re-run; it is a no-op if the lockfile hasn't changed.