From a421d3f0f24b8eb0237bd6c7d36583c7d4e8bcf1 Mon Sep 17 00:00:00 2001 From: echo6-autocommit Date: Sun, 28 Jun 2026 06:00:07 +0000 Subject: [PATCH] auto: docs sync 2026-06-28T06:00:07+00:00 Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/runbooks/central-deploy-cutover.md --- engine/lint-report.md | 2 +- vault/.obsidian/workspace.json | 4 +- vault/runbooks/central-deploy-cutover.md | 219 +++++++++++++++++++++++ 3 files changed, 222 insertions(+), 3 deletions(-) create mode 100644 vault/runbooks/central-deploy-cutover.md diff --git a/engine/lint-report.md b/engine/lint-report.md index d927642..778ab3f 100644 --- a/engine/lint-report.md +++ b/engine/lint-report.md @@ -1,6 +1,6 @@ # Vault Lint Report -Generated: 2026-06-27T18:00:07Z | Docs scanned: 95 | Elapsed: 0.0s +Generated: 2026-06-28T00:00:08Z | Docs scanned: 95 | Elapsed: 0.0s ## Summary diff --git a/vault/.obsidian/workspace.json b/vault/.obsidian/workspace.json index 5b3aa2b..771216b 100644 --- a/vault/.obsidian/workspace.json +++ b/vault/.obsidian/workspace.json @@ -199,6 +199,8 @@ }, "active": "17bd4a6166f789d0", "lastOpenFiles": [ + "runbooks/central-deploy-cutover.md", + "runbooks/central-deploy-cutover.md.tmp.2734058.9aafbee53297", "docs/software/navi.md.tmp.5281.13900bd73182", "docs/software/navi.md.tmp.5281.4b9a09bef001", "projects/nominatim-v5-reimport.md.tmp.5281.7f47896f6abc", @@ -208,8 +210,6 @@ "runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.fad0ae6cac23", "runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.70c8d22f0a1d", "runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.3a384be4f42c", - "runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.32d6e661d1df", - "runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.e72f1c289265", "runbooks/fleet-magicdns-resolved-migration.md", "projects/fleet-platform-baseline.md", "runbooks/toc-cortex-pve9.2-update.md", diff --git a/vault/runbooks/central-deploy-cutover.md b/vault/runbooks/central-deploy-cutover.md new file mode 100644 index 0000000..86c1728 --- /dev/null +++ b/vault/runbooks/central-deploy-cutover.md @@ -0,0 +1,219 @@ +--- +title: "central — Deploy & Cutover Runbook" +type: runbook +tags: [recon] +related: ["[[central]]"] +updated: 2026-06-28 +--- + +# central — Deploy & Cutover Runbook + +## Overview / when to use + +`central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state. + +Use this runbook whenever you need to deploy a new release or roll back. + +--- + +## Key facts + +| Item | Value | +|---|---| +| Deploy dir | `/opt/central` (owned `central:central`) | +| Virtualenv | `/opt/central/.venv` (uv-managed, editable install of `src/central`) | +| Lockfile | `/opt/central/uv.lock` | +| Env file | `/etc/central/central.env` (`CENTRAL_DB_DSN`, NATS URL; readable only by `central`) | +| Migration runner | `central.migrate` / console script `central-migrate`; tracks `public.schema_migrations`; reads `sql/migrations/*.sql` | +| Migration policy | Forward-only, idempotent, **no down-scripts** | +| Systemd units | `central-supervisor`, `central-archive`, `central-gui` (all `User=central`, in `/etc/systemd/system`) | +| Start order | `nats-server` + `postgresql@16-main` must be up first; `central-archive` Requires both; `central-supervisor` Requires nats; `central-gui` has no hard deps | +| Deploy model | Tracks **tags** — `git status` showing "detached HEAD" is normal, not a problem | + +--- + +## Pre-flight (ALWAYS run before any deploy) + +**1. Confirm services are healthy now:** + +```bash +systemctl is-active central-supervisor central-archive central-gui +``` + +**2. Confirm migration state is clean:** + +```bash +sudo -u central /opt/central/.venv/bin/central-migrate --check +``` + +Exits 1 on drift. Resolve before proceeding. + +**3. Manual DB backup — REQUIRED before migrations (no automated backup exists):** + +```bash +sudo -u central pg_dump -Fc central > /tmp/central-pre-.pgdump +``` + +> Rollback of migrations depends ENTIRELY on this dump — there are no down-scripts. + +--- + +## A. General deploy procedure + +Run on CT 104 as `zvx`. Substitute `` throughout. + +```bash +# 1. Fetch tags +sudo -u central git -C /opt/central fetch origin + +# 2. Checkout the target tag (detached HEAD — normal for this box) +sudo -u central git -C /opt/central checkout v + +# 3. Sync venv from uv.lock (handles dep changes; editable install picks up code automatically) +sudo -u central bash -c 'cd /opt/central && uv sync' + +# 4. Preview then apply migrations +sudo -u central /opt/central/.venv/bin/central-migrate --dry-run +sudo -u central /opt/central/.venv/bin/central-migrate + +# 5. Restart (nats/postgres assumed already up) +sudo systemctl restart central-supervisor central-archive central-gui + +# 6. Verify +systemctl is-active central-supervisor central-archive central-gui +sudo -u central /opt/central/.venv/bin/central-migrate --check # should be clean +journalctl -u central-supervisor --since "2 min ago" --no-pager | tail -30 +curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/health # expect 200 +``` + +> Because the install is editable, pure source-only changes are live after restart even without `uv sync`. Always run `uv sync` anyway to be safe. + +--- + +## B. Pending cutover — 3 open PRs + EONET go-global (worked example) + +**Prerequisite:** Deploys are tag-based. You must merge and tag on Forge **before** deploying. + +### Step 1 — Merge the PRs + +| PR | Contents | Notes | +|---|---|---| +| #112 | Migration 036, version bump to 0.14.6, README | Only PR touching `pyproject.toml` — no collisions | +| #113 | EONET `bypass_bbox_filter` flag + archive mirror | Code-only prerequisite for go-global | +| #114 | Supervisor CPU: sat dedup + WAL `cursors.db` | Independent CPU fix | + +Review and merge all three into main. + +### Step 2 — Cut and push the tag + +```bash +git tag v0.14.6 +git push origin v0.14.6 +``` + +### Step 3 — Deploy on CT 104 + +Run **Section A** with `NEW_TAG=0.14.6`. + +> **Migration 036 note:** This migration is already applied on this box. `central-migrate` will skip it (idempotent). That is expected, not an error. + +### Step 4 — EONET go-global DB cutover + +The code flag from PR #113 is necessary but **not sufficient**. After the code deploy, remove the Idaho `region` key from `config.adapters` to stop dropping global EONET events: + +```bash +# Read current setting first +sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';" + +# Remove ONLY the region key (JSONB minus operator) +sudo -u central psql -d central -c "UPDATE config.adapters SET settings = settings - 'region' WHERE name='eonet';" + +# Read back to confirm region is gone +sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';" + +# Ensure supervisor picks up the config change +sudo systemctl restart central-supervisor +``` + +> A config hot-reload mechanism may exist, but a supervisor restart is the guaranteed path. + +--- + +## Post-deploy verification (this cutover specifically) + +**EONET flowing globally:** + +```bash +sudo -u central psql -d central -c "SELECT count(*), max(time) FROM events WHERE source='eonet';" +``` + +Expect a rising count and a `max(time)` within ~30 min (EONET's ingestion cadence). + +**CENTRAL_DISASTER stream gaining messages:** + +```bash +curl -sS http://localhost:8222/jsz?streams=1 | grep -A3 CENTRAL_DISASTER +``` + +Or use `nats stream info CENTRAL_DISASTER`. + +**Supervisor CPU dropped:** + +After ~30–60 min, `systemctl status central-supervisor` CPU time should grow far slower than before (~17% baseline → low single digits). + +**cursors.db shrank:** + +```bash +ls -lh /var/lib/central/cursors.db +``` + +Expect ~600–800 MB smaller than the prior ~1.3 GB once sat dedup sweeps complete (may take a full sweep cycle). + +**[[navi]] tiles unaffected:** + +```bash +curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:8000/api/traffic/flow/8/48/95.png +``` + +Expect `200 image/png`. + +--- + +## Rollback + +### Code rollback + +```bash +sudo -u central git -C /opt/central checkout v0.14.5 +sudo -u central bash -c 'cd /opt/central && uv sync' +sudo systemctl restart central-supervisor central-archive central-gui +``` + +### DB rollback (schema changes) + +Forward-only migrations have **no down-scripts**. To undo schema changes, restore the pre-flight dump: + +```bash +sudo -u central pg_restore -d central --clean /tmp/central-pre-.pgdump +``` + +> Destructive. Understand exactly what you're reverting before running. + +### EONET region key only (no redeploy needed) + +If you only need to revert go-global without rolling back code: + +```bash +sudo -u central psql -d central -c "UPDATE config.adapters SET settings = jsonb_set(settings, '{region}', '{\"east\":-111.0,\"west\":-117.5,\"north\":44.5,\"south\":41.8}') WHERE name='eonet';" +sudo systemctl restart central-supervisor +``` + +--- + +## Notes / gotchas + +- No deploy automation, no CI/CD, no auto-pull — every deploy is manual. +- No automated DB backups on this box. The pre-flight `pg_dump` is the only safety net for migrations. +- Migrations are forward-only and idempotent; re-running `central-migrate` is safe. +- Detached HEAD after checkout is normal — the box tracks tags, not a branch. +- `uv sync` is always safe to re-run; it is a no-op if the lockfile hasn't changed.