auto: docs sync 2026-06-28T06:00:07+00:00
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/runbooks/central-deploy-cutover.md
This commit is contained in:
echo6-autocommit
parent
64097c0cc0
commit
a421d3f0f2
3 changed files with 222 additions and 3 deletions
4
vault/.obsidian/workspace.json
vendored
4
vault/.obsidian/workspace.json
vendored
|
|
@ -199,6 +199,8 @@
|
|||
},
|
||||
"active": "17bd4a6166f789d0",
|
||||
"lastOpenFiles": [
|
||||
"runbooks/central-deploy-cutover.md",
|
||||
"runbooks/central-deploy-cutover.md.tmp.2734058.9aafbee53297",
|
||||
"docs/software/navi.md.tmp.5281.13900bd73182",
|
||||
"docs/software/navi.md.tmp.5281.4b9a09bef001",
|
||||
"projects/nominatim-v5-reimport.md.tmp.5281.7f47896f6abc",
|
||||
|
|
@ -208,8 +210,6 @@
|
|||
"runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.fad0ae6cac23",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.70c8d22f0a1d",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.3a384be4f42c",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.32d6e661d1df",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md.tmp.5281.e72f1c289265",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md",
|
||||
"projects/fleet-platform-baseline.md",
|
||||
"runbooks/toc-cortex-pve9.2-update.md",
|
||||
|
|
|
|||
219
vault/runbooks/central-deploy-cutover.md
Normal file
219
vault/runbooks/central-deploy-cutover.md
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
---
|
||||
title: "central — Deploy & Cutover Runbook"
|
||||
type: runbook
|
||||
tags: [recon]
|
||||
related: ["[[central]]"]
|
||||
updated: 2026-06-28
|
||||
---
|
||||
|
||||
# central — Deploy & Cutover Runbook
|
||||
|
||||
## Overview / when to use
|
||||
|
||||
`central` runs at `/opt/central` on **utility CT 104** — `ssh zvx@100.64.0.12` (mesh; preferred) or `192.168.1.104` (LAN). Runs as user `central`, three systemd units. Deploys are **manual, tag-based, detached-HEAD checkouts** — there is no deploy script and no CI/CD on this box. `zvx` has passwordless sudo. See [[central]] for architecture and current state.
|
||||
|
||||
Use this runbook whenever you need to deploy a new release or roll back.
|
||||
|
||||
---
|
||||
|
||||
## Key facts
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Deploy dir | `/opt/central` (owned `central:central`) |
|
||||
| Virtualenv | `/opt/central/.venv` (uv-managed, editable install of `src/central`) |
|
||||
| Lockfile | `/opt/central/uv.lock` |
|
||||
| Env file | `/etc/central/central.env` (`CENTRAL_DB_DSN`, NATS URL; readable only by `central`) |
|
||||
| Migration runner | `central.migrate` / console script `central-migrate`; tracks `public.schema_migrations`; reads `sql/migrations/*.sql` |
|
||||
| Migration policy | Forward-only, idempotent, **no down-scripts** |
|
||||
| Systemd units | `central-supervisor`, `central-archive`, `central-gui` (all `User=central`, in `/etc/systemd/system`) |
|
||||
| Start order | `nats-server` + `postgresql@16-main` must be up first; `central-archive` Requires both; `central-supervisor` Requires nats; `central-gui` has no hard deps |
|
||||
| Deploy model | Tracks **tags** — `git status` showing "detached HEAD" is normal, not a problem |
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight (ALWAYS run before any deploy)
|
||||
|
||||
**1. Confirm services are healthy now:**
|
||||
|
||||
```bash
|
||||
systemctl is-active central-supervisor central-archive central-gui
|
||||
```
|
||||
|
||||
**2. Confirm migration state is clean:**
|
||||
|
||||
```bash
|
||||
sudo -u central /opt/central/.venv/bin/central-migrate --check
|
||||
```
|
||||
|
||||
Exits 1 on drift. Resolve before proceeding.
|
||||
|
||||
**3. Manual DB backup — REQUIRED before migrations (no automated backup exists):**
|
||||
|
||||
```bash
|
||||
sudo -u central pg_dump -Fc central > /tmp/central-pre-<tag>.pgdump
|
||||
```
|
||||
|
||||
> Rollback of migrations depends ENTIRELY on this dump — there are no down-scripts.
|
||||
|
||||
---
|
||||
|
||||
## A. General deploy procedure
|
||||
|
||||
Run on CT 104 as `zvx`. Substitute `<NEW_TAG>` throughout.
|
||||
|
||||
```bash
|
||||
# 1. Fetch tags
|
||||
sudo -u central git -C /opt/central fetch origin
|
||||
|
||||
# 2. Checkout the target tag (detached HEAD — normal for this box)
|
||||
sudo -u central git -C /opt/central checkout v<NEW_TAG>
|
||||
|
||||
# 3. Sync venv from uv.lock (handles dep changes; editable install picks up code automatically)
|
||||
sudo -u central bash -c 'cd /opt/central && uv sync'
|
||||
|
||||
# 4. Preview then apply migrations
|
||||
sudo -u central /opt/central/.venv/bin/central-migrate --dry-run
|
||||
sudo -u central /opt/central/.venv/bin/central-migrate
|
||||
|
||||
# 5. Restart (nats/postgres assumed already up)
|
||||
sudo systemctl restart central-supervisor central-archive central-gui
|
||||
|
||||
# 6. Verify
|
||||
systemctl is-active central-supervisor central-archive central-gui
|
||||
sudo -u central /opt/central/.venv/bin/central-migrate --check # should be clean
|
||||
journalctl -u central-supervisor --since "2 min ago" --no-pager | tail -30
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/health # expect 200
|
||||
```
|
||||
|
||||
> Because the install is editable, pure source-only changes are live after restart even without `uv sync`. Always run `uv sync` anyway to be safe.
|
||||
|
||||
---
|
||||
|
||||
## B. Pending cutover — 3 open PRs + EONET go-global (worked example)
|
||||
|
||||
**Prerequisite:** Deploys are tag-based. You must merge and tag on Forge **before** deploying.
|
||||
|
||||
### Step 1 — Merge the PRs
|
||||
|
||||
| PR | Contents | Notes |
|
||||
|---|---|---|
|
||||
| #112 | Migration 036, version bump to 0.14.6, README | Only PR touching `pyproject.toml` — no collisions |
|
||||
| #113 | EONET `bypass_bbox_filter` flag + archive mirror | Code-only prerequisite for go-global |
|
||||
| #114 | Supervisor CPU: sat dedup + WAL `cursors.db` | Independent CPU fix |
|
||||
|
||||
Review and merge all three into main.
|
||||
|
||||
### Step 2 — Cut and push the tag
|
||||
|
||||
```bash
|
||||
git tag v0.14.6 <merge-sha>
|
||||
git push origin v0.14.6
|
||||
```
|
||||
|
||||
### Step 3 — Deploy on CT 104
|
||||
|
||||
Run **Section A** with `NEW_TAG=0.14.6`.
|
||||
|
||||
> **Migration 036 note:** This migration is already applied on this box. `central-migrate` will skip it (idempotent). That is expected, not an error.
|
||||
|
||||
### Step 4 — EONET go-global DB cutover
|
||||
|
||||
The code flag from PR #113 is necessary but **not sufficient**. After the code deploy, remove the Idaho `region` key from `config.adapters` to stop dropping global EONET events:
|
||||
|
||||
```bash
|
||||
# Read current setting first
|
||||
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"
|
||||
|
||||
# Remove ONLY the region key (JSONB minus operator)
|
||||
sudo -u central psql -d central -c "UPDATE config.adapters SET settings = settings - 'region' WHERE name='eonet';"
|
||||
|
||||
# Read back to confirm region is gone
|
||||
sudo -u central psql -d central -c "SELECT settings FROM config.adapters WHERE name='eonet';"
|
||||
|
||||
# Ensure supervisor picks up the config change
|
||||
sudo systemctl restart central-supervisor
|
||||
```
|
||||
|
||||
> A config hot-reload mechanism may exist, but a supervisor restart is the guaranteed path.
|
||||
|
||||
---
|
||||
|
||||
## Post-deploy verification (this cutover specifically)
|
||||
|
||||
**EONET flowing globally:**
|
||||
|
||||
```bash
|
||||
sudo -u central psql -d central -c "SELECT count(*), max(time) FROM events WHERE source='eonet';"
|
||||
```
|
||||
|
||||
Expect a rising count and a `max(time)` within ~30 min (EONET's ingestion cadence).
|
||||
|
||||
**CENTRAL_DISASTER stream gaining messages:**
|
||||
|
||||
```bash
|
||||
curl -sS http://localhost:8222/jsz?streams=1 | grep -A3 CENTRAL_DISASTER
|
||||
```
|
||||
|
||||
Or use `nats stream info CENTRAL_DISASTER`.
|
||||
|
||||
**Supervisor CPU dropped:**
|
||||
|
||||
After ~30–60 min, `systemctl status central-supervisor` CPU time should grow far slower than before (~17% baseline → low single digits).
|
||||
|
||||
**cursors.db shrank:**
|
||||
|
||||
```bash
|
||||
ls -lh /var/lib/central/cursors.db
|
||||
```
|
||||
|
||||
Expect ~600–800 MB smaller than the prior ~1.3 GB once sat dedup sweeps complete (may take a full sweep cycle).
|
||||
|
||||
**[[navi]] tiles unaffected:**
|
||||
|
||||
```bash
|
||||
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:8000/api/traffic/flow/8/48/95.png
|
||||
```
|
||||
|
||||
Expect `200 image/png`.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
### Code rollback
|
||||
|
||||
```bash
|
||||
sudo -u central git -C /opt/central checkout v0.14.5
|
||||
sudo -u central bash -c 'cd /opt/central && uv sync'
|
||||
sudo systemctl restart central-supervisor central-archive central-gui
|
||||
```
|
||||
|
||||
### DB rollback (schema changes)
|
||||
|
||||
Forward-only migrations have **no down-scripts**. To undo schema changes, restore the pre-flight dump:
|
||||
|
||||
```bash
|
||||
sudo -u central pg_restore -d central --clean /tmp/central-pre-<tag>.pgdump
|
||||
```
|
||||
|
||||
> Destructive. Understand exactly what you're reverting before running.
|
||||
|
||||
### EONET region key only (no redeploy needed)
|
||||
|
||||
If you only need to revert go-global without rolling back code:
|
||||
|
||||
```bash
|
||||
sudo -u central psql -d central -c "UPDATE config.adapters SET settings = jsonb_set(settings, '{region}', '{\"east\":-111.0,\"west\":-117.5,\"north\":44.5,\"south\":41.8}') WHERE name='eonet';"
|
||||
sudo systemctl restart central-supervisor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes / gotchas
|
||||
|
||||
- No deploy automation, no CI/CD, no auto-pull — every deploy is manual.
|
||||
- No automated DB backups on this box. The pre-flight `pg_dump` is the only safety net for migrations.
|
||||
- Migrations are forward-only and idempotent; re-running `central-migrate` is safe.
|
||||
- Detached HEAD after checkout is normal — the box tracks tags, not a branch.
|
||||
- `uv sync` is always safe to re-run; it is a no-op if the lockfile hasn't changed.
|
||||
Loading…
Add table
Add a link
Reference in a new issue