4e12cc3352
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/projects/fleet-storage-memory-upgrade.md vault/runbooks/navi-lift-to-media.md
224 lines
10 KiB
Markdown
224 lines
10 KiB
Markdown
---
|
||
title: Separate navi onto its own VM on media
|
||
type: runbook
|
||
tags:
|
||
- media
|
||
- proxmox
|
||
aliases: []
|
||
related:
|
||
- [[navi-recon-separation]]
|
||
- [[fleet-storage-memory-upgrade]]
|
||
- [[environment]]
|
||
- [[navi]]
|
||
- [[deployment]]
|
||
updated: 2026-08-16
|
||
---
|
||
# Separate navi onto its own VM on media
|
||
|
||
**Status: COMPLETE through Phase 5 (2026-08-15). Phase 6 deliberately NOT run — rollback stays open.**
|
||
|
||
[[navi]] now runs on **navi-vm (VMID 1131) on media**, tailnet `100.64.0.27`, LAN `192.168.1.132`. Cutover outage was **2 min 10 s** (20:56:47–20:58:57 UTC), navi only — `recon-vm` was never stopped and [[recon]]/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers.
|
||
|
||
Splits [[navi]] out of `recon-vm` (VMID 1130 on data) into its own VM on media, taking **all live data onto local NVMe**. `recon-vm` keeps running throughout — it is never stopped. Background on the entanglement is [[navi-recon-separation]].
|
||
|
||
## What moves and why
|
||
|
||
Two datasets are live for [[navi]], on two different slow tiers:
|
||
|
||
| Data | Size | Currently on | Problem |
|
||
|---|---|---|---|
|
||
| `nav` | 625 GB | data's SATA SSD (virtiofs) | SATA, and the volume is 93% full |
|
||
| `planet-dem.pmtiles` | **658 GB** | **pi-nas over NFS** | 7200 rpm spinning disk across 1 GbE |
|
||
|
||
The DEM is the worse of the two. It is read on **every elevation query** (`NAVI_DEM_PMTILES` in `navi-geo.env`) *and* served **directly to browsers** for terrain (`alias /mnt/nas/nav/planet-dem.pmtiles` in `sites-available/dem-origin`). There is already a `dem-cache.conf` in nginx built specifically to paper over the NFS latency.
|
||
|
||
**The remaining ~540 GB on pi-nas is build material and stays there** — `contour-rebuild` 206 GB, `worldcover` 116 GB, `hillshade-work` 94 GB, `planet-latest.osm.pbf` 87 GB, plus dumps and staging. Nothing reads those at runtime.
|
||
|
||
**[[recon]] keeps `kiwix` (138 GB) and `library` (59 GB)** on data. Once `nav` leaves, data's SATA SSD drops from **93% to about 21%**.
|
||
|
||
## Sizing on media's 1.82 TB
|
||
|
||
| Volume | Virtual | Actual |
|
||
|---|---|---|
|
||
| `scsi0` OS (restored) | 180 GB | ~179 GB |
|
||
| `scsi1` `/data/nav` | 750 GB | 625 GB |
|
||
| `scsi2` `/data/dem` | 700 GB | 658 GB |
|
||
| **Total** | **1.63 TB (90%)** | **1.46 TB (80%)** |
|
||
|
||
qcow2 is sparse, so actual consumption is what matters day to day — but the fully-allocated worst case still fits, which is the point of sizing it this way. **Keep contour rebuilds and other build work on pi-nas**; there is not room for them here.
|
||
|
||
Memory: navi-vm at 32 GB puts media at 49 of 62 GB allocated, leaving room for the viewshed budget from [[navi-recon-separation]].
|
||
|
||
---
|
||
|
||
## Phase 0 — done
|
||
|
||
- **106 GB archive** at `pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst`, verified. Rollback point.
|
||
- **Baseline 9/9** on the navi suite; map renders headless.
|
||
- `discard=on` staged on 1130's `scsi0` — applies on its next restart, which this runbook does **not** trigger.
|
||
|
||
`navi_test.sh` is `BASE=` parameterised and checks page shell, health, geocode, place details, landclass, Valhalla height, Valhalla route and both pmtiles range reads. **Run it plus a headless screenshot after every phase.**
|
||
|
||
---
|
||
|
||
## Phase 1 — build navi-vm on media (no outage)
|
||
|
||
Restore the archive as VMID 1131. This is a lift — every navi service arrives already configured.
|
||
|
||
```bash
|
||
ssh ts-media
|
||
qmrestore pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst 1131 \
|
||
--storage media-nvme
|
||
```
|
||
|
||
**Do not start it yet** — it is byte-identical to the running VM.
|
||
|
||
```bash
|
||
qm set 1131 --delete args # drop virtiofs devices + memory-backend-memfd
|
||
qm set 1131 --name navi-vm --memory 32768
|
||
qm set 1131 --ipconfig0 ip=192.168.1.132/24,gw=192.168.1.1
|
||
qm set 1131 --net0 virtio,bridge=vmbr0,link_down=1
|
||
qm set 1131 --scsi1 media-nvme:750,format=qcow2,discard=on,ssd=1
|
||
qm set 1131 --scsi2 media-nvme:700,format=qcow2,discard=on,ssd=1
|
||
qm start 1131
|
||
```
|
||
|
||
From the console, with the link still down:
|
||
|
||
```bash
|
||
tailscale logout; systemctl stop tailscaled
|
||
rm -f /var/lib/tailscale/tailscaled.state
|
||
rm -f /etc/machine-id && systemd-machine-id-setup
|
||
hostnamectl set-hostname navi-vm
|
||
sed -i '/virtiofs/d;/mnt\/nas/d' /etc/fstab # old share + pi-nas mounts
|
||
|
||
mkfs.ext4 -L navdata /dev/sdb && mkfs.ext4 -L demdata /dev/sdc
|
||
mkdir -p /data/nav /data/dem
|
||
cat >> /etc/fstab <<'EOF'
|
||
LABEL=navdata /data/nav ext4 defaults,noatime,nofail 0 2
|
||
LABEL=demdata /data/dem ext4 defaults,noatime,nofail 0 2
|
||
/data/nav /mnt/nav none bind,nofail 0 0
|
||
EOF
|
||
mount -a
|
||
```
|
||
|
||
The bind keeps `/mnt/nav` valid so **no navi service config changes** — except the two DEM references, handled in Phase 3.
|
||
|
||
Then bring the NIC up (`qm set 1131 --net0 virtio,bridge=vmbr0`) and re-register Tailscale as a **new** node.
|
||
|
||
## Phase 2 — copy 1.28 TB (no outage)
|
||
|
||
`recon-vm` keeps serving throughout.
|
||
|
||
```bash
|
||
# nav, from data's SATA SSD
|
||
rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /data/nav/
|
||
|
||
# DEM, straight from pi-nas rather than through recon-vm
|
||
mkdir -p /mnt/pinas && mount -t nfs -o vers=3,ro 192.168.1.245:/export/data /mnt/pinas
|
||
rsync -aHAX --info=progress2 /mnt/pinas/nav/planet-dem.pmtiles /data/dem/
|
||
umount /mnt/pinas
|
||
```
|
||
|
||
Expect **4–6 hours total**. The DEM leg is bounded by pi-nas's spinning disk, not the network.
|
||
|
||
## Phase 3 — repoint navi at local storage
|
||
|
||
On **navi-vm**, the only config edits in the whole migration:
|
||
|
||
```bash
|
||
sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
|
||
/etc/navi-backend/navi-geo.env
|
||
sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
|
||
/etc/nginx/sites-available/dem-origin
|
||
grep -rn "/mnt/nas" /etc/navi-backend/ /etc/nginx/sites-enabled/ # expect nothing
|
||
nginx -t && systemctl restart nginx navi-geo navi-offroute
|
||
```
|
||
|
||
Strip recon and kiwix from navi-vm — they came along with the clone:
|
||
|
||
```bash
|
||
systemctl disable --now recon recon-watchdog kiwix
|
||
```
|
||
|
||
Leave `/opt/recon` on disk for now; deleting it is a Phase 6 task.
|
||
|
||
## Phase 4 — verify before exposing
|
||
|
||
Test the new VM **directly**, before touching the front door:
|
||
|
||
```bash
|
||
BASE=http://<navi-vm-tailnet-ip>:8440 ./navi_test.sh
|
||
```
|
||
|
||
Do not proceed on anything less than **9/9**. Valhalla needs ~1 minute after boot to load 32,396 tiles; 502s before that are expected.
|
||
|
||
## Phase 5 — cutover (navi outage only, ~2 minutes)
|
||
|
||
`recon-vm` stays **running**. Only navi's [[services]] stop.
|
||
|
||
```bash
|
||
# on recon-vm — quiesce navi writers so the delta is consistent
|
||
systemctl stop navi-geo navi-landclass navi-offroute navi-places \
|
||
navi-contacts navi-admin navi-config photon argus-resolver
|
||
docker stop nominatim-v5 valhalla
|
||
systemctl stop postgresql@16-main
|
||
|
||
# final delta (nav only; the DEM is static)
|
||
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /data/nav/
|
||
```
|
||
|
||
Start navi's stack on navi-vm, then repoint [[caddy]] on utility CT 101 — **four `reverse_proxy` lines** in the `navi.echo6.co` block (`@authed_api`, `@public_api`, `@valhalla`, and the fallthrough) — at navi-vm on 8440.
|
||
|
||
```bash
|
||
pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile
|
||
pct exec 101 -- systemctl reload caddy
|
||
```
|
||
|
||
[[dns]] and [[authentik]] are untouched.
|
||
|
||
## Phase 6 — reclaim (DEFERRED — do not run)
|
||
|
||
Only after navi has been healthy on media for **at least a week**:
|
||
|
||
- `rm -rf /mnt/data/nav` on data → SATA SSD 93% → ~21%
|
||
- `systemctl disable --now virtiofsd-nav` on data
|
||
- `rm -rf /opt/recon` on navi-vm
|
||
- Drop 1130's memory from 24 GB to ~8 GB
|
||
- **Never remove anything from pi-nas** — it is the source tier, not a cache (Matt, 2026-08-15). The 658 GB DEM stays there alongside the build material it was derived from.
|
||
|
||
**Keep the source data and VM 1130 intact until then.** That is the rollback path.
|
||
|
||
---
|
||
|
||
## Rollback
|
||
|
||
| Stage | Recovery |
|
||
|---|---|
|
||
| Phases 1–4 | Nothing changed — navi still served from recon-vm |
|
||
| After Phase 5 | Revert the four [[caddy]] lines, restart navi's [[services]] on 1130 |
|
||
| Catastrophic | `qmrestore` the Phase 0 archive |
|
||
|
||
---
|
||
|
||
## What actually bit us (2026-08-15 run)
|
||
|
||
**`systemctl start` is not `restart`.** Phase 3 ran twice; the first attempt started the navi services *before* the second DEM reference in `navi-offroute.env` was found and fixed. The second run used `start`, which is a no-op on a running unit, so `navi-offroute` kept the stale [[environment]] and threw `FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles'` on every terrain request while the on-disk config looked correct. **Verify the live process environment, not the config file:** `tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>`.
|
||
|
||
**Two files reference the DEM, not one** — `navi-geo.env` *and* `navi-offroute.env`.
|
||
|
||
**Do not rsync onto a running database.** The first Phase 5 delta was taken with navi-vm's postgres still running, overwriting `pgdata` underneath it. Recovered by stopping postgres on **both** sides and re-syncing (17 files, 66 KB — the first pass had been essentially complete). Quiesce the destination as well as the source.
|
||
|
||
**qmrestore clones the MAC and IP.** The restored VM carried recon-vm's `BC:24:11:07:B0:F9` and its cloud-init IP. Both were changed with the NIC `link_down=1`, using the QEMU guest agent over virtio-serial, before it ever saw the network.
|
||
|
||
**Tailscale must be re-registered before nginx will start.** nginx has an upstream on `central.echo6.mesh`, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, `nginx -t` failed with `host not found in upstream`. (That upstream is fine, incidentally — CT 104 kept the name "[[central]]" but runs Conduit.)
|
||
|
||
**A size guard cost a re-run.** `658 GiB` is `705,591,064,874` decimal bytes; a `> 700000000000` sanity check rejected a byte-perfect copy.
|
||
|
||
## Known traps
|
||
|
||
**`virtiofsd` dies with the VM.** Stopping VM 1130 leaves `virtiofsd-{nav,kiwix,library}.service` inactive, and it then **refuses to start** with `Failed to connect to /run/virtiofsd-*.sock`. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this.
|
||
|
||
**nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if [[dns]] is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable.
|
||
|
||
**Do not restart recon-vm.** recon and navi share it until Phase 5 completes, so any restart takes both down.
|