echo6-docs/vault/runbooks/navi-lift-to-media.md

title	type	tags	aliases	related	updated
Separate navi onto its own VM on media	runbook	media proxmox		navi-recon-separation fleet-storage-memory-upgrade environment navi deployment	2026-08-16

Separate navi onto its own VM on media

Status: COMPLETE through Phase 5 (2026-08-15). Phase 6 deliberately NOT run — rollback stays open.

navi now runs on navi-vm (VMID 1131) on media, tailnet 100.64.0.27, LAN 192.168.1.132. Cutover outage was 2 min 10 s (20:56:47–20:58:57 UTC), navi only — recon-vm was never stopped and recon/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers.

Splits navi out of recon-vm (VMID 1130 on data) into its own VM on media, taking all live data onto local NVMe. recon-vm keeps running throughout — it is never stopped. Background on the entanglement is navi-recon-separation.

What moves and why

Two datasets are live for navi, on two different slow tiers:

Data	Size	Currently on	Problem
nav	625 GB	data's SATA SSD (virtiofs)	SATA, and the volume is 93% full
planet-dem.pmtiles	658 GB	pi-nas over NFS	7200 rpm spinning disk across 1 GbE

The DEM is the worse of the two. It is read on every elevation query (NAVI_DEM_PMTILES in navi-geo.env) and served directly to browsers for terrain (alias /mnt/nas/nav/planet-dem.pmtiles in sites-available/dem-origin). There is already a dem-cache.conf in nginx built specifically to paper over the NFS latency.

The remaining ~540 GB on pi-nas is build material and stays there — contour-rebuild 206 GB, worldcover 116 GB, hillshade-work 94 GB, planet-latest.osm.pbf 87 GB, plus dumps and staging. Nothing reads those at runtime.

recon keeps kiwix (138 GB) and library (59 GB) on data. Once nav leaves, data's SATA SSD drops from 93% to about 21%.

Sizing on media's 1.82 TB

Volume	Virtual	Actual
scsi0 OS (restored)	180 GB	~179 GB
scsi1 /data/nav	750 GB	625 GB
scsi2 /data/dem	700 GB	658 GB
Total	1.63 TB (90%)	1.46 TB (80%)

qcow2 is sparse, so actual consumption is what matters day to day — but the fully-allocated worst case still fits, which is the point of sizing it this way. Keep contour rebuilds and other build work on pi-nas; there is not room for them here.

Memory: navi-vm at 32 GB puts media at 49 of 62 GB allocated, leaving room for the viewshed budget from navi-recon-separation.

---

Phase 0 — done

• 106 GB archive at pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst, verified. Rollback point.
• Baseline 9/9 on the navi suite; map renders headless.
• discard=on staged on 1130's scsi0 — applies on its next restart, which this runbook does not trigger.

navi_test.sh is BASE= parameterised and checks page shell, health, geocode, place details, landclass, Valhalla height, Valhalla route and both pmtiles range reads. Run it plus a headless screenshot after every phase.

---

Phase 1 — build navi-vm on media (no outage)

Restore the archive as VMID 1131. This is a lift — every navi service arrives already configured.

ssh ts-media
qmrestore pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst 1131 \
  --storage media-nvme

Do not start it yet — it is byte-identical to the running VM.

qm set 1131 --delete args          # drop virtiofs devices + memory-backend-memfd
qm set 1131 --name navi-vm --memory 32768
qm set 1131 --ipconfig0 ip=192.168.1.132/24,gw=192.168.1.1
qm set 1131 --net0 virtio,bridge=vmbr0,link_down=1
qm set 1131 --scsi1 media-nvme:750,format=qcow2,discard=on,ssd=1
qm set 1131 --scsi2 media-nvme:700,format=qcow2,discard=on,ssd=1
qm start 1131

From the console, with the link still down:

tailscale logout; systemctl stop tailscaled
rm -f /var/lib/tailscale/tailscaled.state
rm -f /etc/machine-id && systemd-machine-id-setup
hostnamectl set-hostname navi-vm
sed -i '/virtiofs/d;/mnt\/nas/d' /etc/fstab    # old share + pi-nas mounts

mkfs.ext4 -L navdata /dev/sdb && mkfs.ext4 -L demdata /dev/sdc
mkdir -p /data/nav /data/dem
cat >> /etc/fstab <<'EOF'
LABEL=navdata /data/nav ext4 defaults,noatime,nofail 0 2
LABEL=demdata /data/dem ext4 defaults,noatime,nofail 0 2
/data/nav /mnt/nav none bind,nofail 0 0
EOF
mount -a

The bind keeps /mnt/nav valid so no navi service config changes — except the two DEM references, handled in Phase 3.

Then bring the NIC up (qm set 1131 --net0 virtio,bridge=vmbr0) and re-register Tailscale as a new node.

Phase 2 — copy 1.28 TB (no outage)

recon-vm keeps serving throughout.

# nav, from data's SATA SSD
rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /data/nav/

# DEM, straight from pi-nas rather than through recon-vm
mkdir -p /mnt/pinas && mount -t nfs -o vers=3,ro 192.168.1.245:/export/data /mnt/pinas
rsync -aHAX --info=progress2 /mnt/pinas/nav/planet-dem.pmtiles /data/dem/
umount /mnt/pinas

Expect 4–6 hours total. The DEM leg is bounded by pi-nas's spinning disk, not the network.

Phase 3 — repoint navi at local storage

On navi-vm, the only config edits in the whole migration:

sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
  /etc/navi-backend/navi-geo.env
sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
  /etc/nginx/sites-available/dem-origin
grep -rn "/mnt/nas" /etc/navi-backend/ /etc/nginx/sites-enabled/   # expect nothing
nginx -t && systemctl restart nginx navi-geo navi-offroute

Strip recon and kiwix from navi-vm — they came along with the clone:

systemctl disable --now recon recon-watchdog kiwix

Leave /opt/recon on disk for now; deleting it is a Phase 6 task.

Phase 4 — verify before exposing

Test the new VM directly, before touching the front door:

BASE=http://<navi-vm-tailnet-ip>:8440 ./navi_test.sh

Do not proceed on anything less than 9/9. Valhalla needs ~1 minute after boot to load 32,396 tiles; 502s before that are expected.

Phase 5 — cutover (navi outage only, ~2 minutes)

recon-vm stays running. Only navi's services stop.

# on recon-vm — quiesce navi writers so the delta is consistent
systemctl stop navi-geo navi-landclass navi-offroute navi-places \
  navi-contacts navi-admin navi-config photon argus-resolver
docker stop nominatim-v5 valhalla
systemctl stop postgresql@16-main

# final delta (nav only; the DEM is static)
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /data/nav/

Start navi's stack on navi-vm, then repoint caddy on utility CT 101 — four reverse_proxy lines in the navi.echo6.co block (@authed_api, @public_api, @valhalla, and the fallthrough) — at navi-vm on 8440.

pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile
pct exec 101 -- systemctl reload caddy

dns and authentik are untouched.

Phase 6 — reclaim (DEFERRED — do not run)

Only after navi has been healthy on media for at least a week:

• rm -rf /mnt/data/nav on data → SATA SSD 93% → ~21%
• systemctl disable --now virtiofsd-nav on data
• rm -rf /opt/recon on navi-vm
• Drop 1130's memory from 24 GB to ~8 GB
• Never remove anything from pi-nas — it is the source tier, not a cache (Matt, 2026-08-15). The 658 GB DEM stays there alongside the build material it was derived from.

Keep the source data and VM 1130 intact until then. That is the rollback path.

---

Rollback

Stage	Recovery
Phases 1–4	Nothing changed — navi still served from recon-vm
After Phase 5	Revert the four caddy lines, restart navi's services on 1130
Catastrophic	qmrestore the Phase 0 archive

---

What actually bit us (2026-08-15 run)

systemctl start is not restart. Phase 3 ran twice; the first attempt started the navi services before the second DEM reference in navi-offroute.env was found and fixed. The second run used start, which is a no-op on a running unit, so navi-offroute kept the stale environment and threw FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles' on every terrain request while the on-disk config looked correct. Verify the live process environment, not the config file: tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>.

Two files reference the DEM, not one — navi-geo.env and navi-offroute.env.

Do not rsync onto a running database. The first Phase 5 delta was taken with navi-vm's postgres still running, overwriting pgdata underneath it. Recovered by stopping postgres on both sides and re-syncing (17 files, 66 KB — the first pass had been essentially complete). Quiesce the destination as well as the source.

qmrestore clones the MAC and IP. The restored VM carried recon-vm's BC:24:11:07:B0:F9 and its cloud-init IP. Both were changed with the NIC link_down=1, using the QEMU guest agent over virtio-serial, before it ever saw the network.

Tailscale must be re-registered before nginx will start. nginx has an upstream on central.echo6.mesh, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, nginx -t failed with host not found in upstream. (That upstream is fine, incidentally — CT 104 kept the name "central" but runs Conduit.)

A size guard cost a re-run. 658 GiB is 705,591,064,874 decimal bytes; a > 700000000000 sanity check rejected a byte-perfect copy.

Known traps

virtiofsd dies with the VM. Stopping VM 1130 leaves virtiofsd-{nav,kiwix,library}.service inactive, and it then refuses to start with Failed to connect to /run/virtiofsd-*.sock. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this.

nginx fails on boot. satellite-cache.conf resolves an upstream hostname at parse time; if dns is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — 18 days of navi 502s, unnoticed. Start it manually after any boot, and fix it properly with a resolver directive plus the upstream in a variable.

Do not restart recon-vm. recon and navi share it until Phase 5 completes, so any restart takes both down.