echo6-docs/vault/runbooks/navi-lift-to-media.md

title	type	tags	aliases	related	updated
Lift navi off recon-vm onto media	runbook	proxmox recon			2026-08-15

Lift navi off recon-vm onto media

Status: planned, not executed. Nothing below has been run.

navi and recon share one VM (recon-vm, VMID 1130 on data), so neither can move without the other — see navi-recon-separation. This lifts the whole VM to media as a clone, then strips each copy down to one platform. No services are rebuilt.

Target is media, which now has 64 GB RAM and a 1.82 TB NVMe volume — see fleet-storage-memory-upgrade.

---

Why lift rather than rebuild

navi is ~20 interlocking services: seven navi-* gunicorns, nginx:8440, Valhalla and Nominatim in Docker, Photon (Java), and PostgreSQL 16 whose data_directory sits at /mnt/nav/overture/pgdata. Standing that up fresh means reproducing every version, path and config. Cloning carries all of it intact; you then delete what doesn't belong.

Cost: the clone inherits recon's files too, which you remove afterwards, and it inherits the source VM's identity, which must be fixed before it ever touches the network.

---

Facts this plan relies on

Item	Value
Source	recon-vm VMID 1130 on data, 4 cores, 24 GB, 180 GB disk
New VMID	1131 (next free; 1130 is the highest in use)
navi data	/mnt/data/nav — 625 GB
Front door	Caddy on utility CT 101 → 100.64.0.24:8440 (nginx in recon-vm)
Auth	Authentik forward_auth at the Caddy layer — unchanged by this
Space needed on media	~180 GB system + 625 GB data = ~805 GB of 1.82 TB

recon does not call navi. Grepping /opt/recon for navi ports (84xx, 8002, 8010, 2322) and hostnames returns nothing. The split creates no cross-host dependency.

Stays on data: recon.service (8420), recon-watchdog, kiwix-serve (8430), and the library + kiwix shares.

---

The three traps

1. Identity collision. The clone is byte-identical: same static IP 192.168.1.130, same hostname, same machine-id, same Tailscale node key. Booting it on the network unfixed will fight the original for its IP and its tailnet identity. Fix identity before first network boot (step 3).

2. The virtiofs args: will not start on media. recon-vm carries raw QEMU args wiring vhost-user-fs-pci devices to /run/virtiofsd-{nav,kiwix,library}.sock, plus a memory-backend-memfd object that virtiofs requires. Those sockets do not exist on media, so the clone fails to start as-is.

Rather than recreate virtiofsd on media, give the new VM a real virtual data disk. That makes it self-contained — no host-side sockets, no raw args — and therefore movable with pve-guest-park-and-adopt, which the current VM is not.

3. PostgreSQL moves with the data. pgdata lives inside /mnt/nav, so it travels with the 625 GB. Nothing separate to dump and restore, but the copy must be consistent — stop postgres before the final sync (step 5).

---

1. Back up the source

ssh ts-data
vzdump 1130 --mode snapshot --storage pinas-backup --compress zstd

--mode snapshot runs live — no downtime. This is both the rollback point and the clone source. Note the archive name.

2. Restore as VMID 1131 on media

ssh ts-media
qmrestore pinas-backup:backup/vzdump-qemu-1130-<ts>.vma.zst 1131 --storage media-nvme

media-nvme is the 1.82 TB volume. Do not start it yet.

3. Fix identity before it ever boots on the network

qm set 1131 --name navi-vm
qm set 1131 --ipconfig0 ip=192.168.1.132/24,gw=192.168.1.1
qm set 1131 --memory 32768          # navi + viewshed headroom
qm set 1131 --delete args           # drop virtiofs wiring

Removing args also removes the memory-backend-memfd object, which is only needed for virtiofs.

Then boot with the NIC detached, and from the console clear the inherited identity:

qm set 1131 --net0 virtio,bridge=vmbr0,link_down=1
qm start 1131
# on the console:
tailscale logout && systemctl stop tailscaled
rm -f /var/lib/tailscale/tailscaled.state
hostnamectl set-hostname navi-vm
rm -f /etc/machine-id && systemd-machine-id-setup

Only then reattach the NIC (qm set 1131 --net0 virtio,bridge=vmbr0) and re-register Tailscale as a new node.

4. Attach and prepare the data disk

qm set 1131 --scsi1 media-nvme:700,format=qcow2
# inside the guest:
mkfs.ext4 -L nav /dev/sdb
mkdir -p /mnt/nav
echo "LABEL=nav /mnt/nav ext4 defaults,noatime,nofail 0 2" >> /etc/fstab
mount /mnt/nav

700 GB gives the 625 GB room to grow. Also remove the three virtiofs lines from the guest's /etc/fstab, which will otherwise fail on boot.

5. Copy the 625 GB

First pass live, while navi keeps serving from data:

rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /mnt/nav/

Expect 2–3 hours over 1 GbE. Then stop the writers on the source and take a final delta:

# on recon-vm
systemctl stop navi-geo navi-landclass navi-offroute navi-places \
  navi-contacts navi-admin navi-config photon argus-resolver
docker stop nominatim-v5 valhalla
systemctl stop postgresql@16-main
# on navi-vm
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /mnt/nav/

The postgres stop is what makes pgdata consistent. Do not skip it.

6. Strip each side

On navi-vm (1131) — remove recon and kiwix:

systemctl disable --now recon recon-watchdog kiwix
rm -rf /opt/recon

On recon-vm (1130) — remove navi and the geo stack:

systemctl disable --now navi-geo navi-landclass navi-offroute navi-places \
  navi-contacts navi-admin navi-config photon argus-resolver postgresql@16-main
docker rm -f nominatim-v5 valhalla

Leave /mnt/data/nav on data untouched until step 9.

7. Cutover — one line

In the Caddyfile on utility CT 101, inside the navi.echo6.co block, repoint every reverse_proxy from 100.64.0.24:8440 to the new VM's tailnet address on 8440. There are four occurrences (@authed_api, @public_api, @valhalla, and the fallthrough handle).

ssh ts-utility
pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile
pct exec 101 -- systemctl reload caddy

DNS and Authentik are untouched — the hostname and auth flow do not change.

8. Verify

Confirm navi answers through the front door, not just that the VM runs — check the map loads, a geocode returns, and a route computes. Follow headless-browser-page-verification for the UI. Then confirm recon is still healthy on 8420 and kiwix on 8430.

9. Reclaim — a week later, not the same day

Once navi has been healthy on media for several days:

# on data
rm -rf /mnt/data/nav
systemctl disable --now virtiofsd-nav
qm set 1130 --memory 8192     # recon alone needs far less than 24 GB

That drops /mnt/data from 93% to roughly 21% and frees ~16 GB of RAM on data — which matters, since data is hard-capped at 32 GB.

---

Rollback

Stage	Recovery
Any point before step 7	Nothing changed for users — navi still serves from data
After cutover	Revert the four Caddy lines to 100.64.0.24:8440, restart navi's services on recon-vm
Catastrophic	qmrestore the step 1 archive from pinas-backup

The rollback path stays open until step 9 deletes the source data. Do not run step 9 early.