194 lines
7.2 KiB
Markdown
194 lines
7.2 KiB
Markdown
|
---
|
|||
|
|
title: Lift navi off recon-vm onto media
|
|||
|
|
type: runbook
|
|||
|
|
tags:
|
|||
|
|
- proxmox
|
|||
|
|
- recon
|
|||
|
|
aliases: []
|
|||
|
|
related: []
|
|||
|
|
updated: 2026-08-15
|
|||
|
|
---
|
|||
|
|
# Lift navi off recon-vm onto media
|
|||
|
|
|
|||
|
|
**Status: planned, not executed.** Nothing below has been run.
|
|||
|
|
|
|||
|
|
[[navi]] and [[recon]] share one VM (`recon-vm`, VMID 1130 on data), so neither can move without the other — see [[navi-recon-separation]]. This lifts the whole VM to media as a clone, then strips each copy down to one platform. No services are rebuilt.
|
|||
|
|
|
|||
|
|
Target is media, which now has 64 GB RAM and a 1.82 TB NVMe volume — see [[fleet-storage-memory-upgrade]].
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Why lift rather than rebuild
|
|||
|
|
|
|||
|
|
navi is ~20 interlocking services: seven `navi-*` gunicorns, nginx:8440, Valhalla and Nominatim in Docker, Photon (Java), and PostgreSQL 16 whose `data_directory` sits at `/mnt/nav/overture/pgdata`. Standing that up fresh means reproducing every version, path and config. Cloning carries all of it intact; you then delete what doesn't belong.
|
|||
|
|
|
|||
|
|
Cost: the clone inherits recon's files too, which you remove afterwards, and it inherits the source VM's identity, which **must** be fixed before it ever touches the network.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Facts this plan relies on
|
|||
|
|
|
|||
|
|
| Item | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Source | `recon-vm` VMID 1130 on data, 4 cores, 24 GB, 180 GB disk |
|
|||
|
|
| New VMID | **1131** (next free; 1130 is the highest in use) |
|
|||
|
|
| navi data | `/mnt/data/nav` — **625 GB** |
|
|||
|
|
| Front door | Caddy on utility CT 101 → `100.64.0.24:8440` (nginx in recon-vm) |
|
|||
|
|
| Auth | Authentik forward_auth at the Caddy layer — unchanged by this |
|
|||
|
|
| Space needed on media | ~180 GB system + 625 GB data = **~805 GB** of 1.82 TB |
|
|||
|
|
|
|||
|
|
**recon does not call navi.** Grepping `/opt/recon` for navi ports (84xx, 8002, 8010, 2322) and hostnames returns nothing. The split creates no cross-host dependency.
|
|||
|
|
|
|||
|
|
**Stays on data:** `recon.service` (8420), `recon-watchdog`, `kiwix-serve` (8430), and the `library` + `kiwix` shares.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## The three traps
|
|||
|
|
|
|||
|
|
**1. Identity collision.** The clone is byte-identical: same static IP `192.168.1.130`, same hostname, same `machine-id`, same Tailscale node key. Booting it on the network unfixed will fight the original for its IP and its tailnet identity. **Fix identity before first network boot** (step 3).
|
|||
|
|
|
|||
|
|
**2. The virtiofs `args:` will not start on media.** `recon-vm` carries raw QEMU args wiring `vhost-user-fs-pci` devices to `/run/virtiofsd-{nav,kiwix,library}.sock`, plus a `memory-backend-memfd` object that virtiofs requires. Those sockets do not exist on media, so the clone fails to start as-is.
|
|||
|
|
|
|||
|
|
Rather than recreate virtiofsd on media, **give the new VM a real virtual data disk**. That makes it self-contained — no host-side sockets, no raw args — and therefore movable with [[pve-guest-park-and-adopt]], which the current VM is not.
|
|||
|
|
|
|||
|
|
**3. PostgreSQL moves with the data.** `pgdata` lives inside `/mnt/nav`, so it travels with the 625 GB. Nothing separate to dump and restore, but the copy must be consistent — stop postgres before the final sync (step 5).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 1. Back up the source
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ssh ts-data
|
|||
|
|
vzdump 1130 --mode snapshot --storage pinas-backup --compress zstd
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`--mode snapshot` runs live — no downtime. This is both the rollback point and the clone source. Note the archive name.
|
|||
|
|
|
|||
|
|
## 2. Restore as VMID 1131 on media
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ssh ts-media
|
|||
|
|
qmrestore pinas-backup:backup/vzdump-qemu-1130-<ts>.vma.zst 1131 --storage media-nvme
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`media-nvme` is the 1.82 TB volume. **Do not start it yet.**
|
|||
|
|
|
|||
|
|
## 3. Fix identity before it ever boots on the network
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
qm set 1131 --name navi-vm
|
|||
|
|
qm set 1131 --ipconfig0 ip=192.168.1.132/24,gw=192.168.1.1
|
|||
|
|
qm set 1131 --memory 32768 # navi + viewshed headroom
|
|||
|
|
qm set 1131 --delete args # drop virtiofs wiring
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Removing `args` also removes the `memory-backend-memfd` object, which is only needed for virtiofs.
|
|||
|
|
|
|||
|
|
Then boot **with the NIC detached**, and from the console clear the inherited identity:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
qm set 1131 --net0 virtio,bridge=vmbr0,link_down=1
|
|||
|
|
qm start 1131
|
|||
|
|
# on the console:
|
|||
|
|
tailscale logout && systemctl stop tailscaled
|
|||
|
|
rm -f /var/lib/tailscale/tailscaled.state
|
|||
|
|
hostnamectl set-hostname navi-vm
|
|||
|
|
rm -f /etc/machine-id && systemd-machine-id-setup
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Only then reattach the NIC (`qm set 1131 --net0 virtio,bridge=vmbr0`) and re-register Tailscale as a new node.
|
|||
|
|
|
|||
|
|
## 4. Attach and prepare the data disk
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
qm set 1131 --scsi1 media-nvme:700,format=qcow2
|
|||
|
|
# inside the guest:
|
|||
|
|
mkfs.ext4 -L nav /dev/sdb
|
|||
|
|
mkdir -p /mnt/nav
|
|||
|
|
echo "LABEL=nav /mnt/nav ext4 defaults,noatime,nofail 0 2" >> /etc/fstab
|
|||
|
|
mount /mnt/nav
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
700 GB gives the 625 GB room to grow. Also remove the three virtiofs lines from the guest's `/etc/fstab`, which will otherwise fail on boot.
|
|||
|
|
|
|||
|
|
## 5. Copy the 625 GB
|
|||
|
|
|
|||
|
|
First pass live, while navi keeps serving from data:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /mnt/nav/
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Expect 2–3 hours over 1 GbE. Then stop the writers on the source and take a final delta:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# on recon-vm
|
|||
|
|
systemctl stop navi-geo navi-landclass navi-offroute navi-places \
|
|||
|
|
navi-contacts navi-admin navi-config photon argus-resolver
|
|||
|
|
docker stop nominatim-v5 valhalla
|
|||
|
|
systemctl stop postgresql@16-main
|
|||
|
|
# on navi-vm
|
|||
|
|
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /mnt/nav/
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The postgres stop is what makes `pgdata` consistent. Do not skip it.
|
|||
|
|
|
|||
|
|
## 6. Strip each side
|
|||
|
|
|
|||
|
|
On **navi-vm** (1131) — remove recon and kiwix:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
systemctl disable --now recon recon-watchdog kiwix
|
|||
|
|
rm -rf /opt/recon
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
On **recon-vm** (1130) — remove navi and the geo stack:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
systemctl disable --now navi-geo navi-landclass navi-offroute navi-places \
|
|||
|
|
navi-contacts navi-admin navi-config photon argus-resolver postgresql@16-main
|
|||
|
|
docker rm -f nominatim-v5 valhalla
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Leave `/mnt/data/nav` on data untouched until step 9.
|
|||
|
|
|
|||
|
|
## 7. Cutover — one line
|
|||
|
|
|
|||
|
|
In the Caddyfile on utility CT 101, inside the `navi.echo6.co` block, repoint every `reverse_proxy` from `100.64.0.24:8440` to the new VM's tailnet address on 8440. There are four occurrences (`@authed_api`, `@public_api`, `@valhalla`, and the fallthrough `handle`).
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ssh ts-utility
|
|||
|
|
pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile
|
|||
|
|
pct exec 101 -- systemctl reload caddy
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
DNS and Authentik are untouched — the hostname and auth flow do not change.
|
|||
|
|
|
|||
|
|
## 8. Verify
|
|||
|
|
|
|||
|
|
Confirm navi answers through the front door, not just that the VM runs — check the map loads, a geocode returns, and a route computes. Follow [[headless-browser-page-verification]] for the UI. Then confirm recon is still healthy on 8420 and kiwix on 8430.
|
|||
|
|
|
|||
|
|
## 9. Reclaim — a week later, not the same day
|
|||
|
|
|
|||
|
|
Once navi has been healthy on media for several days:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# on data
|
|||
|
|
rm -rf /mnt/data/nav
|
|||
|
|
systemctl disable --now virtiofsd-nav
|
|||
|
|
qm set 1130 --memory 8192 # recon alone needs far less than 24 GB
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
That drops `/mnt/data` from 93% to roughly 21% and frees ~16 GB of RAM on data — which matters, since data is hard-capped at 32 GB.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Rollback
|
|||
|
|
|
|||
|
|
| Stage | Recovery |
|
|||
|
|
|---|---|
|
|||
|
|
| Any point before step 7 | Nothing changed for users — navi still serves from data |
|
|||
|
|
| After cutover | Revert the four Caddy lines to `100.64.0.24:8440`, restart navi's services on recon-vm |
|
|||
|
|
| Catastrophic | `qmrestore` the step 1 archive from `pinas-backup` |
|
|||
|
|
|
|||
|
|
The rollback path stays open until step 9 deletes the source data. Do not run step 9 early.
|