echo6-docs/vault/projects/navi-recon-separation.md

title	type	tags	aliases	related	updated
Separating navi from recon	project	recon		navi deployment cc-rules fleet-storage-memory-upgrade recon-operations	2026-08-14

Separating navi from recon

recon and navi are separate functions but they currently live in one VM, so neither can be relocated without the other. This documents what actually belongs to which, established 2026-08-13/14, and what a split would cost.

Platform docs: recon, navi.

---

Current state

recon-vm (VMID 1130) on data — 4 cores, 24 GB allocated, 180 GB disk on local-lvm. It runs roughly twenty services across both platforms:

• recon — recon.service, recon-watchdog.service
• navi — navi-admin, navi-config, navi-contacts, navi-geo, navi-landclass, navi-offroute, navi-places (gunicorn, all bound to 127.0.0.1:84xx)
• geo stack — photon.service, argus-resolver.service, PostgreSQL 16, plus Docker running Nominatim v5 and Valhalla
• content — kiwix.service
• apache2 fronts the localhost-bound services

Calling a relocation of this VM "moving recon" badly undersells it.

---

Ownership, measured

Three virtiofs shares are mounted from data's /mnt/data (the 1 TB SATA SSD at 93% full):

Share	Size	Belongs to	Evidence
nav	625 GB	navi	held open by postgres, valhalla_, java (Photon)
kiwix	138 GB	kiwix-serve, also read by recon	kiwix-ser holds it; recon references /mnt/kiwix/library
library	59 GB	recon	/opt/recon code references /mnt/library/Acquired, _ingest, _acquired

/mnt/nav breakdown: overture 252 GB (contains PostgreSQL's data_directory at /mnt/nav/overture/pgdata), tiles 128 GB (Valhalla), worldcover 85 GB, sources 67 GB, addresses 36 GB, nominatim-v5 29 GB, photon 21 GB, padus 3.8 GB.

recon's own footprint is 25 GB — /opt/recon, of which data is 24 GB. It lives on the VM's disk, not on the full volume. Its environment holds only a profile setting, four Gemini keys and a PeerTube password. recon uses no database at all — the only psycopg2 hits are files inside its venv. PostgreSQL belongs entirely to navi.

---

Which one should move

navi, if either does.

• Storage — navi is 625 GB of the 821 GB on data's full SATA SSD. Moving recon frees zero bytes from the volume that is actually full. Moving navi takes it from 93% to roughly 21%.
• Memory — the 18 GB of page cache in that VM serves Nominatim, Valhalla, Photon and PostgreSQL. recon is a modest Python service. The VM uses 4.8 GB of 23 GB with the rest as working cache.
• Speed — navi's data sits on SATA SSD today. NVMe is materially faster for the random reads a geocoder and routing engine live on.

Counterpoint worth noting: recon holds a PeerTube password and feeds the PeerTube instance on media, so recon has a mild affinity toward media, not away from it.

---

Why this is a rebuild, not a migration

virtiofs is machine-local. The shares are wired through raw QEMU args: to unix sockets on the data host, backed by three systemd units (virtiofsd-nav/kiwix/library.service) each running /usr/libexec/virtiofsd --socket-path=/run/virtiofsd-<n>.sock --shared-dir=/mnt/data/<n> --cache=auto --announce-submounts. It cannot cross hosts, and it blocks live migration outright. vzdump does not capture the contents either — see pve-guest-park-and-adopt.

The VM config references socket paths, not host-specific IDs, so the same units on another host with --shared-dir repointed would work unchanged. Moving the whole VM is therefore tractable. Splitting the two platforms apart is not — it means standing up PostgreSQL with the overture and padus databases, the Nominatim and Valhalla containers, Photon, and the seven navi-* services on a new host, then removing them from the original.

Unresolved before planning: the navi-* services bind to 127.0.0.1 behind apache2, so the external entry point moves with navi. Whether recon calls navi's APIs has not been confirmed.

Memory accounting caution: the virtiofsd process for nav shows ~16.7 GB RSS, but RssAnon is only 20 MB — it is almost entirely RssShmem, the guest's own RAM mapped through the memory-backend-memfd,share=on object that virtiofs requires. Counting it on top of the VM's 24 GB is double-counting.

---

Viewshed — planned, not built

No viewshed or line-of-sight code exists anywhere on the VM. Sizing for the planned feature, since it drives the memory budget:

Elevation data present is SRTM 1 arc-second (~30 m) — 48 .hgt tiles of 3601×3601 int16, 1.2 GB total, covering 41–50°N, 110–118°W (roughly 620 × 400 miles, ~163,000 sq mi). Moving to 10 m 3DEP over the same footprint costs 7–22 GB depending on format; contour generation is where storage actually goes (10–60 GB of vector tiles, plus 100–150 GB of transient scratch during gdal_contour/tippecanoe, which is why cortex carries a 32 GB swapfile). 3DEP 10 m is US-only.

Memory depends entirely on algorithm:

Approach	250 mi radius at 10 m
Ray-cast, tile cache	~8 GB (32-tile LRU) — algorithm itself under 200 MB
Full-grid raster	~84 GB across DEM + clutter + RSSI + working layers

A 250-mile disc is 80,400² ≈ 6.5 billion cells; full-grid is quadratic in radius, ray-casting is linear. Tools in this space — including xarray-spatial, which mesh_terrain uses — implement full-grid, which is why they pull in dask for out-of-core work.

Budget 32 GB if building this. That covers full-grid at 100–150 miles at 10 m, 250 miles at 30 m, or dask-chunked 250 miles at 10 m. Uncompromised 250 miles at 10 m in memory is not reachable on current hardware.

Physical note: with 4/3-Earth refraction, d(km) = 4.12(√h₁ + √h₂). A 250-mile path needs ~2,380 m at both ends — summit-to-summit only. Realistic siting at 1,500 m gives about 200 miles station-to-station.

This budget does not fit on data, which is capped at 32 GB and already runs the whole geo stack. It fits comfortably on media at 64 GB. See fleet-storage-memory-upgrade.