8f95c6e5c3
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/corescope-ingest-stall-oom.md vault/runbooks/edge2-access-reference.md vault/runbooks/edge2-boot-recovery.md vault/runbooks/navi-lift-to-media.md vault/runbooks/peertube-sitemap-redis-oom.md
107 lines
7 KiB
Markdown
107 lines
7 KiB
Markdown
---
|
||
title: Separating navi from recon
|
||
type: project
|
||
tags:
|
||
- recon
|
||
aliases: []
|
||
related:
|
||
- [[navi-lift-to-media]]
|
||
- [[navi]]
|
||
- [[fleet-storage-memory-upgrade]]
|
||
- [[deployment]]
|
||
- [[environment]]
|
||
updated: 2026-08-15
|
||
---
|
||
# Separating navi from recon
|
||
|
||
[[recon]] and [[navi]] are separate functions but they currently live in **one VM**, so neither can be relocated without the other. This documents what actually belongs to which, established 2026-08-13/14, and what a split would cost.
|
||
|
||
Platform docs: [[recon]], [[navi]].
|
||
|
||
---
|
||
|
||
## Current state
|
||
|
||
`recon-vm` (VMID 1130) on data — 4 cores, 24 GB allocated, 180 GB disk on local-lvm. It runs roughly twenty [[services]] across both platforms:
|
||
|
||
- **[[recon]]** — `recon.service`, `recon-watchdog.service`
|
||
- **[[navi]]** — `navi-admin`, `navi-config`, `navi-contacts`, `navi-geo`, `navi-landclass`, `navi-offroute`, `navi-places` (gunicorn, all bound to `127.0.0.1:84xx`)
|
||
- **geo stack** — `photon.service`, `argus-resolver.service`, PostgreSQL 16, plus Docker running Nominatim v5 and Valhalla
|
||
- **content** — `kiwix.service`
|
||
- `apache2` fronts the localhost-bound [[services]]
|
||
|
||
Calling a relocation of this VM "moving recon" badly undersells it.
|
||
|
||
---
|
||
|
||
## Ownership, measured
|
||
|
||
Three virtiofs shares are mounted from data's `/mnt/data` (the 1 TB SATA SSD at 93% full):
|
||
|
||
| Share | Size | Belongs to | Evidence |
|
||
|---|---|---|---|
|
||
| `nav` | 625 GB | **navi** | held open by `postgres`, `valhalla_`, `java` (Photon) |
|
||
| `kiwix` | 138 GB | kiwix-serve, also read by recon | `kiwix-ser` holds it; recon references `/mnt/kiwix/library` |
|
||
| `library` | 59 GB | **recon** | `/opt/recon` code references `/mnt/library/Acquired`, `_ingest`, `_acquired` |
|
||
|
||
`/mnt/nav` breakdown: `overture` 252 GB (contains PostgreSQL's `data_directory` at `/mnt/nav/overture/pgdata`), `tiles` 128 GB (Valhalla), `worldcover` 85 GB, `sources` 67 GB, `addresses` 36 GB, `nominatim-v5` 29 GB, `photon` 21 GB, `padus` 3.8 GB.
|
||
|
||
**recon's own footprint is 25 GB** — `/opt/recon`, of which `data` is 24 GB. It lives on the VM's disk, **not** on the full volume. Its [[environment]] holds only a profile setting, four Gemini keys and a PeerTube password. **recon uses no database at all** — the only `psycopg2` hits are files inside its venv. PostgreSQL belongs entirely to navi.
|
||
|
||
---
|
||
|
||
## Which one should move
|
||
|
||
**navi**, if either does.
|
||
|
||
- **Storage** — navi is 625 GB of the 821 GB on data's full SATA SSD. Moving recon frees *zero bytes* from the volume that is actually full. Moving navi takes it from 93% to roughly 21%.
|
||
- **Memory** — the 18 GB of page cache in that VM serves Nominatim, Valhalla, Photon and PostgreSQL. recon is a modest Python service. The VM uses 4.8 GB of 23 GB with the rest as working cache.
|
||
- **Speed** — navi's data sits on SATA SSD today. NVMe is materially faster for the random reads a geocoder and routing engine live on.
|
||
|
||
Counterpoint worth noting: recon holds a PeerTube password and feeds the PeerTube instance on media, so recon has a mild affinity *toward* media, not away from it.
|
||
|
||
---
|
||
|
||
## Why this is a rebuild, not a migration
|
||
|
||
**virtiofs is machine-local.** The shares are wired through raw QEMU `args:` to unix sockets on the data host, backed by three systemd units (`virtiofsd-nav/kiwix/library.service`) each running `/usr/libexec/virtiofsd --socket-path=/run/virtiofsd-<n>.sock --shared-dir=/mnt/data/<n> --cache=auto --announce-submounts`. It cannot cross hosts, and it blocks live migration outright. `vzdump` does not capture the contents either — see [[pve-guest-park-and-adopt]].
|
||
|
||
The VM config references socket *paths*, not host-specific IDs, so the same units on another host with `--shared-dir` repointed would work unchanged. Moving the whole VM is therefore tractable. Splitting the two platforms apart is not — it means standing up PostgreSQL with the overture and padus databases, the Nominatim and Valhalla containers, Photon, and the seven `navi-*` services on a new host, then removing them from the original.
|
||
|
||
Unresolved before planning: the `navi-*` services bind to `127.0.0.1` behind `apache2`, so the external entry point moves with navi. Whether recon calls navi's APIs has not been confirmed.
|
||
|
||
**Memory accounting caution:** the `virtiofsd` process for `nav` shows ~16.7 GB RSS, but `RssAnon` is only 20 MB — it is almost entirely `RssShmem`, the guest's own RAM mapped through the `memory-backend-memfd,share=on` object that virtiofs requires. Counting it on top of the VM's 24 GB is double-counting.
|
||
|
||
---
|
||
|
||
## Viewshed — planned, not built
|
||
|
||
No viewshed or line-of-sight code exists anywhere on the VM. Sizing for the planned feature, since it drives the memory budget:
|
||
|
||
Elevation data present is **SRTM 1 arc-second (~30 m)** — 48 `.hgt` tiles of 3601×3601 int16, 1.2 GB total, covering **41–50°N, 110–118°W** (roughly 620 × 400 miles, ~163,000 sq mi). Moving to 10 m 3DEP over the same footprint costs 7–22 GB depending on format; contour generation is where storage actually goes (10–60 GB of vector tiles, plus 100–150 GB of transient scratch during `gdal_contour`/`tippecanoe`, which is why cortex carries a 32 GB swapfile). 3DEP 10 m is US-only.
|
||
|
||
Memory depends entirely on algorithm:
|
||
|
||
| Approach | 250 mi radius at 10 m |
|
||
|---|---|
|
||
| Ray-cast, tile cache | ~8 GB (32-tile LRU) — algorithm itself under 200 MB |
|
||
| Full-grid raster | **~84 GB** across DEM + clutter + RSSI + working layers |
|
||
|
||
A 250-mile disc is 80,400² ≈ 6.5 billion cells; full-grid is quadratic in radius, ray-casting is linear. Tools in this space — including `xarray-spatial`, which mesh_terrain uses — implement full-grid, which is why they pull in `dask` for out-of-core work.
|
||
|
||
**Budget 32 GB** if building this. That covers full-grid at 100–150 miles at 10 m, 250 miles at 30 m, or dask-chunked 250 miles at 10 m. Uncompromised 250 miles at 10 m in memory is not reachable on current hardware.
|
||
|
||
Physical note: with 4/3-Earth refraction, `d(km) = 4.12(√h₁ + √h₂)`. A 250-mile path needs ~2,380 m at both ends — summit-to-summit only. Realistic siting at 1,500 m gives about 200 miles station-to-station.
|
||
|
||
This budget does not fit on data, which is capped at 32 GB and already runs the whole geo stack. It fits comfortably on media at 64 GB. See [[fleet-storage-memory-upgrade]].
|
||
|
||
---
|
||
|
||
## Execution plan
|
||
|
||
Decided 2026-08-15: **lift the whole VM and strip each copy**, rather than rebuilding navi's ~20 services from scratch. Step-by-step procedure with rollback points is [[navi-lift-to-media]].
|
||
|
||
Two findings that made this tractable, both verified rather than assumed:
|
||
|
||
- **The front door is a single upstream.** [[caddy]] on utility CT 101 sends all of `navi.echo6.co` to `100.64.0.24:8440` (nginx inside recon-vm, which fans out to the gunicorns, Valhalla, Nominatim and Photon). Cutover is four `reverse_proxy` lines in one Caddyfile block — no [[dns]] change, no [[authentik]] change. `apache2` on the VM is only a default vhost and fronts nothing.
|
||
- **recon has no dependency on navi.** Grepping `/opt/recon` for every navi port (84xx, 8002, 8010, 2322) and hostname returns nothing, so splitting them creates no cross-host call path.
|