matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-08-14T12:00:03+00:00

Browse source 
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/docs/services/services.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/omv-add-nfs-share.md vault/runbooks/peertube-sitemap-redis-oom.md vault/runbooks/pve-guest-park-and-adopt.md
This commit is contained in:
echo6-autocommit 2026-08-14 12:00:03 +00:00
commit afaf019076
11 changed files with 697 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

96
vault/projects/navi-recon-separation.md Normal file
View file

@ -0,0 +1,96 @@
---
title: Separating navi from recon
type: project
tags:
- recon
aliases: []
related:
- [[navi]]
- [[deployment]]
- [[cc-rules]]
- [[fleet-storage-memory-upgrade]]
- [[recon-operations]]
updated: 2026-08-14
---
# Separating navi from recon
[[recon]] and [[navi]] are separate functions but they currently live in **one VM**, so neither can be relocated without the other. This documents what actually belongs to which, established 2026-08-13/14, and what a split would cost.
Platform docs: [[recon]], [[navi]].
---
## Current state
`recon-vm` (VMID 1130) on data — 4 cores, 24 GB allocated, 180 GB disk on local-lvm. It runs roughly twenty [[services]] across both platforms:
- **recon** — `recon.service`, `recon-watchdog.service`
- **navi** — `navi-admin`, `navi-config`, `navi-contacts`, `navi-geo`, `navi-landclass`, `navi-offroute`, `navi-places` (gunicorn, all bound to `127.0.0.1:84xx`)
- **geo stack** — `photon.service`, `argus-resolver.service`, PostgreSQL 16, plus Docker running Nominatim v5 and Valhalla
- **content** — `kiwix.service`
- `apache2` fronts the localhost-bound services
Calling a relocation of this VM "moving recon" badly undersells it.
---
## Ownership, measured
Three virtiofs shares are mounted from data's `/mnt/data` (the 1 TB SATA SSD at 93% full):
| Share | Size | Belongs to | Evidence |
|---|---|---|---|
| `nav` | 625 GB | **navi** | held open by `postgres`, `valhalla_`, `java` (Photon) |
| `kiwix` | 138 GB | kiwix-serve, also read by recon | `kiwix-ser` holds it; recon references `/mnt/kiwix/library` |
| `library` | 59 GB | **recon** | `/opt/recon` code references `/mnt/library/Acquired`, `_ingest`, `_acquired` |
`/mnt/nav` breakdown: `overture` 252 GB (contains PostgreSQL's `data_directory` at `/mnt/nav/overture/pgdata`), `tiles` 128 GB (Valhalla), `worldcover` 85 GB, `sources` 67 GB, `addresses` 36 GB, `nominatim-v5` 29 GB, `photon` 21 GB, `padus` 3.8 GB.
**recon's own footprint is 25 GB** — `/opt/recon`, of which `data` is 24 GB. It lives on the VM's disk, **not** on the full volume. Its [[environment]] holds only a profile setting, four Gemini keys and a PeerTube password. **recon uses no database at all** — the only `psycopg2` hits are files inside its venv. PostgreSQL belongs entirely to navi.
---
## Which one should move
**navi**, if either does.
- **Storage** — navi is 625 GB of the 821 GB on data's full SATA SSD. Moving recon frees *zero bytes* from the volume that is actually full. Moving navi takes it from 93% to roughly 21%.
- **Memory** — the 18 GB of page cache in that VM serves Nominatim, Valhalla, Photon and PostgreSQL. recon is a modest Python service. The VM uses 4.8 GB of 23 GB with the rest as working cache.
- **Speed** — navi's data sits on SATA SSD today. NVMe is materially faster for the random reads a geocoder and routing engine live on.
Counterpoint worth noting: recon holds a PeerTube password and feeds the PeerTube instance on media, so recon has a mild affinity *toward* media, not away from it.
---
## Why this is a rebuild, not a migration
**virtiofs is machine-local.** The shares are wired through raw QEMU `args:` to unix sockets on the data host, backed by three systemd units (`virtiofsd-nav/kiwix/library.service`) each running `/usr/libexec/virtiofsd --socket-path=/run/virtiofsd-<n>.sock --shared-dir=/mnt/data/<n> --cache=auto --announce-submounts`. It cannot cross hosts, and it blocks live migration outright. `vzdump` does not capture the contents either — see [[pve-guest-park-and-adopt]].
The VM config references socket *paths*, not host-specific IDs, so the same units on another host with `--shared-dir` repointed would work unchanged. Moving the whole VM is therefore tractable. Splitting the two platforms apart is not — it means standing up PostgreSQL with the overture and padus databases, the Nominatim and Valhalla containers, Photon, and the seven `navi-*` services on a new host, then removing them from the original.
Unresolved before planning: the `navi-*` services bind to `127.0.0.1` behind `apache2`, so the external entry point moves with navi. Whether recon calls navi's APIs has not been confirmed.
**Memory accounting caution:** the `virtiofsd` process for `nav` shows ~16.7 GB RSS, but `RssAnon` is only 20 MB — it is almost entirely `RssShmem`, the guest's own RAM mapped through the `memory-backend-memfd,share=on` object that virtiofs requires. Counting it on top of the VM's 24 GB is double-counting.
---
## Viewshed — planned, not built
No viewshed or line-of-sight code exists anywhere on the VM. Sizing for the planned feature, since it drives the memory budget:
Elevation data present is **SRTM 1 arc-second (~30 m)** — 48 `.hgt` tiles of 3601×3601 int16, 1.2 GB total, covering **41–50°N, 110–118°W** (roughly 620 × 400 miles, ~163,000 sq mi). Moving to 10 m 3DEP over the same footprint costs 7–22 GB depending on format; contour generation is where storage actually goes (10–60 GB of vector tiles, plus 100–150 GB of transient scratch during `gdal_contour`/`tippecanoe`, which is why cortex carries a 32 GB swapfile). 3DEP 10 m is US-only.
Memory depends entirely on algorithm:
| Approach | 250 mi radius at 10 m |
|---|---|
| Ray-cast, tile cache | ~8 GB (32-tile LRU) — algorithm itself under 200 MB |
| Full-grid raster | **~84 GB** across DEM + clutter + RSSI + working layers |
A 250-mile disc is 80,400² ≈ 6.5 billion cells; full-grid is quadratic in radius, ray-casting is linear. Tools in this space — including `xarray-spatial`, which mesh_terrain uses — implement full-grid, which is why they pull in `dask` for out-of-core work.
**Budget 32 GB** if building this. That covers full-grid at 100–150 miles at 10 m, 250 miles at 30 m, or dask-chunked 250 miles at 10 m. Uncompromised 250 miles at 10 m in memory is not reachable on current hardware.
Physical note: with 4/3-Earth refraction, `d(km) = 4.12(√h₁ + √h₂)`. A 250-mile path needs ~2,380 m at both ends — summit-to-summit only. Realistic siting at 1,500 m gives about 200 miles station-to-station.
This budget does not fit on data, which is capped at 32 GB and already runs the whole geo stack. It fits comfortably on media at 64 GB. See [[fleet-storage-memory-upgrade]].