matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-08-14T12:00:03+00:00

Browse source 
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/docs/services/services.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/omv-add-nfs-share.md vault/runbooks/peertube-sitemap-redis-oom.md vault/runbooks/pve-guest-park-and-adopt.md
This commit is contained in:
echo6-autocommit 2026-08-14 12:00:03 +00:00
commit afaf019076
11 changed files with 697 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

112
vault/projects/fleet-storage-memory-upgrade.md Normal file
View file

@ -0,0 +1,112 @@
---
title: Fleet Storage and Memory Upgrade
type: project
tags:
- storage
aliases: []
related:
- [[environment]]
- [[fleet-platform-baseline]]
- [[toc-cortex-pve9.2-update]]
- [[fleet-patch-audit]]
- [[pve-guest-park-and-adopt]]
updated: 2026-08-14
---
# Fleet Storage and Memory Upgrade
Placing a batch of acquired drives and memory across the fleet, and the storage-architecture decisions that came out of sizing it. All figures read live from the hardware 2026-08-13/14, not from spec sheets. Node reference is [[environment]].
---
## Parts on hand
| Part | Qty | Form factor | Where it fits | Gain |
|---|---|---|---|---|
| 32 GB DDR5 SODIMM | 2 | SODIMM | **media only** | 32 → 64 GB, its board maximum |
| 2 TB NVMe | 1 | M.2 2280 | media's freed M.2 | +2 TB fast local |
| 1 TB NVMe | 2 | M.2 2280 | utility, cloud, or toc | replaces a 512 GB, needs migration |
| 16 GB DDR4 SODIMM | 4 | SODIMM | **nowhere** | cold spares only |
The DDR4 has no home. data, utility and cloud each already run 2 × 16 GB DDR4-3200; data and utility are hard-capped at 32 GB, and cloud would need 32 GB sticks to gain anything. toc's two empty slots take **full-size DIMMs**, not SODIMM. Keep them as field spares for the three ThinkCentres.
Freed by the media work: one 512 GB Intel NVMe (0% wear, 9,791 hrs) and two 16 GB DDR5-4800 ADATA SODIMMs.
---
## Memory ceilings
| Node | Installed | Slots | Board max | Headroom |
|---|---|---|---|---|
| media | 32 GB DDR5-4800 | 2/2 | 64 GB | swap to 2 × 32 GB |
| cloud | 32 GB DDR4-3200 | 2/2 | 64 GB | needs 32 GB sticks |
| data | 32 GB DDR4-3200 | 2/2 | 32 GB | **at ceiling** |
| utility | 32 GB DDR4-3200 | 2/2 | 32 GB | **at ceiling** |
| toc | 64 GB DDR4 | 6/8 | 256 GB | 2 slots free, full-size DIMM |
| pi-nas | 8 GB soldered | — | fixed | none |
| aida-nebra | 906 MiB soldered | — | fixed | none |
**media runs DDR5-4800, not the 5600 previously recorded.**
**toc's memory runs at 2133 MT/s.** Installed modules are rated 3200, 2666 and 2133; DDR4 clocks the whole bus to the slowest module present. Matched sticks would buy speed as well as capacity. DIMM6 is an ECC module but ECC is not active on this board.
---
## Storage positions
| Node | Installed | Free positions |
|---|---|---|
| data | 1 TB NVMe + 1 TB SATA SSD | **none** |
| utility | 512 GB NVMe — 12% wear, 85.2 TB written | 1 × 2.5" SATA bay |
| cloud | 512 GB NVMe — 3% wear, 16.8 TB | 1 × 2.5" SATA bay |
| media | 2 × 512 GB NVMe (one = dead Windows) | 1 M.2 once pulled |
| toc | 512 GB NVMe — 26% wear, 85.6 TB | 8 SATA ports, 4 PCIe slots |
| pi-nas | 2 × 3 TB btrfs RAID1 + 2 × 24 TB single ext4 | 1 SATA port (`ata5`) |
**data has no second M.2.** Verified from PCIe topology, not spec sheets: only three external root ports exist and all are populated — NVMe, onboard NIC, Wi-Fi. The Wi-Fi slot is E-keyed and cannot take an M-keyed 2280 drive. Adding capacity to data means replacing a drive.
**media's second M.2 holds a BitLocker Windows install dated 2022-09-26**, untouched by Proxmox, 0% wear. It is the drive to pull.
Two identical Intel SSDPEKNU512GZH in media — go by serial:
- **Pull `PHKA142402U8512A`** (Windows)
- **Keep `PHKA142504HP512A`** (Proxmox boot and LVM)
Dell firmware reports a placeholder bus address for all three M.2 slots and Intel VMD remaps the PCI addresses, so the label serial is the only reliable identifier. Pulling the wrong one is non-destructive — the node just won't boot. Provisioning the wrong one is not.
**Highest-wear drives** are toc's (26%, 85.6 TB) and utility's (12%, 85.2 TB in only 3,459 hours — about 166 full drive-writes). Both are single drives with no redundancy.
---
## Sequence
1. **media, one shutdown, no migration.** Pull the Windows drive, fit the 2 TB, swap both DDR5 sticks for the 32s. Yields 64 GB and 2 TB of local NVMe. Takes down PeerTube, the *arr stack and mcc for the duration.
2. **utility.** 1 TB replaces the 512 GB SN740 — clone-and-swap, since its only M.2 is the boot drive. It is the fullest node (thin pool 45.4% vs cloud's 23.2%) and the fastest-wearing drive.
3. **toc.** Its lone M.2 holds the most-written drive in the fleet. With four free PCIe slots, an **M.2-to-PCIe adapter** lets the 1 TB be *added* rather than replacing the boot drive — no clone, no migration.
Open question: whether any of the spare 512 GB drives are 2.5" SATA rather than M.2. If so they drop straight into utility's and cloud's empty bays as pure additions with no migration at all.
---
## Ceph — evaluated, not adopted
Goal was migration freedom: move guests between nodes at will and rebuild nodes as needed.
Sizing, from 715 GB of guest data actually in use (~950 GB provisioned):
| Scope | Usable | `size=3` raw | With headroom | Per node × 5 |
|---|---|---|---|---|
| Guest disks only | 1 TB | 3 TB | ~4.7 TB | ~1 TB |
| Guests + nav/kiwix/library | 2 TB | 6 TB | ~9 TB | ~1.8 TB |
Capacity was never the blocker. **Free devices were.** Ceph wants a whole dedicated drive per OSD, and data has none free, while utility and cloud have only one M.2 each holding their boot drive — so the spare M.2 NVMe cannot become OSDs there. It also costs ~4 GB RAM per OSD (`osd_memory_target`) on two nodes hard-capped at 32 GB.
One unlock exists: if [[navi]] moves off data, the 1 TB SATA SSD empties and becomes data's OSD. See [[navi-recon-separation]].
Adopted instead: **park-and-adopt via vzdump** to NAS-backed storage, which achieves the migration-and-rebuild goal with hardware already owned. See [[pve-guest-park-and-adopt]]. Note that LXC cannot live-migrate in PVE 9 under any storage arrangement, so shared storage would have bought fast restart-migration, not zero downtime.
Running guests directly from NFS was also rejected: container volumes are raw, NFS snapshots need qcow2, so `pct snapshot` / `pct rollback` — the rollback path [[fleet-patch-audit]] depends on — would stop working, and pi-nas would become a single point of failure for every guest at once.
---
## Redundancy note
22 TB of bulk data on pi-nas sits on **single ext4 drives with no redundancy**, including PeerTube's 11 TB library. Only the small 3 TB pair is mirrored, and that pair has **73,119 power-on hours** — 8.3 years — while holding Immich and Nextcloud. SMART is clean with zero reallocated sectors on all four drives. One SATA port remains free on the controller. Build detail is [[pi-nas-omv-runbook]].

96
vault/projects/navi-recon-separation.md Normal file
View file

@ -0,0 +1,96 @@
---
title: Separating navi from recon
type: project
tags:
- recon
aliases: []
related:
- [[navi]]
- [[deployment]]
- [[cc-rules]]
- [[fleet-storage-memory-upgrade]]
- [[recon-operations]]
updated: 2026-08-14
---
# Separating navi from recon
[[recon]] and [[navi]] are separate functions but they currently live in **one VM**, so neither can be relocated without the other. This documents what actually belongs to which, established 2026-08-13/14, and what a split would cost.
Platform docs: [[recon]], [[navi]].
---
## Current state
`recon-vm` (VMID 1130) on data — 4 cores, 24 GB allocated, 180 GB disk on local-lvm. It runs roughly twenty [[services]] across both platforms:
- **recon** — `recon.service`, `recon-watchdog.service`
- **navi** — `navi-admin`, `navi-config`, `navi-contacts`, `navi-geo`, `navi-landclass`, `navi-offroute`, `navi-places` (gunicorn, all bound to `127.0.0.1:84xx`)
- **geo stack** — `photon.service`, `argus-resolver.service`, PostgreSQL 16, plus Docker running Nominatim v5 and Valhalla
- **content** — `kiwix.service`
- `apache2` fronts the localhost-bound services
Calling a relocation of this VM "moving recon" badly undersells it.
---
## Ownership, measured
Three virtiofs shares are mounted from data's `/mnt/data` (the 1 TB SATA SSD at 93% full):
| Share | Size | Belongs to | Evidence |
|---|---|---|---|
| `nav` | 625 GB | **navi** | held open by `postgres`, `valhalla_`, `java` (Photon) |
| `kiwix` | 138 GB | kiwix-serve, also read by recon | `kiwix-ser` holds it; recon references `/mnt/kiwix/library` |
| `library` | 59 GB | **recon** | `/opt/recon` code references `/mnt/library/Acquired`, `_ingest`, `_acquired` |
`/mnt/nav` breakdown: `overture` 252 GB (contains PostgreSQL's `data_directory` at `/mnt/nav/overture/pgdata`), `tiles` 128 GB (Valhalla), `worldcover` 85 GB, `sources` 67 GB, `addresses` 36 GB, `nominatim-v5` 29 GB, `photon` 21 GB, `padus` 3.8 GB.
**recon's own footprint is 25 GB** — `/opt/recon`, of which `data` is 24 GB. It lives on the VM's disk, **not** on the full volume. Its [[environment]] holds only a profile setting, four Gemini keys and a PeerTube password. **recon uses no database at all** — the only `psycopg2` hits are files inside its venv. PostgreSQL belongs entirely to navi.
---
## Which one should move
**navi**, if either does.
- **Storage** — navi is 625 GB of the 821 GB on data's full SATA SSD. Moving recon frees *zero bytes* from the volume that is actually full. Moving navi takes it from 93% to roughly 21%.
- **Memory** — the 18 GB of page cache in that VM serves Nominatim, Valhalla, Photon and PostgreSQL. recon is a modest Python service. The VM uses 4.8 GB of 23 GB with the rest as working cache.
- **Speed** — navi's data sits on SATA SSD today. NVMe is materially faster for the random reads a geocoder and routing engine live on.
Counterpoint worth noting: recon holds a PeerTube password and feeds the PeerTube instance on media, so recon has a mild affinity *toward* media, not away from it.
---
## Why this is a rebuild, not a migration
**virtiofs is machine-local.** The shares are wired through raw QEMU `args:` to unix sockets on the data host, backed by three systemd units (`virtiofsd-nav/kiwix/library.service`) each running `/usr/libexec/virtiofsd --socket-path=/run/virtiofsd-<n>.sock --shared-dir=/mnt/data/<n> --cache=auto --announce-submounts`. It cannot cross hosts, and it blocks live migration outright. `vzdump` does not capture the contents either — see [[pve-guest-park-and-adopt]].
The VM config references socket *paths*, not host-specific IDs, so the same units on another host with `--shared-dir` repointed would work unchanged. Moving the whole VM is therefore tractable. Splitting the two platforms apart is not — it means standing up PostgreSQL with the overture and padus databases, the Nominatim and Valhalla containers, Photon, and the seven `navi-*` services on a new host, then removing them from the original.
Unresolved before planning: the `navi-*` services bind to `127.0.0.1` behind `apache2`, so the external entry point moves with navi. Whether recon calls navi's APIs has not been confirmed.
**Memory accounting caution:** the `virtiofsd` process for `nav` shows ~16.7 GB RSS, but `RssAnon` is only 20 MB — it is almost entirely `RssShmem`, the guest's own RAM mapped through the `memory-backend-memfd,share=on` object that virtiofs requires. Counting it on top of the VM's 24 GB is double-counting.
---
## Viewshed — planned, not built
No viewshed or line-of-sight code exists anywhere on the VM. Sizing for the planned feature, since it drives the memory budget:
Elevation data present is **SRTM 1 arc-second (~30 m)** — 48 `.hgt` tiles of 3601×3601 int16, 1.2 GB total, covering **41–50°N, 110–118°W** (roughly 620 × 400 miles, ~163,000 sq mi). Moving to 10 m 3DEP over the same footprint costs 7–22 GB depending on format; contour generation is where storage actually goes (10–60 GB of vector tiles, plus 100–150 GB of transient scratch during `gdal_contour`/`tippecanoe`, which is why cortex carries a 32 GB swapfile). 3DEP 10 m is US-only.
Memory depends entirely on algorithm:
| Approach | 250 mi radius at 10 m |
|---|---|
| Ray-cast, tile cache | ~8 GB (32-tile LRU) — algorithm itself under 200 MB |
| Full-grid raster | **~84 GB** across DEM + clutter + RSSI + working layers |
A 250-mile disc is 80,400² ≈ 6.5 billion cells; full-grid is quadratic in radius, ray-casting is linear. Tools in this space — including `xarray-spatial`, which mesh_terrain uses — implement full-grid, which is why they pull in `dask` for out-of-core work.
**Budget 32 GB** if building this. That covers full-grid at 100–150 miles at 10 m, 250 miles at 30 m, or dask-chunked 250 miles at 10 m. Uncompromised 250 miles at 10 m in memory is not reachable on current hardware.
Physical note: with 4/3-Earth refraction, `d(km) = 4.12(√h₁ + √h₂)`. A 250-mile path needs ~2,380 m at both ends — summit-to-summit only. Realistic siting at 1,500 m gives about 200 miles station-to-station.
This budget does not fit on data, which is capped at 32 GB and already runs the whole geo stack. It fits comfortably on media at 64 GB. See [[fleet-storage-memory-upgrade]].