auto: docs sync 2026-08-15T12:00:04+00:00
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/corescope-ingest-stall-oom.md vault/runbooks/edge2-access-reference.md vault/runbooks/edge2-boot-recovery.md vault/runbooks/navi-lift-to-media.md vault/runbooks/peertube-sitemap-redis-oom.md
This commit is contained in:
echo6-autocommit
parent
38c45a278c
commit
8f95c6e5c3
12 changed files with 166 additions and 152 deletions
File diff suppressed because one or more lines are too long
|
|
@ -247,3 +247,9 @@
|
|||
- mode: incremental
|
||||
- docs selected: 7
|
||||
- processed: 7 | written: 7 | flagged: 0 | errors: 0
|
||||
|
||||
## 2026-08-15T09:00:01Z — sweep run
|
||||
- end: 2026-08-15T09:00:20Z
|
||||
- mode: incremental
|
||||
- docs selected: 8
|
||||
- processed: 8 | written: 8 | flagged: 0 | errors: 0
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Vault Lint Report
|
||||
|
||||
Generated: 2026-08-15T05:04:38Z | Docs scanned: 115 | Elapsed: 0.0s
|
||||
Generated: 2026-08-15T09:00:02Z | Docs scanned: 115 | Elapsed: 0.0s
|
||||
|
||||
## Summary
|
||||
|
||||
|
|
@ -8,7 +8,7 @@ Generated: 2026-08-15T05:04:38Z | Docs scanned: 115 | Elapsed: 0.0s
|
|||
|----------|-------|
|
||||
| ERROR (dead links) | 24 |
|
||||
| WARN (schema) | 0 |
|
||||
| INFO (orphans) | 36 |
|
||||
| INFO (orphans) | 35 |
|
||||
|
||||
### WARN breakdown
|
||||
- Missing frontmatter block: 0
|
||||
|
|
@ -69,7 +69,6 @@ Generated: 2026-08-15T05:04:38Z | Docs scanned: 115 | Elapsed: 0.0s
|
|||
- no incoming links: runbooks/meshai-prod-compose-override.md
|
||||
- no incoming links: runbooks/meshmonitor-password-reset.md
|
||||
- no incoming links: runbooks/meshtasticd-sim-nodes-runbook.md
|
||||
- no incoming links: runbooks/navi-lift-to-media.md
|
||||
- no incoming links: runbooks/nordvpn-lxc.md
|
||||
- no incoming links: runbooks/pg-backup.md
|
||||
- no incoming links: runbooks/pipeline-patterns.md
|
||||
|
|
@ -107,7 +106,7 @@ Matt decides whether to create a real doc — when he does, future sweeps will l
|
|||
| Term | Docs mentioning it |
|
||||
|------|--------------------|
|
||||
| `tailscale` | 42 |
|
||||
| `docker` | 38 |
|
||||
| `docker` | 37 |
|
||||
| `proxmox` | 34 |
|
||||
| `headscale` | 25 |
|
||||
| `peertube` | 25 |
|
||||
|
|
|
|||
4
vault/.obsidian/workspace.json
vendored
4
vault/.obsidian/workspace.json
vendored
|
|
@ -199,6 +199,8 @@
|
|||
},
|
||||
"active": "17bd4a6166f789d0",
|
||||
"lastOpenFiles": [
|
||||
"runbooks/navi-lift-to-media.md.tmp.2730138.5755c5b630d8",
|
||||
"runbooks/navi-lift-to-media.md.tmp.2730138.74d5fc239a82",
|
||||
"projects/navi-recon-separation.md.tmp.2730138.1a9ee67ba5ca",
|
||||
"runbooks/navi-lift-to-media.md",
|
||||
"runbooks/navi-lift-to-media.md.tmp.2730138.56b7ce0eca2e",
|
||||
|
|
@ -209,8 +211,6 @@
|
|||
"runbooks/edge2-access-reference.md.tmp.3701844.be7e6cfc6226",
|
||||
"runbooks/peertube-sitemap-redis-oom.md.tmp.3701844.88863a688de8",
|
||||
"projects/fleet-storage-memory-upgrade.md.tmp.3701844.7763e1bbbff5",
|
||||
"projects/fleet-storage-memory-upgrade.md.tmp.3701844.9037775c0789",
|
||||
"runbooks/edge2-access-reference.md.tmp.3701844.a8a5a00eaf49",
|
||||
"runbooks/edge2-boot-recovery.md",
|
||||
"runbooks/corescope-ingest-stall-oom.md",
|
||||
"projects/navi-recon-separation.md",
|
||||
|
|
|
|||
|
|
@ -9,8 +9,8 @@ related:
|
|||
- [[fleet-storage-memory-upgrade]]
|
||||
- [[ip-allocation]]
|
||||
- [[toc-cortex-pve9.2-update]]
|
||||
- [[proxmox-create-ubuntu-vm]]
|
||||
updated: 2026-08-14
|
||||
- [[pve-guest-park-and-adopt]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# Echo6 Environment Reference
|
||||
|
||||
|
|
@ -35,7 +35,7 @@ Five nodes running Proxmox VE:
|
|||
| media | Intel SSDPEKNU512GZH 512GB (NVMe, boot/`pve`) | WD Green SN350 2TB (NVMe, VG `tank`) |
|
||||
| toc | 512GB NVMe | — |
|
||||
|
||||
**media upgraded 2026-08-15:** 32 → 64 GB DDR5-4800 (2 × 32 GB, board maximum reached), and the second M.2 — which held a leftover BitLocker Windows install — replaced with a 2 TB WD Green SN350. That drive is VG `tank` as a **single 1.82 TB volume** — `tank/data`, ext4, mounted `/mnt/nvme2tb`, registered as Proxmox dir storage `media-nvme` (`nodes media`-scoped). Deliberately not split into separate pools: navi's geo data is the intended consumer and it needs one large contiguous filesystem it can grow into. The displaced Windows data was archived to pi-nas first — see [[fleet-storage-memory-upgrade]].
|
||||
**media upgraded 2026-08-15:** 32 → 64 GB DDR5-4800 (2 × 32 GB, board maximum reached), and the second M.2 — which held a leftover BitLocker Windows install — replaced with a 2 TB WD Green SN350. That drive is VG `tank` as a **single 1.82 TB volume** — `tank/data`, ext4, mounted `/mnt/nvme2tb`, registered as Proxmox dir storage `media-nvme` (`nodes media`-scoped). Deliberately not split into separate pools: [[navi]]'s geo data is the intended consumer and it needs one large contiguous filesystem it can grow into. The displaced Windows data was archived to pi-nas first — see [[fleet-storage-memory-upgrade]].
|
||||
|
||||
**Free positions (verified live 2026-08-14):** data has none — only three external PCIe root ports exist and all are populated (NVMe, NIC, Wi-Fi), and the Wi-Fi M.2 is E-keyed so it cannot take a storage drive. utility and cloud each have an empty 2.5" SATA bay. media's second M.2 holds a leftover BitLocker Windows install (serial `PHKA142402U8512A`; the Proxmox drive is `PHKA142504HP512A`). toc has 8 unpopulated SATA ports and 4 free PCIe slots. pi-nas has one free SATA port (`ata5`). Placement plan is [[fleet-storage-memory-upgrade]].
|
||||
|
||||
|
|
@ -184,7 +184,7 @@ Current registered nodes (25 total):
|
|||
|
||||
| Node | Tailscale IP | Type |
|
||||
|------|-------------|------|
|
||||
| contabo | 100.64.0.40 | VPS (edge1, mail-only, rebuilt 2026-06-19; formerly 100.64.0.1) |
|
||||
| contabo | 100.64.0.40 | VPS (edge1, mail-only, rebuilt [[2026-06-19]]; formerly 100.64.0.1) |
|
||||
| immich | 100.64.0.2 | LXC |
|
||||
| media | 100.64.0.3 | Proxmox |
|
||||
| cloud | 100.64.0.4 | Proxmox |
|
||||
|
|
@ -281,6 +281,6 @@ matt-desktop is accessible via local IP (192.168.1.254) or Tailscale (100.64.0.1
|
|||
|
||||
| Purpose | IP |
|
||||
|---------|-----|
|
||||
| Home external (public services) | 199.6.36.163 |
|
||||
| Home external (public [[services]]) | 199.6.36.163 |
|
||||
| edge1 (mail-only, rebuilt Contabo VPS) | 5.189.158.149 |
|
||||
| edge2 (front door: vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh.com) | 184.174.35.153 |
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ aliases: []
|
|||
related:
|
||||
- [[environment]]
|
||||
- [[fleet-platform-baseline]]
|
||||
- [[navi-lift-to-media]]
|
||||
- [[toc-cortex-pve9.2-update]]
|
||||
- [[fleet-patch-audit]]
|
||||
- [[pve-guest-park-and-adopt]]
|
||||
- [[navi-recon-separation]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# Fleet Storage and Memory Upgrade
|
||||
|
|
|
|||
|
|
@ -5,12 +5,12 @@ tags:
|
|||
- recon
|
||||
aliases: []
|
||||
related:
|
||||
- [[navi-lift-to-media]]
|
||||
- [[navi]]
|
||||
- [[deployment]]
|
||||
- [[cc-rules]]
|
||||
- [[fleet-storage-memory-upgrade]]
|
||||
- [[recon-operations]]
|
||||
updated: 2026-08-14
|
||||
- [[deployment]]
|
||||
- [[environment]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# Separating navi from recon
|
||||
|
||||
|
|
@ -24,11 +24,11 @@ Platform docs: [[recon]], [[navi]].
|
|||
|
||||
`recon-vm` (VMID 1130) on data — 4 cores, 24 GB allocated, 180 GB disk on local-lvm. It runs roughly twenty [[services]] across both platforms:
|
||||
|
||||
- **recon** — `recon.service`, `recon-watchdog.service`
|
||||
- **navi** — `navi-admin`, `navi-config`, `navi-contacts`, `navi-geo`, `navi-landclass`, `navi-offroute`, `navi-places` (gunicorn, all bound to `127.0.0.1:84xx`)
|
||||
- **[[recon]]** — `recon.service`, `recon-watchdog.service`
|
||||
- **[[navi]]** — `navi-admin`, `navi-config`, `navi-contacts`, `navi-geo`, `navi-landclass`, `navi-offroute`, `navi-places` (gunicorn, all bound to `127.0.0.1:84xx`)
|
||||
- **geo stack** — `photon.service`, `argus-resolver.service`, PostgreSQL 16, plus Docker running Nominatim v5 and Valhalla
|
||||
- **content** — `kiwix.service`
|
||||
- `apache2` fronts the localhost-bound services
|
||||
- `apache2` fronts the localhost-bound [[services]]
|
||||
|
||||
Calling a relocation of this VM "moving recon" badly undersells it.
|
||||
|
||||
|
|
@ -103,5 +103,5 @@ Decided 2026-08-15: **lift the whole VM and strip each copy**, rather than rebui
|
|||
|
||||
Two findings that made this tractable, both verified rather than assumed:
|
||||
|
||||
- **The front door is a single upstream.** Caddy on utility CT 101 sends all of `navi.echo6.co` to `100.64.0.24:8440` (nginx inside recon-vm, which fans out to the gunicorns, Valhalla, Nominatim and Photon). Cutover is four `reverse_proxy` lines in one Caddyfile block — no DNS change, no Authentik change. `apache2` on the VM is only a default vhost and fronts nothing.
|
||||
- **The front door is a single upstream.** [[caddy]] on utility CT 101 sends all of `navi.echo6.co` to `100.64.0.24:8440` (nginx inside recon-vm, which fans out to the gunicorns, Valhalla, Nominatim and Photon). Cutover is four `reverse_proxy` lines in one Caddyfile block — no [[dns]] change, no [[authentik]] change. `apache2` on the VM is only a default vhost and fronts nothing.
|
||||
- **recon has no dependency on navi.** Grepping `/opt/recon` for every navi port (84xx, 8002, 8010, 2322) and hostname returns nothing, so splitting them creates no cross-host call path.
|
||||
|
|
|
|||
|
|
@ -3,14 +3,13 @@ title: CoreScope Silent Ingest Stall — Diagnosis and Fix
|
|||
type: runbook
|
||||
tags:
|
||||
- mesh
|
||||
- proxmox
|
||||
aliases: []
|
||||
related:
|
||||
- [[services]]
|
||||
- [[ct-runbook]]
|
||||
- [[peertube-sitemap-redis-oom]]
|
||||
- [[environment]]
|
||||
- [[meshcore-transport]]
|
||||
- [[edge2-boot-recovery]]
|
||||
- [[meshtastic-headscale-runbook]]
|
||||
- [[SESSION-HANDOFF-meshai-test]]
|
||||
- [[services]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# CoreScope Silent Ingest Stall — Diagnosis and Fix
|
||||
|
|
@ -129,7 +128,7 @@ Monitored every three hours for 26 hours after the fix: ingest never stalled, me
|
|||
|
||||
## If it ever recurs
|
||||
|
||||
More memory raises the ceiling; it does not cap growth. The next lever is to bound CoreScope itself in `/opt/corescope-data/config.json` — `packetStore.maxMemoryMB` around 320, `retentionHours` 48, and `runtime.maxMemoryMB` around 512 (that last one applies to **both** the server and the ingestor). The config is a **partial override**, so adding keys will not disturb `mqttSources`. Upstream documents this deployment case in issues #836 and #1010.
|
||||
More memory raises the ceiling; it does not cap growth. The next lever is to bound CoreScope itself in `/opt/corescope-data/config.json` — `packetStore.maxMemoryMB` around 320, `retentionHours` 48, and `runtime.maxMemoryMB` around 512 (that last one applies to **both** the server and the ingestor). The config is a **partial override**, so adding keys will not disturb `mqttSources`. Upstream documents this [[deployment]] case in issues #836 and #1010.
|
||||
|
||||
A flat-`tx_inserted` self-heal watchdog on CT 100 is worth building if the pattern returns, since the container's own watchdog demonstrably stops trying.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,19 +5,18 @@ tags:
|
|||
- proxmox
|
||||
aliases: []
|
||||
related:
|
||||
- [[expose-service-edge2]]
|
||||
- [[proxmox-onboard-node]]
|
||||
- [[expose-service-edge2]]
|
||||
- [[lxc-service-migration]]
|
||||
- [[authentik]]
|
||||
- [[headscale-onboard-node]]
|
||||
- [[edge2-boot-recovery]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# edge2 Access Reference
|
||||
|
||||
## SSH Access
|
||||
|
||||
edge2 is hardened differently from home-cluster Proxmox nodes. If you are here because edge2 rebooted and its services are returning 502, go to [[edge2-boot-recovery]] instead.
|
||||
edge2 is hardened differently from home-cluster Proxmox nodes. If you are here because edge2 rebooted and its [[services]] are returning 502, go to [[edge2-boot-recovery]] instead.
|
||||
|
||||
| Property | edge2 | Home Proxmox (data, utility, etc.) |
|
||||
|----------|-------|------------------------------------|
|
||||
|
|
|
|||
|
|
@ -2,22 +2,19 @@
|
|||
title: edge2 Boot Recovery — compose-reconcile Tailscale Race
|
||||
type: runbook
|
||||
tags:
|
||||
- proxmox
|
||||
- auth
|
||||
- mesh
|
||||
aliases: []
|
||||
related:
|
||||
- [[edge2-access-reference]]
|
||||
- [[expose-service-edge2]]
|
||||
- [[authentik]]
|
||||
- [[caddy]]
|
||||
- [[ct-runbook]]
|
||||
- [[deploy-livesync]]
|
||||
- [[matrix-synapse-deployment]]
|
||||
- [[headscale-oidc-boot-order]]
|
||||
- [[lxc-service-migration]]
|
||||
- [[services]]
|
||||
- [[edge2-access-reference]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# edge2 Boot Recovery — compose-reconcile Tailscale Race
|
||||
|
||||
edge2 reboots and most of its services never come back. Every vhost behind it returns 502 while all nine containers report `running`. Hit on 2026-08-14 for 2h05m. Root cause found and fixed the same day.
|
||||
edge2 reboots and most of its [[services]] never come back. Every vhost behind it returns 502 while all nine containers report `running`. Hit on 2026-08-14 for 2h05m. Root cause found and fixed the same day.
|
||||
|
||||
The June 2026 `compose-reconcile.service` was built to prevent exactly this and had never been tested by a real reboot. This was its first, and it lost.
|
||||
|
||||
|
|
@ -39,7 +36,7 @@ ssh edge2 'sudo -n pct exec 105 -- docker ps -a --format "{{.Names}} | {{.Status
|
|||
```
|
||||
|
||||
- CT 102 vaultwarden, CT 103 forgejo, CT 104 livesync: containers stuck in **`Created`**, never started
|
||||
- CT 105 authentik, CT 106 matrix: app containers crash-looping on `Temporary failure in name resolution` for their compose-sibling database hostnames (`postgresql`, `matrix-postgres`)
|
||||
- CT 105 [[authentik]], CT 106 matrix: app containers crash-looping on `Temporary failure in name resolution` for their compose-sibling database hostnames (`postgresql`, `matrix-postgres`)
|
||||
|
||||
**[[authentik]] being down 502s everything behind forward-auth**, including services that are themselves perfectly healthy. On 2026-08-14 this is how the outage surfaced: `mesh.echo6.co` returned 502, but MeshMonitor was fine the entire time — only its login was broken. Check `auth.echo6.co` before diagnosing any individual service.
|
||||
|
||||
|
|
@ -137,7 +134,7 @@ edge2 died mid-operation at **01:12:45 UTC** after 46 days of uptime and returne
|
|||
- **No panic, oops, OOM, I/O error, hung task or soft lockup** anywhere in the previous boot; the last kernel message was a routine hourly `drop_caches` 56 minutes earlier
|
||||
- **Not the watchdog** — edge2 is not clustered, so there is no HA and `/dev/watchdog` was never armed
|
||||
- **No resource pressure** — last full sample before death: load 3.38, CPU 15.9%, iowait 2.43%, memory 6.2 GB of 25 GB
|
||||
- **Not datacenter-wide** — edge1, the other Contabo VPS, had been up since 2026-06-19 and sailed straight through
|
||||
- **Not datacenter-wide** — edge1, the other Contabo VPS, had been up since [[2026-06-19]] and sailed straight through
|
||||
- **No migration signature** — identical CPU (AMD EPYC family 0x17 model 0x1 stepping 0x2) and memory map before and after
|
||||
|
||||
The boot was a full cold SeaBIOS re-init, so the VM was destroyed and recreated at hypervisor level. Contabo's status page listed no incident, but they do not publish single-node resets.
|
||||
|
|
|
|||
|
|
@ -1,185 +1,191 @@
|
|||
---
|
||||
title: Lift navi off recon-vm onto media
|
||||
title: Separate navi onto its own VM on media
|
||||
type: runbook
|
||||
tags:
|
||||
- media
|
||||
- proxmox
|
||||
- recon
|
||||
aliases: []
|
||||
related: []
|
||||
related:
|
||||
- [[navi-recon-separation]]
|
||||
- [[fleet-storage-memory-upgrade]]
|
||||
- [[navi]]
|
||||
- [[deployment]]
|
||||
- [[environment]]
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# Lift navi off recon-vm onto media
|
||||
# Separate navi onto its own VM on media
|
||||
|
||||
**Status: planned, not executed.** Nothing below has been run.
|
||||
**Status: Phase 0 complete. Nothing from Phase 1 onward has been run.**
|
||||
|
||||
[[navi]] and [[recon]] share one VM (`recon-vm`, VMID 1130 on data), so neither can move without the other — see [[navi-recon-separation]]. This lifts the whole VM to media as a clone, then strips each copy down to one platform. No services are rebuilt.
|
||||
Splits [[navi]] out of `recon-vm` (VMID 1130 on data) into its own VM on media, taking **all live data onto local NVMe**. `recon-vm` keeps running throughout — it is never stopped. Background on the entanglement is [[navi-recon-separation]].
|
||||
|
||||
Target is media, which now has 64 GB RAM and a 1.82 TB NVMe volume — see [[fleet-storage-memory-upgrade]].
|
||||
## What moves and why
|
||||
|
||||
Two datasets are live for [[navi]], on two different slow tiers:
|
||||
|
||||
| Data | Size | Currently on | Problem |
|
||||
|---|---|---|---|
|
||||
| `nav` | 625 GB | data's SATA SSD (virtiofs) | SATA, and the volume is 93% full |
|
||||
| `planet-dem.pmtiles` | **658 GB** | **pi-nas over NFS** | 7200 rpm spinning disk across 1 GbE |
|
||||
|
||||
The DEM is the worse of the two. It is read on **every elevation query** (`NAVI_DEM_PMTILES` in `navi-geo.env`) *and* served **directly to browsers** for terrain (`alias /mnt/nas/nav/planet-dem.pmtiles` in `sites-available/dem-origin`). There is already a `dem-cache.conf` in nginx built specifically to paper over the NFS latency.
|
||||
|
||||
**The remaining ~540 GB on pi-nas is build material and stays there** — `contour-rebuild` 206 GB, `worldcover` 116 GB, `hillshade-work` 94 GB, `planet-latest.osm.pbf` 87 GB, plus dumps and staging. Nothing reads those at runtime.
|
||||
|
||||
**[[recon]] keeps `kiwix` (138 GB) and `library` (59 GB)** on data. Once `nav` leaves, data's SATA SSD drops from **93% to about 21%**.
|
||||
|
||||
## Sizing on media's 1.82 TB
|
||||
|
||||
| Volume | Virtual | Actual |
|
||||
|---|---|---|
|
||||
| `scsi0` OS (restored) | 180 GB | ~179 GB |
|
||||
| `scsi1` `/data/nav` | 750 GB | 625 GB |
|
||||
| `scsi2` `/data/dem` | 700 GB | 658 GB |
|
||||
| **Total** | **1.63 TB (90%)** | **1.46 TB (80%)** |
|
||||
|
||||
qcow2 is sparse, so actual consumption is what matters day to day — but the fully-allocated worst case still fits, which is the point of sizing it this way. **Keep contour rebuilds and other build work on pi-nas**; there is not room for them here.
|
||||
|
||||
Memory: navi-vm at 32 GB puts media at 49 of 62 GB allocated, leaving room for the viewshed budget from [[navi-recon-separation]].
|
||||
|
||||
---
|
||||
|
||||
## Why lift rather than rebuild
|
||||
## Phase 0 — done
|
||||
|
||||
navi is ~20 interlocking services: seven `navi-*` gunicorns, nginx:8440, Valhalla and Nominatim in Docker, Photon (Java), and PostgreSQL 16 whose `data_directory` sits at `/mnt/nav/overture/pgdata`. Standing that up fresh means reproducing every version, path and config. Cloning carries all of it intact; you then delete what doesn't belong.
|
||||
- **106 GB archive** at `pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst`, verified. Rollback point.
|
||||
- **Baseline 9/9** on the navi suite; map renders headless.
|
||||
- `discard=on` staged on 1130's `scsi0` — applies on its next restart, which this runbook does **not** trigger.
|
||||
|
||||
Cost: the clone inherits recon's files too, which you remove afterwards, and it inherits the source VM's identity, which **must** be fixed before it ever touches the network.
|
||||
`navi_test.sh` is `BASE=` parameterised and checks page shell, health, geocode, place details, landclass, Valhalla height, Valhalla route and both pmtiles range reads. **Run it plus a headless screenshot after every phase.**
|
||||
|
||||
---
|
||||
|
||||
## Facts this plan relies on
|
||||
## Phase 1 — build navi-vm on media (no outage)
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Source | `recon-vm` VMID 1130 on data, 4 cores, 24 GB, 180 GB disk |
|
||||
| New VMID | **1131** (next free; 1130 is the highest in use) |
|
||||
| navi data | `/mnt/data/nav` — **625 GB** |
|
||||
| Front door | Caddy on utility CT 101 → `100.64.0.24:8440` (nginx in recon-vm) |
|
||||
| Auth | Authentik forward_auth at the Caddy layer — unchanged by this |
|
||||
| Space needed on media | ~180 GB system + 625 GB data = **~805 GB** of 1.82 TB |
|
||||
|
||||
**recon does not call navi.** Grepping `/opt/recon` for navi ports (84xx, 8002, 8010, 2322) and hostnames returns nothing. The split creates no cross-host dependency.
|
||||
|
||||
**Stays on data:** `recon.service` (8420), `recon-watchdog`, `kiwix-serve` (8430), and the `library` + `kiwix` shares.
|
||||
|
||||
---
|
||||
|
||||
## The three traps
|
||||
|
||||
**1. Identity collision.** The clone is byte-identical: same static IP `192.168.1.130`, same hostname, same `machine-id`, same Tailscale node key. Booting it on the network unfixed will fight the original for its IP and its tailnet identity. **Fix identity before first network boot** (step 3).
|
||||
|
||||
**2. The virtiofs `args:` will not start on media.** `recon-vm` carries raw QEMU args wiring `vhost-user-fs-pci` devices to `/run/virtiofsd-{nav,kiwix,library}.sock`, plus a `memory-backend-memfd` object that virtiofs requires. Those sockets do not exist on media, so the clone fails to start as-is.
|
||||
|
||||
Rather than recreate virtiofsd on media, **give the new VM a real virtual data disk**. That makes it self-contained — no host-side sockets, no raw args — and therefore movable with [[pve-guest-park-and-adopt]], which the current VM is not.
|
||||
|
||||
**3. PostgreSQL moves with the data.** `pgdata` lives inside `/mnt/nav`, so it travels with the 625 GB. Nothing separate to dump and restore, but the copy must be consistent — stop postgres before the final sync (step 5).
|
||||
|
||||
---
|
||||
|
||||
## 1. Back up the source
|
||||
|
||||
```bash
|
||||
ssh ts-data
|
||||
vzdump 1130 --mode snapshot --storage pinas-backup --compress zstd
|
||||
```
|
||||
|
||||
`--mode snapshot` runs live — no downtime. This is both the rollback point and the clone source. Note the archive name.
|
||||
|
||||
## 2. Restore as VMID 1131 on media
|
||||
Restore the archive as VMID 1131. This is a lift — every navi service arrives already configured.
|
||||
|
||||
```bash
|
||||
ssh ts-media
|
||||
qmrestore pinas-backup:backup/vzdump-qemu-1130-<ts>.vma.zst 1131 --storage media-nvme
|
||||
qmrestore pinas-backup:backup/vzdump-qemu-1130-2026_08_15-00_07_11.vma.zst 1131 \
|
||||
--storage media-nvme
|
||||
```
|
||||
|
||||
`media-nvme` is the 1.82 TB volume. **Do not start it yet.**
|
||||
|
||||
## 3. Fix identity before it ever boots on the network
|
||||
**Do not start it yet** — it is byte-identical to the running VM.
|
||||
|
||||
```bash
|
||||
qm set 1131 --name navi-vm
|
||||
qm set 1131 --delete args # drop virtiofs devices + memory-backend-memfd
|
||||
qm set 1131 --name navi-vm --memory 32768
|
||||
qm set 1131 --ipconfig0 ip=192.168.1.132/24,gw=192.168.1.1
|
||||
qm set 1131 --memory 32768 # navi + viewshed headroom
|
||||
qm set 1131 --delete args # drop virtiofs wiring
|
||||
```
|
||||
|
||||
Removing `args` also removes the `memory-backend-memfd` object, which is only needed for virtiofs.
|
||||
|
||||
Then boot **with the NIC detached**, and from the console clear the inherited identity:
|
||||
|
||||
```bash
|
||||
qm set 1131 --net0 virtio,bridge=vmbr0,link_down=1
|
||||
qm set 1131 --scsi1 media-nvme:750,format=qcow2,discard=on,ssd=1
|
||||
qm set 1131 --scsi2 media-nvme:700,format=qcow2,discard=on,ssd=1
|
||||
qm start 1131
|
||||
# on the console:
|
||||
tailscale logout && systemctl stop tailscaled
|
||||
```
|
||||
|
||||
From the console, with the link still down:
|
||||
|
||||
```bash
|
||||
tailscale logout; systemctl stop tailscaled
|
||||
rm -f /var/lib/tailscale/tailscaled.state
|
||||
hostnamectl set-hostname navi-vm
|
||||
rm -f /etc/machine-id && systemd-machine-id-setup
|
||||
hostnamectl set-hostname navi-vm
|
||||
sed -i '/virtiofs/d;/mnt\/nas/d' /etc/fstab # old share + pi-nas mounts
|
||||
|
||||
mkfs.ext4 -L navdata /dev/sdb && mkfs.ext4 -L demdata /dev/sdc
|
||||
mkdir -p /data/nav /data/dem
|
||||
cat >> /etc/fstab <<'EOF'
|
||||
LABEL=navdata /data/nav ext4 defaults,noatime,nofail 0 2
|
||||
LABEL=demdata /data/dem ext4 defaults,noatime,nofail 0 2
|
||||
/data/nav /mnt/nav none bind,nofail 0 0
|
||||
EOF
|
||||
mount -a
|
||||
```
|
||||
|
||||
Only then reattach the NIC (`qm set 1131 --net0 virtio,bridge=vmbr0`) and re-register Tailscale as a new node.
|
||||
The bind keeps `/mnt/nav` valid so **no navi service config changes** — except the two DEM references, handled in Phase 3.
|
||||
|
||||
## 4. Attach and prepare the data disk
|
||||
Then bring the NIC up (`qm set 1131 --net0 virtio,bridge=vmbr0`) and re-register Tailscale as a **new** node.
|
||||
|
||||
## Phase 2 — copy 1.28 TB (no outage)
|
||||
|
||||
`recon-vm` keeps serving throughout.
|
||||
|
||||
```bash
|
||||
qm set 1131 --scsi1 media-nvme:700,format=qcow2
|
||||
# inside the guest:
|
||||
mkfs.ext4 -L nav /dev/sdb
|
||||
mkdir -p /mnt/nav
|
||||
echo "LABEL=nav /mnt/nav ext4 defaults,noatime,nofail 0 2" >> /etc/fstab
|
||||
mount /mnt/nav
|
||||
# nav, from data's SATA SSD
|
||||
rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /data/nav/
|
||||
|
||||
# DEM, straight from pi-nas rather than through recon-vm
|
||||
mkdir -p /mnt/pinas && mount -t nfs -o vers=3,ro 192.168.1.245:/export/data /mnt/pinas
|
||||
rsync -aHAX --info=progress2 /mnt/pinas/nav/planet-dem.pmtiles /data/dem/
|
||||
umount /mnt/pinas
|
||||
```
|
||||
|
||||
700 GB gives the 625 GB room to grow. Also remove the three virtiofs lines from the guest's `/etc/fstab`, which will otherwise fail on boot.
|
||||
Expect **4–6 hours total**. The DEM leg is bounded by pi-nas's spinning disk, not the network.
|
||||
|
||||
## 5. Copy the 625 GB
|
||||
## Phase 3 — repoint navi at local storage
|
||||
|
||||
First pass live, while navi keeps serving from data:
|
||||
On **navi-vm**, the only config edits in the whole migration:
|
||||
|
||||
```bash
|
||||
rsync -aHAX --info=progress2 zvx@100.64.0.24:/mnt/nav/ /mnt/nav/
|
||||
sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
|
||||
/etc/navi-backend/navi-geo.env
|
||||
sed -i 's#/mnt/nas/nav/planet-dem.pmtiles#/data/dem/planet-dem.pmtiles#' \
|
||||
/etc/nginx/sites-available/dem-origin
|
||||
grep -rn "/mnt/nas" /etc/navi-backend/ /etc/nginx/sites-enabled/ # expect nothing
|
||||
nginx -t && systemctl restart nginx navi-geo navi-offroute
|
||||
```
|
||||
|
||||
Expect 2–3 hours over 1 GbE. Then stop the writers on the source and take a final delta:
|
||||
Strip recon and kiwix from navi-vm — they came along with the clone:
|
||||
|
||||
```bash
|
||||
# on recon-vm
|
||||
systemctl disable --now recon recon-watchdog kiwix
|
||||
```
|
||||
|
||||
Leave `/opt/recon` on disk for now; deleting it is a Phase 6 task.
|
||||
|
||||
## Phase 4 — verify before exposing
|
||||
|
||||
Test the new VM **directly**, before touching the front door:
|
||||
|
||||
```bash
|
||||
BASE=http://<navi-vm-tailnet-ip>:8440 ./navi_test.sh
|
||||
```
|
||||
|
||||
Do not proceed on anything less than **9/9**. Valhalla needs ~1 minute after boot to load 32,396 tiles; 502s before that are expected.
|
||||
|
||||
## Phase 5 — cutover (navi outage only, ~2 minutes)
|
||||
|
||||
`recon-vm` stays **running**. Only navi's [[services]] stop.
|
||||
|
||||
```bash
|
||||
# on recon-vm — quiesce navi writers so the delta is consistent
|
||||
systemctl stop navi-geo navi-landclass navi-offroute navi-places \
|
||||
navi-contacts navi-admin navi-config photon argus-resolver
|
||||
docker stop nominatim-v5 valhalla
|
||||
systemctl stop postgresql@16-main
|
||||
# on navi-vm
|
||||
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /mnt/nav/
|
||||
|
||||
# final delta (nav only; the DEM is static)
|
||||
rsync -aHAX --delete zvx@100.64.0.24:/mnt/nav/ /data/nav/
|
||||
```
|
||||
|
||||
The postgres stop is what makes `pgdata` consistent. Do not skip it.
|
||||
|
||||
## 6. Strip each side
|
||||
|
||||
On **navi-vm** (1131) — remove recon and kiwix:
|
||||
Start navi's stack on navi-vm, then repoint [[caddy]] on utility CT 101 — **four `reverse_proxy` lines** in the `navi.echo6.co` block (`@authed_api`, `@public_api`, `@valhalla`, and the fallthrough) — at navi-vm on 8440.
|
||||
|
||||
```bash
|
||||
systemctl disable --now recon recon-watchdog kiwix
|
||||
rm -rf /opt/recon
|
||||
```
|
||||
|
||||
On **recon-vm** (1130) — remove navi and the geo stack:
|
||||
|
||||
```bash
|
||||
systemctl disable --now navi-geo navi-landclass navi-offroute navi-places \
|
||||
navi-contacts navi-admin navi-config photon argus-resolver postgresql@16-main
|
||||
docker rm -f nominatim-v5 valhalla
|
||||
```
|
||||
|
||||
Leave `/mnt/data/nav` on data untouched until step 9.
|
||||
|
||||
## 7. Cutover — one line
|
||||
|
||||
In the Caddyfile on utility CT 101, inside the `navi.echo6.co` block, repoint every `reverse_proxy` from `100.64.0.24:8440` to the new VM's tailnet address on 8440. There are four occurrences (`@authed_api`, `@public_api`, `@valhalla`, and the fallthrough `handle`).
|
||||
|
||||
```bash
|
||||
ssh ts-utility
|
||||
pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile
|
||||
pct exec 101 -- systemctl reload caddy
|
||||
```
|
||||
|
||||
DNS and Authentik are untouched — the hostname and auth flow do not change.
|
||||
[[dns]] and [[authentik]] are untouched.
|
||||
|
||||
## 8. Verify
|
||||
## Phase 6 — reclaim (DEFERRED — do not run)
|
||||
|
||||
Confirm navi answers through the front door, not just that the VM runs — check the map loads, a geocode returns, and a route computes. Follow [[headless-browser-page-verification]] for the UI. Then confirm recon is still healthy on 8420 and kiwix on 8430.
|
||||
Only after navi has been healthy on media for **at least a week**:
|
||||
|
||||
## 9. Reclaim — a week later, not the same day
|
||||
- `rm -rf /mnt/data/nav` on data → SATA SSD 93% → ~21%
|
||||
- `systemctl disable --now virtiofsd-nav` on data
|
||||
- `rm -rf /opt/recon` on navi-vm
|
||||
- Drop 1130's memory from 24 GB to ~8 GB
|
||||
- Optionally remove the 658 GB DEM from pi-nas once the local copy is proven
|
||||
|
||||
Once navi has been healthy on media for several days:
|
||||
|
||||
```bash
|
||||
# on data
|
||||
rm -rf /mnt/data/nav
|
||||
systemctl disable --now virtiofsd-nav
|
||||
qm set 1130 --memory 8192 # recon alone needs far less than 24 GB
|
||||
```
|
||||
|
||||
That drops `/mnt/data` from 93% to roughly 21% and frees ~16 GB of RAM on data — which matters, since data is hard-capped at 32 GB.
|
||||
**Keep the source data and VM 1130 intact until then.** That is the rollback path.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -187,8 +193,16 @@ That drops `/mnt/data` from 93% to roughly 21% and frees ~16 GB of RAM on data
|
|||
|
||||
| Stage | Recovery |
|
||||
|---|---|
|
||||
| Any point before step 7 | Nothing changed for users — navi still serves from data |
|
||||
| After cutover | Revert the four Caddy lines to `100.64.0.24:8440`, restart navi's services on recon-vm |
|
||||
| Catastrophic | `qmrestore` the step 1 archive from `pinas-backup` |
|
||||
| Phases 1–4 | Nothing changed — navi still served from recon-vm |
|
||||
| After Phase 5 | Revert the four Caddy lines, restart navi's services on 1130 |
|
||||
| Catastrophic | `qmrestore` the Phase 0 archive |
|
||||
|
||||
The rollback path stays open until step 9 deletes the source data. Do not run step 9 early.
|
||||
---
|
||||
|
||||
## Known traps
|
||||
|
||||
**`virtiofsd` dies with the VM.** Stopping VM 1130 leaves `virtiofsd-{nav,kiwix,library}.service` inactive, and it then **refuses to start** with `Failed to connect to /run/virtiofsd-*.sock`. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this.
|
||||
|
||||
**nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if DNS is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable.
|
||||
|
||||
**Do not restart recon-vm.** recon and navi share it until Phase 5 completes, so any restart takes both down.
|
||||
|
|
|
|||
|
|
@ -6,11 +6,11 @@ tags:
|
|||
aliases: []
|
||||
related:
|
||||
- [[add-peertube-channel]]
|
||||
- [[caddy]]
|
||||
- [[peertube-remote-runner]]
|
||||
- [[caddy]]
|
||||
- [[corescope-ingest-stall-oom]]
|
||||
- [[central]]
|
||||
- [[recon-operations]]
|
||||
updated: 2026-08-14
|
||||
updated: 2026-08-15
|
||||
---
|
||||
# PeerTube Sitemap Redis OOM — Diagnosis and Fix
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue