auto: docs sync 2026-08-16T12:00:06+00:00

Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/projects/fleet-storage-memory-upgrade.md vault/runbooks/navi-lift-to-media.md
This commit is contained in:
echo6-autocommit 2026-08-16 12:00:06 +00:00
commit 4e12cc3352
5 changed files with 18 additions and 12 deletions

File diff suppressed because one or more lines are too long

View file

@ -253,3 +253,9 @@
- mode: incremental - mode: incremental
- docs selected: 8 - docs selected: 8
- processed: 8 | written: 8 | flagged: 0 | errors: 0 - processed: 8 | written: 8 | flagged: 0 | errors: 0
## 2026-08-16T09:00:01Z — sweep run
- end: 2026-08-16T09:00:10Z
- mode: incremental
- docs selected: 2
- processed: 2 | written: 2 | flagged: 0 | errors: 0

View file

@ -1,6 +1,6 @@
# Vault Lint Report # Vault Lint Report
Generated: 2026-08-16T00:00:04Z | Docs scanned: 115 | Elapsed: 0.0s Generated: 2026-08-16T09:00:01Z | Docs scanned: 115 | Elapsed: 0.0s
## Summary ## Summary

View file

@ -6,11 +6,11 @@ tags:
aliases: [] aliases: []
related: related:
- [[environment]] - [[environment]]
- [[fleet-platform-baseline]]
- [[navi-lift-to-media]] - [[navi-lift-to-media]]
- [[toc-cortex-pve9.2-update]] - [[fleet-platform-baseline]]
- [[navi-recon-separation]] - [[navi-recon-separation]]
updated: 2026-08-15 - [[toc-cortex-pve9.2-update]]
updated: 2026-08-16
--- ---
# Fleet Storage and Memory Upgrade # Fleet Storage and Memory Upgrade

View file

@ -8,16 +8,16 @@ aliases: []
related: related:
- [[navi-recon-separation]] - [[navi-recon-separation]]
- [[fleet-storage-memory-upgrade]] - [[fleet-storage-memory-upgrade]]
- [[environment]]
- [[navi]] - [[navi]]
- [[deployment]] - [[deployment]]
- [[environment]] updated: 2026-08-16
updated: 2026-08-15
--- ---
# Separate navi onto its own VM on media # Separate navi onto its own VM on media
**Status: COMPLETE through Phase 5 (2026-08-15). Phase 6 deliberately NOT run — rollback stays open.** **Status: COMPLETE through Phase 5 (2026-08-15). Phase 6 deliberately NOT run — rollback stays open.**
navi now runs on **navi-vm (VMID 1131) on media**, tailnet `100.64.0.27`, LAN `192.168.1.132`. Cutover outage was **2 min 10 s** (20:56:4720:58:57 UTC), navi only — `recon-vm` was never stopped and recon/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers. [[navi]] now runs on **navi-vm (VMID 1131) on media**, tailnet `100.64.0.27`, LAN `192.168.1.132`. Cutover outage was **2 min 10 s** (20:56:4720:58:57 UTC), navi only — `recon-vm` was never stopped and [[recon]]/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers.
Splits [[navi]] out of `recon-vm` (VMID 1130 on data) into its own VM on media, taking **all live data onto local NVMe**. `recon-vm` keeps running throughout — it is never stopped. Background on the entanglement is [[navi-recon-separation]]. Splits [[navi]] out of `recon-vm` (VMID 1130 on data) into its own VM on media, taking **all live data onto local NVMe**. `recon-vm` keeps running throughout — it is never stopped. Background on the entanglement is [[navi-recon-separation]].
@ -196,14 +196,14 @@ Only after navi has been healthy on media for **at least a week**:
| Stage | Recovery | | Stage | Recovery |
|---|---| |---|---|
| Phases 14 | Nothing changed — navi still served from recon-vm | | Phases 14 | Nothing changed — navi still served from recon-vm |
| After Phase 5 | Revert the four Caddy lines, restart navi's services on 1130 | | After Phase 5 | Revert the four [[caddy]] lines, restart navi's [[services]] on 1130 |
| Catastrophic | `qmrestore` the Phase 0 archive | | Catastrophic | `qmrestore` the Phase 0 archive |
--- ---
## What actually bit us (2026-08-15 run) ## What actually bit us (2026-08-15 run)
**`systemctl start` is not `restart`.** Phase 3 ran twice; the first attempt started the navi services *before* the second DEM reference in `navi-offroute.env` was found and fixed. The second run used `start`, which is a no-op on a running unit, so `navi-offroute` kept the stale environment and threw `FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles'` on every terrain request while the on-disk config looked correct. **Verify the live process environment, not the config file:** `tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>`. **`systemctl start` is not `restart`.** Phase 3 ran twice; the first attempt started the navi services *before* the second DEM reference in `navi-offroute.env` was found and fixed. The second run used `start`, which is a no-op on a running unit, so `navi-offroute` kept the stale [[environment]] and threw `FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles'` on every terrain request while the on-disk config looked correct. **Verify the live process environment, not the config file:** `tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>`.
**Two files reference the DEM, not one** — `navi-geo.env` *and* `navi-offroute.env`. **Two files reference the DEM, not one** — `navi-geo.env` *and* `navi-offroute.env`.
@ -211,7 +211,7 @@ Only after navi has been healthy on media for **at least a week**:
**qmrestore clones the MAC and IP.** The restored VM carried recon-vm's `BC:24:11:07:B0:F9` and its cloud-init IP. Both were changed with the NIC `link_down=1`, using the QEMU guest agent over virtio-serial, before it ever saw the network. **qmrestore clones the MAC and IP.** The restored VM carried recon-vm's `BC:24:11:07:B0:F9` and its cloud-init IP. Both were changed with the NIC `link_down=1`, using the QEMU guest agent over virtio-serial, before it ever saw the network.
**Tailscale must be re-registered before nginx will start.** nginx has an upstream on `central.echo6.mesh`, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, `nginx -t` failed with `host not found in upstream`. (That upstream is fine, incidentally — CT 104 kept the name "central" but runs Conduit.) **Tailscale must be re-registered before nginx will start.** nginx has an upstream on `central.echo6.mesh`, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, `nginx -t` failed with `host not found in upstream`. (That upstream is fine, incidentally — CT 104 kept the name "[[central]]" but runs Conduit.)
**A size guard cost a re-run.** `658 GiB` is `705,591,064,874` decimal bytes; a `> 700000000000` sanity check rejected a byte-perfect copy. **A size guard cost a re-run.** `658 GiB` is `705,591,064,874` decimal bytes; a `> 700000000000` sanity check rejected a byte-perfect copy.
@ -219,6 +219,6 @@ Only after navi has been healthy on media for **at least a week**:
**`virtiofsd` dies with the VM.** Stopping VM 1130 leaves `virtiofsd-{nav,kiwix,library}.service` inactive, and it then **refuses to start** with `Failed to connect to /run/virtiofsd-*.sock`. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this. **`virtiofsd` dies with the VM.** Stopping VM 1130 leaves `virtiofsd-{nav,kiwix,library}.service` inactive, and it then **refuses to start** with `Failed to connect to /run/virtiofsd-*.sock`. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this.
**nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if DNS is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable. **nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if [[dns]] is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable.
**Do not restart recon-vm.** recon and navi share it until Phase 5 completes, so any restart takes both down. **Do not restart recon-vm.** recon and navi share it until Phase 5 completes, so any restart takes both down.