auto: docs sync 2026-08-16T12:00:06+00:00
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/projects/fleet-storage-memory-upgrade.md vault/runbooks/navi-lift-to-media.md
This commit is contained in:
parent
c6e1817b14
commit
4e12cc3352
5 changed files with 18 additions and 12 deletions
File diff suppressed because one or more lines are too long
|
|
@ -253,3 +253,9 @@
|
|||
- mode: incremental
|
||||
- docs selected: 8
|
||||
- processed: 8 | written: 8 | flagged: 0 | errors: 0
|
||||
|
||||
## 2026-08-16T09:00:01Z — sweep run
|
||||
- end: 2026-08-16T09:00:10Z
|
||||
- mode: incremental
|
||||
- docs selected: 2
|
||||
- processed: 2 | written: 2 | flagged: 0 | errors: 0
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Vault Lint Report
|
||||
|
||||
Generated: 2026-08-16T00:00:04Z | Docs scanned: 115 | Elapsed: 0.0s
|
||||
Generated: 2026-08-16T09:00:01Z | Docs scanned: 115 | Elapsed: 0.0s
|
||||
|
||||
## Summary
|
||||
|
||||
|
|
|
|||
|
|
@ -6,11 +6,11 @@ tags:
|
|||
aliases: []
|
||||
related:
|
||||
- [[environment]]
|
||||
- [[fleet-platform-baseline]]
|
||||
- [[navi-lift-to-media]]
|
||||
- [[toc-cortex-pve9.2-update]]
|
||||
- [[fleet-platform-baseline]]
|
||||
- [[navi-recon-separation]]
|
||||
updated: 2026-08-15
|
||||
- [[toc-cortex-pve9.2-update]]
|
||||
updated: 2026-08-16
|
||||
---
|
||||
# Fleet Storage and Memory Upgrade
|
||||
|
||||
|
|
|
|||
|
|
@ -8,16 +8,16 @@ aliases: []
|
|||
related:
|
||||
- [[navi-recon-separation]]
|
||||
- [[fleet-storage-memory-upgrade]]
|
||||
- [[environment]]
|
||||
- [[navi]]
|
||||
- [[deployment]]
|
||||
- [[environment]]
|
||||
updated: 2026-08-15
|
||||
updated: 2026-08-16
|
||||
---
|
||||
# Separate navi onto its own VM on media
|
||||
|
||||
**Status: COMPLETE through Phase 5 (2026-08-15). Phase 6 deliberately NOT run — rollback stays open.**
|
||||
|
||||
navi now runs on **navi-vm (VMID 1131) on media**, tailnet `100.64.0.27`, LAN `192.168.1.132`. Cutover outage was **2 min 10 s** (20:56:47–20:58:57 UTC), navi only — `recon-vm` was never stopped and recon/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers.
|
||||
[[navi]] now runs on **navi-vm (VMID 1131) on media**, tailnet `100.64.0.27`, LAN `192.168.1.132`. Cutover outage was **2 min 10 s** (20:56:47–20:58:57 UTC), navi only — `recon-vm` was never stopped and [[recon]]/kiwix stayed up throughout. Verified 9/9 on the functional suite plus headless confirmation of map render, search, place details, DEM elevation, and the USFS/BLM/Public Lands layers.
|
||||
|
||||
Splits [[navi]] out of `recon-vm` (VMID 1130 on data) into its own VM on media, taking **all live data onto local NVMe**. `recon-vm` keeps running throughout — it is never stopped. Background on the entanglement is [[navi-recon-separation]].
|
||||
|
||||
|
|
@ -196,14 +196,14 @@ Only after navi has been healthy on media for **at least a week**:
|
|||
| Stage | Recovery |
|
||||
|---|---|
|
||||
| Phases 1–4 | Nothing changed — navi still served from recon-vm |
|
||||
| After Phase 5 | Revert the four Caddy lines, restart navi's services on 1130 |
|
||||
| After Phase 5 | Revert the four [[caddy]] lines, restart navi's [[services]] on 1130 |
|
||||
| Catastrophic | `qmrestore` the Phase 0 archive |
|
||||
|
||||
---
|
||||
|
||||
## What actually bit us (2026-08-15 run)
|
||||
|
||||
**`systemctl start` is not `restart`.** Phase 3 ran twice; the first attempt started the navi services *before* the second DEM reference in `navi-offroute.env` was found and fixed. The second run used `start`, which is a no-op on a running unit, so `navi-offroute` kept the stale environment and threw `FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles'` on every terrain request while the on-disk config looked correct. **Verify the live process environment, not the config file:** `tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>`.
|
||||
**`systemctl start` is not `restart`.** Phase 3 ran twice; the first attempt started the navi services *before* the second DEM reference in `navi-offroute.env` was found and fixed. The second run used `start`, which is a no-op on a running unit, so `navi-offroute` kept the stale [[environment]] and threw `FileNotFoundError: '/mnt/nas/nav/planet-dem.pmtiles'` on every terrain request while the on-disk config looked correct. **Verify the live process environment, not the config file:** `tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value <svc>)/environ | grep <VAR>`.
|
||||
|
||||
**Two files reference the DEM, not one** — `navi-geo.env` *and* `navi-offroute.env`.
|
||||
|
||||
|
|
@ -211,7 +211,7 @@ Only after navi has been healthy on media for **at least a week**:
|
|||
|
||||
**qmrestore clones the MAC and IP.** The restored VM carried recon-vm's `BC:24:11:07:B0:F9` and its cloud-init IP. Both were changed with the NIC `link_down=1`, using the QEMU guest agent over virtio-serial, before it ever saw the network.
|
||||
|
||||
**Tailscale must be re-registered before nginx will start.** nginx has an upstream on `central.echo6.mesh`, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, `nginx -t` failed with `host not found in upstream`. (That upstream is fine, incidentally — CT 104 kept the name "central" but runs Conduit.)
|
||||
**Tailscale must be re-registered before nginx will start.** nginx has an upstream on `central.echo6.mesh`, which only resolves via MagicDNS. With tailscaled disabled during identity cleanup, `nginx -t` failed with `host not found in upstream`. (That upstream is fine, incidentally — CT 104 kept the name "[[central]]" but runs Conduit.)
|
||||
|
||||
**A size guard cost a re-run.** `658 GiB` is `705,591,064,874` decimal bytes; a `> 700000000000` sanity check rejected a byte-perfect copy.
|
||||
|
||||
|
|
@ -219,6 +219,6 @@ Only after navi has been healthy on media for **at least a week**:
|
|||
|
||||
**`virtiofsd` dies with the VM.** Stopping VM 1130 leaves `virtiofsd-{nav,kiwix,library}.service` inactive, and it then **refuses to start** with `Failed to connect to /run/virtiofsd-*.sock`. Restart all three first. Confirmed the hard way 2026-08-15. navi-vm has no virtiofs, so it never inherits this.
|
||||
|
||||
**nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if DNS is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable.
|
||||
**nginx fails on boot.** `satellite-cache.conf` resolves an upstream hostname at parse time; if [[dns]] is not ready, nginx dies and stays dead. It did exactly that from 2026-07-28 to 08-15 — **18 days of navi 502s, unnoticed**. Start it manually after any boot, and fix it properly with a `resolver` directive plus the upstream in a variable.
|
||||
|
||||
**Do not restart recon-vm.** recon and navi share it until Phase 5 completes, so any restart takes both down.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue