auto: docs sync 2026-08-14T12:00:03+00:00

Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/docs/services/services.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/omv-add-nfs-share.md vault/runbooks/peertube-sitemap-redis-oom.md vault/runbooks/pve-guest-park-and-adopt.md
2026-08-14 12:00:03 +00:00 · 2026-08-14 12:00:03 +00:00 · afaf019076
commit afaf019076
parent 873fbfc49d
11 changed files with 697 additions and 46 deletions
--- a/vault/runbooks/omv-add-nfs-share.md
+++ b/vault/runbooks/omv-add-nfs-share.md
@ -0,0 +1,147 @@
+---
+title: Add an NFS Share on pi-nas (OpenMediaVault RPC)
+type: runbook
+tags:
+  - storage
+aliases: []
+related:
+  - [[pi-nas-omv-runbook]]
+  - [[pve-guest-park-and-adopt]]
+  - [[proxmox-onboard-node]]
+  - [[proxmox-create-ubuntu-vm]]
+  - [[ct-runbook]]
+updated: 2026-08-14
+---
+# Add an NFS Share on pi-nas (OpenMediaVault RPC)
+
+pi-nas runs OpenMediaVault 8.4.0 (Synchrony). **Never hand-edit `/etc/exports` or `/etc/fstab` on this box.** Both are auto-generated from OMV's config database and carry the warning header; hand edits are silently reverted the next time any OMV config is applied, which will happen eventually whether or not you trigger it.
+
+Create shares through OMV's RPC instead. The web UI does the same thing, but RPC is scriptable and works over SSH.
+
+Initial NAS build is [[pi-nas-omv-runbook]].
+
+---
+
+## Key facts
+
+The **new-object sentinel UUID** is how OMV signals "create this, don't update":
+
+```
+fa4b1c66-ef79-11e5-87a0-0002b3a176b4
+```
+
+Resolve it yourself if ever in doubt:
+
+```bash
+sudo php -r 'require_once("/usr/share/php/openmediavault/autoloader.inc");
+  echo \OMV\Environment::get("OMV_CONFIGOBJECT_NEW_UUID"),"\n";'
+```
+
+**Filesystem references** (`mntentref`) as of 2026-08-14:
+
+| Disk | Size / role | mntentref |
+|---|---|---|
+| `sda`+`sdb` | 2.7 TB btrfs RAID1 — the only mirrored pool | `423365c2-513c-4900-bbde-a2e7b21e6bd5` |
+| `sdc1` | 24 TB ext4, single | `a3c6d71c-527d-45a3-aab7-1868e858456f` |
+| `sdd1` | 24 TB ext4, single — carries PeerTube | `79238c6b-1064-4ac9-9564-b12fc8608b74` |
+
+Re-read them at any time:
+
+```bash
+sudo omv-rpc -u admin "ShareMgmt" "getList" '{"start":0,"limit":-1}'
+```
+
+---
+
+## 1. Create the shared folder
+
+`reldirpath` is relative to the chosen filesystem's mount point and must end in `/`. It may not contain `..`.
+
+```bash
+sudo omv-rpc -u admin "ShareMgmt" "set" '{
+  "uuid":"fa4b1c66-ef79-11e5-87a0-0002b3a176b4",
+  "name":"pvebackup",
+  "reldirpath":"pvebackup/",
+  "comment":"Proxmox vzdump archives",
+  "mntentref":"a3c6d71c-527d-45a3-aab7-1868e858456f",
+  "mode":"775"
+}'
+```
+
+It returns the created object. **Keep the `uuid`** — the NFS share references it.
+
+## 2. Create the NFS share, once per client network
+
+Pass the sentinel for `mntentref` as well; OMV overwrites it with the bind-mount entry it creates for `/export/<name>`.
+
+`no_root_squash` is required for anything Proxmox writes as root, including `vzdump`. Omit it for shares that do not need it.
+
+```bash
+SF=<uuid from step 1>
+NEW=fa4b1c66-ef79-11e5-87a0-0002b3a176b4
+
+# LAN
+sudo omv-rpc -u admin "NFS" "setShare" "{
+  \"uuid\":\"$NEW\",\"sharedfolderref\":\"$SF\",\"mntentref\":\"$NEW\",
+  \"client\":\"192.168.1.0/24\",\"options\":\"rw\",
+  \"extraoptions\":\"subtree_check,insecure,no_root_squash\",
+  \"comment\":\"pvebackup - local\"}"
+
+# tailnet
+sudo omv-rpc -u admin "NFS" "setShare" "{
+  \"uuid\":\"$NEW\",\"sharedfolderref\":\"$SF\",\"mntentref\":\"$NEW\",
+  \"client\":\"100.64.0.0/10\",\"options\":\"rw\",
+  \"extraoptions\":\"subtree_check,insecure,no_root_squash\",
+  \"comment\":\"pvebackup - tailnet\"}"
+```
+
+`extraoptions` is pattern-validated: comma-separated words only, no spaces.
+
+## 3. Apply
+
+Scope it to the modules actually affected rather than applying everything:
+
+```bash
+sudo omv-rpc -u admin "Config" "applyChanges" '{"modules":["fstab","nfs"],"force":false}'
+```
+
+This regenerates `/etc/exports`, creates the bind mount, and re-runs `exportfs`.
+
+## 4. Verify — including that you broke nothing
+
+`exportfs` re-runs against live clients. Existing mounts survived this in practice, but confirm rather than assume:
+
+```bash
+# on pi-nas
+sudo exportfs -v | grep -A1 pvebackup
+df -h /export/pvebackup
+sudo exportfs -s | grep -oE '^/export/[a-z]+' | sort -u
+```
+
+Then check every existing consumer still reads. As of 2026-08-14 those are media (`/mnt/peertube-storage`) and cloud (`/mnt/nfs-immich`, `/mnt/nfs-nextcloud`) — see [[services]].
+
+---
+
+## Mounting it in Proxmox
+
+Storage config is cluster-wide, so run this once on any node and all five pick it up:
+
+```bash
+pvesm add nfs pinas-backup --server 192.168.1.245 \
+  --export /export/pvebackup --content backup --options vers=3
+```
+
+Confirm root can actually write, which is what `no_root_squash` buys:
+
+```bash
+dd if=/dev/zero of=/mnt/pve/pinas-backup/.writetest bs=1M count=32 && \
+  rm /mnt/pve/pinas-backup/.writetest && echo OK
+```
+
+Content types matter. `backup` is for vzdump archives; `images,rootdir` would let guests run directly from NFS, which was deliberately not done — see [[pve-guest-park-and-adopt]] for why.
+
+---
+
+## Redundancy warning
+
+Only `sda`+`sdb` are mirrored (btrfs RAID1, 2.7 TB, ~2.1 TB free). `sdc` and `sdd` are single ext4 drives with no redundancy despite holding the bulk of the data, including PeerTube's library. Put anything that must survive a drive failure on the mirrored pool.
--- a/vault/runbooks/peertube-sitemap-redis-oom.md
+++ b/vault/runbooks/peertube-sitemap-redis-oom.md
@ -0,0 +1,143 @@
+---
+title: PeerTube Sitemap Redis OOM — Diagnosis and Fix
+type: runbook
+tags:
+  - media
+aliases: []
+related:
+  - [[add-peertube-channel]]
+  - [[caddy]]
+  - [[peertube-remote-runner]]
+  - [[central]]
+  - [[recon-operations]]
+updated: 2026-08-14
+---
+# PeerTube Sitemap Redis OOM — Diagnosis and Fix
+
+stream.echo6.co goes down and media's load average climbs past 100 while the CPU sits mostly idle. Root cause found and fixed 2026-08-13.
+
+[[deployment]] layout is [[services]]; PeerTube lives in CT 110 on media.
+
+---
+
+## Signature
+
+Recognise it by this combination — the idle CPU is the tell:
+
+- media load average **>100** with CPU **~85% idle** and high iowait
+- `pct exec 110` hangs and returns nothing
+- Every process in the container — nginx, postgres, redis, sshd — stuck in `D` state
+- Other home [[services]] (echo6.co, jellyfin, immich) respond normally in under 100 ms
+- [[caddy]] on utility times out for stream.echo6.co only
+
+It looks like a dead host. It is one wedged container on a host with plenty of free memory.
+
+```bash
+ssh ts-media 'uptime; ps -eo stat --no-headers | grep -c "^D"'
+cat /sys/fs/cgroup/lxc/110/memory.current /sys/fs/cgroup/lxc/110/memory.max
+grep -E '^oom' /sys/fs/cgroup/lxc/110/memory.events
+cat /sys/fs/cgroup/lxc/110/io.pressure
+```
+
+At the time of the incident: memory 4.269 GB of a 4.294 GB limit, swap 100% full, **268 cgroup OOM kills**, `io.pressure` pinned at 99% — while the host itself had 13 GB free.
+
+---
+
+## Cause
+
+`/sitemap.xml` is **108 MB** and takes ~45 s to generate, because the instance mirrors ~136K videos. PeerTube caches it in redis under:
+
+```
+redis-stream.echo6.co-api-cache-<epoch_ms>-/sitemap.xml
+```
+
+Each cached copy costs **402 MB** — roughly 4× the wire size, from Node string overhead — and carries a multi-hour TTL. Every cache miss mints another copy. Redis runs `maxmemory 0` with `noeviction`, so nothing ever evicts them. Generating the sitemap also spikes the PeerTube Node process to ~2.9 GB RSS on its own.
+
+In a 4 GB container that is fatal. Memory and swap fill, the kernel OOM-kills redis every few hours, and every process ends up in uninterruptible sleep on major page faults.
+
+Three cached copies accounted for 1.13 GB of redis's 1.14 GB. The genuine working set — bull job queues — is about **11 MB**.
+
+---
+
+## Fix
+
+All three parts are reboot-safe and already applied.
+
+### Container memory 4 GB → 8 GB
+
+```bash
+ssh ts-media
+pct stop 110          # stops cleanly despite the D-state pile
+pct set 110 -memory 8192
+pct start 110
+```
+
+Check headroom first — media had 13.3 GB allocated of 31 GB.
+
+### Cap redis
+
+`volatile-lru`, **not** `allkeys-lru`. Bull job-queue keys are mostly TTL-less and must never be evicted; only the API cache entries carry TTLs.
+
+```bash
+R=$(grep -oP '(?<=auth: ")[^"]+' /var/www/peertube/config/production.yaml)
+pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory 1gb
+pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory-policy volatile-lru
+pct exec 110 -- redis-cli -a "$R" --no-auth-warning config rewrite
+```
+
+`config rewrite` persists to `/etc/redis/redis.conf`.
+
+### Stop serving the sitemap
+
+The sitemap TTL is compiled into PeerTube 8.0.2 and is not settable in `production.yaml`, so the durable control is nginx. Add above `location / {` in `sites-available/peertube`:
+
+```nginx
+location = /sitemap.xml {
+    return 404;
+}
+```
+
+Then `nginx -t && systemctl reload nginx`. Result: 404 in 43 ms instead of 108 MB over 45 s. This stops both the redis blob and the Node heap spike at source.
+
+### Reclaim existing blobs
+
+```bash
+pct exec 110 -- bash -c "redis-cli -a $R --no-auth-warning --scan \
+  --pattern '*api-cache*sitemap.xml' | while read k; do \
+  redis-cli -a $R --no-auth-warning del \"\$k\"; done"
+```
+
+Dropped redis from 1.14 GB to 11 MB immediately.
+
+---
+
+## Editing nginx in this container — read this first
+
+`sites-enabled/peertube` is a **symlink** to `sites-available/peertube`. Running `sed -i.bak` against it replaces the symlink with a regular file *and* leaves the `.bak` symlink inside `sites-enabled/`, so nginx loads the server block twice and warns:
+
+```
+conflicting server name "stream.echo6.co" on 0.0.0.0:80, ignored
+```
+
+Edit `sites-available/` directly and never leave backups inside `sites-enabled/`. A pristine pre-change copy is at `/root/peertube-nginx-orig-20260813.bak` inside CT 110.
+
+---
+
+## Verify
+
+```bash
+curl -o /dev/null -w '%{http_code} %{size_download}B %{time_total}s\n' https://stream.echo6.co/sitemap.xml   # expect 404
+curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://stream.echo6.co/                                 # expect 200
+```
+
+Then confirm the page actually renders per [[headless-browser-page-verification]] — a 200 from nginx does not prove PeerTube is serving.
+
+Post-fix: load 136 → 2.6, io.pressure 99% → 5%, swap 0, D-state 0, OOM kills 0.
+
+---
+
+## Still open
+
+8 GB raises the ceiling; it does not stop growth. At ~136K videos and climbing, revisit if the library grows substantially. Fetching `/sitemap.xml` to measure it mints a fresh 402 MB redis entry, so do not casually curl it.
+
+Related pipeline failure modes: [[peertube-remote-runner]], [[add-peertube-channel]].
--- a/vault/runbooks/pve-guest-park-and-adopt.md
+++ b/vault/runbooks/pve-guest-park-and-adopt.md
@ -0,0 +1,134 @@
+---
+title: Park a Proxmox Guest on the NAS and Re-Adopt It
+type: runbook
+tags:
+  - proxmox
+  - storage
+aliases: []
+related:
+  - [[omv-add-nfs-share]]
+  - [[environment]]
+  - [[proxmox-onboard-node]]
+  - [[proxmox-create-ubuntu-vm]]
+  - [[pi-nas-omv-runbook]]
+updated: 2026-08-14
+---
+# Park a Proxmox Guest on the NAS and Re-Adopt It
+
+Shut a guest down, archive the whole thing to pi-nas, rebuild or replace the node, then bring the guest back on **any** node in the cluster. This is the Proxmox equivalent of registering an orphaned VM from an ESXi datastore, and it is the supported way to free a node for rebuild without shared runtime storage.
+
+The archive is self-contained and carries no reference to its origin host, so it can be restored to a different node, a different VMID, different storage, or a freshly reinstalled machine.
+
+---
+
+## Prerequisites
+
+The `pinas-backup` storage must exist. It was added 2026-08-14 and is visible on all five cluster nodes:
+
+| Setting | Value |
+|---|---|
+| Type | NFS, `vers=3` |
+| Server | `192.168.1.245` (pi-nas) |
+| Export | `/export/pvebackup` |
+| Backing disk | `sdc1` — the 19 TB-free drive, deliberately not `sdd1` (PeerTube) |
+| Content | `backup` |
+| Mount point | `/mnt/pve/pinas-backup` |
+
+Confirm before starting:
+
+```bash
+pvesm status | grep pinas-backup
+```
+
+Creating or recreating this share is covered in [[omv-add-nfs-share]]. General NAS build is [[pi-nas-omv-runbook]].
+
+---
+
+## 1. Park the guest
+
+`--mode stop` shuts the guest down cleanly, archives it, and leaves it stopped.
+
+```bash
+# container
+vzdump 111 --mode stop --storage pinas-backup --compress zstd
+
+# VM
+vzdump 105 --mode stop --storage pinas-backup --compress zstd
+```
+
+Note the archive name it prints. Files land in `/mnt/pve/pinas-backup/dump/`:
+
+```
+vzdump-lxc-111-2026_08_14-10_22_31.tar.zst     # container
+vzdump-qemu-105-2026_08_14-10_45_02.vma.zst    # VM
+```
+
+Compression is significant — a 50 GB container typically lands near 1.5 GB.
+
+List what is parked:
+
+```bash
+pvesm list pinas-backup
+```
+
+## 2. Remove the guest from the source node
+
+Only once the archive is verified present and non-zero. This is the destructive step.
+
+```bash
+pct destroy 111        # container
+qm destroy 105         # VM
+```
+
+Skip this entirely if the plan is to wipe the node anyway.
+
+## 3. Rebuild the node
+
+Node rebuild is [[proxmox-onboard-node]]. The archive is untouched by anything done to the node.
+
+## 4. Adopt it back
+
+Run this **on the node you want the guest to live on**. The VMID and target storage are free choices — they do not have to match the original.
+
+```bash
+# container
+pct restore 111 pinas-backup:backup/vzdump-lxc-111-2026_08_14-10_22_31.tar.zst \
+  --storage local-lvm
+
+# VM
+qmrestore pinas-backup:backup/vzdump-qemu-105-2026_08_14-10_45_02.vma.zst 105 \
+  --storage local-lvm
+```
+
+To clone rather than move, restore under a new VMID and leave the original in place.
+
+## 5. Start and verify
+
+```bash
+pct start 111 && pct status 111
+qm start 105 && qm status 105
+```
+
+Verify the service itself, not just that the guest is running. For anything with a web front end, follow [[headless-browser-page-verification]].
+
+---
+
+## Things that will bite you
+
+**Containers cannot live-migrate in PVE 9 at all.** This is a hard platform limitation, not a configuration gap. Restart migration is the only option for LXC. VMs live-migrate normally when storage is shared.
+
+**Bind mounts and device mount points are not captured.** `vzdump` skips their contents. Anything reached through a bind mount, a `lxc.mount.entry`, or virtiofs must be moved separately. This applies directly to recon-vm, whose `nav`, `kiwix` and `library` shares are virtiofs from the host — see [[navi-recon-separation]].
+
+**Passthrough does not survive relocation.** USB devices, PCI passthrough and custom `args:` reference host-specific paths. Guests using them need those recreated on the target node before they will start.
+
+**`sdc` has no redundancy.** It is a single ext4 drive. An archive parked there dies with the drive — acceptable for a staging area you are actively pulling back from, not acceptable as the only copy of something. Only `sda`/`sdb` on pi-nas are mirrored.
+
+**Space is shared.** `/export/pvebackup` and `/export/arr` sit on the same physical drive, so the ~19 TB free is not exclusively yours.
+
+---
+
+## Why not shared runtime storage
+
+Running guests directly off NFS was considered and rejected: container volumes are raw images, NFS snapshots require qcow2, so `pct snapshot` / `pct rollback` stops working — and that rollback path is what [[fleet-patch-audit]] depends on for safe patching. It would also make pi-nas a single point of failure for every guest on every node simultaneously.
+
+Ceph was considered for the same goal. It needs a dedicated whole drive per node, and three of five nodes have no free drive bay; it also costs ~4 GB RAM per OSD on two nodes that are hard-capped at 32 GB. See [[fleet-storage-memory-upgrade]].