8f95c6e5c3
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/corescope-ingest-stall-oom.md vault/runbooks/edge2-access-reference.md vault/runbooks/edge2-boot-recovery.md vault/runbooks/navi-lift-to-media.md vault/runbooks/peertube-sitemap-redis-oom.md
145 lines
5.4 KiB
Markdown
145 lines
5.4 KiB
Markdown
---
|
||
title: PeerTube Sitemap Redis OOM — Diagnosis and Fix
|
||
type: runbook
|
||
tags:
|
||
- media
|
||
aliases: []
|
||
related:
|
||
- [[add-peertube-channel]]
|
||
- [[peertube-remote-runner]]
|
||
- [[caddy]]
|
||
- [[corescope-ingest-stall-oom]]
|
||
- [[central]]
|
||
updated: 2026-08-15
|
||
---
|
||
# PeerTube Sitemap Redis OOM — Diagnosis and Fix
|
||
|
||
stream.echo6.co goes down and media's load average climbs past 100 while the CPU sits mostly idle. Root cause found and fixed 2026-08-13.
|
||
|
||
[[deployment]] layout is [[services]]; PeerTube lives in CT 110 on media.
|
||
|
||
---
|
||
|
||
## Signature
|
||
|
||
Recognise it by this combination — the idle CPU is the tell:
|
||
|
||
- media load average **>100** with CPU **~85% idle** and high iowait
|
||
- `pct exec 110` hangs and returns nothing
|
||
- Every process in the container — nginx, postgres, redis, sshd — stuck in `D` state
|
||
- Other home [[services]] (echo6.co, jellyfin, immich) respond normally in under 100 ms
|
||
- [[caddy]] on utility times out for stream.echo6.co only
|
||
|
||
It looks like a dead host. It is one wedged container on a host with plenty of free memory.
|
||
|
||
```bash
|
||
ssh ts-media 'uptime; ps -eo stat --no-headers | grep -c "^D"'
|
||
cat /sys/fs/cgroup/lxc/110/memory.current /sys/fs/cgroup/lxc/110/memory.max
|
||
grep -E '^oom' /sys/fs/cgroup/lxc/110/memory.events
|
||
cat /sys/fs/cgroup/lxc/110/io.pressure
|
||
```
|
||
|
||
At the time of the incident: memory 4.269 GB of a 4.294 GB limit, swap 100% full, **268 cgroup OOM kills**, `io.pressure` pinned at 99% — while the host itself had 13 GB free.
|
||
|
||
---
|
||
|
||
## Cause
|
||
|
||
`/sitemap.xml` is **108 MB** and takes ~45 s to generate, because the instance mirrors ~136K videos. PeerTube caches it in redis under:
|
||
|
||
```
|
||
redis-stream.echo6.co-api-cache-<epoch_ms>-/sitemap.xml
|
||
```
|
||
|
||
Each cached copy costs **402 MB** — roughly 4× the wire size, from Node string overhead — and carries a multi-hour TTL. Every cache miss mints another copy. Redis runs `maxmemory 0` with `noeviction`, so nothing ever evicts them. Generating the sitemap also spikes the PeerTube Node process to ~2.9 GB RSS on its own.
|
||
|
||
In a 4 GB container that is fatal. Memory and swap fill, the kernel OOM-kills redis every few hours, and every process ends up in uninterruptible sleep on major page faults.
|
||
|
||
Three cached copies accounted for 1.13 GB of redis's 1.14 GB. The genuine working set — bull job queues — is about **11 MB**.
|
||
|
||
---
|
||
|
||
## Fix
|
||
|
||
All three parts are reboot-safe and already applied.
|
||
|
||
### Container memory 4 GB → 8 GB
|
||
|
||
```bash
|
||
ssh ts-media
|
||
pct stop 110 # stops cleanly despite the D-state pile
|
||
pct set 110 -memory 8192
|
||
pct start 110
|
||
```
|
||
|
||
Check headroom first — media had 13.3 GB allocated of 31 GB.
|
||
|
||
### Cap redis
|
||
|
||
`volatile-lru`, **not** `allkeys-lru`. Bull job-queue keys are mostly TTL-less and must never be evicted; only the API cache entries carry TTLs.
|
||
|
||
```bash
|
||
R=$(grep -oP '(?<=auth: ")[^"]+' /var/www/peertube/config/production.yaml)
|
||
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory 1gb
|
||
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory-policy volatile-lru
|
||
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config rewrite
|
||
```
|
||
|
||
`config rewrite` persists to `/etc/redis/redis.conf`.
|
||
|
||
### Stop serving the sitemap
|
||
|
||
The sitemap TTL is compiled into PeerTube 8.0.2 and is not settable in `production.yaml`, so the durable control is nginx. Add above `location / {` in `sites-available/peertube`:
|
||
|
||
```nginx
|
||
location = /sitemap.xml {
|
||
return 404;
|
||
}
|
||
```
|
||
|
||
Then `nginx -t && systemctl reload nginx`. Result: 404 in 43 ms instead of 108 MB over 45 s. This stops both the redis blob and the Node heap spike at source.
|
||
|
||
### Reclaim existing blobs
|
||
|
||
```bash
|
||
pct exec 110 -- bash -c "redis-cli -a $R --no-auth-warning --scan \
|
||
--pattern '*api-cache*sitemap.xml' | while read k; do \
|
||
redis-cli -a $R --no-auth-warning del \"\$k\"; done"
|
||
```
|
||
|
||
Dropped redis from 1.14 GB to 11 MB immediately.
|
||
|
||
---
|
||
|
||
## Editing nginx in this container — read this first
|
||
|
||
`sites-enabled/peertube` is a **symlink** to `sites-available/peertube`. Running `sed -i.bak` against it replaces the symlink with a regular file *and* leaves the `.bak` symlink inside `sites-enabled/`, so nginx loads the server block twice and warns:
|
||
|
||
```
|
||
conflicting server name "stream.echo6.co" on 0.0.0.0:80, ignored
|
||
```
|
||
|
||
Edit `sites-available/` directly and never leave backups inside `sites-enabled/`. A pristine pre-change copy is at `/root/peertube-nginx-orig-20260813.bak` inside CT 110.
|
||
|
||
---
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
curl -o /dev/null -w '%{http_code} %{size_download}B %{time_total}s\n' https://stream.echo6.co/sitemap.xml # expect 404
|
||
curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://stream.echo6.co/ # expect 200
|
||
```
|
||
|
||
Then confirm the page actually renders per [[headless-browser-page-verification]] — a 200 from nginx does not prove PeerTube is serving.
|
||
|
||
Post-fix: load 136 → 2.6, io.pressure 99% → 5%, swap 0, D-state 0, OOM kills 0.
|
||
|
||
---
|
||
|
||
## Still open
|
||
|
||
8 GB raises the ceiling; it does not stop growth. At ~136K videos and climbing, revisit if the library grows substantially. Fetching `/sitemap.xml` to measure it mints a fresh 402 MB redis entry, so do not casually curl it.
|
||
|
||
Related pipeline failure modes: [[peertube-remote-runner]], [[add-peertube-channel]].
|
||
|
||
The same shape — a cgroup ceiling well below what the application assumes, presenting as something other than memory — bit CoreScope on utility. See [[corescope-ingest-stall-oom]]. Worth checking any container whose app keeps a large in-memory cache against its `memory.max`.
|