echo6-docs/vault/runbooks/peertube-sitemap-redis-oom.md

title	type	tags	aliases	related	updated
PeerTube Sitemap Redis OOM — Diagnosis and Fix	runbook	media		add-peertube-channel caddy peertube-remote-runner central recon-operations	2026-08-14

PeerTube Sitemap Redis OOM — Diagnosis and Fix

stream.echo6.co goes down and media's load average climbs past 100 while the CPU sits mostly idle. Root cause found and fixed 2026-08-13.

deployment layout is services; PeerTube lives in CT 110 on media.

---

Signature

Recognise it by this combination — the idle CPU is the tell:

• media load average >100 with CPU ~85% idle and high iowait
• pct exec 110 hangs and returns nothing
• Every process in the container — nginx, postgres, redis, sshd — stuck in D state
• Other home services (echo6.co, jellyfin, immich) respond normally in under 100 ms
• caddy on utility times out for stream.echo6.co only

It looks like a dead host. It is one wedged container on a host with plenty of free memory.

ssh ts-media 'uptime; ps -eo stat --no-headers | grep -c "^D"'
cat /sys/fs/cgroup/lxc/110/memory.current /sys/fs/cgroup/lxc/110/memory.max
grep -E '^oom' /sys/fs/cgroup/lxc/110/memory.events
cat /sys/fs/cgroup/lxc/110/io.pressure

At the time of the incident: memory 4.269 GB of a 4.294 GB limit, swap 100% full, 268 cgroup OOM kills, io.pressure pinned at 99% — while the host itself had 13 GB free.

---

Cause

/sitemap.xml is 108 MB and takes ~45 s to generate, because the instance mirrors ~136K videos. PeerTube caches it in redis under:

redis-stream.echo6.co-api-cache-<epoch_ms>-/sitemap.xml

Each cached copy costs 402 MB — roughly 4× the wire size, from Node string overhead — and carries a multi-hour TTL. Every cache miss mints another copy. Redis runs maxmemory 0 with noeviction, so nothing ever evicts them. Generating the sitemap also spikes the PeerTube Node process to ~2.9 GB RSS on its own.

In a 4 GB container that is fatal. Memory and swap fill, the kernel OOM-kills redis every few hours, and every process ends up in uninterruptible sleep on major page faults.

Three cached copies accounted for 1.13 GB of redis's 1.14 GB. The genuine working set — bull job queues — is about 11 MB.

---

Fix

All three parts are reboot-safe and already applied.

Container memory 4 GB → 8 GB

ssh ts-media
pct stop 110          # stops cleanly despite the D-state pile
pct set 110 -memory 8192
pct start 110

Check headroom first — media had 13.3 GB allocated of 31 GB.

Cap redis

volatile-lru, not allkeys-lru. Bull job-queue keys are mostly TTL-less and must never be evicted; only the API cache entries carry TTLs.

R=$(grep -oP '(?<=auth: ")[^"]+' /var/www/peertube/config/production.yaml)
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory 1gb
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory-policy volatile-lru
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config rewrite

config rewrite persists to /etc/redis/redis.conf.

Stop serving the sitemap

The sitemap TTL is compiled into PeerTube 8.0.2 and is not settable in production.yaml, so the durable control is nginx. Add above location / { in sites-available/peertube:

location = /sitemap.xml {
    return 404;
}

Then nginx -t && systemctl reload nginx. Result: 404 in 43 ms instead of 108 MB over 45 s. This stops both the redis blob and the Node heap spike at source.

Reclaim existing blobs

pct exec 110 -- bash -c "redis-cli -a $R --no-auth-warning --scan \
  --pattern '*api-cache*sitemap.xml' | while read k; do \
  redis-cli -a $R --no-auth-warning del \"\$k\"; done"

Dropped redis from 1.14 GB to 11 MB immediately.

---

Editing nginx in this container — read this first

sites-enabled/peertube is a symlink to sites-available/peertube. Running sed -i.bak against it replaces the symlink with a regular file and leaves the .bak symlink inside sites-enabled/, so nginx loads the server block twice and warns:

conflicting server name "stream.echo6.co" on 0.0.0.0:80, ignored

Edit sites-available/ directly and never leave backups inside sites-enabled/. A pristine pre-change copy is at /root/peertube-nginx-orig-20260813.bak inside CT 110.

---

Verify

curl -o /dev/null -w '%{http_code} %{size_download}B %{time_total}s\n' https://stream.echo6.co/sitemap.xml   # expect 404
curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://stream.echo6.co/                                 # expect 200

Then confirm the page actually renders per headless-browser-page-verification — a 200 from nginx does not prove PeerTube is serving.

Post-fix: load 136 → 2.6, io.pressure 99% → 5%, swap 0, D-state 0, OOM kills 0.

---

Still open

8 GB raises the ceiling; it does not stop growth. At ~136K videos and climbing, revisit if the library grows substantially. Fetching /sitemap.xml to measure it mints a fresh 402 MB redis entry, so do not casually curl it.

Related pipeline failure modes: peertube-remote-runner, add-peertube-channel.

The same shape — a cgroup ceiling well below what the application assumes, presenting as something other than memory — bit CoreScope on utility. See corescope-ingest-stall-oom. Worth checking any container whose app keeps a large in-memory cache against its memory.max.