--- title: PeerTube Sitemap Redis OOM — Diagnosis and Fix type: runbook tags: - media aliases: [] related: - [[add-peertube-channel]] - [[caddy]] - [[peertube-remote-runner]] - [[central]] - [[recon-operations]] updated: 2026-08-14 --- # PeerTube Sitemap Redis OOM — Diagnosis and Fix stream.echo6.co goes down and media's load average climbs past 100 while the CPU sits mostly idle. Root cause found and fixed 2026-08-13. [[deployment]] layout is [[services]]; PeerTube lives in CT 110 on media. --- ## Signature Recognise it by this combination — the idle CPU is the tell: - media load average **>100** with CPU **~85% idle** and high iowait - `pct exec 110` hangs and returns nothing - Every process in the container — nginx, postgres, redis, sshd — stuck in `D` state - Other home [[services]] (echo6.co, jellyfin, immich) respond normally in under 100 ms - [[caddy]] on utility times out for stream.echo6.co only It looks like a dead host. It is one wedged container on a host with plenty of free memory. ```bash ssh ts-media 'uptime; ps -eo stat --no-headers | grep -c "^D"' cat /sys/fs/cgroup/lxc/110/memory.current /sys/fs/cgroup/lxc/110/memory.max grep -E '^oom' /sys/fs/cgroup/lxc/110/memory.events cat /sys/fs/cgroup/lxc/110/io.pressure ``` At the time of the incident: memory 4.269 GB of a 4.294 GB limit, swap 100% full, **268 cgroup OOM kills**, `io.pressure` pinned at 99% — while the host itself had 13 GB free. --- ## Cause `/sitemap.xml` is **108 MB** and takes ~45 s to generate, because the instance mirrors ~136K videos. PeerTube caches it in redis under: ``` redis-stream.echo6.co-api-cache--/sitemap.xml ``` Each cached copy costs **402 MB** — roughly 4× the wire size, from Node string overhead — and carries a multi-hour TTL. Every cache miss mints another copy. Redis runs `maxmemory 0` with `noeviction`, so nothing ever evicts them. Generating the sitemap also spikes the PeerTube Node process to ~2.9 GB RSS on its own. In a 4 GB container that is fatal. Memory and swap fill, the kernel OOM-kills redis every few hours, and every process ends up in uninterruptible sleep on major page faults. Three cached copies accounted for 1.13 GB of redis's 1.14 GB. The genuine working set — bull job queues — is about **11 MB**. --- ## Fix All three parts are reboot-safe and already applied. ### Container memory 4 GB → 8 GB ```bash ssh ts-media pct stop 110 # stops cleanly despite the D-state pile pct set 110 -memory 8192 pct start 110 ``` Check headroom first — media had 13.3 GB allocated of 31 GB. ### Cap redis `volatile-lru`, **not** `allkeys-lru`. Bull job-queue keys are mostly TTL-less and must never be evicted; only the API cache entries carry TTLs. ```bash R=$(grep -oP '(?<=auth: ")[^"]+' /var/www/peertube/config/production.yaml) pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory 1gb pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory-policy volatile-lru pct exec 110 -- redis-cli -a "$R" --no-auth-warning config rewrite ``` `config rewrite` persists to `/etc/redis/redis.conf`. ### Stop serving the sitemap The sitemap TTL is compiled into PeerTube 8.0.2 and is not settable in `production.yaml`, so the durable control is nginx. Add above `location / {` in `sites-available/peertube`: ```nginx location = /sitemap.xml { return 404; } ``` Then `nginx -t && systemctl reload nginx`. Result: 404 in 43 ms instead of 108 MB over 45 s. This stops both the redis blob and the Node heap spike at source. ### Reclaim existing blobs ```bash pct exec 110 -- bash -c "redis-cli -a $R --no-auth-warning --scan \ --pattern '*api-cache*sitemap.xml' | while read k; do \ redis-cli -a $R --no-auth-warning del \"\$k\"; done" ``` Dropped redis from 1.14 GB to 11 MB immediately. --- ## Editing nginx in this container — read this first `sites-enabled/peertube` is a **symlink** to `sites-available/peertube`. Running `sed -i.bak` against it replaces the symlink with a regular file *and* leaves the `.bak` symlink inside `sites-enabled/`, so nginx loads the server block twice and warns: ``` conflicting server name "stream.echo6.co" on 0.0.0.0:80, ignored ``` Edit `sites-available/` directly and never leave backups inside `sites-enabled/`. A pristine pre-change copy is at `/root/peertube-nginx-orig-20260813.bak` inside CT 110. --- ## Verify ```bash curl -o /dev/null -w '%{http_code} %{size_download}B %{time_total}s\n' https://stream.echo6.co/sitemap.xml # expect 404 curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://stream.echo6.co/ # expect 200 ``` Then confirm the page actually renders per [[headless-browser-page-verification]] — a 200 from nginx does not prove PeerTube is serving. Post-fix: load 136 → 2.6, io.pressure 99% → 5%, swap 0, D-state 0, OOM kills 0. --- ## Still open 8 GB raises the ceiling; it does not stop growth. At ~136K videos and climbing, revisit if the library grows substantially. Fetching `/sitemap.xml` to measure it mints a fresh 402 MB redis entry, so do not casually curl it. Related pipeline failure modes: [[peertube-remote-runner]], [[add-peertube-channel]]. The same shape — a cgroup ceiling well below what the application assumes, presenting as something other than memory — bit CoreScope on utility. See [[corescope-ingest-stall-oom]]. Worth checking any container whose app keeps a large in-memory cache against its `memory.max`.