143 lines
5.1 KiB
Markdown
143 lines
5.1 KiB
Markdown
|
---
|
|||
|
|
title: PeerTube Sitemap Redis OOM — Diagnosis and Fix
|
|||
|
|
type: runbook
|
|||
|
|
tags:
|
|||
|
|
- media
|
|||
|
|
aliases: []
|
|||
|
|
related:
|
|||
|
|
- [[add-peertube-channel]]
|
|||
|
|
- [[caddy]]
|
|||
|
|
- [[peertube-remote-runner]]
|
|||
|
|
- [[central]]
|
|||
|
|
- [[recon-operations]]
|
|||
|
|
updated: 2026-08-14
|
|||
|
|
---
|
|||
|
|
# PeerTube Sitemap Redis OOM — Diagnosis and Fix
|
|||
|
|
|
|||
|
|
stream.echo6.co goes down and media's load average climbs past 100 while the CPU sits mostly idle. Root cause found and fixed 2026-08-13.
|
|||
|
|
|
|||
|
|
[[deployment]] layout is [[services]]; PeerTube lives in CT 110 on media.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Signature
|
|||
|
|
|
|||
|
|
Recognise it by this combination — the idle CPU is the tell:
|
|||
|
|
|
|||
|
|
- media load average **>100** with CPU **~85% idle** and high iowait
|
|||
|
|
- `pct exec 110` hangs and returns nothing
|
|||
|
|
- Every process in the container — nginx, postgres, redis, sshd — stuck in `D` state
|
|||
|
|
- Other home [[services]] (echo6.co, jellyfin, immich) respond normally in under 100 ms
|
|||
|
|
- [[caddy]] on utility times out for stream.echo6.co only
|
|||
|
|
|
|||
|
|
It looks like a dead host. It is one wedged container on a host with plenty of free memory.
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ssh ts-media 'uptime; ps -eo stat --no-headers | grep -c "^D"'
|
|||
|
|
cat /sys/fs/cgroup/lxc/110/memory.current /sys/fs/cgroup/lxc/110/memory.max
|
|||
|
|
grep -E '^oom' /sys/fs/cgroup/lxc/110/memory.events
|
|||
|
|
cat /sys/fs/cgroup/lxc/110/io.pressure
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
At the time of the incident: memory 4.269 GB of a 4.294 GB limit, swap 100% full, **268 cgroup OOM kills**, `io.pressure` pinned at 99% — while the host itself had 13 GB free.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Cause
|
|||
|
|
|
|||
|
|
`/sitemap.xml` is **108 MB** and takes ~45 s to generate, because the instance mirrors ~136K videos. PeerTube caches it in redis under:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
redis-stream.echo6.co-api-cache-<epoch_ms>-/sitemap.xml
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Each cached copy costs **402 MB** — roughly 4× the wire size, from Node string overhead — and carries a multi-hour TTL. Every cache miss mints another copy. Redis runs `maxmemory 0` with `noeviction`, so nothing ever evicts them. Generating the sitemap also spikes the PeerTube Node process to ~2.9 GB RSS on its own.
|
|||
|
|
|
|||
|
|
In a 4 GB container that is fatal. Memory and swap fill, the kernel OOM-kills redis every few hours, and every process ends up in uninterruptible sleep on major page faults.
|
|||
|
|
|
|||
|
|
Three cached copies accounted for 1.13 GB of redis's 1.14 GB. The genuine working set — bull job queues — is about **11 MB**.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Fix
|
|||
|
|
|
|||
|
|
All three parts are reboot-safe and already applied.
|
|||
|
|
|
|||
|
|
### Container memory 4 GB → 8 GB
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ssh ts-media
|
|||
|
|
pct stop 110 # stops cleanly despite the D-state pile
|
|||
|
|
pct set 110 -memory 8192
|
|||
|
|
pct start 110
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Check headroom first — media had 13.3 GB allocated of 31 GB.
|
|||
|
|
|
|||
|
|
### Cap redis
|
|||
|
|
|
|||
|
|
`volatile-lru`, **not** `allkeys-lru`. Bull job-queue keys are mostly TTL-less and must never be evicted; only the API cache entries carry TTLs.
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
R=$(grep -oP '(?<=auth: ")[^"]+' /var/www/peertube/config/production.yaml)
|
|||
|
|
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory 1gb
|
|||
|
|
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config set maxmemory-policy volatile-lru
|
|||
|
|
pct exec 110 -- redis-cli -a "$R" --no-auth-warning config rewrite
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`config rewrite` persists to `/etc/redis/redis.conf`.
|
|||
|
|
|
|||
|
|
### Stop serving the sitemap
|
|||
|
|
|
|||
|
|
The sitemap TTL is compiled into PeerTube 8.0.2 and is not settable in `production.yaml`, so the durable control is nginx. Add above `location / {` in `sites-available/peertube`:
|
|||
|
|
|
|||
|
|
```nginx
|
|||
|
|
location = /sitemap.xml {
|
|||
|
|
return 404;
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Then `nginx -t && systemctl reload nginx`. Result: 404 in 43 ms instead of 108 MB over 45 s. This stops both the redis blob and the Node heap spike at source.
|
|||
|
|
|
|||
|
|
### Reclaim existing blobs
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
pct exec 110 -- bash -c "redis-cli -a $R --no-auth-warning --scan \
|
|||
|
|
--pattern '*api-cache*sitemap.xml' | while read k; do \
|
|||
|
|
redis-cli -a $R --no-auth-warning del \"\$k\"; done"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Dropped redis from 1.14 GB to 11 MB immediately.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Editing nginx in this container — read this first
|
|||
|
|
|
|||
|
|
`sites-enabled/peertube` is a **symlink** to `sites-available/peertube`. Running `sed -i.bak` against it replaces the symlink with a regular file *and* leaves the `.bak` symlink inside `sites-enabled/`, so nginx loads the server block twice and warns:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
conflicting server name "stream.echo6.co" on 0.0.0.0:80, ignored
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Edit `sites-available/` directly and never leave backups inside `sites-enabled/`. A pristine pre-change copy is at `/root/peertube-nginx-orig-20260813.bak` inside CT 110.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
curl -o /dev/null -w '%{http_code} %{size_download}B %{time_total}s\n' https://stream.echo6.co/sitemap.xml # expect 404
|
|||
|
|
curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://stream.echo6.co/ # expect 200
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Then confirm the page actually renders per [[headless-browser-page-verification]] — a 200 from nginx does not prove PeerTube is serving.
|
|||
|
|
|
|||
|
|
Post-fix: load 136 → 2.6, io.pressure 99% → 5%, swap 0, D-state 0, OOM kills 0.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Still open
|
|||
|
|
|
|||
|
|
8 GB raises the ceiling; it does not stop growth. At ~136K videos and climbing, revisit if the library grows substantially. Fetching `/sitemap.xml` to measure it mints a fresh 402 MB redis entry, so do not casually curl it.
|
|||
|
|
|
|||
|
|
Related pipeline failure modes: [[peertube-remote-runner]], [[add-peertube-channel]].
|