echo6-docs/vault/runbooks/corescope-ingest-stall-oom.md

title	type	tags	aliases	related	updated
CoreScope Silent Ingest Stall — Diagnosis and Fix	runbook	mesh proxmox		services ct-runbook peertube-sitemap-redis-oom environment meshcore-transport	2026-08-15

CoreScope Silent Ingest Stall — Diagnosis and Fix

CoreScope stops recording MeshCore packets while the website stays up and looks perfectly healthy. It wedged twice this way (2026-08-07 and 2026-08-08), the second time for five days before anyone noticed. Root cause found and fixed 2026-08-13.

CoreScope is a MeshCore packet analyser running as a Docker container on CT 100 on utility, alongside MeshMonitor and PotatoMesh. Public at corescope.idahomesh.com and embedded at idahomesh.com/corescope. Contributors feed it over MQTT.

---

Signature

The dangerous part is that nothing looks broken. The container reports Up N days, the web UI serves data, WebSocket clients connect, and the five-minute stats line keeps printing — with identical numbers forever.

pct exec 100 -- docker logs --since 30m corescope 2>&1 | grep '\[stats\]'

Frozen tx_inserted / tx_dupes / obs_inserted across consecutive ticks is the one reliable tell. Judge whether the counter is moving, not whether it is non-zero.

Corroborate from outside: at https://corescope.idahomesh.com/api/observers, every observer shares the same last_seen and reports packetsLastHour: 0. If all of them stop at the same instant, the fault is ours, not the contributors'.

Earlier in the log you will find the MQTT client giving up:

MQTT [local] disconnected from tcp://localhost:1883: pingresp not received, disconnecting
MQTT [local] WATCHDOG: client reports connected but no messages received for 5m0s
MQTT [local] WATCHDOG forcing reconnect (half-open TCP suspected)
MQTT [local] WATCHDOG reconnect attempt issued
    <- then silence, and no "subscribed to meshcore/#" ever again

The watchdog falling quiet is not recovery. It tried three times and stopped.

Prove the feed is fine before blaming contributors

Subscribe to the broker directly inside the container. If packets are arriving here while tx_inserted is frozen, the broker, the edge2 relay and the contributors are all healthy and the ingestor alone is wedged.

pct exec 100 -- docker exec corescope \
  timeout 25 mosquitto_sub -h 127.0.0.1 -p 1883 -t 'meshcore/#' -v

---

Cause

Not MQTT, and not the disk.

corescope-server keeps a seven-day packet store in memory. Upstream's default budget is packetStore.maxMemoryMB: 1024, from which it derives a Go soft limit of 1.5x = 1536 MiB. CT 100 was a 2 GB container also running MeshMonitor, two PotatoMesh containers, Mosquitto and the ingestor. That budget never fit.

The server creeps to roughly 1.8 GB RSS, the cgroup falls into reclaim thrash, and everything inside CT 100 crawls for about 1h45m. Measured during the second event: a single SQLite InsertTransmission took 6,449 seconds, and a neighbour-build tick took 1h46m. While that is happening the ingestor cannot service its MQTT keepalive, so the connection drops and the reconnects cannot complete either.

Then the kernel kills the largest process:

Aug 06 23:48:55  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1776728kB
Aug 08 17:41:15  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1840364kB

(Host journal is in local time; those are 08-07 05:48 and 08-08 23:41 UTC, matching both stalls to the second.)

Why it never self-heals: supervisord runs corescope-server and corescope-ingestor as two separate programs. The OOM killer takes the server, and supervisord dutifully restarts only that one. The ingestor is never killed, so it is never restarted — it sits wedged while the UI comes back looking healthy.

Why it stays dead for days: once ingest stops, memory stops growing, so there is no second OOM to force the issue. The failure is self-limiting and silent. Time to failure was about eight hours of ingest after a fresh start.

The recurring [db] WAL checkpoint error: disk I/O error (778) messages are swap thrash, not failing storage. Do not chase the disk. There are no vzdump jobs on utility.

---

Fix

Raise the container's memory. LXC applies this live — no restart, nothing drops.

pct set 100 -memory 6144
pct exec 100 -- free -m        # expect 6144 total

Check host headroom first: utility has 30 GB, and although CT 104 is allocated 12 GB it is retired and idle.

CoreScope has no Docker-level memory cap (HostConfig.Memory=0, cgroup max), so it inherits the container's limit — which is why raising the CT is what actually moves its ceiling.

pct exec 100 -- docker inspect corescope --format 'Memory={{.HostConfig.Memory}}'
pct exec 100 -- docker exec corescope cat /sys/fs/cgroup/memory.max

docker stats still prints "/2GiB" for uncapped containers — that is a stale daemon reading of host RAM, purely cosmetic. Verify with docker inspect and memory.max, never with docker stats.

Clearing a stall that has already happened

pct exec 100 -- docker restart corescope

Plain docker run container with --restart unless-stopped and no compose project, so restart rather than recreate is correct. Recovery is immediate. Look for Running — 1 MQTT source(s) connected and, critically, MQTT [local] subscribed to meshcore/# — that subscribe line is the real proof, and it is what was missing the whole time.

Counters reset to zero because they are per-process. Climbing from zero is success, not data loss.

---

Verify

pct exec 100 -- docker logs --since 20m corescope 2>&1 | grep '\[stats\]' | tail -3
pct exec 100 -- free -m
journalctl -k --since '4 hours ago' | grep -c 'Memory cgroup out of memory'

Healthy means tx_inserted increasing across consecutive ticks and an OOM count of zero.

Monitored every three hours for 26 hours after the fix: ingest never stalled, memory stayed flat at about 1.87 GB of 6 GB, zero OOM kills. Against a failure that previously arrived within eight hours, that is the fix holding rather than the clock being reset.

---

If it ever recurs

More memory raises the ceiling; it does not cap growth. The next lever is to bound CoreScope itself in /opt/corescope-data/config.json — packetStore.maxMemoryMB around 320, retentionHours 48, and runtime.maxMemoryMB around 512 (that last one applies to both the server and the ingestor). The config is a partial override, so adding keys will not disturb mqttSources. Upstream documents this deployment case in issues #836 and #1010.

A flat-tx_inserted self-heal watchdog on CT 100 is worth building if the pattern returns, since the container's own watchdog demonstrably stops trying.

Same family of failure as peertube-sitemap-redis-oom — a cgroup ceiling far below what the application assumes, presenting as something else entirely.