echo6-docs/vault/runbooks/corescope-ingest-stall-oom.md

---
title: CoreScope Silent Ingest Stall — Diagnosis and Fix
type: runbook
tags:
  - mesh
  - proxmox
aliases: []
related:
  - [[services]]
  - [[ct-runbook]]
  - [[peertube-sitemap-redis-oom]]
  - [[environment]]
  - [[meshcore-transport]]
updated: 2026-08-15
---
# CoreScope Silent Ingest Stall — Diagnosis and Fix

CoreScope stops recording MeshCore packets while the website stays up and looks perfectly healthy. It wedged twice this way (2026-08-07 and 2026-08-08), the second time for five days before anyone noticed. Root cause found and fixed 2026-08-13.

CoreScope is a MeshCore packet analyser running as a Docker container on **CT 100 on utility**, alongside MeshMonitor and PotatoMesh. Public at `corescope.idahomesh.com` and embedded at `idahomesh.com/corescope`. Contributors feed it over MQTT.

---

## Signature

The dangerous part is that nothing looks broken. The container reports `Up N days`, the web UI serves data, WebSocket clients connect, and the five-minute stats line keeps printing — **with identical numbers forever**.

```bash
pct exec 100 -- docker logs --since 30m corescope 2>&1 | grep '\[stats\]'
```

Frozen `tx_inserted` / `tx_dupes` / `obs_inserted` across consecutive ticks is the one reliable tell. Judge whether the counter is **moving**, not whether it is non-zero.

Corroborate from outside: at `https://corescope.idahomesh.com/api/observers`, every observer shares the same `last_seen` and reports `packetsLastHour: 0`. If all of them stop at the same instant, the fault is ours, not the contributors'.

Earlier in the log you will find the MQTT client giving up:

```
MQTT [local] disconnected from tcp://localhost:1883: pingresp not received, disconnecting
MQTT [local] WATCHDOG: client reports connected but no messages received for 5m0s
MQTT [local] WATCHDOG forcing reconnect (half-open TCP suspected)
MQTT [local] WATCHDOG reconnect attempt issued
    <- then silence, and no "subscribed to meshcore/#" ever again
```

The watchdog falling quiet is **not** recovery. It tried three times and stopped.

### Prove the feed is fine before blaming contributors

Subscribe to the broker directly inside the container. If packets are arriving here while `tx_inserted` is frozen, the broker, the edge2 relay and the contributors are all healthy and the ingestor alone is wedged.

```bash
pct exec 100 -- docker exec corescope \
  timeout 25 mosquitto_sub -h 127.0.0.1 -p 1883 -t 'meshcore/#' -v
```

---

## Cause

Not MQTT, and not the disk.

`corescope-server` keeps a seven-day packet store **in memory**. Upstream's default budget is `packetStore.maxMemoryMB: 1024`, from which it derives a Go soft limit of 1.5x = **1536 MiB**. CT 100 was a **2 GB** container also running MeshMonitor, two PotatoMesh containers, Mosquitto and the ingestor. That budget never fit.

The server creeps to roughly 1.8 GB RSS, the cgroup falls into reclaim thrash, and everything inside CT 100 crawls for about 1h45m. Measured during the second event: a single SQLite `InsertTransmission` took **6,449 seconds**, and a neighbour-build tick took 1h46m. While that is happening the ingestor cannot service its MQTT keepalive, so the connection drops and the reconnects cannot complete either.

Then the kernel kills the largest process:

```
Aug 06 23:48:55  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1776728kB
Aug 08 17:41:15  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1840364kB
```

(Host journal is in local time; those are 08-07 05:48 and 08-08 23:41 UTC, matching both stalls to the second.)

**Why it never self-heals:** supervisord runs `corescope-server` and `corescope-ingestor` as two separate programs. The OOM killer takes the *server*, and supervisord dutifully restarts only that one. The ingestor is never killed, so it is never restarted — it sits wedged while the UI comes back looking healthy.

**Why it stays dead for days:** once ingest stops, memory stops growing, so there is no second OOM to force the issue. The failure is self-limiting and silent. Time to failure was about eight hours of ingest after a fresh start.

The recurring `[db] WAL checkpoint error: disk I/O error (778)` messages are **swap thrash, not failing storage**. Do not chase the disk. There are no vzdump jobs on utility.

---

## Fix

Raise the container's memory. LXC applies this **live** — no restart, nothing drops.

```bash
pct set 100 -memory 6144
pct exec 100 -- free -m        # expect 6144 total
```

Check host headroom first: utility has 30 GB, and although CT 104 is allocated 12 GB it is retired and idle.

CoreScope has **no Docker-level memory cap** (`HostConfig.Memory=0`, cgroup `max`), so it inherits the container's limit — which is why raising the CT is what actually moves its ceiling.

```bash
pct exec 100 -- docker inspect corescope --format 'Memory={{.HostConfig.Memory}}'
pct exec 100 -- docker exec corescope cat /sys/fs/cgroup/memory.max
```

**`docker stats` still prints "/2GiB" for uncapped containers** — that is a stale daemon reading of host RAM, purely cosmetic. Verify with `docker inspect` and `memory.max`, never with `docker stats`.

### Clearing a stall that has already happened

```bash
pct exec 100 -- docker restart corescope
```

Plain `docker run` container with `--restart unless-stopped` and no compose project, so restart rather than recreate is correct. Recovery is immediate. Look for `Running — 1 MQTT source(s) connected` and, critically, **`MQTT [local] subscribed to meshcore/#`** — that subscribe line is the real proof, and it is what was missing the whole time.

Counters reset to zero because they are per-process. **Climbing from zero is success, not data loss.**

---

## Verify

```bash
pct exec 100 -- docker logs --since 20m corescope 2>&1 | grep '\[stats\]' | tail -3
pct exec 100 -- free -m
journalctl -k --since '4 hours ago' | grep -c 'Memory cgroup out of memory'
```

Healthy means `tx_inserted` increasing across consecutive ticks and an OOM count of zero.

Monitored every three hours for 26 hours after the fix: ingest never stalled, memory stayed flat at about 1.87 GB of 6 GB, zero OOM kills. Against a failure that previously arrived within eight hours, that is the fix holding rather than the clock being reset.

---

## If it ever recurs

More memory raises the ceiling; it does not cap growth. The next lever is to bound CoreScope itself in `/opt/corescope-data/config.json` — `packetStore.maxMemoryMB` around 320, `retentionHours` 48, and `runtime.maxMemoryMB` around 512 (that last one applies to **both** the server and the ingestor). The config is a **partial override**, so adding keys will not disturb `mqttSources`. Upstream documents this deployment case in issues #836 and #1010.

A flat-`tx_inserted` self-heal watchdog on CT 100 is worth building if the pattern returns, since the container's own watchdog demonstrably stops trying.

Same family of failure as [[peertube-sitemap-redis-oom]] — a cgroup ceiling far below what the application assumes, presenting as something else entirely.