auto: docs sync 2026-08-15T06:00:04+00:00

Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/corescope-ingest-stall-oom.md vault/runbooks/edge2-access-reference.md vault/runbooks/edge2-boot-recovery.md vault/runbooks/navi-lift-to-media.md vault/runbooks/peertube-sitemap-redis-oom.md

vault/runbooks/corescope-ingest-stall-oom.md

@@ -0,0 +1,136 @@
+---
+title: CoreScope Silent Ingest Stall — Diagnosis and Fix
+type: runbook
+tags:
+  - mesh
+  - proxmox
+aliases: []
+related:
+  - [[services]]
+  - [[ct-runbook]]
+  - [[peertube-sitemap-redis-oom]]
+  - [[environment]]
+  - [[meshcore-transport]]
+updated: 2026-08-15
+---
+# CoreScope Silent Ingest Stall — Diagnosis and Fix
+
+CoreScope stops recording MeshCore packets while the website stays up and looks perfectly healthy. It wedged twice this way (2026-08-07 and 2026-08-08), the second time for five days before anyone noticed. Root cause found and fixed 2026-08-13.
+
+CoreScope is a MeshCore packet analyser running as a Docker container on **CT 100 on utility**, alongside MeshMonitor and PotatoMesh. Public at `corescope.idahomesh.com` and embedded at `idahomesh.com/corescope`. Contributors feed it over MQTT.
+
+---
+
+## Signature
+
+The dangerous part is that nothing looks broken. The container reports `Up N days`, the web UI serves data, WebSocket clients connect, and the five-minute stats line keeps printing — **with identical numbers forever**.
+
+```bash
+pct exec 100 -- docker logs --since 30m corescope 2>&1 | grep '\[stats\]'
+```
+
+Frozen `tx_inserted` / `tx_dupes` / `obs_inserted` across consecutive ticks is the one reliable tell. Judge whether the counter is **moving**, not whether it is non-zero.
+
+Corroborate from outside: at `https://corescope.idahomesh.com/api/observers`, every observer shares the same `last_seen` and reports `packetsLastHour: 0`. If all of them stop at the same instant, the fault is ours, not the contributors'.
+
+Earlier in the log you will find the MQTT client giving up:
+
+```
+MQTT [local] disconnected from tcp://localhost:1883: pingresp not received, disconnecting
+MQTT [local] WATCHDOG: client reports connected but no messages received for 5m0s
+MQTT [local] WATCHDOG forcing reconnect (half-open TCP suspected)
+MQTT [local] WATCHDOG reconnect attempt issued
+    <- then silence, and no "subscribed to meshcore/#" ever again
+```
+
+The watchdog falling quiet is **not** recovery. It tried three times and stopped.
+
+### Prove the feed is fine before blaming contributors
+
+Subscribe to the broker directly inside the container. If packets are arriving here while `tx_inserted` is frozen, the broker, the edge2 relay and the contributors are all healthy and the ingestor alone is wedged.
+
+```bash
+pct exec 100 -- docker exec corescope \
+  timeout 25 mosquitto_sub -h 127.0.0.1 -p 1883 -t 'meshcore/#' -v
+```
+
+---
+
+## Cause
+
+Not MQTT, and not the disk.
+
+`corescope-server` keeps a seven-day packet store **in memory**. Upstream's default budget is `packetStore.maxMemoryMB: 1024`, from which it derives a Go soft limit of 1.5x = **1536 MiB**. CT 100 was a **2 GB** container also running MeshMonitor, two PotatoMesh containers, Mosquitto and the ingestor. That budget never fit.
+
+The server creeps to roughly 1.8 GB RSS, the cgroup falls into reclaim thrash, and everything inside CT 100 crawls for about 1h45m. Measured during the second event: a single SQLite `InsertTransmission` took **6,449 seconds**, and a neighbour-build tick took 1h46m. While that is happening the ingestor cannot service its MQTT keepalive, so the connection drops and the reconnects cannot complete either.
+
+Then the kernel kills the largest process:
+
+```
+Aug 06 23:48:55  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1776728kB
+Aug 08 17:41:15  Memory cgroup out of memory: Killed process (corescope-serve) anon-rss:1840364kB
+```
+
+(Host journal is in local time; those are 08-07 05:48 and 08-08 23:41 UTC, matching both stalls to the second.)
+
+**Why it never self-heals:** supervisord runs `corescope-server` and `corescope-ingestor` as two separate programs. The OOM killer takes the *server*, and supervisord dutifully restarts only that one. The ingestor is never killed, so it is never restarted — it sits wedged while the UI comes back looking healthy.
+
+**Why it stays dead for days:** once ingest stops, memory stops growing, so there is no second OOM to force the issue. The failure is self-limiting and silent. Time to failure was about eight hours of ingest after a fresh start.
+
+The recurring `[db] WAL checkpoint error: disk I/O error (778)` messages are **swap thrash, not failing storage**. Do not chase the disk. There are no vzdump jobs on utility.
+
+---
+
+## Fix
+
+Raise the container's memory. LXC applies this **live** — no restart, nothing drops.
+
+```bash
+pct set 100 -memory 6144
+pct exec 100 -- free -m        # expect 6144 total
+```
+
+Check host headroom first: utility has 30 GB, and although CT 104 is allocated 12 GB it is retired and idle.
+
+CoreScope has **no Docker-level memory cap** (`HostConfig.Memory=0`, cgroup `max`), so it inherits the container's limit — which is why raising the CT is what actually moves its ceiling.
+
+```bash
+pct exec 100 -- docker inspect corescope --format 'Memory={{.HostConfig.Memory}}'
+pct exec 100 -- docker exec corescope cat /sys/fs/cgroup/memory.max
+```
+
+**`docker stats` still prints "/2GiB" for uncapped containers** — that is a stale daemon reading of host RAM, purely cosmetic. Verify with `docker inspect` and `memory.max`, never with `docker stats`.
+
+### Clearing a stall that has already happened
+
+```bash
+pct exec 100 -- docker restart corescope
+```
+
+Plain `docker run` container with `--restart unless-stopped` and no compose project, so restart rather than recreate is correct. Recovery is immediate. Look for `Running — 1 MQTT source(s) connected` and, critically, **`MQTT [local] subscribed to meshcore/#`** — that subscribe line is the real proof, and it is what was missing the whole time.
+
+Counters reset to zero because they are per-process. **Climbing from zero is success, not data loss.**
+
+---
+
+## Verify
+
+```bash
+pct exec 100 -- docker logs --since 20m corescope 2>&1 | grep '\[stats\]' | tail -3
+pct exec 100 -- free -m
+journalctl -k --since '4 hours ago' | grep -c 'Memory cgroup out of memory'
+```
+
+Healthy means `tx_inserted` increasing across consecutive ticks and an OOM count of zero.
+
+Monitored every three hours for 26 hours after the fix: ingest never stalled, memory stayed flat at about 1.87 GB of 6 GB, zero OOM kills. Against a failure that previously arrived within eight hours, that is the fix holding rather than the clock being reset.
+
+---
+
+## If it ever recurs
+
+More memory raises the ceiling; it does not cap growth. The next lever is to bound CoreScope itself in `/opt/corescope-data/config.json` — `packetStore.maxMemoryMB` around 320, `retentionHours` 48, and `runtime.maxMemoryMB` around 512 (that last one applies to **both** the server and the ingestor). The config is a **partial override**, so adding keys will not disturb `mqttSources`. Upstream documents this deployment case in issues #836 and #1010.
+
+A flat-`tx_inserted` self-heal watchdog on CT 100 is worth building if the pattern returns, since the container's own watchdog demonstrably stops trying.
+
+Same family of failure as [[peertube-sitemap-redis-oom]] — a cgroup ceiling far below what the application assumes, presenting as something else entirely.