echo6-docs/vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md

title	type	tags	aliases	related	updated
pymc-repeater KISS TNC Re-enumeration Fix	runbook	mesh		recon-operations SESSION-HANDOFF-meshai-test meshtasticd-sim-nodes-runbook meshtastic-headscale-runbook syncthing-add-node	2026-07-13

pymc-repeater KISS TNC Re-enumeration Fix

pymc-repeater.service on aida-nebra (Tailscale 100.64.0.9) runs the Meshcore LoRa repeater node EchoBase on 910.525 MHz SF7. It talks to a RAK4631 USB KISS TNC (VID 239a / PID 8029) via the /dev/pymc-radio udev symlink. The Nebra SX1262 hat on the same machine runs a separate meshtasticd instance on its own radio — see meshtastic-sidecar-node for that hardware's setup. Do not touch meshtasticd or the SX1262 hat during any pymc-repeater work.

---

Background — Why This Breaks

KissModemWrapper holds the serial file descriptor open but has no reopen logic. When the RAK4631 re-enumerates (power cycle, USB glitch, kernel rescan), the kernel assigns a new node: observed ttyACM0 → ttyACM1 around 2026-06-23. The service keeps a stale fd pointing at /dev/ttyACMx (deleted). Systemd still reports active (running) — the service is silently deaf and mute.

---

Symptom

Every ~30 seconds in the journal:

KissModemWrapper - ERROR - Serial write error: ... [Errno 5] Input/output error
TX frame write failed, dropping frame

Zero RF traffic in or out despite a green systemctl status.

---

Diagnose

1. Check the journal for the I/O error spam

ssh zvx@100.64.0.9
journalctl -u pymc-repeater.service --since '5 min ago' --no-pager

Look for [Errno 5] Input/output error or TX frame write failed — either confirms stale fd.

2. Inspect the process's open file descriptors

sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM

• Stale (broken): symlink target contains (deleted) — smoking gun.
• Healthy: target points at a real /dev/ttyACMx.

3. Confirm where the device landed after re-enumeration

ls -l /dev/pymc-radio

Shows which ttyACM* the live RAK4631 is on now.

---

Manual Recovery

A restart clears the stale fd and re-opens the device via the udev symlink:

sudo systemctl restart pymc-repeater.service

Verify recovery

# fd should now point at a real ttyACMx (not deleted)
sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM

# Journal should show reconnect and normal TX
journalctl -u pymc-repeater.service -n 20 --no-pager

Expected healthy journal lines:

KISS modem connected to /dev/pymc-radio at 115200 baud
Retransmitted packet ...

The [Errno 5] spam must be absent.

---

Permanent Fix (applied 2026-07-01)

Two changes make pymc-repeater self-healing across any future re-enumeration. Original files were backed up as .bak alongside each.

1. udev rule — /etc/udev/rules.d/99-pymc-radio.rules

Added TAG+="systemd" and ENV{SYSTEMD_WANTS}="pymc-repeater.service" to the existing VID 239a / PID 8029 rule:

SUBSYSTEM=="tty", ATTRS{idVendor}=="239a", ATTRS{idProduct}=="8029", \
  SYMLINK+="pymc-radio", \
  TAG+="systemd", \
  ENV{SYSTEMD_WANTS}="pymc-repeater.service"

On any future re-enumeration, udev creates the dev-pymc\x2dradio.device unit for the new node and systemd activates pymc-repeater.service against it automatically.

2. systemd unit — /etc/systemd/system/pymc-repeater.service

[Unit] additions:

BindsTo=dev-pymc\x2dradio.device
After=dev-pymc\x2dradio.device
StartLimitIntervalSec=300
StartLimitBurst=10

[Service] additions:

Restart=always
RestartSec=5

BindsTo stops the service cleanly when the TNC drops (the device unit disappears) and restarts it when the device returns. Restart=always with RestartSec=5 ensures fast recovery from any modem crash without the device unit cycling.

---

Verify the Resilience Wiring Is Intact

Run these on aida-nebra after any future maintenance to confirm the permanent fix is still in place:

# BindsTo and After wired to the device unit
systemctl show pymc-repeater.service -p BindsTo -p After | grep -i pymc

# udev tagging correct
udevadm info -q property -n /dev/pymc-radio | grep -i SYSTEMD

# device unit tracked by systemd
systemctl list-units --type=device --all | grep -i pymc

Expected output:

BindsTo=dev-pymc\x2dradio.device
After=... dev-pymc\x2dradio.device ...
SYSTEMD_WANTS=pymc-repeater.service
dev-pymc\x2dradio.device  loaded active plugged  ...

---

Known Secondary Issue (non-blocking)

RRD graphing fails with rrdtool not available. Metrics/graphing only — no impact on RF operation.

---

Quick Reference

Item	Value
Host	aida-nebra (100.64.0.9)
Service	pymc-repeater.service
Node / freq	EchoBase, 910.525 MHz SF7
TNC hardware	RAK4631 USB KISS TNC (VID 239a / PID 8029)
udev symlink	/dev/pymc-radio
udev rule	/etc/udev/rules.d/99-pymc-radio.rules
Unit file	/etc/systemd/system/pymc-repeater.service
Unrelated radio	Nebra SX1262 hat → meshtasticd (do not touch)

# Restart (manual recovery)
sudo systemctl restart pymc-repeater.service

# Tail journal
journalctl -u pymc-repeater.service -f

# Check fd state
sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM