auto: docs sync 2026-07-01T18:00:05+00:00
Files changed: engine/lint-report.md vault/.obsidian/workspace.json vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md
This commit is contained in:
echo6-autocommit
parent
4082aed827
commit
e7d30b3c48
3 changed files with 190 additions and 3 deletions
4
vault/.obsidian/workspace.json
vendored
4
vault/.obsidian/workspace.json
vendored
|
|
@ -199,6 +199,8 @@
|
|||
},
|
||||
"active": "17bd4a6166f789d0",
|
||||
"lastOpenFiles": [
|
||||
"runbooks/pymc-repeater-kiss-tnc-reenumeration.md",
|
||||
"runbooks/pymc-repeater-kiss-tnc-reenumeration.md.tmp.5281.e47d0cbf9c9a",
|
||||
"runbooks/headscale-oidc-boot-order.md",
|
||||
"runbooks/headscale-oidc-boot-order.md.tmp.246153.e2a5642e10dd",
|
||||
"runbooks/headscale-onboard-node.md.tmp.246153.9adf70459e77",
|
||||
|
|
@ -209,7 +211,6 @@
|
|||
"runbooks/ia-cli-reference.md.tmp.3603785.daa56d288051",
|
||||
"runbooks/pipeline-patterns.md.tmp.3603785.ea95256eaac1",
|
||||
"runbooks/headless-browser-page-verification.md.tmp.3603785.4eb1d115aefe",
|
||||
"runbooks/headless-browser-page-verification.md.tmp.3603785.f017d6928577",
|
||||
"runbooks/headless-browser-page-verification.md",
|
||||
"runbooks/central-deploy-cutover.md",
|
||||
"runbooks/fleet-magicdns-resolved-migration.md",
|
||||
|
|
@ -235,7 +236,6 @@
|
|||
"concepts/ocr.md",
|
||||
"archive/projects/mmud/mmud-prompts/mmud-prompts/01-update-planned.md",
|
||||
"INDEX.md",
|
||||
"glossary.md",
|
||||
"assets/echo6yellow_logo_422x422_square.png",
|
||||
"assets/echo6yellow_logo_422x81.png",
|
||||
"assets/echo6_logo.png",
|
||||
|
|
|
|||
187
vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md
Normal file
187
vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md
Normal file
|
|
@ -0,0 +1,187 @@
|
|||
---
|
||||
title: pymc-repeater KISS TNC Re-enumeration Fix
|
||||
type: runbook
|
||||
tags:
|
||||
- mesh
|
||||
related:
|
||||
- [[meshtastic-sidecar-node]]
|
||||
updated: 2026-07-01
|
||||
---
|
||||
# pymc-repeater KISS TNC Re-enumeration Fix
|
||||
|
||||
`pymc-repeater.service` on **aida-nebra** (Tailscale 100.64.0.9) runs the Meshcore LoRa repeater node `EchoBase` on 910.525 MHz SF7. It talks to a RAK4631 USB KISS TNC (VID 239a / PID 8029) via the `/dev/pymc-radio` udev symlink. The Nebra SX1262 hat on the same machine runs a separate `meshtasticd` instance on its own radio — see [[meshtastic-sidecar-node]] for that hardware's setup. Do not touch `meshtasticd` or the SX1262 hat during any pymc-repeater work.
|
||||
|
||||
---
|
||||
|
||||
## Background — Why This Breaks
|
||||
|
||||
`KissModemWrapper` holds the serial file descriptor open but has no reopen logic. When the RAK4631 re-enumerates (power cycle, USB glitch, kernel rescan), the kernel assigns a new node: observed `ttyACM0` → `ttyACM1` around 2026-06-23. The service keeps a stale fd pointing at `/dev/ttyACMx (deleted)`. Systemd still reports `active (running)` — the service is silently deaf and mute.
|
||||
|
||||
---
|
||||
|
||||
## Symptom
|
||||
|
||||
Every ~30 seconds in the journal:
|
||||
|
||||
```
|
||||
KissModemWrapper - ERROR - Serial write error: ... [Errno 5] Input/output error
|
||||
TX frame write failed, dropping frame
|
||||
```
|
||||
|
||||
Zero RF traffic in or out despite a green `systemctl status`.
|
||||
|
||||
---
|
||||
|
||||
## Diagnose
|
||||
|
||||
### 1. Check the journal for the I/O error spam
|
||||
|
||||
```bash
|
||||
ssh zvx@100.64.0.9
|
||||
journalctl -u pymc-repeater.service --since '5 min ago' --no-pager
|
||||
```
|
||||
|
||||
Look for `[Errno 5] Input/output error` or `TX frame write failed` — either confirms stale fd.
|
||||
|
||||
### 2. Inspect the process's open file descriptors
|
||||
|
||||
```bash
|
||||
sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM
|
||||
```
|
||||
|
||||
- **Stale (broken):** symlink target contains `(deleted)` — smoking gun.
|
||||
- **Healthy:** target points at a real `/dev/ttyACMx`.
|
||||
|
||||
### 3. Confirm where the device landed after re-enumeration
|
||||
|
||||
```bash
|
||||
ls -l /dev/pymc-radio
|
||||
```
|
||||
|
||||
Shows which `ttyACM*` the live RAK4631 is on now.
|
||||
|
||||
---
|
||||
|
||||
## Manual Recovery
|
||||
|
||||
A restart clears the stale fd and re-opens the device via the udev symlink:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart pymc-repeater.service
|
||||
```
|
||||
|
||||
### Verify recovery
|
||||
|
||||
```bash
|
||||
# fd should now point at a real ttyACMx (not deleted)
|
||||
sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM
|
||||
|
||||
# Journal should show reconnect and normal TX
|
||||
journalctl -u pymc-repeater.service -n 20 --no-pager
|
||||
```
|
||||
|
||||
Expected healthy journal lines:
|
||||
|
||||
```
|
||||
KISS modem connected to /dev/pymc-radio at 115200 baud
|
||||
Retransmitted packet ...
|
||||
```
|
||||
|
||||
The `[Errno 5]` spam must be absent.
|
||||
|
||||
---
|
||||
|
||||
## Permanent Fix (applied 2026-07-01)
|
||||
|
||||
Two changes make pymc-repeater self-healing across any future re-enumeration. Original files were backed up as `.bak` alongside each.
|
||||
|
||||
### 1. udev rule — `/etc/udev/rules.d/99-pymc-radio.rules`
|
||||
|
||||
Added `TAG+="systemd"` and `ENV{SYSTEMD_WANTS}="pymc-repeater.service"` to the existing VID 239a / PID 8029 rule:
|
||||
|
||||
```udev
|
||||
SUBSYSTEM=="tty", ATTRS{idVendor}=="239a", ATTRS{idProduct}=="8029", \
|
||||
SYMLINK+="pymc-radio", \
|
||||
TAG+="systemd", \
|
||||
ENV{SYSTEMD_WANTS}="pymc-repeater.service"
|
||||
```
|
||||
|
||||
On any future re-enumeration, udev creates the `dev-pymc\x2dradio.device` unit for the new node and systemd activates `pymc-repeater.service` against it automatically.
|
||||
|
||||
### 2. systemd unit — `/etc/systemd/system/pymc-repeater.service`
|
||||
|
||||
`[Unit]` additions:
|
||||
|
||||
```ini
|
||||
BindsTo=dev-pymc\x2dradio.device
|
||||
After=dev-pymc\x2dradio.device
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
```
|
||||
|
||||
`[Service]` additions:
|
||||
|
||||
```ini
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
```
|
||||
|
||||
`BindsTo` stops the service cleanly when the TNC drops (the device unit disappears) and restarts it when the device returns. `Restart=always` with `RestartSec=5` ensures fast recovery from any modem crash without the device unit cycling.
|
||||
|
||||
---
|
||||
|
||||
## Verify the Resilience Wiring Is Intact
|
||||
|
||||
Run these on aida-nebra after any future maintenance to confirm the permanent fix is still in place:
|
||||
|
||||
```bash
|
||||
# BindsTo and After wired to the device unit
|
||||
systemctl show pymc-repeater.service -p BindsTo -p After | grep -i pymc
|
||||
|
||||
# udev tagging correct
|
||||
udevadm info -q property -n /dev/pymc-radio | grep -i SYSTEMD
|
||||
|
||||
# device unit tracked by systemd
|
||||
systemctl list-units --type=device --all | grep -i pymc
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
BindsTo=dev-pymc\x2dradio.device
|
||||
After=... dev-pymc\x2dradio.device ...
|
||||
SYSTEMD_WANTS=pymc-repeater.service
|
||||
dev-pymc\x2dradio.device loaded active plugged ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Secondary Issue (non-blocking)
|
||||
|
||||
RRD graphing fails with `rrdtool not available`. Metrics/graphing only — no impact on RF operation.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Host | aida-nebra (100.64.0.9) |
|
||||
| Service | `pymc-repeater.service` |
|
||||
| Node / freq | EchoBase, 910.525 MHz SF7 |
|
||||
| TNC hardware | RAK4631 USB KISS TNC (VID 239a / PID 8029) |
|
||||
| udev symlink | `/dev/pymc-radio` |
|
||||
| udev rule | `/etc/udev/rules.d/99-pymc-radio.rules` |
|
||||
| Unit file | `/etc/systemd/system/pymc-repeater.service` |
|
||||
| Unrelated radio | Nebra SX1262 hat → `meshtasticd` (do not touch) |
|
||||
|
||||
```bash
|
||||
# Restart (manual recovery)
|
||||
sudo systemctl restart pymc-repeater.service
|
||||
|
||||
# Tail journal
|
||||
journalctl -u pymc-repeater.service -f
|
||||
|
||||
# Check fd state
|
||||
sudo ls -l /proc/$(systemctl show -p MainPID --value pymc-repeater.service)/fd/ | grep ttyACM
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue