echo6-docs/runbooks/meshtasticd-sim-nodes-runbook.md

# Meshtasticd SIM Node Runbook — LXC Deployment

## Overview

This runbook covers deploying meshtasticd SIM (virtual) nodes inside LXC containers on Proxmox, each paired with a dedicated service (BBS, MeshSense, etc.). SIM nodes communicate with your real radio node over UDP and appear as normal nodes on the mesh — clients, maps, and other services can't tell the difference.

**Design principle:** One container = one SIM daemon + one service. Clean isolation, easy to snapshot, migrate, or tear down without affecting anything else.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        Proxmox Host                             │
│                                                                 │
│  ┌───────────────────┐                                          │
│  │  Real Radio Node   │  (LXC, bare metal, Pi, or USB device)   │
│  │  meshtasticd       │                                         │
│  │  Port 4403         │                                         │
│  │  /dev/ttyUSB0      │                                         │
│  │  UDP enabled       │                                         │
│  └────────┬──────────┘                                          │
│           │  UDP (vmbr0 or dedicated bridge)                    │
│           │                                                     │
│     ┌─────┴─────┬───────────────┐                               │
│     │           │               │                               │
│  ┌──▼────────┐ ┌▼────────────┐ ┌▼────────────┐                  │
│  │ LXC: BBS  │ │ LXC: Sense  │ │ LXC: Bot    │  ...more as     │
│  │           │ │             │ │             │  needed          │
│  │ mesht. sim│ │ mesht. sim  │ │ mesht. sim  │                  │
│  │ port 4403 │ │ port 4403   │ │ port 4403   │                  │
│  │ + BBS svc │ │ + MeshSense │ │ + bot svc   │                  │
│  └───────────┘ └─────────────┘ └─────────────┘                  │
└─────────────────────────────────────────────────────────────────┘
```

Since each SIM daemon is alone in its container, they can all use the default port (4403) internally. No port juggling needed.

---

## Prerequisites

- Proxmox host with LXC support
- A working real radio meshtasticd instance somewhere on the network with UDP enabled
- An LXC template (Ubuntu 22.04/24.04 or Debian 12 recommended)
- Network bridge accessible to both the real radio node and LXC containers

---

## Step 1: Create the LXC Container

From the Proxmox host CLI:

```bash
# Create an unprivileged container with static IP and TUN device for Tailscale
pct create <CTID> local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname mesh-<service> \
  --memory 512 \
  --cores 1 \
  --rootfs local-lvm:4 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.<CTID>/24,gw=192.168.1.1 \
  --unprivileged 1 \
  --features nesting=1 \
  --start 0 \
  --password <from .ref/credentials>

# Add TUN device for Tailscale (must be done before first start)
cat >> /etc/pve/lxc/<CTID>.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF

pct start <CTID>
```

Adjust memory/cores/storage to taste. SIM daemons are lightweight — 512MB RAM and 1 core is plenty for the daemon plus most services.

### Bootstrap standard packages

```bash
echo6-bootstrap-ct.sh <CTID>
```

If the script isn't on the Proxmox host, run `echo6-onboard-node.sh` first. See `runbooks/proxmox-onboard-node.md`.

---

## Step 2: Install meshtasticd Inside the Container

Install from the Meshtastic OBS (OpenSUSE Build Service) repository:

```bash
pct exec <CTID> -- bash -c "
  # Add the Meshtastic beta repo (Debian 12)
  curl -fsSL https://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/Release.key | \
    gpg --dearmor -o /etc/apt/trusted.gpg.d/network_Meshtastic_beta.gpg
  echo 'deb http://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/ /' > \
    /etc/apt/sources.list.d/meshtasticd.list
  apt-get update -qq
  DEBIAN_FRONTEND=noninteractive apt-get install -y meshtasticd
"
```

For Ubuntu 24.04 containers, replace `Debian_12` with `Debian_13` in both URLs above.

Verify it's installed:

```bash
pct exec <CTID> -- meshtasticd --version
```

---

## Step 3: Configure SIM Mode

Create or edit the config file. Since this is a dedicated container, you can use the default paths.

### /etc/meshtasticd/config.yaml

```yaml
Lora:
  Module: sim

General:
  # Prevent loading any default config.d overrides
  # ConfigDirectory: /etc/meshtasticd/config.d/

  # REQUIRED: Set a unique MAC address for this SIM node
  # Last 3 hex pairs = node color in client apps
  MACAddress: "DE:AD:00:FF:00:01"
```

### MAC Address Guidelines

- **Every SIM node must have a unique MAC.** If two nodes share a MAC, you'll get node ID collisions and unpredictable behavior.
- The last 3 byte pairs map to a hex color code displayed in client apps.
- Pick a scheme that makes sense for your deployment, e.g.:
  - `DE:AD:00:FF:00:01` — SIM node 1 (BBS)
  - `DE:AD:00:00:FF:02` — SIM node 2 (MeshSense)
  - `DE:AD:00:FF:FF:03` — SIM node 3 (bot)

---

## Step 4: Enable UDP Bridging

UDP is what connects SIM nodes to the rest of your mesh. Every meshtasticd instance — real radio and all SIM nodes — needs UDP enabled and must be able to reach each other on the network.

Add to your SIM node's config:

```yaml
Networking:
  EnableUDP: true
```

**Also ensure your real radio node has UDP enabled** in its own config.

### Network Considerations

For UDP mesh traffic to flow between LXC containers and your real radio node:

- All containers and the real radio host must be on the **same Layer 2 network** (same bridge, same subnet) — UDP broadcast/multicast needs to reach all instances.
- If your real radio runs on a different host (e.g., a Raspberry Pi), make sure it's on the same VLAN/subnet as the LXC bridge.
- Proxmox's default `vmbr0` bridge works fine if everything is on the same network.
- If you're running the real radio in its own LXC and passing through USB, the same bridge rules apply.

**Firewall note:** If you have Proxmox firewall or iptables rules on the host, ensure UDP traffic between containers is not blocked. Meshtasticd uses UDP broadcast by default — verify your bridge allows broadcast forwarding.

---

## Step 5: Configure the systemd Service

The meshtasticd package likely installs a default unit file. If you need to customize it:

```bash
sudo systemctl edit meshtasticd --full
```

Or create/verify `/etc/systemd/system/meshtasticd.service`:

```ini
[Unit]
Description=Meshtastic Daemon - SIM Node
After=network.target

[Service]
Type=simple
User=meshtasticd
Group=meshtasticd
ExecStart=/usr/bin/meshtasticd -d /var/lib/meshtasticd/vfs -c /etc/meshtasticd/config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Since this is the only meshtasticd instance in the container, you don't need custom port flags — the default port is fine.

Ensure directory ownership:

```bash
sudo mkdir -p /var/lib/meshtasticd/vfs
sudo chown -R meshtasticd:meshtasticd /var/lib/meshtasticd
sudo chown -R meshtasticd:meshtasticd /etc/meshtasticd
```

Start and enable:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now meshtasticd
sudo systemctl status meshtasticd
```

---

## Step 5b: Install Tailscale and Register with Headscale

Install Tailscale inside the container:

```bash
pct exec <CTID> -- bash -c "
  echo nameserver 1.1.1.1 > /etc/resolv.conf
  echo nameserver 8.8.8.8 >> /etc/resolv.conf
  curl -fsSL https://tailscale.com/install.sh | sh
"
```

Generate a preauth key on Contabo (user ID 1 = echo6):

```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```

Register the node:

```bash
pct exec <CTID> -- tailscale up --login-server https://vpn.echo6.co --authkey <PREAUTH_KEY> --hostname mesh-<service>

# Verify
pct exec <CTID> -- tailscale status
```

**Note:** The TUN device must already be configured in the container config (done in Step 1). If Tailscale fails to start, verify `/dev/net/tun` exists inside the container.

---

## Step 6: Configure the SIM Node

With the daemon running and no service connected yet, configure it via CLI:

```bash
# Install the Meshtastic Python CLI
pip install meshtastic

# Check that the node is up
meshtastic --host localhost --info

# Set a descriptive name
meshtastic --host localhost --set-owner "BBS Node"
meshtastic --host localhost --set-owner-short "BBS"

# Optionally set a position (makes it appear on mesh maps)
meshtastic --host localhost --setlat XX.XXXX --setlon -XXX.XXXX --setalt XXXX

# Set the node role as appropriate
meshtastic --host localhost --set device.role CLIENT
```

If you don't set a position, the node still functions on the mesh but won't appear on maps.

---

## Step 7: Install and Connect Your Service

Now install whichever service this container is dedicated to and point it at `localhost:4403` (or whatever the default meshtasticd API port is).

### Example: BBS

```bash
# Install your BBS software of choice
# Point it at the local meshtasticd instance
# BBS_CONFIG: host=localhost, port=4403
```

### Example: MeshSense

```bash
# MeshSense supports non-default ports
# Configure it to connect to localhost:4403
```

**Reminder:** One client API connection per daemon. Once the service is connected, don't also try to connect a client app to the same instance. If you need to reconfigure the node, stop the service first.

---

## Provisioning Additional Containers

For each new service, repeat steps 1–7 with:

1. A new container ID and hostname (e.g., `mesh-sense`, `mesh-bot`)
2. A **unique MAC address** in config.yaml
3. The specific service installed alongside meshtasticd

### Quick Clone Approach

Once you have one container fully set up, you can clone it in Proxmox and just change:

- Container hostname
- MAC address in `/etc/meshtasticd/config.yaml`
- The service installed/configured
- Node owner name via `meshtastic --host localhost --set-owner "NewName"`

```bash
# Clone from Proxmox CLI
pct clone 201 202 --hostname mesh-sense --full
pct start 202
pct enter 202

# Update the MAC address
nano /etc/meshtasticd/config.yaml  # change MACAddress

# Restart meshtasticd to pick up new MAC
systemctl restart meshtasticd

# Reconfigure node identity
meshtastic --host localhost --set-owner "MeshSense"
```

---

## USB Passthrough for Real Radio Container

If you want your real radio node in an LXC too (rather than bare metal), you need to pass the USB device through to the container.

On the Proxmox host, find the device:

```bash
ls -la /dev/serial/by-id/
# or
lsusb
```

Add to the container config (`/etc/pve/lxc/<CTID>.conf`):

```
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.mount.entry: /dev/ttyUSB0 dev/ttyUSB0 none bind,optional,create=file
```

Adjust the device path and cgroup major number as needed for your hardware. The container will also need a `config.yaml` with the real LoRa module config instead of `Module: sim`.

---

## Troubleshooting

### SIM nodes not seeing the real radio (or each other)
- Verify UDP is enabled on **all** instances
- Confirm all containers are on the same bridge/subnet
- Check for firewall rules blocking UDP broadcast between containers
- Test basic connectivity: `ping` between containers

### Node ID collisions / "things get weird"
- Every SIM node must have a **unique MAC address** — check each container's config.yaml
- After changing a MAC, restart meshtasticd and verify with `meshtastic --host localhost --info`

### Service can't connect to meshtasticd
- Is meshtasticd actually running? `systemctl status meshtasticd`
- Is another client already connected? Only one API connection per instance.
- Check the port: `ss -tlnp | grep 4403`

### Config not taking effect
- Make sure `ConfigDirectory` is commented out or pointed somewhere empty so default config.d files don't override your settings
- Restart after config changes: `systemctl restart meshtasticd`

### Permission errors
- VFS and config directories must be owned by the `meshtasticd` user
  ```bash
  chown -R meshtasticd:meshtasticd /var/lib/meshtasticd
  chown -R meshtasticd:meshtasticd /etc/meshtasticd
  ```

---

## Quick Reference

```bash
# Inside any SIM container:
systemctl status meshtasticd          # Check daemon status
journalctl -u meshtasticd -f          # Follow logs
meshtastic --host localhost --info    # Node info (only if no service is connected)

# From Proxmox host:
pct list                              # List all containers
pct enter <CTID>                      # Shell into container
pct exec <CTID> -- systemctl status meshtasticd  # Check without entering
```

---

## Container Inventory Template

Track your deployment:

| CTID | Hostname    | MAC Address         | Service    | Port | Notes           |
|------|-------------|---------------------|------------|------|-----------------|
| 200  | mesh-radio  | (hardware)          | Real radio | 4403 | USB passthrough |
| 201  | mesh-bbs    | DE:AD:00:FF:00:01   | BBS        | 4403 | —               |
| 202  | mesh-sense  | DE:AD:00:00:FF:02   | MeshSense  | 4403 | —               |
| 203  | mesh-bot    | DE:AD:00:FF:FF:03   | Bot        | 4403 | —               |

---

## Credits

Procedure sourced from a community discussion between pdxlocs, tedward, and wehooper4 regarding multi-daemon meshtasticd deployments with SIM mode. Adapted for LXC/Proxmox deployment.