echo6-docs/vault/projects/meshtastic-headscale-runbook.md

---
title: IdahoMesh Tailnet Runbook
type: project
tags:
  - mesh
aliases: []
related:
  - [[idahomesh-bridge-setup]]
  - [[idahomesh-vpn-device-setup]]
  - [[meshtastic-sidecar-node]]
  - [[headscale-onboard-node]]
  - [[caddy]]
updated: 2026-07-11
---
# IdahoMesh Tailnet Runbook

## Overview

Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.

### Architecture

```
Echo6 Headscale (100.64.0.x)
       ↓ (one-way only)
  [Bridge LXC]  ← dual tailscaled, NAT + firewall
       ↓
IdahoMesh Headscale (100.100.0.x)
       ↕                ↕
  Nebra CM3s      Sidpatchy's devices
```

> **Security:** The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.

### IP Allocation

| Tailnet     | Prefix           | Notes                                    |
|-------------|------------------|------------------------------------------|
| Echo6       | 100.64.0.0/10    | Existing, do not change                  |
| IdahoMesh   | 100.100.0.0/16   | Within Tailscale's required 100.64.0.0/10 supernet |

### Infrastructure

| Component | VMID | Host | Local IP | Purpose |
|-----------|------|------|----------|---------|
| meshtastic-hs | CT 106 | utility | 192.168.1.106 | IdahoMesh Headscale server |
| mesh-bridge | CT 107 | utility | 192.168.1.107 | One-way bridge between tailnets |

---

## Phase 1: IdahoMesh Headscale Instance

### 1.1 Create the LXC on utility

```bash
ssh root@192.168.1.241

pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname meshtastic-hs \
  --memory 512 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:4 \
  --unprivileged 1 \
  --onboot 1 \
  --start 1
```

Bootstrap standard packages:

```bash
echo6-bootstrap-ct.sh 106
```

### 1.2 Install Headscale

```bash
pct exec 106 -- bash -c '
apt update && apt install -y curl

HEADSCALE_VERSION="0.28.0"
curl -Lo /usr/local/bin/headscale \
  "https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
chmod +x /usr/local/bin/headscale

mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
'
```

### 1.3 Configure Headscale

Create `/etc/headscale/config.yaml`:

```yaml
server_url: https://vpn.idahomesh.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v4: 100.100.0.0/16
  v6: fd7a:115c:a1e0:ab00::/56
  allocation: sequential

derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  paths: []
  auto_update_enabled: true
  update_frequency: 3h

disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m

database:
  type: sqlite
  debug: false
  gorm:
    prepare_stmt: true
    parameterized_queries: true
    skip_err_record_not_found: true
    slow_threshold: 1000
  sqlite:
    path: /var/lib/headscale/db.sqlite
    write_ahead_log: true
    wal_autocheckpoint: 1000

policy:
  mode: file
  path: /etc/headscale/acl.json

dns:
  magic_dns: true
  base_domain: mesh.local
  override_local_dns: true
  nameservers:
    global:
      - 1.1.1.1
      - 9.9.9.9
    split: {}
  search_domains: []
  extra_records: []

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

logtail:
  enabled: false

randomize_client_port: false

log:
  level: info
  format: text
```

> **Note:** Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind [[caddy]], so TLS termination happens at the reverse proxy.

### 1.4 Create the ACL Policy

Create `/etc/headscale/acl.json`:

```json
{
  "groups": {
    "group:malice":     ["malice@"],
    "group:sidpatchy":  ["sidpatchy@"],
    "group:nebra":      ["nebra@"]
  },

  "acls": [
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:nebra:*"],
      "comment": "Nebra gateways talk to each other"
    },
    {
      "action": "accept",
      "src": ["group:malice"],
      "dst": ["group:nebra:*"],
      "comment": "Echo6 bridge can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:sidpatchy"],
      "dst": ["group:nebra:*"],
      "comment": "Sidpatchy can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:malice:*", "group:sidpatchy:*"],
      "comment": "Nebras can respond back to both"
    }
  ]
}
```

> **Important:** Headscale v0.28.0 requires usernames in ACL groups to have `@` suffix (e.g., `malice@`). The `--user` flag on CLI commands takes user IDs (integers), not names.
>
> **No malice↔Sidpatchy rules.** They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).

### 1.5 Create systemd Service

Create `/etc/systemd/system/headscale.service`:

```ini
[Unit]
Description=Headscale - IdahoMesh Tailnet
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl enable --now headscale
systemctl status headscale
```

### 1.6 Create Users and Preauthkeys

```bash
headscale users create echo6
headscale users create sidpatchy
headscale users create nebra

# For the bridge LXC (your side)
headscale preauthkeys create --user echo6 --expiration 24h
# Save this key ^^^

# For Sidpatchy — send this to him
headscale preauthkeys create --user sidpatchy --expiration 72h
# Save this key ^^^

# For Nebra CM3 gateways (reusable so all Nebras use same key)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# Save this key ^^^
```

---

## Phase 2: Bridge LXC (CT 107 on utility)

This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows **one-way only**: Echo6 → IdahoMesh.

### 2.1 Create the LXC

```bash
ssh root@192.168.1.241

pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname mesh-bridge \
  --memory 256 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:2 \
  --unprivileged 1 \
  --features nesting=1 \
  --onboot 1 \
  --start 1
```

Add TUN device access for Tailscale (on the Proxmox host):

```bash
# Stop the container first
pct stop 107

cat >> /etc/pve/lxc/107.conf << 'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF

pct start 107
```

### 2.2 Install Tailscale

```bash
pct exec 107 -- bash -c '
curl -fsSL https://tailscale.com/install.sh | sh
'
```

### 2.3 Set Up Dual tailscaled

Create directories for the second instance:

```bash
pct exec 107 -- bash -c '
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
'
```

The default tailscaled service handles Echo6. Create a second service for IdahoMesh:

Create `/etc/systemd/system/tailscaled-meshtastic.service`:

```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/sbin/tailscaled \
  --state=/var/lib/tailscale-meshtastic/tailscaled.state \
  --socket=/var/run/tailscale-meshtastic/tailscaled.sock \
  --port=41642 \
  --tun=tailscale1
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```

### 2.4 Enable IP Forwarding

```bash
cat <<EOF > /etc/sysctl.d/99-bridge.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```

### 2.5 Join Both Tailnets

```bash
# Join Echo6 (default tailscaled instance)
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
tailscale up \
  --login-server=https://vpn.echo6.co \
  --advertise-routes=100.100.0.0/16 \
  --accept-routes

# Join IdahoMesh (second instance)
# Do NOT advertise Echo6 routes — one-way only
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<echo6-preauthkey-from-step-1.6> \
  --accept-routes
```

After joining, approve the advertised route on Echo6 Headscale only:

```bash
# On Echo6 Headscale (edge2 CT 107) — enable the 100.100.0.0/16 route
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale routes list"
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale routes enable -r <route-id>"

# NO route approval needed on IdahoMesh Headscale — nothing is advertised
```

### 2.6 Configure One-Way Firewall and NAT

This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.

Install iptables:

```bash
apt install -y iptables iptables-persistent
```

Apply rules:

```bash
# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE

# Allow Echo6 → IdahoMesh (outbound)
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT

# Allow established/related return traffic only (responses to Echo6-initiated connections)
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT

# DROP all new connections from IdahoMesh → Echo6
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP
```

Persist across reboots:

```bash
netfilter-persistent save
```

Verify:

```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```

> **What this achieves:**
> - Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
> - Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
> - Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
> - Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)

---

## Phase 3: Expose vpn.idahomesh.com

### 3.1 Issue SSL Certificate

```bash
ssh root@192.168.1.241

pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
'
```

### 3.2 Install Certificate

```bash
pct exec 101 -- bash -c '
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
  --cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
  --key-file /etc/caddy/certs/vpn.idahomesh.com.key \
  --fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
  --reloadcmd "systemctl reload caddy"

chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
'
```

### 3.3 Add Caddy Site Block

```bash
pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''

vpn.idahomesh.com {
    tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
    reverse_proxy 192.168.1.106:8080
}
EOF
systemctl reload caddy'
```

### 3.4 Add GoDaddy DNS Record

```bash
# On cortex/TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163
```

### 3.5 Verify

```bash
dig +short vpn.idahomesh.com
# Should return 199.6.36.163

curl -I https://vpn.idahomesh.com
```

---

## Phase 4: Register Nebra CM3 Gateways

Only Burley Butte for now. See `idahomesh-vpn-device-setup.md` for the full device onboarding runbook.

```bash
# SSH to Burley Butte
curl -fsSL https://tailscale.com/install.sh | sh

tailscale up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<nebra-preauthkey-from-step-1.6> \
  --hostname=burley-butte
```

Verify on the IdahoMesh Headscale:

```bash
# On CT 106
headscale nodes list
```

---

## Phase 5: Sidpatchy Onboarding

Send Sidpatchy the following:

1. **IdahoMesh VPN URL:** `https://vpn.idahomesh.com`
2. **Preauthkey:** (the one generated in Step 1.6 for sidpatchy)
3. **Device setup runbook:** `idahomesh-vpn-device-setup.md`

---

## Phase 6: Verification

### From the bridge LXC (CT 107)

```bash
# Ping Burley Butte via IdahoMesh tailnet
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte

# Check status on both tailnets
tailscale status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```

### From any Echo6 machine (via bridge routes)

```bash
# Should be routable through the bridge (NAT'd)
ping 100.100.0.x  # Burley Butte's IdahoMesh IP
```

### Verify isolation — from IdahoMesh side

```bash
# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
ping 100.64.0.14  # cortex — should timeout/unreachable
```

---

## Quick Reference

| Component      | Location         | Tailnet    | IP              |
|----------------|------------------|------------|-----------------|
| IdahoMesh HS   | CT 106, utility  | IdahoMesh  | 192.168.1.106   |
| Bridge LXC     | CT 107, utility  | Both       | 192.168.1.107   |
| Burley Butte   | Field site       | IdahoMesh  | 100.100.0.x     |
| Sidpatchy      | Remote           | IdahoMesh  | 100.100.0.x     |

---

## Maintenance Notes

- **Preauthkeys expire.** Generate long-lived reusable keys for Nebras, short-lived for humans.
- **Headscale updates:** Check releases at https://github.com/juanfont/headscale/releases
- **ACL changes:** Edit `/etc/headscale/acl.json` on CT 106, then `systemctl reload headscale`
- **Firewall rules:** Persisted via `netfilter-persistent` on CT 107. Verify after reboot with `iptables -L FORWARD -v -n`
- **If a Nebra goes offline:** Check `headscale nodes list` — may need a new key if expired.
- **Sidpatchy wants off?** `headscale nodes delete -i <node-id>` and revoke the preauthkey.
- **Device setup instructions:** See `idahomesh-vpn-device-setup.md`

---

## Headscale 0.28 → 0.29 Upgrade Guide

**Rehearsed on CT 106 (meshtastic-hs) 2026-06-21. Result: success. Use this as the playbook for CT 107 (mesh-bridge / Echo6 Headscale).**

### Breaking changes that require action before starting

#### 1. `randomize_client_port` removed from config.yaml (HARD BLOCKER)

Headscale 0.29 **refuses to start** if `randomize_client_port` exists in `config.yaml`. Remove the line:

```bash
sed -i '/^randomize_client_port:/d' /etc/headscale/config.yaml
```

If the feature is needed, move it to the ACL policy file (`/etc/headscale/acl.json`) as a top-level key:
```json
{ "randomizeClientPort": true }
```

Default is `false`; simply removing the key is equivalent to `randomize_client_port: false`.

#### 2. `ephemeral_node_inactivity_timeout` removed from top-level config (deprecation → removed)

The old top-level key is gone in 0.29. Replace with the new nested path:

```yaml
# OLD (remove this):
ephemeral_node_inactivity_timeout: 30m

# NEW (replace with):
node:
  ephemeral:
    inactivity_timeout: 30m
```

0.29 accepts the old key with a warning and uses the correct value, but the warning appears on every startup. Fix it to keep logs clean.

#### 3. Strict version upgrade path enforced

0.29 blocks skipping minor versions. If upgrading from 0.27 or earlier, you MUST go 0.27 → 0.28 → 0.29 one step at a time. We were on 0.28, so this is not a blocker for this fleet.

#### 4. ACL wildcard `*` behavior changed (review your policy)

In 0.29, bare `*` in ACL `src`/`dst` resolves to the CGNAT+ULA range (tailnet nodes only) instead of all IPs. If you had `"dst": ["*:*"]` meaning "any internet IP," you must switch to `autogroup:danger-all` as a source or explicit CIDRs. **The IdahoMesh ACL does not use bare `*` — no action needed for this tailnet.**

#### 5. Minimum Tailscale client version raised to v1.80.0

Any tailscale client older than v1.80.0 will be rejected by 0.29. Verify all registered nodes are running a sufficiently recent tailscale before upgrading.

### Other notable 0.29 changes (no action required, FYI)

- GivenName collision suffix changed from random hash (`laptop-abc12xyz`) to monotonic numeric (`laptop`, `laptop-1`). MagicDNS names for nodes with old random suffixes will change on upgrade. The raw Hostname column is unchanged.
- `oidc.expiry` removed; use `node.expiry` for all registration methods.
- `headscale nodes register` deprecated in favour of `headscale auth register --auth-id`.
- `--namespace` flag removed from nodes commands (was already replaced by `--user`).
- New: `trusted_proxies` config option for True-Client-IP headers (previously honoured from any client).
- DB migration is automatic on first start — no manual migration command needed.

### Rollback procedure (if upgrade fails)

1. Stop the service: `systemctl stop headscale`
2. Reinstall the 0.28.0 binary: `cp /usr/local/bin/headscale.bak-v0.28.0-pre029 /usr/local/bin/headscale`
3. Restore config: `cp /etc/headscale/config.yaml.bak-pre029-20260621 /etc/headscale/config.yaml`
4. Restore DB: `cp /root/headscale-db-pre029-20260621.sqlite /var/lib/headscale/db.sqlite`
5. Start: `systemctl start headscale`

> **Note:** 0.29 blocks downgrading once the DB migration has run. The DB backup from step 4 predates the migration, so this rollback is clean.

### CT 106 (meshtastic-hs) upgrade record

- **Date:** 2026-06-21
- **From:** v0.28.0 → **To:** v0.29.1
- **Deployment:** native systemd binary at `/usr/local/bin/headscale`
- **Config changes made:**
  - Removed `randomize_client_port: false`
  - Replaced `ephemeral_node_inactivity_timeout: 30m` with `node.ephemeral.inactivity_timeout: 30m`
- **DB migration:** automatic, clean, no errors in logs
- **Nodes after upgrade:** all 3 registered (burley-butte offline, mesh-bridge online, mt-isr offline) — same as before
- **Backups (pre-upgrade):** `/etc/headscale/config.yaml.bak-pre029-20260621`, `/root/headscale-db-pre029-20260621.sqlite`, `/usr/local/bin/headscale.bak-v0.28.0-pre029`

### CT 107 (mesh-bridge / Echo6 Headscale) upgrade checklist

CT 107 runs Docker (not native binary). Steps will differ:

1. Back up: `docker cp headscale-vanilla:/var/lib/headscale/db.sqlite /root/headscale-db-pre029-$(date +%Y%m%d).sqlite` and `cp /path/to/config.yaml config.yaml.bak-pre029`
2. Apply the two config changes above (remove `randomize_client_port`, migrate ephemeral timeout key)
3. Change the image tag in `docker-compose.yml` from `headscale/headscale:0.28.x` to `headscale/headscale:0.29.1`
4. `docker compose up -d`
5. Validate: `docker exec headscale-vanilla headscale version`, check `docker logs headscale-vanilla` for migration success, `docker exec headscale-vanilla headscale nodes list`

---

*Last updated: 2026-06-21*