echo6-docs/vault/projects/meshtastic-headscale-runbook.md

title	type	tags	aliases	related	updated
IdahoMesh Tailnet Runbook	project	mesh		idahomesh-bridge-setup idahomesh-vpn-device-setup meshtastic-sidecar-node headscale-onboard-node services	2026-07-13

IdahoMesh Tailnet Runbook

Overview

Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.

Architecture

Echo6 Headscale (100.64.0.x)
       ↓ (one-way only)
  [Bridge LXC]  ← dual tailscaled, NAT + firewall
       ↓
IdahoMesh Headscale (100.100.0.x)
       ↕                ↕
  Nebra CM3s      Sidpatchy's devices

Security: The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.

IP Allocation

Tailnet	Prefix	Notes
Echo6	100.64.0.0/10	Existing, do not change
IdahoMesh	100.100.0.0/16	Within Tailscale's required 100.64.0.0/10 supernet

Infrastructure

Component	VMID	Host	Local IP	Purpose
meshtastic-hs	CT 106	utility	192.168.1.106	IdahoMesh Headscale server
mesh-bridge	CT 107	utility	192.168.1.107	One-way bridge between tailnets

---

Phase 1: IdahoMesh Headscale Instance

1.1 Create the LXC on utility

ssh root@192.168.1.241

pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname meshtastic-hs \
  --memory 512 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:4 \
  --unprivileged 1 \
  --onboot 1 \
  --start 1

Bootstrap standard packages:

echo6-bootstrap-ct.sh 106

1.2 Install Headscale

pct exec 106 -- bash -c '
apt update && apt install -y curl

HEADSCALE_VERSION="0.28.0"
curl -Lo /usr/local/bin/headscale \
  "https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
chmod +x /usr/local/bin/headscale

mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
'

1.3 Configure Headscale

Create /etc/headscale/config.yaml:

server_url: https://vpn.idahomesh.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v4: 100.100.0.0/16
  v6: fd7a:115c:a1e0:ab00::/56
  allocation: sequential

derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  paths: []
  auto_update_enabled: true
  update_frequency: 3h

disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m

database:
  type: sqlite
  debug: false
  gorm:
    prepare_stmt: true
    parameterized_queries: true
    skip_err_record_not_found: true
    slow_threshold: 1000
  sqlite:
    path: /var/lib/headscale/db.sqlite
    write_ahead_log: true
    wal_autocheckpoint: 1000

policy:
  mode: file
  path: /etc/headscale/acl.json

dns:
  magic_dns: true
  base_domain: mesh.local
  override_local_dns: true
  nameservers:
    global:
      - 1.1.1.1
      - 9.9.9.9
    split: {}
  search_domains: []
  extra_records: []

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

logtail:
  enabled: false

randomize_client_port: false

log:
  level: info
  format: text

Note: Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind caddy, so TLS termination happens at the reverse proxy.

1.4 Create the ACL Policy

Create /etc/headscale/acl.json:

{
  "groups": {
    "group:malice":     ["malice@"],
    "group:sidpatchy":  ["sidpatchy@"],
    "group:nebra":      ["nebra@"]
  },

  "acls": [
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:nebra:*"],
      "comment": "Nebra gateways talk to each other"
    },
    {
      "action": "accept",
      "src": ["group:malice"],
      "dst": ["group:nebra:*"],
      "comment": "Echo6 bridge can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:sidpatchy"],
      "dst": ["group:nebra:*"],
      "comment": "Sidpatchy can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:malice:*", "group:sidpatchy:*"],
      "comment": "Nebras can respond back to both"
    }
  ]
}

Important: Headscale v0.28.0 requires usernames in ACL groups to have @ suffix (e.g., malice@). The --user flag on CLI commands takes user IDs (integers), not names.

No malice↔Sidpatchy rules. They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).

1.5 Create systemd Service

Create /etc/systemd/system/headscale.service:

[Unit]
Description=Headscale - IdahoMesh Tailnet
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now headscale
systemctl status headscale

1.6 Create Users and Preauthkeys

headscale users create echo6
headscale users create sidpatchy
headscale users create nebra

# For the bridge LXC (your side)
headscale preauthkeys create --user echo6 --expiration 24h
# Save this key ^^^

# For Sidpatchy — send this to him
headscale preauthkeys create --user sidpatchy --expiration 72h
# Save this key ^^^

# For Nebra CM3 gateways (reusable so all Nebras use same key)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# Save this key ^^^

---

Phase 2: Bridge LXC (CT 107 on utility)

This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows one-way only: Echo6 → IdahoMesh.

2.1 Create the LXC

ssh root@192.168.1.241

pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname mesh-bridge \
  --memory 256 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:2 \
  --unprivileged 1 \
  --features nesting=1 \
  --onboot 1 \
  --start 1

Add TUN device access for Tailscale (on the Proxmox host):

# Stop the container first
pct stop 107

cat >> /etc/pve/lxc/107.conf << 'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF

pct start 107

2.2 Install Tailscale

pct exec 107 -- bash -c '
curl -fsSL https://tailscale.com/install.sh | sh
'

2.3 Set Up Dual tailscaled

Create directories for the second instance:

pct exec 107 -- bash -c '
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
'

The default tailscaled service handles Echo6. Create a second service for IdahoMesh:

Create /etc/systemd/system/tailscaled-meshtastic.service:

[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/sbin/tailscaled \
  --state=/var/lib/tailscale-meshtastic/tailscaled.state \
  --socket=/var/run/tailscale-meshtastic/tailscaled.sock \
  --port=41642 \
  --tun=tailscale1
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic

2.4 Enable IP Forwarding

cat <<EOF > /etc/sysctl.d/99-bridge.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf

2.5 Join Both Tailnets

# Join Echo6 (default tailscaled instance)
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
tailscale up \
  --login-server=https://vpn.echo6.co \
  --advertise-routes=100.100.0.0/16 \
  --accept-routes

# Join IdahoMesh (second instance)
# Do NOT advertise Echo6 routes — one-way only
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<echo6-preauthkey-from-step-1.6> \
  --accept-routes

After joining, approve the advertised route on Echo6 Headscale only:

# On Echo6 Headscale (edge2 CT 107) — enable the 100.100.0.0/16 route
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale routes list"
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale routes enable -r <route-id>"

# NO route approval needed on IdahoMesh Headscale — nothing is advertised

2.6 Configure One-Way Firewall and NAT

This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.

Install iptables:

apt install -y iptables iptables-persistent

Apply rules:

# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE

# Allow Echo6 → IdahoMesh (outbound)
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT

# Allow established/related return traffic only (responses to Echo6-initiated connections)
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT

# DROP all new connections from IdahoMesh → Echo6
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP

Persist across reboots:

netfilter-persistent save

Verify:

iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n

What this achieves:

• Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
• Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
• Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
• Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)

---

Phase 3: Expose vpn.idahomesh.com

3.1 Issue SSL Certificate

ssh root@192.168.1.241

pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
'

3.2 Install Certificate

pct exec 101 -- bash -c '
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
  --cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
  --key-file /etc/caddy/certs/vpn.idahomesh.com.key \
  --fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
  --reloadcmd "systemctl reload caddy"

chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
'

3.3 Add Caddy Site Block

pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''

vpn.idahomesh.com {
    tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
    reverse_proxy 192.168.1.106:8080
}
EOF
systemctl reload caddy'

3.4 Add GoDaddy DNS Record

# On cortex/TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163

3.5 Verify

dig +short vpn.idahomesh.com
# Should return 199.6.36.163

curl -I https://vpn.idahomesh.com

---

Phase 4: Register Nebra CM3 Gateways

Only Burley Butte for now. See idahomesh-vpn-device-setup.md for the full device onboarding runbook.

# SSH to Burley Butte
curl -fsSL https://tailscale.com/install.sh | sh

tailscale up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<nebra-preauthkey-from-step-1.6> \
  --hostname=burley-butte

Verify on the IdahoMesh Headscale:

# On CT 106
headscale nodes list

---

Phase 5: Sidpatchy Onboarding

Send Sidpatchy the following:

1. IdahoMesh VPN URL: https://vpn.idahomesh.com
2. Preauthkey: (the one generated in Step 1.6 for sidpatchy)
3. Device setup runbook: idahomesh-vpn-device-setup.md

---

Phase 6: Verification

From the bridge LXC (CT 107)

# Ping Burley Butte via IdahoMesh tailnet
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte

# Check status on both tailnets
tailscale status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status

From any Echo6 machine (via bridge routes)

# Should be routable through the bridge (NAT'd)
ping 100.100.0.x  # Burley Butte's IdahoMesh IP

Verify isolation — from IdahoMesh side

# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
ping 100.64.0.14  # cortex — should timeout/unreachable

---

Quick Reference

Component	Location	Tailnet	IP
IdahoMesh HS	CT 106, utility	IdahoMesh	192.168.1.106
Bridge LXC	CT 107, utility	Both	192.168.1.107
Burley Butte	Field site	IdahoMesh	100.100.0.x
Sidpatchy	Remote	IdahoMesh	100.100.0.x

---

Maintenance Notes

• Preauthkeys expire. Generate long-lived reusable keys for Nebras, short-lived for humans.
• Headscale updates: Check releases at https://github.com/juanfont/headscale/releases
• ACL changes: Edit /etc/headscale/acl.json on CT 106, then systemctl reload headscale
• Firewall rules: Persisted via netfilter-persistent on CT 107. Verify after reboot with iptables -L FORWARD -v -n
• If a Nebra goes offline: Check headscale nodes list — may need a new key if expired.
• Sidpatchy wants off? headscale nodes delete -i <node-id> and revoke the preauthkey.
• Device setup instructions: See idahomesh-vpn-device-setup.md

---

Headscale 0.28 → 0.29 Upgrade Guide

Rehearsed on CT 106 (meshtastic-hs) 2026-06-21. Result: success. Use this as the playbook for CT 107 (mesh-bridge / Echo6 Headscale).

Breaking changes that require action before starting

1. randomize_client_port removed from config.yaml (HARD BLOCKER)

Headscale 0.29 refuses to start if randomize_client_port exists in config.yaml. Remove the line:

sed -i '/^randomize_client_port:/d' /etc/headscale/config.yaml

If the feature is needed, move it to the ACL policy file (/etc/headscale/acl.json) as a top-level key:

{ "randomizeClientPort": true }

Default is false; simply removing the key is equivalent to randomize_client_port: false.

2. ephemeral_node_inactivity_timeout removed from top-level config (deprecation → removed)

The old top-level key is gone in 0.29. Replace with the new nested path:

# OLD (remove this):
ephemeral_node_inactivity_timeout: 30m

# NEW (replace with):
node:
  ephemeral:
    inactivity_timeout: 30m

0.29 accepts the old key with a warning and uses the correct value, but the warning appears on every startup. Fix it to keep logs clean.

3. Strict version upgrade path enforced

0.29 blocks skipping minor versions. If upgrading from 0.27 or earlier, you MUST go 0.27 → 0.28 → 0.29 one step at a time. We were on 0.28, so this is not a blocker for this fleet.

4. ACL wildcard * behavior changed (review your policy)

In 0.29, bare * in ACL src/dst resolves to the CGNAT+ULA range (tailnet nodes only) instead of all IPs. If you had "dst": ["*:*"] meaning "any internet IP," you must switch to autogroup:danger-all as a source or explicit CIDRs. The IdahoMesh ACL does not use bare * — no action needed for this tailnet.

5. Minimum Tailscale client version raised to v1.80.0

Any tailscale client older than v1.80.0 will be rejected by 0.29. Verify all registered nodes are running a sufficiently recent tailscale before upgrading.

Other notable 0.29 changes (no action required, FYI)

• GivenName collision suffix changed from random hash (laptop-abc12xyz) to monotonic numeric (laptop, laptop-1). MagicDNS names for nodes with old random suffixes will change on upgrade. The raw Hostname column is unchanged.
• oidc.expiry removed; use node.expiry for all registration methods.
• headscale nodes register deprecated in favour of headscale auth register --auth-id.
• --namespace flag removed from nodes commands (was already replaced by --user).
• New: trusted_proxies config option for True-Client-IP headers (previously honoured from any client).
• DB migration is automatic on first start — no manual migration command needed.

Rollback procedure (if upgrade fails)

1. Stop the service: systemctl stop headscale
2. Reinstall the 0.28.0 binary: cp /usr/local/bin/headscale.bak-v0.28.0-pre029 /usr/local/bin/headscale
3. Restore config: cp /etc/headscale/config.yaml.bak-pre029-20260621 /etc/headscale/config.yaml
4. Restore DB: cp /root/headscale-db-pre029-20260621.sqlite /var/lib/headscale/db.sqlite
5. Start: systemctl start headscale

Note: 0.29 blocks downgrading once the DB migration has run. The DB backup from step 4 predates the migration, so this rollback is clean.

CT 106 (meshtastic-hs) upgrade record

• Date: 2026-06-21
• From: v0.28.0 → To: v0.29.1
• deployment: native systemd binary at /usr/local/bin/headscale
• Config changes made:
  • Removed randomize_client_port: false
  • Replaced ephemeral_node_inactivity_timeout: 30m with node.ephemeral.inactivity_timeout: 30m
• DB migration: automatic, clean, no errors in logs
• Nodes after upgrade: all 3 registered (burley-butte offline, mesh-bridge online, mt-isr offline) — same as before
• Backups (pre-upgrade): /etc/headscale/config.yaml.bak-pre029-20260621, /root/headscale-db-pre029-20260621.sqlite, /usr/local/bin/headscale.bak-v0.28.0-pre029

CT 107 (mesh-bridge / Echo6 Headscale) upgrade checklist

CT 107 runs Docker (not native binary). Steps will differ:

1. Back up: docker cp headscale-vanilla:/var/lib/headscale/db.sqlite /root/headscale-db-pre029-$(date +%Y%m%d).sqlite and cp /path/to/config.yaml config.yaml.bak-pre029
2. Apply the two config changes above (remove randomize_client_port, migrate ephemeral timeout key)
3. Change the image tag in docker-compose.yml from headscale/headscale:0.28.x to headscale/headscale:0.29.1
4. docker compose up -d
5. Validate: docker exec headscale-vanilla headscale version, check docker logs headscale-vanilla for migration success, docker exec headscale-vanilla headscale nodes list

---

Last updated: 2026-06-21