echo6-docs/projects/meshtastic-headscale-runbook.md

# IdahoMesh Tailnet Runbook

## Overview

Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.

### Architecture

```
Echo6 Headscale (100.64.0.x)
       ↓ (one-way only)
  [Bridge LXC]  ← dual tailscaled, NAT + firewall
       ↓
IdahoMesh Headscale (100.100.0.x)
       ↕                ↕
  Nebra CM3s      Sidpatchy's devices
```

> **Security:** The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.

### IP Allocation

| Tailnet     | Prefix           | Notes                                    |
|-------------|------------------|------------------------------------------|
| Echo6       | 100.64.0.0/10    | Existing, do not change                  |
| IdahoMesh   | 100.100.0.0/16   | Within Tailscale's required 100.64.0.0/10 supernet |

### Infrastructure

| Component | VMID | Host | Local IP | Purpose |
|-----------|------|------|----------|---------|
| meshtastic-hs | CT 106 | utility | 192.168.1.106 | IdahoMesh Headscale server |
| mesh-bridge | CT 107 | utility | 192.168.1.107 | One-way bridge between tailnets |

---

## Phase 1: IdahoMesh Headscale Instance

### 1.1 Create the LXC on utility

```bash
ssh root@192.168.1.241

pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname meshtastic-hs \
  --memory 512 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:4 \
  --unprivileged 1 \
  --onboot 1 \
  --start 1
```

Bootstrap standard packages:

```bash
echo6-bootstrap-ct.sh 106
```

### 1.2 Install Headscale

```bash
pct exec 106 -- bash -c '
apt update && apt install -y curl

HEADSCALE_VERSION="0.28.0"
curl -Lo /usr/local/bin/headscale \
  "https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
chmod +x /usr/local/bin/headscale

mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
'
```

### 1.3 Configure Headscale

Create `/etc/headscale/config.yaml`:

```yaml
server_url: https://vpn.idahomesh.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v4: 100.100.0.0/16
  v6: fd7a:115c:a1e0:ab00::/56
  allocation: sequential

derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  paths: []
  auto_update_enabled: true
  update_frequency: 3h

disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m

database:
  type: sqlite
  debug: false
  gorm:
    prepare_stmt: true
    parameterized_queries: true
    skip_err_record_not_found: true
    slow_threshold: 1000
  sqlite:
    path: /var/lib/headscale/db.sqlite
    write_ahead_log: true
    wal_autocheckpoint: 1000

policy:
  mode: file
  path: /etc/headscale/acl.json

dns:
  magic_dns: true
  base_domain: mesh.local
  override_local_dns: true
  nameservers:
    global:
      - 1.1.1.1
      - 9.9.9.9
    split: {}
  search_domains: []
  extra_records: []

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

logtail:
  enabled: false

randomize_client_port: false

log:
  level: info
  format: text
```

> **Note:** Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind Caddy, so TLS termination happens at the reverse proxy.

### 1.4 Create the ACL Policy

Create `/etc/headscale/acl.json`:

```json
{
  "groups": {
    "group:malice":     ["malice@"],
    "group:sidpatchy":  ["sidpatchy@"],
    "group:nebra":      ["nebra@"]
  },

  "acls": [
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:nebra:*"],
      "comment": "Nebra gateways talk to each other"
    },
    {
      "action": "accept",
      "src": ["group:malice"],
      "dst": ["group:nebra:*"],
      "comment": "Echo6 bridge can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:sidpatchy"],
      "dst": ["group:nebra:*"],
      "comment": "Sidpatchy can reach Nebras"
    },
    {
      "action": "accept",
      "src": ["group:nebra"],
      "dst": ["group:malice:*", "group:sidpatchy:*"],
      "comment": "Nebras can respond back to both"
    }
  ]
}
```

> **Important:** Headscale v0.28.0 requires usernames in ACL groups to have `@` suffix (e.g., `malice@`). The `--user` flag on CLI commands takes user IDs (integers), not names.
>
> **No malice↔Sidpatchy rules.** They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).

### 1.5 Create systemd Service

Create `/etc/systemd/system/headscale.service`:

```ini
[Unit]
Description=Headscale - IdahoMesh Tailnet
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl enable --now headscale
systemctl status headscale
```

### 1.6 Create Users and Preauthkeys

```bash
headscale users create echo6
headscale users create sidpatchy
headscale users create nebra

# For the bridge LXC (your side)
headscale preauthkeys create --user echo6 --expiration 24h
# Save this key ^^^

# For Sidpatchy — send this to him
headscale preauthkeys create --user sidpatchy --expiration 72h
# Save this key ^^^

# For Nebra CM3 gateways (reusable so all Nebras use same key)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# Save this key ^^^
```

---

## Phase 2: Bridge LXC (CT 107 on utility)

This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows **one-way only**: Echo6 → IdahoMesh.

### 2.1 Create the LXC

```bash
ssh root@192.168.1.241

pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname mesh-bridge \
  --memory 256 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
  --storage local-lvm \
  --rootfs local-lvm:2 \
  --unprivileged 1 \
  --features nesting=1 \
  --onboot 1 \
  --start 1
```

Add TUN device access for Tailscale (on the Proxmox host):

```bash
# Stop the container first
pct stop 107

cat >> /etc/pve/lxc/107.conf << 'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF

pct start 107
```

### 2.2 Install Tailscale

```bash
pct exec 107 -- bash -c '
curl -fsSL https://tailscale.com/install.sh | sh
'
```

### 2.3 Set Up Dual tailscaled

Create directories for the second instance:

```bash
pct exec 107 -- bash -c '
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
'
```

The default tailscaled service handles Echo6. Create a second service for IdahoMesh:

Create `/etc/systemd/system/tailscaled-meshtastic.service`:

```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/sbin/tailscaled \
  --state=/var/lib/tailscale-meshtastic/tailscaled.state \
  --socket=/var/run/tailscale-meshtastic/tailscaled.sock \
  --port=41642 \
  --tun=tailscale1
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```

### 2.4 Enable IP Forwarding

```bash
cat <<EOF > /etc/sysctl.d/99-bridge.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```

### 2.5 Join Both Tailnets

```bash
# Join Echo6 (default tailscaled instance)
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
tailscale up \
  --login-server=https://vpn.echo6.co \
  --advertise-routes=100.100.0.0/16 \
  --accept-routes

# Join IdahoMesh (second instance)
# Do NOT advertise Echo6 routes — one-way only
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<echo6-preauthkey-from-step-1.6> \
  --accept-routes
```

After joining, approve the advertised route on Echo6 Headscale only:

```bash
# On Echo6 Headscale (Contabo) — enable the 100.100.0.0/16 route
docker exec headscale-vanilla headscale routes list
docker exec headscale-vanilla headscale routes enable -r <route-id>

# NO route approval needed on IdahoMesh Headscale — nothing is advertised
```

### 2.6 Configure One-Way Firewall and NAT

This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.

Install iptables:

```bash
apt install -y iptables iptables-persistent
```

Apply rules:

```bash
# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE

# Allow Echo6 → IdahoMesh (outbound)
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT

# Allow established/related return traffic only (responses to Echo6-initiated connections)
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT

# DROP all new connections from IdahoMesh → Echo6
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP
```

Persist across reboots:

```bash
netfilter-persistent save
```

Verify:

```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```

> **What this achieves:**
> - Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
> - Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
> - Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
> - Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)

---

## Phase 3: Expose vpn.idahomesh.com

### 3.1 Issue SSL Certificate

```bash
ssh root@192.168.1.241

pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
'
```

### 3.2 Install Certificate

```bash
pct exec 101 -- bash -c '
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
  --cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
  --key-file /etc/caddy/certs/vpn.idahomesh.com.key \
  --fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
  --reloadcmd "systemctl reload caddy"

chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
'
```

### 3.3 Add Caddy Site Block

```bash
pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''

vpn.idahomesh.com {
    tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
    reverse_proxy 192.168.1.106:8080
}
EOF
systemctl reload caddy'
```

### 3.4 Add GoDaddy DNS Record

```bash
# On cortex/TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163
```

### 3.5 Verify

```bash
dig +short vpn.idahomesh.com
# Should return 199.6.36.163

curl -I https://vpn.idahomesh.com
```

---

## Phase 4: Register Nebra CM3 Gateways

Only Burley Butte for now. See `idahomesh-vpn-device-setup.md` for the full device onboarding runbook.

```bash
# SSH to Burley Butte
curl -fsSL https://tailscale.com/install.sh | sh

tailscale up \
  --login-server=https://vpn.idahomesh.com \
  --authkey=<nebra-preauthkey-from-step-1.6> \
  --hostname=burley-butte
```

Verify on the IdahoMesh Headscale:

```bash
# On CT 106
headscale nodes list
```

---

## Phase 5: Sidpatchy Onboarding

Send Sidpatchy the following:

1. **IdahoMesh VPN URL:** `https://vpn.idahomesh.com`
2. **Preauthkey:** (the one generated in Step 1.6 for sidpatchy)
3. **Device setup runbook:** `idahomesh-vpn-device-setup.md`

---

## Phase 6: Verification

### From the bridge LXC (CT 107)

```bash
# Ping Burley Butte via IdahoMesh tailnet
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte

# Check status on both tailnets
tailscale status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```

### From any Echo6 machine (via bridge routes)

```bash
# Should be routable through the bridge (NAT'd)
ping 100.100.0.x  # Burley Butte's IdahoMesh IP
```

### Verify isolation — from IdahoMesh side

```bash
# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
ping 100.64.0.14  # cortex — should timeout/unreachable
```

---

## Quick Reference

| Component      | Location         | Tailnet    | IP              |
|----------------|------------------|------------|-----------------|
| IdahoMesh HS   | CT 106, utility  | IdahoMesh  | 192.168.1.106   |
| Bridge LXC     | CT 107, utility  | Both       | 192.168.1.107   |
| Burley Butte   | Field site       | IdahoMesh  | 100.100.0.x     |
| Sidpatchy      | Remote           | IdahoMesh  | 100.100.0.x     |

---

## Maintenance Notes

- **Preauthkeys expire.** Generate long-lived reusable keys for Nebras, short-lived for humans.
- **Headscale updates:** Check releases at https://github.com/juanfont/headscale/releases
- **ACL changes:** Edit `/etc/headscale/acl.json` on CT 106, then `systemctl reload headscale`
- **Firewall rules:** Persisted via `netfilter-persistent` on CT 107. Verify after reboot with `iptables -L FORWARD -v -n`
- **If a Nebra goes offline:** Check `headscale nodes list` — may need a new key if expired.
- **Sidpatchy wants off?** `headscale nodes delete -i <node-id>` and revoke the preauthkey.
- **Device setup instructions:** See `idahomesh-vpn-device-setup.md`

---

*Last updated: 2026-02-11*