Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
89834796ff
commit
e9231ac24a
93 changed files with 51223 additions and 254 deletions
561
projects/meshtastic-headscale-runbook.md
Normal file
561
projects/meshtastic-headscale-runbook.md
Normal file
|
|
@ -0,0 +1,561 @@
|
|||
# IdahoMesh Tailnet Runbook
|
||||
|
||||
## Overview
|
||||
|
||||
Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Echo6 Headscale (100.64.0.x)
|
||||
↓ (one-way only)
|
||||
[Bridge LXC] ← dual tailscaled, NAT + firewall
|
||||
↓
|
||||
IdahoMesh Headscale (100.100.0.x)
|
||||
↕ ↕
|
||||
Nebra CM3s Sidpatchy's devices
|
||||
```
|
||||
|
||||
> **Security:** The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.
|
||||
|
||||
### IP Allocation
|
||||
|
||||
| Tailnet | Prefix | Notes |
|
||||
|-------------|------------------|------------------------------------------|
|
||||
| Echo6 | 100.64.0.0/10 | Existing, do not change |
|
||||
| IdahoMesh | 100.100.0.0/16 | Within Tailscale's required 100.64.0.0/10 supernet |
|
||||
|
||||
### Infrastructure
|
||||
|
||||
| Component | VMID | Host | Local IP | Purpose |
|
||||
|-----------|------|------|----------|---------|
|
||||
| meshtastic-hs | CT 106 | utility | 192.168.1.106 | IdahoMesh Headscale server |
|
||||
| mesh-bridge | CT 107 | utility | 192.168.1.107 | One-way bridge between tailnets |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: IdahoMesh Headscale Instance
|
||||
|
||||
### 1.1 Create the LXC on utility
|
||||
|
||||
```bash
|
||||
ssh root@192.168.1.241
|
||||
|
||||
pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
|
||||
--hostname meshtastic-hs \
|
||||
--memory 512 \
|
||||
--cores 1 \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
|
||||
--storage local-lvm \
|
||||
--rootfs local-lvm:4 \
|
||||
--unprivileged 1 \
|
||||
--onboot 1 \
|
||||
--start 1
|
||||
```
|
||||
|
||||
Bootstrap standard packages:
|
||||
|
||||
```bash
|
||||
echo6-bootstrap-ct.sh 106
|
||||
```
|
||||
|
||||
### 1.2 Install Headscale
|
||||
|
||||
```bash
|
||||
pct exec 106 -- bash -c '
|
||||
apt update && apt install -y curl
|
||||
|
||||
HEADSCALE_VERSION="0.28.0"
|
||||
curl -Lo /usr/local/bin/headscale \
|
||||
"https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
|
||||
chmod +x /usr/local/bin/headscale
|
||||
|
||||
mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
|
||||
'
|
||||
```
|
||||
|
||||
### 1.3 Configure Headscale
|
||||
|
||||
Create `/etc/headscale/config.yaml`:
|
||||
|
||||
```yaml
|
||||
server_url: https://vpn.idahomesh.com
|
||||
listen_addr: 0.0.0.0:8080
|
||||
metrics_listen_addr: 127.0.0.1:9090
|
||||
grpc_listen_addr: 127.0.0.1:50443
|
||||
grpc_allow_insecure: false
|
||||
|
||||
noise:
|
||||
private_key_path: /var/lib/headscale/noise_private.key
|
||||
|
||||
prefixes:
|
||||
v4: 100.100.0.0/16
|
||||
v6: fd7a:115c:a1e0:ab00::/56
|
||||
allocation: sequential
|
||||
|
||||
derp:
|
||||
server:
|
||||
enabled: false
|
||||
urls:
|
||||
- https://controlplane.tailscale.com/derpmap/default
|
||||
paths: []
|
||||
auto_update_enabled: true
|
||||
update_frequency: 3h
|
||||
|
||||
disable_check_updates: false
|
||||
ephemeral_node_inactivity_timeout: 30m
|
||||
|
||||
database:
|
||||
type: sqlite
|
||||
debug: false
|
||||
gorm:
|
||||
prepare_stmt: true
|
||||
parameterized_queries: true
|
||||
skip_err_record_not_found: true
|
||||
slow_threshold: 1000
|
||||
sqlite:
|
||||
path: /var/lib/headscale/db.sqlite
|
||||
write_ahead_log: true
|
||||
wal_autocheckpoint: 1000
|
||||
|
||||
policy:
|
||||
mode: file
|
||||
path: /etc/headscale/acl.json
|
||||
|
||||
dns:
|
||||
magic_dns: true
|
||||
base_domain: mesh.local
|
||||
override_local_dns: true
|
||||
nameservers:
|
||||
global:
|
||||
- 1.1.1.1
|
||||
- 9.9.9.9
|
||||
split: {}
|
||||
search_domains: []
|
||||
extra_records: []
|
||||
|
||||
unix_socket: /var/run/headscale/headscale.sock
|
||||
unix_socket_permission: "0770"
|
||||
|
||||
logtail:
|
||||
enabled: false
|
||||
|
||||
randomize_client_port: false
|
||||
|
||||
log:
|
||||
level: info
|
||||
format: text
|
||||
```
|
||||
|
||||
> **Note:** Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind Caddy, so TLS termination happens at the reverse proxy.
|
||||
|
||||
### 1.4 Create the ACL Policy
|
||||
|
||||
Create `/etc/headscale/acl.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"groups": {
|
||||
"group:malice": ["malice@"],
|
||||
"group:sidpatchy": ["sidpatchy@"],
|
||||
"group:nebra": ["nebra@"]
|
||||
},
|
||||
|
||||
"acls": [
|
||||
{
|
||||
"action": "accept",
|
||||
"src": ["group:nebra"],
|
||||
"dst": ["group:nebra:*"],
|
||||
"comment": "Nebra gateways talk to each other"
|
||||
},
|
||||
{
|
||||
"action": "accept",
|
||||
"src": ["group:malice"],
|
||||
"dst": ["group:nebra:*"],
|
||||
"comment": "Echo6 bridge can reach Nebras"
|
||||
},
|
||||
{
|
||||
"action": "accept",
|
||||
"src": ["group:sidpatchy"],
|
||||
"dst": ["group:nebra:*"],
|
||||
"comment": "Sidpatchy can reach Nebras"
|
||||
},
|
||||
{
|
||||
"action": "accept",
|
||||
"src": ["group:nebra"],
|
||||
"dst": ["group:malice:*", "group:sidpatchy:*"],
|
||||
"comment": "Nebras can respond back to both"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
> **Important:** Headscale v0.28.0 requires usernames in ACL groups to have `@` suffix (e.g., `malice@`). The `--user` flag on CLI commands takes user IDs (integers), not names.
|
||||
>
|
||||
> **No malice↔Sidpatchy rules.** They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).
|
||||
|
||||
### 1.5 Create systemd Service
|
||||
|
||||
Create `/etc/systemd/system/headscale.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Headscale - IdahoMesh Tailnet
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/headscale serve
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now headscale
|
||||
systemctl status headscale
|
||||
```
|
||||
|
||||
### 1.6 Create Users and Preauthkeys
|
||||
|
||||
```bash
|
||||
headscale users create echo6
|
||||
headscale users create sidpatchy
|
||||
headscale users create nebra
|
||||
|
||||
# For the bridge LXC (your side)
|
||||
headscale preauthkeys create --user echo6 --expiration 24h
|
||||
# Save this key ^^^
|
||||
|
||||
# For Sidpatchy — send this to him
|
||||
headscale preauthkeys create --user sidpatchy --expiration 72h
|
||||
# Save this key ^^^
|
||||
|
||||
# For Nebra CM3 gateways (reusable so all Nebras use same key)
|
||||
headscale preauthkeys create --user nebra --reusable --expiration 8760h
|
||||
# Save this key ^^^
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Bridge LXC (CT 107 on utility)
|
||||
|
||||
This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows **one-way only**: Echo6 → IdahoMesh.
|
||||
|
||||
### 2.1 Create the LXC
|
||||
|
||||
```bash
|
||||
ssh root@192.168.1.241
|
||||
|
||||
pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
|
||||
--hostname mesh-bridge \
|
||||
--memory 256 \
|
||||
--cores 1 \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
|
||||
--storage local-lvm \
|
||||
--rootfs local-lvm:2 \
|
||||
--unprivileged 1 \
|
||||
--features nesting=1 \
|
||||
--onboot 1 \
|
||||
--start 1
|
||||
```
|
||||
|
||||
Add TUN device access for Tailscale (on the Proxmox host):
|
||||
|
||||
```bash
|
||||
# Stop the container first
|
||||
pct stop 107
|
||||
|
||||
cat >> /etc/pve/lxc/107.conf << 'EOF'
|
||||
lxc.cgroup2.devices.allow: c 10:200 rwm
|
||||
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
|
||||
EOF
|
||||
|
||||
pct start 107
|
||||
```
|
||||
|
||||
### 2.2 Install Tailscale
|
||||
|
||||
```bash
|
||||
pct exec 107 -- bash -c '
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
'
|
||||
```
|
||||
|
||||
### 2.3 Set Up Dual tailscaled
|
||||
|
||||
Create directories for the second instance:
|
||||
|
||||
```bash
|
||||
pct exec 107 -- bash -c '
|
||||
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
|
||||
'
|
||||
```
|
||||
|
||||
The default tailscaled service handles Echo6. Create a second service for IdahoMesh:
|
||||
|
||||
Create `/etc/systemd/system/tailscaled-meshtastic.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Tailscale daemon (IdahoMesh tailnet)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/sbin/tailscaled \
|
||||
--state=/var/lib/tailscale-meshtastic/tailscaled.state \
|
||||
--socket=/var/run/tailscale-meshtastic/tailscaled.sock \
|
||||
--port=41642 \
|
||||
--tun=tailscale1
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now tailscaled-meshtastic
|
||||
```
|
||||
|
||||
### 2.4 Enable IP Forwarding
|
||||
|
||||
```bash
|
||||
cat <<EOF > /etc/sysctl.d/99-bridge.conf
|
||||
net.ipv4.ip_forward = 1
|
||||
net.ipv6.conf.all.forwarding = 1
|
||||
EOF
|
||||
sysctl -p /etc/sysctl.d/99-bridge.conf
|
||||
```
|
||||
|
||||
### 2.5 Join Both Tailnets
|
||||
|
||||
```bash
|
||||
# Join Echo6 (default tailscaled instance)
|
||||
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
|
||||
tailscale up \
|
||||
--login-server=https://vpn.echo6.co \
|
||||
--advertise-routes=100.100.0.0/16 \
|
||||
--accept-routes
|
||||
|
||||
# Join IdahoMesh (second instance)
|
||||
# Do NOT advertise Echo6 routes — one-way only
|
||||
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
|
||||
--login-server=https://vpn.idahomesh.com \
|
||||
--authkey=<echo6-preauthkey-from-step-1.6> \
|
||||
--accept-routes
|
||||
```
|
||||
|
||||
After joining, approve the advertised route on Echo6 Headscale only:
|
||||
|
||||
```bash
|
||||
# On Echo6 Headscale (Contabo) — enable the 100.100.0.0/16 route
|
||||
docker exec headscale-vanilla headscale routes list
|
||||
docker exec headscale-vanilla headscale routes enable -r <route-id>
|
||||
|
||||
# NO route approval needed on IdahoMesh Headscale — nothing is advertised
|
||||
```
|
||||
|
||||
### 2.6 Configure One-Way Firewall and NAT
|
||||
|
||||
This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.
|
||||
|
||||
Install iptables:
|
||||
|
||||
```bash
|
||||
apt install -y iptables iptables-persistent
|
||||
```
|
||||
|
||||
Apply rules:
|
||||
|
||||
```bash
|
||||
# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
|
||||
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
|
||||
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE
|
||||
|
||||
# Allow Echo6 → IdahoMesh (outbound)
|
||||
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT
|
||||
|
||||
# Allow established/related return traffic only (responses to Echo6-initiated connections)
|
||||
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT
|
||||
|
||||
# DROP all new connections from IdahoMesh → Echo6
|
||||
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP
|
||||
```
|
||||
|
||||
Persist across reboots:
|
||||
|
||||
```bash
|
||||
netfilter-persistent save
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
iptables -L FORWARD -v -n
|
||||
iptables -t nat -L POSTROUTING -v -n
|
||||
```
|
||||
|
||||
> **What this achieves:**
|
||||
> - Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
|
||||
> - Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
|
||||
> - Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
|
||||
> - Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Expose vpn.idahomesh.com
|
||||
|
||||
### 3.1 Issue SSL Certificate
|
||||
|
||||
```bash
|
||||
ssh root@192.168.1.241
|
||||
|
||||
pct exec 101 -- bash -c '
|
||||
export GD_Key="<from .ref/credentials>"
|
||||
export GD_Secret="<from .ref/credentials>"
|
||||
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
|
||||
'
|
||||
```
|
||||
|
||||
### 3.2 Install Certificate
|
||||
|
||||
```bash
|
||||
pct exec 101 -- bash -c '
|
||||
mkdir -p /etc/caddy/certs
|
||||
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
|
||||
--cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
|
||||
--key-file /etc/caddy/certs/vpn.idahomesh.com.key \
|
||||
--fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
|
||||
--reloadcmd "systemctl reload caddy"
|
||||
|
||||
chown -R caddy:caddy /etc/caddy/certs
|
||||
chmod 600 /etc/caddy/certs/*.key
|
||||
chmod 644 /etc/caddy/certs/*.crt
|
||||
'
|
||||
```
|
||||
|
||||
### 3.3 Add Caddy Site Block
|
||||
|
||||
```bash
|
||||
pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''
|
||||
|
||||
vpn.idahomesh.com {
|
||||
tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
|
||||
reverse_proxy 192.168.1.106:8080
|
||||
}
|
||||
EOF
|
||||
systemctl reload caddy'
|
||||
```
|
||||
|
||||
### 3.4 Add GoDaddy DNS Record
|
||||
|
||||
```bash
|
||||
# On cortex/TOC
|
||||
source /home/zvx/projects/.ref/credentials
|
||||
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163
|
||||
```
|
||||
|
||||
### 3.5 Verify
|
||||
|
||||
```bash
|
||||
dig +short vpn.idahomesh.com
|
||||
# Should return 199.6.36.163
|
||||
|
||||
curl -I https://vpn.idahomesh.com
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Register Nebra CM3 Gateways
|
||||
|
||||
Only Burley Butte for now. See `idahomesh-vpn-device-setup.md` for the full device onboarding runbook.
|
||||
|
||||
```bash
|
||||
# SSH to Burley Butte
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
|
||||
tailscale up \
|
||||
--login-server=https://vpn.idahomesh.com \
|
||||
--authkey=<nebra-preauthkey-from-step-1.6> \
|
||||
--hostname=burley-butte
|
||||
```
|
||||
|
||||
Verify on the IdahoMesh Headscale:
|
||||
|
||||
```bash
|
||||
# On CT 106
|
||||
headscale nodes list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Sidpatchy Onboarding
|
||||
|
||||
Send Sidpatchy the following:
|
||||
|
||||
1. **IdahoMesh VPN URL:** `https://vpn.idahomesh.com`
|
||||
2. **Preauthkey:** (the one generated in Step 1.6 for sidpatchy)
|
||||
3. **Device setup runbook:** `idahomesh-vpn-device-setup.md`
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Verification
|
||||
|
||||
### From the bridge LXC (CT 107)
|
||||
|
||||
```bash
|
||||
# Ping Burley Butte via IdahoMesh tailnet
|
||||
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte
|
||||
|
||||
# Check status on both tailnets
|
||||
tailscale status
|
||||
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
|
||||
```
|
||||
|
||||
### From any Echo6 machine (via bridge routes)
|
||||
|
||||
```bash
|
||||
# Should be routable through the bridge (NAT'd)
|
||||
ping 100.100.0.x # Burley Butte's IdahoMesh IP
|
||||
```
|
||||
|
||||
### Verify isolation — from IdahoMesh side
|
||||
|
||||
```bash
|
||||
# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
|
||||
ping 100.64.0.14 # cortex — should timeout/unreachable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Component | Location | Tailnet | IP |
|
||||
|----------------|------------------|------------|-----------------|
|
||||
| IdahoMesh HS | CT 106, utility | IdahoMesh | 192.168.1.106 |
|
||||
| Bridge LXC | CT 107, utility | Both | 192.168.1.107 |
|
||||
| Burley Butte | Field site | IdahoMesh | 100.100.0.x |
|
||||
| Sidpatchy | Remote | IdahoMesh | 100.100.0.x |
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Notes
|
||||
|
||||
- **Preauthkeys expire.** Generate long-lived reusable keys for Nebras, short-lived for humans.
|
||||
- **Headscale updates:** Check releases at https://github.com/juanfont/headscale/releases
|
||||
- **ACL changes:** Edit `/etc/headscale/acl.json` on CT 106, then `systemctl reload headscale`
|
||||
- **Firewall rules:** Persisted via `netfilter-persistent` on CT 107. Verify after reboot with `iptables -L FORWARD -v -n`
|
||||
- **If a Nebra goes offline:** Check `headscale nodes list` — may need a new key if expired.
|
||||
- **Sidpatchy wants off?** `headscale nodes delete -i <node-id>` and revoke the preauthkey.
|
||||
- **Device setup instructions:** See `idahomesh-vpn-device-setup.md`
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-02-11*
|
||||
Loading…
Add table
Add a link
Reference in a new issue