matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

Browse source 
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup)
- Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing
- Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack
- Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure
- Removes 4 deprecated runbook duplicates (canonical versions live in projects/)
- Adds .gitignore for binary archives and editor temp files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-04-13 06:02:16 +00:00
commit e9231ac24a
93 changed files with 51223 additions and 254 deletions
Download patch file Download diff file Expand all files Collapse all files

337
runbooks/idahomesh-bridge-setup.md Normal file
View file

@ -0,0 +1,337 @@
# IdahoMesh Bridge Setup
Build a one-way bridge between your tailnet and the IdahoMesh Meshtastic network. This lets your devices reach Nebra gateways through IdahoMesh, while preventing IdahoMesh from reaching back into your network.
---
## Architecture
```
Your Tailnet (your Headscale/Tailscale)
↓ (one-way only)
[Bridge Machine] ← dual tailscaled, NAT + firewall
↓
IdahoMesh Tailnet (100.100.0.0/16)
↕
Nebra CM3 Gateways (Meshtastic nodes)
```
**Security model:** Your devices can reach Nebras. Nothing on IdahoMesh can initiate connections back into your network. NAT masquerades your source IPs so IdahoMesh only sees the bridge's IP.
---
## Prerequisites
| Item | Value |
|------|-------|
| IdahoMesh Headscale URL | `https://vpn.idahomesh.com` |
| IdahoMesh prefix | `100.100.0.0/16` |
| Your preauthkey | Provided by IdahoMesh admin |
You also need:
1. A Linux machine on your network (VM, Pi, bare metal — anything running systemd)
2. Root access on that machine
3. Internet access (to reach vpn.idahomesh.com)
4. Your own tailnet's connection details (Headscale URL, or stock Tailscale if using tailscale.com)
---
## Step 1: Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
```
This installs both `tailscale` and `tailscaled`. The default service (`tailscaled.service`) will handle your primary tailnet.
---
## Step 2: Set Up Dual tailscaled
You need two Tailscale daemon instances — one for your tailnet, one for IdahoMesh. The default `tailscaled` service handles your tailnet. Create a second service for IdahoMesh.
### Create directories
```bash
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
```
### Create the second tailscaled service
Write `/etc/systemd/system/tailscaled-meshtastic.service`:
```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/sbin/tailscaled \
--state=/var/lib/tailscale-meshtastic/tailscaled.state \
--socket=/var/run/tailscale-meshtastic/tailscaled.sock \
--port=41642 \
--tun=tailscale1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
> **Critical:** The `--tun=tailscale1` flag is required. Both instances cannot use the default `tailscale0` TUN device — the second one will fail with "TUN device tailscale0 is busy" if you omit this.
Enable and start it:
```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```
---
## Step 3: Enable IP Forwarding
```bash
cat > /etc/sysctl.d/99-bridge.conf << 'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```
---
## Step 4: Join Both Tailnets
### Join your tailnet (default tailscaled)
Advertise the IdahoMesh range so your other devices can route to Meshtastic nodes through this bridge:
```bash
# If you use your own Headscale:
tailscale up \
--login-server=https://YOUR_HEADSCALE_URL \
--advertise-routes=100.100.0.0/16 \
--accept-routes
# If you use stock Tailscale (tailscale.com):
tailscale up \
--advertise-routes=100.100.0.0/16 \
--accept-routes
```
After joining, you need to **approve the advertised route** on your tailnet's admin:
- **Headscale:** `headscale routes list` then `headscale routes enable -r <route-id>`
- **Stock Tailscale:** Go to admin console → Machines → your bridge → approve the `100.100.0.0/16` subnet route
### Join IdahoMesh (second tailscaled)
```bash
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
--login-server=https://vpn.idahomesh.com \
--authkey=YOUR_IDAHOMESH_PREAUTHKEY \
--accept-routes
```
> **Important:** Do NOT advertise your tailnet's routes on IdahoMesh. The bridge is one-way — IdahoMesh should have no route back into your network.
---
## Step 5: Configure One-Way Firewall and NAT
This is the critical security step. Your tailnet can reach IdahoMesh, but nothing on IdahoMesh can reach back into your network.
Replace `YOUR_TAILNET_PREFIX` below with your tailnet's IP range. Common values:
| Tailnet type | Prefix |
|-------------|--------|
| Stock Tailscale | `100.64.0.0/10` |
| Custom Headscale | Check your `config.yaml` → `prefixes.v4` |
### Apply iptables rules
```bash
# NAT: Masquerade your source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not your real tailnet IPs
iptables -t nat -A POSTROUTING -s YOUR_TAILNET_PREFIX -d 100.100.0.0/16 -j MASQUERADE
# Allow your tailnet → IdahoMesh (outbound)
iptables -A FORWARD -s YOUR_TAILNET_PREFIX -d 100.100.0.0/16 -j ACCEPT
# Allow established/related return traffic only (responses to connections you initiated)
iptables -A FORWARD -s 100.100.0.0/16 -d YOUR_TAILNET_PREFIX -m state --state ESTABLISHED,RELATED -j ACCEPT
# DROP all new connections from IdahoMesh → your tailnet
iptables -A FORWARD -s 100.100.0.0/16 -d YOUR_TAILNET_PREFIX -j DROP
```
### Persist rules across reboots
Do **not** use `iptables-persistent` — it hangs on install even with noninteractive mode. Use manual persistence instead:
```bash
# Save current rules
mkdir -p /etc/iptables
iptables-save > /etc/iptables/rules.v4
```
Create `/etc/systemd/system/iptables-restore.service`:
```ini
[Unit]
Description=Restore iptables rules
Before=network-pre.target
Wants=network-pre.target
[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables/rules.v4
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
```
Enable it:
```bash
systemctl daemon-reload
systemctl enable iptables-restore
```
### Verify rules
```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```
You should see:
- MASQUERADE rule on POSTROUTING
- ACCEPT for your prefix → 100.100.0.0/16
- ACCEPT ESTABLISHED,RELATED for 100.100.0.0/16 → your prefix
- DROP for 100.100.0.0/16 → your prefix
---
## Step 6: Enable Routes on Your Other Devices
Any device on your tailnet that wants to reach IdahoMesh Nebras through the bridge needs to accept subnet routes:
```bash
# On each device that needs access
tailscale set --accept-routes
```
Without this, traffic to `100.100.0.x` goes to the default gateway instead of through the Tailscale tunnel.
---
## Step 7: Verify
### Check both tailscaled instances
```bash
# Your tailnet
tailscale status
# IdahoMesh
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```
Both should show "online" with peers listed.
### Ping a Nebra gateway
```bash
# From the bridge itself, via IdahoMesh socket
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte
```
### Ping from another device on your tailnet
```bash
# From any device on your tailnet (with --accept-routes enabled)
# Use the Nebra's IdahoMesh IP
ping 100.100.0.3 # Burley Butte
```
### Verify isolation
From an IdahoMesh device or ask the admin to test — pinging your tailnet IPs from IdahoMesh should time out / be unreachable.
---
## Troubleshooting
### Second tailscaled won't start
- Check `journalctl -u tailscaled-meshtastic -f`
- Most common: forgot `--tun=tailscale1` — both instances fighting over `tailscale0`
- Verify the port isn't in conflict: default uses 41641, second uses 41642
### Can't reach Nebras from other devices on your tailnet
- Verify the bridge advertises `100.100.0.0/16`: `tailscale status` should show it as a subnet router
- Approve the route on your tailnet admin (Headscale or Tailscale admin console)
- Enable `--accept-routes` on the client device trying to reach Nebras
### IdahoMesh preauthkey expired
- Contact the IdahoMesh admin for a new key
- Re-join: `tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up --login-server=https://vpn.idahomesh.com --authkey=NEW_KEY --accept-routes`
### After reboot, only one tailscaled reconnects
- Check both services: `systemctl status tailscaled` and `systemctl status tailscaled-meshtastic`
- Verify iptables rules survived: `iptables -L FORWARD -v -n`
- If the second instance lost state, re-join IdahoMesh with a new preauthkey
### Default tailscaled pointed at wrong server after force-reauth
- If you run `tailscale up --force-reauth` without specifying `--login-server`, it may reconnect to the wrong Headscale
- Always specify `--login-server` explicitly when re-authing:
- Default instance: `tailscale up --login-server=https://YOUR_HEADSCALE_URL --force-reauth`
- IdahoMesh instance: `tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up --login-server=https://vpn.idahomesh.com --force-reauth`
---
## Quick Reference
| Item | Value |
|------|-------|
| IdahoMesh URL | `https://vpn.idahomesh.com` |
| IdahoMesh prefix | `100.100.0.0/16` |
| IdahoMesh socket | `/var/run/tailscale-meshtastic/tailscaled.sock` |
| IdahoMesh TUN device | `tailscale1` |
| IdahoMesh port | `41642` |
| Default Tailscale socket | `/var/run/tailscale/tailscaled.sock` |
| Default TUN device | `tailscale0` |
| Default port | `41641` |
### Useful commands
```bash
# IdahoMesh status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
# IdahoMesh ping
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping <hostname>
# Check firewall rules
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
# Restart services
systemctl restart tailscaled # Your tailnet
systemctl restart tailscaled-meshtastic # IdahoMesh
```
---
*Last updated: 2026-02-11*