Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 06:02:16 +00:00 · 2026-04-13 06:02:16 +00:00 · e9231ac24a
commit e9231ac24a
parent 89834796ff
93 changed files with 51223 additions and 254 deletions
--- a/runbooks/nordvpn-lxc.md
+++ b/runbooks/nordvpn-lxc.md
@ -0,0 +1,343 @@
+# NordVPN / WireGuard in LXC
+
+Set up VPN with IP rotation inside an LXC container. Handles the LXC-specific gotchas: TUN device, systemd compatibility, split tunneling so local services stay reachable.
+
+---
+
+## Prerequisites
+
+- LXC container provisioned and running (see `ct-runbook.md`)
+- SSH access to both the Proxmox host and the container
+- NordVPN account with a service token (from https://my.nordaccount.com/dashboard/nordvpn/access-tokens/)
+
+---
+
+## Inputs
+
+Prompt the user for all of these before executing:
+
+```
+CTID=                   # Container ID on Proxmox host
+CT_HOST=                # SSH alias or IP for the container
+PVE_HOST=               # SSH alias or IP for the Proxmox host
+NORDVPN_TOKEN=          # NordVPN service token
+VPN_COUNTRIES=          # Comma-separated rotation list (e.g., "United_States,Canada,United_Kingdom,Germany,Netherlands,Sweden")
+VPN_CONFIG_DIR=         # Where to store WireGuard configs inside CT (e.g., /opt/vpn)
+```
+
+---
+
+## Step 1: Enable TUN Device on Container
+
+**Run on Proxmox host.** LXC containers don't have `/dev/net/tun` by default — VPN won't work without it.
+
+```bash
+ssh $PVE_HOST "grep -q 'dev/net/tun' /etc/pve/lxc/${CTID}.conf 2>/dev/null || {
+  echo 'lxc.cgroup2.devices.allow: c 10:200 rwm' >> /etc/pve/lxc/${CTID}.conf
+  echo 'lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file' >> /etc/pve/lxc/${CTID}.conf
+  echo 'TUN device added — container restart required'
+}"
+```
+
+If lines were added, restart the container:
+
+```bash
+ssh $PVE_HOST "pct reboot $CTID"
+```
+
+### Gate
+
+```bash
+ssh $CT_HOST 'ls -la /dev/net/tun'
+```
+
+Must show the device. If missing, the cgroup/mount entries didn't take — check `/etc/pve/lxc/${CTID}.conf`.
+
+---
+
+## Step 2: Try NordVPN CLI (Option A)
+
+The CLI is the simplest path but requires working systemd in the container (which most LXCs have, but some stripped-down templates don't).
+
+```bash
+ssh $CT_HOST 'sh <(curl -sSf https://downloads.nordcdn.com/apps/linux/install.sh)'
+```
+
+If the installer completes without errors:
+
+```bash
+ssh $CT_HOST "nordvpn login --token $NORDVPN_TOKEN"
+ssh $CT_HOST 'nordvpn set technology nordlynx'      # WireGuard-based, faster
+ssh $CT_HOST 'nordvpn set killswitch off'            # Don't kill local services
+ssh $CT_HOST 'nordvpn set autoconnect off'           # We control rotation
+ssh $CT_HOST 'nordvpn set dns off'                   # Keep container's DNS
+```
+
+### Test
+
+```bash
+ssh $CT_HOST 'nordvpn connect United_States && sleep 3 && curl -s https://ifconfig.me && nordvpn disconnect'
+```
+
+Must show a non-local IP. If it does, **skip to Step 4** (rotation script).
+
+### Common failures
+
+- **"Whoops! /run/nordvpn/nordvpnd.sock not found"** — nordvpnd service didn't start. Check `systemctl status nordvpnd`. If systemd is broken in this LXC, fall through to Option B.
+- **"Permission denied creating /dev/net/tun"** — Step 1 TUN device not configured. Go back.
+- **Installer hangs on "Starting NordVPN daemon"** — systemd issue. Kill it, fall through to Option B.
+
+---
+
+## Step 3: WireGuard Manual Configs (Option B — Fallback)
+
+Use this if NordVPN CLI doesn't work in the LXC.
+
+### Install WireGuard
+
+```bash
+ssh $CT_HOST 'apt install -y wireguard-tools curl jq'
+```
+
+### Generate NordVPN WireGuard configs
+
+NordVPN provides WireGuard configs via their API. Generate one per country:
+
+```bash
+ssh $CT_HOST "mkdir -p $VPN_CONFIG_DIR"
+
+# Get NordVPN WireGuard private key
+# Method: Use the NordVPN API with your token to get credentials
+# This requires the nordvpn CLI to extract the private key, OR manual setup:
+#
+# 1. Go to https://my.nordaccount.com/dashboard/nordvpn/manual-configuration/
+# 2. Generate WireGuard credentials
+# 3. Download configs for each country
+# 4. SCP them to the container
+
+# Place configs as: $VPN_CONFIG_DIR/us.conf, ca.conf, uk.conf, de.conf, nl.conf, se.conf
+```
+
+**⚠️ Manual step required:** NordVPN's WireGuard config generation requires either the CLI (which didn't work) or manual download from the NordVPN dashboard. Download `.conf` files for each country in the rotation list and SCP them to the container.
+
+### Config format
+
+Each `.conf` file should look like:
+
+```ini
+[Interface]
+PrivateKey = <your-wireguard-private-key>
+Address = 10.5.0.2/16
+DNS = 103.86.96.100
+
+[Peer]
+PublicKey = <server-public-key>
+AllowedIPs = 0.0.0.0/0
+Endpoint = <server-ip>:51820
+PersistentKeepalive = 25
+```
+
+**Critical for LXC:** If the container runs services that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
+
+```ini
+# Route everything EXCEPT local network through VPN
+AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
+# This covers all IPs but lets 192.168.x.x and 100.64.x.x traffic stay local
+```
+
+Or more precisely, exclude your subnets:
+
+```bash
+# Generate AllowedIPs that exclude local networks
+# This sends all traffic through VPN except 192.168.1.0/24 and 100.64.0.0/10
+AllowedIPs = 0.0.0.0/5, 8.0.0.0/7, 11.0.0.0/8, 12.0.0.0/6, 16.0.0.0/4, 32.0.0.0/3, 64.0.0.0/3, 96.0.0.0/6, 100.0.0.0/10, 100.128.0.0/9, 101.0.0.0/8, 102.0.0.0/7, 104.0.0.0/5, 112.0.0.0/4, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/8, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/9, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/6, 200.0.0.0/5, 208.0.0.0/4, 224.0.0.0/3
+```
+
+**Simpler alternative:** Use `wg-quick` post-up/down scripts to manage routes:
+
+```ini
+[Interface]
+PrivateKey = <key>
+Address = 10.5.0.2/16
+PostUp = ip route add 192.168.1.0/24 via $(ip route show default | awk '{print $3}') dev eth0
+PostUp = ip route add 100.64.0.0/10 via $(ip route show default | awk '{print $3}') dev eth0
+PreDown = ip route del 192.168.1.0/24 via $(ip route show default | awk '{print $3}') dev eth0 2>/dev/null; true
+PreDown = ip route del 100.64.0.0/10 via $(ip route show default | awk '{print $3}') dev eth0 2>/dev/null; true
+
+[Peer]
+PublicKey = <key>
+AllowedIPs = 0.0.0.0/0
+Endpoint = <server>:51820
+```
+
+### Test
+
+```bash
+ssh $CT_HOST "wg-quick up $VPN_CONFIG_DIR/us.conf && sleep 2 && curl -s https://ifconfig.me && echo && wg-quick down $VPN_CONFIG_DIR/us.conf"
+```
+
+Must show a NordVPN IP. Verify local services still reachable:
+
+```bash
+# From another machine on the LAN, while VPN is up:
+curl -s http://<CT_LOCAL_IP>:<SERVICE_PORT>/  # Must still respond
+```
+
+---
+
+## Step 4: VPN Rotation Helper Script
+
+Regardless of Option A or B, create a rotation script that other services can call.
+
+```bash
+ssh $CT_HOST "cat > $VPN_CONFIG_DIR/vpn-rotate.sh << 'SCRIPT'
+#!/bin/bash
+# VPN Rotation Script
+# Usage: vpn-rotate.sh [connect|disconnect|rotate|status]
+
+CONFIG_DIR=\"$VPN_CONFIG_DIR\"
+STATE_FILE=\"$VPN_CONFIG_DIR/vpn-state.json\"
+COUNTRIES=($VPN_COUNTRIES)
+
+# Detect VPN method
+if command -v nordvpn &>/dev/null && systemctl is-active --quiet nordvpnd 2>/dev/null; then
+    VPN_METHOD=nordvpn
+else
+    VPN_METHOD=wireguard
+fi
+
+get_current() {
+    if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
+        nordvpn status 2>/dev/null | grep -i country | awk '{print \$NF}'
+    else
+        wg show 2>/dev/null | head -1 | awk '{print \$2}' | sed 's/.conf//'
+    fi
+}
+
+get_public_ip() {
+    curl -s --connect-timeout 5 https://ifconfig.me 2>/dev/null
+}
+
+vpn_connect() {
+    local country=\${1:-\${COUNTRIES[0]}}
+    echo \"Connecting to \$country...\"
+    if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
+        nordvpn connect \"\$country\"
+    else
+        # Disconnect any existing
+        for conf in \$CONFIG_DIR/*.conf; do
+            wg-quick down \"\$conf\" 2>/dev/null
+        done
+        local conf_file=\"\$CONFIG_DIR/\$(echo \$country | tr '[:upper:]' '[:lower:]' | cut -c1-2).conf\"
+        if [ -f \"\$conf_file\" ]; then
+            wg-quick up \"\$conf_file\"
+        else
+            echo \"ERROR: No config for \$country (\$conf_file)\"
+            return 1
+        fi
+    fi
+    sleep 3
+    echo \"Public IP: \$(get_public_ip)\"
+}
+
+vpn_disconnect() {
+    if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
+        nordvpn disconnect
+    else
+        for conf in \$CONFIG_DIR/*.conf; do
+            wg-quick down \"\$conf\" 2>/dev/null
+        done
+    fi
+}
+
+vpn_rotate() {
+    local current=\$(get_current)
+    local next_idx=0
+    for i in \"\${!COUNTRIES[@]}\"; do
+        if echo \"\${COUNTRIES[\$i]}\" | grep -qi \"\$current\"; then
+            next_idx=$(( (i + 1) % \${#COUNTRIES[@]} ))
+            break
+        fi
+    done
+    vpn_disconnect
+    sleep 2
+    vpn_connect \"\${COUNTRIES[\$next_idx]}\"
+}
+
+vpn_status() {
+    echo \"Method:  \$VPN_METHOD\"
+    echo \"Country: \$(get_current || echo 'disconnected')\"
+    echo \"IP:      \$(get_public_ip || echo 'unknown')\"
+}
+
+case \"\${1:-status}\" in
+    connect)    vpn_connect \"\$2\" ;;
+    disconnect) vpn_disconnect ;;
+    rotate)     vpn_rotate ;;
+    status)     vpn_status ;;
+    *)          echo \"Usage: \$0 {connect [country]|disconnect|rotate|status}\" ;;
+esac
+SCRIPT
+chmod +x $VPN_CONFIG_DIR/vpn-rotate.sh"
+```
+
+### Test rotation
+
+```bash
+ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh connect"
+ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh status"
+ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh rotate"
+ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh status"
+ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh disconnect"
+```
+
+Each `rotate` should switch countries and show a different IP.
+
+---
+
+## Verification Checklist
+
+```bash
+echo "=== VPN Setup Check ==="
+echo ""
+echo "TUN device:   $(ls /dev/net/tun 2>/dev/null && echo 'OK' || echo 'MISSING')"
+echo "VPN method:   $(command -v nordvpn >/dev/null && echo 'NordVPN CLI' || echo 'WireGuard')"
+echo "Configs:      $(ls $VPN_CONFIG_DIR/*.conf 2>/dev/null | wc -l) country configs"
+echo "Rotation:     $(ls $VPN_CONFIG_DIR/vpn-rotate.sh 2>/dev/null && echo 'OK' || echo 'MISSING')"
+echo ""
+echo "Quick connect test..."
+$VPN_CONFIG_DIR/vpn-rotate.sh connect
+echo "VPN IP:       $(curl -s https://ifconfig.me)"
+echo "Local access: $(curl -s -o /dev/null -w '%{http_code}' http://localhost:9000/ 2>/dev/null || echo 'N/A')"
+$VPN_CONFIG_DIR/vpn-rotate.sh disconnect
+echo "Home IP:      $(curl -s https://ifconfig.me)"
+```
+
+---
+
+## Troubleshooting
+
+### "RTNETLINK answers: Operation not permitted" on wg-quick up
+
+TUN device not available. Go back to Step 1. Container may need restart after adding cgroup entries.
+
+### VPN connects but local services unreachable
+
+Split tunneling not configured. The VPN is routing ALL traffic including LAN. Fix the `AllowedIPs` or add PostUp routes per Step 3.
+
+### DNS stops working when VPN is up
+
+NordVPN CLI: `nordvpn set dns off` (use container's DNS, not NordVPN's).
+WireGuard: Remove the `DNS =` line from the `.conf` file.
+
+### "Cannot open TUN/TAP dev /dev/net/tun: No such file or directory"
+
+Container config missing TUN mount entry. Check `/etc/pve/lxc/${CTID}.conf` for both the cgroup allow and mount entry lines.
+
+### NordVPN CLI installed but nordvpnd won't start
+
+Common in LXC. `systemctl status nordvpnd` will usually show a cgroup or namespace error. Fall through to WireGuard (Step 3).
+
+---
+
+*Last updated: 2026-02-13*