Initial commit: infrastructure documentation
Includes: - Hardware environment reference (Proxmox cluster, VMs, LXCs) - Services inventory with current deployments - Caddy & DNS configuration reference - Runbooks for common deployment procedures Recent additions: - SearXNG deployment (utility CT 102, search.echo6.co) - TOC conversion to Proxmox with cortex VM - Syncthing sync between Contabo and cortex Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
commit
880ff09c90
14 changed files with 1986 additions and 0 deletions
406
runbooks/headscale-full-deployment.md
Executable file
406
runbooks/headscale-full-deployment.md
Executable file
|
|
@ -0,0 +1,406 @@
|
|||
# Headscale Full Deployment Runbook
|
||||
## Nodes + Headplane + Authentik OIDC
|
||||
|
||||
**Headscale location:** `/opt/headscale-vanilla`
|
||||
**Container name:** `headscale-vanilla`
|
||||
**Domain:** `vpn.echo6.co`
|
||||
**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ`
|
||||
|
||||
---
|
||||
|
||||
## PHASE 1: REGISTER CONTABO (must be first)
|
||||
|
||||
```bash
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname contabo --force-reauth
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
docker exec headscale-vanilla headscale nodes list
|
||||
```
|
||||
**STOP if contabo doesn't appear. Do not continue.**
|
||||
|
||||
---
|
||||
|
||||
## PHASE 2: REGISTER ALL LXC/CT NODES
|
||||
|
||||
SSH into each container. For each one:
|
||||
|
||||
```bash
|
||||
# Check if tailscale is installed
|
||||
which tailscale || echo "NOT INSTALLED"
|
||||
|
||||
# Install if missing
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
```
|
||||
|
||||
Then register. **Do them in this exact order for sequential IPs:**
|
||||
|
||||
```bash
|
||||
# utility (will get 100.64.0.2)
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname utility --force-reauth
|
||||
|
||||
# data (will get 100.64.0.3)
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname data --force-reauth
|
||||
|
||||
# cloud (will get 100.64.0.4)
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname cloud --force-reauth
|
||||
|
||||
# media (will get 100.64.0.5)
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname media --force-reauth
|
||||
|
||||
# aida-nebra (will get 100.64.0.6)
|
||||
tailscale up --login-server https://vpn.echo6.co \
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
|
||||
--hostname aida-nebra --force-reauth
|
||||
```
|
||||
|
||||
After each, verify from Contabo:
|
||||
```bash
|
||||
docker exec headscale-vanilla headscale nodes list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 3: REGISTER DESKTOP + PHONES
|
||||
|
||||
**Desktop (Windows — PowerShell as Admin):**
|
||||
```powershell
|
||||
tailscale up --login-server https://vpn.echo6.co `
|
||||
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
|
||||
--hostname desktop --force-reauth
|
||||
```
|
||||
|
||||
**Phones:**
|
||||
- Open Tailscale app → Settings → Account
|
||||
- Log out if needed
|
||||
- Use "Custom coordination server" or "Alternate server"
|
||||
- Enter: `https://vpn.echo6.co`
|
||||
- Should auto-register with the tailnet
|
||||
|
||||
If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY
|
||||
|
||||
```bash
|
||||
docker exec headscale-vanilla headscale nodes list
|
||||
```
|
||||
|
||||
Expected output: all nodes with sequential 100.64.0.x IPs.
|
||||
|
||||
Test from any node:
|
||||
```bash
|
||||
tailscale ping contabo
|
||||
tailscale ping data
|
||||
tailscale ping utility
|
||||
```
|
||||
|
||||
Test magic DNS:
|
||||
```bash
|
||||
ping data.echo6.mesh
|
||||
ping utility.echo6.mesh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)
|
||||
|
||||
```bash
|
||||
mkdir -p /opt/headscale-vanilla/backups
|
||||
|
||||
# Immediate backup
|
||||
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
|
||||
".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"
|
||||
|
||||
# Set up cron for automatic backups every 6 hours, 7-day retention
|
||||
crontab -e
|
||||
# Add this line:
|
||||
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 6: PERSISTENCE TEST
|
||||
|
||||
```bash
|
||||
cd /opt/headscale-vanilla
|
||||
docker compose down
|
||||
sleep 5
|
||||
ls -la /opt/headscale-vanilla/data/db.sqlite*
|
||||
docker compose up -d
|
||||
sleep 10
|
||||
docker exec headscale-vanilla headscale nodes list
|
||||
```
|
||||
|
||||
**Every node must survive. If any are missing, STOP and report.**
|
||||
|
||||
---
|
||||
|
||||
## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE
|
||||
|
||||
This lets Tailscale clients authenticate via Authentik instead of preauth keys.
|
||||
|
||||
1. Log into Authentik admin panel
|
||||
2. Go to **Applications → Applications → Create with Provider**
|
||||
3. Configure:
|
||||
- **Application name:** Headscale
|
||||
- **Slug:** `headscale` (remember this — it's part of the issuer URL)
|
||||
- **Provider type:** OAuth2/OpenID Connect
|
||||
- **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
|
||||
- **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
|
||||
- **Signing key:** Select any available key
|
||||
- **Scopes:** Ensure these scope mappings are selected:
|
||||
- `openid`
|
||||
- `profile`
|
||||
- `email`
|
||||
- **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
|
||||
4. Note the **Client ID** and **Client Secret**
|
||||
5. Click Submit
|
||||
|
||||
---
|
||||
|
||||
## PHASE 8: CONFIGURE HEADSCALE OIDC
|
||||
|
||||
Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:
|
||||
|
||||
```yaml
|
||||
oidc:
|
||||
only_start_if_oidc_is_available: true
|
||||
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
|
||||
client_id: "<Client ID from Authentik>"
|
||||
client_secret: "<Client Secret from Authentik>"
|
||||
scope: ["openid", "profile", "email", "offline_access"]
|
||||
pkce:
|
||||
enabled: true
|
||||
method: S256
|
||||
strip_email_domain: true
|
||||
```
|
||||
|
||||
Replace:
|
||||
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
|
||||
- `<Client ID from Authentik>` with the actual client ID
|
||||
- `<Client Secret from Authentik>` with the actual client secret
|
||||
|
||||
Restart Headscale:
|
||||
```bash
|
||||
cd /opt/headscale-vanilla
|
||||
docker compose restart
|
||||
sleep 10
|
||||
docker logs headscale-vanilla 2>&1 | tail -20
|
||||
```
|
||||
|
||||
**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**
|
||||
|
||||
Test: From any node, run:
|
||||
```bash
|
||||
tailscale up --login-server https://vpn.echo6.co --force-reauth
|
||||
```
|
||||
It should open a browser → Authentik login → back to terminal, authenticated.
|
||||
|
||||
**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**
|
||||
|
||||
---
|
||||
|
||||
## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE
|
||||
|
||||
This is a SECOND application in Authentik for the web UI login.
|
||||
|
||||
1. Go to **Applications → Applications → Create with Provider**
|
||||
2. Configure:
|
||||
- **Application name:** Headplane
|
||||
- **Slug:** `headplane`
|
||||
- **Provider type:** OAuth2/OpenID Connect
|
||||
- **Authorization flow:** Same as before
|
||||
- **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
|
||||
- **Signing key:** Same key
|
||||
- **Scopes:** `openid`, `profile`, `email`
|
||||
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
|
||||
4. Click Submit
|
||||
|
||||
---
|
||||
|
||||
## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE
|
||||
|
||||
```bash
|
||||
docker exec headscale-vanilla headscale apikeys create --expiration 999d
|
||||
```
|
||||
|
||||
**Save this key — you need it for the Headplane config.**
|
||||
|
||||
---
|
||||
|
||||
## PHASE 11: CREATE HEADPLANE CONFIG
|
||||
|
||||
```bash
|
||||
# Generate a cookie secret
|
||||
openssl rand -hex 16
|
||||
```
|
||||
|
||||
Write `/opt/headscale-vanilla/headplane-config.yaml`:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
host: "0.0.0.0"
|
||||
port: 3000
|
||||
cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
|
||||
cookie_secure: true
|
||||
data_path: "/var/lib/headplane"
|
||||
|
||||
headscale:
|
||||
url: "http://headscale-vanilla:8080"
|
||||
config_path: "/etc/headscale/config.yaml"
|
||||
config_strict: false
|
||||
|
||||
oidc:
|
||||
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
|
||||
client_id: "<Headplane Client ID from Authentik>"
|
||||
client_secret: "<Headplane Client Secret from Authentik>"
|
||||
token_endpoint_auth_method: "client_secret_post"
|
||||
headscale_api_key: "<API_KEY_FROM_PHASE_10>"
|
||||
redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
|
||||
disable_api_key_login: false
|
||||
|
||||
integration:
|
||||
docker:
|
||||
enabled: true
|
||||
container_name: "headscale-vanilla"
|
||||
socket: "/var/run/docker.sock"
|
||||
```
|
||||
|
||||
Replace all `<PLACEHOLDERS>` with actual values.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE
|
||||
|
||||
Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
headscale:
|
||||
# ... your existing headscale service, don't change it ...
|
||||
|
||||
headplane:
|
||||
image: ghcr.io/tale/headplane:latest
|
||||
container_name: headplane
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- headscale
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
volumes:
|
||||
- ./headplane-config.yaml:/etc/headplane/config.yaml:ro
|
||||
- ./headplane-data:/var/lib/headplane
|
||||
- ./config.yaml:/etc/headscale/config.yaml:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
```
|
||||
|
||||
Start it:
|
||||
```bash
|
||||
cd /opt/headscale-vanilla
|
||||
docker compose up -d
|
||||
sleep 10
|
||||
docker logs headplane 2>&1 | tail -20
|
||||
```
|
||||
|
||||
Check for errors. Common issues:
|
||||
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
|
||||
- Can't connect to headscale → ensure `url` matches the container name and internal port
|
||||
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock
|
||||
|
||||
---
|
||||
|
||||
## PHASE 13: UPDATE CADDY FOR HEADPLANE
|
||||
|
||||
Add the `/admin` route to your Caddy config for `vpn.echo6.co`:
|
||||
|
||||
```
|
||||
vpn.echo6.co {
|
||||
handle /admin* {
|
||||
reverse_proxy 127.0.0.1:3000
|
||||
}
|
||||
handle {
|
||||
reverse_proxy 127.0.0.1:8084
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Restart Caddy:
|
||||
```bash
|
||||
# Wherever your Caddy lives — adjust path as needed
|
||||
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 14: TEST HEADPLANE
|
||||
|
||||
1. Browse to `https://vpn.echo6.co/admin`
|
||||
2. You should see the Headplane login page
|
||||
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
|
||||
4. **The FIRST user to log in gets Owner permissions**
|
||||
5. Verify you can see all your nodes in the UI
|
||||
|
||||
If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).
|
||||
|
||||
---
|
||||
|
||||
## PHASE 15: FINAL VERIFICATION
|
||||
|
||||
Run all of these from Contabo:
|
||||
|
||||
```bash
|
||||
# All nodes present?
|
||||
docker exec headscale-vanilla headscale nodes list
|
||||
|
||||
# Both containers healthy?
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||||
|
||||
# Headplane accessible?
|
||||
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
|
||||
# Should return 200 or 302
|
||||
|
||||
# Database backed up?
|
||||
ls -la /opt/headscale-vanilla/backups/
|
||||
|
||||
# Cron running?
|
||||
crontab -l | grep sqlite3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REPORT TEMPLATE
|
||||
|
||||
After each phase, report:
|
||||
|
||||
```
|
||||
Phase X complete:
|
||||
- Output of headscale nodes list:
|
||||
- Any errors:
|
||||
- Logs (last 10 lines):
|
||||
```
|
||||
|
||||
**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**
|
||||
|
||||
---
|
||||
|
||||
## KNOWN GOTCHAS
|
||||
|
||||
1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
|
||||
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
|
||||
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
|
||||
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
|
||||
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
|
||||
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs
|
||||
Loading…
Add table
Add a link
Reference in a new issue