echo6-docs/projects/headscale-full-deployment.md

Headscale Full Deployment Runbook

Nodes + Headplane + Authentik OIDC

Headscale location: /opt/headscale-vanilla Container name: headscale-vanilla Domain: vpn.echo6.co Auth key: hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ

---

PHASE 1: REGISTER CONTABO (must be first)

tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname contabo --force-reauth

Verify:

docker exec headscale-vanilla headscale nodes list

STOP if contabo doesn't appear. Do not continue.

---

PHASE 2: REGISTER ALL LXC/CT NODES

SSH into each container. For each one:

# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"

# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh

Then register. Do them in this exact order for sequential IPs:

# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname utility --force-reauth

# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname data --force-reauth

# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname cloud --force-reauth

# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname media --force-reauth

# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname aida-nebra --force-reauth

After each, verify from Contabo:

docker exec headscale-vanilla headscale nodes list

---

PHASE 3: REGISTER DESKTOP + PHONES

Desktop (Windows — PowerShell as Admin):

tailscale up --login-server https://vpn.echo6.co `
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
  --hostname desktop --force-reauth

Phones:

• Open Tailscale app → Settings → Account
• Log out if needed
• Use "Custom coordination server" or "Alternate server"
• Enter: https://vpn.echo6.co
• Should auto-register with the tailnet

If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.

---

PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY

docker exec headscale-vanilla headscale nodes list

Expected output: all nodes with sequential 100.64.0.x IPs.

Test from any node:

tailscale ping contabo
tailscale ping data
tailscale ping utility

Test magic DNS:

ping data.echo6.mesh
ping utility.echo6.mesh

---

PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)

mkdir -p /opt/headscale-vanilla/backups

# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
  ".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"

# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete

---

PHASE 6: PERSISTENCE TEST

cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list

Every node must survive. If any are missing, STOP and report.

---

PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE

This lets Tailscale clients authenticate via Authentik instead of preauth keys.

1. Log into Authentik admin panel
2. Go to Applications → Applications → Create with Provider
3. Configure:
   • Application name: Headscale
   • Slug: headscale (remember this — it's part of the issuer URL)
   • Provider type: OAuth2/OpenID Connect
   • Authorization flow: default-provider-authorization-implicit-consent (or explicit if you want)
   • Redirect URI (Strict): https://vpn.echo6.co/oidc/callback
   • Signing key: Select any available key
   • Scopes: Ensure these scope mappings are selected:
     • openid
     • profile
     • email
     • offline_access ← CRITICAL — without this, nodes break on Headscale restart
4. Note the Client ID and Client Secret
5. Click Submit

---

PHASE 8: CONFIGURE HEADSCALE OIDC

Edit /opt/headscale-vanilla/config.yaml — add this OIDC block:

oidc:
  only_start_if_oidc_is_available: true
  issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
  client_id: "<Client ID from Authentik>"
  client_secret: "<Client Secret from Authentik>"
  scope: ["openid", "profile", "email", "offline_access"]
  pkce:
    enabled: true
    method: S256
  strip_email_domain: true

Replace:

• <YOUR_AUTHENTIK_DOMAIN> with your Authentik domain (e.g., auth.echo6.co)
• <Client ID from Authentik> with the actual client ID
• <Client Secret from Authentik> with the actual client secret

Restart Headscale:

cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20

Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.

Test: From any node, run:

tailscale up --login-server https://vpn.echo6.co --force-reauth

It should open a browser → Authentik login → back to terminal, authenticated.

Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.

---

PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE

This is a SECOND application in Authentik for the web UI login.

1. Go to Applications → Applications → Create with Provider
2. Configure:
   • Application name: Headplane
   • Slug: headplane
   • Provider type: OAuth2/OpenID Connect
   • Authorization flow: Same as before
   • Redirect URI (Strict): https://vpn.echo6.co/admin/oidc/callback
   • Signing key: Same key
   • Scopes: openid, profile, email
3. Note the Client ID and Client Secret (different from Headscale's)
4. Click Submit

---

PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE

docker exec headscale-vanilla headscale apikeys create --expiration 999d

Save this key — you need it for the Headplane config.

---

PHASE 11: CREATE HEADPLANE CONFIG

# Generate a cookie secret
openssl rand -hex 16

Write /opt/headscale-vanilla/headplane-config.yaml:

server:
  host: "0.0.0.0"
  port: 3000
  cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
  cookie_secure: true
  data_path: "/var/lib/headplane"

headscale:
  url: "http://headscale-vanilla:8080"
  config_path: "/etc/headscale/config.yaml"
  config_strict: false

oidc:
  issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
  client_id: "<Headplane Client ID from Authentik>"
  client_secret: "<Headplane Client Secret from Authentik>"
  token_endpoint_auth_method: "client_secret_post"
  headscale_api_key: "<API_KEY_FROM_PHASE_10>"
  redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
  disable_api_key_login: false

integration:
  docker:
    enabled: true
    container_name: "headscale-vanilla"
    socket: "/var/run/docker.sock"

Replace all <PLACEHOLDERS> with actual values.

---

PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE

Edit /opt/headscale-vanilla/docker-compose.yml — add the headplane service:

services:
  headscale:
    # ... your existing headscale service, don't change it ...

  headplane:
    image: ghcr.io/tale/headplane:latest
    container_name: headplane
    restart: unless-stopped
    depends_on:
      - headscale
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - ./headplane-config.yaml:/etc/headplane/config.yaml:ro
      - ./headplane-data:/var/lib/headplane
      - ./config.yaml:/etc/headscale/config.yaml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

Start it:

cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20

Check for errors. Common issues:

• "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
• Can't connect to headscale → ensure url matches the container name and internal port
• Docker socket permission denied → check that the headplane container can read /var/run/docker.sock

---

PHASE 13: UPDATE CADDY FOR HEADPLANE

Add the /admin route to your Caddy config for vpn.echo6.co:

vpn.echo6.co {
    handle /admin* {
        reverse_proxy 127.0.0.1:3000
    }
    handle {
        reverse_proxy 127.0.0.1:8084
    }
}

Restart Caddy:

# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile

---

PHASE 14: TEST HEADPLANE

1. Browse to https://vpn.echo6.co/admin
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. The FIRST user to log in gets Owner permissions
5. Verify you can see all your nodes in the UI

If OIDC fails, you can still log in with the API key (that's why we set disable_api_key_login: false).

---

PHASE 15: FINAL VERIFICATION

Run all of these from Contabo:

# All nodes present?
docker exec headscale-vanilla headscale nodes list

# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"

# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302

# Database backed up?
ls -la /opt/headscale-vanilla/backups/

# Cron running?
crontab -l | grep sqlite3

---

REPORT TEMPLATE

After each phase, report:

Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):

Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.

---

KNOWN GOTCHAS

1. offline_access scope — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. config_strict: false — Headscale 0.28.0 has config options Headplane may not recognize
3. Headplane needs Docker socket — For the integration that lets it restart Headscale when you change settings
4. First OIDC login = Owner — Don't let random people hit your Headplane URL before you log in first
5. Phones may not support custom servers — Android F-Droid build is more flexible; iOS is limited
6. Two separate OIDC apps — Headscale and Headplane each need their own application in Authentik with different redirect URIs