echo6-docs/projects/headscale-full-deployment.md

# Headscale Full Deployment Runbook
## Nodes + Headplane + Authentik OIDC

**Headscale location:** `/opt/headscale-vanilla`
**Container name:** `headscale-vanilla`
**Domain:** `vpn.echo6.co`
**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ`

---

## PHASE 1: REGISTER CONTABO (must be first)

```bash
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname contabo --force-reauth
```

Verify:
```bash
docker exec headscale-vanilla headscale nodes list
```
**STOP if contabo doesn't appear. Do not continue.**

---

## PHASE 2: REGISTER ALL LXC/CT NODES

SSH into each container. For each one:

```bash
# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"

# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh
```

Then register. **Do them in this exact order for sequential IPs:**

```bash
# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname utility --force-reauth

# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname data --force-reauth

# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname cloud --force-reauth

# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname media --force-reauth

# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
  --hostname aida-nebra --force-reauth
```

After each, verify from Contabo:
```bash
docker exec headscale-vanilla headscale nodes list
```

---

## PHASE 3: REGISTER DESKTOP + PHONES

**Desktop (Windows — PowerShell as Admin):**
```powershell
tailscale up --login-server https://vpn.echo6.co `
  --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
  --hostname desktop --force-reauth
```

**Phones:**
- Open Tailscale app → Settings → Account
- Log out if needed
- Use "Custom coordination server" or "Alternate server"
- Enter: `https://vpn.echo6.co`
- Should auto-register with the tailnet

If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.

---

## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY

```bash
docker exec headscale-vanilla headscale nodes list
```

Expected output: all nodes with sequential 100.64.0.x IPs.

Test from any node:
```bash
tailscale ping contabo
tailscale ping data
tailscale ping utility
```

Test magic DNS:
```bash
ping data.echo6.mesh
ping utility.echo6.mesh
```

---

## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)

```bash
mkdir -p /opt/headscale-vanilla/backups

# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
  ".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"

# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
```

---

## PHASE 6: PERSISTENCE TEST

```bash
cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list
```

**Every node must survive. If any are missing, STOP and report.**

---

## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE

This lets Tailscale clients authenticate via Authentik instead of preauth keys.

1. Log into Authentik admin panel
2. Go to **Applications → Applications → Create with Provider**
3. Configure:
   - **Application name:** Headscale
   - **Slug:** `headscale` (remember this — it's part of the issuer URL)
   - **Provider type:** OAuth2/OpenID Connect
   - **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
   - **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
   - **Signing key:** Select any available key
   - **Scopes:** Ensure these scope mappings are selected:
     - `openid`
     - `profile`
     - `email`
     - **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
4. Note the **Client ID** and **Client Secret**
5. Click Submit

---

## PHASE 8: CONFIGURE HEADSCALE OIDC

Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:

```yaml
oidc:
  only_start_if_oidc_is_available: true
  issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
  client_id: "<Client ID from Authentik>"
  client_secret: "<Client Secret from Authentik>"
  scope: ["openid", "profile", "email", "offline_access"]
  pkce:
    enabled: true
    method: S256
  strip_email_domain: true
```

Replace:
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
- `<Client ID from Authentik>` with the actual client ID
- `<Client Secret from Authentik>` with the actual client secret

Restart Headscale:
```bash
cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20
```

**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**

Test: From any node, run:
```bash
tailscale up --login-server https://vpn.echo6.co --force-reauth
```
It should open a browser → Authentik login → back to terminal, authenticated.

**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**

---

## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE

This is a SECOND application in Authentik for the web UI login.

1. Go to **Applications → Applications → Create with Provider**
2. Configure:
   - **Application name:** Headplane
   - **Slug:** `headplane`
   - **Provider type:** OAuth2/OpenID Connect
   - **Authorization flow:** Same as before
   - **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
   - **Signing key:** Same key
   - **Scopes:** `openid`, `profile`, `email`
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
4. Click Submit

---

## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE

```bash
docker exec headscale-vanilla headscale apikeys create --expiration 999d
```

**Save this key — you need it for the Headplane config.**

---

## PHASE 11: CREATE HEADPLANE CONFIG

```bash
# Generate a cookie secret
openssl rand -hex 16
```

Write `/opt/headscale-vanilla/headplane-config.yaml`:

```yaml
server:
  host: "0.0.0.0"
  port: 3000
  cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
  cookie_secure: true
  data_path: "/var/lib/headplane"

headscale:
  url: "http://headscale-vanilla:8080"
  config_path: "/etc/headscale/config.yaml"
  config_strict: false

oidc:
  issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
  client_id: "<Headplane Client ID from Authentik>"
  client_secret: "<Headplane Client Secret from Authentik>"
  token_endpoint_auth_method: "client_secret_post"
  headscale_api_key: "<API_KEY_FROM_PHASE_10>"
  redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
  disable_api_key_login: false

integration:
  docker:
    enabled: true
    container_name: "headscale-vanilla"
    socket: "/var/run/docker.sock"
```

Replace all `<PLACEHOLDERS>` with actual values.

---

## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE

Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:

```yaml
services:
  headscale:
    # ... your existing headscale service, don't change it ...

  headplane:
    image: ghcr.io/tale/headplane:latest
    container_name: headplane
    restart: unless-stopped
    depends_on:
      - headscale
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - ./headplane-config.yaml:/etc/headplane/config.yaml:ro
      - ./headplane-data:/var/lib/headplane
      - ./config.yaml:/etc/headscale/config.yaml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
```

Start it:
```bash
cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20
```

Check for errors. Common issues:
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
- Can't connect to headscale → ensure `url` matches the container name and internal port
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock

---

## PHASE 13: UPDATE CADDY FOR HEADPLANE

Add the `/admin` route to your Caddy config for `vpn.echo6.co`:

```
vpn.echo6.co {
    handle /admin* {
        reverse_proxy 127.0.0.1:3000
    }
    handle {
        reverse_proxy 127.0.0.1:8084
    }
}
```

Restart Caddy:
```bash
# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```

---

## PHASE 14: TEST HEADPLANE

1. Browse to `https://vpn.echo6.co/admin`
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. **The FIRST user to log in gets Owner permissions**
5. Verify you can see all your nodes in the UI

If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).

---

## PHASE 15: FINAL VERIFICATION

Run all of these from Contabo:

```bash
# All nodes present?
docker exec headscale-vanilla headscale nodes list

# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"

# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302

# Database backed up?
ls -la /opt/headscale-vanilla/backups/

# Cron running?
crontab -l | grep sqlite3
```

---

## REPORT TEMPLATE

After each phase, report:

```
Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):
```

**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**

---

## KNOWN GOTCHAS

1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs