echo6-docs/vault/runbooks/headscale-oidc-boot-order.md

---
title: Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)
type: runbook
tags:
  - auth
aliases: []
related:
  - [[authentik-oidc-application]]
  - [[headscale-onboard-node]]
  - [[authentik]]
  - [[authentik-upgrade]]
  - [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)

Headscale wires up its OIDC provider **once, at process startup**, by fetching [[authentik]]'s discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs **OIDC-disabled until it is restarted**. This is a boot-ordering hazard: the fleet Headscale (edge2 **CT107**) and Authentik (edge2 **CT105**, `auth.echo6.co`) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.

## Symptom

- A client (especially the **Tailscale Android app on a GrapheneOS phone**) hits **Connect** and lands on Headscale's **manual node-registration page** (the "run `headscale nodes register ...`" screen) instead of being redirected to **Authentik** for SSO login.
- New nodes can't self-register via SSO; only the preauth-key / CLI path works.
- `vpn.echo6.co/health` still returns `pass` — the service is *up*, it's just running without OIDC. The failure is silent.

## Root cause

The Headscale config sets `only_start_if_oidc_is_available: false`, so Headscale boots even when OIDC discovery fails. When Authentik was unreachable at startup, the log shows:

```
WRN failed to set up OIDC provider, falling back to CLI based authentication
    error="creating OIDC provider from issuer config: 502 Bad Gateway"
```

From that point the running process serves only manual/CLI registration — even after Authentik recovers — because OIDC is initialized only at startup, never retried.

## Diagnosis

Access path: `ssh edge2` (admin) then `sudo pct exec 107 -- docker exec headscale headscale <cmd>`. See [[edge2-access-reference]].

```bash
# 1. Is OIDC currently broken? Look for the 502 / fallback warning since the container started:
ssh edge2 "sudo pct exec 107 -- docker logs headscale --since 48h 2>&1 | grep -iE 'oidc|502|fallback'"

# 2. Is Authentik discovery healthy NOW (from inside CT107)?
ssh edge2 "sudo pct exec 107 -- curl -fsS --max-time 8 \
  https://auth.echo6.co/application/o/headscale/.well-known/openid-configuration \
  -o /dev/null -w 'HTTP %{http_code}\n'"
```

If step 1 shows the fallback warning AND step 2 returns `HTTP 200`, OIDC is recoverable by a restart.

## Fix

Restart the Headscale container so it re-initializes the OIDC provider against now-healthy Authentik:

```bash
ssh edge2 "sudo pct exec 107 -- docker restart headscale"
```

Verify the fix:

```bash
# Fallback warning should be GONE this start (only the benign 'listening without TLS' WRN is expected):
ssh edge2 "sudo pct exec 107 -- docker logs headscale --since 3m 2>&1 | grep -iE 'oidc|502|fallback|warn'"

# Health + fleet intact:
curl -fsS https://vpn.echo6.co/health         # -> {"status":"pass"}
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale nodes list | grep -c 100.64.0"   # ~34+ nodes
```

Then have the client hit **Connect** again — it should now redirect to Authentik (silently, if the device's browser already holds a live `auth.echo6.co` SSO session).

> **Impact of the restart:** low. Existing WireGuard tunnels stay up (no node drops off the tailnet); only control-plane coordination pauses for a few seconds. Headscale is lockout-critical — drive the restart only from the edge2 out-of-band path above, never over `vpn.echo6.co` itself.

## Prevention / follow-up

- The failure is **silent** (health stays green), so it recurs on any future edge2/Authentik restart-ordering hiccup. Candidate hardening: a health-gated watchdog that restarts Headscale when the logs show the OIDC fallback warning *while* Authentik discovery returns 200; or a systemd boot-order/`ExecStartPre` gate so Headscale waits for Authentik discovery before starting.
- After any **edge2 reboot**, treat "can a fresh client SSO-register?" as a post-reboot check — not just `vpn.echo6.co/health`.

## Reference incident

- **2026-06-30** — GrapheneOS phone reauth landed on the manual-registration page. Headscale had booted ~11h earlier (during an edge2 reboot / DNS-bootstrap window) while Authentik was 502'ing, so OIDC was disabled. Restarting the `headscale` container restored OIDC; the phone then registered via Authentik (silent SSO) as user `matt`. Onboarding control-server path also corrected in [[headscale-onboard-node]] the same day.