echo6-docs/vault/runbooks/headscale-oidc-boot-order.md

title	type	tags	aliases	related	updated
Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)	runbook	auth		authentik-oidc-application headscale-onboard-node authentik authentik-upgrade fleet-magicdns-resolved-migration	2026-07-13

Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)

Headscale wires up its OIDC provider once, at process startup, by fetching authentik's discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs OIDC-disabled until it is restarted. This is a boot-ordering hazard: the fleet Headscale (edge2 CT107) and Authentik (edge2 CT105, auth.echo6.co) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.

Symptom

• A client (especially the Tailscale Android app on a GrapheneOS phone) hits Connect and lands on Headscale's manual node-registration page (the "run headscale nodes register ..." screen) instead of being redirected to Authentik for SSO login.
• New nodes can't self-register via SSO; only the preauth-key / CLI path works.
• vpn.echo6.co/health still returns pass — the service is up, it's just running without OIDC. The failure is silent.

Root cause

The Headscale config sets only_start_if_oidc_is_available: false, so Headscale boots even when OIDC discovery fails. When Authentik was unreachable at startup, the log shows:

WRN failed to set up OIDC provider, falling back to CLI based authentication
    error="creating OIDC provider from issuer config: 502 Bad Gateway"

From that point the running process serves only manual/CLI registration — even after Authentik recovers — because OIDC is initialized only at startup, never retried.

Diagnosis

Access path: ssh edge2 (admin) then sudo pct exec 107 -- docker exec headscale headscale <cmd>. See edge2-access-reference.

# 1. Is OIDC currently broken? Look for the 502 / fallback warning since the container started:
ssh edge2 "sudo pct exec 107 -- docker logs headscale --since 48h 2>&1 | grep -iE 'oidc|502|fallback'"

# 2. Is Authentik discovery healthy NOW (from inside CT107)?
ssh edge2 "sudo pct exec 107 -- curl -fsS --max-time 8 \
  https://auth.echo6.co/application/o/headscale/.well-known/openid-configuration \
  -o /dev/null -w 'HTTP %{http_code}\n'"

If step 1 shows the fallback warning AND step 2 returns HTTP 200, OIDC is recoverable by a restart.

Fix

Restart the Headscale container so it re-initializes the OIDC provider against now-healthy Authentik:

ssh edge2 "sudo pct exec 107 -- docker restart headscale"

Verify the fix:

# Fallback warning should be GONE this start (only the benign 'listening without TLS' WRN is expected):
ssh edge2 "sudo pct exec 107 -- docker logs headscale --since 3m 2>&1 | grep -iE 'oidc|502|fallback|warn'"

# Health + fleet intact:
curl -fsS https://vpn.echo6.co/health         # -> {"status":"pass"}
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale nodes list | grep -c 100.64.0"   # ~34+ nodes

Then have the client hit Connect again — it should now redirect to Authentik (silently, if the device's browser already holds a live auth.echo6.co SSO session).

Impact of the restart: low. Existing WireGuard tunnels stay up (no node drops off the tailnet); only control-plane coordination pauses for a few seconds. Headscale is lockout-critical — drive the restart only from the edge2 out-of-band path above, never over vpn.echo6.co itself.

Prevention / follow-up

• The failure is silent (health stays green), so it recurs on any future edge2/Authentik restart-ordering hiccup. Candidate hardening: a health-gated watchdog that restarts Headscale when the logs show the OIDC fallback warning while Authentik discovery returns 200; or a systemd boot-order/ExecStartPre gate so Headscale waits for Authentik discovery before starting.
• After any edge2 reboot, treat "can a fresh client SSO-register?" as a post-reboot check — not just vpn.echo6.co/health.

Reference incident

• 2026-06-30 — GrapheneOS phone reauth landed on the manual-registration page. Headscale had booted ~11h earlier (during an edge2 reboot / DNS-bootstrap window) while Authentik was 502'ing, so OIDC was disabled. Restarting the headscale container restored OIDC; the phone then registered via Authentik (silent SSO) as user matt. Onboarding control-server path also corrected in headscale-onboard-node the same day.