echo6-docs/vault/runbooks/authentik-upgrade.md

title	type	tags	aliases	related	updated
Authentik: Major Version Upgrade	runbook	auth		authentik-oidc-application authentik headscale-oidc-boot-order lxc-service-migration authentik-create-invitation	2026-07-13

Authentik: Major Version Upgrade

Migrated off Contabo → edge2 CT 105 on 2026-06-18.

Upgrade authentik between major versions on edge2 CT 105. Covers backup, upgrade, verification, and rollback.

---

When to Use This

Any time authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.

---

Prerequisites

• SSH access to edge2 CT 105 via edge2 host: ssh edge2 'sudo pct exec 105 -- bash' (interactive shell inside the CT; root SSH direct to edge2/the CT is refused — see edge2-access-reference)
• Authentik compose directory: /opt/authentik/
• Current version: check with docker exec authentik-server ak --version

---

Inputs

CURRENT_VERSION=2025.12.4    # Current running version
TARGET_VERSION=2026.2.1      # Version to upgrade to

---

Step 1: Check Release Notes

Before upgrading, read the release notes for every major version between current and target:

https://docs.goauthentik.io/docs/releases/

Look for:

• Breaking changes — removed features, changed defaults, API changes
• Dependency changes — added/removed services (e.g., Redis removed in 2025.10)
• Configuration changes — new required env vars, changed mount paths
• Database migrations — large migrations that may take time

Known Breaking Changes (Reference)

Version	Change	Impact
2025.10	Redis completely removed	Delete redis service + all AUTHENTIK_REDIS env vars
2025.10	Default email scope returns email_verified: false	Use custom scope mapping (PK 02c22323) that forces true
2025.10	Worker requires user: root	Add user: root to worker service in compose
2025.12	Stage creation endpoints moved	stages/<type>/stages/ for POST (some types)

---

Step 2: Backup

2a. Snapshot edge2 CT 105 (Proxmox-managed LXC)

CT 105 is an LXC on edge2 — take a Proxmox snapshot before upgrading: ssh edge2 "sudo pct snapshot 105 pre-upgrade-$(date +%Y%m%d)". Also rely on the file-level backups below.

2b. PostgreSQL Dump

ssh edge2 'sudo pct exec 105 -- bash'  # interactive shell inside CT 105; run the rest of this section inside it

cd /opt/authentik
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

docker exec authentik-postgres \
  pg_dump -U authentik -d authentik \
  --clean --if-exists \
  > /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

ls -lh /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

2c. Compose Directory Backup

cp -a /opt/authentik /opt/authentik.bak_${TIMESTAMP}

This preserves docker-compose.yml, .env, certs/, and any custom files.

2d. Record Current State

# Save current version
docker exec authentik-server ak --version

# Save current provider list (for post-upgrade comparison)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/" \
  -H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)" \
  | python3 -c "import sys,json; [print(f'{p[\"pk\"]:3d} {p[\"name\"]}') for p in json.load(sys.stdin)['results']]"

---

Step 3: Apply Compose Changes

Review release notes and update docker-compose.yml before pulling the new image:

cd /opt/authentik
nano docker-compose.yml

Common changes by version:

Removing Redis (2025.10+):

# DELETE the redis service entirely
# DELETE these env vars from server + worker:
#   AUTHENTIK_REDIS__HOST
#   AUTHENTIK_REDIS__PORT

Worker user requirement (2025.10+):

services:
  worker:
    user: root   # ADD this line

New env vars: Check release notes for any new required AUTHENTIK_* env vars. Add to both server and worker services.

---

Step 4: Upgrade

cd /opt/authentik

# Update image tag in docker-compose.yml
# Change: image: ghcr.io/goauthentik/server:CURRENT_VERSION
# To:     image: ghcr.io/goauthentik/server:TARGET_VERSION
nano docker-compose.yml

# Pull new image
docker compose pull

# Stop and recreate containers (migrations run automatically on start)
docker compose down && docker compose up -d

# Watch logs for migration progress
docker compose logs -f server --since 1m

Migrations may take 1–5 minutes depending on database size. Wait until you see:

Starting gunicorn

---

Step 5: Verify

5a. Version Check

docker exec authentik-server ak --version
# Should show TARGET_VERSION

5b. API Token

API tokens are sometimes invalidated during major upgrades. Test:

curl -s -o /dev/null -w "%{http_code}" \
  "https://auth.echo6.co/api/v3/core/applications/" \
  -H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)"

If this returns 403, regenerate the token:

docker exec -i authentik-server ak shell <<'PYEOF'
from authentik.core.models import Token, TokenIntents, User
user = User.objects.get(username="akadmin")
Token.objects.filter(identifier="claude-api-token").delete()
t = Token(identifier="claude-api-token", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
t.save()
print(t.key)
PYEOF

Update /home/zvx/projects/.ref/credentials with the new token.

5c. Email Scope Mapping

Check that the custom email scope (02c22323) is still assigned to providers. The default scope (096b0d6f) may return email_verified: false in 2025.10+:

# Check a canary provider (Forgejo, PK 2)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/2/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  | python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['property_mappings'], indent=2))"

Verify 02c22323-da89-457a-bc12-7f4dd6a3d8ab is in the list. If missing, re-add it to all providers.

5d. Spot-Check OAuth2 Apps

Test SSO login on 2–3 apps as canaries:

1. Forgejo — https://forge.echo6.co → click "Sign in with Authentik"
2. Proxmox — https://proxmox.echo6.co → select OpenID realm

Both should redirect to Authentik, authenticate, and return to the app.

5e. SMTP Delivery

docker exec authentik-server ak test_email matt@echo6.co

Check that the test email arrives. If SMTP auth fails, verify the no-reply@echo6.co mailbox authsource (see Mailcow runbook).

5f. Invitation System

Create a test invitation in Admin UI → Directory → Invitations with email: matt@echo6.co in custom attributes. Confirm the email is sent. Delete the test invitation after.

---

Rollback

If the upgrade breaks critical functionality:

Option A: Roll Back Image (Quick)

cd /opt/authentik

# Revert image tag to previous version
nano docker-compose.yml
# Change TARGET_VERSION back to CURRENT_VERSION

docker compose down && docker compose up -d

This works if no breaking database migrations occurred. Check logs for migration errors.

Option B: Full Restore (Nuclear)

cd /opt

# Stop everything
cd /opt/authentik && docker compose down

# Restore compose directory
rm -rf /opt/authentik
cp -a /opt/authentik.bak_${TIMESTAMP} /opt/authentik

# Restore database
cd /opt/authentik
docker compose up -d postgres
sleep 10

docker exec -i authentik-postgres \
  psql -U authentik -d authentik \
  < /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

docker compose up -d

Post-Rollback

• Verify the old version is running: docker exec authentik-server ak --version
• Test SSO login on Forgejo
• Test SMTP: docker exec authentik-server ak test_email matt@echo6.co

---

Cleanup

After confirming the upgrade is stable (wait at least 24 hours):

# Remove backup
rm -rf /opt/authentik.bak_${TIMESTAMP}

# Keep the SQL dump for archival (or remove if space is needed)
# rm /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

# Prune old Docker images
docker image prune -a --filter "until=168h"

---

Checklist

[ ] Release notes reviewed for all versions between current and target
[ ] PostgreSQL dump taken
[ ] Compose directory backed up
[ ] Compose changes applied (removed/added services, env vars, user directives)
[ ] Image tag updated and pulled
[ ] Containers recreated, migrations completed
[ ] Version confirmed
[ ] API token tested (regenerated if needed)
[ ] Custom email scope verified on providers
[ ] SSO login tested on Forgejo + Proxmox
[ ] SMTP delivery tested
[ ] Invitation system tested
[ ] Backup files cleaned up (after 24h+ stability)

---

Created: 2026-02-16