echo6-docs/runbooks/authentik-upgrade.md

# Authentik: Major Version Upgrade

Upgrade Authentik between major versions on Contabo. Covers backup, upgrade, verification, and rollback.

---

## When to Use This

Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.

---

## Prerequisites

- SSH access to Contabo (`ssh root@100.64.0.1`)
- Authentik compose directory: `/opt/authentik/`
- Current version: check with `docker exec authentik-server ak --version`

---

## Inputs

```
CURRENT_VERSION=2025.12.4    # Current running version
TARGET_VERSION=2026.2.1      # Version to upgrade to
```

---

## Step 1: Check Release Notes

Before upgrading, read the release notes for **every major version between current and target**:

```
https://docs.goauthentik.io/docs/releases/
```

Look for:
- **Breaking changes** — removed features, changed defaults, API changes
- **Dependency changes** — added/removed services (e.g., Redis removed in 2025.10)
- **Configuration changes** — new required env vars, changed mount paths
- **Database migrations** — large migrations that may take time

### Known Breaking Changes (Reference)

| Version | Change | Impact |
|---------|--------|--------|
| 2025.10 | Redis completely removed | Delete redis service + all `AUTHENTIK_REDIS` env vars |
| 2025.10 | Default email scope returns `email_verified: false` | Use custom scope mapping (PK `02c22323`) that forces `true` |
| 2025.10 | Worker requires `user: root` | Add `user: root` to worker service in compose |
| 2025.12 | Stage creation endpoints moved | `stages/<type>/stages/` for POST (some types) |

---

## Step 2: Backup

### 2a. Snapshot Contabo (if Proxmox-managed)

If Contabo were a Proxmox VM, take a snapshot. Since it's a bare-metal VPS, skip this and rely on the file-level backups below.

### 2b. PostgreSQL Dump

```bash
ssh root@100.64.0.1

cd /opt/authentik
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

docker exec authentik-postgres \
  pg_dump -U authentik -d authentik \
  --clean --if-exists \
  > /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

ls -lh /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
```

### 2c. Compose Directory Backup

```bash
cp -a /opt/authentik /opt/authentik.bak_${TIMESTAMP}
```

This preserves `docker-compose.yml`, `.env`, `certs/`, and any custom files.

### 2d. Record Current State

```bash
# Save current version
docker exec authentik-server ak --version

# Save current provider list (for post-upgrade comparison)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/" \
  -H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)" \
  | python3 -c "import sys,json; [print(f'{p[\"pk\"]:3d} {p[\"name\"]}') for p in json.load(sys.stdin)['results']]"
```

---

## Step 3: Apply Compose Changes

Review release notes and update `docker-compose.yml` **before** pulling the new image:

```bash
cd /opt/authentik
nano docker-compose.yml
```

Common changes by version:

**Removing Redis (2025.10+):**
```yaml
# DELETE the redis service entirely
# DELETE these env vars from server + worker:
#   AUTHENTIK_REDIS__HOST
#   AUTHENTIK_REDIS__PORT
```

**Worker user requirement (2025.10+):**
```yaml
services:
  worker:
    user: root   # ADD this line
```

**New env vars:**
Check release notes for any new required `AUTHENTIK_*` env vars. Add to both server and worker services.

---

## Step 4: Upgrade

```bash
cd /opt/authentik

# Update image tag in docker-compose.yml
# Change: image: ghcr.io/goauthentik/server:CURRENT_VERSION
# To:     image: ghcr.io/goauthentik/server:TARGET_VERSION
nano docker-compose.yml

# Pull new image
docker compose pull

# Stop and recreate containers (migrations run automatically on start)
docker compose down && docker compose up -d

# Watch logs for migration progress
docker compose logs -f server --since 1m
```

Migrations may take 1–5 minutes depending on database size. Wait until you see:
```
Starting gunicorn
```

---

## Step 5: Verify

### 5a. Version Check

```bash
docker exec authentik-server ak --version
# Should show TARGET_VERSION
```

### 5b. API Token

API tokens are sometimes invalidated during major upgrades. Test:

```bash
curl -s -o /dev/null -w "%{http_code}" \
  "https://auth.echo6.co/api/v3/core/applications/" \
  -H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)"
```

If this returns `403`, regenerate the token:

```bash
docker exec -i authentik-server ak shell <<'PYEOF'
from authentik.core.models import Token, TokenIntents, User
user = User.objects.get(username="akadmin")
Token.objects.filter(identifier="claude-api-token").delete()
t = Token(identifier="claude-api-token", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
t.save()
print(t.key)
PYEOF
```

Update `/home/zvx/projects/.ref/credentials` with the new token.

### 5c. Email Scope Mapping

Check that the custom email scope (`02c22323`) is still assigned to providers. The default scope (`096b0d6f`) may return `email_verified: false` in 2025.10+:

```bash
# Check a canary provider (Forgejo, PK 2)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/2/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  | python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['property_mappings'], indent=2))"
```

Verify `02c22323-da89-457a-bc12-7f4dd6a3d8ab` is in the list. If missing, re-add it to all providers.

### 5d. Spot-Check OAuth2 Apps

Test SSO login on 2–3 apps as canaries:

1. **Forgejo** — `https://forge.echo6.co` → click "Sign in with Authentik"
2. **Proxmox** — `https://proxmox.echo6.co` → select OpenID realm

Both should redirect to Authentik, authenticate, and return to the app.

### 5e. SMTP Delivery

```bash
docker exec authentik-server ak test_email matt@echo6.co
```

Check that the test email arrives. If SMTP auth fails, verify the no-reply@echo6.co mailbox authsource (see Mailcow runbook).

### 5f. Invitation System

Create a test invitation in Admin UI → Directory → Invitations with `email: matt@echo6.co` in custom attributes. Confirm the email is sent. Delete the test invitation after.

---

## Rollback

If the upgrade breaks critical functionality:

### Option A: Roll Back Image (Quick)

```bash
cd /opt/authentik

# Revert image tag to previous version
nano docker-compose.yml
# Change TARGET_VERSION back to CURRENT_VERSION

docker compose down && docker compose up -d
```

This works if no breaking database migrations occurred. Check logs for migration errors.

### Option B: Full Restore (Nuclear)

```bash
cd /opt

# Stop everything
cd /opt/authentik && docker compose down

# Restore compose directory
rm -rf /opt/authentik
cp -a /opt/authentik.bak_${TIMESTAMP} /opt/authentik

# Restore database
cd /opt/authentik
docker compose up -d postgres
sleep 10

docker exec -i authentik-postgres \
  psql -U authentik -d authentik \
  < /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

docker compose up -d
```

### Post-Rollback

- Verify the old version is running: `docker exec authentik-server ak --version`
- Test SSO login on Forgejo
- Test SMTP: `docker exec authentik-server ak test_email matt@echo6.co`

---

## Cleanup

After confirming the upgrade is stable (wait at least 24 hours):

```bash
# Remove backup
rm -rf /opt/authentik.bak_${TIMESTAMP}

# Keep the SQL dump for archival (or remove if space is needed)
# rm /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql

# Prune old Docker images
docker image prune -a --filter "until=168h"
```

---

## Checklist

```
[ ] Release notes reviewed for all versions between current and target
[ ] PostgreSQL dump taken
[ ] Compose directory backed up
[ ] Compose changes applied (removed/added services, env vars, user directives)
[ ] Image tag updated and pulled
[ ] Containers recreated, migrations completed
[ ] Version confirmed
[ ] API token tested (regenerated if needed)
[ ] Custom email scope verified on providers
[ ] SSO login tested on Forgejo + Proxmox
[ ] SMTP delivery tested
[ ] Invitation system tested
[ ] Backup files cleaned up (after 24h+ stability)
```

---

*Created: 2026-02-16*