Matt Johnson
e9231ac24a
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
314 lines
8.1 KiB
Markdown
314 lines
8.1 KiB
Markdown
# Authentik: Major Version Upgrade
|
||
|
||
Upgrade Authentik between major versions on Contabo. Covers backup, upgrade, verification, and rollback.
|
||
|
||
---
|
||
|
||
## When to Use This
|
||
|
||
Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- SSH access to Contabo (`ssh root@100.64.0.1`)
|
||
- Authentik compose directory: `/opt/authentik/`
|
||
- Current version: check with `docker exec authentik-server ak --version`
|
||
|
||
---
|
||
|
||
## Inputs
|
||
|
||
```
|
||
CURRENT_VERSION=2025.12.4 # Current running version
|
||
TARGET_VERSION=2026.2.1 # Version to upgrade to
|
||
```
|
||
|
||
---
|
||
|
||
## Step 1: Check Release Notes
|
||
|
||
Before upgrading, read the release notes for **every major version between current and target**:
|
||
|
||
```
|
||
https://docs.goauthentik.io/docs/releases/
|
||
```
|
||
|
||
Look for:
|
||
- **Breaking changes** — removed features, changed defaults, API changes
|
||
- **Dependency changes** — added/removed services (e.g., Redis removed in 2025.10)
|
||
- **Configuration changes** — new required env vars, changed mount paths
|
||
- **Database migrations** — large migrations that may take time
|
||
|
||
### Known Breaking Changes (Reference)
|
||
|
||
| Version | Change | Impact |
|
||
|---------|--------|--------|
|
||
| 2025.10 | Redis completely removed | Delete redis service + all `AUTHENTIK_REDIS` env vars |
|
||
| 2025.10 | Default email scope returns `email_verified: false` | Use custom scope mapping (PK `02c22323`) that forces `true` |
|
||
| 2025.10 | Worker requires `user: root` | Add `user: root` to worker service in compose |
|
||
| 2025.12 | Stage creation endpoints moved | `stages/<type>/stages/` for POST (some types) |
|
||
|
||
---
|
||
|
||
## Step 2: Backup
|
||
|
||
### 2a. Snapshot Contabo (if Proxmox-managed)
|
||
|
||
If Contabo were a Proxmox VM, take a snapshot. Since it's a bare-metal VPS, skip this and rely on the file-level backups below.
|
||
|
||
### 2b. PostgreSQL Dump
|
||
|
||
```bash
|
||
ssh root@100.64.0.1
|
||
|
||
cd /opt/authentik
|
||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||
|
||
docker exec authentik-postgres \
|
||
pg_dump -U authentik -d authentik \
|
||
--clean --if-exists \
|
||
> /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||
|
||
ls -lh /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||
```
|
||
|
||
### 2c. Compose Directory Backup
|
||
|
||
```bash
|
||
cp -a /opt/authentik /opt/authentik.bak_${TIMESTAMP}
|
||
```
|
||
|
||
This preserves `docker-compose.yml`, `.env`, `certs/`, and any custom files.
|
||
|
||
### 2d. Record Current State
|
||
|
||
```bash
|
||
# Save current version
|
||
docker exec authentik-server ak --version
|
||
|
||
# Save current provider list (for post-upgrade comparison)
|
||
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/" \
|
||
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)" \
|
||
| python3 -c "import sys,json; [print(f'{p[\"pk\"]:3d} {p[\"name\"]}') for p in json.load(sys.stdin)['results']]"
|
||
```
|
||
|
||
---
|
||
|
||
## Step 3: Apply Compose Changes
|
||
|
||
Review release notes and update `docker-compose.yml` **before** pulling the new image:
|
||
|
||
```bash
|
||
cd /opt/authentik
|
||
nano docker-compose.yml
|
||
```
|
||
|
||
Common changes by version:
|
||
|
||
**Removing Redis (2025.10+):**
|
||
```yaml
|
||
# DELETE the redis service entirely
|
||
# DELETE these env vars from server + worker:
|
||
# AUTHENTIK_REDIS__HOST
|
||
# AUTHENTIK_REDIS__PORT
|
||
```
|
||
|
||
**Worker user requirement (2025.10+):**
|
||
```yaml
|
||
services:
|
||
worker:
|
||
user: root # ADD this line
|
||
```
|
||
|
||
**New env vars:**
|
||
Check release notes for any new required `AUTHENTIK_*` env vars. Add to both server and worker services.
|
||
|
||
---
|
||
|
||
## Step 4: Upgrade
|
||
|
||
```bash
|
||
cd /opt/authentik
|
||
|
||
# Update image tag in docker-compose.yml
|
||
# Change: image: ghcr.io/goauthentik/server:CURRENT_VERSION
|
||
# To: image: ghcr.io/goauthentik/server:TARGET_VERSION
|
||
nano docker-compose.yml
|
||
|
||
# Pull new image
|
||
docker compose pull
|
||
|
||
# Stop and recreate containers (migrations run automatically on start)
|
||
docker compose down && docker compose up -d
|
||
|
||
# Watch logs for migration progress
|
||
docker compose logs -f server --since 1m
|
||
```
|
||
|
||
Migrations may take 1–5 minutes depending on database size. Wait until you see:
|
||
```
|
||
Starting gunicorn
|
||
```
|
||
|
||
---
|
||
|
||
## Step 5: Verify
|
||
|
||
### 5a. Version Check
|
||
|
||
```bash
|
||
docker exec authentik-server ak --version
|
||
# Should show TARGET_VERSION
|
||
```
|
||
|
||
### 5b. API Token
|
||
|
||
API tokens are sometimes invalidated during major upgrades. Test:
|
||
|
||
```bash
|
||
curl -s -o /dev/null -w "%{http_code}" \
|
||
"https://auth.echo6.co/api/v3/core/applications/" \
|
||
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)"
|
||
```
|
||
|
||
If this returns `403`, regenerate the token:
|
||
|
||
```bash
|
||
docker exec -i authentik-server ak shell <<'PYEOF'
|
||
from authentik.core.models import Token, TokenIntents, User
|
||
user = User.objects.get(username="akadmin")
|
||
Token.objects.filter(identifier="claude-api-token").delete()
|
||
t = Token(identifier="claude-api-token", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
|
||
t.save()
|
||
print(t.key)
|
||
PYEOF
|
||
```
|
||
|
||
Update `/home/zvx/projects/.ref/credentials` with the new token.
|
||
|
||
### 5c. Email Scope Mapping
|
||
|
||
Check that the custom email scope (`02c22323`) is still assigned to providers. The default scope (`096b0d6f`) may return `email_verified: false` in 2025.10+:
|
||
|
||
```bash
|
||
# Check a canary provider (Forgejo, PK 2)
|
||
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/2/" \
|
||
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
|
||
| python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['property_mappings'], indent=2))"
|
||
```
|
||
|
||
Verify `02c22323-da89-457a-bc12-7f4dd6a3d8ab` is in the list. If missing, re-add it to all providers.
|
||
|
||
### 5d. Spot-Check OAuth2 Apps
|
||
|
||
Test SSO login on 2–3 apps as canaries:
|
||
|
||
1. **Forgejo** — `https://forge.echo6.co` → click "Sign in with Authentik"
|
||
2. **Proxmox** — `https://proxmox.echo6.co` → select OpenID realm
|
||
|
||
Both should redirect to Authentik, authenticate, and return to the app.
|
||
|
||
### 5e. SMTP Delivery
|
||
|
||
```bash
|
||
docker exec authentik-server ak test_email matt@echo6.co
|
||
```
|
||
|
||
Check that the test email arrives. If SMTP auth fails, verify the no-reply@echo6.co mailbox authsource (see Mailcow runbook).
|
||
|
||
### 5f. Invitation System
|
||
|
||
Create a test invitation in Admin UI → Directory → Invitations with `email: matt@echo6.co` in custom attributes. Confirm the email is sent. Delete the test invitation after.
|
||
|
||
---
|
||
|
||
## Rollback
|
||
|
||
If the upgrade breaks critical functionality:
|
||
|
||
### Option A: Roll Back Image (Quick)
|
||
|
||
```bash
|
||
cd /opt/authentik
|
||
|
||
# Revert image tag to previous version
|
||
nano docker-compose.yml
|
||
# Change TARGET_VERSION back to CURRENT_VERSION
|
||
|
||
docker compose down && docker compose up -d
|
||
```
|
||
|
||
This works if no breaking database migrations occurred. Check logs for migration errors.
|
||
|
||
### Option B: Full Restore (Nuclear)
|
||
|
||
```bash
|
||
cd /opt
|
||
|
||
# Stop everything
|
||
cd /opt/authentik && docker compose down
|
||
|
||
# Restore compose directory
|
||
rm -rf /opt/authentik
|
||
cp -a /opt/authentik.bak_${TIMESTAMP} /opt/authentik
|
||
|
||
# Restore database
|
||
cd /opt/authentik
|
||
docker compose up -d postgres
|
||
sleep 10
|
||
|
||
docker exec -i authentik-postgres \
|
||
psql -U authentik -d authentik \
|
||
< /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||
|
||
docker compose up -d
|
||
```
|
||
|
||
### Post-Rollback
|
||
|
||
- Verify the old version is running: `docker exec authentik-server ak --version`
|
||
- Test SSO login on Forgejo
|
||
- Test SMTP: `docker exec authentik-server ak test_email matt@echo6.co`
|
||
|
||
---
|
||
|
||
## Cleanup
|
||
|
||
After confirming the upgrade is stable (wait at least 24 hours):
|
||
|
||
```bash
|
||
# Remove backup
|
||
rm -rf /opt/authentik.bak_${TIMESTAMP}
|
||
|
||
# Keep the SQL dump for archival (or remove if space is needed)
|
||
# rm /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||
|
||
# Prune old Docker images
|
||
docker image prune -a --filter "until=168h"
|
||
```
|
||
|
||
---
|
||
|
||
## Checklist
|
||
|
||
```
|
||
[ ] Release notes reviewed for all versions between current and target
|
||
[ ] PostgreSQL dump taken
|
||
[ ] Compose directory backed up
|
||
[ ] Compose changes applied (removed/added services, env vars, user directives)
|
||
[ ] Image tag updated and pulled
|
||
[ ] Containers recreated, migrations completed
|
||
[ ] Version confirmed
|
||
[ ] API token tested (regenerated if needed)
|
||
[ ] Custom email scope verified on providers
|
||
[ ] SSO login tested on Forgejo + Proxmox
|
||
[ ] SMTP delivery tested
|
||
[ ] Invitation system tested
|
||
[ ] Backup files cleaned up (after 24h+ stability)
|
||
```
|
||
|
||
---
|
||
|
||
*Created: 2026-02-16*
|