Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
89834796ff
commit
e9231ac24a
93 changed files with 51223 additions and 254 deletions
314
runbooks/authentik-upgrade.md
Normal file
314
runbooks/authentik-upgrade.md
Normal file
|
|
@ -0,0 +1,314 @@
|
|||
# Authentik: Major Version Upgrade
|
||||
|
||||
Upgrade Authentik between major versions on Contabo. Covers backup, upgrade, verification, and rollback.
|
||||
|
||||
---
|
||||
|
||||
## When to Use This
|
||||
|
||||
Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SSH access to Contabo (`ssh root@100.64.0.1`)
|
||||
- Authentik compose directory: `/opt/authentik/`
|
||||
- Current version: check with `docker exec authentik-server ak --version`
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
```
|
||||
CURRENT_VERSION=2025.12.4 # Current running version
|
||||
TARGET_VERSION=2026.2.1 # Version to upgrade to
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Check Release Notes
|
||||
|
||||
Before upgrading, read the release notes for **every major version between current and target**:
|
||||
|
||||
```
|
||||
https://docs.goauthentik.io/docs/releases/
|
||||
```
|
||||
|
||||
Look for:
|
||||
- **Breaking changes** — removed features, changed defaults, API changes
|
||||
- **Dependency changes** — added/removed services (e.g., Redis removed in 2025.10)
|
||||
- **Configuration changes** — new required env vars, changed mount paths
|
||||
- **Database migrations** — large migrations that may take time
|
||||
|
||||
### Known Breaking Changes (Reference)
|
||||
|
||||
| Version | Change | Impact |
|
||||
|---------|--------|--------|
|
||||
| 2025.10 | Redis completely removed | Delete redis service + all `AUTHENTIK_REDIS` env vars |
|
||||
| 2025.10 | Default email scope returns `email_verified: false` | Use custom scope mapping (PK `02c22323`) that forces `true` |
|
||||
| 2025.10 | Worker requires `user: root` | Add `user: root` to worker service in compose |
|
||||
| 2025.12 | Stage creation endpoints moved | `stages/<type>/stages/` for POST (some types) |
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Backup
|
||||
|
||||
### 2a. Snapshot Contabo (if Proxmox-managed)
|
||||
|
||||
If Contabo were a Proxmox VM, take a snapshot. Since it's a bare-metal VPS, skip this and rely on the file-level backups below.
|
||||
|
||||
### 2b. PostgreSQL Dump
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1
|
||||
|
||||
cd /opt/authentik
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
docker exec authentik-postgres \
|
||||
pg_dump -U authentik -d authentik \
|
||||
--clean --if-exists \
|
||||
> /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||||
|
||||
ls -lh /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||||
```
|
||||
|
||||
### 2c. Compose Directory Backup
|
||||
|
||||
```bash
|
||||
cp -a /opt/authentik /opt/authentik.bak_${TIMESTAMP}
|
||||
```
|
||||
|
||||
This preserves `docker-compose.yml`, `.env`, `certs/`, and any custom files.
|
||||
|
||||
### 2d. Record Current State
|
||||
|
||||
```bash
|
||||
# Save current version
|
||||
docker exec authentik-server ak --version
|
||||
|
||||
# Save current provider list (for post-upgrade comparison)
|
||||
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/" \
|
||||
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)" \
|
||||
| python3 -c "import sys,json; [print(f'{p[\"pk\"]:3d} {p[\"name\"]}') for p in json.load(sys.stdin)['results']]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Apply Compose Changes
|
||||
|
||||
Review release notes and update `docker-compose.yml` **before** pulling the new image:
|
||||
|
||||
```bash
|
||||
cd /opt/authentik
|
||||
nano docker-compose.yml
|
||||
```
|
||||
|
||||
Common changes by version:
|
||||
|
||||
**Removing Redis (2025.10+):**
|
||||
```yaml
|
||||
# DELETE the redis service entirely
|
||||
# DELETE these env vars from server + worker:
|
||||
# AUTHENTIK_REDIS__HOST
|
||||
# AUTHENTIK_REDIS__PORT
|
||||
```
|
||||
|
||||
**Worker user requirement (2025.10+):**
|
||||
```yaml
|
||||
services:
|
||||
worker:
|
||||
user: root # ADD this line
|
||||
```
|
||||
|
||||
**New env vars:**
|
||||
Check release notes for any new required `AUTHENTIK_*` env vars. Add to both server and worker services.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Upgrade
|
||||
|
||||
```bash
|
||||
cd /opt/authentik
|
||||
|
||||
# Update image tag in docker-compose.yml
|
||||
# Change: image: ghcr.io/goauthentik/server:CURRENT_VERSION
|
||||
# To: image: ghcr.io/goauthentik/server:TARGET_VERSION
|
||||
nano docker-compose.yml
|
||||
|
||||
# Pull new image
|
||||
docker compose pull
|
||||
|
||||
# Stop and recreate containers (migrations run automatically on start)
|
||||
docker compose down && docker compose up -d
|
||||
|
||||
# Watch logs for migration progress
|
||||
docker compose logs -f server --since 1m
|
||||
```
|
||||
|
||||
Migrations may take 1–5 minutes depending on database size. Wait until you see:
|
||||
```
|
||||
Starting gunicorn
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Verify
|
||||
|
||||
### 5a. Version Check
|
||||
|
||||
```bash
|
||||
docker exec authentik-server ak --version
|
||||
# Should show TARGET_VERSION
|
||||
```
|
||||
|
||||
### 5b. API Token
|
||||
|
||||
API tokens are sometimes invalidated during major upgrades. Test:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" \
|
||||
"https://auth.echo6.co/api/v3/core/applications/" \
|
||||
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)"
|
||||
```
|
||||
|
||||
If this returns `403`, regenerate the token:
|
||||
|
||||
```bash
|
||||
docker exec -i authentik-server ak shell <<'PYEOF'
|
||||
from authentik.core.models import Token, TokenIntents, User
|
||||
user = User.objects.get(username="akadmin")
|
||||
Token.objects.filter(identifier="claude-api-token").delete()
|
||||
t = Token(identifier="claude-api-token", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
|
||||
t.save()
|
||||
print(t.key)
|
||||
PYEOF
|
||||
```
|
||||
|
||||
Update `/home/zvx/projects/.ref/credentials` with the new token.
|
||||
|
||||
### 5c. Email Scope Mapping
|
||||
|
||||
Check that the custom email scope (`02c22323`) is still assigned to providers. The default scope (`096b0d6f`) may return `email_verified: false` in 2025.10+:
|
||||
|
||||
```bash
|
||||
# Check a canary provider (Forgejo, PK 2)
|
||||
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/2/" \
|
||||
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
|
||||
| python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['property_mappings'], indent=2))"
|
||||
```
|
||||
|
||||
Verify `02c22323-da89-457a-bc12-7f4dd6a3d8ab` is in the list. If missing, re-add it to all providers.
|
||||
|
||||
### 5d. Spot-Check OAuth2 Apps
|
||||
|
||||
Test SSO login on 2–3 apps as canaries:
|
||||
|
||||
1. **Forgejo** — `https://forge.echo6.co` → click "Sign in with Authentik"
|
||||
2. **Proxmox** — `https://proxmox.echo6.co` → select OpenID realm
|
||||
|
||||
Both should redirect to Authentik, authenticate, and return to the app.
|
||||
|
||||
### 5e. SMTP Delivery
|
||||
|
||||
```bash
|
||||
docker exec authentik-server ak test_email matt@echo6.co
|
||||
```
|
||||
|
||||
Check that the test email arrives. If SMTP auth fails, verify the no-reply@echo6.co mailbox authsource (see Mailcow runbook).
|
||||
|
||||
### 5f. Invitation System
|
||||
|
||||
Create a test invitation in Admin UI → Directory → Invitations with `email: matt@echo6.co` in custom attributes. Confirm the email is sent. Delete the test invitation after.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the upgrade breaks critical functionality:
|
||||
|
||||
### Option A: Roll Back Image (Quick)
|
||||
|
||||
```bash
|
||||
cd /opt/authentik
|
||||
|
||||
# Revert image tag to previous version
|
||||
nano docker-compose.yml
|
||||
# Change TARGET_VERSION back to CURRENT_VERSION
|
||||
|
||||
docker compose down && docker compose up -d
|
||||
```
|
||||
|
||||
This works if no breaking database migrations occurred. Check logs for migration errors.
|
||||
|
||||
### Option B: Full Restore (Nuclear)
|
||||
|
||||
```bash
|
||||
cd /opt
|
||||
|
||||
# Stop everything
|
||||
cd /opt/authentik && docker compose down
|
||||
|
||||
# Restore compose directory
|
||||
rm -rf /opt/authentik
|
||||
cp -a /opt/authentik.bak_${TIMESTAMP} /opt/authentik
|
||||
|
||||
# Restore database
|
||||
cd /opt/authentik
|
||||
docker compose up -d postgres
|
||||
sleep 10
|
||||
|
||||
docker exec -i authentik-postgres \
|
||||
psql -U authentik -d authentik \
|
||||
< /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||||
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Post-Rollback
|
||||
|
||||
- Verify the old version is running: `docker exec authentik-server ak --version`
|
||||
- Test SSO login on Forgejo
|
||||
- Test SMTP: `docker exec authentik-server ak test_email matt@echo6.co`
|
||||
|
||||
---
|
||||
|
||||
## Cleanup
|
||||
|
||||
After confirming the upgrade is stable (wait at least 24 hours):
|
||||
|
||||
```bash
|
||||
# Remove backup
|
||||
rm -rf /opt/authentik.bak_${TIMESTAMP}
|
||||
|
||||
# Keep the SQL dump for archival (or remove if space is needed)
|
||||
# rm /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
|
||||
|
||||
# Prune old Docker images
|
||||
docker image prune -a --filter "until=168h"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
```
|
||||
[ ] Release notes reviewed for all versions between current and target
|
||||
[ ] PostgreSQL dump taken
|
||||
[ ] Compose directory backed up
|
||||
[ ] Compose changes applied (removed/added services, env vars, user directives)
|
||||
[ ] Image tag updated and pulled
|
||||
[ ] Containers recreated, migrations completed
|
||||
[ ] Version confirmed
|
||||
[ ] API token tested (regenerated if needed)
|
||||
[ ] Custom email scope verified on providers
|
||||
[ ] SSO login tested on Forgejo + Proxmox
|
||||
[ ] SMTP delivery tested
|
||||
[ ] Invitation system tested
|
||||
[ ] Backup files cleaned up (after 24h+ stability)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Created: 2026-02-16*
|
||||
Loading…
Add table
Add a link
Reference in a new issue