matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

Browse source 
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup)
- Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing
- Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack
- Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure
- Removes 4 deprecated runbook duplicates (canonical versions live in projects/)
- Adds .gitignore for binary archives and editor temp files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-04-13 06:02:16 +00:00
commit e9231ac24a
93 changed files with 51223 additions and 254 deletions
Download patch file Download diff file Expand all files Collapse all files

338
docs/software/authentik.md
View file

@ -2,7 +2,7 @@
## Location
- **Server:** Contabo (5.189.158.149 / 100.64.0.6)
- **Server:** Contabo (5.189.158.149 / 100.64.0.1)
- **URL:** https://auth.echo6.co
- **Internal Port:** 9000
@ -10,6 +10,12 @@
API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN`
```bash
# Test API access
curl -s "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" | python3 -m json.tool
```
## Flow UUIDs
Required for OAuth2 provider creation:
@ -21,28 +27,88 @@ Required for OAuth2 provider creation:
| Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` |
| Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` |
## Signing Key
| Key | UUID |
|-----|------|
| authentik Self-signed Certificate | `09f508f0-6b8e-4031-8563-0d3cebf86868` |
## Property Mappings (Standard OIDC Set)
Include these three in every provider:
| Mapping | UUID |
|---------|------|
| OpenID Connect scope: openid | `c6426dad-0d85-4daa-89fe-cb850ad4bfd7` |
| Echo6 OAuth: email (verified=true) | `02c22323-da89-457a-bc12-7f4dd6a3d8ab` |
| OpenID Connect scope: profile | `113ab791-fa04-4e8c-b103-09d706bc21b4` |
**Note:** The default email scope (`096b0d6f`) was changed in 2025.10 to return `email_verified: false`. Our custom scope (`02c22323`) overrides this to always return `true`. All 14 OAuth2 providers were migrated to use the custom scope on 2026-02-16.
## Current OAuth2 Providers
| PK | Name | Client ID | Application | Redirect URI |
|----|------|-----------|-------------|--------------|
| 1 | Mailcow | mailcow | Mailcow | `https://mail.echo6.co/sso/oidc` |
| 2 | Forgejo | forgejo | Forgejo | `https://forge.echo6.co/user/oauth2/Authentik/callback` |
| 3 | Vaultwarden | vaultwarden | Vaultwarden | `https://vault.echo6.co/identity/connect/callback` |
| 4 | Proxmox | proxmox | Proxmox VE | `https://proxmox.echo6.co` (regex) |
| 5 | Headscale | headscale | Headscale VPN | `https://vpn.echo6.co/oidc/callback` |
| 6 | Headplane | headplane | Headplane | `https://vpn.echo6.co/admin/oidc/callback` |
| 8 | Nextcloud | nextcloud | Nextcloud | `https://nextcloud.echo6.co/apps/oidc_login/oidc` |
| 9 | Immich | immich | Immich | `https://immich.echo6.co/auth/login`, `/api/oauth/mobile-redirect` |
| 10 | jellyfin | jellyfin | Jellyfin | `https://jellyfin.echo6.co/sso/OID/redirect/Authentik` |
| 11 | jellyseer | jellyseer | Jellyseer | `https://requests.echo6.co/(login\|api/v1/auth/oidc-callback).*` (regex) |
| 12 | PeerTube | peertube | PeerTube | `https://stream.echo6.co/plugins/auth-openid-connect/...` |
| 13 | WATCHTOWER | watchtower | WATCHTOWER | Forward auth (proxy provider) |
| 14 | Open WebUI | open-webui | Open WebUI | `https://ai.echo6.co/oauth/oidc/callback` |
| 15 | Matrix | 93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T | Matrix | `https://matrix.echo6.co/_synapse/client/oidc/callback` |
| 16 | Files Forward Auth | — | Files | Forward auth (proxy provider) |
| 17 | LiveSync Provisioner | ZBoLdYmxlSUyMqgekswIPS4YuaeBn5uCr8GtWm5H | LiveSync | Forward auth (proxy provider) |
## Groups
| Name | PK | Superuser | Members | Used By |
|------|----|-----------|---------|----- ---|
| authentik Admins | `9944e153-f860-4443-81d1-ae544f611806` | Yes | akadmin, matt | All apps (admin access) |
| media-users | `0820b2b8-6c54-4c20-9a0a-872820e6d9ea` | No | jodie, matt | Jellyfin, Jellyseer, PeerTube |
| ai-users | `0631b273-cfd8-4ed1-afa6-e262d0dc5a69` | No | matt | Open WebUI |
| cloud-users | `db3cbf5d-8057-4e33-8e8d-95bfdb35fbac` | No | — | Immich, Nextcloud |
| communication-users | `31bce176-cd86-4aea-8db3-a57e03d5c2d1` | No | — | Mailcow, Matrix |
| productivity-users | `698d80c7-7c29-43cd-b5d4-9eb24c85a6cc` | No | — | — |
| security-users | `f345a043-c2a4-4906-a43b-9860eae86ee1` | No | — | — |
| proxmox_admins | `d85a868d-7d1e-4585-92a8-b8bb86771b53` | No | akadmin, matt | Proxmox VE |
| proxmox_users | `cf26703a-a824-47dd-9550-30b848a8ce5f` | No | — | Proxmox VE |
| authentik Read-only | `ce03664b-46f3-43b7-9967-5f65d591fdb6` | No | — | RBAC read-only role |
| livesync-users | `8e575a86-326e-4df8-8828-8379d8ab861f` | No | matt | LiveSync |
## Application Access Pattern
Every application uses `policy_engine_mode: "any"` with two group bindings:
1. **authentik Admins** — gives admin/superuser access to all apps
2. **Service-specific group** — controls which regular users can access the app
Users must be in at least one bound group to access the application.
## Create New API Token
```bash
ssh root@100.64.0.6 'docker exec authentik-server ak shell -c "
from authentik.core.models import Token, User
user = User.objects.get(username=\"akadmin\")
token, created = Token.objects.get_or_create(
identifier=\"token-name\",
user=user,
defaults={\"intent\": \"api\", \"expiring\": False}
)
print(token.key)
"'
ssh root@100.64.0.1 'docker exec -i authentik-server ak shell' <<'PYEOF'
from authentik.core.models import Token, TokenIntents, User
user = User.objects.get(username="akadmin")
Token.objects.filter(identifier="token-name").delete()
t = Token(identifier="token-name", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
t.save()
print(t.key)
PYEOF
```
## Quick OAuth2 Provider Creation
## Full OAuth2 App Setup (Provider + Application + Group + Bindings)
### Step 1: Create the OAuth2 provider
```bash
# Source credentials
source /home/zvx/projects/.ref/credentials
# Create provider
curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
@ -50,13 +116,25 @@ curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
"name": "AppName",
"authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd",
"invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a",
"property_mappings": [
"c6426dad-0d85-4daa-89fe-cb850ad4bfd7",
"02c22323-da89-457a-bc12-7f4dd6a3d8ab",
"113ab791-fa04-4e8c-b103-09d706bc21b4"
],
"client_type": "confidential",
"client_id": "appname",
"client_secret": "<generate-a-long-random-secret>",
"redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}],
"sub_mode": "user_username"
"sub_mode": "user_username",
"signing_key": "09f508f0-6b8e-4031-8563-0d3cebf86868",
"include_claims_in_id_token": true
}'
# Note the "pk" from the response — needed for Step 2
```
# Create application (use pk from provider response)
### Step 2: Create the application
```bash
curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
@ -64,14 +142,232 @@ curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
"name": "AppName",
"slug": "appname",
"provider": PROVIDER_PK,
"meta_launch_url": "https://app.echo6.co"
"meta_launch_url": "https://app.echo6.co",
"policy_engine_mode": "any"
}'
# Note the "pk" from the response — needed for Step 4
```
### Step 3: Create the user group
```bash
curl -s -X POST "https://auth.echo6.co/api/v3/core/groups/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "appname-users",
"is_superuser": false,
"users": [7]
}'
# user 7 = matt. Note the "pk" from the response.
```
### Step 4: Bind groups to the application
```bash
# Bind authentik Admins
curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"target": "APPLICATION_PK",
"group": "9944e153-f860-4443-81d1-ae544f611806",
"order": 0, "enabled": true, "negate": false, "timeout": 30
}'
# Bind service-specific group
curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"target": "APPLICATION_PK",
"group": "GROUP_PK",
"order": 0, "enabled": true, "negate": false, "timeout": 30
}'
```
### Step 5: Verify
```bash
# OIDC discovery should return endpoints
curl -s "https://auth.echo6.co/application/o/appname/.well-known/openid-configuration" | python3 -m json.tool
# Check bindings
curl -s "https://auth.echo6.co/api/v3/policies/bindings/?target=APPLICATION_PK" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN"
```
## Common Redirect URI Patterns
| Application Type | Redirect URI Pattern |
|------------------|---------------------|
| Web app | `https://app.echo6.co/callback` |
| Web app (oauth) | `https://app.echo6.co/oauth/callback` |
| Web app (generic) | `https://app.echo6.co/callback` |
| Web app (oauth path) | `https://app.echo6.co/oauth/callback` |
| Open WebUI (OIDC) | `https://app.echo6.co/oauth/oidc/callback` |
| Forgejo | `https://app.echo6.co/user/oauth2/Authentik/callback` |
| Jellyfin (SSO plugin) | `https://app.echo6.co/sso/OID/redirect/Authentik` |
| Caddy forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` |
## Users
| PK | Username | Name | Email |
|----|----------|------|-------|
| 6 | akadmin | authentik Default Admin | root@example.com |
| 7 | matt | Matt Johnson | matt@echo6.co |
| 9 | jodie | Jodie | johnsonsinidaho@gmail.com |
## Email Invitation System
### SMTP Configuration
Authentik sends email via Mailcow:
| Setting | Value |
|---------|-------|
| SMTP Host | mail.echo6.co |
| SMTP Port | 587 (STARTTLS) |
| Username | no-reply@echo6.co |
| From | no-reply@echo6.co |
**Important:** The no-reply@echo6.co mailbox MUST have `authsource=mailcow` in the Mailcow database (not `generic-oidc`). If it gets reset to `generic-oidc`, SMTP auth will fail because Dovecot tries to authenticate via Authentik SSO instead of the local password. Fix with:
```sql
docker exec mailcowdockerized-mysql-mailcow-1 mysql -u mailcow -p<DBPASS> mailcow \
-e "UPDATE mailbox SET authsource='mailcow' WHERE username='no-reply@echo6.co'"
```
### Enrollment Flow
| Component | PK |
|-----------|-----|
| Flow (invitation-enrollment) | `184a9e20-f5c5-4b44-8775-266a568439c0` |
| Invitation stage (enrollment-invitation) | `994252a6-a659-4304-b9aa-a6591857a53b` |
| Prompt stage (enrollment-credentials) | `a5e2a2f1-95c9-4627-a3dd-e0915ff64cda` |
| User write stage (enrollment-user-write) | `05f565e0-9d64-4ecc-b738-8ffc697ad829` |
| User login stage (enrollment-user-login) | `f23958ea-47bd-4fb2-a657-b6f85fc6331a` |
Stage order: Invitation (10) → Prompt (20) → User Write (30) → User Login (40)
Flow settings: `authentication=require_unauthenticated`, `continue_flow_without_invitation=false`
User write creates users under `users/enrolled` path.
### Email Automation
Uses the community pattern from [authentik/discussions/13305](https://github.com/goauthentik/authentik/discussions/13305):
| Component | PK |
|-----------|-----|
| Expression policy (invitation-email-sender) | `770188de-2b60-4e87-820a-fa64d419ed89` |
| Notification rule (invitation-email-trigger) | `a145a8c9-b3eb-440d-8b77-27139c346a17` |
When an invitation is created with an `email` field in custom attributes, the expression policy triggers `ak_send_email()` to send the enrollment link to the invitee automatically.
### How to Use
**Invite via email** (Admin UI → Directory → Invitations → Create):
1. Name it, select **Invitation Enrollment** flow, toggle **Single use** on, set expiry
2. Custom attributes:
```yaml
name: Jane Smith
email: jane@example.com
```
3. Click Create — email is sent automatically
**Invite via link** (no email):
1. Same as above but leave custom attributes empty (or omit `email` field)
2. Click Create → expand the row → copy the invitation link
3. Send the link manually
---
## Branding & Theming
Echo6 cyberpunk branding applied to Authentik 2025.12.4 via System → Brands.
### Brand Settings
| Setting | Value |
|---------|-------|
| Brand title | `echo6` |
| Theme | `dark` (forced, not automatic) |
| Logo | `/media/custom/echo6-logo.png` (uploaded via Customization → Files) |
| Favicon | `/media/custom/echo6-favicon.png` (uploaded via Customization → Files) |
| Custom CSS | Echo6 Authentik CSS (~200 rules, applied via Brand → Custom CSS field) |
### Flow Titles
| Flow | Title |
|------|-------|
| Authentication | `echo6 // login` |
| Invalidation | `echo6 // logout` |
| Recovery | `echo6 // recovery` |
| User Settings | `echo6 // settings` |
### Custom CSS Highlights
- **Font:** JetBrains Mono globally (with `:not()` exclusions for FontAwesome/PatternFly icon fonts)
- **Colors:** Cyan `#28C0E8` primary accent, Yellow `#F0D848` secondary, dark backgrounds `#0a0e17`/`#111827`/`#1a2332`
- **Login card:** Dark background, cyan-glow focus on inputs, branded submit button
- **Admin sidebar:** Dark with cyan hover/active states
- **User dashboard:** 3-column grid layout, dark cards with cyan border on hover
- **Application icons:** Custom SVG icons uploaded for all 15 services
### CSS Storage
The custom CSS is stored in the Brand model's `branding_custom_css` field. To update:
```bash
# Copy CSS to Contabo
scp /path/to/echo6-authentik.css root@100.64.0.1:/opt/authentik/branding/custom.css
# Load into Brand model via ak shell
ssh root@100.64.0.1 'docker exec -i authentik-server ak shell' <<'PYEOF'
from authentik.brands.models import Brand
b = Brand.objects.get(domain="auth.echo6.co")
b.branding_custom_css = open("/media/custom/custom.css").read()
b.save()
PYEOF
# Restart to apply
ssh root@100.64.0.1 'cd /opt/authentik && docker compose restart server worker'
```
### SSO Launch URL Pattern
All authenticated service links use Authentik's application launch URL:
```
https://auth.echo6.co/application/launch/<app-slug>/
```
This provides seamless SSO: authenticated users pass through to the app, unauthenticated users get the login page then redirect to the app. Used by the SearXNG waffle menu and nav bar.
| App Slug | Service | Launch URL |
|----------|---------|-----------|
| open-webui | Aurora (AI) | `https://auth.echo6.co/application/launch/open-webui/` |
| peertube | Stream | `https://auth.echo6.co/application/launch/peertube/` |
| files | Files | `https://auth.echo6.co/application/launch/files/` |
| watchtower | Watchtower | `https://auth.echo6.co/application/launch/watchtower/` |
| immich | Photos | `https://auth.echo6.co/application/launch/immich/` |
| mailcow | Mail | `https://auth.echo6.co/application/launch/mailcow/` |
| nextcloud | Cloud | `https://auth.echo6.co/application/launch/nextcloud/` |
| jellyfin | Jellyfin | `https://auth.echo6.co/application/launch/jellyfin/` |
| jellyseer | Requests | `https://auth.echo6.co/application/launch/jellyseer/` |
### Brand Color Reference
| Color | Hex | Usage |
|-------|-----|-------|
| Cyan | `#28C0E8` | Primary accent, links, focus states |
| Cyan Light | `#5DD4F5` | Hover states |
| Yellow | `#F0D848` | Secondary accent, badges |
| BG Primary | `#0a0e17` | Page backgrounds |
| BG Secondary | `#111827` | Cards, inputs |
| BG Tertiary | `#1a2332` | Elevated surfaces |
| Border | `#1e3a5f` | All borders |
| Text Primary | `#e0e6ed` | Main text |
| Text Muted | `#7a8ca0` | Secondary text |
---
*Last updated: 2026-02-18 — Added LiveSync proxy provider (PK 17) + livesync-users group, Files Forward Auth (PK 16). Authentik 2025.12.4*

74
docs/software/caddy.md
View file

@ -4,19 +4,26 @@
**Config:** `/etc/caddy/Caddyfile` on Contabo (ssh root@100.64.0.1)
**Global options:** `email admin@echo6.co`, `admin off` (no live reload — must `systemctl restart caddy`)
### Current Site Blocks
| Domain | Backend | Service |
|--------|---------|---------|
| auth.echo6.co | 127.0.0.1:9000 | Authentik SSO |
| forge.echo6.co | 127.0.0.1:3001 | Forgejo Git |
| mail.echo6.co | https://127.0.0.1:8443 | Mailcow (tls_insecure_skip_verify) |
| mail.echo6.co | https://127.0.0.1:8453 | Mailcow (tls_insecure_skip_verify, r/w timeout 3600s) |
| vpn.echo6.co | 127.0.0.1:8084 | Headscale |
| vpn.echo6.co/admin* | 127.0.0.1:3100 | Headplane |
| autodiscover.echo6.co | https://127.0.0.1:8443 | Mailcow autodiscover |
| autoconfig.echo6.co | https://127.0.0.1:8443 | Mailcow autoconfig |
| vault.echo6.co | 127.0.0.1:8086 | Vaultwarden |
| proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) |
| wt.echo6.co | 127.0.0.1:8099 (Authentik forward auth) | WATCHTOWER ops dashboard |
| matrix.echo6.co | 127.0.0.1:8008 + 127.0.0.1:8085 | Matrix Synapse + MAS (login/logout/refresh/auth_metadata → MAS:8085, _matrix/* → Synapse:8008, default → MAS:8085) |
| element.echo6.co | 127.0.0.1:8088 | Element Web client |
| notes.echo6.co | 127.0.0.1:5984 + 127.0.0.1:5985 | LiveSync (CouchDB + provisioner, forward auth on /_provision*, CORS for Obsidian) |
| tak.echo6.co | https://100.64.0.1:8446 + 100.64.0.1:8990 | TAK Server admin (8446, Authentik forward auth) + SIGIL console (/sigil, 8990) |
### Commands
@ -41,8 +48,21 @@ journalctl -u caddy -f
| Domain | Backend | Pattern | Service |
|--------|---------|---------|---------|
| mesh.echo6.co | 100.64.0.7:8080 | Tailscale | MeshMonitor |
| search.echo6.co | 100.64.0.15:8080 | Tailscale | SearXNG |
| mesh.echo6.co | 192.168.1.100:8080 | Local IP | MeshMonitor (Authentik forward auth) |
| echo6.co | 100.64.0.15:8080 | Tailscale | Echo6 Search (SearXNG) + Matrix well-known |
| search.echo6.co | — | — | 301 redirect to echo6.co |
| nas.echo6.co | 100.64.0.21:80 | Tailscale | OpenMediaVault (pi-nas) |
| immich.echo6.co | 192.168.1.182:2283 | Local IP | Immich (has 2FA) |
| nextcloud.echo6.co | 192.168.1.183:11000 | Local IP | Nextcloud AIO (SSO via Authentik) |
| jellyfin.echo6.co | 100.64.0.18:8096 | Tailscale | Jellyfin media server (SSO via Authentik) |
| requests.echo6.co | 100.64.0.18:5055 | Tailscale | Jellyseer request management (SSO via Authentik) |
| stream.echo6.co | 192.168.1.170:80 | Local IP | PeerTube video streaming (SSO via Authentik) |
| ai.echo6.co | 100.64.0.14:8080 | Tailscale | Open WebUI (SSO via Authentik) |
| files.echo6.co | 100.64.0.24:8888 | Tailscale | RECON PDF library (Authentik forward auth) |
| recon.echo6.co | 100.64.0.24:8420 | Tailscale | RECON dashboard + API |
| lidarr.echo6.co | 100.64.0.18:8686 | Tailscale | Lidarr music automation (Authentik forward auth) |
| navidrome.echo6.co | 100.64.0.18:4533 | Tailscale | Navidrome music server (Authentik forward auth, /rest/* exempt for Subsonic API) |
| vpn.idahomesh.com | 192.168.1.106:8080 | Local IP | IdahoMesh Headscale VPN coordination |
### Commands
@ -68,10 +88,21 @@ ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
| mail.echo6.co | 100.64.0.1 | Mailcow |
| vpn.echo6.co | 100.64.0.1 | Headscale |
| vault.echo6.co | 100.64.0.1 | Vaultwarden |
| docs.echo6.co | 100.64.0.1 | Wiki.js |
| proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) |
| stream.echo6.co | *TBD* | PeerTube - needs host verification |
| notes.echo6.co | *TBD* | Obsidian LiveSync - needs host verification |
| stream.echo6.co | 100.64.0.8 | PeerTube (via utility Caddy) |
| notes.echo6.co | 100.64.0.1 | LiveSync CouchDB + provisioner (via Contabo Caddy) |
| tak.echo6.co | 100.64.0.1 | TAK Server + SIGIL (via Contabo Caddy) |
| jellyfin.echo6.co | 100.64.0.8 | Jellyfin (via utility Caddy) |
| requests.echo6.co | 100.64.0.8 | Jellyseer (via utility Caddy) |
| wt.echo6.co | 100.64.0.1 | WATCHTOWER ops dashboard |
| ai.echo6.co | 100.64.0.8 | Open WebUI (via utility Caddy) |
| matrix.echo6.co | 100.64.0.1 | Matrix Synapse (via Contabo Caddy) |
| element.echo6.co | 100.64.0.1 | Element Web (via Contabo Caddy) |
| echo6.co | 100.64.0.8 | Echo6 Search homepage (via utility Caddy) |
| files.echo6.co | 100.64.0.8 | RECON PDF library (via utility Caddy) |
| recon.echo6.co | 100.64.0.8 | RECON dashboard (via utility Caddy) |
| lidarr.echo6.co | 100.64.0.8 | Lidarr music automation (via utility Caddy) |
| navidrome.echo6.co | 100.64.0.8 | Navidrome music server (via utility Caddy) |
### Commands
@ -95,19 +126,31 @@ dig +short forge.echo6.co @100.64.0.1 # Test
| mail | Mailcow Email |
| vpn | Headscale VPN |
| vault | Vaultwarden |
| wt | WATCHTOWER ops dashboard |
| matrix | Matrix Synapse |
| element | Element Web |
| notes | LiveSync (CouchDB + provisioner) |
| proxmox | Proxmox VE (via Tailscale to data node) |
| tak | TAK Server + SIGIL |
### Home Services → 199.6.36.163
| Subdomain | Service |
|-----------|---------|
| @ | Main site |
| @ | Echo6 Search homepage (SearXNG) |
| ai | Open WebUI |
| docs | Wiki.js |
| stream | PeerTube |
| notes | Obsidian LiveSync |
| jellyfin | Jellyfin |
| mesh | MeshMonitor |
| search | SearXNG |
| nas | OpenMediaVault (pi-nas) |
| search | SearXNG (redirects to echo6.co) |
| immich | Immich |
| nextcloud | Nextcloud |
| requests | Jellyseer |
| files | RECON PDF library |
| recon | RECON dashboard |
| lidarr | Lidarr music automation |
| navidrome | Navidrome music server |
### Email Records
@ -151,12 +194,21 @@ oidc:
|---------|---------------|--------------|---------------|
| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
| Forgejo SSH | 22 | 0.0.0.0:2222 | Direct (not proxied) |
| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
| WATCHTOWER | 8084 | host network :8099 | wt.echo6.co |
| Matrix Synapse | 8008 | 127.0.0.1:8008 | matrix.echo6.co (/_matrix/*, /_synapse/*) |
| Matrix MAS | 8080 | 127.0.0.1:8085 | matrix.echo6.co (login/logout/refresh/auth_metadata, default) |
| Element Web | 80 | 127.0.0.1:8088 | element.echo6.co |
| LiveSync CouchDB | 5984 | 127.0.0.1:5984 | notes.echo6.co |
| LiveSync Provisioner | 8080 | 127.0.0.1:5985 | notes.echo6.co/_provision/* |
| TAK Server Admin | 8446 | https://100.64.0.1:8446 (Tailscale) | tak.echo6.co |
| SIGIL Console | 8990 | 100.64.0.1:8990 | tak.echo6.co/sigil |
---
*Last updated: 2026-02-06 — Added SearXNG (search.echo6.co) on utility CT 102*
*Last updated: 2026-04-13 — Audit sync: added MAS routing on matrix.echo6.co, lidarr/navidrome/vpn.idahomesh.com to utility Caddy, proxmox/tak to GoDaddy, removed ghost docs.echo6.co entries, added dnsmasq lidarr/navidrome*

131
docs/software/recon.md Normal file
View file

@ -0,0 +1,131 @@
# RECON — Knowledge Extraction Pipeline
## Overview
RECON extracts knowledge from PDFs and web content into a searchable vector database. PDFs are scanned from an NFS-mounted library, text is extracted (with Gemini Vision fallback for scanned docs), concepts are enriched via Gemini, and embeddings are stored in Qdrant. Aurora (Open WebUI) queries the knowledge base via RAG filter.
## Location
- **Host:** recon LXC (CT 130 on data node, 192.168.1.240)
- **IP:** 192.168.1.130 / 100.64.0.24 (Tailscale)
- **Install:** `/opt/recon/`
- **User:** zvx
- **Service:** `recon.service` (systemd, Type=simple, Restart=on-failure)
- **Dashboard:** https://recon.echo6.co (internal: http://100.64.0.24:8420)
- **Health:** https://recon.echo6.co/api/health
## Stack
| Component | Technology | Location |
|-----------|-----------|----------|
| Pipeline + CLI | Python 3.12, argparse | /opt/recon/recon.py |
| Dashboard + API | Flask | /opt/recon/lib/api.py (port 8420) |
| Status DB | SQLite (WAL mode) | /opt/recon/data/recon.db |
| Vector DB | Qdrant | cortex:6333 (Docker) |
| Embeddings | TEI (bge-m3, 1024-dim) | cortex:8090 (Docker) |
| Enrichment | Gemini 2.0 Flash | Google API (4 keys) |
| Vision OCR | Gemini 2.0 Flash | Google API (shared keys) |
| Text extraction | PyPDF2, poppler-utils, Tesseract | Local |
| PDF source | NFS | pi-nas:/export/library → /mnt/library |
| File server | nginx | localhost:8888 → files.echo6.co |
## Pipeline Stages
All stages run concurrently as daemon threads in the service:
1. **Scanner** (hourly) — walks /mnt/library, catalogues new PDFs, queues them
2. **Extract** (4 workers) — PyPDF2 → pdftotext → Tesseract → Gemini Vision per page
3. **Enrich** (16 workers, 4 API keys) — Gemini extracts structured concepts from text windows
4. **Embed** (4 workers) — TEI generates vectors, upserted to Qdrant
## Extraction Chain
Per page, in order. Each method only runs if the previous returned <50 chars:
1. **PyPDF2** — fast, free, works on text-based PDFs
2. **pdftotext** (poppler) — handles some PDFs PyPDF2 misses
3. **Tesseract OCR** — renders page to image, runs local OCR
4. **Gemini Vision** — renders page to PNG, sends to Gemini 2.0 Flash vision API
Method tracking saved in `data/text/{hash}/meta.json` as `ocr_methods` dict.
## Scale (as of Feb 2026)
- ~10,162 documents in pipeline
- ~95,000+ vectors in Qdrant (HNSW index, <10ms search latency)
- Collection: `recon_knowledge`
- ~13,239 PDFs catalogued from NFS library
## Resilience
- **Enricher**: Exponential backoff (5s→80s) for transient errors (429, 500, 503). Window-level failure isolation — partial enrichment beats zero.
- **Extractor**: Per-page timeout (30s), per-document timeout (1800s). Partial extractions saved.
- **Embedder**: Skip-on-failure per concept, batch processing.
- **Service**: Restart=on-failure, RestartSec=30, MemoryMax=3G.
## Configuration
**Config file:** `/opt/recon/config.yaml`
Key sections:
- `processing.extract_workers` (4), `enrich_workers` (16), `embed_workers` (4)
- `processing.extract_timeout` (1800s), `page_timeout` (30s)
- `processing.enrich_max_retries` (5), `enrich_base_delay` (5.0)
- `gemini.model` (gemini-2.0-flash), `gemini.response_mime_type` (application/json)
- `service.scan_interval` (3600), `stage_poll_interval` (30)
**API keys:** `/opt/recon/.env` — GEMINI_KEY_1 through GEMINI_KEY_4
## API Endpoints
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/` | GET | Dashboard HTML |
| `/api/knowledge-stats` | GET | Full pipeline stats, per-source breakdown |
| `/api/health` | GET | Health check (Qdrant, TEI, NFS, Gemini, pipeline) |
| `/api/search` | GET | Vector search (`?q=query&limit=5`) |
| `/api/upload` | POST | Upload PDF (multipart: file + category) |
| `/api/upload/<hash>/status` | GET | Upload status tracking |
| `/api/upload/categories` | GET | Available upload categories |
| `/api/ingest` | POST | Ingest intel JSON data |
| `/api/peertube/channels` | GET | List all channels from channel-map.json with video counts from PeerTube DB |
| `/api/peertube/channels/stats` | GET | Channel count, total videos, downloader status |
| `/api/peertube/channels/add` | POST | Add channel: resolve YT URL, create PeerTube channel, update JSON |
| `/api/peertube/channels/<name>` | DELETE | Remove channel from JSON and optionally from PeerTube |
## Backups
- **Destination:** `root@100.64.0.1:/opt/backups/recon/`
- **Full sync:** every 6 hours (concepts, text, DB, config)
- **DB snapshot:** every 2 hours
- **Recovery:** restore from Contabo → `recon rebuild` (reconstructs Qdrant from concept JSONs)
- **Critical data:** `data/concepts/` — Gemini extraction work, costs money to regenerate
## Key Files
```
/opt/recon/
├── recon.py # CLI entry point + service command
├── config.yaml # Configuration
├── .env # Gemini API keys
├── PROJECT-BIBLE.md # Full documentation
├── lib/
│ ├── api.py # Flask dashboard + API
│ ├── extractor.py # PDF → text (4-method chain)
│ ├── enricher.py # Text → concepts (Gemini)
│ ├── embedder.py # Concepts → vectors (TEI/Qdrant)
│ ├── status.py # SQLite DB (WAL, thread-safe)
│ └── utils.py # Config, hashing, logging
├── scripts/
│ ├── backup.sh # Backup to Contabo
│ ├── validate.py # Pipeline consistency checker
│ └── rebuild_qdrant.py # Nuclear Qdrant rebuild
└── data/
├── recon.db # SQLite status DB
├── concepts/{hash}/ # Enriched concept JSONs
└── text/{hash}/ # Extracted page text
```
---
*Last updated: 2026-02-16 — Initial creation*

130
docs/software/searxng.md Normal file
View file

@ -0,0 +1,130 @@
# SearXNG — Echo6 Search Homepage
## Overview
SearXNG is deployed as the branded Echo6 search homepage at `echo6.co`. The default simple theme is overridden with custom templates (base.html, index.html) and static assets via Docker bind mounts. Features a cyberpunk aesthetic with JetBrains Mono font, cyan/yellow palette, top navigation bar with service links, and a waffle app launcher menu.
## Location
- **Host:** searxng LXC (CT 102 on utility node, 192.168.1.241)
- **IP:** 192.168.1.102 / 100.64.0.15 (Tailscale)
- **Install:** `/opt/searxng/` (Docker Compose)
- **User:** zvx
- **URL:** https://echo6.co
- **Redirect:** https://search.echo6.co → https://echo6.co (301 permanent)
## Stack
| Component | Technology | Location |
|-----------|-----------|----------|
| Search engine | SearXNG (Docker, v2026.2.6) | searxng container |
| Cache | Valkey (Redis-compatible) | valkey container |
| Reverse proxy | Utility Caddy (CT 101) | 192.168.1.101 |
| SSL certs | acme.sh (Let's Encrypt) | /etc/caddy/certs/ on CT 101 |
## Theme Customization
Custom Echo6 theme applied via Docker bind mounts that override SearXNG's simple theme:
### Overridden Files
| Override | Container Path | Purpose |
|----------|---------------|---------|
| `custom/templates/simple/base.html` | `/usr/local/searxng/searx/templates/simple/base.html` | Nav bar, CSS, JS, footer, waffle menu |
| `custom/templates/simple/index.html` | `/usr/local/searxng/searx/templates/simple/index.html` | Homepage (Echo6 logo + search bar) |
| `custom/img/echo6-logo.png` | `/usr/local/searxng/searx/static/themes/simple/img/echo6-logo.png` | Echo6 logo |
| `custom/img/favicon.png` | `/usr/local/searxng/searx/static/themes/simple/img/favicon.png` | Echo6 favicon |
### Brand Palette
| Variable | Value | Usage |
|----------|-------|-------|
| `--bg-primary` | `#0a0e17` | Page background |
| `--bg-secondary` | `#111827` | Cards, nav, waffle |
| `--bg-tertiary` | `#1a2332` | Hover states |
| `--accent-cyan` | `#28C0E8` | Links, focus glow, active states |
| `--accent-yellow` | `#F0D848` | Logo accents, highlights |
| `--text-primary` | `#e0e6ed` | Main text |
| `--text-secondary` | `#7a8ca0` | Muted text |
| `--border` | `#1e3a5f` | Borders |
| `--border-focus` | `#28C0E8` | Focus ring |
| Font | JetBrains Mono | All text |
### Layout
- **Homepage:** Centered Echo6 logo + pill-shaped search bar (viewport-locked, no scroll)
- **Results page:** Two-column grid (`1fr 25rem`) — results left, sidebar right
- **Nav bar:** Left: `.//files`, `.//stream` — Right: `.//photos`, `.//mail`, waffle menu, login avatar
- **Waffle menu:** 3x3 grid of service tiles (Aurora, Stream, Files, Watchtower, Photos, Mail, Cloud, Admin, Search) with inline SVG icons
- **All nav links:** Use Authentik SSO launch URLs (`https://auth.echo6.co/application/launch/<slug>/`)
## Configuration
**Config file:** `/opt/searxng/searxng-config/settings.yml` (mounted to `/etc/searxng/settings.yml`)
Key settings:
- `general.instance_name`: "Echo6"
- `ui.default_theme`: simple
- `ui.theme_args.simple_style`: dark
- `ui.center_alignment`: false (enables full-width results layout)
- `server.base_url`: "https://echo6.co/"
**Environment:** Set in docker-compose.yml:
- `SEARXNG_BASE_URL=https://echo6.co/`
- `INSTANCE_NAME=Echo6`
## Key Files
```
/opt/searxng/
├── docker-compose.yml # Compose config (searxng + valkey)
├── searxng-config/
│ └── settings.yml # SearXNG configuration
└── custom/
├── templates/simple/
│ ├── base.html # Custom base template (nav, CSS, JS, footer, waffle)
│ └── index.html # Custom homepage (Echo6 logo + search bar)
└── img/
├── echo6-logo.png # Echo6 logo
└── favicon.png # Echo6 favicon
```
## Commands
```bash
# Access CT 102 via utility Proxmox host
ssh root@192.168.1.241 'pct exec 102 -- <command>'
# Restart SearXNG
ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose restart searxng'
# View logs
ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose logs -f searxng'
# Full recreate (after compose changes)
ssh zvx@192.168.1.102 'cd /opt/searxng && docker compose down && docker compose up -d'
# Push template file via pct
scp /tmp/file.html root@192.168.1.241:/tmp/
ssh root@192.168.1.241 'pct push 102 /tmp/file.html /opt/searxng/custom/templates/simple/file.html'
# Test locally
curl -s http://192.168.1.102:8080 | head -30
```
## Caddy Configuration
**Utility Caddy (CT 101):**
- `echo6.co` → `100.64.0.15:8080` + Matrix `.well-known` handlers
- `search.echo6.co` → 301 redirect to `https://echo6.co`
**dnsmasq (Contabo):**
- `echo6.co` → `100.64.0.8` (utility Caddy)
**GoDaddy DNS:**
- `@` (echo6.co) → `199.6.36.163` (home)
- `search` → `199.6.36.163` (home)
---
*Last updated: 2026-02-17 — Initial creation after Echo6 homepage deployment*