Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 06:02:16 +00:00 · 2026-04-13 06:02:16 +00:00 · e9231ac24a
commit e9231ac24a
parent 89834796ff
93 changed files with 51223 additions and 254 deletions
--- a/projects/peertube-rebuild.md
+++ b/projects/peertube-rebuild.md
@ -0,0 +1,468 @@
+# Project: PeerTube YouTube Archive Rebuild
+
+**Goal:** Rebuild PeerTube at `stream.echo6.co` with Authentik SSO, 18TB NFS storage, and a bulk import pipeline for 250 YouTube channels (~136K videos).
+
+**Status:** Phase 1 — Complete (2026-02-13). CT 110 on media, 192.168.1.170, TS 100.64.0.23, PeerTube v8.0.2
+
+---
+
+## Architecture
+
+```
+                         ┌─────────────────────────────────┐
+                         │         utility node             │
+Internet ──── DNS ──────▶│  Caddy LXC (CT 101)             │
+                         │  stream.echo6.co → PT LXC:80    │
+                         └──────────────┬──────────────────┘
+                                        │
+                         ┌──────────────▼──────────────────┐
+                         │          media node              │
+                         │  PeerTube LXC (CT 100)            │
+                         │    ├── nginx (port 80)           │
+                         │    ├── PeerTube (port 9000)      │
+                         │    ├── PostgreSQL 16              │
+                         │    ├── Redis                      │
+                         │    └── /var/www/peertube/storage  │
+                         │           └── NFS mount (18TB)    │
+                         └──────────────────────────────────┘
+
+Authentik ◄──── OIDC ────► PeerTube
+
+Phase 2: cortex (VM 150 on TOC, has GPU) = remote transcoding runner
+```
+
+**Key decisions:**
+- LXC on media node (not VM, not Docker) — CT 100
+- Privileged container (NFS bind-mount uid mapping is hell otherwise)
+- Native PeerTube install (Node.js + PostgreSQL + Redis + nginx, no Docker)
+- PeerTube v8.0.2 — config and nginx template may differ from v6.x docs; verify during install
+- Node.js 20 (v8 requirement, not 18)
+- Caddy on utility handles TLS, nginx inside LXC handles WebSocket/static files
+- Caddy proxies to local IP (192.168.1.x) since PeerTube has OIDC
+- Transcoding: 480p + 720p only (storage budget)
+- Built-in channel sync DISABLED — bulk pipeline handles imports
+- Signup disabled — Authentik SSO only
+- NFS storage from pi-nas `/export/peertube` (separate from arr)
+
+---
+
+## Phase 1: PeerTube Up and Secure
+
+### 1.1 Provision LXC on media
+
+**Run:** `runbooks/ct-runbook.md` with these inputs:
+
+| Variable | Value |
+|----------|-------|
+| Host | media (192.168.1.243) |
+| CTID | 100 |
+| Hostname | peertube |
+| Template | Debian 12 (not Ubuntu — PeerTube docs target Debian) |
+| Memory | 4096 MB |
+| Cores | 4 |
+| Disk | 50 GB root |
+| Privileged | YES (override ct-runbook default) |
+| Network | DHCP initially |
+
+**Deviations from ct-runbook:**
+- Use **Debian 12** template instead of Ubuntu 24.04
+- Use **privileged** container (`--unprivileged 0`) for NFS compatibility
+- Skip Docker install — PeerTube runs native
+- Still do: base packages, zvx user, SSH, Tailscale
+
+### 1.2 Mount NFS storage
+
+On the **media host** (not inside LXC):
+
+```bash
+# Mount NFS on host
+mkdir -p /mnt/peertube-storage
+mount -t nfs 192.168.1.245:/export/peertube /mnt/peertube-storage
+
+# Persist
+echo "192.168.1.245:/export/peertube /mnt/peertube-storage nfs defaults,_netdev 0 0" >> /etc/fstab
+
+# Bind-mount into LXC
+echo "mp0: /mnt/peertube-storage,mp=/var/www/peertube/storage" >> /etc/pve/lxc/<CTID>.conf
+
+# Restart LXC to pick up mount
+pct stop <CTID> && pct start <CTID>
+```
+
+**Verify inside LXC:**
+```bash
+df -h /var/www/peertube/storage    # Should show ~18TB
+touch /var/www/peertube/storage/test && rm /var/www/peertube/storage/test
+```
+
+**NFS details:**
+- Server: pi-nas (192.168.1.245 / 100.64.0.21)
+- Export: `/export/peertube`
+- Access: Already configured in OMV for 100.64.0.0/10 and 192.168.1.0/24
+
+### 1.3 Install PeerTube dependencies
+
+Inside the LXC:
+
+```bash
+# PostgreSQL 16
+sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
+apt update && apt install -y postgresql-16 postgresql-contrib-16
+
+sudo -u postgres psql << 'SQL'
+CREATE USER peertube WITH PASSWORD '<PG_PASSWORD>';
+CREATE DATABASE peertube_prod OWNER peertube;
+\c peertube_prod
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+CREATE EXTENSION IF NOT EXISTS unaccent;
+SQL
+
+# Redis
+apt install -y redis-server
+sed -i 's/^# requirepass .*/requirepass <REDIS_PASSWORD>/' /etc/redis/redis.conf
+sed -i 's/^bind .*/bind 127.0.0.1 -::1/' /etc/redis/redis.conf
+systemctl restart redis-server && systemctl enable redis-server
+
+# Node.js 20 (PeerTube v8 requires Node.js 20+)
+curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
+apt install -y nodejs
+npm install -g yarn
+
+# ffmpeg (for transcoding)
+apt install -y ffmpeg
+```
+
+### 1.4 Install PeerTube
+
+```bash
+# Create peertube user
+adduser --system --group --home /var/www/peertube --shell /bin/bash peertube
+chown -R peertube:peertube /var/www/peertube/storage
+
+# Get latest version (joinpeertube.org API is dead, use GitHub)
+PEERTUBE_VERSION=$(curl -s https://api.github.com/repos/Chocobozzz/PeerTube/releases/latest | grep -oP '"tag_name": "v\K[^"]+' || echo "8.0.2")
+
+# Download and install
+cd /var/www/peertube
+sudo -u peertube mkdir -p config
+sudo -u peertube mkdir -p storage/{avatars,caches,captions,logs,plugins,previews,redundancy,streaming-playlists,thumbnails,tmp,torrents,videos,bin,storyboards,web-videos,original-video-files}
+sudo -u peertube wget -q "https://github.com/Chocobozzz/PeerTube/releases/download/v${PEERTUBE_VERSION}/peertube-v${PEERTUBE_VERSION}.tar.xz"
+sudo -u peertube tar xf peertube-v${PEERTUBE_VERSION}.tar.xz
+sudo -u peertube ln -s peertube-v${PEERTUBE_VERSION} peertube-latest
+cd peertube-latest
+sudo -u peertube yarn install --production --pure-lockfile
+```
+
+### 1.5 Configure PeerTube
+
+Create `/var/www/peertube/config/local-production.json`:
+
+```json
+{
+  "listen": { "hostname": "0.0.0.0", "port": 9000 },
+  "webserver": { "https": true, "hostname": "stream.echo6.co", "port": 443 },
+  "database": {
+    "hostname": "localhost", "port": 5432,
+    "name": "peertube_prod", "username": "peertube", "password": "<PG_PASSWORD>"
+  },
+  "redis": { "hostname": "localhost", "port": 6379, "auth": "<REDIS_PASSWORD>" },
+  "storage": {
+    "avatars": "/var/www/peertube/storage/avatars/",
+    "caches": "/var/www/peertube/storage/caches/",
+    "captions": "/var/www/peertube/storage/captions/",
+    "logs": "/var/www/peertube/storage/logs/",
+    "plugins": "/var/www/peertube/storage/plugins/",
+    "previews": "/var/www/peertube/storage/previews/",
+    "redundancy": "/var/www/peertube/storage/redundancy/",
+    "streaming_playlists": "/var/www/peertube/storage/streaming-playlists/",
+    "thumbnails": "/var/www/peertube/storage/thumbnails/",
+    "tmp": "/var/www/peertube/storage/tmp/",
+    "torrents": "/var/www/peertube/storage/torrents/",
+    "videos": "/var/www/peertube/storage/videos/",
+    "bin": "/var/www/peertube/storage/bin/",
+    "storyboards": "/var/www/peertube/storage/storyboards/",
+    "web_videos": "/var/www/peertube/storage/web-videos/",
+    "original_video_files": "/var/www/peertube/storage/original-video-files/"
+  },
+  "admin": { "email": "admin@echo6.co" },
+  "signup": { "enabled": false },
+  "import": {
+    "videos": { "concurrency": 10, "http": { "enabled": true }, "torrent": { "enabled": false } },
+    "video_channel_synchronization": { "enabled": false }
+  },
+  "transcoding": {
+    "enabled": true, "threads": 2, "concurrency": 2,
+    "allow_additional_extensions": true, "allow_audio_files": true,
+    "resolutions": {
+      "0p": false, "144p": false, "240p": false, "360p": false,
+      "480p": true, "720p": true,
+      "1080p": false, "1440p": false, "2160p": false
+    },
+    "hls": { "enabled": true },
+    "web_videos": { "enabled": true }
+  }
+}
+```
+
+```bash
+chown peertube:peertube /var/www/peertube/config/local-production.json
+chmod 600 /var/www/peertube/config/local-production.json
+```
+
+### 1.6 nginx (inside LXC)
+
+```bash
+apt install -y nginx
+rm -f /etc/nginx/sites-enabled/default
+
+cat > /etc/nginx/sites-available/peertube << 'NGINXCONF'
+server {
+    listen 80;
+    server_name stream.echo6.co;
+
+    add_header X-Frame-Options "SAMEORIGIN";
+    add_header X-Content-Type-Options "nosniff";
+
+    client_max_body_size 20G;
+    proxy_connect_timeout 600;
+    proxy_send_timeout 600;
+    proxy_read_timeout 600;
+    send_timeout 600;
+
+    location ~ ^/client/(.*\.(js|css|woff2|otf|ttf|woff|eot|svg|png|jpg|gif|ico|webp))$ {
+        add_header Cache-Control "public, max-age=31536000, immutable";
+        alias /var/www/peertube/peertube-latest/client/dist/$1;
+    }
+
+    location ~ ^(/static/(webseed|web-videos|streaming-playlists|redundancy)/.+)$ {
+        set $upstream_peertube http://127.0.0.1:9000;
+        try_files /var/www/peertube/storage$1 @api;
+        root /;
+        add_header Cache-Control "public, max-age=7200";
+    }
+
+    location @api {
+        proxy_pass http://127.0.0.1:9000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+    }
+
+    location / {
+        proxy_pass http://127.0.0.1:9000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        client_max_body_size 20G;
+    }
+}
+NGINXCONF
+
+ln -s /etc/nginx/sites-available/peertube /etc/nginx/sites-enabled/peertube
+nginx -t && systemctl restart nginx && systemctl enable nginx
+```
+
+### 1.7 systemd service
+
+```bash
+cat > /etc/systemd/system/peertube.service << 'EOF'
+[Unit]
+Description=PeerTube daemon
+After=network.target postgresql.service redis-server.service
+
+[Service]
+Type=simple
+User=peertube
+Group=peertube
+Environment=NODE_ENV=production
+Environment=NODE_CONFIG_DIR=/var/www/peertube/config
+WorkingDirectory=/var/www/peertube/peertube-latest
+ExecStart=/usr/bin/node dist/server
+Restart=always
+RestartSec=10
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=peertube
+TimeoutStartSec=60
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl daemon-reload
+systemctl enable peertube
+systemctl start peertube
+
+# Grab auto-generated root password
+journalctl -u peertube | grep -i "password"
+```
+
+### 1.8 Expose via Caddy
+
+**Run:** `runbooks/expose-service-home.md` with these inputs:
+
+| Variable | Value |
+|----------|-------|
+| Service | stream |
+| Domain | stream.echo6.co |
+| Backend IP | PeerTube LXC local IP (192.168.1.x — has OIDC, use local IP pattern) |
+| Backend port | 80 |
+| Has OIDC | YES (use local IP, not Tailscale) |
+
+DNS record for `stream.echo6.co` does NOT exist in GoDaddy yet — create it pointing to `199.6.36.163`.
+
+Also update dnsmasq split DNS on Contabo — entry exists pointing to old IP `100.64.0.7`, update to `100.64.0.8` (utility Caddy, same pattern as jellyfin/requests).
+
+### 1.9 Authentik OIDC
+
+**Run:** `runbooks/authentik-oidc-application.md` with these inputs:
+
+| Variable | Value |
+|----------|-------|
+| SERVICE_NAME | PeerTube |
+| SERVICE_SLUG | peertube |
+| SERVICE_URL | https://stream.echo6.co |
+| OIDC_CALLBACK_PATH | Check PeerTube OIDC plugin docs — likely `/plugins/auth-openid-connect/router/code-cb` |
+| NEEDS_OFFLINE_ACCESS | yes |
+| CLIENT_TYPE | confidential |
+
+**Note:** No existing PeerTube provider in Authentik — create from scratch using the runbook.
+
+Then install the plugin inside PeerTube:
+
+```bash
+cd /var/www/peertube/peertube-latest
+sudo -u peertube NODE_ENV=production NODE_CONFIG_DIR=/var/www/peertube/config \
+  node dist/server/tools/peertube-plugins.js install \
+  --npm-name peertube-plugin-auth-openid-connect
+systemctl restart peertube
+```
+
+Configure via Admin UI → Plugins → OpenID Connect:
+- Discover URL: `https://auth.echo6.co/application/o/peertube/.well-known/openid-configuration`
+- Client ID/Secret from Authentik
+- Scope: `openid email profile`
+- Username property: `preferred_username`
+- Display name property: `name`
+
+### 1.10 First login and lockdown
+
+1. Log in as `root` with the auto-generated password
+2. Change root password immediately
+3. Test Authentik SSO login
+4. Promote your Authentik user to admin
+5. Admin → Configuration: instance name "Echo6 Archive", signup disabled, HTTP import enabled
+
+### Phase 1 checklist
+
+```
+[x] LXC on media — CT 110, privileged, Debian 12, 4C/4GB/50GB
+[x] NFS 22TB mounted and writable (/export/peertube from 192.168.1.245)
+[x] PostgreSQL 16 + Redis installed
+[x] Node.js 22 + pnpm + ffmpeg installed (v8.0.2 requires Node 22, pnpm not yarn)
+[x] PeerTube v8.0.2 installed and configured
+[x] nginx configured (port 80, WebSocket, static files)
+[x] systemd service running
+[x] Tailscale registered (100.64.0.23)
+[x] Caddy on utility proxying stream.echo6.co → 192.168.1.170:80
+[x] DNS verified (GoDaddy + dnsmasq split DNS → 100.64.0.8)
+[x] Authentik OIDC working (provider pk:12, app slug: peertube)
+[ ] Root password changed, your user promoted to admin (manual step)
+```
+
+**Update after Phase 1:**
+- `docs/hardware/environment.md` — add PeerTube LXC
+- `docs/services/services.md` — add PeerTube entry
+- `docs/software/caddy.md` — add stream.echo6.co site block
+
+---
+
+## Phase 2: Import Pipeline
+
+### 2.1 Create PeerTube channels
+
+Script to create all 250 channels from the master spreadsheet via PeerTube API. One channel per YouTube channel, matching names.
+
+### 2.2 Bulk downloader
+
+- yt-dlp with cookies + PO tokens
+- `--match-filter "duration > 61"` to exclude Shorts
+- Round-robin across channels (5-10 videos per channel, rotate)
+- Download to NFS staging area
+- Track downloaded video IDs in archive file (prevent re-downloads)
+
+### 2.3 Import pipeline
+
+- Watch staging area for new downloads
+- Import to correct PeerTube channel via API
+- Move source file after successful import (or delete if transcoded)
+- Rate limit to avoid overwhelming PeerTube
+
+### 2.4 GPU transcoding on cortex
+
+- PeerTube remote runner protocol
+- cortex already has RTX A4000 + nvidia-container-toolkit
+- NVENC encoding for 480p + 720p HLS
+- PeerTube delegates transcoding jobs to cortex runner
+
+### Phase 2 checklist
+
+```
+[x] 100 channels created in PeerTube (99 planned + extras, channel-map.json at /opt/bulk-import/config/)
+[x] yt-dlp configured (cookies, Shorts filter)
+[x] Bulk downloader script with round-robin (pt-downloader service on CT 110)
+[x] Import pipeline (pt-importer service on CT 110, resumable chunked upload)
+[x] Archive tracking (downloaded.txt, downloader-state.json)
+[x] GPU transcoding runner on cortex (pt-transcoder service, H.265 NVENC)
+[x] PeerTube remote runner on cortex (Whisper auto-captioning, medium model, smart GPU/CPU routing)
+[x] Test: full cycle — download → transcode → import → playable
+[ ] VPN/IP rotation (NordVPN token pending from Matt)
+```
+
+---
+
+## Phase 3: Monitoring
+
+### 3.1 WATCHTOWER dashboard
+
+- Import queue depth and throughput
+- Per-channel video counts vs YouTube totals
+- Storage usage and growth rate
+- Transcoding queue status
+
+### 3.2 Alerts
+
+- Storage threshold warnings (80%, 90%, 95%)
+- Stalled imports (no progress for N hours)
+- Failed downloads (rate limiting, auth issues)
+
+### Phase 3 checklist
+
+```
+[ ] Dashboard showing import progress
+[ ] Per-channel completion tracking
+[ ] Storage alerts configured
+[ ] Stall detection working
+```
+
+---
+
+## Reference
+
+- **Master channel list:** `youtube_archive_master.xlsx` (250 channels, 19 categories)
+- **PeerTube LXC:** CT 110 on media (192.168.1.243)
+- **NFS:** pi-nas (192.168.1.245) export `/export/peertube`
+- **Previous PeerTube Tailscale IP:** 100.64.0.7 (do not reuse — assign fresh)
+- **Previous bulk import map:** `final-channel-map.json` (lost with crash)
+- **Previous download archive:** `downloaded.txt` (21,714 video IDs, lost with crash)
+- **Runbooks used:** ct-runbook.md, expose-service-home.md, authentik-oidc-application.md
+- **Docs to update after Phase 1:** environment.md, services.md, caddy.md, dns.md (dnsmasq entry)