# Project: PeerTube YouTube Archive Rebuild

**Goal:** Rebuild PeerTube at `stream.echo6.co` with Authentik SSO, 18TB NFS storage, and a bulk import pipeline for 250 YouTube channels (~136K videos).

**Status:** Phase 1 — Complete (2026-02-13). CT 110 on media, 192.168.1.170, TS 100.64.0.23, PeerTube v8.0.2

---

## Architecture

```
                         ┌─────────────────────────────────┐
                         │         utility node             │
Internet ──── DNS ──────▶│  Caddy LXC (CT 101)             │
                         │  stream.echo6.co → PT LXC:80    │
                         └──────────────┬──────────────────┘
                                        │
                         ┌──────────────▼──────────────────┐
                         │          media node              │
                         │  PeerTube LXC (CT 100)            │
                         │    ├── nginx (port 80)           │
                         │    ├── PeerTube (port 9000)      │
                         │    ├── PostgreSQL 16              │
                         │    ├── Redis                      │
                         │    └── /var/www/peertube/storage  │
                         │           └── NFS mount (18TB)    │
                         └──────────────────────────────────┘

Authentik ◄──── OIDC ────► PeerTube

Phase 2: cortex (VM 150 on TOC, has GPU) = remote transcoding runner
```

**Key decisions:**
- LXC on media node (not VM, not Docker) — CT 100
- Privileged container (NFS bind-mount uid mapping is hell otherwise)
- Native PeerTube install (Node.js + PostgreSQL + Redis + nginx, no Docker)
- PeerTube v8.0.2 — config and nginx template may differ from v6.x docs; verify during install
- Node.js 20 (v8 requirement, not 18)
- Caddy on utility handles TLS, nginx inside LXC handles WebSocket/static files
- Caddy proxies to local IP (192.168.1.x) since PeerTube has OIDC
- Transcoding: 480p + 720p only (storage budget)
- Built-in channel sync DISABLED — bulk pipeline handles imports
- Signup disabled — Authentik SSO only
- NFS storage from pi-nas `/export/peertube` (separate from arr)

---

## Phase 1: PeerTube Up and Secure

### 1.1 Provision LXC on media

**Run:** `runbooks/ct-runbook.md` with these inputs:

| Variable | Value |
|----------|-------|
| Host | media (192.168.1.243) |
| CTID | 100 |
| Hostname | peertube |
| Template | Debian 12 (not Ubuntu — PeerTube docs target Debian) |
| Memory | 4096 MB |
| Cores | 4 |
| Disk | 50 GB root |
| Privileged | YES (override ct-runbook default) |
| Network | DHCP initially |

**Deviations from ct-runbook:**
- Use **Debian 12** template instead of Ubuntu 24.04
- Use **privileged** container (`--unprivileged 0`) for NFS compatibility
- Skip Docker install — PeerTube runs native
- Still do: base packages, zvx user, SSH, Tailscale

### 1.2 Mount NFS storage

On the **media host** (not inside LXC):

```bash
# Mount NFS on host
mkdir -p /mnt/peertube-storage
mount -t nfs 192.168.1.245:/export/peertube /mnt/peertube-storage

# Persist
echo "192.168.1.245:/export/peertube /mnt/peertube-storage nfs defaults,_netdev 0 0" >> /etc/fstab

# Bind-mount into LXC
echo "mp0: /mnt/peertube-storage,mp=/var/www/peertube/storage" >> /etc/pve/lxc/<CTID>.conf

# Restart LXC to pick up mount
pct stop <CTID> && pct start <CTID>
```

**Verify inside LXC:**
```bash
df -h /var/www/peertube/storage    # Should show ~18TB
touch /var/www/peertube/storage/test && rm /var/www/peertube/storage/test
```

**NFS details:**
- Server: pi-nas (192.168.1.245 / 100.64.0.21)
- Export: `/export/peertube`
- Access: Already configured in OMV for 100.64.0.0/10 and 192.168.1.0/24

### 1.3 Install PeerTube dependencies

Inside the LXC:

```bash
# PostgreSQL 16
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
apt update && apt install -y postgresql-16 postgresql-contrib-16

sudo -u postgres psql << 'SQL'
CREATE USER peertube WITH PASSWORD '<PG_PASSWORD>';
CREATE DATABASE peertube_prod OWNER peertube;
\c peertube_prod
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;
SQL

# Redis
apt install -y redis-server
sed -i 's/^# requirepass .*/requirepass <REDIS_PASSWORD>/' /etc/redis/redis.conf
sed -i 's/^bind .*/bind 127.0.0.1 -::1/' /etc/redis/redis.conf
systemctl restart redis-server && systemctl enable redis-server

# Node.js 20 (PeerTube v8 requires Node.js 20+)
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs
npm install -g yarn

# ffmpeg (for transcoding)
apt install -y ffmpeg
```

### 1.4 Install PeerTube

```bash
# Create peertube user
adduser --system --group --home /var/www/peertube --shell /bin/bash peertube
chown -R peertube:peertube /var/www/peertube/storage

# Get latest version (joinpeertube.org API is dead, use GitHub)
PEERTUBE_VERSION=$(curl -s https://api.github.com/repos/Chocobozzz/PeerTube/releases/latest | grep -oP '"tag_name": "v\K[^"]+' || echo "8.0.2")

# Download and install
cd /var/www/peertube
sudo -u peertube mkdir -p config
sudo -u peertube mkdir -p storage/{avatars,caches,captions,logs,plugins,previews,redundancy,streaming-playlists,thumbnails,tmp,torrents,videos,bin,storyboards,web-videos,original-video-files}
sudo -u peertube wget -q "https://github.com/Chocobozzz/PeerTube/releases/download/v${PEERTUBE_VERSION}/peertube-v${PEERTUBE_VERSION}.tar.xz"
sudo -u peertube tar xf peertube-v${PEERTUBE_VERSION}.tar.xz
sudo -u peertube ln -s peertube-v${PEERTUBE_VERSION} peertube-latest
cd peertube-latest
sudo -u peertube yarn install --production --pure-lockfile
```

### 1.5 Configure PeerTube

Create `/var/www/peertube/config/local-production.json`:

```json
{
  "listen": { "hostname": "0.0.0.0", "port": 9000 },
  "webserver": { "https": true, "hostname": "stream.echo6.co", "port": 443 },
  "database": {
    "hostname": "localhost", "port": 5432,
    "name": "peertube_prod", "username": "peertube", "password": "<PG_PASSWORD>"
  },
  "redis": { "hostname": "localhost", "port": 6379, "auth": "<REDIS_PASSWORD>" },
  "storage": {
    "avatars": "/var/www/peertube/storage/avatars/",
    "caches": "/var/www/peertube/storage/caches/",
    "captions": "/var/www/peertube/storage/captions/",
    "logs": "/var/www/peertube/storage/logs/",
    "plugins": "/var/www/peertube/storage/plugins/",
    "previews": "/var/www/peertube/storage/previews/",
    "redundancy": "/var/www/peertube/storage/redundancy/",
    "streaming_playlists": "/var/www/peertube/storage/streaming-playlists/",
    "thumbnails": "/var/www/peertube/storage/thumbnails/",
    "tmp": "/var/www/peertube/storage/tmp/",
    "torrents": "/var/www/peertube/storage/torrents/",
    "videos": "/var/www/peertube/storage/videos/",
    "bin": "/var/www/peertube/storage/bin/",
    "storyboards": "/var/www/peertube/storage/storyboards/",
    "web_videos": "/var/www/peertube/storage/web-videos/",
    "original_video_files": "/var/www/peertube/storage/original-video-files/"
  },
  "admin": { "email": "admin@echo6.co" },
  "signup": { "enabled": false },
  "import": {
    "videos": { "concurrency": 10, "http": { "enabled": true }, "torrent": { "enabled": false } },
    "video_channel_synchronization": { "enabled": false }
  },
  "transcoding": {
    "enabled": true, "threads": 2, "concurrency": 2,
    "allow_additional_extensions": true, "allow_audio_files": true,
    "resolutions": {
      "0p": false, "144p": false, "240p": false, "360p": false,
      "480p": true, "720p": true,
      "1080p": false, "1440p": false, "2160p": false
    },
    "hls": { "enabled": true },
    "web_videos": { "enabled": true }
  }
}
```

```bash
chown peertube:peertube /var/www/peertube/config/local-production.json
chmod 600 /var/www/peertube/config/local-production.json
```

### 1.6 nginx (inside LXC)

```bash
apt install -y nginx
rm -f /etc/nginx/sites-enabled/default

cat > /etc/nginx/sites-available/peertube << 'NGINXCONF'
server {
    listen 80;
    server_name stream.echo6.co;

    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";

    client_max_body_size 20G;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;

    location ~ ^/client/(.*\.(js|css|woff2|otf|ttf|woff|eot|svg|png|jpg|gif|ico|webp))$ {
        add_header Cache-Control "public, max-age=31536000, immutable";
        alias /var/www/peertube/peertube-latest/client/dist/$1;
    }

    location ~ ^(/static/(webseed|web-videos|streaming-playlists|redundancy)/.+)$ {
        set $upstream_peertube http://127.0.0.1:9000;
        try_files /var/www/peertube/storage$1 @api;
        root /;
        add_header Cache-Control "public, max-age=7200";
    }

    location @api {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        client_max_body_size 20G;
    }
}
NGINXCONF

ln -s /etc/nginx/sites-available/peertube /etc/nginx/sites-enabled/peertube
nginx -t && systemctl restart nginx && systemctl enable nginx
```

### 1.7 systemd service

```bash
cat > /etc/systemd/system/peertube.service << 'EOF'
[Unit]
Description=PeerTube daemon
After=network.target postgresql.service redis-server.service

[Service]
Type=simple
User=peertube
Group=peertube
Environment=NODE_ENV=production
Environment=NODE_CONFIG_DIR=/var/www/peertube/config
WorkingDirectory=/var/www/peertube/peertube-latest
ExecStart=/usr/bin/node dist/server
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube
TimeoutStartSec=60

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable peertube
systemctl start peertube

# Grab auto-generated root password
journalctl -u peertube | grep -i "password"
```

### 1.8 Expose via Caddy

**Run:** `runbooks/expose-service-home.md` with these inputs:

| Variable | Value |
|----------|-------|
| Service | stream |
| Domain | stream.echo6.co |
| Backend IP | PeerTube LXC local IP (192.168.1.x — has OIDC, use local IP pattern) |
| Backend port | 80 |
| Has OIDC | YES (use local IP, not Tailscale) |

DNS record for `stream.echo6.co` does NOT exist in GoDaddy yet — create it pointing to `199.6.36.163`.

Also update dnsmasq split DNS on Contabo — entry exists pointing to old IP `100.64.0.7`, update to `100.64.0.8` (utility Caddy, same pattern as jellyfin/requests).

### 1.9 Authentik OIDC

**Run:** `runbooks/authentik-oidc-application.md` with these inputs:

| Variable | Value |
|----------|-------|
| SERVICE_NAME | PeerTube |
| SERVICE_SLUG | peertube |
| SERVICE_URL | https://stream.echo6.co |
| OIDC_CALLBACK_PATH | Check PeerTube OIDC plugin docs — likely `/plugins/auth-openid-connect/router/code-cb` |
| NEEDS_OFFLINE_ACCESS | yes |
| CLIENT_TYPE | confidential |

**Note:** No existing PeerTube provider in Authentik — create from scratch using the runbook.

Then install the plugin inside PeerTube:

```bash
cd /var/www/peertube/peertube-latest
sudo -u peertube NODE_ENV=production NODE_CONFIG_DIR=/var/www/peertube/config \
  node dist/server/tools/peertube-plugins.js install \
  --npm-name peertube-plugin-auth-openid-connect
systemctl restart peertube
```

Configure via Admin UI → Plugins → OpenID Connect:
- Discover URL: `https://auth.echo6.co/application/o/peertube/.well-known/openid-configuration`
- Client ID/Secret from Authentik
- Scope: `openid email profile`
- Username property: `preferred_username`
- Display name property: `name`

### 1.10 First login and lockdown

1. Log in as `root` with the auto-generated password
2. Change root password immediately
3. Test Authentik SSO login
4. Promote your Authentik user to admin
5. Admin → Configuration: instance name "Echo6 Archive", signup disabled, HTTP import enabled

### Phase 1 checklist

```
[x] LXC on media — CT 110, privileged, Debian 12, 4C/4GB/50GB
[x] NFS 22TB mounted and writable (/export/peertube from 192.168.1.245)
[x] PostgreSQL 16 + Redis installed
[x] Node.js 22 + pnpm + ffmpeg installed (v8.0.2 requires Node 22, pnpm not yarn)
[x] PeerTube v8.0.2 installed and configured
[x] nginx configured (port 80, WebSocket, static files)
[x] systemd service running
[x] Tailscale registered (100.64.0.23)
[x] Caddy on utility proxying stream.echo6.co → 192.168.1.170:80
[x] DNS verified (GoDaddy + dnsmasq split DNS → 100.64.0.8)
[x] Authentik OIDC working (provider pk:12, app slug: peertube)
[ ] Root password changed, your user promoted to admin (manual step)
```

**Update after Phase 1:**
- `docs/hardware/environment.md` — add PeerTube LXC
- `docs/services/services.md` — add PeerTube entry
- `docs/software/caddy.md` — add stream.echo6.co site block

---

## Phase 2: Import Pipeline

### 2.1 Create PeerTube channels

Script to create all 250 channels from the master spreadsheet via PeerTube API. One channel per YouTube channel, matching names.

### 2.2 Bulk downloader

- yt-dlp with cookies + PO tokens
- `--match-filter "duration > 61"` to exclude Shorts
- Round-robin across channels (5-10 videos per channel, rotate)
- Download to NFS staging area
- Track downloaded video IDs in archive file (prevent re-downloads)

### 2.3 Import pipeline

- Watch staging area for new downloads
- Import to correct PeerTube channel via API
- Move source file after successful import (or delete if transcoded)
- Rate limit to avoid overwhelming PeerTube

### 2.4 GPU transcoding on cortex

- PeerTube remote runner protocol
- cortex already has RTX A4000 + nvidia-container-toolkit
- NVENC encoding for 480p + 720p HLS
- PeerTube delegates transcoding jobs to cortex runner

### Phase 2 checklist

```
[x] 100 channels created in PeerTube (99 planned + extras, channel-map.json at /opt/bulk-import/config/)
[x] yt-dlp configured (cookies, Shorts filter)
[x] Bulk downloader script with round-robin (pt-downloader service on CT 110)
[x] Import pipeline (pt-importer service on CT 110, resumable chunked upload)
[x] Archive tracking (downloaded.txt, downloader-state.json)
[x] GPU transcoding runner on cortex (pt-transcoder service, H.265 NVENC)
[x] PeerTube remote runner on cortex (Whisper auto-captioning, medium model, smart GPU/CPU routing)
[x] Test: full cycle — download → transcode → import → playable
[ ] VPN/IP rotation (NordVPN token pending from Matt)
```

---

## Phase 3: Monitoring

### 3.1 WATCHTOWER dashboard

- Import queue depth and throughput
- Per-channel video counts vs YouTube totals
- Storage usage and growth rate
- Transcoding queue status

### 3.2 Alerts

- Storage threshold warnings (80%, 90%, 95%)
- Stalled imports (no progress for N hours)
- Failed downloads (rate limiting, auth issues)

### Phase 3 checklist

```
[ ] Dashboard showing import progress
[ ] Per-channel completion tracking
[ ] Storage alerts configured
[ ] Stall detection working
```

---

## Reference

- **Master channel list:** `youtube_archive_master.xlsx` (250 channels, 19 categories)
- **PeerTube LXC:** CT 110 on media (192.168.1.243)
- **NFS:** pi-nas (192.168.1.245) export `/export/peertube`
- **Previous PeerTube Tailscale IP:** 100.64.0.7 (do not reuse — assign fresh)
- **Previous bulk import map:** `final-channel-map.json` (lost with crash)
- **Previous download archive:** `downloaded.txt` (21,714 video IDs, lost with crash)
- **Runbooks used:** ct-runbook.md, expose-service-home.md, authentik-oidc-application.md
- **Docs to update after Phase 1:** environment.md, services.md, caddy.md, dns.md (dnsmasq entry)