echo6-docs/runbooks/peertube-remote-runner.md

PeerTube Remote Runner — GPU Transcoding

Deploy a PeerTube remote runner with NVENC GPU transcoding. The runner pulls jobs from PeerTube over WebSocket, transcodes with the GPU, and uploads HLS streams back.

Use this when adding a new runner node, rebuilding an existing one, or re-registering after a PeerTube rebuild.

---

Prerequisites

• PeerTube instance running and accessible (HTTP, not necessarily HTTPS)
• PeerTube admin credentials or API access to generate runner registration tokens
• Target machine with:
  • NVIDIA GPU with NVENC support (Maxwell gen 2+ / GTX 950+)
  • NVIDIA drivers installed and working (nvidia-smi returns output)
  • Node.js 18+ installed
  • SSH access from CC host

If the target machine needs NVIDIA drivers or Node.js, see proxmox-create-ubuntu-vm.md Steps 9 and 11.

---

Inputs

Prompt the user for all of these before executing:

RUNNER_HOST=            # SSH alias or IP for the runner machine (e.g., cortex)
RUNNER_NAME=            # Human-readable runner name (e.g., "cortex-nvenc")
RUNNER_USER=            # User to run the service as (e.g., "zvx")
PT_URL=                 # PeerTube instance URL reachable from runner (e.g., "http://100.64.0.23:9000")
PT_HOST_HEADER=         # PeerTube's public hostname for Host header (e.g., "stream.echo6.co")
PT_ADMIN_USER=          # PeerTube admin username (e.g., "root")
PT_ADMIN_PASS=          # PeerTube admin password
INSTALL_DIR=            # Where to put runner config (default: /opt/peertube-runner)

---

Step 1: Verify GPU

ssh $RUNNER_HOST 'nvidia-smi --query-gpu=name,driver_version,memory.total,encoder.stats.sessionCount --format=csv,noheader'

Gate

Must return GPU name, driver version, and VRAM. If it fails:

• No output → NVIDIA drivers not installed. See proxmox-create-ubuntu-vm.md Step 9.
• "NVML: Driver/library version mismatch" → reboot the machine.
• "No devices found" → GPU passthrough not configured (VMs) or hardware issue.

Check NVENC specifically:

ssh $RUNNER_HOST 'nvidia-smi -q | grep -A 5 "Encoder"'

Must show encoder session info. If "N/A", the GPU doesn't support NVENC or drivers are too old.

---

Step 2: Install peertube-runner

PeerTube runner is distributed via npm.

ssh $RUNNER_HOST 'which node && node --version'  # Must be 18+

Install the runner:

ssh $RUNNER_HOST 'sudo npm install -g @peertube/peertube-runner'
ssh $RUNNER_HOST 'which peertube-runner && peertube-runner --version'

Gate

peertube-runner --version must return a version number. If npm install fails, check Node.js version (must be 18+).

Create config directory:

ssh $RUNNER_HOST "sudo mkdir -p $INSTALL_DIR && sudo chown $RUNNER_USER:$RUNNER_USER $INSTALL_DIR"

---

Step 3: Generate Registration Token on PeerTube

Get a registration token from the PeerTube instance. This requires admin access.

Option A: Via API

# Get OAuth client credentials
CLIENT_CREDS=$(ssh $RUNNER_HOST "curl -s $PT_URL/api/v1/oauth-clients/local -H 'Host: $PT_HOST_HEADER'")
CLIENT_ID=$(echo "$CLIENT_CREDS" | jq -r '.client_id')
CLIENT_SECRET=$(echo "$CLIENT_CREDS" | jq -r '.client_secret')

# Get admin token
TOKEN_RESP=$(ssh $RUNNER_HOST "curl -s $PT_URL/api/v1/users/token \
  -H 'Host: $PT_HOST_HEADER' \
  --data 'client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&grant_type=password&username=$PT_ADMIN_USER&password=$PT_ADMIN_PASS'")
ACCESS_TOKEN=$(echo "$TOKEN_RESP" | jq -r '.access_token')

# Generate runner registration token
REG_TOKEN=$(ssh $RUNNER_HOST "curl -s -X POST $PT_URL/api/v1/runners/registration-tokens/generate \
  -H 'Host: $PT_HOST_HEADER' \
  -H 'Authorization: Bearer $ACCESS_TOKEN' | jq -r '.registrationToken'")

echo "Registration token: $REG_TOKEN"

Option B: Via PeerTube Admin UI

1. Log into PeerTube as admin
2. Administration → System → Runners
3. Click "Generate registration token"
4. Copy the token

Gate

Must have a registration token string. If the API returns errors:

• 401 → wrong admin credentials
• 404 → PeerTube version too old (runners require v5.2+)
• Connection refused → PeerTube not reachable from runner. Check URL and network.

---

Step 4: Register Runner

ssh $RUNNER_HOST "peertube-runner register \
  --url $PT_URL \
  --registration-token $REG_TOKEN \
  --runner-name $RUNNER_NAME"

Gate

Must complete without errors. Verify registration:

ssh $RUNNER_HOST 'peertube-runner list-registered'

Must show the PeerTube instance URL. If registration fails:

• "Invalid registration token" → token already used or expired. Generate a new one.
• "ECONNREFUSED" → runner can't reach PeerTube. Test: curl -s $PT_URL/api/v1/config
• "self-signed certificate" → if PeerTube uses HTTPS with self-signed cert, use NODE_TLS_REJECT_UNAUTHORIZED=0 (not recommended) or fix the cert.

---

Step 5: Configure NVENC

The runner auto-detects ffmpeg capabilities, but verify NVENC is available to ffmpeg:

ssh $RUNNER_HOST 'ffmpeg -encoders 2>/dev/null | grep nvenc'

Must show h264_nvenc and hevc_nvenc. If missing, install ffmpeg with NVENC support:

# Ubuntu/Debian — the default ffmpeg usually includes NVENC if drivers are installed
ssh $RUNNER_HOST 'sudo apt install -y ffmpeg'

# Re-check
ssh $RUNNER_HOST 'ffmpeg -encoders 2>/dev/null | grep nvenc'

If still missing, the NVIDIA drivers may not include the encoding libraries. Install:

ssh $RUNNER_HOST 'sudo apt install -y libnvidia-encode-550'  # Match your driver version

---

Step 6: Create systemd Service

ssh $RUNNER_HOST "sudo tee /etc/systemd/system/peertube-runner.service > /dev/null << 'EOF'
[Unit]
Description=PeerTube Remote Runner (NVENC)
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
Requires=nvidia-persistenced.service

[Service]
Type=simple
User=$RUNNER_USER
Group=$RUNNER_USER
Environment=NODE_ENV=production
ExecStart=/usr/bin/peertube-runner server \
  --enable-job vod-hls-transcoding \
  --enable-job vod-audio-merge-transcoding \
  --enable-job live-rtmp-hls-transcoding \
  --enable-job video-studio-transcoding \
  --enable-job video-transcription
WorkingDirectory=/home/$RUNNER_USER
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube-runner
MemoryMax=20G

[Install]
WantedBy=multi-user.target
EOF"

ssh $RUNNER_HOST 'sudo systemctl daemon-reload'
ssh $RUNNER_HOST 'sudo systemctl enable peertube-runner'
ssh $RUNNER_HOST 'sudo systemctl start peertube-runner'

Gate

ssh $RUNNER_HOST 'systemctl is-active peertube-runner'

Must return active. If it fails, check logs:

ssh $RUNNER_HOST 'journalctl -u peertube-runner -n 50 --no-pager'

Common failures:

• "Cannot find module" → peertube-runner not installed globally, or PATH issue. Check which peertube-runner.
• "nvidia-persistenced.service not found" → remove the Requires= line if nvidia-persistenced isn't set up (it's optional but recommended).

---

Step 7: Install Health Check

Cron script that auto-restarts the runner if it crashes or the GPU becomes inaccessible.

ssh $RUNNER_HOST "sudo tee $INSTALL_DIR/health.sh > /dev/null << 'HEALTH'
#!/bin/bash
LOG_TAG=\"peertube-runner-health\"

if ! systemctl is-active --quiet peertube-runner; then
    logger -t \$LOG_TAG \"Runner not active, restarting...\"
    systemctl restart peertube-runner
    sleep 10
fi

if ! pgrep -f \"peertube-runner server\" > /dev/null; then
    logger -t \$LOG_TAG \"Runner process not found, restarting service...\"
    systemctl restart peertube-runner
fi

if ! nvidia-smi > /dev/null 2>&1; then
    logger -t \$LOG_TAG \"GPU not accessible, restarting nvidia-persistenced and runner...\"
    systemctl restart nvidia-persistenced 2>/dev/null
    sleep 5
    systemctl restart peertube-runner
fi
HEALTH
chmod +x $INSTALL_DIR/health.sh"

# Add cron job (every 5 minutes)
ssh $RUNNER_HOST "(crontab -l 2>/dev/null | grep -v peertube-runner-health; echo '*/5 * * * * $INSTALL_DIR/health.sh') | crontab -"

Verify:

ssh $RUNNER_HOST 'crontab -l | grep peertube'

---

Step 8: Test Transcoding

Upload a test video and verify the full pipeline works.

Quick test via API

# Download a short test video
ssh $RUNNER_HOST 'curl -L -o /tmp/test-video.mp4 "https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/360/Big_Buck_Bunny_360_10s_1MB.mp4" 2>/dev/null'

# Upload to PeerTube (reuse ACCESS_TOKEN from Step 3)
ssh $RUNNER_HOST "curl -s -X POST $PT_URL/api/v1/videos/upload \
  -H 'Host: $PT_HOST_HEADER' \
  -H 'Authorization: Bearer $ACCESS_TOKEN' \
  -F 'videofile=@/tmp/test-video.mp4' \
  -F 'name=Runner Test Video' \
  -F 'channelId=1' \
  -F 'privacy=1' \
  -F 'waitTranscoding=true' | jq '{uuid, name, state}'"

Verify GPU is processing

Within 30 seconds of upload:

ssh $RUNNER_HOST 'nvidia-smi --query-gpu=utilization.gpu,utilization.encoder,temperature.gpu,power.draw --format=csv,noheader'

GPU utilization and encoder utilization should be non-zero. If encoder shows 0% but GPU shows activity, NVENC isn't being used — check ffmpeg encoder detection (Step 5).

Check runner logs

ssh $RUNNER_HOST 'journalctl -u peertube-runner -n 20 --no-pager | grep -i "transcod\|job\|error"'

Should show job pickup, transcoding progress, and completion.

Clean up

# Delete test video via API (optional)
ssh $RUNNER_HOST "curl -s -X DELETE $PT_URL/api/v1/videos/<VIDEO_UUID> \
  -H 'Host: $PT_HOST_HEADER' \
  -H 'Authorization: Bearer $ACCESS_TOKEN'"

ssh $RUNNER_HOST 'rm -f /tmp/test-video.mp4'

---

Verification Checklist

echo "=== PeerTube Runner Check ==="
echo ""
echo "GPU:          $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'MISSING')"
echo "NVENC:        $(ffmpeg -encoders 2>/dev/null | grep -c h264_nvenc) encoders"
echo "Runner ver:   $(peertube-runner --version 2>/dev/null || echo 'NOT INSTALLED')"
echo "Registered:   $(peertube-runner list-registered 2>/dev/null | grep -c 'http' || echo '0') instance(s)"
echo "Service:      $(systemctl is-active peertube-runner 2>/dev/null || echo 'NOT RUNNING')"
echo "Health cron:  $(crontab -l 2>/dev/null | grep -c peertube-runner || echo '0') entries"
echo "ffmpeg procs: $(pgrep -c ffmpeg 2>/dev/null || echo '0') active"
echo "GPU util:     $(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader 2>/dev/null || echo 'N/A')"

---

Adding a Second PeerTube Instance

To register the same runner with another PeerTube instance:

peertube-runner register \
  --url <SECOND_PT_URL> \
  --registration-token <TOKEN> \
  --runner-name $RUNNER_NAME

The runner handles multiple registrations automatically — it polls all registered instances for jobs.

Unregistering

# List registrations
peertube-runner list-registered

# Unregister from a specific instance
peertube-runner unregister --url $PT_URL

---

Troubleshooting

Runner picks up jobs but transcoding fails immediately

Check ffmpeg NVENC access:

ffmpeg -y -f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc /tmp/nvenc-test.mp4

If this fails, NVENC isn't accessible to ffmpeg. Common causes: wrong driver version, missing libnvidia-encode, or GPU in use by another process that's holding all NVENC sessions.

Runner shows 0 active jobs despite pending queue

• WebSocket connection issue. Check: journalctl -u peertube-runner | grep -i websocket
• Runner registered with wrong URL. Verify: peertube-runner list-registered
• PeerTube remote runners not enabled. Check PeerTube config: transcoding.remote_runners.enabled must be true

GPU utilization stuck at 100% / NVENC sessions maxed

The RTX A4000 supports ~3 simultaneous NVENC sessions (consumer cards are limited to 3, pro cards vary). If all sessions are in use, new jobs queue on the runner side. This is normal — throughput is limited by NVENC session count, not GPU compute.

To increase throughput: patch the NVENC session limit (search "nvidia nvenc patch") or add a second runner node.

Runner keeps disconnecting / restarting

• Check memory: free -h. The 20GB MemoryMax in the service file may be too low if processing many concurrent jobs. Increase if needed.
• Check disk space: transcoding uses temp space. Ensure /tmp or the runner's working directory has sufficient free space (10GB+ recommended).
• Network instability between runner and PeerTube. Use Tailscale IP instead of public URL for reliability.

After PeerTube rebuild, runner can't connect

Registration is tied to the PeerTube instance. After a rebuild:

1. Unregister: peertube-runner unregister --url $PT_URL
2. Generate new registration token on the new PeerTube
3. Re-register: peertube-runner register --url $PT_URL --registration-token <NEW_TOKEN> --runner-name $RUNNER_NAME
4. Restart: sudo systemctl restart peertube-runner

---

Quick Reference: Current Runners

Runner	Host	GPU	PeerTube Instance	Status
cortex-nvenc	cortex (VM 150 on TOC)	RTX A4000 16GB	stream.echo6.co (CT 110)	Active

---

Whisper Transcription Setup

The runner also handles video-transcription jobs (auto-captioning via Whisper). A smart wrapper routes jobs based on audio duration:

Smart Wrapper (/usr/local/bin/whisper-smart on cortex)

• Model: medium (good accuracy, fits in VRAM on float16)
• GPU path (< 1 hour): --device cuda --compute_type float16 — ~3.7GB VRAM, fast
• CPU path (>= 1 hour): --device cpu --compute_type int8 — ~8-11GB RAM, slow but avoids VRAM exhaustion
• CPU serialization: flock --nonblock /tmp/whisper-cpu.lock — only one CPU transcription at a time. If lock is held, wrapper exits 1 and the runner retries the job later.
• Concurrency: Runner config set to concurrency = 2 — allows one GPU + one CPU job in parallel

Symlink chain

/usr/local/bin/whisper-ctranslate2 → /usr/local/bin/whisper-smart (the smart wrapper)
/usr/local/bin/whisper-ctranslate2-real → /home/zvx/.local/bin/whisper-ctranslate2 (actual Python binary)

Key config

• Runner config: ~/.config/peertube-runner-nodejs/default/config.toml — model = "medium", concurrency = 2
• PeerTube config: /var/www/peertube/config/production.yaml on CT 110 — model-name: medium
• systemd MemoryMax: 20G (CPU int8 medium model peaks at ~11GB)

Wrapper log

tail -f /tmp/whisper-wrapper.log   # Shows mode (GPU/CPU/CPU-BLOCKED), duration, args

---

Last updated: 2026-02-17 — Added Whisper smart transcription setup, MemoryMax 12G→20G, concurrency 1→2