echo6-docs/projects/arr-stack-runbook.md

# CC Runbook: Build ARR Media Stack on Proxmox `media` Node

## Objective

Build a complete media automation stack on the Proxmox node `media` inside a single Ubuntu VM called `arr`. Each service runs in its own Docker container with a shared bridge network for inter-service communication. All services are exposed on the VM's LAN IP on their respective ports.

**Services:**
- Jellyfin (media server, software transcoding — no GPU)
- Jellyseer (request management)
- Sonarr (TV automation)
- Radarr (Movie automation)
- Prowlarr (indexer manager)
- SABnzbd (Usenet download client)

---

## Phase 0: SSH Prereq Check

**CRITICAL — Do this first. Do not skip.**

```bash
ssh media "echo 'SSH OK to media node'"
```

If this fails, stop and fix SSH access before proceeding. Use sshpass or key auth per `~/.ssh/config`. Cortex is the management host — all commands originate from here.

---

## Phase 1: Create Ubuntu VM on `media`

1. SSH to `media` Proxmox node.
2. Find the next available VMID: `pvesh get /cluster/nextid`
3. Download Ubuntu 24.04 cloud image if not already cached:
   - URL: `https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img`
   - Store in appropriate Proxmox storage.
4. Create a VM named `arr` with:
   - **Network:** bridged to the LAN bridge (likely `vmbr0`)
   - **Resource allocation:** Decide based on the combined needs of all six services. Jellyfin (software transcoding) and SABnzbd (decompression) are the heaviest. Sonarr/Radarr/Prowlarr/Jellyseer are lightweight. Size the VM accordingly — suggest at minimum 4 cores and 8GB RAM, but use your judgment.
   - **Disk:** 30GB for OS + container configs (media lives on NFS)
   - Cloud-init configured with:
     - Default user: `zvx`
     - SSH key from cortex (discover from `~/.ssh/id_rsa.pub` or equivalent)
     - Networking: DHCP or static — check the pattern of other VMs on this node and match it
5. Start the VM, wait for boot, discover and record its LAN IP.
6. Verify SSH from cortex → arr VM works.

---

## Phase 2: Base System Setup on `arr` VM

SSH into the `arr` VM:

1. `apt update && apt upgrade -y`
2. Install Docker + Docker Compose via the official Docker apt repo for Ubuntu.
3. Install NFS client: `apt install -y nfs-common`
4. Install Tailscale and join the tailnet:
   - `curl -fsSL https://tailscale.com/install.sh | sh`
   - `tailscale up` — use an auth key if available. Check how other VMs joined (look at Headscale config if self-hosted).
   - Record the Tailscale IP of the `arr` VM.
5. Discover appropriate PUID/PGID:
   - Mount the NFS share temporarily and `ls -ln` to check file ownership.
   - If no files exist, create a `media` user/group (e.g., PUID=1000, PGID=1000) and ensure NFS permissions align.

---

## Phase 3: NFS Mount

1. **Discover the NFS server:**
   - The NFS export is `/export/arr`, accessible from `100.64.0.0/10` (Tailscale) and `192.168.1.0/24` (LAN).
   - Find the NFS server IP by checking:
     - `/etc/fstab` on other VMs on this node
     - `showmount -e <candidate IPs>` on LAN
     - Proxmox storage config: `pvesm status` or `/etc/pve/storage.cfg`
2. `mkdir -p /mnt/arr`
3. `mount -t nfs <NFS_SERVER>:/export/arr /mnt/arr`
4. Create subdirectories if they don't exist:
   ```
   mkdir -p /mnt/arr/{movies,tv,downloads,downloads/complete,downloads/incomplete}
   ```
5. Set ownership to discovered PUID:PGID on all subdirs.
6. Add to `/etc/fstab` for persistence:
   ```
   <NFS_SERVER>:/export/arr /mnt/arr nfs defaults,_netdev 0 0
   ```
7. Verify: `umount /mnt/arr && mount -a && ls /mnt/arr`

---

## Phase 4: Docker Containers

### Setup

```bash
mkdir -p /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
```

Create a Docker bridge network for inter-service communication:
```bash
docker network create arr-net
```

### Container Deployment

Deploy each service as its own standalone container. All containers join `arr-net`. All get `TZ=America/Boise` and the discovered `PUID`/`PGID`.

**Decide per-container resource limits** (CPU shares, memory limits) based on service needs:
- **Heavy:** Jellyfin (transcoding), SABnzbd (decompression) — allocate more CPU/RAM
- **Medium:** Sonarr, Radarr — moderate
- **Light:** Prowlarr, Jellyseer — minimal

Use lightweight images (hotio where available, official otherwise).

#### Jellyfin
- Image: `jellyfin/jellyfin:latest`
- Container name: `jellyfin`
- Port: `8096:8096`
- Volumes:
  - `/opt/arr/jellyfin/config:/config`
  - `/mnt/arr/movies:/data/movies:ro`
  - `/mnt/arr/tv:/data/tv:ro`
- Network: `arr-net`
- Restart: `unless-stopped`

#### Jellyseer
- Image: `fallenbagel/jellyseer:latest`
- Container name: `jellyseer`
- Port: `5055:5055`
- Volumes:
  - `/opt/arr/jellyseer/config:/app/config`
- Network: `arr-net`
- Restart: `unless-stopped`

#### Sonarr
- Image: `ghcr.io/hotio/sonarr:latest`
- Container name: `sonarr`
- Port: `8989:8989`
- Volumes:
  - `/opt/arr/sonarr/config:/config`
  - `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`

#### Radarr
- Image: `ghcr.io/hotio/radarr:latest`
- Container name: `radarr`
- Port: `7878:7878`
- Volumes:
  - `/opt/arr/radarr/config:/config`
  - `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`

#### Prowlarr
- Image: `ghcr.io/hotio/prowlarr:latest`
- Container name: `prowlarr`
- Port: `9696:9696`
- Volumes:
  - `/opt/arr/prowlarr/config:/config`
- Network: `arr-net`
- Restart: `unless-stopped`

#### SABnzbd
- Image: `ghcr.io/hotio/sabnzbd:latest`
- Container name: `sabnzbd`
- Port: `8080:8080`
- Volumes:
  - `/opt/arr/sabnzbd/config:/config`
  - `/mnt/arr/downloads:/data/downloads`
- Network: `arr-net`
- Restart: `unless-stopped`

### Volume Mapping Design

Sonarr and Radarr both map `/mnt/arr:/data` so hardlinks/atomic moves work between `/data/downloads/complete` and `/data/movies` or `/data/tv` without cross-filesystem copies. This is critical for avoiding double disk usage.

### Verify

All six containers are running: `docker ps`
Curl each service on localhost to confirm they respond on their expected ports.

---

## Phase 5: Authentik OIDC Setup

**Discovery:** Find the Authentik instance.
- Check Caddy config on `utility` for an existing Authentik route (likely `auth.echo6.co` or `authentik.echo6.co`).
- Discover the Authentik API URL and obtain/create an API token from Authentik's docker-compose environment or admin API.

### Jellyfin OIDC
1. Create OAuth2/OpenID Provider in Authentik:
   - Name: `jellyfin`, Client type: Confidential
   - Redirect URI: `https://jellyfin.echo6.co/sso/OID/redirect/Authentik`
   - Scopes: `openid profile email`
   - Signing key: use existing or create
2. Create Application: Name `Jellyfin`, slug `jellyfin`, attach provider.
3. Record Client ID + Secret.
4. Install SSO-Auth plugin in Jellyfin and configure with Authentik OIDC details (discovery URL, client ID, secret).

### Jellyseer OIDC
1. Create OAuth2/OpenID Provider in Authentik:
   - Name: `jellyseer`, Client type: Confidential
   - Redirect URI: `https://requests.echo6.co/api/v1/auth/oidc-callback` (verify actual callback path from Jellyseer docs)
   - Scopes: `openid profile email`
2. Create Application: Name `Jellyseer`, slug `jellyseer`, attach provider.
3. Record Client ID + Secret.
4. Configure Jellyseer OIDC via its settings.

---

## Phase 6: Caddy Reverse Proxy on `utility`

SSH to `utility`. Discover the Caddyfile location and how Caddy is managed (docker, systemd, etc.).

Add entries using the **Tailscale IP** of the `arr` VM as the upstream:

```
jellyfin.echo6.co {
    reverse_proxy <ARR_TAILSCALE_IP>:8096
}

requests.echo6.co {
    reverse_proxy <ARR_TAILSCALE_IP>:5055
}
```

**Do NOT expose Sonarr, Radarr, Prowlarr, or SABnzbd via Caddy.** Those are internal-only, accessible via Tailscale or LAN.

Reload Caddy.

---

## Phase 7: GoDaddy DNS

**Discovery:** Check if GoDaddy API key/secret exists on cortex or utility. Look at how existing `echo6.co` subdomains are configured for the pattern.

Create A records (via API if available, otherwise output for manual creation):

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | `jellyfin` | Public IP of Caddy/utility (discover) | 600 |
| A | `requests` | Public IP of Caddy/utility (discover) | 600 |

These are publicly exposed WITHOUT Tailscale. Caddy handles TLS via Let's Encrypt. The upstream uses the Tailscale IP but DNS points to the public-facing Caddy IP.

---

## Phase 8: Validation

1. From `arr` VM, curl all six services on localhost (ports 8096, 5055, 8989, 7878, 9696, 8080)
2. `curl -sI https://jellyfin.echo6.co` → 200 with valid TLS
3. `curl -sI https://requests.echo6.co` → 200 with valid TLS
4. Authentik OIDC login works for both Jellyfin and Jellyseer
5. NFS persists after reboot: `reboot`, wait, `df -h /mnt/arr`
6. All containers auto-start after reboot: `docker ps` shows all six running

---

## Important Notes

- **Do NOT configure** Prowlarr indexers, Sonarr/Radarr API connections, or SABnzbd Usenet provider credentials. That will be done in a separate prompt.
- **All discovery steps are intentional** — do not hardcode IPs or paths. Find them dynamically from the running infrastructure.
- **If any phase fails, stop and report the error.** Do not skip phases.