matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

Browse source 
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup)
- Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing
- Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack
- Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure
- Removes 4 deprecated runbook duplicates (canonical versions live in projects/)
- Adds .gitignore for binary archives and editor temp files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-04-13 06:02:16 +00:00
commit e9231ac24a
93 changed files with 51223 additions and 254 deletions
Download patch file Download diff file Expand all files Collapse all files

173
runbooks/add-peertube-channel.md Normal file
View file

@ -0,0 +1,173 @@
# Add PeerTube Channel
## Overview
Add a YouTube channel to the PeerTube bulk import pipeline. Creates the PeerTube channel, adds to channel-map.json, and the downloader will begin syncing videos automatically.
## Prerequisites
- SSH access from CT 130 (RECON) → CT 110 (PeerTube): working
- Sudoers: `/etc/sudoers.d/recon-mgmt` on CT 110 (allows zvx to run yt-dlp, psql, tee as peertube)
- YouTube cookies at `/opt/bulk-import/config/cookies.txt` on CT 110 (not stale)
## Method 1: Web UI (Preferred)
1. Open **RECON Dashboard** → Upload tab: `http://192.168.1.130:8420/upload`
2. Scroll to **PeerTube Channels** section
3. Enter YouTube URL, category, priority
4. Click **Add Channel**
5. Wait for "Added: ChannelName" confirmation
**Note:** If the channel has members-only content, the API will automatically retry with `--ignore-errors` on the `/videos` tab.
## Method 2: CLI (For Troubleshooting)
Use when the web UI fails or you need manual control.
### Variables
```bash
YT_URL="https://www.youtube.com/@ChannelName"
CATEGORY="CategoryName"
PRIORITY="M" # H, M, or L
```
### Step 1: Resolve Channel Info
```bash
# From CT 130 or cortex:
ssh zvx@192.168.1.170 "sudo -u peertube /usr/local/bin/yt-dlp \
--cookies /opt/bulk-import/config/cookies.txt \
--print channel --print channel_url --print channel_id \
--playlist-items 1 --skip-download '$YT_URL'"
```
**If members-only error:** Append `/videos` to URL and add `--ignore-errors --playlist-items 1:5`:
```bash
ssh zvx@192.168.1.170 "sudo -u peertube /usr/local/bin/yt-dlp \
--cookies /opt/bulk-import/config/cookies.txt \
--print channel --print channel_url --print channel_id \
--ignore-errors --playlist-items 1:5 --skip-download '${YT_URL}/videos' 2>/dev/null" | head -3
```
Record the output:
```
CHANNEL_NAME="Civilian Rifleman"
CHANNEL_URL="https://www.youtube.com/channel/UC..."
CHANNEL_ID="UC..."
```
### Step 2: Slugify Actor Name
```bash
ACTOR_NAME=$(echo "$CHANNEL_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g; s/--*/-/g; s/^-//; s/-$//' | cut -c1-50)
echo "$ACTOR_NAME"
```
### Step 3: Check for Duplicates
```bash
ssh zvx@192.168.1.170 "cat /opt/bulk-import/config/channel-map.json" \
| python3 -c "import sys,json; d=json.load(sys.stdin); \
matches=[c for c in d if c.get('actor_name')=='$ACTOR_NAME' or c.get('youtube_channel_id')=='$CHANNEL_ID']; \
print('DUPLICATE:', matches[0]['channel_name']) if matches else print('OK - no conflicts')"
```
### Step 4: Create PeerTube Channel
```bash
ssh zvx@192.168.1.170 bash << 'REMOTE'
CLIENT=$(curl -s http://localhost:9000/api/v1/oauth-clients/local -H "Host: stream.echo6.co")
CID=$(echo "$CLIENT" | python3 -c "import sys,json; print(json.load(sys.stdin)['client_id'])")
CSEC=$(echo "$CLIENT" | python3 -c "import sys,json; print(json.load(sys.stdin)['client_secret'])")
TOKEN=$(curl -s http://localhost:9000/api/v1/users/token -H "Host: stream.echo6.co" \
--data "client_id=$CID&client_secret=$CSEC&grant_type=password&username=root&password=7redditGold" \
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
curl -s -X POST http://localhost:9000/api/v1/video-channels \
-H "Host: stream.echo6.co" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{\"name\":\"ACTOR_NAME\",\"displayName\":\"(YT)CHANNEL_NAME\"}"
REMOTE
```
Replace `ACTOR_NAME` and `CHANNEL_NAME` in the `-d` payload. Record the returned `videoChannel.id`.
### Step 5: Update channel-map.json
**IMPORTANT:** Write to temp file first, then tee into place. Never pipe directly into tee on the same file being read — it causes a race condition that empties the file.
```bash
ssh zvx@192.168.1.170 bash << 'REMOTE'
python3 -c "
import json
with open('/opt/bulk-import/config/channel-map.json') as f:
channels = json.load(f)
channels.append({
'category': 'CATEGORY',
'channel_name': '(YT)CHANNEL_NAME',
'actor_name': 'ACTOR_NAME',
'youtube_url': 'CHANNEL_URL',
'youtube_channel_id': 'CHANNEL_ID',
'peertube_channel_id': PT_CHANNEL_ID,
'video_count': 0,
'priority': 'PRIORITY',
'est_videos': 0,
'est_gb': 0
})
print(json.dumps(channels, indent=2))
" > /tmp/channel-map-new.json \
&& sudo -u peertube tee /opt/bulk-import/config/channel-map.json < /tmp/channel-map-new.json > /dev/null \
&& rm -f /tmp/channel-map-new.json \
&& echo "OK"
REMOTE
```
Replace all placeholder values (CATEGORY, CHANNEL_NAME, ACTOR_NAME, CHANNEL_URL, CHANNEL_ID, PT_CHANNEL_ID, PRIORITY).
### Step 6: Verify
```bash
# Check channel count
curl -s http://192.168.1.130:8420/api/peertube/channels/stats | python3 -m json.tool
# Verify new channel in list
curl -s http://192.168.1.130:8420/api/peertube/channels \
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d[-1]['actor_name'], d[-1]['category'])"
```
## Recovery: Empty channel-map.json
If `tee` race condition empties the file:
1. Check Contabo backup: `ssh root@100.64.0.1 ls -la /opt/backups/recon/`
2. Or rebuild from PeerTube DB:
```bash
ssh zvx@192.168.1.170 "sudo -u peertube psql peertube_prod -t -A -c \
\"SELECT name, \\\"displayName\\\" FROM \\\"videoChannel\\\" WHERE name != 'root_channel' AND name != 'default' ORDER BY id;\""
```
## API Endpoints (RECON Dashboard)
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/peertube/channels` | GET | List all channels with video counts |
| `/api/peertube/channels/stats` | GET | Total channels, videos, downloader status |
| `/api/peertube/channels/add` | POST | Add channel (JSON: youtube_url, category, priority) |
| `/api/peertube/channels/<actor_name>` | DELETE | Remove channel from JSON and PeerTube |
## Common Issues
| Issue | Cause | Fix |
|-------|-------|-----|
| yt-dlp "Join this channel" error | Members-only first video | API auto-retries with `/videos` tab. CLI: add `--ignore-errors --playlist-items 1:5` and use `/videos` URL |
| channel-map.json empty (0 bytes) | tee race condition | Always write to temp file first, then tee. Restore from backup or Contabo |
| sudo: password required | Sudoers not set up | Create `/etc/sudoers.d/recon-mgmt` via `pct exec 110` from root@192.168.1.243 |
| PeerTube "actor name already exists" | Channel exists in PeerTube but not in JSON | Add entry to JSON manually with correct `peertube_channel_id` |
---
*Last updated: 2026-02-18 — Initial creation*

337
runbooks/authentik-access-groups.md Normal file
View file

@ -0,0 +1,337 @@
# Authentik Access Groups
Manage group-based application access via the Authentik API. No web UI interaction required.
**Authentik instance:** https://auth.echo6.co (Contabo, 100.64.0.1)
**Key behavior:** Users in `authentik Admins` (is_superuser=true) bypass ALL policy checks automatically. Group bindings only restrict non-superuser access.
---
## How It Works
By default, any authenticated Authentik user can access any application. Adding a **policy binding** that ties a **group** to an **application** restricts that app to group members only (plus superusers).
- One binding per group-application pair
- An app can have multiple group bindings (policy_engine_mode=`any` means membership in ANY bound group grants access)
- Apps with zero bindings remain open to all authenticated users
- Superusers always have access regardless of bindings
---
## Setup
```bash
AK_TOKEN="$(grep 'AUTHENTIK_API_TOKEN=' /home/zvx/projects/.ref/credentials | tail -1 | sed 's/.*=//' | tr -d '"')"
AK_API="https://auth.echo6.co/api/v3"
```
Verify:
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" "$AK_API/core/groups/?page_size=1" | jq '.pagination.count'
```
Must return a number. If `403`, the token is invalid or expired.
---
## Procedure A: Create a New Access Group
### Inputs
```
GROUP_NAME= # lowercase, hyphenated (e.g., "finance-users", "dev-users")
```
Convention: `<category>-users` (e.g., `media-users`, `cloud-users`, `security-users`).
### Create the group
```bash
curl -s -X POST "$AK_API/core/groups/" \
-H "Authorization: Bearer $AK_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$GROUP_NAME\"}" | jq '{name: .name, pk: .pk}'
```
Store the returned `pk` as `GROUP_PK`.
### Gate
Response must include a valid UUID `pk`. If it returns an error, the group name likely already exists.
---
## Procedure B: Bind a Group to an Application
This restricts the application so only members of the bound group (and superusers) can access it.
### Inputs
```
APP_SLUG= # Application slug (e.g., "jellyfin", "nextcloud")
GROUP_PK= # Group UUID from Procedure A or the reference table below
```
### Look up the application PK
```bash
APP_PK=$(curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/applications/?slug=$APP_SLUG&superuser_full_list=true" \
| jq -r '.results[0].pk')
echo "App PK: $APP_PK"
```
Must return a UUID. Use `superuser_full_list=true` because apps that already have bindings won't appear without it.
### Check for existing bindings
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/policies/bindings/?target=$APP_PK" \
| jq '.results[] | {pk: .pk, group: .group_obj.name}'
```
Review output. If the desired group is already bound, skip creation.
### Create the binding
```bash
curl -s -X POST "$AK_API/policies/bindings/" \
-H "Authorization: Bearer $AK_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"group\": \"$GROUP_PK\",
\"target\": \"$APP_PK\",
\"order\": 0,
\"enabled\": true,
\"negate\": false,
\"timeout\": 30
}" | jq '{pk: .pk, group: .group_obj.name, target: .target}'
```
### Gate
Response must include a valid UUID `pk`. If it fails:
- **"target" invalid** — the application PK is wrong
- **"group" invalid** — the group PK is wrong
---
## Procedure C: Add a User to a Group
### Inputs
```
USERNAME= # Authentik username (e.g., "jodie")
GROUP_PK= # Group UUID
```
### Look up the user PK
```bash
USER_PK=$(curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/users/?search=$USERNAME" \
| jq -r '.results[0].pk')
echo "User PK: $USER_PK"
```
### Get current group members
```bash
CURRENT_USERS=$(curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/groups/$GROUP_PK/" \
| jq -r '[.users[]] | join(",")')
echo "Current user PKs: $CURRENT_USERS"
```
### Add user to group
```bash
curl -s -X PATCH "$AK_API/core/groups/$GROUP_PK/" \
-H "Authorization: Bearer $AK_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"users\": [$CURRENT_USERS, $USER_PK]}" \
| jq '{name: .name, users: [.users_obj[].username]}'
```
### Gate
Response must list the user in `users`. The `users` field is a **replace** operation — always include existing user PKs to avoid removing them.
---
## Procedure D: Remove a User from a Group
Same as Procedure C, but omit the user PK from the `users` array:
```bash
# Get current members, filter out the target user
NEW_USERS=$(curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/groups/$GROUP_PK/" \
| jq -r "[.users[] | select(. != $USER_PK)] | join(\",\")")
curl -s -X PATCH "$AK_API/core/groups/$GROUP_PK/" \
-H "Authorization: Bearer $AK_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"users\": [$NEW_USERS]}" \
| jq '{name: .name, users: [.users_obj[].username]}'
```
---
## Procedure E: Remove a Group Binding from an Application
This re-opens the application to all authenticated users (if it was the only binding).
### Find the binding PK
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/policies/bindings/?target=$APP_PK" \
| jq '.results[] | {binding_pk: .pk, group: .group_obj.name}'
```
### Delete the binding
```bash
BINDING_PK= # From the output above
curl -s -X DELETE -w "%{http_code}" \
-H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/policies/bindings/$BINDING_PK/"
```
Must return `204`.
---
## Procedure F: Rename a Group
```bash
OLD_GROUP_PK= # UUID of the group to rename
NEW_NAME= # New name (e.g., "media-users")
curl -s -X PATCH "$AK_API/core/groups/$OLD_GROUP_PK/" \
-H "Authorization: Bearer $AK_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$NEW_NAME\"}" \
| jq '{name: .name, pk: .pk}'
```
Renaming propagates to all existing bindings automatically — no need to recreate bindings.
---
## Verification
### List all groups and members
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/groups/?page_size=50" \
| jq '.results[] | {name: .name, pk: .pk, superuser: .is_superuser, users: [.users_obj[].username]}'
```
### List all application bindings
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/applications/?superuser_full_list=true&page_size=50" \
| jq -r '.results[] | .slug' | while read slug; do
echo "--- $slug ---"
APP_PK=$(curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/applications/?slug=$slug&superuser_full_list=true" \
| jq -r '.results[0].pk')
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/policies/bindings/?target=$APP_PK" \
| jq -r 'if .results | length == 0 then " (open to all)" else .results[] | " \(.group_obj.name)" end'
done
```
### Check what a specific user can see
```bash
# This shows apps visible to the API token owner without superuser bypass
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/applications/?superuser_full_list=false" \
| jq '[.results[].name]'
```
For a non-superuser, this returns only apps they have group access to plus unbound apps.
---
## Troubleshooting
### User gets "access denied" after binding was added
1. Verify the user is in the correct group:
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/core/groups/$GROUP_PK/" \
| jq '[.users_obj[].username]'
```
2. Verify the binding exists and is enabled:
```bash
curl -s -H "Authorization: Bearer $AK_TOKEN" \
"$AK_API/policies/bindings/?target=$APP_PK" \
| jq '.results[] | {group: .group_obj.name, enabled: .enabled, negate: .negate}'
```
3. Check that `negate` is `false` — if `true`, the binding denies access instead of granting it.
### Superuser can't see all apps in the UI
The Authentik user library page uses `superuser_full_list=false` by default. Superusers always have SSO access to all apps, but the library page only shows apps the user is explicitly authorized for. This is cosmetic — direct URL access still works.
### App disappeared from user's library after adding first binding
Expected behavior. Before any bindings exist, the app is open to everyone. The moment you add the first group binding, only that group's members (and superusers) see it. Make sure all intended users are in the group before binding.
---
## Quick Reference: Current State
### Groups
| Group | PK | Members |
|-------|----|---------|
| authentik Admins | `9944e153-f860-4443-81d1-ae544f611806` | akadmin, matt (superuser) |
| media-users | `0820b2b8-6c54-4c20-9a0a-872820e6d9ea` | jodie |
| communication-users | `31bce176-cd86-4aea-8db3-a57e03d5c2d1` | — |
| security-users | `f345a043-c2a4-4906-a43b-9860eae86ee1` | — |
| productivity-users | `698d80c7-7c29-43cd-b5d4-9eb24c85a6cc` | — |
| cloud-users | `db3cbf5d-8057-4e33-8e8d-95bfdb35fbac` | — |
| proxmox_admins | `d85a868d-7d1e-4585-92a8-b8bb86771b53` | akadmin, matt |
| proxmox_users | `cf26703a-a824-47dd-9550-30b848a8ce5f` | — |
### Application Bindings
| Application | Slug | Group | Binding PK |
|-------------|------|-------|------------|
| Jellyfin | jellyfin | media-users | `31515ffc-f937-442f-9813-263e68247687` |
| Jellyseer | jellyseer | media-users | *(existing)* |
| PeerTube | peertube | media-users | `c0f79fd3-9270-49f6-8457-42affc96c50a` |
| Mailcow | mailcow | communication-users | `5a8f92de-81d5-4cf6-9273-7093de0f568d` |
| Vaultwarden | vaultwarden | security-users | `a39e9d0c-237d-4a62-976e-b6c74ee31629` |
| Forgejo | forgejo | productivity-users | `49953de3-af0b-4b1b-954d-70684d127445` |
| Nextcloud | nextcloud | cloud-users | `6ac8ccfc-7ca3-4288-a281-b78a1c675e57` |
| Immich | immich | cloud-users | `4fdf2887-e94f-4c24-81d2-d8cd4587ef38` |
### Unbound Applications (open to all authenticated users)
| Application | Slug | Reason |
|-------------|------|--------|
| Headplane | headplane | Admin tool — superuser access only needed |
| Headscale VPN | headscale | Admin tool — superuser access only needed |
| Proxmox VE | proxmox | Admin tool — superuser access only needed |
| WATCHTOWER | watchtower | Admin tool — superuser access only needed |
---
*Last updated: 2026-02-14 — Initial creation with 5 access groups and 8 application bindings*

204
runbooks/authentik-create-invitation.md Normal file
View file

@ -0,0 +1,204 @@
# Authentik: Create Invitation
Create user invitations via the Authentik Admin UI. Supports two modes: email (automatic delivery) and link-sharing (manual delivery).
---
## When to Use This
Any time a new user needs to be invited to Echo6 services. Invitations create a time-limited enrollment link that lets the invitee set up their own username and password.
---
## Prerequisites
- Authentik admin access at https://auth.echo6.co
- For email mode: SMTP must be configured and working (no-reply@echo6.co via Mailcow)
---
## Mode 1: Invite via Email (Automatic)
The invitation email is sent automatically when the invitation is created with an `email` field in custom attributes.
### Steps
1. Log in to https://auth.echo6.co as admin
2. Navigate to **Directory → Invitations → Create**
3. Fill in:
| Field | Value |
|-------|-------|
| Name | Descriptive name (e.g., `jane-smith-2026-02`) |
| Flow | **Invitation Enrollment** |
| Single use | **On** (recommended) |
| Expires | Set appropriately (e.g., 7 days from now) |
4. In **Custom attributes** (YAML format):
```yaml
name: Jane Smith
email: jane@example.com
```
5. Click **Create**
The expression policy (`invitation-email-sender`) detects the `email` field and calls `ak_send_email()` to deliver the enrollment link to the invitee. The email includes the invitation URL with the `?itoken=` parameter.
### What the Invitee Receives
- Email from `no-reply@echo6.co` with subject "You've been invited to join Echo6"
- Contains a link to `https://auth.echo6.co/if/flow/invitation-enrollment/?itoken=<token>`
- The link takes them through the enrollment flow: accept invitation → set username/password → auto-login
---
## Mode 2: Invite via Link (Manual)
For cases where you want to share the link yourself (Slack, Signal, in person, etc.), omit the `email` field.
### Steps
1. Log in to https://auth.echo6.co as admin
2. Navigate to **Directory → Invitations → Create**
3. Fill in:
| Field | Value |
|-------|-------|
| Name | Descriptive name (e.g., `jane-smith-link`) |
| Flow | **Invitation Enrollment** |
| Single use | **On** (recommended) |
| Expires | Set appropriately |
4. **Custom attributes** — either leave empty `{}` or include only the name:
```yaml
name: Jane Smith
```
Do **not** include an `email` field — this prevents the automatic email from being sent.
5. Click **Create**
6. In the invitation list, **expand the row** to reveal the invitation link
7. Copy and share the link manually
---
## Custom Attributes Reference
| Field | Required | Purpose |
|-------|----------|---------|
| `name` | No | Pre-fills the invitee's display name (if enrollment flow uses it) |
| `email` | No | Triggers automatic email delivery. Omit for link-sharing mode |
Only `email` affects system behavior. Any other fields are stored as metadata on the invitation.
---
## Best Practices
### Expiry
- **Email invitations:** 7 days is reasonable — gives time for the email to arrive and the user to act
- **Link invitations:** 24–48 hours if sharing in real-time; 7 days if async
- **Never use no-expiry** — orphaned invitations are a security risk
### Single Use
- **Always enable** for individual invitations — prevents link reuse after the invitee enrolls
- Only disable if you're creating a batch enrollment link for a group (rare)
### Naming Convention
Use `firstname-lastname-YYYY-MM` or `purpose-YYYY-MM` for easy identification:
- `jane-smith-2026-02`
- `jodie-media-access-2026-02`
- `batch-beta-testers-2026-03`
---
## After Enrollment
New users are created under the `users/enrolled` path. To grant them access to services:
1. Navigate to **Directory → Groups**
2. Add the user to the appropriate group(s):
| Group | Grants Access To |
|-------|-----------------|
| media-users | Jellyfin, Jellyseer, PeerTube |
| ai-users | Open WebUI |
| cloud-users | Immich, Nextcloud |
| communication-users | Mailcow, Matrix |
See the [Access Groups runbook](authentik-access-groups.md) for detailed group management procedures.
---
## Troubleshooting
### Email not sent (email mode)
1. **Check custom attributes** — the `email` field must be present and correctly formatted
2. **Check SMTP** — verify Authentik can send email:
```bash
ssh root@100.64.0.1
docker exec authentik-server ak test_email matt@echo6.co
```
3. **Check Mailcow authsource** — if SMTP auth fails, the no-reply@echo6.co mailbox may have reverted to `generic-oidc`. See [Mailcow Create Mailbox runbook](mailcow-create-mailbox.md), Step 2
4. **Check Authentik logs**:
```bash
docker compose -f /opt/authentik/docker-compose.yml logs server --since 5m 2>&1 | grep -i email
```
### "Invalid invite/invite not found" when clicking link
- The invitation has expired or was already used (single-use)
- The invitation was deleted
- The `?itoken=` parameter is missing or malformed in the URL
### User enrolled but can't access any apps
- The user needs to be added to at least one service group (see "After Enrollment" above)
- By default, enrolled users have no group memberships
---
## Managing Existing Invitations
### View All Invitations
Admin UI → **Directory → Invitations** — shows all active invitations with name, expiry, and usage status.
### Delete an Invitation
Click the trash icon next to the invitation. This immediately invalidates the link — anyone who hasn't enrolled yet will see "Invalid invite."
### Via API
```bash
# List all invitations
curl -s "https://auth.echo6.co/api/v3/stages/invitation/invitations/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" | python3 -m json.tool
# Delete by PK
curl -s -X DELETE "https://auth.echo6.co/api/v3/stages/invitation/invitations/<PK>/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN"
```
---
## Checklist
```
[ ] Invitation created with correct flow (Invitation Enrollment)
[ ] Single use enabled
[ ] Expiry set appropriately
[ ] Email mode: email field in custom attributes, delivery confirmed
[ ] Link mode: link copied and shared manually
[ ] After enrollment: user added to appropriate groups
```
---
*Created: 2026-02-16*

353
runbooks/authentik-oidc-application.md Normal file
View file

@ -0,0 +1,353 @@
# Add Authentik OIDC to an Application
Fully automated via Authentik API. No web UI interaction required.
**Prerequisite:** DNS must already exist for the service (run expose-service-contabo.md or expose-service-home.md first).
**Authentik instance:** https://auth.echo6.co (Contabo, 100.64.0.6)
---
## Inputs
Prompt the user for all of these before executing any steps:
```
SERVICE_NAME= # Human-readable (e.g., "Vaultwarden", "Headplane")
SERVICE_SLUG= # URL-safe, lowercase (e.g., "vaultwarden", "headplane")
SERVICE_URL= # Base URL (e.g., "https://vault.echo6.co")
OIDC_CALLBACK_PATH= # App's OIDC callback (e.g., "/oidc/callback")
NEEDS_OFFLINE_ACCESS= # yes/no — does the app need refresh tokens?
CLIENT_TYPE= # confidential (server-side) or public (SPA/mobile)
```
The redirect URI is `${SERVICE_URL}${OIDC_CALLBACK_PATH}`.
### When to set NEEDS_OFFLINE_ACCESS=yes
- The app stores sessions that must survive service restarts (Headscale, Vaultwarden)
- The app uses refresh tokens for long-lived sessions
- Users shouldn't have to re-authenticate after every restart
### Reserved slugs
These conflict with Authentik's internal OAuth2 endpoints and **cannot be used**: `authorize`, `token`, `device`, `userinfo`, `introspect`, `revoke`.
---
## Step 1: Get API Token
Create an API token from the Authentik admin account. This only needs to happen once — reuse the token across all OIDC setups.
```bash
ssh root@100.64.0.6 "docker exec authentik-server \
ak create_token --user akadmin --identifier oidc-automation --expiring 2>/dev/null \
|| echo 'Token may already exist — check credentials file'"
```
If the token already exists, retrieve it from `/home/zvx/projects/.ref/credentials` (`AUTHENTIK_API_TOKEN`).
Store it for use in subsequent steps:
```bash
AK_TOKEN="<token>"
AK_API="https://auth.echo6.co/api/v3"
```
---
## Step 2: Look Up Authentik Internal IDs
The API requires UUIDs for flows, scope mappings, and signing keys. These are stable per Authentik instance but must be looked up once.
### Authorization flow
```bash
ssh root@100.64.0.6 "curl -s \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/flows/instances/?slug=default-provider-authorization-implicit-consent' \
| jq -r '.results[0].pk'"
```
Store as `AUTH_FLOW_PK`.
### Scope mappings
```bash
# Get all scope mapping UUIDs at once
ssh root@100.64.0.6 "curl -s \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/propertymappings/provider/scope/?ordering=scope_name' \
| jq -r '.results[] | select(.scope_name == \"openid\" or .scope_name == \"email\" or .scope_name == \"profile\" or .scope_name == \"offline_access\") | \"\(.scope_name): \(.pk)\"'"
```
Store each UUID: `SCOPE_OPENID_PK`, `SCOPE_EMAIL_PK`, `SCOPE_PROFILE_PK`, `SCOPE_OFFLINE_PK`.
### Signing key
```bash
ssh root@100.64.0.6 "curl -s \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/crypto/certificatekeypairs/?name=authentik+Self-signed+Certificate&has_key=true' \
| jq -r '.results[0].pk'"
```
Store as `SIGNING_KEY_PK`.
### Gate
All five values must be non-null. If any are missing, Authentik's default objects may not have been created yet — check that the instance is healthy.
---
## Step 3: Create the OAuth2 Provider
Build the scope mappings array based on whether offline_access is needed:
```bash
# Base scopes (always included)
SCOPES="[\"$SCOPE_OPENID_PK\", \"$SCOPE_EMAIL_PK\", \"$SCOPE_PROFILE_PK\"]"
# Add offline_access if needed
if [ "$NEEDS_OFFLINE_ACCESS" = "yes" ]; then
SCOPES="[\"$SCOPE_OPENID_PK\", \"$SCOPE_EMAIL_PK\", \"$SCOPE_PROFILE_PK\", \"$SCOPE_OFFLINE_PK\"]"
fi
```
Create the provider:
```bash
PROVIDER_RESPONSE=$(ssh root@100.64.0.6 "curl -s \
-X POST '$AK_API/providers/oauth2/' \
-H 'Authorization: Bearer $AK_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
\"name\": \"$SERVICE_NAME\",
\"authorization_flow\": \"$AUTH_FLOW_PK\",
\"client_type\": \"$CLIENT_TYPE\",
\"redirect_uris\": [{
\"matching_mode\": \"strict\",
\"url\": \"${SERVICE_URL}${OIDC_CALLBACK_PATH}\"
}],
\"signing_key\": \"$SIGNING_KEY_PK\",
\"property_mappings\": $SCOPES,
\"access_token_validity\": \"hours=1\",
\"refresh_token_validity\": \"days=30\"
}'")
# Extract the values we need
PROVIDER_PK=$(echo "$PROVIDER_RESPONSE" | jq -r '.pk')
CLIENT_ID=$(echo "$PROVIDER_RESPONSE" | jq -r '.client_id')
CLIENT_SECRET=$(echo "$PROVIDER_RESPONSE" | jq -r '.client_secret')
echo "Provider PK: $PROVIDER_PK"
echo "Client ID: $CLIENT_ID"
echo "Client Secret: $CLIENT_SECRET"
```
### Gate
`PROVIDER_PK` must be a number (not null or an error). If the API returns an error, common causes:
- **Duplicate name** — a provider with this name already exists
- **Invalid flow PK** — the authorization flow UUID is wrong
- **Invalid scope PK** — one of the scope mapping UUIDs is wrong
---
## Step 4: Create the Application
```bash
ssh root@100.64.0.6 "curl -s \
-X POST '$AK_API/core/applications/' \
-H 'Authorization: Bearer $AK_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
\"name\": \"$SERVICE_NAME\",
\"slug\": \"$SERVICE_SLUG\",
\"provider\": $PROVIDER_PK,
\"meta_launch_url\": \"$SERVICE_URL\"
}' | jq '{name: .name, slug: .slug, provider: .provider}'"
```
### Gate
Response must include the slug and provider PK. If it fails, the slug may already be in use.
---
## Step 5: Verify Authentik Side
### Discovery endpoint
```bash
curl -s "https://auth.echo6.co/application/o/$SERVICE_SLUG/.well-known/openid-configuration" | jq '{issuer, authorization_endpoint, token_endpoint, jwks_uri}'
```
Must return all four fields with valid URLs.
### JWKS endpoint
```bash
curl -s "https://auth.echo6.co/application/o/$SERVICE_SLUG/jwks/" | jq '.keys | length'
```
Must return at least `1`. If it returns `0`, the signing key was not attached to the provider — go back and fix Step 3.
---
## Step 6: Configure the Application
This step varies per application. Use the Client ID, Client Secret, and issuer URL from above.
### OIDC endpoints (all derived from the slug)
```
Issuer: https://auth.echo6.co/application/o/$SERVICE_SLUG/
Authorize: https://auth.echo6.co/application/o/authorize/
Token: https://auth.echo6.co/application/o/token/
User Info: https://auth.echo6.co/application/o/userinfo/
JWKS: https://auth.echo6.co/application/o/$SERVICE_SLUG/jwks/
```
Most apps only need the **Issuer** (or Discovery URL) plus Client ID and Client Secret. The app auto-discovers the rest.
### Common config patterns
**Environment variables (Docker):**
```bash
OIDC_ISSUER=https://auth.echo6.co/application/o/$SERVICE_SLUG/
OIDC_CLIENT_ID=$CLIENT_ID
OIDC_CLIENT_SECRET=$CLIENT_SECRET
OIDC_SCOPES="openid email profile" # add offline_access if needed
OIDC_REDIRECT_URI=${SERVICE_URL}${OIDC_CALLBACK_PATH}
```
**Config file (YAML):**
```yaml
oidc:
issuer: "https://auth.echo6.co/application/o/$SERVICE_SLUG/"
client_id: "$CLIENT_ID"
client_secret: "$CLIENT_SECRET"
scope: ["openid", "profile", "email"] # add "offline_access" if needed
```
### Common alternate names for these values
| Concept | Names you'll see |
|---------|-----------------|
| Issuer | `authority`, `issuer_url`, `sso_authority`, `provider_url` |
| Client ID | `client_id`, `oidc_client_id`, `sso_client_id` |
| Client Secret | `client_secret`, `oidc_client_secret`, `sso_client_secret` |
| Redirect URI | `redirect_uri`, `callback_url`, `oidc_redirect_url` |
| Scopes | `scope`, `scopes`, `oidc_scopes`, `sso_scopes` |
---
## Step 7: Test Login
1. Open `$SERVICE_URL` in a browser
2. Click SSO / OIDC login
3. Should redirect to `auth.echo6.co` → authenticate → redirect back to the app
4. Verify user info is correct (email, display name)
---
## Step 8: Store Credentials
```bash
cat >> /home/zvx/projects/.ref/credentials << EOF
# $SERVICE_NAME OIDC
${SERVICE_SLUG^^}_OIDC_CLIENT_ID=$CLIENT_ID
${SERVICE_SLUG^^}_OIDC_CLIENT_SECRET=$CLIENT_SECRET
${SERVICE_SLUG^^}_OIDC_ISSUER=https://auth.echo6.co/application/o/$SERVICE_SLUG/
EOF
```
---
## Troubleshooting
### SSO login redirects back to login page (loop)
Check in order:
1. **Access token validity too short** — increase to at least `hours=1`
2. **Missing `offline_access` scope** — app can't refresh tokens, session expires immediately
3. **Missing signing key** — JWKS endpoint returns empty, app can't verify tokens
Debug via API:
```bash
ssh root@100.64.0.6 "curl -s \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/providers/oauth2/?search=$SERVICE_NAME' \
| jq '.results[0] | {name, client_id, signing_key, access_token_validity, refresh_token_validity, property_mappings}'"
```
Or via ak shell:
```bash
ssh root@100.64.0.6 "docker exec authentik-server ak shell -c \"
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.get(name='$SERVICE_NAME')
print(f'Access Token: {p.access_token_validity}')
print(f'Refresh Token: {p.refresh_token_validity}')
print(f'Signing Key: {p.signing_key}')
print(f'Scopes: {list(p.property_mappings.values_list(\\\"scope_name\\\", flat=True))}')
\""
```
### "Failed to discover OpenID provider" / discovery error
1. JWKS endpoint is empty → signing key missing from provider
2. Authentik unreachable from the app → test with `curl` from the app's host
3. Wrong issuer URL → must include trailing slash, must match the slug exactly
### "Invalid redirect URI"
The redirect URI in the app config must **exactly** match what's in Authentik — scheme, trailing slashes, path, everything.
### User authenticated but gets "access denied"
User isn't authorized for the application. By default all authenticated users have access. If you've added group restrictions via policy bindings, verify the user is in the correct group:
```bash
ssh root@100.64.0.6 "curl -s \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/core/applications/$SERVICE_SLUG/' \
| jq '{name, slug, policy_engine_mode}'"
```
### Token/session breaks after service restart
Missing `offline_access` scope. Without refresh tokens, sessions only last as long as the access token validity.
### Delete and recreate (nuclear option)
```bash
# Delete application first (it references the provider)
ssh root@100.64.0.6 "curl -s -X DELETE \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/core/applications/$SERVICE_SLUG/'"
# Then delete provider
ssh root@100.64.0.6 "curl -s -X DELETE \
-H 'Authorization: Bearer $AK_TOKEN' \
'$AK_API/providers/oauth2/$PROVIDER_PK/'"
```
Then re-run from Step 3.
---
## Quick Reference: Existing OIDC Applications
| Application | Slug | Redirect URI | offline_access |
|-------------|------|-------------|----------------|
| Headscale | `headscale` | `https://vpn.echo6.co/oidc/callback` | Yes |
| Headplane | `headplane` | `https://vpn.echo6.co/admin/oidc/callback` | No |
| Vaultwarden | `vaultwarden` | `https://vault.echo6.co/identity/connect/oidc-signin` | Yes |

314
runbooks/authentik-upgrade.md Normal file
View file

@ -0,0 +1,314 @@
# Authentik: Major Version Upgrade
Upgrade Authentik between major versions on Contabo. Covers backup, upgrade, verification, and rollback.
---
## When to Use This
Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
---
## Prerequisites
- SSH access to Contabo (`ssh root@100.64.0.1`)
- Authentik compose directory: `/opt/authentik/`
- Current version: check with `docker exec authentik-server ak --version`
---
## Inputs
```
CURRENT_VERSION=2025.12.4 # Current running version
TARGET_VERSION=2026.2.1 # Version to upgrade to
```
---
## Step 1: Check Release Notes
Before upgrading, read the release notes for **every major version between current and target**:
```
https://docs.goauthentik.io/docs/releases/
```
Look for:
- **Breaking changes** — removed features, changed defaults, API changes
- **Dependency changes** — added/removed services (e.g., Redis removed in 2025.10)
- **Configuration changes** — new required env vars, changed mount paths
- **Database migrations** — large migrations that may take time
### Known Breaking Changes (Reference)
| Version | Change | Impact |
|---------|--------|--------|
| 2025.10 | Redis completely removed | Delete redis service + all `AUTHENTIK_REDIS` env vars |
| 2025.10 | Default email scope returns `email_verified: false` | Use custom scope mapping (PK `02c22323`) that forces `true` |
| 2025.10 | Worker requires `user: root` | Add `user: root` to worker service in compose |
| 2025.12 | Stage creation endpoints moved | `stages/<type>/stages/` for POST (some types) |
---
## Step 2: Backup
### 2a. Snapshot Contabo (if Proxmox-managed)
If Contabo were a Proxmox VM, take a snapshot. Since it's a bare-metal VPS, skip this and rely on the file-level backups below.
### 2b. PostgreSQL Dump
```bash
ssh root@100.64.0.1
cd /opt/authentik
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
docker exec authentik-postgres \
pg_dump -U authentik -d authentik \
--clean --if-exists \
> /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
ls -lh /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
```
### 2c. Compose Directory Backup
```bash
cp -a /opt/authentik /opt/authentik.bak_${TIMESTAMP}
```
This preserves `docker-compose.yml`, `.env`, `certs/`, and any custom files.
### 2d. Record Current State
```bash
# Save current version
docker exec authentik-server ak --version
# Save current provider list (for post-upgrade comparison)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/" \
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)" \
| python3 -c "import sys,json; [print(f'{p[\"pk\"]:3d} {p[\"name\"]}') for p in json.load(sys.stdin)['results']]"
```
---
## Step 3: Apply Compose Changes
Review release notes and update `docker-compose.yml` **before** pulling the new image:
```bash
cd /opt/authentik
nano docker-compose.yml
```
Common changes by version:
**Removing Redis (2025.10+):**
```yaml
# DELETE the redis service entirely
# DELETE these env vars from server + worker:
# AUTHENTIK_REDIS__HOST
# AUTHENTIK_REDIS__PORT
```
**Worker user requirement (2025.10+):**
```yaml
services:
worker:
user: root # ADD this line
```
**New env vars:**
Check release notes for any new required `AUTHENTIK_*` env vars. Add to both server and worker services.
---
## Step 4: Upgrade
```bash
cd /opt/authentik
# Update image tag in docker-compose.yml
# Change: image: ghcr.io/goauthentik/server:CURRENT_VERSION
# To: image: ghcr.io/goauthentik/server:TARGET_VERSION
nano docker-compose.yml
# Pull new image
docker compose pull
# Stop and recreate containers (migrations run automatically on start)
docker compose down && docker compose up -d
# Watch logs for migration progress
docker compose logs -f server --since 1m
```
Migrations may take 1–5 minutes depending on database size. Wait until you see:
```
Starting gunicorn
```
---
## Step 5: Verify
### 5a. Version Check
```bash
docker exec authentik-server ak --version
# Should show TARGET_VERSION
```
### 5b. API Token
API tokens are sometimes invalidated during major upgrades. Test:
```bash
curl -s -o /dev/null -w "%{http_code}" \
"https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $(grep AUTHENTIK_API_TOKEN /home/zvx/projects/.ref/credentials | cut -d= -f2)"
```
If this returns `403`, regenerate the token:
```bash
docker exec -i authentik-server ak shell <<'PYEOF'
from authentik.core.models import Token, TokenIntents, User
user = User.objects.get(username="akadmin")
Token.objects.filter(identifier="claude-api-token").delete()
t = Token(identifier="claude-api-token", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
t.save()
print(t.key)
PYEOF
```
Update `/home/zvx/projects/.ref/credentials` with the new token.
### 5c. Email Scope Mapping
Check that the custom email scope (`02c22323`) is still assigned to providers. The default scope (`096b0d6f`) may return `email_verified: false` in 2025.10+:
```bash
# Check a canary provider (Forgejo, PK 2)
curl -s "https://auth.echo6.co/api/v3/providers/oauth2/2/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
| python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['property_mappings'], indent=2))"
```
Verify `02c22323-da89-457a-bc12-7f4dd6a3d8ab` is in the list. If missing, re-add it to all providers.
### 5d. Spot-Check OAuth2 Apps
Test SSO login on 2–3 apps as canaries:
1. **Forgejo** — `https://forge.echo6.co` → click "Sign in with Authentik"
2. **Proxmox** — `https://proxmox.echo6.co` → select OpenID realm
Both should redirect to Authentik, authenticate, and return to the app.
### 5e. SMTP Delivery
```bash
docker exec authentik-server ak test_email matt@echo6.co
```
Check that the test email arrives. If SMTP auth fails, verify the no-reply@echo6.co mailbox authsource (see Mailcow runbook).
### 5f. Invitation System
Create a test invitation in Admin UI → Directory → Invitations with `email: matt@echo6.co` in custom attributes. Confirm the email is sent. Delete the test invitation after.
---
## Rollback
If the upgrade breaks critical functionality:
### Option A: Roll Back Image (Quick)
```bash
cd /opt/authentik
# Revert image tag to previous version
nano docker-compose.yml
# Change TARGET_VERSION back to CURRENT_VERSION
docker compose down && docker compose up -d
```
This works if no breaking database migrations occurred. Check logs for migration errors.
### Option B: Full Restore (Nuclear)
```bash
cd /opt
# Stop everything
cd /opt/authentik && docker compose down
# Restore compose directory
rm -rf /opt/authentik
cp -a /opt/authentik.bak_${TIMESTAMP} /opt/authentik
# Restore database
cd /opt/authentik
docker compose up -d postgres
sleep 10
docker exec -i authentik-postgres \
psql -U authentik -d authentik \
< /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
docker compose up -d
```
### Post-Rollback
- Verify the old version is running: `docker exec authentik-server ak --version`
- Test SSO login on Forgejo
- Test SMTP: `docker exec authentik-server ak test_email matt@echo6.co`
---
## Cleanup
After confirming the upgrade is stable (wait at least 24 hours):
```bash
# Remove backup
rm -rf /opt/authentik.bak_${TIMESTAMP}
# Keep the SQL dump for archival (or remove if space is needed)
# rm /opt/authentik/backups/authentik_pre_upgrade_${TIMESTAMP}.sql
# Prune old Docker images
docker image prune -a --filter "until=168h"
```
---
## Checklist
```
[ ] Release notes reviewed for all versions between current and target
[ ] PostgreSQL dump taken
[ ] Compose directory backed up
[ ] Compose changes applied (removed/added services, env vars, user directives)
[ ] Image tag updated and pulled
[ ] Containers recreated, migrations completed
[ ] Version confirmed
[ ] API token tested (regenerated if needed)
[ ] Custom email scope verified on providers
[ ] SSO login tested on Forgejo + Proxmox
[ ] SMTP delivery tested
[ ] Invitation system tested
[ ] Backup files cleaned up (after 24h+ stability)
```
---
*Created: 2026-02-16*

279
runbooks/binary-wrapper-interception.md Normal file
View file

@ -0,0 +1,279 @@
# Binary Wrapper Interception
Transparently intercept a CLI binary with a wrapper script that adds pre-flight logic (routing, validation, logging) before exec-ing the real binary. The caller — whether a service, cron job, or another script — never knows the difference.
Use this when you need to modify the behavior of a tool that's called by a system you don't control (e.g., a runner, scheduler, or third-party service), without changing the caller's config or code.
---
## Prerequisites
- The binary to intercept is installed and working
- You have root/sudo access on the target machine
- The caller invokes the binary by absolute path or via PATH lookup
---
## Inputs
Prompt the user for all of these before executing:
```
TARGET_HOST= # SSH alias or IP (e.g., cortex). Use "localhost" if local.
BINARY_NAME= # Name of the binary to intercept (e.g., "whisper-ctranslate2")
BINARY_PATH= # Full path to the binary (e.g., "/usr/local/bin/whisper-ctranslate2")
WRAPPER_NAME= # Name for the wrapper script (e.g., "whisper-smart")
WRAPPER_DIR= # Directory for the wrapper (e.g., "/usr/local/bin")
REAL_SUFFIX= # Suffix for the renamed real binary (default: "-real")
```
---
## Step 1: Locate the Real Binary
Find the actual binary or symlink that will be intercepted.
```bash
ssh $TARGET_HOST "ls -la $BINARY_PATH && file $BINARY_PATH"
```
If it's already a symlink, follow it to the real target:
```bash
ssh $TARGET_HOST "readlink -f $BINARY_PATH"
```
### Gate
Must return a valid file. Record the real binary location — you'll need it for the rename.
---
## Step 2: Rename the Real Binary
Move the original binary out of the way so the wrapper can take its place.
```bash
ssh $TARGET_HOST "sudo mv $BINARY_PATH ${BINARY_PATH}${REAL_SUFFIX}"
```
If the original was a symlink (e.g., pip-installed Python tool):
```bash
# Preserve the symlink target
REAL_TARGET=$(ssh $TARGET_HOST "readlink -f $BINARY_PATH")
ssh $TARGET_HOST "sudo rm $BINARY_PATH && sudo ln -s $REAL_TARGET ${BINARY_PATH}${REAL_SUFFIX}"
```
### Gate
```bash
ssh $TARGET_HOST "ls -la ${BINARY_PATH}${REAL_SUFFIX}"
ssh $TARGET_HOST "${BINARY_PATH}${REAL_SUFFIX} --version 2>/dev/null || ${BINARY_PATH}${REAL_SUFFIX} --help 2>/dev/null | head -1"
```
The renamed binary must exist and be executable. If not, undo immediately:
```bash
ssh $TARGET_HOST "sudo mv ${BINARY_PATH}${REAL_SUFFIX} $BINARY_PATH"
```
---
## Step 3: Write the Wrapper Script
Create the wrapper at `$WRAPPER_DIR/$WRAPPER_NAME`. The wrapper must:
1. Accept all original arguments (`$@`)
2. Perform pre-flight logic (inspection, routing, logging)
3. `exec` the real binary with (possibly modified) arguments
4. Never silently swallow errors — if pre-flight fails, exit with a meaningful code
Template:
```bash
#!/bin/bash
# Wrapper for $BINARY_NAME — transparently intercepts calls
# Real binary at: ${BINARY_PATH}${REAL_SUFFIX}
LOGFILE="/tmp/${WRAPPER_NAME}.log"
# ──── Pre-flight logic ────
# Add your inspection, routing, or validation here.
# Example: inspect input files, check resource availability, choose parameters.
# Parse arguments to find relevant inputs (file paths, flags, etc.)
# This section is use-case specific.
# ──── Logging ────
echo "[WRAPPER] $(date) args: $@" >> "$LOGFILE"
# ──── Execute real binary ────
# Use exec to replace this process — caller sees the real binary's exit code,
# stdout, stderr, and signal handling as if wrapper didn't exist.
exec ${BINARY_PATH}${REAL_SUFFIX} "$@"
```
Deploy the wrapper:
```bash
ssh $TARGET_HOST "sudo tee $WRAPPER_DIR/$WRAPPER_NAME > /dev/null << 'WRAPPER'
<paste wrapper script here>
WRAPPER
sudo chmod +x $WRAPPER_DIR/$WRAPPER_NAME"
```
### Gate
```bash
ssh $TARGET_HOST "ls -la $WRAPPER_DIR/$WRAPPER_NAME && head -1 $WRAPPER_DIR/$WRAPPER_NAME"
```
Must show executable permissions and `#!/bin/bash` shebang.
---
## Step 4: Install the Symlink
Replace the original binary path with a symlink to the wrapper.
```bash
ssh $TARGET_HOST "sudo ln -sf $WRAPPER_DIR/$WRAPPER_NAME $BINARY_PATH"
```
### Gate
```bash
ssh $TARGET_HOST "ls -la $BINARY_PATH"
```
Must show: `$BINARY_PATH -> $WRAPPER_DIR/$WRAPPER_NAME`
Verify the full chain:
```bash
ssh $TARGET_HOST "ls -la $BINARY_PATH && ls -la ${BINARY_PATH}${REAL_SUFFIX}"
```
Should show:
```
BINARY_PATH -> WRAPPER_DIR/WRAPPER_NAME (wrapper)
BINARY_PATH-real -> /path/to/actual/binary (real binary)
```
---
## Step 5: Test the Interception
Run the binary as the caller would. The wrapper should intercept transparently.
```bash
# Direct invocation
ssh $TARGET_HOST "$BINARY_PATH --version"
# Check wrapper log
ssh $TARGET_HOST "tail -5 /tmp/${WRAPPER_NAME}.log"
```
The `--version` output should come from the real binary. The log should show the wrapper fired.
Test with actual workload arguments:
```bash
ssh $TARGET_HOST "$BINARY_PATH <typical args here>"
ssh $TARGET_HOST "tail -1 /tmp/${WRAPPER_NAME}.log"
```
### Gate
Both must succeed. If the binary fails or produces different output than before, the wrapper has a bug — check argument passing (quoting, `$@` vs `$*`).
---
## Step 6: Verify Service Integration
If the binary is called by a service (systemd, cron, etc.), restart that service and confirm it picks up the wrapper.
```bash
ssh $TARGET_HOST "sudo systemctl restart <service-name>"
ssh $TARGET_HOST "sleep 5 && tail -5 /tmp/${WRAPPER_NAME}.log"
```
The log should show entries from the service's invocations, not just your manual tests.
---
## Rollback
To remove the wrapper and restore the original binary:
```bash
ssh $TARGET_HOST "sudo rm $BINARY_PATH && sudo mv ${BINARY_PATH}${REAL_SUFFIX} $BINARY_PATH"
# Or if the original was a symlink:
ssh $TARGET_HOST "sudo rm $BINARY_PATH && sudo ln -s <original-target> $BINARY_PATH"
```
No service restart needed — next invocation hits the real binary directly.
---
## Key Principles
1. **`exec` is mandatory.** Without `exec`, the wrapper runs the binary as a child process, which breaks signal handling (SIGTERM won't reach the real binary) and doubles PID usage. `exec` replaces the wrapper process entirely.
2. **Use `"$@"` not `$@` or `$*`.** Quoted `"$@"` preserves argument boundaries. Unquoted `$@` splits arguments with spaces. `$*` merges all arguments into one string.
3. **Appended flags override earlier ones.** Many CLI tools (argparse, getopt) use last-value-wins for duplicate flags. The wrapper can append `--flag value` after `"$@"` to force overrides without removing the caller's original flags.
4. **Exit codes matter.** If pre-flight fails, exit with a non-zero code that the caller understands. Some callers retry on specific exit codes (e.g., PeerTube runner retries on exit 1).
5. **Log to /tmp, not to the service's log directory.** The wrapper log is a debug artifact, not part of the service's data. `/tmp` is cleaned on reboot, which is fine for wrapper logs.
---
## Troubleshooting
### Wrapper not being called
Check the symlink chain: `ls -la $BINARY_PATH`. If the service uses a hardcoded absolute path that bypasses PATH, the symlink might be in the wrong location.
### Arguments with spaces break
Use `"$@"` (quoted) in the exec line, not `$@` (unquoted).
### Service fails after wrapper install
Check the wrapper's shebang (`#!/bin/bash`), permissions (`chmod +x`), and that `exec` is present. Without exec, the wrapper may exit before the binary finishes.
### Wrapper log is empty
The service might be calling a different path than expected. Check: `which $BINARY_NAME` and compare with what the service config specifies.
---
## Usage Examples
### Whisper transcription routing (PeerTube runner on cortex)
The PeerTube remote runner calls `whisper-ctranslate2` for auto-captioning. The smart wrapper intercepts this to route short videos to GPU and long videos to CPU.
```
BINARY_NAME=whisper-ctranslate2
BINARY_PATH=/usr/local/bin/whisper-ctranslate2
WRAPPER_NAME=whisper-smart
REAL_SUFFIX=-real
Symlink chain:
/usr/local/bin/whisper-ctranslate2 → /usr/local/bin/whisper-smart
/usr/local/bin/whisper-ctranslate2-real → /home/zvx/.local/bin/whisper-ctranslate2
Wrapper logic:
- ffprobe audio duration from first non-flag argument
- < 1hr → exec with --device cuda --compute_type float16
- >= 1hr → exec with --device cpu --compute_type int8
- Appends --model medium after $@ (last-value-wins override)
```
---
*Last updated: 2026-02-17*

183
runbooks/contabo-configs.md
View file

@ -1,183 +0,0 @@
# Contabo VPS Current Configurations
**Server:** 5.189.158.149 / 100.64.0.4
**Last Updated:** 2026-02-05
---
## Caddy Configuration
**File:** `/etc/caddy/Caddyfile`
```caddyfile
# Global options
{
email admin@echo6.co
admin off
}
# Main Mailcow hostname
mail.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
read_timeout 3600s
write_timeout 3600s
}
}
}
# Autodiscover for Outlook
autodiscover.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
}
}
}
# Autoconfig for Thunderbird
autoconfig.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
}
}
}
# Headscale VPN + Headplane Admin
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3100
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
# Authentik SSO
auth.echo6.co {
reverse_proxy 127.0.0.1:9000
}
# Forgejo Git Forge
forge.echo6.co {
reverse_proxy 127.0.0.1:3001
}
# Vaultwarden Password Manager
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### Commands
```bash
# Validate
caddy validate --config /etc/caddy/Caddyfile
# Restart (admin off, so reload won't work)
systemctl restart caddy
# Logs
journalctl -u caddy -f
```
---
## dnsmasq Split DNS Configuration
**File:** `/etc/dnsmasq.d/tailscale-dns.conf`
```conf
# DNSmasq config for Tailscale Split DNS
# Listen only on Tailscale interface
listen-address=100.64.0.4
bind-interfaces
# Upstream DNS servers
server=1.1.1.1
server=8.8.8.8
# Local records for echo6.co services (route through Tailscale)
address=/forge.echo6.co/100.64.0.4
address=/auth.echo6.co/100.64.0.4
address=/mail.echo6.co/100.64.0.4
address=/vpn.echo6.co/100.64.0.4
address=/docs.echo6.co/100.64.0.4
address=/vault.echo6.co/100.64.0.4
address=/stream.echo6.co/100.64.0.7
address=/notes.echo6.co/100.64.0.22
# Don't read /etc/hosts
no-hosts
# Cache size
cache-size=1000
# Log queries for debugging
log-queries
```
### Commands
```bash
# Restart
systemctl restart dnsmasq
# Status
systemctl status dnsmasq
# Test resolution
dig +short vault.echo6.co @100.64.0.4
```
---
## Port Mappings Summary
| Service | Container Port | Host Binding | Caddy Proxy |
|---------|---------------|--------------|-------------|
| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
| Forgejo SSH | 22 | 0.0.0.0:2222 | Direct |
| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
---
## DNS Records (GoDaddy → Contabo)
| Subdomain | IP | Service |
|-----------|-----|---------|
| auth | 5.189.158.149 | Authentik |
| forge | 5.189.158.149 | Forgejo |
| mail | 5.189.158.149 | Mailcow |
| vpn | 5.189.158.149 | Headscale |
| vault | 5.189.158.149 | Vaultwarden |
| autodiscover | 5.189.158.149 | Mailcow |
| autoconfig | 5.189.158.149 | Mailcow |
---
## Split DNS Mappings (Tailscale)
| Domain | Tailscale IP | Server |
|--------|-------------|--------|
| auth.echo6.co | 100.64.0.4 | Contabo |
| forge.echo6.co | 100.64.0.4 | Contabo |
| mail.echo6.co | 100.64.0.4 | Contabo |
| vpn.echo6.co | 100.64.0.4 | Contabo |
| vault.echo6.co | 100.64.0.4 | Contabo |
| docs.echo6.co | 100.64.0.4 | Contabo |
| stream.echo6.co | 100.64.0.7 | PeerTube |
| notes.echo6.co | 100.64.0.22 | Cloud |
---
*Last updated: 2026-02-05*

259
runbooks/ct-runbook.md Normal file
View file

@ -0,0 +1,259 @@
# Proxmox CT/LXC Provisioning Runbook
Every container gets the same baseline: local user, Tailscale, SSH, Docker, and common tools. No exceptions.
---
## Prerequisites
- Proxmox VE host with Ubuntu 24.04 LXC template downloaded
- Tailscale auth key (reusable, from https://login.tailscale.com/admin/settings/keys)
- SSH access to Proxmox host
If you don't have the template cached yet:
```bash
pveam update
pveam download local system ubuntu-24.04-standard_24.04-2_amd64.tar.zst
```
---
## 1. Create the Container
Pick the next available CTID. Adjust `--memory`, `--cores`, and `--rootfs` to fit the workload.
```bash
# Variables — edit these per container
CTID=110
HOSTNAME="mycontainer"
STORAGE="local-lvm" # or zfs-pool, ceph, etc.
DISK_SIZE=8 # GB
MEMORY=2048 # MB
CORES=2
BRIDGE="vmbr0"
pct create $CTID local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \
--hostname $HOSTNAME \
--storage $STORAGE \
--rootfs ${STORAGE}:${DISK_SIZE} \
--memory $MEMORY \
--cores $CORES \
--net0 name=eth0,bridge=${BRIDGE},ip=dhcp \
--unprivileged 1 \
--features nesting=1,keyctl=1 \
--onboot 1 \
--start 1
```
`nesting=1` is required for Docker. `keyctl=1` prevents keyring errors in systemd containers.
Wait a few seconds for the container to boot, then enter it:
```bash
pct enter $CTID
```
Everything from here on runs **inside the container**.
---
## 2. Base System Update
```bash
apt update && apt upgrade -y
```
---
## 3. Common Tools
```bash
apt install -y \
curl \
wget \
vim \
htop \
git \
unzip \
jq \
net-tools \
dnsutils \
ca-certificates \
gnupg \
lsb-release \
sudo \
sshpass
```
---
## 4. Create User
```bash
useradd -m -s /bin/bash -G sudo zvx
echo "zvx:7redditGold" | chpasswd
```
Verify:
```bash
su - zvx -c "whoami && sudo -l"
```
---
## 5. SSH Configuration
SSH should already be running in the Ubuntu 24.04 template, but make sure password auth is enabled for sshpass workflows:
```bash
# Ensure SSH is installed and running
apt install -y openssh-server
systemctl enable --now ssh
# Allow password auth (needed for sshpass)
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
systemctl restart ssh
```
Test from the Proxmox host (exit the container first):
```bash
CT_IP=$(pct exec $CTID -- hostname -I | awk '{print $1}')
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=accept-new zvx@$CT_IP "echo 'SSH OK'"
```
---
## 6. Install Docker
```bash
# Add Docker's official GPG key and repo
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
> /etc/apt/sources.list.d/docker.list
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Add zvx to docker group (no sudo needed for docker commands)
usermod -aG docker zvx
# Verify
docker run --rm hello-world
```
If Docker fails to start with an AppArmor or permissions error, confirm `nesting=1` is set on the container (Step 1). You can check/fix from the Proxmox host:
```bash
pct set $CTID --features nesting=1,keyctl=1
pct reboot $CTID
```
---
## 7. Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
```
Bring it up with your auth key:
```bash
# Replace with your actual auth key
tailscale up --authkey=tskey-auth-XXXXXXXXXXXX --ssh
```
If you don't have an auth key handy, run without `--authkey` and it will print a URL to authenticate in a browser:
```bash
tailscale up --ssh
```
Verify:
```bash
tailscale status
tailscale ip -4
```
The `--ssh` flag enables Tailscale SSH, which lets you SSH into the container over Tailscale without managing keys. The container will appear in your tailnet by its hostname.
---
## 8. Verification Checklist
Run this from inside the container to confirm everything:
```bash
echo "=== CT Provisioning Check ==="
echo ""
echo "Hostname: $(hostname)"
echo "User zvx: $(id zvx 2>/dev/null && echo 'OK' || echo 'MISSING')"
echo "sudo: $(sudo -l -U zvx 2>/dev/null | grep -q ALL && echo 'OK' || echo 'MISSING')"
echo "sshpass: $(which sshpass >/dev/null 2>&1 && echo 'OK' || echo 'MISSING')"
echo "SSH: $(systemctl is-active ssh)"
echo "Docker: $(docker --version 2>/dev/null || echo 'MISSING')"
echo "Tailscale: $(tailscale status --self 2>/dev/null | head -1 || echo 'NOT CONNECTED')"
echo "Tailscale IP: $(tailscale ip -4 2>/dev/null || echo 'N/A')"
```
Expected output — everything should say OK/active with a Tailscale IP:
```
=== CT Provisioning Check ===
Hostname: mycontainer
User zvx: uid=1000(zvx) gid=1000(zvx) groups=1000(zvx),27(sudo),998(docker) OK
sudo: OK
sshpass: OK
SSH: active
Docker: Docker version 27.x.x, build xxxxxxx
Tailscale: 100.x.x.x mycontainer tagged-devices linux -
Tailscale IP: 100.x.x.x
```
---
## Quick Reference (Copy/Paste Block)
For the impatient — the whole thing end to end after `pct enter`:
```bash
# Update + tools
apt update && apt upgrade -y
apt install -y curl wget vim htop git unzip jq net-tools dnsutils \
ca-certificates gnupg lsb-release sudo sshpass openssh-server
# User
useradd -m -s /bin/bash -G sudo zvx
echo "zvx:7redditGold" | chpasswd
# SSH
systemctl enable --now ssh
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
systemctl restart ssh
# Docker
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
> /etc/apt/sources.list.d/docker.list
apt update && apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
usermod -aG docker zvx
# Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --ssh
```

333
runbooks/gpu-cpu-fallback-routing.md Normal file
View file

@ -0,0 +1,333 @@
# GPU/CPU Fallback Routing
Route workloads to GPU or CPU based on pre-flight inspection of job properties (duration, file size, resolution, complexity). Small jobs go to GPU for speed; large jobs fall back to CPU to avoid VRAM exhaustion. Concurrent job control via flock prevents OOM kills — excess jobs fail fast and re-queue instead of competing for memory.
Use this when you have a GPU workload where some jobs exceed VRAM capacity, and the system needs to handle both small and large jobs without manual intervention or OOM kills.
---
## Prerequisites
- NVIDIA GPU with working drivers (`nvidia-smi` returns output)
- Both GPU and CPU execution paths available for the workload
- A probe tool to inspect job properties before execution (e.g., `ffprobe`, `mediainfo`, `file`, `wc`)
- A caller that retries on non-zero exit codes (scheduler, job queue, runner)
---
## Inputs
Prompt the user for all of these before executing:
```
TARGET_HOST= # Machine with GPU (e.g., cortex)
WORKLOAD_BINARY= # The tool that processes jobs (e.g., "whisper-ctranslate2-real")
PROBE_TOOL= # Tool to inspect job properties (e.g., "ffprobe", "mediainfo")
GPU_VRAM_MB= # Total VRAM available (e.g., 16384 for 16GB)
WORKLOAD_VRAM_MB= # VRAM used per GPU job (e.g., 3700)
WORKLOAD_RAM_MB= # RAM used per CPU job (e.g., 11000)
THRESHOLD_VALUE= # Cutoff for GPU vs CPU routing (e.g., 3600 for seconds)
THRESHOLD_UNIT= # What the threshold measures (e.g., "seconds", "bytes", "pixels")
MAX_GPU_JOBS= # Max concurrent GPU jobs (e.g., 2)
MAX_CPU_JOBS= # Max concurrent CPU jobs (e.g., 1)
GPU_ARGS= # Arguments for GPU execution (e.g., "--device cuda --compute_type float16")
CPU_ARGS= # Arguments for CPU execution (e.g., "--device cpu --compute_type int8")
```
---
## Step 1: Determine the Routing Threshold
Profile representative workloads to find the VRAM crossover point.
```bash
# Run a small job on GPU, monitor VRAM
ssh $TARGET_HOST "nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits"
# Run the workload...
# Check peak VRAM during execution
# Run a large job on GPU, watch for OOM
# If it OOM-kills or exceeds VRAM, that's your upper bound
```
The threshold should be set conservatively below the point where GPU jobs start failing. Common strategies:
| Workload Type | Probe Property | Typical Threshold |
|---------------|----------------|-------------------|
| Audio transcription | Duration (seconds) | 1-2 hours |
| Image generation | Resolution (megapixels) | Based on model VRAM curve |
| Video encoding | Duration × resolution | Derived from VRAM budget |
| LLM inference | Token count / context length | Model-specific |
### Gate
You must have a clear, measurable property that predicts VRAM usage. If the relationship between job properties and VRAM is unpredictable, this pattern won't work — use a different strategy (e.g., try GPU first, fall back on OOM).
---
## Step 2: Write the Probe Function
The probe function inspects the job input and returns the routing metric.
```bash
# Generic probe template
probe_workload() {
local INPUT="$1"
local METRIC=0
if [[ -n "$INPUT" && -f "$INPUT" ]]; then
# Example: audio/video duration via ffprobe
METRIC=$($PROBE_TOOL -v quiet -show_entries format=duration \
-of csv=p=0 "$INPUT" 2>/dev/null | cut -d. -f1)
METRIC=${METRIC:-0}
# Example: file size in bytes
# METRIC=$(stat -c%s "$INPUT" 2>/dev/null)
# Example: image resolution (width × height)
# METRIC=$($PROBE_TOOL -v quiet -show_entries stream=width,height \
# -of csv=p=0 "$INPUT" 2>/dev/null | awk -F, '{print $1*$2}')
fi
echo "$METRIC"
}
```
### Gate
Test the probe against known inputs:
```bash
# Small workload (should route to GPU)
probe_workload /path/to/small/input # Should be < THRESHOLD_VALUE
# Large workload (should route to CPU)
probe_workload /path/to/large/input # Should be >= THRESHOLD_VALUE
```
---
## Step 3: Implement the Router
The router uses the probe result to select GPU or CPU execution path.
```bash
#!/bin/bash
# GPU/CPU Fallback Router
# Routes jobs based on $THRESHOLD_UNIT inspection
THRESHOLD=$THRESHOLD_VALUE
LOGFILE="/tmp/workload-router.log"
GPU_LOCK="/tmp/gpu-workload.lock"
CPU_LOCK="/tmp/cpu-workload.lock"
# ──── Probe ────
INPUT="<extract from $@>"
METRIC=$(probe_workload "$INPUT")
# ──── Route ────
if (( METRIC < THRESHOLD )); then
MODE="GPU"
DEVICE_ARGS="$GPU_ARGS"
LOCK_FILE="$GPU_LOCK"
MAX_CONCURRENT=$MAX_GPU_JOBS
else
MODE="CPU"
DEVICE_ARGS="$CPU_ARGS"
LOCK_FILE="$CPU_LOCK"
MAX_CONCURRENT=$MAX_CPU_JOBS
fi
# ──── Concurrency control ────
if (( MAX_CONCURRENT == 1 )); then
# Single-job lock: flock with fail-fast
exec 9>"$LOCK_FILE"
if ! flock --nonblock 9; then
echo "[ROUTER] $(date) mode=${MODE}-BLOCKED metric=${METRIC} (slot full, exiting)" >> "$LOGFILE"
exit 1 # Caller should retry later
fi
fi
# For MAX_CONCURRENT > 1, use numbered lock files:
# for i in $(seq 0 $((MAX_CONCURRENT - 1))); do
# SLOT_LOCK="${LOCK_FILE}.${i}"
# exec 9>"$SLOT_LOCK"
# if flock --nonblock 9; then
# break # Got a slot
# fi
# if (( i == MAX_CONCURRENT - 1 )); then
# echo "[ROUTER] $(date) mode=${MODE}-BLOCKED metric=${METRIC} (all slots full)" >> "$LOGFILE"
# exit 1
# fi
# done
# ──── Log and execute ────
echo "[ROUTER] $(date) mode=$MODE metric=${METRIC} args: $@" >> "$LOGFILE"
exec $WORKLOAD_BINARY "$@" $DEVICE_ARGS
```
### Key design decisions
- **`flock --nonblock`**: Non-blocking lock attempt. If the slot is taken, exit immediately instead of waiting. This prevents queue starvation where all runner slots are blocked waiting for CPU jobs.
- **Exit code 1**: The caller (runner, scheduler) should interpret this as "retry later." Most job queues do this by default.
- **`exec`**: Replace the router process with the workload binary. Signals, exit codes, and resource limits pass through cleanly.
- **Lock files in `/tmp`**: Automatically cleaned on reboot. No stale locks after crashes.
---
## Step 4: Integrate with the Caller
Deploy the router using the binary wrapper interception pattern (see `binary-wrapper-interception.md`):
1. Rename the real binary: `mv $BINARY → ${BINARY}-real`
2. Write the router script
3. Symlink: `ln -sf /path/to/router $BINARY`
Or, if the caller supports configurable command paths, point it directly at the router.
---
## Step 5: Verify Both Paths
### GPU path
```bash
# Submit a small job
ssh $TARGET_HOST "$BINARY <small-input-args>"
# Verify GPU usage
ssh $TARGET_HOST "nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader"
# Check log
ssh $TARGET_HOST "tail -1 /tmp/workload-router.log"
# Should show: mode=GPU
```
### CPU path
```bash
# Submit a large job
ssh $TARGET_HOST "$BINARY <large-input-args>"
# Verify CPU usage (no GPU spike)
ssh $TARGET_HOST "nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader"
# GPU should be idle
# Check RAM
ssh $TARGET_HOST "free -h"
# Check log
ssh $TARGET_HOST "tail -1 /tmp/workload-router.log"
# Should show: mode=CPU
```
### Concurrency control
```bash
# Start a CPU job, then immediately try a second one
ssh $TARGET_HOST "$BINARY <large-input-1> &"
sleep 2
ssh $TARGET_HOST "$BINARY <large-input-2>"
# Second job should exit immediately with code 1
# Check log
ssh $TARGET_HOST "grep BLOCKED /tmp/workload-router.log"
# Should show: mode=CPU-BLOCKED
```
---
## Step 6: Tune and Monitor
After initial deployment, monitor for a day and adjust:
```bash
# Distribution of GPU vs CPU jobs
ssh $TARGET_HOST "grep -c 'mode=GPU' /tmp/workload-router.log"
ssh $TARGET_HOST "grep -c 'mode=CPU' /tmp/workload-router.log"
ssh $TARGET_HOST "grep -c 'BLOCKED' /tmp/workload-router.log"
```
If BLOCKED count is high relative to CPU count, the threshold may be too aggressive (routing too many jobs to CPU). Consider raising the threshold or increasing MAX_CPU_JOBS if RAM allows.
---
## Memory Budget Worksheet
```
GPU path:
VRAM per job: $WORKLOAD_VRAM_MB MB
Max GPU jobs: $MAX_GPU_JOBS
Total GPU VRAM: $GPU_VRAM_MB MB
Headroom: GPU_VRAM_MB - (WORKLOAD_VRAM_MB × MAX_GPU_JOBS) MB
→ Headroom must be positive
CPU path:
RAM per job: $WORKLOAD_RAM_MB MB
Max CPU jobs: $MAX_CPU_JOBS
System RAM: $(free -m | awk '/Mem:/{print $2}') MB
Other processes: ~2-4 GB (OS, services, buffers)
Headroom: SystemRAM - OtherProcs - (WORKLOAD_RAM_MB × MAX_CPU_JOBS) MB
→ Headroom must be positive
systemd MemoryMax: Should be set to MAX(GPU peak, CPU peak) + 20% buffer
```
---
## Troubleshooting
### GPU job OOM-kills despite being under threshold
The threshold is too high, or VRAM usage varies by input characteristics beyond what the probe measures. Lower the threshold or add a secondary probe (e.g., check resolution in addition to duration).
### CPU jobs pile up and exhaust RAM
`MAX_CPU_JOBS` is too high, or the `flock` mechanism isn't working. Check that lock files are being created in `/tmp/` and that the `exec 9>` file descriptor redirect is correct.
### All jobs route to CPU
The probe is returning 0 or failing silently. Test the probe manually:
```bash
$PROBE_TOOL -v quiet -show_entries format=duration -of csv=p=0 /path/to/input
```
If it returns empty, the input file may not be accessible to the probe tool (permissions, path issues).
### Blocked jobs never get retried
The caller doesn't retry on exit code 1. Check the caller's retry behavior. Some systems need specific exit codes (e.g., 75 for "temporary failure" in some mail systems). Adjust the exit code in the router to match what the caller expects.
### Lock files persist after crash
`/tmp` is cleaned on reboot, so stale locks self-heal. For immediate cleanup: `rm /tmp/cpu-workload.lock`. The next job will recreate it.
---
## Usage Examples
### Whisper auto-captioning on PeerTube runner (cortex)
```
WORKLOAD_BINARY=/usr/local/bin/whisper-ctranslate2-real
PROBE_TOOL=ffprobe
GPU_VRAM_MB=16384 # RTX A4000
WORKLOAD_VRAM_MB=3700 # Whisper medium on float16
WORKLOAD_RAM_MB=11000 # Whisper medium on CPU int8 (peak for 9.5hr video)
THRESHOLD_VALUE=3600 # 1 hour in seconds
THRESHOLD_UNIT=seconds
MAX_GPU_JOBS=2 # Runner concurrency=2, but both can be GPU
MAX_CPU_JOBS=1 # Only 1 CPU job at a time (11GB peak, 20G MemoryMax)
GPU_ARGS="--device cuda --compute_type float16"
CPU_ARGS="--device cpu --compute_type int8"
Result: 4100+ videos captioned. ~20 videos over 1 hour routed to CPU.
GPU jobs: ~3.7GB VRAM, 88-99% GPU utilization
CPU jobs: ~8-11GB RAM, serialized via flock
MemoryMax=20G on the runner service as safety net.
```
---
*Last updated: 2026-02-17*

406
runbooks/headscale-full-deployment.md
View file

@ -1,406 +0,0 @@
# Headscale Full Deployment Runbook
## Nodes + Headplane + Authentik OIDC
**Headscale location:** `/opt/headscale-vanilla`
**Container name:** `headscale-vanilla`
**Domain:** `vpn.echo6.co`
**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ`
---
## PHASE 1: REGISTER CONTABO (must be first)
```bash
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname contabo --force-reauth
```
Verify:
```bash
docker exec headscale-vanilla headscale nodes list
```
**STOP if contabo doesn't appear. Do not continue.**
---
## PHASE 2: REGISTER ALL LXC/CT NODES
SSH into each container. For each one:
```bash
# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"
# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh
```
Then register. **Do them in this exact order for sequential IPs:**
```bash
# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname utility --force-reauth
# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname data --force-reauth
# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname cloud --force-reauth
# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname media --force-reauth
# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname aida-nebra --force-reauth
```
After each, verify from Contabo:
```bash
docker exec headscale-vanilla headscale nodes list
```
---
## PHASE 3: REGISTER DESKTOP + PHONES
**Desktop (Windows — PowerShell as Admin):**
```powershell
tailscale up --login-server https://vpn.echo6.co `
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
--hostname desktop --force-reauth
```
**Phones:**
- Open Tailscale app → Settings → Account
- Log out if needed
- Use "Custom coordination server" or "Alternate server"
- Enter: `https://vpn.echo6.co`
- Should auto-register with the tailnet
If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.
---
## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY
```bash
docker exec headscale-vanilla headscale nodes list
```
Expected output: all nodes with sequential 100.64.0.x IPs.
Test from any node:
```bash
tailscale ping contabo
tailscale ping data
tailscale ping utility
```
Test magic DNS:
```bash
ping data.echo6.mesh
ping utility.echo6.mesh
```
---
## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)
```bash
mkdir -p /opt/headscale-vanilla/backups
# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"
# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
```
---
## PHASE 6: PERSISTENCE TEST
```bash
cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list
```
**Every node must survive. If any are missing, STOP and report.**
---
## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE
This lets Tailscale clients authenticate via Authentik instead of preauth keys.
1. Log into Authentik admin panel
2. Go to **Applications → Applications → Create with Provider**
3. Configure:
- **Application name:** Headscale
- **Slug:** `headscale` (remember this — it's part of the issuer URL)
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
- **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
- **Signing key:** Select any available key
- **Scopes:** Ensure these scope mappings are selected:
- `openid`
- `profile`
- `email`
- **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
4. Note the **Client ID** and **Client Secret**
5. Click Submit
---
## PHASE 8: CONFIGURE HEADSCALE OIDC
Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:
```yaml
oidc:
only_start_if_oidc_is_available: true
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
client_id: "<Client ID from Authentik>"
client_secret: "<Client Secret from Authentik>"
scope: ["openid", "profile", "email", "offline_access"]
pkce:
enabled: true
method: S256
strip_email_domain: true
```
Replace:
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
- `<Client ID from Authentik>` with the actual client ID
- `<Client Secret from Authentik>` with the actual client secret
Restart Headscale:
```bash
cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20
```
**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**
Test: From any node, run:
```bash
tailscale up --login-server https://vpn.echo6.co --force-reauth
```
It should open a browser → Authentik login → back to terminal, authenticated.
**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**
---
## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE
This is a SECOND application in Authentik for the web UI login.
1. Go to **Applications → Applications → Create with Provider**
2. Configure:
- **Application name:** Headplane
- **Slug:** `headplane`
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** Same as before
- **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
- **Signing key:** Same key
- **Scopes:** `openid`, `profile`, `email`
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
4. Click Submit
---
## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE
```bash
docker exec headscale-vanilla headscale apikeys create --expiration 999d
```
**Save this key — you need it for the Headplane config.**
---
## PHASE 11: CREATE HEADPLANE CONFIG
```bash
# Generate a cookie secret
openssl rand -hex 16
```
Write `/opt/headscale-vanilla/headplane-config.yaml`:
```yaml
server:
host: "0.0.0.0"
port: 3000
cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
cookie_secure: true
data_path: "/var/lib/headplane"
headscale:
url: "http://headscale-vanilla:8080"
config_path: "/etc/headscale/config.yaml"
config_strict: false
oidc:
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
client_id: "<Headplane Client ID from Authentik>"
client_secret: "<Headplane Client Secret from Authentik>"
token_endpoint_auth_method: "client_secret_post"
headscale_api_key: "<API_KEY_FROM_PHASE_10>"
redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
disable_api_key_login: false
integration:
docker:
enabled: true
container_name: "headscale-vanilla"
socket: "/var/run/docker.sock"
```
Replace all `<PLACEHOLDERS>` with actual values.
---
## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE
Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:
```yaml
services:
headscale:
# ... your existing headscale service, don't change it ...
headplane:
image: ghcr.io/tale/headplane:latest
container_name: headplane
restart: unless-stopped
depends_on:
- headscale
ports:
- "127.0.0.1:3000:3000"
volumes:
- ./headplane-config.yaml:/etc/headplane/config.yaml:ro
- ./headplane-data:/var/lib/headplane
- ./config.yaml:/etc/headscale/config.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
```
Start it:
```bash
cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20
```
Check for errors. Common issues:
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
- Can't connect to headscale → ensure `url` matches the container name and internal port
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock
---
## PHASE 13: UPDATE CADDY FOR HEADPLANE
Add the `/admin` route to your Caddy config for `vpn.echo6.co`:
```
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3000
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
```
Restart Caddy:
```bash
# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```
---
## PHASE 14: TEST HEADPLANE
1. Browse to `https://vpn.echo6.co/admin`
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. **The FIRST user to log in gets Owner permissions**
5. Verify you can see all your nodes in the UI
If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).
---
## PHASE 15: FINAL VERIFICATION
Run all of these from Contabo:
```bash
# All nodes present?
docker exec headscale-vanilla headscale nodes list
# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"
# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302
# Database backed up?
ls -la /opt/headscale-vanilla/backups/
# Cron running?
crontab -l | grep sqlite3
```
---
## REPORT TEMPLATE
After each phase, report:
```
Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):
```
**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**
---
## KNOWN GOTCHAS
1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs

300
runbooks/ia-cli-reference.md Normal file
View file

@ -0,0 +1,300 @@
# Internet Archive CLI Reference
Quick reference for the `ia` command-line tool on pi-nas.
---
## Location & Setup
| Detail | Value |
|--------|-------|
| Host | pi-nas (192.168.1.245 / 100.64.0.21) |
| Binary | `ia` (v5.7.2, pip-installed) |
| Config | `~/.config/internetarchive/ia.ini` |
---
## 1. Configure / Authenticate
Required for uploads, metadata edits, and accessing restricted items. Not required for public downloads or searches.
```bash
ia configure
# Prompts for archive.org email + password
# Stores credentials in ~/.config/internetarchive/ia.ini
```
Verify:
```bash
ia configure --help # Should show options without errors
```
---
## 2. Search
Search the archive.org catalog. Returns JSON by default.
### Basic syntax
```bash
ia search '<query>'
```
### Query syntax
Queries use Lucene syntax. Combine fields with AND/OR, quote phrases.
| Field | Example | Notes |
|-------|---------|-------|
| `collection` | `collection:prelinger` | Items in a specific collection |
| `subject` | `subject:"ham radio"` | Subject/tag match |
| `mediatype` | `mediatype:texts` | texts, movies, audio, software, image, data, web, collection |
| `creator` | `creator:"ARRL"` | Author/creator |
| `title` | `title:"emergency"` | Item title |
| `date` | `date:[2020-01-01 TO 2024-12-31]` | Date range (YYYY-MM-DD) |
| `year` | `year:2023` | Shorthand for year |
| `language` | `language:eng` | ISO language code |
| `licenseurl` | `licenseurl:*creativecommons*` | License filter |
### Combined queries
```bash
# PDFs about ham radio published after 2020
ia search 'subject:"ham radio" mediatype:texts date:[2020-01-01 TO 2099-12-31]'
# All items in a specific collection
ia search 'collection:prelinger'
# Creator + mediatype
ia search 'creator:"ARRL" AND mediatype:texts'
```
### Output options
```bash
# Default: JSON objects, one per line
ia search 'collection:prelinger'
# Itemlist mode — outputs only identifiers, one per line
# Pipe this to ia download --itemlist
ia search 'collection:prelinger' --itemlist
# Save itemlist to file
ia search 'collection:prelinger' --itemlist > prelinger-items.txt
# Limit results with parameters
ia search 'subject:radio' --parameters='rows=50'
# Count results without downloading them all
ia search 'collection:prelinger' --num-found
```
### Practical examples
```bash
# Find all items in a collection and count them
ia search 'collection:arrl_qst' --num-found
# Get identifiers for bulk download
ia search 'collection:arrl_qst' --itemlist > arrl-items.txt
# Search within a collection for specific subjects
ia search 'collection:prelinger subject:"san francisco"' --itemlist
# Find audio recordings by a specific creator
ia search 'creator:"Grateful Dead" mediatype:audio' --itemlist
# Search for items with specific file formats available
ia search 'collection:librivoxaudio format:"64Kbps MP3"' --itemlist
```
---
## 3. List Item Contents
View files within an item without downloading.
```bash
# List all files in an item
ia list <identifier>
# Example
ia list prelinger_films
```
Output shows filenames, sizes, and formats.
---
## 4. Metadata
View and modify item metadata.
### Read metadata
```bash
# Full metadata as JSON
ia metadata <identifier>
# Pretty-print with jq
ia metadata <identifier> | jq .
# Get specific fields
ia metadata <identifier> | jq '.metadata.title'
ia metadata <identifier> | jq '.metadata.subject'
ia metadata <identifier> | jq '.metadata.collection'
# List available formats for an item
ia metadata <identifier> --formats
```
### Modify metadata (requires authentication)
```bash
# Set a field
ia metadata <identifier> --modify="description:Updated description"
# Remove a field
ia metadata <identifier> --modify="subject:REMOVE_TAG"
# Append to existing value
ia metadata <identifier> --append="subject:new-tag"
# Add to array field
ia metadata <identifier> --append-list="collection:another-collection"
# Bulk modify from CSV (must have 'identifier' column)
ia metadata --spreadsheet=metadata.csv
```
---
## 5. Upload (requires authentication)
```bash
# Upload files to a new or existing item
ia upload <identifier> file1.pdf file2.pdf \
--metadata="mediatype:texts" \
--metadata="title:My Upload" \
--metadata="subject:test"
# Upload from stdin
curl -sL https://example.com/file.pdf | \
ia upload <identifier> - --remote-name=file.pdf
# Retry on failure
ia upload <identifier> largefile.zip --retries 10
# Bulk upload from CSV (requires 'identifier' and 'file' columns)
ia upload --spreadsheet=uploads.csv
```
**Important:** `mediatype` cannot be changed after initial upload.
---
## 6. Delete (requires authentication)
```bash
# Delete a specific file
ia delete <identifier> filename.pdf
# Delete file and all its derivatives
ia delete <identifier> filename.pdf --cascade
# Delete all files in an item
ia delete <identifier> --all
```
Deleted files are backed up to `history/files/` automatically.
---
## 7. Copy / Move
```bash
# Copy a file between items
ia copy source-item/file.pdf dest-item/file.pdf
# Copy with metadata for new items
ia copy source/file.pdf new-item/file.pdf --metadata="title:Copied Item"
# Move (copy + delete source)
ia move source-item/file.pdf dest-item/file.pdf
```
---
## 8. Tasks
View catalog processing tasks (derive jobs, uploads in progress, etc.).
```bash
# Tasks for a specific item
ia tasks <identifier>
# All your queued/running tasks
ia tasks
```
---
## Command Quick Reference
| Command | Alias | Purpose |
|---------|-------|---------|
| `ia configure` | `ia co` | Set up credentials |
| `ia search` | `ia se` | Search catalog |
| `ia download` | `ia do` | Download files |
| `ia list` | `ia ls` | List item files |
| `ia metadata` | `ia md` | View/edit metadata |
| `ia upload` | `ia up` | Upload files |
| `ia delete` | `ia rm` | Delete files |
| `ia copy` | `ia cp` | Copy between items |
| `ia move` | `ia mv` | Move between items |
| `ia tasks` | `ia ta` | View task queue |
---
## Global Flags
| Flag | Short | Purpose |
|------|-------|---------|
| `--help` | `-h` | Show help |
| `--version` | `-v` | Show version |
| `--config-file FILE` | `-c` | Use alternate config |
| `--log` | `-l` | Enable logging |
| `--debug` | `-d` | Verbose debug output |
| `--insecure` | `-i` | Use HTTP instead of HTTPS |
---
## Troubleshooting
### "You need to be logged in"
Run `ia configure` and enter your archive.org credentials. Verify with:
```bash
cat ~/.config/internetarchive/ia.ini
```
### Search returns no results
- Check query syntax — field names are case-sensitive
- Use quotes around multi-word values: `subject:"ham radio"` not `subject:ham radio`
- Verify the collection/identifier exists: `ia metadata <identifier>`
### Slow searches
Large collections can take minutes to enumerate. Use `--parameters='rows=100'` to limit during testing, or `--num-found` to just get the count first.
### Rate limiting
Archive.org may throttle aggressive requests. Space out bulk operations and use `--retries` on downloads.
---
*Last updated: 2026-02-14*

429
runbooks/ia-download-mirror.md Normal file
View file

@ -0,0 +1,429 @@
# Download & Mirror from Internet Archive
Procedures for downloading items, filtering by format/pattern, bulk downloading from collections, and mirroring entire collections via the `ia` CLI on pi-nas.
---
## Prerequisites
- `ia` CLI installed on pi-nas (192.168.1.245) — v5.7.2
- Authenticated if downloading restricted items: `ia configure`
- Sufficient storage on pi-nas (check with `df -h`)
- Reference: `ia-cli-reference.md` for search/query syntax
---
## 1. Download a Single Item
An "item" is a logical unit on archive.org identified by its identifier (visible in the URL: `archive.org/details/<identifier>`).
```bash
# Download all files in an item to ./<identifier>/
ia download <identifier>
# Example
ia download prelinger_films
```
The default creates a directory named after the identifier containing all files (originals + derivatives).
### Gate
Verify the download directory exists and has files:
```bash
ls -la <identifier>/
```
---
## 2. Filtered Downloads
### By glob pattern
Download only files matching a shell glob pattern.
```bash
# Only PDFs
ia download <identifier> --glob="*.pdf"
# Only MP4 video files
ia download <identifier> --glob="*.mp4"
# Multiple patterns (pipe-separated)
ia download <identifier> --glob="*.pdf|*.epub"
```
### With exclusions
Exclude patterns require `--glob` to also be set.
```bash
# All MP4s except low-quality variants
ia download <identifier> --glob="*.mp4" --exclude="*512kb*"
# All files except metadata/review XMLs
ia download <identifier> --glob="*" --exclude="*_meta.xml|*_reviews.xml|*_files.xml"
# Multiple exclusions
ia download <identifier> --glob="*.mp4" --exclude="*512kb*|*_thumb*"
```
### By format name
Download files of a specific archive.org format (as shown by `ia metadata --formats`).
```bash
# Check available formats first
ia metadata <identifier> --formats
# Download only a specific format
ia download <identifier> --format="512Kb MPEG4"
ia download <identifier> --format="PDF"
ia download <identifier> --format="EPUB"
```
**Note:** `--format` is incompatible with `--glob` and `--exclude`. Use one approach or the other.
### On-the-fly formats
Some formats (EPUB, MOBI, DAISY, MARCXML) are generated on demand.
```bash
ia download <identifier> --on-the-fly --format="EPUB"
```
---
## 3. Download Options
### Control output location
```bash
# Download to a specific directory
ia download <identifier> --destdir=/mnt/archive/downloads/
# Flatten directory structure (no subdirectory per item)
ia download <identifier> --no-directories
```
### Resume interrupted downloads
```bash
# Resume — skips files that already exist and match checksum
ia download <identifier> --checksum
# Checksum mode compares MD5 hashes — safe to re-run
```
### Preserve timestamps
```bash
# Keep original timestamps from archive.org
ia download <identifier> --no-change-timestamp
```
### Dry run
```bash
# See what would be downloaded without actually downloading
ia download <identifier> --dry-run
```
---
## 4. Bulk Download from Search Results
Pipe search results directly into download. This is the primary method for downloading multiple items.
### Basic pattern
```bash
# Search → itemlist → download
ia search 'collection:prelinger mediatype:movies' --itemlist | \
ia download --itemlist -
# The - tells ia download to read identifiers from stdin
```
### With filters
```bash
# Download only PDFs from all items in a collection
ia search 'collection:arrl_qst' --itemlist | \
ia download --itemlist - --glob="*.pdf"
# Download only MP3s from an audio collection
ia search 'collection:librivoxaudio' --itemlist | \
ia download --itemlist - --glob="*.mp3"
```
### With destination directory
```bash
# Download to a specific location
ia search 'collection:prelinger' --itemlist | \
ia download --itemlist - --destdir=/mnt/archive/prelinger/
```
### Save itemlist for reuse
When a search is large, save the itemlist first so you can resume without re-searching.
```bash
# Step 1: Save itemlist
ia search 'collection:prelinger mediatype:movies' --itemlist > prelinger-items.txt
# Step 2: Check count
wc -l prelinger-items.txt
# Step 3: Download from file
ia download --itemlist prelinger-items.txt --glob="*.mp4"
# Step 4: Resume if interrupted (just re-run with --checksum)
ia download --itemlist prelinger-items.txt --glob="*.mp4" --checksum
```
---
## 5. Bulk Download with GNU Parallel
For faster bulk downloads, use GNU Parallel for concurrent item downloads.
```bash
# Install parallel if not present
sudo apt install -y parallel
# Download 5 items concurrently
ia search 'collection:prelinger' --itemlist | \
parallel -j5 'ia download {} --glob="*.mp4"'
# With destination directory
ia search 'collection:prelinger' --itemlist | \
parallel -j5 'ia download {} --glob="*.mp4" --destdir=/mnt/archive/prelinger/'
# From saved itemlist
parallel -j5 'ia download {} --glob="*.pdf"' < items.txt
```
**Caution:** Be respectful of archive.org bandwidth. 3-5 concurrent downloads is reasonable. Higher parallelism may trigger rate limiting.
---
## 6. Mirror an Entire Collection
Mirroring means downloading everything and being able to re-run to pick up new additions.
### Initial mirror
```bash
# Step 1: Create working directory
mkdir -p /mnt/archive/<collection-name>
cd /mnt/archive/<collection-name>
# Step 2: Generate itemlist
ia search 'collection:<collection-name>' --itemlist > itemlist.txt
echo "Found $(wc -l < itemlist.txt) items"
# Step 3: Download all items (adjust --glob as needed)
ia download --itemlist itemlist.txt --destdir=/mnt/archive/<collection-name>/
# Or with format filter
ia download --itemlist itemlist.txt --glob="*.pdf" --destdir=/mnt/archive/<collection-name>/
```
### Update an existing mirror
Re-run the same commands. Use `--checksum` to skip already-downloaded files.
```bash
cd /mnt/archive/<collection-name>
# Refresh itemlist (new items since last run)
ia search 'collection:<collection-name>' --itemlist > itemlist-new.txt
# Download only new/changed files
ia download --itemlist itemlist-new.txt --checksum --destdir=/mnt/archive/<collection-name>/
```
### Mirror with a script
For recurring mirrors, create a simple script:
```bash
#!/bin/bash
# mirror-collection.sh <collection-name> [glob-pattern]
COLLECTION="$1"
GLOB="${2:-*}"
DEST="/mnt/archive/$COLLECTION"
mkdir -p "$DEST"
echo "Refreshing itemlist for $COLLECTION..."
ia search "collection:$COLLECTION" --itemlist > "$DEST/itemlist.txt"
COUNT=$(wc -l < "$DEST/itemlist.txt")
echo "Found $COUNT items"
echo "Downloading (glob: $GLOB)..."
ia download --itemlist "$DEST/itemlist.txt" --glob="$GLOB" --checksum --destdir="$DEST/"
echo "Mirror complete: $DEST"
```
Usage:
```bash
chmod +x mirror-collection.sh
./mirror-collection.sh arrl_qst "*.pdf"
./mirror-collection.sh prelinger "*.mp4"
```
---
## 7. Practical Patterns
### Download all PDFs from a collection
```bash
ia search 'collection:arrl_qst' --itemlist | \
ia download --itemlist - --glob="*.pdf" --destdir=/mnt/archive/arrl-qst/
```
### Download specific media types from a collection
```bash
# High-quality video only
ia search 'collection:prelinger' --itemlist | \
ia download --itemlist - --format="MPEG4" --destdir=/mnt/archive/prelinger-video/
# Audio in MP3 format
ia search 'collection:librivoxaudio creator:"Mark Twain"' --itemlist | \
ia download --itemlist - --glob="*64kb*.mp3" --destdir=/mnt/archive/twain-audio/
```
### Download items matching a date range
```bash
ia search 'collection:arrl_qst date:[1950-01-01 TO 1959-12-31]' --itemlist | \
ia download --itemlist - --glob="*.pdf" --destdir=/mnt/archive/arrl-1950s/
```
### Download a single specific file from an item
```bash
# List files first
ia list <identifier>
# Download just one file
ia download <identifier> specific-file.pdf
```
### Download and preserve directory structure
```bash
# Default behavior — each item gets its own subdirectory
ia download --itemlist items.txt --destdir=/mnt/archive/output/
# Result: /mnt/archive/output/<identifier1>/files...
# /mnt/archive/output/<identifier2>/files...
```
### Pipe a single file to stdout
```bash
# Stream a file without saving to disk
ia download <identifier> specific-file.pdf --stdout | less
ia download <identifier> data.json --stdout | jq .
```
---
## 8. Storage Planning
Before large downloads, estimate storage requirements.
```bash
# Count items in collection
ia search 'collection:<name>' --num-found
# Check a sample item's size
ia metadata <sample-identifier> | jq '[.files[].size | tonumber] | add / 1048576 | floor'
# Output in MB
# Check available storage on pi-nas
df -h /mnt/
```
### Rule of thumb
- Text collections (PDFs, EPUBs): ~10-100 MB per item
- Audio collections: ~100 MB - 1 GB per item
- Video collections: ~1-10 GB per item
- Software archives: highly variable
---
## Troubleshooting
### Download hangs or stalls
```bash
# Kill and resume with checksum verification
# Ctrl+C to stop, then re-run with --checksum
ia download --itemlist items.txt --glob="*.pdf" --checksum
```
### "Item not found" errors in bulk download
Some items in a collection may be restricted or taken down. These will fail individually but the batch continues. Check errors in output.
### Disk full during bulk download
```bash
# Check what's using space
du -sh /mnt/archive/*/ | sort -rh | head -20
# Resume after freeing space — checksum mode skips completed files
ia download --itemlist items.txt --checksum
```
### Rate limiting / 429 errors
Archive.org may throttle aggressive downloads.
- Reduce parallel jobs (if using GNU Parallel)
- Add delays between items: `parallel -j2 --delay 5 'ia download {}' < items.txt`
- Wait and retry later
### Corrupt downloads
```bash
# Re-download with checksum verification — replaces corrupt files
ia download <identifier> --checksum
```
### Permission denied on destination
```bash
# Ensure the download user owns the target directory
sudo chown -R $(whoami):$(whoami) /mnt/archive/
```
---
## Checklist: Collection Mirror
```
[ ] Identify collection identifier on archive.org
[ ] Check available storage (df -h)
[ ] Estimate collection size (--num-found + sample item size)
[ ] Generate itemlist (ia search --itemlist > itemlist.txt)
[ ] Review itemlist count (wc -l itemlist.txt)
[ ] Start download with appropriate filters (--glob, --format)
[ ] Verify downloaded files exist and are non-zero
[ ] If interrupted, resume with --checksum
[ ] Record collection details in project notes
```
---
*Last updated: 2026-02-14*

337
runbooks/idahomesh-bridge-setup.md Normal file
View file

@ -0,0 +1,337 @@
# IdahoMesh Bridge Setup
Build a one-way bridge between your tailnet and the IdahoMesh Meshtastic network. This lets your devices reach Nebra gateways through IdahoMesh, while preventing IdahoMesh from reaching back into your network.
---
## Architecture
```
Your Tailnet (your Headscale/Tailscale)
↓ (one-way only)
[Bridge Machine] ← dual tailscaled, NAT + firewall
↓
IdahoMesh Tailnet (100.100.0.0/16)
↕
Nebra CM3 Gateways (Meshtastic nodes)
```
**Security model:** Your devices can reach Nebras. Nothing on IdahoMesh can initiate connections back into your network. NAT masquerades your source IPs so IdahoMesh only sees the bridge's IP.
---
## Prerequisites
| Item | Value |
|------|-------|
| IdahoMesh Headscale URL | `https://vpn.idahomesh.com` |
| IdahoMesh prefix | `100.100.0.0/16` |
| Your preauthkey | Provided by IdahoMesh admin |
You also need:
1. A Linux machine on your network (VM, Pi, bare metal — anything running systemd)
2. Root access on that machine
3. Internet access (to reach vpn.idahomesh.com)
4. Your own tailnet's connection details (Headscale URL, or stock Tailscale if using tailscale.com)
---
## Step 1: Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
```
This installs both `tailscale` and `tailscaled`. The default service (`tailscaled.service`) will handle your primary tailnet.
---
## Step 2: Set Up Dual tailscaled
You need two Tailscale daemon instances — one for your tailnet, one for IdahoMesh. The default `tailscaled` service handles your tailnet. Create a second service for IdahoMesh.
### Create directories
```bash
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
```
### Create the second tailscaled service
Write `/etc/systemd/system/tailscaled-meshtastic.service`:
```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/sbin/tailscaled \
--state=/var/lib/tailscale-meshtastic/tailscaled.state \
--socket=/var/run/tailscale-meshtastic/tailscaled.sock \
--port=41642 \
--tun=tailscale1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
> **Critical:** The `--tun=tailscale1` flag is required. Both instances cannot use the default `tailscale0` TUN device — the second one will fail with "TUN device tailscale0 is busy" if you omit this.
Enable and start it:
```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```
---
## Step 3: Enable IP Forwarding
```bash
cat > /etc/sysctl.d/99-bridge.conf << 'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```
---
## Step 4: Join Both Tailnets
### Join your tailnet (default tailscaled)
Advertise the IdahoMesh range so your other devices can route to Meshtastic nodes through this bridge:
```bash
# If you use your own Headscale:
tailscale up \
--login-server=https://YOUR_HEADSCALE_URL \
--advertise-routes=100.100.0.0/16 \
--accept-routes
# If you use stock Tailscale (tailscale.com):
tailscale up \
--advertise-routes=100.100.0.0/16 \
--accept-routes
```
After joining, you need to **approve the advertised route** on your tailnet's admin:
- **Headscale:** `headscale routes list` then `headscale routes enable -r <route-id>`
- **Stock Tailscale:** Go to admin console → Machines → your bridge → approve the `100.100.0.0/16` subnet route
### Join IdahoMesh (second tailscaled)
```bash
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
--login-server=https://vpn.idahomesh.com \
--authkey=YOUR_IDAHOMESH_PREAUTHKEY \
--accept-routes
```
> **Important:** Do NOT advertise your tailnet's routes on IdahoMesh. The bridge is one-way — IdahoMesh should have no route back into your network.
---
## Step 5: Configure One-Way Firewall and NAT
This is the critical security step. Your tailnet can reach IdahoMesh, but nothing on IdahoMesh can reach back into your network.
Replace `YOUR_TAILNET_PREFIX` below with your tailnet's IP range. Common values:
| Tailnet type | Prefix |
|-------------|--------|
| Stock Tailscale | `100.64.0.0/10` |
| Custom Headscale | Check your `config.yaml` → `prefixes.v4` |
### Apply iptables rules
```bash
# NAT: Masquerade your source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not your real tailnet IPs
iptables -t nat -A POSTROUTING -s YOUR_TAILNET_PREFIX -d 100.100.0.0/16 -j MASQUERADE
# Allow your tailnet → IdahoMesh (outbound)
iptables -A FORWARD -s YOUR_TAILNET_PREFIX -d 100.100.0.0/16 -j ACCEPT
# Allow established/related return traffic only (responses to connections you initiated)
iptables -A FORWARD -s 100.100.0.0/16 -d YOUR_TAILNET_PREFIX -m state --state ESTABLISHED,RELATED -j ACCEPT
# DROP all new connections from IdahoMesh → your tailnet
iptables -A FORWARD -s 100.100.0.0/16 -d YOUR_TAILNET_PREFIX -j DROP
```
### Persist rules across reboots
Do **not** use `iptables-persistent` — it hangs on install even with noninteractive mode. Use manual persistence instead:
```bash
# Save current rules
mkdir -p /etc/iptables
iptables-save > /etc/iptables/rules.v4
```
Create `/etc/systemd/system/iptables-restore.service`:
```ini
[Unit]
Description=Restore iptables rules
Before=network-pre.target
Wants=network-pre.target
[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables/rules.v4
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
```
Enable it:
```bash
systemctl daemon-reload
systemctl enable iptables-restore
```
### Verify rules
```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```
You should see:
- MASQUERADE rule on POSTROUTING
- ACCEPT for your prefix → 100.100.0.0/16
- ACCEPT ESTABLISHED,RELATED for 100.100.0.0/16 → your prefix
- DROP for 100.100.0.0/16 → your prefix
---
## Step 6: Enable Routes on Your Other Devices
Any device on your tailnet that wants to reach IdahoMesh Nebras through the bridge needs to accept subnet routes:
```bash
# On each device that needs access
tailscale set --accept-routes
```
Without this, traffic to `100.100.0.x` goes to the default gateway instead of through the Tailscale tunnel.
---
## Step 7: Verify
### Check both tailscaled instances
```bash
# Your tailnet
tailscale status
# IdahoMesh
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```
Both should show "online" with peers listed.
### Ping a Nebra gateway
```bash
# From the bridge itself, via IdahoMesh socket
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte
```
### Ping from another device on your tailnet
```bash
# From any device on your tailnet (with --accept-routes enabled)
# Use the Nebra's IdahoMesh IP
ping 100.100.0.3 # Burley Butte
```
### Verify isolation
From an IdahoMesh device or ask the admin to test — pinging your tailnet IPs from IdahoMesh should time out / be unreachable.
---
## Troubleshooting
### Second tailscaled won't start
- Check `journalctl -u tailscaled-meshtastic -f`
- Most common: forgot `--tun=tailscale1` — both instances fighting over `tailscale0`
- Verify the port isn't in conflict: default uses 41641, second uses 41642
### Can't reach Nebras from other devices on your tailnet
- Verify the bridge advertises `100.100.0.0/16`: `tailscale status` should show it as a subnet router
- Approve the route on your tailnet admin (Headscale or Tailscale admin console)
- Enable `--accept-routes` on the client device trying to reach Nebras
### IdahoMesh preauthkey expired
- Contact the IdahoMesh admin for a new key
- Re-join: `tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up --login-server=https://vpn.idahomesh.com --authkey=NEW_KEY --accept-routes`
### After reboot, only one tailscaled reconnects
- Check both services: `systemctl status tailscaled` and `systemctl status tailscaled-meshtastic`
- Verify iptables rules survived: `iptables -L FORWARD -v -n`
- If the second instance lost state, re-join IdahoMesh with a new preauthkey
### Default tailscaled pointed at wrong server after force-reauth
- If you run `tailscale up --force-reauth` without specifying `--login-server`, it may reconnect to the wrong Headscale
- Always specify `--login-server` explicitly when re-authing:
- Default instance: `tailscale up --login-server=https://YOUR_HEADSCALE_URL --force-reauth`
- IdahoMesh instance: `tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up --login-server=https://vpn.idahomesh.com --force-reauth`
---
## Quick Reference
| Item | Value |
|------|-------|
| IdahoMesh URL | `https://vpn.idahomesh.com` |
| IdahoMesh prefix | `100.100.0.0/16` |
| IdahoMesh socket | `/var/run/tailscale-meshtastic/tailscaled.sock` |
| IdahoMesh TUN device | `tailscale1` |
| IdahoMesh port | `41642` |
| Default Tailscale socket | `/var/run/tailscale/tailscaled.sock` |
| Default TUN device | `tailscale0` |
| Default port | `41641` |
### Useful commands
```bash
# IdahoMesh status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
# IdahoMesh ping
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping <hostname>
# Check firewall rules
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
# Restart services
systemctl restart tailscaled # Your tailnet
systemctl restart tailscaled-meshtastic # IdahoMesh
```
---
*Last updated: 2026-02-11*

299
runbooks/idahomesh-vpn-device-setup.md Normal file
View file

@ -0,0 +1,299 @@
# IdahoMesh VPN — Device Setup
Join a device to the IdahoMesh tailnet (Meshtastic mesh network VPN).
---
## Overview
| Item | Value |
|------|-------|
| Headscale URL | `https://vpn.idahomesh.com` |
| Tailnet prefix | 100.100.0.0/16 |
| MagicDNS domain | mesh.local |
| Supported platforms | Linux (x86/ARM), Windows, macOS, Android, iOS |
### Users
| User | Purpose | Key type |
|------|---------|----------|
| malice | Echo6 infrastructure (bridge LXC) | Short-lived, single-use |
| sidpatchy | Sidpatchy's devices | Short-lived, single-use |
| nebra | Nebra CM3 gateways (field devices) | Long-lived, reusable |
---
## Prerequisites
Before starting, you need:
1. **A preauthkey** — generated by the IdahoMesh Headscale admin (see [Generating Keys](#generating-preauthkeys) below)
2. **Internet access** on the device (to reach vpn.idahomesh.com)
3. **Root/admin access** on the device
---
## Linux (Debian/Ubuntu/Raspberry Pi OS)
### Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
```
### Join the IdahoMesh tailnet
```bash
tailscale up \
--login-server=https://vpn.idahomesh.com \
--authkey=<YOUR_PREAUTHKEY> \
--hostname=<DEVICE_NAME>
```
Replace:
- `<YOUR_PREAUTHKEY>` — the key provided to you
- `<DEVICE_NAME>` — short, lowercase hostname (e.g., `burley-butte`, `sid-laptop`)
### Verify
```bash
tailscale status
tailscale ip -4
ping -c 3 100.100.0.1 # Ping another node on the tailnet
```
You should see a `100.100.x.x` IP assigned.
---
## Nebra CM3 Gateway (ARM/Raspberry Pi)
Same as Linux above. The install script auto-detects ARM.
```bash
# Install
curl -fsSL https://tailscale.com/install.sh | sh
# Join — use the reusable nebra preauthkey
tailscale up \
--login-server=https://vpn.idahomesh.com \
--authkey=<NEBRA_PREAUTHKEY> \
--hostname=<GATEWAY_NAME>
```
**Naming convention for Nebra gateways:**
| Gateway | Hostname |
|---------|----------|
| Burley Butte | `burley-butte` |
| Picabo | `picabo` |
| AIDA-NEBRA | `aida-nebra` |
### Enable Tailscale on boot
Tailscale installs as a systemd service and starts on boot automatically. Confirm:
```bash
systemctl is-enabled tailscaled
# Should output: enabled
```
---
## Windows
### Install
Download and install Tailscale from: https://tailscale.com/download/windows
### Join
Open **PowerShell as Administrator:**
```powershell
tailscale up --login-server=https://vpn.idahomesh.com `
--authkey=<YOUR_PREAUTHKEY> `
--hostname=<DEVICE_NAME>
```
### Verify
```powershell
tailscale status
tailscale ip -4
```
---
## macOS
### Install
Download and install Tailscale from: https://tailscale.com/download/mac
Or via Homebrew:
```bash
brew install tailscale
```
### Join
```bash
tailscale up \
--login-server=https://vpn.idahomesh.com \
--authkey=<YOUR_PREAUTHKEY> \
--hostname=<DEVICE_NAME>
```
### Verify
```bash
tailscale status
tailscale ip -4
```
---
## Android
1. Install Tailscale from **F-Droid** (recommended — supports custom servers) or Google Play
2. Open Tailscale → tap the three-dot menu → **Use custom coordination server**
3. Enter: `https://vpn.idahomesh.com`
4. Authenticate (it will open a browser if no authkey is used)
> **Note:** The Google Play version may not support custom coordination servers. Use the F-Droid build if the option is missing.
---
## iOS
1. Install Tailscale from the App Store
2. Open Tailscale → Settings → **Use Alternate Server**
3. Enter: `https://vpn.idahomesh.com`
4. Authenticate via browser
> **Note:** iOS support for custom Headscale servers can be limited. If the option is unavailable, use the CLI on another device and share connectivity via subnet routing.
---
## Verification Checklist
Run from the newly joined device:
```bash
echo "=== IdahoMesh VPN Check ==="
echo "Hostname: $(hostname)"
echo "Tailscale IP: $(tailscale ip -4 2>/dev/null || echo 'N/A')"
echo "Status: $(tailscale status --self 2>/dev/null | head -1 || echo 'NOT CONNECTED')"
echo "Login server: $(tailscale debug prefs 2>/dev/null | grep -o 'ControlURL:[^ ]*' || echo 'unknown')"
```
Expected: a `100.100.x.x` IP and connected status.
### Test connectivity to other nodes
```bash
# List all visible peers
tailscale status
# Ping another node by MagicDNS name
tailscale ping <OTHER_HOSTNAME>
# Or by IP
ping 100.100.0.x
```
---
## Generating Preauthkeys
**Admin only** — run on the IdahoMesh Headscale server (CT 106, 192.168.1.106):
```bash
# For a human user (single-use, expires in 24h)
headscale preauthkeys create --user <USERNAME> --expiration 24h
# For a human user (single-use, longer window)
headscale preauthkeys create --user <USERNAME> --expiration 72h
# For Nebra gateways (reusable, long-lived)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# List existing keys
headscale preauthkeys list --user <USERNAME>
```
Users: `malice` (ID 4), `sidpatchy` (ID 2), `nebra` (ID 3)
> **Note:** Headscale v0.28.0 `--user` flag requires user IDs (integers), not names.
---
## Removing a Device
**Admin only:**
```bash
# List all nodes
headscale nodes list
# Delete a node by ID
headscale nodes delete -i <NODE_ID>
```
On the device itself:
```bash
tailscale logout
```
---
## Troubleshooting
### "connection refused" or timeout on join
- Confirm the device has internet access: `curl -I https://vpn.idahomesh.com`
- Check DNS resolution: `dig vpn.idahomesh.com`
- Verify the preauthkey hasn't expired
### "key expired" or "invalid key"
- Preauthkeys are time-limited. Ask the admin for a new one
- Nebra keys are reusable but still expire — check with `headscale preauthkeys list --user nebra`
### Device shows "offline" in node list
- Check if tailscaled is running: `systemctl status tailscaled`
- Restart: `systemctl restart tailscaled`
- Force re-auth: `tailscale up --login-server=https://vpn.idahomesh.com --force-reauth`
### Can't reach other nodes
- Confirm both devices show as "online" in `tailscale status`
- Check ACL policy — the admin may need to add rules for your user group
- Try direct IP ping before MagicDNS names
### MagicDNS not resolving
- Confirm `--accept-dns=true` (this is the default)
- Check: `tailscale debug prefs | grep CorpDNS`
- Restart tailscaled
---
## Quick Reference
| Item | Value |
|------|-------|
| Headscale URL | `https://vpn.idahomesh.com` |
| Tailnet prefix | 100.100.0.0/16 |
| MagicDNS domain | mesh.local |
| Headscale server | CT 106 on utility (192.168.1.106) |
| Admin access | SSH to 192.168.1.106 or via utility Proxmox |
| ACL policy | `/etc/headscale/acl.json` on CT 106 |
---
*Last updated: 2026-02-11*

259
runbooks/mailcow-create-mailbox.md Normal file
View file

@ -0,0 +1,259 @@
# Mailcow: Create Mailbox
Create a new mailbox in Mailcow on the Contabo VPS. Covers both interactive (UI) and API-driven creation, with the critical authsource fix for service accounts.
---
## When to Use This
Any time a new mailbox is created in Mailcow, but **especially** for service/system accounts that authenticate via SMTP to send mail programmatically (e.g., `no-reply@echo6.co` used by Authentik, `recon@echo6.co` used by the RECON pipeline). These accounts don't log in through the Mailcow web UI or SSO — they pass credentials directly to Postfix over SMTP, so they **must** use local password authentication.
---
## Prerequisites
- SSH access to Contabo (`ssh root@100.64.0.1`)
- Mailcow API key (stored in Mailcow admin UI under System → Configuration → API)
- Mailcow DB password: source from `/opt/mailcow-dockerized/.env` (`DBPASS`)
---
## Inputs
```
LOCAL_PART=no-reply # Left side of the @ sign
DOMAIN=echo6.co # Must already exist in Mailcow
DISPLAY_NAME="Echo6 No Reply" # Friendly name
PASSWORD=<strong-password> # Generate with: openssl rand -base64 24 | tr -d '/+='
QUOTA_MB=256 # Mailbox quota in MB
IS_SERVICE_ACCOUNT=true # true = SMTP sender, false = regular user
MAILCOW_API_KEY=<api-key> # From Mailcow admin UI
```
---
## Step 1: Create the Mailbox
### Option A: Via Mailcow API
```bash
ssh root@100.64.0.1
curl -sk -X POST "https://127.0.0.1:8443/api/v1/add/mailbox" \
-H "X-API-Key: ${MAILCOW_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"local_part": "'${LOCAL_PART}'",
"domain": "'${DOMAIN}'",
"name": "'${DISPLAY_NAME}'",
"password": "'${PASSWORD}'",
"password2": "'${PASSWORD}'",
"quota": '${QUOTA_MB}',
"active": 1,
"force_pw_update": 0,
"tls_enforce_in": 1,
"tls_enforce_out": 1
}'
```
Expected response:
```json
[{"type":"success","msg":["mailbox_added","no-reply@echo6.co"]}]
```
### Option B: Via Mailcow Admin UI
1. Open https://mail.echo6.co (log in as admin)
2. Navigate to **Email → Mailboxes → Add mailbox**
3. Fill in local part, domain, display name, password, quota
4. Click **Add**
### Access Flags
For service accounts (send-only), disable unnecessary access after creation:
```bash
curl -sk -X POST "https://127.0.0.1:8443/api/v1/edit/mailbox" \
-H "X-API-Key: ${MAILCOW_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"items": ["'${LOCAL_PART}@${DOMAIN}'"],
"attr": {
"sogo_access": "0",
"imap_access": "0",
"pop3_access": "0",
"smtp_access": "1"
}
}'
```
Regular user accounts can leave all access flags at their defaults (all enabled).
---
## Step 2: Fix authsource (CRITICAL for Service Accounts)
### The Problem
Mailcow domains configured with OIDC authentication (like `echo6.co` with Authentik SSO) set `authsource=generic-oidc` on **every new mailbox by default**. This tells Dovecot to authenticate the account through the OIDC provider instead of the local password hash.
For service accounts that log in via SMTP with a username and password, this means:
1. Postfix receives the SMTP AUTH credentials
2. Postfix hands them to Dovecot for verification
3. Dovecot's Lua passdb sees `authsource=generic-oidc`
4. Dovecot tries to authenticate via Authentik SSO
5. The service account doesn't exist in Authentik → **auth fails silently**
6. SMTP returns `535 5.7.8 Error: authentication failed: (reason unavailable)`
The failure message gives no indication that OIDC is the cause. The password is correct, the mailbox exists, and the Mailcow API reports success on password changes — but SMTP auth never works.
### The Fix
Change the authsource from `generic-oidc` to `mailcow` in the database:
```bash
ssh root@100.64.0.1
# Source the DB password
DBPASS=$(grep ^DBPASS /opt/mailcow-dockerized/.env | cut -d= -f2)
# Check current authsource
docker exec mailcowdockerized-mysql-mailcow-1 \
mysql -u mailcow -p${DBPASS} mailcow -N \
-e "SELECT username, authsource FROM mailbox WHERE username='${LOCAL_PART}@${DOMAIN}'"
# Fix: set authsource to local password auth
docker exec mailcowdockerized-mysql-mailcow-1 \
mysql -u mailcow -p${DBPASS} mailcow \
-e "UPDATE mailbox SET authsource='mailcow' WHERE username='${LOCAL_PART}@${DOMAIN}'"
```
**When to apply:**
| Account Type | authsource | Reason |
|-------------|-----------|--------|
| Service account (SMTP sender) | `mailcow` | Authenticates with local password via SMTP |
| Regular user (SSO login) | `generic-oidc` | Authenticates through Authentik web SSO |
| Regular user (IMAP/SMTP client) | `mailcow` | Authenticates with local password from mail client |
Rule of thumb: if the account will ever authenticate with a username + password (SMTP, IMAP, POP3), set `authsource=mailcow`. Only leave `generic-oidc` for accounts that exclusively use SSO web login.
---
## Step 3: Verify SMTP Authentication
Wait a few seconds after the authsource fix, then test:
```bash
# From the Contabo host
python3 -c "
import smtplib
s = smtplib.SMTP('mail.echo6.co', 587, timeout=10)
s.starttls()
s.login('${LOCAL_PART}@${DOMAIN}', '${PASSWORD}')
print('SMTP auth: OK')
s.quit()
"
```
If SMTP auth is being used from inside a Docker container (like Authentik), also test from there:
```bash
docker exec authentik-worker python3 -c "
import smtplib
s = smtplib.SMTP('mail.echo6.co', 587, timeout=10)
s.starttls()
s.login('${LOCAL_PART}@${DOMAIN}', '${PASSWORD}')
print('Container SMTP auth: OK')
s.quit()
"
```
Both should print `OK`. If either fails with `535 5.7.8 Error: authentication failed`, re-check the authsource (Step 2).
---
## Step 4: Store Credentials
Add the new mailbox credentials to `/home/zvx/projects/.ref/credentials`:
```
# Mailcow: ${LOCAL_PART}@${DOMAIN}
MAILCOW_${LOCAL_PART^^}_USER=${LOCAL_PART}@${DOMAIN}
MAILCOW_${LOCAL_PART^^}_PASS=${PASSWORD}
```
---
## Troubleshooting
### SMTP auth fails immediately after mailbox creation
**Cause:** `authsource=generic-oidc` (see Step 2).
### SMTP auth fails after it was previously working
**Cause 1:** Mailcow may have reset the authsource during a stack restart or update. Re-apply Step 2.
**Cause 2:** The password hash may have been corrupted by a failed API password reset. Mailcow's password edit API (`/api/v1/edit/mailbox`) sometimes reports success but doesn't actually update the hash. **Workaround:** Delete the mailbox and recreate it from scratch (Step 1), then re-apply the authsource fix (Step 2). Do not attempt to reset the password via API.
### "Unknown user" in Dovecot logs
Check that the mailbox exists and is active:
```bash
curl -sk "https://127.0.0.1:8443/api/v1/get/mailbox/${LOCAL_PART}@${DOMAIN}" \
-H "X-API-Key: ${MAILCOW_API_KEY}" | python3 -m json.tool
```
### Checking Dovecot auth logs
```bash
docker logs mailcowdockerized-dovecot-mailcow-1 --since 5m 2>&1 | grep -i auth
```
### Checking netfilter/fail2ban
Too many failed SMTP login attempts can trigger Mailcow's brute-force protection:
```bash
docker logs mailcowdockerized-netfilter-mailcow-1 --since 10m 2>&1 | grep -i ban
```
If the Contabo IP (5.189.158.149) is banned, restart the netfilter container:
```bash
cd /opt/mailcow-dockerized && docker compose restart netfilter-mailcow
```
---
## Checklist
```
[ ] Mailbox created (API or UI)
[ ] Access flags set appropriately for account type
[ ] authsource checked — set to 'mailcow' if service account
[ ] SMTP auth verified from host
[ ] SMTP auth verified from consuming container (if applicable)
[ ] Credentials stored in /home/zvx/projects/.ref/credentials
```
---
## Reference: Current Service Accounts
| Mailbox | Used By | authsource | Purpose |
|---------|---------|-----------|---------|
| no-reply@echo6.co | Authentik | mailcow | SSO invitation emails, notifications |
| cipher@echo6.co | CIPHER | generic-oidc | Daily intelligence briefs |
| recon@echo6.co | RECON | generic-oidc | Pipeline notifications |
| fulcrum@echo6.co | Fulcrum | generic-oidc | Hub notifications |
**Note:** cipher, recon, and fulcrum currently use `generic-oidc`. If any of these need to send mail via SMTP (not through the SSO web UI), their authsource must be changed to `mailcow` per Step 2.
---
*Created: 2026-02-16*

734
runbooks/meshtastic-sidecar-node.md Normal file
View file

@ -0,0 +1,734 @@
# Meshtastic Sidecar Node — Modular Deployment Runbook
Deploy a Raspberry Pi node with a real Meshtastic radio and an operator-selected combination of software modules. Each module is self-contained — install only what the site needs.
---
## Input Variables
Fill these in before running any module:
| Variable | Value | Description |
|----------|-------|-------------|
| `HOSTNAME` | | Node hostname (e.g., `mt-isr`) |
| `NODE_IP` | | Local IP address (e.g., `192.168.1.112`) |
| `SSH_USER` | | SSH username (e.g., `isr`) |
| `SSH_PASS` | | SSH password (from `.ref/credentials`) |
| `RADIO_DEVICE` | | Serial device for radio (e.g., `/dev/ttyACM0`) |
| `OS_VERSION` | | `debian-12` (bookworm) or `debian-13` (trixie) |
---
## Module Menu
Select which modules to deploy. Check off as completed:
```
Meshtastic Sidecar Node Deployment: $HOSTNAME ($NODE_IP)
Select modules to install:
[ ] Module 1: IP Settings — Static IP, hostname, DNS
[ ] Module 2: Tailscale — Headscale VPN registration
[ ] Module 3: meshtasticd — Meshtastic radio daemon
[ ] Module 4: Meshtastic Python — meshtastic CLI + Python API
[ ] Module 5: advBBS — Bulletin board system
[ ] Module 6: Meshing-Around — Mesh bot + WebGUI
[ ] Module 7: MeshMonitor — Feed mesh data to monitoring
```
### Dependencies
```
Module 1 ─── standalone
Module 2 ─── standalone
Module 3 ─── standalone (but needed by 4, 5, 6)
Module 4 ─── requires Module 3
Module 5 ─── requires Module 3
Module 6 ─── requires Module 3
Module 7 ─── requires Module 2
```
---
## Module 1: IP Settings
### Set hostname
```bash
sshpass -p '$SSH_PASS' ssh $SSH_USER@$NODE_IP
sudo hostnamectl set-hostname $HOSTNAME
echo "127.0.1.1 $HOSTNAME" | sudo tee -a /etc/hosts
```
### Configure static IP (if not using DHCP reservation)
For NetworkManager-managed systems (Raspberry Pi OS with desktop):
```bash
sudo nmcli con mod "preconfigured" \
ipv4.method manual \
ipv4.addresses "$NODE_IP/24" \
ipv4.gateway "192.168.1.1" \
ipv4.dns "1.1.1.1,8.8.8.8"
sudo nmcli con up "preconfigured"
```
For headless systems with `/etc/network/interfaces`:
```bash
sudo tee /etc/network/interfaces.d/eth0 << EOF
auto eth0
iface eth0 inet static
address $NODE_IP/24
gateway 192.168.1.1
dns-nameservers 1.1.1.1 8.8.8.8
EOF
sudo systemctl restart networking
```
For WiFi-only Pis, use `wlan0` instead of `eth0` and configure via `wpa_supplicant` or NetworkManager.
### Configure DNS fallback
```bash
echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf
echo "nameserver 8.8.8.8" | sudo tee -a /etc/resolv.conf
```
### Verify
```bash
hostname # Should show $HOSTNAME
ip addr show # Should show $NODE_IP
ping -c 3 1.1.1.1 # Internet connectivity
ping -c 3 192.168.1.1 # Gateway reachable
```
---
## Module 2: Tailscale
### Choose tailnet
| Tailnet | Headscale URL | Prefix | Key generation |
|---------|---------------|--------|----------------|
| Echo6 | `https://vpn.echo6.co` | 100.64.0.0/10 | On Contabo |
| IdahoMesh | `https://vpn.idahomesh.com` | 100.100.0.0/16 | On CT 106 |
### Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
```
### Generate preauthkey
**Echo6** (from cortex or any machine with Tailscale access to Contabo):
```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```
**IdahoMesh** (from utility Proxmox host):
```bash
ssh root@192.168.1.241 'pct exec 106 -- headscale preauthkeys create --user <USER_ID> --expiration 24h'
```
Users: malice (4), sidpatchy (2), nebra (3)
### Register with Headscale
```bash
sudo tailscale up \
--login-server=$HEADSCALE_URL \
--authkey=$PREAUTH_KEY \
--hostname=$HOSTNAME
```
### Install DNS bootstrap drop-in (reboot-safe)
Prevents chicken-and-egg DNS failure where tailscaled can't resolve the coordination server after reboot:
```bash
sudo mkdir -p /etc/systemd/system/tailscaled.service.d
sudo tee /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf << 'EOF'
[Service]
ExecStartPre=/bin/sh -c "grep -q nameserver /etc/resolv.conf || echo nameserver 1.1.1.1 > /etc/resolv.conf"
EOF
sudo systemctl daemon-reload
```
### Verify
```bash
tailscale status # Should show connected
tailscale ip -4 # Should show 100.64.x.x or 100.100.x.x
ping -c 3 100.64.0.1 # Echo6: ping Contabo
ping -c 3 100.100.0.1 # IdahoMesh: ping Headscale
```
### Enable accept-routes (if needed)
Required to reach subnets advertised by the mesh-bridge (CT 107):
```bash
sudo tailscale up --login-server=$HEADSCALE_URL --accept-routes
```
---
## Module 3: meshtasticd
### Detect OS and add OBS repo
**Debian 12 (bookworm):**
```bash
curl -fsSL https://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/Release.key | \
sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/network_Meshtastic_beta.gpg
echo 'deb http://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/ /' | \
sudo tee /etc/apt/sources.list.d/meshtasticd.list
```
**Debian 13 (trixie) / Ubuntu 24.04:**
```bash
curl -fsSL https://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_13/Release.key | \
sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/network_Meshtastic_beta.gpg
echo 'deb http://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_13/ /' | \
sudo tee /etc/apt/sources.list.d/meshtasticd.list
```
### Install
```bash
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y meshtasticd
```
### Configure — choose hardware type
#### Option A: USB radio (`/dev/ttyACM0` or `/dev/ttyUSB0`)
```bash
sudo tee /etc/meshtasticd/config.yaml << EOF
Lora:
Module: sx1262
DIO2_AS_RF_SWITCH: true
CS: 8
IRQ: 22
Busy: 27
Reset: 17
Serial:
Enabled: true
Device: $RADIO_DEVICE
Networking:
EnableUDP: true
EOF
```
> **Note:** If using a USB-connected Meshtastic device (RAK, T-Beam, etc.), the `Lora` section should be removed entirely and the device is managed via the serial interface. The config becomes:
```bash
sudo tee /etc/meshtasticd/config.yaml << EOF
Serial:
Enabled: true
Device: $RADIO_DEVICE
Networking:
EnableUDP: true
EOF
```
#### Option B: Nebra SX1262 Pi Hat (e.g., aida-nebra)
```bash
sudo tee /etc/meshtasticd/config.yaml << EOF
Lora:
Module: sx1262
DIO2_AS_RF_SWITCH: true
CS: 24
IRQ: 22
Busy: 27
Reset: 17
Networking:
EnableUDP: true
EOF
```
#### Option C: SIM mode (no physical radio, virtual mesh node)
```bash
sudo tee /etc/meshtasticd/config.yaml << EOF
Lora:
Module: sim
General:
MACAddress: "DE:AD:00:XX:XX:XX"
Networking:
EnableUDP: true
EOF
```
### Fix permissions
```bash
sudo mkdir -p /var/lib/meshtasticd/vfs
sudo chown -R meshtasticd:meshtasticd /var/lib/meshtasticd
sudo chown -R meshtasticd:meshtasticd /etc/meshtasticd
```
### Add user to dialout group (for serial access)
```bash
sudo usermod -a -G dialout meshtasticd
```
### Enable and start
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now meshtasticd
```
### Verify
```bash
sudo systemctl status meshtasticd # Should be active (running)
sudo journalctl -u meshtasticd -n 30 # Check for radio detection
ss -tlnp | grep 4403 # API port listening
```
Look for lines like `Connected to radio` or `Detected module: sx1262` in the journal output.
---
## Module 4: Meshtastic Python
**Requires:** Module 3 (meshtasticd must be running)
### Install
```bash
sudo apt-get install -y python3-pip python3-venv
pip3 install --break-system-packages meshtastic
```
Or in a venv if the system enforces PEP 668:
```bash
python3 -m venv /opt/meshtastic-cli
/opt/meshtastic-cli/bin/pip install meshtastic
sudo ln -sf /opt/meshtastic-cli/bin/meshtastic /usr/local/bin/meshtastic
```
### Set node identity
```bash
# Set long name (visible in client apps and mesh maps)
meshtastic --host localhost --set-owner "$HOSTNAME"
# Set short name (4 chars max, shown in compact views)
meshtastic --host localhost --set-owner-short "${HOSTNAME:0:4}"
```
### Set device role
| Role | Use case |
|------|----------|
| CLIENT | Default, standard mesh participant |
| CLIENT_MUTE | Receives but doesn't rebroadcast |
| ROUTER | Prioritizes forwarding, minimal local traffic |
| ROUTER_CLIENT | Router + normal client features |
```bash
meshtastic --host localhost --set device.role CLIENT
```
### Set position (optional, for mesh maps)
```bash
meshtastic --host localhost --setlat XX.XXXX --setlon -XXX.XXXX --setalt XXXX
```
### Verify
```bash
meshtastic --host localhost --info
```
Should show node name, ID, role, and radio parameters. If it says "Error connecting," verify meshtasticd is running and no other client is connected to port 4403.
> **Important:** Only ONE client can connect to the meshtasticd API at a time. If a service (advBBS, Meshing-Around) is already connected, disconnect it first before using the CLI.
---
## Module 5: advBBS
**Requires:** Module 3 (meshtasticd)
### Install prerequisites
```bash
sudo apt-get install -y python3-pip python3-venv git docker.io docker-compose
sudo systemctl enable --now docker
sudo usermod -a -G docker $SSH_USER
```
### Clone and configure
```bash
sudo mkdir -p /opt/advbbs
cd /opt/advbbs
sudo git clone https://forge.echo6.co/advbbs/advbbs.git .
sudo cp config.example.toml config.toml
```
Edit `config.toml`:
```toml
[bbs]
name = "$HOSTNAME BBS"
callsign = "${HOSTNAME^^}"
admin_password = "CHANGE_THIS"
[meshtastic]
connection_type = "tcp"
tcp_host = "localhost"
tcp_port = 4403
[database]
backup_path = "/data/backups"
backup_interval_hours = 24
[sync]
enabled = true
# Add federation peers as needed:
# [[sync.peers]]
# node_id = "!abc12345"
# name = "REMOTE-BBS"
# protocol = "advbbs"
# enabled = true
```
### Deploy with Docker
Standard (x86):
```bash
cd /opt/advbbs
sudo docker-compose up -d
```
Raspberry Pi (memory-optimized):
```bash
cd /opt/advbbs
sudo docker-compose -f docker-compose.rpi.yml build
sudo docker-compose -f docker-compose.rpi.yml up -d
```
### Deploy without Docker (native, for minimal Pi setups)
```bash
cd /opt/advbbs
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Create systemd service
sudo tee /etc/systemd/system/advbbs.service << EOF
[Unit]
Description=advBBS Meshtastic BBS
After=network.target meshtasticd.service
Requires=meshtasticd.service
[Service]
Type=simple
User=$SSH_USER
WorkingDirectory=/opt/advbbs
ExecStart=/opt/advbbs/venv/bin/python -m advbbs
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now advbbs
```
### Verify
```bash
sudo docker-compose logs -f # Docker
sudo journalctl -u advbbs -f # Native
# From another mesh node, DM the BBS node:
# !help
# Should get a response listing commands
```
---
## Module 6: Meshing-Around
**Requires:** Module 3 (meshtasticd)
### Option A: Docker (recommended for nodes with >1GB RAM)
```bash
sudo apt-get install -y docker.io docker-compose
sudo systemctl enable --now docker
sudo mkdir -p /opt/meshing-around
cd /opt/meshing-around
# Clone from upstream
sudo git clone https://github.com/SpudGunMan/meshing-around.git .
# Create config — point at local meshtasticd
# Edit mesh.ini or config file as needed:
# interface_type = tcp
# hostname = localhost
# port = 4403
sudo docker-compose up -d
```
WebGUI accessible at `http://$NODE_IP:8085`
### Option B: Native Python (for RPi Zero 2 W or low-memory nodes)
```bash
sudo mkdir -p /opt/meshing-around
cd /opt/meshing-around
sudo git clone https://github.com/SpudGunMan/meshing-around.git .
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Create systemd service
sudo tee /etc/systemd/system/meshing-around.service << EOF
[Unit]
Description=Meshing-Around Mesh Bot + WebGUI
After=network.target meshtasticd.service
Requires=meshtasticd.service
[Service]
Type=simple
User=$SSH_USER
WorkingDirectory=/opt/meshing-around
ExecStart=/opt/meshing-around/venv/bin/python mesh_bot.py
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now meshing-around
```
### Verify
```bash
sudo docker-compose logs -f # Docker
sudo journalctl -u meshing-around -f # Native
curl -s http://localhost:8085 | head -5 # WebGUI responds
```
---
## Module 7: MeshMonitor
**Requires:** Module 2 (Tailscale, to reach MeshMonitor at 100.64.0.7)
MeshMonitor runs on CT 100 (192.168.1.100 / 100.64.0.7:8080). This module configures the sidecar node to report mesh data to the MeshMonitor instance.
### Configure MeshMonitor connection
MeshMonitor connects to sidecar nodes via the meshtasticd TCP API. The sidecar node needs to be reachable from MeshMonitor over Tailscale.
1. Ensure meshtasticd is running and listening on port 4403 (Module 3)
2. Ensure Tailscale is connected (Module 2) so MeshMonitor can reach this node
3. Register the node in MeshMonitor's web UI:
```bash
# MeshMonitor admin credentials (from .ref/credentials)
# URL: http://100.64.0.7:8080
# User: admin
# Pass: 7redditGold
# Open MeshMonitor web UI and add this node:
# - Node address: $TAILSCALE_IP:4403
# - Node name: $HOSTNAME
```
### Verify
```bash
# Confirm Tailscale can reach MeshMonitor
ping -c 3 100.64.0.7
curl -s http://100.64.0.7:8080 | head -5
# Confirm meshtasticd API is accessible from the network
ss -tlnp | grep 4403
```
---
## Post-Deploy Checklist
```
[ ] IP / hostname configured (Module 1)
[ ] Tailscale connected to correct tailnet (Module 2, if selected)
[ ] meshtasticd running, radio detected (Module 3, if selected)
[ ] Node visible on mesh (check from another node)
[ ] Selected service modules running (advBBS, Meshing-Around, etc.)
[ ] All services survive reboot: sudo reboot && verify after
[ ] Credentials logged in .ref/credentials
[ ] Node added to environment.md (IP, Tailscale IP, purpose)
[ ] Node added to services.md (if running services)
```
---
## Troubleshooting
### meshtasticd won't start
```bash
sudo journalctl -u meshtasticd -n 50 --no-pager
# Common issues:
# - Serial device not found: check ls -la /dev/ttyACM0 (or ttyUSB0)
# - Permission denied: sudo chown -R meshtasticd:meshtasticd /var/lib/meshtasticd /etc/meshtasticd
# - Wrong OBS repo for OS version: Debian 12 vs 13 repo URL mismatch
```
### Radio not detected
```bash
# List serial devices
ls -la /dev/ttyACM* /dev/ttyUSB*
# Check kernel messages for USB events
dmesg | tail -20
# If device was unplugged/replugged, restart meshtasticd
sudo systemctl restart meshtasticd
```
### Meshtastic CLI can't connect
```bash
# Only ONE client can connect at a time
# Stop any running services first:
sudo systemctl stop advbbs meshing-around 2>/dev/null
# Then try CLI
meshtastic --host localhost --info
# Restart services when done
sudo systemctl start advbbs meshing-around 2>/dev/null
```
### Tailscale won't connect after reboot
```bash
# Check if DNS bootstrap drop-in is installed
cat /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf
# If missing, install it (Module 2)
# If present, check resolv.conf
cat /etc/resolv.conf
# Force fallback DNS and restart
echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf
sudo systemctl restart tailscaled
tailscale status
```
### advBBS won't connect to meshtasticd
```bash
# Verify meshtasticd is running and API port is open
ss -tlnp | grep 4403
# Check advBBS config points to localhost:4403
grep -A 3 '\[meshtastic\]' /opt/advbbs/config.toml
# Verify no other client is holding the connection
# meshtasticd only allows ONE concurrent API client
```
### Node not visible on mesh
```bash
# Check meshtasticd logs for radio status
sudo journalctl -u meshtasticd | grep -i -E "radio|connect|error"
# Verify UDP is enabled (for SIM nodes or multi-daemon setups)
grep -i udp /etc/meshtasticd/config.yaml
# Verify node identity is set
meshtastic --host localhost --info 2>/dev/null || echo "Another client connected — stop services first"
```
### Low memory (RPi Zero 2 W)
The RPi Zero 2 W has ~416MB RAM. Monitor usage:
```bash
free -h
# If memory is tight:
# - Use native Python installs instead of Docker
# - Only run ONE service alongside meshtasticd
# - Use advBBS's docker-compose.rpi.yml if using Docker
# - Disable web-reader profile to save memory
```
---
## Quick Reference
```bash
# Service management
sudo systemctl status meshtasticd
sudo systemctl status advbbs
sudo systemctl status meshing-around
sudo journalctl -u meshtasticd -f
# Mesh CLI (stop services first!)
meshtastic --host localhost --info
meshtastic --host localhost --set-owner "NodeName"
meshtastic --host localhost --set device.role CLIENT
# Tailscale
tailscale status
tailscale ip -4
tailscale ping <other-node>
```
---
## Existing Nodes Reference
| Node | IP | User | Radio | OS | Modules |
|------|-----|------|-------|-----|---------|
| aida-nebra | 192.168.1.253 | zvx | Nebra SX1262 Hat | RPi OS | 2 (Echo6), 3, 4 |
| mt-burleybutte | 192.168.1.185 | bb | Nebra SX1262 Hat | RPi OS | 2 (IdahoMesh), 3, 4 |
| mt-isr | 192.168.1.112 | isr | USB `/dev/ttyACM0` | Debian 13 | 3 (installed, inactive) |
---
*Last updated: 2026-02-21*

417
runbooks/meshtasticd-sim-nodes-runbook.md Normal file
View file

@ -0,0 +1,417 @@
# Meshtasticd SIM Node Runbook — LXC Deployment
## Overview
This runbook covers deploying meshtasticd SIM (virtual) nodes inside LXC containers on Proxmox, each paired with a dedicated service (BBS, MeshSense, etc.). SIM nodes communicate with your real radio node over UDP and appear as normal nodes on the mesh — clients, maps, and other services can't tell the difference.
**Design principle:** One container = one SIM daemon + one service. Clean isolation, easy to snapshot, migrate, or tear down without affecting anything else.
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Proxmox Host │
│ │
│ ┌───────────────────┐ │
│ │ Real Radio Node │ (LXC, bare metal, Pi, or USB device) │
│ │ meshtasticd │ │
│ │ Port 4403 │ │
│ │ /dev/ttyUSB0 │ │
│ │ UDP enabled │ │
│ └────────┬──────────┘ │
│ │ UDP (vmbr0 or dedicated bridge) │
│ │ │
│ ┌─────┴─────┬───────────────┐ │
│ │ │ │ │
│ ┌──▼────────┐ ┌▼────────────┐ ┌▼────────────┐ │
│ │ LXC: BBS │ │ LXC: Sense │ │ LXC: Bot │ ...more as │
│ │ │ │ │ │ │ needed │
│ │ mesht. sim│ │ mesht. sim │ │ mesht. sim │ │
│ │ port 4403 │ │ port 4403 │ │ port 4403 │ │
│ │ + BBS svc │ │ + MeshSense │ │ + bot svc │ │
│ └───────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
Since each SIM daemon is alone in its container, they can all use the default port (4403) internally. No port juggling needed.
---
## Prerequisites
- Proxmox host with LXC support
- A working real radio meshtasticd instance somewhere on the network with UDP enabled
- An LXC template (Ubuntu 22.04/24.04 or Debian 12 recommended)
- Network bridge accessible to both the real radio node and LXC containers
---
## Step 1: Create the LXC Container
From the Proxmox host CLI:
```bash
# Create an unprivileged container with static IP and TUN device for Tailscale
pct create <CTID> local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname mesh-<service> \
--memory 512 \
--cores 1 \
--rootfs local-lvm:4 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.<CTID>/24,gw=192.168.1.1 \
--unprivileged 1 \
--features nesting=1 \
--start 0 \
--password <from .ref/credentials>
# Add TUN device for Tailscale (must be done before first start)
cat >> /etc/pve/lxc/<CTID>.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start <CTID>
```
Adjust memory/cores/storage to taste. SIM daemons are lightweight — 512MB RAM and 1 core is plenty for the daemon plus most services.
### Bootstrap standard packages
```bash
echo6-bootstrap-ct.sh <CTID>
```
If the script isn't on the Proxmox host, run `echo6-onboard-node.sh` first. See `runbooks/proxmox-onboard-node.md`.
---
## Step 2: Install meshtasticd Inside the Container
Install from the Meshtastic OBS (OpenSUSE Build Service) repository:
```bash
pct exec <CTID> -- bash -c "
# Add the Meshtastic beta repo (Debian 12)
curl -fsSL https://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/Release.key | \
gpg --dearmor -o /etc/apt/trusted.gpg.d/network_Meshtastic_beta.gpg
echo 'deb http://download.opensuse.org/repositories/network:/Meshtastic:/beta/Debian_12/ /' > \
/etc/apt/sources.list.d/meshtasticd.list
apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y meshtasticd
"
```
For Ubuntu 24.04 containers, replace `Debian_12` with `Debian_13` in both URLs above.
Verify it's installed:
```bash
pct exec <CTID> -- meshtasticd --version
```
---
## Step 3: Configure SIM Mode
Create or edit the config file. Since this is a dedicated container, you can use the default paths.
### /etc/meshtasticd/config.yaml
```yaml
Lora:
Module: sim
General:
# Prevent loading any default config.d overrides
# ConfigDirectory: /etc/meshtasticd/config.d/
# REQUIRED: Set a unique MAC address for this SIM node
# Last 3 hex pairs = node color in client apps
MACAddress: "DE:AD:00:FF:00:01"
```
### MAC Address Guidelines
- **Every SIM node must have a unique MAC.** If two nodes share a MAC, you'll get node ID collisions and unpredictable behavior.
- The last 3 byte pairs map to a hex color code displayed in client apps.
- Pick a scheme that makes sense for your deployment, e.g.:
- `DE:AD:00:FF:00:01` — SIM node 1 (BBS)
- `DE:AD:00:00:FF:02` — SIM node 2 (MeshSense)
- `DE:AD:00:FF:FF:03` — SIM node 3 (bot)
---
## Step 4: Enable UDP Bridging
UDP is what connects SIM nodes to the rest of your mesh. Every meshtasticd instance — real radio and all SIM nodes — needs UDP enabled and must be able to reach each other on the network.
Add to your SIM node's config:
```yaml
Networking:
EnableUDP: true
```
**Also ensure your real radio node has UDP enabled** in its own config.
### Network Considerations
For UDP mesh traffic to flow between LXC containers and your real radio node:
- All containers and the real radio host must be on the **same Layer 2 network** (same bridge, same subnet) — UDP broadcast/multicast needs to reach all instances.
- If your real radio runs on a different host (e.g., a Raspberry Pi), make sure it's on the same VLAN/subnet as the LXC bridge.
- Proxmox's default `vmbr0` bridge works fine if everything is on the same network.
- If you're running the real radio in its own LXC and passing through USB, the same bridge rules apply.
**Firewall note:** If you have Proxmox firewall or iptables rules on the host, ensure UDP traffic between containers is not blocked. Meshtasticd uses UDP broadcast by default — verify your bridge allows broadcast forwarding.
---
## Step 5: Configure the systemd Service
The meshtasticd package likely installs a default unit file. If you need to customize it:
```bash
sudo systemctl edit meshtasticd --full
```
Or create/verify `/etc/systemd/system/meshtasticd.service`:
```ini
[Unit]
Description=Meshtastic Daemon - SIM Node
After=network.target
[Service]
Type=simple
User=meshtasticd
Group=meshtasticd
ExecStart=/usr/bin/meshtasticd -d /var/lib/meshtasticd/vfs -c /etc/meshtasticd/config.yaml
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
```
Since this is the only meshtasticd instance in the container, you don't need custom port flags — the default port is fine.
Ensure directory ownership:
```bash
sudo mkdir -p /var/lib/meshtasticd/vfs
sudo chown -R meshtasticd:meshtasticd /var/lib/meshtasticd
sudo chown -R meshtasticd:meshtasticd /etc/meshtasticd
```
Start and enable:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now meshtasticd
sudo systemctl status meshtasticd
```
---
## Step 5b: Install Tailscale and Register with Headscale
Install Tailscale inside the container:
```bash
pct exec <CTID> -- bash -c "
echo nameserver 1.1.1.1 > /etc/resolv.conf
echo nameserver 8.8.8.8 >> /etc/resolv.conf
curl -fsSL https://tailscale.com/install.sh | sh
"
```
Generate a preauth key on Contabo (user ID 1 = echo6):
```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```
Register the node:
```bash
pct exec <CTID> -- tailscale up --login-server https://vpn.echo6.co --authkey <PREAUTH_KEY> --hostname mesh-<service>
# Verify
pct exec <CTID> -- tailscale status
```
**Note:** The TUN device must already be configured in the container config (done in Step 1). If Tailscale fails to start, verify `/dev/net/tun` exists inside the container.
---
## Step 6: Configure the SIM Node
With the daemon running and no service connected yet, configure it via CLI:
```bash
# Install the Meshtastic Python CLI
pip install meshtastic
# Check that the node is up
meshtastic --host localhost --info
# Set a descriptive name
meshtastic --host localhost --set-owner "BBS Node"
meshtastic --host localhost --set-owner-short "BBS"
# Optionally set a position (makes it appear on mesh maps)
meshtastic --host localhost --setlat XX.XXXX --setlon -XXX.XXXX --setalt XXXX
# Set the node role as appropriate
meshtastic --host localhost --set device.role CLIENT
```
If you don't set a position, the node still functions on the mesh but won't appear on maps.
---
## Step 7: Install and Connect Your Service
Now install whichever service this container is dedicated to and point it at `localhost:4403` (or whatever the default meshtasticd API port is).
### Example: BBS
```bash
# Install your BBS software of choice
# Point it at the local meshtasticd instance
# BBS_CONFIG: host=localhost, port=4403
```
### Example: MeshSense
```bash
# MeshSense supports non-default ports
# Configure it to connect to localhost:4403
```
**Reminder:** One client API connection per daemon. Once the service is connected, don't also try to connect a client app to the same instance. If you need to reconfigure the node, stop the service first.
---
## Provisioning Additional Containers
For each new service, repeat steps 1–7 with:
1. A new container ID and hostname (e.g., `mesh-sense`, `mesh-bot`)
2. A **unique MAC address** in config.yaml
3. The specific service installed alongside meshtasticd
### Quick Clone Approach
Once you have one container fully set up, you can clone it in Proxmox and just change:
- Container hostname
- MAC address in `/etc/meshtasticd/config.yaml`
- The service installed/configured
- Node owner name via `meshtastic --host localhost --set-owner "NewName"`
```bash
# Clone from Proxmox CLI
pct clone 201 202 --hostname mesh-sense --full
pct start 202
pct enter 202
# Update the MAC address
nano /etc/meshtasticd/config.yaml # change MACAddress
# Restart meshtasticd to pick up new MAC
systemctl restart meshtasticd
# Reconfigure node identity
meshtastic --host localhost --set-owner "MeshSense"
```
---
## USB Passthrough for Real Radio Container
If you want your real radio node in an LXC too (rather than bare metal), you need to pass the USB device through to the container.
On the Proxmox host, find the device:
```bash
ls -la /dev/serial/by-id/
# or
lsusb
```
Add to the container config (`/etc/pve/lxc/<CTID>.conf`):
```
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.mount.entry: /dev/ttyUSB0 dev/ttyUSB0 none bind,optional,create=file
```
Adjust the device path and cgroup major number as needed for your hardware. The container will also need a `config.yaml` with the real LoRa module config instead of `Module: sim`.
---
## Troubleshooting
### SIM nodes not seeing the real radio (or each other)
- Verify UDP is enabled on **all** instances
- Confirm all containers are on the same bridge/subnet
- Check for firewall rules blocking UDP broadcast between containers
- Test basic connectivity: `ping` between containers
### Node ID collisions / "things get weird"
- Every SIM node must have a **unique MAC address** — check each container's config.yaml
- After changing a MAC, restart meshtasticd and verify with `meshtastic --host localhost --info`
### Service can't connect to meshtasticd
- Is meshtasticd actually running? `systemctl status meshtasticd`
- Is another client already connected? Only one API connection per instance.
- Check the port: `ss -tlnp | grep 4403`
### Config not taking effect
- Make sure `ConfigDirectory` is commented out or pointed somewhere empty so default config.d files don't override your settings
- Restart after config changes: `systemctl restart meshtasticd`
### Permission errors
- VFS and config directories must be owned by the `meshtasticd` user
```bash
chown -R meshtasticd:meshtasticd /var/lib/meshtasticd
chown -R meshtasticd:meshtasticd /etc/meshtasticd
```
---
## Quick Reference
```bash
# Inside any SIM container:
systemctl status meshtasticd # Check daemon status
journalctl -u meshtasticd -f # Follow logs
meshtastic --host localhost --info # Node info (only if no service is connected)
# From Proxmox host:
pct list # List all containers
pct enter <CTID> # Shell into container
pct exec <CTID> -- systemctl status meshtasticd # Check without entering
```
---
## Container Inventory Template
Track your deployment:
| CTID | Hostname | MAC Address | Service | Port | Notes |
|------|-------------|---------------------|------------|------|-----------------|
| 200 | mesh-radio | (hardware) | Real radio | 4403 | USB passthrough |
| 201 | mesh-bbs | DE:AD:00:FF:00:01 | BBS | 4403 | — |
| 202 | mesh-sense | DE:AD:00:00:FF:02 | MeshSense | 4403 | — |
| 203 | mesh-bot | DE:AD:00:FF:FF:03 | Bot | 4403 | — |
---
## Credits
Procedure sourced from a community discussion between pdxlocs, tedward, and wehooper4 regarding multi-daemon meshtasticd deployments with SIM mode. Adapted for LXC/Proxmox deployment.

343
runbooks/nordvpn-lxc.md Normal file
View file

@ -0,0 +1,343 @@
# NordVPN / WireGuard in LXC
Set up VPN with IP rotation inside an LXC container. Handles the LXC-specific gotchas: TUN device, systemd compatibility, split tunneling so local services stay reachable.
---
## Prerequisites
- LXC container provisioned and running (see `ct-runbook.md`)
- SSH access to both the Proxmox host and the container
- NordVPN account with a service token (from https://my.nordaccount.com/dashboard/nordvpn/access-tokens/)
---
## Inputs
Prompt the user for all of these before executing:
```
CTID= # Container ID on Proxmox host
CT_HOST= # SSH alias or IP for the container
PVE_HOST= # SSH alias or IP for the Proxmox host
NORDVPN_TOKEN= # NordVPN service token
VPN_COUNTRIES= # Comma-separated rotation list (e.g., "United_States,Canada,United_Kingdom,Germany,Netherlands,Sweden")
VPN_CONFIG_DIR= # Where to store WireGuard configs inside CT (e.g., /opt/vpn)
```
---
## Step 1: Enable TUN Device on Container
**Run on Proxmox host.** LXC containers don't have `/dev/net/tun` by default — VPN won't work without it.
```bash
ssh $PVE_HOST "grep -q 'dev/net/tun' /etc/pve/lxc/${CTID}.conf 2>/dev/null || {
echo 'lxc.cgroup2.devices.allow: c 10:200 rwm' >> /etc/pve/lxc/${CTID}.conf
echo 'lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file' >> /etc/pve/lxc/${CTID}.conf
echo 'TUN device added — container restart required'
}"
```
If lines were added, restart the container:
```bash
ssh $PVE_HOST "pct reboot $CTID"
```
### Gate
```bash
ssh $CT_HOST 'ls -la /dev/net/tun'
```
Must show the device. If missing, the cgroup/mount entries didn't take — check `/etc/pve/lxc/${CTID}.conf`.
---
## Step 2: Try NordVPN CLI (Option A)
The CLI is the simplest path but requires working systemd in the container (which most LXCs have, but some stripped-down templates don't).
```bash
ssh $CT_HOST 'sh <(curl -sSf https://downloads.nordcdn.com/apps/linux/install.sh)'
```
If the installer completes without errors:
```bash
ssh $CT_HOST "nordvpn login --token $NORDVPN_TOKEN"
ssh $CT_HOST 'nordvpn set technology nordlynx' # WireGuard-based, faster
ssh $CT_HOST 'nordvpn set killswitch off' # Don't kill local services
ssh $CT_HOST 'nordvpn set autoconnect off' # We control rotation
ssh $CT_HOST 'nordvpn set dns off' # Keep container's DNS
```
### Test
```bash
ssh $CT_HOST 'nordvpn connect United_States && sleep 3 && curl -s https://ifconfig.me && nordvpn disconnect'
```
Must show a non-local IP. If it does, **skip to Step 4** (rotation script).
### Common failures
- **"Whoops! /run/nordvpn/nordvpnd.sock not found"** — nordvpnd service didn't start. Check `systemctl status nordvpnd`. If systemd is broken in this LXC, fall through to Option B.
- **"Permission denied creating /dev/net/tun"** — Step 1 TUN device not configured. Go back.
- **Installer hangs on "Starting NordVPN daemon"** — systemd issue. Kill it, fall through to Option B.
---
## Step 3: WireGuard Manual Configs (Option B — Fallback)
Use this if NordVPN CLI doesn't work in the LXC.
### Install WireGuard
```bash
ssh $CT_HOST 'apt install -y wireguard-tools curl jq'
```
### Generate NordVPN WireGuard configs
NordVPN provides WireGuard configs via their API. Generate one per country:
```bash
ssh $CT_HOST "mkdir -p $VPN_CONFIG_DIR"
# Get NordVPN WireGuard private key
# Method: Use the NordVPN API with your token to get credentials
# This requires the nordvpn CLI to extract the private key, OR manual setup:
#
# 1. Go to https://my.nordaccount.com/dashboard/nordvpn/manual-configuration/
# 2. Generate WireGuard credentials
# 3. Download configs for each country
# 4. SCP them to the container
# Place configs as: $VPN_CONFIG_DIR/us.conf, ca.conf, uk.conf, de.conf, nl.conf, se.conf
```
**⚠️ Manual step required:** NordVPN's WireGuard config generation requires either the CLI (which didn't work) or manual download from the NordVPN dashboard. Download `.conf` files for each country in the rotation list and SCP them to the container.
### Config format
Each `.conf` file should look like:
```ini
[Interface]
PrivateKey = <your-wireguard-private-key>
Address = 10.5.0.2/16
DNS = 103.86.96.100
[Peer]
PublicKey = <server-public-key>
AllowedIPs = 0.0.0.0/0
Endpoint = <server-ip>:51820
PersistentKeepalive = 25
```
**Critical for LXC:** If the container runs services that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
```ini
# Route everything EXCEPT local network through VPN
AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
# This covers all IPs but lets 192.168.x.x and 100.64.x.x traffic stay local
```
Or more precisely, exclude your subnets:
```bash
# Generate AllowedIPs that exclude local networks
# This sends all traffic through VPN except 192.168.1.0/24 and 100.64.0.0/10
AllowedIPs = 0.0.0.0/5, 8.0.0.0/7, 11.0.0.0/8, 12.0.0.0/6, 16.0.0.0/4, 32.0.0.0/3, 64.0.0.0/3, 96.0.0.0/6, 100.0.0.0/10, 100.128.0.0/9, 101.0.0.0/8, 102.0.0.0/7, 104.0.0.0/5, 112.0.0.0/4, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/8, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/9, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/6, 200.0.0.0/5, 208.0.0.0/4, 224.0.0.0/3
```
**Simpler alternative:** Use `wg-quick` post-up/down scripts to manage routes:
```ini
[Interface]
PrivateKey = <key>
Address = 10.5.0.2/16
PostUp = ip route add 192.168.1.0/24 via $(ip route show default | awk '{print $3}') dev eth0
PostUp = ip route add 100.64.0.0/10 via $(ip route show default | awk '{print $3}') dev eth0
PreDown = ip route del 192.168.1.0/24 via $(ip route show default | awk '{print $3}') dev eth0 2>/dev/null; true
PreDown = ip route del 100.64.0.0/10 via $(ip route show default | awk '{print $3}') dev eth0 2>/dev/null; true
[Peer]
PublicKey = <key>
AllowedIPs = 0.0.0.0/0
Endpoint = <server>:51820
```
### Test
```bash
ssh $CT_HOST "wg-quick up $VPN_CONFIG_DIR/us.conf && sleep 2 && curl -s https://ifconfig.me && echo && wg-quick down $VPN_CONFIG_DIR/us.conf"
```
Must show a NordVPN IP. Verify local services still reachable:
```bash
# From another machine on the LAN, while VPN is up:
curl -s http://<CT_LOCAL_IP>:<SERVICE_PORT>/ # Must still respond
```
---
## Step 4: VPN Rotation Helper Script
Regardless of Option A or B, create a rotation script that other services can call.
```bash
ssh $CT_HOST "cat > $VPN_CONFIG_DIR/vpn-rotate.sh << 'SCRIPT'
#!/bin/bash
# VPN Rotation Script
# Usage: vpn-rotate.sh [connect|disconnect|rotate|status]
CONFIG_DIR=\"$VPN_CONFIG_DIR\"
STATE_FILE=\"$VPN_CONFIG_DIR/vpn-state.json\"
COUNTRIES=($VPN_COUNTRIES)
# Detect VPN method
if command -v nordvpn &>/dev/null && systemctl is-active --quiet nordvpnd 2>/dev/null; then
VPN_METHOD=nordvpn
else
VPN_METHOD=wireguard
fi
get_current() {
if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
nordvpn status 2>/dev/null | grep -i country | awk '{print \$NF}'
else
wg show 2>/dev/null | head -1 | awk '{print \$2}' | sed 's/.conf//'
fi
}
get_public_ip() {
curl -s --connect-timeout 5 https://ifconfig.me 2>/dev/null
}
vpn_connect() {
local country=\${1:-\${COUNTRIES[0]}}
echo \"Connecting to \$country...\"
if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
nordvpn connect \"\$country\"
else
# Disconnect any existing
for conf in \$CONFIG_DIR/*.conf; do
wg-quick down \"\$conf\" 2>/dev/null
done
local conf_file=\"\$CONFIG_DIR/\$(echo \$country | tr '[:upper:]' '[:lower:]' | cut -c1-2).conf\"
if [ -f \"\$conf_file\" ]; then
wg-quick up \"\$conf_file\"
else
echo \"ERROR: No config for \$country (\$conf_file)\"
return 1
fi
fi
sleep 3
echo \"Public IP: \$(get_public_ip)\"
}
vpn_disconnect() {
if [ \"\$VPN_METHOD\" = \"nordvpn\" ]; then
nordvpn disconnect
else
for conf in \$CONFIG_DIR/*.conf; do
wg-quick down \"\$conf\" 2>/dev/null
done
fi
}
vpn_rotate() {
local current=\$(get_current)
local next_idx=0
for i in \"\${!COUNTRIES[@]}\"; do
if echo \"\${COUNTRIES[\$i]}\" | grep -qi \"\$current\"; then
next_idx=$(( (i + 1) % \${#COUNTRIES[@]} ))
break
fi
done
vpn_disconnect
sleep 2
vpn_connect \"\${COUNTRIES[\$next_idx]}\"
}
vpn_status() {
echo \"Method: \$VPN_METHOD\"
echo \"Country: \$(get_current || echo 'disconnected')\"
echo \"IP: \$(get_public_ip || echo 'unknown')\"
}
case \"\${1:-status}\" in
connect) vpn_connect \"\$2\" ;;
disconnect) vpn_disconnect ;;
rotate) vpn_rotate ;;
status) vpn_status ;;
*) echo \"Usage: \$0 {connect [country]|disconnect|rotate|status}\" ;;
esac
SCRIPT
chmod +x $VPN_CONFIG_DIR/vpn-rotate.sh"
```
### Test rotation
```bash
ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh connect"
ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh status"
ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh rotate"
ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh status"
ssh $CT_HOST "$VPN_CONFIG_DIR/vpn-rotate.sh disconnect"
```
Each `rotate` should switch countries and show a different IP.
---
## Verification Checklist
```bash
echo "=== VPN Setup Check ==="
echo ""
echo "TUN device: $(ls /dev/net/tun 2>/dev/null && echo 'OK' || echo 'MISSING')"
echo "VPN method: $(command -v nordvpn >/dev/null && echo 'NordVPN CLI' || echo 'WireGuard')"
echo "Configs: $(ls $VPN_CONFIG_DIR/*.conf 2>/dev/null | wc -l) country configs"
echo "Rotation: $(ls $VPN_CONFIG_DIR/vpn-rotate.sh 2>/dev/null && echo 'OK' || echo 'MISSING')"
echo ""
echo "Quick connect test..."
$VPN_CONFIG_DIR/vpn-rotate.sh connect
echo "VPN IP: $(curl -s https://ifconfig.me)"
echo "Local access: $(curl -s -o /dev/null -w '%{http_code}' http://localhost:9000/ 2>/dev/null || echo 'N/A')"
$VPN_CONFIG_DIR/vpn-rotate.sh disconnect
echo "Home IP: $(curl -s https://ifconfig.me)"
```
---
## Troubleshooting
### "RTNETLINK answers: Operation not permitted" on wg-quick up
TUN device not available. Go back to Step 1. Container may need restart after adding cgroup entries.
### VPN connects but local services unreachable
Split tunneling not configured. The VPN is routing ALL traffic including LAN. Fix the `AllowedIPs` or add PostUp routes per Step 3.
### DNS stops working when VPN is up
NordVPN CLI: `nordvpn set dns off` (use container's DNS, not NordVPN's).
WireGuard: Remove the `DNS =` line from the `.conf` file.
### "Cannot open TUN/TAP dev /dev/net/tun: No such file or directory"
Container config missing TUN mount entry. Check `/etc/pve/lxc/${CTID}.conf` for both the cgroup allow and mount entry lines.
### NordVPN CLI installed but nordvpnd won't start
Common in LXC. `systemctl status nordvpnd` will usually show a cgroup or namespace error. Fall through to WireGuard (Step 3).
---
*Last updated: 2026-02-13*

458
runbooks/peertube-remote-runner.md Normal file
View file

@ -0,0 +1,458 @@
# PeerTube Remote Runner — GPU Transcoding
Deploy a PeerTube remote runner with NVENC GPU transcoding. The runner pulls jobs from PeerTube over WebSocket, transcodes with the GPU, and uploads HLS streams back.
Use this when adding a new runner node, rebuilding an existing one, or re-registering after a PeerTube rebuild.
---
## Prerequisites
- PeerTube instance running and accessible (HTTP, not necessarily HTTPS)
- PeerTube admin credentials or API access to generate runner registration tokens
- Target machine with:
- NVIDIA GPU with NVENC support (Maxwell gen 2+ / GTX 950+)
- NVIDIA drivers installed and working (`nvidia-smi` returns output)
- Node.js 18+ installed
- SSH access from CC host
If the target machine needs NVIDIA drivers or Node.js, see `proxmox-create-ubuntu-vm.md` Steps 9 and 11.
---
## Inputs
Prompt the user for all of these before executing:
```
RUNNER_HOST= # SSH alias or IP for the runner machine (e.g., cortex)
RUNNER_NAME= # Human-readable runner name (e.g., "cortex-nvenc")
RUNNER_USER= # User to run the service as (e.g., "zvx")
PT_URL= # PeerTube instance URL reachable from runner (e.g., "http://100.64.0.23:9000")
PT_HOST_HEADER= # PeerTube's public hostname for Host header (e.g., "stream.echo6.co")
PT_ADMIN_USER= # PeerTube admin username (e.g., "root")
PT_ADMIN_PASS= # PeerTube admin password
INSTALL_DIR= # Where to put runner config (default: /opt/peertube-runner)
```
---
## Step 1: Verify GPU
```bash
ssh $RUNNER_HOST 'nvidia-smi --query-gpu=name,driver_version,memory.total,encoder.stats.sessionCount --format=csv,noheader'
```
### Gate
Must return GPU name, driver version, and VRAM. If it fails:
- No output → NVIDIA drivers not installed. See `proxmox-create-ubuntu-vm.md` Step 9.
- "NVML: Driver/library version mismatch" → reboot the machine.
- "No devices found" → GPU passthrough not configured (VMs) or hardware issue.
Check NVENC specifically:
```bash
ssh $RUNNER_HOST 'nvidia-smi -q | grep -A 5 "Encoder"'
```
Must show encoder session info. If "N/A", the GPU doesn't support NVENC or drivers are too old.
---
## Step 2: Install peertube-runner
PeerTube runner is distributed via npm.
```bash
ssh $RUNNER_HOST 'which node && node --version' # Must be 18+
```
Install the runner:
```bash
ssh $RUNNER_HOST 'sudo npm install -g @peertube/peertube-runner'
ssh $RUNNER_HOST 'which peertube-runner && peertube-runner --version'
```
### Gate
`peertube-runner --version` must return a version number. If npm install fails, check Node.js version (must be 18+).
Create config directory:
```bash
ssh $RUNNER_HOST "sudo mkdir -p $INSTALL_DIR && sudo chown $RUNNER_USER:$RUNNER_USER $INSTALL_DIR"
```
---
## Step 3: Generate Registration Token on PeerTube
Get a registration token from the PeerTube instance. This requires admin access.
### Option A: Via API
```bash
# Get OAuth client credentials
CLIENT_CREDS=$(ssh $RUNNER_HOST "curl -s $PT_URL/api/v1/oauth-clients/local -H 'Host: $PT_HOST_HEADER'")
CLIENT_ID=$(echo "$CLIENT_CREDS" | jq -r '.client_id')
CLIENT_SECRET=$(echo "$CLIENT_CREDS" | jq -r '.client_secret')
# Get admin token
TOKEN_RESP=$(ssh $RUNNER_HOST "curl -s $PT_URL/api/v1/users/token \
-H 'Host: $PT_HOST_HEADER' \
--data 'client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&grant_type=password&username=$PT_ADMIN_USER&password=$PT_ADMIN_PASS'")
ACCESS_TOKEN=$(echo "$TOKEN_RESP" | jq -r '.access_token')
# Generate runner registration token
REG_TOKEN=$(ssh $RUNNER_HOST "curl -s -X POST $PT_URL/api/v1/runners/registration-tokens/generate \
-H 'Host: $PT_HOST_HEADER' \
-H 'Authorization: Bearer $ACCESS_TOKEN' | jq -r '.registrationToken'")
echo "Registration token: $REG_TOKEN"
```
### Option B: Via PeerTube Admin UI
1. Log into PeerTube as admin
2. Administration → System → Runners
3. Click "Generate registration token"
4. Copy the token
### Gate
Must have a registration token string. If the API returns errors:
- 401 → wrong admin credentials
- 404 → PeerTube version too old (runners require v5.2+)
- Connection refused → PeerTube not reachable from runner. Check URL and network.
---
## Step 4: Register Runner
```bash
ssh $RUNNER_HOST "peertube-runner register \
--url $PT_URL \
--registration-token $REG_TOKEN \
--runner-name $RUNNER_NAME"
```
### Gate
Must complete without errors. Verify registration:
```bash
ssh $RUNNER_HOST 'peertube-runner list-registered'
```
Must show the PeerTube instance URL. If registration fails:
- "Invalid registration token" → token already used or expired. Generate a new one.
- "ECONNREFUSED" → runner can't reach PeerTube. Test: `curl -s $PT_URL/api/v1/config`
- "self-signed certificate" → if PeerTube uses HTTPS with self-signed cert, use `NODE_TLS_REJECT_UNAUTHORIZED=0` (not recommended) or fix the cert.
---
## Step 5: Configure NVENC
The runner auto-detects ffmpeg capabilities, but verify NVENC is available to ffmpeg:
```bash
ssh $RUNNER_HOST 'ffmpeg -encoders 2>/dev/null | grep nvenc'
```
Must show `h264_nvenc` and `hevc_nvenc`. If missing, install ffmpeg with NVENC support:
```bash
# Ubuntu/Debian — the default ffmpeg usually includes NVENC if drivers are installed
ssh $RUNNER_HOST 'sudo apt install -y ffmpeg'
# Re-check
ssh $RUNNER_HOST 'ffmpeg -encoders 2>/dev/null | grep nvenc'
```
If still missing, the NVIDIA drivers may not include the encoding libraries. Install:
```bash
ssh $RUNNER_HOST 'sudo apt install -y libnvidia-encode-550' # Match your driver version
```
---
## Step 6: Create systemd Service
```bash
ssh $RUNNER_HOST "sudo tee /etc/systemd/system/peertube-runner.service > /dev/null << 'EOF'
[Unit]
Description=PeerTube Remote Runner (NVENC)
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
Requires=nvidia-persistenced.service
[Service]
Type=simple
User=$RUNNER_USER
Group=$RUNNER_USER
Environment=NODE_ENV=production
ExecStart=/usr/bin/peertube-runner server \
--enable-job vod-hls-transcoding \
--enable-job vod-audio-merge-transcoding \
--enable-job live-rtmp-hls-transcoding \
--enable-job video-studio-transcoding \
--enable-job video-transcription
WorkingDirectory=/home/$RUNNER_USER
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube-runner
MemoryMax=20G
[Install]
WantedBy=multi-user.target
EOF"
ssh $RUNNER_HOST 'sudo systemctl daemon-reload'
ssh $RUNNER_HOST 'sudo systemctl enable peertube-runner'
ssh $RUNNER_HOST 'sudo systemctl start peertube-runner'
```
### Gate
```bash
ssh $RUNNER_HOST 'systemctl is-active peertube-runner'
```
Must return `active`. If it fails, check logs:
```bash
ssh $RUNNER_HOST 'journalctl -u peertube-runner -n 50 --no-pager'
```
Common failures:
- "Cannot find module" → peertube-runner not installed globally, or PATH issue. Check `which peertube-runner`.
- "nvidia-persistenced.service not found" → remove the `Requires=` line if nvidia-persistenced isn't set up (it's optional but recommended).
---
## Step 7: Install Health Check
Cron script that auto-restarts the runner if it crashes or the GPU becomes inaccessible.
```bash
ssh $RUNNER_HOST "sudo tee $INSTALL_DIR/health.sh > /dev/null << 'HEALTH'
#!/bin/bash
LOG_TAG=\"peertube-runner-health\"
if ! systemctl is-active --quiet peertube-runner; then
logger -t \$LOG_TAG \"Runner not active, restarting...\"
systemctl restart peertube-runner
sleep 10
fi
if ! pgrep -f \"peertube-runner server\" > /dev/null; then
logger -t \$LOG_TAG \"Runner process not found, restarting service...\"
systemctl restart peertube-runner
fi
if ! nvidia-smi > /dev/null 2>&1; then
logger -t \$LOG_TAG \"GPU not accessible, restarting nvidia-persistenced and runner...\"
systemctl restart nvidia-persistenced 2>/dev/null
sleep 5
systemctl restart peertube-runner
fi
HEALTH
chmod +x $INSTALL_DIR/health.sh"
# Add cron job (every 5 minutes)
ssh $RUNNER_HOST "(crontab -l 2>/dev/null | grep -v peertube-runner-health; echo '*/5 * * * * $INSTALL_DIR/health.sh') | crontab -"
```
Verify:
```bash
ssh $RUNNER_HOST 'crontab -l | grep peertube'
```
---
## Step 8: Test Transcoding
Upload a test video and verify the full pipeline works.
### Quick test via API
```bash
# Download a short test video
ssh $RUNNER_HOST 'curl -L -o /tmp/test-video.mp4 "https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/360/Big_Buck_Bunny_360_10s_1MB.mp4" 2>/dev/null'
# Upload to PeerTube (reuse ACCESS_TOKEN from Step 3)
ssh $RUNNER_HOST "curl -s -X POST $PT_URL/api/v1/videos/upload \
-H 'Host: $PT_HOST_HEADER' \
-H 'Authorization: Bearer $ACCESS_TOKEN' \
-F 'videofile=@/tmp/test-video.mp4' \
-F 'name=Runner Test Video' \
-F 'channelId=1' \
-F 'privacy=1' \
-F 'waitTranscoding=true' | jq '{uuid, name, state}'"
```
### Verify GPU is processing
Within 30 seconds of upload:
```bash
ssh $RUNNER_HOST 'nvidia-smi --query-gpu=utilization.gpu,utilization.encoder,temperature.gpu,power.draw --format=csv,noheader'
```
GPU utilization and encoder utilization should be non-zero. If encoder shows 0% but GPU shows activity, NVENC isn't being used — check ffmpeg encoder detection (Step 5).
### Check runner logs
```bash
ssh $RUNNER_HOST 'journalctl -u peertube-runner -n 20 --no-pager | grep -i "transcod\|job\|error"'
```
Should show job pickup, transcoding progress, and completion.
### Clean up
```bash
# Delete test video via API (optional)
ssh $RUNNER_HOST "curl -s -X DELETE $PT_URL/api/v1/videos/<VIDEO_UUID> \
-H 'Host: $PT_HOST_HEADER' \
-H 'Authorization: Bearer $ACCESS_TOKEN'"
ssh $RUNNER_HOST 'rm -f /tmp/test-video.mp4'
```
---
## Verification Checklist
```bash
echo "=== PeerTube Runner Check ==="
echo ""
echo "GPU: $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'MISSING')"
echo "NVENC: $(ffmpeg -encoders 2>/dev/null | grep -c h264_nvenc) encoders"
echo "Runner ver: $(peertube-runner --version 2>/dev/null || echo 'NOT INSTALLED')"
echo "Registered: $(peertube-runner list-registered 2>/dev/null | grep -c 'http' || echo '0') instance(s)"
echo "Service: $(systemctl is-active peertube-runner 2>/dev/null || echo 'NOT RUNNING')"
echo "Health cron: $(crontab -l 2>/dev/null | grep -c peertube-runner || echo '0') entries"
echo "ffmpeg procs: $(pgrep -c ffmpeg 2>/dev/null || echo '0') active"
echo "GPU util: $(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader 2>/dev/null || echo 'N/A')"
```
---
## Adding a Second PeerTube Instance
To register the same runner with another PeerTube instance:
```bash
peertube-runner register \
--url <SECOND_PT_URL> \
--registration-token <TOKEN> \
--runner-name $RUNNER_NAME
```
The runner handles multiple registrations automatically — it polls all registered instances for jobs.
## Unregistering
```bash
# List registrations
peertube-runner list-registered
# Unregister from a specific instance
peertube-runner unregister --url $PT_URL
```
---
## Troubleshooting
### Runner picks up jobs but transcoding fails immediately
Check ffmpeg NVENC access:
```bash
ffmpeg -y -f lavfi -i testsrc=duration=5:size=1280x720:rate=30 -c:v h264_nvenc /tmp/nvenc-test.mp4
```
If this fails, NVENC isn't accessible to ffmpeg. Common causes: wrong driver version, missing libnvidia-encode, or GPU in use by another process that's holding all NVENC sessions.
### Runner shows 0 active jobs despite pending queue
- WebSocket connection issue. Check: `journalctl -u peertube-runner | grep -i websocket`
- Runner registered with wrong URL. Verify: `peertube-runner list-registered`
- PeerTube remote runners not enabled. Check PeerTube config: `transcoding.remote_runners.enabled` must be `true`
### GPU utilization stuck at 100% / NVENC sessions maxed
The RTX A4000 supports ~3 simultaneous NVENC sessions (consumer cards are limited to 3, pro cards vary). If all sessions are in use, new jobs queue on the runner side. This is normal — throughput is limited by NVENC session count, not GPU compute.
To increase throughput: patch the NVENC session limit (search "nvidia nvenc patch") or add a second runner node.
### Runner keeps disconnecting / restarting
- Check memory: `free -h`. The 20GB MemoryMax in the service file may be too low if processing many concurrent jobs. Increase if needed.
- Check disk space: transcoding uses temp space. Ensure `/tmp` or the runner's working directory has sufficient free space (10GB+ recommended).
- Network instability between runner and PeerTube. Use Tailscale IP instead of public URL for reliability.
### After PeerTube rebuild, runner can't connect
Registration is tied to the PeerTube instance. After a rebuild:
1. Unregister: `peertube-runner unregister --url $PT_URL`
2. Generate new registration token on the new PeerTube
3. Re-register: `peertube-runner register --url $PT_URL --registration-token <NEW_TOKEN> --runner-name $RUNNER_NAME`
4. Restart: `sudo systemctl restart peertube-runner`
---
## Quick Reference: Current Runners
| Runner | Host | GPU | PeerTube Instance | Status |
|--------|------|-----|-------------------|--------|
| cortex-nvenc | cortex (VM 150 on TOC) | RTX A4000 16GB | stream.echo6.co (CT 110) | Active |
---
## Whisper Transcription Setup
The runner also handles `video-transcription` jobs (auto-captioning via Whisper). A smart wrapper routes jobs based on audio duration:
### Smart Wrapper (`/usr/local/bin/whisper-smart` on cortex)
- **Model:** `medium` (good accuracy, fits in VRAM on float16)
- **GPU path (< 1 hour):** `--device cuda --compute_type float16` — ~3.7GB VRAM, fast
- **CPU path (>= 1 hour):** `--device cpu --compute_type int8` — ~8-11GB RAM, slow but avoids VRAM exhaustion
- **CPU serialization:** `flock --nonblock /tmp/whisper-cpu.lock` — only one CPU transcription at a time. If lock is held, wrapper exits 1 and the runner retries the job later.
- **Concurrency:** Runner config set to `concurrency = 2` — allows one GPU + one CPU job in parallel
### Symlink chain
```
/usr/local/bin/whisper-ctranslate2 → /usr/local/bin/whisper-smart (the smart wrapper)
/usr/local/bin/whisper-ctranslate2-real → /home/zvx/.local/bin/whisper-ctranslate2 (actual Python binary)
```
### Key config
- **Runner config:** `~/.config/peertube-runner-nodejs/default/config.toml` — `model = "medium"`, `concurrency = 2`
- **PeerTube config:** `/var/www/peertube/config/production.yaml` on CT 110 — `model-name: medium`
- **systemd MemoryMax:** `20G` (CPU int8 medium model peaks at ~11GB)
### Wrapper log
```bash
tail -f /tmp/whisper-wrapper.log # Shows mode (GPU/CPU/CPU-BLOCKED), duration, args
```
---
*Last updated: 2026-02-17 — Added Whisper smart transcription setup, MemoryMax 12G→20G, concurrency 1→2*

177
runbooks/pg-backup.md Normal file
View file

@ -0,0 +1,177 @@
# PostgreSQL Backup (Docker)
Automated pg_dump backups for any Docker-hosted PostgreSQL instance. Retention, integrity check, and restore testing included.
---
## Inputs
```
CONTAINER_NAME= # Docker container name (e.g., "matrix-postgres")
DB_NAME= # Database to back up (e.g., "synapse")
DB_USER= # Database user (e.g., "synapse")
BACKUP_DIR= # Host directory for backups (e.g., "/opt/matrix/backups")
RETENTION_DAYS=14 # Days to keep backups
CRON_SCHEDULE="0 3 * * *" # Daily at 3AM
```
---
## Step 1: Create Backup Directory
```bash
mkdir -p ${BACKUP_DIR}
chmod 700 ${BACKUP_DIR}
mkdir -p $(dirname ${BACKUP_DIR})/scripts
```
---
## Step 2: Create Backup Script
Create `$(dirname ${BACKUP_DIR})/scripts/pg_backup.sh`:
```bash
#!/usr/bin/env bash
set -euo pipefail
# --- Configuration (edit per service) ---
CONTAINER_NAME="${CONTAINER_NAME}"
DB_NAME="${DB_NAME}"
DB_USER="${DB_USER}"
BACKUP_DIR="${BACKUP_DIR}"
RETENTION_DAYS=${RETENTION_DAYS}
# --- Derived ---
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz"
LOG_FILE="${BACKUP_DIR}/backup.log"
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "${LOG_FILE}"
}
# --- Backup ---
log "Starting backup of ${DB_NAME} from ${CONTAINER_NAME}"
docker exec ${CONTAINER_NAME} pg_dump \
-U ${DB_USER} \
-d ${DB_NAME} \
--format=plain \
--no-owner \
--no-privileges \
| gzip > "${BACKUP_FILE}"
if [ $? -eq 0 ] && [ -s "${BACKUP_FILE}" ]; then
SIZE=$(du -h "${BACKUP_FILE}" | cut -f1)
log "Backup successful: ${BACKUP_FILE} (${SIZE})"
else
log "ERROR: Backup failed or produced empty file"
rm -f "${BACKUP_FILE}"
exit 1
fi
# --- Retention ---
DELETED=$(find ${BACKUP_DIR} -name "${DB_NAME}_*.sql.gz" -mtime +${RETENTION_DAYS} -print -delete | wc -l)
log "Retention cleanup: removed ${DELETED} backups older than ${RETENTION_DAYS} days"
# --- Integrity ---
if gzip -t "${BACKUP_FILE}" 2>/dev/null; then
log "Integrity check: PASS"
else
log "ERROR: Integrity check FAILED — backup is corrupt"
exit 1
fi
log "Backup complete."
```
Make executable:
```bash
chmod +x $(dirname ${BACKUP_DIR})/scripts/pg_backup.sh
```
---
## Step 3: Test Manually
```bash
$(dirname ${BACKUP_DIR})/scripts/pg_backup.sh
ls -lh ${BACKUP_DIR}/*.sql.gz
cat ${BACKUP_DIR}/backup.log
```
---
## Step 4: Schedule via Cron
```bash
(crontab -l 2>/dev/null; echo "${CRON_SCHEDULE} $(dirname ${BACKUP_DIR})/scripts/pg_backup.sh >> ${BACKUP_DIR}/cron.log 2>&1") | crontab -
crontab -l | grep pg_backup
```
---
## Step 5: Test Restore (Non-Destructive)
```bash
# Create throwaway test database
docker exec ${CONTAINER_NAME} psql -U ${DB_USER} -c "CREATE DATABASE ${DB_NAME}_restore_test;"
# Restore latest backup into it
LATEST=$(ls -t ${BACKUP_DIR}/${DB_NAME}_*.sql.gz | head -1)
gunzip -c "${LATEST}" | docker exec -i ${CONTAINER_NAME} psql -U ${DB_USER} -d ${DB_NAME}_restore_test
# Spot-check (adjust table names per service)
docker exec ${CONTAINER_NAME} psql -U ${DB_USER} -d ${DB_NAME}_restore_test -c "\dt" | head -20
# Cleanup
docker exec ${CONTAINER_NAME} psql -U ${DB_USER} -c "DROP DATABASE ${DB_NAME}_restore_test;"
```
---
## Emergency Restore
```bash
# 1. Stop the application container (not postgres)
docker stop <app-container>
# 2. Drop and recreate
docker exec ${CONTAINER_NAME} psql -U ${DB_USER} -c "DROP DATABASE ${DB_NAME};"
docker exec ${CONTAINER_NAME} psql -U ${DB_USER} -c "CREATE DATABASE ${DB_NAME} OWNER ${DB_USER};"
# 3. Restore
LATEST=$(ls -t ${BACKUP_DIR}/${DB_NAME}_*.sql.gz | head -1)
gunzip -c "${LATEST}" | docker exec -i ${CONTAINER_NAME} psql -U ${DB_USER} -d ${DB_NAME}
# 4. Restart app
docker start <app-container>
```
---
## Monitoring Hook (Optional)
Add to your monitoring stack:
```bash
LATEST_AGE=$(( $(date +%s) - $(stat -c %Y $(ls -t ${BACKUP_DIR}/${DB_NAME}_*.sql.gz | head -1)) ))
if [ ${LATEST_AGE} -gt 90000 ]; then
echo "WARNING: Latest ${DB_NAME} backup is more than 25 hours old"
fi
```
---
## Checklist
```
□ Manual backup produces valid .sql.gz
□ Backup log shows success
□ Gzip integrity check passes
□ Cron job installed
□ Test restore succeeds
□ Test database cleaned up
```

201
runbooks/pi-nas-omv-runbook.md Normal file
View file

@ -0,0 +1,201 @@
# Pi 5 NAS — OMV Provisioning Runbook
SSH into the Pi. The Pi should already be booted with Raspberry Pi OS Lite, Ethernet connected, Radxa Penta SATA Hat installed.
---
## 1. Update + Install OMV
```bash
sudo apt update
sudo apt upgrade -y
wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash
```
This takes ~5 minutes. Reboot when done:
```bash
sudo reboot
```
---
## 2. Enable PCIe Port
SSH back in after reboot. Drives won't show up until the PCIe port is enabled.
```bash
sudo tee -a /boot/firmware/config.txt > /dev/null << 'EOF'
# Radxa Penta SATA Hat — enable PCIe Gen 3
dtparam=pciex1
dtparam=pciex1_gen=3
EOF
sudo reboot
```
---
## 3. Verify Drives
SSH back in and confirm all four drives are visible:
```bash
lsblk
```
Expected output (one per drive bay):
```
sda
sdb
sdc
sdd
```
If any are missing, check SATA cable seating on the Radxa hat and verify the PCIe lines were added to config.txt.
---
## 4. Standard Baseline (zvx user, sshpass, Tailscale)
If the Pi wasn't flashed with the zvx user via Raspberry Pi Imager, create it now:
```bash
sudo useradd -m -s /bin/bash -G sudo zvx
echo "zvx:7redditGold" | sudo chpasswd
```
Install sshpass and Tailscale:
```bash
sudo apt install -y sshpass
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh
```
---
## 5. Configure OMV via Web UI
Open a browser and go to the Pi's IP address. Default login:
- **Username:** `admin`
- **Password:** `openmediavault`
Change the admin password immediately.
### Format Drives (ext4, no RAID)
Each drive is used individually — no RAID array.
1. **Storage → Disks** — confirm all 4 drives appear
2. **Storage → File Systems** — for each drive:
- Click **Create**
- Select the drive (sda, sdb, sdc, sdd)
- Type: **ext4**
- Label them something useful (e.g., `bay1`, `bay2`, `bay3`, `bay4`)
- Click **Save**, then **Mount** each one
3. **Apply** pending changes when prompted
### Create Shared Folders
1. **Storage → Shared Folders** — create a folder on each drive as needed, e.g.:
- `share1` on `bay1`
- `media` on `bay2`
- `proxmox-storage` on `bay3`
- `backup` on `bay4`
- (adjust names/layout to your needs)
2. Set permissions: **Administrator: read/write, Users: read/write, Others: read-only** (or as desired)
### Enable SMB (Windows Shares)
1. **Services → SMB/CIFS → Settings** — toggle **Enabled**, click **Save**
2. **Services → SMB/CIFS → Shares** — click **Create** for each shared folder you want accessible from Windows:
- Select the shared folder
- **Public:** No
- **Browseable:** Yes
- Click **Save**
3. **Apply** pending changes
### Enable NFS (Proxmox CT Storage)
1. **Services → NFS → Settings** — toggle **Enabled**, click **Save**
2. **Services → NFS → Shares** — click **Create** for each folder Proxmox needs:
- Select the shared folder (e.g., `proxmox-storage`)
- **Client:** your Proxmox subnet, e.g., `192.168.1.0/24`
- **Privilege:** Read/Write
- **Extra options:** `subtree_check,insecure,no_root_squash`
- Click **Save**
3. **Apply** pending changes
`no_root_squash` is needed because Proxmox CTs write as root. `insecure` allows connections from ports >1024 which some NFS clients use.
### Create OMV User (for SMB access)
1. **Users → Users** — click **Create**
- **Name:** `zvx`
- **Password:** `7redditGold`
- **Groups:** add to `users`
2. **Save** and **Apply**
This user is for SMB authentication. The Linux `zvx` user created earlier is separate from the OMV web UI user system.
---
## 6. Connect from Windows
From a Windows machine on the same network:
```
\\<NAS-IP>\share1
```
Or map as a network drive. Authenticate with `zvx` / `7redditGold`.
---
## 7. Connect from Proxmox
On the Proxmox host, add the NFS share as storage:
**Datacenter → Storage → Add → NFS:**
- **ID:** `nas-storage` (or whatever)
- **Server:** NAS IP address (or Tailscale IP)
- **Export:** `/export/proxmox-storage` (check exact path with `showmount -e <NAS-IP>` from Proxmox)
- **Content:** select what you'll store (Disk image, Container, ISO image, Snippets, Backups, etc.)
Or via CLI on the Proxmox host:
```bash
# Verify the NFS export is visible
showmount -e <NAS-IP>
# Add to Proxmox storage config
pvesm add nfs nas-storage \
--server <NAS-IP> \
--export /export/proxmox-storage \
--content images,rootdir,vztmpl,backup,iso \
--options vers=4
```
---
## 8. Verification Checklist
```bash
echo "=== Pi NAS Provisioning Check ==="
echo ""
echo "Hostname: $(hostname)"
echo "User zvx: $(id zvx 2>/dev/null && echo 'OK' || echo 'MISSING')"
echo "sshpass: $(which sshpass >/dev/null 2>&1 && echo 'OK' || echo 'MISSING')"
echo "Tailscale: $(tailscale status --self 2>/dev/null | head -1 || echo 'NOT CONNECTED')"
echo "Tailscale IP: $(tailscale ip -4 2>/dev/null || echo 'N/A')"
echo "OMV: $(systemctl is-active openmediavault-engined 2>/dev/null || echo 'NOT RUNNING')"
echo "PCIe: $(grep -q 'dtparam=pciex1' /boot/firmware/config.txt && echo 'ENABLED' || echo 'DISABLED')"
echo "Drives: $(lsblk -d -n -o NAME | grep '^sd' | wc -l) detected"
echo "SMB: $(systemctl is-active smbd 2>/dev/null || echo 'NOT RUNNING')"
echo "NFS: $(systemctl is-active nfs-server 2>/dev/null || echo 'NOT RUNNING')"
```

324
runbooks/pipeline-probe-gate.md Normal file
View file

@ -0,0 +1,324 @@
# Pre-Flight Probe Gate for Pipeline Efficiency
Insert a cheap inspection step before expensive processing in a pipeline. Probe the input (ffprobe, mediainfo, file headers, checksums) to skip work that will be wasted — wrong format, already optimized, below quality threshold, or too large to process safely. Log every decision for an audit trail. Keep a post-processing safety net as backup.
Use this when your pipeline processes files in bulk and a significant percentage of inputs don't need the expensive step, or when processing the wrong input would waste time, storage, or GPU cycles.
---
## Prerequisites
- A pipeline with at least one expensive processing step (transcoding, inference, embedding, etc.)
- A probe tool that can inspect inputs cheaply (< 1 second per file)
- Clear criteria for what constitutes a "skip" vs "process" decision
---
## Inputs
Prompt the user for all of these before executing:
```
PIPELINE_NAME= # Human-readable name (e.g., "video-transcoder", "pdf-extractor")
PROBE_TOOL= # Inspection tool (e.g., "ffprobe", "mediainfo", "file", "pdfinfo")
INPUT_DIR= # Where the pipeline reads inputs (e.g., "/opt/pipeline/incoming")
OUTPUT_DIR= # Where processed outputs go (e.g., "/opt/pipeline/processed")
SKIP_DIR= # Where skipped inputs go (e.g., "/opt/pipeline/skipped")
FAIL_DIR= # Where failed inputs go (e.g., "/opt/pipeline/failed")
LOG_FILE= # Decision log path (e.g., "/opt/pipeline/logs/probe-gate.log")
```
---
## Step 1: Define Skip Criteria
Enumerate the conditions under which a file should skip the expensive step. Be specific — vague criteria lead to false positives.
### Common probe checks
| Check | Probe Command | Skip When |
|-------|---------------|-----------|
| Video codec | `ffprobe -show_entries stream=codec_name` | Already target codec (e.g., already HEVC) |
| Audio bitrate | `ffprobe -show_entries stream=bit_rate` | Below minimum quality threshold |
| Resolution | `ffprobe -show_entries stream=width,height` | Below minimum (e.g., < 360p) |
| Duration | `ffprobe -show_entries format=duration` | Exceeds safe processing limit |
| File size | `stat -c%s` | Zero bytes, or exceeds storage budget |
| PDF pages | `pdfinfo file.pdf \| grep Pages` | Too many pages for OCR budget |
| Image format | `file --mime-type` | Already target format |
| Container format | `ffprobe -show_entries format=format_name` | Unsupported container |
| Corruption | `ffprobe -v error` exit code | Non-zero = corrupt file |
| Existing output | `test -f $OUTPUT_DIR/$(basename)` | Output already exists (dedup) |
### Gate
Write your criteria as a decision table:
```
Criterion 1: <property> <operator> <value> → SKIP (reason: "<why>")
Criterion 2: <property> <operator> <value> → SKIP (reason: "<why>")
Criterion 3: <property> not available → SKIP (reason: "probe failed")
Default: → PROCESS
```
---
## Step 2: Write the Probe Gate Function
```bash
#!/bin/bash
# Pre-flight probe gate for $PIPELINE_NAME
# Returns: 0 = process, 1 = skip, 2 = fail (corrupt/unreadable)
LOGFILE="$LOG_FILE"
probe_gate() {
local INPUT="$1"
local BASENAME=$(basename "$INPUT")
local TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
# ──── Existence check ────
if [[ ! -f "$INPUT" ]]; then
echo "[$TIMESTAMP] FAIL $BASENAME reason=file_not_found" >> "$LOGFILE"
return 2
fi
# ──── Size check ────
local SIZE=$(stat -c%s "$INPUT" 2>/dev/null)
if (( SIZE == 0 )); then
echo "[$TIMESTAMP] SKIP $BASENAME reason=zero_bytes size=0" >> "$LOGFILE"
return 1
fi
# ──── Probe the input ────
# Adapt this section to your probe tool and criteria
local PROBE_OUTPUT
PROBE_OUTPUT=$($PROBE_TOOL <probe-specific-flags> "$INPUT" 2>/dev/null)
local PROBE_EXIT=$?
if (( PROBE_EXIT != 0 )); then
echo "[$TIMESTAMP] FAIL $BASENAME reason=probe_failed exit=$PROBE_EXIT" >> "$LOGFILE"
return 2
fi
# ──── Apply skip criteria ────
# Example: check if already target codec
local CODEC=$(echo "$PROBE_OUTPUT" | grep codec_name | head -1 | cut -d= -f2)
if [[ "$CODEC" == "hevc" ]]; then
echo "[$TIMESTAMP] SKIP $BASENAME reason=already_hevc codec=$CODEC" >> "$LOGFILE"
return 1
fi
# Example: check if below minimum resolution
local HEIGHT=$(echo "$PROBE_OUTPUT" | grep '^height=' | head -1 | cut -d= -f2)
if (( HEIGHT < 240 )); then
echo "[$TIMESTAMP] SKIP $BASENAME reason=below_min_resolution height=$HEIGHT" >> "$LOGFILE"
return 1
fi
# ──── Passed all checks ────
echo "[$TIMESTAMP] PASS $BASENAME codec=$CODEC height=${HEIGHT} size=$SIZE" >> "$LOGFILE"
return 0
}
```
### Key design decisions
- **Return codes**: 0 = process (matches shell "success" convention), 1 = skip, 2 = fail. Callers use `$?` to branch.
- **Structured log lines**: Every decision logged with timestamp, verdict, filename, and reason. Parseable by grep/awk for reporting.
- **Probe errors = FAIL, not SKIP**: If the probe itself fails, the file might be corrupt — route to fail directory for manual inspection rather than silently skipping.
---
## Step 3: Integrate into the Pipeline
### Option A: Inline in processing loop
```bash
for INPUT in "$INPUT_DIR"/*; do
probe_gate "$INPUT"
case $? in
0) process_file "$INPUT" # Expensive step
mv "$INPUT" "$OUTPUT_DIR/"
;;
1) mv "$INPUT" "$SKIP_DIR/" # Skipped — preserve for audit
;;
2) mv "$INPUT" "$FAIL_DIR/" # Failed probe — needs investigation
;;
esac
done
```
### Option B: As a pre-filter in a wrapper script
If the expensive step is a binary called by a service (see `binary-wrapper-interception.md`), add the probe gate to the wrapper:
```bash
# In the wrapper script, before exec:
probe_gate "$INPUT_FILE"
GATE_RESULT=$?
if (( GATE_RESULT == 1 )); then
echo "[WRAPPER] $(date) SKIPPED: $INPUT_FILE" >> "$LOGFILE"
exit 0 # Success — nothing to do
fi
if (( GATE_RESULT == 2 )); then
echo "[WRAPPER] $(date) PROBE FAILED: $INPUT_FILE" >> "$LOGFILE"
exit 1 # Error — caller should retry or alert
fi
# Gate passed — proceed with expensive processing
exec $REAL_BINARY "$@"
```
### Option C: In a Python pipeline script
```python
import subprocess, shutil, os
def probe_gate(input_path: str) -> tuple[str, dict]:
"""Returns (verdict, metadata) where verdict is 'process', 'skip', or 'fail'."""
if not os.path.exists(input_path):
return 'fail', {'reason': 'file_not_found'}
size = os.path.getsize(input_path)
if size == 0:
return 'skip', {'reason': 'zero_bytes', 'size': 0}
result = subprocess.run(
['ffprobe', '-v', 'quiet', '-show_entries', 'stream=codec_name,height',
'-of', 'flat', input_path],
capture_output=True, text=True
)
if result.returncode != 0:
return 'fail', {'reason': 'probe_failed', 'exit': result.returncode}
# Parse and apply criteria...
return 'process', {'codec': codec, 'height': height, 'size': size}
```
---
## Step 4: Add Post-Processing Safety Net
The probe gate is the primary filter, but add a post-processing check as backup. This catches cases where the probe was wrong (e.g., file reported as H.264 but was actually corrupt).
```bash
post_process_check() {
local OUTPUT="$1"
local TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
# Size gate: output should be at least 10% of input size
local INPUT_SIZE=$2
local OUTPUT_SIZE=$(stat -c%s "$OUTPUT" 2>/dev/null)
if (( OUTPUT_SIZE < INPUT_SIZE / 10 )); then
echo "[$TIMESTAMP] POST-FAIL $OUTPUT reason=output_too_small input=${INPUT_SIZE} output=${OUTPUT_SIZE}" >> "$LOGFILE"
return 1
fi
# Integrity check: verify output is valid
$PROBE_TOOL -v error "$OUTPUT" 2>/dev/null
if (( $? != 0 )); then
echo "[$TIMESTAMP] POST-FAIL $OUTPUT reason=output_corrupt" >> "$LOGFILE"
return 1
fi
echo "[$TIMESTAMP] POST-PASS $OUTPUT size=$OUTPUT_SIZE" >> "$LOGFILE"
return 0
}
```
---
## Step 5: Reporting
Use the structured log to generate reports:
```bash
# Decision breakdown
echo "=== Probe Gate Report ==="
echo "Processed: $(grep -c ' PASS ' $LOG_FILE)"
echo "Skipped: $(grep -c ' SKIP ' $LOG_FILE)"
echo "Failed: $(grep -c ' FAIL ' $LOG_FILE)"
echo ""
# Top skip reasons
echo "Skip reasons:"
grep ' SKIP ' $LOG_FILE | grep -oP 'reason=\S+' | sort | uniq -c | sort -rn
# Failed files needing attention
echo ""
echo "Failed files:"
grep ' FAIL ' $LOG_FILE | tail -10
```
---
## Troubleshooting
### Probe is slow (> 1 second per file)
Some probe tools read more of the file than necessary. For ffprobe, use `-analyzeduration 1000000 -probesize 1000000` to limit how much of the file it reads. For large PDFs, `pdfinfo` is faster than opening the file in Python.
### Probe reports wrong codec/format
Some files have mismatched container and stream codecs. Probe the stream level, not the container:
```bash
ffprobe -v quiet -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 "$INPUT"
```
### Skipped files that should have been processed
Review the skip log. Lower the threshold or add exceptions for edge cases. The skip directory preserves files for re-processing if criteria change.
### Post-processing catches failures the probe missed
This is the safety net working as intended. Investigate why the probe didn't catch it — the input may have unusual characteristics. Add a new probe criterion if the pattern is common.
---
## Usage Examples
### PeerTube H.265 transcoding pipeline (cortex)
```
PIPELINE_NAME=video-transcoder
PROBE_TOOL=ffprobe
INPUT_DIR=/opt/bulk-import/completed
OUTPUT_DIR=/opt/bulk-import/transcoded
SKIP_DIR=/opt/bulk-import/skipped
FAIL_DIR=/opt/bulk-import/failed
Probe criteria:
- codec_name == "hevc" → SKIP (already H.265)
- height < 240 → SKIP (too low quality to bother)
- duration == 0 → FAIL (corrupt or audio-only)
- probe exit != 0 → FAIL (unreadable)
Post-processing safety net:
- output size < 10% of input → FAIL (transcode produced garbage)
- ffprobe on output fails → FAIL (corrupt output)
Result: Saved ~15% of GPU cycles by skipping already-optimized files.
```
### PDF extraction pipeline (RECON on CT 130)
```
PIPELINE_NAME=pdf-extractor
PROBE_TOOL=pdfinfo
INPUT_DIR=/mnt/library/incoming
OUTPUT_DIR=/opt/recon/extracted
Probe criteria:
- Pages > 500 → route to Gemini Vision (OCR too slow)
- Pages == 0 → FAIL (corrupt PDF)
- File size < 1KB → SKIP (empty/placeholder)
- Encrypted: yes → SKIP (can't extract without password)
- Already in SQLite status table → SKIP (dedup)
```
---
*Last updated: 2026-02-17*

12
runbooks/proxmox-create-ubuntu-vm.md
View file

@ -101,6 +101,16 @@ ssh root@$PVE_HOST "qm set $VMID \
--boot order=scsi0"
```
### Alternative: Use standard Echo6 cloud-init snippet
If the snippet is already on the node (deployed by `echo6-onboard-node.sh`), you can use it instead of the manual `--ciuser`/`--cipassword` config above. This pre-installs sshpass, curl, git, htop, and other standard packages via cloud-init:
```bash
ssh root@$PVE_HOST "qm set $VMID --cicustom \"user=local:snippets/echo6-base-userdata.yml\""
```
**Note:** You still need `--ipconfig0`, `--nameserver`, `--searchdomain`, `--sshkeys`, and `--boot` from the block above. The snippet only covers packages and `manage_etc_hosts`.
## Step 5 — GPU Passthrough (if enabled)
Skip if `GPU_PASSTHROUGH=no`.
@ -144,7 +154,7 @@ ssh root@$PVE_HOST "qm terminal $VMID"
```bash
ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y \
curl wget git htop iotop tmux vim \
sshpass curl wget git htop iotop tmux vim \
rsync tree jq unzip \
net-tools dnsutils \
python3 python3-pip python3-venv \

215
runbooks/proxmox-onboard-node.md Normal file
View file

@ -0,0 +1,215 @@
# Runbook: Onboard a Proxmox Node
You install Proxmox. You give CC an IP and a root password. CC does the rest.
---
## Current Cluster
| Alias | Local IP | Tailscale IP |
|----------|-----------------|-----------------|
| data | 192.168.1.240 | 100.64.0.20 |
| utility | 192.168.1.241 | 100.64.0.19 |
| cloud | 192.168.1.242 | 100.64.0.22 |
| media | 192.168.1.243 | 100.64.0.21 |
Management host: **cortex**
---
## Inputs
```
NODE_IP= # e.g. 192.168.1.244
NODE_ALIAS= # e.g. storage (lowercase, no dots)
ROOT_PASS= # root password for initial key copy
```
---
## Phase 1: SSH Access
Nothing works without this.
```bash
# Ensure sshpass is installed
which sshpass || sudo apt install -y sshpass
# Test access immediately
sshpass -p "$ROOT_PASS" ssh \
-o StrictHostKeyChecking=accept-new \
-o IdentitiesOnly=yes \
-o PreferredAuthentications=password \
root@$NODE_IP 'hostname'
```
### Gate
Must return the hostname. **Stop if this fails.**
### Add host alias
```bash
# Ensure ~/.ssh/config has global defaults (idempotent)
grep -q "IdentitiesOnly yes" ~/.ssh/config 2>/dev/null || cat >> ~/.ssh/config << 'EOF'
Host *
IdentitiesOnly yes
StrictHostKeyChecking accept-new
ConnectTimeout 10
ServerAliveInterval 30
ServerAliveCountMax 3
EOF
# Add alias (idempotent)
grep -q "Host $NODE_ALIAS$" ~/.ssh/config 2>/dev/null || cat >> ~/.ssh/config << EOF
Host $NODE_ALIAS
HostName $NODE_IP
User root
EOF
```
### Optional: Set up key auth
Eliminates the need for sshpass on every command to this node.
```bash
ls ~/.ssh/id_ed25519 || ssh-keygen -t ed25519 -C "cortex" -N "" -f ~/.ssh/id_ed25519
sshpass -p "$ROOT_PASS" ssh-copy-id \
-o StrictHostKeyChecking=accept-new \
-o IdentitiesOnly=yes \
-o PreferredAuthentications=password \
root@$NODE_IP
# Verify key auth works (no password)
ssh $NODE_ALIAS 'hostname'
```
### How CC connects for the rest of this runbook
If key auth is set up:
```bash
ssh $NODE_ALIAS '<command>'
```
If not:
```bash
sshpass -p "$ROOT_PASS" ssh $NODE_ALIAS '<command>'
```
---
## Phase 2: Base Configuration
```bash
ssh $NODE_ALIAS 'apt update && apt dist-upgrade -y'
ssh $NODE_ALIAS 'timedatectl set-timezone America/Boise'
ssh $NODE_ALIAS 'timedatectl status | grep -i sync'
# Disable enterprise repo
ssh $NODE_ALIAS 'sed -i "s/^deb/# deb/" /etc/apt/sources.list.d/pve-enterprise.list 2>/dev/null; true'
# Add no-subscription repo
ssh $NODE_ALIAS 'grep -q "pve-no-subscription" /etc/apt/sources.list.d/pve-no-subscription.list 2>/dev/null || \
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list'
```
---
## Phase 3: Tailscale
```bash
ssh $NODE_ALIAS 'curl -fsSL https://tailscale.com/install.sh | sh'
ssh $NODE_ALIAS 'tailscale up --login-server=https://<HEADSCALE_URL> --auth-key=<PREAUTH_KEY>'
# Get Tailscale IP and add alias
TSIP=$(ssh $NODE_ALIAS 'tailscale ip -4')
echo "Tailscale IP: $TSIP"
grep -q "Host ts-$NODE_ALIAS$" ~/.ssh/config 2>/dev/null || cat >> ~/.ssh/config << EOF
Host ts-$NODE_ALIAS
HostName $TSIP
User root
EOF
ssh ts-$NODE_ALIAS 'hostname'
```
---
## Phase 4: Verify Cluster Membership
You join the node to the cluster. CC verifies it's there.
```bash
ssh $NODE_ALIAS 'pvecm status 2>/dev/null | grep "Cluster Member"'
ssh data 'pvecm nodes'
```
If not in the cluster yet, **stop and tell the user**. Do not run `pvecm add`.
---
## Phase 5: Verify
```bash
# Authentik SSO (syncs via cluster)
ssh $NODE_ALIAS 'pveum realm list | grep authentik'
# Storage
ssh $NODE_ALIAS 'pvesm status'
ssh $NODE_ALIAS 'lsblk && echo "---" && vgs && lvs'
```
---
## Phase 6: Update Inventory
Add to CLAUDE.md cluster table:
```
| <NODE_ALIAS> | <NODE_IP> | <TSIP> |
```
Update any hardcoded node lists:
- proxmox-audit.sh (NODES array)
- Monitoring/backup targets
---
## Final Verification
Every line must say OK.
```bash
echo "=== $NODE_ALIAS ==="
echo -n "SSH (local): "; ssh $NODE_ALIAS 'echo OK' 2>&1
echo -n "SSH (tailscale): "; ssh ts-$NODE_ALIAS 'echo OK' 2>&1
echo -n "Cluster: "; ssh $NODE_ALIAS 'pvecm status 2>/dev/null | grep -q "Cluster Member: Yes" && echo OK || echo FAIL'
echo -n "Tailscale: "; ssh $NODE_ALIAS 'tailscale status --self >/dev/null 2>&1 && echo OK || echo FAIL'
echo -n "OIDC realm: "; ssh $NODE_ALIAS 'pveum realm list 2>/dev/null | grep -q authentik && echo OK || echo FAIL'
echo -n "Storage: "; ssh $NODE_ALIAS 'pvesm status >/dev/null 2>&1 && echo OK || echo FAIL'
echo -n "PVE version: "; ssh $NODE_ALIAS 'pveversion'
echo -n "Time sync: "; ssh $NODE_ALIAS 'timedatectl show -p NTPSynchronized --value'
```
---
## Troubleshooting
**"Too many authentication failures"**
`IdentitiesOnly yes` missing from `Host *` in `~/.ssh/config`.
**sshpass "Permission denied"**
Add `-o PreferredAuthentications=password -o IdentitiesOnly=yes`.
**Cluster join corosync errors**
Check `/etc/hosts` on all nodes includes the new hostname and IP.
**Authentik realm missing**
Check `systemctl status pve-cluster`. Realm syncs via pmxcfs in `/etc/pve/domains.cfg`.
**Can't migrate VMs to node**
Storage mismatch. Compare `pvesm status` on both nodes.

189
runbooks/recon-operations.md Normal file
View file

@ -0,0 +1,189 @@
# RECON Operations Runbook
## Service Info
- **Host:** recon LXC (CT 130 on data node)
- **IP:** 192.168.1.130 / 100.64.0.24
- **Install:** /opt/recon/
- **User:** zvx
- **Service:** `recon.service` (systemd)
## Service Management
```bash
ssh zvx@100.64.0.24
sudo systemctl start|stop|restart|status recon
journalctl -u recon -f
```
## Health Check
```bash
curl -s http://100.64.0.24:8420/api/health | python3 -m json.tool
# Returns: healthy (200), degraded/unhealthy (503)
# Checks: Qdrant, TEI, NFS, Gemini keys, pipeline counts
```
## Pipeline Status
```bash
ssh zvx@100.64.0.24
cd /opt/recon && source venv/bin/activate
python3 recon.py status # Summary counts
python3 recon.py failures # Failed documents
python3 recon.py search "query" # Test search
```
## Dashboard
- **URL:** http://100.64.0.24:8420
- Shows: pipeline progress, per-source breakdown, Qdrant stats
- Auto-refreshes every 30s
## Common Operations
```bash
cd /opt/recon && source venv/bin/activate
# Add a PDF
python3 recon.py upload --file /path/to.pdf --category "Reference"
# Add web content
python3 recon.py ingest-url "https://example.com/article" --process
# Crawl a website
python3 recon.py crawl "https://docs.example.com" --process
# Manual pipeline run (normally automatic via service)
python3 recon.py extract
python3 recon.py enrich
python3 recon.py embed
# Scan library for new PDFs (normally hourly via service)
python3 recon.py scan
python3 recon.py queue
```
## Dependencies
| Service | Host | Port | Purpose |
|---------|------|------|---------|
| Qdrant | cortex | 6333 | Vector DB (recon_knowledge collection) |
| TEI | cortex | 8090 | Text embeddings (bge-m3, 1024-dim) |
| Ollama | cortex | 11434 | Chat model for Aurora RAG |
| NFS | pi-nas | — | /mnt/library (PDF source) |
| Gemini API | Google | — | Enrichment + vision OCR (4 keys in .env) |
| Contabo VPS | 100.64.0.1 | — | Backup destination |
## Backups
- **Destination:** `root@100.64.0.1:/opt/backups/recon/`
- **Full sync (concepts, text, DB, config):** every 6 hours via cron
- **DB snapshot only:** every 2 hours via cron
- **Script:** `/opt/recon/scripts/backup.sh`
### Verify backups
```bash
ssh root@100.64.0.1 'ls -lh /opt/backups/recon/recon_*.db && du -sh /opt/backups/recon/'
```
## Troubleshooting
### Pipeline stalled (no progress)
```bash
journalctl -u recon -n 50 # Check errors
curl -s http://100.64.0.24:8420/api/health # Check dependencies
sudo systemctl restart recon # Restart
```
### Gemini rate limits (429 errors)
Built-in: exponential backoff 5s→10s→20s→40s→80s with jitter. Window failures skip that window and continue — partial enrichment beats zero.
If sustained: reduce `enrich_workers` in config.yaml, restart.
### Qdrant down
```bash
ssh zvx@cortex
docker ps | grep qdrant
docker restart qdrant
# If data lost: ssh zvx@100.64.0.24 'cd /opt/recon && source venv/bin/activate && python3 recon.py rebuild'
```
### TEI down
```bash
ssh zvx@cortex
docker ps | grep tei
docker restart tei
```
### NFS mount lost
```bash
ssh zvx@100.64.0.24
mount | grep library
sudo mount -a
sudo systemctl restart recon
```
### Reset stuck documents
```bash
cd /opt/recon && source venv/bin/activate
# Find stuck transitional states
sqlite3 data/recon.db "SELECT status, COUNT(*) FROM documents WHERE status IN ('extracting','enriching','embedding') GROUP BY status;"
# Reset them
sqlite3 data/recon.db "UPDATE documents SET status='queued' WHERE status='extracting';"
sqlite3 data/recon.db "UPDATE documents SET status='extracted' WHERE status='enriching';"
sqlite3 data/recon.db "UPDATE documents SET status='enriched' WHERE status='embedding';"
```
### Full recovery from Contabo backup
```bash
ssh zvx@100.64.0.24
sudo systemctl stop recon
rsync -av root@100.64.0.1:/opt/backups/recon/concepts/ /opt/recon/data/concepts/
rsync -av root@100.64.0.1:/opt/backups/recon/text/ /opt/recon/data/text/
# Pick the latest DB backup
rsync -av root@100.64.0.1:/opt/backups/recon/recon_latest.db /opt/recon/data/recon.db
cd /opt/recon && source venv/bin/activate
python3 recon.py rebuild # Rebuilds Qdrant from concept JSONs
sudo systemctl start recon
```
## Key Files
| Path | Purpose |
|------|---------|
| `/opt/recon/config.yaml` | All configuration |
| `/opt/recon/.env` | Gemini API keys (GEMINI_KEY_1 through GEMINI_KEY_4) |
| `/opt/recon/data/recon.db` | SQLite status DB |
| `/opt/recon/data/concepts/` | Gemini extraction results (CRITICAL — costs $ to regenerate) |
| `/opt/recon/data/text/` | Extracted page text (regenerable from PDFs) |
| `/opt/recon/PROJECT-BIBLE.md` | Full system documentation |
| `/opt/recon/scripts/backup.sh` | Backup script |
| `/opt/recon/scripts/validate.py` | Pipeline consistency checker |
| `/opt/recon/scripts/rebuild_qdrant.py` | Nuclear Qdrant rebuild |
## Pipeline Architecture
```
/mnt/library/ (NFS)
│
▼ hourly scan
[Catalogue] → [Queue] → [Extract] → [Enrich] → [Embed] → [Complete]
4 workers 16 workers 4 workers
PyPDF2 Gemini TEI+Qdrant
pdftotext 2.0 Flash bge-m3
Tesseract 1024-dim
Gemini Vision
```
---
*Last updated: 2026-02-16 — Initial creation*

464
runbooks/recon-service-integration.md Normal file
View file

@ -0,0 +1,464 @@
# RECON Dashboard Service Integration
Add a management UI for a remote service to a Flask/FastAPI dashboard. The pattern: SSH key trust between the dashboard host and the target, scoped sudoers for specific commands, a REST API layer (`GET /api/{service}/status` + `POST /api/{service}/{action}`), and a frontend panel with status indicator, action buttons, and live feedback.
Use this when you have a service running on a remote LXC/VM that needs a web management interface — start/stop/restart, status checks, log tailing, or config hot-reload — without SSH-ing into the box manually.
---
## Prerequisites
- A running Flask or FastAPI dashboard (e.g., RECON on CT 130, WATCHTOWER on Contabo)
- The target service running on a reachable host (LXC, VM, or bare metal)
- SSH access from the dashboard host to the target host
- The dashboard runs as a known user (e.g., `zvx`, `recon`, `watchtower`)
---
## Inputs
Prompt the user for all of these before executing:
```
DASHBOARD_HOST= # Host running the dashboard (e.g., "192.168.1.130", "CT 130")
DASHBOARD_USER= # User the dashboard runs as (e.g., "zvx")
DASHBOARD_APP_PATH= # Path to the dashboard app (e.g., "/opt/recon/lib/api.py")
DASHBOARD_STATIC_PATH= # Path to frontend files (e.g., "/opt/recon/lib/static/")
TARGET_HOST= # Host running the service to manage (e.g., "192.168.1.170")
TARGET_USER= # User to SSH as on the target (e.g., "zvx")
SERVICE_NAME= # systemd service name (e.g., "peertube", "pt-downloader")
SERVICE_DISPLAY_NAME= # Human-readable name for the UI (e.g., "PeerTube", "Downloader")
SERVICE_SLUG= # URL-safe slug (e.g., "peertube", "downloader")
ALLOWED_ACTIONS= # Comma-separated actions (e.g., "start,stop,restart,status,logs")
```
---
## Step 1: Set Up SSH Key Trust
The dashboard host must be able to SSH to the target without a password prompt.
### Generate key (if not already present)
```bash
ssh $DASHBOARD_HOST "test -f /home/$DASHBOARD_USER/.ssh/id_ed25519 || \
ssh-keygen -t ed25519 -N '' -f /home/$DASHBOARD_USER/.ssh/id_ed25519"
```
### Copy public key to target
```bash
# Get the public key
PUBKEY=$(ssh $DASHBOARD_HOST "cat /home/$DASHBOARD_USER/.ssh/id_ed25519.pub")
# Add to target's authorized_keys
ssh $TARGET_HOST "mkdir -p /home/$TARGET_USER/.ssh && \
echo '$PUBKEY' >> /home/$TARGET_USER/.ssh/authorized_keys && \
chmod 700 /home/$TARGET_USER/.ssh && \
chmod 600 /home/$TARGET_USER/.ssh/authorized_keys && \
chown -R $TARGET_USER:$TARGET_USER /home/$TARGET_USER/.ssh"
```
### Gate
```bash
ssh $DASHBOARD_HOST "ssh -o BatchMode=yes -o ConnectTimeout=5 $TARGET_USER@$TARGET_HOST 'hostname'"
```
Must return the target hostname without prompting for a password. If it fails:
- "Permission denied (publickey)" → key not in authorized_keys, or wrong user
- "Host key verification failed" → add `-o StrictHostKeyChecking=accept-new` for first connection
- Timeout → network issue, firewall, or wrong IP
---
## Step 2: Configure Scoped Sudoers on Target
Grant the target user passwordless sudo for **only** the specific commands the dashboard needs. Never use `NOPASSWD: ALL` for service integrations.
```bash
ssh root@$TARGET_HOST "cat > /etc/sudoers.d/${SERVICE_SLUG}-mgmt << 'SUDOERS'
# Allow $TARGET_USER to manage $SERVICE_NAME via dashboard
$TARGET_USER ALL=(ALL) NOPASSWD: /usr/bin/systemctl start $SERVICE_NAME
$TARGET_USER ALL=(ALL) NOPASSWD: /usr/bin/systemctl stop $SERVICE_NAME
$TARGET_USER ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart $SERVICE_NAME
$TARGET_USER ALL=(ALL) NOPASSWD: /usr/bin/systemctl status $SERVICE_NAME
$TARGET_USER ALL=(ALL) NOPASSWD: /usr/bin/journalctl -u $SERVICE_NAME *
SUDOERS
chmod 440 /etc/sudoers.d/${SERVICE_SLUG}-mgmt"
```
### Gate
```bash
ssh $DASHBOARD_HOST "ssh $TARGET_USER@$TARGET_HOST 'sudo systemctl status $SERVICE_NAME'"
```
Must return service status without a password prompt. If "sudo: a password is required", the sudoers file has a syntax error or isn't being loaded — check `visudo -cf /etc/sudoers.d/${SERVICE_SLUG}-mgmt`.
---
## Step 3: Add API Endpoints
Add REST endpoints to the dashboard app for status checks and actions.
### Flask pattern
```python
import subprocess
import shlex
SERVICE_INTEGRATIONS = {
'$SERVICE_SLUG': {
'display_name': '$SERVICE_DISPLAY_NAME',
'target_host': '$TARGET_USER@$TARGET_HOST',
'service_name': '$SERVICE_NAME',
'allowed_actions': ['start', 'stop', 'restart', 'status', 'logs'],
},
}
def ssh_cmd(host: str, cmd: str, timeout: int = 10) -> dict:
"""Execute a command on a remote host via SSH."""
full_cmd = f"ssh -o BatchMode=yes -o ConnectTimeout=5 {host} {shlex.quote(cmd)}"
try:
result = subprocess.run(
full_cmd, shell=True, capture_output=True, text=True, timeout=timeout
)
return {
'success': result.returncode == 0,
'stdout': result.stdout.strip(),
'stderr': result.stderr.strip(),
'exit_code': result.returncode,
}
except subprocess.TimeoutExpired:
return {'success': False, 'stdout': '', 'stderr': 'SSH command timed out', 'exit_code': -1}
@app.route('/api/services/<slug>/status')
def api_service_status(slug):
svc = SERVICE_INTEGRATIONS.get(slug)
if not svc:
return jsonify({'error': 'Unknown service'}), 404
result = ssh_cmd(svc['target_host'], f"sudo systemctl status {svc['service_name']}")
# Parse systemctl status output
active = 'active (running)' in result.get('stdout', '')
return jsonify({
'service': svc['display_name'],
'active': active,
'raw': result['stdout'],
})
@app.route('/api/services/<slug>/<action>', methods=['POST'])
def api_service_action(slug, action):
svc = SERVICE_INTEGRATIONS.get(slug)
if not svc:
return jsonify({'error': 'Unknown service'}), 404
if action not in svc['allowed_actions']:
return jsonify({'error': f'Action {action} not allowed'}), 403
if action == 'logs':
result = ssh_cmd(
svc['target_host'],
f"sudo journalctl -u {svc['service_name']} -n 50 --no-pager",
timeout=15,
)
elif action in ('start', 'stop', 'restart'):
result = ssh_cmd(svc['target_host'], f"sudo systemctl {action} {svc['service_name']}")
elif action == 'status':
result = ssh_cmd(svc['target_host'], f"sudo systemctl status {svc['service_name']}")
else:
return jsonify({'error': 'Unknown action'}), 400
return jsonify(result)
```
### FastAPI pattern
Same logic, different decorators:
```python
@app.get('/api/services/{slug}/status')
async def api_service_status(slug: str):
# Same implementation, wrapped in run_in_executor for async
@app.post('/api/services/{slug}/{action}')
async def api_service_action(slug: str, action: str):
# Same implementation
```
### Gate
Restart the dashboard and test:
```bash
curl -s http://$DASHBOARD_HOST:8420/api/services/$SERVICE_SLUG/status | python3 -m json.tool
```
Must return JSON with `active: true/false` and service details.
```bash
curl -s -X POST http://$DASHBOARD_HOST:8420/api/services/$SERVICE_SLUG/restart | python3 -m json.tool
```
Must return `success: true`.
---
## Step 4: Add Frontend Panel
Add a service management panel to the dashboard UI. This goes in the appropriate tab (e.g., Upload, Dashboard, or a new Services tab).
```html
<!-- Service Management Panel: $SERVICE_DISPLAY_NAME -->
<div class="service-panel" id="panel-$SERVICE_SLUG">
<h3>$SERVICE_DISPLAY_NAME</h3>
<!-- Status indicator -->
<div class="status-row">
<span class="status-dot" id="status-$SERVICE_SLUG"></span>
<span id="status-text-$SERVICE_SLUG">Checking...</span>
<button onclick="refreshStatus('$SERVICE_SLUG')" class="btn-sm">Refresh</button>
</div>
<!-- Action buttons -->
<div class="action-buttons">
<button onclick="serviceAction('$SERVICE_SLUG', 'restart')" class="btn btn-warning">Restart</button>
<button onclick="serviceAction('$SERVICE_SLUG', 'stop')" class="btn btn-danger">Stop</button>
<button onclick="serviceAction('$SERVICE_SLUG', 'start')" class="btn btn-success">Start</button>
<button onclick="serviceAction('$SERVICE_SLUG', 'logs')" class="btn btn-info">View Logs</button>
</div>
<!-- Feedback area -->
<pre id="feedback-$SERVICE_SLUG" class="feedback-box" style="display:none;"></pre>
</div>
```
```javascript
// Service management JS
async function refreshStatus(slug) {
const dot = document.getElementById(`status-${slug}`);
const text = document.getElementById(`status-text-${slug}`);
try {
const resp = await fetch(`/api/services/${slug}/status`);
const data = await resp.json();
dot.className = data.active ? 'status-dot active' : 'status-dot inactive';
text.textContent = data.active ? 'Running' : 'Stopped';
} catch (e) {
dot.className = 'status-dot error';
text.textContent = 'Unreachable';
}
}
async function serviceAction(slug, action) {
const feedback = document.getElementById(`feedback-${slug}`);
feedback.style.display = 'block';
feedback.textContent = `Executing ${action}...`;
try {
const resp = await fetch(`/api/services/${slug}/${action}`, { method: 'POST' });
const data = await resp.json();
feedback.textContent = data.stdout || data.stderr || (data.success ? 'Done' : 'Failed');
// Refresh status after action
if (['start', 'stop', 'restart'].includes(action)) {
setTimeout(() => refreshStatus(slug), 2000);
}
} catch (e) {
feedback.textContent = `Error: ${e.message}`;
}
}
// Auto-refresh status every 30 seconds
setInterval(() => {
document.querySelectorAll('.service-panel').forEach(panel => {
const slug = panel.id.replace('panel-', '');
refreshStatus(slug);
});
}, 30000);
// Initial load
document.addEventListener('DOMContentLoaded', () => {
document.querySelectorAll('.service-panel').forEach(panel => {
const slug = panel.id.replace('panel-', '');
refreshStatus(slug);
});
});
```
```css
.status-dot {
display: inline-block;
width: 12px;
height: 12px;
border-radius: 50%;
margin-right: 8px;
}
.status-dot.active { background: #22c55e; }
.status-dot.inactive { background: #ef4444; }
.status-dot.error { background: #f59e0b; }
.feedback-box {
background: #1e1e2e;
color: #cdd6f4;
padding: 12px;
border-radius: 4px;
max-height: 300px;
overflow-y: auto;
font-size: 12px;
margin-top: 8px;
}
```
---
## Step 5: Verify End-to-End
1. **Load the dashboard** in a browser: `http://$DASHBOARD_HOST:8420/`
2. **Check status indicator**: should show green dot + "Running" (or red + "Stopped")
3. **Click Restart**: feedback box should show systemctl output, status should flip briefly then return to Running
4. **Click View Logs**: should show last 50 journal lines
5. **Click Stop**: status should change to Stopped (red)
6. **Click Start**: status should change to Running (green)
---
## Adding More Services
To integrate a second service, repeat Steps 1-4 with new inputs. The `SERVICE_INTEGRATIONS` dict supports multiple entries:
```python
SERVICE_INTEGRATIONS = {
'peertube': { ... },
'downloader': {
'display_name': 'Bulk Downloader',
'target_host': 'zvx@192.168.1.170',
'service_name': 'pt-downloader',
'allowed_actions': ['start', 'stop', 'restart', 'status', 'logs'],
},
'transcoder': {
'display_name': 'H.265 Transcoder',
'target_host': 'zvx@192.168.1.150',
'service_name': 'pt-transcoder',
'allowed_actions': ['start', 'stop', 'restart', 'status', 'logs'],
},
}
```
Each service gets its own panel in the UI, its own sudoers file on the target, and its own API routes (all handled by the generic `/<slug>/<action>` pattern).
---
## Security Considerations
- **Scoped sudoers**: Only allow the specific `systemctl` and `journalctl` commands needed. Never `NOPASSWD: ALL`.
- **SSH BatchMode**: `BatchMode=yes` ensures SSH never falls back to interactive password prompt. If key auth fails, the command fails immediately.
- **Action allowlist**: The `allowed_actions` list prevents the API from executing arbitrary commands. Only listed actions are accepted.
- **No shell injection**: Use `shlex.quote()` on any user-provided or variable input before passing to `subprocess.run(shell=True)`. Or use `subprocess.run(cmd_list)` with a list to avoid shell entirely.
- **Timeout on SSH**: Always set `-o ConnectTimeout` and `subprocess.run(timeout=)` to prevent the dashboard from hanging on network issues.
---
## Troubleshooting
### Status shows "Unreachable"
SSH from the dashboard host to the target is failing. Test manually:
```bash
ssh -o BatchMode=yes -o ConnectTimeout=5 $TARGET_USER@$TARGET_HOST 'hostname'
```
Common causes: SSH key not deployed, wrong user, firewall, target host down.
### "sudo: a password is required"
The sudoers file isn't working. Check:
```bash
ssh root@$TARGET_HOST "visudo -cf /etc/sudoers.d/${SERVICE_SLUG}-mgmt"
```
Must say "parsed OK". Also verify the username in the sudoers file matches `$TARGET_USER`.
### Actions work via curl but not from the browser
CORS issue. Add CORS headers to the API:
```python
# Flask
from flask_cors import CORS
CORS(app)
# Or manually:
@app.after_request
def add_cors(response):
response.headers['Access-Control-Allow-Origin'] = '*'
response.headers['Access-Control-Allow-Methods'] = 'GET, POST'
return response
```
### Dashboard hangs when target host is down
The SSH timeout isn't working, or it's set too high. Ensure both `-o ConnectTimeout=5` (SSH) and `timeout=10` (subprocess) are set. The subprocess timeout is the hard limit.
### Log output is truncated
The `-n 50` flag limits journalctl output. Increase it, or add a `lines` query parameter:
```python
lines = request.args.get('lines', 50, type=int)
lines = min(lines, 500) # Cap to prevent abuse
```
---
## Usage Examples
### RECON managing pipeline services (CT 130 dashboard → CT 110 PeerTube)
```
DASHBOARD_HOST=192.168.1.130 (CT 130, data node)
DASHBOARD_USER=zvx
TARGET_HOST=192.168.1.170 (CT 110, media node)
SERVICE_NAME=peertube
SERVICE_SLUG=peertube
Sudoers on CT 110:
zvx ALL=(ALL) NOPASSWD: /usr/bin/systemctl start peertube
zvx ALL=(ALL) NOPASSWD: /usr/bin/systemctl stop peertube
zvx ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart peertube
zvx ALL=(ALL) NOPASSWD: /usr/bin/systemctl status peertube
zvx ALL=(ALL) NOPASSWD: /usr/bin/journalctl -u peertube *
API endpoints:
GET /api/services/peertube/status → returns active/inactive + raw systemctl output
POST /api/services/peertube/restart → restarts PeerTube, returns success/failure
POST /api/services/peertube/logs → returns last 50 journal lines
Dashboard panel: green/red dot + Restart/Stop/Start/Logs buttons + feedback box
```
### WATCHTOWER monitoring remote services (Contabo → multiple hosts)
```
DASHBOARD_HOST=5.189.158.149 (Contabo)
DASHBOARD_USER=root
Services managed:
- peertube (CT 110): start/stop/restart/status/logs
- pt-downloader (CT 110): start/stop/restart/status/logs
- pt-importer (CT 110): start/stop/restart/status/logs
- pt-transcoder (cortex): start/stop/restart/status/logs
- recon (CT 130): start/stop/restart/status/logs
Each service has its own sudoers file on its target host,
its own entry in SERVICE_INTEGRATIONS, and its own UI panel.
```
---
*Last updated: 2026-02-17*

272
runbooks/syncthing-add-node.md Normal file
View file

@ -0,0 +1,272 @@
# Syncthing: Add a New Node to the Project Sync Cluster
## Overview
Adds a new machine to the Syncthing `projects` folder mesh. All nodes sync bidirectionally — new files merge, nothing is overwritten or deleted.
**Current cluster:**
| Node | Device ID (short) | Path | OS |
|------|--------------------|------|----|
| cortex | `6VP7KIB` | `/home/zvx/projects` | Ubuntu 24.04 |
| contabo | `SBYGD4P` | `/home/zvx/projects` | Ubuntu 24.04 |
| bluefin | `5ZTWIXM` | `/var/home/malice/projects` | Fedora Atomic |
| matt-desktop | `GCH6AAG` | `E:\Documents\projects` | Windows |
**Syncthing version:** v2.0.15 (all nodes must run v2.x — v1.x is incompatible)
**Config API:** All configuration changes are done via the REST API at `http://127.0.0.1:8384/rest/config` using the API key from each node's config XML.
---
## Prerequisites
- New node has network access to at least one existing node (Tailscale preferred)
- SSH access to the new node and at least one existing node
---
## Step 1: Install Syncthing v2
### Linux (apt-based)
Download the binary directly — the apt repo may only have v1.x:
```bash
curl -fsSL https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-linux-amd64-v2.0.15.tar.gz -o /tmp/syncthing.tar.gz
tar -xzf /tmp/syncthing.tar.gz -C /tmp
sudo cp /tmp/syncthing-linux-amd64-v2.0.15/syncthing /usr/bin/syncthing
syncthing --version # verify v2.x
```
### Linux (Homebrew — Fedora Atomic/Bluefin)
```bash
brew install syncthing
# Creates ~/.local/state/syncthing/ for config
```
### Windows
```powershell
New-Item -ItemType Directory -Force -Path "$HOME\syncthing"
Invoke-WebRequest -Uri "https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-windows-amd64-v2.0.15.zip" -OutFile "$HOME\syncthing\st.zip"
Expand-Archive -Path "$HOME\syncthing\st.zip" -DestinationPath "$HOME\syncthing" -Force
Copy-Item "$HOME\syncthing\syncthing-windows-amd64-v2.0.15\syncthing.exe" "$HOME\syncthing\syncthing.exe" -Force
Remove-Item -Recurse -Force "$HOME\syncthing\syncthing-windows-amd64-v2.0.15"
Remove-Item "$HOME\syncthing\st.zip"
```
---
## Step 2: Generate Config and Get Device ID
```bash
syncthing generate
# Output includes: device=XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
```
Save the full device ID — you'll need it for all other nodes.
---
## Step 3: Set Up Auto-Start
### Linux (systemd service — existing unit)
If `syncthing@<user>.service` exists (apt installs it):
```bash
sudo systemctl enable --now syncthing@zvx
```
### Linux (systemd user service — manual)
Create `~/.config/systemd/user/syncthing.service`:
```ini
[Unit]
Description=Syncthing - Open Source Continuous File Synchronization
After=network.target
[Service]
ExecStart=/path/to/syncthing serve --no-browser --no-restart --logflags=0
Restart=on-failure
RestartSec=10
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
[Install]
WantedBy=default.target
```
```bash
systemctl --user daemon-reload
systemctl --user enable --now syncthing
```
### Windows (Scheduled Task)
```powershell
schtasks /create /tn Syncthing /tr "C:\Users\administrator\syncthing\syncthing.exe serve --no-browser --no-restart" /sc onlogon /rl highest /f
```
Then start it for the current session:
```powershell
Start-Process -FilePath "$HOME\syncthing\syncthing.exe" -ArgumentList "serve","--no-browser","--no-restart" -WindowStyle Hidden
```
### Windows Firewall
Required — syncthing won't accept connections without this:
```powershell
netsh advfirewall firewall add rule name="Syncthing" dir=in action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
netsh advfirewall firewall add rule name="Syncthing-Out" dir=out action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
```
---
## Step 4: Configure the New Node via REST API
Wait for syncthing to start (~5 seconds), then get the API key:
```bash
# Linux
grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//'
# Windows (PowerShell)
([xml](Get-Content "$env:LOCALAPPDATA\Syncthing\config.xml")).configuration.gui.apikey
```
Use the API to add devices and the projects folder. This Python snippet does it all — run it on the **new node**:
```python
import json, urllib.request
API_KEY = "<apikey from above>"
MY_DEVICE_ID = "<new node device ID>"
PROJECTS_PATH = "<local path to projects folder>" # e.g. /home/zvx/projects
# All cluster nodes — add the new node's ID to this list when updating existing nodes
DEVICES = {
"cortex": {"id": "6VP7KIB-ZHBI3AT-XO5FMY2-LFAZYM6-UMAV75U-MZZADW3-ZOBHJXY-GF26DAC", "addr": "tcp://100.64.0.14:22000"},
"contabo": {"id": "SBYGD4P-BUWMWRQ-JJYYG75-YBR4WOO-OH42WH4-IAAO33D-STJZX6O-SZA2SQ4", "addr": "tcp://100.64.0.1:22000"},
"bluefin": {"id": "5ZTWIXM-XNBUEW5-XWJM7PG-FJDMX5H-YMXM3CC-ZVS2PNO-NG2E3KJ-D5HXKQB", "addr": "dynamic"},
"matt-desktop": {"id": "GCH6AAG-IWPH6TR-7GI7THZ-DIVXRRQ-EQMRBNN-IZG7Y2F-HM6BRLX-AC3MIQ6", "addr": "dynamic"},
}
def api(method, path, data=None):
url = f"http://127.0.0.1:8384{path}"
body = json.dumps(data).encode() if data else None
req = urllib.request.Request(url, data=body, method=method,
headers={"X-API-Key": API_KEY, "Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req).read())
cfg = api("GET", "/rest/config")
existing_ids = [d["deviceID"] for d in cfg["devices"]]
# Add all peer devices
for name, dev in DEVICES.items():
if dev["id"] not in existing_ids and dev["id"] != MY_DEVICE_ID:
cfg["devices"].append({
"deviceID": dev["id"], "name": name,
"addresses": [dev["addr"]], "compression": "metadata",
"paused": False, "autoAcceptFolders": False
})
# Add projects folder if missing
folder_ids = [f["id"] for f in cfg["folders"]]
if "projects" not in folder_ids:
all_device_ids = [d["id"] for d in DEVICES.values()] + [MY_DEVICE_ID]
cfg["folders"].append({
"id": "projects", "label": "projects",
"path": PROJECTS_PATH, "type": "sendreceive",
"rescanIntervalS": 60, "fsWatcherEnabled": True, "fsWatcherDelayS": 10,
"devices": [{"deviceID": did} for did in set(all_device_ids)]
})
api("PUT", "/rest/config", cfg)
print("New node configured")
```
---
## Step 5: Add the New Node to ALL Existing Nodes
For **each** existing node, run the following (substituting the new node's device ID and name):
```bash
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
NEW_ID="<new node device ID>"
NEW_NAME="<new node name>"
CONFIG=$(curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/config)
CONFIG=$(echo "$CONFIG" | python3 -c "
import json,sys
c = json.load(sys.stdin)
did = '$NEW_ID'
ids = [d['deviceID'] for d in c['devices']]
if did not in ids:
c['devices'].append({'deviceID': did, 'name': '$NEW_NAME', 'addresses': ['dynamic'], 'compression': 'metadata', 'paused': False, 'autoAcceptFolders': False})
for f in c['folders']:
if f['id'] == 'projects':
fids = [d['deviceID'] for d in f['devices']]
if did not in fids:
f['devices'].append({'deviceID': did})
json.dump(c, sys.stdout)
")
curl -s -X PUT -H "X-API-Key: $APIKEY" -H 'Content-Type: application/json' -d "$CONFIG" http://127.0.0.1:8384/rest/config
```
> **Important:** The CLI (`syncthing cli config devices add` / `syncthing cli config folders <id> devices add`) panics on v2.0.15 with a `reflect.Value.Elem on slice Value` bug. Always use the REST API instead.
---
## Step 6: Verify
Check connections from the new node:
```bash
syncthing cli show connections
```
Check sync status:
```bash
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/db/status?folder=projects | python3 -m json.tool
```
Key fields: `state` should be `syncing` then `idle`, `needFiles` should reach `0`.
---
## Troubleshooting
| Problem | Cause | Fix |
|---------|-------|-----|
| Connections establish then drop with "reading length: EOF" | Version mismatch (v1 vs v2) | Upgrade all nodes to v2.x |
| Node shows as device but never connects | Firewall blocking port 22000 | Open inbound/outbound for syncthing binary (Windows) or port 22000 (Linux) |
| `syncthing cli config ... add` panics | Known bug in v2.0.15 CLI | Use REST API at `http://127.0.0.1:8384/rest/config` instead |
| Windows: syncthing not listening after start | Process started but exited silently | Check `%LOCALAPPDATA%\Syncthing\` for config issues; restart with `--logfile` flag |
| SSH to Windows mangles backslashes | Bash SSH escaping | Use PowerShell scripts via SCP, or use `$HOME\` which expands server-side |
---
## Config File Locations
| OS | Config XML | Data/Index |
|----|------------|------------|
| Linux (apt) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
| Linux (brew) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
| Windows | `%LOCALAPPDATA%\Syncthing\config.xml` | `%LOCALAPPDATA%\Syncthing\` |
## API Reference
- **Get config:** `GET http://127.0.0.1:8384/rest/config`
- **Set config:** `PUT http://127.0.0.1:8384/rest/config` (full config JSON)
- **Connections:** `GET http://127.0.0.1:8384/rest/system/connections`
- **Folder status:** `GET http://127.0.0.1:8384/rest/db/status?folder=projects`
- **Header:** `X-API-Key: <apikey>`

101
runbooks/utility-caddy-initial-setup.md
View file

@ -1,101 +0,0 @@
# Utility Caddy LXC — Initial Setup
One-time setup. Only needed if rebuilding from scratch.
## Overview
| Item | Value |
|------|-------|
| CT ID | 101 |
| Hostname | caddy |
| Local IP | 192.168.1.101 |
| Tailscale IP | 100.64.0.2 |
| Public access | 199.6.36.163 (router forwards 80/443) |
## 1. Create LXC
```bash
ssh root@192.168.1.241
pct create 101 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname caddy \
--cores 1 \
--memory 512 \
--swap 256 \
--rootfs local-lvm:8 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.101/24,gw=192.168.1.1 \
--features nesting=1 \
--unprivileged 1 \
--password <from .ref/credentials>
# TUN device for Tailscale
cat >> /etc/pve/lxc/101.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 101
```
## 2. Install Tailscale
```bash
pct exec 101 -- bash -c "
echo nameserver 1.1.1.1 > /etc/resolv.conf
apt-get update && apt-get install -y curl
curl -fsSL https://tailscale.com/install.sh | sh
"
```
## 3. Register with Headscale
```bash
pct exec 101 -- tailscale up --login-server https://vpn.echo6.co --hostname caddy
# On Contabo — register the node
ssh root@100.64.0.6 'docker exec headscale-standby headscale nodes register --key <KEY> --user echo6'
# Verify
pct exec 101 -- tailscale status
```
## 4. Install Caddy
```bash
pct exec 101 -- bash -c "
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt | tee /etc/apt/sources.list.d/caddy-stable.list
apt-get update && apt-get install -y caddy
"
```
## 5. Install acme.sh
```bash
pct exec 101 -- bash -c "
curl https://get.acme.sh | sh -s email=admin@echo6.co
"
```
## 6. Create initial Caddyfile
```bash
pct exec 101 -- bash -c "cat > /etc/caddy/Caddyfile << 'EOF'
{
email admin@echo6.co
}
EOF
systemctl enable caddy
systemctl start caddy"
```
## 7. Router port forward
Forward on your router:
- TCP 80 → 192.168.1.101:80
- TCP 443 → 192.168.1.101:443
## Done
Add services using the expose-service-home.md runbook.

222
runbooks/vaultwarden-deployment.md
View file

@ -1,222 +0,0 @@
# Vaultwarden Deployment
**Deployed:** 2026-02-05
**Location:** Contabo VPS (5.189.158.149 / 100.64.0.6)
**URL:** https://vault.echo6.co
---
## Service Details
| Setting | Value |
|---------|-------|
| Container | `vaultwarden` |
| Image | `vaultwarden/server:latest` |
| Port | `127.0.0.1:8086` (web), `127.0.0.1:3012` (websocket) |
| Data | `/opt/vaultwarden/data` |
| Config | `/opt/vaultwarden/.env` |
| SSO | Authentik (enabled) |
| Signups | Disabled (invite-only) |
---
## Access
| Method | URL |
|--------|-----|
| Web Vault | https://vault.echo6.co |
| Admin Panel | https://vault.echo6.co/admin |
| SSO Login | "Enterprise Single Sign-On" button |
---
## Configuration Files
### Docker Compose (`/opt/vaultwarden/docker-compose.yml`)
```yaml
services:
vaultwarden:
image: vaultwarden/server:latest
container_name: vaultwarden
restart: unless-stopped
env_file:
- .env
ports:
- "127.0.0.1:8086:80"
- "127.0.0.1:3012:3012"
volumes:
- ./data:/data
environment:
- TZ=America/Boise
```
### Environment (`.env`)
```bash
# Admin
ADMIN_TOKEN=<see credentials file>
DOMAIN=https://vault.echo6.co
# Security
SIGNUPS_ALLOWED=false
INVITATIONS_ALLOWED=true
SHOW_PASSWORD_HINT=false
# WebSocket
WEBSOCKET_ENABLED=true
# SSO (Authentik)
SSO_ENABLED=true
SSO_ONLY=false
SSO_CLIENT_ID=vaultwarden
SSO_CLIENT_SECRET=<see credentials file>
SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/
SSO_PKCE=true
SSO_SCOPES="openid email profile offline_access"
# Timezone
TZ=America/Boise
LOG_LEVEL=info
```
### Caddy Site Block
```caddyfile
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### dnsmasq Split DNS
```conf
address=/vault.echo6.co/100.64.0.6
```
---
## Authentik SSO Configuration
### Provider Settings (pk=3)
| Setting | Value |
|---------|-------|
| Name | Vaultwarden |
| Client ID | `vaultwarden` |
| Client Type | Confidential |
| Redirect URI | `https://vault.echo6.co/identity/connect/oidc-signin` |
| Signing Key | authentik Internal JWT Certificate (RS256) |
| Access Token Validity | 1 hour |
| Refresh Token Validity | 30 days |
### Scopes
- `openid` - Required for OIDC
- `email` - User email
- `profile` - User profile
- `offline_access` - Refresh tokens
### OIDC Endpoints
| Endpoint | URL |
|----------|-----|
| Discovery | https://auth.echo6.co/application/o/vaultwarden/.well-known/openid-configuration |
| JWKS | https://auth.echo6.co/application/o/vaultwarden/jwks/ |
| Authorize | https://auth.echo6.co/application/o/authorize/ |
| Token | https://auth.echo6.co/application/o/token/ |
---
## Troubleshooting
### SSO Login Loop
**Symptom:** After SSO auth, redirects back to login screen.
**Causes:**
1. Access token too short (< 5 min)
2. Missing `offline_access` scope (no refresh token)
3. Missing signing key (empty JWKS)
**Fix:**
```bash
# Check Authentik provider settings via ak shell
docker exec authentik-server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.get(name='Vaultwarden')
print(f'Access Token: {p.access_token_validity}')
print(f'Signing Key: {p.signing_key}')
print(f'Scopes: {list(p.property_mappings.values_list(\"scope_name\", flat=True))}')"
```
### SSO Discovery Error
**Symptom:** "Failed to discover OpenID provider: Failed to parse server response"
**Causes:**
1. Empty JWKS endpoint (no signing key)
2. Missing property mappings
**Fix:** Add signing key and scopes to Authentik provider.
### View Logs
```bash
# Vaultwarden
docker logs vaultwarden --tail 100 2>&1 | grep -i -E "sso|error"
# Authentik
docker logs authentik-server --tail 100 2>&1 | grep -i vaultwarden
```
---
## Maintenance
### Restart Service
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose restart
```
### Update Image
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose pull
docker compose up -d
```
### Backup Data
```bash
# Stop container first
docker compose stop
tar -czf vaultwarden-backup-$(date +%Y%m%d).tar.gz data/
docker compose start
```
---
## Credentials Reference
All credentials stored in `/home/zvx/projects/.ref/credentials`:
```
VAULTWARDEN_URL
VAULTWARDEN_ADMIN_TOKEN
VAULTWARDEN_ADMIN_URL
VAULTWARDEN_OIDC_PROVIDER_ID
VAULTWARDEN_OIDC_CLIENT_ID
VAULTWARDEN_OIDC_CLIENT_SECRET
VAULTWARDEN_OIDC_ISSUER
```
---
*Last updated: 2026-02-05*