matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit: infrastructure documentation

Browse source 
Includes:
- Hardware environment reference (Proxmox cluster, VMs, LXCs)
- Services inventory with current deployments
- Caddy & DNS configuration reference
- Runbooks for common deployment procedures

Recent additions:
- SearXNG deployment (utility CT 102, search.echo6.co)
- TOC conversion to Proxmox with cortex VM
- Syncthing sync between Contabo and cortex

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-02-06 21:27:29 +01:00
commit 880ff09c90
14 changed files with 1986 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

15
.gitignore vendored Normal file
View file

@ -0,0 +1,15 @@
# Credentials - NEVER commit
credentials
credentials.bak
*.credentials
# Backup files
*.bak
*.backup
*~
# Editor files
.vscode/
.idea/
*.swp
*.swo

107
docs/hardware/environment.md Normal file
View file

@ -0,0 +1,107 @@
# Echo6 Environment Reference
## Proxmox Cluster (echo6-cluster)
Five nodes running Proxmox VE:
| Node | Local IP | Tailscale | Hardware | Purpose |
|------|----------|-----------|----------|---------|
| data | 192.168.1.240 | 100.64.0.6 | Mini PC | Database services |
| utility | 192.168.1.241 | 100.64.0.5 | Mini PC | Utility services, monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | Mini PC | Cloud storage, personal services |
| media | 192.168.1.243 | 100.64.0.3 | Mini PC | Media server, *arr stack |
| toc | 192.168.1.244 | 100.64.0.13 | Workstation | GPU compute, AI/ML workloads |
### TOC Node Details
- **Hardware:** Intel i9-10900X (20 threads), 48GB RAM, 512GB NVMe, RTX A4000
- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
- **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
## Virtual Machines
| VM | Host | VMID | Local IP | Tailscale | Purpose |
|----|------|------|----------|-----------|---------|
| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — LLMs, ARGUS, Aurora, model training |
### cortex VM Details
- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
- **Resources:** 16 threads, 32GB RAM, 300GB disk
- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
- **User:** zvx (sudo, SSH keys from cluster)
- **Claude Code:** v2.1.34 installed
## Key Servers
| Server | Local IP | Tailscale | Purpose |
|--------|----------|-----------|---------|
| aida-nebra | 192.168.1.253 | 100.64.0.9 | Meshtastic node (meshtasticd on Pi) |
| matt-desktop | — | 100.64.0.10 | Personal workstation |
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge |
## LXC Containers
| Container | Host | Local IP | Tailscale | Purpose |
|-----------|------|----------|-----------|---------|
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring |
| caddy | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | SearXNG metasearch engine |
## IP Allocation Scheme
| Range | Purpose |
|-------|---------|
| .1-.10 | Network infrastructure |
| .11-.99 | DHCP clients |
| .100-.149 | LXC containers |
| .150-.199 | VMs |
| .240-.250 | Proxmox hosts + bare metal |
| .251-.254 | Meshtastic nodes |
Full details: `/home/zvx/projects/utility/ip-allocation.md`
## Headscale Node List
Current registered nodes (12 total):
| Node | Tailscale IP | Type |
|------|-------------|------|
| contabo | 100.64.0.1 | VPS |
| media | 100.64.0.3 | Proxmox |
| cloud | 100.64.0.4 | Proxmox |
| utility | 100.64.0.5 | Proxmox |
| data | 100.64.0.6 | Proxmox |
| meshmonitor | 100.64.0.7 | LXC |
| caddy | 100.64.0.8 | LXC |
| aida-nebra | 100.64.0.9 | Pi |
| matt-desktop | 100.64.0.10 | Desktop |
| toc | 100.64.0.13 | Proxmox |
| cortex | 100.64.0.14 | VM |
| searxng | 100.64.0.15 | LXC |
## SSH Access
**Standard user:** `zvx`
**Credentials:** Source from `/home/zvx/projects/.ref/credentials`
```bash
# SSH to any server
ssh zvx@<ip-address>
# Examples
ssh zvx@192.168.1.244 # TOC (Proxmox host)
ssh zvx@192.168.1.150 # cortex VM
ssh zvx@192.168.1.241 # utility Proxmox
ssh root@100.64.0.1 # Contabo (via Tailscale)
ssh zvx@cortex # cortex via Tailscale hostname
```
## Key External IPs
| Purpose | IP |
|---------|-----|
| Home external (public services) | 199.6.36.163 |
| Contabo VPS | 5.189.158.149 |

68
docs/services/services.md Normal file
View file

@ -0,0 +1,68 @@
# Current Services Inventory
## Active Services
| Service | Location | IP:Port | Access | Notes |
|---------|----------|---------|--------|-------|
| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring |
| Utility Caddy | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
| SearXNG | utility (CT 102) | 192.168.1.102:8080 | https://search.echo6.co | Metasearch engine (Docker) |
| meshtasticd | aida-nebra | 192.168.1.253:4403 | Internal | Software Meshtastic node |
| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider |
| Forge | Contabo | 5.189.158.149 | https://forge.echo6.co | Git server |
| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
| Vaultwarden | Contabo | 127.0.0.1:8086 | https://vault.echo6.co | Password manager (SSO enabled) |
| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
## Services by Server
### toc - Proxmox Host (192.168.1.244 / Tailscale: 100.64.0.13)
- Proxmox VE node (echo6-cluster)
- GPU passthrough host for cortex VM
- No direct services — workloads run on cortex VM
### cortex - VM 150 on toc (192.168.1.150 / Tailscale: 100.64.0.14)
- GPU compute VM (RTX A4000)
- Claude Code host
- Syncthing (syncs with Contabo)
- **Planned:** Ollama, Open-WebUI, LiteLLM, ARGUS, Aurora
### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7)
- MeshMonitor (port 8080)
### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8)
- Utility Caddy (reverse proxy for VPN-only services)
### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
- SearXNG metasearch engine (port 8080)
- Redis/Valkey cache
- Compose path: `/opt/searxng/docker-compose.yml`
### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9)
- meshtasticd (software Meshtastic node)
### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
- Authentik (SSO)
- Forge (Git)
- Headscale (mesh VPN)
- Mailcow (email)
- Vaultwarden (passwords)
- Syncthing (syncs with cortex)
## Adding New Services
When deploying a new service, update this file with:
1. Service name
2. Host location (server + container if applicable)
3. IP:Port
4. Access method (internal only vs public URL)
5. Brief description
## Naming Conventions
- **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP
- **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy

77
docs/software/authentik.md Normal file
View file

@ -0,0 +1,77 @@
# Authentik SSO Configuration
## Location
- **Server:** Contabo (5.189.158.149 / 100.64.0.6)
- **URL:** https://auth.echo6.co
- **Internal Port:** 9000
## API Access
API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN`
## Flow UUIDs
Required for OAuth2 provider creation:
| Flow | UUID |
|------|------|
| Authorization (implicit) | `86051292-389f-4bd9-b0f9-53cd32f197fd` |
| Authorization (explicit) | `6f9f5c89-9f98-4776-9e0d-a72a8ad17963` |
| Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` |
| Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` |
## Create New API Token
```bash
ssh root@100.64.0.6 'docker exec authentik-server ak shell -c "
from authentik.core.models import Token, User
user = User.objects.get(username=\"akadmin\")
token, created = Token.objects.get_or_create(
identifier=\"token-name\",
user=user,
defaults={\"intent\": \"api\", \"expiring\": False}
)
print(token.key)
"'
```
## Quick OAuth2 Provider Creation
```bash
# Source credentials
source /home/zvx/projects/.ref/credentials
# Create provider
curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd",
"invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a",
"client_type": "confidential",
"client_id": "appname",
"redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}],
"sub_mode": "user_username"
}'
# Create application (use pk from provider response)
curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"slug": "appname",
"provider": PROVIDER_PK,
"meta_launch_url": "https://app.echo6.co"
}'
```
## Common Redirect URI Patterns
| Application Type | Redirect URI Pattern |
|------------------|---------------------|
| Web app | `https://app.echo6.co/callback` |
| Web app (oauth) | `https://app.echo6.co/oauth/callback` |
| Caddy forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` |

162
docs/software/caddy.md Normal file
View file

@ -0,0 +1,162 @@
# Caddy & DNS Reference
## Contabo Caddy
**Config:** `/etc/caddy/Caddyfile` on Contabo (ssh root@100.64.0.1)
### Current Site Blocks
| Domain | Backend | Service |
|--------|---------|---------|
| auth.echo6.co | 127.0.0.1:9000 | Authentik SSO |
| forge.echo6.co | 127.0.0.1:3001 | Forgejo Git |
| mail.echo6.co | https://127.0.0.1:8443 | Mailcow (tls_insecure_skip_verify) |
| vpn.echo6.co | 127.0.0.1:8084 | Headscale |
| vpn.echo6.co/admin* | 127.0.0.1:3100 | Headplane |
| autodiscover.echo6.co | https://127.0.0.1:8443 | Mailcow autodiscover |
| autoconfig.echo6.co | https://127.0.0.1:8443 | Mailcow autoconfig |
| vault.echo6.co | 127.0.0.1:8086 | Vaultwarden |
| proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) |
### Commands
```bash
ssh root@100.64.0.1
caddy validate --config /etc/caddy/Caddyfile
systemctl restart caddy # admin off, so reload won't work
journalctl -u caddy -f
```
---
## Utility Caddy (Home)
**Location:** CT 101 on utility Proxmox (192.168.1.101)
**Tailscale IP:** 100.64.0.8
**Config:** `/etc/caddy/Caddyfile` inside CT 101
**SSL Certs:** `/etc/caddy/certs/` (managed by acme.sh)
**Port forward:** Router 80/443 → 192.168.1.101
### Current Site Blocks
| Domain | Backend | Pattern | Service |
|--------|---------|---------|---------|
| mesh.echo6.co | 100.64.0.7:8080 | Tailscale | MeshMonitor |
| search.echo6.co | 100.64.0.15:8080 | Tailscale | SearXNG |
### Commands
```bash
ssh root@192.168.1.241 'pct exec 101 -- cat /etc/caddy/Caddyfile'
ssh root@192.168.1.241 'pct exec 101 -- systemctl reload caddy'
ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
```
---
## dnsmasq (Tailscale Split DNS)
**Config:** `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo
**Listens on:** 100.64.0.1:53
### Current Records
| Domain | Tailscale IP | Service |
|--------|-------------|---------|
| auth.echo6.co | 100.64.0.1 | Authentik |
| forge.echo6.co | 100.64.0.1 | Forgejo |
| mail.echo6.co | 100.64.0.1 | Mailcow |
| vpn.echo6.co | 100.64.0.1 | Headscale |
| vault.echo6.co | 100.64.0.1 | Vaultwarden |
| docs.echo6.co | 100.64.0.1 | Wiki.js |
| proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) |
| stream.echo6.co | *TBD* | PeerTube - needs host verification |
| notes.echo6.co | *TBD* | Obsidian LiveSync - needs host verification |
### Commands
```bash
ssh root@100.64.0.1
nano /etc/dnsmasq.d/tailscale-dns.conf
systemctl restart dnsmasq
dig +short forge.echo6.co @100.64.0.1 # Test
```
---
## GoDaddy DNS Records (echo6.co)
### Contabo Services → 5.189.158.149
| Subdomain | Service |
|-----------|---------|
| auth | Authentik SSO |
| forge | Forgejo Git |
| mail | Mailcow Email |
| vpn | Headscale VPN |
| vault | Vaultwarden |
### Home Services → 199.6.36.163
| Subdomain | Service |
|-----------|---------|
| @ | Main site |
| ai | Open WebUI |
| docs | Wiki.js |
| stream | PeerTube |
| notes | Obsidian LiveSync |
| jellyfin | Jellyfin |
| mesh | MeshMonitor |
| search | SearXNG |
### Email Records
| Type | Name | Value |
|------|------|-------|
| MX | @ | mail.echo6.co |
| CNAME | autoconfig | mail.echo6.co |
| CNAME | autodiscover | mail.echo6.co |
| TXT | @ | v=spf1 mx a:mail.echo6.co -all |
| TXT | _dmarc | v=DMARC1; p=quarantine |
| TXT | dkim._domainkey | (DKIM key) |
---
## Headscale Config
**Location:** `/opt/headscale/` on Contabo
**Data:** Named Docker volume `headscale_headscale-data`
**Config:** `/opt/headscale/config.yaml`
```yaml
dns:
base_domain: echo6.mesh
nameservers:
global:
- 1.1.1.1
oidc:
issuer: "https://auth.echo6.co/application/o/headscale/"
client_id: "headscale"
```
**Split DNS:** Configured via dnsmasq on Contabo.
**Headplane:** Deployed at `vpn.echo6.co/admin` - OIDC via Authentik. First login gets Owner.
---
## Port Map (Contabo)
| Service | Container Port | Host Binding | Public Domain |
|---------|---------------|--------------|---------------|
| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
---
*Last updated: 2026-02-06 — Added SearXNG (search.echo6.co) on utility CT 102*

64
docs/software/dns.md Normal file
View file

@ -0,0 +1,64 @@
# GoDaddy DNS Management
## Script Location
`~/bin/godaddy-dns.py`
## API Credentials
Stored in `/home/zvx/projects/.ref/credentials` as:
- `GODADDY_API_KEY`
- `GODADDY_API_SECRET`
## Key IPs for DNS Records
| Purpose | IP |
|---------|-----|
| External (home services) | `199.6.36.163` |
| Contabo Server | `5.189.158.149` |
## Managed Domains
arclightvanguard.com, echo6.co, echo6.org, happylittlellc.com, idahomesh.com, k7zvx.com, lpmesh.com, maliceinwonderland.org, matthewwayne.com, smugglersden.co, underdogs.cc
## Usage Examples
```bash
# List all domains
godaddy-dns.py list-domains
# List records for a domain
godaddy-dns.py list echo6.co
# Add A record
godaddy-dns.py add-a echo6.co www 199.6.36.163
# Add CNAME record
godaddy-dns.py add-cname echo6.co blog www.echo6.co
# Add MX record with priority
godaddy-dns.py add-mx echo6.co mail.echo6.co --priority=10
# Delete record
godaddy-dns.py delete echo6.co A www
# Configure MX for all domains
godaddy-dns.py setup-mail
```
## Common Patterns
### Point subdomain to home network
```bash
godaddy-dns.py add-a echo6.co newservice 199.6.36.163
```
### Point subdomain to Contabo
```bash
godaddy-dns.py add-a echo6.co auth 5.189.158.149
```
### Create CNAME alias
```bash
godaddy-dns.py add-cname echo6.co alias target.echo6.co
```

183
runbooks/contabo-configs.md Normal file
View file

@ -0,0 +1,183 @@
# Contabo VPS Current Configurations
**Server:** 5.189.158.149 / 100.64.0.4
**Last Updated:** 2026-02-05
---
## Caddy Configuration
**File:** `/etc/caddy/Caddyfile`
```caddyfile
# Global options
{
email admin@echo6.co
admin off
}
# Main Mailcow hostname
mail.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
read_timeout 3600s
write_timeout 3600s
}
}
}
# Autodiscover for Outlook
autodiscover.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
}
}
}
# Autoconfig for Thunderbird
autoconfig.echo6.co {
reverse_proxy https://127.0.0.1:8443 {
transport http {
tls_insecure_skip_verify
}
}
}
# Headscale VPN + Headplane Admin
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3100
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
# Authentik SSO
auth.echo6.co {
reverse_proxy 127.0.0.1:9000
}
# Forgejo Git Forge
forge.echo6.co {
reverse_proxy 127.0.0.1:3001
}
# Vaultwarden Password Manager
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### Commands
```bash
# Validate
caddy validate --config /etc/caddy/Caddyfile
# Restart (admin off, so reload won't work)
systemctl restart caddy
# Logs
journalctl -u caddy -f
```
---
## dnsmasq Split DNS Configuration
**File:** `/etc/dnsmasq.d/tailscale-dns.conf`
```conf
# DNSmasq config for Tailscale Split DNS
# Listen only on Tailscale interface
listen-address=100.64.0.4
bind-interfaces
# Upstream DNS servers
server=1.1.1.1
server=8.8.8.8
# Local records for echo6.co services (route through Tailscale)
address=/forge.echo6.co/100.64.0.4
address=/auth.echo6.co/100.64.0.4
address=/mail.echo6.co/100.64.0.4
address=/vpn.echo6.co/100.64.0.4
address=/docs.echo6.co/100.64.0.4
address=/vault.echo6.co/100.64.0.4
address=/stream.echo6.co/100.64.0.7
address=/notes.echo6.co/100.64.0.22
# Don't read /etc/hosts
no-hosts
# Cache size
cache-size=1000
# Log queries for debugging
log-queries
```
### Commands
```bash
# Restart
systemctl restart dnsmasq
# Status
systemctl status dnsmasq
# Test resolution
dig +short vault.echo6.co @100.64.0.4
```
---
## Port Mappings Summary
| Service | Container Port | Host Binding | Caddy Proxy |
|---------|---------------|--------------|-------------|
| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
| Forgejo SSH | 22 | 0.0.0.0:2222 | Direct |
| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
---
## DNS Records (GoDaddy → Contabo)
| Subdomain | IP | Service |
|-----------|-----|---------|
| auth | 5.189.158.149 | Authentik |
| forge | 5.189.158.149 | Forgejo |
| mail | 5.189.158.149 | Mailcow |
| vpn | 5.189.158.149 | Headscale |
| vault | 5.189.158.149 | Vaultwarden |
| autodiscover | 5.189.158.149 | Mailcow |
| autoconfig | 5.189.158.149 | Mailcow |
---
## Split DNS Mappings (Tailscale)
| Domain | Tailscale IP | Server |
|--------|-------------|--------|
| auth.echo6.co | 100.64.0.4 | Contabo |
| forge.echo6.co | 100.64.0.4 | Contabo |
| mail.echo6.co | 100.64.0.4 | Contabo |
| vpn.echo6.co | 100.64.0.4 | Contabo |
| vault.echo6.co | 100.64.0.4 | Contabo |
| docs.echo6.co | 100.64.0.4 | Contabo |
| stream.echo6.co | 100.64.0.7 | PeerTube |
| notes.echo6.co | 100.64.0.22 | Cloud |
---
*Last updated: 2026-02-05*

80
runbooks/expose-service-contabo.md Executable file
View file

@ -0,0 +1,80 @@
# Expose Service on Contabo
## Prerequisites
- Service running in Docker on Contabo
- Port bound to `127.0.0.1` only (never `0.0.0.0`)
## Steps
### 1. Deploy the service
```bash
ssh root@100.64.0.6
mkdir -p /opt/<service>
# Create docker-compose.yml with port bound to 127.0.0.1:<port>
docker compose up -d
```
### 2. Add DNS record
```bash
# On TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a echo6.co <service> 5.189.158.149
dig +short <service>.echo6.co @8.8.8.8 # Verify
```
### 3. Add Caddy site block
```bash
ssh root@100.64.0.6
nano /etc/caddy/Caddyfile
# Add:
# <service>.echo6.co {
# reverse_proxy 127.0.0.1:<port>
# }
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
```
### 4. Add dnsmasq split DNS entry
```bash
ssh root@100.64.0.6
nano /etc/dnsmasq.d/tailscale-dns.conf
# Add:
# address=/<service>.echo6.co/100.64.0.6
systemctl restart dnsmasq
```
### 5. Verify
```bash
# Public
curl -I https://<service>.echo6.co
# Tailscale
dig +short <service>.echo6.co @100.64.0.6 # Should return 100.64.0.6
```
### 6. Update docs
- Update `~/.claude/docs/infrastructure/caddy.md` with new site block
- Update `~/.claude/docs/infrastructure/services.md` with new service
- Add credentials to `/home/zvx/projects/.ref/credentials` if applicable
## Checklist
```
□ Docker container running, port on 127.0.0.1 only
□ GoDaddy DNS → 5.189.158.149
□ Caddy site block added and reloaded
□ dnsmasq entry added and restarted
□ Public access verified
□ Tailscale access verified
□ Docs updated
```

107
runbooks/expose-service-home.md Executable file
View file

@ -0,0 +1,107 @@
# Expose Service on Home Network
## Prerequisites
- Service running on a Proxmox CT/VM or bare metal
- Router forwards 80/443 to Utility Caddy (192.168.1.101) — one-time setup
- Determine pattern: does the service have Authentik OIDC?
## Steps
### 1. Determine backend target
| Has OIDC? | Proxy to | Why |
|-----------|----------|-----|
| YES | Local IP (192.168.1.x:port) | Authentik SSO protects access |
| NO | Tailscale IP (100.64.0.x:port) | Only Caddy can reach backend |
If no OIDC, service MUST have Tailscale installed and registered with Headscale first.
### 2. Issue SSL certificate
```bash
ssh root@192.168.1.241
pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d <service>.echo6.co --server letsencrypt
'
```
### 3. Install certificate
```bash
pct exec 101 -- bash -c "
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d <service>.echo6.co \
--cert-file /etc/caddy/certs/<service>.echo6.co.crt \
--key-file /etc/caddy/certs/<service>.echo6.co.key \
--fullchain-file /etc/caddy/certs/<service>.echo6.co.fullchain.crt \
--reloadcmd 'systemctl reload caddy'
chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
"
```
### 4. Add Caddy site block
```bash
# WITH OIDC — local IP
pct exec 101 -- bash -c "cat >> /etc/caddy/Caddyfile << 'EOF'
<service>.echo6.co {
tls /etc/caddy/certs/<service>.echo6.co.fullchain.crt /etc/caddy/certs/<service>.echo6.co.key
reverse_proxy 192.168.1.<X>:<PORT>
}
EOF
systemctl reload caddy"
# WITHOUT OIDC — Tailscale IP
pct exec 101 -- bash -c "cat >> /etc/caddy/Caddyfile << 'EOF'
<service>.echo6.co {
tls /etc/caddy/certs/<service>.echo6.co.fullchain.crt /etc/caddy/certs/<service>.echo6.co.key
reverse_proxy 100.64.0.<X>:<PORT>
}
EOF
systemctl reload caddy"
```
### 5. Add DNS record
```bash
# On TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a echo6.co <service> 199.6.36.163
```
### 6. Update service CORS (if applicable)
Add `https://<service>.echo6.co` to the service's allowed origins.
### 7. Verify
```bash
curl -I https://<service>.echo6.co
```
### 8. Update docs
- Update `~/.claude/docs/infrastructure/caddy.md` with new site block
- Update `~/.claude/docs/infrastructure/services.md` with new service
- Add credentials to `/home/zvx/projects/.ref/credentials` if applicable
## Checklist
```
□ Backend pattern chosen (OIDC → local IP, no OIDC → Tailscale IP)
□ SSL cert issued and installed via acme.sh
□ Caddy site block added to CT 101 Caddyfile
□ Caddy reloaded
□ GoDaddy DNS → 199.6.36.163
□ CORS updated if needed
□ HTTPS access verified
□ Docs updated
```

406
runbooks/headscale-full-deployment.md Executable file
View file

@ -0,0 +1,406 @@
# Headscale Full Deployment Runbook
## Nodes + Headplane + Authentik OIDC
**Headscale location:** `/opt/headscale-vanilla`
**Container name:** `headscale-vanilla`
**Domain:** `vpn.echo6.co`
**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ`
---
## PHASE 1: REGISTER CONTABO (must be first)
```bash
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname contabo --force-reauth
```
Verify:
```bash
docker exec headscale-vanilla headscale nodes list
```
**STOP if contabo doesn't appear. Do not continue.**
---
## PHASE 2: REGISTER ALL LXC/CT NODES
SSH into each container. For each one:
```bash
# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"
# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh
```
Then register. **Do them in this exact order for sequential IPs:**
```bash
# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname utility --force-reauth
# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname data --force-reauth
# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname cloud --force-reauth
# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname media --force-reauth
# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname aida-nebra --force-reauth
```
After each, verify from Contabo:
```bash
docker exec headscale-vanilla headscale nodes list
```
---
## PHASE 3: REGISTER DESKTOP + PHONES
**Desktop (Windows — PowerShell as Admin):**
```powershell
tailscale up --login-server https://vpn.echo6.co `
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
--hostname desktop --force-reauth
```
**Phones:**
- Open Tailscale app → Settings → Account
- Log out if needed
- Use "Custom coordination server" or "Alternate server"
- Enter: `https://vpn.echo6.co`
- Should auto-register with the tailnet
If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.
---
## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY
```bash
docker exec headscale-vanilla headscale nodes list
```
Expected output: all nodes with sequential 100.64.0.x IPs.
Test from any node:
```bash
tailscale ping contabo
tailscale ping data
tailscale ping utility
```
Test magic DNS:
```bash
ping data.echo6.mesh
ping utility.echo6.mesh
```
---
## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)
```bash
mkdir -p /opt/headscale-vanilla/backups
# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"
# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
```
---
## PHASE 6: PERSISTENCE TEST
```bash
cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list
```
**Every node must survive. If any are missing, STOP and report.**
---
## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE
This lets Tailscale clients authenticate via Authentik instead of preauth keys.
1. Log into Authentik admin panel
2. Go to **Applications → Applications → Create with Provider**
3. Configure:
- **Application name:** Headscale
- **Slug:** `headscale` (remember this — it's part of the issuer URL)
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
- **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
- **Signing key:** Select any available key
- **Scopes:** Ensure these scope mappings are selected:
- `openid`
- `profile`
- `email`
- **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
4. Note the **Client ID** and **Client Secret**
5. Click Submit
---
## PHASE 8: CONFIGURE HEADSCALE OIDC
Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:
```yaml
oidc:
only_start_if_oidc_is_available: true
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
client_id: "<Client ID from Authentik>"
client_secret: "<Client Secret from Authentik>"
scope: ["openid", "profile", "email", "offline_access"]
pkce:
enabled: true
method: S256
strip_email_domain: true
```
Replace:
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
- `<Client ID from Authentik>` with the actual client ID
- `<Client Secret from Authentik>` with the actual client secret
Restart Headscale:
```bash
cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20
```
**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**
Test: From any node, run:
```bash
tailscale up --login-server https://vpn.echo6.co --force-reauth
```
It should open a browser → Authentik login → back to terminal, authenticated.
**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**
---
## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE
This is a SECOND application in Authentik for the web UI login.
1. Go to **Applications → Applications → Create with Provider**
2. Configure:
- **Application name:** Headplane
- **Slug:** `headplane`
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** Same as before
- **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
- **Signing key:** Same key
- **Scopes:** `openid`, `profile`, `email`
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
4. Click Submit
---
## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE
```bash
docker exec headscale-vanilla headscale apikeys create --expiration 999d
```
**Save this key — you need it for the Headplane config.**
---
## PHASE 11: CREATE HEADPLANE CONFIG
```bash
# Generate a cookie secret
openssl rand -hex 16
```
Write `/opt/headscale-vanilla/headplane-config.yaml`:
```yaml
server:
host: "0.0.0.0"
port: 3000
cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
cookie_secure: true
data_path: "/var/lib/headplane"
headscale:
url: "http://headscale-vanilla:8080"
config_path: "/etc/headscale/config.yaml"
config_strict: false
oidc:
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
client_id: "<Headplane Client ID from Authentik>"
client_secret: "<Headplane Client Secret from Authentik>"
token_endpoint_auth_method: "client_secret_post"
headscale_api_key: "<API_KEY_FROM_PHASE_10>"
redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
disable_api_key_login: false
integration:
docker:
enabled: true
container_name: "headscale-vanilla"
socket: "/var/run/docker.sock"
```
Replace all `<PLACEHOLDERS>` with actual values.
---
## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE
Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:
```yaml
services:
headscale:
# ... your existing headscale service, don't change it ...
headplane:
image: ghcr.io/tale/headplane:latest
container_name: headplane
restart: unless-stopped
depends_on:
- headscale
ports:
- "127.0.0.1:3000:3000"
volumes:
- ./headplane-config.yaml:/etc/headplane/config.yaml:ro
- ./headplane-data:/var/lib/headplane
- ./config.yaml:/etc/headscale/config.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
```
Start it:
```bash
cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20
```
Check for errors. Common issues:
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
- Can't connect to headscale → ensure `url` matches the container name and internal port
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock
---
## PHASE 13: UPDATE CADDY FOR HEADPLANE
Add the `/admin` route to your Caddy config for `vpn.echo6.co`:
```
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3000
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
```
Restart Caddy:
```bash
# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```
---
## PHASE 14: TEST HEADPLANE
1. Browse to `https://vpn.echo6.co/admin`
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. **The FIRST user to log in gets Owner permissions**
5. Verify you can see all your nodes in the UI
If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).
---
## PHASE 15: FINAL VERIFICATION
Run all of these from Contabo:
```bash
# All nodes present?
docker exec headscale-vanilla headscale nodes list
# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"
# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302
# Database backed up?
ls -la /opt/headscale-vanilla/backups/
# Cron running?
crontab -l | grep sqlite3
```
---
## REPORT TEMPLATE
After each phase, report:
```
Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):
```
**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**
---
## KNOWN GOTCHAS
1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs

127
runbooks/meshmonitor-password-reset.md Normal file
View file

@ -0,0 +1,127 @@
# MeshMonitor Admin Password Reset
## Overview
Reset the admin password for MeshMonitor web interface.
| Item | Value |
|------|-------|
| Service | MeshMonitor |
| Location | utility CT 100 (192.168.1.100:8080) |
| Database | SQLite at `/data/meshmonitor.db` |
## Prerequisites
- SSH access to Proxmox utility node (192.168.1.241) as root
- `sshpass` installed on local machine
- `python3` with `bcrypt` module (for generating password hash)
## Quick Reset (One-liner)
Reset to a specific password (generates hash inside container to avoid escaping issues):
```bash
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no root@192.168.1.241 'pct exec 100 -- docker exec meshmonitor node -e "
const Database = require(\"better-sqlite3\");
const bcrypt = require(\"bcrypt\");
const db = new Database(\"/data/meshmonitor.db\");
const hash = bcrypt.hashSync(\"7redditGold\", 10);
db.prepare(\"UPDATE users SET password_hash = ? WHERE username = ?\").run(hash, \"admin\");
console.log(\"Password updated to 7redditGold\");
db.close();
"'
```
To use a different password, replace both instances of `7redditGold` (SSH password and new MeshMonitor password).
## Step-by-Step Process
### 1. SSH to Proxmox Host
```bash
ssh root@192.168.1.241
```
### 2. Enter the Container
```bash
pct exec 100 -- bash
```
### 3. Option A: Use Built-in Reset Script (Random Password)
```bash
docker exec meshmonitor node /app/reset-admin.mjs
```
This generates a random password - save the output.
### 3. Option B: Set Specific Password via Node.js
Generate a bcrypt hash first (run on any machine with Node.js):
```bash
node -e "const bcrypt = require('bcrypt'); \
bcrypt.hash('YOUR_PASSWORD_HERE', 10).then(h => console.log(h));"
```
Then update the database:
```bash
docker exec meshmonitor node -e "
const Database = require('better-sqlite3');
const db = new Database('/data/meshmonitor.db');
db.prepare(\"UPDATE users SET password_hash = ? WHERE username = 'admin'\")
.run('\$2b\$10\$YOUR_HASH_HERE');
db.close();
"
```
### 4. Verify Login
Open http://192.168.1.100:8080 and log in with:
- Username: `admin`
- Password: (your new password)
## Troubleshooting
### Check Current Admin User
```bash
docker exec meshmonitor node -e "
const Database = require('better-sqlite3');
const db = new Database('/data/meshmonitor.db');
console.log(db.prepare(\"SELECT * FROM users WHERE username='admin'\").get());
db.close();
"
```
### List All Users
```bash
docker exec meshmonitor node -e "
const Database = require('better-sqlite3');
const db = new Database('/data/meshmonitor.db');
console.log(db.prepare('SELECT id, username, is_admin, is_active FROM users').all());
db.close();
"
```
### Container Not Running
```bash
# On Proxmox host
pct exec 100 -- docker ps -a
pct exec 100 -- docker start meshmonitor
```
## Notes
- The `reset-admin.mjs` script generates random passwords; use direct DB update for specific passwords
- SQLite3 CLI is not installed in the container; use Node.js with `better-sqlite3`
- Password column is `password_hash` in the `users` table (lowercase)
- **Shell escaping gotcha:** Bcrypt hashes contain `$` characters (e.g., `$2b$10$...`) which get interpreted by the shell. Always use single quotes around the outer SSH command to preserve them, or generate the hash inside the container using `bcrypt.hashSync()` directly
---
*Created: 2026-02-04*

267
runbooks/proxmox-create-ubuntu-vm.md Normal file
View file

@ -0,0 +1,267 @@
# Proxmox — Create Ubuntu VM (Cloud-Init)
Automated VM creation using Ubuntu cloud images. No interactive installer needed.
## Prerequisites
- SSH access to the target Proxmox host (directly or via jump box)
- Headscale running on Contabo with a valid preauth key
- Target Proxmox host has sufficient resources (check with `pvesm status`, `free -h`, `nproc`)
## Variables — Prompt the User
**Before executing any steps, prompt the user for ALL of the following values.** Present them one group at a time. Suggest defaults in parentheses where noted. Do not proceed until all values are confirmed.
### Group 1 — Identity
Prompt for these first:
- **VM name** — hostname for the VM (e.g., `cortex`)
- **Proxmox host** — which node to create it on? (e.g., `toc`, `data`, `utility`)
### Group 2 — Networking
Once identity is set, prompt:
- **VMID** — 150-199 range per convention. Suggest next available by checking `qm list` on the host.
- **Static IP** — suggest matching VMID (e.g., VMID 150 → 192.168.1.150). Verify it's not already in use.
- **Gateway** — (default: `192.168.1.1`)
### Group 3 — Resources
Prompt for hardware allocation. Check host resources first (`nproc`, `free -h`, `pvesm status`) and suggest reasonable values:
- **CPU cores/threads** — how many to allocate?
- **RAM (MB)** — how much?
- **Disk (GB)** — how large?
### Group 4 — Features
Prompt yes/no for each:
- **GPU passthrough?** — if yes, detect PCI address automatically via `lspci -nn | grep -i nvidia` on the host. Requires IOMMU+VFIO already configured.
- **Install Docker?**
- **Install NVIDIA drivers?** — only relevant if GPU passthrough is yes.
- **Install Node.js?** — for Claude Code.
- **Register with Tailscale?** — if yes, Tailscale hostname defaults to VM name.
### Summary
After collecting all values, present a summary table and ask for confirmation before executing.
## Step 1 — Download Ubuntu Cloud Image
Check if image already exists on the host. Only download if missing.
```bash
ssh root@$PVE_HOST 'ls /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img 2>/dev/null \
|| wget -P /var/lib/vz/template/iso/ \
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img'
```
## Step 2 — Create VM
```bash
ssh root@$PVE_HOST "qm create $VMID \
--name $VM_NAME \
--memory $RAM_MB \
--cores $CORES \
--cpu cputype=host \
--scsihw virtio-scsi-single \
--net0 virtio,bridge=vmbr0 \
--ostype l26 \
--bios ovmf \
--efidisk0 local-lvm:1 \
--machine q35 \
--agent enabled=1 \
--onboot 1"
```
## Step 3 — Import Cloud Image as Disk
```bash
ssh root@$PVE_HOST "qm importdisk $VMID /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img local-lvm"
# Attach the imported disk (disk name is vm-$VMID-disk-1)
ssh root@$PVE_HOST "qm set $VMID --scsi0 local-lvm:vm-${VMID}-disk-1,iothread=1,discard=on"
# Resize
ssh root@$PVE_HOST "qm resize $VMID scsi0 ${DISK_GB}G"
```
## Step 4 — Configure Cloud-Init
```bash
# Add cloud-init drive
ssh root@$PVE_HOST "qm set $VMID --ide2 local-lvm:cloudinit"
# Ensure SSH keys exist on the Proxmox host
# If not, pull from another node:
ssh root@$PVE_HOST 'test -f /root/.ssh/authorized_keys || echo "ERROR: No SSH keys on host"'
# Configure cloud-init
ssh root@$PVE_HOST "qm set $VMID \
--ciuser zvx \
--cipassword temp-change-me \
--ipconfig0 ip=${VM_IP}/24,gw=${GATEWAY} \
--nameserver 1.1.1.1 \
--searchdomain echo6.co \
--sshkeys /root/.ssh/authorized_keys \
--boot order=scsi0"
```
## Step 5 — GPU Passthrough (if enabled)
Skip if `GPU_PASSTHROUGH=no`.
Requires IOMMU and VFIO already configured on the Proxmox host. Verify first:
```bash
ssh root@$PVE_HOST 'dmesg | grep -i "IOMMU enabled"'
ssh root@$PVE_HOST "lspci -nnk -s $GPU_PCI_ADDR | grep 'Kernel driver in use: vfio-pci'"
```
If both check out:
```bash
ssh root@$PVE_HOST "qm set $VMID --hostpci0 ${GPU_PCI_ADDR},pcie=1,x-vga=0"
```
If VFIO is NOT configured, stop and follow the IOMMU/VFIO setup procedure before continuing.
## Step 6 — Start VM and Wait for Boot
```bash
ssh root@$PVE_HOST "qm config $VMID"
ssh root@$PVE_HOST "qm start $VMID"
echo "Waiting for VM to boot and run cloud-init..."
sleep 60
until ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no zvx@$VM_IP 'hostname' 2>/dev/null; do
echo "Still waiting..."
sleep 15
done
```
If the VM doesn't come up after 3 minutes, check the console via Proxmox web UI or:
```bash
ssh root@$PVE_HOST "qm terminal $VMID"
```
## Step 7 — Base System Setup
```bash
ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y \
curl wget git htop iotop tmux vim \
rsync tree jq unzip \
net-tools dnsutils \
python3 python3-pip python3-venv \
sudo'
```
## Step 8 — SSH Keys
```bash
# Copy authorized_keys from an existing node
scp root@data:/home/zvx/.ssh/authorized_keys /tmp/ak 2>/dev/null \
|| scp root@utility:/home/zvx/.ssh/authorized_keys /tmp/ak 2>/dev/null
scp /tmp/ak zvx@$VM_IP:~/.ssh/authorized_keys
ssh zvx@$VM_IP 'chmod 600 ~/.ssh/authorized_keys'
rm -f /tmp/ak
```
## Step 9 — NVIDIA Drivers (if GPU passthrough)
Skip if `INSTALL_NVIDIA=no`.
```bash
ssh zvx@$VM_IP 'lspci | grep -i nvidia'
# Install driver
ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y nvidia-driver-550'
ssh zvx@$VM_IP 'sudo reboot'
sleep 60
until ssh -o ConnectTimeout=5 zvx@$VM_IP 'hostname' 2>/dev/null; do
echo "Waiting for reboot..."
sleep 15
done
ssh zvx@$VM_IP 'nvidia-smi'
```
Verify: `nvidia-smi` should show the GPU name, driver version, and VRAM.
## Step 10 — Docker (if enabled)
Skip if `INSTALL_DOCKER=no`.
```bash
ssh zvx@$VM_IP 'curl -fsSL https://get.docker.com | sh'
ssh zvx@$VM_IP 'sudo usermod -aG docker zvx'
```
### NVIDIA Container Toolkit (only if GPU + Docker)
```bash
ssh zvx@$VM_IP 'curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && \
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed "s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g" | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list'
ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit'
ssh zvx@$VM_IP 'sudo nvidia-ctk runtime configure --runtime=docker'
ssh zvx@$VM_IP 'sudo systemctl restart docker'
# Test
ssh zvx@$VM_IP 'docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu24.04 nvidia-smi'
```
## Step 11 — Node.js (if enabled)
Skip if `INSTALL_NODEJS=no`.
```bash
ssh zvx@$VM_IP 'curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && \
sudo apt-get install -y nodejs'
```
## Step 12 — Tailscale Registration
Generate a preauth key on Contabo first:
```bash
docker exec headscale-standby headscale preauthkeys create --user echo6 --reusable --expiration 72h
```
Then register the VM:
```bash
ssh zvx@$VM_IP 'curl -fsSL https://tailscale.com/install.sh | sh'
ssh zvx@$VM_IP 'sudo systemctl enable tailscaled && sudo systemctl start tailscaled'
ssh zvx@$VM_IP "sudo tailscale up --login-server https://vpn.echo6.co --auth-key <KEY> --hostname $TAILSCALE_HOSTNAME"
# Verify
ssh zvx@$VM_IP 'tailscale status'
docker exec headscale-standby headscale nodes list
```
## Step 13 — Final Verification
```bash
ssh zvx@$VM_IP "
echo '=== Hostname ===' && hostname
echo '=== IP ===' && ip -4 addr show | grep 'inet 192'
echo '=== Kernel ===' && uname -r
echo '=== GPU ===' && (nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader 2>/dev/null || echo 'No GPU')
echo '=== Docker ===' && (docker --version 2>/dev/null || echo 'Not installed')
echo '=== Docker GPU ===' && (docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu24.04 nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'N/A')
echo '=== Tailscale ===' && tailscale status
echo '=== Node.js ===' && (node --version 2>/dev/null || echo 'Not installed')
echo '=== Python ===' && python3 --version
echo '=== Disk ===' && df -h /
"
docker exec headscale-standby headscale nodes list
```
## Post-Creation
1. Update `~/.claude/docs/infrastructure/environment.md` with the new VM's IP and Tailscale IP
2. Update `~/.claude/docs/infrastructure/services.md` once services are deployed
3. Remove the cloud image ISO if disk space is tight: `ssh root@$PVE_HOST 'rm /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img'`
4. Change the default password: `ssh zvx@$VM_IP 'passwd'`

101
runbooks/utility-caddy-initial-setup.md Executable file
View file

@ -0,0 +1,101 @@
# Utility Caddy LXC — Initial Setup
One-time setup. Only needed if rebuilding from scratch.
## Overview
| Item | Value |
|------|-------|
| CT ID | 101 |
| Hostname | caddy |
| Local IP | 192.168.1.101 |
| Tailscale IP | 100.64.0.2 |
| Public access | 199.6.36.163 (router forwards 80/443) |
## 1. Create LXC
```bash
ssh root@192.168.1.241
pct create 101 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname caddy \
--cores 1 \
--memory 512 \
--swap 256 \
--rootfs local-lvm:8 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.101/24,gw=192.168.1.1 \
--features nesting=1 \
--unprivileged 1 \
--password <from .ref/credentials>
# TUN device for Tailscale
cat >> /etc/pve/lxc/101.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 101
```
## 2. Install Tailscale
```bash
pct exec 101 -- bash -c "
echo nameserver 1.1.1.1 > /etc/resolv.conf
apt-get update && apt-get install -y curl
curl -fsSL https://tailscale.com/install.sh | sh
"
```
## 3. Register with Headscale
```bash
pct exec 101 -- tailscale up --login-server https://vpn.echo6.co --hostname caddy
# On Contabo — register the node
ssh root@100.64.0.6 'docker exec headscale-standby headscale nodes register --key <KEY> --user echo6'
# Verify
pct exec 101 -- tailscale status
```
## 4. Install Caddy
```bash
pct exec 101 -- bash -c "
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt | tee /etc/apt/sources.list.d/caddy-stable.list
apt-get update && apt-get install -y caddy
"
```
## 5. Install acme.sh
```bash
pct exec 101 -- bash -c "
curl https://get.acme.sh | sh -s email=admin@echo6.co
"
```
## 6. Create initial Caddyfile
```bash
pct exec 101 -- bash -c "cat > /etc/caddy/Caddyfile << 'EOF'
{
email admin@echo6.co
}
EOF
systemctl enable caddy
systemctl start caddy"
```
## 7. Router port forward
Forward on your router:
- TCP 80 → 192.168.1.101:80
- TCP 443 → 192.168.1.101:443
## Done
Add services using the expose-service-home.md runbook.

222
runbooks/vaultwarden-deployment.md Normal file
View file

@ -0,0 +1,222 @@
# Vaultwarden Deployment
**Deployed:** 2026-02-05
**Location:** Contabo VPS (5.189.158.149 / 100.64.0.6)
**URL:** https://vault.echo6.co
---
## Service Details
| Setting | Value |
|---------|-------|
| Container | `vaultwarden` |
| Image | `vaultwarden/server:latest` |
| Port | `127.0.0.1:8086` (web), `127.0.0.1:3012` (websocket) |
| Data | `/opt/vaultwarden/data` |
| Config | `/opt/vaultwarden/.env` |
| SSO | Authentik (enabled) |
| Signups | Disabled (invite-only) |
---
## Access
| Method | URL |
|--------|-----|
| Web Vault | https://vault.echo6.co |
| Admin Panel | https://vault.echo6.co/admin |
| SSO Login | "Enterprise Single Sign-On" button |
---
## Configuration Files
### Docker Compose (`/opt/vaultwarden/docker-compose.yml`)
```yaml
services:
vaultwarden:
image: vaultwarden/server:latest
container_name: vaultwarden
restart: unless-stopped
env_file:
- .env
ports:
- "127.0.0.1:8086:80"
- "127.0.0.1:3012:3012"
volumes:
- ./data:/data
environment:
- TZ=America/Boise
```
### Environment (`.env`)
```bash
# Admin
ADMIN_TOKEN=<see credentials file>
DOMAIN=https://vault.echo6.co
# Security
SIGNUPS_ALLOWED=false
INVITATIONS_ALLOWED=true
SHOW_PASSWORD_HINT=false
# WebSocket
WEBSOCKET_ENABLED=true
# SSO (Authentik)
SSO_ENABLED=true
SSO_ONLY=false
SSO_CLIENT_ID=vaultwarden
SSO_CLIENT_SECRET=<see credentials file>
SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/
SSO_PKCE=true
SSO_SCOPES="openid email profile offline_access"
# Timezone
TZ=America/Boise
LOG_LEVEL=info
```
### Caddy Site Block
```caddyfile
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### dnsmasq Split DNS
```conf
address=/vault.echo6.co/100.64.0.6
```
---
## Authentik SSO Configuration
### Provider Settings (pk=3)
| Setting | Value |
|---------|-------|
| Name | Vaultwarden |
| Client ID | `vaultwarden` |
| Client Type | Confidential |
| Redirect URI | `https://vault.echo6.co/identity/connect/oidc-signin` |
| Signing Key | authentik Internal JWT Certificate (RS256) |
| Access Token Validity | 1 hour |
| Refresh Token Validity | 30 days |
### Scopes
- `openid` - Required for OIDC
- `email` - User email
- `profile` - User profile
- `offline_access` - Refresh tokens
### OIDC Endpoints
| Endpoint | URL |
|----------|-----|
| Discovery | https://auth.echo6.co/application/o/vaultwarden/.well-known/openid-configuration |
| JWKS | https://auth.echo6.co/application/o/vaultwarden/jwks/ |
| Authorize | https://auth.echo6.co/application/o/authorize/ |
| Token | https://auth.echo6.co/application/o/token/ |
---
## Troubleshooting
### SSO Login Loop
**Symptom:** After SSO auth, redirects back to login screen.
**Causes:**
1. Access token too short (< 5 min)
2. Missing `offline_access` scope (no refresh token)
3. Missing signing key (empty JWKS)
**Fix:**
```bash
# Check Authentik provider settings via ak shell
docker exec authentik-server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.get(name='Vaultwarden')
print(f'Access Token: {p.access_token_validity}')
print(f'Signing Key: {p.signing_key}')
print(f'Scopes: {list(p.property_mappings.values_list(\"scope_name\", flat=True))}')"
```
### SSO Discovery Error
**Symptom:** "Failed to discover OpenID provider: Failed to parse server response"
**Causes:**
1. Empty JWKS endpoint (no signing key)
2. Missing property mappings
**Fix:** Add signing key and scopes to Authentik provider.
### View Logs
```bash
# Vaultwarden
docker logs vaultwarden --tail 100 2>&1 | grep -i -E "sso|error"
# Authentik
docker logs authentik-server --tail 100 2>&1 | grep -i vaultwarden
```
---
## Maintenance
### Restart Service
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose restart
```
### Update Image
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose pull
docker compose up -d
```
### Backup Data
```bash
# Stop container first
docker compose stop
tar -czf vaultwarden-backup-$(date +%Y%m%d).tar.gz data/
docker compose start
```
---
## Credentials Reference
All credentials stored in `/home/zvx/projects/.ref/credentials`:
```
VAULTWARDEN_URL
VAULTWARDEN_ADMIN_TOKEN
VAULTWARDEN_ADMIN_URL
VAULTWARDEN_OIDC_PROVIDER_ID
VAULTWARDEN_OIDC_CLIENT_ID
VAULTWARDEN_OIDC_CLIENT_SECRET
VAULTWARDEN_OIDC_ISSUER
```
---
*Last updated: 2026-02-05*