commit 880ff09c9052aab53f55575a1911221332ecb273 Author: Matt Johnson Date: Fri Feb 6 21:27:29 2026 +0100 Initial commit: infrastructure documentation Includes: - Hardware environment reference (Proxmox cluster, VMs, LXCs) - Services inventory with current deployments - Caddy & DNS configuration reference - Runbooks for common deployment procedures Recent additions: - SearXNG deployment (utility CT 102, search.echo6.co) - TOC conversion to Proxmox with cortex VM - Syncthing sync between Contabo and cortex Co-Authored-By: Claude Opus 4.5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..96ea0bc --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# Credentials - NEVER commit +credentials +credentials.bak +*.credentials + +# Backup files +*.bak +*.backup +*~ + +# Editor files +.vscode/ +.idea/ +*.swp +*.swo diff --git a/docs/hardware/environment.md b/docs/hardware/environment.md new file mode 100644 index 0000000..78418a1 --- /dev/null +++ b/docs/hardware/environment.md @@ -0,0 +1,107 @@ +# Echo6 Environment Reference + +## Proxmox Cluster (echo6-cluster) + +Five nodes running Proxmox VE: + +| Node | Local IP | Tailscale | Hardware | Purpose | +|------|----------|-----------|----------|---------| +| data | 192.168.1.240 | 100.64.0.6 | Mini PC | Database services | +| utility | 192.168.1.241 | 100.64.0.5 | Mini PC | Utility services, monitoring | +| cloud | 192.168.1.242 | 100.64.0.4 | Mini PC | Cloud storage, personal services | +| media | 192.168.1.243 | 100.64.0.3 | Mini PC | Media server, *arr stack | +| toc | 192.168.1.244 | 100.64.0.13 | Workstation | GPU compute, AI/ML workloads | + +### TOC Node Details + +- **Hardware:** Intel i9-10900X (20 threads), 48GB RAM, 512GB NVMe, RTX A4000 +- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host +- **VMID ranges:** 100-149 (LXC), 150-199 (VMs) +- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config + +## Virtual Machines + +| VM | Host | VMID | Local IP | Tailscale | Purpose | +|----|------|------|----------|-----------|---------| +| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — LLMs, ARGUS, Aurora, model training | + +### cortex VM Details + +- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic +- **Resources:** 16 threads, 32GB RAM, 300GB disk +- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0 +- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3 +- **User:** zvx (sudo, SSH keys from cluster) +- **Claude Code:** v2.1.34 installed + +## Key Servers + +| Server | Local IP | Tailscale | Purpose | +|--------|----------|-----------|---------| +| aida-nebra | 192.168.1.253 | 100.64.0.9 | Meshtastic node (meshtasticd on Pi) | +| matt-desktop | — | 100.64.0.10 | Personal workstation | +| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge | + +## LXC Containers + +| Container | Host | Local IP | Tailscale | Purpose | +|-----------|------|----------|-----------|---------| +| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring | +| caddy | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy | +| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | SearXNG metasearch engine | + +## IP Allocation Scheme + +| Range | Purpose | +|-------|---------| +| .1-.10 | Network infrastructure | +| .11-.99 | DHCP clients | +| .100-.149 | LXC containers | +| .150-.199 | VMs | +| .240-.250 | Proxmox hosts + bare metal | +| .251-.254 | Meshtastic nodes | + +Full details: `/home/zvx/projects/utility/ip-allocation.md` + +## Headscale Node List + +Current registered nodes (12 total): + +| Node | Tailscale IP | Type | +|------|-------------|------| +| contabo | 100.64.0.1 | VPS | +| media | 100.64.0.3 | Proxmox | +| cloud | 100.64.0.4 | Proxmox | +| utility | 100.64.0.5 | Proxmox | +| data | 100.64.0.6 | Proxmox | +| meshmonitor | 100.64.0.7 | LXC | +| caddy | 100.64.0.8 | LXC | +| aida-nebra | 100.64.0.9 | Pi | +| matt-desktop | 100.64.0.10 | Desktop | +| toc | 100.64.0.13 | Proxmox | +| cortex | 100.64.0.14 | VM | +| searxng | 100.64.0.15 | LXC | + +## SSH Access + +**Standard user:** `zvx` +**Credentials:** Source from `/home/zvx/projects/.ref/credentials` + +```bash +# SSH to any server +ssh zvx@ + +# Examples +ssh zvx@192.168.1.244 # TOC (Proxmox host) +ssh zvx@192.168.1.150 # cortex VM +ssh zvx@192.168.1.241 # utility Proxmox +ssh root@100.64.0.1 # Contabo (via Tailscale) +ssh zvx@cortex # cortex via Tailscale hostname +``` + +## Key External IPs + +| Purpose | IP | +|---------|-----| +| Home external (public services) | 199.6.36.163 | +| Contabo VPS | 5.189.158.149 | diff --git a/docs/services/services.md b/docs/services/services.md new file mode 100644 index 0000000..d384f99 --- /dev/null +++ b/docs/services/services.md @@ -0,0 +1,68 @@ +# Current Services Inventory + +## Active Services + +| Service | Location | IP:Port | Access | Notes | +|---------|----------|---------|--------|-------| +| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring | +| Utility Caddy | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services | +| SearXNG | utility (CT 102) | 192.168.1.102:8080 | https://search.echo6.co | Metasearch engine (Docker) | +| meshtasticd | aida-nebra | 192.168.1.253:4403 | Internal | Software Meshtastic node | +| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider | +| Forge | Contabo | 5.189.158.149 | https://forge.echo6.co | Git server | +| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) | +| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) | +| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server | +| Vaultwarden | Contabo | 127.0.0.1:8086 | https://vault.echo6.co | Password manager (SSO enabled) | +| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ | +| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ | +| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) | + +## Services by Server + +### toc - Proxmox Host (192.168.1.244 / Tailscale: 100.64.0.13) +- Proxmox VE node (echo6-cluster) +- GPU passthrough host for cortex VM +- No direct services — workloads run on cortex VM + +### cortex - VM 150 on toc (192.168.1.150 / Tailscale: 100.64.0.14) +- GPU compute VM (RTX A4000) +- Claude Code host +- Syncthing (syncs with Contabo) +- **Planned:** Ollama, Open-WebUI, LiteLLM, ARGUS, Aurora + +### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7) +- MeshMonitor (port 8080) + +### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8) +- Utility Caddy (reverse proxy for VPN-only services) + +### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15) +- SearXNG metasearch engine (port 8080) +- Redis/Valkey cache +- Compose path: `/opt/searxng/docker-compose.yml` + +### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9) +- meshtasticd (software Meshtastic node) + +### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1) +- Authentik (SSO) +- Forge (Git) +- Headscale (mesh VPN) +- Mailcow (email) +- Vaultwarden (passwords) +- Syncthing (syncs with cortex) + +## Adding New Services + +When deploying a new service, update this file with: +1. Service name +2. Host location (server + container if applicable) +3. IP:Port +4. Access method (internal only vs public URL) +5. Brief description + +## Naming Conventions + +- **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP +- **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy diff --git a/docs/software/authentik.md b/docs/software/authentik.md new file mode 100644 index 0000000..f5bea1e --- /dev/null +++ b/docs/software/authentik.md @@ -0,0 +1,77 @@ +# Authentik SSO Configuration + +## Location + +- **Server:** Contabo (5.189.158.149 / 100.64.0.6) +- **URL:** https://auth.echo6.co +- **Internal Port:** 9000 + +## API Access + +API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN` + +## Flow UUIDs + +Required for OAuth2 provider creation: + +| Flow | UUID | +|------|------| +| Authorization (implicit) | `86051292-389f-4bd9-b0f9-53cd32f197fd` | +| Authorization (explicit) | `6f9f5c89-9f98-4776-9e0d-a72a8ad17963` | +| Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` | +| Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` | + +## Create New API Token + +```bash +ssh root@100.64.0.6 'docker exec authentik-server ak shell -c " +from authentik.core.models import Token, User +user = User.objects.get(username=\"akadmin\") +token, created = Token.objects.get_or_create( + identifier=\"token-name\", + user=user, + defaults={\"intent\": \"api\", \"expiring\": False} +) +print(token.key) +"' +``` + +## Quick OAuth2 Provider Creation + +```bash +# Source credentials +source /home/zvx/projects/.ref/credentials + +# Create provider +curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \ + -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "AppName", + "authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd", + "invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a", + "client_type": "confidential", + "client_id": "appname", + "redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}], + "sub_mode": "user_username" + }' + +# Create application (use pk from provider response) +curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \ + -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "AppName", + "slug": "appname", + "provider": PROVIDER_PK, + "meta_launch_url": "https://app.echo6.co" + }' +``` + +## Common Redirect URI Patterns + +| Application Type | Redirect URI Pattern | +|------------------|---------------------| +| Web app | `https://app.echo6.co/callback` | +| Web app (oauth) | `https://app.echo6.co/oauth/callback` | +| Caddy forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` | diff --git a/docs/software/caddy.md b/docs/software/caddy.md new file mode 100644 index 0000000..06dd9d7 --- /dev/null +++ b/docs/software/caddy.md @@ -0,0 +1,162 @@ +# Caddy & DNS Reference + +## Contabo Caddy + +**Config:** `/etc/caddy/Caddyfile` on Contabo (ssh root@100.64.0.1) + +### Current Site Blocks + +| Domain | Backend | Service | +|--------|---------|---------| +| auth.echo6.co | 127.0.0.1:9000 | Authentik SSO | +| forge.echo6.co | 127.0.0.1:3001 | Forgejo Git | +| mail.echo6.co | https://127.0.0.1:8443 | Mailcow (tls_insecure_skip_verify) | +| vpn.echo6.co | 127.0.0.1:8084 | Headscale | +| vpn.echo6.co/admin* | 127.0.0.1:3100 | Headplane | +| autodiscover.echo6.co | https://127.0.0.1:8443 | Mailcow autodiscover | +| autoconfig.echo6.co | https://127.0.0.1:8443 | Mailcow autoconfig | +| vault.echo6.co | 127.0.0.1:8086 | Vaultwarden | +| proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) | + +### Commands + +```bash +ssh root@100.64.0.1 +caddy validate --config /etc/caddy/Caddyfile +systemctl restart caddy # admin off, so reload won't work +journalctl -u caddy -f +``` + +--- + +## Utility Caddy (Home) + +**Location:** CT 101 on utility Proxmox (192.168.1.101) +**Tailscale IP:** 100.64.0.8 +**Config:** `/etc/caddy/Caddyfile` inside CT 101 +**SSL Certs:** `/etc/caddy/certs/` (managed by acme.sh) +**Port forward:** Router 80/443 → 192.168.1.101 + +### Current Site Blocks + +| Domain | Backend | Pattern | Service | +|--------|---------|---------|---------| +| mesh.echo6.co | 100.64.0.7:8080 | Tailscale | MeshMonitor | +| search.echo6.co | 100.64.0.15:8080 | Tailscale | SearXNG | + +### Commands + +```bash +ssh root@192.168.1.241 'pct exec 101 -- cat /etc/caddy/Caddyfile' +ssh root@192.168.1.241 'pct exec 101 -- systemctl reload caddy' +ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f' +``` + +--- + +## dnsmasq (Tailscale Split DNS) + +**Config:** `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo +**Listens on:** 100.64.0.1:53 + +### Current Records + +| Domain | Tailscale IP | Service | +|--------|-------------|---------| +| auth.echo6.co | 100.64.0.1 | Authentik | +| forge.echo6.co | 100.64.0.1 | Forgejo | +| mail.echo6.co | 100.64.0.1 | Mailcow | +| vpn.echo6.co | 100.64.0.1 | Headscale | +| vault.echo6.co | 100.64.0.1 | Vaultwarden | +| docs.echo6.co | 100.64.0.1 | Wiki.js | +| proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) | +| stream.echo6.co | *TBD* | PeerTube - needs host verification | +| notes.echo6.co | *TBD* | Obsidian LiveSync - needs host verification | + +### Commands + +```bash +ssh root@100.64.0.1 +nano /etc/dnsmasq.d/tailscale-dns.conf +systemctl restart dnsmasq +dig +short forge.echo6.co @100.64.0.1 # Test +``` + +--- + +## GoDaddy DNS Records (echo6.co) + +### Contabo Services → 5.189.158.149 + +| Subdomain | Service | +|-----------|---------| +| auth | Authentik SSO | +| forge | Forgejo Git | +| mail | Mailcow Email | +| vpn | Headscale VPN | +| vault | Vaultwarden | + +### Home Services → 199.6.36.163 + +| Subdomain | Service | +|-----------|---------| +| @ | Main site | +| ai | Open WebUI | +| docs | Wiki.js | +| stream | PeerTube | +| notes | Obsidian LiveSync | +| jellyfin | Jellyfin | +| mesh | MeshMonitor | +| search | SearXNG | + +### Email Records + +| Type | Name | Value | +|------|------|-------| +| MX | @ | mail.echo6.co | +| CNAME | autoconfig | mail.echo6.co | +| CNAME | autodiscover | mail.echo6.co | +| TXT | @ | v=spf1 mx a:mail.echo6.co -all | +| TXT | _dmarc | v=DMARC1; p=quarantine | +| TXT | dkim._domainkey | (DKIM key) | + +--- + +## Headscale Config + +**Location:** `/opt/headscale/` on Contabo +**Data:** Named Docker volume `headscale_headscale-data` +**Config:** `/opt/headscale/config.yaml` + +```yaml +dns: + base_domain: echo6.mesh + nameservers: + global: + - 1.1.1.1 + +oidc: + issuer: "https://auth.echo6.co/application/o/headscale/" + client_id: "headscale" +``` + +**Split DNS:** Configured via dnsmasq on Contabo. +**Headplane:** Deployed at `vpn.echo6.co/admin` - OIDC via Authentik. First login gets Owner. + +--- + +## Port Map (Contabo) + +| Service | Container Port | Host Binding | Public Domain | +|---------|---------------|--------------|---------------| +| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co | +| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co | +| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co | +| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin | +| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co | +| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co | +| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub | + +--- + +*Last updated: 2026-02-06 — Added SearXNG (search.echo6.co) on utility CT 102* diff --git a/docs/software/dns.md b/docs/software/dns.md new file mode 100644 index 0000000..3e04e4e --- /dev/null +++ b/docs/software/dns.md @@ -0,0 +1,64 @@ +# GoDaddy DNS Management + +## Script Location + +`~/bin/godaddy-dns.py` + +## API Credentials + +Stored in `/home/zvx/projects/.ref/credentials` as: +- `GODADDY_API_KEY` +- `GODADDY_API_SECRET` + +## Key IPs for DNS Records + +| Purpose | IP | +|---------|-----| +| External (home services) | `199.6.36.163` | +| Contabo Server | `5.189.158.149` | + +## Managed Domains + +arclightvanguard.com, echo6.co, echo6.org, happylittlellc.com, idahomesh.com, k7zvx.com, lpmesh.com, maliceinwonderland.org, matthewwayne.com, smugglersden.co, underdogs.cc + +## Usage Examples + +```bash +# List all domains +godaddy-dns.py list-domains + +# List records for a domain +godaddy-dns.py list echo6.co + +# Add A record +godaddy-dns.py add-a echo6.co www 199.6.36.163 + +# Add CNAME record +godaddy-dns.py add-cname echo6.co blog www.echo6.co + +# Add MX record with priority +godaddy-dns.py add-mx echo6.co mail.echo6.co --priority=10 + +# Delete record +godaddy-dns.py delete echo6.co A www + +# Configure MX for all domains +godaddy-dns.py setup-mail +``` + +## Common Patterns + +### Point subdomain to home network +```bash +godaddy-dns.py add-a echo6.co newservice 199.6.36.163 +``` + +### Point subdomain to Contabo +```bash +godaddy-dns.py add-a echo6.co auth 5.189.158.149 +``` + +### Create CNAME alias +```bash +godaddy-dns.py add-cname echo6.co alias target.echo6.co +``` diff --git a/runbooks/contabo-configs.md b/runbooks/contabo-configs.md new file mode 100644 index 0000000..8f11a02 --- /dev/null +++ b/runbooks/contabo-configs.md @@ -0,0 +1,183 @@ +# Contabo VPS Current Configurations + +**Server:** 5.189.158.149 / 100.64.0.4 +**Last Updated:** 2026-02-05 + +--- + +## Caddy Configuration + +**File:** `/etc/caddy/Caddyfile` + +```caddyfile +# Global options +{ + email admin@echo6.co + admin off +} + +# Main Mailcow hostname +mail.echo6.co { + reverse_proxy https://127.0.0.1:8443 { + transport http { + tls_insecure_skip_verify + read_timeout 3600s + write_timeout 3600s + } + } +} + +# Autodiscover for Outlook +autodiscover.echo6.co { + reverse_proxy https://127.0.0.1:8443 { + transport http { + tls_insecure_skip_verify + } + } +} + +# Autoconfig for Thunderbird +autoconfig.echo6.co { + reverse_proxy https://127.0.0.1:8443 { + transport http { + tls_insecure_skip_verify + } + } +} + +# Headscale VPN + Headplane Admin +vpn.echo6.co { + handle /admin* { + reverse_proxy 127.0.0.1:3100 + } + handle { + reverse_proxy 127.0.0.1:8084 + } +} + +# Authentik SSO +auth.echo6.co { + reverse_proxy 127.0.0.1:9000 +} + +# Forgejo Git Forge +forge.echo6.co { + reverse_proxy 127.0.0.1:3001 +} + +# Vaultwarden Password Manager +vault.echo6.co { + reverse_proxy /notifications/hub 127.0.0.1:3012 + reverse_proxy 127.0.0.1:8086 +} +``` + +### Commands + +```bash +# Validate +caddy validate --config /etc/caddy/Caddyfile + +# Restart (admin off, so reload won't work) +systemctl restart caddy + +# Logs +journalctl -u caddy -f +``` + +--- + +## dnsmasq Split DNS Configuration + +**File:** `/etc/dnsmasq.d/tailscale-dns.conf` + +```conf +# DNSmasq config for Tailscale Split DNS +# Listen only on Tailscale interface +listen-address=100.64.0.4 +bind-interfaces + +# Upstream DNS servers +server=1.1.1.1 +server=8.8.8.8 + +# Local records for echo6.co services (route through Tailscale) +address=/forge.echo6.co/100.64.0.4 +address=/auth.echo6.co/100.64.0.4 +address=/mail.echo6.co/100.64.0.4 +address=/vpn.echo6.co/100.64.0.4 +address=/docs.echo6.co/100.64.0.4 +address=/vault.echo6.co/100.64.0.4 +address=/stream.echo6.co/100.64.0.7 +address=/notes.echo6.co/100.64.0.22 + +# Don't read /etc/hosts +no-hosts + +# Cache size +cache-size=1000 + +# Log queries for debugging +log-queries +``` + +### Commands + +```bash +# Restart +systemctl restart dnsmasq + +# Status +systemctl status dnsmasq + +# Test resolution +dig +short vault.echo6.co @100.64.0.4 +``` + +--- + +## Port Mappings Summary + +| Service | Container Port | Host Binding | Caddy Proxy | +|---------|---------------|--------------|-------------| +| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co | +| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co | +| Forgejo SSH | 22 | 0.0.0.0:2222 | Direct | +| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co | +| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin | +| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co | +| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co | +| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub | + +--- + +## DNS Records (GoDaddy → Contabo) + +| Subdomain | IP | Service | +|-----------|-----|---------| +| auth | 5.189.158.149 | Authentik | +| forge | 5.189.158.149 | Forgejo | +| mail | 5.189.158.149 | Mailcow | +| vpn | 5.189.158.149 | Headscale | +| vault | 5.189.158.149 | Vaultwarden | +| autodiscover | 5.189.158.149 | Mailcow | +| autoconfig | 5.189.158.149 | Mailcow | + +--- + +## Split DNS Mappings (Tailscale) + +| Domain | Tailscale IP | Server | +|--------|-------------|--------| +| auth.echo6.co | 100.64.0.4 | Contabo | +| forge.echo6.co | 100.64.0.4 | Contabo | +| mail.echo6.co | 100.64.0.4 | Contabo | +| vpn.echo6.co | 100.64.0.4 | Contabo | +| vault.echo6.co | 100.64.0.4 | Contabo | +| docs.echo6.co | 100.64.0.4 | Contabo | +| stream.echo6.co | 100.64.0.7 | PeerTube | +| notes.echo6.co | 100.64.0.22 | Cloud | + +--- + +*Last updated: 2026-02-05* diff --git a/runbooks/expose-service-contabo.md b/runbooks/expose-service-contabo.md new file mode 100755 index 0000000..6732f35 --- /dev/null +++ b/runbooks/expose-service-contabo.md @@ -0,0 +1,80 @@ +# Expose Service on Contabo + +## Prerequisites +- Service running in Docker on Contabo +- Port bound to `127.0.0.1` only (never `0.0.0.0`) + +## Steps + +### 1. Deploy the service + +```bash +ssh root@100.64.0.6 +mkdir -p /opt/ +# Create docker-compose.yml with port bound to 127.0.0.1: +docker compose up -d +``` + +### 2. Add DNS record + +```bash +# On TOC +source /home/zvx/projects/.ref/credentials +godaddy-dns.py add-a echo6.co 5.189.158.149 +dig +short .echo6.co @8.8.8.8 # Verify +``` + +### 3. Add Caddy site block + +```bash +ssh root@100.64.0.6 +nano /etc/caddy/Caddyfile + +# Add: +# .echo6.co { +# reverse_proxy 127.0.0.1: +# } + +caddy validate --config /etc/caddy/Caddyfile +systemctl reload caddy +``` + +### 4. Add dnsmasq split DNS entry + +```bash +ssh root@100.64.0.6 +nano /etc/dnsmasq.d/tailscale-dns.conf + +# Add: +# address=/.echo6.co/100.64.0.6 + +systemctl restart dnsmasq +``` + +### 5. Verify + +```bash +# Public +curl -I https://.echo6.co + +# Tailscale +dig +short .echo6.co @100.64.0.6 # Should return 100.64.0.6 +``` + +### 6. Update docs + +- Update `~/.claude/docs/infrastructure/caddy.md` with new site block +- Update `~/.claude/docs/infrastructure/services.md` with new service +- Add credentials to `/home/zvx/projects/.ref/credentials` if applicable + +## Checklist + +``` +□ Docker container running, port on 127.0.0.1 only +□ GoDaddy DNS → 5.189.158.149 +□ Caddy site block added and reloaded +□ dnsmasq entry added and restarted +□ Public access verified +□ Tailscale access verified +□ Docs updated +``` diff --git a/runbooks/expose-service-home.md b/runbooks/expose-service-home.md new file mode 100755 index 0000000..1aac255 --- /dev/null +++ b/runbooks/expose-service-home.md @@ -0,0 +1,107 @@ +# Expose Service on Home Network + +## Prerequisites +- Service running on a Proxmox CT/VM or bare metal +- Router forwards 80/443 to Utility Caddy (192.168.1.101) — one-time setup +- Determine pattern: does the service have Authentik OIDC? + +## Steps + +### 1. Determine backend target + +| Has OIDC? | Proxy to | Why | +|-----------|----------|-----| +| YES | Local IP (192.168.1.x:port) | Authentik SSO protects access | +| NO | Tailscale IP (100.64.0.x:port) | Only Caddy can reach backend | + +If no OIDC, service MUST have Tailscale installed and registered with Headscale first. + +### 2. Issue SSL certificate + +```bash +ssh root@192.168.1.241 + +pct exec 101 -- bash -c ' +export GD_Key="" +export GD_Secret="" +/root/.acme.sh/acme.sh --issue --dns dns_gd -d .echo6.co --server letsencrypt +' +``` + +### 3. Install certificate + +```bash +pct exec 101 -- bash -c " +mkdir -p /etc/caddy/certs +/root/.acme.sh/acme.sh --install-cert -d .echo6.co \ + --cert-file /etc/caddy/certs/.echo6.co.crt \ + --key-file /etc/caddy/certs/.echo6.co.key \ + --fullchain-file /etc/caddy/certs/.echo6.co.fullchain.crt \ + --reloadcmd 'systemctl reload caddy' + +chown -R caddy:caddy /etc/caddy/certs +chmod 600 /etc/caddy/certs/*.key +chmod 644 /etc/caddy/certs/*.crt +" +``` + +### 4. Add Caddy site block + +```bash +# WITH OIDC — local IP +pct exec 101 -- bash -c "cat >> /etc/caddy/Caddyfile << 'EOF' + +.echo6.co { + tls /etc/caddy/certs/.echo6.co.fullchain.crt /etc/caddy/certs/.echo6.co.key + reverse_proxy 192.168.1.: +} +EOF +systemctl reload caddy" + +# WITHOUT OIDC — Tailscale IP +pct exec 101 -- bash -c "cat >> /etc/caddy/Caddyfile << 'EOF' + +.echo6.co { + tls /etc/caddy/certs/.echo6.co.fullchain.crt /etc/caddy/certs/.echo6.co.key + reverse_proxy 100.64.0.: +} +EOF +systemctl reload caddy" +``` + +### 5. Add DNS record + +```bash +# On TOC +source /home/zvx/projects/.ref/credentials +godaddy-dns.py add-a echo6.co 199.6.36.163 +``` + +### 6. Update service CORS (if applicable) + +Add `https://.echo6.co` to the service's allowed origins. + +### 7. Verify + +```bash +curl -I https://.echo6.co +``` + +### 8. Update docs + +- Update `~/.claude/docs/infrastructure/caddy.md` with new site block +- Update `~/.claude/docs/infrastructure/services.md` with new service +- Add credentials to `/home/zvx/projects/.ref/credentials` if applicable + +## Checklist + +``` +□ Backend pattern chosen (OIDC → local IP, no OIDC → Tailscale IP) +□ SSL cert issued and installed via acme.sh +□ Caddy site block added to CT 101 Caddyfile +□ Caddy reloaded +□ GoDaddy DNS → 199.6.36.163 +□ CORS updated if needed +□ HTTPS access verified +□ Docs updated +``` diff --git a/runbooks/headscale-full-deployment.md b/runbooks/headscale-full-deployment.md new file mode 100755 index 0000000..94ac881 --- /dev/null +++ b/runbooks/headscale-full-deployment.md @@ -0,0 +1,406 @@ +# Headscale Full Deployment Runbook +## Nodes + Headplane + Authentik OIDC + +**Headscale location:** `/opt/headscale-vanilla` +**Container name:** `headscale-vanilla` +**Domain:** `vpn.echo6.co` +**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ` + +--- + +## PHASE 1: REGISTER CONTABO (must be first) + +```bash +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname contabo --force-reauth +``` + +Verify: +```bash +docker exec headscale-vanilla headscale nodes list +``` +**STOP if contabo doesn't appear. Do not continue.** + +--- + +## PHASE 2: REGISTER ALL LXC/CT NODES + +SSH into each container. For each one: + +```bash +# Check if tailscale is installed +which tailscale || echo "NOT INSTALLED" + +# Install if missing +curl -fsSL https://tailscale.com/install.sh | sh +``` + +Then register. **Do them in this exact order for sequential IPs:** + +```bash +# utility (will get 100.64.0.2) +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname utility --force-reauth + +# data (will get 100.64.0.3) +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname data --force-reauth + +# cloud (will get 100.64.0.4) +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname cloud --force-reauth + +# media (will get 100.64.0.5) +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname media --force-reauth + +# aida-nebra (will get 100.64.0.6) +tailscale up --login-server https://vpn.echo6.co \ + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \ + --hostname aida-nebra --force-reauth +``` + +After each, verify from Contabo: +```bash +docker exec headscale-vanilla headscale nodes list +``` + +--- + +## PHASE 3: REGISTER DESKTOP + PHONES + +**Desktop (Windows — PowerShell as Admin):** +```powershell +tailscale up --login-server https://vpn.echo6.co ` + --auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ ` + --hostname desktop --force-reauth +``` + +**Phones:** +- Open Tailscale app → Settings → Account +- Log out if needed +- Use "Custom coordination server" or "Alternate server" +- Enter: `https://vpn.echo6.co` +- Should auto-register with the tailnet + +If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device. + +--- + +## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY + +```bash +docker exec headscale-vanilla headscale nodes list +``` + +Expected output: all nodes with sequential 100.64.0.x IPs. + +Test from any node: +```bash +tailscale ping contabo +tailscale ping data +tailscale ping utility +``` + +Test magic DNS: +```bash +ping data.echo6.mesh +ping utility.echo6.mesh +``` + +--- + +## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else) + +```bash +mkdir -p /opt/headscale-vanilla/backups + +# Immediate backup +sqlite3 /opt/headscale-vanilla/data/db.sqlite \ + ".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'" + +# Set up cron for automatic backups every 6 hours, 7-day retention +crontab -e +# Add this line: +0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete +``` + +--- + +## PHASE 6: PERSISTENCE TEST + +```bash +cd /opt/headscale-vanilla +docker compose down +sleep 5 +ls -la /opt/headscale-vanilla/data/db.sqlite* +docker compose up -d +sleep 10 +docker exec headscale-vanilla headscale nodes list +``` + +**Every node must survive. If any are missing, STOP and report.** + +--- + +## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE + +This lets Tailscale clients authenticate via Authentik instead of preauth keys. + +1. Log into Authentik admin panel +2. Go to **Applications → Applications → Create with Provider** +3. Configure: + - **Application name:** Headscale + - **Slug:** `headscale` (remember this — it's part of the issuer URL) + - **Provider type:** OAuth2/OpenID Connect + - **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want) + - **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback` + - **Signing key:** Select any available key + - **Scopes:** Ensure these scope mappings are selected: + - `openid` + - `profile` + - `email` + - **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart +4. Note the **Client ID** and **Client Secret** +5. Click Submit + +--- + +## PHASE 8: CONFIGURE HEADSCALE OIDC + +Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block: + +```yaml +oidc: + only_start_if_oidc_is_available: true + issuer: "https:///application/o/headscale/" + client_id: "" + client_secret: "" + scope: ["openid", "profile", "email", "offline_access"] + pkce: + enabled: true + method: S256 + strip_email_domain: true +``` + +Replace: +- `` with your Authentik domain (e.g., `auth.echo6.co`) +- `` with the actual client ID +- `` with the actual client secret + +Restart Headscale: +```bash +cd /opt/headscale-vanilla +docker compose restart +sleep 10 +docker logs headscale-vanilla 2>&1 | tail -20 +``` + +**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.** + +Test: From any node, run: +```bash +tailscale up --login-server https://vpn.echo6.co --force-reauth +``` +It should open a browser → Authentik login → back to terminal, authenticated. + +**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.** + +--- + +## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE + +This is a SECOND application in Authentik for the web UI login. + +1. Go to **Applications → Applications → Create with Provider** +2. Configure: + - **Application name:** Headplane + - **Slug:** `headplane` + - **Provider type:** OAuth2/OpenID Connect + - **Authorization flow:** Same as before + - **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback` + - **Signing key:** Same key + - **Scopes:** `openid`, `profile`, `email` +3. Note the **Client ID** and **Client Secret** (different from Headscale's) +4. Click Submit + +--- + +## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE + +```bash +docker exec headscale-vanilla headscale apikeys create --expiration 999d +``` + +**Save this key — you need it for the Headplane config.** + +--- + +## PHASE 11: CREATE HEADPLANE CONFIG + +```bash +# Generate a cookie secret +openssl rand -hex 16 +``` + +Write `/opt/headscale-vanilla/headplane-config.yaml`: + +```yaml +server: + host: "0.0.0.0" + port: 3000 + cookie_secret: "" + cookie_secure: true + data_path: "/var/lib/headplane" + +headscale: + url: "http://headscale-vanilla:8080" + config_path: "/etc/headscale/config.yaml" + config_strict: false + +oidc: + issuer: "https:///application/o/headplane/" + client_id: "" + client_secret: "" + token_endpoint_auth_method: "client_secret_post" + headscale_api_key: "" + redirect_uri: "https://vpn.echo6.co/admin/oidc/callback" + disable_api_key_login: false + +integration: + docker: + enabled: true + container_name: "headscale-vanilla" + socket: "/var/run/docker.sock" +``` + +Replace all `` with actual values. + +--- + +## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE + +Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service: + +```yaml +services: + headscale: + # ... your existing headscale service, don't change it ... + + headplane: + image: ghcr.io/tale/headplane:latest + container_name: headplane + restart: unless-stopped + depends_on: + - headscale + ports: + - "127.0.0.1:3000:3000" + volumes: + - ./headplane-config.yaml:/etc/headplane/config.yaml:ro + - ./headplane-data:/var/lib/headplane + - ./config.yaml:/etc/headscale/config.yaml:ro + - /var/run/docker.sock:/var/run/docker.sock:ro +``` + +Start it: +```bash +cd /opt/headscale-vanilla +docker compose up -d +sleep 10 +docker logs headplane 2>&1 | tail -20 +``` + +Check for errors. Common issues: +- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml +- Can't connect to headscale → ensure `url` matches the container name and internal port +- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock + +--- + +## PHASE 13: UPDATE CADDY FOR HEADPLANE + +Add the `/admin` route to your Caddy config for `vpn.echo6.co`: + +``` +vpn.echo6.co { + handle /admin* { + reverse_proxy 127.0.0.1:3000 + } + handle { + reverse_proxy 127.0.0.1:8084 + } +} +``` + +Restart Caddy: +```bash +# Wherever your Caddy lives — adjust path as needed +docker exec caddy caddy reload --config /etc/caddy/Caddyfile +``` + +--- + +## PHASE 14: TEST HEADPLANE + +1. Browse to `https://vpn.echo6.co/admin` +2. You should see the Headplane login page +3. Click "Sign in with OIDC" → redirects to Authentik → authenticate +4. **The FIRST user to log in gets Owner permissions** +5. Verify you can see all your nodes in the UI + +If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`). + +--- + +## PHASE 15: FINAL VERIFICATION + +Run all of these from Contabo: + +```bash +# All nodes present? +docker exec headscale-vanilla headscale nodes list + +# Both containers healthy? +docker ps --format "table {{.Names}}\t{{.Status}}" + +# Headplane accessible? +curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin +# Should return 200 or 302 + +# Database backed up? +ls -la /opt/headscale-vanilla/backups/ + +# Cron running? +crontab -l | grep sqlite3 +``` + +--- + +## REPORT TEMPLATE + +After each phase, report: + +``` +Phase X complete: +- Output of headscale nodes list: +- Any errors: +- Logs (last 10 lines): +``` + +**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.** + +--- + +## KNOWN GOTCHAS + +1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts +2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize +3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings +4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first +5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited +6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs diff --git a/runbooks/meshmonitor-password-reset.md b/runbooks/meshmonitor-password-reset.md new file mode 100644 index 0000000..d234e25 --- /dev/null +++ b/runbooks/meshmonitor-password-reset.md @@ -0,0 +1,127 @@ +# MeshMonitor Admin Password Reset + +## Overview + +Reset the admin password for MeshMonitor web interface. + +| Item | Value | +|------|-------| +| Service | MeshMonitor | +| Location | utility CT 100 (192.168.1.100:8080) | +| Database | SQLite at `/data/meshmonitor.db` | + +## Prerequisites + +- SSH access to Proxmox utility node (192.168.1.241) as root +- `sshpass` installed on local machine +- `python3` with `bcrypt` module (for generating password hash) + +## Quick Reset (One-liner) + +Reset to a specific password (generates hash inside container to avoid escaping issues): + +```bash +sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no root@192.168.1.241 'pct exec 100 -- docker exec meshmonitor node -e " +const Database = require(\"better-sqlite3\"); +const bcrypt = require(\"bcrypt\"); +const db = new Database(\"/data/meshmonitor.db\"); +const hash = bcrypt.hashSync(\"7redditGold\", 10); +db.prepare(\"UPDATE users SET password_hash = ? WHERE username = ?\").run(hash, \"admin\"); +console.log(\"Password updated to 7redditGold\"); +db.close(); +"' +``` + +To use a different password, replace both instances of `7redditGold` (SSH password and new MeshMonitor password). + +## Step-by-Step Process + +### 1. SSH to Proxmox Host + +```bash +ssh root@192.168.1.241 +``` + +### 2. Enter the Container + +```bash +pct exec 100 -- bash +``` + +### 3. Option A: Use Built-in Reset Script (Random Password) + +```bash +docker exec meshmonitor node /app/reset-admin.mjs +``` + +This generates a random password - save the output. + +### 3. Option B: Set Specific Password via Node.js + +Generate a bcrypt hash first (run on any machine with Node.js): + +```bash +node -e "const bcrypt = require('bcrypt'); \ + bcrypt.hash('YOUR_PASSWORD_HERE', 10).then(h => console.log(h));" +``` + +Then update the database: + +```bash +docker exec meshmonitor node -e " +const Database = require('better-sqlite3'); +const db = new Database('/data/meshmonitor.db'); +db.prepare(\"UPDATE users SET password_hash = ? WHERE username = 'admin'\") + .run('\$2b\$10\$YOUR_HASH_HERE'); +db.close(); +" +``` + +### 4. Verify Login + +Open http://192.168.1.100:8080 and log in with: +- Username: `admin` +- Password: (your new password) + +## Troubleshooting + +### Check Current Admin User + +```bash +docker exec meshmonitor node -e " +const Database = require('better-sqlite3'); +const db = new Database('/data/meshmonitor.db'); +console.log(db.prepare(\"SELECT * FROM users WHERE username='admin'\").get()); +db.close(); +" +``` + +### List All Users + +```bash +docker exec meshmonitor node -e " +const Database = require('better-sqlite3'); +const db = new Database('/data/meshmonitor.db'); +console.log(db.prepare('SELECT id, username, is_admin, is_active FROM users').all()); +db.close(); +" +``` + +### Container Not Running + +```bash +# On Proxmox host +pct exec 100 -- docker ps -a +pct exec 100 -- docker start meshmonitor +``` + +## Notes + +- The `reset-admin.mjs` script generates random passwords; use direct DB update for specific passwords +- SQLite3 CLI is not installed in the container; use Node.js with `better-sqlite3` +- Password column is `password_hash` in the `users` table (lowercase) +- **Shell escaping gotcha:** Bcrypt hashes contain `$` characters (e.g., `$2b$10$...`) which get interpreted by the shell. Always use single quotes around the outer SSH command to preserve them, or generate the hash inside the container using `bcrypt.hashSync()` directly + +--- + +*Created: 2026-02-04* diff --git a/runbooks/proxmox-create-ubuntu-vm.md b/runbooks/proxmox-create-ubuntu-vm.md new file mode 100644 index 0000000..c010862 --- /dev/null +++ b/runbooks/proxmox-create-ubuntu-vm.md @@ -0,0 +1,267 @@ +# Proxmox — Create Ubuntu VM (Cloud-Init) + +Automated VM creation using Ubuntu cloud images. No interactive installer needed. + +## Prerequisites + +- SSH access to the target Proxmox host (directly or via jump box) +- Headscale running on Contabo with a valid preauth key +- Target Proxmox host has sufficient resources (check with `pvesm status`, `free -h`, `nproc`) + +## Variables — Prompt the User + +**Before executing any steps, prompt the user for ALL of the following values.** Present them one group at a time. Suggest defaults in parentheses where noted. Do not proceed until all values are confirmed. + +### Group 1 — Identity +Prompt for these first: +- **VM name** — hostname for the VM (e.g., `cortex`) +- **Proxmox host** — which node to create it on? (e.g., `toc`, `data`, `utility`) + +### Group 2 — Networking +Once identity is set, prompt: +- **VMID** — 150-199 range per convention. Suggest next available by checking `qm list` on the host. +- **Static IP** — suggest matching VMID (e.g., VMID 150 → 192.168.1.150). Verify it's not already in use. +- **Gateway** — (default: `192.168.1.1`) + +### Group 3 — Resources +Prompt for hardware allocation. Check host resources first (`nproc`, `free -h`, `pvesm status`) and suggest reasonable values: +- **CPU cores/threads** — how many to allocate? +- **RAM (MB)** — how much? +- **Disk (GB)** — how large? + +### Group 4 — Features +Prompt yes/no for each: +- **GPU passthrough?** — if yes, detect PCI address automatically via `lspci -nn | grep -i nvidia` on the host. Requires IOMMU+VFIO already configured. +- **Install Docker?** +- **Install NVIDIA drivers?** — only relevant if GPU passthrough is yes. +- **Install Node.js?** — for Claude Code. +- **Register with Tailscale?** — if yes, Tailscale hostname defaults to VM name. + +### Summary +After collecting all values, present a summary table and ask for confirmation before executing. + +## Step 1 — Download Ubuntu Cloud Image + +Check if image already exists on the host. Only download if missing. + +```bash +ssh root@$PVE_HOST 'ls /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img 2>/dev/null \ + || wget -P /var/lib/vz/template/iso/ \ + https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img' +``` + +## Step 2 — Create VM + +```bash +ssh root@$PVE_HOST "qm create $VMID \ + --name $VM_NAME \ + --memory $RAM_MB \ + --cores $CORES \ + --cpu cputype=host \ + --scsihw virtio-scsi-single \ + --net0 virtio,bridge=vmbr0 \ + --ostype l26 \ + --bios ovmf \ + --efidisk0 local-lvm:1 \ + --machine q35 \ + --agent enabled=1 \ + --onboot 1" +``` + +## Step 3 — Import Cloud Image as Disk + +```bash +ssh root@$PVE_HOST "qm importdisk $VMID /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img local-lvm" + +# Attach the imported disk (disk name is vm-$VMID-disk-1) +ssh root@$PVE_HOST "qm set $VMID --scsi0 local-lvm:vm-${VMID}-disk-1,iothread=1,discard=on" + +# Resize +ssh root@$PVE_HOST "qm resize $VMID scsi0 ${DISK_GB}G" +``` + +## Step 4 — Configure Cloud-Init + +```bash +# Add cloud-init drive +ssh root@$PVE_HOST "qm set $VMID --ide2 local-lvm:cloudinit" + +# Ensure SSH keys exist on the Proxmox host +# If not, pull from another node: +ssh root@$PVE_HOST 'test -f /root/.ssh/authorized_keys || echo "ERROR: No SSH keys on host"' + +# Configure cloud-init +ssh root@$PVE_HOST "qm set $VMID \ + --ciuser zvx \ + --cipassword temp-change-me \ + --ipconfig0 ip=${VM_IP}/24,gw=${GATEWAY} \ + --nameserver 1.1.1.1 \ + --searchdomain echo6.co \ + --sshkeys /root/.ssh/authorized_keys \ + --boot order=scsi0" +``` + +## Step 5 — GPU Passthrough (if enabled) + +Skip if `GPU_PASSTHROUGH=no`. + +Requires IOMMU and VFIO already configured on the Proxmox host. Verify first: + +```bash +ssh root@$PVE_HOST 'dmesg | grep -i "IOMMU enabled"' +ssh root@$PVE_HOST "lspci -nnk -s $GPU_PCI_ADDR | grep 'Kernel driver in use: vfio-pci'" +``` + +If both check out: + +```bash +ssh root@$PVE_HOST "qm set $VMID --hostpci0 ${GPU_PCI_ADDR},pcie=1,x-vga=0" +``` + +If VFIO is NOT configured, stop and follow the IOMMU/VFIO setup procedure before continuing. + +## Step 6 — Start VM and Wait for Boot + +```bash +ssh root@$PVE_HOST "qm config $VMID" +ssh root@$PVE_HOST "qm start $VMID" + +echo "Waiting for VM to boot and run cloud-init..." +sleep 60 +until ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no zvx@$VM_IP 'hostname' 2>/dev/null; do + echo "Still waiting..." + sleep 15 +done +``` + +If the VM doesn't come up after 3 minutes, check the console via Proxmox web UI or: + +```bash +ssh root@$PVE_HOST "qm terminal $VMID" +``` + +## Step 7 — Base System Setup + +```bash +ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y \ + curl wget git htop iotop tmux vim \ + rsync tree jq unzip \ + net-tools dnsutils \ + python3 python3-pip python3-venv \ + sudo' +``` + +## Step 8 — SSH Keys + +```bash +# Copy authorized_keys from an existing node +scp root@data:/home/zvx/.ssh/authorized_keys /tmp/ak 2>/dev/null \ + || scp root@utility:/home/zvx/.ssh/authorized_keys /tmp/ak 2>/dev/null + +scp /tmp/ak zvx@$VM_IP:~/.ssh/authorized_keys +ssh zvx@$VM_IP 'chmod 600 ~/.ssh/authorized_keys' +rm -f /tmp/ak +``` + +## Step 9 — NVIDIA Drivers (if GPU passthrough) + +Skip if `INSTALL_NVIDIA=no`. + +```bash +ssh zvx@$VM_IP 'lspci | grep -i nvidia' + +# Install driver +ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y nvidia-driver-550' +ssh zvx@$VM_IP 'sudo reboot' + +sleep 60 +until ssh -o ConnectTimeout=5 zvx@$VM_IP 'hostname' 2>/dev/null; do + echo "Waiting for reboot..." + sleep 15 +done + +ssh zvx@$VM_IP 'nvidia-smi' +``` + +Verify: `nvidia-smi` should show the GPU name, driver version, and VRAM. + +## Step 10 — Docker (if enabled) + +Skip if `INSTALL_DOCKER=no`. + +```bash +ssh zvx@$VM_IP 'curl -fsSL https://get.docker.com | sh' +ssh zvx@$VM_IP 'sudo usermod -aG docker zvx' +``` + +### NVIDIA Container Toolkit (only if GPU + Docker) + +```bash +ssh zvx@$VM_IP 'curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ + sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && \ + curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ + sed "s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g" | \ + sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list' + +ssh zvx@$VM_IP 'sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit' +ssh zvx@$VM_IP 'sudo nvidia-ctk runtime configure --runtime=docker' +ssh zvx@$VM_IP 'sudo systemctl restart docker' + +# Test +ssh zvx@$VM_IP 'docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu24.04 nvidia-smi' +``` + +## Step 11 — Node.js (if enabled) + +Skip if `INSTALL_NODEJS=no`. + +```bash +ssh zvx@$VM_IP 'curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && \ + sudo apt-get install -y nodejs' +``` + +## Step 12 — Tailscale Registration + +Generate a preauth key on Contabo first: + +```bash +docker exec headscale-standby headscale preauthkeys create --user echo6 --reusable --expiration 72h +``` + +Then register the VM: + +```bash +ssh zvx@$VM_IP 'curl -fsSL https://tailscale.com/install.sh | sh' +ssh zvx@$VM_IP 'sudo systemctl enable tailscaled && sudo systemctl start tailscaled' +ssh zvx@$VM_IP "sudo tailscale up --login-server https://vpn.echo6.co --auth-key --hostname $TAILSCALE_HOSTNAME" + +# Verify +ssh zvx@$VM_IP 'tailscale status' +docker exec headscale-standby headscale nodes list +``` + +## Step 13 — Final Verification + +```bash +ssh zvx@$VM_IP " + echo '=== Hostname ===' && hostname + echo '=== IP ===' && ip -4 addr show | grep 'inet 192' + echo '=== Kernel ===' && uname -r + echo '=== GPU ===' && (nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader 2>/dev/null || echo 'No GPU') + echo '=== Docker ===' && (docker --version 2>/dev/null || echo 'Not installed') + echo '=== Docker GPU ===' && (docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu24.04 nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'N/A') + echo '=== Tailscale ===' && tailscale status + echo '=== Node.js ===' && (node --version 2>/dev/null || echo 'Not installed') + echo '=== Python ===' && python3 --version + echo '=== Disk ===' && df -h / +" + +docker exec headscale-standby headscale nodes list +``` + +## Post-Creation + +1. Update `~/.claude/docs/infrastructure/environment.md` with the new VM's IP and Tailscale IP +2. Update `~/.claude/docs/infrastructure/services.md` once services are deployed +3. Remove the cloud image ISO if disk space is tight: `ssh root@$PVE_HOST 'rm /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img'` +4. Change the default password: `ssh zvx@$VM_IP 'passwd'` diff --git a/runbooks/utility-caddy-initial-setup.md b/runbooks/utility-caddy-initial-setup.md new file mode 100755 index 0000000..419bfb5 --- /dev/null +++ b/runbooks/utility-caddy-initial-setup.md @@ -0,0 +1,101 @@ +# Utility Caddy LXC — Initial Setup + +One-time setup. Only needed if rebuilding from scratch. + +## Overview + +| Item | Value | +|------|-------| +| CT ID | 101 | +| Hostname | caddy | +| Local IP | 192.168.1.101 | +| Tailscale IP | 100.64.0.2 | +| Public access | 199.6.36.163 (router forwards 80/443) | + +## 1. Create LXC + +```bash +ssh root@192.168.1.241 + +pct create 101 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \ + --hostname caddy \ + --cores 1 \ + --memory 512 \ + --swap 256 \ + --rootfs local-lvm:8 \ + --net0 name=eth0,bridge=vmbr0,ip=192.168.1.101/24,gw=192.168.1.1 \ + --features nesting=1 \ + --unprivileged 1 \ + --password + +# TUN device for Tailscale +cat >> /etc/pve/lxc/101.conf << EOF +lxc.cgroup2.devices.allow: c 10:200 rwm +lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file +EOF + +pct start 101 +``` + +## 2. Install Tailscale + +```bash +pct exec 101 -- bash -c " +echo nameserver 1.1.1.1 > /etc/resolv.conf +apt-get update && apt-get install -y curl +curl -fsSL https://tailscale.com/install.sh | sh +" +``` + +## 3. Register with Headscale + +```bash +pct exec 101 -- tailscale up --login-server https://vpn.echo6.co --hostname caddy + +# On Contabo — register the node +ssh root@100.64.0.6 'docker exec headscale-standby headscale nodes register --key --user echo6' + +# Verify +pct exec 101 -- tailscale status +``` + +## 4. Install Caddy + +```bash +pct exec 101 -- bash -c " +apt-get install -y debian-keyring debian-archive-keyring apt-transport-https +curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg +curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt | tee /etc/apt/sources.list.d/caddy-stable.list +apt-get update && apt-get install -y caddy +" +``` + +## 5. Install acme.sh + +```bash +pct exec 101 -- bash -c " +curl https://get.acme.sh | sh -s email=admin@echo6.co +" +``` + +## 6. Create initial Caddyfile + +```bash +pct exec 101 -- bash -c "cat > /etc/caddy/Caddyfile << 'EOF' +{ + email admin@echo6.co +} +EOF +systemctl enable caddy +systemctl start caddy" +``` + +## 7. Router port forward + +Forward on your router: +- TCP 80 → 192.168.1.101:80 +- TCP 443 → 192.168.1.101:443 + +## Done + +Add services using the expose-service-home.md runbook. diff --git a/runbooks/vaultwarden-deployment.md b/runbooks/vaultwarden-deployment.md new file mode 100644 index 0000000..d1c8000 --- /dev/null +++ b/runbooks/vaultwarden-deployment.md @@ -0,0 +1,222 @@ +# Vaultwarden Deployment + +**Deployed:** 2026-02-05 +**Location:** Contabo VPS (5.189.158.149 / 100.64.0.6) +**URL:** https://vault.echo6.co + +--- + +## Service Details + +| Setting | Value | +|---------|-------| +| Container | `vaultwarden` | +| Image | `vaultwarden/server:latest` | +| Port | `127.0.0.1:8086` (web), `127.0.0.1:3012` (websocket) | +| Data | `/opt/vaultwarden/data` | +| Config | `/opt/vaultwarden/.env` | +| SSO | Authentik (enabled) | +| Signups | Disabled (invite-only) | + +--- + +## Access + +| Method | URL | +|--------|-----| +| Web Vault | https://vault.echo6.co | +| Admin Panel | https://vault.echo6.co/admin | +| SSO Login | "Enterprise Single Sign-On" button | + +--- + +## Configuration Files + +### Docker Compose (`/opt/vaultwarden/docker-compose.yml`) + +```yaml +services: + vaultwarden: + image: vaultwarden/server:latest + container_name: vaultwarden + restart: unless-stopped + env_file: + - .env + ports: + - "127.0.0.1:8086:80" + - "127.0.0.1:3012:3012" + volumes: + - ./data:/data + environment: + - TZ=America/Boise +``` + +### Environment (`.env`) + +```bash +# Admin +ADMIN_TOKEN= +DOMAIN=https://vault.echo6.co + +# Security +SIGNUPS_ALLOWED=false +INVITATIONS_ALLOWED=true +SHOW_PASSWORD_HINT=false + +# WebSocket +WEBSOCKET_ENABLED=true + +# SSO (Authentik) +SSO_ENABLED=true +SSO_ONLY=false +SSO_CLIENT_ID=vaultwarden +SSO_CLIENT_SECRET= +SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/ +SSO_PKCE=true +SSO_SCOPES="openid email profile offline_access" + +# Timezone +TZ=America/Boise +LOG_LEVEL=info +``` + +### Caddy Site Block + +```caddyfile +vault.echo6.co { + reverse_proxy /notifications/hub 127.0.0.1:3012 + reverse_proxy 127.0.0.1:8086 +} +``` + +### dnsmasq Split DNS + +```conf +address=/vault.echo6.co/100.64.0.6 +``` + +--- + +## Authentik SSO Configuration + +### Provider Settings (pk=3) + +| Setting | Value | +|---------|-------| +| Name | Vaultwarden | +| Client ID | `vaultwarden` | +| Client Type | Confidential | +| Redirect URI | `https://vault.echo6.co/identity/connect/oidc-signin` | +| Signing Key | authentik Internal JWT Certificate (RS256) | +| Access Token Validity | 1 hour | +| Refresh Token Validity | 30 days | + +### Scopes + +- `openid` - Required for OIDC +- `email` - User email +- `profile` - User profile +- `offline_access` - Refresh tokens + +### OIDC Endpoints + +| Endpoint | URL | +|----------|-----| +| Discovery | https://auth.echo6.co/application/o/vaultwarden/.well-known/openid-configuration | +| JWKS | https://auth.echo6.co/application/o/vaultwarden/jwks/ | +| Authorize | https://auth.echo6.co/application/o/authorize/ | +| Token | https://auth.echo6.co/application/o/token/ | + +--- + +## Troubleshooting + +### SSO Login Loop + +**Symptom:** After SSO auth, redirects back to login screen. + +**Causes:** +1. Access token too short (< 5 min) +2. Missing `offline_access` scope (no refresh token) +3. Missing signing key (empty JWKS) + +**Fix:** +```bash +# Check Authentik provider settings via ak shell +docker exec authentik-server ak shell -c " +from authentik.providers.oauth2.models import OAuth2Provider +p = OAuth2Provider.objects.get(name='Vaultwarden') +print(f'Access Token: {p.access_token_validity}') +print(f'Signing Key: {p.signing_key}') +print(f'Scopes: {list(p.property_mappings.values_list(\"scope_name\", flat=True))}')" +``` + +### SSO Discovery Error + +**Symptom:** "Failed to discover OpenID provider: Failed to parse server response" + +**Causes:** +1. Empty JWKS endpoint (no signing key) +2. Missing property mappings + +**Fix:** Add signing key and scopes to Authentik provider. + +### View Logs + +```bash +# Vaultwarden +docker logs vaultwarden --tail 100 2>&1 | grep -i -E "sso|error" + +# Authentik +docker logs authentik-server --tail 100 2>&1 | grep -i vaultwarden +``` + +--- + +## Maintenance + +### Restart Service + +```bash +ssh root@5.189.158.149 +cd /opt/vaultwarden +docker compose restart +``` + +### Update Image + +```bash +ssh root@5.189.158.149 +cd /opt/vaultwarden +docker compose pull +docker compose up -d +``` + +### Backup Data + +```bash +# Stop container first +docker compose stop +tar -czf vaultwarden-backup-$(date +%Y%m%d).tar.gz data/ +docker compose start +``` + +--- + +## Credentials Reference + +All credentials stored in `/home/zvx/projects/.ref/credentials`: + +``` +VAULTWARDEN_URL +VAULTWARDEN_ADMIN_TOKEN +VAULTWARDEN_ADMIN_URL +VAULTWARDEN_OIDC_PROVIDER_ID +VAULTWARDEN_OIDC_CLIENT_ID +VAULTWARDEN_OIDC_CLIENT_SECRET +VAULTWARDEN_OIDC_ISSUER +``` + +--- + +*Last updated: 2026-02-05*