matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit: infrastructure documentation

Browse source 
Includes:
- Hardware environment reference (Proxmox cluster, VMs, LXCs)
- Services inventory with current deployments
- Caddy & DNS configuration reference
- Runbooks for common deployment procedures

Recent additions:
- SearXNG deployment (utility CT 102, search.echo6.co)
- TOC conversion to Proxmox with cortex VM
- Syncthing sync between Contabo and cortex

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-02-06 21:27:29 +01:00
commit 880ff09c90
14 changed files with 1986 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

107
docs/hardware/environment.md Normal file
View file

@ -0,0 +1,107 @@
# Echo6 Environment Reference
## Proxmox Cluster (echo6-cluster)
Five nodes running Proxmox VE:
| Node | Local IP | Tailscale | Hardware | Purpose |
|------|----------|-----------|----------|---------|
| data | 192.168.1.240 | 100.64.0.6 | Mini PC | Database services |
| utility | 192.168.1.241 | 100.64.0.5 | Mini PC | Utility services, monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | Mini PC | Cloud storage, personal services |
| media | 192.168.1.243 | 100.64.0.3 | Mini PC | Media server, *arr stack |
| toc | 192.168.1.244 | 100.64.0.13 | Workstation | GPU compute, AI/ML workloads |
### TOC Node Details
- **Hardware:** Intel i9-10900X (20 threads), 48GB RAM, 512GB NVMe, RTX A4000
- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
- **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
## Virtual Machines
| VM | Host | VMID | Local IP | Tailscale | Purpose |
|----|------|------|----------|-----------|---------|
| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — LLMs, ARGUS, Aurora, model training |
### cortex VM Details
- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
- **Resources:** 16 threads, 32GB RAM, 300GB disk
- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
- **User:** zvx (sudo, SSH keys from cluster)
- **Claude Code:** v2.1.34 installed
## Key Servers
| Server | Local IP | Tailscale | Purpose |
|--------|----------|-----------|---------|
| aida-nebra | 192.168.1.253 | 100.64.0.9 | Meshtastic node (meshtasticd on Pi) |
| matt-desktop | — | 100.64.0.10 | Personal workstation |
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge |
## LXC Containers
| Container | Host | Local IP | Tailscale | Purpose |
|-----------|------|----------|-----------|---------|
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring |
| caddy | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
| searxng | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | SearXNG metasearch engine |
## IP Allocation Scheme
| Range | Purpose |
|-------|---------|
| .1-.10 | Network infrastructure |
| .11-.99 | DHCP clients |
| .100-.149 | LXC containers |
| .150-.199 | VMs |
| .240-.250 | Proxmox hosts + bare metal |
| .251-.254 | Meshtastic nodes |
Full details: `/home/zvx/projects/utility/ip-allocation.md`
## Headscale Node List
Current registered nodes (12 total):
| Node | Tailscale IP | Type |
|------|-------------|------|
| contabo | 100.64.0.1 | VPS |
| media | 100.64.0.3 | Proxmox |
| cloud | 100.64.0.4 | Proxmox |
| utility | 100.64.0.5 | Proxmox |
| data | 100.64.0.6 | Proxmox |
| meshmonitor | 100.64.0.7 | LXC |
| caddy | 100.64.0.8 | LXC |
| aida-nebra | 100.64.0.9 | Pi |
| matt-desktop | 100.64.0.10 | Desktop |
| toc | 100.64.0.13 | Proxmox |
| cortex | 100.64.0.14 | VM |
| searxng | 100.64.0.15 | LXC |
## SSH Access
**Standard user:** `zvx`
**Credentials:** Source from `/home/zvx/projects/.ref/credentials`
```bash
# SSH to any server
ssh zvx@<ip-address>
# Examples
ssh zvx@192.168.1.244 # TOC (Proxmox host)
ssh zvx@192.168.1.150 # cortex VM
ssh zvx@192.168.1.241 # utility Proxmox
ssh root@100.64.0.1 # Contabo (via Tailscale)
ssh zvx@cortex # cortex via Tailscale hostname
```
## Key External IPs
| Purpose | IP |
|---------|-----|
| Home external (public services) | 199.6.36.163 |
| Contabo VPS | 5.189.158.149 |

68
docs/services/services.md Normal file
View file

@ -0,0 +1,68 @@
# Current Services Inventory
## Active Services
| Service | Location | IP:Port | Access | Notes |
|---------|----------|---------|--------|-------|
| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 | https://mesh.echo6.co | Meshtastic mesh monitoring |
| Utility Caddy | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
| SearXNG | utility (CT 102) | 192.168.1.102:8080 | https://search.echo6.co | Metasearch engine (Docker) |
| meshtasticd | aida-nebra | 192.168.1.253:4403 | Internal | Software Meshtastic node |
| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider |
| Forge | Contabo | 5.189.158.149 | https://forge.echo6.co | Git server |
| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
| Vaultwarden | Contabo | 127.0.0.1:8086 | https://vault.echo6.co | Password manager (SSO enabled) |
| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
## Services by Server
### toc - Proxmox Host (192.168.1.244 / Tailscale: 100.64.0.13)
- Proxmox VE node (echo6-cluster)
- GPU passthrough host for cortex VM
- No direct services — workloads run on cortex VM
### cortex - VM 150 on toc (192.168.1.150 / Tailscale: 100.64.0.14)
- GPU compute VM (RTX A4000)
- Claude Code host
- Syncthing (syncs with Contabo)
- **Planned:** Ollama, Open-WebUI, LiteLLM, ARGUS, Aurora
### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7)
- MeshMonitor (port 8080)
### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8)
- Utility Caddy (reverse proxy for VPN-only services)
### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
- SearXNG metasearch engine (port 8080)
- Redis/Valkey cache
- Compose path: `/opt/searxng/docker-compose.yml`
### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9)
- meshtasticd (software Meshtastic node)
### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
- Authentik (SSO)
- Forge (Git)
- Headscale (mesh VPN)
- Mailcow (email)
- Vaultwarden (passwords)
- Syncthing (syncs with cortex)
## Adding New Services
When deploying a new service, update this file with:
1. Service name
2. Host location (server + container if applicable)
3. IP:Port
4. Access method (internal only vs public URL)
5. Brief description
## Naming Conventions
- **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP
- **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy

77
docs/software/authentik.md Normal file
View file

@ -0,0 +1,77 @@
# Authentik SSO Configuration
## Location
- **Server:** Contabo (5.189.158.149 / 100.64.0.6)
- **URL:** https://auth.echo6.co
- **Internal Port:** 9000
## API Access
API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN`
## Flow UUIDs
Required for OAuth2 provider creation:
| Flow | UUID |
|------|------|
| Authorization (implicit) | `86051292-389f-4bd9-b0f9-53cd32f197fd` |
| Authorization (explicit) | `6f9f5c89-9f98-4776-9e0d-a72a8ad17963` |
| Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` |
| Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` |
## Create New API Token
```bash
ssh root@100.64.0.6 'docker exec authentik-server ak shell -c "
from authentik.core.models import Token, User
user = User.objects.get(username=\"akadmin\")
token, created = Token.objects.get_or_create(
identifier=\"token-name\",
user=user,
defaults={\"intent\": \"api\", \"expiring\": False}
)
print(token.key)
"'
```
## Quick OAuth2 Provider Creation
```bash
# Source credentials
source /home/zvx/projects/.ref/credentials
# Create provider
curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd",
"invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a",
"client_type": "confidential",
"client_id": "appname",
"redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}],
"sub_mode": "user_username"
}'
# Create application (use pk from provider response)
curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"slug": "appname",
"provider": PROVIDER_PK,
"meta_launch_url": "https://app.echo6.co"
}'
```
## Common Redirect URI Patterns
| Application Type | Redirect URI Pattern |
|------------------|---------------------|
| Web app | `https://app.echo6.co/callback` |
| Web app (oauth) | `https://app.echo6.co/oauth/callback` |
| Caddy forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` |

162
docs/software/caddy.md Normal file
View file

@ -0,0 +1,162 @@
# Caddy & DNS Reference
## Contabo Caddy
**Config:** `/etc/caddy/Caddyfile` on Contabo (ssh root@100.64.0.1)
### Current Site Blocks
| Domain | Backend | Service |
|--------|---------|---------|
| auth.echo6.co | 127.0.0.1:9000 | Authentik SSO |
| forge.echo6.co | 127.0.0.1:3001 | Forgejo Git |
| mail.echo6.co | https://127.0.0.1:8443 | Mailcow (tls_insecure_skip_verify) |
| vpn.echo6.co | 127.0.0.1:8084 | Headscale |
| vpn.echo6.co/admin* | 127.0.0.1:3100 | Headplane |
| autodiscover.echo6.co | https://127.0.0.1:8443 | Mailcow autodiscover |
| autoconfig.echo6.co | https://127.0.0.1:8443 | Mailcow autoconfig |
| vault.echo6.co | 127.0.0.1:8086 | Vaultwarden |
| proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) |
### Commands
```bash
ssh root@100.64.0.1
caddy validate --config /etc/caddy/Caddyfile
systemctl restart caddy # admin off, so reload won't work
journalctl -u caddy -f
```
---
## Utility Caddy (Home)
**Location:** CT 101 on utility Proxmox (192.168.1.101)
**Tailscale IP:** 100.64.0.8
**Config:** `/etc/caddy/Caddyfile` inside CT 101
**SSL Certs:** `/etc/caddy/certs/` (managed by acme.sh)
**Port forward:** Router 80/443 → 192.168.1.101
### Current Site Blocks
| Domain | Backend | Pattern | Service |
|--------|---------|---------|---------|
| mesh.echo6.co | 100.64.0.7:8080 | Tailscale | MeshMonitor |
| search.echo6.co | 100.64.0.15:8080 | Tailscale | SearXNG |
### Commands
```bash
ssh root@192.168.1.241 'pct exec 101 -- cat /etc/caddy/Caddyfile'
ssh root@192.168.1.241 'pct exec 101 -- systemctl reload caddy'
ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
```
---
## dnsmasq (Tailscale Split DNS)
**Config:** `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo
**Listens on:** 100.64.0.1:53
### Current Records
| Domain | Tailscale IP | Service |
|--------|-------------|---------|
| auth.echo6.co | 100.64.0.1 | Authentik |
| forge.echo6.co | 100.64.0.1 | Forgejo |
| mail.echo6.co | 100.64.0.1 | Mailcow |
| vpn.echo6.co | 100.64.0.1 | Headscale |
| vault.echo6.co | 100.64.0.1 | Vaultwarden |
| docs.echo6.co | 100.64.0.1 | Wiki.js |
| proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) |
| stream.echo6.co | *TBD* | PeerTube - needs host verification |
| notes.echo6.co | *TBD* | Obsidian LiveSync - needs host verification |
### Commands
```bash
ssh root@100.64.0.1
nano /etc/dnsmasq.d/tailscale-dns.conf
systemctl restart dnsmasq
dig +short forge.echo6.co @100.64.0.1 # Test
```
---
## GoDaddy DNS Records (echo6.co)
### Contabo Services → 5.189.158.149
| Subdomain | Service |
|-----------|---------|
| auth | Authentik SSO |
| forge | Forgejo Git |
| mail | Mailcow Email |
| vpn | Headscale VPN |
| vault | Vaultwarden |
### Home Services → 199.6.36.163
| Subdomain | Service |
|-----------|---------|
| @ | Main site |
| ai | Open WebUI |
| docs | Wiki.js |
| stream | PeerTube |
| notes | Obsidian LiveSync |
| jellyfin | Jellyfin |
| mesh | MeshMonitor |
| search | SearXNG |
### Email Records
| Type | Name | Value |
|------|------|-------|
| MX | @ | mail.echo6.co |
| CNAME | autoconfig | mail.echo6.co |
| CNAME | autodiscover | mail.echo6.co |
| TXT | @ | v=spf1 mx a:mail.echo6.co -all |
| TXT | _dmarc | v=DMARC1; p=quarantine |
| TXT | dkim._domainkey | (DKIM key) |
---
## Headscale Config
**Location:** `/opt/headscale/` on Contabo
**Data:** Named Docker volume `headscale_headscale-data`
**Config:** `/opt/headscale/config.yaml`
```yaml
dns:
base_domain: echo6.mesh
nameservers:
global:
- 1.1.1.1
oidc:
issuer: "https://auth.echo6.co/application/o/headscale/"
client_id: "headscale"
```
**Split DNS:** Configured via dnsmasq on Contabo.
**Headplane:** Deployed at `vpn.echo6.co/admin` - OIDC via Authentik. First login gets Owner.
---
## Port Map (Contabo)
| Service | Container Port | Host Binding | Public Domain |
|---------|---------------|--------------|---------------|
| Authentik | 9000 | 127.0.0.1:9000 | auth.echo6.co |
| Forgejo | 3000 | 127.0.0.1:3001 | forge.echo6.co |
| Headscale | 8080 | 127.0.0.1:8084 | vpn.echo6.co |
| Headplane | 3000 | 127.0.0.1:3100 | vpn.echo6.co/admin |
| Mailcow | 8443 | 127.0.0.1:8443 | mail.echo6.co |
| Vaultwarden | 80 | 127.0.0.1:8086 | vault.echo6.co |
| Vaultwarden WS | 3012 | 127.0.0.1:3012 | vault.echo6.co/notifications/hub |
---
*Last updated: 2026-02-06 — Added SearXNG (search.echo6.co) on utility CT 102*

64
docs/software/dns.md Normal file
View file

@ -0,0 +1,64 @@
# GoDaddy DNS Management
## Script Location
`~/bin/godaddy-dns.py`
## API Credentials
Stored in `/home/zvx/projects/.ref/credentials` as:
- `GODADDY_API_KEY`
- `GODADDY_API_SECRET`
## Key IPs for DNS Records
| Purpose | IP |
|---------|-----|
| External (home services) | `199.6.36.163` |
| Contabo Server | `5.189.158.149` |
## Managed Domains
arclightvanguard.com, echo6.co, echo6.org, happylittlellc.com, idahomesh.com, k7zvx.com, lpmesh.com, maliceinwonderland.org, matthewwayne.com, smugglersden.co, underdogs.cc
## Usage Examples
```bash
# List all domains
godaddy-dns.py list-domains
# List records for a domain
godaddy-dns.py list echo6.co
# Add A record
godaddy-dns.py add-a echo6.co www 199.6.36.163
# Add CNAME record
godaddy-dns.py add-cname echo6.co blog www.echo6.co
# Add MX record with priority
godaddy-dns.py add-mx echo6.co mail.echo6.co --priority=10
# Delete record
godaddy-dns.py delete echo6.co A www
# Configure MX for all domains
godaddy-dns.py setup-mail
```
## Common Patterns
### Point subdomain to home network
```bash
godaddy-dns.py add-a echo6.co newservice 199.6.36.163
```
### Point subdomain to Contabo
```bash
godaddy-dns.py add-a echo6.co auth 5.189.158.149
```
### Create CNAME alias
```bash
godaddy-dns.py add-cname echo6.co alias target.echo6.co
```