77b4715384
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/software/central.md vault/docs/software/conduit.md vault/runbooks/add-peertube-channel.md vault/runbooks/central-deploy-cutover.md vault/runbooks/conduit-operations.md vault/runbooks/peertube-remote-runner.md
267 lines
15 KiB
Markdown
267 lines
15 KiB
Markdown
---
|
||
title: Echo6 Environment Reference
|
||
type: reference
|
||
tags:
|
||
- proxmox
|
||
aliases: []
|
||
related:
|
||
- [[fleet-platform-baseline]]
|
||
- [[ip-allocation]]
|
||
- [[headscale-onboard-node]]
|
||
- [[proxmox-create-ubuntu-vm]]
|
||
- [[toc-cortex-pve9.2-update]]
|
||
updated: 2026-07-16
|
||
---
|
||
# Echo6 Environment Reference
|
||
|
||
## Proxmox Cluster (echo6-cluster)
|
||
|
||
Five nodes running Proxmox VE:
|
||
|
||
| Node | Local IP | Tailscale | Hardware | RAM | Purpose |
|
||
| ------- | ------------- | ----------- | ----------------------------------------------- | -------------- | -------------------------------- |
|
||
| data | 192.168.1.240 | 100.64.0.6 | AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD | 32GB DDR4-3200 | Database [[services]] |
|
||
| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility [[services]], monitoring |
|
||
| cloud | 192.168.1.242 | 100.64.0.4 | Intel i7-12700T, 512GB NVMe | 32GB DDR4-3200 | Cloud storage, personal [[services]] |
|
||
| media | 192.168.1.243 | 100.64.0.3 | Intel i7-14700T, 2x 512GB NVMe | 32GB DDR5-5600 | Media server, *arr stack |
|
||
| toc | 192.168.1.244 | 100.64.0.13 | Workstation (i9-10900X) | 64GB DDR4 | GPU compute, AI/ML workloads |
|
||
### Node Storage Details
|
||
|
||
| Node | Primary Disk | Secondary Disk |
|
||
|------|-------------|----------------|
|
||
| data | Samsung SSD 980 1TB (NVMe) | SanDisk SDSSDH3 1TB (SATA SSD) |
|
||
| utility | WD PC SN740 512GB (NVMe) | — |
|
||
| cloud | SK Hynix HFS512GEJ9X164N 512GB (NVMe) | — |
|
||
| media | 2x Intel SSDPEKNU512GZH 512GB (NVMe) | — |
|
||
| toc | 512GB NVMe | — |
|
||
|
||
### Node Hardware Identifiers
|
||
|
||
| Node | Make/Model | Serial/Service Tag |
|
||
|------|-----------|---------------------|
|
||
| data | Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) | MZ010LPV |
|
||
| utility | Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) | MJ0LZNYT |
|
||
| cloud | Lenovo ThinkCentre M70q Gen 3 (11T3000RUS) | MJ0LQCGJ |
|
||
| media | Dell OptiPlex Micro 7020 | 58SM6X3 (BIOS 1.20.0, 2025-09-04) |
|
||
| toc | HP Z4 G4 Workstation | MXL2383MVK |
|
||
|
||
**No BMC/IPMI/iDRAC on any node** (verified `dmidecode -t 38` empty on all five, 2026-07-14) — there is no remote power-cycle path for any Proxmox host. Recovery from a hung or powered-off node requires physical access.
|
||
|
||
### Network Notes
|
||
|
||
- **media NIC:** Original Intel e1000e NIC (`nic0`, MAC `e8:cf:83:20:8b:cb`) crashes under sustained NFS load — present but DOWN/unused. Sole uplink is a USB Realtek RTL8153 GbE dongle (MAC `0c:37:96:0e:e8:53`) on vmbr0.
|
||
- **Tailscale [[dns]] bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback [[dns]] exists before tailscaled starts, preventing chicken-and-egg [[dns]] resolution failures on reboot
|
||
|
||
### TOC Node Details
|
||
|
||
- **Hardware:** Intel i9-10900X (20 threads), 64GB RAM (4×8GB + 2×16GB DDR4), 512GB NVMe, RTX A4000
|
||
- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
|
||
- **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
|
||
- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
|
||
|
||
## Virtual Machines
|
||
|
||
| VM | Host | VMID | Local IP | Tailscale | Purpose |
|
||
|----|------|------|----------|-----------|---------|
|
||
| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — Open WebUI, Ollama, Qdrant, TEI, Claude Code |
|
||
| recon-vm | data | 1130 | 192.168.1.130 | 100.64.0.24 | [[recon]] + [[navi]] platforms, Files, Kiwix, geo backends |
|
||
| arr | media | 105 | 192.168.1.160 | 100.64.0.18 | ARR media automation stack (Jellyfin, Sonarr, Radarr, etc.) |
|
||
|
||
### cortex VM Details
|
||
|
||
- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
|
||
- **Resources:** 16 threads, 32GB RAM, 300GB disk
|
||
- **Swap:** 32GB swapfile (`/swapfile`), `vm.swappiness=10` (prefer RAM, swap under pressure only). Provisioned 2026-04-23 to support bursty jobs (gdal_contour, tippecanoe) alongside Docker ML containers.
|
||
- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
|
||
- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
|
||
- **Docker containers:** open-webui (8080), ollama (11434 w/ GPU), qdrant (6333), tei (8090)
|
||
- **User:** zvx (sudo, SSH keys from cluster)
|
||
- **Claude Code:** installed
|
||
|
||
### recon-vm Details
|
||
|
||
- **OS:** Ubuntu 24.04.4 LTS (cloud-init), kernel 6.8.0-110-generic
|
||
- **Resources:** 4 cores, 24GB RAM, 180GB disk
|
||
- **Software:** Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
|
||
- **Platforms:** [[recon]] (knowledge extraction pipeline, :8420) and [[navi]] (offline navigation, navi.echo6.co, :8440) with geo backends (Valhalla :8002, Nominatim :8010, Photon :2322, PostgreSQL/PostGIS :5432)
|
||
- **Systemd services:** [[recon]] (8420), recon-watchdog, kiwix (8430), nginx (8888)
|
||
- **NFS mounts:** pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
|
||
- **User:** zvx (sudo, SSH key auth)
|
||
- **Migrated from:** CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
|
||
- **Note:** Old CT 130 (LXC) has since been destroyed (verified 2026-06-15) — only VM 1130 remains.
|
||
|
||
### arr VM Details
|
||
|
||
- **OS:** Ubuntu 24.04 (cloud-init)
|
||
- **Resources:** 4 cores, 8GB RAM, 30GB disk on local-lvm
|
||
- **Software:** Docker 29.2.1, Tailscale, NFS client, sshpass, qemu-guest-agent
|
||
- **User:** zvx (sudo, SSH key from cortex)
|
||
- **NFS:** pi-nas:/export/arr → /mnt/arr (22TB, movies/tv/downloads)
|
||
- **Docker containers:** jellyfin (8096), jellyseer (5055), sonarr (8989), radarr (7878), prowlarr (9696), sabnzbd (8080), lidarr (8686/6595), navidrome (4533)
|
||
- **Docker network:** arr-net (bridge)
|
||
- **Config dirs:** /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd,lidarr}
|
||
|
||
## Key Servers
|
||
|
||
| Server | Local IP | Tailscale | Purpose |
|
||
|--------|----------|-----------|---------|
|
||
| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. [[meshai]] (CT 108) connects here via TCP |
|
||
| mt-isr | 192.168.1.141 | 100.100.0.5 (IdahoMesh) | Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI) |
|
||
| mt-burleybutte | 192.168.1.185 | — | Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN) |
|
||
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
|
||
| matt-desktop | 192.168.1.254 | 100.64.0.10 | Personal workstation (Windows, your PC) |
|
||
| ha | 192.168.1.151 | 100.64.0.16 | Home Assistant (VM 151 on cloud, Docker, home automation) |
|
||
|
||
| **edge1** (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, **mail-only** — Mailcow in CT 101; host [[caddy]] + mailcow-dnat.service; rebuilt [[2026-06-19]] |
|
||
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB — **permanent front door** for vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh/intermountainmesh |
|
||
|
||
*Last updated: [[2026-06-19]] — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other services; Headscale node `contabo` moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 ([[authentik]]), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
|
||
|
||
## LXC Containers
|
||
|
||
| Container | Host | Local IP | Tailscale | Purpose |
|
||
|-----------|------|----------|-----------|---------|
|
||
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel) |
|
||
| [[caddy]] | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
|
||
| [[searxng]] | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage ([[searxng]], echo6.co) |
|
||
| immich | cloud (CT 120) | 192.168.1.182 | 100.64.0.2 | Immich photo management |
|
||
| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
|
||
| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
|
||
| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
|
||
| [[meshai]] | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | MeshAI - LLM-powered Meshtastic assistant |
|
||
| [[archivist]] | utility (CT 118) | 192.168.1.118 | — | [[archivist]] knowledge pipeline |
|
||
| [[argus]] | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | [[argus]] - OSINT intelligence gathering platform |
|
||
| [[central]] | utility (CT 104) | 192.168.1.104 | 100.64.0.12 | Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see [[central]] |
|
||
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.17 | PeerTube video streaming — Tailscale identity is `peertube-4hve9pdr` (collision-suffixed) since ~2026-06-22; `.23` is stale/gone |
|
||
| mcc | media (CT 111) | 192.168.1.111 | 100.64.0.19 | pymc console web app ([[caddy]] + Postfix, /api+/auth+/ws → aida-nebra :8000) |
|
||
| mailcow | edge1 (CT 101) | 10.10.10.2 | — | Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + Caddy) |
|
||
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
|
||
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | WordPress for intermountainmesh.com |
|
||
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
|
||
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
|
||
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
|
||
| [[authentik]] | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
|
||
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
|
||
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19) |
|
||
|
||
> **Note (2026-06-19):** edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (`forgejo-ssh-dnat.service`) is a permanent systemd unit on edge2 host.
|
||
|
||
## IP Allocation Scheme
|
||
|
||
| Range | Purpose |
|
||
|-------|---------|
|
||
| .1-.10 | Network infrastructure |
|
||
| .11-.99 | DHCP clients |
|
||
| .100-.149 | LXC containers |
|
||
| .150-.199 | VMs |
|
||
| .240-.250 | Proxmox hosts + bare metal |
|
||
| .251-.254 | Meshtastic nodes |
|
||
|
||
Full details: [[ip-allocation]]
|
||
|
||
## Headscale Node List
|
||
|
||
Current registered nodes (25 total):
|
||
|
||
| Node | Tailscale IP | Type |
|
||
|------|-------------|------|
|
||
| contabo | 100.64.0.40 | VPS (edge1, mail-only, rebuilt 2026-06-19; formerly 100.64.0.1) |
|
||
| immich | 100.64.0.2 | LXC |
|
||
| media | 100.64.0.3 | Proxmox |
|
||
| cloud | 100.64.0.4 | Proxmox |
|
||
| utility | 100.64.0.5 | Proxmox |
|
||
| data | 100.64.0.6 | Proxmox |
|
||
| meshmonitor | 100.64.0.7 | LXC |
|
||
| caddy | 100.64.0.8 | LXC |
|
||
| aida-nebra | 100.64.0.9 | Pi |
|
||
| matt-desktop | 100.64.0.10 | Desktop |
|
||
| nextcloud | 100.64.0.11 | LXC |
|
||
| toc | 100.64.0.13 | Proxmox |
|
||
| cortex | 100.64.0.14 | VM |
|
||
| [[searxng]] | 100.64.0.15 | LXC |
|
||
| ha | 100.64.0.16 | VM (cloud VM 151 — node id 54, user echo6; home automation) |
|
||
| arr | 100.64.0.18 | VM |
|
||
| pi-nas | 100.64.0.21 | Pi |
|
||
| mesh-bridge | 100.64.0.22 | LXC |
|
||
| peertube-4hve9pdr | 100.64.0.17 | LXC (collision-suffixed identity since ~2026-06-22; formerly `peertube` at 100.64.0.23, now stale/gone) |
|
||
| [[recon]] | 100.64.0.24 | VM |
|
||
| [[argus]] | 100.64.0.25 | LXC |
|
||
| [[central]] | 100.64.0.12 | LXC (utility CT 104 — central.echo6.mesh) |
|
||
| edge2 | 100.64.0.26 | Proxmox/Contabo VPS |
|
||
| gl-a1300 | 100.64.0.29 | Router |
|
||
| bluefin | 100.64.0.30 | Desktop |
|
||
| wordpress | 100.64.0.31 | LXC |
|
||
| meshai | 100.64.0.32 | LXC |
|
||
| vaultwarden | 100.64.0.33 | LXC (edge2 CT 102) |
|
||
| forgejo | 100.64.0.34 | LXC (edge2 CT 103) — node id 46 |
|
||
| livesync | 100.64.0.35 | LXC (edge2 CT 104) — migrated 2026-06-16 |
|
||
| authentik | 100.64.0.36 | LXC (edge2 CT 105) — node id 48, migrated 2026-06-18 |
|
||
| matrix | 100.64.0.37 | LXC (edge2 CT 106) — migrated 2026-06-18 |
|
||
| headscale | 100.64.0.38 | LXC (edge2 CT 107) — migrated 2026-06-19 |
|
||
|
||
## IdahoMesh Headscale Node List
|
||
|
||
Separate Headscale instance on CT 106 (192.168.1.106), prefix 100.100.0.0/16.
|
||
Reachable from echo6 tailnet via mesh-bridge (CT 107).
|
||
|
||
| Node | Tailscale IP | User | Type |
|
||
|------|-------------|------|------|
|
||
| mesh-bridge | 100.100.0.3 | malice | LXC (bridge) |
|
||
| burley-butte | 100.100.0.1 | nebra | Pi (offline) |
|
||
| mt-isr | 100.100.0.5 | nebra | Pi Zero 2 W |
|
||
|
||
## SSH Access
|
||
|
||
**Standard user:** `zvx`
|
||
**Credentials:** Source from `/home/zvx/projects/.ref/credentials`
|
||
|
||
Most servers use SSH key auth. Exceptions noted below.
|
||
|
||
```bash
|
||
# SSH to any server (key auth)
|
||
ssh zvx@<ip-address>
|
||
|
||
# Examples
|
||
ssh zvx@192.168.1.244 # TOC (Proxmox host)
|
||
ssh zvx@192.168.1.150 # cortex VM
|
||
ssh zvx@192.168.1.241 # utility Proxmox
|
||
ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149 # edge1 (mail-only; formerly Contabo)
|
||
ssh zvx@cortex # cortex via Tailscale hostname
|
||
ssh edge2 # edge2 via SSH config alias (admin@100.64.0.26)
|
||
```
|
||
|
||
### edge2 SSH access
|
||
|
||
edge2 is hardened: `PasswordAuthentication no`, root login disabled. SSH user is `admin` with passwordless sudo. The SSH config alias `edge2` maps to `admin@100.64.0.26`.
|
||
|
||
Authorized keys on edge2 (`/home/admin/.ssh/authorized_keys`):
|
||
- `echo6-contabo2-184.174.35.153` — the original provisioning key (cortex `~/.ssh/contabo2_ed25519`)
|
||
- `cortex` — cortex default `id_ed25519`
|
||
- `matt-desktop-wsl2` — WSL2 default key
|
||
- `cookie-sync` — Windows native SSH key
|
||
|
||
To access CTs on edge2: `ssh edge2` then `sudo pct exec <vmid> -- bash`.
|
||
|
||
### Password-auth hosts
|
||
|
||
These require password authentication (no SSH keys installed):
|
||
|
||
| Host | User | Password | Access |
|
||
|------|------|----------|--------|
|
||
| aida-nebra | zvx | 7redditGold | `sshpass -p '7redditGold' ssh zvx@aida-nebra` |
|
||
| mt-isr | isr | UfPsfwyMIUIKb1 | `sshpass -p 'UfPsfwyMIUIKb1' ssh isr@192.168.1.141` |
|
||
| mt-burleybutte | bb | (see credentials) | `sshpass -p '<pw>' ssh bb@192.168.1.185` |
|
||
| matt-desktop | administrator | Qw1290opzx | `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no administrator@192.168.1.254` |
|
||
| toc | root | 7redditGold | `sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@100.64.0.13` |
|
||
|
||
Use the Tailscale hostname (`aida-nebra`) or local IP (`192.168.1.253`) — both work for aida-nebra.
|
||
mt-isr is on IdahoMesh tailnet (100.100.0.5) — reachable from echo6 via bridge.
|
||
matt-desktop is accessible via local IP (192.168.1.254) or Tailscale (100.64.0.10) — requires explicit password auth flags.
|
||
|
||
## Key External IPs
|
||
|
||
| Purpose | IP |
|
||
|---------|-----|
|
||
| Home external (public services) | 199.6.36.163 |
|
||
| edge1 (mail-only, rebuilt Contabo VPS) | 5.189.158.149 |
|
||
| edge2 (front door: vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh/intermountainmesh) | 184.174.35.153 |
|