matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
echo6-docs/vault/docs/hardware/environment.md
echo6-autocommit afaf019076 auto: docs sync 2026-08-14T12:00:03+00:00 
Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/docs/hardware/environment.md vault/docs/services/services.md vault/projects/fleet-storage-memory-upgrade.md vault/projects/navi-recon-separation.md vault/runbooks/omv-add-nfs-share.md vault/runbooks/peertube-sitemap-redis-oom.md vault/runbooks/pve-guest-park-and-adopt.md
2026-08-14 12:00:03 +00:00

17 KiB
Raw Blame History

title type tags aliases related updated
Echo6 Environment Reference reference
proxmox
fleet-platform-baseline
fleet-storage-memory-upgrade
ip-allocation
toc-cortex-pve9.2-update
proxmox-create-ubuntu-vm
2026-08-14

Echo6 Environment Reference

Proxmox Cluster (echo6-cluster)

Five nodes running Proxmox VE:

Node Local IP Tailscale Hardware RAM Purpose
data 192.168.1.240 100.64.0.6 AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD 32GB DDR4-3200 Database services
utility 192.168.1.241 100.64.0.5 AMD Ryzen 7 PRO 5750GE, 512GB NVMe 32GB DDR4-3200 Utility services, monitoring
cloud 192.168.1.242 100.64.0.4 Intel i7-12700T, 512GB NVMe 32GB DDR4-3200 Cloud storage, personal services
media 192.168.1.243 100.64.0.3 Intel i7-14700T, 2x 512GB NVMe 32GB DDR5-4800 Media server, *arr stack
toc 192.168.1.244 100.64.0.13 Workstation (i9-10900X) 64GB DDR4 GPU compute, AI/ML workloads

Node Storage Details

Node Primary Disk Secondary Disk
data Samsung SSD 980 1TB (NVMe) SanDisk SDSSDH3 1TB (SATA SSD)
utility WD PC SN740 512GB (NVMe) —
cloud SK Hynix HFS512GEJ9X164N 512GB (NVMe) —
media 2x Intel SSDPEKNU512GZH 512GB (NVMe) —
toc 512GB NVMe —

Free positions (verified live 2026-08-14): data has none — only three external PCIe root ports exist and all are populated (NVMe, NIC, Wi-Fi), and the Wi-Fi M.2 is E-keyed so it cannot take a storage drive. utility and cloud each have an empty 2.5" SATA bay. media's second M.2 holds a leftover BitLocker Windows install (serial PHKA142402U8512A; the Proxmox drive is PHKA142504HP512A). toc has 8 unpopulated SATA ports and 4 free PCIe slots. pi-nas has one free SATA port (ata5). Placement plan is fleet-storage-memory-upgrade.

Memory ceilings: data and utility are hard-capped at 32 GB and already there. cloud and media max at 64 GB. toc has 2 of 8 DIMM slots free but takes full-size DDR4 DIMMs, not SODIMM, and currently runs all modules at 2133 MT/s because rated speeds are mixed (3200/2666/2133).

Cluster Backup Storage

pinas-backup — NFS 192.168.1.245:/export/pvebackup, vers=3, content backup, mounted at /mnt/pve/pinas-backup on all five nodes. Backed by pi-nas sdc1 (~19 TB free), deliberately not sdd1 which carries PeerTube. Added 2026-08-14 for park-and-adopt guest relocation — see pve-guest-park-and-adopt and omv-add-nfs-share.

Non-Cluster Physical Hosts

Host Local IP Tailscale Hardware
pi-nas 192.168.1.245 100.64.0.21 Raspberry Pi 5, 8GB soldered, JMicron JMB58x 5-port SATA HBA, 2x 3TB btrfs RAID1 + 2x 24TB single ext4
aida-nebra 192.168.1.253 100.64.0.9 Raspberry Pi Compute Module 3, Cortex-A53, 906MiB soldered, eMMC boot, RAK4631 on ttyACM1

Neither is upgradable — soldered RAM, no DIMM sockets.

Node Hardware Identifiers

Node Make/Model Serial/Service Tag
data Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) MZ010LPV
utility Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) MJ0LZNYT
cloud Lenovo ThinkCentre M70q Gen 3 (11T3000RUS) MJ0LQCGJ
media Dell OptiPlex Micro 7020 58SM6X3 (BIOS 1.20.0, 2025-09-04)
toc HP Z4 G4 Workstation MXL2383MVK

No BMC/IPMI/iDRAC on any node (verified dmidecode -t 38 empty on all five, 2026-07-14) — there is no remote power-cycle path for any Proxmox host. Recovery from a hung or powered-off node requires physical access.

Network Notes

  • media NIC: Original Intel e1000e NIC (nic0, MAC e8:cf:83:20:8b:cb) crashes under sustained NFS load — present but DOWN/unused. Sole uplink is a USB Realtek RTL8153 GbE dongle (MAC 0c:37:96:0e:e8:53) on vmbr0.
  • Tailscale dns bootstrap: All LXC containers with Tailscale have a systemd drop-in (/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf) that ensures fallback dns exists before tailscaled starts, preventing chicken-and-egg dns resolution failures on reboot

TOC Node Details

  • Hardware: Intel i9-10900X (20 threads), 64GB RAM (4×8GB + 2×16GB DDR4), 512GB NVMe, RTX A4000
  • GPU: Passed through via VFIO to VM 150 (cortex), not used on host
  • VMID ranges: 100-149 (LXC), 150-199 (VMs)
  • Presave backup: /home/zvx/toc-presave/ on Contabo (1.8G) — contains old Ubuntu config

Virtual Machines

VM Host VMID Local IP Tailscale Purpose
cortex toc 150 192.168.1.150 100.64.0.14 GPU compute — Open WebUI, Ollama, Qdrant, TEI, Claude Code
recon-vm data 1130 192.168.1.130 100.64.0.24 recon + navi platforms, Files, Kiwix, geo backends
arr media 105 192.168.1.160 100.64.0.18 ARR media automation stack (Jellyfin, Sonarr, Radarr, etc.)

cortex VM Details

  • OS: Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
  • Resources: 16 threads, 32GB RAM, 300GB disk
  • Swap: 32GB swapfile (/swapfile), vm.swappiness=10 (prefer RAM, swap under pressure only). Provisioned 2026-04-23 to support bursty jobs (gdal_contour, tippecanoe) alongside Docker ML containers.
  • GPU: RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
  • Software: Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
  • Docker containers: open-webui (8080), ollama (11434 w/ GPU), qdrant (6333), tei (8090)
  • User: zvx (sudo, SSH keys from cluster)
  • Claude Code: installed

recon-vm Details

  • OS: Ubuntu 24.04.4 LTS (cloud-init), kernel 6.8.0-110-generic
  • Resources: 4 cores, 24GB RAM, 180GB disk
  • Software: Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
  • Platforms: recon (knowledge extraction pipeline, :8420) and navi (offline navigation, navi.echo6.co, :8440) with geo backends (Valhalla :8002, Nominatim :8010, Photon :2322, PostgreSQL/PostGIS :5432)
  • Systemd services: recon (8420), recon-watchdog, kiwix (8430), nginx (8888)
  • NFS mounts: pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
  • User: zvx (sudo, SSH key auth)
  • Migrated from: CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
  • Note: Old CT 130 (LXC) has since been destroyed (verified 2026-06-15) — only VM 1130 remains.

arr VM Details

  • OS: Ubuntu 24.04 (cloud-init)
  • Resources: 4 cores, 8GB RAM, 30GB disk on local-lvm
  • Software: Docker 29.2.1, Tailscale, NFS client, sshpass, qemu-guest-agent
  • User: zvx (sudo, SSH key from cortex)
  • NFS: pi-nas:/export/arr → /mnt/arr (22TB, movies/tv/downloads)
  • Docker containers: jellyfin (8096), jellyseer (5055), sonarr (8989), radarr (7878), prowlarr (9696), sabnzbd (8080), lidarr (8686/6595), navidrome (4533)
  • Docker network: arr-net (bridge)
  • Config dirs: /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd,lidarr}

Key Servers

Server Local IP Tailscale Purpose
aida-nebra 192.168.1.253 100.64.0.9 AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. meshai (CT 108) connects here via TCP
mt-isr 192.168.1.141 100.100.0.5 (IdahoMesh) Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI)
mt-burleybutte 192.168.1.185 — Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN)
pi-nas 192.168.1.245 100.64.0.21 Raspberry Pi NAS
matt-desktop 192.168.1.254 100.64.0.10 Personal workstation (Windows, your PC)
ha 192.168.1.151 100.64.0.16 Home Assistant (VM 151 on cloud, Docker, home automation)

| edge1 (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, mail-only — Mailcow in CT 101; host caddy + mailcow-dnat.service; rebuilt 2026-06-19 | | edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB — permanent front door for vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh.com |

Last updated: 2026-06-19 — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other services; Headscale node contabo moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 (authentik), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)

LXC Containers

Container Host Local IP Tailscale Purpose
meshmonitor utility (CT 100) 192.168.1.100 100.64.0.7 Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel)
caddy utility (CT 101) 192.168.1.101 100.64.0.8 Home reverse proxy
searxng utility (CT 102) 192.168.1.102 100.64.0.15 Echo6 Search homepage (searxng, echo6.co)
immich cloud (CT 120) 192.168.1.182 100.64.0.2 Immich photo management
nextcloud cloud (CT 121) 192.168.1.183 100.64.0.11 Nextcloud AIO
meshtastic-hs utility (CT 106) 192.168.1.106 — IdahoMesh Headscale VPN coordination
mesh-bridge utility (CT 107) 192.168.1.107 100.64.0.22 Dual-tailscaled bridge (echo6 ↔ idahomesh)
meshai utility (CT 108) 192.168.1.144 100.64.0.32 meshai - LLM-powered Meshtastic assistant
archivist utility (CT 118) 192.168.1.118 — archivist knowledge pipeline
argus utility (CT 103) 192.168.1.103 100.64.0.25 argus - OSINT intelligence gathering platform
central utility (CT 104) 192.168.1.104 100.64.0.12 Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see central
peertube media (CT 110) 192.168.1.170 100.64.0.17 PeerTube video streaming — Tailscale identity is peertube-4hve9pdr (collision-suffixed) since ~2026-06-22; .23 is stale/gone
mcc media (CT 111) 192.168.1.111 100.64.0.19 pymc console web app (caddy + Postfix, /api+/auth+/ws → aida-nebra :8000)
mailcow edge1 (CT 101) 10.10.10.2 — Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + caddy)
pdm edge2 (CT 100) 10.10.10.10 100.64.0.28 Proxmox Datacenter Manager
wordpress edge2 (CT 101) 10.10.10.11 100.64.0.31 Grav CMS 2.0.11 for idahomesh.com (migrated from WordPress 2026-07-17; hostname wordpress unchanged; MariaDB purged — flat-file, no DB)
vaultwarden edge2 (CT 102) 10.10.10.20 100.64.0.33 Vaultwarden password manager (migrated from Contabo 2026-06-16)
forgejo edge2 (CT 103) 10.10.10.21 100.64.0.34 Forgejo git server (migrated from Contabo 2026-06-16)
livesync edge2 (CT 104) 10.10.10.22 100.64.0.35 LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16)
authentik edge2 (CT 105) 10.10.10.23 100.64.0.36 authentik SSO platform (migrated from Contabo 2026-06-18)
matrix edge2 (CT 106) 10.10.10.24 100.64.0.37 Matrix stack (synapse + MAS + Element + mautrix_signal; migrated from Contabo 2026-06-18)
headscale edge2 (CT 107) 10.10.10.25 100.64.0.38 Headscale + Headplane tailnet control plane (migrated from Contabo 2026-06-19)

Note (2026-06-19): edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (forgejo-ssh-dnat.service) is a permanent systemd unit on edge2 host.

IP Allocation Scheme

Range Purpose
.1-.10 Network infrastructure
.11-.99 DHCP clients
.100-.149 LXC containers
.150-.199 VMs
.240-.250 Proxmox hosts + bare metal
.251-.254 Meshtastic nodes

Full details: ip-allocation

Headscale Node List

Current registered nodes (25 total):

Node Tailscale IP Type
contabo 100.64.0.40 VPS (edge1, mail-only, rebuilt 2026-06-19; formerly 100.64.0.1)
immich 100.64.0.2 LXC
media 100.64.0.3 Proxmox
cloud 100.64.0.4 Proxmox
utility 100.64.0.5 Proxmox
data 100.64.0.6 Proxmox
meshmonitor 100.64.0.7 LXC
caddy 100.64.0.8 LXC
aida-nebra 100.64.0.9 Pi
matt-desktop 100.64.0.10 Desktop
nextcloud 100.64.0.11 LXC
toc 100.64.0.13 Proxmox
cortex 100.64.0.14 VM
searxng 100.64.0.15 LXC
ha 100.64.0.16 VM (cloud VM 151 — node id 54, user echo6; home automation)
arr 100.64.0.18 VM
pi-nas 100.64.0.21 Pi
mesh-bridge 100.64.0.22 LXC
peertube-4hve9pdr 100.64.0.17 LXC (collision-suffixed identity since ~2026-06-22; formerly peertube at 100.64.0.23, now stale/gone)
recon 100.64.0.24 VM
argus 100.64.0.25 LXC
central 100.64.0.12 LXC (utility CT 104 — central.echo6.mesh)
edge2 100.64.0.26 Proxmox/Contabo VPS
gl-a1300 100.64.0.29 Router
bluefin 100.64.0.30 Desktop
wordpress 100.64.0.31 LXC (edge2 CT 101 — now runs Grav CMS; hostname unchanged)
meshai 100.64.0.32 LXC
vaultwarden 100.64.0.33 LXC (edge2 CT 102)
forgejo 100.64.0.34 LXC (edge2 CT 103) — node id 46
livesync 100.64.0.35 LXC (edge2 CT 104) — migrated 2026-06-16
authentik 100.64.0.36 LXC (edge2 CT 105) — node id 48, migrated 2026-06-18
matrix 100.64.0.37 LXC (edge2 CT 106) — migrated 2026-06-18
headscale 100.64.0.38 LXC (edge2 CT 107) — migrated 2026-06-19

IdahoMesh Headscale Node List

Separate Headscale instance on CT 106 (192.168.1.106), prefix 100.100.0.0/16. Reachable from echo6 tailnet via mesh-bridge (CT 107).

Node Tailscale IP User Type
mesh-bridge 100.100.0.3 malice LXC (bridge)
burley-butte 100.100.0.1 nebra Pi (offline)
mt-isr 100.100.0.5 nebra Pi Zero 2 W

SSH Access

Standard user: zvx Credentials: Source from /home/zvx/projects/.ref/credentials

Most servers use SSH key auth. Exceptions noted below.

# SSH to any server (key auth)
ssh zvx@<ip-address>

# Examples
ssh zvx@192.168.1.244  # TOC (Proxmox host)
ssh zvx@192.168.1.150  # cortex VM
ssh zvx@192.168.1.241  # utility Proxmox
ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149  # edge1 (mail-only; formerly Contabo)
ssh zvx@cortex         # cortex via Tailscale hostname
ssh edge2              # edge2 via SSH config alias (admin@100.64.0.26)

edge2 SSH access

edge2 is hardened: PasswordAuthentication no, root login disabled. SSH user is admin with passwordless sudo. The SSH config alias edge2 maps to admin@100.64.0.26.

Authorized keys on edge2 (/home/admin/.ssh/authorized_keys):

  • echo6-contabo2-184.174.35.153 — the original provisioning key (cortex ~/.ssh/contabo2_ed25519)
  • cortex — cortex default id_ed25519
  • matt-desktop-wsl2 — WSL2 default key
  • cookie-sync — Windows native SSH key

To access CTs on edge2: ssh edge2 then sudo pct exec <vmid> -- bash.

Password-auth hosts

These require password authentication (no SSH keys installed):

Host User Password Access
aida-nebra zvx 7redditGold sshpass -p '7redditGold' ssh zvx@aida-nebra
mt-isr isr UfPsfwyMIUIKb1 sshpass -p 'UfPsfwyMIUIKb1' ssh isr@192.168.1.141
mt-burleybutte bb (see credentials) sshpass -p '<pw>' ssh bb@192.168.1.185
matt-desktop administrator Qw1290opzx ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no administrator@192.168.1.254
toc root 7redditGold sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@100.64.0.13

Use the Tailscale hostname (aida-nebra) or local IP (192.168.1.253) — both work for aida-nebra. mt-isr is on IdahoMesh tailnet (100.100.0.5) — reachable from echo6 via bridge. matt-desktop is accessible via local IP (192.168.1.254) or Tailscale (100.64.0.10) — requires explicit password auth flags.

Key External IPs

Purpose IP
Home external (public services) 199.6.36.163
edge1 (mail-only, rebuilt Contabo VPS) 5.189.158.149
edge2 (front door: vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh.com) 184.174.35.153