matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
echo6-docs/AUDIT-2026-02-21.md
Matt Johnson e9231ac24a Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync 
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup)
- Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing
- Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack
- Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure
- Removes 4 deprecated runbook duplicates (canonical versions live in projects/)
- Adds .gitignore for binary archives and editor temp files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 06:02:16 +00:00

20 KiB
Raw Blame History

.ref/ Directory Audit — 2026-02-21

Auditor: Claude Code (cortex) Hostname: cortex Scope: Every file in /home/zvx/projects/.ref/ cross-referenced against live infrastructure state.

1. Dead Services to Remove

Wiki.js / docs.echo6.co — CONFIRMED DEAD

  • Evidence: No Docker container running on Contabo. docker ps -a shows no wiki-related containers.
  • Stale artifacts:
    • credentials lines 20-23: WIKIJS_ADMIN_EMAIL, WIKIJS_ADMIN_PASSWORD, WIKIJS_URL
    • credentials line 35: WIKIJS_API_TOKEN (JWT, expires 2026-11-07)
    • GoDaddy DNS: docs.echo6.co still resolves to 199.6.36.163
    • dnsmasq: docs.echo6.co still has an entry pointing to 100.64.0.1
    • docs/software/caddy.md: Still referenced under Contabo Caddy (not present in actual Caddyfile)
    • docs/software/dns.md: Still lists docs.echo6.co under home services
  • Action: Remove credentials, delete DNS record, remove dnsmasq entry, remove from caddy.md and dns.md.

Echo6 Portal — CONFIRMED DEAD

  • Evidence: Nothing listening on port 3002 on utility (192.168.1.241). No container found.
  • Stale artifacts:
    • credentials lines 60-69: ECHO6_PORTAL_* credentials (admin user, OIDC client ID/secret, issuer URL)
    • credentials line 61: References deployment at 192.168.1.241:3002
  • Action: Remove credentials. Verify Authentik provider echo6-portal doesn't still exist (clean up if so).

Stalwart Mail — Already Decommissioned (Partially Cleaned)

  • Status: Credentials correctly commented out in credentials file (lines 71-75).
  • credentials line 72: Comment says "Authentik provider PK 53 may still exist" — Verified: PK 53 does NOT exist in Authentik. The comment is now misleading.
  • Action: Update comment to state PK 53 was already cleaned, or remove the entire stale block.

2. Stale Credentials to Clean

OPENWEBUI_API_URL — Wrong IP

  • File: credentials line 52
  • Current value: OPENWEBUI_API_URL=http://192.168.1.239:3000
  • Problem: No known service at 192.168.1.239. Open WebUI runs on cortex at 192.168.1.150:8080.
  • Action: Update to http://192.168.1.150:8080 or http://100.64.0.14:8080 (Tailscale IP).

MESHMONITOR_TAILSCALE_URL — Wrong IP

  • File: credentials lines 124, 127
  • Current value: MESHMONITOR_TAILSCALE_URL="http://100.64.0.1:8080"
  • Problem: 100.64.0.1 is Contabo, not MeshMonitor. MeshMonitor runs on utility CT 100, Tailscale IP 100.64.0.7.
  • Action: Change to http://100.64.0.7:8080.

Contabo Tailscale IP in credentials — Wrong

  • File: credentials line 86
  • Current value: # Tailscale IP: 100.64.0.6
  • Problem: 100.64.0.6 is the data Proxmox node's Tailscale IP. Contabo's Tailscale IP is 100.64.0.1.
  • Action: Change comment to # Tailscale IP: 100.64.0.1.

Stale File Path Reference

  • File: credentials line 33
  • Current value: # - File Location: /home/zvx/projects/selfhosting/.credentials
  • Problem: References an old path that no longer exists. Current location is /home/zvx/projects/.ref/credentials.
  • Action: Update to current path or remove the comment.

3. Incorrect Values

Caddy Port: mail.echo6.co

  • File: docs/software/caddy.md — Contabo Caddy table
  • Documented: reverse_proxy https://127.0.0.1:8443 (port 8443)
  • Actual (live Caddyfile): reverse_proxy https://127.0.0.1:8453 (port 8453)
  • Also wrong in: Port Map table at bottom of caddy.md (shows 8443)
  • Action: Update both references from 8443 to 8453.

Contabo Tailscale IP — Multiple Files Reference 100.64.0.6 Instead of 100.64.0.1

File Line/Section Wrong Value Correct Value
credentials Line 86 100.64.0.6 100.64.0.1
projects/vaultwarden-deployment.md SSH commands 100.64.0.6 100.64.0.1
projects/authentik-oidc-application.md (if exists as runbook) SSH commands 100.64.0.6 100.64.0.1
runbooks/authentik-oidc-application.md SSH commands to Contabo 100.64.0.6 100.64.0.1
runbooks/expose-service-contabo.md Step 5 verification dig command 100.64.0.6 100.64.0.1

Root cause: 100.64.0.6 is the data Proxmox node. This was likely a copy-paste error from early infrastructure setup when Contabo's Tailscale IP may not have been settled.

Headscale Container Name — "headscale-vanilla" vs "headscale"

File Incorrect Correct
projects/headscale-full-deployment.md headscale-vanilla (throughout) headscale
runbooks/proxmox-create-ubuntu-vm.md headscale-standby (lines 245, 257, 277) headscale

Note: MEMORY.md already documents this: "Headscale container on Contabo is named headscale (not headscale-vanilla as in some runbooks)." The runbooks were never updated to match.

Utility Caddy Tailscale IP

  • File: projects/utility-caddy-initial-setup.md
  • Documented: Tailscale IP 100.64.0.2
  • Actual: 100.64.0.8 (per environment.md and live headscale node list)
  • Action: Update to 100.64.0.8.

Proxmox Onboard Node — Wrong Tailscale IPs for Cluster Nodes

  • File: runbooks/proxmox-onboard-node.md — "Current Cluster" table (lines 9-14)
  • Documented:
Node Tailscale IP (documented) Tailscale IP (actual from environment.md)
data 100.64.0.20 100.64.0.6
utility 100.64.0.19 100.64.0.5
cloud 100.64.0.22 100.64.0.4
media 100.64.0.21 100.64.0.3

Every single Tailscale IP in this table is wrong. These appear to be from a completely different assignment scheme. Additionally, the toc node (192.168.1.244 / 100.64.0.13) is missing from this table entirely.

PeerTube Rebuild — Wrong CT ID in Architecture Diagram

  • File: projects/peertube-rebuild.md
  • Phase 1 architecture diagram references CT 100 for PeerTube. Actual is CT 110.
  • Note: The checklist section correctly references CT 110. Only the diagram is wrong.

Matrix Deployment — Stale Host References

  • File: projects/matrix-synapse-deployment.md
  • Problem: Still contains original CT 108 deployment instructions (utility node). Matrix was migrated to Contabo on 2026-02-15.
  • dnsmasq section says point matrix.echo6.co to 100.64.0.8 (utility Caddy). Actual dnsmasq points to 100.64.0.1 (Contabo).
  • Action: Add a prominent note at the top indicating this was migrated, or rewrite for Contabo deployment.

meshtastic-headscale-runbook.md — Wrong Debian Template Version

  • File: projects/meshtastic-headscale-runbook.md
  • References: debian-12-standard_12.7-1_amd64.tar.zst
  • Actual (per MEMORY.md): debian-12-standard_12.12-1_amd64.tar.zst

openwebui-theme-deploy.md — Wrong Home Directory

  • File: projects/openwebui-theme-deploy.md
  • References: /home/matt/ paths in some instructions
  • Actual: cortex user is zvx, home directory is /home/zvx/

arr-wiring-runbook.md — Wrong File Paths

  • File: projects/arr-wiring-runbook.md
  • References: ./ref/services/usenet.md (lines 24, 51, 69)
  • Should be: .ref/docs/services/usenet.md

ct-runbook.md — References stock Tailscale auth pattern

  • File: runbooks/ct-runbook.md
  • Problem: References tskey-auth-* (stock Tailscale preauth key format) instead of Headscale registration pattern. Echo6 uses self-hosted Headscale where keys are generated via headscale preauthkeys create.

WATCHTOWER Deployment — Wrong Caddy Pattern

  • File: projects/cc-deploy-watchtower-v2.md
  • References: Docker-based Caddy with ~/docker/caddy/sites/ site file pattern
  • Actual: Utility Caddy is systemd-based on CT 101 with /etc/caddy/Caddyfile (single file, not per-site directory)

4. Duplicates / Contradictions

AUTHENTIK_API_TOKEN — Defined Twice

  • File: credentials line 14 and line 111
  • Values: Both contain the same token value: YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w
  • Action: Remove one instance. Keep the one at line 111 (has regeneration date context), remove line 14's duplicate.

FORGEJO_URL — Defined Twice

  • File: credentials line 27 (FORGEJO_URL=https://forge.echo6.co) and line 100 (FORGEJO_URL="https://forge.echo6.co")
  • Action: Remove the earlier instance (line 27). The line 100 block has more complete Forgejo credentials.

dnsmasq: notes.echo6.co — Duplicate Entries with Different IPs

  • Actual dnsmasq config has TWO entries for notes.echo6.co:
    • address=/notes.echo6.co/100.64.0.22 (mesh-bridge CT 107 — incorrect target)
    • address=/notes.echo6.co/100.64.0.1 (Contabo — correct, LiveSync runs here)
  • Action: Remove the 100.64.0.22 entry. Only the 100.64.0.1 entry should remain.

authentik-access-groups.md vs authentik.md — Conflicting Group Members

  • authentik.md lists media-users members as: jodie, matt
  • authentik-access-groups.md lists media-users members as: jodie (no matt)
  • Action: Verify live state via Authentik API and update both docs to match. Matt may be in authentik Admins (superuser) which bypasses group checks, making explicit media-users membership unnecessary.

authentik.md vs authentik-access-groups.md — Missing Groups

  • authentik.md lists ai-users group (PK 0631b273-...) with member matt
  • authentik-access-groups.md does NOT list ai-users at all
  • Action: Add ai-users to access groups runbook reference table.

authentik.md vs authentik-access-groups.md — Missing Application Bindings

The access groups runbook is missing bindings for recently added applications:

  • Open WebUI (PK 14) — should be bound to ai-users
  • Matrix (PK 15) — should be bound to communication-users
  • TAK Server — not listed anywhere
  • Action: Update the Quick Reference tables in both documents.

5. Missing Documentation

TAK Server (tak.echo6.co) — Running, Undocumented

  • Status: Deployed 2026-02-20 on Contabo. Docker containers running (tak-server-deploy, sigil). Credentials exist in credentials file (lines 236-249). Caddy site blocks exist on Contabo.
  • Missing from:
    • docs/services/services.md — no TAK Server entry
    • docs/software/caddy.md — no tak.echo6.co site block documented
    • docs/software/authentik.md — no TAK Server provider listed
    • docs/hardware/environment.md — Contabo services list doesn't mention TAK
  • Action: Create docs/software/tak.md or add TAK Server entries to existing docs.

Obsidian LiveSync (notes.echo6.co) — Running, Partially Documented

  • Status: Docker containers running on Contabo (livesync-couchdb, livesync-provisioner). Credentials exist (lines 37-43). Caddy site block exists. projects/deploy livesync.md exists as research doc.
  • Missing from:
    • docs/services/services.md — no LiveSync entry
    • docs/software/caddy.md — no notes.echo6.co Contabo site block documented
  • Action: Add to services.md and caddy.md.

RECON LXC (CT 130) — Running, Missing from Infrastructure Docs

  • Status: docs/software/recon.md exists and is thorough. But CT 130 is missing from:
    • docs/hardware/environment.md — not in LXC Containers table
    • docs/services/services.md — not listed
  • Action: Add | recon | data (CT 130) | 192.168.1.130 | 100.64.0.24 | RECON knowledge extraction pipeline | to both tables.

files.echo6.co — Referenced, Not Documented

  • Status: Referenced in dnsmasq config, RECON docs, and landing page data export. Presumably an nginx file server on RECON (CT 130).
  • Missing from:
    • docs/services/services.md — no entry
    • docs/software/caddy.md — no dnsmasq entry documented
  • Action: Add to services.md and caddy.md dnsmasq section.

Undocumented Headscale Nodes

The headscale node list in docs/hardware/environment.md is missing several entries seen in live headscale nodes list:

Node Tailscale IP Status Notes
recon 100.64.0.24 Online CT 130, documented in recon.md but not in environment.md
localhost 100.64.0.12 Last seen varies Unknown purpose — possibly a test/dev registration
invalid-nwr32bou 100.64.0.16 Last seen varies Unknown — possibly a stale/orphaned node registration
  • Also: meshmon-node appears as hostname "advbbs" in headscale — may have been renamed/repurposed.
  • Action: Audit headscale node list, remove orphaned registrations, update environment.md.

Undocumented Contabo Docker Containers

Live docker ps on Contabo shows containers not documented anywhere:

  • sigil — TAK Server companion (web console), partially covered by TAK credentials
  • termix — Unknown purpose, not documented

Action: Identify what termix is and document both.

Usenet Credentials Not in Credentials File

  • File: docs/services/usenet.md references API keys and passwords with "see .ref/credentials"
  • Problem: No SABnzbd, Sonarr, Radarr, or Prowlarr API keys exist in the credentials file
  • Action: Either add the API keys to the credentials file or remove the reference in usenet.md.

6. Recommendations

Priority 1 — Fix Immediately (Data Integrity / Operational Risk)

  1. Remove dead Wiki.js credentials from credentials file (lines 20-23, 35). These contain valid API tokens for a service that no longer exists.

  2. Fix MESHMONITOR_TAILSCALE_URL (line 127) — currently points to Contabo instead of MeshMonitor. Any automation using this URL will target the wrong host.

  3. Fix OPENWEBUI_API_URL (line 52) — points to a non-existent IP. Any automation using this will fail silently.

  4. Fix dnsmasq duplicate for notes.echo6.co — the incorrect 100.64.0.22 entry could cause intermittent routing failures depending on which entry dnsmasq uses.

  5. Fix proxmox-onboard-node.md Tailscale IPs — every IP in the cluster table is wrong. Anyone following this runbook will get incorrect SSH aliases.

Priority 2 — Documentation Accuracy

  1. Update Contabo Tailscale IP from 100.64.0.6 to 100.64.0.1 across all affected files (5+ files).

  2. Update Headscale container name from headscale-vanilla/headscale-standby to headscale in deployment runbooks.

  3. Fix caddy.md mail port from 8443 to 8453.

  4. Add TAK Server to services.md, caddy.md, and authentik.md.

  5. Add RECON CT 130 to environment.md LXC table and services.md.

  6. Add LiveSync/notes.echo6.co to services.md and caddy.md.

  7. Remove stale AUTHENTIK_API_TOKEN duplicate and FORGEJO_URL duplicate from credentials.

Priority 3 — Cleanup

  1. Remove Echo6 Portal credentials (lines 60-69) — service is dead.

  2. Clean up Stalwart Mail comment (line 72) — remove misleading "PK 53 may still exist" text.

  3. Delete GoDaddy DNS record for docs.echo6.co — service no longer exists.

  4. Remove dnsmasq entry for docs.echo6.co.

  5. Update stale file path reference on credentials line 33.

  6. Audit headscale node list — clean up orphaned nodes (localhost, invalid-nwr32bou), investigate meshmon-node/advbbs rename, update environment.md.

  7. Investigate termix container on Contabo — document or remove.

  8. Update authentik-access-groups.md — add ai-users group, add Open WebUI/Matrix/TAK bindings to reference tables.

Files Audited

Credentials

  • /home/zvx/projects/.ref/credentials

Documentation

  • /home/zvx/projects/.ref/docs/hardware/environment.md
  • /home/zvx/projects/.ref/docs/services/services.md
  • /home/zvx/projects/.ref/docs/services/usenet.md
  • /home/zvx/projects/.ref/docs/software/authentik.md
  • /home/zvx/projects/.ref/docs/software/caddy.md
  • /home/zvx/projects/.ref/docs/software/dns.md
  • /home/zvx/projects/.ref/docs/software/recon.md
  • /home/zvx/projects/.ref/docs/software/searxng.md

Project Files

  • /home/zvx/projects/.ref/projects/DEPLOY-API-KEYS-TAB.md
  • /home/zvx/projects/.ref/projects/advbbs-project.md
  • /home/zvx/projects/.ref/projects/arr-stack-runbook.md
  • /home/zvx/projects/.ref/projects/arr-wiring-runbook.md
  • /home/zvx/projects/.ref/projects/cc-deploy-watchtower-v2.md
  • /home/zvx/projects/.ref/projects/deploy livesync.md
  • /home/zvx/projects/.ref/projects/headscale-full-deployment.md
  • /home/zvx/projects/.ref/projects/matrix-synapse-deployment.md
  • /home/zvx/projects/.ref/projects/meshtastic-headscale-runbook.md
  • /home/zvx/projects/.ref/projects/openwebui-theme-deploy.md
  • /home/zvx/projects/.ref/projects/peertube-phase2-project.md
  • /home/zvx/projects/.ref/projects/peertube-rebuild.md
  • /home/zvx/projects/.ref/projects/utility-caddy-initial-setup.md
  • /home/zvx/projects/.ref/projects/vaultwarden-deployment.md

Runbooks

  • /home/zvx/projects/.ref/runbooks/add-peertube-channel.md
  • /home/zvx/projects/.ref/runbooks/authentik-access-groups.md
  • /home/zvx/projects/.ref/runbooks/authentik-create-invitation.md
  • /home/zvx/projects/.ref/runbooks/authentik-oidc-application.md
  • /home/zvx/projects/.ref/runbooks/authentik-upgrade.md
  • /home/zvx/projects/.ref/runbooks/binary-wrapper-interception.md
  • /home/zvx/projects/.ref/runbooks/ct-runbook.md
  • /home/zvx/projects/.ref/runbooks/expose-service-contabo.md
  • /home/zvx/projects/.ref/runbooks/expose-service-home.md
  • /home/zvx/projects/.ref/runbooks/gpu-cpu-fallback-routing.md
  • /home/zvx/projects/.ref/runbooks/ia-cli-reference.md
  • /home/zvx/projects/.ref/runbooks/ia-download-mirror.md
  • /home/zvx/projects/.ref/runbooks/idahomesh-bridge-setup.md
  • /home/zvx/projects/.ref/runbooks/idahomesh-vpn-device-setup.md
  • /home/zvx/projects/.ref/runbooks/mailcow-create-mailbox.md
  • /home/zvx/projects/.ref/runbooks/meshmonitor-password-reset.md
  • /home/zvx/projects/.ref/runbooks/meshtasticd-sim-nodes-runbook.md
  • /home/zvx/projects/.ref/runbooks/nordvpn-lxc.md
  • /home/zvx/projects/.ref/runbooks/peertube-remote-runner.md
  • /home/zvx/projects/.ref/runbooks/pg-backup.md
  • /home/zvx/projects/.ref/runbooks/pi-nas-omv-runbook.md
  • /home/zvx/projects/.ref/runbooks/pipeline-probe-gate.md
  • /home/zvx/projects/.ref/runbooks/proxmox-create-ubuntu-vm.md
  • /home/zvx/projects/.ref/runbooks/proxmox-onboard-node.md
  • /home/zvx/projects/.ref/runbooks/recon-operations.md
  • /home/zvx/projects/.ref/runbooks/recon-service-integration.md

Misc Files

  • /home/zvx/projects/.ref/echo6-landing-page-data-export.md
  • /home/zvx/projects/.ref/ia-download-queue.md

Not Audited (Low-Risk Assets)

  • /home/zvx/projects/.ref/pp_comparison.json — data file, no infrastructure references
  • /home/zvx/projects/.ref/.gitignore — git config
  • /home/zvx/projects/.ref/assets/ — static assets (CSS, JS, images, key_manager.py)

Summary Statistics

Category Count
Dead services identified 2 (Wiki.js, Echo6 Portal) + 1 already decommissioned (Stalwart)
Stale credentials to clean 4 entries (Wiki.js x3, Echo6 Portal x5, wrong URLs x2, stale path x1)
Incorrect values found 15+ across 10+ files
Duplicate entries 4 (AUTHENTIK_API_TOKEN, FORGEJO_URL, dnsmasq notes.echo6.co, group member lists)
Missing documentation items 7 (TAK Server, LiveSync, RECON in env, files.echo6.co, headscale nodes, termix, usenet creds)
Total files audited 44
Files with issues 24
Files clean 20

Audit completed: 2026-02-21 by Claude Code on cortex