echo6-docs/AUDIT-2026-02-21.md

# .ref/ Directory Audit — 2026-02-21

**Auditor:** Claude Code (cortex)
**Hostname:** cortex
**Scope:** Every file in `/home/zvx/projects/.ref/` cross-referenced against live infrastructure state.

---

## 1. Dead Services to Remove

### Wiki.js / docs.echo6.co — CONFIRMED DEAD

- **Evidence:** No Docker container running on Contabo. `docker ps -a` shows no wiki-related containers.
- **Stale artifacts:**
  - `credentials` lines 20-23: `WIKIJS_ADMIN_EMAIL`, `WIKIJS_ADMIN_PASSWORD`, `WIKIJS_URL`
  - `credentials` line 35: `WIKIJS_API_TOKEN` (JWT, expires 2026-11-07)
  - GoDaddy DNS: `docs.echo6.co` still resolves to `199.6.36.163`
  - dnsmasq: `docs.echo6.co` still has an entry pointing to `100.64.0.1`
  - `docs/software/caddy.md`: Still referenced under Contabo Caddy (not present in actual Caddyfile)
  - `docs/software/dns.md`: Still lists `docs.echo6.co` under home services
- **Action:** Remove credentials, delete DNS record, remove dnsmasq entry, remove from caddy.md and dns.md.

### Echo6 Portal — CONFIRMED DEAD

- **Evidence:** Nothing listening on port 3002 on utility (192.168.1.241). No container found.
- **Stale artifacts:**
  - `credentials` lines 60-69: `ECHO6_PORTAL_*` credentials (admin user, OIDC client ID/secret, issuer URL)
  - `credentials` line 61: References deployment at `192.168.1.241:3002`
- **Action:** Remove credentials. Verify Authentik provider `echo6-portal` doesn't still exist (clean up if so).

### Stalwart Mail — Already Decommissioned (Partially Cleaned)

- **Status:** Credentials correctly commented out in credentials file (lines 71-75).
- **credentials** line 72: Comment says "Authentik provider PK 53 may still exist" — **Verified: PK 53 does NOT exist in Authentik.** The comment is now misleading.
- **Action:** Update comment to state PK 53 was already cleaned, or remove the entire stale block.

---

## 2. Stale Credentials to Clean

### OPENWEBUI_API_URL — Wrong IP

- **File:** `credentials` line 52
- **Current value:** `OPENWEBUI_API_URL=http://192.168.1.239:3000`
- **Problem:** No known service at `192.168.1.239`. Open WebUI runs on cortex at `192.168.1.150:8080`.
- **Action:** Update to `http://192.168.1.150:8080` or `http://100.64.0.14:8080` (Tailscale IP).

### MESHMONITOR_TAILSCALE_URL — Wrong IP

- **File:** `credentials` lines 124, 127
- **Current value:** `MESHMONITOR_TAILSCALE_URL="http://100.64.0.1:8080"`
- **Problem:** `100.64.0.1` is Contabo, not MeshMonitor. MeshMonitor runs on utility CT 100, Tailscale IP `100.64.0.7`.
- **Action:** Change to `http://100.64.0.7:8080`.

### Contabo Tailscale IP in credentials — Wrong

- **File:** `credentials` line 86
- **Current value:** `# Tailscale IP: 100.64.0.6`
- **Problem:** `100.64.0.6` is the data Proxmox node's Tailscale IP. Contabo's Tailscale IP is `100.64.0.1`.
- **Action:** Change comment to `# Tailscale IP: 100.64.0.1`.

### Stale File Path Reference

- **File:** `credentials` line 33
- **Current value:** `# - File Location: /home/zvx/projects/selfhosting/.credentials`
- **Problem:** References an old path that no longer exists. Current location is `/home/zvx/projects/.ref/credentials`.
- **Action:** Update to current path or remove the comment.

---

## 3. Incorrect Values

### Caddy Port: mail.echo6.co

- **File:** `docs/software/caddy.md` — Contabo Caddy table
- **Documented:** `reverse_proxy https://127.0.0.1:8443` (port 8443)
- **Actual (live Caddyfile):** `reverse_proxy https://127.0.0.1:8453` (port 8453)
- **Also wrong in:** Port Map table at bottom of `caddy.md` (shows 8443)
- **Action:** Update both references from 8443 to 8453.

### Contabo Tailscale IP — Multiple Files Reference 100.64.0.6 Instead of 100.64.0.1

| File | Line/Section | Wrong Value | Correct Value |
|------|-------------|-------------|---------------|
| `credentials` | Line 86 | `100.64.0.6` | `100.64.0.1` |
| `projects/vaultwarden-deployment.md` | SSH commands | `100.64.0.6` | `100.64.0.1` |
| `projects/authentik-oidc-application.md` (if exists as runbook) | SSH commands | `100.64.0.6` | `100.64.0.1` |
| `runbooks/authentik-oidc-application.md` | SSH commands to Contabo | `100.64.0.6` | `100.64.0.1` |
| `runbooks/expose-service-contabo.md` | Step 5 verification dig command | `100.64.0.6` | `100.64.0.1` |

**Root cause:** 100.64.0.6 is the **data** Proxmox node. This was likely a copy-paste error from early infrastructure setup when Contabo's Tailscale IP may not have been settled.

### Headscale Container Name — "headscale-vanilla" vs "headscale"

| File | Incorrect | Correct |
|------|-----------|---------|
| `projects/headscale-full-deployment.md` | `headscale-vanilla` (throughout) | `headscale` |
| `runbooks/proxmox-create-ubuntu-vm.md` | `headscale-standby` (lines 245, 257, 277) | `headscale` |

**Note:** MEMORY.md already documents this: "Headscale container on Contabo is named `headscale` (not `headscale-vanilla` as in some runbooks)." The runbooks were never updated to match.

### Utility Caddy Tailscale IP

- **File:** `projects/utility-caddy-initial-setup.md`
- **Documented:** Tailscale IP `100.64.0.2`
- **Actual:** `100.64.0.8` (per environment.md and live headscale node list)
- **Action:** Update to `100.64.0.8`.

### Proxmox Onboard Node — Wrong Tailscale IPs for Cluster Nodes

- **File:** `runbooks/proxmox-onboard-node.md` — "Current Cluster" table (lines 9-14)
- **Documented:**

| Node | Tailscale IP (documented) | Tailscale IP (actual from environment.md) |
|------|--------------------------|------------------------------------------|
| data | 100.64.0.20 | 100.64.0.6 |
| utility | 100.64.0.19 | 100.64.0.5 |
| cloud | 100.64.0.22 | 100.64.0.4 |
| media | 100.64.0.21 | 100.64.0.3 |

**Every single Tailscale IP in this table is wrong.** These appear to be from a completely different assignment scheme. Additionally, the `toc` node (192.168.1.244 / 100.64.0.13) is missing from this table entirely.

### PeerTube Rebuild — Wrong CT ID in Architecture Diagram

- **File:** `projects/peertube-rebuild.md`
- **Phase 1 architecture diagram** references CT 100 for PeerTube. Actual is CT 110.
- **Note:** The checklist section correctly references CT 110. Only the diagram is wrong.

### Matrix Deployment — Stale Host References

- **File:** `projects/matrix-synapse-deployment.md`
- **Problem:** Still contains original CT 108 deployment instructions (utility node). Matrix was migrated to Contabo on 2026-02-15.
- **dnsmasq section** says point `matrix.echo6.co` to `100.64.0.8` (utility Caddy). Actual dnsmasq points to `100.64.0.1` (Contabo).
- **Action:** Add a prominent note at the top indicating this was migrated, or rewrite for Contabo deployment.

### meshtastic-headscale-runbook.md — Wrong Debian Template Version

- **File:** `projects/meshtastic-headscale-runbook.md`
- **References:** `debian-12-standard_12.7-1_amd64.tar.zst`
- **Actual (per MEMORY.md):** `debian-12-standard_12.12-1_amd64.tar.zst`

### openwebui-theme-deploy.md — Wrong Home Directory

- **File:** `projects/openwebui-theme-deploy.md`
- **References:** `/home/matt/` paths in some instructions
- **Actual:** cortex user is `zvx`, home directory is `/home/zvx/`

### arr-wiring-runbook.md — Wrong File Paths

- **File:** `projects/arr-wiring-runbook.md`
- **References:** `./ref/services/usenet.md` (lines 24, 51, 69)
- **Should be:** `.ref/docs/services/usenet.md`

### ct-runbook.md — References stock Tailscale auth pattern

- **File:** `runbooks/ct-runbook.md`
- **Problem:** References `tskey-auth-*` (stock Tailscale preauth key format) instead of Headscale registration pattern. Echo6 uses self-hosted Headscale where keys are generated via `headscale preauthkeys create`.

### WATCHTOWER Deployment — Wrong Caddy Pattern

- **File:** `projects/cc-deploy-watchtower-v2.md`
- **References:** Docker-based Caddy with `~/docker/caddy/sites/` site file pattern
- **Actual:** Utility Caddy is systemd-based on CT 101 with `/etc/caddy/Caddyfile` (single file, not per-site directory)

---

## 4. Duplicates / Contradictions

### AUTHENTIK_API_TOKEN — Defined Twice

- **File:** `credentials` line 14 and line 111
- **Values:** Both contain the same token value: `YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w`
- **Action:** Remove one instance. Keep the one at line 111 (has regeneration date context), remove line 14's duplicate.

### FORGEJO_URL — Defined Twice

- **File:** `credentials` line 27 (`FORGEJO_URL=https://forge.echo6.co`) and line 100 (`FORGEJO_URL="https://forge.echo6.co"`)
- **Action:** Remove the earlier instance (line 27). The line 100 block has more complete Forgejo credentials.

### dnsmasq: notes.echo6.co — Duplicate Entries with Different IPs

- **Actual dnsmasq config** has TWO entries for `notes.echo6.co`:
  - `address=/notes.echo6.co/100.64.0.22` (mesh-bridge CT 107 — incorrect target)
  - `address=/notes.echo6.co/100.64.0.1` (Contabo — correct, LiveSync runs here)
- **Action:** Remove the `100.64.0.22` entry. Only the `100.64.0.1` entry should remain.

### authentik-access-groups.md vs authentik.md — Conflicting Group Members

- `authentik.md` lists `media-users` members as: **jodie, matt**
- `authentik-access-groups.md` lists `media-users` members as: **jodie** (no matt)
- **Action:** Verify live state via Authentik API and update both docs to match. Matt may be in `authentik Admins` (superuser) which bypasses group checks, making explicit media-users membership unnecessary.

### authentik.md vs authentik-access-groups.md — Missing Groups

- `authentik.md` lists `ai-users` group (PK `0631b273-...`) with member matt
- `authentik-access-groups.md` does NOT list `ai-users` at all
- **Action:** Add `ai-users` to access groups runbook reference table.

### authentik.md vs authentik-access-groups.md — Missing Application Bindings

The access groups runbook is missing bindings for recently added applications:
- Open WebUI (PK 14) — should be bound to `ai-users`
- Matrix (PK 15) — should be bound to `communication-users`
- TAK Server — not listed anywhere
- **Action:** Update the Quick Reference tables in both documents.

---

## 5. Missing Documentation

### TAK Server (tak.echo6.co) — Running, Undocumented

- **Status:** Deployed 2026-02-20 on Contabo. Docker containers running (`tak-server-deploy`, `sigil`). Credentials exist in `credentials` file (lines 236-249). Caddy site blocks exist on Contabo.
- **Missing from:**
  - `docs/services/services.md` — no TAK Server entry
  - `docs/software/caddy.md` — no tak.echo6.co site block documented
  - `docs/software/authentik.md` — no TAK Server provider listed
  - `docs/hardware/environment.md` — Contabo services list doesn't mention TAK
- **Action:** Create `docs/software/tak.md` or add TAK Server entries to existing docs.

### Obsidian LiveSync (notes.echo6.co) — Running, Partially Documented

- **Status:** Docker containers running on Contabo (`livesync-couchdb`, `livesync-provisioner`). Credentials exist (lines 37-43). Caddy site block exists. `projects/deploy livesync.md` exists as research doc.
- **Missing from:**
  - `docs/services/services.md` — no LiveSync entry
  - `docs/software/caddy.md` — no notes.echo6.co Contabo site block documented
- **Action:** Add to services.md and caddy.md.

### RECON LXC (CT 130) — Running, Missing from Infrastructure Docs

- **Status:** `docs/software/recon.md` exists and is thorough. But CT 130 is missing from:
  - `docs/hardware/environment.md` — not in LXC Containers table
  - `docs/services/services.md` — not listed
- **Action:** Add `| recon | data (CT 130) | 192.168.1.130 | 100.64.0.24 | RECON knowledge extraction pipeline |` to both tables.

### files.echo6.co — Referenced, Not Documented

- **Status:** Referenced in dnsmasq config, RECON docs, and landing page data export. Presumably an nginx file server on RECON (CT 130).
- **Missing from:**
  - `docs/services/services.md` — no entry
  - `docs/software/caddy.md` — no dnsmasq entry documented
- **Action:** Add to services.md and caddy.md dnsmasq section.

### Undocumented Headscale Nodes

The headscale node list in `docs/hardware/environment.md` is missing several entries seen in live `headscale nodes list`:

| Node | Tailscale IP | Status | Notes |
|------|-------------|--------|-------|
| recon | 100.64.0.24 | Online | CT 130, documented in recon.md but not in environment.md |
| localhost | 100.64.0.12 | Last seen varies | Unknown purpose — possibly a test/dev registration |
| invalid-nwr32bou | 100.64.0.16 | Last seen varies | Unknown — possibly a stale/orphaned node registration |

- **Also:** `meshmon-node` appears as hostname "advbbs" in headscale — may have been renamed/repurposed.
- **Action:** Audit headscale node list, remove orphaned registrations, update environment.md.

### Undocumented Contabo Docker Containers

Live `docker ps` on Contabo shows containers not documented anywhere:
- `sigil` — TAK Server companion (web console), partially covered by TAK credentials
- `termix` — Unknown purpose, not documented

**Action:** Identify what `termix` is and document both.

### Usenet Credentials Not in Credentials File

- **File:** `docs/services/usenet.md` references API keys and passwords with "see .ref/credentials"
- **Problem:** No SABnzbd, Sonarr, Radarr, or Prowlarr API keys exist in the credentials file
- **Action:** Either add the API keys to the credentials file or remove the reference in usenet.md.

---

## 6. Recommendations

### Priority 1 — Fix Immediately (Data Integrity / Operational Risk)

1. **Remove dead Wiki.js credentials** from credentials file (lines 20-23, 35). These contain valid API tokens for a service that no longer exists.

2. **Fix MESHMONITOR_TAILSCALE_URL** (line 127) — currently points to Contabo instead of MeshMonitor. Any automation using this URL will target the wrong host.

3. **Fix OPENWEBUI_API_URL** (line 52) — points to a non-existent IP. Any automation using this will fail silently.

4. **Fix dnsmasq duplicate** for notes.echo6.co — the incorrect `100.64.0.22` entry could cause intermittent routing failures depending on which entry dnsmasq uses.

5. **Fix proxmox-onboard-node.md Tailscale IPs** — every IP in the cluster table is wrong. Anyone following this runbook will get incorrect SSH aliases.

### Priority 2 — Documentation Accuracy

6. **Update Contabo Tailscale IP** from `100.64.0.6` to `100.64.0.1` across all affected files (5+ files).

7. **Update Headscale container name** from `headscale-vanilla`/`headscale-standby` to `headscale` in deployment runbooks.

8. **Fix caddy.md mail port** from 8443 to 8453.

9. **Add TAK Server** to services.md, caddy.md, and authentik.md.

10. **Add RECON CT 130** to environment.md LXC table and services.md.

11. **Add LiveSync/notes.echo6.co** to services.md and caddy.md.

12. **Remove stale AUTHENTIK_API_TOKEN duplicate** and FORGEJO_URL duplicate from credentials.

### Priority 3 — Cleanup

13. **Remove Echo6 Portal credentials** (lines 60-69) — service is dead.

14. **Clean up Stalwart Mail comment** (line 72) — remove misleading "PK 53 may still exist" text.

15. **Delete GoDaddy DNS record** for docs.echo6.co — service no longer exists.

16. **Remove dnsmasq entry** for docs.echo6.co.

17. **Update stale file path reference** on credentials line 33.

18. **Audit headscale node list** — clean up orphaned nodes (`localhost`, `invalid-nwr32bou`), investigate `meshmon-node`/`advbbs` rename, update environment.md.

19. **Investigate `termix` container** on Contabo — document or remove.

20. **Update authentik-access-groups.md** — add `ai-users` group, add Open WebUI/Matrix/TAK bindings to reference tables.

---

## Files Audited

### Credentials
- [x] `/home/zvx/projects/.ref/credentials`

### Documentation
- [x] `/home/zvx/projects/.ref/docs/hardware/environment.md`
- [x] `/home/zvx/projects/.ref/docs/services/services.md`
- [x] `/home/zvx/projects/.ref/docs/services/usenet.md`
- [x] `/home/zvx/projects/.ref/docs/software/authentik.md`
- [x] `/home/zvx/projects/.ref/docs/software/caddy.md`
- [x] `/home/zvx/projects/.ref/docs/software/dns.md`
- [x] `/home/zvx/projects/.ref/docs/software/recon.md`
- [x] `/home/zvx/projects/.ref/docs/software/searxng.md`

### Project Files
- [x] `/home/zvx/projects/.ref/projects/DEPLOY-API-KEYS-TAB.md`
- [x] `/home/zvx/projects/.ref/projects/advbbs-project.md`
- [x] `/home/zvx/projects/.ref/projects/arr-stack-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/arr-wiring-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/cc-deploy-watchtower-v2.md`
- [x] `/home/zvx/projects/.ref/projects/deploy livesync.md`
- [x] `/home/zvx/projects/.ref/projects/headscale-full-deployment.md`
- [x] `/home/zvx/projects/.ref/projects/matrix-synapse-deployment.md`
- [x] `/home/zvx/projects/.ref/projects/meshtastic-headscale-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/openwebui-theme-deploy.md`
- [x] `/home/zvx/projects/.ref/projects/peertube-phase2-project.md`
- [x] `/home/zvx/projects/.ref/projects/peertube-rebuild.md`
- [x] `/home/zvx/projects/.ref/projects/utility-caddy-initial-setup.md`
- [x] `/home/zvx/projects/.ref/projects/vaultwarden-deployment.md`

### Runbooks
- [x] `/home/zvx/projects/.ref/runbooks/add-peertube-channel.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-access-groups.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-create-invitation.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-oidc-application.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-upgrade.md`
- [x] `/home/zvx/projects/.ref/runbooks/binary-wrapper-interception.md`
- [x] `/home/zvx/projects/.ref/runbooks/ct-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/expose-service-contabo.md`
- [x] `/home/zvx/projects/.ref/runbooks/expose-service-home.md`
- [x] `/home/zvx/projects/.ref/runbooks/gpu-cpu-fallback-routing.md`
- [x] `/home/zvx/projects/.ref/runbooks/ia-cli-reference.md`
- [x] `/home/zvx/projects/.ref/runbooks/ia-download-mirror.md`
- [x] `/home/zvx/projects/.ref/runbooks/idahomesh-bridge-setup.md`
- [x] `/home/zvx/projects/.ref/runbooks/idahomesh-vpn-device-setup.md`
- [x] `/home/zvx/projects/.ref/runbooks/mailcow-create-mailbox.md`
- [x] `/home/zvx/projects/.ref/runbooks/meshmonitor-password-reset.md`
- [x] `/home/zvx/projects/.ref/runbooks/meshtasticd-sim-nodes-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/nordvpn-lxc.md`
- [x] `/home/zvx/projects/.ref/runbooks/peertube-remote-runner.md`
- [x] `/home/zvx/projects/.ref/runbooks/pg-backup.md`
- [x] `/home/zvx/projects/.ref/runbooks/pi-nas-omv-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/pipeline-probe-gate.md`
- [x] `/home/zvx/projects/.ref/runbooks/proxmox-create-ubuntu-vm.md`
- [x] `/home/zvx/projects/.ref/runbooks/proxmox-onboard-node.md`
- [x] `/home/zvx/projects/.ref/runbooks/recon-operations.md`
- [x] `/home/zvx/projects/.ref/runbooks/recon-service-integration.md`

### Misc Files
- [x] `/home/zvx/projects/.ref/echo6-landing-page-data-export.md`
- [x] `/home/zvx/projects/.ref/ia-download-queue.md`

### Not Audited (Low-Risk Assets)
- `/home/zvx/projects/.ref/pp_comparison.json` — data file, no infrastructure references
- `/home/zvx/projects/.ref/.gitignore` — git config
- `/home/zvx/projects/.ref/assets/` — static assets (CSS, JS, images, key_manager.py)

---

## Summary Statistics

| Category | Count |
|----------|-------|
| Dead services identified | 2 (Wiki.js, Echo6 Portal) + 1 already decommissioned (Stalwart) |
| Stale credentials to clean | 4 entries (Wiki.js x3, Echo6 Portal x5, wrong URLs x2, stale path x1) |
| Incorrect values found | 15+ across 10+ files |
| Duplicate entries | 4 (AUTHENTIK_API_TOKEN, FORGEJO_URL, dnsmasq notes.echo6.co, group member lists) |
| Missing documentation items | 7 (TAK Server, LiveSync, RECON in env, files.echo6.co, headscale nodes, termix, usenet creds) |
| Total files audited | 44 |
| Files with issues | 24 |
| Files clean | 20 |

---

*Audit completed: 2026-02-21 by Claude Code on cortex*