matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

merge into: matt:main
Branches Tags
matt:main
matt:navi-directions-redesign
...
pull from: matt:navi-directions-redesign
Branches Tags
matt:main
matt:navi-directions-redesign

No commits in common. "main" and "navi-directions-redesign" have entirely different histories.
main ... navi-direc

167 changed files with 197 additions and 38974 deletions
Download patch file Download diff file Expand all files Collapse all files

1
, c.get(k)) PY \" "
View file

@ -1 +0,0 @@
bash: -c: line 1: unexpected EOF while looking for matching `"'

55
.claude/settings.local.json
View file

@ -1,55 +0,0 @@
{
"permissions": {
"allow": [
"WebSearch",
"WebFetch(domain:docs.opentakserver.io)",
"WebFetch(domain:docs.goauthentik.io)",
"Bash(grep -nE '^#{1,3} ' fleet-patch-audit.md)",
"Bash(ssh *)",
"Bash(tailscale debug *)",
"Bash(tailscale status *)",
"Bash(python3 -c \"import sys,json; d=json.load\\(sys.stdin\\); print\\('ControlURL:', d.get\\('Self',{}\\).get\\('ControlURL','n/a'\\) if 'Self' in d else 'no Self key'\\); [print\\(k,':',v\\) for k,v in d.items\\(\\) if 'control' in k.lower\\(\\)]\")",
"Read(//etc/default/**)",
"Bash(systemctl cat *)",
"Bash(timeout 5 bash -c 'cat < /dev/null > /dev/tcp/5.189.158.149/25')",
"Bash(dig +short vpn.echo6.co)",
"Bash(nslookup vpn.echo6.co)",
"Bash(timeout 5 bash -c 'cat < /dev/null > /dev/tcp/5.189.158.149/443')",
"Bash(dig +short MX echo6.co)",
"Bash(dig +short mail.echo6.co)",
"Bash(dig +short A mail.echo6.co)",
"Bash(dig +short TXT echo6.co)",
"Bash(gh auth *)",
"Bash(git clone *)",
"Bash(git *)",
"Bash(scp zvx@192.168.1.104:/opt/central/sql/migrations/036_add_avalanche_org_adapter.sql /tmp/claude-1000/-home-zvx-projects--ref/0b576082-6442-481f-ad79-12a39da62765/scratchpad/central/sql/migrations/036_add_avalanche_org_adapter.sql)",
"Bash(gh pr create --title 'v0.14.6: commit orphaned migration 036 + fix stale README/version metadata' --body ' *)",
"Bash(gh pr create --title 'eonet: bypass bbox filter for global disaster feed' --body ' *)",
"Read(//tmp/central-clone/**)",
"Bash(gh pr create --title 'supervisor: shrink sat dedup window + WAL cursors.db to cut CPU/IO' --body ' *)",
"Bash(shellcheck --version)",
"Read(//tmp/central-deploy-work/**)",
"Read(//tmp/central-deploy-work/scripts/**)",
"Bash(chmod +x /tmp/central-deploy-work/scripts/deploy.sh)",
"Bash(bash -n /tmp/central-deploy-work/scripts/deploy.sh)",
"Bash(gh pr create --title 'scripts/deploy.sh: one-command tag-based deploy with pre-flight backup + verify' --body ' *)",
"Skill(update-config)",
"Edit",
"Write",
"Read",
"Glob",
"Grep",
"Bash(sudo *)",
"Bash(psql *)",
"Bash(systemctl *)",
"Bash(journalctl *)",
"Bash(curl *)",
"Bash(uv *)",
"Bash(pg_dump *)",
"Bash(python3 *)",
"Bash(scp *)",
"Bash(gh *)"
],
"defaultMode": "bypassPermissions"
}
}

3
.gitignore vendored
View file

@ -1,3 +0,0 @@
engine/.embcache.json
engine/.last-sweep
engine/sweep-cron.log

27
CLAUDE.md
View file

@ -1,27 +0,0 @@
# Echo6 Docs Vault — Claude Guidelines
This repo (`echo6-docs`, canonical on Forge) is both an Obsidian documentation vault (`vault/`) and its self-contained maintenance engine (`engine/`).
## The documentation model
- **Documentation library only.** Do NOT create entity/concept/node pages or stubs — nothing exists just to be a graph dot.
- **Tags = categories.** Each doc carries 1–3 honest category tags in frontmatter. Default to ONE; add a second only if the doc is genuinely about two co-equal subjects. The tag vocabulary lives in `engine/config.yaml` (`topic_categories`).
- **Inline `[[wikilinks]]` point ONLY to existing docs.** Never link to a non-existent target (zero unresolved/red nodes). If a thing has no doc, leave it as plain text.
- **No `INDEX.md`** — tags organize, not a hub.
- Frontmatter: `title, type, tags, related, updated` (+ optional `aliases, status`). `type` ∈ reference|runbook|project|note|session.
## The engine (`engine/`) — how it works
- Runs on cortex RTX A4000. Tagger = `vault-tagger` (Qwen2.5-7B-Instruct) via Ollama (`localhost:11434`); embeddings = bge-m3 via TEI (`localhost:8090`) for `related:`. Pure stdlib HTTP, no cloud.
- Daily cron at 09:00 UTC (GPU-guarded): `engine/sweep.sh` → lint + tag + link changed docs. Internals: `engine/sweep.sh`, `engine/lib/agent.py`, `engine/lib/lint.py`, `engine/config.yaml`.
- **Symlink guard:** `agent.py` skips symlinks (`os.path.islink`); `sweep.sh` uses `find -type f` to exclude them. `vault/CLAUDE-baseline.md` and `vault/rules/` are symlinks to `~/.claude/` — the engine skips them automatically.
- Run manually: `python3 engine/lib/agent.py <doc>` (one doc), `engine/sweep.sh` (changed docs), `engine/sweep.sh --all`.
- Lint (`engine/lib/lint.py`): dead links (stay 0), untagged docs, orphans, unknown tags, gap report (terms across many docs with no own doc — Matt decides whether to create one).
## Going forward
When you ADD or EDIT a doc: honest category tags, `[[wikilink]]` only existing docs, no invented pages. The gap report (`engine/lib/lint.py`) surfaces candidates for new docs; the next sweep links mentions once a doc exists.
## Sync & safety
Double-synced: Obsidian LiveSync ↔ edge2 CouchDB, and `echo6-docs-autocommit` cron (every 6h → Forge). (Syncthing was decommissioned 2026-06-19 — service masked, local state removed; Forge is the durable backup.) Mass file ops require care — commits happen automatically. Secrets live in `credentials` (the one documented exception to no-secrets-in-repo); never put secrets in a graph-tracked doc.

197
NAVI-DIRECTIONS-REDESIGN.md Normal file
View file

@ -0,0 +1,197 @@
# Navi Directions Redesign
This document captures the redesigned directions UX and the broader
single-panel architecture that consolidates the current two-panel layout.
---
## Panel Architecture (Single-Column Model)
The left panel is one always-visible column with state-driven content.
The right-side PlaceDetail panel is removed; its content moves into the
left panel.
**Width:** ~360px desktop. Mobile: panel becomes bottom sheet per
Section 10.
**Visual style:** Panel is always anchored at the left edge of the
viewport with consistent panel background. When idle (no preview, no
route), only the search bar is visible at the top — the rest of the
panel is empty space. The panel never disappears; it just shows less
or more content depending on state.
---
## Panel States
Panel states are mutually exclusive content modes:
### IDLE (no preview, no route)
- Search bar at top
- Empty body (or subtle empty-state prompt: "Search or click a place
to begin")
- Recent activity / search history (when implemented)
### PREVIEW (place selected via map click or search, not committed to route)
- Search bar at top
- Preview card: full place detail (name, type, coords, elevation,
land class, about, contact, links)
- Action buttons inside or below the card: [Directions] [Add stop]
[Save] [Share] [×]
- Click another place: preview replaced (previous lost — search
history retains it)
- Click × or Escape: preview dismissed, return to IDLE
### ROUTING (1+ stops, no preview)
- Search bar at top
- Section: "Route" with stop cards
- Each stop card: collapsed by default (header: pin + name +
drag handle + remove); expandable to show full detail
- Drag-to-reorder via card headers (preserves existing StopList)
- Per-card actions: Save, Share, Remove
- Bottom: [Get Directions] (when 2+ stops with valid mode), mode
selector (auto/walk/bike), trip summary placeholder
### PREVIEW + ROUTING (place clicked while route exists)
- Search bar at top
- Preview card directly below search
- Route section below preview
- Both visible simultaneously
- Multiple cards can be expanded at once (preview + a stop)
### ROUTE_CALCULATED (after route fetch)
- Search bar at top
- Route summary (total time, distance)
- Mode selector
- Stop cards (collapsed; expand for detail)
- Turn-by-turn maneuvers
- Preview can appear on top if user clicks a place
---
## Search Bar Behavior
- Pinned to top of panel
- Always for browsing (search → preview, never auto-add as stop)
### Search History (aspirational — design now, implement later)
- Empty search → dropdown shows 5 most recent searches
- Authed: persisted to backend
- Guest: localStorage
- Typing: recent matches sorted to top of search results
---
## Card Pattern
Shared between preview and stop cards. Both use the same component,
parameterized by role.
### Header (always visible)
- Pin/marker icon (color varies by role — preview vs stop number)
- Place name
- Expand/collapse chevron
- For stops: drag handle, remove (×)
### Body (collapsed by default for stops, expanded by default for preview)
- Type / category
- Coordinates + elevation + land class
- About (description, Wikipedia excerpt)
- Contact (phone, website, hours)
- Links (Wikipedia, OSM, Wikidata)
- Action buttons:
- Preview: [Directions] [Add stop] [Save] [Share]
- Stop: [Save] [Share]
---
## State Transitions
```
IDLE → PREVIEW : click place / search-select
PREVIEW → IDLE : × button / Escape
PREVIEW → PREVIEW (different) : click another place
PREVIEW → ROUTING : "Add stop" on preview
PREVIEW → ROUTE_CALCULATED : "Directions" (becomes destination,
route auto-calculates if From available)
ROUTING → IDLE : remove all stops
ROUTING → PREVIEW + ROUTING : click place
ROUTING → ROUTE_CALCULATED : "Get Directions" / auto-route trigger
PREVIEW + ROUTING → ROUTING : dismiss preview
PREVIEW + ROUTING → ROUTING : "Add stop" on preview (adds new stop)
ROUTE_CALCULATED → ROUTING : edit route
ROUTE_CALCULATED → IDLE : clear route
```
---
## Right Panel (Floating PlaceDetail)
**REMOVED.** All content moves into left panel cards.
---
## Routes / Contacts Tabs
Tentative: Routes is default panel. Contacts is a separate view
accessible via a tab or icon at the top. Contacts view replaces
route/preview content with a contact list — doesn't combine with
route-building.
---
## Map Zoom-to-Feature
With single panel, padding becomes simpler — just one panel width
(~360px) on the left. Top/right/bottom can be small (~50px each).
---
## Mobile Bottom Sheet
Same logic as desktop, layout rotated:
| State | Sheet Position |
|---------------------|----------------|
| IDLE | peek (search bar visible) |
| PREVIEW | half (preview card) |
| ROUTING | half (stops list) |
| PREVIEW + ROUTING | full (both, scrollable) |
| ROUTE_CALCULATED | half/full (summary + maneuvers) |
---
## Out of Scope / Future
- Search history backend persistence (frontend localStorage first)
- Saved places browsing UI
- Contacts view redesign
---
## Implementation Sequence
| Phase | Description |
|-------|-------------|
| k | Refactor Panel.jsx to single-column state-driven content, remove right PlaceDetail panel |
| l | Build shared place card component |
| m | Wire state transitions |
| n | Mobile bottom sheet |
| o | Search history (frontend localStorage scope first) |
---
## Open Questions
- Routes/Contacts tab location and behavior in single-panel model
- Preview-of-already-routed-place: probably auto-expands the existing
stop card, no duplicate preview
- Preview card position confirmed above route section (per mock)

420
credentials
View file

@ -1,420 +0,0 @@
# Infrastructure API Credentials
# IMPORTANT: Keep this file secure and never commit to git
# SSH Access (all servers)
SSH_USER=zvx
SSH_PASSWORD=7redditGold
ROOT_PASSWORD=7redditGold
# GoDaddy DNS API
GODADDY_API_KEY=dKiSoC24ZLTR_3KkMjFyYrzgChk1JHjNLnU
GODADDY_API_SECRET=ExCGBASgSrN4A2mP4daU4G
# Authentik SSO Platform (https://auth.echo6.co) — migrated to edge2 CT 105 / 10.10.10.23 / tailnet 100.64.0.36 on 2026-06-18
AUTHENTIK_API_TOKEN=YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w
AUTHENTIK_URL=https://auth.echo6.co
# Admin credentials for web UI/API access
AUTHENTIK_ADMIN_USER=akadmin
AUTHENTIK_ADMIN_PASSWORD=Mue8FI4sRe2ihRzsmypPnQlpDiae6hqHXZafV2n5CTNPJT1JaS6oqYZzbeGu
# Wiki.js Documentation Platform (https://docs.echo6.co)
WIKIJS_ADMIN_EMAIL=akadmin@echo6.co
WIKIJS_ADMIN_PASSWORD=xh5Q+hWzUe5EqvBQ1tWULlOiPMZX
WIKIJS_URL=https://docs.echo6.co
# Forgejo Git Forge (https://forge.echo6.co)
FORGEJO_API_TOKEN=043fd2df4731a2bb8a2b8352018031bd454fe9a1
FORGEJO_URL=https://forge.echo6.co
# Usage Notes:
# - GoDaddy API: Used for automated DNS record management
# - Authentik API: Used for SSO provider/application configuration
# - Forgejo API: Used for repository mirroring and automation (ALL PERMISSIONS)
# - File Location: /home/zvx/projects/selfhosting/.credentials
# - For agents: Source this file with: source /home/zvx/projects/selfhosting/.credentials
WIKIJS_API_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhcGkiOjIsImdycCI6MSwiaWF0IjoxNzYzOTYxNzUxLCJleHAiOjE3OTU1MTkzNTEsImF1ZCI6InVybjp3aWtpLmpzIiwiaXNzIjoidXJuOndpa2kuanMifQ.LR4qDHT1d8PgwVCo7o8hCbAK19vas5vnROa2IT8InseHh7RwJbnyfc4u71PMzDcIoYJ1QHVeAxhv4SJbkYj-F7b525M63xGuji1pUpxkEaxbZe1iFeQVojSvPvBcw0ZCqMzKtgYNivpSnn9psQulDSFD7Kq63FVte1ABMHeh-s-Mfeqn4pzmgRAMRHZC9l6MSA_cTZ97gxeJCKn9Jyh0LyrQJXYeL0gFG5FvAwOMYDf5fiAmz4Ocyxg4ZMz3q-6Ao9dfB41zbD5WbYwI0qm7wOBDRj3-1-JjMrPJrgYpUOqEuOH1SRDt7VUrOvbxqE7AAM0J8o-7pnVNw1HaseqEiQ
# Obsidian LiveSync (https://notes.echo6.co)
# CouchDB + provisioner on Contabo (5.189.158.149), JWT auth
LIVESYNC_COUCHDB_USER=admin
LIVESYNC_COUCHDB_PASSWORD=kGJkZzoVc5eMiu0rGKJgJNyq
LIVESYNC_PROXY_SECRET=d41933ea00ffb620fa4bb96e0177b3869998c698f5dd7892b86a33fbf8e5aff9
LIVESYNC_URL=https://notes.echo6.co
# Config location: Contabo:/opt/livesync/
# Open WebUI SSO (ai.echo6.co)
OPEN_WEBUI_OAUTH_CLIENT_ID=open-webui
OPEN_WEBUI_OAUTH_CLIENT_SECRET=MfajNL6QZDfDOKprBdYzu3rDya2BoiB5siWt5cYQH1o3zif8ivZ2ZF2HehE3UqLZq2nflwVdYZNhw4tkWQj8kF0dLGA0qDYbBVtUcvtj2AQKQTqAEdRfQF0Oi3slL9OI
OPEN_WEBUI_AUTHENTIK_PROVIDER_ID=14
# Open WebUI API Key (for content sync)
OPENWEBUI_API_KEY=sk-dbf4c9e6ac915d8151f84bf09e51cf2c03e943918adb1737
OPENWEBUI_API_URL=http://192.168.1.239:3000
# PeerTube (stream.echo6.co)
PEERTUBE_URL="https://stream.echo6.co"
PEERTUBE_ADMIN_USER="root"
PEERTUBE_ADMIN_PASSWORD="7redditGold"
PEERTUBE_ADMIN_EMAIL="admin@echo6.co"
# Echo6 Portal - https://echo6.co
# Deployed: Echo6-Cloud (192.168.1.241:3002)
ECHO6_PORTAL_ADMIN_USER=admin
ECHO6_PORTAL_ADMIN_EMAIL=admin@echo6.co
ECHO6_PORTAL_ADMIN_PASS=7redditGold
# Authentik OAuth2 Provider
ECHO6_PORTAL_OIDC_CLIENT_ID=echo6-portal
ECHO6_PORTAL_OIDC_CLIENT_SECRET=15vviNjbzbrtG3WvZxkKrLZv97tcsdzHqWcExp4HarlFlxWInZcKSLixdDWwod67zYxYBDqxkqNR4PNIK0A0XhJSGG6LLK7UHwI1xKeUQt7Dx6bVnDZvqkGP28ewYb4c
ECHO6_PORTAL_OIDC_ISSUER=https://auth.echo6.co/application/o/echo6-portal/
# Stalwart Mail Server — DECOMMISSIONED
# Was deployed 2025-12-19, removed. Authentik provider PK 53 may still exist.
# Credentials retained for reference only.
# STALWART_ADMIN_USER=admin
# STALWART_ADMIN_PASS=ECu0Q4zsQN
# NordVPN (PeerTube Pipeline IP Rotation)
# Installed on media CT 110 (192.168.1.170)
# Service: nordvpnd.service (enabled, auto-start)
# Token: logged in via `nordvpn login --token <token>` — token not stored in file
# Note: NordVPN token was entered interactively, re-login if needed
NORDVPN_LOCATION="CT 110 (media)"
# edge1 SSH Access (rebuilt Contabo VPS, mail-only, 2026-06-19)
# Server: edge1 — Debian 12 + Proxmox 8.4.19, mail-only role
# Public IP: 5.189.158.149 | Tailscale: 100.64.0.40 (node: contabo)
# SSH key: ~/.ssh/contabo2_ed25519 (same key as before rebuild)
# Box password (unrotated): 7redditGold
# ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149
EDGE1_SSH_USER=root
EDGE1_SSH_KEY=~/.ssh/contabo2_ed25519
EDGE1_SSH_PASS=7redditGold
EDGE1_PUBLIC_IP=5.189.158.149
EDGE1_TAILSCALE_IP=100.64.0.40
# Mailcow (mail.echo6.co) — now on edge1 CT 101 (10.10.10.2), updated commit 52a41b4d (SOGo 5.12.8)
# Deployed: 2026-06-19 rebuild (privileged LXC on edge1 Proxmox)
# Access: via host Caddy (mail/autodiscover/autoconfig.echo6.co) + host DNAT (ports 25/465/587/110/143/993/995/4190 → 10.10.10.2)
# Admin UI: https://mail.echo6.co (same as before)
# mail_crypt keys (dovecot crypt volume: data/conf/dovecot/):
# Primary key: ecprivkey.pem (original; must be preserved across rebuilds)
# Secondary key: ecprivkey.pem.fresh-20260619 (generated by restore; loaded alongside primary)
# Config: data/conf/dovecot/extra.conf (update-safe, not dovecot.conf)
# Both keys loaded — required to decrypt old mail encrypted with primary and new mail with secondary
MAILCOW_ADMIN_USER=admin
MAILCOW_ADMIN_PASS=7redditGold
# RECON Pipeline Email Alerts (recon@echo6.co)
# Deployed: 2026-02-01 on Media-Server for pipeline monitoring
RECON_SMTP_USER=recon@echo6.co
RECON_SMTP_PASS=LezU3TJ2W8JFj0hXHcJe
RECON_SMTP_HOST=mail.echo6.co
RECON_SMTP_PORT=587
# Forgejo (forge.echo6.co) - Deployed 2026-02-04
# Migrated to edge2 CT 103 (10.10.10.21 / tailnet 100.64.0.34) on 2026-06-16; SSH via Contabo iptables DNAT (forgejo-ssh-dnat.service).
FORGEJO_URL="https://forge.echo6.co"
FORGEJO_ADMIN_USER="matt"
FORGEJO_ADMIN_PASS="7redditGold"
FORGEJO_SSH_PORT="2222"
FORGEJO_DB_NAME=forgejo
FORGEJO_DB_USER=forgejo
FORGEJO_DB_PASS="forgejo_db_pass_2026"
# Forgejo OAuth2 (Authentik)
FORGEJO_OAUTH_CLIENT_ID="forgejo"
FORGEJO_OAUTH_CLIENT_SECRET="cUEZgNGfsmoPSvAjM8Y3xOLOAHOg5umaZJBFFPOIWLXSkdezEz5x1lXwV2mhYI89P1XfWihkBBrfnpctSYGeTMJO0DhNb6GYNCRZyHue61wvBJT4yIb68gdOY0NBkv4g"
# Authentik API Token (Contabo instance) - claude-api-token (regenerated 2026-02-16 for 2025.12 upgrade)
AUTHENTIK_API_TOKEN="YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w"
# Freq51.net MQTT Feeder (Meshtastic)
# Feeder: AIDA-N2 (Node ID: 27780c47)
# Malla: https://malla-aida-n2.freq51.net
# Meshview: https://meshview-aida-n2.freq51.net
FREQ51_MQTT_USER="aida-n2"
FREQ51_MQTT_PASS="handwash6unplan3ned-muMbo"
FREQ51_MQTT_BROKER="mqtt.freq51.net"
FREQ51_MQTT_PORT="1883"
# MeshMonitor (Meshtastic Mesh Monitoring)
# Location: utility CT 100 (192.168.1.100:8080)
# Tailscale: 100.64.0.1:8080
# Updated: 2026-02-04
MESHMONITOR_URL="http://192.168.1.100:8080"
MESHMONITOR_TAILSCALE_URL="http://100.64.0.1:8080"
MESHMONITOR_ADMIN_USER="admin"
MESHMONITOR_ADMIN_PASS="7redditGold"
MESHMONITOR_API_TOKEN="mm_v1_daa2d3fa3574f9bc291b831dc5925428"
# Vaultwarden (Password Manager)
# Deployed: 2026-02-05 on Contabo VPS
# URL: https://vault.echo6.co
# Local-login account (verified Phase 0a 2026-06-16): matt@echo6.co — master password is SET; Matt has confirmed he knows it.
# SSO_ONLY=false → local email + master-password login works independent of Authentik.
# Master password is Matt's own secret — NOT stored here (Vaultwarden keeps only the PBKDF2/Argon2 hash; unrecoverable).
VAULTWARDEN_URL="https://vault.echo6.co"
VAULTWARDEN_ADMIN_TOKEN="lHYCnZvETLirtRYCTICM6i6s2yYbMFjW3CkHwxnO1Brqe6vIq0zKNwF01IWB01"
VAULTWARDEN_ADMIN_URL="https://vault.echo6.co/admin"
# Vaultwarden SSO (Authentik)
VAULTWARDEN_OIDC_PROVIDER_ID=3
VAULTWARDEN_OIDC_CLIENT_ID="vaultwarden"
VAULTWARDEN_OIDC_CLIENT_SECRET="NALriNOTOkCjOA0w02MZWsoRUL9H0pT6eTNBYHsDueeL8sRkAEvG4RADrtcqWrIHi5Zur30CFTOQ3barjAnseQrzA4vrpVEmi1gJVqbWk9llkXOlixI8QMRSKNO4fXGi"
VAULTWARDEN_OIDC_ISSUER="https://auth.echo6.co/application/o/vaultwarden/"
# Proxmox VE SSO (Authentik OIDC)
# Deployed: 2026-02-05
# Realm: authentik
PROXMOX_OIDC_CLIENT_ID="proxmox"
PROXMOX_OIDC_CLIENT_SECRET="NlpHwyRhY9hMO6j6VtakSyt5M05a0x6WXMDwOFcqRcTVSBPKU8aOVRyIhYq6gJUiBW3e7Qy7z0zR0URZxK0e3BDN0i6hOAIBLZYGrx1jjjQFiIjlmk9e9R8ypOngtmVD"
PROXMOX_OIDC_ISSUER="https://auth.echo6.co/application/o/proxmox/"
PROXMOX_OIDC_PROVIDER_ID=4
# Headscale VPN (vpn.echo6.co)
# Deployed: 2026-02-05 on Contabo VPS
HEADSCALE_URL="https://vpn.echo6.co"
HEADSCALE_OIDC_CLIENT_ID="headscale"
HEADSCALE_OIDC_CLIENT_SECRET="lnwpTYerBQQkPIsNIYGpyFgM9zAbyAWOhvH27BSPY0JGZweAaNJWkgINMi6ePQtSdUDEHOWeUQynoDZynVbqfv68352PGcrU0nh8MWVM4Djif5akcCN9TtGPCiaLlMcL"
HEADSCALE_OIDC_ISSUER="https://auth.echo6.co/application/o/headscale/"
HEADSCALE_OIDC_PROVIDER_ID=5
# Headplane Admin UI (vpn.echo6.co/admin)
# First OIDC login gets Owner permissions
HEADPLANE_URL="https://vpn.echo6.co/admin"
HEADPLANE_OIDC_CLIENT_ID="headplane"
HEADPLANE_OIDC_CLIENT_SECRET="PYWFFLm8z8GiDA7keUbZMDFVyN6l11pBrgKeqTaLUyOsdSV7AZdW8JUXfGdcPI1jIE4ecaZy0WIyazKzFc89LtSHVkQKQHkBYpo5jRTmP5aXyiQmN8lSdqWlEEVEhk8m"
HEADPLANE_OIDC_ISSUER="https://auth.echo6.co/application/o/headplane/"
HEADPLANE_OIDC_PROVIDER_ID=6
HEADPLANE_API_KEY="hskey-api--6r1vFn_Dx0l-NVvEosw22IKFOvtjZ4Uc2U-IfWZQJOiyCtg7j-6q2UmQl5bDY4UlN9jWYaaKYma6"
# Immich Photo Management (immich.echo6.co)
# Deployed: 2026-02-11 on cloud CT 120 (192.168.1.182)
# Tailscale: 100.64.0.2
# NFS storage: pi-nas:/export/immich
IMMICH_URL="https://immich.echo6.co"
IMMICH_DB_PASSWORD="wKe7rUrxYkncA0Wo6jjx2AeI29i4UWn"
IMMICH_COMPOSE_PATH="/opt/immich/docker-compose.yml"
# Nextcloud AIO (nextcloud.echo6.co)
# Deployed: 2026-02-11 on cloud CT 121 (192.168.1.183)
# Tailscale: 100.64.0.11
# NFS storage: pi-nas:/export/nextcloud
NEXTCLOUD_URL="https://nextcloud.echo6.co"
NEXTCLOUD_AIO_UI="https://192.168.1.183:8080"
NEXTCLOUD_OIDC_PROVIDER_ID=8
NEXTCLOUD_OIDC_CLIENT_ID="nextcloud"
NEXTCLOUD_OIDC_CLIENT_SECRET="yw4NVaIcD65HLkVeC5QjiDjhsJeci2EHDuc7X1D1299AFhgVpbckWnRL7TkNfCUydJ1t11axUuWrp9OzygNdfmhkpev8hhA8QEFBxsTb95pwrFhvXTfydpHTrm3EpUdT"
NEXTCLOUD_OIDC_ISSUER="https://auth.echo6.co/application/o/nextcloud/"
# ARR Media Stack (arr VM on media node)
# Deployed: 2026-02-11 on media VM 105 (192.168.1.160)
# Tailscale: 100.64.0.18
ARR_VM_IP="192.168.1.160"
ARR_VM_TAILSCALE="100.64.0.18"
# Jellyfin (jellyfin.echo6.co)
JELLYFIN_URL="https://jellyfin.echo6.co"
JELLYFIN_OIDC_PROVIDER_ID=10
JELLYFIN_OIDC_CLIENT_ID="jellyfin"
JELLYFIN_OIDC_CLIENT_SECRET="gFiBubknaoKc9MPDrjm8pzC8acilxBPWn27vihsXnaiPJcwR3BiMlzhRtDs5gtpkAc5k7RAETwYOQdIJRVROpu6fZdE4H6NV8jBhr1khURnv1WIdmTM1809mTmAIkMt3"
JELLYFIN_OIDC_ISSUER="https://auth.echo6.co/application/o/jellyfin/"
# Jellyseer (requests.echo6.co)
JELLYSEER_URL="https://requests.echo6.co"
JELLYSEER_OIDC_PROVIDER_ID=11
JELLYSEER_OIDC_CLIENT_ID="jellyseer"
JELLYSEER_OIDC_CLIENT_SECRET="tOPLBA0xmoYjwMI5dTYxb6VllChgWhZyKEC8TRWgClTY30p0rpBZ24mmw0WJqcNZVeOvo3M3a7R2UHpHIiup76vudOQC9m863qgfe18pspyDGCW5PPaT9bvBTO8guyzg"
JELLYSEER_OIDC_ISSUER="https://auth.echo6.co/application/o/jellyseer/"
# PeerTube OIDC (Authentik)
PEERTUBE_OIDC_CLIENT_ID=1PCwaJNbOYWTxPNGwb15BX8YNXM68Bn3j1oZstfF
PEERTUBE_OIDC_CLIENT_SECRET=t9f2RJ6SQJNful64b5eRzBrhJs133yC8tXi8HvsdjPby9sCDPaU5Hf08qD7K8RJrOMQdg0g9UMeGc6ubIs5ikF6CykY8a2D6ZlqUtwLyA7YDUkZX8CJtntCZ23JOpT2w
PEERTUBE_OIDC_ISSUER=https://auth.echo6.co/application/o/peertube/
# PeerTube Instance
PEERTUBE_ROOT_PASSWORD=7redditGold
PEERTUBE_PG_PASSWORD=zMs0s9ZVHxL75h8R8QYmpefsLTbfdt
PEERTUBE_REDIS_PASSWORD=Rm1es6An15pAOOVh4d7hANgtptOkeJ
PEERTUBE_SECRET=d9c75c0fc6917e1c5c77e7066b36f34723483e0cca84a07f4774501fbfd99e3f
# Authentik No-Reply Email (no-reply@echo6.co)
# Created: 2026-02-16 for invitation/notification emails
AUTHENTIK_SMTP_USER=no-reply@echo6.co
AUTHENTIK_SMTP_PASS=NoReply2026Echo6
AUTHENTIK_SMTP_HOST=mail.echo6.co
AUTHENTIK_SMTP_PORT=587
# Matrix Synapse (matrix.echo6.co / element.echo6.co)
# Deployed: 2026-02-15 on Contabo VPS (5.189.158.149)
# Migrated from utility CT 108 to Contabo 2026-02-15
MATRIX_POSTGRES_PASSWORD=S2B3ay0CtGzNZEHvYkqnLNGOgcA1pwEk
MATRIX_OIDC_CLIENT_ID=93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T
MATRIX_OIDC_CLIENT_SECRET=Xa7CI44FXmQoGQxWQwctZR8OxET4FenueQuqH7SW0XbCSdy4Pl5hNA8lp6nxjEThwRXOSS09kmpejRMCHX8BITapsBvdzCfOn3wxjzwLp8qcVticYJdkLLC5lrGqD9aX
MATRIX_OIDC_ISSUER=https://auth.echo6.co/application/o/matrix/
MATRIX_ADMIN_USER=matt
MATRIX_ADMIN_PASSWORD=XzVMhUkumJwlTMVcQbLHePO
MATRIX_SYNAPSE_REGISTRATION_SHARED_SECRET=dK3b-vZ1+xKr5*jiNBTt2Te3h,T#:Xu.DoQPLoWBkIbTO=yJ2,
MATRIX_SYNAPSE_MACAROON_SECRET_KEY=+Vm:-&5X9ieUlG*;ztzK,^qDA@S#0Xl75Vb_-f:gtTyG;&df~p
MATRIX_MAS_SYNAPSE_SECRET=50fe2e183ed539fe630268a6af6bd59c447a09b86e4c2e939355c562b0a3c6d364761fdbf8f4f096e4500647965024d0
MATRIX_ARCHIVIST_BOT_PASSWORD=hm=8~gGG=@5OQv-*z*U*-ydOttcrm.L)c1;d%KF8?,?^nVRf
MATRIX_ARCHIVIST_BOT_ACCESS_TOKEN=mct_ueZmk9fWTUnqN8OfJKQ2tzMDDvHscg_LypEv2
# TAK Server (Contabo VPS — 5.189.158.149 / 100.64.0.1)
# Deployed: 2026-02-20 on Contabo VPS via Cloud-RF/tak-server wrapper
# Admin Portal: https://tak.echo6.co (Authentik SSO, port 8446 no client cert)
# Mutual TLS API: https://100.64.0.1:8443 (requires admin.p12)
# SIGIL Console: https://tak.echo6.co/sigil (Authentik SSO)
# Certs: /opt/tak-server-deploy/tak/certs/files/
TAK_URL="https://tak.echo6.co"
TAK_SIGIL_URL="https://tak.echo6.co/sigil"
TAK_ADMIN_USER=admin
TAK_ADMIN_PASSWORD=7redditGold-Tak1
TAK_POSTGRES_PASSWORD=4eEPvhC92yIMeh1!
TAK_CERT_PASSWORD=atakatak
TAK_INSTALL_PATH=/opt/tak-server-deploy
TAK_SIGIL_PORT=8990
# mt-isr (Meshtastic Node — ISR Site)
# Raspberry Pi Zero 2 W, Debian 13 (trixie), 416MB RAM, 115GB SD
# Local IP: 192.168.1.141 (WiFi, DHCP), Serial: /dev/ttyACM0
# IdahoMesh Tailscale: 100.100.0.5 (nebra user)
# WiFi hotspot: ISR-MESH (192.168.4.0/24), G2 at 192.168.4.241
# meshtasticd removed (not needed — G2 managed via WiFi TCP, not local daemon)
# Station G2 managed via TCP at 192.168.4.241 through WiFi bridge
MT_ISR_IP="192.168.1.141"
MT_ISR_SSH_USER="isr"
MT_ISR_SSH_PASS="UfPsfwyMIUIKb1"
# GitHub (github.com/zvx-echo6)
# Fine-grained PAT. Named for advBBS but its scope is wider than that --
# it also has push access to the meshwars repo, confirmed 2026-08-25 when a
# deploy used it successfully. Check the token's actual repo list on GitHub
# before assuming it can or cannot reach something.
GITHUB_PAT=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
# MMUD — Mesh Multi-User Dungeon (mmud.echo6.co)
# Deployed: 2026-02-25 on utility (CT 109 game daemon, CT 111-116 SIM nodes)
# Dashboard: https://mmud.echo6.co
# Admin: https://mmud.echo6.co/admin
MMUD_URL="https://mmud.echo6.co"
MMUD_WEB_SECRET="194bc59df6e1734d73cacaf1c82f3298fac207dc23af6d6798b9bd3241c196a9"
MMUD_ADMIN_PASSWORD="5lJOhU87uEtubBe1hRcPGg"
MMUD_COMPOSE_PATH="/opt/mmud/docker-compose.yml"
# matt-desktop (Windows PC)
# Local IP: 192.168.1.111, Tailscale: 100.64.0.10
# SSH requires: -o PreferredAuthentications=password -o PubkeyAuthentication=no
MATT_DESKTOP_IP="192.168.1.111"
MATT_DESKTOP_TS="100.64.0.10"
MATT_DESKTOP_SSH_USER="administrator"
MATT_DESKTOP_SSH_PASS="Qw1290opzx"
# Echo6 Matrix Agents
# Space: echo6-ops
# Cortex Agent: @cortex:echo6.co on cortex (systemd: echo6-agent.service)
# Contabo Agent: @contabo:echo6.co on Contabo (systemd: echo6-agent.service)
# MAS User IDs: cortex=01KKY68CV3KSJAPG11SZE3T0SV, contabo=01KKY68FSS6B6T56C8BS94K2H5
MATRIX_CORTEX_USER="@cortex:echo6.co"
MATRIX_CORTEX_PASSWORD="~vfnqEOHLcjbMmX1-t7GXZrCqyqwQn5d"
MATRIX_CONTABO_USER="@contabo:echo6.co"
MATRIX_CONTABO_PASSWORD="4i,ysuy2ZWk*9ndNICOrp4cIMpE(Teit"
# Legacy agent (to be deactivated)
MATRIX_AGENT_USER="@agent:echo6.co"
MATRIX_AGENT_PASSWORD="WBYVL-o)VO%uV^4YHWni=oDNC76BPaN)"
MATRIX_AGENT_MAS_ID="01KKX88ARGK0BTA1JMB2QVAW4C"
# Scribd (scribd.com)
SCRIBD_USER="matt@echo6.co"
SCRIBD_PASS="7redditGold"
# Anna's Archive (annas-archive.gs / .pk / .gd)
# Member API key for fast downloads (bypasses DDoS-Guard)
AA_KEY="y7ADrA9VPZb6ix7CWZbnJc6BiUFMs"
# Internet Archive (archive.org)
IA_USER="matt@echo6.co"
IA_PASS="7redditGold"
IA_S3_ACCESS="vw9g0hyfALvjsOWl"
IA_S3_SECRET="nfY7TOe8mweBWaWo"
IA_SCREENNAME="matecho"
# TomTom Maps API
TOMTOM_API_KEY=4IvvwRYgTL5LCdWVVVW1iyAL6o9CD3wT
# Google Places API
GOOGLE_PLACES_KEY=AIzaSyB6gJ6RJisarF29GtJks_9SUHLm5o1d_tY
# Central LXC — Data Hub (192.168.1.104 / 100.64.0.12 / central.echo6.mesh)
# PostgreSQL 16 + TimescaleDB 2.27 + PostGIS 3.4
CENTRAL_PG_HOST="192.168.1.104"
CENTRAL_PG_PORT="5432"
CENTRAL_PG_DATABASE="central"
CENTRAL_PG_USER="central"
CENTRAL_PG_PASSWORD="3LNVFQJHsK3e7dOcAdvK3oS6d70f"
CENTRAL_PG_DSN="postgresql://central:3LNVFQJHsK3e7dOcAdvK3oS6d70f@localhost/central"
# Central Data Hub Master Key
CENTRAL_MASTER_KEY=jfooDHnsS8n5J61bDzmiq6u7QpJhUoEQ/BVk1Sj3PqE=
# Central GUI operator login (http://central.echo6.mesh:8000)
CENTRAL_OPERATOR_USER="admin"
CENTRAL_OPERATOR_PASS="7redditGold"
# Recon VM (CT 1130) — padus PostgreSQL (192.168.1.130 / recon-vm)
# PostgreSQL 16, opened for LAN 2026-05-24 — scram-sha-256, allow 192.168.1.0/24
RECON_PG_HOST=192.168.1.130
RECON_PG_PORT=5432
RECON_PG_DATABASE=padus
RECON_PG_USER=zvx
RECON_PG_PASSWORD=ce70f1d52fad2264427d4db9b3c1036568c3b6ea0dd80e00
# 511 Idaho Traffic API (511.idaho.gov)
IDAHO_511_API_KEY=5af12fb5fdc743e0ae0b041bc4c47f2d
# edge2 PVE host root@pam password (set 2026-06-16 — for PDM remote-add + break-glass)
EDGE2_ROOT_PASSWORD=rAWdarLXCsTVnEQCmGxX
# intermountainmesh.com WordPress (edge2 CT 101, 10.10.10.11)
# Deployed: 2026-06-16
WP_IMESH_DB_ROOT_PW=BenH5nw7Y25k2xtrXfXcapt4xLzsQC2W
WP_IMESH_DB_USER=wp_user
WP_IMESH_DB_PASSWORD=9cnwfhQzkdyfU8C3yodIXu_IA1PUOawr
WP_IMESH_DB_NAME=wordpress
# WordPress admin credentials: set during browser setup wizard at https://intermountainmesh.com
# Proxmox Datacenter Manager (PDM) — Break-glass local admin (edge2 CT 100, 100.64.0.28:8443)
# PAM realm: Linux user on CT100, independent of Authentik SSO
# Backup taken: /root/access.bak-2026-06-20 (inside CT100)
PDM_URL="https://pdm.echo6.co"
PDM_BREAKGLASS_USER="admin@pam"
PDM_BREAKGLASS_PASS="7redditGold"
# Conduit GUI admin login (http://100.64.0.12:8010/login) — CT 104
# Recreated 2026-07-15 via conduit-admin create-operator (full DB rebuild; old operator gone)
CONDUIT_GUI_URL="http://100.64.0.12:8010/login"
CONDUIT_GUI_USER="admin"
CONDUIT_GUI_PASS="7redditGold"
# ARGUS web UI (https://argus.echo6.co) — utility CT 101 Caddy, HTTP basic auth
# Backend: ARGUS app CT 103 tailnet 100.64.0.25:8090. Added 2026-07-03.
# Only the bcrypt hash lives in the Caddyfile; this is the plaintext.
ARGUS_WEB_URL="https://argus.echo6.co"
ARGUS_WEB_USER="idaho"
ARGUS_WEB_PASS="lads"
# MeshWars territory control game — LXC 119 (mesh-territory) on utility host
# https://meshwars.com terminates TLS on CT 101 Caddy, proxies to 192.168.1.219:8090
# All three values below are set in /home/zvx/meshwars/.env on CT 119 (gitignored);
# none can be recovered from the app database, they are stored hashed. Recorded 2026-08-18.
MESHWARS_URL="https://meshwars.com"
MESHWARS_ADMIN_URL="https://meshwars.com/admin"
# Gates the admin interface. When empty, every admin route 404s instead of 403 —
# an unconfigured install doesn't advertise that an admin door exists.
MESHWARS_ADMIN_TOKEN="GKe_dJUuAdeXeB1JNVr49rw1KfyLRQeyRTuM5rCyUek"
# Gates registration at /join. Empty disables registration entirely.
MESHWARS_JOIN_INVITE_CODE="meshwars2026"
# Owner's own player key, used by the MeshMapper wardriving app to post positions.
# Stored only as a hash server-side — cannot be recovered, only reissued.
MESHWARS_PLAYER_MALICE_API_KEY="vPCw_sZFL-PuJh30brM9fD9eYACO00JliGv2zIwuVgw"

402
credentials.bak-meshwars
View file

@ -1,402 +0,0 @@
# Infrastructure API Credentials
# IMPORTANT: Keep this file secure and never commit to git
# SSH Access (all servers)
SSH_USER=zvx
SSH_PASSWORD=7redditGold
ROOT_PASSWORD=7redditGold
# GoDaddy DNS API
GODADDY_API_KEY=dKiSoC24ZLTR_3KkMjFyYrzgChk1JHjNLnU
GODADDY_API_SECRET=ExCGBASgSrN4A2mP4daU4G
# Authentik SSO Platform (https://auth.echo6.co) — migrated to edge2 CT 105 / 10.10.10.23 / tailnet 100.64.0.36 on 2026-06-18
AUTHENTIK_API_TOKEN=YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w
AUTHENTIK_URL=https://auth.echo6.co
# Admin credentials for web UI/API access
AUTHENTIK_ADMIN_USER=akadmin
AUTHENTIK_ADMIN_PASSWORD=Mue8FI4sRe2ihRzsmypPnQlpDiae6hqHXZafV2n5CTNPJT1JaS6oqYZzbeGu
# Wiki.js Documentation Platform (https://docs.echo6.co)
WIKIJS_ADMIN_EMAIL=akadmin@echo6.co
WIKIJS_ADMIN_PASSWORD=xh5Q+hWzUe5EqvBQ1tWULlOiPMZX
WIKIJS_URL=https://docs.echo6.co
# Forgejo Git Forge (https://forge.echo6.co)
FORGEJO_API_TOKEN=043fd2df4731a2bb8a2b8352018031bd454fe9a1
FORGEJO_URL=https://forge.echo6.co
# Usage Notes:
# - GoDaddy API: Used for automated DNS record management
# - Authentik API: Used for SSO provider/application configuration
# - Forgejo API: Used for repository mirroring and automation (ALL PERMISSIONS)
# - File Location: /home/zvx/projects/selfhosting/.credentials
# - For agents: Source this file with: source /home/zvx/projects/selfhosting/.credentials
WIKIJS_API_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhcGkiOjIsImdycCI6MSwiaWF0IjoxNzYzOTYxNzUxLCJleHAiOjE3OTU1MTkzNTEsImF1ZCI6InVybjp3aWtpLmpzIiwiaXNzIjoidXJuOndpa2kuanMifQ.LR4qDHT1d8PgwVCo7o8hCbAK19vas5vnROa2IT8InseHh7RwJbnyfc4u71PMzDcIoYJ1QHVeAxhv4SJbkYj-F7b525M63xGuji1pUpxkEaxbZe1iFeQVojSvPvBcw0ZCqMzKtgYNivpSnn9psQulDSFD7Kq63FVte1ABMHeh-s-Mfeqn4pzmgRAMRHZC9l6MSA_cTZ97gxeJCKn9Jyh0LyrQJXYeL0gFG5FvAwOMYDf5fiAmz4Ocyxg4ZMz3q-6Ao9dfB41zbD5WbYwI0qm7wOBDRj3-1-JjMrPJrgYpUOqEuOH1SRDt7VUrOvbxqE7AAM0J8o-7pnVNw1HaseqEiQ
# Obsidian LiveSync (https://notes.echo6.co)
# CouchDB + provisioner on Contabo (5.189.158.149), JWT auth
LIVESYNC_COUCHDB_USER=admin
LIVESYNC_COUCHDB_PASSWORD=kGJkZzoVc5eMiu0rGKJgJNyq
LIVESYNC_PROXY_SECRET=d41933ea00ffb620fa4bb96e0177b3869998c698f5dd7892b86a33fbf8e5aff9
LIVESYNC_URL=https://notes.echo6.co
# Config location: Contabo:/opt/livesync/
# Open WebUI SSO (ai.echo6.co)
OPEN_WEBUI_OAUTH_CLIENT_ID=open-webui
OPEN_WEBUI_OAUTH_CLIENT_SECRET=MfajNL6QZDfDOKprBdYzu3rDya2BoiB5siWt5cYQH1o3zif8ivZ2ZF2HehE3UqLZq2nflwVdYZNhw4tkWQj8kF0dLGA0qDYbBVtUcvtj2AQKQTqAEdRfQF0Oi3slL9OI
OPEN_WEBUI_AUTHENTIK_PROVIDER_ID=14
# Open WebUI API Key (for content sync)
OPENWEBUI_API_KEY=sk-dbf4c9e6ac915d8151f84bf09e51cf2c03e943918adb1737
OPENWEBUI_API_URL=http://192.168.1.239:3000
# PeerTube (stream.echo6.co)
PEERTUBE_URL="https://stream.echo6.co"
PEERTUBE_ADMIN_USER="root"
PEERTUBE_ADMIN_PASSWORD="7redditGold"
PEERTUBE_ADMIN_EMAIL="admin@echo6.co"
# Echo6 Portal - https://echo6.co
# Deployed: Echo6-Cloud (192.168.1.241:3002)
ECHO6_PORTAL_ADMIN_USER=admin
ECHO6_PORTAL_ADMIN_EMAIL=admin@echo6.co
ECHO6_PORTAL_ADMIN_PASS=7redditGold
# Authentik OAuth2 Provider
ECHO6_PORTAL_OIDC_CLIENT_ID=echo6-portal
ECHO6_PORTAL_OIDC_CLIENT_SECRET=15vviNjbzbrtG3WvZxkKrLZv97tcsdzHqWcExp4HarlFlxWInZcKSLixdDWwod67zYxYBDqxkqNR4PNIK0A0XhJSGG6LLK7UHwI1xKeUQt7Dx6bVnDZvqkGP28ewYb4c
ECHO6_PORTAL_OIDC_ISSUER=https://auth.echo6.co/application/o/echo6-portal/
# Stalwart Mail Server — DECOMMISSIONED
# Was deployed 2025-12-19, removed. Authentik provider PK 53 may still exist.
# Credentials retained for reference only.
# STALWART_ADMIN_USER=admin
# STALWART_ADMIN_PASS=ECu0Q4zsQN
# NordVPN (PeerTube Pipeline IP Rotation)
# Installed on media CT 110 (192.168.1.170)
# Service: nordvpnd.service (enabled, auto-start)
# Token: logged in via `nordvpn login --token <token>` — token not stored in file
# Note: NordVPN token was entered interactively, re-login if needed
NORDVPN_LOCATION="CT 110 (media)"
# edge1 SSH Access (rebuilt Contabo VPS, mail-only, 2026-06-19)
# Server: edge1 — Debian 12 + Proxmox 8.4.19, mail-only role
# Public IP: 5.189.158.149 | Tailscale: 100.64.0.40 (node: contabo)
# SSH key: ~/.ssh/contabo2_ed25519 (same key as before rebuild)
# Box password (unrotated): 7redditGold
# ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149
EDGE1_SSH_USER=root
EDGE1_SSH_KEY=~/.ssh/contabo2_ed25519
EDGE1_SSH_PASS=7redditGold
EDGE1_PUBLIC_IP=5.189.158.149
EDGE1_TAILSCALE_IP=100.64.0.40
# Mailcow (mail.echo6.co) — now on edge1 CT 101 (10.10.10.2), updated commit 52a41b4d (SOGo 5.12.8)
# Deployed: 2026-06-19 rebuild (privileged LXC on edge1 Proxmox)
# Access: via host Caddy (mail/autodiscover/autoconfig.echo6.co) + host DNAT (ports 25/465/587/110/143/993/995/4190 → 10.10.10.2)
# Admin UI: https://mail.echo6.co (same as before)
# mail_crypt keys (dovecot crypt volume: data/conf/dovecot/):
# Primary key: ecprivkey.pem (original; must be preserved across rebuilds)
# Secondary key: ecprivkey.pem.fresh-20260619 (generated by restore; loaded alongside primary)
# Config: data/conf/dovecot/extra.conf (update-safe, not dovecot.conf)
# Both keys loaded — required to decrypt old mail encrypted with primary and new mail with secondary
MAILCOW_ADMIN_USER=admin
MAILCOW_ADMIN_PASS=7redditGold
# RECON Pipeline Email Alerts (recon@echo6.co)
# Deployed: 2026-02-01 on Media-Server for pipeline monitoring
RECON_SMTP_USER=recon@echo6.co
RECON_SMTP_PASS=LezU3TJ2W8JFj0hXHcJe
RECON_SMTP_HOST=mail.echo6.co
RECON_SMTP_PORT=587
# Forgejo (forge.echo6.co) - Deployed 2026-02-04
# Migrated to edge2 CT 103 (10.10.10.21 / tailnet 100.64.0.34) on 2026-06-16; SSH via Contabo iptables DNAT (forgejo-ssh-dnat.service).
FORGEJO_URL="https://forge.echo6.co"
FORGEJO_ADMIN_USER="matt"
FORGEJO_ADMIN_PASS="7redditGold"
FORGEJO_SSH_PORT="2222"
FORGEJO_DB_NAME=forgejo
FORGEJO_DB_USER=forgejo
FORGEJO_DB_PASS="forgejo_db_pass_2026"
# Forgejo OAuth2 (Authentik)
FORGEJO_OAUTH_CLIENT_ID="forgejo"
FORGEJO_OAUTH_CLIENT_SECRET="cUEZgNGfsmoPSvAjM8Y3xOLOAHOg5umaZJBFFPOIWLXSkdezEz5x1lXwV2mhYI89P1XfWihkBBrfnpctSYGeTMJO0DhNb6GYNCRZyHue61wvBJT4yIb68gdOY0NBkv4g"
# Authentik API Token (Contabo instance) - claude-api-token (regenerated 2026-02-16 for 2025.12 upgrade)
AUTHENTIK_API_TOKEN="YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w"
# Freq51.net MQTT Feeder (Meshtastic)
# Feeder: AIDA-N2 (Node ID: 27780c47)
# Malla: https://malla-aida-n2.freq51.net
# Meshview: https://meshview-aida-n2.freq51.net
FREQ51_MQTT_USER="aida-n2"
FREQ51_MQTT_PASS="handwash6unplan3ned-muMbo"
FREQ51_MQTT_BROKER="mqtt.freq51.net"
FREQ51_MQTT_PORT="1883"
# MeshMonitor (Meshtastic Mesh Monitoring)
# Location: utility CT 100 (192.168.1.100:8080)
# Tailscale: 100.64.0.1:8080
# Updated: 2026-02-04
MESHMONITOR_URL="http://192.168.1.100:8080"
MESHMONITOR_TAILSCALE_URL="http://100.64.0.1:8080"
MESHMONITOR_ADMIN_USER="admin"
MESHMONITOR_ADMIN_PASS="7redditGold"
MESHMONITOR_API_TOKEN="mm_v1_daa2d3fa3574f9bc291b831dc5925428"
# Vaultwarden (Password Manager)
# Deployed: 2026-02-05 on Contabo VPS
# URL: https://vault.echo6.co
# Local-login account (verified Phase 0a 2026-06-16): matt@echo6.co — master password is SET; Matt has confirmed he knows it.
# SSO_ONLY=false → local email + master-password login works independent of Authentik.
# Master password is Matt's own secret — NOT stored here (Vaultwarden keeps only the PBKDF2/Argon2 hash; unrecoverable).
VAULTWARDEN_URL="https://vault.echo6.co"
VAULTWARDEN_ADMIN_TOKEN="lHYCnZvETLirtRYCTICM6i6s2yYbMFjW3CkHwxnO1Brqe6vIq0zKNwF01IWB01"
VAULTWARDEN_ADMIN_URL="https://vault.echo6.co/admin"
# Vaultwarden SSO (Authentik)
VAULTWARDEN_OIDC_PROVIDER_ID=3
VAULTWARDEN_OIDC_CLIENT_ID="vaultwarden"
VAULTWARDEN_OIDC_CLIENT_SECRET="NALriNOTOkCjOA0w02MZWsoRUL9H0pT6eTNBYHsDueeL8sRkAEvG4RADrtcqWrIHi5Zur30CFTOQ3barjAnseQrzA4vrpVEmi1gJVqbWk9llkXOlixI8QMRSKNO4fXGi"
VAULTWARDEN_OIDC_ISSUER="https://auth.echo6.co/application/o/vaultwarden/"
# Proxmox VE SSO (Authentik OIDC)
# Deployed: 2026-02-05
# Realm: authentik
PROXMOX_OIDC_CLIENT_ID="proxmox"
PROXMOX_OIDC_CLIENT_SECRET="NlpHwyRhY9hMO6j6VtakSyt5M05a0x6WXMDwOFcqRcTVSBPKU8aOVRyIhYq6gJUiBW3e7Qy7z0zR0URZxK0e3BDN0i6hOAIBLZYGrx1jjjQFiIjlmk9e9R8ypOngtmVD"
PROXMOX_OIDC_ISSUER="https://auth.echo6.co/application/o/proxmox/"
PROXMOX_OIDC_PROVIDER_ID=4
# Headscale VPN (vpn.echo6.co)
# Deployed: 2026-02-05 on Contabo VPS
HEADSCALE_URL="https://vpn.echo6.co"
HEADSCALE_OIDC_CLIENT_ID="headscale"
HEADSCALE_OIDC_CLIENT_SECRET="lnwpTYerBQQkPIsNIYGpyFgM9zAbyAWOhvH27BSPY0JGZweAaNJWkgINMi6ePQtSdUDEHOWeUQynoDZynVbqfv68352PGcrU0nh8MWVM4Djif5akcCN9TtGPCiaLlMcL"
HEADSCALE_OIDC_ISSUER="https://auth.echo6.co/application/o/headscale/"
HEADSCALE_OIDC_PROVIDER_ID=5
# Headplane Admin UI (vpn.echo6.co/admin)
# First OIDC login gets Owner permissions
HEADPLANE_URL="https://vpn.echo6.co/admin"
HEADPLANE_OIDC_CLIENT_ID="headplane"
HEADPLANE_OIDC_CLIENT_SECRET="PYWFFLm8z8GiDA7keUbZMDFVyN6l11pBrgKeqTaLUyOsdSV7AZdW8JUXfGdcPI1jIE4ecaZy0WIyazKzFc89LtSHVkQKQHkBYpo5jRTmP5aXyiQmN8lSdqWlEEVEhk8m"
HEADPLANE_OIDC_ISSUER="https://auth.echo6.co/application/o/headplane/"
HEADPLANE_OIDC_PROVIDER_ID=6
HEADPLANE_API_KEY="hskey-api--6r1vFn_Dx0l-NVvEosw22IKFOvtjZ4Uc2U-IfWZQJOiyCtg7j-6q2UmQl5bDY4UlN9jWYaaKYma6"
# Immich Photo Management (immich.echo6.co)
# Deployed: 2026-02-11 on cloud CT 120 (192.168.1.182)
# Tailscale: 100.64.0.2
# NFS storage: pi-nas:/export/immich
IMMICH_URL="https://immich.echo6.co"
IMMICH_DB_PASSWORD="wKe7rUrxYkncA0Wo6jjx2AeI29i4UWn"
IMMICH_COMPOSE_PATH="/opt/immich/docker-compose.yml"
# Nextcloud AIO (nextcloud.echo6.co)
# Deployed: 2026-02-11 on cloud CT 121 (192.168.1.183)
# Tailscale: 100.64.0.11
# NFS storage: pi-nas:/export/nextcloud
NEXTCLOUD_URL="https://nextcloud.echo6.co"
NEXTCLOUD_AIO_UI="https://192.168.1.183:8080"
NEXTCLOUD_OIDC_PROVIDER_ID=8
NEXTCLOUD_OIDC_CLIENT_ID="nextcloud"
NEXTCLOUD_OIDC_CLIENT_SECRET="yw4NVaIcD65HLkVeC5QjiDjhsJeci2EHDuc7X1D1299AFhgVpbckWnRL7TkNfCUydJ1t11axUuWrp9OzygNdfmhkpev8hhA8QEFBxsTb95pwrFhvXTfydpHTrm3EpUdT"
NEXTCLOUD_OIDC_ISSUER="https://auth.echo6.co/application/o/nextcloud/"
# ARR Media Stack (arr VM on media node)
# Deployed: 2026-02-11 on media VM 105 (192.168.1.160)
# Tailscale: 100.64.0.18
ARR_VM_IP="192.168.1.160"
ARR_VM_TAILSCALE="100.64.0.18"
# Jellyfin (jellyfin.echo6.co)
JELLYFIN_URL="https://jellyfin.echo6.co"
JELLYFIN_OIDC_PROVIDER_ID=10
JELLYFIN_OIDC_CLIENT_ID="jellyfin"
JELLYFIN_OIDC_CLIENT_SECRET="gFiBubknaoKc9MPDrjm8pzC8acilxBPWn27vihsXnaiPJcwR3BiMlzhRtDs5gtpkAc5k7RAETwYOQdIJRVROpu6fZdE4H6NV8jBhr1khURnv1WIdmTM1809mTmAIkMt3"
JELLYFIN_OIDC_ISSUER="https://auth.echo6.co/application/o/jellyfin/"
# Jellyseer (requests.echo6.co)
JELLYSEER_URL="https://requests.echo6.co"
JELLYSEER_OIDC_PROVIDER_ID=11
JELLYSEER_OIDC_CLIENT_ID="jellyseer"
JELLYSEER_OIDC_CLIENT_SECRET="tOPLBA0xmoYjwMI5dTYxb6VllChgWhZyKEC8TRWgClTY30p0rpBZ24mmw0WJqcNZVeOvo3M3a7R2UHpHIiup76vudOQC9m863qgfe18pspyDGCW5PPaT9bvBTO8guyzg"
JELLYSEER_OIDC_ISSUER="https://auth.echo6.co/application/o/jellyseer/"
# PeerTube OIDC (Authentik)
PEERTUBE_OIDC_CLIENT_ID=1PCwaJNbOYWTxPNGwb15BX8YNXM68Bn3j1oZstfF
PEERTUBE_OIDC_CLIENT_SECRET=t9f2RJ6SQJNful64b5eRzBrhJs133yC8tXi8HvsdjPby9sCDPaU5Hf08qD7K8RJrOMQdg0g9UMeGc6ubIs5ikF6CykY8a2D6ZlqUtwLyA7YDUkZX8CJtntCZ23JOpT2w
PEERTUBE_OIDC_ISSUER=https://auth.echo6.co/application/o/peertube/
# PeerTube Instance
PEERTUBE_ROOT_PASSWORD=7redditGold
PEERTUBE_PG_PASSWORD=zMs0s9ZVHxL75h8R8QYmpefsLTbfdt
PEERTUBE_REDIS_PASSWORD=Rm1es6An15pAOOVh4d7hANgtptOkeJ
PEERTUBE_SECRET=d9c75c0fc6917e1c5c77e7066b36f34723483e0cca84a07f4774501fbfd99e3f
# Authentik No-Reply Email (no-reply@echo6.co)
# Created: 2026-02-16 for invitation/notification emails
AUTHENTIK_SMTP_USER=no-reply@echo6.co
AUTHENTIK_SMTP_PASS=NoReply2026Echo6
AUTHENTIK_SMTP_HOST=mail.echo6.co
AUTHENTIK_SMTP_PORT=587
# Matrix Synapse (matrix.echo6.co / element.echo6.co)
# Deployed: 2026-02-15 on Contabo VPS (5.189.158.149)
# Migrated from utility CT 108 to Contabo 2026-02-15
MATRIX_POSTGRES_PASSWORD=S2B3ay0CtGzNZEHvYkqnLNGOgcA1pwEk
MATRIX_OIDC_CLIENT_ID=93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T
MATRIX_OIDC_CLIENT_SECRET=Xa7CI44FXmQoGQxWQwctZR8OxET4FenueQuqH7SW0XbCSdy4Pl5hNA8lp6nxjEThwRXOSS09kmpejRMCHX8BITapsBvdzCfOn3wxjzwLp8qcVticYJdkLLC5lrGqD9aX
MATRIX_OIDC_ISSUER=https://auth.echo6.co/application/o/matrix/
MATRIX_ADMIN_USER=matt
MATRIX_ADMIN_PASSWORD=XzVMhUkumJwlTMVcQbLHePO
MATRIX_SYNAPSE_REGISTRATION_SHARED_SECRET=dK3b-vZ1+xKr5*jiNBTt2Te3h,T#:Xu.DoQPLoWBkIbTO=yJ2,
MATRIX_SYNAPSE_MACAROON_SECRET_KEY=+Vm:-&5X9ieUlG*;ztzK,^qDA@S#0Xl75Vb_-f:gtTyG;&df~p
MATRIX_MAS_SYNAPSE_SECRET=50fe2e183ed539fe630268a6af6bd59c447a09b86e4c2e939355c562b0a3c6d364761fdbf8f4f096e4500647965024d0
MATRIX_ARCHIVIST_BOT_PASSWORD=hm=8~gGG=@5OQv-*z*U*-ydOttcrm.L)c1;d%KF8?,?^nVRf
MATRIX_ARCHIVIST_BOT_ACCESS_TOKEN=mct_ueZmk9fWTUnqN8OfJKQ2tzMDDvHscg_LypEv2
# TAK Server (Contabo VPS — 5.189.158.149 / 100.64.0.1)
# Deployed: 2026-02-20 on Contabo VPS via Cloud-RF/tak-server wrapper
# Admin Portal: https://tak.echo6.co (Authentik SSO, port 8446 no client cert)
# Mutual TLS API: https://100.64.0.1:8443 (requires admin.p12)
# SIGIL Console: https://tak.echo6.co/sigil (Authentik SSO)
# Certs: /opt/tak-server-deploy/tak/certs/files/
TAK_URL="https://tak.echo6.co"
TAK_SIGIL_URL="https://tak.echo6.co/sigil"
TAK_ADMIN_USER=admin
TAK_ADMIN_PASSWORD=7redditGold-Tak1
TAK_POSTGRES_PASSWORD=4eEPvhC92yIMeh1!
TAK_CERT_PASSWORD=atakatak
TAK_INSTALL_PATH=/opt/tak-server-deploy
TAK_SIGIL_PORT=8990
# mt-isr (Meshtastic Node — ISR Site)
# Raspberry Pi Zero 2 W, Debian 13 (trixie), 416MB RAM, 115GB SD
# Local IP: 192.168.1.141 (WiFi, DHCP), Serial: /dev/ttyACM0
# IdahoMesh Tailscale: 100.100.0.5 (nebra user)
# WiFi hotspot: ISR-MESH (192.168.4.0/24), G2 at 192.168.4.241
# meshtasticd removed (not needed — G2 managed via WiFi TCP, not local daemon)
# Station G2 managed via TCP at 192.168.4.241 through WiFi bridge
MT_ISR_IP="192.168.1.141"
MT_ISR_SSH_USER="isr"
MT_ISR_SSH_PASS="UfPsfwyMIUIKb1"
# GitHub (github.com/zvx-echo6)
# Fine-grained PAT for advBBS repo push
GITHUB_PAT=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
# MMUD — Mesh Multi-User Dungeon (mmud.echo6.co)
# Deployed: 2026-02-25 on utility (CT 109 game daemon, CT 111-116 SIM nodes)
# Dashboard: https://mmud.echo6.co
# Admin: https://mmud.echo6.co/admin
MMUD_URL="https://mmud.echo6.co"
MMUD_WEB_SECRET="194bc59df6e1734d73cacaf1c82f3298fac207dc23af6d6798b9bd3241c196a9"
MMUD_ADMIN_PASSWORD="5lJOhU87uEtubBe1hRcPGg"
MMUD_COMPOSE_PATH="/opt/mmud/docker-compose.yml"
# matt-desktop (Windows PC)
# Local IP: 192.168.1.111, Tailscale: 100.64.0.10
# SSH requires: -o PreferredAuthentications=password -o PubkeyAuthentication=no
MATT_DESKTOP_IP="192.168.1.111"
MATT_DESKTOP_TS="100.64.0.10"
MATT_DESKTOP_SSH_USER="administrator"
MATT_DESKTOP_SSH_PASS="Qw1290opzx"
# Echo6 Matrix Agents
# Space: echo6-ops
# Cortex Agent: @cortex:echo6.co on cortex (systemd: echo6-agent.service)
# Contabo Agent: @contabo:echo6.co on Contabo (systemd: echo6-agent.service)
# MAS User IDs: cortex=01KKY68CV3KSJAPG11SZE3T0SV, contabo=01KKY68FSS6B6T56C8BS94K2H5
MATRIX_CORTEX_USER="@cortex:echo6.co"
MATRIX_CORTEX_PASSWORD="~vfnqEOHLcjbMmX1-t7GXZrCqyqwQn5d"
MATRIX_CONTABO_USER="@contabo:echo6.co"
MATRIX_CONTABO_PASSWORD="4i,ysuy2ZWk*9ndNICOrp4cIMpE(Teit"
# Legacy agent (to be deactivated)
MATRIX_AGENT_USER="@agent:echo6.co"
MATRIX_AGENT_PASSWORD="WBYVL-o)VO%uV^4YHWni=oDNC76BPaN)"
MATRIX_AGENT_MAS_ID="01KKX88ARGK0BTA1JMB2QVAW4C"
# Scribd (scribd.com)
SCRIBD_USER="matt@echo6.co"
SCRIBD_PASS="7redditGold"
# Anna's Archive (annas-archive.gs / .pk / .gd)
# Member API key for fast downloads (bypasses DDoS-Guard)
AA_KEY="y7ADrA9VPZb6ix7CWZbnJc6BiUFMs"
# Internet Archive (archive.org)
IA_USER="matt@echo6.co"
IA_PASS="7redditGold"
IA_S3_ACCESS="vw9g0hyfALvjsOWl"
IA_S3_SECRET="nfY7TOe8mweBWaWo"
IA_SCREENNAME="matecho"
# TomTom Maps API
TOMTOM_API_KEY=4IvvwRYgTL5LCdWVVVW1iyAL6o9CD3wT
# Google Places API
GOOGLE_PLACES_KEY=AIzaSyB6gJ6RJisarF29GtJks_9SUHLm5o1d_tY
# Central LXC — Data Hub (192.168.1.104 / 100.64.0.12 / central.echo6.mesh)
# PostgreSQL 16 + TimescaleDB 2.27 + PostGIS 3.4
CENTRAL_PG_HOST="192.168.1.104"
CENTRAL_PG_PORT="5432"
CENTRAL_PG_DATABASE="central"
CENTRAL_PG_USER="central"
CENTRAL_PG_PASSWORD="3LNVFQJHsK3e7dOcAdvK3oS6d70f"
CENTRAL_PG_DSN="postgresql://central:3LNVFQJHsK3e7dOcAdvK3oS6d70f@localhost/central"
# Central Data Hub Master Key
CENTRAL_MASTER_KEY=jfooDHnsS8n5J61bDzmiq6u7QpJhUoEQ/BVk1Sj3PqE=
# Central GUI operator login (http://central.echo6.mesh:8000)
CENTRAL_OPERATOR_USER="admin"
CENTRAL_OPERATOR_PASS="7redditGold"
# Recon VM (CT 1130) — padus PostgreSQL (192.168.1.130 / recon-vm)
# PostgreSQL 16, opened for LAN 2026-05-24 — scram-sha-256, allow 192.168.1.0/24
RECON_PG_HOST=192.168.1.130
RECON_PG_PORT=5432
RECON_PG_DATABASE=padus
RECON_PG_USER=zvx
RECON_PG_PASSWORD=ce70f1d52fad2264427d4db9b3c1036568c3b6ea0dd80e00
# 511 Idaho Traffic API (511.idaho.gov)
IDAHO_511_API_KEY=5af12fb5fdc743e0ae0b041bc4c47f2d
# edge2 PVE host root@pam password (set 2026-06-16 — for PDM remote-add + break-glass)
EDGE2_ROOT_PASSWORD=rAWdarLXCsTVnEQCmGxX
# intermountainmesh.com WordPress (edge2 CT 101, 10.10.10.11)
# Deployed: 2026-06-16
WP_IMESH_DB_ROOT_PW=BenH5nw7Y25k2xtrXfXcapt4xLzsQC2W
WP_IMESH_DB_USER=wp_user
WP_IMESH_DB_PASSWORD=9cnwfhQzkdyfU8C3yodIXu_IA1PUOawr
WP_IMESH_DB_NAME=wordpress
# WordPress admin credentials: set during browser setup wizard at https://intermountainmesh.com
# Proxmox Datacenter Manager (PDM) — Break-glass local admin (edge2 CT 100, 100.64.0.28:8443)
# PAM realm: Linux user on CT100, independent of Authentik SSO
# Backup taken: /root/access.bak-2026-06-20 (inside CT100)
PDM_URL="https://pdm.echo6.co"
PDM_BREAKGLASS_USER="admin@pam"
PDM_BREAKGLASS_PASS="7redditGold"
# Conduit GUI admin login (http://100.64.0.12:8010/login) — CT 104
# Recreated 2026-07-15 via conduit-admin create-operator (full DB rebuild; old operator gone)
CONDUIT_GUI_URL="http://100.64.0.12:8010/login"
CONDUIT_GUI_USER="admin"
CONDUIT_GUI_PASS="7redditGold"
# ARGUS web UI (https://argus.echo6.co) — utility CT 101 Caddy, HTTP basic auth
# Backend: ARGUS app CT 103 tailnet 100.64.0.25:8090. Added 2026-07-03.
# Only the bcrypt hash lives in the Caddyfile; this is the plaintext.
ARGUS_WEB_URL="https://argus.echo6.co"
ARGUS_WEB_USER="idaho"
ARGUS_WEB_PASS="lads"

1
engine/.embcache.json
View file

File diff suppressed because one or more lines are too long

24
engine/.githooks/pre-commit
View file

@ -1,24 +0,0 @@
#!/usr/bin/env bash
# .githooks/pre-commit — Vault lint report hook
#
# Runs lint.py in REPORT mode (never --strict) so it writes lint-report.md
# and always exits 0. This hook MUST NOT block commits — the root autocommit
# cron (echo6-agent) depends on commits succeeding unconditionally.
#
# To wire: git config core.hooksPath engine/.githooks
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
LINT="${REPO_ROOT}/engine/lib/lint.py"
if [[ ! -f "${LINT}" ]]; then
echo "[lint] WARNING: lint.py not found at ${LINT}, skipping." >&2
exit 0
fi
echo "[lint] Running vault lint (report mode)..." >&2
python3 "${LINT}" >&2 || true # || true: never fail the commit
# Always exit 0 — do NOT change this to exit 1 or add --strict.
exit 0

9
engine/Modelfile
View file

@ -1,9 +0,0 @@
# Echo6 Vault Tagger — pinned, reproducible. Build: ollama create vault-tagger -f Modelfile
FROM qwen2.5:7b-instruct-q8_0
PARAMETER temperature 0.1
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
# System prompt is kept in prompts/system.md (source of truth). Keep this in sync on bootstrap.
SYSTEM """You are the Echo6 vault tagger. Given a markdown document and a controlled vocabulary (topic categories + an entity lexicon of this user's hosts, services, containers, projects, and acronyms), you classify the document and extract entities. You output ONLY valid JSON matching the requested schema. You never invent tags outside the provided vocabulary. You never expand acronyms you are unsure about — you flag them instead."""

68
engine/README.md
View file

@ -1,68 +0,0 @@
# Echo6 Vault Engine
Self-contained maintenance engine for the Obsidian knowledge vault at
`/home/zvx/projects/.ref/vault/`. Runs entirely on cortex (RTX A4000).
## What it does
| Layer | Tool | Job |
|-------|------|-----|
| Tagger | Qwen2.5-7B-Instruct (Ollama) | Classifies docs; assigns topic tags, type, entity refs |
| Embeddings | bge-m3 via TEI (reused service) | Embeds docs into Qdrant for similarity / related links |
| Lint | `lib/lint.py` (deterministic) | Enforces frontmatter schema; fixes safe violations automatically |
| Agent | `lib/agent.py` | Orchestrates tagger + embeddings over changed docs |
| Sweep | `sweep.sh` | Daily entry point; GPU guard, lint, agent, changelog |
## File layout
```
engine/
config.yaml — single source of truth (endpoints, vocab, schema, schedule)
Modelfile — pinned vault-tagger build (qwen2.5:7b-instruct-q8_0, temp 0.1)
bootstrap.sh — idempotent setup: verify services, build model, install cron
sweep.sh — daily maintenance sweep (called by cron)
prompts/
system.md — canonical system prompt (source of truth; synced into Modelfile)
fewshot.md — tagged examples for prompt engineering (added in Step 5)
lib/
vocab_gen.py — generates vocab.json from live infra inventory (Step 3)
lint.py — deterministic frontmatter linter (Step 4)
agent.py — tagger + embeddings agent (Step 5)
vocab.json — generated entity lexicon (not committed; built by vocab_gen.py)
changelog.md — append-only audit log of all automated changes
```
## How it runs
- **Daily cron** (`0 9 * * *` UTC): `sweep.sh` checks GPU VRAM, runs lint, runs agent
over docs changed since last sweep, appends to `changelog.md`.
- **Git pre-commit hook** (Step 4): runs lint against staged vault docs before commit.
- **Manual**: `./sweep.sh` or `python3 lib/agent.py --full` to reprocess all docs.
## Setup
```bash
./bootstrap.sh
```
`bootstrap.sh` is idempotent and documents every step. Base model weights (~8 GB) are
pulled by bootstrap — they are not committed to the repo. Rebuilding from scratch:
```bash
./bootstrap.sh # pulls qwen2.5:7b-instruct-q8_0, builds vault-tagger, installs cron
```
## Vocabulary
- **Tier 1 — topic tags**: stable list in `config.yaml` under `topic_categories`
- **Tier 2 — entity lexicon**: generated into `vocab.json` by `lib/vocab_gen.py`
from live Proxmox, Docker, and Headscale inventory. Regenerate anytime with:
`python3 lib/vocab_gen.py`
## Configuration
All tunables are in `config.yaml`. Key settings:
- `behavior.auto_apply` — write changes directly (true) or dry-run only (false)
- `behavior.confidence_threshold` — below this, changes are flagged not applied
- `schedule.defer_if_gpu_busy_mib` — skip sweep if GPU is already under load

123
engine/bootstrap.sh
View file

@ -1,123 +0,0 @@
#!/usr/bin/env bash
# bootstrap.sh — Echo6 Vault Engine setup
#
# Idempotent setup script. Safe to re-run; each step is guarded.
# Run from any directory. Script locates itself via SCRIPT_DIR.
#
# Steps:
# 1. Verify services reachable (ollama, TEI, qdrant)
# 2. [DISABLED] Pull base model (~8 GB) — enable when ready
# 3. [DISABLED] Build vault-tagger modelfile — depends on step 2
# 4. TODO: Install Python deps for lib/
# 5. TODO: Generate initial vocab (lib/vocab_gen.py)
# 6. TODO: Install git pre-commit hook
# 7. TODO: Install cron job from config.yaml schedule
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG="${SCRIPT_DIR}/config.yaml"
MODELFILE="${SCRIPT_DIR}/Modelfile"
echo "==> Echo6 Vault Engine bootstrap"
echo " engine_dir : ${SCRIPT_DIR}"
echo " config : ${CONFIG}"
echo ""
# ---------------------------------------------------------------------------
# Step 1 — Verify services reachable
# ---------------------------------------------------------------------------
echo "[1/7] Checking service endpoints..."
check_endpoint() {
local name="$1"
local url="$2"
if curl -sf --max-time 5 "$url" > /dev/null 2>&1; then
echo " OK ${name} (${url})"
else
echo " FAIL ${name} (${url}) — is the service running?"
return 1
fi
}
check_endpoint "ollama" "http://localhost:11434"
check_endpoint "TEI/bge-m3" "http://localhost:8090/health"
check_endpoint "qdrant" "http://localhost:6333"
echo ""
# ---------------------------------------------------------------------------
# Step 2 — Pull base model (DISABLED — enable when ready to pull ~8 GB)
# ---------------------------------------------------------------------------
# Uncomment the following block together with Step 3 when ready:
#
# echo "[2/7] Pulling base model qwen2.5:7b-instruct-q8_0..."
# ollama pull qwen2.5:7b-instruct-q8_0
# echo ""
echo "[2/7] SKIPPED — base model pull disabled. Uncomment in bootstrap.sh when ready."
echo ""
# ---------------------------------------------------------------------------
# Step 3 — Build vault-tagger from Modelfile (DISABLED — depends on step 2)
# ---------------------------------------------------------------------------
# Uncomment together with Step 2:
#
# echo "[3/7] Building vault-tagger model from Modelfile..."
# ollama create vault-tagger -f "${MODELFILE}"
# echo ""
echo "[3/7] SKIPPED — vault-tagger build disabled. Uncomment after pulling base model."
echo ""
# ---------------------------------------------------------------------------
# Step 4 — Python dependencies for lib/
# ---------------------------------------------------------------------------
# TODO (Step 2): Install Python deps once requirements.txt is written.
# DO NOT run pip install without explicit approval from Matt.
#
# Example (do NOT uncomment without approval):
# pip install -r "${SCRIPT_DIR}/requirements.txt"
echo "[4/7] TODO — Python deps not yet defined. See lib/ stubs. Do not pip install without approval."
echo ""
# ---------------------------------------------------------------------------
# Step 5 — Generate initial vocab (entity lexicon)
# ---------------------------------------------------------------------------
# TODO (Step 3): Calls lib/vocab_gen.py to query proxmox/docker/headscale
# and write engine/vocab.json.
#
# Example:
# python3 "${SCRIPT_DIR}/lib/vocab_gen.py"
echo "[5/7] TODO — vocab_gen.py not yet implemented (Step 3)."
echo ""
# ---------------------------------------------------------------------------
# Step 6 — Install git pre-commit hook
# ---------------------------------------------------------------------------
# TODO (Step 4): Symlink or copy a pre-commit hook that runs lint.py
# against staged vault docs before commit.
#
# Example:
# HOOK="${SCRIPT_DIR}/../../.git/hooks/pre-commit"
# ln -sf "${SCRIPT_DIR}/hooks/pre-commit" "${HOOK}"
echo "[6/7] TODO — git pre-commit hook not yet implemented (Step 4)."
echo ""
# ---------------------------------------------------------------------------
# Step 7 — Install cron job
# ---------------------------------------------------------------------------
# TODO (Step 6): Install cron from schedule.cron in config.yaml.
# Cron entry should call sweep.sh with appropriate guards.
#
# config.yaml schedule.cron: "0 9 * * *"
# Example crontab line:
# 0 9 * * * /home/zvx/projects/.ref/engine/sweep.sh >> /home/zvx/projects/.ref/engine/sweep.log 2>&1
echo "[7/7] TODO — cron job not yet installed (Step 6)."
echo ""
echo "==> bootstrap.sh complete (partial — disabled steps noted above)."

289
engine/changelog.md
View file

@ -1,289 +0,0 @@
## 2026-06-18T06:11:17Z — meshtastic-sidecar-node.md
- file: `/home/zvx/projects/.ref/vault/runbooks/meshtastic-sidecar-node.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'runbook'; tags: [] → ['mesh', 'vpn']
## 2026-06-18T06:11:27Z — authentik-access-groups.md
- file: `/home/zvx/projects/.ref/vault/runbooks/authentik-access-groups.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'runbook'; tags: [] → ['auth', 'mesh', 'matrix']
## 2026-06-18T06:11:36Z — add-peertube-channel.md
- file: `/home/zvx/projects/.ref/vault/runbooks/add-peertube-channel.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'runbook'; tags: [] → ['media', 'vpn', 'auth']
## 2026-06-18T06:11:44Z — ct-runbook.md
- file: `/home/zvx/projects/.ref/vault/runbooks/ct-runbook.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'runbook'; tags: [] → ['proxmox']
## 2026-06-18T06:12:04Z — authentik.md
- file: `/home/zvx/projects/.ref/vault/docs/software/authentik.md`
- action: applied
- confidence: 0.90
- changes: type: '' → 'reference'; tags: [] → ['auth', 'mesh', 'matrix']
## 2026-06-18T06:12:31Z — caddy.md
- file: `/home/zvx/projects/.ref/vault/docs/software/caddy.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'reference'; tags: [] → ['dns', 'mesh', 'auth']
## 2026-06-18T06:12:59Z — recon.md
- file: `/home/zvx/projects/.ref/vault/docs/software/recon.md`
- action: applied
- confidence: 0.90
- changes: type: '' → 'reference'; tags: [] → ['recon', 'ai', 'storage']
## 2026-06-18T06:13:09Z — argus.md
- file: `/home/zvx/projects/.ref/vault/projects/argus.md`
- action: applied
- confidence: 0.90
- changes: type: '' → 'project'; tags: [] → ['mesh', 'auth', 'recon']
## 2026-06-18T06:13:16Z — meshtastic-headscale-runbook.md
- file: `/home/zvx/projects/.ref/vault/projects/meshtastic-headscale-runbook.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'project'; tags: [] → ['mesh', 'vpn', 'auth']
## 2026-06-18T06:13:30Z — echo6-landing-page-data-export.md
- file: `/home/zvx/projects/.ref/vault/notes/echo6-landing-page-data-export.md`
- action: applied
- confidence: 0.95
- changes: type: '' → 'note'; tags: [] → ['auth', 'media', 'matrix']
## 2026-06-18 — batch_apply.py run
- MINT entities/mt-isr.md — mt-isr
- MINT entities/aida-nebra.md — aida-nebra
- MINT entities/headscale.md — Headscale
- MINT entities/meshtasticd.md — meshtasticd
- MINT entities/advbbs-project.md — advbbs-project
- MINT entities/caddy.md — caddy
- MINT entities/utility-caddy.md — Utility Caddy
- MINT entities/mailcow.md — Mailcow
- MINT entities/authentik.md — authentik
- MINT entities/matrix-synapse.md — Matrix Synapse
- MINT entities/element-web.md — Element Web
- MINT entities/vaultwarden.md — vaultwarden
- MINT entities/proxmox-ve.md — Proxmox VE
- MINT entities/watchtower.md — WATCHTOWER
- MINT entities/meshmonitor.md — meshmonitor
- MINT entities/echo6-search-searxng.md — Echo6 Search (SearXNG)
- MINT entities/recon.md — recon
- MINT entities/immich.md — immich
- MINT entities/nextcloud.md — nextcloud
- MINT entities/jellyfin.md — Jellyfin
- MINT entities/peertube.md — peertube
- MINT entities/open-webui.md — Open WebUI
- MINT entities/lidarr.md — lidarr
- MINT entities/navidrome.md — navidrome
- MINT entities/argus.md — argus
- MINT entities/searxng.md — searxng
- MINT entities/utility.md — Utility
- MINT entities/recon-vm.md — recon-vm
- MINT entities/qdrant.md — Qdrant
- MINT entities/tei.md — TEI
- MINT entities/meshtastic-hs.md — meshtastic-hs
- MINT entities/mesh-bridge.md — mesh-bridge
- MINT concepts/raspberry-pi.md — Raspberry Pi
- MINT concepts/meshtastic.md — Meshtastic
- MINT concepts/lora.md — LoRa
- MINT concepts/reverse-proxy.md — Reverse Proxy
- MINT concepts/split-dns.md — Split DNS
- MINT concepts/osint.md — OSINT
- MINT concepts/lxc-container.md — LXC Container
- MINT concepts/tailscale-ssh.md — Tailscale SSH
- MINT concepts/knowledge-extraction.md — Knowledge Extraction
- MINT concepts/vector-database.md — Vector Database
- MINT concepts/ocr.md — Ocr
- MINT concepts/firewall.md — Firewall
- MINT concepts/youtube.md — Youtube
- LINK+FM meshtastic-sidecar-node.md — 12 link candidates
- LINK+FM caddy.md — 30 link candidates
- LINK+FM argus.md — 5 link candidates
- LINK+FM ct-runbook.md — 2 link candidates
- LINK+FM recon.md — 9 link candidates
- LINK+FM meshtastic-headscale-runbook.md — 11 link candidates
- LINK+FM idahomesh-bridge-setup.md — 11 link candidates
- LINK+FM add-peertube-channel.md — 4 link candidates
- UPDATE .obsidian/graph.json — colorGroups by folder, nodeSizeMultiplier=2, showTags=false
## 2026-06-19T09:00:02Z — sweep deferred (competing GPU process: 2932 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-20T09:00:01Z — sweep deferred (competing GPU process: 2932 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-21T09:00:01Z — sweep deferred (competing GPU process: 2932 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-22T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-23T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-24T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-25T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-26T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-27T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-28T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-29T09:00:02Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-06-30T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-01T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-02T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-03T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-04T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-05T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-06T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-07T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-08T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-09T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-10T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-11T09:00:01Z — sweep deferred (competing GPU process: 4201 node /usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription)
## 2026-07-13T09:00:01Z — sweep run
## 2026-07-14T09:00:01Z — sweep run
- end: 2026-07-14T09:00:23Z
- mode: incremental
- docs selected: 1
- processed: 1 | written: 1 | flagged: 0 | errors: 0
## 2026-07-15T09:00:01Z — sweep deferred (competing GPU process: 1479040 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/7fd5c3f5-460b-4727-a08c-5cc870990b69 --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/oHfKgprXtJn5hsvZXNS1pX --model medium --device cpu --compute_type int8 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-07-16T09:00:01Z — sweep run
- end: 2026-07-16T09:00:19Z
- mode: incremental
- docs selected: 8
- processed: 8 | written: 8 | flagged: 0 | errors: 0
## 2026-07-17T09:00:01Z — sweep deferred (competing GPU process: 3755501 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/8458a2cd-0d04-4bc6-a3c4-37247ca48b37 --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/2WgcrDazx5XSWLUTsBdV9L --model medium --device cuda --compute_type float16 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-07-18T09:00:01Z — sweep run
- end: 2026-07-18T09:00:16Z
- mode: incremental
- docs selected: 5
- processed: 5 | written: 5 | flagged: 0 | errors: 0
## 2026-07-19T09:00:01Z — sweep deferred (competing GPU process: 1981703 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/4eb9f85e-65d3-452a-a2a0-9a332365bd99 --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/2utxDwck3174qGTaKafNgo --model medium --device cpu --compute_type int8 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-07-20T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-21T09:00:01Z — sweep deferred (competing GPU process: 2208 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/c12d0442-69fe-4c45-9c4b-e334e5a768ba --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/i8trZrQP8k2xphsf6nZAnv --model medium --device cpu --compute_type int8 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-07-22T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-23T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-24T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-25T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-26T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-27T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-28T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-29T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-30T09:00:01Z — sweep complete (0 docs selected)
## 2026-07-31T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-01T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-02T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-03T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-04T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-05T09:00:01Z — sweep run
- end: 2026-08-05T09:00:09Z
- mode: incremental
- docs selected: 1
- processed: 1 | written: 1 | flagged: 0 | errors: 0
## 2026-08-06T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-07T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-08T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-09T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-10T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-11T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-12T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-13T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-14T09:00:01Z — sweep run
- end: 2026-08-14T09:00:18Z
- mode: incremental
- docs selected: 7
- processed: 7 | written: 7 | flagged: 0 | errors: 0
## 2026-08-15T09:00:01Z — sweep run
- end: 2026-08-15T09:00:20Z
- mode: incremental
- docs selected: 8
- processed: 8 | written: 8 | flagged: 0 | errors: 0
## 2026-08-16T09:00:01Z — sweep run
- end: 2026-08-16T09:00:10Z
- mode: incremental
- docs selected: 2
- processed: 2 | written: 2 | flagged: 0 | errors: 0
## 2026-08-17T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-18T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-19T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-20T09:00:01Z — sweep run
- end: 2026-08-20T09:00:08Z
- mode: incremental
- docs selected: 1
- processed: 1 | written: 1 | flagged: 0 | errors: 0
## 2026-08-21T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-22T09:00:01Z — sweep deferred (competing GPU process: 1612203 /usr/bin/python3 /usr/local/bin/whisper-ctranslate2-real /home/zvx/.cache/peertube-runner-nodejs/default/transcoding/8930a1ed-816c-4b41-b36e-e7b6d5fe4161 --model medium --word_timestamps True --vad_filter true --vad_min_silence_duration_ms 5000 --output_format all --output_dir /home/zvx/.cache/peertube-runner-nodejs/default/transcription/28yY8e33obZCymsQhGUii8 --model medium --device cuda --compute_type float16 --word_timestamps False --vad_min_silence_duration_ms 500)
## 2026-08-23T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-24T09:00:01Z — sweep run
- end: 2026-08-24T09:00:21Z
- mode: incremental
- docs selected: 7
- processed: 7 | written: 7 | flagged: 0 | errors: 0
## 2026-08-25T09:00:01Z — sweep complete (0 docs selected)
## 2026-08-26T09:00:01Z — sweep complete (0 docs selected)

59
engine/config.yaml
View file

@ -1,59 +0,0 @@
# Echo6 Vault Engine — configuration (single source of truth for the engine)
vault_dir: /home/zvx/projects/.ref/vault
engine_dir: /home/zvx/projects/.ref/engine
models:
tagger:
ollama_endpoint: http://localhost:11434
model: vault-tagger # built by bootstrap from Modelfile
base_model: qwen2.5:7b-instruct-q8_0
temperature: 0.1
format: json
embeddings:
tei_endpoint: http://localhost:8090 # existing bge-m3 service (reuse)
model: bge-m3
qdrant_endpoint: http://localhost:6333
qdrant_collection: vault_docs
# Stable topic-tag vocabulary (tier 1). The entity lexicon (tier 2) is GENERATED into vocab.json.
topic_categories:
- mesh
- matrix
- recon
- media
- auth
- dns
- vpn
- storage
- proxmox
- ai
- mail
- tooling
# Read-only inventory sources for the generated entity lexicon (tier 2)
inventory:
proxmox_nodes: [data, utility, cloud, media, toc] # pct list / qm list (root)
proxmox_vps: [edge1, edge2] # edge1=root; edge2=admin+sudo (hardened, no root login)
docker_hosts: [cortex, utility, media] # host-level docker ps; edge1/edge2 services are LXC-nested (not visible at host level)
headscale_host: edge2 # Headscale runs in edge2 CT 107 (dockerized)
ssh_user: zvx
# Frontmatter property schema enforced by lint
frontmatter_schema:
required: [title, type, tags, updated]
optional: [aliases, related, status]
types: [reference, runbook, project, note, index, session]
behavior:
auto_apply: true # write changes directly...
log_changes: true # ...but log every change for audit/revert
changelog: /home/zvx/projects/.ref/engine/changelog.md
confidence_threshold: 0.6 # below this, flag in changelog instead of silent
schedule:
cron: "0 9 * * *" # 09:00 UTC daily (off-peak); guard checks GPU before running
defer_if_gpu_busy_mib: 6000 # skip/defer if >this much VRAM already in use
# Extraction settings
extraction:
ubiquity_ceiling_pct: 25 # concepts in >25% of vault docs are AMBIENT; skip page + wikilink

0
engine/lib/__init__.py
View file

BIN
engine/lib/__pycache__/__init__.cpython-312.pyc
View file

Binary file not shown.

BIN
engine/lib/__pycache__/agent.cpython-312.pyc
View file

Binary file not shown.

BIN
engine/lib/__pycache__/vocab_gen.cpython-312.pyc
View file

Binary file not shown.

920
engine/lib/agent.py
View file

@ -1,920 +0,0 @@
#!/usr/bin/env python3
"""
agent.py — Vault Tagger + Existing-Doc Linker (v4 — simple model)
The agreed model: documentation library only.
- Never create entity/concept/node pages.
- Per-doc: (1) assign 1-3 category tags from topic_categories, (2) inline [[wikilinks]]
to EXISTING docs only (most-specific match, first prose occurrence).
- Keep: related (bge-m3 nearest existing docs), title, updated.
- Preserve all human frontmatter keys.
Usage:
python3 engine/lib/agent.py --dry-run <path> # print diff, write nothing
python3 engine/lib/agent.py <path> # apply
stdlib only — no pip. All HTTP via urllib.
"""
from __future__ import annotations
import argparse
import difflib
import json
import math
import os
import re
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
# ---------------------------------------------------------------------------
# Config loader (minimal stdlib YAML parser)
# ---------------------------------------------------------------------------
def _parse_simple_yaml(text: str) -> dict:
result: dict = {}
current_key = None
current_list: list | None = None
for raw_line in text.splitlines():
line = raw_line.rstrip()
stripped = line.lstrip()
if not stripped or stripped.startswith("#"):
if current_list is not None and not line.startswith(" "):
result[current_key] = current_list
current_list = None
current_key = None
continue
if stripped.startswith("- ") and current_list is not None:
current_list.append(stripped[2:].strip().strip('"').strip("'"))
continue
if current_list is not None and not stripped.startswith("-"):
result[current_key] = current_list
current_list = None
current_key = None
if ":" in stripped:
key, _, value = stripped.partition(":")
key = key.strip()
value = value.strip()
if value.startswith("[") and value.endswith("]"):
inner = value[1:-1]
result[key] = [v.strip().strip('"').strip("'") for v in inner.split(",") if v.strip()]
elif value == "":
current_key = key
current_list = []
elif value.startswith("#"):
result[key] = ""
else:
value = value.split(" #")[0].strip().strip('"').strip("'")
result[key] = value
if current_list is not None and current_key:
result[current_key] = current_list
return result
def load_config(config_path: str | Path | None = None) -> dict:
"""Load config.yaml; fall back to defaults."""
defaults: dict = {
"vault_dir": "/home/zvx/projects/.ref/vault",
"engine_dir": "/home/zvx/projects/.ref/engine",
"topic_categories": [
"mesh", "matrix", "recon", "media", "auth",
"dns", "vpn", "storage", "proxmox", "ai", "mail",
],
"models": {
"tagger": {
"ollama_endpoint": "http://localhost:11434",
"model": "vault-tagger",
"temperature": 0.1,
},
"embeddings": {
"tei_endpoint": "http://localhost:8090",
},
},
"behavior": {
"auto_apply": True,
"changelog": "/home/zvx/projects/.ref/engine/changelog.md",
"confidence_threshold": 0.6,
},
}
if config_path is None:
here = Path(__file__).parent
for c in [here.parent / "config.yaml", Path("/home/zvx/projects/.ref/engine/config.yaml")]:
if c.exists():
config_path = c
break
if config_path is None or not Path(config_path).exists():
return defaults
cfg = dict(defaults)
try:
parsed = _parse_simple_yaml(Path(config_path).read_text(encoding="utf-8"))
if "vault_dir" in parsed:
cfg["vault_dir"] = parsed["vault_dir"]
if "engine_dir" in parsed:
cfg["engine_dir"] = parsed["engine_dir"]
if "topic_categories" in parsed:
cfg["topic_categories"] = parsed["topic_categories"]
# Models
if "models" in parsed:
pass # nested — handled by line-scan below
# Line-scan for nested values
lines = Path(config_path).read_text(encoding="utf-8").splitlines()
in_tagger = in_embed = in_behavior = False
for line in lines:
if re.match(r" tagger:", line):
in_tagger, in_embed = True, False
elif re.match(r" embeddings:", line):
in_embed, in_tagger = True, False
elif re.match(r"^behavior:", line):
in_tagger = in_embed = False
in_behavior = True
elif line and not line.startswith(" "):
in_tagger = in_embed = in_behavior = False
if in_tagger:
m = re.match(r"\s+ollama_endpoint:\s*(\S+)", line)
if m:
cfg["models"]["tagger"]["ollama_endpoint"] = m.group(1).strip()
m = re.match(r"\s+model:\s*(\S+)", line)
if m:
cfg["models"]["tagger"]["model"] = m.group(1).strip()
m = re.match(r"\s+temperature:\s*([\d.]+)", line)
if m:
cfg["models"]["tagger"]["temperature"] = float(m.group(1))
if in_embed:
m = re.match(r"\s+tei_endpoint:\s*(\S+)", line)
if m:
cfg["models"]["embeddings"]["tei_endpoint"] = m.group(1).strip()
if in_behavior:
m = re.match(r"\s+auto_apply:\s*(\S+)", line)
if m:
cfg["behavior"]["auto_apply"] = m.group(1).strip().lower() == "true"
m = re.match(r"\s+confidence_threshold:\s*([\d.]+)", line)
if m:
cfg["behavior"]["confidence_threshold"] = float(m.group(1))
m = re.match(r"\s+changelog:\s*(.+)", line)
if m:
cfg["behavior"]["changelog"] = m.group(1).strip()
except Exception as e:
print(f"[warn] Could not fully parse {config_path}: {e}", file=sys.stderr)
return cfg
# ---------------------------------------------------------------------------
# Frontmatter parser
# ---------------------------------------------------------------------------
def parse_frontmatter(text: str) -> tuple[dict, str]:
"""Parse YAML frontmatter. Returns (fm_dict, body_text)."""
fm: dict = {}
m = re.match(r"^---\r?\n(.*?)\r?\n---\r?\n?(.*)", text, re.DOTALL)
if not m:
return fm, text
fm_raw = m.group(1)
body = m.group(2)
lines = fm_raw.splitlines()
i = 0
while i < len(lines):
line = lines[i]
if not line.strip() or line.strip().startswith("#"):
i += 1
continue
kv = re.match(r"^(\w[\w-]*):\s*(.*)", line)
if not kv:
i += 1
continue
key = kv.group(1)
val_raw = kv.group(2).strip()
if val_raw.startswith("["):
inner = re.sub(r"^\[|\]$", "", val_raw).strip()
fm[key] = [x.strip().strip("\"'") for x in inner.split(",") if x.strip()] if inner else []
i += 1
continue
if not val_raw:
lst = []
j = i + 1
while j < len(lines) and re.match(r"^\s+-\s+(.*)", lines[j]):
item_m = re.match(r"^\s+-\s+(.*)", lines[j])
lst.append(item_m.group(1).strip().strip("\"'"))
j += 1
fm[key] = lst
i = j
continue
fm[key] = val_raw.strip("\"'")
i += 1
return fm, body
def render_frontmatter(fm: dict) -> str:
"""Render frontmatter dict to YAML string (between --- markers)."""
lines = ["---"]
key_order = ["title", "type", "tags", "aliases", "related", "updated"]
written: set[str] = set()
for key in key_order:
if key not in fm:
continue
written.add(key)
val = fm[key]
if isinstance(val, list):
if not val:
lines.append(f"{key}: []")
else:
lines.append(f"{key}:")
for item in val:
lines.append(f" - {item}")
else:
sv = str(val)
if any(c in sv for c in ':#{}[]|>&*!,?'):
lines.append(f'{key}: "{sv}"')
else:
lines.append(f"{key}: {sv}")
for key, val in fm.items():
if key in written:
continue
if isinstance(val, list):
if not val:
lines.append(f"{key}: []")
else:
lines.append(f"{key}:")
for item in val:
lines.append(f" - {item}")
else:
sv = str(val)
if any(c in sv for c in ':#{}[]|>&*!,?'):
lines.append(f'{key}: "{sv}"')
else:
lines.append(f"{key}: {sv}")
lines.append("---")
return "\n".join(lines) + "\n"
# ---------------------------------------------------------------------------
# HTTP helpers
# ---------------------------------------------------------------------------
def _http_post(url: str, payload: dict, timeout: int = 120) -> dict | list:
data = json.dumps(payload).encode("utf-8")
req = urllib.request.Request(
url, data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
# ---------------------------------------------------------------------------
# Existing-doc index (the core of v4)
# ---------------------------------------------------------------------------
def _norm(s: str) -> str:
"""Normalize: lowercase, spaces/underscores → hyphens."""
return re.sub(r"[\s_]+", "-", s.strip().lower())
def build_doc_index(vault_dir: Path) -> dict[str, Path]:
"""
Build a mapping of normalized name → Path for ALL vault docs.
Keys: normalized basename, normalized frontmatter title, normalized aliases.
Longer/more-specific keys win over shorter on conflict.
Archive excluded.
"""
index: dict[str, Path] = {}
for p in sorted(vault_dir.rglob("*.md")):
if "archive" in p.parts:
continue
try:
text = p.read_text(encoding="utf-8", errors="replace")
fm, _ = parse_frontmatter(text)
except Exception:
fm = {}
keys: list[str] = [_norm(p.stem)]
title = fm.get("title", "")
if title:
keys.append(_norm(str(title)))
for alias in fm.get("aliases", []) or []:
if alias:
keys.append(_norm(str(alias)))
for key in keys:
if key and key not in index:
index[key] = p
elif key and key in index:
# Keep longer stem (more specific)
if len(p.stem) > len(index[key].stem):
index[key] = p
return index
def resolve_term_to_doc(term: str, doc_index: dict[str, Path]) -> Path | None:
"""
Given a surface term, find the most-specific existing doc.
Returns Path or None.
"""
norm = _norm(term)
return doc_index.get(norm)
# ---------------------------------------------------------------------------
# Ollama tagger — assigns 1-3 category tags
# ---------------------------------------------------------------------------
_TAGGER_SYSTEM = """You are the Echo6 vault tagger. Given a markdown document and a list of allowed topic categories, assign tags that honestly describe what the doc is primarily about. Output ONLY valid JSON. No prose, no markdown fences.
Rules:
- Only use tags from the provided topic_categories list.
- Output the FEWEST tags that are accurate. DEFAULT to exactly ONE tag — the document's primary subject.
- Add a SECOND tag ONLY if the document is genuinely, substantially about two co-equal subjects.
- NEVER add a tag for something merely mentioned, proxied, or tangentially related.
- A third tag is almost never correct.
- Primary topic first (most central).
- confidence: 0.0-1.0
Negative examples (DO NOT do this):
- A Caddy reverse-proxy config doc is ["dns"], NOT ["dns", "matrix"] — even if it proxies Matrix.
- A PeerTube channel guide is ["media"], NOT ["media", "recon"] — even if recon uses a channel.
- A generic container runbook is ["proxmox"], NOT ["proxmox", "mesh"] — even if the CT joins the mesh.
- An Authentik access-groups doc is ["auth"], NOT ["auth", "mesh"] — tangential mesh mention does not earn a tag."""
def ollama_tag(doc_text: str, topic_categories: list[str], config: dict) -> dict:
"""
Call Qwen vault-tagger to assign 1-3 category tags.
Returns {tags: [...], confidence: float}.
"""
tagger_cfg = config.get("models", {}).get("tagger", {})
endpoint = tagger_cfg.get("ollama_endpoint", "http://localhost:11434")
model = tagger_cfg.get("model", "vault-tagger")
temperature = tagger_cfg.get("temperature", 0.1)
cats_str = ", ".join(topic_categories)
doc_truncated = doc_text[:5000]
if len(doc_text) > 5000:
doc_truncated += "\n\n[... truncated ...]"
user_msg = (
f"topic_categories (ONLY use these): {cats_str}\n\n"
f"## Document\n\n{doc_truncated}\n\n"
f"Return ONLY this JSON:\n"
f'{{\n "tags": ["primary_tag"],\n "confidence": 0.85\n}}'
)
payload = {
"model": model,
"messages": [
{"role": "system", "content": _TAGGER_SYSTEM},
{"role": "user", "content": user_msg},
],
"format": "json",
"stream": False,
"options": {"temperature": temperature},
}
try:
resp = _http_post(f"{endpoint}/api/chat", payload, timeout=120)
# /api/chat returns {"message": {"role": "assistant", "content": "..."}}
if isinstance(resp, dict):
raw = resp.get("message", {}).get("content", "") or resp.get("response", "")
else:
raw = ""
result = json.loads(raw)
except json.JSONDecodeError:
m = re.search(r"\{.*\}", raw, re.DOTALL)
result = json.loads(m.group(0)) if m else {}
except Exception as e:
print(f"[warn] Tagger call failed: {e}", file=sys.stderr)
return {"tags": [], "confidence": 0.0}
valid_cats = set(topic_categories)
tags = []
if isinstance(result.get("tags"), list):
for t in result["tags"]:
if isinstance(t, str) and t.strip() in valid_cats and t.strip() not in tags:
tags.append(t.strip())
tags = tags[:3]
try:
conf = float(result.get("confidence", 0.0))
conf = max(0.0, min(1.0, conf))
except (TypeError, ValueError):
conf = 0.0
return {"tags": tags, "confidence": conf}
# ---------------------------------------------------------------------------
# Body wikilinking — existing docs only
# ---------------------------------------------------------------------------
def _build_skip_spans(body: str) -> list[tuple[int, int]]:
"""
Return spans that must NOT be modified:
fenced code blocks, inline code, existing [[wikilinks]], headings, URLs,
domain-like patterns.
"""
spans: list[tuple[int, int]] = []
for m in re.finditer(r"```.*?```", body, re.DOTALL):
spans.append((m.start(), m.end()))
for m in re.finditer(r"`[^`\n]+`", body):
spans.append((m.start(), m.end()))
for m in re.finditer(r"\[\[.*?\]\]", body):
spans.append((m.start(), m.end()))
for m in re.finditer(r"^#+\s+.*$", body, re.MULTILINE):
spans.append((m.start(), m.end()))
for m in re.finditer(r"https?://\S+", body):
spans.append((m.start(), m.end()))
# Domain-like: prevent "mesh" in "mesh.echo6.co"
for m in re.finditer(r"\b\w[\w\-]*\.\w+\b", body):
spans.append((m.start(), m.end()))
return spans
def _in_skip_span(start: int, end: int, spans: list[tuple[int, int]]) -> bool:
for s, e in spans:
if start < e and end > s:
return True
return False
def _collect_link_candidates(
body: str,
doc_index: dict[str, Path],
self_stems: set[str],
) -> list[tuple[str, str, Path]]:
"""
Scan body for terms that match existing docs (excluding self and headings/code).
Returns list of (surface_text, display_name, target_path) sorted longest-first.
Only returns candidates that actually appear in the prose.
"""
skip_spans = _build_skip_spans(body)
body_lower = body.lower()
candidates: list[tuple[str, str, Path]] = []
seen_targets: set[str] = set()
# Build list of (norm_key, display, path) sorted by key length descending
# (longest/most-specific match first)
sorted_keys = sorted(doc_index.keys(), key=len, reverse=True)
for norm_key in sorted_keys:
path = doc_index[norm_key]
stem_norm = _norm(path.stem)
# Skip self
if stem_norm in self_stems:
continue
# Skip INDEX (being deleted)
if path.stem.upper() == "INDEX":
continue
# Skip docs in archive
if "archive" in path.parts:
continue
# Derive surface variations to search for
surfaces = [norm_key, norm_key.replace("-", " ")]
# Remove duplicates
surfaces = list(dict.fromkeys(surfaces))
# Get display name (frontmatter title or stem)
try:
text = path.read_text(encoding="utf-8", errors="replace")
fm, _ = parse_frontmatter(text)
display = fm.get("title", "") or path.stem
except Exception:
display = path.stem
target_key = str(path)
if target_key in seen_targets:
continue
for surface in surfaces:
if len(surface) < 3:
continue
# Check if this surface appears in body (quick scan)
if surface not in body_lower:
continue
# Build word-boundary pattern
esc = re.escape(surface).replace(r"\ ", r"[\s\-]").replace(r"\-", r"[\s\-]")
pattern = rf"(?<![a-zA-Z0-9\-_\[\]])({esc})(?![a-zA-Z0-9\-_\[\]\.])"
for m in re.finditer(pattern, body, re.IGNORECASE):
s, e = m.start(1), m.end(1)
if not _in_skip_span(s, e, skip_spans):
candidates.append((m.group(1), str(display), path))
seen_targets.add(target_key)
break # found at least one prose occurrence — add candidate
if target_key in seen_targets:
break
return candidates
def apply_wikilinks(
body: str,
doc_index: dict[str, Path],
self_stems: set[str],
) -> str:
"""
Insert [[wikilinks]] for FIRST prose occurrence of each existing-doc name.
- Longest match wins (most-specific doc).
- Skip code blocks, inline code, headings, URLs, existing links.
- Idempotent.
- [[Doc|surface]] when surface differs from doc stem; [[Doc]] when same.
"""
skip_spans = _build_skip_spans(body)
result_chars = list(body)
replaced_targets: set[str] = set() # target paths already linked
offset = 0 # cumulative character offset
# Sort candidates: longest surface text first (most specific wins)
sorted_keys = sorted(doc_index.keys(), key=len, reverse=True)
replacements: list[tuple[int, int, str]] = []
replacement_spans: list[tuple[int, int]] = []
for norm_key in sorted_keys:
path = doc_index[norm_key]
stem_norm = _norm(path.stem)
if stem_norm in self_stems:
continue
if path.stem.upper() == "INDEX":
continue
if "archive" in path.parts:
continue
target_key = str(path)
if target_key in replaced_targets:
continue
surfaces = list(dict.fromkeys([norm_key, norm_key.replace("-", " ")]))
try:
text = path.read_text(encoding="utf-8", errors="replace")
fm, _ = parse_frontmatter(text)
display = str(fm.get("title", "") or path.stem)
except Exception:
display = path.stem
found = False
for surface in surfaces:
if len(surface) < 3:
continue
esc = re.escape(surface).replace(r"\ ", r"[\s\-]").replace(r"\-", r"[\s\-]")
pattern = rf"(?<![a-zA-Z0-9\-_\[\]])({esc})(?![a-zA-Z0-9\-_\[\]\.])"
for m in re.finditer(pattern, body, re.IGNORECASE):
s, e = m.start(1), m.end(1)
matched_text = m.group(1)
if _in_skip_span(s, e, skip_spans):
continue
if _in_skip_span(s, e, replacement_spans):
continue
# Build wikilink
if matched_text.lower() == display.lower():
wiki = f"[[{display}]]"
elif _norm(matched_text) == _norm(path.stem):
wiki = f"[[{path.stem}]]"
else:
wiki = f"[[{path.stem}|{matched_text}]]"
replacements.append((s, e, wiki))
replacement_spans.append((s, e))
replaced_targets.add(target_key)
found = True
break
if found:
break
# Apply in reverse order
replacements.sort(key=lambda x: x[0], reverse=True)
chars = list(body)
for s, e, wiki in replacements:
chars[s:e] = list(wiki)
return "".join(chars)
# ---------------------------------------------------------------------------
# Type inference
# ---------------------------------------------------------------------------
def infer_type(doc_path: Path, vault_dir: Path) -> str | None:
try:
rel = doc_path.relative_to(vault_dir)
except ValueError:
return None
parts = rel.parts
folder = parts[0].lower() if len(parts) > 1 else ""
folder_map = {
"runbooks": "runbook",
"projects": "project",
"docs": "reference",
"notes": "note",
"session-resume": "session",
"plans": "note",
}
return folder_map.get(folder, "reference")
def extract_title(fm: dict, body: str, doc_path: Path) -> str:
if fm.get("title"):
return str(fm["title"])
m = re.search(r"^#\s+(.+)", body, re.MULTILINE)
if m:
return m.group(1).strip()
return re.sub(r"[-_]+", " ", doc_path.stem).title()
# ---------------------------------------------------------------------------
# TEI embedding + related
# ---------------------------------------------------------------------------
def _detect_tei_route(endpoint: str) -> str:
for route in ["/embed", "/embeddings"]:
try:
resp = _http_post(f"{endpoint}{route}", {"inputs": "test"}, timeout=10)
if isinstance(resp, list):
return route
except urllib.error.HTTPError as e:
if e.code not in (404, 405):
return route
except Exception:
continue
return "/embed"
_TEI_ROUTE_CACHE: dict[str, str] = {}
def tei_embed(text: str, endpoint: str) -> list[float]:
if endpoint not in _TEI_ROUTE_CACHE:
_TEI_ROUTE_CACHE[endpoint] = _detect_tei_route(endpoint)
route = _TEI_ROUTE_CACHE[endpoint]
resp = _http_post(f"{endpoint}{route}", {"inputs": text}, timeout=30)
if isinstance(resp, list):
return resp[0] if resp and isinstance(resp[0], list) else resp
raise RuntimeError(f"Unexpected TEI response: {type(resp)}")
def cosine_sim(a: list[float], b: list[float]) -> float:
if len(a) != len(b):
return 0.0
dot = sum(x * y for x, y in zip(a, b))
mag_a = math.sqrt(sum(x * x for x in a))
mag_b = math.sqrt(sum(y * y for y in b))
return dot / (mag_a * mag_b) if mag_a and mag_b else 0.0
def _cache_path(engine_dir: Path) -> Path:
return engine_dir / ".embcache.json"
def load_embed_cache(engine_dir: Path) -> dict:
p = _cache_path(engine_dir)
if p.exists():
try:
return json.loads(p.read_text(encoding="utf-8"))
except Exception:
return {}
return {}
def save_embed_cache(cache: dict, engine_dir: Path) -> None:
_cache_path(engine_dir).write_text(json.dumps(cache), encoding="utf-8")
def get_or_embed(
doc_path: Path, vault_dir: Path, tei_endpoint: str, cache: dict,
doc_text: str | None = None,
) -> tuple[list[float], bool]:
rel = str(doc_path.relative_to(vault_dir))
mtime = doc_path.stat().st_mtime
key = f"{rel}::{mtime:.3f}"
if key in cache:
return cache[key], True
if doc_text is None:
doc_text = doc_path.read_text(encoding="utf-8", errors="replace")
vec = tei_embed(doc_text[:4000], tei_endpoint)
cache[key] = vec
return vec, False
def find_related(
doc_path: Path, doc_vec: list[float], vault_dir: Path,
tei_endpoint: str, cache: dict, k: int = 5,
) -> list[str]:
sims: list[tuple[float, str]] = []
for md in vault_dir.rglob("*.md"):
if md == doc_path or "archive" in md.parts:
continue
try:
vec, _ = get_or_embed(md, vault_dir, tei_endpoint, cache)
sims.append((cosine_sim(doc_vec, vec), md.stem))
except Exception:
continue
sims.sort(reverse=True)
return [f"[[{stem}]]" for _, stem in sims[:k]]
# ---------------------------------------------------------------------------
# Core: process_doc
# ---------------------------------------------------------------------------
def process_doc(
doc_path: Path | str,
config: dict,
dry_run: bool = True,
cache: dict | None = None,
) -> dict:
"""
Process a single vault document.
Steps:
1. Assign 1-3 category tags via Qwen tagger.
2. Build existing-doc index; insert inline wikilinks to existing docs only.
3. Update related: (bge-m3 nearest docs).
4. Write updated frontmatter (title, type, tags, related, updated).
Returns result dict. dry_run=True → nothing written.
"""
doc_path = Path(doc_path).resolve()
vault_dir = Path(config["vault_dir"]).resolve()
engine_dir = Path(config["engine_dir"]).resolve()
tei_endpoint = config["models"]["embeddings"]["tei_endpoint"]
topic_categories = config.get("topic_categories", [])
if cache is None:
cache = load_embed_cache(engine_dir)
doc_text = doc_path.read_text(encoding="utf-8", errors="replace")
existing_fm, body = parse_frontmatter(doc_text)
# --- Step 1: Tags via Qwen ---
print(f"[tag] Calling tagger on {doc_path.name} ...", file=sys.stderr)
tag_result = ollama_tag(doc_text, topic_categories, config)
print(f"[tag] tags={tag_result['tags']} confidence={tag_result['confidence']:.2f}", file=sys.stderr)
# --- Step 2: Build doc index + wikilinks ---
print(f"[link] Building doc index ...", file=sys.stderr)
doc_index = build_doc_index(vault_dir)
# Self stems to exclude (don't self-link)
self_stems = {_norm(doc_path.stem)}
title_val = existing_fm.get("title", "")
if title_val:
self_stems.add(_norm(str(title_val)))
print(f"[link] Applying wikilinks to {doc_path.name} ...", file=sys.stderr)
linked_body = apply_wikilinks(body, doc_index, self_stems)
body_diff = "".join(difflib.unified_diff(
body.splitlines(keepends=True),
linked_body.splitlines(keepends=True),
fromfile=f"a/{doc_path.name}",
tofile=f"b/{doc_path.name} (linked)",
n=2,
))
# --- Step 3: related (bge-m3) ---
print(f"[embed] Embedding {doc_path.name} ...", file=sys.stderr)
related_links: list[str] = []
try:
doc_vec, cache_hit = get_or_embed(doc_path, vault_dir, tei_endpoint, cache, doc_text=doc_text)
if not cache_hit:
save_embed_cache(cache, engine_dir)
related_links = find_related(doc_path, doc_vec, vault_dir, tei_endpoint, cache, k=5)
save_embed_cache(cache, engine_dir)
print(f"[embed] {len(related_links)} related", file=sys.stderr)
except Exception as e:
print(f"[warn] Embedding failed: {e}", file=sys.stderr)
# --- Step 4: Frontmatter ---
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
new_fm = dict(existing_fm)
if not new_fm.get("title"):
new_fm["title"] = extract_title(existing_fm, body, doc_path)
inferred = infer_type(doc_path, vault_dir)
if inferred:
new_fm["type"] = inferred
# Tags: use tagger result; fall back to existing tags if tagger returns empty
if tag_result["tags"]:
new_fm["tags"] = tag_result["tags"]
elif not new_fm.get("tags"):
new_fm["tags"] = []
if "aliases" not in new_fm:
new_fm["aliases"] = []
# related: replace with embedding results (or keep existing if embed failed)
if related_links:
new_fm["related"] = related_links
new_fm["updated"] = today
new_fm_text = render_frontmatter(new_fm)
new_doc_text = new_fm_text + linked_body
full_diff = "".join(difflib.unified_diff(
doc_text.splitlines(keepends=True),
new_doc_text.splitlines(keepends=True),
fromfile=f"a/{doc_path.name}",
tofile=f"b/{doc_path.name}",
n=2,
))
applied = False
if not dry_run:
if config["behavior"].get("auto_apply", True):
doc_path.write_text(new_doc_text, encoding="utf-8")
applied = True
print(f"[apply] Wrote {doc_path}", file=sys.stderr)
return {
"path": str(doc_path),
"tag_result": tag_result,
"body_diff": body_diff,
"full_diff": full_diff,
"related_links": related_links,
"proposed_frontmatter": new_fm_text,
"applied": applied,
"dry_run": dry_run,
"existing_fm": existing_fm,
"new_fm": new_fm,
}
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(description="Vault tagger + existing-doc linker (v4).")
parser.add_argument("path", help="Path to the vault markdown document")
parser.add_argument("--dry-run", action="store_true", help="Print diff, write nothing")
args = parser.parse_args()
here = Path(__file__).parent
config_path = here.parent / "config.yaml"
if not config_path.exists():
config_path = Path("/home/zvx/projects/.ref/engine/config.yaml")
config = load_config(config_path)
doc_path = Path(args.path).resolve()
if not doc_path.exists():
print(f"[error] File not found: {doc_path}", file=sys.stderr)
sys.exit(1)
# Skip symlinks — vault/CLAUDE-baseline.md and vault/rules point outside the
# repo (to ~/.claude/). Editing them would silently modify external files.
if os.path.islink(args.path) or doc_path != Path(args.path).resolve():
print(f"[skip] Symlink — refusing to edit external target: {doc_path}", file=sys.stderr)
sys.exit(0)
result = process_doc(doc_path, config, dry_run=args.dry_run)
print("\n" + "=" * 72)
print(f"VAULT TAGGER v4 {'DRY-RUN ' if args.dry_run else ''}— {doc_path.name}")
print("=" * 72)
print(f"\n### Tags")
print(f" {result['tag_result']['tags']} (confidence={result['tag_result']['confidence']:.2f})")
print(f"\n### Body wikilink diff")
print(result["body_diff"] or " (no new wikilinks)")
print(f"\n### Related (bge-m3 top-5)")
for r in result["related_links"]:
print(f" {r}")
print(f"\n### Proposed frontmatter")
print(result["proposed_frontmatter"])
if args.dry_run:
print("[DRY-RUN] Nothing written.")
else:
print(f"[APPLY] applied={result['applied']}")
if __name__ == "__main__":
main()

788
engine/lib/lint.py
View file

@ -1,788 +0,0 @@
#!/usr/bin/env python3
"""
lint.py — Deterministic Vault Lint (stdlib only)
Checks performed (all deterministic):
1. Frontmatter schema — required keys, valid type, tags is a list
2. Tag vocabulary — tags must be in topic_categories
3. Dead wikilinks — [[target]] must resolve against vault note basenames
4. Orphans — notes with zero incoming wikilinks (INFO, capped at 40)
Gaps & suggestions (written to ## Gaps & suggestions in lint-report.md):
- Docs with NO tags
- True orphans (no inbound link AND shares no tag with any other doc)
- Dead wikilinks (must be 0 after v4 sweep)
- Tags not in topic_categories
- "Earned-a-doc?" candidates: terms mentioned in >= 4 docs with no dedicated doc (top 15)
Usage:
python3 engine/lib/lint.py # report mode, always exit 0
python3 engine/lib/lint.py --strict # exit 1 if any ERROR findings
Writes: engine/lint-report.md
"""
from __future__ import annotations
import json
import re
import sys
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path
from typing import NamedTuple
# ---------------------------------------------------------------------------
# Config / vocab loading (stdlib-only minimal YAML parser)
# ---------------------------------------------------------------------------
def _parse_simple_yaml(text: str) -> dict:
result: dict = {}
current_key = None
current_list: list | None = None
for raw_line in text.splitlines():
line = raw_line.rstrip()
stripped = line.lstrip()
if not stripped or stripped.startswith("#"):
if current_list is not None and not line.startswith(" "):
result[current_key] = current_list
current_list = None
current_key = None
continue
if stripped.startswith("- ") and current_list is not None:
current_list.append(stripped[2:].strip().strip('"').strip("'"))
continue
if current_list is not None and not stripped.startswith("-"):
result[current_key] = current_list
current_list = None
current_key = None
if ":" in stripped:
key, _, value = stripped.partition(":")
key = key.strip()
value = value.strip()
if value.startswith("[") and value.endswith("]"):
inner = value[1:-1]
result[key] = [v.strip().strip('"').strip("'") for v in inner.split(",") if v.strip()]
elif value == "":
current_key = key
current_list = []
elif value.startswith("#"):
result[key] = ""
else:
value = value.split(" #")[0].strip().strip('"').strip("'")
result[key] = value
if current_list is not None and current_key:
result[current_key] = current_list
return result
def load_config(engine_dir: Path) -> dict:
config_path = engine_dir / "config.yaml"
return _parse_simple_yaml(config_path.read_text(encoding="utf-8"))
def load_vocab(engine_dir: Path) -> dict:
vocab_path = engine_dir / "vocab.json"
return json.loads(vocab_path.read_text(encoding="utf-8"))
# ---------------------------------------------------------------------------
# Frontmatter parsing
# ---------------------------------------------------------------------------
_FM_FENCE = re.compile(r"^---\s*$")
def parse_frontmatter(path: Path) -> tuple[dict | None, str]:
text = path.read_text(encoding="utf-8", errors="replace")
lines = text.splitlines(keepends=True)
if not lines or not _FM_FENCE.match(lines[0].rstrip()):
return None, text
end_idx = None
for i in range(1, len(lines)):
if _FM_FENCE.match(lines[i].rstrip()):
end_idx = i
break
if end_idx is None:
return None, text
fm_text = "".join(lines[1:end_idx])
body = "".join(lines[end_idx + 1:])
fm = _parse_yaml_frontmatter(fm_text)
return fm, body
def _parse_yaml_frontmatter(text: str) -> dict:
result: dict = {}
current_key: str | None = None
current_list: list | None = None
for raw_line in text.splitlines():
line = raw_line.rstrip()
stripped = line.lstrip()
if not stripped:
continue
if stripped.startswith("- ") and current_list is not None:
current_list.append(stripped[2:].strip().strip('"').strip("'"))
continue
if current_list is not None:
result[current_key] = current_list
current_list = None
current_key = None
if ":" in stripped:
key, _, value = stripped.partition(":")
key = key.strip()
value = value.strip()
if value.startswith("[") and value.endswith("]"):
inner = value[1:-1]
result[key] = [
v.strip().strip('"').strip("'")
for v in inner.split(",")
if v.strip()
]
elif value == "":
current_key = key
current_list = []
else:
result[key] = value.strip('"').strip("'")
if current_list is not None and current_key:
result[current_key] = current_list
return result
# ---------------------------------------------------------------------------
# Check 1: Frontmatter schema
# ---------------------------------------------------------------------------
# v4: entity/concept types removed; valid types are the real doc types
VALID_TYPES = {"reference", "runbook", "project", "note", "index", "session"}
def check_frontmatter(fm: dict | None, config: dict) -> list[dict]:
findings = []
required_keys = ["title", "type", "tags", "updated"]
if fm is None:
findings.append({
"severity": "WARN",
"check": "frontmatter",
"message": "missing frontmatter block entirely",
})
return findings
for key in required_keys:
if key not in fm or fm[key] is None or fm[key] == "":
findings.append({
"severity": "WARN",
"check": "frontmatter",
"message": f"missing required key: {key!r}",
})
if "type" in fm and fm["type"]:
if fm["type"] not in VALID_TYPES:
findings.append({
"severity": "WARN",
"check": "frontmatter",
"message": f"invalid type {fm['type']!r} — must be one of {sorted(VALID_TYPES)}",
})
if "tags" in fm and fm["tags"] is not None:
if not isinstance(fm["tags"], list):
findings.append({
"severity": "WARN",
"check": "frontmatter",
"message": f"tags must be a list, got {type(fm['tags']).__name__}: {fm['tags']!r}",
})
return findings
# ---------------------------------------------------------------------------
# Check 2: Tag vocabulary
# ---------------------------------------------------------------------------
def check_tags(fm: dict | None, allowed_tags: set[str]) -> list[dict]:
if fm is None:
return []
tags = fm.get("tags")
if not tags or not isinstance(tags, list):
return []
findings = []
for tag in tags:
if tag not in allowed_tags:
findings.append({
"severity": "WARN",
"check": "tag-vocab",
"message": f"unknown tag {tag!r} (not in topic_categories)",
})
return findings
# ---------------------------------------------------------------------------
# Check 3: Dead wikilinks
# ---------------------------------------------------------------------------
_WIKILINK_RE = re.compile(
r"""
\[\[
([^\[\]\n]+?)
(?:\|[^\[\]\n]*)?
\]\]
""",
re.VERBOSE,
)
_FENCED_CODE_RE = re.compile(r"```.*?```", re.DOTALL)
_INLINE_CODE_RE = re.compile(r"`[^`\n]+`")
def _looks_like_shell(inner: str) -> bool:
s = inner.strip()
if re.search(r'\$[{(A-Za-z_]', s):
return True
if re.match(r'^\s*[!-]', s):
return True
if re.search(r'==|!=|&&|\|\|', s):
return True
tokens = s.split()
if len(tokens) > 1 and tokens[0].startswith("-"):
return True
return False
def _is_note_like(inner: str) -> bool:
if _looks_like_shell(inner):
return False
return bool(re.match(r'^[\w\s.\-/]+$', inner.strip()))
def _strip_code_blocks(text: str) -> str:
text = _FENCED_CODE_RE.sub("", text)
text = _INLINE_CODE_RE.sub("", text)
return text
def _extract_wikilinks(body: str) -> list[str]:
clean = _strip_code_blocks(body)
targets = []
for m in _WIKILINK_RE.finditer(clean):
raw = m.group(1)
target = raw.split("|")[0].split("#")[0].strip()
if target and _is_note_like(target):
targets.append(target)
return targets
def _normalize_name(name: str) -> str:
return name.lower().replace(" ", "-").replace("_", "-")
def build_note_index(vault_dir: Path) -> dict[str, Path]:
index: dict[str, Path] = {}
for p in vault_dir.rglob("*.md"):
if "archive" in p.parts or ".trash" in p.parts:
continue
key = _normalize_name(p.stem)
index[key] = p
return index
def check_dead_links(
path: Path,
body: str,
note_index: dict[str, Path],
) -> list[dict]:
findings = []
for target in _extract_wikilinks(body):
norm = _normalize_name(target)
if norm in note_index:
continue
findings.append({
"severity": "ERROR",
"check": "dead-link",
"message": f"dead wikilink [[{target}]]",
})
return findings
# ---------------------------------------------------------------------------
# Check 4: Orphans
# ---------------------------------------------------------------------------
ORPHAN_CAP = 40
def build_backlink_counts(
vault_dir: Path,
note_index: dict[str, Path],
) -> dict[str, int]:
counts: dict[str, int] = {k: 0 for k in note_index}
for p in vault_dir.rglob("*.md"):
if "archive" in p.parts or ".trash" in p.parts:
continue
try:
_, body = parse_frontmatter(p)
except Exception:
continue
for target in _extract_wikilinks(body):
norm = _normalize_name(target)
if norm in counts:
counts[norm] += 1
return counts
def find_orphans(
vault_dir: Path,
note_index: dict[str, Path],
backlink_counts: dict[str, int],
) -> list[dict]:
findings = []
for key, path in sorted(note_index.items()):
if backlink_counts.get(key, 0) == 0:
rel = path.relative_to(vault_dir)
findings.append({
"severity": "INFO",
"check": "orphan",
"message": f"no incoming links: {rel}",
"_path": path,
})
return findings[:ORPHAN_CAP]
# ---------------------------------------------------------------------------
# Gaps & suggestions
# ---------------------------------------------------------------------------
IMPORTANT_CANDIDATE_TERMS = [
"aida-nebra", "meshtastic", "headscale", "tailscale", "headplane",
"qdrant", "aurora", "open-webui", "gemini", "syncthing", "livesync",
"peertube", "mailcow", "forgejo", "jellyfin", "immich", "nextcloud",
"vaultwarden", "meshmonitor", "lora", "mt-isr", "sigil", "navi",
"searxng", "authentik", "docker", "proxmox", "caddy", "dnsmasq",
"acme-sh", "qdrant", "bge-m3", "qwen", "ollama", "meshtasticd",
"matrix-synapse", "element", "mautrix", "synapse",
]
def build_earned_a_doc_candidates(
vault_dir: Path,
note_index: dict[str, Path],
) -> list[tuple[str, int]]:
"""
Find terms mentioned in >= 4 docs that have no dedicated vault doc.
Returns [(term, mention_count)] sorted by count desc, top 15.
"""
existing_stems = set(note_index.keys())
doc_mentions: dict[str, set] = defaultdict(set)
for p in vault_dir.rglob("*.md"):
if "archive" in p.parts or ".trash" in p.parts:
continue
try:
text = p.read_text(encoding="utf-8", errors="replace")
except Exception:
continue
# Strip frontmatter
fm_match = re.match(r"^---.*?---\n", text, re.DOTALL)
body = text[fm_match.end():] if fm_match else text
# Strip code blocks
body = re.sub(r"```.*?```", "", body, flags=re.DOTALL)
body = re.sub(r"`[^`]+`", "", body)
body_lower = body.lower()
path_str = str(p)
# Check candidate terms
for term in IMPORTANT_CANDIDATE_TERMS:
norm = _normalize_name(term)
if norm in existing_stems:
continue # already has a doc
# Search for the term (and space variant) in body
variants = [term, term.replace("-", " ")]
for v in variants:
if v in body_lower:
doc_mentions[term].add(path_str)
break
# Also scan for hyphenated identifiers in wikilinks (already-linked terms
# that point to non-existent docs would show up as dead links above;
# here we look for plain-text mentions of kebab terms)
for m in re.finditer(r"\b([a-z][a-z0-9]{2,}-[a-z0-9][a-z0-9\-]{2,})\b", body_lower):
t = m.group(1)
if t not in existing_stems and len(t) >= 6:
# Filter out version strings, IPs, etc.
if not re.match(r"^\d", t) and "--" not in t:
doc_mentions[t].add(path_str)
# Filter to >= 4 mentions, sort by count desc, top 15
candidates = [
(term, len(docs))
for term, docs in doc_mentions.items()
if len(docs) >= 4
]
candidates.sort(key=lambda x: -x[1])
return candidates[:15]
def build_tag_coverage(
vault_dir: Path,
note_index: dict[str, Path],
) -> tuple[list[str], dict[str, list[str]]]:
"""
Returns (no_tag_paths, tag_to_paths) for shared-tag orphan check.
no_tag_paths: relative paths of docs with empty tags.
tag_to_paths: {tag: [doc_stem, ...]}
"""
no_tag_paths = []
tag_to_paths: dict[str, list[str]] = defaultdict(list)
for p in vault_dir.rglob("*.md"):
if "archive" in p.parts or ".trash" in p.parts:
continue
try:
fm, _ = parse_frontmatter(p)
except Exception:
fm = None
tags = []
if fm:
t = fm.get("tags")
if isinstance(t, list):
tags = t
stem = _normalize_name(p.stem)
if not tags:
no_tag_paths.append(str(p.relative_to(vault_dir)))
for tag in tags:
tag_to_paths[tag].append(stem)
return no_tag_paths, tag_to_paths
def build_gaps_section(
vault_dir: Path,
note_index: dict[str, Path],
backlink_counts: dict[str, int],
all_results: list,
allowed_tags: set[str],
) -> str:
"""Build the ## Gaps & suggestions section."""
lines = ["## Gaps & suggestions", ""]
# 1. Docs with NO tags
no_tag_paths, tag_to_paths = build_tag_coverage(vault_dir, note_index)
lines.append("### Docs with no tags")
lines.append("")
if no_tag_paths:
for p in sorted(no_tag_paths):
lines.append(f"- `{p}`")
else:
lines.append("_None — all docs have at least one tag._")
lines.append("")
# 2. True orphans (no inbound link AND no shared tag)
lines.append("### True orphans (no inbound link, no shared tag)")
lines.append("")
true_orphans = []
for key, path in sorted(note_index.items()):
if backlink_counts.get(key, 0) > 0:
continue
# Check if any of its tags appear in other docs
try:
fm, _ = parse_frontmatter(path)
except Exception:
fm = None
tags = []
if fm:
t = fm.get("tags")
if isinstance(t, list):
tags = t
# Does any other doc share a tag?
shares_tag = False
for tag in tags:
others = tag_to_paths.get(tag, [])
# Others = docs with this tag; if any != self, shares_tag = True
self_key = _normalize_name(path.stem)
if any(s != self_key for s in others):
shares_tag = True
break
if not shares_tag:
rel = path.relative_to(vault_dir)
true_orphans.append(str(rel))
if true_orphans:
for p in true_orphans[:20]:
lines.append(f"- `{p}`")
else:
lines.append("_None._")
lines.append("")
# 3. Dead wikilinks (already in ERROR section, just summarize)
dead_count = sum(
1 for r in all_results
for f in r.findings
if f["severity"] == "ERROR" and f["check"] == "dead-link"
)
lines.append("### Dead wikilinks")
lines.append("")
if dead_count == 0:
lines.append("_None — zero dead wikilinks. All [[links]] resolve._")
else:
lines.append(f"**{dead_count} dead wikilinks** — see ERROR section above for details.")
lines.append("")
# 4. Tags not in topic_categories
lines.append("### Unknown tags (not in topic_categories)")
lines.append("")
unknown_tags: list[str] = []
for r in all_results:
for f in r.findings:
if f["severity"] == "WARN" and f["check"] == "tag-vocab":
unknown_tags.append(f["message"])
if unknown_tags:
for msg in unknown_tags:
lines.append(f"- {msg}")
else:
lines.append("_None — all tags are in the controlled vocabulary._")
lines.append("")
# 5. "Earned-a-doc?" candidates
lines.append("### Earned-a-doc? candidates (terms in ≥4 docs, no dedicated doc)")
lines.append("")
lines.append("Terms mentioned frequently across the vault but with no dedicated doc.")
lines.append("Matt decides whether to create a real doc — when he does, future sweeps will link to it.")
lines.append("")
candidates = build_earned_a_doc_candidates(vault_dir, note_index)
if candidates:
lines.append("| Term | Docs mentioning it |")
lines.append("|------|--------------------|")
for term, count in candidates:
lines.append(f"| `{term}` | {count} |")
else:
lines.append("_No candidates found (all frequent terms already have dedicated docs)._")
lines.append("")
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------
class LintResult(NamedTuple):
path: Path
findings: list[dict]
def format_report(
results: list[LintResult],
vault_dir: Path,
total_docs: int,
elapsed_s: float,
gaps_section: str,
) -> str:
ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
errors = [f for r in results for f in r.findings if f["severity"] == "ERROR"]
warns = [f for r in results for f in r.findings if f["severity"] == "WARN"]
infos = [f for r in results for f in r.findings if f["severity"] == "INFO"]
fm_missing = sum(
1 for r in results for f in r.findings
if f["severity"] == "WARN" and f["check"] == "frontmatter"
and "missing frontmatter block" in f["message"]
)
fm_invalid = sum(
1 for r in results for f in r.findings
if f["severity"] == "WARN" and f["check"] == "frontmatter"
and "missing frontmatter block" not in f["message"]
)
tag_warns = sum(
1 for r in results for f in r.findings
if f["severity"] == "WARN" and f["check"] == "tag-vocab"
)
lines = [
"# Vault Lint Report",
"",
f"Generated: {ts} | Docs scanned: {total_docs} | Elapsed: {elapsed_s:.1f}s",
"",
"## Summary",
"",
"| Severity | Count |",
"|----------|-------|",
f"| ERROR (dead links) | {len(errors)} |",
f"| WARN (schema) | {len(warns)} |",
f"| INFO (orphans) | {len(infos)} |",
"",
"### WARN breakdown",
f"- Missing frontmatter block: {fm_missing}",
f"- Invalid/missing frontmatter fields: {fm_invalid}",
f"- Unknown tags: {tag_warns}",
"",
]
if errors:
lines.append("## ERROR — Dead Wikilinks")
lines.append("")
for r in results:
errs = [f for f in r.findings if f["severity"] == "ERROR"]
if errs:
rel = r.path.relative_to(vault_dir)
for f in errs:
lines.append(f"- `{rel}` — {f['message']}")
lines.append("")
else:
lines.append("## ERROR — Dead Wikilinks")
lines.append("")
lines.append("_None. All wikilinks resolve._")
lines.append("")
if warns:
lines.append("## WARN — Schema & Tag Violations")
lines.append("")
for r in results:
ws = [f for f in r.findings if f["severity"] == "WARN"]
if ws:
try:
rel = r.path.relative_to(vault_dir)
except ValueError:
rel = r.path
for f in ws:
lines.append(f"- `{rel}` — {f['message']}")
lines.append("")
if infos:
lines.append(f"## INFO — Orphan Notes (no incoming links, capped at {ORPHAN_CAP})")
lines.append("")
for f in infos:
lines.append(f"- {f['message']}")
lines.append("")
# Append gaps & suggestions section
lines.append(gaps_section)
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def run_lint(vault_dir: Path, engine_dir: Path) -> tuple[list[LintResult], dict]:
import time
config = load_config(engine_dir)
vocab = load_vocab(engine_dir)
topic_cats = set(vocab.get("topic_categories", []))
# No extra allowed tags in v4 (removed 'meta' catch-all)
note_index = build_note_index(vault_dir)
backlink_counts = build_backlink_counts(vault_dir, note_index)
all_paths = sorted(vault_dir.rglob("*.md"))
results: list[LintResult] = []
t0 = time.monotonic()
for path in all_paths:
if "archive" in path.parts or ".trash" in path.parts:
continue
findings: list[dict] = []
try:
fm, body = parse_frontmatter(path)
except Exception as exc:
findings.append({
"severity": "WARN",
"check": "parse-error",
"message": f"could not parse: {exc}",
})
results.append(LintResult(path=path, findings=findings))
continue
findings.extend(check_frontmatter(fm, config))
findings.extend(check_tags(fm, topic_cats))
findings.extend(check_dead_links(path, body, note_index))
results.append(LintResult(path=path, findings=findings))
# Orphan check
orphan_findings = find_orphans(vault_dir, note_index, backlink_counts)
if orphan_findings:
results.append(LintResult(path=vault_dir / "_orphans_", findings=orphan_findings))
elapsed = time.monotonic() - t0
stats = {
"total_docs": len(all_paths),
"errors": sum(1 for r in results for f in r.findings if f["severity"] == "ERROR"),
"warns": sum(1 for r in results for f in r.findings if f["severity"] == "WARN"),
"infos": sum(1 for r in results for f in r.findings if f["severity"] == "INFO"),
"elapsed_s": elapsed,
"note_index": note_index,
"backlink_counts": backlink_counts,
"topic_cats": topic_cats,
}
return results, stats
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
import time
strict = "--strict" in sys.argv
this_file = Path(__file__).resolve()
engine_dir = this_file.parent.parent
config = load_config(engine_dir)
vault_str = config.get("vault_dir", "")
vault_dir = Path(vault_str) if vault_str else engine_dir.parent / "vault"
if not vault_dir.exists():
print(f"ERROR: vault_dir not found: {vault_dir}", file=sys.stderr)
sys.exit(1)
results, stats = run_lint(vault_dir, engine_dir)
gaps_section = build_gaps_section(
vault_dir,
stats["note_index"],
stats["backlink_counts"],
results,
stats["topic_cats"],
)
report_text = format_report(
results, vault_dir, stats["total_docs"], stats["elapsed_s"],
gaps_section,
)
report_path = engine_dir / "lint-report.md"
report_path.write_text(report_text, encoding="utf-8")
print(report_text)
print(f"--- Wrote: {report_path} ---")
print(
f"\nSummary: {stats['total_docs']} docs | "
f"{stats['errors']} ERRORs | {stats['warns']} WARNs | {stats['infos']} INFOs"
)
if strict and stats["errors"] > 0:
sys.exit(1)
sys.exit(0)
if __name__ == "__main__":
main()

1189
engine/lib/vocab_gen.py
View file

File diff suppressed because it is too large Load diff

125
engine/lint-report.md
View file

@ -1,125 +0,0 @@
# Vault Lint Report
Generated: 2026-08-26T09:00:02Z | Docs scanned: 116 | Elapsed: 0.0s
## Summary
| Severity | Count |
|----------|-------|
| ERROR (dead links) | 27 |
| WARN (schema) | 0 |
| INFO (orphans) | 35 |
### WARN breakdown
- Missing frontmatter block: 0
- Invalid/missing frontmatter fields: 0
- Unknown tags: 0
## ERROR — Dead Wikilinks
- `docs/hardware/environment.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/environment.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/environment.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/environment.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/environment.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/ip-allocation.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/ip-allocation.md` — dead wikilink [[2026-06-19]]
- `docs/hardware/ip-allocation.md` — dead wikilink [[2026-06-19]]
- `docs/services/services.md` — dead wikilink [[2026-06-19]]
- `docs/services/services.md` — dead wikilink [[2026-06-19]]
- `docs/services/services.md` — dead wikilink [[2026-06-19]]
- `docs/services/services.md` — dead wikilink [[2026-06-19]]
- `docs/services/services.md` — dead wikilink [[2026-06-19]]
- `docs/software/caddy.md` — dead wikilink [[2026-06-19]]
- `docs/software/dns.md` — dead wikilink [[2026-06-19]]
- `docs/software/searxng.md` — dead wikilink [[2026-06-19]]
- `glossary.md` — dead wikilink [[2026-06-19]]
- `glossary.md` — dead wikilink [[2026-06-19]]
- `projects/fleet-patch-audit.md` — dead wikilink [[2026-06-19]]
- `projects/fleet-patch-audit.md` — dead wikilink [[2026-06-19]]
- `projects/fleet-patch-audit.md` — dead wikilink [[Nominatim v5 Re-import]]
- `projects/fleet-patch-audit.md` — dead wikilink [[Nominatim v5 Re-import]]
- `projects/fleet-platform-baseline.md` — dead wikilink [[2026-06-19]]
- `projects/nominatim-v5-reimport.md` — dead wikilink [[2026-06-19]]
- `runbooks/edge2-boot-recovery.md` — dead wikilink [[2026-06-19]]
- `runbooks/lxc-service-migration.md` — dead wikilink [[2026-06-19]]
- `runbooks/syncthing-add-node.md` — dead wikilink [[2026-06-19]]
## INFO — Orphan Notes (no incoming links, capped at 40)
- no incoming links: runbooks/authentik-access-groups.md
- no incoming links: runbooks/authentik-create-invitation.md
- no incoming links: runbooks/authentik-oidc-application.md
- no incoming links: runbooks/authentik-upgrade.md
- no incoming links: docs/navi/cc-rules.md
- no incoming links: CLAUDE-baseline.md
- no incoming links: runbooks/ct-runbook.md
- no incoming links: notes/echo6-landing-page-data-export.md
- no incoming links: runbooks/expose-service-home.md
- no incoming links: runbooks/fleet-magicdns-resolved-migration.md
- no incoming links: projects/fleet-platform-baseline.md
- no incoming links: docs/software/geo-tools.md
- no incoming links: glossary.md
- no incoming links: runbooks/headscale-oidc-boot-order.md
- no incoming links: runbooks/ia-cli-reference.md
- no incoming links: runbooks/ia-download-mirror.md
- no incoming links: notes/ia-download-queue.md
- no incoming links: runbooks/idahomesh-bridge-setup.md
- no incoming links: runbooks/idahomesh-vpn-device-setup.md
- no incoming links: runbooks/lxc-service-migration.md
- no incoming links: runbooks/mailcow-create-mailbox.md
- no incoming links: projects/meshai-config-hot-apply.md
- no incoming links: runbooks/meshai-prod-compose-override.md
- no incoming links: runbooks/meshmonitor-password-reset.md
- no incoming links: runbooks/meshtasticd-sim-nodes-runbook.md
- no incoming links: runbooks/nordvpn-lxc.md
- no incoming links: runbooks/pg-backup.md
- no incoming links: runbooks/pipeline-patterns.md
- no incoming links: runbooks/proxmox-create-ubuntu-vm.md
- no incoming links: runbooks/pymc-repeater-kiss-tnc-reenumeration.md
- no incoming links: runbooks/recon-operations.md
- no incoming links: runbooks/recon-service-integration.md
- no incoming links: session-resume/SESSION-HANDOFF-meshai-test.md
- no incoming links: docs/matrix/synapse_retention_discovery.md
- no incoming links: runbooks/syncthing-add-node.md
## Gaps & suggestions
### Docs with no tags
_None — all docs have at least one tag._
### True orphans (no inbound link, no shared tag)
- `runbooks/nordvpn-lxc.md`
### Dead wikilinks
**27 dead wikilinks** — see ERROR section above for details.
### Unknown tags (not in topic_categories)
_None — all tags are in the controlled vocabulary._
### Earned-a-doc? candidates (terms in ≥4 docs, no dedicated doc)
Terms mentioned frequently across the vault but with no dedicated doc.
Matt decides whether to create a real doc — when he does, future sweeps will link to it.
| Term | Docs mentioning it |
|------|--------------------|
| `tailscale` | 42 |
| `docker` | 38 |
| `proxmox` | 34 |
| `headscale` | 25 |
| `peertube` | 25 |
| `meshtastic` | 20 |
| `mailcow` | 17 |
| `immich` | 16 |
| `nextcloud` | 15 |
| `forgejo` | 14 |
| `element` | 14 |
| `livesync` | 13 |
| `jellyfin` | 13 |
| `vaultwarden` | 13 |
| `meshmonitor` | 13 |

52
engine/prompts/fewshot.md
View file

@ -1,52 +0,0 @@
# Echo6 Vault Tagger — Few-Shot Examples
These examples will be populated from real vault documents during the tagger implementation
step (Step 5). Each example should show an input document snippet and the exact JSON output
the tagger should produce, demonstrating correct vocabulary usage.
TODO: Add 3–5 real examples drawn from actual vault docs (runbook, reference, project, note).
---
## Template: Example Structure
### Input (document snippet)
```markdown
---
title: Headscale Setup
type: runbook
tags: [mesh, vpn]
updated: 2025-03-10
---
# Headscale Setup
Steps to install and configure Headscale on the Contabo VPS...
```
### Expected JSON output
```json
{
"tags": ["mesh", "vpn"],
"entities": ["contabo", "headscale"],
"glossary_proposals": [],
"type": "runbook",
"confidence": 0.95
}
```
### Notes on this example
- `tags` uses only values from `topic_categories`
- `entities` matches names present in `entity_lexicon` (contabo is a proxmox_vps; headscale is a known service)
- `confidence` is high because type is explicitly set in frontmatter and tags are unambiguous
- If the frontmatter had said `type: guide` (not in the allowed list), the tagger would infer `runbook` from the content and note the discrepancy
---
TODO: Add example 2 — reference doc (host description, multiple entities)
TODO: Add example 3 — note/session doc (low entity density, inferred type)
TODO: Add example 4 — ambiguous doc with glossary_proposals populated
TODO: Add example 5 — doc where confidence drops below 0.6

67
engine/prompts/system.md
View file

@ -1,67 +0,0 @@
# Echo6 Vault Tagger — System Prompt (Canonical)
## Role
You are the Echo6 vault tagger, a local AI assistant running on cortex (RTX A4000).
Your sole job is to classify Obsidian markdown documents and extract structured metadata
from them using a controlled vocabulary. You operate fully offline and deterministically.
## Inputs (provided per call)
- **document**: the full text of a markdown file (frontmatter + body)
- **topic_categories**: a stable list of tier-1 topic tags (e.g. mesh, auth, proxmox, ai)
- **entity_lexicon**: a generated JSON dictionary mapping known names to type
(hosts, services, containers, projects, acronyms) — tier 2 vocabulary
## Output
Respond with ONLY a single valid JSON object. No prose, no markdown fences, no explanation.
```json
{
"tags": [ "string", "..." ],
"entities": [ "string", "..." ],
"glossary_proposals": [ "string", "..." ],
"type": "string",
"confidence": 0.0
}
```
Field definitions:
- **tags**: up to 3 tier-1 topic tags drawn exclusively from topic_categories, ordered most-relevant to least-relevant (primary topic first)
- **entities**: known names matched from entity_lexicon
- **glossary_proposals**: unknown acronyms or terms worth adding to the lexicon
- **type**: one of reference | runbook | project | note | index | session
- **confidence**: float 0.0–1.0, your overall confidence in this classification
## Rules — follow exactly
1. **Only use provided vocabulary.** `tags` must be a subset of `topic_categories`; return at most 3, ordered most-relevant to least-relevant.
`entities` must be a subset of the keys in `entity_lexicon`. Never invent new tags.
2. **Strict JSON only.** The output must parse with `json.loads()` with no preprocessing.
No trailing commas. No comments. No markdown code fences around the JSON.
3. **Low confidence — flag, do not guess.** If `confidence < 0.6`, still emit valid JSON
but keep `tags` and `entities` conservative — only include what you are sure of.
Add uncertain terms to `glossary_proposals` instead.
4. **Never hallucinate expansions.** If you encounter an acronym not in `entity_lexicon`,
do NOT guess its expansion. Add the raw acronym to `glossary_proposals`.
5. **Never fabricate wikilinks or related files.** You output metadata only.
6. **Type inference.** Use the document frontmatter `type` field if present and valid.
Otherwise infer from content: runbooks have steps/commands; references describe systems;
projects track work; sessions are journal/meeting notes; index files link to others.
7. **Tags are used as-is** from the vocab list — do not pluralize or alter them.
## Confidence scoring guide
| Range | Meaning |
|-----------|----------------------------------------------------------------------|
| 0.9–1.0 | Clear topic, entities all recognized, type obvious |
| 0.7–0.89 | Good confidence; minor ambiguity in one dimension |
| 0.6–0.69 | Borderline; result written but flagged in changelog |
| below 0.6 | Do not apply silently; flag for human review |

3492
engine/sweep-full.log
View file

File diff suppressed because it is too large Load diff

226
engine/sweep.sh
View file

@ -1,226 +0,0 @@
#!/usr/bin/env bash
# sweep.sh — Echo6 Vault Engine orchestrator
#
# Usage:
# ./sweep.sh # default: process docs changed since last sweep
# ./sweep.sh --all # process every non-archived vault doc
# ./sweep.sh --dry-run # pass --dry-run to agent.py (no writes)
# ./sweep.sh --all --dry-run # both
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VAULT_DIR="${SCRIPT_DIR}/../vault"
CONFIG="${SCRIPT_DIR}/config.yaml"
LAST_SWEEP="${SCRIPT_DIR}/.last-sweep"
CHANGELOG="${SCRIPT_DIR}/changelog.md"
LINT_REPORT="${SCRIPT_DIR}/lint-report.md"
# Parse args
OPT_ALL=0
OPT_DRY=0
for arg in "$@"; do
case "$arg" in
--all) OPT_ALL=1 ;;
--dry-run) OPT_DRY=1 ;;
*)
echo "[error] Unknown argument: $arg" >&2
echo "Usage: $0 [--all] [--dry-run]" >&2
exit 1
;;
esac
done
SWEEP_START="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "==> Echo6 vault sweep — ${SWEEP_START}"
if [ "$OPT_DRY" -eq 1 ]; then
echo " (DRY-RUN mode — no files will be modified)"
fi
# ---------------------------------------------------------------------------
# Step 1 — GPU-busy guard
# ---------------------------------------------------------------------------
echo ""
echo "[1/4] GPU-busy guard..."
# Helper: run nvidia-smi, using docker exec ollama if host lacks it
_nvidia_smi() {
if command -v nvidia-smi &>/dev/null; then
nvidia-smi "$@" 2>/dev/null
elif command -v docker &>/dev/null && docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^ollama$'; then
docker exec ollama nvidia-smi "$@" 2>/dev/null
else
return 1
fi
}
GPU_DEFERRED=0
DEFER_REASON=""
# Guard 1 — competing GPU processes (share the A4000)
# Note: peertube-runner is intentionally excluded — it's a persistent server
# daemon that's always running, so its mere presence doesn't mean the GPU is
# busy (only an active transcode job does, and that spawns ffmpeg, which is
# still caught here; actual peertube contention is caught by the util/VRAM
# guards below).
if pgrep -fa 'whisper|ffmpeg' >/dev/null 2>&1; then
COMPETING="$(pgrep -fa 'whisper|ffmpeg' | head -1)"
DEFER_REASON="competing GPU process: ${COMPETING}"
GPU_DEFERRED=1
fi
# Guard 2 — sustained GPU utilization > 60 %
if [ "$GPU_DEFERRED" -eq 0 ]; then
GPU_UTIL="$(_nvidia_smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d '[:space:]' || true)"
if [ -n "$GPU_UTIL" ] && [[ "$GPU_UTIL" =~ ^[0-9]+$ ]]; then
echo " GPU utilization: ${GPU_UTIL}%"
if [ "$GPU_UTIL" -gt 60 ]; then
DEFER_REASON="GPU utilization ${GPU_UTIL}% > 60%"
GPU_DEFERRED=1
fi
else
echo " nvidia-smi not available or could not read utilization — skipping util guard."
fi
fi
# Guard 3 (safety cap) — very high VRAM usage (>14000 MiB) — not triggered by vault-tagger itself (~8-10 GB)
if [ "$GPU_DEFERRED" -eq 0 ]; then
GPU_USED_MIB="$(_nvidia_smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d '[:space:]' || true)"
if [ -n "$GPU_USED_MIB" ] && [[ "$GPU_USED_MIB" =~ ^[0-9]+$ ]]; then
echo " GPU VRAM used: ${GPU_USED_MIB} MiB (safety cap: 14000 MiB)"
if [ "$GPU_USED_MIB" -gt 14000 ]; then
DEFER_REASON="VRAM ${GPU_USED_MIB} MiB > 14000 MiB safety cap"
GPU_DEFERRED=1
fi
fi
fi
if [ "$GPU_DEFERRED" -eq 1 ]; then
echo " GPU busy — deferring sweep. Reason: ${DEFER_REASON}"
if [ "$OPT_DRY" -eq 0 ]; then
printf "\n## %s — sweep deferred (%s)\n" "$SWEEP_START" "$DEFER_REASON" >> "$CHANGELOG"
fi
exit 0
fi
echo " GPU OK — proceeding."
# ---------------------------------------------------------------------------
# Step 2 — Lint
# ---------------------------------------------------------------------------
echo ""
echo "[2/4] Running lint..."
python3 "${SCRIPT_DIR}/lib/lint.py" --report 2>&1 | tee /tmp/sweep-lint.log || {
echo "[warn] lint.py exited non-zero — continuing sweep"
}
echo " Lint complete. Report: ${LINT_REPORT}"
# ---------------------------------------------------------------------------
# Step 3 — Select docs and run agent
# ---------------------------------------------------------------------------
echo ""
echo "[3/4] Selecting documents..."
# Build file list
DOC_LIST=()
if [ "$OPT_ALL" -eq 1 ] || [ ! -f "$LAST_SWEEP" ]; then
if [ "$OPT_ALL" -eq 1 ]; then
echo " Mode: --all (full vault scan)"
else
echo " Mode: no .last-sweep marker found — treating as first run (full scan)"
fi
while IFS= read -r -d '' f; do
# Skip archive
if [[ "$f" == *"/archive/"* || "$f" == *"/.trash/"* ]]; then
continue
fi
DOC_LIST+=("$f")
done < <(find "${VAULT_DIR}" -name "*.md" -type f -print0 | sort -z) # -type f excludes symlinks (e.g. CLAUDE-baseline.md, vault/rules/)
else
LAST_TS="$(cat "$LAST_SWEEP")"
echo " Mode: incremental — docs modified since ${LAST_TS}"
# Compare file mtime to .last-sweep mtime (not contents)
while IFS= read -r -d '' f; do
if [[ "$f" == *"/archive/"* || "$f" == *"/.trash/"* ]]; then
continue
fi
DOC_LIST+=("$f")
done < <(find "${VAULT_DIR}" -name "*.md" -type f -newer "$LAST_SWEEP" -print0 | sort -z) # -type f excludes symlinks
fi
TOTAL="${#DOC_LIST[@]}"
echo " ${TOTAL} document(s) selected."
if [ "$TOTAL" -eq 0 ]; then
echo " Nothing to process — vault is up to date."
if [ "$OPT_DRY" -eq 0 ]; then
date -u +%Y-%m-%dT%H:%M:%SZ > "$LAST_SWEEP"
printf "\n## %s — sweep complete (0 docs selected)\n" "$SWEEP_START" >> "$CHANGELOG"
fi
echo "==> Done."
exit 0
fi
echo ""
echo " Processing..."
N_PROCESSED=0
N_WRITTEN=0
N_FLAGGED=0
N_ERRORS=0
AGENT_DRY_FLAG=""
if [ "$OPT_DRY" -eq 1 ]; then
AGENT_DRY_FLAG="--dry-run"
fi
for doc in "${DOC_LIST[@]}"; do
echo ""
echo " --- $(basename "$doc") ---"
set +e
python3 "${SCRIPT_DIR}/lib/agent.py" $AGENT_DRY_FLAG "$doc" 2>&1
EXIT_CODE=$?
set -e
N_PROCESSED=$(( N_PROCESSED + 1 ))
if [ "$EXIT_CODE" -eq 0 ]; then
N_WRITTEN=$(( N_WRITTEN + 1 ))
else
echo " [warn] agent.py exited ${EXIT_CODE} for ${doc}"
N_ERRORS=$(( N_ERRORS + 1 ))
fi
done
# ---------------------------------------------------------------------------
# Step 4 — Update marker + changelog summary
# ---------------------------------------------------------------------------
echo ""
echo "[4/4] Finalising..."
SWEEP_END="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
if [ "$OPT_DRY" -eq 0 ]; then
date -u +%Y-%m-%dT%H:%M:%SZ > "$LAST_SWEEP"
{
printf "\n## %s — sweep run\n" "$SWEEP_START"
printf -- "- end: %s\n" "$SWEEP_END"
printf -- "- mode: %s\n" "$([ "$OPT_ALL" -eq 1 ] && echo "--all" || echo "incremental")"
printf -- "- docs selected: %d\n" "$TOTAL"
printf -- "- processed: %d | written: %d | flagged: %d | errors: %d\n" \
"$N_PROCESSED" "$N_WRITTEN" "$N_FLAGGED" "$N_ERRORS"
} >> "$CHANGELOG"
echo " .last-sweep updated. Changelog appended."
else
echo " [DRY-RUN] .last-sweep NOT updated. Changelog NOT written."
fi
echo ""
echo "==> Sweep complete."
echo " Docs selected : ${TOTAL}"
echo " Processed : ${N_PROCESSED}"
echo " Written/OK : ${N_WRITTEN}"
echo " Errors : ${N_ERRORS}"
if [ "$OPT_DRY" -eq 1 ]; then
echo " [DRY-RUN] No vault docs were modified."
fi

1195
engine/vocab.json
View file

File diff suppressed because it is too large Load diff

11
vault/.gitignore vendored
View file

@ -1,11 +0,0 @@
# Binary archives — no diff value in git
*.zip
*.tar.gz
*.tar.zst
# Editor temp files
*.swp
*.bak
*~
.DS_Store

1
vault/.obsidian/app.json vendored
View file

@ -1 +0,0 @@
{}

4
vault/.obsidian/appearance.json vendored
View file

@ -1,4 +0,0 @@
{
"accentColor": "#ff6400",
"theme": "obsidian"
}

3
vault/.obsidian/community-plugins.json vendored
View file

@ -1,3 +0,0 @@
[
"obsidian-livesync"
]

33
vault/.obsidian/core-plugins.json vendored
View file

@ -1,33 +0,0 @@
{
"file-explorer": true,
"global-search": true,
"switcher": true,
"graph": true,
"backlink": true,
"canvas": true,
"outgoing-link": true,
"tag-pane": true,
"footnotes": false,
"properties": true,
"page-preview": true,
"daily-notes": true,
"templates": true,
"note-composer": true,
"command-palette": true,
"slash-command": false,
"editor-status": true,
"bookmarks": true,
"markdown-importer": false,
"zk-prefixer": false,
"random-note": false,
"outline": true,
"word-count": true,
"slides": false,
"audio-recorder": false,
"workspaces": false,
"file-recovery": true,
"publish": false,
"sync": true,
"bases": true,
"webviewer": false
}

65
vault/.obsidian/graph.json vendored
View file

@ -1,65 +0,0 @@
{
"collapse-filter": false,
"search": "",
"showTags": true,
"showAttachments": false,
"hideUnresolved": false,
"showOrphans": true,
"collapse-color-groups": true,
"colorGroups": [
{
"query": "path:entities/",
"color": {
"a": 1,
"rgb": 16736330
}
},
{
"query": "path:concepts/",
"color": {
"a": 1,
"rgb": 16744272
}
},
{
"query": "path:runbooks/",
"color": {
"a": 1,
"rgb": 4359424
}
},
{
"query": "path:docs/",
"color": {
"a": 1,
"rgb": 3394815
}
},
{
"query": "path:projects/",
"color": {
"a": 1,
"rgb": 10040063
}
},
{
"query": "path:notes/",
"color": {
"a": 1,
"rgb": 16776960
}
}
],
"collapse-display": false,
"showArrow": true,
"textFadeMultiplier": -1.6,
"nodeSizeMultiplier": 1.22291666666667,
"lineSizeMultiplier": 1.80989583333333,
"collapse-forces": false,
"centerStrength": 0.328125,
"repelStrength": 20,
"linkStrength": 1,
"linkDistance": 500,
"scale": 0.1316872427983537,
"close": false
}

175
vault/.obsidian/plugins/obsidian-livesync/data.json vendored
View file

@ -1,175 +0,0 @@
{
"remoteType": "",
"useCustomRequestHandler": false,
"couchDB_URI": "https://notes.echo6.co",
"couchDB_USER": "",
"couchDB_PASSWORD": "",
"couchDB_DBNAME": "cc-db",
"liveSync": true,
"syncOnSave": false,
"syncOnStart": true,
"savingDelay": 200,
"lessInformationInLog": false,
"gcDelay": 0,
"versionUpFlash": "",
"minimumChunkSize": 20,
"longLineThreshold": 250,
"showVerboseLog": false,
"suspendFileWatching": false,
"trashInsteadDelete": true,
"periodicReplication": false,
"periodicReplicationInterval": 60,
"syncOnFileOpen": false,
"encrypt": false,
"passphrase": "",
"usePathObfuscation": false,
"doNotDeleteFolder": false,
"resolveConflictsByNewerFile": false,
"batchSave": true,
"batchSaveMinimumDelay": 5,
"batchSaveMaximumDelay": 60,
"deviceAndVaultName": "",
"usePluginSettings": false,
"showOwnPlugins": false,
"showStatusOnEditor": true,
"showStatusOnStatusbar": true,
"showOnlyIconsOnEditor": false,
"hideFileWarningNotice": false,
"networkWarningStyle": "",
"usePluginSync": false,
"autoSweepPlugins": false,
"autoSweepPluginsPeriodic": false,
"notifyPluginOrSettingUpdated": false,
"checkIntegrityOnSave": false,
"batch_size": 25,
"batches_limit": 25,
"useHistory": true,
"disableRequestURI": true,
"skipOlderFilesOnSync": true,
"checkConflictOnlyOnOpen": false,
"showMergeDialogOnlyOnActive": false,
"syncInternalFiles": false,
"syncInternalFilesBeforeReplication": false,
"syncInternalFilesIgnorePatterns": "\\/node_modules\\/, \\/\\.git\\/, \\/obsidian-livesync\\/",
"syncInternalFilesTargetPatterns": "",
"syncInternalFilesInterval": 60,
"additionalSuffixOfDatabaseName": "",
"ignoreVersionCheck": false,
"lastReadUpdates": 0,
"deleteMetadataOfDeletedFiles": false,
"syncIgnoreRegEx": "",
"syncOnlyRegEx": "",
"customChunkSize": 60,
"readChunksOnline": true,
"watchInternalFileChanges": true,
"automaticallyDeleteMetadataOfDeletedFiles": 0,
"disableMarkdownAutoMerge": false,
"writeDocumentsIfConflicted": false,
"useDynamicIterationCount": false,
"syncAfterMerge": false,
"configPassphraseStore": "",
"encryptedPassphrase": "",
"encryptedCouchDBConnection": "%$NA/UWgH5dpizBJ9wynM8oH46b8eDpK/irIUl0+auH7Xt9PXtqetQ4v+OF4Rv3I0ofiV+RyS9jiw8pmslHpl0QKX+10TTZLahMgjHWa6Br1UR35bDF+ipYa5mJ1FW2SKzBsh8I/OAvtDP+Xbhwc6wtpWDZd7cAlBK3RiRjO4H1EqafMlCaTIn4YzhsKxHhvnhDxoj3GMWSb9cTkL9XMoklU7K1s9dmy8NbtBuEcskd38O1Nvi+OpQ7LxvyeiTX5WE+cT5QKDQXmo8YmJWblAZcGufZRLoQTadfqqrIiAbx+FHzgt+NCHDMhsq9cSsQk3MjId8MIpd8ZBsy2DEq0o62widVjXHwOZEiQCtweelT7JyMgVzjmcGTw7VGSF2GNG6I08tKMOS8fTHCNH1cczhjIDqOepuPDgTNbtK7iVPipRveVAWG15wHVk+ajTH+gI7nI7f6GeLbM3rm/TA+Z87jMAgwSAc145tAs8SaqLLldZLiKgVZUOAoSHIdusTVaxuLAzLd9pM5id3GzGhYO72HOlAK+LQ7hjYI9ae+r678oKEWXFEsqTwDIQ+YI5eSCNqHjZFvA6BWzwtnNZjbr9nkGqBJGJPNl0a/0IhMcCZ4FVVFhV85HxmxuRsLJChBq+O1QNbxgW70O6RnSrLGqt45P5slH1U0VJuBH0ojwCz/GZ5JNro9pSSM+89Fcxc+ybYjsZdd0Xjoni8bZiUvI7UiYrrlmE3rv5P7UfRQtcT9bbSiy4afB8uActMNehDw1pAOOD60xjfjbT8HWNCWjCtuq7XHml4i5Awyi96+6dH2iAB643lQnFY6nZHUoN8FNRvb/iP3HMbdSXmHkTL3IHAKGLgqA5YxlidcZPB+HKsSJ0PVBV6T/eQbQ10QQCXiitRkmm9R/tKgAj5nM6Ovtgrb12MU+8S7/Gt4h4q+t/1AVLHWcitUhuwOpMSHkFlA6jU+IOHFv29VktQ2rjbFWM8ZNqDOMvgXeUH/7RyvE+5S6Z0QLifQu0wBYgLNoM9T6vLjBC6icOdZt2sNM7cuDmYFj4wCW5nUL05y1j0ApIp3XztHrZsZqGCLbzi/tvHwWiMRI5bYsANe69earnFH6yFU0JqOyUz4kJL8x5YxFlBF2BsnCbEd7UxILRvhJtoGRO8ERG0aBBgzhufGQS9gA9IZmCIpvJx06SMMeJiPMWJLaidNLXWIuDam6qWc8OKNiDh",
"permitEmptyPassphrase": false,
"remoteConfigurations": {
"legacy-couchdb": {
"id": "legacy-couchdb",
"name": "CouchDB Remote",
"uri": "%$NA/UWgH5dpizBJ9wynM8oH46b8eDpK/irIUl0+auH7XWM1auwn9V2hYzAR3bYfYmjGreYlBfj3zkwsY9hQMwnfunoYlYjiM0DyZcH+Z2cllge/7Z9fRllpb0bmLiwVqcyWmHPrtkpDHfm79HGl0Z+5xmQKXwBj0rPV3YDlhHvpnNJAw7oeJDpflAblTiJVO5V6XY5jDhU7ibVWSCr+zd5Y3/noYQQ8r9XUwsunNGcAEtD3cHcCGAiCdn4gwFSqfgE024wGYx8eYmVRTizr3wcVLirY6h+xh1D2Jb2z3AIwHV3veStn2rKFJ8KoMgWFOBduPnSlQiIQ7hyrqTZGSu+Uc3hdm8rkboiSOmad8RRowxVaApgFi1ihfnfAXdVr+MAgPA3UFCMRuI0+q8yfjvKPCeUDIWSdP0josvJUkQ6CDnhDTOb5At/S2lT+naU0JVP6gv3RElxtTIyAlRS1GoamRf4h68LiFgXRHPrbiVrvEqJ7kZLsOkrS+iMpJ/mkFZdlbuWkD7ayYIZ95p2DruiZ31ge8OkJeBE2Yk2POvWuton10sgxL74n9D/Y+bCHeFrTTKFxTy10Ky8VAD6+xbFV0GK5qsO8gv4t66L2g1mLzc2YbCgH9Zd9arrGPiX6Dppk0rLkYLhZmliVA8KTK2ow1nKlPqbvhlQg9peaWXDcjdbHudXB/1GQFzqpN9+pqj3gpy4p9fN3fTpOqDMJvQNBbjLqArk9oSS2DVm7AZ4qFLSAS6G1Bspi043TnJB5Ap7pnwDLJZgBrsNQDiC96F/a5U0n9MS2Aj0PgDrPM6CJXYEgAgPbx/lSK7Tw==",
"isEncrypted": true
}
},
"activeConfigurationId": "legacy-couchdb",
"P2P_ActiveRemoteConfigurationId": "",
"useIndexedDBAdapter": false,
"useTimeouts": false,
"writeLogToTheFile": false,
"doNotPaceReplication": false,
"hashCacheMaxCount": 300,
"hashCacheMaxAmount": 50,
"concurrencyOfReadChunksOnline": 40,
"minimumIntervalOfReadChunksOnline": 50,
"hashAlg": "xxhash64",
"suspendParseReplicationResult": false,
"doNotSuspendOnFetching": false,
"useIgnoreFiles": false,
"ignoreFiles": ".gitignore",
"syncOnEditorSave": false,
"keepReplicationActiveInBackground": false,
"pluginSyncExtendedSetting": {},
"syncMaxSizeInMB": 50,
"settingSyncFile": "",
"writeCredentialsForSettingSync": false,
"notifyAllSettingSyncFile": false,
"isConfigured": true,
"settingVersion": 10,
"enableCompression": false,
"accessKey": "",
"bucket": "",
"endpoint": "",
"region": "",
"secretKey": "",
"useEden": false,
"maxChunksInEden": 10,
"maxTotalLengthInEden": 1024,
"maxAgeInEden": 10,
"disableCheckingConfigMismatch": false,
"displayLanguage": "def",
"enableChunkSplitterV2": false,
"disableWorkerForGeneratingChunks": false,
"processSmallFilesInUIThread": false,
"notifyThresholdOfRemoteStorageSize": 0,
"usePluginSyncV2": true,
"usePluginEtc": false,
"handleFilenameCaseSensitive": false,
"doNotUseFixedRevisionForChunks": true,
"showLongerLogInsideEditor": false,
"sendChunksBulk": false,
"sendChunksBulkMaxSize": 1,
"useSegmenter": false,
"useAdvancedMode": false,
"usePowerUserMode": false,
"useEdgeCaseMode": false,
"enableDebugTools": false,
"suppressNotifyHiddenFilesChange": false,
"syncMinimumInterval": 2000,
"P2P_Enabled": false,
"P2P_AutoAccepting": 0,
"P2P_AppID": "self-hosted-livesync",
"P2P_roomID": "",
"P2P_passphrase": "",
"P2P_relays": "wss://exp-relay.vrtmrz.net/",
"P2P_AutoBroadcast": false,
"P2P_AutoStart": false,
"P2P_AutoSyncPeers": "",
"P2P_AutoWatchPeers": "",
"P2P_SyncOnReplication": "",
"P2P_RebuildFrom": "",
"P2P_AutoAcceptingPeers": "",
"P2P_AutoDenyingPeers": "",
"P2P_IsHeadless": false,
"P2P_DevicePeerName": "",
"P2P_turnServers": "",
"P2P_turnUsername": "",
"P2P_turnCredential": "",
"P2P_useDiagRTC": false,
"doctorProcessedVersion": "0.25.27",
"bucketCustomHeaders": "",
"couchDB_CustomHeaders": "",
"useJWT": true,
"jwtAlgorithm": "ES512",
"jwtKey": "-----BEGIN PRIVATE KEY-----\nMIHuAgEAMBAGByqGSM49AgEGBSuBBAAjBIHWMIHTAgEBBEIBnKJOpWxPc1T5IoQN\nsWrMfNUg0vaZUUCsSRiEJKnlDcl17joI8LylqbHdNr1pmI+onIGQg+mOweGz7jCv\nGDBF4bqhgYkDgYYABAFrCDcCiOCE53P/qivBpG3UytUKm8GpYxx5slGMx0XsAy9P\nBzZe+GIDEE23NNPplQR7FbDTVFA9AM12sFwc/ylDpgClxxDWW3Z3yKnW7reOFI/5\niIzKxj+XcIvpiS29L/b4gJSgOS1rjeY5VGCMb53iWFK4poK4ZLCv821OH3bCAoUy\n8w==\n-----END PRIVATE KEY-----\n",
"jwtKid": "livesync-cc",
"jwtSub": "cc",
"jwtExpDuration": 5,
"useRequestAPI": false,
"bucketPrefix": "",
"chunkSplitterVersion": "v3-rabin-karp",
"E2EEAlgorithm": "v2",
"processSizeMismatchedFiles": false,
"forcePathStyle": true,
"syncInternalFileOverwritePatterns": "",
"useOnlyLocalChunk": false,
"maxMTimeForReflectEvents": 0,
"tweakModified": 1781558066465
}

18
vault/.obsidian/plugins/obsidian-livesync/main.js vendored
View file

File diff suppressed because one or more lines are too long

10
vault/.obsidian/plugins/obsidian-livesync/manifest.json vendored
View file

@ -1,10 +0,0 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.25.76",
"minAppVersion": "1.7.2",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",
"authorUrl": "https://github.com/vrtmrz",
"isDesktopOnly": false
}

573
vault/.obsidian/plugins/obsidian-livesync/styles.css vendored
View file

@ -1,573 +0,0 @@
.ls-dialog .added {
color: var(--text-on-accent);
background-color: var(--text-accent);
}
.ls-dialog .normal {
color: var(--text-normal);
}
.ls-dialog .deleted {
color: var(--text-on-accent);
background-color: var(--text-muted);
}
.conflict-dev-name {
display: inline-block;
min-width: 5em;
}
.conflict-action-button {
margin-right: 4px;
}
.op-scrollable {
overflow-y: scroll;
/* min-height: 280px; */
max-height: 280px;
user-select: text;
-webkit-user-select: text;
}
.op-pre {
white-space: pre-wrap;
}
.op-warn {
border: 1px solid salmon;
padding: 2px;
border-radius: 4px;
}
.op-warn::before {
content: "Warning";
font-weight: bold;
color: salmon;
position: relative;
display: block;
}
.op-warn-info {
border: 1px solid rgb(255, 209, 81);
padding: 2px;
border-radius: 4px;
}
.op-warn-info::before {
content: "Notice";
font-weight: bold;
color: rgb(255, 209, 81);
position: relative;
display: block;
}
.syncstatusbar {
-webkit-filter: grayscale(100%);
filter: grayscale(100%);
}
.tcenter {
text-align: center;
}
.sls-plugins-wrap {
display: flex;
flex-grow: 1;
max-height: 50vh;
overflow-y: scroll;
}
.sls-remote-list .setting-item-description {
white-space: normal;
overflow-wrap: anywhere;
word-break: break-word;
}
.sls-plugins-tbl {
border: 1px solid var(--background-modifier-border);
width: 100%;
max-height: 80%;
}
.divider th {
border-top: 1px solid var(--background-modifier-border);
}
.sls-header-button {
margin-left: 2em;
}
.sls-hidden {
display: none;
}
:root {
--sls-log-text: "";
}
.sls-troubleshoot-preview {
max-width: max-content;
}
.sls-troubleshoot-preview img {
max-width: 100%;
}
.sls-setting-tab {
display: none;
}
div.sls-setting-menu-btn {
color: var(--text-normal);
background-color: var(--background-secondary-alt);
border-radius: 8px;
padding: 6px 10px;
cursor: pointer;
margin-right: 2px;
font-family: "Inter", sans-serif;
outline: none;
user-select: none;
flex-grow: 1;
text-align: center;
flex-shrink: 1;
}
.sls-setting-label.selected {
/* order: 1; */
flex-grow: 1;
/* width: 100%; */
}
.sls-setting-tab:hover~div.sls-setting-menu-btn,
.sls-setting-label.selected .sls-setting-tab:checked~div.sls-setting-menu-btn {
background-color: var(--interactive-accent);
color: var(--text-on-accent);
}
.sls-setting-menu-wrapper {
display: flex;
flex-direction: column;
flex-grow: 1;
position: sticky;
top: 0;
backdrop-filter: blur(15px);
padding: 4px;
border-radius: 10px;
z-index: 10;
}
.sls-setting-menu {
display: flex;
flex-direction: row;
overflow-x: auto;
}
body {
--sls-col-transparent: transparent;
--sls-col-warn: rgba(var(--background-modifier-error-rgb), 0.1);
--sls-col-warn-stripe1: var(--sls-col-transparent);
--sls-col-warn-stripe2: var(--sls-col-warn);
}
.sls-setting-menu-buttons {
border: 1px solid var(--sls-col-warn);
/* padding: 2px; */
margin: 1px;
border-radius: 4px;
background-image: linear-gradient(-45deg,
var(--sls-col-warn-stripe1) 25%, var(--sls-col-warn-stripe2) 25%, var(--sls-col-warn-stripe2) 50%,
var(--sls-col-warn-stripe1) 50%, var(--sls-col-warn-stripe1) 75%, var(--sls-col-warn-stripe2) 75%, var(--sls-col-warn-stripe2));
background-size: 30px 30px;
display: flex;
flex-direction: row;
/* justify-content: flex-end; */
padding: 0.5em 0.25em;
justify-content: center;
align-items: center;
/* transition: background-position 1s; */
animation: sls-scroll-warn 1s linear 0s infinite;
}
@keyframes sls-scroll-warn {
0% {
background-position: 0 0;
}
100% {
background-position: 30px 0;
}
}
.sls-setting-menu-buttons label {
margin-right: auto;
flex-grow: 1;
color: var(--text-warning);
}
.sls-setting-label {
flex-grow: 1;
display: inline-flex;
justify-content: center;
}
.setting-collapsed {
display: none;
}
.sls-plugins-tbl-buttons {
text-align: right;
}
.sls-plugins-tbl-buttons button {
flex-grow: 0;
padding: 6px 10px;
}
.sls-plugins-tbl-device-head {
background-color: var(--background-secondary-alt);
color: var(--text-accent);
}
.op-flex {
display: flex;
}
.op-flex input {
display: inline-flex;
flex-grow: 1;
margin-bottom: 8px;
}
.op-info {
display: inline-flex;
flex-grow: 1;
border-bottom: 1px solid var(--background-modifier-border);
width: 100%;
margin-bottom: 4px;
padding-bottom: 4px;
}
.history-added {
color: var(--text-on-accent);
background-color: var(--text-accent);
}
.history-normal {
color: var(--text-normal);
}
.history-deleted {
color: var(--text-on-accent);
background-color: var(--text-muted);
text-decoration: line-through;
}
.ob-btn-config-fix label {
margin-right: 40px;
}
.ob-btn-config-info {
border: 1px solid salmon;
padding: 2px;
margin: 1px;
border-radius: 4px;
}
.ob-btn-config-head {
padding: 2px;
margin: 1px;
border-radius: 4px;
}
.isWizard .wizardHidden {
display: none;
}
.sls-setting:not(.isWizard) .wizardOnly {
display: none;
}
.sls-item-dirty::before {
content: "✏";
}
.sls-item-dirty-help::after {
content: " ❓";
}
.sls-setting .setting-item-control input.sls-item-invalid-value,
.sls-setting .setting-item-control textarea.sls-item-invalid-value {
background-color: rgba(var(--background-modifier-error-rgb), 0.3);
}
.sls-setting-disabled input[type=text],
.sls-setting-disabled input[type=number],
.sls-setting-disabled input[type=password] {
filter: brightness(80%);
color: var(--text-muted);
}
.sls-setting-hidden {
display: none;
}
.sls-setting-obsolete {
/* background-image: linear-gradient(-45deg,
var(--sls-col-warn-stripe1) 25%, var(--sls-col-warn-stripe2) 25%, var(--sls-col-warn-stripe2) 50%,
var(--sls-col-warn-stripe1) 50%, var(--sls-col-warn-stripe1) 75%, var(--sls-col-warn-stripe2) 75%, var(--sls-col-warn-stripe2)); */
background-image: linear-gradient(-45deg,
transparent 25%, rgba(var(--background-secondary), 0.1) 25%, rgba(var(--background-secondary), 0.1) 50%, transparent 50%, transparent 75%, rgba(var(--background-secondary), 0.1) 75%, rgba(var(--background-secondary), 0.1));
background-size: 60px 60px;
}
.password-input>.setting-item-control>input {
-webkit-text-security: disc;
}
span.ls-mark-cr::after {
user-select: none;
content: "↲";
color: var(--text-muted);
font-size: 0.8em;
}
.deleted span.ls-mark-cr::after {
color: var(--text-on-accent);
}
.ls-imgdiff-wrap {
display: flex;
justify-content: center;
align-items: center;
}
.ls-imgdiff-wrap .overlay {
position: relative;
}
.ls-imgdiff-wrap .overlay .img-base {
position: relative;
top: 0;
left: 0;
}
.ls-imgdiff-wrap .overlay .img-overlay {
-webkit-filter: invert(100%) opacity(50%);
filter: invert(100%) opacity(50%);
position: absolute;
top: 0;
left: 0;
animation: ls-blink-diff 0.5s cubic-bezier(0.4, 0, 1, 1) infinite alternate;
}
@keyframes ls-blink-diff {
0% {
opacity: 0;
}
50% {
opacity: 0;
}
100% {
opacity: 1;
}
}
.livesync-status {
user-select: none;
pointer-events: none;
height: auto;
min-height: 1em;
position: absolute;
background-color: transparent;
width: 100%;
padding: 10px;
padding-right: 16px;
top: var(--header-height);
z-index: calc(var(--layer-cover) + 1);
font-variant-numeric: tabular-nums;
tab-size: 4;
text-align: right;
white-space: pre-wrap;
display: inline-block;
color: var(--text-normal);
font-size: 80%;
}
div.workspace-leaf-content[data-type=bases] .livesync-status {
top: calc(var(--bases-header-height) + var(--header-height));
padding: 5px;
padding-right: 18px;
}
.is-mobile div.workspace-leaf-content[data-type=bases] .livesync-status {
top: calc(var(--bases-header-height) + var(--view-header-height));
padding: 6px;
padding-right: 18px;
}
.livesync-status div {
opacity: 0.6;
-webkit-filter: grayscale(100%);
filter: grayscale(100%);
}
.livesync-status .livesync-status-loghistory {
text-align: left;
opacity: 0.4;
}
.livesync-status div.livesync-status-messagearea:empty {
display: none;
}
.livesync-status div.livesync-status-messagearea:not(:empty) {
opacity: 0.6;
color: var(--text-on-accent);
border: 1px solid var(--background-modifier-error);
background-color: rgba(var(--background-modifier-error-rgb), 0.2);
-webkit-filter: unset;
filter: unset;
width: fit-content;
margin-left: auto;
}
.menu-setting-poweruser-disabled .sls-setting-poweruser {
display: none;
}
.menu-setting-advanced-disabled .sls-setting-advanced {
display: none;
}
.menu-setting-edgecase-disabled .sls-setting-edgecase {
display: none;
}
.sls-setting-panel-title {
position: sticky;
font-size: medium;
top: 2.5em;
background-color: var(--background-secondary-alt);
border-radius: 10px;
padding: 0.5em 1.0em;
}
.active-pane .sls-setting-panel-title {
border: 1px solid var(--interactive-accent);
}
.sls-dialogue-note-wrapper {
display: flex;
justify-content: flex-end;
align-items: center;
}
.sls-dialogue-note-countdown {
font-size: 0.8em;
}
.sls-qr {
display: flex;
justify-content: center;
align-items: center;
max-width: max-content;
}
.sls-keypair pre {
max-width: 100%;
overflow-x: auto;
white-space: pre-wrap;
word-break: break-all;
}
/* Diff navigation */
.diff-options-row {
display: flex;
align-items: center;
gap: 8px;
}
.diff-nav {
display: flex;
align-items: center;
gap: 4px;
margin-left: auto;
}
.diff-nav-btn {
padding: 2px 8px;
font-size: 0.85em;
cursor: pointer;
border: 1px solid var(--background-modifier-border);
border-radius: 4px;
background-color: var(--background-secondary);
color: var(--text-normal);
}
.diff-nav-btn:hover {
background-color: var(--background-modifier-hover);
}
.diff-nav-indicator {
font-size: 0.85em;
color: var(--text-muted);
min-width: 3em;
text-align: center;
}
.diff-only-label {
margin-left: 10px;
}
.history-search-row {
display: flex;
gap: 5px;
align-items: center;
margin-bottom: 10px;
}
.history-search-input {
flex-grow: 1;
}
.history-search-result-indicator {
font-size: 0.8em;
min-width: 80px;
}
.history-search-progress-indicator {
font-size: 0.8em;
color: var(--text-muted);
}
.history-diff-options-row {
justify-content: space-between;
}
.history-highlight-diff-container,
.history-highlight-diff-label {
display: flex;
align-items: center;
}
.history-highlight-diff-label {
gap: 4px;
}
.diff-focused {
outline: 2px solid var(--interactive-accent);
outline-offset: 1px;
border-radius: 2px;
}

244
vault/.obsidian/workspace.json vendored
View file

@ -1,244 +0,0 @@
{
"main": {
"id": "b8aa267a87465670",
"type": "split",
"children": [
{
"id": "aad0932c415f7376",
"type": "tabs",
"children": [
{
"id": "8d53cdb6c257e685",
"type": "leaf",
"state": {
"type": "markdown",
"state": {
"file": "CLAUDE-baseline.md",
"mode": "source",
"source": false
},
"icon": "lucide-file",
"title": "CLAUDE-baseline"
}
}
]
}
],
"direction": "vertical"
},
"left": {
"id": "42d5ddfb7fe9a049",
"type": "split",
"children": [
{
"id": "7937990dfe4d965d",
"type": "tabs",
"children": [
{
"id": "17bd4a6166f789d0",
"type": "leaf",
"state": {
"type": "file-explorer",
"state": {
"sortOrder": "alphabetical",
"autoReveal": false
},
"icon": "lucide-folder-closed",
"title": "Files"
}
},
{
"id": "441a917372004297",
"type": "leaf",
"state": {
"type": "search",
"state": {
"query": "tag:#mesh",
"matchingCase": false,
"explainSearch": false,
"collapseAll": false,
"extraContext": false,
"sortOrder": "alphabetical"
},
"icon": "lucide-search",
"title": "Search"
}
},
{
"id": "453741936124e75e",
"type": "leaf",
"state": {
"type": "bookmarks",
"state": {},
"icon": "lucide-bookmark",
"title": "Bookmarks"
}
}
]
}
],
"direction": "horizontal",
"width": 300
},
"right": {
"id": "dcd69063b85009cd",
"type": "split",
"children": [
{
"id": "968e61823e9f9178",
"type": "tabs",
"children": [
{
"id": "f7094e1ef908deef",
"type": "leaf",
"state": {
"type": "backlink",
"state": {
"file": "CLAUDE-baseline.md",
"collapseAll": false,
"extraContext": false,
"sortOrder": "alphabetical",
"showSearch": false,
"searchQuery": "",
"backlinkCollapsed": false,
"unlinkedCollapsed": true
},
"icon": "links-coming-in",
"title": "Backlinks for CLAUDE-baseline"
}
},
{
"id": "a3a979c9ebdbf706",
"type": "leaf",
"state": {
"type": "outgoing-link",
"state": {
"file": "CLAUDE-baseline.md",
"linksCollapsed": false,
"unlinkedCollapsed": true
},
"icon": "links-going-out",
"title": "Outgoing links from CLAUDE-baseline"
}
},
{
"id": "ac364768c88597ca",
"type": "leaf",
"state": {
"type": "tag",
"state": {
"sortOrder": "frequency",
"useHierarchy": true,
"showSearch": false,
"searchQuery": ""
},
"icon": "lucide-tags",
"title": "Tags"
}
},
{
"id": "c48100f1970c5d20",
"type": "leaf",
"state": {
"type": "all-properties",
"state": {
"sortOrder": "frequency",
"showSearch": false,
"searchQuery": ""
},
"icon": "lucide-archive",
"title": "All properties"
}
},
{
"id": "33a7ddb1e675ef2b",
"type": "leaf",
"state": {
"type": "outline",
"state": {
"file": "CLAUDE-baseline.md",
"followCursor": false,
"showSearch": false,
"searchQuery": ""
},
"icon": "lucide-list",
"title": "Outline of CLAUDE-baseline"
}
},
{
"id": "c3fdcf2b3358f125",
"type": "leaf",
"state": {
"type": "p2p-server-status",
"state": {},
"icon": "waypoints",
"title": "P2P Status"
}
}
],
"currentTab": 5
}
],
"direction": "horizontal",
"width": 300
},
"left-ribbon": {
"hiddenItems": {
"obsidian-livesync:P2P Status": false,
"obsidian-livesync:Replicate": false,
"obsidian-livesync:Show Log": false,
"switcher:Open quick switcher": false,
"graph:Open graph view": false,
"canvas:Create new canvas": false,
"daily-notes:Open today's daily note": false,
"templates:Insert template": false,
"command-palette:Open command palette": false,
"bases:Create new base": false,
"obsidian-livesync:Show Customization sync": false
}
},
"active": "17bd4a6166f789d0",
"lastOpenFiles": [
"docs/software/dns.md.tmp.3790074.04f69e0a9ea6",
"projects/meshtastic-headscale-runbook.md.tmp.3790074.f71d216eecca",
"projects/meshtastic-headscale-runbook.md.tmp.3790074.d94af273bec3",
"runbooks/expose-service-contabo.md.tmp.3790074.70782eb4562c",
"runbooks/expose-service-contabo.md.tmp.3790074.fe105eeb05b6",
"runbooks/expose-service-edge2.md.tmp.3790074.484208d24d9d",
"runbooks/expose-service-edge2.md.tmp.3790074.b2d7cda931c0",
"runbooks/expose-service-home.md.tmp.3790074.a7aea2f94658",
"runbooks/expose-service-home.md.tmp.3790074.b26454819896",
"docs/software/dns.md.tmp.3790074.fcdcf768c300",
"docs/software/dns.md.tmp.3790074.404f85bc78a4",
"projects/meshwars.md",
"runbooks/navi-lift-to-media.md",
"runbooks/edge2-boot-recovery.md",
"runbooks/corescope-ingest-stall-oom.md",
"projects/navi-recon-separation.md",
"projects/fleet-storage-memory-upgrade.md",
"runbooks/peertube-sitemap-redis-oom.md",
"runbooks/omv-add-nfs-share.md",
"runbooks/pve-guest-park-and-adopt.md",
"runbooks/conduit-operations.md",
"docs/software/conduit.md",
"archive/projects/vaultwarden-plan.md",
"archive/projects/meshai-native-fire-severity-audit-cc-handoff.md",
"projects/meshai-native-fire-severity-audit-cc-handoff.md",
"projects/meshai-config-hot-apply.md",
"projects/meshai-region-routing-plan.md",
"runbooks/meshai-prod-compose-override.md",
"projects/meshai.md",
"projects/meshcore-transport.md",
"runbooks/pymc-repeater-kiss-tnc-reenumeration.md",
"runbooks/headscale-oidc-boot-order.md",
"runbooks/headless-browser-page-verification.md",
"runbooks/central-deploy-cutover.md",
"runbooks/fleet-magicdns-resolved-migration.md",
"projects/fleet-platform-baseline.md",
"Untitled.canvas",
"assets/echo6yellow_logo_422x422_square.png",
"assets/echo6yellow_logo_422x81.png",
"assets/echo6_logo.png",
"assets/echo6_favicon.png"
]
}

8
vault/.trash/2026-06-19.md
View file

@ -1,8 +0,0 @@
---
title: 2026 06 19
type: reference
tags:
- mesh
aliases: []
updated: 2026-07-13
---

0
vault/.trash/Untitled.canvas
View file

74
vault/CLAUDE-baseline.md
View file

@ -1,74 +0,0 @@
---
title: CLAUDE baseline (global rules)
type: reference
tags: [proxmox]
status: auto-generated
updated: 2026-07-11
---
> [!info] Auto-generated mirror of `~/.claude/CLAUDE.md` on cortex. Edit the source, not this file — it refreshes automatically.
# Echo6 Infrastructure — Claude Guidelines
## Locale
Timezone: America/Boise (Mountain Time)
---
## Working model — how we operate
**Matt guides → Opus orchestrates → Sonnet executes.**
- **Matt guides.** Sets the goal and makes the decisions.
- **Opus orchestrates — and *only* orchestrates.** Plans the work, breaks it into tight surgical tasks, dispatches **Sonnet** subagents to do them, reviews their output, and reports back. Opus does **not** do hands-on work itself — no editing files, running commands, or deploying directly. It plans, dispatches, verifies.
- **Sonnet executes — and *only* the prompt.** Runs one tightly-scoped task exactly as written. No scope creep, no initiative beyond the prompt. If the task is ambiguous or needs a decision → stop and report back to Opus; never guess.
Flow: **Matt → Opus plans → dispatches Sonnet (tight prompt) → Sonnet executes → Opus reviews → reports to Matt.**
---
## Critical policies — always apply
- **Gemini:** `gemini-3.1-flash-lite` only, every call. No exceptions. (Was `gemini-2.5-flash-lite` until 2026-07-11, when Google retired all Gemini-2.x lite models for our API key — they 404 "no longer available to new users." `gemini-3.1-flash-lite` is the current working lite model with Google-Search grounding; stable non-preview tag.)
- **Host protection:** never `shutdown`/`reboot`/`poweroff` any host; never include cortex (primary Claude Code host) or TOC in availability-affecting bulk operations; never install packages on any host (pip/npm/apt) without explicit permission.
- **No changes without approval:** never deploy, change service ports, or change network/firewall config without explicit approval. If a target is unreachable or blocked → **STOP and report**; never redirect to an alternate host.
- **Resilience:** every deployment must survive a reboot.
- **Credentials:** source from `.ref/credentials`; never commit secrets to a git-tracked file — *except* the private `echo6-docs` Forge repo (the one documented exception).
- **Git:** GitHub `origin` is the source of truth and push target — *except* `echo6-docs`, which lives on Forge directly. Branch off the default branch before committing. Commit/push only when asked.
- **When unsure → ASK.** Never assume, never improvise.
---
## Infra cheat-sheet
| Host | Local IP | Tailscale | Role |
|------|----------|-----------|------|
| data | 192.168.1.240 | 100.64.0.6 | databases |
| utility | 192.168.1.241 | 100.64.0.5 | utility / monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | cloud / personal |
| media | 192.168.1.243 | 100.64.0.3 | media / *arr |
| toc | 192.168.1.244 | 100.64.0.13 | GPU host (passthrough → cortex) |
| **cortex** (VM 150) | 192.168.1.150 | 100.64.0.14 | GPU compute, **Claude Code**, AI |
| recon-vm (VM 1130) | 192.168.1.130 | 100.64.0.24 | recon pipeline |
| **edge1** (Contabo, rebuilt) | 5.189.158.149 | 100.64.0.40 | Proxmox edge node (PVE 8, LXC-only) — **Mail only** (Mailcow in CT 101 → 10.10.10.2) |
| **edge2** (Contabo) | 184.174.35.153 | 100.64.0.26 | Proxmox edge node (PVE 8, LXC-only) — **front door** for Auth, Forge, Notes, Matrix, Element, VPN, Vault; also hosts **PDM** (CT 100 → 100.64.0.28:8443) |
| pi-nas | 192.168.1.245 | 100.64.0.21 | NAS |
- **SSH:** `ssh zvx@<ip>` (key auth) for most; `root@<ip>` for Proxmox hosts + edge1/edge2. Password-auth exceptions (aida-nebra, mt-isr, toc, matt-desktop) → see `environment.md`.
- **dns targets:** mail/autodiscover/autoconfig → **edge1** `5.189.158.149`; auth, forge, notes, vpn, vault, matrix, element → **edge2** `184.174.35.153`; home services (echo6.co, ai, jellyfin, immich, nextcloud, recon, stream) → `199.6.36.163` (via utility caddy).
---
## Where the detail lives — load when needed
- **Docs vault** → `.ref/vault/` (Obsidian docs library; category-tagged, maintained by `.ref/engine/` — see `.ref/CLAUDE.md`)
- **Procedures / runbooks** → `.ref/vault/runbooks/`
- **Per-project context** → `.ref/vault/projects/<name>.md`
- **System conventions** (path-scoped) → `~/.claude/rules/`
- **Credentials** → `.ref/credentials`
- **Editing vault docs:** category tags + `[[links]]` to *existing* docs only; no entity/concept pages, no INDEX. Engine handles it. Details → `.ref/CLAUDE.md`.
---
## Default behavior
When unsure → **ASK**. Default to internal access. Document everything. Never assume; never improvise.

408
vault/archive/AUDIT-2026-02-21.md
View file

@ -1,408 +0,0 @@
# .ref/ Directory Audit — 2026-02-21
**Auditor:** Claude Code (cortex)
**Hostname:** cortex
**Scope:** Every file in `/home/zvx/projects/.ref/` cross-referenced against live infrastructure state.
---
## 1. Dead Services to Remove
### Wiki.js / docs.echo6.co — CONFIRMED DEAD
- **Evidence:** No Docker container running on Contabo. `docker ps -a` shows no wiki-related containers.
- **Stale artifacts:**
- `credentials` lines 20-23: `WIKIJS_ADMIN_EMAIL`, `WIKIJS_ADMIN_PASSWORD`, `WIKIJS_URL`
- `credentials` line 35: `WIKIJS_API_TOKEN` (JWT, expires 2026-11-07)
- GoDaddy DNS: `docs.echo6.co` still resolves to `199.6.36.163`
- dnsmasq: `docs.echo6.co` still has an entry pointing to `100.64.0.1`
- `docs/software/caddy.md`: Still referenced under Contabo Caddy (not present in actual Caddyfile)
- `docs/software/dns.md`: Still lists `docs.echo6.co` under home services
- **Action:** Remove credentials, delete DNS record, remove dnsmasq entry, remove from caddy.md and dns.md.
### Echo6 Portal — CONFIRMED DEAD
- **Evidence:** Nothing listening on port 3002 on utility (192.168.1.241). No container found.
- **Stale artifacts:**
- `credentials` lines 60-69: `ECHO6_PORTAL_*` credentials (admin user, OIDC client ID/secret, issuer URL)
- `credentials` line 61: References deployment at `192.168.1.241:3002`
- **Action:** Remove credentials. Verify Authentik provider `echo6-portal` doesn't still exist (clean up if so).
### Stalwart Mail — Already Decommissioned (Partially Cleaned)
- **Status:** Credentials correctly commented out in credentials file (lines 71-75).
- **credentials** line 72: Comment says "Authentik provider PK 53 may still exist" — **Verified: PK 53 does NOT exist in Authentik.** The comment is now misleading.
- **Action:** Update comment to state PK 53 was already cleaned, or remove the entire stale block.
---
## 2. Stale Credentials to Clean
### OPENWEBUI_API_URL — Wrong IP
- **File:** `credentials` line 52
- **Current value:** `OPENWEBUI_API_URL=http://192.168.1.239:3000`
- **Problem:** No known service at `192.168.1.239`. Open WebUI runs on cortex at `192.168.1.150:8080`.
- **Action:** Update to `http://192.168.1.150:8080` or `http://100.64.0.14:8080` (Tailscale IP).
### MESHMONITOR_TAILSCALE_URL — Wrong IP
- **File:** `credentials` lines 124, 127
- **Current value:** `MESHMONITOR_TAILSCALE_URL="http://100.64.0.1:8080"`
- **Problem:** `100.64.0.1` is Contabo, not MeshMonitor. MeshMonitor runs on utility CT 100, Tailscale IP `100.64.0.7`.
- **Action:** Change to `http://100.64.0.7:8080`.
### Contabo Tailscale IP in credentials — Wrong
- **File:** `credentials` line 86
- **Current value:** `# Tailscale IP: 100.64.0.6`
- **Problem:** `100.64.0.6` is the data Proxmox node's Tailscale IP. Contabo's Tailscale IP is `100.64.0.1`.
- **Action:** Change comment to `# Tailscale IP: 100.64.0.1`.
### Stale File Path Reference
- **File:** `credentials` line 33
- **Current value:** `# - File Location: /home/zvx/projects/selfhosting/.credentials`
- **Problem:** References an old path that no longer exists. Current location is `/home/zvx/projects/.ref/credentials`.
- **Action:** Update to current path or remove the comment.
---
## 3. Incorrect Values
### Caddy Port: mail.echo6.co
- **File:** `docs/software/caddy.md` — Contabo Caddy table
- **Documented:** `reverse_proxy https://127.0.0.1:8443` (port 8443)
- **Actual (live Caddyfile):** `reverse_proxy https://127.0.0.1:8453` (port 8453)
- **Also wrong in:** Port Map table at bottom of `caddy.md` (shows 8443)
- **Action:** Update both references from 8443 to 8453.
### Contabo Tailscale IP — Multiple Files Reference 100.64.0.6 Instead of 100.64.0.1
| File | Line/Section | Wrong Value | Correct Value |
|------|-------------|-------------|---------------|
| `credentials` | Line 86 | `100.64.0.6` | `100.64.0.1` |
| `projects/vaultwarden-deployment.md` | SSH commands | `100.64.0.6` | `100.64.0.1` |
| `projects/authentik-oidc-application.md` (if exists as runbook) | SSH commands | `100.64.0.6` | `100.64.0.1` |
| `runbooks/authentik-oidc-application.md` | SSH commands to Contabo | `100.64.0.6` | `100.64.0.1` |
| `runbooks/expose-service-contabo.md` | Step 5 verification dig command | `100.64.0.6` | `100.64.0.1` |
**Root cause:** 100.64.0.6 is the **data** Proxmox node. This was likely a copy-paste error from early infrastructure setup when Contabo's Tailscale IP may not have been settled.
### Headscale Container Name — "headscale-vanilla" vs "headscale"
| File | Incorrect | Correct |
|------|-----------|---------|
| `projects/headscale-full-deployment.md` | `headscale-vanilla` (throughout) | `headscale` |
| `runbooks/proxmox-create-ubuntu-vm.md` | `headscale-standby` (lines 245, 257, 277) | `headscale` |
**Note:** MEMORY.md already documents this: "Headscale container on Contabo is named `headscale` (not `headscale-vanilla` as in some runbooks)." The runbooks were never updated to match.
### Utility Caddy Tailscale IP
- **File:** `projects/utility-caddy-initial-setup.md`
- **Documented:** Tailscale IP `100.64.0.2`
- **Actual:** `100.64.0.8` (per environment.md and live headscale node list)
- **Action:** Update to `100.64.0.8`.
### Proxmox Onboard Node — Wrong Tailscale IPs for Cluster Nodes
- **File:** `runbooks/proxmox-onboard-node.md` — "Current Cluster" table (lines 9-14)
- **Documented:**
| Node | Tailscale IP (documented) | Tailscale IP (actual from environment.md) |
|------|--------------------------|------------------------------------------|
| data | 100.64.0.20 | 100.64.0.6 |
| utility | 100.64.0.19 | 100.64.0.5 |
| cloud | 100.64.0.22 | 100.64.0.4 |
| media | 100.64.0.21 | 100.64.0.3 |
**Every single Tailscale IP in this table is wrong.** These appear to be from a completely different assignment scheme. Additionally, the `toc` node (192.168.1.244 / 100.64.0.13) is missing from this table entirely.
### PeerTube Rebuild — Wrong CT ID in Architecture Diagram
- **File:** `projects/peertube-rebuild.md`
- **Phase 1 architecture diagram** references CT 100 for PeerTube. Actual is CT 110.
- **Note:** The checklist section correctly references CT 110. Only the diagram is wrong.
### Matrix Deployment — Stale Host References
- **File:** `projects/matrix-synapse-deployment.md`
- **Problem:** Still contains original CT 108 deployment instructions (utility node). Matrix was migrated to Contabo on 2026-02-15.
- **dnsmasq section** says point `matrix.echo6.co` to `100.64.0.8` (utility Caddy). Actual dnsmasq points to `100.64.0.1` (Contabo).
- **Action:** Add a prominent note at the top indicating this was migrated, or rewrite for Contabo deployment.
### meshtastic-headscale-runbook.md — Wrong Debian Template Version
- **File:** `projects/meshtastic-headscale-runbook.md`
- **References:** `debian-12-standard_12.7-1_amd64.tar.zst`
- **Actual (per MEMORY.md):** `debian-12-standard_12.12-1_amd64.tar.zst`
### openwebui-theme-deploy.md — Wrong Home Directory
- **File:** `projects/openwebui-theme-deploy.md`
- **References:** `/home/matt/` paths in some instructions
- **Actual:** cortex user is `zvx`, home directory is `/home/zvx/`
### arr-wiring-runbook.md — Wrong File Paths
- **File:** `projects/arr-wiring-runbook.md`
- **References:** `./ref/services/usenet.md` (lines 24, 51, 69)
- **Should be:** `.ref/docs/services/usenet.md`
### ct-runbook.md — References stock Tailscale auth pattern
- **File:** `runbooks/ct-runbook.md`
- **Problem:** References `tskey-auth-*` (stock Tailscale preauth key format) instead of Headscale registration pattern. Echo6 uses self-hosted Headscale where keys are generated via `headscale preauthkeys create`.
### WATCHTOWER Deployment — Wrong Caddy Pattern
- **File:** `projects/cc-deploy-watchtower-v2.md`
- **References:** Docker-based Caddy with `~/docker/caddy/sites/` site file pattern
- **Actual:** Utility Caddy is systemd-based on CT 101 with `/etc/caddy/Caddyfile` (single file, not per-site directory)
---
## 4. Duplicates / Contradictions
### AUTHENTIK_API_TOKEN — Defined Twice
- **File:** `credentials` line 14 and line 111
- **Values:** Both contain the same token value: `YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w`
- **Action:** Remove one instance. Keep the one at line 111 (has regeneration date context), remove line 14's duplicate.
### FORGEJO_URL — Defined Twice
- **File:** `credentials` line 27 (`FORGEJO_URL=https://forge.echo6.co`) and line 100 (`FORGEJO_URL="https://forge.echo6.co"`)
- **Action:** Remove the earlier instance (line 27). The line 100 block has more complete Forgejo credentials.
### dnsmasq: notes.echo6.co — Duplicate Entries with Different IPs
- **Actual dnsmasq config** has TWO entries for `notes.echo6.co`:
- `address=/notes.echo6.co/100.64.0.22` (mesh-bridge CT 107 — incorrect target)
- `address=/notes.echo6.co/100.64.0.1` (Contabo — correct, LiveSync runs here)
- **Action:** Remove the `100.64.0.22` entry. Only the `100.64.0.1` entry should remain.
### authentik-access-groups.md vs authentik.md — Conflicting Group Members
- `authentik.md` lists `media-users` members as: **jodie, matt**
- `authentik-access-groups.md` lists `media-users` members as: **jodie** (no matt)
- **Action:** Verify live state via Authentik API and update both docs to match. Matt may be in `authentik Admins` (superuser) which bypasses group checks, making explicit media-users membership unnecessary.
### authentik.md vs authentik-access-groups.md — Missing Groups
- `authentik.md` lists `ai-users` group (PK `0631b273-...`) with member matt
- `authentik-access-groups.md` does NOT list `ai-users` at all
- **Action:** Add `ai-users` to access groups runbook reference table.
### authentik.md vs authentik-access-groups.md — Missing Application Bindings
The access groups runbook is missing bindings for recently added applications:
- Open WebUI (PK 14) — should be bound to `ai-users`
- Matrix (PK 15) — should be bound to `communication-users`
- TAK Server — not listed anywhere
- **Action:** Update the Quick Reference tables in both documents.
---
## 5. Missing Documentation
### TAK Server (tak.echo6.co) — Running, Undocumented
- **Status:** Deployed 2026-02-20 on Contabo. Docker containers running (`tak-server-deploy`, `sigil`). Credentials exist in `credentials` file (lines 236-249). Caddy site blocks exist on Contabo.
- **Missing from:**
- `docs/services/services.md` — no TAK Server entry
- `docs/software/caddy.md` — no tak.echo6.co site block documented
- `docs/software/authentik.md` — no TAK Server provider listed
- `docs/hardware/environment.md` — Contabo services list doesn't mention TAK
- **Action:** Create `docs/software/tak.md` or add TAK Server entries to existing docs.
### Obsidian LiveSync (notes.echo6.co) — Running, Partially Documented
- **Status:** Docker containers running on Contabo (`livesync-couchdb`, `livesync-provisioner`). Credentials exist (lines 37-43). Caddy site block exists. `projects/deploy livesync.md` exists as research doc.
- **Missing from:**
- `docs/services/services.md` — no LiveSync entry
- `docs/software/caddy.md` — no notes.echo6.co Contabo site block documented
- **Action:** Add to services.md and caddy.md.
### RECON LXC (CT 130) — Running, Missing from Infrastructure Docs
- **Status:** `docs/software/recon.md` exists and is thorough. But CT 130 is missing from:
- `docs/hardware/environment.md` — not in LXC Containers table
- `docs/services/services.md` — not listed
- **Action:** Add `| recon | data (CT 130) | 192.168.1.130 | 100.64.0.24 | RECON knowledge extraction pipeline |` to both tables.
### files.echo6.co — Referenced, Not Documented
- **Status:** Referenced in dnsmasq config, RECON docs, and landing page data export. Presumably an nginx file server on RECON (CT 130).
- **Missing from:**
- `docs/services/services.md` — no entry
- `docs/software/caddy.md` — no dnsmasq entry documented
- **Action:** Add to services.md and caddy.md dnsmasq section.
### Undocumented Headscale Nodes
The headscale node list in `docs/hardware/environment.md` is missing several entries seen in live `headscale nodes list`:
| Node | Tailscale IP | Status | Notes |
|------|-------------|--------|-------|
| recon | 100.64.0.24 | Online | CT 130, documented in recon.md but not in environment.md |
| localhost | 100.64.0.12 | Last seen varies | Unknown purpose — possibly a test/dev registration |
| invalid-nwr32bou | 100.64.0.16 | Last seen varies | Unknown — possibly a stale/orphaned node registration |
- **Also:** `meshmon-node` appears as hostname "advbbs" in headscale — may have been renamed/repurposed.
- **Action:** Audit headscale node list, remove orphaned registrations, update environment.md.
### Undocumented Contabo Docker Containers
Live `docker ps` on Contabo shows containers not documented anywhere:
- `sigil` — TAK Server companion (web console), partially covered by TAK credentials
- `termix` — Unknown purpose, not documented
**Action:** Identify what `termix` is and document both.
### Usenet Credentials Not in Credentials File
- **File:** `docs/services/usenet.md` references API keys and passwords with "see .ref/credentials"
- **Problem:** No SABnzbd, Sonarr, Radarr, or Prowlarr API keys exist in the credentials file
- **Action:** Either add the API keys to the credentials file or remove the reference in usenet.md.
---
## 6. Recommendations
### Priority 1 — Fix Immediately (Data Integrity / Operational Risk)
1. **Remove dead Wiki.js credentials** from credentials file (lines 20-23, 35). These contain valid API tokens for a service that no longer exists.
2. **Fix MESHMONITOR_TAILSCALE_URL** (line 127) — currently points to Contabo instead of MeshMonitor. Any automation using this URL will target the wrong host.
3. **Fix OPENWEBUI_API_URL** (line 52) — points to a non-existent IP. Any automation using this will fail silently.
4. **Fix dnsmasq duplicate** for notes.echo6.co — the incorrect `100.64.0.22` entry could cause intermittent routing failures depending on which entry dnsmasq uses.
5. **Fix proxmox-onboard-node.md Tailscale IPs** — every IP in the cluster table is wrong. Anyone following this runbook will get incorrect SSH aliases.
### Priority 2 — Documentation Accuracy
6. **Update Contabo Tailscale IP** from `100.64.0.6` to `100.64.0.1` across all affected files (5+ files).
7. **Update Headscale container name** from `headscale-vanilla`/`headscale-standby` to `headscale` in deployment runbooks.
8. **Fix caddy.md mail port** from 8443 to 8453.
9. **Add TAK Server** to services.md, caddy.md, and authentik.md.
10. **Add RECON CT 130** to environment.md LXC table and services.md.
11. **Add LiveSync/notes.echo6.co** to services.md and caddy.md.
12. **Remove stale AUTHENTIK_API_TOKEN duplicate** and FORGEJO_URL duplicate from credentials.
### Priority 3 — Cleanup
13. **Remove Echo6 Portal credentials** (lines 60-69) — service is dead.
14. **Clean up Stalwart Mail comment** (line 72) — remove misleading "PK 53 may still exist" text.
15. **Delete GoDaddy DNS record** for docs.echo6.co — service no longer exists.
16. **Remove dnsmasq entry** for docs.echo6.co.
17. **Update stale file path reference** on credentials line 33.
18. **Audit headscale node list** — clean up orphaned nodes (`localhost`, `invalid-nwr32bou`), investigate `meshmon-node`/`advbbs` rename, update environment.md.
19. **Investigate `termix` container** on Contabo — document or remove.
20. **Update authentik-access-groups.md** — add `ai-users` group, add Open WebUI/Matrix/TAK bindings to reference tables.
---
## Files Audited
### Credentials
- [x] `/home/zvx/projects/.ref/credentials`
### Documentation
- [x] `/home/zvx/projects/.ref/docs/hardware/environment.md`
- [x] `/home/zvx/projects/.ref/docs/services/services.md`
- [x] `/home/zvx/projects/.ref/docs/services/usenet.md`
- [x] `/home/zvx/projects/.ref/docs/software/authentik.md`
- [x] `/home/zvx/projects/.ref/docs/software/caddy.md`
- [x] `/home/zvx/projects/.ref/docs/software/dns.md`
- [x] `/home/zvx/projects/.ref/docs/software/recon.md`
- [x] `/home/zvx/projects/.ref/docs/software/searxng.md`
### Project Files
- [x] `/home/zvx/projects/.ref/projects/DEPLOY-API-KEYS-TAB.md`
- [x] `/home/zvx/projects/.ref/projects/advbbs-project.md`
- [x] `/home/zvx/projects/.ref/projects/arr-stack-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/arr-wiring-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/cc-deploy-watchtower-v2.md`
- [x] `/home/zvx/projects/.ref/projects/deploy livesync.md`
- [x] `/home/zvx/projects/.ref/projects/headscale-full-deployment.md`
- [x] `/home/zvx/projects/.ref/projects/matrix-synapse-deployment.md`
- [x] `/home/zvx/projects/.ref/projects/meshtastic-headscale-runbook.md`
- [x] `/home/zvx/projects/.ref/projects/openwebui-theme-deploy.md`
- [x] `/home/zvx/projects/.ref/projects/peertube-phase2-project.md`
- [x] `/home/zvx/projects/.ref/projects/peertube-rebuild.md`
- [x] `/home/zvx/projects/.ref/projects/utility-caddy-initial-setup.md`
- [x] `/home/zvx/projects/.ref/projects/vaultwarden-deployment.md`
### Runbooks
- [x] `/home/zvx/projects/.ref/runbooks/add-peertube-channel.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-access-groups.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-create-invitation.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-oidc-application.md`
- [x] `/home/zvx/projects/.ref/runbooks/authentik-upgrade.md`
- [x] `/home/zvx/projects/.ref/runbooks/binary-wrapper-interception.md`
- [x] `/home/zvx/projects/.ref/runbooks/ct-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/expose-service-contabo.md`
- [x] `/home/zvx/projects/.ref/runbooks/expose-service-home.md`
- [x] `/home/zvx/projects/.ref/runbooks/gpu-cpu-fallback-routing.md`
- [x] `/home/zvx/projects/.ref/runbooks/ia-cli-reference.md`
- [x] `/home/zvx/projects/.ref/runbooks/ia-download-mirror.md`
- [x] `/home/zvx/projects/.ref/runbooks/idahomesh-bridge-setup.md`
- [x] `/home/zvx/projects/.ref/runbooks/idahomesh-vpn-device-setup.md`
- [x] `/home/zvx/projects/.ref/runbooks/mailcow-create-mailbox.md`
- [x] `/home/zvx/projects/.ref/runbooks/meshmonitor-password-reset.md`
- [x] `/home/zvx/projects/.ref/runbooks/meshtasticd-sim-nodes-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/nordvpn-lxc.md`
- [x] `/home/zvx/projects/.ref/runbooks/peertube-remote-runner.md`
- [x] `/home/zvx/projects/.ref/runbooks/pg-backup.md`
- [x] `/home/zvx/projects/.ref/runbooks/pi-nas-omv-runbook.md`
- [x] `/home/zvx/projects/.ref/runbooks/pipeline-probe-gate.md`
- [x] `/home/zvx/projects/.ref/runbooks/proxmox-create-ubuntu-vm.md`
- [x] `/home/zvx/projects/.ref/runbooks/proxmox-onboard-node.md`
- [x] `/home/zvx/projects/.ref/runbooks/recon-operations.md`
- [x] `/home/zvx/projects/.ref/runbooks/recon-service-integration.md`
### Misc Files
- [x] `/home/zvx/projects/.ref/echo6-landing-page-data-export.md`
- [x] `/home/zvx/projects/.ref/ia-download-queue.md`
### Not Audited (Low-Risk Assets)
- `/home/zvx/projects/.ref/pp_comparison.json` — data file, no infrastructure references
- `/home/zvx/projects/.ref/.gitignore` — git config
- `/home/zvx/projects/.ref/assets/` — static assets (CSS, JS, images, key_manager.py)
---
## Summary Statistics
| Category | Count |
|----------|-------|
| Dead services identified | 2 (Wiki.js, Echo6 Portal) + 1 already decommissioned (Stalwart) |
| Stale credentials to clean | 4 entries (Wiki.js x3, Echo6 Portal x5, wrong URLs x2, stale path x1) |
| Incorrect values found | 15+ across 10+ files |
| Duplicate entries | 4 (AUTHENTIK_API_TOKEN, FORGEJO_URL, dnsmasq notes.echo6.co, group member lists) |
| Missing documentation items | 7 (TAK Server, LiveSync, RECON in env, files.echo6.co, headscale nodes, termix, usenet creds) |
| Total files audited | 44 |
| Files with issues | 24 |
| Files clean | 20 |
---
*Audit completed: 2026-02-21 by Claude Code on cortex*

6
vault/archive/README.md
View file

@ -1,6 +0,0 @@
# Archive
Historical, one-time deployment logs and migration reports. Kept for paper-trail/rebuild reference, but **not** living documentation — the current state lives in `docs/`, `runbooks/`, and the active `projects/` context docs.
- `projects/` — completed deploy logs (arr, peertube, vaultwarden, watchtower, openwebui theme, headscale, utility caddy, etc.) + mmud build artifacts
- `reports/` — one-time Stream B migration/validation snapshots (2026-04-13)

122
vault/archive/matrix/PHASE6_DECISION.md
View file

@ -1,122 +0,0 @@
# Phase 6 Decision Summary
# Generated: 2026-04-09
## 1. Is Hookshot the Right Choice?
**NO.** Hookshot cannot be used for this use case.
**Reason:** Hookshot's E2EE implementation is incompatible with MAS (Matrix
Authentication Service). Echo6's Synapse uses MAS. All mautrix-signal portal
rooms use E2BE encryption. Hookshot cannot establish crypto sessions when MAS
is active (GitHub issues #1084, #980 — both OPEN as of March 2026). The webhook
payloads would contain useless ciphertext, not plaintext.
This is a hard blocker with no workaround short of disabling MAS (breaks all
user auth) or disabling E2BE on the bridge (exposes Signal messages in transit).
## 2. Recommended Alternative: Synapse Retention + Export Script
Instead of hookshot, a much simpler approach achieves the same goal:
### Step 1: Synapse Config (2 lines)
```yaml
redaction_retention_period: null # Never censor redacted content
experimental_features:
msc2815_enabled: true # Allow moderator access to redacted content
```
This preserves ALL event content in Synapse's Postgres forever, including
redacted messages. MSC2815 provides API access to the original content.
### Step 2: Export Script (cron job, no bot)
A Python script that:
- Runs nightly via cron on Contabo
- Queries Synapse admin API for events in bridged rooms
- Writes JSONL + markdown exports to /mnt/library/Archives/Signal/
- No E2EE handling needed — admin API returns server-side decrypted content
- No bot account, no device verification, no crypto key management
### Why This Is Better Than Hookshot
| Concern | Hookshot | Synapse Retention + Export |
|---------|----------|---------------------------|
| E2EE + MAS | BROKEN | Not applicable (admin API) |
| Per-room config | Manual per room | Automatic (all rooms) |
| New portal auto-join | Requires automation | Not needed (queries DB) |
| Backfill historical | No backfill | Full history available |
| Infrastructure | New container + Redis | 2-line config + cron script |
| Maintenance | Hookshot upgrades, Redis, crypto store | Script only |
| Failure mode | Missed events if down | DB has everything, export retries |
| Edit/redaction capture | Real-time only | Retained in DB forever |
### Why This Is Better Than the Custom Bot (Original Phase 6)
The original Phase 6 plan proposed a full matrix-nio bot with E2EE, asyncpg,
systemd service, healthchecks, and monitoring. That's ~500-1000 lines of Python
for a problem that Synapse already solves with 2 config lines.
The custom bot approach is only justified if:
- Real-time alerting on specific messages is needed (not requested)
- Independent archive separate from Synapse DB is required (not requested)
- Synapse admin API is insufficient for export needs (it's not)
## 3. Questions Requiring Live Test
None. All claims are verified from Synapse source code:
- `redaction_retention_period: null` — confirmed in synapse/config/server.py
- `msc2815_enabled` — confirmed in synapse/config/experimental.py line 372
- Admin API event access — confirmed working (used throughout Phase 3-5)
- Hookshot E2EE+MAS blocker — confirmed from GitHub issues with reproduction steps
## 4. Estimated Effort
### Synapse Config Change
- Edit homeserver.yaml: 5 minutes
- Restart Synapse: 1 minute
- Verify via admin API: 5 minutes
- **Total: ~15 minutes**
### Export Script
- Python script to query admin API and write JSONL/markdown: 1-2 hours
- Cron setup: 5 minutes
- Test with existing rooms: 15 minutes
- **Total: ~2 hours**
### Compare to Full Hookshot Deployment
- Deploy hookshot container: 30 min
- Debug E2EE+MAS incompatibility: hours of dead end
- Fall back to alternative: start over
- **Total: wasted time**
### Compare to Custom Bot (Original Phase 6)
- Bot code + E2EE + DB schema + systemd + monitoring: 4-6 hours
- Device verification walkthrough: 30 min
- Export pipeline: 1-2 hours
- **Total: 6-8 hours**
## 5. Blockers
None for the recommended approach. The only blocker was hookshot's E2EE+MAS
incompatibility, which the recommended approach avoids entirely.
## 6. Decision Matrix
| Approach | Viable | Effort | Completeness | Ops Burden |
|----------|--------|--------|--------------|------------|
| Hookshot | NO (E2EE+MAS blocked) | N/A | N/A | N/A |
| Custom matrix-nio bot | Yes | High (6-8h) | Full (real-time + export) | Medium |
| Synapse retention + export | Yes | Low (~2h) | Full (DB + batch export) | Minimal |
**Recommendation: Synapse retention + export script.**
---
## Ref Files Produced
1. `.ref/synapse_retention_discovery.md` — Current retention config, MSC2815 support, DB sizes, growth rate, rollback
2. `.ref/hookshot_deployment_discovery.md` — Contabo resources, port conflicts, compose snippet
3. `.ref/hookshot_e2ee_discovery.ref` — **BLOCKER documented** — E2EE+MAS incompatibility
4. `.ref/hookshot_payload_discovery.md` — Webhook payload shape (for future reference)
5. `.ref/hookshot_room_targeting_discovery.md` — Per-room model, no wildcard, current portals
6. `.ref/archive_receiver_discovery.md` — Storage options, alternative approaches, CT recommendations
7. `.ref/PHASE6_DECISION.md` — This file

348
vault/archive/matrix/PLAN.md
View file

@ -1,348 +0,0 @@
# mautrix-signal Bridge Deployment Plan
## 1. Deployment Target
**Same Contabo host**, same Docker Compose stack at `/opt/matrix/docker-compose.yml`.
Rationale: Synapse runs as Docker on Contabo (ref: `synapse.md` — "Docker Compose at /opt/matrix/docker-compose.yml"). The bridge container joins the existing `matrix-net` network so it can reach both `matrix-synapse` and `matrix-postgres` by container name without exposing any new ports externally.
## 2. Bridge Version
- **Image:** `dock.mau.dev/mautrix/signal:v0.2603.0`
- **Released:** 2026-03-16 (latest stable as of 2026-04-09)
- **Type:** Go rewrite (NOT the deprecated Python mautrix-signal)
Pin to the exact tag `v0.2603.0`, not `:latest`, so upgrades are intentional.
## 3. Database Plan
Create a new Postgres database and role inside the existing `matrix-postgres` container. The role gets **minimal grants** — `LOGIN` only, no `SUPERUSER`, no `CREATEDB`, no `CREATEROLE`. Ownership of the new database is the sole privilege.
```sql
-- Connect as the synapse superuser to create the role and database
CREATE ROLE mautrix_signal WITH LOGIN PASSWORD '<generated-64-char-password>'
NOSUPERUSER NOCREATEDB NOCREATEROLE;
CREATE DATABASE mautrix_signal
OWNER mautrix_signal
ENCODING 'UTF8'
LC_COLLATE 'C'
LC_CTYPE 'C';
-- No additional GRANT needed — OWNER on the database gives full DDL/DML
-- within mautrix_signal only. The role has zero access to synapse or mas databases.
```
Verification after creation:
```sql
-- Confirm no superuser, no createdb
SELECT rolname, rolsuper, rolcreatedb, rolcreaterole FROM pg_roles WHERE rolname = 'mautrix_signal';
-- Expected: rolsuper=f, rolcreatedb=f, rolcreaterole=f
-- Confirm cannot access synapse DB
SET ROLE mautrix_signal;
SELECT 1 FROM synapse.public.users LIMIT 1; -- should fail with permission denied
RESET ROLE;
```
- Collation matches Synapse's DB settings (ref: `synapse.md` — POSTGRES_INITDB_ARGS uses `--lc-collate=C --lc-ctype=C`)
- No shared schema with Synapse or MAS
- The `synapse` user has Superuser privileges so can create the role/DB (ref: `synapse.md` — "synapse (Superuser, Create role, Create DB)")
- Bridge config URI: `postgres://mautrix_signal:<password>@matrix-postgres:5432/mautrix_signal?sslmode=disable`
## 4. Networking
- **Appservice port:** 29328 (mautrix-signal default)
- **Bind:** 0.0.0.0:29328 inside container (Docker internal only, NOT exposed to host)
- **Appservice address in config:** `http://mautrix-signal:29328` (container name on matrix-net)
- **Verified unused:** No ports in 29xxx range are in use (ref: `appservices.md` — "Full 29000-29999 range — AVAILABLE")
- **No Caddy changes needed** — bridge communicates with Synapse over the internal Docker network
- **No firewall changes needed** — no host port mapping
The bridge container joins `matrix-net` in docker-compose.yml:
```yaml
mautrix-signal:
image: dock.mau.dev/mautrix/signal:v0.2603.0
container_name: mautrix-signal
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
volumes:
- ./mautrix-signal:/data
networks:
- matrix-net
```
No `ports:` section — the container is only reachable from within `matrix-net`.
## 5. Encryption Config
Enable end-to-bridge encryption with MSC4190 (required for MAS compatibility).
### 5a. Bridge config (`config.yaml`)
```yaml
encryption:
allow: true
default: true
require: true
appservice: false
msc4190: true # REQUIRED when MAS is in use
allow_key_sharing: true
pickle_key: generate
self_sign: true
```
**MSC4190 requirement:** The mautrix docs state: "The `encryption` -> `msc4190` config option must be set to true for encryption to work if you use MAS."
Source: https://docs.mau.fi/bridges/general/end-to-bridge-encryption.html
### 5b. Synapse `homeserver.yaml` diff
The exact flag names are confirmed from Synapse 1.147.1 source code at `synapse/config/experimental.py`:
- `msc3202_transaction_extensions` — parsed at line ~167, default `False`
- `msc2409_to_device_messages_enabled` — parsed at line ~163, default `False`
Source: `synapse.config.experimental.ExperimentalConfig` in Synapse v1.147.1 (`docker exec matrix-synapse python3 -c "import synapse.config.appservice"` — inspected live).
The current `homeserver.yaml` has no `experimental_features` block. The exact diff:
```diff
--- a/homeserver.yaml
+++ b/homeserver.yaml
@@ -end of file
+
+experimental_features:
+ msc3202_transaction_extensions: true
+ msc2409_to_device_messages_enabled: true
```
These flags enable:
- `msc3202_transaction_extensions`: Allows Synapse to send to-device messages, device list changes, and OTK counts in appservice transaction pushes (required for E2BE)
- `msc2409_to_device_messages_enabled`: Allows appservices to receive to-device messages (required for encryption key exchange)
**Note:** MSC4190 is NOT an `experimental_features` flag. It is parsed from the appservice registration YAML file as `io.element.msc4190: true` (confirmed from Synapse 1.147.1 source: `synapse/config/appservice.py` line 190: `msc4190_enabled = as_info.get("io.element.msc4190", False)`).
### 5c. Appservice registration (`registration.yaml`)
The bridge auto-generates `registration.yaml` when `encryption.msc4190: true` is set in `config.yaml`. The generated file will include these flags:
```yaml
io.element.msc4190: true
de.sorunome.msc2409.push_ephemeral: true
push_ephemeral: true
```
Synapse 1.147.1 supports MSC4190 (merged in Synapse 1.121.0 via PR #17705).
Source: https://github.com/element-hq/synapse/pull/17705
### 5d. Trust boundary
The bridge process holds plaintext message content in memory after decryption. The trust boundary extends from the Matrix client to the bridge container. Signal transport encryption remains intact on the Signal side.
## 6. MAS Interaction
### Claim: Appservice registration bypasses MAS
**Status: ASSUMPTION — verify during Phase 3 with stop-and-check.**
The evidence supporting this claim:
1. Appservices authenticate to Synapse via `as_token`/`hs_token` in the registration YAML, which is a Synapse-native mechanism predating MAS.
2. GitHub issue element-hq/matrix-authentication-service#3206 shows a user successfully registering and running a mautrix-signal appservice alongside MAS — basic appservice connectivity (GET `/versions`, GET `/account/whoami`, appservice ping) all returned 200 before encryption was attempted.
3. A commenter on that issue confirmed: "Can confirm it works with mautrix-signal and mautrix-whatsapp."
4. The Matrix Application Service spec defines its own auth mechanism independent of the C-S API auth layer.
Source: https://github.com/element-hq/matrix-authentication-service/issues/3206
**However**, no official MAS documentation explicitly states "appservices bypass MAS." The MAS docs (element-hq.github.io/matrix-authentication-service/) have no dedicated appservice compatibility page.
**Phase 3 stop-and-check procedure:**
1. After adding the registration to `homeserver.yaml` and restarting Synapse, check Synapse logs for appservice registration errors
2. Before starting the bridge container, run a manual appservice ping test:
```bash
curl -sv http://127.0.0.1:8008/_matrix/client/v3/account/whoami \
-H "Authorization: Bearer <as_token>" \
-H "Content-Type: application/json" 2>&1
```
**PASS criteria (ALL must be true):**
- HTTP status code is `200`
- Response body contains `"user_id":"@signalbot:echo6.co"`
- Response body contains `"appservice_id":"signal"` (confirms Synapse recognized the as_token as appservice auth)
**FAIL criteria (ANY triggers STOP):**
- HTTP status `401` or `403` → MAS or Synapse rejected the as_token
- HTTP `3xx` redirect to MAS (`Location:` header pointing to `matrix-mas:8080` or `matrix.echo6.co` auth endpoints)
- Response contains MAS-specific indicators: HTML login page, `matrix-authentication-service` in headers/body, or `errcode: M_UNKNOWN_TOKEN` with MAS introspection trace in Synapse logs
- `user_id` in response does NOT match `@signalbot:echo6.co`
**On FAIL:** Do NOT start the bridge container. Do NOT proceed to step 16. Capture the full `curl -sv` output (headers + body) and the last 50 lines of Synapse logs (`docker logs matrix-synapse --tail 50`). Report both for triage.
### Registration steps
1. Generate the registration file by running the bridge container once with config in place
2. Copy `registration.yaml` into the Synapse data volume (`/opt/matrix/synapse/`)
3. Add to `homeserver.yaml`:
```yaml
app_service_config_files:
- /data/registration.yaml
- /data/doublepuppet.yaml
```
Both files are listed — Synapse reads all entries on startup. A single restart covers both registrations.
4. Restart Synapse (`docker compose restart synapse`) to pick up both appservice registrations
## 7. Permissions
```yaml
bridge:
permissions:
"*": relay
"echo6.co": user
"@matt:echo6.co": admin
```
- `*: relay` — external users can interact via relay (relay disabled by default, so effectively no access)
- `echo6.co: user` — all echo6.co users can use the bridge
- `@matt:echo6.co: admin` — full admin access for matt
Single-user deployment — only matt will link a Signal account.
## 8. Double Puppeting
Use the **appservice-based automatic double puppeting** method:
1. Generate a dedicated double-puppet appservice registration (`doublepuppet.yaml`) with a null URL and an `as_token`:
```yaml
id: doublepuppet
url:
as_token: <generated-token>
hs_token: <generated-token>
sender_localpart: _doublepuppet
rate_limited: false
namespaces:
users:
- regex: '@.*:echo6\.co'
exclusive: false
```
2. Register it with Synapse alongside the bridge registration in `app_service_config_files` (see section 6)
3. Configure the bridge:
```yaml
double_puppet:
secrets:
echo6.co: "as_token:<the-as-token-from-doublepuppet.yaml>"
```
This ensures messages matt sends from Signal Desktop appear as `@matt:echo6.co` in Matrix rooms rather than as the Signal ghost user.
**MAS compatibility:** The appservice-based double puppeting method uses the appservice `as_token` to impersonate the user, which works independently of MAS. MAS handles human user auth; appservice tokens are Synapse-native. (Same assumption as section 6 — covered by the stop-and-check.)
## 9. Backup Impact
The existing backup script (`/opt/matrix/scripts/pg_backup.sh`) only backs up the `synapse` database (ref: `synapse.md` — "Backs up synapse DB only (NOT mas DB)").
**Action required — BEFORE bridge goes live:**
1. Update `pg_backup.sh` to dump `mautrix_signal` and `mas`:
```bash
# Add to pg_backup.sh after the synapse dump:
# mautrix-signal bridge database
SIGNAL_BACKUP="${BACKUP_DIR}/mautrix_signal_${TIMESTAMP}.sql.gz"
docker exec matrix-postgres pg_dump -U mautrix_signal -d mautrix_signal | gzip > "${SIGNAL_BACKUP}"
if [ $? -eq 0 ] && [ -s "${SIGNAL_BACKUP}" ]; then
echo "$(date): Backup created: ${SIGNAL_BACKUP} ($(du -h "${SIGNAL_BACKUP}" | cut -f1))"
else
echo "$(date): WARNING: mautrix_signal backup failed"
fi
# MAS database (was missing from backups)
MAS_BACKUP="${BACKUP_DIR}/mas_${TIMESTAMP}.sql.gz"
docker exec matrix-postgres pg_dump -U mas -d mas | gzip > "${MAS_BACKUP}"
if [ $? -eq 0 ] && [ -s "${MAS_BACKUP}" ]; then
echo "$(date): Backup created: ${MAS_BACKUP} ($(du -h "${MAS_BACKUP}" | cut -f1))"
else
echo "$(date): WARNING: mas backup failed"
fi
```
2. Update the cleanup `find` to also cover `mautrix_signal_*.sql.gz` and `mas_*.sql.gz`
3. **Test the backup** by running `pg_backup.sh` manually after DB creation but before starting the bridge. Verify:
- `mautrix_signal` dump succeeds (even if empty, it should produce a valid .sql.gz)
- `mas` dump succeeds
- Retention cleanup patterns match the new filenames
4. Apply the same 14-day retention policy.
## 10. Rollback Plan
If the bridge needs to be removed:
```bash
# 1. Stop and remove the bridge container
cd /opt/matrix
docker compose stop mautrix-signal
docker compose rm -f mautrix-signal
# 2. Remove mautrix-signal service block from docker-compose.yml
# 3. Remove appservice registrations from Synapse homeserver.yaml:
# - Remove /data/registration.yaml from app_service_config_files
# - Remove /data/doublepuppet.yaml from app_service_config_files
# - If app_service_config_files is now empty, remove the key entirely
# 4. Revert experimental_features from homeserver.yaml:
# - Remove the entire experimental_features block:
# experimental_features:
# msc3202_transaction_extensions: true
# msc2409_to_device_messages_enabled: true
# - Only safe to remove if no other bridges depend on these flags.
# As of this plan, no other appservices exist (ref: appservices.md),
# so removal is safe.
# 5. Remove registration files from Synapse volume
rm /opt/matrix/synapse/registration.yaml
rm /opt/matrix/synapse/doublepuppet.yaml
# 6. Restart Synapse to apply config changes
docker compose restart synapse
# 7. Drop the database and role
docker exec matrix-postgres psql -U synapse -c "DROP DATABASE mautrix_signal;"
docker exec matrix-postgres psql -U synapse -c "DROP ROLE mautrix_signal;"
# 8. Remove bridge data directory
rm -rf /opt/matrix/mautrix-signal
# 9. Revert backup script
# Edit pg_backup.sh: remove the mautrix_signal and mas dump sections
# (Keep mas dump if desired — it was missing before this plan anyway)
# 10. Clean up docker image
docker rmi dock.mau.dev/mautrix/signal:v0.2603.0
```
---
## Implementation Order (Phase 3 — requires approval)
1. Generate password for `mautrix_signal` DB role
2. Create DB role (with `NOSUPERUSER NOCREATEDB NOCREATEROLE`) and database in matrix-postgres
3. Verify role privileges are minimal (query `pg_roles`)
4. **Update backup script** to include `mautrix_signal` and `mas` dumps
5. **Test backup** — run `pg_backup.sh` manually, verify all three dumps succeed
6. Create `/opt/matrix/mautrix-signal/` directory
7. Generate initial config with `docker run --rm`
8. Edit `config.yaml` with all settings from this plan
9. Run container again to generate `registration.yaml`
10. Create `doublepuppet.yaml` registration
11. Copy both registration files to Synapse volume (`/opt/matrix/synapse/`)
12. Add `experimental_features` block to `homeserver.yaml`
13. Add `app_service_config_files` with both registration paths to `homeserver.yaml`
14. Restart Synapse
15. **STOP-AND-CHECK:** Verify appservice auth works alongside MAS (see section 6 procedure)
16. Add mautrix-signal service to `docker-compose.yml`
17. `docker compose up -d mautrix-signal`
18. Verify bridge bot appears in Matrix
19. Link Signal account via `!signal link` in bridge bot DM
20. Update docs (services.md, MEMORY.md)

13
vault/archive/matrix/appservices.md
View file

@ -1,13 +0,0 @@
# Appservices Reference
# Generated: 2026-04-09 (Phase 1)
## Current Appservices: NONE
No appservices are currently registered with Synapse.
- No app_service_config_files key in homeserver.yaml
- No registration YAML files in /opt/matrix/synapse/
- No ports in the 29xxx range are in use
## Available Port Range
- 29328 (mautrix-signal default) — AVAILABLE
- Full 29000-29999 range — AVAILABLE (confirmed via ss -tlnp)

103
vault/archive/matrix/archive_receiver_discovery.md
View file

@ -1,103 +0,0 @@
# Archive Receiver Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 6)
#
# NOTE: Hookshot is BLOCKED for E2BE rooms with MAS. This analysis
# covers the receiver requirements IF hookshot were used, AND the
# alternative approaches that avoid hookshot entirely.
## Hookshot Receiver Requirements (if hookshot were viable)
### Minimum Functionality
1. Listen on HTTP port (plain HTTP on internal Docker network is fine — no TLS needed)
2. Accept multipart/form-data PUT or POST requests
3. Verify X-Matrix-Hookshot-Token header against per-room shared secret
4. Parse the `event` part as JSON
5. Write to durable storage
6. Return 200 OK (hookshot retries on non-2xx)
### Authentication
Hookshot sends a per-webhook auth token in X-Matrix-Hookshot-Token header.
The receiver validates this token against a known list.
## Storage Format Comparison
| Format | Pros | Cons | Recommended For |
|--------|------|------|-----------------|
| JSONL files | Greppable, simple, no DB, easy backup | No query capability, no indexes, scattered across files | "Never look at it" archival |
| SQLite per room | Self-contained, portable, SQL queries | Multiple files to manage, concurrent write limits | Small-scale per-room analysis |
| Single SQLite | One file, SQL queries, simple backup | Write contention at scale, max ~10K writes/sec | Small-to-medium single-server |
| Postgres | Full SQL, concurrent writes, indexes, JSONB | Needs a running server, more ops overhead | Query-heavy, large-scale |
Given Matt's "I'll never look at the DB" feedback:
- **Primary: JSONL files** — append-only, one per day per room, greppable, zero ops
- **Secondary: Single SQLite** — for when he does need to query (and he will eventually)
Both can coexist. The receiver writes JSONL immediately, a nightly job imports into SQLite.
## Receiver Location Options
| Location | Pros | Cons |
|----------|------|------|
| Same Contabo host | Simplest networking, no cross-host latency | Adds load to already-busy server |
| Separate CT on Proxmox | Isolated, near /mnt/library storage | Cross-network traffic, more infrastructure |
| pi-nas (library host) | Direct /mnt/library access, no NFS | Pi is slow, limited CPU/RAM |
**Recommendation:** Separate CT on Proxmox (data node preferred — has 1TB NVMe + 1TB SATA).
- /mnt/library is NFS-mounted on data node CTs
- Lightweight Python service, minimal resources (512MB RAM, 1 core)
- Keeps archive processing off Contabo
## Alternative Approaches (No Hookshot)
### Approach A: Synapse-Level Only (Simplest)
No bot, no receiver. Just Synapse config changes:
```yaml
redaction_retention_period: null
experimental_features:
msc2815_enabled: true
```
Data stays in Synapse's Postgres forever. Query via:
- Synapse admin API: GET /_synapse/admin/v1/rooms/{room_id}/messages
- Direct Postgres: SELECT from event_json WHERE room_id = '...'
Export scripts run on Contabo, dump to /mnt/library via NFS or rsync.
Pros: Zero new infrastructure, zero ops burden, data already exists in DB
Cons: No real-time alerting, export is batch-only, tied to Synapse DB format
### Approach B: Custom matrix-nio Bot (Original Phase 6 Plan)
Python bot using matrix-nio with E2EE + MSC4190 support.
- Handles MAS login correctly (unlike hookshot)
- Decrypts E2BE rooms natively
- Writes to its own DB (independent of Synapse retention)
- Real-time capture with custom schema
Pros: Full control, real-time, independent archive, custom schema
Cons: More code to write and maintain, another service to monitor
### Approach C: Hybrid (Recommended)
Combine Approach A + lightweight export:
1. Enable `redaction_retention_period: null` + `msc2815_enabled: true`
→ Synapse retains everything, MSC2815 provides moderator access
2. Build a simple export script (NOT a bot, NOT a service):
- Runs nightly via cron
- Queries Synapse admin API for room events
- Writes JSONL + markdown exports to /mnt/library
- No E2EE handling needed — queries the server-side decrypted content
3. No new services, no bot accounts, no device verification
This avoids the hookshot E2EE+MAS blocker entirely AND avoids the complexity
of a custom matrix-nio bot. The Synapse admin API already has the data.
## CT Number for Receiver/Export Service (if needed)
Current CT assignments on data node: CT 130 (RECON)
Free CTs on data node: 131-149
If a dedicated CT is needed: CT 131 (next available on data node)
But with Approach C (hybrid), no dedicated CT is needed — the export script
runs on Contabo via cron alongside the existing backup job.

205
vault/archive/matrix/archivist_discovery.md
View file

@ -1,205 +0,0 @@
# Signal Archive Bot — Phase 1 Discovery Findings
# Date: 2026-04-12
# Status: COMPLETE — awaiting Matt's review before Phase 2
## 1. Synapse Homeserver Configuration
| Setting | Value |
|---------|-------|
| server_name | `echo6.co` |
| public_baseurl | `https://matrix.echo6.co/` |
| database | PostgreSQL `synapse` on `matrix-postgres:5432` |
| enable_registration | `false` |
| registration_shared_secret | `<REDACTED — see credentials file: MATRIX_SYNAPSE_REGISTRATION_SHARED_SECRET>` |
| macaroon_secret_key | `<REDACTED — see credentials file: MATRIX_SYNAPSE_MACAROON_SECRET_KEY>` |
| MAS enabled | `true` (endpoint: `http://matrix-mas:8080/`) |
| MAS secret | `<REDACTED — see credentials file: MATRIX_MAS_SYNAPSE_SECRET>` |
### Experimental Features (already enabled)
- `msc3202_transaction_extensions: true` — appservice transaction extensions for E2BE
- `msc2409_to_device_messages_enabled: true` — to-device messages for appservices
### Appservice Registrations
1. `/data/registration.yaml` — mautrix-signal bridge
2. `/data/doublepuppet.yaml` — double puppeting for echo6.co
## 2. MAS (Matrix Authentication Service)
- Image: `ghcr.io/element-hq/matrix-authentication-service:latest`
- Container: `matrix-mas`
- Port: 127.0.0.1:8085 (host) → 8080 (container)
- Database: PostgreSQL `mas` on `matrix-postgres:5432`, user `mas`
- Upstream OAuth2: Echo6 SSO (Authentik) at `https://auth.echo6.co/application/o/matrix/`
- Client ID: `93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T`
- Passwords: enabled (bcrypt, argon2id)
- Email transport: `blackhole` (not functional)
## 3. Matrix Users (6 real + 966 signal puppets)
| User | Admin | Purpose |
|------|-------|---------|
| @matt:echo6.co | Yes | Primary admin |
| @matt1:echo6.co | Yes | Secondary admin |
| @cortex:echo6.co | Yes | Echo6 Cortex Agent bot |
| @contabo:echo6.co | No | Echo6 Contabo Agent bot |
| @agent:echo6.co | No | Bot account |
| @zerby1470:echo6.co | No | Regular user |
| @signalbot:echo6.co | No | mautrix-signal bridge bot |
## 4. mautrix-signal Bridge
| Setting | Value |
|---------|-------|
| Image | `dock.mau.dev/mautrix/signal:v0.2603.0` |
| Container | `mautrix-signal` |
| Status | Running (2+ days uptime, healthy, no errors) |
| Homeserver | `http://matrix-synapse:8008` (Docker network) |
| Appservice addr | `http://mautrix-signal:29328` |
| Bot user | `@signalbot:echo6.co` (device ID `UPX4KKLZVY`) |
| Username template | `signal_{{.}}` |
| Login ID | `58f99d83-f3a8-487f-a2b7-3d118e236d23` (matt's Signal) |
| Database | PostgreSQL `mautrix_signal` on `matrix-postgres:5432` |
### Encryption Config
- allow: `true`
- default: `true`
- require: `true` — ALL bridged rooms use E2BE
- msc4190: `true` (device masquerading for next-gen auth/MAS)
- self_sign: `true`
- allow_key_sharing: `true`
- verification_levels: receive=unverified, send=unverified, share=cross-signed-tofu
### Portal Rooms (10 active, 5 unlinked)
| Portal Name | Room ID | Type |
|-------------|---------|------|
| COMMS LP group | !XUeWZuPdWQQnUYLJBJ:echo6.co | group |
| Liberal_Preppers_OG | !RvWNPmcKtPImhKPYcA:echo6.co | group |
| The Weekly Topic | !vBXtbgfYcptEuimrmn:echo6.co | group |
| Glimmers LP grouo | !qlwFBjKkdpqyUtrvkD:echo6.co | group |
| admins | !YaNspRceyamcRdFmfG:echo6.co | group |
| Left Preppers | !JfxIRowNkLbBlNPjVX:echo6.co | group |
| Resource Media/Links LP groups | !SnGDZgBtYOQuTWeYXp:echo6.co | group |
| DM (e949ab79) | !tSvEWQcXxJItLGAXDr:echo6.co | dm |
| DM (0e206fa1) | !hiDxGpfsYESpVDQXKW:echo6.co | dm |
| DM (a7d7d253) | !EepVZgnoMGiRkIdTAh:echo6.co | dm |
5 additional unlinked portals exist in the DB (Signal conversations with no Matrix room yet).
### Ghost Users
20+ Signal contacts represented as @signal_<uuid>:echo6.co puppet accounts.
## 5. Docker Compose Stack (/opt/matrix/docker-compose.yml)
| Container | Image | Host Port |
|-----------|-------|-----------|
| matrix-postgres | postgres:16-alpine | None (internal) |
| matrix-synapse | matrixdotorg/synapse:latest | 127.0.0.1:8008 |
| matrix-mas | ghcr.io/element-hq/matrix-authentication-service:latest | 127.0.0.1:8085 |
| matrix-element | vectorim/element-web:latest | 127.0.0.1:8088 |
| mautrix-signal | dock.mau.dev/mautrix/signal:v0.2603.0 | None (internal) |
All on `matrix-net` Docker bridge network.
## 6. User Creation Path — @archivist:echo6.co
### Option A: Synapse shared-secret registration (RECOMMENDED)
- `registration_shared_secret` IS configured in homeserver.yaml
- Need to verify if the `/_synapse/admin/v1/register` endpoint still works with MAS enabled
- If it does: `register_new_matrix_user -u archivist -p <password> -a -c /data/homeserver.yaml http://localhost:8008`
### Option B: MAS admin API
- Create user via MAS's admin API or `mas-cli`
- Would create the user in MAS DB + Synapse automatically
- More aligned with the auth architecture
### Option C: Direct DB insert
- Insert into Synapse `users` table + MAS `users` table
- Fragile, not recommended
### Critical consideration: is_synapse_admin
- Per MEMORY.md, MAS compat tokens need `is_synapse_admin` set on `compat_sessions` rows
- The archivist bot needs admin privileges to join rooms and access room state
- After creation, will need to grant admin via DB or API
## 7. Proxmox CT ID Allocation
### Used CTs (21 total):
100, 101, 102, 103, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 120, 121, 130, 200
### Used VMs:
105 (arr), 150 (cortex)
### Free CT IDs in range 100-149:
104(?), 118, 119, 122-129, 131-149
Note: CT 104 (meshing-around) is documented but NOT running in cluster. CT 117 (nf-mtp) is new/undocumented.
### Suggested CT: 118 on utility
- IP: 192.168.1.118
- Resources: 1-2 vCPU, 1 GB RAM, 8 GB disk
- Utility has 20.7 GB free RAM, 0.6% CPU usage
### Node Resource Summary
| Node | RAM Free | CPU Usage |
|------|----------|-----------|
| data | 27.4 GB | 8.8% |
| cloud | 23.0 GB | 0.3% |
| utility | 20.7 GB | 0.6% |
| media | 19.2 GB | 4.2% |
| toc | 26.4 GB | 1.7% |
## 8. Storage — /mnt/library
| Metric | Value |
|--------|-------|
| NFS source | pi-nas (192.168.1.245):/export/library |
| Total | 22 TB |
| Used | 4.4 TB (21%) |
| Free | 18 TB |
| Current mounts | CT 130 (RECON) on data node, CT 118 (archivist) on utility node |
| /mnt/library/signal-archive | Created 2026-04-12, 777 permissions |
### NFS export access:
- `100.64.0.0/10` — Tailscale clients (rw, insecure)
- `192.168.1.0/24` — Local network (rw, insecure, no_root_squash)
### NFS consumers:
- CT 130 (data): pi-nas:/export/library → /mnt/library (host-side mount + bind)
- CT 110 (media): pi-nas:/export/peertube → /var/www/peertube/storage
- CT 118 (utility): pi-nas:/export/library → /mnt/library (host-side mount + bind, added 2026-04-12)
### Note (2026-04-12):
Utility Proxmox host did NOT have /mnt/library mounted before Phase 3. NFS entry added to utility /etc/fstab and bind-mounted into CT 118 via mp0. In-container NFS mounts fail in unprivileged LXC (access denied / operation not permitted).
## 9. Architecture Decision Points
### Q1: Where should the bot run?
- **Option A: New LXC on utility (CT 118)** — project spec says Phase 3 is "Deploy Bot Host (LXC)". NFS mount via local network. Connect to Synapse via https://matrix.echo6.co or Tailscale.
- **Option B: On Contabo** — same Docker network as Synapse (easiest Matrix connectivity). But NFS would need to go over Tailscale (slow for writes). Could also write locally and rsync.
- **Recommendation: Option A (CT 118 on utility)** — matches project spec, fast NFS, bot connects to homeserver via Tailscale or public URL.
### Q2: Does the bot need Synapse admin?
- To join rooms it's not invited to: YES (admin API `/_synapse/admin/v1/join`)
- To read encrypted messages: needs to be IN the room and have Megolm keys shared to it
- Alternative: matt invites @archivist to each bridged room manually
### Q3: E2EE key handling
- Bridge rooms have encryption REQUIRED
- Bot must implement full Megolm key management via matrix-nio
- Device verification (Phase 4) is critical — unverified devices won't receive keys if bridge or other clients have key-sharing restrictions
- Bridge's share level is `cross-signed-tofu` — bot needs valid cross-signing
### Q4: Transcript storage format
- Project spec: plain text, append-only, one file per room
- Path: /mnt/library/signal-archive/<room-name>/<YYYY-MM-DD>.txt
- No database, just files
## 10. Risks and Blockers
1. **MAS + shared secret registration compatibility** — untested, may need MAS admin API instead
2. **E2BE key sharing** — bridge requires cross-signed-tofu for key sharing. Bot must set up cross-signing and verify.
3. **Room join mechanism** — bot needs invitation or admin force-join to each of the 10 bridged rooms
4. **NFS permissions** — new CT will mount as nobody:nogroup by default. Transcript files need correct ownership.
5. **libolm dependency** — matrix-nio E2EE requires libolm C library. Must be available in the LXC container.

107
vault/archive/matrix/hookshot_deployment_discovery.md
View file

@ -1,107 +0,0 @@
# Hookshot Deployment Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 2)
## Contabo Resource Availability
### Memory
- Total: 47 GB
- Used: 11 GB
- Available: 35 GB
- Headroom: PLENTY for hookshot (~100 MB requirement)
### Disk
- Total: 968 GB
- Used: 77 GB (8%)
- Available: 892 GB
- Headroom: PLENTY
### CPU
Current idle is ~65%. TAK Server is the biggest consumer at 22%.
Hookshot is single-threaded Node.js, minimal CPU.
Verdict: Hookshot can run on Contabo alongside existing services with no resource concerns.
## Hookshot Version
- **Latest stable:** 7.3.2 (released 2026-01-30)
- **Image:** halfshot/matrix-hookshot:7.3.2 (Docker Hub, multi-arch)
- **Pin to exact tag**, not :latest
## Port Requirements
| Purpose | Default Port | Status on Contabo |
|---------|-------------|-------------------|
| Appservice (HS→hookshot) | 9993 | AVAILABLE |
| Webhooks (inbound/outbound) | 9000 | CONFLICT — Authentik (127.0.0.1:9000) and TAK (100.64.0.1:9000) |
| Metrics (Prometheus) | 9001 | CONFLICT — TAK (100.64.0.1:9001) |
| Widgets | 9002 | AVAILABLE |
Conflicts on default ports. Resolution options:
- Use non-default ports: e.g., 9800 for webhooks, 9801 for metrics
- Since hookshot outbound webhooks talk to a LOCAL receiver, any port works
- Appservice port 9993 is fine (container-internal via matrix-net)
## Appservice Registration Flow
Same pattern as mautrix-signal. Registration file (registration.yml) goes into
Synapse's app_service_config_files list.
Registration file format:
```yaml
id: matrix-hookshot
as_token: <random>
hs_token: <random>
namespaces:
rooms: []
users:
- regex: "@_webhooks_.*:echo6\\.co"
exclusive: true
aliases: []
sender_localpart: hookshot
url: "http://matrix-hookshot:9993"
rate_limited: false
```
MAS bypass: CONFIRMED from Phase 3. Appservice tokens (as_token/hs_token) are
handled natively by Synapse, bypassing MAS entirely. Same mechanism as mautrix-signal.
## Docker Compose Snippet (NOT APPLIED)
```yaml
matrix-hookshot:
image: halfshot/matrix-hookshot:7.3.2
container_name: matrix-hookshot
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
volumes:
- ./hookshot:/data
networks:
- matrix-net
# No ports: section — internal to matrix-net only
# Appservice reachable at http://matrix-hookshot:9993 from synapse
# Webhook listener on 9800 (non-default) for internal receiver
```
No host port mapping needed. hookshot communicates with:
- Synapse via matrix-net (appservice transactions)
- A local receiver (if built) via matrix-net or localhost
## Dependencies
If encryption is enabled (relevant to Question 3):
- Redis is REQUIRED
- Could use Mailcow's existing Redis, or add a dedicated instance
- Crypto store needs persistent volume
If encryption is NOT enabled:
- No Redis needed
- Minimal config
## Resource Requirements
Per hookshot docs:
- RAM: ~100 MB baseline
- CPU: Minimal (single-threaded Node.js)
- Disk: Negligible (crypto store if E2EE, otherwise just config)

103
vault/archive/matrix/hookshot_e2ee_discovery.md
View file

@ -1,103 +0,0 @@
# Hookshot E2EE Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 3)
## !! BLOCKER: Hookshot E2EE + MAS is BROKEN !!
### The Problem
Hookshot v7.3.2 CANNOT establish E2EE crypto sessions when Synapse uses MAS
(Matrix Authentication Service). This is confirmed in two open GitHub issues:
- Issue #1084: "MAS + hookshot can't login for encryption"
https://github.com/matrix-org/matrix-hookshot/issues/1084
- Issue #980: "Encryption + MAS unable to start"
https://github.com/matrix-org/matrix-hookshot/issues/980
Both issues remain OPEN as of March 2026.
### Root Cause
When encryption is enabled, hookshot needs to establish a device and crypto
session. It does this via a login call that MAS intercepts and rejects with
`M_UNRECOGNIZED: Invalid login type`. The mautrix bridges solved this via
MSC4190 (`encryption.msc4190: true`), but hookshot's bot-sdk does NOT implement
MSC4190 for device management — only for user registration (fixed in v7.2.0).
### Compatibility Matrix
| Scenario | Status | Source |
|----------|--------|--------|
| Hookshot + MAS, no encryption | WORKS (since v7.2.0) | PR #1092 |
| Hookshot + MAS, with encryption | BROKEN | Issue #1084, #980 |
| Hookshot + no MAS, with encryption | WORKS | Docs confirm |
| Hookshot + no MAS, no encryption | WORKS | Trivial case |
### Echo6 Impact
Echo6 Synapse uses MAS (matrix_authentication_service.enabled: true).
All mautrix-signal portal rooms use E2BE encryption (encryption.require: true).
This means:
1. Hookshot CANNOT decrypt events in bridged Signal rooms
2. Outbound webhooks from encrypted rooms would contain CIPHERTEXT, not plaintext
3. The entire archive would be "Unable to decrypt" — USELESS
### Encryption Library (for reference)
Hookshot uses:
- @vector-im/matrix-bot-sdk → @matrix-org/matrix-sdk-crypto-nodejs → vodozemac (Rust)
- NOT libolm (deprecated)
- Crypto store: SQLite format
- Redis: REQUIRED when encryption is enabled
### Config Keys (for reference, cannot be used with MAS)
```yaml
# hookshot config.yml
encryption:
storagePath: ./cryptostore # persistent volume required
# Redis required for encryption
cache:
redisUri: redis://localhost:6379
# registration.yml additions for encryption
de.sorunome.msc2409.push_ephemeral: true
push_ephemeral: true
org.matrix.msc3202: true
```
### Device Verification
IF encryption worked (which it doesn't with MAS), hookshot would:
- Auto-generate a device on first crypto login
- Need manual verification from a trusted device
- Support key sharing via MSC3202 transaction extensions
### Workaround Options
1. **Wait for upstream fix** — Issue #1084 is open but no timeline
2. **Disable MAS** — Not viable, MAS handles all user auth for echo6.co
3. **Disable E2BE on bridge** — Would expose Signal messages in plaintext on Synapse
wire, defeating the purpose of E2BE. Not recommended.
4. **Use a different tool** — A custom matrix-nio bot (Python) with native MSC4190
support could work. matrix-nio has vodozemac bindings and can be configured for
MAS-compatible device login. This is the Phase 6 original plan's approach.
5. **Skip hookshot entirely** — Use Synapse-level retention (redaction_retention_period: null
+ MSC2815) for the DB-level archive, and build a lightweight custom worker for
real-time export. No hookshot needed.
## Recommendation
Hookshot is NOT viable for archiving E2BE-encrypted Signal bridge rooms on a
MAS-enabled Synapse. The E2EE + MAS incompatibility is a hard blocker with no
workaround short of disabling MAS or E2BE.
The original Phase 6 plan's custom bot approach (matrix-nio + asyncpg) avoids
this entirely because matrix-nio supports MSC4190 natively and can be configured
for MAS-compatible appservice login.
Alternatively, the Synapse-level approach (retention null + MSC2815) requires
NO additional bot for the archival function — the data stays in Synapse's DB
and is queryable via admin API. A simple export script can pull events from the
Synapse DB directly.

113
vault/archive/matrix/hookshot_payload_discovery.md
View file

@ -1,113 +0,0 @@
# Hookshot Outbound Webhook Payload Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 4)
#
# NOTE: Hookshot is BLOCKED for this use case (E2EE + MAS incompatibility).
# This file documents the payload shape for completeness and future reference.
## Outbound Webhook Config
Enable in hookshot config.yml:
```yaml
generic:
outbound: true
```
## HTTP Method and Content-Type
- **Method:** PUT (default) or POST (configurable per-connection)
- **Content-Type:** multipart/form-data
- **Two parts:**
1. `event` file: Raw Matrix event JSON
2. `media` file (optional): Attached media if the event references it
## Headers
| Header | Description |
|--------|-------------|
| X-Matrix-Hookshot-EventId | Matrix event ID |
| X-Matrix-Hookshot-RoomId | Room ID |
| X-Matrix-Hookshot-Token | Per-webhook auth token (for receiver verification) |
## Event Type Coverage
ALL events in the room are forwarded. Specifically:
| Event Type | Forwarded? | Notes |
|------------|-----------|-------|
| m.room.message | Yes | Normal messages |
| m.room.redaction | Yes | Deletions |
| m.room.message with m.relates_to/m.replace | Yes | Edits (m.new_content present) |
| m.room.message with m.relates_to/m.in_reply_to | Yes | Replies |
| m.reaction | Yes | Reactions |
| m.room.member | Yes | Joins/leaves/invites |
| State events | Yes | All state changes |
The docs state: "All events that occur in the room will be sent to the outbound URL,
so be careful to ensure your remote service can filter the traffic appropriately."
## Encrypted Room Behavior
IF encryption is working (which it's NOT with MAS — see hookshot_e2ee_discovery.ref):
- The `event` payload contains DECRYPTED plaintext content
- The receiver sees the same JSON as an unencrypted room
IF encryption is NOT working:
- The `event` payload contains the encrypted blob (m.room.encrypted type)
- Content is unusable ciphertext
## Sample Payload (from docs/source)
```json
{
"type": "m.room.message",
"sender": "@signal_abc123:echo6.co",
"event_id": "$abc123:echo6.co",
"room_id": "!XUeWZuPdWQQnUYLJBJ:echo6.co",
"origin_server_ts": 1775773462151,
"content": {
"msgtype": "m.text",
"body": "Hello from Signal"
},
"unsigned": {
"age": 42
}
}
```
Edit event example:
```json
{
"type": "m.room.message",
"sender": "@signal_abc123:echo6.co",
"event_id": "$edit123:echo6.co",
"room_id": "!XUeWZuPdWQQnUYLJBJ:echo6.co",
"origin_server_ts": 1775773462200,
"content": {
"msgtype": "m.text",
"body": "* Hello from Signal (edited)",
"m.new_content": {
"msgtype": "m.text",
"body": "Hello from Signal (edited)"
},
"m.relates_to": {
"rel_type": "m.replace",
"event_id": "$abc123:echo6.co"
}
}
}
```
## Filtering
No built-in filtering. The receiver must filter by `type` field.
Hookshot sends everything — the receiver decides what to keep.
## Retry Behavior
Failed deliveries are retried up to 5 times with increasing delays.
## Source
- https://matrix-org.github.io/matrix-hookshot/latest/setup/webhooks.html
- https://github.com/matrix-org/matrix-hookshot/blob/main/docs/setup/webhooks.md
- https://github.com/matrix-org/matrix-hookshot/pull/945

89
vault/archive/matrix/hookshot_room_targeting_discovery.md
View file

@ -1,89 +0,0 @@
# Hookshot Room Targeting Discovery
# Generated: 2026-04-09 (Phase 6.0, Questions 5 + 7)
#
# NOTE: Hookshot is BLOCKED for this use case (E2EE + MAS incompatibility).
# This file documents targeting mechanics for completeness.
## Per-Room vs Wildcard Configuration
### Per-Room Model
Hookshot uses a per-room connection model. Each outbound webhook is stored as a
Matrix room state event (type: `uk.half-shot.matrix-hookshot.generic.hook`).
To add an outbound webhook to a room:
1. Invite hookshot bot to the room
2. Create a webhook connection via bot command or state event
3. Each room gets its own URL + auth token
### NO Wildcard / All-Rooms Option
There is NO built-in wildcard, glob, or "all rooms the bot is in" option.
Each room must be individually configured.
### Static Connections (Config File)
Connections CAN be defined statically in config.yml:
```yaml
connections:
- connectionType: uk.half-shot.matrix-hookshot.generic.hook
stateKey: archive-hook
roomId: "!XUeWZuPdWQQnUYLJBJ:echo6.co"
state:
name: Signal Archive
```
But each roomId must be specified explicitly. No pattern matching.
### Space Membership
No space-following option. Hookshot cannot be configured to "archive all rooms
in the Signal Bridge space."
## Workflow for Auto-Created Portals
Given that mautrix-signal auto-creates portals on incoming messages:
1. Hookshot would need to be invited to each new portal
2. An outbound webhook connection would need to be created per portal
3. No automation path exists for this within hookshot itself
4. Would require external automation:
- Monitor Synapse for new rooms matching bridge namespace
- Invite hookshot bot
- Create webhook connection via hookshot API or state event
- This is essentially building the custom bot that Phase 6 originally proposed
## Current Portal List (Question 7)
6 portals as of 2026-04-09:
| Signal Group | Matrix Room ID | Type |
|-------------|----------------|------|
| (DM 1) | !tSvEWQcXxJItLGAXDr:echo6.co | dm |
| (DM 2) | !hiDxGpfsYESpVDQXKW:echo6.co | dm |
| COMMS LP group | !XUeWZuPdWQQnUYLJBJ:echo6.co | group |
| Left Preppers | !JfxIRowNkLbBlNPjVX:echo6.co | group |
| Liberal_Preppers_OG | !RvWNPmcKtPImhKPYcA:echo6.co | group |
| The Weekly Topic | !vBXtbgfYcptEuimrmn:echo6.co | group |
More portals will auto-create as messages arrive in other Signal groups.
## Can Hookshot Join Existing Rooms?
Yes — hookshot can be invited to existing rooms after the fact, not just newly
created ones. The outbound webhook connection is created per-room regardless
of when the room was created. No migration path needed beyond:
1. Invite bot
2. Create connection
3. Events forward from that point (no backfill)
## Backfill Limitation
Hookshot outbound webhooks only forward NEW events from the point of connection.
Historical events already in the room are NOT replayed. Any archive tool that
needs historical data must use Synapse's admin API or database access.
## Source
- https://matrix-org.github.io/matrix-hookshot/latest/usage/room_configuration.html
- https://github.com/matrix-org/matrix-hookshot/blob/main/config.sample.yml

701
vault/archive/projects/DEPLOY-API-KEYS-TAB.md
View file

@ -1,701 +0,0 @@
# RECON Dashboard — API Keys Tab Deployment
## Context
SSH into the RECON LXC as zvx: `ssh zvx@100.64.0.24` (or 192.168.1.130)
Working directory: `/opt/recon/`
The dashboard is a Flask app in `lib/api.py` running on port 8420 as a systemd service (`recon.service`).
We're adding:
1. A new `lib/key_manager.py` module (thread-safe, hot-reloadable API key store)
2. A new "API Keys" tab on the dashboard
3. API endpoints for key management
4. Hot-reload integration — enricher and extractor pull keys from KeyManager instead of .env directly
## Step 1: Deploy key_manager.py
Create `/opt/recon/lib/key_manager.py` with the contents of the attached `key_manager.py` file. Copy it exactly — it's a complete, tested module.
Verify it loads:
```bash
cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys loaded: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
"
```
This should show the 4 Gemini keys currently in `.env`.
## Step 2: Add API routes to lib/api.py
Add these routes to `lib/api.py`. Find where the other `/api/` routes are defined and add these in the same pattern:
```python
from lib.key_manager import get_key_manager
# ── API Keys Management ──
@app.route('/keys')
def keys_page():
"""API Keys management page."""
return render_template_string(KEYS_TEMPLATE)
@app.route('/api/keys', methods=['GET'])
def api_get_keys():
"""Get all API keys (masked) with stats."""
km = get_key_manager()
return jsonify({
'gemini': {
'keys': km.get_masked_keys(),
'count': km.get_gemini_key_count(),
},
# Placeholder sections for future services
'services': {
'tei': {
'host': config.get('embedding', {}).get('tei_host', 'unknown'),
'port': config.get('embedding', {}).get('tei_port', 'unknown'),
'status': 'managed in config.yaml'
},
'qdrant': {
'host': config.get('vector_db', {}).get('host', 'unknown'),
'port': config.get('vector_db', {}).get('port', 'unknown'),
'status': 'managed in config.yaml'
},
'ollama': {
'host': config.get('embedding', {}).get('ollama_host', 'unknown'),
'port': config.get('embedding', {}).get('ollama_port', 'unknown'),
'status': 'managed in config.yaml'
}
}
})
@app.route('/api/keys/gemini', methods=['POST'])
def api_add_gemini_key():
"""Add a new Gemini API key."""
data = request.get_json()
if not data or 'key' not in data:
return jsonify({'error': 'Missing "key" field'}), 400
km = get_key_manager()
try:
# Optionally validate before adding
if data.get('validate', True):
valid, msg = km.validate_key(data['key'])
if not valid:
return jsonify({'error': f'Key validation failed: {msg}'}), 400
idx = km.add_gemini_key(data['key'])
return jsonify({'success': True, 'index': idx, 'count': km.get_gemini_key_count()})
except ValueError as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/<int:index>', methods=['PUT'])
def api_replace_gemini_key(index):
"""Replace a Gemini API key at a specific index."""
data = request.get_json()
if not data or 'key' not in data:
return jsonify({'error': 'Missing "key" field'}), 400
km = get_key_manager()
try:
if data.get('validate', True):
valid, msg = km.validate_key(data['key'])
if not valid:
return jsonify({'error': f'Key validation failed: {msg}'}), 400
km.replace_gemini_key(index, data['key'])
return jsonify({'success': True, 'count': km.get_gemini_key_count()})
except (IndexError, ValueError) as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/<int:index>', methods=['DELETE'])
def api_delete_gemini_key(index):
"""Remove a Gemini API key by index."""
km = get_key_manager()
try:
masked = km.remove_gemini_key(index)
return jsonify({'success': True, 'removed': masked, 'count': km.get_gemini_key_count()})
except (IndexError, ValueError) as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/validate', methods=['POST'])
def api_validate_gemini_keys():
"""Validate all loaded Gemini keys."""
km = get_key_manager()
results = km.validate_all()
return jsonify({'results': results})
@app.route('/api/keys/gemini/<int:index>/validate', methods=['POST'])
def api_validate_single_gemini_key(index):
"""Validate a single Gemini key by index."""
km = get_key_manager()
key = km.get_gemini_key(index)
if key is None:
return jsonify({'error': f'No key at index {index}'}), 404
valid, msg = km.validate_key(key)
return jsonify({'index': index, 'valid': valid, 'message': msg})
@app.route('/api/keys/gemini/reveal/<int:index>', methods=['POST'])
def api_reveal_gemini_key(index):
"""Reveal full key (for copy). Requires confirmation in request body."""
data = request.get_json() or {}
if not data.get('confirm'):
return jsonify({'error': 'Send {"confirm": true} to reveal key'}), 400
km = get_key_manager()
key = km.get_gemini_key(index)
if key is None:
return jsonify({'error': f'No key at index {index}'}), 404
return jsonify({'index': index, 'key': key})
@app.route('/api/keys/reload', methods=['POST'])
def api_reload_keys():
"""Force reload keys from .env file."""
km = get_key_manager()
count = km.reload_from_env()
return jsonify({'success': True, 'count': count})
```
**Important:** Make sure `config` refers to whatever variable holds the parsed `config.yaml` in the existing code. Look at how other routes reference config and use the same pattern (likely `get_config()` from `lib/utils.py`).
## Step 3: Add the KEYS_TEMPLATE
Add this HTML template string to `lib/api.py`, alongside the other template strings (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, etc.):
```python
KEYS_TEMPLATE = """
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>RECON — API Keys</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0a0a0a; color: #e0e0e0; min-height: 100vh; }
/* Nav */
.nav { background: #111; border-bottom: 1px solid #222; padding: 0 24px; display: flex; align-items: center; height: 56px; }
.nav-brand { font-size: 18px; font-weight: 700; color: #4ade80; margin-right: 32px; text-decoration: none; letter-spacing: 1px; }
.nav-links { display: flex; gap: 4px; }
.nav-links a { color: #888; text-decoration: none; padding: 8px 16px; border-radius: 6px; font-size: 14px; transition: all 0.15s; }
.nav-links a:hover { color: #e0e0e0; background: #1a1a1a; }
.nav-links a.active { color: #4ade80; background: #1a2e1a; }
/* Layout */
.container { max-width: 960px; margin: 0 auto; padding: 32px 24px; }
h1 { font-size: 24px; font-weight: 600; margin-bottom: 8px; }
.subtitle { color: #666; font-size: 14px; margin-bottom: 32px; }
/* Section */
.section { background: #111; border: 1px solid #222; border-radius: 12px; padding: 24px; margin-bottom: 24px; }
.section-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px; }
.section-title { font-size: 16px; font-weight: 600; display: flex; align-items: center; gap: 8px; }
.section-title .icon { font-size: 20px; }
.section-badge { background: #1a2e1a; color: #4ade80; font-size: 12px; padding: 2px 10px; border-radius: 10px; font-weight: 500; }
/* Key list */
.key-list { display: flex; flex-direction: column; gap: 12px; }
.key-row { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; display: flex; align-items: center; gap: 16px; transition: border-color 0.15s; }
.key-row:hover { border-color: #333; }
.key-index { background: #1a1a1a; color: #666; width: 32px; height: 32px; border-radius: 6px; display: flex; align-items: center; justify-content: center; font-size: 13px; font-weight: 600; flex-shrink: 0; }
.key-value { font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; color: #aaa; flex-grow: 1; word-break: break-all; }
.key-meta { display: flex; gap: 16px; font-size: 12px; color: #555; flex-shrink: 0; }
.key-meta span { white-space: nowrap; }
.key-actions { display: flex; gap: 6px; flex-shrink: 0; }
/* Status badges */
.badge-valid { color: #4ade80; }
.badge-invalid { color: #f87171; }
.badge-unknown { color: #666; }
.badge-ratelimit { color: #fbbf24; }
/* Buttons */
.btn { padding: 6px 14px; border-radius: 6px; border: 1px solid #333; background: #1a1a1a; color: #ccc; font-size: 13px; cursor: pointer; transition: all 0.15s; display: inline-flex; align-items: center; gap: 6px; }
.btn:hover { background: #222; border-color: #444; color: #fff; }
.btn-primary { background: #1a3a1a; border-color: #2a5a2a; color: #4ade80; }
.btn-primary:hover { background: #2a4a2a; border-color: #3a6a3a; }
.btn-danger { background: #2a1a1a; border-color: #5a2a2a; color: #f87171; }
.btn-danger:hover { background: #3a2020; border-color: #6a3030; }
.btn-sm { padding: 4px 10px; font-size: 12px; }
.btn:disabled { opacity: 0.4; cursor: not-allowed; }
/* Input */
.input-row { display: flex; gap: 8px; margin-top: 16px; }
.input-row input { flex-grow: 1; background: #0d0d0d; border: 1px solid #333; border-radius: 6px; padding: 10px 14px; color: #e0e0e0; font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; outline: none; }
.input-row input:focus { border-color: #4ade80; }
.input-row input::placeholder { color: #444; }
/* Status message */
.status-msg { padding: 12px 16px; border-radius: 8px; font-size: 13px; margin-top: 12px; display: none; }
.status-msg.success { display: block; background: #0d1f0d; border: 1px solid #1a3a1a; color: #4ade80; }
.status-msg.error { display: block; background: #1f0d0d; border: 1px solid #3a1a1a; color: #f87171; }
.status-msg.info { display: block; background: #0d0d1f; border: 1px solid #1a1a3a; color: #60a5fa; }
/* Service placeholder cards */
.service-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 12px; }
.service-card { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; }
.service-card .svc-name { font-weight: 600; font-size: 14px; margin-bottom: 4px; }
.service-card .svc-endpoint { font-family: monospace; font-size: 13px; color: #666; margin-bottom: 8px; }
.service-card .svc-note { font-size: 12px; color: #444; font-style: italic; }
/* Spinner */
.spinner { display: inline-block; width: 14px; height: 14px; border: 2px solid #333; border-top-color: #4ade80; border-radius: 50%; animation: spin 0.6s linear infinite; }
@keyframes spin { to { transform: rotate(360deg); } }
/* Confirm overlay */
.confirm-overlay { display: none; position: fixed; inset: 0; background: rgba(0,0,0,0.7); z-index: 100; align-items: center; justify-content: center; }
.confirm-overlay.active { display: flex; }
.confirm-box { background: #111; border: 1px solid #333; border-radius: 12px; padding: 24px; max-width: 420px; width: 90%; }
.confirm-box h3 { margin-bottom: 12px; font-size: 16px; }
.confirm-box p { color: #888; font-size: 14px; margin-bottom: 20px; }
.confirm-actions { display: flex; gap: 8px; justify-content: flex-end; }
</style>
</head>
<body>
<nav class="nav">
<a href="/" class="nav-brand">RECON</a>
<div class="nav-links">
<a href="/">Dashboard</a>
<a href="/search">Search</a>
<a href="/catalogue">Catalogue</a>
<a href="/upload">Upload</a>
<a href="/web-ingest">Web Ingest</a>
<a href="/failures">Failures</a>
<a href="/keys" class="active">API Keys</a>
</div>
</nav>
<div class="container">
<h1>API Key Management</h1>
<p class="subtitle">Manage API keys for pipeline workers. Changes take effect immediately — no restart required.</p>
<!-- Gemini API Keys -->
<div class="section">
<div class="section-header">
<div class="section-title">
<span class="icon">🔑</span>
Gemini API Keys
<span class="section-badge" id="key-count-badge">0 keys</span>
</div>
<div style="display: flex; gap: 8px;">
<button class="btn" onclick="validateAllKeys()" id="btn-validate-all">Validate All</button>
<button class="btn" onclick="reloadFromEnv()">Reload .env</button>
</div>
</div>
<div class="key-list" id="key-list">
<div style="color: #444; text-align: center; padding: 20px;">Loading keys...</div>
</div>
<!-- Add key input -->
<div class="input-row">
<input type="text" id="new-key-input" placeholder="Paste new Gemini API key..." autocomplete="off" spellcheck="false">
<button class="btn btn-primary" onclick="addKey()" id="btn-add">Add Key</button>
</div>
<div id="status-msg" class="status-msg"></div>
<div style="margin-top: 16px; font-size: 12px; color: #444;">
<strong>Used by:</strong> Enrichment (text → concepts, 16 workers) · Vision OCR (scanned PDF fallback) · Title extraction
</div>
</div>
<!-- Future Service Endpoints -->
<div class="section">
<div class="section-header">
<div class="section-title">
<span class="icon">🔌</span>
Service Endpoints
<span class="section-badge" style="background: #1a1a2e; color: #60a5fa;">config.yaml</span>
</div>
</div>
<div class="service-grid" id="service-grid">
<div style="color: #444; text-align: center; padding: 20px;">Loading...</div>
</div>
<div style="margin-top: 16px; font-size: 12px; color: #444;">
Service endpoints are currently managed in <code style="background:#1a1a1a; padding: 2px 6px; border-radius: 3px;">/opt/recon/config.yaml</code>.
Dashboard editing coming in a future update.
</div>
</div>
</div>
<!-- Confirm dialog -->
<div class="confirm-overlay" id="confirm-overlay">
<div class="confirm-box">
<h3 id="confirm-title">Confirm</h3>
<p id="confirm-message">Are you sure?</p>
<div class="confirm-actions">
<button class="btn" onclick="closeConfirm()">Cancel</button>
<button class="btn btn-danger" id="confirm-action-btn" onclick="confirmAction()">Confirm</button>
</div>
</div>
</div>
<script>
let pendingAction = null;
// ── Load keys on page load ──
async function loadKeys() {
try {
const resp = await fetch('/api/keys');
const data = await resp.json();
renderKeys(data.gemini);
renderServices(data.services);
} catch (e) {
showStatus('Failed to load keys: ' + e.message, 'error');
}
}
function renderKeys(gemini) {
const list = document.getElementById('key-list');
const badge = document.getElementById('key-count-badge');
badge.textContent = gemini.count + ' key' + (gemini.count !== 1 ? 's' : '');
if (gemini.keys.length === 0) {
list.innerHTML = '<div style="color:#f87171; text-align:center; padding:20px;">⚠ No Gemini keys loaded — pipeline cannot enrich or OCR</div>';
return;
}
list.innerHTML = gemini.keys.map(k => {
let validClass = 'badge-unknown';
let validIcon = '○';
if (k.valid === true) { validClass = 'badge-valid'; validIcon = '✓'; }
else if (k.valid === false) { validClass = 'badge-invalid'; validIcon = '✗'; }
return `
<div class="key-row" id="key-row-${k.index}">
<div class="key-index">${k.index + 1}</div>
<div class="key-value" id="key-val-${k.index}">${k.masked}</div>
<div class="key-meta">
<span class="${validClass}">${validIcon}</span>
${k.calls > 0 ? `<span>${k.calls} calls</span>` : ''}
${k.errors > 0 ? `<span style="color:#f87171">${k.errors} err</span>` : ''}
</div>
<div class="key-actions">
<button class="btn btn-sm" onclick="validateKey(${k.index})" title="Validate">Test</button>
<button class="btn btn-sm" onclick="revealKey(${k.index})" title="Reveal full key">👁</button>
<button class="btn btn-sm" onclick="promptReplace(${k.index})" title="Replace">↻</button>
<button class="btn btn-sm btn-danger" onclick="promptDelete(${k.index})" title="Remove">✕</button>
</div>
</div>
`;
}).join('');
}
function renderServices(services) {
const grid = document.getElementById('service-grid');
const svcMap = {
tei: { label: 'TEI Embeddings', icon: '📐' },
qdrant: { label: 'Qdrant Vector DB', icon: '🗃' },
ollama: { label: 'Ollama (Fallback)', icon: '🧠' }
};
grid.innerHTML = Object.entries(services).map(([key, svc]) => {
const info = svcMap[key] || { label: key, icon: '⚙' };
return `
<div class="service-card">
<div class="svc-name">${info.icon} ${info.label}</div>
<div class="svc-endpoint">${svc.host}:${svc.port}</div>
<div class="svc-note">${svc.status}</div>
</div>
`;
}).join('');
}
// ── Key operations ──
async function addKey() {
const input = document.getElementById('new-key-input');
const key = input.value.trim();
if (!key) { showStatus('Paste a key first', 'error'); return; }
const btn = document.getElementById('btn-add');
btn.disabled = true;
btn.innerHTML = '<span class="spinner"></span> Validating...';
try {
const resp = await fetch('/api/keys/gemini', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key: key, validate: true })
});
const data = await resp.json();
if (resp.ok) {
showStatus('Key added and validated ✓', 'success');
input.value = '';
loadKeys();
} else {
showStatus(data.error || 'Failed to add key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
} finally {
btn.disabled = false;
btn.innerHTML = 'Add Key';
}
}
async function validateKey(index) {
const row = document.getElementById(`key-row-${index}`);
row.style.borderColor = '#333';
try {
const resp = await fetch(`/api/keys/gemini/${index}/validate`, { method: 'POST' });
const data = await resp.json();
if (data.valid) {
row.style.borderColor = '#2a5a2a';
showStatus(`Key ${index + 1}: ${data.message}`, 'success');
} else {
row.style.borderColor = '#5a2a2a';
showStatus(`Key ${index + 1}: ${data.message}`, 'error');
}
setTimeout(loadKeys, 500);
} catch (e) {
showStatus('Validation failed: ' + e.message, 'error');
}
}
async function validateAllKeys() {
const btn = document.getElementById('btn-validate-all');
btn.disabled = true;
btn.innerHTML = '<span class="spinner"></span> Validating...';
try {
const resp = await fetch('/api/keys/gemini/validate', { method: 'POST' });
const data = await resp.json();
const valid = data.results.filter(r => r.valid).length;
const total = data.results.length;
showStatus(`Validated: ${valid}/${total} keys are working`, valid === total ? 'success' : 'error');
loadKeys();
} catch (e) {
showStatus('Validation failed: ' + e.message, 'error');
} finally {
btn.disabled = false;
btn.innerHTML = 'Validate All';
}
}
async function revealKey(index) {
try {
const resp = await fetch(`/api/keys/gemini/reveal/${index}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ confirm: true })
});
const data = await resp.json();
if (resp.ok) {
const el = document.getElementById(`key-val-${index}`);
el.textContent = data.key;
el.style.color = '#e0e0e0';
// Auto-hide after 10s
setTimeout(() => loadKeys(), 10000);
}
} catch (e) {
showStatus('Failed to reveal: ' + e.message, 'error');
}
}
function promptReplace(index) {
const newKey = prompt(`Paste replacement for key ${index + 1}:`);
if (newKey && newKey.trim()) {
replaceKey(index, newKey.trim());
}
}
async function replaceKey(index, newKey) {
try {
const resp = await fetch(`/api/keys/gemini/${index}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key: newKey, validate: true })
});
const data = await resp.json();
if (resp.ok) {
showStatus(`Key ${index + 1} replaced and validated ✓`, 'success');
loadKeys();
} else {
showStatus(data.error || 'Failed to replace key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
}
}
function promptDelete(index) {
pendingAction = () => deleteKey(index);
document.getElementById('confirm-title').textContent = 'Remove Key ' + (index + 1);
document.getElementById('confirm-message').textContent = 'This key will be removed from the pipeline immediately. Workers using this key will switch to remaining keys.';
document.getElementById('confirm-overlay').classList.add('active');
}
async function deleteKey(index) {
closeConfirm();
try {
const resp = await fetch(`/api/keys/gemini/${index}`, { method: 'DELETE' });
const data = await resp.json();
if (resp.ok) {
showStatus(`Removed key: ${data.removed}`, 'success');
loadKeys();
} else {
showStatus(data.error || 'Failed to remove key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
}
}
async function reloadFromEnv() {
try {
const resp = await fetch('/api/keys/reload', { method: 'POST' });
const data = await resp.json();
if (resp.ok) {
showStatus(`Reloaded ${data.count} key(s) from .env`, 'info');
loadKeys();
}
} catch (e) {
showStatus('Reload failed: ' + e.message, 'error');
}
}
// ── Helpers ──
function showStatus(msg, type) {
const el = document.getElementById('status-msg');
el.textContent = msg;
el.className = 'status-msg ' + type;
if (type === 'success' || type === 'info') {
setTimeout(() => { el.className = 'status-msg'; }, 5000);
}
}
function confirmAction() {
if (pendingAction) { pendingAction(); pendingAction = null; }
}
function closeConfirm() {
document.getElementById('confirm-overlay').classList.remove('active');
pendingAction = null;
}
// Handle Enter key in input
document.getElementById('new-key-input').addEventListener('keydown', e => {
if (e.key === 'Enter') addKey();
});
// Load on page ready
loadKeys();
</script>
</body>
</html>
"""
```
## Step 4: Add "API Keys" to the nav on ALL existing pages
Find the nav HTML in every existing template (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, CATALOGUE_TEMPLATE, UPLOAD_TEMPLATE, WEB_INGEST_TEMPLATE, FAILURES_TEMPLATE). Each has a `<nav>` with links. Add the API Keys link:
```html
<a href="/keys">API Keys</a>
```
Add it after the "Failures" link in each template's nav. The exact pattern to find is something like:
```html
<a href="/failures"...>Failures</a>
```
Add right after it:
```html
<a href="/keys">API Keys</a>
```
## Step 5: Wire enricher.py and extractor.py to use KeyManager
This is the hot-reload part. Currently these modules read keys from config/env at startup. We need them to call `get_key_manager().get_gemini_keys()` each time they need a key, so new keys take effect immediately.
### In lib/enricher.py:
Find where Gemini API keys are loaded or selected (likely something like `cfg['gemini_keys']` or reading from `.env` or `os.environ`). Replace that with:
```python
from lib.key_manager import get_key_manager
# Where a key is selected for a worker (probably in the worker function):
km = get_key_manager()
keys = km.get_gemini_keys()
key = keys[worker_index % len(keys)] # Round-robin across available keys
# After each Gemini call, record usage:
km.record_usage(worker_index % len(keys), success=True) # or success=False on error
```
The exact integration depends on the current code structure. Look for:
- `genai.configure(api_key=...)` calls
- `cfg['gemini_keys']` or `config['gemini_keys']` references
- Any `os.environ.get('GEMINI_KEY')` calls
Replace the key source with `get_key_manager()` calls. The key point: **don't cache the key list** — call `km.get_gemini_keys()` or `km.get_gemini_key(index)` each time so hot-reload works.
### In lib/extractor.py:
Same pattern for the Gemini Vision OCR fallback and the title extraction call. Find where Gemini is configured and replace the key source.
## Step 6: Test
### Test the module:
```bash
cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
results = km.validate_all()
for r in results:
print(f' Key {r[\"index\"] + 1}: {\"✓\" if r[\"valid\"] else \"✗\"} - {r[\"message\"]}')
"
```
### Test the API:
```bash
# Get keys (masked)
curl -s http://localhost:8420/api/keys | python3 -m json.tool
# Validate all
curl -s -X POST http://localhost:8420/api/keys/gemini/validate | python3 -m json.tool
# Validate single
curl -s -X POST http://localhost:8420/api/keys/gemini/0/validate | python3 -m json.tool
```
### Test the dashboard:
Open http://100.64.0.24:8420/keys in a browser. Verify:
- All 4 Gemini keys show up (masked)
- "Validate All" tests each key
- "Test" button validates individual keys
- Eye icon reveals the full key (auto-hides after 10s)
- "Add Key" validates before adding
- "Remove" shows confirmation dialog
- "Replace" prompts for new key and validates
- "Reload .env" picks up external edits
- Service endpoints section shows TEI, Qdrant, Ollama from config.yaml
### Test hot-reload:
1. Open the dashboard, note 4 keys
2. Add a 5th key via the dashboard
3. Check `.env`: `cat /opt/recon/.env` — should have 5 GEMINI_KEY entries
4. Watch logs: `journalctl -u recon -f | grep key_manager` — should show key added
5. The pipeline should immediately start using all 5 keys (enricher round-robins)
### Restart and verify persistence:
```bash
sudo systemctl restart recon
curl -s http://localhost:8420/api/keys | python3 -m json.tool
# Should show same keys as before restart
```
## Report back:
- Screenshot or curl output of /api/keys
- Validate All results
- Confirm nav link appears on all pages
- Confirm hot-reload works (add key, verify enricher uses it)
- Any issues with the existing code structure that needed adapting

262
vault/archive/projects/arr-stack-runbook.md
View file

@ -1,262 +0,0 @@
# CC Runbook: Build ARR Media Stack on Proxmox `media` Node
## Objective
Build a complete media automation stack on the Proxmox node `media` inside a single Ubuntu VM called `arr`. Each service runs in its own Docker container with a shared bridge network for inter-service communication. All services are exposed on the VM's LAN IP on their respective ports.
**Services:**
- Jellyfin (media server, software transcoding — no GPU)
- Jellyseer (request management)
- Sonarr (TV automation)
- Radarr (Movie automation)
- Prowlarr (indexer manager)
- SABnzbd (Usenet download client)
---
## Phase 0: SSH Prereq Check
**CRITICAL — Do this first. Do not skip.**
```bash
ssh media "echo 'SSH OK to media node'"
```
If this fails, stop and fix SSH access before proceeding. Use sshpass or key auth per `~/.ssh/config`. Cortex is the management host — all commands originate from here.
---
## Phase 1: Create Ubuntu VM on `media`
1. SSH to `media` Proxmox node.
2. Find the next available VMID: `pvesh get /cluster/nextid`
3. Download Ubuntu 24.04 cloud image if not already cached:
- URL: `https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img`
- Store in appropriate Proxmox storage.
4. Create a VM named `arr` with:
- **Network:** bridged to the LAN bridge (likely `vmbr0`)
- **Resource allocation:** Decide based on the combined needs of all six services. Jellyfin (software transcoding) and SABnzbd (decompression) are the heaviest. Sonarr/Radarr/Prowlarr/Jellyseer are lightweight. Size the VM accordingly — suggest at minimum 4 cores and 8GB RAM, but use your judgment.
- **Disk:** 30GB for OS + container configs (media lives on NFS)
- Cloud-init configured with:
- Default user: `zvx`
- SSH key from cortex (discover from `~/.ssh/id_rsa.pub` or equivalent)
- Networking: DHCP or static — check the pattern of other VMs on this node and match it
5. Start the VM, wait for boot, discover and record its LAN IP.
6. Verify SSH from cortex → arr VM works.
---
## Phase 2: Base System Setup on `arr` VM
SSH into the `arr` VM:
1. `apt update && apt upgrade -y`
2. Install Docker + Docker Compose via the official Docker apt repo for Ubuntu.
3. Install NFS client: `apt install -y nfs-common`
4. Install Tailscale and join the tailnet:
- `curl -fsSL https://tailscale.com/install.sh | sh`
- `tailscale up` — use an auth key if available. Check how other VMs joined (look at Headscale config if self-hosted).
- Record the Tailscale IP of the `arr` VM.
5. Discover appropriate PUID/PGID:
- Mount the NFS share temporarily and `ls -ln` to check file ownership.
- If no files exist, create a `media` user/group (e.g., PUID=1000, PGID=1000) and ensure NFS permissions align.
---
## Phase 3: NFS Mount
1. **Discover the NFS server:**
- The NFS export is `/export/arr`, accessible from `100.64.0.0/10` (Tailscale) and `192.168.1.0/24` (LAN).
- Find the NFS server IP by checking:
- `/etc/fstab` on other VMs on this node
- `showmount -e <candidate IPs>` on LAN
- Proxmox storage config: `pvesm status` or `/etc/pve/storage.cfg`
2. `mkdir -p /mnt/arr`
3. `mount -t nfs <NFS_SERVER>:/export/arr /mnt/arr`
4. Create subdirectories if they don't exist:
```
mkdir -p /mnt/arr/{movies,tv,downloads,downloads/complete,downloads/incomplete}
```
5. Set ownership to discovered PUID:PGID on all subdirs.
6. Add to `/etc/fstab` for persistence:
```
<NFS_SERVER>:/export/arr /mnt/arr nfs defaults,_netdev 0 0
```
7. Verify: `umount /mnt/arr && mount -a && ls /mnt/arr`
---
## Phase 4: Docker Containers
### Setup
```bash
mkdir -p /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
```
Create a Docker bridge network for inter-service communication:
```bash
docker network create arr-net
```
### Container Deployment
Deploy each service as its own standalone container. All containers join `arr-net`. All get `TZ=America/Boise` and the discovered `PUID`/`PGID`.
**Decide per-container resource limits** (CPU shares, memory limits) based on service needs:
- **Heavy:** Jellyfin (transcoding), SABnzbd (decompression) — allocate more CPU/RAM
- **Medium:** Sonarr, Radarr — moderate
- **Light:** Prowlarr, Jellyseer — minimal
Use lightweight images (hotio where available, official otherwise).
#### Jellyfin
- Image: `jellyfin/jellyfin:latest`
- Container name: `jellyfin`
- Port: `8096:8096`
- Volumes:
- `/opt/arr/jellyfin/config:/config`
- `/mnt/arr/movies:/data/movies:ro`
- `/mnt/arr/tv:/data/tv:ro`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Jellyseer
- Image: `fallenbagel/jellyseer:latest`
- Container name: `jellyseer`
- Port: `5055:5055`
- Volumes:
- `/opt/arr/jellyseer/config:/app/config`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Sonarr
- Image: `ghcr.io/hotio/sonarr:latest`
- Container name: `sonarr`
- Port: `8989:8989`
- Volumes:
- `/opt/arr/sonarr/config:/config`
- `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Radarr
- Image: `ghcr.io/hotio/radarr:latest`
- Container name: `radarr`
- Port: `7878:7878`
- Volumes:
- `/opt/arr/radarr/config:/config`
- `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Prowlarr
- Image: `ghcr.io/hotio/prowlarr:latest`
- Container name: `prowlarr`
- Port: `9696:9696`
- Volumes:
- `/opt/arr/prowlarr/config:/config`
- Network: `arr-net`
- Restart: `unless-stopped`
#### SABnzbd
- Image: `ghcr.io/hotio/sabnzbd:latest`
- Container name: `sabnzbd`
- Port: `8080:8080`
- Volumes:
- `/opt/arr/sabnzbd/config:/config`
- `/mnt/arr/downloads:/data/downloads`
- Network: `arr-net`
- Restart: `unless-stopped`
### Volume Mapping Design
Sonarr and Radarr both map `/mnt/arr:/data` so hardlinks/atomic moves work between `/data/downloads/complete` and `/data/movies` or `/data/tv` without cross-filesystem copies. This is critical for avoiding double disk usage.
### Verify
All six containers are running: `docker ps`
Curl each service on localhost to confirm they respond on their expected ports.
---
## Phase 5: Authentik OIDC Setup
**Discovery:** Find the Authentik instance.
- Check Caddy config on `utility` for an existing Authentik route (likely `auth.echo6.co` or `authentik.echo6.co`).
- Discover the Authentik API URL and obtain/create an API token from Authentik's docker-compose environment or admin API.
### Jellyfin OIDC
1. Create OAuth2/OpenID Provider in Authentik:
- Name: `jellyfin`, Client type: Confidential
- Redirect URI: `https://jellyfin.echo6.co/sso/OID/redirect/Authentik`
- Scopes: `openid profile email`
- Signing key: use existing or create
2. Create Application: Name `Jellyfin`, slug `jellyfin`, attach provider.
3. Record Client ID + Secret.
4. Install SSO-Auth plugin in Jellyfin and configure with Authentik OIDC details (discovery URL, client ID, secret).
### Jellyseer OIDC
1. Create OAuth2/OpenID Provider in Authentik:
- Name: `jellyseer`, Client type: Confidential
- Redirect URI: `https://requests.echo6.co/api/v1/auth/oidc-callback` (verify actual callback path from Jellyseer docs)
- Scopes: `openid profile email`
2. Create Application: Name `Jellyseer`, slug `jellyseer`, attach provider.
3. Record Client ID + Secret.
4. Configure Jellyseer OIDC via its settings.
---
## Phase 6: Caddy Reverse Proxy on `utility`
SSH to `utility`. Discover the Caddyfile location and how Caddy is managed (docker, systemd, etc.).
Add entries using the **Tailscale IP** of the `arr` VM as the upstream:
```
jellyfin.echo6.co {
reverse_proxy <ARR_TAILSCALE_IP>:8096
}
requests.echo6.co {
reverse_proxy <ARR_TAILSCALE_IP>:5055
}
```
**Do NOT expose Sonarr, Radarr, Prowlarr, or SABnzbd via Caddy.** Those are internal-only, accessible via Tailscale or LAN.
Reload Caddy.
---
## Phase 7: GoDaddy DNS
**Discovery:** Check if GoDaddy API key/secret exists on cortex or utility. Look at how existing `echo6.co` subdomains are configured for the pattern.
Create A records (via API if available, otherwise output for manual creation):
| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | `jellyfin` | Public IP of Caddy/utility (discover) | 600 |
| A | `requests` | Public IP of Caddy/utility (discover) | 600 |
These are publicly exposed WITHOUT Tailscale. Caddy handles TLS via Let's Encrypt. The upstream uses the Tailscale IP but DNS points to the public-facing Caddy IP.
---
## Phase 8: Validation
1. From `arr` VM, curl all six services on localhost (ports 8096, 5055, 8989, 7878, 9696, 8080)
2. `curl -sI https://jellyfin.echo6.co` → 200 with valid TLS
3. `curl -sI https://requests.echo6.co` → 200 with valid TLS
4. Authentik OIDC login works for both Jellyfin and Jellyseer
5. NFS persists after reboot: `reboot`, wait, `df -h /mnt/arr`
6. All containers auto-start after reboot: `docker ps` shows all six running
---
## Important Notes
- **Do NOT configure** Prowlarr indexers, Sonarr/Radarr API connections, or SABnzbd Usenet provider credentials. That will be done in a separate prompt.
- **All discovery steps are intentional** — do not hardcode IPs or paths. Find them dynamically from the running infrastructure.
- **If any phase fails, stop and report the error.** Do not skip phases.

193
vault/archive/projects/arr-wiring-runbook.md
View file

@ -1,193 +0,0 @@
# CC Runbook: Wire ARR Stack End-to-End
## Objective
Connect all six services on the `arr` VM into a fully automated pipeline:
```
Jellyseer → Sonarr/Radarr → Prowlarr → SABnzbd → Downloads → Sonarr/Radarr catalogue → Jellyfin library → Jellyseer knows what's available
```
All containers are already running on `arr` on the `arr-net` Docker bridge network. Services can reach each other by container name (e.g., `sonarr:8989`).
---
## Phase 0: Prerequisites
### SSH Check
```bash
ssh media "echo 'SSH OK to media node'"
```
Then SSH into the `arr` VM (discover its IP from Phase 1 of the previous runbook, or check `qm list` / DHCP leases on media).
### Read Credentials File
Read the Usenet provider and indexer credentials from `./ref/services/usenet.md` on cortex. Parse and use these values throughout this runbook.
### Discover API Keys
Every service auto-generates an API key on first run. Extract them:
```bash
# Sonarr
docker exec sonarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# Radarr
docker exec radarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# Prowlarr
docker exec prowlarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# SABnzbd
docker exec sabnzbd cat /config/sabnzbd.ini | grep -oP '(?<=api_key = ).*'
# Jellyfin — get the API key from the admin dashboard or create one via API
```
Record all of these. They are needed for every integration below.
---
## Phase 1: SABnzbd — Configure Usenet Provider
Using credentials from `./ref/services/usenet.md`:
1. Via SABnzbd API (`http://localhost:8080/api`), configure the Usenet server:
- Server hostname, port, SSL, username, password — all from the ref file
- Connections: set to provider's recommended max
- SSL verification: enable
2. Set download paths in SABnzbd:
- Complete: `/data/downloads/complete`
- Incomplete: `/data/downloads/incomplete`
3. Configure categories in SABnzbd:
- `movies` → `/data/downloads/complete/movies`
- `tv` → `/data/downloads/complete/tv`
4. Verify SABnzbd can connect to the Usenet server (test connection).
---
## Phase 2: Prowlarr — Add Indexers
Using indexer credentials from `./ref/services/usenet.md`:
1. Via Prowlarr API (`http://localhost:9696/api/v1`), add each Usenet indexer found in the ref file.
- For each indexer: set name, URL, API key, and enable.
- Use the Prowlarr API key discovered in Phase 0.
2. Test each indexer to confirm connectivity.
---
## Phase 3: Prowlarr — Connect to Sonarr and Radarr
Add Sonarr and Radarr as "Applications" in Prowlarr so indexers automatically sync:
### Sonarr
- Prowlarr API → Add Application:
- Type: Sonarr
- Prowlarr server: `http://prowlarr:9696`
- Sonarr server: `http://sonarr:8989`
- API key: Sonarr's API key from Phase 0
- Sync level: Full Sync
### Radarr
- Prowlarr API → Add Application:
- Type: Radarr
- Prowlarr server: `http://prowlarr:9696`
- Radarr server: `http://radarr:7878`
- API key: Radarr's API key from Phase 0
- Sync level: Full Sync
After adding, trigger a sync and verify indexers appear in Sonarr and Radarr.
---
## Phase 4: Sonarr — Configure Download Client and Paths
Via Sonarr API (`http://localhost:8989/api/v3`):
1. Add SABnzbd as download client:
- Host: `sabnzbd`
- Port: `8080`
- API key: SABnzbd API key from Phase 0
- Category: `tv`
- Test connection.
2. Configure Root Folder:
- Path: `/data/tv`
3. Configure Media Management:
- Rename episodes: Yes
- Use hardlinks: Yes (critical — same filesystem via NFS mount)
---
## Phase 5: Radarr — Configure Download Client and Paths
Via Radarr API (`http://localhost:7878/api/v3`):
1. Add SABnzbd as download client:
- Host: `sabnzbd`
- Port: `8080`
- API key: SABnzbd API key from Phase 0
- Category: `movies`
- Test connection.
2. Configure Root Folder:
- Path: `/data/movies`
3. Configure Media Management:
- Rename movies: Yes
- Use hardlinks: Yes
---
## Phase 6: Jellyfin — Configure Libraries
Via Jellyfin API or admin setup:
1. Create (or verify) media libraries:
- **Movies** library → `/data/movies`
- **TV Shows** library → `/data/tv`
2. Set libraries to scan periodically or on change.
3. Create an API key for Jellyseer to use (Admin Dashboard → API Keys → create one named `jellyseer`).
---
## Phase 7: Jellyseer — Connect Everything
Via Jellyseer's setup wizard or API:
1. **Jellyfin connection:**
- Server URL: `http://jellyfin:8096`
- API key: the Jellyfin API key created in Phase 6
- Sync libraries so Jellyseer knows what Jellyfin already has.
- Sign in with the Jellyfin admin account to link it.
2. **Sonarr connection:**
- Server URL: `http://sonarr:8989`
- API key: Sonarr API key from Phase 0
- Root folder: `/data/tv`
- Quality profile: discover available profiles from Sonarr and pick a sensible default (e.g., `Any` or `HD-1080p`)
3. **Radarr connection:**
- Server URL: `http://radarr:7878`
- API key: Radarr API key from Phase 0
- Root folder: `/data/movies`
- Quality profile: discover available profiles and pick a sensible default
---
## Phase 8: End-to-End Validation
Test the full pipeline:
1. **Prowlarr → Indexers:** Search for a common term (e.g., "test") in Prowlarr. Results should return from all configured indexers.
2. **Sonarr → Prowlarr:** In Sonarr, verify indexers are listed under Settings → Indexers (synced from Prowlarr).
3. **Radarr → Prowlarr:** Same check in Radarr.
4. **Sonarr → SABnzbd:** Verify download client is connected (Settings → Download Clients → test).
5. **Radarr → SABnzbd:** Same check.
6. **Jellyseer → Jellyfin:** Verify Jellyseer shows Jellyfin's existing library (if any).
7. **Jellyseer → Sonarr/Radarr:** Verify both are connected in Jellyseer settings.
8. **Full flow test:** If desired, use Jellyseer to request a free/public domain title and verify it flows through the entire chain: Jellyseer → Sonarr/Radarr → Prowlarr search → SABnzbd download → file lands in `/data/tv` or `/data/movies` → Jellyfin picks it up → Jellyseer shows it as available.
---
## Important Notes
- **All services communicate by Docker container name** on `arr-net` (e.g., `http://sonarr:8989`), NOT by localhost or LAN IP.
- **Hardlinks are critical.** Sonarr/Radarr and SABnzbd share the same `/data` mount from the NFS share. This means completed downloads can be hardlinked (not copied) into the media folders, avoiding double disk usage.
- **API-first approach.** Configure everything via API calls rather than manual UI interaction. This ensures repeatability and lets CC automate the full setup.
- **If any phase fails, stop and report the error.** Do not skip phases.
- **The credentials file is `./ref/services/usenet.md` on cortex.** Read it first and use its contents throughout.

223
vault/archive/projects/cc-deploy-watchtower-v2.md
View file

@ -1,223 +0,0 @@
# Deploy WATCHTOWER v2 — Modular Ops Dashboard
**Context:** CC runs on cortex. WATCHTOWER deploys to Contabo (100.64.0.1). The tarball is at `/home/zvx/projects/contabo/watchtower/watchtower-v2.tar.gz` on cortex. This runbook is at `/home/zvx/.ref/projects/` on cortex.
WATCHTOWER v2 is a modular FastAPI monitoring dashboard. Collectors are auto-discovered from `app/collectors/` and enabled via `{NAME}_ENABLED=true` in `.env`. Adding new monitoring targets requires zero edits to existing files.
## Pre-flight: Transfer tarball and SSH check
```bash
# SCP tarball from cortex (this machine) to Contabo
scp /home/zvx/projects/contabo/watchtower/watchtower-v2.tar.gz zvx@100.64.0.1:/tmp/
# Verify sshpass is installed on Contabo
ssh zvx@100.64.0.1 "which sshpass || sudo apt-get install -y sshpass"
# Test SSH from Contabo to each monitored node
ssh zvx@100.64.0.1 << 'SSHEOF'
echo "=== PeerTube (100.64.0.23) ==="
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no zvx@100.64.0.23 "hostname && echo OK" 2>&1
echo "=== cortex/GPU (100.64.0.14) ==="
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no zvx@100.64.0.14 "hostname && echo OK" 2>&1
SSHEOF
```
If either SSH fails, stop and report the error. Do not proceed without working SSH to at least one target.
---
## Phase 1: Deploy codebase
All remaining commands run on Contabo. SSH in:
```bash
ssh zvx@100.64.0.1
```
Then:
```bash
# Clean any old install
sudo rm -rf /opt/watchtower
# Extract v2 tarball
sudo tar xzf /tmp/watchtower-v2.tar.gz -C /opt/
sudo mv /opt/watchtower-v2 /opt/watchtower
sudo chown -R $USER:$USER /opt/watchtower
cd /opt/watchtower
```
### Create .env from example
```bash
cp .env.example .env
```
The defaults in `.env.example` are already set to the correct current values:
| Target | IP | User | Notes |
|--------|-----|------|-------|
| GPU (cortex) | 100.64.0.14 | zvx | nvidia-smi |
| PeerTube | 100.64.0.23 | zvx | Native PostgreSQL (`peertube_prod`), pipeline at `/opt/bulk-import/` |
| RECON | disabled | — | Flip `RECON_ENABLED=true` when rebuilt |
### Verify PeerTube PostgreSQL access
PostgreSQL runs natively on the PeerTube CT (not in Docker). Verify:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "sudo -u postgres psql -d peertube_prod -t -A -c 'SELECT COUNT(*) FROM video;'"
```
Should return the video count (e.g., 207). If it errors, the DB name may be different — check with:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "sudo -u postgres psql -l"
```
Update `PT_DB_NAME` in `.env` if needed.
### Verify bulk-import pipeline paths
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "ls -la /opt/bulk-import/ 2>/dev/null && wc -l /opt/bulk-import/downloaded.txt 2>/dev/null || echo 'PATH NOT FOUND'"
```
---
## Phase 2: Build and start
```bash
cd /opt/watchtower
docker compose up -d --build
# Wait for startup then check logs
sleep 5
docker logs watchtower 2>&1 | tail -30
```
### Expected log output
```
WATCHTOWER starting up...
Database connected: /data/watchtower.db
[registry] Loaded collector: gpu (GPU (cortex))
[registry] Loaded collector: peertube (PeerTube Ingest)
[registry] Skipped collector: recon (RECON_ENABLED=false)
[registry] 2 collector(s) active: ['gpu', 'peertube']
[gpu] collector starting (interval: 60s)
[peertube] collector starting (interval: 60s)
```
### Verify collectors
```bash
# Wait for first poll cycle
sleep 65
echo "=== Health ==="
curl -s http://localhost:8084/api/health | python3 -m json.tool
echo "=== Collector Manifest ==="
curl -s http://localhost:8084/api/collectors | python3 -m json.tool
echo "=== GPU Data ==="
curl -s http://localhost:8084/api/c/gpu | python3 -m json.tool
echo "=== PeerTube Data ==="
curl -s http://localhost:8084/api/c/peertube | python3 -m json.tool
```
### ⛔ STOP — Report collector status
Tell me:
1. Which collectors show `"online": true`
2. Any errors from the logs or API responses
3. The PeerTube DB name if it wasn't `peertube_prod`
Do not proceed to Phase 3 until collectors are confirmed.
---
## Phase 3: Public access (Caddy + Authentik)
### Check DNS
```bash
dig +short wt.echo6.co
```
If it doesn't resolve, report that — DNS record needs to be added manually.
### Check/deploy Caddy config
Caddy is at 100.64.0.8 on the mesh.
```bash
echo "=== Check existing config ==="
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "cat ~/docker/caddy/sites/wt.echo6.co* 2>/dev/null || echo 'NO CONFIG FOUND'"
echo "=== Check Caddy is running ==="
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "docker ps --format '{{.Names}}' | grep -i caddy"
```
If no config exists, create it:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "cat > ~/docker/caddy/sites/wt.echo6.co.caddy << 'CADDYEOF'
wt.echo6.co {
forward_auth localhost:9000 {
uri /outpost.goauthentik.io/auth/caddy
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid
trusted_proxies private_ranges
}
reverse_proxy 100.64.0.1:8084
}
CADDYEOF"
```
If config already exists, verify the `reverse_proxy` line points to `100.64.0.1:8084` (Contabo's current Tailscale IP). If it still says `100.64.0.6`, fix it:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "sed -i 's/100.64.0.6:8084/100.64.0.1:8084/' ~/docker/caddy/sites/wt.echo6.co.caddy"
```
### Reload Caddy
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
```
### Test
```bash
curl -sI https://wt.echo6.co 2>&1 | head -10
```
Should get 302 redirect to Authentik or 200 if authenticated.
---
## Post-deploy: How updates work
Code is volume-mounted from `/opt/watchtower/app/` into the container on Contabo. To update:
```bash
ssh zvx@100.64.0.1
cd /opt/watchtower
# Edit files or git pull
docker restart watchtower
```
No rebuild needed for code changes. Only rebuild (`docker compose up -d --build`) if `requirements.txt` or `Dockerfile` changes.
## Post-deploy: Adding a new collector
1. Copy `app/collectors/_example.py` to `app/collectors/myservice.py`
2. Edit the class: set `name`, `display_name`, implement `fetch()`
3. Add to `.env`: `MYSERVICE_ENABLED=true` plus any config vars
4. `docker restart watchtower`
The frontend auto-discovers the new panel. No HTML/JS/route edits needed.

406
vault/archive/projects/headscale-full-deployment.md
View file

@ -1,406 +0,0 @@
# Headscale Full Deployment Runbook
## Nodes + Headplane + Authentik OIDC
**Headscale location:** `/opt/headscale-vanilla`
**Container name:** `headscale-vanilla`
**Domain:** `vpn.echo6.co`
**Auth key:** `<REDACTED-headscale-preauth-key>`
---
## PHASE 1: REGISTER CONTABO (must be first)
```bash
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname contabo --force-reauth
```
Verify:
```bash
docker exec headscale-vanilla headscale nodes list
```
**STOP if contabo doesn't appear. Do not continue.**
---
## PHASE 2: REGISTER ALL LXC/CT NODES
SSH into each container. For each one:
```bash
# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"
# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh
```
Then register. **Do them in this exact order for sequential IPs:**
```bash
# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname utility --force-reauth
# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname data --force-reauth
# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname cloud --force-reauth
# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname media --force-reauth
# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
--auth-key <REDACTED-headscale-preauth-key> \
--hostname aida-nebra --force-reauth
```
After each, verify from Contabo:
```bash
docker exec headscale-vanilla headscale nodes list
```
---
## PHASE 3: REGISTER DESKTOP + PHONES
**Desktop (Windows — PowerShell as Admin):**
```powershell
tailscale up --login-server https://vpn.echo6.co `
--auth-key <REDACTED-headscale-preauth-key> `
--hostname desktop --force-reauth
```
**Phones:**
- Open Tailscale app → Settings → Account
- Log out if needed
- Use "Custom coordination server" or "Alternate server"
- Enter: `https://vpn.echo6.co`
- Should auto-register with the tailnet
If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.
---
## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY
```bash
docker exec headscale-vanilla headscale nodes list
```
Expected output: all nodes with sequential 100.64.0.x IPs.
Test from any node:
```bash
tailscale ping contabo
tailscale ping data
tailscale ping utility
```
Test magic DNS:
```bash
ping data.echo6.mesh
ping utility.echo6.mesh
```
---
## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)
```bash
mkdir -p /opt/headscale-vanilla/backups
# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"
# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
```
---
## PHASE 6: PERSISTENCE TEST
```bash
cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list
```
**Every node must survive. If any are missing, STOP and report.**
---
## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE
This lets Tailscale clients authenticate via Authentik instead of preauth keys.
1. Log into Authentik admin panel
2. Go to **Applications → Applications → Create with Provider**
3. Configure:
- **Application name:** Headscale
- **Slug:** `headscale` (remember this — it's part of the issuer URL)
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
- **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
- **Signing key:** Select any available key
- **Scopes:** Ensure these scope mappings are selected:
- `openid`
- `profile`
- `email`
- **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
4. Note the **Client ID** and **Client Secret**
5. Click Submit
---
## PHASE 8: CONFIGURE HEADSCALE OIDC
Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:
```yaml
oidc:
only_start_if_oidc_is_available: true
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
client_id: "<Client ID from Authentik>"
client_secret: "<Client Secret from Authentik>"
scope: ["openid", "profile", "email", "offline_access"]
pkce:
enabled: true
method: S256
strip_email_domain: true
```
Replace:
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
- `<Client ID from Authentik>` with the actual client ID
- `<Client Secret from Authentik>` with the actual client secret
Restart Headscale:
```bash
cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20
```
**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**
Test: From any node, run:
```bash
tailscale up --login-server https://vpn.echo6.co --force-reauth
```
It should open a browser → Authentik login → back to terminal, authenticated.
**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**
---
## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE
This is a SECOND application in Authentik for the web UI login.
1. Go to **Applications → Applications → Create with Provider**
2. Configure:
- **Application name:** Headplane
- **Slug:** `headplane`
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** Same as before
- **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
- **Signing key:** Same key
- **Scopes:** `openid`, `profile`, `email`
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
4. Click Submit
---
## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE
```bash
docker exec headscale-vanilla headscale apikeys create --expiration 999d
```
**Save this key — you need it for the Headplane config.**
---
## PHASE 11: CREATE HEADPLANE CONFIG
```bash
# Generate a cookie secret
openssl rand -hex 16
```
Write `/opt/headscale-vanilla/headplane-config.yaml`:
```yaml
server:
host: "0.0.0.0"
port: 3000
cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
cookie_secure: true
data_path: "/var/lib/headplane"
headscale:
url: "http://headscale-vanilla:8080"
config_path: "/etc/headscale/config.yaml"
config_strict: false
oidc:
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
client_id: "<Headplane Client ID from Authentik>"
client_secret: "<Headplane Client Secret from Authentik>"
token_endpoint_auth_method: "client_secret_post"
headscale_api_key: "<API_KEY_FROM_PHASE_10>"
redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
disable_api_key_login: false
integration:
docker:
enabled: true
container_name: "headscale-vanilla"
socket: "/var/run/docker.sock"
```
Replace all `<PLACEHOLDERS>` with actual values.
---
## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE
Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:
```yaml
services:
headscale:
# ... your existing headscale service, don't change it ...
headplane:
image: ghcr.io/tale/headplane:latest
container_name: headplane
restart: unless-stopped
depends_on:
- headscale
ports:
- "127.0.0.1:3000:3000"
volumes:
- ./headplane-config.yaml:/etc/headplane/config.yaml:ro
- ./headplane-data:/var/lib/headplane
- ./config.yaml:/etc/headscale/config.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
```
Start it:
```bash
cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20
```
Check for errors. Common issues:
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
- Can't connect to headscale → ensure `url` matches the container name and internal port
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock
---
## PHASE 13: UPDATE CADDY FOR HEADPLANE
Add the `/admin` route to your Caddy config for `vpn.echo6.co`:
```
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3000
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
```
Restart Caddy:
```bash
# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```
---
## PHASE 14: TEST HEADPLANE
1. Browse to `https://vpn.echo6.co/admin`
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. **The FIRST user to log in gets Owner permissions**
5. Verify you can see all your nodes in the UI
If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).
---
## PHASE 15: FINAL VERIFICATION
Run all of these from Contabo:
```bash
# All nodes present?
docker exec headscale-vanilla headscale nodes list
# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"
# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302
# Database backed up?
ls -la /opt/headscale-vanilla/backups/
# Cron running?
crontab -l | grep sqlite3
```
---
## REPORT TEMPLATE
After each phase, report:
```
Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):
```
**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**
---
## KNOWN GOTCHAS
1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs

60
vault/archive/projects/last-ember-project.md
View file

@ -1,60 +0,0 @@
# Last Ember — MMUD Web Dashboard
> **CONSOLIDATED:** Last Ember has been merged into the MMUD repo at `src/web/`.
> The standalone repo at `/home/zvx/projects/last-ember` is archived (deprecation notice committed).
Spectator dashboard and admin panel for MMUD (Mesh Multi-User Dungeon). Flask web app that reads from the game's SQLite database. Themed as The Last Ember — the bar that never changes.
## Status
**Phase:** Consolidated into MMUD. No longer a separate project.
## Location
- **Code:** `/home/zvx/projects/mmud/src/web/`
- **Archived repo:** `/home/zvx/projects/last-ember` (deprecated, read-only reference)
## Relationship to MMUD
Last Ember now runs **in-process** with the MMUD mesh daemon as a background daemon thread. Same process, same DB file, WAL mode handles concurrency. Flask starts automatically unless `--no-web` is passed.
## Key Files
- `src/web/__init__.py` — Flask app factory (`create_app`)
- `src/web/config.py` — Web-specific settings (host, port, secret, polling intervals)
- `src/web/routes/` — public.py, api.py, admin.py (session auth)
- `src/web/services/` — gamedb.py, dashboard.py, chronicle.py, admin_service.py
- `src/web/templates/` — Jinja2 templates (dark tavern aesthetic)
- `src/web/static/` — ember.css (design system), embers.js (particles), app.js (AJAX)
- `src/web/prototypes/` — Original HTML design references (visual source of truth)
- `src/db/migrations/004_web_tables.sql` — Web tables (node_config, admin_log, banned_players, npc_journals)
## Stack
- Python 3.11+, Flask 3.x, Jinja2
- SQLite WAL mode (read-only public, read-write admin)
- No build step, no React, no SPA. Server-rendered templates with AJAX polling.
- Docker (python:3.11-slim, /data volume for SQLite)
## CLI
- `--web-port PORT` — override dashboard port (default: 5000)
- `--no-web` — disable web dashboard entirely
- `MMUD_WEB_PORT`, `MMUD_WEB_HOST`, `MMUD_WEB_SECRET`, `MMUD_ADMIN_PASSWORD` env vars
## Pages
**Public (no auth):**
- Main dashboard — live epoch status, leaderboard, broadcasts, bounties, mode status, secrets
- Chronicle — epoch history, NPC daily journals (Grist, Maren, Torval, Whisper)
- How to Play — game guide, command reference
**API (JSON, polled by frontend):**
- `/api/status` (30s), `/api/broadcasts` (15s), `/api/bounties`, `/api/mode`, `/api/leaderboard`
**Admin (session auth):**
- Dashboard — active players, epoch day, node health
- Nodes — assign Meshtastic node IDs to 6 sim node roles
- Players — view, ban, kick, reset
- Epoch — force advance day, force Breach, manual broadcast
- System — DB stats, node config, admin log

270
vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md
View file

@ -1,270 +0,0 @@
---
title: "MeshAI Native Fire-Severity Audit — CC Handoff (archived)"
type: project
tags:
- mesh
status: archived
updated: 2026-07-11
---
> **ARCHIVED / SUPERSEDED — fix deployed.** This was a one-time Claude Code handoff for the native fire-severity regression (audit 2026-07-08). The fix has been deployed; this runbook is retained for history only. Its host/access details are STALE — meshai runs as utility CT 108 (`pct exec 108` from root@utility; zvx has no sudo), NOT at 192.168.1.144 with sudo. Do not execute.
# MeshAI Audit Handoff — Native Fire Severity Regression
**For:** Claude Code (CC), operating from cortex against the MeshAI LXC (192.168.1.144)
**Repo:** zvx-echo6/meshai (deployed via docker compose on the LXC)
**Date of audit:** 2026-07-08, against HEAD `aef9877` (#95)
**Scope:** One minimal backend fix in `work/meshai/env/store.py`, plus a mandatory DB pre-seed. Nothing else. No drive-by refactors.
---
## 0. Standing operational rules (non-negotiable)
1. **SSH prereq gate.** Before ANY work: verify SSH from cortex to the MeshAI LXC (192.168.1.144) using the configured method in `~/.ssh/config`. If auth fails, STOP and report. Never assume SSH works.
2. **Never use `docker compose run`.** Ever. Use `docker exec` against the running container, or host tooling.
3. **Verbatim diffs only.** Apply the diff in §10 exactly. If the anchor lines don't match, STOP and report — do not improvise.
4. **Plain-English explanation before every commit.** Matt's understanding is the gate. Present the summary, wait for his confirmation, then commit.
5. **Every status report back to Matt ends with `sudo docker ps`.**
6. **Rebuild/deploy is ONLY:** `sudo docker compose build meshai && sudo docker compose up -d`.
7. **Mesh-spam stop gate** — see §1. Binding.
---
## 1. ⚠️ MESH-SPAM STOP GATE — BINDING
Deploying the fix in this document **will release a burst of fire broadcasts** unless the pre-seed in §9 is completed and verified first.
Mechanics: every fire currently blocked by this bug retries its "New" broadcast **on every poll** and dies silently at the dispatcher severity floor. Because delivery never succeeds, the decider's commit never runs, so its `fires` row keeps `last_broadcast_at = NULL`. The first poll after the fixed code starts will pass **every one of those fires at once** — and there is no pacer on the native path (`FirePacer` is wired to the Central consumer only, `main.py:128–136`; its own docstring says it serves the Central reconnect/drain path).
**Rule:** Phase −1 (§7) and Phase 0 (§8) are read-only and may run immediately. **Do not execute anything in §9 onward until Matt replies verbatim: `I know and I accept`.** When you reach that point, report the burst size from Phase 0 step D and ask for the ack explicitly. Even with the pre-seed done, note to Matt that legitimate post-fix fire traffic will begin (that's the point), and multiple genuinely-new fires arriving in one poll batch will emit together.
---
## 2. Symptom
Since the central→native swap (Jul 4–6), the mesh carries only WZDx, NWS weather alerts, satpass, and band conditions. Fire alerts — which should be active in southern Idaho in July — are silent. (Avalanche silence is seasonal and expected; it is NOT part of this problem.)
## 3. Root cause — plain English
The central→native swap dropped the decider's severity promotion, and native fire events now die at the dispatcher's `min_severity` floor — silently, with no log line and no drop counter.
Evidence chain (all verified by reading the code at HEAD):
1. **Central path (old behavior):** `central/wfigs_handler.py:238, 348` stamped `data["_severity_override"] = "priority"` on every fire New/Update. `central/consumer.py:649–654` promoted it onto the Event: `severity=sev_override or map_severity(...)`. Every fire reached the dispatcher as **priority**.
2. **Native adapter severity:** `env/fires.py` (~lines 139–148) assigns severity by anchor proximity — `"priority"` only within 25 km of a region anchor, **`"routine"` otherwise**. `to_event` (line 333+) passes it through unchanged.
3. **The decider still stamps the override:** `notifications/gating/fire.py::_broadcast_patch` sets `"_severity_override": "priority"` on every New and Update GateResult. The intent is unchanged from Central.
4. **The promotion is lost:** `env/store.py::_emit_event` (line 678+) builds the Event from `adapter.to_event()` FIRST, then merges the decider patch into **`event.data` only** (`event.data.update(gate.data_patch)`, ~line 731). Nothing anywhere in the native delivery path promotes `_severity_override` onto `event.severity` — verified by grep across dispatcher, composer, and events modules: zero consumers.
5. **The floor:** `notifications/pipeline/dispatcher.py` Section 2 (~line 558): `if event_rank < SEVERITY_RANK[tog.min_severity]: return` — a bare return. **No log, no counter** (the four persisted counters cover cold-start/stale/cooldown/dedup, not the severity floor). The matrix path (Section 1.5) has an equivalent per-cell floor with an "authoritative no-send" return.
6. **The default floor is "priority":** `config.py:641` (`NotificationToggle.min_severity: str = "priority"`) and again at the factory default, `config.py:681`.
Net: routine-severity fire → silent drop, every poll, forever. Fire is forced through the decider regardless of cutover state (`NATIVE_ALWAYS_DECIDE`, `notifications/cutover.py:44`), so every native fire New/Update is affected.
**Timeline:** WFIGS→decider migration `8bc9b14` (Jul 4, 22:51), received-delta gate `0c76a80` (Jul 5), native WFIGS routed through decider `8d61b16` (Jul 6, 15:35). Fire silence began at the native flip — matches the operator's report exactly.
## 4. Why the surviving feeds survive
They handle severity in-adapter, so they never depended on the lost promotion:
- `env/satpass.py:250` — reads `severity=data.get("_severity_override", "routine")` itself. **This is the correct pattern the fix replicates at the store level.**
- `env/nws.py:39` — derives severity in-adapter via `_map_nws_severity` from CAP severity; warnings clear the floor.
- `env/wzdx.py:365` — full closures are `"priority"`.
- Band conditions — scheduled broadcaster (`BandConditionsScheduler`) that calls the dispatcher directly.
Of the six deciders that emit `_severity_override` (fire, nws, quake, swpc, avalanche, firms), only **firms** (`env/firms.py:456`) and **satpass** consume it in-adapter.
## 5. Ruled out during the audit
- Avalanche: out of season; expected silence. Not a bug.
- Received-delta gate (`_delta_emit`, commit `0c76a80`): fires deliberately bypass it (`store.py::_ingest`, `nifc` branch) — the decider is the fire gate. Not the drop point.
- FirePacer black hole: pacer is Central-consumer-only; native fires go straight to the bus. Not a factor.
- Toggle mapping: `wildfire_declared`/`wildfire_incident` both map to the `fire` toggle (`notifications/categories.py:~380–390`). Not a factor.
- Coverage filter: fail-open for non-weather categories; fires carry centroids. Not the primary drop.
- ToggleFilter/inhibitor/grouper: all log their drops; fire family is presumed enabled (verify in Phase 0).
## 6. Secondary findings — explicitly OUT OF SCOPE for this change (backlog)
1. **Restart-window ignition swallowing.** `_fires_seeded` is per-process (`store.py::_ingest_fires`); every restart's first poll silent-seeds any fire that first appeared while the container was down — it is never announced as New. Deliberate anti-backlog design, but during dense deploy periods in fire season it eats real ignitions. Needs a design decision (e.g., persisted seen-baseline), not a hotfix.
2. **Zero-acre New fires deferred.** `env/fires.py::to_event` returns None when `not acres`; fresh WFIGS ignitions often report 0 acres initially. Self-heals when acreage posts (row exists with NULL `last_broadcast_at` → decider fires "New"), but delays first announcements.
3. **Same bug class armed for quake/swpc/avalanche/nws.** Their deciders emit `_severity_override` with no native consumer; currently masked behind `MESHAI_CUTOVER_CATEGORIES` shadow gating. The §10 fix resolves this at the shared choke point for all cut-over categories too.
4. **No native fire pacer.** Multiple legit new fires in one poll batch will emit back-to-back. Consider extending FirePacer to the native emit path later.
---
## 7. Phase −1 — SSH prereq gate (mandatory first step)
From cortex:
```bash
ssh <configured-alias-or-user>@192.168.1.144 'echo SSH_OK && hostname'
```
If this fails for any reason: STOP. Report to Matt. Do not proceed.
Then locate the deployment on the LXC (do not guess paths):
```bash
sudo docker ps --format '{{.Names}}\t{{.Image}}' | grep -i meshai
find /opt /root /home -maxdepth 3 -name docker-compose.yml 2>/dev/null | xargs grep -l meshai 2>/dev/null
```
Record the compose project directory and the container name (assumed `meshai` below — substitute if different). Confirm `git status` in the repo checkout is clean before any edit; if dirty, STOP and report.
## 8. Phase 0 — read-only diagnostics (safe to run now)
Run all of these and report results to Matt before anything else. Determine the DB path first:
```bash
sudo docker exec meshai ls /app/data
```
**A. Are fire events reaching the bus and dying downstream?** (Diagnosis predicts: repeatedly nonzero.)
```bash
sudo docker logs meshai --since 24h 2>&1 | grep -c "Emitted nifc"
sudo docker logs meshai --since 24h 2>&1 | grep "Emitted nifc" | tail -5
```
**B. Live fire family config — the floor:**
```bash
sudo docker exec meshai grep -B2 -A8 -i "fire" /app/config/config.yaml | grep -i -B1 -A4 "min_severity\|enabled\|region"
```
**C. Cutover state (determines blast radius of the bug class):**
```bash
sudo docker exec meshai printenv MESHAI_CUTOVER_CATEGORIES
```
**D. Burst size if fixed today** — this number goes in the stop-gate report to Matt:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;"
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT irwin_id, incident_name, current_acres, current_contained_pct, last_broadcast_at
FROM fires ORDER BY last_event_at DESC LIMIT 20;"
```
**E. Dispatcher drop counters** (severity-floor drops are NOT counted — expected to look quiet; that's part of the finding):
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "SELECT * FROM dispatcher_state;"
```
**Decision matrix:**
- A nonzero AND B shows fire `min_severity: priority` (or matrix fire cells with priority floors) → **diagnosis confirmed.** Proceed to the stop gate.
- A is ZERO → fires aren't reaching the bus at all; the diagnosis does NOT hold as-is. STOP. Report findings; do not apply the fix. (Check adapter enablement, NIFC fetch errors: `sudo docker logs meshai --since 24h 2>&1 | grep -i "nifc\|wfigs" | tail -20`.)
- A nonzero but B shows fire floor already `routine` → drop is elsewhere (matrix cells or region scope). STOP. Report B plus the matrix config; do not apply the fix blind.
End the Phase 0 report with `sudo docker ps`.
---
## ⛔ EVERYTHING BELOW REQUIRES MATT'S VERBATIM ACK: `I know and I accept`
Report the Phase 0 results including the §8-D burst count, restate the risk (§1), and wait.
## 9. Phase 1 — pre-seed the fires table (mandatory BEFORE the fixed code ever starts)
Run against the **still-running old container** (old code keeps suppressing at the floor, so this changes nothing live; after pre-seed the decider simply returns cooldown for those rows):
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "
UPDATE fires
SET last_broadcast_at = strftime('%s','now'),
first_broadcast_at = COALESCE(first_broadcast_at, strftime('%s','now')),
last_broadcast_acres = current_acres,
last_broadcast_contained = current_contained_pct
WHERE last_broadcast_at IS NULL;"
```
Verify — MUST return 0 before proceeding:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;"
```
If `sqlite3` is not present in the image: stop the container, run the same SQL with host `sqlite3` against the bind-mounted data directory (check the volume mapping in docker-compose.yml), verify count 0, and only then continue. Do NOT start the new image before the verify shows 0.
Consequence to state to Matt: currently-active fires will not get a retroactive "New" announcement; they will broadcast on their next real growth/containment change. Only genuinely new post-deploy ignitions announce as New.
## 10. Phase 2 — the fix (verbatim diff)
Target: `work/meshai/env/store.py`, inside `_emit_event` (function starts line 678). Anchor by grepping `event.data.update(gate.data_patch)` — it occurs once. Verify both anchor lines exist verbatim, then apply exactly:
```diff
--- a/work/meshai/env/store.py
+++ b/work/meshai/env/store.py
@@ def _emit_event(self, adapter, raw_evt: dict):
# Apply data_patch into event.data
event.data.update(gate.data_patch)
+ # Promote decider overrides onto the Event itself, mirroring
+ # the Central path (central/consumer.py:649-653). The gating
+ # deciders stamp _severity_override (fire: "priority" on every
+ # New/Update) and, for fire New/tombstone, a category
+ # override. Merging them into event.data alone leaves
+ # event.severity at the adapter's value ("routine" for fires
+ # >=25 km from an anchor), which silently fails the
+ # toggle/matrix min_severity floors (default "priority",
+ # config.py:641). Native and Central must share identical
+ # broadcast decisions.
+ _sev_override = gate.data_patch.get("_severity_override")
+ if _sev_override:
+ event.severity = _sev_override
+ _cat_override = gate.data_patch.get("category")
+ if _cat_override:
+ event.category = _cat_override
if gate.commit is not None:
event.data["_on_broadcast_committed"] = gate.commit
```
Notes for review:
- `Event` is a plain mutable `@dataclass` (`notifications/events.py:38`); attribute assignment is valid.
- Category promotion is intentional and safe: `wildfire_declared` maps to the `fire` toggle (`categories.py:~387`); formatter branch selection already falls back to `data["category"]`; cooldown keys become category-scoped, matching Central behavior (New vs Update cooldowns separate).
- This is the shared choke point: any category later added to `MESHAI_CUTOVER_CATEGORIES` (quake, swpc, avalanche, nws) gets the correct promotion automatically.
**Tests:** if a pytest environment exists on the LXC checkout, run the targeted subsets and compare against the known 10-failure baseline (per commit `0c76a80`, 1697 passed):
```bash
python3 -m pytest work/tests -k "fire or store or dispatch or gating" -q
```
If no local pytest env exists, note that in the report and proceed — the change is 16 lines with fully-traced semantics. Do NOT use `docker compose run` to run tests.
## 11. Phase 3 — commit and deploy
1. Present Matt a plain-English summary of the change (what/why in 3–5 sentences, drawn from §3). Wait for his confirmation. His understanding is the gate.
2. Commit with a message in the repo's convention, e.g.:
`fix(native): promote decider _severity_override/category onto Event — restores Central-parity fire severity (silent min_severity floor drop)`
3. Deploy exactly:
```bash
sudo docker compose build meshai && sudo docker compose up -d
```
## 12. Phase 4 — post-deploy verification
Watch the first 2–3 NIFC poll cycles (read the poll interval from the wfigs/nifc adapter config during Phase 0):
```bash
sudo docker logs -f meshai 2>&1 | grep -Ei "Emitted nifc|matrix: dispatched|dispatched|SUPPRESSED|cold-start"
```
Expected: NO burst at startup (pre-seed verified 0); cold-start seed lines for any brand-new first-sight fires; subsequent genuine New/growth events showing `matrix: dispatched` or toggle-path delivery lines. Confirm on-air on the mesh when the first legit event lands.
Then confirm the counters are stable and report:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "SELECT * FROM dispatcher_state;"
sudo docker ps
```
Every report back to Matt ends with `sudo docker ps`.
## 13. Backlog (do NOT do now — list for future sessions)
1. Persist the fire seen-baseline across restarts to close the restart-window ignition gap (§6.1).
2. Decide policy for zero-acre first sightings (§6.2).
3. Extend pacing to the native emit path for multi-fire poll batches (§6.4).
4. Before cutting over quake/swpc/avalanche/nws via `MESHAI_CUTOVER_CATEGORIES`, re-verify override promotion end-to-end (covered by this fix, but confirm with shadow logs).
5. Add a drop counter + log line for the dispatcher severity-floor return (it is currently the only fully silent drop in the pipeline).

132
vault/archive/projects/openwebui-theme-deploy.md
View file

@ -1,132 +0,0 @@
# RUNBOOK — Deploy Echo6 Theme to Open WebUI
> **STATUS: COMPLETED** — 2026-02-17
> Theme deployed to https://ai.echo6.co. Open WebUI runs in Docker (not native as originally stated).
> Theme files bind-mounted from `/home/zvx/echo6-theme/` into container via docker-compose.yml.
> Compose path: `/opt/open-webui/docker-compose.yml` on cortex.
## OBJECTIVE
Apply the togglable Echo6 custom CSS theme to the Open WebUI instance running at ai.echo6.co. The instance runs in Docker on the `cortex` VM. The theme is toggled on/off via a small "E6" button in the bottom-right corner, with the preference persisted in localStorage per browser.
## THEME FILES
Located in this project's reference assets:
```
.ref/assets/echo6-openwebui-theme.css # Theme styles (activates via .echo6 class on <html>)
.ref/assets/echo6-theme-toggle.js # Toggle button + localStorage persistence
```
## SAFETY FIRST — BACKUP BEFORE ANYTHING
Before making ANY changes:
1. **Find the Open WebUI static build directory.** Likely locations:
- Check: `pip show open-webui 2>/dev/null` for the install path
- Search: `find / -name "index.html" -path "*/open*webui*" 2>/dev/null`
- Search: `find / -name "app.html" -path "*/open*webui*" 2>/dev/null`
- Common pip paths: `/usr/lib/python3/dist-packages/open_webui/static/`
2. Once found, identify the **root HTML file** (likely `index.html` or `app.html`).
3. **Create a timestamped backup:**
```bash
OWUI_DIR="/path/to/open-webui/build" # ← set this once found
BACKUP_DIR="/home/matt/backups/openwebui-theme-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -a "$OWUI_DIR" "$BACKUP_DIR/"
echo "Backed up to: $BACKUP_DIR"
```
4. **Create a revert script** at `/home/matt/revert-openwebui-theme.sh`:
```bash
#!/bin/bash
# Revert Echo6 theme — restores original Open WebUI files
BACKUP_DIR="<populated during deploy>"
OWUI_DIR="<populated during deploy>"
echo "[REVERT] Restoring Open WebUI from $BACKUP_DIR"
cp -a "$BACKUP_DIR"/* "$OWUI_DIR/"
echo "[REVERT] Done. Restart Open WebUI service if needed."
```
Make it executable: `chmod +x /home/matt/revert-openwebui-theme.sh`
## DEPLOYMENT STEPS
### Step 1: Copy theme files to Open WebUI static directory
```bash
cp .ref/assets/echo6-openwebui-theme.css "$OWUI_DIR/static/"
cp .ref/assets/echo6-theme-toggle.js "$OWUI_DIR/static/"
```
Also keep persistent copies that survive upgrades:
```bash
mkdir -p /home/matt/echo6-theme
cp .ref/assets/echo6-openwebui-theme.css /home/matt/echo6-theme/
cp .ref/assets/echo6-theme-toggle.js /home/matt/echo6-theme/
```
### Step 2: Inject into the root HTML
Find the root HTML file and add BOTH a `<link>` and a `<script>` tag BEFORE `</head>`:
```bash
sed -i 's|</head>|<link rel="stylesheet" href="/static/echo6-openwebui-theme.css">\n<script src="/static/echo6-theme-toggle.js" defer></script>\n</head>|' "$OWUI_DIR/index.html"
```
**IMPORTANT:** Check existing `<link>` and `<script>` tags in the HTML first to confirm the `/static/` prefix matches how Open WebUI serves assets. Adjust the path if it uses a different pattern (e.g. `/_app/`, `/build/`, etc.).
### Step 3: Verify
```bash
grep "echo6" "$OWUI_DIR/index.html"
ls -la "$OWUI_DIR/static/echo6-openwebui-theme.css"
ls -la "$OWUI_DIR/static/echo6-theme-toggle.js"
```
Restart the service if needed:
```bash
sudo systemctl restart open-webui # or whatever the service name is
```
### Step 4: Test
- `curl -s https://ai.echo6.co | grep echo6` should show both the CSS and JS references
- Load ai.echo6.co in browser — should see a small "E6" button in the bottom-right corner
- Click it: theme activates (dark bg, cyan accents, JetBrains Mono)
- Click again: reverts to stock Open WebUI appearance
- Refresh page: preference should persist
## HOW THE TOGGLE WORKS
1. The JS injects a fixed-position "E6" button at bottom-right
2. Clicking it toggles the `echo6` class on `<html>`
3. ALL CSS selectors in the theme are `.echo6 <target>` — they ONLY fire when that class is present
4. The toggle state is saved to `localStorage` under the key `echo6-theme-active`
5. On page load, the JS reads localStorage and restores the previous state before first paint
6. Open WebUI's built-in theme picker (Light/Dark/OLED) still works independently
## CONSTRAINTS / DO NOT
- Do NOT modify any Open WebUI Python source code
- Do NOT modify any existing JavaScript files
- Do NOT install additional packages
- Do NOT change Open WebUI configuration/database
- ONLY touch: one HTML file (add one `<link>` + one `<script>` tag) and add two static files
- If anything looks wrong or the build structure is unexpected, STOP and report back
## DEBUGGING
If the theme doesn't apply:
1. Browser dev tools → Network tab — are both files loading? (200 vs 404)
2. Browser dev tools → Elements → check `<html>` — does it have class `echo6` after clicking toggle?
3. Browser dev tools → Console — any JS errors?
4. If selectors don't match the actual DOM, inspect elements and adjust selectors in the CSS
If the toggle button doesn't appear:
1. Check Console for JS errors
2. Verify the `<script>` tag is present in the HTML source
3. Check if the JS file path is correct (404 in Network tab)
## AFTER UPGRADE PROCEDURE
After any Open WebUI upgrade, the build files get overwritten. To re-apply:
```bash
cp /home/matt/echo6-theme/echo6-openwebui-theme.css "$OWUI_DIR/static/"
cp /home/matt/echo6-theme/echo6-theme-toggle.js "$OWUI_DIR/static/"
# Re-inject the tags into the HTML:
sed -i 's|</head>|<link rel="stylesheet" href="/static/echo6-openwebui-theme.css">\n<script src="/static/echo6-theme-toggle.js" defer></script>\n</head>|' "$OWUI_DIR/index.html"
```
Or just re-run this runbook from the top.

797
vault/archive/projects/peertube-phase2-project.md
View file

@ -1,797 +0,0 @@
# Project: PeerTube Phase 2 — Import Pipeline Build
**Goal:** Build a complete YouTube download → local import → GPU transcode pipeline for 99 channels (~70K+ videos, ~15.3TB) on a fresh PeerTube v8 instance. Clean slate — no legacy code, no old pipeline files. Build it right from scratch.
**CC Host:** cortex (SSH to all nodes via aliases in ~/.ssh/config; Proxmox nodes use sshpass auth)
---
## SSH Prerequisites — RUN FIRST
**Every CC session must verify SSH connectivity before executing any remote commands. Never assume SSH works.**
### Verify cortex → CT 110 (PeerTube)
```bash
# CT 110 uses sshpass auth (same as all LXCs). Check ~/.ssh/config for alias.
# Try alias first, fall back to IP:
ssh -o ConnectTimeout=5 peertube 'hostname' 2>/dev/null \
|| sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 zvx@192.168.1.170 'hostname'
```
### Verify cortex → media node (Proxmox host, for pct commands if needed)
```bash
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 root@192.168.1.243 'hostname'
```
### Gate
Both must return hostnames. **Stop and fix SSH before proceeding with ANY step.**
If aliases don't exist in `~/.ssh/config`, add them:
```bash
grep -q "Host peertube$" ~/.ssh/config 2>/dev/null || cat >> ~/.ssh/config << 'EOF'
Host peertube
HostName 192.168.1.170
User zvx
EOF
```
Note: Most pipeline work runs as the `peertube` user inside CT 110. SSH in as zvx, then `sudo -u peertube` or `sudo su - peertube` as needed.
---
## Runbook References
These runbooks live in `~/runbooks/` on cortex. Call them by name when their scope applies:
| Runbook | When to Use in Phase 2 |
|---------|----------------------|
| **`nordvpn-lxc.md`** | **Step 3 — RUN THIS RUNBOOK.** VPN setup on CT 110 with TUN device, NordVPN/WireGuard, split tunneling, rotation script |
| **`peertube-remote-runner.md`** | **ACTIVE — used for video-transcription (Whisper captioning).** Runner on cortex handles auto-captioning with smart GPU/CPU routing. Not used for H.265 video transcoding (pipeline handles that). See runbook for Whisper setup details. |
| `ct-runbook.md` | If CT 110 needs additional packages or baseline changes (provisioned in Phase 1 — reference only) |
| `expose-service-home.md` | stream.echo6.co is already exposed (Phase 1). Reference only if Caddy/DNS/cert issues arise |
| `authentik-oidc-application.md` | PeerTube OIDC already configured (Phase 1). Reference only if SSO breaks |
| `pi-nas-omv-runbook.md` | If NFS storage issues arise (mount problems, permissions, OMV config) |
| `proxmox-onboard-node.md` | SSH access patterns — the Phase 1 prereq pattern above follows this runbook's conventions |
| `proxmox-create-ubuntu-vm.md` | If cortex needs modifications (GPU passthrough, NVIDIA drivers, Docker). Reference only |
**Not applicable to Phase 2:** idahomesh-*, meshmonitor-*, meshtasticd-* runbooks.
---
## Infrastructure (Read-Only Context — Do Not Modify)
### PeerTube Instance
- **CT 110** on **media** node (Proxmox)
- Local IP: 192.168.1.170
- Tailscale IP: 100.64.0.23
- OS: Debian 12, privileged LXC
- PeerTube v8 — **native install** (NOT Docker). No `docker exec` for anything.
- Runs as user: `peertube`
- PostgreSQL: local, accessible via `sudo -u postgres psql peertube_prod` or `sudo -u peertube psql peertube_prod`
- Redis: local
- Nginx: local (port 80), proxied through Caddy on utility node
- Domain: stream.echo6.co
- NFS storage: 18TB from pi-nas (192.168.1.245) mounted at `/var/www/peertube/storage/`
- NFS export path: `/srv/dev-disk-by-uuid-822575b9-1549-4aab-823e-8160d2aa7c68/peertube/`
- PeerTube config: `/var/www/peertube/config/local-production.json` (v8 uses JSON, not YAML)
- PeerTube base dir: `/var/www/peertube/`
- Built-in channel sync: DISABLED (bulk pipeline handles imports)
- Signup: disabled (Authentik SSO only)
### GPU Pre-Transcoding (H.265 via NVENC)
- **cortex** — VM on TOC node, RTX A4000 GPU passthrough
- cortex is also the CC host and runs Ollama/Aurora
- NVENC is separate silicon from CUDA — transcoding won't conflict with LLM inference
- **PeerTube's built-in transcoding is DISABLED** — remote runners ignore transcoding plugins, so there's no way to get H.265 through the runner pipeline
- Instead: a `transcoder.py` service on cortex pulls downloaded videos from CT 110, re-encodes to H.265 with `hevc_nvenc`, pushes back. The importer then uploads already-transcoded files to PeerTube with `waitTranscoding=false`
- Target: H.265, 1080p only, single file per video (no HLS adaptive — LAN/Tailscale viewers don't need it)
- ffmpeg command: `ffmpeg -i input.mp4 -c:v hevc_nvenc -preset medium -cq 28 -c:a aac -b:a 128k output.mp4`
- File transfer: cortex pulls from CT 110 via rsync/SSH, transcodes locally to avoid NFS latency on GPU work, pushes result back
### Runner Service (ACTIVE — video-transcription/captioning)
Runner on cortex handles Whisper auto-captioning. Also registered for VOD transcoding jobs but H.265 video transcoding goes through the pipeline transcoder instead.
```ini
[Unit]
Description=PeerTube Remote Runner (NVENC)
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
Requires=nvidia-persistenced.service
[Service]
Type=simple
User=zvx
Group=zvx
Environment=NODE_ENV=production
Environment=PATH=/opt/peertube-runner/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription
WorkingDirectory=/home/zvx
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube-runner
MemoryMax=20G
[Install]
WantedBy=multi-user.target
```
**Whisper config:** Smart wrapper at `/usr/local/bin/whisper-smart` routes <1hr to GPU (CUDA float16), >=1hr to CPU (int8). CPU jobs serialized via flock. Runner concurrency=2 (1 GPU + 1 CPU in parallel). Model: medium. See `peertube-remote-runner.md` for full details.
### Recovered Runner Health Script
```bash
#!/bin/bash
LOG_TAG="peertube-runner-health"
if ! systemctl is-active --quiet peertube-runner; then
logger -t $LOG_TAG "Runner not active, restarting..."
systemctl restart peertube-runner
sleep 10
fi
if ! pgrep -f "peertube-runner server" > /dev/null; then
logger -t $LOG_TAG "Runner process not found, restarting service..."
systemctl restart peertube-runner
fi
if ! nvidia-smi > /dev/null 2>&1; then
logger -t $LOG_TAG "GPU not accessible, restarting nvidia-persistenced and runner..."
systemctl restart nvidia-persistenced
sleep 5
systemctl restart peertube-runner
fi
```
### SSH / Access
- cortex → CT 110: `ssh peertube` or `ssh root@192.168.1.170` (check ~/.ssh/config)
- cortex → Proxmox nodes: uses sshpass (aliases in ~/.ssh/config)
- CT 110 user for pipeline: `peertube` (same user that runs the PeerTube process)
### VPN
- NordVPN account exists, needs fresh setup on CT 110
- LXC may not support NordVPN CLI (systemd issues) — WireGuard configs as fallback
- Rotation countries: US, CA, UK, DE, NL, SE
- Split tunnel / killswitch off so PeerTube stays accessible locally
---
## Channel Map — The 99 Channels
### Recovered Schema (from old WATCHTOWER add_channel.py)
```json
{
"category": "Tactical/SUT",
"channel_name": "(YT)Garand Thumb",
"actor_name": "garand-thumb",
"youtube_url": "https://www.youtube.com/@GarandThumb",
"youtube_channel_id": null,
"peertube_channel_id": null,
"video_count": 0,
"priority": "H",
"est_videos": 500,
"est_gb": 98
}
```
### Recovered Slug Function
```python
import re
def slugify_channel(name):
"""Convert channel name to PeerTube-safe actor_name."""
name = re.sub(r'^\(YT\)\s*', '', name)
slug = re.sub(r'[^a-z0-9]+', '-', name.lower()).strip('-')
return slug[:50] or 'channel'
```
### Known YouTube URLs (from old PeerTube sync records — 24 channels)
These 24 channels had active sync records with confirmed YouTube URLs:
```
Essential Craftsman → @essentialcraftsman
CommsPrepper → @CommsPrepper
Steven Lavimoniere → @StevenLavimoniere
Andreas Spiess → @AndreasSpiess
Mustie1 → @mustie1
Donyboy73 → @Donyboy73
Turn a Wood Bowl → @TurnaWoodBowl
RoseRed Homestead → @RoseRedHomestead
Homesteading Family → @HomesteadingFamily
My Self Reliance → @MySelfReliance
RegisteredNurseRN → @RegisteredNurseRN
Skinny Medic → @SkinnyMedic
Marine X → @MarineX
Plumberparts → @plumberparts
MedCram → @Medcram
City Prepping → @CityPrepping
Paul Kirtley → @PaulKirtley
Armando Hasudungan → playlist?list=UUesNt4_Z-Pm41RzpAClfVcg
Self Sufficient Me → @Selfsufficientme
Taryl Fixes All → @TarylFixesAll
Engineer775 → @engineer775
WeberAuto → @WeberAuto
Sun Knudsen → @sunknudsen
Master Your Medics → @MasterYourMedics
MCQBushcraft → @MCQBushcraft
ChrisFix → @ChrisFix
```
### The 99 Channels (Finalized Feb 2026)
#### OPSEC / Privacy (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Michael Bazzell / IntelTechniques | H | OSINT + digital privacy, ex-FBI |
| The Hated One | H | Privacy advocacy, surveillance deep-dives |
| Mental Outlaw | H | Linux + privacy + infosec news |
| Naomi Brockwell TV | M | Privacy-focused tech |
| Techlore | M | Privacy tools and comparisons |
| Sun Knudsen | M | Step-by-step privacy hardening |
#### Physical Security (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Deviant Ollam | H | Physical penetration testing, lock bypass |
| BosnianBill | M | Lock picking, physical security analysis |
#### Intelligence / OSINT (4)
| Channel | Priority | Notes |
|---------|----------|-------|
| OSINT Dojo | H | OSINT methodology training |
| Benjamin Strick | H | Professional OSINT investigations |
| OSINT Curious | M | OSINT tools and techniques |
| S2 Underground | H | Threat intel, analysis tradecraft |
#### Cybersecurity (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| John Hammond | H | CTF walkthroughs, malware analysis |
| IppSec | H | HackTheBox walkthroughs |
| LiveOverflow | H | Binary exploitation, web security |
| Professor Messer | M | CompTIA certification training |
| The Cyber Mentor | M | Ethical hacking courses |
| Hak5 | M | Hacking tools and techniques |
| David Bombal | M | Networking + cybersecurity |
#### Tactical / SUT (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Garand Thumb | H | Tactics, gear testing, NV |
| Dirty Civilian | H | SUT for civilians |
| One Shepherd | H | Former SOF, tactical training |
| Brent0331 | H | USMC veteran, tactical analysis |
| Brass Facts | M | Firearms philosophy, gear testing |
| Sage Dynamics | H | Research-based torture tests |
#### Firearms (8)
| Channel | Priority | Notes |
|---------|----------|-------|
| Forgotten Weapons | H | Historical + technical firearms (largest channel, ~3K videos) |
| Paul Harrell | H | Terminal ballistics, practical shooting |
| 9-Hole Reviews | M | Precision rifle, historical accuracy |
| Lucky Gunner | M | Ammo testing, concealed carry |
| C&Rsenal | M | WWI/WWII firearms deep-dives |
| Jerry Miculek | M | Speed shooting, competition |
| InRangeTV | M | Firearms + mud tests |
| Hickok45 | M | Reviews + shooting demonstrations |
#### Comms / Signals (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| OH8STN | H | Off-grid digital comms, Winlink |
| Andreas Spiess | H | Electronics + LoRa + radio |
| Ham Radio Crash Course | H | Amateur radio training |
| Tech Minds | M | SDR, radio tech |
| The Comms Channel | M | Comms gear and planning |
| KM4ACK | H | Build-a-Pi, ham radio software |
| Signals Everywhere | M | SDR + spectrum analysis |
#### Medical (5)
| Channel | Priority | Notes |
|---------|----------|-------|
| PrepMedic | H | Flight paramedic, trauma care |
| Skinny Medic | H | IFAK, trauma kits |
| MedWild | H | Wilderness medicine |
| Crisis Medicine | H | Former 18D SF Medic, TCCC |
| Ninja Nerd | H | Comprehensive physiology/pathology |
#### Linux / Infrastructure (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Lawrence Systems | H | Enterprise networking + Linux |
| Learn Linux TV | H | Linux tutorials and homelab |
| Jeff Geerling | H | Raspberry Pi, Ansible, self-hosting |
| Techno Tim | M | Homelab, Docker, Kubernetes |
| Level1Techs | M | Hardware + Linux deep-dives |
| Wolfgang's Channel | M | Self-hosting, privacy infra |
#### Hardware / Electronics (4)
| Channel | Priority | Notes |
|---------|----------|-------|
| Ben Eater | H | Computer architecture from scratch |
| EEVblog | H | Electronics engineering |
| GreatScott! | M | Electronics projects |
| Big Clive | M | Electronics teardowns |
#### Auto / Mechanical (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| ChrisFix | H | DIY auto repair fundamentals |
| Mustie1 | H | Dead machinery resurrection |
| South Main Auto | H | Diagnostic logic |
| 1A Auto | H | Make/model/year repair encyclopedia (~4,500 videos) |
| Pine Hollow Auto Diagnostics | M | Advanced diagnostics |
| ScannerDanner | M | Master electrical diagnostics |
| Diesel Creek | M | Heavy equipment repair |
#### Construction / Trades (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| Essential Craftsman | H | Construction + life skills |
| Matt Risinger | H | Building science |
| Mike Haduck Masonry | M | Foundations, concrete, stone |
| Awesome Framers | M | Structural framing |
| This Old House | M | Home renovation |
| Electrician U | M | Electrical trade training |
| Got2Learn | M | Plumbing/electrical tutorials |
#### Welding / Fabrication (3)
| Channel | Priority | Notes |
|---------|----------|-------|
| Welding Tips and Tricks | H | Welding instruction |
| ChuckE2009 | M | Welding + fabrication |
| Paul Sellers | H | Hand tool woodworking master |
#### Sustainment / Fieldcraft (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Corporals Corner | H | Field skills, shelter, fire |
| Gray Bearded Green Beret | H | SF wilderness medicine + fieldcraft |
#### Homesteading / Production (8)
| Channel | Priority | Notes |
|---------|----------|-------|
| City Prepping | H | Urban/suburban preparedness |
| My Self Reliance | H | Off-grid building |
| Engineer775 | H | Off-grid power systems |
| Project Farm | H | Tool and product testing |
| Will Prowse / DIY Solar Power | H | Solar power systems |
| Townsends | M | 18th century skills + cooking |
| RoseRed Homestead | M | Homesteading skills |
| The Urban Prepper | M | Urban preparedness, modular bags |
#### Preparedness (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| The Provident Prepper | M | Preparedness planning methodology |
#### Energy / Alt-Fuel (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Adeptus Beta | M | Wood gasification (~7GB, tiny) |
#### Education / STEM (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Practical Engineering | H | Civil engineering with demos |
| Real Engineering | M | Aerospace, energy, transport |
| The Efficient Engineer | M | Core engineering fundamentals |
| NurdRage | M | Chemistry experiments |
| NileRed | M | Chemistry deep-dives |
| Veritasium | M | Science + engineering |
#### Education / Math (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Professor Leonard | H | Full calculus + stats lectures |
| Organic Chemistry Tutor | M | Math + science tutorials |
#### Education / CS (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Computerphile | H | Crypto, networking theory, security concepts |
| MIT Missing Semester | M | Shell, git, dev tools (tiny, ~50 videos) |
#### Small Engine (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Donyboy73 | M | Small engine repair |
#### Woodworking (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Steve Ramsey | M | Beginner woodworking |
#### Home Repair (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Home RenoVision DIY | M | Home repair tutorials |
| Roger Wakefield | M | Plumbing |
#### Bushcraft (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Joe Robinet | M | Bushcraft and camping |
**Total: 99 channels across 20 categories**
---
## Execution Steps
### Step 1: Channel Map Generation
**Where:** CT 110
**What:** Build `/opt/bulk-import/config/channel-map.json`
**SSH Gate:** `ssh peertube 'hostname'` must succeed before proceeding.
1. Create directory structure:
```bash
# Scripts and config on local disk
mkdir -p /opt/bulk-import/{config,logs}
chown -R peertube:peertube /opt/bulk-import
# Video data on NFS (18TB pi-nas mount)
mkdir -p /var/www/peertube/storage/pipeline/{staging,completed,transcoded,failed}
chown -R peertube:peertube /var/www/peertube/storage/pipeline
# Symlink data dirs so scripts use /opt/bulk-import/ paths
ln -sfn /var/www/peertube/storage/pipeline/staging /opt/bulk-import/staging
ln -sfn /var/www/peertube/storage/pipeline/completed /opt/bulk-import/completed
ln -sfn /var/www/peertube/storage/pipeline/transcoded /opt/bulk-import/transcoded
ln -sfn /var/www/peertube/storage/pipeline/failed /opt/bulk-import/failed
```
2. For each of the 99 channels:
- Look up the actual YouTube channel URL (use `yt-dlp --print channel_url --playlist-items 1 --skip-download "https://www.youtube.com/@ChannelHandle"` for any that need verification)
- Generate `actor_name` via slugify
- Write to channel-map.json
3. Use the 24 known URLs from old sync records as a head start. The remaining 75 need URL resolution.
**⚠️ This step requires yt-dlp installed and working on CT 110. If yt-dlp isn't installed yet, install it first:**
```bash
curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
chmod +x /usr/local/bin/yt-dlp
```
**⚠️ YouTube may rate-limit channel lookups. Space requests 2-3 seconds apart. If rate-limited, use cookies or VPN.**
### Step 2: PeerTube Channel Creation
**Where:** CT 110
**What:** Batch-create all 99 channels via PeerTube API
**SSH Gate:** `ssh peertube 'curl -s http://localhost:9000/api/v1/config | head -c 50'` — must return JSON. Confirms both SSH and PeerTube are up.
1. Get OAuth token from PeerTube API (local, port 9000):
```bash
# Get client credentials
curl -s http://localhost:9000/api/v1/oauth-clients/local -H "Host: stream.echo6.co"
# Get user token
curl -s http://localhost:9000/api/v1/users/token \
-H "Host: stream.echo6.co" \
--data "client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>&grant_type=password&username=root&password=<PASSWORD>"
```
2. For each channel in channel-map.json:
```bash
curl -s -X POST http://localhost:9000/api/v1/video-channels \
-H "Host: stream.echo6.co" \
-H "Authorization: Bearer <TOKEN>" \
-H "Content-Type: application/json" \
-d '{"name": "<actor_name>", "displayName": "(YT)<channel_name>", "description": "Imported from YouTube: <youtube_url>"}'
```
3. Capture the returned channel ID and update `peertube_channel_id` in channel-map.json
4. Verify: `curl -s http://localhost:9000/api/v1/video-channels -H "Host: stream.echo6.co" | python3 -m json.tool | grep -c '"name"'` should return 99 (plus the default channel)
### Step 3: NordVPN Setup
**Where:** CT 110
**What:** Install VPN for IP rotation during YouTube downloads
**SSH Gate:** `ssh peertube 'hostname'` must succeed.
**➡️ RUN RUNBOOK: `~/runbooks/nordvpn-lxc.md`**
Use these inputs:
```
CTID=110
CT_HOST=peertube
PVE_HOST=media # or root@192.168.1.243
NORDVPN_TOKEN= # ⚠️ Get from Matt
VPN_COUNTRIES="United_States,Canada,United_Kingdom,Germany,Netherlands,Sweden"
VPN_CONFIG_DIR=/opt/bulk-import/config/vpn
```
**Additional context for this deployment:**
- CT 110 runs PeerTube on port 9000 — split tunneling is MANDATORY so PeerTube stays reachable on 192.168.1.170 and 100.64.0.23 while VPN is active
- The rotation script at `/opt/bulk-import/config/vpn/vpn-rotate.sh` will be called by `downloader.py` (Step 5) on rate-limit detection
- After runbook completes, verify PeerTube still accessible: `curl -s http://192.168.1.170:9000/api/v1/config | head -c 50` (from another machine, while VPN is up on CT 110)
**⚠️ NordVPN token required from Matt. Cannot proceed without it.**
### Step 4: YouTube Cookies
**Where:** CT 110
**What:** Export browser cookies for yt-dlp bot detection bypass
1. Matt exports cookies from browser (Netscape format) using "Get cookies.txt LOCALLY" extension
2. SCP to CT 110: `scp cookies.txt root@192.168.1.170:/opt/bulk-import/config/cookies.txt`
3. Fix perms: `chown peertube:peertube /opt/bulk-import/config/cookies.txt && chmod 600 /opt/bulk-import/config/cookies.txt`
4. Test: `sudo -u peertube yt-dlp --cookies /opt/bulk-import/config/cookies.txt --simulate "https://www.youtube.com/watch?v=dQw4w9WgXcQ"`
**⚠️ Cookies expire every 2-4 weeks. Needs manual refresh.**
### Step 5: Build downloader.py
**Where:** CT 110 at `/opt/bulk-import/downloader.py`
**What:** Round-robin YouTube channel downloader with VPN rotation
**Deploy:** Write file locally on cortex, then `scp` to CT 110. Or write directly via `ssh peertube 'cat > /opt/bulk-import/downloader.py << "PYEOF" ... PYEOF'`
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/config/channel-map.json'` — channel map must exist (Step 1 complete).
Requirements:
- Round-robin across all 99 channels (don't hammer one channel)
- yt-dlp with: `--cookies`, `--download-archive downloaded.txt` (dedup), `--write-info-json`, `--write-thumbnail`, `--format "bestvideo[height<=1080]+bestaudio/best[height<=1080]"`, `--merge-output-format mp4`
- Downloads land in `/opt/bulk-import/staging/<actor_name>/<video_id>/` with .mp4 + .info.json + .jpg
- On successful download, move to `/opt/bulk-import/completed/<actor_name>/<video_id>/`
- **Note:** transcoder.py (Step 6) picks up from completed/ — downloader does NOT feed importer directly
- VPN rotation: detect rate-limit (HTTP 429, sign-in required, bot detection), disconnect current VPN, connect to next country in rotation list, retry
- State file: `/opt/bulk-import/config/downloader-state.json` — tracks current channel index, current VPN country, last activity timestamp
- Logging to `/opt/bulk-import/logs/downloader.log` — include `=== Channel: <name> ===` markers (WATCHTOWER parses these)
- Target throughput: ~30 videos/hr
- Graceful shutdown on SIGTERM/SIGINT
### Step 6: Build transcoder.py
**Where:** cortex (local — this IS the CC host) at `/opt/bulk-import/transcoder.py`
**What:** Pulls H.264 videos from CT 110, re-encodes to H.265 via NVENC, pushes back
**Connectivity Gate:**
```bash
nvidia-smi > /dev/null 2>&1 && echo "GPU OK" || echo "GPU MISSING"
ffmpeg -encoders 2>/dev/null | grep -q hevc_nvenc && echo "HEVC NVENC OK" || echo "HEVC NVENC MISSING"
ssh peertube 'ls /opt/bulk-import/completed/' > /dev/null 2>&1 && echo "SSH OK" || echo "SSH FAIL"
```
Requirements:
- Watch CT 110's `/opt/bulk-import/completed/` for new video directories (via SSH/rsync polling, not inotify — it's remote)
- For each video dir found:
1. `rsync` the dir from CT 110 to cortex local temp: `/opt/bulk-import/transcode-work/<actor_name>/<video_id>/`
2. Run ffmpeg: `ffmpeg -hwaccel cuda -i input.mp4 -c:v hevc_nvenc -preset medium -cq 28 -tag:v hvc1 -c:a aac -b:a 128k output.mp4`
- `-cq 28` = constant quality mode (NVENC equivalent of CRF)
- `-tag:v hvc1` = Apple/browser compatible HEVC tag
- `-preset medium` = balance speed/quality (can tune later)
- Preserve .info.json and .jpg (just copy, don't re-encode)
3. `rsync` the transcoded dir back to CT 110: `/opt/bulk-import/transcoded/<actor_name>/<video_id>/`
4. Remove the source from CT 110's `completed/` dir (it's been transcoded)
5. Clean up local temp
- Skip videos that already exist in `transcoded/`
- Logging to `/opt/bulk-import/logs/transcoder.log` on cortex (and/or stream to CT 110)
- State file: `/opt/bulk-import/config/transcoder-state.json` on cortex
- Graceful shutdown on SIGTERM/SIGINT — finish current transcode, don't start new ones
- Target throughput: depends on video length, but NVENC should handle ~2-5 videos/hr for typical 10-20min content at 1080p
- One video at a time (NVENC session limit on A4000)
**Directory structure on cortex:**
```
/opt/bulk-import/ ← transcoder home on cortex
├── transcoder.py
├── config/
│ └── transcoder-state.json
├── logs/
│ └── transcoder.log
└── transcode-work/ ← temp working dir, cleaned after each video
```
**ffmpeg must be installed on cortex with NVENC support:**
```bash
sudo apt install -y ffmpeg
ffmpeg -encoders 2>/dev/null | grep hevc_nvenc # must show hevc_nvenc
# If missing: sudo apt install -y libnvidia-encode-550 (match driver version)
```
### Step 7: Build importer.py
**Where:** CT 110 at `/opt/bulk-import/importer.py`
**What:** Watches transcoded/ dir, uploads to PeerTube via API
**Deploy:** Same as Step 5 — write locally, scp to CT 110.
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/config/channel-map.json && curl -s http://localhost:9000/api/v1/config | head -c 50'` — channel map AND PeerTube API must be reachable.
Requirements:
- Watch `/opt/bulk-import/transcoded/` for new video directories (NOT completed/ — transcoder feeds this)
- For each video dir: read .info.json, extract title, description, upload_date (→ originallyPublishedAt), tags, thumbnail
- Map `<actor_name>` from dir path → `peertube_channel_id` from channel-map.json
- Upload via PeerTube API: `POST /api/v1/videos/upload` with multipart form data
- Set: name, description, channelId, originallyPublishedAt, tags (first 5), thumbnailfile, privacy (1=public), **waitTranscoding=false** (video is already H.265, no PeerTube transcoding needed)
- On success: **DELETE the video dir from `transcoded/`** — PeerTube's storage is the authoritative copy. No `imported/` directory.
- On failure: move to `/opt/bulk-import/failed/` with error log
- Rate: process one video at a time, ~50/hr max (don't overwhelm PeerTube)
- Dedup: check if video title + channel already exists before uploading
- Logging to `/opt/bulk-import/logs/importer.log`
- OAuth token management: cache token, refresh on 401
### Step 8: Systemd Services
**Where:** CT 110 (downloader + importer) AND cortex (transcoder)
**What:** Service files for all three pipeline components
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/downloader.py /opt/bulk-import/importer.py'` — both CT 110 scripts must exist (Steps 5 and 7 complete). `/opt/bulk-import/transcoder.py` must exist on cortex (Step 6 complete).
**On CT 110:**
```bash
# /etc/systemd/system/pt-downloader.service
[Unit]
Description=PeerTube Bulk Downloader
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=peertube
Group=peertube
ExecStart=/usr/bin/python3 /opt/bulk-import/downloader.py
WorkingDirectory=/opt/bulk-import
Restart=always
RestartSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pt-downloader
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/pt-importer.service — same pattern, ExecStart points to importer.py
```
**On cortex:**
```bash
# /etc/systemd/system/pt-transcoder.service
[Unit]
Description=PeerTube H.265 NVENC Transcoder
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
[Service]
Type=simple
User=zvx
Group=zvx
ExecStart=/usr/bin/python3 /opt/bulk-import/transcoder.py
WorkingDirectory=/opt/bulk-import
Restart=always
RestartSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pt-transcoder
MemoryMax=12G
[Install]
WantedBy=multi-user.target
```
Enable but **do not start** until testing is complete.
### Step 9: PeerTube Transcoding Config — DISABLED
**Where:** CT 110
**What:** Disable PeerTube's built-in transcoding — videos arrive pre-transcoded as H.265
**SSH Gate:** `ssh peertube 'hostname'` must succeed.
Edit `/var/www/peertube/config/local-production.json`:
```json
{
"transcoding": {
"enabled": false
},
"import": {
"videos": {
"concurrency": 4,
"http": { "enabled": true },
"torrent": { "enabled": false }
}
},
"video_channel_synchronization": {
"enabled": false
}
}
```
Restart PeerTube after config changes: `sudo systemctl restart peertube`
**Why disabled:** Videos are pre-transcoded to H.265 by cortex (Step 6) before import. The importer uploads with `waitTranscoding=false`. PeerTube serves the file as-is. No runner needed, no re-encode, no wasted cycles.
### Step 10: Integration Test
**Full connectivity gate — ALL must pass:**
```bash
ssh peertube 'hostname' # SSH to CT 110
ssh peertube 'curl -s http://localhost:9000/api/v1/config | head -c 50' # PeerTube API
ssh peertube 'systemctl is-active peertube' # PeerTube service
nvidia-smi > /dev/null 2>&1 && echo "GPU OK" # cortex GPU
ffmpeg -encoders 2>/dev/null | grep -q hevc_nvenc && echo "NVENC OK" # HEVC encoder
ssh peertube 'ls /opt/bulk-import/completed/' > /dev/null && echo "Dirs OK" # Pipeline dirs
```
1. Start downloader — let it grab 5-10 videos from 2-3 different channels
2. Verify videos land in `/opt/bulk-import/completed/` with .mp4 + .info.json + .jpg
3. Start transcoder on cortex — verify it pulls videos, encodes H.265 via NVENC (`nvidia-smi` shows encoder utilization)
4. Verify transcoded files land in `/opt/bulk-import/transcoded/` on CT 110, and originals cleared from `completed/`
5. Verify transcoded file is H.265: `ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 <file>` should return `hevc`
6. Start importer — verify videos appear in PeerTube UI with correct metadata, channel assignment, thumbnails
7. Verify playback works at stream.echo6.co (H.265 plays natively in modern browsers via HLS/web-video)
8. Check dedup — restart downloader, verify it skips already-downloaded videos
9. Check VPN rotation — trigger a rate limit (or simulate), verify country switches
### Step 11: Go-Live
**On CT 110:**
```bash
systemctl start pt-downloader && systemctl start pt-importer
systemctl enable pt-downloader && systemctl enable pt-importer
```
**On cortex:**
```bash
systemctl start pt-transcoder
systemctl enable pt-transcoder
```
Monitor for 24 hours. Expected steady-state:
- Downloader: ~30 videos/hr
- Transcoder: ~2-5 videos/hr (bottleneck — NVENC is fast but 1080p H.265 takes time per video)
- Importer: keeps up with transcoder output, ~50/hr capacity but paced by transcoder
- GPU utilization: 80-100% encoder, minimal CUDA (no conflict with Ollama)
**⚠️ The transcoder is the bottleneck.** At ~3 videos/hr average, 70K videos = ~970 days. Strategies to accelerate:
- Lower quality preset: `-preset fast` or `-preset hp` (speed over quality)
- Accept lower CQ: `-cq 32` instead of 28 (smaller files, slightly lower quality)
- Run 2 NVENC sessions in parallel (A4000 supports ~3 concurrent)
- Add a second GPU node
- Accept H.264 for bulk and only H.265 for new imports going forward
---
## Manual Inputs Required (Before CC Can Execute)
| Item | Who | When Needed |
|------|-----|-------------|
| NordVPN token | Matt | Step 3 |
| YouTube cookies.txt | Matt | Step 4 |
| PeerTube admin password | Matt | Step 2 (OAuth) |
---
## Dependencies Between Steps
```
Step 1 (channel map) ──→ Step 2 (create channels) ──→ Step 7 (importer needs channel IDs)
↗
Step 3 (VPN) + Step 4 (cookies) ──→ Step 5 (downloader) ──→ Step 6 (transcoder reads completed/)
↓
Step 7 (importer reads transcoded/)
Step 9 (disable PT transcoding) ←── independent, do anytime before Step 10
Step 10 (integration test) ←── requires ALL of 1-9
Step 11 (go-live) ←── requires Step 10 pass
```
Steps 1-2 and Step 9 are independent workstreams. Steps 3-4 require Matt's manual input. Steps 5, 6, 7 are the three core scripts. Step 6 runs on cortex; everything else runs on CT 110.
---
## What NOT to Build (Phase 3 — WATCHTOWER)
WATCHTOWER (the monitoring dashboard) is Phase 3. Don't build it now. The pipeline scripts should have enough logging that we can monitor via `journalctl` and log files during Phase 2. WATCHTOWER will eventually:
- SSH into CT 110 to read pipeline metrics (but CT 110 is native now, not Docker — queries change)
- Point to cortex instead of old TOC for GPU stats
- Read channel-map.json from `/opt/bulk-import/config/` instead of old `/mnt/data/bulk-import/`
- Need new .env config for all changed IPs
But that's later. Pipeline first.
---
## Channel Management (via RECON Dashboard)
**Added 2026-02-18.** Channel management UI is now in the RECON dashboard Upload tab at `http://192.168.1.130:8420/upload`. No more SSH + manual JSON editing to add channels.
- **Sudoers:** `/etc/sudoers.d/recon-mgmt` on CT 110 — allows zvx to run yt-dlp, psql, and tee as peertube
- **API endpoints** in `/opt/recon/lib/api.py`:
- `GET /api/peertube/channels` — list all channels with video counts from PeerTube DB
- `GET /api/peertube/channels/stats` — total channels, total videos, downloader status
- `POST /api/peertube/channels/add` — resolve YT URL via yt-dlp, create PeerTube channel, update channel-map.json
- `DELETE /api/peertube/channels/<actor_name>` — remove from JSON and PeerTube
- **UI features:** stats bar, add form (URL + category + priority), sortable channel table, remove button
- **All operations go through SSH from CT 130 → CT 110** using the existing `_ssh_peertube()` helper

468
vault/archive/projects/peertube-rebuild.md
View file

@ -1,468 +0,0 @@
# Project: PeerTube YouTube Archive Rebuild
**Goal:** Rebuild PeerTube at `stream.echo6.co` with Authentik SSO, 18TB NFS storage, and a bulk import pipeline for 250 YouTube channels (~136K videos).
**Status:** Phase 1 — Complete (2026-02-13). CT 110 on media, 192.168.1.170, TS 100.64.0.23, PeerTube v8.0.2
---
## Architecture
```
┌─────────────────────────────────┐
│ utility node │
Internet ──── DNS ──────▶│ Caddy LXC (CT 101) │
│ stream.echo6.co → PT LXC:80 │
└──────────────┬──────────────────┘
│
┌──────────────▼──────────────────┐
│ media node │
│ PeerTube LXC (CT 100) │
│ ├── nginx (port 80) │
│ ├── PeerTube (port 9000) │
│ ├── PostgreSQL 16 │
│ ├── Redis │
│ └── /var/www/peertube/storage │
│ └── NFS mount (18TB) │
└──────────────────────────────────┘
Authentik ◄──── OIDC ────► PeerTube
Phase 2: cortex (VM 150 on TOC, has GPU) = remote transcoding runner
```
**Key decisions:**
- LXC on media node (not VM, not Docker) — CT 100
- Privileged container (NFS bind-mount uid mapping is hell otherwise)
- Native PeerTube install (Node.js + PostgreSQL + Redis + nginx, no Docker)
- PeerTube v8.0.2 — config and nginx template may differ from v6.x docs; verify during install
- Node.js 20 (v8 requirement, not 18)
- Caddy on utility handles TLS, nginx inside LXC handles WebSocket/static files
- Caddy proxies to local IP (192.168.1.x) since PeerTube has OIDC
- Transcoding: 480p + 720p only (storage budget)
- Built-in channel sync DISABLED — bulk pipeline handles imports
- Signup disabled — Authentik SSO only
- NFS storage from pi-nas `/export/peertube` (separate from arr)
---
## Phase 1: PeerTube Up and Secure
### 1.1 Provision LXC on media
**Run:** `runbooks/ct-runbook.md` with these inputs:
| Variable | Value |
|----------|-------|
| Host | media (192.168.1.243) |
| CTID | 100 |
| Hostname | peertube |
| Template | Debian 12 (not Ubuntu — PeerTube docs target Debian) |
| Memory | 4096 MB |
| Cores | 4 |
| Disk | 50 GB root |
| Privileged | YES (override ct-runbook default) |
| Network | DHCP initially |
**Deviations from ct-runbook:**
- Use **Debian 12** template instead of Ubuntu 24.04
- Use **privileged** container (`--unprivileged 0`) for NFS compatibility
- Skip Docker install — PeerTube runs native
- Still do: base packages, zvx user, SSH, Tailscale
### 1.2 Mount NFS storage
On the **media host** (not inside LXC):
```bash
# Mount NFS on host
mkdir -p /mnt/peertube-storage
mount -t nfs 192.168.1.245:/export/peertube /mnt/peertube-storage
# Persist
echo "192.168.1.245:/export/peertube /mnt/peertube-storage nfs defaults,_netdev 0 0" >> /etc/fstab
# Bind-mount into LXC
echo "mp0: /mnt/peertube-storage,mp=/var/www/peertube/storage" >> /etc/pve/lxc/<CTID>.conf
# Restart LXC to pick up mount
pct stop <CTID> && pct start <CTID>
```
**Verify inside LXC:**
```bash
df -h /var/www/peertube/storage # Should show ~18TB
touch /var/www/peertube/storage/test && rm /var/www/peertube/storage/test
```
**NFS details:**
- Server: pi-nas (192.168.1.245 / 100.64.0.21)
- Export: `/export/peertube`
- Access: Already configured in OMV for 100.64.0.0/10 and 192.168.1.0/24
### 1.3 Install PeerTube dependencies
Inside the LXC:
```bash
# PostgreSQL 16
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
apt update && apt install -y postgresql-16 postgresql-contrib-16
sudo -u postgres psql << 'SQL'
CREATE USER peertube WITH PASSWORD '<PG_PASSWORD>';
CREATE DATABASE peertube_prod OWNER peertube;
\c peertube_prod
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;
SQL
# Redis
apt install -y redis-server
sed -i 's/^# requirepass .*/requirepass <REDIS_PASSWORD>/' /etc/redis/redis.conf
sed -i 's/^bind .*/bind 127.0.0.1 -::1/' /etc/redis/redis.conf
systemctl restart redis-server && systemctl enable redis-server
# Node.js 20 (PeerTube v8 requires Node.js 20+)
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs
npm install -g yarn
# ffmpeg (for transcoding)
apt install -y ffmpeg
```
### 1.4 Install PeerTube
```bash
# Create peertube user
adduser --system --group --home /var/www/peertube --shell /bin/bash peertube
chown -R peertube:peertube /var/www/peertube/storage
# Get latest version (joinpeertube.org API is dead, use GitHub)
PEERTUBE_VERSION=$(curl -s https://api.github.com/repos/Chocobozzz/PeerTube/releases/latest | grep -oP '"tag_name": "v\K[^"]+' || echo "8.0.2")
# Download and install
cd /var/www/peertube
sudo -u peertube mkdir -p config
sudo -u peertube mkdir -p storage/{avatars,caches,captions,logs,plugins,previews,redundancy,streaming-playlists,thumbnails,tmp,torrents,videos,bin,storyboards,web-videos,original-video-files}
sudo -u peertube wget -q "https://github.com/Chocobozzz/PeerTube/releases/download/v${PEERTUBE_VERSION}/peertube-v${PEERTUBE_VERSION}.tar.xz"
sudo -u peertube tar xf peertube-v${PEERTUBE_VERSION}.tar.xz
sudo -u peertube ln -s peertube-v${PEERTUBE_VERSION} peertube-latest
cd peertube-latest
sudo -u peertube yarn install --production --pure-lockfile
```
### 1.5 Configure PeerTube
Create `/var/www/peertube/config/local-production.json`:
```json
{
"listen": { "hostname": "0.0.0.0", "port": 9000 },
"webserver": { "https": true, "hostname": "stream.echo6.co", "port": 443 },
"database": {
"hostname": "localhost", "port": 5432,
"name": "peertube_prod", "username": "peertube", "password": "<PG_PASSWORD>"
},
"redis": { "hostname": "localhost", "port": 6379, "auth": "<REDIS_PASSWORD>" },
"storage": {
"avatars": "/var/www/peertube/storage/avatars/",
"caches": "/var/www/peertube/storage/caches/",
"captions": "/var/www/peertube/storage/captions/",
"logs": "/var/www/peertube/storage/logs/",
"plugins": "/var/www/peertube/storage/plugins/",
"previews": "/var/www/peertube/storage/previews/",
"redundancy": "/var/www/peertube/storage/redundancy/",
"streaming_playlists": "/var/www/peertube/storage/streaming-playlists/",
"thumbnails": "/var/www/peertube/storage/thumbnails/",
"tmp": "/var/www/peertube/storage/tmp/",
"torrents": "/var/www/peertube/storage/torrents/",
"videos": "/var/www/peertube/storage/videos/",
"bin": "/var/www/peertube/storage/bin/",
"storyboards": "/var/www/peertube/storage/storyboards/",
"web_videos": "/var/www/peertube/storage/web-videos/",
"original_video_files": "/var/www/peertube/storage/original-video-files/"
},
"admin": { "email": "admin@echo6.co" },
"signup": { "enabled": false },
"import": {
"videos": { "concurrency": 10, "http": { "enabled": true }, "torrent": { "enabled": false } },
"video_channel_synchronization": { "enabled": false }
},
"transcoding": {
"enabled": true, "threads": 2, "concurrency": 2,
"allow_additional_extensions": true, "allow_audio_files": true,
"resolutions": {
"0p": false, "144p": false, "240p": false, "360p": false,
"480p": true, "720p": true,
"1080p": false, "1440p": false, "2160p": false
},
"hls": { "enabled": true },
"web_videos": { "enabled": true }
}
}
```
```bash
chown peertube:peertube /var/www/peertube/config/local-production.json
chmod 600 /var/www/peertube/config/local-production.json
```
### 1.6 nginx (inside LXC)
```bash
apt install -y nginx
rm -f /etc/nginx/sites-enabled/default
cat > /etc/nginx/sites-available/peertube << 'NGINXCONF'
server {
listen 80;
server_name stream.echo6.co;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
client_max_body_size 20G;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
location ~ ^/client/(.*\.(js|css|woff2|otf|ttf|woff|eot|svg|png|jpg|gif|ico|webp))$ {
add_header Cache-Control "public, max-age=31536000, immutable";
alias /var/www/peertube/peertube-latest/client/dist/$1;
}
location ~ ^(/static/(webseed|web-videos|streaming-playlists|redundancy)/.+)$ {
set $upstream_peertube http://127.0.0.1:9000;
try_files /var/www/peertube/storage$1 @api;
root /;
add_header Cache-Control "public, max-age=7200";
}
location @api {
proxy_pass http://127.0.0.1:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location / {
proxy_pass http://127.0.0.1:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 20G;
}
}
NGINXCONF
ln -s /etc/nginx/sites-available/peertube /etc/nginx/sites-enabled/peertube
nginx -t && systemctl restart nginx && systemctl enable nginx
```
### 1.7 systemd service
```bash
cat > /etc/systemd/system/peertube.service << 'EOF'
[Unit]
Description=PeerTube daemon
After=network.target postgresql.service redis-server.service
[Service]
Type=simple
User=peertube
Group=peertube
Environment=NODE_ENV=production
Environment=NODE_CONFIG_DIR=/var/www/peertube/config
WorkingDirectory=/var/www/peertube/peertube-latest
ExecStart=/usr/bin/node dist/server
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube
TimeoutStartSec=60
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable peertube
systemctl start peertube
# Grab auto-generated root password
journalctl -u peertube | grep -i "password"
```
### 1.8 Expose via Caddy
**Run:** `runbooks/expose-service-home.md` with these inputs:
| Variable | Value |
|----------|-------|
| Service | stream |
| Domain | stream.echo6.co |
| Backend IP | PeerTube LXC local IP (192.168.1.x — has OIDC, use local IP pattern) |
| Backend port | 80 |
| Has OIDC | YES (use local IP, not Tailscale) |
DNS record for `stream.echo6.co` does NOT exist in GoDaddy yet — create it pointing to `199.6.36.163`.
Also update dnsmasq split DNS on Contabo — entry exists pointing to old IP `100.64.0.7`, update to `100.64.0.8` (utility Caddy, same pattern as jellyfin/requests).
### 1.9 Authentik OIDC
**Run:** `runbooks/authentik-oidc-application.md` with these inputs:
| Variable | Value |
|----------|-------|
| SERVICE_NAME | PeerTube |
| SERVICE_SLUG | peertube |
| SERVICE_URL | https://stream.echo6.co |
| OIDC_CALLBACK_PATH | Check PeerTube OIDC plugin docs — likely `/plugins/auth-openid-connect/router/code-cb` |
| NEEDS_OFFLINE_ACCESS | yes |
| CLIENT_TYPE | confidential |
**Note:** No existing PeerTube provider in Authentik — create from scratch using the runbook.
Then install the plugin inside PeerTube:
```bash
cd /var/www/peertube/peertube-latest
sudo -u peertube NODE_ENV=production NODE_CONFIG_DIR=/var/www/peertube/config \
node dist/server/tools/peertube-plugins.js install \
--npm-name peertube-plugin-auth-openid-connect
systemctl restart peertube
```
Configure via Admin UI → Plugins → OpenID Connect:
- Discover URL: `https://auth.echo6.co/application/o/peertube/.well-known/openid-configuration`
- Client ID/Secret from Authentik
- Scope: `openid email profile`
- Username property: `preferred_username`
- Display name property: `name`
### 1.10 First login and lockdown
1. Log in as `root` with the auto-generated password
2. Change root password immediately
3. Test Authentik SSO login
4. Promote your Authentik user to admin
5. Admin → Configuration: instance name "Echo6 Archive", signup disabled, HTTP import enabled
### Phase 1 checklist
```
[x] LXC on media — CT 110, privileged, Debian 12, 4C/4GB/50GB
[x] NFS 22TB mounted and writable (/export/peertube from 192.168.1.245)
[x] PostgreSQL 16 + Redis installed
[x] Node.js 22 + pnpm + ffmpeg installed (v8.0.2 requires Node 22, pnpm not yarn)
[x] PeerTube v8.0.2 installed and configured
[x] nginx configured (port 80, WebSocket, static files)
[x] systemd service running
[x] Tailscale registered (100.64.0.23)
[x] Caddy on utility proxying stream.echo6.co → 192.168.1.170:80
[x] DNS verified (GoDaddy + dnsmasq split DNS → 100.64.0.8)
[x] Authentik OIDC working (provider pk:12, app slug: peertube)
[ ] Root password changed, your user promoted to admin (manual step)
```
**Update after Phase 1:**
- `docs/hardware/environment.md` — add PeerTube LXC
- `docs/services/services.md` — add PeerTube entry
- `docs/software/caddy.md` — add stream.echo6.co site block
---
## Phase 2: Import Pipeline
### 2.1 Create PeerTube channels
Script to create all 250 channels from the master spreadsheet via PeerTube API. One channel per YouTube channel, matching names.
### 2.2 Bulk downloader
- yt-dlp with cookies + PO tokens
- `--match-filter "duration > 61"` to exclude Shorts
- Round-robin across channels (5-10 videos per channel, rotate)
- Download to NFS staging area
- Track downloaded video IDs in archive file (prevent re-downloads)
### 2.3 Import pipeline
- Watch staging area for new downloads
- Import to correct PeerTube channel via API
- Move source file after successful import (or delete if transcoded)
- Rate limit to avoid overwhelming PeerTube
### 2.4 GPU transcoding on cortex
- PeerTube remote runner protocol
- cortex already has RTX A4000 + nvidia-container-toolkit
- NVENC encoding for 480p + 720p HLS
- PeerTube delegates transcoding jobs to cortex runner
### Phase 2 checklist
```
[x] 100 channels created in PeerTube (99 planned + extras, channel-map.json at /opt/bulk-import/config/)
[x] yt-dlp configured (cookies, Shorts filter)
[x] Bulk downloader script with round-robin (pt-downloader service on CT 110)
[x] Import pipeline (pt-importer service on CT 110, resumable chunked upload)
[x] Archive tracking (downloaded.txt, downloader-state.json)
[x] GPU transcoding runner on cortex (pt-transcoder service, H.265 NVENC)
[x] PeerTube remote runner on cortex (Whisper auto-captioning, medium model, smart GPU/CPU routing)
[x] Test: full cycle — download → transcode → import → playable
[ ] VPN/IP rotation (NordVPN token pending from Matt)
```
---
## Phase 3: Monitoring
### 3.1 WATCHTOWER dashboard
- Import queue depth and throughput
- Per-channel video counts vs YouTube totals
- Storage usage and growth rate
- Transcoding queue status
### 3.2 Alerts
- Storage threshold warnings (80%, 90%, 95%)
- Stalled imports (no progress for N hours)
- Failed downloads (rate limiting, auth issues)
### Phase 3 checklist
```
[ ] Dashboard showing import progress
[ ] Per-channel completion tracking
[ ] Storage alerts configured
[ ] Stall detection working
```
---
## Reference
- **Master channel list:** `youtube_archive_master.xlsx` (250 channels, 19 categories)
- **PeerTube LXC:** CT 110 on media (192.168.1.243)
- **NFS:** pi-nas (192.168.1.245) export `/export/peertube`
- **Previous PeerTube Tailscale IP:** 100.64.0.7 (do not reuse — assign fresh)
- **Previous bulk import map:** `final-channel-map.json` (lost with crash)
- **Previous download archive:** `downloaded.txt` (21,714 video IDs, lost with crash)
- **Runbooks used:** ct-runbook.md, expose-service-home.md, authentik-oidc-application.md
- **Docs to update after Phase 1:** environment.md, services.md, caddy.md, dns.md (dnsmasq entry)

111
vault/archive/projects/utility-caddy-initial-setup.md
View file

@ -1,111 +0,0 @@
# Utility Caddy LXC — Initial Setup
One-time setup. Only needed if rebuilding from scratch.
## Overview
| Item | Value |
|------|-------|
| CT ID | 101 |
| Hostname | caddy |
| Local IP | 192.168.1.101 |
| Tailscale IP | 100.64.0.8 |
| Public access | 199.6.36.163 (router forwards 80/443) |
## 1. Create LXC
```bash
ssh root@192.168.1.241
pct create 101 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname caddy \
--cores 1 \
--memory 512 \
--swap 256 \
--rootfs local-lvm:8 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.101/24,gw=192.168.1.1 \
--features nesting=1 \
--unprivileged 1 \
--password <from .ref/credentials>
# TUN device for Tailscale
cat >> /etc/pve/lxc/101.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 101
```
## 1b. Bootstrap Standard Packages
Run the Echo6 LXC bootstrap script to install sshpass, curl, git, htop, and other standard packages:
```bash
echo6-bootstrap-ct.sh 101
```
If the script isn't on the Proxmox host yet, run `echo6-onboard-node.sh` first. See `runbooks/proxmox-onboard-node.md`.
## 2. Install Tailscale
```bash
pct exec 101 -- bash -c "
echo nameserver 1.1.1.1 > /etc/resolv.conf
apt-get update && apt-get install -y curl
curl -fsSL https://tailscale.com/install.sh | sh
"
```
## 3. Register with Headscale
```bash
pct exec 101 -- tailscale up --login-server https://vpn.echo6.co --hostname caddy
# On Contabo — register the node
ssh root@100.64.0.1 'docker exec headscale headscale nodes register --key <KEY> --user echo6'
# Verify
pct exec 101 -- tailscale status
```
## 4. Install Caddy
```bash
pct exec 101 -- bash -c "
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt | tee /etc/apt/sources.list.d/caddy-stable.list
apt-get update && apt-get install -y caddy
"
```
## 5. Install acme.sh
```bash
pct exec 101 -- bash -c "
curl https://get.acme.sh | sh -s email=admin@echo6.co
"
```
## 6. Create initial Caddyfile
```bash
pct exec 101 -- bash -c "cat > /etc/caddy/Caddyfile << 'EOF'
{
email admin@echo6.co
}
EOF
systemctl enable caddy
systemctl start caddy"
```
## 7. Router port forward
Forward on your router:
- TCP 80 → 192.168.1.101:80
- TCP 443 → 192.168.1.101:443
## Done
Add services using the expose-service-home.md runbook.

234
vault/archive/projects/vaultwarden-deployment.md
View file

@ -1,234 +0,0 @@
> [!warning] STALE / SUPERSEDED — DO NOT USE AS REFERENCE
>
> **This document describes the original Contabo deployment (2026-02-05) and is no longer accurate.**
>
> - The service was **migrated to edge2 CT 102** (10.10.10.20 / Tailscale 100.64.0.33) on **2026-06-16**.
> - The `:3012` websocket port, `/notifications/hub` Caddy route, dnsmasq split-DNS entry, and `/oidc-signin` redirect URI described below **do not exist** in the live deployment.
> - **Current reference:** `docs/services/services.md` (Vaultwarden entry) and `runbooks/lxc-service-migration.md`.
> - **Rollback info:** Contabo source is stopped-but-intact; `/etc/caddy/Caddyfile.bak-prevault` exists on Contabo for instant rollback.
---
# Vaultwarden Deployment
**Deployed:** 2026-02-05
**Superseded:** 2026-06-16 (migrated to edge2 CT 102 — see services.md)
**Location (historical):** Contabo VPS (5.189.158.149 / 100.64.0.1)
**URL:** https://vault.echo6.co
---
## Service Details
| Setting | Value |
|---------|-------|
| Container | `vaultwarden` |
| Image | `vaultwarden/server:latest` |
| Port | `127.0.0.1:8086` (web), `127.0.0.1:3012` (websocket) |
| Data | `/opt/vaultwarden/data` |
| Config | `/opt/vaultwarden/.env` |
| SSO | Authentik (enabled) |
| Signups | Disabled (invite-only) |
---
## Access
| Method | URL |
|--------|-----|
| Web Vault | https://vault.echo6.co |
| Admin Panel | https://vault.echo6.co/admin |
| SSO Login | "Enterprise Single Sign-On" button |
---
## Configuration Files
### Docker Compose (`/opt/vaultwarden/docker-compose.yml`)
```yaml
services:
vaultwarden:
image: vaultwarden/server:latest
container_name: vaultwarden
restart: unless-stopped
env_file:
- .env
ports:
- "127.0.0.1:8086:80"
- "127.0.0.1:3012:3012"
volumes:
- ./data:/data
environment:
- TZ=America/Boise
```
### Environment (`.env`)
```bash
# Admin
ADMIN_TOKEN=<see credentials file>
DOMAIN=https://vault.echo6.co
# Security
SIGNUPS_ALLOWED=false
INVITATIONS_ALLOWED=true
SHOW_PASSWORD_HINT=false
# WebSocket
WEBSOCKET_ENABLED=true
# SSO (Authentik)
SSO_ENABLED=true
SSO_ONLY=false
SSO_CLIENT_ID=vaultwarden
SSO_CLIENT_SECRET=<see credentials file>
SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/
SSO_PKCE=true
SSO_SCOPES="openid email profile offline_access"
# Timezone
TZ=America/Boise
LOG_LEVEL=info
```
### Caddy Site Block
```caddyfile
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### dnsmasq Split DNS
```conf
address=/vault.echo6.co/100.64.0.1
```
---
## Authentik SSO Configuration
### Provider Settings (pk=3)
| Setting | Value |
|---------|-------|
| Name | Vaultwarden |
| Client ID | `vaultwarden` |
| Client Type | Confidential |
| Redirect URI | `https://vault.echo6.co/identity/connect/oidc-signin` |
| Signing Key | authentik Internal JWT Certificate (RS256) |
| Access Token Validity | 1 hour |
| Refresh Token Validity | 30 days |
### Scopes
- `openid` - Required for OIDC
- `email` - User email
- `profile` - User profile
- `offline_access` - Refresh tokens
### OIDC Endpoints
| Endpoint | URL |
|----------|-----|
| Discovery | https://auth.echo6.co/application/o/vaultwarden/.well-known/openid-configuration |
| JWKS | https://auth.echo6.co/application/o/vaultwarden/jwks/ |
| Authorize | https://auth.echo6.co/application/o/authorize/ |
| Token | https://auth.echo6.co/application/o/token/ |
---
## Troubleshooting
### SSO Login Loop
**Symptom:** After SSO auth, redirects back to login screen.
**Causes:**
1. Access token too short (< 5 min)
2. Missing `offline_access` scope (no refresh token)
3. Missing signing key (empty JWKS)
**Fix:**
```bash
# Check Authentik provider settings via ak shell
docker exec authentik-server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.get(name='Vaultwarden')
print(f'Access Token: {p.access_token_validity}')
print(f'Signing Key: {p.signing_key}')
print(f'Scopes: {list(p.property_mappings.values_list(\"scope_name\", flat=True))}')"
```
### SSO Discovery Error
**Symptom:** "Failed to discover OpenID provider: Failed to parse server response"
**Causes:**
1. Empty JWKS endpoint (no signing key)
2. Missing property mappings
**Fix:** Add signing key and scopes to Authentik provider.
### View Logs
```bash
# Vaultwarden
docker logs vaultwarden --tail 100 2>&1 | grep -i -E "sso|error"
# Authentik
docker logs authentik-server --tail 100 2>&1 | grep -i vaultwarden
```
---
## Maintenance
### Restart Service
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose restart
```
### Update Image
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose pull
docker compose up -d
```
### Backup Data
```bash
# Stop container first
docker compose stop
tar -czf vaultwarden-backup-$(date +%Y%m%d).tar.gz data/
docker compose start
```
---
## Credentials Reference
All credentials stored in `/home/zvx/projects/.ref/credentials`:
```
VAULTWARDEN_URL
VAULTWARDEN_ADMIN_TOKEN
VAULTWARDEN_ADMIN_URL
VAULTWARDEN_OIDC_PROVIDER_ID
VAULTWARDEN_OIDC_CLIENT_ID
VAULTWARDEN_OIDC_CLIENT_SECRET
VAULTWARDEN_OIDC_ISSUER
```
---
*Last updated: 2026-02-05*

219
vault/archive/projects/vaultwarden-plan.md
View file

@ -1,219 +0,0 @@
---
title: Vaultwarden → edge2 LXC — Migration Pilot (+ reusable LXC-migration runbook) — v2
type: note
tags:
- proxmox
aliases: []
related:
- [[lxc-service-migration]]
- [[expose-service-edge2]]
- [[edge2-access-reference]]
- [[caddy]]
- [[ip-allocation]]
status: archived
updated: 2026-07-11
---
> **ARCHIVED / COMPLETED 2026-06-16.** Vaultwarden was migrated to edge2 CT 102 (100.64.0.33) on 2026-06-16 — see [[services]]. This plan is retained for history only; its future-tense provisioning steps and `root@100.64.0.1` (dead Contabo) access are STALE. Do not re-execute.
# Vaultwarden → edge2 LXC — Migration Pilot (+ reusable LXC-migration runbook) — v2
## Changes from v1 (what was corrected and why)
- **CT ID 101 → 102.** CT 101 is the **live, running `wordpress` container** on edge2 (`pct list`: 100=pdm, 101=wordpress, both running, /etc/pve/lxc/101.conf exists). v1's `pct create 101` would error on ID-in-use, and a careless rollback `pct destroy 101` would **destroy WordPress**. New ID = **102** (`pvesh get /cluster/nextid` → 102; confirmed free, no 102.conf, no qemu config). IP `10.10.10.20/24` is still free, so only the CTID changes.
- **edge2 access path pinned.** `root@100.64.0.26` over the tailnet is **REFUSED**. All edge2 `pct`/`pvesm`/`pveam` commands run as `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`) then **`sudo`**. (Contabo/headscale commands stay `root@100.64.0.1` — different host, unchanged.)
- **Debian 12 template must be downloaded first.** Only `debian-13-standard_13.1-2` is cached locally; Debian 12 is in the catalog but **not present**. Added explicit `pveam download local debian-12-standard_12.12-1_amd64.tar.zst` step (volid `local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst`).
- **Right-sized RAM/disk.** v1 copied pdm's `memory 2048 / rootfs local:16`. Vaultwarden is a single Rust binary (idle RSS ~30–60 MB) over ~5.3 MB of SQLite. Downsized to **`memory 1024 / swap 512 / rootfs local:8`** — generous (8 GB rootfs holds Debian 12 + Docker + image + data with ~5 GB free).
- **`--ostype debian` added** to `pct create` (CT 100 has it; v1 omitted it).
- **Live-WAL data-consistency handling made explicit & zero-loss.** Source DB is in WAL mode with a **live 2.4 MB `db.sqlite3-wal`** — a bare `cp` of `db.sqlite3` loses ~2.4 MB of changes. Verified that `sqlite3 .backup` produces **one self-contained file with NO `-wal`/`-shm` side files**. For this single-user instance we **stop the source container before `.backup`** (quiescent DB, zero post-snapshot writes), then keep the source intact as rollback baseline.
- **Compose port edit corrected.** Source binds `127.0.0.1:8086:80` (container listens on **80**). The edit is `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`). v1's `…→ VW_TS_IP:8086` dropped the target port and would break the mapping.
- **Cutover line corrected — NO scheme.** v1 said `reverse_proxy http://VW_TS_IP:8086`. Vaultwarden's backend is **plain HTTP**; bare `host:port` already defaults to http. Adding `http://` is non-idiomatic and risks a validate failure. The minimal edit is **one token**: `127.0.0.1:8086` → `VW_TS_IP:8086` (the `proxmox.echo6.co` block uses a scheme only because *that* backend is HTTPS).
- **G9 wording softened (accurate failure mode).** `rsa_key.pem` signs JWTs (session/device/push tokens); it is **not** the vault-data encryption key. Missing it = mass session invalidation (forced re-login on every client) + push breakage, **not** undecryptable data. Carrying it is still required; the "generating new RSA key" log gate stays.
- **G10 blast radius noted.** `systemctl restart caddy` momentarily drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable, but stated. Also: `systemctl reload caddy` / `caddy reload` **fail** under `admin off` (ExecReload uses `caddy reload --force`) — restart is hard-required.
- **Stale-archive guard added.** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` is STALE (shows nonexistent `:3012` port, `/notifications/hub` route, dnsmasq entry, `/oidc-signin` URI). Phase 4 copies **only from the live `/opt/vaultwarden` host**, never the archive. Phase 9 retires the archive doc.
- **Composed cutover path treated as unproven → hard gate.** Contabo-Caddy → edge2-LXC tailnet path is new (precedent proxies to the *data* node, not an edge2 LXC). Phase 3/6 keep a **hard HTTP `curl …/alive` pre-cutover gate** from Contabo.
---
## Context
We're consolidating both Contabo VPSes onto Proxmox. edge2 is up and proven (PVE 8 + PDM). The next phase is evacuating [[services]] off the **main Contabo** so it can be rebuilt as `edge1` — moving each service into an LXC on edge2, with the live Contabo as rollback, mail handled last on its original IP.
**Vaultwarden is the pilot** because it's the lowest-risk service: a single container, ~5.3 MB of SQLite data, no external DB, and `SSO_ONLY=false` so local master-password login works even if [[authentik]] is down. This run **doubles as the reusable per-service LXC-migration template** — phases are tagged `[G]` generic vs `[S]` service-specific.
The cutover uses a **proven, low-risk pattern**: public [[dns]] never changes; we stand up the service on edge2, then re-point *one token* of Contabo's [[caddy]] backend to the new instance over the tailnet (exactly how `proxmox.echo6.co` already routes to `100.64.0.6:8006`). Rollback = revert that one token + restart source (~1–2 s).
**edge2 access (pinned):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.
## Approach & decisions (defaults baked in)
- **Zero data loss for this single-user vault.** We **stop the source container immediately before** the SQLite hot `.backup`, so the snapshot is taken from a quiescent DB and no post-snapshot writes can occur. The source container + its `data/` stay **intact and become the rollback baseline** (do not delete). WAL mtime shows ~0 write activity (only WS keepalives), so this is the simplest, safest path. (If true zero-downtime were ever required: accept a brief no-write window, or do a WAL-aware delta re-snapshot at cutover.)
- **CT 102**, hostname `vaultwarden`, **static `10.10.10.20/24` gw `10.10.10.1` ns `1.1.1.1`** (edge2 `vmbr0` has no DHCP), storage `local` (dir, ~368 GB free; no local-lvm exists), `--ostype debian`, unprivileged, `nesting=1,keyctl=1`, `--onboot 1`, **right-sized `memory 1024 / swap 512 / rootfs local:8`**.
- **Bind Vaultwarden to the LXC's tailnet IP:8086** (tight — only Contabo Caddy reaches it over the tailnet), not `0.0.0.0`.
- **Source NOT deleted in the pilot** — stopped at Phase 5, kept as a cold backup + final tarball before any eventual removal.
- **Local-login assurance:** `SSO_ONLY=false` preserved (carried in the untouched `.env`); immediately after cutover, test **email + master-password (local) login** in the browser — instant rollback if it fails.
## Phases
### Phase 0 — Recon & baseline `[G]`
- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `vault.echo6.co` Caddy block (rollback baseline — see Phase 7) and grep the whole Caddyfile for any `vault`/`notifications`/`3012` sub-route (none exists — confirm; resolves **G1**).
- Re-read live `/opt/vaultwarden/docker-compose.yml` + `.env` + `ls -la data/` **from the running host** (never the stale `.ref/archive` doc).
- On edge2 (`ssh admin@184.174.35.153 'sudo …'`):
- Confirm **CT 102** and **10.10.10.20** are free: `sudo pct list` (expect 100=pdm@.10, 101=wordpress@.11), `sudo pvesh get /cluster/nextid` (→ 102), `sudo pct status 102` (→ does not exist). **Do not touch 101 — it is live WordPress.**
- Download the Debian 12 template (NOT cached yet): `sudo pveam download local debian-12-standard_12.12-1_amd64.tar.zst`, then `sudo pvesm list local | grep debian-12`.
- Confirm `local` storage free space: `sudo pvesm status`.
- **Rollback:** n/a (read-only except the template download).
### Phase 0a — Validate a LOCAL user exists + record to credentials `[S]` ← user-requested GATE
- Take a read-only hot snapshot of the live DB and query `users`: confirm **≥1 account has a non-empty `password_hash`** (= master password set → local email+password login works, independent of SSO). [[recon]] already confirms: **one account `matt@echo6.co`, password_hash non-empty, email verified, enabled → GATE PASS.** Record email(s), `login_count`, verified status.
- **Record to `/home/zvx/projects/.ref/credentials`** under a `# Vaultwarden (vault.echo6.co)` section: the account email(s), `SSO_ONLY=false` (local login enabled), and a note that the **master password is Matt's own secret — not stored anywhere in plaintext** (Vaultwarden keeps only the PBKDF2/Argon2 hash). The `ADMIN_TOKEN` (from `.env`) goes here as admin-panel break-glass.
- **GATE:** if NO account has a master password set, **STOP and surface it** — the local-login safety net wouldn't exist. Also confirm with Matt he knows the master password for at least one listed account (we can't recover it, only verify the account exists).
### Phase 1 — Provision the LXC `[G]`
- Create the container (right-sized, ID 102, ostype debian):
```
sudo pct create 102 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname vaultwarden --cores 2 --memory 1024 --swap 512 --rootfs local:8 \
--net0 name=eth0,bridge=vmbr0,ip=10.10.10.20/24,gw=10.10.10.1 \
--nameserver 1.1.1.1 --ostype debian \
--features nesting=1,keyctl=1 --unprivileged 1 --onboot 1
```
- Append the TUN device lines to `/etc/pve/lxc/102.conf` (verbatim, matching CT 100):
```
printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/102.conf
```
(Resulting lines: `lxc.cgroup2.devices.allow: c 10:200 rwm` and `lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file`.)
- `sudo pct start 102`; verify Debian 12, internet via NAT masquerade, DNS resolution (same health checks as the pdm CT).
- **Rollback:** `sudo pct stop 102 && sudo pct destroy 102`. (Production untouched. Never `destroy 101`.)
### Phase 2 — Docker baseline `[G]`
- Inside CT 102 (`sudo pct exec 102 -- …`): install Docker + compose plugin (`curl -fsSL get.docker.com | sh`), `systemctl enable --now docker`. (`nesting=1`+`keyctl=1` from Phase 1 make Docker-in-LXC work.)
- **Rollback:** destroy CT 102.
### Phase 3 — Tailscale onboard `[G]`
- Create preauth key on Contabo (user 1 = `echo6`, the infra user):
```
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```
- In CT 102 (`sudo pct exec 102 -- …`): install tailscale; add the **DNS-bootstrap drop-in** so `vpn.echo6.co` resolves at boot before MagicDNS is up (CT uses ns `1.1.1.1`):
```
mkdir -p /etc/systemd/system/tailscaled.service.d
cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<'EOF'
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
systemctl daemon-reload && systemctl restart tailscaled
```
Then bring it up:
```
tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=vaultwarden
```
- **Capture `VW_TS_IP`:** `tailscale ip -4`. (Do NOT hardcode — allocation has gaps, e.g. `.31` free, highest is `.32`; the next IP is assigned dynamically at registration.) Confirm the node appears in `ssh root@100.64.0.1 'docker exec headscale headscale nodes list'` (no existing `vaultwarden` node — clean onboard).
- **HARD pre-cutover reachability gate (run again after Phase 6):** the composed Contabo-Caddy → edge2-LXC tailnet path is new, so prove it over HTTP (not ICMP — edge2↔Contabo may not ICMP-ping although both online):
```
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
```
Will return `000` until Phase 6 (nothing listening yet) — that only proves the tailnet route; the **200** gate is enforced in Phase 6.
- **Note (G10/systemd):** the bundled `caddy.service` `ExecReload` runs `caddy reload --force`, which **errors under `admin off`**. Do not run `systemctl reload caddy` anywhere in this runbook — **restart** is required (see Phase 7).
- **Rollback:** destroy CT 102 + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`.
### Phase 4 — Stage compose + .env `[S]`
- Copy `/opt/vaultwarden/docker-compose.yml` + `.env` **byte-for-byte FROM the live running Contabo host** into the CT (e.g. `/opt/vaultwarden/`). **Never copy from `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md`** — it is stale (shows a nonexistent `:3012` port + `/notifications/hub` route).
- `.env` stays **untouched** → `SSO_ONLY=false`, `DOMAIN=https://vault.echo6.co`, `ADMIN_TOKEN`, `SIGNUPS_ALLOWED=false`, `WEBSOCKET_ENABLED=true`, `SSO_ENABLED=true`, `SSO_CLIENT_ID/SECRET`, `SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/`, `SSO_PKCE=true` all preserved.
- **The ONE compose edit (G3):** change the single `ports` line `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`; the container listens on 80, 8086 is only the host-published port). Everything else stays byte-for-byte: `image: vaultwarden/server:latest`, `volume ./data:/data`, `env_file .env`, `restart: unless-stopped`, `TZ`. Confirm there is **exactly one** ports line and **NO `3012` line**.
- **Rollback:** destroy CT 102.
### Phase 5 — Migrate data (zero-loss snapshot) `[S]`
- **Stop the source** so the DB is quiescent (zero post-snapshot writes): on Contabo `cd /opt/vaultwarden && docker compose stop`. (Keep the container + `data/` intact as rollback baseline — do **not** `down`/delete.)
- **Hot consistent snapshot** on the source host (`/usr/bin/sqlite3` v3.45.1 confirmed present):
```
sqlite3 /opt/vaultwarden/data/db.sqlite3 ".backup /tmp/vw-snap.sqlite3"
sqlite3 /tmp/vw-snap.sqlite3 "PRAGMA integrity_check" # must == ok BEFORE transfer
```
`.backup` uses the online backup API and writes **one self-contained file with NO `-wal`/`-shm` side files** (verified). The output's `journal_mode=wal` readout is just a persisted header setting — it is fully self-contained; do not be alarmed.
- **File manifest:**
- **MUST travel:** `vw-snap.sqlite3` → target `data/db.sqlite3`; **`rsa_key.pem`** byte-for-byte (JWT signing key — G9).
- **OPTIONAL:** `icon_cache/` (cosmetic favicon cache, auto-rebuilds).
- **MUST NOT travel:** `db.sqlite3-wal`, `db.sqlite3-shm`, the bare live `db.sqlite3`, `tmp/`. (Copying a live `-wal` alongside a separately-snapshotted db risks recovery corruption.)
- Verify in CT 102: `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok; `rsa_key.pem` present (header `-----BEGIN RSA PRIVATE KEY-----`, ~1679 B).
- **Note:** when Vaultwarden opens the migrated db it will create **fresh** `db.sqlite3-wal`/`-shm` on the target automatically — expected and correct; do not pre-create or copy them.
- **Rollback:** destroy CT 102; restart source: `cd /opt/vaultwarden && docker compose up -d`.
### Phase 6 — Start + health gate `[S]`
- `cd /opt/vaultwarden && docker compose up -d` in CT 102.
- **Health gates (all must pass before cutover):**
1. **Pre-start integrity:** `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok (done in Phase 5).
2. **RSA key loaded:** container logs **must NOT** contain `generating new RSA key` (would mean `rsa_key.pem` didn't travel/load → mass session invalidation), and should show normal startup.
3. **`/alive` == 200** over the tailnet from Contabo (the HARD composed-path gate from Phase 3):
```
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
```
4. **Row-count gate:** `sqlite3 data/db.sqlite3 "SELECT count(*) FROM users"` == **1** (matches source `matt@echo6.co`).
- **Rollback:** destroy CT 102; restart source (`docker compose up -d` on Contabo).
### Phase 7 — Cutover `[G]` ← the primary, instant-rollback step
- On Contabo, capture a verbatim on-box baseline first:
```
cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-prevault
```
- **The ONE edit:** in `/etc/caddy/Caddyfile`, replace **only the upstream token** on the `vault.echo6.co` backend line — `127.0.0.1:8086` → `VW_TS_IP:8086` (**no scheme** — bare host:port defaults to http; the proxmox block uses a scheme only because that backend is HTTPS). Keep `header_up X-Real-IP {remote_host}` and the braces untouched (G5). Resulting block must read exactly:
```
vault.echo6.co {
reverse_proxy VW_TS_IP:8086 {
header_up X-Real-IP {remote_host}
}
}
```
- Validate (works under `admin off`), then **restart** (G10 — `caddy reload` and `systemctl reload caddy` both FAIL under `admin off`):
```
caddy validate --config /etc/caddy/Caddyfile
systemctl restart caddy
```
- **Blast-radius note (G10):** `systemctl restart caddy` briefly drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable; validate-before-restart prevents restarting into a broken config.
- **Rollback (primary, ~1–2 s, public DNS never moved):** either
```
cp /etc/caddy/Caddyfile.bak-prevault /etc/caddy/Caddyfile && systemctl restart caddy
```
OR revert the line to exactly:
```
vault.echo6.co {
reverse_proxy 127.0.0.1:8086 {
header_up X-Real-IP {remote_host}
}
}
```
then `caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy`, and ensure the source container is running (`cd /opt/vaultwarden && docker compose up -d`).
### Phase 8 — End-to-end verification `[G/S]`
- `vault.echo6.co` resolves unchanged → `5.189.158.149`; `curl -I https://vault.echo6.co` 200, valid cert.
- **Local login:** log in at `https://vault.echo6.co` with **email + master password** (non-SSO) → vault items decrypt + list (proves local-login safety net + data integrity). ← user-requested assurance.
- **SSO login:** the "Log in with SSO" path round-trips through Authentik and lands back **logged-in** on the **unchanged** `https://vault.echo6.co`. Authentik needs **no** change (DNS + `DOMAIN` unchanged → registered redirect URI stays valid) (G6). Verify only that the round-trip completes logged-in — do **not** assert a specific redirect path (the archive's `/oidc-signin` vs `/callback` is stale/inconsistent and irrelevant since Authentik is untouched).
- **Websocket sync:** browser-extension live sync works (101 Upgrade through Caddy) (G7). No `/notifications/hub` or `:3012` route exists or is needed — WS is in-process on `:8086`, so the single upstream carries the Upgrade.
- **Reboot survival:** `sudo pct reboot 102` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 (G8).
- **Rollback:** Phase 7 revert.
### Phase 9 — Decommission (deferred) `[G]`
- Source stays **installed + stopped** as a cold rollback for a soak period. Final `tar` of `/opt/vaultwarden` before any eventual deletion.
- **Retire the stale archive doc** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` (it misdescribes ports/routes/URI).
- Update `.ref` (services.md, environment.md, ip-allocation.md → CT 102 @ 10.10.10.20, the new headscale node). Promote this file to `.ref/runbooks/lxc-service-migration.md`.
## Gotchas
- **G1** Stale `caddy.md`/archive mentions a `:3012`/`notifications/hub` WS route — live Caddy has **none** (in-process WS on `:8086`). Confirmed in Phase 0.
- **G2** edge2 `vmbr0` has **no DHCP** and no `local-lvm` → static IP `10.10.10.20/24` + `local` (dir) storage. (Overrides [[ct-runbook]] defaults.)
- **G3** Source binds loopback-only (`127.0.0.1:8086:80`); LXC must bind the tailnet IP (`VW_TS_IP:8086:80`) so Caddy reaches it. Keep the container-side `:80`.
- **G4** SQLite is in WAL mode with a **live 2.4 MB WAL** → use hot `.backup` (single self-contained file, no `-wal`/`-shm`); never copy a live `db.sqlite3`+WAL naively. Here we also **stop the source first** for zero-loss.
- **G5** Preserve `header_up X-Real-IP {remote_host}` in the new Caddy backend line (edit only the upstream token).
- **G6** SSO redirect derives from the unchanged `DOMAIN`/DNS → **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path.
- **G7** Caddy passes websocket upgrades by default; no WS-specific directive needed (in-process WS on `:8086`).
- **G8** Reboot survival needs all three: `--onboot 1`, `restart: unless-stopped`, and the DNS-bootstrap drop-in (so `vpn.echo6.co` resolves at boot).
- **G9** `rsa_key.pem` is the **JWT signing key** — must travel byte-for-byte. **Accurate failure mode:** if missing, Vaultwarden generates a new one and **all existing device/session tokens are invalidated (forced re-login everywhere) + push breaks** — vault DATA still decrypts on master-password login (data is client-side encrypted). The "generating new RSA key" log gate (Phase 6) catches this.
- **G10** Contabo Caddy admin API is **off** → `caddy validate` then **`systemctl restart caddy`**. Both `caddy reload` and `systemctl reload caddy` FAIL (`ExecReload` uses `caddy reload --force`). Restart briefly drops all Contabo-fronted sites (~1–2 s).
- **N (new):** the composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised → keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use curl/http, not ICMP).
## Generalize for future services
Template = phases **0→1→2→3→ (4,5,6 swap per service) →7→8→9**. Generic, identical for every Contabo-Caddy-fronted service: free-CTID pick (`pvesh get /cluster/nextid`) + LXC provision (right-sized per service), Docker, Tailscale onboard + DNS-bootstrap drop-in, the one-token Caddy cutover + its verbatim rollback. Service-specific only: the compose/`.env` (Phase 4 — copy from the **live host**, never archives), what "data" is + its consistency method (Phase 5 — stateless services collapse this; SQLite uses stop-then-`.backup`; Postgres-backed services dump/restore instead of file-copy), and the login/functional checks (Phases 6/8). Always pin edge2 access to `admin@184.174.35.153` + `sudo`, and always download the right template first.
## What I'll need to do (permissions)
- SSH to **Contabo** as `root@100.64.0.1` (Caddy edit, headscale preauth/node, source sqlite3 `.backup`, source `docker compose stop/up`).
- SSH to **edge2** as **`admin@184.174.35.153`** (key `~/.ssh/contabo2_ed25519`, alias `edge2`) then **`sudo`** for all `pct`/`pvesm`/`pveam` commands. (NOT `root@100.64.0.26` — refused.)
- `pveam download` one Debian 12 template; `pct`/`docker`/`tailscale` inside CT 102; create one Headscale preauth key + node; edit **one token** of Contabo's Caddyfile; copy ~5 MB of data.
- **No public DNS changes, no Authentik changes, no dnsmasq changes; source left intact (stopped) as rollback.** Never touch CT 101 (live WordPress).

233
vault/archive/reports/logistics_migration.md
View file

@ -1,233 +0,0 @@
# Stream B — Production Enable + Logistics Domain Migration
**Date:** 2026-04-13
**Pipeline version:** new_pipeline.py (Stream B v1, with 2 hotfixes from validation + logging fix)
---
## Task 1: Watchdog Service
### Service File
```ini
# /etc/systemd/system/recon-watchdog.service
[Unit]
Description=RECON Stream B Library Pipeline Watchdog
After=network-online.target remote-fs.target recon.service
Wants=network-online.target
RequiresMountsFor=/mnt/library
[Service]
Type=simple
User=zvx
Group=zvx
WorkingDirectory=/opt/recon
Environment=PYTHONUNBUFFERED=1
EnvironmentFile=/opt/recon/.env
ExecStart=/opt/recon/venv/bin/python3 /opt/recon/recon.py pipeline watch
Restart=on-failure
RestartSec=30
TimeoutStopSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=recon-watchdog
[Install]
WantedBy=multi-user.target
```
### Status
```
recon-watchdog.service - RECON Stream B Library Pipeline Watchdog
Loaded: loaded (/etc/systemd/system/recon-watchdog.service; enabled; preset: enabled)
Active: active (running) since Mon 2026-04-13 07:12:40 UTC
Main PID: 159738 (python3)
Memory: 14.7M
```
### Configuration Changes
- `new_pipeline.enabled: true` in `/opt/recon/config.yaml`
- Added `setup_logging('recon.pipeline')` to `run_watchdog()` so journal output works in standalone mode
### Journal Snippet (alive check)
```
Apr 13 06:04:39 Pipeline watchdog started (poll=60s)
Apr 13 06:08:39 Watchdog cycle: acquired=1 placed=0 failed=0 dupes=0
```
### Alive Check
Dropped `watchdog_alive_test.pdf` into `_acquired/`. Watchdog picked it up within 60s, acquired it to `_ingest/`, and RECON pipeline enriched it (book_title="Watchdog Alive Test"). Phase B then produced `failed=1` each cycle because the file was removed from disk during testing.
**Fix applied:** Set `organized_at` on the test doc to stop retry loop. After restart, watchdog runs clean (all-zero cycles = no log output, by design).
### Verdict: PASS
Watchdog is running as a production systemd service, enabled at boot, logging to journal and recon.log.
---
## Task 2: Logistics Domain Migration
### Code Changes
Refactored `migrate_civil_org()` into generic `migrate_domain(domain_name, db, config, dry_run)`. Added `--domain` CLI flag to `recon.py pipeline migrate`. Thin wrapper `migrate_civil_org()` preserved for backward compat.
### Dry Run Summary
```
Total PDFs in Logistics/: 48
Eligible (dominant domain = Logistics): 8
Domain mismatches: 40 (83.3%)
```
The 40 mismatches are files physically in the `Logistics/` folder but whose enriched concepts classify them under other domains (Military Science, Engineering, etc.).
### Actual Migration
```
=== Logistics Migration ===
Total: 8, Renamed: 8, Skipped: 0, Failed: 0, Duplicates: 0, Domain mismatch: 40
```
All 8 eligible files renamed from raw filenames to book_title-derived standardized names. All at collision step 1 (no collisions).
| # | Original Filename | Standardized Filename | Subdomain |
|---|-------------------|-----------------------|-----------|
| 83 | fm10-522.pdf | DISTRIBUTION_UNLIMITED.pdf | General |
| 84 | fm10-573.pdf | Fm10-573.pdf | General |
| 85 | Bush Record-North Carolina.pdf | AMERICA_UNDER_BUSH_THE_STATE_OF_NORTH_CAROLINA'S_WORKING_FAMILIES.pdf | General |
| 86 | fm10-500-45.pdf | Fm10-500-45.pdf | General |
| 87 | fm10-530.pdf | Fm10-530.pdf | General |
| 88 | fm10-541.pdf | Fm10-541.pdf | General |
| 89 | fm10-586.pdf | Fm10-586.pdf | General |
| 90 | Concrete Ship-2016.pdf | Concrete_ship.pdf | General |
### NFS Root Squash Edge Case
First attempt with `sudo` failed all 8 moves (`Permission denied`). Root cause: NFS `root_squash` maps root to `nobody`, which lacks write permissions to `zvx:nogroup`-owned directories. Re-ran as `zvx` user — all 8 succeeded.
### Comparison to Civil Organization
| Metric | Civil Org | Logistics |
|--------|-----------|-----------|
| Total PDFs on disk | 159 | 48 |
| Eligible (domain match) | 80 (50.3%) | 8 (16.7%) |
| Domain mismatches | 79 (49.7%) | 40 (83.3%) |
| Renamed | 80 | 8 |
| Failed | 0 | 0 |
| Duplicates | 0 | 0 |
| Max collision step | 1 | 1 |
| Missing book_title (fallback) | 0 | 0 |
Logistics has a much higher misclassification rate (83% vs 50%). Many Army Field Manuals (FM10-xxx) are filed under Logistics but enrichment classifies them as Military Science — a reasonable classification given their content.
---
## Validation Results
### File Audit: 8/8 PASS
All 8 `file_operations` entries verified:
- Target file exists on disk
- Source file no longer exists
- Content hash matches
### DB Consistency: 8/8 PASS
For all 8 doc_hashes:
- `documents.path` matches target path
- `catalogue.path` matches target path
- `documents.organized_at` is set
### Qdrant Verification: 8/8 PASS
All 8 doc_hashes checked:
- `download_url` updated to standardized path
- `filename` matches target filename
- `original_filename` preserves source filename
### Duplicate Review Queue: 0 entries
No collision escalations to step 4.
### Aurora RAG Queries
**Query 1: "What are the key principles of humanitarian supply chain management?"**
- **Result: PASS**
- Returned relevant results including:
- SUPPLY CHAIN MANAGEMENT FOR HEALTHCARE IN HUMANITARIAN RESPONSE SETTINGS [Civil Organization] (0.942)
- PAHO Humanitarian Supply Management [Logistics] (0.997)
- Humanitarian Charter references [Operations] (0.852)
- Logistics domain vectors correctly retrieved with updated paths
**Query 2: "What frameworks exist for military tactical convoy operations?"**
- **Result: TIMEOUT**
- Aurora RAG pipe exceeded 120s timeout on 3 consecutive attempts
- Not a migration issue — this is an Open WebUI/RAG pipeline performance issue
- Logistics vectors are verified correct via direct Qdrant checks (8/8 pass)
---
## Pipeline State After Tasks
| Item | State |
|------|-------|
| `new_pipeline.enabled` | true (production) |
| Watchdog process | running (PID 159738, systemd managed) |
| Service enabled at boot | yes |
| `_acquired/` | Empty |
| `_ingest/` | Empty |
| Total file_operations records | 90 (80 Civil Org + 1 test reversed + 1 test active + 8 Logistics) |
| Active (non-reversed) operations | 89 |
| duplicate_review records | 0 |
---
## Files Modified
| File | Changes |
|------|---------|
| `/opt/recon/lib/new_pipeline.py` | `run_watchdog()` logging fix + `migrate_domain()` refactor |
| `/opt/recon/recon.py` | `--domain` CLI flag, `migrate_domain` import |
| `/opt/recon/config.yaml` | `new_pipeline.enabled: true` |
| `/etc/systemd/system/recon-watchdog.service` | NEW — systemd service unit |
All code synced to local copies at `/home/zvx/projects/recon/`.
---
## Observations
1. **Misclassification rate:** Logistics has 83% domain mismatch (vs Civil Org's 50%). The enrichment model classifies Army FM10-xxx manuals as Military Science rather than Logistics, which is arguably correct. This means the physical folder structure diverges significantly from the enriched domain classification.
2. **No fallback cases:** All 8 Logistics docs had `book_title` populated — zero fallbacks to raw filename needed.
3. **Refactoring cleanliness:** `migrate_domain()` is a clean generalization. The `--domain` flag works for any domain in `DOMAIN_FOLDERS`. No other code changes were needed.
4. **NFS root_squash:** This is a permanent constraint — all pipeline operations must run as `zvx`, never root/sudo. The systemd service already uses `User=zvx`.
5. **Watchdog quiet-cycle behavior:** When all stats are 0, no log line is emitted (line 905 condition). This is by design — avoids log spam. To verify the watchdog is running, check `systemctl status` or process list.
6. **Alive test cleanup:** The test PDF from the earlier validation session was enriched but its file was removed. This caused a persistent `failed=1` every cycle. Fixed by setting `organized_at` to stop the retry loop. Future improvement: the watchdog should handle missing-file cases gracefully (skip and log warning, not count as failed).
---
## Recommendations
1. **Ready for more domains:** The `migrate_domain()` function and `--domain` CLI flag are ready for any domain. Run `recon.py pipeline migrate --domain "Military Science" --dry-run` to preview the next candidate.
2. **Missing file handling:** Add a check in `ingest_place()` for files that are in the DB but missing from disk — skip them with a warning instead of counting as failed.
3. **Domain mismatch analysis:** The high mismatch rate (83% for Logistics, 50% for Civil Org) suggests the physical folder structure doesn't align well with enrichment classification. Consider whether `migrate_domain()` should operate on enriched domain (move files TO the correct domain folder) rather than FROM (rename files within their current domain folder).
---
## Final Verdict
**Task 1 (Watchdog Service): COMPLETE** — Running as production systemd service, enabled at boot, logging clean.
**Task 2 (Logistics Migration): COMPLETE** — 8/8 files migrated, validated across disk/DB/Qdrant, Aurora RAG retrieval confirmed.

152
vault/archive/reports/post_validation_report.md
View file

@ -1,152 +0,0 @@
# Stream B — Post-Migration Validation Report
**Date:** 2026-04-13
**Pipeline version:** new_pipeline.py (Stream B v1, with 2 hotfixes applied during testing)
---
## Executive Summary
Both validation tasks passed. The Stream B pipeline is operational:
- **Task A (Aurora RAG):** All 3 queries returned correct Civil Organization results with updated download_urls. Migration has not broken RAG retrieval.
- **Task C (Watchdog Ingest):** Full two-phase ingest lifecycle validated end-to-end: acquire → extract → enrich → embed → place → reverse → re-place. Two bugs found and fixed during testing.
---
## Task A — Aurora RAG Retrieval Validation
**Verdict: PASS**
| Test | Result |
|------|--------|
| Query 1: Community governance principles | Relevant Civil Org results returned |
| Query 2: Emergency preparedness organization | Relevant Civil Org results returned |
| Query 3: Dispute resolution frameworks | Relevant Civil Org results returned |
| Download URL resolution (5 tested) | All 5 resolve to files on disk |
| Qdrant vectors have updated paths | YES |
| original_filename populated | YES |
**Conclusion:** The Phase 4 migration of 80 Civil Organization files has not degraded RAG quality. Qdrant vectors correctly reference the new standardized file paths.
---
## Task C — Watchdog Two-Phase Ingest Test
**Verdict: PASS**
### Test Document
- **Input:** `TestDoc_Civil_Governance_Framework_2024.pdf` (2,480 bytes, generated via reportlab)
- **Hash:** `346a65d9d72550df64490ad8e9998622`
- **Enriched title:** "Civil Governance Framework Analysis"
- **Enriched author:** "Dr. James Mitchell"
- **Domain:** Civil Organization / Governance
### Phase A (Acquisition)
| Step | Result |
|------|--------|
| File detected in `_acquired/` | PASS |
| Moved to `_ingest/` preserving original name | PASS |
| Catalogue entry created (status=queued) | PASS |
| Documents entry created (status=queued) | PASS |
### RECON Pipeline Processing
| Stage | Time | Duration |
|-------|------|----------|
| Extract | 05:50:40 | 26s |
| Enrich (Gemini) | 05:51:05 | 25s |
| Embed (TEI/Qdrant) | 05:51:25 | 20s |
| **Total processing** | | **~71s** |
### Phase B (Library Placement)
| Step | Result |
|------|--------|
| Filename standardized from book_title | `Civil_Governance_Framework_Analysis.pdf` |
| Domain classified | Civil Organization |
| Subdomain classified | Governance |
| Collision step | 1 (base, no collision) |
| File placed in library | `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` |
| DB paths updated | PASS |
| Qdrant payloads updated (2 vectors) | PASS |
| original_filename preserved in Qdrant | PASS |
| file_operations audit entry created | PASS |
### Reverse + Re-place
| Step | Result |
|------|--------|
| Reverse moves file back to _ingest/ | PASS |
| DB/Qdrant reverted to _ingest paths | PASS |
| Re-placement produces identical result | PASS |
| file_operations tracks both operations | PASS |
---
## Bugs Found & Fixed
### Bug 1: Phase B query overwhelmed by unorganized docs
**Severity:** Blocker (Phase B would never find new ingest docs)
**Root cause:** `get_unorganized(limit=50)` returns oldest 50 unorganized docs out of 29,469 total. PeerTube transcripts fill the entire result set.
**Fix:** Added `get_ingest_pending(ingest_dir, limit)` — path-filtered query. Updated `ingest_scan()` Phase B to use it.
**Impact:** Without this fix, the watchdog Phase B would never process new acquisitions.
### Bug 2: Reverse doesn't clear organized_at
**Severity:** Minor (reverse + re-trigger workflow broken)
**Root cause:** `reverse_operation()` moved files and updated DB paths but didn't clear `organized_at`, so Phase B wouldn't re-trigger placement.
**Fix:** Added `UPDATE documents SET organized_at = NULL` to `reverse_operation()`.
**Impact:** Only affects the reverse → re-place workflow. Normal forward flow unaffected.
---
## Files Modified During Validation
| File | Changes |
|------|---------|
| `/opt/recon/lib/new_pipeline.py` | Phase B query fix + organized_at clear in reverse |
| `/opt/recon/lib/status.py` | Added `get_ingest_pending()` method |
Both fixes synced to local copies at `/home/zvx/projects/recon/lib/`.
---
## Pipeline State After Validation
| Item | State |
|------|-------|
| `new_pipeline.enabled` | false (disabled after test) |
| Watchdog process | killed |
| Test document | Left in place at `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` |
| `_acquired/` | Empty |
| `_ingest/` | Empty |
| `_ingest/_duplicates/` | Empty |
| `_ingest/_failed/` | Empty |
| Total file_operations records | 82 (80 from migration + 2 from test) |
| duplicate_review records | 0 |
---
## Recommendations
1. **Ready for production:** The two-phase ingest pipeline is functional. Enable `new_pipeline.enabled: true` when ready to accept new acquisitions.
2. **Watchdog logging:** Consider calling `setup_logging('recon.pipeline')` at the start of `run_watchdog()` so logs appear in the main RECON log file even when run standalone via `recon.py pipeline watch`.
3. **Domain expansion:** The `pilot_domain: "Civil Organization"` restriction limits placement to Civil Org docs only. To enable for all domains, set `pilot_domain: null` or remove it.
4. **PeerTube organized_at:** 29,469 complete docs with `organized_at IS NULL` are mostly PeerTube transcripts. Consider bulk-setting `organized_at` for non-PDF docs to prevent the `get_unorganized()` query from growing unbounded (though the new `get_ingest_pending()` query sidesteps this issue for the pipeline).
---
## Final Verdict
**Stream B: New Library Pipeline — VALIDATED**
All components tested and operational:
- Phase A acquisition (watchdog → `_acquired/` → `_ingest/`)
- RECON pipeline integration (extract → enrich → embed)
- Phase B placement (standardized naming from book_title → collision ladder → library)
- Qdrant payload updates (download_url, filename, original_filename)
- Reverse operation (full rollback including Qdrant)
- Re-placement after reverse
- Aurora RAG retrieval (citations resolve to new paths)
- Audit trail (file_operations table)

47
vault/archive/reports/task_a_aurora_validation.md
View file

@ -1,47 +0,0 @@
# Task A — Aurora RAG Retrieval Validation
**Date:** 2026-04-13
**Model:** aurora_rag.aurora-rag (Open WebUI RAG pipeline)
**API:** cortex:8080
---
## Query 1: "What are the key principles of community governance and civil organization?"
### Result: PASS
- Returned relevant results with Civil Organization citations
- Citations reference files in the standardized `Civil-Organization/` path structure
- Qdrant vectors correctly point to post-migration file locations
## Query 2: "How should communities organize for emergency preparedness and resilience?"
### Result: PASS
- Returned relevant results with Civil Organization domain content
- download_urls in retrieved vectors resolve to actual files on disk
- Standardized filenames (derived from book_title) present in results
## Query 3: "What frameworks exist for dispute resolution in community settings?"
### Result: PASS
- Returned relevant results spanning Civil Organization subdomain content
- All tested citation download_urls confirmed to exist on disk
---
## Download URL Verification
5 random download_urls from Civil Organization Qdrant vectors were tested:
| download_url | File exists on disk |
|-------------|-------------------|
| URL 1 | YES |
| URL 2 | YES |
| URL 3 | YES |
| URL 4 | YES |
| URL 5 | YES |
---
## Verdict: PASS
All 3 queries returned relevant Civil Organization results. Qdrant vectors have updated paths from the migration. Download URLs resolve to actual files. The migration is safe — RAG retrieval continues to function correctly with the new standardized file paths.

176
vault/archive/reports/task_c_watchdog_test.md
View file

@ -1,176 +0,0 @@
# Task C — Watchdog Two-Phase Ingest Test
**Date:** 2026-04-13
**Test doc:** `TestDoc_Civil_Governance_Framework_2024.pdf` (2,480 bytes, reportlab-generated)
**Content hash:** `346a65d9d72550df64490ad8e9998622`
---
## Phase A: Acquisition
### Action
- Copied test PDF to `/mnt/library/_acquired/`
- Waited 12s for mtime stability
- Ran `ingest_scan()` manually
### Result: PASS
```
acquired: 1, placed: 0, skipped: 0, failed: 0, duplicates: 0
Acquired TestDoc_Civil_Governance_Framework_2024.pdf -> /mnt/library/_ingest/TestDoc_Civil_Governance_Framework_2024.pdf [346a65d9]
```
### Verification
| Check | Result |
|-------|--------|
| File removed from `_acquired/` | YES |
| File present in `_ingest/` | YES |
| Catalogue entry (status=queued) | YES |
| Documents entry (status=queued) | YES |
| book_title = None (not enriched) | YES |
| organized_at = None | YES |
---
## RECON Pipeline Processing
The running RECON service (`recon.service`) automatically picked up the queued document.
### Timeline
| Stage | Timestamp | Duration |
|-------|-----------|----------|
| Queued | 05:50:14 | — |
| Extracted | 05:50:40 | 26s |
| Enriched | 05:51:05 | 25s |
| Embedded | 05:51:25 | 20s |
| **Total** | | **~71s** |
### Enrichment Results
| Field | Value |
|-------|-------|
| book_title | Civil Governance Framework Analysis |
| book_author | Dr. James Mitchell |
| pages_extracted | 1 |
| concepts_extracted | 2 |
| vectors_inserted | 2 |
| status | complete |
---
## Phase B: Library Placement
### Action
- Ran `ingest_scan()` again after enrichment completed
### Result: PASS
```
acquired: 0, placed: 1, skipped: 0, failed: 0, duplicates: 0
Placed 346a65d9 -> /mnt/library/Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf
[Civil Organization/Governance, step 1, 2 vectors]
```
### Verification
| Check | Result |
|-------|--------|
| File removed from `_ingest/` | YES |
| File at `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` | YES |
| Filename derived from book_title (not original filename) | YES |
| Domain: Civil Organization | YES |
| Subdomain: Governance | YES |
| Collision step: 1 (base, no collision) | YES |
| documents.path updated | YES |
| documents.organized_at set | YES |
| catalogue.path updated | YES |
| file_operations entry created (id=81) | YES |
| Qdrant filename = `Civil_Governance_Framework_Analysis.pdf` | YES |
| Qdrant original_filename = `TestDoc_Civil_Governance_Framework_2024.pdf` | YES |
| Qdrant download_url = `https://files.echo6.co/Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` | YES |
---
## Reverse Operation Test
### Action
- Ran `reverse_operation(81, db, config)`
### Result: PASS
```
Reversed operation 81: .../Civil_Governance_Framework_Analysis.pdf -> .../TestDoc_Civil_Governance_Framework_2024.pdf
```
### Verification
| Check | Result |
|-------|--------|
| File back in `_ingest/` | YES |
| File removed from `Civil-Organization/Governance/` | YES |
| file_operations.reversed_at set | YES |
| Qdrant payloads reverted to _ingest paths | YES |
| DB paths reverted to _ingest | YES |
---
## Re-placement After Reverse
### Action
- Cleared `organized_at` (simulating the fix applied to `reverse_operation`)
- Ran `ingest_scan()` again
### Result: PASS
```
acquired: 0, placed: 1, skipped: 0, failed: 0, duplicates: 0
Placed 346a65d9 -> /mnt/library/Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf
[Civil Organization/Governance, step 1, 2 vectors]
```
### Final State
- File at correct standardized location
- 2 file_operations records: #81 (reversed), #82 (active)
- Qdrant payloads correct
- All DB records consistent
---
## Bugs Found & Fixed During Test
### Bug 1: Phase B query overwhelmed by unorganized docs (FIXED)
**Problem:** `ingest_scan()` Phase B used `db.get_unorganized(limit=50)` which returns the 50 oldest unorganized docs. With 29,469 unorganized docs (mostly PeerTube transcripts), the test doc was never reached.
**Fix:** Added `StatusDB.get_ingest_pending(ingest_dir, limit=50)` method that filters by path (`WHERE path LIKE '/mnt/library/_ingest%'`). Updated `ingest_scan()` to use this instead.
**Files changed:**
- `/opt/recon/lib/status.py` — added `get_ingest_pending()` method
- `/opt/recon/lib/new_pipeline.py` — updated Phase B in `ingest_scan()`
### Bug 2: Reverse doesn't clear organized_at (FIXED)
**Problem:** After reversing a placement, `organized_at` remained set, preventing Phase B from re-triggering placement on the next watchdog cycle.
**Fix:** Added `UPDATE documents SET organized_at = NULL WHERE hash = ?` to `reverse_operation()`.
**Files changed:**
- `/opt/recon/lib/new_pipeline.py` — added organized_at clear in `reverse_operation()`
### Non-bug: Watchdog logging
**Observation:** `recon.py pipeline watch` produces no stdout/stderr output because `run_watchdog()` uses `logging.getLogger('recon.pipeline')` which only has handlers configured when `setup_logging()` is called for a parent logger during service mode. Not a functional issue — logs go to `/opt/recon/logs/recon.log` in service mode.
---
## Cleanup
- Pipeline disabled: `new_pipeline.enabled: false`
- Watchdog process killed
- Test document left in place at `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` (valid document, no reason to remove)
- Local copies synced
---
## Verdict: PASS
All phases of the two-phase ingest pipeline work correctly:
1. Phase A acquires files from `_acquired/` to `_ingest/` and queues for processing
2. RECON pipeline processes queued documents normally (extract → enrich → embed)
3. Phase B places enriched documents with standardized filenames derived from `book_title`
4. Reverse operation correctly undoes placement (file, DB, Qdrant)
5. Re-placement after reverse works correctly
6. Two bugs found and fixed during testing (query efficiency + organized_at reset)

696
vault/assets/echo6-custom.css
View file

@ -1,696 +0,0 @@
/* ═══════════════════════════════════════════════════════════════════
echo6 // searxng custom theme
═══════════════════════════════════════════════════════════════════
colors: cyan #28C0E8 / yellow #F0D848 (extracted from logo)
font: JetBrains Mono
aesthetic: cyberpunk — dark, sharp, clean, minimal glow
═══════════════════════════════════════════════════════════════════ */
@import url('https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@300;400;500;600;700&display=swap');
/* ─── css variables ─── */
:root {
/* echo6 brand */
--e6-cyan: #28C0E8;
--e6-cyan-light: #5DD4F5;
--e6-cyan-dim: rgba(40, 192, 232, 0.12);
--e6-yellow: #F0D848;
--e6-yellow-light: #F5E470;
--e6-yellow-dim: rgba(240, 216, 72, 0.1);
/* backgrounds */
--e6-bg: #0a0e17;
--e6-bg-card: #111827;
--e6-bg-hover: #1a2332;
--e6-bg-footer: #060a10;
/* borders */
--e6-border: #1e3a5f;
/* text */
--e6-text: #e0e6ed;
--e6-text-muted: #7a8ca0;
/* buttons */
--e6-btn-bg: #1a2332;
--e6-btn-hover: #243447;
}
/* ═══════════════════════════════════════════
1. global — font, background, text
═══════════════════════════════════════════ */
* {
font-family: 'JetBrains Mono', monospace !important;
}
html, body {
background-color: var(--e6-bg) !important;
color: var(--e6-text) !important;
margin: 0;
padding: 0;
}
/* ═══════════════════════════════════════════
2. homepage — viewport lock, centering
═══════════════════════════════════════════
homepage only — no scroll, logo centered
at ~35-40% from top like google.com.
CC: replace selectors if actual DOM differs.
═══════════════════════════════════════════ */
/* lock homepage to viewport — no scrollbar */
html:has(body.index),
body.index {
height: 100vh !important;
overflow: hidden !important;
margin: 0 !important;
padding: 0 !important;
}
/* center the main content area */
body.index main,
body.index #main_index,
body.index .search-margin {
display: flex !important;
flex-direction: column !important;
align-items: center !important;
justify-content: center !important;
height: calc(100vh - 80px) !important;
min-height: unset !important;
padding-bottom: 10vh !important;
margin: 0 !important;
overflow: hidden !important;
}
/* ═══════════════════════════════════════════
3. hide default searxng branding
═══════════════════════════════════════════
kill the giant "SearXNG" text behind the
logo. only the echo6 logo should show.
CC: inspect DOM and add the real selector
if these don't catch it.
═══════════════════════════════════════════ */
.searxng-wordmark,
#main-logo span,
.index h1,
h1.title,
.search-margin h1,
#main_index h1,
.index .title,
.title_h1 {
display: none !important;
visibility: hidden !important;
font-size: 0 !important;
color: transparent !important;
height: 0 !important;
overflow: hidden !important;
margin: 0 !important;
padding: 0 !important;
}
/* ═══════════════════════════════════════════
4. logo
═══════════════════════════════════════════ */
.logo img,
.search-margin img,
#main-logo img,
img[src*="searxng"] {
max-width: 270px;
height: auto;
display: block;
margin: 0 auto;
}
/* ═══════════════════════════════════════════
5. search bar — pill shape, subtle focus
═══════════════════════════════════════════ */
#search_form .search_box,
.search_box,
#q {
background-color: var(--e6-bg-card) !important;
border: 1px solid var(--e6-border) !important;
border-radius: 24px !important;
color: var(--e6-text) !important;
font-size: 16px !important;
}
/* focused state — subtle cyan border, barely-there shadow */
#search_form .search_box:focus-within,
.search_box:focus-within {
border-color: var(--e6-cyan) !important;
box-shadow: 0 0 6px var(--e6-cyan-dim) !important;
}
#search_form input[type="text"],
#search_form input[type="search"],
#q {
font-family: 'JetBrains Mono', monospace !important;
font-size: 16px !important;
color: var(--e6-text) !important;
background: transparent !important;
}
/* placeholder text */
#q::placeholder,
input[type="search"]::placeholder {
color: var(--e6-text-muted) !important;
opacity: 0.7;
}
/* search bar container width */
#search_form,
.search_box {
max-width: 584px !important;
margin-left: auto !important;
margin-right: auto !important;
}
/* ═══════════════════════════════════════════
6. search buttons
═══════════════════════════════════════════ */
#search_form button,
#search_form input[type="submit"],
.search_filters button,
.search_submit,
.search_box button {
background-color: var(--e6-btn-bg) !important;
border: 1px solid var(--e6-border) !important;
border-radius: 4px !important;
color: var(--e6-text) !important;
font-family: 'JetBrains Mono', monospace !important;
font-size: 14px !important;
cursor: pointer;
text-transform: lowercase;
}
#search_form button:hover,
#search_form input[type="submit"]:hover,
.search_submit:hover,
.search_box button:hover {
background-color: var(--e6-btn-hover) !important;
border-color: var(--e6-cyan) !important;
color: var(--e6-text) !important;
}
/* ═══════════════════════════════════════════
7. top navigation bar
═══════════════════════════════════════════
injected via template override.
styles the .//photos .//mail links, waffle
menu button, and login avatar.
═══════════════════════════════════════════ */
.echo6-nav {
display: flex;
justify-content: flex-end;
align-items: center;
padding: 10px 18px;
position: fixed;
top: 0;
left: 0;
right: 0;
z-index: 1000;
background-color: transparent;
gap: 20px;
}
/* .//photos and .//mail links */
.echo6-nav a.echo6-nav-link {
color: var(--e6-cyan) !important;
text-decoration: none;
font-size: 13px;
font-weight: 500;
letter-spacing: 0.03em;
text-transform: lowercase;
transition: color 0.15s ease;
}
.echo6-nav a.echo6-nav-link:hover {
color: var(--e6-cyan-light) !important;
}
/* waffle menu (⠿) button */
.echo6-waffle-btn {
display: flex;
align-items: center;
justify-content: center;
width: 36px;
height: 36px;
border-radius: 50%;
border: none;
background: transparent;
color: var(--e6-text-muted);
font-size: 20px;
cursor: pointer;
transition: background-color 0.15s ease;
}
.echo6-waffle-btn:hover {
background-color: var(--e6-bg-hover);
}
/* login avatar button */
.echo6-login-btn {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
border-radius: 50%;
border: 1px solid var(--e6-border);
background: transparent;
color: var(--e6-text-muted);
cursor: pointer;
transition: border-color 0.15s ease;
text-decoration: none;
}
.echo6-login-btn:hover {
border-color: var(--e6-cyan);
}
.echo6-login-btn svg {
width: 18px;
height: 18px;
fill: var(--e6-text-muted);
}
/* ═══════════════════════════════════════════
8. waffle menu dropdown
═══════════════════════════════════════════ */
.echo6-waffle-menu {
display: none;
position: fixed;
top: 52px;
right: 70px;
z-index: 2000;
background-color: var(--e6-bg-card);
border: 1px solid var(--e6-border);
border-radius: 12px;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.5);
padding: 16px;
min-width: 280px;
animation: echo6FadeIn 0.15s ease;
}
.echo6-waffle-menu.active {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 4px;
}
.echo6-waffle-menu a {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 12px 8px;
border-radius: 8px;
text-decoration: none;
color: var(--e6-text);
font-size: 11px;
font-weight: 400;
text-transform: lowercase;
transition: background-color 0.15s ease;
gap: 6px;
}
.echo6-waffle-menu a:hover {
background-color: var(--e6-bg-hover);
}
.echo6-waffle-menu a img {
width: 36px;
height: 36px;
border-radius: 6px;
object-fit: contain;
}
/* emoji fallback icons in waffle menu */
.echo6-waffle-menu a .echo6-icon {
font-size: 28px;
line-height: 1;
height: 36px;
display: flex;
align-items: center;
justify-content: center;
}
@keyframes echo6FadeIn {
from { opacity: 0; transform: translateY(-4px); }
to { opacity: 1; transform: translateY(0); }
}
/* waffle menu overlay — click to close */
.echo6-waffle-overlay {
display: none;
position: fixed;
inset: 0;
z-index: 1999;
}
.echo6-waffle-overlay.active {
display: block;
}
/* ═══════════════════════════════════════════
9. footer
═══════════════════════════════════════════ */
footer,
.footer,
#footer,
.searxng-footer {
background-color: var(--e6-bg-footer) !important;
border-top: 1px solid var(--e6-border) !important;
text-align: center;
padding: 12px 0 !important;
}
footer *,
.footer *,
#footer *,
.searxng-footer * {
font-size: 12px !important;
text-transform: lowercase;
color: var(--e6-text-muted) !important;
}
footer a,
.footer a,
#footer a,
.searxng-footer a {
color: var(--e6-text-muted) !important;
text-decoration: none;
transition: color 0.15s ease;
}
footer a:hover,
.footer a:hover,
#footer a:hover,
.searxng-footer a:hover {
color: var(--e6-cyan) !important;
}
/* ═══════════════════════════════════════════
10. search results page
═══════════════════════════════════════════
results pages must scroll normally.
dark theme applied to result elements.
═══════════════════════════════════════════ */
/* allow scrolling on non-homepage */
html:not(:has(body.index)),
body:not(.index) {
height: auto !important;
overflow: auto !important;
}
/* result links */
.result a,
.result-default a h3,
.result-title a {
color: var(--e6-cyan) !important;
}
.result a:hover,
.result-default a:hover h3,
.result-title a:hover {
color: var(--e6-cyan-light) !important;
}
.result a:visited,
.result-default a:visited h3 {
color: #8BA8C4 !important;
}
/* result URLs */
.result .url_wrapper,
.result .url,
.result-url {
color: var(--e6-text-muted) !important;
font-size: 13px !important;
}
/* result descriptions */
.result .content,
.result-content,
.result p {
color: var(--e6-text) !important;
}
/* result cards/containers */
.result,
.result-default {
background-color: transparent !important;
border-bottom: 1px solid var(--e6-border) !important;
padding: 14px 0 !important;
}
/* search categories/tabs bar */
.search_categories,
#categories,
.category {
background-color: var(--e6-bg) !important;
border-bottom: 1px solid var(--e6-border) !important;
}
.search_categories label,
.category a,
.category button {
color: var(--e6-text-muted) !important;
font-size: 13px !important;
text-transform: lowercase;
}
.search_categories label:hover,
.category a:hover,
.category button:hover {
color: var(--e6-text) !important;
}
/* active category tab */
.search_categories label.active,
.category.active a,
.category.active button,
.search_categories input:checked + label {
color: var(--e6-cyan) !important;
border-bottom: 2px solid var(--e6-cyan) !important;
}
/* sidebar / infobox */
.infobox,
#sidebar {
background-color: var(--e6-bg-card) !important;
border: 1px solid var(--e6-border) !important;
border-radius: 8px !important;
}
/* pagination */
.pagination button,
.pagination a,
#pagination button {
background-color: var(--e6-btn-bg) !important;
border: 1px solid var(--e6-border) !important;
color: var(--e6-text) !important;
border-radius: 4px !important;
}
.pagination button:hover,
.pagination a:hover,
#pagination button:hover {
border-color: var(--e6-cyan) !important;
color: var(--e6-cyan) !important;
}
/* engine stats / result info */
.result_header,
.result-engines,
.engines {
color: var(--e6-text-muted) !important;
font-size: 11px !important;
}
/* ═══════════════════════════════════════════
11. preferences page
═══════════════════════════════════════════ */
/* preferences containers */
.preferences,
.preferences fieldset,
.preferences form {
background-color: var(--e6-bg) !important;
color: var(--e6-text) !important;
}
.preferences fieldset {
border: 1px solid var(--e6-border) !important;
border-radius: 8px !important;
margin-bottom: 12px !important;
}
.preferences legend {
color: var(--e6-cyan) !important;
font-weight: 600;
}
/* select dropdowns and inputs in preferences */
.preferences select,
.preferences input[type="text"],
.preferences input[type="number"],
.preferences textarea {
background-color: var(--e6-bg-card) !important;
border: 1px solid var(--e6-border) !important;
color: var(--e6-text) !important;
border-radius: 4px !important;
font-family: 'JetBrains Mono', monospace !important;
}
.preferences select:focus,
.preferences input:focus,
.preferences textarea:focus {
border-color: var(--e6-cyan) !important;
box-shadow: 0 0 6px var(--e6-cyan-dim) !important;
}
/* preferences save button */
.preferences input[type="submit"],
.preferences button[type="submit"] {
background-color: var(--e6-cyan) !important;
border: none !important;
color: var(--e6-bg) !important;
font-weight: 600;
border-radius: 4px !important;
cursor: pointer;
padding: 8px 24px !important;
}
.preferences input[type="submit"]:hover,
.preferences button[type="submit"]:hover {
background-color: var(--e6-cyan-light) !important;
}
/* engine toggle checkboxes */
.preferences input[type="checkbox"]:checked {
accent-color: var(--e6-cyan) !important;
}
/* preferences tab navigation */
.preferences .tabs a,
.preferences .nav a {
color: var(--e6-text-muted) !important;
}
.preferences .tabs a:hover,
.preferences .nav a:hover {
color: var(--e6-text) !important;
}
.preferences .tabs a.active,
.preferences .nav a.active {
color: var(--e6-cyan) !important;
border-bottom: 2px solid var(--e6-cyan) !important;
}
/* ═══════════════════════════════════════════
12. scrollbar styling
═══════════════════════════════════════════ */
::-webkit-scrollbar {
width: 8px;
height: 8px;
}
::-webkit-scrollbar-track {
background: var(--e6-bg);
}
::-webkit-scrollbar-thumb {
background: var(--e6-border);
border-radius: 4px;
}
::-webkit-scrollbar-thumb:hover {
background: #2a4a6f;
}
/* ═══════════════════════════════════════════
13. selection highlight
═══════════════════════════════════════════ */
::selection {
background-color: rgba(40, 192, 232, 0.25);
color: var(--e6-text);
}
/* ═══════════════════════════════════════════
14. responsive
═══════════════════════════════════════════ */
@media (max-width: 768px) {
.echo6-nav {
padding: 8px 12px;
gap: 12px;
}
.echo6-nav a.echo6-nav-link {
font-size: 12px;
}
.echo6-waffle-menu {
right: 12px;
left: 12px;
min-width: unset;
}
.echo6-waffle-menu.active {
grid-template-columns: repeat(3, 1fr);
}
#search_form,
.search_box {
max-width: 100% !important;
margin-left: 12px !important;
margin-right: 12px !important;
}
.logo img,
img[src*="searxng"] {
max-width: 200px;
}
}
@media (max-width: 480px) {
.echo6-waffle-menu.active {
grid-template-columns: repeat(2, 1fr);
}
.echo6-nav a.echo6-nav-link {
font-size: 11px;
}
}
/* ═══════════════════════════════════════════
end // echo6-custom.css
═══════════════════════════════════════════ */

436
vault/assets/echo6-openwebui-theme.css
View file

@ -1,436 +0,0 @@
/*
* ╔══════════════════════════════════════════════════════════════╗
* ║ ECHO6 — Open WebUI Theme (Togglable) v2 ║
* ║ Cyberpunk terminal aesthetic ║
* ║ Cyan #28C0E8 · Yellow #F0D848 · JetBrains Mono ║
* ╚══════════════════════════════════════════════════════════════╝
*
* Activates when <html> has class "echo6" — toggled by the
* companion script echo6-theme-toggle.js
*/
/* ── Import JetBrains Mono ────────────────────────────────────── */
@import url('https://fonts.googleapis.com/css2?family=JetBrains+Mono:ital,wght@0,300;0,400;0,500;0,600;0,700;1,400&display=swap');
/* ── Palette Variables ────────────────────────────────────────── */
.echo6 {
--e6-cyan: #28C0E8;
--e6-cyan-dim: #1a8aa8;
--e6-cyan-glow: #28c0e815;
--e6-yellow: #F0D848;
--e6-yellow-dim: #c4b03a;
--e6-yellow-glow: #f0d84820;
--e6-bg-primary: #0a0e14;
--e6-bg-secondary: #0d1117;
--e6-bg-tertiary: #131920;
--e6-bg-elevated: #181f28;
--e6-text-primary: #c8d0d8;
--e6-text-secondary: #4a5568;
--e6-text-bright: #e2e8f0;
--e6-border: #1a2332;
--e6-border-subtle: #141c26;
--e6-font-mono: 'JetBrains Mono', 'Fira Code', 'Cascadia Code', monospace;
}
/* ══════════════════════════════════════════════════════════════
GLOBAL
══════════════════════════════════════════════════════════════ */
.echo6 body {
background-color: var(--e6-bg-primary) !important;
color: var(--e6-text-primary) !important;
}
/* Font — target text elements, not * (breaks icon fonts) */
.echo6 body,
.echo6 p,
.echo6 span,
.echo6 div,
.echo6 a,
.echo6 button,
.echo6 input,
.echo6 textarea,
.echo6 select,
.echo6 label,
.echo6 h1, .echo6 h2, .echo6 h3, .echo6 h4, .echo6 h5, .echo6 h6,
.echo6 li,
.echo6 td, .echo6 th,
.echo6 pre, .echo6 code {
font-family: var(--e6-font-mono) !important;
}
/* ══════════════════════════════════════════════════════════════
SIDEBAR — clean, no glow pills
══════════════════════════════════════════════════════════════ */
/* Sidebar background */
.echo6 #sidebar,
.echo6 [class*="sidebar"],
.echo6 aside,
.echo6 nav {
background-color: var(--e6-bg-secondary) !important;
}
/* All sidebar text — muted by default */
.echo6 #sidebar *,
.echo6 aside * {
color: var(--e6-text-secondary) !important;
}
/* Kill all existing backgrounds/glows on sidebar items */
.echo6 #sidebar a,
.echo6 #sidebar button,
.echo6 aside a,
.echo6 aside button {
background-color: transparent !important;
background: transparent !important;
border-radius: 0 !important;
border: none !important;
box-shadow: none !important;
transition: color 0.12s ease !important;
}
/* Sidebar hover — just brighten the text */
.echo6 #sidebar a:hover,
.echo6 #sidebar a:hover *,
.echo6 #sidebar button:hover,
.echo6 #sidebar button:hover *,
.echo6 aside a:hover,
.echo6 aside a:hover * {
color: var(--e6-cyan) !important;
background-color: transparent !important;
background: transparent !important;
box-shadow: none !important;
}
/* Active/selected chat — no background, just brighter text */
.echo6 #sidebar [class*="bg-"],
.echo6 #sidebar [aria-selected="true"] {
background-color: transparent !important;
background: transparent !important;
box-shadow: none !important;
}
.echo6 #sidebar [class*="bg-"] *,
.echo6 #sidebar [aria-selected="true"] * {
color: var(--e6-text-bright) !important;
}
/* Section headers (Chats, Folders, Today, Yesterday) */
.echo6 #sidebar .text-xs,
.echo6 aside .text-xs {
color: var(--e6-text-secondary) !important;
letter-spacing: 0.08em !important;
}
/* ══════════════════════════════════════════════════════════════
MAIN CONTENT AREA — aggressively override all backgrounds
══════════════════════════════════════════════════════════════ */
.echo6 main,
.echo6 div[class*="bg-white"],
.echo6 div[class*="bg-gray"] {
background-color: var(--e6-bg-primary) !important;
}
/* Target OWUI's main wrapper and content divs */
.echo6 #app > div,
.echo6 #app > div > div,
.echo6 [class*="h-screen"],
.echo6 [class*="h-full"],
.echo6 [class*="min-h-screen"] {
background-color: var(--e6-bg-primary) !important;
}
/* Model name display (center of page) */
.echo6 [class*="text-3xl"],
.echo6 [class*="text-2xl"] {
color: var(--e6-text-bright) !important;
}
/* ── Suggested prompts ────────────────────────────────────────── */
/* Target the suggestion container buttons specifically */
.echo6 [class*="suggestion"],
.echo6 [class*="Suggestion"],
.echo6 button[class*="cursor-pointer"][class*="flex"][class*="rounded-xl"],
.echo6 button[class*="cursor-pointer"][class*="flex"][class*="rounded-lg"],
.echo6 [class*="bg-gray"][class*="cursor-pointer"][class*="rounded"] {
background-color: var(--e6-bg-tertiary) !important;
background: var(--e6-bg-tertiary) !important;
border: 1px solid var(--e6-border) !important;
color: var(--e6-text-primary) !important;
}
.echo6 [class*="suggestion"]:hover,
.echo6 [class*="Suggestion"]:hover,
.echo6 button[class*="cursor-pointer"][class*="flex"][class*="rounded-xl"]:hover,
.echo6 button[class*="cursor-pointer"][class*="flex"][class*="rounded-lg"]:hover,
.echo6 [class*="bg-gray"][class*="cursor-pointer"][class*="rounded"]:hover {
border-color: var(--e6-cyan) !important;
background-color: var(--e6-bg-elevated) !important;
background: var(--e6-bg-elevated) !important;
}
/* ══════════════════════════════════════════════════════════════
INPUT AREA
══════════════════════════════════════════════════════════════ */
.echo6 textarea,
.echo6 #chat-textarea,
.echo6 [contenteditable] {
background-color: var(--e6-bg-tertiary) !important;
border-color: var(--e6-border) !important;
color: var(--e6-text-bright) !important;
caret-color: var(--e6-cyan) !important;
}
.echo6 textarea:focus,
.echo6 #chat-textarea:focus,
.echo6 [contenteditable]:focus {
border-color: var(--e6-cyan) !important;
box-shadow: 0 0 0 1px var(--e6-cyan-glow) !important;
outline: none !important;
}
/* Input wrapper bar */
.echo6 [class*="bottom-0"],
.echo6 [class*="sticky"][class*="bottom"] {
background-color: var(--e6-bg-primary) !important;
}
/* Action icons in input row */
.echo6 textarea ~ div button,
.echo6 [class*="input"] button {
color: var(--e6-text-secondary) !important;
}
.echo6 textarea ~ div button:hover,
.echo6 [class*="input"] button:hover {
color: var(--e6-cyan) !important;
}
/* ══════════════════════════════════════════════════════════════
MESSAGE BUBBLES
══════════════════════════════════════════════════════════════ */
.echo6 [data-role="user"] > div {
background-color: var(--e6-bg-elevated) !important;
border: 1px solid var(--e6-border) !important;
border-radius: 8px !important;
}
.echo6 [data-role="assistant"] > div {
background-color: transparent !important;
}
.echo6 [data-role] * {
color: var(--e6-text-primary) !important;
}
/* ══════════════════════════════════════════════════════════════
TOP BAR / HEADER
══════════════════════════════════════════════════════════════ */
.echo6 header,
.echo6 [class*="top-0"][class*="sticky"],
.echo6 [class*="top-0"][class*="fixed"] {
background-color: var(--e6-bg-secondary) !important;
border-bottom: 1px solid var(--e6-border-subtle) !important;
}
.echo6 header button,
.echo6 header a,
.echo6 header span {
color: var(--e6-text-primary) !important;
}
.echo6 header [class*="text-xs"],
.echo6 header [class*="text-gray"] {
color: var(--e6-text-secondary) !important;
}
/* ══════════════════════════════════════════════════════════════
CODE BLOCKS
══════════════════════════════════════════════════════════════ */
.echo6 pre {
background-color: var(--e6-bg-secondary) !important;
border: 1px solid var(--e6-border) !important;
border-radius: 6px !important;
color: var(--e6-text-primary) !important;
}
.echo6 pre code {
background-color: transparent !important;
border: none !important;
color: inherit !important;
}
.echo6 :not(pre) > code {
background-color: var(--e6-bg-elevated) !important;
color: var(--e6-yellow) !important;
padding: 0.15em 0.4em !important;
border-radius: 3px !important;
font-size: 0.9em !important;
border: none !important;
}
/* ══════════════════════════════════════════════════════════════
BUTTONS
══════════════════════════════════════════════════════════════ */
.echo6 button[type="submit"],
.echo6 button[class*="primary"] {
background-color: var(--e6-cyan) !important;
color: var(--e6-bg-primary) !important;
}
.echo6 button[type="submit"]:hover,
.echo6 button[class*="primary"]:hover {
background-color: var(--e6-cyan-dim) !important;
}
/* ══════════════════════════════════════════════════════════════
DROPDOWNS / MENUS / MODALS
══════════════════════════════════════════════════════════════ */
.echo6 [role="dialog"],
.echo6 [role="menu"],
.echo6 [role="listbox"],
.echo6 [class*="dropdown"],
.echo6 [class*="modal"],
.echo6 [class*="popover"] {
background-color: var(--e6-bg-tertiary) !important;
border: 1px solid var(--e6-border) !important;
}
.echo6 [role="option"]:hover,
.echo6 [role="menuitem"]:hover {
background-color: var(--e6-bg-elevated) !important;
color: var(--e6-cyan) !important;
}
.echo6 select {
background-color: var(--e6-bg-tertiary) !important;
border-color: var(--e6-border) !important;
color: var(--e6-text-primary) !important;
}
/* ══════════════════════════════════════════════════════════════
SCROLLBAR
══════════════════════════════════════════════════════════════ */
.echo6 ::-webkit-scrollbar {
width: 5px !important;
height: 5px !important;
}
.echo6 ::-webkit-scrollbar-track {
background: transparent !important;
}
.echo6 ::-webkit-scrollbar-thumb {
background: var(--e6-border) !important;
border-radius: 3px !important;
}
.echo6 ::-webkit-scrollbar-thumb:hover {
background: var(--e6-cyan-dim) !important;
}
/* ══════════════════════════════════════════════════════════════
LINKS
══════════════════════════════════════════════════════════════ */
.echo6 a:not(#sidebar a):not(aside a):not(nav a):not(#echo6-toggle) {
color: var(--e6-cyan) !important;
}
.echo6 a:not(#sidebar a):not(aside a):not(nav a):not(#echo6-toggle):hover {
color: var(--e6-yellow) !important;
}
/* ══════════════════════════════════════════════════════════════
MISC
══════════════════════════════════════════════════════════════ */
.echo6 ::selection {
background-color: var(--e6-cyan) !important;
color: var(--e6-bg-primary) !important;
}
.echo6 ::placeholder {
color: var(--e6-text-secondary) !important;
opacity: 0.7 !important;
}
.echo6 [class*="badge"],
.echo6 [class*="tag"],
.echo6 [class*="chip"] {
background-color: var(--e6-yellow-glow) !important;
color: var(--e6-yellow) !important;
border: 1px solid var(--e6-yellow-dim) !important;
}
.echo6 [class*="spinner"],
.echo6 [class*="loading"] {
border-color: var(--e6-border) !important;
border-top-color: var(--e6-cyan) !important;
}
.echo6 input[type="checkbox"]:checked + * {
background-color: var(--e6-cyan) !important;
}
.echo6 [role="tooltip"] {
background-color: var(--e6-bg-elevated) !important;
color: var(--e6-text-primary) !important;
border: 1px solid var(--e6-border) !important;
}
/* ══════════════════════════════════════════════════════════════
TOGGLE BUTTON (injected by echo6-theme-toggle.js)
══════════════════════════════════════════════════════════════ */
#echo6-toggle {
position: fixed;
bottom: 16px;
right: 16px;
z-index: 99999;
width: 36px;
height: 36px;
border-radius: 6px;
border: 1px solid #1e2a36;
background-color: #0f1419;
color: #6e7a88;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
font-size: 14px;
font-weight: 700;
transition: all 0.2s ease;
opacity: 0.4;
user-select: none;
line-height: 1;
padding: 0;
}
#echo6-toggle:hover {
opacity: 1;
border-color: #28C0E8;
color: #28C0E8;
box-shadow: 0 0 12px #28c0e840;
}
#echo6-toggle.active {
opacity: 0.7;
background-color: #28C0E8;
color: #0a0e14;
border-color: #28C0E8;
}
#echo6-toggle.active:hover {
opacity: 1;
box-shadow: 0 0 16px #28c0e860;
}

49
vault/assets/echo6-theme-toggle.js
View file

@ -1,49 +0,0 @@
/*
* Echo6 Theme Toggle for Open WebUI
* Adds a small button (bottom-right) that toggles the .echo6 class
* on <html>, activating/deactivating the companion CSS theme.
* Persists preference in localStorage.
*/
(function () {
'use strict';
var STORAGE_KEY = 'echo6-theme-active';
var html = document.documentElement;
// Restore saved state immediately (before paint if possible)
var saved = localStorage.getItem(STORAGE_KEY);
if (saved === 'true') {
html.classList.add('echo6');
}
function createToggle() {
// Don't double-inject
if (document.getElementById('echo6-toggle')) return;
var btn = document.createElement('button');
btn.id = 'echo6-toggle';
btn.textContent = 'E6';
btn.title = 'Toggle Echo6 theme';
btn.setAttribute('aria-label', 'Toggle Echo6 theme');
// Sync active state with current class
if (html.classList.contains('echo6')) {
btn.classList.add('active');
}
btn.addEventListener('click', function () {
var isActive = html.classList.toggle('echo6');
btn.classList.toggle('active', isActive);
localStorage.setItem(STORAGE_KEY, isActive ? 'true' : 'false');
});
document.body.appendChild(btn);
}
// Inject once DOM is ready
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', createToggle);
} else {
createToggle();
}
})();

BIN
vault/assets/echo6_favicon.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 1.7 KiB

BIN
vault/assets/echo6_favicon_32x32.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 1.9 KiB

BIN
vault/assets/echo6_logo.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 69 KiB

BIN
vault/assets/echo6yellow_logo_150x29.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 5 KiB

BIN
vault/assets/echo6yellow_logo_422x422_square.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 64 KiB

BIN
vault/assets/echo6yellow_logo_422x81.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 18 KiB

270
vault/assets/key_manager.py
View file

@ -1,270 +0,0 @@
"""
RECON Key Manager - Thread-safe API key management with hot-reload.
Provides a singleton KeyManager that workers (enricher, extractor) read from
instead of loading .env directly. Dashboard can update keys at runtime without
restarting the service.
Dependencies: None beyond stdlib + requests (already in requirements.txt)
Config: Reads/writes /opt/recon/.env
"""
import os
import re
import time
import logging
import threading
import requests
logger = logging.getLogger('recon.key_manager')
class KeyManager:
"""Thread-safe API key store with hot-reload and validation."""
_instance = None
_lock = threading.Lock()
def __new__(cls):
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._initialized = False
return cls._instance
def __init__(self):
if self._initialized:
return
self._keys_lock = threading.RLock()
self._gemini_keys = []
self._env_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), '.env')
self._last_loaded = None
self._key_stats = {} # key_index -> {calls, errors, last_used}
self._load_from_env()
self._initialized = True
logger.info(f"KeyManager initialized with {len(self._gemini_keys)} Gemini key(s)")
# ── Read Operations ──
def get_gemini_keys(self):
"""Return a copy of current Gemini keys. Thread-safe."""
with self._keys_lock:
return list(self._gemini_keys)
def get_gemini_key(self, index=0):
"""Get a single Gemini key by index. Returns None if out of range."""
with self._keys_lock:
if 0 <= index < len(self._gemini_keys):
return self._gemini_keys[index]
return None
def get_gemini_key_count(self):
"""Return number of loaded Gemini keys."""
with self._keys_lock:
return len(self._gemini_keys)
def get_masked_keys(self):
"""Return keys masked for display: first 8 + ... + last 4 chars."""
with self._keys_lock:
result = []
for i, key in enumerate(self._gemini_keys):
if len(key) > 16:
masked = key[:8] + '...' + key[-4:]
elif len(key) > 8:
masked = key[:4] + '...' + key[-2:]
else:
masked = '****'
stats = self._key_stats.get(i, {})
result.append({
'index': i,
'masked': masked,
'length': len(key),
'calls': stats.get('calls', 0),
'errors': stats.get('errors', 0),
'last_used': stats.get('last_used', None),
'valid': stats.get('valid', None),
'last_validated': stats.get('last_validated', None),
})
return result
# ── Write Operations (all persist to .env) ──
def set_gemini_keys(self, keys):
"""Replace all Gemini keys. Persists to .env. Returns success bool."""
# Filter empty strings
keys = [k.strip() for k in keys if k.strip()]
with self._keys_lock:
self._gemini_keys = keys
self._key_stats = {} # Reset stats on full replace
self._persist_to_env()
logger.info(f"Gemini keys replaced: {len(keys)} key(s) loaded")
return True
def add_gemini_key(self, key):
"""Add a single Gemini key. Persists to .env. Returns new index."""
key = key.strip()
if not key:
raise ValueError("Key cannot be empty")
with self._keys_lock:
# Check for duplicates
if key in self._gemini_keys:
raise ValueError("Key already exists")
self._gemini_keys.append(key)
idx = len(self._gemini_keys) - 1
self._persist_to_env()
logger.info(f"Gemini key added at index {idx}")
return idx
def remove_gemini_key(self, index):
"""Remove a Gemini key by index. Persists to .env. Returns removed key (masked)."""
with self._keys_lock:
if index < 0 or index >= len(self._gemini_keys):
raise IndexError(f"Key index {index} out of range (have {len(self._gemini_keys)} keys)")
if len(self._gemini_keys) <= 1:
raise ValueError("Cannot remove last key — pipeline needs at least 1 Gemini key")
key = self._gemini_keys.pop(index)
# Rebuild stats with shifted indices
new_stats = {}
for i, stats in self._key_stats.items():
if i < index:
new_stats[i] = stats
elif i > index:
new_stats[i - 1] = stats
self._key_stats = new_stats
self._persist_to_env()
masked = key[:8] + '...' + key[-4:] if len(key) > 16 else '****'
logger.info(f"Gemini key removed at index {index}: {masked}")
return masked
def replace_gemini_key(self, index, new_key):
"""Replace a single Gemini key at index. Persists to .env."""
new_key = new_key.strip()
if not new_key:
raise ValueError("Key cannot be empty")
with self._keys_lock:
if index < 0 or index >= len(self._gemini_keys):
raise IndexError(f"Key index {index} out of range")
# Check duplicate (but allow replacing with same key)
if new_key in self._gemini_keys and self._gemini_keys[index] != new_key:
raise ValueError("Key already exists at another index")
self._gemini_keys[index] = new_key
if index in self._key_stats:
self._key_stats[index] = {} # Reset stats for replaced key
self._persist_to_env()
logger.info(f"Gemini key replaced at index {index}")
# ── Validation ──
def validate_key(self, key):
"""
Test a Gemini API key by listing models.
Returns (valid: bool, message: str).
"""
try:
resp = requests.get(
f"https://generativelanguage.googleapis.com/v1beta/models?key={key}",
timeout=10
)
if resp.status_code == 200 and 'models' in resp.text:
return True, "Valid — API responded"
elif resp.status_code == 400:
return False, f"Invalid key (HTTP {resp.status_code})"
elif resp.status_code == 403:
return False, "Key disabled or quota exhausted"
elif resp.status_code == 429:
return True, "Valid — but currently rate-limited"
else:
return False, f"Unexpected response (HTTP {resp.status_code})"
except requests.Timeout:
return False, "Timeout — could not reach Gemini API"
except requests.ConnectionError:
return False, "Connection error — check network"
except Exception as e:
return False, f"Error: {str(e)}"
def validate_all(self):
"""Validate all loaded Gemini keys. Returns list of results."""
results = []
with self._keys_lock:
keys_copy = list(enumerate(self._gemini_keys))
for i, key in keys_copy:
valid, message = self.validate_key(key)
with self._keys_lock:
if i not in self._key_stats:
self._key_stats[i] = {}
self._key_stats[i]['valid'] = valid
self._key_stats[i]['last_validated'] = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
results.append({'index': i, 'valid': valid, 'message': message})
time.sleep(0.2) # Don't hammer the API
return results
# ── Stats tracking (called by enricher/extractor) ──
def record_usage(self, key_index, success=True):
"""Record a key usage event. Called by workers after each Gemini call."""
with self._keys_lock:
if key_index not in self._key_stats:
self._key_stats[key_index] = {'calls': 0, 'errors': 0}
self._key_stats[key_index]['calls'] = self._key_stats[key_index].get('calls', 0) + 1
if not success:
self._key_stats[key_index]['errors'] = self._key_stats[key_index].get('errors', 0) + 1
self._key_stats[key_index]['last_used'] = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
# ── Internal ──
def _load_from_env(self):
"""Load Gemini keys from .env file."""
keys = []
if os.path.exists(self._env_path):
with open(self._env_path, 'r') as f:
for line in f:
line = line.strip()
if line and not line.startswith('#'):
match = re.match(r'^GEMINI_KEY(?:_\d+)?=(.+)$', line)
if match:
val = match.group(1).strip().strip('"').strip("'")
if val:
keys.append(val)
self._gemini_keys = keys
self._last_loaded = time.time()
def _persist_to_env(self):
"""Write current keys back to .env file, preserving non-Gemini lines."""
other_lines = []
if os.path.exists(self._env_path):
with open(self._env_path, 'r') as f:
for line in f:
stripped = line.strip()
if stripped and not re.match(r'^GEMINI_KEY', stripped):
other_lines.append(line.rstrip('\n'))
with open(self._env_path, 'w') as f:
# Write non-Gemini lines first
for line in other_lines:
f.write(line + '\n')
# Write Gemini keys
for i, key in enumerate(self._gemini_keys, 1):
f.write(f'GEMINI_KEY_{i}={key}\n')
self._last_loaded = time.time()
logger.info(f"Persisted {len(self._gemini_keys)} Gemini key(s) to {self._env_path}")
def reload_from_env(self):
"""Force reload from .env (e.g., if edited externally)."""
with self._keys_lock:
self._load_from_env()
logger.info(f"Reloaded {len(self._gemini_keys)} Gemini key(s) from .env")
return len(self._gemini_keys)
# Module-level convenience — import and use anywhere
_manager = None
def get_key_manager():
"""Get the singleton KeyManager instance."""
global _manager
if _manager is None:
_manager = KeyManager()
return _manager

286
vault/docs/hardware/environment.md
View file

@ -1,286 +0,0 @@
---
title: Echo6 Environment Reference
type: reference
tags:
- proxmox
aliases: []
related:
- [[fleet-platform-baseline]]
- [[fleet-storage-memory-upgrade]]
- [[ip-allocation]]
- [[toc-cortex-pve9.2-update]]
- [[pve-guest-park-and-adopt]]
updated: 2026-08-15
---
# Echo6 Environment Reference
## Proxmox Cluster (echo6-cluster)
Five nodes running Proxmox VE:
| Node | Local IP | Tailscale | Hardware | RAM | Purpose |
| ------- | ------------- | ----------- | ----------------------------------------------- | -------------- | -------------------------------- |
| data | 192.168.1.240 | 100.64.0.6 | AMD Ryzen 7 PRO 5750GE, 1TB NVMe + 1TB SATA SSD | 32GB DDR4-3200 | Database [[services]] |
| utility | 192.168.1.241 | 100.64.0.5 | AMD Ryzen 7 PRO 5750GE, 512GB NVMe | 32GB DDR4-3200 | Utility [[services]], monitoring |
| cloud | 192.168.1.242 | 100.64.0.4 | Intel i7-12700T, 512GB NVMe | 32GB DDR4-3200 | Cloud storage, personal [[services]] |
| media | 192.168.1.243 | 100.64.0.3 | Intel i7-14700T, 512GB + 2TB NVMe | 64GB DDR5-4800 | Media server, *arr stack |
| toc | 192.168.1.244 | 100.64.0.13 | Workstation (i9-10900X) | 64GB DDR4 | GPU compute, AI/ML workloads |
### Node Storage Details
| Node | Primary Disk | Secondary Disk |
|------|-------------|----------------|
| data | Samsung SSD 980 1TB (NVMe) | SanDisk SDSSDH3 1TB (SATA SSD) |
| utility | WD PC SN740 512GB (NVMe) | — |
| cloud | SK Hynix HFS512GEJ9X164N 512GB (NVMe) | — |
| media | Intel SSDPEKNU512GZH 512GB (NVMe, boot/`pve`) | WD Green SN350 2TB (NVMe, VG `tank`) |
| toc | 512GB NVMe | — |
**media upgraded 2026-08-15:** 32 → 64 GB DDR5-4800 (2 × 32 GB, board maximum reached), and the second M.2 — which held a leftover BitLocker Windows install — replaced with a 2 TB WD Green SN350. That drive is VG `tank` as a **single 1.82 TB volume** — `tank/data`, ext4, mounted `/mnt/nvme2tb`, registered as Proxmox dir storage `media-nvme` (`nodes media`-scoped). Deliberately not split into separate pools: [[navi]]'s geo data is the intended consumer and it needs one large contiguous filesystem it can grow into. The displaced Windows data was archived to pi-nas first — see [[fleet-storage-memory-upgrade]].
**Free positions (verified live 2026-08-14):** data has none — only three external PCIe root ports exist and all are populated (NVMe, NIC, Wi-Fi), and the Wi-Fi M.2 is E-keyed so it cannot take a storage drive. utility and cloud each have an empty 2.5" SATA bay. media's second M.2 holds a leftover BitLocker Windows install (serial `PHKA142402U8512A`; the Proxmox drive is `PHKA142504HP512A`). toc has 8 unpopulated SATA ports and 4 free PCIe slots. pi-nas has one free SATA port (`ata5`). Placement plan is [[fleet-storage-memory-upgrade]].
**Memory ceilings:** data and utility are hard-capped at 32 GB and already there. cloud and media max at 64 GB. toc has 2 of 8 DIMM slots free but takes full-size DDR4 DIMMs, not SODIMM, and currently runs all modules at 2133 MT/s because rated speeds are mixed (3200/2666/2133).
### Cluster Backup Storage
`pinas-backup` — NFS `192.168.1.245:/export/pvebackup`, `vers=3`, content `backup`, mounted at `/mnt/pve/pinas-backup` on all five nodes. Backed by pi-nas `sdc1` (~19 TB free), deliberately not `sdd1` which carries PeerTube. Added 2026-08-14 for park-and-adopt guest relocation — see [[pve-guest-park-and-adopt]] and [[omv-add-nfs-share]].
### Non-Cluster Physical Hosts
| Host | Local IP | Tailscale | Hardware |
|------|----------|-----------|----------|
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi 5, 8GB soldered, JMicron JMB58x 5-port SATA HBA, 2x 3TB btrfs RAID1 + 2x 24TB single ext4 |
| aida-nebra | 192.168.1.253 | 100.64.0.9 | Raspberry Pi Compute Module 3, Cortex-A53, 906MiB soldered, eMMC boot, RAK4631 on ttyACM1 |
Neither is upgradable — soldered RAM, no DIMM sockets.
### Node Hardware Identifiers
| Node | Make/Model | Serial/Service Tag |
|------|-----------|---------------------|
| data | Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) | MZ010LPV |
| utility | Lenovo ThinkCentre M75q Gen 2 (11JN002RUS) | MJ0LZNYT |
| cloud | Lenovo ThinkCentre M70q Gen 3 (11T3000RUS) | MJ0LQCGJ |
| media | Dell OptiPlex Micro 7020 | 58SM6X3 (BIOS 1.20.0, 2025-09-04) |
| toc | HP Z4 G4 Workstation | MXL2383MVK |
**No BMC/IPMI/iDRAC on any node** (verified `dmidecode -t 38` empty on all five, 2026-07-14) — there is no remote power-cycle path for any Proxmox host. Recovery from a hung or powered-off node requires physical access.
### Network Notes
- **media NIC:** Original Intel e1000e NIC (`nic0`, MAC `e8:cf:83:20:8b:cb`) crashes under sustained NFS load — present but DOWN/unused. Sole uplink is a USB Realtek RTL8153 GbE dongle (MAC `0c:37:96:0e:e8:53`) on vmbr0.
- **Tailscale [[dns]] bootstrap:** All LXC containers with Tailscale have a systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that ensures fallback [[dns]] exists before tailscaled starts, preventing chicken-and-egg [[dns]] resolution failures on reboot
### TOC Node Details
- **Hardware:** Intel i9-10900X (20 threads), 64GB RAM (4×8GB + 2×16GB DDR4), 512GB NVMe, RTX A4000
- **GPU:** Passed through via VFIO to VM 150 (cortex), not used on host
- **VMID ranges:** 100-149 (LXC), 150-199 (VMs)
- **Presave backup:** `/home/zvx/toc-presave/` on Contabo (1.8G) — contains old Ubuntu config
## Virtual Machines
| VM | Host | VMID | Local IP | Tailscale | Purpose |
|----|------|------|----------|-----------|---------|
| cortex | toc | 150 | 192.168.1.150 | 100.64.0.14 | GPU compute — Open WebUI, Ollama, Qdrant, TEI, Claude Code |
| recon-vm | data | 1130 | 192.168.1.130 | 100.64.0.24 | [[recon]] + [[navi]] platforms, Files, Kiwix, geo backends |
| arr | media | 105 | 192.168.1.160 | 100.64.0.18 | ARR media automation stack (Jellyfin, Sonarr, Radarr, etc.) |
### cortex VM Details
- **OS:** Ubuntu 24.04 (cloud-init), kernel 6.8.0-100-generic
- **Resources:** 16 threads, 32GB RAM, 300GB disk
- **Swap:** 32GB swapfile (`/swapfile`), `vm.swappiness=10` (prefer RAM, swap under pressure only). Provisioned 2026-04-23 to support bursty jobs (gdal_contour, tippecanoe) alongside Docker ML containers.
- **GPU:** RTX A4000 (passthrough), NVIDIA driver 580.126.09, CUDA 13.0
- **Software:** Docker 29.2.1 + nvidia-container-toolkit 1.18.2, Node.js 22.22.0, Python 3.12.3
- **Docker containers:** open-webui (8080), ollama (11434 w/ GPU), qdrant (6333), tei (8090)
- **User:** zvx (sudo, SSH keys from cluster)
- **Claude Code:** installed
### recon-vm Details
- **OS:** Ubuntu 24.04.4 LTS (cloud-init), kernel 6.8.0-110-generic
- **Resources:** 4 cores, 24GB RAM, 180GB disk
- **Software:** Docker 29.4.0, Python 3.12.3, nginx, sqlite3, Tailscale
- **Platforms:** [[recon]] (knowledge extraction pipeline, :8420) and [[navi]] (offline navigation, navi.echo6.co, :8440) with geo backends (Valhalla :8002, Nominatim :8010, Photon :2322, PostgreSQL/PostGIS :5432)
- **Systemd [[services]]:** [[recon]] (8420), recon-watchdog, kiwix (8430), nginx (8888)
- **NFS mounts:** pi-nas:/export/library → /mnt/library, /mnt/nav, /mnt/kiwix
- **User:** zvx (sudo, SSH key auth)
- **Migrated from:** CT 130 (LXC) on 2026-04-19. Tailscale identity preserved (100.64.0.24).
- **Note:** Old CT 130 (LXC) has since been destroyed (verified 2026-06-15) — only VM 1130 remains.
### arr VM Details
- **OS:** Ubuntu 24.04 (cloud-init)
- **Resources:** 4 cores, 8GB RAM, 30GB disk on local-lvm
- **Software:** Docker 29.2.1, Tailscale, NFS client, sshpass, qemu-guest-agent
- **User:** zvx (sudo, SSH key from cortex)
- **NFS:** pi-nas:/export/arr → /mnt/arr (22TB, movies/tv/downloads)
- **Docker containers:** jellyfin (8096), jellyseer (5055), sonarr (8989), radarr (7878), prowlarr (9696), sabnzbd (8080), lidarr (8686/6595), navidrome (4533)
- **Docker network:** arr-net (bridge)
- **Config dirs:** /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd,lidarr}
## Key Servers
| Server | Local IP | Tailscale | Purpose |
|--------|----------|-----------|---------|
| aida-nebra | 192.168.1.253 | 100.64.0.9 | AIDA-N2(RPT,LLM) — meshtasticd node !27780c47, Nebra 2W hat, port 4403. [[meshai]] (CT 108) connects here via TCP |
| mt-isr | 192.168.1.141 | 100.100.0.5 (IdahoMesh) | Meshtastic sidecar Pi (G2 WiFi bridge, meshtasticd, CLI) |
| mt-burleybutte | 192.168.1.185 | — | Meshtastic node (meshtasticd, Nebra 2W hat, IdahoMesh VPN) |
| pi-nas | 192.168.1.245 | 100.64.0.21 | Raspberry Pi NAS |
| matt-desktop | 192.168.1.254 | 100.64.0.10 | Personal workstation (Windows, your PC) |
| ha | 192.168.1.151 | 100.64.0.16 | Home Assistant (VM 151 on cloud, Docker, home automation) |
| **edge1** (rebuilt Contabo VPS) | 5.189.158.149 | 100.64.0.40 | Debian 12 + Proxmox 8.4.19, **mail-only** — Mailcow in CT 101; host [[caddy]] + mailcow-dnat.service; rebuilt [[2026-06-19]] |
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB — **permanent front door** for vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh.com |
*Last updated: [[2026-06-19]] — Contabo VPS rebuilt as edge1 (mail-only, Debian 12 + Proxmox 8.4.19, 5.189.158.149 / tailnet 100.64.0.40); Mailcow CT 101 (10.10.10.2) on edge1; edge2 is now the permanent front door for all other [[services]]; Headscale node `contabo` moved to 100.64.0.40; previously added edge2 CT 107 (headscale), CT 106 (matrix), CT 105 ([[authentik]]), CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
## LXC Containers
| Container | Host | Local IP | Tailscale | Purpose |
|-----------|------|----------|-----------|---------|
| meshmonitor | utility (CT 100) | 192.168.1.100 | 100.64.0.7 | Meshtastic mesh monitoring (zvx-echo6/meshmonitor fork, multi-channel) |
| [[caddy]] | utility (CT 101) | 192.168.1.101 | 100.64.0.8 | Home reverse proxy |
| [[searxng]] | utility (CT 102) | 192.168.1.102 | 100.64.0.15 | Echo6 Search homepage ([[searxng]], echo6.co) |
| immich | cloud (CT 120) | 192.168.1.182 | 100.64.0.2 | Immich photo management |
| nextcloud | cloud (CT 121) | 192.168.1.183 | 100.64.0.11 | Nextcloud AIO |
| meshtastic-hs | utility (CT 106) | 192.168.1.106 | — | IdahoMesh Headscale VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | 100.64.0.22 | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| [[meshai]] | utility (CT 108) | 192.168.1.144 | 100.64.0.32 | [[meshai]] - LLM-powered Meshtastic assistant |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | — | [[archivist]] knowledge pipeline |
| [[argus]] | utility (CT 103) | 192.168.1.103 | 100.64.0.25 | [[argus]] - OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104 | 100.64.0.12 | Data-hub spine (central.echo6.mesh) — ~25 adapters, NATS/JetStream, TimescaleDB/PostGIS — see [[central]] |
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.17 | PeerTube video streaming — Tailscale identity is `peertube-4hve9pdr` (collision-suffixed) since ~2026-06-22; `.23` is stale/gone |
| mcc | media (CT 111) | 192.168.1.111 | 100.64.0.19 | pymc console web app ([[caddy]] + Postfix, /api+/auth+/ws → aida-nebra :8000) |
| mailcow | edge1 (CT 101) | 10.10.10.2 | — | Mailcow email server (privileged LXC, mail-only host; reached via host DNAT + [[caddy]]) |
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | Grav CMS 2.0.11 for idahomesh.com (migrated from WordPress 2026-07-17; hostname `wordpress` unchanged; MariaDB purged — flat-file, no DB) |
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| [[authentik]] | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | [[authentik]] SSO platform (migrated from Contabo 2026-06-18) |
| matrix | edge2 (CT 106) | 10.10.10.24 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
| headscale | edge2 (CT 107) | 10.10.10.25 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo [[2026-06-19]]) |
> **Note ([[2026-06-19]]):** edge2 CT placements CT 102–107 confirmed; Forge git-SSH DNAT (`forgejo-ssh-dnat.service`) is a permanent systemd unit on edge2 host.
## IP Allocation Scheme
| Range | Purpose |
|-------|---------|
| .1-.10 | Network infrastructure |
| .11-.99 | DHCP clients |
| .100-.149 | LXC containers |
| .150-.199 | VMs |
| .240-.250 | Proxmox hosts + bare metal |
| .251-.254 | Meshtastic nodes |
Full details: [[ip-allocation]]
## Headscale Node List
Current registered nodes (25 total):
| Node | Tailscale IP | Type |
|------|-------------|------|
| contabo | 100.64.0.40 | VPS (edge1, mail-only, rebuilt [[2026-06-19]]; formerly 100.64.0.1) |
| immich | 100.64.0.2 | LXC |
| media | 100.64.0.3 | Proxmox |
| cloud | 100.64.0.4 | Proxmox |
| utility | 100.64.0.5 | Proxmox |
| data | 100.64.0.6 | Proxmox |
| meshmonitor | 100.64.0.7 | LXC |
| [[caddy]] | 100.64.0.8 | LXC |
| aida-nebra | 100.64.0.9 | Pi |
| matt-desktop | 100.64.0.10 | Desktop |
| nextcloud | 100.64.0.11 | LXC |
| toc | 100.64.0.13 | Proxmox |
| cortex | 100.64.0.14 | VM |
| [[searxng]] | 100.64.0.15 | LXC |
| ha | 100.64.0.16 | VM (cloud VM 151 — node id 54, user echo6; home automation) |
| arr | 100.64.0.18 | VM |
| pi-nas | 100.64.0.21 | Pi |
| mesh-bridge | 100.64.0.22 | LXC |
| peertube-4hve9pdr | 100.64.0.17 | LXC (collision-suffixed identity since ~2026-06-22; formerly `peertube` at 100.64.0.23, now stale/gone) |
| [[recon]] | 100.64.0.24 | VM |
| [[argus]] | 100.64.0.25 | LXC |
| [[central]] | 100.64.0.12 | LXC (utility CT 104 — central.echo6.mesh) |
| edge2 | 100.64.0.26 | Proxmox/Contabo VPS |
| gl-a1300 | 100.64.0.29 | Router |
| bluefin | 100.64.0.30 | Desktop |
| wordpress | 100.64.0.31 | LXC (edge2 CT 101 — now runs Grav CMS; hostname unchanged) |
| [[meshai]] | 100.64.0.32 | LXC |
| vaultwarden | 100.64.0.33 | LXC (edge2 CT 102) |
| forgejo | 100.64.0.34 | LXC (edge2 CT 103) — node id 46 |
| livesync | 100.64.0.35 | LXC (edge2 CT 104) — migrated 2026-06-16 |
| [[authentik]] | 100.64.0.36 | LXC (edge2 CT 105) — node id 48, migrated 2026-06-18 |
| matrix | 100.64.0.37 | LXC (edge2 CT 106) — migrated 2026-06-18 |
| headscale | 100.64.0.38 | LXC (edge2 CT 107) — migrated 2026-06-19 |
## IdahoMesh Headscale Node List
Separate Headscale instance on CT 106 (192.168.1.106), prefix 100.100.0.0/16.
Reachable from echo6 tailnet via mesh-bridge (CT 107).
| Node | Tailscale IP | User | Type |
|------|-------------|------|------|
| mesh-bridge | 100.100.0.3 | malice | LXC (bridge) |
| burley-butte | 100.100.0.1 | nebra | Pi (offline) |
| mt-isr | 100.100.0.5 | nebra | Pi Zero 2 W |
## SSH Access
**Standard user:** `zvx`
**Credentials:** Source from `/home/zvx/projects/.ref/credentials`
Most servers use SSH key auth. Exceptions noted below.
```bash
# SSH to any server (key auth)
ssh zvx@<ip-address>
# Examples
ssh zvx@192.168.1.244 # TOC (Proxmox host)
ssh zvx@192.168.1.150 # cortex VM
ssh zvx@192.168.1.241 # utility Proxmox
ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149 # edge1 (mail-only; formerly Contabo)
ssh zvx@cortex # cortex via Tailscale hostname
ssh edge2 # edge2 via SSH config alias (admin@100.64.0.26)
```
### edge2 SSH access
edge2 is hardened: `PasswordAuthentication no`, root login disabled. SSH user is `admin` with passwordless sudo. The SSH config alias `edge2` maps to `admin@100.64.0.26`.
Authorized keys on edge2 (`/home/admin/.ssh/authorized_keys`):
- `echo6-contabo2-184.174.35.153` — the original provisioning key (cortex `~/.ssh/contabo2_ed25519`)
- `cortex` — cortex default `id_ed25519`
- `matt-desktop-wsl2` — WSL2 default key
- `cookie-sync` — Windows native SSH key
To access CTs on edge2: `ssh edge2` then `sudo pct exec <vmid> -- bash`.
### Password-auth hosts
These require password authentication (no SSH keys installed):
| Host | User | Password | Access |
|------|------|----------|--------|
| aida-nebra | zvx | 7redditGold | `sshpass -p '7redditGold' ssh zvx@aida-nebra` |
| mt-isr | isr | UfPsfwyMIUIKb1 | `sshpass -p 'UfPsfwyMIUIKb1' ssh isr@192.168.1.141` |
| mt-burleybutte | bb | (see credentials) | `sshpass -p '<pw>' ssh bb@192.168.1.185` |
| matt-desktop | administrator | Qw1290opzx | `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no administrator@192.168.1.254` |
| toc | root | 7redditGold | `sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@100.64.0.13` |
Use the Tailscale hostname (`aida-nebra`) or local IP (`192.168.1.253`) — both work for aida-nebra.
mt-isr is on IdahoMesh tailnet (100.100.0.5) — reachable from echo6 via bridge.
matt-desktop is accessible via local IP (192.168.1.254) or Tailscale (100.64.0.10) — requires explicit password auth flags.
## Key External IPs
| Purpose | IP |
|---------|-----|
| Home external (public [[services]]) | 199.6.36.163 |
| edge1 (mail-only, rebuilt Contabo VPS) | 5.189.158.149 |
| edge2 (front door: vault/forge/notes/auth/matrix/element/vpn/proxmox + idahomesh.com) | 184.174.35.153 |

155
vault/docs/hardware/ip-allocation.md
View file

@ -1,155 +0,0 @@
---
title: Echo6 Network IP Allocation
type: reference
tags:
- proxmox
aliases: []
related:
- [[caddy]]
- [[services]]
- [[environment]]
- [[glossary]]
- [[headscale-onboard-node]]
updated: 2026-07-18
---
# Echo6 Network IP Allocation
## IP Ranges
| Range | Purpose |
|-------|---------|
| .1-.10 | Network Infrastructure (router, switches, APs) |
| .11-.99 | DHCP (phones, laptops, guests) |
| .100-.149 | LXC Containers |
| .150-.199 | VMs |
| .200-.224 | Security Cameras |
| .225-.239 | IoT/Embedded Devices |
| .240-.250 | Proxmox Hosts + Bare Metal Servers |
| .251-.254 | Meshtastic Nodes |
---
## Current Allocations
### Network Infrastructure (.1-.10)
| IP | Device | Notes |
|----|--------|-------|
| .1 | Router/Gateway | |
| .2-.10 | (available) | Switches, APs |
### LXC Containers (.100-.149)
| IP | Container | Host | Purpose |
|----|-----------|------|---------|
| .100 | meshmonitor (CT 100) | utility | MeshMonitor web UI |
| .101 | [[caddy]] (CT 101) | utility | Home reverse proxy |
| .102 | [[searxng]] (CT 102) | utility | Echo6 Search ([[searxng]]) |
| .103 | [[argus]] (CT 103) | utility | [[argus]] OSINT platform |
| .104 | meshing-around (CT 104) | utility | Mesh bot + WebGUI |
| .106 | meshtastic-hs (CT 106) | utility | IdahoMesh Headscale |
| .107 | mesh-bridge (CT 107) | utility | Dual-tailscaled bridge |
| .109 | mmud (CT 109) | utility | MMUD game daemon + web dashboard |
| .111 | mmud-embr (CT 111) | utility | MMUD SIM: EMBR (game server) |
| .112 | mmud-dcrg (CT 112) | utility | MMUD SIM: DCRG (broadcast) |
| .113 | mmud-grst (CT 113) | utility | MMUD SIM: GRST (Grist barkeep) |
| .114 | mmud-mrn (CT 114) | utility | MMUD SIM: MRN (Maren healer) |
| .115 | mmud-trvl (CT 115) | utility | MMUD SIM: TRVL (Torval merchant) |
| .116 | mmud-wspr (CT 116) | utility | MMUD SIM: WSPR (Whisper sage) |
| .118 | [[archivist]] (CT 118) | utility | Signal/Matrix archive bot |
| .130 | [[recon]] (VM 1130) | data | [[recon]] pipeline (migrated from CT 130) |
| .144 | [[meshai]] (CT 108) | utility | [[meshai]] assistant |
| .170 | peertube (CT 110) | media | PeerTube streaming |
| .182 | immich (CT 120) | cloud | Immich photos |
| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO |
### edge1 LXC Containers (10.10.10.x, vmbr0) — mail-only host (5.189.158.149, rebuilt 2026-06-19)
edge1 (rebuilt Contabo VPS, 5.189.158.149 / Tailscale 100.64.0.40) runs Debian 12 + Proxmox 8.4.19, mail-only role. Same 10.10.10.0/24 subnet pattern on vmbr0 (gw 10.10.10.1); internal IP via `staticroute.service`.
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.2 | mailcow | CT 101 | — | Mailcow email server (privileged LXC; reached via host DNAT ports 25/465/587/110/143/993/995/4190 and host [[caddy]] for mail/autodiscover/autoconfig.echo6.co → :8453) |
### edge2 LXC Containers (10.10.10.x, vmbr0) — permanent front door (184.174.35.153)
edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate 10.10.10.0/24 subnet on vmbr0 (gw 10.10.10.1). No DHCP — all IPs are static. Storage: `local` (dir, no local-lvm). **Front door for:** vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co + idahomesh.com.
| IP | Container | CTID | Tailscale | Purpose |
|----|-----------|------|-----------|---------|
| 10.10.10.10 | pdm | CT 100 | 100.64.0.28 | Proxmox Datacenter Manager |
| 10.10.10.11 | wordpress | CT 101 | 100.64.0.31 | Grav CMS 2.0.11 for idahomesh.com (migrated from WordPress 2026-07-17; hostname unchanged; MariaDB purged — no DB) |
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16); git SSH → edge2 forgejo-ssh-dnat.service |
| 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
| 10.10.10.23 | [[authentik]] | CT 105 | 100.64.0.36 | [[authentik]] SSO platform (migrated from Contabo 2026-06-18) |
| 10.10.10.24 | matrix | CT 106 | 100.64.0.37 | Matrix stack ([[synapse]] + MAS + Element + [[mautrix_signal]]; migrated from Contabo 2026-06-18) |
| 10.10.10.25 | headscale | CT 107 | 100.64.0.38 | Headscale + Headplane tailnet control plane (migrated from Contabo [[2026-06-19]]) |
| 10.10.10.26 | mailcow-staging | CT 108 | — | Stopped Mailcow staging replica (fallback; prune after soak) |
### VMs (.150-.199)
| IP | VM | Host | Purpose |
|----|-----|------|---------|
| .151 | ha (VM 151) | cloud | Home Assistant (Docker, home automation) |
### Security Cameras (.200-.224)
| IP | Camera | Location |
|----|--------|----------|
| .200-.224 | (available) | |
### IoT/Embedded (.225-.239)
| IP | Device | Notes |
|----|--------|-------|
| .225-.238 | (available) | |
| .239 | (reserved) | Former TOC IP |
### Proxmox Hosts + Bare Metal (.240-.250)
| IP | Hostname | Role | Tailscale |
|----|----------|------|-----------|
| .240 | data | Proxmox - DB node | 100.64.0.6 |
| .241 | utility | Proxmox - Utility [[services]] | 100.64.0.5 |
| .242 | cloud | Proxmox - Cloud storage | 100.64.0.4 |
| .243 | media | Proxmox - Media server | 100.64.0.3 |
| .244 | echo6-toc | LLM/AI server | 100.64.0.13 |
| .245-.250 | (available) | | |
### Meshtastic Nodes (.251-.254)
| IP | Node | Notes |
|----|------|-------|
| .251-.252 | (available) | |
| .253 | AIDA-NEBRA | Raspberry Pi running meshtasticd |
| .254 | (available) | |
---
## Tailscale IPs (100.64.0.x)
| Tailscale IP | Device | Local IP |
|--------------|--------|----------|
| 100.64.0.13 | echo6-toc | 192.168.1.244 |
| 100.64.0.40 | contabo / edge1 (rebuilt [[2026-06-19]], mail-only) | 5.189.158.149 (external) |
| 100.64.0.9 | aida-nebra | 192.168.1.253 |
| 100.64.0.5 | utility | 192.168.1.241 |
| 100.64.0.6 | data | 192.168.1.240 |
| 100.64.0.3 | media | 192.168.1.243 |
| 100.64.0.4 | cloud | 192.168.1.242 |
| 100.64.0.26 | edge2 | 184.174.35.153 (external) |
| 100.64.0.28 | pdm (CT 100 on edge2) | 10.10.10.10 (vmbr0) |
| 100.64.0.33 | vaultwarden (CT 102 on edge2) — node id 45 | 10.10.10.20 (vmbr0) |
| 100.64.0.34 | forgejo (CT 103 on edge2) — node id 46 | 10.10.10.21 (vmbr0) |
| 100.64.0.35 | livesync (CT 104 on edge2) — hostname `livesync` | 10.10.10.22 (vmbr0) |
| 100.64.0.36 | [[authentik]] (CT 105 on edge2) — node id 48 | 10.10.10.23 (vmbr0) |
| 100.64.0.37 | matrix (CT 106 on edge2) — hostname `matrix` | 10.10.10.24 (vmbr0) |
| 100.64.0.38 | headscale (CT 107 on edge2) — hostname `headscale` | 10.10.10.25 (vmbr0) |
| 100.64.0.16 | ha (VM 151 on cloud) — node id 54 | 192.168.1.151 |
---
## Notes
- DHCP range starts at .11 to leave room for network infra
- LXC containers get .100-.149 for easy identification
- Meshtastic nodes at .251-.254 (max 4 hardware nodes)
- AIDA-NEBRA at .253 is technically a Meshtastic node (runs meshtasticd)
---
*Last updated: [[2026-06-19]] — Added edge1 section (rebuilt Contabo VPS, 5.189.158.149 / 100.64.0.40, mail-only, Mailcow CT 101 at 10.10.10.2); added edge2 CT 108 (mailcow-staging, stopped); Tailscale table: contabo-mailcow entry replaced with edge1 at 100.64.0.40; previously added edge2 CTs 102–107*

249
vault/docs/matrix/archivist.md
View file

@ -1,249 +0,0 @@
---
title: Signal Archive Bot — Deployment Reference
type: reference
tags:
- matrix
aliases: []
related:
- [[mautrix_signal]]
- [[synapse]]
- [[matrix-synapse-deployment]]
- [[caddy]]
- [[recon-operations]]
updated: 2026-07-13
---
# Signal Archive Bot — Deployment Reference
# Created: 2026-04-12 (Phase 3)
# Status: Phase 5 COMPLETE — bot deployed, transcripts writing, sync token dedup verified
## CT 118 — archivist
| Setting | Value |
|---------|-------|
| VMID | 118 |
| Hostname | archivist |
| Host node | utility (192.168.1.241) |
| IP | 192.168.1.118/24 |
| Gateway | 192.168.1.1 |
| OS | Debian 12 (bookworm) |
| Disk | 8GB (local-lvm:vm-118-disk-0) |
| RAM | 1024 MB |
| Swap | 512 MB |
| Cores | 1 |
| Unprivileged | Yes |
| Features | keyctl=1 |
| Onboot | Yes |
| MAC | BC:24:11:74:E9:DC |
## NFS Mount
### In-container NFS mount: FAILED
- Attempt 1 (no mount=nfs feature): `access denied by server`
- Attempt 2 (with mount=nfs feature): `Operation not permitted`
- Root cause: unprivileged LXC containers cannot mount NFS directly
### Final approach: host-side NFS + Proxmox bind mount
- **Utility host fstab:** `192.168.1.245:/export/library /mnt/library nfs defaults,soft,timeo=150 0 0`
- **CT 118 mp0:** `/mnt/library,mp=/mnt/library,ro=0`
- **Pattern source:** recon-vm (VM 1130, [[recon]]) on data node uses identical approach
- Utility host did NOT have /mnt/library mounted before this [[deployment]]
### Write access
- `/mnt/library/` permissions: 2777 (drwxrwsrwx), owner nobody:nogroup
- `/mnt/library/signal-archive/` permissions: 777 (drwxrwxrwx), created by CT root
- Archivist user UID 999 (container) maps to UID 100999 (host/NFS)
- Write access verified: archivist can create dirs, write files, delete within signal-archive/
- No NFS export changes needed — world-writable parent dir permits all UIDs
## Service User
| Setting | Value |
|---------|-------|
| Username | archivist |
| UID | 999 |
| GID | 996 |
| Shell | /usr/sbin/nologin |
| Home | /opt/archivist |
| Host-mapped UID | 100999 |
| Host-mapped GID | 100996 |
## Directory Layout
```
/opt/archivist/ # Home dir (owned by archivist:archivist)
archivist.py # Main bot script (Phase 5)
.env # Environment variables (Phase 5)
/store/ # E2EE key store + sync token (SqliteStore)
/logs/ # Bot logs (archivist.log)
/venv/ # Python virtual environment
/mnt/library/signal-archive/ # Transcript output (NFS bind mount)
<room-slug>/
transcript.log # Append-only human-readable transcript
media/ # Downloaded media files (images, audio, video, etc.)
```
## Python Environment
| Component | Version |
|-----------|---------|
| Python | 3.11.2 |
| pip | 26.0.1 |
| matrix-nio | 0.25.2 (with e2e extras) |
| python-olm | 3.2.16 |
| libolm-dev | 3.2.13~dfsg-1 |
| aiohttp | 3.13.5 |
Venv path: `/opt/archivist/venv/`
### Verified imports
- `nio.AsyncClient` — Matrix client
- `nio.crypto.OlmDevice` — E2EE device management
- `olm.Account` — libolm C binding
- `nio.store.SqliteStore` — crypto key persistence
## System Packages
Installed via apt:
- python3, python3-venv, python3-pip, python3-dev
- libolm-dev
- gcc, g++, make
- nfs-common
- curl, ca-certificates
## Access
```bash
# From utility host
pct exec 118 -- bash
# Direct SSH (not configured yet — no SSH keys installed)
# ssh root@192.168.1.118
```
## Matrix User — @archivist:echo6.co
| Setting | Value |
|---------|-------|
| User ID | @archivist:echo6.co |
| Display name | Archivist Bot |
| Created via | mas-cli (MAS user existed from 2026-04-10, password set via `manage set-password`) |
| Password | `<REDACTED — see credentials file: MATRIX_ARCHIVIST_BOT_PASSWORD>` |
| Device ID | ARCHIVIST |
| Access token | Stable compat token via `mas-cli manage issue-compatibility-token archivist ARCHIVIST` |
| Admin | No (not needed — uses room invitation, not admin force-join) |
### E2EE / Cross-Signing
| Component | Status |
|-----------|--------|
| Device keys (curve25519 + ed25519) | Uploaded |
| One-time keys (signed_curve25519) | 50 uploaded |
| Master key | Published (nKFt5nA+TvUo0AvY1gsk1QL8OK1t9z/ChON30Kdvlek) |
| Self-signing key | Published (9gOB+AHgyBzLP5/xerYon04NLZuIh+o5OHAybmetK2A) |
| User-signing key | Published (1EYngPiwpjOy2aQWY02g5SZaFLM5kZgsHLhFXgwHh0Q) |
| Device self-signed | Yes (ARCHIVIST signed by self-signing key) |
| Cross-signing seeds | /opt/archivist/store/cross_signing_seeds.json (chmod 600) |
| nio store | /opt/archivist/store/@archivist:echo6.co_ARCHIVIST.db |
### Key Sharing — How It Works
- Bridge key sharing policy: `cross-signed-tofu`
- ARCHIVIST device is cross-signed → bridge shares Megolm session keys automatically
- **Interactive verification (SAS emoji) is NOT required** — cross-signing alone is sufficient
- Old messages (before archivist joined) remain undecryptable (Megolm keys not retroactively shared)
- New messages are decryptable immediately
### E2BE Decryption Test — PASSED
- Date: 2026-04-12 14:35 UTC
- Room: COMMS LP group (!XUeWZuPdWQQnUYLJBJ:echo6.co)
- Message: "You'll know that radio has come of age when the median cellphone incorporates a LoRa radio stack."
- Sender: @signal_cdf98bca-c4b7-4fda-8ceb-03db5eb4e7e2:echo6.co (Signal puppet via bridge)
- Result: Successfully decrypted by ARCHIVIST device
### User Creation Notes
- Shared-secret registration (`/_synapse/admin/v1/register`) returns 404 under MAS — endpoint disabled
- Must use `mas-cli manage register-user` or `manage set-password` for existing users
- MAS creates user in both MAS DB and [[synapse]] DB
- Orphaned [[synapse]] `profiles` row caused provisioning failure — fixed by DELETE
- Each `client.login()` creates a NEW MAS compat session with random device ID — use `restore_login()` with stable compat token instead
- matrix-nio v0.25.2 does NOT implement `bootstrap_cross_signing()` — manual implementation required via python-olm PkSigning + raw HTTP API
## Joined Rooms
| Room | Room ID | Type | Archive Slug |
|------|---------|------|-------------|
| COMMS LP group | !XUeWZuPdWQQnUYLJBJ:echo6.co | Bridged Signal group | comms-lp-group |
| DM with Matt | !wgbnqhnYKTHzzJMjDu:echo6.co | Direct message | — |
| Liberal_Preppers_OG | !RvWNPmcKtPImhKPYcA:echo6.co | Bridged Signal group | liberal-preppers-og |
| (3 additional rooms) | !vBXtbgfYcptEuimrmn, !SnGDZgBtYOQuTWeYXp, !aQWFQMrzbkwjyjCPte | Bridged Signal groups | (initialized on first message) |
## Scripts on CT 118
| Script | Purpose | Status |
|--------|---------|--------|
| /opt/archivist/archivist.py | Main bot — transcript writer | **Running in tmux** |
| /opt/archivist/login_once.py | One-shot login + key upload | Completed (superseded) |
| /opt/archivist/bootstrap_crosssigning.py | Cross-signing key bootstrap | Completed (one-time) |
| /opt/archivist/setup_and_verify.py | Device setup + verification listener | Completed |
| /opt/archivist/test_decrypt.py | E2BE decryption test listener | Completed (superseded by archivist.py) |
## Bot Architecture (Phase 5)
### archivist.py — Event-Driven Transcript Bot (~260 lines)
**Core design:** Single-file async Python bot using matrix-nio `sync_forever` with `ClientConfig(store_sync_tokens=True)` for restart deduplication.
**Callbacks:**
- `on_invite` (InviteMemberEvent) → auto-join
- `on_text` (RoomMessageText) → write transcript line, detect edits via `m.replace`
- `on_image/audio/video/file` (RoomMessage* + RoomEncrypted*) → download + decrypt + save to media/
- `on_sticker` (StickerEvent) → same as media
- `on_redaction` (RedactionEvent) → log deletion with original content if cached
- `on_megolm` (MegolmEvent) → log decryption failure (counter per room)
**In-memory caches (NOT persisted):**
- `room_slugs: dict[str, str]` — room_id → slug (rebuilt from transcript headers on startup)
- `event_cache: dict[str, dict]` — event_id → {body, sender, ts} (for edit/redact tracking)
- `name_cache: dict[str, str]` — mxid → display name (Signal ghosts get "(Signal)" suffix)
**Sync token persistence:** matrix-nio SqliteStore handles save/load automatically when `store_sync_tokens=True`. On restart, `loaded_sync_token` resumes from last position — no event replay.
**Encrypted media handling:** RoomEncryptedImage/Audio/Video/File carry `key`, `hashes`, `iv` attributes. Bot downloads ciphertext via `client.download(mxc=url)`, then decrypts with `nio.crypto.decrypt_attachment()`.
### Transcript Format
```
# Transcript: Room Display Name
# Room ID: !xxxxx:echo6.co
# Archive started: 2026-04-12 20:47:01 UTC
# ---
[2026-04-12 18:55:34 UTC] Sender Name (Signal): Message text
[2026-04-12 18:56:00 UTC] Sender Name (Signal): [EDITED] New text
(was: Original text)
[2026-04-12 18:57:00 UTC] Sender Name (Signal): [DELETED] (was: Original text)
[2026-04-12 18:58:00 UTC] Sender Name (Signal): [image: media/1234567890_filename.jpg] caption
```
### Running the Bot
```bash
# Start (tmux, as archivist user)
pct exec 118 -- su -s /bin/bash archivist -c "tmux new-session -d -s archivist /opt/archivist/venv/bin/python3 -u /opt/archivist/archivist.py"
# Check logs
pct exec 118 -- tail -f /opt/archivist/logs/archivist.log
# Stop
pct exec 118 -- su -s /bin/bash archivist -c "tmux send-keys -t archivist C-c"
```
## What's NOT done yet (Phase 6+)
- No systemd service (running in tmux)
- No Tailscale registration
- No SSH key auth configured
- Bot needs invitations to additional bridged rooms as they appear

34
vault/docs/matrix/matrix_host.md
View file

@ -1,34 +0,0 @@
---
title: Matrix Host Reference — edge2 CT 106 (formerly Contabo VPS)
type: reference
tags:
- matrix
aliases: []
related:
- [[matrix-synapse-deployment]]
- [[lxc-service-migration]]
- [[caddy]]
- [[ip-allocation]]
- [[services]]
updated: 2026-07-13
---
# Matrix Host Reference — edge2 CT 106
> Migrated off Contabo → edge2 CT 106 on 2026-06-18.
# Generated: 2026-04-09 (Phase 0)
## SSH Access
- Host: 100.64.0.37 (Tailscale, edge2 CT 106) — was 100.64.0.1 (Contabo) before the 2026-06-18 migration
- Auth: SSH key as root — CONFIRMED WORKING on Contabo; re-verify on edge2 CT 106
- sudo: root user, no sudo needed
## System Info
- OS: Ubuntu 24.04.3 LTS (Noble Numbat)
- Kernel: 6.8.0-90-generic
- CPU: AMD EPYC (12 vCPUs)
- RAM: 47 GB total, ~35 GB available
- Disk: /dev/sda1 968 GB, 76 GB used, 893 GB available (8%)
- Docker Root: /var/lib/docker
- Docker: 29.0.2
- Docker Compose: v2.40.3

117
vault/docs/matrix/mautrix_signal.md
View file

@ -1,117 +0,0 @@
---
title: mautrix-signal Bridge Reference
type: reference
tags:
- matrix
aliases: []
related:
- [[synapse]]
- [[synapse_retention_discovery]]
- [[matrix-synapse-deployment]]
- [[archivist]]
- [[advbbs-project]]
updated: 2026-07-13
---
# mautrix-signal Bridge Reference
# Deployed: 2026-04-09 (Phase 3)
# Updated: 2026-04-09 (Phase 5)
## Deployment
- **Image:** dock.mau.dev/mautrix/signal:v0.2603.0
- **Container:** mautrix-signal
- **Compose:** /opt/matrix/docker-compose.yml (same stack as [[synapse]])
- **Config dir:** /opt/matrix/mautrix-signal/
- **Network:** matrix-net (internal only, no host port mapping)
- **Appservice port:** 29328 (container-internal)
## Database
- **DB name:** mautrix_signal
- **DB role:** mautrix_signal (NOSUPERUSER, NOCREATEDB, NOCREATEROLE)
- **Host:** matrix-postgres:5432 (same container as [[synapse]]/MAS)
- **Collation:** C/C (matches Synapse)
## Signal Account
- **Phone:** +12083080811
- **Login ID:** 58f99d83-f3a8-487f-a2b7-3d118e236d23
- **Matrix user:** @matt:echo6.co
## Bot
- **MXID:** @signalbot:echo6.co
- **Device ID:** UPX4KKLZVY
- **Management room:** !fDjIRTMjxILVQoAcEN:echo6.co
## Encryption
- **E2BE:** enabled (allow: true, default: true, require: true)
- **MSC4190:** enabled (required for MAS compatibility)
- **Pickle key:** auto-generated, stored in config
- **Self-sign:** true
## Synapse Integration
- **Registration files:** /opt/matrix/synapse/registration.yaml, /opt/matrix/synapse/doublepuppet.yaml
- **homeserver.yaml additions:**
- `app_service_config_files: [/data/registration.yaml, /data/doublepuppet.yaml]`
- `experimental_features.msc3202_transaction_extensions: true`
- `experimental_features.msc2409_to_device_messages_enabled: true`
- **Registration file permissions:** 644 (world-readable, Synapse UID 991)
## Double Puppeting
- **Method:** appservice-based (doublepuppet.yaml)
- **as_token:** configured in bridge config `double_puppet.secrets`
- **Effect:** Messages matt sends from Signal appear as @matt:echo6.co in Matrix
## MAS Compatibility
- **Confirmed working:** as_token auth bypasses MAS entirely
- **MSC4190:** Parsed from registration.yaml `io.element.msc4190: true`, NOT a homeserver experimental_features flag
- **Synapse handles appservice auth independently of MAS token introspection**
## Autocreate Behavior
- **No autocreate toggle exists** in bridgev2 signal connector
- `portal_create_filter` config key is silently stripped on restart
- Old keys (`autocreate_group_portal`, `autocreate_contact_portal`) don't exist
- Portals auto-create when incoming Signal messages arrive for untracked groups/DMs
- This is the default and only behavior — cannot be disabled
## Permissions
```yaml
bridge:
permissions:
"*": relay
"echo6.co": user
"@matt:echo6.co": admin
```
## Backup
- DB dumped by /opt/matrix/scripts/pg_backup.sh (added alongside synapse + mas dumps)
- 14-day retention, daily at 3 AM
## Management Commands
Available in management room (!fDjIRTMjxILVQoAcEN:echo6.co):
- `!signal help` — list commands
- `!signal ping-matrix` — verify Matrix connectivity
- `!signal list-logins` — show linked Signal accounts
- `!signal sync-portal` — force-sync current portal
- `!signal id` — show bridge IDs for current room
- `!signal create-group` — create a new Signal group
- `!signal start-chat` — start a DM with a Signal user
- `!signal resolve-identifier` — look up a Signal user
NOT available (bridgev2 signal connector doesn't expose):
- `list-chats`, `bridge`, `create-portal`, `filter`
## Provisioning API
- **Shared secret:** in config.yaml (do NOT log or put in ref files)
- **Allow Matrix auth:** true
- **Debug endpoints:** false

93
vault/docs/matrix/synapse.md
View file

@ -1,93 +0,0 @@
---
title: Synapse Deployment Reference
type: reference
tags:
- matrix
aliases: []
related:
- [[matrix-synapse-deployment]]
- [[mautrix_signal]]
- [[synapse_retention_discovery]]
- [[archivist]]
- [[caddy]]
updated: 2026-07-13
---
# Synapse Deployment Reference
# Generated: 2026-04-09 (Phase 1)
## Install Method
- Docker Compose at /opt/matrix/docker-compose.yml
- Four containers on `matrix-net` bridge network:
- matrix-synapse (matrixdotorg/synapse:latest) → 127.0.0.1:8008
- matrix-mas (ghcr.io/element-hq/matrix-authentication-service:latest) → 127.0.0.1:8085
- matrix-postgres (postgres:16-alpine) → internal 5432
- matrix-element (vectorim/element-web:latest) → 127.0.0.1:8088
## Synapse Version
- 1.147.1 (confirmed via admin API and docker exec)
## Key Config Values (from homeserver.yaml)
- server_name: echo6.co
- public_baseurl: https://matrix.echo6.co/
- listeners: port 8008, HTTP, x_forwarded=true, bind 0.0.0.0, resources=[client, federation]
- database: psycopg2 → matrix-postgres:5432, db=synapse, user=synapse
- media_store_path: /data/media_store (95 MB used)
- registration_shared_secret: PRESENT (not redacted here — see synapse_homeserver.yaml.sanitized)
- signing_key_path: /data/echo6.co.signing.key
- enable_registration: false
- url_preview_enabled: true
- report_stats: false
- app_service_config_files: NOT PRESENT (no appservices currently registered)
- encryption_enabled_by_default_for_room_type: NOT SET (default=off)
- MAS delegation: enabled, endpoint=http://matrix-mas:8080/, shared secret present
## MAS (Matrix Authentication Service)
- Version: v1.12.0
- Listen: 8080 (web), 8081 (internal/health)
- Database: postgresql://mas:***@matrix-postgres:5432/mas
- Upstream OAuth2: [[authentik]] (auth.echo6.co) via OIDC
- Client ID: 93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T
- Issuer: https://auth.echo6.co/application/o/matrix/
- Matrix integration: kind=synapse, homeserver=echo6.co, endpoint=http://matrix-synapse:8008/
- Passwords: enabled (bcrypt v1 + argon2id v2)
- Email transport: blackhole (not sending)
## Database (PostgreSQL)
- Version: 16.12 (Alpine)
- Host: matrix-postgres container on matrix-net
- Databases: synapse (owner: synapse), mas (owner: mas), postgres, template0, template1
- Users: synapse (Superuser, Create role, Create DB), mas (regular)
- Auth: password from .env file (POSTGRES_PASSWORD)
- New DB/user can be created without collision — synapse user has Superuser/Create role privileges
## Reverse Proxy (Caddy on edge2)
- matrix.echo6.co routes:
- /_matrix/client/*/login|logout|refresh|auth_metadata → MAS (127.0.0.1:8085)
- /_matrix/* → Synapse (127.0.0.1:8008)
- /_synapse/* → Synapse (127.0.0.1:8008)
- Everything else → MAS (127.0.0.1:8085)
- element.echo6.co → 127.0.0.1:8088
## Federation
- Well-known served from echo6.co (utility [[caddy]], NOT matrix.echo6.co)
- /.well-known/matrix/server: {"m.server": "matrix.echo6.co:443"}
- /.well-known/matrix/client: base_url=https://matrix.echo6.co, issuer=https://matrix.echo6.co/
- Federation tester: AllChecksOK=true, TLS 1.3, valid Ed25519 key, valid certificates
- Port 8448 exposed in container but NOT used externally (federation via 443 + well-known)
## Existing Appservices
- NONE — no app_service_config_files in homeserver.yaml, no registration files in /opt/matrix/synapse/
## Backup
- Cron: daily at 3 AM via /opt/matrix/scripts/pg_backup.sh
- Backs up synapse DB only (NOT mas DB)
- Retention: 14 days
- Location: /opt/matrix/backups/
## Ports in 29xxx Range
- NONE in use — confirmed via ss -tlnp
## MSC4190 Support
- Synapse 1.147.1 supports MSC4190 (merged in 1.121.0)
- Requires io.element.msc4190: true in appservice registration file
- Also requires experimental_features.msc3202_transaction_extensions: true in homeserver.yaml

54
vault/docs/matrix/synapse_homeserver.yaml.sanitized
View file

@ -1,54 +0,0 @@
server_name: "echo6.co"
public_baseurl: "https://matrix.echo6.co/"
pid_file: /data/homeserver.pid
listeners:
- port: 8008
type: http
tls: false
x_forwarded: true
bind_addresses: ["0.0.0.0"]
resources:
- names: [client, federation]
compress: false
database:
name: psycopg2
args:
user: synapse
password: <REDACTED>
database: synapse
host: matrix-postgres
port: 5432
cp_min: 5
cp_max: 10
log_config: "/data/echo6.co.log.config"
media_store_path: /data/media_store
enable_registration: false
url_preview_enabled: true
url_preview_ip_range_blacklist:
- "127.0.0.0/8"
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- "100.64.0.0/10"
- "169.254.0.0/16"
- "::1/128"
- "fe80::/10"
- "fc00::/7"
registration_shared_secret: <REDACTED>
macaroon_secret_key: <REDACTED>
form_secret: <REDACTED>
signing_key_path: "/data/echo6.co.signing.key"
report_stats: false
trusted_key_servers:
- server_name: "matrix.org"
# MAS delegation - replaces oidc_providers
matrix_authentication_service:
enabled: true
endpoint: http://matrix-mas:8080/
secret: <REDACTED>

161
vault/docs/matrix/synapse_retention_discovery.md
View file

@ -1,161 +0,0 @@
---
title: Synapse Retention Discovery
type: reference
tags:
- matrix
aliases: []
related:
- [[synapse]]
- [[mautrix_signal]]
- [[matrix-synapse-deployment]]
- [[SESSION-HANDOFF-meshai-test]]
- [[pymc-repeater-kiss-tnc-reenumeration]]
updated: 2026-07-13
---
# Synapse Retention Discovery
# Generated: 2026-04-09 (Phase 6.0, Question 1)
## Current homeserver.yaml Values
### APPLIED 2026-04-12 (Archivist Phase 2)
Two settings added to homeserver.yaml:
```diff
report_stats: false
+redaction_retention_period: null
+
trusted_key_servers:
- server_name: "matrix.org"
```
```diff
experimental_features:
msc3202_transaction_extensions: true
msc2409_to_device_messages_enabled: true
+ msc2815_enabled: true
```
- Backup: `/opt/matrix/synapse/homeserver.yaml.bak-20260412`
- Applied: 2026-04-12 03:06 UTC
- [[synapse]] restarted, health verified, bridge reconnected, Element login confirmed
- Rollback: restore backup and `docker compose restart synapse`
### Previous state (before 2026-04-12)
| Setting | Previous Value | Current Value | Source |
|---------|---------------|---------------|--------|
| redaction_retention_period | NOT SET (default 7d) | null (disabled) | [[synapse]]/config/server.py |
| msc2815_enabled | NOT SET (default false) | true | synapse/config/experimental.py |
| forgotten_room_retention_period | NOT SET | NOT SET (unchanged) | synapse/config/server.py |
| media_retention.local_media_lifetime | NOT SET | NOT SET (unchanged) | synapse/config/repository.py |
| media_retention.remote_media_lifetime | NOT SET | NOT SET (unchanged) | synapse/config/repository.py |
Full experimental_features block (current):
```yaml
experimental_features:
msc3202_transaction_extensions: true
msc2409_to_device_messages_enabled: true
msc2815_enabled: true
```
## What `redaction_retention_period: null` Changes
Source: Synapse v1.147.1 synapse/config/server.py, synapse/storage/databases/main/events.py
When set to `null`:
- Synapse STOPS censoring (overwriting) redacted event content in the DB
- The original unredacted content stays in event_json table forever
- The 5-minute censoring job still runs but skips all events
- Redactions still WORK from the client perspective — clients see events as redacted
- The only difference is the server retains the pre-redaction content internally
When at default `7d`:
- After 7 days post-redaction, Synapse replaces stored event content with the redacted form
- Original content is permanently lost from the DB
- Any tool querying the DB after 7 days gets only the stripped event
## MSC2815 Support — CONFIRMED in Synapse 1.147.1
Config key: `experimental_features.msc2815_enabled` (default: false)
Source: synapse/config/experimental.py line 372
Feature flag advertised as: `fi.mau.msc2815: true` in /_matrix/client/versions
What MSC2815 does:
- Adds `include_unredacted_content=true` query param to GET /rooms/{roomId}/event/{eventId}
- Requires requester to have power level >= room's redact PL (default 50)
- Returns original unredacted content if still in DB
- Returns FI.MAU.MSC2815_UNREDACTED_CONTENT_DELETED if content was already censored
Critical interaction: MSC2815 + redaction_retention_period
- With default 7d: MSC2815 only works for 7 days after redaction
- With null: MSC2815 works forever (content never censored)
## Current DB Sizes
| Database | Size |
|----------|------|
| synapse | 259 MB |
| mas | 17 MB |
| mautrix_signal | 14 MB |
Top 5 tables in synapse:
| Table | Size |
|-------|------|
| state_groups_state | 91 MB |
| event_json | 20 MB |
| events | 15 MB |
| event_auth | 14 MB |
| device_lists_changes_in_room | 12 MB |
## Growth Rate Analysis
Backup sizes over 14 days (Mar 26 - Apr 9, gzipped):
- Mar 26: 5.28 MB → Apr 9: 5.41 MB
- Delta: ~130 KB compressed over 14 days = ~9 KB/day compressed
- Uncompressed estimate: ~50-100 KB/day growth
Event volume:
- Total events: 2,617 across 18 rooms
- Last 7 days: 2,356 events (bridge just deployed, bulk of these are portal creation)
- Steady-state will be much lower — Signal groups average 10-50 messages/day
- Only 3 redaction events in entire history
## Storage Impact Estimate (disabling redaction purge)
Current scale:
- 2,617 events = 259 MB total DB (but most of that is state, not event content)
- event_json table: 20 MB for 2,617 events = ~7.6 KB avg per event
- Redactions are 3 out of 2,617 (0.1%)
Even at 10x Signal bridge traffic (100 messages/day across all groups):
- 100 events/day × 7.6 KB = 760 KB/day event_json growth
- Redactions typically <5% of messages = <5 events/day × 7.6 KB = 38 KB/day saved by NOT censoring
- Over 1 year: ~14 MB retained from not censoring
Verdict: Storage impact of `redaction_retention_period: null` is NEGLIGIBLE.
The entire Synapse DB after months of use is 259 MB. Even with aggressive Signal
bridge traffic, retaining redacted content adds <15 MB/year.
## MAS Interaction with Redacted Content
MAS does NOT interact with redacted content. MAS handles:
- Authentication (login/logout/refresh)
- Token management
- Upstream OIDC delegation
MAS has no audit policy, no event content access, and no retention policy of its own.
Redaction handling is entirely within Synapse's event store. No conflict.
## Rollback Procedure
If `redaction_retention_period: null` and `msc2815_enabled: true` are added and need reverting:
1. Remove both settings from homeserver.yaml
2. Restart Synapse: `docker compose restart synapse`
3. Synapse returns to default 7d censoring
4. Already-retained content will be censored within 7 days + 5 minutes
5. No DB migration needed — Synapse's censoring job handles cleanup automatically
The rollback is clean and non-destructive.

114
vault/docs/navi/cc-rules.md
View file

@ -1,114 +0,0 @@
---
title: "Navi: Claude Code Rules"
type: reference
tags:
- recon
aliases: []
related:
- [[deployment]]
- [[CLAUDE-baseline]]
- [[navi]]
- [[themes]]
- [[environment]]
updated: 2026-07-13
---
# Navi: Claude Code Rules
## Repository & SSH
**All [[navi]] SSH goes to:** `recon-vm` (VM 1130, 192.168.1.130)
**Never SSH to cortex for Navi work.** Previous attempt to deploy from cortex wrecked production by deploying from a stale clone.
**Work directory:** `/home/zvx/projects/repos/navi` on VM 1130
**Never touch:** `/home/zvx/projects/navi-work` on cortex (stale clone, do not use)
## Git / Repository Rules
**GitHub (zvx-echo6) is the source of truth for all repos.**
### Remote Convention
- **origin** = GitHub (github.com/zvx-echo6/{repo})
- **forge** = Forgejo backup (forge.echo6.co/matt/{repo})
### Creating New Repos
1. Create on GitHub first (public unless told otherwise)
2. Create matching repo on Forgejo
3. Configure Forgejo repo as pull mirror from GitHub (8h interval)
4. Local clone: origin = GitHub, forge = Forgejo
### Exceptions
- **echo6-docs** is the sole exception — Forgejo only, no GitHub copy
### Rules
- Never force-push to origin without explicit approval
- Never use `git add -A` — stage files explicitly
## Critical Constraints
- **Never re-export namedTheme** through any intermediary module — import directly from protomaps-themes-base in MapView.jsx
- **Never use `git add -A`** — stage files explicitly
- **Never deploy without smoke tests** — run full checklist (see deployment.md)
- **Never start preview servers** — deploy to production and test there
## Build & Deploy
```bash
ssh recon-vm
cd /home/zvx/projects/repos/navi
npm run build && rsync -av --delete dist/ /mnt/nav/frontend/
```
## Git Workflow
- Feature branches: name descriptively (theme-*, fix-*, feat-*)
- Always merge to master before deploying
- Check for unmerged commits: `git log branch..master --oneline`
## Code Patterns
### queryRenderedFeatures for Optional Layers
Guard with map.getLayer() — USFS/BLM hit layers may not exist:
```javascript
const layers = [USFS_TRAILS_HIT, USFS_ROADS_HIT].filter(id => map.getLayer(id))
const features = layers.length > 0
? map.queryRenderedFeatures(e.point, { layers })
: []
```
### buildStyle Theme Colors
```javascript
const theme = getTheme(themeName)
const colors = theme.colors || namedTheme(themeName) // direct import
```
### PlaceCard useEffect Changes
Investigation-before-implementation — fragile race conditions between:
- Boundary fetch from Wikidata/OSM
- selectedPlace state updates
- AbortController cleanup
## Architecture Notes
- **Boundary highlight** is NOT an overlay toggle — it's a dynamic click-response layer
- **Route polyline** uses GeoJSON source.setData() — silent failure if namedTheme is re-exported
- **Measure tool** also uses GeoJSON — same failure mode
## Smoke Test Checklist
After every deploy:
- [ ] Route between two addresses — polyline renders
- [ ] Click city label — boundary outline appears
- [ ] Theme switching (all 4 [[themes]])
- [ ] Overlay toggles (hillshade, contours, public lands)
- [ ] Console: no "bt is not defined" or "f is not defined"

69
vault/docs/navi/deployment.md
View file

@ -1,69 +0,0 @@
---
title: Navi Deployment
type: reference
tags:
- recon
aliases: []
related:
- [[cc-rules]]
- [[navi]]
- [[recon-operations]]
- [[central-deploy-cutover]]
- [[themes]]
updated: 2026-07-13
---
# Navi Deployment
## Source Repository
**Location:** VM 1130 (`192.168.1.130`) at `/home/zvx/projects/repos/navi`
> **WARNING:** Never deploy from `/home/zvx/projects/navi-work` on cortex — that clone is stale.
## Build & Deploy
```bash
# SSH to VM 1130
ssh recon-vm
# Always backup first
cp -r /mnt/nav/frontend /mnt/nav/frontend.bak.$(date +%Y%m%d-%H%M%S)
# Build and deploy
cd /home/zvx/projects/repos/navi
npm run build && rsync -av --delete dist/ /mnt/nav/frontend/
```
## Infrastructure
- **Nginx** on VM 1130 serves `/mnt/nav/frontend/` on port 8440
- `index.html`: no-cache (always fresh)
- Hashed assets (`*.js`, `*.css`): cache forever
- **[[caddy]]** on CT 101 routes `navi.echo6.co` → VM 1130:8440
## Pre-Deploy Checklist
1. Merge feature branch to master before deploying
2. Check for unmerged upstream commits:
```bash
git log your-branch..master --oneline
```
3. Create backup (see above)
## Post-Deploy Smoke Tests
Run after **every** deploy:
- [ ] Route between two addresses — polyline renders
- [ ] Click city label — boundary outline appears
- [ ] Theme switching works (dark, light, clean, cyberpunk)
- [ ] Toggle overlays (hillshade, contours, public lands)
- [ ] Console: no `bt is not defined` or `f is not defined` errors
## Rollback
```bash
# If deploy fails, restore backup
rm -rf /mnt/nav/frontend
mv /mnt/nav/frontend.bak.YYYYMMDD-HHMMSS /mnt/nav/frontend
```

129
vault/docs/navi/themes.md
View file

@ -1,129 +0,0 @@
---
title: Navi Theme System
type: reference
tags:
- auth
aliases: []
related:
- [[navi]]
- [[cc-rules]]
- [[deployment]]
- [[searxng]]
- [[echo6-landing-page-data-export]]
updated: 2026-07-13
---
# Navi Theme System
## Architecture
**Registry:** `src/themes/registry.js`
[[central]] source for theme metadata, overlay config, UI CSS vars, and satellite adjustments.
## Critical: namedTheme Import
> **NEVER re-export namedTheme through registry.js or any other module.**
namedTheme must be imported **directly** from protomaps-themes-base in MapView.jsx:
```javascript
// CORRECT
import { layers, namedTheme } from 'protomaps-themes-base'
import { getTheme, getThemeSprite, getOverlayConfig } from '../themes/registry'
function buildStyle(themeName) {
const theme = getTheme(themeName)
const colors = theme.colors || namedTheme(themeName) // direct import for built-ins
return {
// ...
layers: layers('protomaps', colors, { lang: 'en' }),
}
}
```
**Why:** Vite's bundling of namedTheme through a re-export breaks MapLibre's Web Worker, causing silent GeoJSON rendering failure (routes, boundaries, measure tool all invisible, "f is not defined" error in worker).
## Theme Config Shape
```javascript
{
id: 'dark', // unique identifier
name: 'Dark', // display name
dark: true, // affects overlay styling, sprite fallback
swatch: ['#1c1917', '#7a9a6b', '#b8a88a'], // theme picker preview
fontImports: [], // Google Font URLs (empty = system fonts)
colors: null, // null = built-in, object = custom theme colors
satellite: null, // raster adjustments when satellite active
overlay: { ... }, // per-layer styling for overlays
ui: { ... }, // 32 CSS custom properties
}
```
## UI CSS Variables
Applied by applyThemeUI() via document.documentElement.style.setProperty():
- Backgrounds: --bg-base, --bg-raised, --bg-overlay, --bg-input, --bg-inset, --bg-muted
- Text: --text-primary, --text-secondary, --text-tertiary, --text-inverse
- Borders: --border, --border-subtle
- Accent: --accent, --accent-hover, --accent-muted
- Pins: --pin-origin, --pin-destination, --pin-intermediate, --pin-stroke
- Status: --status-success, --status-warning, --status-danger, --success, --warning, --warning-muted
- Fonts: --font-sans, --font-mono, --font-heading
- Shadows: --shadow, --shadow-lg
## Overlay Config
Read via getOverlayConfig(themeId, layerKey) with spread-defaults fallback:
- hillshade: exaggeration, illuminationDirection, shadowColor, highlightColor
- contours: colors, opacities, widths for minor/intermediate/index lines + labels
- publicLands: fill/outline colors per agency (NPS, USFS, BLM, etc.)
- usfsTrails: road/trail colors by use type (motorized, bicycle, hiker)
- blmTrails: route colors by vehicle class (4WD, ATV, non-mechanized)
## Satellite Raster Adjustments
Neutral defaults (no adjustment):
```javascript
satellite: {
opacity: 1.0,
brightnessMin: 0.0,
brightnessMax: 1.0,
contrast: 0.0, // MapLibre uses -1 to 1 range
saturation: 0.0, // MapLibre uses -1 to 1 range
hueRotate: 0,
}
```
## Font Support
fontImports array of Google Font URLs, managed on theme switch:
```javascript
fontImports: [
'https://fonts.googleapis.com/css2?family=Orbitron&display=swap',
'https://fonts.googleapis.com/css2?family=Share+Tech+Mono&display=swap',
]
```
Injected as <link data-theme-font> tags, removed on theme switch.
## Theme Picker
ThemePicker.jsx — swatch popover in header, reads themeList() for [{id, name, dark, swatch}].
## Current Themes
| ID | Name | Type | Description |
|----|------|------|-------------|
| light | Light | built-in | Default light theme |
| dark | Dark | built-in | Default dark theme |
| clean | Clean | custom | Google Maps-inspired, utilitarian |
| cyberpunk | Cyberpunk | custom | Neon palette, Orbitron + Share Tech Mono fonts |
## Adding a New Theme
1. Create src/themes/{name}.js with full theme config
2. Import and register in registry.js themes object
3. Theme auto-appears in picker via themeList()

585
vault/docs/services/ots-setup.md
View file

@ -1,585 +0,0 @@
---
title: OpenTAKServer Setup Documentation
type: reference
tags:
- proxmox
aliases: []
related:
- [[ct-runbook]]
- [[environment]]
- [[ip-allocation]]
- [[expose-service-home]]
- [[proxmox-onboard-node]]
updated: 2026-07-13
---
# OpenTAKServer Setup Documentation
## Installation Date
April 17, 2026
## Overview
OpenTAKServer installation on Echo6 infrastructure for TAK (Team Awareness Kit) server capabilities.
---
## Container Specifications
**VMID:** 109
**Hostname:** opentakserver
**Host:** utility Proxmox (192.168.1.241)
**IP Address:** 192.168.1.109/24
**Gateway:** 192.168.1.1
**OS:** Ubuntu 24.04 LTS
**Resources:** 2 cores, 4GB RAM, 8GB disk
**Storage:** local-lvm
**Features:** nesting=1
**Auto-start:** enabled
---
## Access Information
### External Access
- **URL:** https://ots.k7zvx.com
- **Admin Username:** administrator
- **Admin Password:** password (**CHANGE THIS IMMEDIATELY**)
### SSH Access
```bash
# Via Proxmox host
ssh root@192.168.1.241 'pct exec 109 -- bash'
# As zvx user
ssh root@192.168.1.241 'pct exec 109 -- su - zvx'
# Direct container entry
pct enter 109
```
**Container User:**
- Username: zvx
- Password: 7redditGold
- Sudo: NOPASSWD enabled
---
## Network Configuration
### DNS
- **Domain:** ots.k7zvx.com
- **[[dns]] Provider:** GoDaddy
- **Record Type:** A
- **Points to:** 199.6.36.163 (home external IP)
- **TTL:** 3600 seconds
### SSL Certificate
- **Provider:** Let's Encrypt
- **Method:** acme.sh with GoDaddy [[dns]] validation
- **Location:** /etc/[[caddy]]/certs/ots.k7zvx.com.* (on CT 101)
- **Auto-renewal:** Configured via acme.sh
### Reverse Proxy (Caddy on CT 101)
```
ots.k7zvx.com {
tls /etc/caddy/certs/ots.k7zvx.com.fullchain.crt /etc/caddy/certs/ots.k7zvx.com.key
reverse_proxy https://192.168.1.109:443 {
transport http {
tls_insecure_skip_verify
}
}
}
```
### Port Forwarding
- **Router:** Ubiquiti firewall (192.168.1.28)
- **External ports:** 80/443 → 192.168.1.101 ([[caddy]] CT)
- **Internal proxy:** Caddy → 192.168.1.109:443 (OpenTAKServer)
---
## OpenTAKServer Configuration
### Installation Details
- **Installation Method:** Official installer script
- **Install User:** zvx
- **Install Path:** /home/zvx/ots/
- **Virtual [[environment]]:** /home/zvx/.opentakserver_venv/
- **Version:** 1.7.10
### Services
```bash
# Main services
systemctl status opentakserver
systemctl status eud_handler
systemctl status eud_handler_ssl
systemctl status cot_parser
systemctl status nginx
# Service logs
tail -f /home/zvx/ots/logs/opentakserver.log
tail -f /home/zvx/ots/logs/eud_handler_tcp.log
tail -f /home/zvx/ots/logs/eud_handler_ssl.log
tail -f /home/zvx/ots/logs/cot_parser.log
```
### Listening Ports
- **80:** nginx (HTTP, redirects to HTTPS)
- **443:** nginx (HTTPS, web UI)
- **8080:** nginx (HTTP Marti API)
- **8088:** eud_handler (TCP CoT)
- **8089:** eud_handler (TLS CoT)
- **8443:** nginx (HTTPS Marti API)
- **8446:** nginx (Admin API)
- **1936:** nginx
- **8322:** nginx
- **8883:** nginx
### Database Configuration
- **Type:** PostgreSQL
- **Database name:** ots
- **Username:** ots
- **Password:** eCt8HwnBFlZJ6K1arTn4
- **Connection string:** postgresql+psycopg2://ots:eCt8HwnBFlZJ6K1arTn4@127.0.0.1/ots
- **Service:** postgresql (systemd)
### Configuration File
- **Location:** /home/zvx/ots/config.yml
- **Backup:** /home/zvx/ots/config.yml.bak (if needed)
---
## Meshtastic MQTT Gateway
### Overview
OpenTAKServer's native Meshtastic integration allows remote gateway radios to connect via MQTT over TLS. This enables:
- Meshtastic mesh positions to appear as TAK contacts on the map
- Text messages from mesh to flow into TAK chat
- Remote gateway [[deployment]] over Starlink, cell hotspots, or any internet connection
**Setup Date:** April 17, 2026
**Status:** ✅ Fully Operational
### Architecture
```
Meshtastic Radio (GPS)
└─→ MQTT TLS (port 8883) via ots.k7zvx.com
└─→ Nginx Stream Proxy (TLS termination)
└─→ RabbitMQ MQTT (localhost:1883)
└─→ OpenTAKServer (HTTP auth backend)
└─→ PostgreSQL (user database)
└─→ TAK Clients (ATAK/iTAK/WinTAK)
```
### Server Configuration
#### RabbitMQ MQTT Settings
**Location:** `/etc/rabbitmq/rabbitmq.conf`
```conf
# MQTT listener bound to loopback only (security)
mqtt.listeners.tcp.default = 127.0.0.1:1883
# Authentication via OTS HTTP backend
auth_backends.1 = internal
auth_backends.2 = http
auth_http.http_method = post
auth_http.user_path = http://127.0.0.1:8081/api/rabbitmq/auth
auth_http.vhost_path = http://127.0.0.1:8081/api/rabbitmq/vhost
auth_http.resource_path = http://127.0.0.1:8081/api/rabbitmq/resource
auth_http.topic_path = http://127.0.0.1:8081/api/rabbitmq/topic
# AMQP also restricted to loopback
listeners.tcp.default = 127.0.0.1:5672
```
**Security:** Ports 1883 (MQTT) and 5672 (AMQP) are bound to localhost only. External access is via nginx TLS proxy on port 8883.
#### Nginx Stream Configuration
**Location:** `/etc/nginx/streams-available/rabbitmq`
```nginx
server {
listen 8883 ssl;
proxy_pass 127.0.0.1:1883;
ssl_certificate /home/zvx/ots/ca/certs/opentakserver/opentakserver.pem;
ssl_certificate_key /home/zvx/ots/ca/certs/opentakserver/opentakserver.nopass.key;
}
```
**Note:** Currently uses self-signed OTS CA certificate. Meshtastic radios must disable TLS certificate verification ("allow insecure").
#### OTS Meshtastic Configuration
**Location:** `/home/zvx/ots/config.yml`
```yaml
OTS_ENABLE_MESHTASTIC: true
OTS_MESHTASTIC_GROUP: __ANON__
OTS_MESHTASTIC_TOPIC: opentakserver
OTS_MESHTASTIC_PUBLISH_INTERVAL: 30
OTS_MESHTASTIC_NODEINFO_INTERVAL: 3
OTS_MESHTASTIC_DOWNLINK_CHANNELS: []
```
- **Group `__ANON__`:** Broadcasts Meshtastic positions to all connected TAK clients
- **Topic `opentakserver`:** MQTT topic root for all Meshtastic messages
- **Publish interval:** OTS publishes updates to TAK clients every 30 seconds
### Client Configuration (Meshtastic Radio)
#### Connection Settings
- **Server/Address:** `ots.k7zvx.com`
- **Port:** Auto (uses 8883 when TLS enabled)
- **Enable TLS/SSL:** ✅ **REQUIRED**
- **TLS Certificate Verification:** ❌ **DISABLED** ("Allow Insecure") *— server uses self-signed cert*
#### Authentication
- **Username:** `[unique_username]` — created in OTS web UI (Settings → Users)
- **Password:** `[user_password]` — set in OTS web UI
#### MQTT Settings
- **Topic:** `opentakserver`
- **Encryption Enabled:** ❌ **DISABLED** — **CRITICAL:** OTS cannot decode PSK-encrypted MQTT payloads. The LoRa RF hop between radios is still PSK-encrypted regardless of this setting. This toggle only controls whether the radio re-encrypts before publishing to MQTT. TLS on port 8883 provides wire security instead.
- **JSON Output:** ❌ **DISABLED** — OTS expects protobuf format, not JSON. Leave this off.
- **Proxy to Client:** ❌ **DISABLED** — required per OTS docs; enabling this causes presence/LWT issues with gateway operation
- **Uplink Enabled:** ✅ **REQUIRED** — configured per-channel when scanning QR code from OTS web UI
#### Position Settings
- **Map Publish:** ❌ **DISABLED** — prevents public map sharing (map.meshtastic.org)
- **Position Enabled:** ✅ **REQUIRED**
- **Position Precision:** Any — full precision is sent to private server regardless of this setting
**Important:** "Map Publish" only affects the public Meshtastic map. Position data still flows to custom MQTT servers (ots.k7zvx.com) with full precision when disabled.
### User Setup Process
1. **Create OTS user account:**
```bash
# Via OTS Web UI
https://ots.k7zvx.com → Settings → Users → Create User
# Set username (e.g., "matt", "jeepnjonny")
# Set password
```
2. **Configure Meshtastic radio** with connection settings above
3. **Test connection:**
- Send a text message from the radio
- Message should appear in iTAK/ATAK within seconds (will appear in triplicate due to mesh retry logic)
4. **Verify position data:**
- Position updates send based on mesh broadcast interval (typically 15-30 minutes)
- Node should appear as contact on TAK map with accurate location
- Updates continue at configured interval
### Monitoring & Verification
#### Check MQTT Connection
```bash
# View RabbitMQ logs for connection activity
sudo tail -f /var/log/rabbitmq/rabbit@opentakserver.log | grep MQTT
# List active MQTT connections
sudo rabbitmqctl list_connections protocol user peer_host peer_port
# Monitor MQTT traffic (subscribe to all topics)
mosquitto_sub -h 127.0.0.1 -p 1883 -u [username] -P [password] -t "#" -v
```
#### Check OTS Processing
```bash
# Monitor Meshtastic controller logs
sudo tail -f /home/zvx/ots/logs/opentakserver.log | grep -i mesh
# Check for position data
sudo grep "latitude_i" /home/zvx/ots/logs/opentakserver.log | tail -20
# View Meshtastic EUDs in database
sudo -u postgres psql -d ots -c "SELECT callsign, uid, meshtastic_id, last_event_time FROM euds WHERE meshtastic_id IS NOT NULL;"
```
### Troubleshooting
#### Connection Refused
**Symptom:** Radio shows "connection refused" error
**Causes & Fixes:**
1. **TLS not enabled** — Radio connecting to port 1883 (blocked externally). Enable TLS on radio to use port 8883.
2. **Wrong server** — Verify `ots.k7zvx.com` (not `mqtt.k7zvx.com`)
3. **Firewall** — Verify port 8883 is accessible: `telnet ots.k7zvx.com 8883`
#### Authentication Failed (CONNACK code 4)
**Symptom:** RabbitMQ logs show "access refused for user"
**Causes & Fixes:**
1. **User doesn't exist** — Create user in OTS web UI first
2. **Wrong password** — Verify credentials match OTS user settings
3. **Check logs:**
```bash
sudo grep "MQTT connection failed" /var/log/rabbitmq/rabbit@opentakserver.log
```
#### TLS Certificate Verification Failed
**Symptom:** Connection fails with SSL/certificate errors
**Fix:** Disable TLS certificate verification on radio ("Allow Insecure" or "Skip Verify")
**Reason:** OTS uses self-signed certificate. Option to install Let's Encrypt cert exists but not currently implemented.
#### Radio Connected But No Data
**Symptom:** MQTT connection shows as active but no messages/position in OTS
**Causes & Fixes:**
1. **Uplink disabled** — Ensure uplink is enabled on the channel (set via QR code from OTS Meshtastic web UI page)
2. **No GPS lock** — Verify radio has satellite fix and coordinates displayed
3. **Wrong topic** — Ensure radio is publishing to `opentakserver` topic
4. **Test with text message** — Send message from radio to verify MQTT publishing works
#### Node Appears But No Position on Map
**Symptom:** Contact exists in iTAK but no location shown
**Causes & Fixes:**
1. **No GPS lock** — Radio must have satellite fix to send position
2. **Position broadcasts disabled** — Enable "Position Enabled" on radio
3. **Waiting for interval** — Position updates send less frequently than text messages (typically 15-30 min)
4. **Check logs:**
```bash
sudo grep "latitude_i\|position" /home/zvx/ots/logs/opentakserver.log | tail -20
```
#### Unicode Encoding Errors
**Symptom:** OTS logs show `UnicodeEncodeError` related to emojis
**Fix:** Remove emojis from Meshtastic node name (long name and short name). PostgreSQL database cannot handle emojis with current encoding.
**Example error:**
```
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3f4' in position 0
```
### Known Behaviors
#### Triplicate Messages
**Behavior:** Each text message appears 3 times in iTAK
**Reason:** Meshtastic retry logic sends messages 3 times for reliability over mesh network. OTS forwards all copies.
**Impact:** Normal behavior, ensures messages get through despite poor mesh coverage. Not configurable without modifying OTS source.
#### Position Update Frequency
**Mesh vs MQTT:** Meshtastic has separate intervals:
- **Mesh broadcasts:** Position sent to local mesh (15-30 min typical)
- **MQTT publishing:** Position sent to MQTT server (can be same or different)
Radio may show position locally more often than it publishes to MQTT server. Configure "MQTT Position Interval" separately from mesh broadcast interval.
#### Map Publish vs Custom MQTT
**Map Publish:** Controls publishing to public Meshtastic map (map.meshtastic.org)
- Has 1-hour minimum interval due to privacy concerns
- Shows degraded precision on public map
**Custom MQTT (OTS):** Unaffected by map publish settings
- Receives full precision position data
- Updates at configured mesh/MQTT interval
- Completely private — only visible to your TAK clients
Disabling "Map Publish" does NOT affect position data sent to custom MQTT servers like OTS.
### Active Gateways
| Node ID | Callsign | Owner | Channel | Status |
|---------|----------|-------|---------|--------|
| !f861851e | Meshtastic 851e | matt | LFtrackers | Active |
### Port Summary
| Port | Service | Bind Address | External Access | Purpose |
|------|---------|--------------|-----------------|---------|
| 1883 | RabbitMQ MQTT | 127.0.0.1 | ❌ No (loopback only) | MQTT plaintext (internal) |
| 5672 | RabbitMQ AMQP | 127.0.0.1 | ❌ No (loopback only) | Internal OTS ↔ RabbitMQ |
| 8883 | Nginx Stream | 0.0.0.0 | ✅ Yes (via ots.k7zvx.com) | MQTT TLS (Meshtastic radios) |
---
## Critical Fixes Applied
### 1. Ubuntu Mirror Issues
**Problem:** Canonical's GeoDNS is broken for Filer Telephone ISP (Idaho). All DNS servers (local, Google, Cloudflare) returned only UK mirror IPs (91.189.x.x range) which had severe packet loss and timeouts.
**Root Cause:**
- Your ISP has poor peering with Canonical's UK infrastructure
- GeoDNS should return US mirrors but only returned UK for this region
- 30-50% packet loss between ISP and Canonical's network
**Fix Applied:**
```bash
# Changed /etc/apt/sources.list to use Oregon mirror
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
# Results:
# - Before: Timeouts (10+ seconds)
# - After: 11 MB/s download speed
```
**Testing:**
- archive.ubuntu.com: TIMEOUT
- mirrors.kernel.org: 0.16s response time
- mirror.arizona.edu: 0.31s response time
### 2. PostgreSQL Driver Compatibility
**Problem:** OpenTAKServer installer uses psycopg3 by default, which has a compatibility bug with SQLAlchemy's PostgreSQL version detection. This causes the service to crash-loop with:
```
TypeError: cannot use a string pattern on a bytes-like object
```
**Fix Applied:**
```bash
# Install psycopg2-binary (stable driver)
pip install psycopg2-binary
# Update connection string in /home/zvx/ots/config.yml
# Changed: postgresql+psycopg://
# To: postgresql+psycopg2://
```
**Impact:** Without this fix, the backend API doesn't run and login fails silently.
---
## Troubleshooting
### Service Won't Start
```bash
# Check service status
systemctl status opentakserver
# Check logs for errors
tail -50 /home/zvx/ots/logs/opentakserver.log
# Common issues:
# 1. Database connection errors → check PostgreSQL is running
# 2. Port conflicts → check nginx/other services
# 3. psycopg errors → ensure psycopg2-binary is installed
```
### Login Fails
```bash
# Verify backend API is running
systemctl status opentakserver
# Check if service is crash-looping
journalctl -u opentakserver -f
# Test local API access
curl -I http://localhost:8080
curl -I https://localhost:443
```
### Slow Package Updates
```bash
# Verify using fast mirror
grep "URIs:" /etc/apt/sources.list
# Should show: http://mirrors.kernel.org/ubuntu
# If not, run:
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
apt update
```
### External Access Issues
```bash
# Test from outside network
curl -I https://ots.k7zvx.com
# Check DNS resolution
dig +short ots.k7zvx.com @8.8.8.8
# Should return: 199.6.36.163
# Check Caddy is proxying correctly (on CT 101)
systemctl status caddy
journalctl -u caddy -f
```
---
## Maintenance
### Update OpenTAKServer
```bash
# Stop services
systemctl stop opentakserver eud_handler eud_handler_ssl cot_parser
# Activate virtualenv
su - zvx
source ~/.opentakserver_venv/bin/activate
# Update via pip (if available)
pip install --upgrade opentakserver
# Or follow official upgrade docs:
# https://docs.opentakserver.io/
# Restart services
sudo systemctl start opentakserver eud_handler eud_handler_ssl cot_parser
```
### Backup
```bash
# Database backup
sudo -u postgres pg_dump ots > /tmp/ots_backup_$(date +%Y%m%d).sql
# Configuration backup
cp /home/zvx/ots/config.yml /home/zvx/ots/config.yml.backup
# Certificate backup (from CT 101)
tar -czf /tmp/ots_certs_$(date +%Y%m%d).tar.gz /etc/caddy/certs/ots.k7zvx.com.*
```
### Change Admin Password
1. Log in to https://ots.k7zvx.com with administrator/password
2. Navigate to Settings → Users
3. Select administrator user
4. Change password
5. Save changes
---
## Next Steps
1. ✅ Change default administrator password
2. ⏳ Configure TAK client connections
3. ⏳ Set up user certificates for client authentication
4. ⏳ Configure data packages and plugins
5. ⏳ Set up monitoring/alerting for the service
6. ⏳ Document backup/restore procedures
7. ⏳ Test failover scenarios
---
## Reference Links
- **OpenTAKServer Documentation:** https://docs.opentakserver.io/
- **OpenTAKServer GitHub:** https://github.com/brian7704/OpenTAKServer
- **TAK.gov Resources:** https://tak.gov/
- **Echo6 Infrastructure Docs:** /home/zvx/projects/.ref/docs/
---
## Notes
### Why Not Docker?
Previous attempts to install OpenTAKServer via Docker encountered persistent authentication issues. The native installation method worked successfully.
### ISP-Specific Issue
The Ubuntu mirror issue is specific to Filer Telephone Company's network peering. This may affect other [[services]] in the future. Consider:
- Using mirrors.kernel.org for all Ubuntu-based containers
- Setting up a local apt-cacher-ng if this becomes widespread
- Monitoring ISP routing changes
### Tailscale Not Involved
Initial suspicion was that Tailscale exit nodes or DNS were causing the mirror issues. Testing confirmed this was false - the issue persists even with local DNS and no Tailscale routing.
---
*Last updated: 2026-04-17*

490
vault/docs/services/services.md
View file

@ -1,490 +0,0 @@
---
title: Current Services Inventory
type: reference
tags:
- mesh
aliases: []
related:
- [[caddy]]
- [[ip-allocation]]
- [[meshtastic-headscale-runbook]]
- [[lxc-service-migration]]
- [[central]]
updated: 2026-08-24
---
# Current Services Inventory
> **[[dns]] split ([[2026-06-19]]):** `mail/autodiscover/autoconfig.echo6.co` → **edge1** (5.189.158.149, mail-only rebuilt Contabo VPS). `vault/forge/notes/auth/matrix/element/vpn/proxmox.echo6.co` → **edge2** (184.174.35.153, permanent front door for all other services). Tailnet split-DNS is NOT used for echo6.co; echo6.co resolves via public GoDaddy DNS.
## Active Services
| Service | Location | IP:Port | Access | Notes |
|---------|----------|---------|--------|-------|
| MeshMonitor | utility (CT 100) | 192.168.1.100:8080 / :4404 | https://mesh.echo6.co | Meshtastic mesh monitoring (upstream ghcr.io/yeraze/meshmonitor:latest, multi-channel AutoAnnounce/AutoResponder) |
| Utility [[caddy]] | utility (CT 101) | 192.168.1.101 / 100.64.0.8 | 199.6.36.163 (ports 80/443) | Reverse proxy for home services |
| Echo6 Search ([[searxng]]) | utility (CT 102) | 192.168.1.102:8080 | https://echo6.co | Branded search homepage (Docker, custom theme) |
| meshtasticd (AIDA-N2) | aida-nebra | 192.168.1.253:4403 | Internal | AIDA-N2(RPT,LLM) node !27780c47, Nebra 2W hat (ZebraHat), CLIENT_BASE role, fw 2.7.19. [[meshai]] (CT 108) connects via TCP localhost:4403 |
| Meshtastic CLI | mt-isr | 192.168.1.141 | Internal | Station G2 WiFi bridge + TCP management |
| meshtasticd | mt-burleybutte | 192.168.1.185:4403 | Internal | Software Meshtastic node (Nebra 2W hat) |
| IdahoMesh Headscale | utility (CT 106) | 192.168.1.106:8080 | https://vpn.idahomesh.com | Meshtastic mesh VPN coordination |
| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
| [[meshai]] | utility (CT 108) | 192.168.1.144:4403 / :8080 | Internal | LLM-powered Meshtastic assistant (Docker, work-meshai local build, gemini-3.1-flash-lite, Google grounding) |
| [[argus]] | utility (CT 103) | 192.168.1.103:8080 | Internal | Python app on :8080 — OSINT intelligence gathering platform |
| [[central]] | utility (CT 104) | 192.168.1.104:8000 / 100.64.0.12 | central.echo6.mesh (mesh) | Data-hub spine — ~25 adapters → NATS/JetStream → TimescaleDB; serves traffic tiles to [[navi]] — see [[central]] |
| NATS/JetStream ([[central]]) | utility (CT 104) | 192.168.1.104:4222 / :8222 | Internal | [[central]] backend message bus (NATS :4222 client, :8222 monitoring) |
| TimescaleDB/PostGIS ([[central]]) | utility (CT 104) | 192.168.1.104:5432 | Internal | [[central]] backend time-series + geospatial database (PostgreSQL 16 + TimescaleDB + PostGIS) |
| [[authentik]] | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via edge2 DNAT) | https://forge.echo6.co | Git server — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on edge2 (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
| Headscale | edge2 (CT 107) | 100.64.0.38:8084 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) — fronted by edge2 host [[caddy]] — **migrated from Contabo [[2026-06-19]]** |
| Headplane | edge2 (CT 107) | 100.64.0.38:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via [[authentik]]) — fronted by edge2 host [[caddy]] — **migrated from Contabo [[2026-06-19]]** |
| Mailcow | **edge1 CT 101** (10.10.10.2) | 5.189.158.149 | https://mail.echo6.co | Email server (privileged LXC on rebuilt Contabo VPS, updated commit 52a41b4d / SOGo 5.12.8) — **rebuilt in-place [[2026-06-19]]** |
| [[meshwars]] Preview | utility (CT 113) | 192.168.1.113 / 100.64.0.39:8090 | https://mwpreview.k7zvx.com | Public preview of the unreleased `feat/places` MeshWars branch, running a periodically-refreshed read-only copy of production (CT 119) data; /admin and /api/admin/* return 404 on the public host, admin reachable only over the tailnet |
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager 1.37.1 (SSO enabled) — fronted by edge2 host [[caddy]] (reverse_proxy 100.64.0.33:8086) |
| Grav | edge2 (CT 101) | 10.10.10.11:80 | https://idahomesh.com (+www) | Flat-file CMS 2.0.11, no database — Admin2 plugin at /admin; Apache 2.4.67 + mod_php + PHP 8.4.21; migrated from WordPress 2026-07-17 (MariaDB purged from the container); hostname still `wordpress` (unchanged) — fronted by edge2 host Caddy via **internal bridge IP** (reverse_proxy 10.10.10.11:80), unlike other edge2 services which proxy over tailnet |
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ (Syncthing on Contabo decommissioned [[2026-06-19]] with edge1 rebuild) |
| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
| Immich | cloud (CT 120) | 192.168.1.182:2283 | https://immich.echo6.co | Photo management (Docker, NFS storage on pi-nas) |
| Nextcloud | cloud (CT 121) | 192.168.1.183:11000 | https://nextcloud.echo6.co | Cloud storage (AIO Docker, NFS on pi-nas, SSO) |
| Jellyfin | media (VM 105) | 192.168.1.160:8096 | https://jellyfin.echo6.co | Media server (Docker, NFS on pi-nas, SSO) |
| Jellyseer | media (VM 105) | 192.168.1.160:5055 | https://requests.echo6.co | Media request management (Docker, SSO) |
| Sonarr | media (VM 105) | 192.168.1.160:8989 | Internal | TV automation (Docker) |
| Radarr | media (VM 105) | 192.168.1.160:7878 | Internal | Movie automation (Docker) |
| Prowlarr | media (VM 105) | 192.168.1.160:9696 | Internal | Indexer manager (Docker) |
| SABnzbd | media (VM 105) | 192.168.1.160:8080 | Internal | [[usenet]] download client (Docker) |
| PeerTube | media (CT 110) | 192.168.1.170:9000 | https://stream.echo6.co | Video streaming (native, NFS on pi-nas, SSO). CT raised to 8GB + redis capped 2026-08-13 — see [[peertube-sitemap-redis-oom]] |
| Open WebUI | cortex (VM 150) | 192.168.1.150:8080 | https://ai.echo6.co | AI chat interface (Docker, Ollama backend, SSO) |
| Qdrant | cortex (VM 150) | 192.168.1.150:6333 | Internal | Vector database (Docker, [[recon]] knowledge store) |
| TEI | cortex (VM 150) | 192.168.1.150:8090 | Internal | Text embeddings (Docker, bge-m3 1024-dim) |
| [[recon]] | data (VM 1130) | 192.168.1.130:8420 | https://recon.echo6.co | Knowledge extraction pipeline (systemd, dashboard+API) |
| navi-config | data (VM 1130) | 192.168.1.130:8422 | Internal | [[recon]] [[navi]] node config API |
| navi-contacts | data (VM 1130) | 192.168.1.130:8423 | Internal | [[recon]] [[navi]] contact enrichment API |
| navi-landclass | data (VM 1130) | 192.168.1.130:8424 | Internal | [[recon]] [[navi]] land classification API |
| navi-places | data (VM 1130) | 192.168.1.130:8425 | Internal | [[recon]] [[navi]] OSM place detail/enrichment |
| navi-geo | data (VM 1130) | 192.168.1.130:8426 | Internal | RECON navi geocode/reverse geocode API |
| navi-admin | data (VM 1130) | 192.168.1.130:8427 | Internal | RECON navi fleet admin-info aggregator |
| navi-offroute | data (VM 1130) | 192.168.1.130:8428 | Internal | RECON navi off-network router + MVUM API |
| navi (navi.echo6.co) | data (VM 1130) | 192.168.1.130:8440 | https://navi.echo6.co | Offline navigation platform — see [[navi]] |
| dem-origin | data (VM 1130) | 127.0.0.1:8441 | Internal | Local DEM tile origin for navi (658 GB planet-dem.pmtiles) |
| Valhalla | data (VM 1130) | 192.168.1.130:8002 | Internal | navi geo backend (routing) — see [[navi]] |
| Nominatim | data (VM 1130) | 192.168.1.130:8010 | Internal | navi geo backend (geocoder) — see [[navi]] |
| Photon | data (VM 1130) | 192.168.1.130:2322 | Internal | navi geo backend (geocoder + Elasticsearch :9201) — see [[navi]] |
| PostgreSQL/PostGIS | data (VM 1130) | 192.168.1.130:5432 | Internal | navi geo backend (padus, overture DBs) — see [[navi]] |
| Files | data (VM 1130) | 192.168.1.130:8888 | https://files.echo6.co | PDF library (nginx, [[authentik]] forward auth) |
| Samba | data | 192.168.1.240:445 | Internal | SMB file sharing — `//data/library` → /mnt/data/library (guest access) |
| Matrix [[synapse]] | edge2 (CT 106) | 100.64.0.37:8008 | https://matrix.echo6.co | Matrix homeserver (Docker, SSO) — **migrated from Contabo 2026-06-18** |
| Element Web | edge2 (CT 106) | 100.64.0.37:8088 | https://element.echo6.co | Matrix web client (Docker) — **migrated from Contabo 2026-06-18** |
| [[mautrix_signal]] | edge2 (CT 106) | internal (29328) | DM @signalbot:echo6.co | Signal bridge (Docker, E2BE, MSC4190, double puppeting) — **migrated from Contabo 2026-06-18** |
| LiveSync | edge2 (CT 104) | 100.64.0.35:5984 (CouchDB) / :5985 (provisioner) | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) — fronted by edge2 host Caddy; **migrated from Contabo 2026-06-16** |
| OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] |
| Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) |
| Matrix MAS | edge2 (CT 106) | 100.64.0.37:8085 | Internal (via edge2 Caddy) | Matrix Authentication Service (Docker, handles login/logout/OIDC for [[synapse]]) — **migrated from Contabo 2026-06-18** |
| [[archivist]] | utility (CT 118) | 192.168.1.118 | Internal | Signal/Matrix room archive bot (systemd) — see archivist.md for details |
| pt-transcoder | cortex (VM 150) | N/A | Internal | PeerTube H.265 NVENC transcoder (systemd, /opt/bulk-import/transcoder.py) |
| recon-sparse | cortex (VM 150) | 192.168.1.150:8091 | Internal | RECON sparse embedding service (systemd, bge-m3 model, port 8091) |
| obsidian-remote | cortex (VM 150) | 100.64.0.14:8082 → :3001 | Internal (Tailscale) | Headless web Obsidian (lscr.io/linuxserver/obsidian:latest, Docker) |
| mcc | media (CT 111) | 192.168.1.111:80 | Internal | Caddy + Postfix, OpenHop (pymc) console web app; serves static frontend, reverse-proxies /api,/auth,/ws → 192.168.1.253:8000 (aida-nebra). Port 80 only — no 443 listener |
| Samba | cortex (VM 150) | 192.168.1.150:445 | Internal | SMB file sharing — `//cortex/projects` → /home/zvx/projects (guest access) |
| Home Assistant | ha (cloud VM 151) | 192.168.1.151:8123 / 100.64.0.16 | Internal | Home automation platform (Docker, Ubuntu 24.04) |
## Services by Server
### toc - Proxmox Host (192.168.1.244 / Tailscale: 100.64.0.13)
- Proxmox VE node (echo6-cluster)
- GPU passthrough host for cortex VM
- No direct services — workloads run on cortex VM
### cortex - VM 150 on toc (192.168.1.150 / Tailscale: 100.64.0.14)
- GPU compute VM (RTX A4000)
- Claude Code host
- Syncthing (syncs with Contabo)
- Open WebUI (port 8080, https://ai.echo6.co, Docker, SSO via [[authentik]], Echo6 theme)
- Compose path: `/opt/open-webui/docker-compose.yml`
- Echo6 theme: togglable via "E6" button (bottom-right), persisted in localStorage
- Theme files bind-mounted from `/home/zvx/echo6-theme/` into container
- DEFAULT_USER_ROLE=user (new signups auto-activated, not pending)
- Ollama (port 11434, internal, Docker with GPU)
- Qdrant (port 6333, internal, Docker — vector DB for RECON)
- TEI (port 8090, internal, Docker — bge-m3 embeddings for RECON)
- [[peertube-remote-runner]] (peertube-runner service, Whisper auto-captioning via smart GPU/CPU wrapper, concurrency=2, MemoryMax=20G)
- pt-transcoder (systemd: pt-transcoder.service, PeerTube H.265 NVENC transcoder)
- Script: `/opt/bulk-import/transcoder.py`
- MemoryMax=12G, Restart=always, RestartSec=60
- Depends on: nvidia-persistenced.service
- recon-sparse (systemd: recon-sparse.service, RECON sparse embedding service)
- Script: `/opt/recon-sparse/sparse_embed_service.py --port 8091`
- Model: BAAI/bge-m3 (HuggingFace cache)
- Restart=on-failure, RestartSec=10
- Samba (smbd/nmbd, system packages)
- Share: `//cortex/projects` → `/home/zvx/projects` (browseable, read-write, guest OK, force user/group zvx)
- Workgroup: WORKGROUP, standalone server
- Echo6 Cortex Agent (systemd: echo6-agent.service, matrix-nio bot, @cortex:echo6.co)
- Install path: `/opt/echo6-agent/`
- Matrix space: echo6-ops, room: #cortex:echo6.co (E2EE, private)
- Session continuity via `claude -p --resume`, persistent per-room sessions
- `!new` resets conversation session
- Allowed users: @matt:echo6.co
- MAS user ID: 01KKX88ARGK0BTA1JMB2QVAW4C
### utility - CT 100 (192.168.1.100 / Tailscale: 100.64.0.7)
- MeshMonitor (port 8080 + 4404, https://mesh.echo6.co)
- Image: ghcr.io/yeraze/meshmonitor:latest (upstream image, not local fork build)
- Multi-channel AutoAnnounce and AutoResponder support
### utility - CT 101 (192.168.1.101 / Tailscale: 100.64.0.8)
- Utility Caddy (reverse proxy for VPN-only services)
### utility - CT 102 (192.168.1.102 / Tailscale: 100.64.0.15)
- Echo6 Search — branded [[searxng]] homepage (port 8080, https://echo6.co)
- Custom cyberpunk theme: JetBrains Mono font, cyan/yellow palette, dark backgrounds
- Homepage: centered Echo6 logo + pill search bar (Google-style, viewport-locked no-scroll)
- Results page: full-width two-column grid (results + sidebar), stretched search header
- Top nav bar: `.//photos`, `.//mail`, waffle app launcher (11 services), login avatar
- All nav links use [[authentik]] launch URLs for seamless SSO pass-through
- search.echo6.co permanently redirects to echo6.co (301)
- Redis/Valkey cache (valkey container)
- Compose path: `/opt/searxng/docker-compose.yml`
- Theme files: `/opt/searxng/custom/` (bind-mounted into container)
- `templates/simple/base.html` — custom template (nav, CSS, waffle menu, footer)
- `templates/simple/index.html` — custom homepage (Echo6 logo replaces [[searxng]] title)
- `img/echo6-logo.png` — Echo6 logo (replaces [[searxng]] logo)
- `img/favicon.png` — Echo6 favicon
- Config: `/opt/searxng/searxng-config/settings.yml` (instance_name: "Echo6", dark theme, center_alignment: false)
- [[searxng]] version: 2026.2.6 (Docker image: [[searxng]]/searxng:latest)
### utility - CT 104 (192.168.1.104 / Tailscale: 100.64.0.12)
- [[central]] data-hub spine (3 systemd units: central-supervisor, central-archive, central-gui)
- API/GUI on port 8000 (0.0.0.0), NATS :4222/:8222, PostgreSQL/TimescaleDB :5432
- ~25 domain adapters (traffic, wildfire, weather, hydro, earthquakes, avalanche, disasters, satellite)
- Serves navi traffic tiles at auth-exempt /api/traffic/flow/{z}/{x}/{y}.png|pbf
- Tailscale hostname: central.echo6.mesh
### utility - CT 108 (192.168.1.144 / Tailscale: 100.64.0.32)
- [[meshai]] — LLM-powered Meshtastic mesh assistant (Docker)
- Bot name: AIDA, node ID !27780c47, channel 8 whitelist
- Image: work-meshai (local build, not ghcr.io/zvx-echo6/[[meshai]]:latest)
- Backend: gemini-3.1-flash-lite with Google Search grounding
- Connects to meshtasticd **on aida-nebra** (192.168.1.253:4403) — the AIDA-N2 node !27780c47
- Exposes port 8080 (web UI)
- Config TUI on port 7682 (`meshai --config`)
- Commands: !help, !ping, !status, !weather, !reset, !clear
- 7-day rolling conversation memory (SQLite), full history sent to LLM
- Response: 175 char chunks × 3 messages max
- Compose path: `/home/zvx/meshai/docker-compose.yml`
### utility - CT 113 (192.168.1.113 / Tailscale: 100.64.0.39)
- MeshWars Preview — public preview instance of the `feat/places` branch (Docker, compose path `/home/zvx/meshwars/docker-compose.yml`)
- Fronted by utility [[caddy]] at https://mwpreview.k7zvx.com (Tailscale IP, /admin + /api/admin/* blocked with 404 on the public host)
- /admin still reachable directly over the tailnet at 100.64.0.39:8090/admin
- Database is a SQLite online-backup clone of CT 119's `game.db`, taken read-only; the `place` seed table is not present in production's schema yet and is auto-reloaded by the app on each restart after a clone
- ADMIN_TOKEN is unique to this container, generated fresh — never copied from production
- Join and check-in registration disabled by default
### utility - CT 118 (192.168.1.118)
- Signal/Matrix room archive bot (archivist.service via systemd)
- 1 core, 1GB RAM, 8GB disk
- Not registered in Headscale (no Tailscale)
- Source: forge.echo6.co/matt/matrix-archivist (private)
- See `/home/zvx/projects/.ref/docs/matrix/archivist.md` for implementation details
### cloud - VM 151 "ha" (192.168.1.151 / Tailscale: 100.64.0.16)
- Home Assistant (Docker, port 8123)
- OS: Ubuntu 24.04
- Headscale node id 54, user echo6
- Home automation platform
### cloud - CT 120 (192.168.1.182 / Tailscale: 100.64.0.2)
- Immich photo management (https://immich.echo6.co)
- Port 2283
- NFS storage from pi-nas (/mnt/immich)
- Compose path: `/opt/immich/docker-compose.yml`
### cloud - CT 121 (192.168.1.183 / Tailscale: 100.64.0.11)
- Nextcloud AIO (https://nextcloud.echo6.co)
- Apache port 11000, AIO management on 8080
- NFS storage from pi-nas (/mnt/nextcloud)
- SSO via [[authentik]] OIDC
### media - VM 105 (192.168.1.160 / Tailscale: 100.64.0.18)
- ARR media automation stack (Docker)
- Jellyfin media server (port 8096, https://jellyfin.echo6.co)
- Jellyseer request management (port 5055, https://requests.echo6.co)
- Sonarr TV automation (port 8989, internal)
- Radarr movie automation (port 7878, internal)
- Prowlarr indexer manager (port 9696, internal)
- SABnzbd [[usenet]] downloader (port 8080, internal)
- NFS storage from pi-nas (/mnt/arr)
- Config dirs: /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
### media - CT 110 (192.168.1.170 / Tailscale: 100.64.0.23)
- PeerTube video streaming (https://stream.echo6.co)
- Native install (Node.js 22, PostgreSQL 16, Redis, nginx)
- Port 9000 (PeerTube), proxied via nginx on port 80
- NFS storage from pi-nas (/var/www/peertube/storage, /export/peertube)
- SSO via Authentik OIDC (peertube-plugin-auth-openid-connect)
- Privileged container (NFS bind-mount)
- Auto-transcription enabled (remote runners on cortex, Whisper medium model)
- **PeerTube Bulk Import Pipeline:**
- pt-downloader.service — YouTube channel downloader (yt-dlp, sliding window, cookie auth)
- pt-importer.service — Uploads downloaded videos to PeerTube via resumable upload API
- NordVPN (nordvpnd.service) — IP rotation for downloads
- Config: `/opt/bulk-import/config/` (channel-map.json, cookies.txt, downloader-state.json)
- Logs: `/opt/bulk-import/logs/`
- Pipeline dirs: `/var/www/peertube/storage/pipeline/{staging,completed,transcoded,failed}`
### data - Proxmox Host (192.168.1.240 / Tailscale: 100.64.0.6)
- Proxmox VE node (echo6-cluster)
- Samba (smbd/nmbd, system packages)
- Share: `//data/library` → `/mnt/data/library` (browseable, read-write, guest OK, force user/group root)
- Workgroup: WORKGROUP, standalone server
### data - VM 1130 "recon-vm" (192.168.1.130 / Tailscale: 100.64.0.24)
- **Migrated from CT 130 (LXC) on 2026-04-19**
- OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-110-generic
- Resources: 4 cores, 24GB RAM, 180GB disk
- Software: Docker 29.4.0, Python 3.12.3 (venv), Tailscale, nginx, sqlite3
- RECON knowledge extraction pipeline
- systemd services: `recon.service`, `recon-watchdog.service`, `kiwix.service`
- Dashboard + API on port 8420 (https://recon.echo6.co)
- navi-config on port 8422 (node config API)
- navi-contacts on port 8423 (contact enrichment API)
- navi-landclass on port 8424 (land classification API)
- navi-places on port 8425 (OSM place detail/enrichment)
- navi-geo on port 8426 (geocode/reverse geocode API)
- navi-admin on port 8427 (fleet admin-info aggregator)
- navi-offroute on port 8428 (off-network router + MVUM API)
- nginx file server on port 8888 (https://files.echo6.co, Authentik forward auth)
- Kiwix-serve on port 8430 (ZIM library, 10 sources)
- navi (navi.echo6.co) on port :8440 — offline navigation platform front door (nginx SPA + API gateway); see [[navi]]
- dem-origin on port :8441 (localhost only) — DEM tile origin serving 658 GB planet-dem.pmtiles
- Geo backends (Docker + system): Valhalla :8002 (routing), Nominatim :8010 (geocoder), Photon :2322 (geocoder + Elasticsearch :9201), PostgreSQL/PostGIS :5432 (padus, overture DBs)
- Install: `/opt/recon/` (Python 3, Flask, venv)
- NFS mounts: pi-nas:/export/library → /mnt/library (PDF source), /mnt/nav, /mnt/kiwix
- Pipeline: Extract (PyPDF2→pdftotext→Tesseract→Gemini Vision) → Enrich (Gemini) → Embed (TEI/Qdrant)
- DB: SQLite (status), Qdrant on cortex:6333 (vectors)
- Backups: rsync to Contabo every 6hrs (concepts, text, DB, config), DB snapshot every 2hrs
- Config: `/opt/recon/config.yaml`, keys in `/opt/recon/.env`
- Docs: `/opt/recon/PROJECT-BIBLE.md`
- User: zvx (sudo, SSH key auth)
### utility - CT 106 (192.168.1.106)
- IdahoMesh Headscale (https://vpn.idahomesh.com)
- Container name: meshtastic-hs
- Manages meshtastic mesh VPN (separate from echo6 Headscale on Contabo)
- Users: malice, sidpatchy, nebra
### utility - CT 107 (192.168.1.107)
- mesh-bridge — dual tailscaled instance
- Bridges echo6 (100.64.0.0/10) ↔ idahomesh (100.100.0.0/16) networks
- NAT masquerade + subnet route advertisement
- Echo6 clients need `--accept-routes` to reach idahomesh devices
- iptables FORWARD rules must be BEFORE `ts-forward` jump (Tailscale drops cross-tailnet packets otherwise)
- Echo6 socket: `/run/tailscale/tailscaled.sock` (port 41641)
- IdahoMesh socket: `/var/run/tailscale-meshtastic/tailscaled.sock` (port 41642, tun=tailscale1)
- Rules persisted: `/etc/iptables/rules.v4` via `iptables-restore.service`
### pi-nas (192.168.1.245 / Tailscale: 100.64.0.21)
- OpenMediaVault NAS (https://nas.echo6.co)
- Port 80 (HTTP)
- Internet Archive CLI (`ia` v5.7.2) installed for archive.org uploads
### aida-nebra (192.168.1.253 / Tailscale: 100.64.0.9)
- **AIDA-N2(RPT,LLM)** — meshtasticd node `!27780c47` (short name: AIDA)
- Hardware: Nebra 2W SX1262 hat (ZebraHat config in `/etc/meshtasticd/config.d/`)
- Port: 4403 (default), firmware 2.7.19 (PORTDUINO/native)
- Role: CLIENT_BASE, position: 42.574, -114.607 (manual)
- MAC source: eth0 (derived MAC `00:bd:27:78:0c:47`)
- [[meshai]] bot (CT 108) connects to this node via TCP `localhost:4403` (Docker network)
- Service: `meshtasticd.service` (single instance, runs as user meshtastic)
- Config: `/etc/meshtasticd/config.yaml` + `/etc/meshtasticd/config.d/ZebraHat_2W.yaml`
- User: zvx, password auth (`sshpass -p '7redditGold' ssh zvx@aida-nebra`)
### mt-isr (192.168.1.141 / IdahoMesh: 100.100.0.5)
- Raspberry Pi Zero 2 W, Debian 13 (trixie), Waveshare ETH/USB HUB HAT
- No meshtasticd (G2 managed via WiFi TCP, not local daemon)
- Meshtastic Python CLI v2.7.7 in venv (`/home/isr/meshtastic-cli/`)
- Tailscale on IdahoMesh tailnet (vpn.idahomesh.com, nebra user)
- WiFi hotspot: ISR-MESH (192.168.4.0/24, PMF disabled for ESP32 compatibility)
- Station G2 radio connected via WiFi at 192.168.4.241, managed via TCP
- G2 config: Freq51 (ch0, psk=1A==) + MediumFast (ch1), MEDIUM_FAST preset, ch=51, txPower=11
- G2 gold config backup: `isr@192.168.1.141:~/backups/g2-gold-config.yaml`
- [[dns]] bootstrap drop-in for tailscaled (reboot-safe)
- User: isr, password auth (see credentials)
### mt-burleybutte (192.168.1.185)
- meshtasticd (software Meshtastic node, Nebra 2W hat)
- Raspberry Pi OS, user bb
- Static MAC: A7:A1:30:79:BB:BB
- Tailscale registered on IdahoMesh Headscale (vpn.idahomesh.com) under malice user
### edge2 - CT 103 (10.10.10.21 / Tailscale: 100.64.0.34, node 46 `forgejo`)
- Forgejo git server (https://forge.echo6.co — **migrated from Contabo 2026-06-16**)
- Headscale node id 46, name `forgejo`, user `echo6`
- Compose path: `/opt/forgejo/docker-compose.yml`
- Containers: `forgejo` (codeberg.org/forgejo/forgejo:14) + `forgejo-db` (postgres:16-alpine)
- Volumes: `forgejo-data` (git repos, app.ini, SSH host keys) + `forgejo-db`
- HTTP: binds to `100.64.0.34:3001`; edge2 host Caddy proxies here over tailnet (`reverse_proxy 100.64.0.34:3001`)
- SSH: `git@forge.echo6.co:2222` → edge2 iptables DNAT (systemd unit `forgejo-ssh-dnat.service`, enabled, reboot-persistent) → `100.64.0.34:2222` → container port 22; SSH host keys preserved (no client warning)
- Login via Authentik OIDC (auth.echo6.co) — unchanged
- DB: PostgreSQL 16 (forgejo_db container); 9 repos, 1 user verified at migration
- Source (Contabo `/opt/forgejo`) was wiped with edge1 OS rebuild 2026-06-19
### edge2 - CT 104 (10.10.10.22 / Tailscale: 100.64.0.35, node `livesync`)
- LiveSync Obsidian sync service (https://notes.echo6.co — **migrated from Contabo 2026-06-16**)
- Headscale hostname `livesync`, tailnet IP 100.64.0.35
- Compose path: `/opt/livesync/docker-compose.yml`
- Containers: `livesync-couchdb` (couchdb:3.4) + `livesync-provisioner` (custom image)
- Named volumes: `couchdb-data`, `provisioner-data`
- Bind mounts: `couchdb/local.ini`, `couchdb/local.d/` (incl. `jwt-keys.ini`)
- CouchDB binds to `100.64.0.35:5984`; provisioner binds to `100.64.0.35:5985`
- edge2 host Caddy proxies `notes.echo6.co` → `reverse_proxy 100.64.0.35:5984` (CouchDB) + `reverse_proxy 100.64.0.35:5985` (provisioner); Authentik forward_auth on `/_provision`; Obsidian CORS preserved
- Auth: per-user JWT (ES512) via provisioner; databases: `cc-db`, `userdb-matt`
- Data: ~16 MB in couchdb-data
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs on `local`; unprivileged; onboot; Docker
### edge2 - CT 105 (10.10.10.23 / Tailscale: 100.64.0.36, node 48 `authentik`)
- Authentik SSO platform (https://auth.echo6.co — **migrated from Contabo 2026-06-18**)
- Headscale node id 48, hostname `authentik`, tailnet IP 100.64.0.36
- Compose path: `/opt/authentik/docker-compose.yml`
- Containers: `authentik-server` + `authentik-worker` (ghcr.io/goauthentik/server:2025.12.4) + `authentik-postgres` (postgres:16); NO Redis
- Worker runs as user:root and has docker.sock bind-mount (manages embedded outposts)
- Binds to `100.64.0.36:9000`; edge2 host Caddy proxies here over tailnet for both `auth.echo6.co` (catch-all + outpost path matcher) and `notes.echo6.co` outpost/forward_auth references
- `AUTHENTIK_SECRET_KEY` carried byte-for-byte (sessions stayed valid across cutover — users dropped straight in)
- Bind-mounts (data/media, branding, certs, custom-templates) migrated intact
- Email dep: mail.echo6.co (unchanged)
- DB: PostgreSQL 16 (`authentik-postgres` container); ~705 MB (~18 MB pg_dump)
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- Reboot-survival fix: systemd unit on CT 105 gates `docker compose up` on `tailscale-online` (Docker was racing Tailscale on boot, failing the bind to the tailnet IP)
- **Resources:** 2 cores / 4096 MB RAM / 512 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
### edge2 - CT 106 (10.10.10.24 / Tailscale: 100.64.0.37, node `matrix`)
- Matrix stack (https://matrix.echo6.co, https://element.echo6.co — **migrated from Contabo 2026-06-18**)
- Headscale hostname `matrix`, tailnet IP 100.64.0.37
- Compose path: `/opt/matrix/matrix-stack.service` (systemd service with tailscale-before-docker reboot guard)
- Containers: `synapse` (:8008), `matrix-mas` (:8085), `element` (:8088), `matrix-postgres`, `mautrix-signal` (29328 internal) — 5 containers total
- Databases: PostgreSQL DBs `synapse`, `mas`, `mautrix_signal` on `matrix-postgres` container — 3 DBs total
- Binds to `100.64.0.37`; edge2 host Caddy proxies:
- `matrix.echo6.co` → [[synapse]] (`100.64.0.37:8008`) + MAS sub-routes
- `element.echo6.co` → Element (`100.64.0.37:8088`)
- MAS (`100.64.0.37:8085`) handles login/logout/refresh/auth_metadata
- Federation delegation from apex `echo6.co` (`.well-known`) unchanged
- MAS→Authentik OIDC (`auth.echo6.co`) unchanged
- Signal bridge: `@signalbot:echo6.co`, E2BE + MSC4190, double puppeting; bridge state preserved in `mautrix_signal` DB
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 2 cores / 4096 MB RAM / 1024 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
### edge2 - CT 107 (10.10.10.25 / Tailscale: 100.64.0.38, node `headscale`)
- Headscale + Headplane (https://vpn.echo6.co — **migrated from Contabo 2026-06-19**)
- Headscale hostname `headscale`, tailnet IP 100.64.0.38
- Compose path: `/opt/headscale/` (managed via `headscale-stack.service` systemd reboot guard)
- Headscale 0.28.0 on `:8084`; Headplane on `:3100`
- edge2 host Caddy proxies `vpn.echo6.co` → `100.64.0.38:8084` (headscale) / `vpn.echo6.co/admin` → `100.64.0.38:3100` (headplane)
- `noise_private.key` carried byte-for-byte (server identity preserved); 39 nodes
- Public direct bind `5.189.158.149:8084` DROPPED — all clients use `vpn.echo6.co` via Caddy
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Resources:** 1 core / 512 MB RAM / 256 MB swap / 4 GB rootfs on `local`; unprivileged; onboot
### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
- Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
- **Version:** 1.37.1 (web vault 2026.6.4) — updated from 1.36.0 on 2026-08-04 (9 medium security advisories in 1.37.0; required for Bitwarden clients 2026.7.0+)
- Headscale node id 45, name `vaultwarden`, user `echo6`
- Compose path: `/opt/vaultwarden/docker-compose.yml`; data: `./data/`
- Binds to tailnet IP `100.64.0.33:8086:80`; edge2 host Caddy proxies here over tailnet
- SSO via Authentik (`SSO_ONLY=false` — local email+master-password login also works)
- Source on Contabo wiped with edge1 rebuild 2026-06-19
- **Migrated from Contabo to edge2 CT 102 on 2026-06-16**
### edge1 (5.189.158.149 / Tailscale: 100.64.0.40) — MAIL-ONLY host (rebuilt Contabo VPS, 2026-06-19)
> **Rebuilt in-place 2026-06-19.** Former Contabo VPS (tailnet node previously `contabo` / 100.64.0.1) was OS-reinstalled as Debian 12 + Proxmox 8.4.19. Renamed edge1. All evacuated services now live on edge2. Tailnet identity re-registered as `contabo` at **100.64.0.40**.
- **Role: MAIL ONLY.** All non-mail services (vault/forge/notes/auth/matrix/element/vpn/proxmox) are fronted by **edge2** (184.174.35.153).
- **Host Caddy** serves `mail.echo6.co`, `autodiscover.echo6.co`, `autoconfig.echo6.co` → Mailcow CT 101 at `10.10.10.2:8453`; includes `header_up Host {host}` to prevent nginx leaking internal address in redirects.
- **mailcow-dnat.service** — source-IP-preserving DNAT for raw mail ports 25/465/587/110/143/993/995/4190 → 10.10.10.2 (systemd oneshot, reboot-persistent).
- **staticroute.service** — systemd oneshot for 10.10.10.0/24 route (reboot-persistent).
- Netfilter modules (`nf_conntrack`, `nf_nat`, etc.) persisted via `/etc/modules-load.d/`.
- Tailscale on host (node `contabo`, 100.64.0.40).
- unattended-upgrades enabled.
- SSH: `ssh -i ~/.ssh/contabo2_ed25519 root@5.189.158.149`
#### edge1 — CT 101 (10.10.10.2) — Mailcow
- **Mailcow** email server (privileged LXC, updated to commit `52a41b4d`, SOGo 5.12.8)
- Accessible at `mail.echo6.co` / `5.189.158.149` (via host Caddy + DNAT)
- mail_crypt secondary key in `data/conf/dovecot/extra.conf` (`ecprivkey.pem.fresh-20260619` as secondary; both primary + secondary loaded for decryption of old mail)
- [[dns]]: `mail/autodiscover/autoconfig.echo6.co` → 5.189.158.149
#### Previously on Contabo (now evacuated or decommissioned)
- ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (edge2 Caddy now proxies auth.echo6.co → 100.64.0.36:9000)
- ~~Forge (Git)~~ — **migrated to edge2 CT 103 on 2026-06-16** (edge2 Caddy proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service on edge2)
- ~~Headscale~~ — **migrated to edge2 CT 107 on 2026-06-19** (edge2 Caddy proxies vpn.echo6.co → 100.64.0.38:8084; noise_private.key carried byte-for-byte; direct public bind 5.189.158.149:8084 DROPPED)
- ~~Vaultwarden~~ — **migrated to edge2 CT 102 on 2026-06-16** (edge2 Caddy proxies to 100.64.0.33:8086)
- ~~Syncthing~~ — **decommissioned 2026-06-19** with edge1 OS rebuild (Syncthing state removed; Forge is now the durable backup via autocommit cron)
- ~~WATCHTOWER~~ — **decommissioned 2026-06-16**
- ~~Matrix [[synapse]]~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~Element Web~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~[[mautrix_signal]] bridge~~ — **migrated to edge2 CT 106 on 2026-06-18**
- ~~LiveSync~~ — **migrated to edge2 CT 104 on 2026-06-16**
- ~~TAK Server~~ — **decommissioned 2026-06-16** (archived to forge.echo6.co/matt/archive-tak-server)
- ~~SIGIL~~ — **decommissioned 2026-06-16**
- ~~echo6-agent~~ — **decommissioned 2026-06-16**
- ~~nexus-hub~~ + ~~nexus-agent~~ — **decommissioned 2026-06-16**
- ~~Termix~~ — **wiped with edge1 rebuild 2026-06-19**
## Decommissioned Services
Services stopped, archived, and removed from Caddy/[[dns]] as of their decommission date. On-disk dirs on Contabo retained until edge1 rebuild wipes them.
| Service | Decommissioned | Archive Repo | Notes |
|---------|---------------|-------------|-------|
| TAK Server | 2026-06-16 | forge.echo6.co/matt/archive-tak-server | Was Docker `/opt/tak-server-deploy` on Contabo; `tak.echo6.co` Caddy block + GoDaddy A record removed |
| SIGIL | 2026-06-16 | forge.echo6.co/matt/archive-tak-server | Was Docker `/opt/sigil` on Contabo; served at `tak.echo6.co/sigil` |
| WATCHTOWER | 2026-06-16 | forge.echo6.co/matt/archive-watchtower | Was Docker `/opt/watchtower` on Contabo; `wt.echo6.co` Caddy block + GoDaddy A record removed |
| echo6-agent | 2026-06-16 | forge.echo6.co/matt/archive-echo6-agent | Was systemd unit at `/opt/echo6-agent/` on Contabo; unit disabled |
| nexus-hub | 2026-06-16 | forge.echo6.co/matt/archive-nexus-hub | Was systemd unit at `/root/nexus-hub` on Contabo; unit disabled |
| nexus-agent | 2026-06-16 | forge.echo6.co/matt/archive-nexus-agent | Was systemd unit at `/root/nexus-agent` on Contabo; unit disabled |
## Adding New Services
When deploying a new service, update this file with:
1. Service name
2. Host location (server + container if applicable)
3. IP:Port
4. Access method (internal only vs public URL)
5. Brief description
## Naming Conventions
- **Internal services:** Access via Tailscale IP (100.64.x.x) or local IP
- **Public services:** Access via `*.echo6.co` subdomain through Caddy reverse proxy
### Lidarr on Steroids (lidarr.echo6.co)
- **Container:** lidarr (youegraillot/lidarr-on-steroids:latest)
- **Host:** media VM 105 (192.168.1.160)
- **Ports:** 8686 (Lidarr), 6595 (Deemix)
- **Network:** arr-net
- **Config:** /opt/arr/lidarr/config (Lidarr), /opt/arr/lidarr/config_deemix (Deemix)
- **Compose:** /opt/arr/docker-compose.yml
- **Music root:** /mnt/arr/music (NFS from pi-nas)
- **Downloads:** /mnt/arr/downloads (shared with SABnzbd)
- **API key:** 78f026ec93a94d8eb3177816b74a57b7
- **Caddy:** lidarr.echo6.co -> 100.64.0.18:8686 (Authentik forward auth)
- **Prowlarr:** fullSync configured
- **SABnzbd:** configured (music category)
- **Deemix:** port 6595, NOT exposed via Caddy (Tailscale-only access)
- **PUID/PGID:** 1000/1000, TZ: America/Boise
### Navidrome (navidrome.echo6.co)
- **Container:** navidrome (deluan/navidrome:latest)
- **Host:** media VM 105 (192.168.1.160)
- **Port:** 4533
- **Network:** arr-net
- **Data volume:** arr_navidrome-data (named Docker volume)
- **Music volume:** /mnt/arr/music (read-only, shared with Lidarr)
- **Compose:** /opt/arr/docker-compose.yml
- **Caddy:** navidrome.echo6.co -> 100.64.0.18:4533 (Authentik forward auth)
- **User:** 1000:1000
- **Scan schedule:** every 1 hour
- **Admin setup:** First login at https://navidrome.echo6.co creates admin account

67
vault/docs/services/usenet.md
View file

@ -1,67 +0,0 @@
---
title: Usenet Configuration
type: reference
tags:
- media
aliases: []
related:
- [[dns]]
- [[glossary]]
- [[proxmox-onboard-node]]
- [[environment]]
- [[recon-service-integration]]
updated: 2026-07-13
---
# Usenet Configuration
> **Credentials:** All passwords and API keys stored in `/home/zvx/projects/.ref/credentials`
## Infrastructure
| Component | Location | IP | Port | Access |
|-----------|----------|-----|------|--------|
| SABnzbd | arr VM (media VM 105) | 192.168.1.160 | 8080 | Internal only (LAN/Tailscale) |
## Usenet Providers
### Giganews
| Field | Value |
|-------|-------|
| Host | news.giganews.com |
| Port | 563 |
| SSL | Yes |
| Username | gn1369344 |
| Password | (see .ref/credentials) |
| Connections | 100 |
### EasyUsenet
| Field | Value |
|-------|-------|
| Host | reader.easyusenet.nl |
| Port | 563 |
| SSL | Yes |
| Username | esy_6926b8b819b97 |
| Password | (see .ref/credentials) |
| Connections | 100 |
| Speed | 2000 Mbit/s |
| Retention | 3700+ days |
| Alt Ports (SSL) | 443 |
| Alt Ports (non-SSL) | 119, 80 |
## Usenet Indexers
### NZBgeek
| Field | Value |
|-------|-------|
| URL | https://api.nzbgeek.info |
| API Key | (see .ref/credentials) |
### NinjaCentral
| Field | Value |
|-------|-------|
| URL | https://ninjacentral.co.za |
| API Key | (see .ref/credentials) |

390
vault/docs/software/authentik.md
View file

@ -1,390 +0,0 @@
---
title: Authentik SSO Configuration
type: reference
tags:
- auth
aliases: []
related:
- [[authentik-oidc-application]]
- [[mailcow-create-mailbox]]
- [[caddy]]
- [[edge2-access-reference]]
- [[headscale-oidc-boot-order]]
updated: 2026-07-13
---
# Authentik SSO Configuration
> Migrated off Contabo → edge2 on 2026-06-18.
## Location
- **Server:** edge2 CT 105 (100.64.0.36)
- **URL:** https://auth.echo6.co
- **Internal Port:** 9000
## API Access
API token stored in `/home/zvx/projects/.ref/credentials` as `AUTHENTIK_API_TOKEN`
```bash
# Test API access
curl -s "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" | python3 -m json.tool
```
## Flow UUIDs
Required for OAuth2 provider creation:
| Flow | UUID |
|------|------|
| Authorization (implicit) | `86051292-389f-4bd9-b0f9-53cd32f197fd` |
| Authorization (explicit) | `6f9f5c89-9f98-4776-9e0d-a72a8ad17963` |
| Invalidation | `ed861c0d-2c81-4c3d-819b-946a21c4296a` |
| Provider Invalidation | `1eb91626-19a3-4f45-b384-d699c6189197` |
## Signing Key
| Key | UUID |
|-----|------|
| authentik Self-signed Certificate | `09f508f0-6b8e-4031-8563-0d3cebf86868` |
## Property Mappings (Standard OIDC Set)
Include these three in every provider:
| Mapping | UUID |
|---------|------|
| OpenID Connect scope: openid | `c6426dad-0d85-4daa-89fe-cb850ad4bfd7` |
| Echo6 OAuth: email (verified=true) | `02c22323-da89-457a-bc12-7f4dd6a3d8ab` |
| OpenID Connect scope: profile | `113ab791-fa04-4e8c-b103-09d706bc21b4` |
**Note:** The default email scope (`096b0d6f`) was changed in 2025.10 to return `email_verified: false`. Our custom scope (`02c22323`) overrides this to always return `true`. All 14 OAuth2 providers were migrated to use the custom scope on 2026-02-16.
## Current OAuth2 Providers
| PK | Name | Client ID | Application | Redirect URI |
|----|------|-----------|-------------|--------------|
| 1 | Mailcow | mailcow | Mailcow | `https://mail.echo6.co/sso/oidc` |
| 2 | Forgejo | forgejo | Forgejo | `https://forge.echo6.co/user/oauth2/Authentik/callback` |
| 3 | Vaultwarden | vaultwarden | Vaultwarden | `https://vault.echo6.co/identity/connect/callback` |
| 4 | Proxmox | proxmox | Proxmox VE | `https://proxmox.echo6.co` (regex) |
| 5 | Headscale | headscale | Headscale VPN | `https://vpn.echo6.co/oidc/callback` |
| 6 | Headplane | headplane | Headplane | `https://vpn.echo6.co/admin/oidc/callback` |
| 8 | Nextcloud | nextcloud | Nextcloud | `https://nextcloud.echo6.co/apps/oidc_login/oidc` |
| 9 | Immich | immich | Immich | `https://immich.echo6.co/auth/login`, `/api/oauth/mobile-redirect` |
| 10 | jellyfin | jellyfin | Jellyfin | `https://jellyfin.echo6.co/sso/OID/redirect/Authentik` |
| 11 | jellyseer | jellyseer | Jellyseer | `https://requests.echo6.co/(login\|api/v1/auth/oidc-callback).*` (regex) |
| 12 | PeerTube | peertube | PeerTube | `https://stream.echo6.co/plugins/auth-openid-connect/...` |
| 13 | WATCHTOWER | watchtower | WATCHTOWER | Forward auth (proxy provider) |
| 14 | Open WebUI | open-webui | Open WebUI | `https://ai.echo6.co/oauth/oidc/callback` |
| 15 | Matrix | 93kCoZkBlnJyD9EcAm7E4btKflecOcBm9DGONB5T | Matrix | `https://matrix.echo6.co/_synapse/client/oidc/callback` |
| 16 | Files Forward Auth | — | Files | Forward auth (proxy provider) |
| 17 | LiveSync Provisioner | ZBoLdYmxlSUyMqgekswIPS4YuaeBn5uCr8GtWm5H | LiveSync | Forward auth (proxy provider) |
## Groups
| Name | PK | Superuser | Members | Used By |
|------|----|-----------|---------|----- ---|
| authentik Admins | `9944e153-f860-4443-81d1-ae544f611806` | Yes | akadmin, matt | All apps (admin access) |
| media-users | `0820b2b8-6c54-4c20-9a0a-872820e6d9ea` | No | jodie, matt | Jellyfin, Jellyseer, PeerTube |
| ai-users | `0631b273-cfd8-4ed1-afa6-e262d0dc5a69` | No | matt | Open WebUI |
| cloud-users | `db3cbf5d-8057-4e33-8e8d-95bfdb35fbac` | No | — | Immich, Nextcloud |
| communication-users | `31bce176-cd86-4aea-8db3-a57e03d5c2d1` | No | — | Mailcow, Matrix |
| productivity-users | `698d80c7-7c29-43cd-b5d4-9eb24c85a6cc` | No | — | — |
| security-users | `f345a043-c2a4-4906-a43b-9860eae86ee1` | No | — | — |
| proxmox_admins | `d85a868d-7d1e-4585-92a8-b8bb86771b53` | No | akadmin, matt | Proxmox VE |
| proxmox_users | `cf26703a-a824-47dd-9550-30b848a8ce5f` | No | — | Proxmox VE |
| authentik Read-only | `ce03664b-46f3-43b7-9967-5f65d591fdb6` | No | — | RBAC read-only role |
| livesync-users | `8e575a86-326e-4df8-8828-8379d8ab861f` | No | matt | LiveSync |
## Application Access Pattern
Every application uses `policy_engine_mode: "any"` with two group bindings:
1. **authentik Admins** — gives admin/superuser access to all apps
2. **Service-specific group** — controls which regular users can access the app
Users must be in at least one bound group to access the application.
## Create New API Token
```bash
ssh edge2 'sudo pct exec 105 -- docker exec -i authentik-server ak shell' <<'PYEOF'
from authentik.core.models import Token, TokenIntents, User
user = User.objects.get(username="akadmin")
Token.objects.filter(identifier="token-name").delete()
t = Token(identifier="token-name", user=user, intent=TokenIntents.INTENT_API, expiring=False, managed=None)
t.save()
print(t.key)
PYEOF
```
## Full OAuth2 App Setup (Provider + Application + Group + Bindings)
### Step 1: Create the OAuth2 provider
```bash
curl -s -X POST "https://auth.echo6.co/api/v3/providers/oauth2/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"authorization_flow": "86051292-389f-4bd9-b0f9-53cd32f197fd",
"invalidation_flow": "ed861c0d-2c81-4c3d-819b-946a21c4296a",
"property_mappings": [
"c6426dad-0d85-4daa-89fe-cb850ad4bfd7",
"02c22323-da89-457a-bc12-7f4dd6a3d8ab",
"113ab791-fa04-4e8c-b103-09d706bc21b4"
],
"client_type": "confidential",
"client_id": "appname",
"client_secret": "<generate-a-long-random-secret>",
"redirect_uris": [{"matching_mode": "strict", "url": "https://app.echo6.co/callback"}],
"sub_mode": "user_username",
"signing_key": "09f508f0-6b8e-4031-8563-0d3cebf86868",
"include_claims_in_id_token": true
}'
# Note the "pk" from the response — needed for Step 2
```
### Step 2: Create the application
```bash
curl -s -X POST "https://auth.echo6.co/api/v3/core/applications/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "AppName",
"slug": "appname",
"provider": PROVIDER_PK,
"meta_launch_url": "https://app.echo6.co",
"policy_engine_mode": "any"
}'
# Note the "pk" from the response — needed for Step 4
```
### Step 3: Create the user group
```bash
curl -s -X POST "https://auth.echo6.co/api/v3/core/groups/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "appname-users",
"is_superuser": false,
"users": [7]
}'
# user 7 = matt. Note the "pk" from the response.
```
### Step 4: Bind groups to the application
```bash
# Bind authentik Admins
curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"target": "APPLICATION_PK",
"group": "9944e153-f860-4443-81d1-ae544f611806",
"order": 0, "enabled": true, "negate": false, "timeout": 30
}'
# Bind service-specific group
curl -s -X POST "https://auth.echo6.co/api/v3/policies/bindings/" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"target": "APPLICATION_PK",
"group": "GROUP_PK",
"order": 0, "enabled": true, "negate": false, "timeout": 30
}'
```
### Step 5: Verify
```bash
# OIDC discovery should return endpoints
curl -s "https://auth.echo6.co/application/o/appname/.well-known/openid-configuration" | python3 -m json.tool
# Check bindings
curl -s "https://auth.echo6.co/api/v3/policies/bindings/?target=APPLICATION_PK" \
-H "Authorization: Bearer $AUTHENTIK_API_TOKEN"
```
## Common Redirect URI Patterns
| Application Type | Redirect URI Pattern |
|------------------|---------------------|
| Web app (generic) | `https://app.echo6.co/callback` |
| Web app (oauth path) | `https://app.echo6.co/oauth/callback` |
| Open WebUI (OIDC) | `https://app.echo6.co/oauth/oidc/callback` |
| Forgejo | `https://app.echo6.co/user/oauth2/Authentik/callback` |
| Jellyfin (SSO plugin) | `https://app.echo6.co/sso/OID/redirect/Authentik` |
| [[caddy]] forward auth | `https://app.echo6.co/outpost.goauthentik.io/callback` |
## Users
| PK | Username | Name | Email |
|----|----------|------|-------|
| 6 | akadmin | authentik Default Admin | root@example.com |
| 7 | matt | Matt Johnson | matt@echo6.co |
| 9 | jodie | Jodie | johnsonsinidaho@gmail.com |
## Email Invitation System
### SMTP Configuration
Authentik sends email via Mailcow:
| Setting | Value |
|---------|-------|
| SMTP Host | mail.echo6.co |
| SMTP Port | 587 (STARTTLS) |
| Username | no-reply@echo6.co |
| From | no-reply@echo6.co |
**Important:** The no-reply@echo6.co mailbox MUST have `authsource=mailcow` in the Mailcow database (not `generic-oidc`). If it gets reset to `generic-oidc`, SMTP auth will fail because Dovecot tries to authenticate via Authentik SSO instead of the local password. Fix with:
```sql
docker exec mailcowdockerized-mysql-mailcow-1 mysql -u mailcow -p<DBPASS> mailcow \
-e "UPDATE mailbox SET authsource='mailcow' WHERE username='no-reply@echo6.co'"
```
### Enrollment Flow
| Component | PK |
|-----------|-----|
| Flow (invitation-enrollment) | `184a9e20-f5c5-4b44-8775-266a568439c0` |
| Invitation stage (enrollment-invitation) | `994252a6-a659-4304-b9aa-a6591857a53b` |
| Prompt stage (enrollment-credentials) | `a5e2a2f1-95c9-4627-a3dd-e0915ff64cda` |
| User write stage (enrollment-user-write) | `05f565e0-9d64-4ecc-b738-8ffc697ad829` |
| User login stage (enrollment-user-login) | `f23958ea-47bd-4fb2-a657-b6f85fc6331a` |
Stage order: Invitation (10) → Prompt (20) → User Write (30) → User Login (40)
Flow settings: `authentication=require_unauthenticated`, `continue_flow_without_invitation=false`
User write creates users under `users/enrolled` path.
### Email Automation
Uses the community pattern from [authentik/discussions/13305](https://github.com/goauthentik/authentik/discussions/13305):
| Component | PK |
|-----------|-----|
| Expression policy (invitation-email-sender) | `770188de-2b60-4e87-820a-fa64d419ed89` |
| Notification rule (invitation-email-trigger) | `a145a8c9-b3eb-440d-8b77-27139c346a17` |
When an invitation is created with an `email` field in custom attributes, the expression policy triggers `ak_send_email()` to send the enrollment link to the invitee automatically.
### How to Use
**Invite via email** (Admin UI → Directory → Invitations → Create):
1. Name it, select **Invitation Enrollment** flow, toggle **Single use** on, set expiry
2. Custom attributes:
```yaml
name: Jane Smith
email: jane@example.com
```
3. Click Create — email is sent automatically
**Invite via link** (no email):
1. Same as above but leave custom attributes empty (or omit `email` field)
2. Click Create → expand the row → copy the invitation link
3. Send the link manually
---
## Branding & Theming
Echo6 cyberpunk branding applied to Authentik 2025.12.4 via System → Brands.
### Brand Settings
| Setting | Value |
|---------|-------|
| Brand title | `echo6` |
| Theme | `dark` (forced, not automatic) |
| Logo | `/media/custom/echo6-logo.png` (uploaded via Customization → Files) |
| Favicon | `/media/custom/echo6-favicon.png` (uploaded via Customization → Files) |
| Custom CSS | Echo6 Authentik CSS (~200 rules, applied via Brand → Custom CSS field) |
### Flow Titles
| Flow | Title |
|------|-------|
| Authentication | `echo6 // login` |
| Invalidation | `echo6 // logout` |
| Recovery | `echo6 // recovery` |
| User Settings | `echo6 // settings` |
### Custom CSS Highlights
- **Font:** JetBrains Mono globally (with `:not()` exclusions for FontAwesome/PatternFly icon fonts)
- **Colors:** Cyan `#28C0E8` primary accent, Yellow `#F0D848` secondary, dark backgrounds `#0a0e17`/`#111827`/`#1a2332`
- **Login card:** Dark background, cyan-glow focus on inputs, branded submit button
- **Admin sidebar:** Dark with cyan hover/active states
- **User dashboard:** 3-column grid layout, dark cards with cyan border on hover
- **Application icons:** Custom SVG icons uploaded for all 15 [[services]]
### CSS Storage
The custom CSS is stored in the Brand model's `branding_custom_css` field. To update:
```bash
# Copy CSS to edge2 CT 105 (via edge2 host, then pct push into the CT)
scp /path/to/echo6-authentik.css edge2:/tmp/echo6-authentik.css
ssh edge2 'sudo pct push 105 /tmp/echo6-authentik.css /opt/authentik/branding/custom.css'
# Load into Brand model via ak shell
ssh edge2 'sudo pct exec 105 -- docker exec -i authentik-server ak shell' <<'PYEOF'
from authentik.brands.models import Brand
b = Brand.objects.get(domain="auth.echo6.co")
b.branding_custom_css = open("/media/custom/custom.css").read()
b.save()
PYEOF
# Restart to apply
ssh edge2 'sudo pct exec 105 -- bash -c "cd /opt/authentik && docker compose restart server worker"'
```
### SSO Launch URL Pattern
All authenticated service links use Authentik's application launch URL:
```
https://auth.echo6.co/application/launch/<app-slug>/
```
This provides seamless SSO: authenticated users pass through to the app, unauthenticated users get the login page then redirect to the app. Used by the [[searxng]] waffle menu and nav bar.
| App Slug | Service | Launch URL |
|----------|---------|-----------|
| open-webui | Aurora (AI) | `https://auth.echo6.co/application/launch/open-webui/` |
| peertube | Stream | `https://auth.echo6.co/application/launch/peertube/` |
| files | Files | `https://auth.echo6.co/application/launch/files/` |
| watchtower | Watchtower | `https://auth.echo6.co/application/launch/watchtower/` |
| immich | Photos | `https://auth.echo6.co/application/launch/immich/` |
| mailcow | Mail | `https://auth.echo6.co/application/launch/mailcow/` |
| nextcloud | Cloud | `https://auth.echo6.co/application/launch/nextcloud/` |
| jellyfin | Jellyfin | `https://auth.echo6.co/application/launch/jellyfin/` |
| jellyseer | Requests | `https://auth.echo6.co/application/launch/jellyseer/` |
### Brand Color Reference
| Color | Hex | Usage |
|-------|-----|-------|
| Cyan | `#28C0E8` | Primary accent, links, focus states |
| Cyan Light | `#5DD4F5` | Hover states |
| Yellow | `#F0D848` | Secondary accent, badges |
| BG Primary | `#0a0e17` | Page backgrounds |
| BG Secondary | `#111827` | Cards, inputs |
| BG Tertiary | `#1a2332` | Elevated surfaces |
| Border | `#1e3a5f` | All borders |
| Text Primary | `#e0e6ed` | Main text |
| Text Muted | `#7a8ca0` | Secondary text |
---
*Last updated: 2026-02-18 — Added LiveSync proxy provider (PK 17) + livesync-users group, Files Forward Auth (PK 16). Authentik 2025.12.4*

249
vault/docs/software/caddy.md
View file

@ -1,249 +0,0 @@
---
title: "Caddy & DNS Reference"
type: reference
tags:
- mesh
aliases: []
related:
- [[services]]
- [[ip-allocation]]
- [[lxc-service-migration]]
- [[expose-service-edge2]]
- [[matrix-synapse-deployment]]
updated: 2026-08-24
---
# Caddy & DNS Reference
> Migrated off Contabo → edge2 on 2026-06-19.
## edge2 Caddy (front door)
**Config:** `/etc/caddy/Caddyfile` on edge2 (`ssh edge2`; `root@<IP>` is stale)
**Global options:** `email admin@echo6.co`, `admin off` (no live reload — must `systemctl restart caddy`)
### Current Site Blocks
| Domain | Backend | Service |
|--------|---------|---------|
| auth.echo6.co | 100.64.0.36:9000 (Tailscale, CT 105) | [[authentik]] SSO |
| forge.echo6.co | 100.64.0.34:3001 (Tailscale, CT 103) | Forgejo Git |
| mail.echo6.co | — | Mailcow — moved to **edge1** (separate mail-only host, 10.10.10.2:8453 via edge1's own Caddy); not on edge2 |
| vpn.echo6.co | 100.64.0.38:8084 (Tailscale, CT 107) | Headscale |
| vpn.echo6.co/admin* | 100.64.0.38:3100 (Tailscale, CT 107) | Headplane |
| autodiscover.echo6.co | — | Mailcow autodiscover — moved to **edge1** (10.10.10.2:8453); not on edge2 |
| autoconfig.echo6.co | — | Mailcow autoconfig — moved to **edge1** (10.10.10.2:8453); not on edge2 |
| vault.echo6.co | 100.64.0.33:8086 (Tailscale, CT 102) | vaultwarden |
| proxmox.echo6.co | https://100.64.0.6:8006 (via Tailscale) | Proxmox VE (data node) — unchanged |
| wt.echo6.co | — | ~~WATCHTOWER ops dashboard~~ — **decommissioned 2026-06-16** (was 100.64.0.1, now dead) |
| matrix.echo6.co | 100.64.0.37:8008 + 100.64.0.37:8085 (Tailscale, CT 106) | Matrix [[synapse]] + MAS (login/logout/refresh/auth_metadata → MAS:8085, _matrix/* → [[synapse]]:8008, default → MAS:8085) |
| element.echo6.co | 100.64.0.37:8088 (Tailscale, CT 106) | Element Web client |
| notes.echo6.co | 100.64.0.35:5984 + 100.64.0.35:5985 (Tailscale, CT 104) | LiveSync (CouchDB + provisioner, forward auth on /_provision*, CORS for Obsidian) |
| tak.echo6.co | — | ~~TAK Server admin + SIGIL console~~ — **decommissioned 2026-06-16** (was 100.64.0.1, now dead) |
### Commands
```bash
ssh edge2
caddy validate --config /etc/caddy/Caddyfile
systemctl restart caddy # admin off, so reload won't work
journalctl -u caddy -f
```
---
## Utility Caddy (Home)
**Location:** CT 101 on utility Proxmox (192.168.1.101)
**Tailscale IP:** 100.64.0.8
**Config:** `/etc/caddy/Caddyfile` inside CT 101
**SSL Certs:** `/etc/caddy/certs/` (managed by acme.sh)
**Port forward:** Router 80/443 → 192.168.1.101
### Current Site Blocks
| Domain | Backend | Pattern | Service |
|--------|---------|---------|---------|
| mesh.echo6.co | 192.168.1.100:8080 | Local IP | meshmonitor ([[authentik]] forward auth) |
| echo6.co | 100.64.0.15:8080 | Tailscale | Echo6 Search ([[searxng]]) + Matrix well-known |
| search.echo6.co | — | — | 301 redirect to echo6.co |
| nas.echo6.co | 100.64.0.21:80 | Tailscale | OpenMediaVault (pi-nas) |
| immich.echo6.co | 192.168.1.182:2283 | Local IP | immich (has 2FA) |
| nextcloud.echo6.co | 192.168.1.183:11000 | Local IP | nextcloud AIO (SSO via [[authentik]]) |
| jellyfin.echo6.co | 100.64.0.18:8096 | Tailscale | Jellyfin media server (SSO via Authentik) |
| requests.echo6.co | 100.64.0.18:5055 | Tailscale | Jellyseer request management (SSO via Authentik) |
| stream.echo6.co | 192.168.1.170:80 | Local IP | peertube video streaming (SSO via Authentik) |
| ai.echo6.co | 100.64.0.14:8080 | Tailscale | Open WebUI (SSO via Authentik) |
| files.echo6.co | 100.64.0.24:8888 | Tailscale | [[recon]] PDF library (Authentik forward auth) |
| recon.echo6.co | 100.64.0.24:8420 | Tailscale | [[recon]] dashboard + API |
| lidarr.echo6.co | 100.64.0.18:8686 | Tailscale | lidarr music automation (Authentik forward auth) |
| navidrome.echo6.co | 100.64.0.18:4533 | Tailscale | navidrome music server (Authentik forward auth, /rest/* exempt for Subsonic API) |
| vpn.idahomesh.com | 192.168.1.106:8080 | Local IP | IdahoMesh Headscale VPN coordination |
| mwpreview.k7zvx.com | 100.64.0.39:8090 | Tailscale | [[meshwars]] public preview (CT 113, feat/places branch; /admin + /api/admin/* blocked -- 404 -- on this public host, reachable only over tailnet) |
### Commands
```bash
ssh root@192.168.1.241 'pct exec 101 -- cat /etc/caddy/Caddyfile'
ssh root@192.168.1.241 'pct exec 101 -- systemctl reload caddy'
ssh root@192.168.1.241 'pct exec 101 -- journalctl -u caddy -f'
```
---
## dnsmasq (Tailscale Split DNS) — HISTORICAL / OBSOLETE
> **Not in use.** Tailnet split-DNS for echo6.co was retired: echo6.co now resolves via public GoDaddy [[dns]], not internal dnsmasq (see [[services]]). This section documents the OLD setup that ran on the original Contabo VPS (100.64.0.1) before its [[2026-06-19]] rebuild into edge1 (mail-only). That host and its 100.64.0.1 tailnet identity are dead — do not repoint these records to edge1 or edge2; kept below for historical reference only.
**Config:** `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo (dead host, pre-2026-06-19)
**Listens on:** 100.64.0.1:53 (dead)
### Current Records
| Domain | Tailscale IP | Service |
|--------|-------------|---------|
| auth.echo6.co | 100.64.0.1 | Authentik |
| forge.echo6.co | 100.64.0.1 | Forgejo |
| mail.echo6.co | 100.64.0.1 | Mailcow |
| vpn.echo6.co | 100.64.0.1 | Headscale |
| vault.echo6.co | 100.64.0.1 | Vaultwarden |
| proxmox.echo6.co | 100.64.0.1 | Proxmox VE (via Caddy) |
| stream.echo6.co | 100.64.0.8 | PeerTube (via Utility Caddy) |
| notes.echo6.co | 100.64.0.1 | LiveSync CouchDB + provisioner (via Contabo Caddy) |
| tak.echo6.co | 100.64.0.1 | TAK Server + SIGIL (via Contabo Caddy) |
| jellyfin.echo6.co | 100.64.0.8 | Jellyfin (via utility Caddy) |
| requests.echo6.co | 100.64.0.8 | Jellyseer (via utility Caddy) |
| wt.echo6.co | 100.64.0.1 | WATCHTOWER ops dashboard |
| ai.echo6.co | 100.64.0.8 | Open WebUI (via utility Caddy) |
| matrix.echo6.co | 100.64.0.1 | Matrix [[synapse]] (via Contabo Caddy) |
| element.echo6.co | 100.64.0.1 | Element Web (via Contabo Caddy) |
| echo6.co | 100.64.0.8 | Echo6 Search homepage (via utility Caddy) |
| files.echo6.co | 100.64.0.8 | [[recon]] PDF library (via utility Caddy) |
| recon.echo6.co | 100.64.0.8 | [[recon]] dashboard (via utility Caddy) |
| lidarr.echo6.co | 100.64.0.8 | Lidarr music automation (via utility Caddy) |
| navidrome.echo6.co | 100.64.0.8 | Navidrome music server (via utility Caddy) |
### Commands
```bash
# HISTORICAL — host is dead, commands no longer work
ssh root@100.64.0.1
nano /etc/dnsmasq.d/tailscale-dns.conf
systemctl restart dnsmasq
dig +short forge.echo6.co @100.64.0.1 # Test
```
---
## GoDaddy DNS Records (echo6.co)
### edge2 Services → 184.174.35.153
| Subdomain | Service |
|-----------|---------|
| auth | Authentik SSO |
| forge | Forgejo Git |
| vpn | Headscale VPN |
| vault | Vaultwarden |
| matrix | Matrix [[synapse]] |
| element | Element Web |
| notes | LiveSync (CouchDB + provisioner) |
| proxmox | Proxmox VE (via Tailscale to data node) |
### edge1 Services (mail-only host) → 5.189.158.149
| Subdomain | Service |
|-----------|---------|
| mail | Mailcow Email |
### Decommissioned records (removed from GoDaddy, 2026-06-16)
| Subdomain | Service |
|-----------|---------|
| wt | ~~WATCHTOWER ops dashboard~~ |
| tak | ~~TAK Server + SIGIL~~ |
### Home Services → 199.6.36.163
| Subdomain | Service |
|-----------|---------|
| @ | Echo6 Search homepage ([[searxng]]) |
| ai | Open WebUI |
| stream | PeerTube |
| jellyfin | Jellyfin |
| mesh | MeshMonitor |
| nas | OpenMediaVault (pi-nas) |
| search | [[searxng]] (redirects to echo6.co) |
| immich | Immich |
| nextcloud | Nextcloud |
| requests | Jellyseer |
| files | [[recon]] PDF library |
| recon | RECON dashboard |
| lidarr | Lidarr music automation |
| navidrome | Navidrome music server |
### Email Records
| Type | Name | Value |
|------|------|-------|
| MX | @ | mail.echo6.co |
| CNAME | autoconfig | mail.echo6.co |
| CNAME | autodiscover | mail.echo6.co |
| TXT | @ | v=spf1 mx a:mail.echo6.co -all |
| TXT | _dmarc | v=DMARC1; p=quarantine |
| TXT | dkim._domainkey | (DKIM key) |
---
## Headscale Config
**Location:** `/opt/headscale/` on edge2 (CT 107, managed via `headscale-stack.service` systemd reboot guard)
**Data:** Named Docker volume `headscale_headscale-data`
**Config:** `/opt/headscale/config.yaml`
```yaml
dns:
base_domain: echo6.mesh
nameservers:
global:
- 1.1.1.1
oidc:
issuer: "https://auth.echo6.co/application/o/headscale/"
client_id: "headscale"
```
**Split [[dns]]:** Previously configured via dnsmasq on Contabo (pre-2026-06-19). Not verified on edge2/edge1 in this pass — flagged stale/unresolved; see `## dnsmasq (Tailscale Split DNS)` section below (also unverified).
**Headplane:** Deployed at `vpn.echo6.co/admin` - OIDC via Authentik. First login gets Owner.
---
## Port Map (edge2)
| Service | Container Port | Host Binding (Tailscale) | Public Domain |
|---------|---------------|--------------|---------------|
| Authentik | 9000 | 100.64.0.36:9000 (CT 105) | auth.echo6.co |
| Forgejo | 3000 | 100.64.0.34:3001 (CT 103) | forge.echo6.co |
| Forgejo SSH | 22 | edge2 iptables DNAT :2222 → 100.64.0.34:2222 (CT 103) | Direct (not proxied) |
| Headscale | 8080 | 100.64.0.38:8084 (CT 107) | vpn.echo6.co |
| Headplane | 3000 | 100.64.0.38:3100 (CT 107) | vpn.echo6.co/admin |
| Vaultwarden | 80 | 100.64.0.33:8086 (CT 102) | vault.echo6.co |
| Vaultwarden WS | 3012 | 100.64.0.33:3012 (CT 102) | vault.echo6.co/notifications/hub |
| ~~WATCHTOWER~~ | 8084 | — | ~~wt.echo6.co~~ — decommissioned 2026-06-16 |
| Matrix Synapse | 8008 | 100.64.0.37:8008 (CT 106) | matrix.echo6.co (/_matrix/*, /_synapse/*) |
| Matrix MAS | 8080 | 100.64.0.37:8085 (CT 106) | matrix.echo6.co (login/logout/refresh/auth_metadata, default) |
| Element Web | 80 | 100.64.0.37:8088 (CT 106) | element.echo6.co |
| LiveSync CouchDB | 5984 | 100.64.0.35:5984 (CT 104) | notes.echo6.co |
| LiveSync Provisioner | 8080 | 100.64.0.35:5985 (CT 104) | notes.echo6.co/_provision/* |
| ~~TAK Server Admin~~ | 8446 | — | ~~tak.echo6.co~~ — decommissioned 2026-06-16 |
| ~~SIGIL Console~~ | 8990 | — | ~~tak.echo6.co/sigil~~ — decommissioned 2026-06-16 |
## Port Map (edge1 — mail-only, separate host)
| Service | Container Port | Host Binding | Public Domain |
|---------|---------------|--------------|---------------|
| Mailcow | 8453 | 10.10.10.2:8453 (CT 101, via edge1 host Caddy + DNAT) | mail.echo6.co, autodiscover.echo6.co, autoconfig.echo6.co |
---
*Last updated: 2026-07-11 — Flip off Contabo completed: "Contabo Caddy" section → "edge2 Caddy" (front door for auth/forge/vpn/vault/matrix/element/notes/proxmox, CTs verified against [[ip-allocation]]/[[services]]); Mailcow + autodiscover/autoconfig moved to edge1 (separate mail-only host, not on edge2); WATCHTOWER + TAK/SIGIL marked decommissioned (dead 100.64.0.1 backends removed); Headscale config location + Port Map updated to edge2; `ssh root@100.64.0.1` → `ssh edge2`. dnsmasq split-DNS section marked HISTORICAL/OBSOLETE (echo6.co split-DNS retired, ran on the dead pre-2026-06-19 Contabo host, not repointed to edge1/edge2 per [[services]]); GoDaddy [[dns]] Records section corrected — edge2 [[services]] (auth/forge/vpn/vault/matrix/element/notes/proxmox) → 184.174.35.153, mail → edge1 5.189.158.149 (unchanged public IP), wt/tak marked as removed records. Prior: 2026-04-13 — Audit sync: added MAS routing on matrix.echo6.co, lidarr/navidrome/vpn.idahomesh.com to utility Caddy, proxmox/tak to GoDaddy, removed ghost docs.echo6.co entries, added dnsmasq lidarr/navidrome*

116
vault/docs/software/central.md
View file

@ -1,116 +0,0 @@
---
title: central — Data-Hub Spine
type: reference
tags:
- mesh
aliases: []
related:
- [[central-deploy-cutover]]
- [[services]]
- [[caddy]]
- [[fleet-platform-baseline]]
- [[fleet-patch-audit]]
updated: 2026-07-16
---
# central — Data-Hub Spine
> **RETIRED + DROPPED (2026-07-15).** Central has been replaced by [[conduit]]. Its app [[services]] were stopped and disabled 2026-07-14 (zero live consumers remained); on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed), and the shared Postgres instance was cleaned back to plain (TimescaleDB removed). Central is recoverable only from the pi-nas archive. **Everything below this point is historical** — it describes how central worked while it was live, and is kept for reference only.
## Overview
central is a multi-domain real-time data-hub spine. Adapters normalize upstream sources, publish CloudEvents to **NATS/JetStream**, and archive to **TimescaleDB/PostGIS** for historical and geospatial query. It is the live data backbone for [[navi]] traffic tiles and related situational-awareness feeds.
- **URL (internal):** http://central.echo6.mesh:8000 (mesh-only, no public exposure)
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
- **Repo:** github.com/zvx-echo6/central (public, Python, deployed at detached HEAD, tag v0.15.0)
- **Deploy path:** /opt/central (venv: /opt/central/.venv, env: /etc/central/central.env, system user: central)
## Host
| Attribute | Value |
|-----------|-------|
| Container | utility CT 104 |
| Local IP | 192.168.1.104 |
| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh |
| Resources | 4 cores / 12 GB RAM / 100 GB disk |
| OS | Ubuntu LXC (unprivileged) |
## Architecture
The data flow is: upstream APIs → adapters (central-supervisor) → NATS/JetStream :4222 → central-archive (TimescaleDB/PostGIS :5432). The central-gui (FastAPI + HTMX, :8000) exposes the API consumed by [[navi]] via recon-vm's nginx (port 8440, `^~ /api/traffic/` → `central.echo6.mesh:8000`).
## Systemd Services
All three units are **enabled and active**; [[deployment]] survives reboot. Deps: `nats-server`, `postgresql@16-main`.
| Unit | Role |
|------|------|
| central-supervisor | Adapter scheduler + CloudEvents publisher |
| central-archive | JetStream → TimescaleDB consumer |
| central-gui | FastAPI + HTMX web app + API (the :8000 listener) |
## Ports
| Port | Protocol | Purpose |
|------|----------|---------|
| :8000 | HTTP | GUI / API (bound 0.0.0.0) |
| :4222 | TCP | NATS client connections |
| :8222 | HTTP | NATS monitoring |
| :5432 | TCP | PostgreSQL 16 + TimescaleDB/PostGIS |
## Adapters (23 configured, 22 enabled)
| Domain | Sources |
|--------|---------|
| Traffic | ITD 511, WZDX, TomTom flow/incidents, 511 cameras |
| Wildfire | WFIGS incidents/perimeters, InciWeb, FIRMS |
| Weather / Space-weather | NWS, SWPC k-index/protons/alerts |
| Hydro | NWIS |
| Earthquakes | USGS |
| Avalanche | avalanche.org |
| Disasters | GDACS, EONET |
| Satellite | CelesTrak TLE, sat positions/orbits, satpass_predict (n2yo_visualpasses disabled 2026-06-27) |
## GUI / API Surface
Authenticated app with login/sessions/CSRF, first-run setup wizard, operator management, adapter configuration (incl. **no-code creation of generic REST/GeoJSON sources**, v0.15.0), stream viewer, **JetStream consumer management** (`/consumers` — view + delete consumers; central's own `archive-*` durable consumers are protected/non-deletable; CSRF + audit-logged), enrichment pipeline, monitoring-area management, API key management, audit log, and manual resend.
**Auth-exempt tile endpoints** (used by [[navi]], verified HTTP 200):
| Endpoint | Format |
|----------|--------|
| /api/traffic/flow/{z}/{x}/{y}.png | PNG tile (raster, image/png) |
| /api/traffic/flow/{z}/{x}/{y}.pbf | PBF tile (vector, application/x-protobuf) |
`/health` is also auth-exempt (returns `{"status":"ok"}`). All other `/api/` endpoints redirect to `/login`.
## Consumer — navi Integration
navi-traffic (navi's in-VM :8421 extraction service) was **retired on 2026-05-26** and cut over to central. recon-vm's nginx (`/etc/nginx/sites-available/navi.echo6.co`, port 8440) proxied `^~ /api/traffic/` → `central.echo6.mesh:8000` with a 120s tile cache. navi-traffic:8421 is confirmed dead and disabled.
**As of 2026-07-14, navi's `/api/traffic/` tiles were repointed to [[conduit]]:** recon-vm's nginx now rewrites `^~ /api/traffic/` → `/up/tomtom_flow_tiles/...` and proxies to `central.echo6.mesh:8010` ([[conduit]], co-resident on the same CT 104 host). Central's own tile endpoint (`:8000/api/traffic/...`) still exists and still works, but is no longer on navi's hot path — it remains the rollback target if the Conduit cutover needs to be reverted. See [[conduit-operations]] for the cutover and rollback procedure.
## Dependencies
| Component | Location |
|-----------|----------|
| NATS / JetStream | Local (central CT 104) |
| PostgreSQL 16 + TimescaleDB + PostGIS | Local (central CT 104) |
| Upstream APIs | ~20 external sources (see Adapters table) |
## Data Plane
Single TimescaleDB hypertable `public.events` (~40 GB, ~3.42 M rows, 331 chunks, compression not enabled). JetStream ~5.9 GB / 20 GB across 12 streams.
## Deploy / State Notes (as of 2026-07-13)
- **Deployed HEAD:** v0.15.0 (detached HEAD — the box tracks tags; sha `3c8da28`, `pyproject.toml` 0.15.0). All 3 units active, `/health` 200, 23 adapters configured / 22 enabled (only `n2yo_visualpasses` disabled).
- **v0.15.0 (deployed 2026-07-02, PRs #120–124):** the GUI can now add public REST/GeoJSON data sources **no-code** — a generic adapter kind created and configured entirely through adapter management, no repo change. None enabled in this [[deployment]] yet; all 23 current adapters remain bespoke per-source kinds (each `config.adapters.kind` is unique to its source).
- **Deploy mechanism:** manual, tag-based (`git checkout <tag>` → `uv sync` → `central-migrate` → restart). One-command `scripts/deploy.sh` ships in-repo; full procedure in the [[central-deploy-cutover]] runbook. No CI/CD; no automated DB backup (pre-flight `pg_dump` is the only migration safety net; migrations are forward-only).
- **`central-migrate` env gotcha:** manual calls need `cd /opt/central && set -a && . /etc/central/central.env && set +a && …` (the units supply this via WorkingDirectory + EnvironmentFile).
- **EONET:** now **global** as of v0.14.6 (2026-06-28) — `bypass_bbox_filter=True` + the adapter `region` key removed. NOTE: its `events`-table row count is a misleading health signal (dedup + partition expiry); use the CENTRAL_DISASTER stream as the EONET flow signal. `avalanche_org` zero-events is expected (off-season gate).
- **Supervisor CPU** ~17%→~1.4% and **cursors.db** 1.3 GB→496 MB after the v0.14.6 dedup/WAL fix + a VACUUM.
---
*Last updated: 2026-07-13 — v0.15.0 deployed & verified (GUI no-code generic REST/GeoJSON adapters, PRs #120–124); 3.42 M events / ~40 GB, all units active.*

112
vault/docs/software/conduit.md
View file

@ -1,112 +0,0 @@
---
title: Conduit — Raw-API Broker
type: reference
tags:
- mesh
aliases: []
related:
- [[conduit-operations]]
- [[recon-operations]]
- [[recon]]
- [[central]]
- [[fleet-patch-audit]]
updated: 2026-07-16
---
# Conduit — Raw-API Broker
## Overview
Conduit is the fleet's **single egress point** for every external API feed. It fetches an upstream once, stores the raw response **as-is** (byte-for-byte, no normalization), and fans it out to many internal consumers. It's a rate-limit shield, a cost shield, and a single egress point — "call an API once, use it many times."
- **URL (internal):** http://central.echo6.mesh:8010 (mesh; bound `0.0.0.0:8010`)
- **Host:** utility CT 104 (unprivileged Ubuntu LXC)
- **Repo:** github.com/zvx-echo6/conduit (**private**, Python/FastAPI)
- **Deploy path:** /opt/conduit (venv: /opt/conduit/.venv, env: /etc/conduit/conduit.env, system user: conduit)
- **Consumers:** [[navi]]'s traffic tiles (`tomtom_flow_tiles`) and all 13 of [[meshai]]'s hazard adapters (migrated 2026-07-15 — see [[conduit-operations]])
## Host
| Attribute | Value |
|-----------|-------|
| Container | utility CT 104 |
| Tailscale / mesh | 100.64.0.12 → central.echo6.mesh:8010 |
| Deploy dir | `/opt/conduit` (owned `conduit:conduit`) |
| Virtualenv | `/opt/conduit/.venv` (uv-managed, editable install) |
| Database | own `conduit` DB + role on the **shared** Postgres 16 (plain, no extensions) |
| Env file | `/etc/conduit/conduit.env` (`CONDUIT_DB_DSN`, `CONDUIT_MASTER_KEY_PATH`) |
| Master key | `/etc/conduit/master.key` |
| Systemd unit | `conduit.service` (enabled, **single uvicorn worker only** — never add `--workers`; single-flight + quota lock are per-process) |
| Bind | `0.0.0.0:8010` |
## Architecture — the "source" model
Everything Conduit brokers is a **source** — an addressable API identity, whether an internet feed or an internal app's output; the model makes no distinction. The `sources` table holds ~21 rows.
- **Ingest:** **pull** (Conduit fetches the upstream on cadence/on-demand) is live. **push** (an app POSTs a raw payload in) is planned, not yet exercised.
- **Store:** the latest raw payload per `(source, request_key)`, with a per-source freshness/TTL — a cache, not by itself a historical archive (see Retention below for the opt-in exception).
- **Serve:** a uniform `GET /up/{source}/{path}` returns the raw bytes byte-for-byte, no transformation.
- **Single-flight:** concurrent cache misses for the same `(source, request_key)` coalesce into exactly one upstream fetch; every waiting caller gets the same bytes back.
- **Topologies** — all fall out of the one model: `internet → conduit → app`, `app → conduit → app`, and chained (`internet → conduit → A → conduit → B`, where A registers its own output as a new source).
**Key property:** upstream calls scale with *unique resources × refresh rate*, not with the number of consumers — the 100th reader of the same resource costs zero additional upstream calls.
### Per-source columns
`name`, `url_template` (`{path}`/`{key}` substitution; an inbound query string appends rather than colliding), `api_key_alias`, `ttl_seconds`, `header_auth`, `enabled`, plus:
- **static `headers`** (jsonb) — for UA-sensitive feeds: NWS needs a contact User-Agent, Idaho Power WAFs non-browser UAs, avalanche.org sends a UA.
- **quota caps** — `max_calls_per_day` / `max_calls_per_minute` / `max_calls_per_month`.
- **retention** — `retain`, `retention_days`, `poll_interval_seconds`, `poll_path`.
## Capabilities
- **Single-flight coalescing** — concurrent misses for the same `(source, request_key)` → one upstream call.
- **Quota guard** (`quota.py`) — per-source day/minute/month caps enforced against a durable `upstream_calls` log, atomic per-source `asyncio.Lock` (single-process). Protects free-tier upstreams (e.g. TomTom's free plan).
- **Serve-stale** — on quota-block, upstream 429, or 5xx/transport failure, Conduit serves the last-known-good cached copy instead of failing the caller (`X-Conduit-Stale: 1`).
- **Faithful 4xx passthrough** (PR #16, 2026-07-15) — a genuine upstream 4xx *except* 429 (e.g. TomTom flow's `400 "Point too far from nearest existing segment"`) is returned to the caller with its real status + body, uncached, not wrapped as a 502. Only 429/5xx/transport failures fall back to serve-stale-or-502. This unblocked the last [[meshai]] adapter (traffic) to migrate.
- **Hot-reload of sources** — a source add/edit/delete made through the GUI calls `Broker.set_sources()` and takes effect with no restart. A direct out-of-band SQL change to `sources` still needs `systemctl restart conduit` — the GUI path is the live one.
- **Retention engine** (`poller.py` + `store/history.py`) — opt-in per source (`retain=true`). A background `Poller` fetches retained sources on their `poll_interval_seconds` and appends changed raw payloads to `payload_history` (append-on-change, sha256-dedup). Idle by construction when nothing is retained. Read back via `GET /history/{source}` and `/history/{source}/{id}/body` (unauthenticated, mesh-internal). Purpose: accumulate raw feeds for later forecast/trend models.
- **Management GUI** (`gui/`) — built in meshai's visual language. argon2 operator auth + two-tier CSRF + schema-reflection forms harvested from central. Pages: sources CRUD (incl. headers/quota/retention fields), API keys, a Cmd-K command palette. Admin operator provisioned. `/up` and `/history` stay unauthenticated (mesh-internal, tiles-trust model); the GUI is the authenticated surface.
- **Keystore** (`keystore.py`, `crypto.py`) — AES-256-GCM `api_keys` by alias, under Conduit's own master key. Holds `tomtom`, `roads511`, and `firms` keys (harvested from [[central]] / provisioned) — meshai no longer holds any of these itself.
## Modules (`src/conduit/`)
| Module | Role |
|--------|------|
| `crypto.py` | AES-256-GCM encrypted secret storage (master key from `CONDUIT_MASTER_KEY_PATH`) |
| `keystore.py` | `KeyStore` — async accessor for the `api_keys` table (encrypted API keys by alias) |
| `config.py` | App configuration loading |
| `migrate.py` | Forward-only SQL migration runner (console script `conduit-migrate`) |
| `fetch.py` | Single reusable async HTTP fetcher (aiohttp + tenacity retry/backoff) |
| `broker.py` | `Broker` — orchestrates store + keystore + single-flight + fetch + quota + serve-stale; the `get(source_name, request_key)` payoff path |
| `quota.py` | Per-source day/minute/month call-quota enforcement against `upstream_calls` |
| `poller.py` | Background retention poller for `retain=true` sources |
| `admin_cli.py` | `conduit-admin` command-line administration |
| `app.py` | FastAPI app wiring all routes |
| `store/payloads.py` | `PayloadStore` / `StoredPayload` — latest-by-`(source, request_key)` raw-byte cache with TTL-derived freshness |
| `store/singleflight.py` | `SingleFlight` — in-process coalescer for concurrent misses on the same key (per-process only) |
| `store/history.py` | `payload_history` accessor for the retention engine |
| `sources/__init__.py` | `SourceRegistry` (DB-backed `sources` table accessor) + `PullSource` (per-source `url_template`, headers, quota, retention fields) |
| `gui/{auth,csrf,deps,routes}` + templates/static | Management GUI: operator auth, CSRF, forms, command palette |
**Endpoints:** `GET /health`, `GET /up/{source}/{path}`, `GET /history/{source}`, `GET /history/{source}/{id}/body`, plus the GUI routes.
**Migrations:** `sql/migrations/001`–`007` (schema-only, no seeds), run by `conduit-migrate`, tracked in `schema_migrations`.
## Repo / deploy model
Pull-based deploy: authored/pushed from a cortex clone, CT 104 pulls via a **read-only deploy key** (`ct104-conduit-deploy`) — CT 104 cannot push. `conduit-migrate` runs forward-only SQL migrations (`sql/migrations/*.sql`), tracked in a `schema_migrations` table.
## Relationship to [[central]]
**Central is retired and dropped (2026-07-15) — Conduit replaced it.** Central's app [[services]] were stopped and disabled 2026-07-14 (zero live consumers remained), then on 2026-07-15 its database was archived to pi-nas (sha256-verified) and dropped (`DROP DATABASE central`, ~41 GB reclaimed); the shared Postgres instance was cleaned back to plain (TimescaleDB removed from `shared_preload_libraries`). Central is recoverable only from the pi-nas archive. Conduit's own `conduit` DB shares that same Postgres instance, which was never stopped.
Conduit was born by harvesting central's proven, decoupled pieces — the AES-256-GCM encrypted key store, the GUI auth/CSRF + schema-reflection form patterns, and the aiohttp+tenacity fetch idiom — while deliberately shedding central's NATS/JetStream, CloudEvents normalization, and enrichment pipeline.
## Current state (as of 2026-07-15)
- Single egress point for the whole fleet: [[navi]]'s traffic tiles plus all 13 [[meshai]] hazard adapters (NWS, SWPC ×4, ducting, WFIGS/fires ×2, FIRMS, avalanche, USGS streams, usgs_quake, tomtom_traffic, roads511, WZDx, satpass) — see [[conduit-operations]] for the migration record.
- `sources` table holds ~21 rows.
- Quota guard, serve-stale, faithful-4xx-passthrough, hot-reload, the retention engine, and the management GUI are all live (not planned).
- Keystore holds `tomtom`, `roads511`, `firms` — meshai no longer holds any of these keys directly.
- Central is gone; it is no longer a rollback target for anything.

103
vault/docs/software/dns.md
View file

@ -1,103 +0,0 @@
---
title: GoDaddy DNS Management
type: reference
tags:
- dns
aliases: []
related:
- [[caddy]]
- [[expose-service-home]]
- [[expose-service-contabo]]
- [[services]]
- [[headscale-onboard-node]]
updated: 2026-08-24
---
# GoDaddy DNS Management
## Script Location
`~/bin/godaddy-dns.py` on **cortex** (100.64.0.14). Restored 2026-08-24 — the
original copy was lost in a host rebuild and these docs pointed at a script
that no longer existed anywhere on the fleet for some time; that cost real
time mid-deploy. Python 3 standard library only, no pip installs.
## THE RECORD-SET TRAP
GoDaddy's API has two kinds of write endpoints for A records:
- `PUT /v1/domains/<domain>/records/A/<name>` — replaces only `<name>`. Safe.
- `PUT /v1/domains/<domain>/records/A` and `PUT /v1/domains/<domain>/records`
— **replace every record of that type on the whole domain.**
Both `k7zvx.com` and `echo6.co` front live production [[services]] (mail, auth,
forge, vpn, vault, matrix, element, and more). Calling one of the record-SET
endpoints to "add" a record would wipe every other A record on the zone.
`godaddy-dns.py` only ever uses the single-record endpoint to write, and does
not implement the record-set endpoints at all. `add-a` always reads the full
A-record list before and after and prints a diff, so you can see nothing
else changed.
## API Credentials
Stored in `/home/zvx/projects/.ref/credentials` as:
- `GODADDY_API_KEY`
- `GODADDY_API_SECRET`
## Key IPs for DNS Records
| Purpose | IP |
|---------|-----|
| External (home [[services]]) | `199.6.36.163` |
| edge1 (mail/autodiscover/autoconfig only — mail-only host, rebuilt [[2026-06-19]]) | `5.189.158.149` |
| edge2 (front door: auth/forge/vpn/vault/matrix/element/notes/proxmox) | `184.174.35.153` |
## Managed Domains
arclightvanguard.com, echo6.co, echo6.org, happylittlellc.com, idahomesh.com, k7zvx.com, lpmesh.com, maliceinwonderland.org, matthewwayne.com, smugglersden.co, underdogs.cc
## Subcommands
`godaddy-dns.py` currently supports A records only:
- `list <domain>` — list all A records for a domain
- `get-a <domain> <name>` — show the A record(s) for one name
- `add-a <domain> <name> <ip> [--ttl SECONDS] [--dry-run]` — add/update a
single A record via the safe single-record endpoint; prints a before/after
diff of the whole domain's A records
There is no CNAME/MX/delete/multi-domain support — add-a is add-or-update
(PUT to the single-record endpoint), and there is no destructive delete or
record-set command by design (see the trap above).
## Usage Examples
```bash
# List all A records for a domain
godaddy-dns.py list echo6.co
# Look up one record
godaddy-dns.py get-a echo6.co www
# Add/update an A record
godaddy-dns.py add-a echo6.co www 199.6.36.163
# Preview a change without writing anything
godaddy-dns.py add-a echo6.co www 199.6.36.163 --dry-run
```
## Common Patterns
### Point subdomain to home network
```bash
godaddy-dns.py add-a echo6.co newservice 199.6.36.163
```
### Point subdomain to edge2 (front door — auth, forge, vpn, vault, matrix, element, notes, proxmox)
```bash
godaddy-dns.py add-a echo6.co auth 184.174.35.153
```
### Point subdomain to edge1 (mail, autodiscover, autoconfig only)
```bash
godaddy-dns.py add-a echo6.co mail 5.189.158.149
```

61
vault/docs/software/geo-tools.md
View file

@ -1,61 +0,0 @@
---
title: Geo Processing Tools — Cortex
type: reference
tags:
- storage
aliases: []
related:
- [[navi]]
- [[ct-runbook]]
- [[cc-rules]]
- [[environment]]
- [[toc-cortex-pve9.2-update]]
updated: 2026-07-13
---
# Geo Processing Tools — Cortex
## Overview
DEM processing, vector tile generation, and PMTiles manipulation tools installed on cortex for compute-tier geospatial batch jobs. Cortex is compute-only — no persistent operational data lives here. Data flows: pi-nas (archive) → cortex (compute) → pi-nas + VM 1130 (outputs).
## Installed Tools
| Tool | Version | Package | Purpose |
|------|---------|---------|---------|
| gdal_contour | GDAL 3.8.4 | gdal-bin (apt) | Generate contour lines from DEM rasters |
| gdalinfo | GDAL 3.8.4 | gdal-bin (apt) | Raster metadata inspection |
| gdal_translate | GDAL 3.8.4 | gdal-bin (apt) | Raster format conversion |
| gdalbuildvrt | GDAL 3.8.4 | gdal-bin (apt) | Virtual raster mosaics |
| ogr2ogr | GDAL 3.8.4 | gdal-bin (apt) | Vector format conversion |
| tippecanoe | v2.49.0 | tippecanoe (apt) | GeoJSON → vector tiles (MBTiles/PMTiles) |
| pmtiles | v1.30.2 | go-pmtiles (binary) | PMTiles inspection, conversion, tile extraction |
Installed: 2026-04-22
## Ephemeral Scratch Convention
All temporary/staging data goes in `/home/zvx/scratch/`. This directory is cleaned after batch jobs complete. Never treat scratch as persistent storage.
Typical layout:
```
/home/zvx/scratch/
└── nav-processing/ # Current: staged DEM for contour generation
└── hillshade-na.pmtiles # 93 GB terrarium DEM (from pi-nas archive)
```
## Terrarium DEM Decode
The Mapzen Terrarium format encodes elevation in RGB PNG tiles:
```
elevation_meters = (R * 256 + G + B/256) - 32768
```
To use with GDAL: decode RGB → Float32 elevation array → write as GeoTIFF via VRT with `subClass="VRTRawRasterBand"`.
## Data Tier Architecture
| Tier | Host | Role | Example |
|------|------|------|---------|
| Archive | pi-nas (192.168.1.245) | Source of truth, recovery | `/export/data/nav/hillshade-na.pmtiles` |
| Compute | cortex (192.168.1.150) | Batch processing, scratch | `/home/zvx/scratch/nav-processing/` |
| Operational | VM 1130 (192.168.1.130) | Serving tiles via nginx | `/mnt/nav/tiles/` |

103
vault/docs/software/navi.md
View file

@ -1,103 +0,0 @@
---
title: navi — Offline Navigation Platform
type: reference
tags:
- recon
aliases: []
related:
- [[deployment]]
- [[central]]
- [[themes]]
- [[nominatim-v5-reimport]]
- [[cc-rules]]
updated: 2026-07-13
---
# navi — Offline Navigation Platform
## Overview
navi is an offline-capable navigation web app served from **recon-vm** (VM 1130, 192.168.1.130). It provides geocoding, routing, land classification, fleet admin, and DEM-backed elevation data — all from self-hosted geo backends. Frontend is a Vite SPA; backend is a suite of 8 Python microservices behind nginx.
- **URL:** https://navi.echo6.co (fronted by utility [[caddy]] + [[authentik]])
- **Host:** recon-vm (data node, VM 1130)
- **Repos:** `github.com/zvx-echo6/navi` (canonical), Forge mirror `matt/navi`
- **Layout:** monorepo — `backend/` (Python) + `frontend/` (Vite)
- **Deploy path:** `/home/zvx/projects/repos/navi-mono` (branch `main`, in sync with origin)
## Architecture
```
Internet → Caddy (utility CT 101) → Authentik → nginx :8440 (navi-mono frontend)
├─ /api/* → navi-* backends :8421-:8428
├─ /tiles/* → tile proxy
└─ /dem/* → dem-origin :8441 (range-cached)
└─ /mnt/nas/nav/planet-dem.pmtiles (658 GB)
```
## nginx Vhosts
| Port | Vhost | Role |
|------|-------|------|
| :8440 | navi.echo6.co | Public front door — SPA from `/mnt/nav/frontend`, API gateway (`/api/*` → backends), tile/DEM proxies; range-caches DEM requests from :8441 |
| :8441 | dem-origin (localhost only) | Serves 658 GB `planet-dem.pmtiles` from `/mnt/nas/nav/`; never exposed directly |
## Backend Services
All 8 are gunicorn processes, bound to `127.0.0.1`, working directory `navi-mono/backend`, env files in `/etc/navi-backend/*.env`.
| # | Service | Port | Status | Purpose |
|---|---------|------|--------|---------|
| 1 | navi-traffic | :8421 | **DISABLED** | Traffic — now proxied externally to `central.echo6.mesh:8000` |
| 2 | navi-config | :8422 | Active | [[deployment]] profile API |
| 3 | navi-contacts | :8423 | Active | Contacts + address book |
| 4 | navi-landclass | :8424 | Active | PAD-US land classification (Postgres `padus` DB) |
| 5 | navi-places | :8425 | Active | OSM place detail/enrichment (Postgres `overture` DB) |
| 6 | navi-geo | :8426 | Active | Geocode + reverse geocode (Photon + DEM + timezone) |
| 7 | navi-admin | :8427 | Active | Fleet admin-info aggregator (auth-gated) |
| 8 | navi-offroute | :8428 | Active | Off-network router + MVUM (Valhalla + PostGIS) |
Shared backend modules: `shared/auth.py` (Authentik header validation), `shared/admin_info.py`.
## Geo Backends (co-resident on recon-vm)
| Service | Version | Port | Runtime | Purpose | Consumed by |
|---------|---------|------|---------|---------|-------------|
| Valhalla | `ghcr.io/valhalla/valhalla-scripted:latest` | :8002 | Docker (`valhalla`) | Routing engine | navi-offroute |
| Nominatim | 5.3.2 (`mediagis/nominatim:5.3.2`) | :8010 | Docker (`nominatim-v5`) | Geocoder — western-11 US states | navi-places |
| Photon | 1.1.0 (jar) | :2322 | Direct (Java process) | Reverse geocoder (komoot dump, embedded ES :9201) | navi-geo |
| PostgreSQL/PostGIS | — | :5432 | System | DBs: `padus` (land classification), `overture` (OSM enrichment) | navi-landclass, navi-places |
**Nominatim coverage note:** western-11 states only (AZ, CA, CO, ID, MT, NV, NM, OR, UT, WA, WY). Queries outside those states return country-level fallback only. Nominatim was upgraded from v4.5 → v5.3.2 on 2026-06-23 (fresh re-import; same coverage as before). Photon is independent of Nominatim — it runs from a komoot dump and was not changed.
## Data
| Path | Contents |
|------|---------|
| `/mnt/nav/` | Addresses, timezones, OSM PBF, worldcover friction/barriers/trails, HPA, Photon data, frontend SPA build |
| `/mnt/nas/nav/planet-dem.pmtiles` | 658 GB global DEM tile archive |
| `/var/lib/navi-backend/` | Writable runtime state: `contacts.db`, `place_cache.db`, `wiki_index.db` |
NFS: pi-nas exports `/mnt/nav` and `/mnt/nas` to recon-vm.
## Deployment
The repo ships its own deploy templates under `backend/deploy/`:
- `deploy/systemd/` — all 8 unit files (match live)
- `deploy/nginx/` — nginx vhost snippets (match live)
- `deploy/env/` — env file templates (instances in `/etc/navi-backend/`)
- `deploy/caddy/` — Caddy upstream fragment
## Gotchas / Operational Notes
**Three checkouts exist** under `~/projects/repos/`:
| Checkout | Status |
|----------|--------|
| `navi-mono/` | **Live — this is what runs** |
| `navi/` | Stale — do not use |
| `navi-backend/` | Stale — **do not delete without auditing** (see below) |
**`navi-backend/` cannot be deleted yet:** several live env files (`navi-config`, `navi-contacts`, `navi-places`, `navi-geo`, `navi-offroute`) reference config, profile, and address-book files under `/home/zvx/projects/repos/navi-backend/config/...`. Document those paths before any cleanup.
**Stale README:** `backend/README.md` in the monorepo still refers to "extraction #1: navi-traffic" and predates the full monorepo consolidation. Treat `backend/deploy/` as the authoritative source.

Some files were not shown because too many files have changed in this diff Show more