matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: migrate Authentik (SSO keystone) to edge2 CT 105

Browse source 
- Authentik -> edge2 CT 105 (Postgres pg_dump/restore; SECRET_KEY carried verbatim; zero-downtime until ~2s cutover)
- Multi-block Caddy cutover: auth.echo6.co + notes.echo6.co outpost/forward_auth -> 100.64.0.36:9000
- runbook: add reboot tailscale-before-docker gotcha; clarify dnsmasq must NOT be repointed (points at Caddy host)
- source left stopped + intact on Contabo as cold rollback

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-18 05:49:07 +00:00
commit 44f0257376
140 changed files with 4013 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

245
vault/projects/advbbs-project.md Normal file
View file

@ -0,0 +1,245 @@
# advBBS — Claude Code Project Context
## Source of Truth
**GitHub repo**: https://github.com/zvx-echo6/advbbs (always pull latest before working)
## What is advBBS?
A federated, encryption-first BBS for Meshtastic mesh radio networks. Users interact by sending text DMs to a Meshtastic node running the BBS. Multi-hop mail routing between BBS nodes over LoRa radio. Runs on Raspberry Pi Zero 2 W (~100MB RAM).
Built with Python 3.11, SQLite (WAL mode), Meshtastic Python API. Docker-deployed. This is a "vibe-coded" project built with AI assistance — functional but may have rough edges.
---
## Package Structure
```
advbbs/
├── __init__.py
├── __main__.py # Entry point
├── config.py # TOML config loading, dataclasses
├── cli/
│ ├── config_rich.py # Rich-based interactive config TUI
├── commands/
│ ├── dispatcher.py # Command parser + all !commands
├── core/
│ ├── bbs.py # Main BBS class, event loop, session mgmt
│ ├── boards.py # Board service (CRUD, access control)
│ ├── crypto.py # Argon2id + ChaCha20-Poly1305 encryption
│ ├── mail.py # Mail service (inbox, send, read, delete)
│ ├── maintenance.py # Scheduled cleanup tasks
│ ├── rate_limiter.py # Per-node rate limiting
├── db/
│ ├── connection.py # SQLite connection, schema, migrations
│ ├── models.py # Dataclasses (User, Message, Board, etc.)
│ ├── messages.py # MessageRepository (CRUD)
│ ├── users.py # UserRepository, NodeRepository, UserNodeRepository
├── mesh/
│ ├── interface.py # Meshtastic radio interface, send/receive DMs
├── sync/
│ ├── manager.py # SyncManager — federation orchestrator (RAP, mail routing, retry logic)
│ ├── compat/
│ │ ├── advbbs_native.py # Wire protocol handler (HELLO, SYNC_ACK, bulletin format)
├── utils/
│ ├── formatting.py # Text formatting helpers
│ ├── pagination.py # Message pagination for mesh constraints
tests/
├── test_boards.py
├── test_crypto.py
├── test_mail.py
├── test_maintenance.py
├── test_pagination.py
├── test_sync.py
docs/
├── commands.md
├── mail.md
├── boards.md
├── sync.md # Federation + RAP protocol docs
├── configuration.md
├── deployment.md
├── security.md
├── rap-testing.md # Multi-hop RAP test procedures
├── migration.md # fq51bbs → advBBS migration
├── quickstart.md
├── USER-QUICKSTART.md
├── ELI5.md
```
### Non-obvious file placements
- `crypto.py` and `rate_limiter.py` → `core/` (not root)
- `interface.py` (Meshtastic mesh interface) → `mesh/` (not root)
- `advbbs_native.py` (wire protocol) → `sync/compat/` (not root)
- `dispatcher.py` (all user commands) → `commands/` (not root)
- `config_rich.py` (TUI config) → `cli/` (not root)
- `formatting.py`, `pagination.py` → `utils/`
---
## Database
SQLite with WAL mode, autocommit, `check_same_thread=False`. Row factory enabled for dict-like access.
### Schema (3 migrations)
**Migration 001 — Core tables:**
- `users` — id, username, password_hash, salt, encryption_key, recovery_key_enc, is_admin, is_banned, ban fields
- `nodes` — Meshtastic nodes (node_id like `!abcdef12`, short_name, long_name, SNR/RSSI)
- `user_nodes` — Multi-node identity (user_id ↔ node_id, is_primary)
- `messages` — uuid (UNIQUE), msg_type (`mail`/`bulletin`/`system`), board_id, sender/recipient user/node IDs, subject_enc, body_enc (BLOB NOT NULL), timestamps, origin_bbs, forwarded_to, hop_count, delivery_attempts
- `boards` — name, description, board_type, board_key_enc
- `board_access` — Per-user restricted board access
- `board_states` — Per-user read position
- `bbs_peers` — node_id, bbs_name, protocol, sync_enabled, trust_level
- `sync_log` — message_uuid, peer_id, direction, status, attempts
**Migration 002 — Settings/maintenance:**
- Added columns: `messages.deleted_at_us`, `bbs_peers.callsign/name/capabilities/last_seen_us`
- New tables: `bbs_settings` (KV store), `board_read_positions`
**Migration 003 — RAP:**
- Added peer columns: `health_status`, `failed_heartbeats`, `last_heartbeat_us`, `last_pong_us`, `quality_score`
- New tables: `rap_routes` (dest_bbs, via_peer_id, hop_count, quality_score, expires_at_us), `rap_pending_mail` (queued mail for offline routes)
### Timestamps
All timestamps are microseconds since epoch (`int(time.time() * 1_000_000)`), stored as INTEGER. Column suffix `_us`.
---
## Wire Protocol
All inter-BBS messages sent as Meshtastic DMs. Format: `advBBS|1|<MSG_TYPE>|<payload>`
### RAP Messages (Route Announcement Protocol)
| Message | Purpose | Payload |
|---------|---------|---------|
| `RAP_PING` | Heartbeat | `timestamp_us` |
| `RAP_PONG` | Response + routes | `timestamp_us\|route_table` |
| `RAP_ROUTES` | Route table broadcast | `route_table` |
Route table format: `BBS1:hop:quality;BBS2:hop:quality` (e.g., `MV51:0:1.0;J51B:1:1.00`)
### Mail Protocol Messages
| Message | Format | Purpose |
|---------|--------|---------|
| `MAILREQ` | `MAILREQ\|uuid\|from_user\|from_bbs\|to_user\|to_bbs\|hop\|num_parts\|route` | Request delivery |
| `MAILACK` | `MAILACK\|uuid\|OK` | Accept, ready for chunks |
| `MAILNAK` | `MAILNAK\|uuid\|reason` | Reject (NOUSER, NOROUTE, MAXHOPS, LOOP) |
| `MAILDAT` | `MAILDAT\|uuid\|part/total\|data` | Message chunk (max 150 chars × 3) |
| `MAILDLV` | `MAILDLV\|uuid\|OK\|user@BBS` | Delivery confirmation |
### Mail Flow
```
Sender BBS Destination BBS
│ │
│── MAILREQ ────────────▶│ (pre-flight: user exists?)
│◀── MAILACK ────────────│ (ready for chunks)
│── MAILDAT 1/1 ────────▶│ (body chunk)
│ │ (store in DB)
│◀── MAILDLV ────────────│ (confirmed)
```
Multi-hop: intermediate BBS relays MAILREQ/MAILDAT, tracked via `_relay_mail` dict. Max 5 hops. Route list in MAILREQ prevents loops.
---
## Key Architecture Patterns
### Threading Model
- **Main thread**: asyncio event loop (`bbs._loop`) — runs tick(), scheduled tasks
- **Meshtastic callback thread**: `on_receive` fires from Meshtastic library thread
- **Bridge**: `_schedule_async(coro)` uses `asyncio.run_coroutine_threadsafe()` to schedule work from callback thread onto main loop
### Session Management
Sessions keyed by Meshtastic node_id. Login requires both password AND a registered node (node-based 2FA). Sessions expire after inactivity.
### Encryption
- **At rest**: All message bodies encrypted with user-derived keys (Argon2id KDF → ChaCha20-Poly1305)
- **Transport**: Meshtastic PSK encryption (AES-256) recommended
- **Remote mail**: Stored plaintext on receiving BBS (encrypted at read time by recipient's key)
### Message Constraints
- LoRa max ~150 bytes usable per packet
- Remote mail body max 450 chars (3 chunks × 150)
- Pagination helper chunks long responses for mesh delivery
- TX queue collision avoidance: 2.5s delay between protocol DMs
### Peer Security
Federation traffic whitelisted by peer — only configured peers accepted. Non-peer protocol messages rejected.
---
## Configuration
TOML config file. Key sections: `[bbs]`, `[database]`, `[meshtastic]`, `[crypto]`, `[features]`, `[operating_mode]`, `[sync]`, `[rate_limits]`, `[web_reader]`, `[cli_config]`, `[logging]`.
Operating modes: `full`, `mail_only`, `boards_only`, `repeater`.
Peers configured as `[[sync.peers]]` arrays with `node_id`, `name`, `protocol`, `enabled`.
RAP timing defaults are conservative for mesh (12h heartbeat, 36h route expiry, 24h route share).
---
## Current Live Federation Topology
```
MV51 (Old Man Malice / Matt) ◀──▶ J51B (JeepnJonny)
node: !00ff0001 node: !60a43e58
```
Both running Docker containers. Meshtastic simulator (meshtasticd) for testing.
---
## Known Bug: Federation Mail Delivery Failure
### Symptom
```
[ERROR] advbbs.sync.manager: DELIVER b8e78195: Failed to store in database
```
### Root Cause
`create_incoming_remote_mail()` in `db/messages.py` returns `None` for both duplicates (logged at DEBUG — invisible) and real DB errors. Mesh radio retransmissions deliver the same MAILDAT twice, triggering duplicate detection, but the caller can't distinguish this from a real failure.
Additionally, when a duplicate IS detected, no MAILDLV confirmation is sent back, causing the sender to retry indefinitely.
### Fix Required (3 changes)
1. **`db/messages.py`** — `create_incoming_remote_mail`: Return `"duplicate"` sentinel instead of `None` for duplicate UUID. Promote log from DEBUG → INFO. Add traceback to exception path.
2. **`sync/manager.py`** — `_deliver_remote_mail`: Handle `"duplicate"` return: log at INFO, still send MAILDLV confirmation, clean up state. Improve error message for real failures.
3. **`sync/manager.py`** — `handle_maildat` (the `_handle_maildat` section around line 1068): Add `delivering` flag guard to prevent double scheduling from mesh retransmissions.
---
## Development Notes
- Tests: `pytest tests/` — unit tests for crypto, mail, boards, maintenance, pagination, sync
- Docker build: `docker compose build` (can take 10-15 min on Pi, may need swap)
- Config TUI: `advbbs-config` or `python -m advbbs.cli.config_rich`
- Logs: `docker compose logs -f`
- DB inspection: `sqlite3 /data/advbbs.db ".tables"` inside container
---
## Style / Conventions
- Logging: `logger = logging.getLogger(__name__)` per module
- DB access: Repository pattern (MessageRepository, UserRepository, etc.) wrapping Database methods
- All DB timestamps: microseconds (`_us` suffix)
- UUIDs: `str(uuid.uuid4())` for message dedup
- Meshtastic node IDs: hex string with `!` prefix (e.g., `!00ff0001`)
- Commands: `!` prefix, case-insensitive, short aliases
- Config: TOML with dataclass parsing in `config.py`

310
vault/projects/argus.md Normal file
View file

@ -0,0 +1,310 @@
# ARGUS - OSINT Intelligence Platform
**Status:** Container provisioned, baseline installed, awaiting application deployment
**Last Updated:** 2026-06-14
---
## Overview
ARGUS (Automated Reconnaissance & Gathering for Unified Situational-awareness) is an OSINT intelligence gathering platform combining SearXNG with local LLM analysis for automated threat intelligence collection and processing.
**Architecture:**
- Search backend: SearXNG (self-hosted)
- Analysis: Local LLM models (no cloud APIs)
- Scopes: Local, regional, national, global threat levels
- Privacy-first: No PII collection, focus on events/trends/policies
---
## Container Specifications
| Property | Value |
|----------|-------|
| **CTID** | 103 |
| **Hostname** | argus |
| **Host** | utility (192.168.1.241 / 100.64.0.5) |
| **Local IP** | 192.168.1.103 (static) |
| **Tailscale IP** | 100.64.0.25 |
| **Gateway** | 192.168.1.1 |
| **Container Type** | Privileged (unprivileged=0) |
| **Resources** | 4 cores, 8GB RAM, 30GB disk |
| **Storage** | local-lvm:vm-103-disk-0 |
| **Network** | vmbr0, eth0 |
| **Features** | nesting=1 (Docker support) |
| **Autostart** | Yes (onboot=1) |
| **OS** | Ubuntu 24.04 LTS |
**Why privileged:** Required for /dev/net/tun access (Tailscale). Attempted unprivileged initially but tailscaled failed with "CreateTUN failed; /dev/net/tun does not exist".
---
## Installed Software (Baseline)
- **Docker:** 29.5.3 + docker-compose plugin
- **Tailscale:** 1.98.4 (registered with Headscale at vpn.echo6.co)
- **User:** zvx (uid=1000, groups: sudo, docker)
- **Common tools:** curl, wget, vim, htop, git, jq, net-tools, dnsutils, sshpass
- **SSH:** OpenSSH server (password auth enabled)
---
## Tailscale Configuration
**Headscale server:** https://vpn.echo6.co
**User:** echo6 (user ID 1)
**Tailscale IP:** 100.64.0.25
**Registration:** `tailscale up --login-server=https://vpn.echo6.co --authkey=<key> --ssh --accept-routes`
**DNS Bootstrap Fix:**
Systemd drop-in at `/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf` ensures fallback DNS (1.1.1.1, 8.8.8.8) exists before tailscaled starts, preventing chicken-and-egg DNS resolution failures on reboot.
```bash
[Service]
# Ensure fallback DNS exists before tailscaled starts
# Prevents chicken-and-egg DNS resolution failures on reboot
ExecStartPre=/bin/sh -c "echo nameserver 1.1.1.1 > /etc/resolv.conf; echo nameserver 8.8.8.8 >> /etc/resolv.conf"
```
---
## Network Configuration
**Static IP:** Configured via Proxmox (`pct set 103 -net0 name=eth0,bridge=vmbr0,ip=192.168.1.103/24,gw=192.168.1.1`)
**Container config** (`/etc/pve/lxc/103.conf`):
```
arch: amd64
cores: 4
features: nesting=1
hostname: argus
memory: 8192
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:EA:8B:21,ip=192.168.1.103/24,gw=192.168.1.1,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-103-disk-0,size=30G
swap: 512
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
```
**TUN device:** Added manually via `lxc.cgroup2.devices.allow` and `lxc.mount.entry` to support Tailscale in privileged container.
---
## Access Methods
### SSH Access
```bash
# Local network (static IP)
ssh zvx@192.168.1.103
# Tailscale VPN
ssh zvx@100.64.0.25
ssh zvx@argus
# With password (for sshpass workflows)
sshpass -p '7redditGold' ssh zvx@192.168.1.103
```
**Credentials:**
- User: `zvx`
- Password: `7redditGold`
- Sudo: Enabled (no password prompt)
### From Proxmox Host
```bash
# Execute commands in container
pct exec 103 -- <command>
# Enter container shell
pct enter 103
# Container management
pct start 103
pct stop 103
pct reboot 103
pct status 103
```
---
## ARGUS Application Architecture (Planned)
### Geographic Scope Hierarchy
| Scope | Description | Update Frequency |
|-------|-------------|------------------|
| LOCAL | Idaho, immediate region | High |
| REGIONAL | Pacific Northwest, neighboring states | Medium |
| NATIONAL | US-wide threats, policy changes | Medium |
| GLOBAL | International, geopolitical | Low |
### Data Storage
- **Raw search results:** `data/raw/{scope}/{date}/`
- **Processed intel:** `data/processed/{scope}/`
- **Alerts:** `data/alerts/`
- **Timestamps:** All in UTC
### Privacy Rules
- No PII collection on individuals
- Focus on events, trends, policies — not people
- Scrub any inadvertent PII before storage
- Logs must not contain search queries with personal info
### LLM Analysis
- **Model hosting:** Local only (no cloud APIs)
- **Functions:** Summarization, threat classification, entity extraction, sentiment/threat scoring
- **Entities:** Locations, organizations (not individuals)
---
## Provisioning History
**2026-06-14 03:00 UTC** - Initial provisioning
1. **First attempt (unprivileged):** Failed - tailscaled couldn't access /dev/net/tun
2. **Second attempt (privileged):** Success
- Created CT 103 with `--unprivileged 0`
- Installed baseline (apt update/upgrade, common tools, Docker, Tailscale)
- DNS fix required post-restart (resolv.conf reset to 100.100.100.100)
- Added DNS bootstrap systemd drop-in to prevent future DNS failures
- Configured static IP 192.168.1.103 (originally got .142 via DHCP)
- Added TUN device support via lxc.cgroup2 and lxc.mount.entry
**Headscale registration:**
- Created preauth key via `docker exec headscale headscale preauthkeys create --user 1 --expiration 24h --reusable`
- Registered successfully after DNS fix
- Assigned Tailscale IP: 100.64.0.25
---
## Verification Checklist
Run inside container to verify baseline:
```bash
pct exec 103 -- bash -c '
echo "=== CT Provisioning Check ==="
echo ""
echo "Hostname: $(hostname)"
echo "User zvx: $(id zvx 2>/dev/null && echo OK || echo MISSING)"
echo "sudo: $(sudo -l -U zvx 2>/dev/null | grep -q ALL && echo OK || echo MISSING)"
echo "sshpass: $(which sshpass >/dev/null 2>&1 && echo OK || echo MISSING)"
echo "SSH: $(systemctl is-active ssh)"
echo "Docker: $(docker --version 2>/dev/null || echo MISSING)"
echo "Tailscale: $(tailscale status --self 2>/dev/null | head -1 || echo NOT CONNECTED)"
echo "Tailscale IP: $(tailscale ip -4 2>/dev/null || echo N/A)"
echo "Local IP: $(hostname -I | awk \"{print \$1}\")"
'
```
**Expected output:**
```
=== CT Provisioning Check ===
Hostname: argus
User zvx: uid=1000(zvx) gid=1000(zvx) groups=1000(zvx),27(sudo),990(docker) OK
sudo: OK
sshpass: OK
SSH: active
Docker: Docker version 29.5.3, build d1c06ef
Tailscale: 100.64.0.25 argus echo6 linux -
Tailscale IP: 100.64.0.25
Local IP: 192.168.1.103
```
---
## Known Issues & Resolutions
### Issue: DNS resolution fails after container restart
**Symptom:** `resolv.conf` gets reset to invalid nameserver (100.100.100.100), breaking apt and network connectivity.
**Root cause:** LXC containers sometimes reset DNS on boot before networking is fully initialized.
**Resolution:** Installed systemd drop-in (`/etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf`) that sets fallback DNS before tailscaled starts. Prevents chicken-and-egg failure where Tailscale can't resolve vpn.echo6.co because DNS is broken.
### Issue: Tailscaled fails with "/dev/net/tun does not exist"
**Symptom:** Tailscaled crashes on startup with `CreateTUN("tailscale0") failed; /dev/net/tun does not exist`.
**Root cause:** Unprivileged LXC containers don't have access to /dev/net/tun by default.
**Resolution:** Recreated container as privileged (`--unprivileged 0`) and added TUN device to container config:
```
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
```
---
## Next Steps (Application Deployment)
1. **SearXNG deployment:** Docker container for self-hosted search aggregation
2. **LLM integration:** Local model for analysis (Ollama on cortex or self-hosted)
3. **Database:** SQLite for processed intel, possibly Qdrant for vector search (cortex:6333 available)
4. **Scheduler:** Cron or systemd timers for automated collection
5. **Web dashboard:** Flask/FastAPI for threat intel visualization
6. **Alerting:** Integration with Matrix/email for high-priority threats
---
## Operational Notes
- **Backup strategy:** TBD (Docker volumes + application data)
- **Log rotation:** TBD
- **Monitoring:** TBD (consider adding to WATCHTOWER ops dashboard)
- **Updates:** Standard Ubuntu + Docker update procedures
- **Resource scaling:** Can adjust cores/RAM via `pct set 103 -cores X -memory Y` (requires container restart)
---
## Related Documentation
- **CT provisioning:** `/home/zvx/projects/.ref/runbooks/ct-runbook.md`
- **ARGUS rules:** `~/.claude/rules/argus.md`
- **Environment:** `/home/zvx/projects/.ref/docs/hardware/environment.md`
- **Services:** `/home/zvx/projects/.ref/docs/services/services.md`
- **Headscale:** `/home/zvx/projects/.ref/docs/software/caddy.md` (dnsmasq split DNS)
---
## Quick Command Reference
```bash
# Container management (from Proxmox host)
pct start 103
pct stop 103
pct reboot 103
pct enter 103
# SSH access
ssh zvx@192.168.1.103
ssh zvx@argus # via Tailscale DNS
# Check Tailscale status
pct exec 103 -- tailscale status
pct exec 103 -- tailscale ip -4
# Docker commands (as zvx user)
ssh zvx@argus "docker ps"
ssh zvx@argus "docker compose up -d"
# View container config
cat /etc/pve/lxc/103.conf
# Check resource usage
pct status 103 --verbose
```
---
**Provisioned by:** Claude Code
**Container ready for:** ARGUS application deployment

242
vault/projects/deploy-livesync.md Normal file
View file

@ -0,0 +1,242 @@
# Deploying CouchDB with JWT auth for Obsidian LiveSync via Authentik
**LiveSync has native client-side JWT support that eliminates the need for a browser-based OIDC flow.** The plugin generates and signs JWTs internally using a stored private key, sending `Authorization: Bearer` headers directly to CouchDB. This fundamentally changes the architecture: instead of proxying OIDC tokens, you provision per-user key pairs, configure CouchDB with the public keys, and distribute setup URIs containing the private keys. Authentik serves as the identity backbone for a provisioning service — not as a runtime token issuer. No one has publicly documented a complete LiveSync + SSO deployment, making this guide a synthesis of the Kishieel Keycloak series, CouchDB JWT internals, Authentik's claim customization, and the LiveSync plugin's JWT implementation.
---
## CouchDB's JWT engine and the exact local.ini configuration
CouchDB 3.3+ includes a built-in JWT authentication handler requiring zero plugins. From Kishieel's Keycloak series and the official docs, here is the complete `local.ini`:
```ini
[couchdb]
single_node = true
[chttpd]
bind_address = 0.0.0.0
port = 5984
require_valid_user_except_for_up = true
authentication_handlers = {chttpd_auth, jwt_authentication_handler}, {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
[jwt_auth]
required_claims = exp,iat
roles_claim_path = _couchdb\.roles
[jwt_keys]
; EC key for LiveSync plugin (ES512 with P-521 curve)
ec:livesync-user1 = -----BEGIN PUBLIC KEY-----\nMHYwEAYHK...AzztRs\n-----END PUBLIC KEY-----\n
; RSA key from Authentik JWKS (for service/API access)
rsa:authentik-kid-here = -----BEGIN PUBLIC KEY-----\nMIIBIjAN...IDAQAB\n-----END PUBLIC KEY-----\n
[chttpd_auth]
secret = generate-a-long-random-secret-here
[cors]
origins = app://obsidian.md,capacitor://localhost,http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE
max_age = 3600
[admins]
admin = your-admin-password
```
**Critical details on `roles_claim_path`**: The backslash in `_couchdb\.roles` is mandatory. Without it, CouchDB interprets the dot as JSON nesting and looks for `{"_couchdb": {"roles": [...]}}` instead of the flat key `{"_couchdb.roles": [...]}`. This was a long-standing bug (issue #3176, #3758) that caused JWT roles to silently fail until the `roles_claim_path` syntax was added in CouchDB 3.3. The deprecated `roles_claim_name` setting did not have this problem but is ignored when `roles_claim_path` is set.
**Key format in `[jwt_keys]`** follows the pattern `{algorithm}:{kid} = {value}`. The algorithm prefix (`hmac:`, `rsa:`, `ec:`) is mandatory and prevents algorithm-confusion attacks. CouchDB reads the JWT header's `alg` claim to determine the prefix and the `kid` claim to select the specific key. If no `kid` is present in the JWT, CouchDB falls back to `{algorithm}:_default`. For asymmetric keys, the value is the PEM-encoded public key with literal `\n` replacing newlines. For HMAC, it's a base64-encoded secret. Since CouchDB 3.3, `=` characters in key names (common in base64 key IDs) are supported when the name-value separator uses spaces: `rsa:kid-with-base64= = -----BEGIN...`.
**On `required_claims`**: By default this is empty, meaning **CouchDB does not validate token expiration**. Always set `required_claims = exp` at minimum. The `sub` claim is always mandatory regardless of this setting and maps directly to the CouchDB username.
**Key rotation via the HTTP config API** takes effect immediately without restart:
```bash
curl -u admin:password -X PUT \
"http://localhost:5984/_node/_local/_config/jwt_keys/ec:new-kid" \
-H "Content-Type: text/plain" \
-d '"-----BEGIN PUBLIC KEY-----\nMHYw...\n-----END PUBLIC KEY-----\n"'
```
However, **CouchDB bug #5091** reports that `PUT /_node/{node}/_config/jwt_keys/{key}` returns HTTP 400 for valid PEM keys in some CouchDB versions. The workaround is writing keys to a `.ini` file in `/opt/couchdb/etc/local.d/` and restarting via `POST /_node/_local/_restart`. Changes to `local.ini` directly always require a restart; API-based changes do not.
---
## Kishieel's Keycloak pattern adapted for Authentik
Kishieel's two-part series provides the only complete, proven CouchDB + OIDC reference implementation. The architecture uses OpenResty (Nginx + Lua) as a proxy that performs OIDC authentication for browser clients and injects a Bearer token before forwarding to CouchDB. Here's how each component maps to the Authentik equivalent:
**Keycloak groups → Authentik groups with attributes**: Kishieel created Keycloak groups `/couchdb/admins` and `/couchdb/users` with a group attribute `_couchdb.roles` set to `["_admin"]` and `["_user"]` respectively. In Authentik, you'd create groups named `couchdb-admins` and `couchdb-users` with custom attributes `{"couchdb_role": "_admin"}` and `{"couchdb_role": "_user"}` respectively.
**Keycloak protocol mapper → Authentik scope mapping**: Kishieel used an `oidc-usermodel-attribute-mapper` with `claim.name: "_couchdb\\.roles"` (double-escaped backslash to produce a literal dot in the JWT claim). The mapper was `multivalued: true` and `aggregate.attrs: true` to collect roles from all groups. In Authentik, create a **Scope Mapping** under Customization → Property Mappings:
- **Name**: `CouchDB Roles`
- **Scope name**: `couchdb`
- **Expression**:
```python
return {
"_couchdb.roles": list(set(
str(g.attributes.get("couchdb_role"))
for g in request.user.ak_groups.all()
if "couchdb_role" in g.attributes
))
}
```
This iterates all user groups, extracts the `couchdb_role` attribute where it exists, deduplicates, and returns it as the `_couchdb.roles` claim. Values returned by scope mappings are added as custom claims to **both access tokens and ID tokens**.
**Keycloak client scope → Authentik OAuth2 provider scope**: Kishieel created a `couchdb` client scope containing the mapper, then assigned it as an optional scope on both the `couchdb-proxy` (confidential) and `couchdb-cli` (public) clients. In Authentik, assign the scope mapping to your OAuth2 provider's **Selected Scopes** list alongside `openid`, `profile`, and `email`. Check **"Include claims in id_token"** in the provider settings.
**Kishieel's Lua proxy script** (`access.lua`) is the key innovation. It uses `lua-resty-openidc` to perform the full OIDC authorization code flow for browser requests, then sets `Authorization: Bearer <access_token>` before proxying to CouchDB. Critically, the Part 2 update added an early return: if the request already has an `Authorization` header (from a CLI or API client), the Lua script skips the OIDC flow entirely. This dual-path design — browser SSO via proxy, direct Bearer token for programmatic access — is the pattern to replicate.
---
## LiveSync's native JWT: how the plugin signs its own tokens
The Obsidian LiveSync plugin has **built-in JWT generation** that changes the deployment model fundamentally. Instead of obtaining tokens from an IdP at runtime, the plugin stores a private key and signs short-lived JWTs client-side. The relevant plugin settings are:
| Setting | Type | Default | Purpose |
|---------|------|---------|---------|
| `useJWT` | boolean | `false` | Enable JWT authentication |
| `jwtAlgorithm` | string | `""` | JWT algorithm (e.g., `ES512`, `RS256`) |
| `jwtKey` | string | `""` | **Private key** in PEM format |
| `jwtKid` | string | `""` | Key ID matching CouchDB's `[jwt_keys]` entry |
| `jwtSub` | string | `""` | Subject claim → CouchDB username |
| `jwtExpDuration` | number | `5` | Token lifetime in minutes |
**Token lifecycle**: Tokens are cached and reused until **10% of the expiration duration remains or 10 seconds**, whichever is longer (capped at 1 minute maximum). With the default 5-minute expiration, tokens refresh at the 30-second mark. The plugin generates a new token by signing with the stored private key — no network call to an IdP.
**Key generation for ES512** (P-521 elliptic curve):
```bash
# Generate private key
openssl ecparam -genkey -name secp521r1 -noout -out private_key.pem
# Extract public key
openssl ec -in private_key.pem -pubout -out public_key.pem
```
The private key goes into the plugin's `jwtKey` setting. The public key (with newlines escaped as `\n`) goes into CouchDB's `[jwt_keys]` as `ec:<kid> = <pem>`.
**The setup URI gap**: The `generate_setupuri.ts` script (at `utils/flyio/generate_setupuri.ts`) only accepts basic auth parameters (`hostname`, `database`, `username`, `password`, `passphrase`). It does not support JWT settings. GitHub issue #729 documents this limitation. To generate a setup URI with JWT config, you must construct the full settings object (including `useJWT`, `jwtAlgorithm`, `jwtKey`, `jwtKid`, `jwtSub`, `jwtExpDuration`), encrypt it with a passphrase, and format it as `obsidian://setuplivesync?settings=[encrypted_data]`. The encryption mechanism uses passphrase-based AES encryption.
---
## Authentik configuration for service tokens and JWKS
**OAuth2 provider setup** at `auth.echo6.co`: Create an OAuth2/OIDC provider for the CouchDB service. Set the signing key to an RSA certificate (Authentik defaults to its self-signed certificate using RS256). The JWKS endpoint is at `https://auth.echo6.co/application/o/<app-slug>/jwks/` and the OpenID configuration at `https://auth.echo6.co/application/o/<app-slug>/.well-known/openid-configuration`.
**Token lifetimes** are configurable per-provider. The **access token defaults to 5 minutes** (format: `minutes=5`), refresh token to 30 days. For a provisioning service that generates long-lived tokens, extend to `hours=1` or more. The syntax accepts `hours=1,minutes=30,seconds=0`.
**Getting tokens programmatically** via `client_credentials`:
```bash
# Method 1: Client ID + Secret (auto-creates service account)
curl -X POST 'https://auth.echo6.co/application/o/token/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials' \
-d 'client_id=<client_id>' \
-d 'client_secret=<client_secret>' \
-d 'scope=openid profile couchdb'
# Method 2: Service account credentials
curl -X POST 'https://auth.echo6.co/application/o/token/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials' \
-d 'client_id=<client_id>' \
-d 'username=my-service-account' \
-d 'password=my-app-password-token' \
-d 'scope=openid profile couchdb'
```
Authentik supports `client_credentials`, `password` (ROPC — treated identically to client_credentials), `authorization_code`, `refresh_token`, `implicit`, and `urn:ietf:params:oauth:grant-type:device_code`. All endpoint URLs: token at `/application/o/token/`, authorize at `/application/o/authorize/`, device at `/application/o/device/`.
**JWKS-to-PEM conversion** for injecting Authentik's signing key into CouchDB is handled by the `couchdb-idp-updater` tool (GitHub: beyonddemise/couchdb-idp-updater). This NodeJS tool by Stephan Wissel periodically fetches the JWKS from an IdP's `.well-known/openid-configuration`, converts JWK keys to PEM format, and updates CouchDB's `[jwt_keys]` config. Due to bug #5091, it may need to write directly to an INI file rather than using the REST API. The manual conversion script (`jwks2couch.mjs`) is available as a GitHub gist. The process is: fetch JWKS → for each key, convert JWK to PEM using `jwk-to-pem` npm package → collapse newlines to `\n` → write to `[jwt_keys]` as `rsa:<kid> = <collapsed-pem>`.
---
## Per-database security without the _users database
**JWT users do not need to exist in CouchDB's `_users` database.** The user context is constructed entirely from the JWT: `sub` becomes the username, and the roles claim provides roles. CouchDB never queries `_users` during JWT authentication.
**The `_security` document** controls per-database access:
```json
{
"admins": {
"names": [],
"roles": ["_admin"]
},
"members": {
"names": ["specific-jwt-sub-value"],
"roles": []
}
}
```
Set this via:
```bash
curl -u admin:password -X PUT \
"http://localhost:5984/userdb-alice/_security" \
-H "Content-Type: application/json" \
-d '{"admins":{"names":[],"roles":["_admin"]},"members":{"names":["alice"],"roles":[]}}'
```
CouchDB matches the JWT `sub` against `members.names` and `admins.names`, and the JWT roles against `members.roles` and `admins.roles`. If `_security` has any members defined, only matching users can access the database. The `members` role grants read access to all documents and write access to non-design documents. The `admins` role additionally allows writing design documents and modifying `_security`.
**Do not use `couch_peruser` with JWT.** The Plexify article documents that CouchDB's built-in `couch_peruser` feature only auto-creates databases for admin users under JWT auth — requiring you to grant `_admin` to everyone, which is dangerous. Instead, create databases and set `_security` programmatically from a provisioning service using admin credentials.
---
## Proven deployment patterns and what breaks
**No one has publicly deployed LiveSync with full SSO end-to-end.** GitHub discussion #484 captures the core problem: *"For the auth, I use Authentik for my self hosted programs, however I am unsure if it will work with the obsidian extension since there is no user interface to login."* The plugin runs inside Obsidian's Electron shell — it cannot redirect to a browser for an OIDC login flow.
**Token expiration causes PouchDB replication failures.** When a JWT expires, CouchDB returns 401 with a `WWW-Authenticate: Basic` header, triggering an unwanted browser auth popup in Electron. The Plexify article documented this and recommended suppressing the header via reverse proxy or CouchDB config.
**CORS is the most common failure mode.** Issue #628 documents that LiveSync does not send the `Origin` header on non-preflight requests, causing CouchDB's CORS handler to omit `Access-Control-Allow-Origin` from responses. The fix is configuring CORS in `local.ini` (shown above) rather than relying on the reverse proxy alone. Required origins: `app://obsidian.md`, `capacitor://localhost`, `http://localhost`.
**The Caddy reverse proxy config** for `notes.echo6.co`:
```
notes.echo6.co {
reverse_proxy couchdb:5984
header {
Access-Control-Allow-Origin "app://obsidian.md"
Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
Access-Control-Allow-Headers "Content-Type, Authorization"
Access-Control-Allow-Credentials "true"
Access-Control-Max-Age "86400"
}
@options method OPTIONS
handle @options {
respond 204
}
}
```
---
## The recommended architecture for notes.echo6.co
Given the constraints — LiveSync can't do OIDC flows, but it can sign JWTs client-side — the architecture has three components:
**1. CouchDB container** at `notes.echo6.co` behind Caddy, configured with JWT auth handler, CORS, and per-user databases with `_security` documents.
**2. A provisioning service** (a small web app hosted on `forge.echo6.co` or as a Docker container) that:
- Is protected by Authentik forward auth (browser-based OIDC login)
- On first login, generates an EC key pair (ES512/P-521) for the user
- Creates a per-user CouchDB database (`userdb-<username>`)
- Sets the `_security` document to restrict access to that user's `sub`
- Injects the public key into CouchDB's `[jwt_keys]` via the config API or INI file
- Constructs and displays a setup URI containing all JWT settings (`useJWT: true`, `jwtAlgorithm: ES512`, `jwtKey: <private_key>`, `jwtKid: <kid>`, `jwtSub: <username>`, `jwtExpDuration: 5`)
- Encrypts the URI with a per-user passphrase and presents it as a clickable `obsidian://setuplivesync?settings=[...]` link
**3. couchdb-idp-updater sidecar** (optional, only needed if you also want Authentik-issued JWTs accepted directly by CouchDB for API access). This periodically syncs Authentik's JWKS public keys into CouchDB's config.
The provisioning service is the critical custom component. It bridges the gap between Authentik's identity management and LiveSync's key-based JWT model. Users authenticate once through their browser via Authentik SSO, receive their setup URI, paste it into Obsidian, and from that point the plugin handles all authentication autonomously by signing its own tokens.
## Conclusion
The deployment hinges on a non-obvious insight: **LiveSync's JWT support is self-contained, not IdP-dependent**. The plugin signs tokens locally using a stored private key, which means the OIDC provider's role shifts from runtime token issuer to user provisioning backbone. CouchDB's `roles_claim_path = _couchdb\.roles` with the escaped dot, `required_claims = exp,iat`, and per-kid key entries in `[jwt_keys]` form the server-side foundation. The Kishieel blog's Lua proxy pattern remains valuable for browser-based CouchDB admin access but is unnecessary for the Obsidian plugin itself. The main engineering work is building the provisioning service that generates key pairs, configures CouchDB databases, and outputs encrypted setup URIs — a task well-suited to a Claude Code automation prompt targeting Docker Compose on Contabo with Caddy as the edge proxy.

469
vault/projects/matrix-synapse-deployment.md Normal file
View file

@ -0,0 +1,469 @@
# Matrix Synapse Deployment
**Status:** Deployed 2026-02-15, migrated to Contabo 2026-02-15
**Target:** Contabo VPS (5.189.158.149 / 100.64.0.1)
**URLs:** https://matrix.echo6.co (Synapse), https://element.echo6.co (Element Web)
**Server Name:** echo6.co (federated identity: @user:echo6.co)
---
## Architecture
| Component | Detail |
|-----------|--------|
| Host | Contabo VPS (5.189.158.149 / 100.64.0.1) |
| Docker services | Synapse (127.0.0.1:8008), Element Web (127.0.0.1:8088), PostgreSQL 16 |
| Reverse proxy | Contabo Caddy (auto ACME certs) |
| SSO | Authentik OIDC → communication-users group |
| Federation | Well-known delegation on echo6.co base domain (served by utility Caddy) |
| Compose path | `/opt/matrix/docker-compose.yml` |
| Backup | Daily at 3AM, 14-day retention, `/opt/matrix/backups/` |
The server name is `echo6.co` (not `matrix.echo6.co`) so federated user IDs are `@user:echo6.co`. The Synapse instance lives at `matrix.echo6.co` and delegation is handled via `.well-known` endpoints on the base domain.
---
## Step 1: Provision LXC Container
Run **ct-runbook.md** on the utility node with these parameters:
```
CTID=108
HOSTNAME=matrix
STORAGE=local-lvm
DISK_SIZE=16
MEMORY=2048
CORES=2
BRIDGE=vmbr0
```
After the runbook completes (user, SSH, Docker, Tailscale all verified), continue here.
---
## Step 2: Create Project Structure
SSH into CT 108:
```bash
CT_IP=$(ssh root@192.168.1.241 "pct exec 108 -- hostname -I | awk '{print \$1}'")
sshpass -p '7redditGold' ssh zvx@$CT_IP
```
Create directories:
```bash
sudo mkdir -p /opt/matrix/{synapse,postgres,element,backups,scripts}
sudo chown -R zvx:zvx /opt/matrix
```
---
## Step 3: Create Docker Compose
Create `/opt/matrix/docker-compose.yml`:
```yaml
services:
postgres:
image: postgres:16-alpine
container_name: matrix-postgres
restart: unless-stopped
environment:
POSTGRES_DB: synapse
POSTGRES_USER: synapse
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_INITDB_ARGS: "--encoding=UTF8 --lc-collate=C --lc-ctype=C"
volumes:
- ./postgres:/var/lib/postgresql/data
networks:
- matrix-net
healthcheck:
test: ["CMD-SHELL", "pg_isready -U synapse -d synapse"]
interval: 10s
timeout: 5s
retries: 5
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix-synapse
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
environment:
SYNAPSE_CONFIG_PATH: /data/homeserver.yaml
volumes:
- ./synapse:/data
ports:
- "8008:8008"
networks:
- matrix-net
element:
image: vectorim/element-web:latest
container_name: matrix-element
restart: unless-stopped
volumes:
- ./element/config.json:/app/config.json:ro
ports:
- "8080:80"
networks:
- matrix-net
networks:
matrix-net:
driver: bridge
```
Create `/opt/matrix/.env`:
```bash
POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 32)
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" > /opt/matrix/.env
chmod 600 /opt/matrix/.env
echo "Save this password to /home/zvx/projects/.ref/credentials"
cat /opt/matrix/.env
```
---
## Step 4: Generate Synapse Config
```bash
cd /opt/matrix
docker run -it --rm \
-v ./synapse:/data \
-e SYNAPSE_SERVER_NAME=echo6.co \
-e SYNAPSE_REPORT_STATS=no \
matrixdotorg/synapse:latest generate
```
Edit `synapse/homeserver.yaml` — replace the full `database` section and add OIDC config:
```yaml
server_name: "echo6.co"
public_baseurl: "https://matrix.echo6.co/"
listeners:
- port: 8008
type: http
tls: false
x_forwarded: true
bind_addresses: ['0.0.0.0']
resources:
- names: [client, federation]
compress: false
database:
name: psycopg2
args:
user: synapse
password: <POSTGRES_PASSWORD from .env>
database: synapse
host: matrix-postgres
port: 5432
cp_min: 5
cp_max: 10
media_store_path: /data/media_store
enable_registration: false
url_preview_enabled: true
# Authentik OIDC — fill client_id and client_secret after running authentik-oidc-application.md
oidc_providers:
- idp_id: authentik
idp_name: "Echo6 SSO"
discover: true
issuer: "https://auth.echo6.co/application/o/matrix/"
client_id: "<from authentik-oidc-application.md>"
client_secret: "<from authentik-oidc-application.md>"
scopes: ["openid", "profile", "email"]
user_mapping_provider:
config:
localpart_template: "{{ user.preferred_username }}"
display_name_template: "{{ user.name }}"
```
---
## Step 5: Configure Element Web
Create `/opt/matrix/element/config.json`:
```json
{
"default_server_config": {
"m.homeserver": {
"base_url": "https://matrix.echo6.co",
"server_name": "echo6.co"
}
},
"brand": "Echo6 Chat",
"disable_guests": true,
"disable_3pid_login": false
}
```
---
## Step 6: Start Services
```bash
cd /opt/matrix
docker compose up -d
docker compose ps
```
Wait for Synapse to initialize the database (watch logs):
```bash
docker compose logs -f synapse
# Wait for "Synapse now listening on TCP port 8008"
```
---
## Step 7: Expose via Caddy and DNS
Run **expose-service-home.md** twice — once for `matrix.echo6.co` and once for `element.echo6.co`.
This service has OIDC, so use local IP per the runbook's decision table.
### matrix.echo6.co
- Backend: `192.168.1.108:8008` (local IP, has OIDC)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS
Caddy site block (note the path-based routing for Matrix):
```caddyfile
matrix.echo6.co {
tls /etc/caddy/certs/matrix.echo6.co.fullchain.crt /etc/caddy/certs/matrix.echo6.co.key
reverse_proxy /_matrix/* 192.168.1.108:8008
reverse_proxy /_synapse/* 192.168.1.108:8008
}
```
### element.echo6.co
- Backend: `192.168.1.108:8080` (local IP)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS
```caddyfile
element.echo6.co {
tls /etc/caddy/certs/element.echo6.co.fullchain.crt /etc/caddy/certs/element.echo6.co.key
reverse_proxy 192.168.1.108:8080
}
```
### Well-known delegation (federation)
This must go on the `echo6.co` base domain. Check if there's already an `echo6.co` block in the Utility Caddy Caddyfile — if so, merge these `handle` directives into it. If not, add a new block:
```caddyfile
echo6.co {
tls /etc/caddy/certs/echo6.co.fullchain.crt /etc/caddy/certs/echo6.co.key
handle /.well-known/matrix/server {
header Content-Type application/json
respond `{"m.server": "matrix.echo6.co:443"}`
}
handle /.well-known/matrix/client {
header Content-Type application/json
header Access-Control-Allow-Origin *
respond `{"m.homeserver": {"base_url": "https://matrix.echo6.co"}}`
}
# ... any existing handlers for echo6.co ...
}
```
If `echo6.co` doesn't have a cert yet, issue one via acme.sh following the same pattern in expose-service-home.md.
### dnsmasq split DNS
Add to `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo:
```
address=/matrix.echo6.co/100.64.0.8
address=/element.echo6.co/100.64.0.8
```
Both point to the Utility Caddy Tailscale IP (100.64.0.8), which proxies to CT 108.
Restart dnsmasq:
```bash
ssh root@100.64.0.1 "systemctl restart dnsmasq"
```
---
## Step 8: Configure Authentik SSO
Run **authentik-oidc-application.md** with these inputs:
```
SERVICE_NAME=Matrix
SERVICE_SLUG=matrix
SERVICE_URL=https://matrix.echo6.co
OIDC_CALLBACK_PATH=/_synapse/client/oidc/callback
NEEDS_OFFLINE_ACCESS=no
CLIENT_TYPE=confidential
```
After completing the runbook, take the Client ID and Client Secret and update `synapse/homeserver.yaml` (Step 4) with the real values. Then restart Synapse:
```bash
cd /opt/matrix && docker compose restart synapse
```
---
## Step 9: Bind Access Group
Run **authentik-access-groups.md** Procedure B to bind the `matrix` application to the `communication-users` group.
```
APP_SLUG=matrix
GROUP_PK=31bce176-cd86-4aea-8db3-a57e03d5c2d1 # communication-users
```
This shares the same access group as Mailcow.
---
## Step 10: Create Admin User
```bash
docker exec -it matrix-synapse register_new_matrix_user \
-u matt \
-p <secure-password> \
-a \
-c /data/homeserver.yaml \
http://localhost:8008
```
---
## Step 11: Schedule PostgreSQL Backups
Run **pg-backup.md** with these inputs:
```
CONTAINER_NAME=matrix-postgres
DB_NAME=synapse
DB_USER=synapse
BACKUP_DIR=/opt/matrix/backups
RETENTION_DAYS=14
CRON_SCHEDULE="0 3 * * *"
```
---
## Verification
### Internal (from CT 108)
```bash
curl -s http://localhost:8008/_matrix/client/versions | jq .
curl -s http://localhost:8008/_matrix/federation/v1/version | jq .
curl -s http://localhost:8080 | head -5
```
### External (from cortex or any tailnet device)
```bash
curl -s https://matrix.echo6.co/_matrix/client/versions | jq .
curl -s https://matrix.echo6.co/_matrix/federation/v1/version | jq .
curl -sI https://element.echo6.co | head -5
curl -s https://echo6.co/.well-known/matrix/server | jq .
curl -s https://echo6.co/.well-known/matrix/client | jq .
```
### Federation
```bash
curl -s "https://federationtester.matrix.org/api/report?server_name=echo6.co" | jq '.FederationOK'
```
Must return `true`.
### SSO
1. Open https://element.echo6.co
2. Click SSO login
3. Should redirect to auth.echo6.co → authenticate → redirect back to Element
4. Verify user identity matches Authentik profile
---
## Troubleshooting
### Synapse won't start
```bash
docker compose logs synapse 2>&1 | tail -50
```
Common causes: bad YAML indentation in homeserver.yaml, wrong PostgreSQL password, database not ready.
### Federation test fails
Check in order:
1. `.well-known/matrix/server` returns `{"m.server": "matrix.echo6.co:443"}`
2. `/_matrix/federation/v1/version` is accessible from the public internet
3. Caddy is routing `/_matrix/*` paths correctly (not just root)
4. GoDaddy DNS for `echo6.co` points to 199.6.36.163
### SSO login loop
See troubleshooting in authentik-oidc-application.md. Most common cause: missing signing key on the Authentik provider, or wrong callback path.
### Element can't connect
Verify Element's `config.json` has `base_url` set to `https://matrix.echo6.co` (not `http://`, not `localhost`).
---
## Runbook References
| Step | Runbook | Purpose |
|------|---------|---------|
| 1 | ct-runbook.md | LXC provisioning, Docker, user, SSH, Tailscale |
| 7 | expose-service-home.md | SSL cert, Caddy site block, GoDaddy DNS |
| 8 | authentik-oidc-application.md | Create OIDC provider + application |
| 9 | authentik-access-groups.md | Bind communication-users group |
| 11 | pg-backup.md | Scheduled PostgreSQL backup with retention |
---
## Credentials Reference
Store in `/home/zvx/projects/.ref/credentials`:
```
# Matrix Synapse
MATRIX_POSTGRES_PASSWORD=<from .env>
MATRIX_OIDC_CLIENT_ID=<from authentik-oidc-application.md>
MATRIX_OIDC_CLIENT_SECRET=<from authentik-oidc-application.md>
MATRIX_OIDC_ISSUER=https://auth.echo6.co/application/o/matrix/
MATRIX_ADMIN_USER=matt
```
---
## Post-Deploy Updates
After deployment, update these docs:
- `docs/services/services.md` — add Matrix entry
- `docs/software/caddy.md` — add matrix.echo6.co and element.echo6.co site blocks
- `docs/software/dns.md` — note well-known delegation on echo6.co
- `docs/hardware/environment.md` — add CT 108 to LXC table and Headscale node list
- `runbooks/authentik-access-groups.md` — add Matrix to application bindings table
---
*Created: 2026-02-15*

561
vault/projects/meshtastic-headscale-runbook.md Normal file
View file

@ -0,0 +1,561 @@
# IdahoMesh Tailnet Runbook
## Overview
Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.
### Architecture
```
Echo6 Headscale (100.64.0.x)
↓ (one-way only)
[Bridge LXC] ← dual tailscaled, NAT + firewall
↓
IdahoMesh Headscale (100.100.0.x)
↕ ↕
Nebra CM3s Sidpatchy's devices
```
> **Security:** The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.
### IP Allocation
| Tailnet | Prefix | Notes |
|-------------|------------------|------------------------------------------|
| Echo6 | 100.64.0.0/10 | Existing, do not change |
| IdahoMesh | 100.100.0.0/16 | Within Tailscale's required 100.64.0.0/10 supernet |
### Infrastructure
| Component | VMID | Host | Local IP | Purpose |
|-----------|------|------|----------|---------|
| meshtastic-hs | CT 106 | utility | 192.168.1.106 | IdahoMesh Headscale server |
| mesh-bridge | CT 107 | utility | 192.168.1.107 | One-way bridge between tailnets |
---
## Phase 1: IdahoMesh Headscale Instance
### 1.1 Create the LXC on utility
```bash
ssh root@192.168.1.241
pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
--hostname meshtastic-hs \
--memory 512 \
--cores 1 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
--storage local-lvm \
--rootfs local-lvm:4 \
--unprivileged 1 \
--onboot 1 \
--start 1
```
Bootstrap standard packages:
```bash
echo6-bootstrap-ct.sh 106
```
### 1.2 Install Headscale
```bash
pct exec 106 -- bash -c '
apt update && apt install -y curl
HEADSCALE_VERSION="0.28.0"
curl -Lo /usr/local/bin/headscale \
"https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
chmod +x /usr/local/bin/headscale
mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
'
```
### 1.3 Configure Headscale
Create `/etc/headscale/config.yaml`:
```yaml
server_url: https://vpn.idahomesh.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false
noise:
private_key_path: /var/lib/headscale/noise_private.key
prefixes:
v4: 100.100.0.0/16
v6: fd7a:115c:a1e0:ab00::/56
allocation: sequential
derp:
server:
enabled: false
urls:
- https://controlplane.tailscale.com/derpmap/default
paths: []
auto_update_enabled: true
update_frequency: 3h
disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m
database:
type: sqlite
debug: false
gorm:
prepare_stmt: true
parameterized_queries: true
skip_err_record_not_found: true
slow_threshold: 1000
sqlite:
path: /var/lib/headscale/db.sqlite
write_ahead_log: true
wal_autocheckpoint: 1000
policy:
mode: file
path: /etc/headscale/acl.json
dns:
magic_dns: true
base_domain: mesh.local
override_local_dns: true
nameservers:
global:
- 1.1.1.1
- 9.9.9.9
split: {}
search_domains: []
extra_records: []
unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"
logtail:
enabled: false
randomize_client_port: false
log:
level: info
format: text
```
> **Note:** Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind Caddy, so TLS termination happens at the reverse proxy.
### 1.4 Create the ACL Policy
Create `/etc/headscale/acl.json`:
```json
{
"groups": {
"group:malice": ["malice@"],
"group:sidpatchy": ["sidpatchy@"],
"group:nebra": ["nebra@"]
},
"acls": [
{
"action": "accept",
"src": ["group:nebra"],
"dst": ["group:nebra:*"],
"comment": "Nebra gateways talk to each other"
},
{
"action": "accept",
"src": ["group:malice"],
"dst": ["group:nebra:*"],
"comment": "Echo6 bridge can reach Nebras"
},
{
"action": "accept",
"src": ["group:sidpatchy"],
"dst": ["group:nebra:*"],
"comment": "Sidpatchy can reach Nebras"
},
{
"action": "accept",
"src": ["group:nebra"],
"dst": ["group:malice:*", "group:sidpatchy:*"],
"comment": "Nebras can respond back to both"
}
]
}
```
> **Important:** Headscale v0.28.0 requires usernames in ACL groups to have `@` suffix (e.g., `malice@`). The `--user` flag on CLI commands takes user IDs (integers), not names.
>
> **No malice↔Sidpatchy rules.** They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).
### 1.5 Create systemd Service
Create `/etc/systemd/system/headscale.service`:
```ini
[Unit]
Description=Headscale - IdahoMesh Tailnet
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
```bash
systemctl daemon-reload
systemctl enable --now headscale
systemctl status headscale
```
### 1.6 Create Users and Preauthkeys
```bash
headscale users create echo6
headscale users create sidpatchy
headscale users create nebra
# For the bridge LXC (your side)
headscale preauthkeys create --user echo6 --expiration 24h
# Save this key ^^^
# For Sidpatchy — send this to him
headscale preauthkeys create --user sidpatchy --expiration 72h
# Save this key ^^^
# For Nebra CM3 gateways (reusable so all Nebras use same key)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# Save this key ^^^
```
---
## Phase 2: Bridge LXC (CT 107 on utility)
This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows **one-way only**: Echo6 → IdahoMesh.
### 2.1 Create the LXC
```bash
ssh root@192.168.1.241
pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
--hostname mesh-bridge \
--memory 256 \
--cores 1 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
--storage local-lvm \
--rootfs local-lvm:2 \
--unprivileged 1 \
--features nesting=1 \
--onboot 1 \
--start 1
```
Add TUN device access for Tailscale (on the Proxmox host):
```bash
# Stop the container first
pct stop 107
cat >> /etc/pve/lxc/107.conf << 'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 107
```
### 2.2 Install Tailscale
```bash
pct exec 107 -- bash -c '
curl -fsSL https://tailscale.com/install.sh | sh
'
```
### 2.3 Set Up Dual tailscaled
Create directories for the second instance:
```bash
pct exec 107 -- bash -c '
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
'
```
The default tailscaled service handles Echo6. Create a second service for IdahoMesh:
Create `/etc/systemd/system/tailscaled-meshtastic.service`:
```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/sbin/tailscaled \
--state=/var/lib/tailscale-meshtastic/tailscaled.state \
--socket=/var/run/tailscale-meshtastic/tailscaled.sock \
--port=41642 \
--tun=tailscale1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```
### 2.4 Enable IP Forwarding
```bash
cat <<EOF > /etc/sysctl.d/99-bridge.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```
### 2.5 Join Both Tailnets
```bash
# Join Echo6 (default tailscaled instance)
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
tailscale up \
--login-server=https://vpn.echo6.co \
--advertise-routes=100.100.0.0/16 \
--accept-routes
# Join IdahoMesh (second instance)
# Do NOT advertise Echo6 routes — one-way only
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
--login-server=https://vpn.idahomesh.com \
--authkey=<echo6-preauthkey-from-step-1.6> \
--accept-routes
```
After joining, approve the advertised route on Echo6 Headscale only:
```bash
# On Echo6 Headscale (Contabo) — enable the 100.100.0.0/16 route
docker exec headscale-vanilla headscale routes list
docker exec headscale-vanilla headscale routes enable -r <route-id>
# NO route approval needed on IdahoMesh Headscale — nothing is advertised
```
### 2.6 Configure One-Way Firewall and NAT
This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.
Install iptables:
```bash
apt install -y iptables iptables-persistent
```
Apply rules:
```bash
# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE
# Allow Echo6 → IdahoMesh (outbound)
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT
# Allow established/related return traffic only (responses to Echo6-initiated connections)
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT
# DROP all new connections from IdahoMesh → Echo6
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP
```
Persist across reboots:
```bash
netfilter-persistent save
```
Verify:
```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```
> **What this achieves:**
> - Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
> - Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
> - Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
> - Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)
---
## Phase 3: Expose vpn.idahomesh.com
### 3.1 Issue SSL Certificate
```bash
ssh root@192.168.1.241
pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
'
```
### 3.2 Install Certificate
```bash
pct exec 101 -- bash -c '
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
--cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
--key-file /etc/caddy/certs/vpn.idahomesh.com.key \
--fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
--reloadcmd "systemctl reload caddy"
chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
'
```
### 3.3 Add Caddy Site Block
```bash
pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''
vpn.idahomesh.com {
tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
reverse_proxy 192.168.1.106:8080
}
EOF
systemctl reload caddy'
```
### 3.4 Add GoDaddy DNS Record
```bash
# On cortex/TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163
```
### 3.5 Verify
```bash
dig +short vpn.idahomesh.com
# Should return 199.6.36.163
curl -I https://vpn.idahomesh.com
```
---
## Phase 4: Register Nebra CM3 Gateways
Only Burley Butte for now. See `idahomesh-vpn-device-setup.md` for the full device onboarding runbook.
```bash
# SSH to Burley Butte
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up \
--login-server=https://vpn.idahomesh.com \
--authkey=<nebra-preauthkey-from-step-1.6> \
--hostname=burley-butte
```
Verify on the IdahoMesh Headscale:
```bash
# On CT 106
headscale nodes list
```
---
## Phase 5: Sidpatchy Onboarding
Send Sidpatchy the following:
1. **IdahoMesh VPN URL:** `https://vpn.idahomesh.com`
2. **Preauthkey:** (the one generated in Step 1.6 for sidpatchy)
3. **Device setup runbook:** `idahomesh-vpn-device-setup.md`
---
## Phase 6: Verification
### From the bridge LXC (CT 107)
```bash
# Ping Burley Butte via IdahoMesh tailnet
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte
# Check status on both tailnets
tailscale status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```
### From any Echo6 machine (via bridge routes)
```bash
# Should be routable through the bridge (NAT'd)
ping 100.100.0.x # Burley Butte's IdahoMesh IP
```
### Verify isolation — from IdahoMesh side
```bash
# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
ping 100.64.0.14 # cortex — should timeout/unreachable
```
---
## Quick Reference
| Component | Location | Tailnet | IP |
|----------------|------------------|------------|-----------------|
| IdahoMesh HS | CT 106, utility | IdahoMesh | 192.168.1.106 |
| Bridge LXC | CT 107, utility | Both | 192.168.1.107 |
| Burley Butte | Field site | IdahoMesh | 100.100.0.x |
| Sidpatchy | Remote | IdahoMesh | 100.100.0.x |
---
## Maintenance Notes
- **Preauthkeys expire.** Generate long-lived reusable keys for Nebras, short-lived for humans.
- **Headscale updates:** Check releases at https://github.com/juanfont/headscale/releases
- **ACL changes:** Edit `/etc/headscale/acl.json` on CT 106, then `systemctl reload headscale`
- **Firewall rules:** Persisted via `netfilter-persistent` on CT 107. Verify after reboot with `iptables -L FORWARD -v -n`
- **If a Nebra goes offline:** Check `headscale nodes list` — may need a new key if expired.
- **Sidpatchy wants off?** `headscale nodes delete -i <node-id>` and revoke the preauthkey.
- **Device setup instructions:** See `idahomesh-vpn-device-setup.md`
---
*Last updated: 2026-02-11*

92
vault/projects/mmud-project.md Normal file
View file

@ -0,0 +1,92 @@
# MMUD — Mesh Multi-User Dungeon
Text-based multiplayer dungeon crawler for Meshtastic LoRa mesh networks. BBS door games (LORD, TradeWars) adapted for 150-char mesh radio constraints, async play, 30-day wipe cycles.
## Status
**Phase:** Deployed and running — all 6 phases implemented, NPC conversation system live with Gemini 2.5 Flash.
## Deployment
- **Game Daemon:** CT 109 (192.168.1.109) on utility node, Docker container, Flask dashboard on port 5000
- **Dashboard:** https://mmud.echo6.co (Last Ember — dark tavern aesthetic)
- **SIM Nodes:** 6 meshtasticd LXC containers (CT 111-116) on utility
- EMBR (CT 111) — game server
- DCRG (CT 112) — broadcast
- GRST (CT 113) — Grist barkeep NPC
- MRN (CT 114) — Maren healer NPC
- TRVL (CT 115) — Torval merchant NPC
- WSPR (CT 116) — Whisper sage NPC
- **LLM Backend:** Gemini 2.5 Flash via Google genai SDK (configured in DB `llm_config` table)
- **Compose:** `/opt/mmud/docker-compose.yml` on CT 109
- **Admin:** https://mmud.echo6.co/admin (session auth, password in docker env)
## Repo
`/home/zvx/projects/mmud` (GitHub: zvx-echo6/mmud)
The repo contains a `CLAUDE.md` with full architecture, directory structure, and implementation guidance. **Read it first before any work.**
## Key Files
- `CLAUDE.md` — Architecture, patterns, gotchas
- `docs/planned.md` — Complete game design document (~950 lines). Source of truth for all mechanics.
- `docs/npc-lore.md` — NPC deep lore bible (5-layer Hearth-Sworn backstories, 514 lines)
- `docs/worldbuilding.md` — Surface-level worldbuilding (Legend of Oryn, floor identities)
- `config.py` — All game constants with rationale
- `src/db/schema.sql` — Full database schema (migrations in `src/db/migrations/`)
## Architecture
- **Python 3.11+**, SQLite, Meshtastic Python API, Flask 3.x, Jinja2
- **Docker:** python:3.11-slim, /data volume for SQLite DB
- **6-node mesh topology:** EMBR (game), DCRG (broadcast), 4 NPC nodes
- **150 characters per message** — hard ceiling from Meshtastic LoRa
- **12 dungeon actions/day**, 30-day epochs, async-first
- **828+ tests** — `python3 -m pytest tests/ -x -v`
## NPC Conversation System
NPCs use runtime LLM calls (the one exception to the "no runtime LLM" rule). Key features:
- **TX Tag System:** LLM prefixes responses with `[TX:action:detail]` for game mechanics (heal, buy, sell, browse, gamble, hint, recap)
- **Session Memory:** Per-player per-NPC conversation history with persistent memory summaries
- **Deep Lore:** Five-layer backstory system (Surface → Observations → History → Truth → Unspeakable)
- Trigger word detection pushes toward deeper layers
- Interaction count tracks conversation depth per player per NPC
- "Soren" is a Layer 5 nuclear trigger for all 4 NPCs
- **Easter Eggs:** Death memory (Maren), daily gamble (Torval), countdown (Whisper), late-epoch vulnerability (Maren), inter-NPC secret (Torval/Whisper)
- **DummyBackend:** Keyword-based offline mode for testing without LLM
## LLM Configuration
- Model configured via `llm_config` table in SQLite DB (not env vars)
- Currently: Gemini 2.5 Flash (`gemini-2.5-flash`)
- API key stored in DB, manageable via admin panel at /admin/llm
- **No `max_output_tokens` restrictions** — Gemini 2.5 Flash thinking tokens consume the budget, causing truncation. All backends have token limits removed.
- Supports: Google (Gemini), Anthropic (Claude), OpenAI-compatible backends
## Development Phases (All Complete)
1. **Core Loop** — Message handling, parser, player creation, navigation, combat, death, action budget
2. **Economy & Progression** — XP, leveling, gold, shops, gear, bank, healer
3. **Social Systems** — Broadcasts, barkeep, bounty board, player messages, mail
4. **Epoch Generation** — World gen, LLM narrative pipeline, secrets, bounty pools
5. **Endgame Modes** — Hold the Line, Raid Boss, Retrieve & Escape, epoch vote
6. **The Breach** — Breach zone gen, 4 mini-events (Heist, Emergence, Incursion, Resonance)
## Gotchas
- Gemini 2.5 Flash thinking tokens count against `max_output_tokens` — never set token limits
- NPC greeting path uses `complete()`, conversation path uses `chat()` — different code paths
- `npc_memory` table stores `turn_count` for interaction depth tracking
- Death log table (`death_log`) tracks monster kills for Maren's memory feature
- Town actions are always free — never charge dungeon actions in town
- The design doc (`docs/planned.md`) is the source of truth — if code contradicts it, code is wrong
## Notes
- All regen/HP/damage numbers are targets — will need playtesting
- SQLite single file DB, no ORM, raw parameterized SQL
- Every outbound message must fit 150 chars — the formatter is the final gate
- Container on CT 109 connects to 6 SIM nodes via TCP (ports 4403)