Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
89834796ff
commit
e9231ac24a
93 changed files with 51223 additions and 254 deletions
103
archive_receiver_discovery.ref
Normal file
103
archive_receiver_discovery.ref
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
# Archive Receiver Discovery
|
||||
# Generated: 2026-04-09 (Phase 6.0, Question 6)
|
||||
#
|
||||
# NOTE: Hookshot is BLOCKED for E2BE rooms with MAS. This analysis
|
||||
# covers the receiver requirements IF hookshot were used, AND the
|
||||
# alternative approaches that avoid hookshot entirely.
|
||||
|
||||
## Hookshot Receiver Requirements (if hookshot were viable)
|
||||
|
||||
### Minimum Functionality
|
||||
1. Listen on HTTP port (plain HTTP on internal Docker network is fine — no TLS needed)
|
||||
2. Accept multipart/form-data PUT or POST requests
|
||||
3. Verify X-Matrix-Hookshot-Token header against per-room shared secret
|
||||
4. Parse the `event` part as JSON
|
||||
5. Write to durable storage
|
||||
6. Return 200 OK (hookshot retries on non-2xx)
|
||||
|
||||
### Authentication
|
||||
Hookshot sends a per-webhook auth token in X-Matrix-Hookshot-Token header.
|
||||
The receiver validates this token against a known list.
|
||||
|
||||
## Storage Format Comparison
|
||||
|
||||
| Format | Pros | Cons | Recommended For |
|
||||
|--------|------|------|-----------------|
|
||||
| JSONL files | Greppable, simple, no DB, easy backup | No query capability, no indexes, scattered across files | "Never look at it" archival |
|
||||
| SQLite per room | Self-contained, portable, SQL queries | Multiple files to manage, concurrent write limits | Small-scale per-room analysis |
|
||||
| Single SQLite | One file, SQL queries, simple backup | Write contention at scale, max ~10K writes/sec | Small-to-medium single-server |
|
||||
| Postgres | Full SQL, concurrent writes, indexes, JSONB | Needs a running server, more ops overhead | Query-heavy, large-scale |
|
||||
|
||||
Given Matt's "I'll never look at the DB" feedback:
|
||||
- **Primary: JSONL files** — append-only, one per day per room, greppable, zero ops
|
||||
- **Secondary: Single SQLite** — for when he does need to query (and he will eventually)
|
||||
|
||||
Both can coexist. The receiver writes JSONL immediately, a nightly job imports into SQLite.
|
||||
|
||||
## Receiver Location Options
|
||||
|
||||
| Location | Pros | Cons |
|
||||
|----------|------|------|
|
||||
| Same Contabo host | Simplest networking, no cross-host latency | Adds load to already-busy server |
|
||||
| Separate CT on Proxmox | Isolated, near /mnt/library storage | Cross-network traffic, more infrastructure |
|
||||
| pi-nas (library host) | Direct /mnt/library access, no NFS | Pi is slow, limited CPU/RAM |
|
||||
|
||||
**Recommendation:** Separate CT on Proxmox (data node preferred — has 1TB NVMe + 1TB SATA).
|
||||
- /mnt/library is NFS-mounted on data node CTs
|
||||
- Lightweight Python service, minimal resources (512MB RAM, 1 core)
|
||||
- Keeps archive processing off Contabo
|
||||
|
||||
## Alternative Approaches (No Hookshot)
|
||||
|
||||
### Approach A: Synapse-Level Only (Simplest)
|
||||
|
||||
No bot, no receiver. Just Synapse config changes:
|
||||
```yaml
|
||||
redaction_retention_period: null
|
||||
experimental_features:
|
||||
msc2815_enabled: true
|
||||
```
|
||||
|
||||
Data stays in Synapse's Postgres forever. Query via:
|
||||
- Synapse admin API: GET /_synapse/admin/v1/rooms/{room_id}/messages
|
||||
- Direct Postgres: SELECT from event_json WHERE room_id = '...'
|
||||
|
||||
Export scripts run on Contabo, dump to /mnt/library via NFS or rsync.
|
||||
|
||||
Pros: Zero new infrastructure, zero ops burden, data already exists in DB
|
||||
Cons: No real-time alerting, export is batch-only, tied to Synapse DB format
|
||||
|
||||
### Approach B: Custom matrix-nio Bot (Original Phase 6 Plan)
|
||||
|
||||
Python bot using matrix-nio with E2EE + MSC4190 support.
|
||||
- Handles MAS login correctly (unlike hookshot)
|
||||
- Decrypts E2BE rooms natively
|
||||
- Writes to its own DB (independent of Synapse retention)
|
||||
- Real-time capture with custom schema
|
||||
|
||||
Pros: Full control, real-time, independent archive, custom schema
|
||||
Cons: More code to write and maintain, another service to monitor
|
||||
|
||||
### Approach C: Hybrid (Recommended)
|
||||
|
||||
Combine Approach A + lightweight export:
|
||||
1. Enable `redaction_retention_period: null` + `msc2815_enabled: true`
|
||||
→ Synapse retains everything, MSC2815 provides moderator access
|
||||
2. Build a simple export script (NOT a bot, NOT a service):
|
||||
- Runs nightly via cron
|
||||
- Queries Synapse admin API for room events
|
||||
- Writes JSONL + markdown exports to /mnt/library
|
||||
- No E2EE handling needed — queries the server-side decrypted content
|
||||
3. No new services, no bot accounts, no device verification
|
||||
|
||||
This avoids the hookshot E2EE+MAS blocker entirely AND avoids the complexity
|
||||
of a custom matrix-nio bot. The Synapse admin API already has the data.
|
||||
|
||||
## CT Number for Receiver/Export Service (if needed)
|
||||
|
||||
Current CT assignments on data node: CT 130 (RECON)
|
||||
Free CTs on data node: 131-149
|
||||
|
||||
If a dedicated CT is needed: CT 131 (next available on data node)
|
||||
But with Approach C (hybrid), no dedicated CT is needed — the export script
|
||||
runs on Contabo via cron alongside the existing backup job.
|
||||
Loading…
Add table
Add a link
Reference in a new issue