matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync

Browse source 
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup)
- Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing
- Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack
- Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure
- Removes 4 deprecated runbook duplicates (canonical versions live in projects/)
- Adds .gitignore for binary archives and editor temp files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-04-13 06:02:16 +00:00
commit e9231ac24a
93 changed files with 51223 additions and 254 deletions
Download patch file Download diff file Expand all files Collapse all files

701
projects/DEPLOY-API-KEYS-TAB.md Normal file
View file

@ -0,0 +1,701 @@
# RECON Dashboard — API Keys Tab Deployment
## Context
SSH into the RECON LXC as zvx: `ssh zvx@100.64.0.24` (or 192.168.1.130)
Working directory: `/opt/recon/`
The dashboard is a Flask app in `lib/api.py` running on port 8420 as a systemd service (`recon.service`).
We're adding:
1. A new `lib/key_manager.py` module (thread-safe, hot-reloadable API key store)
2. A new "API Keys" tab on the dashboard
3. API endpoints for key management
4. Hot-reload integration — enricher and extractor pull keys from KeyManager instead of .env directly
## Step 1: Deploy key_manager.py
Create `/opt/recon/lib/key_manager.py` with the contents of the attached `key_manager.py` file. Copy it exactly — it's a complete, tested module.
Verify it loads:
```bash
cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys loaded: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
"
```
This should show the 4 Gemini keys currently in `.env`.
## Step 2: Add API routes to lib/api.py
Add these routes to `lib/api.py`. Find where the other `/api/` routes are defined and add these in the same pattern:
```python
from lib.key_manager import get_key_manager
# ── API Keys Management ──
@app.route('/keys')
def keys_page():
"""API Keys management page."""
return render_template_string(KEYS_TEMPLATE)
@app.route('/api/keys', methods=['GET'])
def api_get_keys():
"""Get all API keys (masked) with stats."""
km = get_key_manager()
return jsonify({
'gemini': {
'keys': km.get_masked_keys(),
'count': km.get_gemini_key_count(),
},
# Placeholder sections for future services
'services': {
'tei': {
'host': config.get('embedding', {}).get('tei_host', 'unknown'),
'port': config.get('embedding', {}).get('tei_port', 'unknown'),
'status': 'managed in config.yaml'
},
'qdrant': {
'host': config.get('vector_db', {}).get('host', 'unknown'),
'port': config.get('vector_db', {}).get('port', 'unknown'),
'status': 'managed in config.yaml'
},
'ollama': {
'host': config.get('embedding', {}).get('ollama_host', 'unknown'),
'port': config.get('embedding', {}).get('ollama_port', 'unknown'),
'status': 'managed in config.yaml'
}
}
})
@app.route('/api/keys/gemini', methods=['POST'])
def api_add_gemini_key():
"""Add a new Gemini API key."""
data = request.get_json()
if not data or 'key' not in data:
return jsonify({'error': 'Missing "key" field'}), 400
km = get_key_manager()
try:
# Optionally validate before adding
if data.get('validate', True):
valid, msg = km.validate_key(data['key'])
if not valid:
return jsonify({'error': f'Key validation failed: {msg}'}), 400
idx = km.add_gemini_key(data['key'])
return jsonify({'success': True, 'index': idx, 'count': km.get_gemini_key_count()})
except ValueError as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/<int:index>', methods=['PUT'])
def api_replace_gemini_key(index):
"""Replace a Gemini API key at a specific index."""
data = request.get_json()
if not data or 'key' not in data:
return jsonify({'error': 'Missing "key" field'}), 400
km = get_key_manager()
try:
if data.get('validate', True):
valid, msg = km.validate_key(data['key'])
if not valid:
return jsonify({'error': f'Key validation failed: {msg}'}), 400
km.replace_gemini_key(index, data['key'])
return jsonify({'success': True, 'count': km.get_gemini_key_count()})
except (IndexError, ValueError) as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/<int:index>', methods=['DELETE'])
def api_delete_gemini_key(index):
"""Remove a Gemini API key by index."""
km = get_key_manager()
try:
masked = km.remove_gemini_key(index)
return jsonify({'success': True, 'removed': masked, 'count': km.get_gemini_key_count()})
except (IndexError, ValueError) as e:
return jsonify({'error': str(e)}), 400
@app.route('/api/keys/gemini/validate', methods=['POST'])
def api_validate_gemini_keys():
"""Validate all loaded Gemini keys."""
km = get_key_manager()
results = km.validate_all()
return jsonify({'results': results})
@app.route('/api/keys/gemini/<int:index>/validate', methods=['POST'])
def api_validate_single_gemini_key(index):
"""Validate a single Gemini key by index."""
km = get_key_manager()
key = km.get_gemini_key(index)
if key is None:
return jsonify({'error': f'No key at index {index}'}), 404
valid, msg = km.validate_key(key)
return jsonify({'index': index, 'valid': valid, 'message': msg})
@app.route('/api/keys/gemini/reveal/<int:index>', methods=['POST'])
def api_reveal_gemini_key(index):
"""Reveal full key (for copy). Requires confirmation in request body."""
data = request.get_json() or {}
if not data.get('confirm'):
return jsonify({'error': 'Send {"confirm": true} to reveal key'}), 400
km = get_key_manager()
key = km.get_gemini_key(index)
if key is None:
return jsonify({'error': f'No key at index {index}'}), 404
return jsonify({'index': index, 'key': key})
@app.route('/api/keys/reload', methods=['POST'])
def api_reload_keys():
"""Force reload keys from .env file."""
km = get_key_manager()
count = km.reload_from_env()
return jsonify({'success': True, 'count': count})
```
**Important:** Make sure `config` refers to whatever variable holds the parsed `config.yaml` in the existing code. Look at how other routes reference config and use the same pattern (likely `get_config()` from `lib/utils.py`).
## Step 3: Add the KEYS_TEMPLATE
Add this HTML template string to `lib/api.py`, alongside the other template strings (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, etc.):
```python
KEYS_TEMPLATE = """
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>RECON — API Keys</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0a0a0a; color: #e0e0e0; min-height: 100vh; }
/* Nav */
.nav { background: #111; border-bottom: 1px solid #222; padding: 0 24px; display: flex; align-items: center; height: 56px; }
.nav-brand { font-size: 18px; font-weight: 700; color: #4ade80; margin-right: 32px; text-decoration: none; letter-spacing: 1px; }
.nav-links { display: flex; gap: 4px; }
.nav-links a { color: #888; text-decoration: none; padding: 8px 16px; border-radius: 6px; font-size: 14px; transition: all 0.15s; }
.nav-links a:hover { color: #e0e0e0; background: #1a1a1a; }
.nav-links a.active { color: #4ade80; background: #1a2e1a; }
/* Layout */
.container { max-width: 960px; margin: 0 auto; padding: 32px 24px; }
h1 { font-size: 24px; font-weight: 600; margin-bottom: 8px; }
.subtitle { color: #666; font-size: 14px; margin-bottom: 32px; }
/* Section */
.section { background: #111; border: 1px solid #222; border-radius: 12px; padding: 24px; margin-bottom: 24px; }
.section-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px; }
.section-title { font-size: 16px; font-weight: 600; display: flex; align-items: center; gap: 8px; }
.section-title .icon { font-size: 20px; }
.section-badge { background: #1a2e1a; color: #4ade80; font-size: 12px; padding: 2px 10px; border-radius: 10px; font-weight: 500; }
/* Key list */
.key-list { display: flex; flex-direction: column; gap: 12px; }
.key-row { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; display: flex; align-items: center; gap: 16px; transition: border-color 0.15s; }
.key-row:hover { border-color: #333; }
.key-index { background: #1a1a1a; color: #666; width: 32px; height: 32px; border-radius: 6px; display: flex; align-items: center; justify-content: center; font-size: 13px; font-weight: 600; flex-shrink: 0; }
.key-value { font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; color: #aaa; flex-grow: 1; word-break: break-all; }
.key-meta { display: flex; gap: 16px; font-size: 12px; color: #555; flex-shrink: 0; }
.key-meta span { white-space: nowrap; }
.key-actions { display: flex; gap: 6px; flex-shrink: 0; }
/* Status badges */
.badge-valid { color: #4ade80; }
.badge-invalid { color: #f87171; }
.badge-unknown { color: #666; }
.badge-ratelimit { color: #fbbf24; }
/* Buttons */
.btn { padding: 6px 14px; border-radius: 6px; border: 1px solid #333; background: #1a1a1a; color: #ccc; font-size: 13px; cursor: pointer; transition: all 0.15s; display: inline-flex; align-items: center; gap: 6px; }
.btn:hover { background: #222; border-color: #444; color: #fff; }
.btn-primary { background: #1a3a1a; border-color: #2a5a2a; color: #4ade80; }
.btn-primary:hover { background: #2a4a2a; border-color: #3a6a3a; }
.btn-danger { background: #2a1a1a; border-color: #5a2a2a; color: #f87171; }
.btn-danger:hover { background: #3a2020; border-color: #6a3030; }
.btn-sm { padding: 4px 10px; font-size: 12px; }
.btn:disabled { opacity: 0.4; cursor: not-allowed; }
/* Input */
.input-row { display: flex; gap: 8px; margin-top: 16px; }
.input-row input { flex-grow: 1; background: #0d0d0d; border: 1px solid #333; border-radius: 6px; padding: 10px 14px; color: #e0e0e0; font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; outline: none; }
.input-row input:focus { border-color: #4ade80; }
.input-row input::placeholder { color: #444; }
/* Status message */
.status-msg { padding: 12px 16px; border-radius: 8px; font-size: 13px; margin-top: 12px; display: none; }
.status-msg.success { display: block; background: #0d1f0d; border: 1px solid #1a3a1a; color: #4ade80; }
.status-msg.error { display: block; background: #1f0d0d; border: 1px solid #3a1a1a; color: #f87171; }
.status-msg.info { display: block; background: #0d0d1f; border: 1px solid #1a1a3a; color: #60a5fa; }
/* Service placeholder cards */
.service-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 12px; }
.service-card { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; }
.service-card .svc-name { font-weight: 600; font-size: 14px; margin-bottom: 4px; }
.service-card .svc-endpoint { font-family: monospace; font-size: 13px; color: #666; margin-bottom: 8px; }
.service-card .svc-note { font-size: 12px; color: #444; font-style: italic; }
/* Spinner */
.spinner { display: inline-block; width: 14px; height: 14px; border: 2px solid #333; border-top-color: #4ade80; border-radius: 50%; animation: spin 0.6s linear infinite; }
@keyframes spin { to { transform: rotate(360deg); } }
/* Confirm overlay */
.confirm-overlay { display: none; position: fixed; inset: 0; background: rgba(0,0,0,0.7); z-index: 100; align-items: center; justify-content: center; }
.confirm-overlay.active { display: flex; }
.confirm-box { background: #111; border: 1px solid #333; border-radius: 12px; padding: 24px; max-width: 420px; width: 90%; }
.confirm-box h3 { margin-bottom: 12px; font-size: 16px; }
.confirm-box p { color: #888; font-size: 14px; margin-bottom: 20px; }
.confirm-actions { display: flex; gap: 8px; justify-content: flex-end; }
</style>
</head>
<body>
<nav class="nav">
<a href="/" class="nav-brand">RECON</a>
<div class="nav-links">
<a href="/">Dashboard</a>
<a href="/search">Search</a>
<a href="/catalogue">Catalogue</a>
<a href="/upload">Upload</a>
<a href="/web-ingest">Web Ingest</a>
<a href="/failures">Failures</a>
<a href="/keys" class="active">API Keys</a>
</div>
</nav>
<div class="container">
<h1>API Key Management</h1>
<p class="subtitle">Manage API keys for pipeline workers. Changes take effect immediately — no restart required.</p>
<!-- Gemini API Keys -->
<div class="section">
<div class="section-header">
<div class="section-title">
<span class="icon">🔑</span>
Gemini API Keys
<span class="section-badge" id="key-count-badge">0 keys</span>
</div>
<div style="display: flex; gap: 8px;">
<button class="btn" onclick="validateAllKeys()" id="btn-validate-all">Validate All</button>
<button class="btn" onclick="reloadFromEnv()">Reload .env</button>
</div>
</div>
<div class="key-list" id="key-list">
<div style="color: #444; text-align: center; padding: 20px;">Loading keys...</div>
</div>
<!-- Add key input -->
<div class="input-row">
<input type="text" id="new-key-input" placeholder="Paste new Gemini API key..." autocomplete="off" spellcheck="false">
<button class="btn btn-primary" onclick="addKey()" id="btn-add">Add Key</button>
</div>
<div id="status-msg" class="status-msg"></div>
<div style="margin-top: 16px; font-size: 12px; color: #444;">
<strong>Used by:</strong> Enrichment (text → concepts, 16 workers) · Vision OCR (scanned PDF fallback) · Title extraction
</div>
</div>
<!-- Future Service Endpoints -->
<div class="section">
<div class="section-header">
<div class="section-title">
<span class="icon">🔌</span>
Service Endpoints
<span class="section-badge" style="background: #1a1a2e; color: #60a5fa;">config.yaml</span>
</div>
</div>
<div class="service-grid" id="service-grid">
<div style="color: #444; text-align: center; padding: 20px;">Loading...</div>
</div>
<div style="margin-top: 16px; font-size: 12px; color: #444;">
Service endpoints are currently managed in <code style="background:#1a1a1a; padding: 2px 6px; border-radius: 3px;">/opt/recon/config.yaml</code>.
Dashboard editing coming in a future update.
</div>
</div>
</div>
<!-- Confirm dialog -->
<div class="confirm-overlay" id="confirm-overlay">
<div class="confirm-box">
<h3 id="confirm-title">Confirm</h3>
<p id="confirm-message">Are you sure?</p>
<div class="confirm-actions">
<button class="btn" onclick="closeConfirm()">Cancel</button>
<button class="btn btn-danger" id="confirm-action-btn" onclick="confirmAction()">Confirm</button>
</div>
</div>
</div>
<script>
let pendingAction = null;
// ── Load keys on page load ──
async function loadKeys() {
try {
const resp = await fetch('/api/keys');
const data = await resp.json();
renderKeys(data.gemini);
renderServices(data.services);
} catch (e) {
showStatus('Failed to load keys: ' + e.message, 'error');
}
}
function renderKeys(gemini) {
const list = document.getElementById('key-list');
const badge = document.getElementById('key-count-badge');
badge.textContent = gemini.count + ' key' + (gemini.count !== 1 ? 's' : '');
if (gemini.keys.length === 0) {
list.innerHTML = '<div style="color:#f87171; text-align:center; padding:20px;">⚠ No Gemini keys loaded — pipeline cannot enrich or OCR</div>';
return;
}
list.innerHTML = gemini.keys.map(k => {
let validClass = 'badge-unknown';
let validIcon = '○';
if (k.valid === true) { validClass = 'badge-valid'; validIcon = '✓'; }
else if (k.valid === false) { validClass = 'badge-invalid'; validIcon = '✗'; }
return `
<div class="key-row" id="key-row-${k.index}">
<div class="key-index">${k.index + 1}</div>
<div class="key-value" id="key-val-${k.index}">${k.masked}</div>
<div class="key-meta">
<span class="${validClass}">${validIcon}</span>
${k.calls > 0 ? `<span>${k.calls} calls</span>` : ''}
${k.errors > 0 ? `<span style="color:#f87171">${k.errors} err</span>` : ''}
</div>
<div class="key-actions">
<button class="btn btn-sm" onclick="validateKey(${k.index})" title="Validate">Test</button>
<button class="btn btn-sm" onclick="revealKey(${k.index})" title="Reveal full key">👁</button>
<button class="btn btn-sm" onclick="promptReplace(${k.index})" title="Replace">↻</button>
<button class="btn btn-sm btn-danger" onclick="promptDelete(${k.index})" title="Remove">✕</button>
</div>
</div>
`;
}).join('');
}
function renderServices(services) {
const grid = document.getElementById('service-grid');
const svcMap = {
tei: { label: 'TEI Embeddings', icon: '📐' },
qdrant: { label: 'Qdrant Vector DB', icon: '🗃' },
ollama: { label: 'Ollama (Fallback)', icon: '🧠' }
};
grid.innerHTML = Object.entries(services).map(([key, svc]) => {
const info = svcMap[key] || { label: key, icon: '⚙' };
return `
<div class="service-card">
<div class="svc-name">${info.icon} ${info.label}</div>
<div class="svc-endpoint">${svc.host}:${svc.port}</div>
<div class="svc-note">${svc.status}</div>
</div>
`;
}).join('');
}
// ── Key operations ──
async function addKey() {
const input = document.getElementById('new-key-input');
const key = input.value.trim();
if (!key) { showStatus('Paste a key first', 'error'); return; }
const btn = document.getElementById('btn-add');
btn.disabled = true;
btn.innerHTML = '<span class="spinner"></span> Validating...';
try {
const resp = await fetch('/api/keys/gemini', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key: key, validate: true })
});
const data = await resp.json();
if (resp.ok) {
showStatus('Key added and validated ✓', 'success');
input.value = '';
loadKeys();
} else {
showStatus(data.error || 'Failed to add key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
} finally {
btn.disabled = false;
btn.innerHTML = 'Add Key';
}
}
async function validateKey(index) {
const row = document.getElementById(`key-row-${index}`);
row.style.borderColor = '#333';
try {
const resp = await fetch(`/api/keys/gemini/${index}/validate`, { method: 'POST' });
const data = await resp.json();
if (data.valid) {
row.style.borderColor = '#2a5a2a';
showStatus(`Key ${index + 1}: ${data.message}`, 'success');
} else {
row.style.borderColor = '#5a2a2a';
showStatus(`Key ${index + 1}: ${data.message}`, 'error');
}
setTimeout(loadKeys, 500);
} catch (e) {
showStatus('Validation failed: ' + e.message, 'error');
}
}
async function validateAllKeys() {
const btn = document.getElementById('btn-validate-all');
btn.disabled = true;
btn.innerHTML = '<span class="spinner"></span> Validating...';
try {
const resp = await fetch('/api/keys/gemini/validate', { method: 'POST' });
const data = await resp.json();
const valid = data.results.filter(r => r.valid).length;
const total = data.results.length;
showStatus(`Validated: ${valid}/${total} keys are working`, valid === total ? 'success' : 'error');
loadKeys();
} catch (e) {
showStatus('Validation failed: ' + e.message, 'error');
} finally {
btn.disabled = false;
btn.innerHTML = 'Validate All';
}
}
async function revealKey(index) {
try {
const resp = await fetch(`/api/keys/gemini/reveal/${index}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ confirm: true })
});
const data = await resp.json();
if (resp.ok) {
const el = document.getElementById(`key-val-${index}`);
el.textContent = data.key;
el.style.color = '#e0e0e0';
// Auto-hide after 10s
setTimeout(() => loadKeys(), 10000);
}
} catch (e) {
showStatus('Failed to reveal: ' + e.message, 'error');
}
}
function promptReplace(index) {
const newKey = prompt(`Paste replacement for key ${index + 1}:`);
if (newKey && newKey.trim()) {
replaceKey(index, newKey.trim());
}
}
async function replaceKey(index, newKey) {
try {
const resp = await fetch(`/api/keys/gemini/${index}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key: newKey, validate: true })
});
const data = await resp.json();
if (resp.ok) {
showStatus(`Key ${index + 1} replaced and validated ✓`, 'success');
loadKeys();
} else {
showStatus(data.error || 'Failed to replace key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
}
}
function promptDelete(index) {
pendingAction = () => deleteKey(index);
document.getElementById('confirm-title').textContent = 'Remove Key ' + (index + 1);
document.getElementById('confirm-message').textContent = 'This key will be removed from the pipeline immediately. Workers using this key will switch to remaining keys.';
document.getElementById('confirm-overlay').classList.add('active');
}
async function deleteKey(index) {
closeConfirm();
try {
const resp = await fetch(`/api/keys/gemini/${index}`, { method: 'DELETE' });
const data = await resp.json();
if (resp.ok) {
showStatus(`Removed key: ${data.removed}`, 'success');
loadKeys();
} else {
showStatus(data.error || 'Failed to remove key', 'error');
}
} catch (e) {
showStatus('Network error: ' + e.message, 'error');
}
}
async function reloadFromEnv() {
try {
const resp = await fetch('/api/keys/reload', { method: 'POST' });
const data = await resp.json();
if (resp.ok) {
showStatus(`Reloaded ${data.count} key(s) from .env`, 'info');
loadKeys();
}
} catch (e) {
showStatus('Reload failed: ' + e.message, 'error');
}
}
// ── Helpers ──
function showStatus(msg, type) {
const el = document.getElementById('status-msg');
el.textContent = msg;
el.className = 'status-msg ' + type;
if (type === 'success' || type === 'info') {
setTimeout(() => { el.className = 'status-msg'; }, 5000);
}
}
function confirmAction() {
if (pendingAction) { pendingAction(); pendingAction = null; }
}
function closeConfirm() {
document.getElementById('confirm-overlay').classList.remove('active');
pendingAction = null;
}
// Handle Enter key in input
document.getElementById('new-key-input').addEventListener('keydown', e => {
if (e.key === 'Enter') addKey();
});
// Load on page ready
loadKeys();
</script>
</body>
</html>
"""
```
## Step 4: Add "API Keys" to the nav on ALL existing pages
Find the nav HTML in every existing template (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, CATALOGUE_TEMPLATE, UPLOAD_TEMPLATE, WEB_INGEST_TEMPLATE, FAILURES_TEMPLATE). Each has a `<nav>` with links. Add the API Keys link:
```html
<a href="/keys">API Keys</a>
```
Add it after the "Failures" link in each template's nav. The exact pattern to find is something like:
```html
<a href="/failures"...>Failures</a>
```
Add right after it:
```html
<a href="/keys">API Keys</a>
```
## Step 5: Wire enricher.py and extractor.py to use KeyManager
This is the hot-reload part. Currently these modules read keys from config/env at startup. We need them to call `get_key_manager().get_gemini_keys()` each time they need a key, so new keys take effect immediately.
### In lib/enricher.py:
Find where Gemini API keys are loaded or selected (likely something like `cfg['gemini_keys']` or reading from `.env` or `os.environ`). Replace that with:
```python
from lib.key_manager import get_key_manager
# Where a key is selected for a worker (probably in the worker function):
km = get_key_manager()
keys = km.get_gemini_keys()
key = keys[worker_index % len(keys)] # Round-robin across available keys
# After each Gemini call, record usage:
km.record_usage(worker_index % len(keys), success=True) # or success=False on error
```
The exact integration depends on the current code structure. Look for:
- `genai.configure(api_key=...)` calls
- `cfg['gemini_keys']` or `config['gemini_keys']` references
- Any `os.environ.get('GEMINI_KEY')` calls
Replace the key source with `get_key_manager()` calls. The key point: **don't cache the key list** — call `km.get_gemini_keys()` or `km.get_gemini_key(index)` each time so hot-reload works.
### In lib/extractor.py:
Same pattern for the Gemini Vision OCR fallback and the title extraction call. Find where Gemini is configured and replace the key source.
## Step 6: Test
### Test the module:
```bash
cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
results = km.validate_all()
for r in results:
print(f' Key {r[\"index\"] + 1}: {\"✓\" if r[\"valid\"] else \"✗\"} - {r[\"message\"]}')
"
```
### Test the API:
```bash
# Get keys (masked)
curl -s http://localhost:8420/api/keys | python3 -m json.tool
# Validate all
curl -s -X POST http://localhost:8420/api/keys/gemini/validate | python3 -m json.tool
# Validate single
curl -s -X POST http://localhost:8420/api/keys/gemini/0/validate | python3 -m json.tool
```
### Test the dashboard:
Open http://100.64.0.24:8420/keys in a browser. Verify:
- All 4 Gemini keys show up (masked)
- "Validate All" tests each key
- "Test" button validates individual keys
- Eye icon reveals the full key (auto-hides after 10s)
- "Add Key" validates before adding
- "Remove" shows confirmation dialog
- "Replace" prompts for new key and validates
- "Reload .env" picks up external edits
- Service endpoints section shows TEI, Qdrant, Ollama from config.yaml
### Test hot-reload:
1. Open the dashboard, note 4 keys
2. Add a 5th key via the dashboard
3. Check `.env`: `cat /opt/recon/.env` — should have 5 GEMINI_KEY entries
4. Watch logs: `journalctl -u recon -f | grep key_manager` — should show key added
5. The pipeline should immediately start using all 5 keys (enricher round-robins)
### Restart and verify persistence:
```bash
sudo systemctl restart recon
curl -s http://localhost:8420/api/keys | python3 -m json.tool
# Should show same keys as before restart
```
## Report back:
- Screenshot or curl output of /api/keys
- Validate All results
- Confirm nav link appears on all pages
- Confirm hot-reload works (add key, verify enricher uses it)
- Any issues with the existing code structure that needed adapting

245
projects/advbbs-project.md Normal file
View file

@ -0,0 +1,245 @@
# advBBS — Claude Code Project Context
## Source of Truth
**GitHub repo**: https://github.com/zvx-echo6/advbbs (always pull latest before working)
## What is advBBS?
A federated, encryption-first BBS for Meshtastic mesh radio networks. Users interact by sending text DMs to a Meshtastic node running the BBS. Multi-hop mail routing between BBS nodes over LoRa radio. Runs on Raspberry Pi Zero 2 W (~100MB RAM).
Built with Python 3.11, SQLite (WAL mode), Meshtastic Python API. Docker-deployed. This is a "vibe-coded" project built with AI assistance — functional but may have rough edges.
---
## Package Structure
```
advbbs/
├── __init__.py
├── __main__.py # Entry point
├── config.py # TOML config loading, dataclasses
├── cli/
│ ├── config_rich.py # Rich-based interactive config TUI
├── commands/
│ ├── dispatcher.py # Command parser + all !commands
├── core/
│ ├── bbs.py # Main BBS class, event loop, session mgmt
│ ├── boards.py # Board service (CRUD, access control)
│ ├── crypto.py # Argon2id + ChaCha20-Poly1305 encryption
│ ├── mail.py # Mail service (inbox, send, read, delete)
│ ├── maintenance.py # Scheduled cleanup tasks
│ ├── rate_limiter.py # Per-node rate limiting
├── db/
│ ├── connection.py # SQLite connection, schema, migrations
│ ├── models.py # Dataclasses (User, Message, Board, etc.)
│ ├── messages.py # MessageRepository (CRUD)
│ ├── users.py # UserRepository, NodeRepository, UserNodeRepository
├── mesh/
│ ├── interface.py # Meshtastic radio interface, send/receive DMs
├── sync/
│ ├── manager.py # SyncManager — federation orchestrator (RAP, mail routing, retry logic)
│ ├── compat/
│ │ ├── advbbs_native.py # Wire protocol handler (HELLO, SYNC_ACK, bulletin format)
├── utils/
│ ├── formatting.py # Text formatting helpers
│ ├── pagination.py # Message pagination for mesh constraints
tests/
├── test_boards.py
├── test_crypto.py
├── test_mail.py
├── test_maintenance.py
├── test_pagination.py
├── test_sync.py
docs/
├── commands.md
├── mail.md
├── boards.md
├── sync.md # Federation + RAP protocol docs
├── configuration.md
├── deployment.md
├── security.md
├── rap-testing.md # Multi-hop RAP test procedures
├── migration.md # fq51bbs → advBBS migration
├── quickstart.md
├── USER-QUICKSTART.md
├── ELI5.md
```
### Non-obvious file placements
- `crypto.py` and `rate_limiter.py` → `core/` (not root)
- `interface.py` (Meshtastic mesh interface) → `mesh/` (not root)
- `advbbs_native.py` (wire protocol) → `sync/compat/` (not root)
- `dispatcher.py` (all user commands) → `commands/` (not root)
- `config_rich.py` (TUI config) → `cli/` (not root)
- `formatting.py`, `pagination.py` → `utils/`
---
## Database
SQLite with WAL mode, autocommit, `check_same_thread=False`. Row factory enabled for dict-like access.
### Schema (3 migrations)
**Migration 001 — Core tables:**
- `users` — id, username, password_hash, salt, encryption_key, recovery_key_enc, is_admin, is_banned, ban fields
- `nodes` — Meshtastic nodes (node_id like `!abcdef12`, short_name, long_name, SNR/RSSI)
- `user_nodes` — Multi-node identity (user_id ↔ node_id, is_primary)
- `messages` — uuid (UNIQUE), msg_type (`mail`/`bulletin`/`system`), board_id, sender/recipient user/node IDs, subject_enc, body_enc (BLOB NOT NULL), timestamps, origin_bbs, forwarded_to, hop_count, delivery_attempts
- `boards` — name, description, board_type, board_key_enc
- `board_access` — Per-user restricted board access
- `board_states` — Per-user read position
- `bbs_peers` — node_id, bbs_name, protocol, sync_enabled, trust_level
- `sync_log` — message_uuid, peer_id, direction, status, attempts
**Migration 002 — Settings/maintenance:**
- Added columns: `messages.deleted_at_us`, `bbs_peers.callsign/name/capabilities/last_seen_us`
- New tables: `bbs_settings` (KV store), `board_read_positions`
**Migration 003 — RAP:**
- Added peer columns: `health_status`, `failed_heartbeats`, `last_heartbeat_us`, `last_pong_us`, `quality_score`
- New tables: `rap_routes` (dest_bbs, via_peer_id, hop_count, quality_score, expires_at_us), `rap_pending_mail` (queued mail for offline routes)
### Timestamps
All timestamps are microseconds since epoch (`int(time.time() * 1_000_000)`), stored as INTEGER. Column suffix `_us`.
---
## Wire Protocol
All inter-BBS messages sent as Meshtastic DMs. Format: `advBBS|1|<MSG_TYPE>|<payload>`
### RAP Messages (Route Announcement Protocol)
| Message | Purpose | Payload |
|---------|---------|---------|
| `RAP_PING` | Heartbeat | `timestamp_us` |
| `RAP_PONG` | Response + routes | `timestamp_us\|route_table` |
| `RAP_ROUTES` | Route table broadcast | `route_table` |
Route table format: `BBS1:hop:quality;BBS2:hop:quality` (e.g., `MV51:0:1.0;J51B:1:1.00`)
### Mail Protocol Messages
| Message | Format | Purpose |
|---------|--------|---------|
| `MAILREQ` | `MAILREQ\|uuid\|from_user\|from_bbs\|to_user\|to_bbs\|hop\|num_parts\|route` | Request delivery |
| `MAILACK` | `MAILACK\|uuid\|OK` | Accept, ready for chunks |
| `MAILNAK` | `MAILNAK\|uuid\|reason` | Reject (NOUSER, NOROUTE, MAXHOPS, LOOP) |
| `MAILDAT` | `MAILDAT\|uuid\|part/total\|data` | Message chunk (max 150 chars × 3) |
| `MAILDLV` | `MAILDLV\|uuid\|OK\|user@BBS` | Delivery confirmation |
### Mail Flow
```
Sender BBS Destination BBS
│ │
│── MAILREQ ────────────▶│ (pre-flight: user exists?)
│◀── MAILACK ────────────│ (ready for chunks)
│── MAILDAT 1/1 ────────▶│ (body chunk)
│ │ (store in DB)
│◀── MAILDLV ────────────│ (confirmed)
```
Multi-hop: intermediate BBS relays MAILREQ/MAILDAT, tracked via `_relay_mail` dict. Max 5 hops. Route list in MAILREQ prevents loops.
---
## Key Architecture Patterns
### Threading Model
- **Main thread**: asyncio event loop (`bbs._loop`) — runs tick(), scheduled tasks
- **Meshtastic callback thread**: `on_receive` fires from Meshtastic library thread
- **Bridge**: `_schedule_async(coro)` uses `asyncio.run_coroutine_threadsafe()` to schedule work from callback thread onto main loop
### Session Management
Sessions keyed by Meshtastic node_id. Login requires both password AND a registered node (node-based 2FA). Sessions expire after inactivity.
### Encryption
- **At rest**: All message bodies encrypted with user-derived keys (Argon2id KDF → ChaCha20-Poly1305)
- **Transport**: Meshtastic PSK encryption (AES-256) recommended
- **Remote mail**: Stored plaintext on receiving BBS (encrypted at read time by recipient's key)
### Message Constraints
- LoRa max ~150 bytes usable per packet
- Remote mail body max 450 chars (3 chunks × 150)
- Pagination helper chunks long responses for mesh delivery
- TX queue collision avoidance: 2.5s delay between protocol DMs
### Peer Security
Federation traffic whitelisted by peer — only configured peers accepted. Non-peer protocol messages rejected.
---
## Configuration
TOML config file. Key sections: `[bbs]`, `[database]`, `[meshtastic]`, `[crypto]`, `[features]`, `[operating_mode]`, `[sync]`, `[rate_limits]`, `[web_reader]`, `[cli_config]`, `[logging]`.
Operating modes: `full`, `mail_only`, `boards_only`, `repeater`.
Peers configured as `[[sync.peers]]` arrays with `node_id`, `name`, `protocol`, `enabled`.
RAP timing defaults are conservative for mesh (12h heartbeat, 36h route expiry, 24h route share).
---
## Current Live Federation Topology
```
MV51 (Old Man Malice / Matt) ◀──▶ J51B (JeepnJonny)
node: !00ff0001 node: !60a43e58
```
Both running Docker containers. Meshtastic simulator (meshtasticd) for testing.
---
## Known Bug: Federation Mail Delivery Failure
### Symptom
```
[ERROR] advbbs.sync.manager: DELIVER b8e78195: Failed to store in database
```
### Root Cause
`create_incoming_remote_mail()` in `db/messages.py` returns `None` for both duplicates (logged at DEBUG — invisible) and real DB errors. Mesh radio retransmissions deliver the same MAILDAT twice, triggering duplicate detection, but the caller can't distinguish this from a real failure.
Additionally, when a duplicate IS detected, no MAILDLV confirmation is sent back, causing the sender to retry indefinitely.
### Fix Required (3 changes)
1. **`db/messages.py`** — `create_incoming_remote_mail`: Return `"duplicate"` sentinel instead of `None` for duplicate UUID. Promote log from DEBUG → INFO. Add traceback to exception path.
2. **`sync/manager.py`** — `_deliver_remote_mail`: Handle `"duplicate"` return: log at INFO, still send MAILDLV confirmation, clean up state. Improve error message for real failures.
3. **`sync/manager.py`** — `handle_maildat` (the `_handle_maildat` section around line 1068): Add `delivering` flag guard to prevent double scheduling from mesh retransmissions.
---
## Development Notes
- Tests: `pytest tests/` — unit tests for crypto, mail, boards, maintenance, pagination, sync
- Docker build: `docker compose build` (can take 10-15 min on Pi, may need swap)
- Config TUI: `advbbs-config` or `python -m advbbs.cli.config_rich`
- Logs: `docker compose logs -f`
- DB inspection: `sqlite3 /data/advbbs.db ".tables"` inside container
---
## Style / Conventions
- Logging: `logger = logging.getLogger(__name__)` per module
- DB access: Repository pattern (MessageRepository, UserRepository, etc.) wrapping Database methods
- All DB timestamps: microseconds (`_us` suffix)
- UUIDs: `str(uuid.uuid4())` for message dedup
- Meshtastic node IDs: hex string with `!` prefix (e.g., `!00ff0001`)
- Commands: `!` prefix, case-insensitive, short aliases
- Config: TOML with dataclass parsing in `config.py`

262
projects/arr-stack-runbook.md Normal file
View file

@ -0,0 +1,262 @@
# CC Runbook: Build ARR Media Stack on Proxmox `media` Node
## Objective
Build a complete media automation stack on the Proxmox node `media` inside a single Ubuntu VM called `arr`. Each service runs in its own Docker container with a shared bridge network for inter-service communication. All services are exposed on the VM's LAN IP on their respective ports.
**Services:**
- Jellyfin (media server, software transcoding — no GPU)
- Jellyseer (request management)
- Sonarr (TV automation)
- Radarr (Movie automation)
- Prowlarr (indexer manager)
- SABnzbd (Usenet download client)
---
## Phase 0: SSH Prereq Check
**CRITICAL — Do this first. Do not skip.**
```bash
ssh media "echo 'SSH OK to media node'"
```
If this fails, stop and fix SSH access before proceeding. Use sshpass or key auth per `~/.ssh/config`. Cortex is the management host — all commands originate from here.
---
## Phase 1: Create Ubuntu VM on `media`
1. SSH to `media` Proxmox node.
2. Find the next available VMID: `pvesh get /cluster/nextid`
3. Download Ubuntu 24.04 cloud image if not already cached:
- URL: `https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img`
- Store in appropriate Proxmox storage.
4. Create a VM named `arr` with:
- **Network:** bridged to the LAN bridge (likely `vmbr0`)
- **Resource allocation:** Decide based on the combined needs of all six services. Jellyfin (software transcoding) and SABnzbd (decompression) are the heaviest. Sonarr/Radarr/Prowlarr/Jellyseer are lightweight. Size the VM accordingly — suggest at minimum 4 cores and 8GB RAM, but use your judgment.
- **Disk:** 30GB for OS + container configs (media lives on NFS)
- Cloud-init configured with:
- Default user: `zvx`
- SSH key from cortex (discover from `~/.ssh/id_rsa.pub` or equivalent)
- Networking: DHCP or static — check the pattern of other VMs on this node and match it
5. Start the VM, wait for boot, discover and record its LAN IP.
6. Verify SSH from cortex → arr VM works.
---
## Phase 2: Base System Setup on `arr` VM
SSH into the `arr` VM:
1. `apt update && apt upgrade -y`
2. Install Docker + Docker Compose via the official Docker apt repo for Ubuntu.
3. Install NFS client: `apt install -y nfs-common`
4. Install Tailscale and join the tailnet:
- `curl -fsSL https://tailscale.com/install.sh | sh`
- `tailscale up` — use an auth key if available. Check how other VMs joined (look at Headscale config if self-hosted).
- Record the Tailscale IP of the `arr` VM.
5. Discover appropriate PUID/PGID:
- Mount the NFS share temporarily and `ls -ln` to check file ownership.
- If no files exist, create a `media` user/group (e.g., PUID=1000, PGID=1000) and ensure NFS permissions align.
---
## Phase 3: NFS Mount
1. **Discover the NFS server:**
- The NFS export is `/export/arr`, accessible from `100.64.0.0/10` (Tailscale) and `192.168.1.0/24` (LAN).
- Find the NFS server IP by checking:
- `/etc/fstab` on other VMs on this node
- `showmount -e <candidate IPs>` on LAN
- Proxmox storage config: `pvesm status` or `/etc/pve/storage.cfg`
2. `mkdir -p /mnt/arr`
3. `mount -t nfs <NFS_SERVER>:/export/arr /mnt/arr`
4. Create subdirectories if they don't exist:
```
mkdir -p /mnt/arr/{movies,tv,downloads,downloads/complete,downloads/incomplete}
```
5. Set ownership to discovered PUID:PGID on all subdirs.
6. Add to `/etc/fstab` for persistence:
```
<NFS_SERVER>:/export/arr /mnt/arr nfs defaults,_netdev 0 0
```
7. Verify: `umount /mnt/arr && mount -a && ls /mnt/arr`
---
## Phase 4: Docker Containers
### Setup
```bash
mkdir -p /opt/arr/{jellyfin,jellyseer,sonarr,radarr,prowlarr,sabnzbd}
```
Create a Docker bridge network for inter-service communication:
```bash
docker network create arr-net
```
### Container Deployment
Deploy each service as its own standalone container. All containers join `arr-net`. All get `TZ=America/Boise` and the discovered `PUID`/`PGID`.
**Decide per-container resource limits** (CPU shares, memory limits) based on service needs:
- **Heavy:** Jellyfin (transcoding), SABnzbd (decompression) — allocate more CPU/RAM
- **Medium:** Sonarr, Radarr — moderate
- **Light:** Prowlarr, Jellyseer — minimal
Use lightweight images (hotio where available, official otherwise).
#### Jellyfin
- Image: `jellyfin/jellyfin:latest`
- Container name: `jellyfin`
- Port: `8096:8096`
- Volumes:
- `/opt/arr/jellyfin/config:/config`
- `/mnt/arr/movies:/data/movies:ro`
- `/mnt/arr/tv:/data/tv:ro`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Jellyseer
- Image: `fallenbagel/jellyseer:latest`
- Container name: `jellyseer`
- Port: `5055:5055`
- Volumes:
- `/opt/arr/jellyseer/config:/app/config`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Sonarr
- Image: `ghcr.io/hotio/sonarr:latest`
- Container name: `sonarr`
- Port: `8989:8989`
- Volumes:
- `/opt/arr/sonarr/config:/config`
- `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Radarr
- Image: `ghcr.io/hotio/radarr:latest`
- Container name: `radarr`
- Port: `7878:7878`
- Volumes:
- `/opt/arr/radarr/config:/config`
- `/mnt/arr:/data`
- Network: `arr-net`
- Restart: `unless-stopped`
#### Prowlarr
- Image: `ghcr.io/hotio/prowlarr:latest`
- Container name: `prowlarr`
- Port: `9696:9696`
- Volumes:
- `/opt/arr/prowlarr/config:/config`
- Network: `arr-net`
- Restart: `unless-stopped`
#### SABnzbd
- Image: `ghcr.io/hotio/sabnzbd:latest`
- Container name: `sabnzbd`
- Port: `8080:8080`
- Volumes:
- `/opt/arr/sabnzbd/config:/config`
- `/mnt/arr/downloads:/data/downloads`
- Network: `arr-net`
- Restart: `unless-stopped`
### Volume Mapping Design
Sonarr and Radarr both map `/mnt/arr:/data` so hardlinks/atomic moves work between `/data/downloads/complete` and `/data/movies` or `/data/tv` without cross-filesystem copies. This is critical for avoiding double disk usage.
### Verify
All six containers are running: `docker ps`
Curl each service on localhost to confirm they respond on their expected ports.
---
## Phase 5: Authentik OIDC Setup
**Discovery:** Find the Authentik instance.
- Check Caddy config on `utility` for an existing Authentik route (likely `auth.echo6.co` or `authentik.echo6.co`).
- Discover the Authentik API URL and obtain/create an API token from Authentik's docker-compose environment or admin API.
### Jellyfin OIDC
1. Create OAuth2/OpenID Provider in Authentik:
- Name: `jellyfin`, Client type: Confidential
- Redirect URI: `https://jellyfin.echo6.co/sso/OID/redirect/Authentik`
- Scopes: `openid profile email`
- Signing key: use existing or create
2. Create Application: Name `Jellyfin`, slug `jellyfin`, attach provider.
3. Record Client ID + Secret.
4. Install SSO-Auth plugin in Jellyfin and configure with Authentik OIDC details (discovery URL, client ID, secret).
### Jellyseer OIDC
1. Create OAuth2/OpenID Provider in Authentik:
- Name: `jellyseer`, Client type: Confidential
- Redirect URI: `https://requests.echo6.co/api/v1/auth/oidc-callback` (verify actual callback path from Jellyseer docs)
- Scopes: `openid profile email`
2. Create Application: Name `Jellyseer`, slug `jellyseer`, attach provider.
3. Record Client ID + Secret.
4. Configure Jellyseer OIDC via its settings.
---
## Phase 6: Caddy Reverse Proxy on `utility`
SSH to `utility`. Discover the Caddyfile location and how Caddy is managed (docker, systemd, etc.).
Add entries using the **Tailscale IP** of the `arr` VM as the upstream:
```
jellyfin.echo6.co {
reverse_proxy <ARR_TAILSCALE_IP>:8096
}
requests.echo6.co {
reverse_proxy <ARR_TAILSCALE_IP>:5055
}
```
**Do NOT expose Sonarr, Radarr, Prowlarr, or SABnzbd via Caddy.** Those are internal-only, accessible via Tailscale or LAN.
Reload Caddy.
---
## Phase 7: GoDaddy DNS
**Discovery:** Check if GoDaddy API key/secret exists on cortex or utility. Look at how existing `echo6.co` subdomains are configured for the pattern.
Create A records (via API if available, otherwise output for manual creation):
| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | `jellyfin` | Public IP of Caddy/utility (discover) | 600 |
| A | `requests` | Public IP of Caddy/utility (discover) | 600 |
These are publicly exposed WITHOUT Tailscale. Caddy handles TLS via Let's Encrypt. The upstream uses the Tailscale IP but DNS points to the public-facing Caddy IP.
---
## Phase 8: Validation
1. From `arr` VM, curl all six services on localhost (ports 8096, 5055, 8989, 7878, 9696, 8080)
2. `curl -sI https://jellyfin.echo6.co` → 200 with valid TLS
3. `curl -sI https://requests.echo6.co` → 200 with valid TLS
4. Authentik OIDC login works for both Jellyfin and Jellyseer
5. NFS persists after reboot: `reboot`, wait, `df -h /mnt/arr`
6. All containers auto-start after reboot: `docker ps` shows all six running
---
## Important Notes
- **Do NOT configure** Prowlarr indexers, Sonarr/Radarr API connections, or SABnzbd Usenet provider credentials. That will be done in a separate prompt.
- **All discovery steps are intentional** — do not hardcode IPs or paths. Find them dynamically from the running infrastructure.
- **If any phase fails, stop and report the error.** Do not skip phases.

193
projects/arr-wiring-runbook.md Normal file
View file

@ -0,0 +1,193 @@
# CC Runbook: Wire ARR Stack End-to-End
## Objective
Connect all six services on the `arr` VM into a fully automated pipeline:
```
Jellyseer → Sonarr/Radarr → Prowlarr → SABnzbd → Downloads → Sonarr/Radarr catalogue → Jellyfin library → Jellyseer knows what's available
```
All containers are already running on `arr` on the `arr-net` Docker bridge network. Services can reach each other by container name (e.g., `sonarr:8989`).
---
## Phase 0: Prerequisites
### SSH Check
```bash
ssh media "echo 'SSH OK to media node'"
```
Then SSH into the `arr` VM (discover its IP from Phase 1 of the previous runbook, or check `qm list` / DHCP leases on media).
### Read Credentials File
Read the Usenet provider and indexer credentials from `./ref/services/usenet.md` on cortex. Parse and use these values throughout this runbook.
### Discover API Keys
Every service auto-generates an API key on first run. Extract them:
```bash
# Sonarr
docker exec sonarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# Radarr
docker exec radarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# Prowlarr
docker exec prowlarr cat /config/config.xml | grep -oP '(?<=<ApiKey>).*(?=</ApiKey>)'
# SABnzbd
docker exec sabnzbd cat /config/sabnzbd.ini | grep -oP '(?<=api_key = ).*'
# Jellyfin — get the API key from the admin dashboard or create one via API
```
Record all of these. They are needed for every integration below.
---
## Phase 1: SABnzbd — Configure Usenet Provider
Using credentials from `./ref/services/usenet.md`:
1. Via SABnzbd API (`http://localhost:8080/api`), configure the Usenet server:
- Server hostname, port, SSL, username, password — all from the ref file
- Connections: set to provider's recommended max
- SSL verification: enable
2. Set download paths in SABnzbd:
- Complete: `/data/downloads/complete`
- Incomplete: `/data/downloads/incomplete`
3. Configure categories in SABnzbd:
- `movies` → `/data/downloads/complete/movies`
- `tv` → `/data/downloads/complete/tv`
4. Verify SABnzbd can connect to the Usenet server (test connection).
---
## Phase 2: Prowlarr — Add Indexers
Using indexer credentials from `./ref/services/usenet.md`:
1. Via Prowlarr API (`http://localhost:9696/api/v1`), add each Usenet indexer found in the ref file.
- For each indexer: set name, URL, API key, and enable.
- Use the Prowlarr API key discovered in Phase 0.
2. Test each indexer to confirm connectivity.
---
## Phase 3: Prowlarr — Connect to Sonarr and Radarr
Add Sonarr and Radarr as "Applications" in Prowlarr so indexers automatically sync:
### Sonarr
- Prowlarr API → Add Application:
- Type: Sonarr
- Prowlarr server: `http://prowlarr:9696`
- Sonarr server: `http://sonarr:8989`
- API key: Sonarr's API key from Phase 0
- Sync level: Full Sync
### Radarr
- Prowlarr API → Add Application:
- Type: Radarr
- Prowlarr server: `http://prowlarr:9696`
- Radarr server: `http://radarr:7878`
- API key: Radarr's API key from Phase 0
- Sync level: Full Sync
After adding, trigger a sync and verify indexers appear in Sonarr and Radarr.
---
## Phase 4: Sonarr — Configure Download Client and Paths
Via Sonarr API (`http://localhost:8989/api/v3`):
1. Add SABnzbd as download client:
- Host: `sabnzbd`
- Port: `8080`
- API key: SABnzbd API key from Phase 0
- Category: `tv`
- Test connection.
2. Configure Root Folder:
- Path: `/data/tv`
3. Configure Media Management:
- Rename episodes: Yes
- Use hardlinks: Yes (critical — same filesystem via NFS mount)
---
## Phase 5: Radarr — Configure Download Client and Paths
Via Radarr API (`http://localhost:7878/api/v3`):
1. Add SABnzbd as download client:
- Host: `sabnzbd`
- Port: `8080`
- API key: SABnzbd API key from Phase 0
- Category: `movies`
- Test connection.
2. Configure Root Folder:
- Path: `/data/movies`
3. Configure Media Management:
- Rename movies: Yes
- Use hardlinks: Yes
---
## Phase 6: Jellyfin — Configure Libraries
Via Jellyfin API or admin setup:
1. Create (or verify) media libraries:
- **Movies** library → `/data/movies`
- **TV Shows** library → `/data/tv`
2. Set libraries to scan periodically or on change.
3. Create an API key for Jellyseer to use (Admin Dashboard → API Keys → create one named `jellyseer`).
---
## Phase 7: Jellyseer — Connect Everything
Via Jellyseer's setup wizard or API:
1. **Jellyfin connection:**
- Server URL: `http://jellyfin:8096`
- API key: the Jellyfin API key created in Phase 6
- Sync libraries so Jellyseer knows what Jellyfin already has.
- Sign in with the Jellyfin admin account to link it.
2. **Sonarr connection:**
- Server URL: `http://sonarr:8989`
- API key: Sonarr API key from Phase 0
- Root folder: `/data/tv`
- Quality profile: discover available profiles from Sonarr and pick a sensible default (e.g., `Any` or `HD-1080p`)
3. **Radarr connection:**
- Server URL: `http://radarr:7878`
- API key: Radarr API key from Phase 0
- Root folder: `/data/movies`
- Quality profile: discover available profiles and pick a sensible default
---
## Phase 8: End-to-End Validation
Test the full pipeline:
1. **Prowlarr → Indexers:** Search for a common term (e.g., "test") in Prowlarr. Results should return from all configured indexers.
2. **Sonarr → Prowlarr:** In Sonarr, verify indexers are listed under Settings → Indexers (synced from Prowlarr).
3. **Radarr → Prowlarr:** Same check in Radarr.
4. **Sonarr → SABnzbd:** Verify download client is connected (Settings → Download Clients → test).
5. **Radarr → SABnzbd:** Same check.
6. **Jellyseer → Jellyfin:** Verify Jellyseer shows Jellyfin's existing library (if any).
7. **Jellyseer → Sonarr/Radarr:** Verify both are connected in Jellyseer settings.
8. **Full flow test:** If desired, use Jellyseer to request a free/public domain title and verify it flows through the entire chain: Jellyseer → Sonarr/Radarr → Prowlarr search → SABnzbd download → file lands in `/data/tv` or `/data/movies` → Jellyfin picks it up → Jellyseer shows it as available.
---
## Important Notes
- **All services communicate by Docker container name** on `arr-net` (e.g., `http://sonarr:8989`), NOT by localhost or LAN IP.
- **Hardlinks are critical.** Sonarr/Radarr and SABnzbd share the same `/data` mount from the NFS share. This means completed downloads can be hardlinked (not copied) into the media folders, avoiding double disk usage.
- **API-first approach.** Configure everything via API calls rather than manual UI interaction. This ensures repeatability and lets CC automate the full setup.
- **If any phase fails, stop and report the error.** Do not skip phases.
- **The credentials file is `./ref/services/usenet.md` on cortex.** Read it first and use its contents throughout.

223
projects/cc-deploy-watchtower-v2.md Normal file
View file

@ -0,0 +1,223 @@
# Deploy WATCHTOWER v2 — Modular Ops Dashboard
**Context:** CC runs on cortex. WATCHTOWER deploys to Contabo (100.64.0.1). The tarball is at `/home/zvx/projects/contabo/watchtower/watchtower-v2.tar.gz` on cortex. This runbook is at `/home/zvx/.ref/projects/` on cortex.
WATCHTOWER v2 is a modular FastAPI monitoring dashboard. Collectors are auto-discovered from `app/collectors/` and enabled via `{NAME}_ENABLED=true` in `.env`. Adding new monitoring targets requires zero edits to existing files.
## Pre-flight: Transfer tarball and SSH check
```bash
# SCP tarball from cortex (this machine) to Contabo
scp /home/zvx/projects/contabo/watchtower/watchtower-v2.tar.gz zvx@100.64.0.1:/tmp/
# Verify sshpass is installed on Contabo
ssh zvx@100.64.0.1 "which sshpass || sudo apt-get install -y sshpass"
# Test SSH from Contabo to each monitored node
ssh zvx@100.64.0.1 << 'SSHEOF'
echo "=== PeerTube (100.64.0.23) ==="
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no zvx@100.64.0.23 "hostname && echo OK" 2>&1
echo "=== cortex/GPU (100.64.0.14) ==="
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=no zvx@100.64.0.14 "hostname && echo OK" 2>&1
SSHEOF
```
If either SSH fails, stop and report the error. Do not proceed without working SSH to at least one target.
---
## Phase 1: Deploy codebase
All remaining commands run on Contabo. SSH in:
```bash
ssh zvx@100.64.0.1
```
Then:
```bash
# Clean any old install
sudo rm -rf /opt/watchtower
# Extract v2 tarball
sudo tar xzf /tmp/watchtower-v2.tar.gz -C /opt/
sudo mv /opt/watchtower-v2 /opt/watchtower
sudo chown -R $USER:$USER /opt/watchtower
cd /opt/watchtower
```
### Create .env from example
```bash
cp .env.example .env
```
The defaults in `.env.example` are already set to the correct current values:
| Target | IP | User | Notes |
|--------|-----|------|-------|
| GPU (cortex) | 100.64.0.14 | zvx | nvidia-smi |
| PeerTube | 100.64.0.23 | zvx | Native PostgreSQL (`peertube_prod`), pipeline at `/opt/bulk-import/` |
| RECON | disabled | — | Flip `RECON_ENABLED=true` when rebuilt |
### Verify PeerTube PostgreSQL access
PostgreSQL runs natively on the PeerTube CT (not in Docker). Verify:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "sudo -u postgres psql -d peertube_prod -t -A -c 'SELECT COUNT(*) FROM video;'"
```
Should return the video count (e.g., 207). If it errors, the DB name may be different — check with:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "sudo -u postgres psql -l"
```
Update `PT_DB_NAME` in `.env` if needed.
### Verify bulk-import pipeline paths
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.23 "ls -la /opt/bulk-import/ 2>/dev/null && wc -l /opt/bulk-import/downloaded.txt 2>/dev/null || echo 'PATH NOT FOUND'"
```
---
## Phase 2: Build and start
```bash
cd /opt/watchtower
docker compose up -d --build
# Wait for startup then check logs
sleep 5
docker logs watchtower 2>&1 | tail -30
```
### Expected log output
```
WATCHTOWER starting up...
Database connected: /data/watchtower.db
[registry] Loaded collector: gpu (GPU (cortex))
[registry] Loaded collector: peertube (PeerTube Ingest)
[registry] Skipped collector: recon (RECON_ENABLED=false)
[registry] 2 collector(s) active: ['gpu', 'peertube']
[gpu] collector starting (interval: 60s)
[peertube] collector starting (interval: 60s)
```
### Verify collectors
```bash
# Wait for first poll cycle
sleep 65
echo "=== Health ==="
curl -s http://localhost:8084/api/health | python3 -m json.tool
echo "=== Collector Manifest ==="
curl -s http://localhost:8084/api/collectors | python3 -m json.tool
echo "=== GPU Data ==="
curl -s http://localhost:8084/api/c/gpu | python3 -m json.tool
echo "=== PeerTube Data ==="
curl -s http://localhost:8084/api/c/peertube | python3 -m json.tool
```
### ⛔ STOP — Report collector status
Tell me:
1. Which collectors show `"online": true`
2. Any errors from the logs or API responses
3. The PeerTube DB name if it wasn't `peertube_prod`
Do not proceed to Phase 3 until collectors are confirmed.
---
## Phase 3: Public access (Caddy + Authentik)
### Check DNS
```bash
dig +short wt.echo6.co
```
If it doesn't resolve, report that — DNS record needs to be added manually.
### Check/deploy Caddy config
Caddy is at 100.64.0.8 on the mesh.
```bash
echo "=== Check existing config ==="
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "cat ~/docker/caddy/sites/wt.echo6.co* 2>/dev/null || echo 'NO CONFIG FOUND'"
echo "=== Check Caddy is running ==="
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "docker ps --format '{{.Names}}' | grep -i caddy"
```
If no config exists, create it:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "cat > ~/docker/caddy/sites/wt.echo6.co.caddy << 'CADDYEOF'
wt.echo6.co {
forward_auth localhost:9000 {
uri /outpost.goauthentik.io/auth/caddy
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid
trusted_proxies private_ranges
}
reverse_proxy 100.64.0.1:8084
}
CADDYEOF"
```
If config already exists, verify the `reverse_proxy` line points to `100.64.0.1:8084` (Contabo's current Tailscale IP). If it still says `100.64.0.6`, fix it:
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "sed -i 's/100.64.0.6:8084/100.64.0.1:8084/' ~/docker/caddy/sites/wt.echo6.co.caddy"
```
### Reload Caddy
```bash
sshpass -p '7redditGold' ssh zvx@100.64.0.8 "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
```
### Test
```bash
curl -sI https://wt.echo6.co 2>&1 | head -10
```
Should get 302 redirect to Authentik or 200 if authenticated.
---
## Post-deploy: How updates work
Code is volume-mounted from `/opt/watchtower/app/` into the container on Contabo. To update:
```bash
ssh zvx@100.64.0.1
cd /opt/watchtower
# Edit files or git pull
docker restart watchtower
```
No rebuild needed for code changes. Only rebuild (`docker compose up -d --build`) if `requirements.txt` or `Dockerfile` changes.
## Post-deploy: Adding a new collector
1. Copy `app/collectors/_example.py` to `app/collectors/myservice.py`
2. Edit the class: set `name`, `display_name`, implement `fetch()`
3. Add to `.env`: `MYSERVICE_ENABLED=true` plus any config vars
4. `docker restart watchtower`
The frontend auto-discovers the new panel. No HTML/JS/route edits needed.

242
projects/deploy livesync.md Normal file
View file

@ -0,0 +1,242 @@
# Deploying CouchDB with JWT auth for Obsidian LiveSync via Authentik
**LiveSync has native client-side JWT support that eliminates the need for a browser-based OIDC flow.** The plugin generates and signs JWTs internally using a stored private key, sending `Authorization: Bearer` headers directly to CouchDB. This fundamentally changes the architecture: instead of proxying OIDC tokens, you provision per-user key pairs, configure CouchDB with the public keys, and distribute setup URIs containing the private keys. Authentik serves as the identity backbone for a provisioning service — not as a runtime token issuer. No one has publicly documented a complete LiveSync + SSO deployment, making this guide a synthesis of the Kishieel Keycloak series, CouchDB JWT internals, Authentik's claim customization, and the LiveSync plugin's JWT implementation.
---
## CouchDB's JWT engine and the exact local.ini configuration
CouchDB 3.3+ includes a built-in JWT authentication handler requiring zero plugins. From Kishieel's Keycloak series and the official docs, here is the complete `local.ini`:
```ini
[couchdb]
single_node = true
[chttpd]
bind_address = 0.0.0.0
port = 5984
require_valid_user_except_for_up = true
authentication_handlers = {chttpd_auth, jwt_authentication_handler}, {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
[jwt_auth]
required_claims = exp,iat
roles_claim_path = _couchdb\.roles
[jwt_keys]
; EC key for LiveSync plugin (ES512 with P-521 curve)
ec:livesync-user1 = -----BEGIN PUBLIC KEY-----\nMHYwEAYHK...AzztRs\n-----END PUBLIC KEY-----\n
; RSA key from Authentik JWKS (for service/API access)
rsa:authentik-kid-here = -----BEGIN PUBLIC KEY-----\nMIIBIjAN...IDAQAB\n-----END PUBLIC KEY-----\n
[chttpd_auth]
secret = generate-a-long-random-secret-here
[cors]
origins = app://obsidian.md,capacitor://localhost,http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE
max_age = 3600
[admins]
admin = your-admin-password
```
**Critical details on `roles_claim_path`**: The backslash in `_couchdb\.roles` is mandatory. Without it, CouchDB interprets the dot as JSON nesting and looks for `{"_couchdb": {"roles": [...]}}` instead of the flat key `{"_couchdb.roles": [...]}`. This was a long-standing bug (issue #3176, #3758) that caused JWT roles to silently fail until the `roles_claim_path` syntax was added in CouchDB 3.3. The deprecated `roles_claim_name` setting did not have this problem but is ignored when `roles_claim_path` is set.
**Key format in `[jwt_keys]`** follows the pattern `{algorithm}:{kid} = {value}`. The algorithm prefix (`hmac:`, `rsa:`, `ec:`) is mandatory and prevents algorithm-confusion attacks. CouchDB reads the JWT header's `alg` claim to determine the prefix and the `kid` claim to select the specific key. If no `kid` is present in the JWT, CouchDB falls back to `{algorithm}:_default`. For asymmetric keys, the value is the PEM-encoded public key with literal `\n` replacing newlines. For HMAC, it's a base64-encoded secret. Since CouchDB 3.3, `=` characters in key names (common in base64 key IDs) are supported when the name-value separator uses spaces: `rsa:kid-with-base64= = -----BEGIN...`.
**On `required_claims`**: By default this is empty, meaning **CouchDB does not validate token expiration**. Always set `required_claims = exp` at minimum. The `sub` claim is always mandatory regardless of this setting and maps directly to the CouchDB username.
**Key rotation via the HTTP config API** takes effect immediately without restart:
```bash
curl -u admin:password -X PUT \
"http://localhost:5984/_node/_local/_config/jwt_keys/ec:new-kid" \
-H "Content-Type: text/plain" \
-d '"-----BEGIN PUBLIC KEY-----\nMHYw...\n-----END PUBLIC KEY-----\n"'
```
However, **CouchDB bug #5091** reports that `PUT /_node/{node}/_config/jwt_keys/{key}` returns HTTP 400 for valid PEM keys in some CouchDB versions. The workaround is writing keys to a `.ini` file in `/opt/couchdb/etc/local.d/` and restarting via `POST /_node/_local/_restart`. Changes to `local.ini` directly always require a restart; API-based changes do not.
---
## Kishieel's Keycloak pattern adapted for Authentik
Kishieel's two-part series provides the only complete, proven CouchDB + OIDC reference implementation. The architecture uses OpenResty (Nginx + Lua) as a proxy that performs OIDC authentication for browser clients and injects a Bearer token before forwarding to CouchDB. Here's how each component maps to the Authentik equivalent:
**Keycloak groups → Authentik groups with attributes**: Kishieel created Keycloak groups `/couchdb/admins` and `/couchdb/users` with a group attribute `_couchdb.roles` set to `["_admin"]` and `["_user"]` respectively. In Authentik, you'd create groups named `couchdb-admins` and `couchdb-users` with custom attributes `{"couchdb_role": "_admin"}` and `{"couchdb_role": "_user"}` respectively.
**Keycloak protocol mapper → Authentik scope mapping**: Kishieel used an `oidc-usermodel-attribute-mapper` with `claim.name: "_couchdb\\.roles"` (double-escaped backslash to produce a literal dot in the JWT claim). The mapper was `multivalued: true` and `aggregate.attrs: true` to collect roles from all groups. In Authentik, create a **Scope Mapping** under Customization → Property Mappings:
- **Name**: `CouchDB Roles`
- **Scope name**: `couchdb`
- **Expression**:
```python
return {
"_couchdb.roles": list(set(
str(g.attributes.get("couchdb_role"))
for g in request.user.ak_groups.all()
if "couchdb_role" in g.attributes
))
}
```
This iterates all user groups, extracts the `couchdb_role` attribute where it exists, deduplicates, and returns it as the `_couchdb.roles` claim. Values returned by scope mappings are added as custom claims to **both access tokens and ID tokens**.
**Keycloak client scope → Authentik OAuth2 provider scope**: Kishieel created a `couchdb` client scope containing the mapper, then assigned it as an optional scope on both the `couchdb-proxy` (confidential) and `couchdb-cli` (public) clients. In Authentik, assign the scope mapping to your OAuth2 provider's **Selected Scopes** list alongside `openid`, `profile`, and `email`. Check **"Include claims in id_token"** in the provider settings.
**Kishieel's Lua proxy script** (`access.lua`) is the key innovation. It uses `lua-resty-openidc` to perform the full OIDC authorization code flow for browser requests, then sets `Authorization: Bearer <access_token>` before proxying to CouchDB. Critically, the Part 2 update added an early return: if the request already has an `Authorization` header (from a CLI or API client), the Lua script skips the OIDC flow entirely. This dual-path design — browser SSO via proxy, direct Bearer token for programmatic access — is the pattern to replicate.
---
## LiveSync's native JWT: how the plugin signs its own tokens
The Obsidian LiveSync plugin has **built-in JWT generation** that changes the deployment model fundamentally. Instead of obtaining tokens from an IdP at runtime, the plugin stores a private key and signs short-lived JWTs client-side. The relevant plugin settings are:
| Setting | Type | Default | Purpose |
|---------|------|---------|---------|
| `useJWT` | boolean | `false` | Enable JWT authentication |
| `jwtAlgorithm` | string | `""` | JWT algorithm (e.g., `ES512`, `RS256`) |
| `jwtKey` | string | `""` | **Private key** in PEM format |
| `jwtKid` | string | `""` | Key ID matching CouchDB's `[jwt_keys]` entry |
| `jwtSub` | string | `""` | Subject claim → CouchDB username |
| `jwtExpDuration` | number | `5` | Token lifetime in minutes |
**Token lifecycle**: Tokens are cached and reused until **10% of the expiration duration remains or 10 seconds**, whichever is longer (capped at 1 minute maximum). With the default 5-minute expiration, tokens refresh at the 30-second mark. The plugin generates a new token by signing with the stored private key — no network call to an IdP.
**Key generation for ES512** (P-521 elliptic curve):
```bash
# Generate private key
openssl ecparam -genkey -name secp521r1 -noout -out private_key.pem
# Extract public key
openssl ec -in private_key.pem -pubout -out public_key.pem
```
The private key goes into the plugin's `jwtKey` setting. The public key (with newlines escaped as `\n`) goes into CouchDB's `[jwt_keys]` as `ec:<kid> = <pem>`.
**The setup URI gap**: The `generate_setupuri.ts` script (at `utils/flyio/generate_setupuri.ts`) only accepts basic auth parameters (`hostname`, `database`, `username`, `password`, `passphrase`). It does not support JWT settings. GitHub issue #729 documents this limitation. To generate a setup URI with JWT config, you must construct the full settings object (including `useJWT`, `jwtAlgorithm`, `jwtKey`, `jwtKid`, `jwtSub`, `jwtExpDuration`), encrypt it with a passphrase, and format it as `obsidian://setuplivesync?settings=[encrypted_data]`. The encryption mechanism uses passphrase-based AES encryption.
---
## Authentik configuration for service tokens and JWKS
**OAuth2 provider setup** at `auth.echo6.co`: Create an OAuth2/OIDC provider for the CouchDB service. Set the signing key to an RSA certificate (Authentik defaults to its self-signed certificate using RS256). The JWKS endpoint is at `https://auth.echo6.co/application/o/<app-slug>/jwks/` and the OpenID configuration at `https://auth.echo6.co/application/o/<app-slug>/.well-known/openid-configuration`.
**Token lifetimes** are configurable per-provider. The **access token defaults to 5 minutes** (format: `minutes=5`), refresh token to 30 days. For a provisioning service that generates long-lived tokens, extend to `hours=1` or more. The syntax accepts `hours=1,minutes=30,seconds=0`.
**Getting tokens programmatically** via `client_credentials`:
```bash
# Method 1: Client ID + Secret (auto-creates service account)
curl -X POST 'https://auth.echo6.co/application/o/token/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials' \
-d 'client_id=<client_id>' \
-d 'client_secret=<client_secret>' \
-d 'scope=openid profile couchdb'
# Method 2: Service account credentials
curl -X POST 'https://auth.echo6.co/application/o/token/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials' \
-d 'client_id=<client_id>' \
-d 'username=my-service-account' \
-d 'password=my-app-password-token' \
-d 'scope=openid profile couchdb'
```
Authentik supports `client_credentials`, `password` (ROPC — treated identically to client_credentials), `authorization_code`, `refresh_token`, `implicit`, and `urn:ietf:params:oauth:grant-type:device_code`. All endpoint URLs: token at `/application/o/token/`, authorize at `/application/o/authorize/`, device at `/application/o/device/`.
**JWKS-to-PEM conversion** for injecting Authentik's signing key into CouchDB is handled by the `couchdb-idp-updater` tool (GitHub: beyonddemise/couchdb-idp-updater). This NodeJS tool by Stephan Wissel periodically fetches the JWKS from an IdP's `.well-known/openid-configuration`, converts JWK keys to PEM format, and updates CouchDB's `[jwt_keys]` config. Due to bug #5091, it may need to write directly to an INI file rather than using the REST API. The manual conversion script (`jwks2couch.mjs`) is available as a GitHub gist. The process is: fetch JWKS → for each key, convert JWK to PEM using `jwk-to-pem` npm package → collapse newlines to `\n` → write to `[jwt_keys]` as `rsa:<kid> = <collapsed-pem>`.
---
## Per-database security without the _users database
**JWT users do not need to exist in CouchDB's `_users` database.** The user context is constructed entirely from the JWT: `sub` becomes the username, and the roles claim provides roles. CouchDB never queries `_users` during JWT authentication.
**The `_security` document** controls per-database access:
```json
{
"admins": {
"names": [],
"roles": ["_admin"]
},
"members": {
"names": ["specific-jwt-sub-value"],
"roles": []
}
}
```
Set this via:
```bash
curl -u admin:password -X PUT \
"http://localhost:5984/userdb-alice/_security" \
-H "Content-Type: application/json" \
-d '{"admins":{"names":[],"roles":["_admin"]},"members":{"names":["alice"],"roles":[]}}'
```
CouchDB matches the JWT `sub` against `members.names` and `admins.names`, and the JWT roles against `members.roles` and `admins.roles`. If `_security` has any members defined, only matching users can access the database. The `members` role grants read access to all documents and write access to non-design documents. The `admins` role additionally allows writing design documents and modifying `_security`.
**Do not use `couch_peruser` with JWT.** The Plexify article documents that CouchDB's built-in `couch_peruser` feature only auto-creates databases for admin users under JWT auth — requiring you to grant `_admin` to everyone, which is dangerous. Instead, create databases and set `_security` programmatically from a provisioning service using admin credentials.
---
## Proven deployment patterns and what breaks
**No one has publicly deployed LiveSync with full SSO end-to-end.** GitHub discussion #484 captures the core problem: *"For the auth, I use Authentik for my self hosted programs, however I am unsure if it will work with the obsidian extension since there is no user interface to login."* The plugin runs inside Obsidian's Electron shell — it cannot redirect to a browser for an OIDC login flow.
**Token expiration causes PouchDB replication failures.** When a JWT expires, CouchDB returns 401 with a `WWW-Authenticate: Basic` header, triggering an unwanted browser auth popup in Electron. The Plexify article documented this and recommended suppressing the header via reverse proxy or CouchDB config.
**CORS is the most common failure mode.** Issue #628 documents that LiveSync does not send the `Origin` header on non-preflight requests, causing CouchDB's CORS handler to omit `Access-Control-Allow-Origin` from responses. The fix is configuring CORS in `local.ini` (shown above) rather than relying on the reverse proxy alone. Required origins: `app://obsidian.md`, `capacitor://localhost`, `http://localhost`.
**The Caddy reverse proxy config** for `notes.echo6.co`:
```
notes.echo6.co {
reverse_proxy couchdb:5984
header {
Access-Control-Allow-Origin "app://obsidian.md"
Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
Access-Control-Allow-Headers "Content-Type, Authorization"
Access-Control-Allow-Credentials "true"
Access-Control-Max-Age "86400"
}
@options method OPTIONS
handle @options {
respond 204
}
}
```
---
## The recommended architecture for notes.echo6.co
Given the constraints — LiveSync can't do OIDC flows, but it can sign JWTs client-side — the architecture has three components:
**1. CouchDB container** at `notes.echo6.co` behind Caddy, configured with JWT auth handler, CORS, and per-user databases with `_security` documents.
**2. A provisioning service** (a small web app hosted on `forge.echo6.co` or as a Docker container) that:
- Is protected by Authentik forward auth (browser-based OIDC login)
- On first login, generates an EC key pair (ES512/P-521) for the user
- Creates a per-user CouchDB database (`userdb-<username>`)
- Sets the `_security` document to restrict access to that user's `sub`
- Injects the public key into CouchDB's `[jwt_keys]` via the config API or INI file
- Constructs and displays a setup URI containing all JWT settings (`useJWT: true`, `jwtAlgorithm: ES512`, `jwtKey: <private_key>`, `jwtKid: <kid>`, `jwtSub: <username>`, `jwtExpDuration: 5`)
- Encrypts the URI with a per-user passphrase and presents it as a clickable `obsidian://setuplivesync?settings=[...]` link
**3. couchdb-idp-updater sidecar** (optional, only needed if you also want Authentik-issued JWTs accepted directly by CouchDB for API access). This periodically syncs Authentik's JWKS public keys into CouchDB's config.
The provisioning service is the critical custom component. It bridges the gap between Authentik's identity management and LiveSync's key-based JWT model. Users authenticate once through their browser via Authentik SSO, receive their setup URI, paste it into Obsidian, and from that point the plugin handles all authentication autonomously by signing its own tokens.
## Conclusion
The deployment hinges on a non-obvious insight: **LiveSync's JWT support is self-contained, not IdP-dependent**. The plugin signs tokens locally using a stored private key, which means the OIDC provider's role shifts from runtime token issuer to user provisioning backbone. CouchDB's `roles_claim_path = _couchdb\.roles` with the escaped dot, `required_claims = exp,iat`, and per-kid key entries in `[jwt_keys]` form the server-side foundation. The Kishieel blog's Lua proxy pattern remains valuable for browser-based CouchDB admin access but is unnecessary for the Obsidian plugin itself. The main engineering work is building the provisioning service that generates key pairs, configures CouchDB databases, and outputs encrypted setup URIs — a task well-suited to a Claude Code automation prompt targeting Docker Compose on Contabo with Caddy as the edge proxy.

406
projects/headscale-full-deployment.md Normal file
View file

@ -0,0 +1,406 @@
# Headscale Full Deployment Runbook
## Nodes + Headplane + Authentik OIDC
**Headscale location:** `/opt/headscale-vanilla`
**Container name:** `headscale-vanilla`
**Domain:** `vpn.echo6.co`
**Auth key:** `hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ`
---
## PHASE 1: REGISTER CONTABO (must be first)
```bash
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname contabo --force-reauth
```
Verify:
```bash
docker exec headscale-vanilla headscale nodes list
```
**STOP if contabo doesn't appear. Do not continue.**
---
## PHASE 2: REGISTER ALL LXC/CT NODES
SSH into each container. For each one:
```bash
# Check if tailscale is installed
which tailscale || echo "NOT INSTALLED"
# Install if missing
curl -fsSL https://tailscale.com/install.sh | sh
```
Then register. **Do them in this exact order for sequential IPs:**
```bash
# utility (will get 100.64.0.2)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname utility --force-reauth
# data (will get 100.64.0.3)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname data --force-reauth
# cloud (will get 100.64.0.4)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname cloud --force-reauth
# media (will get 100.64.0.5)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname media --force-reauth
# aida-nebra (will get 100.64.0.6)
tailscale up --login-server https://vpn.echo6.co \
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ \
--hostname aida-nebra --force-reauth
```
After each, verify from Contabo:
```bash
docker exec headscale-vanilla headscale nodes list
```
---
## PHASE 3: REGISTER DESKTOP + PHONES
**Desktop (Windows — PowerShell as Admin):**
```powershell
tailscale up --login-server https://vpn.echo6.co `
--auth-key hskey-auth-LOd5lzxvsHaP-GP9K6QkG6UW60UFeoDbKv5OxR9yJXupFvfy-Ps_SGmYu5QxG5g-I7JsVDEebZpVJ `
--hostname desktop --force-reauth
```
**Phones:**
- Open Tailscale app → Settings → Account
- Log out if needed
- Use "Custom coordination server" or "Alternate server"
- Enter: `https://vpn.echo6.co`
- Should auto-register with the tailnet
If the app doesn't support custom servers natively, you may need the F-Droid build on Android or the CLI on a jailbroken iOS device.
---
## PHASE 4: VERIFY ALL NODES + TEST CONNECTIVITY
```bash
docker exec headscale-vanilla headscale nodes list
```
Expected output: all nodes with sequential 100.64.0.x IPs.
Test from any node:
```bash
tailscale ping contabo
tailscale ping data
tailscale ping utility
```
Test magic DNS:
```bash
ping data.echo6.mesh
ping utility.echo6.mesh
```
---
## PHASE 5: BACKUP THE DATABASE (do this NOW before anything else)
```bash
mkdir -p /opt/headscale-vanilla/backups
# Immediate backup
sqlite3 /opt/headscale-vanilla/data/db.sqlite \
".backup '/opt/headscale-vanilla/backups/db-$(date +%Y%m%d-%H%M).sqlite'"
# Set up cron for automatic backups every 6 hours, 7-day retention
crontab -e
# Add this line:
0 */6 * * * sqlite3 /opt/headscale-vanilla/data/db.sqlite ".backup '/opt/headscale-vanilla/backups/db-$(date +\%Y\%m\%d-\%H\%M).sqlite'" && find /opt/headscale-vanilla/backups -name "db-*.sqlite" -mtime +7 -delete
```
---
## PHASE 6: PERSISTENCE TEST
```bash
cd /opt/headscale-vanilla
docker compose down
sleep 5
ls -la /opt/headscale-vanilla/data/db.sqlite*
docker compose up -d
sleep 10
docker exec headscale-vanilla headscale nodes list
```
**Every node must survive. If any are missing, STOP and report.**
---
## PHASE 7: CREATE AUTHENTIK OIDC PROVIDER FOR HEADSCALE
This lets Tailscale clients authenticate via Authentik instead of preauth keys.
1. Log into Authentik admin panel
2. Go to **Applications → Applications → Create with Provider**
3. Configure:
- **Application name:** Headscale
- **Slug:** `headscale` (remember this — it's part of the issuer URL)
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** default-provider-authorization-implicit-consent (or explicit if you want)
- **Redirect URI (Strict):** `https://vpn.echo6.co/oidc/callback`
- **Signing key:** Select any available key
- **Scopes:** Ensure these scope mappings are selected:
- `openid`
- `profile`
- `email`
- **`offline_access`** ← CRITICAL — without this, nodes break on Headscale restart
4. Note the **Client ID** and **Client Secret**
5. Click Submit
---
## PHASE 8: CONFIGURE HEADSCALE OIDC
Edit `/opt/headscale-vanilla/config.yaml` — add this OIDC block:
```yaml
oidc:
only_start_if_oidc_is_available: true
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headscale/"
client_id: "<Client ID from Authentik>"
client_secret: "<Client Secret from Authentik>"
scope: ["openid", "profile", "email", "offline_access"]
pkce:
enabled: true
method: S256
strip_email_domain: true
```
Replace:
- `<YOUR_AUTHENTIK_DOMAIN>` with your Authentik domain (e.g., `auth.echo6.co`)
- `<Client ID from Authentik>` with the actual client ID
- `<Client Secret from Authentik>` with the actual client secret
Restart Headscale:
```bash
cd /opt/headscale-vanilla
docker compose restart
sleep 10
docker logs headscale-vanilla 2>&1 | tail -20
```
**Check logs for OIDC errors. If it fails to start, remove the OIDC block and restart.**
Test: From any node, run:
```bash
tailscale up --login-server https://vpn.echo6.co --force-reauth
```
It should open a browser → Authentik login → back to terminal, authenticated.
**Your existing preauth-key nodes still work. OIDC is for NEW registrations and re-auths.**
---
## PHASE 9: CREATE AUTHENTIK OIDC PROVIDER FOR HEADPLANE
This is a SECOND application in Authentik for the web UI login.
1. Go to **Applications → Applications → Create with Provider**
2. Configure:
- **Application name:** Headplane
- **Slug:** `headplane`
- **Provider type:** OAuth2/OpenID Connect
- **Authorization flow:** Same as before
- **Redirect URI (Strict):** `https://vpn.echo6.co/admin/oidc/callback`
- **Signing key:** Same key
- **Scopes:** `openid`, `profile`, `email`
3. Note the **Client ID** and **Client Secret** (different from Headscale's)
4. Click Submit
---
## PHASE 10: GENERATE HEADSCALE API KEY FOR HEADPLANE
```bash
docker exec headscale-vanilla headscale apikeys create --expiration 999d
```
**Save this key — you need it for the Headplane config.**
---
## PHASE 11: CREATE HEADPLANE CONFIG
```bash
# Generate a cookie secret
openssl rand -hex 16
```
Write `/opt/headscale-vanilla/headplane-config.yaml`:
```yaml
server:
host: "0.0.0.0"
port: 3000
cookie_secret: "<OUTPUT_OF_OPENSSL_RAND_HEX_16>"
cookie_secure: true
data_path: "/var/lib/headplane"
headscale:
url: "http://headscale-vanilla:8080"
config_path: "/etc/headscale/config.yaml"
config_strict: false
oidc:
issuer: "https://<YOUR_AUTHENTIK_DOMAIN>/application/o/headplane/"
client_id: "<Headplane Client ID from Authentik>"
client_secret: "<Headplane Client Secret from Authentik>"
token_endpoint_auth_method: "client_secret_post"
headscale_api_key: "<API_KEY_FROM_PHASE_10>"
redirect_uri: "https://vpn.echo6.co/admin/oidc/callback"
disable_api_key_login: false
integration:
docker:
enabled: true
container_name: "headscale-vanilla"
socket: "/var/run/docker.sock"
```
Replace all `<PLACEHOLDERS>` with actual values.
---
## PHASE 12: ADD HEADPLANE TO DOCKER COMPOSE
Edit `/opt/headscale-vanilla/docker-compose.yml` — add the headplane service:
```yaml
services:
headscale:
# ... your existing headscale service, don't change it ...
headplane:
image: ghcr.io/tale/headplane:latest
container_name: headplane
restart: unless-stopped
depends_on:
- headscale
ports:
- "127.0.0.1:3000:3000"
volumes:
- ./headplane-config.yaml:/etc/headplane/config.yaml:ro
- ./headplane-data:/var/lib/headplane
- ./config.yaml:/etc/headscale/config.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
```
Start it:
```bash
cd /opt/headscale-vanilla
docker compose up -d
sleep 10
docker logs headplane 2>&1 | tail -20
```
Check for errors. Common issues:
- "OIDC configuration is incomplete" → double-check all OIDC values in headplane-config.yaml
- Can't connect to headscale → ensure `url` matches the container name and internal port
- Docker socket permission denied → check that the headplane container can read /var/run/docker.sock
---
## PHASE 13: UPDATE CADDY FOR HEADPLANE
Add the `/admin` route to your Caddy config for `vpn.echo6.co`:
```
vpn.echo6.co {
handle /admin* {
reverse_proxy 127.0.0.1:3000
}
handle {
reverse_proxy 127.0.0.1:8084
}
}
```
Restart Caddy:
```bash
# Wherever your Caddy lives — adjust path as needed
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```
---
## PHASE 14: TEST HEADPLANE
1. Browse to `https://vpn.echo6.co/admin`
2. You should see the Headplane login page
3. Click "Sign in with OIDC" → redirects to Authentik → authenticate
4. **The FIRST user to log in gets Owner permissions**
5. Verify you can see all your nodes in the UI
If OIDC fails, you can still log in with the API key (that's why we set `disable_api_key_login: false`).
---
## PHASE 15: FINAL VERIFICATION
Run all of these from Contabo:
```bash
# All nodes present?
docker exec headscale-vanilla headscale nodes list
# Both containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"
# Headplane accessible?
curl -s -o /dev/null -w "%{http_code}" https://vpn.echo6.co/admin
# Should return 200 or 302
# Database backed up?
ls -la /opt/headscale-vanilla/backups/
# Cron running?
crontab -l | grep sqlite3
```
---
## REPORT TEMPLATE
After each phase, report:
```
Phase X complete:
- Output of headscale nodes list:
- Any errors:
- Logs (last 10 lines):
```
**Do NOT skip phases. Do NOT combine phases. If something fails, stop and report.**
---
## KNOWN GOTCHAS
1. **offline_access scope** — If you forget this in Authentik, nodes lose auth after Headscale restarts
2. **config_strict: false** — Headscale 0.28.0 has config options Headplane may not recognize
3. **Headplane needs Docker socket** — For the integration that lets it restart Headscale when you change settings
4. **First OIDC login = Owner** — Don't let random people hit your Headplane URL before you log in first
5. **Phones may not support custom servers** — Android F-Droid build is more flexible; iOS is limited
6. **Two separate OIDC apps** — Headscale and Headplane each need their own application in Authentik with different redirect URIs

60
projects/last-ember-project.md Normal file
View file

@ -0,0 +1,60 @@
# Last Ember — MMUD Web Dashboard
> **CONSOLIDATED:** Last Ember has been merged into the MMUD repo at `src/web/`.
> The standalone repo at `/home/zvx/projects/last-ember` is archived (deprecation notice committed).
Spectator dashboard and admin panel for MMUD (Mesh Multi-User Dungeon). Flask web app that reads from the game's SQLite database. Themed as The Last Ember — the bar that never changes.
## Status
**Phase:** Consolidated into MMUD. No longer a separate project.
## Location
- **Code:** `/home/zvx/projects/mmud/src/web/`
- **Archived repo:** `/home/zvx/projects/last-ember` (deprecated, read-only reference)
## Relationship to MMUD
Last Ember now runs **in-process** with the MMUD mesh daemon as a background daemon thread. Same process, same DB file, WAL mode handles concurrency. Flask starts automatically unless `--no-web` is passed.
## Key Files
- `src/web/__init__.py` — Flask app factory (`create_app`)
- `src/web/config.py` — Web-specific settings (host, port, secret, polling intervals)
- `src/web/routes/` — public.py, api.py, admin.py (session auth)
- `src/web/services/` — gamedb.py, dashboard.py, chronicle.py, admin_service.py
- `src/web/templates/` — Jinja2 templates (dark tavern aesthetic)
- `src/web/static/` — ember.css (design system), embers.js (particles), app.js (AJAX)
- `src/web/prototypes/` — Original HTML design references (visual source of truth)
- `src/db/migrations/004_web_tables.sql` — Web tables (node_config, admin_log, banned_players, npc_journals)
## Stack
- Python 3.11+, Flask 3.x, Jinja2
- SQLite WAL mode (read-only public, read-write admin)
- No build step, no React, no SPA. Server-rendered templates with AJAX polling.
- Docker (python:3.11-slim, /data volume for SQLite)
## CLI
- `--web-port PORT` — override dashboard port (default: 5000)
- `--no-web` — disable web dashboard entirely
- `MMUD_WEB_PORT`, `MMUD_WEB_HOST`, `MMUD_WEB_SECRET`, `MMUD_ADMIN_PASSWORD` env vars
## Pages
**Public (no auth):**
- Main dashboard — live epoch status, leaderboard, broadcasts, bounties, mode status, secrets
- Chronicle — epoch history, NPC daily journals (Grist, Maren, Torval, Whisper)
- How to Play — game guide, command reference
**API (JSON, polled by frontend):**
- `/api/status` (30s), `/api/broadcasts` (15s), `/api/bounties`, `/api/mode`, `/api/leaderboard`
**Admin (session auth):**
- Dashboard — active players, epoch day, node health
- Nodes — assign Meshtastic node IDs to 6 sim node roles
- Players — view, ban, kick, reset
- Epoch — force advance day, force Breach, manual broadcast
- System — DB stats, node config, admin log

469
projects/matrix-synapse-deployment.md Normal file
View file

@ -0,0 +1,469 @@
# Matrix Synapse Deployment
**Status:** Deployed 2026-02-15, migrated to Contabo 2026-02-15
**Target:** Contabo VPS (5.189.158.149 / 100.64.0.1)
**URLs:** https://matrix.echo6.co (Synapse), https://element.echo6.co (Element Web)
**Server Name:** echo6.co (federated identity: @user:echo6.co)
---
## Architecture
| Component | Detail |
|-----------|--------|
| Host | Contabo VPS (5.189.158.149 / 100.64.0.1) |
| Docker services | Synapse (127.0.0.1:8008), Element Web (127.0.0.1:8088), PostgreSQL 16 |
| Reverse proxy | Contabo Caddy (auto ACME certs) |
| SSO | Authentik OIDC → communication-users group |
| Federation | Well-known delegation on echo6.co base domain (served by utility Caddy) |
| Compose path | `/opt/matrix/docker-compose.yml` |
| Backup | Daily at 3AM, 14-day retention, `/opt/matrix/backups/` |
The server name is `echo6.co` (not `matrix.echo6.co`) so federated user IDs are `@user:echo6.co`. The Synapse instance lives at `matrix.echo6.co` and delegation is handled via `.well-known` endpoints on the base domain.
---
## Step 1: Provision LXC Container
Run **ct-runbook.md** on the utility node with these parameters:
```
CTID=108
HOSTNAME=matrix
STORAGE=local-lvm
DISK_SIZE=16
MEMORY=2048
CORES=2
BRIDGE=vmbr0
```
After the runbook completes (user, SSH, Docker, Tailscale all verified), continue here.
---
## Step 2: Create Project Structure
SSH into CT 108:
```bash
CT_IP=$(ssh root@192.168.1.241 "pct exec 108 -- hostname -I | awk '{print \$1}'")
sshpass -p '7redditGold' ssh zvx@$CT_IP
```
Create directories:
```bash
sudo mkdir -p /opt/matrix/{synapse,postgres,element,backups,scripts}
sudo chown -R zvx:zvx /opt/matrix
```
---
## Step 3: Create Docker Compose
Create `/opt/matrix/docker-compose.yml`:
```yaml
services:
postgres:
image: postgres:16-alpine
container_name: matrix-postgres
restart: unless-stopped
environment:
POSTGRES_DB: synapse
POSTGRES_USER: synapse
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_INITDB_ARGS: "--encoding=UTF8 --lc-collate=C --lc-ctype=C"
volumes:
- ./postgres:/var/lib/postgresql/data
networks:
- matrix-net
healthcheck:
test: ["CMD-SHELL", "pg_isready -U synapse -d synapse"]
interval: 10s
timeout: 5s
retries: 5
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix-synapse
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
environment:
SYNAPSE_CONFIG_PATH: /data/homeserver.yaml
volumes:
- ./synapse:/data
ports:
- "8008:8008"
networks:
- matrix-net
element:
image: vectorim/element-web:latest
container_name: matrix-element
restart: unless-stopped
volumes:
- ./element/config.json:/app/config.json:ro
ports:
- "8080:80"
networks:
- matrix-net
networks:
matrix-net:
driver: bridge
```
Create `/opt/matrix/.env`:
```bash
POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 32)
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" > /opt/matrix/.env
chmod 600 /opt/matrix/.env
echo "Save this password to /home/zvx/projects/.ref/credentials"
cat /opt/matrix/.env
```
---
## Step 4: Generate Synapse Config
```bash
cd /opt/matrix
docker run -it --rm \
-v ./synapse:/data \
-e SYNAPSE_SERVER_NAME=echo6.co \
-e SYNAPSE_REPORT_STATS=no \
matrixdotorg/synapse:latest generate
```
Edit `synapse/homeserver.yaml` — replace the full `database` section and add OIDC config:
```yaml
server_name: "echo6.co"
public_baseurl: "https://matrix.echo6.co/"
listeners:
- port: 8008
type: http
tls: false
x_forwarded: true
bind_addresses: ['0.0.0.0']
resources:
- names: [client, federation]
compress: false
database:
name: psycopg2
args:
user: synapse
password: <POSTGRES_PASSWORD from .env>
database: synapse
host: matrix-postgres
port: 5432
cp_min: 5
cp_max: 10
media_store_path: /data/media_store
enable_registration: false
url_preview_enabled: true
# Authentik OIDC — fill client_id and client_secret after running authentik-oidc-application.md
oidc_providers:
- idp_id: authentik
idp_name: "Echo6 SSO"
discover: true
issuer: "https://auth.echo6.co/application/o/matrix/"
client_id: "<from authentik-oidc-application.md>"
client_secret: "<from authentik-oidc-application.md>"
scopes: ["openid", "profile", "email"]
user_mapping_provider:
config:
localpart_template: "{{ user.preferred_username }}"
display_name_template: "{{ user.name }}"
```
---
## Step 5: Configure Element Web
Create `/opt/matrix/element/config.json`:
```json
{
"default_server_config": {
"m.homeserver": {
"base_url": "https://matrix.echo6.co",
"server_name": "echo6.co"
}
},
"brand": "Echo6 Chat",
"disable_guests": true,
"disable_3pid_login": false
}
```
---
## Step 6: Start Services
```bash
cd /opt/matrix
docker compose up -d
docker compose ps
```
Wait for Synapse to initialize the database (watch logs):
```bash
docker compose logs -f synapse
# Wait for "Synapse now listening on TCP port 8008"
```
---
## Step 7: Expose via Caddy and DNS
Run **expose-service-home.md** twice — once for `matrix.echo6.co` and once for `element.echo6.co`.
This service has OIDC, so use local IP per the runbook's decision table.
### matrix.echo6.co
- Backend: `192.168.1.108:8008` (local IP, has OIDC)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS
Caddy site block (note the path-based routing for Matrix):
```caddyfile
matrix.echo6.co {
tls /etc/caddy/certs/matrix.echo6.co.fullchain.crt /etc/caddy/certs/matrix.echo6.co.key
reverse_proxy /_matrix/* 192.168.1.108:8008
reverse_proxy /_synapse/* 192.168.1.108:8008
}
```
### element.echo6.co
- Backend: `192.168.1.108:8080` (local IP)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS
```caddyfile
element.echo6.co {
tls /etc/caddy/certs/element.echo6.co.fullchain.crt /etc/caddy/certs/element.echo6.co.key
reverse_proxy 192.168.1.108:8080
}
```
### Well-known delegation (federation)
This must go on the `echo6.co` base domain. Check if there's already an `echo6.co` block in the Utility Caddy Caddyfile — if so, merge these `handle` directives into it. If not, add a new block:
```caddyfile
echo6.co {
tls /etc/caddy/certs/echo6.co.fullchain.crt /etc/caddy/certs/echo6.co.key
handle /.well-known/matrix/server {
header Content-Type application/json
respond `{"m.server": "matrix.echo6.co:443"}`
}
handle /.well-known/matrix/client {
header Content-Type application/json
header Access-Control-Allow-Origin *
respond `{"m.homeserver": {"base_url": "https://matrix.echo6.co"}}`
}
# ... any existing handlers for echo6.co ...
}
```
If `echo6.co` doesn't have a cert yet, issue one via acme.sh following the same pattern in expose-service-home.md.
### dnsmasq split DNS
Add to `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo:
```
address=/matrix.echo6.co/100.64.0.8
address=/element.echo6.co/100.64.0.8
```
Both point to the Utility Caddy Tailscale IP (100.64.0.8), which proxies to CT 108.
Restart dnsmasq:
```bash
ssh root@100.64.0.1 "systemctl restart dnsmasq"
```
---
## Step 8: Configure Authentik SSO
Run **authentik-oidc-application.md** with these inputs:
```
SERVICE_NAME=Matrix
SERVICE_SLUG=matrix
SERVICE_URL=https://matrix.echo6.co
OIDC_CALLBACK_PATH=/_synapse/client/oidc/callback
NEEDS_OFFLINE_ACCESS=no
CLIENT_TYPE=confidential
```
After completing the runbook, take the Client ID and Client Secret and update `synapse/homeserver.yaml` (Step 4) with the real values. Then restart Synapse:
```bash
cd /opt/matrix && docker compose restart synapse
```
---
## Step 9: Bind Access Group
Run **authentik-access-groups.md** Procedure B to bind the `matrix` application to the `communication-users` group.
```
APP_SLUG=matrix
GROUP_PK=31bce176-cd86-4aea-8db3-a57e03d5c2d1 # communication-users
```
This shares the same access group as Mailcow.
---
## Step 10: Create Admin User
```bash
docker exec -it matrix-synapse register_new_matrix_user \
-u matt \
-p <secure-password> \
-a \
-c /data/homeserver.yaml \
http://localhost:8008
```
---
## Step 11: Schedule PostgreSQL Backups
Run **pg-backup.md** with these inputs:
```
CONTAINER_NAME=matrix-postgres
DB_NAME=synapse
DB_USER=synapse
BACKUP_DIR=/opt/matrix/backups
RETENTION_DAYS=14
CRON_SCHEDULE="0 3 * * *"
```
---
## Verification
### Internal (from CT 108)
```bash
curl -s http://localhost:8008/_matrix/client/versions | jq .
curl -s http://localhost:8008/_matrix/federation/v1/version | jq .
curl -s http://localhost:8080 | head -5
```
### External (from cortex or any tailnet device)
```bash
curl -s https://matrix.echo6.co/_matrix/client/versions | jq .
curl -s https://matrix.echo6.co/_matrix/federation/v1/version | jq .
curl -sI https://element.echo6.co | head -5
curl -s https://echo6.co/.well-known/matrix/server | jq .
curl -s https://echo6.co/.well-known/matrix/client | jq .
```
### Federation
```bash
curl -s "https://federationtester.matrix.org/api/report?server_name=echo6.co" | jq '.FederationOK'
```
Must return `true`.
### SSO
1. Open https://element.echo6.co
2. Click SSO login
3. Should redirect to auth.echo6.co → authenticate → redirect back to Element
4. Verify user identity matches Authentik profile
---
## Troubleshooting
### Synapse won't start
```bash
docker compose logs synapse 2>&1 | tail -50
```
Common causes: bad YAML indentation in homeserver.yaml, wrong PostgreSQL password, database not ready.
### Federation test fails
Check in order:
1. `.well-known/matrix/server` returns `{"m.server": "matrix.echo6.co:443"}`
2. `/_matrix/federation/v1/version` is accessible from the public internet
3. Caddy is routing `/_matrix/*` paths correctly (not just root)
4. GoDaddy DNS for `echo6.co` points to 199.6.36.163
### SSO login loop
See troubleshooting in authentik-oidc-application.md. Most common cause: missing signing key on the Authentik provider, or wrong callback path.
### Element can't connect
Verify Element's `config.json` has `base_url` set to `https://matrix.echo6.co` (not `http://`, not `localhost`).
---
## Runbook References
| Step | Runbook | Purpose |
|------|---------|---------|
| 1 | ct-runbook.md | LXC provisioning, Docker, user, SSH, Tailscale |
| 7 | expose-service-home.md | SSL cert, Caddy site block, GoDaddy DNS |
| 8 | authentik-oidc-application.md | Create OIDC provider + application |
| 9 | authentik-access-groups.md | Bind communication-users group |
| 11 | pg-backup.md | Scheduled PostgreSQL backup with retention |
---
## Credentials Reference
Store in `/home/zvx/projects/.ref/credentials`:
```
# Matrix Synapse
MATRIX_POSTGRES_PASSWORD=<from .env>
MATRIX_OIDC_CLIENT_ID=<from authentik-oidc-application.md>
MATRIX_OIDC_CLIENT_SECRET=<from authentik-oidc-application.md>
MATRIX_OIDC_ISSUER=https://auth.echo6.co/application/o/matrix/
MATRIX_ADMIN_USER=matt
```
---
## Post-Deploy Updates
After deployment, update these docs:
- `docs/services/services.md` — add Matrix entry
- `docs/software/caddy.md` — add matrix.echo6.co and element.echo6.co site blocks
- `docs/software/dns.md` — note well-known delegation on echo6.co
- `docs/hardware/environment.md` — add CT 108 to LXC table and Headscale node list
- `runbooks/authentik-access-groups.md` — add Matrix to application bindings table
---
*Created: 2026-02-15*

561
projects/meshtastic-headscale-runbook.md Normal file
View file

@ -0,0 +1,561 @@
# IdahoMesh Tailnet Runbook
## Overview
Stand up a dedicated Headscale instance for the IdahoMesh Meshtastic network, separate from Echo6. This tailnet will be shared between Echo6 (via a one-way bridge LXC) and Sidpatchy (direct join). Nebra CM3 gateways register directly on this Headscale.
### Architecture
```
Echo6 Headscale (100.64.0.x)
↓ (one-way only)
[Bridge LXC] ← dual tailscaled, NAT + firewall
↓
IdahoMesh Headscale (100.100.0.x)
↕ ↕
Nebra CM3s Sidpatchy's devices
```
> **Security:** The bridge is one-way. Echo6 can reach Meshtastic devices, but Meshtastic devices (including Sidpatchy) CANNOT reach back into Echo6. NAT masquerades the source and iptables drops inbound initiation.
### IP Allocation
| Tailnet | Prefix | Notes |
|-------------|------------------|------------------------------------------|
| Echo6 | 100.64.0.0/10 | Existing, do not change |
| IdahoMesh | 100.100.0.0/16 | Within Tailscale's required 100.64.0.0/10 supernet |
### Infrastructure
| Component | VMID | Host | Local IP | Purpose |
|-----------|------|------|----------|---------|
| meshtastic-hs | CT 106 | utility | 192.168.1.106 | IdahoMesh Headscale server |
| mesh-bridge | CT 107 | utility | 192.168.1.107 | One-way bridge between tailnets |
---
## Phase 1: IdahoMesh Headscale Instance
### 1.1 Create the LXC on utility
```bash
ssh root@192.168.1.241
pct create 106 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
--hostname meshtastic-hs \
--memory 512 \
--cores 1 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.106/24,gw=192.168.1.1 \
--storage local-lvm \
--rootfs local-lvm:4 \
--unprivileged 1 \
--onboot 1 \
--start 1
```
Bootstrap standard packages:
```bash
echo6-bootstrap-ct.sh 106
```
### 1.2 Install Headscale
```bash
pct exec 106 -- bash -c '
apt update && apt install -y curl
HEADSCALE_VERSION="0.28.0"
curl -Lo /usr/local/bin/headscale \
"https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64"
chmod +x /usr/local/bin/headscale
mkdir -p /etc/headscale /var/lib/headscale /var/run/headscale
'
```
### 1.3 Configure Headscale
Create `/etc/headscale/config.yaml`:
```yaml
server_url: https://vpn.idahomesh.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false
noise:
private_key_path: /var/lib/headscale/noise_private.key
prefixes:
v4: 100.100.0.0/16
v6: fd7a:115c:a1e0:ab00::/56
allocation: sequential
derp:
server:
enabled: false
urls:
- https://controlplane.tailscale.com/derpmap/default
paths: []
auto_update_enabled: true
update_frequency: 3h
disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m
database:
type: sqlite
debug: false
gorm:
prepare_stmt: true
parameterized_queries: true
skip_err_record_not_found: true
slow_threshold: 1000
sqlite:
path: /var/lib/headscale/db.sqlite
write_ahead_log: true
wal_autocheckpoint: 1000
policy:
mode: file
path: /etc/headscale/acl.json
dns:
magic_dns: true
base_domain: mesh.local
override_local_dns: true
nameservers:
global:
- 1.1.1.1
- 9.9.9.9
split: {}
search_domains: []
extra_records: []
unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"
logtail:
enabled: false
randomize_client_port: false
log:
level: info
format: text
```
> **Note:** Embedded DERP is disabled — we use Tailscale's public DERP relays. The server is behind Caddy, so TLS termination happens at the reverse proxy.
### 1.4 Create the ACL Policy
Create `/etc/headscale/acl.json`:
```json
{
"groups": {
"group:malice": ["malice@"],
"group:sidpatchy": ["sidpatchy@"],
"group:nebra": ["nebra@"]
},
"acls": [
{
"action": "accept",
"src": ["group:nebra"],
"dst": ["group:nebra:*"],
"comment": "Nebra gateways talk to each other"
},
{
"action": "accept",
"src": ["group:malice"],
"dst": ["group:nebra:*"],
"comment": "Echo6 bridge can reach Nebras"
},
{
"action": "accept",
"src": ["group:sidpatchy"],
"dst": ["group:nebra:*"],
"comment": "Sidpatchy can reach Nebras"
},
{
"action": "accept",
"src": ["group:nebra"],
"dst": ["group:malice:*", "group:sidpatchy:*"],
"comment": "Nebras can respond back to both"
}
]
}
```
> **Important:** Headscale v0.28.0 requires usernames in ACL groups to have `@` suffix (e.g., `malice@`). The `--user` flag on CLI commands takes user IDs (integers), not names.
>
> **No malice↔Sidpatchy rules.** They can only see each other's Nebra traffic. The bridge firewall provides additional isolation (see Phase 2.6).
### 1.5 Create systemd Service
Create `/etc/systemd/system/headscale.service`:
```ini
[Unit]
Description=Headscale - IdahoMesh Tailnet
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
```bash
systemctl daemon-reload
systemctl enable --now headscale
systemctl status headscale
```
### 1.6 Create Users and Preauthkeys
```bash
headscale users create echo6
headscale users create sidpatchy
headscale users create nebra
# For the bridge LXC (your side)
headscale preauthkeys create --user echo6 --expiration 24h
# Save this key ^^^
# For Sidpatchy — send this to him
headscale preauthkeys create --user sidpatchy --expiration 72h
# Save this key ^^^
# For Nebra CM3 gateways (reusable so all Nebras use same key)
headscale preauthkeys create --user nebra --reusable --expiration 8760h
# Save this key ^^^
```
---
## Phase 2: Bridge LXC (CT 107 on utility)
This LXC lives on Echo6's network and runs two tailscaled instances — one on Echo6, one on IdahoMesh. Traffic flows **one-way only**: Echo6 → IdahoMesh.
### 2.1 Create the LXC
```bash
ssh root@192.168.1.241
pct create 107 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
--hostname mesh-bridge \
--memory 256 \
--cores 1 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.107/24,gw=192.168.1.1 \
--storage local-lvm \
--rootfs local-lvm:2 \
--unprivileged 1 \
--features nesting=1 \
--onboot 1 \
--start 1
```
Add TUN device access for Tailscale (on the Proxmox host):
```bash
# Stop the container first
pct stop 107
cat >> /etc/pve/lxc/107.conf << 'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 107
```
### 2.2 Install Tailscale
```bash
pct exec 107 -- bash -c '
curl -fsSL https://tailscale.com/install.sh | sh
'
```
### 2.3 Set Up Dual tailscaled
Create directories for the second instance:
```bash
pct exec 107 -- bash -c '
mkdir -p /var/lib/tailscale-meshtastic /var/run/tailscale-meshtastic
'
```
The default tailscaled service handles Echo6. Create a second service for IdahoMesh:
Create `/etc/systemd/system/tailscaled-meshtastic.service`:
```ini
[Unit]
Description=Tailscale daemon (IdahoMesh tailnet)
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/sbin/tailscaled \
--state=/var/lib/tailscale-meshtastic/tailscaled.state \
--socket=/var/run/tailscale-meshtastic/tailscaled.sock \
--port=41642 \
--tun=tailscale1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
```bash
systemctl daemon-reload
systemctl enable --now tailscaled-meshtastic
```
### 2.4 Enable IP Forwarding
```bash
cat <<EOF > /etc/sysctl.d/99-bridge.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl -p /etc/sysctl.d/99-bridge.conf
```
### 2.5 Join Both Tailnets
```bash
# Join Echo6 (default tailscaled instance)
# Advertise IdahoMesh range so Echo6 devices can route to Meshtastic nodes
tailscale up \
--login-server=https://vpn.echo6.co \
--advertise-routes=100.100.0.0/16 \
--accept-routes
# Join IdahoMesh (second instance)
# Do NOT advertise Echo6 routes — one-way only
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock up \
--login-server=https://vpn.idahomesh.com \
--authkey=<echo6-preauthkey-from-step-1.6> \
--accept-routes
```
After joining, approve the advertised route on Echo6 Headscale only:
```bash
# On Echo6 Headscale (Contabo) — enable the 100.100.0.0/16 route
docker exec headscale-vanilla headscale routes list
docker exec headscale-vanilla headscale routes enable -r <route-id>
# NO route approval needed on IdahoMesh Headscale — nothing is advertised
```
### 2.6 Configure One-Way Firewall and NAT
This is the critical security step. Echo6 can reach IdahoMesh devices, but nothing on IdahoMesh can reach back into Echo6.
Install iptables:
```bash
apt install -y iptables iptables-persistent
```
Apply rules:
```bash
# NAT: Masquerade Echo6 source IPs when going to IdahoMesh
# Nebras see the bridge's IdahoMesh IP, not real Echo6 IPs
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -d 100.100.0.0/16 -j MASQUERADE
# Allow Echo6 → IdahoMesh (outbound)
iptables -A FORWARD -s 100.64.0.0/10 -d 100.100.0.0/16 -j ACCEPT
# Allow established/related return traffic only (responses to Echo6-initiated connections)
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -m state --state ESTABLISHED,RELATED -j ACCEPT
# DROP all new connections from IdahoMesh → Echo6
iptables -A FORWARD -s 100.100.0.0/16 -d 100.64.0.0/10 -j DROP
```
Persist across reboots:
```bash
netfilter-persistent save
```
Verify:
```bash
iptables -L FORWARD -v -n
iptables -t nat -L POSTROUTING -v -n
```
> **What this achieves:**
> - Echo6 devices can SSH/ping Nebras through the bridge (NAT handles return path)
> - Nebras see the bridge's 100.100.0.x IP as source, never real Echo6 IPs
> - Sidpatchy has NO routable path into Echo6 — no route is advertised and the firewall drops it
> - Sidpatchy can still reach Nebras directly within the IdahoMesh tailnet (no bridge involved)
---
## Phase 3: Expose vpn.idahomesh.com
### 3.1 Issue SSL Certificate
```bash
ssh root@192.168.1.241
pct exec 101 -- bash -c '
export GD_Key="<from .ref/credentials>"
export GD_Secret="<from .ref/credentials>"
/root/.acme.sh/acme.sh --issue --dns dns_gd -d vpn.idahomesh.com --server letsencrypt
'
```
### 3.2 Install Certificate
```bash
pct exec 101 -- bash -c '
mkdir -p /etc/caddy/certs
/root/.acme.sh/acme.sh --install-cert -d vpn.idahomesh.com \
--cert-file /etc/caddy/certs/vpn.idahomesh.com.crt \
--key-file /etc/caddy/certs/vpn.idahomesh.com.key \
--fullchain-file /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt \
--reloadcmd "systemctl reload caddy"
chown -R caddy:caddy /etc/caddy/certs
chmod 600 /etc/caddy/certs/*.key
chmod 644 /etc/caddy/certs/*.crt
'
```
### 3.3 Add Caddy Site Block
```bash
pct exec 101 -- bash -c 'cat >> /etc/caddy/Caddyfile << '\''EOF'\''
vpn.idahomesh.com {
tls /etc/caddy/certs/vpn.idahomesh.com.fullchain.crt /etc/caddy/certs/vpn.idahomesh.com.key
reverse_proxy 192.168.1.106:8080
}
EOF
systemctl reload caddy'
```
### 3.4 Add GoDaddy DNS Record
```bash
# On cortex/TOC
source /home/zvx/projects/.ref/credentials
godaddy-dns.py add-a idahomesh.com vpn 199.6.36.163
```
### 3.5 Verify
```bash
dig +short vpn.idahomesh.com
# Should return 199.6.36.163
curl -I https://vpn.idahomesh.com
```
---
## Phase 4: Register Nebra CM3 Gateways
Only Burley Butte for now. See `idahomesh-vpn-device-setup.md` for the full device onboarding runbook.
```bash
# SSH to Burley Butte
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up \
--login-server=https://vpn.idahomesh.com \
--authkey=<nebra-preauthkey-from-step-1.6> \
--hostname=burley-butte
```
Verify on the IdahoMesh Headscale:
```bash
# On CT 106
headscale nodes list
```
---
## Phase 5: Sidpatchy Onboarding
Send Sidpatchy the following:
1. **IdahoMesh VPN URL:** `https://vpn.idahomesh.com`
2. **Preauthkey:** (the one generated in Step 1.6 for sidpatchy)
3. **Device setup runbook:** `idahomesh-vpn-device-setup.md`
---
## Phase 6: Verification
### From the bridge LXC (CT 107)
```bash
# Ping Burley Butte via IdahoMesh tailnet
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock ping burley-butte
# Check status on both tailnets
tailscale status
tailscale --socket=/var/run/tailscale-meshtastic/tailscaled.sock status
```
### From any Echo6 machine (via bridge routes)
```bash
# Should be routable through the bridge (NAT'd)
ping 100.100.0.x # Burley Butte's IdahoMesh IP
```
### Verify isolation — from IdahoMesh side
```bash
# This MUST fail — Sidpatchy or Nebras should NOT reach Echo6 IPs
ping 100.64.0.14 # cortex — should timeout/unreachable
```
---
## Quick Reference
| Component | Location | Tailnet | IP |
|----------------|------------------|------------|-----------------|
| IdahoMesh HS | CT 106, utility | IdahoMesh | 192.168.1.106 |
| Bridge LXC | CT 107, utility | Both | 192.168.1.107 |
| Burley Butte | Field site | IdahoMesh | 100.100.0.x |
| Sidpatchy | Remote | IdahoMesh | 100.100.0.x |
---
## Maintenance Notes
- **Preauthkeys expire.** Generate long-lived reusable keys for Nebras, short-lived for humans.
- **Headscale updates:** Check releases at https://github.com/juanfont/headscale/releases
- **ACL changes:** Edit `/etc/headscale/acl.json` on CT 106, then `systemctl reload headscale`
- **Firewall rules:** Persisted via `netfilter-persistent` on CT 107. Verify after reboot with `iptables -L FORWARD -v -n`
- **If a Nebra goes offline:** Check `headscale nodes list` — may need a new key if expired.
- **Sidpatchy wants off?** `headscale nodes delete -i <node-id>` and revoke the preauthkey.
- **Device setup instructions:** See `idahomesh-vpn-device-setup.md`
---
*Last updated: 2026-02-11*

92
projects/mmud-project.md Normal file
View file

@ -0,0 +1,92 @@
# MMUD — Mesh Multi-User Dungeon
Text-based multiplayer dungeon crawler for Meshtastic LoRa mesh networks. BBS door games (LORD, TradeWars) adapted for 150-char mesh radio constraints, async play, 30-day wipe cycles.
## Status
**Phase:** Deployed and running — all 6 phases implemented, NPC conversation system live with Gemini 2.5 Flash.
## Deployment
- **Game Daemon:** CT 109 (192.168.1.109) on utility node, Docker container, Flask dashboard on port 5000
- **Dashboard:** https://mmud.echo6.co (Last Ember — dark tavern aesthetic)
- **SIM Nodes:** 6 meshtasticd LXC containers (CT 111-116) on utility
- EMBR (CT 111) — game server
- DCRG (CT 112) — broadcast
- GRST (CT 113) — Grist barkeep NPC
- MRN (CT 114) — Maren healer NPC
- TRVL (CT 115) — Torval merchant NPC
- WSPR (CT 116) — Whisper sage NPC
- **LLM Backend:** Gemini 2.5 Flash via Google genai SDK (configured in DB `llm_config` table)
- **Compose:** `/opt/mmud/docker-compose.yml` on CT 109
- **Admin:** https://mmud.echo6.co/admin (session auth, password in docker env)
## Repo
`/home/zvx/projects/mmud` (GitHub: zvx-echo6/mmud)
The repo contains a `CLAUDE.md` with full architecture, directory structure, and implementation guidance. **Read it first before any work.**
## Key Files
- `CLAUDE.md` — Architecture, patterns, gotchas
- `docs/planned.md` — Complete game design document (~950 lines). Source of truth for all mechanics.
- `docs/npc-lore.md` — NPC deep lore bible (5-layer Hearth-Sworn backstories, 514 lines)
- `docs/worldbuilding.md` — Surface-level worldbuilding (Legend of Oryn, floor identities)
- `config.py` — All game constants with rationale
- `src/db/schema.sql` — Full database schema (migrations in `src/db/migrations/`)
## Architecture
- **Python 3.11+**, SQLite, Meshtastic Python API, Flask 3.x, Jinja2
- **Docker:** python:3.11-slim, /data volume for SQLite DB
- **6-node mesh topology:** EMBR (game), DCRG (broadcast), 4 NPC nodes
- **150 characters per message** — hard ceiling from Meshtastic LoRa
- **12 dungeon actions/day**, 30-day epochs, async-first
- **828+ tests** — `python3 -m pytest tests/ -x -v`
## NPC Conversation System
NPCs use runtime LLM calls (the one exception to the "no runtime LLM" rule). Key features:
- **TX Tag System:** LLM prefixes responses with `[TX:action:detail]` for game mechanics (heal, buy, sell, browse, gamble, hint, recap)
- **Session Memory:** Per-player per-NPC conversation history with persistent memory summaries
- **Deep Lore:** Five-layer backstory system (Surface → Observations → History → Truth → Unspeakable)
- Trigger word detection pushes toward deeper layers
- Interaction count tracks conversation depth per player per NPC
- "Soren" is a Layer 5 nuclear trigger for all 4 NPCs
- **Easter Eggs:** Death memory (Maren), daily gamble (Torval), countdown (Whisper), late-epoch vulnerability (Maren), inter-NPC secret (Torval/Whisper)
- **DummyBackend:** Keyword-based offline mode for testing without LLM
## LLM Configuration
- Model configured via `llm_config` table in SQLite DB (not env vars)
- Currently: Gemini 2.5 Flash (`gemini-2.5-flash`)
- API key stored in DB, manageable via admin panel at /admin/llm
- **No `max_output_tokens` restrictions** — Gemini 2.5 Flash thinking tokens consume the budget, causing truncation. All backends have token limits removed.
- Supports: Google (Gemini), Anthropic (Claude), OpenAI-compatible backends
## Development Phases (All Complete)
1. **Core Loop** — Message handling, parser, player creation, navigation, combat, death, action budget
2. **Economy & Progression** — XP, leveling, gold, shops, gear, bank, healer
3. **Social Systems** — Broadcasts, barkeep, bounty board, player messages, mail
4. **Epoch Generation** — World gen, LLM narrative pipeline, secrets, bounty pools
5. **Endgame Modes** — Hold the Line, Raid Boss, Retrieve & Escape, epoch vote
6. **The Breach** — Breach zone gen, 4 mini-events (Heist, Emergence, Incursion, Resonance)
## Gotchas
- Gemini 2.5 Flash thinking tokens count against `max_output_tokens` — never set token limits
- NPC greeting path uses `complete()`, conversation path uses `chat()` — different code paths
- `npc_memory` table stores `turn_count` for interaction depth tracking
- Death log table (`death_log`) tracks monster kills for Maren's memory feature
- Town actions are always free — never charge dungeon actions in town
- The design doc (`docs/planned.md`) is the source of truth — if code contradicts it, code is wrong
## Notes
- All regen/HP/damage numbers are targets — will need playtesting
- SQLite single file DB, no ORM, raw parameterized SQL
- Every outbound message must fit 150 chars — the formatter is the final gate
- Container on CT 109 connects to 6 SIM nodes via TCP (ports 4403)

823
projects/mmud/last-ember-chronicle.html Normal file
View file

@ -0,0 +1,823 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>The Last Ember — Chronicle</title>
<link href="https://fonts.googleapis.com/css2?family=Cinzel:wght@400;600;700&family=Crimson+Text:ital,wght@0,400;0,600;1,400&family=JetBrains+Mono:wght@300;400;500&display=swap" rel="stylesheet">
<style>
:root {
--ember: #e8713a;
--ember-glow: #ff9d5c;
--ember-deep: #c44e1a;
--ash: #1a1714;
--charcoal: #0d0b09;
--smoke: #2a2520;
--smoke-light: #3d3630;
--parchment: #d4c4a8;
--parchment-dark: #b8a88c;
--parchment-faded: #a89878;
--bone: #c8b898;
--blood: #8b2020;
--blood-bright: #cc3333;
--gold: #c4a44a;
--gold-dim: #8a7a3a;
--frost: #7a9ab0;
--poison: #5a8a4a;
--text-bright: #e8dcc8;
--text-dim: #9a8e78;
--text-ghost: #5a5244;
--victory: #5a8a4a;
--defeat: #8b2020;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--charcoal);
color: var(--text-bright);
font-family: 'Crimson Text', Georgia, serif;
min-height: 100vh;
overflow-x: hidden;
}
#ember-canvas {
position: fixed;
top: 0; left: 0;
width: 100%; height: 100%;
pointer-events: none;
z-index: 0;
opacity: 0.4;
}
.page-wrap {
position: relative;
z-index: 2;
max-width: 800px;
margin: 0 auto;
padding: 0 24px;
}
/* ═══ NAV ═══ */
.nav-bar {
display: flex;
align-items: center;
justify-content: center;
gap: 32px;
padding: 20px 0;
border-bottom: 1px solid rgba(90,82,68,0.15);
margin-bottom: 12px;
}
.nav-link {
font-family: 'Cinzel', serif;
font-size: 10px;
letter-spacing: 0.25em;
color: var(--text-ghost);
text-decoration: none;
text-transform: uppercase;
padding: 6px 0;
border-bottom: 1px solid transparent;
transition: all 0.3s;
cursor: pointer;
}
.nav-link:hover { color: var(--parchment-faded); }
.nav-link.active {
color: var(--parchment);
border-bottom-color: var(--ember);
}
.nav-home {
font-family: 'Cinzel', serif;
font-size: 14px;
color: var(--parchment-faded);
text-decoration: none;
letter-spacing: 0.1em;
transition: color 0.3s;
}
.nav-home:hover { color: var(--ember-glow); }
/* ═══ PAGE HEADER ═══ */
.page-header {
text-align: center;
padding: 40px 0 12px;
}
.page-title {
font-family: 'Cinzel', serif;
font-size: clamp(22px, 4vw, 32px);
font-weight: 700;
letter-spacing: 0.12em;
color: var(--parchment);
text-shadow: 0 0 30px rgba(232,113,58,0.2);
margin-bottom: 6px;
}
.page-subtitle {
font-size: 15px;
font-style: italic;
color: var(--text-ghost);
max-width: 500px;
margin: 0 auto;
line-height: 1.5;
}
.divider {
display: flex;
align-items: center;
gap: 16px;
margin: 28px 0;
color: var(--text-ghost);
font-size: 11px;
letter-spacing: 0.2em;
font-family: 'Cinzel', serif;
}
.divider::before, .divider::after {
content: '';
flex: 1;
height: 1px;
background: linear-gradient(90deg, transparent, var(--smoke-light), transparent);
}
/* ═══ CHRONICLE — EPOCH CARDS ═══ */
.epoch-card {
position: relative;
margin-bottom: 40px;
padding: 28px 32px;
background: linear-gradient(180deg, rgba(26,23,20,0.95), rgba(13,11,9,0.95));
border: 1px solid var(--smoke-light);
border-radius: 2px;
overflow: hidden;
}
.epoch-card::before {
content: '';
position: absolute;
top: 0; left: 0; right: 0;
height: 2px;
}
.epoch-card.victory::before {
background: linear-gradient(90deg, transparent, var(--victory), transparent);
}
.epoch-card.defeat::before {
background: linear-gradient(90deg, transparent, var(--defeat), transparent);
}
.epoch-card.current::before {
background: linear-gradient(90deg, transparent, var(--ember), transparent);
animation: current-pulse 3s ease-in-out infinite;
}
@keyframes current-pulse {
0%, 100% { opacity: 0.5; }
50% { opacity: 1; }
}
.epoch-header {
display: flex;
align-items: flex-start;
justify-content: space-between;
margin-bottom: 16px;
}
.epoch-number {
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
letter-spacing: 0.2em;
color: var(--text-ghost);
margin-bottom: 4px;
}
.epoch-title {
font-family: 'Cinzel', serif;
font-size: 20px;
font-weight: 600;
color: var(--parchment);
line-height: 1.3;
}
.epoch-outcome {
flex-shrink: 0;
padding: 4px 14px;
font-family: 'Cinzel', serif;
font-size: 10px;
letter-spacing: 0.2em;
border-radius: 1px;
text-transform: uppercase;
}
.epoch-outcome.victory {
border: 1px solid var(--victory);
color: var(--victory);
background: rgba(90,138,74,0.08);
}
.epoch-outcome.defeat {
border: 1px solid var(--defeat);
color: var(--blood-bright);
background: rgba(139,32,32,0.08);
}
.epoch-outcome.ongoing {
border: 1px solid var(--ember);
color: var(--ember-glow);
background: rgba(232,113,58,0.08);
animation: ongoing-pulse 2s ease-in-out infinite;
}
@keyframes ongoing-pulse {
0%, 100% { opacity: 0.7; }
50% { opacity: 1; }
}
.epoch-meta {
display: flex;
gap: 20px;
margin-bottom: 16px;
flex-wrap: wrap;
}
.epoch-meta-item {
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
color: var(--text-ghost);
letter-spacing: 0.05em;
}
.epoch-meta-item .meta-val {
color: var(--text-dim);
}
.epoch-summary {
font-size: 16px;
line-height: 1.7;
color: var(--text-dim);
}
.epoch-summary p {
margin-bottom: 12px;
}
.epoch-summary p:last-child { margin-bottom: 0; }
.epoch-summary .name { color: var(--parchment); font-weight: 600; }
.epoch-summary .place { color: var(--ember-glow); font-style: italic; }
.epoch-summary .item { color: var(--gold); }
.epoch-roster {
margin-top: 16px;
padding-top: 12px;
border-top: 1px solid rgba(90,82,68,0.12);
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
color: var(--text-ghost);
letter-spacing: 0.05em;
}
.epoch-roster .roster-names {
color: var(--text-dim);
margin-left: 4px;
}
/* ═══ JOURNALS — TAB SYSTEM ═══ */
.journal-section { display: none; }
.journal-section.active { display: block; }
.chronicle-section { display: none; }
.chronicle-section.active { display: block; }
.npc-tabs {
display: flex;
gap: 0;
margin-bottom: 28px;
border-bottom: 1px solid var(--smoke-light);
}
.npc-tab {
flex: 1;
text-align: center;
padding: 14px 8px;
font-family: 'Cinzel', serif;
font-size: 11px;
letter-spacing: 0.15em;
color: var(--text-ghost);
cursor: pointer;
border-bottom: 2px solid transparent;
transition: all 0.3s;
position: relative;
}
.npc-tab:hover { color: var(--parchment-faded); }
.npc-tab.active {
color: var(--parchment);
border-bottom-color: var(--ember);
}
.npc-tab .tab-icon {
display: block;
font-size: 18px;
margin-bottom: 4px;
opacity: 0.6;
}
.npc-tab.active .tab-icon { opacity: 0.9; }
/* Journal entries */
.journal-feed { display: none; }
.journal-feed.active { display: block; }
.journal-entry {
margin-bottom: 32px;
padding: 24px 28px;
background: linear-gradient(180deg, rgba(26,23,20,0.9), rgba(13,11,9,0.9));
border: 1px solid var(--smoke-light);
border-radius: 2px;
position: relative;
overflow: hidden;
}
.journal-entry::before {
content: '';
position: absolute;
top: 0; left: 0;
width: 3px; height: 100%;
}
.journal-entry.grist::before { background: var(--ember); opacity: 0.4; }
.journal-entry.maren::before { background: var(--blood-bright); opacity: 0.4; }
.journal-entry.torval::before { background: var(--gold); opacity: 0.4; }
.journal-entry.whisper::before { background: var(--frost); opacity: 0.4; }
.journal-date {
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
color: var(--text-ghost);
letter-spacing: 0.15em;
margin-bottom: 12px;
}
.journal-text {
font-size: 16px;
line-height: 1.75;
color: var(--text-dim);
}
.journal-text p {
margin-bottom: 10px;
text-indent: 1.5em;
}
.journal-text p:first-child { text-indent: 0; }
.journal-text p:last-child { margin-bottom: 0; }
/* Voice-specific styling */
.journal-entry.grist .journal-text {
font-size: 15px;
line-height: 1.65;
}
.journal-entry.whisper .journal-text {
font-style: italic;
letter-spacing: 0.01em;
}
.journal-entry.maren .journal-text {
font-size: 15px;
}
.journal-entry.torval .journal-text {
font-size: 16px;
}
.journal-npc-sig {
margin-top: 14px;
text-align: right;
font-family: 'Cinzel', serif;
font-size: 11px;
color: var(--text-ghost);
letter-spacing: 0.1em;
font-style: italic;
}
/* ═══ FOOTER ═══ */
.page-footer {
text-align: center;
padding: 32px 0 48px;
border-top: 1px solid rgba(90,82,68,0.15);
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
color: var(--text-ghost);
letter-spacing: 0.15em;
}
.page-footer a {
color: var(--text-ghost);
text-decoration: none;
transition: color 0.2s;
}
.page-footer a:hover { color: var(--ember); }
/* Responsive */
@media (max-width: 600px) {
.epoch-header { flex-direction: column; gap: 10px; }
.epoch-card { padding: 20px; }
.journal-entry { padding: 18px 20px; }
.npc-tab { font-size: 9px; padding: 10px 4px; }
.npc-tab .tab-icon { font-size: 16px; }
}
</style>
</head>
<body>
<canvas id="ember-canvas"></canvas>
<div class="page-wrap">
<!-- NAV -->
<nav class="nav-bar">
<a class="nav-home" href="#">The Last Ember</a>
<span style="color:var(--smoke-light)">·</span>
<a class="nav-link active" data-page="chronicle" onclick="showPage('chronicle')">Chronicle</a>
<a class="nav-link" data-page="journals" onclick="showPage('journals')">Journals</a>
<a class="nav-link" href="#">Board</a>
</nav>
<!-- ════════════════════════════════════ -->
<!-- CHRONICLE PAGE -->
<!-- ════════════════════════════════════ -->
<div class="chronicle-section active" id="page-chronicle">
<div class="page-header">
<h1 class="page-title">Chronicle</h1>
<p class="page-subtitle">Every epoch leaves its mark. The dungeon forgets. We do not.</p>
</div>
<div class="divider">CURRENT EPOCH</div>
<!-- CURRENT EPOCH -->
<div class="epoch-card current">
<div class="epoch-header">
<div>
<div class="epoch-number">EPOCH VII · DAY 17 OF 30</div>
<div class="epoch-title">The Siege of the Drowned Mines</div>
</div>
<span class="epoch-outcome ongoing">In Progress</span>
</div>
<div class="epoch-meta">
<span class="epoch-meta-item">MODE: <span class="meta-val">Hold the Line</span></span>
<span class="epoch-meta-item">BREACH: <span class="meta-val">The Emergence (open)</span></span>
<span class="epoch-meta-item">PLAYERS: <span class="meta-val">7</span></span>
<span class="epoch-meta-item">SECRETS: <span class="meta-val">11/20</span></span>
</div>
<div class="epoch-summary">
<p>Seventeen days in and the water still rises. <span class="name">Kael</span> has led the push through the second depth, establishing <span class="place">Checkpoint Alpha</span> through sheer attrition — three deaths, two retreats, and a final dawn push that cleared the cluster in a single session. The Bounty Troll that haunted <span class="place">the Sunken Gallery</span> for nine days fell to a combined effort, its last breath echoing through flooded corridors that have already begun to reclaim the rooms behind the front line.</p>
<p><span class="name">Mira</span> has proven the epoch's quiet weapon — twelve secrets uncovered, including the <span class="item">Ancient Ward</span> that halved the second floor's regen and gave the fighters a window they desperately needed. The Breach opened two days ago and something massive stirs within. Floor 3 is barely mapped. Floor 4 is a rumor. Thirteen days remain, and the mines are not finished with them yet.</p>
</div>
<div class="epoch-roster">
ROSTER: <span class="roster-names">Kael · Mira · Torr · Sable · Dren · Ash · Vex</span>
</div>
</div>
<div class="divider">PAST EPOCHS</div>
<!-- EPOCH VI — VICTORY -->
<div class="epoch-card victory">
<div class="epoch-header">
<div>
<div class="epoch-number">EPOCH VI · 30 DAYS · FEB 2026</div>
<div class="epoch-title">The Crown of the Ember Wyrm</div>
</div>
<span class="epoch-outcome victory">Victory</span>
</div>
<div class="epoch-meta">
<span class="epoch-meta-item">MODE: <span class="meta-val">Retrieve & Escape</span></span>
<span class="epoch-meta-item">BREACH: <span class="meta-val">The Resonance</span></span>
<span class="epoch-meta-item">PLAYERS: <span class="meta-val">9</span></span>
<span class="epoch-meta-item">SECRETS: <span class="meta-val">18/20</span></span>
</div>
<div class="epoch-summary">
<p>They called it the impossible run. <span class="name">Torr</span> claimed the <span class="item">Crown of the Ember Wyrm</span> on the twenty-second day, four floors deep in chambers that burned with a heat that had no source. The Pursuer awakened three rooms behind — an eyeless thing that moved without sound and killed without hesitation. <span class="name">Mira</span> had spent six days warding the third floor, and <span class="name">Sable</span> held the chokepoint between the second and third depths for eleven hours before the Pursuer caught her. She died on her feet. The Crown passed to <span class="name">Ash</span> through the relay, and the final sprint through the first floor took four minutes of real time and a year off everyone's nerves.</p>
<p>The Resonance Breach had been the epoch's turning point — a puzzle dungeon between floors two and three that <span class="name">Dren</span> solved alone over three quiet days while the rest of the server fought for every room. The shortcut it opened shaved two floors off the escape route and made the impossible merely improbable. Eighteen of twenty secrets fell. The last two died with the epoch, their locations known to no one. <span class="name">Kael</span> finished at level ten — the first to cap since Epoch III. The Crown rests in the Hall. The Wyrm's chambers have already begun to reshape.</p>
</div>
<div class="epoch-roster">
ROSTER: <span class="roster-names">Kael · Mira · Torr · Sable · Dren · Ash · Vex · Lira · Puck</span>
</div>
</div>
<!-- EPOCH V — DEFEAT -->
<div class="epoch-card defeat">
<div class="epoch-header">
<div>
<div class="epoch-number">EPOCH V · 30 DAYS · JAN 2026</div>
<div class="epoch-title">The Warden of the Bone Pits</div>
</div>
<span class="epoch-outcome defeat">Defeat</span>
</div>
<div class="epoch-meta">
<span class="epoch-meta-item">MODE: <span class="meta-val">Hold the Line</span></span>
<span class="epoch-meta-item">BREACH: <span class="meta-val">The Incursion</span></span>
<span class="epoch-meta-item">PLAYERS: <span class="meta-val">5</span></span>
<span class="epoch-meta-item">SECRETS: <span class="meta-val">13/20</span></span>
</div>
<div class="epoch-summary">
<p>Five adventurers against a dungeon that fought back with everything it had. The <span class="place">Bone Pits</span> earned their name — floors slick with calcite, walls studded with things that used to be alive, and a regen rate on the third depth that three players simply could not outpace. <span class="name">Kael</span> and <span class="name">Mira</span> held the second floor for twelve consecutive days, a feat of endurance that the barkeep still recounts to anyone who'll listen, but the third floor's checkpoints required a coordination window that never came. The Incursion Breach on day fifteen made it worse — monsters pouring upward through the new passage, forcing <span class="name">Torr</span> to abandon the push and defend cleared ground.</p>
<p>The Warden never spawned. They never reached it. On day twenty-eight, the front line collapsed back to <span class="place">Checkpoint Beta</span> on floor two and held there, grim and exhausted, while the last rooms fell dark around them. <span class="name">Dren</span> joined on day nineteen — too late to turn the tide, but early enough to witness the slow retreat. Thirteen secrets found, seven left buried. The epoch ended not with a killing blow but with a long silence, the dungeon reclaiming what it had never truly lost. Grist poured five drinks that night. Nobody ordered them.</p>
</div>
<div class="epoch-roster">
ROSTER: <span class="roster-names">Kael · Mira · Torr · Dren · Sable</span>
</div>
</div>
<!-- EPOCH IV — VICTORY -->
<div class="epoch-card victory">
<div class="epoch-header">
<div>
<div class="epoch-number">EPOCH IV · 30 DAYS · DEC 2025</div>
<div class="epoch-title">The Fall of the Iron Colossus</div>
</div>
<span class="epoch-outcome victory">Victory</span>
</div>
<div class="epoch-meta">
<span class="epoch-meta-item">MODE: <span class="meta-val">Raid Boss</span></span>
<span class="epoch-meta-item">BREACH: <span class="meta-val">The Heist</span></span>
<span class="epoch-meta-item">PLAYERS: <span class="meta-val">11</span></span>
<span class="epoch-meta-item">SECRETS: <span class="meta-val">20/20</span></span>
</div>
<div class="epoch-summary">
<p>Eleven adventurers. Three thousand three hundred hit points of ancient iron and malice squatting in the deepest chamber of the fourth floor. The <span class="place">Iron Colossus</span> rolled Armor Phase and No Escape — a combination that meant once you committed below twenty-five percent, you were finishing the fight or dying in it. The first week was pure scouting. <span class="name">Lira</span> lost two days' gold learning what the phase transitions looked like. <span class="name">Puck</span> discovered the armor weakness on day nine — a ritual hidden behind a stat-gated secret on floor three that permanently stripped the Colossus's defenses. The tide turned.</p>
<p>By day twenty, every player on the server had contributed damage. The final phase began on a Tuesday morning when <span class="name">Kael</span> pushed it below the threshold and the exits sealed. He died. <span class="name">Vex</span> went in next and died. <span class="name">Ash</span> went in third with stacked discovery buffs, two consumables, and a borrowed <span class="item">Runed Maul</span> from Torval's back shelf. The Colossus fell in six rounds. The only epoch where every secret was found. <span class="name">Mira</span> found the twentieth on day twenty-nine — a lore secret hidden in something Whisper had said on day three that nobody thought to write down.</p>
</div>
<div class="epoch-roster">
ROSTER: <span class="roster-names">Kael · Mira · Torr · Sable · Dren · Ash · Vex · Lira · Puck · Strand · Wick</span>
</div>
</div>
</div>
<!-- ════════════════════════════════════ -->
<!-- JOURNALS PAGE -->
<!-- ════════════════════════════════════ -->
<div class="journal-section" id="page-journals">
<div class="page-header">
<h1 class="page-title">Journals</h1>
<p class="page-subtitle">Four voices. Same day. Different truths.</p>
</div>
<!-- NPC TABS -->
<div class="npc-tabs">
<div class="npc-tab active" data-npc="grist" onclick="showJournal('grist')">
<span class="tab-icon">🍺</span>
Grist
</div>
<div class="npc-tab" data-npc="maren" onclick="showJournal('maren')">
<span class="tab-icon">🩸</span>
Maren
</div>
<div class="npc-tab" data-npc="torval" onclick="showJournal('torval')">
<span class="tab-icon">⚖</span>
Torval
</div>
<div class="npc-tab" data-npc="whisper" onclick="showJournal('whisper')">
<span class="tab-icon">👁</span>
Whisper
</div>
</div>
<!-- GRIST'S JOURNAL -->
<div class="journal-feed active" id="journal-grist">
<div class="journal-entry grist">
<div class="journal-date">EPOCH VII · DAY 17</div>
<div class="journal-text">
<p>Kael came in bloody again. Wouldn't say from what. Ordered two drinks, finished one, stared at the wall for ten minutes, then asked about the bounty board. I told him the troll was done. He already knew. He's the one who killed it.</p>
<p>Mira stopped by after. She found something on the second floor — wouldn't say what exactly, but she had that look. The one where she knows something the dungeon doesn't want her to know. Traded a token for a hint about floor three. I gave her what I had. She'll figure out the rest.</p>
<p>Sable came in late. Died again. Third time this epoch. Didn't want to talk about it. I poured her something warm and told her the front line held. It did. Barely. Floor two lost two rooms overnight but Alpha's holding. That checkpoint isn't going anywhere.</p>
<p>Seven of them now. Seven against whatever's down there. Thirteen days left. The Breach opened yesterday and something's moving inside it. Big. They can feel it through the floor when it shifts. I can feel it through the bar.</p>
</div>
<div class="journal-npc-sig">— Grist</div>
</div>
<div class="journal-entry grist">
<div class="journal-date">EPOCH VII · DAY 16</div>
<div class="journal-text">
<p>The troll died today. Took nine days. Kael landed the killing blow but Mira and Torr chipped it down to nothing over the past week. That thing regenerated every night and every morning someone went back in. That's what this place does to people. It makes them stubborn.</p>
<p>New bounty went up. Spiders on the eastern branch of floor two. Six of them. Torr's already on it. He likes the quiet work — finds the nest, clears what he can, gets out. No glory, just progress. Good kid.</p>
<p>The Breach cracked open sometime after midnight. I heard it. Everyone heard it. The lanterns flickered for the first time in longer than I can remember. Something poured through that crack that wasn't light and wasn't dark. Dren was the first one down to look. Hasn't come back to report yet.</p>
</div>
<div class="journal-npc-sig">— Grist</div>
</div>
<div class="journal-entry grist">
<div class="journal-date">EPOCH VII · DAY 15</div>
<div class="journal-text">
<p>Told them. Three days I've been saying the walls were getting thin. Nobody listens to the barkeep until the ground starts shaking. The Breach is open. The passage sits between the second and third depths, and whatever's inside it is not from either floor.</p>
<p>Quiet day otherwise. Everyone's saving their actions for tomorrow. Smart. The dungeon doesn't care about smart, but it helps.</p>
</div>
<div class="journal-npc-sig">— Grist</div>
</div>
</div>
<!-- MAREN'S JOURNAL -->
<div class="journal-feed" id="journal-maren">
<div class="journal-entry maren">
<div class="journal-date">EPOCH VII · DAY 17</div>
<div class="journal-text">
<p>Three today. Kael first — deep lacerations across the forearms, consistent with something that grabs before it bites. He sat still while I worked. Didn't flinch. That's not bravery. That's numbness. I've seen the difference.</p>
<p>Sable second. Blunt force trauma to the ribs, probably from a charging attack she didn't sidestep. I asked her why she rushed the room. She said she thought she could make it. They always think they can make it. I set the rib and told her to stay above floor one for two days. She won't.</p>
<p>Torr came in for a routine patch. Minor cuts, nothing structural. He's careful. Moves like someone who's been hurt enough times to know exactly how much it costs. I appreciate that. More of them should learn it before they learn it the hard way.</p>
<p>The Breach is open. I can smell it from here — ozone and something older. I know what's on the other side of cracks like that. I know what lives in the spaces between floors. I went there once. I'm not going back. But they will. And I'll be here when they crawl out.</p>
</div>
<div class="journal-npc-sig">— Maren</div>
</div>
<div class="journal-entry maren">
<div class="journal-date">EPOCH VII · DAY 16</div>
<div class="journal-text">
<p>Sable again. That's twice in three days. This time it was the Gallery — took a hit from the troll's replacement spawn that she wasn't expecting. The original was stronger, she said. As if that's an excuse for not respecting the weaker one. The weaker ones still kill you. I've stitched enough of them to know.</p>
<p>No other patients. The troll's death seems to have given them confidence. Confidence is when I get busy.</p>
</div>
<div class="journal-npc-sig">— Maren</div>
</div>
<div class="journal-entry maren">
<div class="journal-date">EPOCH VII · DAY 15</div>
<div class="journal-text">
<p>No injuries today. Unusual. They're all resting, saving themselves for whatever the Breach brings. The smart ones prepare. The others will be my patients tomorrow.</p>
<p>The scar on my palm aches when the dungeon shifts. It ached all night.</p>
</div>
<div class="journal-npc-sig">— Maren</div>
</div>
</div>
<!-- TORVAL'S JOURNAL -->
<div class="journal-feed" id="journal-torval">
<div class="journal-entry torval">
<div class="journal-date">EPOCH VII · DAY 17</div>
<div class="journal-text">
<p>Good day! Sold a reinforced buckler to Sable — she needed it after, well, you know. Third death this run. I didn't mention that. Just told her the buckler was "lightly used, deeply reliable." She didn't laugh. They never do. But she bought it, and that's what matters. For her, I mean. Protection. Very important.</p>
<p>Kael came in to appraise something from the second floor. Tapped it on the counter. Listened. Heavy, good ring, slight harmonic on the follow-through. Tier four, easily. Named a fair price. He sold it back for the upgrade fund. Practical man, Kael. No sentiment about gear. I respect that. I also profit from it, which I respect slightly more.</p>
<p>Dren bought three smoke bombs. Three. For one person. I asked if he was planning something specific. He said "the Breach." I said "ah." I wrapped them carefully. Something about the way he said it made me think he might actually need all three.</p>
<p>The ledger gains another page. The pages at the front are still unreadable. I've stopped trying.</p>
</div>
<div class="journal-npc-sig">— Torval</div>
</div>
<div class="journal-entry torval">
<div class="journal-date">EPOCH VII · DAY 16</div>
<div class="journal-text">
<p>Inventory refresh day! Somehow the stock always matches what they'll need. I've stopped questioning it. New shipment includes tier three weapons appropriate for the second floor push and a few trinkets I haven't seen before. One of them hums. Not loudly. Not unpleasantly. But it hums. Priced it accordingly.</p>
<p>The troll is dead. Good for morale, bad for my potion sales. When the big threat goes away, they get brave and stop buying healing supplies. I'll give it two days before Sable's back at my counter buying bandages.</p>
</div>
<div class="journal-npc-sig">— Torval</div>
</div>
<div class="journal-entry torval">
<div class="journal-date">EPOCH VII · DAY 15</div>
<div class="journal-text">
<p>The ground cracked. Stock fell off two shelves. Nothing broke — I pack carefully, because I know where I work. The Breach is open. New territory means new drops means new customers means new pages in the ledger. I love this job.</p>
<p>Restocked the smoke bombs. I have a feeling.</p>
</div>
<div class="journal-npc-sig">— Torval</div>
</div>
</div>
<!-- WHISPER'S JOURNAL -->
<div class="journal-feed" id="journal-whisper">
<div class="journal-entry whisper">
<div class="journal-date">EPOCH VII · DAY 17</div>
<div class="journal-text">
<p>The second floor remembers being whole. It pushes back at night — not the monsters, the stone itself. Rooms seal shut like wounds closing. Two lost since dawn. Alpha holds because something older than the mine agreed it should. I don't know what. I heard it once, through the wall between the second and third depths. It was counting.</p>
<p>Mira came to the corner today. She found the ward — I could see it on her, the residue of old mechanisms waking up. She asked about the eastern branch. I told her what I could. The words come in pieces. A door. A serpent that isn't a serpent. The sound of water where no water runs. She wrote it down. Good. I can't always say it twice.</p>
<p>The Breach breathes. I can hear it from here. Two floors away and I can hear it like it's sitting next to me. Something large. Something that was here before the mines. Before the bar. Before the lanterns. Not before me. I was here first. I think. The memory is thin today.</p>
</div>
<div class="journal-npc-sig">— Whisper</div>
</div>
<div class="journal-entry whisper">
<div class="journal-date">EPOCH VII · DAY 16</div>
<div class="journal-text">
<p>The troll stopped. Its voice left the stone. A small silence where there used to be weight. Kael ended it but the dungeon let it end. Some things are allowed to die. Others aren't. The replacement is weaker — a shadow of a shadow. It serves the room but the room doesn't respect it.</p>
<p>Three secrets on the eastern branch. I can feel them like teeth in a jaw. The first is behind something carved. The second requires a key that isn't a key. The third — I lose the third when I try to look at it directly. It moves. Or I move. One of us does.</p>
</div>
<div class="journal-npc-sig">— Whisper</div>
</div>
<div class="journal-entry whisper">
<div class="journal-date">EPOCH VII · DAY 15</div>
<div class="journal-text">
<p>It opened. The thin place between. I told Grist three days ago. He listens, in his way. He told them. They listened, in theirs.</p>
<p>What came through the crack is not new. It has been waiting underneath the underneath, patient as geology. The rooms between the floors are not rooms. They are the dungeon dreaming about itself. The secrets in there are different — not hidden, just not yet decided. They will become what they need to become when someone looks at them long enough.</p>
<p>The lanterns flickered. They have never flickered. I watched them very carefully afterward to make sure they were still the same lanterns. They are. But they noticed too.</p>
</div>
<div class="journal-npc-sig">— Whisper</div>
</div>
</div>
</div>
<!-- FOOTER -->
<div class="page-footer">
<a href="#">The Last Ember</a> · meshMUD
</div>
</div>
<script>
// ═══ EMBER PARTICLES (same as main page) ═══
const canvas = document.getElementById('ember-canvas');
const ctx = canvas.getContext('2d');
let embers = [];
function resize() {
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
}
resize();
window.addEventListener('resize', resize);
class Ember {
constructor() { this.reset(); }
reset() {
this.x = Math.random() * canvas.width;
this.y = canvas.height + 10;
this.size = Math.random() * 2 + 0.5;
this.speedY = -(Math.random() * 0.3 + 0.08);
this.speedX = (Math.random() - 0.5) * 0.2;
this.opacity = Math.random() * 0.4 + 0.15;
this.decay = Math.random() * 0.0008 + 0.0003;
this.wobble = Math.random() * Math.PI * 2;
this.wobbleSpeed = Math.random() * 0.015 + 0.003;
const t = Math.random();
this.r = Math.floor(200 + t * 55);
this.g = Math.floor(80 + t * 80);
this.b = Math.floor(20 + t * 30);
}
update() {
this.wobble += this.wobbleSpeed;
this.x += this.speedX + Math.sin(this.wobble) * 0.12;
this.y += this.speedY;
this.opacity -= this.decay;
if (this.opacity <= 0 || this.y < -20) this.reset();
}
draw() {
ctx.beginPath();
ctx.arc(this.x, this.y, this.size, 0, Math.PI * 2);
ctx.fillStyle = `rgba(${this.r},${this.g},${this.b},${this.opacity})`;
ctx.fill();
ctx.beginPath();
ctx.arc(this.x, this.y, this.size * 2.5, 0, Math.PI * 2);
ctx.fillStyle = `rgba(${this.r},${this.g},${this.b},${this.opacity * 0.12})`;
ctx.fill();
}
}
for (let i = 0; i < 30; i++) {
const e = new Ember();
e.y = Math.random() * canvas.height;
embers.push(e);
}
function animateEmbers() {
ctx.clearRect(0, 0, canvas.width, canvas.height);
embers.forEach(e => { e.update(); e.draw(); });
requestAnimationFrame(animateEmbers);
}
animateEmbers();
// ═══ PAGE SWITCHING ═══
function showPage(page) {
document.querySelectorAll('.chronicle-section, .journal-section').forEach(el => el.classList.remove('active'));
document.getElementById('page-' + page).classList.add('active');
document.querySelectorAll('.nav-link[data-page]').forEach(el => {
el.classList.toggle('active', el.dataset.page === page);
});
window.scrollTo({ top: 0, behavior: 'smooth' });
}
// ═══ NPC JOURNAL TABS ═══
function showJournal(npc) {
document.querySelectorAll('.npc-tab').forEach(el => {
el.classList.toggle('active', el.dataset.npc === npc);
});
document.querySelectorAll('.journal-feed').forEach(el => {
el.classList.toggle('active', el.id === 'journal-' + npc);
});
}
</script>
</body>
</html>

915
projects/mmud/last-ember-howto.html Normal file
View file

@ -0,0 +1,915 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>The Last Ember — How to Play</title>
<link href="https://fonts.googleapis.com/css2?family=Cinzel:wght@400;600;700&family=Crimson+Text:ital,wght@0,400;0,600;1,400&family=JetBrains+Mono:wght@300;400;500&display=swap" rel="stylesheet">
<style>
:root {
--ember: #e8713a;
--ember-glow: #ff9d5c;
--ember-deep: #c44e1a;
--ash: #1a1714;
--charcoal: #0d0b09;
--smoke: #2a2520;
--smoke-light: #3d3630;
--parchment: #d4c4a8;
--parchment-dark: #b8a88c;
--parchment-faded: #a89878;
--bone: #c8b898;
--blood: #8b2020;
--blood-bright: #cc3333;
--gold: #c4a44a;
--gold-dim: #8a7a3a;
--frost: #7a9ab0;
--poison: #5a8a4a;
--text-bright: #e8dcc8;
--text-dim: #9a8e78;
--text-ghost: #5a5244;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--charcoal);
color: var(--text-bright);
font-family: 'Crimson Text', Georgia, serif;
min-height: 100vh;
overflow-x: hidden;
}
#ember-canvas {
position: fixed;
top: 0; left: 0;
width: 100%; height: 100%;
pointer-events: none;
z-index: 0;
opacity: 0.35;
}
.page-wrap {
position: relative;
z-index: 2;
max-width: 720px;
margin: 0 auto;
padding: 0 24px;
}
/* ═══ NAV ═══ */
.nav-bar {
display: flex;
align-items: center;
justify-content: center;
gap: 32px;
padding: 20px 0;
border-bottom: 1px solid rgba(90,82,68,0.15);
margin-bottom: 12px;
}
.nav-link {
font-family: 'Cinzel', serif;
font-size: 10px;
letter-spacing: 0.25em;
color: var(--text-ghost);
text-decoration: none;
text-transform: uppercase;
padding: 6px 0;
border-bottom: 1px solid transparent;
transition: all 0.3s;
}
.nav-link:hover { color: var(--parchment-faded); }
.nav-link.active { color: var(--parchment); border-bottom-color: var(--ember); }
.nav-home {
font-family: 'Cinzel', serif;
font-size: 14px;
color: var(--parchment-faded);
text-decoration: none;
letter-spacing: 0.1em;
transition: color 0.3s;
}
.nav-home:hover { color: var(--ember-glow); }
/* ═══ HEADER ═══ */
.page-header {
text-align: center;
padding: 48px 0 12px;
}
.page-title {
font-family: 'Cinzel', serif;
font-size: clamp(22px, 4vw, 32px);
font-weight: 700;
letter-spacing: 0.12em;
color: var(--parchment);
text-shadow: 0 0 30px rgba(232,113,58,0.2);
margin-bottom: 6px;
}
.page-subtitle {
font-size: 15px;
font-style: italic;
color: var(--text-ghost);
max-width: 460px;
margin: 0 auto;
line-height: 1.5;
}
.divider {
display: flex;
align-items: center;
gap: 16px;
margin: 36px 0 28px;
color: var(--text-ghost);
font-size: 10px;
letter-spacing: 0.25em;
font-family: 'Cinzel', serif;
text-transform: uppercase;
}
.divider::before, .divider::after {
content: '';
flex: 1;
height: 1px;
background: linear-gradient(90deg, transparent, var(--smoke-light), transparent);
}
/* ═══ PROSE SECTIONS ═══ */
.prose {
font-size: 17px;
line-height: 1.8;
color: var(--text-dim);
margin-bottom: 20px;
}
.prose strong {
color: var(--parchment);
font-weight: 600;
}
.prose em.place {
color: var(--ember-glow);
font-style: italic;
}
.prose em.npc {
color: var(--parchment-dark);
font-style: normal;
font-weight: 600;
}
.prose em.cmd {
color: var(--gold);
font-style: normal;
font-family: 'JetBrains Mono', monospace;
font-size: 14px;
background: rgba(196,164,74,0.08);
padding: 1px 6px;
border-radius: 2px;
border: 1px solid rgba(196,164,74,0.15);
}
.prose em.item {
color: var(--gold);
font-style: italic;
}
/* ═══ CALLOUT BOXES ═══ */
.callout {
padding: 20px 24px;
margin: 24px 0;
background: linear-gradient(135deg, rgba(26,23,20,0.95), rgba(42,37,32,0.7));
border: 1px solid var(--smoke-light);
border-radius: 2px;
position: relative;
overflow: hidden;
}
.callout::before {
content: '';
position: absolute;
top: 0; left: 0;
width: 3px;
height: 100%;
}
.callout.ember::before { background: var(--ember); opacity: 0.5; }
.callout.gold::before { background: var(--gold); opacity: 0.5; }
.callout.frost::before { background: var(--frost); opacity: 0.5; }
.callout.blood::before { background: var(--blood-bright); opacity: 0.5; }
.callout-label {
font-family: 'Cinzel', serif;
font-size: 10px;
letter-spacing: 0.2em;
color: var(--text-ghost);
text-transform: uppercase;
margin-bottom: 10px;
}
.callout .prose { margin-bottom: 0; }
.callout .prose:not(:last-child) { margin-bottom: 12px; }
/* ═══ COMMAND REFERENCE ═══ */
.cmd-grid {
display: grid;
grid-template-columns: auto 1fr;
gap: 4px 16px;
padding: 4px 0;
}
.cmd-key {
font-family: 'JetBrains Mono', monospace;
font-size: 13px;
color: var(--gold);
padding: 3px 0;
white-space: nowrap;
}
.cmd-desc {
font-size: 14px;
color: var(--text-dim);
padding: 3px 0;
line-height: 1.5;
}
.cmd-unlock {
font-family: 'JetBrains Mono', monospace;
font-size: 9px;
color: var(--text-ghost);
background: rgba(90,82,68,0.15);
padding: 1px 6px;
border-radius: 1px;
margin-left: 6px;
vertical-align: middle;
}
/* ═══ CLASS CARDS ═══ */
.class-cards {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 16px;
margin: 20px 0;
}
@media (max-width: 560px) {
.class-cards { grid-template-columns: 1fr; }
}
.class-card {
padding: 20px 16px;
background: linear-gradient(180deg, rgba(26,23,20,0.95), rgba(13,11,9,0.95));
border: 1px solid var(--smoke-light);
border-radius: 2px;
text-align: center;
position: relative;
overflow: hidden;
}
.class-card::before {
content: '';
position: absolute;
top: 0; left: 0; right: 0;
height: 2px;
}
.class-card.fighter::before { background: linear-gradient(90deg, transparent, var(--blood-bright), transparent); }
.class-card.caster::before { background: linear-gradient(90deg, transparent, var(--poison), transparent); }
.class-card.rogue::before { background: linear-gradient(90deg, transparent, var(--frost), transparent); }
.class-card-icon {
font-size: 28px;
margin-bottom: 8px;
opacity: 0.8;
}
.class-card-name {
font-family: 'Cinzel', serif;
font-size: 14px;
font-weight: 600;
letter-spacing: 0.1em;
color: var(--parchment);
margin-bottom: 2px;
}
.class-card-stat {
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
letter-spacing: 0.1em;
margin-bottom: 10px;
}
.class-card.fighter .class-card-stat { color: var(--blood-bright); }
.class-card.caster .class-card-stat { color: var(--poison); }
.class-card.rogue .class-card-stat { color: var(--frost); }
.class-card-desc {
font-size: 13px;
line-height: 1.6;
color: var(--text-dim);
}
/* ═══ FLOW DIAGRAM ═══ */
.flow-steps {
display: flex;
flex-direction: column;
gap: 0;
margin: 20px 0;
}
.flow-step {
display: flex;
gap: 16px;
align-items: flex-start;
padding: 14px 0;
}
.flow-step-num {
width: 28px;
height: 28px;
border-radius: 50%;
border: 1px solid var(--smoke-light);
display: flex;
align-items: center;
justify-content: center;
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-ghost);
flex-shrink: 0;
position: relative;
}
.flow-step:not(:last-child) .flow-step-num::after {
content: '';
position: absolute;
top: 28px;
left: 50%;
width: 1px;
height: calc(100% + 16px);
background: var(--smoke-light);
transform: translateX(-50%);
}
.flow-step-content {
padding-top: 3px;
}
.flow-step-label {
font-family: 'Cinzel', serif;
font-size: 12px;
letter-spacing: 0.1em;
color: var(--parchment-faded);
margin-bottom: 4px;
}
.flow-step-text {
font-size: 14px;
line-height: 1.6;
color: var(--text-dim);
}
.flow-step-text code {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: var(--gold);
background: rgba(196,164,74,0.08);
padding: 1px 5px;
border-radius: 2px;
border: 1px solid rgba(196,164,74,0.12);
}
/* ═══ MESSAGE EXAMPLE ═══ */
.msg-example {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
line-height: 2;
padding: 16px 20px;
background: rgba(13,11,9,0.9);
border: 1px solid var(--smoke-light);
border-radius: 2px;
margin: 16px 0;
overflow-x: auto;
}
.msg-server { color: var(--parchment-faded); }
.msg-player { color: var(--frost); }
.msg-system { color: var(--text-ghost); font-style: italic; }
.msg-broadcast { color: var(--ember-glow); }
.msg-gold { color: var(--gold); }
/* ═══ TIP STRIP ═══ */
.tip-strip {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 16px;
margin: 20px 0;
}
@media (max-width: 560px) {
.tip-strip { grid-template-columns: 1fr; }
}
.tip-card {
padding: 16px 18px;
background: linear-gradient(135deg, rgba(26,23,20,0.9), rgba(42,37,32,0.5));
border: 1px solid rgba(90,82,68,0.15);
border-radius: 2px;
}
.tip-card-label {
font-family: 'Cinzel', serif;
font-size: 10px;
letter-spacing: 0.15em;
color: var(--text-ghost);
text-transform: uppercase;
margin-bottom: 6px;
}
.tip-card-text {
font-size: 14px;
line-height: 1.6;
color: var(--text-dim);
}
/* ═══ NPC GUIDE ═══ */
.npc-guide {
display: flex;
gap: 16px;
align-items: flex-start;
padding: 16px 0;
border-bottom: 1px solid rgba(90,82,68,0.1);
}
.npc-guide:last-child { border-bottom: none; }
.npc-guide-icon {
font-size: 24px;
flex-shrink: 0;
opacity: 0.7;
margin-top: 2px;
}
.npc-guide-name {
font-family: 'Cinzel', serif;
font-size: 14px;
font-weight: 600;
color: var(--parchment);
margin-bottom: 2px;
}
.npc-guide-role {
font-family: 'JetBrains Mono', monospace;
font-size: 9px;
color: var(--text-ghost);
letter-spacing: 0.15em;
text-transform: uppercase;
margin-bottom: 6px;
}
.npc-guide-desc {
font-size: 14px;
line-height: 1.6;
color: var(--text-dim);
}
/* ═══ FOOTER ═══ */
.page-footer {
text-align: center;
padding: 36px 0 48px;
border-top: 1px solid rgba(90,82,68,0.15);
margin-top: 20px;
font-family: 'JetBrains Mono', monospace;
font-size: 10px;
color: var(--text-ghost);
letter-spacing: 0.15em;
}
.page-footer a {
color: var(--text-ghost);
text-decoration: none;
transition: color 0.2s;
}
.page-footer a:hover { color: var(--ember); }
</style>
</head>
<body>
<canvas id="ember-canvas"></canvas>
<div class="page-wrap">
<!-- NAV -->
<nav class="nav-bar">
<a class="nav-home" href="#">The Last Ember</a>
<span style="color:var(--smoke-light)">·</span>
<a class="nav-link" href="#">Board</a>
<a class="nav-link" href="#">Chronicle</a>
<a class="nav-link" href="#">Journals</a>
<a class="nav-link active">How to Play</a>
</nav>
<!-- HEADER -->
<div class="page-header">
<h1 class="page-title">How to Play</h1>
<p class="page-subtitle">A text adventure played over radio. Five minutes a day. Thirty days an epoch. No internet required.</p>
</div>
<!-- ════════════════════════════ -->
<!-- WHAT IS THIS -->
<!-- ════════════════════════════ -->
<div class="divider">What is meshMUD</div>
<p class="prose">meshMUD is a multiplayer text adventure that runs over <strong>Meshtastic</strong> — a long-range radio mesh network. There is no internet connection, no app store, no account creation. You play by sending short text messages from your Meshtastic node. The game responds. Everything happens in 150 characters or less.</p>
<p class="prose">It plays like the BBS door games of the early '90s — <em class="place">Legend of the Red Dragon</em>, <em class="place">TradeWars 2002</em> — adapted for radio. Short daily sessions. Asynchronous multiplayer. A shared world where you see evidence of other players without needing to be online at the same time. A dungeon that resets every 30 days.</p>
<p class="prose">You don't need to be a gamer. You don't need to be fast. You need a Meshtastic radio and five minutes.</p>
<div class="callout ember">
<div class="callout-label">The basics</div>
<p class="prose">You wake up in a tavern called <em class="place">The Last Ember</em>. Below it is a dungeon that changes every 30 days. You explore it, fight monsters, find secrets, and help other players push deeper — all by typing short commands over your radio. When the 30 days end, the dungeon resets. Your character persists. The stories stay.</p>
</div>
<!-- ════════════════════════════ -->
<!-- GETTING STARTED -->
<!-- ════════════════════════════ -->
<div class="divider">Getting Started</div>
<p class="prose">If your mesh network is running meshMUD, the game server listens for direct messages from any node. Send it a DM and it responds. That's it.</p>
<div class="flow-steps">
<div class="flow-step">
<div class="flow-step-num">1</div>
<div class="flow-step-content">
<div class="flow-step-label">Send a DM to the game node</div>
<div class="flow-step-text">Find the meshMUD node on your Meshtastic client and send any message. The server responds with a welcome and asks you to pick a class.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">2</div>
<div class="flow-step-content">
<div class="flow-step-label">Pick your class</div>
<div class="flow-step-text">One letter. <code>F</code> for Fighter, <code>C</code> for Caster, <code>R</code> for Rogue. That's your only creation choice — everything else emerges through play.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">3</div>
<div class="flow-step-content">
<div class="flow-step-label">You're in</div>
<div class="flow-step-text">The server drops you in <em class="place">The Last Ember</em> with starting gear and a handful of gold. Type <code>L</code> to look around. Type <code>H</code> for help. You're playing.</div>
</div>
</div>
</div>
<div class="msg-example">
<span class="msg-player">You → meshMUD:</span> <span class="msg-gold">hello</span><br>
<span class="msg-server">meshMUD:</span> Welcome to The Last Ember. Pick a class: (F)ighter (C)aster (R)ogue<br>
<span class="msg-player">You:</span> <span class="msg-gold">F</span><br>
<span class="msg-server">meshMUD:</span> Kael the Fighter. POW:5 DEF:4 SPD:3 HP:30. You stand in the tavern. Type L.<br>
<span class="msg-player">You:</span> <span class="msg-gold">L</span><br>
<span class="msg-server">meshMUD:</span> The Last Ember. Lanterns burn without oil. Grist polishes a glass. Exits: dungeon.<br>
</div>
<!-- ════════════════════════════ -->
<!-- CLASSES -->
<!-- ════════════════════════════ -->
<div class="divider">The Three Classes</div>
<p class="prose">Three stats govern everything: <strong>POW</strong> (offense), <strong>DEF</strong> (survivability), and <strong>SPD</strong> (evasion, initiative, spellcasting). Each class leans into one. You earn 2 stat points per level to allocate however you want — that's where your build takes shape.</p>
<div class="class-cards">
<div class="class-card fighter">
<div class="class-card-icon">⚔</div>
<div class="class-card-name">Fighter</div>
<div class="class-card-stat">POW-FOCUSED</div>
<div class="class-card-desc">High HP. Hits hard. Takes hits. Abilities like Strike, Bash, Rally, Cleave. Passive damage reduction. The front line.</div>
</div>
<div class="class-card caster">
<div class="class-card-icon">✦</div>
<div class="class-card-name">Caster</div>
<div class="class-card-stat">SPD-FOCUSED</div>
<div class="class-card-desc">Low HP. Spells scale on SPD. Bolt, Ward, Blast, Drain. Passive: see enemy stats. Knowledge is power. Fragile is the cost.</div>
</div>
<div class="class-card rogue">
<div class="class-card-icon">◈</div>
<div class="class-card-name">Rogue</div>
<div class="class-card-stat">SPD / BALANCED</div>
<div class="class-card-desc">Stealth and utility. Stab, Dodge, Ambush, Steal. Passive evasion chance. Thrives in the spaces between fights.</div>
</div>
</div>
<!-- ════════════════════════════ -->
<!-- A TYPICAL DAY -->
<!-- ════════════════════════════ -->
<div class="divider">A Typical Day</div>
<p class="prose">A session takes <strong>five to fifteen minutes</strong>. You get 12 dungeon actions per day — enough to explore a few rooms, fight a few monsters, and make progress without burning out. Town actions are always free.</p>
<div class="flow-steps">
<div class="flow-step">
<div class="flow-step-num">1</div>
<div class="flow-step-content">
<div class="flow-step-label">Visit Grist</div>
<div class="flow-step-text">The barkeep tells you what happened while you were gone. Who died, what fell, what the front line looks like. Always free. This is how the world stays alive between sessions.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">2</div>
<div class="flow-step-content">
<div class="flow-step-label">Check the bounty board</div>
<div class="flow-step-text">Shared objectives the whole server works toward. A monster with a communal HP pool. An exploration target. You chip away at it — so does everyone else.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">3</div>
<div class="flow-step-content">
<div class="flow-step-label">Gear up</div>
<div class="flow-step-text">Buy supplies from Torval, heal up with Maren if you need it, spend a bard token at the bar for a hint or buff. All free actions.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">4</div>
<div class="flow-step-content">
<div class="flow-step-label">Enter the dungeon</div>
<div class="flow-step-text">Move room to room, fight what you find, look for secrets, leave messages for other players. Each move or fight costs an action. Twelve per day — spend them wisely.</div>
</div>
</div>
<div class="flow-step">
<div class="flow-step-num">5</div>
<div class="flow-step-content">
<div class="flow-step-label">Return to town</div>
<div class="flow-step-text">Bank your gold before the dungeon takes it. Tomorrow the rooms may have changed, the bounty may be weaker, and someone may have left you a message you need to read.</div>
</div>
</div>
</div>
<!-- ════════════════════════════ -->
<!-- THE TOWN -->
<!-- ════════════════════════════ -->
<div class="divider">The Last Ember — Your Town</div>
<p class="prose">The tavern is the one room that never changes. Epochs wipe the dungeon, reshuffle everything, reshape the world — but <em class="place">The Last Ember</em> stays. Same bar. Same people. Same lanterns that burn without oil and nobody questions anymore.</p>
<p class="prose">Four people live here. They remember you across every wipe.</p>
<div class="npc-guide">
<span class="npc-guide-icon">🍺</span>
<div>
<div class="npc-guide-name">Grist</div>
<div class="npc-guide-role">Barkeep</div>
<div class="npc-guide-desc">Knows everything that happens in the dungeon because everyone tells him and he never forgets. Visit him first every session — he'll catch you up on what you missed. He also runs the bounty board, handles the epoch vote, and trades bard tokens for hints, buffs, and secrets. He doesn't trade because he's kind. He trades because he collects.</div>
</div>
</div>
<div class="npc-guide">
<span class="npc-guide-icon">🩸</span>
<div>
<div class="npc-guide-name">Maren</div>
<div class="npc-guide-role">Healer</div>
<div class="npc-guide-desc">Used to be an adventurer. Went deeper than anyone. Came back done. Heals with her hands, not magic, and it hurts. She charges gold because free healing breeds carelessness. She's the reason you survive long enough to learn from your mistakes.</div>
</div>
</div>
<div class="npc-guide">
<span class="npc-guide-icon">⚖</span>
<div>
<div class="npc-guide-name">Torval</div>
<div class="npc-guide-role">Merchant</div>
<div class="npc-guide-desc">Buys and sells gear. Appraises items by weight and sound. His inventory somehow matches what's in the dungeon each epoch. Nobody asks how. His prices are fair and his stock is real, which is more than you can say for most people in a town built around a hole full of monsters.</div>
</div>
</div>
<div class="npc-guide">
<span class="npc-guide-icon">👁</span>
<div>
<div class="npc-guide-name">Whisper</div>
<div class="npc-guide-role">Sage</div>
<div class="npc-guide-desc">Sits in the same corner. Knows things about the dungeon that change each epoch — lore, connections, what the symbols mean. Speaks in fragments because that's how the information comes to her. Pay attention to her exact words. Players who dismiss her as flavor text miss half the game.</div>
</div>
</div>
<!-- ════════════════════════════ -->
<!-- THE DUNGEON -->
<!-- ════════════════════════════ -->
<div class="divider">The Dungeon</div>
<p class="prose">Four floors. Each one deeper, harder, and stranger than the last. Monsters get meaner. Secrets get subtler. The rooms change every epoch but the structure holds — floor one is where you learn, floor four is where legends are made.</p>
<p class="prose">You carry three pieces of gear: a <strong>weapon</strong>, <strong>armor</strong>, and a <strong>trinket</strong>. The trinket is the wildcard — it might grant a passive ability, boost an unexpected stat, or do something no other slot can. Six tiers of gear across the dungeon. The best stuff doesn't come from shops.</p>
<div class="callout frost">
<div class="callout-label">Death</div>
<p class="prose">Death costs you all the gold you're carrying. Not your gear. Not your level. Just your gold. The question is always the same: do you bank it before you go in, or carry it and risk losing everything? The dungeon teaches you the answer. Usually the hard way.</p>
</div>
<!-- ════════════════════════════ -->
<!-- MULTIPLAYER -->
<!-- ════════════════════════════ -->
<div class="divider">Playing Together</div>
<p class="prose">meshMUD is <strong>asynchronous multiplayer</strong>. You don't need to be online at the same time as anyone else. You see other players through what they leave behind — messages scratched on dungeon walls, bounty progress that wasn't there yesterday, broadcasts announcing who found what and who fell where.</p>
<div class="tip-strip">
<div class="tip-card">
<div class="tip-card-label">Bounties</div>
<div class="tip-card-text">Shared objectives with communal HP pools. You chip away at a target over days. Everyone who contributes shares the reward when it falls.</div>
</div>
<div class="tip-card">
<div class="tip-card-label">Messages</div>
<div class="tip-card-text">Leave 15-character notes in dungeon rooms for others to find. Warnings, tips, coordinates. Dark Souls soapstone, over LoRa.</div>
</div>
<div class="tip-card">
<div class="tip-card-label">Mail</div>
<div class="tip-card-text">Send direct messages to specific players through the barkeep. Coordinate strategy, share secrets, warn someone about what's ahead.</div>
</div>
<div class="tip-card">
<div class="tip-card-label">Broadcasts</div>
<div class="tip-card-text">Major events announce to the whole mesh. Boss kills, rare finds, deaths, front line changes. The world narrates itself.</div>
</div>
</div>
<p class="prose">There is no PvP. All competition runs through leaderboards, bounty races, and endgame objectives. On a small mesh network where everyone knows each other, cooperation is the game.</p>
<!-- ════════════════════════════ -->
<!-- EPOCHS -->
<!-- ════════════════════════════ -->
<div class="divider">The 30-Day Epoch</div>
<p class="prose">Every 30 days, the dungeon resets. New rooms, new monsters, new secrets, new narrative. Your character keeps their name and their history, but gear and gold start fresh. Each epoch has an <strong>endgame mode</strong> — a shared objective the whole server works toward. On day 30, players vote on the next epoch's mode.</p>
<div class="callout gold">
<div class="callout-label">Three endgame modes</div>
<p class="prose"><strong>Hold the Line</strong> — the dungeon regenerates rooms. Push the front line deeper, establish checkpoints that lock in progress. The whole server descends together.</p>
<p class="prose"><strong>Raid Boss</strong> — a massive enemy with thousands of HP squats on the lowest floor. The server chips away over days. Discover its weaknesses. Coordinate the kill.</p>
<p class="prose"><strong>Retrieve &amp; Escape</strong> — an artifact on floor four. Grab it, carry it to the surface. Something unkillable chases the carrier. Other players clear the path, block the pursuer, relay the objective hand-to-hand.</p>
</div>
<p class="prose">On day 15, the <strong>Breach</strong> opens — a surprise mini-zone between floors two and three with its own challenge, its own loot, and its own secrets. You don't know what's inside until it opens.</p>
<!-- ════════════════════════════ -->
<!-- SECRETS -->
<!-- ════════════════════════════ -->
<div class="divider">Secrets &amp; Discovery</div>
<p class="prose">Twenty secrets hide in the dungeon each epoch. Some are behind walls that need a strong arm to break. Some are puzzles spread across multiple rooms. Some are hidden in things Whisper says that nobody thinks to write down. Finding them isn't required — but every secret you uncover gives a real mechanical advantage, and some of them benefit the entire server.</p>
<p class="prose"><strong>Read the room descriptions carefully.</strong> The dungeon tells you where its secrets are. It just doesn't tell you plainly.</p>
<!-- ════════════════════════════ -->
<!-- COMMANDS -->
<!-- ════════════════════════════ -->
<div class="divider">Quick Command Reference</div>
<p class="prose">Every command fits in a short message. Most have single-letter shortcuts. New commands unlock as you level up — the game teaches you as you go.</p>
<div class="callout ember">
<div class="callout-label">Movement &amp; Awareness</div>
<div class="cmd-grid">
<span class="cmd-key">n s e w</span><span class="cmd-desc">Move north, south, east, west</span>
<span class="cmd-key">l</span><span class="cmd-desc">Look — describe current room, show exits</span>
<span class="cmd-key">x [thing]</span><span class="cmd-desc">Examine something in the room <span class="cmd-unlock">LV5</span></span>
<span class="cmd-key">who</span><span class="cmd-desc">List active players <span class="cmd-unlock">LV3</span></span>
</div>
</div>
<div class="callout blood">
<div class="callout-label">Combat</div>
<div class="cmd-grid">
<span class="cmd-key">f</span><span class="cmd-desc">Fight — engage the monster in this room</span>
<span class="cmd-key">a</span><span class="cmd-desc">Attack — basic melee/spell attack</span>
<span class="cmd-key">flee</span><span class="cmd-desc">Attempt to escape combat (SPD-based chance)</span>
</div>
</div>
<div class="callout gold">
<div class="callout-label">Town &amp; NPCs</div>
<div class="cmd-grid">
<span class="cmd-key">barkeep</span><span class="cmd-desc">Talk to Grist — recap, tokens, bounties <span class="cmd-unlock">LV3</span></span>
<span class="cmd-key">heal</span><span class="cmd-desc">Visit Maren — restore HP for gold <span class="cmd-unlock">LV3</span></span>
<span class="cmd-key">shop</span><span class="cmd-desc">Browse Torval's inventory <span class="cmd-unlock">LV3</span></span>
<span class="cmd-key">bank</span><span class="cmd-desc">Deposit gold safely <span class="cmd-unlock">LV3</span></span>
<span class="cmd-key">board</span><span class="cmd-desc">View active bounties <span class="cmd-unlock">LV3</span></span>
</div>
</div>
<div class="callout frost">
<div class="callout-label">Inventory &amp; Character</div>
<div class="cmd-grid">
<span class="cmd-key">i</span><span class="cmd-desc">Inventory — show gear and backpack <span class="cmd-unlock">LV2</span></span>
<span class="cmd-key">st</span><span class="cmd-desc">Stats — show POW, DEF, SPD, HP, gold, level</span>
<span class="cmd-key">equip [item]</span><span class="cmd-desc">Equip an item from your backpack <span class="cmd-unlock">LV2</span></span>
<span class="cmd-key">use [item]</span><span class="cmd-desc">Use a consumable <span class="cmd-unlock">LV2</span></span>
</div>
</div>
<div class="callout ember">
<div class="callout-label">Social</div>
<div class="cmd-grid">
<span class="cmd-key">msg [text]</span><span class="cmd-desc">Leave a 15-char message in this room <span class="cmd-unlock">LV5</span></span>
<span class="cmd-key">read</span><span class="cmd-desc">Read messages in this room</span>
<span class="cmd-key">rate</span><span class="cmd-desc">Mark a message as helpful</span>
<span class="cmd-key">mail</span><span class="cmd-desc">Check your inbox <span class="cmd-unlock">LV3</span></span>
<span class="cmd-key">mail [who] [text]</span><span class="cmd-desc">Send mail to a player <span class="cmd-unlock">LV3</span></span>
</div>
</div>
<div class="callout gold">
<div class="callout-label">Meta</div>
<div class="cmd-grid">
<span class="cmd-key">h</span><span class="cmd-desc">Help — list all available commands</span>
<span class="cmd-key">h [cmd]</span><span class="cmd-desc">Help on a specific command</span>
<span class="cmd-key">vote</span><span class="cmd-desc">Vote for next epoch's mode (day 30 only) <span class="cmd-unlock">LV1</span></span>
</div>
</div>
<!-- ════════════════════════════ -->
<!-- TIPS -->
<!-- ════════════════════════════ -->
<div class="divider">Grist's Advice for the New Arrival</div>
<div class="callout ember">
<div class="callout-label">Things nobody tells you</div>
<p class="prose"><strong>Bank before you descend.</strong> Death takes everything you're carrying. Not your gear, not your level — just your gold. The bank is free. Use it.</p>
<p class="prose"><strong>Visit Grist every session.</strong> His recap costs nothing and tells you everything you missed. The bounty board is there too. Five seconds of reading saves you from walking into something that killed Sable yesterday.</p>
<p class="prose"><strong>Leave messages.</strong> A 15-character note in a dangerous room saves someone's life tomorrow. This is a small network. Help each other.</p>
<p class="prose"><strong>Read room descriptions.</strong> The dungeon hides things in plain sight. If the text mentions scratches on a wall, there's a reason. If Whisper mumbles about the eastern branch, there's a reason. The game rewards attention.</p>
<p class="prose"><strong>You don't have to fight everything.</strong> Twelve actions is enough for a good day, not enough for a reckless one. Know when to push and when to walk away. The dungeon will be here tomorrow.</p>
<p class="prose"><strong>Bard tokens accrue whether you log in or not.</strong> One per day, cap at five. Spend them at the barkeep for things gold can't buy — hints, buffs, intel. A patient player who saves five tokens gets information that changes everything.</p>
</div>
<!-- FOOTER -->
<div class="page-footer">
<a href="#">The Last Ember</a> · meshMUD
</div>
</div>
<script>
// ═══ EMBER PARTICLES ═══
const canvas = document.getElementById('ember-canvas');
const ctx = canvas.getContext('2d');
let embers = [];
function resize() {
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
}
resize();
window.addEventListener('resize', resize);
class Ember {
constructor() { this.reset(); }
reset() {
this.x = Math.random() * canvas.width;
this.y = canvas.height + 10;
this.size = Math.random() * 2 + 0.5;
this.speedY = -(Math.random() * 0.3 + 0.08);
this.speedX = (Math.random() - 0.5) * 0.2;
this.opacity = Math.random() * 0.35 + 0.1;
this.decay = Math.random() * 0.0007 + 0.0003;
this.wobble = Math.random() * Math.PI * 2;
this.wobbleSpeed = Math.random() * 0.015 + 0.003;
const t = Math.random();
this.r = Math.floor(200 + t * 55);
this.g = Math.floor(80 + t * 80);
this.b = Math.floor(20 + t * 30);
}
update() {
this.wobble += this.wobbleSpeed;
this.x += this.speedX + Math.sin(this.wobble) * 0.1;
this.y += this.speedY;
this.opacity -= this.decay;
if (this.opacity <= 0 || this.y < -20) this.reset();
}
draw() {
ctx.beginPath();
ctx.arc(this.x, this.y, this.size, 0, Math.PI * 2);
ctx.fillStyle = `rgba(${this.r},${this.g},${this.b},${this.opacity})`;
ctx.fill();
ctx.beginPath();
ctx.arc(this.x, this.y, this.size * 2.5, 0, Math.PI * 2);
ctx.fillStyle = `rgba(${this.r},${this.g},${this.b},${this.opacity * 0.1})`;
ctx.fill();
}
}
for (let i = 0; i < 25; i++) {
const e = new Ember();
e.y = Math.random() * canvas.height;
embers.push(e);
}
function animateEmbers() {
ctx.clearRect(0, 0, canvas.width, canvas.height);
embers.forEach(e => { e.update(); e.draw(); });
requestAnimationFrame(animateEmbers);
}
animateEmbers();
</script>
</body>
</html>

1442
projects/mmud/last-ember.html Normal file
View file

File diff suppressed because it is too large Load diff

215
projects/mmud/mmud-phase5-prompt.md Normal file
View file

@ -0,0 +1,215 @@
# Task: MMUD Phase 5 — Endgame Modes
## Before Writing Any Code
Re-read these sections of `/home/zvx/projects/mmud/docs/planned.md`:
- Endgame: Three Rotating Modes (all of it — mode selection, R&E with Pursuer + support roles, Raid Boss with mechanic tables + phases, Hold the Line with regen + checkpoints + floor bosses)
- The Darkcragg Depths (dungeon name context)
- Floor Boss Mechanic Tables (all 4 floor tables)
- Bounties During Hold the Line
Also re-read `config.py` for: HTL_REGEN_ROOMS_PER_DAY, FLOOR_BOSS_MECHANICS, WARDEN_HP/REGEN, RAID_BOSS_HP_PER_PLAYER/CAP/REGEN/MECHANIC_TABLE/PHASES, PURSUER_ADVANCE_RATE/SPAWN_DISTANCE/RELAY_RESET_DISTANCE, WARD/LURE constants, ENDGAME_MODES.
Phase 4 already generates floor bosses, raid boss pre-config, and bounties. This phase wires the runtime game logic for all three modes.
## Phase 5 Deliverables
All three endgame modes are playable. The epoch vote selects the mode. Each mode has its own win condition, progression mechanics, and broadcasts.
### 1. Epoch Vote System
`src/systems/vote.py` (new file)
- Day 30 trigger (already in daytick.py — wire it to the vote system)
- `vote <mode>` command at barkeep — free action. Modes: `retrieve`, `raid`, `hold` (or numbers 1/2/3).
- Votes are public — broadcast on cast: "🗳 {name} voted {mode}."
- Votes can be changed up until epoch ends. UPSERT into epoch_votes table.
- Tally on epoch end: most votes wins. Tiebreak: longest-unplayed mode. No quorum — 1 vote decides if only 1 player votes. Zero votes → longest-unplayed auto-selected.
- `epoch_generate.py` already accepts endgame mode — wire the vote result into the next epoch's generation.
### 2. Hold the Line — Runtime Logic
`src/systems/endgame_htl.py` (new file)
**Room clearing:**
- All dungeon rooms start hostile (`htl_cleared = 0`)
- Killing all monsters in a room sets `htl_cleared = 1` with timestamp
- Regen ticks (already in daytick.py) revert rooms: pick N random cleared non-checkpoint rooms per floor per day, set `htl_cleared = 0`, respawn monsters. Spread ticks across the day (floor 2 at 5/day = 1 room every ~5 hours).
- Cleared rooms are safe — no random encounters. Reverted rooms respawn monsters.
**Checkpoints:**
- Checkpoint rooms are defined during worldgen (is_checkpoint = 1 in rooms table). 3 per floor (hub, midpoint, stairway), 1 on floor 4 (Warden).
- Establishment condition: all rooms in the checkpoint cluster (checkpoint room + all adjacent rooms) must be cleared within one regen window. Check on each room clear — if the cluster is complete, spawn the floor boss.
- Once the floor boss dies, checkpoint locks permanently. `htl_checkpoints.established = 1`. Regen can never revert rooms behind an established checkpoint.
- Final checkpoint on each floor (stairway) unlocks the next floor for all players.
**Floor bosses:**
- Already generated with rolled mechanics (from Phase 4 bossgen).
- Floor boss spawns in the checkpoint room when cluster is cleared. Uses the same shared HP pool / chip-and-run combat as bounties.
- Boss mechanic implementation — each mechanic modifies combat behavior:
- `armored` — damage halved until boss below 50% HP
- `enraged` — boss deals double damage below 50% HP, takes 25% more
- `regenerator` — boss heals 10% HP between sessions (check on engagement, apply since last fight)
- `stalwart` — first flee attempt per engagement always fails
- `warded` — boss has +50% DEF until a discovery secret on the same floor is found (check secret_progress)
- `phasing` — boss is immune to damage on even-numbered epoch days (check epoch.day_number)
- `draining` — boss steals 10% of damage dealt as HP from the attacker
- `splitting` — at 50% HP, boss splits into two half-HP monsters in adjacent rooms. Both must die.
- `rotating_resistance` — immune to the highest stat used by the last player who fought it. Track in DB.
- `retaliator` — reflects 20% of damage back to attacker
- `summoner` — spawns 1 add at start of each engagement. Add must die before boss can be damaged.
- `cursed` — player who dealt most damage last session gets -2 to a random stat next login
- Floor 4 Warden: shared HP pool 300-500, regen at 3%/8h, rolls 2 mechanics from the full table.
- Warden kill = epoch win. Broadcast: "🏆 The Warden has fallen! The Darkcragg Depths are conquered!"
**Broadcasts from DCRG:**
- "🏰 Floor {n} Checkpoint {name} established!"
- "⚠ Floor {n} lost {x} rooms. Frontline at {room}."
- "🏰 Floor {n} unlocked! The descent continues."
- "💀 Floor {n} frontline collapsed to Checkpoint {name}. Rally!"
- Floor boss spawned, floor boss killed, Warden progress.
**Barkeep integration:**
- Grist's recap includes HtL status: floors cleared percentage, checkpoint status, frontline position.
### 3. Raid Boss — Runtime Logic
`src/systems/endgame_raid.py` (new file)
**Activation:**
- On epoch start (if mode is raid_boss), calculate active player count (anyone who entered the dungeon in first 3 days).
- Set raid_boss.hp = 300 × active players, cap 6000. Set raid_boss.hp_max to same value.
- Place boss in a room on floor 3-4 (from pre-generated config in Phase 4).
**Combat:**
- Same chip-and-run as bounties — shared HP pool, engage/damage/flee.
- Regen: 3%/8h (lazy-evaluated like bounties — calculate on engagement).
- Track contributions in raid_boss_contributors.
**Mechanic implementation (2-3 rolled):**
- `windup_strike` — every 3rd combat round, next round deals triple damage unless player uses `defend` or `dodge` action. Add `defend` and `dodge` as combat commands (cost 1 action, negate the windup).
- `flat_damage_boost` — boss damage multiplied by 1.5x
- `retribution` — at 75%/50%/25% HP thresholds, burst damage (2x normal) to the player who pushed it past the threshold
- `aura_damage` — player takes 5% max HP unavoidable damage each combat round regardless of DEF
- `extra_regen` — regen rate becomes 5%/8h instead of 3%/8h
- `armor_phase` — boss takes half damage until: a discovery secret on the floor is found, OR 5+ unique players have contributed damage
- `boss_flees` — at 75%/50%/25% HP, boss relocates to random room on same floor. Broadcast from DCRG: "🐉 The {boss} has fled to somewhere on Floor {n}!" Players must find it.
- `regen_burst` — once per day at a random hour, boss heals 15% max HP in one tick. Trackable through observation.
- `no_escape` — below 25% HP, all flee attempts fail. Fight to the death.
- `summoner` — 1-2 adds spawn per engagement, must be killed before boss takes damage
- `lockout` — after engaging, player can't reengage for 24 hours. Store lockout_until in raid_boss_contributors.
- `enrage_timer` — after 5 combat rounds in a single engagement, boss damage doubles each subsequent round
**Phases:**
- Phase transitions at 66% and 33% HP (`RAID_BOSS_PHASES`).
- At each threshold, rolled mechanics intensify. Implementation: each mechanic has a `phase_modifier(phase_num)` that scales its effect. E.g., summoner spawns 1 add in phase 1, 2 in phase 2, 3 in phase 3. Windup goes from every 3rd round to every 2nd.
- Phase transition broadcasts from DCRG: "🐉 The {boss} enters its second phase!"
**Win condition:** Boss HP reaches 0. All contributors rewarded. Killing blow gets bonus. Broadcast: "🏆 The {boss} has been slain! Victory belongs to the Darkcragg!"
### 4. Retrieve and Escape — Runtime Logic
`src/systems/endgame_rne.py` (new file)
**Setup:**
- Guardian monster placed on floor 4 during epoch generation (add to bossgen if not already there — a strong but non-boss monster guarding the objective).
- `escape_run` table tracks run state.
**Claiming the objective:**
- Player defeats the guardian on floor 4 → objective claimed. `escape_run.active = 1`, carrier set, pursuer spawns.
- Broadcast from DCRG: "👑 {name} claimed the {objective}! The Pursuer stirs."
- Monster spawn rates double on all floors (multiply spawn chance by `ESCAPE_SPAWN_RATE_MULTIPLIER`).
**Pursuer:**
- Tracks carrier. Advances 1 room toward carrier every 2 carrier actions (`PURSUER_ADVANCE_RATE`).
- Spawns 3 rooms behind carrier (`PURSUER_SPAWN_DISTANCE`).
- Track pursuer position in `escape_run.pursuer_room_id`. Track fractional ticks in `pursuer_ticks`.
- On every carrier action: increment pursuer_ticks. When pursuer_ticks >= PURSUER_ADVANCE_RATE, advance pursuer 1 room toward carrier (pathfind shortest route), reset ticks.
- When pursuer enters carrier's room: forced combat. Pursuer is invulnerable (takes no damage). Hits hard. Carrier can only flee. Flee uses normal SPD check. Success = carrier moves 1 room. Failure = take damage + try again next action.
**Carrier death and relay:**
- Carrier dies → objective drops at death room. Broadcast from DCRG: "💀 The carrier has fallen on Floor {n}. The {objective} lies unguarded."
- `escape_run.objective_dropped = 1`, `dropped_room_id` set.
- Any player can `pickup` the objective in that room.
- On pickup: pursuer resets to 5 rooms behind new carrier (`PURSUER_RELAY_RESET_DISTANCE`). Broadcast: "👑 {name} picks up the {objective}! The Pursuer resets."
- Death penalty still applies to the dead carrier (gold loss, respawn in town).
**Three support roles:**
**Blockers:**
- Non-carrier in a room between pursuer and carrier. When pursuer reaches a blocker's room, forced combat with the blocker instead of advancing.
- Blocker can't kill pursuer (invulnerable). Each round blocker survives = 1 round pursuer isn't moving.
- Blocker can flee (normal SPD check). Blocker can die.
- Broadcast: "🛡 {name} is blocking the Pursuer on Floor {n}!" and "💀 {name} fell holding the line. The Pursuer advances."
- Implementation: on pursuer advance, check if any player is in the target room. If yes, pursuer enters combat with them instead of continuing.
**Warders:**
- `ward` command in a cleared dungeon room (1 extra action after clearing = `WARD_ACTION_COST`). Sets `rooms.ward_active = 1`.
- Warded room slows pursuer — takes 2 advance ticks to pass through instead of 1 (`WARD_PURSUER_SLOWDOWN`).
- Ward breaks after one use (reset to 0 when pursuer passes through).
- No broadcast on warding — silent preparation.
**Lures:**
- `lure` command when on same floor as pursuer. Costs 2 actions (`LURE_ACTION_COST`).
- Pursuer diverts toward lure player for 3 ticks (`LURE_DIVERT_TICKS`), then snaps back to carrier tracking. Total delay ~6 ticks (`LURE_TOTAL_DELAY_TICKS`) including backtrack.
- Broadcast: "🎯 {name} lured the Pursuer into {room}! It diverts."
- After divert expires: "👁 The Pursuer has reacquired the carrier."
**Pursuer distance broadcasts (from DCRG):**
- "👁 The Pursuer is {n} rooms behind the carrier." (every 5 carrier actions)
- "👁 The Pursuer is 3 rooms behind. It's closing."
- "⚠ The Pursuer has reached the carrier!"
**Win condition:** Any player delivers objective to town (The Last Ember). Broadcast: "🏆 The {objective} has reached the surface! Victory belongs to the Darkcragg!"
- All participants get epoch win credit (tracked in escape_participants by role).
### 5. Mode Activation in Engine
Update `src/core/engine.py` and `src/core/actions.py`:
- On game start, check epoch.endgame_mode. Load the appropriate endgame system.
- Mode-specific commands only available when that mode is active:
- HtL: checkpoint status command, floor control display
- Raid: raid boss status command (`boss` — show HP, phase, mechanics discovered so far)
- R&E: `pickup`, `ward`, `lure`, `block` commands. Carrier status. Pursuer distance.
- Combat system needs to dispatch to endgame boss combat (floor boss, raid boss, pursuer) when the target is a special entity. Same chip-and-run framework but with mechanic overlays.
- Endgame status integrated into barkeep recap and stats display.
### 6. New Combat Commands
For raid boss mechanics:
- `defend` / `def` — defensive stance. Negates windup strike. Costs 1 dungeon action. Does no damage that round.
- `dodge` / `dge` — evasion. Negates windup strike. Costs 1 dungeon action. Does no damage that round.
For R&E:
- `pickup` — pick up dropped objective in current room. Free action.
- `ward` — ward current room after clearing it. 1 dungeon action.
- `lure` — lure the Pursuer. 2 dungeon actions.
- `block` — (passive) just being in the pursuer's path triggers blocking. No explicit command needed — the system detects it. But add a `block` info command that shows: "Stand in the Pursuer's path to block. It will fight you instead of advancing."
## Rules
- All responses under 150 chars. Test this.
- All broadcasts route through DCRG node, not EMBR.
- Endgame mode commands are only available when that mode is active. Other mode commands return: "That doesn't apply this epoch."
- Boss combat uses the same chip-and-run framework as bounties — shared HP pool, damage persists, flee to disengage.
- Floor boss and raid boss regen is lazy-evaluated (calculate accumulated regen on engagement).
- Use constants from `config.py`.
- Raw parameterized SQL, no ORM.
- Commit after each mode is working (3 major commits minimum).
## Testing
Add to `tests/`:
- `tests/test_vote.py` — vote casting, changing, public broadcast, tally, tiebreak, zero-vote fallback
- `tests/test_htl.py` — room clearing, regen ticks, checkpoint cluster detection, checkpoint establishment, floor boss spawn on cluster clear, floor unlock, Warden kill = win, rooms behind checkpoint immune to regen
- `tests/test_boss_mechanics.py` — test each of the 12 mechanic implementations: armored, enraged, regenerator, stalwart, warded, phasing, draining, splitting, rotating_resistance, retaliator, summoner, cursed. Test phase scaling for raid boss.
- `tests/test_raid.py` — HP scaling from active players, cap at 6000, regen, phase transitions at 66%/33%, contribution tracking, lockout mechanic, completion + rewards
- `tests/test_rne.py` — objective claim, pursuer advancement (2:1 ratio), pursuer in carrier room triggers combat, carrier death drops objective, relay pickup resets pursuer, ward slows pursuer, lure diverts pursuer, blocker intercepts pursuer, win condition on town delivery
- `tests/test_rne_broadcasts.py` — all R&E broadcasts fire correctly (claim, distance, blocker, lure, death, relay, victory)
Use in-memory SQLite for tests. All endgame tests should generate a proper epoch first (use epoch_generate with DummyBackend).
## Done When
All three endgame modes are playable end-to-end. A Hold the Line epoch can be won by clearing all floors and killing the Warden. A Raid Boss epoch can be won by depleting the boss HP pool through coordinated chip-and-run combat with mechanic discovery. A Retrieve and Escape epoch can be won through a relay of carriers with blockers, warders, and lures supporting. The epoch vote selects the next mode. All broadcasts route through DCRG. All responses under 150 chars, all tests passing. Commit and report.

120
projects/mmud/mmud-phase6-prompt.md Normal file
View file

@ -0,0 +1,120 @@
# Task: MMUD Phase 6 — The Breach
## Before Writing Any Code
Re-read these sections of `/home/zvx/projects/mmud/docs/planned.md`:
- The Breach — Mid-Epoch Event (Day 15) (all of it — 4 mini-events, endgame interaction, design rationale)
- Breach secrets (the 3 breach-type secrets)
Also re-read `config.py` for: BREACH_ROOMS_MIN/MAX, BREACH_CONNECTS_FLOORS, BREACH_SECRETS, BREACH_MINI_EVENTS, EMERGENCE_HP, INCURSION_REGEN/HOLD_HOURS.
Phase 4 already generates the Breach zone (breachgen.py) and Phase 4's daytick.py already handles the day 15 trigger and days 12-13 foreshadowing. Phase 4's breach.py has basic state management. This phase wires the full runtime logic for all 4 mini-events.
## Phase 6 Deliverables
The Breach opens on day 15 with a random mini-event. Each of the 4 types plays differently. The Breach interacts with whichever endgame mode is active.
### 1. Breach Activation (verify/extend existing)
The day 15 trigger should already be in daytick.py. Verify it:
- Day 12-13: barkeep foreshadowing broadcasts from DCRG: "The walls grow thin between the second and third depths. Something stirs."
- Day 15: Breach opens. Set `breach.active = 1`. Open the room exits connecting Breach zone to floors 2 and 3. Broadcast from DCRG: "⚡ The ground splits. A new passage has opened between Floors 2 and 3. Strange light pours from within."
- Players can now enter Breach rooms via the new exits from floors 2 and 3.
- The permanent shortcut between floors 2 and 3 persists for the rest of the epoch.
### 2. Mini-Event: The Heist (mini Retrieve & Escape)
`src/systems/breach_heist.py` (new file)
- Artifact in the deepest Breach room, guarded by the Breach mini-boss.
- Kill mini-boss → claim artifact. Carrier must bring it back to town.
- Pursuer spawns (slower, Breach-only — only operates within the 5-8 Breach rooms + the floors 2-3 connection).
- If carrier dies, artifact drops. Any player can pick up.
- Relay mechanics same as R&E but compressed — 5-8 rooms, not 4 floors.
- 3 Breach secrets scattered along the escape route. Found under pressure.
- Completion: artifact delivered to town. Breach rewards distributed. Broadcast from DCRG: "🏆 The artifact has been extracted from the Breach!"
Reuse as much R&E logic from Phase 5 as possible — shared carrier/pursuer/relay patterns.
### 3. Mini-Event: The Emergence (mini Raid Boss)
`src/systems/breach_emergence.py` (new file)
- Creature with shared HP pool (500-800 HP, `EMERGENCE_HP_MIN/MAX`) sits in central Breach room.
- Surrounding rooms spawn minions on a timer (respawn every 8 hours).
- Same chip-and-run combat as bounties/raid boss. Regen at 3%/8h.
- 3 Breach secrets are in the minion rooms — discovered while contributing to the kill.
- Completion: creature HP reaches 0. Broadcast: "🏆 The Breach creature has been destroyed!"
Reuse raid boss combat framework from Phase 5.
### 4. Mini-Event: The Incursion (mini Hold the Line)
`src/systems/breach_incursion.py` (new file)
- Breach rooms start fully hostile. Regen at 2 rooms/day (`INCURSION_REGEN_ROOMS_PER_DAY`) within just 5-8 rooms.
- Players must clear ALL Breach rooms and hold them all for 48 hours (`INCURSION_HOLD_HOURS`).
- If any room reverts during the hold timer, the clock resets.
- 3 Breach secrets behind the hardest rooms, found as part of the push.
- Track hold start time in `breach.incursion_hold_started_at`. On each regen tick, check if any Breach room reverted — if so, reset the timer.
- Completion: 48 hours with all rooms held. Broadcast: "🏆 The Breach has been secured! The incursion is contained."
Reuse HtL room clearing/regen logic from Phase 5.
### 5. Mini-Event: The Resonance (puzzle dungeon)
`src/systems/breach_resonance.py` (new file)
- No combat focus. Breach rooms contain environmental puzzles.
- 3 Breach secrets ARE the puzzle rewards. Finding all 3 unlocks a bonus cache in the deepest room.
- Puzzles are generated in Phase 4 (breachgen already places Breach secrets). This phase adds the interaction logic:
- `examine` objects in Breach rooms triggers puzzle checks
- Puzzle state tracked per-player in secret_progress
- Sequence puzzles, item-interaction puzzles, cross-room clue puzzles (use the same multi-room puzzle archetypes from the main dungeon)
- Completion: all 3 Breach secrets found by any player(s). Bonus cache unlocked. Broadcast: "🏆 The Resonance has been understood. The Breach yields its secrets."
Soloable by nature — knowledge not stats.
### 6. Breach Interaction with Endgame Modes
Regardless of which mini-event is running, the Breach benefits the active endgame mode:
- **Retrieve & Escape:** The Breach shortcut (floors 2↔3) becomes an alternate escape route. Carrier can path through it. Shorter but Breach content (mini-boss, minions, etc.) may still be there.
- **Raid Boss:** Breach completion (any mini-event) drops a buff item granting +20% damage vs the raid boss for the rest of the epoch. Add to player inventory on Breach completion.
- **Hold the Line:** Breach rooms count as bonus territory toward checkpoint progress on both floors 2 and 3. Cleared Breach rooms contribute to the cleared room count for both floor 2 and floor 3 checkpoints.
### 7. Breach Secret Integration
Verify that the 3 Breach secrets work with the existing discovery system:
- `secrets` command includes Breach secrets in the count after day 15
- Secret milestones (5/10/15/20) fire correctly with Breach secrets included
- Barkeep hints for Breach secrets only available after day 15
- Breach secrets contribute to the completionist reward (all 20 found)
## Rules
- All responses under 150 chars.
- All Breach broadcasts route through DCRG.
- Breach mini-event is always random (selected at epoch gen, never voted).
- Reuse combat/territory frameworks from Phase 5 — don't duplicate code.
- Breach content is inaccessible before day 15. Exits to Breach rooms don't exist until activation.
- Use constants from `config.py`.
- Commit after each mini-event works.
## Testing
Add to `tests/`:
- `tests/test_breach_activation.py` — day 15 trigger, foreshadowing on days 12-13, exits open, Breach accessible, inaccessible before day 15
- `tests/test_breach_heist.py` — mini-boss, artifact claim, mini-pursuer, relay, completion, secrets under pressure
- `tests/test_breach_emergence.py` — shared HP pool, minion respawn, chip-and-run, completion, secrets in minion rooms
- `tests/test_breach_incursion.py` — room clearing, regen within Breach, 48h hold timer, timer reset on revert, completion
- `tests/test_breach_resonance.py` — puzzle interaction, secret discovery, bonus cache unlock, no combat required
- `tests/test_breach_endgame.py` — R&E shortcut, raid boss damage buff, HtL bonus territory
Use in-memory SQLite for tests. Generate full epoch with DummyBackend for each test.
## Done When
The Breach opens on day 15 with one of four randomly selected mini-events. Each mini-event is playable end-to-end with its own win condition. Breach secrets integrate cleanly with the discovery system. The Breach interacts with whichever endgame mode is active. All broadcasts route through DCRG. All responses under 150 chars, all tests passing. Commit and report.
This is the final gameplay phase. After this, the full 30-day epoch loop is complete: epoch generates → players explore and progress → Breach opens day 15 → endgame mode pushes through days 20-30 → epoch vote → wipe → new epoch.

152
projects/mmud/mmud-prompts/mmud-prompts/01-update-planned.md Normal file
View file

@ -0,0 +1,152 @@
# Task: Update planned.md with new design sections
Edit `/home/zvx/projects/mmud/docs/planned.md` in place. Three additions plus resolved decisions updates.
---
## Addition 1: The Last Ember — Town Hub
Find the `## Atmosphere & Writing` section. **BEFORE** the `---` divider that separates Atmosphere from `## New Player Onboarding`, insert this new section:
```markdown
---
## The Last Ember — Town Hub
The Last Ember is the one room that never changes. Epochs wipe the dungeon, reskin the narrative, randomize everything — but players always wake up in the same bar, with the same people, who remember them. The lanterns don't burn oil — they just burn. Nobody lights them. Nobody replaces them. The dungeon reshapes itself every 30 days but the Last Ember sits at the mouth of it like a tooth that won't come loose.
The Last Ember is the constant across every epoch, every server, every wipe. It is the frame for the entire game.
### Grist — The Barkeep
Has never left the bar. Not once. Players who've been around for dozens of epochs start to wonder if he *can*. He knows everything that happens in the dungeon — not because he goes there, but because everyone who comes back tells him, and he never forgets. He speaks in short, deliberate sentences. Never wastes a word. He pours drinks that are always exactly what you needed, even if you didn't order.
His recap isn't a service — it's a compulsion. He *has* to tell you what happened. Like the information would burn him if he held it.
He's the bard token system. He trades in stories, not gold. Bring him something interesting — a secret, a discovery, something nobody else knows — and he gives you something back. Information, a temporary edge, a nudge in the right direction. He doesn't trade because he's kind. He trades because he *collects*.
**Mechanical role:** Recap (free), bard token exchange, hints, epoch vote ballot, bounty board.
### Maren — The Healer
Used to be an adventurer. Went deeper than anyone. Came back wrong — not injured, just *done*. She won't say what she saw on the lowest floor. She heals with her hands, not magic, and it hurts. She's efficient, not gentle. She charges gold because she says free healing breeds carelessness, and she's tired of patching people up who didn't respect the dungeon.
She's the only NPC who will occasionally refuse to talk to you if you died doing something stupid — but she still heals you.
She has a scar across her left palm that she got "the last time." She won't say the last time of what.
**Mechanical role:** HP restoration for gold.
### Torval — The Merchant
Doesn't go into the dungeon either, but somehow his inventory matches what's down there each epoch. Nobody asks how. He appraises items by weight and sound — taps gear on the counter, listens, names a price. He's cheerful in a way that feels slightly wrong given where he operates. He tells bad jokes. He calls everyone "friend" and means it exactly zero percent. He'd sell you a cursed sword and sleep fine.
But his prices are fair and his stock is real, which is more than you can say for most people in a town built around a hole full of monsters.
He keeps a ledger that goes back further than the bar. The pages at the front are in a language nobody can read.
**Mechanical role:** Buy, sell, item appraisal.
### Whisper — The Sage
Nobody knows if Whisper is her name or a description of how she talks. She sits in the corner of the Last Ember, always the same corner, and she knows things about the dungeon that change each epoch — lore, history, connections between rooms, what the symbols mean. She speaks in fragments and riddles not because she's trying to be mysterious but because that's how the information comes to her. She describes it like listening to a conversation through a wall.
Her clues are genuine but filtered through whatever broke her ability to just *say things plainly*. Players who pay attention to her exact phrasing find secrets faster. Players who dismiss her as flavor text miss half the game.
She has been the same age for as long as anyone can remember.
**Mechanical role:** Lore hints, secret clues, puzzle guidance (via bard tokens).
### NPC Live Conversations — LLM at Runtime
The "zero LLM at runtime" rule has one exception: talking to NPCs in the Last Ember. Walking up to Grist and having an actual conversation, asking Maren about her scar, trying to get Whisper to speak plainly — these interactions use a live LLM call.
The 150-character limit IS the NPC's personality. Grist is terse by nature. Maren doesn't waste words. Whisper speaks in fragments. Torval talks fast. The constraint is the flavor.
**Command:** `talk <npc>` or `talk <npc> <message>` — free action (in town only). Opens or continues a conversation.
**System prompt per NPC includes:**
- Full backstory and personality card
- Current game state injection: active bounties, recent deaths, Breach status, epoch day, floor control percentages, raid boss HP — whatever is relevant. The NPC *knows what's happening.*
- Hard rules: respond in character, NEVER break character, response MUST be under 150 characters, never reveal exact secret locations or puzzle solutions (hints only), never acknowledge being an AI, never discuss anything outside the game world.
**What each NPC brings:**
- **Grist** — gossip and world state. Knows everything from broadcast logs. Ask about another player and he'll tell you what they've been up to. Dry, factual, slightly unsettling in how much he knows.
- **Maren** — the human element. Comments on your injuries, your play pattern, your stubbornness. Has opinions about the dungeon. Will never talk about what she saw on the lowest floor no matter how hard you try.
- **Torval** — comic relief and commerce. Banter about items, terrible jokes, comments on your gear. "You're wearing THAT to floor 3? Bold." Embellished sales pitches.
- **Whisper** — lore oracle. High-skill conversation. Speaks in fragments. Ask the right questions and get real, useful information about secrets. Her cryptic style is the LLM prompt, not a gimmick — talking to Whisper IS a puzzle.
**Guardrails:**
- Conversation memory is session-only — NPCs don't remember yesterday's chat. Keeps context windows small and prevents exploit accumulation.
- If the LLM fails or times out, fall back to a random pre-generated dialogue snippet from the batch pipeline (20 per NPC already generated at epoch start).
- No rate limit on NPC conversations. Players can talk as long as they want. The NPCs are storytellers and historians — extended conversation is a feature, not abuse.
- Uses the same pluggable LLM backend as the epoch generation pipeline (Anthropic, OpenAI, Google, or Dummy).
**Server History Seed — 2 Years of Lore:**
Before the server goes live, generate 24 epochs (2 years) of simulated history. Each epoch gets: number, endgame mode, Breach type, narrative theme, win/loss result, 3-5 notable players (generated names, classes, what they did), 1-2 memorable moments, hall of fame entries, titles earned. Stored in the persistent tables. When the real server starts on epoch 25, the NPCs have 24 epochs of stories to tell. A compressed lore packet (20-30 sentences of highlights) is injected into every NPC system prompt and regenerated each epoch as real player history accumulates and blends with seeded history.
**Cost math:** At Haiku-tier pricing, ~500 tokens per turn. Even heavy usage (50+ turns/day across all players) is ~$0.006/day. Unlimited conversation is essentially free.
```
---
## Addition 2: Command Discovery
Find the `## New Player Onboarding` section. After the "Daily Tips" subsection and before the `---` divider that separates it from `## Resolved Decisions`, insert:
```markdown
### Command Discovery — No Guessing on Slow Radio
On a 45-60 second radio round-trip, guessing a command and getting "Unknown command" is unacceptable. Every interaction point should make available commands visible.
**First connect message:** Include core commands explicitly. Not "type H for help" — actually list them. `Move:N/S/E/W Fight:F Look:L Flee:FL Stats:ST Help:H` fits in 150 chars and gives a new player everything for their first session.
**Smart error responses:** Never just "Unknown command." Always suggest valid commands based on current player state:
- In town: `Unknown. Try: BAR SHOP HEAL BANK TRAIN ENTER H(elp)`
- In dungeon: `Unknown. Try: F(ight) FL(ee) L(ook) N/S/E/W H(elp)`
- In combat: `Unknown. Try: F(ight) FL(ee) STATS`
- Dead: `Unknown. You're dead. Type RESPAWN.`
**Context-sensitive help (H command):** `H` alone shows commands available in current state. `H <cmd>` gives specific help. All fits 150 chars. Help output changes based on player level — only shows unlocked commands.
**Barkeep nudges:** When a player visits Grist but hasn't used a system yet, the recap appends a tip: "Tip: try BOUNTY to see active hunts" or "Tip: use MSG to leave notes in rooms." One tip per visit, rotating through unused systems. Stops once the player has tried everything.
**Progressive unlock announcements:** When a command unlocks at a new level, announce it explicitly with usage: "⬆ Level 3! New: SHOP(buy gear) BANK(save gold) MAIL(send messages)"
**Last Ember quick reference:** The spectator web page includes a printable command cheat sheet — a one-page reference players can keep next to their Meshtastic device. Physical reference for a physical radio game.
```
---
## Addition 3: Resolved Decisions
Find the `## Resolved Decisions` section. Add these lines at the end of the list:
```markdown
- Town hub: The Last Ember — persistent bar across all epochs, all servers. Four permanent NPCs: Grist (barkeep), Maren (healer), Torval (merchant), Whisper (sage).
- NPC live conversations: NPCs are sim nodes on the mesh (GRST, MRN, TRVL, WSPR). Players DM them directly. Three rule layers: unknown node gets static onboarding, known player not in bar gets static rejection, known player in bar gets full LLM conversation. No rate limit. Session-only memory. Falls back to pre-generated dialogue on failure. 24-epoch history seed provides 2 years of lore.
- Command discovery: smart error responses show valid commands for current state, barkeep nudges for unused systems, explicit command listing on first connect.
```
---
## Addition 4: LLM Content Pipeline update
Find the `## LLM Content Pipeline` section. Find the line that says `### Decision Rule` and the text `Use LLMs for content that can be validated offline. Use deterministic templates for anything that must be correct in real-time.`
Replace that with:
```markdown
### Decision Rule
Use LLMs for content that can be validated offline. Use deterministic templates for anything that must be correct in real-time. **One exception:** NPC conversations in The Last Ember use live LLM calls — the 150-char response constraint, personality cards, and session-only memory make this safe, cheap, and in-character. See The Last Ember section for details.
```
---
## Commit
```bash
git add -A && git commit -m "Design doc: add Last Ember NPCs, live NPC conversations, command discovery"
```

102
projects/mmud/mmud-prompts/mmud-prompts/02-npc-nodes.md Normal file
View file

@ -0,0 +1,102 @@
# Task: Add NPC sim node architecture and rules to planned.md
Edit `/home/zvx/projects/mmud/docs/planned.md` in place.
---
## Edit 1: NPC Sim Nodes section
Find the `### NPC Live Conversations — LLM at Runtime` subsection inside `## The Last Ember — Town Hub`. Replace the **Command** line and everything after it in that subsection (from `**Command:**` through the end of `**Cost math:**`) with the following:
```markdown
**Network Architecture — NPCs as Mesh Nodes:**
The NPCs are literal Meshtastic nodes on the mesh network. Five sim nodes, all backed by the same game database:
- **EMBR** — The Last Ember. The game server. All game commands go here.
- **GRST** — Grist. DM this node to talk to the barkeep.
- **MRN** — Maren. DM this node to talk to the healer.
- **TRVL** — Torval. DM this node to talk to the merchant.
- **WSPR** — Whisper. DM this node to talk to the sage.
Players don't issue a `talk` command — they DM the NPC's node directly. The game server sees inbound on the NPC node ID, checks the rules below, and routes the response back through that NPC's node. The NPCs are *people on the network*, not menu options.
**Three rule layers (checked in order):**
**Rule 1 — Unknown node (not in the game):** Static in-character rejection with onboarding instructions. No LLM call. Each NPC has a fixed response:
- Grist: `"Don't know you. DM EMBR to start. Then we'll talk."`
- Maren: `"I only patch up adventurers. DM EMBR to become one."`
- Torval: `"No account, no credit, friend. DM EMBR to join up."`
- Whisper: `"...not yet. EMBR. Begin there."`
**Rule 2 — Known player, not in the bar:** Static in-character refusal. Player is in the dungeon, dead, or otherwise not in town. No LLM call.
- Grist: `"You're not here, {name}. Come back to the bar first."`
- Maren: `"I can hear you're still down there. Come back alive."`
- Torval: `"I don't do deliveries. Get back to the Ember."`
- Whisper: `"...too far. Return."`
**Rule 3 — Known player, in the bar:** Full LLM conversation. This is the only case that triggers a live LLM call.
**System prompt per NPC includes:**
- Full backstory and personality card
- Current game state injection: active bounties, recent deaths, Breach status, epoch day, floor control percentages, raid boss HP — whatever is relevant. The NPC *knows what's happening.*
- Hard rules: respond in character, NEVER break character, response MUST be under 150 characters, never reveal exact secret locations or puzzle solutions (hints only), never acknowledge being an AI, never discuss anything outside the game world.
**What each NPC brings:**
- **Grist** — gossip and world state. Knows everything from broadcast logs. Ask about another player and he'll tell you what they've been up to. Dry, factual, slightly unsettling in how much he knows.
- **Maren** — the human element. Comments on your injuries, your play pattern, your stubbornness. Has opinions about the dungeon. Will never talk about what she saw on the lowest floor no matter how hard you try.
- **Torval** — comic relief and commerce. Banter about items, terrible jokes, comments on your gear. "You're wearing THAT to floor 3? Bold." Embellished sales pitches.
- **Whisper** — lore oracle. High-skill conversation. Speaks in fragments. Ask the right questions and get real, useful information about secrets. Her cryptic style is the LLM prompt, not a gimmick — talking to Whisper IS a puzzle.
**Guardrails:**
- Conversation memory is session-only — NPCs don't remember yesterday's chat. Keeps context windows small and prevents exploit accumulation.
- If the LLM fails or times out, fall back to a random pre-generated dialogue snippet from the batch pipeline (20 per NPC already generated at epoch start).
- No rate limit on NPC conversations. Players can talk as long as they want. The NPCs are storytellers, historians, and characters — extended conversation is a feature, not abuse.
- Uses the same pluggable LLM backend as the epoch generation pipeline (Anthropic, OpenAI, Google, or Dummy).
**Server History Seed — 2 Years of Lore:**
Before the server goes live, generate 24 epochs (2 years) of simulated history. For each epoch:
- Epoch number, endgame mode, Breach type, narrative theme
- Whether the server won or lost (mix of both — some epic victories, some heartbreaking failures)
- 3-5 notable players per epoch (generated names, classes, levels reached, what they did)
- 1-2 memorable moments per epoch ("Kira carried the Crown from floor 4 to floor 1 with 3 HP", "The Warden stood for 28 days — the server failed on the final push", "Epoch 11's Raid Boss had No Escape + Enraged — three players died on the killing blow")
- Hall of fame entries, titles earned
Stored in the persistent tables (accounts, hall_of_fame, hall_of_fame_participants, titles). When the real server starts on epoch 25, the NPCs have 24 epochs of history to draw from. Grist drops names of old champions. Maren compares your injuries to legends. Torval mentions gear from epochs past. Whisper sees patterns across cycles that nobody else notices.
**NPC context injection includes a lore packet:** A compressed 20-30 sentence summary of server history highlights pulled from the hall of fame tables. Regenerated at each epoch start so it stays current as real player history accumulates and blends with the seed history. The NPCs don't distinguish between seeded and real history — it's all the same to them.
**Cost math:** At Haiku-tier pricing, ~500 tokens per conversation turn. Even heavy usage (50 turns/day across all players) is ~25,000 tokens/day ≈ $0.006/day. Unlimited conversation is essentially free.
```
---
## Edit 2: Update Resolved Decisions
Find the resolved decision line that says:
```
- NPC live conversations: LLM at runtime exception for talk command in town. Session-only memory, 5/day rate limit per NPC, falls back to pre-generated dialogue on failure.
```
Replace it with:
```
- NPC live conversations: NPCs are sim nodes on the mesh (GRST, MRN, TRVL, WSPR). Players DM them directly. Three rule layers: unknown node gets static onboarding response, known player not in bar gets static rejection, known player in bar gets full LLM conversation. No rate limit. Session-only memory. Falls back to pre-generated dialogue on failure. 24-epoch history seed provides 2 years of lore for NPCs to draw from.
```
---
## Edit 3: Add to Open Questions
Add this to the end of the `## Open Questions` list:
```
- NPC sim node deployment — which host runs meshtasticd with 5 identities, TCP routing to game LXC
```
---
## Commit
```bash
git add -A && git commit -m "Design doc: NPCs as mesh sim nodes, three-layer access rules, onboarding funnel"
```

61
projects/mmud/mmud-prompts/mmud-prompts/03-darkcragg.md Normal file
View file

@ -0,0 +1,61 @@
# Task: Add Darkcragg Depths dungeon name to planned.md
Edit `/home/zvx/projects/mmud/docs/planned.md` in place.
---
## Edit 1: Dungeon section
Find the `## Dungeon` section. Find the line `### Hub-Spoke Layout with Loops`. Insert a new subsection BEFORE it:
```markdown
### The Darkcragg Depths
The dungeon is always the Darkcragg Depths. Like the Last Ember, the name is a constant — it persists across every epoch, every server, every wipe. The floors reskin, the layout regenerates, the monsters change, but the Depths are always the Depths. Players descend into the Darkcragg. They talk about the Darkcragg. It's a proper noun, not a generic dungeon.
The four floors are narratively re-skinned each epoch (Sunken Halls, Fungal Depths, Ember Caverns, Void Reach are defaults — the LLM pipeline may rename them) but the Darkcragg Depths is the name on the door every time.
```
---
## Edit 2: Update The Last Ember section
Find the paragraph in `## The Last Ember — Town Hub` that starts with "The Last Ember is the constant across every epoch". Replace that single line with:
```markdown
The Last Ember and the Darkcragg Depths are the two constants across every epoch, every server, every wipe. The bar and the hole it sits on top of. Everything else changes. These don't.
```
---
## Edit 3: Update NPC rejection messages
Find the Rule 2 static responses for known players not in the bar. Update Maren's line:
Replace:
```
- Maren: `"I can hear you're still down there. Come back alive."`
```
With:
```
- Maren: `"I can hear you're still in the Darkcragg. Come back alive."`
```
---
## Edit 4: Resolved Decisions
Add to the end of the `## Resolved Decisions` list:
```markdown
- Dungeon name: The Darkcragg Depths — persistent across all epochs like the Last Ember. Floor names reskin per epoch but the Darkcragg is always the Darkcragg.
```
---
## Commit
```bash
git add -A && git commit -m "Design doc: the dungeon is the Darkcragg Depths"
```

75
projects/mmud/mmud-prompts/mmud-prompts/04-dcrg-node.md Normal file
View file

@ -0,0 +1,75 @@
# Task: Add Darkcragg Depths broadcast node to planned.md
Edit `/home/zvx/projects/mmud/docs/planned.md` in place.
---
## Edit 1: Update NPC Sim Nodes section
Find the `**Network Architecture — NPCs as Mesh Nodes:**` block inside `## The Last Ember — Town Hub`. Replace the node list (the 5-item bullet list starting with `- **EMBR**` through `- **WSPR**`) with:
```markdown
- **EMBR** — The Last Ember. The game server. All game commands go here. Responds with direct action results only.
- **DCRG** — The Darkcragg Depths. One-way broadcast node. All dungeon events come from here — deaths, bounty progress, Breach opening, regen ticks, boss phase transitions, discoveries, level-ups. Does not accept commands. The dungeon is alive on the network.
- **GRST** — Grist. DM this node to talk to the barkeep.
- **MRN** — Maren. DM this node to talk to the healer.
- **TRVL** — Torval. DM this node to talk to the merchant.
- **WSPR** — Whisper. DM this node to talk to the sage.
This splits two distinct streams: EMBR only sends direct responses to your actions. DCRG is the ambient feed of what's happening in the world. The NPCs are people you talk to. Six nodes total, one game DB backing all of them.
```
---
## Edit 2: Add DCRG rules to the rule layers
Find `**Three rule layers (checked in order):**`. Insert a new section BEFORE Rule 1:
```markdown
**DCRG rules (broadcast node):**
- DCRG never accepts inbound messages. If a player or unknown node DMs DCRG, it responds with a static message: `"The Darkcragg does not answer. It only speaks. DM EMBR to play."`
- All tier 1 and tier 2 broadcasts are sent FROM the DCRG node, not EMBR.
- Targeted broadcasts (multi-room puzzle feedback) are also sent from DCRG as DMs to qualifying players.
- DCRG is the voice of the dungeon. When someone dies, when the Breach opens, when a bounty falls — it comes from the Darkcragg.
```
---
## Edit 3: Update Broadcast System section
Find `## Broadcast System`. Find the first paragraph or description of how broadcasts work. Add this line at the end of the introductory text, before any subsections:
```markdown
All broadcasts are sent from the DCRG (Darkcragg Depths) sim node, not the main EMBR game node. This separates the ambient world feed from direct command responses. EMBR talks to you. The Darkcragg talks about everyone.
```
---
## Edit 4: Update Resolved Decisions
Find the resolved decision about NPC live conversations that starts with `- NPC live conversations: NPCs are sim nodes`. Replace it with:
```markdown
- Mesh node architecture: 6 sim nodes — EMBR (game commands + responses), DCRG (one-way dungeon broadcasts), GRST/MRN/TRVL/WSPR (NPC conversations). One game DB backs all of them.
- NPC conversations: Players DM NPC nodes directly. Three rule layers: unknown node gets static onboarding, known player not in bar gets static rejection, known player in bar gets full LLM conversation. Session-only memory, 5/day rate limit per NPC, falls back to pre-generated dialogue on failure.
- DCRG is broadcast-only — does not accept commands. All tier 1/2 and targeted broadcasts route through DCRG.
```
---
## Edit 5: Update Open Questions
Find the open question `- NPC sim node deployment`. Replace it with:
```markdown
- Sim node deployment — which host runs meshtasticd with 6 identities (EMBR, DCRG, GRST, MRN, TRVL, WSPR), TCP routing to game LXC
```
---
## Commit
```bash
git add -A && git commit -m "Design doc: DCRG broadcast node — the dungeon speaks on the mesh"
```

215
projects/mmud/mmud-prompts/mmud-prompts/05-phase5.md Normal file
View file

@ -0,0 +1,215 @@
# Task: MMUD Phase 5 — Endgame Modes
## Before Writing Any Code
Re-read these sections of `/home/zvx/projects/mmud/docs/planned.md`:
- Endgame: Three Rotating Modes (all of it — mode selection, R&E with Pursuer + support roles, Raid Boss with mechanic tables + phases, Hold the Line with regen + checkpoints + floor bosses)
- The Darkcragg Depths (dungeon name context)
- Floor Boss Mechanic Tables (all 4 floor tables)
- Bounties During Hold the Line
Also re-read `config.py` for: HTL_REGEN_ROOMS_PER_DAY, FLOOR_BOSS_MECHANICS, WARDEN_HP/REGEN, RAID_BOSS_HP_PER_PLAYER/CAP/REGEN/MECHANIC_TABLE/PHASES, PURSUER_ADVANCE_RATE/SPAWN_DISTANCE/RELAY_RESET_DISTANCE, WARD/LURE constants, ENDGAME_MODES.
Phase 4 already generates floor bosses, raid boss pre-config, and bounties. This phase wires the runtime game logic for all three modes.
## Phase 5 Deliverables
All three endgame modes are playable. The epoch vote selects the mode. Each mode has its own win condition, progression mechanics, and broadcasts.
### 1. Epoch Vote System
`src/systems/vote.py` (new file)
- Day 30 trigger (already in daytick.py — wire it to the vote system)
- `vote <mode>` command at barkeep — free action. Modes: `retrieve`, `raid`, `hold` (or numbers 1/2/3).
- Votes are public — broadcast on cast: "🗳 {name} voted {mode}."
- Votes can be changed up until epoch ends. UPSERT into epoch_votes table.
- Tally on epoch end: most votes wins. Tiebreak: longest-unplayed mode. No quorum — 1 vote decides if only 1 player votes. Zero votes → longest-unplayed auto-selected.
- `epoch_generate.py` already accepts endgame mode — wire the vote result into the next epoch's generation.
### 2. Hold the Line — Runtime Logic
`src/systems/endgame_htl.py` (new file)
**Room clearing:**
- All dungeon rooms start hostile (`htl_cleared = 0`)
- Killing all monsters in a room sets `htl_cleared = 1` with timestamp
- Regen ticks (already in daytick.py) revert rooms: pick N random cleared non-checkpoint rooms per floor per day, set `htl_cleared = 0`, respawn monsters. Spread ticks across the day (floor 2 at 5/day = 1 room every ~5 hours).
- Cleared rooms are safe — no random encounters. Reverted rooms respawn monsters.
**Checkpoints:**
- Checkpoint rooms are defined during worldgen (is_checkpoint = 1 in rooms table). 3 per floor (hub, midpoint, stairway), 1 on floor 4 (Warden).
- Establishment condition: all rooms in the checkpoint cluster (checkpoint room + all adjacent rooms) must be cleared within one regen window. Check on each room clear — if the cluster is complete, spawn the floor boss.
- Once the floor boss dies, checkpoint locks permanently. `htl_checkpoints.established = 1`. Regen can never revert rooms behind an established checkpoint.
- Final checkpoint on each floor (stairway) unlocks the next floor for all players.
**Floor bosses:**
- Already generated with rolled mechanics (from Phase 4 bossgen).
- Floor boss spawns in the checkpoint room when cluster is cleared. Uses the same shared HP pool / chip-and-run combat as bounties.
- Boss mechanic implementation — each mechanic modifies combat behavior:
- `armored` — damage halved until boss below 50% HP
- `enraged` — boss deals double damage below 50% HP, takes 25% more
- `regenerator` — boss heals 10% HP between sessions (check on engagement, apply since last fight)
- `stalwart` — first flee attempt per engagement always fails
- `warded` — boss has +50% DEF until a discovery secret on the same floor is found (check secret_progress)
- `phasing` — boss is immune to damage on even-numbered epoch days (check epoch.day_number)
- `draining` — boss steals 10% of damage dealt as HP from the attacker
- `splitting` — at 50% HP, boss splits into two half-HP monsters in adjacent rooms. Both must die.
- `rotating_resistance` — immune to the highest stat used by the last player who fought it. Track in DB.
- `retaliator` — reflects 20% of damage back to attacker
- `summoner` — spawns 1 add at start of each engagement. Add must die before boss can be damaged.
- `cursed` — player who dealt most damage last session gets -2 to a random stat next login
- Floor 4 Warden: shared HP pool 300-500, regen at 3%/8h, rolls 2 mechanics from the full table.
- Warden kill = epoch win. Broadcast: "🏆 The Warden has fallen! The Darkcragg Depths are conquered!"
**Broadcasts from DCRG:**
- "🏰 Floor {n} Checkpoint {name} established!"
- "⚠ Floor {n} lost {x} rooms. Frontline at {room}."
- "🏰 Floor {n} unlocked! The descent continues."
- "💀 Floor {n} frontline collapsed to Checkpoint {name}. Rally!"
- Floor boss spawned, floor boss killed, Warden progress.
**Barkeep integration:**
- Grist's recap includes HtL status: floors cleared percentage, checkpoint status, frontline position.
### 3. Raid Boss — Runtime Logic
`src/systems/endgame_raid.py` (new file)
**Activation:**
- On epoch start (if mode is raid_boss), calculate active player count (anyone who entered the dungeon in first 3 days).
- Set raid_boss.hp = 300 × active players, cap 6000. Set raid_boss.hp_max to same value.
- Place boss in a room on floor 3-4 (from pre-generated config in Phase 4).
**Combat:**
- Same chip-and-run as bounties — shared HP pool, engage/damage/flee.
- Regen: 3%/8h (lazy-evaluated like bounties — calculate on engagement).
- Track contributions in raid_boss_contributors.
**Mechanic implementation (2-3 rolled):**
- `windup_strike` — every 3rd combat round, next round deals triple damage unless player uses `defend` or `dodge` action. Add `defend` and `dodge` as combat commands (cost 1 action, negate the windup).
- `flat_damage_boost` — boss damage multiplied by 1.5x
- `retribution` — at 75%/50%/25% HP thresholds, burst damage (2x normal) to the player who pushed it past the threshold
- `aura_damage` — player takes 5% max HP unavoidable damage each combat round regardless of DEF
- `extra_regen` — regen rate becomes 5%/8h instead of 3%/8h
- `armor_phase` — boss takes half damage until: a discovery secret on the floor is found, OR 5+ unique players have contributed damage
- `boss_flees` — at 75%/50%/25% HP, boss relocates to random room on same floor. Broadcast from DCRG: "🐉 The {boss} has fled to somewhere on Floor {n}!" Players must find it.
- `regen_burst` — once per day at a random hour, boss heals 15% max HP in one tick. Trackable through observation.
- `no_escape` — below 25% HP, all flee attempts fail. Fight to the death.
- `summoner` — 1-2 adds spawn per engagement, must be killed before boss takes damage
- `lockout` — after engaging, player can't reengage for 24 hours. Store lockout_until in raid_boss_contributors.
- `enrage_timer` — after 5 combat rounds in a single engagement, boss damage doubles each subsequent round
**Phases:**
- Phase transitions at 66% and 33% HP (`RAID_BOSS_PHASES`).
- At each threshold, rolled mechanics intensify. Implementation: each mechanic has a `phase_modifier(phase_num)` that scales its effect. E.g., summoner spawns 1 add in phase 1, 2 in phase 2, 3 in phase 3. Windup goes from every 3rd round to every 2nd.
- Phase transition broadcasts from DCRG: "🐉 The {boss} enters its second phase!"
**Win condition:** Boss HP reaches 0. All contributors rewarded. Killing blow gets bonus. Broadcast: "🏆 The {boss} has been slain! Victory belongs to the Darkcragg!"
### 4. Retrieve and Escape — Runtime Logic
`src/systems/endgame_rne.py` (new file)
**Setup:**
- Guardian monster placed on floor 4 during epoch generation (add to bossgen if not already there — a strong but non-boss monster guarding the objective).
- `escape_run` table tracks run state.
**Claiming the objective:**
- Player defeats the guardian on floor 4 → objective claimed. `escape_run.active = 1`, carrier set, pursuer spawns.
- Broadcast from DCRG: "👑 {name} claimed the {objective}! The Pursuer stirs."
- Monster spawn rates double on all floors (multiply spawn chance by `ESCAPE_SPAWN_RATE_MULTIPLIER`).
**Pursuer:**
- Tracks carrier. Advances 1 room toward carrier every 2 carrier actions (`PURSUER_ADVANCE_RATE`).
- Spawns 3 rooms behind carrier (`PURSUER_SPAWN_DISTANCE`).
- Track pursuer position in `escape_run.pursuer_room_id`. Track fractional ticks in `pursuer_ticks`.
- On every carrier action: increment pursuer_ticks. When pursuer_ticks >= PURSUER_ADVANCE_RATE, advance pursuer 1 room toward carrier (pathfind shortest route), reset ticks.
- When pursuer enters carrier's room: forced combat. Pursuer is invulnerable (takes no damage). Hits hard. Carrier can only flee. Flee uses normal SPD check. Success = carrier moves 1 room. Failure = take damage + try again next action.
**Carrier death and relay:**
- Carrier dies → objective drops at death room. Broadcast from DCRG: "💀 The carrier has fallen on Floor {n}. The {objective} lies unguarded."
- `escape_run.objective_dropped = 1`, `dropped_room_id` set.
- Any player can `pickup` the objective in that room.
- On pickup: pursuer resets to 5 rooms behind new carrier (`PURSUER_RELAY_RESET_DISTANCE`). Broadcast: "👑 {name} picks up the {objective}! The Pursuer resets."
- Death penalty still applies to the dead carrier (gold loss, respawn in town).
**Three support roles:**
**Blockers:**
- Non-carrier in a room between pursuer and carrier. When pursuer reaches a blocker's room, forced combat with the blocker instead of advancing.
- Blocker can't kill pursuer (invulnerable). Each round blocker survives = 1 round pursuer isn't moving.
- Blocker can flee (normal SPD check). Blocker can die.
- Broadcast: "🛡 {name} is blocking the Pursuer on Floor {n}!" and "💀 {name} fell holding the line. The Pursuer advances."
- Implementation: on pursuer advance, check if any player is in the target room. If yes, pursuer enters combat with them instead of continuing.
**Warders:**
- `ward` command in a cleared dungeon room (1 extra action after clearing = `WARD_ACTION_COST`). Sets `rooms.ward_active = 1`.
- Warded room slows pursuer — takes 2 advance ticks to pass through instead of 1 (`WARD_PURSUER_SLOWDOWN`).
- Ward breaks after one use (reset to 0 when pursuer passes through).
- No broadcast on warding — silent preparation.
**Lures:**
- `lure` command when on same floor as pursuer. Costs 2 actions (`LURE_ACTION_COST`).
- Pursuer diverts toward lure player for 3 ticks (`LURE_DIVERT_TICKS`), then snaps back to carrier tracking. Total delay ~6 ticks (`LURE_TOTAL_DELAY_TICKS`) including backtrack.
- Broadcast: "🎯 {name} lured the Pursuer into {room}! It diverts."
- After divert expires: "👁 The Pursuer has reacquired the carrier."
**Pursuer distance broadcasts (from DCRG):**
- "👁 The Pursuer is {n} rooms behind the carrier." (every 5 carrier actions)
- "👁 The Pursuer is 3 rooms behind. It's closing."
- "⚠ The Pursuer has reached the carrier!"
**Win condition:** Any player delivers objective to town (The Last Ember). Broadcast: "🏆 The {objective} has reached the surface! Victory belongs to the Darkcragg!"
- All participants get epoch win credit (tracked in escape_participants by role).
### 5. Mode Activation in Engine
Update `src/core/engine.py` and `src/core/actions.py`:
- On game start, check epoch.endgame_mode. Load the appropriate endgame system.
- Mode-specific commands only available when that mode is active:
- HtL: checkpoint status command, floor control display
- Raid: raid boss status command (`boss` — show HP, phase, mechanics discovered so far)
- R&E: `pickup`, `ward`, `lure`, `block` commands. Carrier status. Pursuer distance.
- Combat system needs to dispatch to endgame boss combat (floor boss, raid boss, pursuer) when the target is a special entity. Same chip-and-run framework but with mechanic overlays.
- Endgame status integrated into barkeep recap and stats display.
### 6. New Combat Commands
For raid boss mechanics:
- `defend` / `def` — defensive stance. Negates windup strike. Costs 1 dungeon action. Does no damage that round.
- `dodge` / `dge` — evasion. Negates windup strike. Costs 1 dungeon action. Does no damage that round.
For R&E:
- `pickup` — pick up dropped objective in current room. Free action.
- `ward` — ward current room after clearing it. 1 dungeon action.
- `lure` — lure the Pursuer. 2 dungeon actions.
- `block` — (passive) just being in the pursuer's path triggers blocking. No explicit command needed — the system detects it. But add a `block` info command that shows: "Stand in the Pursuer's path to block. It will fight you instead of advancing."
## Rules
- All responses under 150 chars. Test this.
- All broadcasts route through DCRG node, not EMBR.
- Endgame mode commands are only available when that mode is active. Other mode commands return: "That doesn't apply this epoch."
- Boss combat uses the same chip-and-run framework as bounties — shared HP pool, damage persists, flee to disengage.
- Floor boss and raid boss regen is lazy-evaluated (calculate accumulated regen on engagement).
- Use constants from `config.py`.
- Raw parameterized SQL, no ORM.
- Commit after each mode is working (3 major commits minimum).
## Testing
Add to `tests/`:
- `tests/test_vote.py` — vote casting, changing, public broadcast, tally, tiebreak, zero-vote fallback
- `tests/test_htl.py` — room clearing, regen ticks, checkpoint cluster detection, checkpoint establishment, floor boss spawn on cluster clear, floor unlock, Warden kill = win, rooms behind checkpoint immune to regen
- `tests/test_boss_mechanics.py` — test each of the 12 mechanic implementations: armored, enraged, regenerator, stalwart, warded, phasing, draining, splitting, rotating_resistance, retaliator, summoner, cursed. Test phase scaling for raid boss.
- `tests/test_raid.py` — HP scaling from active players, cap at 6000, regen, phase transitions at 66%/33%, contribution tracking, lockout mechanic, completion + rewards
- `tests/test_rne.py` — objective claim, pursuer advancement (2:1 ratio), pursuer in carrier room triggers combat, carrier death drops objective, relay pickup resets pursuer, ward slows pursuer, lure diverts pursuer, blocker intercepts pursuer, win condition on town delivery
- `tests/test_rne_broadcasts.py` — all R&E broadcasts fire correctly (claim, distance, blocker, lure, death, relay, victory)
Use in-memory SQLite for tests. All endgame tests should generate a proper epoch first (use epoch_generate with DummyBackend).
## Done When
All three endgame modes are playable end-to-end. A Hold the Line epoch can be won by clearing all floors and killing the Warden. A Raid Boss epoch can be won by depleting the boss HP pool through coordinated chip-and-run combat with mechanic discovery. A Retrieve and Escape epoch can be won through a relay of carriers with blockers, warders, and lures supporting. The epoch vote selects the next mode. All broadcasts route through DCRG. All responses under 150 chars, all tests passing. Commit and report.

120
projects/mmud/mmud-prompts/mmud-prompts/06-phase6.md Normal file
View file

@ -0,0 +1,120 @@
# Task: MMUD Phase 6 — The Breach
## Before Writing Any Code
Re-read these sections of `/home/zvx/projects/mmud/docs/planned.md`:
- The Breach — Mid-Epoch Event (Day 15) (all of it — 4 mini-events, endgame interaction, design rationale)
- Breach secrets (the 3 breach-type secrets)
Also re-read `config.py` for: BREACH_ROOMS_MIN/MAX, BREACH_CONNECTS_FLOORS, BREACH_SECRETS, BREACH_MINI_EVENTS, EMERGENCE_HP, INCURSION_REGEN/HOLD_HOURS.
Phase 4 already generates the Breach zone (breachgen.py) and Phase 4's daytick.py already handles the day 15 trigger and days 12-13 foreshadowing. Phase 4's breach.py has basic state management. This phase wires the full runtime logic for all 4 mini-events.
## Phase 6 Deliverables
The Breach opens on day 15 with a random mini-event. Each of the 4 types plays differently. The Breach interacts with whichever endgame mode is active.
### 1. Breach Activation (verify/extend existing)
The day 15 trigger should already be in daytick.py. Verify it:
- Day 12-13: barkeep foreshadowing broadcasts from DCRG: "The walls grow thin between the second and third depths. Something stirs."
- Day 15: Breach opens. Set `breach.active = 1`. Open the room exits connecting Breach zone to floors 2 and 3. Broadcast from DCRG: "⚡ The ground splits. A new passage has opened between Floors 2 and 3. Strange light pours from within."
- Players can now enter Breach rooms via the new exits from floors 2 and 3.
- The permanent shortcut between floors 2 and 3 persists for the rest of the epoch.
### 2. Mini-Event: The Heist (mini Retrieve & Escape)
`src/systems/breach_heist.py` (new file)
- Artifact in the deepest Breach room, guarded by the Breach mini-boss.
- Kill mini-boss → claim artifact. Carrier must bring it back to town.
- Pursuer spawns (slower, Breach-only — only operates within the 5-8 Breach rooms + the floors 2-3 connection).
- If carrier dies, artifact drops. Any player can pick up.
- Relay mechanics same as R&E but compressed — 5-8 rooms, not 4 floors.
- 3 Breach secrets scattered along the escape route. Found under pressure.
- Completion: artifact delivered to town. Breach rewards distributed. Broadcast from DCRG: "🏆 The artifact has been extracted from the Breach!"
Reuse as much R&E logic from Phase 5 as possible — shared carrier/pursuer/relay patterns.
### 3. Mini-Event: The Emergence (mini Raid Boss)
`src/systems/breach_emergence.py` (new file)
- Creature with shared HP pool (500-800 HP, `EMERGENCE_HP_MIN/MAX`) sits in central Breach room.
- Surrounding rooms spawn minions on a timer (respawn every 8 hours).
- Same chip-and-run combat as bounties/raid boss. Regen at 3%/8h.
- 3 Breach secrets are in the minion rooms — discovered while contributing to the kill.
- Completion: creature HP reaches 0. Broadcast: "🏆 The Breach creature has been destroyed!"
Reuse raid boss combat framework from Phase 5.
### 4. Mini-Event: The Incursion (mini Hold the Line)
`src/systems/breach_incursion.py` (new file)
- Breach rooms start fully hostile. Regen at 2 rooms/day (`INCURSION_REGEN_ROOMS_PER_DAY`) within just 5-8 rooms.
- Players must clear ALL Breach rooms and hold them all for 48 hours (`INCURSION_HOLD_HOURS`).
- If any room reverts during the hold timer, the clock resets.
- 3 Breach secrets behind the hardest rooms, found as part of the push.
- Track hold start time in `breach.incursion_hold_started_at`. On each regen tick, check if any Breach room reverted — if so, reset the timer.
- Completion: 48 hours with all rooms held. Broadcast: "🏆 The Breach has been secured! The incursion is contained."
Reuse HtL room clearing/regen logic from Phase 5.
### 5. Mini-Event: The Resonance (puzzle dungeon)
`src/systems/breach_resonance.py` (new file)
- No combat focus. Breach rooms contain environmental puzzles.
- 3 Breach secrets ARE the puzzle rewards. Finding all 3 unlocks a bonus cache in the deepest room.
- Puzzles are generated in Phase 4 (breachgen already places Breach secrets). This phase adds the interaction logic:
- `examine` objects in Breach rooms triggers puzzle checks
- Puzzle state tracked per-player in secret_progress
- Sequence puzzles, item-interaction puzzles, cross-room clue puzzles (use the same multi-room puzzle archetypes from the main dungeon)
- Completion: all 3 Breach secrets found by any player(s). Bonus cache unlocked. Broadcast: "🏆 The Resonance has been understood. The Breach yields its secrets."
Soloable by nature — knowledge not stats.
### 6. Breach Interaction with Endgame Modes
Regardless of which mini-event is running, the Breach benefits the active endgame mode:
- **Retrieve & Escape:** The Breach shortcut (floors 2↔3) becomes an alternate escape route. Carrier can path through it. Shorter but Breach content (mini-boss, minions, etc.) may still be there.
- **Raid Boss:** Breach completion (any mini-event) drops a buff item granting +20% damage vs the raid boss for the rest of the epoch. Add to player inventory on Breach completion.
- **Hold the Line:** Breach rooms count as bonus territory toward checkpoint progress on both floors 2 and 3. Cleared Breach rooms contribute to the cleared room count for both floor 2 and floor 3 checkpoints.
### 7. Breach Secret Integration
Verify that the 3 Breach secrets work with the existing discovery system:
- `secrets` command includes Breach secrets in the count after day 15
- Secret milestones (5/10/15/20) fire correctly with Breach secrets included
- Barkeep hints for Breach secrets only available after day 15
- Breach secrets contribute to the completionist reward (all 20 found)
## Rules
- All responses under 150 chars.
- All Breach broadcasts route through DCRG.
- Breach mini-event is always random (selected at epoch gen, never voted).
- Reuse combat/territory frameworks from Phase 5 — don't duplicate code.
- Breach content is inaccessible before day 15. Exits to Breach rooms don't exist until activation.
- Use constants from `config.py`.
- Commit after each mini-event works.
## Testing
Add to `tests/`:
- `tests/test_breach_activation.py` — day 15 trigger, foreshadowing on days 12-13, exits open, Breach accessible, inaccessible before day 15
- `tests/test_breach_heist.py` — mini-boss, artifact claim, mini-pursuer, relay, completion, secrets under pressure
- `tests/test_breach_emergence.py` — shared HP pool, minion respawn, chip-and-run, completion, secrets in minion rooms
- `tests/test_breach_incursion.py` — room clearing, regen within Breach, 48h hold timer, timer reset on revert, completion
- `tests/test_breach_resonance.py` — puzzle interaction, secret discovery, bonus cache unlock, no combat required
- `tests/test_breach_endgame.py` — R&E shortcut, raid boss damage buff, HtL bonus territory
Use in-memory SQLite for tests. Generate full epoch with DummyBackend for each test.
## Done When
The Breach opens on day 15 with one of four randomly selected mini-events. Each mini-event is playable end-to-end with its own win condition. Breach secrets integrate cleanly with the discovery system. The Breach interacts with whichever endgame mode is active. All broadcasts route through DCRG. All responses under 150 chars, all tests passing. Commit and report.
This is the final gameplay phase. After this, the full 30-day epoch loop is complete: epoch generates → players explore and progress → Breach opens day 15 → endgame mode pushes through days 20-30 → epoch vote → wipe → new epoch.

32
projects/mmud/mmud-prompts/mmud-prompts/README.md Normal file
View file

@ -0,0 +1,32 @@
# MMUD Prompt Bundle
## Status
- Phases 1-4: COMPLETE (265 tests passing)
- Phases 5-6: Prompts ready
- Design doc: NEEDS UPDATES (4 prompts below, run before Phase 5)
## Run Order
### Step 1: Design Doc Updates (run in CC in this exact order)
These update `/home/zvx/projects/mmud/docs/planned.md` in place:
1. `01-update-planned.md` — Adds The Last Ember (bar + 4 NPC bios), NPC live conversations, command discovery
2. `02-npc-nodes.md` — Replaces talk command with sim node architecture (6 nodes), three rule layers, onboarding funnel
3. `03-darkcragg.md` — Names the dungeon "The Darkcragg Depths"
4. `04-dcrg-node.md` — Adds DCRG as one-way broadcast node, separates broadcast stream from EMBR
### Step 2: Build Phases
5. `05-phase5.md` — Endgame modes: Hold the Line (regen, checkpoints, floor bosses, 12 mechanics), Raid Boss (HP scaling, 12 mechanics, 3 phases), Retrieve & Escape (Pursuer, blockers, warders, lures), epoch vote
6. `06-phase6.md` — The Breach: 4 mini-events (Heist, Emergence, Incursion, Resonance), endgame interaction, day 15 trigger. Final gameplay phase.
### If Needed
- `mmud-project.md` — Drop into `/home/zvx/projects/.ref/projects/` if not already there
## What's Complete After Phase 6
The full 30-day epoch loop: generate → explore → Breach day 15 → endgame push days 20-30 → vote → wipe → new epoch. All three endgame modes, all four Breach mini-events, 20 secrets, 40 bounties, floor bosses, raid boss, Pursuer + support roles.
## What Comes After Phase 6
- NPC live conversations (LLM runtime for talk via sim nodes) — needs implementation prompt
- Sim node deployment (meshtasticd with 6 identities) — needs infrastructure work
- Last Ember spectator web dashboard — separate project, parallel track
- Playtesting and number tuning

48
projects/mmud/mmud-prompts/mmud-prompts/mmud-project.md Normal file
View file

@ -0,0 +1,48 @@
# MMUD — Mesh Multi-User Dungeon
Text-based multiplayer dungeon crawler for Meshtastic LoRa mesh networks. BBS door games (LORD, TradeWars) adapted for 150-char mesh radio constraints, async play, 30-day wipe cycles.
## Status
**Phase:** Pre-development — design complete, repo scaffolded, implementation not started.
## Repo
`/home/zvx/projects/mmud`
The repo contains a `CLAUDE.md` with full architecture, directory structure, development phases, and implementation guidance. **Read it first before any implementation work.**
## Key Files
- `CLAUDE.md` — Architecture, patterns, dev phases, gotchas
- `docs/planned.md` — Complete game design document (~950 lines). Source of truth for all mechanics. If code contradicts this, code is wrong.
- `config.py` — All game constants with rationale
- `src/db/schema.sql` — Full database schema
## Design Constraints
- 150 characters per Meshtastic LoRa message (hard ceiling)
- Zero runtime LLM calls — all text batch-generated at epoch start
- Async-first — all multiplayer through shared DB state
- 12 dungeon actions/day, 30-day epochs
- Python 3.11+, SQLite, Meshtastic Python API
## Development Phases
1. **Core Loop** — Meshtastic message handling, command parser, player creation, room navigation, basic combat, death, action budget
2. **Economy & Progression** — XP, leveling, gold, shops, gear (weapon/armor/trinket), bank, healer
3. **Social Systems** — Broadcasts (tier 1/2/targeted), barkeep (recap, tokens, hints), bounty board, player messages, mail
4. **Epoch Generation** — World gen, LLM narrative pipeline (batch + validation), secret placement, bounty pool generation
5. **Endgame Modes** — Hold the Line (regen, checkpoints, floor bosses), Raid Boss (HP scaling, mechanic tables, phases), Retrieve & Escape (Pursuer, blockers, warders, lures), epoch vote
6. **The Breach** — Breach zone gen, 4 mini-event types (Heist, Emergence, Incursion, Resonance), day 15 trigger
## No Runbooks Needed
This is a pure software project — no LXC provisioning, no Caddy config, no Authentik integration. Runs as a Python daemon connected to a Meshtastic device via USB/serial or TCP. No infrastructure runbooks apply.
## Notes
- All regen/HP/damage numbers in the design doc are targets, not validated — will need playtesting
- The game runs on a Meshtastic mesh network, not a web server
- SQLite single file DB, no ORM, raw parameterized SQL
- Every outbound message must fit 150 chars — the formatter is the final gate

132
projects/openwebui-theme-deploy.md Normal file
View file

@ -0,0 +1,132 @@
# RUNBOOK — Deploy Echo6 Theme to Open WebUI
> **STATUS: COMPLETED** — 2026-02-17
> Theme deployed to https://ai.echo6.co. Open WebUI runs in Docker (not native as originally stated).
> Theme files bind-mounted from `/home/zvx/echo6-theme/` into container via docker-compose.yml.
> Compose path: `/opt/open-webui/docker-compose.yml` on cortex.
## OBJECTIVE
Apply the togglable Echo6 custom CSS theme to the Open WebUI instance running at ai.echo6.co. The instance runs in Docker on the `cortex` VM. The theme is toggled on/off via a small "E6" button in the bottom-right corner, with the preference persisted in localStorage per browser.
## THEME FILES
Located in this project's reference assets:
```
.ref/assets/echo6-openwebui-theme.css # Theme styles (activates via .echo6 class on <html>)
.ref/assets/echo6-theme-toggle.js # Toggle button + localStorage persistence
```
## SAFETY FIRST — BACKUP BEFORE ANYTHING
Before making ANY changes:
1. **Find the Open WebUI static build directory.** Likely locations:
- Check: `pip show open-webui 2>/dev/null` for the install path
- Search: `find / -name "index.html" -path "*/open*webui*" 2>/dev/null`
- Search: `find / -name "app.html" -path "*/open*webui*" 2>/dev/null`
- Common pip paths: `/usr/lib/python3/dist-packages/open_webui/static/`
2. Once found, identify the **root HTML file** (likely `index.html` or `app.html`).
3. **Create a timestamped backup:**
```bash
OWUI_DIR="/path/to/open-webui/build" # ← set this once found
BACKUP_DIR="/home/matt/backups/openwebui-theme-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -a "$OWUI_DIR" "$BACKUP_DIR/"
echo "Backed up to: $BACKUP_DIR"
```
4. **Create a revert script** at `/home/matt/revert-openwebui-theme.sh`:
```bash
#!/bin/bash
# Revert Echo6 theme — restores original Open WebUI files
BACKUP_DIR="<populated during deploy>"
OWUI_DIR="<populated during deploy>"
echo "[REVERT] Restoring Open WebUI from $BACKUP_DIR"
cp -a "$BACKUP_DIR"/* "$OWUI_DIR/"
echo "[REVERT] Done. Restart Open WebUI service if needed."
```
Make it executable: `chmod +x /home/matt/revert-openwebui-theme.sh`
## DEPLOYMENT STEPS
### Step 1: Copy theme files to Open WebUI static directory
```bash
cp .ref/assets/echo6-openwebui-theme.css "$OWUI_DIR/static/"
cp .ref/assets/echo6-theme-toggle.js "$OWUI_DIR/static/"
```
Also keep persistent copies that survive upgrades:
```bash
mkdir -p /home/matt/echo6-theme
cp .ref/assets/echo6-openwebui-theme.css /home/matt/echo6-theme/
cp .ref/assets/echo6-theme-toggle.js /home/matt/echo6-theme/
```
### Step 2: Inject into the root HTML
Find the root HTML file and add BOTH a `<link>` and a `<script>` tag BEFORE `</head>`:
```bash
sed -i 's|</head>|<link rel="stylesheet" href="/static/echo6-openwebui-theme.css">\n<script src="/static/echo6-theme-toggle.js" defer></script>\n</head>|' "$OWUI_DIR/index.html"
```
**IMPORTANT:** Check existing `<link>` and `<script>` tags in the HTML first to confirm the `/static/` prefix matches how Open WebUI serves assets. Adjust the path if it uses a different pattern (e.g. `/_app/`, `/build/`, etc.).
### Step 3: Verify
```bash
grep "echo6" "$OWUI_DIR/index.html"
ls -la "$OWUI_DIR/static/echo6-openwebui-theme.css"
ls -la "$OWUI_DIR/static/echo6-theme-toggle.js"
```
Restart the service if needed:
```bash
sudo systemctl restart open-webui # or whatever the service name is
```
### Step 4: Test
- `curl -s https://ai.echo6.co | grep echo6` should show both the CSS and JS references
- Load ai.echo6.co in browser — should see a small "E6" button in the bottom-right corner
- Click it: theme activates (dark bg, cyan accents, JetBrains Mono)
- Click again: reverts to stock Open WebUI appearance
- Refresh page: preference should persist
## HOW THE TOGGLE WORKS
1. The JS injects a fixed-position "E6" button at bottom-right
2. Clicking it toggles the `echo6` class on `<html>`
3. ALL CSS selectors in the theme are `.echo6 <target>` — they ONLY fire when that class is present
4. The toggle state is saved to `localStorage` under the key `echo6-theme-active`
5. On page load, the JS reads localStorage and restores the previous state before first paint
6. Open WebUI's built-in theme picker (Light/Dark/OLED) still works independently
## CONSTRAINTS / DO NOT
- Do NOT modify any Open WebUI Python source code
- Do NOT modify any existing JavaScript files
- Do NOT install additional packages
- Do NOT change Open WebUI configuration/database
- ONLY touch: one HTML file (add one `<link>` + one `<script>` tag) and add two static files
- If anything looks wrong or the build structure is unexpected, STOP and report back
## DEBUGGING
If the theme doesn't apply:
1. Browser dev tools → Network tab — are both files loading? (200 vs 404)
2. Browser dev tools → Elements → check `<html>` — does it have class `echo6` after clicking toggle?
3. Browser dev tools → Console — any JS errors?
4. If selectors don't match the actual DOM, inspect elements and adjust selectors in the CSS
If the toggle button doesn't appear:
1. Check Console for JS errors
2. Verify the `<script>` tag is present in the HTML source
3. Check if the JS file path is correct (404 in Network tab)
## AFTER UPGRADE PROCEDURE
After any Open WebUI upgrade, the build files get overwritten. To re-apply:
```bash
cp /home/matt/echo6-theme/echo6-openwebui-theme.css "$OWUI_DIR/static/"
cp /home/matt/echo6-theme/echo6-theme-toggle.js "$OWUI_DIR/static/"
# Re-inject the tags into the HTML:
sed -i 's|</head>|<link rel="stylesheet" href="/static/echo6-openwebui-theme.css">\n<script src="/static/echo6-theme-toggle.js" defer></script>\n</head>|' "$OWUI_DIR/index.html"
```
Or just re-run this runbook from the top.

797
projects/peertube-phase2-project.md Normal file
View file

@ -0,0 +1,797 @@
# Project: PeerTube Phase 2 — Import Pipeline Build
**Goal:** Build a complete YouTube download → local import → GPU transcode pipeline for 99 channels (~70K+ videos, ~15.3TB) on a fresh PeerTube v8 instance. Clean slate — no legacy code, no old pipeline files. Build it right from scratch.
**CC Host:** cortex (SSH to all nodes via aliases in ~/.ssh/config; Proxmox nodes use sshpass auth)
---
## SSH Prerequisites — RUN FIRST
**Every CC session must verify SSH connectivity before executing any remote commands. Never assume SSH works.**
### Verify cortex → CT 110 (PeerTube)
```bash
# CT 110 uses sshpass auth (same as all LXCs). Check ~/.ssh/config for alias.
# Try alias first, fall back to IP:
ssh -o ConnectTimeout=5 peertube 'hostname' 2>/dev/null \
|| sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 zvx@192.168.1.170 'hostname'
```
### Verify cortex → media node (Proxmox host, for pct commands if needed)
```bash
sshpass -p '7redditGold' ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 root@192.168.1.243 'hostname'
```
### Gate
Both must return hostnames. **Stop and fix SSH before proceeding with ANY step.**
If aliases don't exist in `~/.ssh/config`, add them:
```bash
grep -q "Host peertube$" ~/.ssh/config 2>/dev/null || cat >> ~/.ssh/config << 'EOF'
Host peertube
HostName 192.168.1.170
User zvx
EOF
```
Note: Most pipeline work runs as the `peertube` user inside CT 110. SSH in as zvx, then `sudo -u peertube` or `sudo su - peertube` as needed.
---
## Runbook References
These runbooks live in `~/runbooks/` on cortex. Call them by name when their scope applies:
| Runbook | When to Use in Phase 2 |
|---------|----------------------|
| **`nordvpn-lxc.md`** | **Step 3 — RUN THIS RUNBOOK.** VPN setup on CT 110 with TUN device, NordVPN/WireGuard, split tunneling, rotation script |
| **`peertube-remote-runner.md`** | **ACTIVE — used for video-transcription (Whisper captioning).** Runner on cortex handles auto-captioning with smart GPU/CPU routing. Not used for H.265 video transcoding (pipeline handles that). See runbook for Whisper setup details. |
| `ct-runbook.md` | If CT 110 needs additional packages or baseline changes (provisioned in Phase 1 — reference only) |
| `expose-service-home.md` | stream.echo6.co is already exposed (Phase 1). Reference only if Caddy/DNS/cert issues arise |
| `authentik-oidc-application.md` | PeerTube OIDC already configured (Phase 1). Reference only if SSO breaks |
| `pi-nas-omv-runbook.md` | If NFS storage issues arise (mount problems, permissions, OMV config) |
| `proxmox-onboard-node.md` | SSH access patterns — the Phase 1 prereq pattern above follows this runbook's conventions |
| `proxmox-create-ubuntu-vm.md` | If cortex needs modifications (GPU passthrough, NVIDIA drivers, Docker). Reference only |
**Not applicable to Phase 2:** idahomesh-*, meshmonitor-*, meshtasticd-* runbooks.
---
## Infrastructure (Read-Only Context — Do Not Modify)
### PeerTube Instance
- **CT 110** on **media** node (Proxmox)
- Local IP: 192.168.1.170
- Tailscale IP: 100.64.0.23
- OS: Debian 12, privileged LXC
- PeerTube v8 — **native install** (NOT Docker). No `docker exec` for anything.
- Runs as user: `peertube`
- PostgreSQL: local, accessible via `sudo -u postgres psql peertube_prod` or `sudo -u peertube psql peertube_prod`
- Redis: local
- Nginx: local (port 80), proxied through Caddy on utility node
- Domain: stream.echo6.co
- NFS storage: 18TB from pi-nas (192.168.1.245) mounted at `/var/www/peertube/storage/`
- NFS export path: `/srv/dev-disk-by-uuid-822575b9-1549-4aab-823e-8160d2aa7c68/peertube/`
- PeerTube config: `/var/www/peertube/config/local-production.json` (v8 uses JSON, not YAML)
- PeerTube base dir: `/var/www/peertube/`
- Built-in channel sync: DISABLED (bulk pipeline handles imports)
- Signup: disabled (Authentik SSO only)
### GPU Pre-Transcoding (H.265 via NVENC)
- **cortex** — VM on TOC node, RTX A4000 GPU passthrough
- cortex is also the CC host and runs Ollama/Aurora
- NVENC is separate silicon from CUDA — transcoding won't conflict with LLM inference
- **PeerTube's built-in transcoding is DISABLED** — remote runners ignore transcoding plugins, so there's no way to get H.265 through the runner pipeline
- Instead: a `transcoder.py` service on cortex pulls downloaded videos from CT 110, re-encodes to H.265 with `hevc_nvenc`, pushes back. The importer then uploads already-transcoded files to PeerTube with `waitTranscoding=false`
- Target: H.265, 1080p only, single file per video (no HLS adaptive — LAN/Tailscale viewers don't need it)
- ffmpeg command: `ffmpeg -i input.mp4 -c:v hevc_nvenc -preset medium -cq 28 -c:a aac -b:a 128k output.mp4`
- File transfer: cortex pulls from CT 110 via rsync/SSH, transcodes locally to avoid NFS latency on GPU work, pushes result back
### Runner Service (ACTIVE — video-transcription/captioning)
Runner on cortex handles Whisper auto-captioning. Also registered for VOD transcoding jobs but H.265 video transcoding goes through the pipeline transcoder instead.
```ini
[Unit]
Description=PeerTube Remote Runner (NVENC)
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
Requires=nvidia-persistenced.service
[Service]
Type=simple
User=zvx
Group=zvx
Environment=NODE_ENV=production
Environment=PATH=/opt/peertube-runner/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/usr/bin/peertube-runner server --enable-job vod-hls-transcoding --enable-job vod-audio-merge-transcoding --enable-job live-rtmp-hls-transcoding --enable-job video-studio-transcoding --enable-job video-transcription
WorkingDirectory=/home/zvx
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube-runner
MemoryMax=20G
[Install]
WantedBy=multi-user.target
```
**Whisper config:** Smart wrapper at `/usr/local/bin/whisper-smart` routes <1hr to GPU (CUDA float16), >=1hr to CPU (int8). CPU jobs serialized via flock. Runner concurrency=2 (1 GPU + 1 CPU in parallel). Model: medium. See `peertube-remote-runner.md` for full details.
### Recovered Runner Health Script
```bash
#!/bin/bash
LOG_TAG="peertube-runner-health"
if ! systemctl is-active --quiet peertube-runner; then
logger -t $LOG_TAG "Runner not active, restarting..."
systemctl restart peertube-runner
sleep 10
fi
if ! pgrep -f "peertube-runner server" > /dev/null; then
logger -t $LOG_TAG "Runner process not found, restarting service..."
systemctl restart peertube-runner
fi
if ! nvidia-smi > /dev/null 2>&1; then
logger -t $LOG_TAG "GPU not accessible, restarting nvidia-persistenced and runner..."
systemctl restart nvidia-persistenced
sleep 5
systemctl restart peertube-runner
fi
```
### SSH / Access
- cortex → CT 110: `ssh peertube` or `ssh root@192.168.1.170` (check ~/.ssh/config)
- cortex → Proxmox nodes: uses sshpass (aliases in ~/.ssh/config)
- CT 110 user for pipeline: `peertube` (same user that runs the PeerTube process)
### VPN
- NordVPN account exists, needs fresh setup on CT 110
- LXC may not support NordVPN CLI (systemd issues) — WireGuard configs as fallback
- Rotation countries: US, CA, UK, DE, NL, SE
- Split tunnel / killswitch off so PeerTube stays accessible locally
---
## Channel Map — The 99 Channels
### Recovered Schema (from old WATCHTOWER add_channel.py)
```json
{
"category": "Tactical/SUT",
"channel_name": "(YT)Garand Thumb",
"actor_name": "garand-thumb",
"youtube_url": "https://www.youtube.com/@GarandThumb",
"youtube_channel_id": null,
"peertube_channel_id": null,
"video_count": 0,
"priority": "H",
"est_videos": 500,
"est_gb": 98
}
```
### Recovered Slug Function
```python
import re
def slugify_channel(name):
"""Convert channel name to PeerTube-safe actor_name."""
name = re.sub(r'^\(YT\)\s*', '', name)
slug = re.sub(r'[^a-z0-9]+', '-', name.lower()).strip('-')
return slug[:50] or 'channel'
```
### Known YouTube URLs (from old PeerTube sync records — 24 channels)
These 24 channels had active sync records with confirmed YouTube URLs:
```
Essential Craftsman → @essentialcraftsman
CommsPrepper → @CommsPrepper
Steven Lavimoniere → @StevenLavimoniere
Andreas Spiess → @AndreasSpiess
Mustie1 → @mustie1
Donyboy73 → @Donyboy73
Turn a Wood Bowl → @TurnaWoodBowl
RoseRed Homestead → @RoseRedHomestead
Homesteading Family → @HomesteadingFamily
My Self Reliance → @MySelfReliance
RegisteredNurseRN → @RegisteredNurseRN
Skinny Medic → @SkinnyMedic
Marine X → @MarineX
Plumberparts → @plumberparts
MedCram → @Medcram
City Prepping → @CityPrepping
Paul Kirtley → @PaulKirtley
Armando Hasudungan → playlist?list=UUesNt4_Z-Pm41RzpAClfVcg
Self Sufficient Me → @Selfsufficientme
Taryl Fixes All → @TarylFixesAll
Engineer775 → @engineer775
WeberAuto → @WeberAuto
Sun Knudsen → @sunknudsen
Master Your Medics → @MasterYourMedics
MCQBushcraft → @MCQBushcraft
ChrisFix → @ChrisFix
```
### The 99 Channels (Finalized Feb 2026)
#### OPSEC / Privacy (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Michael Bazzell / IntelTechniques | H | OSINT + digital privacy, ex-FBI |
| The Hated One | H | Privacy advocacy, surveillance deep-dives |
| Mental Outlaw | H | Linux + privacy + infosec news |
| Naomi Brockwell TV | M | Privacy-focused tech |
| Techlore | M | Privacy tools and comparisons |
| Sun Knudsen | M | Step-by-step privacy hardening |
#### Physical Security (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Deviant Ollam | H | Physical penetration testing, lock bypass |
| BosnianBill | M | Lock picking, physical security analysis |
#### Intelligence / OSINT (4)
| Channel | Priority | Notes |
|---------|----------|-------|
| OSINT Dojo | H | OSINT methodology training |
| Benjamin Strick | H | Professional OSINT investigations |
| OSINT Curious | M | OSINT tools and techniques |
| S2 Underground | H | Threat intel, analysis tradecraft |
#### Cybersecurity (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| John Hammond | H | CTF walkthroughs, malware analysis |
| IppSec | H | HackTheBox walkthroughs |
| LiveOverflow | H | Binary exploitation, web security |
| Professor Messer | M | CompTIA certification training |
| The Cyber Mentor | M | Ethical hacking courses |
| Hak5 | M | Hacking tools and techniques |
| David Bombal | M | Networking + cybersecurity |
#### Tactical / SUT (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Garand Thumb | H | Tactics, gear testing, NV |
| Dirty Civilian | H | SUT for civilians |
| One Shepherd | H | Former SOF, tactical training |
| Brent0331 | H | USMC veteran, tactical analysis |
| Brass Facts | M | Firearms philosophy, gear testing |
| Sage Dynamics | H | Research-based torture tests |
#### Firearms (8)
| Channel | Priority | Notes |
|---------|----------|-------|
| Forgotten Weapons | H | Historical + technical firearms (largest channel, ~3K videos) |
| Paul Harrell | H | Terminal ballistics, practical shooting |
| 9-Hole Reviews | M | Precision rifle, historical accuracy |
| Lucky Gunner | M | Ammo testing, concealed carry |
| C&Rsenal | M | WWI/WWII firearms deep-dives |
| Jerry Miculek | M | Speed shooting, competition |
| InRangeTV | M | Firearms + mud tests |
| Hickok45 | M | Reviews + shooting demonstrations |
#### Comms / Signals (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| OH8STN | H | Off-grid digital comms, Winlink |
| Andreas Spiess | H | Electronics + LoRa + radio |
| Ham Radio Crash Course | H | Amateur radio training |
| Tech Minds | M | SDR, radio tech |
| The Comms Channel | M | Comms gear and planning |
| KM4ACK | H | Build-a-Pi, ham radio software |
| Signals Everywhere | M | SDR + spectrum analysis |
#### Medical (5)
| Channel | Priority | Notes |
|---------|----------|-------|
| PrepMedic | H | Flight paramedic, trauma care |
| Skinny Medic | H | IFAK, trauma kits |
| MedWild | H | Wilderness medicine |
| Crisis Medicine | H | Former 18D SF Medic, TCCC |
| Ninja Nerd | H | Comprehensive physiology/pathology |
#### Linux / Infrastructure (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Lawrence Systems | H | Enterprise networking + Linux |
| Learn Linux TV | H | Linux tutorials and homelab |
| Jeff Geerling | H | Raspberry Pi, Ansible, self-hosting |
| Techno Tim | M | Homelab, Docker, Kubernetes |
| Level1Techs | M | Hardware + Linux deep-dives |
| Wolfgang's Channel | M | Self-hosting, privacy infra |
#### Hardware / Electronics (4)
| Channel | Priority | Notes |
|---------|----------|-------|
| Ben Eater | H | Computer architecture from scratch |
| EEVblog | H | Electronics engineering |
| GreatScott! | M | Electronics projects |
| Big Clive | M | Electronics teardowns |
#### Auto / Mechanical (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| ChrisFix | H | DIY auto repair fundamentals |
| Mustie1 | H | Dead machinery resurrection |
| South Main Auto | H | Diagnostic logic |
| 1A Auto | H | Make/model/year repair encyclopedia (~4,500 videos) |
| Pine Hollow Auto Diagnostics | M | Advanced diagnostics |
| ScannerDanner | M | Master electrical diagnostics |
| Diesel Creek | M | Heavy equipment repair |
#### Construction / Trades (7)
| Channel | Priority | Notes |
|---------|----------|-------|
| Essential Craftsman | H | Construction + life skills |
| Matt Risinger | H | Building science |
| Mike Haduck Masonry | M | Foundations, concrete, stone |
| Awesome Framers | M | Structural framing |
| This Old House | M | Home renovation |
| Electrician U | M | Electrical trade training |
| Got2Learn | M | Plumbing/electrical tutorials |
#### Welding / Fabrication (3)
| Channel | Priority | Notes |
|---------|----------|-------|
| Welding Tips and Tricks | H | Welding instruction |
| ChuckE2009 | M | Welding + fabrication |
| Paul Sellers | H | Hand tool woodworking master |
#### Sustainment / Fieldcraft (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Corporals Corner | H | Field skills, shelter, fire |
| Gray Bearded Green Beret | H | SF wilderness medicine + fieldcraft |
#### Homesteading / Production (8)
| Channel | Priority | Notes |
|---------|----------|-------|
| City Prepping | H | Urban/suburban preparedness |
| My Self Reliance | H | Off-grid building |
| Engineer775 | H | Off-grid power systems |
| Project Farm | H | Tool and product testing |
| Will Prowse / DIY Solar Power | H | Solar power systems |
| Townsends | M | 18th century skills + cooking |
| RoseRed Homestead | M | Homesteading skills |
| The Urban Prepper | M | Urban preparedness, modular bags |
#### Preparedness (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| The Provident Prepper | M | Preparedness planning methodology |
#### Energy / Alt-Fuel (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Adeptus Beta | M | Wood gasification (~7GB, tiny) |
#### Education / STEM (6)
| Channel | Priority | Notes |
|---------|----------|-------|
| Practical Engineering | H | Civil engineering with demos |
| Real Engineering | M | Aerospace, energy, transport |
| The Efficient Engineer | M | Core engineering fundamentals |
| NurdRage | M | Chemistry experiments |
| NileRed | M | Chemistry deep-dives |
| Veritasium | M | Science + engineering |
#### Education / Math (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Professor Leonard | H | Full calculus + stats lectures |
| Organic Chemistry Tutor | M | Math + science tutorials |
#### Education / CS (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Computerphile | H | Crypto, networking theory, security concepts |
| MIT Missing Semester | M | Shell, git, dev tools (tiny, ~50 videos) |
#### Small Engine (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Donyboy73 | M | Small engine repair |
#### Woodworking (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Steve Ramsey | M | Beginner woodworking |
#### Home Repair (2)
| Channel | Priority | Notes |
|---------|----------|-------|
| Home RenoVision DIY | M | Home repair tutorials |
| Roger Wakefield | M | Plumbing |
#### Bushcraft (1)
| Channel | Priority | Notes |
|---------|----------|-------|
| Joe Robinet | M | Bushcraft and camping |
**Total: 99 channels across 20 categories**
---
## Execution Steps
### Step 1: Channel Map Generation
**Where:** CT 110
**What:** Build `/opt/bulk-import/config/channel-map.json`
**SSH Gate:** `ssh peertube 'hostname'` must succeed before proceeding.
1. Create directory structure:
```bash
# Scripts and config on local disk
mkdir -p /opt/bulk-import/{config,logs}
chown -R peertube:peertube /opt/bulk-import
# Video data on NFS (18TB pi-nas mount)
mkdir -p /var/www/peertube/storage/pipeline/{staging,completed,transcoded,failed}
chown -R peertube:peertube /var/www/peertube/storage/pipeline
# Symlink data dirs so scripts use /opt/bulk-import/ paths
ln -sfn /var/www/peertube/storage/pipeline/staging /opt/bulk-import/staging
ln -sfn /var/www/peertube/storage/pipeline/completed /opt/bulk-import/completed
ln -sfn /var/www/peertube/storage/pipeline/transcoded /opt/bulk-import/transcoded
ln -sfn /var/www/peertube/storage/pipeline/failed /opt/bulk-import/failed
```
2. For each of the 99 channels:
- Look up the actual YouTube channel URL (use `yt-dlp --print channel_url --playlist-items 1 --skip-download "https://www.youtube.com/@ChannelHandle"` for any that need verification)
- Generate `actor_name` via slugify
- Write to channel-map.json
3. Use the 24 known URLs from old sync records as a head start. The remaining 75 need URL resolution.
**⚠️ This step requires yt-dlp installed and working on CT 110. If yt-dlp isn't installed yet, install it first:**
```bash
curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
chmod +x /usr/local/bin/yt-dlp
```
**⚠️ YouTube may rate-limit channel lookups. Space requests 2-3 seconds apart. If rate-limited, use cookies or VPN.**
### Step 2: PeerTube Channel Creation
**Where:** CT 110
**What:** Batch-create all 99 channels via PeerTube API
**SSH Gate:** `ssh peertube 'curl -s http://localhost:9000/api/v1/config | head -c 50'` — must return JSON. Confirms both SSH and PeerTube are up.
1. Get OAuth token from PeerTube API (local, port 9000):
```bash
# Get client credentials
curl -s http://localhost:9000/api/v1/oauth-clients/local -H "Host: stream.echo6.co"
# Get user token
curl -s http://localhost:9000/api/v1/users/token \
-H "Host: stream.echo6.co" \
--data "client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>&grant_type=password&username=root&password=<PASSWORD>"
```
2. For each channel in channel-map.json:
```bash
curl -s -X POST http://localhost:9000/api/v1/video-channels \
-H "Host: stream.echo6.co" \
-H "Authorization: Bearer <TOKEN>" \
-H "Content-Type: application/json" \
-d '{"name": "<actor_name>", "displayName": "(YT)<channel_name>", "description": "Imported from YouTube: <youtube_url>"}'
```
3. Capture the returned channel ID and update `peertube_channel_id` in channel-map.json
4. Verify: `curl -s http://localhost:9000/api/v1/video-channels -H "Host: stream.echo6.co" | python3 -m json.tool | grep -c '"name"'` should return 99 (plus the default channel)
### Step 3: NordVPN Setup
**Where:** CT 110
**What:** Install VPN for IP rotation during YouTube downloads
**SSH Gate:** `ssh peertube 'hostname'` must succeed.
**➡️ RUN RUNBOOK: `~/runbooks/nordvpn-lxc.md`**
Use these inputs:
```
CTID=110
CT_HOST=peertube
PVE_HOST=media # or root@192.168.1.243
NORDVPN_TOKEN= # ⚠️ Get from Matt
VPN_COUNTRIES="United_States,Canada,United_Kingdom,Germany,Netherlands,Sweden"
VPN_CONFIG_DIR=/opt/bulk-import/config/vpn
```
**Additional context for this deployment:**
- CT 110 runs PeerTube on port 9000 — split tunneling is MANDATORY so PeerTube stays reachable on 192.168.1.170 and 100.64.0.23 while VPN is active
- The rotation script at `/opt/bulk-import/config/vpn/vpn-rotate.sh` will be called by `downloader.py` (Step 5) on rate-limit detection
- After runbook completes, verify PeerTube still accessible: `curl -s http://192.168.1.170:9000/api/v1/config | head -c 50` (from another machine, while VPN is up on CT 110)
**⚠️ NordVPN token required from Matt. Cannot proceed without it.**
### Step 4: YouTube Cookies
**Where:** CT 110
**What:** Export browser cookies for yt-dlp bot detection bypass
1. Matt exports cookies from browser (Netscape format) using "Get cookies.txt LOCALLY" extension
2. SCP to CT 110: `scp cookies.txt root@192.168.1.170:/opt/bulk-import/config/cookies.txt`
3. Fix perms: `chown peertube:peertube /opt/bulk-import/config/cookies.txt && chmod 600 /opt/bulk-import/config/cookies.txt`
4. Test: `sudo -u peertube yt-dlp --cookies /opt/bulk-import/config/cookies.txt --simulate "https://www.youtube.com/watch?v=dQw4w9WgXcQ"`
**⚠️ Cookies expire every 2-4 weeks. Needs manual refresh.**
### Step 5: Build downloader.py
**Where:** CT 110 at `/opt/bulk-import/downloader.py`
**What:** Round-robin YouTube channel downloader with VPN rotation
**Deploy:** Write file locally on cortex, then `scp` to CT 110. Or write directly via `ssh peertube 'cat > /opt/bulk-import/downloader.py << "PYEOF" ... PYEOF'`
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/config/channel-map.json'` — channel map must exist (Step 1 complete).
Requirements:
- Round-robin across all 99 channels (don't hammer one channel)
- yt-dlp with: `--cookies`, `--download-archive downloaded.txt` (dedup), `--write-info-json`, `--write-thumbnail`, `--format "bestvideo[height<=1080]+bestaudio/best[height<=1080]"`, `--merge-output-format mp4`
- Downloads land in `/opt/bulk-import/staging/<actor_name>/<video_id>/` with .mp4 + .info.json + .jpg
- On successful download, move to `/opt/bulk-import/completed/<actor_name>/<video_id>/`
- **Note:** transcoder.py (Step 6) picks up from completed/ — downloader does NOT feed importer directly
- VPN rotation: detect rate-limit (HTTP 429, sign-in required, bot detection), disconnect current VPN, connect to next country in rotation list, retry
- State file: `/opt/bulk-import/config/downloader-state.json` — tracks current channel index, current VPN country, last activity timestamp
- Logging to `/opt/bulk-import/logs/downloader.log` — include `=== Channel: <name> ===` markers (WATCHTOWER parses these)
- Target throughput: ~30 videos/hr
- Graceful shutdown on SIGTERM/SIGINT
### Step 6: Build transcoder.py
**Where:** cortex (local — this IS the CC host) at `/opt/bulk-import/transcoder.py`
**What:** Pulls H.264 videos from CT 110, re-encodes to H.265 via NVENC, pushes back
**Connectivity Gate:**
```bash
nvidia-smi > /dev/null 2>&1 && echo "GPU OK" || echo "GPU MISSING"
ffmpeg -encoders 2>/dev/null | grep -q hevc_nvenc && echo "HEVC NVENC OK" || echo "HEVC NVENC MISSING"
ssh peertube 'ls /opt/bulk-import/completed/' > /dev/null 2>&1 && echo "SSH OK" || echo "SSH FAIL"
```
Requirements:
- Watch CT 110's `/opt/bulk-import/completed/` for new video directories (via SSH/rsync polling, not inotify — it's remote)
- For each video dir found:
1. `rsync` the dir from CT 110 to cortex local temp: `/opt/bulk-import/transcode-work/<actor_name>/<video_id>/`
2. Run ffmpeg: `ffmpeg -hwaccel cuda -i input.mp4 -c:v hevc_nvenc -preset medium -cq 28 -tag:v hvc1 -c:a aac -b:a 128k output.mp4`
- `-cq 28` = constant quality mode (NVENC equivalent of CRF)
- `-tag:v hvc1` = Apple/browser compatible HEVC tag
- `-preset medium` = balance speed/quality (can tune later)
- Preserve .info.json and .jpg (just copy, don't re-encode)
3. `rsync` the transcoded dir back to CT 110: `/opt/bulk-import/transcoded/<actor_name>/<video_id>/`
4. Remove the source from CT 110's `completed/` dir (it's been transcoded)
5. Clean up local temp
- Skip videos that already exist in `transcoded/`
- Logging to `/opt/bulk-import/logs/transcoder.log` on cortex (and/or stream to CT 110)
- State file: `/opt/bulk-import/config/transcoder-state.json` on cortex
- Graceful shutdown on SIGTERM/SIGINT — finish current transcode, don't start new ones
- Target throughput: depends on video length, but NVENC should handle ~2-5 videos/hr for typical 10-20min content at 1080p
- One video at a time (NVENC session limit on A4000)
**Directory structure on cortex:**
```
/opt/bulk-import/ ← transcoder home on cortex
├── transcoder.py
├── config/
│ └── transcoder-state.json
├── logs/
│ └── transcoder.log
└── transcode-work/ ← temp working dir, cleaned after each video
```
**ffmpeg must be installed on cortex with NVENC support:**
```bash
sudo apt install -y ffmpeg
ffmpeg -encoders 2>/dev/null | grep hevc_nvenc # must show hevc_nvenc
# If missing: sudo apt install -y libnvidia-encode-550 (match driver version)
```
### Step 7: Build importer.py
**Where:** CT 110 at `/opt/bulk-import/importer.py`
**What:** Watches transcoded/ dir, uploads to PeerTube via API
**Deploy:** Same as Step 5 — write locally, scp to CT 110.
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/config/channel-map.json && curl -s http://localhost:9000/api/v1/config | head -c 50'` — channel map AND PeerTube API must be reachable.
Requirements:
- Watch `/opt/bulk-import/transcoded/` for new video directories (NOT completed/ — transcoder feeds this)
- For each video dir: read .info.json, extract title, description, upload_date (→ originallyPublishedAt), tags, thumbnail
- Map `<actor_name>` from dir path → `peertube_channel_id` from channel-map.json
- Upload via PeerTube API: `POST /api/v1/videos/upload` with multipart form data
- Set: name, description, channelId, originallyPublishedAt, tags (first 5), thumbnailfile, privacy (1=public), **waitTranscoding=false** (video is already H.265, no PeerTube transcoding needed)
- On success: **DELETE the video dir from `transcoded/`** — PeerTube's storage is the authoritative copy. No `imported/` directory.
- On failure: move to `/opt/bulk-import/failed/` with error log
- Rate: process one video at a time, ~50/hr max (don't overwhelm PeerTube)
- Dedup: check if video title + channel already exists before uploading
- Logging to `/opt/bulk-import/logs/importer.log`
- OAuth token management: cache token, refresh on 401
### Step 8: Systemd Services
**Where:** CT 110 (downloader + importer) AND cortex (transcoder)
**What:** Service files for all three pipeline components
**SSH Gate:** `ssh peertube 'ls /opt/bulk-import/downloader.py /opt/bulk-import/importer.py'` — both CT 110 scripts must exist (Steps 5 and 7 complete). `/opt/bulk-import/transcoder.py` must exist on cortex (Step 6 complete).
**On CT 110:**
```bash
# /etc/systemd/system/pt-downloader.service
[Unit]
Description=PeerTube Bulk Downloader
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=peertube
Group=peertube
ExecStart=/usr/bin/python3 /opt/bulk-import/downloader.py
WorkingDirectory=/opt/bulk-import
Restart=always
RestartSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pt-downloader
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/pt-importer.service — same pattern, ExecStart points to importer.py
```
**On cortex:**
```bash
# /etc/systemd/system/pt-transcoder.service
[Unit]
Description=PeerTube H.265 NVENC Transcoder
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
[Service]
Type=simple
User=zvx
Group=zvx
ExecStart=/usr/bin/python3 /opt/bulk-import/transcoder.py
WorkingDirectory=/opt/bulk-import
Restart=always
RestartSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pt-transcoder
MemoryMax=12G
[Install]
WantedBy=multi-user.target
```
Enable but **do not start** until testing is complete.
### Step 9: PeerTube Transcoding Config — DISABLED
**Where:** CT 110
**What:** Disable PeerTube's built-in transcoding — videos arrive pre-transcoded as H.265
**SSH Gate:** `ssh peertube 'hostname'` must succeed.
Edit `/var/www/peertube/config/local-production.json`:
```json
{
"transcoding": {
"enabled": false
},
"import": {
"videos": {
"concurrency": 4,
"http": { "enabled": true },
"torrent": { "enabled": false }
}
},
"video_channel_synchronization": {
"enabled": false
}
}
```
Restart PeerTube after config changes: `sudo systemctl restart peertube`
**Why disabled:** Videos are pre-transcoded to H.265 by cortex (Step 6) before import. The importer uploads with `waitTranscoding=false`. PeerTube serves the file as-is. No runner needed, no re-encode, no wasted cycles.
### Step 10: Integration Test
**Full connectivity gate — ALL must pass:**
```bash
ssh peertube 'hostname' # SSH to CT 110
ssh peertube 'curl -s http://localhost:9000/api/v1/config | head -c 50' # PeerTube API
ssh peertube 'systemctl is-active peertube' # PeerTube service
nvidia-smi > /dev/null 2>&1 && echo "GPU OK" # cortex GPU
ffmpeg -encoders 2>/dev/null | grep -q hevc_nvenc && echo "NVENC OK" # HEVC encoder
ssh peertube 'ls /opt/bulk-import/completed/' > /dev/null && echo "Dirs OK" # Pipeline dirs
```
1. Start downloader — let it grab 5-10 videos from 2-3 different channels
2. Verify videos land in `/opt/bulk-import/completed/` with .mp4 + .info.json + .jpg
3. Start transcoder on cortex — verify it pulls videos, encodes H.265 via NVENC (`nvidia-smi` shows encoder utilization)
4. Verify transcoded files land in `/opt/bulk-import/transcoded/` on CT 110, and originals cleared from `completed/`
5. Verify transcoded file is H.265: `ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 <file>` should return `hevc`
6. Start importer — verify videos appear in PeerTube UI with correct metadata, channel assignment, thumbnails
7. Verify playback works at stream.echo6.co (H.265 plays natively in modern browsers via HLS/web-video)
8. Check dedup — restart downloader, verify it skips already-downloaded videos
9. Check VPN rotation — trigger a rate limit (or simulate), verify country switches
### Step 11: Go-Live
**On CT 110:**
```bash
systemctl start pt-downloader && systemctl start pt-importer
systemctl enable pt-downloader && systemctl enable pt-importer
```
**On cortex:**
```bash
systemctl start pt-transcoder
systemctl enable pt-transcoder
```
Monitor for 24 hours. Expected steady-state:
- Downloader: ~30 videos/hr
- Transcoder: ~2-5 videos/hr (bottleneck — NVENC is fast but 1080p H.265 takes time per video)
- Importer: keeps up with transcoder output, ~50/hr capacity but paced by transcoder
- GPU utilization: 80-100% encoder, minimal CUDA (no conflict with Ollama)
**⚠️ The transcoder is the bottleneck.** At ~3 videos/hr average, 70K videos = ~970 days. Strategies to accelerate:
- Lower quality preset: `-preset fast` or `-preset hp` (speed over quality)
- Accept lower CQ: `-cq 32` instead of 28 (smaller files, slightly lower quality)
- Run 2 NVENC sessions in parallel (A4000 supports ~3 concurrent)
- Add a second GPU node
- Accept H.264 for bulk and only H.265 for new imports going forward
---
## Manual Inputs Required (Before CC Can Execute)
| Item | Who | When Needed |
|------|-----|-------------|
| NordVPN token | Matt | Step 3 |
| YouTube cookies.txt | Matt | Step 4 |
| PeerTube admin password | Matt | Step 2 (OAuth) |
---
## Dependencies Between Steps
```
Step 1 (channel map) ──→ Step 2 (create channels) ──→ Step 7 (importer needs channel IDs)
↗
Step 3 (VPN) + Step 4 (cookies) ──→ Step 5 (downloader) ──→ Step 6 (transcoder reads completed/)
↓
Step 7 (importer reads transcoded/)
Step 9 (disable PT transcoding) ←── independent, do anytime before Step 10
Step 10 (integration test) ←── requires ALL of 1-9
Step 11 (go-live) ←── requires Step 10 pass
```
Steps 1-2 and Step 9 are independent workstreams. Steps 3-4 require Matt's manual input. Steps 5, 6, 7 are the three core scripts. Step 6 runs on cortex; everything else runs on CT 110.
---
## What NOT to Build (Phase 3 — WATCHTOWER)
WATCHTOWER (the monitoring dashboard) is Phase 3. Don't build it now. The pipeline scripts should have enough logging that we can monitor via `journalctl` and log files during Phase 2. WATCHTOWER will eventually:
- SSH into CT 110 to read pipeline metrics (but CT 110 is native now, not Docker — queries change)
- Point to cortex instead of old TOC for GPU stats
- Read channel-map.json from `/opt/bulk-import/config/` instead of old `/mnt/data/bulk-import/`
- Need new .env config for all changed IPs
But that's later. Pipeline first.
---
## Channel Management (via RECON Dashboard)
**Added 2026-02-18.** Channel management UI is now in the RECON dashboard Upload tab at `http://192.168.1.130:8420/upload`. No more SSH + manual JSON editing to add channels.
- **Sudoers:** `/etc/sudoers.d/recon-mgmt` on CT 110 — allows zvx to run yt-dlp, psql, and tee as peertube
- **API endpoints** in `/opt/recon/lib/api.py`:
- `GET /api/peertube/channels` — list all channels with video counts from PeerTube DB
- `GET /api/peertube/channels/stats` — total channels, total videos, downloader status
- `POST /api/peertube/channels/add` — resolve YT URL via yt-dlp, create PeerTube channel, update channel-map.json
- `DELETE /api/peertube/channels/<actor_name>` — remove from JSON and PeerTube
- **UI features:** stats bar, add form (URL + category + priority), sortable channel table, remove button
- **All operations go through SSH from CT 130 → CT 110** using the existing `_ssh_peertube()` helper

468
projects/peertube-rebuild.md Normal file
View file

@ -0,0 +1,468 @@
# Project: PeerTube YouTube Archive Rebuild
**Goal:** Rebuild PeerTube at `stream.echo6.co` with Authentik SSO, 18TB NFS storage, and a bulk import pipeline for 250 YouTube channels (~136K videos).
**Status:** Phase 1 — Complete (2026-02-13). CT 110 on media, 192.168.1.170, TS 100.64.0.23, PeerTube v8.0.2
---
## Architecture
```
┌─────────────────────────────────┐
│ utility node │
Internet ──── DNS ──────▶│ Caddy LXC (CT 101) │
│ stream.echo6.co → PT LXC:80 │
└──────────────┬──────────────────┘
│
┌──────────────▼──────────────────┐
│ media node │
│ PeerTube LXC (CT 100) │
│ ├── nginx (port 80) │
│ ├── PeerTube (port 9000) │
│ ├── PostgreSQL 16 │
│ ├── Redis │
│ └── /var/www/peertube/storage │
│ └── NFS mount (18TB) │
└──────────────────────────────────┘
Authentik ◄──── OIDC ────► PeerTube
Phase 2: cortex (VM 150 on TOC, has GPU) = remote transcoding runner
```
**Key decisions:**
- LXC on media node (not VM, not Docker) — CT 100
- Privileged container (NFS bind-mount uid mapping is hell otherwise)
- Native PeerTube install (Node.js + PostgreSQL + Redis + nginx, no Docker)
- PeerTube v8.0.2 — config and nginx template may differ from v6.x docs; verify during install
- Node.js 20 (v8 requirement, not 18)
- Caddy on utility handles TLS, nginx inside LXC handles WebSocket/static files
- Caddy proxies to local IP (192.168.1.x) since PeerTube has OIDC
- Transcoding: 480p + 720p only (storage budget)
- Built-in channel sync DISABLED — bulk pipeline handles imports
- Signup disabled — Authentik SSO only
- NFS storage from pi-nas `/export/peertube` (separate from arr)
---
## Phase 1: PeerTube Up and Secure
### 1.1 Provision LXC on media
**Run:** `runbooks/ct-runbook.md` with these inputs:
| Variable | Value |
|----------|-------|
| Host | media (192.168.1.243) |
| CTID | 100 |
| Hostname | peertube |
| Template | Debian 12 (not Ubuntu — PeerTube docs target Debian) |
| Memory | 4096 MB |
| Cores | 4 |
| Disk | 50 GB root |
| Privileged | YES (override ct-runbook default) |
| Network | DHCP initially |
**Deviations from ct-runbook:**
- Use **Debian 12** template instead of Ubuntu 24.04
- Use **privileged** container (`--unprivileged 0`) for NFS compatibility
- Skip Docker install — PeerTube runs native
- Still do: base packages, zvx user, SSH, Tailscale
### 1.2 Mount NFS storage
On the **media host** (not inside LXC):
```bash
# Mount NFS on host
mkdir -p /mnt/peertube-storage
mount -t nfs 192.168.1.245:/export/peertube /mnt/peertube-storage
# Persist
echo "192.168.1.245:/export/peertube /mnt/peertube-storage nfs defaults,_netdev 0 0" >> /etc/fstab
# Bind-mount into LXC
echo "mp0: /mnt/peertube-storage,mp=/var/www/peertube/storage" >> /etc/pve/lxc/<CTID>.conf
# Restart LXC to pick up mount
pct stop <CTID> && pct start <CTID>
```
**Verify inside LXC:**
```bash
df -h /var/www/peertube/storage # Should show ~18TB
touch /var/www/peertube/storage/test && rm /var/www/peertube/storage/test
```
**NFS details:**
- Server: pi-nas (192.168.1.245 / 100.64.0.21)
- Export: `/export/peertube`
- Access: Already configured in OMV for 100.64.0.0/10 and 192.168.1.0/24
### 1.3 Install PeerTube dependencies
Inside the LXC:
```bash
# PostgreSQL 16
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
apt update && apt install -y postgresql-16 postgresql-contrib-16
sudo -u postgres psql << 'SQL'
CREATE USER peertube WITH PASSWORD '<PG_PASSWORD>';
CREATE DATABASE peertube_prod OWNER peertube;
\c peertube_prod
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;
SQL
# Redis
apt install -y redis-server
sed -i 's/^# requirepass .*/requirepass <REDIS_PASSWORD>/' /etc/redis/redis.conf
sed -i 's/^bind .*/bind 127.0.0.1 -::1/' /etc/redis/redis.conf
systemctl restart redis-server && systemctl enable redis-server
# Node.js 20 (PeerTube v8 requires Node.js 20+)
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs
npm install -g yarn
# ffmpeg (for transcoding)
apt install -y ffmpeg
```
### 1.4 Install PeerTube
```bash
# Create peertube user
adduser --system --group --home /var/www/peertube --shell /bin/bash peertube
chown -R peertube:peertube /var/www/peertube/storage
# Get latest version (joinpeertube.org API is dead, use GitHub)
PEERTUBE_VERSION=$(curl -s https://api.github.com/repos/Chocobozzz/PeerTube/releases/latest | grep -oP '"tag_name": "v\K[^"]+' || echo "8.0.2")
# Download and install
cd /var/www/peertube
sudo -u peertube mkdir -p config
sudo -u peertube mkdir -p storage/{avatars,caches,captions,logs,plugins,previews,redundancy,streaming-playlists,thumbnails,tmp,torrents,videos,bin,storyboards,web-videos,original-video-files}
sudo -u peertube wget -q "https://github.com/Chocobozzz/PeerTube/releases/download/v${PEERTUBE_VERSION}/peertube-v${PEERTUBE_VERSION}.tar.xz"
sudo -u peertube tar xf peertube-v${PEERTUBE_VERSION}.tar.xz
sudo -u peertube ln -s peertube-v${PEERTUBE_VERSION} peertube-latest
cd peertube-latest
sudo -u peertube yarn install --production --pure-lockfile
```
### 1.5 Configure PeerTube
Create `/var/www/peertube/config/local-production.json`:
```json
{
"listen": { "hostname": "0.0.0.0", "port": 9000 },
"webserver": { "https": true, "hostname": "stream.echo6.co", "port": 443 },
"database": {
"hostname": "localhost", "port": 5432,
"name": "peertube_prod", "username": "peertube", "password": "<PG_PASSWORD>"
},
"redis": { "hostname": "localhost", "port": 6379, "auth": "<REDIS_PASSWORD>" },
"storage": {
"avatars": "/var/www/peertube/storage/avatars/",
"caches": "/var/www/peertube/storage/caches/",
"captions": "/var/www/peertube/storage/captions/",
"logs": "/var/www/peertube/storage/logs/",
"plugins": "/var/www/peertube/storage/plugins/",
"previews": "/var/www/peertube/storage/previews/",
"redundancy": "/var/www/peertube/storage/redundancy/",
"streaming_playlists": "/var/www/peertube/storage/streaming-playlists/",
"thumbnails": "/var/www/peertube/storage/thumbnails/",
"tmp": "/var/www/peertube/storage/tmp/",
"torrents": "/var/www/peertube/storage/torrents/",
"videos": "/var/www/peertube/storage/videos/",
"bin": "/var/www/peertube/storage/bin/",
"storyboards": "/var/www/peertube/storage/storyboards/",
"web_videos": "/var/www/peertube/storage/web-videos/",
"original_video_files": "/var/www/peertube/storage/original-video-files/"
},
"admin": { "email": "admin@echo6.co" },
"signup": { "enabled": false },
"import": {
"videos": { "concurrency": 10, "http": { "enabled": true }, "torrent": { "enabled": false } },
"video_channel_synchronization": { "enabled": false }
},
"transcoding": {
"enabled": true, "threads": 2, "concurrency": 2,
"allow_additional_extensions": true, "allow_audio_files": true,
"resolutions": {
"0p": false, "144p": false, "240p": false, "360p": false,
"480p": true, "720p": true,
"1080p": false, "1440p": false, "2160p": false
},
"hls": { "enabled": true },
"web_videos": { "enabled": true }
}
}
```
```bash
chown peertube:peertube /var/www/peertube/config/local-production.json
chmod 600 /var/www/peertube/config/local-production.json
```
### 1.6 nginx (inside LXC)
```bash
apt install -y nginx
rm -f /etc/nginx/sites-enabled/default
cat > /etc/nginx/sites-available/peertube << 'NGINXCONF'
server {
listen 80;
server_name stream.echo6.co;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
client_max_body_size 20G;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
location ~ ^/client/(.*\.(js|css|woff2|otf|ttf|woff|eot|svg|png|jpg|gif|ico|webp))$ {
add_header Cache-Control "public, max-age=31536000, immutable";
alias /var/www/peertube/peertube-latest/client/dist/$1;
}
location ~ ^(/static/(webseed|web-videos|streaming-playlists|redundancy)/.+)$ {
set $upstream_peertube http://127.0.0.1:9000;
try_files /var/www/peertube/storage$1 @api;
root /;
add_header Cache-Control "public, max-age=7200";
}
location @api {
proxy_pass http://127.0.0.1:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location / {
proxy_pass http://127.0.0.1:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 20G;
}
}
NGINXCONF
ln -s /etc/nginx/sites-available/peertube /etc/nginx/sites-enabled/peertube
nginx -t && systemctl restart nginx && systemctl enable nginx
```
### 1.7 systemd service
```bash
cat > /etc/systemd/system/peertube.service << 'EOF'
[Unit]
Description=PeerTube daemon
After=network.target postgresql.service redis-server.service
[Service]
Type=simple
User=peertube
Group=peertube
Environment=NODE_ENV=production
Environment=NODE_CONFIG_DIR=/var/www/peertube/config
WorkingDirectory=/var/www/peertube/peertube-latest
ExecStart=/usr/bin/node dist/server
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=peertube
TimeoutStartSec=60
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable peertube
systemctl start peertube
# Grab auto-generated root password
journalctl -u peertube | grep -i "password"
```
### 1.8 Expose via Caddy
**Run:** `runbooks/expose-service-home.md` with these inputs:
| Variable | Value |
|----------|-------|
| Service | stream |
| Domain | stream.echo6.co |
| Backend IP | PeerTube LXC local IP (192.168.1.x — has OIDC, use local IP pattern) |
| Backend port | 80 |
| Has OIDC | YES (use local IP, not Tailscale) |
DNS record for `stream.echo6.co` does NOT exist in GoDaddy yet — create it pointing to `199.6.36.163`.
Also update dnsmasq split DNS on Contabo — entry exists pointing to old IP `100.64.0.7`, update to `100.64.0.8` (utility Caddy, same pattern as jellyfin/requests).
### 1.9 Authentik OIDC
**Run:** `runbooks/authentik-oidc-application.md` with these inputs:
| Variable | Value |
|----------|-------|
| SERVICE_NAME | PeerTube |
| SERVICE_SLUG | peertube |
| SERVICE_URL | https://stream.echo6.co |
| OIDC_CALLBACK_PATH | Check PeerTube OIDC plugin docs — likely `/plugins/auth-openid-connect/router/code-cb` |
| NEEDS_OFFLINE_ACCESS | yes |
| CLIENT_TYPE | confidential |
**Note:** No existing PeerTube provider in Authentik — create from scratch using the runbook.
Then install the plugin inside PeerTube:
```bash
cd /var/www/peertube/peertube-latest
sudo -u peertube NODE_ENV=production NODE_CONFIG_DIR=/var/www/peertube/config \
node dist/server/tools/peertube-plugins.js install \
--npm-name peertube-plugin-auth-openid-connect
systemctl restart peertube
```
Configure via Admin UI → Plugins → OpenID Connect:
- Discover URL: `https://auth.echo6.co/application/o/peertube/.well-known/openid-configuration`
- Client ID/Secret from Authentik
- Scope: `openid email profile`
- Username property: `preferred_username`
- Display name property: `name`
### 1.10 First login and lockdown
1. Log in as `root` with the auto-generated password
2. Change root password immediately
3. Test Authentik SSO login
4. Promote your Authentik user to admin
5. Admin → Configuration: instance name "Echo6 Archive", signup disabled, HTTP import enabled
### Phase 1 checklist
```
[x] LXC on media — CT 110, privileged, Debian 12, 4C/4GB/50GB
[x] NFS 22TB mounted and writable (/export/peertube from 192.168.1.245)
[x] PostgreSQL 16 + Redis installed
[x] Node.js 22 + pnpm + ffmpeg installed (v8.0.2 requires Node 22, pnpm not yarn)
[x] PeerTube v8.0.2 installed and configured
[x] nginx configured (port 80, WebSocket, static files)
[x] systemd service running
[x] Tailscale registered (100.64.0.23)
[x] Caddy on utility proxying stream.echo6.co → 192.168.1.170:80
[x] DNS verified (GoDaddy + dnsmasq split DNS → 100.64.0.8)
[x] Authentik OIDC working (provider pk:12, app slug: peertube)
[ ] Root password changed, your user promoted to admin (manual step)
```
**Update after Phase 1:**
- `docs/hardware/environment.md` — add PeerTube LXC
- `docs/services/services.md` — add PeerTube entry
- `docs/software/caddy.md` — add stream.echo6.co site block
---
## Phase 2: Import Pipeline
### 2.1 Create PeerTube channels
Script to create all 250 channels from the master spreadsheet via PeerTube API. One channel per YouTube channel, matching names.
### 2.2 Bulk downloader
- yt-dlp with cookies + PO tokens
- `--match-filter "duration > 61"` to exclude Shorts
- Round-robin across channels (5-10 videos per channel, rotate)
- Download to NFS staging area
- Track downloaded video IDs in archive file (prevent re-downloads)
### 2.3 Import pipeline
- Watch staging area for new downloads
- Import to correct PeerTube channel via API
- Move source file after successful import (or delete if transcoded)
- Rate limit to avoid overwhelming PeerTube
### 2.4 GPU transcoding on cortex
- PeerTube remote runner protocol
- cortex already has RTX A4000 + nvidia-container-toolkit
- NVENC encoding for 480p + 720p HLS
- PeerTube delegates transcoding jobs to cortex runner
### Phase 2 checklist
```
[x] 100 channels created in PeerTube (99 planned + extras, channel-map.json at /opt/bulk-import/config/)
[x] yt-dlp configured (cookies, Shorts filter)
[x] Bulk downloader script with round-robin (pt-downloader service on CT 110)
[x] Import pipeline (pt-importer service on CT 110, resumable chunked upload)
[x] Archive tracking (downloaded.txt, downloader-state.json)
[x] GPU transcoding runner on cortex (pt-transcoder service, H.265 NVENC)
[x] PeerTube remote runner on cortex (Whisper auto-captioning, medium model, smart GPU/CPU routing)
[x] Test: full cycle — download → transcode → import → playable
[ ] VPN/IP rotation (NordVPN token pending from Matt)
```
---
## Phase 3: Monitoring
### 3.1 WATCHTOWER dashboard
- Import queue depth and throughput
- Per-channel video counts vs YouTube totals
- Storage usage and growth rate
- Transcoding queue status
### 3.2 Alerts
- Storage threshold warnings (80%, 90%, 95%)
- Stalled imports (no progress for N hours)
- Failed downloads (rate limiting, auth issues)
### Phase 3 checklist
```
[ ] Dashboard showing import progress
[ ] Per-channel completion tracking
[ ] Storage alerts configured
[ ] Stall detection working
```
---
## Reference
- **Master channel list:** `youtube_archive_master.xlsx` (250 channels, 19 categories)
- **PeerTube LXC:** CT 110 on media (192.168.1.243)
- **NFS:** pi-nas (192.168.1.245) export `/export/peertube`
- **Previous PeerTube Tailscale IP:** 100.64.0.7 (do not reuse — assign fresh)
- **Previous bulk import map:** `final-channel-map.json` (lost with crash)
- **Previous download archive:** `downloaded.txt` (21,714 video IDs, lost with crash)
- **Runbooks used:** ct-runbook.md, expose-service-home.md, authentik-oidc-application.md
- **Docs to update after Phase 1:** environment.md, services.md, caddy.md, dns.md (dnsmasq entry)

111
projects/utility-caddy-initial-setup.md Executable file
View file

@ -0,0 +1,111 @@
# Utility Caddy LXC — Initial Setup
One-time setup. Only needed if rebuilding from scratch.
## Overview
| Item | Value |
|------|-------|
| CT ID | 101 |
| Hostname | caddy |
| Local IP | 192.168.1.101 |
| Tailscale IP | 100.64.0.2 |
| Public access | 199.6.36.163 (router forwards 80/443) |
## 1. Create LXC
```bash
ssh root@192.168.1.241
pct create 101 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname caddy \
--cores 1 \
--memory 512 \
--swap 256 \
--rootfs local-lvm:8 \
--net0 name=eth0,bridge=vmbr0,ip=192.168.1.101/24,gw=192.168.1.1 \
--features nesting=1 \
--unprivileged 1 \
--password <from .ref/credentials>
# TUN device for Tailscale
cat >> /etc/pve/lxc/101.conf << EOF
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start 101
```
## 1b. Bootstrap Standard Packages
Run the Echo6 LXC bootstrap script to install sshpass, curl, git, htop, and other standard packages:
```bash
echo6-bootstrap-ct.sh 101
```
If the script isn't on the Proxmox host yet, run `echo6-onboard-node.sh` first. See `runbooks/proxmox-onboard-node.md`.
## 2. Install Tailscale
```bash
pct exec 101 -- bash -c "
echo nameserver 1.1.1.1 > /etc/resolv.conf
apt-get update && apt-get install -y curl
curl -fsSL https://tailscale.com/install.sh | sh
"
```
## 3. Register with Headscale
```bash
pct exec 101 -- tailscale up --login-server https://vpn.echo6.co --hostname caddy
# On Contabo — register the node
ssh root@100.64.0.6 'docker exec headscale-standby headscale nodes register --key <KEY> --user echo6'
# Verify
pct exec 101 -- tailscale status
```
## 4. Install Caddy
```bash
pct exec 101 -- bash -c "
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt | tee /etc/apt/sources.list.d/caddy-stable.list
apt-get update && apt-get install -y caddy
"
```
## 5. Install acme.sh
```bash
pct exec 101 -- bash -c "
curl https://get.acme.sh | sh -s email=admin@echo6.co
"
```
## 6. Create initial Caddyfile
```bash
pct exec 101 -- bash -c "cat > /etc/caddy/Caddyfile << 'EOF'
{
email admin@echo6.co
}
EOF
systemctl enable caddy
systemctl start caddy"
```
## 7. Router port forward
Forward on your router:
- TCP 80 → 192.168.1.101:80
- TCP 443 → 192.168.1.101:443
## Done
Add services using the expose-service-home.md runbook.

222
projects/vaultwarden-deployment.md Normal file
View file

@ -0,0 +1,222 @@
# Vaultwarden Deployment
**Deployed:** 2026-02-05
**Location:** Contabo VPS (5.189.158.149 / 100.64.0.6)
**URL:** https://vault.echo6.co
---
## Service Details
| Setting | Value |
|---------|-------|
| Container | `vaultwarden` |
| Image | `vaultwarden/server:latest` |
| Port | `127.0.0.1:8086` (web), `127.0.0.1:3012` (websocket) |
| Data | `/opt/vaultwarden/data` |
| Config | `/opt/vaultwarden/.env` |
| SSO | Authentik (enabled) |
| Signups | Disabled (invite-only) |
---
## Access
| Method | URL |
|--------|-----|
| Web Vault | https://vault.echo6.co |
| Admin Panel | https://vault.echo6.co/admin |
| SSO Login | "Enterprise Single Sign-On" button |
---
## Configuration Files
### Docker Compose (`/opt/vaultwarden/docker-compose.yml`)
```yaml
services:
vaultwarden:
image: vaultwarden/server:latest
container_name: vaultwarden
restart: unless-stopped
env_file:
- .env
ports:
- "127.0.0.1:8086:80"
- "127.0.0.1:3012:3012"
volumes:
- ./data:/data
environment:
- TZ=America/Boise
```
### Environment (`.env`)
```bash
# Admin
ADMIN_TOKEN=<see credentials file>
DOMAIN=https://vault.echo6.co
# Security
SIGNUPS_ALLOWED=false
INVITATIONS_ALLOWED=true
SHOW_PASSWORD_HINT=false
# WebSocket
WEBSOCKET_ENABLED=true
# SSO (Authentik)
SSO_ENABLED=true
SSO_ONLY=false
SSO_CLIENT_ID=vaultwarden
SSO_CLIENT_SECRET=<see credentials file>
SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/
SSO_PKCE=true
SSO_SCOPES="openid email profile offline_access"
# Timezone
TZ=America/Boise
LOG_LEVEL=info
```
### Caddy Site Block
```caddyfile
vault.echo6.co {
reverse_proxy /notifications/hub 127.0.0.1:3012
reverse_proxy 127.0.0.1:8086
}
```
### dnsmasq Split DNS
```conf
address=/vault.echo6.co/100.64.0.6
```
---
## Authentik SSO Configuration
### Provider Settings (pk=3)
| Setting | Value |
|---------|-------|
| Name | Vaultwarden |
| Client ID | `vaultwarden` |
| Client Type | Confidential |
| Redirect URI | `https://vault.echo6.co/identity/connect/oidc-signin` |
| Signing Key | authentik Internal JWT Certificate (RS256) |
| Access Token Validity | 1 hour |
| Refresh Token Validity | 30 days |
### Scopes
- `openid` - Required for OIDC
- `email` - User email
- `profile` - User profile
- `offline_access` - Refresh tokens
### OIDC Endpoints
| Endpoint | URL |
|----------|-----|
| Discovery | https://auth.echo6.co/application/o/vaultwarden/.well-known/openid-configuration |
| JWKS | https://auth.echo6.co/application/o/vaultwarden/jwks/ |
| Authorize | https://auth.echo6.co/application/o/authorize/ |
| Token | https://auth.echo6.co/application/o/token/ |
---
## Troubleshooting
### SSO Login Loop
**Symptom:** After SSO auth, redirects back to login screen.
**Causes:**
1. Access token too short (< 5 min)
2. Missing `offline_access` scope (no refresh token)
3. Missing signing key (empty JWKS)
**Fix:**
```bash
# Check Authentik provider settings via ak shell
docker exec authentik-server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.get(name='Vaultwarden')
print(f'Access Token: {p.access_token_validity}')
print(f'Signing Key: {p.signing_key}')
print(f'Scopes: {list(p.property_mappings.values_list(\"scope_name\", flat=True))}')"
```
### SSO Discovery Error
**Symptom:** "Failed to discover OpenID provider: Failed to parse server response"
**Causes:**
1. Empty JWKS endpoint (no signing key)
2. Missing property mappings
**Fix:** Add signing key and scopes to Authentik provider.
### View Logs
```bash
# Vaultwarden
docker logs vaultwarden --tail 100 2>&1 | grep -i -E "sso|error"
# Authentik
docker logs authentik-server --tail 100 2>&1 | grep -i vaultwarden
```
---
## Maintenance
### Restart Service
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose restart
```
### Update Image
```bash
ssh root@5.189.158.149
cd /opt/vaultwarden
docker compose pull
docker compose up -d
```
### Backup Data
```bash
# Stop container first
docker compose stop
tar -czf vaultwarden-backup-$(date +%Y%m%d).tar.gz data/
docker compose start
```
---
## Credentials Reference
All credentials stored in `/home/zvx/projects/.ref/credentials`:
```
VAULTWARDEN_URL
VAULTWARDEN_ADMIN_TOKEN
VAULTWARDEN_ADMIN_URL
VAULTWARDEN_OIDC_PROVIDER_ID
VAULTWARDEN_OIDC_CLIENT_ID
VAULTWARDEN_OIDC_CLIENT_SECRET
VAULTWARDEN_OIDC_ISSUER
```
---
*Last updated: 2026-02-05*