echo6-docs/projects/DEPLOY-API-KEYS-TAB.md

RECON Dashboard — API Keys Tab Deployment

Context

SSH into the RECON LXC as zvx: ssh zvx@100.64.0.24 (or 192.168.1.130) Working directory: /opt/recon/ The dashboard is a Flask app in lib/api.py running on port 8420 as a systemd service (recon.service).

We're adding:

1. A new lib/key_manager.py module (thread-safe, hot-reloadable API key store)
2. A new "API Keys" tab on the dashboard
3. API endpoints for key management
4. Hot-reload integration — enricher and extractor pull keys from KeyManager instead of .env directly

Step 1: Deploy key_manager.py

Create /opt/recon/lib/key_manager.py with the contents of the attached key_manager.py file. Copy it exactly — it's a complete, tested module.

Verify it loads:

cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys loaded: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
"

This should show the 4 Gemini keys currently in .env.

Step 2: Add API routes to lib/api.py

Add these routes to lib/api.py. Find where the other /api/ routes are defined and add these in the same pattern:

from lib.key_manager import get_key_manager

# ── API Keys Management ──

@app.route('/keys')
def keys_page():
    """API Keys management page."""
    return render_template_string(KEYS_TEMPLATE)

@app.route('/api/keys', methods=['GET'])
def api_get_keys():
    """Get all API keys (masked) with stats."""
    km = get_key_manager()
    return jsonify({
        'gemini': {
            'keys': km.get_masked_keys(),
            'count': km.get_gemini_key_count(),
        },
        # Placeholder sections for future services
        'services': {
            'tei': {
                'host': config.get('embedding', {}).get('tei_host', 'unknown'),
                'port': config.get('embedding', {}).get('tei_port', 'unknown'),
                'status': 'managed in config.yaml'
            },
            'qdrant': {
                'host': config.get('vector_db', {}).get('host', 'unknown'),
                'port': config.get('vector_db', {}).get('port', 'unknown'),
                'status': 'managed in config.yaml'
            },
            'ollama': {
                'host': config.get('embedding', {}).get('ollama_host', 'unknown'),
                'port': config.get('embedding', {}).get('ollama_port', 'unknown'),
                'status': 'managed in config.yaml'
            }
        }
    })

@app.route('/api/keys/gemini', methods=['POST'])
def api_add_gemini_key():
    """Add a new Gemini API key."""
    data = request.get_json()
    if not data or 'key' not in data:
        return jsonify({'error': 'Missing "key" field'}), 400
    km = get_key_manager()
    try:
        # Optionally validate before adding
        if data.get('validate', True):
            valid, msg = km.validate_key(data['key'])
            if not valid:
                return jsonify({'error': f'Key validation failed: {msg}'}), 400
        idx = km.add_gemini_key(data['key'])
        return jsonify({'success': True, 'index': idx, 'count': km.get_gemini_key_count()})
    except ValueError as e:
        return jsonify({'error': str(e)}), 400

@app.route('/api/keys/gemini/<int:index>', methods=['PUT'])
def api_replace_gemini_key(index):
    """Replace a Gemini API key at a specific index."""
    data = request.get_json()
    if not data or 'key' not in data:
        return jsonify({'error': 'Missing "key" field'}), 400
    km = get_key_manager()
    try:
        if data.get('validate', True):
            valid, msg = km.validate_key(data['key'])
            if not valid:
                return jsonify({'error': f'Key validation failed: {msg}'}), 400
        km.replace_gemini_key(index, data['key'])
        return jsonify({'success': True, 'count': km.get_gemini_key_count()})
    except (IndexError, ValueError) as e:
        return jsonify({'error': str(e)}), 400

@app.route('/api/keys/gemini/<int:index>', methods=['DELETE'])
def api_delete_gemini_key(index):
    """Remove a Gemini API key by index."""
    km = get_key_manager()
    try:
        masked = km.remove_gemini_key(index)
        return jsonify({'success': True, 'removed': masked, 'count': km.get_gemini_key_count()})
    except (IndexError, ValueError) as e:
        return jsonify({'error': str(e)}), 400

@app.route('/api/keys/gemini/validate', methods=['POST'])
def api_validate_gemini_keys():
    """Validate all loaded Gemini keys."""
    km = get_key_manager()
    results = km.validate_all()
    return jsonify({'results': results})

@app.route('/api/keys/gemini/<int:index>/validate', methods=['POST'])
def api_validate_single_gemini_key(index):
    """Validate a single Gemini key by index."""
    km = get_key_manager()
    key = km.get_gemini_key(index)
    if key is None:
        return jsonify({'error': f'No key at index {index}'}), 404
    valid, msg = km.validate_key(key)
    return jsonify({'index': index, 'valid': valid, 'message': msg})

@app.route('/api/keys/gemini/reveal/<int:index>', methods=['POST'])
def api_reveal_gemini_key(index):
    """Reveal full key (for copy). Requires confirmation in request body."""
    data = request.get_json() or {}
    if not data.get('confirm'):
        return jsonify({'error': 'Send {"confirm": true} to reveal key'}), 400
    km = get_key_manager()
    key = km.get_gemini_key(index)
    if key is None:
        return jsonify({'error': f'No key at index {index}'}), 404
    return jsonify({'index': index, 'key': key})

@app.route('/api/keys/reload', methods=['POST'])
def api_reload_keys():
    """Force reload keys from .env file."""
    km = get_key_manager()
    count = km.reload_from_env()
    return jsonify({'success': True, 'count': count})

Important: Make sure config refers to whatever variable holds the parsed config.yaml in the existing code. Look at how other routes reference config and use the same pattern (likely get_config() from lib/utils.py).

Step 3: Add the KEYS_TEMPLATE

Add this HTML template string to lib/api.py, alongside the other template strings (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, etc.):

KEYS_TEMPLATE = """
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>RECON — API Keys</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0a0a0a; color: #e0e0e0; min-height: 100vh; }
        
        /* Nav */
        .nav { background: #111; border-bottom: 1px solid #222; padding: 0 24px; display: flex; align-items: center; height: 56px; }
        .nav-brand { font-size: 18px; font-weight: 700; color: #4ade80; margin-right: 32px; text-decoration: none; letter-spacing: 1px; }
        .nav-links { display: flex; gap: 4px; }
        .nav-links a { color: #888; text-decoration: none; padding: 8px 16px; border-radius: 6px; font-size: 14px; transition: all 0.15s; }
        .nav-links a:hover { color: #e0e0e0; background: #1a1a1a; }
        .nav-links a.active { color: #4ade80; background: #1a2e1a; }
        
        /* Layout */
        .container { max-width: 960px; margin: 0 auto; padding: 32px 24px; }
        h1 { font-size: 24px; font-weight: 600; margin-bottom: 8px; }
        .subtitle { color: #666; font-size: 14px; margin-bottom: 32px; }
        
        /* Section */
        .section { background: #111; border: 1px solid #222; border-radius: 12px; padding: 24px; margin-bottom: 24px; }
        .section-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px; }
        .section-title { font-size: 16px; font-weight: 600; display: flex; align-items: center; gap: 8px; }
        .section-title .icon { font-size: 20px; }
        .section-badge { background: #1a2e1a; color: #4ade80; font-size: 12px; padding: 2px 10px; border-radius: 10px; font-weight: 500; }
        
        /* Key list */
        .key-list { display: flex; flex-direction: column; gap: 12px; }
        .key-row { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; display: flex; align-items: center; gap: 16px; transition: border-color 0.15s; }
        .key-row:hover { border-color: #333; }
        .key-index { background: #1a1a1a; color: #666; width: 32px; height: 32px; border-radius: 6px; display: flex; align-items: center; justify-content: center; font-size: 13px; font-weight: 600; flex-shrink: 0; }
        .key-value { font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; color: #aaa; flex-grow: 1; word-break: break-all; }
        .key-meta { display: flex; gap: 16px; font-size: 12px; color: #555; flex-shrink: 0; }
        .key-meta span { white-space: nowrap; }
        .key-actions { display: flex; gap: 6px; flex-shrink: 0; }
        
        /* Status badges */
        .badge-valid { color: #4ade80; }
        .badge-invalid { color: #f87171; }
        .badge-unknown { color: #666; }
        .badge-ratelimit { color: #fbbf24; }
        
        /* Buttons */
        .btn { padding: 6px 14px; border-radius: 6px; border: 1px solid #333; background: #1a1a1a; color: #ccc; font-size: 13px; cursor: pointer; transition: all 0.15s; display: inline-flex; align-items: center; gap: 6px; }
        .btn:hover { background: #222; border-color: #444; color: #fff; }
        .btn-primary { background: #1a3a1a; border-color: #2a5a2a; color: #4ade80; }
        .btn-primary:hover { background: #2a4a2a; border-color: #3a6a3a; }
        .btn-danger { background: #2a1a1a; border-color: #5a2a2a; color: #f87171; }
        .btn-danger:hover { background: #3a2020; border-color: #6a3030; }
        .btn-sm { padding: 4px 10px; font-size: 12px; }
        .btn:disabled { opacity: 0.4; cursor: not-allowed; }
        
        /* Input */
        .input-row { display: flex; gap: 8px; margin-top: 16px; }
        .input-row input { flex-grow: 1; background: #0d0d0d; border: 1px solid #333; border-radius: 6px; padding: 10px 14px; color: #e0e0e0; font-family: 'SF Mono', 'Consolas', monospace; font-size: 14px; outline: none; }
        .input-row input:focus { border-color: #4ade80; }
        .input-row input::placeholder { color: #444; }
        
        /* Status message */
        .status-msg { padding: 12px 16px; border-radius: 8px; font-size: 13px; margin-top: 12px; display: none; }
        .status-msg.success { display: block; background: #0d1f0d; border: 1px solid #1a3a1a; color: #4ade80; }
        .status-msg.error { display: block; background: #1f0d0d; border: 1px solid #3a1a1a; color: #f87171; }
        .status-msg.info { display: block; background: #0d0d1f; border: 1px solid #1a1a3a; color: #60a5fa; }
        
        /* Service placeholder cards */
        .service-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 12px; }
        .service-card { background: #0d0d0d; border: 1px solid #1a1a1a; border-radius: 8px; padding: 16px; }
        .service-card .svc-name { font-weight: 600; font-size: 14px; margin-bottom: 4px; }
        .service-card .svc-endpoint { font-family: monospace; font-size: 13px; color: #666; margin-bottom: 8px; }
        .service-card .svc-note { font-size: 12px; color: #444; font-style: italic; }
        
        /* Spinner */
        .spinner { display: inline-block; width: 14px; height: 14px; border: 2px solid #333; border-top-color: #4ade80; border-radius: 50%; animation: spin 0.6s linear infinite; }
        @keyframes spin { to { transform: rotate(360deg); } }
        
        /* Confirm overlay */
        .confirm-overlay { display: none; position: fixed; inset: 0; background: rgba(0,0,0,0.7); z-index: 100; align-items: center; justify-content: center; }
        .confirm-overlay.active { display: flex; }
        .confirm-box { background: #111; border: 1px solid #333; border-radius: 12px; padding: 24px; max-width: 420px; width: 90%; }
        .confirm-box h3 { margin-bottom: 12px; font-size: 16px; }
        .confirm-box p { color: #888; font-size: 14px; margin-bottom: 20px; }
        .confirm-actions { display: flex; gap: 8px; justify-content: flex-end; }
    </style>
</head>
<body>
    <nav class="nav">
        <a href="/" class="nav-brand">RECON</a>
        <div class="nav-links">
            <a href="/">Dashboard</a>
            <a href="/search">Search</a>
            <a href="/catalogue">Catalogue</a>
            <a href="/upload">Upload</a>
            <a href="/web-ingest">Web Ingest</a>
            <a href="/failures">Failures</a>
            <a href="/keys" class="active">API Keys</a>
        </div>
    </nav>
    
    <div class="container">
        <h1>API Key Management</h1>
        <p class="subtitle">Manage API keys for pipeline workers. Changes take effect immediately — no restart required.</p>
        
        <!-- Gemini API Keys -->
        <div class="section">
            <div class="section-header">
                <div class="section-title">
                    <span class="icon">🔑</span>
                    Gemini API Keys
                    <span class="section-badge" id="key-count-badge">0 keys</span>
                </div>
                <div style="display: flex; gap: 8px;">
                    <button class="btn" onclick="validateAllKeys()" id="btn-validate-all">Validate All</button>
                    <button class="btn" onclick="reloadFromEnv()">Reload .env</button>
                </div>
            </div>
            
            <div class="key-list" id="key-list">
                <div style="color: #444; text-align: center; padding: 20px;">Loading keys...</div>
            </div>
            
            <!-- Add key input -->
            <div class="input-row">
                <input type="text" id="new-key-input" placeholder="Paste new Gemini API key..." autocomplete="off" spellcheck="false">
                <button class="btn btn-primary" onclick="addKey()" id="btn-add">Add Key</button>
            </div>
            
            <div id="status-msg" class="status-msg"></div>
            
            <div style="margin-top: 16px; font-size: 12px; color: #444;">
                <strong>Used by:</strong> Enrichment (text → concepts, 16 workers) · Vision OCR (scanned PDF fallback) · Title extraction
            </div>
        </div>
        
        <!-- Future Service Endpoints -->
        <div class="section">
            <div class="section-header">
                <div class="section-title">
                    <span class="icon">🔌</span>
                    Service Endpoints
                    <span class="section-badge" style="background: #1a1a2e; color: #60a5fa;">config.yaml</span>
                </div>
            </div>
            
            <div class="service-grid" id="service-grid">
                <div style="color: #444; text-align: center; padding: 20px;">Loading...</div>
            </div>
            
            <div style="margin-top: 16px; font-size: 12px; color: #444;">
                Service endpoints are currently managed in <code style="background:#1a1a1a; padding: 2px 6px; border-radius: 3px;">/opt/recon/config.yaml</code>. 
                Dashboard editing coming in a future update.
            </div>
        </div>
    </div>
    
    <!-- Confirm dialog -->
    <div class="confirm-overlay" id="confirm-overlay">
        <div class="confirm-box">
            <h3 id="confirm-title">Confirm</h3>
            <p id="confirm-message">Are you sure?</p>
            <div class="confirm-actions">
                <button class="btn" onclick="closeConfirm()">Cancel</button>
                <button class="btn btn-danger" id="confirm-action-btn" onclick="confirmAction()">Confirm</button>
            </div>
        </div>
    </div>
    
    <script>
        let pendingAction = null;
        
        // ── Load keys on page load ──
        async function loadKeys() {
            try {
                const resp = await fetch('/api/keys');
                const data = await resp.json();
                renderKeys(data.gemini);
                renderServices(data.services);
            } catch (e) {
                showStatus('Failed to load keys: ' + e.message, 'error');
            }
        }
        
        function renderKeys(gemini) {
            const list = document.getElementById('key-list');
            const badge = document.getElementById('key-count-badge');
            badge.textContent = gemini.count + ' key' + (gemini.count !== 1 ? 's' : '');
            
            if (gemini.keys.length === 0) {
                list.innerHTML = '<div style="color:#f87171; text-align:center; padding:20px;">⚠ No Gemini keys loaded — pipeline cannot enrich or OCR</div>';
                return;
            }
            
            list.innerHTML = gemini.keys.map(k => {
                let validClass = 'badge-unknown';
                let validIcon = '○';
                if (k.valid === true) { validClass = 'badge-valid'; validIcon = '✓'; }
                else if (k.valid === false) { validClass = 'badge-invalid'; validIcon = '✗'; }
                
                return `
                <div class="key-row" id="key-row-${k.index}">
                    <div class="key-index">${k.index + 1}</div>
                    <div class="key-value" id="key-val-${k.index}">${k.masked}</div>
                    <div class="key-meta">
                        <span class="${validClass}">${validIcon}</span>
                        ${k.calls > 0 ? `<span>${k.calls} calls</span>` : ''}
                        ${k.errors > 0 ? `<span style="color:#f87171">${k.errors} err</span>` : ''}
                    </div>
                    <div class="key-actions">
                        <button class="btn btn-sm" onclick="validateKey(${k.index})" title="Validate">Test</button>
                        <button class="btn btn-sm" onclick="revealKey(${k.index})" title="Reveal full key">👁</button>
                        <button class="btn btn-sm" onclick="promptReplace(${k.index})" title="Replace">↻</button>
                        <button class="btn btn-sm btn-danger" onclick="promptDelete(${k.index})" title="Remove">✕</button>
                    </div>
                </div>
                `;
            }).join('');
        }
        
        function renderServices(services) {
            const grid = document.getElementById('service-grid');
            const svcMap = {
                tei: { label: 'TEI Embeddings', icon: '📐' },
                qdrant: { label: 'Qdrant Vector DB', icon: '🗃' },
                ollama: { label: 'Ollama (Fallback)', icon: '🧠' }
            };
            
            grid.innerHTML = Object.entries(services).map(([key, svc]) => {
                const info = svcMap[key] || { label: key, icon: '⚙' };
                return `
                <div class="service-card">
                    <div class="svc-name">${info.icon} ${info.label}</div>
                    <div class="svc-endpoint">${svc.host}:${svc.port}</div>
                    <div class="svc-note">${svc.status}</div>
                </div>
                `;
            }).join('');
        }
        
        // ── Key operations ──
        
        async function addKey() {
            const input = document.getElementById('new-key-input');
            const key = input.value.trim();
            if (!key) { showStatus('Paste a key first', 'error'); return; }
            
            const btn = document.getElementById('btn-add');
            btn.disabled = true;
            btn.innerHTML = '<span class="spinner"></span> Validating...';
            
            try {
                const resp = await fetch('/api/keys/gemini', {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ key: key, validate: true })
                });
                const data = await resp.json();
                if (resp.ok) {
                    showStatus('Key added and validated ✓', 'success');
                    input.value = '';
                    loadKeys();
                } else {
                    showStatus(data.error || 'Failed to add key', 'error');
                }
            } catch (e) {
                showStatus('Network error: ' + e.message, 'error');
            } finally {
                btn.disabled = false;
                btn.innerHTML = 'Add Key';
            }
        }
        
        async function validateKey(index) {
            const row = document.getElementById(`key-row-${index}`);
            row.style.borderColor = '#333';
            
            try {
                const resp = await fetch(`/api/keys/gemini/${index}/validate`, { method: 'POST' });
                const data = await resp.json();
                if (data.valid) {
                    row.style.borderColor = '#2a5a2a';
                    showStatus(`Key ${index + 1}: ${data.message}`, 'success');
                } else {
                    row.style.borderColor = '#5a2a2a';
                    showStatus(`Key ${index + 1}: ${data.message}`, 'error');
                }
                setTimeout(loadKeys, 500);
            } catch (e) {
                showStatus('Validation failed: ' + e.message, 'error');
            }
        }
        
        async function validateAllKeys() {
            const btn = document.getElementById('btn-validate-all');
            btn.disabled = true;
            btn.innerHTML = '<span class="spinner"></span> Validating...';
            
            try {
                const resp = await fetch('/api/keys/gemini/validate', { method: 'POST' });
                const data = await resp.json();
                const valid = data.results.filter(r => r.valid).length;
                const total = data.results.length;
                showStatus(`Validated: ${valid}/${total} keys are working`, valid === total ? 'success' : 'error');
                loadKeys();
            } catch (e) {
                showStatus('Validation failed: ' + e.message, 'error');
            } finally {
                btn.disabled = false;
                btn.innerHTML = 'Validate All';
            }
        }
        
        async function revealKey(index) {
            try {
                const resp = await fetch(`/api/keys/gemini/reveal/${index}`, {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ confirm: true })
                });
                const data = await resp.json();
                if (resp.ok) {
                    const el = document.getElementById(`key-val-${index}`);
                    el.textContent = data.key;
                    el.style.color = '#e0e0e0';
                    // Auto-hide after 10s
                    setTimeout(() => loadKeys(), 10000);
                }
            } catch (e) {
                showStatus('Failed to reveal: ' + e.message, 'error');
            }
        }
        
        function promptReplace(index) {
            const newKey = prompt(`Paste replacement for key ${index + 1}:`);
            if (newKey && newKey.trim()) {
                replaceKey(index, newKey.trim());
            }
        }
        
        async function replaceKey(index, newKey) {
            try {
                const resp = await fetch(`/api/keys/gemini/${index}`, {
                    method: 'PUT',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ key: newKey, validate: true })
                });
                const data = await resp.json();
                if (resp.ok) {
                    showStatus(`Key ${index + 1} replaced and validated ✓`, 'success');
                    loadKeys();
                } else {
                    showStatus(data.error || 'Failed to replace key', 'error');
                }
            } catch (e) {
                showStatus('Network error: ' + e.message, 'error');
            }
        }
        
        function promptDelete(index) {
            pendingAction = () => deleteKey(index);
            document.getElementById('confirm-title').textContent = 'Remove Key ' + (index + 1);
            document.getElementById('confirm-message').textContent = 'This key will be removed from the pipeline immediately. Workers using this key will switch to remaining keys.';
            document.getElementById('confirm-overlay').classList.add('active');
        }
        
        async function deleteKey(index) {
            closeConfirm();
            try {
                const resp = await fetch(`/api/keys/gemini/${index}`, { method: 'DELETE' });
                const data = await resp.json();
                if (resp.ok) {
                    showStatus(`Removed key: ${data.removed}`, 'success');
                    loadKeys();
                } else {
                    showStatus(data.error || 'Failed to remove key', 'error');
                }
            } catch (e) {
                showStatus('Network error: ' + e.message, 'error');
            }
        }
        
        async function reloadFromEnv() {
            try {
                const resp = await fetch('/api/keys/reload', { method: 'POST' });
                const data = await resp.json();
                if (resp.ok) {
                    showStatus(`Reloaded ${data.count} key(s) from .env`, 'info');
                    loadKeys();
                }
            } catch (e) {
                showStatus('Reload failed: ' + e.message, 'error');
            }
        }
        
        // ── Helpers ──
        
        function showStatus(msg, type) {
            const el = document.getElementById('status-msg');
            el.textContent = msg;
            el.className = 'status-msg ' + type;
            if (type === 'success' || type === 'info') {
                setTimeout(() => { el.className = 'status-msg'; }, 5000);
            }
        }
        
        function confirmAction() {
            if (pendingAction) { pendingAction(); pendingAction = null; }
        }
        
        function closeConfirm() {
            document.getElementById('confirm-overlay').classList.remove('active');
            pendingAction = null;
        }
        
        // Handle Enter key in input
        document.getElementById('new-key-input').addEventListener('keydown', e => {
            if (e.key === 'Enter') addKey();
        });
        
        // Load on page ready
        loadKeys();
    </script>
</body>
</html>
"""

Step 4: Add "API Keys" to the nav on ALL existing pages

Find the nav HTML in every existing template (DASHBOARD_TEMPLATE, SEARCH_TEMPLATE, CATALOGUE_TEMPLATE, UPLOAD_TEMPLATE, WEB_INGEST_TEMPLATE, FAILURES_TEMPLATE). Each has a <nav> with links. Add the API Keys link:

<a href="/keys">API Keys</a>

Add it after the "Failures" link in each template's nav. The exact pattern to find is something like:

<a href="/failures"...>Failures</a>

Add right after it:

<a href="/keys">API Keys</a>

Step 5: Wire enricher.py and extractor.py to use KeyManager

This is the hot-reload part. Currently these modules read keys from config/env at startup. We need them to call get_key_manager().get_gemini_keys() each time they need a key, so new keys take effect immediately.

In lib/enricher.py:

Find where Gemini API keys are loaded or selected (likely something like cfg['gemini_keys'] or reading from .env or os.environ). Replace that with:

from lib.key_manager import get_key_manager

# Where a key is selected for a worker (probably in the worker function):
km = get_key_manager()
keys = km.get_gemini_keys()
key = keys[worker_index % len(keys)]  # Round-robin across available keys

# After each Gemini call, record usage:
km.record_usage(worker_index % len(keys), success=True)  # or success=False on error

The exact integration depends on the current code structure. Look for:

• genai.configure(api_key=...) calls
• cfg['gemini_keys'] or config['gemini_keys'] references
• Any os.environ.get('GEMINI_KEY') calls

Replace the key source with get_key_manager() calls. The key point: don't cache the key list — call km.get_gemini_keys() or km.get_gemini_key(index) each time so hot-reload works.

In lib/extractor.py:

Same pattern for the Gemini Vision OCR fallback and the title extraction call. Find where Gemini is configured and replace the key source.

Step 6: Test

Test the module:

cd /opt/recon && source venv/bin/activate
python3 -c "
from lib.key_manager import get_key_manager
km = get_key_manager()
print(f'Keys: {km.get_gemini_key_count()}')
print(f'Masked: {km.get_masked_keys()}')
results = km.validate_all()
for r in results:
    print(f'  Key {r[\"index\"] + 1}: {\"✓\" if r[\"valid\"] else \"✗\"} - {r[\"message\"]}')
"

Test the API:

# Get keys (masked)
curl -s http://localhost:8420/api/keys | python3 -m json.tool

# Validate all
curl -s -X POST http://localhost:8420/api/keys/gemini/validate | python3 -m json.tool

# Validate single
curl -s -X POST http://localhost:8420/api/keys/gemini/0/validate | python3 -m json.tool

Test the dashboard:

Open http://100.64.0.24:8420/keys in a browser. Verify:

• All 4 Gemini keys show up (masked)
• "Validate All" tests each key
• "Test" button validates individual keys
• Eye icon reveals the full key (auto-hides after 10s)
• "Add Key" validates before adding
• "Remove" shows confirmation dialog
• "Replace" prompts for new key and validates
• "Reload .env" picks up external edits
• Service endpoints section shows TEI, Qdrant, Ollama from config.yaml

Test hot-reload:

1. Open the dashboard, note 4 keys
2. Add a 5th key via the dashboard
3. Check .env: cat /opt/recon/.env — should have 5 GEMINI_KEY entries
4. Watch logs: journalctl -u recon -f | grep key_manager — should show key added
5. The pipeline should immediately start using all 5 keys (enricher round-robins)

Restart and verify persistence:

sudo systemctl restart recon
curl -s http://localhost:8420/api/keys | python3 -m json.tool
# Should show same keys as before restart

Report back:

• Screenshot or curl output of /api/keys
• Validate All results
• Confirm nav link appears on all pages
• Confirm hot-reload works (add key, verify enricher uses it)
• Any issues with the existing code structure that needed adapting