cce29c595d
Files changed: ", c.get(k))\nPY\n\\\"\n\"" engine/config.yaml engine/lib/lint.py engine/lib/vocab_gen.py engine/lint-report.md engine/sweep.sh vault/.obsidian/workspace.json vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md vault/archive/projects/vaultwarden-plan.md vault/docs/matrix/matrix_host.md vault/docs/matrix/synapse.md vault/docs/services/services.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/dns.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/expose-service-contabo.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md
148 lines
6.7 KiB
Markdown
148 lines
6.7 KiB
Markdown
---
|
|
title: RECON — Knowledge Extraction Pipeline
|
|
type: reference
|
|
tags:
|
|
- recon
|
|
aliases: []
|
|
related:
|
|
- [[recon-operations]]
|
|
- [[ia-download-queue]]
|
|
- [[services]]
|
|
- [[usenet]]
|
|
- [[caddy]]
|
|
updated: 2026-07-11
|
|
---
|
|
# RECON — Knowledge Extraction Pipeline
|
|
|
|
## Overview
|
|
|
|
RECON extracts knowledge from PDFs and web content into a searchable vector database. PDFs are scanned from an NFS-mounted library, text is extracted (with Gemini Vision fallback for scanned docs), concepts are enriched via Gemini, and embeddings are stored in Qdrant. Aurora (Open WebUI) queries the knowledge base via RAG filter.
|
|
|
|
## Location
|
|
|
|
- **Host:** recon-vm (VM 1130 on data node, 192.168.1.240) — migrated from CT 130 on 2026-04-19
|
|
- **IP:** 192.168.1.130 / 100.64.0.24 (Tailscale)
|
|
- **Repo:** github.com/zvx-echo6/recon (branch `master`; Forge mirror `matt/recon`)
|
|
- **Install:** `/opt/recon/` (Python venv, in sync with origin)
|
|
- **User:** zvx
|
|
- **Service:** `recon.service`, `recon-watchdog.service`, `kiwix.service` (systemd)
|
|
- **Entrypoints:** `recon.py service` (dashboard/API :8420) + `recon.py pipeline watch` (watchdog, auto-ingests new PDFs in `/mnt/library`)
|
|
- **Dashboard:** https://recon.echo6.co (internal: http://100.64.0.24:8420)
|
|
- **Health:** https://recon.echo6.co/api/health
|
|
|
|
## Stack
|
|
|
|
| Component | Technology | Location |
|
|
|-----------|-----------|----------|
|
|
| Pipeline + CLI | Python 3.12, argparse | /opt/recon/recon.py |
|
|
| Dashboard + API | Flask | /opt/recon/lib/api.py (port 8420) |
|
|
| Status DB | SQLite (WAL mode) | /opt/recon/data/recon.db |
|
|
| Vector DB | Qdrant | cortex:6333 (Docker) |
|
|
| Embeddings | TEI (bge-m3, 1024-dim) | cortex:8090 (Docker) |
|
|
| Sparse embeddings | recon-sparse (bge-m3 SPLADE) | cortex:8091 (systemd) |
|
|
| Enrichment | Gemini `gemini-3.1-flash-lite` (enforced) | Google API (4 keys) |
|
|
| Vision Ocr | gemini-3.1-flash-lite | Google API (shared keys) |
|
|
| Text extraction | PyPDF2, poppler-utils, Tesseract | Local |
|
|
| PDF source | NFS | pi-nas:/export/library → /mnt/library |
|
|
| File server | nginx | localhost:8888 → files.echo6.co |
|
|
|
|
## Pipeline Stages
|
|
|
|
All stages run concurrently as daemon threads in the service:
|
|
|
|
1. **Scanner** (hourly) — walks /mnt/library, catalogues new PDFs, queues them
|
|
2. **Extract** (4 workers) — PyPDF2 → pdftotext → Tesseract → Gemini Vision per page
|
|
3. **Enrich** (16 workers, 4 API keys) — Gemini extracts structured concepts from text windows
|
|
4. **Embed** (4 workers) — TEI generates vectors, upserted to Qdrant
|
|
|
|
## Extraction Chain
|
|
|
|
Per page, in order. Each method only runs if the previous returned <50 chars:
|
|
|
|
1. **PyPDF2** — fast, free, works on text-based PDFs
|
|
2. **pdftotext** (poppler) — handles some PDFs PyPDF2 misses
|
|
3. **Tesseract OCR** — renders page to image, runs local OCR
|
|
4. **Gemini Vision** — renders page to PNG, sends to gemini-3.1-flash-lite vision API
|
|
|
|
Method tracking saved in `data/text/{hash}/meta.json` as `ocr_methods` dict.
|
|
|
|
## Scale (as of Feb 2026)
|
|
|
|
- ~10,162 documents in pipeline
|
|
- ~95,000+ vectors in Qdrant (HNSW index, <10ms search latency)
|
|
- Collection: `recon_knowledge_hybrid`
|
|
- ~13,239 PDFs catalogued from NFS library
|
|
|
|
## Resilience
|
|
|
|
- **Enricher**: Exponential backoff (5s→80s) for transient errors (429, 500, 503). Window-level failure isolation — partial enrichment beats zero.
|
|
- **Extractor**: Per-page timeout (30s), per-document timeout (1800s). Partial extractions saved.
|
|
- **Embedder**: Skip-on-failure per concept, batch processing.
|
|
- **Service**: Restart=on-failure, RestartSec=30, MemoryMax=3G.
|
|
|
|
## Configuration
|
|
|
|
**Config file:** `/opt/recon/config.yaml`
|
|
|
|
Key sections:
|
|
- `processing.extract_workers` (4), `enrich_workers` (16), `embed_workers` (4)
|
|
- `processing.extract_timeout` (1800s), `page_timeout` (30s)
|
|
- `processing.enrich_max_retries` (5), `enrich_base_delay` (5.0)
|
|
- `gemini.model` (gemini-3.1-flash-lite), `gemini.response_mime_type` (application/json)
|
|
- `service.scan_interval` (3600), `stage_poll_interval` (30)
|
|
|
|
**API keys:** `/opt/recon/.env` — GEMINI_KEY_1 through GEMINI_KEY_4
|
|
|
|
## API Endpoints
|
|
|
|
| Endpoint | Method | Purpose |
|
|
|----------|--------|---------|
|
|
| `/` | GET | Dashboard HTML |
|
|
| `/api/knowledge-stats` | GET | Full pipeline stats, per-source breakdown |
|
|
| `/api/health` | GET | Health check (Qdrant, TEI, NFS, Gemini, pipeline) |
|
|
| `/api/search` | GET | Vector search (`?q=query&limit=5`) |
|
|
| `/api/upload` | POST | Upload PDF (multipart: file + category) |
|
|
| `/api/upload/<hash>/status` | GET | Upload status tracking |
|
|
| `/api/upload/categories` | GET | Available upload categories |
|
|
| `/api/ingest` | POST | Ingest intel JSON data |
|
|
| `/api/peertube/channels` | GET | List all channels from channel-map.json with video counts from PeerTube DB |
|
|
| `/api/peertube/channels/stats` | GET | Channel count, total videos, downloader status |
|
|
| `/api/peertube/channels/add` | POST | Add channel: resolve YT URL, create PeerTube channel, update JSON |
|
|
| `/api/peertube/channels/<name>` | DELETE | Remove channel from JSON and optionally from PeerTube |
|
|
|
|
## Backups
|
|
|
|
- **Destination:** `root@100.64.0.40:/opt/backups/recon/` (edge1 — retains the storage/public IP post-rebuild)
|
|
- **Full sync:** every 6 hours (concepts, text, DB, config)
|
|
- **DB snapshot:** every 2 hours
|
|
- **Recovery:** restore from edge1 → `recon rebuild` (reconstructs Qdrant from concept JSONs)
|
|
- **Critical data:** `data/concepts/` — Gemini extraction work, costs money to regenerate
|
|
|
|
## Key Files
|
|
|
|
```
|
|
/opt/recon/
|
|
├── recon.py # CLI entry point + service command
|
|
├── config.yaml # Configuration
|
|
├── .env # Gemini API keys
|
|
├── PROJECT-BIBLE.md # Full documentation
|
|
├── lib/
|
|
│ ├── api.py # Flask dashboard + API
|
|
│ ├── extractor.py # PDF → text (4-method chain)
|
|
│ ├── enricher.py # Text → concepts (Gemini)
|
|
│ ├── embedder.py # Concepts → vectors (TEI/Qdrant)
|
|
│ ├── status.py # SQLite DB (WAL, thread-safe)
|
|
│ └── utils.py # Config, hashing, logging
|
|
├── scripts/
|
|
│ ├── backup.sh # Backup to edge1
|
|
│ ├── validate.py # Pipeline consistency checker
|
|
│ └── rebuild_qdrant.py # Nuclear Qdrant rebuild
|
|
└── data/
|
|
├── recon.db # SQLite status DB
|
|
├── concepts/{hash}/ # Enriched concept JSONs
|
|
└── text/{hash}/ # Extracted page text
|
|
```
|
|
|
|
---
|
|
|
|
*Last updated: 2026-07-11 — Gemini model refs updated to gemini-3.1-flash-lite (retired gemini-2.5-flash-lite); backup destination corrected Contabo (100.64.0.1, dead) → edge1 (100.64.0.40). Prior: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current deployment) — verified against live 2026-06-18*
|