echo6-docs/docs/software/recon.md

RECON — Knowledge Extraction Pipeline

Overview

RECON extracts knowledge from PDFs and web content into a searchable vector database. PDFs are scanned from an NFS-mounted library, text is extracted (with Gemini Vision fallback for scanned docs), concepts are enriched via Gemini, and embeddings are stored in Qdrant. Aurora (Open WebUI) queries the knowledge base via RAG filter.

Location

• Host: recon LXC (CT 130 on data node, 192.168.1.240)
• IP: 192.168.1.130 / 100.64.0.24 (Tailscale)
• Install: /opt/recon/
• User: zvx
• Service: recon.service (systemd, Type=simple, Restart=on-failure)
• Dashboard: https://recon.echo6.co (internal: http://100.64.0.24:8420)
• Health: https://recon.echo6.co/api/health

Stack

Component	Technology	Location
Pipeline + CLI	Python 3.12, argparse	/opt/recon/recon.py
Dashboard + API	Flask	/opt/recon/lib/api.py (port 8420)
Status DB	SQLite (WAL mode)	/opt/recon/data/recon.db
Vector DB	Qdrant	cortex:6333 (Docker)
Embeddings	TEI (bge-m3, 1024-dim)	cortex:8090 (Docker)
Enrichment	Gemini 2.0 Flash	Google API (4 keys)
Vision OCR	Gemini 2.0 Flash	Google API (shared keys)
Text extraction	PyPDF2, poppler-utils, Tesseract	Local
PDF source	NFS	pi-nas:/export/library → /mnt/library
File server	nginx	localhost:8888 → files.echo6.co

Pipeline Stages

All stages run concurrently as daemon threads in the service:

1. Scanner (hourly) — walks /mnt/library, catalogues new PDFs, queues them
2. Extract (4 workers) — PyPDF2 → pdftotext → Tesseract → Gemini Vision per page
3. Enrich (16 workers, 4 API keys) — Gemini extracts structured concepts from text windows
4. Embed (4 workers) — TEI generates vectors, upserted to Qdrant

Extraction Chain

Per page, in order. Each method only runs if the previous returned <50 chars:

1. PyPDF2 — fast, free, works on text-based PDFs
2. pdftotext (poppler) — handles some PDFs PyPDF2 misses
3. Tesseract OCR — renders page to image, runs local OCR
4. Gemini Vision — renders page to PNG, sends to Gemini 2.0 Flash vision API

Method tracking saved in data/text/{hash}/meta.json as ocr_methods dict.

Scale (as of Feb 2026)

• ~10,162 documents in pipeline
• ~95,000+ vectors in Qdrant (HNSW index, <10ms search latency)
• Collection: recon_knowledge
• ~13,239 PDFs catalogued from NFS library

Resilience

• Enricher: Exponential backoff (5s→80s) for transient errors (429, 500, 503). Window-level failure isolation — partial enrichment beats zero.
• Extractor: Per-page timeout (30s), per-document timeout (1800s). Partial extractions saved.
• Embedder: Skip-on-failure per concept, batch processing.
• Service: Restart=on-failure, RestartSec=30, MemoryMax=3G.

Configuration

Config file: /opt/recon/config.yaml

Key sections:

• processing.extract_workers (4), enrich_workers (16), embed_workers (4)
• processing.extract_timeout (1800s), page_timeout (30s)
• processing.enrich_max_retries (5), enrich_base_delay (5.0)
• gemini.model (gemini-2.0-flash), gemini.response_mime_type (application/json)
• service.scan_interval (3600), stage_poll_interval (30)

API keys: /opt/recon/.env — GEMINI_KEY_1 through GEMINI_KEY_4

API Endpoints

Endpoint	Method	Purpose
/	GET	Dashboard HTML
/api/knowledge-stats	GET	Full pipeline stats, per-source breakdown
/api/health	GET	Health check (Qdrant, TEI, NFS, Gemini, pipeline)
/api/search	GET	Vector search (?q=query&limit=5)
/api/upload	POST	Upload PDF (multipart: file + category)
/api/upload/<hash>/status	GET	Upload status tracking
/api/upload/categories	GET	Available upload categories
/api/ingest	POST	Ingest intel JSON data
/api/peertube/channels	GET	List all channels from channel-map.json with video counts from PeerTube DB
/api/peertube/channels/stats	GET	Channel count, total videos, downloader status
/api/peertube/channels/add	POST	Add channel: resolve YT URL, create PeerTube channel, update JSON
/api/peertube/channels/<name>	DELETE	Remove channel from JSON and optionally from PeerTube

Backups

• Destination: root@100.64.0.1:/opt/backups/recon/
• Full sync: every 6 hours (concepts, text, DB, config)
• DB snapshot: every 2 hours
• Recovery: restore from Contabo → recon rebuild (reconstructs Qdrant from concept JSONs)
• Critical data: data/concepts/ — Gemini extraction work, costs money to regenerate

Key Files

/opt/recon/
├── recon.py              # CLI entry point + service command
├── config.yaml           # Configuration
├── .env                  # Gemini API keys
├── PROJECT-BIBLE.md      # Full documentation
├── lib/
│   ├── api.py            # Flask dashboard + API
│   ├── extractor.py      # PDF → text (4-method chain)
│   ├── enricher.py       # Text → concepts (Gemini)
│   ├── embedder.py       # Concepts → vectors (TEI/Qdrant)
│   ├── status.py         # SQLite DB (WAL, thread-safe)
│   └── utils.py          # Config, hashing, logging
├── scripts/
│   ├── backup.sh         # Backup to Contabo
│   ├── validate.py       # Pipeline consistency checker
│   └── rebuild_qdrant.py # Nuclear Qdrant rebuild
└── data/
    ├── recon.db           # SQLite status DB
    ├── concepts/{hash}/   # Enriched concept JSONs
    └── text/{hash}/       # Extracted page text

---

Last updated: 2026-02-16 — Initial creation