echo6-docs/vault/docs/software/recon.md

title	type	tags	aliases	related	updated
RECON — Knowledge Extraction Pipeline	reference	recon		recon-operations ia-download-queue services usenet caddy	2026-07-11

RECON — Knowledge Extraction Pipeline

Overview

RECON extracts knowledge from PDFs and web content into a searchable vector database. PDFs are scanned from an NFS-mounted library, text is extracted (with Gemini Vision fallback for scanned docs), concepts are enriched via Gemini, and embeddings are stored in Qdrant. Aurora (Open WebUI) queries the knowledge base via RAG filter.

Location

• Host: recon-vm (VM 1130 on data node, 192.168.1.240) — migrated from CT 130 on 2026-04-19
• IP: 192.168.1.130 / 100.64.0.24 (Tailscale)
• Repo: github.com/zvx-echo6/recon (branch master; Forge mirror matt/recon)
• Install: /opt/recon/ (Python venv, in sync with origin)
• User: zvx
• Service: recon.service, recon-watchdog.service, kiwix.service (systemd)
• Entrypoints: recon.py service (dashboard/API :8420) + recon.py pipeline watch (watchdog, auto-ingests new PDFs in /mnt/library)
• Dashboard: https://recon.echo6.co (internal: http://100.64.0.24:8420)
• Health: https://recon.echo6.co/api/health

Stack

Component	Technology	Location
Pipeline + CLI	Python 3.12, argparse	/opt/recon/recon.py
Dashboard + API	Flask	/opt/recon/lib/api.py (port 8420)
Status DB	SQLite (WAL mode)	/opt/recon/data/recon.db
Vector DB	Qdrant	cortex:6333 (Docker)
Embeddings	TEI (bge-m3, 1024-dim)	cortex:8090 (Docker)
Sparse embeddings	recon-sparse (bge-m3 SPLADE)	cortex:8091 (systemd)
Enrichment	Gemini gemini-3.1-flash-lite (enforced)	Google API (4 keys)
Vision Ocr	gemini-3.1-flash-lite	Google API (shared keys)
Text extraction	PyPDF2, poppler-utils, Tesseract	Local
PDF source	NFS	pi-nas:/export/library → /mnt/library
File server	nginx	localhost:8888 → files.echo6.co

Pipeline Stages

All stages run concurrently as daemon threads in the service:

1. Scanner (hourly) — walks /mnt/library, catalogues new PDFs, queues them
2. Extract (4 workers) — PyPDF2 → pdftotext → Tesseract → Gemini Vision per page
3. Enrich (16 workers, 4 API keys) — Gemini extracts structured concepts from text windows
4. Embed (4 workers) — TEI generates vectors, upserted to Qdrant

Extraction Chain

Per page, in order. Each method only runs if the previous returned <50 chars:

1. PyPDF2 — fast, free, works on text-based PDFs
2. pdftotext (poppler) — handles some PDFs PyPDF2 misses
3. Tesseract OCR — renders page to image, runs local OCR
4. Gemini Vision — renders page to PNG, sends to gemini-3.1-flash-lite vision API

Method tracking saved in data/text/{hash}/meta.json as ocr_methods dict.

Scale (as of Feb 2026)

• ~10,162 documents in pipeline
• ~95,000+ vectors in Qdrant (HNSW index, <10ms search latency)
• Collection: recon_knowledge_hybrid
• ~13,239 PDFs catalogued from NFS library

Resilience

• Enricher: Exponential backoff (5s→80s) for transient errors (429, 500, 503). Window-level failure isolation — partial enrichment beats zero.
• Extractor: Per-page timeout (30s), per-document timeout (1800s). Partial extractions saved.
• Embedder: Skip-on-failure per concept, batch processing.
• Service: Restart=on-failure, RestartSec=30, MemoryMax=3G.

Configuration

Config file: /opt/recon/config.yaml

Key sections:

• processing.extract_workers (4), enrich_workers (16), embed_workers (4)
• processing.extract_timeout (1800s), page_timeout (30s)
• processing.enrich_max_retries (5), enrich_base_delay (5.0)
• gemini.model (gemini-3.1-flash-lite), gemini.response_mime_type (application/json)
• service.scan_interval (3600), stage_poll_interval (30)

API keys: /opt/recon/.env — GEMINI_KEY_1 through GEMINI_KEY_4

API Endpoints

Endpoint	Method	Purpose
/	GET	Dashboard HTML
/api/knowledge-stats	GET	Full pipeline stats, per-source breakdown
/api/health	GET	Health check (Qdrant, TEI, NFS, Gemini, pipeline)
/api/search	GET	Vector search (?q=query&limit=5)
/api/upload	POST	Upload PDF (multipart: file + category)
/api/upload/<hash>/status	GET	Upload status tracking
/api/upload/categories	GET	Available upload categories
/api/ingest	POST	Ingest intel JSON data
/api/peertube/channels	GET	List all channels from channel-map.json with video counts from PeerTube DB
/api/peertube/channels/stats	GET	Channel count, total videos, downloader status
/api/peertube/channels/add	POST	Add channel: resolve YT URL, create PeerTube channel, update JSON
/api/peertube/channels/<name>	DELETE	Remove channel from JSON and optionally from PeerTube

Backups

• Destination: root@100.64.0.40:/opt/backups/recon/ (edge1 — retains the storage/public IP post-rebuild)
• Full sync: every 6 hours (concepts, text, DB, config)
• DB snapshot: every 2 hours
• Recovery: restore from edge1 → recon rebuild (reconstructs Qdrant from concept JSONs)
• Critical data: data/concepts/ — Gemini extraction work, costs money to regenerate

Key Files

/opt/recon/
├── recon.py              # CLI entry point + service command
├── config.yaml           # Configuration
├── .env                  # Gemini API keys
├── PROJECT-BIBLE.md      # Full documentation
├── lib/
│   ├── api.py            # Flask dashboard + API
│   ├── extractor.py      # PDF → text (4-method chain)
│   ├── enricher.py       # Text → concepts (Gemini)
│   ├── embedder.py       # Concepts → vectors (TEI/Qdrant)
│   ├── status.py         # SQLite DB (WAL, thread-safe)
│   └── utils.py          # Config, hashing, logging
├── scripts/
│   ├── backup.sh         # Backup to edge1
│   ├── validate.py       # Pipeline consistency checker
│   └── rebuild_qdrant.py # Nuclear Qdrant rebuild
└── data/
    ├── recon.db           # SQLite status DB
    ├── concepts/{hash}/   # Enriched concept JSONs
    └── text/{hash}/       # Extracted page text

---

Last updated: 2026-07-11 — Gemini model refs updated to gemini-3.1-flash-lite (retired gemini-2.5-flash-lite); backup destination corrected Contabo (100.64.0.1, dead) → edge1 (100.64.0.40). Prior: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current deployment) — verified against live 2026-06-18