matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-07-12T00:00:23+00:00

Browse source 
Files changed: ", c.get(k))\nPY\n\\\"\n\"" engine/config.yaml engine/lib/lint.py engine/lib/vocab_gen.py engine/lint-report.md engine/sweep.sh vault/.obsidian/workspace.json vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md vault/archive/projects/vaultwarden-plan.md vault/docs/matrix/matrix_host.md vault/docs/matrix/synapse.md vault/docs/services/services.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/dns.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/expose-service-contabo.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md
This commit is contained in:
echo6-autocommit 2026-07-12 00:00:23 +00:00
commit cce29c595d
37 changed files with 336 additions and 262 deletions
Download patch file Download diff file Expand all files Collapse all files

18
vault/docs/software/recon.md
View file

@ -10,7 +10,7 @@ related:
- [[services]]
- [[usenet]]
- [[caddy]]
updated: 2026-06-18
updated: 2026-07-11
---
# RECON — Knowledge Extraction Pipeline
@ -40,8 +40,8 @@ RECON extracts knowledge from PDFs and web content into a searchable vector data
| Vector DB | Qdrant | cortex:6333 (Docker) |
| Embeddings | TEI (bge-m3, 1024-dim) | cortex:8090 (Docker) |
| Sparse embeddings | recon-sparse (bge-m3 SPLADE) | cortex:8091 (systemd) |
| Enrichment | Gemini `gemini-2.5-flash-lite` (enforced) | Google API (4 keys) |
| Vision Ocr | Gemini 2.5 Flash Lite | Google API (shared keys) |
| Enrichment | Gemini `gemini-3.1-flash-lite` (enforced) | Google API (4 keys) |
| Vision Ocr | gemini-3.1-flash-lite | Google API (shared keys) |
| Text extraction | PyPDF2, poppler-utils, Tesseract | Local |
| PDF source | NFS | pi-nas:/export/library → /mnt/library |
| File server | nginx | localhost:8888 → files.echo6.co |
@ -62,7 +62,7 @@ Per page, in order. Each method only runs if the previous returned <50 chars:
1. **PyPDF2** — fast, free, works on text-based PDFs
2. **pdftotext** (poppler) — handles some PDFs PyPDF2 misses
3. **Tesseract OCR** — renders page to image, runs local OCR
4. **Gemini Vision** — renders page to PNG, sends to Gemini 2.5 Flash Lite vision API
4. **Gemini Vision** — renders page to PNG, sends to gemini-3.1-flash-lite vision API
Method tracking saved in `data/text/{hash}/meta.json` as `ocr_methods` dict.
@ -88,7 +88,7 @@ Key sections:
- `processing.extract_workers` (4), `enrich_workers` (16), `embed_workers` (4)
- `processing.extract_timeout` (1800s), `page_timeout` (30s)
- `processing.enrich_max_retries` (5), `enrich_base_delay` (5.0)
- `gemini.model` (gemini-2.5-flash-lite), `gemini.response_mime_type` (application/json)
- `gemini.model` (gemini-3.1-flash-lite), `gemini.response_mime_type` (application/json)
- `service.scan_interval` (3600), `stage_poll_interval` (30)
**API keys:** `/opt/recon/.env` — GEMINI_KEY_1 through GEMINI_KEY_4
@ -112,10 +112,10 @@ Key sections:
## Backups
- **Destination:** `root@100.64.0.1:/opt/backups/recon/`
- **Destination:** `root@100.64.0.40:/opt/backups/recon/` (edge1 — retains the storage/public IP post-rebuild)
- **Full sync:** every 6 hours (concepts, text, DB, config)
- **DB snapshot:** every 2 hours
- **Recovery:** restore from Contabo → `recon rebuild` (reconstructs Qdrant from concept JSONs)
- **Recovery:** restore from edge1 → `recon rebuild` (reconstructs Qdrant from concept JSONs)
- **Critical data:** `data/concepts/` — Gemini extraction work, costs money to regenerate
## Key Files
@ -134,7 +134,7 @@ Key sections:
│ ├── status.py # SQLite DB (WAL, thread-safe)
│ └── utils.py # Config, hashing, logging
├── scripts/
│ ├── backup.sh # Backup to Contabo
│ ├── backup.sh # Backup to edge1
│ ├── validate.py # Pipeline consistency checker
│ └── rebuild_qdrant.py # Nuclear Qdrant rebuild
└── data/
@ -145,4 +145,4 @@ Key sections:
---
*Last updated: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current deployment) — verified against live 2026-06-18*
*Last updated: 2026-07-11 — Gemini model refs updated to gemini-3.1-flash-lite (retired gemini-2.5-flash-lite); backup destination corrected Contabo (100.64.0.1, dead) → edge1 (100.64.0.40). Prior: 2026-06-18 — Updated: repo/branch, recon-sparse :8091, recon_knowledge_hybrid collection, Entrypoints; PROJECT-BIBLE.md dated 2026-02-16 (predates current deployment) — verified against live 2026-06-18*