mirror of
https://github.com/zvx-echo6/recon.git
synced 2026-06-10 00:44:37 +02:00
No description
- Python 83.3%
- JavaScript 7.8%
- HTML 6.4%
- CSS 1.4%
- Shell 1.1%
malice
14ad2cd34aPer-tag HTTP wrapper over wiki_rewrite.rewrite_wiki_link so the (future) navi-places service can rewrite OSM wiki tags to local Kiwix URLs over HTTP instead of importing recon's wiki_rewrite module (which talks to Kiwix on localhost:8430 and the wiki_cache table in /opt/recon/data/place_cache.db). Companion to PR #8 (/api/wiki-enrich) — Matt picked option B (HTTP-couple the Kiwix offline-wiki rewriting too, since it matters in prod). GET /api/wiki-rewrite?tag=<wikipedia|wikidata|wikivoyage|appropedia>&value=<raw> -> 200 {url, status} where status is "local" | "public" | "original" -> 400 on missing value or unknown tag -> no 404 (unclassifiable value echoes back with status "original", mirroring rewrite_wiki_link) Public (no auth), like /api/place/* and /api/wiki-enrich. Changes (additive only): - lib/wiki_rewrite_api.py: new wiki_rewrite_bp blueprint. Thin route directly over the existing rewrite_wiki_link(tag, value) — no extraction needed (it's already a clean standalone function, unlike wiki-enrich's lookup). - lib/api.py: register the blueprint (one block). - lib/wiki_rewrite_api_test.py: 5 tests (local Kiwix hit, public fallback, unclassifiable -> original, missing value -> 400, unknown tag -> 400), stubbing check_kiwix_has_article (no Kiwix/DB), plain-assert + __main__ runner. Verified green against recon's venv (flask 3.1.2). Does NOT touch place_detail's in-process _enrich_wiki_links — that gets removed in a later PR once navi-places is live (same as PR #8). wiki_cache stays in recon's own place_cache.db post-cutover (harmless positive-cache duplication). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| config | ||
| lib | ||
| scripts | ||
| static | ||
| templates | ||
| .gitignore | ||
| api.py | ||
| config.yaml | ||
| enricher.py | ||
| migrate_paths.py | ||
| PROJECT-BIBLE.md | ||
| README.md | ||
| recon.py | ||
| requirements.txt | ||
| run-pipeline-now.sh | ||
| sweep_gated.sh | ||
RECON -- Knowledge Extraction Pipeline
Extracts structured knowledge from PDFs and web content into a Qdrant vector database for RAG retrieval by Aurora.
Quick Start
# Activate
cd /opt/recon && source venv/bin/activate
# Scan library for new PDFs
recon scan
# Queue and process
recon queue
recon extract
recon enrich
recon embed
# Or run full pipeline
recon run
# Ingest a web page
recon ingest-url "https://example.com/article" --category "Category" --process
# Crawl an entire docs site
recon crawl "https://docs.example.com" --include /docs/ --category "Category" --process
# Upload a PDF
recon upload --file /path/to/document.pdf --category "Category"
# Search
recon search "water purification methods"
# Check status
recon status
recon failures
Dashboard
Services
| Service | Location | Purpose |
|---|---|---|
| RECON Dashboard | recon:8420 | Pipeline management + API |
| Qdrant | cortex:6333 | Vector database |
| TEI | cortex:8090 | Embeddings (1,711/sec) |
| Ollama | cortex:11434 | Chat + fallback embeddings |
| OpenWebUI | cortex:8080 (ai.echo6.co) | Aurora chat with RAG |
| File Server | recon:8888 (files.echo6.co) | PDF downloads |
Key Paths
| Path | Contents |
|---|---|
| /opt/recon/ | Application code |
| /opt/recon/data/concepts/ | Gemini extractions (CRITICAL -- back these up) |
| /opt/recon/data/text/ | Extracted text |
| /opt/recon/data/recon.db | SQLite status DB |
| /mnt/library/ | PDF library (NFS from pi-nas) |
Backups
Automated every 6 hours to Contabo VPS via /opt/recon/scripts/backup.sh.
Concept JSONs are the most valuable data ($130+ of Gemini API work).
Qdrant is NOT backed up -- rebuilt from JSONs in ~10 minutes via recon rebuild.
Monitoring
# Pipeline status
recon status
# Tail logs
tail -f /opt/recon/logs/recon.log
# Pipeline run log
tail -f /opt/recon/pipeline.log
# Validate consistency
recon validate --deep
Full Documentation
See PROJECT-BIBLE.md for complete system documentation.