matt/recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/recon.git synced 2026-06-10 00:44:37 +02:00
Code Issues Projects Releases Packages Wiki Activity
No description
103 commits 12 branches 1 tag 1.2 MiB
  • Python 83.3%
  • JavaScript 7.8%
  • HTML 6.4%
  • CSS 1.4%
  • Shell 1.1%
Find a file
malice 75664c7d02 recon: remove /api/traffic/flow handler (now served by navi-traffic, extraction #1) 
The /api/traffic/flow/<z>/<x>/<y>.png handler is dead code in recon. As of
extraction #1 of the recon<->Navi decoupling, this path is served by the
standalone navi-traffic service. Live request flow is now:

    Caddy (CT 101, navi.echo6.co @authed_api, forward_auth)
      -> nginx :8440 (location ^~ /api/traffic/ -> proxy_cache traffic_cache)
        -> navi-traffic gunicorn :8421 (services/navi_traffic)

Cutover verified live: authenticated browser fetch to
https://navi.echo6.co/api/traffic/flow/... returns 200 image/png with
X-Cache-Status MISS then HIT (120s cache), Server: gunicorn.

navi-backend (github.com/zvx-echo6/navi-backend):
  - dae54f3  Initial scaffold: navi-backend + navi-traffic
  - 311cb8f  nginx: use ^~ prefix on /api/traffic/ to beat .png regex catch-all

Caddy cutover (@authed_api upstream 8420 -> nginx 8440) applied on Utility
CT 101. Also drops the now-unused make_response flask import (no other uses
in lib/api.py). os and http_requests remain (used elsewhere).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 01:12:20 -06:00
config offroute: PostGIS entry points with 100m densification and land_status tagging 2026-05-09 03:28:58 +00:00
lib recon: remove /api/traffic/flow handler (now served by navi-traffic, extraction #1) 2026-05-22 01:12:20 -06:00
scripts Add Overture Maps POI enrichment layer for place details 2026-04-21 16:51:25 +00:00
static Replace wget/SingleFile/Playwright backends with Zimit 2026-04-19 14:06:23 +00:00
templates Add Nav-I API key management UI 2026-04-23 06:50:44 +00:00
.gitignore Extract _full_zim_cleanup helper, add SIGHUP + scrape_jobs cleanup 2026-04-19 02:28:49 +00:00
api.py Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
config.yaml Replace wget/SingleFile/Playwright backends with Zimit 2026-04-19 14:06:23 +00:00
enricher.py Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
migrate_paths.py Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
PROJECT-BIBLE.md Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
README.md Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
recon.py Add scraper Phase 2: smart crawl mode detection + browser fallback 2026-04-18 18:26:43 +00:00
requirements.txt Add /api/reverse/<lat>/<lon> localhost-sourced enrichment bundle 2026-05-20 05:33:45 +00:00
run-pipeline-now.sh Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00
sweep_gated.sh Initial commit: RECON codebase baseline 2026-04-14 14:57:23 +00:00

README.md

RECON -- Knowledge Extraction Pipeline

Extracts structured knowledge from PDFs and web content into a Qdrant vector database for RAG retrieval by Aurora.

Quick Start

# Activate
cd /opt/recon && source venv/bin/activate

# Scan library for new PDFs
recon scan

# Queue and process
recon queue
recon extract
recon enrich
recon embed

# Or run full pipeline
recon run

# Ingest a web page
recon ingest-url "https://example.com/article" --category "Category" --process

# Crawl an entire docs site
recon crawl "https://docs.example.com" --include /docs/ --category "Category" --process

# Upload a PDF
recon upload --file /path/to/document.pdf --category "Category"

# Search
recon search "water purification methods"

# Check status
recon status
recon failures

Dashboard

http://100.64.0.24:8420

Services

Service Location Purpose
RECON Dashboard recon:8420 Pipeline management + API
Qdrant cortex:6333 Vector database
TEI cortex:8090 Embeddings (1,711/sec)
Ollama cortex:11434 Chat + fallback embeddings
OpenWebUI cortex:8080 (ai.echo6.co) Aurora chat with RAG
File Server recon:8888 (files.echo6.co) PDF downloads

Key Paths

Path Contents
/opt/recon/ Application code
/opt/recon/data/concepts/ Gemini extractions (CRITICAL -- back these up)
/opt/recon/data/text/ Extracted text
/opt/recon/data/recon.db SQLite status DB
/mnt/library/ PDF library (NFS from pi-nas)

Backups

Automated every 6 hours to Contabo VPS via /opt/recon/scripts/backup.sh. Concept JSONs are the most valuable data ($130+ of Gemini API work). Qdrant is NOT backed up -- rebuilt from JSONs in ~10 minutes via recon rebuild.

Monitoring

# Pipeline status
recon status

# Tail logs
tail -f /opt/recon/logs/recon.log

# Pipeline run log
tail -f /opt/recon/pipeline.log

# Validate consistency
recon validate --deep

Full Documentation

See PROJECT-BIBLE.md for complete system documentation.