checkpoint: pre-audit working tree state — 4 untracked design docs

NAV-INTEGRATION-v4.md

@@ -0,0 +1,363 @@
+# NAV-INTEGRATION-v4.md — Echo6 Navi Module
+
+**Status:** Active  
+**Created:** 2026-04-17  
+**Updated:** 2026-04-19  
+**Author:** Matt + Claude  
+**Repo:** forge.echo6.co/matt/refactored-recon  
+
+---
+
+## System Context
+
+- **RECON** — Backend of everything. Acquires, processes, enriches, embeds, files. Manages all data including Navi datasets.
+- **Aurora** — Eyes, ears, and mouth. Speaks to humans via Open WebUI or mesh. Queries tools, synthesizes answers.
+- **Navi** — Navigation module. Routing, geocoding, tiles, mesh bridge, weather, trail/land overlays.
+- **Kiwix** — Offline internet. ZIM files via kiwix-serve. Aurora integration via tool-callable search.
+- **Library** — files.echo6.co. PDFs across 21 domains.
+
+---
+
+## Infrastructure
+
+- **VM 130** — 192.168.1.130 on data node (192.168.1.240)
+- **OS:** Ubuntu 24.04 LTS (migrated from LXC to VM for Docker support)
+- **RAM:** 16 GB
+- **vCPU:** 4
+- **Boot disk:** 80-100 GB
+- **Docker:** Yes
+- **User:** zvx
+- **Mounts:**
+  - /mnt/library/ — ~67 GB library (virtiofs from data host /mnt/data/library/)
+  - /mnt/nav/ — ~200 GB nav data (virtiofs from data host /mnt/data/nav/)
+  - /mnt/kiwix/ — Kiwix ZIM storage (virtiofs from data host /mnt/data/kiwix/)
+  - /mnt/nas/ — pi-nas NFS share (192.168.1.245)
+
+---
+
+## Completed Work
+
+### Phase H1a: Valhalla Routing Engine ✅
+- **Docker:** `ghcr.io/valhalla/valhalla-scripted:latest` v3.6.3
+- **Port:** 8002
+- **Data:** Idaho PBF, 540 tiles built
+- **Validated:** Buhl → Boise (127.1 mi, 144 min, auto + pedestrian costing)
+- **Branch:** feature/navi on forge.echo6.co/matt/recon
+
+### Phase H1b: Photon Geocoding ✅
+- **Service:** systemd unit, Java jar, Xmx10g
+- **Port:** 2322
+- **Data:** Full planet import, 281.4M documents, 85 GB index
+- **Validated:** Forward + reverse geocoding, worldwide coverage
+
+### Phase H2: Aurora Nav Tools ✅
+- **Files:**
+  - /opt/recon/lib/nav_tools.py — route(), reverse_geocode()
+  - /opt/recon/lib/aurora_nav_tool.py — Open WebUI tool wrapper
+- **Registered:** Navigation tool visible in Open WebUI
+- **Tested:** 5/5 tests passing
+
+### Phase H2b: Semantic Query Router ✅
+- **Files:**
+  - /opt/recon/lib/query_router.py — standalone router (38 example queries, 4 routes)
+  - recon_rag_tool.py v4.2.0 on cortex — router gate integrated into inlet()
+- **Routes:**
+  - nav_route (0.735 confidence) → Valhalla directions, skip RAG
+  - nav_reverse_geocode (0.871) → Photon reverse, skip RAG
+  - direct_answer (0.877) → pass through, skip RAG
+  - rag_search (0.751) → full RAG pipeline
+- **Safety:** TEI down → falls to RAG. Nav fails → falls to RAG.
+- **Expandable:** Adding new routes = embed 10-20 examples, compute centroid, add to dict. No retraining.
+
+### Pi-nas Country Index ✅
+- **File:** /export/data/nav/photon-country-index.txt (also as .md)
+- **Coverage:** 282.5M lines indexed by country code
+- **US records:** 52.8M starting at line 122,520,379
+- **Use:** Enables regional Photon builds for Pi deployment
+
+---
+
+## Active Services on VM 130
+
+| Service | Type | Port | Status |
+|---------|------|------|--------|
+| recon.service | systemd (native) | 8420 (dashboard), 8888 (files) | Running |
+| Valhalla | Docker | 8002 | Running (540 tiles, Idaho) |
+| Photon | systemd (Java) | 2322 | Running (281M docs, 85 GB) |
+
+---
+
+## Git Repos
+
+| Repo | What | Language | Branch |
+|------|------|----------|--------|
+| `matt/recon` | Backend, pipeline, nav_tools, mesh_bridge, router | Python | feature/navi |
+| `matt/refactored-recon` | Design docs, bibles, plans | Markdown | main |
+| `matt/navi` | Web map frontend (TO CREATE) | JS/HTML/CSS | — |
+| `matt/navi-mobile` | Ferrostar Android app (FUTURE) | Kotlin | — |
+
+---
+
+## Priority Tiers (Revised)
+
+### HIGH — Core Proposition (DONE)
+1. ✅ Aurora gives turn-by-turn directions
+2. ✅ Semantic router for intelligent tool selection
+3. Meshtastic waypoint delivery (BLOCKED — needs hardware decisions)
+
+### MEDIUM — Enhanced Capabilities (IN PROGRESS)
+4. Address book in RECON — pre-Photon geocoding for saved locations
+5. Netsyms Address DB — 160M USPS-validated addresses, SQLite
+6. Web frontend — navi.echo6.co (NEW REPO: matt/navi)
+7. Ferrostar mobile app — Android turn-by-turn pointed at Valhalla
+8. TomTom traffic integration — routing-level, not just overlay
+9. Self-hosted weather (Open-Meteo)
+10. Selectable offline region downloads for designated AO
+11. Trail and land ownership overlays (USFS/BLM/PAD-US)
+
+### LOW — Future Enrichment
+12. Wilderness data layers (foraging, water, fauna)
+
+---
+
+## Next Phases (Ordered)
+
+### Phase M-AB: Address Book + Netsyms Download
+
+**Goal:** Saved locations resolve instantly without geocoding. Netsyms provides USPS-validated address precision.
+
+**Address Book:**
+- YAML or SQLite in /opt/recon/data/ or /opt/recon/config/
+- Checked BEFORE Photon in nav_tools geocoding chain
+- Feeds: Aurora web ("how do I get home"), mesh bridge ("nav home"), Navi frontend (starred locations)
+- Structure: key, name, lat, lon, address, aliases[]
+- Future expansion: contacts, callsigns, frequencies (RECON-managed rolodex)
+
+**Netsyms:**
+- Download: https://dl.netsyms.net/gis/addresses/2025/AddressDatabase2025.zip (11 GB compressed, 35 GB uncompressed)
+- SHA256: 3deb85a37c6a4d027dd35fcbc1084e577b06a95be471042acc17fe21dedc3d8e
+- Format: SQLite, 160M addresses (US + Canada), USPS ZIP+4 validated
+- Schema: zipcode, number, street, street2, city, state, plus4, country, latitude, longitude, source
+- Source data: National Address Database + OpenAddresses.io + USPS ZIP+4
+- License: Public domain (facts cannot be copyrighted under US law)
+- Comparison: Photon (OSM crowd-sourced) vs Netsyms (government records). Test side-by-side.
+- Pi deployment: "lite" version at 6.7 GB compressed, no lat/lon, optimized for autocomplete
+
+**Photon vs Netsyms vs Address Book — geocoding chain:**
+1. Address book → exact match on saved locations (instant, zero network)
+2. Netsyms SQLite → street address precision (local query, no service)
+3. Photon → place names, POIs, worldwide ("Boise Airport", "Sawtooth NF")
+
+### Phase M-WEB: Web Frontend (navi.echo6.co)
+
+**Goal:** Google Maps-style web experience with self-hosted tiles, routing, search.
+
+**Repo:** forge.echo6.co/matt/navi (new)
+**Deploy:** nginx on VM 130, port 8440, navi.echo6.co
+
+**Stack:**
+- MapLibre GL JS — map rendering
+- PMTiles — vector tiles served by nginx with Range Requests
+- Valhalla API at :8002 — routing with polyline + maneuvers
+- Photon API at :2322 — search/geocode
+- Netsyms SQLite — address autocomplete (via small API or direct query)
+- TomTom traffic overlay — raster tiles via API key (visual only for now)
+- Address book locations — starred markers on map
+
+**MVP Features:**
+- Search bar (Photon forward geocode)
+- Click-to-route (two points → Valhalla → draw polyline + maneuver list)
+- Mode selector (auto, pedestrian, bicycle)
+- GPS dot (browser Geolocation API)
+- TomTom traffic tile overlay
+- Responsive mobile layout
+
+**Future Features:**
+- Layer switcher (street / topo / satellite)
+- Saved locations from address book
+- Turn-by-turn panel with voice (Web Speech API)
+- Offline PWA with Service Worker caching
+- USFS trail / BLM land / PAD-US overlays
+- Contour/hillshade overlay
+
+**Tile Data (needs building):**
+- Basemap PMTiles from Planetiler or `pmtiles extract` from Protomaps daily builds
+- Start with Idaho regional extract, expand to CONUS
+- Contours from SRTM via phyghtmap (future)
+
+### Phase M-APP: Ferrostar Mobile App
+
+**Goal:** Native Android turn-by-turn navigation app pointed at your Valhalla.
+
+**Repo:** forge.echo6.co/matt/navi-mobile (future)
+
+**Stack:**
+- Ferrostar SDK (BSD license, Rust core, Kotlin/Jetpack Compose UI)
+- Built-in Valhalla route provider: `WellKnownRouteProvider.Valhalla("http://192.168.1.130:8002/route/v1", "auto")`
+- MapLibre Native for map rendering
+- GPS snapping, off-route detection, automatic rerouting
+- Voice guidance via SpokenInstructionObserver
+- Your PMTiles for offline map tiles
+
+**Effort:** Fork the Ferrostar demo app, swap three URLs (Valhalla, tile source, search). Half-day to functional, a few days to polish.
+
+### Phase M-TRAFFIC: TomTom → Valhalla Traffic Integration
+
+**Goal:** Valhalla routes based on real-time traffic, not just static road speeds.
+
+**Architecture:**
+1. Cron job (every 5 min while internet available) → poll TomTom Traffic Flow Segments API for AO
+2. Mapping service → translate TomTom segment IDs to Valhalla edge IDs (built once via `valhalla_ways_to_edges`, stored as lookup table)
+3. Traffic tile writer → pack speeds into Valhalla's binary traffic tile format → update traffic.tar
+4. Valhalla hot-reloads traffic data without restart
+
+**TomTom free tier:** 2,500 requests/day — sufficient for single AO polled every 5 min during waking hours.
+
+**Key insight:** Traffic-aware routing is a peacetime feature. Grid-down = no TomTom feed = Valhalla routes without traffic = fine (traffic patterns irrelevant without grid).
+
+**Effort:** 2-3 CC sessions. Requires understanding Valhalla's internal edge ID system and binary traffic tile format.
+
+**Reference:** Christian Beiwinkel's "Ultimate Guide to Traffic in Valhalla" and his "Valhalla Orbis Tools" for TomTom integration.
+
+### Phase H3: Meshtastic Mesh Bridge + Waypoints + Position Logging
+
+**BLOCKED on hardware decisions:**
+1. Which physical Meshtastic node is the gateway?
+2. Gateway IP on LAN?
+3. Private nav channel PSK
+
+**Architecture (Option B — Aurora-mediated):**
+```
+Mesh DM → mesh_bridge receives text
+  → Injects sender GPS + context
+  → Sends to Aurora (via Ollama API or Open WebUI API)
+  → Semantic router classifies intent
+  → Aurora calls appropriate tool (nav, RAG, weather, etc.)
+  → Response formatted + compressed
+  → Waypoints + text sent back over mesh DM
+```
+
+Fast-path bypass for `n`, `next`, `cancel`, `sitrep` — skip Aurora.
+
+**Position logging:**
+- Subscribe to POSITION_APP packets
+- SQLite table: node_id, lat, lon, altitude, speed, heading, timestamp
+- Enables: "get me home" (earliest position), breadcrumb trails, team tracking
+
+**Waypoint delivery:**
+- Sliding window (3 at a time, 6s pacing on MEDIUM_FAST)
+- Compressed names ≤30 chars ("L Main St 0.3")
+- Emoji icons per maneuver type
+- 24h TTL expiry
+- Private channel to avoid polluting public LongFast
+
+**Gateway config:**
+- Role: CLIENT or ROUTER_CLIENT
+- TCP: Enabled :4403
+- Channel 0: Public LongFast
+- Channel 1: Private nav, PSK-protected, MEDIUM_FAST
+- GPS: Enabled
+
+---
+
+## Data Sources Available
+
+### Geocoding
+| Source | Records | Size | Coverage | Best For |
+|--------|---------|------|----------|----------|
+| Photon (running) | 281M | 85 GB | Planet | Place names, POIs, worldwide |
+| Netsyms (to download) | 160M | 35 GB | US + Canada | Street addresses, USPS precision |
+| OpenAddresses.io | 600M+ | Varies | Worldwide | Future international expansion |
+| Address book (to build) | User-defined | <1 MB | Personal | Saved locations, zero latency |
+
+### Trail and Land Data (Phase M future)
+| Dataset | Source | License | Size as PMTiles |
+|---------|--------|---------|----------------|
+| USFS NFS Trails | data.fs.usda.gov | Public domain | ~150-300 MB |
+| USFS MVUM Roads | data.fs.usda.gov | Public domain | ~250-450 MB |
+| USFS MVUM Trails | data.fs.usda.gov | Public domain | ~50-100 MB |
+| PAD-US 4.1 | usgs.gov | Public domain | ~400-800 MB |
+| BLM SMA | gbp-blm-egis.hub.arcgis.com | Public domain | ~80-150 MB |
+| IDFG Hunt Units | idfg.idaho.gov | Free | ~10 MB |
+| IDL State Trust | idl.idaho.gov | Free | ~20 MB |
+| IDPR OHV/Snow Trails | idpr-data-idaho.hub.arcgis.com | Free | ~10 MB |
+
+**Conversion pipeline:** shapefile → ogr2ogr (EPSG:4326) → tippecanoe → PMTiles → nginx
+**Keep as display-only overlays.** Do NOT merge into OSM PBF for routing.
+
+---
+
+## Pi 5 Deployment Target
+
+Everything built on the homelab should have a "Pi profile" — regional extracts, configurable scope, no 16+ GB RAM dependency.
+
+| Component | Homelab | Pi 5 (8 GB) |
+|-----------|---------|-------------|
+| Valhalla | CONUS (60 GB, Docker) | Regional (2-5 GB, native or Docker) |
+| Geocoding | Photon planet (85 GB, JVM 10g) | Netsyms SQLite (35 GB, zero overhead) |
+| Place names | Photon | Lightweight OSM place extract (~500 MB) |
+| Tiles | CONUS PMTiles (15-20 GB) | Regional PMTiles (2-5 GB) |
+| LLM | Qwen3 8B on cortex GPU | 1-3B model on CPU (slow but functional) |
+| Embeddings | bge-m3 on cortex GPU | Dense-only, 256-dim, int8 |
+| Meshtastic | TCP to gateway | USB serial to node |
+| Weather | Open-Meteo (50-100 GB) | Subset or skip |
+| Storage | 1 TB SSD shared | 1-2 TB NVMe |
+
+---
+
+## Semantic Router — Current Routes
+
+| Route | Confidence | Action | Expandable |
+|-------|-----------|--------|------------|
+| nav_route | 0.735 | Valhalla directions, skip RAG | ✅ |
+| nav_reverse_geocode | 0.871 | Photon reverse, skip RAG | ✅ |
+| direct_answer | 0.877 | Pass through, skip RAG | ✅ |
+| rag_search | 0.751 | Full RAG pipeline | ✅ |
+| kiwix_search | — | PLANNED: kiwix-serve search | Add 10-20 examples |
+| weather | — | PLANNED: Open-Meteo query | Add 10-20 examples |
+| mesh_sitrep | — | PLANNED: node positions summary | Add 10-20 examples |
+
+---
+
+## Storage Budget (Shared 938 GB SSD)
+
+| Mount | Current | Planned Additions | Total |
+|-------|---------|-------------------|-------|
+| /mnt/library/ | 67 GB | — | 67 GB |
+| /mnt/kiwix/ | ~40 GB | +110 GB (Wikipedia + others) | ~150 GB |
+| /mnt/nav/ | ~111 GB (Valhalla 2 GB + Photon 85 GB + sources + PBF) | +35 GB Netsyms, +20 GB PMTiles, +60 GB CONUS Valhalla | ~226 GB |
+| Other host data | ~400 GB | — | ~400 GB |
+| **Total** | | | **~843 GB** |
+| **Free** | | | **~95 GB** |
+
+**Action needed:** Trim Photon to US-only to reclaim ~70 GB, bringing free to ~165 GB. Or accept planet coverage and manage tightly.
+
+---
+
+## Risk Register
+
+| Risk | Impact | Mitigation |
+|------|--------|------------|
+| Valhalla CONUS tile build OOMs | Expansion blocked | Build on cortex, rsync to data node |
+| Photon 85 GB crowds out Kiwix ZIMs | Storage pressure | Trim to US-only (~15 GB), reclaim ~70 GB |
+| TomTom free tier rate limit | Traffic routing degraded | Single AO polling only, cache aggressively |
+| Meshtastic gateway TCP flaky | H3 degraded | Reconnect loop with backoff |
+| Ferrostar SDK breaking changes (pre-1.0) | App maintenance | Pin SDK version, update deliberately |
+| Netsyms lacks place names/POIs | "Boise Airport" fails | Keep Photon as fallback for non-address queries |
+| Address book file not synced to cortex | OWUI can't use saved locations | Inline in Valve JSON or add HTTP endpoint |
+
+---
+
+## Estimated Effort (Remaining)
+
+| Phase | CC Sessions | Blocked By |
+|-------|------------|------------|
+| M-AB: Address book + Netsyms | 1 | Nothing |
+| M-WEB: Web frontend MVP | 2-3 | PMTiles built |
+| M-APP: Ferrostar mobile | 1-2 | PMTiles built |
+| M-TRAFFIC: TomTom → Valhalla | 2-3 | M-WEB validated |
+| H3: Mesh bridge | 2-3 | Meshtastic hardware |
+| Trail/land overlays | 1-2 | tippecanoe installed |
+| Weather (Open-Meteo) | 1 | Nothing |
+| Photon trim to US-only | 0.5 | Nothing |
+| Expand Valhalla to CONUS | 0.5 | Cortex RAM for build |