matt/refactored-recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/refactored-recon.git synced 2026-05-20 14:44:39 +02:00
Code Issues Projects Releases Packages Wiki Activity
refactored-recon/NAV-INTEGRATION-v4.md

15 KiB
Raw Blame History

NAV-INTEGRATION-v4.md — Echo6 Navi Module

Status: Active
Created: 2026-04-17
Updated: 2026-04-19
Author: Matt + Claude
Repo: forge.echo6.co/matt/refactored-recon

System Context

  • RECON — Backend of everything. Acquires, processes, enriches, embeds, files. Manages all data including Navi datasets.
  • Aurora — Eyes, ears, and mouth. Speaks to humans via Open WebUI or mesh. Queries tools, synthesizes answers.
  • Navi — Navigation module. Routing, geocoding, tiles, mesh bridge, weather, trail/land overlays.
  • Kiwix — Offline internet. ZIM files via kiwix-serve. Aurora integration via tool-callable search.
  • Library — files.echo6.co. PDFs across 21 domains.

Infrastructure

  • VM 130 — 192.168.1.130 on data node (192.168.1.240)
  • OS: Ubuntu 24.04 LTS (migrated from LXC to VM for Docker support)
  • RAM: 16 GB
  • vCPU: 4
  • Boot disk: 80-100 GB
  • Docker: Yes
  • User: zvx
  • Mounts:
    • /mnt/library/ — ~67 GB library (virtiofs from data host /mnt/data/library/)
    • /mnt/nav/ — ~200 GB nav data (virtiofs from data host /mnt/data/nav/)
    • /mnt/kiwix/ — Kiwix ZIM storage (virtiofs from data host /mnt/data/kiwix/)
    • /mnt/nas/ — pi-nas NFS share (192.168.1.245)

Completed Work

Phase H1a: Valhalla Routing Engine ✅

  • Docker: ghcr.io/valhalla/valhalla-scripted:latest v3.6.3
  • Port: 8002
  • Data: Idaho PBF, 540 tiles built
  • Validated: Buhl → Boise (127.1 mi, 144 min, auto + pedestrian costing)
  • Branch: feature/navi on forge.echo6.co/matt/recon

Phase H1b: Photon Geocoding ✅

  • Service: systemd unit, Java jar, Xmx10g
  • Port: 2322
  • Data: Full planet import, 281.4M documents, 85 GB index
  • Validated: Forward + reverse geocoding, worldwide coverage

Phase H2: Aurora Nav Tools ✅

  • Files:
    • /opt/recon/lib/nav_tools.py — route(), reverse_geocode()
    • /opt/recon/lib/aurora_nav_tool.py — Open WebUI tool wrapper
  • Registered: Navigation tool visible in Open WebUI
  • Tested: 5/5 tests passing

Phase H2b: Semantic Query Router ✅

  • Files:
    • /opt/recon/lib/query_router.py — standalone router (38 example queries, 4 routes)
    • recon_rag_tool.py v4.2.0 on cortex — router gate integrated into inlet()
  • Routes:
    • nav_route (0.735 confidence) → Valhalla directions, skip RAG
    • nav_reverse_geocode (0.871) → Photon reverse, skip RAG
    • direct_answer (0.877) → pass through, skip RAG
    • rag_search (0.751) → full RAG pipeline
  • Safety: TEI down → falls to RAG. Nav fails → falls to RAG.
  • Expandable: Adding new routes = embed 10-20 examples, compute centroid, add to dict. No retraining.

Pi-nas Country Index ✅

  • File: /export/data/nav/photon-country-index.txt (also as .md)
  • Coverage: 282.5M lines indexed by country code
  • US records: 52.8M starting at line 122,520,379
  • Use: Enables regional Photon builds for Pi deployment

Active Services on VM 130

Service Type Port Status
recon.service systemd (native) 8420 (dashboard), 8888 (files) Running
Valhalla Docker 8002 Running (540 tiles, Idaho)
Photon systemd (Java) 2322 Running (281M docs, 85 GB)

Git Repos

Repo What Language Branch
matt/recon Backend, pipeline, nav_tools, mesh_bridge, router Python feature/navi
matt/refactored-recon Design docs, bibles, plans Markdown main
matt/navi Web map frontend (TO CREATE) JS/HTML/CSS —
matt/navi-mobile Ferrostar Android app (FUTURE) Kotlin —

Priority Tiers (Revised)

HIGH — Core Proposition (DONE)

  1. ✅ Aurora gives turn-by-turn directions
  2. ✅ Semantic router for intelligent tool selection
  3. Meshtastic waypoint delivery (BLOCKED — needs hardware decisions)

MEDIUM — Enhanced Capabilities (IN PROGRESS)

  1. Address book in RECON — pre-Photon geocoding for saved locations
  2. Netsyms Address DB — 160M USPS-validated addresses, SQLite
  3. Web frontend — navi.echo6.co (NEW REPO: matt/navi)
  4. Ferrostar mobile app — Android turn-by-turn pointed at Valhalla
  5. TomTom traffic integration — routing-level, not just overlay
  6. Self-hosted weather (Open-Meteo)
  7. Selectable offline region downloads for designated AO
  8. Trail and land ownership overlays (USFS/BLM/PAD-US)

LOW — Future Enrichment

  1. Wilderness data layers (foraging, water, fauna)

Next Phases (Ordered)

Phase M-AB: Address Book + Netsyms Download

Goal: Saved locations resolve instantly without geocoding. Netsyms provides USPS-validated address precision.

Address Book:

  • YAML or SQLite in /opt/recon/data/ or /opt/recon/config/
  • Checked BEFORE Photon in nav_tools geocoding chain
  • Feeds: Aurora web ("how do I get home"), mesh bridge ("nav home"), Navi frontend (starred locations)
  • Structure: key, name, lat, lon, address, aliases[]
  • Future expansion: contacts, callsigns, frequencies (RECON-managed rolodex)

Netsyms:

  • Download: https://dl.netsyms.net/gis/addresses/2025/AddressDatabase2025.zip (11 GB compressed, 35 GB uncompressed)
  • SHA256: 3deb85a37c6a4d027dd35fcbc1084e577b06a95be471042acc17fe21dedc3d8e
  • Format: SQLite, 160M addresses (US + Canada), USPS ZIP+4 validated
  • Schema: zipcode, number, street, street2, city, state, plus4, country, latitude, longitude, source
  • Source data: National Address Database + OpenAddresses.io + USPS ZIP+4
  • License: Public domain (facts cannot be copyrighted under US law)
  • Comparison: Photon (OSM crowd-sourced) vs Netsyms (government records). Test side-by-side.
  • Pi deployment: "lite" version at 6.7 GB compressed, no lat/lon, optimized for autocomplete

Photon vs Netsyms vs Address Book — geocoding chain:

  1. Address book → exact match on saved locations (instant, zero network)
  2. Netsyms SQLite → street address precision (local query, no service)
  3. Photon → place names, POIs, worldwide ("Boise Airport", "Sawtooth NF")

Phase M-WEB: Web Frontend (navi.echo6.co)

Goal: Google Maps-style web experience with self-hosted tiles, routing, search.

Repo: forge.echo6.co/matt/navi (new) Deploy: nginx on VM 130, port 8440, navi.echo6.co

Stack:

  • MapLibre GL JS — map rendering
  • PMTiles — vector tiles served by nginx with Range Requests
  • Valhalla API at :8002 — routing with polyline + maneuvers
  • Photon API at :2322 — search/geocode
  • Netsyms SQLite — address autocomplete (via small API or direct query)
  • TomTom traffic overlay — raster tiles via API key (visual only for now)
  • Address book locations — starred markers on map

MVP Features:

  • Search bar (Photon forward geocode)
  • Click-to-route (two points → Valhalla → draw polyline + maneuver list)
  • Mode selector (auto, pedestrian, bicycle)
  • GPS dot (browser Geolocation API)
  • TomTom traffic tile overlay
  • Responsive mobile layout

Future Features:

  • Layer switcher (street / topo / satellite)
  • Saved locations from address book
  • Turn-by-turn panel with voice (Web Speech API)
  • Offline PWA with Service Worker caching
  • USFS trail / BLM land / PAD-US overlays
  • Contour/hillshade overlay

Tile Data (needs building):

  • Basemap PMTiles from Planetiler or pmtiles extract from Protomaps daily builds
  • Start with Idaho regional extract, expand to CONUS
  • Contours from SRTM via phyghtmap (future)

Phase M-APP: Ferrostar Mobile App

Goal: Native Android turn-by-turn navigation app pointed at your Valhalla.

Repo: forge.echo6.co/matt/navi-mobile (future)

Stack:

  • Ferrostar SDK (BSD license, Rust core, Kotlin/Jetpack Compose UI)
  • Built-in Valhalla route provider: WellKnownRouteProvider.Valhalla("http://192.168.1.130:8002/route/v1", "auto")
  • MapLibre Native for map rendering
  • GPS snapping, off-route detection, automatic rerouting
  • Voice guidance via SpokenInstructionObserver
  • Your PMTiles for offline map tiles

Effort: Fork the Ferrostar demo app, swap three URLs (Valhalla, tile source, search). Half-day to functional, a few days to polish.

Phase M-TRAFFIC: TomTom → Valhalla Traffic Integration

Goal: Valhalla routes based on real-time traffic, not just static road speeds.

Architecture:

  1. Cron job (every 5 min while internet available) → poll TomTom Traffic Flow Segments API for AO
  2. Mapping service → translate TomTom segment IDs to Valhalla edge IDs (built once via valhalla_ways_to_edges, stored as lookup table)
  3. Traffic tile writer → pack speeds into Valhalla's binary traffic tile format → update traffic.tar
  4. Valhalla hot-reloads traffic data without restart

TomTom free tier: 2,500 requests/day — sufficient for single AO polled every 5 min during waking hours.

Key insight: Traffic-aware routing is a peacetime feature. Grid-down = no TomTom feed = Valhalla routes without traffic = fine (traffic patterns irrelevant without grid).

Effort: 2-3 CC sessions. Requires understanding Valhalla's internal edge ID system and binary traffic tile format.

Reference: Christian Beiwinkel's "Ultimate Guide to Traffic in Valhalla" and his "Valhalla Orbis Tools" for TomTom integration.

Phase H3: Meshtastic Mesh Bridge + Waypoints + Position Logging

BLOCKED on hardware decisions:

  1. Which physical Meshtastic node is the gateway?
  2. Gateway IP on LAN?
  3. Private nav channel PSK

Architecture (Option B — Aurora-mediated):

Mesh DM → mesh_bridge receives text
  → Injects sender GPS + context
  → Sends to Aurora (via Ollama API or Open WebUI API)
  → Semantic router classifies intent
  → Aurora calls appropriate tool (nav, RAG, weather, etc.)
  → Response formatted + compressed
  → Waypoints + text sent back over mesh DM

Fast-path bypass for n, next, cancel, sitrep — skip Aurora.

Position logging:

  • Subscribe to POSITION_APP packets
  • SQLite table: node_id, lat, lon, altitude, speed, heading, timestamp
  • Enables: "get me home" (earliest position), breadcrumb trails, team tracking

Waypoint delivery:

  • Sliding window (3 at a time, 6s pacing on MEDIUM_FAST)
  • Compressed names ≤30 chars ("L Main St 0.3")
  • Emoji icons per maneuver type
  • 24h TTL expiry
  • Private channel to avoid polluting public LongFast

Gateway config:

  • Role: CLIENT or ROUTER_CLIENT
  • TCP: Enabled :4403
  • Channel 0: Public LongFast
  • Channel 1: Private nav, PSK-protected, MEDIUM_FAST
  • GPS: Enabled

Data Sources Available

Geocoding

Source Records Size Coverage Best For
Photon (running) 281M 85 GB Planet Place names, POIs, worldwide
Netsyms (to download) 160M 35 GB US + Canada Street addresses, USPS precision
OpenAddresses.io 600M+ Varies Worldwide Future international expansion
Address book (to build) User-defined <1 MB Personal Saved locations, zero latency

Trail and Land Data (Phase M future)

Dataset Source License Size as PMTiles
USFS NFS Trails data.fs.usda.gov Public domain ~150-300 MB
USFS MVUM Roads data.fs.usda.gov Public domain ~250-450 MB
USFS MVUM Trails data.fs.usda.gov Public domain ~50-100 MB
PAD-US 4.1 usgs.gov Public domain ~400-800 MB
BLM SMA gbp-blm-egis.hub.arcgis.com Public domain ~80-150 MB
IDFG Hunt Units idfg.idaho.gov Free ~10 MB
IDL State Trust idl.idaho.gov Free ~20 MB
IDPR OHV/Snow Trails idpr-data-idaho.hub.arcgis.com Free ~10 MB

Conversion pipeline: shapefile → ogr2ogr (EPSG:4326) → tippecanoe → PMTiles → nginx Keep as display-only overlays. Do NOT merge into OSM PBF for routing.

Pi 5 Deployment Target

Everything built on the homelab should have a "Pi profile" — regional extracts, configurable scope, no 16+ GB RAM dependency.

Component Homelab Pi 5 (8 GB)
Valhalla CONUS (60 GB, Docker) Regional (2-5 GB, native or Docker)
Geocoding Photon planet (85 GB, JVM 10g) Netsyms SQLite (35 GB, zero overhead)
Place names Photon Lightweight OSM place extract (~500 MB)
Tiles CONUS PMTiles (15-20 GB) Regional PMTiles (2-5 GB)
LLM Qwen3 8B on cortex GPU 1-3B model on CPU (slow but functional)
Embeddings bge-m3 on cortex GPU Dense-only, 256-dim, int8
Meshtastic TCP to gateway USB serial to node
Weather Open-Meteo (50-100 GB) Subset or skip
Storage 1 TB SSD shared 1-2 TB NVMe

Semantic Router — Current Routes

Route Confidence Action Expandable
nav_route 0.735 Valhalla directions, skip RAG ✅
nav_reverse_geocode 0.871 Photon reverse, skip RAG ✅
direct_answer 0.877 Pass through, skip RAG ✅
rag_search 0.751 Full RAG pipeline ✅
kiwix_search — PLANNED: kiwix-serve search Add 10-20 examples
weather — PLANNED: Open-Meteo query Add 10-20 examples
mesh_sitrep — PLANNED: node positions summary Add 10-20 examples

Storage Budget (Shared 938 GB SSD)

Mount Current Planned Additions Total
/mnt/library/ 67 GB — 67 GB
/mnt/kiwix/ ~40 GB +110 GB (Wikipedia + others) ~150 GB
/mnt/nav/ ~111 GB (Valhalla 2 GB + Photon 85 GB + sources + PBF) +35 GB Netsyms, +20 GB PMTiles, +60 GB CONUS Valhalla ~226 GB
Other host data ~400 GB — ~400 GB
Total ~843 GB
Free ~95 GB

Action needed: Trim Photon to US-only to reclaim ~70 GB, bringing free to ~165 GB. Or accept planet coverage and manage tightly.

Risk Register

Risk Impact Mitigation
Valhalla CONUS tile build OOMs Expansion blocked Build on cortex, rsync to data node
Photon 85 GB crowds out Kiwix ZIMs Storage pressure Trim to US-only (~15 GB), reclaim ~70 GB
TomTom free tier rate limit Traffic routing degraded Single AO polling only, cache aggressively
Meshtastic gateway TCP flaky H3 degraded Reconnect loop with backoff
Ferrostar SDK breaking changes (pre-1.0) App maintenance Pin SDK version, update deliberately
Netsyms lacks place names/POIs "Boise Airport" fails Keep Photon as fallback for non-address queries
Address book file not synced to cortex OWUI can't use saved locations Inline in Valve JSON or add HTTP endpoint

Estimated Effort (Remaining)

Phase CC Sessions Blocked By
M-AB: Address book + Netsyms 1 Nothing
M-WEB: Web frontend MVP 2-3 PMTiles built
M-APP: Ferrostar mobile 1-2 PMTiles built
M-TRAFFIC: TomTom → Valhalla 2-3 M-WEB validated
H3: Mesh bridge 2-3 Meshtastic hardware
Trail/land overlays 1-2 tippecanoe installed
Weather (Open-Meteo) 1 Nothing
Photon trim to US-only 0.5 Nothing
Expand Valhalla to CONUS 0.5 Cortex RAM for build