matt/refactored-recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/refactored-recon.git synced 2026-05-20 14:44:39 +02:00
Code Issues Projects Releases Packages Wiki Activity
refactored-recon/NAV-INTEGRATION-v3.md

14 KiB
Raw Blame History

NAV-INTEGRATION-v3.md — Echo6 Navi Module

Status: Active
Created: 2026-04-17
Updated: 2026-04-18
Author: Matt + Claude
Repo: forge.echo6.co/matt/refactored-recon

System Context

  • RECON — Backend of everything. Acquires, processes, enriches, embeds, files. Manages all data including Navi datasets.
  • Aurora — Eyes, ears, and mouth. Speaks to humans via Open WebUI or mesh. Queries tools, synthesizes answers.
  • Navi — Navigation module. Routing, geocoding, tiles, mesh bridge, weather, trail/land overlays.
  • Kiwix — Offline internet. ZIM files via kiwix-serve.
  • Library — files.echo6.co. PDFs across 21 domains.

Infrastructure (Post VM Migration)

  • VM 130 — 192.168.1.130 on data node (192.168.1.240)
  • OS: Ubuntu 24.04 LTS
  • RAM: 16 GB
  • vCPU: 4
  • Boot disk: 80-100 GB
  • Docker: Yes (available post-migration)
  • User: zvx
  • Mounts:
    • /mnt/library/ — existing ~67 GB library (bind-mount from data host /mnt/data/library/)
    • /mnt/nav/ — NEW ~200 GB nav data (bind-mount from data host /mnt/data/nav/)

All services run on VM 130. No separate LXC for Navi.

Priority Tiers

HIGH — The Core Proposition

  1. Aurora gives turn-by-turn directions
  2. Meshtastic waypoint delivery native to the Meshtastic app
  3. Mesh bridge logs node position data (breadcrumb trails for "get me home")

MEDIUM — Enhanced Capabilities

  1. Self-hosted weather (Open-Meteo)
  2. Web frontend for directions (OSM tiles + topo/elevation)
  3. Selectable offline region downloads for designated AO
  4. Trail and land ownership overlays (USFS/BLM/PAD-US — free federal data, OnX alternative)

LOW — Future Enrichment

  1. Wilderness data layers (foraging, water, fauna)

Architecture

VM 130 (192.168.1.130)
├── RECON (existing)
│   ├── recon.service (7 daemon threads)
│   ├── /opt/recon/ (Python venv, SQLite, Flask)
│   ├── recon.echo6.co :8420
│   └── files.echo6.co :8888 (nginx)
│
├── Navi Services
│   ├── Valhalla (Docker) :8002 — routing
│   ├── Photon (native Java) :2322 — geocoding
│   ├── nginx :8440 — PMTiles + MapLibre frontend (Phase M)
│   └── Open-Meteo (Docker or .deb) :8080 (Phase M)
│
├── Aurora Integration
│   ├── /opt/recon/lib/nav_tools.py — route(), reverse_geocode(), weather()
│   └── Aurora pipe function tool registration
│
├── Mesh Integration
│   ├── /opt/recon/services/mesh_bridge.py — nav-bridge.service
│   ├── TCPInterface → Meshtastic gateway :4403
│   ├── Waypoint + text delivery
│   └── Position logging (breadcrumb trails per node)
│
└── External Dependencies (unchanged)
    ├── Qdrant — cortex 192.168.1.150:6333
    ├── bge-m3 dense — cortex:8090
    ├── bge-m3 sparse — cortex:8091
    └── Meshtastic gateway — [IP TBD]:4403

Storage Layout

/mnt/nav/
├── sources/
│   ├── idaho-latest.osm.pbf          # ~250 MB (start here)
│   └── us-latest.osm.pbf             # ~11 GB (expand later)
├── valhalla/                          # Docker volume
│   └── (tiles built automatically)   # ~2 GB Idaho, ~60 GB CONUS
├── photon/
│   └── photon_data/                   # ~95 GB planet, or smaller US-only
├── tiles/                             # Phase M
│   ├── basemap.pmtiles                # ~15-20 GB CONUS or extract
│   ├── contours.pmtiles               # ~5-10 GB
│   ├── usfs_trails.pmtiles            # ~150-300 MB
│   ├── usfs_mvum_roads.pmtiles        # ~250-450 MB
│   ├── padus.pmtiles                  # ~400-800 MB
│   ├── blm_sma.pmtiles                # ~80-150 MB
│   └── idfg_hunt_units.pmtiles        # ~10 MB
├── overlays/                          # Raw shapefiles/GeoJSON (source)
│   ├── S_USA.TrailNFS_Publish/
│   ├── S_USA.Road_MVUM/
│   ├── S_USA.Trail_MVUM/
│   ├── PADUS4_1/
│   ├── BLM_SMA/
│   ├── IDFG/
│   └── IDL/
├── weather/                           # Phase M
│   └── open-meteo-data/               # ~50-100 GB
├── frontend/                          # Phase M
│   ├── index.html
│   ├── style.json
│   └── sw.js
└── docker-compose.yml

Decision Log

# Decision Rationale
D1 Everything on VM 130 RECON is the backend of everything. No separate LXC.
D2 VM not LXC Docker needed for Valhalla. Migrated from CT 130 LXC.
D3 Valhalla via Docker C++ build from source is fragile. Docker is the community standard.
D4 Photon native (Java jar) Simple JRE + jar, no Docker needed. Systemd unit.
D5 Idaho-first, expand later Validate the stack on a 250 MB PBF before committing to 11 GB CONUS.
D6 US-only Photon 3-4 GB heap vs 8 GB for planet. Sufficient for prepper nav use case.
D7 Trail data as display overlay, not routed USFS trails don't merge cleanly into OSM PBF. OnX does it the same way.
D8 No private parcel ownership Irrelevant for nav use case. Saves $80K/yr Regrid license.
D9 Log mesh node positions Enables "how do I get back to base camp?" from breadcrumb history.
D10 Private mesh channel for nav Avoid polluting public LongFast. MEDIUM_FAST or SHORT_FAST preset.

HIGH PRIORITY PHASES

Phase H1: Routing + Geocoding Infrastructure

Goal: Valhalla and Photon running on VM 130, answering API calls.

H1a — Valhalla (Idaho-first):

  1. Create directory structure under /mnt/nav/
  2. Download idaho-latest.osm.pbf from Geofabrik (~250 MB)
  3. Deploy Valhalla via Docker (ghcr.io/valhalla/valhalla-scripted:latest)
  4. Build tiles from Idaho PBF (auto on first start)
  5. Validate: Buhl → Boise route with narrative maneuvers
  6. Test pedestrian + bicycle costing modes

H1b — Photon:

  1. Install JRE on VM 130
  2. Download Photon jar + planet dump (or US-only extract)
  3. Configure systemd unit for Photon on port 2322
  4. Validate forward geocoding: "Buhl Idaho"
  5. Validate reverse geocoding: 42.6, -114.46

H1c — Expand to CONUS (after validation):

  1. Download us-latest.osm.pbf (~11 GB)
  2. Build Valhalla tiles on cortex (needs 32 GB peak RAM), rsync to /mnt/nav/valhalla/
  3. Swap Valhalla to use CONUS tiles
  4. Re-validate routing

Validation: Buhl → Boise returns JSON with maneuvers including verbal_succinct_transition_instruction. Photon resolves "Twin Falls Idaho" to correct coordinates.

Phase H2: Aurora Nav Tools

Goal: Ask Aurora "How do I get from Buhl to Boise?" and get turn-by-turn.

Prerequisites: Phase H1a+H1b complete.

Tasks:

  1. Create /opt/recon/lib/nav_tools.py
    • route(origin, destination, mode) → geocode via Photon → route via Valhalla → formatted result
    • reverse_geocode(lat, lon) → Photon reverse lookup
  2. Register route tool in Aurora's Open WebUI Pipe Function
  3. Handle natural language variations
  4. Format response: summary + numbered maneuvers

Validation: In Open WebUI, ask Aurora: "How do I get from Buhl to Boise?" → get directions.

Phase H3: Meshtastic Mesh Bridge + Waypoints + Position Logging

Goal: Text nav Boise on a Meshtastic radio → receive waypoint pins + text directions. All node positions logged for breadcrumb trails.

Prerequisites: Phase H2 complete. Meshtastic gateway accessible via TCP.

Pre-build decisions needed:

  • Gateway node hardware + IP
  • Private channel PSK
  • Position logging schema (SQLite table on VM 130)

Components:

mesh_bridge.py daemon:

  • TCPInterface to gateway :4403
  • Command handlers: nav <dest>, n/next, cancel, where am i, home (route back to base/earliest position)
  • Waypoint delivery with sliding window (3 at a time, 6s pacing)
  • Route state per node with 24h TTL

Position logger:

  • Subscribe to POSITION_APP packets
  • Store: node_id, lat, lon, altitude, speed, heading, timestamp
  • SQLite table at /opt/recon/data/positions.db (or same recon.db)
  • Enables "get me home" by looking up earliest position or pre-registered base camp
  • Retention: configurable, default 30 days

Waypoint compression:

Maneuver Name (≤30 chars) Icon
Start START Buhl 📍
Left L Main St 0.3 ⬅️
Right R Oak Ave 1.2 ➡️
Straight S US-93 N 4.7 ⬆️
Destination DST Twin Falls 🏁

Meshtastic Gateway Config:

  • Role: CLIENT or ROUTER_CLIENT
  • TCP Server: Enabled :4403
  • Channel 0: Public LongFast (untouched)
  • Channel 1: Private nav, PSK-protected, MEDIUM_FAST
  • GPS: Enabled

Validation:

  1. From handheld, text nav Twin Falls on channel 1
  2. Receive text summary + 3 waypoint pins on Meshtastic app map
  3. Text n → next waypoints
  4. Text home → route back to earliest logged position
  5. Verify positions.db is accumulating node position data

MEDIUM PRIORITY PHASES

Start after HIGH phases are validated in the field.

Phase M1: Weather — Open-Meteo

Goal: Aurora answers "what's the weather?" offline. Mesh command: wx.

  • Deploy Open-Meteo (Docker or .deb) with GFS surface subset
  • ~50-100 GB storage, syncs while internet available
  • Add weather() tool to nav_tools.py
  • Add wx command to mesh_bridge

Phase M2: Web Frontend + Tile Serving

Goal: navi.echo6.co — self-hosted map UI with directions, search, layers.

  • PMTiles basemap via Planetiler or Protomaps extract
  • MapLibre GL JS frontend with Valhalla routing + Photon search
  • Layer switcher: street / topo / satellite(NAIP AO-only)
  • GPS tracking dot
  • Start with Headway, evaluate fork vs custom

Phase M3: Offline Region Downloads

Goal: RECON dashboard region picker → triggers tile + routing build.

  • v1: Geofabrik region tree dropdown (simple)
  • v2: Draw-a-bbox on map (complex)
  • Pipeline: osmium extract → planetiler → valhalla_build_tiles
  • Store in /mnt/nav/regions/{name}/

Phase M4: Trail and Land Overlays (OnX Alternative)

Goal: Public land boundaries, USFS trails, MVUM access on the map.

Data sources (all free, public domain):

Dataset Source Size (CONUS)
USFS NFS Trails data.fs.usda.gov ~150-300 MB as PMTiles
USFS MVUM Roads data.fs.usda.gov ~250-450 MB as PMTiles
USFS MVUM Trails data.fs.usda.gov ~50-100 MB as PMTiles
PAD-US 4.1 Fee usgs.gov ~400-800 MB as PMTiles
BLM SMA gbp-blm-egis.hub.arcgis.com ~80-150 MB as PMTiles
IDFG Hunt Units idfg.idaho.gov ~10 MB as PMTiles
IDL State Trust idl.idaho.gov ~20 MB as PMTiles
IDPR OHV/Snow Trails idpr-data-idaho.hub.arcgis.com ~10 MB as PMTiles

Conversion pipeline:

Download shapefile → ogr2ogr (reproject to EPSG:4326, select fields)
→ tippecanoe (GeoJSON → PMTiles) → serve via nginx

MapLibre styling:

  • BLM = yellow, USFS = green, NPS = purple, FWS = orange
  • State = blue, Private = transparent, Wilderness = dark green overlay
  • Motorized trails = solid red, non-motorized = dashed green
  • MVUM open roads = green, seasonal = orange, closed = red

Keep as display-only overlays. Do NOT merge into OSM PBF for routing.

Refresh schedule: PAD-US annually, BLM SMA quarterly, USFS per-unit, IDFG annually. Automate with cron.

LOW PRIORITY PHASES

Phase L1: Wilderness Data

Defer until HIGH and MEDIUM are stable. See NAV-INTEGRATION-v2.md for details.

Dependency Graph

Phase H1a (Valhalla Idaho)
    │
    ├── Phase H1b (Photon)
    │       │
    │       └── Phase H2 (Aurora nav tools)
    │               │
    │               ├── Phase H3 (Mesh bridge + waypoints + position logging)
    │               │
    │               ├── Phase M1 (Weather) — independent of H3
    │               │
    │               └── Phase L1 (Wilderness) — independent
    │
    ├── Phase H1c (Expand to CONUS) — independent of H2
    │
    ├── Phase M2 (Web frontend) — needs H1, enhanced by M4
    │
    ├── Phase M3 (Region downloads) — needs H1
    │
    └── Phase M4 (Trail/land overlays) — independent of routing, pairs with M2

H1a → H1b → H2 → H3 is the critical path.
M phases can proceed in any order after H1.

Estimated Effort

Phase CC Sessions Blocked By
H1a: Valhalla (Idaho) 1 /mnt/nav/ mount
H1b: Photon 0.5 H1a
H1c: Expand to CONUS 0.5 H1a validated
H2: Aurora nav tools 1 H1a + H1b
H3: Mesh bridge 2-3 H2 + Meshtastic hardware
M1: Weather 1 H1
M2: Web frontend 2-4 H1
M3: Region downloads 2-3 H1
M4: Trail/land overlays 1-2 tippecanoe installed
L1: Wilderness 2-3 H2

HIGH total: 5-6 CC sessions MEDIUM total: 6-10 CC sessions

Open Decisions (Resolve Before H3)

  1. Meshtastic gateway: Which physical node? What IP?
  2. Private nav channel PSK: Generate before H3
  3. Position logging location: Separate positions.db or table in recon.db?
  4. Photon scope: Planet dump (95 GB) or US-only extract?
  5. Base camp concept: Pre-registered waypoint, or earliest position in log?

Risk Register

Risk Impact Mitigation
Valhalla CONUS tile build OOMs H1c blocked Build on cortex, rsync to data node
Photon 95 GB download fails H1b delayed wget --continue, checksum verify
Gateway TCP flaky H3 degraded Reconnect loop with backoff
No GPS fix on field handheld H3 edge case Graceful error + manual coord input
Valhalla can't route backcountry trails Trail nav limited Trails are display-only anyway; use OSM foot paths where they exist
USFS trail data quality varies by forest Overlay gaps Supplement with user GPS tracks over time
VM 130 RAM pressure Service degradation Monitor, bump to 20 GB if needed (headroom exists)