matt/refactored-recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/refactored-recon.git synced 2026-05-20 14:44:39 +02:00
Code Issues Projects Releases Packages Wiki Activity

checkpoint: pre-audit working tree state — 4 untracked design docs

Browse source
This commit is contained in:
Matt 2026-04-27 02:08:28 +00:00
commit 3b5c24c7e7
4 changed files with 1181 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

399
NAV-INTEGRATION-v3.md Normal file
View file

@ -0,0 +1,399 @@
# NAV-INTEGRATION-v3.md — Echo6 Navi Module
**Status:** Active
**Created:** 2026-04-17
**Updated:** 2026-04-18
**Author:** Matt + Claude
**Repo:** forge.echo6.co/matt/refactored-recon
---
## System Context
- **RECON** — Backend of everything. Acquires, processes, enriches, embeds, files. Manages all data including Navi datasets.
- **Aurora** — Eyes, ears, and mouth. Speaks to humans via Open WebUI or mesh. Queries tools, synthesizes answers.
- **Navi** — Navigation module. Routing, geocoding, tiles, mesh bridge, weather, trail/land overlays.
- **Kiwix** — Offline internet. ZIM files via kiwix-serve.
- **Library** — files.echo6.co. PDFs across 21 domains.
---
## Infrastructure (Post VM Migration)
- **VM 130** — 192.168.1.130 on data node (192.168.1.240)
- **OS:** Ubuntu 24.04 LTS
- **RAM:** 16 GB
- **vCPU:** 4
- **Boot disk:** 80-100 GB
- **Docker:** Yes (available post-migration)
- **User:** zvx
- **Mounts:**
- /mnt/library/ — existing ~67 GB library (bind-mount from data host /mnt/data/library/)
- /mnt/nav/ — NEW ~200 GB nav data (bind-mount from data host /mnt/data/nav/)
All services run on VM 130. No separate LXC for Navi.
---
## Priority Tiers
### HIGH — The Core Proposition
1. Aurora gives turn-by-turn directions
2. Meshtastic waypoint delivery native to the Meshtastic app
3. Mesh bridge logs node position data (breadcrumb trails for "get me home")
### MEDIUM — Enhanced Capabilities
4. Self-hosted weather (Open-Meteo)
5. Web frontend for directions (OSM tiles + topo/elevation)
6. Selectable offline region downloads for designated AO
7. Trail and land ownership overlays (USFS/BLM/PAD-US — free federal data, OnX alternative)
### LOW — Future Enrichment
8. Wilderness data layers (foraging, water, fauna)
---
## Architecture
```
VM 130 (192.168.1.130)
├── RECON (existing)
│ ├── recon.service (7 daemon threads)
│ ├── /opt/recon/ (Python venv, SQLite, Flask)
│ ├── recon.echo6.co :8420
│ └── files.echo6.co :8888 (nginx)
│
├── Navi Services
│ ├── Valhalla (Docker) :8002 — routing
│ ├── Photon (native Java) :2322 — geocoding
│ ├── nginx :8440 — PMTiles + MapLibre frontend (Phase M)
│ └── Open-Meteo (Docker or .deb) :8080 (Phase M)
│
├── Aurora Integration
│ ├── /opt/recon/lib/nav_tools.py — route(), reverse_geocode(), weather()
│ └── Aurora pipe function tool registration
│
├── Mesh Integration
│ ├── /opt/recon/services/mesh_bridge.py — nav-bridge.service
│ ├── TCPInterface → Meshtastic gateway :4403
│ ├── Waypoint + text delivery
│ └── Position logging (breadcrumb trails per node)
│
└── External Dependencies (unchanged)
├── Qdrant — cortex 192.168.1.150:6333
├── bge-m3 dense — cortex:8090
├── bge-m3 sparse — cortex:8091
└── Meshtastic gateway — [IP TBD]:4403
```
---
## Storage Layout
```
/mnt/nav/
├── sources/
│ ├── idaho-latest.osm.pbf # ~250 MB (start here)
│ └── us-latest.osm.pbf # ~11 GB (expand later)
├── valhalla/ # Docker volume
│ └── (tiles built automatically) # ~2 GB Idaho, ~60 GB CONUS
├── photon/
│ └── photon_data/ # ~95 GB planet, or smaller US-only
├── tiles/ # Phase M
│ ├── basemap.pmtiles # ~15-20 GB CONUS or extract
│ ├── contours.pmtiles # ~5-10 GB
│ ├── usfs_trails.pmtiles # ~150-300 MB
│ ├── usfs_mvum_roads.pmtiles # ~250-450 MB
│ ├── padus.pmtiles # ~400-800 MB
│ ├── blm_sma.pmtiles # ~80-150 MB
│ └── idfg_hunt_units.pmtiles # ~10 MB
├── overlays/ # Raw shapefiles/GeoJSON (source)
│ ├── S_USA.TrailNFS_Publish/
│ ├── S_USA.Road_MVUM/
│ ├── S_USA.Trail_MVUM/
│ ├── PADUS4_1/
│ ├── BLM_SMA/
│ ├── IDFG/
│ └── IDL/
├── weather/ # Phase M
│ └── open-meteo-data/ # ~50-100 GB
├── frontend/ # Phase M
│ ├── index.html
│ ├── style.json
│ └── sw.js
└── docker-compose.yml
```
---
## Decision Log
| # | Decision | Rationale |
|---|----------|-----------|
| D1 | **Everything on VM 130** | RECON is the backend of everything. No separate LXC. |
| D2 | **VM not LXC** | Docker needed for Valhalla. Migrated from CT 130 LXC. |
| D3 | **Valhalla via Docker** | C++ build from source is fragile. Docker is the community standard. |
| D4 | **Photon native (Java jar)** | Simple JRE + jar, no Docker needed. Systemd unit. |
| D5 | **Idaho-first, expand later** | Validate the stack on a 250 MB PBF before committing to 11 GB CONUS. |
| D6 | **US-only Photon** | 3-4 GB heap vs 8 GB for planet. Sufficient for prepper nav use case. |
| D7 | **Trail data as display overlay, not routed** | USFS trails don't merge cleanly into OSM PBF. OnX does it the same way. |
| D8 | **No private parcel ownership** | Irrelevant for nav use case. Saves $80K/yr Regrid license. |
| D9 | **Log mesh node positions** | Enables "how do I get back to base camp?" from breadcrumb history. |
| D10 | **Private mesh channel for nav** | Avoid polluting public LongFast. MEDIUM_FAST or SHORT_FAST preset. |
---
## HIGH PRIORITY PHASES
---
### Phase H1: Routing + Geocoding Infrastructure
**Goal:** Valhalla and Photon running on VM 130, answering API calls.
**H1a — Valhalla (Idaho-first):**
1. Create directory structure under /mnt/nav/
2. Download idaho-latest.osm.pbf from Geofabrik (~250 MB)
3. Deploy Valhalla via Docker (ghcr.io/valhalla/valhalla-scripted:latest)
4. Build tiles from Idaho PBF (auto on first start)
5. Validate: Buhl → Boise route with narrative maneuvers
6. Test pedestrian + bicycle costing modes
**H1b — Photon:**
1. Install JRE on VM 130
2. Download Photon jar + planet dump (or US-only extract)
3. Configure systemd unit for Photon on port 2322
4. Validate forward geocoding: "Buhl Idaho"
5. Validate reverse geocoding: 42.6, -114.46
**H1c — Expand to CONUS (after validation):**
1. Download us-latest.osm.pbf (~11 GB)
2. Build Valhalla tiles on cortex (needs 32 GB peak RAM), rsync to /mnt/nav/valhalla/
3. Swap Valhalla to use CONUS tiles
4. Re-validate routing
**Validation:** Buhl → Boise returns JSON with maneuvers including verbal_succinct_transition_instruction. Photon resolves "Twin Falls Idaho" to correct coordinates.
---
### Phase H2: Aurora Nav Tools
**Goal:** Ask Aurora "How do I get from Buhl to Boise?" and get turn-by-turn.
**Prerequisites:** Phase H1a+H1b complete.
**Tasks:**
1. Create /opt/recon/lib/nav_tools.py
- route(origin, destination, mode) → geocode via Photon → route via Valhalla → formatted result
- reverse_geocode(lat, lon) → Photon reverse lookup
2. Register route tool in Aurora's Open WebUI Pipe Function
3. Handle natural language variations
4. Format response: summary + numbered maneuvers
**Validation:** In Open WebUI, ask Aurora: "How do I get from Buhl to Boise?" → get directions.
---
### Phase H3: Meshtastic Mesh Bridge + Waypoints + Position Logging
**Goal:** Text `nav Boise` on a Meshtastic radio → receive waypoint pins + text directions. All node positions logged for breadcrumb trails.
**Prerequisites:** Phase H2 complete. Meshtastic gateway accessible via TCP.
**Pre-build decisions needed:**
- Gateway node hardware + IP
- Private channel PSK
- Position logging schema (SQLite table on VM 130)
**Components:**
**mesh_bridge.py daemon:**
- TCPInterface to gateway :4403
- Command handlers: `nav <dest>`, `n`/`next`, `cancel`, `where am i`, `home` (route back to base/earliest position)
- Waypoint delivery with sliding window (3 at a time, 6s pacing)
- Route state per node with 24h TTL
**Position logger:**
- Subscribe to POSITION_APP packets
- Store: node_id, lat, lon, altitude, speed, heading, timestamp
- SQLite table at /opt/recon/data/positions.db (or same recon.db)
- Enables "get me home" by looking up earliest position or pre-registered base camp
- Retention: configurable, default 30 days
**Waypoint compression:**
| Maneuver | Name (≤30 chars) | Icon |
|----------|------------------|------|
| Start | `START Buhl` | 📍 |
| Left | `L Main St 0.3` | ⬅️ |
| Right | `R Oak Ave 1.2` | ➡️ |
| Straight | `S US-93 N 4.7` | ⬆️ |
| Destination | `DST Twin Falls` | 🏁 |
**Meshtastic Gateway Config:**
- Role: CLIENT or ROUTER_CLIENT
- TCP Server: Enabled :4403
- Channel 0: Public LongFast (untouched)
- Channel 1: Private nav, PSK-protected, MEDIUM_FAST
- GPS: Enabled
**Validation:**
1. From handheld, text `nav Twin Falls` on channel 1
2. Receive text summary + 3 waypoint pins on Meshtastic app map
3. Text `n` → next waypoints
4. Text `home` → route back to earliest logged position
5. Verify positions.db is accumulating node position data
---
## MEDIUM PRIORITY PHASES
Start after HIGH phases are validated in the field.
---
### Phase M1: Weather — Open-Meteo
**Goal:** Aurora answers "what's the weather?" offline. Mesh command: `wx`.
- Deploy Open-Meteo (Docker or .deb) with GFS surface subset
- ~50-100 GB storage, syncs while internet available
- Add weather() tool to nav_tools.py
- Add `wx` command to mesh_bridge
---
### Phase M2: Web Frontend + Tile Serving
**Goal:** navi.echo6.co — self-hosted map UI with directions, search, layers.
- PMTiles basemap via Planetiler or Protomaps extract
- MapLibre GL JS frontend with Valhalla routing + Photon search
- Layer switcher: street / topo / satellite(NAIP AO-only)
- GPS tracking dot
- Start with Headway, evaluate fork vs custom
---
### Phase M3: Offline Region Downloads
**Goal:** RECON dashboard region picker → triggers tile + routing build.
- v1: Geofabrik region tree dropdown (simple)
- v2: Draw-a-bbox on map (complex)
- Pipeline: osmium extract → planetiler → valhalla_build_tiles
- Store in /mnt/nav/regions/{name}/
---
### Phase M4: Trail and Land Overlays (OnX Alternative)
**Goal:** Public land boundaries, USFS trails, MVUM access on the map.
**Data sources (all free, public domain):**
| Dataset | Source | Size (CONUS) |
|---------|--------|-------------|
| USFS NFS Trails | data.fs.usda.gov | ~150-300 MB as PMTiles |
| USFS MVUM Roads | data.fs.usda.gov | ~250-450 MB as PMTiles |
| USFS MVUM Trails | data.fs.usda.gov | ~50-100 MB as PMTiles |
| PAD-US 4.1 Fee | usgs.gov | ~400-800 MB as PMTiles |
| BLM SMA | gbp-blm-egis.hub.arcgis.com | ~80-150 MB as PMTiles |
| IDFG Hunt Units | idfg.idaho.gov | ~10 MB as PMTiles |
| IDL State Trust | idl.idaho.gov | ~20 MB as PMTiles |
| IDPR OHV/Snow Trails | idpr-data-idaho.hub.arcgis.com | ~10 MB as PMTiles |
**Conversion pipeline:**
```
Download shapefile → ogr2ogr (reproject to EPSG:4326, select fields)
→ tippecanoe (GeoJSON → PMTiles) → serve via nginx
```
**MapLibre styling:**
- BLM = yellow, USFS = green, NPS = purple, FWS = orange
- State = blue, Private = transparent, Wilderness = dark green overlay
- Motorized trails = solid red, non-motorized = dashed green
- MVUM open roads = green, seasonal = orange, closed = red
**Keep as display-only overlays.** Do NOT merge into OSM PBF for routing.
**Refresh schedule:** PAD-US annually, BLM SMA quarterly, USFS per-unit, IDFG annually. Automate with cron.
---
## LOW PRIORITY PHASES
### Phase L1: Wilderness Data
Defer until HIGH and MEDIUM are stable. See NAV-INTEGRATION-v2.md for details.
---
## Dependency Graph
```
Phase H1a (Valhalla Idaho)
│
├── Phase H1b (Photon)
│ │
│ └── Phase H2 (Aurora nav tools)
│ │
│ ├── Phase H3 (Mesh bridge + waypoints + position logging)
│ │
│ ├── Phase M1 (Weather) — independent of H3
│ │
│ └── Phase L1 (Wilderness) — independent
│
├── Phase H1c (Expand to CONUS) — independent of H2
│
├── Phase M2 (Web frontend) — needs H1, enhanced by M4
│
├── Phase M3 (Region downloads) — needs H1
│
└── Phase M4 (Trail/land overlays) — independent of routing, pairs with M2
H1a → H1b → H2 → H3 is the critical path.
M phases can proceed in any order after H1.
```
---
## Estimated Effort
| Phase | CC Sessions | Blocked By |
|-------|------------|------------|
| H1a: Valhalla (Idaho) | 1 | /mnt/nav/ mount |
| H1b: Photon | 0.5 | H1a |
| H1c: Expand to CONUS | 0.5 | H1a validated |
| H2: Aurora nav tools | 1 | H1a + H1b |
| H3: Mesh bridge | 2-3 | H2 + Meshtastic hardware |
| M1: Weather | 1 | H1 |
| M2: Web frontend | 2-4 | H1 |
| M3: Region downloads | 2-3 | H1 |
| M4: Trail/land overlays | 1-2 | tippecanoe installed |
| L1: Wilderness | 2-3 | H2 |
**HIGH total: 5-6 CC sessions**
**MEDIUM total: 6-10 CC sessions**
---
## Open Decisions (Resolve Before H3)
1. **Meshtastic gateway:** Which physical node? What IP?
2. **Private nav channel PSK:** Generate before H3
3. **Position logging location:** Separate positions.db or table in recon.db?
4. **Photon scope:** Planet dump (95 GB) or US-only extract?
5. **Base camp concept:** Pre-registered waypoint, or earliest position in log?
---
## Risk Register
| Risk | Impact | Mitigation |
|------|--------|------------|
| Valhalla CONUS tile build OOMs | H1c blocked | Build on cortex, rsync to data node |
| Photon 95 GB download fails | H1b delayed | wget --continue, checksum verify |
| Gateway TCP flaky | H3 degraded | Reconnect loop with backoff |
| No GPS fix on field handheld | H3 edge case | Graceful error + manual coord input |
| Valhalla can't route backcountry trails | Trail nav limited | Trails are display-only anyway; use OSM foot paths where they exist |
| USFS trail data quality varies by forest | Overlay gaps | Supplement with user GPS tracks over time |
| VM 130 RAM pressure | Service degradation | Monitor, bump to 20 GB if needed (headroom exists) |