diff --git a/docs/OFFROUTE-ARCHITECTURE.md b/docs/OFFROUTE-ARCHITECTURE.md new file mode 100644 index 0000000..a4b4fd8 --- /dev/null +++ b/docs/OFFROUTE-ARCHITECTURE.md @@ -0,0 +1,396 @@ +# OFFROUTE — Off-Network Effort-Based Routing Architecture + +**Status:** Draft +**Author:** Matt / Claude +**Date:** 2026-05-07 +**Canonical location:** `matt/refactored-recon` alongside PROJECT-BIBLE.md, NAV-INTEGRATION-v4.md + +--- + +## 1. Vision + +From any arbitrary point in the backcountry — no trails, no roads, no signal — route via effort cost and safety to the nearest trail, to a BLM/forest road, to a paved road, to home. Four segments, one continuous path, one GeoJSON response. + +The system serves two interfaces: +- **Navi frontend** (`navi.echo6.co`) — visual route overlay on the map +- **Aurora via Meshtastic** — text-based step-by-step directions for a lost person with no map display + +This capability does not exist in any open-source consumer product. CalTopo, OnX, Gaia GPS, AllTrails — all route on-network only. The military has Primordial Ground Guidance (closed-source ATAK plugin). We are building the open, self-hosted equivalent. + +--- + +## 2. The Routing Chain + +``` +[Lost person] + │ + ▼ + ┌──────────────────────────────────────────┐ + │ Segment 1: WILDERNESS → TRAIL │ + │ Engine: Raster cost-surface pathfinder │ + │ Cost: slope effort + vegetation + │ + │ water barriers + land ownership │ + │ Output: lat/lon waypoint sequence │ + └──────────────────────────────────────────┘ + │ snap to nearest trail entry point + ▼ + ┌──────────────────────────────────────────┐ + │ Segment 2: TRAIL → BLM/FOREST ROAD │ + │ Engine: Valhalla (pedestrian/MTB) │ + │ Cost: elevation-aware hike/bike profile │ + └──────────────────────────────────────────┘ + │ transition to road network + ▼ + ┌──────────────────────────────────────────┐ + │ Segment 3: BLM ROAD → PAVED ROAD │ + │ Engine: Valhalla (auto/motorcycle) │ + │ Cost: standard + surface preference │ + └──────────────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────┐ + │ Segment 4: PAVED ROAD → HOME │ + │ Engine: Valhalla (auto) │ + │ Cost: standard routing │ + └──────────────────────────────────────────┘ +``` + +Segments 2–4 already work today via Valhalla. **Segment 1 is the engineering gap.** + +--- + +## 3. Endpoint Design + +### `POST /api/offroute` + +**Request:** +```json +{ + "start": { "lat": 43.512, "lon": -114.823 }, + "destination": { "lat": 42.736, "lon": -114.514 }, + "mode": "foot", + "max_search_km": 15 +} +``` + +**Modes:** `foot` | `mtb` | `atv` + +**Response:** +```json +{ + "segments": [ + { + "type": "wilderness", + "geometry": { "type": "LineString", "coordinates": [...] }, + "distance_m": 4200, + "elevation_gain_m": 310, + "elevation_loss_m": 85, + "estimated_time_min": 72, + "surface": "cross-country", + "instructions": [ + { "bearing": 245, "distance_m": 320, "terrain": "sagebrush slope", "grade_pct": 8 }, + { "bearing": 260, "distance_m": 510, "terrain": "drainage crossing", "grade_pct": -12 } + ] + }, + { + "type": "trail", + "geometry": { "type": "LineString", "coordinates": [...] }, + "trail_name": "Pioneer Cabin Trail", + "distance_m": 6100, + "estimated_time_min": 85 + }, + { + "type": "road_unpaved", + "geometry": { "type": "LineString", "coordinates": [...] }, + "road_name": "FR-227", + "distance_m": 12400, + "estimated_time_min": 22 + }, + { + "type": "road_paved", + "geometry": { "type": "LineString", "coordinates": [...] }, + "distance_m": 34000, + "estimated_time_min": 28 + } + ], + "total_distance_m": 56700, + "total_time_min": 207, + "confidence": 0.82 +} +``` + +**Aurora tool integration:** Add `offroute` to `nav_tools.py` alongside existing `route()` and `reverse_geocode()`. The semantic query router gets a new embedding for "I'm lost, help me get home" / "navigate to nearest road" type queries. + +--- + +## 4. Pathfinder Architecture (Segment 1) + +### 4.1 No Pre-Rendered Slope Rasters + +The pathfinder does NOT need pre-computed slope layers, GDAL processing, or reprojection. It reads elevation directly: + +1. Routing request arrives with a start point and search radius +2. Determine which PMTiles z12 tiles cover the search area +3. Fetch + decode Terrarium tiles from `planet-dem.pmtiles` → numpy elevation arrays +4. Cache decoded arrays keyed by (z, x, y) — LRU, in-memory +5. A* / Dijkstra runs on the elevation grid, computing grade between neighbors on the fly +6. Cost function = `grade → effort model → multiply by land-cover friction → check barriers` + +### 4.2 Elevation Data Source + +**Primary:** `planet-dem.pmtiles` (658GB on pi-nas, served via nginx at `/tiles/planet-dem.pmtiles`) +- Mapterhorn, Copernicus GLO-30 source, Terrarium encoding (lossless WebP) +- z12 with 512px tiles = ~13–16m pixels at Idaho latitude +- 30m effective resolution (upsampled from source) +- Decode: `elevation = (R * 256 + G + B/256) - 32768` (metres, EGM2008) +- Precision: ~3.9mm quantization — far below source noise (~4m RMSE) + +**Upgrade path:** USGS 3DEP 1/3 arc-second (10m bare-earth DTM, CONUS). Same architecture, denser grid. Free download. Address when/if 30m proves insufficient for safety. + +**Regional GeoTIFFs** (203GB on NAS at `/mnt/nas/nav/contour-rebuild/dem/`): Keep as insurance until this pipeline is validated, then delete. + +### 4.3 Cost Function + +For each candidate move from cell A to cell B: + +```python +def travel_cost(elev_a, elev_b, distance_m, friction_ab): + grade = (elev_b - elev_a) / distance_m + + # Safety gate — impassable above threshold + slope_deg = math.degrees(math.atan(abs(grade))) + if slope_deg > MAX_SLOPE[mode]: # foot=40°, mtb=25°, atv=30° + return INF + + # Effort model (speed in km/h) + if mode == "foot": + # Tobler off-path hiking function + speed = 0.6 * 6.0 * math.exp(-3.5 * abs(grade + 0.05)) + elif mode == "mtb": + # Herzog wheeled-transport polynomial (crit_slope=8%) + speed = herzog_wheeled(grade, crit_slope=0.08, base_speed=12) + elif mode == "atv": + # Herzog with higher base speed and slope tolerance + speed = herzog_wheeled(grade, crit_slope=0.15, base_speed=25) + + # Time cost (seconds to traverse this cell) + time_s = (distance_m / 1000.0) / speed * 3600.0 + + # Multiply by land-cover friction + time_s *= friction_ab + + return time_s +``` + +**Tobler off-path:** `W = 0.6 × 6 × exp(-3.5 × |S + 0.05|)` km/h +Peak speed 3.6 km/h at ~-2.86° (slight downhill). The 0.6 multiplier is the off-trail penalty. + +**Herzog wheeled-transport:** sixth-degree polynomial fitted to wheeled vehicle energy expenditure. Has a `crit_slope` parameter where switchbacks become more efficient than direct climb. Best published proxy for MTB/ATV in open-source literature. + +**Reference implementations:** R `leastcostpath` package contains 30+ validated cost functions including Tobler, Tobler off-path, Irmischer-Clarke (male/female/off-path, fitted to USMA cadets), Naismith-Langmuir, Herzog, Minetti, Campbell 2019 percentiles. Port as needed. + +### 4.4 Friction Layers (Cost Surface Inputs) + +All pre-computed offline, tiled, cached. Updated infrequently. + +| Layer | Source | Resolution | Purpose | Update Frequency | +|---|---|---|---|---| +| Elevation | planet-dem.pmtiles | ~30m (z12) | Slope/grade calculation | Static | +| Land cover | NLCD | 30m | Vegetation traversal friction | ~Annual | +| Waterways | OSM | Rasterized from vectors | Barrier (∞ cost) except at bridges/fords | Weekly from planet PBF | +| Water bodies | OSM `natural=water` | Rasterized polygons | Barrier (∞) | Weekly | +| Cliffs | OSM `natural=cliff` | Rasterized lines | Barrier (∞) | Weekly | +| Land ownership | PAD-US | Polygon raster | Access restrictions per mode | ~Quarterly | +| Trails/roads | OSM + USFS | Rasterized lines | Low-cost corridors (negative friction) | Weekly | + +**NLCD friction mapping (foot mode example):** + +| NLCD Class | Description | Friction Multiplier | +|---|---|---| +| 11 | Open Water | ∞ | +| 21 | Developed, Open Space | 1.0 | +| 22 | Developed, Low Intensity | 1.2 | +| 31 | Barren Land | 1.1 | +| 41 | Deciduous Forest | 1.8 | +| 42 | Evergreen Forest | 2.0 | +| 43 | Mixed Forest | 1.9 | +| 52 | Shrub/Scrub | 1.5 | +| 71 | Grassland/Herbaceous | 1.2 | +| 90 | Woody Wetlands | 3.5 | +| 95 | Emergent Herbaceous Wetlands | 4.0 | + +Mode-specific adjustments: MTB and ATV get higher penalties on forest/wetland. ATV gets ∞ on wilderness-designated areas (PAD-US `Des_Tp = WA`). + +**Trail burn-in:** Rasterize OSM trails/tracks as cells with reduced friction (trail cell = 0.5× base, track = 0.3×, road = 0.1×). The pathfinder naturally gravitates toward and follows these corridors without special logic. + +### 4.5 Engine Choice + +**Recommended: scikit-image `MCP_Geometric` for initial build.** + +- Cython Dijkstra, 1–5 seconds on 2–4M cell grids +- `find_costs(start)` computes cumulative cost surface once; `traceback(target)` for any target is O(path length) — reuse for "nearest trail," "nearest road," and destination all in one pass +- `MCP_Flexible` subclass allows overriding `_travel_cost()` for anisotropic costs (uphill ≠ downhill) +- Pure Python integration with Flask backend +- Memory OK up to ~20–40M cells on 24GB + +**Performance path: Rust `pathfinding` crate as a microservice.** + +- A*, Dijkstra, HPA* (hierarchical) all available +- Custom successor function encodes anisotropic cost +- Sub-second on 4M cells +- `hierarchical_pathfinding` crate enables multi-resolution: coarse pass → refine in corridor +- Wrap in Axum HTTP server, call from Flask + +**Decision:** Start with scikit-image Python. If latency is a problem, rewrite the inner loop in Rust. The cost function, data pipeline, and API don't change. + +### 4.6 Multi-Resolution Strategy + +For routes where the wilderness segment exceeds ~10km, full-resolution pathfinding on the entire search area gets expensive. Use the Primordial Ground Guidance approach: + +1. **Coarse pass:** Downsample cost grid 4× (120m cells). Solve A*. Sub-second. +2. **Corridor extraction:** Buffer the coarse path by 200m. +3. **Fine pass:** Re-solve at native 30m resolution only within the corridor. Sub-second. +4. **Total:** <2 seconds for a 15km wilderness segment. + +### 4.7 Network Hand-Off + +The raster pathfinder needs to know where the trail/road network starts so it can stop: + +1. **Pre-compute trail entry points:** Extract from OSM all endpoints and intersections of `highway=path|track|footway|bridleway|unclassified|tertiary|secondary|primary`. Store as a PostGIS point table (or SQLite spatial index in `navi.db`). +2. **Rasterize entry points** onto the cost grid as target cells. +3. **Run `MCP.find_costs(start)`** — the Dijkstra wave expands until it reaches any entry-point cell. Use `goal_reached()` override in `MCP_Flexible` for early termination. +4. **Snap** the reached entry point to its nearest Valhalla graph node. +5. **Call Valhalla** from that node to destination with appropriate costing profile. +6. **Concatenate** raster path + Valhalla path into one GeoJSON with per-segment metadata. + +--- + +## 5. Data Acquisition Checklist + +| Dataset | Status | Size | Action | +|---|---|---|---| +| DEM (planet-dem.pmtiles) | ✅ Have it | 658GB | Serving via nginx from pi-nas | +| NLCD Land Cover (CONUS) | ❌ Not acquired | ~5GB | Download from USGS MRLC | +| NLCD Tree Canopy (CONUS) | ❌ Not acquired | ~2GB | Optional — continuous friction surface | +| OSM Planet PBF | ❌ Not acquired for this use | ~70GB | Extract waterways, cliffs, trails via osmium | +| PAD-US | ✅ Have source | 1.6GB in /mnt/nav/padus/ | Rasterize by access class | +| USFS Trail/Road layers | ✅ Have PMTiles | 848MB + 496MB | Need raw vectors for rasterization | +| Trail entry points index | ❌ Not built | ~50MB | Extract from OSM + USFS | + +**First acquisition:** NLCD. It's the single most impactful layer after the DEM — without land cover, the pathfinder can't distinguish open meadow from dense forest. + +--- + +## 6. Safety Considerations + +This system may guide people through dangerous terrain. Design constraints: + +- **Hard slope cutoffs are non-negotiable.** No route segment should ever cross terrain above the mode's max slope threshold, regardless of how much faster the direct path would be. +- **Confidence scoring:** Every response includes a `confidence` field (0.0–1.0) based on: DEM resolution vs route steepness, distance from nearest verified trail data, land cover data freshness, number of barrier crossings. +- **Fallback behaviors:** If no safe route exists within `max_search_km`, return an error with the direction and distance to the nearest trail (as a bearing, not a route). Never hallucinate a route through impassable terrain. +- **Per-step user confirmation (Aurora/Meshtastic):** In text mode, Aurora should confirm each major terrain transition ("You will cross a drainage heading southwest — confirm you can see safe footing"). A lost person should never blindly follow instructions into terrain they can't visually verify. +- **DSM vs DTM caveat:** Copernicus GLO-30 is a Digital Surface Model (includes treetops, buildings). A flat meadow next to tall pines will show false slope at the treeline. The system should note this in Aurora's instructions for forested areas. +- **30m resolution risk:** A 15m-wide cliff band can be smoothed into a single "steep but passable" cell. The safety gate catches obvious cliffs but may miss narrow features. Documented limitation; mitigated by upgrading to 10m USGS 3DEP in the future. + +--- + +## 7. Implementation Phases + +### Phase O1: Foundation +- Acquire NLCD CONUS land cover +- Build PMTiles elevation decoder + tile cache module +- Implement Tobler off-path cost function +- Prototype: scikit-image MCP on a small Idaho bbox (e.g., 20km × 20km around Sun Valley) +- Validate: does the path avoid canyons, prefer gentle slopes, follow drainages? + +### Phase O2: Friction Integration +- Rasterize NLCD into friction grid +- Rasterize OSM waterways/cliffs as barriers +- Rasterize PAD-US access restrictions +- Burn OSM trails/roads as low-cost corridors +- Combined cost surface for foot mode + +### Phase O3: Network Hand-Off +- Build trail entry point index from OSM + USFS +- Implement MCP → Valhalla stitching +- `/api/offroute` endpoint (foot mode only) +- GeoJSON response with per-segment metadata + +### Phase O4: Multi-Mode + Aurora +- Add MTB cost function (Herzog wheeled-transport) +- Add ATV cost function +- Mode-specific barrier rules (wilderness restrictions for MTB/ATV) +- Aurora tool integration — `offroute` in nav_tools.py +- Meshtastic text-based instruction generation (bearings, terrain descriptions) + +### Phase O5: Performance + Polish +- Multi-resolution pathfinding (coarse → corridor → fine) +- Rust pathfinder microservice (if Python latency is insufficient) +- Confidence scoring +- Navi frontend route visualization with segment coloring +- Elevation profile display per segment + +### Phase O6: Pi 5 Field Kit +- Offline PMTiles elevation access +- Pre-baked cost tiles for Idaho/CONUS-West +- Bbox-filter packager for all spatial datasets +- Full offline operation via Meshtastic ↔ Aurora ↔ offroute chain + +--- + +## 8. Infrastructure + +**Runtime services (VM 1130):** +- `/api/offroute` — Flask endpoint in RECON dashboard +- Tile cache — LRU in-memory decoded elevation arrays +- Valhalla Docker :8002 — on-network routing (already running) + +**Data (VM 1130 /mnt/nav/):** +- Pre-baked friction rasters (NLCD, barriers, trails) — tiled GeoTIFF or COG +- Trail entry point index — SQLite spatial in navi.db + +**Data (pi-nas /mnt/nas/nav/):** +- planet-dem.pmtiles — 658GB, served via nginx +- Regional GeoTIFF DEMs — 203GB, insurance until pipeline validated + +**Compute (cortex or matt-desktop):** +- One-time cost surface generation jobs (NLCD rasterization, OSM extraction, barrier tiling) + +--- + +## 9. Key Decisions Made + +| Decision | Rationale | +|---|---| +| No pre-rendered slope rasters | Pathfinder computes grade on the fly from cached elevation arrays. Simpler, no GDAL dependency at runtime. | +| planet-dem.pmtiles as single elevation source | Same data already drives contours + hillshade. 30m sufficient for first build. Global coverage. | +| scikit-image MCP for initial engine | Cython Dijkstra, proven on 2–4M cell grids, Python-native, anisotropic via MCP_Flexible. Rust upgrade path if needed. | +| Tobler off-path as primary foot cost model | Best-validated off-trail hiking function. Inherently anisotropic. 0.6× off-trail multiplier built in. | +| Trail burn-in (not separate hand-off logic) | Rasterizing trails as low-cost cells lets the pathfinder naturally follow them without mode-switching logic. | +| Pre-baked friction rasters (offline) | NLCD, barriers, and land ownership change slowly. Build once, cache, update periodically. | +| Multi-resolution for long routes | Coarse pass → corridor → fine pass. Standard technique from military route planning (Primordial Ground Guidance). | +| Confidence scoring on every response | Safety-critical system. User must know when to trust vs. verify the route. | + +--- + +## 10. Open Questions + +- [ ] What is the right `max_slope` cutoff per mode? Needs field testing / literature review. +- [ ] Should the pathfinder use A* (faster, needs admissible heuristic) or Dijkstra (guaranteed optimal, slower)? MCP uses Dijkstra; pyastar2d uses A*. +- [ ] How to generate natural-language terrain descriptions for Aurora from raster data? (e.g., "sagebrush slope" vs. "forested drainage") +- [ ] Should we pre-compute the full cost surface for Idaho/CONUS-West, or generate it on demand per request? +- [ ] How to handle seasonal/weather variations? (Snow, spring runoff, wildfire closures) +- [ ] Valhalla pedestrian elevation costing (PR #3234) — test and validate before relying on it for segments 2–4. +- [ ] USFS MVUM (Motor Vehicle Use Maps) — authoritative ATV/4WD legal access layer. Acquire and integrate for ATV mode. + +--- + +## References + +- Tobler, W. (1993). Three Presentations on Geographical Analysis and Modeling. NCGIA TR 93-1. +- Irmischer, I.J. & Clarke, K.C. (2018). Measuring and modeling the speed of human navigation. *Cartography and GIS*, 45(2), 177-186. +- Herzog, I. (2020). Spatial Analysis Based on Cost Functions. In *Archaeological Spatial Analysis*. +- Lewis, J. (2023). `leastcostpath` R package. CRAN. +- GRASS GIS. `r.walk` manual. grass.osgeo.org. +- Hoover, B. et al. (2019). CostMAP: An open-source software package for developing cost surfaces. LANL. +- Mapterhorn project. mapterhorn.com. BSD-3.