matt/navi

mirror of https://github.com/zvx-echo6/navi.git synced 2026-08-26 17:31:37 +00:00

MVUM Layer 3b: OSM parking as multi-modal Auto transition candidates (#31 )

* MVUM Layer 3b: OSM parking as multi-modal Auto transition candidates

Adds OSM parking lots as a third multi-modal-Auto transition source alongside
MVUM trailheads (3a) and surface-change points (3c), so Auto can suggest
"drive to a parking lot, switch to foot/2w/4w" trips where no MVUM trailhead
exists -- BLM/state land, urban edges, anywhere OSM has parking but the USFS
trailhead layer does not. Backend-only; consumes the already-ingested
/mnt/nav/osm-parking.db read-only (no data-pipeline change).

- mvum_parking.py: OSMParkingIndex (process-wide singleton via load_parking_index)
  over a shapely STRtree of parking points, mirroring MVUMSpatialIndex /
  TrailheadIndex. Read-only SQLite. Drops access in (private,no,permit) at load.
  query_parking_near_line(coords, buffer_m=2000) with the same coarse-bbox +
  precise-distance filter as TrailheadIndex. Records carry
  {lat, lon, name, road_class="parking", parking_type, access}.
  Perf note: the ingest already stored representative_point() in lat/lon, so the
  STRtree is built straight from those columns -- parsing the 1.6M WKB blobs at
  boot would add minutes for an identical point.
- router.py: _try_hybrid_auto generalized to gather candidates from each AVAILABLE
  source (trailhead index if present + surface-change always + parking index if
  present) instead of hard-returning when trailhead_index is None, so parking-only
  candidates still work. Combined list keeps the existing closest-first sort +
  HYBRID_MAX_TRAILHEADS cap. Signature unchanged; record shape already compatible.
- app.py / offroute_route.py: load + inject the OSM parking singleton, mirroring
  MVUM_SPATIAL_INDEX / MVUM_TRAILHEAD_INDEX. Failure logs a warning, degrades None.
- admin.py: GET /api/admin/osm-parking/info -> {count, build_time_seconds,
  memory_estimate_mb}, mirroring /api/admin/mvum-spatial/info.
- backend/scripts/ingest_parking.py + README-osm-parking-ingest.md: the
  data-pipeline ingest lifted to the repo with argparse (--geojsonseq/--db, no
  /tmp) + the download/filter/export/ingest/restart refresh recipe.

Tests: test_mvum_parking.py (loads, near-line close-only, private/no/permit
filtered, null-access kept) + test_offroute.py::test_hybrid_consumes_parking_
candidates (parking-only source probed as a leg-1 destination). Full offroute
suite: 82 passed.

Real-DB sanity (not deployed): index loads 1,489,054 usable parking objects
(182,945 access-blocked dropped) in ~12 s using ~950 MB RSS per worker; a Redfish
Lake/Sawtooth corridor query returns 8 lots. The ~950 MB/worker memory cost is
notable -- flagging for review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Numpy-pack OSMParkingIndex coords + lazy records to cut RSS (~950->~570 MB/worker)

Store parking coords as packed float64 numpy arrays (_lats/_lons) and the
attribute columns as interned lists (_names/_parking_types/_accesses), and build
candidate record dicts lazily in query_parking_near_line instead of materializing
1.5M dicts + 1.5M shapely Point objects up front. road_class is the constant
"parking" so it is not stored per row.

Measured on the real /mnt/nav/osm-parking.db (1,489,054 usable rows):
RSS/worker ~950 MB -> ~570 MB (~40%), build ~11 s. Across 2 gunicorn workers that
is ~1.9 GB -> ~1.14 GB.

NOTE: this does NOT reach the ~250 MB originally targeted. The remaining cost is
the shapely STRtree itself: it permanently retains the input geometries
(tree.geometries len == row count), so the transient `del points` does not free
them. Attribution on the real DB: columns-only 137 MB, retained Point objects
+230 MB, STRtree index +110 MB. Reaching ~250 MB would require dropping the
shapely STRtree for a coordinate-only structure (e.g. scipy cKDTree over the
lon/lat arrays), which changes the line-buffer query into a per-vertex radius
query -- a behavior change beyond this fix-up's scope. Flagged for a follow-up.

Tests unchanged except one assertion (`len(idx.records) == idx.count`); full
offroute suite 82 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 14:48:35 -06:00

3.1 KiB

Raw Blame History

OSM parking ingest

How /mnt/nav/osm-parking.db is produced from OSM data. navi-offroute's Layer 3b (backend/services/navi_offroute/mvum_parking.py, OSMParkingIndex) consumes it read-only as multi-modal Auto transition candidates (drive → parking → foot/2w/4w). This is the producer; nothing here touches navi.db.

Source

geofabrik regional extract (North America used in production): https://download.geofabrik.de/north-america-latest.osm.pbf (+ the .md5).

Keep downloads/intermediates under /mnt/nav/sources/osm/. The final DB lives at /mnt/nav/osm-parking.db (separate from navi.db).

Tools: osmium (osmium-tool, CLI) and the repo venv's Python (shapely — no pyosmium/GDAL needed). No service writes here; the index loads the DB read-only.

Refresh recipe

cd /mnt/nav/sources/osm

# 1. download + verify
curl -L --fail -o north-america-latest.osm.pbf.md5 \
  https://download.geofabrik.de/north-america-latest.osm.pbf.md5
curl -L --fail -C - -o north-america-latest.osm.pbf \
  https://download.geofabrik.de/north-america-latest.osm.pbf
exp=$(awk '{print $1}' north-america-latest.osm.pbf.md5)
act=$(md5sum north-america-latest.osm.pbf | awk '{print $1}')
[ "$exp" = "$act" ] && echo "MD5 OK" || { echo "MD5 MISMATCH"; exit 1; }

# 2. filter to amenity=parking (nwr = nodes+ways+relations; referenced nodes kept
#    by default so way/relation polygons stay buildable)
osmium tags-filter north-america-latest.osm.pbf nwr/amenity=parking \
  -o north-america-parking.osm.pbf --overwrite

# 3. export to GeoJSONSeq with osm type+id attributes
osmium export north-america-parking.osm.pbf -f geojsonseq -a type,id \
  -o north-america-parking.geojsonseq --overwrite

# 4. ingest -> osm-parking.db  (repo venv python; ~3 min for NA)
/home/zvx/projects/repos/navi-mono/backend/.venv/bin/python \
  /home/zvx/projects/repos/navi-mono/backend/scripts/ingest_parking.py \
  --geojsonseq north-america-parking.geojsonseq --db /mnt/nav/osm-parking.db

# 5. (optional) reclaim space after the dedupe re-write
sqlite3 /mnt/nav/osm-parking.db "VACUUM;"

# 6. pick up the new data: each navi-offroute worker rebuilds OSMParkingIndex at boot
sudo systemctl restart navi-offroute

Schema

parking(id INTEGER PK, osm_id TEXT, osm_type TEXT, name TEXT, capacity INTEGER NULL, access TEXT NULL, parking_type TEXT NULL, lat REAL, lon REAL, shape BLOB) + index idx_parking_latlon(lat, lon). Geometry: WKB Point for nodes, (Multi)Polygon for areas; lat/lon hold an interior representative_point() (the index builds its STRtree from these directly, skipping per-row WKB parsing at boot).

Notes

osmium export emits each closed area-way twice (raw LineString + assembled polygon); the ingest drops Line geometries to keep one geometry per object. Rare genuinely-open parking ways (data errors) are dropped with them.
OSMParkingIndex further drops access in (private, no, permit) at load — off-limits lots are useless as transition candidates.
North America ≈ 1.67M parking objects after dedupe (~500 MB DB). Verify: sqlite3 /mnt/nav/osm-parking.db "SELECT COUNT(*), osm_type FROM parking GROUP BY osm_type;"

3.1 KiB Raw Blame History