mirror of
https://github.com/zvx-echo6/navi.git
synced 2026-08-26 17:31:37 +00:00
malice
f9f2eb9b8f* MVUM Layer 3b: OSM parking as multi-modal Auto transition candidates
Adds OSM parking lots as a third multi-modal-Auto transition source alongside
MVUM trailheads (3a) and surface-change points (3c), so Auto can suggest
"drive to a parking lot, switch to foot/2w/4w" trips where no MVUM trailhead
exists -- BLM/state land, urban edges, anywhere OSM has parking but the USFS
trailhead layer does not. Backend-only; consumes the already-ingested
/mnt/nav/osm-parking.db read-only (no data-pipeline change).
- mvum_parking.py: OSMParkingIndex (process-wide singleton via load_parking_index)
over a shapely STRtree of parking points, mirroring MVUMSpatialIndex /
TrailheadIndex. Read-only SQLite. Drops access in (private,no,permit) at load.
query_parking_near_line(coords, buffer_m=2000) with the same coarse-bbox +
precise-distance filter as TrailheadIndex. Records carry
{lat, lon, name, road_class="parking", parking_type, access}.
Perf note: the ingest already stored representative_point() in lat/lon, so the
STRtree is built straight from those columns -- parsing the 1.6M WKB blobs at
boot would add minutes for an identical point.
- router.py: _try_hybrid_auto generalized to gather candidates from each AVAILABLE
source (trailhead index if present + surface-change always + parking index if
present) instead of hard-returning when trailhead_index is None, so parking-only
candidates still work. Combined list keeps the existing closest-first sort +
HYBRID_MAX_TRAILHEADS cap. Signature unchanged; record shape already compatible.
- app.py / offroute_route.py: load + inject the OSM parking singleton, mirroring
MVUM_SPATIAL_INDEX / MVUM_TRAILHEAD_INDEX. Failure logs a warning, degrades None.
- admin.py: GET /api/admin/osm-parking/info -> {count, build_time_seconds,
memory_estimate_mb}, mirroring /api/admin/mvum-spatial/info.
- backend/scripts/ingest_parking.py + README-osm-parking-ingest.md: the
data-pipeline ingest lifted to the repo with argparse (--geojsonseq/--db, no
/tmp) + the download/filter/export/ingest/restart refresh recipe.
Tests: test_mvum_parking.py (loads, near-line close-only, private/no/permit
filtered, null-access kept) + test_offroute.py::test_hybrid_consumes_parking_
candidates (parking-only source probed as a leg-1 destination). Full offroute
suite: 82 passed.
Real-DB sanity (not deployed): index loads 1,489,054 usable parking objects
(182,945 access-blocked dropped) in ~12 s using ~950 MB RSS per worker; a Redfish
Lake/Sawtooth corridor query returns 8 lots. The ~950 MB/worker memory cost is
notable -- flagging for review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Numpy-pack OSMParkingIndex coords + lazy records to cut RSS (~950->~570 MB/worker)
Store parking coords as packed float64 numpy arrays (_lats/_lons) and the
attribute columns as interned lists (_names/_parking_types/_accesses), and build
candidate record dicts lazily in query_parking_near_line instead of materializing
1.5M dicts + 1.5M shapely Point objects up front. road_class is the constant
"parking" so it is not stored per row.
Measured on the real /mnt/nav/osm-parking.db (1,489,054 usable rows):
RSS/worker ~950 MB -> ~570 MB (~40%), build ~11 s. Across 2 gunicorn workers that
is ~1.9 GB -> ~1.14 GB.
NOTE: this does NOT reach the ~250 MB originally targeted. The remaining cost is
the shapely STRtree itself: it permanently retains the input geometries
(tree.geometries len == row count), so the transient `del points` does not free
them. Attribution on the real DB: columns-only 137 MB, retained Point objects
+230 MB, STRtree index +110 MB. Reaching ~250 MB would require dropping the
shapely STRtree for a coordinate-only structure (e.g. scipy cKDTree over the
lon/lat arrays), which changes the line-buffer query into a per-vertex radius
query -- a behavior change beyond this fix-up's scope. Flagged for a follow-up.
Tests unchanged except one assertion (`len(idx.records) == idx.count`); full
offroute suite 82 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
70 lines
3.1 KiB
Markdown
70 lines
3.1 KiB
Markdown
# OSM parking ingest
|
|
|
|
How `/mnt/nav/osm-parking.db` is **produced** from OSM data. navi-offroute's Layer 3b
|
|
(`backend/services/navi_offroute/mvum_parking.py`, `OSMParkingIndex`) **consumes** it
|
|
read-only as multi-modal Auto transition candidates (drive → parking → foot/2w/4w).
|
|
This is the producer; nothing here touches `navi.db`.
|
|
|
|
## Source
|
|
|
|
geofabrik regional extract (North America used in production):
|
|
<https://download.geofabrik.de/north-america-latest.osm.pbf> (+ the `.md5`).
|
|
|
|
Keep downloads/intermediates under `/mnt/nav/sources/osm/`. The final DB lives at
|
|
`/mnt/nav/osm-parking.db` (separate from `navi.db`).
|
|
|
|
> Tools: `osmium` (osmium-tool, CLI) and the repo venv's Python (`shapely` — no
|
|
> pyosmium/GDAL needed). No service writes here; the index loads the DB read-only.
|
|
|
|
## Refresh recipe
|
|
|
|
```bash
|
|
cd /mnt/nav/sources/osm
|
|
|
|
# 1. download + verify
|
|
curl -L --fail -o north-america-latest.osm.pbf.md5 \
|
|
https://download.geofabrik.de/north-america-latest.osm.pbf.md5
|
|
curl -L --fail -C - -o north-america-latest.osm.pbf \
|
|
https://download.geofabrik.de/north-america-latest.osm.pbf
|
|
exp=$(awk '{print $1}' north-america-latest.osm.pbf.md5)
|
|
act=$(md5sum north-america-latest.osm.pbf | awk '{print $1}')
|
|
[ "$exp" = "$act" ] && echo "MD5 OK" || { echo "MD5 MISMATCH"; exit 1; }
|
|
|
|
# 2. filter to amenity=parking (nwr = nodes+ways+relations; referenced nodes kept
|
|
# by default so way/relation polygons stay buildable)
|
|
osmium tags-filter north-america-latest.osm.pbf nwr/amenity=parking \
|
|
-o north-america-parking.osm.pbf --overwrite
|
|
|
|
# 3. export to GeoJSONSeq with osm type+id attributes
|
|
osmium export north-america-parking.osm.pbf -f geojsonseq -a type,id \
|
|
-o north-america-parking.geojsonseq --overwrite
|
|
|
|
# 4. ingest -> osm-parking.db (repo venv python; ~3 min for NA)
|
|
/home/zvx/projects/repos/navi-mono/backend/.venv/bin/python \
|
|
/home/zvx/projects/repos/navi-mono/backend/scripts/ingest_parking.py \
|
|
--geojsonseq north-america-parking.geojsonseq --db /mnt/nav/osm-parking.db
|
|
|
|
# 5. (optional) reclaim space after the dedupe re-write
|
|
sqlite3 /mnt/nav/osm-parking.db "VACUUM;"
|
|
|
|
# 6. pick up the new data: each navi-offroute worker rebuilds OSMParkingIndex at boot
|
|
sudo systemctl restart navi-offroute
|
|
```
|
|
|
|
## Schema
|
|
|
|
`parking(id INTEGER PK, osm_id TEXT, osm_type TEXT, name TEXT, capacity INTEGER NULL,
|
|
access TEXT NULL, parking_type TEXT NULL, lat REAL, lon REAL, shape BLOB)` + index
|
|
`idx_parking_latlon(lat, lon)`. Geometry: WKB Point for nodes, (Multi)Polygon for
|
|
areas; `lat`/`lon` hold an interior `representative_point()` (the index builds its
|
|
STRtree from these directly, skipping per-row WKB parsing at boot).
|
|
|
|
## Notes
|
|
|
|
- `osmium export` emits each closed area-way **twice** (raw LineString + assembled
|
|
polygon); the ingest drops Line geometries to keep one geometry per object. Rare
|
|
genuinely-open parking ways (data errors) are dropped with them.
|
|
- `OSMParkingIndex` further drops `access` in (`private`, `no`, `permit`) at load —
|
|
off-limits lots are useless as transition candidates.
|
|
- North America ≈ 1.67M parking objects after dedupe (~500 MB DB). Verify:
|
|
`sqlite3 /mnt/nav/osm-parking.db "SELECT COUNT(*), osm_type FROM parking GROUP BY osm_type;"`
|