mirror of
https://github.com/zvx-echo6/navi.git
synced 2026-08-26 17:31:37 +00:00
malice
f9f2eb9b8f* MVUM Layer 3b: OSM parking as multi-modal Auto transition candidates
Adds OSM parking lots as a third multi-modal-Auto transition source alongside
MVUM trailheads (3a) and surface-change points (3c), so Auto can suggest
"drive to a parking lot, switch to foot/2w/4w" trips where no MVUM trailhead
exists -- BLM/state land, urban edges, anywhere OSM has parking but the USFS
trailhead layer does not. Backend-only; consumes the already-ingested
/mnt/nav/osm-parking.db read-only (no data-pipeline change).
- mvum_parking.py: OSMParkingIndex (process-wide singleton via load_parking_index)
over a shapely STRtree of parking points, mirroring MVUMSpatialIndex /
TrailheadIndex. Read-only SQLite. Drops access in (private,no,permit) at load.
query_parking_near_line(coords, buffer_m=2000) with the same coarse-bbox +
precise-distance filter as TrailheadIndex. Records carry
{lat, lon, name, road_class="parking", parking_type, access}.
Perf note: the ingest already stored representative_point() in lat/lon, so the
STRtree is built straight from those columns -- parsing the 1.6M WKB blobs at
boot would add minutes for an identical point.
- router.py: _try_hybrid_auto generalized to gather candidates from each AVAILABLE
source (trailhead index if present + surface-change always + parking index if
present) instead of hard-returning when trailhead_index is None, so parking-only
candidates still work. Combined list keeps the existing closest-first sort +
HYBRID_MAX_TRAILHEADS cap. Signature unchanged; record shape already compatible.
- app.py / offroute_route.py: load + inject the OSM parking singleton, mirroring
MVUM_SPATIAL_INDEX / MVUM_TRAILHEAD_INDEX. Failure logs a warning, degrades None.
- admin.py: GET /api/admin/osm-parking/info -> {count, build_time_seconds,
memory_estimate_mb}, mirroring /api/admin/mvum-spatial/info.
- backend/scripts/ingest_parking.py + README-osm-parking-ingest.md: the
data-pipeline ingest lifted to the repo with argparse (--geojsonseq/--db, no
/tmp) + the download/filter/export/ingest/restart refresh recipe.
Tests: test_mvum_parking.py (loads, near-line close-only, private/no/permit
filtered, null-access kept) + test_offroute.py::test_hybrid_consumes_parking_
candidates (parking-only source probed as a leg-1 destination). Full offroute
suite: 82 passed.
Real-DB sanity (not deployed): index loads 1,489,054 usable parking objects
(182,945 access-blocked dropped) in ~12 s using ~950 MB RSS per worker; a Redfish
Lake/Sawtooth corridor query returns 8 lots. The ~950 MB/worker memory cost is
notable -- flagging for review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Numpy-pack OSMParkingIndex coords + lazy records to cut RSS (~950->~570 MB/worker)
Store parking coords as packed float64 numpy arrays (_lats/_lons) and the
attribute columns as interned lists (_names/_parking_types/_accesses), and build
candidate record dicts lazily in query_parking_near_line instead of materializing
1.5M dicts + 1.5M shapely Point objects up front. road_class is the constant
"parking" so it is not stored per row.
Measured on the real /mnt/nav/osm-parking.db (1,489,054 usable rows):
RSS/worker ~950 MB -> ~570 MB (~40%), build ~11 s. Across 2 gunicorn workers that
is ~1.9 GB -> ~1.14 GB.
NOTE: this does NOT reach the ~250 MB originally targeted. The remaining cost is
the shapely STRtree itself: it permanently retains the input geometries
(tree.geometries len == row count), so the transient `del points` does not free
them. Attribution on the real DB: columns-only 137 MB, retained Point objects
+230 MB, STRtree index +110 MB. Reaching ~250 MB would require dropping the
shapely STRtree for a coordinate-only structure (e.g. scipy cKDTree over the
lon/lat arrays), which changes the line-buffer query into a per-vertex radius
query -- a behavior change beyond this fix-up's scope. Flagged for a follow-up.
Tests unchanged except one assertion (`len(idx.records) == idx.count`); full
offroute suite 82 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| ingest_parking.py | ||
| mvum_backfill.py | ||
| overture_import.py | ||
| README-mvum-ingest.md | ||
| README-osm-parking-ingest.md | ||
| README.md | ||
navi-backend scripts
Operational scripts for navi-backend. Not part of any Flask service — run manually.
overture_import.py — Overture Places ETL
Loads Overture Maps Places into the host overture PostgreSQL/PostGIS database
(table places), which navi-places consumes for place enrichment (phone,
website, brand, OSM cross-refs). Relocated from recon in the navi↔recon
decoupling (recon produced this data but consumed none of it).
- Source: public S3 Parquet —
s3://overturemaps-us-west-2/release/<OVERTURE_RELEASE>/theme=places/type=place/*, read via DuckDB (httpfs, no credentials). - Release: pinned in-code —
OVERTURE_RELEASE = '2026-04-15.0'. Bump it (and re-run) when a newer Overture release is desired. - Filter: North America bounding box.
- Write semantics: idempotent UPSERT (
INSERT … ON CONFLICT (id) DO UPDATE), batched. Safe to re-run. - Config (env):
OVERTURE_DB_HOST/OVERTURE_DB_PORT/OVERTURE_DB_NAME/OVERTURE_DB_USER/OVERTURE_DB_PASSWORD(defaultslocalhost/5432/overture/overture/ empty). On VM 1130 these come from the same host PG cluster navi-places reads.
Trigger: manual-only
There is no cron job or systemd timer — run it on demand (e.g. when a new
Overture release is published and OVERTURE_RELEASE is bumped):
cd /home/zvx/projects/repos/navi-backend && .venv/bin/python scripts/overture_import.py
The full S3 query takes several minutes; progress is logged to stdout.