mirror of
https://github.com/zvx-echo6/navi.git
synced 2026-08-26 17:31:37 +00:00
89 lines
4.1 KiB
Markdown
89 lines
4.1 KiB
Markdown
|
# MVUM ingest pipeline
|
|||
|
|
|
|||
|
|
How the `mvum_roads` / `mvum_trails` tables in `navi.db` are **produced** from USFS data.
|
|||
|
|
The navi-offroute service code only **consumes** these tables — see
|
|||
|
|
`backend/services/navi_offroute/mvum.py` (access logic + `MVUMSpatialIndex`),
|
|||
|
|
`mvum_annotate.py` (Layer 1 per-edge annotation), and `mvum_exclude.py` (Layer 2c
|
|||
|
|
`exclude_polygons`). The scripts here are the producers.
|
|||
|
|
|
|||
|
|
## Source datasets (USFS EDW)
|
|||
|
|
|
|||
|
|
Published as File Geodatabases (national rollups):
|
|||
|
|
|
|||
|
|
- Roads: <https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_MVUM_Road.gdb.zip>
|
|||
|
|
- Trails (MVUM legal access): <https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_MVUM_Trail.gdb.zip>
|
|||
|
|
- Trail centerlines (geometry donor for the backfill): <https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_Trail_NFS_Publish.gdb.zip>
|
|||
|
|
|
|||
|
|
Keep the unzipped `.gdb`s under `/mnt/nav/sources/mvum/`. Both MVUM and NFS data are
|
|||
|
|
NAD83 / EPSG:4269, and `ogr2ogr` is loaded with `SPATIALITE=NO` so geometry is stored as
|
|||
|
|
plain WKB blobs that shapely and `MVUMSpatialIndex` read directly.
|
|||
|
|
|
|||
|
|
> **Always, before any write to `navi.db`:** snapshot to a named, dated, persistent file
|
|||
|
|
> (never `/tmp`) and stop the service:
|
|||
|
|
> ```
|
|||
|
|
> sudo systemctl stop navi-offroute
|
|||
|
|
> sqlite3 /mnt/nav/navi.db ".backup /mnt/nav/navi-pre-<change>-<YYYY-MM-DD>.db"
|
|||
|
|
> # ... run the ingest/backfill ...
|
|||
|
|
> sudo systemctl start navi-offroute
|
|||
|
|
> ```
|
|||
|
|
> Rollback = restore the snapshot over `navi.db` and restart.
|
|||
|
|
|
|||
|
|
## 1. Initial ingest (one-time, building navi.db from scratch)
|
|||
|
|
|
|||
|
|
Load each MVUM GDB layer into its table. `ogr2ogr` shape (run from `/mnt/nav/sources/mvum/`):
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
# roads -> mvum_roads
|
|||
|
|
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Road.gdb Trans_MVUM_Road \
|
|||
|
|
-dsco SPATIALITE=NO -nln mvum_roads -update -overwrite
|
|||
|
|
|
|||
|
|
# trails -> mvum_trails
|
|||
|
|
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Trail.gdb Trans_MVUM_Trail \
|
|||
|
|
-dsco SPATIALITE=NO -nln mvum_trails -update -overwrite
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`-update` appends to the existing `navi.db` (it holds other navi tables); `-overwrite`
|
|||
|
|
replaces just the named table. This yields `ogc_fid`, a `shape` WKB blob, and the MVUM
|
|||
|
|
attribute columns (`id`, `name`, `forestname`, `symbol`, `atv`, `highclearancevehicle`,
|
|||
|
|
`e_bike_class1`, the `*_datesopen` fields, etc.). Then run the trail backfill (step 3) to
|
|||
|
|
fill the geometry-sparse trail rows.
|
|||
|
|
|
|||
|
|
## 2. Refresh (USFS published new MVUM data)
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
cd /mnt/nav/sources/mvum
|
|||
|
|
for f in Trans_MVUM_Road Trans_MVUM_Trail Trans_Trail_NFS_Publish; do
|
|||
|
|
curl -L --fail -o "$f.gdb.zip" \
|
|||
|
|
"https://data.fs.usda.gov/geodata/edw/edw_resources/fc/$f.gdb.zip"
|
|||
|
|
rm -rf "$f.gdb" && unzip -o "$f.gdb.zip"
|
|||
|
|
done
|
|||
|
|
# (snapshot + stop service as above)
|
|||
|
|
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Road.gdb Trans_MVUM_Road -dsco SPATIALITE=NO -nln mvum_roads -update -overwrite
|
|||
|
|
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Trail.gdb Trans_MVUM_Trail -dsco SPATIALITE=NO -nln mvum_trails -update -overwrite
|
|||
|
|
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db # then restart service
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 3. Repair (mvum_trails has NULL shapes — the gap fixed in P1–P3)
|
|||
|
|
|
|||
|
|
When only the trail geometry is missing (no full re-ingest needed), just run the NFS
|
|||
|
|
backfill. Preview first, then apply:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
# preview (no writes)
|
|||
|
|
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db --dry-run
|
|||
|
|
|
|||
|
|
# apply (after snapshot + stopping the service)
|
|||
|
|
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`mvum_backfill.py` extracts the NFS centerlines from `--nfs-gdb` (default
|
|||
|
|
`/mnt/nav/sources/mvum/Trans_Trail_NFS_Publish.gdb`) into a `_nfs_centerline_extract.sqlite`
|
|||
|
|
cache beside the `.gdb`, matches each null `mvum_trails` row by `TRAIL_NO`+`TRAIL_NAME`
|
|||
|
|
within the forest's `ADMIN_ORG` prefix, merges the matched segments, and writes WKB.
|
|||
|
|
It is idempotent (`UPDATE ... WHERE shape IS NULL`) and prints counters
|
|||
|
|
(`rows_attempted`, `rows_updated`, `rows_skipped_no_match`, `rows_skipped_nfs_null_geom`).
|
|||
|
|
Typical full-population recovery is ~97%; the residual is rows with no join key or trails
|
|||
|
|
absent from NFS / null in NFS too.
|
|||
|
|
|
|||
|
|
After any write: `sudo systemctl start navi-offroute` (each worker rebuilds
|
|||
|
|
`MVUMSpatialIndex` at boot, ~8 s).
|