navi/backend/scripts/README-mvum-ingest.md

MVUM ingest pipeline

How the mvum_roads / mvum_trails tables in navi.db are produced from USFS data. The navi-offroute service code only consumes these tables — see backend/services/navi_offroute/mvum.py (access logic + MVUMSpatialIndex), mvum_annotate.py (Layer 1 per-edge annotation), and mvum_exclude.py (Layer 2c exclude_polygons). The scripts here are the producers.

Source datasets (USFS EDW)

Published as File Geodatabases (national rollups):

• Roads: https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_MVUM_Road.gdb.zip
• Trails (MVUM legal access): https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_MVUM_Trail.gdb.zip
• Trail centerlines (geometry donor for the backfill): https://data.fs.usda.gov/geodata/edw/edw_resources/fc/Trans_Trail_NFS_Publish.gdb.zip

Keep the unzipped .gdbs under /mnt/nav/sources/mvum/. Both MVUM and NFS data are NAD83 / EPSG:4269, and ogr2ogr is loaded with SPATIALITE=NO so geometry is stored as plain WKB blobs that shapely and MVUMSpatialIndex read directly.

Always, before any write to navi.db: snapshot to a named, dated, persistent file (never /tmp) and stop the service:

sudo systemctl stop navi-offroute
sqlite3 /mnt/nav/navi.db ".backup /mnt/nav/navi-pre-<change>-<YYYY-MM-DD>.db"
#   ... run the ingest/backfill ...
sudo systemctl start navi-offroute

Rollback = restore the snapshot over navi.db and restart.

1. Initial ingest (one-time, building navi.db from scratch)

Load each MVUM GDB layer into its table. ogr2ogr shape (run from /mnt/nav/sources/mvum/):

# roads -> mvum_roads
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Road.gdb  Trans_MVUM_Road \
        -dsco SPATIALITE=NO -nln mvum_roads  -update -overwrite

# trails -> mvum_trails
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Trail.gdb Trans_MVUM_Trail \
        -dsco SPATIALITE=NO -nln mvum_trails -update -overwrite

-update appends to the existing navi.db (it holds other navi tables); -overwrite replaces just the named table. This yields ogc_fid, a shape WKB blob, and the MVUM attribute columns (id, name, forestname, symbol, atv, highclearancevehicle, e_bike_class1, the *_datesopen fields, etc.). Then run the trail backfill (step 3) to fill the geometry-sparse trail rows.

2. Refresh (USFS published new MVUM data)

cd /mnt/nav/sources/mvum
for f in Trans_MVUM_Road Trans_MVUM_Trail Trans_Trail_NFS_Publish; do
  curl -L --fail -o "$f.gdb.zip" \
    "https://data.fs.usda.gov/geodata/edw/edw_resources/fc/$f.gdb.zip"
  rm -rf "$f.gdb" && unzip -o "$f.gdb.zip"
done
# (snapshot + stop service as above)
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Road.gdb  Trans_MVUM_Road  -dsco SPATIALITE=NO -nln mvum_roads  -update -overwrite
ogr2ogr -f SQLite /mnt/nav/navi.db Trans_MVUM_Trail.gdb Trans_MVUM_Trail -dsco SPATIALITE=NO -nln mvum_trails -update -overwrite
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db   # then restart service

3. Repair (mvum_trails has NULL shapes — the gap fixed in P1–P3)

When only the trail geometry is missing (no full re-ingest needed), just run the NFS backfill. Preview first, then apply:

# preview (no writes)
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db --dry-run

# apply (after snapshot + stopping the service)
python backend/scripts/mvum_backfill.py --db-path /mnt/nav/navi.db

mvum_backfill.py extracts the NFS centerlines from --nfs-gdb (default /mnt/nav/sources/mvum/Trans_Trail_NFS_Publish.gdb) into a _nfs_centerline_extract.sqlite cache beside the .gdb, matches each null mvum_trails row by TRAIL_NO+TRAIL_NAME within the forest's ADMIN_ORG prefix, merges the matched segments, and writes WKB. It is idempotent (UPDATE ... WHERE shape IS NULL) and prints counters (rows_attempted, rows_updated, rows_skipped_no_match, rows_skipped_nfs_null_geom). Typical full-population recovery is ~97%; the residual is rows with no join key or trails absent from NFS / null in NFS too.

After any write: sudo systemctl start navi-offroute (each worker rebuilds MVUMSpatialIndex at boot, ~8 s).