matt/navi
1
0
Fork 0
mirror of https://github.com/zvx-echo6/navi.git synced 2026-08-26 17:31:37 +00:00
Code Issues Projects Releases Packages Wiki Activity
navi/backend/scripts
History
malice 0c43c872bb
 Commit reproducible MVUM ingest script (P4) (#25) 
Operational scripts only (no service code, no tests).

- backend/scripts/mvum_backfill.py: cleaned, repo-ready version of the P2/P3 NFS-centerline
  backfill. argparse --db-path (default /mnt/nav/navi.db) / --nfs-gdb (default the EDW
  Trans_Trail_NFS_Publish.gdb) / --dry-run; no /tmp, no test-DB paths, no prod-refusing
  guard (intended for prod, gated by --dry-run + the README snapshot step). Extracts NFS
  geometry from the .gdb via ogr2ogr into a cache beside it (no GDAL Python bindings here),
  matches null mvum_trails rows by TRAIL_NO+TRAIL_NAME within the forest ADMIN_ORG prefix,
  merges segments, writes WKB. Idempotent (UPDATE WHERE shape IS NULL). Prints
  rows_attempted/rows_updated/rows_skipped_no_match/rows_skipped_nfs_null_geom.
- backend/scripts/README-mvum-ingest.md: source URLs, initial ingest (ogr2ogr shapes),
  refresh, repair (the NULL-shape gap), snapshot-first + stop-service guardrails, and the
  produce-vs-consume pointer to mvum.py / mvum_annotate.py / mvum_exclude.py.

Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:36:41 -06:00
..
mvum_backfill.py Commit reproducible MVUM ingest script (P4) (#25) 2026-05-25 20:36:41 -06:00
overture_import.py decouple: add scripts/overture_import.py (relocating from recon) 2026-05-23 13:49:32 -06:00
README-mvum-ingest.md Commit reproducible MVUM ingest script (P4) (#25) 2026-05-25 20:36:41 -06:00
README.md decouple: add scripts/overture_import.py (relocating from recon) 2026-05-23 13:49:32 -06:00

README.md

navi-backend scripts

Operational scripts for navi-backend. Not part of any Flask service — run manually.

overture_import.py — Overture Places ETL

Loads Overture Maps Places into the host overture PostgreSQL/PostGIS database (table places), which navi-places consumes for place enrichment (phone, website, brand, OSM cross-refs). Relocated from recon in the navi↔recon decoupling (recon produced this data but consumed none of it).

  • Source: public S3 Parquet — s3://overturemaps-us-west-2/release/<OVERTURE_RELEASE>/theme=places/type=place/*, read via DuckDB (httpfs, no credentials).
  • Release: pinned in-code — OVERTURE_RELEASE = '2026-04-15.0'. Bump it (and re-run) when a newer Overture release is desired.
  • Filter: North America bounding box.
  • Write semantics: idempotent UPSERT (INSERT … ON CONFLICT (id) DO UPDATE), batched. Safe to re-run.
  • Config (env): OVERTURE_DB_HOST / OVERTURE_DB_PORT / OVERTURE_DB_NAME / OVERTURE_DB_USER / OVERTURE_DB_PASSWORD (defaults localhost / 5432 / overture / overture / empty). On VM 1130 these come from the same host PG cluster navi-places reads.

Trigger: manual-only

There is no cron job or systemd timer — run it on demand (e.g. when a new Overture release is published and OVERTURE_RELEASE is bumped):

cd /home/zvx/projects/repos/navi-backend && .venv/bin/python scripts/overture_import.py

The full S3 query takes several minutes; progress is logged to stdout.