matt/navi
1
0
Fork 0
mirror of https://github.com/zvx-echo6/navi.git synced 2026-08-26 17:31:37 +00:00
Code Issues Projects Releases Packages Wiki Activity
navi/backend/scripts/README.md
malice d0e357a3bb decouple: add scripts/overture_import.py (relocating from recon) 
PR-A of the overture-import relocation. The Overture Places ETL moves from
recon (where it produced data nothing in recon consumes) to navi-backend (the
side that owns the consumer, navi-places). Additive: recon's copy stays live
until PR-B; this just establishes the navi-side copy + deps + docs.

- scripts/overture_import.py: verbatim port of recon's script (recon master
  879df84). The ONLY non-verbatim change is the line-8 docstring usage hint,
  swapped from `/opt/recon` + venv to the navi-backend path + .venv.
- pyproject.toml: add `duckdb>=1.5` (recon runs 1.5.2; psycopg2-binary already
  present). It's the only new dep.
- scripts/README.md: document the manual-only trigger + invocation.

Source release is pinned in-code: OVERTURE_RELEASE = '2026-04-15.0'.
No tests (ETL; none on recon either). Per cleanup #29, the script has no lib/
imports — fully self-contained (stdlib + duckdb + psycopg2).

Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 13:49:32 -06:00

1.5 KiB
Raw Blame History

navi-backend scripts

Operational scripts for navi-backend. Not part of any Flask service — run manually.

overture_import.py — Overture Places ETL

Loads Overture Maps Places into the host overture PostgreSQL/PostGIS database (table places), which navi-places consumes for place enrichment (phone, website, brand, OSM cross-refs). Relocated from recon in the navi↔recon decoupling (recon produced this data but consumed none of it).

  • Source: public S3 Parquet — s3://overturemaps-us-west-2/release/<OVERTURE_RELEASE>/theme=places/type=place/*, read via DuckDB (httpfs, no credentials).
  • Release: pinned in-code — OVERTURE_RELEASE = '2026-04-15.0'. Bump it (and re-run) when a newer Overture release is desired.
  • Filter: North America bounding box.
  • Write semantics: idempotent UPSERT (INSERT … ON CONFLICT (id) DO UPDATE), batched. Safe to re-run.
  • Config (env): OVERTURE_DB_HOST / OVERTURE_DB_PORT / OVERTURE_DB_NAME / OVERTURE_DB_USER / OVERTURE_DB_PASSWORD (defaults localhost / 5432 / overture / overture / empty). On VM 1130 these come from the same host PG cluster navi-places reads.

Trigger: manual-only

There is no cron job or systemd timer — run it on demand (e.g. when a new Overture release is published and OVERTURE_RELEASE is bumped):

cd /home/zvx/projects/repos/navi-backend && .venv/bin/python scripts/overture_import.py

The full S3 query takes several minutes; progress is logged to stdout.