navi/backend
Matt f537a55266 wiki_rewrite: paginate fully + extend rewriter to wikivoyage
The Twin Falls Wikivoyage investigation surfaced two coordinated gaps that kept
OSM wikivoyage tags from rewriting to local Kiwix URLs even though the
wikivoyage ZIM is loaded and serving.

Fix 1 (pagination) -- wiki_rewrite.py: append ?count=-1 to the Kiwix OPDS catalog
fetch. kiwix-serve's /catalog/v2/entries defaults to the first 10 entries; the
library has 17, so wikivoyage (and other page-2 ZIMs) were never seen by
_discover_zims and never entered _zim_map, so their tags always fell back to public.

Fix 2 (tag passthrough) -- place_detail.py: add wikivoyage to both nominatim
extratags whitelists, mirroring the existing wikipedia/wikidata lines. The
rewriter (classify_wiki_link / build_kiwix_url / rewrite_wiki_link) and the
_enrich_wiki_links loop were already source_type-generic and covered wikivoyage;
the only missing link was that the nominatim parser dropped the wikivoyage tag
before enrichment ever saw it. No rewriter refactor was needed.

Tests (test_place.py):
- test_catalog_url_requests_full_library: the OPDS fetch URL contains count=-1.
- test_wikivoyage_tag_rewrites_to_local: a wikivoyage OSM tag for a mirrored
  article rewrites to https://wiki.echo6.co/content/wikivoyage_en_all_maxi_<date>/...
  with sources.wiki_rewrites.wikivoyage == "local" (Kiwix mocked).
Full navi-places suite: 15 passed.

Follow-up (separate ops step, not in this PR): prune 3 dangling library.xml
entries on the Kiwix host (wikiloc.com_eng_2026-04_18, meshtastic.org_eng_2026-04_14,
meshtastic.org_eng_2026-04_15) whose ZIM files are absent; kiwix-serve silently
skips them.

Note: Twin Falls (R/121355) itself has no OSM wikivoyage tag, so it still will not
get a local Wikivoyage link from tag rewrite -- that needs the separate name-based
discovery feature (discover_wikivoyage_article stub).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 06:23:24 +00:00
..
config Add navi-contacts service (extraction #3) 2026-05-22 10:59:37 -06:00
deploy Add navi-offroute service (extraction #8 — final) (#10) 2026-05-22 23:30:43 -06:00
scripts Commit reproducible MVUM ingest script (P4) (#25) 2026-05-25 20:36:41 -06:00
services wiki_rewrite: paginate fully + extend rewriter to wikivoyage 2026-05-26 06:23:24 +00:00
shared shared: promote dem.py to shared/ (prep for navi-offroute) (#9) 2026-05-22 22:42:47 -06:00
.gitignore Initial scaffold: navi-backend + navi-traffic (extraction #1) 2026-05-21 22:26:50 -06:00
LICENSE Initial scaffold: navi-backend + navi-traffic (extraction #1) 2026-05-21 22:26:50 -06:00
pyproject.toml feat(offroute): numba A* with anisotropic Tobler, exponentially-inflated cost grid (combined #17+#18) 2026-05-25 15:15:36 +00:00
README.md Add navi-offroute service (extraction #8 — final) (#10) 2026-05-22 23:30:43 -06:00

navi-backend

Monorepo of small, single-responsibility HTTP services extracted from the recon codebase as part of the recon ↔ Navi decoupling project. Each service owns a slice of the /api/* surface that navi.echo6.co depends on, runs behind the existing Caddy/Authentik edge, and is fronted by the navi.echo6.co nginx vhost.

See HANDOFF-recon-navi-decoupling-v3.md for the full plan. This repo is extraction #1: navi-traffic.

Layout

navi-backend/
├── shared/                  # cross-service helpers, imported by every service
│   ├── auth.py              # get_user_id(req), require_auth decorator (Authentik header)
│   └── admin_info.py        # build_info_response(), mask_key(), time_dependency()
├── services/
│   └── navi_traffic/        # extraction #1 — TomTom traffic tile proxy (:8421)
│       ├── app.py           # Flask factory (create_app) + gunicorn entry
│       ├── traffic.py       # /api/traffic/flow/<z>/<x>/<y>.png  (ported from recon)
│       ├── admin.py         # /api/admin/navi-traffic/info  (§4.5 admin convention)
│       └── tests/
└── deploy/
    ├── systemd/navi-traffic.service
    └── nginx/navi-traffic.conf.snippet

Service directories use an underscore (navi_traffic) so they're importable Python packages; the service name stays navi-traffic (hyphen) in systemd, nginx, and the admin-info service field.

Setup

Single workspace, single virtualenv:

python -m venv .venv
.venv/bin/pip install -e .

Test

.venv/bin/pytest services/navi_traffic/tests/ -v

Run (local)

TOMTOM_API_KEY=... .venv/bin/gunicorn 'services.navi_traffic.app:create_app()' \
    --bind 127.0.0.1:8421 --workers 2

Run (local) — navi-geo (extraction #6)

.venv/bin/pytest services/navi_geo/tests/ -v

# All paths/URLs are env-overridable (see deploy/env/navi-geo.env.example).
# No secrets — landclass is HTTP-delegated to navi-landclass (:8424).
.venv/bin/gunicorn 'services.navi_geo.app:create_app()' \
    --bind 127.0.0.1:8426 --workers 2

navi-geo serves /api/geocode, /api/reverse?lat=&lon=, and the reverse enrichment bundle /api/reverse/<lat>/<lon> (Central's 9-key contract). All public. The reverse bundle fans out to Photon, the SpatiaLite timezone DB, navi-landclass (HTTP), and the planet-DEM PMTiles — each degrading to null independently, never 5xx.

Run (local) — navi-admin (extraction #7)

.venv/bin/pytest services/navi_admin/tests/ -v

# No secrets — read-only HTTP fan-out over localhost (see
# deploy/env/navi-admin.env.example). Owns no DB.
.venv/bin/gunicorn 'services.navi_admin.app:create_app()' \
    --bind 127.0.0.1:8427 --workers 2

navi-admin is the fleet admin front door: /api/admin/fleet fans out to every navi-* service's localhost /api/admin/<svc>/info + recon's /api/health (merged, never 5xx — failures land in errors[]); /api/admin/recon/info wraps recon's health into the uniform shape; /api/admin/navi-admin/info self-describes. All @require_auth. The per-service admin endpoints stay localhost-only; this is the single edge-exposed admin surface (needs a Caddy @authed_api edit — see deploy/caddy/navi-admin.caddy.notes.md).

Run (local) — navi-offroute (extraction #8)

.venv/bin/pytest services/navi_offroute/tests/ -v

# All paths/URLs env-overridable (deploy/env/navi-offroute.env.example).
# No secrets — PADUS via libpq peer-auth (dbname=padus). DEM via shared/dem.py.
# Needs osmium-tool on the host + scikit-image/rasterio in the venv.
.venv/bin/gunicorn 'services.navi_offroute.app:create_app()' \
    --bind 127.0.0.1:8428 --workers 2 --timeout 130

navi-offroute serves POST /api/offroute (off-network effort-based routing — in-Python least-cost path over a DEM/friction/barriers/trails/MVUM cost grid, stitched to the road network via Valhalla) and GET /api/mvum (Motor Vehicle Use Map road/trail access lookup). Both public. The ^~ /api/offroute nginx block needs a long proxy_read_timeout (130s); routes can take ~2 min.

The admin-info convention (§4.5)

Every service exposes GET /api/admin/<service-name>/info, gated by require_auth, returning a uniform shape: service, version (git SHA), port, config, env (names + masked values), dependencies (upstream health checks), filesystem, runtime (uptime / request count / last error). No aggregator — a future admin panel fans out to each service in parallel.