navi/backend/README.md

115 lines
4.5 KiB
Markdown
Raw Normal View History

# navi-backend
Monorepo of small, single-responsibility HTTP services extracted from the `recon`
codebase as part of the **recon ↔ Navi decoupling** project. Each service owns a
slice of the `/api/*` surface that `navi.echo6.co` depends on, runs behind the
existing Caddy/Authentik edge, and is fronted by the `navi.echo6.co` nginx vhost.
See `HANDOFF-recon-navi-decoupling-v3.md` for the full plan. This repo is
extraction **#1**: `navi-traffic`.
## Layout
```
navi-backend/
├── shared/ # cross-service helpers, imported by every service
│ ├── auth.py # get_user_id(req), require_auth decorator (Authentik header)
│ └── admin_info.py # build_info_response(), mask_key(), time_dependency()
├── services/
│ └── navi_traffic/ # extraction #1 — TomTom traffic tile proxy (:8421)
│ ├── app.py # Flask factory (create_app) + gunicorn entry
│ ├── traffic.py # /api/traffic/flow/<z>/<x>/<y>.png (ported from recon)
│ ├── admin.py # /api/admin/navi-traffic/info (§4.5 admin convention)
│ └── tests/
└── deploy/
├── systemd/navi-traffic.service
└── nginx/navi-traffic.conf.snippet
```
Service directories use an underscore (`navi_traffic`) so they're importable
Python packages; the **service name** stays `navi-traffic` (hyphen) in systemd,
nginx, and the admin-info `service` field.
## Setup
Single workspace, single virtualenv:
```bash
python -m venv .venv
.venv/bin/pip install -e .
```
## Test
```bash
.venv/bin/pytest services/navi_traffic/tests/ -v
```
## Run (local)
```bash
TOMTOM_API_KEY=... .venv/bin/gunicorn 'services.navi_traffic.app:create_app()' \
--bind 127.0.0.1:8421 --workers 2
```
Add navi-geo service (extraction #6) (#6) * Add navi-geo service (extraction #6) Faithful port of recon's geocode/reverse family to a new :8426 service: GET /api/geocode?q=&limit=&lat=&lon=&zoom= Photon-first ranked search GET /api/reverse?lat=&lon= reverse geocode (Photon) GET /api/reverse/<lat>/<lon> reverse enrichment bundle (Central) Ported modules: geocode.py (engine), netsyms.py (address SQLite), dem.py (planet-DEM reader), address_book.py (reader copy), and the three handlers + four bundle helpers from netsyms_api.py. All three routes public, behaviour- identical to recon. Behaviour-changing edges (both pre-decided in Phase A/B, called out in the PR): - landclass: in-process call replaced with HTTP GET to navi-landclass :8424, reading .summary (the same most-specific unit-name string). First navi→navi edge after landclass itself. - hardcoded paths/URLs → env vars (PHOTON_URL, NAVI_NETSYMS_DB, NAVI_TIMEZONE_DB, NAVI_DEM_PMTILES, NAVI_ADDRESS_BOOK_YAML, NAVI_LANDCLASS_URL); rerank trace log opt-in (NAVI_GEO_RERANK_TRACE_LOG, default off — recon always wrote /tmp). No secrets in this service: PADUS_DB_* disappears because landclass is HTTP- delegated (Phase A §10). Address book uses Option B (shared-file read), the same pattern navi-contacts already uses. Bundle 9-key contract preserved exactly (name/city/county/state/country/ postal_code/timezone/landclass/elevation_m), same null-on-component-failure semantics, same in-memory TTLCache(10_000, 86_400) per worker. Tests: 28 passing, 1 skipped (real timezone DB, off-box). Ported the 9 recon reverse-bundle tests + added the HTTP-landclass coupling tests + hermetic geocode reranker/intent-classifier tests (recon's geocode_test.py was a live smoke test). Adds usaddress/rapidfuzz/cachetools/shapely/numpy/Pillow/pmtiles to deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR #6 review fixes 1. Rename geocode._setup_trace_logger → setup_trace_logger (public hook) 2. Hoist `import requests as http_requests` to module level in geo_route.py 3. Wire netsyms.health() into admin.py (enriches the netsyms filesystem entry with row_count/file_size_bytes/indexed_countries; no shared-builder change) 4. Fix misleading LANDCLASS_TIMEOUT_S comment (recon had no timeout) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zvx-echo6 <mj@k7zvx.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 20:29:22 -06:00
## Run (local) — navi-geo (extraction #6)
```bash
.venv/bin/pytest services/navi_geo/tests/ -v
# All paths/URLs are env-overridable (see deploy/env/navi-geo.env.example).
# No secrets — landclass is HTTP-delegated to navi-landclass (:8424).
.venv/bin/gunicorn 'services.navi_geo.app:create_app()' \
--bind 127.0.0.1:8426 --workers 2
```
`navi-geo` serves `/api/geocode`, `/api/reverse?lat=&lon=`, and the reverse
enrichment bundle `/api/reverse/<lat>/<lon>` (Central's 9-key contract). All
public. The reverse bundle fans out to Photon, the SpatiaLite timezone DB,
navi-landclass (HTTP), and the planet-DEM PMTiles — each degrading to `null`
independently, never 5xx.
Add navi-admin service (extraction #7) (#7) * Add navi-admin service (extraction #7) Net-new fleet admin aggregator on :8427 — no port from recon (recon has no /api/admin route; Phase A §3). Three @require_auth routes: GET /api/admin/fleet fan-out to all 6 navi-* /api/admin/<svc>/info + recon /api/health, merged; never 5xx (failures land in errors[]) GET /api/admin/recon/info recon /api/health wrapped in the info shape GET /api/admin/navi-admin/info self-describe Fan-out forwards the caller's X-Authentik-Username so the @require_auth upstreams accept it; per-service admin endpoints stay localhost-only (this is the single edge-exposed admin surface). Service discovery: hardcoded list in fleet.py (Option B). No secrets, no DB. Deploy artifacts (NOT applied here): navi-admin.env.example, systemd unit, nginx ^~ /api/admin snippet, and deploy/caddy notes for the @authed_api edit (first Caddy change since #2). 12 hermetic tests (fleet happy-path, per-service timeout/500 → errors[], auth-header forwarding, recon-down degraded-not-5xx, self-info no-secrets, auth-required). Full monorepo suite green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR #7 review fixes 1. Symmetric degraded-entry handling in fleet.build_fleet — every probed service now appears in `services` with a uniform degraded dict on failure (matches recon's existing pattern), AND in errors[]. Operators see "everything I tried + which broke" consistently. 2. Catch ValueError specifically in _get_json — non-JSON 200 responses now surface as `error: 'invalid JSON'` instead of opaque 'ValueError'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR #7 review fixes (round 2) 1. Unified degraded shape: wrap_recon_health calls _degraded_entry on failure — no more runtime.status vs runtime.recon_status asymmetry. Every probed service has the same shape on failure (runtime.status == 'unreachable'). recon-specific runtime fields (recon_status/recon_uptime/pipeline) remain only on the success path. 2. DRY'd git short-SHA helper into shared/git_sha.py — was duplicated in 7 service app.py files + fleet.recon_git_sha. One implementation, one place to fix when behavior changes. Adds shared/tests (testpaths now includes "shared"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zvx-echo6 <mj@k7zvx.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 21:21:00 -06:00
## Run (local) — navi-admin (extraction #7)
```bash
.venv/bin/pytest services/navi_admin/tests/ -v
# No secrets — read-only HTTP fan-out over localhost (see
# deploy/env/navi-admin.env.example). Owns no DB.
.venv/bin/gunicorn 'services.navi_admin.app:create_app()' \
--bind 127.0.0.1:8427 --workers 2
```
`navi-admin` is the fleet admin front door: `/api/admin/fleet` fans out to every
navi-* service's localhost `/api/admin/<svc>/info` + recon's `/api/health`
(merged, never 5xx — failures land in `errors[]`); `/api/admin/recon/info` wraps
recon's health into the uniform shape; `/api/admin/navi-admin/info` self-describes.
All `@require_auth`. The per-service admin endpoints stay localhost-only; this is
the single edge-exposed admin surface (needs a Caddy `@authed_api` edit — see
`deploy/caddy/navi-admin.caddy.notes.md`).
Add navi-offroute service (extraction #8 — final) (#10) * Add navi-offroute service (extraction #8 — the last one) Faithful port of recon's /api/offroute (POST) + /api/mvum (GET) and the runtime offroute modules into a new :8428 service. Closes the loop: after this, navi-frontend talks only to navi-backend. Ported: router.py (OffrouteRouter, EntryPointIndex, 4 route strategies, in-Python MCP_Geometric least-cost path, Valhalla integration, per-request osmium extract), mvum.py (MVUMReader over navi.db), cost.py, friction.py, trails.py, and barriers.py (runtime BarrierReader/WildernessReader only). NOT ported (per Phase A §3/§15): prototype.py (dead at runtime), barriers.py build_*_raster (offline GDB→raster prep). DEM imported from shared/dem.py (PR #9), not duplicated. Behaviour-faithful changes: hardcoded paths/URLs → env vars; the profile.offroute.* config (osm_pbf_path/postgis_dsn/densify_interval_m) → dedicated env vars (router drops deployment_config). Both routes public (no auth, matching recon). PADUS via libpq peer-auth DSN (dbname=padus) — NO secret. Owns no DB. 15 hermetic tests (offroute validation + mocked-router shape + close-always; fixture-SQLite MVUM roads/trails/fallback/null; admin auth + no-secrets + probe shape). Full suite 119 passed / 1 skipped. Adds scikit-image + rasterio. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * navi-offroute: PR #10 review cleanups (4 faithful-port deviations) 1. trails.py — drop recon-era "Run the Phase B rasterization script" reference from the not-found error (confusing in navi-offroute context). 2. friction.py — add FileNotFoundError-before-rasterio-open check to match barriers/trails consistency. 3. mvum.py — remove dead try/except shapely import + warnings.warn at 2 sites (shapely is a hard pyproject dep; the fallback was unreachable). 4. router.py — declare psutil in pyproject, drop the silent fallback; the MEMORY_LIMIT_GB safety check was silently disabled in prod. Adds test_friction_reader_raises_file_not_found_when_missing (16 navi-offroute tests; full suite 120 passed / 1 skipped). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zvx-echo6 <mj@k7zvx.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 23:30:43 -06:00
## Run (local) — navi-offroute (extraction #8)
```bash
.venv/bin/pytest services/navi_offroute/tests/ -v
# All paths/URLs env-overridable (deploy/env/navi-offroute.env.example).
# No secrets — PADUS via libpq peer-auth (dbname=padus). DEM via shared/dem.py.
# Needs osmium-tool on the host + scikit-image/rasterio in the venv.
.venv/bin/gunicorn 'services.navi_offroute.app:create_app()' \
--bind 127.0.0.1:8428 --workers 2 --timeout 130
```
`navi-offroute` serves `POST /api/offroute` (off-network effort-based routing —
in-Python least-cost path over a DEM/friction/barriers/trails/MVUM cost grid,
stitched to the road network via Valhalla) and `GET /api/mvum` (Motor Vehicle
Use Map road/trail access lookup). Both public. The `^~ /api/offroute` nginx
block needs a long `proxy_read_timeout` (130s); routes can take ~2 min.
## The admin-info convention (§4.5)
Every service exposes `GET /api/admin/<service-name>/info`, gated by `require_auth`,
returning a uniform shape: `service`, `version` (git SHA), `port`, `config`, `env`
(names + masked values), `dependencies` (upstream health checks), `filesystem`,
`runtime` (uptime / request count / last error). No aggregator — a future admin
panel fans out to each service in parallel.