* MVUM Layer 0: spatial index foundation
Additive only — no routing logic, no response-format, no Valhalla changes.
- mvum.py: add MVUMSpatialIndex. Loads mvum_roads + mvum_trails from navi.db (read-only),
decodes the pure-WKB shape blobs with shapely, builds a shapely.strtree.STRtree, and
keeps a parallel list of full feature records (all columns minus the blob, plus the
parsed geometry) with a by_id lookup. Exposes query_bbox(min_lat,min_lon,max_lat,max_lon)
and query_buffered_line(coords, tolerance_m) returning candidate records (coarse bbox +
buffer; full parallelism filter is a TODO for PR-B). Reports road_count, trail_count,
bbox, build_time_seconds, memory_estimate_mb (RSS delta).
- app.py: build the index once per process (singleton) at service init; stored on
app.config[MVUM_SPATIAL_INDEX]. Failure is logged and degrades to None.
- admin.py: GET /api/admin/mvum-spatial/info (Authentik-gated, read-only) returning the
counts/bbox/build-time/memory stats.
- tests/test_mvum_spatial.py: index loads, query_bbox returns Boise-area features,
query_buffered_line returns a feature, admin endpoint returns counts.
Diagnostic before coding (read-only): roads_with_shape=150568/null=68,
trails_with_shape=21995/null=6746 (green), shape blobs are pure WKB MultiLineString.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix lat-aware buffer + count WKB parse failures
- query_buffered_line: replace the static _DEG_PER_M with _buffer_degrees_for_meters(),
which scales longitude degrees by cos(lat) and uses the larger lat/lon equivalent;
buffer at the polyline avg latitude. Early-return [] for empty coords.
- _load_table: count WKB parse failures and logger.warning once per table when > 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_route_auto now probes every eligible candidate (eligibility filter unchanged) and
picks the one with the smallest summary.total_effort_minutes, instead of returning the
first that succeeds. Fixes the case where a wilderness start + on-road end fell through
to foot and computed the entire network leg at foot pace. selected_mode = winning
candidate; ties keep AUTO_MODE_PRIORITY order; all-fail still returns the last error.
Wilderness leg still always foot (unchanged).
Per-leg breakdown: summary now carries wilderness_minutes + network_minutes (mirrors the
existing wilderness_effort_minutes/network_duration_minutes) in _build_response (A/B/C)
and _route_D. Frontend DirectionsPanel shows a transition badge "Auto: Foot Xmin +
<mode> Ymin" when wilderness_minutes>0 and selected_mode!=foot, else the existing
"Auto chose <mode>".
Tests: add min-time pick + per-leg breakdown; update probe-all assertions (no early
return) and make the selected-mode test time-based.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rename mode constants: mtb->2w, atv->4w
Rename the OFFROUTE travel-mode identifiers mtb->2w and atv->4w across backend and
frontend. MVUM vehicle-access classes/columns (atv, motorcycle) are a separate
vocabulary and are left untouched. UI labels (MTB/ATV) and the cost.py __main__ demo
local variables (cannot be digit-initial identifiers) are unchanged.
Backend: MODE_PROFILES, MODE_TO_COSTING, MODE_TO_VALID_HIGHWAYS, AUTO_MODE_PRIORITY,
_MODES_* sets, CATEGORY_ELIGIBLE_MODES, route()/compute_cost*/_pathfind_wilderness
Literals, VALID_MODES, and tests. offroute_route.py adds a backward-compat shim mapping
legacy mode=mtb->2w and mode=atv->4w before validation so bookmarked URLs still work.
Frontend: store.js routeMode doc, DirectionsPanel TRAVEL_MODES ids + SELECTED_MODE_LABEL
keys, Panel TRAVEL_MODES ids, ManeuverList network_mode->verb map keys, api.js jsdoc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix user-visible labels: MTB->2W, ATV->4W + grep cleanup
Update the user-facing travel-mode labels to match the renamed ids:
- DirectionsPanel TRAVEL_MODES (2w -> "2W", 4w -> "4W") + SELECTED_MODE_LABEL values.
- Panel TRAVEL_MODES labels.
- Stale test comment: mode=mtb -> mode=2w.
Grep pass over frontend/src + backend/services/navi_offroute found no remaining
mode-identifier string literals to rename. Residual hits are all out-of-scope: MVUM
vehicle-access vocabulary (mvum.py, /api/mvum output, single-quoted test fixtures), the
cost.py __main__ demo locals, the offroute_route.py back-compat shim, and comments
referencing the historical names.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A single noisy DEM cell could fabricate a huge fake slope and make an edge
unconditionally impassable, forcing the pathfinder to route around passable terrain.
Replace the hard cliff (|grade|>max_grade -> skip) with a smooth exponential penalty:
no penalty up to max_grade, then base_time *= exp(overshoot * SLOPE_PENALTY_SCALE);
only grades whose penalty exceeds SLOPE_PENALTY_CAP (true bad data / vertical) are
dropped. Routing can now see through noisy cells while still strongly avoiding real
cliffs. Penalty only raises edge cost, so the heuristic stays admissible.
2 tests: smooth penalty traverses a >max_grade gap (finite, raised cost) yet routes
around / drops a past-cap grade; exactly-at-threshold grade incurs no penalty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace MCP_Geometric in _pathfind_wilderness with a numba-jit anisotropic A* (new
astar.py): signed-slope speed (climbing != descending; tobler peaks at -0.05), hard
cliff, per-edge avg context multiplier, trail-takes-both via 256-entry lookup, per-edge
barriers (strict/pragmatic/emergency), multi-goal A* (first popped wins) with admissible
distance/base-speed heuristic. New compute_cost_multiplier_grid (slope-free context
multiplier) + exponential inflation (sigma=1.8; inf->HARD=50*p95 for blur, inf re-imposed).
numba>=0.59 added (numba 0.65.1).
fix: wilderness leg is always foot effort; mode parameter reserved for future flexibility.
_pathfind_wilderness keeps the mode param (threaded from _route_A/B/C) but hardcodes
cost_mode=foot for the cost grid, trail friction, speed function, base speed, and max
grade. Off-trail math for MTB/ATV/vehicle is not well-grounded and real-world wilderness
traversal is foot regardless (push the bike, walk past where the vehicle stops). User mode
still drives entry-point eligibility (query_radius highway filter) and Valhalla network
costing. Matches the original pre-#17 design: wilderness ALWAYS uses foot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old ST_DWithin(50km) scan returned ~226k candidate points near dense areas before
sorting (~3-10s/call). Replace with PostGIS k-NN ordering: ORDER BY geom::geography <->
point LIMIT k, which the existing GiST index on (geom::geography) walks nearest-first and
stops after K rows. SELECT still computes ST_Distance AS distance_m so callers see real
meters. radius_km is kept as a Python soft cap applied after fetch (drops rows beyond it),
preserving the caller expanded-radius fallback (now effectively a no-op).
2 tests: SQL contains <-> + LIMIT and no ST_DWithin; radius_km soft cap filters beyond-cap rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three wilderness-scenario guards used table_exists() OR get_entry_point_count()==0
to check the index is non-empty — a full SELECT COUNT(*) that scanned the entire
2.94M-row table (~9-73s depending on load/cache). Add EntryPointIndex.has_entry_points()
using SELECT EXISTS (SELECT 1 ... LIMIT 1), which short-circuits at the first row, and
swap it into _route_A/_route_B/_route_C. get_entry_point_count() kept for admin-info/tests.
3 new tests: table missing -> False, empty -> False, rows -> True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align spatial eligibility with Valhalla's actual /locate vocabulary:
- road grade lives in classification.classification (8-value enum incl service_other,
which was missing from PAVED_HIGHWAY_CLASSES); track/path/footway are NOT grades,
they live in classification.use.
- _locate_on_network now also returns use (defensive .get chain; None in fallback).
- Renamed TRACK/PATH_HIGHWAY_CLASSES -> TRACK_USE_VALUES/PATH_USE_VALUES; atv/mtb now
match on use, vehicle still on the paved grade.
- 4 tests: service_other->vehicle, use=track->atv/mtb/foot, use=footway->mtb/foot,
none->foot.
Live-verified: the Boise end point (43.626,-116.215, service_other) now yields
[atv,mtb,vehicle,foot] instead of [foot]; Auto intersection picks vehicle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #10s _spatial_eligible_modes read edge.get("road_class"), but Valhalla /locate
returns no class without verbose, and even with verbose=true the class is NOT named
road_class — it lives at edge.classification.classification (lowercase, e.g.
"secondary"). Without it, every paved/track/path gate failed and Auto collapsed to
foot for untyped endpoints.
Fix: add verbose=true to the /locate body and read the class via a defensive
edge.get("edge",{}).get("classification",{}).get("classification") chain, kept under
the same road_class key so downstream eligibility is unchanged. Live-verified against
Valhalla: (43.621,-116.205) -> road_class=secondary, snap 5.0m, PAVED=True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the 3-tier vehicle-only discriminator with a per-endpoint eligible-mode-set
intersection:
- CATEGORY_ELIGIBLE_MODES maps OSM key:value (and key:* wildcards) to eligible modes;
_eligible_modes_from_category resolves a hint, or None when untyped.
- Untyped endpoints fall back to _spatial_eligible_modes: parallel Valhalla /locate
(auto/pedestrian/bicycle) + 3-tier snap/road-class rules (vehicle needs paved, and
paved+flat in the 5-100m grace zone; atv paved/track; mtb paved/track/path; foot always).
- _route_auto intersects both endpoints eligible sets, probes AUTO_MODE_PRIORITY within
it, and adds selected_mode + selected_mode_set. Both untyped endpoints resolve in parallel.
- _locate_on_network now returns road_class.
- api_offroute accepts optional start_category/end_category and forwards them (auto only).
Frontend: requestOffroute takes startCategory/endCategory; computeRoute passes
routeStart/routeEnd.category; startDirections preserves place.category.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vehicle is pure Valhalla road routing — Valhalla snaps endpoints to the nearest
road automatically, so the off-network classifier (OFF_NETWORK_THRESHOLD_M) is
irrelevant for it. Add an early branch in route() that sends vehicle straight to
_route_D_network_only, instead of bandaiding the threshold. Foot/MTB/ATV keep the
gate so users can intentionally pin backcountry points; Auto inherits the skip via
its recursive mode=vehicle probe. Threshold stays at 10.
Frontend: boundary-mode chips now show for all modes except Drive (vehicle).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When mode="auto", OffrouteRouter._route_auto() probes AUTO_MODE_PRIORITY
(vehicle -> atv -> mtb -> foot) and returns the first mode whose network can
serve the route, tagging the result with selected_mode. route() already errors
when a mode cannot reach an endpoint, so the first status==ok is the most
road-capable feasible mode. Adds 4 isolation tests (route monkeypatched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
navi-offroute (:8428, extraction #8) was missing from fleet.py's SERVICES list,
so it never appeared in /api/admin/fleet. Add the one (name, port) row — the
single source of truth that build_fleet, dependency_summaries, and the
self-info fanned_services all derive from. 8427/navi-admin is the aggregator
itself (self-poll), correctly not in its own fan-out.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-REWRITE — the LAST navi→recon coupling. navi-places now
decides "is this wiki article in the local Kiwix mirror?" in-process instead of
HTTP-calling recon's /api/wiki-rewrite. Intra-process swap, no nginx changes.
Mirrors decouple #4-READ (which moved wiki_index.db reads the same way).
- NEW services/navi_places/wiki_rewrite.py: verbatim port of recon's
lib/wiki_rewrite.py. Only adjustments: setup_logging -> stdlib logging;
KIWIX_BASE -> NAVI_KIWIX_BASE_URL env; KIWIX_PUBLIC_BASE -> NAVI_KIWIX_PUBLIC_BASE
env; cache DB -> NAVI_WIKI_CACHE_DB (default /var/lib/navi-backend/wiki_cache.db,
auto-created); + a reset() to match the place_cache/wiki_index per-worker pattern.
No logic changes — same classify, same lazy hourly catalog refresh, same HEAD
timeout, same status values (local/public/original), same cache semantics.
- place_detail.py: _enrich_wiki_links_via_http -> _enrich_wiki_links; calls
wiki_rewrite.rewrite_wiki_link(tag,value) (TUPLE) and unpacks it, replacing
the dict-returning HTTP client. Import + docstrings updated.
- app.py: wiki_rewrite.reset() per worker/test (alongside place_cache/wiki_index).
- DELETE services/navi_places/wiki_rewrite_client.py (HTTP shim dead).
- tests: the 2 wiki-rewrite tests now monkeypatch the local
wiki_rewrite.rewrite_wiki_link (tuple) instead of the deleted client.
Recon's endpoint stays live but unused until PR-B (safe co-existence).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-READ. navi-places now reads its own wiki_index.db directly
(NAVI_WIKI_INDEX_DB) instead of HTTP-calling recon's /api/wiki-enrich — same
pattern it already uses for place_cache.db. The 2.1GB DB was copied to
/var/lib/navi-backend/wiki_index.db out-of-band (5,061,763 rows verified).
- NEW services/navi_places/wiki_index.py: verbatim port of recon's
lookup_wiki_index + _get_wiki_index_db, reading NAVI_WIKI_INDEX_DB, mirroring
place_cache.py's db_path()/lazy-conn/reset() pattern. Returns the same
{wiki_summary, wiki_population, wiki_url, wikivoyage_url} shape /api/wiki-enrich
did, so it's a drop-in for the HTTP client.
- place_detail.py: _enrich_with_wiki_via_http -> _enrich_with_wiki_index; call
wiki_index.lookup() instead of wiki_client.enrich_via_recon(); docstrings.
- app.py: wiki_index.reset() per worker/test (alongside place_cache.reset_cache()).
- admin.py: drop the recon-wiki-enrich dependency probe; add NAVI_WIKI_INDEX_DB
env + a read-only filesystem entry. (recon-wiki-rewrite probe kept — separate
decouple.)
- DELETE wiki_client.py (fully replaced).
- test_place.py: convert the wiki test from a monkeypatched HTTP client to a
hermetic tmp wiki_index.db.
Internal localhost migration — no nginx/edge involvement. recon's /api/wiki-enrich
stays live until PR-B (deploy PR-A first so nothing calls the route after removal).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of the overture-import relocation. The Overture Places ETL moves from
recon (where it produced data nothing in recon consumes) to navi-backend (the
side that owns the consumer, navi-places). Additive: recon's copy stays live
until PR-B; this just establishes the navi-side copy + deps + docs.
- scripts/overture_import.py: verbatim port of recon's script (recon master
879df84). The ONLY non-verbatim change is the line-8 docstring usage hint,
swapped from `/opt/recon` + venv to the navi-backend path + .venv.
- pyproject.toml: add `duckdb>=1.5` (recon runs 1.5.2; psycopg2-binary already
present). It's the only new dep.
- scripts/README.md: document the manual-only trigger + invocation.
Source release is pinned in-code: OVERTURE_RELEASE = '2026-04-15.0'.
No tests (ETL; none on recon either). Per cleanup #29, the script has no lib/
imports — fully self-contained (stdlib + duckdb + psycopg2).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of the 2-PR whoami migration. Net-new, additive endpoint in navi-admin
matching recon's existing handler shape exactly. Recon's handler stays live in
this PR; once nginx routes /api/auth/whoami to :8427 (out-of-band) and recon's
handler is removed (PR-B), navi-admin is the sole owner.
- New services/navi_admin/auth_route.py with its own blueprint (navi_admin_auth):
GET /api/auth/whoami reads X-Authentik-Username, returns {authenticated,
username}. NOT @require_auth — it's the "am I logged in?" check, must answer
the unauthenticated case (mirrors recon).
- app.py: register the new blueprint (2 lines).
- test_auth.py: header-present + header-absent cases.
Kept in its own blueprint/file so admin_route.py's "all routes @require_auth"
invariant stays true. recon and nginx untouched (additive only).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Matt's directive that navi-* should not call any /api/* on recon.
navi-admin was the only navi service doing so (polling recon's /api/health
and surfacing it in /api/admin/recon/info + the /api/admin/fleet fan-out).
navi-admin is now the navi-only fleet view; recon has its own dashboard for
recon-pipeline health.
- admin_route.py: delete the /api/admin/recon/info handler; drop the recon
config entry + RECON_HEALTH_URL/RECON_REPO_PATH env entries from
/api/admin/navi-admin/info; refresh docstrings.
- fleet.py: remove recon constants, recon_health_url/recon_repo_path/
recon_git_sha/wrap_recon_health, the now-unused shared.git_sha import, and
the recon arms in build_fleet + dependency_summaries. /api/admin/fleet now
reports only the 6 navi-* services.
- tests: drop the 2 recon/info tests + recon scaffolding; strip recon
assertions from fleet + self-info tests. 12 relevant tests pass (10 admin
+ 2 git_sha).
shared/git_sha.py KEPT unchanged — it's a generic git_short_sha(path) helper
used by every service's create_app(), not recon-specific.
RECON_HEALTH_URL + RECON_REPO_PATH in /etc/navi-backend/navi-admin.env are now
dead — flagged for out-of-band post-merge cleanup.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-offroute service (extraction #8 — the last one)
Faithful port of recon's /api/offroute (POST) + /api/mvum (GET) and the
runtime offroute modules into a new :8428 service. Closes the loop: after
this, navi-frontend talks only to navi-backend.
Ported: router.py (OffrouteRouter, EntryPointIndex, 4 route strategies,
in-Python MCP_Geometric least-cost path, Valhalla integration, per-request
osmium extract), mvum.py (MVUMReader over navi.db), cost.py, friction.py,
trails.py, and barriers.py (runtime BarrierReader/WildernessReader only).
NOT ported (per Phase A §3/§15): prototype.py (dead at runtime), barriers.py
build_*_raster (offline GDB→raster prep). DEM imported from shared/dem.py
(PR #9), not duplicated.
Behaviour-faithful changes: hardcoded paths/URLs → env vars; the
profile.offroute.* config (osm_pbf_path/postgis_dsn/densify_interval_m) →
dedicated env vars (router drops deployment_config). Both routes public (no
auth, matching recon). PADUS via libpq peer-auth DSN (dbname=padus) — NO
secret. Owns no DB.
15 hermetic tests (offroute validation + mocked-router shape + close-always;
fixture-SQLite MVUM roads/trails/fallback/null; admin auth + no-secrets +
probe shape). Full suite 119 passed / 1 skipped. Adds scikit-image + rasterio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* navi-offroute: PR #10 review cleanups (4 faithful-port deviations)
1. trails.py — drop recon-era "Run the Phase B rasterization script"
reference from the not-found error (confusing in navi-offroute context).
2. friction.py — add FileNotFoundError-before-rasterio-open check to
match barriers/trails consistency.
3. mvum.py — remove dead try/except shapely import + warnings.warn at
2 sites (shapely is a hard pyproject dep; the fallback was unreachable).
4. router.py — declare psutil in pyproject, drop the silent fallback;
the MEMORY_LIMIT_GB safety check was silently disabled in prod.
Adds test_friction_reader_raises_file_not_found_when_missing (16 navi-offroute
tests; full suite 120 passed / 1 skipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure refactor, no behavior change. Moves services/navi_geo/dem.py to
shared/dem.py (verbatim logic + env override; only docstring + location
changed) and re-points navi-geo's two imports (geo_route.py, admin.py) to
`from shared.dem import ...`.
Per extraction-8-phase-a.md §5/§13.1: navi-offroute (#18) needs the same
DEMReader, so a single source of truth in shared/ beats a third copy. Second
shared/ promotion after PR #7 round-2's shared/git_sha.py; navi-offroute will
`from shared.dem import DEMReader` directly.
Adds shared/tests/test_dem.py (dem_path default + NAVI_DEM_PMTILES override).
navi-geo behavior unchanged (test_reverse_bundle mocks geo_route._DEM, agnostic
to DEMReader's location). Full suite: 104 passed / 1 skipped (+2).
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #6 round-1 fixup #3 wired netsyms.health() (COUNT + DISTINCT on
35 GB) into _netsyms_fs_entry, adding >3s latency to cold admin-info
calls. navi-admin's fleet fan-out (3s timeout) caught it after #7
deploy. Reverting to the cheap _file_entry shape; deleting health()
per the no-dead-code rule (the original fallback option).
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-admin service (extraction #7)
Net-new fleet admin aggregator on :8427 — no port from recon (recon has no
/api/admin route; Phase A §3). Three @require_auth routes:
GET /api/admin/fleet fan-out to all 6 navi-* /api/admin/<svc>/info
+ recon /api/health, merged; never 5xx
(failures land in errors[])
GET /api/admin/recon/info recon /api/health wrapped in the info shape
GET /api/admin/navi-admin/info self-describe
Fan-out forwards the caller's X-Authentik-Username so the @require_auth
upstreams accept it; per-service admin endpoints stay localhost-only (this is
the single edge-exposed admin surface). Service discovery: hardcoded list in
fleet.py (Option B). No secrets, no DB.
Deploy artifacts (NOT applied here): navi-admin.env.example, systemd unit,
nginx ^~ /api/admin snippet, and deploy/caddy notes for the @authed_api edit
(first Caddy change since #2).
12 hermetic tests (fleet happy-path, per-service timeout/500 → errors[],
auth-header forwarding, recon-down degraded-not-5xx, self-info no-secrets,
auth-required). Full monorepo suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #7 review fixes
1. Symmetric degraded-entry handling in fleet.build_fleet — every
probed service now appears in `services` with a uniform degraded
dict on failure (matches recon's existing pattern), AND in errors[].
Operators see "everything I tried + which broke" consistently.
2. Catch ValueError specifically in _get_json — non-JSON 200 responses
now surface as `error: 'invalid JSON'` instead of opaque 'ValueError'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #7 review fixes (round 2)
1. Unified degraded shape: wrap_recon_health calls _degraded_entry on
failure — no more runtime.status vs runtime.recon_status asymmetry.
Every probed service has the same shape on failure
(runtime.status == 'unreachable'). recon-specific runtime fields
(recon_status/recon_uptime/pipeline) remain only on the success path.
2. DRY'd git short-SHA helper into shared/git_sha.py — was duplicated in
7 service app.py files + fleet.recon_git_sha. One implementation,
one place to fix when behavior changes. Adds shared/tests (testpaths
now includes "shared").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-geo service (extraction #6)
Faithful port of recon's geocode/reverse family to a new :8426 service:
GET /api/geocode?q=&limit=&lat=&lon=&zoom= Photon-first ranked search
GET /api/reverse?lat=&lon= reverse geocode (Photon)
GET /api/reverse/<lat>/<lon> reverse enrichment bundle (Central)
Ported modules: geocode.py (engine), netsyms.py (address SQLite), dem.py
(planet-DEM reader), address_book.py (reader copy), and the three handlers +
four bundle helpers from netsyms_api.py. All three routes public, behaviour-
identical to recon.
Behaviour-changing edges (both pre-decided in Phase A/B, called out in the PR):
- landclass: in-process call replaced with HTTP GET to navi-landclass :8424,
reading .summary (the same most-specific unit-name string). First navi→navi
edge after landclass itself.
- hardcoded paths/URLs → env vars (PHOTON_URL, NAVI_NETSYMS_DB,
NAVI_TIMEZONE_DB, NAVI_DEM_PMTILES, NAVI_ADDRESS_BOOK_YAML,
NAVI_LANDCLASS_URL); rerank trace log opt-in (NAVI_GEO_RERANK_TRACE_LOG,
default off — recon always wrote /tmp).
No secrets in this service: PADUS_DB_* disappears because landclass is HTTP-
delegated (Phase A §10). Address book uses Option B (shared-file read), the
same pattern navi-contacts already uses.
Bundle 9-key contract preserved exactly (name/city/county/state/country/
postal_code/timezone/landclass/elevation_m), same null-on-component-failure
semantics, same in-memory TTLCache(10_000, 86_400) per worker.
Tests: 28 passing, 1 skipped (real timezone DB, off-box). Ported the 9 recon
reverse-bundle tests + added the HTTP-landclass coupling tests + hermetic
geocode reranker/intent-classifier tests (recon's geocode_test.py was a live
smoke test). Adds usaddress/rapidfuzz/cachetools/shapely/numpy/Pillow/pmtiles
to deps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #6 review fixes
1. Rename geocode._setup_trace_logger → setup_trace_logger (public hook)
2. Hoist `import requests as http_requests` to module level in geo_route.py
3. Wire netsyms.health() into admin.py (enriches the netsyms filesystem entry
with row_count/file_size_bytes/indexed_countries; no shared-builder change)
4. Fix misleading LANDCLASS_TIMEOUT_S comment (recon had no timeout)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_places/ on :8425 — the heaviest extraction. Ports recon's
/api/place family with the two wiki dependencies decoupled to HTTP.
Routes (public, mirroring recon):
GET /api/place/<osm_type>/<int:osm_id> (Nominatim -> Overpass fallback + enrich)
GET /api/place/wikidata/<wikidata_id> (Wikidata entity)
-> 200 / 400 / 404 / 502, same response shapes as recon.
Enrichment chain (recon order): Overture (PostGIS) -> Google Places -> wiki
rewrite -> wiki index. The two wiki paths are now HTTP to recon (the 2.1 GB
wiki_index.db and Kiwix/wiki_cache stay in recon — see [[reference-echo6-edge-topology]]):
- wiki_client.enrich_via_recon -> recon /api/wiki-enrich (PR #8) [has_kiwix_wiki]
- wiki_rewrite_client.rewrite_via_recon -> recon /api/wiki-rewrite (PR #9) [has_wiki_rewriting]
(per-tag loop over the <=4 wiki extratags, mirroring recon's _enrich_wiki_links)
Both clients degrade gracefully (None / status 'original') on error/timeout.
Data ownership (see [[feedback-navi-backend-data-ownership]]):
- place_cache.db migrates to /var/lib/navi-backend/place_cache.db (env
NAVI_PLACE_CACHE_DB). place_cache.py auto-creates the FULL schema on first
open — place_cache (incl. the google_place_id/google_data/google_fetched_at
columns recon added by migration) + google_api_calls — so a fresh DB serves
both cache_put and the Google daily-cap/cache. WAL, shared module conn.
- overture stays in external PG (OVERTURE_DB_* env), verbatim port of recon's
pool (1,3) + _pool_failed latch, with reset_pool()+probe_db() added.
- wiki_index.db / Kiwix stay in recon, reached via the two HTTP endpoints.
Modules: place_cache.py, overture.py (verbatim+probe), google_places.py
(daily cap via env GOOGLE_PLACES_DAILY_CAP; DB via shared place_cache conn),
wiki_client.py + wiki_rewrite_client.py (HTTP, RECON_BASE_URL default
http://127.0.0.1:8420), osm_categories.py (vendored for humanize_category),
place_detail.py (orchestrator), config.py (feature flags from the vendored
profile via NAVI_PROFILES_DIR), place_route.py, admin.py, app.py.
Feature gates read from the vendored profile (config.py), matching recon:
has_overture_enrichment / has_google_places_enrichment / has_kiwix_wiki /
has_wiki_rewriting — flag off => that enricher is skipped entirely.
admin.py (§4.5): 2 secrets masked (OVERTURE_DB_PASSWORD, GOOGLE_PLACES_API_KEY);
3 dependency probes — overture-postgis (SELECT 1), recon-wiki-enrich and
recon-wiki-rewrite (GET with no params, expect HTTP 400 = route alive).
Deploy: systemd unit (:8425) + nginx snippet (^~ /api/place, no trailing slash,
no proxy_cache; public, no Caddy edit — TIER 2 already through nginx since #2).
Tests (13; recon had zero for this module): validation (400), cache hit (no
upstream), nominatim hit, nominatim-miss->overpass fallback, both-fail 502,
not-found 404, wikidata happy + invalid, overture gated-off (no PG call),
wiki-rewrite-via-http local hit + original pass-through, wiki-enrich-via-http
field merge. Full suite 59. See ../recon_refactor/extraction-5-phase-a.md,
-wiki-enrich-investigation.md, -wiki-rewrite-investigation.md, and PRs #8/#9.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_landclass/ on :8424 — single blueprint, behavior-identical
port of recon's lib/landclass.py + the /api/landclass handler.
GET /api/landclass?lat=&lon= -> { lat, lon, classifications[], count,
is_public, is_private, summary }; 400 on bad/out-of-range lat/lon.
db.py: faithful port of recon's PostGIS module — lazy module-level
psycopg2.pool.SimpleConnectionPool(minconn=1, maxconn=3) from PADUS_DB_* env;
the ST_Intersects query on pad_units (antimeridian filter, acres-ordered,
limit 10); all PAD-US code->label maps verbatim; graceful degradation
(returns [] when PG is unreachable, never raises/500). Adds reset_pool()
(create_app resets per worker) and probe_db() (SELECT 1) for admin health.
No filesystem state — PostGIS is external. No DB-on-disk migration; only the
5 PADUS_DB_* env vars (PADUS_DB_PASSWORD is a real secret, masked in
admin-info via mask_key; the other 4 shown plain). adds psycopg2-binary>=2.9.
Decision — DROPPED the recon `has_landclass` profile-flag gate: the frontend
already gates on its own has_landclass feature flag, and removing the
cross-service config dependency keeps navi-landclass self-contained per the
"only API" rule (the service's existence is the feature being available).
navi-geo coupling (reverse-bundle needs landclass) — per Phase A, recommend
Option B: navi-geo HTTP-calls /api/landclass and reads `.summary` (the
endpoint already returns it); no shared module. Decided when #6 lands.
Tests (8; recon had 2): point-with-coverage -> classification + decoded
labels, ocean point -> empty, bad/missing/out-of-range lat/lon -> 400, PG
down -> graceful 200 empty (not 500), format_summary unit. Full suite 46.
Deploy: systemd unit (:8424) + nginx snippet (one ^~ /api/landclass block, no
proxy_cache; /api/landclass is public so no Caddy edit — TIER 2 already
routes through nginx since extraction #2).
See ../recon_refactor/extraction-4-phase-a.md (which also corrects the handoff:
/mnt/nav/padus/ is source GIS files, NOT a runtime path — this service has no
/mnt/nav dependency, only PADUS_DB_* + PG network access).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_contacts/ on :8423 — two blueprints, behavior-identical
ports of recon's contacts + address_book code.
Routes:
contacts (10, all @require_auth, per-user via X-Authentik-Username):
GET/POST /api/contacts; GET /api/contacts/nearby; GET /api/contacts/deleted;
GET/PATCH/DELETE /api/contacts/<id>;
POST /api/contacts/<id>/restore; POST /api/contacts/<id>/restore-as;
DELETE /api/contacts/<id>/purge
address_book (2, public):
GET /api/address_book/lookup?q= ; GET /api/address_book/list
Data ownership (per Matt's rule — DBs live in navi-backend territory,
auto-create on first run; only massive tilesets stay external):
- contacts.db: env NAVI_CONTACTS_DB (default /var/lib/navi-backend/contacts.db).
ContactsDB auto-creates the schema (table + 5 indexes incl. the partial-
unique Home/Work index) on first open via CREATE ... IF NOT EXISTS — this
is recon's own behavior, ported verbatim. WAL + busy_timeout=5000 preserved.
- address_book.yaml: vendored into config/address_book.yaml (read-only, like
the deployment profiles in extraction #2); path via NAVI_ADDRESS_BOOK_YAML.
Tests (28 new; recon had none for contacts): full ContactsDB CRUD, soft-delete/
restore/restore-as/purge, Home/Work 409 (create + restore conflict), nearby
proximity, search/category filter, per-user partitioning, auth-required, and
DB auto-create; plus address_book ported from recon's test (exact/partial/
case-insensitive/alias/miss/empty/list/hot-reload/missing-file). Full suite 38.
Timestamp fix (diverges from recon on purpose): restore_as builds updated_at
with Python's strftime. recon uses the bare '%Y-%m-%dT%H:%M:%fZ' there — but
Python's %f is microseconds-only (no seconds), so that yields malformed ISO
strings like "...T15:30:123456Z". recon's own update()/soft_delete() use the
correct '%Y-%m-%dT%H:%M:%S.%fZ'. This port uses the correct format in all three
places and adds a regression guard (strptime) in test_restore_as_relabels.
This is a PRE-EXISTING recon bug; we fix it here. The recon-side restore_as
retires with extraction #7 (Jinja /nav-i + /deleted-contacts removal), so the
recon refactor doesn't need to touch it.
Also: shared/auth.py require_auth now sets request.user_id (recon's contract —
the contacts routes read it). Backward-compatible: navi-traffic/navi-config
admin endpoints don't use it.
Deploy artifacts: systemd unit (:8423) + nginx snippet with two ^~ blocks.
NOTE the nginx prefixes are `^~ /api/contacts` and `^~ /api/address_book`
WITHOUT a trailing slash, so the bare `/api/contacts` (list/create) is matched
too — a trailing-slash prefix would miss it and fall through to recon.
See ../recon_refactor/extraction-3-phase-a.md for the route/schema/ownership
analysis (which also corrects the handoff: contacts is /opt/recon/data/
contacts.db, NOT /mnt/nav/navi.db).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendors the three deployment profile YAMLs (home, minimal_pi, regional_pi)
into navi-backend's own repo at config/profiles/. Copied verbatim from
/opt/recon/config/profiles/ post-PR-A, so home.yaml carries the auth block and
the two Pi profiles carry the TODO(matt) logout-host note.
Architectural reason: services in this monorepo should depend on each other
only through their HTTP APIs, never each other's filesystem. navi-config
currently defaults to reading /opt/recon/config/profiles/ — a cross-service
filesystem coupling. With the profiles vendored here, navi-config will be
pointed at this repo's copy via NAVI_CONFIG_PROFILES_DIR in
/etc/navi-backend/navi-config.env at deploy time, so it no longer reaches into
recon's tree.
recon keeps its own /opt/recon/config/profiles/ for its internal use; the two
copies are independent for now. Reconciling them — ideally by having recon
HTTP-fetch /api/config from navi-config rather than re-parsing YAML — is a
tracked follow-up, out of scope for extraction #2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_config/ on :8422, mirroring recon's /api/config contract:
- config_route.py: GET /api/config -> jsonify(get_deployment_config())
with Cache-Control: public, max-age=300 (byte-for-byte recon's response).
- config_loader.py: faithful port of recon lib/deployment_config.py. Reads
RECON_PROFILE (default "home") and NAVI_CONFIG_PROFILES_DIR (default
/opt/recon/config/profiles, so it serves the SAME files recon does during
cutover). yaml.safe_load, module-level cache. Lazy load (vs recon's eager
import-time load) so the module imports cleanly off-VM and a missing
profile surfaces as HTTP 500 at request time rather than a failed import.
- admin.py: /api/admin/navi-config/info per handoff §4.5, require_auth gated.
env values (NAVI_CONFIG_PROFILES_DIR, RECON_PROFILE) are non-secret paths/
names, shown as-is (no mask_key); dependencies=[]; filesystem reports the
active profile path + exists/readable.
- app.py: create_app() factory mirroring navi_traffic, same metrics wiring;
resets the loader cache per instance so each worker/test reloads fresh.
Deploy artifacts: systemd unit (:8422) and an nginx snippet using
`location ^~ /api/config` (the ^~ convention from extraction #1 so the asset
.png/.css regex can't shadow it). No proxy_cache zone — the response is
already cached in-process and via Cache-Control: max-age=300; emits a literal
X-Cache-Status: BYPASS for parity with navi-traffic.
Adds PyYAML>=6 to deps. Tests (services/navi_config/tests): 200 + parsed dict,
Cache-Control header, RECON_PROFILE override, default=home, missing profile=500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
