* navi-offroute: Auto classify-once / route-once (kill 4-mode contest)
PR 1 of the Auto rewrite. _route_auto no longer routes all four modes and keeps a
min-time winner; it classifies each endpoint (category map, spatial probe only as
untagged-click fallback), picks the first AUTO_MODE_PRIORITY mode in the eligible
intersection, and routes ONCE. This removes the measured ~3.03s in-town 4-mode
probe (single-mode probing line in journald) -- in-town Auto drops from ~6s toward ~1s.
Trade-off (intentional, PR1): no routing-failure fall-through and no min-time
refinement -- if the capability-picked mode cannot route, the error is returned.
The hybrid path (unchanged here, still gated on the 24km MIN_HYBRID_DISTANCE_KM)
recovers road->offroad plans. Semantic hybrid gate is PR 2.
Scope: router.py (_route_auto contest loop only) + test_offroute.py (contest tests
-> capability-pick tests + new tagged-no-spatial and untagged-spatial-once tests).
_try_hybrid_auto body, the hybrid gate, AUTO_MODE_PRIORITY/MODE_PROFILES, and all
other modules untouched. Full offroute suite: 84 passed.
Design: recon_refactor/auto-rewrite-plan.md (artifacts dir).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Auto: foot-as-last-resort fallback when picked mode fails to route
If the capability-picked mode cannot route, retry foot ONCE (foot always routes
modulo bbox limits) instead of surfacing a wall to the user. On success, ship the
foot route tagged with auto_fallback_from=<picked mode> for the UI; on foot failure,
return the original error. No fallback when the picked mode is already foot.
selected_mode_set still reflects the original capability intersection.
Tests: no_fallthrough -> falls_back_to_foot_on_error; + both-fail returns error;
+ no-fallback-when-picked-is-foot. Full offroute suite: 86 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #34 added per-stage Auto timing logs (single-mode probing took..., hybrid
candidate gathering..., hybrid probing took...) at logger.info level, but
navi-offroute has no logging config, so Python's last-resort handler dropped
everything below WARNING and the lines never reached stderr/journal.
This one-line logging.basicConfig(level=logging.INFO) in app.py opens the gate so
those lines surface in journald, letting us localize the 6-7s baseline Auto
latency on in-town routes. No other changes.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PROBLEM 1 (latency): short in-town Auto routes (Twin Falls->Filer ~7 mi) took 60+
seconds because urban OSM-parking density exploded hybrid candidate evaluation.
Fixes in router.py:
- MIN_HYBRID_DISTANCE_KM 8.0 -> 24.0 (~15 mi): in-town trips never enter hybrid eval
at all -- this alone eliminates the reported latency on Matt's routes.
- HYBRID_MAX_TRAILHEADS 20 -> 8: fewer candidates even on long trips.
- HYBRID_OVERALL_TIMEOUT_S = 6.0: a wall-clock check inside the candidate loop bails
hybrid eval past 6 s (logger.warning) and keeps the single-mode / best-so-far winner.
- HYBRID_EARLY_ABORT_MIN = 30.0: once a candidate beats the single-mode winner by 30+
min, stop probing the rest and ship it.
- Per-stage timing logs (logger.info) in _route_auto / _try_hybrid_auto:
"single-mode probing took Xs", "hybrid candidate gathering: N candidates in Xs",
"hybrid probing took Xs across N tested candidates".
PROBLEM 2 (hybrid render): investigated the missing network polyline. The stated
hypothesis (a hybrid drive leg using a non-"network" segment_type) is DISPROVEN --
_build_hybrid_response emits segment_type=="network" for BOTH the drive and offroad
legs (verified against the live response), the OFFROUTE_NETWORK_LAYER filter matches
it, MODE_COLORS is fully defined, and the store passes data.route correctly. The one
real fragility is the MapLibre color match: if network_mode is ever null the whole
layer can fail to paint (wilderness still draws via its static color -- matching the
exact symptom). Hardened it with ["to-string", ["get","network_mode"]] so a
missing/unknown mode falls through to the blue fallback and the layer always paints.
I could not reproduce the exact missing-leg render headlessly (all backend shapes +
frontend filters are correct), so a Chrome MCP repro is recommended to confirm #2 is
resolved; if a render issue remains it should be diagnosed in-browser.
Tests: hybrid synthetic-trip distances bumped 20 -> 30 km (past the new 24 km gate);
new test_hybrid_early_abort_stops_probing; surface-change integration savings lowered
into the 15-30 min band so both candidate sources are still probed (not early-aborted).
Full offroute suite: 84 passed. npm run build: clean.
PROBLEM 3 (off-road wilderness timeout) is out of scope -- separate follow-up; the
wilderness pathfinder is untouched here.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When PR #30 (wikivoyage name-based discovery) landed, every place cached before
the fix kept returning stale (no-wikivoyage) responses until a manual TRUNCATE of
place_cache. A TTL makes enrichment changes propagate automatically.
- place_cache.py: cache_get now treats a hit older than the TTL as a miss, so the
caller refetches + re-enriches and cache_put overwrites the row (no delete on
read). TTL is NAVI_PLACE_CACHE_TTL_DAYS (default 30), via _ttl_seconds(). Entries
with unknown age (cached_at 0/NULL, e.g. legacy rows) are treated as expired.
- No column migration needed: the schema already has cached_at INTEGER NOT NULL and
cache_put already writes now(). Added an idempotent guard in get_conn anyway
(PRAGMA table_info check -> ALTER TABLE ADD COLUMN cached_at INTEGER DEFAULT 0)
so a hypothetical legacy on-disk DB predating the column self-heals; on the live
DB and fresh DBs it is a no-op since CREATE TABLE already includes cached_at.
Tests (test_place.py): within-TTL hit served from cache (no refetch); past-TTL hit
refetches + refreshes cached_at; NAVI_PLACE_CACHE_TTL_DAYS=1 override expires a
2-day-old entry. Full navi-places suite: 21 passed.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the proven OSMParkingIndex (PR #31) memory-pack pattern to TrailheadIndex,
which the 2026-05-26 memory audit flagged at ~300-450 MB/worker using the old
list[dict]+list[Point] storage.
- mvum_transitions.py: store coords as packed float64 numpy arrays (_lats/_lons)
and attributes as interned lists (_names/_road_classes); build candidate record
dicts lazily via _record(i) in query_trailheads_near_line instead of holding
740k dicts + 740k shapely Point objects. Points are built only to construct the
STRtree, then released. Adds a records property (lazy, for tests/introspection)
and tracks build_time_seconds + memory_estimate_mb (psutil RSS delta) like
OSMParkingIndex. Query logic (coarse STRtree bbox + precise degree-distance
check) unchanged.
- admin.py: GET /api/admin/trailhead/info -> {count, build_time_seconds,
memory_estimate_mb}, mirroring /api/admin/osm-parking/info.
Tests: existing test updated (len(records)==count; the removed _points assertion)
plus a numpy-backing test (_lats/_lons dtype float64, len==count). Full offroute
suite: 83 passed.
Real-DB sanity (not deployed): index loads 740,430 entry points in ~4.8 s using
~285 MB RSS (down from the audit's inferred ~300-450 MB; same ~40% pack ratio as
parking), query returns 317 trailheads on a Redfish Lake corridor.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* MVUM Layer 3b: OSM parking as multi-modal Auto transition candidates
Adds OSM parking lots as a third multi-modal-Auto transition source alongside
MVUM trailheads (3a) and surface-change points (3c), so Auto can suggest
"drive to a parking lot, switch to foot/2w/4w" trips where no MVUM trailhead
exists -- BLM/state land, urban edges, anywhere OSM has parking but the USFS
trailhead layer does not. Backend-only; consumes the already-ingested
/mnt/nav/osm-parking.db read-only (no data-pipeline change).
- mvum_parking.py: OSMParkingIndex (process-wide singleton via load_parking_index)
over a shapely STRtree of parking points, mirroring MVUMSpatialIndex /
TrailheadIndex. Read-only SQLite. Drops access in (private,no,permit) at load.
query_parking_near_line(coords, buffer_m=2000) with the same coarse-bbox +
precise-distance filter as TrailheadIndex. Records carry
{lat, lon, name, road_class="parking", parking_type, access}.
Perf note: the ingest already stored representative_point() in lat/lon, so the
STRtree is built straight from those columns -- parsing the 1.6M WKB blobs at
boot would add minutes for an identical point.
- router.py: _try_hybrid_auto generalized to gather candidates from each AVAILABLE
source (trailhead index if present + surface-change always + parking index if
present) instead of hard-returning when trailhead_index is None, so parking-only
candidates still work. Combined list keeps the existing closest-first sort +
HYBRID_MAX_TRAILHEADS cap. Signature unchanged; record shape already compatible.
- app.py / offroute_route.py: load + inject the OSM parking singleton, mirroring
MVUM_SPATIAL_INDEX / MVUM_TRAILHEAD_INDEX. Failure logs a warning, degrades None.
- admin.py: GET /api/admin/osm-parking/info -> {count, build_time_seconds,
memory_estimate_mb}, mirroring /api/admin/mvum-spatial/info.
- backend/scripts/ingest_parking.py + README-osm-parking-ingest.md: the
data-pipeline ingest lifted to the repo with argparse (--geojsonseq/--db, no
/tmp) + the download/filter/export/ingest/restart refresh recipe.
Tests: test_mvum_parking.py (loads, near-line close-only, private/no/permit
filtered, null-access kept) + test_offroute.py::test_hybrid_consumes_parking_
candidates (parking-only source probed as a leg-1 destination). Full offroute
suite: 82 passed.
Real-DB sanity (not deployed): index loads 1,489,054 usable parking objects
(182,945 access-blocked dropped) in ~12 s using ~950 MB RSS per worker; a Redfish
Lake/Sawtooth corridor query returns 8 lots. The ~950 MB/worker memory cost is
notable -- flagging for review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Numpy-pack OSMParkingIndex coords + lazy records to cut RSS (~950->~570 MB/worker)
Store parking coords as packed float64 numpy arrays (_lats/_lons) and the
attribute columns as interned lists (_names/_parking_types/_accesses), and build
candidate record dicts lazily in query_parking_near_line instead of materializing
1.5M dicts + 1.5M shapely Point objects up front. road_class is the constant
"parking" so it is not stored per row.
Measured on the real /mnt/nav/osm-parking.db (1,489,054 usable rows):
RSS/worker ~950 MB -> ~570 MB (~40%), build ~11 s. Across 2 gunicorn workers that
is ~1.9 GB -> ~1.14 GB.
NOTE: this does NOT reach the ~250 MB originally targeted. The remaining cost is
the shapely STRtree itself: it permanently retains the input geometries
(tree.geometries len == row count), so the transient `del points` does not free
them. Attribution on the real DB: columns-only 137 MB, retained Point objects
+230 MB, STRtree index +110 MB. Reaching ~250 MB would require dropping the
shapely STRtree for a coordinate-only structure (e.g. scipy cKDTree over the
lon/lat arrays), which changes the line-buffer query into a per-vertex radius
query -- a behavior change beyond this fix-up's scope. Flagged for a follow-up.
Tests unchanged except one assertion (`len(idx.records) == idx.count`); full
offroute suite 82 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Places without an OSM wikivoyage tag (the entire current dataset -- 0 of placex
rows carry the tag, vs 32,862 wikipedia) never got a local Wikivoyage link even
when the article exists in the mirror by name. This implements the long-standing
discover_wikivoyage_article stub (a never-finished placeholder ported verbatim
from recon -- not disabled for any flaw) so a place name can resolve to a local
Kiwix article.
- wiki_rewrite.py: discover_wikivoyage_article(name, ...) normalizes the name to
a MediaWiki title and runs it through the existing check_kiwix_has_article
('wikivoyage', ...) path -- same catalog discovery, HEAD probe, and positive
cache as tag rewriting. Returns (kiwix_url, "local") on a HEAD 200, else
(None, None). No public fallback: without an OSM tag we can't confirm a public
Wikivoyage article exists, so a name miss yields no link rather than a guessed
(possibly dead) public URL.
- place_detail._enrich_wiki_links: after the tag-rewrite loop, when extratags has
no wikivoyage value, attempt name-based discovery on result["name"] and, on a
local hit, set extratags["wikivoyage"] + sources.wiki_rewrites["wikivoyage"] =
"local". Tag rewrite always wins when a tag is present (discovery only fills the
gap). Gated by the existing has_wiki_rewriting flag (discovery lives inside that
already-enabled, flag-gated function) -- no new flag / cross-repo profile edit;
the docstring-only has_wiki_discovery flag was never defined in any profile.
Tests (test_place.py): finds-local (HEAD 200 -> local URL + source), 404 ->
no link / no source, and runs-only-when-tag-missing (tag present -> tag rewrite
wins, discovery not consulted). Full navi-places suite: 18 passed.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Twin Falls Wikivoyage investigation surfaced two coordinated gaps that kept
OSM wikivoyage tags from rewriting to local Kiwix URLs even though the
wikivoyage ZIM is loaded and serving.
Fix 1 (pagination) -- wiki_rewrite.py: append ?count=-1 to the Kiwix OPDS catalog
fetch. kiwix-serve's /catalog/v2/entries defaults to the first 10 entries; the
library has 17, so wikivoyage (and other page-2 ZIMs) were never seen by
_discover_zims and never entered _zim_map, so their tags always fell back to public.
Fix 2 (tag passthrough) -- place_detail.py: add wikivoyage to both nominatim
extratags whitelists, mirroring the existing wikipedia/wikidata lines. The
rewriter (classify_wiki_link / build_kiwix_url / rewrite_wiki_link) and the
_enrich_wiki_links loop were already source_type-generic and covered wikivoyage;
the only missing link was that the nominatim parser dropped the wikivoyage tag
before enrichment ever saw it. No rewriter refactor was needed.
Tests (test_place.py):
- test_catalog_url_requests_full_library: the OPDS fetch URL contains count=-1.
- test_wikivoyage_tag_rewrites_to_local: a wikivoyage OSM tag for a mirrored
article rewrites to https://wiki.echo6.co/content/wikivoyage_en_all_maxi_<date>/...
with sources.wiki_rewrites.wikivoyage == "local" (Kiwix mocked).
Full navi-places suite: 15 passed.
Follow-up (separate ops step, not in this PR): prune 3 dangling library.xml
entries on the Kiwix host (wikiloc.com_eng_2026-04_18, meshtastic.org_eng_2026-04_14,
meshtastic.org_eng_2026-04_15) whose ZIM files are absent; kiwix-serve silently
skips them.
Note: Twin Falls (R/121355) itself has no OSM wikivoyage tag, so it still will not
get a local Wikivoyage link from tag rewrite -- that needs the separate name-based
discovery feature (discover_wikivoyage_article stub).
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract surface-category boundaries along the winning single-mode polyline as
additional multi-modal-Auto transition candidates, so Auto can suggest "pull off
where the pavement turns to dirt and switch vehicles" trips even with no MVUM
trailhead nearby. Candidates share the trailhead record shape, so _try_hybrid_auto
consumes them with no restructuring.
Backend-only:
- mvum_surface_change.py: get_surface_change_candidates(coords, valhalla_url) walks
the polyline through Valhalla trace_attributes (action=include, costing=auto,
edge.surface/road_class/use/begin_shape_index/end_shape_index). classify_surface
buckets each edge into PAVED/UNPAVED/TRACK/TRAIL; adjacent edges are grouped into
runs, runs shorter than MIN_STRETCH_M (100 m, measured by haversine along the input
coords) are collapsed to suppress noise, and each surviving category boundary emits
{lat, lon, name: "Surface change: <from>-><to>", road_class}. Capped at 10. Adds an
encode_polyline6 helper (the inverse of the router _decode_polyline method).
- router.py: _try_hybrid_auto concatenates trailheads + surface-change candidates,
then re-sorts by distance to the route and applies the existing
HYBRID_MAX_TRAILHEADS cap. Probing logic unchanged.
Verified trace_attributes on the live Valhalla before coding (returns the requested
edge fields). Two empirically-driven deviations from the spec, flagged:
1. This Valhalla normalizes OSM surface tags into its own enum (paved_smooth/paved/
paved_rough/compacted/dirt/gravel/path/impassable); classify_surface keys on that
enum AND the raw OSM names for robustness.
2. Urban alleys come back as road_class=service_other with surface=paved_smooth, so
the service_other->TRACK rule is gated on a non-paved surface to avoid classifying
paved alleys as tracks.
Tests: test_mvum_surface_change.py (6) -- classify spot-check, paved->unpaved boundary,
sub-100 m noise suppression, uniform-surface empty, encoder round-trip vs the router
decoder, and hybrid integration (both trailhead + surface candidates probed). Full
offroute suite: 77 passed.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auto now also considers "drive in to a trailhead, switch vehicles, continue
on foot/2w/4w" trips and picks one when it is meaningfully faster than the
single-mode winner. Implicit — no new chip; Auto just returns the fastest plan.
Backend:
- mvum_transitions.py: TrailheadIndex (STRtree over trail_entry_points), built
once per process via load_trailheads() (mirrors the MVUMSpatialIndex singleton).
query_trailheads_near_line(coords, buffer_m=2000) with a precise distance filter.
- router.py: _route_auto, after the single-mode probe and only when the winner is
ok AND total_distance_km >= MIN_HYBRID_DISTANCE_KM (8.0), tries hybrids. For each
candidate trailhead near the winning polyline (closest first, capped at 20) and
each (drive, offroad) pair in HYBRID_PAIRS, it routes both legs (annotate_mvum
off) and sums leg times with NO transition cost. A hybrid wins only if it beats
the single-mode winner by >= HYBRID_MIN_TIME_SAVINGS_MIN (15 min); trivial
offroad detours (< HYBRID_MIN_OFFROAD_KM = 0.8 km) are skipped. The winner is
combined into a new "multi" scenario: leg1 features + a kind=transition marker
+ leg2 features; summary carries total_*, per-leg legs[], summed MVUM counts;
selected_mode="hybrid". Each leg is annotated separately.
- app.py / offroute_route.py: load + inject the trailhead index singleton.
Frontend (additive — no api.js signature change):
- DirectionsPanel: per-leg breakdown row for hybrid/multi ("Drive X mi (Ymin)
-> 4W X mi (Zmin) - total Wmin", lucide Repeat between legs); existing Auto
badge still shows.
- MapView: network polylines colored by network_mode (vehicle/auto blue, 4w
orange, 2w green, foot red); transition points rendered as a circle marker with
the lucide Repeat icon + "Switch to <mode>" tooltip; bounds fit skips Points.
Tests: test_mvum_transitions.py — index load, near-line close-only query, short
trip stays single-mode, big-savings hybrid wins, trivial-detour + below-threshold
+ no-trailheads all fall back. 7 new tests; full offroute suite 71 passed.
Note: the DB column is trail_entry_points.highway_class; surfaced as record
"road_class" per the spec.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operational scripts only (no service code, no tests).
- backend/scripts/mvum_backfill.py: cleaned, repo-ready version of the P2/P3 NFS-centerline
backfill. argparse --db-path (default /mnt/nav/navi.db) / --nfs-gdb (default the EDW
Trans_Trail_NFS_Publish.gdb) / --dry-run; no /tmp, no test-DB paths, no prod-refusing
guard (intended for prod, gated by --dry-run + the README snapshot step). Extracts NFS
geometry from the .gdb via ogr2ogr into a cache beside it (no GDAL Python bindings here),
matches null mvum_trails rows by TRAIL_NO+TRAIL_NAME within the forest ADMIN_ORG prefix,
merges segments, writes WKB. Idempotent (UPDATE WHERE shape IS NULL). Prints
rows_attempted/rows_updated/rows_skipped_no_match/rows_skipped_nfs_null_geom.
- backend/scripts/README-mvum-ingest.md: source URLs, initial ingest (ogr2ogr shapes),
refresh, repair (the NULL-shape gap), snapshot-first + stop-service guardrails, and the
produce-vs-consume pointer to mvum.py / mvum_annotate.py / mvum_exclude.py.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend-only. For strict-boundary motorized routes, turn MVUM-closed segments into
buffered Valhalla exclude_polygons so routing actively avoids them. pragmatic/emergency
keep Layer-1 annotate-only behavior; foot is never excluded.
- mvum_exclude.py (new): build_exclude_polygons(start,end,mode,spatial_index,on_date,
boundary_mode) -> GeoJSON Polygon dicts, or None when not applicable (foot / non-strict
/ no index / unmappable mode). Queries the Layer-0 index over a 5km-expanded bbox,
keeps only features closed to the mode (via mvum_annotate._status_for_feature), buffers
each ~15m (lat-corrected), emits one Polygon per part (MultiPolygon split). Caps at 500
with a warning.
- router.py: route() computes self._exclude_polygons once per call (after mode validation;
has boundary_mode), so each Auto candidate probes against its own exclusions. Both
Valhalla /route builders (_route_D_network_only and _valhalla_route) inject
exclude_polygons in array-of-rings form (outer ring per Polygon); omitted when None/empty.
- 6 tests: strict builds polygons, pragmatic/emergency/foot -> None, open not excluded,
1000 closed -> capped at 500 + warning.
No frontend changes (Layer-1 closure warning still fires for residual closures). Wilderness
pathfinder, Layer-0 index, and Layer-1 annotation untouched.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* MVUM Layer 1: per-edge annotation + accurate closure counts
Annotate each network-leg segment with its MVUM access status for the selected mode,
using the Layer-0 MVUMSpatialIndex. No routing-decision or Valhalla changes.
- mvum_annotate.py (new): annotate_network_edges(coords,(lat,lon)), mode, spatial_index,
on_date) -> [EdgeAnnotation{coord_pair_index, matched_features, mvum_status}]. Walks
consecutive pairs, queries the index (10m buffer), applies a parallelism filter (acute
angle to the edge <=45deg, both directions), resolves per-mode access via the existing
check_access/get_mode_field/symbol_to_access (worst/most-restrictive across matches).
Mode map: foot->open (skip), 2w->e_bike_class1, 4w->atv, vehicle->highclearancevehicle;
auto is already resolved to a concrete candidate upstream.
- router.py: _route_D_network_only and _build_response annotate the network leg using the
injected self.spatial_index (getattr-guarded; skipped + debug-logged if None), attach
edge_mvum to the network feature, and add summary.mvum_closed_crossings +
summary.mvum_segments_annotated.
- offroute_route.py: inject app.config[MVUM_SPATIAL_INDEX] onto the router per request.
- DirectionsPanel.jsx: warning row when mvum_closed_crossings>0.
- tests/test_mvum_annotate.py: parallel match, perpendicular reject, seasonal closure,
symbol fallback, summary count.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix seasonal date default + hoist annotation, single annotate for Auto
- Default mvum_on_date to datetime.now() at annotation time so seasonal MVUM
openings/closings actually fire in production (was effectively None -> no seasonal).
- Hoist per-edge annotation out of _route_D_network_only and _build_response into a new
central OffrouteRouter._annotate_network_segments(result, mode), invoked once at the end
of route() (annotate_mvum=True). _route_auto probes with annotate_mvum=False and
annotates only the winning candidate -> Auto runs annotation once instead of up to 4x.
Removed the inline annotation/edge_mvum/summary blocks from both scenario handlers.
Note: the central pass filters network features on properties.segment_type == "network"
(the actual tag) rather than the spec-suggested "kind", which is not a field here.
1 new test: test_route_auto_annotates_only_winner.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* MVUM Layer 0: spatial index foundation
Additive only — no routing logic, no response-format, no Valhalla changes.
- mvum.py: add MVUMSpatialIndex. Loads mvum_roads + mvum_trails from navi.db (read-only),
decodes the pure-WKB shape blobs with shapely, builds a shapely.strtree.STRtree, and
keeps a parallel list of full feature records (all columns minus the blob, plus the
parsed geometry) with a by_id lookup. Exposes query_bbox(min_lat,min_lon,max_lat,max_lon)
and query_buffered_line(coords, tolerance_m) returning candidate records (coarse bbox +
buffer; full parallelism filter is a TODO for PR-B). Reports road_count, trail_count,
bbox, build_time_seconds, memory_estimate_mb (RSS delta).
- app.py: build the index once per process (singleton) at service init; stored on
app.config[MVUM_SPATIAL_INDEX]. Failure is logged and degrades to None.
- admin.py: GET /api/admin/mvum-spatial/info (Authentik-gated, read-only) returning the
counts/bbox/build-time/memory stats.
- tests/test_mvum_spatial.py: index loads, query_bbox returns Boise-area features,
query_buffered_line returns a feature, admin endpoint returns counts.
Diagnostic before coding (read-only): roads_with_shape=150568/null=68,
trails_with_shape=21995/null=6746 (green), shape blobs are pure WKB MultiLineString.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix lat-aware buffer + count WKB parse failures
- query_buffered_line: replace the static _DEG_PER_M with _buffer_degrees_for_meters(),
which scales longitude degrees by cos(lat) and uses the larger lat/lon equivalent;
buffer at the polyline avg latitude. Early-return [] for empty coords.
- _load_table: count WKB parse failures and logger.warning once per table when > 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_route_auto now probes every eligible candidate (eligibility filter unchanged) and
picks the one with the smallest summary.total_effort_minutes, instead of returning the
first that succeeds. Fixes the case where a wilderness start + on-road end fell through
to foot and computed the entire network leg at foot pace. selected_mode = winning
candidate; ties keep AUTO_MODE_PRIORITY order; all-fail still returns the last error.
Wilderness leg still always foot (unchanged).
Per-leg breakdown: summary now carries wilderness_minutes + network_minutes (mirrors the
existing wilderness_effort_minutes/network_duration_minutes) in _build_response (A/B/C)
and _route_D. Frontend DirectionsPanel shows a transition badge "Auto: Foot Xmin +
<mode> Ymin" when wilderness_minutes>0 and selected_mode!=foot, else the existing
"Auto chose <mode>".
Tests: add min-time pick + per-leg breakdown; update probe-all assertions (no early
return) and make the selected-mode test time-based.
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rename mode constants: mtb->2w, atv->4w
Rename the OFFROUTE travel-mode identifiers mtb->2w and atv->4w across backend and
frontend. MVUM vehicle-access classes/columns (atv, motorcycle) are a separate
vocabulary and are left untouched. UI labels (MTB/ATV) and the cost.py __main__ demo
local variables (cannot be digit-initial identifiers) are unchanged.
Backend: MODE_PROFILES, MODE_TO_COSTING, MODE_TO_VALID_HIGHWAYS, AUTO_MODE_PRIORITY,
_MODES_* sets, CATEGORY_ELIGIBLE_MODES, route()/compute_cost*/_pathfind_wilderness
Literals, VALID_MODES, and tests. offroute_route.py adds a backward-compat shim mapping
legacy mode=mtb->2w and mode=atv->4w before validation so bookmarked URLs still work.
Frontend: store.js routeMode doc, DirectionsPanel TRAVEL_MODES ids + SELECTED_MODE_LABEL
keys, Panel TRAVEL_MODES ids, ManeuverList network_mode->verb map keys, api.js jsdoc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix user-visible labels: MTB->2W, ATV->4W + grep cleanup
Update the user-facing travel-mode labels to match the renamed ids:
- DirectionsPanel TRAVEL_MODES (2w -> "2W", 4w -> "4W") + SELECTED_MODE_LABEL values.
- Panel TRAVEL_MODES labels.
- Stale test comment: mode=mtb -> mode=2w.
Grep pass over frontend/src + backend/services/navi_offroute found no remaining
mode-identifier string literals to rename. Residual hits are all out-of-scope: MVUM
vehicle-access vocabulary (mvum.py, /api/mvum output, single-quoted test fixtures), the
cost.py __main__ demo locals, the offroute_route.py back-compat shim, and comments
referencing the historical names.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A single noisy DEM cell could fabricate a huge fake slope and make an edge
unconditionally impassable, forcing the pathfinder to route around passable terrain.
Replace the hard cliff (|grade|>max_grade -> skip) with a smooth exponential penalty:
no penalty up to max_grade, then base_time *= exp(overshoot * SLOPE_PENALTY_SCALE);
only grades whose penalty exceeds SLOPE_PENALTY_CAP (true bad data / vertical) are
dropped. Routing can now see through noisy cells while still strongly avoiding real
cliffs. Penalty only raises edge cost, so the heuristic stays admissible.
2 tests: smooth penalty traverses a >max_grade gap (finite, raised cost) yet routes
around / drops a past-cap grade; exactly-at-threshold grade incurs no penalty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace MCP_Geometric in _pathfind_wilderness with a numba-jit anisotropic A* (new
astar.py): signed-slope speed (climbing != descending; tobler peaks at -0.05), hard
cliff, per-edge avg context multiplier, trail-takes-both via 256-entry lookup, per-edge
barriers (strict/pragmatic/emergency), multi-goal A* (first popped wins) with admissible
distance/base-speed heuristic. New compute_cost_multiplier_grid (slope-free context
multiplier) + exponential inflation (sigma=1.8; inf->HARD=50*p95 for blur, inf re-imposed).
numba>=0.59 added (numba 0.65.1).
fix: wilderness leg is always foot effort; mode parameter reserved for future flexibility.
_pathfind_wilderness keeps the mode param (threaded from _route_A/B/C) but hardcodes
cost_mode=foot for the cost grid, trail friction, speed function, base speed, and max
grade. Off-trail math for MTB/ATV/vehicle is not well-grounded and real-world wilderness
traversal is foot regardless (push the bike, walk past where the vehicle stops). User mode
still drives entry-point eligibility (query_radius highway filter) and Valhalla network
costing. Matches the original pre-#17 design: wilderness ALWAYS uses foot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old ST_DWithin(50km) scan returned ~226k candidate points near dense areas before
sorting (~3-10s/call). Replace with PostGIS k-NN ordering: ORDER BY geom::geography <->
point LIMIT k, which the existing GiST index on (geom::geography) walks nearest-first and
stops after K rows. SELECT still computes ST_Distance AS distance_m so callers see real
meters. radius_km is kept as a Python soft cap applied after fetch (drops rows beyond it),
preserving the caller expanded-radius fallback (now effectively a no-op).
2 tests: SQL contains <-> + LIMIT and no ST_DWithin; radius_km soft cap filters beyond-cap rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three wilderness-scenario guards used table_exists() OR get_entry_point_count()==0
to check the index is non-empty — a full SELECT COUNT(*) that scanned the entire
2.94M-row table (~9-73s depending on load/cache). Add EntryPointIndex.has_entry_points()
using SELECT EXISTS (SELECT 1 ... LIMIT 1), which short-circuits at the first row, and
swap it into _route_A/_route_B/_route_C. get_entry_point_count() kept for admin-info/tests.
3 new tests: table missing -> False, empty -> False, rows -> True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align spatial eligibility with Valhalla's actual /locate vocabulary:
- road grade lives in classification.classification (8-value enum incl service_other,
which was missing from PAVED_HIGHWAY_CLASSES); track/path/footway are NOT grades,
they live in classification.use.
- _locate_on_network now also returns use (defensive .get chain; None in fallback).
- Renamed TRACK/PATH_HIGHWAY_CLASSES -> TRACK_USE_VALUES/PATH_USE_VALUES; atv/mtb now
match on use, vehicle still on the paved grade.
- 4 tests: service_other->vehicle, use=track->atv/mtb/foot, use=footway->mtb/foot,
none->foot.
Live-verified: the Boise end point (43.626,-116.215, service_other) now yields
[atv,mtb,vehicle,foot] instead of [foot]; Auto intersection picks vehicle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #10s _spatial_eligible_modes read edge.get("road_class"), but Valhalla /locate
returns no class without verbose, and even with verbose=true the class is NOT named
road_class — it lives at edge.classification.classification (lowercase, e.g.
"secondary"). Without it, every paved/track/path gate failed and Auto collapsed to
foot for untyped endpoints.
Fix: add verbose=true to the /locate body and read the class via a defensive
edge.get("edge",{}).get("classification",{}).get("classification") chain, kept under
the same road_class key so downstream eligibility is unchanged. Live-verified against
Valhalla: (43.621,-116.205) -> road_class=secondary, snap 5.0m, PAVED=True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the 3-tier vehicle-only discriminator with a per-endpoint eligible-mode-set
intersection:
- CATEGORY_ELIGIBLE_MODES maps OSM key:value (and key:* wildcards) to eligible modes;
_eligible_modes_from_category resolves a hint, or None when untyped.
- Untyped endpoints fall back to _spatial_eligible_modes: parallel Valhalla /locate
(auto/pedestrian/bicycle) + 3-tier snap/road-class rules (vehicle needs paved, and
paved+flat in the 5-100m grace zone; atv paved/track; mtb paved/track/path; foot always).
- _route_auto intersects both endpoints eligible sets, probes AUTO_MODE_PRIORITY within
it, and adds selected_mode + selected_mode_set. Both untyped endpoints resolve in parallel.
- _locate_on_network now returns road_class.
- api_offroute accepts optional start_category/end_category and forwards them (auto only).
Frontend: requestOffroute takes startCategory/endCategory; computeRoute passes
routeStart/routeEnd.category; startDirections preserves place.category.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vehicle is pure Valhalla road routing — Valhalla snaps endpoints to the nearest
road automatically, so the off-network classifier (OFF_NETWORK_THRESHOLD_M) is
irrelevant for it. Add an early branch in route() that sends vehicle straight to
_route_D_network_only, instead of bandaiding the threshold. Foot/MTB/ATV keep the
gate so users can intentionally pin backcountry points; Auto inherits the skip via
its recursive mode=vehicle probe. Threshold stays at 10.
Frontend: boundary-mode chips now show for all modes except Drive (vehicle).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When mode="auto", OffrouteRouter._route_auto() probes AUTO_MODE_PRIORITY
(vehicle -> atv -> mtb -> foot) and returns the first mode whose network can
serve the route, tagging the result with selected_mode. route() already errors
when a mode cannot reach an endpoint, so the first status==ok is the most
road-capable feasible mode. Adds 4 isolation tests (route monkeypatched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
navi-offroute (:8428, extraction #8) was missing from fleet.py's SERVICES list,
so it never appeared in /api/admin/fleet. Add the one (name, port) row — the
single source of truth that build_fleet, dependency_summaries, and the
self-info fanned_services all derive from. 8427/navi-admin is the aggregator
itself (self-poll), correctly not in its own fan-out.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-REWRITE — the LAST navi→recon coupling. navi-places now
decides "is this wiki article in the local Kiwix mirror?" in-process instead of
HTTP-calling recon's /api/wiki-rewrite. Intra-process swap, no nginx changes.
Mirrors decouple #4-READ (which moved wiki_index.db reads the same way).
- NEW services/navi_places/wiki_rewrite.py: verbatim port of recon's
lib/wiki_rewrite.py. Only adjustments: setup_logging -> stdlib logging;
KIWIX_BASE -> NAVI_KIWIX_BASE_URL env; KIWIX_PUBLIC_BASE -> NAVI_KIWIX_PUBLIC_BASE
env; cache DB -> NAVI_WIKI_CACHE_DB (default /var/lib/navi-backend/wiki_cache.db,
auto-created); + a reset() to match the place_cache/wiki_index per-worker pattern.
No logic changes — same classify, same lazy hourly catalog refresh, same HEAD
timeout, same status values (local/public/original), same cache semantics.
- place_detail.py: _enrich_wiki_links_via_http -> _enrich_wiki_links; calls
wiki_rewrite.rewrite_wiki_link(tag,value) (TUPLE) and unpacks it, replacing
the dict-returning HTTP client. Import + docstrings updated.
- app.py: wiki_rewrite.reset() per worker/test (alongside place_cache/wiki_index).
- DELETE services/navi_places/wiki_rewrite_client.py (HTTP shim dead).
- tests: the 2 wiki-rewrite tests now monkeypatch the local
wiki_rewrite.rewrite_wiki_link (tuple) instead of the deleted client.
Recon's endpoint stays live but unused until PR-B (safe co-existence).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-READ. navi-places now reads its own wiki_index.db directly
(NAVI_WIKI_INDEX_DB) instead of HTTP-calling recon's /api/wiki-enrich — same
pattern it already uses for place_cache.db. The 2.1GB DB was copied to
/var/lib/navi-backend/wiki_index.db out-of-band (5,061,763 rows verified).
- NEW services/navi_places/wiki_index.py: verbatim port of recon's
lookup_wiki_index + _get_wiki_index_db, reading NAVI_WIKI_INDEX_DB, mirroring
place_cache.py's db_path()/lazy-conn/reset() pattern. Returns the same
{wiki_summary, wiki_population, wiki_url, wikivoyage_url} shape /api/wiki-enrich
did, so it's a drop-in for the HTTP client.
- place_detail.py: _enrich_with_wiki_via_http -> _enrich_with_wiki_index; call
wiki_index.lookup() instead of wiki_client.enrich_via_recon(); docstrings.
- app.py: wiki_index.reset() per worker/test (alongside place_cache.reset_cache()).
- admin.py: drop the recon-wiki-enrich dependency probe; add NAVI_WIKI_INDEX_DB
env + a read-only filesystem entry. (recon-wiki-rewrite probe kept — separate
decouple.)
- DELETE wiki_client.py (fully replaced).
- test_place.py: convert the wiki test from a monkeypatched HTTP client to a
hermetic tmp wiki_index.db.
Internal localhost migration — no nginx/edge involvement. recon's /api/wiki-enrich
stays live until PR-B (deploy PR-A first so nothing calls the route after removal).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of the overture-import relocation. The Overture Places ETL moves from
recon (where it produced data nothing in recon consumes) to navi-backend (the
side that owns the consumer, navi-places). Additive: recon's copy stays live
until PR-B; this just establishes the navi-side copy + deps + docs.
- scripts/overture_import.py: verbatim port of recon's script (recon master
879df84). The ONLY non-verbatim change is the line-8 docstring usage hint,
swapped from `/opt/recon` + venv to the navi-backend path + .venv.
- pyproject.toml: add `duckdb>=1.5` (recon runs 1.5.2; psycopg2-binary already
present). It's the only new dep.
- scripts/README.md: document the manual-only trigger + invocation.
Source release is pinned in-code: OVERTURE_RELEASE = '2026-04-15.0'.
No tests (ETL; none on recon either). Per cleanup #29, the script has no lib/
imports — fully self-contained (stdlib + duckdb + psycopg2).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of the 2-PR whoami migration. Net-new, additive endpoint in navi-admin
matching recon's existing handler shape exactly. Recon's handler stays live in
this PR; once nginx routes /api/auth/whoami to :8427 (out-of-band) and recon's
handler is removed (PR-B), navi-admin is the sole owner.
- New services/navi_admin/auth_route.py with its own blueprint (navi_admin_auth):
GET /api/auth/whoami reads X-Authentik-Username, returns {authenticated,
username}. NOT @require_auth — it's the "am I logged in?" check, must answer
the unauthenticated case (mirrors recon).
- app.py: register the new blueprint (2 lines).
- test_auth.py: header-present + header-absent cases.
Kept in its own blueprint/file so admin_route.py's "all routes @require_auth"
invariant stays true. recon and nginx untouched (additive only).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Matt's directive that navi-* should not call any /api/* on recon.
navi-admin was the only navi service doing so (polling recon's /api/health
and surfacing it in /api/admin/recon/info + the /api/admin/fleet fan-out).
navi-admin is now the navi-only fleet view; recon has its own dashboard for
recon-pipeline health.
- admin_route.py: delete the /api/admin/recon/info handler; drop the recon
config entry + RECON_HEALTH_URL/RECON_REPO_PATH env entries from
/api/admin/navi-admin/info; refresh docstrings.
- fleet.py: remove recon constants, recon_health_url/recon_repo_path/
recon_git_sha/wrap_recon_health, the now-unused shared.git_sha import, and
the recon arms in build_fleet + dependency_summaries. /api/admin/fleet now
reports only the 6 navi-* services.
- tests: drop the 2 recon/info tests + recon scaffolding; strip recon
assertions from fleet + self-info tests. 12 relevant tests pass (10 admin
+ 2 git_sha).
shared/git_sha.py KEPT unchanged — it's a generic git_short_sha(path) helper
used by every service's create_app(), not recon-specific.
RECON_HEALTH_URL + RECON_REPO_PATH in /etc/navi-backend/navi-admin.env are now
dead — flagged for out-of-band post-merge cleanup.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-offroute service (extraction #8 — the last one)
Faithful port of recon's /api/offroute (POST) + /api/mvum (GET) and the
runtime offroute modules into a new :8428 service. Closes the loop: after
this, navi-frontend talks only to navi-backend.
Ported: router.py (OffrouteRouter, EntryPointIndex, 4 route strategies,
in-Python MCP_Geometric least-cost path, Valhalla integration, per-request
osmium extract), mvum.py (MVUMReader over navi.db), cost.py, friction.py,
trails.py, and barriers.py (runtime BarrierReader/WildernessReader only).
NOT ported (per Phase A §3/§15): prototype.py (dead at runtime), barriers.py
build_*_raster (offline GDB→raster prep). DEM imported from shared/dem.py
(PR #9), not duplicated.
Behaviour-faithful changes: hardcoded paths/URLs → env vars; the
profile.offroute.* config (osm_pbf_path/postgis_dsn/densify_interval_m) →
dedicated env vars (router drops deployment_config). Both routes public (no
auth, matching recon). PADUS via libpq peer-auth DSN (dbname=padus) — NO
secret. Owns no DB.
15 hermetic tests (offroute validation + mocked-router shape + close-always;
fixture-SQLite MVUM roads/trails/fallback/null; admin auth + no-secrets +
probe shape). Full suite 119 passed / 1 skipped. Adds scikit-image + rasterio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* navi-offroute: PR #10 review cleanups (4 faithful-port deviations)
1. trails.py — drop recon-era "Run the Phase B rasterization script"
reference from the not-found error (confusing in navi-offroute context).
2. friction.py — add FileNotFoundError-before-rasterio-open check to
match barriers/trails consistency.
3. mvum.py — remove dead try/except shapely import + warnings.warn at
2 sites (shapely is a hard pyproject dep; the fallback was unreachable).
4. router.py — declare psutil in pyproject, drop the silent fallback;
the MEMORY_LIMIT_GB safety check was silently disabled in prod.
Adds test_friction_reader_raises_file_not_found_when_missing (16 navi-offroute
tests; full suite 120 passed / 1 skipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure refactor, no behavior change. Moves services/navi_geo/dem.py to
shared/dem.py (verbatim logic + env override; only docstring + location
changed) and re-points navi-geo's two imports (geo_route.py, admin.py) to
`from shared.dem import ...`.
Per extraction-8-phase-a.md §5/§13.1: navi-offroute (#18) needs the same
DEMReader, so a single source of truth in shared/ beats a third copy. Second
shared/ promotion after PR #7 round-2's shared/git_sha.py; navi-offroute will
`from shared.dem import DEMReader` directly.
Adds shared/tests/test_dem.py (dem_path default + NAVI_DEM_PMTILES override).
navi-geo behavior unchanged (test_reverse_bundle mocks geo_route._DEM, agnostic
to DEMReader's location). Full suite: 104 passed / 1 skipped (+2).
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #6 round-1 fixup #3 wired netsyms.health() (COUNT + DISTINCT on
35 GB) into _netsyms_fs_entry, adding >3s latency to cold admin-info
calls. navi-admin's fleet fan-out (3s timeout) caught it after #7
deploy. Reverting to the cheap _file_entry shape; deleting health()
per the no-dead-code rule (the original fallback option).
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-admin service (extraction #7)
Net-new fleet admin aggregator on :8427 — no port from recon (recon has no
/api/admin route; Phase A §3). Three @require_auth routes:
GET /api/admin/fleet fan-out to all 6 navi-* /api/admin/<svc>/info
+ recon /api/health, merged; never 5xx
(failures land in errors[])
GET /api/admin/recon/info recon /api/health wrapped in the info shape
GET /api/admin/navi-admin/info self-describe
Fan-out forwards the caller's X-Authentik-Username so the @require_auth
upstreams accept it; per-service admin endpoints stay localhost-only (this is
the single edge-exposed admin surface). Service discovery: hardcoded list in
fleet.py (Option B). No secrets, no DB.
Deploy artifacts (NOT applied here): navi-admin.env.example, systemd unit,
nginx ^~ /api/admin snippet, and deploy/caddy notes for the @authed_api edit
(first Caddy change since #2).
12 hermetic tests (fleet happy-path, per-service timeout/500 → errors[],
auth-header forwarding, recon-down degraded-not-5xx, self-info no-secrets,
auth-required). Full monorepo suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #7 review fixes
1. Symmetric degraded-entry handling in fleet.build_fleet — every
probed service now appears in `services` with a uniform degraded
dict on failure (matches recon's existing pattern), AND in errors[].
Operators see "everything I tried + which broke" consistently.
2. Catch ValueError specifically in _get_json — non-JSON 200 responses
now surface as `error: 'invalid JSON'` instead of opaque 'ValueError'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #7 review fixes (round 2)
1. Unified degraded shape: wrap_recon_health calls _degraded_entry on
failure — no more runtime.status vs runtime.recon_status asymmetry.
Every probed service has the same shape on failure
(runtime.status == 'unreachable'). recon-specific runtime fields
(recon_status/recon_uptime/pipeline) remain only on the success path.
2. DRY'd git short-SHA helper into shared/git_sha.py — was duplicated in
7 service app.py files + fleet.recon_git_sha. One implementation,
one place to fix when behavior changes. Adds shared/tests (testpaths
now includes "shared").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add navi-geo service (extraction #6)
Faithful port of recon's geocode/reverse family to a new :8426 service:
GET /api/geocode?q=&limit=&lat=&lon=&zoom= Photon-first ranked search
GET /api/reverse?lat=&lon= reverse geocode (Photon)
GET /api/reverse/<lat>/<lon> reverse enrichment bundle (Central)
Ported modules: geocode.py (engine), netsyms.py (address SQLite), dem.py
(planet-DEM reader), address_book.py (reader copy), and the three handlers +
four bundle helpers from netsyms_api.py. All three routes public, behaviour-
identical to recon.
Behaviour-changing edges (both pre-decided in Phase A/B, called out in the PR):
- landclass: in-process call replaced with HTTP GET to navi-landclass :8424,
reading .summary (the same most-specific unit-name string). First navi→navi
edge after landclass itself.
- hardcoded paths/URLs → env vars (PHOTON_URL, NAVI_NETSYMS_DB,
NAVI_TIMEZONE_DB, NAVI_DEM_PMTILES, NAVI_ADDRESS_BOOK_YAML,
NAVI_LANDCLASS_URL); rerank trace log opt-in (NAVI_GEO_RERANK_TRACE_LOG,
default off — recon always wrote /tmp).
No secrets in this service: PADUS_DB_* disappears because landclass is HTTP-
delegated (Phase A §10). Address book uses Option B (shared-file read), the
same pattern navi-contacts already uses.
Bundle 9-key contract preserved exactly (name/city/county/state/country/
postal_code/timezone/landclass/elevation_m), same null-on-component-failure
semantics, same in-memory TTLCache(10_000, 86_400) per worker.
Tests: 28 passing, 1 skipped (real timezone DB, off-box). Ported the 9 recon
reverse-bundle tests + added the HTTP-landclass coupling tests + hermetic
geocode reranker/intent-classifier tests (recon's geocode_test.py was a live
smoke test). Adds usaddress/rapidfuzz/cachetools/shapely/numpy/Pillow/pmtiles
to deps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #6 review fixes
1. Rename geocode._setup_trace_logger → setup_trace_logger (public hook)
2. Hoist `import requests as http_requests` to module level in geo_route.py
3. Wire netsyms.health() into admin.py (enriches the netsyms filesystem entry
with row_count/file_size_bytes/indexed_countries; no shared-builder change)
4. Fix misleading LANDCLASS_TIMEOUT_S comment (recon had no timeout)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_places/ on :8425 — the heaviest extraction. Ports recon's
/api/place family with the two wiki dependencies decoupled to HTTP.
Routes (public, mirroring recon):
GET /api/place/<osm_type>/<int:osm_id> (Nominatim -> Overpass fallback + enrich)
GET /api/place/wikidata/<wikidata_id> (Wikidata entity)
-> 200 / 400 / 404 / 502, same response shapes as recon.
Enrichment chain (recon order): Overture (PostGIS) -> Google Places -> wiki
rewrite -> wiki index. The two wiki paths are now HTTP to recon (the 2.1 GB
wiki_index.db and Kiwix/wiki_cache stay in recon — see [[reference-echo6-edge-topology]]):
- wiki_client.enrich_via_recon -> recon /api/wiki-enrich (PR #8) [has_kiwix_wiki]
- wiki_rewrite_client.rewrite_via_recon -> recon /api/wiki-rewrite (PR #9) [has_wiki_rewriting]
(per-tag loop over the <=4 wiki extratags, mirroring recon's _enrich_wiki_links)
Both clients degrade gracefully (None / status 'original') on error/timeout.
Data ownership (see [[feedback-navi-backend-data-ownership]]):
- place_cache.db migrates to /var/lib/navi-backend/place_cache.db (env
NAVI_PLACE_CACHE_DB). place_cache.py auto-creates the FULL schema on first
open — place_cache (incl. the google_place_id/google_data/google_fetched_at
columns recon added by migration) + google_api_calls — so a fresh DB serves
both cache_put and the Google daily-cap/cache. WAL, shared module conn.
- overture stays in external PG (OVERTURE_DB_* env), verbatim port of recon's
pool (1,3) + _pool_failed latch, with reset_pool()+probe_db() added.
- wiki_index.db / Kiwix stay in recon, reached via the two HTTP endpoints.
Modules: place_cache.py, overture.py (verbatim+probe), google_places.py
(daily cap via env GOOGLE_PLACES_DAILY_CAP; DB via shared place_cache conn),
wiki_client.py + wiki_rewrite_client.py (HTTP, RECON_BASE_URL default
http://127.0.0.1:8420), osm_categories.py (vendored for humanize_category),
place_detail.py (orchestrator), config.py (feature flags from the vendored
profile via NAVI_PROFILES_DIR), place_route.py, admin.py, app.py.
Feature gates read from the vendored profile (config.py), matching recon:
has_overture_enrichment / has_google_places_enrichment / has_kiwix_wiki /
has_wiki_rewriting — flag off => that enricher is skipped entirely.
admin.py (§4.5): 2 secrets masked (OVERTURE_DB_PASSWORD, GOOGLE_PLACES_API_KEY);
3 dependency probes — overture-postgis (SELECT 1), recon-wiki-enrich and
recon-wiki-rewrite (GET with no params, expect HTTP 400 = route alive).
Deploy: systemd unit (:8425) + nginx snippet (^~ /api/place, no trailing slash,
no proxy_cache; public, no Caddy edit — TIER 2 already through nginx since #2).
Tests (13; recon had zero for this module): validation (400), cache hit (no
upstream), nominatim hit, nominatim-miss->overpass fallback, both-fail 502,
not-found 404, wikidata happy + invalid, overture gated-off (no PG call),
wiki-rewrite-via-http local hit + original pass-through, wiki-enrich-via-http
field merge. Full suite 59. See ../recon_refactor/extraction-5-phase-a.md,
-wiki-enrich-investigation.md, -wiki-rewrite-investigation.md, and PRs #8/#9.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_landclass/ on :8424 — single blueprint, behavior-identical
port of recon's lib/landclass.py + the /api/landclass handler.
GET /api/landclass?lat=&lon= -> { lat, lon, classifications[], count,
is_public, is_private, summary }; 400 on bad/out-of-range lat/lon.
db.py: faithful port of recon's PostGIS module — lazy module-level
psycopg2.pool.SimpleConnectionPool(minconn=1, maxconn=3) from PADUS_DB_* env;
the ST_Intersects query on pad_units (antimeridian filter, acres-ordered,
limit 10); all PAD-US code->label maps verbatim; graceful degradation
(returns [] when PG is unreachable, never raises/500). Adds reset_pool()
(create_app resets per worker) and probe_db() (SELECT 1) for admin health.
No filesystem state — PostGIS is external. No DB-on-disk migration; only the
5 PADUS_DB_* env vars (PADUS_DB_PASSWORD is a real secret, masked in
admin-info via mask_key; the other 4 shown plain). adds psycopg2-binary>=2.9.
Decision — DROPPED the recon `has_landclass` profile-flag gate: the frontend
already gates on its own has_landclass feature flag, and removing the
cross-service config dependency keeps navi-landclass self-contained per the
"only API" rule (the service's existence is the feature being available).
navi-geo coupling (reverse-bundle needs landclass) — per Phase A, recommend
Option B: navi-geo HTTP-calls /api/landclass and reads `.summary` (the
endpoint already returns it); no shared module. Decided when #6 lands.
Tests (8; recon had 2): point-with-coverage -> classification + decoded
labels, ocean point -> empty, bad/missing/out-of-range lat/lon -> 400, PG
down -> graceful 200 empty (not 500), format_summary unit. Full suite 46.
Deploy: systemd unit (:8424) + nginx snippet (one ^~ /api/landclass block, no
proxy_cache; /api/landclass is public so no Caddy edit — TIER 2 already
routes through nginx since extraction #2).
See ../recon_refactor/extraction-4-phase-a.md (which also corrects the handoff:
/mnt/nav/padus/ is source GIS files, NOT a runtime path — this service has no
/mnt/nav dependency, only PADUS_DB_* + PG network access).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_contacts/ on :8423 — two blueprints, behavior-identical
ports of recon's contacts + address_book code.
Routes:
contacts (10, all @require_auth, per-user via X-Authentik-Username):
GET/POST /api/contacts; GET /api/contacts/nearby; GET /api/contacts/deleted;
GET/PATCH/DELETE /api/contacts/<id>;
POST /api/contacts/<id>/restore; POST /api/contacts/<id>/restore-as;
DELETE /api/contacts/<id>/purge
address_book (2, public):
GET /api/address_book/lookup?q= ; GET /api/address_book/list
Data ownership (per Matt's rule — DBs live in navi-backend territory,
auto-create on first run; only massive tilesets stay external):
- contacts.db: env NAVI_CONTACTS_DB (default /var/lib/navi-backend/contacts.db).
ContactsDB auto-creates the schema (table + 5 indexes incl. the partial-
unique Home/Work index) on first open via CREATE ... IF NOT EXISTS — this
is recon's own behavior, ported verbatim. WAL + busy_timeout=5000 preserved.
- address_book.yaml: vendored into config/address_book.yaml (read-only, like
the deployment profiles in extraction #2); path via NAVI_ADDRESS_BOOK_YAML.
Tests (28 new; recon had none for contacts): full ContactsDB CRUD, soft-delete/
restore/restore-as/purge, Home/Work 409 (create + restore conflict), nearby
proximity, search/category filter, per-user partitioning, auth-required, and
DB auto-create; plus address_book ported from recon's test (exact/partial/
case-insensitive/alias/miss/empty/list/hot-reload/missing-file). Full suite 38.
Timestamp fix (diverges from recon on purpose): restore_as builds updated_at
with Python's strftime. recon uses the bare '%Y-%m-%dT%H:%M:%fZ' there — but
Python's %f is microseconds-only (no seconds), so that yields malformed ISO
strings like "...T15:30:123456Z". recon's own update()/soft_delete() use the
correct '%Y-%m-%dT%H:%M:%S.%fZ'. This port uses the correct format in all three
places and adds a regression guard (strptime) in test_restore_as_relabels.
This is a PRE-EXISTING recon bug; we fix it here. The recon-side restore_as
retires with extraction #7 (Jinja /nav-i + /deleted-contacts removal), so the
recon refactor doesn't need to touch it.
Also: shared/auth.py require_auth now sets request.user_id (recon's contract —
the contacts routes read it). Backward-compatible: navi-traffic/navi-config
admin endpoints don't use it.
Deploy artifacts: systemd unit (:8423) + nginx snippet with two ^~ blocks.
NOTE the nginx prefixes are `^~ /api/contacts` and `^~ /api/address_book`
WITHOUT a trailing slash, so the bare `/api/contacts` (list/create) is matched
too — a trailing-slash prefix would miss it and fall through to recon.
See ../recon_refactor/extraction-3-phase-a.md for the route/schema/ownership
analysis (which also corrects the handoff: contacts is /opt/recon/data/
contacts.db, NOT /mnt/nav/navi.db).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendors the three deployment profile YAMLs (home, minimal_pi, regional_pi)
into navi-backend's own repo at config/profiles/. Copied verbatim from
/opt/recon/config/profiles/ post-PR-A, so home.yaml carries the auth block and
the two Pi profiles carry the TODO(matt) logout-host note.
Architectural reason: services in this monorepo should depend on each other
only through their HTTP APIs, never each other's filesystem. navi-config
currently defaults to reading /opt/recon/config/profiles/ — a cross-service
filesystem coupling. With the profiles vendored here, navi-config will be
pointed at this repo's copy via NAVI_CONFIG_PROFILES_DIR in
/etc/navi-backend/navi-config.env at deploy time, so it no longer reaches into
recon's tree.
recon keeps its own /opt/recon/config/profiles/ for its internal use; the two
copies are independent for now. Reconciling them — ideally by having recon
HTTP-fetch /api/config from navi-config rather than re-parsing YAML — is a
tracked follow-up, out of scope for extraction #2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_config/ on :8422, mirroring recon's /api/config contract:
- config_route.py: GET /api/config -> jsonify(get_deployment_config())
with Cache-Control: public, max-age=300 (byte-for-byte recon's response).
- config_loader.py: faithful port of recon lib/deployment_config.py. Reads
RECON_PROFILE (default "home") and NAVI_CONFIG_PROFILES_DIR (default
/opt/recon/config/profiles, so it serves the SAME files recon does during
cutover). yaml.safe_load, module-level cache. Lazy load (vs recon's eager
import-time load) so the module imports cleanly off-VM and a missing
profile surfaces as HTTP 500 at request time rather than a failed import.
- admin.py: /api/admin/navi-config/info per handoff §4.5, require_auth gated.
env values (NAVI_CONFIG_PROFILES_DIR, RECON_PROFILE) are non-secret paths/
names, shown as-is (no mask_key); dependencies=[]; filesystem reports the
active profile path + exists/readable.
- app.py: create_app() factory mirroring navi_traffic, same metrics wiring;
resets the loader cache per instance so each worker/test reloads fresh.
Deploy artifacts: systemd unit (:8422) and an nginx snippet using
`location ^~ /api/config` (the ^~ convention from extraction #1 so the asset
.png/.css regex can't shadow it). No proxy_cache zone — the response is
already cached in-process and via Cache-Control: max-age=300; emits a literal
X-Cache-Status: BYPASS for parity with navi-traffic.
Adds PyYAML>=6 to deps. Tests (services/navi_config/tests): 200 + parsed dict,
Cache-Control header, RECON_PROFILE override, default=home, missing profile=500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
