Per-tag HTTP wrapper over wiki_rewrite.rewrite_wiki_link so the (future)
navi-places service can rewrite OSM wiki tags to local Kiwix URLs over HTTP
instead of importing recon's wiki_rewrite module (which talks to Kiwix on
localhost:8430 and the wiki_cache table in /opt/recon/data/place_cache.db).
Companion to PR #8 (/api/wiki-enrich) — Matt picked option B (HTTP-couple the
Kiwix offline-wiki rewriting too, since it matters in prod).
GET /api/wiki-rewrite?tag=<wikipedia|wikidata|wikivoyage|appropedia>&value=<raw>
-> 200 {url, status} where status is "local" | "public" | "original"
-> 400 on missing value or unknown tag
-> no 404 (unclassifiable value echoes back with status "original",
mirroring rewrite_wiki_link)
Public (no auth), like /api/place/* and /api/wiki-enrich.
Changes (additive only):
- lib/wiki_rewrite_api.py: new wiki_rewrite_bp blueprint. Thin route directly
over the existing rewrite_wiki_link(tag, value) — no extraction needed
(it's already a clean standalone function, unlike wiki-enrich's lookup).
- lib/api.py: register the blueprint (one block).
- lib/wiki_rewrite_api_test.py: 5 tests (local Kiwix hit, public fallback,
unclassifiable -> original, missing value -> 400, unknown tag -> 400),
stubbing check_kiwix_has_article (no Kiwix/DB), plain-assert + __main__
runner. Verified green against recon's venv (flask 3.1.2).
Does NOT touch place_detail's in-process _enrich_wiki_links — that gets removed
in a later PR once navi-places is live (same as PR #8). wiki_cache stays in
recon's own place_cache.db post-cutover (harmless positive-cache duplication).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HTTP wrapper over the wiki_index lookup so the (future) navi-places service can
fetch wiki enrichment over HTTP instead of reading recon's 2.1 GB
data/wiki_index.db directly (Phase A option B — HTTP coupling).
GET /api/wiki-enrich?wikidata=<Qid> (primary key)
GET /api/wiki-enrich?name=<name>&country=<cc> (fallback key)
-> 200 {wiki_summary?, wiki_population?, wiki_url?, wikivoyage_url?}
-> 400 if no usable key; 404 on no match. Public (no auth, like /api/place/*).
Route keys are wikidata_id / name+country — NOT osm_type/osm_id — because that
is how wiki_index is actually queried (the in-process _enrich_with_wiki_index
looks up by result['wikidata_id'] then name+country_code, never by OSM id; see
extraction-5-wiki-enrich-investigation.md). An osm-keyed route would have forced
a redundant in-recon place lookup.
Changes (additive only):
- lib/place_detail.py: new standalone lookup_wiki_index(wikidata_id, name,
country_code) doing the same two SELECTs + field/URL mapping as the
in-process path, returning a dict or None. Pure DB read, never raises.
`_enrich_with_wiki_index` is LEFT UNTOUCHED — it can be DRY-refactored to
delegate to this in a later PR; the in-process enrichment path is unchanged.
- lib/wiki_enrich_api.py: new wiki_enrich_bp blueprint with the route.
- lib/api.py: register the blueprint (one block).
- lib/wiki_enrich_api_test.py: 4 tests (hit-by-wikidata + decoded fields,
no-match -> 404, name+country fallback, no-key -> 400) over an in-memory
fixture DB; plain-assert style + __main__ runner (recon venv has no pytest).
Verified green against recon's venv (flask 3.1.2).
Does NOT remove the in-process _enrich_with_wiki_index call from place_detail —
that happens in a later PR once navi-places is live and serving.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /api/traffic/flow/<z>/<x>/<y>.png handler is dead code in recon. As of
extraction #1 of the recon<->Navi decoupling, this path is served by the
standalone navi-traffic service. Live request flow is now:
Caddy (CT 101, navi.echo6.co @authed_api, forward_auth)
-> nginx :8440 (location ^~ /api/traffic/ -> proxy_cache traffic_cache)
-> navi-traffic gunicorn :8421 (services/navi_traffic)
Cutover verified live: authenticated browser fetch to
https://navi.echo6.co/api/traffic/flow/... returns 200 image/png with
X-Cache-Status MISS then HIT (120s cache), Server: gunicorn.
navi-backend (github.com/zvx-echo6/navi-backend):
- dae54f3 Initial scaffold: navi-backend + navi-traffic
- 311cb8f nginx: use ^~ prefix on /api/traffic/ to beat .png regex catch-all
Caddy cutover (@authed_api upstream 8420 -> nginx 8440) applied on Utility
CT 101. Also drops the now-unused make_response flask import (no other uses
in lib/api.py). os and http_requests remain (used elsewhere).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enriches place API responses with wiki_summary, wiki_url, wiki_population,
and wikivoyage_url from wiki_index.db. Lookups by wikidata_id first,
then falls back to name + country_code.
Called from Nominatim, Overpass, and Wikidata endpoints.
Gated by has_kiwix_wiki feature flag.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
47 PAD-US units (Aleutian/Bering-Sea BOEM marine features, all is_valid=False)
are stored as antimeridian-wrapping polygons whose bbox spans ~360 deg of
longitude. Their invalid planar geometry forms latitude bands that ST_Intersects
false-matches for non-US points (e.g. London/Germany at ~51N matched
"Rat Islands" ogc_fid 3974).
Fix: add `AND (ST_XMax(geom) - ST_XMin(geom)) < 60` to the lookup_landclass
SELECT. No DB writes; two cheap ST_XMax/XMin evals on the already
spatial-index-filtered result set. Verified live: total 651088 rows,
filtered 651041 (exactly 47 excluded); Yosemite/Grand Canyon retained,
London/Germany now empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per OFFROUTE-ARCHITECTURE.md §9 ("planet-dem.pmtiles as single elevation
source"). The bundle endpoint previously called Valhalla /height, which only
has 48 Idaho HGT tiles; it now reads the planet-scale Terrarium PMTiles that
already back the frontend hillshade and contours.
- dem.py: add DEMReader.sample_point(lat, lon) — one z12 tile (LRU-cached),
Web-Mercator pixel index, None outside the +/-85.05 pole cap or when untiled.
- netsyms_api.py: module-level DEMReader singleton (lazy mmap, None if init
fails); _reverse_elevation now calls _DEM.sample_point; drop the Valhalla
HTTP call and _VALHALLA_HEIGHT_URL.
- tests: DEM-mock and DEM-unavailable cases; EXPECTED_KEYS derives from
_BUNDLE_KEYS. All 9 tests pass.
Verified live: Boise 824m, London 8m, Tokyo 35m, Yosemite 2804m, pole -> None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New geocode_bp sibling to the existing /api/reverse?lat=&lon= route (which
is unchanged). Returns a flat 9-field bundle for the Central enrichment
framework: name, city, county, state, country, postal_code (Photon),
timezone (timezones.sqlite via R-tree + shapely), landclass (in-process
lookup_landclass), elevation_m (Valhalla /height).
- Each component lookup is independent and wrapped in try/except: a failure
logs a warning and yields null, never a 5xx. 400 only on unparseable /
out-of-range coordinates.
- lat/lon parsed manually rather than via Flask <float:>, which rejects
negative and integer coordinates and would 404 instead of 400.
- 10k-entry / 24h TTLCache keyed on coords rounded to 4 decimals.
- Tests mock Photon/Valhalla/landclass; one test exercises the real
timezones.sqlite. cachetools pinned in requirements.txt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Migrate EntryPointIndex from SQLite to PostGIS (padus database)
- Densify highway LineStrings at 100m intervals via Shapely interpolate
- 2.94M entry points from 476k lines (4x more coverage)
- Tag each entry point with land_status via ST_Intersects against padus_sub
- 1.64M public (56%), 1.30M unknown (44%)
- Add geography GIST index for fast radius queries (~25ms)
- Increase OFF_NETWORK_THRESHOLD_M from 10m to 50m for GPS accuracy
- PBF path and PostGIS DSN configurable via home.yaml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The wilderness segment now ALWAYS uses foot mode for MCP pathfinding.
The user's selected mode only affects:
1. Entry point selection (MODE_TO_VALID_HIGHWAYS filtering)
2. Valhalla costing for the network segment
This ensures vehicles can navigate through wilderness (on foot) to
reach roads, rather than failing when no vehicle-accessible path exists.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
MVUM Data Import:
- Downloaded USFS MVUM Roads (150,636 features) and Trails (28,741 features)
- Imported to navi.db as mvum_roads and mvum_trails tables
- Idaho coverage: ~8,994 roads and ~4,504 trails across 7 national forests
- Preserved all vehicle-class fields (ATV, MOTORCYCLE, HIGHCLEARANCEVEHICLE, etc.)
- Preserved seasonal date ranges (*_DATESOPEN fields)
New mvum.py module:
- MVUMReader class for querying MVUM data by bbox and nearest point
- parse_date_range() for seasonal date string parsing (MM/DD-MM/DD format)
- check_access() for determining open/closed status with date checking
- symbol_to_access() fallback when per-vehicle fields are null
- get_mvum_access_grid() for rasterizing MVUM to pathfinder grid
Cost function integration:
- Added mvum parameter to compute_cost_grid()
- MVUM closures respond to boundary_mode:
* strict = impassable (np.inf)
* pragmatic = 5x friction penalty
* emergency = ignored entirely
- Foot mode skips MVUM (motor-vehicle specific)
Router integration:
- Loads MVUM access grid for motorized modes (mtb, atv, vehicle)
- Tracks mvum_closed_crossings in path summary
Places Panel API:
- GET /api/mvum?lat=XX&lon=XX&radius=50
- Returns MVUM feature with access status for all vehicle classes
- Includes seasonal date ranges, maintenance level, forest/district info
- GeoJSON geometry for map display
Validation:
- MVUM places endpoint tested with Sawtooth NF road
- All four modes validated with strict/pragmatic/emergency boundary modes
- Foot mode correctly ignores MVUM restrictions
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- New friction.py: reads WorldCover friction VRT, resamples to match
elevation grid, provides point sampling for validation
- Modified cost.py: accepts optional friction array, multiplies Tobler
time cost by friction multiplier, inf for water/nodata (255/0)
- Modified prototype.py: loads friction layer, passes to cost function,
validates path avoids water cells (friction=255)
Validated on Idaho test bbox:
- Path avoids Murtaugh Lake (no water cells on path)
- Friction along path: min=10, max=20, mean=10.2
- Effort increased 3.4% vs Phase O1 due to friction multipliers
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds _text_quality_ok() gate that replaces the bare 50-char length
check at each stage of the extraction fallback chain. Checks:
- Word-boundary ratio (≥60% of tokens must be real words)
- Concatenation ratio (lc→UC transitions must be <10% of word count)
When PyPDF2 default extraction fails quality check, retries with
space_width=100 for tighter word-boundary detection. This fixes
Haynes/workshop manuals where tight kerning produces concatenated
words like 'byMike' and 'oftheGuild'.
Also adds -layout flag to pdftotext subprocess calls for better
spatial awareness in the poppler fallback stage.
Note: PyPDF2 3.0.1 does not support layout=True parameter.
The space_width parameter serves the same purpose.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pre-processes HTML tree before lxml .text_content() to prevent
element concatenation:
- <table> cells joined with ' | ' delimiter, rows with newlines
- <br> tags produce newlines
- <li> items get '- ' prefix and newline separation
- <dt>/<dd> definition list items get newline separation
Fixes ~868 mangled Qdrant points where table content was jammed
together (e.g. 'Freq51Primary1A==' instead of 'Freq51 | Primary | 1A==').
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Returns {authenticated: bool, username: string|null} based on
X-Authentik-Username header presence. Used by Navi frontend to
detect auth state without triggering SSO redirect.
The /api/geocode endpoint blended Photon and Netsyms results, but only
Photon respected viewport bias from prior work. Address queries to
Netsyms/AddressDB returned globally-sorted matches regardless of where
the user was looking — searching '214 North St' from Idaho returned
Illinois results.
Now fetches up to 200 Netsyms results when viewport lat/lon provided,
sorts by squared distance from viewport center, then returns top N.
Falls back to default ordering when viewport absent. Photon path
unchanged.
Request polygon_geojson=1 from Nominatim to include admin boundary
polygons in place detail responses. Also fetch boundary via OSM
relation ID for wikidata lookups.
- Add get_place_by_wikidata() to place_detail.py
- Queries Wikidata API for entity details (name, description, coords)
- Extracts population, instance_of, OSM relation ID, Wikipedia link
- Add /api/place/wikidata/<id> route to api.py
Supports Navi basemap label enrichment when OSM details unavailable.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add lat/lon/zoom params to geocode() and _retrieve_photon_freetext()
- Update nav_tools.py wrapper to pass through viewport params
- Add /api/geocode handler support for lat/lon/zoom query params
- Add _safe_float() helper for param validation
- Cast zoom to int for Photon compatibility
Allows the frontend to pass current map center/zoom to bias
search results toward the visible area.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Guest users receive local and cached data only. New Google Places API
calls are only triggered for authenticated users, protecting against
cost exploitation on the public navi.echo6.co frontend.
The pattern: cached Google data flows freely (already paid for by an
authed lookup). New API calls require X-Authentik-Username via
get_user_id() check.
Replace /nav-i/api-keys stub with functional admin page for managing
third-party API keys (Gemini, TomTom, Google Places).
- New lib/api_keys_admin.py: list/update/test operations with masked
display, atomic .env writes (.env.bak backup), provider-specific
test calls (Gemini models.list, TomTom geocode, Google Places
searchText)
- 4 new endpoints: GET /api/nav-i/api-keys/list, POST .../update,
POST .../test, POST .../restart-recon
- Full UI: key table with masked values, per-key update modal with
show/hide toggle, inline test results with latency, Gemini detail
sub-table with per-key stats, RECON restart with confirmation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrites OSM wikipedia/wikidata/wikivoyage/appropedia extratag values
to local Kiwix URLs (wiki.echo6.co) when the article exists in a loaded
ZIM, falling back silently to public URLs otherwise.
- New lib/wiki_rewrite.py: URL classification, Kiwix OPDS catalog
discovery (xml.etree.ElementTree), HEAD-based availability check,
positive-only SQLite cache, disabled discovery stubs
- place_detail.py: _enrich_wiki_links() at both Nominatim and Overpass
enrichment sites, before cache_put
- Profile flags: has_wiki_rewriting (home/regional: true, minimal: false),
has_wiki_discovery (all: false, stubs for future activation)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates USGS PAD-US 4.0 (651k features) into a local PostGIS database
for point-in-polygon land ownership queries. Adds /api/landclass endpoint
returning classifications, public/private status, and management hierarchy.
- lib/landclass.py: connection pool, lookup_landclass(), domain label maps
- lib/api.py: GET /api/landclass?lat=&lon= (feature-flag gated)
- home.yaml: enable has_landclass flag
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fills opening_hours, phone, and website gaps when OSM + Overture data
is incomplete. Only fires for business-class POIs (amenity, shop, tourism,
leisure, office, craft). Daily API call cap with SQLite tracking.
cache_put now preserves google columns across cache refreshes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a query contains no road-type keywords (st, blvd, ave, etc.),
boost amenity/shop/tourism/leisure/office/craft results (+3.0) and
penalize highway/route results (-4.0). This fixes searches like
"starbucks twin falls" where a named service road outranked the
actual business POI due to Photon position tiebreaking.
Also fixes:
- Intent classifier now recognizes full state names ("idaho" not
just "ID") for LOCALITY classification
- Locality-type Photon results now populate _city from name field
so they participate in locality_fuzz scoring
- Trace logging expanded to all candidates with osm_key/value
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Businesses with housenumbers (e.g. M&W Markets at 130 US-30) were
classified as street_address because the housenumber check fired before
the osm_key check. Reorder so osm_key in amenity/shop/tourism/leisure/office
is evaluated first, ensuring businesses get type=poi regardless of
whether they have a street address. Also adds office to the POI key set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ingests 20.9M North America places from Overture Maps Foundation
(release 2026-04-15.0) into PostgreSQL. Enriches /api/place responses
with phone, website, and brand data via spatial + fuzzy name matching
when OSM extratags are sparse.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New /api/place/<osm_type>/<osm_id> endpoint returns cleaned OSM tag data
for PlaceDetail panel enrichment. Routes to local Nominatim (Idaho coverage)
first, falls back to Overpass public API for out-of-region queries. Responses
cached in SQLite (data/place_cache.db) with no expiry.
New modules: lib/place_detail.py (proxy + cache), lib/osm_categories.py
(~50 category humanization mappings). Profile YAMLs updated with
place_details config block and has_nominatim_details flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add /api/traffic/flow proxy route to hide TomTom API key from frontend
- Add tileset_hillshade and traffic config blocks to all three profiles
- Flip has_hillshade and has_traffic_overlay flags in home and regional profiles
- Minimal profile has config blocks but flags remain false (dormant)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add profile-driven config infrastructure:
- config/profiles/{home,regional_pi,minimal_pi}.yaml templates
- lib/deployment_config.py loader (reads RECON_PROFILE env var)
- GET /api/config returns active profile as JSON (5min cache)
Frontend reads this on startup to determine tile source, defaults,
and feature flags. No existing behavior changed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Accepts lat/lon query params, calls Photon /reverse, returns same
response shape as /api/geocode. Returns 200 with empty results on
no match (graceful degradation for ocean/unmapped areas).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inverts the /api/geocode chain. Photon is now the primary search
engine; the hand-rolled Netsyms free-text parser is removed.
Address book short-circuits nicknames only ("home", "work") —
full-address queries flow through Photon and address book
entries within 75m annotate matching results with labeled_as.
Coordinate strings detected before search.
Response shape: /api/geocode now returns a ranked candidates
list (always 200 OK, empty list if no match). No more 404 for
unmatched queries. Users can type messy input — wrong case,
missing punctuation, abbreviations, typos — and get results
or close matches.
Netsyms preserved at /api/netsyms/lookup for direct access.
USPS plus4 enrichment of Photon street-address hits is a
planned follow-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
