The Twin Falls Wikivoyage investigation surfaced two coordinated gaps that kept
OSM wikivoyage tags from rewriting to local Kiwix URLs even though the
wikivoyage ZIM is loaded and serving.
Fix 1 (pagination) -- wiki_rewrite.py: append ?count=-1 to the Kiwix OPDS catalog
fetch. kiwix-serve's /catalog/v2/entries defaults to the first 10 entries; the
library has 17, so wikivoyage (and other page-2 ZIMs) were never seen by
_discover_zims and never entered _zim_map, so their tags always fell back to public.
Fix 2 (tag passthrough) -- place_detail.py: add wikivoyage to both nominatim
extratags whitelists, mirroring the existing wikipedia/wikidata lines. The
rewriter (classify_wiki_link / build_kiwix_url / rewrite_wiki_link) and the
_enrich_wiki_links loop were already source_type-generic and covered wikivoyage;
the only missing link was that the nominatim parser dropped the wikivoyage tag
before enrichment ever saw it. No rewriter refactor was needed.
Tests (test_place.py):
- test_catalog_url_requests_full_library: the OPDS fetch URL contains count=-1.
- test_wikivoyage_tag_rewrites_to_local: a wikivoyage OSM tag for a mirrored
article rewrites to https://wiki.echo6.co/content/wikivoyage_en_all_maxi_<date>/...
with sources.wiki_rewrites.wikivoyage == "local" (Kiwix mocked).
Full navi-places suite: 15 passed.
Follow-up (separate ops step, not in this PR): prune 3 dangling library.xml
entries on the Kiwix host (wikiloc.com_eng_2026-04_18, meshtastic.org_eng_2026-04_14,
meshtastic.org_eng_2026-04_15) whose ZIM files are absent; kiwix-serve silently
skips them.
Note: Twin Falls (R/121355) itself has no OSM wikivoyage tag, so it still will not
get a local Wikivoyage link from tag rewrite -- that needs the separate name-based
discovery feature (discover_wikivoyage_article stub).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-REWRITE — the LAST navi→recon coupling. navi-places now
decides "is this wiki article in the local Kiwix mirror?" in-process instead of
HTTP-calling recon's /api/wiki-rewrite. Intra-process swap, no nginx changes.
Mirrors decouple #4-READ (which moved wiki_index.db reads the same way).
- NEW services/navi_places/wiki_rewrite.py: verbatim port of recon's
lib/wiki_rewrite.py. Only adjustments: setup_logging -> stdlib logging;
KIWIX_BASE -> NAVI_KIWIX_BASE_URL env; KIWIX_PUBLIC_BASE -> NAVI_KIWIX_PUBLIC_BASE
env; cache DB -> NAVI_WIKI_CACHE_DB (default /var/lib/navi-backend/wiki_cache.db,
auto-created); + a reset() to match the place_cache/wiki_index per-worker pattern.
No logic changes — same classify, same lazy hourly catalog refresh, same HEAD
timeout, same status values (local/public/original), same cache semantics.
- place_detail.py: _enrich_wiki_links_via_http -> _enrich_wiki_links; calls
wiki_rewrite.rewrite_wiki_link(tag,value) (TUPLE) and unpacks it, replacing
the dict-returning HTTP client. Import + docstrings updated.
- app.py: wiki_rewrite.reset() per worker/test (alongside place_cache/wiki_index).
- DELETE services/navi_places/wiki_rewrite_client.py (HTTP shim dead).
- tests: the 2 wiki-rewrite tests now monkeypatch the local
wiki_rewrite.rewrite_wiki_link (tuple) instead of the deleted client.
Recon's endpoint stays live but unused until PR-B (safe co-existence).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-A of decouple #4-READ. navi-places now reads its own wiki_index.db directly
(NAVI_WIKI_INDEX_DB) instead of HTTP-calling recon's /api/wiki-enrich — same
pattern it already uses for place_cache.db. The 2.1GB DB was copied to
/var/lib/navi-backend/wiki_index.db out-of-band (5,061,763 rows verified).
- NEW services/navi_places/wiki_index.py: verbatim port of recon's
lookup_wiki_index + _get_wiki_index_db, reading NAVI_WIKI_INDEX_DB, mirroring
place_cache.py's db_path()/lazy-conn/reset() pattern. Returns the same
{wiki_summary, wiki_population, wiki_url, wikivoyage_url} shape /api/wiki-enrich
did, so it's a drop-in for the HTTP client.
- place_detail.py: _enrich_with_wiki_via_http -> _enrich_with_wiki_index; call
wiki_index.lookup() instead of wiki_client.enrich_via_recon(); docstrings.
- app.py: wiki_index.reset() per worker/test (alongside place_cache.reset_cache()).
- admin.py: drop the recon-wiki-enrich dependency probe; add NAVI_WIKI_INDEX_DB
env + a read-only filesystem entry. (recon-wiki-rewrite probe kept — separate
decouple.)
- DELETE wiki_client.py (fully replaced).
- test_place.py: convert the wiki test from a monkeypatched HTTP client to a
hermetic tmp wiki_index.db.
Internal localhost migration — no nginx/edge involvement. recon's /api/wiki-enrich
stays live until PR-B (deploy PR-A first so nothing calls the route after removal).
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New services/navi_places/ on :8425 — the heaviest extraction. Ports recon's
/api/place family with the two wiki dependencies decoupled to HTTP.
Routes (public, mirroring recon):
GET /api/place/<osm_type>/<int:osm_id> (Nominatim -> Overpass fallback + enrich)
GET /api/place/wikidata/<wikidata_id> (Wikidata entity)
-> 200 / 400 / 404 / 502, same response shapes as recon.
Enrichment chain (recon order): Overture (PostGIS) -> Google Places -> wiki
rewrite -> wiki index. The two wiki paths are now HTTP to recon (the 2.1 GB
wiki_index.db and Kiwix/wiki_cache stay in recon — see [[reference-echo6-edge-topology]]):
- wiki_client.enrich_via_recon -> recon /api/wiki-enrich (PR #8) [has_kiwix_wiki]
- wiki_rewrite_client.rewrite_via_recon -> recon /api/wiki-rewrite (PR #9) [has_wiki_rewriting]
(per-tag loop over the <=4 wiki extratags, mirroring recon's _enrich_wiki_links)
Both clients degrade gracefully (None / status 'original') on error/timeout.
Data ownership (see [[feedback-navi-backend-data-ownership]]):
- place_cache.db migrates to /var/lib/navi-backend/place_cache.db (env
NAVI_PLACE_CACHE_DB). place_cache.py auto-creates the FULL schema on first
open — place_cache (incl. the google_place_id/google_data/google_fetched_at
columns recon added by migration) + google_api_calls — so a fresh DB serves
both cache_put and the Google daily-cap/cache. WAL, shared module conn.
- overture stays in external PG (OVERTURE_DB_* env), verbatim port of recon's
pool (1,3) + _pool_failed latch, with reset_pool()+probe_db() added.
- wiki_index.db / Kiwix stay in recon, reached via the two HTTP endpoints.
Modules: place_cache.py, overture.py (verbatim+probe), google_places.py
(daily cap via env GOOGLE_PLACES_DAILY_CAP; DB via shared place_cache conn),
wiki_client.py + wiki_rewrite_client.py (HTTP, RECON_BASE_URL default
http://127.0.0.1:8420), osm_categories.py (vendored for humanize_category),
place_detail.py (orchestrator), config.py (feature flags from the vendored
profile via NAVI_PROFILES_DIR), place_route.py, admin.py, app.py.
Feature gates read from the vendored profile (config.py), matching recon:
has_overture_enrichment / has_google_places_enrichment / has_kiwix_wiki /
has_wiki_rewriting — flag off => that enricher is skipped entirely.
admin.py (§4.5): 2 secrets masked (OVERTURE_DB_PASSWORD, GOOGLE_PLACES_API_KEY);
3 dependency probes — overture-postgis (SELECT 1), recon-wiki-enrich and
recon-wiki-rewrite (GET with no params, expect HTTP 400 = route alive).
Deploy: systemd unit (:8425) + nginx snippet (^~ /api/place, no trailing slash,
no proxy_cache; public, no Caddy edit — TIER 2 already through nginx since #2).
Tests (13; recon had zero for this module): validation (400), cache hit (no
upstream), nominatim hit, nominatim-miss->overpass fallback, both-fail 502,
not-found 404, wikidata happy + invalid, overture gated-off (no PG call),
wiki-rewrite-via-http local hit + original pass-through, wiki-enrich-via-http
field merge. Full suite 59. See ../recon_refactor/extraction-5-phase-a.md,
-wiki-enrich-investigation.md, -wiki-rewrite-investigation.md, and PRs #8/#9.
Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
