refactored-recon/PROJECT-STATE.md

PROJECT-STATE.md — Echo6 Navigation Stack

Last verified: 2026-04-21

1. Overview

Navi is a self-hosted navigation and geocoding application. The frontend is a MapLibre-based React SPA with layer controls for hillshade terrain and live traffic overlays. The backend is the RECON knowledge-extraction pipeline, extended with geocoding, address book, reverse-geocode, traffic proxy, and deployment-profile endpoints. Routing is provided by Valhalla (Docker). Tile serving uses PMTiles (planet-scale, self-contained). Place detail enrichment uses a local Nominatim instance (Idaho coverage, expansion planned). The long-term ambition is for this stack to be strippable down to a Raspberry Pi for field/offline use.

2. Repositories

Repo	Forge URL	Master HEAD	Purpose
matt/recon	https://forge.echo6.co/matt/recon	2121ee4	Backend: RECON pipeline + Navi API endpoints
matt/navi	https://forge.echo6.co/matt/navi	4020d5a	Frontend: React + MapLibre + Tailwind SPA
matt/refactored-recon	https://forge.echo6.co/matt/refactored-recon	22dd6c1	Design docs (PROJECT-BIBLE, architecture, phase specs)

3. Deployment Topology

Host	IP	Role
data (Proxmox)	192.168.1.240	VM host, SSD (virtiofs → VM 1130), Samba
VM 1130 "recon-vm"	192.168.1.130 / 100.64.0.24	RECON backend, Navi frontend, Photon, Valhalla, Nominatim, nginx, Kiwix
cortex (VM 150)	192.168.1.150 / 100.64.0.14	Qdrant (6333), TEI embeddings (8090), sparse embeddings (8091), Open WebUI, design-docs repo
pi-nas	192.168.1.245 / 100.64.0.21	Archive storage (Photon recovery JSONL, NFS /export/data)
Utility CT 101	192.168.1.101 / 100.64.0.8	Caddy reverse proxy (TLS termination, Authentik forward auth)

VM 1130 mount topology

All mounts except /mnt/nas are virtiofs from the Proxmox data host SSD — not NFS.

Mount	Type	Source	Purpose
/mnt/nav	virtiofs	data host SSD	Tiles, Photon index, addresses, frontend, Nominatim, Valhalla sources
/mnt/library	virtiofs	data host SSD	PDF source library (RECON ingest)
/mnt/kiwix	virtiofs	data host SSD	ZIM files for Kiwix-serve
/mnt/nas	NFS	pi-nas 192.168.1.245:/export/data	Archive-tier (recovery dumps, cold storage)

4. /mnt/nav Breakdown

Path	Size	Contents
photon/	21 GB	photon_data/ (OpenSearch index), photon-1.1.0.jar
tiles/na.pmtiles	31 GB	North America tileset (active, referenced by frontend)
tiles/planet/	126 GB	Full planet PMTiles archive (planet-20260420.pmtiles + current.pmtiles symlink)
tiles/hillshade-na.pmtiles	93 GB	Terrarium DEM hillshade tiles (Mapzen, z0-12, 1.8M tiles). Backup: pi-nas:/export/data/nav/hillshade-na.pmtiles
addresses/	36 GB	AddressDatabase2025.sqlite — Netsyms 160M-row US/CA address DB
sources/	1.9 GB	Valhalla build inputs: valhalla_tiles/ (213M), elevation_data/ (1.2G), idaho-latest.osm.pbf (119M)
nominatim/	2.3 GB	postgres-data/ (2.0G Idaho DB), data/ (119M PBF), backup dump (168M), README
frontend/	1.5 MB	Built Navi SPA (index.html + assets)
valhalla/	4 KB	Empty (Valhalla config lives in Docker, tiles in sources/)

Total used: 497 GB of 938 GB (56%)

5. Services on VM 1130

Service	Unit / Container	Port	Purpose
RECON	recon.service	8420	Knowledge pipeline + Navi API (Flask)
RECON Watchdog	recon-watchdog.service	—	Library pipeline file watcher (recon.py pipeline watch)
Photon	photon.service	2322	Geocoding service (Java, OpenSearch-backed)
nginx	nginx.service	8440, 8888	Navi frontend (8440), PDF library (8888)
Kiwix	kiwix.service	8430	ZIM file server (kiwix-serve)
Nominatim	nominatim.service (Docker wrapper)	8010	Place detail geocoding (Idaho coverage). Internal only — not exposed externally.
Valhalla	Docker container valhalla	8002	Turn-by-turn routing engine (Idaho OSM extract)

Note: PostgreSQL 16 apt packages are installed on the host but the postgresql service is disabled. Nominatim's Docker container (mediagis/nominatim:4.5) manages its own internal PostgreSQL instance. The persistent DB data lives at /mnt/nav/nominatim/postgres-data/.

nginx virtual hosts

Port	Server	Root / Proxy	Function
8440	navi.echo6.co	/mnt/nav/frontend + proxy /api/ → :8420, /valhalla/ → :8002, /tiles/ → alias	Navi SPA with backend proxying
8888	_	/mnt/library (autoindex)	PDF library browsing (Authentik forward auth)

6. External-Facing Hostnames

Hostname	DNS Target	Internal Backend	Auth	Service
navi.echo6.co	199.6.36.163 → CT 101	100.64.0.24:8440	Authentik forward auth (except /tiles/* unauthenticated)	Navi frontend + API proxy
recon.echo6.co	199.6.36.163 → CT 101	100.64.0.24:8420	None (direct)	RECON dashboard + Navi API
files.echo6.co	199.6.36.163 → CT 101	100.64.0.24:8888	Authentik forward auth	PDF library
forge.echo6.co	5.189.158.149 → Contabo	127.0.0.1:3001	Authentik SSO	Git server
ai.echo6.co	199.6.36.163 → CT 101	100.64.0.14:8080	Authentik SSO	Open WebUI (Aurora)
stream.echo6.co	199.6.36.163 → CT 101	192.168.1.170:80	Authentik SSO	PeerTube

navi.echo6.co routing chain: GoDaddy DNS → 199.6.36.163 (home public IP) → Caddy on CT 101 (TLS via acme.sh cert, Authentik forward auth with /tiles/* exempt) → nginx on VM 1130:8440 (server_name navi.echo6.co, serves /mnt/nav/frontend, proxies /api/ → :8420, /valhalla/ → :8002, /tiles/ → alias /mnt/nav/tiles/). Tailscale split DNS resolves to 100.64.0.8 (CT 101).

7. API Endpoints on RECON (:8420)

Navi-specific (blueprint-registered)

Endpoint	Method	Blueprint	Purpose
/api/geocode	GET	geocode_bp	Forward geocode (address book → Netsyms → Photon cascade)
/api/reverse	GET	geocode_bp	Reverse geocode (lat/lon → address)
/api/address_book/lookup	GET	address_book_bp	Saved-place lookup by name
/api/address_book/list	GET	address_book_bp	List all saved places
/api/netsyms/lookup	GET	netsyms_bp	Direct Netsyms address DB query
/api/netsyms/health	GET	netsyms_bp	Netsyms DB status (row count, file size)

Deployment config and overlays

Endpoint	Method	Purpose
/api/config	GET	Deployment profile + feature flags (cached 300s)
/api/traffic/flow/<z>/<x>/<y>.png	GET	TomTom traffic tile proxy (hides API key, cached 120s)

| /api/place/<osm_type>/<osm_id> | GET | Place detail proxy (Nominatim-first for Idaho bbox, Overpass fallback, SQLite cache at data/place_cache.db) |

Kiwix / Scraper

Endpoint	Method	Purpose
/api/kiwix/sources	GET	List ZIM sources
/api/kiwix/upload	POST	Upload ZIM file
/api/kiwix/toggle-ingest/<id>	POST	Enable/disable source ingestion
/api/kiwix/trigger-ingest/<id>	POST	Force re-ingest
/api/kiwix/remove/<id>	POST	Remove ZIM source
/api/scraper/submit	POST	Submit URL for scraping
/api/scraper/jobs	GET	List scraper jobs
/api/scraper/cancel/<id>	POST	Cancel running job
/api/scraper/retry/<id>	POST	Retry failed job
/api/scraper/delete/<id>	POST	Delete job record
/api/scraper/clear-failed	POST	Purge all failed jobs

Knowledge pipeline

Endpoint	Method	Purpose
/api/upload	POST	Upload PDF/document
/api/upload/<hash>/status	GET	Check upload processing status
/api/ingest-url	POST	Ingest single URL
/api/ingest-urls	POST	Batch URL ingest
/api/crawl	POST	Start web crawl
/api/crawl/<id>/status	GET	Crawl progress
/api/search	POST	Semantic search (Qdrant)
/api/knowledge-stats	GET	Pipeline statistics
/api/quick-stats	GET	Dashboard quick stats
/api/status	GET	Full pipeline status
/api/health	GET	Service health check
/api/metrics/history	GET	Historical metrics

PeerTube integration

Endpoint	Method	Purpose
/api/ingest-peertube	POST	Ingest PeerTube channel
/api/peertube/stats	GET	PeerTube ingest stats
/api/peertube/channels	GET	List tracked channels
/api/peertube/channels/add	POST	Add channel
/api/peertube/dashboard	GET	PeerTube dashboard data

Settings / Admin

Endpoint	Method	Purpose
/api/keys	GET/POST	API key management
/api/cookies/status	GET	YouTube cookie status
/api/cookies/upload	POST	Upload cookies
/api/vpn/status	GET	NordVPN status
/api/vpn/rotate	POST	Rotate VPN IP
/api/service/restart	POST	Restart RECON service

8. Navi Frontend Feature State

Working

• Forward geocode search (address book → Netsyms → Photon cascade)
• Multi-stop turn-by-turn routing (Valhalla, draggable stops)
• Map-click reverse geocode with pin drop
• Place detail panel (address, coords, city/state)
• Copy popover (address / lat,lon / Plus Code)
• GPS origin (browser geolocation as route start)
• Light/dark theme with design tokens
• SPA with tile serving (PMTiles, North America)
• API proxy through nginx (no CORS issues)
• Deployment profiles + /api/config endpoint (home, regional_pi, minimal_pi)
• Layer control popover with hillshade and traffic toggles
• Traffic overlay live (TomTom flow tiles via backend proxy, 120s cache)
• Nominatim Idaho infrastructure standing by for C2b proxy wiring
• Place detail enrichment panel (C2b+C2c: category badge, hours, contact, details, wiki links via Nominatim/Overpass)
• Hillshade terrain overlay (93 GB terrarium DEM, toggle in layer control, renders under vector basemap in both themes)

Stubbed / Partial

• Save button — shows toast "Coming soon", not wired to address book write API
• Share — UI present, no implementation

Planned

Phase	Feature
C2a.2	Nominatim coverage expansion to 11 western states (ID + MT, WY, NV, UT, OR, WA, CO, NM, AZ, CA)
C2.5	Save wiring — persist places to address_book.yaml
C3	Kiwix Wikipedia summary integration in place panel
F	PAD-US parcel/public-land data layer
N	Aurora place-details tool extension

9. Operational Runbook

Deploy Navi frontend

ssh zvx@192.168.1.130
cd /home/zvx/projects/repos/navi
npm run build && rsync -av --delete dist/ /mnt/nav/frontend/

Restart RECON backend

ssh zvx@192.168.1.130
sudo systemctl restart recon
# Cache rebuild takes 2-3 min; enrichment/embedding workers restart

Check Photon geocoder

systemctl is-active photon
curl -s "http://localhost:2322/api?q=filer+idaho&limit=1" | python3 -m json.tool

Check deployment config

curl -s http://localhost:8420/api/config | python3 -m json.tool
# Returns active profile, feature flags, tile sources, service URLs

Restart Nominatim

sudo systemctl restart nominatim
# Or directly: docker restart nominatim
curl http://localhost:8010/status.php  # Should return "OK"

Query Nominatim locally

# Search
curl -s "http://localhost:8010/search.php?q=boise&format=json&limit=1" | python3 -m json.tool

# Details by OSM ID
curl -s "http://localhost:8010/details.php?osmtype=R&osmid=121350&addressdetails=1&format=json" | python3 -m json.tool

Back up Nominatim DB

docker exec nominatim bash -c 'su - postgres -c "pg_dump nominatim" | gzip > /tmp/nom-dump.gz'
docker cp nominatim:/tmp/nom-dump.gz /mnt/nav/nominatim/nominatim-idaho-postimport-$(date +%Y%m%d).dump.gz
# Existing backup: /mnt/nav/nominatim/nominatim-idaho-postimport-20260421.dump.gz (168 MB)

Regenerate na.pmtiles from planet archive

# IMPORTANT: virtiofs corrupts pmtiles extract output. Extract to /tmp first.
pmtiles extract /mnt/nav/tiles/planet/current.pmtiles /tmp/na.pmtiles \
  --bbox=-125.0,24.0,-66.0,50.0
cp /tmp/na.pmtiles /mnt/nav/tiles/na.pmtiles
rm /tmp/na.pmtiles

Restart Valhalla

docker restart valhalla
curl -s http://localhost:8002/status  # Should return 200

Check all Navi endpoints

curl -s "http://localhost:8420/api/geocode?q=filer+idaho" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/reverse?lat=42.5736&lon=-114.6066" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/address_book/lookup?q=home" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/netsyms/health" | python3 -m json.tool
curl -sI http://localhost:8420/api/traffic/flow/10/188/362.png | head -3
curl -s http://localhost:8010/status.php
curl -s http://localhost:8002/status

10. Known Gotchas

1. PMTiles + virtiofs corruption — pmtiles extract (v1.30.1) writes all-zero headers when the output file is on a virtiofs mount. Always extract to /tmp first, then cp to the virtiofs-backed path. The copy preserves valid headers.

2. virtiofsd restart after VM hard-stop — When VM 1130 is hard-stopped (e.g. OOM hang requiring qm stop 1130 from the Proxmox host), the virtiofsd daemons on the data host die. They must be restarted before the VM can boot:

   sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@192.168.1.240 \
     'systemctl start virtiofsd-nav virtiofsd-library virtiofsd-kiwix'
   # Then: qm start 1130
   

3. Nominatim flatnode trap — The mediagis/nominatim container auto-enables --flat-nodes if a /nominatim/flatnode directory exists inside the container. This is triggered by volume mounts, not by environment variables. The flatnode file mmap's 64+ GB and will OOM on this 24 GB VM. Never mount a flatnode volume. For regional imports (state-sized), the container's default --cache mode works fine.

4. Canonical frontend location — The Navi frontend repo lives at /home/zvx/projects/repos/navi on VM 1130 only. A stale copy on cortex was deleted 2026-04-20. Never recreate there.

5. Photon JVM memory — Photon runs with -Xmx10g on a 24 GB VM. If the OpenSearch index is rebuilt, JVM heap pressure can cause OOM kills if other services (RECON, Valhalla, Nominatim) spike simultaneously. Monitor with journalctl -u photon -f.

6. Tailscale split DNS — Internal (Tailscale-connected) clients resolve *.echo6.co to Utility Caddy (100.64.0.8) via dnsmasq on Contabo. External DNS resolves to home public IP (199.6.36.163). Both paths terminate at the same Caddy on CT 101.

7. RECON restart disruption — RECON has long-running enrichment/embedding workers and a cache warm-up phase (~2-3 min). Never restart without confirming no active batch work.

8. Valhalla is Idaho-only — Current routing graph is built from idaho-latest.osm.pbf. Routes outside Idaho will fail or produce nonsensical results. Expanding coverage requires rebuilding tiles from a larger extract.

9. Address book is YAML, not DB — Saved places live in /opt/recon/config/address_book.yaml. No concurrent-write safety. The Save button in the frontend is deliberately stubbed until a proper write path with conflict handling is designed.

10. Planet tiles are large — The full planet PMTiles archive is 126 GB. Don't delete tiles/planet/ thinking it's unused — it's the source for regional extracts.

11. Pi-NAS /tmp is a 4 GB tmpfs — pmtiles convert (and other tools) write temp files to /tmp by default. On pi-nas that's a tiny SD-card-backed tmpfs that fills instantly with large workloads. Always set TMPDIR=/export/data/... (the 22 TB data disk) before running conversion tools on pi-nas.

12. Hillshade recovery artifact — The hillshade PMTiles archive copy lives permanently at pi-nas:/export/data/nav/hillshade-na.pmtiles (93 GB). Download script and logs archived at pi-nas:/export/data/nav/hillshade-work/. SHA256: 7095ef0fbbe42be85b3d5fe86ea13e20cd0a9628af4daa54cb841749b42efc35.

13. RECON api.py merge caution — During the refactored-recon merge, lib/api.py had "both blocks kept" conflict resolution for blueprint registrations. If the merge is ever redone or the file is forked, manually verify all blueprint registrations remain intact.

11. Long-Term Endpoint: Wilderness Navigation

Goal

User is lost in wilderness — no roads, trails, or visible landmarks. Messages Aurora via Meshtastic: "route me back home." Receives step-by-step text directions with cardinal bearings, elevation cues, and landmark references. Runs offline from a Raspberry Pi field node.

Current proof-of-concept status

Component	State
Meshtastic ↔ Aurora text exchange	Working (MeshAI on CT 108, AIDA-N2 node)
Aurora turn-by-turn instruction generation	Working (nav_tools over existing Valhalla routes)
Off-network pathfinding	Not built — this is the unique engineering gap
Controlled field test	Not attempted

Building blocks in place or planned

Block	Status	Role in wilderness-nav
Terrain DEM (hillshade-na.pmtiles, 93 GB)	Live	Cost-surface elevation input
Vector basemap (na.pmtiles, 31 GB)	Live	Trail data, landmark labels
Nominatim + OSM tags	Live (Idaho)	Landmark identification, trail refs
Aurora tool router	Live	LLM text-generation layer for directions
Meshtastic integration	Live	Delivery channel (text over LoRa)
PAD-US public/private land (Phase F)	Planned	Routing constraint — avoid private land
Topo contours (Phase T)	Planned	Visual reference layer for user
Off-road Valhalla (Phase K1-K2)	Planned	Trail-network routing where trails exist
Pi deployment profile (Phase K4)	Planned	Offline field node build
NLCD land-cover ingest	Future	Traversability classification (forest, scrub, water, rock)

Missing pieces specific to wilderness-nav

1. Cost-surface data pipeline — DEM slope + NLCD land cover + hydrography + PAD-US boundaries → single traversability raster. Each cell gets a movement cost based on slope steepness, vegetation density, water crossings, and land access restrictions.
2. Off-network pathfinder — A* (or Dijkstra) on the cost-surface raster for segments where no trail or road network exists. Returns a waypoint sequence with bearings and elevation deltas.
3. /api/offroute endpoint — Hybrid router combining Valhalla network routing (where trails exist) with the off-network pathfinder (where they don't). Stitches segments into a single route with unified instruction format.
4. Aurora tool extension — New tool exposing /api/offroute to the LLM, with prompt engineering for cardinal-bearing + landmark instruction style suitable for text-only delivery over Meshtastic.
5. Field-testing discipline — Controlled validation in known wilderness areas before any real reliance. GPS ground-truth comparison of generated routes.

Sequencing

Aspirational. Prerequisites F, T, and K1-K2 are in the active planning queue. After those complete, the wilderness-specific work (cost surface + pathfinder + offroute endpoint + Aurora integration) is tractable — estimated 1-2 weeks of focused work. Proof-of-concept validation happens in controlled field tests before any serious commitment.

Why this endpoint matters

No commercial product combines self-hosted, offline-capable, LLM-generated natural-language directions, off-network terrain routing, and Meshtastic delivery. This is the reason the navigation stack exists.