matt/refactored-recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/refactored-recon.git synced 2026-05-20 06:34:34 +02:00
Code Issues Projects Releases Packages Wiki Activity

PROJECT-STATE.md: hillshade live, wilderness-nav endpoint, Pi tmpfs gotcha

Browse source 
Hillshade DEM tiles (93 GB, Mapzen terrarium z0-12) now live at
/mnt/nav/tiles/hillshade-na.pmtiles. Place detail proxy (C2b) and
enrichment frontend (C2c) also reflected as complete. Added gotchas
for pi-nas /tmp tmpfs limit and hillshade recovery artifact location.

New Section 11 documents the long-term wilderness-nav endpoint:
off-network pathfinding over cost-surface raster, LLM-generated
cardinal-bearing directions delivered via Meshtastic. Captures
current PoC status, building blocks, missing pieces, and sequencing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt 2026-04-21 07:40:45 +00:00
commit 3d3b13fc2a
1 changed files with 419 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

419
PROJECT-STATE.md Normal file
View file

@ -0,0 +1,419 @@
# PROJECT-STATE.md — Echo6 Navigation Stack
*Last verified: 2026-04-21*
## 1. Overview
Navi is a self-hosted navigation and geocoding application. The frontend is a MapLibre-based
React SPA with layer controls for hillshade terrain and live traffic overlays. The backend is
the RECON knowledge-extraction pipeline, extended with geocoding, address book, reverse-geocode,
traffic proxy, and deployment-profile endpoints. Routing is provided by Valhalla (Docker).
Tile serving uses PMTiles (planet-scale, self-contained). Place detail enrichment uses a local
Nominatim instance (Idaho coverage, expansion planned). The long-term ambition is for this
stack to be strippable down to a Raspberry Pi for field/offline use.
## 2. Repositories
| Repo | Forge URL | Master HEAD | Purpose |
|------|-----------|-------------|---------|
| matt/recon | https://forge.echo6.co/matt/recon | `2121ee4` | Backend: RECON pipeline + Navi API endpoints |
| matt/navi | https://forge.echo6.co/matt/navi | `4020d5a` | Frontend: React + MapLibre + Tailwind SPA |
| matt/refactored-recon | https://forge.echo6.co/matt/refactored-recon | `22dd6c1` | Design docs (PROJECT-BIBLE, architecture, phase specs) |
## 3. Deployment Topology
| Host | IP | Role |
|------|-----|------|
| data (Proxmox) | 192.168.1.240 | VM host, SSD (virtiofs → VM 1130), Samba |
| VM 1130 "recon-vm" | 192.168.1.130 / 100.64.0.24 | RECON backend, Navi frontend, Photon, Valhalla, Nominatim, nginx, Kiwix |
| cortex (VM 150) | 192.168.1.150 / 100.64.0.14 | Qdrant (6333), TEI embeddings (8090), sparse embeddings (8091), Open WebUI, design-docs repo |
| pi-nas | 192.168.1.245 / 100.64.0.21 | Archive storage (Photon recovery JSONL, NFS /export/data) |
| Utility CT 101 | 192.168.1.101 / 100.64.0.8 | Caddy reverse proxy (TLS termination, Authentik forward auth) |
### VM 1130 mount topology
All mounts except `/mnt/nas` are **virtiofs** from the Proxmox data host SSD — not NFS.
| Mount | Type | Source | Purpose |
|-------|------|--------|---------|
| `/mnt/nav` | virtiofs | data host SSD | Tiles, Photon index, addresses, frontend, Nominatim, Valhalla sources |
| `/mnt/library` | virtiofs | data host SSD | PDF source library (RECON ingest) |
| `/mnt/kiwix` | virtiofs | data host SSD | ZIM files for Kiwix-serve |
| `/mnt/nas` | NFS | pi-nas 192.168.1.245:/export/data | Archive-tier (recovery dumps, cold storage) |
## 4. /mnt/nav Breakdown
| Path | Size | Contents |
|------|------|----------|
| `photon/` | 21 GB | `photon_data/` (OpenSearch index), `photon-1.1.0.jar` |
| `tiles/na.pmtiles` | 31 GB | North America tileset (active, referenced by frontend) |
| `tiles/planet/` | 126 GB | Full planet PMTiles archive (`planet-20260420.pmtiles` + `current.pmtiles` symlink) |
| `tiles/hillshade-na.pmtiles` | 93 GB | Terrarium DEM hillshade tiles (Mapzen, z0-12, 1.8M tiles). Backup: `pi-nas:/export/data/nav/hillshade-na.pmtiles` |
| `addresses/` | 36 GB | `AddressDatabase2025.sqlite` — Netsyms 160M-row US/CA address DB |
| `sources/` | 1.9 GB | Valhalla build inputs: `valhalla_tiles/` (213M), `elevation_data/` (1.2G), `idaho-latest.osm.pbf` (119M) |
| `nominatim/` | 2.3 GB | `postgres-data/` (2.0G Idaho DB), `data/` (119M PBF), backup dump (168M), README |
| `frontend/` | 1.5 MB | Built Navi SPA (index.html + assets) |
| `valhalla/` | 4 KB | Empty (Valhalla config lives in Docker, tiles in sources/) |
**Total used:** 497 GB of 938 GB (56%)
## 5. Services on VM 1130
| Service | Unit / Container | Port | Purpose |
|---------|-----------------|------|---------|
| RECON | `recon.service` | 8420 | Knowledge pipeline + Navi API (Flask) |
| RECON Watchdog | `recon-watchdog.service` | — | Library pipeline file watcher (`recon.py pipeline watch`) |
| Photon | `photon.service` | 2322 | Geocoding service (Java, OpenSearch-backed) |
| nginx | `nginx.service` | 8440, 8888 | Navi frontend (8440), PDF library (8888) |
| Kiwix | `kiwix.service` | 8430 | ZIM file server (kiwix-serve) |
| Nominatim | `nominatim.service` (Docker wrapper) | 8010 | Place detail geocoding (Idaho coverage). Internal only — not exposed externally. |
| Valhalla | Docker container `valhalla` | 8002 | Turn-by-turn routing engine (Idaho OSM extract) |
**Note:** PostgreSQL 16 apt packages are installed on the host but the `postgresql` service is
disabled. Nominatim's Docker container (mediagis/nominatim:4.5) manages its own internal
PostgreSQL instance. The persistent DB data lives at `/mnt/nav/nominatim/postgres-data/`.
### nginx virtual hosts
| Port | Server | Root / Proxy | Function |
|------|--------|-------------|----------|
| 8440 | navi.echo6.co | `/mnt/nav/frontend` + proxy `/api/` → :8420, `/valhalla/` → :8002, `/tiles/` → alias | Navi SPA with backend proxying |
| 8888 | _ | `/mnt/library` (autoindex) | PDF library browsing (Authentik forward auth) |
## 6. External-Facing Hostnames
| Hostname | DNS Target | Internal Backend | Auth | Service |
|----------|-----------|-----------------|------|---------|
| navi.echo6.co | 199.6.36.163 → CT 101 | 100.64.0.24:8440 | Authentik forward auth (except `/tiles/*` unauthenticated) | Navi frontend + API proxy |
| recon.echo6.co | 199.6.36.163 → CT 101 | 100.64.0.24:8420 | None (direct) | RECON dashboard + Navi API |
| files.echo6.co | 199.6.36.163 → CT 101 | 100.64.0.24:8888 | Authentik forward auth | PDF library |
| forge.echo6.co | 5.189.158.149 → Contabo | 127.0.0.1:3001 | Authentik SSO | Git server |
| ai.echo6.co | 199.6.36.163 → CT 101 | 100.64.0.14:8080 | Authentik SSO | Open WebUI (Aurora) |
| stream.echo6.co | 199.6.36.163 → CT 101 | 192.168.1.170:80 | Authentik SSO | PeerTube |
**navi.echo6.co routing chain:** GoDaddy DNS `→` 199.6.36.163 (home public IP) `→` Caddy on CT 101
(TLS via acme.sh cert, Authentik forward auth with `/tiles/*` exempt) `→` nginx on VM 1130:8440
(`server_name navi.echo6.co`, serves `/mnt/nav/frontend`, proxies `/api/` → :8420, `/valhalla/` → :8002,
`/tiles/` → alias `/mnt/nav/tiles/`). Tailscale split DNS resolves to 100.64.0.8 (CT 101).
## 7. API Endpoints on RECON (:8420)
### Navi-specific (blueprint-registered)
| Endpoint | Method | Blueprint | Purpose |
|----------|--------|-----------|---------|
| `/api/geocode` | GET | geocode_bp | Forward geocode (address book → Netsyms → Photon cascade) |
| `/api/reverse` | GET | geocode_bp | Reverse geocode (lat/lon → address) |
| `/api/address_book/lookup` | GET | address_book_bp | Saved-place lookup by name |
| `/api/address_book/list` | GET | address_book_bp | List all saved places |
| `/api/netsyms/lookup` | GET | netsyms_bp | Direct Netsyms address DB query |
| `/api/netsyms/health` | GET | netsyms_bp | Netsyms DB status (row count, file size) |
### Deployment config and overlays
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/config` | GET | Deployment profile + feature flags (cached 300s) |
| `/api/traffic/flow/<z>/<x>/<y>.png` | GET | TomTom traffic tile proxy (hides API key, cached 120s) |
| `/api/place/<osm_type>/<osm_id>` | GET | Place detail proxy (Nominatim-first for Idaho bbox, Overpass fallback, SQLite cache at `data/place_cache.db`) |
### Kiwix / Scraper
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/kiwix/sources` | GET | List ZIM sources |
| `/api/kiwix/upload` | POST | Upload ZIM file |
| `/api/kiwix/toggle-ingest/<id>` | POST | Enable/disable source ingestion |
| `/api/kiwix/trigger-ingest/<id>` | POST | Force re-ingest |
| `/api/kiwix/remove/<id>` | POST | Remove ZIM source |
| `/api/scraper/submit` | POST | Submit URL for scraping |
| `/api/scraper/jobs` | GET | List scraper jobs |
| `/api/scraper/cancel/<id>` | POST | Cancel running job |
| `/api/scraper/retry/<id>` | POST | Retry failed job |
| `/api/scraper/delete/<id>` | POST | Delete job record |
| `/api/scraper/clear-failed` | POST | Purge all failed jobs |
### Knowledge pipeline
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/upload` | POST | Upload PDF/document |
| `/api/upload/<hash>/status` | GET | Check upload processing status |
| `/api/ingest-url` | POST | Ingest single URL |
| `/api/ingest-urls` | POST | Batch URL ingest |
| `/api/crawl` | POST | Start web crawl |
| `/api/crawl/<id>/status` | GET | Crawl progress |
| `/api/search` | POST | Semantic search (Qdrant) |
| `/api/knowledge-stats` | GET | Pipeline statistics |
| `/api/quick-stats` | GET | Dashboard quick stats |
| `/api/status` | GET | Full pipeline status |
| `/api/health` | GET | Service health check |
| `/api/metrics/history` | GET | Historical metrics |
### PeerTube integration
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/ingest-peertube` | POST | Ingest PeerTube channel |
| `/api/peertube/stats` | GET | PeerTube ingest stats |
| `/api/peertube/channels` | GET | List tracked channels |
| `/api/peertube/channels/add` | POST | Add channel |
| `/api/peertube/dashboard` | GET | PeerTube dashboard data |
### Settings / Admin
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/keys` | GET/POST | API key management |
| `/api/cookies/status` | GET | YouTube cookie status |
| `/api/cookies/upload` | POST | Upload cookies |
| `/api/vpn/status` | GET | NordVPN status |
| `/api/vpn/rotate` | POST | Rotate VPN IP |
| `/api/service/restart` | POST | Restart RECON service |
## 8. Navi Frontend Feature State
### Working
- Forward geocode search (address book → Netsyms → Photon cascade)
- Multi-stop turn-by-turn routing (Valhalla, draggable stops)
- Map-click reverse geocode with pin drop
- Place detail panel (address, coords, city/state)
- Copy popover (address / lat,lon / Plus Code)
- GPS origin (browser geolocation as route start)
- Light/dark theme with design tokens
- SPA with tile serving (PMTiles, North America)
- API proxy through nginx (no CORS issues)
- Deployment profiles + `/api/config` endpoint (home, regional_pi, minimal_pi)
- Layer control popover with hillshade and traffic toggles
- Traffic overlay live (TomTom flow tiles via backend proxy, 120s cache)
- Nominatim Idaho infrastructure standing by for C2b proxy wiring
- Place detail enrichment panel (C2b+C2c: category badge, hours, contact, details, wiki links via Nominatim/Overpass)
- Hillshade terrain overlay (93 GB terrarium DEM, toggle in layer control, renders under vector basemap in both themes)
### Stubbed / Partial
- **Save button** — shows toast "Coming soon", not wired to address book write API
- **Share** — UI present, no implementation
### Planned
| Phase | Feature |
|-------|---------|
| C2a.2 | Nominatim coverage expansion to 11 western states (ID + MT, WY, NV, UT, OR, WA, CO, NM, AZ, CA) |
| C2.5 | Save wiring — persist places to address_book.yaml |
| C3 | Kiwix Wikipedia summary integration in place panel |
| F | PAD-US parcel/public-land data layer |
| N | Aurora place-details tool extension |
## 9. Operational Runbook
### Deploy Navi frontend
```bash
ssh zvx@192.168.1.130
cd /home/zvx/projects/repos/navi
npm run build && rsync -av --delete dist/ /mnt/nav/frontend/
```
### Restart RECON backend
```bash
ssh zvx@192.168.1.130
sudo systemctl restart recon
# Cache rebuild takes 2-3 min; enrichment/embedding workers restart
```
### Check Photon geocoder
```bash
systemctl is-active photon
curl -s "http://localhost:2322/api?q=filer+idaho&limit=1" | python3 -m json.tool
```
### Check deployment config
```bash
curl -s http://localhost:8420/api/config | python3 -m json.tool
# Returns active profile, feature flags, tile sources, service URLs
```
### Restart Nominatim
```bash
sudo systemctl restart nominatim
# Or directly: docker restart nominatim
curl http://localhost:8010/status.php # Should return "OK"
```
### Query Nominatim locally
```bash
# Search
curl -s "http://localhost:8010/search.php?q=boise&format=json&limit=1" | python3 -m json.tool
# Details by OSM ID
curl -s "http://localhost:8010/details.php?osmtype=R&osmid=121350&addressdetails=1&format=json" | python3 -m json.tool
```
### Back up Nominatim DB
```bash
docker exec nominatim bash -c 'su - postgres -c "pg_dump nominatim" | gzip > /tmp/nom-dump.gz'
docker cp nominatim:/tmp/nom-dump.gz /mnt/nav/nominatim/nominatim-idaho-postimport-$(date +%Y%m%d).dump.gz
# Existing backup: /mnt/nav/nominatim/nominatim-idaho-postimport-20260421.dump.gz (168 MB)
```
### Regenerate na.pmtiles from planet archive
```bash
# IMPORTANT: virtiofs corrupts pmtiles extract output. Extract to /tmp first.
pmtiles extract /mnt/nav/tiles/planet/current.pmtiles /tmp/na.pmtiles \
--bbox=-125.0,24.0,-66.0,50.0
cp /tmp/na.pmtiles /mnt/nav/tiles/na.pmtiles
rm /tmp/na.pmtiles
```
### Restart Valhalla
```bash
docker restart valhalla
curl -s http://localhost:8002/status # Should return 200
```
### Check all Navi endpoints
```bash
curl -s "http://localhost:8420/api/geocode?q=filer+idaho" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/reverse?lat=42.5736&lon=-114.6066" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/address_book/lookup?q=home" | python3 -m json.tool | head -5
curl -s "http://localhost:8420/api/netsyms/health" | python3 -m json.tool
curl -sI http://localhost:8420/api/traffic/flow/10/188/362.png | head -3
curl -s http://localhost:8010/status.php
curl -s http://localhost:8002/status
```
## 10. Known Gotchas
1. **PMTiles + virtiofs corruption** — `pmtiles extract` (v1.30.1) writes all-zero headers
when the output file is on a virtiofs mount. Always extract to `/tmp` first, then `cp`
to the virtiofs-backed path. The copy preserves valid headers.
2. **virtiofsd restart after VM hard-stop** — When VM 1130 is hard-stopped (e.g. OOM hang
requiring `qm stop 1130` from the Proxmox host), the virtiofsd daemons on the data host
die. They must be restarted before the VM can boot:
```bash
sshpass -p '7redditGold' ssh -o PubkeyAuthentication=no root@192.168.1.240 \
'systemctl start virtiofsd-nav virtiofsd-library virtiofsd-kiwix'
# Then: qm start 1130
```
3. **Nominatim flatnode trap** — The mediagis/nominatim container auto-enables `--flat-nodes`
if a `/nominatim/flatnode` directory exists inside the container. This is triggered by
volume mounts, not by environment variables. The flatnode file mmap's 64+ GB and will OOM
on this 24 GB VM. **Never mount a flatnode volume.** For regional imports (state-sized),
the container's default `--cache` mode works fine.
4. **Canonical frontend location** — The Navi frontend repo lives at
`/home/zvx/projects/repos/navi` on **VM 1130 only**. A stale copy on cortex was deleted
2026-04-20. Never recreate there.
5. **Photon JVM memory** — Photon runs with `-Xmx10g` on a 24 GB VM. If the OpenSearch
index is rebuilt, JVM heap pressure can cause OOM kills if other services (RECON, Valhalla,
Nominatim) spike simultaneously. Monitor with `journalctl -u photon -f`.
6. **Tailscale split DNS** — Internal (Tailscale-connected) clients resolve `*.echo6.co`
to Utility Caddy (100.64.0.8) via dnsmasq on Contabo. External DNS resolves to
home public IP (199.6.36.163). Both paths terminate at the same Caddy on CT 101.
7. **RECON restart disruption** — RECON has long-running enrichment/embedding workers and
a cache warm-up phase (~2-3 min). Never restart without confirming no active batch work.
8. **Valhalla is Idaho-only** — Current routing graph is built from `idaho-latest.osm.pbf`.
Routes outside Idaho will fail or produce nonsensical results. Expanding coverage
requires rebuilding tiles from a larger extract.
9. **Address book is YAML, not DB** — Saved places live in `/opt/recon/config/address_book.yaml`.
No concurrent-write safety. The Save button in the frontend is deliberately stubbed until
a proper write path with conflict handling is designed.
10. **Planet tiles are large** — The full planet PMTiles archive is 126 GB. Don't delete
`tiles/planet/` thinking it's unused — it's the source for regional extracts.
12. **Pi-NAS /tmp is a 4 GB tmpfs** — `pmtiles convert` (and other tools) write temp files
to `/tmp` by default. On pi-nas that's a tiny SD-card-backed tmpfs that fills instantly
with large workloads. Always set `TMPDIR=/export/data/...` (the 22 TB data disk) before
running conversion tools on pi-nas.
13. **Hillshade recovery artifact** — The hillshade PMTiles archive copy lives permanently at
`pi-nas:/export/data/nav/hillshade-na.pmtiles` (93 GB). Download script and logs archived
at `pi-nas:/export/data/nav/hillshade-work/`. SHA256:
`7095ef0fbbe42be85b3d5fe86ea13e20cd0a9628af4daa54cb841749b42efc35`.
14. **RECON api.py merge caution** — During the refactored-recon merge, `lib/api.py` had
"both blocks kept" conflict resolution for blueprint registrations. If the merge is ever
redone or the file is forked, manually verify all blueprint registrations remain intact.
## 11. Long-Term Endpoint: Wilderness Navigation
### Goal
User is lost in wilderness — no roads, trails, or visible landmarks. Messages Aurora via
Meshtastic: "route me back home." Receives step-by-step text directions with cardinal
bearings, elevation cues, and landmark references. Runs offline from a Raspberry Pi field node.
### Current proof-of-concept status
| Component | State |
|-----------|-------|
| Meshtastic ↔ Aurora text exchange | Working (MeshAI on CT 108, AIDA-N2 node) |
| Aurora turn-by-turn instruction generation | Working (nav_tools over existing Valhalla routes) |
| Off-network pathfinding | Not built — this is the unique engineering gap |
| Controlled field test | Not attempted |
### Building blocks in place or planned
| Block | Status | Role in wilderness-nav |
|-------|--------|----------------------|
| Terrain DEM (`hillshade-na.pmtiles`, 93 GB) | Live | Cost-surface elevation input |
| Vector basemap (`na.pmtiles`, 31 GB) | Live | Trail data, landmark labels |
| Nominatim + OSM tags | Live (Idaho) | Landmark identification, trail refs |
| Aurora tool router | Live | LLM text-generation layer for directions |
| Meshtastic integration | Live | Delivery channel (text over LoRa) |
| PAD-US public/private land (Phase F) | Planned | Routing constraint — avoid private land |
| Topo contours (Phase T) | Planned | Visual reference layer for user |
| Off-road Valhalla (Phase K1-K2) | Planned | Trail-network routing where trails exist |
| Pi deployment profile (Phase K4) | Planned | Offline field node build |
| NLCD land-cover ingest | Future | Traversability classification (forest, scrub, water, rock) |
### Missing pieces specific to wilderness-nav
1. **Cost-surface data pipeline** — DEM slope + NLCD land cover + hydrography + PAD-US
boundaries → single traversability raster. Each cell gets a movement cost based on
slope steepness, vegetation density, water crossings, and land access restrictions.
2. **Off-network pathfinder** — A* (or Dijkstra) on the cost-surface raster for segments
where no trail or road network exists. Returns a waypoint sequence with bearings and
elevation deltas.
3. **`/api/offroute` endpoint** — Hybrid router combining Valhalla network routing (where
trails exist) with the off-network pathfinder (where they don't). Stitches segments
into a single route with unified instruction format.
4. **Aurora tool extension** — New tool exposing `/api/offroute` to the LLM, with prompt
engineering for cardinal-bearing + landmark instruction style suitable for text-only
delivery over Meshtastic.
5. **Field-testing discipline** — Controlled validation in known wilderness areas before
any real reliance. GPS ground-truth comparison of generated routes.
### Sequencing
Aspirational. Prerequisites F, T, and K1-K2 are in the active planning queue. After those
complete, the wilderness-specific work (cost surface + pathfinder + offroute endpoint +
Aurora integration) is tractable — estimated 1-2 weeks of focused work. Proof-of-concept
validation happens in controlled field tests before any serious commitment.
### Why this endpoint matters
No commercial product combines self-hosted, offline-capable, LLM-generated natural-language
directions, off-network terrain routing, and Meshtastic delivery. This is the reason the
navigation stack exists.