Add navi-admin service (extraction #7) (#7)

* Add navi-admin service (extraction #7)

Net-new fleet admin aggregator on :8427 — no port from recon (recon has no
/api/admin route; Phase A §3). Three @require_auth routes:
  GET /api/admin/fleet            fan-out to all 6 navi-* /api/admin/<svc>/info
                                  + recon /api/health, merged; never 5xx
                                  (failures land in errors[])
  GET /api/admin/recon/info       recon /api/health wrapped in the info shape
  GET /api/admin/navi-admin/info  self-describe

Fan-out forwards the caller's X-Authentik-Username so the @require_auth
upstreams accept it; per-service admin endpoints stay localhost-only (this is
the single edge-exposed admin surface). Service discovery: hardcoded list in
fleet.py (Option B). No secrets, no DB.

Deploy artifacts (NOT applied here): navi-admin.env.example, systemd unit,
nginx ^~ /api/admin snippet, and deploy/caddy notes for the @authed_api edit
(first Caddy change since #2).

12 hermetic tests (fleet happy-path, per-service timeout/500 → errors[],
auth-header forwarding, recon-down degraded-not-5xx, self-info no-secrets,
auth-required). Full monorepo suite green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR #7 review fixes

1. Symmetric degraded-entry handling in fleet.build_fleet — every
   probed service now appears in `services` with a uniform degraded
   dict on failure (matches recon's existing pattern), AND in errors[].
   Operators see "everything I tried + which broke" consistently.
2. Catch ValueError specifically in _get_json — non-JSON 200 responses
   now surface as `error: 'invalid JSON'` instead of opaque 'ValueError'.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR #7 review fixes (round 2)

1. Unified degraded shape: wrap_recon_health calls _degraded_entry on
   failure — no more runtime.status vs runtime.recon_status asymmetry.
   Every probed service has the same shape on failure
   (runtime.status == 'unreachable'). recon-specific runtime fields
   (recon_status/recon_uptime/pipeline) remain only on the success path.
2. DRY'd git short-SHA helper into shared/git_sha.py — was duplicated in
   7 service app.py files + fleet.recon_git_sha. One implementation,
   one place to fix when behavior changes. Adds shared/tests (testpaths
   now includes "shared").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: zvx-echo6 <mj@k7zvx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/README.md

@@ -69,6 +69,25 @@ public. The reverse bundle fans out to Photon, the SpatiaLite timezone DB,
 navi-landclass (HTTP), and the planet-DEM PMTiles — each degrading to `null`
 independently, never 5xx.
 
+## Run (local) — navi-admin (extraction #7)
+
+```bash
+.venv/bin/pytest services/navi_admin/tests/ -v
+
+# No secrets — read-only HTTP fan-out over localhost (see
+# deploy/env/navi-admin.env.example). Owns no DB.
+.venv/bin/gunicorn 'services.navi_admin.app:create_app()' \
+    --bind 127.0.0.1:8427 --workers 2
+```
+
+`navi-admin` is the fleet admin front door: `/api/admin/fleet` fans out to every
+navi-* service's localhost `/api/admin/<svc>/info` + recon's `/api/health`
+(merged, never 5xx — failures land in `errors[]`); `/api/admin/recon/info` wraps
+recon's health into the uniform shape; `/api/admin/navi-admin/info` self-describes.
+All `@require_auth`. The per-service admin endpoints stay localhost-only; this is
+the single edge-exposed admin surface (needs a Caddy `@authed_api` edit — see
+`deploy/caddy/navi-admin.caddy.notes.md`).
+
 ## The admin-info convention (§4.5)
 
 Every service exposes `GET /api/admin/<service-name>/info`, gated by `require_auth`,

backend/deploy/caddy/navi-admin.caddy.notes.md

@@ -0,0 +1,54 @@
+# navi-admin — Caddy edit notes (deploy task, NOT done in the PR)
+
+navi-admin's `/api/admin/*` is **auth-gated** (every admin-info is `@require_auth`).
+Today the `navi.echo6.co` Caddy block on **CT 101** (`192.168.1.241 → pct 101`,
+`/etc/caddy/Caddyfile`) routes `/api/admin/*` through `@public_api` (path `/api/*`)
+with **no auth** → it would serve the fleet/admin endpoints unauthenticated.
+
+So the deploy task must add `/api/admin/*` to the `@authed_api` matcher. **This is
+the first Caddy edit since extraction #2** (Phase A §6/§8).
+
+## The edit
+
+Inside the `navi.echo6.co { ... }` block, the `@authed_api` matcher:
+
+**Before**
+```caddyfile
+@authed_api {
+    path /api/contacts /api/contacts/* /api/auth/whoami /api/traffic /api/traffic/*
+}
+```
+
+**After** (append `/api/admin/*`)
+```caddyfile
+@authed_api {
+    path /api/contacts /api/contacts/* /api/auth/whoami /api/traffic /api/traffic/* /api/admin/*
+}
+```
+
+No other change. `@authed_api` already runs `forward_auth https://auth.echo6.co`
+then `reverse_proxy 100.64.0.24:8440`; adding the path makes `/api/admin/*` take
+that authed path instead of falling through to `@public_api`. Because Caddy
+evaluates the `handle` blocks in source order and `@authed_api` is defined before
+`@public_api`, the new path wins for `/api/admin/*` while `/api/*` still catches
+everything else publicly.
+
+## Apply (on CT 101)
+
+```bash
+ssh root@192.168.1.241 "pct exec 101 -- caddy validate --config /etc/caddy/Caddyfile"
+ssh root@192.168.1.241 "pct exec 101 -- systemctl reload caddy"   # acme/admin off → reload is fine here
+```
+
+## Why this is the only Caddy entry needed
+
+The per-service `/api/admin/<svc>/info` endpoints stay **localhost-only** — they
+are never edge-routed (nginx has no per-service admin block; navi-admin reaches
+them over `127.0.0.1`). Only navi-admin's single `/api/admin` front door is
+edge-exposed, so only this one path needs adding to `@authed_api`.
+
+## nginx side (VM 1130)
+
+Pair this with the `^~ /api/admin → 127.0.0.1:8427` block in
+`deploy/nginx/navi-admin.conf.snippet`, added before the `location /api/`
+catch-all in `/etc/nginx/sites-available/navi.echo6.co`.

backend/deploy/env/navi-admin.env.example

@@ -0,0 +1,21 @@
+# navi-admin — /etc/navi-backend/navi-admin.env
+# Net-new fleet admin aggregator (extraction #7). NO SECRETS — read-only HTTP
+# fan-out over localhost to each navi-* service's /api/admin/<svc>/info plus
+# recon's /api/health. Owns no DB / no writable state (nothing under
+# /var/lib/navi-backend/).
+
+# recon's pipeline-health endpoint, wrapped by /api/admin/recon/info.
+RECON_HEALTH_URL=http://127.0.0.1:8420/api/health
+
+# recon's deploy clone, for git-SHA discovery in /api/admin/recon/info.
+# NOTE: recon deploys at /opt/recon on VM 1130 (a git repo, readable by zvx) —
+# NOT /home/zvx/projects/repos/recon as the Phase B prompt's example assumed
+# (that path doesn't exist). Verified during the build; see PR description.
+RECON_REPO_PATH=/opt/recon
+
+# OPTIONAL: per-service fan-out timeout (seconds). Default 3.0 in code.
+# NAVI_ADMIN_FANOUT_TIMEOUT_S=3
+
+# Service discovery is a hardcoded list in services/navi_admin/fleet.py
+# (Option B) — the set of navi-* services only changes when a new extraction
+# ships, the same moment we'd edit this service to add it. No env list.

backend/deploy/nginx/navi-admin.conf.snippet

@@ -0,0 +1,30 @@
+# =============================================================================
+# navi-admin — nginx integration for the navi.echo6.co vhost
+#
+# ONE block. Add INSIDE the existing
+#     server { server_name navi.echo6.co; ... }
+# block, BEFORE the existing `location /api/ { ... }` catch-all (which currently
+# sends /api/admin/* to recon:8420 → 404, since recon has no /api/admin route).
+#
+# `^~ /api/admin` (NO trailing slash): the `^~` (extraction-#1 lesson) makes
+# nginx skip regex evaluation; the no-slash prefix catches /api/admin/fleet,
+# /api/admin/recon/info, and /api/admin/navi-admin/info.
+#
+# IMPORTANT — this route is AUTH-GATED, unlike the public navi-* prefixes. It
+# needs the matching Caddy edit (see deploy/caddy/navi-admin.caddy.notes.md):
+# add /api/admin/* to @authed_api so forward_auth runs before this proxy. The
+# X-Authentik-Username header set below is what navi-admin forwards on its
+# localhost fan-out to each per-service @require_auth /api/admin/<svc>/info.
+#
+# The per-service /api/admin/<svc>/info endpoints stay localhost-only — they are
+# NOT exposed here. This single /api/admin front door (the fleet aggregator) is
+# the only edge-reachable admin surface.
+# -----------------------------------------------------------------------------
+    location ^~ /api/admin {
+        proxy_pass http://127.0.0.1:8427;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Authentik-Username $http_x_authentik_username;
+        proxy_read_timeout 15s;
+        add_header X-Cache-Status BYPASS;
+    }

backend/deploy/systemd/navi-admin.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=navi-admin — fleet admin-info aggregator (Echo6 navi-backend, extraction #7)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+User=zvx
+WorkingDirectory=/home/zvx/projects/repos/navi-backend
+EnvironmentFile=/etc/navi-backend/navi-admin.env
+ExecStart=/home/zvx/projects/repos/navi-backend/.venv/bin/gunicorn 'services.navi_admin.app:create_app()' --bind 127.0.0.1:8427 --workers 2
+Restart=on-failure
+RestartSec=2
+
+[Install]
+WantedBy=multi-user.target

backend/pyproject.toml

@@ -32,4 +32,4 @@ include = ["shared*", "services*"]
 namespaces = true
 
 [tool.pytest.ini_options]
-testpaths = ["services"]
+testpaths = ["services", "shared"]

backend/services/navi_admin/admin_route.py

@@ -0,0 +1,76 @@
+"""navi-admin routes — the fleet admin front door (extraction #7).
+
+Net-new (recon has no admin-info to port — Phase A §3). Three routes, all
+``@require_auth`` (auth is also enforced at the Caddy edge in prod; the Flask
+gate matches every other navi-* service and protects the localhost :8427 path):
+
+  GET /api/admin/recon/info       recon's /api/health wrapped in the info shape
+  GET /api/admin/fleet            fan-out aggregator across all navi-* + recon
+  GET /api/admin/navi-admin/info  navi-admin's own admin-info (self-describe)
+
+The per-service /api/admin/<svc>/info endpoints stay localhost-only (Phase A
+§3/§7); this fleet endpoint is the single edge-exposed front door.
+"""
+import os
+import time
+
+from flask import Blueprint, jsonify, current_app, request
+
+from shared.auth import require_auth
+from shared.admin_info import build_info_response
+
+from . import fleet
+
+bp = Blueprint('navi_admin', __name__)
+
+PORT = 8427
+
+
+@bp.route('/api/admin/recon/info')
+@require_auth
+def recon_info():
+    """recon's pipeline /api/health, wrapped into the uniform info shape.
+    recon down → a degraded info dict, not a 5xx."""
+    _, health, error = fleet.probe('recon', fleet.recon_health_url(), request.user_id)
+    return jsonify(fleet.wrap_recon_health(health, error))
+
+
+@bp.route('/api/admin/fleet')
+@require_auth
+def fleet_info():
+    """Fan out to every navi-* service + recon over localhost, merged. Forwards
+    the caller's X-Authentik-Username so the @require_auth upstreams accept it.
+    Never 5xx — per-service failures land in `errors`."""
+    return jsonify(fleet.build_fleet(request.user_id))
+
+
+@bp.route('/api/admin/navi-admin/info')
+@require_auth
+def navi_admin_info():
+    """navi-admin's own admin-info. No secrets (Phase A §9) — only non-secret
+    URLs/paths. `dependencies` reuses the same probes the fleet runs."""
+    metrics = current_app.config['METRICS']
+    info = build_info_response(
+        service='navi-admin',
+        version=current_app.config.get('VERSION', 'unknown'),
+        port=PORT,
+        # What it aggregates — non-secret, documents the fleet membership.
+        config={
+            'fanned_services': [{'name': n, 'port': p} for n, p in fleet.SERVICES],
+            'recon': {'name': fleet.RECON_SERVICE_NAME, 'port': fleet.RECON_PORT,
+                      'git_sha': fleet.recon_git_sha()},
+        },
+        env=[
+            {'name': 'RECON_HEALTH_URL', 'value': fleet.recon_health_url()},
+            {'name': 'RECON_REPO_PATH', 'value': fleet.recon_repo_path()},
+            {'name': 'NAVI_ADMIN_FANOUT_TIMEOUT_S', 'value': str(fleet.fanout_timeout())},
+        ],
+        dependencies=fleet.dependency_summaries(request.user_id),
+        filesystem=[],   # navi-admin owns no files / no DB (Phase A §9)
+        runtime={
+            'uptime_s': round(time.time() - metrics['start_time'], 1),
+            'request_count': metrics['request_count'],
+            'last_error_at': metrics['last_error_at'],
+        },
+    )
+    return jsonify(info)

backend/services/navi_admin/app.py

@@ -0,0 +1,38 @@
+"""navi-admin Flask application factory + gunicorn entry.
+
+Gunicorn entry:
+    gunicorn 'services.navi_admin.app:create_app()' --bind 127.0.0.1:8427 --workers 2
+"""
+import time
+
+from flask import Flask
+
+from shared.git_sha import git_short_sha
+
+from . import admin_route
+
+
+def create_app():
+    app = Flask(__name__)
+
+    app.config['VERSION'] = git_short_sha()
+    app.config['METRICS'] = {
+        'start_time': time.time(),
+        'request_count': 0,
+        'last_error_at': None,
+    }
+
+    @app.before_request
+    def _count_request():
+        app.config['METRICS']['request_count'] += 1
+
+    @app.after_request
+    def _track_errors(response):
+        if response.status_code >= 500:
+            app.config['METRICS']['last_error_at'] = time.strftime(
+                '%Y-%m-%dT%H:%M:%SZ', time.gmtime()
+            )
+        return response
+
+    app.register_blueprint(admin_route.bp)
+    return app

backend/services/navi_admin/fleet.py

@@ -0,0 +1,189 @@
+"""Fleet fan-out + recon-health wrapping for navi-admin.
+
+navi-admin is a stateless aggregator: it fans out over localhost to each
+navi-* service's ``/api/admin/<svc>/info`` endpoint (and recon's pipeline
+``/api/health``), merging them into one fleet response. Every per-service admin
+endpoint is ``@require_auth``, so the fan-out forwards the caller's validated
+``X-Authentik-Username`` header — otherwise the upstreams would 401.
+
+Service discovery: a hardcoded module-level list (Option B). The set of navi-*
+services changes only when we ship a new extraction — the same moment we'd be
+editing this file to add it — so an env list would add a moving part with no
+payoff. (One source of truth: the ports/names live here only.)
+"""
+import os
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+import requests
+
+from shared.git_sha import git_short_sha
+
+# (service-name, port) for every shipped navi-* service. The admin-info path is
+# always /api/admin/<service-name>/info. Add a row when a new extraction ships.
+SERVICES = [
+    ('navi-traffic', 8421),    # #1 TomTom traffic tile proxy
+    ('navi-config', 8422),     # #2 deployment profile API
+    ('navi-contacts', 8423),   # #3 contacts + address book
+    ('navi-landclass', 8424),  # #4 PAD-US land classification
+    ('navi-places', 8425),     # #5 OSM place detail + enrichment
+    ('navi-geo', 8426),        # #6 geocode + reverse + reverse bundle
+]
+
+RECON_SERVICE_NAME = 'recon'
+RECON_PORT = 8420
+
+DEFAULT_RECON_HEALTH_URL = 'http://127.0.0.1:8420/api/health'
+DEFAULT_RECON_REPO_PATH = '/opt/recon'   # actual deploy path on VM 1130 (a git repo)
+DEFAULT_FANOUT_TIMEOUT_S = 3.0
+
+
+def recon_health_url():
+    return os.environ.get('RECON_HEALTH_URL', DEFAULT_RECON_HEALTH_URL)
+
+
+def recon_repo_path():
+    return os.environ.get('RECON_REPO_PATH', DEFAULT_RECON_REPO_PATH)
+
+
+def fanout_timeout():
+    try:
+        return float(os.environ.get('NAVI_ADMIN_FANOUT_TIMEOUT_S', DEFAULT_FANOUT_TIMEOUT_S))
+    except (ValueError, TypeError):
+        return DEFAULT_FANOUT_TIMEOUT_S
+
+
+def service_info_url(name, port):
+    return f'http://127.0.0.1:{port}/api/admin/{name}/info'
+
+
+def recon_git_sha():
+    """recon's deployed git SHA, or 'unknown'. Reads from recon_repo_path()
+    (default /opt/recon on VM 1130, Phase A-verified)."""
+    return git_short_sha(recon_repo_path())
+
+
+def _get_json(url, auth_user, timeout):
+    """GET url, forwarding the auth header. Returns (json_or_None, latency_ms,
+    error_or_None) where error is 'timeout' | 'HTTP <code>' | exception name."""
+    headers = {'X-Authentik-Username': auth_user} if auth_user else {}
+    start = time.monotonic()
+    try:
+        resp = requests.get(url, headers=headers, timeout=timeout)
+        latency_ms = round((time.monotonic() - start) * 1000, 1)
+        if resp.status_code != 200:
+            return None, latency_ms, f'HTTP {resp.status_code}'
+        return resp.json(), latency_ms, None
+    except requests.Timeout:
+        return None, round((time.monotonic() - start) * 1000, 1), 'timeout'
+    except ValueError:
+        # 200 with a non-JSON body (e.g. a misrouted upstream serving HTML).
+        # json.JSONDecodeError subclasses ValueError — report it plainly.
+        return None, round((time.monotonic() - start) * 1000, 1), 'invalid JSON'
+    except Exception as exc:
+        return None, round((time.monotonic() - start) * 1000, 1), type(exc).__name__
+
+
+def probe(name, url, auth_user, timeout=None):
+    """One GET → (summary, full_json, error). summary is the {name, status,
+    latency_ms[, error]} shape the per-service admin endpoints use for deps."""
+    full, latency_ms, error = _get_json(url, auth_user, timeout or fanout_timeout())
+    summary = {'name': name, 'status': 'ok' if error is None else 'error',
+               'latency_ms': latency_ms}
+    if error:
+        summary['error'] = error
+    return summary, full, error
+
+
+def _degraded_entry(name, port, error):
+    """Uniform 'service was probed but failed' entry — matches the
+    build_info_response shape so callers see the same keys whether the service is
+    healthy or down. Used for every failure path (navi-* AND recon), so there is
+    exactly one degraded shape: runtime.status == 'unreachable'."""
+    return {
+        'service': name, 'version': 'unknown', 'port': port,
+        'config': {}, 'env': [],
+        'dependencies': [{'name': f'{name}-info', 'status': 'error', 'error': error}],
+        'filesystem': [],
+        'runtime': {'status': 'unreachable'},
+    }
+
+
+def wrap_recon_health(health, error):
+    """Wrap recon's /api/health into an admin-info-shaped dict (service:'recon').
+
+    recon has no admin-info endpoint (Phase A §3); /api/health is the closest
+    input. On success its components become `dependencies` and its pipeline/status
+    become recon-specific `runtime` fields (meaningful only when recon is up). On
+    failure → the same uniform _degraded_entry shape as every other service (no
+    recon-specific data exists to preserve when recon is unreachable)."""
+    if health is None:
+        return _degraded_entry(RECON_SERVICE_NAME, RECON_PORT, error or 'unreachable')
+    version = recon_git_sha()
+    components = health.get('components', {})
+    dependencies = [
+        {'name': cname, 'status': cval.get('status'), **{k: v for k, v in cval.items() if k != 'status'}}
+        for cname, cval in components.items()
+    ]
+    return {
+        'service': RECON_SERVICE_NAME, 'version': version, 'port': RECON_PORT,
+        'config': {}, 'env': [],
+        'dependencies': dependencies,
+        'filesystem': [],
+        'runtime': {
+            'recon_status': health.get('status'),
+            'recon_uptime': health.get('uptime'),
+            'pipeline': health.get('pipeline', {}),
+        },
+    }
+
+
+def build_fleet(auth_user):
+    """Fan out to all navi-* services + recon in parallel; merge. Never raises.
+
+    Returns {services: {<name>: <info>}, fetched_at: ISO8601, errors: [{service, error}]}.
+    Invariant: EVERY probed service appears in `services` — a full info dict when
+    healthy, the uniform `_degraded_entry` shape (runtime.status == 'unreachable')
+    when it fails — and `errors` is a parallel listing of which ones failed and
+    why. No special cases: recon failures use the same degraded shape."""
+    timeout = fanout_timeout()
+    targets = [(name, port, service_info_url(name, port)) for name, port in SERVICES]
+
+    def _fetch(name, port, url):
+        _, full, error = probe(name, url, auth_user, timeout)
+        return name, port, full, error
+
+    services = {}
+    errors = []
+    with ThreadPoolExecutor(max_workers=len(targets) + 1) as ex:
+        futures = [ex.submit(_fetch, name, port, url) for name, port, url in targets]
+        recon_future = ex.submit(probe, RECON_SERVICE_NAME, recon_health_url(), auth_user, timeout)
+        for fut in futures:
+            name, port, full, error = fut.result()
+            services[name] = full if full is not None else _degraded_entry(name, port, error)
+            if error:
+                errors.append({'service': name, 'error': error})
+        _, recon_health, recon_error = recon_future.result()
+        services[RECON_SERVICE_NAME] = wrap_recon_health(recon_health, recon_error)
+        if recon_error:
+            errors.append({'service': RECON_SERVICE_NAME, 'error': recon_error})
+
+    return {
+        'services': services,
+        'fetched_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
+        'errors': errors,
+    }
+
+
+def dependency_summaries(auth_user):
+    """{name, status, latency_ms[, error]} for recon-health + each navi-* admin
+    endpoint — the same probes the fleet runs, reused for navi-admin's own
+    /info `dependencies`. Sequential (it's a rare, auth-gated call)."""
+    timeout = fanout_timeout()
+    summaries = []
+    s, _, _ = probe('recon-health', recon_health_url(), auth_user, timeout)
+    summaries.append(s)
+    for name, port in SERVICES:
+        s, _, _ = probe(name, service_info_url(name, port), auth_user, timeout)
+        summaries.append(s)
+    return summaries

backend/services/navi_admin/tests/test_admin.py

@@ -0,0 +1,212 @@
+"""Hermetic tests for navi-admin's fleet aggregator (extraction #7).
+
+No live calls: services.navi_admin.fleet.requests.get is mocked to a router
+keyed on the port in the URL, so we exercise the real _get_json mapping
+(timeout -> 'timeout', non-200 -> 'HTTP <code>') and the real fan-out/merge.
+recon_git_sha is stubbed so tests don't depend on /opt/recon.
+"""
+import pytest
+import requests
+
+import services.navi_admin.fleet as fleet
+from services.navi_admin.app import create_app
+
+AUTH = {'X-Authentik-Username': 'matt'}
+
+
+def _svc_info(name, port):
+    """A canonical per-service admin-info (build_info_response shape)."""
+    return {'service': name, 'version': 'abc1234', 'port': port, 'config': {},
+            'env': [], 'dependencies': [], 'filesystem': [],
+            'runtime': {'uptime_s': 1.0, 'request_count': 0, 'last_error_at': None}}
+
+
+def _recon_health(status='healthy'):
+    return {'status': status, 'uptime': '2026-05-23T00:00:00Z',
+            'components': {'qdrant': {'status': 'up', 'vectors': 42},
+                           'tei': {'status': 'up'},
+                           'nfs': {'status': 'up'}},
+            'pipeline': {'total': 100, 'done': 90}}
+
+
+class _FakeResp:
+    def __init__(self, status_code, json_data):
+        self.status_code = status_code
+        self._json = json_data
+
+    def json(self):
+        return self._json
+
+
+def _router(behaviors, captured_headers):
+    """behaviors: {port:int -> ('ok', json) | ('http', code) | 'timeout' | 'conn'}."""
+    def fake_get(url, headers=None, timeout=None):
+        captured_headers.append(headers or {})
+        for port, behavior in behaviors.items():
+            if f':{port}/' in url:
+                if behavior == 'timeout':
+                    raise requests.Timeout()
+                if behavior == 'conn':
+                    raise requests.ConnectionError()
+                kind, payload = behavior
+                if kind == 'http':
+                    return _FakeResp(payload, {})
+                return _FakeResp(200, payload)
+        raise AssertionError(f'unexpected GET {url}')
+    return fake_get
+
+
+def _all_ok():
+    """Every navi-* service 200 + recon health 200."""
+    b = {port: ('ok', _svc_info(name, port)) for name, port in fleet.SERVICES}
+    b[fleet.RECON_PORT] = ('ok', _recon_health())
+    return b
+
+
+@pytest.fixture
+def client(monkeypatch):
+    monkeypatch.setattr(fleet, 'recon_git_sha', lambda: 'recon99')
+    return create_app().test_client()
+
+
+@pytest.fixture
+def captured():
+    return []
+
+
+def _wire(monkeypatch, behaviors, captured):
+    monkeypatch.setattr(fleet.requests, 'get', _router(behaviors, captured))
+
+
+# ── fleet fan-out ─────────────────────────────────────────────────────────
+
+def test_fleet_happy_path(client, monkeypatch, captured):
+    _wire(monkeypatch, _all_ok(), captured)
+    data = client.get('/api/admin/fleet', headers=AUTH).get_json()
+    expected = {name for name, _ in fleet.SERVICES} | {'recon'}
+    assert set(data['services'].keys()) == expected
+    assert data['errors'] == []
+    assert data['fetched_at'].endswith('Z')
+    assert data['services']['navi-geo']['port'] == 8426
+    assert data['services']['recon']['runtime']['recon_status'] == 'healthy'
+
+
+def test_fleet_service_timeout_lands_in_errors(client, monkeypatch, captured):
+    b = _all_ok()
+    b[8425] = 'timeout'                       # navi-places times out
+    _wire(monkeypatch, b, captured)
+    data = client.get('/api/admin/fleet', headers=AUTH).get_json()
+    # Invariant: still present in services, as a uniform degraded entry...
+    assert data['services']['navi-places']['runtime']['status'] == 'unreachable'
+    assert data['services']['navi-places']['port'] == 8425
+    # ...and recorded in errors.
+    assert {'service': 'navi-places', 'error': 'timeout'} in data['errors']
+
+
+def test_fleet_service_http_500_lands_in_errors(client, monkeypatch, captured):
+    b = _all_ok()
+    b[8426] = ('http', 500)                   # navi-geo 500s
+    _wire(monkeypatch, b, captured)
+    data = client.get('/api/admin/fleet', headers=AUTH).get_json()
+    assert data['services']['navi-geo']['runtime']['status'] == 'unreachable'
+    assert {'service': 'navi-geo', 'error': 'HTTP 500'} in data['errors']
+
+
+def test_fleet_forwards_auth_header(client, monkeypatch, captured):
+    _wire(monkeypatch, _all_ok(), captured)
+    client.get('/api/admin/fleet', headers=AUTH)
+    assert captured and all(h.get('X-Authentik-Username') == 'matt' for h in captured)
+
+
+def test_fleet_never_5xx_when_recon_down_and_a_service_errors(client, monkeypatch, captured):
+    b = _all_ok()
+    b[fleet.RECON_PORT] = 'timeout'           # recon health down
+    b[8421] = ('http', 502)                   # navi-traffic 502
+    _wire(monkeypatch, b, captured)
+    resp = client.get('/api/admin/fleet', headers=AUTH)
+    assert resp.status_code == 200            # never 5xx
+    data = resp.get_json()
+    # recon still present as a degraded entry (same uniform shape), AND in errors
+    assert data['services']['recon']['runtime']['status'] == 'unreachable'
+    assert {'service': 'recon', 'error': 'timeout'} in data['errors']
+    # navi-traffic also present (degraded) AND in errors — the uniform invariant
+    assert data['services']['navi-traffic']['runtime']['status'] == 'unreachable'
+    assert {'service': 'navi-traffic', 'error': 'HTTP 502'} in data['errors']
+
+
+def test_fleet_service_returns_html_lands_as_invalid_json(client, monkeypatch, captured):
+    """200 OK with a non-JSON body (e.g. a misrouted upstream returning HTML) must
+    surface as 'invalid JSON' in errors[], not the opaque 'ValueError'."""
+    b = _all_ok()
+
+    class _HtmlResp:
+        status_code = 200
+
+        def json(self):
+            raise ValueError('not JSON')
+
+    def fake_get(url, headers=None, timeout=None):
+        captured.append(headers or {})
+        if ':8425/' in url:
+            return _HtmlResp()
+        return _router(b, [])(url, headers=headers, timeout=timeout)
+
+    monkeypatch.setattr(fleet.requests, 'get', fake_get)
+    data = client.get('/api/admin/fleet', headers=AUTH).get_json()
+    assert {'service': 'navi-places', 'error': 'invalid JSON'} in data['errors']
+    assert data['services']['navi-places']['runtime']['status'] == 'unreachable'
+
+
+# ── recon/info wrapper ──────────────────────────────────────────────────────
+
+def test_recon_info_wraps_health(client, monkeypatch, captured):
+    _wire(monkeypatch, {fleet.RECON_PORT: ('ok', _recon_health())}, captured)
+    data = client.get('/api/admin/recon/info', headers=AUTH).get_json()
+    assert data['service'] == 'recon' and data['port'] == 8420
+    assert data['version'] == 'recon99'
+    dep_names = {d['name'] for d in data['dependencies']}
+    assert {'qdrant', 'tei', 'nfs'} <= dep_names
+    assert data['runtime']['recon_status'] == 'healthy'
+    assert data['runtime']['pipeline'] == {'total': 100, 'done': 90}
+
+
+def test_recon_info_recon_down_is_degraded_not_5xx(client, monkeypatch, captured):
+    _wire(monkeypatch, {fleet.RECON_PORT: 'timeout'}, captured)
+    resp = client.get('/api/admin/recon/info', headers=AUTH)
+    assert resp.status_code == 200            # degraded, not 5xx
+    data = resp.get_json()
+    assert data['service'] == 'recon'
+    # Uniform degraded shape (same as every other service) — no recon_status special case.
+    assert data['runtime']['status'] == 'unreachable'
+    assert data['dependencies'][0]['status'] == 'error'
+    assert data['dependencies'][0]['name'] == 'recon-info'
+
+
+# ── self-info ───────────────────────────────────────────────────────────────
+
+def test_self_info_no_secrets_and_lists_deps(client, monkeypatch, captured):
+    _wire(monkeypatch, _all_ok(), captured)
+    data = client.get('/api/admin/navi-admin/info', headers=AUTH).get_json()
+    assert data['service'] == 'navi-admin' and data['port'] == 8427
+    assert data['filesystem'] == []           # owns no files/DB
+    # No masked secrets anywhere in env (Phase A §9 — none exist).
+    assert all('...' not in str(e['value']) and e['value'] != '****' for e in data['env'])
+    dep_names = {d['name'] for d in data['dependencies']}
+    assert 'recon-health' in dep_names
+    assert {name for name, _ in fleet.SERVICES} <= dep_names
+
+
+def test_self_info_config_lists_fanned_services(client, monkeypatch, captured):
+    _wire(monkeypatch, _all_ok(), captured)
+    data = client.get('/api/admin/navi-admin/info', headers=AUTH).get_json()
+    fanned = {s['name']: s['port'] for s in data['config']['fanned_services']}
+    assert fanned == {name: port for name, port in fleet.SERVICES}
+    assert data['config']['recon']['git_sha'] == 'recon99'
+
+
+# ── auth gating ─────────────────────────────────────────────────────────────
+
+@pytest.mark.parametrize('path', [
+    '/api/admin/fleet', '/api/admin/recon/info', '/api/admin/navi-admin/info'])
+def test_auth_required(client, path):
+    assert client.get(path).status_code == 401

backend/services/navi_config/app.py

@@ -3,35 +3,23 @@
 Gunicorn entry:
     gunicorn 'services.navi_config.app:create_app()' --bind 127.0.0.1:8422 --workers 2
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import config_route, admin
 from .config_loader import reset_cache
 
 
-def _git_sha():
-    """Short git SHA of the working tree at startup, or 'unknown' off-repo."""
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL,
-            text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
     # Version + lightweight runtime metrics, read by the admin-info endpoint.
     # NOTE: with gunicorn --workers 2 these counters are per-worker (each worker
     # is its own process); they are indicative, not cluster-wide totals.
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/services/navi_contacts/app.py

@@ -6,34 +6,22 @@ Gunicorn entry:
 Serves two blueprints: contacts (10 routes, auth-gated) and address_book
 (2 routes, public), plus the §4.5 admin-info endpoint.
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import contacts_route, address_book_route, admin
 from . import address_book as address_book_mod
 
 
-def _git_sha():
-    """Short git SHA of the working tree at startup, or 'unknown' off-repo."""
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL,
-            text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
     # Version + lightweight runtime metrics, read by the admin-info endpoint.
     # NOTE: with gunicorn --workers 2 these counters are per-worker.
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/services/navi_geo/app.py

@@ -3,32 +3,22 @@
 Gunicorn entry:
     gunicorn 'services.navi_geo.app:create_app()' --bind 127.0.0.1:8426 --workers 2
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import geo_route, admin
 from . import geocode as geocode_mod
 from . import netsyms
 from . import address_book
 
 
-def _git_sha():
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL, text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/services/navi_landclass/app.py

@@ -3,34 +3,22 @@
 Gunicorn entry:
     gunicorn 'services.navi_landclass.app:create_app()' --bind 127.0.0.1:8424 --workers 2
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import landclass_route, admin
 from . import db
 
 
-def _git_sha():
-    """Short git SHA of the working tree at startup, or 'unknown' off-repo."""
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL,
-            text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
     # Version + lightweight runtime metrics, read by the admin-info endpoint.
     # NOTE: with gunicorn --workers 2 these counters are per-worker.
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/services/navi_places/app.py

@@ -3,32 +3,22 @@
 Gunicorn entry:
     gunicorn 'services.navi_places.app:create_app()' --bind 127.0.0.1:8425 --workers 2
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import place_route, admin
 from . import overture
 from . import place_cache
 from . import config as places_config
 
 
-def _git_sha():
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL, text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/services/navi_traffic/app.py

@@ -3,34 +3,22 @@
 Gunicorn entry:
     gunicorn 'services.navi_traffic.app:create_app()' --bind 127.0.0.1:8421 --workers 2
 """
-import subprocess
 import time
 
 from flask import Flask
 
+from shared.git_sha import git_short_sha
+
 from . import traffic, admin
 
 
-def _git_sha():
-    """Short git SHA of the working tree at startup, or 'unknown' off-repo."""
-    try:
-        sha = subprocess.check_output(
-            ['git', 'rev-parse', '--short', 'HEAD'],
-            stderr=subprocess.DEVNULL,
-            text=True,
-        ).strip()
-        return sha or 'unknown'
-    except Exception:
-        return 'unknown'
-
-
 def create_app():
     app = Flask(__name__)
 
     # Version + lightweight runtime metrics, read by the admin-info endpoint.
     # NOTE: with gunicorn --workers 2 these counters are per-worker (each worker
     # is its own process); they are indicative, not cluster-wide totals.
-    app.config['VERSION'] = _git_sha()
+    app.config['VERSION'] = git_short_sha()
     app.config['METRICS'] = {
         'start_time': time.time(),
         'request_count': 0,

backend/shared/git_sha.py

@@ -0,0 +1,29 @@
+"""Shared git short-SHA helper.
+
+Used by every navi-* service's create_app() for the `version` field in
+admin-info, and by navi-admin's fleet.recon_git_sha to read recon's deployed
+SHA from its clone path. One implementation, one place to fix when behavior
+needs changing.
+"""
+import subprocess
+
+
+def git_short_sha(repo_path: str | None = None) -> str:
+    """Return ``git rev-parse --short HEAD`` for the given repo path, or for the
+    current working directory if path is None. Returns 'unknown' on any failure
+    (no git, no repo, permission denied, detached HEAD, etc.) — never raises.
+
+    repo_path: explicit repo to query (uses ``git -C <path>``); None = current
+               cwd (the systemd unit's WorkingDirectory in prod).
+    """
+    cmd = ['git']
+    if repo_path is not None:
+        cmd.extend(['-C', repo_path])
+    cmd.extend(['rev-parse', '--short', 'HEAD'])
+    try:
+        sha = subprocess.check_output(
+            cmd, stderr=subprocess.DEVNULL, text=True, timeout=3,
+        ).strip()
+        return sha or 'unknown'
+    except Exception:
+        return 'unknown'

backend/shared/tests/test_git_sha.py

@@ -0,0 +1,14 @@
+"""Hermetic tests for shared.git_sha.git_short_sha."""
+from shared.git_sha import git_short_sha
+
+
+def test_git_short_sha_returns_unknown_on_bad_path(tmp_path):
+    # tmp_path is an empty dir, not a git repo → graceful 'unknown', no raise.
+    assert git_short_sha(str(tmp_path)) == 'unknown'
+
+
+def test_git_short_sha_in_real_repo():
+    # The suite runs from inside the navi-backend repo, so the cwd lookup works.
+    sha = git_short_sha()
+    assert sha != 'unknown'
+    assert len(sha) >= 7   # short SHA, sanity bound