fix(v0.7-fire-tracker-4-revised): rip ?status; LLM DM 7-path verification 3 of 7 pass (NOT verified)

Matt review caught a scope error: ?status was a hypothetical sketch
in the design doc ("a node could ping ?status cache peak") treated as
authorization without asking. Ripping the structured-command path
entirely. The LLM DM path with env_reporter injection is the natural-
language interface; ?status was redundant infrastructure parallel to
the path the design depends on.

What landed:
- router.py: _maybe_rewrite_status_query + _lookup_fire_fuzzy +
  _build_fire_status_context removed. route() restored to:
  bang -> IGNORE-empty -> LLM with verbatim query.
- tests/test_fire_tracker_phase4.py: 5 ?status tests removed; replaced
  with two regression guards:
    test_natural_language_fire_question_routes_to_llm -- "how's the
      cache peak fire?" returns RouteType.LLM with the verbatim query
      (no in-router rewriting).
    test_status_helpers_removed_from_router -- hard-block on
      _maybe_rewrite_status_query / _lookup_fire_fuzzy / "?status"
      appearing anywhere in router.py source. If anyone adds a
      structured-command path for fires, this test fails and the
      author has to talk to Matt first.
- 56 passed in 3.80s across phase1+phase2+phase3+phase4+or-arch+
  include-roundtrip.

What stays (NOT ripped):
- Daily fire digest -- scheduled broadcaster, not a command. Its 4
  adapter_config rows (fires.digest_enabled / digest_schedule /
  digest_timezone / digest_max_chars) stay GUI-editable.
- Bug A fix (UnboundLocalError at router.py:745) -- independent of
  ?status. Confirmed still in effect.

LLM DM 7-path verification result -- 3 of 7 pass, INCOMPLETE:

| # | query                                         | env_reporter         | verdict |
|---|-----------------------------------------------|----------------------|---------|
| 1 | "are there any fires near me?"                | build_fires_detail   | PASS    |
| 2 | "any weather alerts?"                         | build_alerts_detail  | FAIL    |
| 3 | "any earthquakes nearby?"                     | build_quakes_detail  | FAIL    |
| 4 | "how's traffic on I-84?"                      | build_traffic_detail | FAIL    |
| 5 | "what's the snake river level?"               | build_gauges_detail  | PASS    |
| 6 | "what are the band conditions?"               | build_swpc_detail    | PASS    |
| 7 | "why didn't I hear about anything today?"     | build_drop_audit     | FAIL    |

Two distinct failure classes:

Class A -- routing miss (#4 traffic, #7 drop):
  _ENV_KEYWORDS_TO_SUBTYPE lacks "traffic" (only road/jam/crash/
  closure/511/incident map to "traffic"), so a query literally
  mentioning "traffic" never triggers env scope -> build_traffic_detail
  never runs even though traffic_events has 9 rows on disk. The LLM
  fell back to training data and hallucinated I-84 conditions.
  build_drop_audit has no natural-language trigger phrase at all;
  "why didn't I hear about anything today?" has no env keyword.

Class B -- empty data + LLM hallucination (#2 alerts, #3 quakes):
  Env scope IS detected, build_alerts_detail and build_quakes_detail
  DO run, but return empty because nws_alerts has 0 rows and
  quake_events 24h-window has 0 rows (legitimate empty state). The
  LLM has no env block to ground on and hallucinated "144 earthquakes
  worldwide" -- sounds authoritative, is fabricated.

Not fixed in this commit -- needs Matt's call on:
  (a) keyword additions to _ENV_KEYWORDS_TO_SUBTYPE for traffic +
      drop_audit triggers (risk: false-positive env-scope triggers
      for unrelated phrases).
  (b) anti-hallucination prompt clamp: "If a topic's env block is
      missing/empty, say you don't have live data instead of
      answering from general knowledge." (risk: bot apologizes
      every other message.)

Per the "STOP if any path fails" instruction, this commit does NOT
claim verification done; the report at
v0.7-firetracker-phase4.md has the full table + per-row mesh-receiver
wire + per-failure root cause analysis.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson (via Claude) 2026-06-06 07:33:11 +00:00
commit 89640f624d
2 changed files with 52 additions and 197 deletions

View file

@ -131,57 +131,63 @@ def test_render_digest_uses_llm_when_available():
# ===========================================================================
# Fuzzy fire lookup for ?status
# Natural-language fire DMs route to the LLM (no ?status fallback)
# ===========================================================================
def test_status_lookup_exact_name():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
f = _lookup_fire_fuzzy("Cache Peak")
assert f is not None
assert f["incident_name"] == "Cache Peak"
def test_natural_language_fire_question_routes_to_llm():
"""The LLM DM path is the sole interface for natural-language fire
questions. Pre-revised commit there was a `?status` intent that
rewrote the query in-router; this test confirms the rewrite is gone
and that a plain English question is forwarded verbatim."""
import asyncio
from meshai.router import MessageRouter, RouteType
from meshai.config_loader import load_config
from meshai.history import ConversationHistory
from meshai.commands.dispatcher import create_dispatcher
cfg = load_config()
history = ConversationHistory(cfg.history)
async def _run():
await history.initialize()
dispatcher = create_dispatcher(
prefix=cfg.commands.prefix,
disabled_commands=cfg.commands.disabled_commands,
custom_commands=cfg.commands.custom_commands,
)
class FakeConnector:
my_node_id = "!THIS_BOT"
class FakeMessage:
text = "how's the cache peak fire?"
sender_id = "!T"
sender_name = "t"
is_dm = True
channel = 0
router = MessageRouter(
config=cfg, connector=FakeConnector(),
history=history, dispatcher=dispatcher,
llm_backend=None, # we only inspect the route() decision
)
result = await router.route(FakeMessage())
return result
result = asyncio.run(_run())
assert result.route_type == RouteType.LLM
# Critical: the query must be the verbatim user text, not a rewrite
# synthesized by an in-router intent helper.
assert result.query == "how's the cache peak fire?"
def test_status_lookup_trims_trailing_fire_word():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
f = _lookup_fire_fuzzy("cache peak fire")
assert f is not None
assert f["incident_name"] == "Cache Peak"
def test_status_lookup_word_overlap_fallback():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
f = _lookup_fire_fuzzy("how is peak doing")
assert f is not None
assert f["incident_name"] == "Cache Peak"
def test_status_lookup_returns_none_on_no_match():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
assert _lookup_fire_fuzzy("nonexistent ranger station") is None
def test_status_query_rewrite_includes_fire_context():
from meshai.router import _maybe_rewrite_status_query
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847, contained=23)
out = _maybe_rewrite_status_query("?status Cache Peak", router=None)
assert out is not None
assert "Cache Peak" in out
assert "1847" in out
# Must instruct the LLM to be terse mesh format.
assert "mesh" in out.lower()
def test_status_query_rewrite_returns_none_when_not_status():
from meshai.router import _maybe_rewrite_status_query
out = _maybe_rewrite_status_query("how's the weather?", router=None)
assert out is None
def test_status_helpers_removed_from_router():
"""Hard guard against ?status helpers sneaking back in. If anyone
adds a structured-command path to router.py for fires, this test
fails and the author has to talk to Matt first."""
from pathlib import Path
src = Path("/opt/meshai/meshai/router.py").read_text()
assert "_maybe_rewrite_status_query" not in src
assert "_lookup_fire_fuzzy" not in src
assert "?status" not in src