fix(v0.7-fire-tracker-4-revised): rip ?status; LLM DM 7-path verification 3 of 7 pass (NOT verified)

Matt review caught a scope error: ?status was a hypothetical sketch
in the design doc ("a node could ping ?status cache peak") treated as
authorization without asking. Ripping the structured-command path
entirely. The LLM DM path with env_reporter injection is the natural-
language interface; ?status was redundant infrastructure parallel to
the path the design depends on.

What landed:
- router.py: _maybe_rewrite_status_query + _lookup_fire_fuzzy +
  _build_fire_status_context removed. route() restored to:
  bang -> IGNORE-empty -> LLM with verbatim query.
- tests/test_fire_tracker_phase4.py: 5 ?status tests removed; replaced
  with two regression guards:
    test_natural_language_fire_question_routes_to_llm -- "how's the
      cache peak fire?" returns RouteType.LLM with the verbatim query
      (no in-router rewriting).
    test_status_helpers_removed_from_router -- hard-block on
      _maybe_rewrite_status_query / _lookup_fire_fuzzy / "?status"
      appearing anywhere in router.py source. If anyone adds a
      structured-command path for fires, this test fails and the
      author has to talk to Matt first.
- 56 passed in 3.80s across phase1+phase2+phase3+phase4+or-arch+
  include-roundtrip.

What stays (NOT ripped):
- Daily fire digest -- scheduled broadcaster, not a command. Its 4
  adapter_config rows (fires.digest_enabled / digest_schedule /
  digest_timezone / digest_max_chars) stay GUI-editable.
- Bug A fix (UnboundLocalError at router.py:745) -- independent of
  ?status. Confirmed still in effect.

LLM DM 7-path verification result -- 3 of 7 pass, INCOMPLETE:

| # | query                                         | env_reporter         | verdict |
|---|-----------------------------------------------|----------------------|---------|
| 1 | "are there any fires near me?"                | build_fires_detail   | PASS    |
| 2 | "any weather alerts?"                         | build_alerts_detail  | FAIL    |
| 3 | "any earthquakes nearby?"                     | build_quakes_detail  | FAIL    |
| 4 | "how's traffic on I-84?"                      | build_traffic_detail | FAIL    |
| 5 | "what's the snake river level?"               | build_gauges_detail  | PASS    |
| 6 | "what are the band conditions?"               | build_swpc_detail    | PASS    |
| 7 | "why didn't I hear about anything today?"     | build_drop_audit     | FAIL    |

Two distinct failure classes:

Class A -- routing miss (#4 traffic, #7 drop):
  _ENV_KEYWORDS_TO_SUBTYPE lacks "traffic" (only road/jam/crash/
  closure/511/incident map to "traffic"), so a query literally
  mentioning "traffic" never triggers env scope -> build_traffic_detail
  never runs even though traffic_events has 9 rows on disk. The LLM
  fell back to training data and hallucinated I-84 conditions.
  build_drop_audit has no natural-language trigger phrase at all;
  "why didn't I hear about anything today?" has no env keyword.

Class B -- empty data + LLM hallucination (#2 alerts, #3 quakes):
  Env scope IS detected, build_alerts_detail and build_quakes_detail
  DO run, but return empty because nws_alerts has 0 rows and
  quake_events 24h-window has 0 rows (legitimate empty state). The
  LLM has no env block to ground on and hallucinated "144 earthquakes
  worldwide" -- sounds authoritative, is fabricated.

Not fixed in this commit -- needs Matt's call on:
  (a) keyword additions to _ENV_KEYWORDS_TO_SUBTYPE for traffic +
      drop_audit triggers (risk: false-positive env-scope triggers
      for unrelated phrases).
  (b) anti-hallucination prompt clamp: "If a topic's env block is
      missing/empty, say you don't have live data instead of
      answering from general knowledge." (risk: bot apologizes
      every other message.)

Per the "STOP if any path fails" instruction, this commit does NOT
claim verification done; the report at
v0.7-firetracker-phase4.md has the full table + per-row mesh-receiver
wire + per-failure root cause analysis.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson (via Claude) 2026-06-06 07:33:11 +00:00
commit 89640f624d
2 changed files with 52 additions and 197 deletions

View file

@ -373,17 +373,9 @@ class MessageRouter:
if not query: if not query:
return RouteResult(RouteType.IGNORE) return RouteResult(RouteType.IGNORE)
# v0.7-fire-tracker-4: ?status <fire_name> intent. # v0.7-fire-tracker-4-revised: the LLM DM path with env_reporter
# Matches the leading "?status" sigil or a bare "status <name>"; # injection is the natural-language interface. No bolt-on
# falls through to the normal LLM path on no match. We do the # structured-command parallel.
# fire lookup here but return RouteType.LLM with a synthesized
# query so generate_llm_response runs the normal injection +
# chunking path with the fire's context attached.
status_query = _maybe_rewrite_status_query(query, self)
if status_query is not None:
return RouteResult(RouteType.LLM, query=status_query)
# Route to LLM
return RouteResult(RouteType.LLM, query=query) return RouteResult(RouteType.LLM, query=query)
def _is_mesh_question(self, message: str) -> bool: def _is_mesh_question(self, message: str) -> bool:
@ -957,146 +949,3 @@ class MessageRouter:
connector=self.connector, connector=self.connector,
history=self.history, history=self.history,
) )
# ============================================================================
# v0.7-fire-tracker-4: ?status <fire> intent helper
# ============================================================================
_STATUS_PREFIXES = ("?status ", "status ", "?status:", "status:")
def _maybe_rewrite_status_query(query: str, router) -> "Optional[str]":
"""If `query` looks like a fire status request, rewrite it with the
fire's persisted context inlined. Return None to let the normal LLM
path handle the message verbatim.
Triggers on the leading word patterns in _STATUS_PREFIXES OR an
interrogative referencing a known fire (e.g. "how is the X fire?").
"""
q = query.strip()
ql = q.lower()
target_phrase = None
for prefix in _STATUS_PREFIXES:
if ql.startswith(prefix):
target_phrase = q[len(prefix):].strip()
break
if target_phrase is None:
# Heuristic for "how is <name> fire?" style without a sigil.
triggers = ("how is ", "tell me about ", "status of ",
"what about ", "any update on ")
for t in triggers:
if ql.startswith(t):
target_phrase = q[len(t):].rstrip("?!. ").strip()
if "fire" in target_phrase.lower():
break
target_phrase = None
if target_phrase is None:
return None
if not target_phrase:
return None
fire = _lookup_fire_fuzzy(target_phrase)
if fire is None:
# No match -- leave the query alone; the LLM with env_reporter
# injection may still answer reasonably.
return None
context = _build_fire_status_context(fire)
return (
f"User asked for the status of {fire['incident_name']}. "
f"Reply with ONE short paragraph (<= 300 chars total) for mesh "
f"radio operators. No markdown.\n\n"
f"FIRE DATA:\n{context}\n\n"
f"Original question: {query}"
)
def _lookup_fire_fuzzy(phrase: str):
"""Find a fire whose incident_name fuzzy-matches phrase. Returns the
sqlite3.Row or None.
Match priority: exact (case-insensitive) -> startswith ->
contains -> word-overlap. Active fires (tombstoned_at IS NULL)
rank above closed ones."""
from meshai.persistence import get_db
conn = get_db()
phrase_l = phrase.lower().strip().rstrip("?!.").rstrip()
# Drop trailing " fire" so "cache peak fire" matches "Cache Peak".
if phrase_l.endswith(" fire"):
phrase_l = phrase_l[:-5].strip()
candidates = conn.execute(
"SELECT irwin_id, incident_name, current_acres, "
"current_contained_pct, state, county, "
"tombstoned_at, last_pass_at "
"FROM fires "
"ORDER BY (tombstoned_at IS NULL) DESC, "
"COALESCE(current_acres, 0) DESC",
).fetchall()
if not candidates:
return None
# Tier 1: exact match.
for c in candidates:
if (c["incident_name"] or "").lower() == phrase_l:
return c
# Tier 2: startswith.
for c in candidates:
if (c["incident_name"] or "").lower().startswith(phrase_l):
return c
# Tier 3: contains.
for c in candidates:
if phrase_l in (c["incident_name"] or "").lower():
return c
# Tier 4: word-overlap (>= 1 token).
tokens = set(phrase_l.split())
if tokens:
best = None
best_overlap = 0
for c in candidates:
name_tokens = set((c["incident_name"] or "").lower().split())
overlap = len(tokens & name_tokens)
if overlap > best_overlap:
best_overlap = overlap
best = c
if best is not None and best_overlap > 0:
return best
return None
def _build_fire_status_context(fire) -> str:
"""Compose the context block for the status query LLM prompt."""
from meshai.persistence import get_db
conn = get_db()
passes = conn.execute(
"SELECT pass_id, drift_mi_from_prev, drift_direction, "
"drift_mi_per_hour, pixel_count, pass_ended_at "
"FROM fire_passes WHERE irwin_id=? "
"ORDER BY pass_ended_at DESC LIMIT 3",
(fire["irwin_id"],),
).fetchall()
lines = [
f"name: {fire['incident_name']}",
f"acres: {fire['current_acres'] or 0}",
f"contained: {fire['current_contained_pct'] or 0}%",
f"county/state: {fire['county'] or '?'}/{fire['state'] or '?'}",
f"closed: {bool(fire['tombstoned_at'])}",
]
if passes:
lines.append("recent passes (newest first):")
for p in passes:
drift = ""
if (p["drift_mi_from_prev"] is not None
and p["drift_direction"] is not None):
drift = (f", drift {p['drift_mi_from_prev']:.1f}mi "
f"{p['drift_direction']}")
lines.append(
f" - pass {p['pass_id']}: {p['pixel_count']} pixel(s)"
f"{drift}")
return "\n".join(lines)

View file

@ -131,57 +131,63 @@ def test_render_digest_uses_llm_when_available():
# =========================================================================== # ===========================================================================
# Fuzzy fire lookup for ?status # Natural-language fire DMs route to the LLM (no ?status fallback)
# =========================================================================== # ===========================================================================
def test_status_lookup_exact_name(): def test_natural_language_fire_question_routes_to_llm():
from meshai.router import _lookup_fire_fuzzy """The LLM DM path is the sole interface for natural-language fire
_seed_fire(irwin_id="ID-A", name="Cache Peak", questions. Pre-revised commit there was a `?status` intent that
lat=42.0, lon=-114.0, acres=1847) rewrote the query in-router; this test confirms the rewrite is gone
f = _lookup_fire_fuzzy("Cache Peak") and that a plain English question is forwarded verbatim."""
assert f is not None import asyncio
assert f["incident_name"] == "Cache Peak" from meshai.router import MessageRouter, RouteType
from meshai.config_loader import load_config
from meshai.history import ConversationHistory
from meshai.commands.dispatcher import create_dispatcher
cfg = load_config()
history = ConversationHistory(cfg.history)
async def _run():
await history.initialize()
dispatcher = create_dispatcher(
prefix=cfg.commands.prefix,
disabled_commands=cfg.commands.disabled_commands,
custom_commands=cfg.commands.custom_commands,
)
class FakeConnector:
my_node_id = "!THIS_BOT"
class FakeMessage:
text = "how's the cache peak fire?"
sender_id = "!T"
sender_name = "t"
is_dm = True
channel = 0
router = MessageRouter(
config=cfg, connector=FakeConnector(),
history=history, dispatcher=dispatcher,
llm_backend=None, # we only inspect the route() decision
)
result = await router.route(FakeMessage())
return result
result = asyncio.run(_run())
assert result.route_type == RouteType.LLM
# Critical: the query must be the verbatim user text, not a rewrite
# synthesized by an in-router intent helper.
assert result.query == "how's the cache peak fire?"
def test_status_lookup_trims_trailing_fire_word(): def test_status_helpers_removed_from_router():
from meshai.router import _lookup_fire_fuzzy """Hard guard against ?status helpers sneaking back in. If anyone
_seed_fire(irwin_id="ID-A", name="Cache Peak", adds a structured-command path to router.py for fires, this test
lat=42.0, lon=-114.0, acres=1847) fails and the author has to talk to Matt first."""
f = _lookup_fire_fuzzy("cache peak fire") from pathlib import Path
assert f is not None src = Path("/opt/meshai/meshai/router.py").read_text()
assert f["incident_name"] == "Cache Peak" assert "_maybe_rewrite_status_query" not in src
assert "_lookup_fire_fuzzy" not in src
assert "?status" not in src
def test_status_lookup_word_overlap_fallback():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
f = _lookup_fire_fuzzy("how is peak doing")
assert f is not None
assert f["incident_name"] == "Cache Peak"
def test_status_lookup_returns_none_on_no_match():
from meshai.router import _lookup_fire_fuzzy
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847)
assert _lookup_fire_fuzzy("nonexistent ranger station") is None
def test_status_query_rewrite_includes_fire_context():
from meshai.router import _maybe_rewrite_status_query
_seed_fire(irwin_id="ID-A", name="Cache Peak",
lat=42.0, lon=-114.0, acres=1847, contained=23)
out = _maybe_rewrite_status_query("?status Cache Peak", router=None)
assert out is not None
assert "Cache Peak" in out
assert "1847" in out
# Must instruct the LLM to be terse mesh format.
assert "mesh" in out.lower()
def test_status_query_rewrite_returns_none_when_not_status():
from meshai.router import _maybe_rewrite_status_query
out = _maybe_rewrite_status_query("how's the weather?", router=None)
assert out is None