mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-10 17:04:45 +02:00
fix(v0.7-fire-tracker-4-revised): rip ?status; LLM DM 7-path verification 3 of 7 pass (NOT verified)
Matt review caught a scope error: ?status was a hypothetical sketch
in the design doc ("a node could ping ?status cache peak") treated as
authorization without asking. Ripping the structured-command path
entirely. The LLM DM path with env_reporter injection is the natural-
language interface; ?status was redundant infrastructure parallel to
the path the design depends on.
What landed:
- router.py: _maybe_rewrite_status_query + _lookup_fire_fuzzy +
_build_fire_status_context removed. route() restored to:
bang -> IGNORE-empty -> LLM with verbatim query.
- tests/test_fire_tracker_phase4.py: 5 ?status tests removed; replaced
with two regression guards:
test_natural_language_fire_question_routes_to_llm -- "how's the
cache peak fire?" returns RouteType.LLM with the verbatim query
(no in-router rewriting).
test_status_helpers_removed_from_router -- hard-block on
_maybe_rewrite_status_query / _lookup_fire_fuzzy / "?status"
appearing anywhere in router.py source. If anyone adds a
structured-command path for fires, this test fails and the
author has to talk to Matt first.
- 56 passed in 3.80s across phase1+phase2+phase3+phase4+or-arch+
include-roundtrip.
What stays (NOT ripped):
- Daily fire digest -- scheduled broadcaster, not a command. Its 4
adapter_config rows (fires.digest_enabled / digest_schedule /
digest_timezone / digest_max_chars) stay GUI-editable.
- Bug A fix (UnboundLocalError at router.py:745) -- independent of
?status. Confirmed still in effect.
LLM DM 7-path verification result -- 3 of 7 pass, INCOMPLETE:
| # | query | env_reporter | verdict |
|---|-----------------------------------------------|----------------------|---------|
| 1 | "are there any fires near me?" | build_fires_detail | PASS |
| 2 | "any weather alerts?" | build_alerts_detail | FAIL |
| 3 | "any earthquakes nearby?" | build_quakes_detail | FAIL |
| 4 | "how's traffic on I-84?" | build_traffic_detail | FAIL |
| 5 | "what's the snake river level?" | build_gauges_detail | PASS |
| 6 | "what are the band conditions?" | build_swpc_detail | PASS |
| 7 | "why didn't I hear about anything today?" | build_drop_audit | FAIL |
Two distinct failure classes:
Class A -- routing miss (#4 traffic, #7 drop):
_ENV_KEYWORDS_TO_SUBTYPE lacks "traffic" (only road/jam/crash/
closure/511/incident map to "traffic"), so a query literally
mentioning "traffic" never triggers env scope -> build_traffic_detail
never runs even though traffic_events has 9 rows on disk. The LLM
fell back to training data and hallucinated I-84 conditions.
build_drop_audit has no natural-language trigger phrase at all;
"why didn't I hear about anything today?" has no env keyword.
Class B -- empty data + LLM hallucination (#2 alerts, #3 quakes):
Env scope IS detected, build_alerts_detail and build_quakes_detail
DO run, but return empty because nws_alerts has 0 rows and
quake_events 24h-window has 0 rows (legitimate empty state). The
LLM has no env block to ground on and hallucinated "144 earthquakes
worldwide" -- sounds authoritative, is fabricated.
Not fixed in this commit -- needs Matt's call on:
(a) keyword additions to _ENV_KEYWORDS_TO_SUBTYPE for traffic +
drop_audit triggers (risk: false-positive env-scope triggers
for unrelated phrases).
(b) anti-hallucination prompt clamp: "If a topic's env block is
missing/empty, say you don't have live data instead of
answering from general knowledge." (risk: bot apologizes
every other message.)
Per the "STOP if any path fails" instruction, this commit does NOT
claim verification done; the report at
v0.7-firetracker-phase4.md has the full table + per-row mesh-receiver
wire + per-failure root cause analysis.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f69a05dd6d
commit
89640f624d
2 changed files with 52 additions and 197 deletions
|
|
@ -131,57 +131,63 @@ def test_render_digest_uses_llm_when_available():
|
|||
|
||||
|
||||
# ===========================================================================
|
||||
# Fuzzy fire lookup for ?status
|
||||
# Natural-language fire DMs route to the LLM (no ?status fallback)
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
def test_status_lookup_exact_name():
|
||||
from meshai.router import _lookup_fire_fuzzy
|
||||
_seed_fire(irwin_id="ID-A", name="Cache Peak",
|
||||
lat=42.0, lon=-114.0, acres=1847)
|
||||
f = _lookup_fire_fuzzy("Cache Peak")
|
||||
assert f is not None
|
||||
assert f["incident_name"] == "Cache Peak"
|
||||
def test_natural_language_fire_question_routes_to_llm():
|
||||
"""The LLM DM path is the sole interface for natural-language fire
|
||||
questions. Pre-revised commit there was a `?status` intent that
|
||||
rewrote the query in-router; this test confirms the rewrite is gone
|
||||
and that a plain English question is forwarded verbatim."""
|
||||
import asyncio
|
||||
from meshai.router import MessageRouter, RouteType
|
||||
from meshai.config_loader import load_config
|
||||
from meshai.history import ConversationHistory
|
||||
from meshai.commands.dispatcher import create_dispatcher
|
||||
|
||||
cfg = load_config()
|
||||
history = ConversationHistory(cfg.history)
|
||||
|
||||
async def _run():
|
||||
await history.initialize()
|
||||
dispatcher = create_dispatcher(
|
||||
prefix=cfg.commands.prefix,
|
||||
disabled_commands=cfg.commands.disabled_commands,
|
||||
custom_commands=cfg.commands.custom_commands,
|
||||
)
|
||||
|
||||
class FakeConnector:
|
||||
my_node_id = "!THIS_BOT"
|
||||
|
||||
class FakeMessage:
|
||||
text = "how's the cache peak fire?"
|
||||
sender_id = "!T"
|
||||
sender_name = "t"
|
||||
is_dm = True
|
||||
channel = 0
|
||||
|
||||
router = MessageRouter(
|
||||
config=cfg, connector=FakeConnector(),
|
||||
history=history, dispatcher=dispatcher,
|
||||
llm_backend=None, # we only inspect the route() decision
|
||||
)
|
||||
result = await router.route(FakeMessage())
|
||||
return result
|
||||
|
||||
result = asyncio.run(_run())
|
||||
assert result.route_type == RouteType.LLM
|
||||
# Critical: the query must be the verbatim user text, not a rewrite
|
||||
# synthesized by an in-router intent helper.
|
||||
assert result.query == "how's the cache peak fire?"
|
||||
|
||||
|
||||
def test_status_lookup_trims_trailing_fire_word():
|
||||
from meshai.router import _lookup_fire_fuzzy
|
||||
_seed_fire(irwin_id="ID-A", name="Cache Peak",
|
||||
lat=42.0, lon=-114.0, acres=1847)
|
||||
f = _lookup_fire_fuzzy("cache peak fire")
|
||||
assert f is not None
|
||||
assert f["incident_name"] == "Cache Peak"
|
||||
|
||||
|
||||
def test_status_lookup_word_overlap_fallback():
|
||||
from meshai.router import _lookup_fire_fuzzy
|
||||
_seed_fire(irwin_id="ID-A", name="Cache Peak",
|
||||
lat=42.0, lon=-114.0, acres=1847)
|
||||
f = _lookup_fire_fuzzy("how is peak doing")
|
||||
assert f is not None
|
||||
assert f["incident_name"] == "Cache Peak"
|
||||
|
||||
|
||||
def test_status_lookup_returns_none_on_no_match():
|
||||
from meshai.router import _lookup_fire_fuzzy
|
||||
_seed_fire(irwin_id="ID-A", name="Cache Peak",
|
||||
lat=42.0, lon=-114.0, acres=1847)
|
||||
assert _lookup_fire_fuzzy("nonexistent ranger station") is None
|
||||
|
||||
|
||||
def test_status_query_rewrite_includes_fire_context():
|
||||
from meshai.router import _maybe_rewrite_status_query
|
||||
_seed_fire(irwin_id="ID-A", name="Cache Peak",
|
||||
lat=42.0, lon=-114.0, acres=1847, contained=23)
|
||||
out = _maybe_rewrite_status_query("?status Cache Peak", router=None)
|
||||
assert out is not None
|
||||
assert "Cache Peak" in out
|
||||
assert "1847" in out
|
||||
# Must instruct the LLM to be terse mesh format.
|
||||
assert "mesh" in out.lower()
|
||||
|
||||
|
||||
def test_status_query_rewrite_returns_none_when_not_status():
|
||||
from meshai.router import _maybe_rewrite_status_query
|
||||
out = _maybe_rewrite_status_query("how's the weather?", router=None)
|
||||
assert out is None
|
||||
def test_status_helpers_removed_from_router():
|
||||
"""Hard guard against ?status helpers sneaking back in. If anyone
|
||||
adds a structured-command path to router.py for fires, this test
|
||||
fails and the author has to talk to Matt first."""
|
||||
from pathlib import Path
|
||||
src = Path("/opt/meshai/meshai/router.py").read_text()
|
||||
assert "_maybe_rewrite_status_query" not in src
|
||||
assert "_lookup_fire_fuzzy" not in src
|
||||
assert "?status" not in src
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue