meshai/work/tests/test_channel_rendering.py
malice a7b7f5a6a4
feat(transport): per-radio serialized+paced outbound send queue (#93)
* feat(transport): per-radio serialized+paced outbound send queue

Prevents simultaneous LoRa transmissions when N events arrive at once.

## Mechanism

Two `RadioSendQueue` instances (one MT, one MC), each a FIFO asyncio.Queue
with a long-running drain task.  The MT queue drains on the main asyncio
loop; the MC queue drains on MeshCore's dedicated event-loop thread.

- MT sends: `run_in_executor` offloads the blocking `sendText` call;
  queue started in `set_message_callback`, cancelled in `disconnect`.
- MC sends: drain loop runs pure-async MC lib coroutines directly on the
  MC loop (no `_run_coro` deadlock); cross-loop callers bridge via
  `concurrent.futures.Future` + `asyncio.wrap_future`.
- Pacing: `await asyncio.sleep(pacing_seconds)` between items; read live
  from config per iteration; floor clamped to 0.25 s.
- Config knobs: `meshtastic_send_pacing_seconds` (default 2.0) and
  `meshcore_send_pacing_seconds` (default 2.0) on `ConnectionConfig`.

## Send sites rerouted

All callers now `await connector.send_message_async(...)`:
- `notifications/channels.py` — MeshBroadcast/MeshCoreBroadcast/MeshDM/
  MeshCoreDM deliver(), test_connection(), deliver_test()
- `responder.py` — DM replies in send_response()
- `transport/meshcore_transport.py` — periodic_advert_loop, telemetry
  poll loop, send_advert() → send_advert_async(), req_telemetry()
  → req_telemetry_async() (all queue-routed from main loop)
- `dashboard/api/mesh_send_routes.py` — test-send, advert, telemetry poll

## Audit accuracy

`deliver()` now returns the actual bool from the radio send (not
optimistic True), so `mesh_broadcasts_out` reflects the real result.

## Tests

17 new tests in tests/test_send_queue.py covering FIFO ordering, no drops,
pacing gap, pacing floor enforcement, event-loop non-blocking, serialization,
lifecycle, MT fallback, config round-trip.  Existing test stubs updated to
wire `send_message_async = AsyncMock(side_effect=send_message)` so prior
call_count / call_args assertions remain valid without changes.

Full suite: 2135 passed, 17 pre-existing failures (unchanged), 0 new regressions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(send-queue): resolve MC telemetry self-deadlock + resolve pending futures on teardown/reconnect; composite MC-channel kwarg; audit no-op false

BLOCKER 1 — req_telemetry_async self-deadlock (meshcore_transport.py):
_req_telemetry_async was calling _enqueue_mc_loop_send inside itself;
when _telem_job_outer ran inside the drain it nested another enqueue+await
on the same single-threaded drain — permanent deadlock on first telemetry poll.
Fix: _req_telemetry_async is now fully inline (no _enqueue_mc_loop_send).
_telemetry_poll_loop wraps its call in _enqueue_mc_loop_send for serialization.
req_telemetry_async's outer job calls _req_telemetry_async inline (safe).

BLOCKER 2 — pending futures abandoned on teardown/reconnect:
RadioSendQueue.stop() only cancelled the drain task; queue-sitting items had
their concurrent.futures.Futures left unresolved, causing wrap_future() callers
to hang indefinitely. Fix: stop() drains the remaining queue with get_nowait()
and cancels every pending cfut. _cancel_mc_queue() schedules the same drain-
and-cancel via call_soon_threadsafe. _start_mc_queue() cancels old drain task
and drains old queue cfuts before arming the new queue (reconnect path).
connector.disconnect() now .result(timeout=5) on stop() instead of fire-and-forget.

SHOULD-FIX 3 — composite passes MC channel as wrong kwarg (composite_transport.py):
_broadcast_async no-hint loop was calling send_message_async(channel=child_channel)
for the meshcore child; should be meshcore_channel=child_channel. Silent drop fixed.

NIT 5 — false success on zero-channel MC send (meshcore_transport.py):
send_message_async returned True when meshcore_channel is None (nothing sent).
Now returns False so audit does not record a success for a no-op.

NIT 7 — config comment contradiction (config.py):
meshtastic_send_pacing_seconds comment said "0 disables the floor" while
simultaneously stating "still floored at 0.25". Removed the contradiction.

Regression tests (tests/test_send_queue.py — 3 new, all in TestDeadlockRegression):
- test_telemetry_queue_no_deadlock: drives req_telemetry_async through a real
  _mc_send_queue with fake MC commands; times out on pre-fix code (deadlock).
- test_teardown_resolves_pending_futures: enqueues slow+fast jobs, stops mid-drain,
  asserts every task resolves promptly; hangs on pre-fix code.
- test_reconnect_resolves_old_futures: calls _start_mc_queue twice, asserts old
  cfuts are cancelled; pre-fix leaves them unresolved.

All 17 pre-existing send-queue tests still pass (20 total now).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-08 09:38:08 -06:00

343 lines
11 KiB
Python

"""Tests for channel-renderer integration (Phase 2.5b)."""
import asyncio
import time
from unittest.mock import MagicMock, AsyncMock, patch
import pytest
from meshai.notifications.events import NotificationPayload
from meshai.notifications.channels import (
MeshBroadcastChannel,
MeshDMChannel,
EmailChannel,
WebhookChannel,
)
def _mock_conn():
"""Create a MagicMock connector with send_message_async wired to send_message.
Channels now call send_message_async (async) instead of send_message (sync).
side_effect delegates to send_message so existing call_count / call_args
assertions remain valid.
"""
c = MagicMock()
c.send_message_async = AsyncMock(side_effect=lambda *a, **kw: c.send_message(*a, **kw))
return c
# ============================================================
# MESH CHANNEL RENDERING TESTS
# ============================================================
def test_mesh_channel_uses_mesh_renderer():
"""MeshBroadcastChannel renders long messages to multiple chunks."""
mock_connector = _mock_conn()
channel = MeshBroadcastChannel(
connector=mock_connector,
channel_index=0,
)
# Build a long message that will require chunking
long_message = "This is a very long alert message that exceeds the character limit. " * 5
payload = NotificationPayload(
message=long_message,
category="weather_warning",
severity="priority",
timestamp=time.time(),
event_type="weather_warning",
)
asyncio.run(channel.deliver(payload, None))
# Should have called send_message multiple times (once per chunk)
assert mock_connector.send_message.call_count >= 2
# Each call's text should be <= 200 chars
for call in mock_connector.send_message.call_args_list:
text = call.kwargs.get("text", call.args[0] if call.args else "")
assert len(text) <= 200
def test_mesh_channel_uses_payload_message_directly_when_chunk_metadata_set():
"""Pre-chunked payloads (from digest) skip re-rendering."""
mock_connector = _mock_conn()
channel = MeshBroadcastChannel(
connector=mock_connector,
channel_index=0,
)
# Payload with chunk metadata set (from digest scheduler)
payload = NotificationPayload(
message="pre-chunked text",
category="digest",
severity="routine",
timestamp=time.time(),
chunk_index=1,
chunk_total=3,
)
asyncio.run(channel.deliver(payload, None))
# Should have called send_message exactly once
assert mock_connector.send_message.call_count == 1
# Should use the message directly
call = mock_connector.send_message.call_args
text = call.kwargs.get("text", call.args[0] if call.args else "")
assert text == "pre-chunked text"
def test_mesh_dm_channel_uses_mesh_renderer():
"""MeshDMChannel renders long messages to chunks for each recipient."""
mock_connector = _mock_conn()
channel = MeshDMChannel(
connector=mock_connector,
node_ids=["!node1", "!node2"],
)
long_message = "This is a long DM message that should be chunked. " * 4
payload = NotificationPayload(
message=long_message,
category="test",
severity="routine",
timestamp=time.time(),
)
asyncio.run(channel.deliver(payload, None))
# Should have called send_message multiple times
# (chunks * nodes)
assert mock_connector.send_message.call_count >= 2
def test_mesh_dm_channel_uses_payload_message_directly_when_chunk_metadata_set():
"""Pre-chunked DM payloads skip re-rendering."""
mock_connector = _mock_conn()
channel = MeshDMChannel(
connector=mock_connector,
node_ids=["!node1"],
)
payload = NotificationPayload(
message="pre-chunked DM",
category="digest",
severity="routine",
timestamp=time.time(),
chunk_index=2,
chunk_total=5,
)
asyncio.run(channel.deliver(payload, None))
# Should use message directly, once per node
assert mock_connector.send_message.call_count == 1
call = mock_connector.send_message.call_args
text = call.kwargs.get("text", call.args[0] if call.args else "")
assert text == "pre-chunked DM"
# ============================================================
# EMAIL CHANNEL RENDERING TESTS
# ============================================================
def test_email_channel_uses_email_renderer():
"""EmailChannel uses renderer for subject and body."""
channel = EmailChannel(
smtp_host="localhost",
smtp_port=25,
smtp_user="",
smtp_password="",
smtp_tls=False,
from_address="test@example.com",
recipients=["user@example.com"],
)
payload = NotificationPayload(
message="Test alert message",
category="weather_warning",
severity="immediate",
timestamp=time.time(),
event_type="weather_warning",
)
# Mock the _send_email method
with patch.object(channel, "_send_email") as mock_send:
asyncio.run(channel.deliver(payload, None))
# Should have been called with renderer output
mock_send.assert_called_once()
call_args = mock_send.call_args
subject = call_args.args[0]
body = call_args.args[1]
# Renderer format checks
assert "[MeshAI]" in subject
assert "IMMEDIATE" in subject
assert "Test alert message" in body
assert "Severity:" in body
# ============================================================
# WEBHOOK CHANNEL RENDERING TESTS
# ============================================================
def test_webhook_channel_uses_webhook_renderer():
"""WebhookChannel uses renderer for JSON payload."""
channel = WebhookChannel(
url="https://example.com/webhook",
headers={},
)
payload = NotificationPayload(
message="Test webhook message",
category="test",
severity="priority",
timestamp=time.time(),
event_type="battery_warning",
)
# Mock httpx
with patch("meshai.notifications.channels.httpx.AsyncClient") as mock_client_class:
mock_client = MagicMock()
mock_response = MagicMock()
mock_response.status_code = 200
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
asyncio.run(channel.deliver(payload, None))
# Check the POST was called
mock_client.post.assert_called_once()
call_kwargs = mock_client.post.call_args.kwargs
# Should have JSON payload with schema_version
json_payload = call_kwargs.get("json", {})
assert "schema_version" in json_payload
assert json_payload["schema_version"] == "1.0"
assert json_payload["message"] == "Test webhook message"
# ============================================================
# PER-FAMILY MESHCORE ROUTING — end-to-end threading guard
# Updated for the explicit-per-mesh model (meshcore_broadcast/mesh_broadcast)
# ============================================================
def test_mesh_broadcast_routes_to_meshtastic_only():
"""mesh_broadcast passes transport='meshtastic' and channel index to
send_message. meshcore_channel is NOT passed (auto-fan removed).
Regression guard: before this model, mesh_broadcast also threaded
meshcore_channel through; now it is Meshtastic-only.
"""
from meshai.config import NotificationRuleConfig
from meshai.notifications.channels import create_channel
mock_connector = _mock_conn()
rule = NotificationRuleConfig(
name="toggle:fire",
delivery_type="mesh_broadcast",
broadcast_channel=1,
meshcore_channel="AIDA", # present in config but must NOT flow to send_message
)
channel = create_channel(rule, mock_connector)
# Pre-chunked payload => exactly one deterministic send_message call.
payload = NotificationPayload(
message="fire alert",
category="fire",
severity="immediate",
timestamp=time.time(),
event_type="fire",
chunk_index=0,
)
assert asyncio.run(channel.deliver(payload, rule)) is True
mock_connector.send_message.assert_called_once()
kwargs = mock_connector.send_message.call_args.kwargs
assert kwargs.get("channel") == 1
assert kwargs.get("transport") == "meshtastic"
# meshcore_channel must NOT be present (no auto-fan).
assert "meshcore_channel" not in kwargs or kwargs.get("meshcore_channel") is None
def test_meshcore_broadcast_routes_to_meshcore_only():
"""meshcore_broadcast passes meshcore_channel=name and transport='meshcore'
to send_message. This is the explicit MeshCore-only delivery path."""
from meshai.config import NotificationRuleConfig
from meshai.notifications.channels import MeshCoreBroadcastChannel, create_channel
# Simulate a CompositeTransport connector with a meshcore child.
mock_connector = _mock_conn()
mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()}
mock_connector.send_message.return_value = True
rule = NotificationRuleConfig(
name="toggle:fire",
delivery_type="meshcore_broadcast",
meshcore_channel="AIDA",
)
channel = create_channel(rule, mock_connector)
assert isinstance(channel, MeshCoreBroadcastChannel)
payload = NotificationPayload(
message="fire alert",
category="fire",
severity="immediate",
timestamp=time.time(),
event_type="fire",
chunk_index=0,
)
assert asyncio.run(channel.deliver(payload, rule)) is True
mock_connector.send_message.assert_called_once()
kwargs = mock_connector.send_message.call_args.kwargs
assert kwargs.get("meshcore_channel") == "AIDA"
assert kwargs.get("transport") == "meshcore"
assert kwargs.get("destination") is None
def test_broadcast_render_loop_threads_meshcore_channel():
"""Non-prechunked path (renderer loop) for meshcore_broadcast threads
meshcore_channel on every chunk send."""
from meshai.config import NotificationRuleConfig
from meshai.notifications.channels import create_channel
mock_connector = _mock_conn()
# Connector has a meshcore child so the no-op guard passes.
mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()}
mock_connector.send_message.return_value = True
rule = NotificationRuleConfig(
name="toggle:fire",
delivery_type="meshcore_broadcast",
meshcore_channel="AIDA",
)
channel = create_channel(rule, mock_connector)
long_message = "This is a very long alert message that exceeds the limit. " * 5
payload = NotificationPayload(
message=long_message,
category="fire",
severity="immediate",
timestamp=time.time(),
event_type="fire",
)
assert asyncio.run(channel.deliver(payload, rule)) is True
assert mock_connector.send_message.call_count >= 2
for call in mock_connector.send_message.call_args_list:
assert call.kwargs.get("meshcore_channel") == "AIDA"
assert call.kwargs.get("transport") == "meshcore"