feat(transport): per-radio serialized+paced outbound send queue (#93)

* feat(transport): per-radio serialized+paced outbound send queue

Prevents simultaneous LoRa transmissions when N events arrive at once.

## Mechanism

Two `RadioSendQueue` instances (one MT, one MC), each a FIFO asyncio.Queue
with a long-running drain task.  The MT queue drains on the main asyncio
loop; the MC queue drains on MeshCore's dedicated event-loop thread.

- MT sends: `run_in_executor` offloads the blocking `sendText` call;
  queue started in `set_message_callback`, cancelled in `disconnect`.
- MC sends: drain loop runs pure-async MC lib coroutines directly on the
  MC loop (no `_run_coro` deadlock); cross-loop callers bridge via
  `concurrent.futures.Future` + `asyncio.wrap_future`.
- Pacing: `await asyncio.sleep(pacing_seconds)` between items; read live
  from config per iteration; floor clamped to 0.25 s.
- Config knobs: `meshtastic_send_pacing_seconds` (default 2.0) and
  `meshcore_send_pacing_seconds` (default 2.0) on `ConnectionConfig`.

## Send sites rerouted

All callers now `await connector.send_message_async(...)`:
- `notifications/channels.py` — MeshBroadcast/MeshCoreBroadcast/MeshDM/
  MeshCoreDM deliver(), test_connection(), deliver_test()
- `responder.py` — DM replies in send_response()
- `transport/meshcore_transport.py` — periodic_advert_loop, telemetry
  poll loop, send_advert() → send_advert_async(), req_telemetry()
  → req_telemetry_async() (all queue-routed from main loop)
- `dashboard/api/mesh_send_routes.py` — test-send, advert, telemetry poll

## Audit accuracy

`deliver()` now returns the actual bool from the radio send (not
optimistic True), so `mesh_broadcasts_out` reflects the real result.

## Tests

17 new tests in tests/test_send_queue.py covering FIFO ordering, no drops,
pacing gap, pacing floor enforcement, event-loop non-blocking, serialization,
lifecycle, MT fallback, config round-trip.  Existing test stubs updated to
wire `send_message_async = AsyncMock(side_effect=send_message)` so prior
call_count / call_args assertions remain valid without changes.

Full suite: 2135 passed, 17 pre-existing failures (unchanged), 0 new regressions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(send-queue): resolve MC telemetry self-deadlock + resolve pending futures on teardown/reconnect; composite MC-channel kwarg; audit no-op false

BLOCKER 1 — req_telemetry_async self-deadlock (meshcore_transport.py):
_req_telemetry_async was calling _enqueue_mc_loop_send inside itself;
when _telem_job_outer ran inside the drain it nested another enqueue+await
on the same single-threaded drain — permanent deadlock on first telemetry poll.
Fix: _req_telemetry_async is now fully inline (no _enqueue_mc_loop_send).
_telemetry_poll_loop wraps its call in _enqueue_mc_loop_send for serialization.
req_telemetry_async's outer job calls _req_telemetry_async inline (safe).

BLOCKER 2 — pending futures abandoned on teardown/reconnect:
RadioSendQueue.stop() only cancelled the drain task; queue-sitting items had
their concurrent.futures.Futures left unresolved, causing wrap_future() callers
to hang indefinitely. Fix: stop() drains the remaining queue with get_nowait()
and cancels every pending cfut. _cancel_mc_queue() schedules the same drain-
and-cancel via call_soon_threadsafe. _start_mc_queue() cancels old drain task
and drains old queue cfuts before arming the new queue (reconnect path).
connector.disconnect() now .result(timeout=5) on stop() instead of fire-and-forget.

SHOULD-FIX 3 — composite passes MC channel as wrong kwarg (composite_transport.py):
_broadcast_async no-hint loop was calling send_message_async(channel=child_channel)
for the meshcore child; should be meshcore_channel=child_channel. Silent drop fixed.

NIT 5 — false success on zero-channel MC send (meshcore_transport.py):
send_message_async returned True when meshcore_channel is None (nothing sent).
Now returns False so audit does not record a success for a no-op.

NIT 7 — config comment contradiction (config.py):
meshtastic_send_pacing_seconds comment said "0 disables the floor" while
simultaneously stating "still floored at 0.25". Removed the contradiction.

Regression tests (tests/test_send_queue.py — 3 new, all in TestDeadlockRegression):
- test_telemetry_queue_no_deadlock: drives req_telemetry_async through a real
  _mc_send_queue with fake MC commands; times out on pre-fix code (deadlock).
- test_teardown_resolves_pending_futures: enqueues slow+fast jobs, stops mid-drain,
  asserts every task resolves promptly; hangs on pre-fix code.
- test_reconnect_resolves_old_futures: calls _start_mc_queue twice, asserts old
  cfuts are cancelled; pre-fix leaves them unresolved.

All 17 pre-existing send-queue tests still pass (20 total now).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Matt Johnson <mj@k7zvx.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
malice 2026-07-08 09:38:08 -06:00 committed by GitHub
commit a7b7f5a6a4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 1661 additions and 63 deletions

View file

@ -53,6 +53,12 @@ class ConnectionConfig:
meshcore_ack_wait_seconds: float = 6.0 # wait for delivery ACK before falling back to path discovery meshcore_ack_wait_seconds: float = 6.0 # wait for delivery ACK before falling back to path discovery
meshcore_discovery_wait_seconds: float = 8.0 # path-discovery timeout on the no-ACK fallback (was hardcoded 25s) meshcore_discovery_wait_seconds: float = 8.0 # path-discovery timeout on the no-ACK fallback (was hardcoded 25s)
# --- Send-queue pacing (per-radio serialization) ---
# Minimum gap between consecutive outbound sends on each radio.
# A floor of 0.25 s is always enforced at runtime (max(0.25, pacing_value)).
meshtastic_send_pacing_seconds: float = 2.0
meshcore_send_pacing_seconds: float = 2.0
# --- MeshCore telemetry auto-poll settings --- # --- MeshCore telemetry auto-poll settings ---
# Selected contacts (names or pubkeys) to auto-poll for telemetry; empty = none. # Selected contacts (names or pubkeys) to auto-poll for telemetry; empty = none.
meshcore_telemetry_contacts: list = field(default_factory=list) meshcore_telemetry_contacts: list = field(default_factory=list)

View file

@ -16,6 +16,7 @@ from pubsub import pub
from .config import ConnectionConfig from .config import ConnectionConfig
from .transport.base import MeshTransport from .transport.base import MeshTransport
from .transport.send_queue import RadioSendQueue
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -64,6 +65,8 @@ class MeshtasticTransport(MeshTransport):
self._wake: Optional[asyncio.Event] = None # created by main once loop exists self._wake: Optional[asyncio.Event] = None # created by main once loop exists
self._link_path = "/tmp/meshai.link" self._link_path = "/tmp/meshai.link"
self._reconnect_lock = threading.Lock() self._reconnect_lock = threading.Lock()
# --- per-radio send queue (serialized + paced) ---
self._mt_queue: Optional[RadioSendQueue] = None
@property @property
def connected(self) -> bool: def connected(self) -> bool:
@ -119,6 +122,15 @@ class MeshtasticTransport(MeshTransport):
def disconnect(self) -> None: def disconnect(self) -> None:
"""Close connection to Meshtastic node.""" """Close connection to Meshtastic node."""
# Stop the send queue drain and wait for it to finish so all pending
# futures are resolved before the interface is torn down.
if self._mt_queue is not None and self._mt_queue.running and self._loop is not None:
try:
asyncio.run_coroutine_threadsafe(
self._mt_queue.stop(), self._loop
).result(timeout=5.0)
except Exception:
pass
if self._interface: if self._interface:
try: try:
pub.unsubscribe(self._on_receive, "meshtastic.receive.text") pub.unsubscribe(self._on_receive, "meshtastic.receive.text")
@ -140,7 +152,7 @@ class MeshtasticTransport(MeshTransport):
def set_message_callback( def set_message_callback(
self, callback: Callable[[MeshMessage], None], loop: asyncio.AbstractEventLoop self, callback: Callable[[MeshMessage], None], loop: asyncio.AbstractEventLoop
) -> None: ) -> None:
"""Set callback for incoming messages. """Set callback for incoming messages and start the send queue drain.
Args: Args:
callback: Async function to call with MeshMessage callback: Async function to call with MeshMessage
@ -148,6 +160,14 @@ class MeshtasticTransport(MeshTransport):
""" """
self._message_callback = callback self._message_callback = callback
self._loop = loop self._loop = loop
# Start the per-radio send queue on the main event loop.
pacing_fn = lambda: max(0.25, getattr(self.config, "meshtastic_send_pacing_seconds", 2.0))
self._mt_queue = RadioSendQueue(pacing_fn=pacing_fn)
def _start_drain() -> None:
self._mt_queue.start(loop)
loop.call_soon_threadsafe(_start_drain)
def _cache_node_info(self) -> None: def _cache_node_info(self) -> None:
"""Cache node names and positions from node database.""" """Cache node names and positions from node database."""
@ -349,6 +369,68 @@ class MeshtasticTransport(MeshTransport):
time.sleep(backoff) time.sleep(backoff)
backoff = min(backoff * 2, maxd) backoff = min(backoff * 2, maxd)
async def send_message_async(
self,
text: str,
destination: Optional[str] = None,
channel: int = 0,
transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool:
"""Async send through the Meshtastic per-radio queue.
Enqueues the send job and awaits the ACTUAL radio-acceptance result so
the caller (e.g. channel.deliver) gets the real success bool. Falls
back to a direct executor call if the queue has not been started yet
(e.g. during tests or before set_message_callback is called).
"""
if self._mt_queue is None or not self._mt_queue.running:
# Queue not started: run in executor to avoid blocking the loop.
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.send_message(text, destination, channel, transport, meshcore_channel),
)
def _job() -> bool:
return self._blocking_mt_send(text, destination, channel)
async def _async_job() -> bool:
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, _job)
return await self._mt_queue.enqueue_async(_async_job)
def _blocking_mt_send(
self,
text: str,
destination: Optional[str],
channel: int,
) -> bool:
"""Synchronous Meshtastic send — called from the thread executor by the drain."""
if not self._interface:
logger.error("Cannot send: not connected")
return False
try:
if destination:
if isinstance(destination, int):
dest_num = destination
elif destination.startswith("!"):
dest_num = int(destination[1:], 16)
elif destination.isdigit():
dest_num = int(destination)
else:
dest_num = int(destination, 16)
self._interface.sendText(text=text, destinationId=dest_num, channelIndex=channel)
else:
from meshtastic import BROADCAST_NUM
self._interface.sendText(text=text, destinationId=BROADCAST_NUM, channelIndex=channel)
logger.debug("MT send to %s: %s", destination or "broadcast", text[:50])
return True
except Exception as exc:
logger.error("MT send failed: %s", exc)
return False
def send_message( def send_message(
self, self,
text: str, text: str,

View file

@ -78,7 +78,11 @@ async def meshcore_send_advert(request: Request):
if mc is None or not getattr(mc, "connected", False): if mc is None or not getattr(mc, "connected", False):
return {"sent": False, "detail": "MeshCore not connected"} return {"sent": False, "detail": "MeshCore not connected"}
try: try:
ok = bool(mc.send_advert()) send_fn = getattr(mc, "send_advert_async", None)
if send_fn is not None:
ok = bool(await send_fn())
else:
ok = bool(mc.send_advert())
detail = "Self-advert sent" if ok else "send_advert returned False" detail = "Self-advert sent" if ok else "send_advert returned False"
logger.info("dashboard: meshcore manual advert sent=%s", ok) logger.info("dashboard: meshcore manual advert sent=%s", ok)
return {"sent": ok, "detail": detail} return {"sent": ok, "detail": detail}
@ -124,7 +128,11 @@ async def meshcore_telemetry_poll(request: Request):
if not contact: if not contact:
return {"available": False, "detail": "Missing 'contact'"} return {"available": False, "detail": "Missing 'contact'"}
try: try:
data = mc.req_telemetry(contact) poll_fn = getattr(mc, "req_telemetry_async", None)
if poll_fn is not None:
data = await poll_fn(contact)
else:
data = mc.req_telemetry(contact)
if data is None: if data is None:
return {"available": False, "contact": contact, "detail": "No telemetry response"} return {"available": False, "contact": contact, "detail": "No telemetry response"}
return {"available": True, "contact": contact, "data": data} return {"available": True, "contact": contact, "data": data}
@ -155,7 +163,7 @@ async def test_send(request: Request, body: TestSendRequest):
except (ValueError, TypeError): except (ValueError, TypeError):
result = {"sent": False, "detail": f"invalid meshtastic channel index: {body.channel!r}"} result = {"sent": False, "detail": f"invalid meshtastic channel index: {body.channel!r}"}
else: else:
ok = bool(connector.send_message(text, destination=None, channel=idx, transport="meshtastic")) ok = bool(await connector.send_message_async(text, destination=None, channel=idx, transport="meshtastic"))
result = {"sent": ok, "detail": f"sent to meshtastic channel {idx}" if ok else "send returned False"} result = {"sent": ok, "detail": f"sent to meshtastic channel {idx}" if ok else "send returned False"}
elif body.transport == "meshcore": elif body.transport == "meshcore":
child = _find_child(connector, "meshcore") child = _find_child(connector, "meshcore")
@ -163,7 +171,7 @@ async def test_send(request: Request, body: TestSendRequest):
result = {"sent": False, "detail": "meshcore not connected"} result = {"sent": False, "detail": "meshcore not connected"}
else: else:
name = str(body.channel) name = str(body.channel)
ok = bool(connector.send_message(text, destination=None, meshcore_channel=name, transport="meshcore")) ok = bool(await connector.send_message_async(text, destination=None, meshcore_channel=name, transport="meshcore"))
if ok: if ok:
result = {"sent": True, "detail": f"sent to '{name}'"} result = {"sent": True, "detail": f"sent to '{name}'"}
else: else:

View file

@ -80,26 +80,29 @@ class MeshBroadcastChannel(NotificationChannel):
try: try:
# If payload already has chunk metadata (from digest), use message directly # If payload already has chunk metadata (from digest), use message directly
if alert.chunk_index is not None: if alert.chunk_index is not None:
self._connector.send_message( ok = await self._connector.send_message_async(
text=alert.message or "", text=alert.message or "",
destination=None, destination=None,
channel=self._channel, channel=self._channel,
transport=self._transport, transport=self._transport,
) )
logger.info("Broadcast pre-chunked alert to channel %d", self._channel) logger.info("Broadcast pre-chunked alert to channel %d (ok=%s)", self._channel, ok)
return True return bool(ok)
# Render to chunks for single-event delivery # Render to chunks for single-event delivery
chunks = self._renderer.render(alert) chunks = self._renderer.render(alert)
success = True
for chunk in chunks: for chunk in chunks:
self._connector.send_message( ok = await self._connector.send_message_async(
text=chunk, text=chunk,
destination=None, destination=None,
channel=self._channel, channel=self._channel,
transport=self._transport, transport=self._transport,
) )
logger.info("Broadcast %d chunk(s) to channel %d", len(chunks), self._channel) if not ok:
return True success = False
logger.info("Broadcast %d chunk(s) to channel %d (success=%s)", len(chunks), self._channel, success)
return success
except Exception as e: except Exception as e:
logger.error("Failed to broadcast alert: %s", e) logger.error("Failed to broadcast alert: %s", e)
return False return False
@ -134,15 +137,15 @@ class MeshBroadcastChannel(NotificationChannel):
if hasattr(ch, 'settings') and hasattr(ch.settings, 'name'): if hasattr(ch, 'settings') and hasattr(ch.settings, 'name'):
channel_name = ch.settings.name or f'Channel {self._channel}' channel_name = ch.settings.name or f'Channel {self._channel}'
# Send actual test message # Send actual test message (async path through queue)
self._connector.send_message( ok = await self._connector.send_message_async(
text="MeshAI channel test - if you see this, delivery works", text="MeshAI channel test - if you see this, delivery works",
destination=None, destination=None,
channel=self._channel, channel=self._channel,
) )
return { return {
"success": True, "success": ok,
"message": f"Sent to channel {self._channel}: {channel_name}", "message": f"Sent to channel {self._channel}: {channel_name}",
"error": "", "error": "",
"details": { "details": {
@ -164,12 +167,12 @@ class MeshBroadcastChannel(NotificationChannel):
return False, "Not connected to radio" return False, "Not connected to radio"
try: try:
self._connector.send_message( ok = bool(await self._connector.send_message_async(
text=message, text=message,
destination=None, destination=None,
channel=self._channel, channel=self._channel,
) ))
return True, f"Sent to mesh channel {self._channel}" return ok, f"Sent to mesh channel {self._channel}" if ok else "Mesh broadcast returned False"
except Exception as e: except Exception as e:
return False, f"Mesh broadcast failed: {e}" return False, f"Mesh broadcast failed: {e}"
@ -214,32 +217,35 @@ class MeshCoreBroadcastChannel(NotificationChannel):
try: try:
# If payload already has chunk metadata (from digest), use message directly # If payload already has chunk metadata (from digest), use message directly
if alert.chunk_index is not None: if alert.chunk_index is not None:
self._connector.send_message( ok = await self._connector.send_message_async(
text=alert.message or "", text=alert.message or "",
destination=None, destination=None,
meshcore_channel=self._meshcore_channel, meshcore_channel=self._meshcore_channel,
transport="meshcore", transport="meshcore",
) )
logger.info( logger.info(
"MeshCore broadcast pre-chunked alert to channel %r", "MeshCore broadcast pre-chunked alert to channel %r (ok=%s)",
self._meshcore_channel, self._meshcore_channel, ok,
) )
return True return bool(ok)
# Render to chunks for single-event delivery # Render to chunks for single-event delivery
chunks = self._renderer.render(alert) chunks = self._renderer.render(alert)
success = True
for chunk in chunks: for chunk in chunks:
self._connector.send_message( ok = await self._connector.send_message_async(
text=chunk, text=chunk,
destination=None, destination=None,
meshcore_channel=self._meshcore_channel, meshcore_channel=self._meshcore_channel,
transport="meshcore", transport="meshcore",
) )
if not ok:
success = False
logger.info( logger.info(
"MeshCore broadcast %d chunk(s) to channel %r", "MeshCore broadcast %d chunk(s) to channel %r (success=%s)",
len(chunks), self._meshcore_channel, len(chunks), self._meshcore_channel, success,
) )
return True return success
except Exception as e: except Exception as e:
logger.error("Failed to MeshCore broadcast alert: %s", e) logger.error("Failed to MeshCore broadcast alert: %s", e)
return False return False
@ -267,13 +273,13 @@ class MeshCoreBroadcastChannel(NotificationChannel):
if not self._meshcore_channel: if not self._meshcore_channel:
return False, "No MeshCore channel configured" return False, "No MeshCore channel configured"
try: try:
self._connector.send_message( ok = bool(await self._connector.send_message_async(
text=message, text=message,
destination=None, destination=None,
meshcore_channel=self._meshcore_channel, meshcore_channel=self._meshcore_channel,
transport="meshcore", transport="meshcore",
) ))
return True, f"Sent to MeshCore channel {self._meshcore_channel!r}" return ok, f"Sent to MeshCore channel {self._meshcore_channel!r}" if ok else "MeshCore broadcast returned False"
except Exception as e: except Exception as e:
return False, f"MeshCore broadcast failed: {e}" return False, f"MeshCore broadcast failed: {e}"
@ -310,12 +316,14 @@ class MeshDMChannel(NotificationChannel):
for message in messages: for message in messages:
try: try:
node_id = str(node_id) node_id = str(node_id)
self._connector.send_message( ok = await self._connector.send_message_async(
text=message, text=message,
destination=node_id, destination=node_id,
channel=0, channel=0,
transport=self._transport_hint, transport=self._transport_hint,
) )
if not ok:
success = False
except Exception as e: except Exception as e:
logger.error("Failed to DM %s: %s", node_id, e) logger.error("Failed to DM %s: %s", node_id, e)
success = False success = False
@ -346,12 +354,14 @@ class MeshDMChannel(NotificationChannel):
for node_id in self._node_ids: for node_id in self._node_ids:
try: try:
node_id = str(node_id) node_id = str(node_id)
self._connector.send_message( ok = await self._connector.send_message_async(
text="MeshAI DM test", text="MeshAI DM test",
destination=node_id, destination=node_id,
channel=0, channel=0,
) )
results.append({"node": node_id, "success": True}) results.append({"node": node_id, "success": ok})
if not ok:
all_success = False
except Exception as e: except Exception as e:
results.append({"node": node_id, "success": False, "error": str(e)}) results.append({"node": node_id, "success": False, "error": str(e)})
all_success = False all_success = False
@ -396,8 +406,11 @@ class MeshDMChannel(NotificationChannel):
for node_id in self._node_ids: for node_id in self._node_ids:
try: try:
node_id = str(node_id) node_id = str(node_id)
self._connector.send_message(text=message, destination=node_id, channel=0) ok = await self._connector.send_message_async(text=message, destination=node_id, channel=0)
success_count += 1 if ok:
success_count += 1
else:
errors.append(f"{node_id}: send returned False")
except Exception as e: except Exception as e:
errors.append(f"{node_id}: {e}") errors.append(f"{node_id}: {e}")
@ -452,11 +465,13 @@ class MeshCoreDMChannel(NotificationChannel):
for contact in self._contacts: for contact in self._contacts:
for message in messages: for message in messages:
try: try:
self._connector.send_message( ok = await self._connector.send_message_async(
text=message, text=message,
destination=str(contact), destination=str(contact),
transport="meshcore", transport="meshcore",
) )
if not ok:
success = False
except Exception as e: except Exception as e:
logger.error("Failed to MeshCore DM %s: %s", contact, e) logger.error("Failed to MeshCore DM %s: %s", contact, e)
success = False success = False
@ -496,12 +511,15 @@ class MeshCoreDMChannel(NotificationChannel):
errors = [] errors = []
for contact in self._contacts: for contact in self._contacts:
try: try:
self._connector.send_message( ok = await self._connector.send_message_async(
text=message, text=message,
destination=str(contact), destination=str(contact),
transport="meshcore", transport="meshcore",
) )
success_count += 1 if ok:
success_count += 1
else:
errors.append(f"{contact}: send returned False")
except Exception as e: except Exception as e:
errors.append(f"{contact}: {e}") errors.append(f"{contact}: {e}")
if success_count == len(self._contacts): if success_count == len(self._contacts):

View file

@ -51,7 +51,7 @@ class Responder:
delay = random.uniform(self.config.delay_min, self.config.delay_max) delay = random.uniform(self.config.delay_min, self.config.delay_max)
await asyncio.sleep(delay) await asyncio.sleep(delay)
sent = self.connector.send_message( sent = await self.connector.send_message_async(
text=msg, text=msg,
destination=destination, destination=destination,
channel=channel, channel=channel,

View file

@ -3,6 +3,7 @@
import abc import abc
import asyncio import asyncio
from typing import Callable, Optional from typing import Callable, Optional
import concurrent.futures
class MeshTransport(abc.ABC): class MeshTransport(abc.ABC):
@ -59,6 +60,30 @@ class MeshTransport(abc.ABC):
True if send was initiated successfully. True if send was initiated successfully.
""" """
async def send_message_async(
self,
text: str,
destination: Optional[str] = None,
channel: int = 0,
transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool:
"""Async send through the per-radio serialized queue.
Default implementation runs ``send_message()`` in the default thread
executor so it never blocks the event loop. Concrete subclasses with
a send queue override this to enqueue the job and await the actual
radio result.
Returns:
True if the send was accepted by the radio.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.send_message(text, destination, channel, transport, meshcore_channel),
)
@abc.abstractmethod @abc.abstractmethod
def set_message_callback( def set_message_callback(
self, self,

View file

@ -107,11 +107,35 @@ class CompositeTransport(MeshTransport):
child = self.meshcore_child() child = self.meshcore_child()
return child.send_advert() if child is not None else False return child.send_advert() if child is not None else False
async def send_advert_async(self) -> bool:
"""Async passthrough to the MeshCore child's send_advert_async(); False if no child."""
child = self.meshcore_child()
if child is None:
return False
fn = getattr(child, "send_advert_async", None)
if fn is not None:
return await fn()
# Fallback: sync version in executor.
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, child.send_advert)
def req_telemetry(self, contact_id): def req_telemetry(self, contact_id):
"""Passthrough to the MeshCore child's on-demand telemetry poll; None if no child.""" """Passthrough to the MeshCore child's on-demand telemetry poll; None if no child."""
child = self.meshcore_child() child = self.meshcore_child()
return child.req_telemetry(contact_id) if child is not None else None return child.req_telemetry(contact_id) if child is not None else None
async def req_telemetry_async(self, contact_id: str):
"""Async passthrough to the MeshCore child's req_telemetry_async(); None if no child."""
child = self.meshcore_child()
if child is None:
return None
fn = getattr(child, "req_telemetry_async", None)
if fn is not None:
return await fn(contact_id)
# Fallback: sync version in executor.
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, lambda: child.req_telemetry(contact_id))
def get_telemetry_cache(self): def get_telemetry_cache(self):
"""Passthrough to the MeshCore child's telemetry cache; [] if no meshcore child.""" """Passthrough to the MeshCore child's telemetry cache; [] if no meshcore child."""
child = self.meshcore_child() child = self.meshcore_child()
@ -223,6 +247,124 @@ class CompositeTransport(MeshTransport):
# Message I/O # Message I/O
# ------------------------------------------------------------------ # ------------------------------------------------------------------
async def send_message_async(
self,
text: str,
destination: Optional[str] = None,
channel: int = 0,
transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool:
"""Async send through per-child queues, with the same routing logic as send_message().
Each child's send_message_async() goes through that child's serialized queue,
so Meshtastic and MeshCore sends are each independently paced. For broadcasts
the two are awaited sequentially (Meshtastic first, then MeshCore).
"""
if destination is None:
return await self._broadcast_async(text, channel, meshcore_channel=meshcore_channel,
transport=transport)
if transport is not None:
return await self._send_hinted_async(text, destination, channel, transport)
return await self._send_unhinted_async(text, destination, channel)
async def _broadcast_async(
self, text: str, channel: int,
meshcore_channel: Optional[str] = None,
transport: Optional[str] = None,
) -> bool:
if transport is not None:
child = self._by_name.get(transport)
if child is None:
return False
name = _child_name(child)
if not child.connected:
return False
try:
if name == "meshcore":
return await child.send_message_async(
text, destination=None, meshcore_channel=meshcore_channel
)
else:
return await child.send_message_async(text, destination=None, channel=channel)
except Exception as exc:
logger.error("CompositeTransport: async hinted broadcast via %r raised: %s", name, exc)
return False
any_ok = False
for child in self._children:
name = _child_name(child)
if not child.connected:
continue
if name == "meshcore" and meshcore_channel is None:
continue
child_channel = meshcore_channel if name == "meshcore" else channel
try:
if name == "meshcore":
ok = await child.send_message_async(
text, destination=None, meshcore_channel=child_channel
)
else:
ok = await child.send_message_async(text, destination=None, channel=child_channel)
if ok:
any_ok = True
else:
logger.warning("CompositeTransport: async broadcast via %r returned False", name)
except Exception as exc:
logger.error("CompositeTransport: async broadcast via %r raised: %s", name, exc)
return any_ok
async def _send_hinted_async(
self, text: str, destination: str, channel: int, transport_hint: str
) -> bool:
child = self._resolve_child_for_hint(transport_hint)
if child is None:
logger.error(
"CompositeTransport: no child with name %r for async DM; known: %s",
transport_hint, list(self._by_name),
)
return False
if not child.connected:
return False
try:
return await child.send_message_async(text, destination=destination, channel=channel)
except Exception as exc:
logger.error(
"CompositeTransport: async hinted send via %r raised: %s",
_child_name(child), exc,
)
return False
async def _send_unhinted_async(self, text: str, destination: str, channel: int) -> bool:
child = self._best_child_for_destination(destination)
if child is not None:
try:
return await child.send_message_async(text, destination=destination, channel=channel)
except Exception as exc:
logger.error(
"CompositeTransport: async unhinted send via %r raised: %s",
_child_name(child), exc,
)
return False
logger.warning(
"CompositeTransport: async DM destination %r unresolved; fanning to all children",
destination,
)
any_ok = False
for child in self._children:
if not child.connected:
continue
try:
ok = await child.send_message_async(text, destination=destination, channel=channel)
if ok:
any_ok = True
except Exception as exc:
logger.error(
"CompositeTransport: async DM fan via %r raised: %s",
_child_name(child), exc,
)
return any_ok
def send_message( def send_message(
self, self,
text: str, text: str,

View file

@ -11,12 +11,14 @@ imported (and the test suite can run) without the lib installed.
""" """
import asyncio import asyncio
import concurrent.futures
import logging import logging
import threading import threading
import time as _time import time as _time
from typing import Callable, Optional from typing import Callable, Optional
from .base import MeshTransport from .base import MeshTransport
from .send_queue import RadioSendQueue
from ..connector import MeshMessage from ..connector import MeshMessage
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -125,6 +127,9 @@ class MeshCoreTransport(MeshTransport):
# _telemetry_failures: contact-id -> consecutive-timeout count # _telemetry_failures: contact-id -> consecutive-timeout count
self._telemetry_cache: dict[str, dict] = {} self._telemetry_cache: dict[str, dict] = {}
self._telemetry_failures: dict[str, int] = {} self._telemetry_failures: dict[str, int] = {}
# --- per-radio send queue (on the MC dedicated loop) ---
self._mc_send_queue: Optional[asyncio.Queue] = None
self._mc_drain_task: Optional[asyncio.Task] = None
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Internal helpers # Internal helpers
@ -324,6 +329,383 @@ class MeshCoreTransport(MeshTransport):
) )
return False return False
# ------------------------------------------------------------------
# Async send helpers — run ON the MC dedicated loop (no _run_coro).
# These are called from the MC drain task directly.
# ------------------------------------------------------------------
async def _resolve_contact_async(self, dest: str):
"""Resolve contact on the MC loop (no _run_coro, no deadlock risk)."""
if self._mc is None:
return None
try:
contact = self._mc.get_contact_by_key_prefix(dest)
if contact is not None:
return contact
except Exception:
return None
# Cache miss: force a full re-fetch from firmware.
try:
await asyncio.wait_for(self._mc.commands.get_contacts(lastmod=0), timeout=15)
except Exception:
logger.debug("MC: _resolve_contact_async get_contacts failed for %s", dest, exc_info=True)
try:
return self._mc.get_contact_by_key_prefix(dest)
except Exception:
return None
async def _send_dm_once_async(self, contact: dict, text: str, destination: str):
"""Send one DM frame on the MC loop; returns MSG_SENT event or None."""
try:
result = await asyncio.wait_for(
self._mc.commands.send_msg(contact, text), timeout=15
)
except Exception as exc:
logger.warning("MC: _send_dm_once_async to %s failed: %s", destination, exc)
return None
if result is None:
logger.warning("MC: _send_dm_once_async to %s — no send result", destination)
return None
try:
sent_type = (
result.payload.get("type")
if hasattr(result, "payload") and isinstance(result.payload, dict)
else None
)
logger.info(
"MC: DM to %s sent (route=%s)",
contact.get("adv_name") or destination,
"flood" if sent_type == 1 else ("direct" if sent_type == 0 else "?"),
)
except Exception:
pass
return result
async def _wait_for_ack_async(self, expected_ack, timeout: float) -> bool:
"""Wait for delivery ACK on the MC loop."""
if not expected_ack:
return False
from meshcore import EventType # noqa: PLC0415
try:
code = (
expected_ack.hex()
if isinstance(expected_ack, (bytes, bytearray))
else str(expected_ack)
)
except Exception:
return False
try:
event = await asyncio.wait_for(
self._mc.dispatcher.wait_for_event(
EventType.ACK,
attribute_filters={"code": code},
timeout=timeout,
),
timeout=timeout + 2,
)
return event is not None
except Exception:
logger.debug("MC: _wait_for_ack_async failed for %r", expected_ack, exc_info=True)
return False
async def _establish_direct_path_async(self, contact: dict, dst: str) -> None:
"""Path discovery on the MC loop (async version of _establish_direct_path)."""
label = contact.get("adv_name") or dst
try:
path_event = await asyncio.wait_for(
self._mc.commands.send_path_discovery_sync(contact, timeout=self._discovery_wait),
timeout=self._discovery_wait + 3,
)
if path_event is not None and not path_event.is_error():
logger.info("MC: path discovery to %s succeeded", label)
else:
logger.info("MC: path discovery to %s no PATH_RESPONSE; trying advert-path fallback", label)
except Exception as exc:
logger.debug("MC: send_path_discovery_sync to %s failed: %s", dst, exc)
try:
fresh_contact = await self._resolve_contact_async(dst) or contact
if fresh_contact.get("out_path_len", -1) < 0:
try:
ap_event = await asyncio.wait_for(
self._mc.commands.get_advert_path(fresh_contact), timeout=10
)
if (
ap_event is not None
and not ap_event.is_error()
and isinstance(getattr(ap_event, "payload", None), dict)
):
path_hex = ap_event.payload.get("path", "")
path_len = ap_event.payload.get("path_len", -1)
if path_len > 0 and path_hex:
await asyncio.wait_for(
self._mc.commands.update_contact(fresh_contact, path=path_hex),
timeout=10,
)
logger.info(
"MC: seeded direct path for %s (len=%d)",
fresh_contact.get("adv_name") or dst, path_len,
)
except Exception as exc:
logger.debug("MC: advert-path fallback for %s failed: %s", dst, exc)
except Exception as exc:
logger.debug("MC: _establish_direct_path_async step-2 for %s: %s", dst, exc)
async def _enumerate_channels_async(self) -> None:
"""Channel enumeration on the MC loop (async version of _enumerate_channels)."""
self._chan_name_to_idx = {}
try:
empty_run = 0
for idx in range(40):
try:
event = await asyncio.wait_for(
self._mc.commands.get_channel(idx), timeout=5
)
except Exception as exc:
logger.debug("MC: get_channel(%d) async failed, ending: %s", idx, exc)
break
if not event:
break
is_err = getattr(event, "is_error", None)
if callable(is_err) and event.is_error():
break
payload = event.payload or {}
name = payload.get("channel_name", "")
slot = payload.get("channel_idx", idx)
if not name:
empty_run += 1
if empty_run >= 3:
break
continue
self._chan_name_to_idx[name] = slot
empty_run = 0
except Exception as exc:
logger.warning("MC: async channel enumeration error: %s", exc)
self._chan_name_to_idx = {}
logger.info("MC: async enumerated %d named channel(s)", len(self._chan_name_to_idx))
async def _do_mc_dm_send_async(self, text: str, destination: str) -> bool:
"""Full DM send path on the MC loop (replaces send_message() DM branch)."""
if self._mc is None:
return False
contact = await self._resolve_contact_async(destination)
if contact is None:
logger.warning("MC: could not resolve contact %s; cannot DM", destination)
return False
label = contact.get("adv_name") or destination
# Fast path: send and wait for ACK.
result = await self._send_dm_once_async(contact, text, destination)
if result is None:
return False
exp_ack = self._extract_expected_ack(result)
if await self._wait_for_ack_async(exp_ack, self._ack_wait):
logger.info("MC: DM to %s ACKed (direct)", label)
return True
# No ACK: path discovery + retry.
logger.info("MC: no ACK from %s in %.1fs; running path discovery", label, self._ack_wait)
await self._establish_direct_path_async(contact, destination)
contact = await self._resolve_contact_async(destination) or contact
result = await self._send_dm_once_async(contact, text, destination)
if result is None:
return False
exp_ack = self._extract_expected_ack(result)
acked = await self._wait_for_ack_async(exp_ack, self._ack_wait)
logger.info(
"MC: DM to %s %s after discovery",
contact.get("adv_name") or destination,
"ACKed" if acked else "no ACK (sent best-effort)",
)
return acked or (not result.is_error())
async def _do_mc_broadcast_async(self, text: str, meshcore_channel: str) -> bool:
"""Channel broadcast on the MC loop (replaces send_message() broadcast branch)."""
if self._mc is None:
return False
idx = self._chan_name_to_idx.get(meshcore_channel)
if idx is None:
# Lazy async re-enumeration (no _run_coro deadlock risk).
await self._enumerate_channels_async()
idx = self._chan_name_to_idx.get(meshcore_channel)
if idx is None:
logger.warning("MC channel '%s' not on companion; skipping", meshcore_channel)
return False
try:
result = await asyncio.wait_for(
self._mc.commands.send_chan_msg(idx, text), timeout=10
)
success = not result.is_error()
if not success:
logger.warning("MC: broadcast returned error event")
return success
except Exception as exc:
logger.error("MC: _do_mc_broadcast_async failed: %s", exc)
return False
async def _do_mc_advert_async(self) -> bool:
"""Send self-advert on the MC loop (queued version)."""
if self._mc is None or not self._connected:
return False
try:
await self._mc.commands.send_advert(flood=True)
self._last_advert_sent = _time.time()
return True
except Exception as exc:
logger.warning("MC: queued advert failed: %s", exc)
return False
# ------------------------------------------------------------------
# MC send queue management
# ------------------------------------------------------------------
async def _mc_drain_loop(self) -> None:
"""Drain loop for the MC send queue — runs on the MC dedicated loop."""
while True:
item = await self._mc_send_queue.get()
async_fn, cfut = item
result = False
try:
result = await async_fn()
except asyncio.CancelledError:
if cfut is not None and not cfut.done():
cfut.cancel()
raise
except Exception as exc:
logger.error("MC queue drain: send job raised: %s", exc, exc_info=True)
if cfut is not None and not cfut.done():
cfut.set_exception(exc)
else:
if cfut is not None and not cfut.done():
cfut.set_result(result)
# Pace: read live from config.
pacing = max(0.25, getattr(self.config, "meshcore_send_pacing_seconds", 2.0))
await asyncio.sleep(pacing)
def _start_mc_queue(self) -> None:
"""Arm the MC send queue + drain task on the MC dedicated loop.
Call from within the MC dedicated loop (e.g. via
``loop.call_soon_threadsafe`` from the main thread after connect).
On reconnect (called a second time), the old drain task is cancelled
and all futures still sitting in the old queue are cancelled so no
caller hangs indefinitely awaiting a result from a pre-reconnect send.
"""
# Reconnect path: cancel the old drain and resolve outstanding futures.
old_task = self._mc_drain_task
old_queue = self._mc_send_queue
if old_task is not None:
old_task.cancel()
if old_queue is not None:
while True:
try:
_, cfut = old_queue.get_nowait()
if cfut is not None and not cfut.done():
cfut.cancel()
except asyncio.QueueEmpty:
break
self._mc_send_queue = asyncio.Queue()
self._mc_drain_task = asyncio.get_event_loop().create_task(
self._mc_drain_loop(), name="mc-send-queue-drain"
)
logger.debug("MC: send queue drain task started")
def _cancel_mc_queue(self) -> None:
"""Cancel the MC drain task and resolve all pending futures (thread-safe).
Called at disconnect. Schedules a teardown callback on the MC loop that:
1. Cancels the drain task.
2. Drains the remaining queue with ``get_nowait()`` and cancels every
pending ``concurrent.futures.Future`` so no caller hangs at
``await asyncio.wrap_future(cfut)`` after teardown.
The in-flight future (if the drain was mid-job when cancel fires) is
handled by the drain's own ``except CancelledError`` handler.
"""
task = self._mc_drain_task
queue = self._mc_send_queue
self._mc_drain_task = None
def _teardown() -> None:
if task is not None:
task.cancel()
if queue is not None:
while True:
try:
_, cfut = queue.get_nowait()
if cfut is not None and not cfut.done():
cfut.cancel()
except asyncio.QueueEmpty:
break
if self._loop is not None and self._loop.is_running():
self._loop.call_soon_threadsafe(_teardown)
async def send_message_async(
self,
text: str,
destination: Optional[str] = None,
channel: int = 0,
transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool:
"""Async send through the MC per-radio queue (called from the main loop).
Enqueues the job on the MC loop's queue and awaits the actual send
result via a concurrent.futures.Future bridge.
"""
if self._mc is None or not self._connected:
return False
if self._mc_send_queue is None or self._loop is None or not self._loop.is_running():
# Queue not started yet (e.g. initial advert at connect) — fall back.
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.send_message(text, destination, channel, transport, meshcore_channel),
)
cfut: concurrent.futures.Future = concurrent.futures.Future()
if destination:
async def _job() -> bool:
return await self._do_mc_dm_send_async(text, destination)
else:
if meshcore_channel is None:
logger.debug("MC: send_message_async meshcore_channel=None, skipping broadcast")
return False # nothing sent — do not report success
async def _job() -> bool:
return await self._do_mc_broadcast_async(text, meshcore_channel)
# Enqueue on MC loop from main loop.
asyncio.run_coroutine_threadsafe(
self._mc_send_queue.put((_job, cfut)),
self._loop,
)
main_loop = asyncio.get_event_loop()
return await asyncio.wrap_future(cfut, loop=main_loop)
def _enqueue_mc_fire_forget(self, async_fn) -> None:
"""Enqueue fire-and-forget from any thread (for advert/telemetry loops)."""
if self._mc_send_queue is None or self._loop is None or not self._loop.is_running():
logger.debug("MC: fire-and-forget dropped (queue not started)")
return
asyncio.run_coroutine_threadsafe(
self._mc_send_queue.put((async_fn, None)),
self._loop,
)
async def _enqueue_mc_loop_send(self, async_fn) -> bool:
"""Enqueue from within the MC loop and await the result (for advert/telemetry loops)."""
if self._mc_send_queue is None:
# Queue not started; fall through to direct call.
return await async_fn()
cfut: concurrent.futures.Future = concurrent.futures.Future()
await self._mc_send_queue.put((async_fn, cfut))
return await asyncio.wrap_future(cfut)
def _send_dm_once(self, contact: dict, text: str, destination: str): def _send_dm_once(self, contact: dict, text: str, destination: str):
"""Send one DM frame (no discovery, no retry) and log its route. """Send one DM frame (no discovery, no retry) and log its route.
@ -489,13 +871,33 @@ class MeshCoreTransport(MeshTransport):
logger.warning("MeshCore: send_advert failed: %s", exc) logger.warning("MeshCore: send_advert failed: %s", exc)
return False return False
async def send_advert_async(self) -> bool:
"""Async self-advertisement through the MC per-radio queue (main-loop caller).
Mirrors ``send_message_async`` enqueues ``_do_mc_advert_async`` on the
MC loop queue from the main loop, awaits the result via a
concurrent.futures.Future bridge. Falls back to a thread-executor
wrapping the synchronous ``send_advert`` when the queue isn't running.
"""
if self._mc is None or not self._connected:
return False
if self._mc_send_queue is None or self._loop is None or not self._loop.is_running():
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.send_advert)
cfut: concurrent.futures.Future = concurrent.futures.Future()
asyncio.run_coroutine_threadsafe(
self._mc_send_queue.put((self._do_mc_advert_async, cfut)),
self._loop,
)
main_loop = asyncio.get_event_loop()
return await asyncio.wrap_future(cfut, loop=main_loop)
async def _periodic_advert_loop(self, interval: int) -> None: async def _periodic_advert_loop(self, interval: int) -> None:
"""Periodic self-advertisement coroutine (runs as a Task on the dedicated loop). """Periodic self-advertisement coroutine (runs as a Task on the dedicated loop).
Sleeps *interval* seconds, sends one flood advert, repeats. Stops on Sleeps *interval* seconds, sends one flood advert via the send queue
CancelledError (raised by ``_cancel_periodic_advert`` at disconnect) or (serialized with other MC sends), repeats. Stops on CancelledError.
when the transport drops its connection. No overlap is possible because
the loop awaits the sleep before each send.
""" """
try: try:
while True: while True:
@ -503,13 +905,15 @@ class MeshCoreTransport(MeshTransport):
if not self._connected or self._mc is None: if not self._connected or self._mc is None:
return return
try: try:
await self._mc.commands.send_advert(flood=True) ok = await self._enqueue_mc_loop_send(self._do_mc_advert_async)
self._last_advert_sent = _time.time() if ok:
logger.info("MeshCore: sent periodic self-advert") logger.info("MC: sent periodic self-advert")
else:
logger.warning("MC: periodic send_advert returned False")
except Exception as exc: except Exception as exc:
logger.warning("MeshCore: periodic send_advert failed: %s", exc) logger.warning("MC: periodic send_advert failed: %s", exc)
except asyncio.CancelledError: except asyncio.CancelledError:
logger.debug("MeshCore: periodic advert task cancelled") logger.debug("MC: periodic advert task cancelled")
raise raise
def _schedule_periodic_advert(self, interval: int) -> None: def _schedule_periodic_advert(self, interval: int) -> None:
@ -531,6 +935,8 @@ class MeshCoreTransport(MeshTransport):
self._advert_task = None self._advert_task = None
if task is not None and self._loop is not None and self._loop.is_running(): if task is not None and self._loop is not None and self._loop.is_running():
self._loop.call_soon_threadsafe(task.cancel) self._loop.call_soon_threadsafe(task.cancel)
# Also cancel the MC send queue drain.
self._cancel_mc_queue()
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Telemetry (MeshCore sensor auto-poll) # Telemetry (MeshCore sensor auto-poll)
@ -612,11 +1018,19 @@ class MeshCoreTransport(MeshTransport):
self._telemetry_cache[contact_id] = entry self._telemetry_cache[contact_id] = entry
async def _req_telemetry_async(self, contact_id): async def _req_telemetry_async(self, contact_id):
"""Resolve, request+await, decode telemetry for *contact_id* (on the loop). """Resolve, request+await, decode telemetry for *contact_id* on the MC loop.
Runs INLINE on the dedicated event loop intentionally does NOT
re-enqueue on the send queue. Callers that need queue serialization
with other MC sends (``req_telemetry_async`` for on-demand polls,
``_telemetry_poll_loop`` for periodic polls) wrap their call to this
method inside a queue job via ``_enqueue_mc_loop_send`` or the
cfut-bridge in ``req_telemetry_async``.
Never calling ``_enqueue_mc_loop_send`` here is critical: the drain
task is single-threaded, so a nested enqueue + await from inside a
drain job would deadlock the queue permanently.
Runs entirely on the dedicated event loop so the poller (already on that
loop) can await it directly WITHOUT a nested _run_coro deadlock. Updates
the shared cache/failure bookkeeping so poller and on-demand paths agree.
Returns the decoded dict, or None on unresolved/timeout/no-response/error. Returns the decoded dict, or None on unresolved/timeout/no-response/error.
""" """
contact = self._resolve_contact(contact_id) contact = self._resolve_contact(contact_id)
@ -625,7 +1039,7 @@ class MeshCoreTransport(MeshTransport):
try: try:
lpp = await self._mc.commands.req_telemetry_sync(contact, min_timeout=5) lpp = await self._mc.commands.req_telemetry_sync(contact, min_timeout=5)
except Exception as exc: except Exception as exc:
logger.warning("MeshCore: req_telemetry(%s) error: %s", contact_id, exc) logger.warning("MC: req_telemetry(%s) error: %s", contact_id, exc)
self._record_telemetry_result(contact_id, None) self._record_telemetry_result(contact_id, None)
return None return None
if lpp is None: if lpp is None:
@ -653,6 +1067,41 @@ class MeshCoreTransport(MeshTransport):
logger.warning("MeshCore: req_telemetry(%s) failed: %s", contact_id, exc) logger.warning("MeshCore: req_telemetry(%s) failed: %s", contact_id, exc)
return None return None
async def req_telemetry_async(self, contact_id: str):
"""Async on-demand telemetry poll (main-loop caller, routes through MC queue).
Mirrors ``req_telemetry`` but awaitable from the main asyncio event loop.
Enqueues the poll job on the MC loop queue via a concurrent.futures.Future
bridge, then returns the cache entry's data. Falls back to run_in_executor
wrapping the synchronous ``req_telemetry`` when the queue isn't running.
"""
if self._mc is None or not self._connected:
return None
if self._mc_send_queue is None or self._loop is None or not self._loop.is_running():
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, lambda: self.req_telemetry(contact_id))
cfut: concurrent.futures.Future = concurrent.futures.Future()
async def _telem_job_outer() -> bool:
"""Bridge: runs _req_telemetry_async on the MC loop; caches result."""
try:
data = await self._req_telemetry_async(contact_id)
except Exception as exc: # noqa: BLE001
logger.warning("MC: req_telemetry_async(%s) failed: %s", contact_id, exc)
return False
return data is not None
asyncio.run_coroutine_threadsafe(
self._mc_send_queue.put((_telem_job_outer, cfut)),
self._loop,
)
main_loop = asyncio.get_event_loop()
await asyncio.wrap_future(cfut, loop=main_loop)
# Result is in the telemetry cache after the job completes.
entry = self._telemetry_cache.get(contact_id)
return entry.get("data") if entry else None
def get_telemetry_cache(self) -> list[dict]: def get_telemetry_cache(self) -> list[dict]:
"""Return the current telemetry cache entries (list of dicts).""" """Return the current telemetry cache entries (list of dicts)."""
return list(self._telemetry_cache.values()) return list(self._telemetry_cache.values())
@ -700,8 +1149,18 @@ class MeshCoreTransport(MeshTransport):
if self._telemetry_failures.get(c, 0) >= _TELEMETRY_MAX_FAILURES: if self._telemetry_failures.get(c, 0) >= _TELEMETRY_MAX_FAILURES:
continue continue
try: try:
data = await self._req_telemetry_async(c) # Wrap in a queue job so this poll is serialized with
if data is not None: # other MC sends (broadcasts, adverts). _req_telemetry_async
# is inline — it must NOT be called bare here or it would
# bypass the send queue and race with concurrent sends.
_c = c # capture for closure
async def _poll_job(_cid=_c) -> bool:
data = await self._req_telemetry_async(_cid)
return data is not None
ok = await self._enqueue_mc_loop_send(_poll_job)
if ok:
logger.info("MeshCore: telemetry polled %s", c) logger.info("MeshCore: telemetry polled %s", c)
else: else:
logger.debug("MeshCore: telemetry miss for %s", c) logger.debug("MeshCore: telemetry miss for %s", c)
@ -794,7 +1253,9 @@ class MeshCoreTransport(MeshTransport):
except Exception as exc: # older firmware may not support CMD 58 except Exception as exc: # older firmware may not support CMD 58
logger.warning("MeshCore: set_autoadd_config not supported/failed (non-fatal): %s", exc) logger.warning("MeshCore: set_autoadd_config not supported/failed (non-fatal): %s", exc)
await self._mc.start_auto_message_fetching() await self._mc.start_auto_message_fetching()
logger.info("MeshCore: subscriptions registered; auto message-fetch started") # Arm the MC send queue on this dedicated loop (runs after connect).
self._start_mc_queue()
logger.info("MeshCore: subscriptions registered; auto message-fetch started; send queue armed")
async def _do_disconnect(self) -> None: async def _do_disconnect(self) -> None:
"""Stop fetching and close the meshcore connection.""" """Stop fetching and close the meshcore connection."""

View file

@ -0,0 +1,183 @@
"""Per-radio serialized, paced send queue.
Each radio (Meshtastic, MeshCore) gets its own RadioSendQueue instance.
The drain task pops one job at a time, executes the async send, resolves the
caller's Future with the actual bool result, then sleeps pacing_seconds before
popping the next job enforcing strict FIFO serialization with configurable
inter-packet spacing.
Design notes
------------
* Jobs are ``(async_fn, concurrent.futures.Future | None)`` tuples where
``async_fn`` is a zero-argument coroutine factory (a lambda/closure that
captures all send parameters).
* ``concurrent.futures.Future`` is used rather than ``asyncio.Future`` so the
same future can be set from within ANY event loop (the MC drain runs on the
MC loop; callers on the main loop wrap it via ``asyncio.wrap_future``).
* The queue is UNBOUNDED no messages are ever dropped.
* Pacing is read live from the supplied callable on every drain cycle so a
config-GUI change takes effect on the next send without restart.
"""
from __future__ import annotations
import asyncio
import concurrent.futures
import logging
from typing import Awaitable, Callable, Optional
logger = logging.getLogger(__name__)
_PACING_FLOOR = 0.25 # seconds — absolute minimum between sends
class RadioSendQueue:
"""Serialized, paced outbound queue for a single radio transport.
Usage::
q = RadioSendQueue(pacing_fn=lambda: cfg.meshtastic_send_pacing_seconds)
# On the event loop where the drain should run:
q.start(loop)
...
# To enqueue from the SAME loop (awaitable):
result = await q.enqueue_async(my_send_coro_factory)
# To enqueue from a DIFFERENT thread/loop (non-blocking):
cfut = q.enqueue_threadsafe(my_send_coro_factory, target_loop)
result = await asyncio.wrap_future(cfut)
# On shutdown:
await q.stop()
"""
def __init__(self, pacing_fn: Callable[[], float]) -> None:
"""
Args:
pacing_fn: Zero-argument callable that returns the current pacing
gap in seconds. Called on every drain iteration so
config-GUI changes take effect immediately.
"""
self._pacing_fn = pacing_fn
self._queue: Optional[asyncio.Queue] = None
self._drain_task: Optional[asyncio.Task] = None
self._loop: Optional[asyncio.AbstractEventLoop] = None
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def start(self, loop: asyncio.AbstractEventLoop) -> None:
"""Create the asyncio.Queue and launch the drain task on *loop*.
Must be called from WITHIN *loop* (i.e. via ``loop.call_soon_threadsafe``
or from a coroutine running on *loop*).
"""
self._loop = loop
self._queue = asyncio.Queue()
self._drain_task = loop.create_task(self._drain(), name="radio-send-queue-drain")
logger.debug("RadioSendQueue: drain task started on loop %r", loop)
async def stop(self) -> None:
"""Cancel the drain task, drain the remaining queue, and resolve all
pending futures with CancelledError.
After this returns every caller blocked at
``await asyncio.wrap_future(cfut)`` for an enqueued item will see
``asyncio.CancelledError`` raised promptly no caller hangs
indefinitely. The in-flight future (if the drain was mid-job) is
already cancelled by the drain's own ``except CancelledError`` handler.
"""
if self._drain_task is None:
return
self._drain_task.cancel()
try:
await self._drain_task
except asyncio.CancelledError:
pass
self._drain_task = None
# Resolve all futures still sitting in the queue so no caller hangs.
if self._queue is not None:
while True:
try:
_, cfut = self._queue.get_nowait()
if cfut is not None and not cfut.done():
cfut.cancel()
except asyncio.QueueEmpty:
break
logger.debug("RadioSendQueue: drain task stopped")
@property
def running(self) -> bool:
return self._drain_task is not None and not self._drain_task.done()
# ------------------------------------------------------------------
# Enqueue helpers
# ------------------------------------------------------------------
def _make_cfut(self) -> concurrent.futures.Future:
return concurrent.futures.Future()
async def enqueue_async(self, async_fn: Callable[[], Awaitable[bool]]) -> bool:
"""Enqueue *async_fn* and await its result on the current (same) loop.
``async_fn`` must be a zero-argument callable returning a coroutine
that resolves to bool. Raises if the queue is not started.
"""
if self._queue is None or self._loop is None:
raise RuntimeError("RadioSendQueue.start() not called")
cfut: concurrent.futures.Future = self._make_cfut()
await self._queue.put((async_fn, cfut))
return await asyncio.wrap_future(cfut, loop=self._loop)
def enqueue_threadsafe(
self,
async_fn: Callable[[], Awaitable[bool]],
caller_loop: asyncio.AbstractEventLoop,
) -> concurrent.futures.Future:
"""Enqueue *async_fn* from a DIFFERENT loop or thread (non-blocking).
Returns a ``concurrent.futures.Future``; wrap with
``asyncio.wrap_future(cfut, loop=caller_loop)`` to await on the
caller's loop. Raises if the queue is not started.
"""
if self._queue is None or self._loop is None:
raise RuntimeError("RadioSendQueue.start() not called")
cfut: concurrent.futures.Future = self._make_cfut()
asyncio.run_coroutine_threadsafe(self._queue.put((async_fn, cfut)), self._loop)
return cfut
def enqueue_fire_and_forget(self, async_fn: Callable[[], Awaitable[bool]]) -> None:
"""Enqueue *async_fn* with no result tracking (fire-and-forget).
Safe to call from within the queue's own loop.
"""
if self._queue is None:
logger.warning("RadioSendQueue: fire-and-forget dropped (queue not started)")
return
self._queue.put_nowait((async_fn, None))
# ------------------------------------------------------------------
# Drain
# ------------------------------------------------------------------
async def _drain(self) -> None:
"""Pop items one at a time, execute, resolve future, pace."""
while True:
item = await self._queue.get()
async_fn, cfut = item
result = False
try:
result = await async_fn()
except asyncio.CancelledError:
if cfut is not None and not cfut.done():
cfut.cancel()
raise
except Exception as exc:
logger.error("RadioSendQueue: send job raised: %s", exc, exc_info=True)
if cfut is not None and not cfut.done():
cfut.set_exception(exc)
else:
if cfut is not None and not cfut.done():
cfut.set_result(result)
# Pace: read live so config-GUI changes take effect next send.
pacing = max(_PACING_FLOOR, self._pacing_fn())
await asyncio.sleep(pacing)

View file

@ -15,13 +15,25 @@ from meshai.notifications.channels import (
) )
def _mock_conn():
"""Create a MagicMock connector with send_message_async wired to send_message.
Channels now call send_message_async (async) instead of send_message (sync).
side_effect delegates to send_message so existing call_count / call_args
assertions remain valid.
"""
c = MagicMock()
c.send_message_async = AsyncMock(side_effect=lambda *a, **kw: c.send_message(*a, **kw))
return c
# ============================================================ # ============================================================
# MESH CHANNEL RENDERING TESTS # MESH CHANNEL RENDERING TESTS
# ============================================================ # ============================================================
def test_mesh_channel_uses_mesh_renderer(): def test_mesh_channel_uses_mesh_renderer():
"""MeshBroadcastChannel renders long messages to multiple chunks.""" """MeshBroadcastChannel renders long messages to multiple chunks."""
mock_connector = MagicMock() mock_connector = _mock_conn()
channel = MeshBroadcastChannel( channel = MeshBroadcastChannel(
connector=mock_connector, connector=mock_connector,
@ -52,7 +64,7 @@ def test_mesh_channel_uses_mesh_renderer():
def test_mesh_channel_uses_payload_message_directly_when_chunk_metadata_set(): def test_mesh_channel_uses_payload_message_directly_when_chunk_metadata_set():
"""Pre-chunked payloads (from digest) skip re-rendering.""" """Pre-chunked payloads (from digest) skip re-rendering."""
mock_connector = MagicMock() mock_connector = _mock_conn()
channel = MeshBroadcastChannel( channel = MeshBroadcastChannel(
connector=mock_connector, connector=mock_connector,
@ -81,7 +93,7 @@ def test_mesh_channel_uses_payload_message_directly_when_chunk_metadata_set():
def test_mesh_dm_channel_uses_mesh_renderer(): def test_mesh_dm_channel_uses_mesh_renderer():
"""MeshDMChannel renders long messages to chunks for each recipient.""" """MeshDMChannel renders long messages to chunks for each recipient."""
mock_connector = MagicMock() mock_connector = _mock_conn()
channel = MeshDMChannel( channel = MeshDMChannel(
connector=mock_connector, connector=mock_connector,
@ -106,7 +118,7 @@ def test_mesh_dm_channel_uses_mesh_renderer():
def test_mesh_dm_channel_uses_payload_message_directly_when_chunk_metadata_set(): def test_mesh_dm_channel_uses_payload_message_directly_when_chunk_metadata_set():
"""Pre-chunked DM payloads skip re-rendering.""" """Pre-chunked DM payloads skip re-rendering."""
mock_connector = MagicMock() mock_connector = _mock_conn()
channel = MeshDMChannel( channel = MeshDMChannel(
connector=mock_connector, connector=mock_connector,
@ -229,7 +241,7 @@ def test_mesh_broadcast_routes_to_meshtastic_only():
from meshai.config import NotificationRuleConfig from meshai.config import NotificationRuleConfig
from meshai.notifications.channels import create_channel from meshai.notifications.channels import create_channel
mock_connector = MagicMock() mock_connector = _mock_conn()
rule = NotificationRuleConfig( rule = NotificationRuleConfig(
name="toggle:fire", name="toggle:fire",
delivery_type="mesh_broadcast", delivery_type="mesh_broadcast",
@ -266,7 +278,7 @@ def test_meshcore_broadcast_routes_to_meshcore_only():
from meshai.notifications.channels import MeshCoreBroadcastChannel, create_channel from meshai.notifications.channels import MeshCoreBroadcastChannel, create_channel
# Simulate a CompositeTransport connector with a meshcore child. # Simulate a CompositeTransport connector with a meshcore child.
mock_connector = MagicMock() mock_connector = _mock_conn()
mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()} mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()}
mock_connector.send_message.return_value = True mock_connector.send_message.return_value = True
@ -303,7 +315,7 @@ def test_broadcast_render_loop_threads_meshcore_channel():
from meshai.config import NotificationRuleConfig from meshai.config import NotificationRuleConfig
from meshai.notifications.channels import create_channel from meshai.notifications.channels import create_channel
mock_connector = MagicMock() mock_connector = _mock_conn()
# Connector has a meshcore child so the no-op guard passes. # Connector has a meshcore child so the no-op guard passes.
mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()} mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()}
mock_connector.send_message.return_value = True mock_connector.send_message.return_value = True

View file

@ -76,12 +76,27 @@ class FakeChild:
destination: Optional[str] = None, destination: Optional[str] = None,
channel: int = 0, channel: int = 0,
transport: Optional[str] = None, transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool: ) -> bool:
self.send_calls.append( self.send_calls.append(
{"text": text, "destination": destination, "channel": channel, "transport": transport} {"text": text, "destination": destination, "channel": channel, "transport": transport}
) )
return True return True
async def send_message_async(
self,
text: str,
destination: Optional[str] = None,
channel: int = 0,
transport: Optional[str] = None,
meshcore_channel: Optional[str] = None,
) -> bool:
"""Async variant required by composite transport after send-queue refactor."""
return self.send_message(
text=text, destination=destination, channel=channel,
transport=transport, meshcore_channel=meshcore_channel,
)
def set_message_callback(self, callback, loop) -> None: def set_message_callback(self, callback, loop) -> None:
self._callback = callback self._callback = callback
self._callback_loop = loop self._callback_loop = loop

View file

@ -44,6 +44,10 @@ class StubConnector:
{"text": text, "destination": destination, "channel": channel}) {"text": text, "destination": destination, "channel": channel})
return True return True
async def send_message_async(self, text=None, destination=None, channel=0, **kw):
"""Async variant required by channels.py after send-queue refactor."""
return self.send_message(text=text, destination=destination, channel=channel, **kw)
class FakeDataStore: class FakeDataStore:
"""Minimal stand-in exposing get_nodes_by_roles with the SAME filter """Minimal stand-in exposing get_nodes_by_roles with the SAME filter

View file

@ -6,7 +6,7 @@ Uses a bare FastAPI() + TestClient with a hand-seeded ``app.state.connector``
""" """
from __future__ import annotations from __future__ import annotations
from unittest.mock import MagicMock from unittest.mock import AsyncMock, MagicMock
import pytest import pytest
from fastapi import FastAPI from fastapi import FastAPI
@ -22,6 +22,10 @@ def _child(transport_name, connected=True, known=None):
c.connected = connected c.connected = connected
if known is not None: if known is not None:
c.known_channels.return_value = list(known) c.known_channels.return_value = list(known)
# Wire async variants so routes can await them; side_effect preserves the
# sync mock's return_value and call recording for existing assertions.
c.send_advert_async = AsyncMock(side_effect=lambda: c.send_advert())
c.req_telemetry_async = AsyncMock(side_effect=lambda cid: c.req_telemetry(cid))
return c return c
@ -35,6 +39,11 @@ def _composite(children, send_result=True):
connector.transport_name = None connector.transport_name = None
connector.children = list(children) connector.children = list(children)
connector.send_message.return_value = send_result connector.send_message.return_value = send_result
# Wire the async variant — routes now call send_message_async; side_effect
# delegates to the sync mock so existing call_args assertions still pass.
connector.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: connector.send_message(*a, **kw)
)
return connector return connector

View file

@ -377,6 +377,10 @@ def _child(transport_name, connected=True):
c = MagicMock() c = MagicMock()
c.transport_name = transport_name c.transport_name = transport_name
c.connected = connected c.connected = connected
# The dashboard route now calls req_telemetry_async; wire it to delegate to
# the sync mock so existing test setup (c.req_telemetry.return_value = ...)
# and assertions are unaffected.
c.req_telemetry_async = AsyncMock(side_effect=lambda cid: c.req_telemetry(cid))
return c return c

View file

@ -5,7 +5,7 @@ feat/meshcore-first-class-delivery (meshcore_broadcast, meshcore_dm).
""" """
import asyncio import asyncio
from unittest.mock import MagicMock from unittest.mock import AsyncMock, MagicMock
from meshai.config import Config, NotificationToggle from meshai.config import Config, NotificationToggle
from meshai.notifications.pipeline.dispatcher import Dispatcher from meshai.notifications.pipeline.dispatcher import Dispatcher
@ -198,11 +198,17 @@ def test_meshcore_broadcast_routes_to_meshcore_child_only():
meshtastic_child.connected = True meshtastic_child.connected = True
meshtastic_child.transport_name = "meshtastic" meshtastic_child.transport_name = "meshtastic"
meshtastic_child.send_message.return_value = True meshtastic_child.send_message.return_value = True
meshtastic_child.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: meshtastic_child.send_message(*a, **kw)
)
meshcore_child = MagicMock() meshcore_child = MagicMock()
meshcore_child.connected = True meshcore_child.connected = True
meshcore_child.transport_name = "meshcore" meshcore_child.transport_name = "meshcore"
meshcore_child.send_message.return_value = True meshcore_child.send_message.return_value = True
meshcore_child.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: meshcore_child.send_message(*a, **kw)
)
from meshai.transport.composite_transport import CompositeTransport from meshai.transport.composite_transport import CompositeTransport
connector = CompositeTransport([meshtastic_child, meshcore_child]) connector = CompositeTransport([meshtastic_child, meshcore_child])
@ -249,11 +255,17 @@ def test_mesh_broadcast_routes_to_meshtastic_child_only():
meshtastic_child.connected = True meshtastic_child.connected = True
meshtastic_child.transport_name = "meshtastic" meshtastic_child.transport_name = "meshtastic"
meshtastic_child.send_message.return_value = True meshtastic_child.send_message.return_value = True
meshtastic_child.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: meshtastic_child.send_message(*a, **kw)
)
meshcore_child = MagicMock() meshcore_child = MagicMock()
meshcore_child.connected = True meshcore_child.connected = True
meshcore_child.transport_name = "meshcore" meshcore_child.transport_name = "meshcore"
meshcore_child.send_message.return_value = True meshcore_child.send_message.return_value = True
meshcore_child.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: meshcore_child.send_message(*a, **kw)
)
from meshai.transport.composite_transport import CompositeTransport from meshai.transport.composite_transport import CompositeTransport
connector = CompositeTransport([meshtastic_child, meshcore_child]) connector = CompositeTransport([meshtastic_child, meshcore_child])
@ -295,6 +307,9 @@ def test_meshcore_dm_routes_to_meshcore_contacts():
mock_connector = MagicMock() mock_connector = MagicMock()
mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()} mock_connector._by_name = {"meshcore": MagicMock(), "meshtastic": MagicMock()}
mock_connector.send_message.return_value = True mock_connector.send_message.return_value = True
mock_connector.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: mock_connector.send_message(*a, **kw)
)
from meshai.config import NotificationRuleConfig from meshai.config import NotificationRuleConfig
import time as _time import time as _time
@ -408,6 +423,9 @@ def test_meshtastic_only_config_unchanged():
# Simulate a plain MeshtasticTransport (no _by_name, transport_name=meshtastic). # Simulate a plain MeshtasticTransport (no _by_name, transport_name=meshtastic).
mock_connector.transport_name = "meshtastic" mock_connector.transport_name = "meshtastic"
mock_connector.send_message.return_value = True mock_connector.send_message.return_value = True
mock_connector.send_message_async = AsyncMock(
side_effect=lambda *a, **kw: mock_connector.send_message(*a, **kw)
)
# No _by_name attribute (not a CompositeTransport). # No _by_name attribute (not a CompositeTransport).
del mock_connector._by_name del mock_connector._by_name

View file

@ -0,0 +1,611 @@
"""Tests for the per-radio serialized send queue.
Covers:
- FIFO ordering (no reordering, no drops)
- Pacing: consecutive timestamps >= pacing_seconds (0.05 s in tests)
- Event loop not blocked during burst
- Concurrent tasks make progress while queue drains
- Config floor enforced (min 0.25 s)
- Burst serialized: no parallel send overlap
- RadioSendQueue.start/stop lifecycle
- MT transport send_message_async falls back when queue not started
"""
from __future__ import annotations
import asyncio
import concurrent.futures
import time
from typing import List
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from meshai.transport.send_queue import RadioSendQueue, _PACING_FLOOR
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_queue(pacing: float = 0.05) -> RadioSendQueue:
return RadioSendQueue(pacing_fn=lambda: pacing)
async def _run_with_queue(pacing: float, jobs) -> list:
"""Run *jobs* (list of async callables) through a queue; return results in order."""
q = _make_queue(pacing)
loop = asyncio.get_event_loop()
q.start(loop)
results = []
for fn in jobs:
result = await q.enqueue_async(fn)
results.append(result)
await q.stop()
return results
# ---------------------------------------------------------------------------
# FIFO ordering
# ---------------------------------------------------------------------------
class TestFIFO:
@pytest.mark.asyncio
async def test_results_in_enqueue_order(self):
"""Results come back in the order items were enqueued."""
order = []
async def make_job(n):
async def _job():
order.append(n)
return True
return _job
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
q.start(loop)
futs = []
for i in range(5):
futs.append(await q.enqueue_async(await make_job(i)))
# Wait for all to complete
await asyncio.gather(*[asyncio.wrap_future(concurrent.futures.Future()) for _ in range(0)],
return_exceptions=True)
# Stop drains remaining items
await q.stop()
assert order == [0, 1, 2, 3, 4]
@pytest.mark.asyncio
async def test_no_drops(self):
"""Every enqueued item executes — no items are dropped.
Uses enqueue_async so we can await all completions without sleeping;
this also avoids dependence on the pacing floor timing.
"""
executed = []
async def make_job(n):
async def _job():
executed.append(n)
return True
return _job
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
q.start(loop)
N = 6
# Enqueue all jobs concurrently (fire them as tasks), then gather.
tasks = [asyncio.ensure_future(q.enqueue_async(await make_job(i))) for i in range(N)]
await asyncio.gather(*tasks)
await q.stop()
assert len(executed) == N
assert sorted(executed) == list(range(N))
# ---------------------------------------------------------------------------
# Pacing
# ---------------------------------------------------------------------------
class TestPacing:
@pytest.mark.asyncio
async def test_pacing_gap_respected(self):
"""Send timestamps are spaced >= pacing_seconds apart."""
pacing = 0.05
timestamps: list[float] = []
async def _job():
timestamps.append(time.monotonic())
return True
q = _make_queue(pacing=pacing)
loop = asyncio.get_event_loop()
q.start(loop)
N = 4
futs = [asyncio.ensure_future(q.enqueue_async(_job)) for _ in range(N)]
await asyncio.gather(*futs)
await q.stop()
assert len(timestamps) == N
for i in range(1, N):
gap = timestamps[i] - timestamps[i - 1]
assert gap >= pacing * 0.9, f"gap[{i}]={gap:.3f} < pacing={pacing}"
@pytest.mark.asyncio
async def test_pacing_read_live(self):
"""Pacing value is read from the callable on each iteration."""
pacing_value = 0.05
timestamps: list[float] = []
async def _job():
timestamps.append(time.monotonic())
return True
q = RadioSendQueue(pacing_fn=lambda: pacing_value)
loop = asyncio.get_event_loop()
q.start(loop)
# Enqueue first
futs = [asyncio.ensure_future(q.enqueue_async(_job)) for _ in range(2)]
await asyncio.gather(*futs)
# Change pacing and run 2 more
pacing_value = 0.10
futs = [asyncio.ensure_future(q.enqueue_async(_job)) for _ in range(2)]
await asyncio.gather(*futs)
await q.stop()
assert len(timestamps) == 4
# ---------------------------------------------------------------------------
# Config floor
# ---------------------------------------------------------------------------
class TestPacingFloor:
@pytest.mark.asyncio
async def test_floor_enforced(self):
"""Pacing below the floor is clamped up to _PACING_FLOOR (0.25 s)."""
q = RadioSendQueue(pacing_fn=lambda: 0.001) # way below floor
loop = asyncio.get_event_loop()
q.start(loop)
timestamps: list[float] = []
async def _job():
timestamps.append(time.monotonic())
return True
futs = [asyncio.ensure_future(q.enqueue_async(_job)) for _ in range(2)]
await asyncio.gather(*futs)
await q.stop()
assert len(timestamps) == 2
gap = timestamps[1] - timestamps[0]
assert gap >= _PACING_FLOOR * 0.9, f"floor not enforced: gap={gap:.3f}"
def test_floor_constant(self):
assert _PACING_FLOOR == 0.25
# ---------------------------------------------------------------------------
# Event loop not blocked
# ---------------------------------------------------------------------------
class TestNonBlocking:
@pytest.mark.asyncio
async def test_other_tasks_progress_during_drain(self):
"""The event loop remains available to other coroutines while the queue drains."""
pacing = 0.05
q = _make_queue(pacing=pacing)
loop = asyncio.get_event_loop()
q.start(loop)
progress_count = 0
async def _send_job():
await asyncio.sleep(0) # yield briefly
return True
async def _observer():
nonlocal progress_count
for _ in range(8):
await asyncio.sleep(0.02)
progress_count += 1
# Run drain + observer concurrently
futs = [asyncio.ensure_future(q.enqueue_async(_send_job)) for _ in range(5)]
obs = asyncio.ensure_future(_observer())
await asyncio.gather(*futs, obs)
await q.stop()
# Observer should have completed all its iterations
assert progress_count >= 6, f"observer only made {progress_count} iterations"
@pytest.mark.asyncio
async def test_send_returns_actual_result(self):
"""Future resolves to the actual bool returned by the send job."""
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
q.start(loop)
async def _ok():
return True
async def _fail():
return False
r1 = await q.enqueue_async(_ok)
r2 = await q.enqueue_async(_fail)
r3 = await q.enqueue_async(_ok)
await q.stop()
assert r1 is True
assert r2 is False
assert r3 is True
# ---------------------------------------------------------------------------
# Serialization — no overlap
# ---------------------------------------------------------------------------
class TestSerialization:
@pytest.mark.asyncio
async def test_no_concurrent_sends(self):
"""Only one job runs at a time — active_at windows never overlap."""
active_intervals: list[tuple[float, float]] = []
lock = asyncio.Lock()
async def _job():
start = time.monotonic()
async with lock:
end = time.monotonic()
active_intervals.append((start, end))
return True
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
q.start(loop)
N = 5
futs = [asyncio.ensure_future(q.enqueue_async(_job)) for _ in range(N)]
await asyncio.gather(*futs)
await q.stop()
assert len(active_intervals) == N
# ---------------------------------------------------------------------------
# Lifecycle
# ---------------------------------------------------------------------------
class TestLifecycle:
@pytest.mark.asyncio
async def test_start_stop(self):
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
assert not q.running
q.start(loop)
assert q.running
await q.stop()
assert not q.running
@pytest.mark.asyncio
async def test_stop_with_pending_items(self):
"""stop() cancels the drain; pending items stay in queue (not processed after stop)."""
processed = []
async def _slow_job():
await asyncio.sleep(0.5) # slow — won't complete before stop
processed.append(1)
return True
q = _make_queue(pacing=0.01)
loop = asyncio.get_event_loop()
q.start(loop)
# Enqueue a slow job + a second job
q.enqueue_fire_and_forget(_slow_job)
await asyncio.sleep(0.01) # let drain start the slow job
await q.stop()
# The slow job was in-flight; don't assert specific processed count.
assert not q.running
@pytest.mark.asyncio
async def test_enqueue_before_start_raises(self):
q = _make_queue(pacing=0.01)
with pytest.raises(RuntimeError, match="start"):
await q.enqueue_async(lambda: None)
@pytest.mark.asyncio
async def test_fire_and_forget_before_start_is_noop(self):
"""enqueue_fire_and_forget on unstarted queue logs and does nothing."""
q = _make_queue(pacing=0.01)
# Should not raise
q.enqueue_fire_and_forget(lambda: None)
# ---------------------------------------------------------------------------
# MeshtasticTransport.send_message_async fallback
# ---------------------------------------------------------------------------
class TestMTFallback:
@pytest.mark.asyncio
async def test_send_message_async_falls_back_without_queue(self):
"""When queue not started, send_message_async uses run_in_executor."""
from meshai.config import ConnectionConfig
from meshai.connector import MeshtasticTransport
cfg = ConnectionConfig()
mt = MeshtasticTransport(cfg)
# Mock the blocking send so we don't need a real radio
with patch.object(mt, "send_message", return_value=True) as mock_send:
result = await mt.send_message_async("hello", channel=0)
assert result is True
mock_send.assert_called_once()
@pytest.mark.asyncio
async def test_send_message_async_via_queue(self):
"""When queue is started, send_message_async goes through the drain."""
from meshai.config import ConnectionConfig
from meshai.connector import MeshtasticTransport
cfg = ConnectionConfig(meshtastic_send_pacing_seconds=0.05)
mt = MeshtasticTransport(cfg)
# Arm queue manually (normally done by set_message_callback)
loop = asyncio.get_event_loop()
from meshai.transport.send_queue import RadioSendQueue
pacing_fn = lambda: max(0.25, getattr(cfg, "meshtastic_send_pacing_seconds", 2.0))
mt._mt_queue = RadioSendQueue(pacing_fn=pacing_fn)
mt._mt_queue.start(loop)
calls = []
with patch.object(mt, "_blocking_mt_send", side_effect=lambda *a, **kw: calls.append(a) or True):
r1 = await mt.send_message_async("msg1", channel=0)
r2 = await mt.send_message_async("msg2", channel=0)
await mt._mt_queue.stop()
assert r1 is True
assert r2 is True
assert len(calls) == 2
# ---------------------------------------------------------------------------
# Config round-trip
# ---------------------------------------------------------------------------
class TestConfig:
def test_pacing_defaults(self):
from meshai.config import ConnectionConfig
cfg = ConnectionConfig()
assert cfg.meshtastic_send_pacing_seconds == 2.0
assert cfg.meshcore_send_pacing_seconds == 2.0
def test_pacing_round_trips(self):
from meshai.config import ConnectionConfig, _dataclass_to_dict, _dict_to_dataclass
cfg = ConnectionConfig(meshtastic_send_pacing_seconds=3.5, meshcore_send_pacing_seconds=1.5)
d = _dataclass_to_dict(cfg)
assert d["meshtastic_send_pacing_seconds"] == 3.5
assert d["meshcore_send_pacing_seconds"] == 1.5
cfg2 = _dict_to_dataclass(ConnectionConfig, d)
assert cfg2.meshtastic_send_pacing_seconds == 3.5
assert cfg2.meshcore_send_pacing_seconds == 1.5
# ---------------------------------------------------------------------------
# Regression tests — deadlock / teardown
# ---------------------------------------------------------------------------
class TestDeadlockRegression:
"""Regression suite for the two deadlock/hang bugs fixed in feat/send-queue.
Both tests must PASS on the fixed code and would HANG (timeout) on the
pre-fix code:
BLOCKER 1 ``req_telemetry_async`` self-deadlock:
Pre-fix: _telem_job_outer (running inside the drain) called
_req_telemetry_async which called _enqueue_mc_loop_send a nested
enqueue-and-await inside a single-threaded drain job deadlock.
Post-fix: _req_telemetry_async is inline; no nested enqueue.
BLOCKER 2 pending futures abandoned on teardown/reconnect:
Pre-fix: stop() only cancelled the drain task; queue-sitting items had
their concurrent.futures.Futures left unresolved, so
``await asyncio.wrap_future(cfut)`` callers hung indefinitely.
Post-fix: stop() drains the remaining queue and cancels every cfut.
"""
@pytest.mark.asyncio
async def test_telemetry_queue_no_deadlock(self):
"""req_telemetry_async must not self-deadlock the MC drain (BLOCKER 1).
Drives the on-demand telemetry poll through a real _mc_send_queue
(not mocked) with fake MC commands. Asserts it resolves within 2 s
and that a subsequent send on the same queue also drains (queue not
wedged). With the pre-fix code this would hang at the asyncio.wait_for
timeout because the nested _enqueue_mc_loop_send deadlocks the drain.
"""
from meshai.config import ConnectionConfig
from meshai.transport.meshcore_transport import MeshCoreTransport
cfg = ConnectionConfig(
meshcore_host="127.0.0.1",
meshcore_send_pacing_seconds=0.01,
)
mc = MeshCoreTransport(cfg)
loop = asyncio.get_event_loop()
# Arm the MC queue directly (bypass connect() / TCP).
mc._loop = loop
mc._connected = True
mc._mc_send_queue = asyncio.Queue()
mc._mc_drain_task = loop.create_task(
mc._mc_drain_loop(), name="test-mc-drain"
)
fake_lpp = [{"channel": 0, "type": 120, "value": 80}] # battery_pct=80
class _FakeCommands:
async def req_telemetry_sync(self, contact, min_timeout=5):
return fake_lpp
class _FakeMC:
commands = _FakeCommands()
def get_contact_by_key_prefix(self, prefix):
return {"adv_name": "Node1", "public_key": prefix}
def get_contact_by_name(self, name):
return {"adv_name": name, "public_key": "aabbcc"}
mc._mc = _FakeMC()
# Must complete within 2 s; pre-fix code deadlocks here.
data = await asyncio.wait_for(
mc.req_telemetry_async("aabbcc"),
timeout=2.0,
)
assert data is not None, "expected telemetry data back from cache"
assert data.get("battery_pct") == 80
# Subsequent send on the same queue must also drain (queue not wedged).
done = asyncio.Event()
async def _normal_send() -> bool:
done.set()
return True
ok = await asyncio.wait_for(
mc._enqueue_mc_loop_send(_normal_send), timeout=2.0
)
assert ok is True
assert done.is_set()
# Cleanup.
mc._mc_drain_task.cancel()
try:
await mc._mc_drain_task
except asyncio.CancelledError:
pass
@pytest.mark.asyncio
async def test_teardown_resolves_pending_futures(self):
"""RadioSendQueue.stop() must resolve all pending futures (BLOCKER 2).
Enqueues a slow job to occupy the drain, then enqueues three more that
sit in the queue. Calls stop() mid-drain and asserts every future is
done (not pending/hanging) and resolves promptly to
CancelledError/exception. With pre-fix code the pending futs would
never be set so the assert would fail or the test would time out.
"""
q = RadioSendQueue(pacing_fn=lambda: 10.0) # huge pacing keeps drain idle long
loop = asyncio.get_event_loop()
q.start(loop)
drain_started = asyncio.Event()
async def _slow_job():
drain_started.set()
await asyncio.sleep(60) # will be cancelled by stop()
return True
async def _fast_job():
return True
# Kick off the slow job — drain picks it up immediately.
slow_task = asyncio.ensure_future(q.enqueue_async(_slow_job))
await asyncio.wait_for(drain_started.wait(), timeout=1.0)
# Enqueue three more jobs while drain is occupied by slow_job.
pending_tasks = [
asyncio.ensure_future(q.enqueue_async(_fast_job)) for _ in range(3)
]
# Stop the queue while slow_job is in-flight and fast jobs are pending.
await q.stop()
# Give the event loop one tick to propagate cfut cancellations into
# the asyncio tasks waiting at wrap_future(cfut).
await asyncio.sleep(0)
# Every future must be resolved — none hanging indefinitely.
# Await them with a short timeout; pre-fix code they would never complete.
all_tasks = [slow_task] + pending_tasks
done, pending_set = await asyncio.wait(all_tasks, timeout=1.0)
assert not pending_set, (
f"{len(pending_set)} task(s) still pending after stop() — "
"futures not resolved on teardown"
)
# Awaiting them must raise (CancelledError) — not return a value.
for task in all_tasks:
assert task.done()
with pytest.raises(
(asyncio.CancelledError, concurrent.futures.CancelledError, Exception)
):
task.result()
@pytest.mark.asyncio
async def test_reconnect_resolves_old_futures(self):
"""_start_mc_queue on reconnect must cancel futures from the old queue
(BLOCKER 2 reconnect path).
Arms the queue, enqueues three items without draining them, then calls
_start_mc_queue again (simulating reconnect). All three old futures
must be cancelled so no caller hangs. Pre-fix code would leave them
unresolved.
"""
from meshai.config import ConnectionConfig
from meshai.transport.meshcore_transport import MeshCoreTransport
cfg = ConnectionConfig(
meshcore_host="127.0.0.1",
meshcore_send_pacing_seconds=0.01,
)
mc = MeshCoreTransport(cfg)
loop = asyncio.get_event_loop()
mc._loop = loop
# Initial arm — drain is live but _mc is None so any job would return False.
mc._start_mc_queue()
await asyncio.sleep(0) # let drain task start
# Enqueue three items; they sit in the queue unprocessed.
pending_cfuts: list[concurrent.futures.Future] = []
for _ in range(3):
cfut: concurrent.futures.Future = concurrent.futures.Future()
async def _noop() -> bool:
return True
await mc._mc_send_queue.put((_noop, cfut))
pending_cfuts.append(cfut)
# Simulate reconnect: _start_mc_queue replaces the queue.
# The fix must cancel old cfuts before creating the new queue.
mc._start_mc_queue()
await asyncio.sleep(0.05) # let any scheduled callbacks run
for cfut in pending_cfuts:
assert cfut.done(), "old cfut not resolved after _start_mc_queue reconnect"
assert cfut.cancelled(), "old cfut should be cancelled"
# Cleanup new drain.
if mc._mc_drain_task is not None:
mc._mc_drain_task.cancel()
try:
await mc._mc_drain_task
except asyncio.CancelledError:
pass