mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-08-26 17:31:34 +00:00
feat(transport): per-radio serialized+paced outbound send queue (#93)
* feat(transport): per-radio serialized+paced outbound send queue Prevents simultaneous LoRa transmissions when N events arrive at once. ## Mechanism Two `RadioSendQueue` instances (one MT, one MC), each a FIFO asyncio.Queue with a long-running drain task. The MT queue drains on the main asyncio loop; the MC queue drains on MeshCore's dedicated event-loop thread. - MT sends: `run_in_executor` offloads the blocking `sendText` call; queue started in `set_message_callback`, cancelled in `disconnect`. - MC sends: drain loop runs pure-async MC lib coroutines directly on the MC loop (no `_run_coro` deadlock); cross-loop callers bridge via `concurrent.futures.Future` + `asyncio.wrap_future`. - Pacing: `await asyncio.sleep(pacing_seconds)` between items; read live from config per iteration; floor clamped to 0.25 s. - Config knobs: `meshtastic_send_pacing_seconds` (default 2.0) and `meshcore_send_pacing_seconds` (default 2.0) on `ConnectionConfig`. ## Send sites rerouted All callers now `await connector.send_message_async(...)`: - `notifications/channels.py` — MeshBroadcast/MeshCoreBroadcast/MeshDM/ MeshCoreDM deliver(), test_connection(), deliver_test() - `responder.py` — DM replies in send_response() - `transport/meshcore_transport.py` — periodic_advert_loop, telemetry poll loop, send_advert() → send_advert_async(), req_telemetry() → req_telemetry_async() (all queue-routed from main loop) - `dashboard/api/mesh_send_routes.py` — test-send, advert, telemetry poll ## Audit accuracy `deliver()` now returns the actual bool from the radio send (not optimistic True), so `mesh_broadcasts_out` reflects the real result. ## Tests 17 new tests in tests/test_send_queue.py covering FIFO ordering, no drops, pacing gap, pacing floor enforcement, event-loop non-blocking, serialization, lifecycle, MT fallback, config round-trip. Existing test stubs updated to wire `send_message_async = AsyncMock(side_effect=send_message)` so prior call_count / call_args assertions remain valid without changes. Full suite: 2135 passed, 17 pre-existing failures (unchanged), 0 new regressions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(send-queue): resolve MC telemetry self-deadlock + resolve pending futures on teardown/reconnect; composite MC-channel kwarg; audit no-op false BLOCKER 1 — req_telemetry_async self-deadlock (meshcore_transport.py): _req_telemetry_async was calling _enqueue_mc_loop_send inside itself; when _telem_job_outer ran inside the drain it nested another enqueue+await on the same single-threaded drain — permanent deadlock on first telemetry poll. Fix: _req_telemetry_async is now fully inline (no _enqueue_mc_loop_send). _telemetry_poll_loop wraps its call in _enqueue_mc_loop_send for serialization. req_telemetry_async's outer job calls _req_telemetry_async inline (safe). BLOCKER 2 — pending futures abandoned on teardown/reconnect: RadioSendQueue.stop() only cancelled the drain task; queue-sitting items had their concurrent.futures.Futures left unresolved, causing wrap_future() callers to hang indefinitely. Fix: stop() drains the remaining queue with get_nowait() and cancels every pending cfut. _cancel_mc_queue() schedules the same drain- and-cancel via call_soon_threadsafe. _start_mc_queue() cancels old drain task and drains old queue cfuts before arming the new queue (reconnect path). connector.disconnect() now .result(timeout=5) on stop() instead of fire-and-forget. SHOULD-FIX 3 — composite passes MC channel as wrong kwarg (composite_transport.py): _broadcast_async no-hint loop was calling send_message_async(channel=child_channel) for the meshcore child; should be meshcore_channel=child_channel. Silent drop fixed. NIT 5 — false success on zero-channel MC send (meshcore_transport.py): send_message_async returned True when meshcore_channel is None (nothing sent). Now returns False so audit does not record a success for a no-op. NIT 7 — config comment contradiction (config.py): meshtastic_send_pacing_seconds comment said "0 disables the floor" while simultaneously stating "still floored at 0.25". Removed the contradiction. Regression tests (tests/test_send_queue.py — 3 new, all in TestDeadlockRegression): - test_telemetry_queue_no_deadlock: drives req_telemetry_async through a real _mc_send_queue with fake MC commands; times out on pre-fix code (deadlock). - test_teardown_resolves_pending_futures: enqueues slow+fast jobs, stops mid-drain, asserts every task resolves promptly; hangs on pre-fix code. - test_reconnect_resolves_old_futures: calls _start_mc_queue twice, asserts old cfuts are cancelled; pre-fix leaves them unresolved. All 17 pre-existing send-queue tests still pass (20 total now). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
6e74b82d51
commit
a7b7f5a6a4
16 changed files with 1661 additions and 63 deletions
|
|
@ -76,12 +76,27 @@ class FakeChild:
|
|||
destination: Optional[str] = None,
|
||||
channel: int = 0,
|
||||
transport: Optional[str] = None,
|
||||
meshcore_channel: Optional[str] = None,
|
||||
) -> bool:
|
||||
self.send_calls.append(
|
||||
{"text": text, "destination": destination, "channel": channel, "transport": transport}
|
||||
)
|
||||
return True
|
||||
|
||||
async def send_message_async(
|
||||
self,
|
||||
text: str,
|
||||
destination: Optional[str] = None,
|
||||
channel: int = 0,
|
||||
transport: Optional[str] = None,
|
||||
meshcore_channel: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""Async variant required by composite transport after send-queue refactor."""
|
||||
return self.send_message(
|
||||
text=text, destination=destination, channel=channel,
|
||||
transport=transport, meshcore_channel=meshcore_channel,
|
||||
)
|
||||
|
||||
def set_message_callback(self, callback, loop) -> None:
|
||||
self._callback = callback
|
||||
self._callback_loop = loop
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue