mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-08-26 17:31:34 +00:00
docs: add routing revamp architecture plan with Phase A amendment
Includes multi-file config layout finding (Amendment A1) discovered during Phase A implementation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
3d7a0349a4
commit
612b76cd10
1 changed files with 267 additions and 0 deletions
267
docs/meshai-routing-revamp-plan.md
Normal file
267
docs/meshai-routing-revamp-plan.md
Normal file
|
|
@ -0,0 +1,267 @@
|
||||||
|
# MeshAI Broadcast Routing Revamp — Architecture Review & Plan
|
||||||
|
**Reviewed at:** main @ 98e3fcf (2026-06-10)
|
||||||
|
**Method:** three-pass read of the notification subsystem — full read, plan construction, line-level verification of every claim.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PART 1 — How routing actually works today
|
||||||
|
|
||||||
|
There are **five parallel delivery paths**, not one system with layers.
|
||||||
|
|
||||||
|
### Path 1: Legacy NotificationRouter (`notifications/router.py`)
|
||||||
|
Fed by `main._dispatch_alerts()` (main.py:644-684) ← `alert_engine.check()` — mesh-health
|
||||||
|
and env-poll alerts as raw dicts. Iterates `config.notifications.rules`
|
||||||
|
(NotificationRuleConfig): category match → `min_severity` gate → per-rule
|
||||||
|
`cooldown_minutes` dedup keyed on `(rule_name, category, event_id-or-message[:50])`
|
||||||
|
(router.py:165-172) → builds channel from the rule's **inline transport fields** → delivers.
|
||||||
|
- Writes **no** `mesh_broadcasts_out` audit row.
|
||||||
|
- Rule stats in `/data/rule_stats.json` (router.py:24), separate from SQLite.
|
||||||
|
- Mesh messages >200 chars go through **LLM summarization** (router.py:187-193).
|
||||||
|
- Also backs the over-the-mesh `!subscribe` command — `add_mesh_subscription()`
|
||||||
|
(router.py:737) dynamically creates a mesh_dm rule per subscriber node — and
|
||||||
|
`generate_report()` used by report delivery (router.py:628).
|
||||||
|
|
||||||
|
### Path 2: Dispatcher rules path (`pipeline/dispatcher.py:252-279`)
|
||||||
|
Fed by the EventBus. Evaluates **the same rules list** against bus Events
|
||||||
|
(`_matching_rules`, dispatcher.py:632-653: enabled → condition → category →
|
||||||
|
min_severity → region_scope). Then delivers.
|
||||||
|
- **No cooldown. No dedup. No staleness. No audit row.** None of the toggle-path
|
||||||
|
guards apply here.
|
||||||
|
|
||||||
|
### Path 3: Dispatcher toggle path (`pipeline/dispatcher.py:281-456`) — the good one
|
||||||
|
Fed by the EventBus through the pipeline: `bus → Inhibitor (inhibit_keys, ttl 1800s,
|
||||||
|
persisted) → Grouper (group_key window 60s, persisted) → ToggleFilter (family enabled
|
||||||
|
set) → tee(dispatch + DigestAccumulator)`.
|
||||||
|
|
||||||
|
Inside `_dispatch_toggles`, in order:
|
||||||
|
0. Cold-start grace (60s after first event, persisted anchor)
|
||||||
|
1. Staleness — `toggle.freshness_seconds` (fire family reads `wfigs.freshness_seconds`)
|
||||||
|
2. Cooldown — per `(toggle, category, region|_cooldown_suffix)`, persisted; **immediate
|
||||||
|
severity bypasses** (dispatcher.py:363-364)
|
||||||
|
3. Dedup — `(source, event.id)` LRU 10k, 7-day SQLite window
|
||||||
|
4. Region scope → **`min_severity` gate (line 420-422)** → **`severity_channels`
|
||||||
|
matrix (line 434-435)** → composer (150-byte budget) → channel per matrix entry →
|
||||||
|
`_post_broadcast_commit`: `mesh_broadcasts_out` audit row + handler callback.
|
||||||
|
|
||||||
|
### Path 4: Scheduled broadcasts (`dispatch_scheduled_broadcast`, dispatcher.py:475-561)
|
||||||
|
Band conditions (3×/day), fire digest (2×/day), reminders — all bypass the pipeline.
|
||||||
|
Only cold-start grace applies. **All three hardcode-route through
|
||||||
|
`rf_propagation.broadcast_channel`** (dispatcher.py:512-519). Writes audit row.
|
||||||
|
|
||||||
|
### Path 5: Fallbacks
|
||||||
|
`mesh_intelligence.alert_channel` + subscriber DMs when no NotificationRouter
|
||||||
|
(main.py:660-682); SubscriptionManager scheduled DM reports (main.py:686+).
|
||||||
|
|
||||||
|
### Pre-pipeline gates (upstream of all of the above)
|
||||||
|
Adapter floors (swpc kp/flare/proton), handler change-detection + cooldowns (fire 8h
|
||||||
|
update etc.), consumer **default-deny** — no synthesized wire string → Event never
|
||||||
|
enters the bus (consumer.py:548-567).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PART 2 — Verified defects and redundancies
|
||||||
|
|
||||||
|
Every item re-checked against source in pass 3.
|
||||||
|
|
||||||
|
| # | Finding | Evidence |
|
||||||
|
|---|---------|----------|
|
||||||
|
| B1 | **Threshold + matrix double-gate.** `min_severity` gates before `severity_channels`; a routine matrix row is dead config when threshold=priority. Exactly the GUI trap from the screenshot. | dispatcher.py:420-422 vs 434-435 |
|
||||||
|
| B2 | **The "digest" matrix column is a no-op.** Dispatcher skips it (`if ch_type == "digest": continue`); digest membership is actually `digest.include` (toggle-name list). Checking the box changes nothing. By design per `test_digest_channel_skipped_in_live_dispatch`, but the GUI presents it as live routing. | dispatcher.py:436-437; pipeline/__init__.py include_toggles |
|
||||||
|
| B3 | **Rules-path deliveries are invisible.** Neither Path 1 nor Path 2 writes `mesh_broadcasts_out`. Forensics on that table only sees toggle + scheduled traffic. | router.py:208; dispatcher.py:263-279 |
|
||||||
|
| B4 | **Path 2 has zero spam protection.** A rule matching a chatty central category re-delivers every event. | dispatcher.py:252-279 |
|
||||||
|
| B5 | **Double-delivery by design.** `dispatch()` runs rules AND toggles; both matching = same event broadcast twice, possibly same channel. Tested as intended behavior (`test_rules_and_toggles_both_fire`). | dispatcher.py:247-250 |
|
||||||
|
| B6 | **Channel 0 is falsy.** `if rf is None or not getattr(rf, "broadcast_channel", None)` — `rf_propagation.broadcast_channel = 0` silently drops ALL scheduled broadcasts (band conditions, fire digest, reminders). Channel 0 is a legitimate primary channel. | dispatcher.py:515 |
|
||||||
|
| B7 | **Fire digest + reminders ride the RF toggle's channel.** Fire content routed by RF-propagation transport config; disabling/misconfiguring the RF toggle silently kills fire digests. | dispatcher.py:512-519; fire_digest.py:255; reminders/__init__.py:315 |
|
||||||
|
| B8 | **Severity fails open** in the legacy router — unknown severity string returns True. | router.py:223-224 |
|
||||||
|
| B9 | **Transport config duplicated everywhere.** Full SMTP credential block inline in every rule AND every toggle (config.py:503-556, 558-580). The stale-SMTP-per-rule failure mode is structural. |
|
||||||
|
| B10 | **Two mesh formatting regimes.** Legacy: LLM-summarize >200 chars. Toggle: deterministic composer, 150-byte budget. Same radio, different text rules. | router.py:187-193 vs composer.py:31 |
|
||||||
|
| B11 | **Three incompatible dedup keys** (legacy `(rule,cat,event_id|msg[:50])` in-memory; toggle `(source,event.id)` persisted; rules-path none). |
|
||||||
|
| B12 | GUI copy bug: severity helper says "Warning" recommended — not a severity level in this system (routine/priority/immediate). Same line still has stale `text-slate-600`. | Notifications.tsx:605 |
|
||||||
|
|
||||||
|
**Bearing on today's "only band conditions" mystery:** fires/traffic went out via the
|
||||||
|
toggle path on `fires`/`roads` toggles' `broadcast_channel` (default → 0 via
|
||||||
|
`_toggle_to_rule`'s `or 0`); band conditions went out on `rf_propagation.broadcast_channel`,
|
||||||
|
which (per B6) **must be non-zero or they'd have been dropped**. So band conditions and
|
||||||
|
fire/traffic provably left on **different channel indices**. If the node was only
|
||||||
|
listening where band conditions land, that's the whole mystery. The forensics query
|
||||||
|
(`select channel, source_event_table from mesh_broadcasts_out`) confirms in one pass.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PART 3 — The revamp plan
|
||||||
|
|
||||||
|
**Principle: consolidate onto the toggle path.** It's the battle-tested one — guards,
|
||||||
|
persistence, audit, composer. We're not building a third system; we're deleting the
|
||||||
|
other four and fixing the toggle path's warts.
|
||||||
|
|
||||||
|
### A. Sinks — transports defined once
|
||||||
|
New `SinkConfig` dataclass + `sinks: dict[str, SinkConfig]` on NotificationsConfig:
|
||||||
|
```yaml
|
||||||
|
notifications:
|
||||||
|
sinks:
|
||||||
|
mesh-primary: {type: mesh_broadcast, channel: 0}
|
||||||
|
mesh-alerts: {type: mesh_broadcast, channel: 2}
|
||||||
|
dm-ops: {type: mesh_dm, node_ids: ["!abcd1234"]}
|
||||||
|
email-ops: {type: email, smtp_host: ..., recipients: [...]}
|
||||||
|
```
|
||||||
|
- `channels.py` gains `create_channel_from_sink(sink, connector)`. Existing channel
|
||||||
|
classes unchanged.
|
||||||
|
- All transport fields (`broadcast_channel`, `node_ids`, `smtp_*`, `webhook_*`)
|
||||||
|
**removed** from NotificationToggle and NotificationRuleConfig (kept dataclass-side
|
||||||
|
only during migration window).
|
||||||
|
- Channel index stored as `int`, validated `>= 0` — kills B6's falsy-zero class of bug.
|
||||||
|
|
||||||
|
### B. One severity mechanism — matrix only
|
||||||
|
- Delete `NotificationToggle.min_severity`. The `severity_channels` matrix becomes the
|
||||||
|
only gate; values are **sink names**, not channel types:
|
||||||
|
`severity_channels: {routine: [], priority: [mesh-primary], immediate: [mesh-primary, dm-ops]}`
|
||||||
|
- Empty list = that severity doesn't deliver. Threshold semantics are now expressible,
|
||||||
|
visible, and conflict-free (kills B1).
|
||||||
|
- Remove the "digest" pseudo-channel from the matrix (kills B2). Digest membership
|
||||||
|
stays `digest.include`; GUI gets a separate per-family "include in digest" checkbox
|
||||||
|
wired to it honestly.
|
||||||
|
- Migration: for each existing toggle, blank matrix rows below old `min_severity`,
|
||||||
|
map `mesh_broadcast`→auto-created sink from its `broadcast_channel`, etc. Write the
|
||||||
|
migration as `scripts/migrate_config_routing.py` mirroring `migrate_config_v03.py`.
|
||||||
|
|
||||||
|
### C. Retire the rules system (both evaluators)
|
||||||
|
- Delete `Dispatcher._dispatch_rules` and the legacy `NotificationRouter.process_alert`
|
||||||
|
delivery path (kills B3/B4/B5/B8/B10/B11).
|
||||||
|
- `main._dispatch_alerts` is replaced by: **alert_engine emits Events onto the bus**
|
||||||
|
(`make_event(source="alert_engine", category=<existing category ids>, ...)`).
|
||||||
|
Categories already map to the `mesh_health` family via `categories.py`, so
|
||||||
|
mesh-health alerts inherit ToggleFilter, guards, composer, audit — everything.
|
||||||
|
- The `!subscribe` mesh command moves off dynamic rules: subscriptions table already
|
||||||
|
exists; implement as **per-node DM sinks auto-managed by the command**, matched by
|
||||||
|
family in the dispatcher (small `subscriber_index: {family: [node_ids]}` consulted
|
||||||
|
after the matrix). `generate_report()` and the band-conditions compute helpers move
|
||||||
|
out of router.py into `notifications/reports.py`; router.py is then deleted.
|
||||||
|
- `rule_stats.json` → replaced by routing_log (below).
|
||||||
|
|
||||||
|
### D. Routing trace — `routing_log` table
|
||||||
|
```
|
||||||
|
routing_log(id, ts, event_source, event_category, event_id, severity,
|
||||||
|
family, stage, outcome, sink, detail)
|
||||||
|
```
|
||||||
|
- One row per decision: `stage` ∈ {toggle_filter, cold_start, staleness, cooldown,
|
||||||
|
dedup, region, matrix, delivery}; `outcome` ∈ {pass, drop, sent, failed}.
|
||||||
|
- Dispatcher's four counters stay (cheap), but every drop now also writes its reason
|
||||||
|
row. Retention: 14 days, pruned on insert like dispatcher_dedup.
|
||||||
|
- Surface: `GET /api/routing/log?source=&category=&since=` + a Carbon dashboard panel.
|
||||||
|
"Why didn't I get the fire alert" becomes one query.
|
||||||
|
|
||||||
|
### E. Dry-run endpoint
|
||||||
|
`POST /api/routing/test` with `{source, category, severity, region, lat, lon}` →
|
||||||
|
synthesize an Event, run the FULL pipeline with delivery stubbed → return the
|
||||||
|
stage-by-stage trace (same shape as routing_log rows) without keying the radio.
|
||||||
|
Pairs with `central.<domain>.test.>` subjects for end-to-end with-radio tests.
|
||||||
|
|
||||||
|
### F. Scheduled broadcasts get their own sinks
|
||||||
|
- `band_conditions.sink`, `fires.digest_sink`, per-reminder `sink` in adapter_config
|
||||||
|
(defaults: `mesh-primary`). Kills B6/B7 — fire digests no longer depend on the RF
|
||||||
|
toggle, and channel 0 works.
|
||||||
|
- `dispatch_scheduled_broadcast(text, sink_name, ...)` — grace check unchanged,
|
||||||
|
audit row unchanged.
|
||||||
|
|
||||||
|
### G. GUI rebuild (Notifications.tsx)
|
||||||
|
- New **Sinks** section (one-time setup, test button per sink — `test_connection()`
|
||||||
|
already exists per channel class).
|
||||||
|
- Family cards shrink to: enabled, regions, freshness, cooldown, matrix of
|
||||||
|
severity → sink multi-select, digest-include checkbox. Threshold dropdown, channel
|
||||||
|
config block, SMTP block all deleted from the card.
|
||||||
|
- Routing log panel + dry-run form.
|
||||||
|
- Fix B12 copy + stale slate class while in there.
|
||||||
|
|
||||||
|
### Phasing for CC (each phase independently shippable)
|
||||||
|
1. **Phase A** — SinkConfig + sinks in config + `create_channel_from_sink` + migration
|
||||||
|
script + tests. No behavior change yet (toggles still read inline config if sinks absent).
|
||||||
|
2. **Phase B** — matrix-to-sinks + min_severity removal + migration + dispatcher change
|
||||||
|
+ GUI family cards + tests.
|
||||||
|
3. **Phase C** — routing_log + dry-run endpoint + GUI panel.
|
||||||
|
4. **Phase D** — alert_engine → bus; delete _dispatch_rules, legacy router delivery,
|
||||||
|
_dispatch_alerts; move reports/subscribe surfaces; delete rule_stats.
|
||||||
|
5. **Phase E** — scheduled-broadcast sinks (B6/B7 fix).
|
||||||
|
|
||||||
|
Order rationale: A is pure addition (zero risk); B delivers the conflict fix you hit
|
||||||
|
today; C delivers the observability that prevents the next debugging afternoon; D is
|
||||||
|
the big deletion and needs C's trace to validate; E is small and independent (can be
|
||||||
|
pulled forward if fire digests misbehave before then).
|
||||||
|
|
||||||
|
### Risks / blast radius
|
||||||
|
- **Config migration** is the dangerous part: live `/data/config/env_feeds.yaml` +
|
||||||
|
notifications config carry real toggles. Migration must be idempotent, back up the
|
||||||
|
file first, and refuse to run twice.
|
||||||
|
- **Tests:** 68 test files; `test_v052_dispatcher.py`, `test_notification_toggles.py`,
|
||||||
|
`test_cold_start_grace.py`, `test_dispatcher_persistence.py`, `test_pipeline_*.py`
|
||||||
|
directly assert current dual-path semantics. `test_rules_and_toggles_both_fire`
|
||||||
|
inverts (must assert single delivery). Budget real time per phase for test rework.
|
||||||
|
- **`/api/config` PUT live-refresh** (`_refresh_toggle_filter`, config_routes.py:228+)
|
||||||
|
must learn to refresh sinks too, or channel edits need restart.
|
||||||
|
- **Mesh `!subscribe`** is user-facing on-air behavior — Phase D must not break
|
||||||
|
existing subscriber DMs; migrate the subscriptions table contents.
|
||||||
|
- Magic data-key contract (`_broadcast_audit`, `_on_broadcast_committed`,
|
||||||
|
`_severity_override`, `_meshai_precomposed`, `_cooldown_suffix`) spans 7 central
|
||||||
|
handlers — **unchanged by this plan**; the revamp is strictly downstream of the bus.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PART 4 — Verification pass results
|
||||||
|
|
||||||
|
Claims re-checked against source after plan construction:
|
||||||
|
|
||||||
|
- Dual gate (B1): confirmed at dispatcher.py:420-422 (rank check) preceding 434
|
||||||
|
(matrix read). The matrix is unreachable for severities below threshold.
|
||||||
|
- Matrix digest no-op (B2): the ONLY reader of `severity_channels` in the codebase is
|
||||||
|
dispatcher.py:434 (grep verified); digest content driven solely by
|
||||||
|
`DigestAccumulator(include_toggles=digest.include)`.
|
||||||
|
- Audit asymmetry (B3): `_post_broadcast_commit` called only at dispatcher.py:452
|
||||||
|
(toggle path) and inline insert at :549 (scheduled). No insert in `_dispatch_rules`
|
||||||
|
or router.py.
|
||||||
|
- Channel-0 falsy (B6): `not getattr(rf, "broadcast_channel", None)` — `not 0 == True`.
|
||||||
|
Confirmed your live system must have rf channel ≠ 0 (band conditions are delivering),
|
||||||
|
which also confirms the channel-split explanation for today's symptom.
|
||||||
|
- Double-delivery intended (B5): `dispatch()` awaits both paths unconditionally;
|
||||||
|
covered by `test_rules_and_toggles_both_fire`.
|
||||||
|
- Composer is sole mesh-format authority on the toggle path (mesh.py `_format_one_line`
|
||||||
|
passes message through verbatim, per the v0.5.7-regression note) — so sink
|
||||||
|
consolidation does not change wire format. Legacy-path retirement (Phase D) removes
|
||||||
|
the only competing formatter.
|
||||||
|
- Pipeline order: bus → inhibitor → grouper → toggle_filter → tee
|
||||||
|
(pipeline/__init__.py:118-121) — sinks plan touches none of these stages.
|
||||||
|
- Scheduled paths all funnel through `dispatch_scheduled_broadcast` (band_conditions,
|
||||||
|
fire_digest.py:255, reminders/__init__.py:315) — single choke point makes Phase E a
|
||||||
|
one-function change plus three config keys.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PART 5 — Implementation amendments
|
||||||
|
|
||||||
|
### Amendment A1: Multi-file config layout (Phase A discovery)
|
||||||
|
|
||||||
|
**Finding:** The live system uses a **multi-file config layout** where
|
||||||
|
`/data/config/notifications.yaml` IS the notifications config directly — it contains
|
||||||
|
`toggles:`, `rules:`, etc. at the root level with **no `notifications:` wrapper key**.
|
||||||
|
|
||||||
|
This differs from the monolithic layout shown in the plan examples, where notifications
|
||||||
|
config would be nested under a `notifications:` key in a larger config file.
|
||||||
|
|
||||||
|
**Impact on all phases:**
|
||||||
|
- Migration scripts must detect layout: if `"notifications" in config`, use nested
|
||||||
|
access; if `"toggles" in config or "rules" in config`, treat file root as the
|
||||||
|
notifications block.
|
||||||
|
- `load_notifications_config()` in `migrate_config_routing.py` implements this detection.
|
||||||
|
- GUI/API code reading `config.notifications.sinks` is unaffected — the `_dict_to_dataclass`
|
||||||
|
conversion handles both layouts identically at runtime.
|
||||||
|
- **All later phases** should follow this pattern when accessing the raw YAML for migration.
|
||||||
|
|
||||||
|
**Code pattern (from Phase A migration script):**
|
||||||
|
```python
|
||||||
|
if "notifications" in config:
|
||||||
|
notifications = config["notifications"]
|
||||||
|
elif "toggles" in config or "rules" in config:
|
||||||
|
notifications = config # Multi-file: file IS the notifications config
|
||||||
|
else:
|
||||||
|
notifications = {}
|
||||||
|
```
|
||||||
|
|
||||||
|
*Recorded: 2026-06-10, during Phase A implementation.*
|
||||||
Loading…
Add table
Add a link
Reference in a new issue