meshai/tests/test_central_region_routing.py

110 lines
4.9 KiB
Python
Raw Normal View History

feat(central): v0.5.4 -- region-aware subscriptions using Central v0.9.20 regional subjects Pre-v0.5.4 every Central subscription used a bare wildcard (central.wx.>, central.fire.>, central.traffic.>, central.quake.>, central.hydro.>, central.space.>), so a Magic Valley operator flipping nws -> central was in fact subscribing to the all-US firehose and discarding 95% of events locally. Central v0.9.20 (2026-05-28) added per-region subject suffixes so the firehose can be filtered server-side. This wires meshai to use them. Backend (meshai/central/consumer.py): - New _subjects_for(adapter, region) replaces the static ADAPTER_SUBJECTS dict. ADAPTER_SUBJECTS is retained as an alias to _SUBJECTS_BARE for any legacy importers; the dispatcher path is unchanged. - Per-adapter subject patterns (region='us.id' default): nws -> central.wx.alert.us.id.> (region BEFORE wildcard) usgs_quake -> central.quake.event.>.us.id (region AFTER wildcard) firms -> central.fire.hotspot.>.us.id fires -> central.fire.incident.id.> (state token at fixed depth) central.fire.perimeter.id.> traffic -> central.traffic.>.id (bare state, no us. prefix) roads511 -> central.traffic.>.id (shared with traffic, sub-adapter routing) usgs -> central.hydro.>.us.id central.hydro.>.unknown (workaround until v0.9.20.1) swpc -> central.space.> (planetary; region ignored) - Empty/None region falls back to bare wildcards (pre-v0.9.20 behaviour). - _subject_owned() pulls region from env.central.region and routes through _subjects_for; v0.5.3 sub-adapter routing (owned-sources set) still applies on shared subjects like central.traffic.>.id. - start() logs the active region at connect-time for ops visibility. Config (meshai/config.py): - CentralConsumerConfig.region: str = "us.id". One region per consumer applies to every central-flipped adapter; per-adapter overrides can land in v0.6 when there is a real use case. Frontend (dashboard-frontend/src/pages/Environment.tsx): - Central Connection panel gets a Region text input next to URL/Durable. - EnvConfig.central type extended with region: string. - Static bundle rebuilt; index-DCFmSeOM.js -> index-B24tHcYj.js. Tests: - tests/test_central_region_routing.py (new, 9 cases): asserts the exact v0.9.20 subject string for each adapter at region='us.id', the SWPC global-stays-global rule, the USGS .unknown workaround, the empty-region backward-compat fallback for all 8 adapters, and integration through CentralConsumer._subject_owned() with the default region. - tests/test_central_consumer.py + tests/test_central_sub_adapter_routing.py: the two tests that asserted bare-wildcard subjects now set env.central.region = "" explicitly to preserve their original concern (no region semantics — backward-compat path only). Why swpc stays global: space weather is planetary -- a CME is detected on the sun, the geomagnetic response is hemispheric. There is no Idaho-only solar event; subscribing per-region would only drop events we want. Why hydro has the .unknown workaround: Central v0.9.20 leaves gauges whose USGS state can't be inferred on central.hydro.>.unknown. Until v0.9.20.1 backfills the state tag we subscribe to both filters to avoid silently losing those rows. Idaho downstream-filtering on data['_enriched']['usgs_site']['state'] is future v0.6 work. Orthogonal to v0.5.2 dispatcher guards (staleness / cooldown / dedup) and v0.5.3 sub-adapter routing: the region filter operates at the NATS subscription layer (server-side), upstream of everything else. Verified: pytest 327 passed (318 prior + 9 new region-routing tests); py_compile clean; frontend build clean. Safe-mode preserved -- no toggle enabled, no master enabled, no central enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 02:30:33 +00:00
"""v0.5.4: Central v0.9.20 region-aware subject building.
Exercises `_subjects_for(adapter, region)` and the wiring through
`CentralConsumer._subject_owned()`. The spec is hard-coded in the test
strings on purpose so a future drift in the v0.9.20 subject scheme
fails noisily here instead of silently shipping wrong filters.
"""
from meshai.central.consumer import (
CentralConsumer,
_subjects_for,
_SUBJECTS_BARE,
)
from meshai.config import EnvironmentalConfig
# --------------------------------------------------------------------- per-adapter
def test_subjects_for_nws_us_id():
"""NWS: region BEFORE wildcard (matches alert.<region>.<...>)."""
assert _subjects_for("nws", "us.id") == ["central.wx.alert.us.id.>"]
def test_subjects_for_usgs_quake_us_id():
"""USGS quake: region AFTER wildcard."""
assert _subjects_for("usgs_quake", "us.id") == ["central.quake.event.>.us.id"]
fix(fire): v0.5.7-fire -- FIRMS NATS pattern + WFIGS tombstone dedup + remove fire_proximity + categories audit Third family of the v0.5.7 NATS-and-categories campaign. Fire is the heaviest of the campaign -- four distinct fixes plus a category audit. Two of the four were broken in production: FIRMS subscribed to a syntactically invalid pattern, and WFIGS tombstones were silently dropped. FIX 1 -- FIRMS NATS pattern (the canonical bug). Pre-v0.5.7-fire `_subjects_for("firms","us.id")` returned `["central.fire.hotspot.>.us.id"]`, which is INVALID NATS (the `>` multi-level wildcard is only legal at the tail token). It also wouldn't have matched anything Central publishes: per the Central v0.10.0 consumer integration guide §firms, the actual published pattern is `central.fire.hotspot.<satellite>.<confidence>` (5 tokens, no us.<state> suffix). The two slots after "hotspot" are satellite name and confidence band -- NOT tile coordinates or region tokens. Note on prompt vs. guide discrepancy: the v0.5.7-fire task spec described a tile-coord/state pattern `central.fire.hotspot.*.*.us.id` (7 tokens with us.<state> tail). That's neither what Central v0.10.0 publishes nor what its guide documents. We follow the guide. Subscribing to the prompt's 7-token pattern would silently match zero messages in production (token-count mismatch). State filtering for FIRMS happens client-side via data.latitude / data.longitude against the configured region bbox. New subscription: `central.fire.hotspot.>` -- tail-only `>`, NATS-legal, matches all <satellite>.<confidence> combinations. FIX 2 -- WFIGS tombstone subjects. Per guide §wfigs_incidents and §wfigs_perimeters, WFIGS publishes: active: central.fire.incident.<state>.<county> (Convention A, depth-3 state) active: central.fire.perimeter.<state>.<county> tombstone: central.fire.incident.removed.<state> (5 tokens, "removed" at depth-3) tombstone: central.fire.perimeter.removed.<state> Pre-v0.5.7-fire `_subjects_for("fires","us.id")` subscribed only to the active subjects (`central.fire.incident.id.>` and `central.fire.perimeter.id.>`). The tombstone subjects have "removed" at depth-3 instead of the state token, so the active-subject `>` filters silently dropped EVERY tombstone. Fall-off signals never reached meshai's inhibitor, so old incidents stayed "live" in the pipeline indefinitely. Added the two tombstone subjects to the subscription list. Both are 5-token literals with no wildcards -- trivially NATS-legal. FIX 3 -- WFIGS tombstone dedup. Per guide §wfigs_incidents removal semantics, the tombstone env_id has the shape `<IrwinID>:removed:<iso_now>` -- the `:removed:` is sandwiched in the middle, with a timestamp tail. Pre-v0.5.7-fire the consumer.py group_key recovery was `re.sub(r":removed$", "", group_key)` -- a literal trailing `:removed` match -- which DID NOT FIRE on the WFIGS form (the regex required `:removed` at the very end of the string, but the WFIGS form has `:<iso>` after it). Consequence: WFIGS tombstones' group_key was the full `<IrwinID>:removed:<iso>` string instead of the bare `<IrwinID>`. The pipeline grouper/inhibitor never matched tombstones to their original incidents, so the lapse signal was lost. Fixed by switching the regex to `re.sub(r":removed(:.*)?$", "", group_key)` -- handles both the WFIGS `<IrwinID>:removed:<iso>` form AND the legacy GDACS `<id>:removed` form. The `is_tombstone` detection also gained an explicit `":removed:" in env_id` check for the WFIGS shape. Per the guide: "the same incident can have one or more removal tombstones over its lifecycle" (it can re-enter and re-fall-off). To preserve per-tombstone distinctness for downstream lifecycle accounting, the full env_id is stashed on `Event.data["_central_tombstone_id"]` (the group_key collapses to the IrwinID by design, but the original env_id with the :<iso> tail survives on data). FIX 4 -- ALERT_CATEGORIES fire-family audit + removed parametric entries. Per Matt's direct feedback ("fire near mesh has its own set of parameters that I don't even know what they could be. like how far is near mesh? I don't know I can't set that."), the parametric `fire_proximity` and the duplicate-named `wildfire_proximity` (both labeled "Fire Near Mesh" with parametric radius-based descriptions) were unselectable in the new Advanced Rules UI. Removed both. Cross-referenced what FIRMS and WFIGS actually emit (per the guide and the native adapter code) and audited the registry: Native emit: firms.py -> new_ignition (when adapter flags new_ignition) or wildfire_hotspot (otherwise) [v0.5.7-fire: was wildfire_proximity] fires.py -> wildfire_incident Central path emit (via map_category): fire.hotspot.* -> wildfire_hotspot fire.incident.* -> wildfire_incident fire.perimeter.* -> wildfire_incident (perimeters merge to the incident) fire.<other> -> wildfire_incident (catchall) Registry after v0.5.7-fire: {new_ignition, wildfire_hotspot, wildfire_incident} Parity confirmed. No orphans, no missing. Aligning firms.py to emit `wildfire_hotspot` (matching the central FIRMS map) means native + central FIRMS produce identical categories regardless of which feed path is enabled. Composer (`_CATEGORY_EMOJI`, `_CATEGORY_LABEL`) and router (three source-attribution tables) updated to drop the removed categories and add the new ones. Deferred to v0.5.8: distance_max_km field on rules for actual proximity filtering. Replaces the parametric fire_proximity registry entry with a parameterized rule field that the user CAN configure ("alert me about wildfire_incident within 30 km" instead of an opaque "Fire Near Mesh" toggle). Tests ----- PYTHONPATH=. pytest -q: 380 passed (was 366; +14 net). - tests/test_fire_v057.py (new): FIRMS subject is tail-only `>` with no mid-subject placement; WFIGS subjects cover active + four tombstones; WFIGS tombstone strips `:removed(:.*)?$` for group_key; two same-IrwinID tombstones both propagate through _handle and share group_key, with the original env_id preserved on data["_central_tombstone_id"]; legacy GDACS `:removed` shape still strips cleanly; fire_proximity / wildfire_proximity absent from ALERT_CATEGORIES; no "Fire Near Mesh" name duplicates; fire-family parity (native + central emit == registry); required-fields check on the three fire entries. - tests/test_central_region_routing.py: updated FIRMS test (tail-only `>`) and WFIGS test (includes tombstone subjects). - tests/test_pipeline_toggle_filter.py, tests/test_adapter_firms.py, tests/test_v052_dispatcher.py, tests/test_pipeline_digest.py: bulk-migrated obsolete category references (wildfire_proximity -> wildfire_hotspot, fire_proximity -> wildfire_incident) so the existing test suites continue to exercise the same routing/digest/dispatch paths with the new category names. Safe-mode preserved (master off, all family toggles off, all adapters native, central disabled). No live toggle flipped. Not tagging yet -- v0.5.7 tag waits until all families ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 06:25:42 +00:00
def test_subjects_for_firms_us_id_uses_tail_only_wildcard():
"""v0.5.7-fire: FIRMS publishes `central.fire.hotspot.<satellite>.<confidence>`
with NO region in the subject (per Central v0.10.0 guide §firms). The
pre-v0.5.7-fire `central.fire.hotspot.>.us.id` was syntactically invalid
(`>` mid-subject) AND wouldn't have matched anything Central actually
publishes. Region filtering for FIRMS now happens client-side via
data.latitude/longitude. Subscription uses tail-only `>` (NATS-legal)."""
assert _subjects_for("firms", "us.id") == ["central.fire.hotspot.>"]
feat(central): v0.5.4 -- region-aware subscriptions using Central v0.9.20 regional subjects Pre-v0.5.4 every Central subscription used a bare wildcard (central.wx.>, central.fire.>, central.traffic.>, central.quake.>, central.hydro.>, central.space.>), so a Magic Valley operator flipping nws -> central was in fact subscribing to the all-US firehose and discarding 95% of events locally. Central v0.9.20 (2026-05-28) added per-region subject suffixes so the firehose can be filtered server-side. This wires meshai to use them. Backend (meshai/central/consumer.py): - New _subjects_for(adapter, region) replaces the static ADAPTER_SUBJECTS dict. ADAPTER_SUBJECTS is retained as an alias to _SUBJECTS_BARE for any legacy importers; the dispatcher path is unchanged. - Per-adapter subject patterns (region='us.id' default): nws -> central.wx.alert.us.id.> (region BEFORE wildcard) usgs_quake -> central.quake.event.>.us.id (region AFTER wildcard) firms -> central.fire.hotspot.>.us.id fires -> central.fire.incident.id.> (state token at fixed depth) central.fire.perimeter.id.> traffic -> central.traffic.>.id (bare state, no us. prefix) roads511 -> central.traffic.>.id (shared with traffic, sub-adapter routing) usgs -> central.hydro.>.us.id central.hydro.>.unknown (workaround until v0.9.20.1) swpc -> central.space.> (planetary; region ignored) - Empty/None region falls back to bare wildcards (pre-v0.9.20 behaviour). - _subject_owned() pulls region from env.central.region and routes through _subjects_for; v0.5.3 sub-adapter routing (owned-sources set) still applies on shared subjects like central.traffic.>.id. - start() logs the active region at connect-time for ops visibility. Config (meshai/config.py): - CentralConsumerConfig.region: str = "us.id". One region per consumer applies to every central-flipped adapter; per-adapter overrides can land in v0.6 when there is a real use case. Frontend (dashboard-frontend/src/pages/Environment.tsx): - Central Connection panel gets a Region text input next to URL/Durable. - EnvConfig.central type extended with region: string. - Static bundle rebuilt; index-DCFmSeOM.js -> index-B24tHcYj.js. Tests: - tests/test_central_region_routing.py (new, 9 cases): asserts the exact v0.9.20 subject string for each adapter at region='us.id', the SWPC global-stays-global rule, the USGS .unknown workaround, the empty-region backward-compat fallback for all 8 adapters, and integration through CentralConsumer._subject_owned() with the default region. - tests/test_central_consumer.py + tests/test_central_sub_adapter_routing.py: the two tests that asserted bare-wildcard subjects now set env.central.region = "" explicitly to preserve their original concern (no region semantics — backward-compat path only). Why swpc stays global: space weather is planetary -- a CME is detected on the sun, the geomagnetic response is hemispheric. There is no Idaho-only solar event; subscribing per-region would only drop events we want. Why hydro has the .unknown workaround: Central v0.9.20 leaves gauges whose USGS state can't be inferred on central.hydro.>.unknown. Until v0.9.20.1 backfills the state tag we subscribe to both filters to avoid silently losing those rows. Idaho downstream-filtering on data['_enriched']['usgs_site']['state'] is future v0.6 work. Orthogonal to v0.5.2 dispatcher guards (staleness / cooldown / dedup) and v0.5.3 sub-adapter routing: the region filter operates at the NATS subscription layer (server-side), upstream of everything else. Verified: pytest 327 passed (318 prior + 9 new region-routing tests); py_compile clean; frontend build clean. Safe-mode preserved -- no toggle enabled, no master enabled, no central enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 02:30:33 +00:00
fix(fire): v0.5.7-fire -- FIRMS NATS pattern + WFIGS tombstone dedup + remove fire_proximity + categories audit Third family of the v0.5.7 NATS-and-categories campaign. Fire is the heaviest of the campaign -- four distinct fixes plus a category audit. Two of the four were broken in production: FIRMS subscribed to a syntactically invalid pattern, and WFIGS tombstones were silently dropped. FIX 1 -- FIRMS NATS pattern (the canonical bug). Pre-v0.5.7-fire `_subjects_for("firms","us.id")` returned `["central.fire.hotspot.>.us.id"]`, which is INVALID NATS (the `>` multi-level wildcard is only legal at the tail token). It also wouldn't have matched anything Central publishes: per the Central v0.10.0 consumer integration guide §firms, the actual published pattern is `central.fire.hotspot.<satellite>.<confidence>` (5 tokens, no us.<state> suffix). The two slots after "hotspot" are satellite name and confidence band -- NOT tile coordinates or region tokens. Note on prompt vs. guide discrepancy: the v0.5.7-fire task spec described a tile-coord/state pattern `central.fire.hotspot.*.*.us.id` (7 tokens with us.<state> tail). That's neither what Central v0.10.0 publishes nor what its guide documents. We follow the guide. Subscribing to the prompt's 7-token pattern would silently match zero messages in production (token-count mismatch). State filtering for FIRMS happens client-side via data.latitude / data.longitude against the configured region bbox. New subscription: `central.fire.hotspot.>` -- tail-only `>`, NATS-legal, matches all <satellite>.<confidence> combinations. FIX 2 -- WFIGS tombstone subjects. Per guide §wfigs_incidents and §wfigs_perimeters, WFIGS publishes: active: central.fire.incident.<state>.<county> (Convention A, depth-3 state) active: central.fire.perimeter.<state>.<county> tombstone: central.fire.incident.removed.<state> (5 tokens, "removed" at depth-3) tombstone: central.fire.perimeter.removed.<state> Pre-v0.5.7-fire `_subjects_for("fires","us.id")` subscribed only to the active subjects (`central.fire.incident.id.>` and `central.fire.perimeter.id.>`). The tombstone subjects have "removed" at depth-3 instead of the state token, so the active-subject `>` filters silently dropped EVERY tombstone. Fall-off signals never reached meshai's inhibitor, so old incidents stayed "live" in the pipeline indefinitely. Added the two tombstone subjects to the subscription list. Both are 5-token literals with no wildcards -- trivially NATS-legal. FIX 3 -- WFIGS tombstone dedup. Per guide §wfigs_incidents removal semantics, the tombstone env_id has the shape `<IrwinID>:removed:<iso_now>` -- the `:removed:` is sandwiched in the middle, with a timestamp tail. Pre-v0.5.7-fire the consumer.py group_key recovery was `re.sub(r":removed$", "", group_key)` -- a literal trailing `:removed` match -- which DID NOT FIRE on the WFIGS form (the regex required `:removed` at the very end of the string, but the WFIGS form has `:<iso>` after it). Consequence: WFIGS tombstones' group_key was the full `<IrwinID>:removed:<iso>` string instead of the bare `<IrwinID>`. The pipeline grouper/inhibitor never matched tombstones to their original incidents, so the lapse signal was lost. Fixed by switching the regex to `re.sub(r":removed(:.*)?$", "", group_key)` -- handles both the WFIGS `<IrwinID>:removed:<iso>` form AND the legacy GDACS `<id>:removed` form. The `is_tombstone` detection also gained an explicit `":removed:" in env_id` check for the WFIGS shape. Per the guide: "the same incident can have one or more removal tombstones over its lifecycle" (it can re-enter and re-fall-off). To preserve per-tombstone distinctness for downstream lifecycle accounting, the full env_id is stashed on `Event.data["_central_tombstone_id"]` (the group_key collapses to the IrwinID by design, but the original env_id with the :<iso> tail survives on data). FIX 4 -- ALERT_CATEGORIES fire-family audit + removed parametric entries. Per Matt's direct feedback ("fire near mesh has its own set of parameters that I don't even know what they could be. like how far is near mesh? I don't know I can't set that."), the parametric `fire_proximity` and the duplicate-named `wildfire_proximity` (both labeled "Fire Near Mesh" with parametric radius-based descriptions) were unselectable in the new Advanced Rules UI. Removed both. Cross-referenced what FIRMS and WFIGS actually emit (per the guide and the native adapter code) and audited the registry: Native emit: firms.py -> new_ignition (when adapter flags new_ignition) or wildfire_hotspot (otherwise) [v0.5.7-fire: was wildfire_proximity] fires.py -> wildfire_incident Central path emit (via map_category): fire.hotspot.* -> wildfire_hotspot fire.incident.* -> wildfire_incident fire.perimeter.* -> wildfire_incident (perimeters merge to the incident) fire.<other> -> wildfire_incident (catchall) Registry after v0.5.7-fire: {new_ignition, wildfire_hotspot, wildfire_incident} Parity confirmed. No orphans, no missing. Aligning firms.py to emit `wildfire_hotspot` (matching the central FIRMS map) means native + central FIRMS produce identical categories regardless of which feed path is enabled. Composer (`_CATEGORY_EMOJI`, `_CATEGORY_LABEL`) and router (three source-attribution tables) updated to drop the removed categories and add the new ones. Deferred to v0.5.8: distance_max_km field on rules for actual proximity filtering. Replaces the parametric fire_proximity registry entry with a parameterized rule field that the user CAN configure ("alert me about wildfire_incident within 30 km" instead of an opaque "Fire Near Mesh" toggle). Tests ----- PYTHONPATH=. pytest -q: 380 passed (was 366; +14 net). - tests/test_fire_v057.py (new): FIRMS subject is tail-only `>` with no mid-subject placement; WFIGS subjects cover active + four tombstones; WFIGS tombstone strips `:removed(:.*)?$` for group_key; two same-IrwinID tombstones both propagate through _handle and share group_key, with the original env_id preserved on data["_central_tombstone_id"]; legacy GDACS `:removed` shape still strips cleanly; fire_proximity / wildfire_proximity absent from ALERT_CATEGORIES; no "Fire Near Mesh" name duplicates; fire-family parity (native + central emit == registry); required-fields check on the three fire entries. - tests/test_central_region_routing.py: updated FIRMS test (tail-only `>`) and WFIGS test (includes tombstone subjects). - tests/test_pipeline_toggle_filter.py, tests/test_adapter_firms.py, tests/test_v052_dispatcher.py, tests/test_pipeline_digest.py: bulk-migrated obsolete category references (wildfire_proximity -> wildfire_hotspot, fire_proximity -> wildfire_incident) so the existing test suites continue to exercise the same routing/digest/dispatch paths with the new category names. Safe-mode preserved (master off, all family toggles off, all adapters native, central disabled). No live toggle flipped. Not tagging yet -- v0.5.7 tag waits until all families ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 06:25:42 +00:00
def test_subjects_for_fires_us_id_includes_tombstones():
"""v0.5.7-fire: WFIGS subjects -- active state-token at depth-3 + the
removal-tombstone subjects (`central.fire.{incident,perimeter}.removed.<state>`)
per Central v0.10.0 guide §wfigs_incidents §wfigs_perimeters. Pre-v0.5.7-fire
we only subscribed to active subjects, silently dropping fall-off signals."""
feat(central): v0.5.4 -- region-aware subscriptions using Central v0.9.20 regional subjects Pre-v0.5.4 every Central subscription used a bare wildcard (central.wx.>, central.fire.>, central.traffic.>, central.quake.>, central.hydro.>, central.space.>), so a Magic Valley operator flipping nws -> central was in fact subscribing to the all-US firehose and discarding 95% of events locally. Central v0.9.20 (2026-05-28) added per-region subject suffixes so the firehose can be filtered server-side. This wires meshai to use them. Backend (meshai/central/consumer.py): - New _subjects_for(adapter, region) replaces the static ADAPTER_SUBJECTS dict. ADAPTER_SUBJECTS is retained as an alias to _SUBJECTS_BARE for any legacy importers; the dispatcher path is unchanged. - Per-adapter subject patterns (region='us.id' default): nws -> central.wx.alert.us.id.> (region BEFORE wildcard) usgs_quake -> central.quake.event.>.us.id (region AFTER wildcard) firms -> central.fire.hotspot.>.us.id fires -> central.fire.incident.id.> (state token at fixed depth) central.fire.perimeter.id.> traffic -> central.traffic.>.id (bare state, no us. prefix) roads511 -> central.traffic.>.id (shared with traffic, sub-adapter routing) usgs -> central.hydro.>.us.id central.hydro.>.unknown (workaround until v0.9.20.1) swpc -> central.space.> (planetary; region ignored) - Empty/None region falls back to bare wildcards (pre-v0.9.20 behaviour). - _subject_owned() pulls region from env.central.region and routes through _subjects_for; v0.5.3 sub-adapter routing (owned-sources set) still applies on shared subjects like central.traffic.>.id. - start() logs the active region at connect-time for ops visibility. Config (meshai/config.py): - CentralConsumerConfig.region: str = "us.id". One region per consumer applies to every central-flipped adapter; per-adapter overrides can land in v0.6 when there is a real use case. Frontend (dashboard-frontend/src/pages/Environment.tsx): - Central Connection panel gets a Region text input next to URL/Durable. - EnvConfig.central type extended with region: string. - Static bundle rebuilt; index-DCFmSeOM.js -> index-B24tHcYj.js. Tests: - tests/test_central_region_routing.py (new, 9 cases): asserts the exact v0.9.20 subject string for each adapter at region='us.id', the SWPC global-stays-global rule, the USGS .unknown workaround, the empty-region backward-compat fallback for all 8 adapters, and integration through CentralConsumer._subject_owned() with the default region. - tests/test_central_consumer.py + tests/test_central_sub_adapter_routing.py: the two tests that asserted bare-wildcard subjects now set env.central.region = "" explicitly to preserve their original concern (no region semantics — backward-compat path only). Why swpc stays global: space weather is planetary -- a CME is detected on the sun, the geomagnetic response is hemispheric. There is no Idaho-only solar event; subscribing per-region would only drop events we want. Why hydro has the .unknown workaround: Central v0.9.20 leaves gauges whose USGS state can't be inferred on central.hydro.>.unknown. Until v0.9.20.1 backfills the state tag we subscribe to both filters to avoid silently losing those rows. Idaho downstream-filtering on data['_enriched']['usgs_site']['state'] is future v0.6 work. Orthogonal to v0.5.2 dispatcher guards (staleness / cooldown / dedup) and v0.5.3 sub-adapter routing: the region filter operates at the NATS subscription layer (server-side), upstream of everything else. Verified: pytest 327 passed (318 prior + 9 new region-routing tests); py_compile clean; frontend build clean. Safe-mode preserved -- no toggle enabled, no master enabled, no central enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 02:30:33 +00:00
assert _subjects_for("fires", "us.id") == [
"central.fire.incident.id.>",
"central.fire.perimeter.id.>",
fix(fire): v0.5.7-fire -- FIRMS NATS pattern + WFIGS tombstone dedup + remove fire_proximity + categories audit Third family of the v0.5.7 NATS-and-categories campaign. Fire is the heaviest of the campaign -- four distinct fixes plus a category audit. Two of the four were broken in production: FIRMS subscribed to a syntactically invalid pattern, and WFIGS tombstones were silently dropped. FIX 1 -- FIRMS NATS pattern (the canonical bug). Pre-v0.5.7-fire `_subjects_for("firms","us.id")` returned `["central.fire.hotspot.>.us.id"]`, which is INVALID NATS (the `>` multi-level wildcard is only legal at the tail token). It also wouldn't have matched anything Central publishes: per the Central v0.10.0 consumer integration guide §firms, the actual published pattern is `central.fire.hotspot.<satellite>.<confidence>` (5 tokens, no us.<state> suffix). The two slots after "hotspot" are satellite name and confidence band -- NOT tile coordinates or region tokens. Note on prompt vs. guide discrepancy: the v0.5.7-fire task spec described a tile-coord/state pattern `central.fire.hotspot.*.*.us.id` (7 tokens with us.<state> tail). That's neither what Central v0.10.0 publishes nor what its guide documents. We follow the guide. Subscribing to the prompt's 7-token pattern would silently match zero messages in production (token-count mismatch). State filtering for FIRMS happens client-side via data.latitude / data.longitude against the configured region bbox. New subscription: `central.fire.hotspot.>` -- tail-only `>`, NATS-legal, matches all <satellite>.<confidence> combinations. FIX 2 -- WFIGS tombstone subjects. Per guide §wfigs_incidents and §wfigs_perimeters, WFIGS publishes: active: central.fire.incident.<state>.<county> (Convention A, depth-3 state) active: central.fire.perimeter.<state>.<county> tombstone: central.fire.incident.removed.<state> (5 tokens, "removed" at depth-3) tombstone: central.fire.perimeter.removed.<state> Pre-v0.5.7-fire `_subjects_for("fires","us.id")` subscribed only to the active subjects (`central.fire.incident.id.>` and `central.fire.perimeter.id.>`). The tombstone subjects have "removed" at depth-3 instead of the state token, so the active-subject `>` filters silently dropped EVERY tombstone. Fall-off signals never reached meshai's inhibitor, so old incidents stayed "live" in the pipeline indefinitely. Added the two tombstone subjects to the subscription list. Both are 5-token literals with no wildcards -- trivially NATS-legal. FIX 3 -- WFIGS tombstone dedup. Per guide §wfigs_incidents removal semantics, the tombstone env_id has the shape `<IrwinID>:removed:<iso_now>` -- the `:removed:` is sandwiched in the middle, with a timestamp tail. Pre-v0.5.7-fire the consumer.py group_key recovery was `re.sub(r":removed$", "", group_key)` -- a literal trailing `:removed` match -- which DID NOT FIRE on the WFIGS form (the regex required `:removed` at the very end of the string, but the WFIGS form has `:<iso>` after it). Consequence: WFIGS tombstones' group_key was the full `<IrwinID>:removed:<iso>` string instead of the bare `<IrwinID>`. The pipeline grouper/inhibitor never matched tombstones to their original incidents, so the lapse signal was lost. Fixed by switching the regex to `re.sub(r":removed(:.*)?$", "", group_key)` -- handles both the WFIGS `<IrwinID>:removed:<iso>` form AND the legacy GDACS `<id>:removed` form. The `is_tombstone` detection also gained an explicit `":removed:" in env_id` check for the WFIGS shape. Per the guide: "the same incident can have one or more removal tombstones over its lifecycle" (it can re-enter and re-fall-off). To preserve per-tombstone distinctness for downstream lifecycle accounting, the full env_id is stashed on `Event.data["_central_tombstone_id"]` (the group_key collapses to the IrwinID by design, but the original env_id with the :<iso> tail survives on data). FIX 4 -- ALERT_CATEGORIES fire-family audit + removed parametric entries. Per Matt's direct feedback ("fire near mesh has its own set of parameters that I don't even know what they could be. like how far is near mesh? I don't know I can't set that."), the parametric `fire_proximity` and the duplicate-named `wildfire_proximity` (both labeled "Fire Near Mesh" with parametric radius-based descriptions) were unselectable in the new Advanced Rules UI. Removed both. Cross-referenced what FIRMS and WFIGS actually emit (per the guide and the native adapter code) and audited the registry: Native emit: firms.py -> new_ignition (when adapter flags new_ignition) or wildfire_hotspot (otherwise) [v0.5.7-fire: was wildfire_proximity] fires.py -> wildfire_incident Central path emit (via map_category): fire.hotspot.* -> wildfire_hotspot fire.incident.* -> wildfire_incident fire.perimeter.* -> wildfire_incident (perimeters merge to the incident) fire.<other> -> wildfire_incident (catchall) Registry after v0.5.7-fire: {new_ignition, wildfire_hotspot, wildfire_incident} Parity confirmed. No orphans, no missing. Aligning firms.py to emit `wildfire_hotspot` (matching the central FIRMS map) means native + central FIRMS produce identical categories regardless of which feed path is enabled. Composer (`_CATEGORY_EMOJI`, `_CATEGORY_LABEL`) and router (three source-attribution tables) updated to drop the removed categories and add the new ones. Deferred to v0.5.8: distance_max_km field on rules for actual proximity filtering. Replaces the parametric fire_proximity registry entry with a parameterized rule field that the user CAN configure ("alert me about wildfire_incident within 30 km" instead of an opaque "Fire Near Mesh" toggle). Tests ----- PYTHONPATH=. pytest -q: 380 passed (was 366; +14 net). - tests/test_fire_v057.py (new): FIRMS subject is tail-only `>` with no mid-subject placement; WFIGS subjects cover active + four tombstones; WFIGS tombstone strips `:removed(:.*)?$` for group_key; two same-IrwinID tombstones both propagate through _handle and share group_key, with the original env_id preserved on data["_central_tombstone_id"]; legacy GDACS `:removed` shape still strips cleanly; fire_proximity / wildfire_proximity absent from ALERT_CATEGORIES; no "Fire Near Mesh" name duplicates; fire-family parity (native + central emit == registry); required-fields check on the three fire entries. - tests/test_central_region_routing.py: updated FIRMS test (tail-only `>`) and WFIGS test (includes tombstone subjects). - tests/test_pipeline_toggle_filter.py, tests/test_adapter_firms.py, tests/test_v052_dispatcher.py, tests/test_pipeline_digest.py: bulk-migrated obsolete category references (wildfire_proximity -> wildfire_hotspot, fire_proximity -> wildfire_incident) so the existing test suites continue to exercise the same routing/digest/dispatch paths with the new category names. Safe-mode preserved (master off, all family toggles off, all adapters native, central disabled). No live toggle flipped. Not tagging yet -- v0.5.7 tag waits until all families ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 06:25:42 +00:00
"central.fire.incident.removed.id",
"central.fire.perimeter.removed.id",
feat(central): v0.5.4 -- region-aware subscriptions using Central v0.9.20 regional subjects Pre-v0.5.4 every Central subscription used a bare wildcard (central.wx.>, central.fire.>, central.traffic.>, central.quake.>, central.hydro.>, central.space.>), so a Magic Valley operator flipping nws -> central was in fact subscribing to the all-US firehose and discarding 95% of events locally. Central v0.9.20 (2026-05-28) added per-region subject suffixes so the firehose can be filtered server-side. This wires meshai to use them. Backend (meshai/central/consumer.py): - New _subjects_for(adapter, region) replaces the static ADAPTER_SUBJECTS dict. ADAPTER_SUBJECTS is retained as an alias to _SUBJECTS_BARE for any legacy importers; the dispatcher path is unchanged. - Per-adapter subject patterns (region='us.id' default): nws -> central.wx.alert.us.id.> (region BEFORE wildcard) usgs_quake -> central.quake.event.>.us.id (region AFTER wildcard) firms -> central.fire.hotspot.>.us.id fires -> central.fire.incident.id.> (state token at fixed depth) central.fire.perimeter.id.> traffic -> central.traffic.>.id (bare state, no us. prefix) roads511 -> central.traffic.>.id (shared with traffic, sub-adapter routing) usgs -> central.hydro.>.us.id central.hydro.>.unknown (workaround until v0.9.20.1) swpc -> central.space.> (planetary; region ignored) - Empty/None region falls back to bare wildcards (pre-v0.9.20 behaviour). - _subject_owned() pulls region from env.central.region and routes through _subjects_for; v0.5.3 sub-adapter routing (owned-sources set) still applies on shared subjects like central.traffic.>.id. - start() logs the active region at connect-time for ops visibility. Config (meshai/config.py): - CentralConsumerConfig.region: str = "us.id". One region per consumer applies to every central-flipped adapter; per-adapter overrides can land in v0.6 when there is a real use case. Frontend (dashboard-frontend/src/pages/Environment.tsx): - Central Connection panel gets a Region text input next to URL/Durable. - EnvConfig.central type extended with region: string. - Static bundle rebuilt; index-DCFmSeOM.js -> index-B24tHcYj.js. Tests: - tests/test_central_region_routing.py (new, 9 cases): asserts the exact v0.9.20 subject string for each adapter at region='us.id', the SWPC global-stays-global rule, the USGS .unknown workaround, the empty-region backward-compat fallback for all 8 adapters, and integration through CentralConsumer._subject_owned() with the default region. - tests/test_central_consumer.py + tests/test_central_sub_adapter_routing.py: the two tests that asserted bare-wildcard subjects now set env.central.region = "" explicitly to preserve their original concern (no region semantics — backward-compat path only). Why swpc stays global: space weather is planetary -- a CME is detected on the sun, the geomagnetic response is hemispheric. There is no Idaho-only solar event; subscribing per-region would only drop events we want. Why hydro has the .unknown workaround: Central v0.9.20 leaves gauges whose USGS state can't be inferred on central.hydro.>.unknown. Until v0.9.20.1 backfills the state tag we subscribe to both filters to avoid silently losing those rows. Idaho downstream-filtering on data['_enriched']['usgs_site']['state'] is future v0.6 work. Orthogonal to v0.5.2 dispatcher guards (staleness / cooldown / dedup) and v0.5.3 sub-adapter routing: the region filter operates at the NATS subscription layer (server-side), upstream of everything else. Verified: pytest 327 passed (318 prior + 9 new region-routing tests); py_compile clean; frontend build clean. Safe-mode preserved -- no toggle enabled, no master enabled, no central enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 02:30:33 +00:00
]
fix(traffic): v0.5.7-traffic -- NATS pattern fix + itd_511 sub-adapter routing + categories audit Second family of the v0.5.7 NATS-and-categories campaign. Weather went first because its NWS pattern was already legal; traffic was carrying invalid NATS syntax in production. FIX 1 -- Invalid `>` mid-subject in traffic. Pre-v0.5.7-traffic the subject builder shipped `central.traffic.>.{state}` for both the traffic and roads511 adapters. NATS rules say `>` is only legal at the tail token; mid-subject `>` is rejected by the broker at subscribe time (or silently delivers nothing depending on server version). Replaced with Convention B (per Central v0.10.0 meshai_integration_guide.md): single-token `*` in the event_type slot, bare state suffix -- `central.traffic.*.id` for Idaho. Shared by the wzdx, tomtom_incidents and state_511_atis adapters. FIX 2 -- roads511 dual subscribe. The new Idaho-only itd_511 adapter in Central v0.10.0 uses Convention A (`central.traffic.<event_type>.us.<state>`, the us.<state> form). Convention B (bare state) is shared with the rest of the traffic family. roads511 now owns BOTH: central.traffic.*.id (Convention B, shared with traffic via _subject_owned) central.traffic.*.us.id (Convention A, itd_511-only) Sub-adapter routing in CentralConsumer._subject_owned (v0.5.1) already keeps shared subjects scoped to the right meshai source -- no change needed. FIX 3 -- itd_511 -> roads511 in CENTRAL_ADAPTER_TO_SOURCE. Mirrors state_511_atis (added v0.5.3). Both Idaho 511 feeds collapse to a single meshai source for UX simplicity; future v0.6 may split them if Matt needs differential rules. FIX 4 -- Roads-family categories audit + finer event_type mapping. Pre-v0.5.7-traffic the central path flattened every traffic-domain event to `traffic_congestion` because work_zone / incident / closure had no entries in _CATEGORY_MAP and fell through to the `traffic.` catchall (then the subject-domain fallback). Added three explicit map entries before the catchall: ("work_zone", "work_zone") # catches "work_zone" and "work_zone.wzdx" ("incident", "road_incident") # catches incident.tomtom_incidents + bare ("closure", "road_closure") # catches closure + closure.itd_511 ALERT_CATEGORIES gains two new roads-family entries so the Advanced Rules editor can target them: work_zone -- Active construction/maintenance work zone road_incident -- Reported incident (crash, hazard, debris) Existing entries `road_closure` and `traffic_congestion` kept. composer._CATEGORY_EMOJI gains matching glyphs (🚧 work_zone, 🚨 road_incident) so the live LoRa rendering lines up with the category example_message glyphs. Audit cross-check (test_alert_categories_roads_complete enforces parity): Native emit: traffic.py -> traffic_congestion; roads511.py -> road_closure Central path emit (via map_category): {road_closure, traffic_congestion, work_zone, road_incident} ALERT_CATEGORIES{toggle=roads}: {road_closure, traffic_congestion, work_zone, road_incident} Parity. No orphans, no missing. DEFERRED to v0.5.8: itd_511_cameras / traffic_cameras stream lives at a different subject domain (central.traffic_cameras.>) and needs a new meshai source (roads_cameras or similar). Out of scope for v0.5.7. Tests ----- PYTHONPATH=. pytest -q: 366 passed (was 345; +21 net). - tests/test_traffic_v057.py (new): NATS-syntax checks (`>` only at tail, single-token `*`), traffic Convention B, roads511 dual-subscribe, shared bare-state subject, itd_511 + state_511_atis remap, map_category event_type preservation, ALERT_CATEGORIES roads parity (reflection-based scan of native emit + central path), required-fields check on the four roads entries. - tests/test_central_region_routing.py: updated `test_subjects_for_traffic_and_roads511_share_state_token` -> two new tests covering Convention B (traffic) and dual-subscribe (roads511). - tests/test_central_consumer.py: updated `test_subject_domain_fallback_for_unmapped_category` (work_zone.wzdx is now mapped, switched to a genuinely-unmapped category) + new `test_v057_traffic_work_zone_now_mapped` asserting wzdx envelopes land on ev.category=="work_zone". Safe-mode preserved (master off, all family toggles off, all adapters native, central disabled). No live toggle flipped. Not tagging yet -- v0.5.7 tag waits until all families ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 06:10:12 +00:00
def test_subjects_for_traffic_uses_convention_b():
"""v0.5.7-traffic: traffic adapter -> bare-state Convention B with `*`
in the event_type slot. Pre-v0.5.7-traffic this was `>.{state}` which
is invalid NATS (`>` must be at the tail). The bare-state subject is
shared with roads511 (sub-adapter routing picks the right meshai source)."""
assert _subjects_for("traffic", "us.id") == ["central.traffic.*.id"]
def test_subjects_for_roads511_dual_subscribes_convention_a_and_b():
"""v0.5.7-traffic: roads511 owns BOTH the shared bare-state subject
(Convention B, shared with traffic) AND the us.<state> subject
(Convention A) where the new Idaho-only itd_511 adapter publishes."""
assert _subjects_for("roads511", "us.id") == [
"central.traffic.*.id",
"central.traffic.*.us.id",
]
feat(central): v0.5.4 -- region-aware subscriptions using Central v0.9.20 regional subjects Pre-v0.5.4 every Central subscription used a bare wildcard (central.wx.>, central.fire.>, central.traffic.>, central.quake.>, central.hydro.>, central.space.>), so a Magic Valley operator flipping nws -> central was in fact subscribing to the all-US firehose and discarding 95% of events locally. Central v0.9.20 (2026-05-28) added per-region subject suffixes so the firehose can be filtered server-side. This wires meshai to use them. Backend (meshai/central/consumer.py): - New _subjects_for(adapter, region) replaces the static ADAPTER_SUBJECTS dict. ADAPTER_SUBJECTS is retained as an alias to _SUBJECTS_BARE for any legacy importers; the dispatcher path is unchanged. - Per-adapter subject patterns (region='us.id' default): nws -> central.wx.alert.us.id.> (region BEFORE wildcard) usgs_quake -> central.quake.event.>.us.id (region AFTER wildcard) firms -> central.fire.hotspot.>.us.id fires -> central.fire.incident.id.> (state token at fixed depth) central.fire.perimeter.id.> traffic -> central.traffic.>.id (bare state, no us. prefix) roads511 -> central.traffic.>.id (shared with traffic, sub-adapter routing) usgs -> central.hydro.>.us.id central.hydro.>.unknown (workaround until v0.9.20.1) swpc -> central.space.> (planetary; region ignored) - Empty/None region falls back to bare wildcards (pre-v0.9.20 behaviour). - _subject_owned() pulls region from env.central.region and routes through _subjects_for; v0.5.3 sub-adapter routing (owned-sources set) still applies on shared subjects like central.traffic.>.id. - start() logs the active region at connect-time for ops visibility. Config (meshai/config.py): - CentralConsumerConfig.region: str = "us.id". One region per consumer applies to every central-flipped adapter; per-adapter overrides can land in v0.6 when there is a real use case. Frontend (dashboard-frontend/src/pages/Environment.tsx): - Central Connection panel gets a Region text input next to URL/Durable. - EnvConfig.central type extended with region: string. - Static bundle rebuilt; index-DCFmSeOM.js -> index-B24tHcYj.js. Tests: - tests/test_central_region_routing.py (new, 9 cases): asserts the exact v0.9.20 subject string for each adapter at region='us.id', the SWPC global-stays-global rule, the USGS .unknown workaround, the empty-region backward-compat fallback for all 8 adapters, and integration through CentralConsumer._subject_owned() with the default region. - tests/test_central_consumer.py + tests/test_central_sub_adapter_routing.py: the two tests that asserted bare-wildcard subjects now set env.central.region = "" explicitly to preserve their original concern (no region semantics — backward-compat path only). Why swpc stays global: space weather is planetary -- a CME is detected on the sun, the geomagnetic response is hemispheric. There is no Idaho-only solar event; subscribing per-region would only drop events we want. Why hydro has the .unknown workaround: Central v0.9.20 leaves gauges whose USGS state can't be inferred on central.hydro.>.unknown. Until v0.9.20.1 backfills the state tag we subscribe to both filters to avoid silently losing those rows. Idaho downstream-filtering on data['_enriched']['usgs_site']['state'] is future v0.6 work. Orthogonal to v0.5.2 dispatcher guards (staleness / cooldown / dedup) and v0.5.3 sub-adapter routing: the region filter operates at the NATS subscription layer (server-side), upstream of everything else. Verified: pytest 327 passed (318 prior + 9 new region-routing tests); py_compile clean; frontend build clean. Safe-mode preserved -- no toggle enabled, no master enabled, no central enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 02:30:33 +00:00
def test_subjects_for_usgs_includes_unknown_workaround():
"""USGS hydro: subscribes to BOTH the region-tagged filter and the
".unknown" filter to cover gauges whose state Central v0.9.20 can't
infer yet (workaround until v0.9.20.1 backfills the tag)."""
assert _subjects_for("usgs", "us.id") == [
"central.hydro.>.us.id",
"central.hydro.>.unknown",
]
def test_subjects_for_swpc_stays_global():
"""SWPC: space weather is planetary; region argument is ignored."""
assert _subjects_for("swpc", "us.id") == ["central.space.>"]
assert _subjects_for("swpc", "us.mt") == ["central.space.>"] # same regardless
assert _subjects_for("swpc", "") == ["central.space.>"]
# --------------------------------------------------------------------- backward compat
def test_subjects_for_empty_region_falls_back_to_bare_wildcards():
"""Empty/None region = pre-v0.9.20 behaviour for every adapter, byte-identical
to the legacy _SUBJECTS_BARE map. Adapters absent from the map return []."""
for adapter, expected in _SUBJECTS_BARE.items():
assert _subjects_for(adapter, "") == expected, f"empty region mismatch for {adapter}"
assert _subjects_for(adapter, None) == expected, f"None region mismatch for {adapter}"
# Unknown adapters return empty regardless of region.
assert _subjects_for("ducting", "us.id") == []
assert _subjects_for("avalanche", "") == []
# --------------------------------------------------------------------- integration
def test_central_region_default_propagates_to_consumer_subjects():
"""Default region = 'us.id': flipping nws to central → consumer subscribes
to the region-aware subject, not the bare wildcard."""
env = EnvironmentalConfig()
assert env.central.region == "us.id" # spec default
env.nws.feed_source = "central"
so = CentralConsumer(env, None)._subject_owned()
assert list(so.keys()) == ["central.wx.alert.us.id.>"]
assert so["central.wx.alert.us.id.>"] == {"nws"}