meshai/tests/test_config_source_field.py

50 lines
1.7 KiB
Python
Raw Normal View History

feat(central): v0.4 C.1 Central connector backend (no-op until adapter source flipped) Adds the backend for sourcing environmental feeds from Central's NATS JetStream firehose instead of (or alongside) meshai's native adapters. Architecture is Matt-approved Option 3' (dedicated package + per-adapter source switch surfaced on the existing Environmental config). NO-OP POSTURE (intentional): every adapter defaults to feed_source="native" and environmental.central.enabled defaults false, so on a stock config the CentralConsumer starts and subscribes to nothing -- behavior is byte-for-byte v0.3. Live env_feeds.yaml is unchanged on disk; an operator who touches nothing sees no change. Flipping an adapter to central is Phase C.3; the dashboard UI for it is Phase C.2. What landed: - meshai/central/ package (CentralConsumer): async start()/stop(), JetStream durable subscribe to subjects derived from adapters with feed_source=central, and _on_message -> normalize -> bus.emit. nats-py is lazy-imported only on the connect path, so no-op boot has zero NATS dependency. - Normalization (CloudEvents envelope -> Central Event -> upstream data): source = inner Event.adapter category = Central hierarchical string -> meshai flat, via a small table-driven prefix map (map_category) severity = 0|1->routine, 2->priority, 3|4->immediate, null->routine lat/lon = geo.centroid, swapped from GeoJSON [lon,lat] -> (lat,lon) group_key/inhibit = outer envelope id (dedup parity with native adapters) expires/timestamp parsed from ISO-8601 Event.data = upstream payload verbatim (generic _enriched merge, preserved as-is incl. hydro's extra usgs_site/usgs_stats bundles) - Tombstone (`.removed.` subject or `:removed` id suffix) -> a "clear" Event carrying the ORIGINAL group_key (`:removed` stripped) + data._central_tombstone so the grouper/inhibitor lets the prior event lapse naturally. - config.py: a `_SourcedFeed` mixin adds `feed_source: native|central` (validated in __post_init__) to all 10 adapter configs; new CentralConsumerConfig as environmental.central { enabled, url, durable, connect_timeout }. Both ride the generic _dict_to_dataclass coercion, so they are GUI-editable via PUT /config/environmental (Rule 17) -- frontend fields come in C.2. - env/store.py: each adapter is instantiated only when enabled AND feed_source=="native"; a feed_source=central adapter is skipped natively (debug-logged) so Central can own it without a duplicate. - main.py: CentralConsumer constructed + started after start_pipeline(), stopped in stop(). DEVIATION FROM SPEC (documented): the spec named the new field `source`, but FIRMSConfig already has a `source` field (the satellite product, "VIIRS_SNPP_NRT"). To avoid the collision the field is named **feed_source** across all adapters. Everything else follows the spec. NETWORKING: zero infra change required. The meshai container already reaches the Central NATS server directly (TCP to 100.64.0.12:4222 OK) and resolves central.echo6.mesh via the Phase 2.6.6 MagicDNS fix. No docker-compose edit; default bridge works (LXC host masquerades to the Tailscale CGNAT range). The lighter bridge-route / host-net / sidecar fallbacks were not needed. Tests: tests/test_central_consumer.py (11) + tests/test_config_source_field.py (6): no-op-when-native, subjects-when-central, source-gate skips native instantiation, normalize+emit, _enriched preserved verbatim, tombstone->clear, severity map (0-4/null), category map (>=4 strings), async _on_message emits+acks, start() no-op without NATS, feed_source default/validate/reject/ dict-coercion. Full suite: 269 passed (was 253 + 16 new). Verification: (A) no bare self._x() in consumer.py. (B) py_compile clean. (C) 269 passed. (D) rebuilt prod -- 8 native adapters, pipeline started, native nifc/traffic emissions still flowing, healthy, no errors, log "CentralConsumer started; 0 subjects subscribed -- no adapters set to central". (E) in-container synthetic _on_message injection normalized correctly (usgs_quake/earthquake_event/immediate, centroid swapped, _enriched preserved) and reached the bus; ephemeral, no config change to roll back. C.2 (dashboard frontend for the feed_source switch + central connection) is next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 02:28:19 +00:00
"""v0.4 C.1: per-adapter `source` field + CentralConsumerConfig."""
import pytest
from meshai.config import (
NWSConfig, FIRMSConfig, USGSQuakeConfig,
EnvironmentalConfig, CentralConsumerConfig,
)
_ADAPTERS = ("nws", "swpc", "ducting", "fires", "avalanche",
"usgs", "usgs_quake", "traffic", "roads511", "firms")
def test_source_defaults_native():
assert NWSConfig().feed_source == "native"
assert FIRMSConfig().feed_source == "native"
assert USGSQuakeConfig().feed_source == "native"
def test_all_adapters_default_native():
env = EnvironmentalConfig()
for attr in _ADAPTERS:
assert getattr(env, attr).feed_source == "native", attr
def test_source_central_validates():
assert NWSConfig(feed_source="central").feed_source == "central"
assert USGSQuakeConfig(feed_source="central").feed_source == "central"
def test_source_garbage_rejects():
with pytest.raises(ValueError):
NWSConfig(feed_source="garbage")
with pytest.raises(ValueError):
FIRMSConfig(feed_source="")
def test_environmental_has_central_default():
env = EnvironmentalConfig()
assert isinstance(env.central, CentralConsumerConfig)
assert env.central.enabled is False
assert env.central.url.startswith("nats://")
def test_source_field_survives_dict_coercion():
"""A `source` in yaml/dict is coerced onto the adapter config."""
from meshai.config import Config, _dict_to_dataclass
cfg = _dict_to_dataclass(Config, {"environmental": {"usgs_quake": {"enabled": True, "feed_source": "central"}}})
assert cfg.environmental.usgs_quake.feed_source == "central"
assert cfg.environmental.nws.feed_source == "native" # untouched default