mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-11 09:24:44 +02:00
feat(central): v0.4 C.1 Central connector backend (no-op until adapter source flipped)
Adds the backend for sourcing environmental feeds from Central's NATS
JetStream firehose instead of (or alongside) meshai's native adapters.
Architecture is Matt-approved Option 3' (dedicated package + per-adapter
source switch surfaced on the existing Environmental config).
NO-OP POSTURE (intentional): every adapter defaults to feed_source="native"
and environmental.central.enabled defaults false, so on a stock config the
CentralConsumer starts and subscribes to nothing -- behavior is byte-for-byte
v0.3. Live env_feeds.yaml is unchanged on disk; an operator who touches
nothing sees no change. Flipping an adapter to central is Phase C.3; the
dashboard UI for it is Phase C.2.
What landed:
- meshai/central/ package (CentralConsumer): async start()/stop(), JetStream
durable subscribe to subjects derived from adapters with feed_source=central,
and _on_message -> normalize -> bus.emit. nats-py is lazy-imported only on
the connect path, so no-op boot has zero NATS dependency.
- Normalization (CloudEvents envelope -> Central Event -> upstream data):
source = inner Event.adapter
category = Central hierarchical string -> meshai flat, via a small
table-driven prefix map (map_category)
severity = 0|1->routine, 2->priority, 3|4->immediate, null->routine
lat/lon = geo.centroid, swapped from GeoJSON [lon,lat] -> (lat,lon)
group_key/inhibit = outer envelope id (dedup parity with native adapters)
expires/timestamp parsed from ISO-8601
Event.data = upstream payload verbatim (generic _enriched merge, preserved
as-is incl. hydro's extra usgs_site/usgs_stats bundles)
- Tombstone (`.removed.` subject or `:removed` id suffix) -> a "clear" Event
carrying the ORIGINAL group_key (`:removed` stripped) + data._central_tombstone
so the grouper/inhibitor lets the prior event lapse naturally.
- config.py: a `_SourcedFeed` mixin adds `feed_source: native|central`
(validated in __post_init__) to all 10 adapter configs; new
CentralConsumerConfig as environmental.central { enabled, url, durable,
connect_timeout }. Both ride the generic _dict_to_dataclass coercion, so
they are GUI-editable via PUT /config/environmental (Rule 17) -- frontend
fields come in C.2.
- env/store.py: each adapter is instantiated only when
enabled AND feed_source=="native"; a feed_source=central adapter is skipped
natively (debug-logged) so Central can own it without a duplicate.
- main.py: CentralConsumer constructed + started after start_pipeline(),
stopped in stop().
DEVIATION FROM SPEC (documented): the spec named the new field `source`, but
FIRMSConfig already has a `source` field (the satellite product,
"VIIRS_SNPP_NRT"). To avoid the collision the field is named **feed_source**
across all adapters. Everything else follows the spec.
NETWORKING: zero infra change required. The meshai container already reaches
the Central NATS server directly (TCP to 100.64.0.12:4222 OK) and resolves
central.echo6.mesh via the Phase 2.6.6 MagicDNS fix. No docker-compose edit;
default bridge works (LXC host masquerades to the Tailscale CGNAT range). The
lighter bridge-route / host-net / sidecar fallbacks were not needed.
Tests: tests/test_central_consumer.py (11) + tests/test_config_source_field.py
(6): no-op-when-native, subjects-when-central, source-gate skips native
instantiation, normalize+emit, _enriched preserved verbatim, tombstone->clear,
severity map (0-4/null), category map (>=4 strings), async _on_message
emits+acks, start() no-op without NATS, feed_source default/validate/reject/
dict-coercion. Full suite: 269 passed (was 253 + 16 new).
Verification: (A) no bare self._x() in consumer.py. (B) py_compile clean.
(C) 269 passed. (D) rebuilt prod -- 8 native adapters, pipeline started,
native nifc/traffic emissions still flowing, healthy, no errors, log
"CentralConsumer started; 0 subjects subscribed -- no adapters set to central".
(E) in-container synthetic _on_message injection normalized correctly
(usgs_quake/earthquake_event/immediate, centroid swapped, _enriched preserved)
and reached the bus; ephemeral, no config change to roll back.
C.2 (dashboard frontend for the feed_source switch + central connection) is next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
20e0dec28a
commit
73c007d227
8 changed files with 500 additions and 20 deletions
24
meshai/env/store.py
vendored
24
meshai/env/store.py
vendored
|
|
@ -29,49 +29,53 @@ class EnvironmentalStore:
|
|||
self._region_anchors = region_anchors or []
|
||||
|
||||
# Create adapter instances based on config
|
||||
if config.nws.enabled:
|
||||
if config.nws.enabled and config.nws.feed_source == "native":
|
||||
from .nws import NWSAlertsAdapter
|
||||
self._adapters["nws"] = NWSAlertsAdapter(config.nws)
|
||||
|
||||
if config.swpc.enabled:
|
||||
if config.swpc.enabled and config.swpc.feed_source == "native":
|
||||
from .swpc import SWPCAdapter
|
||||
self._adapters["swpc"] = SWPCAdapter(config.swpc)
|
||||
|
||||
if config.ducting.enabled:
|
||||
if config.ducting.enabled and config.ducting.feed_source == "native":
|
||||
from .ducting import DuctingAdapter
|
||||
self._adapters["ducting"] = DuctingAdapter(config.ducting)
|
||||
|
||||
if config.fires.enabled:
|
||||
if config.fires.enabled and config.fires.feed_source == "native":
|
||||
from .fires import NICFFiresAdapter
|
||||
self._adapters["nifc"] = NICFFiresAdapter(config.fires, self._region_anchors)
|
||||
|
||||
if config.avalanche.enabled:
|
||||
if config.avalanche.enabled and config.avalanche.feed_source == "native":
|
||||
from .avalanche import AvalancheAdapter
|
||||
self._adapters["avalanche"] = AvalancheAdapter(config.avalanche)
|
||||
|
||||
if config.usgs.enabled:
|
||||
if config.usgs.enabled and config.usgs.feed_source == "native":
|
||||
from .usgs import USGSStreamsAdapter
|
||||
self._adapters["usgs"] = USGSStreamsAdapter(config.usgs)
|
||||
|
||||
if config.usgs_quake.enabled:
|
||||
if config.usgs_quake.enabled and config.usgs_quake.feed_source == "native":
|
||||
from .usgs_quake import USGSQuakeAdapter
|
||||
self._adapters["usgs_quake"] = USGSQuakeAdapter(config.usgs_quake)
|
||||
|
||||
if config.traffic.enabled:
|
||||
if config.traffic.enabled and config.traffic.feed_source == "native":
|
||||
from .traffic import TomTomTrafficAdapter
|
||||
self._adapters["traffic"] = TomTomTrafficAdapter(config.traffic)
|
||||
|
||||
if config.roads511.enabled:
|
||||
if config.roads511.enabled and config.roads511.feed_source == "native":
|
||||
from .roads511 import Roads511Adapter
|
||||
self._adapters["roads511"] = Roads511Adapter(config.roads511)
|
||||
|
||||
# FIRMS needs reference to NIFC adapter for cross-referencing
|
||||
if config.firms.enabled:
|
||||
if config.firms.enabled and config.firms.feed_source == "native":
|
||||
from .firms import FIRMSAdapter
|
||||
fires_adapter = self._adapters.get("nifc")
|
||||
self._firms = FIRMSAdapter(config.firms, self._region_anchors, fires_adapter)
|
||||
self._adapters["firms"] = self._firms
|
||||
|
||||
_central = [n for n in ("nws", "swpc", "ducting", "fires", "avalanche", "usgs", "usgs_quake", "traffic", "roads511", "firms")
|
||||
if getattr(getattr(config, n, None), "feed_source", "native") == "central"]
|
||||
if _central:
|
||||
logger.debug("Adapters sourced from Central (native skipped): %s", _central)
|
||||
logger.info(f"EnvironmentalStore initialized with {len(self._adapters)} adapters")
|
||||
|
||||
def refresh(self) -> bool:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue