feat(v0.6-2): dispatcher state persistence -- cold-start, cooldowns, dedup LRU to SQLite
Closes Rule-20 dispatcher gap from audit doc v0.6-phase1-audit.md finding #1.
Pre-this-commit the cold-start anchor, 4 drop counters, per-toggle cooldown
map, and dedup OrderedDict all lived in Dispatcher instance memory and were
lost on every container restart.
v5.sql adds three tables:
- dispatcher_state (singleton id=1): cold_start_anchor + 4 drop counters
- dispatcher_cooldowns ((toggle,category,region) keyed): last_fired_at
- dispatcher_dedup ((source,event_id) keyed): seen_at
Dispatcher refactor:
- __init__ calls _restore_from_db -- counters, cold-start anchor, cooldown
map, and dedup LRU (most-recent 10k by seen_at) all rehydrated from the
three new tables
- write-through on every mutation: _persist_state for counter/anchor,
_persist_cooldown for cooldown UPSERT + 2*cooldown_s prune,
_persist_dedup for dedup INSERT OR REPLACE + 7-day cleanup
- in-memory caches stay authoritative on the fast read path
- cumulative-since-install counters (NOT since-boot); LLM will be able
to answer "we have dropped 47 stale events this week" after commit #5
(env_reporter) lands
- graceful degrade: missing v5 tables / persistence outage falls back to
fresh in-memory state without crashing the constructor
Tests:
- tests/test_dispatcher_persistence.py (17 tests): state restore on init,
counter+cooldown+dedup survival across simulated restart, cooldown rearm
within 2x window, dedup LRU rebuild caps at 10k, 7-day cleanup on insert,
INSERT OR REPLACE on duplicate source+event_id, v5 migration idempotent,
synthetic storm (50 events) -> restart -> replay (5 incl 1 duplicate)
with the duplicate dedup-rejected and counters NOT resetting
- tests/conftest.py (new): autouse MESHAI_DB_PATH redirection to per-test
tmp file, so the dispatcher_* tables on production /data dont get
polluted by tests that construct Dispatcher() without an explicit fixture
- tests/test_notification_toggles.py: _dispatch helper wipes dedup/cooldown/
state tables between calls (per-call independence preserved; pre-v0.6-2
in-memory-only Dispatcher reset naturally per instance)
Test count: 680 -> 697 (+17 new, 0 regressions).
Refs audit doc v0.6-phase1-audit.md finding #1.
2026-06-05 16:35:40 +00:00
|
|
|
"""Pytest fixture isolation for meshai persistence (v0.6-2).
|
|
|
|
|
|
|
|
|
|
Before v0.6-2 the dispatcher held all state in instance memory, so tests
|
|
|
|
|
that constructed `Dispatcher(...)` were inert w.r.t. SQLite. v0.6-2 made
|
|
|
|
|
`Dispatcher.__init__` read/restore from the persistence layer, which by
|
|
|
|
|
default points at `/data/meshai.sqlite`. Without isolation every test
|
|
|
|
|
would now read+write production state, polluting across tests and across
|
|
|
|
|
pytest invocations.
|
|
|
|
|
|
|
|
|
|
This autouse fixture redirects `MESHAI_DB_PATH` to a per-test tmp file
|
|
|
|
|
and clears the persistence-layer threading.local caches around each test.
|
|
|
|
|
Existing tests that don't reference any fixture get isolation for free;
|
|
|
|
|
tests that explicitly use a `db_path` (or similar) fixture can still
|
|
|
|
|
override the env var inside their own fixture body -- last setenv wins.
|
|
|
|
|
"""
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from meshai.persistence import close_thread_connection
|
|
|
|
|
from meshai.persistence import db as _persistence_db
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
|
|
|
def _isolate_meshai_db(tmp_path, monkeypatch):
|
feat(v0.6-3a): adapter_config foundation -- migration + defaults registry + typed accessor
Closes the foundation slice of audit doc Section A (Rule 17). Lands two
SQLite tables, the seed routine that populates them from a Python
defaults registry, and a typed accessor that handler code will read in
v0.6-3b. No handler changes in this commit -- ZERO behavior risk, every
existing test still passes (721 / 69 skipped / 0 failed).
v6.sql tables:
- adapter_config(adapter, key, value_json, default_json, type, description,
updated_at) PRIMARY KEY(adapter, key) -- JSON-encoded
values flow through a single column uniformly. CHECK
constraint on `type` closes the vocab (int/float/str/
bool/json).
- adapter_meta(adapter PK, display_name, include_in_llm_context,
description, updated_at) -- per-adapter metadata + the
user-scopable LLM-context toggle (Matt refinement #5).
meshai/adapter_config/ package:
- defaults.py: REGISTRY dict mapping (adapter, key) -> {default, type,
description}. Covers audit doc sections A.1-A.12: wfigs, nws,
usgs_quake, swpc, usgs_nwis, incident family (tomtom_incidents,
state_511_atis, itd_511, shared "incident"), central consumer,
dispatcher, band_conditions, geocoder, firms, pipeline (Inhibitor +
Grouper). ~85 keys total. ADAPTER_META covers 15 adapters with
display_name + include_in_llm_context defaulting to True. Per Matt
refinement #3, every default matches the current handler constant
EXACTLY -- first deploy behavior is unchanged.
- _accessor.py: AdapterConfig class with `adapter_config.<adapter>.<key>`
syntax. Read pipeline: in-memory cache hit -> SQL -> registry
fallback (with WARNING) -> AttributeError. Process-wide cache; PUT
via v0.6-3c REST API calls invalidate_cache() to drop the cache.
GIL-atomic dict reads on the fast path (handlers call this hot).
- __init__.py: seed_defaults(conn) -- INSERT OR IGNOREs one row per
registry entry. Idempotent, never overwrites user edits.
Wiring:
- meshai/persistence/db.py: SCHEMA_VERSION 5 -> 6, and init_db() now
calls seed_defaults() after migrations apply.
- meshai/main.py: _init_components() now calls init_db() FIRST (per
commit #1 lessons-learned: a startup-time migration is required
when handlers will rely on the new schema; lazy-on-first-handler
is fine for v4/v5 but not for v6 where handler reads start in
v0.6-3b).
- tests/conftest.py: autouse fixture now calls init_db() + clears
the accessor cache around each test, so every test gets the v6
seed AND a clean cache without per-test boilerplate.
Tests (tests/test_adapter_config_foundation.py, 24 cases):
- v6 tables exist + schema_meta at 6 + type-vocabulary CHECK enforced
- seed populates every REGISTRY + ADAPTER_META row, value_json ==
default_json on first seed, type matches
- seed is idempotent + does not overwrite user edits
- accessor returns correctly typed values for int/float/str/bool/
json list/json dict/json None
- cache hit: second read does not touch the DB (patched _load_from_db
raises, accessor still succeeds)
- invalidate_cache forces a re-read; mutated DB value wins
- registry fallback path triggers when a row is missing (with WARNING)
- unknown key raises AttributeError
- setattr blocked (writes go via the REST API in 3c)
- every default JSON round-trips cleanly; every type is in vocabulary
- ADAPTER_META covers every adapter in REGISTRY
Test count: 697 -> 721 (+24 new, 0 regressions).
v0.6-3b will wire handlers one at a time (wfigs, nws, quake, swpc, nwis,
incident, central, dispatcher, band_conditions, geocoder, firms). Per
the audit lock, defaults match exactly so each wiring step is a pure
refactor -- bisect-safe.
v0.6-3c lands the /api/adapter-config CRUD + the AdapterConfig.tsx
dashboard editor + cache invalidation on PUT.
Refs audit doc v0.6-phase1-audit.md Section A + finding #4.
2026-06-05 17:06:51 +00:00
|
|
|
"""Point MESHAI_DB_PATH at a tmp file per test + run init_db so the
|
|
|
|
|
v6 adapter_config seed lands before any test code runs.
|
|
|
|
|
|
|
|
|
|
v0.6-3a: init_db() applies pending migrations AND seeds
|
|
|
|
|
adapter_config from the defaults registry. Tests that read via
|
|
|
|
|
need the seed in place.
|
|
|
|
|
"""
|
feat(v0.6-2): dispatcher state persistence -- cold-start, cooldowns, dedup LRU to SQLite
Closes Rule-20 dispatcher gap from audit doc v0.6-phase1-audit.md finding #1.
Pre-this-commit the cold-start anchor, 4 drop counters, per-toggle cooldown
map, and dedup OrderedDict all lived in Dispatcher instance memory and were
lost on every container restart.
v5.sql adds three tables:
- dispatcher_state (singleton id=1): cold_start_anchor + 4 drop counters
- dispatcher_cooldowns ((toggle,category,region) keyed): last_fired_at
- dispatcher_dedup ((source,event_id) keyed): seen_at
Dispatcher refactor:
- __init__ calls _restore_from_db -- counters, cold-start anchor, cooldown
map, and dedup LRU (most-recent 10k by seen_at) all rehydrated from the
three new tables
- write-through on every mutation: _persist_state for counter/anchor,
_persist_cooldown for cooldown UPSERT + 2*cooldown_s prune,
_persist_dedup for dedup INSERT OR REPLACE + 7-day cleanup
- in-memory caches stay authoritative on the fast read path
- cumulative-since-install counters (NOT since-boot); LLM will be able
to answer "we have dropped 47 stale events this week" after commit #5
(env_reporter) lands
- graceful degrade: missing v5 tables / persistence outage falls back to
fresh in-memory state without crashing the constructor
Tests:
- tests/test_dispatcher_persistence.py (17 tests): state restore on init,
counter+cooldown+dedup survival across simulated restart, cooldown rearm
within 2x window, dedup LRU rebuild caps at 10k, 7-day cleanup on insert,
INSERT OR REPLACE on duplicate source+event_id, v5 migration idempotent,
synthetic storm (50 events) -> restart -> replay (5 incl 1 duplicate)
with the duplicate dedup-rejected and counters NOT resetting
- tests/conftest.py (new): autouse MESHAI_DB_PATH redirection to per-test
tmp file, so the dispatcher_* tables on production /data dont get
polluted by tests that construct Dispatcher() without an explicit fixture
- tests/test_notification_toggles.py: _dispatch helper wipes dedup/cooldown/
state tables between calls (per-call independence preserved; pre-v0.6-2
in-memory-only Dispatcher reset naturally per instance)
Test count: 680 -> 697 (+17 new, 0 regressions).
Refs audit doc v0.6-phase1-audit.md finding #1.
2026-06-05 16:35:40 +00:00
|
|
|
p = str(tmp_path / "meshai-test-isolated.sqlite")
|
|
|
|
|
monkeypatch.setenv("MESHAI_DB_PATH", p)
|
|
|
|
|
_persistence_db._initialised.clear()
|
|
|
|
|
close_thread_connection()
|
feat(v0.6-3a): adapter_config foundation -- migration + defaults registry + typed accessor
Closes the foundation slice of audit doc Section A (Rule 17). Lands two
SQLite tables, the seed routine that populates them from a Python
defaults registry, and a typed accessor that handler code will read in
v0.6-3b. No handler changes in this commit -- ZERO behavior risk, every
existing test still passes (721 / 69 skipped / 0 failed).
v6.sql tables:
- adapter_config(adapter, key, value_json, default_json, type, description,
updated_at) PRIMARY KEY(adapter, key) -- JSON-encoded
values flow through a single column uniformly. CHECK
constraint on `type` closes the vocab (int/float/str/
bool/json).
- adapter_meta(adapter PK, display_name, include_in_llm_context,
description, updated_at) -- per-adapter metadata + the
user-scopable LLM-context toggle (Matt refinement #5).
meshai/adapter_config/ package:
- defaults.py: REGISTRY dict mapping (adapter, key) -> {default, type,
description}. Covers audit doc sections A.1-A.12: wfigs, nws,
usgs_quake, swpc, usgs_nwis, incident family (tomtom_incidents,
state_511_atis, itd_511, shared "incident"), central consumer,
dispatcher, band_conditions, geocoder, firms, pipeline (Inhibitor +
Grouper). ~85 keys total. ADAPTER_META covers 15 adapters with
display_name + include_in_llm_context defaulting to True. Per Matt
refinement #3, every default matches the current handler constant
EXACTLY -- first deploy behavior is unchanged.
- _accessor.py: AdapterConfig class with `adapter_config.<adapter>.<key>`
syntax. Read pipeline: in-memory cache hit -> SQL -> registry
fallback (with WARNING) -> AttributeError. Process-wide cache; PUT
via v0.6-3c REST API calls invalidate_cache() to drop the cache.
GIL-atomic dict reads on the fast path (handlers call this hot).
- __init__.py: seed_defaults(conn) -- INSERT OR IGNOREs one row per
registry entry. Idempotent, never overwrites user edits.
Wiring:
- meshai/persistence/db.py: SCHEMA_VERSION 5 -> 6, and init_db() now
calls seed_defaults() after migrations apply.
- meshai/main.py: _init_components() now calls init_db() FIRST (per
commit #1 lessons-learned: a startup-time migration is required
when handlers will rely on the new schema; lazy-on-first-handler
is fine for v4/v5 but not for v6 where handler reads start in
v0.6-3b).
- tests/conftest.py: autouse fixture now calls init_db() + clears
the accessor cache around each test, so every test gets the v6
seed AND a clean cache without per-test boilerplate.
Tests (tests/test_adapter_config_foundation.py, 24 cases):
- v6 tables exist + schema_meta at 6 + type-vocabulary CHECK enforced
- seed populates every REGISTRY + ADAPTER_META row, value_json ==
default_json on first seed, type matches
- seed is idempotent + does not overwrite user edits
- accessor returns correctly typed values for int/float/str/bool/
json list/json dict/json None
- cache hit: second read does not touch the DB (patched _load_from_db
raises, accessor still succeeds)
- invalidate_cache forces a re-read; mutated DB value wins
- registry fallback path triggers when a row is missing (with WARNING)
- unknown key raises AttributeError
- setattr blocked (writes go via the REST API in 3c)
- every default JSON round-trips cleanly; every type is in vocabulary
- ADAPTER_META covers every adapter in REGISTRY
Test count: 697 -> 721 (+24 new, 0 regressions).
v0.6-3b will wire handlers one at a time (wfigs, nws, quake, swpc, nwis,
incident, central, dispatcher, band_conditions, geocoder, firms). Per
the audit lock, defaults match exactly so each wiring step is a pure
refactor -- bisect-safe.
v0.6-3c lands the /api/adapter-config CRUD + the AdapterConfig.tsx
dashboard editor + cache invalidation on PUT.
Refs audit doc v0.6-phase1-audit.md Section A + finding #4.
2026-06-05 17:06:51 +00:00
|
|
|
# Clear the adapter_config accessor cache so the prior tests
|
|
|
|
|
# in-memory cache does not leak into this test.
|
|
|
|
|
try:
|
|
|
|
|
from meshai.adapter_config import invalidate_cache
|
|
|
|
|
invalidate_cache()
|
|
|
|
|
except Exception:
|
|
|
|
|
pass
|
|
|
|
|
# Eager init: applies v1..v6 migrations + seeds adapter_config.
|
|
|
|
|
from meshai.persistence import init_db
|
|
|
|
|
init_db()
|
feat(v0.6-2): dispatcher state persistence -- cold-start, cooldowns, dedup LRU to SQLite
Closes Rule-20 dispatcher gap from audit doc v0.6-phase1-audit.md finding #1.
Pre-this-commit the cold-start anchor, 4 drop counters, per-toggle cooldown
map, and dedup OrderedDict all lived in Dispatcher instance memory and were
lost on every container restart.
v5.sql adds three tables:
- dispatcher_state (singleton id=1): cold_start_anchor + 4 drop counters
- dispatcher_cooldowns ((toggle,category,region) keyed): last_fired_at
- dispatcher_dedup ((source,event_id) keyed): seen_at
Dispatcher refactor:
- __init__ calls _restore_from_db -- counters, cold-start anchor, cooldown
map, and dedup LRU (most-recent 10k by seen_at) all rehydrated from the
three new tables
- write-through on every mutation: _persist_state for counter/anchor,
_persist_cooldown for cooldown UPSERT + 2*cooldown_s prune,
_persist_dedup for dedup INSERT OR REPLACE + 7-day cleanup
- in-memory caches stay authoritative on the fast read path
- cumulative-since-install counters (NOT since-boot); LLM will be able
to answer "we have dropped 47 stale events this week" after commit #5
(env_reporter) lands
- graceful degrade: missing v5 tables / persistence outage falls back to
fresh in-memory state without crashing the constructor
Tests:
- tests/test_dispatcher_persistence.py (17 tests): state restore on init,
counter+cooldown+dedup survival across simulated restart, cooldown rearm
within 2x window, dedup LRU rebuild caps at 10k, 7-day cleanup on insert,
INSERT OR REPLACE on duplicate source+event_id, v5 migration idempotent,
synthetic storm (50 events) -> restart -> replay (5 incl 1 duplicate)
with the duplicate dedup-rejected and counters NOT resetting
- tests/conftest.py (new): autouse MESHAI_DB_PATH redirection to per-test
tmp file, so the dispatcher_* tables on production /data dont get
polluted by tests that construct Dispatcher() without an explicit fixture
- tests/test_notification_toggles.py: _dispatch helper wipes dedup/cooldown/
state tables between calls (per-call independence preserved; pre-v0.6-2
in-memory-only Dispatcher reset naturally per instance)
Test count: 680 -> 697 (+17 new, 0 regressions).
Refs audit doc v0.6-phase1-audit.md finding #1.
2026-06-05 16:35:40 +00:00
|
|
|
yield p
|
|
|
|
|
close_thread_connection()
|
|
|
|
|
_persistence_db._initialised.discard(p)
|
feat(v0.6-3a): adapter_config foundation -- migration + defaults registry + typed accessor
Closes the foundation slice of audit doc Section A (Rule 17). Lands two
SQLite tables, the seed routine that populates them from a Python
defaults registry, and a typed accessor that handler code will read in
v0.6-3b. No handler changes in this commit -- ZERO behavior risk, every
existing test still passes (721 / 69 skipped / 0 failed).
v6.sql tables:
- adapter_config(adapter, key, value_json, default_json, type, description,
updated_at) PRIMARY KEY(adapter, key) -- JSON-encoded
values flow through a single column uniformly. CHECK
constraint on `type` closes the vocab (int/float/str/
bool/json).
- adapter_meta(adapter PK, display_name, include_in_llm_context,
description, updated_at) -- per-adapter metadata + the
user-scopable LLM-context toggle (Matt refinement #5).
meshai/adapter_config/ package:
- defaults.py: REGISTRY dict mapping (adapter, key) -> {default, type,
description}. Covers audit doc sections A.1-A.12: wfigs, nws,
usgs_quake, swpc, usgs_nwis, incident family (tomtom_incidents,
state_511_atis, itd_511, shared "incident"), central consumer,
dispatcher, band_conditions, geocoder, firms, pipeline (Inhibitor +
Grouper). ~85 keys total. ADAPTER_META covers 15 adapters with
display_name + include_in_llm_context defaulting to True. Per Matt
refinement #3, every default matches the current handler constant
EXACTLY -- first deploy behavior is unchanged.
- _accessor.py: AdapterConfig class with `adapter_config.<adapter>.<key>`
syntax. Read pipeline: in-memory cache hit -> SQL -> registry
fallback (with WARNING) -> AttributeError. Process-wide cache; PUT
via v0.6-3c REST API calls invalidate_cache() to drop the cache.
GIL-atomic dict reads on the fast path (handlers call this hot).
- __init__.py: seed_defaults(conn) -- INSERT OR IGNOREs one row per
registry entry. Idempotent, never overwrites user edits.
Wiring:
- meshai/persistence/db.py: SCHEMA_VERSION 5 -> 6, and init_db() now
calls seed_defaults() after migrations apply.
- meshai/main.py: _init_components() now calls init_db() FIRST (per
commit #1 lessons-learned: a startup-time migration is required
when handlers will rely on the new schema; lazy-on-first-handler
is fine for v4/v5 but not for v6 where handler reads start in
v0.6-3b).
- tests/conftest.py: autouse fixture now calls init_db() + clears
the accessor cache around each test, so every test gets the v6
seed AND a clean cache without per-test boilerplate.
Tests (tests/test_adapter_config_foundation.py, 24 cases):
- v6 tables exist + schema_meta at 6 + type-vocabulary CHECK enforced
- seed populates every REGISTRY + ADAPTER_META row, value_json ==
default_json on first seed, type matches
- seed is idempotent + does not overwrite user edits
- accessor returns correctly typed values for int/float/str/bool/
json list/json dict/json None
- cache hit: second read does not touch the DB (patched _load_from_db
raises, accessor still succeeds)
- invalidate_cache forces a re-read; mutated DB value wins
- registry fallback path triggers when a row is missing (with WARNING)
- unknown key raises AttributeError
- setattr blocked (writes go via the REST API in 3c)
- every default JSON round-trips cleanly; every type is in vocabulary
- ADAPTER_META covers every adapter in REGISTRY
Test count: 697 -> 721 (+24 new, 0 regressions).
v0.6-3b will wire handlers one at a time (wfigs, nws, quake, swpc, nwis,
incident, central, dispatcher, band_conditions, geocoder, firms). Per
the audit lock, defaults match exactly so each wiring step is a pure
refactor -- bisect-safe.
v0.6-3c lands the /api/adapter-config CRUD + the AdapterConfig.tsx
dashboard editor + cache invalidation on PUT.
Refs audit doc v0.6-phase1-audit.md Section A + finding #4.
2026-06-05 17:06:51 +00:00
|
|
|
try:
|
|
|
|
|
from meshai.adapter_config import invalidate_cache
|
|
|
|
|
invalidate_cache()
|
|
|
|
|
except Exception:
|
|
|
|
|
pass
|