meshai/work/tests/test_satpass_persisted_timer.py

122 lines
5.3 KiB
Python
Raw Normal View History

chore: excise the dead Central NATS consumer path (-11,328 LOC) (#144) * chore: excise the dead Central NATS consumer path Central was retired and its database dropped 2026-07-15; its NATS broker no longer exists. Verified against the live CT108 deployment: all 12 adapters run feed_source=native, zero on central, and central.enabled is False (default, never overridden). The consumer and its handlers were unreachable. Removed: - central/consumer.py and 6 dead handlers (nws, quake, swpc, nwis, avy, incident) -- their handle_* entrypoints were reachable only from the consumer's dispatch - the Central wiring in main.py (init, guarded start, retry loop, stop path) - the dead config surface: CentralConsumerConfig, EnvironmentalConfig.central, adapter_config ("central","severity_thresholds") and its display block - the nats-py dependency (consumer.py was its only importer) - 19 test files that exercised only the dead path KEPT -- these live under central/ but are imported directly by native adapters, and deleting them would break production: - wfigs_handler.py: firms_handler._handle_pass_boundary() calls its _render() on the live FIRMS growth-fire path (env/firms.py -> ingest_hotspot_pixel) - firms_handler, satpass_handler, tle_handler: split files whose handle_* entrypoints are dead but whose engines are live. Left intact; splitting them is separate work. - pass_predictor, budget, idaho_gauge_sites: fully live. The usgs_quake keys global_mag_floor / regional_mag_floor / regional_centroid / regional_radius_mi / broadcast_pager_alerts are NOT removed despite comments labelling them "CENTRAL-PATH ONLY" -- notifications/gating/quake.py reads them unconditionally in the native path. Those comments are corrected separately. Test-count note: the suite drops ~425 tests. Most were migration PARITY tests whose sole purpose was proving the native rewrite byte-matched the Central handler (golden byte-parity, cross-source identity, gate-sequence replay). With the handler deleted there is nothing left to compare against, so they cannot exist. Native-only tests were kept and reworked where a test reached for a central symbol incidentally. This is a real coverage loss, accepted deliberately: the parity harness proved the refactor faithful, and git history preserves the originals. Suite: 1984 passed, 6 failed -- the same 6 pre-existing failures as main (stale SCHEMA_VERSION x3, expired TLE fixtures x2, one order-dependent), all being fixed on fix/green-test-suite. No new failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(nws): restore native-only golden coverage for the wire formatter Commit ca751fb5 deleted the Central nws_handler parity harness along with the handler itself, which took the ONLY tests that pinned formatters.nws .format()'s literal wire output. Gate-sequence and schema-conformance tests already survived natively; the formatter's actual rendered text did not have any native-only regression net. Add TestFormatterGolden to test_nws_refactor.py: 3 real-fixture cases plus 6 hand-built pathological cases mined from the deleted test_nws_handler.py (SVR path-sampling, the "no dangling separator" regression, TOR on-ground vs radar-indicated, FFW flood-cause detection). Every literal was verified by temporarily restoring the pre-excision central.nws_handler._render() from git history (ca751fb5^) in a throwaway, uncommitted script, confirming byte-identical output against the current native format() for all 37 real fixtures (nws/ + nws_last/) and all 9 pathological cases, then pinning the confirmed-matching string as the literal -- not a blind snapshot of current behavior. quake/swpc/avalanche/hydro/incident/fire were checked and already carry equivalent native-only golden coverage (added directly in ca751fb5), so no changes were needed there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 14:03:40 -06:00
"""Tests for the satpass persisted-timer `due_at` column.
Pending satellite-pass consolidations used to be scheduled only as in-memory
asyncio TimerHandles, so a restart orphaned any satpass_pending rows: the row
chore: excise the dead Central NATS consumer path (-11,328 LOC) (#144) * chore: excise the dead Central NATS consumer path Central was retired and its database dropped 2026-07-15; its NATS broker no longer exists. Verified against the live CT108 deployment: all 12 adapters run feed_source=native, zero on central, and central.enabled is False (default, never overridden). The consumer and its handlers were unreachable. Removed: - central/consumer.py and 6 dead handlers (nws, quake, swpc, nwis, avy, incident) -- their handle_* entrypoints were reachable only from the consumer's dispatch - the Central wiring in main.py (init, guarded start, retry loop, stop path) - the dead config surface: CentralConsumerConfig, EnvironmentalConfig.central, adapter_config ("central","severity_thresholds") and its display block - the nats-py dependency (consumer.py was its only importer) - 19 test files that exercised only the dead path KEPT -- these live under central/ but are imported directly by native adapters, and deleting them would break production: - wfigs_handler.py: firms_handler._handle_pass_boundary() calls its _render() on the live FIRMS growth-fire path (env/firms.py -> ingest_hotspot_pixel) - firms_handler, satpass_handler, tle_handler: split files whose handle_* entrypoints are dead but whose engines are live. Left intact; splitting them is separate work. - pass_predictor, budget, idaho_gauge_sites: fully live. The usgs_quake keys global_mag_floor / regional_mag_floor / regional_centroid / regional_radius_mi / broadcast_pager_alerts are NOT removed despite comments labelling them "CENTRAL-PATH ONLY" -- notifications/gating/quake.py reads them unconditionally in the native path. Those comments are corrected separately. Test-count note: the suite drops ~425 tests. Most were migration PARITY tests whose sole purpose was proving the native rewrite byte-matched the Central handler (golden byte-parity, cross-source identity, gate-sequence replay). With the handler deleted there is nothing left to compare against, so they cannot exist. Native-only tests were kept and reworked where a test reached for a central symbol incidentally. This is a real coverage loss, accepted deliberately: the parity harness proved the refactor faithful, and git history preserves the originals. Suite: 1984 passed, 6 failed -- the same 6 pre-existing failures as main (stale SCHEMA_VERSION x3, expired TLE fixtures x2, one order-dependent), all being fixed on fix/green-test-suite. No new failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(nws): restore native-only golden coverage for the wire formatter Commit ca751fb5 deleted the Central nws_handler parity harness along with the handler itself, which took the ONLY tests that pinned formatters.nws .format()'s literal wire output. Gate-sequence and schema-conformance tests already survived natively; the formatter's actual rendered text did not have any native-only regression net. Add TestFormatterGolden to test_nws_refactor.py: 3 real-fixture cases plus 6 hand-built pathological cases mined from the deleted test_nws_handler.py (SVR path-sampling, the "no dangling separator" regression, TOR on-ground vs radar-indicated, FFW flood-cause detection). Every literal was verified by temporarily restoring the pre-excision central.nws_handler._render() from git history (ca751fb5^) in a throwaway, uncommitted script, confirming byte-identical output against the current native format() for all 37 real fixtures (nws/ + nws_last/) and all 9 pathological cases, then pinning the confirmed-matching string as the literal -- not a blind snapshot of current behavior. quake/swpc/avalanche/hydro/incident/fire were checked and already carry equivalent native-only golden coverage (added directly in ca751fb5), so no changes were needed there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 14:03:40 -06:00
survived but its timer did not, and it was never consolidated/broadcast. The
fix persisted a durable `due_at` on each pending row and added a startup
sweep (`CentralConsumer._sweep_pending_satpass`) that reconstructed a timer
for every pending consolidated_id off its persisted due_at.
The Central NATS consumer (and `_sweep_pending_satpass` with it) was retired
2026-07 -- the sweep's tests are gone with it. The native satpass path
(env/satpass.py) never used the satpass_pending buffer or this sweep in the
first place (it consolidates in-memory within a single tick), so nothing
live is affected. What remains here:
- `due_at` is persisted on the normal ingest path (satpass_handler.py,
still live -- shared by both paths historically, now native-only)
- SCHEMA_VERSION == 26 and the v22 migration (which added the due_at
column) still applies cleanly on a fresh DB
"""
from __future__ import annotations
import json
import time
import pytest
from meshai.persistence import get_db, init_db, SCHEMA_VERSION
from meshai.adapter_config import invalidate_cache
# ── helpers ───────────────────────────────────────────────────────────
def _enable_satpass_db(norad_ids=(25544,), dry_run=True):
"""Enable satpass and set opt-in norad_ids in the test DB."""
conn = get_db()
conn.execute("UPDATE adapter_config SET value_json='true' "
"WHERE adapter='satpass' AND key='enabled'")
conn.execute("UPDATE adapter_config SET value_json=? "
"WHERE adapter='satpass' AND key='dry_run'",
(json.dumps(bool(dry_run)),))
conn.execute("UPDATE adapter_config SET value_json=? "
"WHERE adapter='satpass' AND key='norad_ids'",
(json.dumps(list(norad_ids)),))
invalidate_cache()
def _ingest_envelope(norad_id=25544, observer="Boise", max_el=72.5,
aos="2026-06-12T03:32:00Z", los="2026-06-12T03:38:00Z"):
return {
"specversion": "1.0",
"type": "central.sat.pass",
"source": "central",
"id": f"pass-{norad_id}-{aos}",
"data": {
"adapter": "n2yo_visualpasses",
"category": "pass.n2yo_visualpasses",
"data": {
"norad_id": norad_id,
"satellite_name": "ISS",
"observer_name": observer,
"max_elevation_deg": max_el,
"aos_time": aos,
"los_time": los,
"azimuth_at_peak_compass": "S",
"azimuth_at_aos_compass": "SW",
"azimuth_at_los_compass": "NE",
},
},
}
# ── schema / migration ───────────────────────────────────────────────
feat(phase4c): native SGP4 satpass — standalone satellite passes, no Central (#39) meshai can now predict + broadcast satellite passes locally without Central. Data plumbing (4c-1): - env/tle_fetch.py: keyless Celestrak GP fetcher (GROUP/CATNR, FORMAT=tle) → upserts the existing sat_tles table via a shared upsert_tle() helper extracted into tle_handler (Central ingest refactored to call it, unchanged) - observer_locations table (v23, SCHEMA_VERSION 22->23) + persistence helpers; seeded from SatpassConfig.observers in main._init_components - SatpassConfig: observers, tle_groups, norad_ids, tle_refresh_seconds, min_elevation_deg, window_hours Predictor + source-agnostic gate (4c-2): - extracted gate_consolidated_pass(consolidated, *, now) from consolidate_satpass_pending: dedup-vs-satpass_events + rate cap + format_pass + deferred commit. Central path byte-identical (114 tests unchanged) - env/satpass.py: native adapter predicts passes for each sat x observer via pass_predictor.compute_passes, consolidates IN-MEMORY per canonical hour bucket (earliest AOS / latest LOS / max-el observer supplies peak_compass + entry/exit observers), runs the shared gate, emits sat_pass. Commit rides event.data so satpass_events dedups across ticks — NO satpass_pending, NO Central-consumer timer dependency (works with Central off) - registered in env/store.py gated on enabled and feed_source==native 32 new tests (tle_fetch 16, observer_locations 14... satpass_native 8, minus overlaps); full suite 10-failure baseline (1672 passed). Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:25:55 -06:00
def test_schema_version_is_current():
fix(persistence): derive SCHEMA_VERSION from migrations; unbreak the red suite (#140) * fix(persistence): derive SCHEMA_VERSION from the migrations directory db.py hardcoded SCHEMA_VERSION = 26 while migrations/ had already reached v29 (v27 dispatcher floor-drop counter, v28 mesh_observations, v29 IPAWS). The migration runner globs the directory and applies every vN.sql it finds regardless of the constant, so a fresh DB actually landed at 29 while the constant claimed 26 -- a three-version drift that three tests were correctly catching. Derive it from the highest vN.sql present instead of bumping the literal, so it cannot drift again the next time someone adds a migration. Falls back to 0 if the directory is missing so import never fails; the migrations dir sits alongside db.py and ships with the package (Dockerfile COPYs meshai/). Adds a regression guard asserting the constant matches the highest migration file, and updates three tests that hardcoded 26 as a literal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(tle): make TLE fixtures time-relative so they cannot expire The ISS fixtures hardcoded epochs of 2026-06-30/07-01/07-02. tle_handler sets STALE_DAYS = 14 and get_tle_by_norad() filters on epoch >= now - 14d, so the fixtures silently aged out on 2026-07-02 and the tests began failing -- a time bomb, not a regression. Compute epochs relative to wall-clock now (base = now - 2d, +/-1d for newer/older) with correct TLE epoch-field encoding and mod-10 checksum. STALE_DAYS is untouched -- widening it in product code would have changed production behavior to paper over a test bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(fire-tracker): pin config + history db to tmp_path load_config() defaults HistoryConfig.database to the relative path "conversations.db", resolved against the process CWD, so every test calling load_config() with no override shares one file for the whole session. The conftest DB-isolation fixture only covers MESHAI_DB_PATH, not this. Point both the config dir and the history database at the test's tmp_path. NOTE: this does NOT resolve the order-dependent failure -- the test still passes standalone and fails in a full run, so the polluting state lives somewhere other than config/history. Left failing rather than weakened; root cause still unidentified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 13:03:40 -06:00
# SCHEMA_VERSION is derived from the highest vN.sql in migrations/, so
# this just guards against the derivation returning something bogus
# (e.g. 0, which would mean the migrations dir wasn't found). The v22
# migration (this file's focus) must always be <= the current version.
assert SCHEMA_VERSION >= 22
def test_v22_migration_applies_and_adds_due_at_column(tmp_path, monkeypatch):
feat(phase4c): native SGP4 satpass — standalone satellite passes, no Central (#39) meshai can now predict + broadcast satellite passes locally without Central. Data plumbing (4c-1): - env/tle_fetch.py: keyless Celestrak GP fetcher (GROUP/CATNR, FORMAT=tle) → upserts the existing sat_tles table via a shared upsert_tle() helper extracted into tle_handler (Central ingest refactored to call it, unchanged) - observer_locations table (v23, SCHEMA_VERSION 22->23) + persistence helpers; seeded from SatpassConfig.observers in main._init_components - SatpassConfig: observers, tle_groups, norad_ids, tle_refresh_seconds, min_elevation_deg, window_hours Predictor + source-agnostic gate (4c-2): - extracted gate_consolidated_pass(consolidated, *, now) from consolidate_satpass_pending: dedup-vs-satpass_events + rate cap + format_pass + deferred commit. Central path byte-identical (114 tests unchanged) - env/satpass.py: native adapter predicts passes for each sat x observer via pass_predictor.compute_passes, consolidates IN-MEMORY per canonical hour bucket (earliest AOS / latest LOS / max-el observer supplies peak_compass + entry/exit observers), runs the shared gate, emits sat_pass. Commit rides event.data so satpass_events dedups across ticks — NO satpass_pending, NO Central-consumer timer dependency (works with Central off) - registered in env/store.py gated on enabled and feed_source==native 32 new tests (tle_fetch 16, observer_locations 14... satpass_native 8, minus overlaps); full suite 10-failure baseline (1672 passed). Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:25:55 -06:00
"""Fresh DB migrates cleanly and satpass_pending has due_at (v22 column)."""
from meshai.persistence import close_thread_connection
from meshai.persistence import db as persistence_db
db = str(tmp_path / "fresh-v22.sqlite")
monkeypatch.setenv("MESHAI_DB_PATH", db)
persistence_db._initialised.clear()
close_thread_connection()
conn = init_db()
row = conn.execute("SELECT value FROM schema_meta WHERE key='version'").fetchone()
fix(persistence): derive SCHEMA_VERSION from migrations; unbreak the red suite (#140) * fix(persistence): derive SCHEMA_VERSION from the migrations directory db.py hardcoded SCHEMA_VERSION = 26 while migrations/ had already reached v29 (v27 dispatcher floor-drop counter, v28 mesh_observations, v29 IPAWS). The migration runner globs the directory and applies every vN.sql it finds regardless of the constant, so a fresh DB actually landed at 29 while the constant claimed 26 -- a three-version drift that three tests were correctly catching. Derive it from the highest vN.sql present instead of bumping the literal, so it cannot drift again the next time someone adds a migration. Falls back to 0 if the directory is missing so import never fails; the migrations dir sits alongside db.py and ships with the package (Dockerfile COPYs meshai/). Adds a regression guard asserting the constant matches the highest migration file, and updates three tests that hardcoded 26 as a literal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(tle): make TLE fixtures time-relative so they cannot expire The ISS fixtures hardcoded epochs of 2026-06-30/07-01/07-02. tle_handler sets STALE_DAYS = 14 and get_tle_by_norad() filters on epoch >= now - 14d, so the fixtures silently aged out on 2026-07-02 and the tests began failing -- a time bomb, not a regression. Compute epochs relative to wall-clock now (base = now - 2d, +/-1d for newer/older) with correct TLE epoch-field encoding and mod-10 checksum. STALE_DAYS is untouched -- widening it in product code would have changed production behavior to paper over a test bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(fire-tracker): pin config + history db to tmp_path load_config() defaults HistoryConfig.database to the relative path "conversations.db", resolved against the process CWD, so every test calling load_config() with no override shares one file for the whole session. The conftest DB-isolation fixture only covers MESHAI_DB_PATH, not this. Point both the config dir and the history database at the test's tmp_path. NOTE: this does NOT resolve the order-dependent failure -- the test still passes standalone and fails in a full run, so the polluting state lives somewhere other than config/history. Left failing rather than weakened; root cause still unidentified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Johnson <mj@k7zvx.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 13:03:40 -06:00
assert int(row["value"]) == SCHEMA_VERSION
cols = {r["name"] for r in conn.execute("PRAGMA table_info(satpass_pending)")}
assert "due_at" in cols
close_thread_connection()
persistence_db._initialised.discard(db)
# ── due_at persisted on normal ingest ────────────────────────────────
def test_due_at_persisted_on_normal_ingest():
"""handle_satpass writes due_at = received_at + CONSOLIDATION_DELAY."""
from meshai.central.satpass_handler import (
handle_satpass, CONSOLIDATION_DELAY, _parse_iso_epoch)
_enable_satpass_db(norad_ids=[25544], dry_run=True)
for attr in ("_disabled_logged", "_no_norad_ids_logged"):
if hasattr(handle_satpass, attr):
delattr(handle_satpass, attr)
env = _ingest_envelope()
aos_epoch = _parse_iso_epoch("2026-06-12T03:32:00Z")
now = aos_epoch - 300 # inside horizon, before los
assert handle_satpass(env, "central.sat.pass.iss", now=now) is None
conn = get_db()
row = conn.execute(
"SELECT received_at, due_at FROM satpass_pending "
"WHERE norad_id=25544").fetchone()
assert row is not None, "ingest did not write a pending row"
assert row["due_at"] is not None
assert row["due_at"] == row["received_at"] + CONSOLIDATION_DELAY
assert row["due_at"] == now + CONSOLIDATION_DELAY