mirror of
https://github.com/zvx-echo6/meshai.git
synced 2026-06-11 01:14:45 +02:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| e3bf53ade4 |
feat(v0.6-4): gauge_sites + town_anchors curation tables + GUI CRUD
Closes Section A.5 (gauge_sites) and A.12 (town_anchors) of the audit
doc by lifting both Python-dict curation lists into editable SQLite
tables. Operators can add/edit/disable rows from the dashboard without
a deploy; runtime reads go through cached accessors that invalidate
when the REST API mutates state.
Schema:
v8.sql adds gauge_sites(site_id PK, gauge_name, lat, lon, action_ft,
flood_minor_ft, flood_moderate_ft, flood_major_ft, enabled, updated_at).
v9.sql adds town_anchors(anchor_id AUTOINC PK, name UNIQUE, lat, lon,
state, enabled, updated_at).
SCHEMA_VERSION 7 -> 9.
Seed (meshai/persistence/curation.py):
_GAUGE_SITES_SEED carries the original 9 Idaho rows from
IDAHO_CURATED_SITES verbatim.
_TOWN_ANCHORS_SEED carries the 29 Idaho-and-neighbor towns from
_TOWN_COORDS verbatim.
seed_gauge_sites() / seed_town_anchors() INSERT OR IGNORE -- safe to
re-run; never overwrites user edits.
Handler integration:
- meshai/central/idaho_gauge_sites.py: IDAHO_CURATED_SITES dict deleted.
lookup_site() now calls meshai.persistence.curation.lookup_gauge_site()
which reads the table. THRESHOLD_RANK, normalize_site_id, and
compute_threshold_state remain in this module (CODE per Matt s rule).
- meshai/central/nwis_handler.py drops IDAHO_CURATED_SITES from its
import list; the table-backed lookup_site() is API-compatible.
- meshai/central_normalizer.py: _TOWN_COORDS dict deleted.
_compute_distance_bearing() now calls
meshai.persistence.curation.lookup_town_anchor() with the same
lowercased-name semantics it always used.
REST API (meshai/dashboard/api/curation_routes.py):
/api/gauge-sites GET list, GET one, POST add, PUT update, DELETE
/api/town-anchors GET list, GET one, POST add, PUT update, DELETE
Every mutation calls invalidate_curation_cache() so handler reads see
the new state on the next call -- no container restart.
Dashboard (dashboard-frontend/src/pages/):
- GaugeSites.tsx: table view with Add row / Edit row inline / Delete
confirm + per-row enabled toggle. 8 columns mirror the schema.
- TownAnchors.tsx: same pattern, 5 columns. Name is lowercased on
save to match the lookup key.
- Left-nav entries "Gauge Sites" (Droplets icon) and "Town Anchors"
(MapPin icon) added to Layout.tsx; routes added to App.tsx.
Tests (tests/test_curation.py, 18 cases):
- v8/v9 tables exist
- Seed lands every row from both dicts
- Seed idempotent; never overwrites user edits
- lookup_gauge_site hits/miss, disabled rows are invisible
- lookup_town_anchor case-insensitive
- REST API: GET list, GET one, GET 404, POST add, PUT update, DELETE,
POST missing-field 400; both gauge_sites + town_anchors
- Accessor reflects API mutations after invalidate_curation_cache()
tests/test_nwis_handler.py back-compat: IDAHO_CURATED_SITES dict alias
points at _GAUGE_SITES_SEED so the existing assertion suite still passes.
tests/test_adapter_config_foundation.py schema_meta v7 -> v9 bump.
Test count: 797 -> 819 (+18 curation cases + 4 maintenance updates).
|
|||
| 914d38c907 |
feat(v0.6-3b): wire every handler to adapter_config + v7.sql firms dedup_key column
Replaces module-level magic numbers in 12 handlers with reads via the
v0.6-3a.1 typed accessor. Every default matches the prior hardcoded
value exactly, so first-deploy behavior is unchanged.
Handlers wired (43 keys across the 43-row registry):
wfigs cooldown_seconds, anchor_max_mi, broadcast_on_acres,
broadcast_on_contained
nws broadcast_severities, tombstone_msgtypes,
warning_suffix_promotes
usgs_quake regional_centroid, regional_radius_mi,
broadcast_pager_alerts, global_mag_floor,
regional_mag_floor, escalate_mag_floor
swpc geomag_kp_floor (extends G-scale down to Kp 5 when
lowered), flare_class_floor (R-scale
extended to M-class when lowered),
proton_pfu_floor
usgs_nwis parameter_codes, broadcast_on_recede
incident freshness_seconds, broadcast_on_update (Update path
re-implemented when toggled True:
magnitude step-up / delay doubling /
icon_category change)
tomtom_incidents drop_zero_magnitude, drop_non_present
state_511_atis skipped_states (case-insensitive match against both
state_code and primary_region suffix)
central severity_thresholds (immediate_min check ordered before
priority_max so the +inf clamp still
works)
dispatcher dedup_lru_max, cooldown_prune_size,
cooldown_prune_multiplier, dedup_db_retention_days
band_conditions swpc_freshness_seconds, hamqsl_url, hamqsl_timeout_s
geocoder (photon_url/timeout/radius/limit/town_osm_values/
h3_cache_max -- module-level constants kept as
backward-compat aliases; runtime reads via accessor)
pipeline Inhibitor.ttl_seconds + Grouper.window_seconds now
default to None, falling back to
adapter_config.pipeline.{inhibitor_ttl_seconds,
grouper_window_seconds}. Explicit constructor values
still win (test fixtures unchanged).
firms confidence_floor, frp_floor, bbox, dedup_distance_m
Schema:
v7.sql adds firms_pixels.dedup_key column + drops the old hardcoded
round(lat,5) UNIQUE INDEX, replaces with UNIQUE (dedup_key, acq_time,
satellite). The firms_handler quantizes lat/lon to
(dedup_distance_m / 111000) degrees at INSERT time -- meters-based
precision per Matt s spec, tunable via the GUI without schema changes.
SCHEMA_VERSION 6 -> 7. firms_pixels has 0 rows in production so no
backfill needed.
CODE preserved (Matt s rule): sentence templates, emoji literals, the
TomTom icon_map / ITD sub_type_map / Central adapter_map / category_map
translation tables, the band_conditions Kp/SFI -> Good/Fair/Poor
heuristic, anchor-priority ordering, expires-bucket boundaries, the
NOAA G/R/S scale tables. None of these reach the GUI.
Hot-path performance: every accessor read hits the in-memory cache after
the first call; cache hit is one dict get. Per-event reads (e.g. WFIGS
cooldown_seconds on every WFIGS poll-cycle) add a single dict lookup
to existing pipelines.
Backward-compat aliases retained for module-level imports that exist in
test code: WFIGS_BROADCAST_COOLDOWN_S, FIRMS_CONFIDENCE_FLOOR,
FIRMS_FRP_FLOOR, FIRMS_BBOX_OPTIONAL, INCIDENT_FRESHNESS_MAX_S,
PHOTON_BASE_URL/TIMEOUT_S/RADIUS_KM/LIMIT. Handler code reads via
adapter_config; tests can either monkeypatch the module attribute (firms)
or mutate adapter_config DB values.
Test count: 731 -> 731 (no new tests in 3b -- handler wiring is a pure
refactor; coverage comes from the existing handler test suites passing
unchanged).
Refs audit doc v0.6-phase1-audit.md Section A + Matt s CONFIG-vs-CODE rule.
|
|||
| 68dcbc74d0 |
feat(v0.6-3a.1): trim adapter_config registry to CONFIG-only per Matt config-vs-code rule + log-on-delete safety net for orphan cleanup
Drops 35 of the v0.6-3a-draft 77 keys + adds 1 net-new key
(firms.dedup_distance_m) for a final count of 43. The trim rules:
CONFIG (lives in adapter_config, surfaces in the GUI):
where we send (channels), how often (cadences/schedules),
thresholds (magnitude floors, severity gates, distance radius,
cooldown durations, freshness windows), curation data (which
sites/states/codes), toggles (enabled, include_in_llm_context,
drop_zero_magnitude).
CODE (stays in handlers, never reaches the GUI):
sentence templates, emoji choices, mapping/translation functions
(TomTom icon_map, ITD sub_type_map, Central adapter_map and
category_map), rendering logic (anchor priority order,
expires-bucket formatting, threshold-state labels), heuristic
logic (band_conditions Kp/SFI -> Good/Fair/Poor function).
Per-adapter outcome (kept | killed):
wfigs 4 | 4 (cooldown_seconds, anchor_max_mi, two re-broadcast toggles)
nws 3 | 4 (broadcast_severities, tombstone_msgtypes, warning_suffix_promotes)
usgs_quake 6 | 3 (centroid, radius, PAGER list, 3 mag floors)
swpc 3 | 7 (three storm-tier floors)
usgs_nwis 2 | 4 (parameter_codes, broadcast_on_recede)
incident 2 | 0 (freshness_seconds, broadcast_on_update)
tomtom_incidents 2 | 1 (drop_zero_magnitude, drop_non_present)
state_511_atis 1 | 0 (skipped_states)
itd_511 0 | 3 (all sub_type maps/emoji/phrase = CODE)
central 1 | 2 (severity_thresholds)
dispatcher 4 | 0 (LRU cap, prune params, retention days)
band_conditions 3 | 6 (SWPC freshness + HamQSL endpoint config)
geocoder 6 | 1 (Photon endpoint + town-OSM curation + cache cap)
firms 4 | 1* (confidence_floor, frp_floor, bbox, dedup_distance_m)
pipeline 2 | 0 (inhibitor TTL, grouper window)
* firms: dedup_lat_lon_decimals is replaced by dedup_distance_m=5 per
Matt s call (user-facing unit is meters, not decimal places; the
handler will internally translate to quantization step in v0.6-3b).
adapter_meta stays at 15 rows -- itd_511 keeps its include_in_llm_context
toggle even with zero config keys.
Live-DB cleanup:
meshai/adapter_config/__init__.py:prune_orphans(conn) DELETEs every
adapter_config row whose (adapter, key) is no longer in REGISTRY. Each
delete is INFO-logged with the prefix "adapter_config orphan removed:"
so docker logs carry the paper trail. Called from init_db() after
seed_defaults; idempotent (zero deletes on every subsequent boot).
Cache is invalidated when any orphan is removed.
adapter_meta is NOT pruned -- meta rows are cheap and useful even for
adapters that ended up with zero config keys.
Tests (34 cases, replaces v0.6-3a 24-case set):
- Registry count is 43; ADAPTER_META is 15
- Seed lands every REGISTRY + ADAPTER_META row; idempotent; never
overwrites user edits
- prune_orphans removes a synthetic legacy row, logs at INFO with the
exact prefix, leaves known keys untouched, leaves adapter_meta
untouched, invalidates the accessor cache
- Accessor returns correctly-typed values incl new
firms.dedup_distance_m
- Guard tests: no key in REGISTRY contains "emoji", ends with "_map",
or contains "template" / "prefix" (catches CODE leaking back in)
Test count: 721 -> 731 (+10 net: +5 prune cases, +1 firms.dedup_distance_m,
+3 CODE-guard cases, +1 registry-count assertion).
Refs Matt s locked CONFIG-vs-CODE rule.
|
|||
| cb3c5aec7e |
feat(v0.6-3a): adapter_config foundation -- migration + defaults registry + typed accessor
Closes the foundation slice of audit doc Section A (Rule 17). Lands two
SQLite tables, the seed routine that populates them from a Python
defaults registry, and a typed accessor that handler code will read in
v0.6-3b. No handler changes in this commit -- ZERO behavior risk, every
existing test still passes (721 / 69 skipped / 0 failed).
v6.sql tables:
- adapter_config(adapter, key, value_json, default_json, type, description,
updated_at) PRIMARY KEY(adapter, key) -- JSON-encoded
values flow through a single column uniformly. CHECK
constraint on `type` closes the vocab (int/float/str/
bool/json).
- adapter_meta(adapter PK, display_name, include_in_llm_context,
description, updated_at) -- per-adapter metadata + the
user-scopable LLM-context toggle (Matt refinement #5).
meshai/adapter_config/ package:
- defaults.py: REGISTRY dict mapping (adapter, key) -> {default, type,
description}. Covers audit doc sections A.1-A.12: wfigs, nws,
usgs_quake, swpc, usgs_nwis, incident family (tomtom_incidents,
state_511_atis, itd_511, shared "incident"), central consumer,
dispatcher, band_conditions, geocoder, firms, pipeline (Inhibitor +
Grouper). ~85 keys total. ADAPTER_META covers 15 adapters with
display_name + include_in_llm_context defaulting to True. Per Matt
refinement #3, every default matches the current handler constant
EXACTLY -- first deploy behavior is unchanged.
- _accessor.py: AdapterConfig class with `adapter_config.<adapter>.<key>`
syntax. Read pipeline: in-memory cache hit -> SQL -> registry
fallback (with WARNING) -> AttributeError. Process-wide cache; PUT
via v0.6-3c REST API calls invalidate_cache() to drop the cache.
GIL-atomic dict reads on the fast path (handlers call this hot).
- __init__.py: seed_defaults(conn) -- INSERT OR IGNOREs one row per
registry entry. Idempotent, never overwrites user edits.
Wiring:
- meshai/persistence/db.py: SCHEMA_VERSION 5 -> 6, and init_db() now
calls seed_defaults() after migrations apply.
- meshai/main.py: _init_components() now calls init_db() FIRST (per
commit #1 lessons-learned: a startup-time migration is required
when handlers will rely on the new schema; lazy-on-first-handler
is fine for v4/v5 but not for v6 where handler reads start in
v0.6-3b).
- tests/conftest.py: autouse fixture now calls init_db() + clears
the accessor cache around each test, so every test gets the v6
seed AND a clean cache without per-test boilerplate.
Tests (tests/test_adapter_config_foundation.py, 24 cases):
- v6 tables exist + schema_meta at 6 + type-vocabulary CHECK enforced
- seed populates every REGISTRY + ADAPTER_META row, value_json ==
default_json on first seed, type matches
- seed is idempotent + does not overwrite user edits
- accessor returns correctly typed values for int/float/str/bool/
json list/json dict/json None
- cache hit: second read does not touch the DB (patched _load_from_db
raises, accessor still succeeds)
- invalidate_cache forces a re-read; mutated DB value wins
- registry fallback path triggers when a row is missing (with WARNING)
- unknown key raises AttributeError
- setattr blocked (writes go via the REST API in 3c)
- every default JSON round-trips cleanly; every type is in vocabulary
- ADAPTER_META covers every adapter in REGISTRY
Test count: 697 -> 721 (+24 new, 0 regressions).
v0.6-3b will wire handlers one at a time (wfigs, nws, quake, swpc, nwis,
incident, central, dispatcher, band_conditions, geocoder, firms). Per
the audit lock, defaults match exactly so each wiring step is a pure
refactor -- bisect-safe.
v0.6-3c lands the /api/adapter-config CRUD + the AdapterConfig.tsx
dashboard editor + cache invalidation on PUT.
Refs audit doc v0.6-phase1-audit.md Section A + finding #4.
|
