matt/central
1
0
Fork 0
mirror of https://github.com/zvx-echo6/central.git synced 2026-06-10 20:04:43 +02:00
Code Issues Projects Releases Packages Wiki Activity
184 commits 8 branches 67 tags 4.7 MiB
Commit graph

4 commits

Author SHA1 Message Date
Matt Johnson
8d193d3266 feat(telemetry-separation): dedicated /telemetry tab split from /events by adapter data_class (v0.7.4) 
PR #5 of the v0.7.x GUI rework arc. Production code; central-gui restart only
(supervisor untouched -- data_class is read only by central-gui per request).

- SourceAdapter gains a `data_class` class attr (Literal["event","telemetry"],
  default "event"). NWIS opts in as "telemetry" (continuous high-volume water
  gauges); every other adapter stays "event". The /events vs /telemetry split is
  thus registry-derived from class attrs -- no hardcoded adapter-name lists.
- routes.py refactor: `_class_adapter_names(data_class)` and a `data_class` arg
  on `_adapter_filter_options` scope the flat + domain-grouped chip/legend lists
  to a class (colors stay keyed to the FULL registry, so an adapter keeps one
  color across tabs). `_fetch_events` accepts `class_adapters` and adds an
  `adapter = ANY(...)` condition. Shared `_events_query`, `_events_page(data_class,
  base_path)` and `_events_rows_fragment(...)` back both tabs; `/events`,
  `/events/rows`, `/telemetry`, `/telemetry/rows` are thin wrappers.
- Templates parameterized with a `base_path` context var (form action, hx-get,
  hx-push-url header, clear-all redirect, JS BASE_PATH const); the `_events_rows`
  paginator macro takes `base`. Same templates serve both tabs; nav gains a
  Telemetry link.
- /events.json UNCHANGED -- the cursor path sets no `class_adapters`, so the
  subject + pagination contract is intact (TestEventsJsonSubject still passes).

Adds TestTelemetrySeparation (data_class defaults, registry split 11 event / 1
telemetry, class-scoped filter options, color stability, and the `adapter =
ANY(...)` SQL shape incl. the no-class events.json path). Updates the events
frontend tests for the base_path-parameterized templates.

Full suite: 682 passed, 1 skipped (central and unprivileged zvx, 3x each).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 07:34:08 +00:00
Matt Johnson
47e7b4f267 feat(gui-bugs): fix eonet dashboard exception + out-of-range map bbox 
Kickoff of the v0.7.x GUI rework arc. Two operator-facing bugs confirmed
live; production code, central-gui + central-supervisor restart required.

Bug 1 (eonet exception leaking to /dashboard):
The supervisor calls adapter.bump_last_seen on every dedup hit, but only
4 of 12 adapters defined it and the base class did not. Adapters that
re-emit already-published events (eonet re-lists open natural events each
poll) raised AttributeError; the supervisor published it as the adapter's
status error, which /dashboard rendered as literal text in the Last Poll
cell. Fix: add bump_last_seen to the SourceAdapter base class (guarded on
getattr(self, "_db", None)); remove the 4 now-redundant identical
overrides. Fixes all 8 affected adapters, not just eonet. Documents the
method in PRODUCER-INTEGRATION.md 4.3 (producer-doc API guard).

Bug 2 (map bbox out of valid range):
applyViewportFilter serialized raw Leaflet getEast()/getWest(), which
exceed [-180,180] when panned past the dateline at low zoom (e.g.
region_east=411.3281, region_west=-608.2031), and _parse_events_params
passed them straight to ST_MakeEnvelope. Fix (JS): normalize longitudes
into [-180,180]; when the visible span exceeds ~350 deg, omit the bbox
entirely. Fix (backend, defense in depth): _parse_events_params treats an
out-of-range or inverted envelope as "no bbox" rather than erroring or
querying a bogus envelope.

Bugs 3 (FIRMS "duplicates") and 4 (missing expand buttons) from the
planning walkthrough were investigated and refuted (FIRMS rows are
distinct fire pixels, not satellite dupes -- dropping satellite collapses
0 rows; the expand button is present and functional on main), so they are
not part of this PR.

Tests: registry-derived guard that every adapter resolves bump_last_seen +
base-method behavior test; 3 bbox-guard unit tests on _parse_events_params.
Full suite: 634 passed, 1 skipped (central and unprivileged zvx).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:38:13 +00:00
Matt Johnson
98b050b2af feat(3-K): real geocoder backends + producer-doc reframe + consumer-doc enrichment 
Second of three PRs for v0.5.0 (J shipped the framework; this fills in real
backends + documents the reframed design principle in-tree; L is the events
tab + map fix, then tag).

Backends (all satisfy GeocoderBackend; never raise, all-null on any failure):
- NaviBackend — composed Navi /api/reverse/<lat>/<lon> (name/address + timezone
  + landclass + elevation in one call). Near-passthrough: response already
  matches the canonical 9-field shape. Best-effort warmup ping (Boise) on
  construction when a loop is running; config `headers` slot for a future
  Authorization: Bearer (config-only, no code change). Default base_url
  http://192.168.1.130:8440.
- PhotonBackend — raw Photon /reverse?lat&lon&limit=1 (name/address only).
  Maps features[0].properties; postal_code <- postcode; timezone/landclass/
  elevation_m null (Navi-composed-endpoint extras).
- NominatimBackend — OSM Nominatim /reverse?format=jsonv2 (name/address only).
  Configurable rate limit (default 1/sec; 0 disables for self-hosted) +
  required User-Agent. Maps the address block; landclass/elevation_m/timezone
  null.

Registered all three in supervisor _BACKEND_REGISTRY (resolved by EnrichmentConfig
backend_class name).

Docs — design pivot now in-tree:
- PRODUCER §2 reframed: the verbatim Matt quote stays; the translation inverts.
  Central is the consumer's only data plane (consumers can't do follow-up
  lookups), so enrich deliberately and centrally, namespaced under _enriched,
  failing to null. "No enrichment" is gone.
- PRODUCER §10.1 inverted: enrichment is expected; the anti-pattern is doing it
  OUTSIDE the framework (inline in poll(), bypassing cache + _enriched
  namespacing + the never-raise safety net).
- PRODUCER new §13 Enrichment contract: Enricher / GeocoderEnricher /
  GeocoderBackend Protocols, NoOpBackend default, sqlite cache + TTL +
  cache-all-null + don't-cache-on-raise semantics, _enriched.<name> provenance,
  per-field coverage matrix (cross-checked against GEOCODER_FIELDS), and the
  landclass antimeridian known wrinkle.
- CONSUMER FIRMS section: documents the data._enriched.geocoder bundle (9
  fields), per-region coverage (US-full, non-US timezone+elevation), and the
  antimeridian landclass caveat.

Tests:
- test_navi/photon/nominatim_backend.py — happy-path field mapping, null
  handling, extra-key drop, network/timeout/non-200/malformed -> all-null
  (never raises), Nominatim rate-limit (disabled + spacing) + User-Agent.
  Env-gated live Navi smoke (NAVI_INTEGRATION_TEST=1; skipped by default — the
  192.168.1.130 endpoint isn't reachable from CT104's segment).
- test_producer_doc.py — +4: §2 verbatim quote present, §10.1 subsection exists,
  §13 names all four protocol types, §13 coverage matrix == GEOCODER_FIELDS
  (derived from code, not hardcoded).

Verification: full pytest 525 passed, 1 skipped (was 495; +30 backend +
4 doc tests, -1 the env-gated skip). grep subject_for_event/_ADAPTER_REGISTRY
clean. All three backends import + resolve via the registry.

Flagged for later (NOT done here): adapters besides FIRMS that should declare
enrichment_locations (nwis, eonet, gdacs, usgs_quake, wfigs_*) — that's PR L
scope alongside the events tab. See PR description.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:10:44 +00:00
Matt Johnson
6afe80ded3 docs(2-I): producer integration spec — docs/PRODUCER-INTEGRATION.md 
The producer-side contract for adapter authors, mirroring PR H's consumer
spec. Self-contained — readers should not need to grep the codebase to
understand what a new SourceAdapter subclass must implement.

Bakes in the Phase 2 design principle ("Central takes it all and gives it
all. It's up to the pipe to do with it what it will.") so future authors
reject enrichment / silent-drop / opinionated-translation proposals on
sight. The previously-proposed Phase 3 NWIS metadata-enrichment ticket is
called out by name as an example of what gets rejected.

12-section outline locked with PM: design principle, quick start (clone
swpc_kindex), SourceAdapter base class, settings, subject namespace,
dedup keys, StreamEntry registry, removal/fall-off, anti-patterns,
preview hook, acceptance gate.

Sibling test (tests/test_producer_doc.py) mirrors test_consumer_doc.py
discipline:
  - bidirectional == between SourceAdapter API and §4 method coverage
  - preview_for_settings contract verbatim against live docstring
  - top-level domain enumeration vs central.streams.STREAMS prefixes
  - §8 STREAMS snippet vs central.streams.STREAMS
  - anti-patterns adapter-name examples vs discover_adapters()

No hardcoded stream / adapter / domain lists anywhere in the test —
every expected value derives from central.streams,
central.adapter_discovery, or central.adapter at runtime.

Honest about the pre-existing `:` vs `|` dedup-key separator
inconsistency (swpc_alerts and swpc_protons use `|`; everyone else
uses `:`). Recommends `:` for new adapters without forcing a rename PR
on the SWPC pair (separators are persisted in cursors.db rows).

Acceptance bars:
  (a) grep -rn 'subject_for_event\|_ADAPTER_REGISTRY' src tests → empty
  (b) bidirectional override-method coverage asserted in test
  (c) tests/test_producer_doc.py → 6/6 pass
  (d) full pytest suite → 469 pass (was 463 pre-PR; +6 new)
  (e) doc length: 823 lines (within 500–1200 envelope)
  (f) code fences balanced; JSON/Python blocks parse

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 21:17:48 +00:00