From 581f0017f0a10c88f8f14e548eae171b0f40c05b Mon Sep 17 00:00:00 2001 From: Matt Date: Tue, 14 Apr 2026 18:33:15 +0000 Subject: [PATCH] Phase 5c-1: service rewire (code only, not started) Co-Authored-By: Claude Opus 4.6 --- phases/phase-5c1-service-rewire.md | 111 +++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 phases/phase-5c1-service-rewire.md diff --git a/phases/phase-5c1-service-rewire.md b/phases/phase-5c1-service-rewire.md new file mode 100644 index 0000000..1e62be7 --- /dev/null +++ b/phases/phase-5c1-service-rewire.md @@ -0,0 +1,111 @@ +# Phase 5c-1: Service Loop Rewire + +**Executed:** 2026-04-14T18:10–18:30Z UTC + +--- + +## Backup + +| Item | Location | MD5 Hash | +|------|----------|----------| +| recon.db (pre-Phase 5c-1) | CT 130: `/tmp/recon.db.phase5c1.20260414.bak` | `db48369c9fd0937d1b4869196d3fc19b` | + +--- + +## What This Phase Does + +Rewires the RECON service loop (`cmd_service()` in `recon.py`) to use the new dispatcher + filing worker architecture built in Phases 2–4. **Code changes only — the service has NOT been started.** Phase 5c-2 will do the first live run. + +Three files modified, zero data changes. + +--- + +## Files Changed + +| File | Lines Added | Lines Removed | Changes | +|------|-------------|---------------|---------| +| `lib/dispatcher.py` | 28 | 0 | Added `dispatch_loop()` — service-thread wrapper around `dispatch_once()` | +| `lib/filing.py` | 57 | 0 | Added `filing_worker_loop()` — watches for `status=complete` items in processing/ | +| `recon.py` | 98 | 242 | Rewired `cmd_service()`, removed 4 old loop definitions | + +--- + +## Service Thread List: Before vs After + +| # | Before (old) | After (new) | Notes | +|---|-------------|-------------|-------| +| 1 | extract (stage_loop) | dispatcher (dispatch_loop) | **NEW** — scans acquired/ subfolders | +| 2 | enrich (stage_loop) | extract (stage_loop) | **KEPT** — vestigial, will be removed in Phase 6 | +| 3 | embed (stage_loop) | enrich (stage_loop) | **KEPT** | +| 4 | scanner (scanner_loop) | embed (stage_loop) | **KEPT** | +| 5 | peertube (peertube_scanner_loop) | filing (filing_worker_loop) | **NEW** — files completed items to library | +| 6 | crawler (crawler_scheduler_loop) | progress (progress_loop) | **KEPT** | +| 7 | organizer (organizer_loop) | dashboard (start_dashboard) | **KEPT** | +| 8 | progress (progress_loop) | — | — | +| 9 | dashboard (start_dashboard) | — | — | + +**Removed threads:** +- `scanner_loop` — scanned library tree for new PDFs; replaced by dispatcher scanning acquired/ dirs +- `peertube_scanner_loop` — ingested PeerTube transcripts; replaced by dispatcher + transcript processor +- `crawler_scheduler_loop` — crawled configured websites; disabled in config, superseded by acquisition module approach +- `organizer_loop` — filed completed docs using old `organize_document()`; replaced by `filing_worker_loop` + +**Added threads:** +- `dispatch_loop` — runs `dispatch_once()` every 30s (configurable via `service.dispatch_interval`), dispatching content from `acquired/{subfolder}/` to the appropriate processor +- `filing_worker_loop` — queries for `status='complete' AND organized_at IS NULL AND path LIKE '/opt/recon/data/processing/%'` every 30s (configurable via `service.filing_interval`), files items to `library/Domain/Subdomain/` + +**Vestigial extract worker:** The extract stage worker is kept as a no-op safety net. Processors do their own extraction inline, so the extract worker will find nothing to do. Will be removed in Phase 6 cleanup. + +--- + +## Key Design Decisions + +### Filing worker safety rail +The filing worker query includes `AND path LIKE '/opt/recon/data/processing/%'` to ensure it only touches items that went through the new dispatcher → processor pipeline. Legacy items at other paths are left alone. + +### Resilient loops +Both `dispatch_loop()` and `filing_worker_loop()` catch all exceptions from their inner logic and log + continue. Service threads never raise to the caller, preventing a single processing error from killing the thread. + +### Config-driven intervals +New config keys `service.dispatch_interval` and `service.filing_interval` control polling frequency (both default to 30s if not specified in config.yaml). + +--- + +## Verification Results + +| Check | Result | +|-------|--------| +| `py_compile recon.py` | OK | +| `py_compile lib/dispatcher.py` | OK | +| `py_compile lib/filing.py` | OK | +| `import dispatch_once, dispatch_loop` | OK | +| `import file_processed_item, filing_worker_loop` | OK | +| `import recon` | OK | +| `scanner_loop` not in cmd_service | OK | +| `peertube_scanner_loop` not in cmd_service | OK | +| `crawler_scheduler_loop` not in cmd_service | OK | +| `organizer_loop` not in cmd_service | OK | +| `dispatch_loop` in cmd_service | OK | +| `filing_worker_loop` in cmd_service | OK | +| recon.service | inactive | +| recon-watchdog.service | inactive | +| Hopper (acquired/stream/) | 2,259 pairs (unchanged) | +| DB counts | 27,553/27,553 (unchanged) | +| Qdrant points | 2,309,260 (unchanged) | + +--- + +## Service NOT Started + +This phase is code-only. The service has NOT been started. Phase 5c-2 will: +1. Start the service +2. Watch the dispatcher pick up the 2,259 hopper items +3. Monitor the pipeline processing them through enrich → embed → filing + +--- + +## Commit + +| Repo | Branch | Hash | Message | +|------|--------|------|---------| +| matt/recon | refactor | `d9aed35` | Phase 5c-1: dispatcher loop, filing worker loop, service rewire |