Phase 5c-1: dispatcher loop, filing worker loop, service rewire

Adds dispatch_loop() alongside dispatch_once() for service-thread use.
Adds filing_worker_loop() that watches for status=complete items in
/opt/recon/data/processing/ and files them to library/Domain/Subdomain/.

Rewires cmd_service() to run the new architecture:
- Removed: scanner_loop, peertube_scanner_loop, crawler_scheduler_loop,
  organizer_loop (all replaced by dispatcher + new filing worker)
- Kept: enrich and embed stage workers, progress, dashboard
- Kept (vestigial): extract stage worker — will be removed in Phase 6 cleanup
- Added: dispatcher loop thread, filing worker thread

Phase 5c-1 of the refactor. Service not yet started — Phase 5c-2 will do that.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt 2026-04-14 18:30:58 +00:00
commit d9aed35fd7
3 changed files with 98 additions and 242 deletions

View file

@ -142,3 +142,31 @@ def dispatch_once():
})
return results
def dispatch_loop(stop_event, db, config, interval=30):
"""Run dispatch_once() on a loop until stop_event is set.
Designed to run as a service thread. Never raises to the caller.
"""
logger.info("[dispatcher] Loop started (interval: %ds)", interval)
while not stop_event.is_set():
try:
results = dispatch_once()
if results:
actions = {}
for r in results:
a = r.get('action', 'unknown')
actions[a] = actions.get(a, 0) + 1
logger.info("[dispatcher] Dispatched %d items: %s",
len(results),
", ".join(f"{k}={v}" for k, v in sorted(actions.items())))
else:
logger.debug("[dispatcher] No items to dispatch")
except Exception as e:
logger.error("[dispatcher] Error in dispatch_once: %s", e, exc_info=True)
stop_event.wait(interval)
logger.info("[dispatcher] Loop stopped")