refactored-recon/phases/phase-5c2-service-start.md

# Phase 5c-2: Service Start & Transcript Drain

## Objective

Start `recon.service` with the refactored dispatch loop and drain the 2,259
PeerTube transcripts (unprocessed in Phase 5b) through the full pipeline:
dispatch → extract → enrich → embed → complete.

## Initial Attempt (2026-04-14 ~18:42 UTC)

### What happened

The service started cleanly — all threads (extract, enrich, embed, dispatcher,
filing, progress, dashboard) came up. The dispatcher found 2,259 `.txt` +
`.meta.json` pairs in `acquired/stream/` and began dispatching them to
`transcript_processor.pre_flight()`.

**All 2,259 items crashed** with:
```
Permission denied: '/opt/recon/data/processing/{hash}/meta.json'
```

### Root cause

The Phase 5b staging script (`stage_hopper.py`) used `shutil.copy2()` to copy
`meta.json` sidecars from `/mnt/library/_sources/streamecho6/` into the hopper.
The source files were root-owned, and `copy2` preserves permissions — so the
hopper copies were also root-owned.

When `transcript_processor.pre_flight()` ran:
1. It hashed the content file (worked — `.txt` was world-readable)
2. It ran stale cleanup: `shutil.rmtree(proc_dir, ignore_errors=True)`
3. The `ignore_errors=True` **silently failed** to remove root-owned files
   inside processing dirs that already existed from a previous (legacy)
   PeerTube scraper run
4. It tried to read the (now root-owned) `meta.json` inside the half-cleaned
   processing dir → **Permission denied**

The `ignore_errors=True` was the critical bug — it turned a loud, diagnosable
failure into a silent state corruption that cascaded into a different error
2 steps later.

### Resulting state

- 2,259 "dirty" processing dirs with root-owned `meta.json` + `transcript.txt`
  and zvx-owned `page_*.txt` (from the partially-completed page split)
- Zero DB rows for any of the 2,259 items (pre_flight crashed before
  `add_to_catalogue` / `queue_document`)
- 3 "clean" processing dirs from Phase 3/4 test items (intact, with DB rows)
- Hopper drained (all files consumed by dispatcher before crash)

---

## Recovery (2026-04-14 ~20:14 UTC)

### Pre-recovery verification

| Check | Expected | Actual |
|-------|----------|--------|
| recon.service | inactive | inactive |
| processing/ dirs | 2,262 | 2,262 |
| acquired/stream/ | empty | empty |
| catalogue rows | 27,553 | 27,553 |
| documents rows | 27,553 | 27,553 |

DB backup: `/tmp/recon.db.phase5c2recovery.20260414.bak`
(md5: `99031c0994160e3a844375a9a635d616`)

### Section 1: Classify processing dirs

- **3 PRESERVE** — have catalogue rows (Phase 3/4 test items):
  `172f39ae`, `380dbc78`, `93aad72f`
- **2,259 ORPHAN** — no catalogue or documents rows

### Section 2: Reconstitute hopper

Copied `transcript.txt` → `{hash}.txt` and `meta.json` → `{hash}.meta.json`
from each orphan processing dir back into `acquired/stream/`.

Result: 2,259 `.txt` + 2,259 `.meta.json`, all zvx-owned, 0 errors.

### Section 3: Delete orphan processing dirs

`sudo rm -rf` on all 2,259 orphan dirs (sudo required for root-owned files
on local disk). 3 preserved dirs confirmed intact.

### Section 4: Fix the bug

Replaced `shutil.rmtree(proc_dir, ignore_errors=True)` with explicit
try/except that logs and re-raises:

```python
if os.path.exists(proc_dir):
    try:
        shutil.rmtree(proc_dir)
    except Exception as e:
        logger.error("Stale cleanup failed for %s: %s", proc_dir, e)
        raise
```

Applied to both `transcript_processor.py` and `pdf_processor.py`.

### Section 5: Commit

- **Commit:** `9fa60f9` on `refactor` branch
- **Message:** "Fix: stale cleanup in processors must fail loudly on permission errors"
- **Pushed to:** `forge.echo6.co/matt/recon` (origin/refactor)

### Section 6: Retry service start

Service started at **20:15:57 UTC**. All threads came up cleanly.

Dispatcher consumed all 2,259 hopper items on first cycle (~20:16). Items
immediately began flowing through the pipeline:

| Time (UTC) | extracted | enriching | enriched | complete |
|------------|-----------|-----------|----------|----------|
| 20:17:51 | 2,137 | 16 | 31 | 27,634 |
| 20:18:51 | 2,061 | 16 | 15 | 27,717 |
| 20:19:51 | 1,987 | 16 | 3 | 27,803 |
| 20:20:51 | 1,917 | 16 | 36 | 27,840 |
| 20:21:51 | 1,849 | 16 | 26 | 27,918 |

Throughput: ~57 items/min completing. Zero errors in 5-minute stability window.

ETA for full drain: ~32 minutes from 20:22 UTC (~20:54 UTC).

Background monitor running (PID 554723), writing to
`/tmp/phase5c2recovery_initial_log.txt`.

---

## Files Modified

| File | Change |
|------|--------|
| `lib/processors/transcript_processor.py` | Stale cleanup: fail loudly |
| `lib/processors/pdf_processor.py` | Stale cleanup: fail loudly |