# Phase 5b: Transcript Unprocess

**Executed:** 2026-04-14T17:35–18:00Z UTC

---

## Backup

| Item | Location | MD5 Hash |
|------|----------|----------|
| recon.db (pre-Phase 5b) | CT 130: `/tmp/recon.db.phase5b.20260414.bak` | `ba114d3d733f15c9c292f7daca1052b9` |
| recon.db (pre-Phase 5b) | cortex: `/tmp/recon.db.phase5b.20260414.bak` | `ba114d3d733f15c9c292f7daca1052b9` |
| Qdrant baseline | cortex:6333 `recon_knowledge_hybrid` | status=green, 2,320,710 points |
| Unprocess list | CT 130: `/tmp/transcript_unprocess_list.20260414.json` | — |

---

## What This Phase Does

Takes the 2,259 transcripts flagged `skip_unclassified_phase5a` (no viable domain classification) and resets them to virgin state for re-processing by the refactored pipeline:

1. **Stage:** Concatenate page files into `acquired/stream/{hash}.txt` + copy `meta.json` as `{hash}.meta.json`
2. **Delete:** Remove DB rows (catalogue + documents), Qdrant vectors, source directories, and concepts directories

End state: 2,259 content+sidecar pairs sitting in the hopper, ready for the dispatcher to pick up in Phase 5c.

---

## Discovery: Skip Entries Have Qdrant Vectors

Pre-flight revealed that all 2,259 skip_unclassified entries had `vectors_inserted > 0` in the DB (total: 11,450 points, avg 5.1/doc). These were embedded by the original pipeline before classification was attempted.

Confirmed with Qdrant count queries — the vectors exist in the collection. These must be deleted alongside DB rows to avoid orphaned data. Qdrant point deletion was added to the execution script.

---

## Plan Summary

| Metric | Count |
|--------|-------|
| Transcripts to unprocess | 2,259 |
| Unique channels | 113 |
| Total page files | 3,444 |
| Total Qdrant vectors to delete | 11,450 |

---

## Execution

### Section 2: Stage Hopper

Copied all 2,259 transcripts to `acquired/stream/` in 23 chunks of 100.

- **Time:** 0.4 seconds (5,462 entries/sec)
- **Errors:** 0
- **Result:** 2,259 `.txt` + 2,259 `.meta.json` in hopper

### Section 3: Delete Sources, DB Rows, Qdrant Vectors

Processed in 23 chunks of 100 with DB count and Qdrant health verification at each boundary.

- **Time:** 2.2 seconds (1,017 entries/sec)
- **Errors:** 0
- **DB rows deleted:** 2,259 catalogue + 2,259 documents
- **Qdrant points deleted:** 11,450 (2,320,710 → 2,309,260)
- **Source directories deleted:** 2,259
- **Qdrant stayed green/ok throughout**

#### NFS Root Squash Issue

Initial deletion script ran as root (via sudo). NFS root_squash prevented root from deleting zvx-owned files under `_sources/streamecho6/`. The `shutil.rmtree(ignore_errors=True)` silently ate the permission errors.

**Fix:** Ran a separate deletion script as the zvx user. All 2,259 source dirs deleted successfully. Also removed all 131 now-empty channel directories.

---

## Post-Execution Verification

| Check | Expected | Actual |
|-------|----------|--------|
| Catalogue count | 27,553 | 27,553 |
| Documents count | 27,553 | 27,553 |
| skip_unclassified remaining | 0 | 0 |
| Orphaned documents | 0 | 0 |
| Orphaned catalogue | 0 | 0 |
| Hopper .txt files | 2,259 | 2,259 |
| Hopper .meta.json files | 2,259 | 2,259 |
| Hopper spot check (10 random) | All valid | 10/10 OK |
| _sources/streamecho6 transcript dirs | 0 | 0 |
| _sources/streamecho6 channel dirs | 0 | 0 |
| Qdrant status | green/ok | green/ok |
| Qdrant points | 2,309,260 | 2,309,260 |
| Qdrant spot check (5 random deleted) | 0 points each | 5/5 confirmed |
| Services | inactive | inactive |

---

## Anomalies

- **NFS root_squash:** `sudo python3` couldn't delete zvx-owned files on NFS mount. Required a second pass running as zvx. `shutil.rmtree(ignore_errors=True)` masked the failure — caught during verification.
- **Vectors existed (correction):** Initial assumption was 0 Qdrant vectors for these entries. All 2,259 actually had vectors (11,450 total). Deletion added to execution scope.

---

## No Code Changes

Phase 5b is pure data migration. No files in the recon repo were modified or committed.