# Stream B — Post-Migration Validation Report

**Date:** 2026-04-13
**Pipeline version:** new_pipeline.py (Stream B v1, with 2 hotfixes applied during testing)

---

## Executive Summary

Both validation tasks passed. The Stream B pipeline is operational:
- **Task A (Aurora RAG):** All 3 queries returned correct Civil Organization results with updated download_urls. Migration has not broken RAG retrieval.
- **Task C (Watchdog Ingest):** Full two-phase ingest lifecycle validated end-to-end: acquire → extract → enrich → embed → place → reverse → re-place. Two bugs found and fixed during testing.

---

## Task A — Aurora RAG Retrieval Validation

**Verdict: PASS**

| Test | Result |
|------|--------|
| Query 1: Community governance principles | Relevant Civil Org results returned |
| Query 2: Emergency preparedness organization | Relevant Civil Org results returned |
| Query 3: Dispute resolution frameworks | Relevant Civil Org results returned |
| Download URL resolution (5 tested) | All 5 resolve to files on disk |
| Qdrant vectors have updated paths | YES |
| original_filename populated | YES |

**Conclusion:** The Phase 4 migration of 80 Civil Organization files has not degraded RAG quality. Qdrant vectors correctly reference the new standardized file paths.

---

## Task C — Watchdog Two-Phase Ingest Test

**Verdict: PASS**

### Test Document
- **Input:** `TestDoc_Civil_Governance_Framework_2024.pdf` (2,480 bytes, generated via reportlab)
- **Hash:** `346a65d9d72550df64490ad8e9998622`
- **Enriched title:** "Civil Governance Framework Analysis"
- **Enriched author:** "Dr. James Mitchell"
- **Domain:** Civil Organization / Governance

### Phase A (Acquisition)
| Step | Result |
|------|--------|
| File detected in `_acquired/` | PASS |
| Moved to `_ingest/` preserving original name | PASS |
| Catalogue entry created (status=queued) | PASS |
| Documents entry created (status=queued) | PASS |

### RECON Pipeline Processing
| Stage | Time | Duration |
|-------|------|----------|
| Extract | 05:50:40 | 26s |
| Enrich (Gemini) | 05:51:05 | 25s |
| Embed (TEI/Qdrant) | 05:51:25 | 20s |
| **Total processing** | | **~71s** |

### Phase B (Library Placement)
| Step | Result |
|------|--------|
| Filename standardized from book_title | `Civil_Governance_Framework_Analysis.pdf` |
| Domain classified | Civil Organization |
| Subdomain classified | Governance |
| Collision step | 1 (base, no collision) |
| File placed in library | `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` |
| DB paths updated | PASS |
| Qdrant payloads updated (2 vectors) | PASS |
| original_filename preserved in Qdrant | PASS |
| file_operations audit entry created | PASS |

### Reverse + Re-place
| Step | Result |
|------|--------|
| Reverse moves file back to _ingest/ | PASS |
| DB/Qdrant reverted to _ingest paths | PASS |
| Re-placement produces identical result | PASS |
| file_operations tracks both operations | PASS |

---

## Bugs Found & Fixed

### Bug 1: Phase B query overwhelmed by unorganized docs

**Severity:** Blocker (Phase B would never find new ingest docs)
**Root cause:** `get_unorganized(limit=50)` returns oldest 50 unorganized docs out of 29,469 total. PeerTube transcripts fill the entire result set.
**Fix:** Added `get_ingest_pending(ingest_dir, limit)` — path-filtered query. Updated `ingest_scan()` Phase B to use it.
**Impact:** Without this fix, the watchdog Phase B would never process new acquisitions.

### Bug 2: Reverse doesn't clear organized_at

**Severity:** Minor (reverse + re-trigger workflow broken)
**Root cause:** `reverse_operation()` moved files and updated DB paths but didn't clear `organized_at`, so Phase B wouldn't re-trigger placement.
**Fix:** Added `UPDATE documents SET organized_at = NULL` to `reverse_operation()`.
**Impact:** Only affects the reverse → re-place workflow. Normal forward flow unaffected.

---

## Files Modified During Validation

| File | Changes |
|------|---------|
| `/opt/recon/lib/new_pipeline.py` | Phase B query fix + organized_at clear in reverse |
| `/opt/recon/lib/status.py` | Added `get_ingest_pending()` method |

Both fixes synced to local copies at `/home/zvx/projects/recon/lib/`.

---

## Pipeline State After Validation

| Item | State |
|------|-------|
| `new_pipeline.enabled` | false (disabled after test) |
| Watchdog process | killed |
| Test document | Left in place at `Civil-Organization/Governance/Civil_Governance_Framework_Analysis.pdf` |
| `_acquired/` | Empty |
| `_ingest/` | Empty |
| `_ingest/_duplicates/` | Empty |
| `_ingest/_failed/` | Empty |
| Total file_operations records | 82 (80 from migration + 2 from test) |
| duplicate_review records | 0 |

---

## Recommendations

1. **Ready for production:** The two-phase ingest pipeline is functional. Enable `new_pipeline.enabled: true` when ready to accept new acquisitions.

2. **Watchdog logging:** Consider calling `setup_logging('recon.pipeline')` at the start of `run_watchdog()` so logs appear in the main RECON log file even when run standalone via `recon.py pipeline watch`.

3. **Domain expansion:** The `pilot_domain: "Civil Organization"` restriction limits placement to Civil Org docs only. To enable for all domains, set `pilot_domain: null` or remove it.

4. **PeerTube organized_at:** 29,469 complete docs with `organized_at IS NULL` are mostly PeerTube transcripts. Consider bulk-setting `organized_at` for non-PDF docs to prevent the `get_unorganized()` query from growing unbounded (though the new `get_ingest_pending()` query sidesteps this issue for the pipeline).

---

## Final Verdict

**Stream B: New Library Pipeline — VALIDATED**

All components tested and operational:
- Phase A acquisition (watchdog → `_acquired/` → `_ingest/`)
- RECON pipeline integration (extract → enrich → embed)
- Phase B placement (standardized naming from book_title → collision ladder → library)
- Qdrant payload updates (download_url, filename, original_filename)
- Reverse operation (full rollback including Qdrant)
- Re-placement after reverse
- Aurora RAG retrieval (citations resolve to new paths)
- Audit trail (file_operations table)