refactored-recon/phases/phase-3-transcript-processor.md

Phase 3: First End-to-End Test (Transcript Processor)

Executed: 2026-04-14T15:40Z UTC

---

Backup

Item	Location	MD5 Hash
recon.db (pre-Phase 3)	CT 130: /tmp/recon.db.phase2.20260414.bak	20ec1fec2247a999e7d42f6a716481b0
Test row SQL backup	CT 130: /tmp/recon_phase3_testrow_172f39ae.sql	—

---

What Was Created

1. lib/dispatcher.py — dispatch_once() one-shot dispatcher

Scans each configured acquired/<subfolder>/ for stable content+sidecar pairs (.txt + .meta.json sharing a basename). When a pair has been stable for pipeline.mtime_stability_seconds, hands it to the registered processor's pre_flight().

• Dynamically imports processor modules via importlib
• Unknown/missing processors log an error and skip (don't crash)
• Returns list of result dicts from all processor calls

2. lib/processors/transcript_processor.py — pre_flight()

Handles transcript content from acquired/stream/:

1. Reads content file, computes MD5 hash
2. Deduplicates against catalogue by hash
3. Reads .meta.json sidecar
4. Moves pair to processing/{hash}/ (renames content to transcript.txt)
5. Splits raw text into page_NNNN.txt files using chunk_text() (2000 words/page)
6. Registers in catalogue + documents tables
7. Sets documents.text_dir to the processing directory
8. Sets documents.page_count from page split
9. Advances status to extracted

TODO (flagged for later phases): Level-4 name deduplication not implemented for transcripts.

3. lib/utils.py — resolve_text_dir()

Helper function that resolves the text directory for a document:

• If documents.text_dir is set, use that
• Otherwise fall back to legacy config['paths']['text']/{hash}/

4. Package scaffolding

• lib/processors/__init__.py — empty
• lib/acquisition/__init__.py — empty

---

What Changed in Existing Code

lib/enricher.py (line 28, 349)

• Added: from .utils import resolve_text_dir
• Changed: text_dir = os.path.join(config['paths']['text'], file_hash) → text_dir = resolve_text_dir(file_hash, config, db)

lib/embedder.py (line 24, 278)

• Added: from .utils import resolve_text_dir
• Changed: text_dir = os.path.join(config['paths']['text'], file_hash) → text_dir = resolve_text_dir(file_hash, config, db)

Both changes are minimal and additive — the resolve_text_dir() function falls back to the legacy path when text_dir is NULL, so existing documents are unaffected.

---

End-to-End Test

Test transcript

Field	Value
Hash	172f39ae7fc6f5b02e0fabcea450c0e4
Title	Welcome to YouTube Memberships!
Channel	stefan-sobkowiak-miracle-farms
Source	stream.echo6.co
Duration	331 seconds
Pages	1 (3,747 bytes)

Test workflow

Copy-and-unprocess approach: Concatenated the existing page_0001.txt back to raw text, staged in acquired/stream/ with meta.json sidecar, deleted existing catalogue+documents rows (backup in /tmp/).

Pipeline execution (all manual one-shot calls):

Step	Command	Result
1. Dispatch	dispatch_once()	action: extracted — pair found, stable, routed to transcript_processor
2. Enrich	enrich_single(hash, db, config, rotator)	True — text_dir resolved correctly, Gemini returned 0 concepts
3. Embed	embed_single(hash, db, config)	True — 0 concepts → status=complete, 0 vectors
4. File	file_processed_item(hash, source_path, db, config)	action: skip_unclassified — no domain from empty concepts

Path traversal

acquired/stream/172f39ae...txt + .meta.json
  → processing/172f39ae.../transcript.txt + meta.json + page_0001.txt
  → (enriched, 0 concepts → skip_unclassified, not filed to library)

What skip_unclassified means

The transcript is a YouTube Memberships announcement with no extractable knowledge concepts. Gemini correctly returned 0 concepts, and the filing function correctly refused to classify it. This is the expected behavior for low-content transcripts.

---

Bug Found and Fixed During Testing

page_count NULL in documents table: The queue_document() method copies filename/path/size from catalogue but doesn't set page_count. The enricher then fails at line 451: doc.get('page_count', 0) >= 3 — .get() returns None when the key exists with a NULL value, not the default 0.

Fix: Transcript processor now sets page_count alongside text_dir in the documents UPDATE. Commit f69c04a.

---

Verification

Check	Result
catalogue rows	29,812 (removed 1 old, added 1 new)
documents rows	29,812 (same)
recon.service	inactive
recon-watchdog.service	inactive
processing/ dir	172f39ae... with transcript.txt, meta.json, page_0001.txt
acquired/stream/	empty (pair consumed)
text_dir set	/opt/recon/data/processing/172f39ae7fc6f5b02e0fabcea450c0e4
Import tests	all 6 modules pass (dispatcher, transcript_processor, filing, enricher, embedder, resolve_text_dir)

---

Commits

matt/recon (refactor branch):

Hash	Message
66fadb7	Phase 3: dispatcher, transcript processor, text_dir resolution
f69c04a	Phase 3: fix page_count in transcript processor

---

Original text directory preserved

The original /opt/recon/data/text/172f39ae7fc6f5b02e0fabcea450c0e4/ directory was NOT deleted (copy-not-destroy approach). The original concept directory was removed during testing to allow re-enrichment, and the new concept output is in data/concepts/172f39ae.../window_0001.json (empty concepts, as expected).