diff --git a/phases/phase-4-pdf-processor.md b/phases/phase-4-pdf-processor.md new file mode 100644 index 0000000..83c3cb9 --- /dev/null +++ b/phases/phase-4-pdf-processor.md @@ -0,0 +1,139 @@ +# Phase 4: PDF Processor with Layered Metadata Extraction + +**Executed:** 2026-04-14T16:40Z UTC + +--- + +## Backup + +| Item | Location | MD5 Hash | +|------|----------|----------| +| recon.db (pre-Phase 4) | CT 130: `/tmp/recon.db.phase4.20260414.bak` | `1d76f8ba0f169f9a77666af56707f71d` | +| Test row SQL backup | CT 130: `/tmp/recon_phase4_test_93aad72f.sql` | — | + +--- + +## Schema Change + +Added `metadata_provenance TEXT` column to `documents` table. Stores JSON with voted metadata fields, per-field provenance (which source won), and raw source data from all three extraction sources. + +--- + +## What Was Created + +### `lib/processors/pdf_processor.py` — `pre_flight()` + +Handles PDF content from `acquired/pdf/`. Implements a 17-step pipeline: + +1. **Hash** — MD5 of PDF content via `content_hash()` +2. **Stale cleanup** — removes pre-existing `processing/{hash}/` and `concepts/{hash}/` directories +3. **Hash dedupe** — exact content match against catalogue; removes pair if duplicate +4. **Size check** — rejects PDFs exceeding `processing.max_pdf_size_mb` (default 200MB) +5. **Open PDF** — PyPDF2 `PdfReader` with pdfinfo fallback for page count +6. **Source A** — PDF info dictionary metadata (title, author, edition, year) +7. **Source B** — Filename parsing via `clean_filename_to_title()` + regex patterns +8. **Extract first 3 pages** — for Source C input, using existing `extract_text_from_page()` fallback chain +9. **Source C** — Gemini LLM metadata extraction from first 3 pages (retries 3x with 30s backoff) +10. **Vote** — per-field voting across sources; 2+ agreement wins, else priority C > A > B +11. **Level-4 dedupe** — strict check requiring ALL FOUR fields (title, author, edition, year) present and matching an existing document +12. **Move to processing** — PDF → `processing/{hash}/source.pdf`, sidecar → `sidecar.meta.json` +13. **Full text extraction** — all pages via `extract_text_from_page()` (PyPDF2 → pdftotext → Tesseract → Gemini Vision) +14. **Write meta.json** — extraction stats, voted metadata, provenance record +15. **Register in DB** — `add_to_catalogue()` + `queue_document()` +16. **Update documents row** — sets `text_dir`, `page_count`, `book_title`, `book_author`, `metadata_provenance` +17. **Status = extracted** — advances to next pipeline stage + +### Failure Modes + +| Type | Behavior | +|------|----------| +| **Hash duplicate** | Removes pair from acquired/, returns `action='duplicate'` | +| **Content failure** (unreadable PDF) | Moves to `/mnt/library/_review/rejected_pdfs/`, returns `action='content_failure'` | +| **Level-4 duplicate** | Moves to `/mnt/library/_review/duplicate_quarantine/`, queues for human review, returns `action='level4_duplicate'` | +| **Gemini API transient** | Retries 3x with 30s backoff; continues without Source C if exhausted | +| **Oversized PDF** | Moves to rejected_pdfs, returns `action='content_failure'` | + +### Metadata Voting Example + +From the end-to-end test (`93aad72f` — hydro-electric installation): + +| Field | Source A (PDF dict) | Source B (Filename) | Source C (Gemini) | Winner | +|-------|---------------------|---------------------|-------------------|--------| +| Title | `ew-FinishHydro70g.PDF` | `Finalizing A Hydro Electric Installation Hackleman` | `Finalizing a hydro-electric installation` | gemini | +| Author | `Dave` | — | `Michael Hackleman` | gemini | +| Edition | — | — | — | null | +| Year | `2001` | — | `2001` | agreed(pdf_dict,gemini) | + +--- + +## Phase 3 Cleanup Fixes (also committed in this phase) + +### Fix 1.1: Extension preservation in `filing.py` + +`_build_target_path()` calls `sanitize_filename()` which defaults to `.pdf`. For transcripts (`.txt` files), this caused incorrect extensions. Fix: after `_build_target_path()`, replace the target extension with the source file's actual extension. + +### Fix 1.2: Back-fix soldering transcript + +One-off script renamed the filed soldering transcript from `.pdf` to `.txt` in filesystem, catalogue, documents, and Qdrant (5 points). + +### Fix 1.3: Dispatcher log noise + +`_load_processor()` now catches `ModuleNotFoundError` at DEBUG level (not ERROR). Only actual `ImportError` from broken modules logs as ERROR. + +### Fix 1.4: Stale state cleanup in transcript processor + +`pre_flight()` now removes pre-existing `processing/{hash}/` and `concepts/{hash}/` directories before processing, preventing stale concept JSONs from interfering with re-enrichment. + +### Fix 1.5: Solo content files in dispatcher + +`_find_pairs()` now has a second pass that picks up content files without a `.meta.json` sidecar, passing `meta_path=None` to the processor. + +--- + +## Directories Created + +| Path | Purpose | +|------|---------| +| `/mnt/library/_review/rejected_pdfs/` | Unreadable PDFs (0 pages, corrupt) | +| `/mnt/library/_review/duplicate_quarantine/` | Level-4 metadata-duplicate PDFs for human review | +| `/opt/recon/data/acquired/pdf/` | Intake directory for PDF dispatcher | + +--- + +## End-to-End Test + +**Test document:** `93aad72f49207f72af77b90aa7e62016` — "Finalizing a hydro-electric installation" by Michael Hackleman (12 pages, 468KB) + +### Pipeline Execution + +| Stage | Result | +|-------|--------| +| Dispatch + pre_flight | `action='extracted'`, 12/12 pages, metadata voted | +| Enrich | 26 concepts from 3 windows | +| Embed | 26 vectors inserted into Qdrant | +| File | Filed to `/mnt/library/Power-Systems/Hydroelectric-Systems/`, 35 Qdrant points updated | + +### Comparison to Baseline + +| Metric | Baseline | Phase 4 | +|--------|----------|---------| +| Status | complete | complete | +| Pages extracted | 12 | 12 | +| Concepts | 20 | 26 | +| Vectors | 20 | 26 | +| Title | Finalizing a hydro-electric installation | Finalizing a hydro-electric installation | +| Author | Michael Hackleman | Michael Hackleman | +| DB totals | 29812 | 29812 | + +Concept count difference (20 → 26) is expected — enrichment is non-deterministic. Domain classification changed from "Off-grid Systems" to "Power Systems" due to fresh concept extraction. + +--- + +## Commits + +| Hash | Message | +|------|---------| +| `9fe6a0a` | Phase 4: Phase 3 cleanup fixes | +| `96e1e64` | Phase 4: PDF processor with layered metadata extraction | + +Branch: `refactor` on `forge.echo6.co/matt/recon`