Same fix as text_processor — Gemini sometimes returns the literal
string "null" instead of JSON null for empty metadata fields. The
voting logic and Gemini extraction now both treat "null" strings
as None.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 5c-2 failed because shutil.rmtree(ignore_errors=True) silently
failed to clean up root-owned legacy files in processing/{hash}/,
letting the processor proceed into a half-cleaned directory and then
crash on subsequent file writes.
Changes: removed ignore_errors=True, wrapped in try/except that logs
and re-raises, so the processor fails early and visibly if stale
cleanup fails.
Recovery from Phase 5c-2 failure.
- Add lib/processors/pdf_processor.py with full pre_flight pipeline
- Layered metadata: Source A (PDF dict), Source B (filename), Source C (Gemini)
- Field-by-field voting with provenance tracking (metadata_provenance column)
- Level-4 strict dedupe (title+author+edition+year)
- Content failures route to _review/rejected_pdfs/
- Level-4 duplicates route to _review/duplicate_quarantine/
- Full text extraction using existing extract_text_from_page fallback chain
- Schema: added metadata_provenance TEXT to documents table
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
