refactored-recon

Design documents for the RECON pipeline refactor. The goal is to restructure RECON's ingestion pipeline into a hopper-based, type-dispatched architecture where new content sources can be added by writing a small acquisition module and a small processor module without touching shared infrastructure.

This repo is design-only. Implementation happens in the RECON repo; this repo tracks the thinking, the decisions, and the phased migration plan with git history so the architecture can evolve visibly over time.

Status

• Design drafted: 2026-04-14
• Implementation status: not started
• Current system: recon.service stopped pending refactor

Documents

• architecture.md — target architecture. The hopper model, processor pattern, lifecycle, contracts.
• current-state.md — where the system is today, what works, what's broken, what's technical debt.
• migration-plan.md — phased plan to get from current to target without losing data or extended downtime.
• decisions.md — architectural decision record. The forks we considered and why we chose what we chose.
• phases/ — detailed per-phase execution plans (to be filled in as each phase is scoped).

Read order

If you're new to this design, read in this order:

1. current-state.md — understand what exists
2. architecture.md — understand the target
3. decisions.md — understand why the target looks the way it does
4. migration-plan.md — understand how we get there

Principles

Three principles shaped every decision in this design. When in doubt on a detail, fall back to these:

Modularity on the edges, uniformity in the middle. Each content source (PDFs, transcripts, HTML, future types) is its own acquisition module and its own processor. They share nothing except the enrich/embed infrastructure and the filesystem contract. Adding a new type touches only the two new modules and one line of config.

State is a directory. A file's location on disk tells you what stage of the pipeline it's in. Acquired but unprocessed → sitting in _acquired/. Being worked on → sitting in _processing/. Done → sitting in the library under its final name. No status tracking that isn't reflected in where the file actually lives.

Small atomic transitions. Files move between stages as complete units with all their metadata updated together — filesystem, catalogue, documents table, and Qdrant payloads in one transition. Partial state is the enemy. If any part of a transition fails, the file stays where it was.