matt/recon
1
0
Fork 0
mirror of https://github.com/zvx-echo6/recon.git synced 2026-05-20 06:34:40 +02:00
Code Issues Projects Releases Packages Wiki Activity

Add langdetect language filter to enricher + purge non-English ZIM articles

Browse source 
- Install langdetect package for content-level language detection
- Add _check_language() to enricher.py: reads first 1500 chars of first
  page, detects language via langdetect, skips if not in allowed list
- Configurable via config.yaml pipeline.language_filter and
  pipeline.allowed_languages (default: en only)
- Catches non-English content from ANY source (PDF, web, ZIM, PeerTube)
  before burning Gemini API quota on enrichment
- Add scan_zims retry logic (3 attempts, 2s delay) for upload handler
- Purged 6,483 stale non-English zim_articles rows from DB

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt 2026-04-17 14:37:13 +00:00
commit 6f2a1d206e
3 changed files with 67 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

4
config.yaml
View file

@ -440,3 +440,7 @@ pipeline:
text: text_processor
# mtime stability threshold for picking up files from acquired/
mtime_stability_seconds: 10
# Language filter: skip non-English content before Gemini enrichment
language_filter: true # Enable langdetect-based filtering
allowed_languages: # ISO 639-1 codes allowed through enrichment
- en