|
|
da50e5f0b8
|
Add scraper Phase 2: smart crawl mode detection + browser fallback
- Pre-flight detection: wget + Playwright probe to auto-detect if site
needs browser rendering (JS apps, parking page redirects)
- SingleFile CLI crawl backend for JS-rendered sites
- crawl_mode column in scrape_jobs (static/browser/redirect/auto)
- API: optional crawl_mode param on submit, cleared on retry
- Config: rate_limit_delay 2.0→0.5, /api/ reject pattern, preflight
+ singlefile config sections
- Prerequisites: Node.js 22, single-file-cli, Playwright + Chromium
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
2026-04-18 18:26:43 +00:00 |
|
