mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 02:48:38 -04:00
b810ab2fc0
Scraped pages can carry half an emoji in their titles; sqlite3 refuses to store lone surrogates, so one poisoned title failed archive_modeler's batch commit and ended the whole modeling run on 10.0.0.9. Every archive after it stayed unmodeled, and cleanup then deleted their 'empty' domain collections (1,670 -> 578). Strip surrogates at every point Archive ingests scraped strings (titles, author, description, url, article text), and make the modeler skip a batch whose commit fails instead of dying. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| archive | ||
| docs | ||
| flasher | ||
| inventory | ||
| map | ||
| videos | ||
| zim | ||
| __init__.py | ||