mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 02:48:38 -04:00
b810ab2fc0
Scraped pages can carry half an emoji in their titles; sqlite3 refuses to store lone surrogates, so one poisoned title failed archive_modeler's batch commit and ended the whole modeling run on 10.0.0.9. Every archive after it stayed unmodeled, and cleanup then deleted their 'empty' domain collections (1,670 -> 578). Strip surrogates at every point Archive ingests scraped strings (titles, author, description, url, article text), and make the modeler skip a batch whose commit fails instead of dying. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| test | ||
| __init__.py | ||
| api.py | ||
| conftest.py | ||
| errors.py | ||
| lib.py | ||
| models.py | ||
| schema.py | ||