Commit graph

2 commits

Author SHA1 Message Date
Roland
b810ab2fc0
Archives: an unpaired-surrogate title must not kill the modeler.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9.  Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).

Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 19:25:30 -06:00
Roland
3e3694869b
Fixing bug where not all archives were modeled. 2025-12-18 17:19:57 -07:00