mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 19:01:39 -04:00
1f07b5f944
* Readability extraction no longer takes minutes on image-heavy pages SingleFile inlines every image as a base64 data URI, and readability-extractor (jsdom) slows to minutes on a page carrying megabytes of them, so these pages hit the readability timeout and were saved without readability text. Replace each base64 payload with a short placeholder before running readability-extractor, then restore the payloads in its output. Applied in both the archive container and the native (Raspberry Pi/Debian) path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data URI placeholders cannot be forged by page text The placeholder was a fixed string, so a page containing placeholder-like text could index past the payloads (IndexError, failing the archive on the native path) or repeat a placeholder to copy a large image into the output many times. Each extraction now uses a random token, and text that is not a known placeholder is left as-is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data URI placeholders survive Readability's lazy-image cleanup Readability's _fixLazyImages removes an <img> src whose base64 payload is shorter than 133 characters when another attribute names an image file (alt="cat.png", title, a leftover data-src), treating it as a lazy-load placeholder. The ~50 character placeholder triggered this, so those images lost their inline data. The token is now 160 hex characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_api.py | ||
| test_archive_downloader.py | ||
| test_archive_modeler_loop.py | ||
| test_collection_id_bug.py | ||
| test_compressed_singlefile.py | ||
| test_config_import_scenarios.py | ||
| test_domain_tagging.py | ||
| test_domains_config.py | ||
| test_lib.py | ||
| test_models.py | ||