Commit graph

28 commits

Author SHA1 Message Date
lrnselfreliance
f1270b4e6c
archive: raise archive service request limit to 500MB (#692)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Uploaded singlefiles are base64-encoded into JSON before being sent to the
archive container, so an 88MB upload becomes a ~118MB request.  Sanic's default
REQUEST_MAX_SIZE of 100MB rejected it with a 413 before the handler ran, and the
API surfaced this as KeyError: 'singlefile'.

Only the Docker deployment is affected; Debian/RPi extract in-process.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-01 20:09:35 -06:00
lrnselfreliance
1acb148c90
Single file deno (#478)
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.

single-file-cli is designed for Deno.  Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.

The native path and the Archive docker container now run single-file with
"deno run".  Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum).  Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.

Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind.  The Archive
downloader now has a ten minute timeout.  Install scripts install Deno
(previously only upgrade.sh and the image builds did).

* Singlefile writes to a file instead of dumping to stdout.

Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives).  single-file now writes its result to a
temporary file which WROLPi reads back.

As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.

* Compressed Archive screenshots and readability images are no longer broken.

The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths.  Screenshots are now taken from the fully
extracted ZIP so those references resolve.  Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).

* Compressed Archives are created by converting the uncompressed capture.

The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed).  When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file.  The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages.  If the
conversion fails, the uncompressed singlefile is stored instead.

* Consolidating duplicate compressed-singlefile tests.

The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one.  The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.

* Addressing review comments: removeprefix, PermissionError when killing.

* Bumping version to 0.24.1-beta.

* Install scripts assert the deno.zip hash to avoid malicious or invalid files.

The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.

* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:28 -06:00
lrnselfreliance
0a776d030f
Archives can be compressed (SingleFileZ self-extracting HTML) (#477)
* Archives can be compressed (SingleFileZ self-extracting HTML).

A new compress_singlefile download setting runs single-file with
--compress-content, creating a self-extracting HTML file which is also a
valid ZIP.  Only the singlefile .html is compressed; readability files are
always written uncompressed.  Readability, screenshots, titles, and URLs
are extracted from the index.html inside the ZIP.  A Compress checkbox in
the Archive and RSS download forms remembers the user's choice in local
storage.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 19:46:35 -06:00
Roland
1ed0d63bac Creating Generate Screenshot button for Archives which are missing a screenshot. 2025-12-06 15:11:37 -07:00
Roland
5bd71f9664
Adding timeouts to archive docker container. 2025-05-14 17:09:53 -06:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
86355b664d
Using older version of single-file-cli to avoid error. 2024-05-14 20:29:57 -06:00
Roland
2924867f55
Bumping single-file-cli version for all systems. 2024-05-02 09:09:21 -06:00
Roland
73cb142ae1
Reverting to older singlefile container to keep archive screenshot ratio. Using FileGroup.url on ArchivePage. 2023-10-31 10:20:45 -06:00
Roland
097c3143e1
Taking screenshot of singlefile during archiving, then URL if that fails. 2023-07-24 17:17:40 -06:00
Roland
8d0f30566f Updating Sanic requirement for Archive container. 2023-04-28 13:56:21 -06:00
Roland
46cf28dee8
Restoring Archive docker container. Allowing readability to fail. 2023-04-01 15:50:11 -06:00
Roland
f3b493a5a8
Wrapping URL in archive download commands. 2022-10-25 15:07:17 -06:00
Roland
9a47135c99
Using asyncio subprocess for all archive service calls. Returning more informative errors when archive service fails. 2022-10-04 13:15:45 -06:00
Roland
c235ef65f3 Monitoring bandwidth for 2 seconds when running status main.
Updating archive's single-file call to use asyncio as well as fixing JSON error that used to be thrown.

Trying two place for single-file executable.

Adding more docs to bandwidth functions.  Extracting reused functions out.  Testing worker.

Ordering bandwidth statuses by their name.

Breaking down bandwidth by interface.

Splitting download buttons to avoid overflow.

Creating bandwidth worker which tracks server bandwidth usage to display on the status page.

Testing get_bandwidth_info() with no psutil.

Using psutil to fetch status.  Adding bandwidth status.
2022-10-03 16:44:56 -06:00
Roland
e80f24f690
Moving periodic download logic into manager so the manager can call it when it is enabled. 2022-09-25 17:27:48 -06:00
dependabot[bot]
b838164a9f
Bump sanic from 21.9.1 to 21.12.2 in /docker/archive
Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
Roland
f9b98efa8b
Removing /etc/hosts because this does not work in docker. 2022-02-04 10:37:59 -07:00
Roland
7244f2c557
Mounting test directory to api's /media/wrolpi. Updating Archive's Sanic to avoid depreciation error with uvloop. 2022-01-22 19:54:46 -07:00
Roland
b33dabcbb0
Adding comment about logs. 2021-11-15 10:35:49 -07:00
Roland
9a94de0314
Increasing archive response timeout. 2021-11-12 12:38:54 -07:00
Roland
aa1e2f266e
Returning error when archive failed. 2021-11-10 11:19:07 -07:00
Roland
40f493ee68
Checking that file exists before checking size. 2021-11-10 11:16:10 -07:00
Roland
90d3b8ca65
Moving size check into try/except to catch FileNotFoundError. 2021-11-10 10:20:15 -07:00
Roland
d6769c4922
Compressing archive bytes before response. 2021-11-10 10:11:02 -07:00
Roland
7807c44fad
Installing curl into archive container so health check works. 2021-11-09 13:27:22 -07:00
Roland
d9fc3e3d0d
Blocking ads using /etc/hosts for archive. 2021-11-08 15:06:17 -07:00
Roland
9b899f6fc0
A new start
Signed-off-by: Roland <roland@learningselfreliance.com>
2021-11-01 16:43:09 -06:00