wrolpi/docker/archive/Dockerfile

45 lines
2 KiB
Text
Raw Normal View History

FROM browserless/chrome:1.61.1-chrome-stable
USER root
WORKDIR /app
# Install Python for main.py
ENV PYTHONUNBUFFERED=1
Single file deno (#478) * Singlefile runs under Deno 2.8.3; browser leftovers are reaped. single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to ::1 first with no IPv4 fallback) and it never exits after a successful save, blocking the download worker forever. The native path and the Archive docker container now run single-file with "deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09 requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22 minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr that single-file never reads), so the browser is run through singlefile_browser_wrapper.sh which discards its stderr. Commands are run in their own process group which is killed when they finish, reaping the browser single-file leaves behind. The Archive downloader now has a ten minute timeout. Install scripts install Deno (previously only upgrade.sh and the image builds did). * Singlefile writes to a file instead of dumping to stdout. Deno >= 2.3 truncates large --dump-content output at exit; compressed (SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the compression check in write_archive_files, and were destroyed by the HTML prettifier (blank white archives). single-file now writes its result to a temporary file which WROLPi reads back. As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is unreadable is now stored verbatim instead of being prettified. * Compressed Archive screenshots and readability images are no longer broken. The index.html inside a compressed (SingleFileZ) singlefile references its resources by relative ZIP paths. Screenshots are now taken from the fully extracted ZIP so those references resolve. Readability images are inlined as data URIs read from the ZIP (readability-extractor absolutizes relative srcs against the page URL, so both forms are matched). * Compressed Archives are created by converting the uncompressed capture. The page is always captured uncompressed; readability, screenshot, and title come from that canonical page (its images are inline data URIs, so no ZIP extraction or image inlining is needed). When compress_singlefile is set, a second single-file run converts the captured file into a compressed (SingleFileZ) self-extracting file. The conversion is a local file:// operation - the page is not fetched again, and single-file preserves the original URL header when re-saving its own pages. If the conversion fails, the uncompressed singlefile is stored instead. * Consolidating duplicate compressed-singlefile tests. The write_archive_files tests (verbatim write, title fallback, broken-ZIP guard, readability inlining) are combined into one test; the helper probes and the setting-plumbing tests are each merged into one. The do_singlefile compression-failure scenario and the timeout constant are folded into the main do_singlefile test, which shared their setup. * Addressing review comments: removeprefix, PermissionError when killing. * Bumping version to 0.24.1-beta. * Install scripts assert the deno.zip hash to avoid malicious or invalid files. The hashes are the official sha256sum assets of the Deno v2.8.3 release, per architecture. * Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:10 -06:00
RUN apt update && apt install -y python3 python3-dev python3-venv curl unzip && ln -sf python3 /usr/bin/python
RUN python3 -m venv /app/venv
RUN . /app/venv/bin/activate
# Install single-file and Readability from the committed lockfile. Not in /usr/src/app, which is browserless's own app.
COPY npm-tools/package.json npm-tools/package-lock.json /opt/npm-tools/
COPY scripts/install_npm_tools.sh /opt/install_npm_tools.sh
RUN /opt/install_npm_tools.sh /opt/npm-tools
Single file deno (#478) * Singlefile runs under Deno 2.8.3; browser leftovers are reaped. single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to ::1 first with no IPv4 fallback) and it never exits after a successful save, blocking the download worker forever. The native path and the Archive docker container now run single-file with "deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09 requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22 minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr that single-file never reads), so the browser is run through singlefile_browser_wrapper.sh which discards its stderr. Commands are run in their own process group which is killed when they finish, reaping the browser single-file leaves behind. The Archive downloader now has a ten minute timeout. Install scripts install Deno (previously only upgrade.sh and the image builds did). * Singlefile writes to a file instead of dumping to stdout. Deno >= 2.3 truncates large --dump-content output at exit; compressed (SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the compression check in write_archive_files, and were destroyed by the HTML prettifier (blank white archives). single-file now writes its result to a temporary file which WROLPi reads back. As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is unreadable is now stored verbatim instead of being prettified. * Compressed Archive screenshots and readability images are no longer broken. The index.html inside a compressed (SingleFileZ) singlefile references its resources by relative ZIP paths. Screenshots are now taken from the fully extracted ZIP so those references resolve. Readability images are inlined as data URIs read from the ZIP (readability-extractor absolutizes relative srcs against the page URL, so both forms are matched). * Compressed Archives are created by converting the uncompressed capture. The page is always captured uncompressed; readability, screenshot, and title come from that canonical page (its images are inline data URIs, so no ZIP extraction or image inlining is needed). When compress_singlefile is set, a second single-file run converts the captured file into a compressed (SingleFileZ) self-extracting file. The conversion is a local file:// operation - the page is not fetched again, and single-file preserves the original URL header when re-saving its own pages. If the conversion fails, the uncompressed singlefile is stored instead. * Consolidating duplicate compressed-singlefile tests. The write_archive_files tests (verbatim write, title fallback, broken-ZIP guard, readability inlining) are combined into one test; the helper probes and the setting-plumbing tests are each merged into one. The do_singlefile compression-failure scenario and the timeout constant are folded into the main do_singlefile test, which shared their setup. * Addressing review comments: removeprefix, PermissionError when killing. * Bumping version to 0.24.1-beta. * Install scripts assert the deno.zip hash to avoid malicious or invalid files. The hashes are the official sha256sum assets of the Deno v2.8.3 release, per architecture. * Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:10 -06:00
# Install Deno; single-file-cli is designed for it (under node 18 it cannot reach the browser CDP
# on IPv6-enabled hosts and never exits). Match the version pinned in scripts/upgrade.sh.
RUN DENO_VERSION="v2.8.3" \
&& ARCH=$(uname -m) \
&& if [ "$ARCH" = "aarch64" ]; then \
DENO_ARCH="aarch64-unknown-linux-gnu"; \
DENO_SHA256="d4589cc1ffcbf1995c92a0127d932aaf832ac70cfdcc6d5b7bf38043cf303575"; \
else \
DENO_ARCH="x86_64-unknown-linux-gnu"; \
DENO_SHA256="30455b845ffa6082209c3590269c910ad3b7efdf28c9879afd4006c47ae54197"; \
fi \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-${DENO_ARCH}.zip" -o /tmp/deno.zip \
# Assert the hash to avoid malicious or invalid files.
&& echo "${DENO_SHA256} /tmp/deno.zip" | sha256sum -c - \
&& unzip -o /tmp/deno.zip -d /usr/local/bin/ \
&& chmod +x /usr/local/bin/deno \
&& rm /tmp/deno.zip \
&& deno --version
# Install the Python wrapper and it's requirements.
Python packages install from hash-locked requirements; yt-dlp still floats (#913) * Python packages install from hash-locked requirements; yt-dlp still floats Every pip install trusted PyPI over TLS: requirements.txt pins only top-level versions (some with >=), the controller's are almost all >=, and dependencies were never pinned, so a forged certificate (as the eMudhra key compromise allowed) could change code in the API, the controller, or the archive container. scripts/lock_requirements.sh generates a universal, hash-locked .lock beside each requirements.txt, and install_python_requirements.sh installs it with --require-hashes at every site. yt-dlp is left out of the lock and upgraded afterwards so devices still get YouTube fixes; its dependencies are locked. The unpinned `pip install --upgrade pip` steps are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * The unhashed yt-dlp install cannot replace locked packages, and nothing builds from source Review of #913 found two holes. The yt-dlp step ran without --no-deps, so pip's only-if-needed strategy would still fetch an unhashed dependency whenever yt-dlp's metadata asked for a version the lock did not install. And srt is published only as a source archive, whose build installed setuptools with hash checking off on every install. The yt-dlp step now uses --no-deps (its dependencies come only from the lock), both steps are retried as the image builds did before, and installs and locks are wheels-only. srt 3.5.3 is vendored unmodified as wrolpi/vendor/srt.py (MIT), verified against the sdist hash the lock pinned; webvtt-py's SRT reader rejects WROLPi's own test file, so it is not a replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:00:29 -07:00
COPY docker/archive/requirements.txt docker/archive/requirements.lock /app/
COPY scripts/install_python_requirements.sh /opt/install_python_requirements.sh
RUN /opt/install_python_requirements.sh /app/venv/bin/pip3 /app/requirements.txt
COPY docker/archive /app
Single file deno (#478) * Singlefile runs under Deno 2.8.3; browser leftovers are reaped. single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to ::1 first with no IPv4 fallback) and it never exits after a successful save, blocking the download worker forever. The native path and the Archive docker container now run single-file with "deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09 requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22 minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr that single-file never reads), so the browser is run through singlefile_browser_wrapper.sh which discards its stderr. Commands are run in their own process group which is killed when they finish, reaping the browser single-file leaves behind. The Archive downloader now has a ten minute timeout. Install scripts install Deno (previously only upgrade.sh and the image builds did). * Singlefile writes to a file instead of dumping to stdout. Deno >= 2.3 truncates large --dump-content output at exit; compressed (SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the compression check in write_archive_files, and were destroyed by the HTML prettifier (blank white archives). single-file now writes its result to a temporary file which WROLPi reads back. As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is unreadable is now stored verbatim instead of being prettified. * Compressed Archive screenshots and readability images are no longer broken. The index.html inside a compressed (SingleFileZ) singlefile references its resources by relative ZIP paths. Screenshots are now taken from the fully extracted ZIP so those references resolve. Readability images are inlined as data URIs read from the ZIP (readability-extractor absolutizes relative srcs against the page URL, so both forms are matched). * Compressed Archives are created by converting the uncompressed capture. The page is always captured uncompressed; readability, screenshot, and title come from that canonical page (its images are inline data URIs, so no ZIP extraction or image inlining is needed). When compress_singlefile is set, a second single-file run converts the captured file into a compressed (SingleFileZ) self-extracting file. The conversion is a local file:// operation - the page is not fetched again, and single-file preserves the original URL header when re-saving its own pages. If the conversion fails, the uncompressed singlefile is stored instead. * Consolidating duplicate compressed-singlefile tests. The write_archive_files tests (verbatim write, title fallback, broken-ZIP guard, readability inlining) are combined into one test; the helper probes and the setting-plumbing tests are each merged into one. The do_singlefile compression-failure scenario and the timeout constant are folded into the main do_singlefile test, which shared their setup. * Addressing review comments: removeprefix, PermissionError when killing. * Bumping version to 0.24.1-beta. * Install scripts assert the deno.zip hash to avoid malicious or invalid files. The hashes are the official sha256sum assets of the Deno v2.8.3 release, per architecture. * Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:10 -06:00
COPY scripts/singlefile_browser_wrapper.sh /app/singlefile_browser_wrapper.sh
# Launch Archive app in the virtual environment.
ENTRYPOINT [ "/app/venv/bin/python3", "/app/main.py"]