wrolpi/wrolpi/cmd.py
lrnselfreliance 1acb148c90
Single file deno (#478)
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.

single-file-cli is designed for Deno.  Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.

The native path and the Archive docker container now run single-file with
"deno run".  Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum).  Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.

Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind.  The Archive
downloader now has a ten minute timeout.  Install scripts install Deno
(previously only upgrade.sh and the image builds did).

* Singlefile writes to a file instead of dumping to stdout.

Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives).  single-file now writes its result to a
temporary file which WROLPi reads back.

As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.

* Compressed Archive screenshots and readability images are no longer broken.

The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths.  Screenshots are now taken from the fully
extracted ZIP so those references resolve.  Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).

* Compressed Archives are created by converting the uncompressed capture.

The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed).  When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file.  The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages.  If the
conversion fails, the uncompressed singlefile is stored instead.

* Consolidating duplicate compressed-singlefile tests.

The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one.  The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.

* Addressing review comments: removeprefix, PermissionError when killing.

* Bumping version to 0.24.1-beta.

* Install scripts assert the deno.zip hash to avoid malicious or invalid files.

The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.

* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:28 -06:00

340 lines
12 KiB
Python

"""Paths for third-party commands that WROLPi may require."""
import asyncio
import dataclasses
import os
import pathlib
import shlex
import shutil
import signal
import tempfile
from asyncio import CancelledError
from pathlib import Path
from time import time
from typing import Optional
from wrolpi.common import logger, TRACE_LEVEL
from wrolpi.vars import PYTEST, DOCKERIZED
logger = logger.getChild(__name__)
def is_executable(path: Path) -> bool:
"""Returns True if the file is executable."""
return path.is_file() and os.access(path, os.X_OK)
def which(*possible_paths: str, warn: bool = False) -> Optional[Path]:
"""
Find an executable in the system $PATH. If the executable cannot be found in $PATH, then return the first
executable found in possible_paths.
Returns None if no executable can be found.
"""
found = None
possible_paths = list(map(Path, possible_paths))
for path in possible_paths:
if is_executable(path):
# `path` was an absolute file which is executable.
found = path
break
path = shutil.which(path)
if path and (path := Path(path).absolute()) and is_executable(path):
# `path` was a name or relative path.
found = path
if not found and warn and not PYTEST and not DOCKERIZED:
logger.warning(f'Cannot find executable {possible_paths[0]}')
return
elif found:
return found.absolute()
def pid_is_running(pid: int) -> bool:
"""Return True if a process with PID exists."""
try:
os.kill(pid, 0)
return True
except OSError:
return False
# Admin
SUDO_BIN = which('sudo', '/usr/bin/sudo')
NMCLI_BIN = which('nmcli', '/usr/bin/nmcli')
CPUFREQ_INFO_BIN = which('cpufreq-info', '/usr/bin/cpufreq-info')
CPUFREQ_SET_BIN = which('cpufreq-set', '/usr/bin/cpufreq-set')
# Files
WGET_BIN = which('wget', '/usr/bin/wget')
# Map
BASH_BIN = which('bash', '/bin/bash')
# Archive
SINGLE_FILE_BIN = which('single-file',
'/usr/bin/single-file', # rpi os
'/usr/local/bin/single-file', # debian
)
DENO_BIN = which('deno', '/usr/local/bin/deno')
def _find_single_file_deno_script() -> Optional[Path]:
"""Find the Deno entry script of the single-file-cli package.
npm's `single-file` bin is a symlink to the package's single-file-node.js; the Deno script
(`single-file`, shebang `deno run`) lives next to it. single-file-cli is designed for Deno —
under node 18 it cannot reach the browser CDP on IPv6-enabled hosts and never exits."""
if not SINGLE_FILE_BIN:
return None
script = SINGLE_FILE_BIN.resolve().parent / 'single-file'
try:
with script.open('rb') as fh:
if b'deno run' in fh.readline():
return script
except OSError:
pass
return None
SINGLE_FILE_DENO_SCRIPT = _find_single_file_deno_script()
CHROMIUM = which('chromium-browser', 'chromium',
'/usr/bin/chromium-browser', # rpi os
'/usr/bin/chromium', # debian
)
CHROMEDRIVER_BIN = which('chromedriver',
'/usr/bin/chromedriver', # rpi os / debian
)
FIREFOX = which('firefox', 'firefox-esr',
'/usr/bin/firefox',
'/usr/bin/firefox-esr',
)
READABILITY_BIN = which('readability-extractor',
'/usr/bin/readability-extractor', # rpi os
'/usr/local/bin/readability-extractor', # debian
)
# Known browser definitions for SingleFile
KNOWN_BROWSERS = {
'chromium': {
'paths': ['chromium-browser', 'chromium', '/usr/bin/chromium-browser', '/usr/bin/chromium'],
'name': 'Chromium',
},
'firefox': {
'paths': ['firefox', 'firefox-esr', '/usr/bin/firefox', '/usr/bin/firefox-esr'],
'name': 'Firefox',
},
'brave': {
'paths': ['brave-browser', '/usr/bin/brave-browser'],
'name': 'Brave',
}
}
def get_installed_browsers() -> list:
"""
Detect browsers installed on the system that can be used with SingleFile.
Returns a list of dicts with:
- key: The browser identifier (e.g., 'chromium', 'firefox')
- name: Human-readable name (e.g., 'Chromium', 'Firefox')
- path: Absolute path to the executable
"""
browsers = []
for key, info in KNOWN_BROWSERS.items():
path = which(*info['paths'])
if path:
browsers.append({
'key': key,
'name': info['name'],
'path': str(path),
})
return browsers
# Videos
YT_DLP_BIN = which(
'/opt/wrolpi/venv/bin/yt-dlp', # Use virtual environment location
'yt-dlp',
'/usr/local/bin/yt-dlp', # Location in docker container
)
FFPROBE_BIN = which('ffprobe', '/usr/bin/ffprobe')
FFMPEG_BIN = which('ffmpeg', '/usr/bin/ffmpeg')
# Documents
CATDOC_PATH = which('catdoc')
TEXTUTIL_PATH = which('textutil')
@dataclasses.dataclass
class CommandResult:
return_code: int
cancelled: bool
stdout: bytes
stderr: bytes
elapsed: int
pid: int = None
TESTING_RUN_COMMAND_RESULT = None
async def run_command(cmd: tuple[str | pathlib.Path, ...], cwd: pathlib.Path | str = None,
timeout: int = 600, log_command: bool = True,
stdout_callback: callable = None, env: dict = None,
start_new_session: bool = False) -> CommandResult:
"""Run a shell command, return the results (stdout/stderr/return code).
When stdout_callback is provided, stdout is piped and each line is passed to the callback
in real-time. Otherwise, stdout is written to a temporary file.
:param log_command: Enable debug logging.
:param stdout_callback: Called with each decoded stdout line (str). May be None.
:param env: Environment variables for the subprocess. If None, inherits the current environment.
:param start_new_session: Run the command in its own session/process group; when it finishes
(or is killed by timeout/cancel), the whole group is killed. Use for commands that may
leave children behind (e.g. single-file's browser).
"""
if not isinstance(cmd, (list, tuple)):
raise RuntimeError('Command must be a list or tuple')
if PYTEST and TESTING_RUN_COMMAND_RESULT:
logger.debug('run_command: returning mock result')
return TESTING_RUN_COMMAND_RESULT() # call the mock
streaming = stdout_callback is not None
with tempfile.NamedTemporaryFile() as stderr_fh:
stderr_file = pathlib.Path(stderr_fh.name)
cmd = tuple(str(i) for i in cmd)
cmd_str = ' '.join(shlex.quote(i) for i in cmd)
if log_command:
logger.info(f'run_command: running ({timeout=}): {cmd_str}')
elif __debug__ and logger.isEnabledFor(TRACE_LEVEL):
logger.trace(f'run_command: running ({timeout=}): {cmd_str}')
stdout_fh = None
stdout_file = None
reader_task = None
stdout_buf = None
pid = None
if streaming:
stdout_arg = asyncio.subprocess.PIPE
stdout_buf = bytearray()
else:
stdout_fh = tempfile.NamedTemporaryFile()
stdout_file = pathlib.Path(stdout_fh.name)
stdout_arg = stdout_fh
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=stdout_arg,
stderr=stderr_fh,
cwd=cwd,
env=env,
start_new_session=start_new_session,
)
pid = proc.pid
if streaming:
async def _read_stdout():
buf = ''
while True:
chunk = await proc.stdout.read(4096)
if not chunk:
# Flush remaining buffer.
if buf.strip():
try:
stdout_callback(buf.strip())
except Exception as e:
logger.warning(f'run_command: stdout_callback error: {e}')
break
stdout_buf.extend(chunk)
buf += chunk.decode(errors='replace')
# Split on both \n and \r to handle progress bars.
while '\n' in buf or '\r' in buf:
# Find the earliest delimiter.
ni = buf.find('\n')
ri = buf.find('\r')
if ni == -1:
idx, skip = ri, 1
elif ri == -1:
idx, skip = ni, 1
else:
idx, skip = min(ni, ri), 1
line = buf[:idx]
buf = buf[idx + skip:]
if line.strip():
try:
stdout_callback(line.strip())
except Exception as e:
logger.warning(f'run_command: stdout_callback error: {e}')
reader_task = asyncio.create_task(_read_stdout())
cancelled = False
start = time()
try:
while True:
elapsed = int(time() - start)
if timeout and elapsed >= timeout:
logger.warning(f'run_command: timeout exceeded, killing... {cmd=}')
proc.kill()
await proc.wait()
break
try:
await asyncio.wait_for(proc.wait(), timeout=1)
break
except asyncio.TimeoutError:
continue
except CancelledError as e:
logger.warning(f'run_command: cancelled, killing... {cmd=}', exc_info=e)
cancelled = True
proc.kill()
await proc.wait()
if reader_task:
reader_task.cancel()
try:
await reader_task
except (CancelledError, asyncio.CancelledError):
pass
elapsed = int(time() - start)
if streaming:
stdout = bytes(stdout_buf)
else:
stdout = stdout_file.read_bytes() or b''
stderr = stderr_file.read_bytes() or b''
if log_command:
logger.debug(
f'run_command: finished ({elapsed=}s) with stdout={len(stdout)} stderr={len(stderr)}: {cmd_str=}')
elif __debug__ and logger.isEnabledFor(TRACE_LEVEL):
logger.trace(
f'run_command: finished ({elapsed=}s) with stdout={len(stdout)} stderr={len(stderr)}: {cmd_str=}')
return CommandResult(
return_code=proc.returncode,
cancelled=cancelled,
stdout=stdout,
stderr=stderr,
elapsed=elapsed,
pid=pid,
)
finally:
if start_new_session and pid:
# The command was its own session/group leader; kill anything it left behind
# (e.g. the browser single-file spawns but fails to close).
try:
os.killpg(pid, signal.SIGKILL)
logger.info(f'run_command: killed lingering process group {pid}')
except ProcessLookupError:
pass # Nothing left behind; the common case.
except PermissionError as e:
logger.error(f'run_command: failed to kill process group {pid}', exc_info=e)
if stdout_fh:
stdout_fh.close()