mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 10:51:36 -04:00
1acb148c90
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.
single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.
The native path and the Archive docker container now run single-file with
"deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.
Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind. The Archive
downloader now has a ten minute timeout. Install scripts install Deno
(previously only upgrade.sh and the image builds did).
* Singlefile writes to a file instead of dumping to stdout.
Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives). single-file now writes its result to a
temporary file which WROLPi reads back.
As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.
* Compressed Archive screenshots and readability images are no longer broken.
The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths. Screenshots are now taken from the fully
extracted ZIP so those references resolve. Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).
* Compressed Archives are created by converting the uncompressed capture.
The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed). When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file. The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages. If the
conversion fails, the uncompressed singlefile is stored instead.
* Consolidating duplicate compressed-singlefile tests.
The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one. The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.
* Addressing review comments: removeprefix, PermissionError when killing.
* Bumping version to 0.24.1-beta.
* Install scripts assert the deno.zip hash to avoid malicious or invalid files.
The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.
* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| api | ||
| app | ||
| archive | ||
| controller | ||
| help | ||
| samba | ||
| web | ||
| zim | ||
| README.md | ||
WROLPi Docker
Development and testing environment using Docker containers.
Quick Start
git clone https://github.com/lrnselfreliance/wrolpi.git
cd wrolpi
git submodule update --init
docker-compose build --parallel
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
docker-compose up -d db
docker-compose run --rm api db upgrade
docker-compose up
Browse to https://localhost:8443
Services
| Service | Port(s) | Description |
|---|---|---|
| web | 8443 (HTTPS), 8084-8086 | Caddy reverse proxy |
| api | 8081 | Sanic API server |
| app | (via web) | React frontend |
| controller | 8080 | System management (FastAPI) |
| db | 5432 | PostgreSQL 12 |
| archive | 8083 | SingleFile/Readability |
| zim | (via web:8085) | Kiwix |
| help | (via web:8086) | MkDocs documentation |
| map | (via web:8084) | OpenStreetMap tile server |
TLS Certificates
WROLPi uses a two-tier PKI for HTTPS. A root CA is generated once and persists in the media directory. A leaf certificate is regenerated on every container start to pick up hostname and IP changes.
How it works
- On first start, the
webcontainer generates a root CA at/opt/media/config/ssl/. - A leaf cert signed by the root CA is created at
/etc/ssl/caddy/inside the container. - Caddy serves HTTPS on ports 443, 8084, 8085, and 8086 using the leaf cert.
- The Controller serves on port 80 with a certificate trust banner when needed.
Trusting the certificate
- Browse to
http://<host-ip>to see the Controller. If the HTTPS certificate is not yet trusted, a banner will appear with the download link and platform-specific trust instructions. - Download and install the root CA on your device.
- All HTTPS ports will work without browser warnings.
The root CA only needs to be trusted once. Leaf certs can be regenerated (hostname change, container rebuild) without needing to re-trust.
Name Constraints
The root CA is restricted to local/private networks only via X.509 Name Constraints. Even if the CA key were compromised, it cannot sign certificates for public domains.
Permitted:
- DNS:
*.local,localhost - IPv4:
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 - IPv6:
::1,fc00::/7(unique local),fe80::/10(link-local)
EXTRA_SANS
The Docker container cannot discover the host's real LAN IP. Set EXTRA_SANS in a
.env file so the leaf certificate includes your host's IP and hostname:
# .env
EXTRA_SANS=IP:10.0.0.5,DNS:myhost.local
Each entry must have a type prefix (IP: or DNS:). Multiple entries are
comma-separated. After changing EXTRA_SANS, restart the web container:
docker compose up -d --build web
Regenerating the root CA
If you need to regenerate the root CA (e.g. after updating the certificate generation script), delete the old CA files and rebuild:
rm test/config/ssl/ca.crt test/config/ssl/ca.key
docker compose up -d --build web
All devices will need to re-download and re-trust the new CA from
http://<host-ip>/ca.crt.
Volumes
The test/ directory is mounted to /opt/media inside containers, serving as the
media directory for development. Certificate files, configs, and downloaded content
are stored here.
Two external volumes are required for the map service:
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
Environment Variables
| Variable | Default | Description |
|---|---|---|
MEDIA_DIRECTORY |
/opt/media |
Media directory inside containers |
EXTRA_SANS |
(empty) | Additional SANs for the TLS leaf certificate |
WEB_HOST |
0.0.0.0 |
Host bind address for the web service |
WEB_HTTPS_PORT |
8443 |
HTTPS port |
REACT_APP_API |
0.0.0.0:8081 |
API bind address |
UID |
1000 |
User ID for api/app containers |
GID |
1000 |
Group ID for api/app containers |
Upgrading
See UPGRADE.md for upgrade instructions.