1f07b5f944
* Readability extraction no longer takes minutes on image-heavy pages SingleFile inlines every image as a base64 data URI, and readability-extractor (jsdom) slows to minutes on a page carrying megabytes of them, so these pages hit the readability timeout and were saved without readability text. Replace each base64 payload with a short placeholder before running readability-extractor, then restore the payloads in its output. Applied in both the archive container and the native (Raspberry Pi/Debian) path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data URI placeholders cannot be forged by page text The placeholder was a fixed string, so a page containing placeholder-like text could index past the payloads (IndexError, failing the archive on the native path) or repeat a placeholder to copy a large image into the output many times. Each extraction now uses a random token, and text that is not a known placeholder is left as-is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data URI placeholders survive Readability's lazy-image cleanup Readability's _fixLazyImages removes an <img> src whose base64 payload is shorter than 133 characters when another attribute names an image file (alt="cat.png", title, a leftover data-src), treating it as a lazy-load placeholder. The ~50 character placeholder triggered this, so those images lost their inline data. The token is now 160 hex characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api | ||
| app | ||
| archive | ||
| controller | ||
| help | ||
| samba | ||
| web | ||
| zim | ||
| README.md | ||
WROLPi Docker
Development and testing environment using Docker containers.
Quick Start
git clone https://github.com/lrnselfreliance/wrolpi.git
cd wrolpi
git submodule update --init
docker-compose build --parallel
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
docker-compose run --rm api db upgrade
docker-compose up
Browse to https://localhost:8443
Services
| Service | Port(s) | Description |
|---|---|---|
| web | 8443 (HTTPS), 8084-8086 | Caddy reverse proxy |
| api | 8081 | Sanic API server |
| app | (via web) | React frontend |
| controller | 8080 | System management (FastAPI) |
| archive | 8083 | SingleFile/Readability |
| zim | (via web:8085) | Kiwix |
| help | (via web:8086) | MkDocs documentation |
| map | (via web:8084) | OpenStreetMap tile server |
TLS Certificates
WROLPi uses a two-tier PKI for HTTPS. A root CA is generated once and persists in the media directory. A leaf certificate is regenerated on every container start to pick up hostname and IP changes.
How it works
- On first start, the
webcontainer generates a root CA at/opt/media/config/ssl/. - A leaf cert signed by the root CA is created at
/etc/ssl/caddy/inside the container. - Caddy serves HTTPS on ports 443, 8084, 8085, and 8086 using the leaf cert.
- The Controller serves on port 80 with a certificate trust banner when needed.
Trusting the certificate
- Browse to
http://<host-ip>to see the Controller. If the HTTPS certificate is not yet trusted, a banner will appear with the download link and platform-specific trust instructions. - Download and install the root CA on your device.
- All HTTPS ports will work without browser warnings.
The root CA only needs to be trusted once. Leaf certs can be regenerated (hostname change, container rebuild) without needing to re-trust.
Name Constraints
The root CA is restricted to local/private networks only via X.509 Name Constraints. Even if the CA key were compromised, it cannot sign certificates for public domains.
Permitted:
- DNS:
*.local,localhost - IPv4:
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 - IPv6:
::1,fc00::/7(unique local),fe80::/10(link-local)
EXTRA_SANS
The Docker container cannot discover the host's real LAN IP. Set EXTRA_SANS in a
.env file so the leaf certificate includes your host's IP and hostname:
# .env
EXTRA_SANS=IP:10.0.0.5,DNS:myhost.local
Each entry must have a type prefix (IP: or DNS:). Multiple entries are
comma-separated. After changing EXTRA_SANS, restart the web container:
docker compose up -d --build web
Regenerating the root CA
If you need to regenerate the root CA (e.g. after updating the certificate generation script), delete the old CA files and rebuild:
rm test/config/ssl/ca.crt test/config/ssl/ca.key
docker compose up -d --build web
All devices will need to re-download and re-trust the new CA from
http://<host-ip>/ca.crt.
Volumes
The test/ directory is mounted to /opt/media inside containers, serving as the
media directory for development. Certificate files, configs, and downloaded content
are stored here.
Two external volumes are required for the map service:
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
Environment Variables
| Variable | Default | Description |
|---|---|---|
MEDIA_DIRECTORY |
/opt/media |
Media directory inside containers |
EXTRA_SANS |
(empty) | Additional SANs for the TLS leaf certificate |
WEB_HOST |
0.0.0.0 |
Host bind address for the web service |
WEB_HTTPS_PORT |
8443 |
HTTPS port |
REACT_APP_API |
0.0.0.0:8081 |
API bind address |
UID |
1000 |
User ID for api/app containers |
GID |
1000 |
Group ID for api/app containers |
Upgrading
See UPGRADE.md for upgrade instructions.