* A secondary WROLPi copies its primary's library over HTTP
A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them. wrolpi.yaml contributes only its library
settings. Files deleted on the primary are kept.
A secondary never saves the synced configs and never downloads. Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy. It syncs in a daily window or on "Sync now"; WROL Mode stops it.
On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync has its own /admin tab; a sync's reads from the primary are bounded
The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.
A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped. A primary address may not carry a
user name or password (it is logged and shown). While a sync runs, the API
answers only the exact requests that show its progress or stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync keeps the secondary's replaced configs in today's config backup
The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Docker's Caddy logs every request
`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync reports its progress as it downloads, and refuses quietly
The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds. Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time. The primary's version is stored
without its trailing newline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A WROLPi can back up to a second drive; sync settings live in sync.yaml
The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted. The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start. A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears. Not in Docker, which
cannot reach other drives.
Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Sync tab shows the configured backup drive
getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured. It also dropped
`version`. A test now fails for any setting the API sends that getSettings
drops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup can go to a folder on its drive, and remembers its last success
The Backup Drive panel takes a folder on the chosen drive. The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused. A missing folder is created once the drive is checked.
When the last backup succeeded is saved in sync.yaml, so it is known after a
restart. "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The backup drive has its own window
The sync window was shown in the Sync panel but also scheduled the backup
drive. The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync and the backup drive can each exclude folders
A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory. Excluded
folders are never copied; what was copied before is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The folder browser does not offer lost+found
Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync never writes into config or tags, whatever their case
On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup writes the database snapshot only when the database changed
Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove the unused save_sync_config switch
Nothing activated it; SyncConfig saves directly. The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Python packages install from hash-locked requirements; yt-dlp still floats
Every pip install trusted PyPI over TLS: requirements.txt pins only top-level versions
(some with >=), the controller's are almost all >=, and dependencies were never pinned, so
a forged certificate (as the eMudhra key compromise allowed) could change code in the API,
the controller, or the archive container. scripts/lock_requirements.sh generates a
universal, hash-locked .lock beside each requirements.txt, and
install_python_requirements.sh installs it with --require-hashes at every site. yt-dlp is
left out of the lock and upgraded afterwards so devices still get YouTube fixes; its
dependencies are locked. The unpinned `pip install --upgrade pip` steps are gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The unhashed yt-dlp install cannot replace locked packages, and nothing builds from source
Review of #913 found two holes. The yt-dlp step ran without --no-deps, so pip's
only-if-needed strategy would still fetch an unhashed dependency whenever yt-dlp's metadata
asked for a version the lock did not install. And srt is published only as a source
archive, whose build installed setuptools with hash checking off on every install.
The yt-dlp step now uses --no-deps (its dependencies come only from the lock), both steps
are retried as the image builds did before, and installs and locks are wheels-only. srt
3.5.3 is vendored unmodified as wrolpi/vendor/srt.py (MIT), verified against the sdist
hash the lock pinned; webvtt-py's SRT reader rejects WROLPi's own test file, so it is not
a replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
They were installed with `npm i -g`, which pinned only the three top-level versions and
resolved their 156 dependencies fresh from the registry, so a forged TLS certificate (as
the eMudhra key compromise allowed) could change code WROLPi runs on every archive.
npm-tools/package-lock.json pins every package with its sha512, and install_npm_tools.sh
runs `npm ci` and links the commands into /usr/local/bin for every install path, upgrades,
both images and the archive container. The app container's unpinned, unused global serve
is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The install scripts, the nginx-to-Caddy upgrade, and the Pi image downloaded Caddy's apt
signing key and source list over TLS and trusted them as-is, so a forged certificate could
serve a key that signs other caddy packages. add_caddy_apt_repo.sh installs the key the
ISO already uses (fingerprint 65760C51EDEA2017CEA2CA15155B6D79CA56EA34) as an armored
keyring and names it in the source's signed-by.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
upgrade.sh downloaded Deno on every device whose Deno was older than the pinned
version and installed it without a check, so a forged TLS certificate (as the eMudhra
key compromise allowed) could replace the runtime yt-dlp uses. The zip is now checked
against the same sha256 the install scripts pin; a mismatch skips the Deno install and
the upgrade continues. A test requires every Deno download in the repo to be checked.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fail CI when vulture finds new dead code
scripts/check_dead_code.py runs vulture and compares its findings with
scripts/vulture_baseline.txt. The baseline is keyed by file and name, not
line number, so unrelated edits do not invalidate it. Class fields,
attributes, test arguments (side-effect fixtures), and alembic revision
names are not reported, because nearly all of them are false positives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* TEMP: add a dead function to prove the Dead Code check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Revert the temporary dead function
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
install.sh only accepted Debian 12, which cannot run this version, and
rejected Debian 13. It now requires Debian 13 or newer on both PCs and
Raspberry Pis. The install scripts point to the drive wizard instead of
manual mounting.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add a Solar calculator with offline NASA POWER data
Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.
The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter. "Best for the year" and "Best for winter" sweep
the tilt.
Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens. Users can replace it with their own monthly values.
Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.
Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Open the Solar calculator from the map, pins and current location
The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location. A button beside Latitude fills in the browser's current
location. Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.
The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button). It now only cleans up while the
URL is still a map page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add off-grid sizing to the Solar calculator
A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):
- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
at 12, 24 or 48 V. Outdoor lead-acid is enlarged for the cold; outdoor
lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
angle, with a button to apply the count. A PWM controller's loss (battery
voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
against the controller's limit, since cold panels exceed their rated Voc.
The coldest morning defaults to 25 C below the coldest monthly average.
All inputs live in the URL like the rest of the calculator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Bump wrolpi-help for the Solar calculator help page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Credit NASA POWER as it asks, and say how the data was changed
- The calculator gains a "Data attribution" link under the location with
NASA POWER's requested acknowledgment, the data reference (Climatology API
v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
modified the data (rounded to one byte per value, temperature averaged
into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
beside the data, with the API version taken from NASA's responses and the
date the tiles were downloaded. A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: fix PWM sizing, Voc sign, unit switching and loading on review
- PWM usable power now uses the string's voltage (Vmp x panels in series),
delivers nothing when the charging voltage is at or above the string's Voc,
and falls off between Vmp and Voc. The "too little voltage" warning covers
PWM as well as MPPT. Previously two panels in series counted double, and a
panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
negative, in the math, the URL and the input, so the cold-morning check
cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
than asking for a location, and holds the cold-morning verdict until the
site's temperatures are known or one has been entered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: move the data attribution to the bottom of the page
The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location. The source line under the
location points to it. Bumps wrolpi-help to match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: drop the data source line under the location
The attribution at the bottom of the page is enough. The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values. Bumps wrolpi-help for the
refreshed screenshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* files: SQL FileGroup compare, stem column, drop unused discovery path
Refresh walked the tree twice (find, then stat every unchanged group),
and a two-directory refresh scanned the whole media directory. Compare
now diffs in SQL on a TEMP table that already has mtime, accepts multiple
roots, and groups by a stored stem that is versioned so a suffix-rule
change cannot delete-and-reinsert tagged files.
Deletes refresh_discover_paths and /api/files/refresh_progress, which
nothing in production called. Video/doc/zim modelers use the same
(no domain row OR indexed != True) gate.
* files: keep stem current on reorganize; NUL-safe find parse
Reorganize wrote the new primary_path but not stem, so the next refresh
treated the FileGroup as deleted+new and dropped tags. GNU find now
emits NUL-delimited records so a tab or newline in a filename cannot
abort a global refresh. Stem backfill skips the table scan once the
algorithm version matches, and runs off the event loop. An empty roots
list is a global refresh, not invalid SQL.
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page
Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:
* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending
(the normal state) this froze the loop for a second at a time; about a third
of all requests, including trivial ones, took ~1,000 ms. It now polls
`is_set()` and returns immediately.
* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
ran synchronous SQLite queries on the event loop, so one slow search stalled
every other request on that worker. They now run via `asyncio.to_thread`.
* The unfiltered Videos page (`POST /api/videos/search` with only an order)
used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
every video's index entry before LIMIT applies: seconds cold on a large
library on a spinning disk, and ~10 s hot for a deep page. It now seeks each
exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
rows and merges them, with the total from a separate count cached for 60 s.
Results are identical to the generic query.
Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a
gitignored JSON file; an example is included.
Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve the video description from its own endpoint instead of Video.__json__
`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk. The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays. The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.
Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions. The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored. The player page fetches it through
`useVideoExtras` alongside the comments.
Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: thread-local zim sessions, fast-path equivalence test, comments
* Zim search, single-zim search, and search estimates run libzim off the event
loop, each opening its own session inside the worker thread rather than
passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
query and asserts identical rows and paging for both date orders. The one
intentional difference is documented: past the end, the fast path still
reports the true total.
* switch_worker returns before its try block so the empty path does not clear
the event; its comment states the invariant only. The copied "synchronous
SQLite" comments are reduced to one line in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
wrolpi-app crash-loops with 203/EXEC when /usr/local/bin/serve is missing.
Two independent causes, both fixed here:
1. The `single-file --version ||` guard in the install scripts only installed
serve (and readability) when single-file was ABSENT. On a machine that
already has single-file (a prior install or a baked image), serve was
skipped, and re-running install.sh could never repair it. Install each npm
global independently based on its own presence.
2. The service hardcoded /usr/local/bin/serve, which breaks when npm's global
prefix puts serve in /usr/bin (depends on how node was installed). Resolve
serve via PATH with `/usr/bin/env serve`.
Verified on a fresh Raspberry Pi OS Trixie install (10.0.0.8): wrolpi-app is
active with 0 restarts and serves 200 with the PATH-resolved ExecStart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Preserve MEDIA_DIRECTORY/WROLPI_DB across the sudo re-exec (sudo
strips them from the environment, discarding the override).
- Reject unknown options and a second positional path with a usage
error instead of silently checking the wrong database.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It has a python3 shebang but was missing the executable bit; all other
scripts in ./scripts are executable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRAGMA quick_check reads every page of the database, so on a large DB or
a slow (e.g. USB spinning) disk it can run for many minutes, making
help.sh (run by the wrolpi-help-script service) appear to hang.
Drop the quick_check from help.sh's fast startup diagnostics and add a
standalone scripts/check_database_integrity.sh for on-demand checks.
The new script runs sqlite3 as the wrolpi user (WAL -shm access),
reports elapsed time, and supports --full for an exhaustive
integrity_check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fresh installs (and SSH sessions that forward LC_* variables from the client)
can reference an ungenerated locale, causing apt/perl/python to emit
"Cannot set LC_CTYPE to default locale" warnings throughout the install log.
Export LC_ALL=C.UTF-8 for the install run. C.UTF-8 is built into glibc, always
present, and locale-neutral, so it silences the warnings without generating a
locale, without adding a dependency, and without changing the user's configured
locale (e.g. an en_GB.UTF-8 system stays en_GB.UTF-8).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Controller venv (/opt/wrolpi/controller/venv) was only ever created by
upgrade.sh and reset_virtual_environments.sh, neither of which runs during a
fresh install. install_raspberrypios.sh and install_debian_12.sh created only
the main API venv, so on a clean machine wrolpi-controller crash-looped with
203/EXEC ("Unable to locate executable /opt/wrolpi/controller/venv/bin/python")
because repair.sh unconditionally starts the service.
Create the Controller venv inline right after the main venv, mirroring how the
main venv is already built in these scripts. Verified on a fresh Raspberry Pi OS
Trixie install (10.0.0.8): wrolpi-controller comes up active with 0 restarts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triage of Greptile findings on the default-Zim / configurable-directory PR:
- Stale config on restart (P1): settings update() only queues an async
background_save, and the restart switch has no ordering guarantee (the switch
worker pops LIFO), so start_kiwix_serve.sh could read the old zims_destination.
Save the config synchronously before activating the restart switch.
- YAML null (P1): read_config_value.sh now treats an unquoted null/~ as unset
(returns the default), matching PyYAML/the Python config layer. A quoted
"null" remains the literal string.
- Path traversal (P1/security): the settings API now rejects a zims_destination
containing "..", and start_kiwix_serve.sh defends in depth by falling back to
`zims` for any absolute/traversal/empty value, so Kiwix can never scan or
rewrite library.xml outside the media root.
- Reproducible build (P2): pin the zim-tools image to 3.7.0 instead of :latest.
Tests added for null handling, traversal rejection (API + script).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related gaps in how WROLPi serves Kiwix content:
1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
Kiwix. Every image now ships a small built-in default Zim that Kiwix falls
back to, telling the user to download real Zim files.
2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
`zims_destination` (which the Python side already honors).
Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
non-Python scripts can reliably read top-level wrolpi.yaml scalars. Also a
config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
(ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback. It needs no
config read: it already scans the whole mounted media dir, and
zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
serves the new directory without a reboot.
Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A corrupt or partial *.zim (e.g. an interrupted download) is rejected by
kiwix-manage ("Cannot add zim ..."). start_kiwix_serve.sh could then leave
Kiwix down and reported a misleading "Could not find any Zim files" — and
if the bad zim was the only one, exit 2 killed the service with no restart.
- Import each zim independently; log and skip a failed add so the remaining
valid zims are still served.
- Always ensure a library.xml exists (empty if needed) and exec kiwix-serve
unconditionally, so a corrupt/absent zim never stops the server starting.
- Fix the mount-wait loop, which never incremented COUNT (infinite wait
instead of a 30s timeout).
- Add MEDIA_DIRECTORY override so the script is testable.
- Add Restart=on-failure to wrolpi-kiwix.service (matches wrolpi-api.service)
so transient failures self-heal.
- New test harness runs the script with fake kiwix-manage/kiwix-serve shims.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Debian major upgrade (e.g. 12 -> 13) replaces /usr/bin/python3
(3.11 -> 3.13). Existing venvs still resolve `python3` to the system
interpreter, but their packages live in lib/python3.11/site-packages,
which the new interpreter never searches -- every import fails and
`pip install --upgrade` cannot repair them, taking down the API,
Controller, yt-dlp, and Help.
upgrade.sh now compares each venv's recorded Python version against the
current one and does a full rebuild via reset_virtual_environments.sh
when they differ. Detecting a mismatch (rather than literally "3.11")
also self-heals future bumps and is a no-op once versions match or on a
fresh install with no pyvenv.cfg yet.
reset_virtual_environments.sh gains a -y/--yes flag to skip its
confirmation prompt; the interactive yes_or_no read loops forever on EOF
under the tty-less wrolpi-upgrade.service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Debian 12 guard fetched the cutoff tag with a non-forcing refspec, so a
box that had already fetched an earlier `debian12-final` kept its stale local
tag ("would clobber existing tag" is rejected). When the tag later moved
(e.g. a version-bumped final release), the guard pinned to the OLD target
instead of the new one. Force the fetch (+ / --force) so a moved tag wins.
Safe: the commit's signature is still verified before checkout.
Caught on real Debian 12 hardware (10.0.0.8) after moving the tag for a patch
bump; added a sandbox scenario covering the moved-tag/stale-local-tag case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Debian 12 (bookworm) cannot run the Postgres->SQLite / Debian 13 rebase,
and there is no in-place 12->13 upgrade. Guard the on-device upgrade so a
Debian 12 machine pins to the signed `debian12-final` tag and never advances
past it, while Debian 13+ upgrades normally.
./upgrade.sh already fetched+checked out the latest release by the time
scripts/upgrade.sh runs, so the guard runs first: on Debian 12 it verifies
the guard commit's signature, rolls back to debian12-final, re-execs that
version's script (which warns to reinstall on Debian 13 and then runs its
normal upgrade so services restart). Aborts rather than run newer code if the
tag is missing or unsigned. Inert on Debian 13+.
Verified with a sandbox harness covering: D13 pass-through, D12-at-tag
warn+proceed, D12-past-tag verify+rollback+re-exec (no loop), and
missing-tag refusal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.
- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
arch-independent SHA). Version-guarded, so an image that already baked in the
pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
call the shared script. This also adds SHA pinning to the on-device pmtiles
path (previously unpinned).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bake the app build into images; rebuild on-device only when stale
Previously the Raspberry Pi image shipped without app/build/ and compiled
the React bundle on first boot via repair.sh -- a multi-minute, memory-hungry
step that can OOM a 2GB Pi. The Debian Live ISO already baked the build in,
so the two paths had drifted.
Add scripts/build_app.sh as the single source of truth: it hashes the build
inputs (src/, public/, package.json, package-lock.json, tsconfig.json) and
runs `npm run build` (with GENERATE_SOURCEMAP=false) only when build/ is
missing or those inputs changed, then writes build/.build-stamp.
Wire all three call sites to it:
- pi-gen 03-run-chroot.sh now builds after npm install, baking build/ into
the Pi image like the Debian hook already does.
- debian-live 9999-wrolpi.hook.chroot uses it so the baked build is stamped.
- repair.sh uses it, so first-boot/manual-repair/install/upgrade all become
no-ops when the shipped build is current.
Result: fresh images ship a stamped build/, so a weak Pi never compiles on
boot or repair; an on-device rebuild happens only when an upgrade actually
changes app/. build_app.sh never runs npm install, so repair.sh keeps its
offline guarantee.
* build_app.sh: rebuild when built output is incomplete, not just missing
Addresses review feedback: a matching stamp only proves the inputs are
unchanged, not that the output survived. repair.sh exists to fix corruption,
so a build/ with index.html + stamp but a missing JS/CSS asset must still
rebuild. Verify every file referenced by build/asset-manifest.json exists
(and that the manifest itself parses); rebuild otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Debian 13 ships kiwix-tools 3.7.0, whose kiwix-serve rejects the literal
--address 0.0.0.0 ("IP address is not available on this system: 0.0.0.0")
and exits 1, so wrolpi-kiwix fails to start on a fresh trixie image.
kiwix-serve binds all interfaces by default when --address is omitted, on
both the old (bookworm) and new (trixie) versions, so dropping the flag is
portable.
Verified on a trixie Pi (10.0.0.8): wrolpi-kiwix active, serving 200 on
9085 and via Caddy on 8085.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
help.sh now detects invalid Caddy certificates: expired or unparseable
CA/leaf certs, a leaf expiring within 30 days, a leaf that does not match
its private key or the current hostname, and Caddy serving a stale cert
after regeneration. It also checks that the renewal script is executable
and its timer is enabled.
scripts/renew_certificates.sh was committed without the executable bit,
so wrolpi-cert-renew.service failed with "Permission denied". Set the
bit and make repair.sh restore executable bits on all scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On systems that already have dnsmasq (e.g. Ubuntu-style installs), the
previous guard skipped the whole block, silently leaving iw and rfkill
uninstalled. apt-get is a no-op for packages already present, so one
combined install stays idempotent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The amd64 ISO is built with --apt-recommends false, so network-manager's
Recommends (including dnsmasq-base) were never installed. NetworkManager's
shared/hotspot mode spawns dnsmasq for DHCP, so the hotspot failed to
activate on every ISO install. Add it to the package list (which
install_debian_12.sh also installs from) and as an idempotent upgrade.sh
step so existing installs are fixed on their next upgrade.
Also add iw and rfkill, the standard WiFi diagnostic tools.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/upgrade.sh ran `main.py db upgrade` under set -e before
repair.sh, so a broken Postgres bricked the very upgrade carrying the
fix for it (seen live: the bootstrap bind-mount fix could not be
delivered because the binds had already killed the cluster). repair.sh
had the same problem with initialize_api_db.sh and the superuser psql.
Warn and continue instead: the migration is retried by
initialize_api_db.sh during repair, and if the database is still down
afterwards the API reports it loudly while the system-level work
(configs, services, bootstrap refresh) still completes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If cluster files lose postgres ownership (stray recursive chown,
interrupted install), the postmaster stays up but every new backend
dies with FATAL: could not open file "global/pg_filenode.map":
Permission denied — and dropdb/psql cannot work either. Restore
ownership of /var/lib/postgresql and /etc/postgresql and restart
postgres before recreating the database, and start the API when done.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The caddy package install hit a dpkg conffile prompt because repair.sh
already wrote /etc/caddy/Caddyfile; under the tty-less
wrolpi-upgrade.service the prompt got EOF on stdin and apt exited 100.
Run the caddy/samba/gnome-disk-utility installs with
DEBIAN_FRONTEND=noninteractive and --force-confdef/--force-confold
(keep the current conffile; repair.sh manages the Caddyfile anyway),
and run `dpkg --configure -a` first so a box left half-configured by a
previously interrupted run is healed automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If the caddy keyring file already exists (left by a prior partial
install), gpg prompts to overwrite via /dev/tty, which fails under the
tty-less wrolpi-upgrade.service ("cannot open '/dev/tty'") and aborts
the upgrade. --batch --yes overwrites silently, making the caddy
install idempotent in all four scripts that share this pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Global IPs from `hostname -I` (e.g. routable IPv6 assigned via SLAAC)
violated the Root CA's critical nameConstraints, failing leaf verification
with "permitted subtree violation" (error 47) and aborting
install/repair/upgrade. Filter SAN candidates, including EXTRA_SANS, to the
permitted private ranges and log skipped entries; normalize the IP:/DNS:
prefix case in EXTRA_SANS since openssl rejects lowercase prefixes.
Also guard the service stops in upgrade.sh so upgrading a box whose
install died before the units were installed can still recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etcher is privacy-invasive; replace it in INSTALL.md, the USB upgrade
tool messages, and the live-boot ephemeral warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.
single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.
The native path and the Archive docker container now run single-file with
"deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.
Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind. The Archive
downloader now has a ten minute timeout. Install scripts install Deno
(previously only upgrade.sh and the image builds did).
* Singlefile writes to a file instead of dumping to stdout.
Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives). single-file now writes its result to a
temporary file which WROLPi reads back.
As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.
* Compressed Archive screenshots and readability images are no longer broken.
The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths. Screenshots are now taken from the fully
extracted ZIP so those references resolve. Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).
* Compressed Archives are created by converting the uncompressed capture.
The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed). When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file. The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages. If the
conversion fails, the uncompressed singlefile is stored instead.
* Consolidating duplicate compressed-singlefile tests.
The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one. The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.
* Addressing review comments: removeprefix, PermissionError when killing.
* Bumping version to 0.24.1-beta.
* Install scripts assert the deno.zip hash to avoid malicious or invalid files.
The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.
* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Old single-file versions (1.x) create compressed Archives without the
SingleFile comment header in the self-extracting prelude, which WROLPi
rejects as invalid. The install scripts only install single-file when it
is missing, so existing machines were never upgraded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New INSTALL.md consolidates all install paths: WROLPi Portable (Live
USB) as the headline option, plus Debian 12, Raspberry Pi, and Docker.
- README install sections collapsed into a single Install section that
links to INSTALL.md.
- UPGRADE.md gains an "Upgrading a WROLPi Portable USB" section covering
scripts/wrolpi-usb.sh upgrade, and documents that in-place conversion
of an existing data drive is not supported (backup -> flash -> restore).
- wrolpi-usb.sh install now prints that supported procedure instead of a
"not implemented" stub; debian-live-config/README.md documents the same.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
numfmt with a negative argument:
Under `set -euo pipefail` the upgrade subcommand would die at
`numfmt --to=iec ... $headroom_bytes` whenever the new ISO was larger
than the headroom — numfmt treats the leading `-` as an option flag,
exits non-zero, and the script aborts before printing the descriptive
back-up-and-reflash error message just below. Splitting positive
and negative cases and feeding numfmt the absolute value fixes it;
the eventual error message now actually shows up.
Silent umount during sanity check:
`umount "$child" 2>/dev/null || true` was swallowing both stderr and
the exit code. A busy partition (open file handle from a daemon
that auto-opened the USB) would survive the loop, and `cmd_upgrade`
would proceed straight to `dd if=<iso> of=<dev>` — writing to a
block device whose partition is still mounted, corrupting the
filesystem on that partition. Adds a post-loop assertion: re-read
MOUNTPOINTS and die with an `lsof` hint if anything is still mounted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`wrolpi-usb.sh upgrade <iso> <device>` replaces the boot partition of an
existing WROLPi USB with a new ISO while preserving the persistence
partition.
Flow:
1. Locate the persistence partition by label.
2. Read its sector range from sfdisk -d.
3. Confirm the new ISO fits in the headroom (i.e. its byte count <=
persistence_start_sector * 512). If not, bail with instructions
telling the user to back up /media/wrolpi, re-flash from scratch,
and restore. This is the case for USBs flashed before the 8 GiB-
headroom commit.
4. Save the partition table to /tmp/ for emergency recovery.
5. Prompt for "yes".
6. dd the new ISO onto the device. The persistence partition's
sectors are past the ISO's end, so its data is untouched.
7. sfdisk --append to re-add the persistence partition entry at the
saved sector range. The data on disk is still there; we are just
fixing up the partition table the new ISO wrote.
8. partx -u to refresh the kernel view; verify the label is reachable.
`install` subcommand is wired up to a die() for now; will land in a
follow-up commit.
The script refuses to operate on non-removable disks (foot-gun guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Map viewer has simple search.
Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
- Replace nginx with Caddy as the web server for both Docker and native
(RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
proxying to all services (API, React app, Controller, Kiwix, Help, Map),
and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
uses Name Constraints to restrict signing to local addresses only
(.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
certificate installation on iOS, Android, macOS, Windows, Linux, and
Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
/download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.