A Debian major upgrade (e.g. 12 -> 13) replaces /usr/bin/python3
(3.11 -> 3.13). Existing venvs still resolve `python3` to the system
interpreter, but their packages live in lib/python3.11/site-packages,
which the new interpreter never searches -- every import fails and
`pip install --upgrade` cannot repair them, taking down the API,
Controller, yt-dlp, and Help.
upgrade.sh now compares each venv's recorded Python version against the
current one and does a full rebuild via reset_virtual_environments.sh
when they differ. Detecting a mismatch (rather than literally "3.11")
also self-heals future bumps and is a no-op once versions match or on a
fresh install with no pyvenv.cfg yet.
reset_virtual_environments.sh gains a -y/--yes flag to skip its
confirmation prompt; the interactive yes_or_no read loops forever on EOF
under the tty-less wrolpi-upgrade.service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Debian 12 guard fetched the cutoff tag with a non-forcing refspec, so a
box that had already fetched an earlier `debian12-final` kept its stale local
tag ("would clobber existing tag" is rejected). When the tag later moved
(e.g. a version-bumped final release), the guard pinned to the OLD target
instead of the new one. Force the fetch (+ / --force) so a moved tag wins.
Safe: the commit's signature is still verified before checkout.
Caught on real Debian 12 hardware (10.0.0.8) after moving the tag for a patch
bump; added a sandbox scenario covering the moved-tag/stale-local-tag case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Debian 12 (bookworm) cannot run the Postgres->SQLite / Debian 13 rebase,
and there is no in-place 12->13 upgrade. Guard the on-device upgrade so a
Debian 12 machine pins to the signed `debian12-final` tag and never advances
past it, while Debian 13+ upgrades normally.
./upgrade.sh already fetched+checked out the latest release by the time
scripts/upgrade.sh runs, so the guard runs first: on Debian 12 it verifies
the guard commit's signature, rolls back to debian12-final, re-execs that
version's script (which warns to reinstall on Debian 13 and then runs its
normal upgrade so services restart). Aborts rather than run newer code if the
tag is missing or unsigned. Inert on Debian 13+.
Verified with a sandbox harness covering: D13 pass-through, D12-at-tag
warn+proceed, D12-past-tag verify+rollback+re-exec (no loop), and
missing-tag refusal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.
- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
arch-independent SHA). Version-guarded, so an image that already baked in the
pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
call the shared script. This also adds SHA pinning to the on-device pmtiles
path (previously unpinned).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bake the app build into images; rebuild on-device only when stale
Previously the Raspberry Pi image shipped without app/build/ and compiled
the React bundle on first boot via repair.sh -- a multi-minute, memory-hungry
step that can OOM a 2GB Pi. The Debian Live ISO already baked the build in,
so the two paths had drifted.
Add scripts/build_app.sh as the single source of truth: it hashes the build
inputs (src/, public/, package.json, package-lock.json, tsconfig.json) and
runs `npm run build` (with GENERATE_SOURCEMAP=false) only when build/ is
missing or those inputs changed, then writes build/.build-stamp.
Wire all three call sites to it:
- pi-gen 03-run-chroot.sh now builds after npm install, baking build/ into
the Pi image like the Debian hook already does.
- debian-live 9999-wrolpi.hook.chroot uses it so the baked build is stamped.
- repair.sh uses it, so first-boot/manual-repair/install/upgrade all become
no-ops when the shipped build is current.
Result: fresh images ship a stamped build/, so a weak Pi never compiles on
boot or repair; an on-device rebuild happens only when an upgrade actually
changes app/. build_app.sh never runs npm install, so repair.sh keeps its
offline guarantee.
* build_app.sh: rebuild when built output is incomplete, not just missing
Addresses review feedback: a matching stamp only proves the inputs are
unchanged, not that the output survived. repair.sh exists to fix corruption,
so a build/ with index.html + stamp but a missing JS/CSS asset must still
rebuild. Verify every file referenced by build/asset-manifest.json exists
(and that the manifest itself parses); rebuild otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Debian 13 ships kiwix-tools 3.7.0, whose kiwix-serve rejects the literal
--address 0.0.0.0 ("IP address is not available on this system: 0.0.0.0")
and exits 1, so wrolpi-kiwix fails to start on a fresh trixie image.
kiwix-serve binds all interfaces by default when --address is omitted, on
both the old (bookworm) and new (trixie) versions, so dropping the flag is
portable.
Verified on a trixie Pi (10.0.0.8): wrolpi-kiwix active, serving 200 on
9085 and via Caddy on 8085.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
help.sh now detects invalid Caddy certificates: expired or unparseable
CA/leaf certs, a leaf expiring within 30 days, a leaf that does not match
its private key or the current hostname, and Caddy serving a stale cert
after regeneration. It also checks that the renewal script is executable
and its timer is enabled.
scripts/renew_certificates.sh was committed without the executable bit,
so wrolpi-cert-renew.service failed with "Permission denied". Set the
bit and make repair.sh restore executable bits on all scripts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On systems that already have dnsmasq (e.g. Ubuntu-style installs), the
previous guard skipped the whole block, silently leaving iw and rfkill
uninstalled. apt-get is a no-op for packages already present, so one
combined install stays idempotent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The amd64 ISO is built with --apt-recommends false, so network-manager's
Recommends (including dnsmasq-base) were never installed. NetworkManager's
shared/hotspot mode spawns dnsmasq for DHCP, so the hotspot failed to
activate on every ISO install. Add it to the package list (which
install_debian_12.sh also installs from) and as an idempotent upgrade.sh
step so existing installs are fixed on their next upgrade.
Also add iw and rfkill, the standard WiFi diagnostic tools.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/upgrade.sh ran `main.py db upgrade` under set -e before
repair.sh, so a broken Postgres bricked the very upgrade carrying the
fix for it (seen live: the bootstrap bind-mount fix could not be
delivered because the binds had already killed the cluster). repair.sh
had the same problem with initialize_api_db.sh and the superuser psql.
Warn and continue instead: the migration is retried by
initialize_api_db.sh during repair, and if the database is still down
afterwards the API reports it loudly while the system-level work
(configs, services, bootstrap refresh) still completes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If cluster files lose postgres ownership (stray recursive chown,
interrupted install), the postmaster stays up but every new backend
dies with FATAL: could not open file "global/pg_filenode.map":
Permission denied — and dropdb/psql cannot work either. Restore
ownership of /var/lib/postgresql and /etc/postgresql and restart
postgres before recreating the database, and start the API when done.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The caddy package install hit a dpkg conffile prompt because repair.sh
already wrote /etc/caddy/Caddyfile; under the tty-less
wrolpi-upgrade.service the prompt got EOF on stdin and apt exited 100.
Run the caddy/samba/gnome-disk-utility installs with
DEBIAN_FRONTEND=noninteractive and --force-confdef/--force-confold
(keep the current conffile; repair.sh manages the Caddyfile anyway),
and run `dpkg --configure -a` first so a box left half-configured by a
previously interrupted run is healed automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If the caddy keyring file already exists (left by a prior partial
install), gpg prompts to overwrite via /dev/tty, which fails under the
tty-less wrolpi-upgrade.service ("cannot open '/dev/tty'") and aborts
the upgrade. --batch --yes overwrites silently, making the caddy
install idempotent in all four scripts that share this pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Global IPs from `hostname -I` (e.g. routable IPv6 assigned via SLAAC)
violated the Root CA's critical nameConstraints, failing leaf verification
with "permitted subtree violation" (error 47) and aborting
install/repair/upgrade. Filter SAN candidates, including EXTRA_SANS, to the
permitted private ranges and log skipped entries; normalize the IP:/DNS:
prefix case in EXTRA_SANS since openssl rejects lowercase prefixes.
Also guard the service stops in upgrade.sh so upgrading a box whose
install died before the units were installed can still recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etcher is privacy-invasive; replace it in INSTALL.md, the USB upgrade
tool messages, and the live-boot ephemeral warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.
single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.
The native path and the Archive docker container now run single-file with
"deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.
Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind. The Archive
downloader now has a ten minute timeout. Install scripts install Deno
(previously only upgrade.sh and the image builds did).
* Singlefile writes to a file instead of dumping to stdout.
Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives). single-file now writes its result to a
temporary file which WROLPi reads back.
As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.
* Compressed Archive screenshots and readability images are no longer broken.
The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths. Screenshots are now taken from the fully
extracted ZIP so those references resolve. Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).
* Compressed Archives are created by converting the uncompressed capture.
The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed). When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file. The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages. If the
conversion fails, the uncompressed singlefile is stored instead.
* Consolidating duplicate compressed-singlefile tests.
The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one. The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.
* Addressing review comments: removeprefix, PermissionError when killing.
* Bumping version to 0.24.1-beta.
* Install scripts assert the deno.zip hash to avoid malicious or invalid files.
The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.
* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Old single-file versions (1.x) create compressed Archives without the
SingleFile comment header in the self-extracting prelude, which WROLPi
rejects as invalid. The install scripts only install single-file when it
is missing, so existing machines were never upgraded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New INSTALL.md consolidates all install paths: WROLPi Portable (Live
USB) as the headline option, plus Debian 12, Raspberry Pi, and Docker.
- README install sections collapsed into a single Install section that
links to INSTALL.md.
- UPGRADE.md gains an "Upgrading a WROLPi Portable USB" section covering
scripts/wrolpi-usb.sh upgrade, and documents that in-place conversion
of an existing data drive is not supported (backup -> flash -> restore).
- wrolpi-usb.sh install now prints that supported procedure instead of a
"not implemented" stub; debian-live-config/README.md documents the same.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
numfmt with a negative argument:
Under `set -euo pipefail` the upgrade subcommand would die at
`numfmt --to=iec ... $headroom_bytes` whenever the new ISO was larger
than the headroom — numfmt treats the leading `-` as an option flag,
exits non-zero, and the script aborts before printing the descriptive
back-up-and-reflash error message just below. Splitting positive
and negative cases and feeding numfmt the absolute value fixes it;
the eventual error message now actually shows up.
Silent umount during sanity check:
`umount "$child" 2>/dev/null || true` was swallowing both stderr and
the exit code. A busy partition (open file handle from a daemon
that auto-opened the USB) would survive the loop, and `cmd_upgrade`
would proceed straight to `dd if=<iso> of=<dev>` — writing to a
block device whose partition is still mounted, corrupting the
filesystem on that partition. Adds a post-loop assertion: re-read
MOUNTPOINTS and die with an `lsof` hint if anything is still mounted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`wrolpi-usb.sh upgrade <iso> <device>` replaces the boot partition of an
existing WROLPi USB with a new ISO while preserving the persistence
partition.
Flow:
1. Locate the persistence partition by label.
2. Read its sector range from sfdisk -d.
3. Confirm the new ISO fits in the headroom (i.e. its byte count <=
persistence_start_sector * 512). If not, bail with instructions
telling the user to back up /media/wrolpi, re-flash from scratch,
and restore. This is the case for USBs flashed before the 8 GiB-
headroom commit.
4. Save the partition table to /tmp/ for emergency recovery.
5. Prompt for "yes".
6. dd the new ISO onto the device. The persistence partition's
sectors are past the ISO's end, so its data is untouched.
7. sfdisk --append to re-add the persistence partition entry at the
saved sector range. The data on disk is still there; we are just
fixing up the partition table the new ISO wrote.
8. partx -u to refresh the kernel view; verify the label is reachable.
`install` subcommand is wired up to a die() for now; will land in a
follow-up commit.
The script refuses to operate on non-removable disks (foot-gun guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Map viewer has simple search.
Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
- Replace nginx with Caddy as the web server for both Docker and native
(RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
proxying to all services (API, React app, Controller, Kiwix, Help, Map),
and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
uses Name Constraints to restrict signing to local addresses only
(.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
certificate installation on iOS, Android, macOS, Windows, Linux, and
Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
/download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
