Commit graph

86 commits

Author SHA1 Message Date
Roland
19f4958fb5 zim: ship a default fallback Zim and honor the configured Zim directory
Two related gaps in how WROLPi serves Kiwix content:

1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
   Kiwix.  Every image now ships a small built-in default Zim that Kiwix falls
   back to, telling the user to download real Zim files.

2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
   `zims_destination` (which the Python side already honors).

Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
  non-Python scripts can reliably read top-level wrolpi.yaml scalars.  Also a
  config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
  (ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
  the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
  Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback.  It needs no
  config read: it already scans the whole mounted media dir, and
  zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
  serves the new directory without a reboot.

Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 19:09:57 -06:00
Roland
a3a712ead1 debian-live hook: set safe.directory before the pre-stamp reset
Add `git config --global --add safe.directory /opt/wrolpi` before the
`git reset --hard`, guarding against git's "dubious ownership" refusal.
Set HOME=/root first so the --global write cannot fail (and abort the ISO
build under `set -e`) if the live-build chroot leaves HOME unset -- the
same pattern the pi-gen hook already uses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 13:39:50 -06:00
Roland
575d2f9744 image builds: reset repo to HEAD before stamping the app build
The pi-gen and Debian Live chroot hooks run `npm install`, which rewrites
the tracked app/package-lock.json, then build_app.sh stamps
build/.build-stamp from that dirty working tree.  On first boot repair.sh
runs `git reset --hard` (reverting the lock) *before* it recomputes the
stamp, so the baked stamp can never match -- and the device rebuilds the
React app every time, the exact OOM-prone step baking the build in (#536)
was meant to avoid.

Verified against WROLPi-v0.27.2: the shipped image had `M app/package-lock.json`,
baked stamp a9c6ef61..., post-reset stamp 2b908c6c... (guaranteed mismatch).

Mirror repair.sh in both image hooks: `git reset --hard HEAD` right after
npm install and before build_app.sh, so the baked stamp represents committed
source and matches what the device recomputes.  Only the lock is reverted;
build/, node_modules, and the venvs are untracked and survive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 13:39:50 -06:00
lrnselfreliance
899ab13964
Consolidate map-tool install into wrolpi/scripts/install_map_tools.sh (#538)
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.

- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
  SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
  arch-independent SHA). Version-guarded, so an image that already baked in the
  pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
  call the shared script. This also adds SHA pinning to the on-device pmtiles
  path (previously unpinned).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:55:06 -06:00
lrnselfreliance
944d95c717
Bake the app build into images; rebuild on-device only when stale (#536)
* Bake the app build into images; rebuild on-device only when stale

Previously the Raspberry Pi image shipped without app/build/ and compiled
the React bundle on first boot via repair.sh -- a multi-minute, memory-hungry
step that can OOM a 2GB Pi.  The Debian Live ISO already baked the build in,
so the two paths had drifted.

Add scripts/build_app.sh as the single source of truth: it hashes the build
inputs (src/, public/, package.json, package-lock.json, tsconfig.json) and
runs `npm run build` (with GENERATE_SOURCEMAP=false) only when build/ is
missing or those inputs changed, then writes build/.build-stamp.

Wire all three call sites to it:
- pi-gen 03-run-chroot.sh now builds after npm install, baking build/ into
  the Pi image like the Debian hook already does.
- debian-live 9999-wrolpi.hook.chroot uses it so the baked build is stamped.
- repair.sh uses it, so first-boot/manual-repair/install/upgrade all become
  no-ops when the shipped build is current.

Result: fresh images ship a stamped build/, so a weak Pi never compiles on
boot or repair; an on-device rebuild happens only when an upgrade actually
changes app/.  build_app.sh never runs npm install, so repair.sh keeps its
offline guarantee.

* build_app.sh: rebuild when built output is incomplete, not just missing

Addresses review feedback: a matching stamp only proves the inputs are
unchanged, not that the output survived.  repair.sh exists to fix corruption,
so a build/ with index.html + stamp but a missing JS/CSS asset must still
rebuild.  Verify every file referenced by build/asset-manifest.json exists
(and that the manifest itself parses); rebuild otherwise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:19:25 -06:00
Roland
e01795ad7e
Debian 13 (trixie) rebase: docker, debian-live-config, and pi-gen builds.
All three build targets move from Debian 12/bookworm to Debian 13/trixie
for fresh installs; existing Debian 12 systems remain supported in place.

Docker: api/help on python:3.13-trixie, controller on 3.13-slim-trixie,
app on node:20-trixie (matches trixie's nodejs 20), samba/zim on
debian:13.  Full backend suite passes on Python 3.13 in the api container.

debian-live-config: --distribution trixie; postgres teardown for the
SQLite cutover (pg_dropcluster hook block removed, wrolpi-bootstrap.sh no
longer creates a cluster or calls the deleted initialize_api_db.sh — the
API creates wrolpi.db itself on first start).  README documents the
SQLite layout and drops Etcher recommendations.

pi-gen: upstream has no trixie branch — the arm64 branch now defaults to
RELEASE=trixie; KEYBOARD_KEYMAP=en_US is rejected by trixie's
console-setup (ckbcomp validates), use 'us'; stop stamping a Postgres
.pgpass into /etc/skel.

Package lists (both targets): postgresql-15/postgresql-doc/libpq-dev
deleted; libtiff5-dev renamed to libtiff-dev; software-properties-common
and cpufrequtils removed from trixie (Controller CPU throttle falls back
to sysfs, dead cpufreq-set sudoers rule removed); locales and gnupg added
(no longer arrive transitively under --apt-recommends false); sqlite3 CLI
added for reset_tags.sh.

help.sh: replace the Postgres service/port/socket/cluster/psql checks
(all false FAILEDs after the SQLite cutover) with SQLite checks:
wrolpi.db exists, PRAGMA quick_check, file_group table + row count, run
as the wrolpi user so root never creates the -shm sidecar.

Both build scripts take a WROLPI_BUILD_DIR override so development builds
can run beside the release pipeline's build trees; the sudo re-exec
passes it through explicitly (a plain sudo would drop it).

test_common.py: the truncate tests hard-coded sys.getsizeof results from
the 3.11 object layout (str/list shrank in 3.12); assert the contract
(fits budget, prefix, monotonic in budget) so they pass on 3.11 and 3.13.

Verified: trixie ISO (3.7G) and pi-gen desktop image (4.3G) both built
from scratch and passed their sanity checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 15:43:08 -06:00
Roland
93d9af1480 Hotspot: ship dnsmasq-base; NetworkManager requires it for hotspot DHCP.
The amd64 ISO is built with --apt-recommends false, so network-manager's
Recommends (including dnsmasq-base) were never installed. NetworkManager's
shared/hotspot mode spawns dnsmasq for DHCP, so the hotspot failed to
activate on every ISO install. Add it to the package list (which
install_debian_12.sh also installs from) and as an idempotent upgrade.sh
step so existing installs are fixed on their next upgrade.

Also add iw and rfkill, the standard WiFi diagnostic tools.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 07:51:13 -06:00
Roland
f500600025
Bootstrap: Postgres bind-mounts on live boots only.
wrolpi-bootstrap.service applied the /media/wrolpi/config/postgresql
bind-mounts over /var/lib/postgresql and /etc/postgresql at every boot,
including on installed systems.  The binds exist for live boots, where
the root filesystem is an ephemeral overlay; on an installed system they
put the cluster in the path of repair.sh's config chown
(pg_filenode.map permission errors) and, once the user mounts an
NTFS/exFAT primary drive over /media/wrolpi, point Postgres at a
filesystem that cannot host a data directory at all — no clusters,
connection refused, while the umbrella postgresql.service stays green.

Installed systems now keep Postgres at the normal Debian paths on the
root filesystem.  A box that still has a stale bind active from an old
boot gets a log note to reboot; on the first clean boot the existing
pg_createcluster/initialize_api_db logic builds the root-disk cluster
automatically and the database rebuilds from configs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 15:33:54 -06:00
Roland
ebbc60992c
Docs: recommend Raspberry Pi Imager instead of Etcher.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Etcher is privacy-invasive; replace it in INSTALL.md, the USB upgrade
tool messages, and the live-boot ephemeral warning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:58:54 -06:00
lrnselfreliance
1acb148c90
Single file deno (#478)
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.

single-file-cli is designed for Deno.  Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.

The native path and the Archive docker container now run single-file with
"deno run".  Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum).  Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.

Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind.  The Archive
downloader now has a ten minute timeout.  Install scripts install Deno
(previously only upgrade.sh and the image builds did).

* Singlefile writes to a file instead of dumping to stdout.

Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives).  single-file now writes its result to a
temporary file which WROLPi reads back.

As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.

* Compressed Archive screenshots and readability images are no longer broken.

The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths.  Screenshots are now taken from the fully
extracted ZIP so those references resolve.  Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).

* Compressed Archives are created by converting the uncompressed capture.

The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed).  When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file.  The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages.  If the
conversion fails, the uncompressed singlefile is stored instead.

* Consolidating duplicate compressed-singlefile tests.

The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one.  The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.

* Addressing review comments: removeprefix, PermissionError when killing.

* Bumping version to 0.24.1-beta.

* Install scripts assert the deno.zip hash to avoid malicious or invalid files.

The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.

* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:28 -06:00
Roland
d85c7721b5
Fixing call to Chrome to generate Archive screenshots. 2026-06-11 19:46:34 -06:00
Roland
68d8cd39c1
Including hdparm for drive handling.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-06-01 19:31:17 -06:00
Roland
dcfad8b4af
debian-live-config: copy built ISO by name, not a SCRIPT_DIR glob
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
The build ended with `mv "${SCRIPT_DIR}"/*.iso "${DEST}"`. A leftover ISO from
a previous build made the glob match two files, so `mv` treated the target as
a directory and failed ("No such file or directory") -- after a successful
37-minute lb build. Take the single ISO out of the freshly-wiped BUILD_DIR via
a nullglob array and copy it to the final name; robust to leftovers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 20:27:11 -06:00
Roland
0e3564b704 Automating build and upload of WROLPi release images. 2026-05-30 14:23:08 -06:00
Roland
960eb81880
Documenting WROLPi Portable USB, splitting install docs into INSTALL.md.
- New INSTALL.md consolidates all install paths: WROLPi Portable (Live
  USB) as the headline option, plus Debian 12, Raspberry Pi, and Docker.
- README install sections collapsed into a single Install section that
  links to INSTALL.md.
- UPGRADE.md gains an "Upgrading a WROLPi Portable USB" section covering
  scripts/wrolpi-usb.sh upgrade, and documents that in-place conversion
  of an existing data drive is not supported (backup -> flash -> restore).
- wrolpi-usb.sh install now prints that supported procedure instead of a
  "not implemented" stub; debian-live-config/README.md documents the same.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 21:33:22 -06:00
Roland
1bc337bfb4
Addressing Greptile P2s on the Ventoy ephemeral fallback.
- Treat virtual disks (vd*/xvd*/hd*) as direct, not ephemeral.  A blank
  VM disk imaged with the ISO can safely hold a persistence partition;
  previously these fell through the wildcard to loop mode and never
  accumulated state.
- Move the ephemeral note_action inside the mountpoint guard so a manual
  wrolpi-bootstrap.service restart mid-session doesn't re-fire the "NOT
  writing to disk" first-boot dialog.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:13:12 -06:00
Roland
82dd7782d3 Falling back to tmpfs when booted via Ventoy or other loop-mounted ISOs.
wrolpi-bootstrap.sh now classifies the live medium by device-node
pattern.  Real partitions (/dev/sd*, /dev/nvme*, /dev/mmcblk*) continue
to get a persistence partition appended.  Loop/device-mapper sources
(/dev/loop*, /dev/dm-*, /dev/mapper/*) — the marker for Ventoy/YUMI/
Easy2Boot/etc — mount a tmpfs at /media/wrolpi instead, so we never
sfdisk-append a partition onto the user's multiboot stick.

First-boot zenity dialog branches on mode and tells ephemeral users
their data won't persist; README documents the behaviour and the
deferred persistence.dat work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:12:52 -06:00
Roland
662306a458 Blocking lightdm until wrolpi-bootstrap finishes — fixes first-boot zenity race.
wrolpi-bootstrap.service is `WantedBy=multi-user.target` and `Before=`
the wrolpi-api/app/kiwix/controller services, but had no ordering
constraint against lightdm.  Systemd was free to start lightdm in
parallel with bootstrap, so XFCE autologin would land before bootstrap
had written /tmp/wrolpi-firstboot-summary.txt — and the
wrolpi-firstboot-notify autostart would fire, find no summary, and
exit silently.

On the live test this manifested as the user seeing the desktop come
up immediately but never seeing the welcome zenity dialog.  The notify
script's own log confirmed the timing race:

  ===== Mon May 25 08:01:46 PM UTC 2026 invoked ... =====
  no summary file at /tmp/wrolpi-firstboot-summary.txt — nothing to do

bootstrap was still in the middle of `pg_createcluster` at 20:01:46
and didn't write the summary file until ~20:01:48.

Adding `Before=display-manager.service lightdm.service` makes systemd
wait for bootstrap to complete before bringing up the display manager.
Tested live by logging out + back in on the running ISO (re-firing
autostart with the summary already in place) — the zenity dialog
appeared.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
677483a502 Using existing repo .desktop files instead of hand-rolled variants.
Earlier commit hand-rolled six Type=Link launchers from scratch.  The
repo already had the canonical files at etc/raspberrypios/wrolpi.desktop,
wrolpi-help.desktop, wrolpi-help-files.desktop, and controller.desktop,
used on the RPi/Debian bare-metal deployments.

Switching the chroot hook to `cp` those files into ~/Desktop/ keeps the
live ISO in sync with the other deployments — when those .desktop files
get edited (URL changes, new icon, etc.), the live ISO inherits the
change with no chroot-hook edit needed.

Drops the Kiwix Library, WROLPi Files, and Disk Management entries the
previous commit added; the user requested the four canonical entries
only.  The trust autostart from the previous commit continues to mark
all *.desktop in ~/Desktop, so the surviving icons launch on
double-click.

All four icons tested on the live ISO at 10.0.0.17:

  WROLPi App           https://127.0.0.1            ✓
  WROLPi Controller    http://127.0.0.1             ✓
  WROLPi Help          http://127.0.0.1:9086        ✓
  WROLPi Help Files    file:///opt/wrolpi-help/docs ✓

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
4f721274c6 Adding WROLPi Help Files desktop launcher.
A Type=Link to file:///opt/wrolpi-help/docs/, which dispatches via
xdg-open to Thunar.  Mirrors the wrolpi-help-files.desktop entry from
etc/raspberrypios/ used on the RPi/Debian deployments.  Lets the user
browse the raw markdown sources for the Help site directly, in addition
to the rendered HTML at https://localhost:8086/.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
d88417312a Adding desktop shortcuts for WROLPi services + trust-on-login autostart.
Five Type=Link launchers drop on /home/wrolpi/Desktop/ via the chroot
hook:

  WROLPi              https://localhost/             /opt/wrolpi/icon.png
  WROLPi Controller   https://localhost/controller/  controller-logo.png
  WROLPi Help         https://localhost:8086/        help-browser (icon theme)
  Kiwix Library       https://localhost:8085/        accessories-dictionary
  WROLPi Files        file:///media/wrolpi/          folder

All HTTPS URLs land on Caddy on localhost; the file:// URL is
dispatched via xdg-open to Thunar.

XFCE Bookworm rejects any .desktop file in $HOME/Desktop without
metadata::trusted set, even for Type=Link entries (verified live).  So
this also restores the trust autostart we previously had for the now-
removed Calamares icon — generalised to walk every *.desktop in
$HOME/Desktop and stamp the trust metadata on each, every session
start.  Idempotent; gio set is a no-op when values already match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
f8071ea5f4 Fixing first-boot zenity autostart that XFCE was silently ignoring.
The original wrolpi-firstboot-notify.desktop was XDG-valid (per
`desktop-file-validate`) but xfce4-session would not pick it up: zero
log entries on first boot, no zenity dialog, no process spawned.

Bringing the file in line with the upstream autostart entries that we
saw working (nm-applet, xfce4-notifyd, etc.) fixes it.  The likely
required field is one of `TryExec=` / `Icon=` / `Categories=` —
desktop-file-validate didn't flag the absence of any of them, but
adding them as a set makes the autostart fire.

Also instruments the notify script: it now writes a one-line log entry
to /tmp/wrolpi-firstboot-notify.log on every invocation so a future
"didn't see the dialog" report has a smoking gun.

And the rm of the (root-owned) summary file now tries via sudo first.
/tmp's sticky bit prevented the unprivileged user from removing a
root-owned file — left a stale summary that would re-trigger the
dialog on a subsequent login.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
b7ff9f7389 Reserving 8 GiB headroom between ISO and persistence partitions.
The persistence partition is created immediately after the ISO partition
ends (1-MiB aligned).  When `wrolpi-usb.sh upgrade` overwrites the ISO
partition with a future, possibly-larger ISO, growth past the current
ISO size would overwrite persistence-partition sectors.

Reserve 8 GiB between them on first boot.  Future ISO releases up to
roughly 2x the current size (~3.3 GB) can be written into that gap
without touching persistence.

The cost is ~8 GiB of persistence the user can't reclaim.  On a typical
128 GB USB this is ~6% of total capacity, which is comparable to
filesystem overhead and rounding loss.

Existing USBs flashed before this commit have no headroom — they need
a one-time backup-and-reflash to pick it up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
217ec2da70 Polishing first-boot UX: better splash placement + post-bootstrap notification.
Splash:
  Shifted the logo + text block up by half a logo-height.  GRUB's menu
  renders centred over the splash; the previous placement put the icon
  directly behind the menu entries.

First-boot summary:
  wrolpi-bootstrap.sh now accumulates a short list of notable actions
  (partition created, Postgres cluster set up, DB initialised) and
  writes them to /tmp/wrolpi-firstboot-summary.txt.  A new XDG autostart
  (wrolpi-firstboot-notify) reads that file once XFCE comes up and shows
  a zenity dialog telling the user what just happened — answering the
  "wait, what did this thing just do to my USB drive?" question.

Subsequent boots where the bootstrap is a no-op write no summary file,
so the dialog only appears the boot it actually matters.  /tmp is tmpfs,
so the file is also naturally per-boot.

Adds zenity to the package list (not pulled by --apt-recommends false).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
0dc7f60a37 Fixing rebrand binary hook (wrong cwd) and using the new transparent logo.
The rebrand hook from the previous commit failed during lb build because
it referenced files as `binary/boot/grub/grub.cfg` etc., but
live-build's binary_hooks runs each hook with cwd already set to
binary/ (it does `cd binary && ../$HOOK`).  The paths now drop the
`binary/` prefix.

Splash regenerated from the new transparent logo
(app/public/icon-transparent.png) so the wifi-pie has a clean white
outline on the dark splash background — better visual separation than
the white-stripped icon.png the previous splash used.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
e58f7cf8ce Fixing splash typography: subtitle was overlapping the title.
Pushes "Off-grid library" down twice its own height from the bottom of
"WROLPi", clearing the visual collision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
6fe3a8311d Rebranding the boot menu: WROLPi splash, WROLPi menu entries.
The auto-generated live-build splash.png composes a yellow "build worker"
hardhat with "Debian GNU/Linux 12 (bookworm) amd64" baked in as a PNG
overlay.  Both GRUB and ISOLINUX use it as the menu background, so it's
the first thing users see at boot.

Two replacement splash images (800x600 for GRUB, 640x480 for ISOLINUX)
land via config/includes.binary/, generated once from the project's
icon.png (the wifi-pie favicon).  No build-time composition needed; the
files are checked in as binary blobs.

A *.binary hook then seds the live-build-generated menu entries so the
GRUB and ISOLINUX text reads:

  Run WROLPi (live)             ← was "Live system (amd64)"
  Run WROLPi (live, fail-safe)  ← was "Live system (amd64 fail-safe mode)"
  Install WROLPi                ← was "Start installer"
  Install WROLPi (speech)       ← was "Start installer with speech synthesis"

We can't override live-build's substitution source without patching the
distro-installed `binary_grub_cfg` script; doing it after the fact in a
.binary hook keeps the customisation in our config tree.

Note: this rebrands what GRUB / ISOLINUX render *around* the d-i menus.
Once the user selects "Install WROLPi" and d-i actually launches, the
yellow hardhat reappears inside d-i's own screens — that logo
(logo_installer.png) lives inside the d-i initrd and would require a
custom initrd rebuild to swap.  Punting on that for now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
f54b5f0dd8 Reverting to Debian Installer (--debian-installer live) over Calamares.
The Calamares desktop-icon path doesn't launch reliably in our wrolpi
auto-login session.  We chased two stacks of fixes — gvfs + tumbler
for xfdesktop's launch path, lxpolkit for pkexec's GUI auth agent,
metadata::trusted + xfce-exe-checksum for XFCE's untrusted-launcher
check — and the icon still does nothing when double-clicked on a fresh
boot.  The upstream Debian Installer reached via GRUB has none of these
problems and is the better-tested path for installing onto disk.

build.sh adds:
  --debian-installer live          live-installer copies the squashfs
                                   onto the target instead of doing a
                                   network re-install
  --debian-installer-gui true      graphical d-i frontend available

GRUB will now show Live / Install / Graphical Install / Advanced
entries; the user picks at boot.

Drops:
- calamares + calamares-settings-debian from the package list
- /etc/calamares/branding/wrolpi/ + settings.conf + modules
- The install-wrolpi.desktop launcher + the trust autostart we shipped
  to try to make it click-able

Net ISO size will drop ~50-100 MB (Calamares + Qt5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
8b1abd104c Polishing the combined ISO: audio, WiFi, installer launcher.
Three classes of UX gaps surfaced when test-booting the previous build:

  Audio silent.
    --apt-recommends false stripped task-xfce-desktop's recommend chain,
    which is what normally pulls the PulseAudio server.  We only had the
    panel plugin + client libs.  Adding pulseaudio + pulseaudio-utils +
    pavucontrol + alsa-utils explicitly.

  No WiFi UI.
    NetworkManager daemon was running and could see the radio, but
    network-manager-gnome (nm-applet) wasn't pulled by recommends either.
    Adding it explicitly.

  Desktop "Install WROLPi" icon does nothing.
    Two missing pieces stacked: (a) xfdesktop's launch path needs gvfs
    (for metadata::trusted) and tumbler (thumbnail dispatch raises a
    GLib critical without it).  (b) Even when xfdesktop dispatches, pkexec
    can't authenticate without a registered GUI polkit agent — task-xfce-
    desktop's recommend chain would have provided one, but it didn't, so
    pkexec falls back to a TTY agent that fails in a Thunar-spawned
    process (no /dev/tty).  Adding gvfs, lxpolkit, tumbler.  Rewrites
    the desktop launcher as Type=Link (skips the executable-trust check
    entirely) and ships a session autostart that gio-sets
    metadata::trusted + metadata::xfce-exe-checksum so xfdesktop accepts
    user-owned .desktop files.

Also fixes a Calamares FATAL on boot: branding.desc requires a slideshow
key even when there's no real slideshow content; ships a minimal show.qml
that just renders the WROLPi logo + a welcome line.

Also adds xfce4-notifyd and xfce4-power-manager to desktop.list.chroot —
not directly tied to the three issues above, but obvious omissions from
the same recommend-stripped XFCE install.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
2acb80c1ae
Branding Calamares as WROLPi; post-install cleanup of the orphan launcher.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
The installer now identifies as WROLPi rather than Debian: product name,
sidebar logo (the wifi-pie favicon from /icon.png), bootloader entry name,
and URLs all point at WROLPi.  Sidebar colors are WROLPi purple.

Three new files under etc/calamares/:
- branding/wrolpi/branding.desc  — product strings + image + color scheme
- branding/wrolpi/wrolpi-logo.png — copy of the project favicon
- settings.conf                  — override picks branding=wrolpi and adds
                                   shellprocess to the exec sequence
- modules/shellprocess.conf      — rm /home/wrolpi/Desktop/install-wrolpi.desktop
                                   on the target so the installed system
                                   doesn't ship with a broken icon

Slideshow QML is intentionally omitted: producing our own QML/asset pack
is deferred polish.  Calamares falls back to a static welcome screen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:49:22 -06:00
Roland
2524a017e1 Adding Calamares as the graphical installer.
Pulls in `calamares` + `calamares-settings-debian` (the upstream Debian
theme/config package).  Calamares runs from the desktop via the
"Install WROLPi to disk" icon: pkexec elevates, the user types the
wrolpi account password, the standard install wizard opens.

The launcher is the upstream install-debian.desktop with Name and
Comment rebranded to "Install WROLPi to disk" / "Install WROLPi onto
this computer's internal drive".  We keep the upstream install-debian
wrapper script (it handles xhost + pkexec calamares correctly) and just
override the visible labels.

After the chroot creates the wrolpi user the hook copies the launcher
to /home/wrolpi/Desktop/install-wrolpi.desktop and marks it executable
so XFCE launches it without an "untrusted application" warning.

Behaviour:

  Live USB boot          XFCE autologs to wrolpi.  Install icon visible.
                         Click → polkit prompts → Calamares wizard.

  After install + reboot wrolpi-bootstrap.service sets up Postgres on
                         /media/wrolpi/config/postgresql/, runs DB init,
                         starts wrolpi-api / app / kiwix / controller.

Two known phase-3 polish items deferred:
- WROLPi branding for Calamares (logo, colors, slideshow); using stock
  Debian branding for now.
- An orphan install-wrolpi.desktop icon survives on the installed
  system (Calamares removes calamares-settings-debian, taking
  install-debian with it, but not the copy in the user's home).  A
  Calamares shellprocess will rm it in phase 3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:44:05 -06:00
Roland
3c168a4f2d Merging wrolpi-portable into debian-live-config; unifying Postgres on /media/wrolpi/config/postgresql.
Two parallel live-build configs (debian-live-config/ for the installer
ISO and wrolpi-portable/ for the Live USB) carry the same chroot hook
with subtly different fixes — the portable side had accumulated the
better implementation (npm build, SHA-pinned blobs, dynamic Postgres
version, partition-by-start-sector, controller.yaml sentinel, autologin,
branding).  This commit folds those improvements into debian-live-config/
and retires wrolpi-portable/.

The Postgres data + config directories now bind-mount to
/media/wrolpi/config/postgresql/{data,config} regardless of deployment
shape (Live USB, Calamares-installed system with external drive,
Calamares-installed system with no external drive).  Result: pulling
a WROLPi external drive from one host and plugging into another
brings the entire indexed library along — no re-indexing.  Previously
the DB lived on /mnt/persistence/postgresql-{data,config} on Portable
and on /var/lib/postgresql (root) everywhere else; moving a drive
required rebuilding the DB on the new host.

Single wrolpi-bootstrap.{sh,service} replaces wrolpi-portable-bootstrap.
The script detects deployment by probing for a live-boot medium mount:
present → Live (create persistence partition if missing); absent →
Installed (skip partition step; Postgres bind-mount and cluster init
still run).  --debian-installer is dropped from lb config; Calamares
will land in a follow-up commit as the graphical installer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:44:05 -06:00
lrnselfreliance
8f0e4cd93f
Adding support for mounting NTFS drives in the Controller. (#440) 2026-05-15 19:25:43 -06:00
Roland
e0324ce0fc Migrating Controller from port 8087 to 80 so that it is more discoverable. Reformatting Controller dashboard to make header at the top. Reformatting navigation bar to handle the new links which prevented Admin from being visible in some screens. 2026-04-09 15:01:05 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
ce0cdd3a2e Samba is installed and can be configured in the Controller. 2026-03-29 10:19:28 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
fbb71e012d Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
  (RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
  proxying to all services (API, React app, Controller, Kiwix, Help, Map),
  and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
  uses Name Constraints to restrict signing to local addresses only
  (.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
  risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
  certificate installation on iOS, Android, macOS, Windows, Linux, and
  Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
  /download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
  status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 18:48:44 -07:00
Roland
ebac4fdfa2
Removing redundant web app build during image builds. 2026-02-27 17:22:46 -07:00
Roland
ae72b9491d
Fetching latest release version when building images so they are named correctly.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-26 17:45:39 -07:00
Roland
8fab32ef75
Copying WROLPi nginx config on first startup so Controller is accessible. 2026-02-26 17:38:36 -07:00
Roland
f375e4c955
Generating nginx certs on first startup. 2026-02-24 17:14:56 -07:00
Roland
c6393437af
Building pi and debian images using branch by default, allowing specifying branch with -b option.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-02-22 15:09:15 -07:00
Roland
b8d3b5318b
Removing blob file protection during build cleanup.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-02-16 08:14:44 -07:00
Roland
adb8fefd67
Including Deno and curl-cffi which are now required for yt-dlp functionality. Installing both of these packages during upgrade. 2026-02-16 08:08:46 -07:00
Roland
50782f47cc
Checking that map dump is valid before building image. Protecting map
dump file in built image so that it cannot be modified.
2026-02-15 14:49:19 -07:00
Roland
56bdd0cf75
Starting controller on first startup.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-14 17:38:12 -07:00
Roland
18d3cfd686
Setting debian locale.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-02-13 06:49:17 -07:00
Roland
8bb240c996
Using /var/tmp to build debian image.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-12 18:13:00 -07:00
Roland
51b8972de9
Setting locale to avoid perl warnings. 2026-02-01 14:35:56 -07:00