Filtering media was keyed on `data-theme="night"`, which made it a property of
the theme that no user could turn off. Some people want their media tinted to
match the interface; some want to see the actual image.
A theme now declares a filter and whether it starts on, and the user decides:
night night-red on by default. An unfiltered thumbnail undoes dark
adaptation, which is the whole point of the mode.
amber amber-mono off by default. Amber is a look, not a night-vision
aid, so media keeps its real colors until asked for.
light - no filter offered, so no toggle is shown.
dark -
The choice is stored per theme, so wanting amber's tint says nothing about
wanting night's. The toggle lives in the theme picker, which puts it on the
Settings page and in the gallery at once.
Mechanically the rules now key on a `data-media-filter` attribute rather than
`data-theme`: a rule keyed on the theme could not be turned off. index.html
stamps that attribute before first paint alongside data-theme -- without it,
night mode paints a screen of unfiltered thumbnails while the bundle arrives
over a hotspot, costing the user the adaptation the theme exists to protect.
Tests cover both the attribute rule and the fact that the inline script's
duplicated ids and defaults still agree with names.ts.
The amber matrix is the same luminance collapse tinted to #ff9500 (red full,
green 0.584, blue zero) with a gentler gamma than night's: amber carries far
more luminance and clips to a flat wash otherwise.
The gallery gains a Media filtering section showing the WROLPi logo filtered,
beside an unfiltered copy that opts out with `night-unfiltered`. That copy is
behind a click on purpose -- an unfiltered image is a bright patch, and night
mode should not spring one on someone reading in the dark.
ui-design.md gains the media filtering contract it never got in Phase 0,
including the note that a Phase 3 custom theme names an existing filter id and
cannot ship its own matrix, since that would be code rather than data.
Phase 2, part 1. Three files leave Semantic UI (baseline 75 -> 72), and the
components they needed join the library and the gallery:
Header Semantic's shape (as/icon/subheader/dividing). 47 files use
<Header>, and the level now picks the size, so the type scale
stays a scale instead of each call site choosing a font size.
Message gains an icon and a dismiss button.
ActionInput input and button sharing one outline, for <Input action={...}/>.
Loading centered loader with a caption, replacing the
<Loader active inline='centered'> + wrapper div in 19 places.
Modal becomes compound - Modal.Header/.Content/.Actions, plus
Semantic's open, closeIcon, and size names - because 34 files
are written against that shape.
test-utils mounts MantineProvider with ThemeProvider's exact configuration:
migrated components render Mantine internals and every one of their test files
would otherwise need its own provider.
One Semantic bleed found and fixed: semantic.min.css sets
`h1,h2,h3,h4,h5 { font-family: Lato }` and loads after Mantine's element rule,
so headings ignored the theme's body face. Only visible in amber, where the
body is mono and the headings were not. A class-level restatement outranks it;
the rule comes out when Semantic does.
Verified: 925 jest tests, clean production build, and a night-mode audit of
every painted pixel in the gallery (red-only, one known false positive). The
Configs backup and restore flow was exercised against the real API.
Addresses the automated review on #583; all three were real.
The night filter only applied to elements tagged `.media`, and nothing in the
app carried that class - so selecting Night filtered the interface but left
every video, thumbnail, PDF and map full-colour, which is precisely the thing
the mode exists to prevent. The Settings copy promised otherwise, so it was
also wrong.
It now targets the element types instead of an opt-in class, because night mode
has to fail safe: a library of thousands of files reaches the screen through
many components, and one that forgot to opt in would undo half an hour of dark
adaptation. `.media` still works for wrappers that must be filtered as a unit,
and `.night-unfiltered` opts out. The filter is restated on `:fullscreen`, which
a filter on an ancestor stops reaching. Verified on the videos page: all 25
media elements filtered.
Three of the five theme choices rendered a fallback glyph and logged errors,
because the icon map was built by grepping literal name='...' strings and the
navigation picker passes its icon through a variable. Adds sun, moon and
lightbulb, plus a test asserting every theme choice resolves to a real icon.
The pre-paint script wrapped the `matchMedia` fallback in the same try as the
storage read, so a device with storage blocked painted light even when it
prefers dark - recreating the flash the script exists to prevent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gallery at /theme-sample is now a normal route rather than a development
one, and Settings links to it, so a user can see what a theme actually looks
like before committing to it. Every new component belongs in the gallery: it is
how the design gets reviewed, and how a component that only looks wrong in one
of the four themes gets caught.
Settings gains a Theme section with the new ThemePicker, which explains what
each theme is for - notably that night mode filters videos, documents and maps
to red as well, not just the interface.
Theme names and picker options move to themes/names.ts so the component library
can read them without importing components/Theme.tsx, which still pulls in
Semantic UI. Theme.tsx re-exports them, so existing call sites are unchanged.
The navigation bar's compact picker now reads the same list, so the two cannot
drift apart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds src/components/ui: buttons, messages, labels, progress, status, tables,
form controls, panels, cards, statistics, modals, tooltips, tabs, and icons.
Everything resolves its colors through theme tokens, so no component branches
on the current theme, and Semantic UI keeps rendering the app meanwhile.
Mantine is bridged rather than restyled: its own CSS variables are pointed at
our tokens. Four details were not obvious, and each caused a visible defect
in night mode before being fixed:
- Structural colors must go in the resolver's light/dark blocks. Mantine
emits its values under [data-mantine-color-scheme], which outranks the
:root block that `variables` becomes, so its grays were winning.
- The dark and gray palettes are mapped shade by shade to tokens. Mantine
reads them directly for table borders, zebra stripes and input chrome;
they can be neither flattened nor left alone.
- Mantine writes --button-* as inline styles, which no stylesheet can
outrank, so the dashed danger treatment sets real properties instead.
- Switch colors are written onto inner elements, so they need scoped CSS
rather than a variable on the root. Adds a --knob token for control
handles, because Mantine's default thumb follows theme.white, which is
near-black in the dark-based themes.
An audit of every painted pixel in the /theme-sample gallery now finds no
non-red channel in night mode.
Icons come from @tabler/icons-react with the 102 Semantic names the app uses
mapped to components; they stroke with currentColor, so they follow the theme
without any rule of their own. Toasts keep the react-semantic-toasts-2
signature so those call sites migrate by changing an import.
TypeScript 5 is required: Mantine's declarations use `const` type parameters,
which 4.9 cannot parse (Mantine 7 is the same, so downgrading is no escape).
That needs two npm overrides to keep clean installs working, both verified
against a Docker build: react-scripts declares peerOptional typescript ^3||^4,
and fork-ts-checker 6.5.2 assigns typescript.performance.mark, which TS 5
made getter-only - crashing the dev server's type checker on every rebuild.
semantic-baseline.test.js ratchets the removal: it fails if a file not already
listed imports Semantic UI. An ESLint rule would have printed 75 warnings per
rebuild, and CI unsets CI= for its build step, so it would guard nothing there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every color now resolves through CSS custom properties on
html[data-theme=...], so a theme is a table of values rather than code.
Four themes ship: light, dark, night (red hues only, to preserve dark
adaptation off-grid) and amber (monochrome amber terminal).
Fonts are exposed as two roles, --font-body and --font-mono, which a theme
assigns a bundled face to; amber uses the mono face. Faces must be
self-hosted, never fetched from a CDN, so the app keeps working offline.
Night mode also filters media that cannot consume tokens - video, images,
PDFs, embedded pages, canvases - through an SVG color matrix that maps
luminance onto the red channel. Tag such an element with `.media`; a filter
on an app-level ancestor would create a containing block and break fixed
positioning.
ThemeProvider persists the choice and index.html stamps data-theme from
localStorage before the bundle loads, so the first paint already matches the
saved theme on a slow connection. The OS preference is followed only while
"system" is selected; night and amber are explicit choices. The nav toggle
becomes a five-item picker.
Semantic components still render themselves, so only the shell is
token-driven for now; the inverted props stay in place until each component
is migrated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The toggle-hook de-duplication replaced a range of customHooks.js that also
contained these two unrelated hooks, breaking the production build. Restored
byte-identical from master; exports now match master exactly.
Jest missed this because it only compiles what tests import — `npm run build`
is what catches a missing export.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hotspot, Bluetooth, Desktop, VNC and Throttle toggles had each been written
out longhand at four layers. Collapse the shared parts:
- useSubsystemToggle() replaces five near-identical hooks. This also fixes a
latent bug: only useDesktop/useVnc restored state when a request failed, so a
failed Bluetooth/Throttle/Hotspot toggle left at null and — since the
status poll sees no change — the toggle stayed disabled and read as
unsupported until it remounted.
- SubsystemToggle renders all five, with the Desktop's stop confirmation and
VNC's desktop gating as declared options rather than bespoke copies.
- The fallback UI's four update*/toggle* pairs become one config-driven
implementation (-100 lines).
- _raise_for_action() centralises the 13 endpoint failure checks, including
VNC's 409-vs-500 split.
Deliberately not shared: the six get_*_status_dict() builders (payloads
genuinely differ) and the per-subsystem mechanisms (nmcli/rfkill/sysfs/systemctl).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Greptile P1 (security): with both stacks installed, selection keyed only off
wayvnc's unit and config. If RealVNC was the one running, status reported VNC
as off and stop targeted wayvnc — leaving the screen remotely accessible while
the UI claimed VNC was off. Reproduced on 10.0.0.8: RealVNC serving on 5900
reported as running=false.
Now every installed backend is reported with its own active state: status is on
when any backend runs and names the one actually serving, and stop stops all
running backends. wayvnc is still preferred for starting.
Greptile P2 (latency): both units are probed in a single multi-unit
`systemctl show`, and the status dict no longer re-probes, so VNC adds one
subprocess to /api/stats rather than three.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a VNC toggle to the Controller (both UIs) that starts/stops whichever VNC
backend is installed: wayvnc for Wayland (Raspberry Pi OS), RealVNC for X11.
Runtime-only, like the Desktop toggle: the unit is never enabled, so VNC does
not return after a reboot. Users who want it permanently on can enable it with
raspi-config.
VNC serves the desktop session, so starting it without a desktop gives clients
a blank screen. The gate lives in the backend, not just the UI, so curl callers
get it too (409, distinct from a systemctl failure). Stopping is never gated so
a running VNC server can't be stranded on with no way to turn it off.
Backend selection deliberately avoids raspi-config's live-compositor probe
(pgrep labwc), which reports X11 — and would pick the wrong backend — whenever
the desktop is stopped. wayvnc also requires its config to exist; that is
checked directly rather than via the unit's ConditionResult, which reports 'no'
for any unit that has never been started.
Extracts _unit_properties()/_systemctl_unit() so Desktop and VNC share one
systemd probe and start/stop path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stopping the display manager left the Pi's monitor black with only a blinking
cursor. The console VT is already in the foreground with a getty and a live
shell on it (verified: /dev/vcs1 holds the login banner and prompt) — the
display manager just hands the display back without the console being
repainted, so nothing is visible.
Activate the console VT, then bounce off another VT to force a modeset and
redraw, and print a short message telling the user the Controller stopped the
desktop and how to start it again.
Best-effort throughout: skipped when no getty is on the console VT, a chvt or
console-write failure never fails the stop, and being stranded on the repaint
VT is reported rather than claimed as success. Verified on 10.0.0.8 hardware.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
git rev-parse --abbrev-ref HEAD reports "heads/release" when a stale
tag named release exists alongside the release branch (tags outrank
heads in git ref resolution). check_for_update then looked up the
nonexistent origin/heads/release and permanently reported no updates
available, and the Controller UI pre-filled "heads/release" as the
upgrade branch.
Use git branch --show-current everywhere instead; it never
disambiguates. A detached HEAD (empty output) now defaults to the
release branch for upgrades; a genuine git failure still returns None.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Greptile P1: on failure the status poll never re-syncs (the desktop did
not change), so the toggle stayed disabled until remount.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a Desktop toggle to the Controller (both UIs) that starts/stops
display-manager.service. Fail open: the unit is never enabled/disabled,
so the desktop always returns on reboot; users who want it disabled
across reboots can set the default target themselves.
Rename the hotspot and bluetooth actions to reflect their real behavior:
- enable/disable_hotspot -> start/stop_hotspot (/api/hotspot/start|stop)
- enable/disable_bluetooth -> unblock/block_bluetooth (rfkill semantics,
/api/bluetooth/unblock|block)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add daily_limit_reached to the DownloadManager summary (/api/status):
True when due (new, non-recurring) downloads exist and every one of
them is blocked by the global or per-domain daily download limit.
Show a "Max Daily Downloads Reached" banner on the Downloads page when
set (and the manager is neither disabled nor stopped).
Previously a queue stalled by the daily limit looked broken: the UI
reported 0 pending with no explanation, and the skip reason was only
visible at DEBUG log level.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Reset a saved protocol the newly probed device does not support
instead of offering it (both UIs); a successful probe is
authoritative, the saved value is only kept when probing fails.
- Ignore out-of-order protocol probe responses when switching devices
quickly (both UIs).
- Verify the WPA2 rollback after a failed WPA3 activation; report that
the hotspot may be down instead of claiming the revert succeeded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe the selected WiFi device with iw to determine which hotspot
protocols it can host: WPA2 requires AP mode; WPA3 requires SAE in AP
mode (SAE_OFFLOAD_AP, or a mac80211 driver with the CMAC cipher for
software SAE). The Raspberry Pi's brcmfmac supports neither, so Pis
correctly offer WPA2 only (verified against real Pi 4/Pi 5 iw output).
The protocol is saved as hotspot.protocol in controller.yaml.
enable_hotspot() now always sets the Hotspot connection's key-mgmt
explicitly (sae + required PMF for WPA3) so a reused profile cannot
keep a stale protocol, and reverts to WPA2 if the driver rejects SAE.
New GET /api/hotspot/protocols endpoint; protocol dropdowns added to
both the ControllerPage React UI and the fallback controller UI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RPi 5 firmware stops the fan entirely when cool, so a 0 RPM
reading is normal and must not be hidden. On multi-fan machines,
prefer a CPU-labeled hwmon group over faster case fans.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RPi 5 fan connector exposes a tachometer via the pwmfan hwmon
device. Report it as fan_rpm in cpu_stats and show it in both the
Controller fallback UI and the React Status page. Hidden when no fan
is present.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review feedback: skip the service stop when the primary mount is already
unmounted (a no-op request must not cause an API outage), and restart
wrolpi-api (best effort) when umount fails since the drive is still
mounted and usable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Support swapping in a backup drive: /api/disks/unmount now accepts the
primary mount (previously rejected as reserved). It stops wrolpi-api so
open files are released, then umounts directly since the reconciler
never manages the primary mount. The /etc/fstab entry is left in place
so a reboot restores the mount.
Both UIs now offer Unmount for the primary mount and warn that the API
service will be stopped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The React ControllerPage partitions services on `service.group`, but
ServiceStatusResponse did not declare the new kind/group fields, so
Pydantic silently stripped them from /api/services and the React UI
rendered one flat, ungrouped table. The fallback UI was unaffected
because it is server-rendered without the response model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wrolpi-app crash-loops with 203/EXEC when /usr/local/bin/serve is missing.
Two independent causes, both fixed here:
1. The `single-file --version ||` guard in the install scripts only installed
serve (and readability) when single-file was ABSENT. On a machine that
already has single-file (a prior install or a baked image), serve was
skipped, and re-running install.sh could never repair it. Install each npm
global independently based on its own presence.
2. The service hardcoded /usr/local/bin/serve, which breaks when npm's global
prefix puts serve in /usr/bin (depends on how node was installed). Resolve
serve via PATH with `/usr/bin/env serve`.
Verified on a fresh Raspberry Pi OS Trixie install (10.0.0.8): wrolpi-app is
active with 0 restarts and serves 200 with the PATH-resolved ExecStart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses two processing_domains leaks raised in review; each could permanently
exclude a domain and eventually stall the scheduler until restart.
1. Claim-transaction failure (Greptile P1): `_add_processing_domain` mutated shared,
non-transactional state before the claim transaction committed. A commit failure
skipped dispatch but left the domain claimed forever. Fix: defer both the
processing_domain add AND the signal dispatch to after the transaction commits;
track claimed domains in a local `claimed` list so the loop's own dedup /
concurrency cap is unchanged.
2. Row deleted before dispatch (Greptile P1): if a claimed download was deleted
between commit and the signal running, `Download.find_by_id` raised before
`download_domain` was set, so the handler's `finally` released nothing. Fix:
derive `download_domain` from the URL (`urlparse(url).netloc`, the same value the
dispatcher claimed) up front, so cleanup always runs.
Regression tests written bug-first (both fail before this change):
- test_signal_releases_processing_domain_when_download_missing
- test_dispatch_releases_claimed_domains_when_transaction_fails
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_dispatch_new_downloads` claimed downloads inside a `get_immediate_db_session()`
(BEGIN IMMEDIATE) transaction and then `await`ed `api_app.dispatch(...)` while that
write transaction was still open. Under NullPool (production, one connection per
session) the dispatched `signal_download_download` opens its OWN connection and —
inheriting the thread-local `_immediate_txn` flag — also issues BEGIN IMMEDIATE,
which blocks on the write lock the outer transaction still holds until busy_timeout
(30s) expires: `(sqlite3.OperationalError) database is locked [SQL: BEGIN IMMEDIATE]`.
Every dispatch cycle died on the first download, so nothing downloaded at all.
Fix: claim the downloads inside the transaction, collect (id, url), and dispatch the
signals AFTER the `with` block commits and releases the write lock.
Diagnosed live on 10.0.0.9 (downloads had silently stopped); with this change the
14 stuck recurring downloads drained immediately and no lock errors recurred.
Why tests missed it: the `test_session` fixture returns a single shared
(engine, session) for every `_get_db_session` call, so all sessions in a test ride
one connection -- the second-connection write-lock contention that causes the
deadlock cannot occur. Added a regression test asserting the invariant instead: the
download signal must be dispatched with no immediate transaction active.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigating downloads stalling on 10.0.0.9 (recurring downloads going overdue
without dispatching). The dispatch path had blind spots that hid the cause, so add
targeted DEBUG logging, change-detected where the cadence would otherwise flood:
- blocked_download_reason(): single source of truth for can_download that returns
*why* downloading is blocked. dispatch_downloads() logs the reason (and the
unblock transition) only when it changes, instead of silently returning every 5s.
- is_within_download_window(): logs the OPEN/CLOSED transition with the computed
local time, window, and timezone — surfaces any clock/timezone mismatch (the box
shows system UTC-5 vs configured America/Denver) at the moment of evaluation.
- _dispatch_new_downloads(): per-cycle snapshot (new count, processing_domains,
daily global_count, download_wait) plus a bounded tally of why due downloads were
NOT dispatched. Prime suspect is a leaked processing_domains slot (a dispatch that
dies without cleanup blocks that domain, and once 4 leak, everything) — that path
previously had no logging.
Behavior-preserving: can_download semantics are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Preserve MEDIA_DIRECTORY/WROLPI_DB across the sudo re-exec (sudo
strips them from the environment, discarding the override).
- Reject unknown options and a second positional path with a usage
error instead of silently checking the wrong database.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It has a python3 shebang but was missing the executable bit; all other
scripts in ./scripts are executable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRAGMA quick_check reads every page of the database, so on a large DB or
a slow (e.g. USB spinning) disk it can run for many minutes, making
help.sh (run by the wrolpi-help-script service) appear to hang.
Drop the quick_check from help.sh's fast startup diagnostics and add a
standalone scripts/check_database_integrity.sh for on-demand checks.
The new script runs sqlite3 as the wrolpi user (WAL -shm access),
reports elapsed time, and supports --full for an exhaustive
integrity_check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Controller listed every wrolpi-* service in one flat table, which was
confusing for new users who couldn't tell always-on infrastructure from
one-off/dev/maintenance units.
Services are now split into two groups computed once in the backend and
exposed as `service.group`:
* Core Services - persistent services enabled at boot (api, app, caddy,
kiwix, ...), the normal running system.
* Optional & Maintenance - one-off/dev/maintenance tasks (bootstrap, repair,
upgrade, *-dev, discovered wrolpi-*), plus persistent
services the user has disabled at boot (e.g. Samba with
no shares configured).
Rule: core iff kind == "persistent" and enabled; otherwise optional. Group
membership only changes when the user toggles Boot, never on start/stop or
polling, so rows stay put. Within each group, failed/stopped services sort
first so problems stay visible.
Feature parity across all three render paths: the React ControllerPage, the
server-side Jinja template, and the client-side poll refresh. Docker
containers have no group field and fall back to a single ungrouped list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fresh installs (and SSH sessions that forward LC_* variables from the client)
can reference an ungenerated locale, causing apt/perl/python to emit
"Cannot set LC_CTYPE to default locale" warnings throughout the install log.
Export LC_ALL=C.UTF-8 for the install run. C.UTF-8 is built into glibc, always
present, and locale-neutral, so it silences the warnings without generating a
locale, without adding a dependency, and without changing the user's configured
locale (e.g. an en_GB.UTF-8 system stays en_GB.UTF-8).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Controller venv (/opt/wrolpi/controller/venv) was only ever created by
upgrade.sh and reset_virtual_environments.sh, neither of which runs during a
fresh install. install_raspberrypios.sh and install_debian_12.sh created only
the main API venv, so on a clean machine wrolpi-controller crash-looped with
203/EXEC ("Unable to locate executable /opt/wrolpi/controller/venv/bin/python")
because repair.sh unconditionally starts the service.
Create the Controller venv inline right after the main venv, mirroring how the
main venv is already built in these scripts. Verified on a fresh Raspberry Pi OS
Trixie install (10.0.0.8): wrolpi-controller comes up active with 0 restarts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes from PR review:
- `PRAGMA journal_mode=WAL` returns the mode SQLite actually selected and can leave the
prior rollback journal without raising. Confirm the returned mode instead of assuming
the request succeeded, so a silent WAL refusal correctly falls back to TRUNCATE +
synchronous=FULL rather than caching WAL with the weaker synchronous=NORMAL.
- `fuseblk` (block-backed FUSE such as ntfs-3g) has unreliable byte-range locking that
can corrupt the database under the workers' concurrent access. Move it out of the
WAL-incompatible allow-list into an UNSAFE_LOCKING_FS_TYPES refuse-list (renamed from
NETWORK_FS_TYPES). NTFS on a modern kernel mounts via the native `ntfs3` driver, which
stays allowed; exFAT is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media drive formatted for Windows compatibility (exFAT/FAT/NTFS) cannot support
SQLite WAL: it needs mmap'd shared-memory those filesystems don't provide, so
`PRAGMA journal_mode=WAL` raised `(sqlite3.OperationalError) disk I/O error` on
every connection, taking down the whole API (observed on 10.0.0.8; the tell was
`Media directory has the wrong permissions: 0o40777`, since these filesystems can't
store Unix permission bits).
The engine now tries WAL first and, on failure, falls back to a TRUNCATE rollback
journal with synchronous=FULL (crash-safe, since WROLPi is off-grid). The decision
is memoized per-engine so WAL isn't re-attempted or re-logged on every fresh
NullPool connection. The bootstrap environment check no longer refuses these
drives; it warns once that write-concurrency is reduced and recommends ext4.
Network filesystems remain refused: their locking is unsafe regardless of journal mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The download dispatcher (_dispatch_new_downloads) reads the `new` downloads
then claims each by writing last_download_attempt, all in one ORM
transaction. Since ORM transactions begin *deferred*, that write is a lock
upgrade (SHARED->RESERVED). SQLite skips busy_timeout for lock upgrades to
avoid deadlock, so under concurrency the write failed instantly with
"database is locked" instead of waiting out the 30s busy_timeout. This
surfaced repeatedly on the SQLite build after the Postgres->SQLite cutover.
Add get_immediate_db_session(), a committing session that takes the write
lock up front (BEGIN IMMEDIATE) so busy_timeout absorbs the contention --
mirroring what the raw-cursor get_db_curs path already does. Plain
get_db_session stays deferred so ordinary writes and reads keep WAL
read-concurrency; only read-then-write call sites opt in. Convert the
dispatcher to use it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triage of Greptile findings on the default-Zim / configurable-directory PR:
- Stale config on restart (P1): settings update() only queues an async
background_save, and the restart switch has no ordering guarantee (the switch
worker pops LIFO), so start_kiwix_serve.sh could read the old zims_destination.
Save the config synchronously before activating the restart switch.
- YAML null (P1): read_config_value.sh now treats an unquoted null/~ as unset
(returns the default), matching PyYAML/the Python config layer. A quoted
"null" remains the literal string.
- Path traversal (P1/security): the settings API now rejects a zims_destination
containing "..", and start_kiwix_serve.sh defends in depth by falling back to
`zims` for any absolute/traversal/empty value, so Kiwix can never scan or
rewrite library.xml outside the media root.
- Reproducible build (P2): pin the zim-tools image to 3.7.0 instead of :latest.
Tests added for null handling, traversal rejection (API + script).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image builds (pi-gen, debian-live, Docker) already ship wrolpi/blobs, but a
fresh install on existing Debian and an in-place upgrade did not pick up the
default Zim. repair.sh runs on both paths (both install scripts and upgrade.sh
call it) and already manages /opt/wrolpi-blobs immutability, so copy the
committed blobs there -- after the immutable flag is cleared and before it is
re-applied. Local copy only, preserving repair.sh's no-internet guarantee.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related gaps in how WROLPi serves Kiwix content:
1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
Kiwix. Every image now ships a small built-in default Zim that Kiwix falls
back to, telling the user to download real Zim files.
2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
`zims_destination` (which the Python side already honors).
Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
non-Python scripts can reliably read top-level wrolpi.yaml scalars. Also a
config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
(ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback. It needs no
config read: it already scans the whole mounted media dir, and
zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
serves the new directory without a reboot.
Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Kiwix section only checked library.xml and the Caddy HTTPS proxy on 8085,
which made a common failure hard to diagnose: kiwix-serve down while the
wrolpi-kiwix unit still reads "running" (its ExecStart is a bash wrapper), and
corrupt/incomplete zims silently missing from the library.
- Check the wrolpi-kiwix systemd unit exists and the service is up (parity
with the api/app/controller sections).
- Check kiwix-serve is actually listening on 9085 and responds over plain
HTTP there, isolating a kiwix failure from a Caddy proxy failure on 8085.
- Compare *.zim files on disk to <book> entries loaded in library.xml; a
shortfall flags corrupt or incompletely-downloaded zims.
Verified against a healthy Debian 13 RPi (10.0.0.9).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When downloading is enabled at startup, the download worker races a burst
of concurrent SQLite writes on the single DB file: retry_downloads() has
just marked every download 'new', RSS catalog refreshes call
create_downloads() (which fires its own background dispatch_downloads()),
multiple download workers claim downloads, and save_downloads_config
re-saves the config. dispatch_downloads reads-then-commits in a deferred
transaction, so when another writer commits mid-cycle it fails with
SQLITE_BUSY_SNAPSHOT ("database is locked", not absorbed by busy_timeout).
This is self-healing -- the perpetual worker retries on its next cycle and
downloads succeed -- but the raised OperationalError flooded the logs with
full tracebacks and tripped the Sanic traceback renderer (tracerite) into
its "Variable inspector failed" secondary errors.
Split the claim-and-dispatch DB work into _dispatch_new_downloads() and
have dispatch_downloads() swallow a transient lock (debug log, return) so
the next cycle retries cleanly. Non-lock database errors still propagate.
The split keeps the retry decision outside the transaction that also
awaits the download signal dispatches, so a lock is never handled by
re-dispatching (no double-dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A refresh on a large library aborted the whole doc_modeler run on the
first lock error, so the ~20k stranded docs never got modeled.
doc_modeler held one write transaction across a whole batch of 10 docs
while doing slow per-doc extraction (a PDF/epub can be tens of MB). When
another connection committed mid-batch, the deferred transaction failed
to upgrade to a write with SQLITE_BUSY_SNAPSHOT ("database is locked",
which busy_timeout does not absorb -- the same trap that motivated the
short-transaction rewrite of apply_indexers). The per-item error handler
then did `logger.error(f'... {file_group}')`, which lazy-loaded an
attribute on the now-rolled-back session and raised a *second* exception,
escaping the handler and aborting the entire modeler.
Restructure:
- Pick each batch of candidate ids in a short read-only transaction, then
model each doc in its own short write transaction. A lock/error on one
doc rolls back only that doc and cannot poison the rest of the run.
- Retry a doc a few times on "database is locked" -- a fresh transaction
gets a fresh snapshot and succeeds.
- Log the plain file_group id on failure; never touch the ORM object on a
possibly rolled-back session.
- Track only *failed* ids for the termination guard (successes are
excluded by the Doc.id IS NULL gate), keeping the notin_ set small and
well under SQLite's bound-variable limit.
Adds test_doc_modeler_survives_database_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The zim Docker image is FROM debian:13, whose kiwix-tools 3.7.0 rejects the
literal "--address 0.0.0.0" ("IP address is not available on this system"),
so the container failed to start after loading the library. Omit --address;
kiwix-serve then defaults to binding all interfaces.
This mirrors 9e92732a, which applied the same fix to the systemd
start_kiwix_serve.sh path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: the new `Doc.id IS NULL` gate keeps matching a
doc whose _model_doc fails without persisting a Doc row. With
>=DOC_PROCESSING_LIMIT such docs, every batch re-selected the same failing
files, `processed` never dropped below the limit, and the loop hung --
starving later docs. Reproduced with a patched _model_doc that always
raises.
Track attempted FileGroup ids per run and exclude them from the query, so
the candidate pool strictly shrinks and the loop always terminates. Failed
docs are retried on the next refresh (unchanged behavior).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc_modeler gated its query on `FileGroup.indexed == False`, but
`indexed` is a full-text-search flag that `apply_indexers` sets on every
FileGroup during a refresh. Because the refresh pipeline runs
apply_modelers -> apply_indexers, any doc file that reached apply_indexers
without first receiving a Doc row (modeler errored, was interrupted, or
files imported already-indexed) got `indexed=1` and was then excluded by
the `indexed == False` gate forever -- leaving it with model=NULL, no Doc
row, and invisible in /api/docs.
On one library this stranded 20,795 PDFs/epubs/mobis (only the 10
individually-tagged docs, modeled via the per-file do_model path, showed
up). No doc-mimetype files were left with indexed=0, so nothing would
ever recover them.
Gate the modeler on "no Doc row yet OR explicitly flagged for
re-indexing" instead, so already-indexed docs are picked up on the next
refresh while preserving re-index behavior and skipping already-modeled
docs. Self-healing: a global refresh now models the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A corrupt or partial *.zim (e.g. an interrupted download) is rejected by
kiwix-manage ("Cannot add zim ..."). start_kiwix_serve.sh could then leave
Kiwix down and reported a misleading "Could not find any Zim files" — and
if the bad zim was the only one, exit 2 killed the service with no restart.
- Import each zim independently; log and skip a failed add so the remaining
valid zims are still served.
- Always ensure a library.xml exists (empty if needed) and exec kiwix-serve
unconditionally, so a corrupt/absent zim never stops the server starting.
- Fix the mount-wait loop, which never incremented COUNT (infinite wait
instead of a 30s timeout).
- Add MEDIA_DIRECTORY override so the script is testable.
- Add Restart=on-failure to wrolpi-kiwix.service (matches wrolpi-api.service)
so transient failures self-heal.
- New test harness runs the script with fake kiwix-manage/kiwix-serve shims.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
