POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between. They now write inside request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/files/tag, /untag and /delete_groups read, then wrote, in the
request's deferred transaction. SQLite fails that lock upgrade instantly
("database is locked") when another connection wrote in between. Tag and
untag returned a 500; delete_groups returned 204 after FileGroup.delete had
unlinked the files, then its commit failed in the response middleware, so
the FileGroup row outlived its files. They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deleting, restarting, clearing and batch-editing Downloads, and creating,
renaming and deleting Tags, read then wrote in the request's deferred
transaction. SQLite fails that lock upgrade instantly ("database is
locked") when another connection wrote in between, so these failed with a
500 while a download or refresh was writing. They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #846, from review:
- The reorganize conflicts endpoint read the preview, ran ffprobe, then
flushed and committed in the request's deferred transaction. It now
reads, runs ffprobe through get_or_create_ffprobe_json (no session),
and stores the results in one short request.ctx.db.write().
- Its ffprobe step never ran: it matched model_type == 'channel', but a
conflicting video's model_type is its FileGroup's model, 'video'.
- execute_reorganization's no-move path stored the new file_format with
session.commit() on the caller's deferred session, from the reorganize
endpoint and from the batch job. The scan stays a read; the store is a
short write (request.ctx.db.write() in a request, its own connection
otherwise). Called without a session it now opens a read session, so
that write does not wait on its own caller's lock.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
These endpoints read, then wrote, in the request's deferred transaction.
SQLite fails a deferred transaction's lock upgrade instantly ("database
is locked") when another connection wrote in between, so a collection
change during a refresh or download could fail with a 500. They now
write inside request.ctx.db.write(), which begins as a writer and lets
busy_timeout absorb the contention; errors are mapped outside the block
so a 4xx rolls back.
Fixes#674
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
live-config's openssh-server component rewrites sshd_config to
"PasswordAuthentication no" on every live boot, so the live session only
accepted SSH keys. An sshd_config.d drop-in sets it back to yes; sshd uses
the first value it reads and sshd_config includes the drop-ins first, so the
drop-in wins. The login is user / live (live-config's default), now noted
in INSTALL.md. check-iso.sh fails the build if the drop-in is missing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The installer opened with Debian's "debian 13" banner. Debian's
derivative guidelines say derivatives must not be presented as Debian,
and its trademark policy forbids implying endorsement, so the installer
now shows a WROLPi banner in the boot splash's style (pie logo, WROLPi,
"Off-grid library", purple accent).
The banner replaces rootskel-gtk's logo_debian.png and logo_debian_dark.png
(logo_installer*.png link to them) via config/includes.installer, which
live-build adds to the installer initrd. It is 800x75 like Debian's; the
installer stretches its edge columns to the screen width, so the edges are
solid color. check-iso.sh fails the build if the graphical installer's
initrd does not carry it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The library-ownership migration compared the owner of
/mnt/persistence/wrolpi after creating it, so on a new persistence
partition the root-owned directory always looked like an old library and
the first-boot dialog said "Updated the library's ownership for this
WROLPi version". Only re-own a library that already existed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
INSTALL.md and the onboarding wizard send people to wrolpi.local, but the
Debian image had no mDNS responder, so the name never resolved on a Debian
install or Live USB. Raspberry Pi OS ships Avahi; the ISO now does too.
libnss-mdns lets the WROLPi's own desktop resolve .local names (apt
recommends are off in the build, so it is listed explicitly). check-iso.sh
fails the build if either package is missing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Readability extraction no longer takes minutes on image-heavy pages
SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.
Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output. Applied in
both the archive container and the native (Raspberry Pi/Debian) path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders cannot be forged by page text
The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times. Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders survive Readability's lazy-image cleanup
Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder. The ~50 character placeholder triggered this, so those images
lost their inline data. The token is now 160 hex characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The archive service is PID 1 in its container and never reaps orphans.
check_output kills each command's process group, so every archive leaves
chrome, cat, and node grandchildren behind as zombies. A long-running
container accumulated hundreds of them, counting against its PID limit.
init: true runs tini as PID 1, matching the web container.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISO: hostname defaults to wrolpi, boot menu times out
The installer's hostname question defaulted to "debian", so an install that
kept the default was not reachable at wrolpi.local. A preseed in the
installer initrd now defaults it to wrolpi; the question is still shown (and
a DHCP-provided hostname still wins).
The ISO's GRUB and ISOLINUX menus had no timeout, so a Live USB waited at the
boot menu forever after any reboot or power loss. The menus now boot the
default entry ("Run WROLPi (live)") after 10 seconds.
check-iso.sh fails the build if either regresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISO: say what the wrolpi hostname is for
The image does not include avahi-daemon, so the hostname alone does not make
wrolpi.local resolve. The default still names the TLS certificate, the
SMB/NetBIOS share, and the DHCP hostname a router may publish.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ISOs before wrolpi became a system account wrote the Live USB's library on
the persistence partition as UID 1000. Upgrading such a USB with a newer
ISO (wrolpi-usb.sh upgrade) keeps the persistence partition, so the library
would belong to the live session's "user" and the services could not write
to it. Bootstrap now re-owns the library once when its owner is not this
image's wrolpi. Live USBs only: installed systems' drives are left alone.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ISO's chroot hook created wrolpi as a login user (UID 1000, password
"wrolpi", full sudo, LightDM autologin). The Debian installer creates the
user it asks for only when no UID 1000-59999 user exists, so it silently
skipped that user: the only login on an installed system was wrolpi/wrolpi,
with SSH and sudo.
wrolpi is now a system account with no password or login sudo, as on every
other WROLPi deployment. The installer creates the user it asks for (with
sudo, no autologin). The live session logs in as live-config's "user".
Desktop shortcuts moved to /etc/skel so every user gets them.
check-iso.sh now fails the build when the image has a UID 1000-59999 user.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On a Debian install the onboarding wizard offered the EFI system partition
(unlabeled vfat at /boot/efi) as the primary WROLPi drive; choosing it would
mount the boot partition over /media/wrolpi. Candidates were filtered by
the "EFI" label only, and the Debian installer leaves the ESP unlabeled.
/boot/efi is now an excluded mountpoint, and partitions with the EFI system
partition type (GPT GUID or MBR 0xef, from lsblk PARTTYPE) are excluded even
when unmounted, e.g. on a second internal disk.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
"Run repair without external drive" kept the library on the internal disk,
but the API's unmounted-media guard refuses to create or use a database
beneath /media/wrolpi when it is not a mount. After onboarding without a
drive the API restarted forever with no database, and downloads were
blocked as "media not mounted".
The Controller now records the choice as drives.library_on_internal_disk in
controller.yaml -- the only source of it. The database guard and the
media_mounted check both read that marker, so require_media_mounted keeps
its meaning and needs no change. Onboarding a drive later clears the
marker, so it is never copied onto the drive.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
install.sh only accepted Debian 12, which cannot run this version, and
rejected Debian 13. It now requires Debian 13 or newer on both PCs and
Raspberry Pis. The install scripts point to the drive wizard instead of
manual mounting.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Installed Debian systems keep Caddy
The Debian installer purges every package listed in the ISO's
live/filesystem.packages-remove. live-build lists there every package
installed after the package-lists install pass, which includes anything a
chroot hook installs with apt -- and the hook installed Caddy that way.
An installed system therefore had no Caddy: wrolpi-bootstrap failed copying
the Caddyfile, the API/app/Kiwix were never enabled, and HTTPS never came up.
Caddy now comes from package-lists/, with its repository and signing key in
config/archives/ (installed into both the build chroot and the image).
check-iso.sh fails the build when the installer would remove any package,
and can be run by hand against any ISO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISOs install Caddy from Caddy's repository
Moving caddy into package-lists/ made apt read config/archives/ before
ca-certificates was installed, so the HTTPS fetch of Caddy's repository
failed certificate verification. apt ignores a failed repository, and the
image silently got Debian's much older caddy 2.6.2 instead of 2.11.
ca-certificates is now part of the bootstrap, and Debian's caddy is pinned
to -1, so an unreachable Caddy repository fails the build and installed
systems never switch to Debian's caddy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nothing started the hotspot since hotspot_on_startup moved out of the
API, and NetworkManager does not autoconnect the Hotspot connection.
The Controller now starts it once per boot unless hotspot.on_startup is
false; a legacy wrolpi.yaml hotspot_on_startup: false is migrated.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The onboarding wizard deleted /media/wrolpi/config before mounting a
drive, and the shadowed data check ignored config/. A library kept on
the internal disk keeps wrolpi.db and its configs there, so onboarding
a drive later deleted them without warning. Only a config/ holding
nothing but the ssl/ CA is now treated as transient.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Download and Upload are disabled until a full refresh completes, and a
fresh install never had one until the user found the refresh link.
Startup now queues it, and the dashboard says why Download and Upload
are disabled meanwhile.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* INSTALL.md points to the Controller drive wizard
Debian installs are Debian 13 from the amd64 ISO. The Raspberry Pi
steps no longer hand-edit fstab or run repair.sh; a shared First Boot
section covers the drive, certificate trust, and the first refresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* INSTALL.md: Debian installs use the drive wizard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wrolpi-bootstrap.sh created controller.yaml on installed systems, which
hid the onboarding wizard. Installed systems now wait for the wizard,
whose repair.sh finishes setup.
"Run repair without external drive" now writes controller.yaml before
starting repair, so the wizard does not return after repair.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Transcode, job, and Kiwix notifications send no URL, but every user notify
toast said "Click here to view the shared page" and opened /null on click.
Only a notification carrying a URL is followable now.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Without them a model guessed joins: it joined video.channel_id to collection.id
(it is channel.id), and the query silently returned the wrong channel names.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
systemd prints the start timestamp in local time, which was subtracted from
utcnow(), so elapsed_seconds was off by the UTC offset (6 hours in MDT). Ask
systemd for a Unix timestamp instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
upgrade.sh only installs a branch tip signed with the WROLPi key; the tip was a
GitHub squash merge (#820).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read-only SQL query endpoint for advanced users
POST /api/db/query runs one read-only statement; GET /api/db/schema describes
the tables. Both are refused unless `allow_database_query: true` is set in
wrolpi.yaml (there is no UI for it). The connection is opened with mode=ro and
an authorizer that allows only reads; queries are interrupted after 10 seconds,
rows are capped, and long values are truncated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Database queries cannot be enabled through the API, and use bounded memory
The config update API refuses to change `allow_database_query`; only an edit of
wrolpi.yaml on the WROLPi can enable queries. Query connections refuse values
over 32 MB and rows over 100 columns, rows are read one at a time, and the
response stops at 5 million characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The database query setting can be changed like any other config
WROLPi is meant to be as secure as a bookshelf in your home; a LAN user who can
reach the API is trusted, so gating one key in the config API adds nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Database queries can search FTS5 tables on SQLite 3.46
FTS5 reads `PRAGMA data_version` through the authorizer when it opens a table
on SQLite 3.46 (Debian 13), so every MATCH and the schema endpoint failed with
"vtable constructor failed". A denied VACUUM INTO also explains itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A config status test attaches the shared context it reads
It passed only when an earlier test in the same worker had attached it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A repr runs inside log calls. Reading a relationship or an expired column there runs a query, and
on an object whose session is gone it raises DetachedInstanceError instead of logging. Archive's
repr also stat'ed its SingleFile file on disk.
`wrolpi.common.loaded()` follows attributes through state that is already loaded (primary keys
from the identity key) and never runs SQL. Every model repr uses it; with everything loaded the
output is unchanged, except Video and Archive now show the FileGroup's primary path.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest. A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted. The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.
Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before. The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`get_primary_file` returned the first video/audio file in whatever order it
was given, so a group holding both `<stem>.mp4` and a leftover video-only
`<stem>.f313.webm` could make the leftover the primary file. The Video then
pointed at a file without audio, and a download of it was deferred with
"Download did not contain audio stream" on every retry.
Skip yt-dlp per-format files (`.fNNN` stems) when a merged media file is in
the group. A group of only format files still gets one as its primary.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A merge that never finished leaves `<stem>.f313.webm` (video-only) next to
where the merged file belongs. A later download of the same video globs
every `<stem>.*` file into its FileGroup, but only `.part` files and format
files with the *same* extension as the merged video were deleted. A `.webm`
leftover beside a new `.mp4` survived, joined the new FileGroup, and could
become the Video's file.
Delete every `<stem>.fNNN.*` file in the group, whatever its extension, which
also covers WROLPi's poster/ffprobe sidecars of an indexed leftover and
yt-dlp's `.ytdl` state files.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nothing uses sqlalchemy.ext.asyncio; an AsyncSession pilot on the request path was measured on a
Pi and dropped. SQLAlchemy 2.0 still requires greenlet on aarch64 and x86_64, so installs on the
Pi, Debian and Docker are unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.
Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* File downloads: return plain values from upsert_file
Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`. `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone. SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends. Failed downloads retry, re-downloading their files.
`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Archive uploads: return plain values from singlefile_to_archive
Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved. It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query. An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async. Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.
Collection, CollectionItem and Channel dict() now run inside the serializer guard. Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
the session.
Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CustomJSONEncoder looked up the media directory and called Path.relative_to
for every path it encoded. A response with hundreds of paths (an Archive's
page with 51 history entries holds 416) spent most of its encoding time
there: 18.8 ms of 19 ms per response, measured on a Mac. The encoder now
looks up the media directory once per response and strips it as a string
prefix: 1.5 ms for the same response.
Paths outside the media directory, relative paths, and siblings that share
the media directory's name prefix are still returned unchanged; the media
directory itself and "." are still "".
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive. An
Archive with 51 history entries ran 107 statements per request. Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
