domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format. An unchanged
import writes nothing.
A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it. An invalid entry (name
or directory) is skipped, counted, and its domain kept. Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).
Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml imports by diffing
repos.yaml declares one section, its Repos keyed by repo_url_key (another
URL of the same Repo is that Repo), comparing name, description,
directory, Tag, mode, branch, submodules, frequency and clone token (a
missing directory or clone token keeps the database's). An unchanged
import writes nothing (every import rewrote every Repo and its
Download).
An invalid entry is skipped and counted, and its Repo is kept; before,
a Repo whose entry failed validation was deleted. New Repos are created
inside the import's one write: recurring_download (via
get_or_create_download and _set_repository_download) takes commit=False,
and the URL is not looked up again for each new Repo (O(N^2)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml: a branch URL is keyed by its Repo's URL
file_rows keyed the raw URL, but stores parse_repo_url's URL, which
turns a forge's branch URL (GitHub /tree/<branch>, GitLab
/-/tree/<branch>) into the repo URL and a branch. The entry looked like
a new Repo: its insert hit the existing Repo's name (rolling back every
import of that file), and an invalid such entry did not protect its Repo
from deletion. Rows and invalid keys now use the parsed URL; a URL that
cannot be parsed is skipped by its raw key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* playlists.yaml imports by diffing
playlists.yaml declares two sections: playlists, keyed by name, and
their items, keyed by what each points to (a file's or Zim's
media-relative path and entry, or a URL) with position and title
compared. A playlist has at most one item per reference, so a moved
item changes only its position and keeps its row; an unchanged import
writes nothing (every import deleted and re-created every item).
prepare skips items whose file or Zim is not indexed before the write
(their place is kept), drops a Tag that no longer exists once tags.yaml
is imported (the config is then saved), and, without deletes, keeps the
database's order and adds the file's other items after the item that
precedes them.
Moving a FileGroup in a playlist counts as a playlists.yaml change (ORM,
the bulk move, and a refresh that changes the primary file).
media_relative moves to wrolpi.config_import for both configs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Playlist items: keep database order without deletes; empty lists delete
Without a database version, every playlist with an added or changed item
keeps the database's order (the file's new items go after their
predecessor; only titles come from the file), and a row whose merged
position and title equal the database's is not written. Before, only
playlists that gained an item were merged, so a file that only reordered
(or lacked an item the database had) gave two items one position.
A section can say it is present without rows (`ConfigSection.present`):
a listed playlist with an empty items list has its items removed; an
empty or missing playlists list still removes nothing. An item that
cannot be read, or repeats another, is counted in import_skipped, so the
config is not saved over it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* tags.yaml imports by diffing
tags.yaml declares three sections (Tags, tagged files, tagged Zim
entries), so its import reads the file once, diffs it against a snapshot,
and writes only the differences, with the file's version, in one write.
An unchanged import writes nothing.
A removed Tag is deleted with its tagged files and Zim entries; a
Collection using it loses only the Tag. A removed tagged file or Zim
entry is untagged. A tagged file whose Tag the config does not list is
skipped. prepare finds files without a FileGroup on disk, skips missing
ones before the write, moves Zim entries of a replaced Zim to the newest
one, and maps a file named by a secondary member to its group; the
config is then saved with the database's paths.
Moving a tagged FileGroup counts as a tags.yaml change (ORM, the bulk
move, and a refresh that changes the primary file), so a restart before
the save does not untag it.
Sections can define `same` (a tagged file without a date matches any
date), and prepare can count skipped rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A removed Tag leaves no Download that would apply it
The tags import removes a deleted Tag's name from every Download's
tag_names and from the tag_names and channel_tag_name its settings pass
on (a change to download_manager.yaml, which is then saved). Applying a
Tag that does not exist fails the download, and it is retried. Once
tags.yaml is imported, a newer download_manager.yaml drops names no Tag
has, and is saved without them.
A replaced Zim's entry tags move to the newest Zim of the same parsed
name, by the date in its name (as migrate_outdated_zim_tags picks it),
not by name prefix and path order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs import by diffing; download_manager.yaml first
Step 2 of the config import rework. A database config can now declare a
ConfigSection for each kind of row it saves, and ConfigFile.import_config runs
a fixed sequence for it:
- read and validate the file once;
- decide from the versions (wrolpi.config_version) what the import may do;
- snapshot each section's rows as plain data keyed the way the file names them,
and diff: added, changed, removed (none from an empty section, or without a
current version);
- prepare (files, network) with no transaction open;
- write only the differences, and the file's version, in one short write. A
failure writes nothing. If the database's version changed since the
snapshot, the import decides again (and saves a newer database).
Subclasses override the section methods (file_rows, db_rows, upsert, delete)
and the prepare/after_import hooks. The other database configs keep their
importers until their own steps.
download_manager.yaml is the first: an unchanged config imports with no write
(was 5,000 statements for 2,500 Downloads), new Downloads are one INSERT, and a
missing ZimSubscription is restored without rewriting the rest. Settings equal
to the global defaults compare as unset, as they are saved. A destination
written relative to the media directory is stored absolute. Completed
once-downloads are never removed (they are not saved; #679).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync Zim subscriptions directly during the downloads import
A row now compares its subscription's (name, language), derived from the
URL, instead of whether one exists. The import moves a subscription of
that name onto the written Download (a changed language or URL), removes
one the file no longer implies, and leaves the Download's own columns as
the file has them; change_download renamed the old Download's URL into
the one just inserted, and reset attempts and the downloaders. A Kiwix
URL that names no known subscription is a plain Download instead of
failing the whole write.
prepare runs only when there are differences, and its docstring states
that it must be safe to repeat and to abandon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Tests fail when a request writes outside request.ctx.db.write()
A request's transaction begins deferred. A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent. #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.
Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test. allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.
It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
row's delete when the middleware's commit failed under contention. It
now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
Session, after the request ended. set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
deferred get_db_session(). It now begins as a writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tasks don't inherit a declared write; tags import change backed out
From review of #864:
- background_task's task copied its creator's Context, including
_write_intent when started inside get_db_session(commit=True) or
RequestDB.write(), so the guard ignored the task's own writes for its
lifetime. The task now starts with no declared write. That exposes
fetch_channel_source_id (started by the channels config import): it read
the Channel, asked the remote site for its source_id, then committed in
the same deferred transaction. It now reads, fetches with no
transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
walk, and its exit commit saved changes the need_commit gate used to
drop. Reverted to master; splitting its reads from a short write is its
own change. The two tests that run it through a request are marked
allow_deferred_request_writes until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Remove a file's Tags Directory links when WROLPi deletes it (#861)
* Remove a file's Tags Directory links when WROLPi deletes it
When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not. The link became the only copy, so the tags sync
refused to delete it and warned on every sync. The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove Tags Directory links on path deletes; keep a valid poster's link
The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links. It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.
A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Renaming a directory with Collections waits for a concurrent writer
POST /api/files/rename on a directory changed Collections on the request's
deferred transaction. For a plain directory, that transaction was opened
before the FileWorker moved the files and wrote only after, so if anything
else wrote meanwhile the Collection update failed ("database is locked"):
the request returned 409 with the files already moved, and the nested
Collections still pointed at the old directory. Renaming a Collection's
own directory failed the same way, before moving anything.
rename_directory now changes Collections in a `write` (request.ctx.db.write
from the endpoint), which begins as a writer. Files are moved outside it;
nested Collections are read inside it, after the move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Remove a file's Tags Directory links when WROLPi deletes it
When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not. The link became the only copy, so the tags sync
refused to delete it and warned on every sync. The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove Tags Directory links on path deletes; keep a valid poster's link
The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links. It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.
A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test harness resets the shared file worker public queue for each test,
but the FileWorker's private LifoQueue is per-process state that survived
from one test to the next. Tests that process only a count task leave its
chained refresh task queued, and it later ran inside an unrelated test.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Distrust compromised eMudhra intermediates in certifi and Deno (#854)
* Distrust compromised eMudhra intermediates in certifi and Deno
The private keys of eight eMudhra intermediate CAs can be recovered from
their certificates, so anyone can mint a trusted certificate for any host.
* Require certifi>=2026.10.19; yt-dlp, requests and curl-cffi use its bundle,
and 2026.10.19 removes the emSign ECC Root CA - G3 root they chain to.
* Set DENO_TLS_CA_STORE=system for the API (systemd + Docker) and the archive
container so Deno follows the apt-patched system store instead of roots
that only change with a Deno upgrade.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep Deno's bundled roots in the archive container
browserless/chrome:1.61.1 is Ubuntu 22.04 whose newest ca-certificates is
from 2023, older than Deno 2.8.3's bundled roots. The system store only
becomes the better choice once the archive base image is upgraded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): bump dompurify from 3.4.16 to 3.4.17 in /app
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.16 to 3.4.17.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.16...3.4.17)
---
updated-dependencies:
- dependency-name: dompurify
dependency-version: 3.4.17
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
When refresh finds FileGroups sharing (directory, stem) it keeps the highest-id row and deletes the
rest. Deleting cascades, so the loser's tags and collection membership were silently lost, and the
next tags config save persisted the loss. Before the duplicate is deleted, its tags and collection
membership now move onto the FileGroup that survives it, reusing the merge the stem migration uses.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg exits 0 but prints "non monotonically increasing dts to muxer"
from the null output when two frames share a timestamp, collapsed into
"Last message repeated N times" when piped. Any stderr failed the
completeness check, so such videos (common on X) were rejected as
incomplete on every retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Distrust compromised eMudhra intermediates in certifi and Deno
The private keys of eight eMudhra intermediate CAs can be recovered from
their certificates, so anyone can mint a trusted certificate for any host.
* Require certifi>=2026.10.19; yt-dlp, requests and curl-cffi use its bundle,
and 2026.10.19 removes the emSign ECC Root CA - G3 root they chain to.
* Set DENO_TLS_CA_STORE=system for the API (systemd + Docker) and the archive
container so Deno follows the apt-patched system store instead of roots
that only change with a Deno upgrade.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep Deno's bundled roots in the archive container
browserless/chrome:1.61.1 is Ubuntu 22.04 whose newest ca-certificates is
from 2023, older than Deno 2.8.3's bundled roots. The system store only
becomes the better choice once the archive base image is upgraded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Adding, importing, changing, deleting and updating a Repo read, then
wrote, in the request's deferred transaction, so they failed with a 500
("database is locked") when another connection wrote in between. They
now write inside request.ctx.db.write().
Importing reads the clone with git and walks it for its size; those run
before the write transaction, so it is not held for them. Everything
that can refuse the import still runs before the clone is changed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
which() kept looking after shutil.which found a name in $PATH, so a later
fallback path (e.g. '/usr/bin/git' in which('git', '/usr/bin/git'))
always replaced it. Its docstring says $PATH comes first; stop at the
first match.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between. They now write inside
request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between. They now write inside request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/files/tag, /untag and /delete_groups read, then wrote, in the
request's deferred transaction. SQLite fails that lock upgrade instantly
("database is locked") when another connection wrote in between. Tag and
untag returned a 500; delete_groups returned 204 after FileGroup.delete had
unlinked the files, then its commit failed in the response middleware, so
the FileGroup row outlived its files. They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deleting, restarting, clearing and batch-editing Downloads, and creating,
renaming and deleting Tags, read then wrote in the request's deferred
transaction. SQLite fails that lock upgrade instantly ("database is
locked") when another connection wrote in between, so these failed with a
500 while a download or refresh was writing. They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #846, from review:
- The reorganize conflicts endpoint read the preview, ran ffprobe, then
flushed and committed in the request's deferred transaction. It now
reads, runs ffprobe through get_or_create_ffprobe_json (no session),
and stores the results in one short request.ctx.db.write().
- Its ffprobe step never ran: it matched model_type == 'channel', but a
conflicting video's model_type is its FileGroup's model, 'video'.
- execute_reorganization's no-move path stored the new file_format with
session.commit() on the caller's deferred session, from the reorganize
endpoint and from the batch job. The scan stays a read; the store is a
short write (request.ctx.db.write() in a request, its own connection
otherwise). Called without a session it now opens a read session, so
that write does not wait on its own caller's lock.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
These endpoints read, then wrote, in the request's deferred transaction.
SQLite fails a deferred transaction's lock upgrade instantly ("database
is locked") when another connection wrote in between, so a collection
change during a refresh or download could fail with a 500. They now
write inside request.ctx.db.write(), which begins as a writer and lets
busy_timeout absorb the contention; errors are mapped outside the block
so a 4xx rolls back.
Fixes#674
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
live-config's openssh-server component rewrites sshd_config to
"PasswordAuthentication no" on every live boot, so the live session only
accepted SSH keys. An sshd_config.d drop-in sets it back to yes; sshd uses
the first value it reads and sshd_config includes the drop-ins first, so the
drop-in wins. The login is user / live (live-config's default), now noted
in INSTALL.md. check-iso.sh fails the build if the drop-in is missing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The installer opened with Debian's "debian 13" banner. Debian's
derivative guidelines say derivatives must not be presented as Debian,
and its trademark policy forbids implying endorsement, so the installer
now shows a WROLPi banner in the boot splash's style (pie logo, WROLPi,
"Off-grid library", purple accent).
The banner replaces rootskel-gtk's logo_debian.png and logo_debian_dark.png
(logo_installer*.png link to them) via config/includes.installer, which
live-build adds to the installer initrd. It is 800x75 like Debian's; the
installer stretches its edge columns to the screen width, so the edges are
solid color. check-iso.sh fails the build if the graphical installer's
initrd does not carry it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The library-ownership migration compared the owner of
/mnt/persistence/wrolpi after creating it, so on a new persistence
partition the root-owned directory always looked like an old library and
the first-boot dialog said "Updated the library's ownership for this
WROLPi version". Only re-own a library that already existed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
INSTALL.md and the onboarding wizard send people to wrolpi.local, but the
Debian image had no mDNS responder, so the name never resolved on a Debian
install or Live USB. Raspberry Pi OS ships Avahi; the ISO now does too.
libnss-mdns lets the WROLPi's own desktop resolve .local names (apt
recommends are off in the build, so it is listed explicitly). check-iso.sh
fails the build if either package is missing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Readability extraction no longer takes minutes on image-heavy pages
SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.
Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output. Applied in
both the archive container and the native (Raspberry Pi/Debian) path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders cannot be forged by page text
The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times. Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders survive Readability's lazy-image cleanup
Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder. The ~50 character placeholder triggered this, so those images
lost their inline data. The token is now 160 hex characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The archive service is PID 1 in its container and never reaps orphans.
check_output kills each command's process group, so every archive leaves
chrome, cat, and node grandchildren behind as zombies. A long-running
container accumulated hundreds of them, counting against its PID limit.
init: true runs tini as PID 1, matching the web container.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISO: hostname defaults to wrolpi, boot menu times out
The installer's hostname question defaulted to "debian", so an install that
kept the default was not reachable at wrolpi.local. A preseed in the
installer initrd now defaults it to wrolpi; the question is still shown (and
a DHCP-provided hostname still wins).
The ISO's GRUB and ISOLINUX menus had no timeout, so a Live USB waited at the
boot menu forever after any reboot or power loss. The menus now boot the
default entry ("Run WROLPi (live)") after 10 seconds.
check-iso.sh fails the build if either regresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISO: say what the wrolpi hostname is for
The image does not include avahi-daemon, so the hostname alone does not make
wrolpi.local resolve. The default still names the TLS certificate, the
SMB/NetBIOS share, and the DHCP hostname a router may publish.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ISOs before wrolpi became a system account wrote the Live USB's library on
the persistence partition as UID 1000. Upgrading such a USB with a newer
ISO (wrolpi-usb.sh upgrade) keeps the persistence partition, so the library
would belong to the live session's "user" and the services could not write
to it. Bootstrap now re-owns the library once when its owner is not this
image's wrolpi. Live USBs only: installed systems' drives are left alone.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ISO's chroot hook created wrolpi as a login user (UID 1000, password
"wrolpi", full sudo, LightDM autologin). The Debian installer creates the
user it asks for only when no UID 1000-59999 user exists, so it silently
skipped that user: the only login on an installed system was wrolpi/wrolpi,
with SSH and sudo.
wrolpi is now a system account with no password or login sudo, as on every
other WROLPi deployment. The installer creates the user it asks for (with
sudo, no autologin). The live session logs in as live-config's "user".
Desktop shortcuts moved to /etc/skel so every user gets them.
check-iso.sh now fails the build when the image has a UID 1000-59999 user.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On a Debian install the onboarding wizard offered the EFI system partition
(unlabeled vfat at /boot/efi) as the primary WROLPi drive; choosing it would
mount the boot partition over /media/wrolpi. Candidates were filtered by
the "EFI" label only, and the Debian installer leaves the ESP unlabeled.
/boot/efi is now an excluded mountpoint, and partitions with the EFI system
partition type (GPT GUID or MBR 0xef, from lsblk PARTTYPE) are excluded even
when unmounted, e.g. on a second internal disk.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
"Run repair without external drive" kept the library on the internal disk,
but the API's unmounted-media guard refuses to create or use a database
beneath /media/wrolpi when it is not a mount. After onboarding without a
drive the API restarted forever with no database, and downloads were
blocked as "media not mounted".
The Controller now records the choice as drives.library_on_internal_disk in
controller.yaml -- the only source of it. The database guard and the
media_mounted check both read that marker, so require_media_mounted keeps
its meaning and needs no change. Onboarding a drive later clears the
marker, so it is never copied onto the drive.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
install.sh only accepted Debian 12, which cannot run this version, and
rejected Debian 13. It now requires Debian 13 or newer on both PCs and
Raspberry Pis. The install scripts point to the drive wizard instead of
manual mounting.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Installed Debian systems keep Caddy
The Debian installer purges every package listed in the ISO's
live/filesystem.packages-remove. live-build lists there every package
installed after the package-lists install pass, which includes anything a
chroot hook installs with apt -- and the hook installed Caddy that way.
An installed system therefore had no Caddy: wrolpi-bootstrap failed copying
the Caddyfile, the API/app/Kiwix were never enabled, and HTTPS never came up.
Caddy now comes from package-lists/, with its repository and signing key in
config/archives/ (installed into both the build chroot and the image).
check-iso.sh fails the build when the installer would remove any package,
and can be run by hand against any ISO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Debian ISOs install Caddy from Caddy's repository
Moving caddy into package-lists/ made apt read config/archives/ before
ca-certificates was installed, so the HTTPS fetch of Caddy's repository
failed certificate verification. apt ignores a failed repository, and the
image silently got Debian's much older caddy 2.6.2 instead of 2.11.
ca-certificates is now part of the bootstrap, and Debian's caddy is pinned
to -1, so an unreachable Caddy repository fails the build and installed
systems never switch to Debian's caddy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nothing started the hotspot since hotspot_on_startup moved out of the
API, and NetworkManager does not autoconnect the Hotspot connection.
The Controller now starts it once per boot unless hotspot.on_startup is
false; a legacy wrolpi.yaml hotspot_on_startup: false is migrated.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The onboarding wizard deleted /media/wrolpi/config before mounting a
drive, and the shadowed data check ignored config/. A library kept on
the internal disk keeps wrolpi.db and its configs there, so onboarding
a drive later deleted them without warning. Only a config/ holding
nothing but the ssl/ CA is now treated as transient.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Download and Upload are disabled until a full refresh completes, and a
fresh install never had one until the user found the refresh link.
Startup now queues it, and the dashboard says why Download and Upload
are disabled meanwhile.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* INSTALL.md points to the Controller drive wizard
Debian installs are Debian 13 from the amd64 ISO. The Raspberry Pi
steps no longer hand-edit fstab or run repair.sh; a shared First Boot
section covers the drive, certificate trust, and the first refresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* INSTALL.md: Debian installs use the drive wizard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wrolpi-bootstrap.sh created controller.yaml on installed systems, which
hid the onboarding wizard. Installed systems now wait for the wizard,
whose repair.sh finishes setup.
"Run repair without external drive" now writes controller.yaml before
starting repair, so the wizard does not return after repair.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Transcode, job, and Kiwix notifications send no URL, but every user notify
toast said "Click here to view the shared page" and opened /null on click.
Only a notification carrying a URL is followable now.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
