Commit graph

2642 commits

Author SHA1 Message Date
lrnselfreliance
5be899bf1f
Saving zims.yaml keeps the entries of Zims the database does not have yet (#935)
A rebuilt database imports zims.yaml before the refresh has modeled a single
Zim.  That import has no database version to compare with, so it saves the
config afterward, and the save wrote only the Zims in the database: none.  The
file's choices were erased and every Zim was then modeled with the default.

The dump now starts from the file's entries and overlays the database's, so a
choice survives until its Zim is modeled again, and a Zim in the database still
decides its own line.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:11:41 -07:00
lrnselfreliance
c0452e29e6
Hiding a Calibre book's cover and metadata uses the modeler's session (#936)
Doc._hide_calibre_files opened a second write session while the modeler held
the item's transaction.  SQLite has one writer, so the second session waited
out the 30 s busy timeout and failed with "database is locked"; the modeler
retried three times, so every Calibre epub cost about 98 s and stayed
unmodeled, and the cover.jpg / metadata.opf FileGroups it should have deleted
survived.  On a library of 1663 Calibre books the global refresh's modeling
phase would have taken about two days.

The nested session dates from before the SQLite cutover, when Postgres row
locks let it work.  The delete now runs in the caller's session.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:04:28 -07:00
lrnselfreliance
e6b7b9e5dd
Map downloads are checked against the hash in the signed manifest (#934)
* Map downloads are checked against the hash in the signed manifest

A meta4 signs only a bare hash, so any genuinely signed meta4 (an older version of a region, another region) passed
its own check.  The manifest now lists each file's sha256; it is signed as a whole, so the hash is tied to the URL.
The catalog copies each hash into its download's settings, and aria2c checks the file against it.  Downloads from a
manifest without hashes still require the file's signed meta4.

build_map_regions.py writes the hashes, and hash_map_manifest.py adds them to the published manifest from the
verified meta4s without rebuilding any maps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hash_map_manifest.py hashes each file's bytes instead of trusting its meta4's name

Only the bare hash in a meta4 is signed, so an older version's meta4 with its name rewritten (or terrain's, whose name
never changes) passed the name and duplicate-hash checks.  Each file is now hashed from a local copy of the published
files (--files) or downloaded from the CDN, and must match its signed meta4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 20:38:57 -07:00
lrnselfreliance
ee6c384dc6
A refresh keeps a FileGroup's tags and viewing state when its primary file changes (#932)
_upsert_files deleted the FileGroup of every companion file before
inserting, so a FileGroup whose primary file became a companion (foo.jpg,
then foo.mp4 arrives) was replaced by a new, untagged row.  A FileGroup
whose primary file was replaced (foo.txt deleted, foo.mp4 added) still owned
the (directory, stem), and the INSERT failed on the unique index.

Before inserting, the FileGroup owning each incoming (directory, stem) is
renamed to the new primary path, keeping its id, tags, collection
membership, model and viewing state.  When a different FileGroup owns the
primary path, the two collapse into the one dedupe prefers (modeled, then
tagged).  A companion's leftover FileGroup gives its tags, collection
membership and viewing state to its group's FileGroup before it is deleted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 20:21:34 -07:00
lrnselfreliance
851b12b060
A Zim's Search toggle is saved in zims.yaml so a rebuilt database or a newer Zim keeps it (#931)
The toggle was only the zim.auto_search column, which defaults to on.  The
database is rebuilt from files and configs, and a Zim's newer version is a new
row, so every rebuild and every subscription update turned search back on for
every Zim.  On a large library the global search then searched all of them.

zims.yaml records auto_search for each Zim, keyed by the Zim's name without
its date, so a newer version of a Zim is the same Zim.  A Zim the file does not
name is searched (the default stays on).  The config is imported on startup
with the usual version gating, saved when the toggle changes, and a newly
modeled Zim takes its setting from it.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:09:52 -07:00
lrnselfreliance
6fc147ccdf
A refresh re-indexes a FileGroup whose files changed even when their count did not (#933)
_upsert_files kept a FileGroup's files and indexed state when the size,
newest modification time and number of files were unchanged.  A file that
replaced another (foo.png for foo.jpg, same size, older than foo.mp4) left the
old file in `files` and the FileGroup indexed.

The upsert now compares the sorted file names.  FileGroups that still store
absolute paths (the old format) compare by name too, so they are not
re-indexed when nothing changed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:04:32 -07:00
lrnselfreliance
21006eae09
Map files are never downloaded without a signed hash (#930)
* Map files are never downloaded without a signed hash

A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later)
instead of downloading with TLS as its only check.  The search index downloader had no fallback branch at all, so it
downloaded with no hash whenever the .meta4 was missing.

A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when
the meta4's unsigned name differs from the URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* aria2c checks map files only against the signed hash, and a rejected file is never kept

Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4.  A sha-512 of a tampered
file added beside the signed sha-256 made aria2c accept the tampered file.  aria2c now gets a meta4 built from the
signed hash and the download URL alone.

A rejected download was left in the map directory under its final name, where the next attempt took it as already
downloaded.  Map files now download into map/.partial and move into place only once aria2c accepts them; a file that
fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Only the map file aria2c checked is moved into place

download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c
saves and checks the file under the name in the meta4 WROLPi writes.  When they differed, an unchecked file (such as
an interrupted download's leftover in map/.partial) could be moved into place.  The meta4 name now comes from the
URL, must be a plain file name, and any other reported path fails the download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
lrnselfreliance
bfe7b8afbd
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker

The global search estimates every auto-search Zim on each keystroke.  On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it.  Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.

Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them).  It reads the Zim paths and search string on stdin and
writes the counts on stdout.  A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Zim searches run in the libzim worker process too

The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.

The estimate script is now a general libzim worker with estimate, search and
entries jobs.  A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop.  Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker

Review of the earlier commits found two gaps.  Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
lrnselfreliance
9c258b41ee
A refresh groups each stem's files together even when another stem sorts between them (#929)
group_files_by_stem and chunks_by_stem assumed a stem's files are adjacent
once sorted.  They are not: foo.f137.mp4 (stem foo.f137) sorts between
foo.en.vtt and foo.mp4 (stem foo), and foo.tar.gz between foo.jpg and
foo.txt.  _upsert_files then built two FileGroups for one (directory, stem)
and the refresh failed with "UNIQUE constraint failed: file_group.directory,
file_group.stem".  A chunk boundary could split a stem the same way.

Both helpers now group by (directory, stem).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:38:48 -07:00
lrnselfreliance
8edace08bd
Python packages install from hash-locked requirements; yt-dlp still floats (#913)
* Python packages install from hash-locked requirements; yt-dlp still floats

Every pip install trusted PyPI over TLS: requirements.txt pins only top-level versions
(some with >=), the controller's are almost all >=, and dependencies were never pinned, so
a forged certificate (as the eMudhra key compromise allowed) could change code in the API,
the controller, or the archive container.  scripts/lock_requirements.sh generates a
universal, hash-locked .lock beside each requirements.txt, and
install_python_requirements.sh installs it with --require-hashes at every site.  yt-dlp is
left out of the lock and upgraded afterwards so devices still get YouTube fixes; its
dependencies are locked.  The unpinned `pip install --upgrade pip` steps are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The unhashed yt-dlp install cannot replace locked packages, and nothing builds from source

Review of #913 found two holes.  The yt-dlp step ran without --no-deps, so pip's
only-if-needed strategy would still fetch an unhashed dependency whenever yt-dlp's metadata
asked for a version the lock did not install.  And srt is published only as a source
archive, whose build installed setuptools with hash checking off on every install.

The yt-dlp step now uses --no-deps (its dependencies come only from the lock), both steps
are retried as the image builds did before, and installs and locks are wheels-only.  srt
3.5.3 is vendored unmodified as wrolpi/vendor/srt.py (MIT), verified against the sdist
hash the lock pinned; webvtt-py's SRT reader rejects WROLPi's own test file, so it is not
a replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:00:39 -07:00
Joel Shearon
fe85815c35
Format inventory expiration dates instead of epoch timestamps (#901)
Legacy inventory items stored date fields as Unix epoch seconds. The table showed the raw numbers, and sorting and expired highlighting did not treat them as dates.

- Frontend parses date values as YYYY-MM-DD or legacy epoch seconds; display, mobile view, export, search, sort and the expired check all use calendar dates with no timezone shift.
- The legacy migration converts epoch seconds to YYYY-MM-DD, and import repairs already-migrated files once.
- Jest runs with a pinned timezone; the expiry tests pass under any process timezone.

Closes #41

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 12:27:21 -06:00
lrnselfreliance
d69f8317aa
serve, single-file and readability-extractor install from a committed lockfile (#912)
They were installed with `npm i -g`, which pinned only the three top-level versions and
resolved their 156 dependencies fresh from the registry, so a forged TLS certificate (as
the eMudhra key compromise allowed) could change code WROLPi runs on every archive.
npm-tools/package-lock.json pins every package with its sha512, and install_npm_tools.sh
runs `npm ci` and links the commands into /usr/local/bin for every install path, upgrades,
both images and the archive container.  The app container's unpinned, unused global serve
is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:39:08 -06:00
lrnselfreliance
ecdad6c3e0
Caddy's apt repository is trusted through the key committed with WROLPi (#911)
The install scripts, the nginx-to-Caddy upgrade, and the Pi image downloaded Caddy's apt
signing key and source list over TLS and trusted them as-is, so a forged certificate could
serve a key that signs other caddy packages.  add_caddy_apt_repo.sh installs the key the
ISO already uses (fingerprint 65760C51EDEA2017CEA2CA15155B6D79CA56EA34) as an armored
keyring and names it in the source's signed-by.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:22:19 -06:00
lrnselfreliance
bde1209c63
An upgrade refuses to roll back to an older commit of the current branch (#910)
* An upgrade refuses to roll back to an older commit of the current branch

upgrade.sh verifies the fetched commit's signature, but a signature proves a commit is
WROLPi's, not that it is the newest; an intercepted fetch could serve an old signed
release and the forced checkout would roll the device back to it.  refuse_rollback stops
when the target is an older commit of the branch the device is on (a detached HEAD from a
release image counts as that branch).  Branch switches, the same commit, and history the
device does not contain (Debian 12 devices on debian12-final) are allowed, and -d allows
a deliberate downgrade; it reaches the upgrade service through the env file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An upgrade also refuses a signed commit with a lower version from other history

Ancestry alone only caught older commits in the device's own history; any other signed
commit, such as debian12-final or an old experimental branch, passed.  refuse_rollback
now also compares wrolpi/version.txt, which is inside the signed tree, and refuses a lower
or missing version.  A force-pushed branch at the same or a higher version still upgrades.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tests pin the two parts of refuse_rollback that only one rule catches

An earlier commit of the same version is refused only by the ancestry check, and
0.30.10-beta after 0.30.3-beta only orders correctly with sort -V; the fixture's
single-digit versions passed with either part removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 09:33:25 -06:00
lrnselfreliance
fb3b959fff
install.sh and both image builds run WROLPi only from a signed checkout (#909)
Only upgrade.sh checked the release key's signature.  install.sh and the Pi and ISO
builds cloned by branch or tag name and ran what GitHub returned, so a forged TLS
certificate (as the eMudhra key compromise allowed) could change what a fresh install
or a published image runs.  verify_wrolpi_commit.sh requires a VALIDSIG from the pinned
release-key fingerprint.  The image builds verify their clone with the verifier and key
from the build checkout; install.sh carries an identical copy of the function, since it
runs before any verified checkout exists.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 21:27:41 -06:00
lrnselfreliance
70e915f610
An upgrade installs Deno only when its sha256 matches (#908)
upgrade.sh downloaded Deno on every device whose Deno was older than the pinned
version and installed it without a check, so a forged TLS certificate (as the eMudhra
key compromise allowed) could replace the runtime yt-dlp uses.  The zip is now checked
against the same sha256 the install scripts pin; a mismatch skips the Deno install and
the upgrade continues.  A test requires every Deno download in the repo to be checked.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:48:12 -06:00
lrnselfreliance
dc3a8bb313
The Pi image builds from a pinned pi-gen commit and a checksummed Deno (#907)
The Pi build downloaded pi-gen's `arm64` branch head as a zip and ran it as root, and
downloaded Deno without checking it, so a forged TLS certificate (as the eMudhra key
compromise allowed) could change what went into the image.  pi-gen is now fetched by
commit hash, which git verifies against the objects, and the Deno zip is checked
against the same sha256 the other installers pin.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:36:00 -06:00
Roland
2caa2609ec
Bump version to 0.30.3-beta 2026-10-30 19:44:38 -06:00
lrnselfreliance
78e2741db8
A transcode or remux refreshes the sha256 sums a download recorded (#906)
The Job replaces the video file (and its ffprobe json), so the sums in the info json's
wrolpi section described files that were gone.  Once the refresh has re-modeled the
Video, its files are hashed again, without a session held, and the sums replaced.
A Video without recorded sums gains none.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:41:29 -06:00
lrnselfreliance
4c8e9fb68c
The modelers and indexers find unindexed FileGroups through an index (#905)
apply_modelers and apply_indexers count `file_group WHERE indexed != 1` for progress
and apply_indexers selects its batches with it.  SQLite cannot use an index for `!=`,
so each of these read every file_group row; on a large library with a cold disk cache
the first count took close to a minute.  A partial index holds only the unindexed rows.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:40:46 -06:00
lrnselfreliance
32405a9132
A download records the sha256 of its files in the wrolpi section (#904)
* A download records the sha256 of its files in the wrolpi section

A video download writes `wrolpi.sha256` into its .info.json; an archive download writes it into
its .readability.json.  Sums are keyed by suffix, so a rename or Reorganize keeps them valid,
and the metadata JSON holding them is not hashed.

Large files are hashed in execute_download (in a thread, without the DB session).  Finalize
reuses those sums while a file's mtime is unchanged and hashes only files finalize created or
changed (a converted poster, the ffprobe json).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A video download hashes only for an info json, and finalize never suspends

Finalize holds the write session; awaiting the hash let the loop run a switch whose
write waited on that session, stalling the loop for the busy timeout.  Finalize now
hashes inline: the large files were hashed in execute_download.

A download that writes no info json has nowhere to keep the sums, so it no longer
reads its files to hash them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:34:49 -06:00
lrnselfreliance
a40beac0f6
A delete returns once its FileGroups are removed, and models the library afterward (#903)
lib.delete() ran the modelers, indexers, and refresh cleanup inline.  Those cover the
whole library (any unmodeled or unindexed FileGroup anywhere), so a single-file delete
waited on unrelated work and could exceed the response timeout.  The deleted FileGroups
are still removed before the delete returns; the post-processing now runs in the
FileWorker through a queued refresh of the same paths.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:02:01 -06:00
lrnselfreliance
44052741d3
An author or subject beginning with a non-ASCII capital finds its collection (#902)
The lookup compared SQLite's lower(name), which folds only ASCII, with Python's
str.lower(), which folds Unicode.  "Édouard Hospitalier" never matched its own
collection, so the doc modeler inserted a duplicate, hit the unique constraint, and
re-extracted and failed on that doc on every modeler run.  Both sides now use
SQLite's lower().

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 18:37:50 -06:00
lrnselfreliance
56fee562b7
A directory lookup uses the directory index and matches only that directory (#900)
Uploads, deletes, and file refreshes looked up a missing path's descendants with
`directory LIKE 'path/%'`.  SQLite cannot use an index for that LIKE, so each lookup
scanned the whole file_group table; with a cold page cache on a large library this
held the first upload chunk or a delete for tens of seconds.  LIKE also treated `_`
and `%` in a path as wildcards, so refreshing a missing `a_b` removed the FileGroups of
an existing `aXb/sub`.

`directory_or_descendant` matches with a range the directory index can serve.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 19:57:00 -06:00
Roland
49eeef68e8
Bump version to 0.30.2-beta 2026-10-29 18:55:44 -06:00
lrnselfreliance
576f631d95
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work

The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop.  On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.

- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
  internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
  up when either answers.  Either may be empty.  The INTERNET_SERVER environment
  variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Internet check tries its IP address first

socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:57 -06:00
lrnselfreliance
a44cc8a456
A channels.yaml import keeps a Channel's once-Downloads (#898)
channels.yaml saves only a Channel's recurring Downloads (Channel.downloads), but
its import compared every Download of a Channel's Collection.  A once-Download of
a Channel's video was "removed from the file", so an import with deletes deleted
it, even one that had not run yet.  The import now compares, and deletes, only
recurring Downloads.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:59:56 -06:00
lrnselfreliance
ce7095c0a1
A config import records the version of the file it read (#897)
* A config import records the version of the file it read

Every Sanic process initializes the configs it shares with the others while one of them imports the
database configs.  `ConfigFile.initialize` reset the shared dict to the defaults (version 0) before
loading the file, and the import read the file's version from that shared dict.  When the two
overlapped, the import recorded version 0 in `config_version`; every later save of that config was
refused as older than its file, and the next startup's import undid the unsaved changes.

- `ConfigFile.initialize` sets the shared dict in one call, and `MultiFileConfig.initialize` no longer
  empties it first, so other processes never see the defaults in place of the file.
- An import reads the version and rows from the file it read (`import_file_config`), never the shared
  dict.

Tests run the race for every config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Playlists import steps read the file's config; tests restore switch handlers

- `PlaylistsConfig._keep_unreadable_items` and `_merge_items` read `import_file_config`, not the
  shared `_config`.
- `test_import_steps_read_the_file`: no versioned config's `prepare`/`after_import` (or the methods
  they call) reads the shared config.
- The new tests create configs, which replaced the global configs' switch handlers for later tests
  (repos tests then dumped a test's config).  `restore_switch_handlers` puts them back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Only the first process loads a shared config; the others attach

A restarted perpetual process initializes the configs while the other processes serve them.
Re-loading the files then reverted or dropped newer changes in the shared dict (an inventory
created or saved since, a config updated but not saved yet).

- `ConfigFile.initialize` loads the file only into an empty shared dict, under
  `config_update_lock`; otherwise it keeps what the dict holds.
- `MultiFileConfig.initialize` imports only into an empty shared store, under `save_lock()`, and
  points `_configs` back at the shared store if the import fails.
- `MultiFileConfig.create` checks for the slug and inserts it under `save_lock()`.

`test_initialize_keeps_the_shared_config` now changes the shared config between two processes'
initializes (an inventory renamed and another created), and requires the second to keep it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 21:03:52 -06:00
lrnselfreliance
98ceb67321
A malformed entry stops a backup preview or merge with an error naming it (#896)
* A malformed entry stops a backup preview or merge with an error naming it

Backup previews and merges read every config entry as well-formed: a malformed one (an entry that is not a
mapping, a missing or non-text identity, a short tag_files line) raised an AttributeError or TypeError, so the
preview was a 500 and a merge failed with a generic message.  `ConfigFile.backup_entries` now checks each entry
a preview or merge reads, in the backup and in the current config, and raises InvalidConfig naming the file and
the entry; the merge endpoint passes that message on.  A malformed entry may be corrupt, so it is not skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A falsy "tags" that is not a mapping is an error in a backup preview or merge

`tags: []`, `''`, `0` or `false` became no tags through `or dict()` before the mapping check.  Only a missing
or null `tags` means no tags now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:59:49 -06:00
lrnselfreliance
3c815245b4
An archive detects a browser when the configured one is not installed (#895)
A configured browser_executable was used without checking it exists.  A config on the media drive can outlive the
OS it was written on: Raspberry Pi OS's /usr/bin/chromium-browser is /usr/bin/chromium on Debian.  The wrapper
discards the browser's stderr, so its failed exec only showed as single-file's "Connection refused ... Child
process has already terminated", and every archive failed.  Now a missing configured browser is logged and one is
detected, as when none is configured.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:59:00 -06:00
lrnselfreliance
9352002d07
Map downloads compare a Download's region with SQLite's JSON operator (#894)
`Download.settings['region'].astext` is PostgreSQL's JSONB operator.  On SQLite it raises AttributeError, so the
map catalog failed before queuing any region Download, and an extract never checked for a CDN search index.  Use
`.as_string()`.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:58:19 -06:00
lrnselfreliance
f4cd3b6f77
Fail CI when vulture finds new dead code (#889)
* Fail CI when vulture finds new dead code

scripts/check_dead_code.py runs vulture and compares its findings with
scripts/vulture_baseline.txt.  The baseline is keyed by file and name, not
line number, so unrelated edits do not invalidate it.  Class fields,
attributes, test arguments (side-effect fixtures), and alembic revision
names are not reported, because nearly all of them are false positives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* TEMP: add a dead function to prove the Dead Code check fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Revert the temporary dead function

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:10:52 -06:00
lrnselfreliance
4365e23327
Remove code that only tests called (#888)
Deletes functions nothing in the app calls, along with the tests that only
exercised them.  Tests that covered live logic through a dead wrapper now call
the live functions: the Download-linking tests use `domain_download_links` /
`channel_download_links` with `_link_downloads`, the archive tests create
Archives with `write_archive_files` + `register_archive`, and the collection
lifecycle test removes an item with `Collection.remove_item`.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:00:27 -06:00
lrnselfreliance
edca3f610f
Channel downloads follow yt-dlp's playlist redirect and refuse non-channel URLs (#886)
A video URL that names a playlist (watch?v=...&list=...) makes yt-dlp return a
redirect to the playlist instead of the playlist itself.  The ChannelDownloader
now follows up to 10 redirects.  Info without "entries" is not a channel or
playlist, so it is rejected before a Channel is created from it.  Previously
such a Channel was named after the URL ("watch"), and the download then failed
with KeyError: 'entries'.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:00:00 -06:00
lrnselfreliance
8befb23e38
Skip a playlist whose directory is outside the media directory (#884)
playlists.yaml's directory was only made media-relative, so a directory with ".." parts (e.g. "../outside")
was stored as a path outside the media directory, where the playlist's files are synced.  It is now validated
with validate_collection_directory, as domains and channels are; such an entry is skipped and counted, and the
playlist keeps what the database has.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:58:07 -06:00
lrnselfreliance
b14a32c7bc
A recurring Download with more than one URL is an error (#887)
POST /api/download created a recurring Download from the first URL and silently dropped the rest, while answering
201.  No client sends more than one URL with a frequency (each recurring form has one URL field), so reject it, as
PUT /api/download/<id> already does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:57:47 -06:00
lrnselfreliance
22ada7eb57
Remove unused code found by vulture (#885)
Deletes functions, methods, classes, schema dataclasses, error classes, Events
senders, and module-level constants that nothing references, including tests.
Also drops the unused `doc_cleanup(test_directory)` and
`_wait_for_sub_job_with_progress(collection_idx)` parameters, and the imports
that only served removed code.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:50:09 -06:00
lrnselfreliance
a248104b2f
Starting a Controller script writes its env file every time and rejects unknown parameters (#883)
The upgrade service reads BRANCH from /tmp/wrolpi-upgrade.env, which upgrade.sh removes when it finishes.  The
Controller wrote that file only when a request had parameters, so starting an upgrade without a branch (the simple UI
sends no body when the field is empty) left the service unable to start, while the API answered success.  A request
like {"branch": "master"} was accepted with the key ignored, the same way.

Now the env file is always written, and a missing branch is the current one, as upgrade.sh chooses; a file left by an
interrupted run no longer chooses it.  Unknown parameters and unknown request keys are errors.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:35:23 -06:00
lrnselfreliance
bc58a5cc35
Save download_manager.yaml after new Downloads are committed (#882)
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
create_downloads activated save_downloads_config right after its flush, before the caller committed.  The
switch runs in the perpetual process, which cannot see uncommitted Downloads, so a save that ran first wrote
the file without them and nothing saved them later.  The switch is now activated when the session commits, and
not at all when it rolls back.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 20:32:18 -06:00
lrnselfreliance
eb82489426
Keep playlists.yaml in shared memory so the perpetual process can save it (#881)
playlists_config was the only config not attached to a shared_ctx dict.  Switches save configs in the perpetual
process, which never imports the files, so its playlists_config kept the default version 0.  While the database had
no version row for playlists.yaml, the dump wrote version 1 and `save` refused to overwrite the newer file; the
refused save never recorded the row, so every later save was refused too, and new playlists never reached the
config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:57:29 -06:00
lrnselfreliance
ae97727b98
Save channels.yaml with the database's version (#880)
save_channels_config wrote the Channels with `update()`, which keeps the version the config already had in
memory.  After a Channel changed, the database's version was ahead of the saved file's, so the next import
treated the file as older and saved the database over it: an edit made to channels.yaml in the meantime was
lost.  It now saves with `dump_config`, which writes the database's version, as every other database config
does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:39:45 -06:00
lrnselfreliance
d16bac29e8
Renaming a Tag commits its Download renames with the request (#879)
* Renaming a Tag commits its Download renames with the request

update_tag renamed Downloads in a background task using the request's
session, after RequestDB.write() had committed.  Its flush was never
committed, so the rename was lost, and the open write transaction held
SQLite's write lock: every later write failed with "database is locked"
until the API restarted.  Only the Channel moves stay in the background.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Save the Downloads config after a Tag rename commits; scan Downloads only on a rename

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:34:34 -06:00
lrnselfreliance
fffb2ed981
Link a Download to its domain by its destination column (#878)
`domain_download_links` read the destination only from a Download's settings, but the destination is now a column
(the config import moves an old settings destination into it), so no Download was linked to a domain by its
directory.  Read the column, and the settings for an old Download, as channels already do.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:21:55 -06:00
lrnselfreliance
c577cae9d8
Renaming a Tag renames it in Download settings (#877)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
update_tag never renamed settings.tag_names, and its channel_tag_name
edit mutated the JSON in place, so it was never saved.  The old name
stayed, and a Download that later found a new Channel failed (UnknownTag).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:34:47 -06:00
lrnselfreliance
79526afebe
A malformed config entry is skipped and counted instead of failing the import (#875)
* A malformed config entry is skipped and counted instead of failing the import

An entry that is not a mapping (or whose key is not text, or a list that is not a list) made the whole file fail to
import in playlists.yaml, repos.yaml, download_manager.yaml and tags.yaml (tag_files/tag_zims), and a non-text name
did in domains.yaml.  Each is now skipped and counted in import_skipped, so the config is not saved over it, and the
rest of the file imports.  A Channel whose `downloads` is not a list keeps its Downloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An unreadable entry keeps no row and does not make a section present

The key of an entry too malformed to name a row is now `Unreadable(position)`.  It is still counted in
import_skipped, but `present` ignores it, so a section whose entries are all unreadable removes nothing (as an empty
section), instead of deleting every row.  A playlist whose `items` is not a list keeps its items, and a Channel
Download whose url is not text is counted (its Channel keeps its Downloads) instead of failing the import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:28:34 -06:00
lrnselfreliance
0dd61b8e64
Strip whitespace from Tag names (#876)
* Tag names have leading and trailing whitespace stripped

A padded name now finds the existing Tag instead of creating a
near-duplicate, and a whitespace-only name is rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Migration strips whitespace from Tag names and merges collisions

Tags whose names match once stripped are merged (tagged files, Zim
entries, Collections); the first color by id is kept.  Downloads and the
configs that name Tags are rewritten too, so the next import is in step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tags.yaml keeps the database's color for merged Tags

The file was merged in sorted key order, not by Tag id, and the next
import copies the file's color over the database's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:22:16 -06:00
lrnselfreliance
c3e25b84d6
Configs without database rows read and validate only; imports run off the event loop (#874)
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request).  The
per-config overrides that only set successful_import are removed.

wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.

The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:17:31 -06:00
lrnselfreliance
ee10b34b15
recently_viewed.yaml imports by diffing (#873)
* recently_viewed.yaml imports by diffing

recently_viewed.yaml declares one section, its files keyed by
media-relative path, comparing viewed, progress and position.  The
database's rows are the window the config saves (the newest
RECENTLY_VIEWED_LIMIT views, inside the media directory), so an
unchanged import writes nothing.

A file removed from the config has its viewing history cleared
(decision 6), but only when its view is at least as recent as the
config's oldest entry: an older view fell off the end of the window.  A
malformed entry keeps its file's history and is not counted as skipped
(that would stop every recently viewed save).  Without deletes, progress
written after the config is kept, and a newer plain view keeps its
`viewed` but takes the config's position (as before).  prepare finds the
files of new entries on disk before the write; a deleted file's history
goes with it.

Moving a viewed FileGroup counts as a recently_viewed.yaml change (ORM,
the bulk move, and a refresh that changes the primary file).  The media
directory checks are batched (each resolved the media directory again).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* recently_viewed.yaml: the no-delete merge covers files outside the window

Without a database version, newer history in the database wins (its
progress is kept; a newer plain view keeps its `viewed` and takes the
config's position).  That applied only to files among the newest views;
a file listed in the config that had fallen out of them was "added",
and its stale entry replaced the newer history.  prepare now reads the
database's history of added files and applies the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 14:39:45 -06:00
lrnselfreliance
315b60e9eb
channels.yaml imports by diffing (#872)
* channels.yaml imports by diffing

channels.yaml declares two sections: Channels keyed by directory
(comparing name, URL, Tag, file format, and the source id and flags,
which keep the database's value when missing), and the Downloads listed
under each Channel, keyed by (directory, URL); an existing Download keeps
its own frequency.  An unchanged import writes nothing (it ran about a
thousand queries and committed several times).

Removed Channels are deleted with their Collection (and Downloads);
a Download removed from a Channel is deleted.  A channel Collection
without a Channel is deleted unless listed (then it gets its Channel).
A Channel whose directory changed, with the same name, is moved.  An
invalid entry is skipped and counted, and keeps its Channel (by name when
its directory is invalid) and that Channel's Downloads; so does an entry
without a list of Downloads.  Duplicate directories still refuse the
import.  Unknown Tags are dropped once tags.yaml is imported.

New Downloads are created in the import's one write (commit=False on
Channel.get_or_create_download, Download.delete and delete_with_videos).
After the import, Videos are claimed and source ids fetched only for new
and changed Channels; Downloads are linked without a query per Download,
and not while entries are skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* channels.yaml: moves, invalid Downloads, and saves after the write

- A moved Channel's Download diff is retargeted to its new directory:
  one its new entry lists is the same Download, one it does not is
  deleted in the same import (the old directory was treated as kept).
- An unreadable Download is counted as skipped, and its Channel keeps
  its Downloads (it was dropped, and the Channel's Download deleted).
- A new Download without a frequency is dropped before the write and
  counted (it re-opened a write on every import).
- No source id is fetched while entries are skipped (its save would be
  refused, leaving the database newer than the file); a fetched one is
  saved with dump_config, which records the database's version.
- Download.delete(commit=False) neither changes the skip list nor starts
  config saves, which would read the database before the import commits
  (and write a removed Download back into channels.yaml).  The import
  saves download_manager.yaml after its write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 13:13:42 -06:00
lrnselfreliance
57c2a29022
domains.yaml imports by diffing (#871)
* domains.yaml imports by diffing

domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format.  An unchanged
import writes nothing.

A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it.  An invalid entry (name
or directory) is skipped, counted, and its domain kept.  Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).

Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test_bookmarks_config_import starts the app like its siblings

It imports a config without `async_client`, so it fails (no
shared_ctx.configs_imported) whenever it is the first test on its
worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* domains.yaml: converge without a save; no links while entries are skipped

A save after the import is refused while an entry is skipped, so the
import must converge without one:
- a removed domain with Archives that is already reset is left as it is
  (prepare reads `import_db_rows`); before, every import reset it again;
- after prepare, the framework drops changed rows that now match the
  database (a dropped unknown Tag), so they are not written again.

Linking Downloads changes domains.yaml after its version is recorded;
while entries are skipped the save is refused, the database stays newer,
and the next startup would save over the skipped entries.  Linking now
waits for an import without skipped entries.

An entry that is not a mapping is skipped and counted, instead of failing
the whole file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 11:41:59 -06:00