The Job replaces the video file (and its ffprobe json), so the sums in the info json's
wrolpi section described files that were gone. Once the refresh has re-modeled the
Video, its files are hashed again, without a session held, and the sums replaced.
A Video without recorded sums gains none.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A download records the sha256 of its files in the wrolpi section
A video download writes `wrolpi.sha256` into its .info.json; an archive download writes it into
its .readability.json. Sums are keyed by suffix, so a rename or Reorganize keeps them valid,
and the metadata JSON holding them is not hashed.
Large files are hashed in execute_download (in a thread, without the DB session). Finalize
reuses those sums while a file's mtime is unchanged and hashes only files finalize created or
changed (a converted poster, the ffprobe json).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A video download hashes only for an info json, and finalize never suspends
Finalize holds the write session; awaiting the hash let the loop run a switch whose
write waited on that session, stalling the loop for the busy timeout. Finalize now
hashes inline: the large files were hashed in execute_download.
A download that writes no info json has nowhere to keep the sums, so it no longer
reads its files to hash them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The lookup compared SQLite's lower(name), which folds only ASCII, with Python's
str.lower(), which folds Unicode. "Édouard Hospitalier" never matched its own
collection, so the doc modeler inserted a duplicate, hit the unique constraint, and
re-extracted and failed on that doc on every modeler run. Both sides now use
SQLite's lower().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
channels.yaml saves only a Channel's recurring Downloads (Channel.downloads), but
its import compared every Download of a Channel's Collection. A once-Download of
a Channel's video was "removed from the file", so an import with deletes deleted
it, even one that had not run yet. The import now compares, and deletes, only
recurring Downloads.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A config import records the version of the file it read
Every Sanic process initializes the configs it shares with the others while one of them imports the
database configs. `ConfigFile.initialize` reset the shared dict to the defaults (version 0) before
loading the file, and the import read the file's version from that shared dict. When the two
overlapped, the import recorded version 0 in `config_version`; every later save of that config was
refused as older than its file, and the next startup's import undid the unsaved changes.
- `ConfigFile.initialize` sets the shared dict in one call, and `MultiFileConfig.initialize` no longer
empties it first, so other processes never see the defaults in place of the file.
- An import reads the version and rows from the file it read (`import_file_config`), never the shared
dict.
Tests run the race for every config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Playlists import steps read the file's config; tests restore switch handlers
- `PlaylistsConfig._keep_unreadable_items` and `_merge_items` read `import_file_config`, not the
shared `_config`.
- `test_import_steps_read_the_file`: no versioned config's `prepare`/`after_import` (or the methods
they call) reads the shared config.
- The new tests create configs, which replaced the global configs' switch handlers for later tests
(repos tests then dumped a test's config). `restore_switch_handlers` puts them back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Only the first process loads a shared config; the others attach
A restarted perpetual process initializes the configs while the other processes serve them.
Re-loading the files then reverted or dropped newer changes in the shared dict (an inventory
created or saved since, a config updated but not saved yet).
- `ConfigFile.initialize` loads the file only into an empty shared dict, under
`config_update_lock`; otherwise it keeps what the dict holds.
- `MultiFileConfig.initialize` imports only into an empty shared store, under `save_lock()`, and
points `_configs` back at the shared store if the import fails.
- `MultiFileConfig.create` checks for the slug and inserts it under `save_lock()`.
`test_initialize_keeps_the_shared_config` now changes the shared config between two processes'
initializes (an inventory renamed and another created), and requires the second to keep it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A malformed entry stops a backup preview or merge with an error naming it
Backup previews and merges read every config entry as well-formed: a malformed one (an entry that is not a
mapping, a missing or non-text identity, a short tag_files line) raised an AttributeError or TypeError, so the
preview was a 500 and a merge failed with a generic message. `ConfigFile.backup_entries` now checks each entry
a preview or merge reads, in the backup and in the current config, and raises InvalidConfig naming the file and
the entry; the merge endpoint passes that message on. A malformed entry may be corrupt, so it is not skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A falsy "tags" that is not a mapping is an error in a backup preview or merge
`tags: []`, `''`, `0` or `false` became no tags through `or dict()` before the mapping check. Only a missing
or null `tags` means no tags now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A configured browser_executable was used without checking it exists. A config on the media drive can outlive the
OS it was written on: Raspberry Pi OS's /usr/bin/chromium-browser is /usr/bin/chromium on Debian. The wrapper
discards the browser's stderr, so its failed exec only showed as single-file's "Connection refused ... Child
process has already terminated", and every archive failed. Now a missing configured browser is logged and one is
detected, as when none is configured.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`Download.settings['region'].astext` is PostgreSQL's JSONB operator. On SQLite it raises AttributeError, so the
map catalog failed before queuing any region Download, and an extract never checked for a CDN search index. Use
`.as_string()`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deletes functions nothing in the app calls, along with the tests that only
exercised them. Tests that covered live logic through a dead wrapper now call
the live functions: the Download-linking tests use `domain_download_links` /
`channel_download_links` with `_link_downloads`, the archive tests create
Archives with `write_archive_files` + `register_archive`, and the collection
lifecycle test removes an item with `Collection.remove_item`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A video URL that names a playlist (watch?v=...&list=...) makes yt-dlp return a
redirect to the playlist instead of the playlist itself. The ChannelDownloader
now follows up to 10 redirects. Info without "entries" is not a channel or
playlist, so it is rejected before a Channel is created from it. Previously
such a Channel was named after the URL ("watch"), and the download then failed
with KeyError: 'entries'.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deletes functions, methods, classes, schema dataclasses, error classes, Events
senders, and module-level constants that nothing references, including tests.
Also drops the unused `doc_cleanup(test_directory)` and
`_wait_for_sub_job_with_progress(collection_idx)` parameters, and the imports
that only served removed code.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
save_channels_config wrote the Channels with `update()`, which keeps the version the config already had in
memory. After a Channel changed, the database's version was ahead of the saved file's, so the next import
treated the file as older and saved the database over it: an edit made to channels.yaml in the meantime was
lost. It now saves with `dump_config`, which writes the database's version, as every other database config
does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`domain_download_links` read the destination only from a Download's settings, but the destination is now a column
(the config import moves an old settings destination into it), so no Download was linked to a domain by its
directory. Read the column, and the settings for an old Download, as channels already do.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A malformed config entry is skipped and counted instead of failing the import
An entry that is not a mapping (or whose key is not text, or a list that is not a list) made the whole file fail to
import in playlists.yaml, repos.yaml, download_manager.yaml and tags.yaml (tag_files/tag_zims), and a non-text name
did in domains.yaml. Each is now skipped and counted in import_skipped, so the config is not saved over it, and the
rest of the file imports. A Channel whose `downloads` is not a list keeps its Downloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* An unreadable entry keeps no row and does not make a section present
The key of an entry too malformed to name a row is now `Unreadable(position)`. It is still counted in
import_skipped, but `present` ignores it, so a section whose entries are all unreadable removes nothing (as an empty
section), instead of deleting every row. A playlist whose `items` is not a list keeps its items, and a Channel
Download whose url is not text is counted (its Channel keeps its Downloads) instead of failing the import.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request). The
per-config overrides that only set successful_import are removed.
wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.
The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* channels.yaml imports by diffing
channels.yaml declares two sections: Channels keyed by directory
(comparing name, URL, Tag, file format, and the source id and flags,
which keep the database's value when missing), and the Downloads listed
under each Channel, keyed by (directory, URL); an existing Download keeps
its own frequency. An unchanged import writes nothing (it ran about a
thousand queries and committed several times).
Removed Channels are deleted with their Collection (and Downloads);
a Download removed from a Channel is deleted. A channel Collection
without a Channel is deleted unless listed (then it gets its Channel).
A Channel whose directory changed, with the same name, is moved. An
invalid entry is skipped and counted, and keeps its Channel (by name when
its directory is invalid) and that Channel's Downloads; so does an entry
without a list of Downloads. Duplicate directories still refuse the
import. Unknown Tags are dropped once tags.yaml is imported.
New Downloads are created in the import's one write (commit=False on
Channel.get_or_create_download, Download.delete and delete_with_videos).
After the import, Videos are claimed and source ids fetched only for new
and changed Channels; Downloads are linked without a query per Download,
and not while entries are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* channels.yaml: moves, invalid Downloads, and saves after the write
- A moved Channel's Download diff is retargeted to its new directory:
one its new entry lists is the same Download, one it does not is
deleted in the same import (the old directory was treated as kept).
- An unreadable Download is counted as skipped, and its Channel keeps
its Downloads (it was dropped, and the Channel's Download deleted).
- A new Download without a frequency is dropped before the write and
counted (it re-opened a write on every import).
- No source id is fetched while entries are skipped (its save would be
refused, leaving the database newer than the file); a fetched one is
saved with dump_config, which records the database's version.
- Download.delete(commit=False) neither changes the skip list nor starts
config saves, which would read the database before the import commits
(and write a removed Download back into channels.yaml). The import
saves download_manager.yaml after its write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* domains.yaml imports by diffing
domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format. An unchanged
import writes nothing.
A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it. An invalid entry (name
or directory) is skipped, counted, and its domain kept. Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).
Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test_bookmarks_config_import starts the app like its siblings
It imports a config without `async_client`, so it fails (no
shared_ctx.configs_imported) whenever it is the first test on its
worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* domains.yaml: converge without a save; no links while entries are skipped
A save after the import is refused while an entry is skipped, so the
import must converge without one:
- a removed domain with Archives that is already reset is left as it is
(prepare reads `import_db_rows`); before, every import reset it again;
- after prepare, the framework drops changed rows that now match the
database (a dropped unknown Tag), so they are not written again.
Linking Downloads changes domains.yaml after its version is recorded;
while entries are skipped the save is refused, the database stays newer,
and the next startup would save over the skipped entries. Linking now
waits for an import without skipped entries.
An entry that is not a mapping is skipped and counted, instead of failing
the whole file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml imports by diffing
repos.yaml declares one section, its Repos keyed by repo_url_key (another
URL of the same Repo is that Repo), comparing name, description,
directory, Tag, mode, branch, submodules, frequency and clone token (a
missing directory or clone token keeps the database's). An unchanged
import writes nothing (every import rewrote every Repo and its
Download).
An invalid entry is skipped and counted, and its Repo is kept; before,
a Repo whose entry failed validation was deleted. New Repos are created
inside the import's one write: recurring_download (via
get_or_create_download and _set_repository_download) takes commit=False,
and the URL is not looked up again for each new Repo (O(N^2)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml: a branch URL is keyed by its Repo's URL
file_rows keyed the raw URL, but stores parse_repo_url's URL, which
turns a forge's branch URL (GitHub /tree/<branch>, GitLab
/-/tree/<branch>) into the repo URL and a branch. The entry looked like
a new Repo: its insert hit the existing Repo's name (rolling back every
import of that file), and an invalid such entry did not protect its Repo
from deletion. Rows and invalid keys now use the parsed URL; a URL that
cannot be parsed is skipped by its raw key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Tests fail when a request writes outside request.ctx.db.write()
A request's transaction begins deferred. A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent. #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.
Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test. allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.
It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
row's delete when the middleware's commit failed under contention. It
now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
Session, after the request ended. set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
deferred get_db_session(). It now begins as a writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tasks don't inherit a declared write; tags import change backed out
From review of #864:
- background_task's task copied its creator's Context, including
_write_intent when started inside get_db_session(commit=True) or
RequestDB.write(), so the guard ignored the task's own writes for its
lifetime. The task now starts with no declared write. That exposes
fetch_channel_source_id (started by the channels config import): it read
the Channel, asked the remote site for its source_id, then committed in
the same deferred transaction. It now reads, fetches with no
transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
walk, and its exit commit saved changes the need_commit gate used to
drop. Reverted to master; splitting its reads from a short write is its
own change. The two tests that run it through a request are marked
allow_deferred_request_writes until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Remove a file's Tags Directory links when WROLPi deletes it
When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not. The link became the only copy, so the tags sync
refused to delete it and warned on every sync. The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove Tags Directory links on path deletes; keep a valid poster's link
The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links. It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.
A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg exits 0 but prints "non monotonically increasing dts to muxer"
from the null output when two frames share a timestamp, collapsed into
"Last message repeated N times" when piped. Any stderr failed the
completeness check, so such videos (common on X) were rejected as
incomplete on every retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding, importing, changing, deleting and updating a Repo read, then
wrote, in the request's deferred transaction, so they failed with a 500
("database is locked") when another connection wrote in between. They
now write inside request.ctx.db.write().
Importing reads the clone with git and walks it for its size; those run
before the write transaction, so it is not held for them. Everything
that can refuse the import still runs before the clone is changed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between. They now write inside
request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between. They now write inside request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Readability extraction no longer takes minutes on image-heavy pages
SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.
Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output. Applied in
both the archive container and the native (Raspberry Pi/Debian) path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders cannot be forged by page text
The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times. Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders survive Readability's lazy-image cleanup
Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder. The ~50 character placeholder triggered this, so those images
lost their inline data. The token is now 160 hex characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A repr runs inside log calls. Reading a relationship or an expired column there runs a query, and
on an object whose session is gone it raises DetachedInstanceError instead of logging. Archive's
repr also stat'ed its SingleFile file on disk.
`wrolpi.common.loaded()` follows attributes through state that is already loaded (primary keys
from the identity key) and never runs SQL. Every model repr uses it; with everything loaded the
output is unchanged, except Video and Archive now show the FileGroup's primary path.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest. A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted. The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.
Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before. The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A merge that never finished leaves `<stem>.f313.webm` (video-only) next to
where the merged file belongs. A later download of the same video globs
every `<stem>.*` file into its FileGroup, but only `.part` files and format
files with the *same* extension as the merged video were deleted. A `.webm`
leftover beside a new `.mp4` survived, joined the new FileGroup, and could
become the Video's file.
Delete every `<stem>.fNNN.*` file in the group, whatever its extension, which
also covers WROLPi's poster/ffprobe sidecars of an indexed leftover and
yt-dlp's `.ytdl` state files.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.
Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* File downloads: return plain values from upsert_file
Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`. `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone. SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends. Failed downloads retry, re-downloading their files.
`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Archive uploads: return plain values from singlefile_to_archive
Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved. It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query. An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async. Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.
Collection, CollectionItem and Channel dict() now run inside the serializer guard. Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
the session.
Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive. An
Archive with 51 history entries ran 107 statements per request. Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2. This is a mechanical port; the code
stays synchronous.
- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
statement committed on its own. 2.0 autobegins, so the unique-stem
migration's read snapshot could not become a writer after its helpers wrote
on another connection ("database is locked"). The begin listener emits no
BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CircleCI is gone, so the skip and its flag (CIRCLECI → IS_CI) are named for
CI in general, and the flag reads only GITHUB_ACTIONS.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test_ownership.py's git() helper did not use TEST_GIT_ENV, so its commits
read the machine's git identity. CI runners have none and cannot guess one,
so seven tests failed there on every run.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Opening a doc on its page never set FileGroup.viewed, so docs were missing
from Recently Viewed. GET /api/docs/:id now marks the doc viewed, with the
same skip_viewed parameter as archives and videos.
The file preview marked a file viewed more than once: opening from a
?preview= URL sent two tracked requests, and every tag change re-fetched the
file with tracking on. The fetches now skip tracking, and the preview marks
each opening viewed once.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Count Channels with any of the Tags in the Other search estimate
The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Search Channels by name in the Other search tab
The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing. Channels now match the search text by name,
and the Tags when there are both. The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Domains to the Other search tab
Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Playlists to the Other search tab
Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint. The search and the estimate share search_collections_query. The
search now loads each Collection's Downloads up front instead of once per
Collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.
- A Repo is a Collection (kind='repo') plus a `repository` row. The new `git` downloader clones it, then updates it
(weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
An existing clone can be imported where it is. repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download. READMEs are searchable
under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
updated.
- Tagging a Repo moves its clone as one directory. Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
destination. A failed tag move restores the Collection's tag. A Collection's directory cannot leave the media
directory.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse playlist changes until playlists.yaml is imported
While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change). Playlist changes now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse domain changes until domains.yaml is imported
A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import. These now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* archive: importing domains.yaml keeps domains which have archives
Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it. A domain with Archives is now kept (and saved to the config with the next change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* videos: refuse Channel changes until channels.yaml is imported
A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deleting a Zim cascade-deletes its TagZimEntry rows, and the next tags
save writes them out of the config, so every outdated-Zim delete lost
its entry tags. The config-import migration only runs at startup, too
late to help. Move each outdated Zim's tags to the current Zim of the
same name before deleting, then save the tags config.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Reset file worker status when a synchronous refresh is cancelled
refresh_sync runs inside request handlers. When Sanic cancelled one on
response timeout, the phase flag cleared as its `with` block unwound but
the shared status did not, so the worker reported 'modeling' forever while
nothing ran. Reset to idle on cancellation.
Deleting outdated Zims also ran the global modelers inside the request,
which is what timed out. Skip post-processing there and update the
outdated_zims flag directly, the only cleanup step it relied on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: observe the modeling flag in the cancelled refresh_sync test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: one request-scoped database handle per API request
Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that
Session's connection, so a request needs at most one SQLite connection. Creating it does no I/O; requests
that never touch the database never connect. `request.ctx.session` remains the handle's Session.
A `RequestSession` binds itself to one lazily-opened Connection. An engine-bound Session releases its
connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected.
Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the
write lock is not held for the rest of the handler or by background work it starts. This replaces the
handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder,
channel create). The zim search handlers use the request's Session in their worker thread.
Tests count the connections a request opens with production's per-call sessions
(`count_request_connections`). Endpoints whose library functions still open their own connection are
strict xfails, to be removed as those functions take the request's session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: name the real extra connections in the request connection xfails
Channel detail: get_channel already uses the request session; the extra connection is
Channel.get_statistics opening a raw cursor (get_db_curs). Search: count_file_groups opens a
cursor for the total, then handle_file_group_search_results opens a cursor and a session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
cachetools 6 removed cachetools.func.mru_cache; the Zim search estimate
cache was its only user, so it becomes an lru_cache and the pin moves
to 7.2.0 with the Dependabot ignore removed.
reportWebVitals still destructured the web-vitals 2.x names (getCLS,
getFID, ...) which no longer exist, so it silently did nothing. Use
the on* names, replace FID with INP, return the import promise, and
accept any callable rather than instanceof Function.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The listing still read every channel row (for the summary columns) and every download
row of the listed Collections (for the minimum frequency). Both tables store yt-dlp
info_json, so rows are tens of KB and each sits on its own scattered page; selecting
fewer columns did not help because the seek per row is the cost. From a cold disk
cache that was 5 s per table, 10+ s total, on a large library.
Two covering indexes, channel (collection_id, video_count, total_size) and
download (collection_id, frequency), answer both queries without touching a row. The
listing no longer loads Download objects at all; it computes min_download_frequency
with one grouped query and drops the unused `downloads` array from listing rows (the
detail endpoints keep it; the list views only read min_download_frequency).
Adds a models-vs-migrations test (alembic check) so an index or column added to a
model without a migration, or vice versa, fails the suite.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>