All three bugs confirmed; the two suggestions taken.
[bug] `video_modeler` awaited ffprobe -- a subprocess per video, up to 20
per batch -- inside its `commit=True` session. With IMMEDIATE as the
default that holds the write lock for the whole batch, which is the same
starvation this branch is curing in `video_cleanup`. (It was already
partly true before: the batch's first `session.flush([video])` took the
lock ahead of the remaining ffprobes.) The batch is now read, ffprobed
with no transaction open, then written. `get_or_create_ffprobe_json` in
modules/videos/common.py is the path-only entry point that makes that
possible; `Video.get_ffprobe_json` delegates to it and keeps appending
the cache file to its FileGroup. `validate()` -- and the poster
generation it can trigger -- is still inside the write transaction, and
now says so.
[bug] `test_download_dispatched_outside_immediate_transaction` probed
`getattr(_immediate_txn, 'active', False)`, which a ContextVar never has,
so it passed unconditionally. It now probes the actual write lock from a
competing connection under `production_like_sessions`. Mutation-checked:
moving the dispatch back inside the claim transaction fails it.
[bug] `asyncio.create_task` snapshots the current Context, so a flag held
for a write session's lifetime was inherited by every task spawned inside
it -- and the parent's `reset()` cannot reach a child's copy, leaving
that task issuing BEGIN IMMEDIATE for its *read* sessions.
`create_downloads` spawns `dispatch_downloads` exactly that way, so every
user-created download hit it. The flag now lives only across the
session's opening `connection()` call, which also means the write lock is
taken when the session opens rather than at the caller's first statement.
[suggestion] `renew_recurring_downloads` scanned every recurring Download
under the write lock, including the usual cycle where nothing is due. It
reads first now and takes the lock only for the rows that change,
re-checking status since the dispatcher may have claimed one in between.
[nit] Parenthesized the directory OR in the claim join.
`probe_write_lock_is_held` moved to conftest alongside
`write_lock_held_briefly`; three suites use it now.
Suite: 1712 passed, 5 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three "database is locked" failures on the docker WROLPi, one cause each:
* `INSERT INTO download` from POST /api/download -- a user creating a
video download while a refresh ran (2026-08-06 14:06:46);
* six `INSERT INTO file_group` from `upsert_file`, uploads landing in
the same three-minute window;
* `UPDATE download SET status` from `renew_recurring_downloads`.
All three are the deferred read-then-write trap: a transaction that
begins as a reader and upgrades at its first write is refused
*instantly* when another connection wrote in between, because
busy_timeout is skipped for lock upgrades. `get_immediate_db_session`
existed to avoid it, but an opt-in helper only works if every author
remembers, and read-then-write is the ordinary shape of a write. So it
is gone: `get_db_session(commit=True)` -- the existing declaration of
write intent -- now begins IMMEDIATE. Rails 7.1 and Django 5.1 made the
same call, defaulting SQLite transactions to IMMEDIATE rather than
asking application code to opt in.
That default is only safe if write transactions stay short, and one did
not. `video_cleanup` ran its whole Channel-claiming join --
`UPDATE video ... FROM channel ... LEFT JOIN file_group`, O(channels x
file_groups) -- inside one write transaction: 110 to 130 seconds each of
the eight times it appears in the log, on a library with ~1500 channels.
While it ran, *every* writer on the box failed, including ones that
began IMMEDIATE, because the hold far exceeds the 30s busy_timeout. The
14:04-14:06 failures above sit inside its 130-second window. The join
is now a read -- WAL readers never block writers -- and only the
resulting UPDATEs take the lock, 500 rows per transaction.
`claim_videos_for_channels` had the same join and the same fix; channel
import calls it with every channel in channels.yaml.
Also here:
* `renew_recurring_downloads` was a read session with an inner commit;
it is a write session now. Its computed `next_download` values now
persist even when nothing was renewed -- that is the point of
computing them.
* POST /api/download no longer writes through `request.ctx.session`.
That session is committed by the response middleware, so a writer
would hold the lock across the whole request; a write session
commits before the handler returns, ahead of the background dispatch
it fires.
* `_immediate_txn` is a ContextVar, not a threading.local. The
writers that set it include coroutines, and one asyncio thread runs
many of them interleaved.
* `write_lock_held_briefly` moved from the move tests into conftest,
since two suites now need it.
Both regression tests were verified by mutation: reverting either site
alone fails its own test, and the API one fails with the production
error verbatim.
Sources: fractaledmind.com/2023/12/11/sqlite-on-rails-improving-concurrency,
tenthousandmeters.com/blog/sqlite-concurrent-writes-and-database-is-locked-errors,
berthub.eu/articles/posts/a-brief-post-on-sqlite3-database-locked-despite-timeout
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the flush to immediately before FileGroup.move so it covers every
deletion queued in this function, not only the one found by
get_by_path. The earlier missing-file deletions were already reaching
the DB -- the get_by_path query autoflushes them -- but that is an
incidental property of the query sitting between them and the rename,
not something the rename should depend on.
Also assert the kept Video's old path is gone, so the test reads a move
rather than a copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`delete_duplicate_videos` deletes the smaller copy sitting at the
destination path, then renames the copy it keeps onto that path.
`FileGroup.delete` unlinks the files right away but only queues the row
deletion on the session, and SQLAlchemy emits UPDATEs before DELETEs
within a flush -- so the rename's UPDATE of `primary_path` collided with
the row that was about to be deleted:
sqlite3.IntegrityError: UNIQUE constraint failed: file_group.primary_path
The whole download transaction rolled back, so a re-downloaded video
lost its DB record while its files stayed on disk (and the previous
copy's files were already unlinked).
Flush the delete before claiming the path. The earlier missing-file
deletions in the same function never hit this because the `get_by_path`
query between them autoflushes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The yt-dlp `-f` selector was built by joining every selector of every
requested resolution with a comma. In yt-dlp a comma means "download all
of these formats"; `/` is the fallback operator. The resolution map is
ordered best-to-worst and its entries already use `/` internally, so the
comma was never the intent.
With the default resolutions this handed yt-dlp ~45 comma-separated
selectors and asked it to download every one that matched. A 720p video
matched 8 of them (including duplicates, which yt-dlp does not dedupe), so
the whole per-download pipeline ran 8 times against the same output path:
subtitles downloaded, thumbnail downloaded, info.json written, subtitles
converted, thumbnail converted, existing subtitles deleted, repeat. The
duplicate work was invisible because each pass overwrites the last.
Join with `/` so the resolution list behaves as the preference chain it
reads as.
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
- strip_youtube_caption_positioning now writes to a temp file in the same
directory and os.replace()s the original, so a failed/partial write
(e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
which parses the hostname and matches youtube.com / youtu.be (and
subdomains), rejecting lookalikes like youtube.com.example.org and
notyoutube.com. Add a parametrized test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (via yt-dlp) stamp every cue timing line
with `align:start position:0%`, pinning the on-screen text to the
bottom-left of the player. This strips that exact cue setting from the
downloaded .vtt files so browsers render the captions centered.
- add strip_youtube_caption_positioning() to wrolpi/captions.py — a
targeted text replacement (not a webvtt round-trip) so the `<c>`
word-timing tags in the cue text are preserved; matches only the exact
auto-caption signature, so deliberately positioned cues (e.g. with a
`line` setting) are left untouched; idempotent.
- call it from VideoDownloader after normalize_video_file_names, gated on
a youtube.com / youtu.be source URL. It runs before the FileGroup is
created, so the first index reads the corrected file — no re-index
plumbing needed.
Complements the client-side fix in #543 (which centers captions in the
player for existing/any-source files); this makes new YouTube downloads
correct at the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.
Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.
channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.
Unlike the old implementation, the sort degrades gracefully:
- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
download fails cleanly with UnrecoverableDownloadError telling the
user to update yt-dlp or switch to Newest/Oldest — no retry loop.
Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a Channel refresh, log how many video/audio FileGroups are in the
Channel's directory, and how many are assigned to this Channel, other
Channels, or no Channel. Helps debug Videos missing from their Channel
after files are moved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag_file join in Channel.get_statistics() produced one row per tag,
so a Video with N Tags was counted N times in video_count, size, and
length. Count tagged Videos in a subquery instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
YouTube's new listing layout means yt-dlp no longer returns `view_count`
in flat channel/playlist entries, so sorting by views raised KeyError and
the download failed and retried forever.
Sorting by views now requires per-video extraction (hundreds of requests
per channel), so the feature is deprecated rather than degraded:
- downloader: the 'views' branch raises UnrecoverableDownloadError so the
download fails cleanly (no retry loop) with a message to switch to
Newest/Oldest. May be revisited via the YouTube Data API.
- schema: download_order validation drops 'views' (newest/oldest/null only).
- UI: removed "Most Views" from the download-order dropdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day
Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.
Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
stops dispatching counted downloads for the cycle. daily_download_counts derives the
day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
(base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
validation, 0 = unlimited), and the React Settings page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore the censored-video search filter
Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.
Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored). When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`download_info_json` called `Video.get_by_url(url, session)`, but the method
signature is `get_by_url(session, url)`. This passed the URL string where the
session was expected, raising `AttributeError: 'str' object has no attribute
'query'` whenever a Video metadata refresh ran.
Swap the arguments at both call sites and add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- In get_effective_video_settings: on unparseable sleep_requests, delete
the key instead of forcing 0.0. This lets the global config default
(already present from _extract_global_inheritable) take effect, as
suggested by Greptile.
- In handleInputEvent: add explicit `value !== ''` guard before the
isNaN check. Prevents silent NaN in form state for empty number
fields (isNaN('') === false in JS).
Both changes are strict improvements with no behavior change for
valid inputs.
`sleep_requests` (and other numeric inheritable video download settings)
could arrive as strings ('' or '0.75') from the frontend NumberInputForm
and DownloadRequest bodies. These were persisted into Download.settings
(JSONB) and later triggered:
if effective['sleep_requests'] > 0:
TypeError: '>' not supported between instances of 'str' and 'int'
in VideoDownloader.execute_download (and similar sites).
Root cause: DownloadSettings.__post_init__ only filtered None/[] and did
no type coercion, even though the dataclass declared Optional[float]/int.
Changes (minimal targeted fix):
- Add _coerce_optional_number helper and apply it to all numeric fields
in DownloadSettings.__post_init__ (the existing sanitization boundary).
- Centralize legacy string coercion for persisted data in the single
read chokepoint: get_effective_video_settings.
- Simplify VideoDownloaderConfig.sleep_requests getter.
- Frontend: use parseFloat for type=number inputs + null defaults for
sleep_requests (prevents '' from ever being submitted).
- Add focused regression test for string coercion path.
- Revert unrelated change to string field filtering behavior.
No NaN/Inf speculation, no duplicated try/float guards at every call site,
no behavior change for non-numeric settings.
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version