* Jobs queue and video transcode from the video page
Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time. Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.
Add a transcode Job and POST /api/videos/<id>/transcode. ffmpeg reports
progress via -progress pipe:1. The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.
Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode). The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: move Delete into the Edit dropdown
The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist. Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: Edit > Edit to change a video's title
PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow. Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: edit the description in Edit > Edit
PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it. The
modal sends only what changed and refetches the description after saving.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode modal: allow a container-only remux
Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start. The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* WROL Mode: refuse video edits and transcodes past the API
The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live. The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: verify the output before replacing the original
After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.
The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete
Review fixes. The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock. enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs. JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.
The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original). A crash in between leaves
both files with the database pointing at a real one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: the Edit menu owns the transcode Job; polling survives Close and errors
Review fixes. The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed). While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits. A failed poll
request is retried instead of ending the polling. getJobs checks the
response like its siblings. Design narration trimmed from comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: audio files, and Remove video to turn a video into an audio file
video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused. Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one. faststart only for mp4/m4a. Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).
The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: yellow Edit button
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost
Review fixes. The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index. Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation). Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Jobs: _claim_job docstring states the invariant, not the old race
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: fragmented mp4 option for very long videos
A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it. `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly. mp4/m4a only; a remux, so it takes seconds per GB.
Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode modal: the Fragmented label says audio for audio output
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section
Editing a title or description no longer overwrites yt-dlp's keys. The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file. An empty custom description means none.
Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download. FileGroup.get_wrolpi_json reads it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page
Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:
* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending
(the normal state) this froze the loop for a second at a time; about a third
of all requests, including trivial ones, took ~1,000 ms. It now polls
`is_set()` and returns immediately.
* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
ran synchronous SQLite queries on the event loop, so one slow search stalled
every other request on that worker. They now run via `asyncio.to_thread`.
* The unfiltered Videos page (`POST /api/videos/search` with only an order)
used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
every video's index entry before LIMIT applies: seconds cold on a large
library on a spinning disk, and ~10 s hot for a deep page. It now seeks each
exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
rows and merges them, with the total from a separate count cached for 60 s.
Results are identical to the generic query.
Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a
gitignored JSON file; an example is included.
Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve the video description from its own endpoint instead of Video.__json__
`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk. The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays. The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.
Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions. The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored. The player page fetches it through
`useVideoExtras` alongside the comments.
Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: thread-local zim sessions, fast-path equivalence test, comments
* Zim search, single-zim search, and search estimates run libzim off the event
loop, each opening its own session inside the worker thread rather than
passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
query and asserts identical rows and paging for both date orders. The one
intentional difference is documented: past the end, the fast path still
reports the true total.
* switch_worker returns before its try block so the empty path does not clear
the event; its comment states the invariant only. The copied "synchronous
SQLite" comments are reduced to one line in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Restore the censored-video search filter
Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.
Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored). When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
Displaying star for Uploader's comments, heart for favorited comments.
Adding wrolpi help files desktop shortcut to rpi image.
Using zim file names as FileGroup.title.
Removing accidental output commit.
* A new zim module has been added, this module allows several things to interact with Zim files.
* Allow searching of Zim files using libzim.
* Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
* Creation of a new zim docker container which allows viewing of the Zim file contents.
* This commit also adds move/rename/mkdir features in the File Browser.
* File tags are preserved when moving/renaming.
Tags can be created and deleted in UI.
Adding tag_file using id to avoid session error.
Using UTC in test.
Matching tag_names style in test.
Tag icon is displayed on file rows.
Archives, Videos and Files can be filtered by their tags.
Tags can be added and removed from UI.
Tags are displayed in VideoPlayer.
The tags of a file are returned in it's JSON.
Video favorites are converted to Favorite tag on channel config import.
Updating tests to match frontend usage.
PDF creator and modification time are displayed.
Reworking file search to use Video.__json__ for complex video relations.
Archives can be viewed.
Removing ebook.title.
Reworking alembic migration to be an upgrade rather than a replace.
Skipping test on circleci.
Clearing out video caption index only if extract_subtitles if off.
Channel tests pass.
Some video tests pass.
Missing videos can be found. A video can be downloaded.
Orphaned files can be found.
Videos can be tagged.
Videos can be searched.
Videos can be modeled.
Converting some tests to use FileGroup.primary_path.
Adding archive and domains to alembic migration.
Testing cascading Archive titles.
Archives now uses FileGroup.
archive.lib tests pass.
Archive is partially converted to FileGroup.
Calculating FileGroup.primary file on refresh, rather than on select.
wrolpi tests pass.
Using FilesType to convert between str/pathlib.Path for FileGroup.files.
Using list of files in FileGroup.files instead of dictionary.
Storing files in dedicated FileGroup.files to avoid overwriting them.
Adding comment explaining CASE.
Not overwriting FileGroup.data on refresh if no change has been detected.
Removing File model in favor of FileGroup.data.
Adding typing to Files methods.
Files are indexed using the primary mimetype of the FileGroup.
files test_lib tests pass.
Removing file_group_file in favor of simple one-to-many relationship.
PDF files are indexed during refresh.
Adding Tag and TagFile to tag files with specific tags.
Only iterating when full_stem_ids has contents.
Testing that FileGroups are related to their Files.
Files and file_groups are created and linked.
Bump webpack from 5.75.0 to 5.76.1 in /app (#160)
Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)
---
updated-dependencies:
- dependency-name: webpack
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>