* Remove a file's Tags Directory links when WROLPi deletes it
When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not. The link became the only copy, so the tags sync
refused to delete it and warned on every sync. The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove Tags Directory links on path deletes; keep a valid poster's link
The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links. It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.
A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest. A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted. The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.
Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before. The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Move the flush to immediately before FileGroup.move so it covers every
deletion queued in this function, not only the one found by
get_by_path. The earlier missing-file deletions were already reaching
the DB -- the get_by_path query autoflushes them -- but that is an
incidental property of the query sitting between them and the rename,
not something the rename should depend on.
Also assert the kept Video's old path is gone, so the test reads a move
rather than a copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`delete_duplicate_videos` deletes the smaller copy sitting at the
destination path, then renames the copy it keeps onto that path.
`FileGroup.delete` unlinks the files right away but only queues the row
deletion on the session, and SQLAlchemy emits UPDATEs before DELETEs
within a flush -- so the rename's UPDATE of `primary_path` collided with
the row that was about to be deleted:
sqlite3.IntegrityError: UNIQUE constraint failed: file_group.primary_path
The whole download transaction rolled back, so a re-downloaded video
lost its DB record while its files stayed on disk (and the previous
copy's files were already unlinked).
Flush the delete before claiming the path. The earlier missing-file
deletions in the same function never hit this because the `get_by_path`
query between them autoflushes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
Python 3.11 is now required, old versions were causing unpredictability in tests. (Sanic does not yet
support 3.12)
Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
https://github.com/sanic-org/sanic/issues/2921
New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes. The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation. Now all multiprocessing tools are attached to the
app.shared_ctx. This means `api_app` is imported in many, many places.
This forced a change in how the DownloadManager works. Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue. Now, a single worker checks
for new downloads and sends a Sanic signal.
Flags have been reworked to use the `api_app`. I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
* A new zim module has been added, this module allows several things to interact with Zim files.
* Allow searching of Zim files using libzim.
* Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
* Creation of a new zim docker container which allows viewing of the Zim file contents.
* This commit also adds move/rename/mkdir features in the File Browser.
* File tags are preserved when moving/renaming.
Tags can be created and deleted in UI.
Adding tag_file using id to avoid session error.
Using UTC in test.
Matching tag_names style in test.
Tag icon is displayed on file rows.
Archives, Videos and Files can be filtered by their tags.
Tags can be added and removed from UI.
Tags are displayed in VideoPlayer.
The tags of a file are returned in it's JSON.
Video favorites are converted to Favorite tag on channel config import.
Updating tests to match frontend usage.
PDF creator and modification time are displayed.
Reworking file search to use Video.__json__ for complex video relations.
Archives can be viewed.
Removing ebook.title.
Reworking alembic migration to be an upgrade rather than a replace.
Skipping test on circleci.
Clearing out video caption index only if extract_subtitles if off.
Channel tests pass.
Some video tests pass.
Missing videos can be found. A video can be downloaded.
Orphaned files can be found.
Videos can be tagged.
Videos can be searched.
Videos can be modeled.
Converting some tests to use FileGroup.primary_path.
Adding archive and domains to alembic migration.
Testing cascading Archive titles.
Archives now uses FileGroup.
archive.lib tests pass.
Archive is partially converted to FileGroup.
Calculating FileGroup.primary file on refresh, rather than on select.
wrolpi tests pass.
Using FilesType to convert between str/pathlib.Path for FileGroup.files.
Using list of files in FileGroup.files instead of dictionary.
Storing files in dedicated FileGroup.files to avoid overwriting them.
Adding comment explaining CASE.
Not overwriting FileGroup.data on refresh if no change has been detected.
Removing File model in favor of FileGroup.data.
Adding typing to Files methods.
Files are indexed using the primary mimetype of the FileGroup.
files test_lib tests pass.
Removing file_group_file in favor of simple one-to-many relationship.
PDF files are indexed during refresh.
Adding Tag and TagFile to tag files with specific tags.
Only iterating when full_stem_ids has contents.
Testing that FileGroups are related to their Files.
Files and file_groups are created and linked.
Bump webpack from 5.75.0 to 5.76.1 in /app (#160)
Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)
---
updated-dependencies:
- dependency-name: webpack
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
