Commit graph

123 commits

Author SHA1 Message Date
Roland
0ffbc05e91
Unifying video download settings so that global/channel/rss/video
download settings have inheritance.  Allowing more granular control of
potentially failed downloads.
2026-03-05 17:40:17 -07:00
Roland
bdc29dace1
Skipping recent file tracking for ignored directories, and video and archive pages. 2026-02-28 08:39:10 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
a6d6aaf9b4
Adding .ffprobe.json for video file groups to speed up re-indexing in the future; adding save_ffprobe_json to the WROLPiConfig to disable this feature. 2026-02-07 18:40:56 -07:00
Roland
63c370b073
Creating Collection reorganization conflict resolution. 2026-02-06 15:05:15 -07:00
Roland
2c8f765f2b
Adding statistics page for Archives. 2026-02-02 15:15:28 -07:00
Roland
9005c0deff
Adding toggle to disable the tags directory feature. 2026-01-31 18:08:42 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
6a9ffcaadc
Adding selection to the Downloads page, user can delete or retry multiple downloads at once. 2025-12-26 10:35:30 -07:00
Roland
1ebcae0965
Allowing user to configure collection download destination directories, as well as directories in the file structure. This will allow sorting archives by destination year, for example. 2025-12-23 16:46:15 -07:00
Roland
78251b067e
Removing WROLPi admin functionality that is now in Controller.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-19 18:41:27 -07:00
Roland
c6df730dcc
Adding bulk tag button to the file browser. User can add tags to many files, but can only remove tags if all files share the tags to be removed. 2025-12-06 19:06:33 -07:00
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
5b29144d70 Testing that nearly-empty configs are ignored on import. Adding setting to disable automatic upgrade checking. 2025-12-06 15:11:37 -07:00
Roland
63a77b87e6 Removing collection metadata. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
1ed0d63bac Creating Generate Screenshot button for Archives which are missing a screenshot. 2025-12-06 15:11:37 -07:00
Roland
00c86e1a1a
Searching wrolpi user's home directory for browser profiles. 2025-05-02 17:24:51 -06:00
Roland
3d6ccad0c0
Adding feature which adds toggles to Zim files to enable/disable auto search. 2025-02-02 12:43:04 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
01338e4d89
Displaying plug icon when API fails to respond. Using window.apiDown to communicate to other hooks to stop fetching, the status interval will keep checking if the API is back up. 2025-01-24 15:35:26 -07:00
Roland
74a1486f0e
Hiding syntax errors when API is down. 2024-12-18 17:32:00 -07:00
Roland
376850ceeb
Moving config api endpoints to their own file.. 2024-12-03 22:33:19 -07:00
Roland
fe521c99f3
Adding edit form for Scrape downloader. 2024-12-03 12:44:38 -07:00
Roland
bf29bdcff7
Displaying error when video file name format is invalid. 2024-11-29 17:01:16 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
d4fc3d1c07
Moving Configs to SettingsPage. 2024-10-16 18:56:55 -06:00
Roland
4b6fb1d37f
Updating DirectorySearch to allow creation of new directories. 2024-10-16 18:56:51 -06:00
Roland
acf3dee179
Configs are displayed in new ConfigsPage under Admin.
- Configs can be imported and saved.
- Configs are validated before imported.
- API will refuse to save a config if it was not imported successfully.
2024-10-16 18:56:45 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
53d2402d32
Only files that have been tagged will be returned when new 'Any' tag button is used to filter file search results. 2024-10-16 18:55:56 -06:00
Roland
37ff1ab2cd
Changing title_match to title_include for clarity. 2024-10-16 18:55:51 -06:00
Roland
7a784d7c45
Excluded URLs was not displayed correctly. 2024-10-16 18:55:48 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
63381c1d77
Created download retry button. 2024-05-08 17:05:12 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
969a065a6c
Splitting FileGroups from search suggestions to speed up channel/domain response time. 2024-04-16 09:46:47 -06:00
Roland
ae5d47fc21
Shortening refresh button for mobile. 2024-03-11 17:47:27 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
f865d9fb0f
Videos display comments. User can download comments using Refresh button. 2024-02-20 13:47:12 -07:00
Roland
6d5a56cacc
A user can ignore a directory by selecting it and ignoring it in the FileBrowser. 2024-02-19 14:44:20 -07:00
Roland
1bb333e43b
Adding preview URL query which allows sharing of a specific file. 2024-02-17 14:53:39 -07:00
Roland
23d84ed107
Search suggestions can filter by mimetypes. 2024-02-05 16:04:19 -07:00
Roland
3ad7b8af00
Global search waits for selection before submitting. 2024-01-01 15:16:03 -07:00
Roland
377c69e8a7
Adding search icon button to navbar. 2024-01-01 11:31:17 -07:00
Roland
d87bc7b314
Fetching search estimate with tags separate from suggestions. 2023-11-29 21:09:39 -07:00
Roland
bbfe76890b
Converting Search Suggetions to a Context. Search estimates is removed and suggestions is used instead. 2023-11-22 12:19:49 -07:00