Commit graph

122 commits

Author SHA1 Message Date
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
5b29144d70 Testing that nearly-empty configs are ignored on import. Adding setting to disable automatic upgrade checking. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
c635267eda
Fixing bug where Archive screenshots would not be tracked when a user uploads a screenshot.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-10-12 17:19:54 -06:00
Roland
76a454ea34
Linking to uploaded file on download after a user uploads a file with the same URL.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-08-11 16:54:10 -06:00
Roland
57081e327e Converting singlefile_upload_switch_handler to use multiprocessing.Queue to enable uploading of multiple singlefile archives at once. 2025-08-06 14:05:21 -06:00
Roland
3725f0f39d Adding edit button so user can edit downloads, currently supports videos and archives. 2025-08-06 13:03:32 -06:00
Roland
00c86e1a1a
Searching wrolpi user's home directory for browser profiles. 2025-05-02 17:24:51 -06:00
Roland
399cb0f107
Testing that downloads have absolute destinations. Handling 'undefined', 'null', etc, dates. 2025-03-21 17:14:53 -06:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
Roland
15f6e7ed5c
Handling new issue with youtube caption file names. Adding QR Code calculator. 2025-02-07 11:44:17 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
d1f9837ac8
Deleting leftover .part files after a video has been successfully downloaded. 2025-01-12 17:20:29 -07:00
Roland
7d6b7114a7
Adding TRACE level for logging beyond debug. Reworking async_client for tests, not all tests need the App to handle signals, so any test that needs signals can use the new await_switches fixture which primes the app. 2024-12-11 11:52:33 -07:00
Roland
376850ceeb
Moving config api endpoints to their own file.. 2024-12-03 22:33:19 -07:00
Roland
fe521c99f3
Adding edit form for Scrape downloader. 2024-12-03 12:44:38 -07:00
Roland
bf29bdcff7
Displaying error when video file name format is invalid. 2024-11-29 17:01:16 -07:00
Roland
b7d163e7ff
Config files are initialized on startup, if they are missing. 2024-11-28 18:13:43 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
04d92dca09
Not deleting video poster when converting to jpg. 2024-10-16 18:57:22 -06:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
Roland
d3e5b56520
Bumping python requirements versions. 2024-10-16 18:56:41 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
d8ef7e175e
Preserving FileGroup.title when moving. 2024-10-16 18:56:23 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
905042b600
Files which do not have a model can still be uploaded, they are indexed instead. 2024-10-16 18:56:15 -06:00
Roland
3906223844
Moving Channel attachment logic to recurring_download method because only recurring downloads can have Channels. 2024-10-16 18:56:13 -06:00
Roland
a98aebfdda
A video that is uploaded is attached to its Channel. Saving Channels config with both download url and frequency in case downloads config is corrupted. 2024-10-16 18:56:11 -06:00
Roland
c839acb270
Converting Tag methods to use tag id or name. 2024-10-16 18:56:01 -06:00
Roland
2ad363281f
Delaying map startup because it prevents the API from starting. 2024-10-16 18:56:00 -06:00
Roland
af8bd7ba34
Cleaning up some extra data from video info json. 2024-10-16 18:55:59 -06:00
Roland
7933d9625c
Downloads without a frequency cannot be a Channel Download. 2024-10-16 18:55:57 -06:00
Roland
63738bd70f
Removing ChannelDownload in favor of Download.channel_id. 2024-10-16 18:55:54 -06:00
Roland
37ff1ab2cd
Changing title_match to title_include for clarity. 2024-10-16 18:55:51 -06:00
Roland
287beabb9c
Creating missing ChannelDownload when importing channels config. 2024-10-16 18:55:49 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
0fd545d2e8
Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
Roland
ae61977f0b
Reformatting. 2024-06-12 16:47:31 -06:00
Roland
f9227f234c
Fixing logic error where multiprocessing dict was being ignored during initialization. Fixing test which expected the config to be a dict. 2024-05-19 15:46:43 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
947135985d
Reformatting code. 2024-03-12 17:20:51 -06:00
Roland
ce0ae66a9e
Storing video download_datetime. 2024-03-12 11:06:30 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories. 2024-03-09 16:51:26 -07:00
Roland
0d6edc05d0
Duplicate videos are deleted when a video is downloaded. 2024-02-21 12:42:30 -07:00
Roland
cccdd8fe4e
Removing extra parameters from video urls. 2023-11-24 19:06:32 -07:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
ae9d188391
Adding FileGroup.published_date/author/etc. Creating script which converts a Singlefile to an Archive FileGroup files.
Parsing SingleFile datetime using dates.strpdate.

Adding timezone when no timezone info is available.

Installing python-dateutil to assist in handling diverse datetime strings.

Reworking refresh progress to use idempotency to display more accurate progress.  Only logging that files have been invalidated if they used to be indexed.

Clearing archives to force re-index.

Ignoring 'Unknown' author from PDF metadata. Using file_title when pdf metadata title is empty.
2023-10-23 18:09:08 -06:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00