Commit graph

139 commits

Author SHA1 Message Date
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
b98ded62e2
Updating default user agent. 2024-10-20 15:37:07 -06:00
Roland
0784f740bb
Reworking Downloader.process_runner to use real stdout/stderr files to avoid issue with pipes being truncated. 2024-10-16 18:57:07 -06:00
Roland
cb024ab57d
Changing archive download process to output html file, rather than capturing stdout. 2024-10-16 18:57:05 -06:00
Roland
82acca00b6
Changing aiohttp function to be context managers to simplify calls. 2024-10-16 18:56:49 -06:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
d5ca3e7f02
Including Country/Region codes in caption file names. 2024-10-16 18:56:31 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
905042b600
Files which do not have a model can still be uploaded, they are indexed instead. 2024-10-16 18:56:15 -06:00
Roland
c839acb270
Converting Tag methods to use tag id or name. 2024-10-16 18:56:01 -06:00
Roland
2ad363281f
Delaying map startup because it prevents the API from starting. 2024-10-16 18:56:00 -06:00
Roland
53d2402d32
Only files that have been tagged will be returned when new 'Any' tag button is used to filter file search results. 2024-10-16 18:55:56 -06:00
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long. 2024-10-16 18:55:52 -06:00
Roland
638e92bd4e
Do not log entire file when singlefile creation fails. 2024-10-16 18:55:44 -06:00
Roland
0bee458b79
Sorting keys when saving json file. 2024-06-13 14:08:07 -06:00
Roland
ae61977f0b
Reformatting. 2024-06-12 16:47:31 -06:00
Roland
2dae32fbcc
Moving singlefile checking to inside of do_singlefile function. Using User Agent when doing singlefile. 2024-05-28 19:18:51 -06:00
Roland
bd8411bcd7
Updating test with Big Solidus. 2024-05-25 09:45:16 -06:00
Roland
d10a228692
Using stdout as error log when singlefile fails. 2024-05-25 09:40:29 -06:00
Roland
18373a466b
Preserving stdout to Download when singlefile fails. 2024-05-15 12:59:13 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
947135985d
Reformatting code. 2024-03-12 17:20:51 -06:00
Roland
2ec267a65b
Automatically backing up config files each day in config/backup directory. 2024-03-11 17:47:35 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories. 2024-03-09 16:51:26 -07:00
Roland
df849ffb42
Ordering video/archive results when no published_datetime exists. Adding Video.location property which returns the App location of the video. 2024-02-26 13:40:39 -07:00
Roland
61a7fdab10
Creating HTMLIndexer which indexes the text of HTML files. 2024-02-19 15:48:17 -07:00
Roland
59b23b3134
Creating function which appends tag names sub-select. 2024-02-10 21:11:19 -07:00
Roland
1b92d1e580
Global search uses Semantic SearchResultsInput component, other components use SearchInput. 2023-11-21 13:01:52 -07:00
Roland
73cb142ae1
Reverting to older singlefile container to keep archive screenshot ratio. Using FileGroup.url on ArchivePage. 2023-10-31 10:20:45 -06:00
Roland
9ea2dadde5
Raising specific error when datetime cannot be parsed. 2023-10-29 22:15:57 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
ae9d188391
Adding FileGroup.published_date/author/etc. Creating script which converts a Singlefile to an Archive FileGroup files.
Parsing SingleFile datetime using dates.strpdate.

Adding timezone when no timezone info is available.

Installing python-dateutil to assist in handling diverse datetime strings.

Reworking refresh progress to use idempotency to display more accurate progress.  Only logging that files have been invalidated if they used to be indexed.

Clearing archives to force re-index.

Ignoring 'Unknown' author from PDF metadata. Using file_title when pdf metadata title is empty.
2023-10-23 18:09:08 -06:00
Roland
b817534c37
Adding Download.is_new/etc. for code suggestions. Fixing async test warnings. 2023-09-20 12:58:35 -06:00
Roland
a510960c8c
Fixing syntax error. 2023-09-15 16:32:09 -06:00
Roland
f14df90174
Removing unused imports. 2023-09-12 17:49:26 -06:00
Roland
17b3923747
Do not need to fetch all archives when checking if domain has archives. 2023-09-12 17:49:17 -06:00
Roland
7611cf8ea1
Converted ArchiveDownloader to use Downloader.process_runner so archiving can be canceled.
Moving Archive download functions to ArchiveDownloader.

Wrapping all do_download with cancel_wrapper which allows canceling of downloads.
2023-09-12 17:32:29 -06:00
Roland
6959c7a3b4
HTML and JSON files are reformatted when downloaded. 2023-08-26 21:49:24 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
59c4ab2502
Replacing icon property when icon has been requested.
Creating <APIButton/> which handles success/failure/pending during API calls.
2023-07-21 16:52:15 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
bc2ca0e512
Replacing tag filtering with new tag_names_to_sub_select function which uses a single sub-select to filter FileGroups. 2023-05-18 22:39:19 -06:00
Roland
fedbbe31d7
Changing default Survival tag to Primitive. 2023-05-18 17:25:33 -06:00
Roland
ac06a9b8b2
Joining when sorting by size. 2023-05-16 14:59:35 -06:00
Roland
7c1cc07365
Only joining FileGroup table when searching Archives when necessary. 2023-05-16 14:56:53 -06:00