Commit graph

125 commits

Author SHA1 Message Date
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long. 2024-10-16 18:55:52 -06:00
Roland
638e92bd4e
Do not log entire file when singlefile creation fails. 2024-10-16 18:55:44 -06:00
Roland
0bee458b79
Sorting keys when saving json file. 2024-06-13 14:08:07 -06:00
Roland
ae61977f0b
Reformatting. 2024-06-12 16:47:31 -06:00
Roland
2dae32fbcc
Moving singlefile checking to inside of do_singlefile function. Using User Agent when doing singlefile. 2024-05-28 19:18:51 -06:00
Roland
bd8411bcd7
Updating test with Big Solidus. 2024-05-25 09:45:16 -06:00
Roland
d10a228692
Using stdout as error log when singlefile fails. 2024-05-25 09:40:29 -06:00
Roland
18373a466b
Preserving stdout to Download when singlefile fails. 2024-05-15 12:59:13 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
947135985d
Reformatting code. 2024-03-12 17:20:51 -06:00
Roland
2ec267a65b
Automatically backing up config files each day in config/backup directory. 2024-03-11 17:47:35 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories. 2024-03-09 16:51:26 -07:00
Roland
df849ffb42
Ordering video/archive results when no published_datetime exists. Adding Video.location property which returns the App location of the video. 2024-02-26 13:40:39 -07:00
Roland
61a7fdab10
Creating HTMLIndexer which indexes the text of HTML files. 2024-02-19 15:48:17 -07:00
Roland
59b23b3134
Creating function which appends tag names sub-select. 2024-02-10 21:11:19 -07:00
Roland
1b92d1e580
Global search uses Semantic SearchResultsInput component, other components use SearchInput. 2023-11-21 13:01:52 -07:00
Roland
73cb142ae1
Reverting to older singlefile container to keep archive screenshot ratio. Using FileGroup.url on ArchivePage. 2023-10-31 10:20:45 -06:00
Roland
9ea2dadde5
Raising specific error when datetime cannot be parsed. 2023-10-29 22:15:57 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
ae9d188391
Adding FileGroup.published_date/author/etc. Creating script which converts a Singlefile to an Archive FileGroup files.
Parsing SingleFile datetime using dates.strpdate.

Adding timezone when no timezone info is available.

Installing python-dateutil to assist in handling diverse datetime strings.

Reworking refresh progress to use idempotency to display more accurate progress.  Only logging that files have been invalidated if they used to be indexed.

Clearing archives to force re-index.

Ignoring 'Unknown' author from PDF metadata. Using file_title when pdf metadata title is empty.
2023-10-23 18:09:08 -06:00
Roland
b817534c37
Adding Download.is_new/etc. for code suggestions. Fixing async test warnings. 2023-09-20 12:58:35 -06:00
Roland
a510960c8c
Fixing syntax error. 2023-09-15 16:32:09 -06:00
Roland
f14df90174
Removing unused imports. 2023-09-12 17:49:26 -06:00
Roland
17b3923747
Do not need to fetch all archives when checking if domain has archives. 2023-09-12 17:49:17 -06:00
Roland
7611cf8ea1
Converted ArchiveDownloader to use Downloader.process_runner so archiving can be canceled.
Moving Archive download functions to ArchiveDownloader.

Wrapping all do_download with cancel_wrapper which allows canceling of downloads.
2023-09-12 17:32:29 -06:00
Roland
6959c7a3b4
HTML and JSON files are reformatted when downloaded. 2023-08-26 21:49:24 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
59c4ab2502
Replacing icon property when icon has been requested.
Creating <APIButton/> which handles success/failure/pending during API calls.
2023-07-21 16:52:15 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
bc2ca0e512
Replacing tag filtering with new tag_names_to_sub_select function which uses a single sub-select to filter FileGroups. 2023-05-18 22:39:19 -06:00
Roland
fedbbe31d7
Changing default Survival tag to Primitive. 2023-05-18 17:25:33 -06:00
Roland
ac06a9b8b2
Joining when sorting by size. 2023-05-16 14:59:35 -06:00
Roland
7c1cc07365
Only joining FileGroup table when searching Archives when necessary. 2023-05-16 14:56:53 -06:00
Roland
e06f846c8c
Updating function names to match their functionality. 2023-05-03 13:05:57 -06:00
Roland
8de805b6d1
Updating tests to match searching for Channel and Domain names for directories. 2023-05-03 12:59:47 -06:00
Roland
387282b445
Adding my public gpg key.
Moving upload destination input out of drop Segment.

Disabling uploads when downloads are disabled.

Inverting labels for checkboxes in inverted modal.

Deleting file_groups which no longer contain the exclusive primary path.

Adding debug logging for all API errors.

Checking that upload output is not a directory.

Expanding nginx request size.

Adding docs to upload endpoint.

Using dot-circle icon for more image mimetypes.

Using Form.Field to make upload destination required.

Adding border to inverted Segments.

Many files can be uploaded in chunks.

Files can be uploaded, progress is displayed.

Showing white border around share QR code.

Displaying WROLPi colors for download icons.

Creating button to clear DirectorySearch.

Excluding channel/domain directories so other directories can be returned.

Ordering search results.

Ignoring when no channel is found only if we have a destination.

Limiting directory search results.

Displaying categories when searching destination directories.

Handling when live_chat.json file exists.

Allowing refresh of directory directly.

Logging which download was successful.

Removing timeout.

Creating index for directory names.

Sleeping to catch cancel.

Arbitrary files can be downloaded using the new FileDownloader.
2023-04-28 13:56:32 -06:00
Roland
f9ad9981db
Testing Archive headline search. 2023-04-12 19:07:16 -06:00
Roland
bba6a341b7
Only fetching headlines if requested to speed up RPi responses.g 2023-04-12 18:51:26 -06:00
Roland
29e562c520
Headlines can be displayed for file search results. 2023-04-12 18:09:10 -06:00
Roland
5e206fd9a2
Adding more details to Archive slow_logger. 2023-04-12 11:51:28 -06:00
Roland
6fe536558a
You cannot delete a tagged FileGroup. 2023-04-12 11:42:13 -06:00
Roland
18273286dc
Removing FileGroup.full_stem in favor of primary_path. This is to handle when two unrelated files share a name ('some text.txt' and 'some text.jpg'). 2023-04-09 16:59:20 -06:00
Roland
bab10a0c96
Creating sort button which allows the user to select a sort order via modal. 2023-04-09 14:13:36 -06:00
Roland
9ca099267b
Refreshing file when attempting to add Tag to file that has not been refreshed. Adding url to skip list when deleting download. 2023-04-05 10:00:08 -06:00
Roland
ede9d2236b
Video and Archive can be tagged when downloading. 2023-04-03 09:34:42 -06:00
Roland
55cf76a884
Requiring downloader name to be specified. 2023-04-01 15:34:46 -06:00
Roland
ae2b6daa33
FileBrowser files can be tagged. A Tag icon is displayed on each file row in the browser. 2023-04-01 11:52:41 -06:00