Commit graph

109 commits

Author SHA1 Message Date
Roland
bf29bdcff7
Displaying error when video file name format is invalid. 2024-11-29 17:01:16 -07:00
Roland
b7d163e7ff
Config files are initialized on startup, if they are missing. 2024-11-28 18:13:43 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
4905235083
Trimming file names more because yt-dlp uses huge temporary names while downloading. 2024-10-16 18:57:10 -06:00
Roland
0784f740bb
Reworking Downloader.process_runner to use real stdout/stderr files to avoid issue with pipes being truncated. 2024-10-16 18:57:07 -06:00
Roland
584fdadcb7
Getting file name from yt-dlp error when video file name is too long, trimming that long file name. 2024-10-16 18:56:56 -06:00
Roland
b3380aeacb
Trimming video file names when downloading. Using yt-dlp error to get filename to help yt-dlp extract info json. 2024-10-16 18:56:53 -06:00
Roland
77ad52473a
Fixing upsert_file usage. Adding log details for error. 2024-10-16 18:56:43 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
af8bd7ba34
Cleaning up some extra data from video info json. 2024-10-16 18:55:59 -06:00
Roland
958dbb899d
Using uploader data as fallback for channel creation when downloading video. 2024-10-16 18:55:53 -06:00
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long. 2024-10-16 18:55:52 -06:00
Roland
37ff1ab2cd
Changing title_match to title_include for clarity. 2024-10-16 18:55:51 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
0fd545d2e8
Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
Roland
0bee458b79
Sorting keys when saving json file. 2024-06-13 14:08:07 -06:00
Roland
0fbbd36297
Creating perpetual funciton which checks if the internet is accessible. This is used to prevent downloads from starting when no internet exists. 2024-05-31 23:54:00 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
ba2dd6dd11
Many fixes split out from the feature/unified-query branch. Installing diagnostic tools in the built images. Displaying video comment statistics. Echoing error when install.sh fails. 2024-04-14 13:13:30 -06:00
Roland
a665677ecc
Adding more lower-quality video formats to download when no other is available. Getting video ffprobe json when downloading. 2024-03-16 19:58:58 -06:00
Roland
4b8b897eb3
Unifying YoutubeDL. 2024-03-12 16:52:26 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
Displaying star for Uploader's comments, heart for favorited comments.

Adding wrolpi help files desktop shortcut to rpi image.

Using zim file names as FileGroup.title.

Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
df849ffb42
Ordering video/archive results when no published_datetime exists. Adding Video.location property which returns the App location of the video. 2024-02-26 13:40:39 -07:00
Roland
b705e08e48
Handling if multiple videos exist at start of video download. 2024-02-22 13:12:26 -07:00
Roland
2aa30e1674
Getting video location to report errors when getting video metadata again. 2024-02-22 08:47:35 -07:00
Roland
0d6edc05d0
Duplicate videos are deleted when a video is downloaded. 2024-02-21 12:42:30 -07:00
Roland
08fa3af453
Moving video about pane to it's own segment to avoid tabs going out of mobile viewing area. 2024-02-20 15:13:18 -07:00
Roland
5629adc84c
Limiting normal video comment download limit. 2024-02-20 14:06:13 -07:00
Roland
f865d9fb0f
Videos display comments. User can download comments using Refresh button. 2024-02-20 13:47:12 -07:00
Roland
760d949f5e
Adding debugging when finding channel. 2023-11-24 19:23:25 -07:00
Roland
cccdd8fe4e
Removing extra parameters from video urls. 2023-11-24 19:06:32 -07:00
Roland
73cb142ae1
Reverting to older singlefile container to keep archive screenshot ratio. Using FileGroup.url on ArchivePage. 2023-10-31 10:20:45 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
Roland
7611cf8ea1
Converted ArchiveDownloader to use Downloader.process_runner so archiving can be canceled.
Moving Archive download functions to ArchiveDownloader.

Wrapping all do_download with cancel_wrapper which allows canceling of downloads.
2023-09-12 17:32:29 -06:00
Roland
da6d6a8094
Allowing starting a download of a channel without frequency. Doing this creates a once-download. 2023-08-30 14:01:58 -06:00
Roland
6959c7a3b4
HTML and JSON files are reformatted when downloaded. 2023-08-26 21:49:24 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
7f54e592ed
A user can define a destination when creating a channel/playlist download. This destination is then sent to VideoDownloader. 2023-08-17 18:11:57 -06:00
Roland
b9d79d5440
Downloads can be re-created and their settings are preserved. 2023-08-13 10:55:10 -06:00
Roland
e8e5b5727f
Creating Video.validated which is used by find_corrupt_videos.py to search. 2023-08-08 13:42:55 -06:00
Roland
dec72f8d45
Converting captions to vtt when downloading videos. 2023-08-04 11:02:03 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
bab7082634
Downloading video even if the channel cannot be found. 2023-06-09 16:04:44 -06:00
Roland
387282b445
Adding my public gpg key.
Moving upload destination input out of drop Segment.

Disabling uploads when downloads are disabled.

Inverting labels for checkboxes in inverted modal.

Deleting file_groups which no longer contain the exclusive primary path.

Adding debug logging for all API errors.

Checking that upload output is not a directory.

Expanding nginx request size.

Adding docs to upload endpoint.

Using dot-circle icon for more image mimetypes.

Using Form.Field to make upload destination required.

Adding border to inverted Segments.

Many files can be uploaded in chunks.

Files can be uploaded, progress is displayed.

Showing white border around share QR code.

Displaying WROLPi colors for download icons.

Creating button to clear DirectorySearch.

Excluding channel/domain directories so other directories can be returned.

Ordering search results.

Ignoring when no channel is found only if we have a destination.

Limiting directory search results.

Displaying categories when searching destination directories.

Handling when live_chat.json file exists.

Allowing refresh of directory directly.

Logging which download was successful.

Removing timeout.

Creating index for directory names.

Sleeping to catch cancel.

Arbitrary files can be downloaded using the new FileDownloader.
2023-04-28 13:56:32 -06:00
Roland
ede9d2236b
Video and Archive can be tagged when downloading. 2023-04-03 09:34:42 -06:00