Commit graph

97 commits

Author SHA1 Message Date
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long. 2024-10-16 18:55:52 -06:00
Roland
37ff1ab2cd
Changing title_match to title_include for clarity. 2024-10-16 18:55:51 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
0fd545d2e8
Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
Roland
0bee458b79
Sorting keys when saving json file. 2024-06-13 14:08:07 -06:00
Roland
0fbbd36297
Creating perpetual funciton which checks if the internet is accessible. This is used to prevent downloads from starting when no internet exists. 2024-05-31 23:54:00 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
ba2dd6dd11
Many fixes split out from the feature/unified-query branch. Installing diagnostic tools in the built images. Displaying video comment statistics. Echoing error when install.sh fails. 2024-04-14 13:13:30 -06:00
Roland
a665677ecc
Adding more lower-quality video formats to download when no other is available. Getting video ffprobe json when downloading. 2024-03-16 19:58:58 -06:00
Roland
4b8b897eb3
Unifying YoutubeDL. 2024-03-12 16:52:26 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
Displaying star for Uploader's comments, heart for favorited comments.

Adding wrolpi help files desktop shortcut to rpi image.

Using zim file names as FileGroup.title.

Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
df849ffb42
Ordering video/archive results when no published_datetime exists. Adding Video.location property which returns the App location of the video. 2024-02-26 13:40:39 -07:00
Roland
b705e08e48
Handling if multiple videos exist at start of video download. 2024-02-22 13:12:26 -07:00
Roland
2aa30e1674
Getting video location to report errors when getting video metadata again. 2024-02-22 08:47:35 -07:00
Roland
0d6edc05d0
Duplicate videos are deleted when a video is downloaded. 2024-02-21 12:42:30 -07:00
Roland
08fa3af453
Moving video about pane to it's own segment to avoid tabs going out of mobile viewing area. 2024-02-20 15:13:18 -07:00
Roland
5629adc84c
Limiting normal video comment download limit. 2024-02-20 14:06:13 -07:00
Roland
f865d9fb0f
Videos display comments. User can download comments using Refresh button. 2024-02-20 13:47:12 -07:00
Roland
760d949f5e
Adding debugging when finding channel. 2023-11-24 19:23:25 -07:00
Roland
cccdd8fe4e
Removing extra parameters from video urls. 2023-11-24 19:06:32 -07:00
Roland
73cb142ae1
Reverting to older singlefile container to keep archive screenshot ratio. Using FileGroup.url on ArchivePage. 2023-10-31 10:20:45 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
Roland
7611cf8ea1
Converted ArchiveDownloader to use Downloader.process_runner so archiving can be canceled.
Moving Archive download functions to ArchiveDownloader.

Wrapping all do_download with cancel_wrapper which allows canceling of downloads.
2023-09-12 17:32:29 -06:00
Roland
da6d6a8094
Allowing starting a download of a channel without frequency. Doing this creates a once-download. 2023-08-30 14:01:58 -06:00
Roland
6959c7a3b4
HTML and JSON files are reformatted when downloaded. 2023-08-26 21:49:24 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
7f54e592ed
A user can define a destination when creating a channel/playlist download. This destination is then sent to VideoDownloader. 2023-08-17 18:11:57 -06:00
Roland
b9d79d5440
Downloads can be re-created and their settings are preserved. 2023-08-13 10:55:10 -06:00
Roland
e8e5b5727f
Creating Video.validated which is used by find_corrupt_videos.py to search. 2023-08-08 13:42:55 -06:00
Roland
dec72f8d45
Converting captions to vtt when downloading videos. 2023-08-04 11:02:03 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
bab7082634
Downloading video even if the channel cannot be found. 2023-06-09 16:04:44 -06:00
Roland
387282b445
Adding my public gpg key.
Moving upload destination input out of drop Segment.

Disabling uploads when downloads are disabled.

Inverting labels for checkboxes in inverted modal.

Deleting file_groups which no longer contain the exclusive primary path.

Adding debug logging for all API errors.

Checking that upload output is not a directory.

Expanding nginx request size.

Adding docs to upload endpoint.

Using dot-circle icon for more image mimetypes.

Using Form.Field to make upload destination required.

Adding border to inverted Segments.

Many files can be uploaded in chunks.

Files can be uploaded, progress is displayed.

Showing white border around share QR code.

Displaying WROLPi colors for download icons.

Creating button to clear DirectorySearch.

Excluding channel/domain directories so other directories can be returned.

Ordering search results.

Ignoring when no channel is found only if we have a destination.

Limiting directory search results.

Displaying categories when searching destination directories.

Handling when live_chat.json file exists.

Allowing refresh of directory directly.

Logging which download was successful.

Removing timeout.

Creating index for directory names.

Sleeping to catch cancel.

Arbitrary files can be downloaded using the new FileDownloader.
2023-04-28 13:56:32 -06:00
Roland
ede9d2236b
Video and Archive can be tagged when downloading. 2023-04-03 09:34:42 -06:00
Roland
fbfb3458e3
Logging channel when it is found. 2023-04-02 14:55:23 -06:00
Roland
06dd8a168e
ChannelDownloader sets sub-downloader for it's videos to download. 2023-04-02 11:20:57 -06:00
Roland
55cf76a884
Requiring downloader name to be specified. 2023-04-01 15:34:46 -06:00
Roland
ae2b6daa33
FileBrowser files can be tagged. A Tag icon is displayed on each file row in the browser. 2023-04-01 11:52:41 -06:00
Roland
5615ce542d
Making example label a tag.
Tags can be created and deleted in UI.

Adding tag_file using id to avoid session error.

Using UTC in test.

Matching tag_names style in test.

Tag icon is displayed on file rows.

Archives, Videos and Files can be filtered by their tags.

Tags can be added and removed from UI.

Tags are displayed in VideoPlayer.

The tags of a file are returned in it's JSON.

Video favorites are converted to Favorite tag on channel config import.

Updating tests to match frontend usage.

PDF creator and modification time are displayed.

Reworking file search to use Video.__json__ for complex video relations.

Archives can be viewed.

Removing ebook.title.

Reworking alembic migration to be an upgrade rather than a replace.

Skipping test on circleci.

Clearing out video caption index only if extract_subtitles if off.

Channel tests pass.

Some video tests pass.

Missing videos can be found.  A video can be downloaded.

Orphaned files can be found.

Videos can be tagged.

Videos can be searched.

Videos can be modeled.

Converting some tests to use FileGroup.primary_path.

Adding archive and domains to alembic migration.

Testing cascading Archive titles.

Archives now uses FileGroup.

archive.lib tests pass.

Archive is partially converted to FileGroup.

Calculating FileGroup.primary file on refresh, rather than on select.

wrolpi tests pass.

Using FilesType to convert between str/pathlib.Path for FileGroup.files.

Using list of files in FileGroup.files instead of dictionary.

Storing files in dedicated FileGroup.files to avoid overwriting them.

Adding comment explaining CASE.

Not overwriting FileGroup.data on refresh if no change has been detected.

Removing File model in favor of FileGroup.data.

Adding typing to Files methods.

Files are indexed using the primary mimetype of the FileGroup.

files test_lib tests pass.

Removing file_group_file in favor of simple one-to-many relationship.

PDF files are indexed during refresh.

Adding Tag and TagFile to tag files with specific tags.

Only iterating when full_stem_ids has contents.

Testing that FileGroups are related to their Files.

Files and file_groups are created and linked.

Bump webpack from 5.75.0 to 5.76.1 in /app (#160)

Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-25 17:54:41 -06:00
Roland
942a35e5d8
Resolving Channel download entries. Detecting and raising error when user has not specified exactly which video URL they want to download. 2023-02-21 15:12:45 -07:00
Roland
df79aa13b1
Handling when channel info json may not have channel_id. 2023-02-17 16:30:13 -07:00
Roland
9dfaabdb9b
Not sending refresh events for channel downloads. 2022-12-29 09:36:24 -07:00
Roland
0152b8648e
Attaching Channel's info_json even if the download is a playlist. 2022-12-28 19:36:01 -07:00
Roland
93f0e7494f
Creating Event and Flag systems. Events are used to send and display a toast to the App. Flags are used to signal if some action has been performed, or if a package is installed. 2022-12-23 16:04:09 -07:00
Roland
d0d69519cd
Bumping version for new feature.
Skipping video epoch test on circleci.

Created script which finds and cleans up (renames/deletes) orphaned video meta files.

Reworking orphaned video query, some video files were leaking through.

Modifying File table to allow finding of orphaned video files.

Extracting epoch from video info_json for more accurate upload_date.

Extracting epoch from video info_json for more accurate upload_date.
2022-12-07 18:15:28 -07:00
Roland
6a67883354
Remuxing video using yt-dlp to handle when output format is wrong. Checking for remuxed destination video file and using it. 2022-11-23 17:30:25 -07:00