Commit graph

78 commits

Author SHA1 Message Date
Roland
287beabb9c
Creating missing ChannelDownload when importing channels config. 2024-10-16 18:55:49 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
0fd545d2e8
Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
Roland
0bee458b79
Sorting keys when saving json file. 2024-06-13 14:08:07 -06:00
Roland
0fbbd36297
Creating perpetual funciton which checks if the internet is accessible. This is used to prevent downloads from starting when no internet exists. 2024-05-31 23:54:00 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
a665677ecc
Adding more lower-quality video formats to download when no other is available. Getting video ffprobe json when downloading. 2024-03-16 19:58:58 -06:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
Displaying star for Uploader's comments, heart for favorited comments.

Adding wrolpi help files desktop shortcut to rpi image.

Using zim file names as FileGroup.title.

Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
df849ffb42
Ordering video/archive results when no published_datetime exists. Adding Video.location property which returns the App location of the video. 2024-02-26 13:40:39 -07:00
Roland
0d6edc05d0
Duplicate videos are deleted when a video is downloaded. 2024-02-21 12:42:30 -07:00
Roland
f865d9fb0f
Videos display comments. User can download comments using Refresh button. 2024-02-20 13:47:12 -07:00
Roland
1b92d1e580
Global search uses Semantic SearchResultsInput component, other components use SearchInput. 2023-11-21 13:01:52 -07:00
Roland
97b214d387
Adding Total Duration to EditChannel page. Handling when searching channel names, but channels have not been fetched yet. 2023-11-02 10:28:00 -06:00
Roland
440b770a11
Fixing Video.get_surrounding_videos after moving video upload date. 2023-10-29 18:44:50 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
76212cf3de
Created script which searches for videos that have similar names, interacts with the user to choose which to delete. 2023-10-09 08:44:30 -06:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
Roland
e7582944b9
Channels not in the channels config are deleted on import. 2023-09-12 20:24:02 -06:00
Roland
3e39102a55
ffprobe json may not be valid, the video should still be retrieved. 2023-09-07 13:01:35 -06:00
Roland
da6d6a8094
Allowing starting a download of a channel without frequency. Doing this creates a once-download. 2023-08-30 14:01:58 -06:00
Roland
6959c7a3b4
HTML and JSON files are reformatted when downloaded. 2023-08-26 21:49:24 -06:00
Roland
9dbd43cfdd
Videos near a video file without and upload date are returned as the prev/next videos. 2023-08-17 19:32:59 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
2821d6d7b8
Removing unused column. 2023-08-15 14:14:37 -06:00
Roland
e8e5b5727f
Creating Video.validated which is used by find_corrupt_videos.py to search. 2023-08-08 13:42:55 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
ce34721c4c
Videos are disowned when their Channel is deleted. A Channel directory cannot be in another Channel's directory. 2023-07-14 13:56:23 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
8de805b6d1
Updating tests to match searching for Channel and Domain names for directories. 2023-05-03 12:59:47 -06:00
Roland
be3e8375a9
Creating FileGroup.merge to be used in FileGroup.from_paths. 2023-05-03 08:28:53 -06:00
Roland
83f9f6f085 Trusting Video.file_group.primary_path instead of checking mimetype each time. 2023-04-28 13:56:21 -06:00
Roland
6fe536558a
You cannot delete a tagged FileGroup. 2023-04-12 11:42:13 -06:00
Roland
75fb9c4351
A download will be created when a Channel is created. 2023-04-02 11:47:16 -06:00
Roland
ae2b6daa33
FileBrowser files can be tagged. A Tag icon is displayed on each file row in the browser. 2023-04-01 11:52:41 -06:00
Roland
5615ce542d
Making example label a tag.
Tags can be created and deleted in UI.

Adding tag_file using id to avoid session error.

Using UTC in test.

Matching tag_names style in test.

Tag icon is displayed on file rows.

Archives, Videos and Files can be filtered by their tags.

Tags can be added and removed from UI.

Tags are displayed in VideoPlayer.

The tags of a file are returned in it's JSON.

Video favorites are converted to Favorite tag on channel config import.

Updating tests to match frontend usage.

PDF creator and modification time are displayed.

Reworking file search to use Video.__json__ for complex video relations.

Archives can be viewed.

Removing ebook.title.

Reworking alembic migration to be an upgrade rather than a replace.

Skipping test on circleci.

Clearing out video caption index only if extract_subtitles if off.

Channel tests pass.

Some video tests pass.

Missing videos can be found.  A video can be downloaded.

Orphaned files can be found.

Videos can be tagged.

Videos can be searched.

Videos can be modeled.

Converting some tests to use FileGroup.primary_path.

Adding archive and domains to alembic migration.

Testing cascading Archive titles.

Archives now uses FileGroup.

archive.lib tests pass.

Archive is partially converted to FileGroup.

Calculating FileGroup.primary file on refresh, rather than on select.

wrolpi tests pass.

Using FilesType to convert between str/pathlib.Path for FileGroup.files.

Using list of files in FileGroup.files instead of dictionary.

Storing files in dedicated FileGroup.files to avoid overwriting them.

Adding comment explaining CASE.

Not overwriting FileGroup.data on refresh if no change has been detected.

Removing File model in favor of FileGroup.data.

Adding typing to Files methods.

Files are indexed using the primary mimetype of the FileGroup.

files test_lib tests pass.

Removing file_group_file in favor of simple one-to-many relationship.

PDF files are indexed during refresh.

Adding Tag and TagFile to tag files with specific tags.

Only iterating when full_stem_ids has contents.

Testing that FileGroups are related to their Files.

Files and file_groups are created and linked.

Bump webpack from 5.75.0 to 5.76.1 in /app (#160)

Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-25 17:54:41 -06:00
Roland
942a35e5d8
Resolving Channel download entries. Detecting and raising error when user has not specified exactly which video URL they want to download. 2023-02-21 15:12:45 -07:00
Roland
aa15f5db80
Updating Channels UI table to use iso datetime. 2022-12-29 09:58:29 -07:00
Roland
9dfaabdb9b
Not sending refresh events for channel downloads. 2022-12-29 09:36:24 -07:00
Roland
0152b8648e
Attaching Channel's info_json even if the download is a playlist. 2022-12-28 19:36:01 -07:00
Roland
35145d0335
Searching for censored videos when refreshing a channel. 2022-12-26 11:30:28 -07:00
Roland
875f6f2606
Creating ebook details modal for ebook card. Linking to epub reader when possible in ebook card. 2022-12-19 10:40:01 -07:00
Roland
20860fdae8
Detecting more video caption files, indexing them. Only WebVTT captions are displayed on VideoPage. Creating Files tab on VideoPage which displays all files found for the video. 2022-11-30 18:26:38 -07:00
Roland
cc80929723
Clearing out bad json file if it cannot be found. 2022-11-12 13:43:18 -07:00
Roland
2c6fdc42ad
Channel may not have a directory while testing. 2022-11-10 08:36:36 -07:00
Roland
83bd19d91f
Returned Video.censored when fetching video. 2022-11-10 08:21:54 -07:00
Roland
d575b087b3
Associating Video files when downloaded. 2022-10-25 08:15:35 -06:00
Roland
76a9c0d88c
Creating Video/Archive my_files to avoid checking all files. 2022-10-24 17:50:25 -06:00
Roland
e504e595c0
Deleting Video/Archive File records when deleting them. 2022-10-24 17:37:40 -06:00
Roland
797bdc431c
Creating invert/clear buttons for SelectableTable. 2022-10-24 17:15:15 -06:00