Commit graph

93 commits

Author SHA1 Message Date
Roland
7d6b7114a7
Adding TRACE level for logging beyond debug. Reworking async_client for tests, not all tests need the App to handle signals, so any test that needs signals can use the new await_switches fixture which primes the app. 2024-12-11 11:52:33 -07:00
Roland
fe521c99f3
Adding edit form for Scrape downloader. 2024-12-03 12:44:38 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
11dc370ed9
Channel can be created with Tag. 2024-10-16 18:57:01 -06:00
Roland
77c94da3a3
Removing channel mkdir, it will always make the channel. 2024-10-16 18:56:59 -06:00
Roland
584fdadcb7
Getting file name from yt-dlp error when video file name is too long, trimming that long file name. 2024-10-16 18:56:56 -06:00
Roland
d3e5b56520
Bumping python requirements versions. 2024-10-16 18:56:41 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
d5ca3e7f02
Including Country/Region codes in caption file names. 2024-10-16 18:56:31 -06:00
Roland
317f3f3dfb
Top processes are displayed on Status page. Nice of renderd is lowered to avoid clobbering system at startup. 2024-10-16 18:56:27 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
6fe8a7b704
Refusing to rename Tag or move Channel when files are refreshing. 2024-10-16 18:56:24 -06:00
Roland
50d133151d
Refusing to change Channel Downloads or tag Channel when wrol_mode is enabled. 2024-10-16 18:56:21 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
3906223844
Moving Channel attachment logic to recurring_download method because only recurring downloads can have Channels. 2024-10-16 18:56:13 -06:00
Roland
a98aebfdda
A video that is uploaded is attached to its Channel. Saving Channels config with both download url and frequency in case downloads config is corrupted. 2024-10-16 18:56:11 -06:00
Roland
63738bd70f
Removing ChannelDownload in favor of Download.channel_id. 2024-10-16 18:55:54 -06:00
Roland
287beabb9c
Creating missing ChannelDownload when importing channels config. 2024-10-16 18:55:49 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
ae61977f0b
Reformatting. 2024-06-12 16:47:31 -06:00
Roland
8a1b75855d
Testing search_channels_by_name. 2024-05-25 11:15:24 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
2c127a03f5
Testing channel suggestions being ordered by video count. 2024-04-16 15:48:09 -06:00
Roland
7669cfe361
Ordering suggested channel names by video count. 2024-04-16 10:04:06 -06:00
Roland
947135985d
Reformatting code. 2024-03-12 17:20:51 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
85a1360a25
Suggesting channels that names match, even without spaces. 2024-02-19 14:55:53 -07:00
Roland
1b92d1e580
Global search uses Semantic SearchResultsInput component, other components use SearchInput. 2023-11-21 13:01:52 -07:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
b817534c37
Adding Download.is_new/etc. for code suggestions. Fixing async test warnings. 2023-09-20 12:58:35 -06:00
Roland
f14df90174
Removing unused imports. 2023-09-12 17:49:26 -06:00
Roland
da6d6a8094
Allowing starting a download of a channel without frequency. Doing this creates a once-download. 2023-08-30 14:01:58 -06:00
Roland
14017d43bb
Creating Video/Archive get_by_id/find_by_id methods. find_by_id will raise an exception when not found, get_by_id will not. 2023-08-17 18:46:42 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
6458316d1a
Removing unused /videos/tag. Sending events when Channels are created or deleted. 2023-07-17 17:32:29 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
ce34721c4c
Videos are disowned when their Channel is deleted. A Channel directory cannot be in another Channel's directory. 2023-07-14 13:56:23 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
7c5caf1dfd
Summarizing Channel size on Channels page, rather than info age. 2023-05-10 22:59:16 -06:00
Roland
e06f846c8c
Updating function names to match their functionality. 2023-05-03 13:05:57 -06:00
Roland
2cbc6e00e0
Handling situation where React Router may not be available for a Tag Label. 2023-05-03 12:46:57 -06:00
Roland
387282b445
Adding my public gpg key.
Moving upload destination input out of drop Segment.

Disabling uploads when downloads are disabled.

Inverting labels for checkboxes in inverted modal.

Deleting file_groups which no longer contain the exclusive primary path.

Adding debug logging for all API errors.

Checking that upload output is not a directory.

Expanding nginx request size.

Adding docs to upload endpoint.

Using dot-circle icon for more image mimetypes.

Using Form.Field to make upload destination required.

Adding border to inverted Segments.

Many files can be uploaded in chunks.

Files can be uploaded, progress is displayed.

Showing white border around share QR code.

Displaying WROLPi colors for download icons.

Creating button to clear DirectorySearch.

Excluding channel/domain directories so other directories can be returned.

Ordering search results.

Ignoring when no channel is found only if we have a destination.

Limiting directory search results.

Displaying categories when searching destination directories.

Handling when live_chat.json file exists.

Allowing refresh of directory directly.

Logging which download was successful.

Removing timeout.

Creating index for directory names.

Sleeping to catch cancel.

Arbitrary files can be downloaded using the new FileDownloader.
2023-04-28 13:56:32 -06:00
Roland
bab10a0c96
Creating sort button which allows the user to select a sort order via modal. 2023-04-09 14:13:36 -06:00
Roland
75fb9c4351
A download will be created when a Channel is created. 2023-04-02 11:47:16 -06:00
Roland
55cf76a884
Requiring downloader name to be specified. 2023-04-01 15:34:46 -06:00
Roland
5615ce542d
Making example label a tag.
Tags can be created and deleted in UI.

Adding tag_file using id to avoid session error.

Using UTC in test.

Matching tag_names style in test.

Tag icon is displayed on file rows.

Archives, Videos and Files can be filtered by their tags.

Tags can be added and removed from UI.

Tags are displayed in VideoPlayer.

The tags of a file are returned in it's JSON.

Video favorites are converted to Favorite tag on channel config import.

Updating tests to match frontend usage.

PDF creator and modification time are displayed.

Reworking file search to use Video.__json__ for complex video relations.

Archives can be viewed.

Removing ebook.title.

Reworking alembic migration to be an upgrade rather than a replace.

Skipping test on circleci.

Clearing out video caption index only if extract_subtitles if off.

Channel tests pass.

Some video tests pass.

Missing videos can be found.  A video can be downloaded.

Orphaned files can be found.

Videos can be tagged.

Videos can be searched.

Videos can be modeled.

Converting some tests to use FileGroup.primary_path.

Adding archive and domains to alembic migration.

Testing cascading Archive titles.

Archives now uses FileGroup.

archive.lib tests pass.

Archive is partially converted to FileGroup.

Calculating FileGroup.primary file on refresh, rather than on select.

wrolpi tests pass.

Using FilesType to convert between str/pathlib.Path for FileGroup.files.

Using list of files in FileGroup.files instead of dictionary.

Storing files in dedicated FileGroup.files to avoid overwriting them.

Adding comment explaining CASE.

Not overwriting FileGroup.data on refresh if no change has been detected.

Removing File model in favor of FileGroup.data.

Adding typing to Files methods.

Files are indexed using the primary mimetype of the FileGroup.

files test_lib tests pass.

Removing file_group_file in favor of simple one-to-many relationship.

PDF files are indexed during refresh.

Adding Tag and TagFile to tag files with specific tags.

Only iterating when full_stem_ids has contents.

Testing that FileGroups are related to their Files.

Files and file_groups are created and linked.

Bump webpack from 5.75.0 to 5.76.1 in /app (#160)

Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-25 17:54:41 -06:00
Roland
942a35e5d8
Resolving Channel download entries. Detecting and raising error when user has not specified exactly which video URL they want to download. 2023-02-21 15:12:45 -07:00
Roland
81d80238be
Fixing changes before merge.
Moving FilesRefreshButton and DirectoryRefreshButton out of browser file.

Moving file fetching tino FileBrowser.

Removing sub-folders from Files display when any parent folder is clicked.

Using closed captioning for vtt/srt file icon.

Using my own react-semantic-toasts-2 to use toasts in Node 18.

Fixing bug where more than the last page was displayed.

Upgrading yt-dlp.

Fixing sub-folder file selection.

Adding rowKey to SortableTable to give React a key for each row.

Renaming key to path.

Removing duplicate urls.

Unifying file modals.

Displaying modals for image and text files.

Handling refresh loading in FileBrowser.  Creating refreshing_directory flag.

Removing accidental commit.

Sorting by url_count.

Using SortableTable on Domains page.

Using SortableTable for Channels page.

Created new FileBrowser which allows browsing of files in the media directory.

Using Node 18 in pi-gen.

Creating new FileBrowser to replace outdated react-keyed-file-browser.

Returning mimetype with file list.

Do not display 'Do not download' on Channel download in Dashboard.

Allowing null channel download frequency so user can stop downloading a channel.

Bumping version for Node 18.

Using my own semantic-ui-offline to avoid pulling Google fonts.

Upgrading app docker container to use Node 18.  This requires the removal of several npm packages.  File browser will require re-write.  It now pulls fonts from Google, this will need to be fixed.

Testing DownloadsToggle click.

Testing trinary states of DownloadsToggle.

Installing @testing-library/user-event for testing App.
2023-02-08 21:18:51 -07:00
Roland
9dfaabdb9b
Not sending refresh events for channel downloads. 2022-12-29 09:36:24 -07:00