Commit graph

152 commits

Author SHA1 Message Date
Roland
aa15f5db80
Updating Channels UI table to use iso datetime. 2022-12-29 09:58:29 -07:00
Roland
9dfaabdb9b
Not sending refresh events for channel downloads. 2022-12-29 09:36:24 -07:00
Roland
0152b8648e
Attaching Channel's info_json even if the download is a playlist. 2022-12-28 19:36:01 -07:00
Roland
95c8610df2
File.d_text is reformatted to break up long lines. 2022-12-27 15:41:55 -07:00
Roland
35145d0335
Searching for censored videos when refreshing a channel. 2022-12-26 11:30:28 -07:00
Roland
93f0e7494f
Creating Event and Flag systems. Events are used to send and display a toast to the App. Flags are used to signal if some action has been performed, or if a package is installed. 2022-12-23 16:04:09 -07:00
Roland
875f6f2606
Creating ebook details modal for ebook card. Linking to epub reader when possible in ebook card. 2022-12-19 10:40:01 -07:00
Roland
a322d1a0a9
Removing unnecessary flush. 2022-12-16 09:21:22 -07:00
Roland
3751ed7999
Reformatting files. 2022-12-14 19:11:11 -07:00
Roland
a149cc8f69
Reworking modelers to only fetch matching models once to avoid making many queries per group. 2022-12-14 18:52:42 -07:00
Roland
cdc02ecaf2
Mocking apply_indexers because it is unecessary for this test. 2022-12-14 13:16:42 -07:00
Roland
9b048af7fd
Removing unused code. Matching use of style for file name parsing. 2022-12-14 10:33:11 -07:00
Roland
9ffe6ee3f6
Info JSON epoch is time of download, not time of upload. 2022-12-08 13:45:46 -07:00
Roland
d0d69519cd
Bumping version for new feature.
Skipping video epoch test on circleci.

Created script which finds and cleans up (renames/deletes) orphaned video meta files.

Reworking orphaned video query, some video files were leaking through.

Modifying File table to allow finding of orphaned video files.

Extracting epoch from video info_json for more accurate upload_date.

Extracting epoch from video info_json for more accurate upload_date.
2022-12-07 18:15:28 -07:00
Roland
8360e4fb82
Re-indexing video if its files change. 2022-11-30 18:36:22 -07:00
Roland
20860fdae8
Detecting more video caption files, indexing them. Only WebVTT captions are displayed on VideoPage. Creating Files tab on VideoPage which displays all files found for the video. 2022-11-30 18:26:38 -07:00
Roland
6a67883354
Remuxing video using yt-dlp to handle when output format is wrong. Checking for remuxed destination video file and using it. 2022-11-23 17:30:25 -07:00
Roland
c2d25f7717
Only generating video poster if no path or file exists. 2022-11-16 16:52:49 -07:00
Roland
053bc7eaeb
Updating test when videos are deleted. 2022-11-16 12:50:20 -07:00
Roland
003304046c
Commiting final deletion of video. 2022-11-15 13:03:26 -07:00
Roland
02124307e4
Downloads can be deleted. 2022-11-12 14:17:21 -07:00
Roland
cc80929723
Clearing out bad json file if it cannot be found. 2022-11-12 13:43:18 -07:00
Roland
2c6fdc42ad
Channel may not have a directory while testing. 2022-11-10 08:36:36 -07:00
Roland
83bd19d91f
Returned Video.censored when fetching video. 2022-11-10 08:21:54 -07:00
Roland
a0ad941425
Getting all channel properties during session. 2022-11-09 19:41:29 -07:00
Roland
be852e3d46
Fixing bug where empty destination would be coerced into 'None'. 2022-11-09 19:19:06 -07:00
Roland
ddb35448aa
Finding channel by destination directory if any. 2022-11-09 08:54:56 -07:00
Roland
f1bc29d76f
Adding feature where Videos can be downloaded to a destination directory. 2022-11-09 08:33:55 -07:00
Roland
1e1c8f0b85
ChannelDownloader will add itself to Download.Settings, so VideoDownloader can download to the channel if yt-dlp cannot find the channel. 2022-11-08 22:41:50 -07:00
Roland
456a83b8f4
Fixing when path stem's would sometimes not parse. 2022-11-05 19:31:30 -06:00
Roland
6c2a5d723e
Downloads are saved to the download manager config. They are imported on startup. 2022-11-04 19:59:42 -06:00
Roland
3d15ba2091
Removing unused function. 2022-11-03 20:27:59 -06:00
Roland
d575b087b3
Associating Video files when downloaded. 2022-10-25 08:15:35 -06:00
Roland
76a9c0d88c
Creating Video/Archive my_files to avoid checking all files. 2022-10-24 17:50:25 -06:00
Roland
e504e595c0
Deleting Video/Archive File records when deleting them. 2022-10-24 17:37:40 -06:00
Roland
797bdc431c
Creating invert/clear buttons for SelectableTable. 2022-10-24 17:15:15 -06:00
Roland
b81e00ff3d
Perserving ts_rank when searching. 2022-10-24 15:34:13 -06:00
Roland
6fa6be50fe
Stopping renderd when reseting map db.
Skipping channel import during test unless explicitly requested.

Deleting gis database during upgrade, if user agrees.  Increasing API verbosity.

Using status context to prevent fetching status in useHotspot.

Disabling wifi toggle button when pending.

Toggling hotspot when clicking nav hotspot icon.

Displaying Hotspot status icon in nav menu.

Importing channels/favorites after refreshing.

Refresh tasks cancel when server shuts down.

Bumping version for release speed.

Refreshing files is 3x faster.

Creating useRecurringTimeout hook which repeatedly calls a callback after a timeout.

Changing intervals to timeout so fetches don't stack up while server is loaded.

Checking that video file exists before fetching.

Creating release candidate branch.  Not indexing model associated files.
2022-10-21 15:21:57 -06:00
Roland
ddcc2c0674
Increasing test coverage. Removing unused code. 2022-10-15 17:41:10 -06:00
Roland
dad00ad065
Using itertools.cycle to commit occasionally. 2022-10-09 13:12:59 -06:00
Roland
00acd9d891
Multiple videos can be deleted using Videos list view. 2022-09-23 16:43:45 -06:00
Roland
5cc15c9328
Restoring channel.match_regex functionality. 2022-09-23 15:32:28 -06:00
dependabot[bot]
d4ed5fa5ce Fixing JS warnings.
Handling all combinations of toggle in light and dark.

Changing CPUUsageProgress colors to be more readable in dark mode.

Moving theme handling into Theme.js.  Importing element wrappers from Theme.js to handle element theming.

Changing video menu to dark mode.

Creating ThemeContext to share Dark Mode setting.

Removing duplicate code for search functions.

Displaying X when downloads are disabled.

Using react context to share settings across components.

Refusing to start workers when WROL Mode is enabled.

Changing warning to debug.

Queuing recurring downloads first.

Adding Video.url index.

Adding special handling for youtube shorts to videos.

Reworking Downloads to have one worker per process.

Testing creating downloads and getting downloaders.

Testing video validation and poster generation.  Testing video statistics.

Increasing inventory test coverage.

Adding pytest.ini to automatically detect async test.

Adding more archive test coverage.

Testing Archive CRUD.

Catching and logging when encountering error during indexing.

Using FileIcon when Archive has no screenshot.

A Video will be assigned to a Channel by its info json first, then to its directory.

Including suffix in File json.

Adding index for File.suffix.

Gathering suffix during indexing.

Passing entire file down to FileIcon.

Moving logging up.

Returning File.associated in file json.

Associated files are last in file search.

Removing duplicate refreshing code by creating new _refresh_directory function.

Changing File.idempotency to a datetime to avoid having to clear idempotency.

Truncating indexes to avoid error with large files.

Don't overwrite video data when info json is empty.

Using already_downloaded for channel videos.

Awaiting refresh in Channel.refresh.

Testing that videos already downloaded aren't downloaded again.

Adding indexes for video/archive referenced paths.

Changing grid width for mobile or computer.

Adding button to refresh specific directories.

Restoring Channel edit link.

Fixing caption path in video_modeler.

Adding ZIP files to mimetype colors.

Using model instead of first file's model.

Fetching File models in batches to avoid issuing a SELECT for every File with a model.

Flushing generated poster file.

Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.

Attaching Video to it's Channel during download.

Using ExternalCardLink for ArchivePage singlefile link.

Validating and indexing videos after download.

Handling glob when file name contains matching characters.

Reworking and testing split_path_stem_and_suffix and glob_shared_stem.

Adding associated video files to DB during upsert.

Adding pending downloads to dashboard.

Hiding status when download is open.

Handling when load is empty.

Adding load statistics to Dashboard.

Limiting concurrent after_refresh functions.

Handling when no onSelect is declared.

Adding ImageRow for FileTable.

Moving image date into Card.Meta.

Using ExternalLink for FileCard.

Using ExternalLink for ImageCard.

Handling when error occurs during refresh of a directory.  Committing during refresh to avoid losing all files if an error occurs.

Removing api health check to avoid spamming logs.

Creating ImageCard to display image posters when they aren't too large.

Removing archive.domain_id not null.

Reworking file refresh to process all files in a single directory.

Bumping version for major file rework.

No longer checking for video corruption.

Adding duration by default to speed up tests.

Using register_indexer for all indexers.

Adding CPU usage progress bar in Welcome.

Modifying TextIndexer to ignore large text files.

Counting all files during refresh.

Adding refresh_logger to clarify logging.  Checking for wrol_mode before refresh functions.

Removing unused idempotency from video/channel tables.

Using magic module to speed up mimetype identification.

All tests pass.

Archive tests pass.

Changing video get endpoint to return 'file' to avoid ambiguity.

Video tests pass.

Simplifying regex filtering.

Using regex to ignore case when filtering.

Converting all channel classes to functions.

Restoring channel refresh and download buttons.

Matching format of Archive and Video pages.

Adding sorting to Archives search.

Reworking video orders for file search.  Using SearchInput for videos search.

Archives can be searched on the Archives page.  Domains can be filtered.

Reworking Archive search to use Files.  Videos, Archives and Global search use the FilesView.

Videos are claimed by their Channel based on the Channel's directory.  Prev and Next video cards are displayed on the VideoPage.

Creating useVideo for reworked VideoWrapper and VideoPage.

Adding Paginator to FilesView.

Moving file refresh button out of manage tab.  Removing Files page tabs.  Limiting refresh to one at a time.

Adding class for card icon to match poster height.

Video and Archives always re-claim their files on refresh.

Archive files are indexed and claimed.

FileView table is compact and striped.

Videos page uses FileView.

Updated navlinks to router v6.

Removing ! character from file names.

Disabling select button when cards are visible.

Adding show/hide button for FileTable selection.

The Archive Page is displayed.

Adding mimetype dropdown.

Upgrading to React Router v6 for useSearchParams.

Validating video during modeling.  Removing screenshot during corruption check because it was very slow.  Updating app to use video files as search results.

Updating video search to return Files.

Video and WROLPi tests pass.

Creating function to refresh files within a channel's directory.

Removing troubleshooting print.  Updating result order in test.

Reworking video_factory to handle a specific video file location.

Returning video model on file search.

Getting session using video_file.

Screenshotting when checking for video corruption.

Reducing ffmpeg calls by gathering duration during poster generation.

Moving video file matching into model.

VideoIndexer is moved into video module.  Video files are validated.

Moving files module into wrolpi in preparation to use it in modules.

Title and contents are indexed for text file.

Files are indexed on refresh by using their indexer.

Zip files are indexed using their contents.

Testing multiple zip files.

Adding unique to file.path.  Checking that the correct indexer is used.

Adding index to file.modification_datetime and file.size.

Handling file info gathering in File.do_index.  Only reading contents of file if file.indexed is False.

Video model is found during file refresh.  A Video record is created that has the associated files.

Videos without associated files are found.

Videos are found using registered video_modeler() rather than in refresh code.

Removing old video columns.

Handling video info json files.

Trying to import each module individually.  Will continue to import modules even if one fails.

Reworking refresh so it performs less DB calls.

Converting file.path to primary key.

file.path is always a pathlib.Path.

Updating video_factory to use new file system, testing it.

Bump sanic from 21.9.1 to 21.12.2 in /docker/archive

Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-09-23 11:25:55 -06:00
Roland
3bfbaa4c22
Fixing bug where new downloads may not refresh without a next_download. 2022-07-23 16:12:28 -06:00
Roland
b6b04f2739
Removing unused code. Updating comments.
Removing timeout.  Updating tests now that downloads work.

Bumping version for async downloads.

Downloads run asynchronously.  Using aiohttp to make archive http request so ArchiveDownloader.do_download can be asynchronous.

Removing session for create_download(s) usages.

Reworkingk create_downloads to use optional_session.

Canceling download workers after tests and when stopping the manager.

All tests passing.

Tests pass!

Downloader tests pass.

DownloadManager uses multiple workers to download domains in parallel.
2022-07-22 16:35:42 -06:00
Roland
4cc7ef907f
Ignoring live chat when downloading videos. 2022-07-06 17:44:37 -06:00
Roland
59b9f3f81e
Using which() to locate ffmpeg. 2022-06-29 17:32:28 -06:00
Roland
862b12019b
Ignoring dot directories when importing modules. Raising error when ffprobe is not installed. 2022-06-29 17:27:43 -06:00
Roland
e774f81869
Removing duplicate try/excepts in ChannelDownloader. 2022-06-28 12:49:11 -06:00
Roland
c772b57ddb
Returning all urls from a playlist, unless they're already downloaded. Changing all which() calls to contain BIN for clarity. 2022-06-28 10:09:35 -06:00