Commit graph

377 commits

Author SHA1 Message Date
Roland
78218d3559
Decreasing some log warnings to debugs. Decreasing CPU temperature warning color. 2022-10-16 10:33:43 -06:00
Roland
ddcc2c0674
Increasing test coverage. Removing unused code. 2022-10-15 17:41:10 -06:00
Roland
acf4453a52
Increasing test coverage. 2022-10-15 10:37:49 -06:00
Roland
e7fe65258c
Using divmod to calculate date parts. Using recommended datetime.now. 2022-10-10 08:51:46 -06:00
Roland
bc7ebd4090
Indexing Archives after downloading. 2022-10-09 13:47:49 -06:00
Roland
dad00ad065
Using itertools.cycle to commit occasionally. 2022-10-09 13:12:59 -06:00
Roland
9a47135c99
Using asyncio subprocess for all archive service calls. Returning more informative errors when archive service fails. 2022-10-04 13:15:45 -06:00
Roland
e80f24f690
Moving periodic download logic into manager so the manager can call it when it is enabled. 2022-09-25 17:27:48 -06:00
Roland
00acd9d891
Multiple videos can be deleted using Videos list view. 2022-09-23 16:43:45 -06:00
Roland
69e899e032
Multiple archives can be deleted using Archive list view. 2022-09-23 16:22:57 -06:00
Roland
5cc15c9328
Restoring channel.match_regex functionality. 2022-09-23 15:32:28 -06:00
dependabot[bot]
d4ed5fa5ce Fixing JS warnings.
Handling all combinations of toggle in light and dark.

Changing CPUUsageProgress colors to be more readable in dark mode.

Moving theme handling into Theme.js.  Importing element wrappers from Theme.js to handle element theming.

Changing video menu to dark mode.

Creating ThemeContext to share Dark Mode setting.

Removing duplicate code for search functions.

Displaying X when downloads are disabled.

Using react context to share settings across components.

Refusing to start workers when WROL Mode is enabled.

Changing warning to debug.

Queuing recurring downloads first.

Adding Video.url index.

Adding special handling for youtube shorts to videos.

Reworking Downloads to have one worker per process.

Testing creating downloads and getting downloaders.

Testing video validation and poster generation.  Testing video statistics.

Increasing inventory test coverage.

Adding pytest.ini to automatically detect async test.

Adding more archive test coverage.

Testing Archive CRUD.

Catching and logging when encountering error during indexing.

Using FileIcon when Archive has no screenshot.

A Video will be assigned to a Channel by its info json first, then to its directory.

Including suffix in File json.

Adding index for File.suffix.

Gathering suffix during indexing.

Passing entire file down to FileIcon.

Moving logging up.

Returning File.associated in file json.

Associated files are last in file search.

Removing duplicate refreshing code by creating new _refresh_directory function.

Changing File.idempotency to a datetime to avoid having to clear idempotency.

Truncating indexes to avoid error with large files.

Don't overwrite video data when info json is empty.

Using already_downloaded for channel videos.

Awaiting refresh in Channel.refresh.

Testing that videos already downloaded aren't downloaded again.

Adding indexes for video/archive referenced paths.

Changing grid width for mobile or computer.

Adding button to refresh specific directories.

Restoring Channel edit link.

Fixing caption path in video_modeler.

Adding ZIP files to mimetype colors.

Using model instead of first file's model.

Fetching File models in batches to avoid issuing a SELECT for every File with a model.

Flushing generated poster file.

Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.

Attaching Video to it's Channel during download.

Using ExternalCardLink for ArchivePage singlefile link.

Validating and indexing videos after download.

Handling glob when file name contains matching characters.

Reworking and testing split_path_stem_and_suffix and glob_shared_stem.

Adding associated video files to DB during upsert.

Adding pending downloads to dashboard.

Hiding status when download is open.

Handling when load is empty.

Adding load statistics to Dashboard.

Limiting concurrent after_refresh functions.

Handling when no onSelect is declared.

Adding ImageRow for FileTable.

Moving image date into Card.Meta.

Using ExternalLink for FileCard.

Using ExternalLink for ImageCard.

Handling when error occurs during refresh of a directory.  Committing during refresh to avoid losing all files if an error occurs.

Removing api health check to avoid spamming logs.

Creating ImageCard to display image posters when they aren't too large.

Removing archive.domain_id not null.

Reworking file refresh to process all files in a single directory.

Bumping version for major file rework.

No longer checking for video corruption.

Adding duration by default to speed up tests.

Using register_indexer for all indexers.

Adding CPU usage progress bar in Welcome.

Modifying TextIndexer to ignore large text files.

Counting all files during refresh.

Adding refresh_logger to clarify logging.  Checking for wrol_mode before refresh functions.

Removing unused idempotency from video/channel tables.

Using magic module to speed up mimetype identification.

All tests pass.

Archive tests pass.

Changing video get endpoint to return 'file' to avoid ambiguity.

Video tests pass.

Simplifying regex filtering.

Using regex to ignore case when filtering.

Converting all channel classes to functions.

Restoring channel refresh and download buttons.

Matching format of Archive and Video pages.

Adding sorting to Archives search.

Reworking video orders for file search.  Using SearchInput for videos search.

Archives can be searched on the Archives page.  Domains can be filtered.

Reworking Archive search to use Files.  Videos, Archives and Global search use the FilesView.

Videos are claimed by their Channel based on the Channel's directory.  Prev and Next video cards are displayed on the VideoPage.

Creating useVideo for reworked VideoWrapper and VideoPage.

Adding Paginator to FilesView.

Moving file refresh button out of manage tab.  Removing Files page tabs.  Limiting refresh to one at a time.

Adding class for card icon to match poster height.

Video and Archives always re-claim their files on refresh.

Archive files are indexed and claimed.

FileView table is compact and striped.

Videos page uses FileView.

Updated navlinks to router v6.

Removing ! character from file names.

Disabling select button when cards are visible.

Adding show/hide button for FileTable selection.

The Archive Page is displayed.

Adding mimetype dropdown.

Upgrading to React Router v6 for useSearchParams.

Validating video during modeling.  Removing screenshot during corruption check because it was very slow.  Updating app to use video files as search results.

Updating video search to return Files.

Video and WROLPi tests pass.

Creating function to refresh files within a channel's directory.

Removing troubleshooting print.  Updating result order in test.

Reworking video_factory to handle a specific video file location.

Returning video model on file search.

Getting session using video_file.

Screenshotting when checking for video corruption.

Reducing ffmpeg calls by gathering duration during poster generation.

Moving video file matching into model.

VideoIndexer is moved into video module.  Video files are validated.

Moving files module into wrolpi in preparation to use it in modules.

Title and contents are indexed for text file.

Files are indexed on refresh by using their indexer.

Zip files are indexed using their contents.

Testing multiple zip files.

Adding unique to file.path.  Checking that the correct indexer is used.

Adding index to file.modification_datetime and file.size.

Handling file info gathering in File.do_index.  Only reading contents of file if file.indexed is False.

Video model is found during file refresh.  A Video record is created that has the associated files.

Videos without associated files are found.

Videos are found using registered video_modeler() rather than in refresh code.

Removing old video columns.

Handling video info json files.

Trying to import each module individually.  Will continue to import modules even if one fails.

Reworking refresh so it performs less DB calls.

Converting file.path to primary key.

file.path is always a pathlib.Path.

Updating video_factory to use new file system, testing it.

Bump sanic from 21.9.1 to 21.12.2 in /docker/archive

Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-09-23 11:25:55 -06:00
Roland
3bfbaa4c22
Fixing bug where new downloads may not refresh without a next_download. 2022-07-23 16:12:28 -06:00
Roland
b6b04f2739
Removing unused code. Updating comments.
Removing timeout.  Updating tests now that downloads work.

Bumping version for async downloads.

Downloads run asynchronously.  Using aiohttp to make archive http request so ArchiveDownloader.do_download can be asynchronous.

Removing session for create_download(s) usages.

Reworkingk create_downloads to use optional_session.

Canceling download workers after tests and when stopping the manager.

All tests passing.

Tests pass!

Downloader tests pass.

DownloadManager uses multiple workers to download domains in parallel.
2022-07-22 16:35:42 -06:00
Roland
4cc7ef907f
Ignoring live chat when downloading videos. 2022-07-06 17:44:37 -06:00
Roland
59b9f3f81e
Using which() to locate ffmpeg. 2022-06-29 17:32:28 -06:00
Roland
862b12019b
Ignoring dot directories when importing modules. Raising error when ffprobe is not installed. 2022-06-29 17:27:43 -06:00
Roland
f3b61f8c18
Modifying OTP pad to match printer page. 2022-06-29 11:25:07 -06:00
Roland
e774f81869
Removing duplicate try/excepts in ChannelDownloader. 2022-06-28 12:49:11 -06:00
Roland
c772b57ddb
Returning all urls from a playlist, unless they're already downloaded. Changing all which() calls to contain BIN for clarity. 2022-06-28 10:09:35 -06:00
Roland
f639e01951
Removing Latex as dependency. Using browser to convert OTP pads into printable documents. 2022-06-26 16:27:56 -06:00
Roland
776ac7edb0
Removing python-magic requirement in favor of file command. 2022-06-26 16:16:45 -06:00
Roland
fa83728f52
Including process logs in video result. Upgrading yt-dlp. 2022-06-26 15:45:45 -06:00
Roland
d617c88343
Increasing file size column to bigint to allow for larger files. 2022-06-22 17:36:49 -06:00
Roland
3efd47d004
Global timeout can be set in the config and UI.
Downloaders can specify a process timeout.
2022-06-14 20:45:34 -06:00
Roland
ff379185fe
Adding config fixture to download manager fixture to avoid calling both in every test. 2022-06-08 12:21:59 -06:00
Roland
f14a230bc6
RSSDownloader can schedule downloads for other Downloaders using sub_downloader.
RSS download can be submitted in UI.

Checking if entry has link.

A video playlist can be downloaded.

Passing sub_downloader when submitting RSS feed.

Adding Archive downloader button.

Playlists need more info.

Removing expunge, it causes errors elsewhere.  Fixing tests.

Showing errors on recurring downloads.  Ignoring CharacterEncodingOverride bozo error for rss feed.

Changing download button order.

Fixing bug where channel attributes were unavailable after session.  Simplifying video format selection.

Calling validate_video in Video.validate() to avoid duplicate code.

Adding more frequent downloads for RSS Feeds.

DownloadManager skips downloads in it's list.

Renaming get_next_download to more clear calculate_next_download.

Stopping download manager when server is shutting down.

Not shutting down downloads during tests, not all tests use downloads.

Making methods static.
2022-06-08 11:56:42 -06:00
Roland
f2d57fcb23
Reformatting. 2022-05-19 21:38:55 -06:00
Roland
8f62cb895f
Don't process channel downloads. 2022-05-18 19:58:21 -06:00
Roland
c681c500e8
Checking for corruption in bytes to avoid unicode error. 2022-05-18 18:36:12 -06:00
Roland
dd07236834
Checking for video corruption. Processing video when preparing filname to avoid corrupt videos. 2022-05-18 17:24:50 -06:00
lrnselfreliance
0765890b8b
Feature/127 archive UI rework (#131)
* Archives have their own page.  Delete/etc buttons are moved into ArchivePage.

* Alternatives for an Archive are displayed in the ArchivePage.

* ArchiveDownloader returns the location.

* Removing color from Details button.

* Making Archive screenshot and title a link.

* Preserving next_download when updating a Channel.

* Removing button icons from Archive page.
2022-05-16 17:57:18 -06:00
Roland
f9dd056cc9
Fixing error where all Videos of a Channel would be downloaded because upsert_video() did not update the correct record. 2022-05-10 22:44:59 -06:00
Roland
785ad8274c
DownloadResult.try_again is not necessary.
Removing unused fixture in test.

Simplifying logic after download completes.  Removing unused download sorting method.

ChannelDownloader returns location.

Hiding View link when no location is provided.

Performing video upsert in try/except so any errors are attached to the Download.

Including stderr only.

Including logs when returning Downloader.process_runner.

Warning when extention is replaced.

Providing location link on Downloads page.  Showing error modal on errors that fail.

Handling empty URL strings on import.

Fixing channel download test.

Reworking how Channels schedule Video downloads.

Bumping version for DownloadResult.

Updating tests to use DownloadResult.

VideoDownloader returns a DownloadResult.
2022-05-10 21:20:36 -06:00
Roland
4058f22428
Warning when validation fails on video upsert. 2022-04-29 19:30:53 -06:00
Roland
8318224db6
Removing troubleshooting print. 2022-04-29 19:19:12 -06:00
Roland
d75ddf097d
Removing unused function. 2022-04-29 19:09:49 -06:00
Roland
aa203cc102
Moving non-video functions to wrolpi.common. Handling Channel/Downlaod relationship in Channel.update.
Removing default download freqeuncy so channels will not be downloaded without a url and frequency.

Adding channel unique url constraint.

Handling when a Channel's directory may still be relative in DB.

Adding TODOs to remove old favorites.

Removing outdated favorites from config after favorites have been created.

Testing that old-style favorited videos in the config are imported.

Removing channels_by_link.

Using list to store channels in config.

Moving channel import function into lib.

Supporting old channel import with links.

Bumping version for channel.link removal.

Removing all reference to Channel.link in app and api.

Adding relative property to MediaPathType.

Testing that a channel can be downloaded after it has a frequency added.

All tests pass.

Video and Channel tests pass without channel.link.

Removing channel.link usage in channel api.

Changing video Statistics page to About.
2022-04-29 19:09:25 -06:00
lrnselfreliance
fda8636c92
Fixing bug in archives search where offset could not be greater than 100. (#117) 2022-04-14 09:21:26 -06:00
Roland
372056eefd
Removing troubleshooting print. 2022-04-08 09:34:18 -06:00
Roland
7656890835
Testing that video size is gathered during validation. 2022-04-07 13:42:01 -06:00
lrnselfreliance
35f5662571
Bugfix/channel download on import (#116)
* A Download record is created for every Channel imported.

* Downloads must have a URL.

* Handling when channel has download frequency but no url.

* Fetching downloads after clearing.

* Setting next download when importing channels.

* Not deleting recurring completed downloads.

* Fixing comments.

* Skipping some service tests in CircleCI because they shouldn't be tested off a local machine.

Signed-off-by: Roland <roland@learningselfreliance.com>
2022-04-07 11:18:15 -06:00
Roland
43709ac9e7
Creatind download manager config which will store skipped urls. 2022-04-02 16:01:43 -06:00
Roland
5178f72b66
Reworking map import to handle postgres data dumps. 2022-03-31 07:31:15 -06:00
Roland
33fa1444c6
Allowing multiple names for which function. 2022-03-19 17:08:44 -06:00
Roland
a02d1c4527
Using which for chromium-browser. 2022-03-19 16:58:49 -06:00
Roland
5c3c400dd1
Creating cmd.which which will find executables. 2022-03-19 15:18:40 -06:00
Roland
849a3fd43f
Creating debian install scripts.
Modifying Ubuntu 20.04 install script to work with Debian 11.

Should not need to compile mod_tile.

Uncommenting branch checkout.

Not removing nodejs.

Not installing nodejs.

Installing nodejs instead of npm??

Using debian 11 npm sources.

Installing archive tools using git+https

Updating carto location.

Using global renderd.conf in apache config.

Using -I to get debian ip.

Updating more file locations for debian 11.

Checking alternative locations for single-file and readability-extractor
2022-03-16 21:10:49 -06:00
Roland
c815aa68c6
Calculating time-to-import for map pbf files in the backend. These times were created using a test RPi4. Encoding URL of archive files to prevent error which blocked displaying of archives. 2022-03-12 09:50:16 -07:00
Roland
6a22090e54
Logging nicer timestamp for logging import elapsed time. 2022-03-10 13:17:12 -07:00