Python 3.11 is now required, old versions were causing unpredictability in tests. (Sanic does not yet
support 3.12)
Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
https://github.com/sanic-org/sanic/issues/2921
New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes. The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation. Now all multiprocessing tools are attached to the
app.shared_ctx. This means `api_app` is imported in many, many places.
This forced a change in how the DownloadManager works. Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue. Now, a single worker checks
for new downloads and sends a Sanic signal.
Flags have been reworked to use the `api_app`. I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
Parsing SingleFile datetime using dates.strpdate.
Adding timezone when no timezone info is available.
Installing python-dateutil to assist in handling diverse datetime strings.
Reworking refresh progress to use idempotency to display more accurate progress. Only logging that files have been invalidated if they used to be indexed.
Clearing archives to force re-index.
Ignoring 'Unknown' author from PDF metadata. Using file_title when pdf metadata title is empty.
* A new zim module has been added, this module allows several things to interact with Zim files.
* Allow searching of Zim files using libzim.
* Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
* Creation of a new zim docker container which allows viewing of the Zim file contents.
* This commit also adds move/rename/mkdir features in the File Browser.
* File tags are preserved when moving/renaming.
Moving upload destination input out of drop Segment.
Disabling uploads when downloads are disabled.
Inverting labels for checkboxes in inverted modal.
Deleting file_groups which no longer contain the exclusive primary path.
Adding debug logging for all API errors.
Checking that upload output is not a directory.
Expanding nginx request size.
Adding docs to upload endpoint.
Using dot-circle icon for more image mimetypes.
Using Form.Field to make upload destination required.
Adding border to inverted Segments.
Many files can be uploaded in chunks.
Files can be uploaded, progress is displayed.
Showing white border around share QR code.
Displaying WROLPi colors for download icons.
Creating button to clear DirectorySearch.
Excluding channel/domain directories so other directories can be returned.
Ordering search results.
Ignoring when no channel is found only if we have a destination.
Limiting directory search results.
Displaying categories when searching destination directories.
Handling when live_chat.json file exists.
Allowing refresh of directory directly.
Logging which download was successful.
Removing timeout.
Creating index for directory names.
Sleeping to catch cancel.
Arbitrary files can be downloaded using the new FileDownloader.
Tags can be created and deleted in UI.
Adding tag_file using id to avoid session error.
Using UTC in test.
Matching tag_names style in test.
Tag icon is displayed on file rows.
Archives, Videos and Files can be filtered by their tags.
Tags can be added and removed from UI.
Tags are displayed in VideoPlayer.
The tags of a file are returned in it's JSON.
Video favorites are converted to Favorite tag on channel config import.
Updating tests to match frontend usage.
PDF creator and modification time are displayed.
Reworking file search to use Video.__json__ for complex video relations.
Archives can be viewed.
Removing ebook.title.
Reworking alembic migration to be an upgrade rather than a replace.
Skipping test on circleci.
Clearing out video caption index only if extract_subtitles if off.
Channel tests pass.
Some video tests pass.
Missing videos can be found. A video can be downloaded.
Orphaned files can be found.
Videos can be tagged.
Videos can be searched.
Videos can be modeled.
Converting some tests to use FileGroup.primary_path.
Adding archive and domains to alembic migration.
Testing cascading Archive titles.
Archives now uses FileGroup.
archive.lib tests pass.
Archive is partially converted to FileGroup.
Calculating FileGroup.primary file on refresh, rather than on select.
wrolpi tests pass.
Using FilesType to convert between str/pathlib.Path for FileGroup.files.
Using list of files in FileGroup.files instead of dictionary.
Storing files in dedicated FileGroup.files to avoid overwriting them.
Adding comment explaining CASE.
Not overwriting FileGroup.data on refresh if no change has been detected.
Removing File model in favor of FileGroup.data.
Adding typing to Files methods.
Files are indexed using the primary mimetype of the FileGroup.
files test_lib tests pass.
Removing file_group_file in favor of simple one-to-many relationship.
PDF files are indexed during refresh.
Adding Tag and TagFile to tag files with specific tags.
Only iterating when full_stem_ids has contents.
Testing that FileGroups are related to their Files.
Files and file_groups are created and linked.
Bump webpack from 5.75.0 to 5.76.1 in /app (#160)
Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)
---
updated-dependencies:
- dependency-name: webpack
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Skipping video epoch test on circleci.
Created script which finds and cleans up (renames/deletes) orphaned video meta files.
Reworking orphaned video query, some video files were leaking through.
Modifying File table to allow finding of orphaned video files.
Extracting epoch from video info_json for more accurate upload_date.
Extracting epoch from video info_json for more accurate upload_date.
Skipping channel import during test unless explicitly requested.
Deleting gis database during upgrade, if user agrees. Increasing API verbosity.
Using status context to prevent fetching status in useHotspot.
Disabling wifi toggle button when pending.
Toggling hotspot when clicking nav hotspot icon.
Displaying Hotspot status icon in nav menu.
Importing channels/favorites after refreshing.
Refresh tasks cancel when server shuts down.
Bumping version for release speed.
Refreshing files is 3x faster.
Creating useRecurringTimeout hook which repeatedly calls a callback after a timeout.
Changing intervals to timeout so fetches don't stack up while server is loaded.
Checking that video file exists before fetching.
Creating release candidate branch. Not indexing model associated files.
Handling all combinations of toggle in light and dark.
Changing CPUUsageProgress colors to be more readable in dark mode.
Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming.
Changing video menu to dark mode.
Creating ThemeContext to share Dark Mode setting.
Removing duplicate code for search functions.
Displaying X when downloads are disabled.
Using react context to share settings across components.
Refusing to start workers when WROL Mode is enabled.
Changing warning to debug.
Queuing recurring downloads first.
Adding Video.url index.
Adding special handling for youtube shorts to videos.
Reworking Downloads to have one worker per process.
Testing creating downloads and getting downloaders.
Testing video validation and poster generation. Testing video statistics.
Increasing inventory test coverage.
Adding pytest.ini to automatically detect async test.
Adding more archive test coverage.
Testing Archive CRUD.
Catching and logging when encountering error during indexing.
Using FileIcon when Archive has no screenshot.
A Video will be assigned to a Channel by its info json first, then to its directory.
Including suffix in File json.
Adding index for File.suffix.
Gathering suffix during indexing.
Passing entire file down to FileIcon.
Moving logging up.
Returning File.associated in file json.
Associated files are last in file search.
Removing duplicate refreshing code by creating new _refresh_directory function.
Changing File.idempotency to a datetime to avoid having to clear idempotency.
Truncating indexes to avoid error with large files.
Don't overwrite video data when info json is empty.
Using already_downloaded for channel videos.
Awaiting refresh in Channel.refresh.
Testing that videos already downloaded aren't downloaded again.
Adding indexes for video/archive referenced paths.
Changing grid width for mobile or computer.
Adding button to refresh specific directories.
Restoring Channel edit link.
Fixing caption path in video_modeler.
Adding ZIP files to mimetype colors.
Using model instead of first file's model.
Fetching File models in batches to avoid issuing a SELECT for every File with a model.
Flushing generated poster file.
Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.
Attaching Video to it's Channel during download.
Using ExternalCardLink for ArchivePage singlefile link.
Validating and indexing videos after download.
Handling glob when file name contains matching characters.
Reworking and testing split_path_stem_and_suffix and glob_shared_stem.
Adding associated video files to DB during upsert.
Adding pending downloads to dashboard.
Hiding status when download is open.
Handling when load is empty.
Adding load statistics to Dashboard.
Limiting concurrent after_refresh functions.
Handling when no onSelect is declared.
Adding ImageRow for FileTable.
Moving image date into Card.Meta.
Using ExternalLink for FileCard.
Using ExternalLink for ImageCard.
Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs.
Removing api health check to avoid spamming logs.
Creating ImageCard to display image posters when they aren't too large.
Removing archive.domain_id not null.
Reworking file refresh to process all files in a single directory.
Bumping version for major file rework.
No longer checking for video corruption.
Adding duration by default to speed up tests.
Using register_indexer for all indexers.
Adding CPU usage progress bar in Welcome.
Modifying TextIndexer to ignore large text files.
Counting all files during refresh.
Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions.
Removing unused idempotency from video/channel tables.
Using magic module to speed up mimetype identification.
All tests pass.
Archive tests pass.
Changing video get endpoint to return 'file' to avoid ambiguity.
Video tests pass.
Simplifying regex filtering.
Using regex to ignore case when filtering.
Converting all channel classes to functions.
Restoring channel refresh and download buttons.
Matching format of Archive and Video pages.
Adding sorting to Archives search.
Reworking video orders for file search. Using SearchInput for videos search.
Archives can be searched on the Archives page. Domains can be filtered.
Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView.
Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage.
Creating useVideo for reworked VideoWrapper and VideoPage.
Adding Paginator to FilesView.
Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time.
Adding class for card icon to match poster height.
Video and Archives always re-claim their files on refresh.
Archive files are indexed and claimed.
FileView table is compact and striped.
Videos page uses FileView.
Updated navlinks to router v6.
Removing ! character from file names.
Disabling select button when cards are visible.
Adding show/hide button for FileTable selection.
The Archive Page is displayed.
Adding mimetype dropdown.
Upgrading to React Router v6 for useSearchParams.
Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results.
Updating video search to return Files.
Video and WROLPi tests pass.
Creating function to refresh files within a channel's directory.
Removing troubleshooting print. Updating result order in test.
Reworking video_factory to handle a specific video file location.
Returning video model on file search.
Getting session using video_file.
Screenshotting when checking for video corruption.
Reducing ffmpeg calls by gathering duration during poster generation.
Moving video file matching into model.
VideoIndexer is moved into video module. Video files are validated.
Moving files module into wrolpi in preparation to use it in modules.
Title and contents are indexed for text file.
Files are indexed on refresh by using their indexer.
Zip files are indexed using their contents.
Testing multiple zip files.
Adding unique to file.path. Checking that the correct indexer is used.
Adding index to file.modification_datetime and file.size.
Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False.
Video model is found during file refresh. A Video record is created that has the associated files.
Videos without associated files are found.
Videos are found using registered video_modeler() rather than in refresh code.
Removing old video columns.
Handling video info json files.
Trying to import each module individually. Will continue to import modules even if one fails.
Reworking refresh so it performs less DB calls.
Converting file.path to primary key.
file.path is always a pathlib.Path.
Updating video_factory to use new file system, testing it.
Bump sanic from 21.9.1 to 21.12.2 in /docker/archive
Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)
---
updated-dependencies:
- dependency-name: sanic
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
RSS download can be submitted in UI.
Checking if entry has link.
A video playlist can be downloaded.
Passing sub_downloader when submitting RSS feed.
Adding Archive downloader button.
Playlists need more info.
Removing expunge, it causes errors elsewhere. Fixing tests.
Showing errors on recurring downloads. Ignoring CharacterEncodingOverride bozo error for rss feed.
Changing download button order.
Fixing bug where channel attributes were unavailable after session. Simplifying video format selection.
Calling validate_video in Video.validate() to avoid duplicate code.
Adding more frequent downloads for RSS Feeds.
DownloadManager skips downloads in it's list.
Renaming get_next_download to more clear calculate_next_download.
Stopping download manager when server is shutting down.
Not shutting down downloads during tests, not all tests use downloads.
Making methods static.
Removing unused fixture in test.
Simplifying logic after download completes. Removing unused download sorting method.
ChannelDownloader returns location.
Hiding View link when no location is provided.
Performing video upsert in try/except so any errors are attached to the Download.
Including stderr only.
Including logs when returning Downloader.process_runner.
Warning when extention is replaced.
Providing location link on Downloads page. Showing error modal on errors that fail.
Handling empty URL strings on import.
Fixing channel download test.
Reworking how Channels schedule Video downloads.
Bumping version for DownloadResult.
Updating tests to use DownloadResult.
VideoDownloader returns a DownloadResult.
Removing default download freqeuncy so channels will not be downloaded without a url and frequency.
Adding channel unique url constraint.
Handling when a Channel's directory may still be relative in DB.
Adding TODOs to remove old favorites.
Removing outdated favorites from config after favorites have been created.
Testing that old-style favorited videos in the config are imported.
Removing channels_by_link.
Using list to store channels in config.
Moving channel import function into lib.
Supporting old channel import with links.
Bumping version for channel.link removal.
Removing all reference to Channel.link in app and api.
Adding relative property to MediaPathType.
Testing that a channel can be downloaded after it has a frequency added.
All tests pass.
Video and Channel tests pass without channel.link.
Removing channel.link usage in channel api.
Changing video Statistics page to About.
* A Download record is created for every Channel imported.
* Downloads must have a URL.
* Handling when channel has download frequency but no url.
* Fetching downloads after clearing.
* Setting next download when importing channels.
* Not deleting recurring completed downloads.
* Fixing comments.
* Skipping some service tests in CircleCI because they shouldn't be tested off a local machine.
Signed-off-by: Roland <roland@learningselfreliance.com>
commit 54d06bf8045fceaa2684348d110792e6833b170c
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 16:46:14 2022 -0700
Can't delete files when WROL Mode is enabled.
commit 2d91f1983aae47bdb3f6dc7fc2bd2ae642c1126a
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 16:37:42 2022 -0700
Fixing bug where files could not be deleted in series.
commit eed818c50682b19de9f18da2d4daedc54408cc49
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 16:26:31 2022 -0700
Fixing fake_now to reset now() when complete.
commit 183e98cbec25784720ebaeabfac8af4a3fc87da8
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 16:08:14 2022 -0700
Files can be deleted and downloaded.
commit c5bcd25a3da041f42b9e11eae804aa359ab9e985
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 14:41:07 2022 -0700
Files can be deleted or downloaded.
commit 690b332bb4f89aa4e97ce2590b35bd4a71fefd87
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 14:11:05 2022 -0700
File is opened in new tab when file is selected.
commit ba8ead6f3185d0a398c4f89cecb5958f5e7731a3
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 13:32:32 2022 -0700
Removing files sort because React FE sorts them itself.
commit 4e04753234b11d615c5992bc08e9dd9fb7226c05
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 13:29:52 2022 -0700
Duplicate directories are ignored.
commit c2f7d7929f37655e5c2d11767a512571263b35ef
Author: Roland <roland@learningselfreliance.com>
Date: Sat Jan 29 12:22:38 2022 -0700
Multiple directories can be requested.
commit 31678139c0799d0034d564cc1f8ae448234df90c
Author: Roland <roland@learningselfreliance.com>
Date: Fri Jan 28 17:21:33 2022 -0700
Files are displayed with the correct modified time. Files can be recursively viewed.
commit a2d10a1dd792e518c8c6ad8f699b5d416c3d07e4
Author: Roland <roland@learningselfreliance.com>
Date: Thu Jan 27 22:37:19 2022 -0700
Files can be viewed. Folders display their children.
