wrolpi/modules/videos/channel/api.py

153 lines
5.5 KiB
Python
Raw Normal View History

import asyncio
Fixing JS warnings. Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
import pathlib
from http import HTTPStatus
from sanic import response, Blueprint
from sanic.request import Request
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
2024-06-12 16:47:31 -06:00
from wrolpi.api_utils import json_response
from wrolpi.common import logger, wrol_mode_check, get_media_directory, get_relative_to_media_directory
from wrolpi.events import Events
from wrolpi.schema import JSONErrorResponse
from wrolpi.vars import PYTEST
from . import lib
from .. import schema
from ..errors import UnknownChannel
from ..lib import save_channels_config
from ..models import Channel
channel_bp = Blueprint('Channel', url_prefix='/api/videos/channels')
logger = logger.getChild(__name__)
@channel_bp.get('/')
@openapi.description('Get a list of all Channels')
@openapi.response(HTTPStatus.OK, schema.ChannelsResponse)
async def get_channels(_: Request):
channels = await lib.get_minimal_channels()
return json_response({'channels': channels})
@channel_bp.get('/<channel_id:int>')
@openapi.description('Get a Channel')
@openapi.response(HTTPStatus.OK, schema.ChannelResponse)
def channel_get(request: Request, channel_id: int = None):
session = request.ctx.session
channel = lib.get_channel(session, channel_id=channel_id, return_dict=True)
channel.pop('info_json')
return json_response({'channel': channel})
@channel_bp.post('/')
@openapi.definition(
description='Insert a Channel',
body=schema.ChannelPostRequest,
)
@openapi.response(HTTPStatus.OK, schema.ChannelPostResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@validate(schema.ChannelPostRequest)
@wrol_mode_check
def channel_post(request: Request, body: schema.ChannelPostRequest):
body.directory = get_media_directory() / body.directory
db: one request-scoped database handle per API request (#773) * db: one request-scoped database handle per API request Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that Session's connection, so a request needs at most one SQLite connection. Creating it does no I/O; requests that never touch the database never connect. `request.ctx.session` remains the handle's Session. A `RequestSession` binds itself to one lazily-opened Connection. An engine-bound Session releases its connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected. Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the write lock is not held for the rest of the handler or by background work it starts. This replaces the handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder, channel create). The zim search handlers use the request's Session in their worker thread. Tests count the connections a request opens with production's per-call sessions (`count_request_connections`). Endpoints whose library functions still open their own connection are strict xfails, to be removed as those functions take the request's session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: name the real extra connections in the request connection xfails Channel detail: get_channel already uses the request session; the extra connection is Channel.get_statistics opening a raw cursor (get_db_curs). Search: count_file_groups opens a cursor for the total, then handle_file_group_search_results opens a cursor and a session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 15:42:06 -06:00
# A write block: creating a Channel reads (the URL/name/directory conflict check) before it
# INSERTs, and a deferred transaction fails that lock upgrade *instantly* with "database is
# locked" if anything else wrote in between -- which is what a user creating a Channel while
# background workers saved configs hit. The block also commits before the refresh below is
# fired, so the refresh does not wait on a lock this request still holds.
with request.ctx.db.write() as session:
channel = lib.create_channel(session, data=body, return_dict=False)
channel_id, channel_name = channel.id, channel.name
# Refresh the videos asynchronously
Channel.refresh_files(channel_id)
Events.send_created(f'Created Channel: {channel_name}')
return response.json({'success': 'Channel created successfully'}, HTTPStatus.CREATED,
{'Location': f'/api/videos/channels/{channel_id}'})
@channel_bp.put('/<channel_id:int>')
@openapi.definition(
description='Update a Channel',
body=schema.ChannelPutRequest,
)
@openapi.response(HTTPStatus.NO_CONTENT)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@validate(schema.ChannelPutRequest)
@wrol_mode_check
async def channel_update(request: Request, channel_id: int, body: schema.ChannelPutRequest):
session = request.ctx.session
Fixing JS warnings. Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
body.directory = pathlib.Path(body.directory) if body.directory else None
channel = await lib.update_channel(session, data=body, channel_id=channel_id)
save_channels_config.activate_switch()
return response.raw('', HTTPStatus.NO_CONTENT,
headers={'Location': f'/api/videos/channels/{channel.id}'})
@channel_bp.delete('/<channel_id:int>')
@openapi.description('Delete a Channel')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_delete(request: Request, channel_id: int):
session = request.ctx.session
channel = lib.delete_channel(session, channel_id=channel_id)
Events.send_deleted(f'Deleted Channel: {channel["name"]}')
return response.raw('', HTTPStatus.NO_CONTENT)
Fixing JS warnings. Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
@channel_bp.post('/refresh/<channel_id:int>')
@openapi.description('Refresh all files in the Channel directory')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_refresh(request: Request, channel_id: int):
session = request.ctx.session
channel: Channel = lib.get_channel(session, channel_id=channel_id, return_dict=False)
Fixing JS warnings. Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
if not channel:
raise UnknownChannel()
channel.refresh_files(channel.id)
directory = get_relative_to_media_directory(channel.directory)
Events.send_directory_refresh(f'Refreshing: {directory}')
Fixing JS warnings. Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
2022-08-06 05:22:07 +00:00
return response.empty()
@channel_bp.post('/<channel_id:int>/tag')
@openapi.definition(
description='Tag/untag a Channel with a single tag',
body=schema.ChannelTagRequest,
)
@openapi.response(HTTPStatus.NO_CONTENT)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@validate(schema.ChannelTagRequest)
@wrol_mode_check
async def channel_tag(request: Request, channel_id: int, body: schema.ChannelTagRequest):
session = request.ctx.session
directory = None
if body.directory:
directory = pathlib.Path(body.directory)
directory = get_media_directory() / directory
await lib.tag_channel(session, body.tag_name, directory, channel_id)
return response.raw('', HTTPStatus.NO_CONTENT)
@channel_bp.post('/search')
@openapi.definition(
description='Search Channels',
body=schema.ChannelSearchRequest,
)
@validate(schema.ChannelSearchRequest)
async def channel_search(request: Request, body: schema.ChannelSearchRequest):
session = request.ctx.session
channels = await lib.search_channels(session, body.tag_names)
ret = dict(
channels=channels,
)
2024-09-17 18:04:00 -07:00
return json_response(ret)