mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-11-22 16:06:19 -05:00
Fixing JS warnings.
Handling all combinations of toggle in light and dark. Changing CPUUsageProgress colors to be more readable in dark mode. Moving theme handling into Theme.js. Importing element wrappers from Theme.js to handle element theming. Changing video menu to dark mode. Creating ThemeContext to share Dark Mode setting. Removing duplicate code for search functions. Displaying X when downloads are disabled. Using react context to share settings across components. Refusing to start workers when WROL Mode is enabled. Changing warning to debug. Queuing recurring downloads first. Adding Video.url index. Adding special handling for youtube shorts to videos. Reworking Downloads to have one worker per process. Testing creating downloads and getting downloaders. Testing video validation and poster generation. Testing video statistics. Increasing inventory test coverage. Adding pytest.ini to automatically detect async test. Adding more archive test coverage. Testing Archive CRUD. Catching and logging when encountering error during indexing. Using FileIcon when Archive has no screenshot. A Video will be assigned to a Channel by its info json first, then to its directory. Including suffix in File json. Adding index for File.suffix. Gathering suffix during indexing. Passing entire file down to FileIcon. Moving logging up. Returning File.associated in file json. Associated files are last in file search. Removing duplicate refreshing code by creating new _refresh_directory function. Changing File.idempotency to a datetime to avoid having to clear idempotency. Truncating indexes to avoid error with large files. Don't overwrite video data when info json is empty. Using already_downloaded for channel videos. Awaiting refresh in Channel.refresh. Testing that videos already downloaded aren't downloaded again. Adding indexes for video/archive referenced paths. Changing grid width for mobile or computer. Adding button to refresh specific directories. Restoring Channel edit link. Fixing caption path in video_modeler. Adding ZIP files to mimetype colors. Using model instead of first file's model. Fetching File models in batches to avoid issuing a SELECT for every File with a model. Flushing generated poster file. Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id. Attaching Video to it's Channel during download. Using ExternalCardLink for ArchivePage singlefile link. Validating and indexing videos after download. Handling glob when file name contains matching characters. Reworking and testing split_path_stem_and_suffix and glob_shared_stem. Adding associated video files to DB during upsert. Adding pending downloads to dashboard. Hiding status when download is open. Handling when load is empty. Adding load statistics to Dashboard. Limiting concurrent after_refresh functions. Handling when no onSelect is declared. Adding ImageRow for FileTable. Moving image date into Card.Meta. Using ExternalLink for FileCard. Using ExternalLink for ImageCard. Handling when error occurs during refresh of a directory. Committing during refresh to avoid losing all files if an error occurs. Removing api health check to avoid spamming logs. Creating ImageCard to display image posters when they aren't too large. Removing archive.domain_id not null. Reworking file refresh to process all files in a single directory. Bumping version for major file rework. No longer checking for video corruption. Adding duration by default to speed up tests. Using register_indexer for all indexers. Adding CPU usage progress bar in Welcome. Modifying TextIndexer to ignore large text files. Counting all files during refresh. Adding refresh_logger to clarify logging. Checking for wrol_mode before refresh functions. Removing unused idempotency from video/channel tables. Using magic module to speed up mimetype identification. All tests pass. Archive tests pass. Changing video get endpoint to return 'file' to avoid ambiguity. Video tests pass. Simplifying regex filtering. Using regex to ignore case when filtering. Converting all channel classes to functions. Restoring channel refresh and download buttons. Matching format of Archive and Video pages. Adding sorting to Archives search. Reworking video orders for file search. Using SearchInput for videos search. Archives can be searched on the Archives page. Domains can be filtered. Reworking Archive search to use Files. Videos, Archives and Global search use the FilesView. Videos are claimed by their Channel based on the Channel's directory. Prev and Next video cards are displayed on the VideoPage. Creating useVideo for reworked VideoWrapper and VideoPage. Adding Paginator to FilesView. Moving file refresh button out of manage tab. Removing Files page tabs. Limiting refresh to one at a time. Adding class for card icon to match poster height. Video and Archives always re-claim their files on refresh. Archive files are indexed and claimed. FileView table is compact and striped. Videos page uses FileView. Updated navlinks to router v6. Removing ! character from file names. Disabling select button when cards are visible. Adding show/hide button for FileTable selection. The Archive Page is displayed. Adding mimetype dropdown. Upgrading to React Router v6 for useSearchParams. Validating video during modeling. Removing screenshot during corruption check because it was very slow. Updating app to use video files as search results. Updating video search to return Files. Video and WROLPi tests pass. Creating function to refresh files within a channel's directory. Removing troubleshooting print. Updating result order in test. Reworking video_factory to handle a specific video file location. Returning video model on file search. Getting session using video_file. Screenshotting when checking for video corruption. Reducing ffmpeg calls by gathering duration during poster generation. Moving video file matching into model. VideoIndexer is moved into video module. Video files are validated. Moving files module into wrolpi in preparation to use it in modules. Title and contents are indexed for text file. Files are indexed on refresh by using their indexer. Zip files are indexed using their contents. Testing multiple zip files. Adding unique to file.path. Checking that the correct indexer is used. Adding index to file.modification_datetime and file.size. Handling file info gathering in File.do_index. Only reading contents of file if file.indexed is False. Video model is found during file refresh. A Video record is created that has the associated files. Videos without associated files are found. Videos are found using registered video_modeler() rather than in refresh code. Removing old video columns. Handling video info json files. Trying to import each module individually. Will continue to import modules even if one fails. Reworking refresh so it performs less DB calls. Converting file.path to primary key. file.path is always a pathlib.Path. Updating video_factory to use new file system, testing it. Bump sanic from 21.9.1 to 21.12.2 in /docker/archive Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2. - [Release notes](https://github.com/sanic-org/sanic/releases) - [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst) - [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2) --- updated-dependencies: - dependency-name: sanic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
This commit is contained in:
parent
1ede5f3f55
commit
d4ed5fa5ce
98 changed files with 7441 additions and 5257 deletions
|
|
@ -1,20 +1,28 @@
|
|||
import json
|
||||
import pathlib
|
||||
from abc import ABC
|
||||
from typing import Tuple, Optional
|
||||
from typing import Tuple, Dict, List
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from wrolpi.common import logger
|
||||
from wrolpi.db import optional_session
|
||||
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
|
||||
from wrolpi.db import optional_session, get_db_session
|
||||
from wrolpi.downloader import Downloader, Download, DownloadResult
|
||||
from wrolpi.errors import UnrecoverableDownloadError
|
||||
from wrolpi.files.indexers import register_indexer, Indexer
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PYTEST
|
||||
from . import lib
|
||||
from .api import bp # noqa
|
||||
from .models import Archive
|
||||
from .lib import match_archive_files
|
||||
from .models import Archive, Domain
|
||||
|
||||
PRETTY_NAME = 'Archive'
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
__all__ = ['ArchiveDownloader', 'archive_downloader']
|
||||
|
||||
|
||||
class ArchiveDownloader(Downloader, ABC):
|
||||
name = 'archive'
|
||||
|
|
@ -38,9 +46,134 @@ class ArchiveDownloader(Downloader, ABC):
|
|||
return DownloadResult(success=True, location=f'/archive/{archive.id}')
|
||||
|
||||
@optional_session
|
||||
def already_downloaded(self, url: str, session: Session = None) -> bool:
|
||||
return bool(session.query(Archive).filter_by(url=url).count())
|
||||
def already_downloaded(self, urls: List[str], session: Session = None) -> List:
|
||||
archives = list(session.query(Archive).filter(Archive.url.in_(urls)))
|
||||
return archives
|
||||
|
||||
|
||||
# Archive downloader is the last downloader which should be used.
|
||||
archive_downloader = ArchiveDownloader(priority=100)
|
||||
|
||||
|
||||
@register_modeler
|
||||
def archive_modeler(groups: Dict[str, List[File]], session: Session):
|
||||
archive_groups = {}
|
||||
for stem, group in groups.items():
|
||||
if any(lib.is_singlefile_file(i.path) for i in group):
|
||||
archive_groups[stem] = group
|
||||
|
||||
count = 0
|
||||
for stem, group in archive_groups.items():
|
||||
readability_file, singlefile_file, readability_json_file, readability_txt_file, screenshot_file = \
|
||||
match_archive_files(group)
|
||||
archive: Archive = Archive.find_by_path(singlefile_file.path, session)
|
||||
if not archive:
|
||||
archive = Archive(singlefile_file=singlefile_file)
|
||||
session.add(archive)
|
||||
archive.readability_file = readability_file
|
||||
archive.readability_json_file = readability_json_file
|
||||
archive.readability_txt_file = readability_txt_file
|
||||
archive.screenshot_file = screenshot_file
|
||||
|
||||
if archive.readability_file:
|
||||
archive.readability_file.associated = True
|
||||
archive.readability_file.do_index()
|
||||
if archive.readability_json_file:
|
||||
archive.readability_json_file.associated = True
|
||||
archive.readability_json_file.do_index()
|
||||
if archive.readability_txt_file:
|
||||
archive.readability_txt_file.associated = True
|
||||
archive.readability_txt_file.do_index()
|
||||
if archive.screenshot_file:
|
||||
archive.screenshot_file.associated = True
|
||||
archive.screenshot_file.do_index()
|
||||
|
||||
archive.singlefile_file.model = Archive.__tablename__
|
||||
|
||||
archive.singlefile_file.do_index()
|
||||
archive.validate()
|
||||
|
||||
# Remove this group, it will not be processed by other modelers.
|
||||
del groups[stem]
|
||||
|
||||
count += 1
|
||||
if not count % 100:
|
||||
# Commit occasionally.
|
||||
session.commit()
|
||||
|
||||
|
||||
@register_after_refresh
|
||||
@limit_concurrent(1)
|
||||
def archive_cleanup():
|
||||
# Process all Archives that have not been validated. Delete any that no longer exist, or are not real Archives.
|
||||
# Validate all other Archives.
|
||||
logger.info('Searching for invalid Archives')
|
||||
offset = 0
|
||||
limit = 100
|
||||
while True:
|
||||
with get_db_session(commit=True) as session:
|
||||
archives: List[Archive] = session.query(Archive).order_by(Archive.id).limit(limit).offset(offset)
|
||||
processed = 0
|
||||
deleted = 0
|
||||
for archive in archives:
|
||||
processed += 1
|
||||
if not lib.is_singlefile_file(archive.singlefile_path):
|
||||
# Archive is not a real archive.
|
||||
archive.singlefile_file.model = None
|
||||
session.delete(archive)
|
||||
deleted += 1
|
||||
if processed < limit:
|
||||
# Validated all archives.
|
||||
break
|
||||
offset += (limit - deleted)
|
||||
|
||||
with get_db_session(commit=True) as session:
|
||||
# Remove any Domains without any Archives.
|
||||
domain_ids = [i[0] for i in session.execute('SELECT DISTINCT domain_id FROM archive') if i[0]]
|
||||
deleted = 0
|
||||
for domain in session.query(Domain):
|
||||
if domain.id not in domain_ids:
|
||||
deleted += 1
|
||||
session.delete(domain)
|
||||
|
||||
|
||||
@register_indexer('text/html')
|
||||
class ArchiveIndexer(Indexer, ABC):
|
||||
|
||||
@staticmethod
|
||||
def get_title(file: File):
|
||||
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
|
||||
readability_json_path = singlefile_path.with_suffix('.readability.json')
|
||||
if readability_json_path.is_file():
|
||||
with readability_json_path.open('rt') as fh:
|
||||
try:
|
||||
return json.load(fh)['title']
|
||||
except Exception:
|
||||
if not PYTEST:
|
||||
logger.warning(f'Archive readability json file exists, but cannot get title. {file}')
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def get_article(file: File):
|
||||
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
|
||||
readability_txt_path = singlefile_path.with_suffix('.readability.txt')
|
||||
if readability_txt_path.is_file():
|
||||
return readability_txt_path.read_text()
|
||||
|
||||
@classmethod
|
||||
def create_index(cls, file: File) -> Tuple:
|
||||
"""
|
||||
Index an Archive file and it's associated files. This mimetype matches non-archive (non-singlefile) html files,
|
||||
so this index will return empty if a readability file is found.
|
||||
|
||||
a = title
|
||||
b = <empty>
|
||||
c = <empty>
|
||||
d = readability article
|
||||
"""
|
||||
if file.path.name.endswith('.readability.html'):
|
||||
# Readability files are not the primary Archive file.
|
||||
return None, None, None, None
|
||||
a = cls.get_title(file)
|
||||
d = cls.get_article(file)
|
||||
return a, None, None, d
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
import asyncio
|
||||
from http import HTTPStatus
|
||||
|
||||
from sanic import response, Request
|
||||
|
|
@ -22,8 +21,9 @@ logger = logger.getChild(__name__)
|
|||
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
|
||||
async def get_archive(_: Request, archive_id: int):
|
||||
archive = lib.get_archive(archive_id=archive_id)
|
||||
alternatives = archive.alternatives
|
||||
return json_response({'archive': archive, 'alternatives': alternatives})
|
||||
archive_file = archive.singlefile_file.__json__()
|
||||
alternatives = [i.singlefile_file.__json__() for i in archive.alternatives]
|
||||
return json_response({'file': archive_file, 'alternatives': alternatives})
|
||||
|
||||
|
||||
@bp.delete('/<archive_id:int>')
|
||||
|
|
@ -35,13 +35,6 @@ async def delete_archive(_: Request, archive_id: int):
|
|||
return response.empty()
|
||||
|
||||
|
||||
@bp.post('/refresh')
|
||||
@wrol_mode_check
|
||||
async def refresh_archives(_: Request):
|
||||
asyncio.ensure_future(lib.refresh_archives())
|
||||
return response.empty()
|
||||
|
||||
|
||||
@bp.get('/domains')
|
||||
async def fetch_domains(_: Request):
|
||||
domains = lib.get_domains()
|
||||
|
|
@ -65,6 +58,6 @@ async def search_archives(_: Request, body: schema.ArchiveSearchRequest):
|
|||
limit = archive_limit_limiter(body.limit)
|
||||
offset = body.offset or 0
|
||||
|
||||
archives, total = lib.search(search_str, domain, limit, offset)
|
||||
archives, total = lib.archive_search(search_str, domain, limit, offset, body.order_by)
|
||||
ret = dict(archives=archives, totals=dict(archives=total))
|
||||
return json_response(ret)
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import pytest
|
|||
from modules.archive.lib import archive_strftime
|
||||
from modules.archive.models import Archive, Domain
|
||||
from wrolpi.dates import local_timezone
|
||||
from wrolpi.files.models import File
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
|
@ -18,7 +19,7 @@ def archive_directory(test_directory) -> pathlib.Path:
|
|||
|
||||
|
||||
@pytest.fixture
|
||||
def archive_factory(test_session, archive_directory):
|
||||
def archive_factory(test_session, archive_directory, make_files_structure):
|
||||
def time_generator():
|
||||
timestamp = datetime.datetime(2000, 1, 1, 0, 0, 0)
|
||||
while True:
|
||||
|
|
@ -27,12 +28,12 @@ def archive_factory(test_session, archive_directory):
|
|||
|
||||
now = time_generator()
|
||||
|
||||
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None) -> Archive:
|
||||
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None,
|
||||
singlefile_contents: str = None) -> Archive:
|
||||
if domain:
|
||||
assert '/' not in domain
|
||||
|
||||
title = title or str(uuid4())
|
||||
url = url or f'https://example.com/{title}'
|
||||
domain_dir = domain or title
|
||||
domain_dir = archive_directory / domain_dir
|
||||
domain_dir.mkdir(exist_ok=True)
|
||||
|
|
@ -40,19 +41,25 @@ def archive_factory(test_session, archive_directory):
|
|||
archive_datetime = local_timezone(next(now))
|
||||
timestamp = archive_strftime(archive_datetime)
|
||||
|
||||
singlefile_path = domain_dir / f'{timestamp}_{title}.html'
|
||||
screenshot_path = domain_dir / f'{timestamp}_{title}.png'
|
||||
readability_path = domain_dir / f'{timestamp}_{title}.readability.html'
|
||||
readability_txt_path = domain_dir / f'{timestamp}_{title}.readability.txt'
|
||||
readability_json_path = domain_dir / f'{timestamp}_{title}.readability.json'
|
||||
|
||||
with readability_json_path.open('wt') as fh:
|
||||
json.dump({'url': url}, fh)
|
||||
json_contents = {}
|
||||
if title != 'NA':
|
||||
json_contents['title'] = title
|
||||
if url:
|
||||
json_contents['url'] = url
|
||||
json_contents = json.dumps(json_contents)
|
||||
singlefile_path, screenshot_path, readability_path, readability_txt_path, readability_json_path = \
|
||||
make_files_structure({
|
||||
str(domain_dir / f'{timestamp}_{title}.html'): singlefile_contents or '<html></html>',
|
||||
str(domain_dir / f'{timestamp}_{title}.png'): None,
|
||||
str(domain_dir / f'{timestamp}_{title}.readability.html'): '<html></html>',
|
||||
str(domain_dir / f'{timestamp}_{title}.readability.txt'): contents,
|
||||
str(domain_dir / f'{timestamp}_{title}.readability.json'): json_contents,
|
||||
})
|
||||
|
||||
if domain:
|
||||
domain = test_session.query(Domain).filter_by(domain=domain).one_or_none()
|
||||
|
||||
if not domain:
|
||||
if not domain and domain_dir.name != 'NA':
|
||||
domain = Domain(
|
||||
domain=domain_dir.name,
|
||||
directory=domain_dir,
|
||||
|
|
@ -61,23 +68,18 @@ def archive_factory(test_session, archive_directory):
|
|||
test_session.flush([domain])
|
||||
|
||||
archive = Archive(
|
||||
readability_json_path=readability_json_path,
|
||||
readability_path=readability_path,
|
||||
readability_txt_path=readability_txt_path,
|
||||
screenshot_path=screenshot_path,
|
||||
singlefile_path=singlefile_path,
|
||||
readability_json_file=File(path=readability_json_path) if readability_json_path else None,
|
||||
readability_file=File(path=readability_path) if readability_path else None,
|
||||
readability_txt_file=File(path=readability_txt_path) if readability_txt_path else None,
|
||||
screenshot_file=File(path=screenshot_path) if screenshot_path else None,
|
||||
singlefile_file=File(path=singlefile_path, model='archive'),
|
||||
title=title,
|
||||
url=url,
|
||||
domain_id=domain.id,
|
||||
contents=contents,
|
||||
domain_id=domain.id if domain else None,
|
||||
archive_datetime=archive_datetime,
|
||||
)
|
||||
test_session.add(archive)
|
||||
|
||||
test_session.commit()
|
||||
|
||||
for path in archive.my_paths():
|
||||
path.touch()
|
||||
archive.singlefile_file.do_index()
|
||||
|
||||
return archive
|
||||
|
||||
|
|
|
|||
|
|
@ -7,21 +7,22 @@ import subprocess
|
|||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from functools import partial
|
||||
from itertools import groupby
|
||||
from typing import Iterator, Optional, Tuple, List
|
||||
|
||||
import aiohttp
|
||||
from bs4 import BeautifulSoup
|
||||
from selenium import webdriver
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from modules.archive.models import Domain, Archive
|
||||
from wrolpi.cmd import which
|
||||
from wrolpi.common import get_media_directory, logger, chunks, extract_domain, chdir, escape_file_name, walk, \
|
||||
aiohttp_post
|
||||
from wrolpi.common import get_media_directory, logger, extract_domain, chdir, escape_file_name, walk, \
|
||||
aiohttp_post, match_paths_to_suffixes, iterify
|
||||
from wrolpi.dates import now, Seconds, local_timezone
|
||||
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models, optional_session
|
||||
from wrolpi.errors import InvalidDomain, UnknownURL, InvalidArchive
|
||||
from wrolpi.db import get_db_session, get_db_curs, optional_session
|
||||
from wrolpi.errors import UnknownArchive, InvalidOrderBy
|
||||
from wrolpi.files.lib import handle_search_results
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import DOCKERIZED, PYTEST
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
|
@ -284,12 +285,11 @@ async def do_archive(url: str) -> Archive:
|
|||
archive = Archive(
|
||||
title=title,
|
||||
archive_datetime=now(),
|
||||
singlefile_path=archive_files.singlefile,
|
||||
readability_path=archive_files.readability,
|
||||
readability_json_path=archive_files.readability_json,
|
||||
readability_txt_path=archive_files.readability_txt,
|
||||
screenshot_path=archive_files.screenshot,
|
||||
contents=readability_txt,
|
||||
singlefile_file=File(path=archive_files.singlefile, model='archive'),
|
||||
readability_file=File(path=archive_files.readability) if archive_files.readability else None,
|
||||
readability_json_file=File(path=archive_files.readability_json) if archive_files.readability_json else None,
|
||||
readability_txt_file=File(path=archive_files.readability_txt) if archive_files.readability_txt else None,
|
||||
screenshot_file=File(path=archive_files.screenshot) if archive_files.screenshot else None,
|
||||
url=url,
|
||||
domain_id=domain.id,
|
||||
)
|
||||
|
|
@ -323,19 +323,12 @@ def get_title_from_html(html: str, url: str = None) -> str:
|
|||
logger.info(f'Unable to extract title {url}')
|
||||
|
||||
|
||||
def get_domain(session, domain: str) -> Domain:
|
||||
domain_ = session.query(Domain).filter_by(domain=domain).one_or_none()
|
||||
if not domain_:
|
||||
raise InvalidDomain(f'Invalid domain: {domain}')
|
||||
return domain_
|
||||
|
||||
|
||||
@optional_session
|
||||
def get_archive(session, archive_id: int) -> Archive:
|
||||
"""Get an Archive."""
|
||||
archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
|
||||
if not archive:
|
||||
raise InvalidArchive(f'Invalid archive id: {archive_id}')
|
||||
raise UnknownArchive(f'Could not find archive with id: {archive_id}')
|
||||
return archive
|
||||
|
||||
|
||||
|
|
@ -344,7 +337,7 @@ def delete_archive(archive_id: int):
|
|||
with get_db_session(commit=True) as session:
|
||||
archive: Archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
|
||||
if not archive:
|
||||
raise UnknownURL(f'Unknown Archive with id: {archive_id}')
|
||||
raise UnknownArchive(f'Unknown Archive with id: {archive_id}')
|
||||
|
||||
# Delete any files associated with this URL.
|
||||
archive.delete()
|
||||
|
|
@ -476,95 +469,13 @@ def is_archive_file(path: pathlib.Path) -> bool:
|
|||
return path.is_file() and path.suffix.lower() in ARCHIVE_SUFFIXES and bool(ARCHIVE_MATCHER.match(path.name))
|
||||
|
||||
|
||||
def _refresh_archives():
|
||||
def is_singlefile_file(path: pathlib.Path) -> bool:
|
||||
"""
|
||||
Search the Archives directory for archive files, update the database if new files are found. Remove any orphan
|
||||
URLs or Domains.
|
||||
Archive singlefile files are expected to start with the following: %Y-%m-%d-%H-%M-%S
|
||||
they must end with .html, but never with .readability.html.
|
||||
"""
|
||||
archive_directory = get_archive_directory()
|
||||
|
||||
# TODO remove this later when everyone has migrated their files.
|
||||
migrate_archive_files()
|
||||
|
||||
singlefile_paths = set()
|
||||
for domain_directory in filter(lambda i: i.is_dir(), archive_directory.iterdir()):
|
||||
logger.debug(f'Refreshing directory: {domain_directory}')
|
||||
all_archives_files = filter(is_archive_file, walk(domain_directory))
|
||||
archive_groups = group_archive_files(all_archives_files)
|
||||
archive_groups = list(archive_groups)
|
||||
archive_count = 0
|
||||
for chunk in chunks(archive_groups, 20):
|
||||
with get_db_session(commit=True) as session:
|
||||
for dt, archive_files in chunk:
|
||||
archive_count += 1
|
||||
singlefile_paths.add(str(archive_files.singlefile))
|
||||
upsert_archive(dt, archive_files, session)
|
||||
|
||||
if archive_count:
|
||||
logger.info(f'Inserted/updated {archive_count} archives in {domain_directory}')
|
||||
|
||||
singlefile_paths = list(singlefile_paths)
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('DELETE FROM archive WHERE singlefile_path != ALL(%s)', (singlefile_paths,))
|
||||
|
||||
with get_db_curs(commit=True) as curs:
|
||||
stmt = '''
|
||||
DELETE FROM domains WHERE
|
||||
id NOT IN (select distinct domain_id from archive)
|
||||
RETURNING domains.id
|
||||
'''
|
||||
curs.execute(stmt)
|
||||
domains = list(map(dict, curs.fetchall()))
|
||||
logger.info(f'Deleted {len(domains)} Domains')
|
||||
|
||||
with get_db_session(commit=True) as session:
|
||||
# Get all archives that have no text contents, but have txt files.
|
||||
archives = session.query(Archive).filter(
|
||||
Archive.contents == None, # noqa
|
||||
Archive.readability_txt_path != None,
|
||||
).all()
|
||||
for archive in archives:
|
||||
with archive.readability_txt_path.path.open('rt') as fh:
|
||||
archive.contents = fh.read()
|
||||
|
||||
|
||||
async def refresh_archives():
|
||||
_refresh_archives()
|
||||
|
||||
|
||||
def upsert_archive(dt: str, archive_files: ArchiveFiles, session: Session):
|
||||
"""Get or create an Archive, and it's URL/Domain. If it already exists, update it with these new files."""
|
||||
# Extract the URL from the JSON. Fail if this is not possible.
|
||||
try:
|
||||
with archive_files.readability_json.open() as fh:
|
||||
json_contents = json.load(fh)
|
||||
url = json_contents['url']
|
||||
title = json_contents.get('title')
|
||||
except Exception as e:
|
||||
raise InvalidArchive() from e
|
||||
|
||||
domain = get_or_create_domain(session, url)
|
||||
# Get the existing Archive, or create a new one.
|
||||
archive = session.query(Archive).filter_by(singlefile_path=archive_files.singlefile).one_or_none()
|
||||
if not archive:
|
||||
# No archive matches this singlefile_path, create a new one.
|
||||
archive = Archive(url=url, domain_id=domain.id)
|
||||
session.add(archive)
|
||||
session.flush()
|
||||
|
||||
if not archive.title and title:
|
||||
archive.title = title
|
||||
if not archive.title and archive_files.singlefile:
|
||||
# As a last resort, get the title from the HTML.
|
||||
archive.title = get_title_from_html(archive_files.singlefile.read_text(), url)
|
||||
|
||||
# Update the archive with the files that we have.
|
||||
archive.archive_datetime = dt
|
||||
archive.singlefile_path = archive_files.singlefile
|
||||
archive.readability_path = archive_files.readability
|
||||
archive.readability_txt_path = archive_files.readability_txt
|
||||
archive.readability_json_path = archive_files.readability_json
|
||||
archive.screenshot_path = archive_files.screenshot
|
||||
return path.is_file() and path.suffix.lower() == '.html' and not path.name.endswith('.readability.html') \
|
||||
and bool(ARCHIVE_MATCHER.match(path.name))
|
||||
|
||||
|
||||
def get_domains():
|
||||
|
|
@ -581,46 +492,69 @@ def get_domains():
|
|||
return domains
|
||||
|
||||
|
||||
def search(search_str: str, domain: str, limit: int, offset: int) -> Tuple[List[Archive], int]:
|
||||
with get_db_curs() as curs:
|
||||
columns = 'id, COUNT(*) OVER() AS total'
|
||||
params = dict(offset=offset, limit=limit)
|
||||
wheres = ''
|
||||
order_by = '1 DESC'
|
||||
ARCHIVE_ORDERS = {
|
||||
'date': 'a.archive_datetime ASC, LOWER(a.singlefile_path) ASC',
|
||||
'-date': 'a.archive_datetime DESC NULLS LAST, LOWER(a.singlefile_path) DESC',
|
||||
'rank': '2 DESC, a.archive_datetime DESC',
|
||||
'-rank': '2 ASC, a.archive_datetime ASC',
|
||||
}
|
||||
|
||||
if search_str:
|
||||
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
|
||||
params['search_str'] = search_str
|
||||
wheres += '\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)'
|
||||
# highest rank, then most recent
|
||||
order_by = '2 DESC, archive_datetime DESC'
|
||||
|
||||
if domain:
|
||||
curs.execute('SELECT id FROM domains WHERE domain=%s', (domain,))
|
||||
try:
|
||||
domain_id = curs.fetchone()[0]
|
||||
except TypeError:
|
||||
# No domains match the provided domain.
|
||||
return [], 0
|
||||
params['domain_id'] = domain_id
|
||||
wheres += '\nAND domain_id = %(domain_id)s'
|
||||
def archive_search(search_str: str, domain: str, limit: int, offset: int, order_by: str) -> Tuple[List[Archive], int]:
|
||||
wheres = []
|
||||
params = dict(offset=offset, limit=limit)
|
||||
order = '1 DESC'
|
||||
|
||||
# TODO handle different Archive search orders.
|
||||
stmt = f'''
|
||||
SELECT {columns}
|
||||
FROM archive
|
||||
WHERE
|
||||
singlefile_path IS NOT NULL AND singlefile_path != ''
|
||||
{wheres}
|
||||
ORDER BY {order_by}
|
||||
OFFSET %(offset)s
|
||||
LIMIT %(limit)s
|
||||
'''
|
||||
curs.execute(stmt, params)
|
||||
results = [dict(i) for i in curs.fetchall()]
|
||||
total = results[0]['total'] if results else 0
|
||||
ranked_ids = [i['id'] for i in results]
|
||||
if search_str:
|
||||
columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
|
||||
params['search_str'] = search_str
|
||||
wheres.append('\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)')
|
||||
# highest rank, then most recent
|
||||
if not order_by:
|
||||
order = ARCHIVE_ORDERS['rank']
|
||||
else:
|
||||
# No search_str provided, get path and total only.
|
||||
columns = 'f.path, COUNT(*) OVER() AS total'
|
||||
|
||||
results = get_ranked_models(ranked_ids, Archive)
|
||||
if order_by:
|
||||
try:
|
||||
order = ARCHIVE_ORDERS[order_by]
|
||||
except KeyError:
|
||||
raise InvalidOrderBy(f'Invalid order by: {order_by}')
|
||||
|
||||
if domain:
|
||||
params['domain'] = domain
|
||||
wheres.append('AND domain_id = (select id from domains where domains.domain = %(domain)s)')
|
||||
|
||||
wheres = '\n'.join(wheres)
|
||||
stmt = f'''
|
||||
SELECT {columns}
|
||||
FROM archive a
|
||||
LEFT JOIN file f on f.path = a.singlefile_path
|
||||
WHERE
|
||||
a.singlefile_path != ''
|
||||
{wheres}
|
||||
ORDER BY {order}
|
||||
OFFSET %(offset)s
|
||||
LIMIT %(limit)s
|
||||
'''
|
||||
logger.warning(stmt, params)
|
||||
|
||||
results, total = handle_search_results(stmt, params)
|
||||
return results, total
|
||||
|
||||
|
||||
match_archive_paths = partial(match_paths_to_suffixes, suffix_groups=(
|
||||
('.readability.html',),
|
||||
('.html',),
|
||||
('.readability.json',),
|
||||
('.readability.txt',),
|
||||
('.jpg', '.jpeg', '.webp', '.png'),
|
||||
))
|
||||
|
||||
|
||||
@iterify(tuple)
|
||||
def match_archive_files(files: List[File]) -> Tuple[File, File, File, File, File]:
|
||||
archive_paths = match_archive_paths([i.path for i in files])
|
||||
for path in archive_paths:
|
||||
yield next((i for i in files if i.path == path), None)
|
||||
|
|
|
|||
|
|
@ -1,13 +1,22 @@
|
|||
import datetime
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
from typing import Generator
|
||||
|
||||
from sqlalchemy import Column, Integer, String, ForeignKey, Computed
|
||||
from sqlalchemy.orm import relationship, Session
|
||||
import pytz
|
||||
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean
|
||||
from sqlalchemy.orm import relationship, Session, validates
|
||||
from sqlalchemy.orm.collections import InstrumentedList
|
||||
|
||||
from wrolpi.common import ModelHelper, Base, tsvector
|
||||
from wrolpi.common import ModelHelper, Base, logger
|
||||
from wrolpi.dates import TZDateTime
|
||||
from wrolpi.errors import InvalidArchive
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.media_path import MediaPathType
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
|
||||
class Archive(Base, ModelHelper):
|
||||
__tablename__ = 'archive'
|
||||
|
|
@ -16,34 +25,40 @@ class Archive(Base, ModelHelper):
|
|||
domain_id = Column(Integer, ForeignKey('domains.id'))
|
||||
domain = relationship('Domain', primaryjoin='Archive.domain_id==Domain.id')
|
||||
|
||||
singlefile_path = Column(MediaPathType)
|
||||
readability_path = Column(MediaPathType)
|
||||
readability_json_path = Column(MediaPathType)
|
||||
readability_txt_path = Column(MediaPathType)
|
||||
screenshot_path = Column(MediaPathType)
|
||||
|
||||
url = Column(String)
|
||||
title = Column(String)
|
||||
archive_datetime = Column(TZDateTime)
|
||||
contents = Column(String)
|
||||
validated = Column(Boolean)
|
||||
|
||||
textsearch = Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, contents), 'D')'''))
|
||||
# Associated Files.
|
||||
singlefile_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
singlefile_file: File = relationship('File', primaryjoin='Archive.singlefile_path==File.path')
|
||||
readability_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
readability_file: File = relationship('File', primaryjoin='Archive.readability_path==File.path')
|
||||
readability_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
readability_json_file: File = relationship('File', primaryjoin='Archive.readability_json_path==File.path')
|
||||
readability_txt_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
readability_txt_file: File = relationship('File', primaryjoin='Archive.readability_txt_path==File.path')
|
||||
screenshot_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
screenshot_file: File = relationship('File', primaryjoin='Archive.screenshot_path==File.path')
|
||||
|
||||
def __repr__(self):
|
||||
return f'<Archive id={self.id} url={self.url} singlefile={self.singlefile_path}>'
|
||||
if self.domain:
|
||||
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))} ' \
|
||||
f'domain={self.domain.domain}>'
|
||||
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))}>'
|
||||
|
||||
def my_paths(self) -> Generator:
|
||||
if self.singlefile_path:
|
||||
yield self.singlefile_path.path
|
||||
yield self.singlefile_path
|
||||
if self.readability_path:
|
||||
yield self.readability_path.path
|
||||
yield self.readability_path
|
||||
if self.readability_json_path:
|
||||
yield self.readability_json_path.path
|
||||
yield self.readability_json_path
|
||||
if self.readability_txt_path:
|
||||
yield self.readability_txt_path.path
|
||||
yield self.readability_txt_path
|
||||
if self.screenshot_path:
|
||||
yield self.screenshot_path.path
|
||||
yield self.screenshot_path
|
||||
|
||||
def unlink(self):
|
||||
"""
|
||||
|
|
@ -73,6 +88,7 @@ class Archive(Base, ModelHelper):
|
|||
readability_txt_path=self.readability_txt_path,
|
||||
screenshot_path=self.screenshot_path,
|
||||
singlefile_path=self.singlefile_path,
|
||||
stem=self.singlefile_path.stem if self.singlefile_path else None,
|
||||
title=self.title,
|
||||
url=self.url,
|
||||
)
|
||||
|
|
@ -96,11 +112,103 @@ class Archive(Base, ModelHelper):
|
|||
Get a list of Archives that share my URL.
|
||||
"""
|
||||
session = Session.object_session(self)
|
||||
i = list(session.query(Archive).filter(
|
||||
alternatives = list(session.query(Archive).filter(
|
||||
Archive.id != self.id,
|
||||
Archive.url == self.url,
|
||||
).order_by(Archive.archive_datetime))
|
||||
return i
|
||||
return alternatives
|
||||
|
||||
def my_files(self) -> Generator[pathlib.Path, None, None]:
|
||||
"""Return all paths of this Archive. Returns nothing if all paths are None.""" # noqa
|
||||
if self.singlefile_path:
|
||||
yield self.singlefile_path
|
||||
if self.readability_path:
|
||||
yield self.readability_path
|
||||
if self.readability_txt_path:
|
||||
yield self.readability_txt_path
|
||||
if self.readability_json_path:
|
||||
yield self.readability_json_path
|
||||
if self.screenshot_path:
|
||||
yield self.screenshot_path
|
||||
|
||||
@staticmethod
|
||||
def find_by_path(path, session: Session) -> Base:
|
||||
archive = session.query(Archive).filter_by(singlefile_path=path).one_or_none()
|
||||
return archive
|
||||
|
||||
def read_singlefile_data(self):
|
||||
"""
|
||||
Read the start of the singlefile (if any) and decode the Archive information.
|
||||
"""
|
||||
match_url = re.compile(r'^\s+?url:\s+?(http.*)', re.MULTILINE)
|
||||
match_date = re.compile(r'^\s+?saved date:\s+?(.*)', re.MULTILINE)
|
||||
|
||||
if self.singlefile_path:
|
||||
with self.singlefile_path.open('rt') as fh:
|
||||
head = fh.read(500)
|
||||
if 'Page saved with SingleFile' not in head:
|
||||
return
|
||||
try:
|
||||
if match := match_url.findall(head):
|
||||
self.url = match[0].strip()
|
||||
except Exception as e:
|
||||
logger.error(f'Could not get URL from singlefile {self.singlefile_path}', exc_info=e)
|
||||
try:
|
||||
if match := match_date.findall(head):
|
||||
dt = match[0].strip()
|
||||
dt = ' '.join(dt.split(' ')[:5])
|
||||
# SingleFile uses GMT.
|
||||
dt = datetime.datetime.strptime(
|
||||
dt,
|
||||
'%a %b %d %Y %H:%M:%S' # Fri Jun 17 2022 19:24:52
|
||||
).replace(tzinfo=pytz.timezone('GMT'))
|
||||
self.archive_datetime = dt
|
||||
except Exception as e:
|
||||
logger.error(f'Could not get archive date from singlefile {self.singlefile_path}', exc_info=e)
|
||||
|
||||
def read_readability_data(self):
|
||||
"""Read the Readability JSON file, apply its contents to this record."""
|
||||
from modules.archive.lib import get_or_create_domain, get_title_from_html
|
||||
|
||||
readability_json_file = self.readability_json_file
|
||||
if not readability_json_file or not readability_json_file.path.is_file():
|
||||
raise InvalidArchive(f'Archive must have a readability json file: {self.singlefile_file.path}')
|
||||
|
||||
try:
|
||||
with readability_json_file.path.open() as fh:
|
||||
json_contents = json.load(fh)
|
||||
url = json_contents.get('url')
|
||||
title = json_contents.get('title')
|
||||
except Exception as e:
|
||||
raise InvalidArchive() from e
|
||||
|
||||
domain = None
|
||||
if url:
|
||||
domain = get_or_create_domain(Session.object_session(self), url)
|
||||
|
||||
if not title:
|
||||
title = get_title_from_html(self.singlefile_path.read_text())
|
||||
|
||||
self.url = url
|
||||
self.title = title
|
||||
self.domain_id = domain.id if domain else None
|
||||
|
||||
def validate(self):
|
||||
try:
|
||||
self.read_readability_data()
|
||||
self.read_singlefile_data()
|
||||
self.validated = True
|
||||
except Exception as e:
|
||||
logger.warning(f'Unable to validate {self}', exc_info=e)
|
||||
|
||||
@staticmethod
|
||||
def find_by_paths(paths, session):
|
||||
archives = list(session.query(Archive).filter(Archive.singlefile_path.in_(paths)))
|
||||
return archives
|
||||
|
||||
@property
|
||||
def primary_path(self):
|
||||
return self.singlefile_path
|
||||
|
||||
|
||||
class Domain(Base, ModelHelper):
|
||||
|
|
@ -119,3 +227,11 @@ class Domain(Base, ModelHelper):
|
|||
session = Session.object_session(self)
|
||||
session.execute('DELETE FROM archive WHERE domain_id=:id', dict(id=self.id))
|
||||
session.query(Domain).filter_by(id=self.id).delete()
|
||||
|
||||
@validates('domain')
|
||||
def validate_domain(self, key, value: str):
|
||||
if not isinstance(value, str):
|
||||
raise ValueError('Domain must be a string')
|
||||
if len(value.split('.')) < 2:
|
||||
raise ValueError(f'Domain must contain at least one "." domain={repr(value)}')
|
||||
return value
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ class ArchiveSearchRequest:
|
|||
domain: Optional[str] = None
|
||||
offset: Optional[int] = None
|
||||
limit: Optional[int] = None
|
||||
order_by: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
|
|||
|
|
@ -5,16 +5,22 @@ from http import HTTPStatus
|
|||
def check_results(test_client, data, ids):
|
||||
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.OK, response.json
|
||||
assert [i['id'] for i in response.json['archives']] == ids, \
|
||||
f'{response.json["archives"][0]["id"]}..{response.json["archives"][19]["id"]}'
|
||||
if ids:
|
||||
assert response.json['archives'], f'Expected {len(ids)} archives but did not receive any'
|
||||
assert [i['archive']['id'] for i in response.json['archives']] == ids, \
|
||||
f'{response.json["archives"][0]["archive"]["id"]}..{response.json["archives"]["archive"][19]["id"]}'
|
||||
|
||||
|
||||
def test_archives_search(test_session, archive_directory, archive_factory, test_client):
|
||||
"""Archives can be searched by their title and their contents."""
|
||||
# Search with no archives.
|
||||
check_results(test_client, {'search_str': 'foo'}, [])
|
||||
|
||||
archive_factory('example.com', 'https://example.com/one', 'my archive', 'foo bar qux')
|
||||
archive_factory('example.com', 'https://example.com/one', 'other archive', 'foo baz qux qux')
|
||||
archive_factory('example.org', title='archive third', contents='baz qux qux qux')
|
||||
archive_factory('example.org') # has no contents
|
||||
test_session.commit()
|
||||
|
||||
# 1 and 2 contain "foo".
|
||||
data = {'search_str': 'foo'}
|
||||
|
|
@ -44,15 +50,20 @@ def test_archives_search(test_session, archive_directory, archive_factory, test_
|
|||
data = {'search_str': 'third'}
|
||||
check_results(test_client, data, [3, ])
|
||||
|
||||
# All titles contain "qux", but they contain different amounts. They are ordered by the amount.
|
||||
# All contents contain "qux", but they contain different amounts. They are ordered by the amount.
|
||||
data = {'search_str': 'qux'}
|
||||
check_results(test_client, data, [3, 2, 1])
|
||||
|
||||
data = {'search_str': 'qux', 'order_by': 'bad order_by'}
|
||||
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.BAD_REQUEST
|
||||
|
||||
def test_search_offset(archive_factory, test_client):
|
||||
|
||||
def test_search_offset(test_session, archive_factory, test_client):
|
||||
"""Archive search can be offset."""
|
||||
for i in range(500):
|
||||
archive_factory('example.com', f'https://example.com/{i}', contents='foo bar')
|
||||
test_session.commit()
|
||||
|
||||
data = {'search_str': None, 'offset': 0}
|
||||
check_results(test_client, data, list(range(500, 480, -1)))
|
||||
|
|
@ -66,28 +77,87 @@ def test_search_offset(archive_factory, test_client):
|
|||
check_results(test_client, data, [])
|
||||
|
||||
|
||||
def test_archives_search_no_query(archive_factory, test_client):
|
||||
def test_archives_search_no_query(test_session, archive_factory, test_client):
|
||||
"""Archive Search API endpoint does not require data in the body."""
|
||||
# Add 100 random archives.
|
||||
for _ in range(100):
|
||||
archive_factory('example.com')
|
||||
test_session.commit()
|
||||
|
||||
# All archives are returned when no `search_str` is passed.
|
||||
request, response = test_client.post('/api/archive/search', content='{}')
|
||||
assert response.status_code == HTTPStatus.OK, response.json
|
||||
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
|
||||
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
|
||||
assert response.json['totals']['archives'] == 100
|
||||
|
||||
# All archives are from "example.com".
|
||||
data = dict(domain='example.com')
|
||||
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.OK, response.json
|
||||
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
|
||||
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
|
||||
assert response.json['totals']['archives'] == 100
|
||||
|
||||
# No archives are from "example.org".
|
||||
data = dict(domain='example.org')
|
||||
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.OK, response.json
|
||||
assert [i['id'] for i in response.json['archives']] == []
|
||||
assert [i['archive']['id'] for i in response.json['archives']] == []
|
||||
assert response.json['totals']['archives'] == 0
|
||||
|
||||
|
||||
def test_archive_and_domain_crud(test_session, test_client, archive_factory):
|
||||
"""
|
||||
Getting an Archive returns it's File. Testing deleting Archives.
|
||||
"""
|
||||
# Can get empty results.
|
||||
request, response = test_client.get(f'/api/archive/1')
|
||||
assert response.status_code == HTTPStatus.NOT_FOUND
|
||||
request, response = test_client.get(f'/api/archive/domains')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['domains'] == []
|
||||
|
||||
archive1 = archive_factory(domain='example.com')
|
||||
archive2 = archive_factory(domain='example.com')
|
||||
test_session.commit()
|
||||
|
||||
# Archive1 has Archive2 as alternative.
|
||||
request, response = test_client.get(f'/api/archive/{archive1.id}')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['file']['archive']['id'] == archive1.id
|
||||
assert response.json['alternatives'][0]['archive']['id'] == archive2.id
|
||||
|
||||
# Archive2 has Archive1 as alternative.
|
||||
request, response = test_client.get(f'/api/archive/{archive2.id}')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert archive2.id == response.json['file']['archive']['id']
|
||||
assert response.json['alternatives'][0]['archive']['id'] == archive1.id
|
||||
|
||||
# Only one domain.
|
||||
request, response = test_client.get(f'/api/archive/domains')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['domains'][0]['domain'] == 'example.com'
|
||||
|
||||
# Deleting works.
|
||||
request, response = test_client.delete(f'/api/archive/{archive1.id}')
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT
|
||||
|
||||
# Trying to delete again returns NOT_FOUND.
|
||||
request, response = test_client.delete(f'/api/archive/{archive1.id}')
|
||||
assert response.status_code == HTTPStatus.NOT_FOUND
|
||||
|
||||
# Can't get deleted Archive.
|
||||
request, response = test_client.get(f'/api/archive/{archive1.id}')
|
||||
assert response.status_code == HTTPStatus.NOT_FOUND
|
||||
|
||||
# Archive2 no longer has alternatives.
|
||||
request, response = test_client.get(f'/api/archive/{archive2.id}')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['file']['archive']['id'] == archive2.id
|
||||
assert response.json['alternatives'] == []
|
||||
|
||||
# No Archives, no Domains.
|
||||
request, response = test_client.delete(f'/api/archive/{archive2.id}')
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT
|
||||
request, response = test_client.get(f'/api/archive/domains')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['domains'] == []
|
||||
|
|
|
|||
|
|
@ -1,17 +1,18 @@
|
|||
import json
|
||||
import pathlib
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timedelta
|
||||
from http import HTTPStatus
|
||||
|
||||
import mock
|
||||
import pytest
|
||||
|
||||
from modules.archive import lib
|
||||
from modules.archive.lib import get_or_create_domain, _refresh_archives, \
|
||||
get_new_archive_files, delete_archive, do_archive, get_domains, get_archive_directory, group_archive_files, \
|
||||
ArchiveFiles
|
||||
from modules.archive.lib import get_or_create_domain, get_new_archive_files, delete_archive, do_archive, get_domains, \
|
||||
group_archive_files, ArchiveFiles
|
||||
from modules.archive.models import Archive, Domain
|
||||
from wrolpi.dates import local_timezone
|
||||
from wrolpi.db import get_db_session
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.media_path import MediaPath
|
||||
from wrolpi.root_api import CustomJSONEncoder
|
||||
from wrolpi.test.common import skip_circleci
|
||||
|
|
@ -35,8 +36,8 @@ def make_fake_request_archive(readability=True, screenshot=True, title=True):
|
|||
async def test_no_screenshot(test_session):
|
||||
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(screenshot=False)):
|
||||
archive = await do_archive('https://example.com')
|
||||
assert isinstance(archive.singlefile_path, MediaPath)
|
||||
assert isinstance(archive.readability_path, MediaPath)
|
||||
assert isinstance(archive.singlefile_path, pathlib.Path)
|
||||
assert isinstance(archive.readability_path, pathlib.Path)
|
||||
# Screenshot was empty
|
||||
assert archive.screenshot_path is None
|
||||
|
||||
|
|
@ -45,8 +46,8 @@ async def test_no_screenshot(test_session):
|
|||
async def test_no_readability(test_session):
|
||||
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(readability=False)):
|
||||
archive = await do_archive('https://example.com')
|
||||
assert isinstance(archive.singlefile_path, MediaPath)
|
||||
assert isinstance(archive.screenshot_path, MediaPath)
|
||||
assert isinstance(archive.singlefile_path, pathlib.Path)
|
||||
assert isinstance(archive.screenshot_path, pathlib.Path)
|
||||
# Readability empty
|
||||
assert archive.readability_path is None
|
||||
assert archive.readability_txt_path is None
|
||||
|
|
@ -66,7 +67,7 @@ async def test_relationships(test_session, test_directory):
|
|||
url = 'https://wrolpi.org:443'
|
||||
domain = get_or_create_domain(session, url)
|
||||
archive = Archive(
|
||||
singlefile_path=f'{test_directory.name}/wrolpi.org:443/foo',
|
||||
singlefile_file=File(path=test_directory / 'wrolpi.org:443/foo'),
|
||||
title='bar',
|
||||
url=url,
|
||||
domain_id=domain.id,
|
||||
|
|
@ -77,18 +78,7 @@ async def test_relationships(test_session, test_directory):
|
|||
assert archive.domain == domain
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validate_paths(test_session):
|
||||
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
|
||||
archive = await do_archive('https://example.com')
|
||||
try:
|
||||
with get_db_session(commit=True):
|
||||
archive.singlefile_path = 'asdf'
|
||||
except ValueError as e:
|
||||
assert 'relative' in str(e), f'Relative path error was not raised'
|
||||
|
||||
|
||||
def test_archive_refresh_deleted_archive(test_session, archive_directory, archive_factory):
|
||||
def test_archive_refresh_deleted_archive(test_client, test_session, archive_directory, archive_factory):
|
||||
"""
|
||||
Archives/Domains should be deleted when archive files are deleted.
|
||||
"""
|
||||
|
|
@ -97,58 +87,58 @@ def test_archive_refresh_deleted_archive(test_session, archive_directory, archiv
|
|||
archive3 = archive_factory('example.com')
|
||||
archive4 = archive_factory('example.org')
|
||||
archive5 = archive_factory()
|
||||
test_session.commit()
|
||||
|
||||
# Empty directories should be ignored.
|
||||
empty = archive_directory / 'empty'
|
||||
empty.mkdir()
|
||||
(archive_directory / 'empty').mkdir()
|
||||
|
||||
def check_counts(archive_count, domain_count):
|
||||
assert test_session.query(Archive).count() == archive_count
|
||||
assert test_session.query(Domain).count() == domain_count
|
||||
assert test_session.query(Archive).count() == archive_count, 'Archive count does not match'
|
||||
assert test_session.query(Domain).count() == domain_count, 'Domain count does not match'
|
||||
|
||||
# All 3 archives are already in the DB.
|
||||
check_counts(archive_count=5, domain_count=3)
|
||||
_refresh_archives()
|
||||
check_counts(archive_count=5, domain_count=3)
|
||||
# All 5 archives are already in the DB.
|
||||
check_counts(archive_count=5, domain_count=2)
|
||||
test_client.post('/api/files/refresh')
|
||||
check_counts(archive_count=5, domain_count=1)
|
||||
|
||||
# Delete archive2's files, it's the latest for 'https://example.com/1'
|
||||
for path in archive2.my_paths():
|
||||
for path in archive2.my_files():
|
||||
path.unlink()
|
||||
_refresh_archives()
|
||||
check_counts(archive_count=4, domain_count=3)
|
||||
test_client.post('/api/files/refresh')
|
||||
check_counts(archive_count=4, domain_count=1)
|
||||
|
||||
# Delete archive1's files, now the URL is empty.
|
||||
for path in archive1.my_paths():
|
||||
for path in archive1.my_files():
|
||||
path.unlink()
|
||||
_refresh_archives()
|
||||
check_counts(archive_count=3, domain_count=3)
|
||||
test_client.post('/api/files/refresh')
|
||||
check_counts(archive_count=3, domain_count=0)
|
||||
|
||||
# Delete archive3, now there is now example.com domain
|
||||
for path in archive3.my_paths():
|
||||
for path in archive3.my_files():
|
||||
path.unlink()
|
||||
_refresh_archives()
|
||||
check_counts(archive_count=2, domain_count=2)
|
||||
test_client.post('/api/files/refresh')
|
||||
check_counts(archive_count=2, domain_count=0)
|
||||
|
||||
# Delete all the rest of the archives
|
||||
for path in archive4.my_paths():
|
||||
for path in archive4.my_files():
|
||||
path.unlink()
|
||||
for path in archive5.my_paths():
|
||||
for path in archive5.my_files():
|
||||
path.unlink()
|
||||
_refresh_archives()
|
||||
test_client.post('/api/files/refresh')
|
||||
check_counts(archive_count=0, domain_count=0)
|
||||
|
||||
|
||||
def test_fills_contents_with_refresh(test_session, archive_factory, test_client):
|
||||
def test_fills_contents_with_refresh(test_client, test_session, archive_factory):
|
||||
"""Refreshing archives fills in any missing contents."""
|
||||
archive1 = archive_factory('example.com', 'https://example.com/one')
|
||||
archive2 = archive_factory('example.com', 'https://example.com/one')
|
||||
archive3 = archive_factory('example.org')
|
||||
# Title can be found in archive3's singlefile
|
||||
archive3 = archive_factory('example.org',
|
||||
singlefile_contents='<html> some stuff<title>last title</title> other stuff</html>')
|
||||
archive4 = archive_factory('example.org') # has no contents
|
||||
|
||||
# Clear the title, this will be filled in.
|
||||
archive1.title = archive2.title = archive3.title = archive4.title = None
|
||||
# Title can be found in archive3's singlefile
|
||||
archive3.singlefile_path.path.write_text('<html> some stuff<title>last title</title> other stuff</html>')
|
||||
|
||||
contents_title_archive = [
|
||||
('foo bar qux', 'my archive', archive1),
|
||||
|
|
@ -156,26 +146,26 @@ def test_fills_contents_with_refresh(test_session, archive_factory, test_client)
|
|||
('baz qux qux qux', None, archive3),
|
||||
]
|
||||
for contents, json_title, archive in contents_title_archive:
|
||||
with archive.readability_txt_path.path.open('wt') as fh:
|
||||
with archive.readability_txt_path.open('wt') as fh:
|
||||
fh.write(contents)
|
||||
if json_title:
|
||||
archive.readability_json_path.path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
|
||||
archive.readability_json_path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
|
||||
test_session.commit()
|
||||
|
||||
# Contents are empty. Archives are all missing the title.
|
||||
assert not archive1.contents
|
||||
assert not archive2.contents
|
||||
assert not archive3.contents
|
||||
assert not archive4.contents
|
||||
assert not archive1.singlefile_file.d_text
|
||||
assert not archive2.singlefile_file.d_text
|
||||
assert not archive3.singlefile_file.d_text
|
||||
assert not archive4.singlefile_file.d_text
|
||||
|
||||
# Fill the contents.
|
||||
_refresh_archives()
|
||||
test_client.post('/api/files/refresh')
|
||||
# The archives will be renamed with their title.
|
||||
archive1, archive2, archive3, archive4 = test_session.query(Archive).order_by(Archive.id)
|
||||
assert archive1.contents
|
||||
assert archive2.contents
|
||||
assert archive3.contents
|
||||
assert not archive4.contents
|
||||
assert archive1.singlefile_file.title
|
||||
assert archive2.singlefile_file.title
|
||||
assert archive3.singlefile_file.title
|
||||
assert not archive4.singlefile_file.d_text
|
||||
|
||||
# Missing title is filled.
|
||||
assert archive1.title == 'my archive'
|
||||
|
|
@ -207,6 +197,7 @@ def test_get_domains(test_session, archive_factory):
|
|||
archive1 = archive_factory('example.com')
|
||||
archive2 = archive_factory('example.com')
|
||||
archive3 = archive_factory('example.org')
|
||||
test_session.commit()
|
||||
|
||||
assert [i['domain'] for i in get_domains()] == ['example.com', 'example.org']
|
||||
|
||||
|
|
@ -231,22 +222,22 @@ async def test_new_archive(test_session, fake_now):
|
|||
# Everything is filled out.
|
||||
assert isinstance(archive1, Archive)
|
||||
assert archive1.archive_datetime is not None
|
||||
assert isinstance(archive1.singlefile_path, MediaPath)
|
||||
assert isinstance(archive1.readability_path, MediaPath)
|
||||
assert isinstance(archive1.readability_txt_path, MediaPath)
|
||||
assert isinstance(archive1.screenshot_path, MediaPath)
|
||||
assert isinstance(archive1.singlefile_path, pathlib.Path)
|
||||
assert isinstance(archive1.readability_path, pathlib.Path)
|
||||
assert isinstance(archive1.readability_txt_path, pathlib.Path)
|
||||
assert isinstance(archive1.screenshot_path, pathlib.Path)
|
||||
assert archive1.title == 'ジにてこちら'
|
||||
assert archive1.url is not None
|
||||
assert archive1.domain is not None
|
||||
|
||||
# The actual files were dumped and read correctly.
|
||||
with open(archive1.singlefile_path.path) as fh:
|
||||
with open(archive1.singlefile_path) as fh:
|
||||
assert fh.read() == '<html>\ntest single-file\nジにてこちら\n<title>some title</title></html>'
|
||||
with open(archive1.readability_path.path) as fh:
|
||||
with open(archive1.readability_path) as fh:
|
||||
assert fh.read() == '<html>test readability content</html>'
|
||||
with open(archive1.readability_txt_path.path) as fh:
|
||||
with open(archive1.readability_txt_path) as fh:
|
||||
assert fh.read() == '<html>test readability textContent</html>'
|
||||
with open(archive1.readability_json_path.path) as fh:
|
||||
with open(archive1.readability_json_path) as fh:
|
||||
assert json.load(fh) == {'title': 'ジにてこちら', 'url': 'https://example.com'}
|
||||
|
||||
fake_now(datetime(2000, 1, 2))
|
||||
|
|
@ -259,7 +250,7 @@ async def test_new_archive(test_session, fake_now):
|
|||
async def test_get_title_from_html(test_session, fake_now):
|
||||
fake_now(datetime(2000, 1, 1))
|
||||
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
|
||||
archive = await do_archive('example.com')
|
||||
archive = await do_archive('https://example.com')
|
||||
assert archive.title == 'ジにてこちら'
|
||||
|
||||
async def fake_request_archive(_):
|
||||
|
|
@ -273,7 +264,7 @@ async def test_get_title_from_html(test_session, fake_now):
|
|||
|
||||
fake_now(datetime(2000, 1, 2))
|
||||
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
|
||||
archive = await do_archive('example.com')
|
||||
archive = await do_archive('https://example.com')
|
||||
assert archive.title == 'some title'
|
||||
|
||||
async def fake_request_archive(_):
|
||||
|
|
@ -287,7 +278,7 @@ async def test_get_title_from_html(test_session, fake_now):
|
|||
|
||||
fake_now(datetime(2000, 1, 3))
|
||||
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
|
||||
archive = await do_archive('example.com')
|
||||
archive = await do_archive('https://example.com')
|
||||
assert archive.title is None
|
||||
|
||||
|
||||
|
|
@ -321,20 +312,20 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
|
|||
fake_now(datetime(2000, 1, 1))
|
||||
|
||||
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
|
||||
archive1 = await do_archive('example.com')
|
||||
archive1 = await do_archive('https://example.com')
|
||||
|
||||
assert archive1.title == 'ジにてこちら'
|
||||
|
||||
assert str(archive1.singlefile_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_ジにてこちら.html'
|
||||
assert str(archive1.readability_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_ジにてこちら.readability.html'
|
||||
assert str(archive1.readability_txt_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_ジにてこちら.readability.txt'
|
||||
assert str(archive1.readability_json_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_ジにてこちら.readability.json'
|
||||
assert str(archive1.screenshot_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_ジにてこちら.png'
|
||||
assert str(archive1.singlefile_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.html'
|
||||
assert str(archive1.readability_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.html'
|
||||
assert str(archive1.readability_txt_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.txt'
|
||||
assert str(archive1.readability_json_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.json'
|
||||
assert str(archive1.screenshot_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.png'
|
||||
|
||||
async def fake_request_archive(_):
|
||||
singlefile = '<html>\ntest single-file\nジにてこちら\n</html>' # no title in HTML
|
||||
|
|
@ -347,18 +338,18 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
|
|||
return singlefile, r, s
|
||||
|
||||
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
|
||||
archive2 = await do_archive('example.com')
|
||||
archive2 = await do_archive('https://example.com')
|
||||
|
||||
assert str(archive2.singlefile_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_NA.html'
|
||||
assert str(archive2.readability_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_NA.readability.html'
|
||||
assert str(archive2.readability_txt_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_NA.readability.txt'
|
||||
assert str(archive2.readability_json_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_NA.readability.json'
|
||||
assert str(archive2.screenshot_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_NA.png'
|
||||
assert str(archive2.singlefile_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_NA.html'
|
||||
assert str(archive2.readability_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_NA.readability.html'
|
||||
assert str(archive2.readability_txt_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_NA.readability.txt'
|
||||
assert str(archive2.readability_json_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_NA.readability.json'
|
||||
assert str(archive2.screenshot_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_NA.png'
|
||||
|
||||
async def fake_request_archive(_):
|
||||
singlefile = '<html>\ntest single-file\nジにてこちら\n<title>dangerous ;\\//_title</html></html>'
|
||||
|
|
@ -371,62 +362,101 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
|
|||
return singlefile, r, s
|
||||
|
||||
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
|
||||
archive3 = await do_archive('example.com')
|
||||
archive3 = await do_archive('https://example.com')
|
||||
|
||||
assert str(archive3.singlefile_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_dangerous ;_title.html'
|
||||
assert str(archive3.readability_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.html'
|
||||
assert str(archive3.readability_txt_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
|
||||
assert str(archive3.readability_json_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.json'
|
||||
assert str(archive3.screenshot_path.path.relative_to(test_directory)) == \
|
||||
'archive/2000-01-01-00-00-00_dangerous ;_title.png'
|
||||
assert str(archive3.singlefile_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.html'
|
||||
assert str(archive3.readability_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.html'
|
||||
assert str(archive3.readability_txt_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
|
||||
assert str(archive3.readability_json_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.json'
|
||||
assert str(archive3.screenshot_path.relative_to(test_directory)) == \
|
||||
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.png'
|
||||
|
||||
|
||||
@skip_circleci
|
||||
def test_refresh_archives(test_session, test_directory):
|
||||
"""Archives can be found and put in the database. A single Archive will have multiple files."""
|
||||
archive_directory = get_archive_directory()
|
||||
def test_refresh_archives(test_session, test_directory, test_client, make_files_structure):
|
||||
"""
|
||||
Archives can be found and put in the database. A single Archive will have multiple files.
|
||||
"""
|
||||
# The start of a typical singlefile html file.
|
||||
singlefile_contents = '''<!DOCTYPE html> <html lang="en"><!--
|
||||
Page saved with SingleFile
|
||||
url: https://example.com
|
||||
saved date: Mon May 16 2022 23:51:35 GMT+0000 (Coordinated Universal Time)
|
||||
--><head><meta charset="utf-8">'''
|
||||
|
||||
# Make some test files to refresh.
|
||||
example_dir = archive_directory / 'example.com'
|
||||
example_dir.mkdir()
|
||||
(example_dir / '2021-10-05 16:20:10.346823.html').touch() # renames to 2021-10-05-16-20-10_NA.html
|
||||
(example_dir / '2021-10-05 16:20:10.346823.png').touch()
|
||||
(example_dir / '2021-10-05 16:20:10.346823-readability.txt').touch()
|
||||
(example_dir / '2021-10-05 16:20:10.346823-readability.json').touch()
|
||||
(example_dir / '2021-10-05 16:20:10.346823-readability.html').touch()
|
||||
with (example_dir / '2021-10-05 16:20:10.346823-readability.json').open('wt') as fh:
|
||||
json.dump({'url': 'foo'}, fh)
|
||||
|
||||
# These should log an error because they are missing the singlefile path.
|
||||
(example_dir / '2021-10-05 16:20:10.readability.html').touch()
|
||||
(example_dir / '2021-10-05 16:20:11.readability.json').touch()
|
||||
# Bad files should also be ignored.
|
||||
(example_dir / 'random file').touch()
|
||||
(example_dir / '2021-10-05 16:20:10.346827-something.html').mkdir()
|
||||
(example_dir / '2021-10-05 16:20:10 no extension').touch()
|
||||
make_files_structure({
|
||||
# These are all for an individual Archive.
|
||||
'archive/example.com/2021-10-05 16:20:10.346823.html': singlefile_contents, # renames to 2021-10-05-16-20-10_NA
|
||||
'archive/example.com/2021-10-05 16:20:10.346823.png': None,
|
||||
'archive/example.com/2021-10-05 16:20:10.346823-readability.txt': 'article text contents',
|
||||
'archive/example.com/2021-10-05 16:20:10.346823-readability.json': '{"url": "https://example.com"}',
|
||||
'archive/example.com/2021-10-05 16:20:10.346823-readability.html': '<html></html>',
|
||||
# These should log an error because they are missing the singlefile path.
|
||||
'archive/example.com/2021-10-05 16:20:10.readability.html': '<html></html>',
|
||||
'archive/example.com/2021-10-05 16:20:10.readability.json': None,
|
||||
# Bad files should also be ignored.
|
||||
'archive/example.com/random file': 'hello',
|
||||
'archive/example.com/2021-10-05 16:20:10 no extension': None,
|
||||
})
|
||||
|
||||
# The single archive is found.
|
||||
_refresh_archives()
|
||||
with get_db_session() as session:
|
||||
assert session.query(Archive).count() == 1
|
||||
test_client.post('/api/files/refresh')
|
||||
assert test_session.query(Archive).count() == 1
|
||||
|
||||
# Running the refresh does not result in a new archive.
|
||||
_refresh_archives()
|
||||
with get_db_session() as session:
|
||||
assert session.query(Archive).count() == 1
|
||||
test_client.post('/api/files/refresh')
|
||||
assert test_session.query(Archive).count() == 1
|
||||
|
||||
# Archives file format was changed, lets check the new formats are found.
|
||||
(example_dir / '2021-10-05-16-20-11_The Title.html').touch()
|
||||
(example_dir / '2021-10-05-16-20-11_The Title.readability.json').write_text(json.dumps({'url': 'bar'}))
|
||||
_refresh_archives()
|
||||
make_files_structure({
|
||||
'archive/example.com/2021-10-05-16-20-11_The Title.html': '<html></html>',
|
||||
'archive/example.com/2021-10-05-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
|
||||
})
|
||||
test_client.post('/api/files/refresh')
|
||||
# The old formatted archive above is renamed.
|
||||
assert (example_dir / '2021-10-05-16-20-10_NA.html').is_file()
|
||||
with get_db_session() as session:
|
||||
assert session.query(Archive).count() == 2
|
||||
assert (test_directory / 'archive/example.com/2021-10-05-16-20-10_NA.html').is_file()
|
||||
assert test_session.query(Archive).count() == 2
|
||||
|
||||
content = json.dumps({'search_str': 'text', 'model': 'archive'})
|
||||
request, response = test_client.post('/api/files/search', content=content)
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert response.json['files'], 'No files matched "text"'
|
||||
assert response.json['files'][0]['model'] == 'archive', 'Returned file was not an archive'
|
||||
assert response.json['files'][0].get('path') == 'archive/example.com/2021-10-05-16-20-10_NA.html', \
|
||||
'Could not find readability text file containing "text"'
|
||||
assert response.json['files'][0]['archive']['archive_datetime'], 'Archive has no datetime'
|
||||
|
||||
|
||||
def test_refresh_archives_invalid_file(test_session, test_directory, test_client, make_files_structure):
|
||||
"""
|
||||
Invalid Archives should be removed during refresh.
|
||||
"""
|
||||
*_, bogus_path = make_files_structure({
|
||||
'2022-09-04-16-20-11_The Title.html': '<html></html>',
|
||||
'2022-09-04-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
|
||||
'bogus file': None,
|
||||
})
|
||||
test_client.post('/api/files/refresh')
|
||||
assert test_session.query(File).count() == 3 # Three files total.
|
||||
assert test_session.query(Archive).count() == 1 # Only one real archive.
|
||||
|
||||
# Create an invalid Archive, it should be removed during refresh.
|
||||
test_session.add(Archive(singlefile_path=bogus_path))
|
||||
# Claim the bogus file as an Archive, this should be removed.
|
||||
bogus_file: File = test_session.query(File).filter_by(path=bogus_path).one()
|
||||
bogus_file.model = 'archive'
|
||||
test_session.commit()
|
||||
assert test_session.query(File).count() == 3
|
||||
assert test_session.query(Archive).count() == 2
|
||||
|
||||
test_client.post('/api/files/refresh')
|
||||
assert test_session.query(File).count() == 3, 'More files in DB than we created'
|
||||
assert test_session.query(Archive).count() == 1, 'The bogus Archive was not removed'
|
||||
assert bogus_file.model is None, 'Model was not removed'
|
||||
|
||||
|
||||
def test_group_archive_files(test_directory):
|
||||
|
|
@ -488,9 +518,10 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
|
|||
archive1 = archive_factory('example.com', title='the title')
|
||||
archive2 = archive_factory('example.com')
|
||||
archive3 = archive_factory('example.org')
|
||||
assert archive1.singlefile_path.path.is_file()
|
||||
assert archive2.singlefile_path.path.is_file()
|
||||
assert archive3.singlefile_path.path.is_file()
|
||||
test_session.commit()
|
||||
assert archive1.singlefile_path.is_file()
|
||||
assert archive2.singlefile_path.is_file()
|
||||
assert archive3.singlefile_path.is_file()
|
||||
|
||||
example_com_directory = archive_directory / 'example.com'
|
||||
example_org_directory = archive_directory / 'example.org'
|
||||
|
|
@ -532,6 +563,25 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
|
|||
'example.org/2000-01-01-00-00-00_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.readability.json')),
|
||||
]
|
||||
# New Archive files still exist.
|
||||
assert all(i.is_file() for i in archive1.my_paths())
|
||||
assert all(i.is_file() for i in archive2.my_paths())
|
||||
assert all(i.is_file() for i in archive3.my_paths())
|
||||
assert all(i.is_file() for i in archive1.my_files())
|
||||
assert all(i.is_file() for i in archive2.my_files())
|
||||
assert all(i.is_file() for i in archive3.my_files())
|
||||
|
||||
|
||||
def test_archive_files(test_directory):
|
||||
af = lib.ArchiveFiles(
|
||||
singlefile=test_directory / 'archive/singlefile.html',
|
||||
)
|
||||
assert repr(af) == \
|
||||
"<ArchiveFiles singlefile='singlefile.html' readability=None readability_txt=None readability_json=None" \
|
||||
" screenshot=None>"
|
||||
|
||||
|
||||
def test_archive_order(test_session, test_directory, archive_factory):
|
||||
archive1 = archive_factory()
|
||||
archive2 = archive_factory()
|
||||
archive2.archive_datetime += timedelta(seconds=10)
|
||||
test_session.commit()
|
||||
|
||||
assert archive2 > archive1
|
||||
assert archive1 < archive2
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
from .api import bp
|
||||
|
|
@ -1,91 +0,0 @@
|
|||
import pathlib
|
||||
from typing import List
|
||||
from urllib.request import Request
|
||||
|
||||
from sanic import response
|
||||
from sanic_ext import validate
|
||||
from sanic_ext.extensions.openapi import openapi
|
||||
|
||||
from wrolpi.common import get_media_directory, api_param_limiter
|
||||
from wrolpi.errors import InvalidFile
|
||||
from wrolpi.root_api import get_blueprint, json_response
|
||||
from . import lib, schema
|
||||
|
||||
bp = get_blueprint('Files', '/api/files')
|
||||
|
||||
|
||||
def paths_to_files(paths: List[pathlib.Path]):
|
||||
"""Convert Paths to what the React UI expects."""
|
||||
media_directory = get_media_directory()
|
||||
new_files = []
|
||||
for path in paths:
|
||||
stat = path.stat()
|
||||
key = path.relative_to(media_directory)
|
||||
modified = stat.st_mtime
|
||||
if path.is_dir():
|
||||
key = f'{key}/'
|
||||
new_files.append(dict(
|
||||
key=key,
|
||||
modified=modified,
|
||||
url=key,
|
||||
name=path.name,
|
||||
))
|
||||
else:
|
||||
# A File should know it's size.
|
||||
new_files.append(dict(
|
||||
key=key,
|
||||
modified=modified,
|
||||
size=stat.st_size,
|
||||
url=key,
|
||||
name=path.name,
|
||||
))
|
||||
return new_files
|
||||
|
||||
|
||||
@bp.post('/')
|
||||
@openapi.definition(
|
||||
summary='List files in a directory',
|
||||
body=schema.FilesRequest,
|
||||
)
|
||||
@validate(schema.FilesRequest)
|
||||
async def get_files(_: Request, body: schema.FilesRequest):
|
||||
directories = body.directories or []
|
||||
|
||||
files = lib.list_files(directories)
|
||||
files = paths_to_files(files)
|
||||
return json_response({'files': files})
|
||||
|
||||
|
||||
@bp.post('/delete')
|
||||
@openapi.definition(
|
||||
summary='Delete a single file. Returns an error if WROL Mode is enabled.',
|
||||
body=schema.DeleteRequest,
|
||||
)
|
||||
@validate(schema.DeleteRequest)
|
||||
async def delete_file(_: Request, body: schema.DeleteRequest):
|
||||
if not body.file:
|
||||
raise InvalidFile('file cannot be empty')
|
||||
lib.delete_file(body.file)
|
||||
return response.empty()
|
||||
|
||||
|
||||
files_limit_limiter = api_param_limiter(100)
|
||||
files_offset_limiter = api_param_limiter(100, 0)
|
||||
|
||||
|
||||
@bp.post('/refresh')
|
||||
@openapi.description('Find and index all files')
|
||||
async def refresh(_: Request):
|
||||
lib.refresh_files()
|
||||
return response.empty()
|
||||
|
||||
|
||||
@bp.post('/search')
|
||||
@openapi.definition(
|
||||
summary='Search Files',
|
||||
body=schema.FilesSearchRequest,
|
||||
)
|
||||
@validate(schema.FilesSearchRequest)
|
||||
async def search_files(_: Request, body: schema.FilesSearchRequest):
|
||||
files, total = lib.search(body.search_str, body.limit, body.offset)
|
||||
return json_response(dict(files=files, totals=dict(files=total)))
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
import pathlib
|
||||
from typing import List, Callable
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_files_structure(test_directory) -> Callable[[List[str]], List[pathlib.Path]]:
|
||||
def create_files(paths: List[str]) -> List[pathlib.Path]:
|
||||
files = []
|
||||
for name in paths:
|
||||
path = test_directory / name
|
||||
if name.endswith('/'):
|
||||
path.mkdir()
|
||||
else:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.touch()
|
||||
files.append(path)
|
||||
return files
|
||||
|
||||
return create_files
|
||||
|
|
@ -1,185 +0,0 @@
|
|||
import asyncio
|
||||
import re
|
||||
import subprocess
|
||||
from functools import wraps
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg2
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from modules.files.models import File
|
||||
from wrolpi.cmd import which
|
||||
from wrolpi.common import get_media_directory, wrol_mode_check, walk, chunks, logger
|
||||
from wrolpi.dates import from_timestamp
|
||||
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
|
||||
from wrolpi.errors import InvalidFile
|
||||
from wrolpi.vars import PYTEST
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
__all__ = ['list_files', 'delete_file', 'split_file_name', 'upsert_file', 'refresh_files', 'search', 'get_mimetype']
|
||||
|
||||
|
||||
def filter_parent_directories(directories: List[Path]) -> List[Path]:
|
||||
"""
|
||||
Remove parent directories if their children are in the list.
|
||||
|
||||
>>> filter_parent_directories([Path('foo'), Path('foo/bar'), Path('baz')])
|
||||
[Path('foo/bar'), Path('baz')]
|
||||
"""
|
||||
unique_children = set()
|
||||
for directory in sorted(directories):
|
||||
for parent in directory.parents:
|
||||
# Remove any parent of this child.
|
||||
if parent in unique_children:
|
||||
unique_children.remove(parent)
|
||||
unique_children.add(directory)
|
||||
|
||||
# Restore the original order.
|
||||
new_directories = [i for i in directories if i in unique_children]
|
||||
return new_directories
|
||||
|
||||
|
||||
def list_files(directories: List[str]) -> List[Path]:
|
||||
"""
|
||||
List all files down to the directories provided. This includes all parent directories of the directories.
|
||||
"""
|
||||
media_directory = get_media_directory()
|
||||
|
||||
# Always display the media_directory files.
|
||||
paths = list(media_directory.iterdir())
|
||||
|
||||
if directories:
|
||||
directories = [media_directory / i for i in directories if i]
|
||||
directories = filter_parent_directories(directories)
|
||||
for directory in directories:
|
||||
directory = media_directory / directory
|
||||
for parent in directory.parents:
|
||||
is_relative_to = str(media_directory).startswith(str(parent))
|
||||
if parent == Path('.') or is_relative_to:
|
||||
continue
|
||||
paths.extend(parent.iterdir())
|
||||
paths.extend(directory.iterdir())
|
||||
|
||||
return paths
|
||||
|
||||
|
||||
@wrol_mode_check
|
||||
def delete_file(file: str):
|
||||
"""Delete a file in the media directory."""
|
||||
file = get_media_directory() / file
|
||||
if file.is_dir() or not file.is_file():
|
||||
raise InvalidFile(f'Invalid file {file}')
|
||||
file.unlink()
|
||||
|
||||
|
||||
FILE_NAME_REGEX = re.compile(r'[_ ]')
|
||||
|
||||
|
||||
def split_file_name(path: Path) -> List[str]:
|
||||
"""
|
||||
Split a file name into words.
|
||||
|
||||
>>> split_file_name(Path('foo.mp4'))
|
||||
['foo']
|
||||
>>> split_file_name(Path('foo bar.mp4'))
|
||||
['foo', 'bar']
|
||||
"""
|
||||
words = [str(i) for i in FILE_NAME_REGEX.split(path.stem)]
|
||||
return words
|
||||
|
||||
|
||||
FILE_BIN = which('file', '/usr/bin/file')
|
||||
|
||||
|
||||
def get_mimetype(path: Path) -> str:
|
||||
"""Get the mimetype using the builtin `file` command."""
|
||||
cmd = (FILE_BIN, '--mime-type', str(path.absolute()))
|
||||
output = subprocess.check_output(cmd)
|
||||
output = output.decode()
|
||||
mimetype = output.split(' ')[-1].strip()
|
||||
return mimetype
|
||||
|
||||
|
||||
def upsert_file(path: Path, session: Session) -> File:
|
||||
"""Update/insert a File in the DB. Gather metadata about it."""
|
||||
file = session.query(File).filter_by(path=path).one_or_none()
|
||||
if not file:
|
||||
file = File(path=path)
|
||||
session.add(file)
|
||||
if not file.mimetype:
|
||||
file.mimetype = get_mimetype(path)
|
||||
if not file.title:
|
||||
file.title = split_file_name(path)
|
||||
if not file.size or not file.modification_datetime:
|
||||
stat = path.stat()
|
||||
file.size = stat.st_size
|
||||
file.modification_datetime = file.modification_datetime or from_timestamp(stat.st_mtime)
|
||||
|
||||
return file
|
||||
|
||||
|
||||
def _refresh_files():
|
||||
"""Find and index all files"""
|
||||
logger.info('Refreshing Files')
|
||||
# Mark all files as unverified. Any records that are unverified after refresh will be deleted.
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('UPDATE file SET idempotency=null') # noqa
|
||||
|
||||
paths = filter(lambda i: i.is_file(), walk(get_media_directory()))
|
||||
idempotency = str(uuid4())
|
||||
for chunk in chunks(paths, 20):
|
||||
with get_db_session(commit=True) as session:
|
||||
for path in chunk:
|
||||
file = upsert_file(path, session)
|
||||
file.idempotency = idempotency
|
||||
|
||||
# Remove any records where the file no longer exists.
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('DELETE FROM file WHERE idempotency IS NULL')
|
||||
|
||||
logger.info('Done refreshing Files')
|
||||
|
||||
|
||||
@wraps(_refresh_files)
|
||||
def refresh_files():
|
||||
"""Schedule a refresh task if not testing. If testing, do a synchronous refresh."""
|
||||
if PYTEST:
|
||||
return _refresh_files()
|
||||
|
||||
async def _():
|
||||
return _refresh_files()
|
||||
|
||||
asyncio.create_task(_())
|
||||
|
||||
|
||||
def search(search_str: str, limit: int, offset: int):
|
||||
"""
|
||||
Search the "title" of each file. The title is the words in the file's stem.
|
||||
"""
|
||||
with get_db_curs() as curs:
|
||||
stmt = '''
|
||||
SELECT id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total
|
||||
FROM file
|
||||
WHERE textsearch @@ websearch_to_tsquery(%(search_str)s)
|
||||
OFFSET %(offset)s LIMIT %(limit)s
|
||||
'''
|
||||
params = dict(search_str=search_str, offset=offset, limit=limit)
|
||||
curs.execute(stmt, params)
|
||||
|
||||
try:
|
||||
results = list(map(dict, curs.fetchall()))
|
||||
except psycopg2.ProgrammingError:
|
||||
# No files.
|
||||
return [], 0
|
||||
|
||||
total = results[0]['total'] if results else 0
|
||||
ranked_ids = [i['id'] for i in results]
|
||||
|
||||
with get_db_session() as session:
|
||||
results = get_ranked_models(ranked_ids, File, session)
|
||||
results = [i.__json__() for i in results]
|
||||
|
||||
return results, total
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
from sqlalchemy import Integer, Column, String, Computed
|
||||
from sqlalchemy.orm import deferred
|
||||
|
||||
from wrolpi.common import Base, ModelHelper, get_media_directory, tsvector
|
||||
from wrolpi.dates import TZDateTime
|
||||
from wrolpi.media_path import MediaPathType
|
||||
|
||||
|
||||
class File(ModelHelper, Base):
|
||||
__tablename__ = 'file'
|
||||
id = Column(Integer, primary_key=True)
|
||||
path = Column(MediaPathType)
|
||||
|
||||
idempotency = Column(String)
|
||||
mimetype = Column(String)
|
||||
modification_datetime = Column(TZDateTime)
|
||||
size = Column(Integer)
|
||||
title = deferred(Column(String)) # used by textsearch column
|
||||
|
||||
textsearch = deferred(
|
||||
Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A'::"char")''')))
|
||||
|
||||
def __repr__(self):
|
||||
return f'<File id={self.id} path={self.path.relative_to(get_media_directory())} mime={self.mimetype}>'
|
||||
|
||||
def __json__(self):
|
||||
path = self.path.path.relative_to(get_media_directory())
|
||||
d = dict(
|
||||
id=self.id,
|
||||
mimetype=self.mimetype,
|
||||
modified=self.modification_datetime, # React browser expects this.
|
||||
path=path,
|
||||
size=self.size,
|
||||
key=path,
|
||||
)
|
||||
return d
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class FilesRequest:
|
||||
directories: List[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DeleteRequest:
|
||||
file: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class FilesSearchRequest:
|
||||
search_str: str
|
||||
limit: Optional[int] = 20
|
||||
offset: Optional[int] = 0
|
||||
|
|
@ -1,159 +0,0 @@
|
|||
import json
|
||||
from http import HTTPStatus
|
||||
from itertools import zip_longest
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
from wrolpi.errors import API_ERRORS, WROLModeEnabled
|
||||
from wrolpi.test.common import assert_dict_contains
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
def test_list_files_api(test_client, make_files_structure, test_directory):
|
||||
files = [
|
||||
'archives/bar.txt',
|
||||
'archives/baz/bar.txt',
|
||||
'archives/baz/foo.txt',
|
||||
'archives/foo.txt',
|
||||
'empty directory/',
|
||||
'videos/other video.mp4',
|
||||
'videos/some video.mp4',
|
||||
]
|
||||
files = make_files_structure(files)
|
||||
files[0].write_text('bar contents')
|
||||
|
||||
def check_get_files(directories, expected_files):
|
||||
request, response = test_client.post('/api/files', content=json.dumps({'directories': directories}))
|
||||
assert not response.json.get('errors')
|
||||
results = sorted(response.json['files'], key=lambda i: i['key'])
|
||||
for f1, f2 in zip_longest(results, expected_files):
|
||||
if f1 is None or f2 is None:
|
||||
assert f1 == f2
|
||||
for key in f2.keys():
|
||||
assert f1[key] == f2[key], f'{f1} != {f2}'
|
||||
|
||||
# Requesting no directories results in the top-level results.
|
||||
expected = [dict(key='archives/'), dict(key='empty directory/'), dict(key='videos/')]
|
||||
check_get_files([], expected)
|
||||
# empty directory is empty
|
||||
check_get_files(['empty directory'], expected)
|
||||
|
||||
expected = [
|
||||
dict(key='archives/'),
|
||||
dict(key='archives/bar.txt', size=12),
|
||||
dict(key='archives/baz/'),
|
||||
dict(key='archives/foo.txt'),
|
||||
dict(key='empty directory/'),
|
||||
dict(key='videos/'),
|
||||
]
|
||||
check_get_files(['archives'], expected)
|
||||
|
||||
# Sub-directories are supported.
|
||||
expected = [
|
||||
dict(key='archives/'),
|
||||
dict(key='archives/bar.txt', size=12),
|
||||
dict(key='archives/baz/'),
|
||||
dict(key='archives/baz/bar.txt'),
|
||||
dict(key='archives/baz/foo.txt'),
|
||||
dict(key='archives/foo.txt'),
|
||||
dict(key='empty directory/'),
|
||||
dict(key='videos/'),
|
||||
]
|
||||
check_get_files(['archives/baz'], expected)
|
||||
|
||||
expected = [
|
||||
dict(key='archives/'),
|
||||
dict(key='archives/bar.txt', size=12),
|
||||
dict(key='archives/baz/'),
|
||||
dict(key='archives/foo.txt'),
|
||||
dict(key='empty directory/'),
|
||||
dict(key='videos/'),
|
||||
dict(key='videos/other video.mp4'),
|
||||
dict(key='videos/some video.mp4'),
|
||||
]
|
||||
check_get_files(['archives', 'videos'], expected)
|
||||
# Order does not matter.
|
||||
check_get_files(['archives', 'videos', 'empty directory'], expected)
|
||||
check_get_files(['archives', 'empty directory', 'videos'], expected)
|
||||
|
||||
|
||||
def test_delete_file(test_client, make_files_structure, test_directory):
|
||||
files = ['bar.txt', 'baz/']
|
||||
make_files_structure(files)
|
||||
|
||||
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'bar.txt'}))
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT
|
||||
assert not (test_directory / 'bar.txt').is_file()
|
||||
assert (test_directory / 'baz').is_dir()
|
||||
|
||||
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'baz'}))
|
||||
assert response.status_code == HTTPStatus.BAD_REQUEST
|
||||
assert not (test_directory / 'bar.txt').is_file()
|
||||
assert (test_directory / 'baz').is_dir()
|
||||
|
||||
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'bad file'}))
|
||||
assert response.status_code == HTTPStatus.BAD_REQUEST
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'file', [
|
||||
'',
|
||||
]
|
||||
)
|
||||
def test_delete_invalid_file(test_client, file):
|
||||
with mock.patch('modules.files.api.lib.delete_file') as mock_delete_file:
|
||||
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': file}))
|
||||
assert response.status_code == HTTPStatus.BAD_REQUEST
|
||||
mock_delete_file.assert_not_called()
|
||||
|
||||
|
||||
def test_delete_wrol_mode(test_client):
|
||||
"""
|
||||
Can't delete a file when WROL Mode is enabled.
|
||||
"""
|
||||
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
|
||||
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'foo'}))
|
||||
assert response.status_code == HTTPStatus.FORBIDDEN
|
||||
assert response.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
|
||||
|
||||
def do_search(test_client, search_str, total, expected):
|
||||
content = json.dumps({'search_str': search_str})
|
||||
request, response = test_client.post('/api/files/search', content=content)
|
||||
assert response.json['totals']['files'] == total
|
||||
for file, expected in zip_longest(response.json['files'], expected):
|
||||
assert_dict_contains(file, expected)
|
||||
# FileBrowser in React requires a key. We use the path.
|
||||
assert file['path'] == file['key']
|
||||
|
||||
|
||||
def test_files_search(test_session, test_client, make_files_structure):
|
||||
# You can search an empty directory.
|
||||
do_search(test_client, 'nothing', 0, [])
|
||||
|
||||
# Create files in the temporary directory. Add some contents so the mimetype can be tested.
|
||||
files = [
|
||||
'foo_is_the_name.txt',
|
||||
'archives/bar.txt',
|
||||
'baz.mp4',
|
||||
'baz baz two.mp4'
|
||||
]
|
||||
foo, bar, baz, baz2 = make_files_structure(files)
|
||||
foo.write_text('foo contents')
|
||||
bar.write_text('the bar contents')
|
||||
baz.write_bytes((PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4').read_bytes())
|
||||
baz2.write_bytes((PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4').read_bytes())
|
||||
|
||||
# Refresh so files can be searched.
|
||||
request, response = test_client.post('/api/files/refresh')
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT
|
||||
|
||||
do_search(test_client, 'foo', 1, [dict(path='foo_is_the_name.txt', mimetype='text/plain', size=12)])
|
||||
do_search(test_client, 'bar', 1, [dict(path='archives/bar.txt', mimetype='text/plain', size=16)])
|
||||
do_search(test_client, 'baz', 2, [
|
||||
dict(path='baz.mp4', mimetype='video/mp4', size=1055736),
|
||||
dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736),
|
||||
])
|
||||
do_search(test_client, 'two', 1, [dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736)])
|
||||
do_search(test_client, 'nothing', 0, [])
|
||||
|
|
@ -1,207 +0,0 @@
|
|||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import List, Iterable
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from modules.files import lib
|
||||
from modules.files.models import File
|
||||
from wrolpi.errors import InvalidFile
|
||||
from wrolpi.media_path import MediaPathType
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
def assert_files(session: Session, expected):
|
||||
files = {str(i.path.relative) for i in session.query(File).all()}
|
||||
assert files == set(expected)
|
||||
|
||||
|
||||
def test_list_files(make_files_structure, test_directory):
|
||||
files = [
|
||||
'archives/bar.txt',
|
||||
'archives/baz/bar.txt',
|
||||
'archives/baz/foo.txt',
|
||||
'archives/foo.txt',
|
||||
'empty directory/',
|
||||
'videos/other video.mp4',
|
||||
'videos/some video.mp4',
|
||||
]
|
||||
make_files_structure(files)
|
||||
|
||||
def check_list_files(path, expected):
|
||||
result = lib.list_files(path)
|
||||
result = sorted(result)
|
||||
assert [str(i.relative_to(test_directory)) for i in result] == expected
|
||||
|
||||
check_list_files([], ['archives', 'empty directory', 'videos'])
|
||||
# Falsey directories are ignored.
|
||||
check_list_files([None, ''], ['archives', 'empty directory', 'videos'])
|
||||
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
|
||||
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
|
||||
# Trailing slash is ignored.
|
||||
check_list_files(['empty directory/'], ['archives', 'empty directory', 'videos'])
|
||||
|
||||
check_list_files(['archives'], [
|
||||
'archives',
|
||||
'archives/bar.txt',
|
||||
'archives/baz',
|
||||
'archives/foo.txt',
|
||||
'empty directory',
|
||||
'videos',
|
||||
])
|
||||
check_list_files(['archives/baz'], [
|
||||
'archives',
|
||||
'archives/bar.txt',
|
||||
'archives/baz',
|
||||
'archives/baz/bar.txt',
|
||||
'archives/baz/foo.txt',
|
||||
'archives/foo.txt',
|
||||
'empty directory',
|
||||
'videos',
|
||||
])
|
||||
# Including duplicate directories does not change results.
|
||||
check_list_files(['archives/baz', 'archives'], [
|
||||
'archives',
|
||||
'archives/bar.txt',
|
||||
'archives/baz',
|
||||
'archives/baz/bar.txt',
|
||||
'archives/baz/foo.txt',
|
||||
'archives/foo.txt',
|
||||
'empty directory',
|
||||
'videos',
|
||||
])
|
||||
check_list_files(['videos'], [
|
||||
'archives',
|
||||
'empty directory',
|
||||
'videos',
|
||||
'videos/other video.mp4',
|
||||
'videos/some video.mp4',
|
||||
])
|
||||
# All files and directories are listed.
|
||||
check_list_files(['videos', 'archives/baz'], [
|
||||
'archives',
|
||||
'archives/bar.txt',
|
||||
'archives/baz',
|
||||
'archives/baz/bar.txt',
|
||||
'archives/baz/foo.txt',
|
||||
'archives/foo.txt',
|
||||
'empty directory',
|
||||
'videos',
|
||||
'videos/other video.mp4',
|
||||
'videos/some video.mp4',
|
||||
])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'directories,expected', [
|
||||
([Path('foo')], [Path('foo')]),
|
||||
([Path('foo'), Path('bar')], [Path('foo'), Path('bar')]),
|
||||
([Path('foo'), Path('foo/bar'), Path('baz')], [Path('foo/bar'), Path('baz')]),
|
||||
([Path('foo/bar'), Path('foo'), Path('baz')], [Path('foo/bar'), Path('baz')]),
|
||||
([Path('baz'), Path('foo/bar'), Path('foo')], [Path('baz'), Path('foo/bar')]),
|
||||
([Path('foo/bar'), Path('foo'), Path('foo/bar/baz')], [Path('foo/bar/baz')]),
|
||||
]
|
||||
)
|
||||
def test_filter_parent_directories(directories, expected):
|
||||
assert lib.filter_parent_directories(directories) == expected
|
||||
|
||||
|
||||
def test_delete_file(make_files_structure, test_directory):
|
||||
"""
|
||||
File in the media directory can be deleted.
|
||||
"""
|
||||
files = [
|
||||
'archives/foo.txt',
|
||||
'bar.txt',
|
||||
'baz/',
|
||||
]
|
||||
make_files_structure(files)
|
||||
|
||||
lib.delete_file('bar.txt')
|
||||
assert (test_directory / 'archives/foo.txt').is_file()
|
||||
assert not (test_directory / 'bar.txt').is_file()
|
||||
assert (test_directory / 'baz').is_dir()
|
||||
|
||||
lib.delete_file('archives/foo.txt')
|
||||
assert not (test_directory / 'archives/foo.txt').is_file()
|
||||
assert not (test_directory / 'bar.txt').is_file()
|
||||
|
||||
with pytest.raises(InvalidFile):
|
||||
lib.delete_file('baz')
|
||||
with pytest.raises(InvalidFile):
|
||||
lib.delete_file('does not exist')
|
||||
|
||||
assert (test_directory / 'baz').is_dir()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'path,expected',
|
||||
[
|
||||
('foo.mp4', ['foo']),
|
||||
('foo bar.txt', ['foo', 'bar']),
|
||||
('foo_bar.mp4', ['foo', 'bar']),
|
||||
('foo-bar', ['foo-bar']),
|
||||
('123 foo bar', ['123', 'foo', 'bar']),
|
||||
('123foo bar', ['123foo', 'bar']),
|
||||
]
|
||||
)
|
||||
def test_split_file_name(path, expected):
|
||||
assert lib.split_file_name(Path(path)) == expected
|
||||
|
||||
|
||||
def test_refresh_files(test_session, make_files_structure, test_directory):
|
||||
files = make_files_structure([
|
||||
'foo.txt',
|
||||
'bar.txt',
|
||||
'baz.txt',
|
||||
])
|
||||
foo, bar, baz = files
|
||||
|
||||
def get_relative_strs(files_: Iterable[MediaPathType]) -> List[str]:
|
||||
return sorted([str(i.path.path.relative_to(test_directory)) for i in files_])
|
||||
|
||||
lib.refresh_files()
|
||||
results = test_session.query(File)
|
||||
assert get_relative_strs(results) == ['bar.txt', 'baz.txt', 'foo.txt']
|
||||
|
||||
baz.unlink()
|
||||
|
||||
lib.refresh_files()
|
||||
results = test_session.query(File)
|
||||
assert get_relative_strs(results) == ['bar.txt', 'foo.txt']
|
||||
|
||||
foo.unlink()
|
||||
bar.unlink()
|
||||
|
||||
lib.refresh_files()
|
||||
results = test_session.query(File)
|
||||
assert get_relative_strs(results) == []
|
||||
|
||||
|
||||
def test_mime_type(test_session, make_files_structure, test_directory):
|
||||
"""Files module uses the `file` command to get the mimetype of each file."""
|
||||
from PIL import Image
|
||||
|
||||
foo, bar, baz, empty = make_files_structure([
|
||||
'dir/foo text.txt',
|
||||
'dir/bar.jpeg',
|
||||
'dir/baz.mp4',
|
||||
'dir/empty',
|
||||
])
|
||||
foo.write_text('some text')
|
||||
Image.new('RGB', (25, 25), color='grey').save(bar)
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', baz)
|
||||
|
||||
lib.refresh_files()
|
||||
assert_files(test_session, ['dir/foo text.txt', 'dir/bar.jpeg', 'dir/baz.mp4', 'dir/empty'])
|
||||
|
||||
foo = test_session.query(File).filter_by(path=f'{test_directory}/dir/foo text.txt').one()
|
||||
bar = test_session.query(File).filter_by(path=f'{test_directory}/dir/bar.jpeg').one()
|
||||
baz = test_session.query(File).filter_by(path=f'{test_directory}/dir/baz.mp4').one()
|
||||
empty = test_session.query(File).filter_by(path=f'{test_directory}/dir/empty').one()
|
||||
|
||||
assert foo.mimetype == 'text/plain'
|
||||
assert bar.mimetype == 'image/jpeg'
|
||||
assert baz.mimetype == 'video/mp4'
|
||||
assert empty.mimetype == 'inode/x-empty'
|
||||
|
|
@ -8,7 +8,7 @@ from sanic_ext.extensions.openapi import openapi
|
|||
from wrolpi.common import run_after, recursive_map
|
||||
from wrolpi.errors import ValidationError
|
||||
from wrolpi.root_api import get_blueprint, json_response
|
||||
from . import common, inventory, schema
|
||||
from modules.inventory import common, inventory, schema
|
||||
|
||||
NAME = 'inventory'
|
||||
|
||||
|
|
@ -16,25 +16,25 @@ bp = get_blueprint('Inventory', '/api/inventory')
|
|||
|
||||
|
||||
@bp.get('/categories')
|
||||
def _get_categories(_: Request):
|
||||
def get_categories(_: Request):
|
||||
categories = inventory.get_categories()
|
||||
return json_response(dict(categories=categories))
|
||||
|
||||
|
||||
@bp.get('/brands')
|
||||
def _get_brands(_: Request):
|
||||
def get_brands(_: Request):
|
||||
brands = inventory.get_brands()
|
||||
return json_response(dict(brands=brands))
|
||||
|
||||
|
||||
@bp.get('/')
|
||||
def inventories_get(_: Request):
|
||||
def get_inventories(_: Request):
|
||||
inventories = inventory.get_inventories()
|
||||
return json_response(dict(inventories=inventories))
|
||||
|
||||
|
||||
@bp.get('/<inventory_id:int>')
|
||||
def inventory_get(_: Request, inventory_id: int):
|
||||
def get_inventory(_: Request, inventory_id: int):
|
||||
by_category = common.get_inventory_by_category(inventory_id)
|
||||
by_subcategory = common.get_inventory_by_subcategory(inventory_id)
|
||||
by_name = common.get_inventory_by_name(inventory_id)
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ def get_categories() -> List[Tuple[str, str]]:
|
|||
"""
|
||||
categories = DEFAULT_CATEGORIES.copy()
|
||||
with get_db_curs() as curs:
|
||||
curs.execute('SELECT DISTINCT subcategory, category FROM item ORDER BY 1, 2')
|
||||
curs.execute("SELECT DISTINCT COALESCE(subcategory, ''), COALESCE(category, '') FROM item ORDER BY 1, 2")
|
||||
try:
|
||||
db_categories = curs.fetchall()
|
||||
db_categories = [tuple(i) for i in db_categories]
|
||||
|
|
|
|||
|
|
@ -84,6 +84,15 @@ def test_inventory_api(test_session, test_client):
|
|||
inventory = response.json['inventories'][0]
|
||||
inventory_id = inventory['id']
|
||||
|
||||
request, response = test_client.get(f'/api/inventory/{inventory_id}')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
|
||||
request, response = test_client.get('/api/inventory/categories')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
|
||||
request, response = test_client.get('/api/inventory/brands')
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
|
||||
inventory = dict(name='new name')
|
||||
request, response = test_client.put(f'/api/inventory/{inventory_id}', content=json.dumps(inventory))
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code
|
||||
|
|
|
|||
|
|
@ -35,5 +35,5 @@ class MapFile(Base, ModelHelper):
|
|||
'path': self.path,
|
||||
'imported': self.imported,
|
||||
'size': self.size,
|
||||
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path.path, self.size)),
|
||||
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path, self.size)),
|
||||
}
|
||||
|
|
|
|||
21
modules/map/test/test_lib.py
Normal file
21
modules/map/test/test_lib.py
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
from unittest import mock
|
||||
|
||||
from modules.map import lib
|
||||
|
||||
|
||||
def test_get_map_paths(test_directory):
|
||||
pbf_dir = test_directory / 'map/pbf'
|
||||
pbf_dir.mkdir(parents=True)
|
||||
|
||||
pbf = pbf_dir / 'map.osm.pbf'
|
||||
pbf.touch()
|
||||
dump = pbf_dir / 'map.dump'
|
||||
dump.touch()
|
||||
|
||||
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
|
||||
mock_check_output.return_value = b'OpenStreetMap Protocolbuffer Binary Format'
|
||||
assert lib.get_map_paths() == [pbf, ]
|
||||
|
||||
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
|
||||
mock_check_output.return_value = b'PostgreSQL custom database dump'
|
||||
assert lib.get_map_paths() == [dump, ]
|
||||
|
|
@ -1 +1,123 @@
|
|||
from modules.videos import downloader
|
||||
import json
|
||||
import pathlib
|
||||
from abc import ABC
|
||||
from typing import List, Dict
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from modules.videos.models import Video, Channel
|
||||
from wrolpi.captions import read_captions, extract_captions
|
||||
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
|
||||
from wrolpi.db import get_db_curs
|
||||
from wrolpi.files.indexers import Indexer, register_indexer
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PYTEST
|
||||
from .downloader import video_downloader # Import downloaders so they are registered.
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
__all__ = ['video_modeler', 'VideoIndexer']
|
||||
|
||||
|
||||
@register_modeler
|
||||
def video_modeler(groups: Dict[str, List[File]], session: Session):
|
||||
# Only process groups that contain a video file.
|
||||
count = 0
|
||||
local_groups = groups.copy()
|
||||
for stem, group in local_groups.items():
|
||||
video_file = next((i for i in group if i.mimetype.split('/')[0] == 'video' and i.path.suffix != '.part'), None)
|
||||
if not video_file:
|
||||
# Not a video group.
|
||||
continue
|
||||
|
||||
session.flush(group)
|
||||
poster_file = next((i for i in group if i.mimetype.split('/')[0] == 'image'), None)
|
||||
caption_file = next((i for i in group if i.path.name.endswith('.en.vtt') or i.path.name.endswith('.en.srt')),
|
||||
None)
|
||||
info_json_file = next((i for i in group if i.path.name.endswith('.info.json')), None)
|
||||
|
||||
if poster_file:
|
||||
poster_file.associated = True
|
||||
poster_file.do_index()
|
||||
if caption_file:
|
||||
caption_file.associated = True
|
||||
caption_file.do_index()
|
||||
if info_json_file:
|
||||
info_json_file.associated = True
|
||||
info_json_file.do_index()
|
||||
|
||||
video: Video = Video.upsert(video_file, session)
|
||||
video.poster_file = poster_file
|
||||
video.caption_file = caption_file
|
||||
video.info_json_file = info_json_file
|
||||
|
||||
video.video_file.do_index()
|
||||
video.validate(session)
|
||||
|
||||
# Remove this group, it will not be processed by other modelers.
|
||||
del groups[stem]
|
||||
|
||||
count += 1
|
||||
if not count % 100:
|
||||
# Commit occasionally.
|
||||
session.commit()
|
||||
|
||||
|
||||
@register_after_refresh
|
||||
@limit_concurrent(1)
|
||||
def video_cleanup():
|
||||
# Claim all Videos in a Channel's directory for that Channel.
|
||||
logger.info('Claiming Videos for their Channels')
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('''
|
||||
UPDATE video v
|
||||
SET channel_id = c.id
|
||||
FROM channel c
|
||||
WHERE
|
||||
v.video_path LIKE c.directory || '/%'::VARCHAR
|
||||
AND v.channel_id IS NULL
|
||||
''')
|
||||
|
||||
|
||||
EXTRACT_SUBTITLES = False
|
||||
|
||||
|
||||
@register_indexer('video')
|
||||
class VideoIndexer(Indexer, ABC):
|
||||
"""Handles video files like mp4/ogg."""
|
||||
|
||||
@staticmethod
|
||||
def get_description(file):
|
||||
video_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
|
||||
info_json_path = video_path.with_suffix('.info.json')
|
||||
if info_json_path.is_file():
|
||||
with info_json_path.open('rt') as fh:
|
||||
try:
|
||||
return json.load(fh)['description']
|
||||
except Exception:
|
||||
if not PYTEST:
|
||||
logger.warning(f'Video info json file exists, but cannot get description. {file}')
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def create_index(cls, file):
|
||||
"""
|
||||
Index the video file and it's associated files.
|
||||
|
||||
a = title.
|
||||
b = <empty>
|
||||
c = description from info json.
|
||||
d = captions from info json, or extracted from video file.
|
||||
"""
|
||||
a = cls.get_title(file)
|
||||
c = cls.get_description(file)
|
||||
|
||||
# Detect the associated caption file.
|
||||
caption_path = file.path.with_suffix('.en.vtt')
|
||||
d = None
|
||||
if caption_path.is_file():
|
||||
d = read_captions(caption_path)
|
||||
elif EXTRACT_SUBTITLES:
|
||||
d = extract_captions(file.path) or ''
|
||||
|
||||
return a, None, c, d
|
||||
|
|
|
|||
|
|
@ -1,16 +1,12 @@
|
|||
import asyncio
|
||||
from functools import wraps
|
||||
from http import HTTPStatus
|
||||
from typing import List
|
||||
|
||||
from sanic import Blueprint, response
|
||||
from sanic.request import Request
|
||||
from sanic_ext import validate
|
||||
from sanic_ext.extensions.openapi import openapi
|
||||
|
||||
from wrolpi.common import create_websocket_feed, get_sanic_url, \
|
||||
wrol_mode_check
|
||||
from wrolpi.common import logger
|
||||
from wrolpi.common import wrol_mode_check
|
||||
from wrolpi.root_api import add_blueprint, json_response
|
||||
from wrolpi.schema import JSONErrorResponse
|
||||
from . import lib, schema
|
||||
|
|
@ -29,43 +25,6 @@ add_blueprint(bp)
|
|||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
refresh_queue, refresh_event = create_websocket_feed('refresh', '/feeds/refresh', content_bp)
|
||||
|
||||
|
||||
@content_bp.post('/refresh')
|
||||
@content_bp.post('/refresh/<channel_id:str>')
|
||||
@openapi.description('Search for videos that have previously been downloaded and stored.')
|
||||
@openapi.response(HTTPStatus.OK, schema.StreamResponse)
|
||||
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
|
||||
@wrol_mode_check
|
||||
async def refresh(_, channel_id: int = None):
|
||||
refresh_logger = logger.getChild('refresh')
|
||||
stream_url = get_sanic_url(scheme='ws', path='/api/videos/feeds/refresh')
|
||||
|
||||
# Only one refresh can run at a time
|
||||
if refresh_event.is_set():
|
||||
return response.json({'error': 'Refresh already running', 'stream_url': stream_url}, HTTPStatus.CONFLICT)
|
||||
|
||||
refresh_event.set()
|
||||
|
||||
async def do_refresh():
|
||||
try:
|
||||
refresh_logger.info('refresh started')
|
||||
|
||||
channel_ids = [channel_id] if channel_id else None
|
||||
await refresh_videos(channel_ids)
|
||||
|
||||
refresh_logger.info('refresh complete')
|
||||
except Exception:
|
||||
raise
|
||||
finally:
|
||||
refresh_event.clear()
|
||||
|
||||
coro = do_refresh()
|
||||
asyncio.ensure_future(coro)
|
||||
refresh_logger.debug('do_refresh scheduled')
|
||||
return response.json({'code': 'stream-started', 'stream_url': stream_url})
|
||||
|
||||
|
||||
@content_bp.post('/download/<channel_id:int>')
|
||||
@openapi.description('Update a channel catalog, download any missing videos')
|
||||
|
|
@ -77,11 +36,6 @@ def download(_, channel_id: int = None):
|
|||
return response.empty()
|
||||
|
||||
|
||||
@wraps(lib.refresh_videos)
|
||||
async def refresh_videos(channel_ids: List[int] = None):
|
||||
return lib.refresh_videos(channel_ids=channel_ids)
|
||||
|
||||
|
||||
@content_bp.post('/favorite')
|
||||
@openapi.definition(
|
||||
description='Toggle the favorite flag on a video',
|
||||
|
|
|
|||
|
|
@ -1,55 +0,0 @@
|
|||
#! /usr/bin/env python3
|
||||
from pathlib import Path
|
||||
from typing import Generator, Union
|
||||
|
||||
import srt
|
||||
import webvtt
|
||||
|
||||
from wrolpi.common import logger
|
||||
|
||||
__all__ = ['get_captions']
|
||||
|
||||
|
||||
def get_caption_text(caption_path: Union[str, Path]) -> Generator:
|
||||
"""Return all text from each caption of a caption file."""
|
||||
if str(caption_path).endswith('vtt'):
|
||||
for caption in webvtt.read(caption_path):
|
||||
text = str(caption.text).strip()
|
||||
yield text
|
||||
else:
|
||||
with open(caption_path, 'rt') as fh:
|
||||
contents = fh.read()
|
||||
for subtitle in srt.parse(contents):
|
||||
yield subtitle.content
|
||||
|
||||
|
||||
def get_unique_caption_lines(caption_path: Union[str, Path]) -> Generator:
|
||||
"""
|
||||
Return all unique lines from each caption of a caption file.
|
||||
"""
|
||||
last_line = None
|
||||
for text in get_caption_text(caption_path):
|
||||
for line in text.split('\n'):
|
||||
if line and line != last_line:
|
||||
last_line = line
|
||||
yield line
|
||||
|
||||
|
||||
def get_captions(caption_path: Path):
|
||||
"""Parse video captions from the video's captions file."""
|
||||
try:
|
||||
lines = get_unique_caption_lines(str(caption_path))
|
||||
block = '\n'.join(lines)
|
||||
return block
|
||||
except UnicodeDecodeError:
|
||||
# Failed to decode the caption file
|
||||
# TODO handle this error
|
||||
pass
|
||||
except webvtt.errors.MalformedFileError:
|
||||
# File format is broken
|
||||
pass
|
||||
except webvtt.errors.MalformedCaptionError:
|
||||
# Captions form is broken somehow
|
||||
pass
|
||||
|
||||
logger.debug(f'Failed to parse caption file {caption_path}')
|
||||
|
|
@ -1,4 +1,5 @@
|
|||
import asyncio
|
||||
import pathlib
|
||||
from http import HTTPStatus
|
||||
|
||||
from sanic import response, Blueprint
|
||||
|
|
@ -8,11 +9,12 @@ from sanic_ext.extensions.openapi import openapi
|
|||
|
||||
from wrolpi.common import logger, wrol_mode_check, make_media_directory, \
|
||||
get_media_directory, run_after
|
||||
from wrolpi.errors import UnknownDirectory, UnknownChannel
|
||||
from wrolpi.root_api import json_response
|
||||
from wrolpi.schema import JSONErrorResponse
|
||||
from wrolpi.vars import PYTEST
|
||||
from . import lib
|
||||
from .. import schema
|
||||
from .. import schema, Channel
|
||||
from ..lib import save_channels_config
|
||||
|
||||
channel_bp = Blueprint('Channel', url_prefix='/api/videos/channels')
|
||||
|
|
@ -32,7 +34,7 @@ async def get_channels(_: Request):
|
|||
@openapi.description('Get a Channel')
|
||||
@openapi.response(HTTPStatus.OK, schema.ChannelResponse)
|
||||
def channel_get(_: Request, channel_id: int = None):
|
||||
channel = lib.get_channel(channel_id=channel_id)
|
||||
channel = lib.get_channel(channel_id=channel_id, return_dict=True)
|
||||
channel.pop('info_json')
|
||||
return json_response({'channel': channel})
|
||||
|
||||
|
|
@ -55,9 +57,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
|
|||
|
||||
# Refresh the videos asynchronously
|
||||
if not PYTEST:
|
||||
from ..api import refresh_videos
|
||||
coro = refresh_videos([channel.id])
|
||||
asyncio.ensure_future(coro)
|
||||
asyncio.ensure_future(channel.refresh_files())
|
||||
|
||||
return response.json({'success': 'Channel created successfully'}, HTTPStatus.CREATED,
|
||||
{'Location': f'/api/videos/channels/{channel.id}'})
|
||||
|
|
@ -74,6 +74,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
|
|||
@run_after(save_channels_config)
|
||||
@wrol_mode_check
|
||||
def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
|
||||
body.directory = pathlib.Path(body.directory) if body.directory else None
|
||||
channel = lib.update_channel(data=body, channel_id=channel_id)
|
||||
return response.raw('', HTTPStatus.NO_CONTENT,
|
||||
headers={'Location': f'/api/videos/channels/{channel.id}'})
|
||||
|
|
@ -86,3 +87,25 @@ def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
|
|||
def channel_delete(_: Request, channel_id: int):
|
||||
lib.delete_channel(channel_id=channel_id)
|
||||
return response.raw('', HTTPStatus.NO_CONTENT)
|
||||
|
||||
|
||||
@channel_bp.post('/refresh/<channel_id:int>')
|
||||
@openapi.description('Refresh all files in the Channel directory')
|
||||
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
|
||||
@wrol_mode_check
|
||||
def channel_refresh(_: Request, channel_id: int):
|
||||
channel: Channel = lib.get_channel(channel_id=channel_id, return_dict=False)
|
||||
if not channel:
|
||||
raise UnknownChannel()
|
||||
|
||||
asyncio.ensure_future(channel.refresh_files())
|
||||
return response.empty()
|
||||
|
||||
|
||||
@channel_bp.post('/download/<channel_id:int>')
|
||||
@openapi.description('Download the latest catalog for a Channel, download any missing videos.')
|
||||
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
|
||||
@wrol_mode_check
|
||||
def channel_download(_: Request, channel_id: int):
|
||||
lib.download_channel(channel_id)
|
||||
return response.empty()
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import functools
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Union
|
||||
from typing import List, Dict, Union, Optional
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.orm.exc import NoResultFound
|
||||
|
|
@ -7,7 +8,6 @@ from sqlalchemy.orm.exc import NoResultFound
|
|||
from wrolpi.common import run_after, logger, \
|
||||
get_media_directory
|
||||
from wrolpi.db import get_db_curs, optional_session
|
||||
from wrolpi.downloader import download_manager
|
||||
from wrolpi.errors import UnknownChannel, UnknownDirectory, APIError, ValidationError, InvalidDownload
|
||||
from .. import schema
|
||||
from ..common import check_for_channel_conflicts
|
||||
|
|
@ -125,6 +125,9 @@ def update_channel(session: Session, *, data: schema.ChannelPutRequest, channel_
|
|||
data.directory.mkdir()
|
||||
else:
|
||||
raise UnknownDirectory()
|
||||
elif data.directory:
|
||||
# Keep the old directory because the user can't change a Channel's directory.
|
||||
data.directory = channel.directory
|
||||
|
||||
# Verify that the URL/Name/directory aren't taken
|
||||
check_for_channel_conflicts(
|
||||
|
|
@ -165,7 +168,7 @@ def create_channel(session: Session, data: schema.ChannelPostRequest, return_dic
|
|||
name=data.name,
|
||||
url=data.url or None,
|
||||
match_regex=data.match_regex,
|
||||
directory=data.directory, # noqa
|
||||
directory=get_media_directory() / data.directory,
|
||||
download_frequency=data.download_frequency,
|
||||
source_id=data.source_id,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -9,11 +9,10 @@ import mock
|
|||
import pytest
|
||||
|
||||
from modules.videos.channel.lib import delete_channel
|
||||
from modules.videos.common import get_no_channel_directory
|
||||
from modules.videos.models import Channel, Video
|
||||
from modules.videos.models import Channel
|
||||
from wrolpi.dates import now
|
||||
from wrolpi.db import get_db_session
|
||||
from wrolpi.downloader import download_manager, Download
|
||||
from wrolpi.downloader import download_manager, Download, DownloadResult
|
||||
from wrolpi.test.common import assert_dict_contains
|
||||
|
||||
|
||||
|
|
@ -26,31 +25,8 @@ def test_get_channels(test_directory, channel_factory, test_client):
|
|||
assert len(response.json['channels']) == 3
|
||||
|
||||
|
||||
@mock.patch('modules.videos.lib.validate_video')
|
||||
def test_refresh_no_channel(_, test_session, test_client, test_directory, simple_channel):
|
||||
# A regular user-defined channel
|
||||
vid1 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title1.mp4'
|
||||
vid1.touch()
|
||||
vid2 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title2.mp4'
|
||||
vid2.touch()
|
||||
|
||||
# The special NO CHANNEL directory.
|
||||
no_channel_path = get_no_channel_directory()
|
||||
vid3 = no_channel_path / 'some video.mp4'
|
||||
vid3.touch()
|
||||
no_channel_subdir = no_channel_path / 'subdir'
|
||||
no_channel_subdir.mkdir()
|
||||
vid4 = no_channel_subdir / 'other video.mp4'
|
||||
vid4.touch()
|
||||
|
||||
test_client.post('/api/videos/refresh')
|
||||
assert test_session.query(Video).count() == 4
|
||||
|
||||
|
||||
def test_get_video(test_client, test_session, simple_channel, video_factory):
|
||||
"""
|
||||
Test that you get can information about a video. Test that video file can be gotten.
|
||||
"""
|
||||
"""Test that you get can information about a video. Test that video file can be gotten."""
|
||||
now_ = now()
|
||||
video1 = video_factory(channel_id=simple_channel.id)
|
||||
video1.upload_date, video1.title = now_, 'vid1'
|
||||
|
|
@ -66,11 +42,11 @@ def test_get_video(test_client, test_session, simple_channel, video_factory):
|
|||
# Get the video info we inserted
|
||||
_, response = test_client.get('/api/videos/video/1')
|
||||
assert response.status_code == HTTPStatus.OK, response.json
|
||||
assert_dict_contains(response.json['video'], {'title': 'vid1'})
|
||||
assert_dict_contains(response.json['file']['video'], {'title': 'vid1'})
|
||||
|
||||
# The next video is included.
|
||||
assert response.json['prev'] is None
|
||||
assert_dict_contains(response.json['next'], {'title': 'vid2'})
|
||||
assert_dict_contains(response.json['next']['video'], {'title': 'vid2'})
|
||||
|
||||
|
||||
def test_channel_no_download_frequency(test_client, test_session, test_directory, simple_channel):
|
||||
|
|
@ -118,7 +94,7 @@ def test_channel_frequency_update(download_channel, test_client, test_session, t
|
|||
assert old_frequency
|
||||
|
||||
data = dict(
|
||||
directory=str(download_channel.directory.path),
|
||||
directory=str(download_channel.directory),
|
||||
name=download_channel.name,
|
||||
url=download_channel.url,
|
||||
download_frequency=100,
|
||||
|
|
@ -192,73 +168,12 @@ def test_channel_download_crud(test_session, simple_channel):
|
|||
assert test_session.query(Download).count() == 0
|
||||
|
||||
|
||||
def test_video_search(test_client, test_session):
|
||||
"""
|
||||
Test that videos can be searched and that their order is by their textsearch rank.
|
||||
"""
|
||||
# These captions have repeated letters, so they will be higher in the ranking.
|
||||
videos = [
|
||||
('1', 'b b b b e d d'),
|
||||
('2', '2 b b b d'),
|
||||
('3', 'b b'),
|
||||
('4', 'b e e'),
|
||||
('5', ''),
|
||||
]
|
||||
for title, caption in videos:
|
||||
test_session.add(Video(title=title, caption=caption, video_path='foo'))
|
||||
test_session.commit()
|
||||
|
||||
def do_search(search_str, limit=20):
|
||||
d = json.dumps({'search_str': search_str, 'limit': limit})
|
||||
_, resp = test_client.post('/api/videos/search', content=d)
|
||||
return resp
|
||||
|
||||
def search_is_as_expected(resp, expected):
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
response_ids = [i['id'] for i in resp.json['videos']]
|
||||
assert response_ids == expected
|
||||
assert resp.json['totals']['videos'] == len(expected)
|
||||
|
||||
# Repeated runs should return the same result
|
||||
for _ in range(2):
|
||||
# Only videos with a b are returned, ordered by the amount of b's
|
||||
response = do_search('b')
|
||||
search_is_as_expected(response, [1, 2, 3, 4])
|
||||
|
||||
# Only two captions have e
|
||||
response = do_search('e')
|
||||
search_is_as_expected(response, [4, 1])
|
||||
|
||||
# Only two captions have d
|
||||
response = do_search('d')
|
||||
search_is_as_expected(response, [1, 2])
|
||||
|
||||
# 5 can be gotten by it's title
|
||||
response = do_search('5')
|
||||
search_is_as_expected(response, [5])
|
||||
|
||||
# only video 1 has e and d
|
||||
response = do_search('e d')
|
||||
search_is_as_expected(response, [1])
|
||||
|
||||
# video 1 and 4 have b and e, but 1 has more
|
||||
response = do_search('b e')
|
||||
search_is_as_expected(response, [1, 4])
|
||||
|
||||
# Check totals are correct even with a limit
|
||||
response = do_search('b', 2)
|
||||
assert [i['id'] for i in response.json['videos']] == [1, 2]
|
||||
assert response.json['totals']['videos'] == 4
|
||||
|
||||
|
||||
def test_video_file_name(test_session, simple_video, test_client):
|
||||
"""
|
||||
If a Video has no title, the front-end can use the file name as the title.
|
||||
"""
|
||||
"""If a Video has no title, the front-end can use the file name as the title."""
|
||||
_, resp = test_client.get(f'/api/videos/video/{simple_video.id}')
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
assert resp.json['video']['video_path'] == 'simple_video.mp4'
|
||||
assert resp.json['video'].get('stem') == 'simple_video'
|
||||
assert resp.json['file']['video']['video_path'] == 'simple_video.mp4'
|
||||
assert resp.json['file']['video'].get('stem') == 'simple_video'
|
||||
|
||||
|
||||
def test_channel_conflicts(test_client, test_session, test_directory):
|
||||
|
|
@ -339,26 +254,46 @@ def test_channel_empty_url_doesnt_conflict(test_client, test_session, test_direc
|
|||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager):
|
||||
"""A Channel cannot be downloaded until it has been refreshed."""
|
||||
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager, video_factory):
|
||||
"""A Channel cannot be downloaded until it has been refreshed.
|
||||
|
||||
Videos already downloaded are not downloaded again."""
|
||||
vid = video_factory(channel_id=download_channel.id, with_video_file=True, with_poster_ext='jpg')
|
||||
vid.url = 'https://example.com/1'
|
||||
d = download_channel.get_download()
|
||||
test_session.commit()
|
||||
assert not d.next_download
|
||||
|
||||
def check_refreshed(expected: bool):
|
||||
def assert_refreshed(expected: bool):
|
||||
channel = test_session.query(Channel).one()
|
||||
assert channel.refreshed == expected
|
||||
|
||||
check_refreshed(False)
|
||||
assert_refreshed(False)
|
||||
test_session.commit()
|
||||
|
||||
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info:
|
||||
mock_extract_info.return_value = {'entries': [], 'url': 'foo', 'uploader': 'some uploader',
|
||||
downloaded_urls = []
|
||||
|
||||
async def do_download(_, download):
|
||||
# Track all downloaded URLs.
|
||||
downloaded_urls.append(download.url)
|
||||
return DownloadResult(success=True)
|
||||
|
||||
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
|
||||
mock.patch('modules.videos.downloader.VideoDownloader.do_download', do_download):
|
||||
entries = [
|
||||
dict(id=vid.source_id, view_count=0, webpage_url='https://example.com/1'), # Already downloaded.
|
||||
dict(id='not downloaded', view_count=0, webpage_url='https://example.com/2'),
|
||||
]
|
||||
mock_extract_info.return_value = {'entries': entries, 'url': 'foo', 'uploader': 'some uploader',
|
||||
'channel_id': 'the id', 'id': 'the id'}
|
||||
await video_download_manager.do_downloads()
|
||||
await video_download_manager.wait_for_all_downloads()
|
||||
|
||||
check_refreshed(True)
|
||||
# Channel was refreshed before downloading videos.
|
||||
assert_refreshed(True)
|
||||
assert d.next_download
|
||||
# Only the missing video was downloaded.
|
||||
assert downloaded_urls == ['https://example.com/2']
|
||||
|
||||
|
||||
def test_channel_post_directory(test_session, test_client, test_directory):
|
||||
|
|
@ -367,7 +302,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
|
|||
data = dict(name='foo', directory='foo')
|
||||
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.CREATED
|
||||
directory = test_session.query(Channel).filter_by(id=1).one().directory.path
|
||||
directory = test_session.query(Channel).filter_by(id=1).one().directory
|
||||
assert (test_directory / 'foo') == directory
|
||||
assert not directory.is_dir()
|
||||
assert directory.is_absolute()
|
||||
|
|
@ -376,7 +311,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
|
|||
data = dict(name='bar', directory='bar', mkdir=True)
|
||||
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
|
||||
assert response.status_code == HTTPStatus.CREATED
|
||||
directory = test_session.query(Channel).filter_by(id=2).one().directory.path
|
||||
directory = test_session.query(Channel).filter_by(id=2).one().directory
|
||||
assert (test_directory / 'bar') == directory
|
||||
assert directory.is_dir()
|
||||
assert directory.is_absolute()
|
||||
|
|
@ -455,7 +390,7 @@ def test_channel_crud(test_session, test_client, test_directory):
|
|||
assert response.status_code == HTTPStatus.NOT_FOUND
|
||||
|
||||
|
||||
def test_search_channel_videos(test_client, test_session):
|
||||
def test_search_videos_channel(test_client, test_session, video_factory):
|
||||
with get_db_session(commit=True) as session:
|
||||
channel1 = Channel(name='Foo')
|
||||
channel2 = Channel(name='Bar')
|
||||
|
|
@ -469,29 +404,29 @@ def test_search_channel_videos(test_client, test_session):
|
|||
d = dict(channel_id=channel1.id)
|
||||
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert len(response.json['videos']) == 0
|
||||
assert len(response.json['files']) == 0
|
||||
|
||||
with get_db_session(commit=True) as session:
|
||||
vid1 = Video(title='vid1', channel_id=channel2.id, video_path='foo')
|
||||
vid2 = Video(title='vid2', channel_id=channel1.id, video_path='foo')
|
||||
vid1 = video_factory(title='vid1', channel_id=channel2.id)
|
||||
vid2 = video_factory(title='vid2', channel_id=channel1.id)
|
||||
session.add(vid1)
|
||||
session.add(vid2)
|
||||
|
||||
# Videos are gotten by their respective channels
|
||||
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert len(response.json['videos']) == 1
|
||||
assert response.json['totals']['videos'] == 1
|
||||
assert_dict_contains(response.json['videos'][0], dict(id=2, title='vid2', channel_id=channel1.id))
|
||||
assert len(response.json['files']) == 1
|
||||
assert response.json['totals']['files'] == 1
|
||||
assert_dict_contains(response.json['files'][0], dict(path='vid2.mp4', video=dict(channel_id=channel1.id)))
|
||||
|
||||
d = dict(channel_id=channel2.id)
|
||||
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert len(response.json['videos']) == 1
|
||||
assert_dict_contains(response.json['videos'][0], dict(id=1, title='vid1', channel_id=channel2.id))
|
||||
assert len(response.json['files']) == 1
|
||||
assert_dict_contains(response.json['files'][0], dict(path='vid1.mp4', video=dict(channel_id=channel2.id)))
|
||||
|
||||
|
||||
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory):
|
||||
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory, assert_video_search):
|
||||
for i in range(50):
|
||||
video_factory(channel_id=simple_channel.id)
|
||||
|
||||
|
|
@ -500,7 +435,7 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
|
|||
test_session.flush()
|
||||
test_session.refresh(channel2)
|
||||
video_factory(channel_id=channel2.id)
|
||||
test_session.add(Video(title='vid2', channel_id=channel2.id, video_path='foo'))
|
||||
video_factory(title='vid2', channel_id=channel2.id, with_video_file=True)
|
||||
test_session.commit()
|
||||
|
||||
# Get first, second, third, and empty pages of videos.
|
||||
|
|
@ -513,10 +448,8 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
|
|||
]
|
||||
last_ids = []
|
||||
for offset, video_count in tests:
|
||||
d = dict(channel_id=simple_channel.id, order_by='id', offset=offset)
|
||||
_, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
assert len(response.json['videos']) == video_count
|
||||
current_ids = [i['id'] for i in response.json['videos']]
|
||||
_, response = assert_video_search(channel_id=simple_channel.id, order_by='id', offset=offset, limit=20)
|
||||
assert len(response.json['files']) == video_count, 'Returned videos does not match'
|
||||
current_ids = [i['video']['id'] for i in response.json['files']]
|
||||
assert current_ids != last_ids, f'IDs are unchanged current_ids={current_ids}'
|
||||
last_ids = current_ids
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
from modules.videos import schema
|
||||
from modules.videos.channel import lib
|
||||
from modules.videos.lib import save_channels_config, import_channels_config
|
||||
from modules.videos.models import Channel
|
||||
from modules.videos.models import Channel, Video
|
||||
from wrolpi.errors import UnknownChannel
|
||||
|
||||
|
||||
|
|
@ -89,3 +91,39 @@ def test_channels_no_url(test_session, test_directory):
|
|||
|
||||
save_channels_config()
|
||||
import_channels_config()
|
||||
|
||||
|
||||
def test_refresh_videos_finds_channel(test_session, test_client, channel_factory, video_factory):
|
||||
"""A Video will be assigned to a Channel by its info json first, then to its directory."""
|
||||
channel1 = channel_factory(source_id='channel1')
|
||||
channel2 = channel_factory(source_id='channel2')
|
||||
|
||||
# Put the video in the wrong directory. It should be matched to the info json channel first, then to it's directory
|
||||
channel1.source_id = 'channel source id'
|
||||
vid1 = video_factory(
|
||||
with_video_file=(channel2.directory / 'video1.mp4'),
|
||||
with_info_json={'channel_id': channel1.source_id},
|
||||
)
|
||||
test_session.commit()
|
||||
assert not vid1.channel_id
|
||||
|
||||
with mock.patch('wrolpi.common.after_refresh') as mock_after_refresh:
|
||||
# Channel can be found in `video_cleanup`, prevent that with mock.
|
||||
mock_after_refresh.side_effect = []
|
||||
test_client.post('/api/files/refresh')
|
||||
|
||||
# Channel in info_json was found during validation.
|
||||
assert vid1.channel_id == channel1.id
|
||||
|
||||
# Refresh using cleanup, info json should be trusted.
|
||||
test_client.post('/api/files/refresh')
|
||||
assert vid1.channel_id == channel1.id
|
||||
|
||||
# Remove the info json, the Video's Channel will be its directory.
|
||||
vid1.channel_id = None
|
||||
vid1.info_json_path.unlink()
|
||||
test_session.commit()
|
||||
test_client.post('/api/files/refresh')
|
||||
test_session.refresh(vid1)
|
||||
vid1 = test_session.query(Video).one()
|
||||
assert vid1.channel_id == channel2.id
|
||||
|
|
|
|||
|
|
@ -1,12 +1,15 @@
|
|||
import json
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
from datetime import timedelta
|
||||
from decimal import Decimal
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
from typing import Union, Tuple, List, Set, Iterable, Optional
|
||||
from typing import Union, List, Set, Iterable, Optional, Tuple
|
||||
|
||||
import PIL
|
||||
from PIL import Image
|
||||
|
|
@ -14,13 +17,13 @@ from sqlalchemy.orm import Session
|
|||
|
||||
from wrolpi.cmd import which
|
||||
from wrolpi.common import logger, iterify, get_media_directory, \
|
||||
minimize_dict, any_extensions
|
||||
minimize_dict, any_extensions, match_paths_to_suffixes
|
||||
from wrolpi.db import get_db_session, get_db_curs
|
||||
from wrolpi.errors import UnknownFile, ChannelNameConflict, ChannelURLConflict, \
|
||||
from wrolpi.errors import ChannelNameConflict, ChannelURLConflict, \
|
||||
ChannelDirectoryConflict, ChannelSourceIdConflict
|
||||
from wrolpi.media_path import MediaPath
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import DEFAULT_FILE_PERMISSIONS
|
||||
from .models import Channel, Video
|
||||
from .models import Channel
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
|
|
@ -32,6 +35,7 @@ MINIMUM_VIDEO_KEYS = {'id', 'title', 'upload_date', 'duration', 'channel', 'chan
|
|||
'view_count'}
|
||||
# These are the supported video formats. These are in order of their preference.
|
||||
VIDEO_EXTENSIONS = ('mp4', 'ogg', 'webm', 'flv')
|
||||
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
|
||||
|
||||
FFMPEG_BIN = which('ffmpeg', '/usr/bin/ffmpeg', warn=True)
|
||||
|
||||
|
|
@ -49,40 +53,13 @@ def get_videos_directory() -> pathlib.Path:
|
|||
|
||||
|
||||
def get_no_channel_directory() -> pathlib.Path:
|
||||
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist."""
|
||||
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist.""" # noqa
|
||||
directory = get_videos_directory() / 'NO CHANNEL'
|
||||
if not directory.is_dir():
|
||||
directory.mkdir(parents=True)
|
||||
return directory
|
||||
|
||||
|
||||
VALID_VIDEO_KINDS = {'video', 'caption', 'poster', 'description', 'info_json'}
|
||||
|
||||
|
||||
def get_absolute_video_path(video: Video, kind: str = 'video') -> MediaPath:
|
||||
if kind not in VALID_VIDEO_KINDS:
|
||||
raise Exception(f'Unknown video path kind {kind}')
|
||||
path = getattr(video, f'{kind}_path')
|
||||
if path:
|
||||
return path
|
||||
raise UnknownFile()
|
||||
|
||||
|
||||
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
|
||||
|
||||
|
||||
def generate_video_paths(directory: Union[str, pathlib.Path]) -> Tuple[str, pathlib.Path]:
|
||||
"""Generate a list of video paths in the provided directory."""
|
||||
directory = pathlib.Path(directory)
|
||||
|
||||
for child in directory.iterdir():
|
||||
if child.is_file() and match_video_extensions(child.name):
|
||||
child = child.absolute()
|
||||
yield child
|
||||
elif child.is_dir():
|
||||
yield from generate_video_paths(child)
|
||||
|
||||
|
||||
@iterify(set)
|
||||
def remove_duplicate_video_paths(paths: Iterable[Path]) -> Set[Path]:
|
||||
"""
|
||||
|
|
@ -190,13 +167,46 @@ def get_matching_directories(path: Union[str, Path]) -> List[str]:
|
|||
return paths
|
||||
|
||||
|
||||
def generate_video_poster(video_path: Path) -> Path:
|
||||
"""Create a poster (aka thumbnail) next to the provided video_path."""
|
||||
def duration_to_seconds(duration: str) -> int:
|
||||
"""Convert ffmpeg duration to seconds"""
|
||||
hours, minutes, seconds = duration.split(':')
|
||||
seconds, microseconds = seconds.split('.')
|
||||
return int(timedelta(
|
||||
hours=int(hours),
|
||||
minutes=int(minutes),
|
||||
seconds=int(seconds),
|
||||
microseconds=int(microseconds),
|
||||
).total_seconds())
|
||||
|
||||
|
||||
def seconds_to_duration(seconds: int) -> str:
|
||||
"""Convert integer seconds to ffmpeg duration"""
|
||||
hours, seconds = divmod(seconds, 3600)
|
||||
minutes, seconds = divmod(seconds, 60)
|
||||
return f'{hours:02}:{minutes:02}:{seconds:02}.000'
|
||||
|
||||
|
||||
DURATION_REGEX = re.compile(r'Duration: ([\d:.]+),')
|
||||
|
||||
|
||||
def get_duration_from_ffmpeg(stderr: str) -> int:
|
||||
"""Extract the duration from FFPEG's stderr, return this duration as integer seconds."""
|
||||
duration = DURATION_REGEX.findall(stderr)
|
||||
duration = duration_to_seconds(duration[0]) if duration else None
|
||||
return duration
|
||||
|
||||
|
||||
def generate_video_poster(video_path: Path) -> Tuple[Path, Optional[int]]:
|
||||
"""Create a poster (aka thumbnail) next to the provided video_path. Also returns the video duration."""
|
||||
poster_path = video_path.with_suffix('.jpg')
|
||||
cmd = (FFMPEG_BIN, '-n', '-i', str(video_path), '-f', 'mjpeg', '-vframes', '1', '-ss', '00:00:05.000',
|
||||
str(poster_path))
|
||||
if poster_path.exists():
|
||||
# Poster already exists.
|
||||
return poster_path, None
|
||||
try:
|
||||
subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
stderr = proc.stderr.decode()
|
||||
logger.info(f'Generated poster at {poster_path}')
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.warning(f'FFMPEG poster generation failed with stdout: {e.stdout.decode()}')
|
||||
|
|
@ -205,7 +215,8 @@ def generate_video_poster(video_path: Path) -> Path:
|
|||
if not poster_path.exists():
|
||||
raise Exception(f'Failed to find generated poster! {poster_path}')
|
||||
|
||||
return poster_path
|
||||
duration = get_duration_from_ffmpeg(stderr)
|
||||
return poster_path, duration
|
||||
|
||||
|
||||
def convert_image(existing_path: Path, destination_path: Path, ext: str = 'jpeg'):
|
||||
|
|
@ -237,7 +248,7 @@ def is_valid_poster(poster_path: Path) -> bool:
|
|||
FFPROBE_BIN = which('ffprobe', '/usr/bin/ffprobe', warn=True)
|
||||
|
||||
|
||||
def get_video_duration(video_path: Path) -> Optional[int]:
|
||||
def get_video_duration(video_path: Path) -> int:
|
||||
"""Get the duration of a video in seconds. Do this using ffprobe."""
|
||||
if not isinstance(video_path, Path):
|
||||
video_path = Path(video_path)
|
||||
|
|
@ -261,25 +272,29 @@ def get_video_duration(video_path: Path) -> Optional[int]:
|
|||
|
||||
|
||||
def check_for_video_corruption(video_path: Path) -> bool:
|
||||
"""Uses ffprobe to check for specific ways a video file can be corrupt."""
|
||||
if not isinstance(video_path, Path):
|
||||
video_path = Path(video_path)
|
||||
"""Uses ffprobe to check for specific ways a video file can be corrupt.
|
||||
|
||||
Also attempts to screenshot the video near the end of it's duration, this check if the video is corrupted somewhere
|
||||
in the middle."""
|
||||
video_path = Path(video_path) if not isinstance(video_path, Path) else video_path
|
||||
|
||||
if not video_path.is_file():
|
||||
raise FileNotFoundError(f'{video_path} does not exist!')
|
||||
if not FFPROBE_BIN:
|
||||
raise SystemError('ffprobe is not installed!')
|
||||
|
||||
cmd = [FFPROBE_BIN, str(video_path)]
|
||||
# Just read the video using FFMPEG, this prints out useful information about the video.
|
||||
cmd = (FFPROBE_BIN, str(video_path))
|
||||
try:
|
||||
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.warning(f'FFPROBE failed to check for corruption with stdout: {e.stdout.decode()}')
|
||||
logger.warning(f'FFPROBE failed to check for corruption with stderr: {e.stderr.decode()}')
|
||||
return True # video is corrupt.
|
||||
|
||||
messages = (
|
||||
b'Invalid NAL unit size',
|
||||
b'Error splitting the input into NAL units',
|
||||
b'Invalid data found when processing input',
|
||||
)
|
||||
corrupt = False
|
||||
for error in messages:
|
||||
|
|
@ -347,3 +362,19 @@ def minimize_video(video: dict) -> dict:
|
|||
video['info_json'] = minimize_video_info_json(video['info_json'])
|
||||
|
||||
return video
|
||||
|
||||
|
||||
match_video_paths = partial(match_paths_to_suffixes, suffix_groups=(
|
||||
tuple(f'.{i}' for i in VIDEO_EXTENSIONS),
|
||||
('.jpg', '.jpeg', '.webp', '.png'),
|
||||
('.description',),
|
||||
('.en.vtt', '.en.srt'),
|
||||
('.info.json',),
|
||||
))
|
||||
|
||||
|
||||
@iterify(tuple)
|
||||
def match_video_files(files: List[File]) -> Tuple[File, File, File, File, File]:
|
||||
video_paths = match_video_paths([i.path for i in files])
|
||||
for path in video_paths:
|
||||
yield next((i for i in files if i.path == path), None)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
from http import HTTPStatus
|
||||
from itertools import zip_longest
|
||||
from typing import List
|
||||
from uuid import uuid4
|
||||
|
|
@ -13,6 +14,7 @@ from modules.videos.downloader import VideoDownloader, ChannelDownloader
|
|||
from modules.videos.lib import set_test_channels_config, set_test_downloader_config
|
||||
from modules.videos.models import Channel, Video
|
||||
from wrolpi.downloader import DownloadFrequency, DownloadManager, Download
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
|
|
@ -54,9 +56,7 @@ def channel_factory(test_session, test_directory):
|
|||
|
||||
@pytest.fixture
|
||||
def download_channel(test_session, test_directory, video_download_manager) -> Channel:
|
||||
"""
|
||||
Get a test Channel that has a download frequency.
|
||||
"""
|
||||
"""Get a test Channel that has a download frequency."""
|
||||
# Add a frequency to the test channel, then give it a download.
|
||||
channel = Channel(
|
||||
directory=test_directory,
|
||||
|
|
@ -75,64 +75,75 @@ def simple_video(test_session, test_directory, simple_channel) -> Video:
|
|||
"""A Video with an empty video file whose channel is the Simple Channel."""
|
||||
video_path = test_directory / 'simple_video.mp4'
|
||||
video_path.touch()
|
||||
video = Video(video_path=video_path, channel_id=simple_channel.id)
|
||||
video_file = File(path=video_path, model='video')
|
||||
video = Video(video_file=video_file, channel_id=simple_channel.id)
|
||||
test_session.add(video)
|
||||
test_session.commit()
|
||||
video = test_session.query(Video).filter_by(id=video.id).one()
|
||||
return video
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def video_file(test_directory) -> pathlib.Path:
|
||||
destination = test_directory / f'{uuid4()}.mp4'
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
|
||||
|
||||
yield destination
|
||||
|
||||
destination.unlink()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def video_factory(test_session, test_directory):
|
||||
"""Creates Videos for testing."""
|
||||
|
||||
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file: bool = False,
|
||||
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False):
|
||||
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file=None,
|
||||
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False,
|
||||
source_id: str = None):
|
||||
title = title or str(uuid4())
|
||||
if channel_id:
|
||||
|
||||
if with_video_file and isinstance(with_video_file, (pathlib.Path, str)):
|
||||
# Put the video exactly where specified for the test.
|
||||
path = pathlib.Path(with_video_file) if not isinstance(with_video_file, pathlib.Path) else with_video_file
|
||||
elif channel_id:
|
||||
# Put the video in its Channel's directory.
|
||||
channel = test_session.query(Channel).filter_by(id=channel_id).one()
|
||||
path = (channel.directory or test_directory) / f'{title}.mp4'
|
||||
else:
|
||||
(test_directory / 'videos/NO CHANNEL').mkdir(parents=True, exist_ok=True)
|
||||
# Put any video not in a Channel and without a specified file in the NO CHANNEL directory.
|
||||
(test_directory / 'videos/NO CHANNEL').mkdir(exist_ok=True, parents=True)
|
||||
path = test_directory / f'videos/NO CHANNEL/{title}.mp4'
|
||||
|
||||
video_file = File(path=path, model='video')
|
||||
test_session.add(video_file)
|
||||
|
||||
if with_video_file:
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', path)
|
||||
else:
|
||||
path.touch()
|
||||
|
||||
info_json_path = None
|
||||
if with_info_json is True:
|
||||
# User requests a info json file, but does not provide one.
|
||||
info_json_path = path.with_suffix('.info.json')
|
||||
info_json_path.write_text(json.dumps({}))
|
||||
elif with_info_json:
|
||||
if with_info_json:
|
||||
# Use the provided object as the json.
|
||||
with_info_json = {'duration': 5} if with_info_json is True else with_info_json
|
||||
info_json_path = path.with_suffix('.info.json')
|
||||
info_json_path.write_text(json.dumps(with_info_json))
|
||||
test_session.add(File(path=info_json_path))
|
||||
|
||||
poster_path = None
|
||||
if with_poster_ext:
|
||||
poster_path = path.with_suffix(f'.{with_poster_ext}')
|
||||
Image.new('RGB', (25, 25)).save(poster_path)
|
||||
test_session.add(File(path=poster_path))
|
||||
|
||||
caption_path = None
|
||||
if with_caption_file:
|
||||
caption_path = path.with_suffix('.en.vtt')
|
||||
caption_path.touch()
|
||||
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', caption_path)
|
||||
test_session.add(File(path=caption_path))
|
||||
|
||||
video = Video(video_path=str(path), title=title, channel_id=channel_id, source_id=title,
|
||||
info_json_path=info_json_path, poster_path=poster_path, caption_path=caption_path,
|
||||
upload_date=upload_date)
|
||||
video = Video(
|
||||
video_path=path,
|
||||
title=title,
|
||||
channel_id=channel_id,
|
||||
source_id=source_id or title,
|
||||
info_json_path=info_json_path,
|
||||
poster_path=poster_path,
|
||||
caption_path=caption_path,
|
||||
upload_date=upload_date,
|
||||
)
|
||||
test_session.add(video)
|
||||
video_file.do_index()
|
||||
return video
|
||||
|
||||
return factory
|
||||
|
|
@ -199,3 +210,71 @@ def assert_video_ids(test_session):
|
|||
assert id_ == expected, f'Video ids were not as expected: {expected_video_ids=} != {video_ids=}'
|
||||
|
||||
return checker
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def video_with_search_factory(test_session, test_directory):
|
||||
"""A factory that creates a Video record with an associated video File record."""
|
||||
|
||||
def video_with_search(path: str = None, title: str = None, b_text: str = None, c_text: str = None,
|
||||
d_text: str = None, mimetype: str = 'video/mp4'):
|
||||
path = path or (test_directory / str(uuid4())).absolute()
|
||||
video_file = File(path=path, a_text=title, d_text=d_text, mimetype=mimetype, model='video')
|
||||
video = Video(title=title, video_file=video_file)
|
||||
test_session.add(video)
|
||||
return video
|
||||
|
||||
return video_with_search
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def assert_video_search(test_client):
|
||||
"""A fixture which performs a video search request against the API.
|
||||
|
||||
If assert_* params are passed, the response is checked."""
|
||||
|
||||
def search_videos(
|
||||
assert_total: int = None,
|
||||
assert_ids: List[int] = None,
|
||||
assert_paths: List[str] = None,
|
||||
search_str: str = None,
|
||||
filters: List[str] = None,
|
||||
offset: int = None,
|
||||
limit: int = None,
|
||||
order_by: str = None,
|
||||
channel_id: int = None,
|
||||
):
|
||||
content = dict()
|
||||
if search_str is not None:
|
||||
content['search_str'] = search_str
|
||||
if filters is not None:
|
||||
content['filters'] = filters
|
||||
if offset is not None:
|
||||
content['offset'] = offset
|
||||
if limit is not None:
|
||||
content['limit'] = limit
|
||||
if order_by is not None:
|
||||
content['order_by'] = order_by
|
||||
if channel_id is not None:
|
||||
content['channel_id'] = channel_id
|
||||
|
||||
request, response = test_client.post('/api/videos/search', content=json.dumps(content))
|
||||
|
||||
assert response.json
|
||||
assert response.status_code == HTTPStatus.OK
|
||||
|
||||
if assert_ids is not None:
|
||||
assert 'files' in response.json, 'No video files in response'
|
||||
assert [i['video']['id'] for i in response.json['files']] == assert_ids, 'Video IDs do not match'
|
||||
|
||||
if assert_paths is not None:
|
||||
assert 'files' in response.json, 'No video files in response'
|
||||
assert [i['video']['video_path'] for i in response.json['files']] == assert_paths, \
|
||||
'Video paths do not match'
|
||||
|
||||
if assert_total:
|
||||
assert response.json['totals']['files'] == int(assert_total), 'Total video files does not match'
|
||||
|
||||
return request, response
|
||||
|
||||
return search_videos
|
||||
|
|
|
|||
|
|
@ -13,19 +13,20 @@ from yt_dlp.extractor import YoutubeTabIE # noqa
|
|||
from yt_dlp.utils import UnsupportedError, DownloadError
|
||||
|
||||
from wrolpi.cmd import which
|
||||
from wrolpi.common import logger, extract_domain, get_media_directory
|
||||
from wrolpi.common import logger, get_media_directory, escape_file_name
|
||||
from wrolpi.dates import now
|
||||
from wrolpi.db import get_db_session, get_db_curs
|
||||
from wrolpi.db import optional_session
|
||||
from wrolpi.downloader import Downloader, Download, DownloadResult
|
||||
from wrolpi.errors import UnknownChannel, ChannelURLEmpty, UnrecoverableDownloadError
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PYTEST
|
||||
from .channel.lib import create_channel, get_channel
|
||||
from .common import apply_info_json, get_no_channel_directory, get_videos_directory
|
||||
from .lib import upsert_video, refresh_channel_videos, get_downloader_config
|
||||
from .lib import get_downloader_config
|
||||
from .models import Video, Channel
|
||||
from .schema import ChannelPostRequest
|
||||
from .video_url_resolver import video_url_resolver
|
||||
from .video_url_resolver import normalize_youtube_shorts_url
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
ydl_logger = logger.getChild('youtube-dl')
|
||||
|
|
@ -84,16 +85,16 @@ class ChannelDownloader(Downloader, ABC):
|
|||
async def do_download(self, download: Download) -> DownloadResult:
|
||||
"""Update a Channel's catalog, then schedule downloads of every missing video."""
|
||||
info = extract_info(download.url, process=False)
|
||||
# Resolve the entries generator.
|
||||
# Resolve the "entries" generator.
|
||||
info['entries'] = list(info['entries'])
|
||||
download.info_json = info
|
||||
if session := Session.object_session(download):
|
||||
# May not have a session during testing.
|
||||
session.commit()
|
||||
|
||||
name = info['uploader']
|
||||
kind = info['uploader']
|
||||
channel_source_id = info['channel_id']
|
||||
channel = get_or_create_channel(channel_source_id, download.url, name)
|
||||
channel = get_or_create_channel(channel_source_id, download.url, kind)
|
||||
channel.dict() # get all attributes while we have the session.
|
||||
|
||||
location = f'/videos/channel/{channel.id}/video' if channel and channel.id else None
|
||||
|
|
@ -103,33 +104,49 @@ class ChannelDownloader(Downloader, ABC):
|
|||
if is_a_playlist:
|
||||
downloads = self.get_playlist_downloads(download)
|
||||
else:
|
||||
downloads = self.get_channel_downloads(download, info, channel)
|
||||
downloads = await self.get_channel_downloads(download, info, channel)
|
||||
return DownloadResult(success=True, location=location, downloads=downloads)
|
||||
except Exception:
|
||||
name = 'playlist' if is_a_playlist else 'channel'
|
||||
logger.warning(f'Failed to update catalog of {name} {download.url}')
|
||||
if PYTEST:
|
||||
raise
|
||||
kind = 'playlist' if is_a_playlist else 'channel'
|
||||
logger.warning(f'Failed to update catalog of {kind} {download.url}')
|
||||
return DownloadResult(success=False, location=location, error=str(traceback.format_exc()))
|
||||
|
||||
@staticmethod
|
||||
def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
|
||||
async def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
|
||||
"""Get a list of all videos in a Channel, schedule downloads for any missing videos."""
|
||||
channel_id = channel.id
|
||||
|
||||
update_channel_catalog(channel, info)
|
||||
|
||||
domain = extract_domain(download.url)
|
||||
missing_videos = find_all_missing_videos(channel.id)
|
||||
downloads = []
|
||||
for _, _, missing_video in missing_videos:
|
||||
url = missing_video.get('webpage_url') or video_url_resolver(domain, missing_video)
|
||||
downloads.append(url)
|
||||
with get_db_session(commit=True) as session:
|
||||
channel = get_channel(session, channel_id=channel_id, return_dict=False)
|
||||
if not channel.videos or not channel.refreshed:
|
||||
logger.warning(f'Refreshing videos in {channel.directory} because {channel} has no video records!')
|
||||
await channel.refresh_files()
|
||||
session.commit() # Commit the refresh.
|
||||
session.refresh(channel)
|
||||
|
||||
# Get all videos in the channel.
|
||||
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
|
||||
# YouTube Shorts are handled specially.
|
||||
downloads = list(map(normalize_youtube_shorts_url, downloads))
|
||||
# Only download those that have not yet been downloaded.
|
||||
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
|
||||
downloads = [i for i in downloads if i not in already_downloaded]
|
||||
return downloads
|
||||
|
||||
@staticmethod
|
||||
def get_playlist_downloads(download: Download) -> List[str]:
|
||||
"""Get a list of all videos in a playlist, schedule downloads for any missing videos."""
|
||||
# Get all videos in the playlist
|
||||
downloads = [i['url'] for i in download.info_json['entries']]
|
||||
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
|
||||
# YouTube Shorts are handled specially.
|
||||
downloads = [normalize_youtube_shorts_url(i) for i in downloads]
|
||||
# Only download those that have not yet been downloaded.
|
||||
downloads = [i for i in downloads if not video_downloader.already_downloaded(i)]
|
||||
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
|
||||
downloads = [i for i in downloads if i not in already_downloaded]
|
||||
return downloads
|
||||
|
||||
|
||||
|
|
@ -153,9 +170,13 @@ class VideoDownloader(Downloader, ABC):
|
|||
return f'<VideoDownloader>'
|
||||
|
||||
@optional_session
|
||||
def already_downloaded(self, url: str, session: Session = None) -> bool:
|
||||
def already_downloaded(self, *urls: str, session: Session = None) -> List:
|
||||
# We only consider a video record with a video file as "downloaded".
|
||||
return bool(session.query(Video).filter(Video.url == url, Video.video_path != None, ).count()) # noqa
|
||||
videos = list(session.query(Video).filter(
|
||||
Video.url.in_(urls),
|
||||
Video.video_path != None, # noqa
|
||||
))
|
||||
return videos
|
||||
|
||||
@classmethod
|
||||
def valid_url(cls, url) -> Tuple[bool, Optional[dict]]:
|
||||
|
|
@ -200,7 +221,7 @@ class VideoDownloader(Downloader, ABC):
|
|||
# Use the default directory if this video has no channel.
|
||||
out_dir = get_no_channel_directory()
|
||||
if channel:
|
||||
out_dir = channel.directory.path
|
||||
out_dir = channel.directory
|
||||
out_dir.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
logs = None # noqa
|
||||
|
|
@ -244,10 +265,14 @@ class VideoDownloader(Downloader, ABC):
|
|||
)
|
||||
|
||||
with get_db_session(commit=True) as session:
|
||||
# If the video is from a channel, it will already be in the database.
|
||||
source_id = entry['id']
|
||||
existing_video = session.query(Video).filter_by(source_id=source_id).one_or_none()
|
||||
video = upsert_video(session, video_path, channel, id_=existing_video.id if existing_video else None)
|
||||
video_file = File.upsert(video_path, session)
|
||||
video: Video = Video.upsert(video_file, session)
|
||||
video.source_id = entry['id']
|
||||
if channel:
|
||||
video.channel_id = channel.id
|
||||
video.video_file.do_index()
|
||||
video.validate(session)
|
||||
session.flush([video])
|
||||
video_id = video.id
|
||||
except UnrecoverableDownloadError:
|
||||
raise
|
||||
|
|
@ -335,7 +360,7 @@ def get_or_create_channel(source_id: str = None, url: str = None, name: str = No
|
|||
raise ValueError(f'Cannot create channel without a name')
|
||||
|
||||
# Channel does not exist. Create one in the video directory.
|
||||
channel_directory = get_videos_directory() / name
|
||||
channel_directory = get_videos_directory() / escape_file_name(name)
|
||||
if not channel_directory.is_dir():
|
||||
channel_directory.mkdir(parents=True)
|
||||
data = ChannelPostRequest(
|
||||
|
|
@ -409,7 +434,7 @@ def update_channel_catalog(channel: Channel, info: dict):
|
|||
|
||||
# Write the Channel's info to a JSON file.
|
||||
if channel.directory:
|
||||
info_json_path = channel.directory.path / f'{channel.name}.info.json'
|
||||
info_json_path = channel.directory / f'{channel.name}.info.json'
|
||||
with info_json_path.open('wt') as fh:
|
||||
json.dump(info, fh, indent=2)
|
||||
|
||||
|
|
@ -440,7 +465,7 @@ def _find_all_missing_videos(channel_id: id) -> List[Tuple]:
|
|||
return missing_videos
|
||||
|
||||
|
||||
def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
|
||||
async def find_all_missing_videos(channel_id: int = None):
|
||||
"""
|
||||
Find all videos that don't have a video file, but are found in the DB (taken from the channel's info_json).
|
||||
|
||||
|
|
@ -453,7 +478,7 @@ def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
|
|||
# Check that the channel has some videos. We can't be sure what is missing if we don't know what we have.
|
||||
if channel.refreshed is False:
|
||||
# Refresh this channel's videos.
|
||||
refresh_channel_videos(channel)
|
||||
await channel.refresh_files()
|
||||
|
||||
match_regex = re.compile(channel.match_regex) if channel.match_regex else None
|
||||
|
||||
|
|
|
|||
|
|
@ -1,24 +1,23 @@
|
|||
import functools
|
||||
import html
|
||||
import pathlib
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from typing import Tuple, Optional, List, Union
|
||||
from uuid import uuid1
|
||||
from typing import Tuple, Optional, Union
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
from yt_dlp import YoutubeDL
|
||||
|
||||
from wrolpi import before_startup
|
||||
from wrolpi.common import chunks, ConfigFile, get_media_directory, sanitize_link
|
||||
from wrolpi.common import ConfigFile, get_media_directory, sanitize_link
|
||||
from wrolpi.dates import from_timestamp, Seconds
|
||||
from wrolpi.db import get_db_curs, get_db_session, optional_session
|
||||
from wrolpi.media_path import MediaPath
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PYTEST
|
||||
from .captions import get_captions
|
||||
from .common import generate_video_paths, remove_duplicate_video_paths, apply_info_json, get_video_duration, \
|
||||
is_valid_poster, convert_image, generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
|
||||
get_no_channel_directory, check_for_video_corruption
|
||||
from .common import is_valid_poster, convert_image, \
|
||||
generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
|
||||
get_no_channel_directory, get_video_duration
|
||||
from .models import Channel, Video
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
|
@ -26,156 +25,66 @@ logger = logger.getChild(__name__)
|
|||
DEFAULT_DOWNLOAD_FREQUENCY = Seconds.week
|
||||
|
||||
|
||||
def refresh_channel_videos(channel: Channel):
|
||||
"""
|
||||
Find all video files in a channel's directory. Add any videos not in the DB to the DB.
|
||||
"""
|
||||
# Set the idempotency key, we can remove any videos not touched during this search.
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('UPDATE video SET idempotency=NULL WHERE channel_id=%s', (channel.id,))
|
||||
|
||||
idempotency = str(uuid1())
|
||||
directory = channel.directory.path
|
||||
|
||||
# A set of absolute paths that exist in the file system
|
||||
possible_new_paths = generate_video_paths(directory)
|
||||
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
|
||||
|
||||
# Update all videos that match the current video paths
|
||||
new_paths = [str(i) for i in possible_new_paths]
|
||||
with get_db_curs(commit=True) as curs:
|
||||
query = 'UPDATE video SET idempotency = %s WHERE channel_id = %s AND video_path = ANY(%s) RETURNING video_path'
|
||||
curs.execute(query, (idempotency, channel.id, new_paths))
|
||||
existing_paths = {i for (i,) in curs.fetchall()}
|
||||
|
||||
# Get the paths for any video not yet in the DB
|
||||
# (paths in DB are relative, but we need to pass an absolute path)
|
||||
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
|
||||
|
||||
for chunk in chunks(new_videos, 20):
|
||||
with get_db_session(commit=True) as session:
|
||||
for video_path in chunk:
|
||||
video_path = pathlib.Path(video_path)
|
||||
upsert_video(session, video_path, channel, idempotency=idempotency)
|
||||
logger.debug(f'{channel.name}: Added {video_path.name}')
|
||||
|
||||
with get_db_curs(commit=True) as curs:
|
||||
stmt = 'DELETE FROM video WHERE channel_id=%s AND idempotency IS NULL AND video_path IS NOT NULL RETURNING id'
|
||||
curs.execute(stmt, (channel.id,))
|
||||
deleted_count = len(curs.fetchall())
|
||||
|
||||
if deleted_count:
|
||||
deleted_status = f'Deleted {deleted_count} video records from channel {channel.name}'
|
||||
logger.info(deleted_status)
|
||||
|
||||
logger.info(f'{channel.name}: {len(new_videos)} new videos, {len(existing_paths)} already existed. ')
|
||||
|
||||
with get_db_session(commit=True) as session:
|
||||
channel = session.query(Channel).filter_by(id=channel.id).one()
|
||||
channel.refreshed = True
|
||||
|
||||
apply_info_json(channel.id)
|
||||
class VideoInfoJSON(object):
|
||||
channel_source_id = None
|
||||
title = None
|
||||
duration = None
|
||||
view_count = None
|
||||
url = None
|
||||
|
||||
|
||||
def refresh_no_channel_videos():
|
||||
"""
|
||||
Refresh the Videos in the NO CHANNEL directory.
|
||||
"""
|
||||
from modules.videos.downloader import get_no_channel_directory
|
||||
directory = get_no_channel_directory()
|
||||
if not directory.is_dir():
|
||||
return
|
||||
|
||||
logger.info('Refreshing NO CHANNEL videos')
|
||||
|
||||
idempotency = str(uuid1())
|
||||
|
||||
# Get all WROLPi compatible videos, remove any duplicates (different formats).
|
||||
possible_new_paths = generate_video_paths(directory)
|
||||
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
|
||||
|
||||
new_paths = [str(i) for i in possible_new_paths]
|
||||
with get_db_curs(commit=True) as curs:
|
||||
query = 'UPDATE video SET idempotency = %s WHERE video_path = ANY(%s) RETURNING video_path'
|
||||
curs.execute(query, (idempotency, new_paths))
|
||||
existing_paths = {i for (i,) in curs.fetchall()}
|
||||
|
||||
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
|
||||
|
||||
for chunk in chunks(new_videos, 20):
|
||||
with get_db_session(commit=True) as session:
|
||||
for video_path in chunk:
|
||||
upsert_video(session, pathlib.Path(video_path), idempotency=idempotency)
|
||||
logger.debug(f'Added NO CHANNEL video {video_path}')
|
||||
|
||||
with get_db_curs(commit=True) as curs:
|
||||
curs.execute('DELETE FROM video WHERE channel_id IS NULL AND idempotency IS NULL RETURNING id')
|
||||
deleted_count = len(curs.fetchall())
|
||||
|
||||
if deleted_count:
|
||||
deleted_status = f'Deleted {deleted_count} video records in NO CHANNEL.'
|
||||
logger.info(deleted_status)
|
||||
|
||||
|
||||
def process_video_info_json(video: Video):
|
||||
def process_video_info_json(video: Video) -> VideoInfoJSON:
|
||||
"""
|
||||
Parse the Video's info json file, return the relevant data.
|
||||
"""
|
||||
title = duration = view_count = url = None
|
||||
video_info_json = VideoInfoJSON()
|
||||
if info_json := video.get_info_json():
|
||||
title = info_json.get('fulltitle') or info_json.get('title')
|
||||
title = html.unescape(title) if title else None
|
||||
video_info_json.title = html.unescape(title) if title else None
|
||||
|
||||
duration = info_json.get('duration')
|
||||
view_count = info_json.get('view_count')
|
||||
url = info_json.get('webpage_url') or info_json.get('url')
|
||||
video_info_json.duration = info_json.get('duration')
|
||||
video_info_json.view_count = info_json.get('view_count')
|
||||
video_info_json.url = info_json.get('webpage_url') or info_json.get('url')
|
||||
video_info_json.channel_source_id = info_json.get('channel_id')
|
||||
|
||||
return title, duration, view_count, url
|
||||
return video_info_json
|
||||
|
||||
|
||||
def validate_videos():
|
||||
@optional_session
|
||||
@functools.lru_cache
|
||||
def get_channel_id_by_source_id(session: Session, source_id: str) -> Optional[int]:
|
||||
"""Return the id of the Channel which matches the provided `source_id`.
|
||||
|
||||
These are cached results."""
|
||||
channel = session.query(Channel).filter_by(source_id=source_id).one_or_none()
|
||||
return channel.id if channel else None
|
||||
|
||||
|
||||
def validate_video(video: Video, channel_generate_poster: bool, session: Session):
|
||||
"""
|
||||
Validate all Videos not yet validated. A Video is validated when we have attempted to find its: title, duration,
|
||||
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
|
||||
found, it will be generated from the video file.
|
||||
Validate a single video.
|
||||
|
||||
This function marks the Video as validated, even if no data can be found so a Video will not be validated multiple
|
||||
times.
|
||||
"""
|
||||
with get_db_curs() as curs:
|
||||
curs.execute('SELECT id FROM video WHERE video_path IS NOT NULL AND validated IS FALSE')
|
||||
video_ids = [i['id'] for i in curs.fetchall()]
|
||||
curs.execute('SELECT id, generate_posters FROM channel')
|
||||
channel_generate_posters = {i['id']: i['generate_posters'] for i in curs.fetchall()}
|
||||
A Video is validated when we have attempted to find its: title, duration, view_count, url, caption, size, and
|
||||
channel.
|
||||
|
||||
logger.info(f'Validating {len(video_ids)} videos.')
|
||||
for chunk in chunks(video_ids, 20):
|
||||
with get_db_session(commit=True) as session:
|
||||
videos = session.query(Video).filter(Video.id.in_(chunk)).all()
|
||||
for video in videos:
|
||||
video.validate()
|
||||
|
||||
|
||||
def validate_video(video: Video, channel_generate_poster: bool):
|
||||
"""
|
||||
Validate a single video. A Video is validated when we have attempted to find its: title, duration,
|
||||
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
|
||||
A Video is also valid when it has a JPEG poster, if any. If no poster can be
|
||||
found, it will be generated from the video file.
|
||||
"""
|
||||
if not video.title or not video.duration or not video.view_count or not video.url:
|
||||
info_json_path = video.info_json_file.path if video.info_json_file else video.info_json_path
|
||||
if info_json_path and (not video.title or not video.duration or not video.view_count or not video.url):
|
||||
# These properties can be found in the info json.
|
||||
title, duration, view_count, url = process_video_info_json(video)
|
||||
video.title = title
|
||||
video.duration = duration
|
||||
video.url = url
|
||||
video_info_json = process_video_info_json(video)
|
||||
video.title = video_info_json.title
|
||||
video.duration = video_info_json.duration
|
||||
video.url = video_info_json.url
|
||||
# View count will probably be overwritten by more recent data when this Video's Channel is
|
||||
# updated.
|
||||
video.view_count = video.view_count or view_count
|
||||
video.view_count = video.view_count or video_info_json.view_count
|
||||
|
||||
video_path = video.video_path.path if isinstance(video.video_path, MediaPath) else video.video_path
|
||||
if video_info_json.channel_source_id:
|
||||
video.channel_id = get_channel_id_by_source_id(source_id=video_info_json.channel_source_id)
|
||||
|
||||
# Check for corruption, but don't throw away the video even if it is corrupt. Errors will be logged in this call.
|
||||
check_for_video_corruption(video_path)
|
||||
video_path = video.video_path
|
||||
|
||||
if not video.title or not video.upload_date or not video.source_id:
|
||||
# Video is missing things that can be extracted from the video file name.
|
||||
|
|
@ -184,41 +93,41 @@ def validate_video(video: Video, channel_generate_poster: bool):
|
|||
video.title = video.title or html.unescape(title)
|
||||
video.upload_date = video.upload_date or upload_date
|
||||
video.source_id = video.source_id or source_id
|
||||
if not video.duration:
|
||||
# Video duration was not in the info json, use ffprobe.
|
||||
video.duration = get_video_duration(video_path)
|
||||
if not video.caption and video.caption_path:
|
||||
video.caption = get_captions(video.caption_path.path)
|
||||
|
||||
if not video.size or not video.modification_datetime:
|
||||
stat = video_path.stat()
|
||||
video.size = video.size or stat.st_size
|
||||
video.modification_datetime = video.modification_datetime or from_timestamp(stat.st_mtime)
|
||||
|
||||
if not video.poster_path:
|
||||
# Video poster is not found, lets check near the video file.
|
||||
for ext in ('.jpg', '.jpeg', '.webp', '.png'):
|
||||
if (poster_path := video_path.with_suffix(ext)).is_file():
|
||||
video.poster_path = poster_path
|
||||
break
|
||||
if channel_generate_poster:
|
||||
# Try to convert/generate, but keep the old poster if those fail.
|
||||
video.poster_path = convert_or_generate_poster(video) or video.poster_path
|
||||
new_poster_path, duration = convert_or_generate_poster(video) or video.poster_path
|
||||
if new_poster_path:
|
||||
# Poster was created/updated.
|
||||
poster_file = File.upsert(new_poster_path, session)
|
||||
session.flush([poster_file])
|
||||
video.poster_file = poster_file
|
||||
video.poster_path = new_poster_path
|
||||
if duration:
|
||||
video.duration = duration
|
||||
|
||||
if not video.duration:
|
||||
# Duration was not retrieved during poster generation.
|
||||
video.duration = get_video_duration(video_path)
|
||||
|
||||
|
||||
def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
|
||||
def convert_or_generate_poster(video: Video) -> Tuple[Optional[pathlib.Path], Optional[int]]:
|
||||
"""
|
||||
If a Video has a poster, but the poster is invalid, convert it. If a Video has no poster, generate one from the
|
||||
video file.
|
||||
|
||||
Returns None if the poster was not converted, and not generated.
|
||||
"""
|
||||
video_path = video.video_path.path
|
||||
video_path = video.video_path
|
||||
# Modification/generation of poster is enabled for this channel.
|
||||
if video.poster_path:
|
||||
if video.poster_path or video.poster_file:
|
||||
# Check that the poster is a more universally supported JPEG.
|
||||
old: pathlib.Path = video.poster_path.path if \
|
||||
isinstance(video.poster_path, MediaPath) else video.poster_path
|
||||
old: pathlib.Path = video.poster_path if video.poster_path else video.poster_file.path
|
||||
new = old.with_suffix('.jpg')
|
||||
|
||||
if old != new and new.exists():
|
||||
|
|
@ -231,63 +140,22 @@ def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
|
|||
try:
|
||||
convert_image(old, new)
|
||||
old.unlink(missing_ok=True)
|
||||
logger.info(f'Converted invalid poster {old} to {new}')
|
||||
return new
|
||||
logger.info(f'Converted invalid poster {repr(str(old))} to {repr(str(new))}')
|
||||
return new, None
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to convert invalid poster {old} to {new}', exc_info=e)
|
||||
return
|
||||
return None, None
|
||||
|
||||
if not video.poster_path:
|
||||
if not video.poster_path or not video.poster_file:
|
||||
# Video poster was not discovered, or converted. Let's generate it.
|
||||
try:
|
||||
poster_path = generate_video_poster(video_path)
|
||||
poster_path, duration = generate_video_poster(video_path)
|
||||
logger.debug(f'Generated poster for {video}')
|
||||
return poster_path
|
||||
return poster_path, duration
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to generate poster for {video}', exc_info=e)
|
||||
|
||||
|
||||
def refresh_videos(channel_ids: List[int] = None):
|
||||
"""
|
||||
Find any videos in the channel directories and add them to the DB. Delete DB records of any videos not in the
|
||||
file system.
|
||||
|
||||
Yields status updates to be passed to the UI.
|
||||
|
||||
:return:
|
||||
"""
|
||||
logger.info('Refreshing video files')
|
||||
with get_db_session() as session:
|
||||
if channel_ids:
|
||||
channels = session.query(Channel).filter(Channel.id.in_(channel_ids))
|
||||
else:
|
||||
channels = session.query(Channel).all()
|
||||
|
||||
channels = list(channels)
|
||||
|
||||
if not channels and channel_ids:
|
||||
raise Exception(f'No channels match id(s): {channel_ids}')
|
||||
elif not channels:
|
||||
raise Exception(f'No channels in DB. Have you created any?')
|
||||
|
||||
for channel in channels:
|
||||
try:
|
||||
refresh_channel_videos(channel)
|
||||
except Exception as e:
|
||||
logger.fatal(f'Failed to refresh videos for channel {channel.name}!', exc_info=e)
|
||||
pass
|
||||
|
||||
if not channel_ids:
|
||||
# Refresh NO CHANNEL videos when not refreshing a specific channel.
|
||||
refresh_no_channel_videos()
|
||||
|
||||
# Fill in any missing data for all videos.
|
||||
if not PYTEST:
|
||||
import_channels_config()
|
||||
|
||||
validate_videos()
|
||||
|
||||
logger.info('Refresh of video files complete')
|
||||
return None, None
|
||||
|
||||
|
||||
class ChannelsConfig(ConfigFile):
|
||||
|
|
@ -470,9 +338,10 @@ def get_channels_config_from_db(session: Session) -> dict:
|
|||
favorites = defaultdict(lambda: {})
|
||||
for video in favorite_videos:
|
||||
if video.channel:
|
||||
favorites[str(video.channel.directory.relative)][video.video_path.path.name] = dict(favorite=video.favorite)
|
||||
directory = video.channel.directory.relative_to(get_media_directory())
|
||||
favorites[str(directory)][video.video_path.name] = dict(favorite=video.favorite)
|
||||
else:
|
||||
favorites['NO CHANNEL'][video.video_path.path.name] = dict(favorite=video.favorite)
|
||||
favorites['NO CHANNEL'][video.video_path.name] = dict(favorite=video.favorite)
|
||||
favorites = dict(favorites)
|
||||
|
||||
return dict(channels=channels, favorites=favorites)
|
||||
|
|
@ -510,7 +379,6 @@ def import_channels_config():
|
|||
"""Import channel settings to the DB. Existing channels will be updated."""
|
||||
logger.info('Importing videos config')
|
||||
try:
|
||||
from .downloader import ChannelDownloader
|
||||
config = get_channels_config()
|
||||
channels, favorites = config.channels, config.favorites
|
||||
|
||||
|
|
@ -569,7 +437,7 @@ def import_channels_config():
|
|||
if not channel:
|
||||
logger.warning(f'Cannot find channel {directory=} for favorites!')
|
||||
continue
|
||||
channel_dir = channel.directory.path
|
||||
channel_dir = channel.directory
|
||||
else:
|
||||
channel_dir = get_no_channel_directory()
|
||||
|
||||
|
|
@ -698,74 +566,3 @@ def parse_video_file_name(video_path: pathlib.Path) -> \
|
|||
# Return the stem as a last resort
|
||||
title = pathlib.Path(video_path).stem.strip()
|
||||
return None, None, None, title
|
||||
|
||||
|
||||
def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel = None, idempotency: str = None,
|
||||
id_: str = None) -> Video:
|
||||
"""
|
||||
Insert a video into the DB. Also, find any meta-files near the video file and store them on the video row.
|
||||
|
||||
If id_ is provided, update that entry.
|
||||
"""
|
||||
if not video_path.is_absolute():
|
||||
raise ValueError(f'Video path is not absolute: {video_path}')
|
||||
|
||||
# This function can update or insert a Video.
|
||||
video = None
|
||||
if id_:
|
||||
video = session.query(Video).filter_by(id=id_).one()
|
||||
if not video:
|
||||
# The video may already be in the database, match the source_id to any existing video.
|
||||
_, _, source_id, _ = parse_video_file_name(video_path)
|
||||
if source_id:
|
||||
video = session.query(Video).filter_by(source_id=source_id).one_or_none()
|
||||
if not video:
|
||||
video = Video()
|
||||
session.add(video)
|
||||
|
||||
# Set the file values, all other things can be found using these files.
|
||||
poster_path, description_path, caption_path, info_json_path = find_meta_files(video_path)
|
||||
video.caption_path = caption_path
|
||||
video.description_path = description_path
|
||||
video.info_json_path = info_json_path
|
||||
video.poster_path = poster_path
|
||||
video.video_path = video_path
|
||||
if channel and not str(video_path).startswith(str(channel.directory.path)):
|
||||
raise ValueError(f'Video path is not within its channel {video_path=} not in {channel.directory=}')
|
||||
|
||||
if channel:
|
||||
video.channel_id = channel.id
|
||||
video.idempotency = idempotency
|
||||
|
||||
session.flush()
|
||||
session.refresh(video)
|
||||
|
||||
video.validate()
|
||||
|
||||
session.flush()
|
||||
|
||||
return video
|
||||
|
||||
|
||||
def find_meta_files(path: pathlib.Path) -> Tuple[pathlib.Path, pathlib.Path, pathlib.Path, pathlib.Path]:
|
||||
"""
|
||||
Find all files that share a file's full path, except their extensions. It is assumed that file with the
|
||||
same name, but different extension is related to that file. A None will be yielded if the meta file doesn't exist.
|
||||
|
||||
Example:
|
||||
>>> foo = pathlib.Path('foo.bar')
|
||||
>>> find_meta_files(foo)
|
||||
(pathlib.Path('foo.jpg'), pathlib.Path('foo.description'),
|
||||
pathlib.Path('foo.en.vtt'), pathlib.Path('foo.info.json'))
|
||||
"""
|
||||
suffix = path.suffix
|
||||
name, suffix, _ = str(path.name).rpartition(suffix)
|
||||
meta_file_exts = (('.jpg', '.webp', '.png'), ('.description',), ('.en.vtt', '.en.srt'), ('.info.json',))
|
||||
for meta_exts in meta_file_exts:
|
||||
for meta_ext in meta_exts:
|
||||
meta_path = path.with_suffix(meta_ext)
|
||||
if meta_path.exists():
|
||||
yield meta_path
|
||||
break
|
||||
else:
|
||||
yield None
|
||||
|
|
|
|||
|
|
@ -1,18 +1,21 @@
|
|||
import json
|
||||
import pathlib
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Generator
|
||||
from typing import Optional, Generator, Dict, List
|
||||
|
||||
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey, Computed
|
||||
from sqlalchemy.orm import relationship, Session, deferred
|
||||
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey
|
||||
from sqlalchemy.orm import relationship, Session
|
||||
from sqlalchemy.orm.collections import InstrumentedList
|
||||
|
||||
from wrolpi.common import Base, tsvector, ModelHelper, logger, get_media_directory
|
||||
from wrolpi.common import Base, ModelHelper, logger, get_media_directory
|
||||
from wrolpi.dates import now, TZDateTime
|
||||
from wrolpi.db import get_db_curs
|
||||
from wrolpi.downloader import Download, download_manager
|
||||
from wrolpi.errors import UnknownVideo, UnknownFile, UnknownDirectory
|
||||
from wrolpi.media_path import MediaPathType, MediaPath
|
||||
from wrolpi.errors import UnknownVideo
|
||||
from wrolpi.files.lib import refresh_directory_files_recursively, glob_shared_stem
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.media_path import MediaPathType
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
|
|
@ -22,18 +25,8 @@ class Video(ModelHelper, Base):
|
|||
id = Column(Integer, primary_key=True)
|
||||
channel_id = Column(Integer, ForeignKey('channel.id'))
|
||||
channel = relationship('Channel', primaryjoin='Video.channel_id==Channel.id', back_populates='videos')
|
||||
idempotency = Column(String)
|
||||
validated = Column(Boolean, default=False)
|
||||
|
||||
# File paths
|
||||
caption_path = Column(MediaPathType)
|
||||
description_path = Column(MediaPathType)
|
||||
ext = Column(String)
|
||||
info_json_path = Column(MediaPathType)
|
||||
poster_path = Column(MediaPathType)
|
||||
video_path = Column(MediaPathType)
|
||||
|
||||
caption = deferred(Column(String)) # slow to fetch
|
||||
censored = Column(Boolean, default=False)
|
||||
duration = Column(Integer)
|
||||
favorite = Column(TZDateTime)
|
||||
|
|
@ -46,18 +39,42 @@ class Video(ModelHelper, Base):
|
|||
view_count = Column(Integer)
|
||||
viewed = Column(TZDateTime)
|
||||
|
||||
textsearch = deferred(
|
||||
Column(tsvector, Computed('''to_tsvector('english'::regconfig,
|
||||
((COALESCE(title, ''::text) || ' '::text) ||
|
||||
COALESCE(caption, ''::text)))''')))
|
||||
# Associated Files
|
||||
caption_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
caption_file: File = relationship('File', primaryjoin='Video.caption_path==File.path')
|
||||
info_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
info_json_file: File = relationship('File', primaryjoin='Video.info_json_path==File.path')
|
||||
poster_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
poster_file: File = relationship('File', primaryjoin='Video.poster_path==File.path')
|
||||
video_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
|
||||
video_file: File = relationship('File', primaryjoin='Video.video_path==File.path')
|
||||
|
||||
def __repr__(self):
|
||||
video_path = self.video_path
|
||||
if self.video_path and isinstance(self.video_path, MediaPath):
|
||||
video_path = self.video_path.path.relative_to(get_media_directory())
|
||||
return f'<Video id={self.id} title={repr(self.title)} path={video_path} channel={self.channel_id} ' \
|
||||
v = self.video_file
|
||||
if self.video_path:
|
||||
video_path = self.video_path.path if hasattr(self.video_path, 'path') else self.video_path
|
||||
v = video_path.relative_to(get_media_directory())
|
||||
return f'<Video id={self.id} title={repr(self.title)} path={v} channel={self.channel_id} ' \
|
||||
f'source_id={repr(self.source_id)}>'
|
||||
|
||||
@classmethod
|
||||
def upsert(cls, video_file: File, session: Session):
|
||||
video = session.query(Video).filter_by(video_file=video_file).one_or_none()
|
||||
if not video:
|
||||
video = Video(video_file=video_file)
|
||||
video: Video
|
||||
video.video_path = video.video_path or video_file.path
|
||||
video.video_file = video.video_file or video_file
|
||||
video.size = video.video_path.stat().st_size
|
||||
session.add(video)
|
||||
|
||||
video.find_my_files()
|
||||
|
||||
# Claim the video file for this model.
|
||||
video.video_file.model = cls.__tablename__
|
||||
|
||||
return video
|
||||
|
||||
def dict(self) -> dict:
|
||||
d = super().dict()
|
||||
if self.channel_id:
|
||||
|
|
@ -72,35 +89,34 @@ class Video(ModelHelper, Base):
|
|||
from .common import minimize_video
|
||||
return minimize_video(self.dict())
|
||||
|
||||
def _clear_paths(self):
|
||||
self.caption_path = None
|
||||
self.description_path = None
|
||||
self.ext = None
|
||||
self.info_json_path = None
|
||||
self.poster_path = None
|
||||
self.video_path = None
|
||||
def _clear_files(self):
|
||||
# Use setattr to avoid confusing PyCharm
|
||||
setattr(self, 'caption_file', None)
|
||||
setattr(self, 'description_file', None)
|
||||
setattr(self, 'ext', None)
|
||||
setattr(self, 'info_json_file', None)
|
||||
setattr(self, 'poster_file', None)
|
||||
setattr(self, 'video_file', None)
|
||||
|
||||
def my_paths(self) -> Generator[Path, None, None]:
|
||||
def my_files(self) -> Generator[Path, None, None]:
|
||||
"""Return all paths of this Video. Returns nothing if all paths are None.""" # noqa
|
||||
if self.description_path: # noqa
|
||||
yield self.description_path.path
|
||||
if self.poster_path:
|
||||
yield self.poster_path.path
|
||||
if self.caption_path:
|
||||
yield self.caption_path.path
|
||||
if self.info_json_path:
|
||||
yield self.info_json_path.path
|
||||
if self.video_path:
|
||||
yield self.video_path.path
|
||||
if self.poster_file:
|
||||
yield self.poster_file.path
|
||||
if self.caption_file:
|
||||
yield self.caption_file.path
|
||||
if self.info_json_file:
|
||||
yield self.info_json_file.path
|
||||
if self.video_file:
|
||||
yield self.video_file.path
|
||||
|
||||
def delete(self):
|
||||
"""
|
||||
Remove all files related to this video. Add it to it's Channel's skip list.
|
||||
"""
|
||||
for path in self.my_paths():
|
||||
for path in self.my_files():
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
self._clear_paths()
|
||||
self._clear_files()
|
||||
|
||||
needs_save = bool(self.favorite)
|
||||
|
||||
|
|
@ -126,21 +142,15 @@ class Video(ModelHelper, Base):
|
|||
def set_viewed(self):
|
||||
self.viewed = now()
|
||||
|
||||
def get_info_json(self) -> Optional[JSON]:
|
||||
def get_info_json(self) -> Optional[Dict]:
|
||||
"""If this Video has an info_json file, return it's contents. Otherwise, return None."""
|
||||
if not self.info_json_path:
|
||||
info_json_path = self.info_json_file.path if self.info_json_file else self.info_json_path
|
||||
if not info_json_path:
|
||||
return
|
||||
|
||||
try:
|
||||
info_json_path = self.info_json_path.path if \
|
||||
isinstance(self.info_json_path, MediaPath) else self.info_json_path
|
||||
with open(info_json_path, 'rb') as fh:
|
||||
contents = json.load(fh)
|
||||
return contents
|
||||
except UnknownFile:
|
||||
pass
|
||||
except UnknownDirectory:
|
||||
pass
|
||||
with info_json_path.open('rb') as fh:
|
||||
return json.load(fh)
|
||||
except Exception as e:
|
||||
logger.warning(f'Unable to parse info json {self.info_json_path}', exc_info=e)
|
||||
return None
|
||||
|
|
@ -230,7 +240,7 @@ class Video(ModelHelper, Base):
|
|||
def __json__(self):
|
||||
from modules.videos.common import minimize_video_info_json
|
||||
info_json = minimize_video_info_json(self.get_info_json()) if self.info_json_path else None
|
||||
stem = self.video_path.path.stem if self.video_path else None
|
||||
stem = self.video_path.stem if self.video_path else None
|
||||
|
||||
d = dict(
|
||||
caption_path=self.caption_path,
|
||||
|
|
@ -249,44 +259,78 @@ class Video(ModelHelper, Base):
|
|||
upload_date=self.upload_date,
|
||||
url=self.url,
|
||||
validated=self.validated,
|
||||
video_path=self.video_path,
|
||||
view_count=self.view_count,
|
||||
video_path=self.video_path.relative_to(get_media_directory()),
|
||||
viewed=self.viewed,
|
||||
)
|
||||
return d
|
||||
|
||||
def validate(self):
|
||||
def validate(self, session: Session):
|
||||
"""Perform a validation of this video and it's files. Mark this video as validated if no errors occur."""
|
||||
if not self.video_path:
|
||||
if not self.video_file:
|
||||
# Can't validate if there is no video file.
|
||||
return False
|
||||
|
||||
from .lib import validate_video
|
||||
try:
|
||||
validate_video(self, self.channel.generate_posters if self.channel else False)
|
||||
validate_video(self, self.channel.generate_posters if self.channel else False, session)
|
||||
self.validated = True
|
||||
except Exception as e:
|
||||
logger.warning(f'Failed to validate video {self}', exc_info=e)
|
||||
|
||||
return self.validated
|
||||
|
||||
def check_for_corruption(self):
|
||||
"""Uses ffprobe to check if this video is corrupt."""
|
||||
def find_my_files(self):
|
||||
"""Search near the video file for it's associated files. (poster, caption, etc.)"""
|
||||
from modules.videos.common import match_video_files
|
||||
if not self.video_path:
|
||||
return
|
||||
raise ValueError("Can't find files when I don't have a video path!")
|
||||
|
||||
from .lib import check_for_video_corruption
|
||||
return check_for_video_corruption(self.video_path.path)
|
||||
# Associated files share the same stem, but different extensions.
|
||||
associated_paths = glob_shared_stem(self.video_path)
|
||||
associated_paths.pop(associated_paths.index(self.video_path))
|
||||
|
||||
# Get the session from the video first, fallback to the video file.
|
||||
session: Session = Session.object_session(self) or Session.object_session(self.video_file)
|
||||
|
||||
files = [File.upsert(i, session) for i in associated_paths]
|
||||
_, poster_file, description_file, caption_file, info_json_file = match_video_files(files)
|
||||
|
||||
if poster_file:
|
||||
poster_file.do_index()
|
||||
if caption_file:
|
||||
caption_file.do_index()
|
||||
if info_json_file:
|
||||
info_json_file.do_index()
|
||||
|
||||
session.flush([i for i in files])
|
||||
|
||||
self.poster_file = poster_file
|
||||
self.caption_file = caption_file
|
||||
self.info_json_file = info_json_file
|
||||
|
||||
@staticmethod
|
||||
def find_by_path(path, session):
|
||||
video = session.query(Video).filter_by(video_path=path).one_or_none()
|
||||
return video
|
||||
|
||||
@staticmethod
|
||||
def find_by_paths(paths: List[pathlib.Path], session) -> List:
|
||||
videos = list(session.query(Video).filter(Video.video_path.in_(paths)))
|
||||
return videos
|
||||
|
||||
@property
|
||||
def primary_path(self):
|
||||
return self.video_path
|
||||
|
||||
|
||||
class Channel(ModelHelper, Base):
|
||||
__tablename__ = 'channel'
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String)
|
||||
idempotency = Column(String)
|
||||
url = Column(String, unique=True)
|
||||
match_regex = Column(String)
|
||||
directory: MediaPath = Column(MediaPathType)
|
||||
directory: pathlib.Path = Column(MediaPathType)
|
||||
skip_download_videos = Column(ARRAY(String))
|
||||
generate_posters = Column(Boolean, default=False) # generating posters may delete files, and can be slow.
|
||||
calculate_duration = Column(Boolean, default=True)
|
||||
|
|
@ -337,6 +381,12 @@ class Channel(ModelHelper, Base):
|
|||
for key, value in data.items():
|
||||
setattr(self, key, value)
|
||||
|
||||
# We need an absolute directory.
|
||||
if isinstance(self.directory, pathlib.Path) and not self.directory.is_absolute():
|
||||
self.directory = get_media_directory() / self.directory
|
||||
elif isinstance(self.directory, str) and not pathlib.Path(self.directory).is_absolute():
|
||||
self.directory = get_media_directory() / self.directory
|
||||
|
||||
# All channels with a URL and download_frequency should have a download.
|
||||
session = Session.object_session(self)
|
||||
if download and not self.download_frequency:
|
||||
|
|
@ -360,7 +410,7 @@ class Channel(ModelHelper, Base):
|
|||
"""
|
||||
config = dict(
|
||||
calculate_duration=self.calculate_duration,
|
||||
directory=str(self.directory.path),
|
||||
directory=str(self.directory),
|
||||
download_frequency=self.download_frequency,
|
||||
favorites={},
|
||||
generate_posters=self.generate_posters,
|
||||
|
|
@ -400,6 +450,7 @@ class Channel(ModelHelper, Base):
|
|||
|
||||
def dict(self, with_statistics: bool = False):
|
||||
d = super(Channel, self).dict()
|
||||
d['directory'] = self.directory.relative_to(get_media_directory())
|
||||
if with_statistics:
|
||||
d['statistics'] = self.get_statistics()
|
||||
return d
|
||||
|
|
@ -425,3 +476,8 @@ class Channel(ModelHelper, Base):
|
|||
largest_video=largest_video,
|
||||
)
|
||||
return statistics
|
||||
|
||||
async def refresh_files(self):
|
||||
"""Refresh all files within this Channel's directory. Mark this channel as refreshed."""
|
||||
await refresh_directory_files_recursively(self.directory)
|
||||
self.refreshed = True
|
||||
|
|
|
|||
|
|
@ -131,7 +131,9 @@ class ExtendedVideo(Video):
|
|||
|
||||
@dataclass
|
||||
class VideoResponse:
|
||||
video = ExtendedVideo
|
||||
file = ExtendedVideo
|
||||
prev = ExtendedVideo
|
||||
next = ExtendedVideo
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
import json
|
||||
import shutil
|
||||
from http import HTTPStatus
|
||||
from pathlib import Path
|
||||
|
||||
from modules.videos.models import Video
|
||||
from modules.videos.models import Video, Channel
|
||||
from modules.videos.video import lib as video_lib
|
||||
from wrolpi.downloader import DownloadFrequency
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
|
|
@ -11,18 +14,17 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
|
|||
subdir = test_directory / 'subdir'
|
||||
subdir.mkdir()
|
||||
|
||||
# video1 is in a subdirectory.
|
||||
video1 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
|
||||
# video1 is in a subdirectory, move its files into the subdirectory.
|
||||
video1 = video_factory(channel_id=simple_channel.id, with_video_file=subdir / 'video1.mp4',
|
||||
with_info_json=True,
|
||||
with_poster_ext='jpg')
|
||||
test_session.commit()
|
||||
shutil.move(video1.video_path.path, subdir / video1.video_path.path.name)
|
||||
shutil.move(video1.poster_path.path, subdir / video1.poster_path.path.name)
|
||||
video1.video_path = subdir / video1.video_path.path.name
|
||||
test_session.delete(video1.poster_file)
|
||||
video1.poster_path = None
|
||||
# video2 is in the test directory.
|
||||
video2 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
|
||||
with_poster_ext='jpg')
|
||||
video2.poster_path = video1.poster_path = None
|
||||
video2.poster_path = video2.poster_file = video1.poster_path = None
|
||||
test_session.commit()
|
||||
|
||||
assert not video1.size, 'video1 should not have size during creation'
|
||||
|
|
@ -36,27 +38,73 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
|
|||
orphan_poster.touch()
|
||||
|
||||
# Create a bogus video in the channel.
|
||||
bogus = Video(video_path='foo', channel_id=simple_channel.id)
|
||||
video_file = File(path=test_directory / 'foo')
|
||||
video_file.idempotency = None # File was not added in the current refresh.
|
||||
bogus = Video(video_file=video_file, channel_id=simple_channel.id)
|
||||
test_session.add(bogus)
|
||||
test_session.commit()
|
||||
|
||||
test_client.post('/api/videos/refresh')
|
||||
test_client.post('/api/files/refresh')
|
||||
|
||||
# Posters were found during refresh.
|
||||
assert video1.poster_path
|
||||
assert 'subdir' in str(video1.poster_path.path)
|
||||
assert 'subdir' in str(video1.poster_path)
|
||||
assert video2.poster_path
|
||||
# Missing video3 was found
|
||||
video3: Video = test_session.query(Video).filter_by(id=4).one()
|
||||
assert video3.video_path.path == vid3
|
||||
assert video3.video_path != video1.video_path
|
||||
assert video3.video_path != video2.video_path
|
||||
assert video3.video_path == vid3
|
||||
assert len(set(test_session.query(Video.video_path))) == 3, 'Should have 3 distinct video files'
|
||||
# Bogus video was removed.
|
||||
assert not any('foo' in str(i.video_path.path) for i in test_session.query(Video).all())
|
||||
assert not any('foo' in str(i.video_path) for i in test_session.query(Video).all())
|
||||
# Orphan file was not deleted.
|
||||
assert orphan_poster.is_file(), 'Orphan poster was removed!'
|
||||
|
||||
assert video1.size, 'video1 size was not found'
|
||||
|
||||
# Remove video1's poster, video1 should be updated.
|
||||
video1_video_path = str(video1.video_path)
|
||||
video1.poster_path.unlink()
|
||||
test_client.post('/api/files/refresh')
|
||||
video1 = test_session.query(Video).filter_by(video_path=video1_video_path).one()
|
||||
assert not video1.poster_path
|
||||
|
||||
|
||||
def test_channels_with_videos(test_session, test_client, test_directory, channel_factory, video_factory):
|
||||
channel1 = channel_factory('channel1', name='channel1')
|
||||
channel2 = channel_factory('channel2', name='channel2')
|
||||
vid1 = video_factory(channel_id=channel1.id, with_video_file=True)
|
||||
vid2 = video_factory(channel_id=channel2.id, with_video_file=True)
|
||||
vid3 = video_factory(with_video_file=True)
|
||||
test_session.commit()
|
||||
|
||||
vid1_path = vid1.video_path
|
||||
vid2_path = vid2.video_path
|
||||
vid3_path = vid3.video_path
|
||||
|
||||
test_session.delete(vid1)
|
||||
test_session.delete(vid2)
|
||||
test_session.delete(vid3)
|
||||
test_session.commit()
|
||||
|
||||
assert test_session.query(Video).count() == 0, 'Videos were not deleted'
|
||||
assert test_session.query(Channel).count() == 2, 'Channels were deleted!'
|
||||
assert vid1_path.is_file(), 'Video file was deleted'
|
||||
assert vid2_path.is_file(), 'Video file was deleted'
|
||||
assert vid3_path.is_file(), 'Video file was deleted'
|
||||
|
||||
test_client.post('/api/files/refresh')
|
||||
|
||||
assert test_session.query(Video).count() == 3, 'Did not find correct amount of video files.'
|
||||
assert {i[0] for i in test_session.query(Channel.name)} == {'channel1', 'channel2'}, 'Channels were changed.'
|
||||
|
||||
vid1: Video = test_session.query(Video).filter_by(video_path=vid1_path).one()
|
||||
vid2: Video = test_session.query(Video).filter_by(video_path=vid2_path).one()
|
||||
vid3: Video = test_session.query(Video).filter_by(video_path=vid3_path).one()
|
||||
|
||||
assert vid1.channel == channel1
|
||||
assert vid2.channel == channel2
|
||||
assert vid3.channel is None
|
||||
|
||||
|
||||
def test_api_download_channel(test_session, test_client, simple_channel):
|
||||
"""A Channel download (a catalog update) can be triggered via the API."""
|
||||
|
|
@ -70,3 +118,75 @@ def test_api_download_channel(test_session, test_client, simple_channel):
|
|||
test_session.commit()
|
||||
request, response = test_client.post(f'/api/videos/download/{simple_channel.id}')
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT
|
||||
|
||||
|
||||
def test_search_videos_file(test_client, test_session, test_directory, video_with_search_factory):
|
||||
"""Test that videos can be searched and that their order is by their textsearch rank."""
|
||||
# These captions have repeated letters, so they will be higher in the ranking.
|
||||
videos = [
|
||||
('1', 'b b b b e d d'),
|
||||
('2', '2 b b b d'),
|
||||
('3', 'b b'),
|
||||
('4', 'b e e'),
|
||||
('5', ''),
|
||||
]
|
||||
for title, caption in videos:
|
||||
video_with_search_factory(title=title, d_text=caption)
|
||||
test_session.commit()
|
||||
|
||||
def assert_search(search_str, expected, limit=20, total=None):
|
||||
d = json.dumps({'search_str': search_str, 'limit': limit})
|
||||
_, resp = test_client.post('/api/videos/search', content=d)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
response_ids = [i['video']['id'] for i in resp.json['files']]
|
||||
assert response_ids == expected
|
||||
assert resp.json['totals']['files'] == (total or len(expected))
|
||||
|
||||
# Repeated runs should return the same result
|
||||
for _ in range(2):
|
||||
# Only videos with a b are returned, ordered by the amount of b's
|
||||
assert_search('b', [1, 2, 3, 4])
|
||||
|
||||
# Only two captions have e
|
||||
assert_search('e', [4, 1])
|
||||
|
||||
# Only two captions have d
|
||||
assert_search('d', [1, 2])
|
||||
|
||||
# 5 can be gotten by it's title
|
||||
assert_search('5', [5])
|
||||
|
||||
# only video 1 has e and d
|
||||
assert_search('e d', [1])
|
||||
|
||||
# video 1 and 4 have b and e, but 1 has more
|
||||
assert_search('b e', [1, 4])
|
||||
|
||||
# Check totals are correct even with a limit
|
||||
assert_search('b', [1, 2], limit=2, total=4)
|
||||
|
||||
|
||||
def test_search_videos(test_client, test_session, video_factory, assert_video_search, simple_channel):
|
||||
"""Search the Video table. This does not need to use a join with the File table."""
|
||||
vid1 = video_factory(upload_date='2022-09-16', with_video_file=True)
|
||||
vid2 = video_factory(upload_date='2022-09-17', with_video_file=True, channel_id=simple_channel.id)
|
||||
vid3 = video_factory(upload_date='2022-09-18', with_video_file=True)
|
||||
vid1.set_favorite(True)
|
||||
test_session.commit()
|
||||
assert test_session.query(Video).count() == 3
|
||||
assert test_session.query(File).count() == 3
|
||||
|
||||
assert_video_search(assert_total=3, assert_ids=[vid1.id, vid2.id, vid3.id], order_by='upload_date')
|
||||
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id, vid1.id], order_by='-upload_date')
|
||||
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id], order_by='-upload_date', limit=2)
|
||||
assert_video_search(assert_total=3, assert_ids=[vid2.id, vid1.id], order_by='-upload_date', limit=2, offset=1)
|
||||
assert_video_search(assert_total=1, assert_ids=[vid2.id], order_by='-upload_date', channel_id=simple_channel.id)
|
||||
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'])
|
||||
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'], limit=1)
|
||||
|
||||
# No results, no total is returned from the DB.
|
||||
assert_video_search(assert_total=0, assert_ids=[], filters=['favorite'], offset=1)
|
||||
|
||||
# Check all order_by.
|
||||
for order_by in video_lib.VIDEO_ORDERS:
|
||||
assert_video_search(order_by=order_by)
|
||||
|
|
|
|||
|
|
@ -1,104 +0,0 @@
|
|||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from modules.videos import captions
|
||||
from modules.videos.models import Video
|
||||
from wrolpi.db import get_db_curs, get_db_context
|
||||
from wrolpi.test.common import wrap_test_db, TestAPI
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
class TestCaption(TestAPI):
|
||||
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
self.vtt_path1 = Path(self.tmp_dir.name) / 'example1.en.vtt'
|
||||
self.vtt_path1.touch()
|
||||
self.vtt_path1.write_text((PROJECT_DIR / 'test/example1.en.vtt').read_text())
|
||||
self.vtt_path2 = Path(self.tmp_dir.name) / 'example2.en.vtt'
|
||||
self.vtt_path2.touch()
|
||||
self.vtt_path2.write_text((PROJECT_DIR / 'test/example2.en.vtt').read_text())
|
||||
|
||||
def test_example_en_vtt(self):
|
||||
expected = ['okay welcome to this session this is',
|
||||
'okay welcome to this session this is',
|
||||
'okay welcome to this session this is\ncalled the kinetic bunny need to meet',
|
||||
'called the kinetic bunny need to meet',
|
||||
'called the kinetic bunny need to meet\nthe virtual bonnie emilio stop this is',
|
||||
'the virtual bonnie emilio stop this is',
|
||||
'the virtual bonnie emilio stop this is\njana car who just to give a little bit',
|
||||
'jana car who just to give a little bit',
|
||||
'jana car who just to give a little bit\nof background information we come from a',
|
||||
'of background information we come from a',
|
||||
'of background information we come from a\ncompany called le code from finland and']
|
||||
self.assertEqual(list(captions.get_caption_text(self.vtt_path1)), expected)
|
||||
|
||||
expected = ['okay welcome to this session this is',
|
||||
'called the kinetic bunny need to meet',
|
||||
'the virtual bonnie emilio stop this is',
|
||||
'jana car who just to give a little bit',
|
||||
'of background information we come from a',
|
||||
'company called le code from finland and']
|
||||
self.assertEqual(list(captions.get_unique_caption_lines(self.vtt_path1)), expected)
|
||||
|
||||
@wrap_test_db
|
||||
def test_process_captions(self):
|
||||
_, session = get_db_context()
|
||||
video1 = Video(title='scream', caption_path=str(self.vtt_path1))
|
||||
session.add(video1)
|
||||
session.commit()
|
||||
video1.caption = captions.get_captions(video1.caption_path.path)
|
||||
video2 = Video(title='bar', caption_path=str(self.vtt_path2))
|
||||
session.add(video2)
|
||||
session.commit()
|
||||
video2.caption = captions.get_captions(video2.caption_path.path)
|
||||
|
||||
session.commit()
|
||||
session.flush()
|
||||
session.refresh(video1)
|
||||
session.refresh(video2)
|
||||
|
||||
# Get the video from the DB
|
||||
video1 = session.query(Video).filter_by(id=video1.id).one()
|
||||
self.assertIsNotNone(video1.caption)
|
||||
video2 = session.query(Video).filter_by(id=video2.id).one()
|
||||
self.assertIsNotNone(video2.caption)
|
||||
|
||||
# Search using the tsvector, "sessions" never actually appears in the text, but "session" does
|
||||
with get_db_curs() as curs:
|
||||
def select_textsearch(*args):
|
||||
curs.execute('SELECT id FROM video WHERE textsearch @@ to_tsquery(%s) ORDER BY id', args)
|
||||
|
||||
select_textsearch('sessions')
|
||||
self.assertEqual(curs.fetchall(), [[1, ]])
|
||||
# Matches video1.title and video2.caption
|
||||
select_textsearch('scream')
|
||||
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
|
||||
# Matches video1.title and video2.caption
|
||||
select_textsearch('scream | sessions')
|
||||
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
|
||||
# Only matches video1.title
|
||||
select_textsearch('scream & sessions')
|
||||
self.assertEqual(curs.fetchall(), [[1, ]])
|
||||
# Matches neither
|
||||
select_textsearch('scream & sess')
|
||||
self.assertEqual(curs.fetchall(), [])
|
||||
# Matches video2.caption
|
||||
select_textsearch('yawn | sess')
|
||||
self.assertEqual(curs.fetchall(), [[2, ]])
|
||||
# Matches video2.caption
|
||||
select_textsearch('yawn')
|
||||
self.assertEqual(curs.fetchall(), [[2, ]])
|
||||
# Matches video2.title
|
||||
select_textsearch('bar')
|
||||
self.assertEqual(curs.fetchall(), [[2, ]])
|
||||
|
||||
|
||||
def test_bad_caption(test_directory):
|
||||
"""A caption file that cannot be parsed is ignored."""
|
||||
bad_caption_file = PROJECT_DIR / 'test/bad_caption.en.vtt'
|
||||
test_file = test_directory / 'bad_caption.en.vtt'
|
||||
shutil.copy(bad_caption_file, test_file)
|
||||
|
||||
result = captions.get_captions(test_file)
|
||||
assert result is None
|
||||
|
|
@ -244,7 +244,7 @@ def test_update_censored_videos(test_session, video_factory, simple_channel):
|
|||
check_censored([(vid1.id, True), (vid2.id, True), (vid3.id, True), (vid4.id, False)])
|
||||
|
||||
|
||||
def test_import_favorites(test_session, simple_channel, video_factory, test_channels_config):
|
||||
def test_import_favorites(test_session, test_directory, simple_channel, video_factory, test_channels_config):
|
||||
"""
|
||||
A favorited Video will be preserved through everything (channel deletion, DB wipe) and can be imported and will be
|
||||
favorited again.
|
||||
|
|
@ -254,14 +254,13 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
|
|||
video_factory(channel_id=simple_channel.id) # never favorited
|
||||
vid2 = video_factory(channel_id=None) # has no channel
|
||||
vid3 = video_factory(channel_id=simple_channel.id)
|
||||
favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
|
||||
vid3.favorite = favorite
|
||||
favorite = vid3.favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
|
||||
test_session.commit()
|
||||
vid2_video_path = vid2.video_path.path
|
||||
vid2_video_path = vid2.video_path
|
||||
|
||||
favorites = {
|
||||
str(simple_channel.directory.relative): {vid3.video_path.path.name: {'favorite': favorite}},
|
||||
'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}},
|
||||
str(simple_channel.directory.relative_to(test_directory)): {vid3.video_path.name: {'favorite': favorite}},
|
||||
'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}},
|
||||
}
|
||||
|
||||
# Save config, verify that favorite is set.
|
||||
|
|
@ -299,7 +298,7 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
|
|||
# Deleting the Video in the model really removes the favorite status.
|
||||
vid3.delete()
|
||||
config = get_channels_config()
|
||||
assert config.favorites == {'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}}}
|
||||
assert config.favorites == {'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}}}
|
||||
import_and_verify([vid2.id])
|
||||
|
||||
|
||||
|
|
@ -369,11 +368,11 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
|
|||
vid4 = video_factory(title='vid4')
|
||||
test_session.commit()
|
||||
|
||||
channel1_name, channel_1_directory = channel1.name, channel1.directory.path
|
||||
channel2_name, channel_2_directory = channel2.name, channel2.directory.path
|
||||
channel1_name, channel_1_directory = channel1.name, channel1.directory
|
||||
channel2_name, channel_2_directory = channel2.name, channel2.directory
|
||||
|
||||
assert channel1.directory.path.exists() and channel1.directory.path.is_absolute()
|
||||
assert channel2.directory.path.exists() and channel2.directory.path.is_absolute()
|
||||
assert channel1.directory.exists() and channel1.directory.is_absolute()
|
||||
assert channel2.directory.exists() and channel2.directory.is_absolute()
|
||||
|
||||
# Channel's used to have a relative directory in the DB.
|
||||
test_session.execute('UPDATE channel SET directory=:directory WHERE id=2',
|
||||
|
|
@ -390,9 +389,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
|
|||
channel['download_frequency'] = DownloadFrequency.biweekly
|
||||
# Change the favorites to the old "link" method as well.
|
||||
channels_config.favorites = {
|
||||
sanitize_link(channel1.name): {str(vid1.video_path.path): {'favorite': now()}},
|
||||
sanitize_link(channel2.name): {str(vid2.video_path.path): {'favorite': now()}},
|
||||
'NO CHANNEL': {str(vid3.video_path.path): {'favorite': now()}},
|
||||
sanitize_link(channel1.name): {str(vid1.video_path): {'favorite': now()}},
|
||||
sanitize_link(channel2.name): {str(vid2.video_path): {'favorite': now()}},
|
||||
'NO CHANNEL': {str(vid3.video_path): {'favorite': now()}},
|
||||
}
|
||||
channels_config.save()
|
||||
|
||||
|
|
@ -407,9 +406,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
|
|||
# No new channels were created. Existing channels were updated.
|
||||
assert test_session.query(Channel).count() == 2
|
||||
channel1, channel2 = test_session.query(Channel).order_by(Channel.id)
|
||||
assert channel1.name == channel1_name and str(channel1.directory.path) == str(channel_1_directory)
|
||||
assert channel1.name == channel1_name and str(channel1.directory) == str(channel_1_directory)
|
||||
# Channel2's directory is now absolute.
|
||||
assert channel2.name == channel2_name and str(channel2.directory.path) == str(channel_2_directory)
|
||||
assert channel2.name == channel2_name and str(channel2.directory) == str(channel_2_directory)
|
||||
assert channel1.download_frequency == DownloadFrequency.biweekly
|
||||
assert channel2.download_frequency == DownloadFrequency.biweekly
|
||||
# Videos were favorited by their channel "link".
|
||||
|
|
|
|||
|
|
@ -14,12 +14,14 @@ from modules.videos.models import Channel, Video
|
|||
from wrolpi.db import get_db_context
|
||||
from wrolpi.downloader import DownloadManager, Download, DownloadResult
|
||||
from wrolpi.errors import InvalidDownload
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.test.common import TestAPI
|
||||
from wrolpi.test.test_downloader import HTTPDownloader
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
|
||||
|
||||
def test_find_all_missing_videos(test_session, channel_factory, video_factory):
|
||||
@pytest.mark.asyncio
|
||||
async def test_find_all_missing_videos(test_session, channel_factory, video_factory):
|
||||
channel1 = channel_factory(url='some url')
|
||||
channel1.info_json = {'entries': [{'id': 'foo', 'view_count': 0}]}
|
||||
test_session.commit()
|
||||
|
|
@ -29,14 +31,14 @@ def test_find_all_missing_videos(test_session, channel_factory, video_factory):
|
|||
video_factory()
|
||||
|
||||
# No missing videos
|
||||
assert [] == list(find_all_missing_videos(channel1.id))
|
||||
assert [] == [i async for i in find_all_missing_videos(channel1.id)]
|
||||
|
||||
# Create a video that has no video file.
|
||||
video = Video(title='needs to be downloaded', channel_id=channel1.id, source_id='foo')
|
||||
test_session.add(video)
|
||||
test_session.commit()
|
||||
|
||||
missing = list(find_all_missing_videos(channel1.id))
|
||||
missing = [i async for i in find_all_missing_videos(channel1.id)]
|
||||
assert len(missing) == 1
|
||||
|
||||
# The video returned is the one we faked.
|
||||
|
|
@ -175,7 +177,7 @@ async def test_video_download_no_channel(test_session, video_download_manager, v
|
|||
mock_video_process_runner.assert_called_once()
|
||||
|
||||
video: Video = test_session.query(Video).one()
|
||||
assert str(video.video_path.path) == f'{channel_dir}/video.mp4'
|
||||
assert str(video.video_path) == f'{channel_dir}/video.mp4'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -364,19 +366,23 @@ async def test_invalid_download_url(test_session, test_download_manager, mock_vi
|
|||
|
||||
@pytest.mark.asyncio
|
||||
async def test_video_download_1(test_session, test_directory, simple_channel, video_download_manager,
|
||||
mock_video_process_runner):
|
||||
mock_video_process_runner, image_file):
|
||||
"""A video download is performed, files are associated."""
|
||||
simple_channel.source_id = example_video_json['channel_id']
|
||||
simple_channel.directory = test_directory / 'videos/channel name'
|
||||
simple_channel.directory.mkdir(parents=True)
|
||||
|
||||
video_file = simple_channel.directory / 'a video.mp4'
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_file)
|
||||
video_path = simple_channel.directory / 'a video.mp4'
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
|
||||
# Create a poster file which was downloaded.
|
||||
poster_path = video_path.with_suffix('.png')
|
||||
image_file.rename(poster_path)
|
||||
|
||||
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
|
||||
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
|
||||
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename:
|
||||
mock_extract_info.return_value = example_video_json
|
||||
mock_prepare_filename.return_value = (video_file, {'id': 'foo'})
|
||||
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
|
||||
|
||||
video_download_manager.create_download(url)
|
||||
await video_download_manager.wait_for_all_downloads()
|
||||
|
|
@ -389,7 +395,10 @@ async def test_video_download_1(test_session, test_directory, simple_channel, vi
|
|||
assert test_session.query(Channel).one()
|
||||
|
||||
video: Video = test_session.query(Video).one()
|
||||
assert video.video_path.path.is_absolute()
|
||||
assert video.video_path.is_absolute(), 'Video path is not absolute'
|
||||
assert video.poster_path == poster_path, 'Video poster was not discovered'
|
||||
assert video.validated, 'Video was not validated'
|
||||
assert video.video_file and video.video_file.indexed, 'Video was not indexed'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -428,9 +437,9 @@ example_playlist_json = {
|
|||
'ie_key': 'Youtube',
|
||||
'title': 'video 2 title',
|
||||
'uploader': None,
|
||||
'url': 'https://youtube.com/watch?v=video_2_url',
|
||||
'url': 'https://www.youtube.com/shorts/video_2_url',
|
||||
'view_count': 1413,
|
||||
'webpage_url': 'https://youtube.com/watch?v=video_2_url'},
|
||||
'webpage_url': 'https://www.youtube.com/shorts/video_2_url'},
|
||||
{'_type': 'url',
|
||||
'description': None,
|
||||
'duration': None,
|
||||
|
|
@ -438,9 +447,9 @@ example_playlist_json = {
|
|||
'ie_key': 'Youtube',
|
||||
'title': 'video 1 title',
|
||||
'uploader': None,
|
||||
'url': 'https://youtube.com/watch?v=video_1_url',
|
||||
'url': 'https://www.youtube.com/watch?v=video_1_url',
|
||||
'view_count': 58504,
|
||||
'webpage_url': 'https://youtube.com/watch?v=video_1_url'},
|
||||
'webpage_url': 'https://www.youtube.com/watch?v=video_1_url'},
|
||||
{'_type': 'url',
|
||||
'description': None,
|
||||
'duration': None,
|
||||
|
|
@ -473,15 +482,14 @@ example_playlist_json = {
|
|||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_playlist(test_session, test_directory, mock_video_extract_info, video_download_manager):
|
||||
"""
|
||||
All videos in a playlist can be downloaded for it's Channel.
|
||||
"""
|
||||
"""All videos in a playlist can be downloaded for it's Channel."""
|
||||
download = Download(url='playlist url')
|
||||
test_session.add(download)
|
||||
channel = get_or_create_channel(example_playlist_json['channel_id'], download.url, example_channel_json['uploader'])
|
||||
test_session.add(Video(url='https://youtube.com/watch?v=video_1_url',
|
||||
source_id='video 1 id', channel_id=channel.id, video_path='video exists',
|
||||
poster_path='poster exists'))
|
||||
video_file = File(path=test_directory / 'video file.mp4')
|
||||
test_session.add(video_file)
|
||||
test_session.add(Video(url='https://www.youtube.com/watch?v=video_1_url',
|
||||
source_id='video 1 id', channel_id=channel.id, video_file=video_file))
|
||||
test_session.commit()
|
||||
|
||||
mock_video_extract_info.return_value = example_playlist_json # Playlist info is fetched first.
|
||||
|
|
@ -491,6 +499,10 @@ async def test_download_playlist(test_session, test_directory, mock_video_extrac
|
|||
result = await channel_downloader.do_download(download)
|
||||
assert result.success is True, 'Download was not successful'
|
||||
assert set(result.downloads) == {
|
||||
'https://youtube.com/watch?v=video_2_url',
|
||||
'https://www.youtube.com/watch?v=video_2_url', # Shorts is converted to regular URL.
|
||||
'https://youtube.com/watch?v=video_3_url',
|
||||
}
|
||||
|
||||
|
||||
def test_channel_download_match_title(test_session, test_directory, simple_channel, video_download_manager):
|
||||
pass
|
||||
|
|
|
|||
|
|
@ -1,25 +1,20 @@
|
|||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
from datetime import datetime
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
import sqlalchemy
|
||||
|
||||
from modules.videos import lib
|
||||
from modules.videos.common import apply_info_json
|
||||
from modules.videos.lib import validate_videos, parse_video_file_name, upsert_video
|
||||
from modules.videos.lib import parse_video_file_name, validate_video, get_statistics
|
||||
from modules.videos.models import Video
|
||||
from modules.videos.video.lib import video_search
|
||||
from wrolpi.dates import local_timezone
|
||||
from wrolpi.vars import PROJECT_DIR
|
||||
from wrolpi.files import lib as files_lib
|
||||
from wrolpi.files.models import File
|
||||
|
||||
|
||||
def test_search_censored_videos(test_session, simple_channel):
|
||||
def test_search_censored_videos(test_session, simple_channel, video_factory):
|
||||
for i in map(str, range(50)):
|
||||
test_session.add(Video(source_id=i, channel=simple_channel, video_path='foo'))
|
||||
vid = Video(source_id='51', video_path='bar') # this should never be modified because it has no channel
|
||||
video_factory(channel_id=simple_channel.id, source_id=i)
|
||||
vid = video_factory(source_id=51) # this should never be modified because it has no channel
|
||||
test_session.add(vid)
|
||||
test_session.commit()
|
||||
|
||||
|
|
@ -32,135 +27,23 @@ def test_search_censored_videos(test_session, simple_channel):
|
|||
|
||||
# All source_ids are in the entries.
|
||||
set_entries(map(str, range(50)))
|
||||
videos, total = video_search(filters=['censored'], order_by='id')
|
||||
assert [i['source_id'] for i in videos] == []
|
||||
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
|
||||
assert [i['video']['source_id'] for i in videos] == []
|
||||
assert total == 0
|
||||
|
||||
# First 5 are censored.
|
||||
set_entries(map(str, range(5, 50)))
|
||||
videos, total = video_search(filters=['censored'], order_by='id')
|
||||
assert [i['source_id'] for i in videos] == [str(i) for i in range(5)]
|
||||
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
|
||||
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(5)]
|
||||
assert total == 5
|
||||
|
||||
# First 25 are censored.
|
||||
set_entries(map(str, range(25, 50)))
|
||||
videos, total = video_search(filters=['censored'], order_by='id')
|
||||
assert [i['source_id'] for i in videos] == [str(i) for i in range(20)]
|
||||
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
|
||||
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(20)]
|
||||
assert total == 25
|
||||
|
||||
|
||||
def test_validate_videos(test_session, simple_channel, video_factory):
|
||||
"""
|
||||
Videos that aren't validated should have their data filled in while being validated.
|
||||
"""
|
||||
simple_channel.generate_posters = True
|
||||
|
||||
vid1_json = {'title': 'info's title'}
|
||||
vid3_json = {'duration': 100}
|
||||
vid5_json = {'view_count': 42}
|
||||
vid6_json = {'webpage_url': 'https://example.com/webpage', 'url': 'https://example.com/url'}
|
||||
|
||||
vid1 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid1_json)
|
||||
vid2 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
|
||||
vid3 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid3_json)
|
||||
vid4 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
|
||||
vid5 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid5_json)
|
||||
vid6 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid6_json)
|
||||
vid7 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg')
|
||||
vid8 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='webp')
|
||||
vid9 = video_factory(simple_channel.id, with_video_file=True)
|
||||
|
||||
# These videos are missing this data, and a refresh should fill them in.
|
||||
vid1.title = None
|
||||
vid2.caption = None
|
||||
vid2.title = None
|
||||
vid3.duration = None
|
||||
vid4.size = None
|
||||
vid5.view_count = None
|
||||
vid6.url = None
|
||||
vid7.poster_path = None
|
||||
test_session.commit()
|
||||
|
||||
# Write the associated files. This data should be processed.
|
||||
vid2.caption_path = vid2.video_path.path.with_suffix('.en.vtt')
|
||||
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', vid2.caption_path)
|
||||
test_session.commit()
|
||||
|
||||
# All Videos are validated, all data is filled out. The info json data is trusted first.
|
||||
validate_videos()
|
||||
assert all([i.validated for i in test_session.query(Video)])
|
||||
assert vid1.title == "info's title"
|
||||
assert vid2.caption.startswith('okay welcome')
|
||||
assert 'mp4' not in vid2.title
|
||||
assert vid3.duration == 100
|
||||
assert vid4.size == 1055736
|
||||
assert vid5.view_count == 42
|
||||
assert vid6.url == 'https://example.com/webpage'
|
||||
assert str(vid7.poster_path).endswith('.jpg')
|
||||
assert str(vid8.poster_path).endswith('.jpg')
|
||||
assert str(vid9.poster_path).endswith('.jpg') # this was generated from the video file.
|
||||
|
||||
vid1.title = None
|
||||
vid3.duration = None
|
||||
vid5.view_count = None
|
||||
vid6.url = None
|
||||
vid9.poster_path = None
|
||||
test_session.commit()
|
||||
|
||||
# Validation does not happen because all are still validated.
|
||||
validate_videos()
|
||||
|
||||
# All videos need to be validated again.
|
||||
for i in test_session.query(Video):
|
||||
i.validated = False
|
||||
# Use backup methods to fetch data.
|
||||
vid1.info_json_path = None
|
||||
vid3.info_json_path = None
|
||||
vid5.info_json_path = None
|
||||
vid6.info_json_path.path.write_text(json.dumps({'url': 'https://example.com/url'}))
|
||||
test_session.commit()
|
||||
|
||||
validate_videos()
|
||||
assert all([i.validated for i in test_session.query(Video)])
|
||||
assert vid1.title != "info's title" # file name is used.
|
||||
assert vid3.duration == 5 # video file duration is used.
|
||||
assert vid5.view_count is None
|
||||
assert vid6.url == 'https://example.com/url' # url is used as backup url.
|
||||
# Posters are both JPEG format. Webp file is removed.
|
||||
assert vid7.poster_path.path.is_file()
|
||||
assert vid8.poster_path.path.is_file()
|
||||
assert not vid8.poster_path.path.with_suffix('.webp').is_file()
|
||||
assert str(vid9.poster_path).endswith('.jpg') # the generated file was rediscovered.
|
||||
|
||||
|
||||
def test_validate_video_exception(test_session, simple_channel, video_factory):
|
||||
"""
|
||||
Test that even if a Video cannot be validated, the other Videos will still be validated.
|
||||
"""
|
||||
vid1 = video_factory(simple_channel.id, with_video_file=True)
|
||||
vid2 = video_factory(simple_channel.id, with_video_file=True)
|
||||
# The validator should not even check this Video because it does not have a video file.
|
||||
vid3 = video_factory(simple_channel.id)
|
||||
vid3.video_path = None
|
||||
assert not all([i.validated for i in test_session.query(Video)])
|
||||
test_session.commit()
|
||||
|
||||
with mock.patch('modules.videos.lib.process_video_info_json') as mock_process_video_info_json:
|
||||
mock_process_video_info_json.side_effect = [
|
||||
Exception('skip this video'),
|
||||
(None, None, None, None),
|
||||
('should not be set to video 3', None, None, None), # This should not be called!
|
||||
]
|
||||
validate_videos()
|
||||
# Only first two videos are validated.
|
||||
assert mock_process_video_info_json.call_count == 2, mock_process_video_info_json.call_count
|
||||
test_session.commit()
|
||||
|
||||
assert not vid1.validated
|
||||
assert vid2.validated
|
||||
assert not vid3.validated
|
||||
|
||||
|
||||
@pytest.mark.parametrize('file_name,expected', [
|
||||
('channel_20000101_12345678910_ some title.mp4', ('channel', '20000101', '12345678910', 'some title')),
|
||||
('channel name_NA_12345678910_ some title.mp4', ('channel name', None, '12345678910', 'some title')),
|
||||
|
|
@ -186,67 +69,120 @@ def test_parse_video_file_name(file_name, expected):
|
|||
assert parse_video_file_name(video_path) == expected, f'{file_name} != {expected}'
|
||||
|
||||
|
||||
def test_validate_does_not_generate_when_disabled(test_session, channel_factory, video_factory):
|
||||
@pytest.mark.asyncio
|
||||
async def test_associated_files(test_session, test_directory, video_file, image_file, vtt_file1):
|
||||
"""A Video file has files related to it. They should be attached to the Video record."""
|
||||
caption_file = video_file.with_suffix('.en.vtt')
|
||||
vtt_file1.rename(caption_file)
|
||||
poster_file = video_file.with_suffix('.jpeg')
|
||||
image_file.rename(poster_file)
|
||||
|
||||
await files_lib._refresh_all_files()
|
||||
test_session.commit()
|
||||
|
||||
# All files were found.
|
||||
file_paths = {i.path for i in test_session.query(File)}
|
||||
assert len(file_paths) == 3
|
||||
|
||||
video: Video = test_session.query(Video).one()
|
||||
video_id = video.id
|
||||
assert video.video_file == video_file
|
||||
assert video.poster_file == poster_file
|
||||
assert video.caption_file == caption_file
|
||||
assert video.video_file.modification_datetime
|
||||
assert video.video_file.size
|
||||
assert video.video_file.indexed
|
||||
|
||||
await files_lib._refresh_all_files()
|
||||
test_session.commit()
|
||||
|
||||
# No new files were added.
|
||||
assert {i.path for i in test_session.query(File)} == file_paths
|
||||
# Same video exists.
|
||||
video: Video = test_session.query(Video).one()
|
||||
assert video.id == video_id, 'Video id changed. Was the video recreated?'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unassociated_video(test_session, test_directory, video_file):
|
||||
"""A Video file without associated files is still a Video."""
|
||||
# Move the video file into a new subdirectory. The video file should still be found.
|
||||
sub_dir = test_directory / 'subdir'
|
||||
sub_dir.mkdir()
|
||||
new_video_file_path = sub_dir / video_file.name
|
||||
video_file.rename(new_video_file_path)
|
||||
video_file = new_video_file_path
|
||||
|
||||
await files_lib._refresh_all_files()
|
||||
test_session.commit()
|
||||
|
||||
video: Video = test_session.query(Video).one()
|
||||
assert video.video_file == video_file
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_video_factory(test_session, video_factory, channel_factory):
|
||||
"""
|
||||
A poster should not be generated if the Video's Channel forbids it.
|
||||
The `video_factory` pytest fixture is used in many video tests. Test all it's functionality.
|
||||
"""
|
||||
channel1 = channel_factory() # posters should not be generated by default.
|
||||
channel = channel_factory()
|
||||
video = video_factory(
|
||||
channel_id=channel.id,
|
||||
with_video_file=True,
|
||||
with_info_json={'description': 'hello'},
|
||||
with_caption_file=True,
|
||||
with_poster_ext='png',
|
||||
source_id='some id',
|
||||
)
|
||||
test_session.commit()
|
||||
|
||||
assert video.video_path and video.video_file and video.video_path.is_file()
|
||||
assert video.caption_path and video.caption_file and video.caption_path.is_file()
|
||||
assert video.poster_path and video.poster_file and video.poster_path.is_file()
|
||||
assert video.info_json_path and video.info_json_file and video.info_json_path.is_file()
|
||||
|
||||
assert video.video_file.a_text # title
|
||||
assert video.video_file.b_text is None
|
||||
assert video.video_file.c_text and 'hello' in video.video_file.c_text # description
|
||||
assert video.video_file.d_text # captions
|
||||
assert video.source_id == 'some id'
|
||||
|
||||
|
||||
def test_validate_video(test_session, test_directory, video_factory):
|
||||
"""A video poster will be generated only if the channel permits."""
|
||||
vid1 = video_factory(with_video_file=True)
|
||||
assert not vid1.poster_path
|
||||
|
||||
validate_video(vid1, True, test_session)
|
||||
assert vid1.poster_path, 'Poster was not created'
|
||||
assert vid1.poster_path.is_file(), 'Poster path does not exist'
|
||||
assert vid1.video_path.stem == vid1.poster_path.stem
|
||||
assert vid1.poster_path.suffix == '.jpg'
|
||||
|
||||
# A PNG is replaced.
|
||||
vid2 = video_factory(with_video_file=True, with_poster_ext='.png')
|
||||
vid2.poster_path.with_suffix('.jpg').touch() # New poster exists, replace it.
|
||||
assert vid2.poster_path and vid2.poster_path.is_file() and vid2.poster_path.suffix == '.png', \
|
||||
'Poster was not initialized'
|
||||
validate_video(vid2, True, test_session)
|
||||
assert vid2.poster_path.is_file(), 'New poster was not generated'
|
||||
assert vid2.poster_path.suffix == '.jpg'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_statistics(test_session, video_factory, channel_factory):
|
||||
# Can get statistics in empty DB.
|
||||
await get_statistics()
|
||||
|
||||
channel1 = channel_factory()
|
||||
channel2 = channel_factory()
|
||||
channel2.generate_posters = True
|
||||
vid1 = video_factory(channel1.id, with_video_file=True)
|
||||
vid2 = video_factory(channel2.id, with_video_file=True)
|
||||
# webp will not be converted unless generate_posters is enabled.
|
||||
vid3 = video_factory(channel1.id, with_video_file=True, with_poster_ext='webp')
|
||||
test_session.commit()
|
||||
assert not vid1.poster_path
|
||||
assert not vid2.poster_path
|
||||
assert str(vid3.poster_path).endswith('webp')
|
||||
video_factory(channel_id=channel1.id)
|
||||
video_factory(channel_id=channel1.id)
|
||||
video_factory(channel_id=channel2.id)
|
||||
video_factory()
|
||||
|
||||
validate_videos()
|
||||
|
||||
assert not vid1.poster_path
|
||||
assert vid2.poster_path
|
||||
assert str(vid3.poster_path).endswith('webp')
|
||||
|
||||
|
||||
def test_video_modification_datetime(test_session, simple_channel, video_file):
|
||||
"""
|
||||
A Video has a modification_datetime which represents the video file's mtime.
|
||||
"""
|
||||
expected = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
|
||||
with mock.patch('modules.videos.lib.from_timestamp') as mock_from_timestamp:
|
||||
mock_from_timestamp.return_value = expected
|
||||
video = upsert_video(test_session, video_file, simple_channel)
|
||||
test_session.commit()
|
||||
assert video.modification_datetime == expected
|
||||
|
||||
|
||||
def test_refresh_channel_videos(test_session, channel_factory, video_factory, video_file, assert_video_ids):
|
||||
assert_video_ids([])
|
||||
|
||||
# Simulate a channel's catalog being updated.
|
||||
channel1 = channel_factory(name='channel1')
|
||||
test_session.add(Video(source_id='source1', channel_id=channel1.id))
|
||||
test_session.add(Video(source_id='source2', channel_id=channel1.id))
|
||||
|
||||
# Add some real files to be found.
|
||||
destination = channel1.directory / f'channel1_20220510_source1_the title.mp4'
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
|
||||
destination = channel1.directory / f'channel1_20220511_source2_the title.mp4'
|
||||
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
|
||||
|
||||
lib.refresh_channel_videos(channel1)
|
||||
assert_video_ids([1, 2])
|
||||
|
||||
# These video files were deleted.
|
||||
test_session.add(Video(source_id='this should be removed', channel_id=channel1.id, video_path='foo'))
|
||||
test_session.add(Video(source_id='this should also be removed', video_path='bar'))
|
||||
test_session.commit()
|
||||
|
||||
assert_video_ids([1, 2, 3, 4])
|
||||
|
||||
lib.refresh_channel_videos(channel1)
|
||||
assert_video_ids([1, 2, 4])
|
||||
|
||||
lib.refresh_videos()
|
||||
assert_video_ids([1, 2])
|
||||
result = await get_statistics()
|
||||
assert 'statistics' in result
|
||||
assert 'videos' in result['statistics']
|
||||
assert 'channels' in result['statistics']
|
||||
assert 'historical' in result['statistics']
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ logger = logger.getChild(__name__)
|
|||
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
|
||||
def video_get(_: Request, video_id: int):
|
||||
video, previous_video, next_video = lib.get_video_for_app(video_id)
|
||||
return json_response({'video': video, 'prev': previous_video, 'next': next_video})
|
||||
return json_response({'file': video, 'prev': previous_video, 'next': next_video})
|
||||
|
||||
|
||||
@video_bp.post('/search')
|
||||
|
|
@ -40,7 +40,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
|
|||
if body.order_by not in lib.VIDEO_ORDERS:
|
||||
raise InvalidOrderBy('Invalid order by')
|
||||
|
||||
videos, videos_total = lib.video_search(
|
||||
files, videos_total = lib.video_search(
|
||||
body.search_str,
|
||||
body.offset,
|
||||
body.limit,
|
||||
|
|
@ -49,7 +49,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
|
|||
body.filters,
|
||||
)
|
||||
|
||||
ret = {'videos': list(videos), 'totals': {'videos': videos_total}}
|
||||
ret = {'files': list(files), 'totals': {'files': videos_total}}
|
||||
return json_response(ret)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import pathlib
|
||||
from datetime import datetime
|
||||
from typing import Tuple, Optional, List
|
||||
|
||||
|
|
@ -8,6 +9,9 @@ from sqlalchemy.orm.exc import NoResultFound
|
|||
from wrolpi.common import run_after, logger
|
||||
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
|
||||
from wrolpi.errors import UnknownVideo
|
||||
from wrolpi.files.lib import assign_file_prefetched_models, handle_search_results
|
||||
from wrolpi.files.models import File
|
||||
from wrolpi.vars import PYTEST
|
||||
from ..lib import save_channels_config
|
||||
from ..models import Video
|
||||
|
||||
|
|
@ -31,49 +35,49 @@ def get_video_for_app(video_id: int) -> Tuple[dict, Optional[dict], Optional[dic
|
|||
video.set_viewed()
|
||||
previous_video, next_video = video.get_surrounding_videos()
|
||||
|
||||
caption = video.caption
|
||||
video = video.__json__()
|
||||
video['caption'] = caption
|
||||
previous_video = previous_video.__json__() if previous_video else None
|
||||
next_video = next_video.__json__() if next_video else None
|
||||
caption = video.video_file.d_text
|
||||
video = video.video_file.__json__()
|
||||
video['video']['caption'] = caption
|
||||
previous_video = previous_video.video_file.__json__() if previous_video else None
|
||||
next_video = next_video.video_file.__json__() if next_video else None
|
||||
|
||||
return video, previous_video, next_video
|
||||
|
||||
|
||||
VIDEO_ORDERS = {
|
||||
'upload_date': 'upload_date ASC, LOWER(video_path) ASC',
|
||||
'-upload_date': 'upload_date DESC NULLS LAST, LOWER(video_path) DESC',
|
||||
'rank': '2 DESC, LOWER(video_path) DESC',
|
||||
'-rank': '2 ASC, LOWER(video_path) ASC',
|
||||
'id': 'id ASC',
|
||||
'-id': 'id DESC',
|
||||
'size': 'size ASC, LOWER(video_path) ASC',
|
||||
'-size': 'size DESC, LOWER(video_path) DESC',
|
||||
'duration': 'duration ASC, LOWER(video_path) ASC',
|
||||
'-duration': 'duration DESC, LOWER(video_path) DESC',
|
||||
'favorite': 'favorite ASC, LOWER(video_path) ASC',
|
||||
'-favorite': 'favorite DESC, LOWER(video_path) DESC',
|
||||
'viewed': 'viewed ASC',
|
||||
'-viewed': 'viewed DESC',
|
||||
'view_count': 'view_count ASC',
|
||||
'-view_count': 'view_count DESC',
|
||||
'modification_datetime': 'modification_datetime ASC',
|
||||
'-modification_datetime': 'modification_datetime DESC',
|
||||
'upload_date': 'v.upload_date ASC, LOWER(v.video_path) ASC',
|
||||
'-upload_date': 'v.upload_date DESC NULLS LAST, LOWER(v.video_path) DESC',
|
||||
'rank': '2 DESC, LOWER(v.video_path) DESC',
|
||||
'-rank': '2 ASC, LOWER(v.video_path) ASC',
|
||||
'id': 'v.id ASC',
|
||||
'-id': 'v.id DESC',
|
||||
'size': 'f.size ASC, LOWER(v.video_path) ASC',
|
||||
'-size': 'f.size DESC, LOWER(v.video_path) DESC',
|
||||
'duration': 'duration ASC, LOWER(v.video_path) ASC',
|
||||
'-duration': 'duration DESC, LOWER(v.video_path) DESC',
|
||||
'favorite': 'favorite ASC, LOWER(v.video_path) ASC',
|
||||
'-favorite': 'favorite DESC, LOWER(v.video_path) DESC',
|
||||
'viewed': 'v.viewed ASC',
|
||||
'-viewed': 'v.viewed DESC',
|
||||
'view_count': 'v.view_count ASC',
|
||||
'-view_count': 'v.view_count DESC',
|
||||
'modification_datetime': 'f.modification_datetime ASC',
|
||||
'-modification_datetime': 'f.modification_datetime DESC',
|
||||
}
|
||||
NO_NULL_ORDERS = {
|
||||
'viewed': '\nAND viewed IS NOT NULL',
|
||||
'-viewed': '\nAND viewed IS NOT NULL',
|
||||
'duration': '\nAND duration IS NOT NULL',
|
||||
'-duration': '\nAND duration IS NOT NULL',
|
||||
'size': '\nAND size IS NOT NULL',
|
||||
'-size': '\nAND size IS NOT NULL',
|
||||
'view_count': '\nAND view_count IS NOT NULL',
|
||||
'-view_count': '\nAND view_count IS NOT NULL',
|
||||
'modification_datetime': '\nAND modification_datetime IS NOT NULL',
|
||||
'-modification_datetime': '\nAND modification_datetime IS NOT NULL',
|
||||
'viewed': 'v.viewed IS NOT NULL',
|
||||
'-viewed': 'v.viewed IS NOT NULL',
|
||||
'duration': 'v.duration IS NOT NULL',
|
||||
'-duration': 'v.duration IS NOT NULL',
|
||||
'size': 'f.size IS NOT NULL',
|
||||
'-size': 'f.size IS NOT NULL',
|
||||
'view_count': 'v.view_count IS NOT NULL',
|
||||
'-view_count': 'v.view_count IS NOT NULL',
|
||||
'modification_datetime': 'f.modification_datetime IS NOT NULL',
|
||||
'-modification_datetime': 'f.modification_datetime IS NOT NULL',
|
||||
}
|
||||
DEFAULT_VIDEO_ORDER = 'rank'
|
||||
VIDEO_QUERY_LIMIT = 20
|
||||
VIDEO_QUERY_LIMIT = 24
|
||||
|
||||
|
||||
def video_search(
|
||||
|
|
@ -84,71 +88,61 @@ def video_search(
|
|||
order_by: str = None,
|
||||
filters: List[str] = None,
|
||||
) -> Tuple[List[dict], int]:
|
||||
with get_db_curs() as curs:
|
||||
params = dict(search_str=search_str, offset=offset)
|
||||
channel_where = ''
|
||||
if channel_id:
|
||||
channel_where = 'AND channel_id = %(channel_id)s'
|
||||
params['channel_id'] = channel_id
|
||||
wheres = ['v.video_path IS NOT NULL']
|
||||
|
||||
# Filter for/against favorites, if it was provided
|
||||
favorites_where = ''
|
||||
if isinstance(filters, list) and 'favorite' in filters:
|
||||
favorites_where = 'AND favorite IS NOT NULL'
|
||||
params = dict(search_str=search_str, offset=offset)
|
||||
if channel_id:
|
||||
wheres.append('v.channel_id = %(channel_id)s')
|
||||
params['channel_id'] = channel_id
|
||||
|
||||
censored_where = ''
|
||||
if isinstance(filters, list) and 'censored' in filters:
|
||||
censored_where = 'AND censored = true'
|
||||
# Filter for/against favorites, if it was provided
|
||||
if isinstance(filters, list) and 'favorite' in filters:
|
||||
wheres.append('v.favorite IS NOT NULL')
|
||||
|
||||
where = ''
|
||||
if search_str:
|
||||
# A search_str was provided by the user, modify the query to filter by it.
|
||||
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
|
||||
where = 'AND textsearch @@ websearch_to_tsquery(%(search_str)s)'
|
||||
params['search_str'] = search_str
|
||||
else:
|
||||
# No search_str provided. Get id and total only.
|
||||
columns = 'id, COUNT(*) OVER() AS total'
|
||||
if isinstance(filters, list) and 'censored' in filters:
|
||||
wheres.append('v.censored = true')
|
||||
|
||||
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
|
||||
# whitelist.
|
||||
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
|
||||
if order_by:
|
||||
try:
|
||||
order = VIDEO_ORDERS[order_by]
|
||||
except KeyError:
|
||||
raise
|
||||
if order_by in NO_NULL_ORDERS:
|
||||
where += NO_NULL_ORDERS[order_by]
|
||||
if search_str:
|
||||
# A search_str was provided by the user, modify the query to filter by it.
|
||||
select_columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), ' \
|
||||
'COUNT(*) OVER() AS total'
|
||||
wheres.append('f.textsearch @@ websearch_to_tsquery(%(search_str)s)')
|
||||
params['search_str'] = search_str
|
||||
join = 'LEFT JOIN file f on f.path = v.video_path'
|
||||
else:
|
||||
# No search_str provided. Get path and total only.
|
||||
select_columns = 'v.video_path AS path, COUNT(*) OVER() AS total'
|
||||
join = ''
|
||||
|
||||
query = f'''
|
||||
SELECT
|
||||
{columns}
|
||||
FROM video
|
||||
WHERE
|
||||
video_path IS NOT NULL
|
||||
{where}
|
||||
{channel_where}
|
||||
{favorites_where}
|
||||
{censored_where}
|
||||
ORDER BY {order}
|
||||
OFFSET %(offset)s LIMIT {int(limit)}
|
||||
'''.strip()
|
||||
logger.debug(query)
|
||||
if order_by in {'size', '-size', 'modification_datetime', '-modification_datetime'}:
|
||||
# Size and modification_datetime are from the video file.
|
||||
join = 'LEFT JOIN file f on f.path = v.video_path'
|
||||
|
||||
curs.execute(query, params)
|
||||
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
|
||||
# whitelist.
|
||||
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
|
||||
if order_by:
|
||||
try:
|
||||
results = [dict(i) for i in curs.fetchall()]
|
||||
except psycopg2.ProgrammingError:
|
||||
# No videos
|
||||
return [], 0
|
||||
total = results[0]['total'] if results else 0
|
||||
ranked_ids = [i['id'] for i in results]
|
||||
order = VIDEO_ORDERS[order_by]
|
||||
except KeyError:
|
||||
raise
|
||||
if order_by in NO_NULL_ORDERS:
|
||||
wheres.append(NO_NULL_ORDERS[order_by])
|
||||
|
||||
with get_db_session() as session:
|
||||
results = get_ranked_models(ranked_ids, Video, session=session)
|
||||
results = [i.__json__() for i in results]
|
||||
wheres = '\n AND '.join(wheres)
|
||||
where = f'WHERE\n{wheres}' if wheres else ''
|
||||
stmt = f'''
|
||||
SELECT
|
||||
{select_columns}
|
||||
FROM video v
|
||||
{join}
|
||||
{where}
|
||||
ORDER BY {order}
|
||||
OFFSET %(offset)s LIMIT {int(limit)}
|
||||
'''.strip()
|
||||
logger.debug(stmt, params)
|
||||
|
||||
results, total = handle_search_results(stmt, params)
|
||||
return results, total
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
from datetime import timedelta
|
||||
from http import HTTPStatus
|
||||
from json import dumps
|
||||
from unittest import mock
|
||||
|
||||
from modules.videos.models import Video
|
||||
from modules.videos.video.lib import get_video_for_app
|
||||
|
|
@ -71,7 +70,7 @@ def test_get_video_prev_next(test_session, channel_factory, video_factory):
|
|||
|
||||
def test_get_video_for_app(test_session, simple_channel, simple_video):
|
||||
vid, prev, next_ = get_video_for_app(simple_video.id)
|
||||
assert vid['id'] == simple_video.id
|
||||
assert vid['video']['id'] == simple_video.id
|
||||
|
||||
|
||||
def test_video_delete(test_session, test_directory, channel_factory, video_factory):
|
||||
|
|
@ -81,8 +80,8 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
|
|||
vid2 = video_factory(channel_id=channel2.id, with_video_file=True, with_info_json=True)
|
||||
test_session.commit()
|
||||
|
||||
vid1_video_path, vid1_caption_path = vid1.video_path.path, vid1.caption_path.path
|
||||
vid2_video_path, vid2_info_json_path = vid2.video_path.path, vid2.info_json_path.path
|
||||
vid1_video_path, vid1_caption_path = vid1.video_path, vid1.caption_path
|
||||
vid2_video_path, vid2_info_json_path = vid2.video_path, vid2.info_json_path
|
||||
|
||||
# No videos have been deleted yet.
|
||||
assert vid1_video_path.is_file() and vid1_caption_path.is_file()
|
||||
|
|
@ -106,36 +105,42 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
|
|||
assert vid2_video_path.is_file() is False and vid2_info_json_path.is_file() is False
|
||||
|
||||
|
||||
def test_wrol_mode(test_directory, simple_channel, simple_video):
|
||||
def test_wrol_mode(test_directory, simple_channel, simple_video, wrol_mode_fixture, test_download_manager):
|
||||
"""Many methods are blocked when WROL Mode is enabled."""
|
||||
channel = dumps(dict(name=simple_channel.name, directory=str(simple_channel.directory)))
|
||||
favorite = dumps(dict(video_id=simple_video.id, favorite=True))
|
||||
|
||||
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
|
||||
# Can't create, update, or delete a channel.
|
||||
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
wrol_mode_fixture(True)
|
||||
|
||||
# Can't delete a video
|
||||
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
# Can't create, update, or delete a channel.
|
||||
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
|
||||
# Can't refresh or download
|
||||
_, resp = api_app.test_client.post('/api/videos/refresh')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
# Can't delete a video
|
||||
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
|
||||
# THE REST OF THESE METHODS ARE ALLOWED
|
||||
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
# Can't refresh or download
|
||||
_, resp = api_app.test_client.post('/api/files/refresh')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
|
||||
assert resp.status_code == HTTPStatus.FORBIDDEN
|
||||
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
|
||||
|
||||
# THE REST OF THESE METHODS ARE ALLOWED
|
||||
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
|
||||
assert resp.status_code == HTTPStatus.OK
|
||||
|
||||
assert test_download_manager.stopped.is_set()
|
||||
|
||||
wrol_mode_fixture(False)
|
||||
assert not test_download_manager.stopped.is_set()
|
||||
|
|
|
|||
|
|
@ -1,3 +1,9 @@
|
|||
import re
|
||||
|
||||
from wrolpi.common import logger
|
||||
|
||||
logger = logger.getChild(__name__)
|
||||
|
||||
formatters = {
|
||||
'youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
|
||||
'www.youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
|
||||
|
|
@ -22,3 +28,16 @@ def video_url_resolver(domain: str, entry: dict) -> str:
|
|||
except KeyError:
|
||||
# Unable to resolve, try the URL as is.
|
||||
return entry['url']
|
||||
|
||||
|
||||
YOUTUBE_SHORT_REGEX = re.compile(r'https?:\/\/w{3}?\.?youtube.com\/shorts\/(.{5,15})')
|
||||
|
||||
|
||||
def normalize_youtube_shorts_url(url: str) -> str:
|
||||
"""Convert a YouTube "shorts" URL to a regular video URL.
|
||||
|
||||
This is to avoid downloading shorts twice."""
|
||||
if match := YOUTUBE_SHORT_REGEX.match(url.strip()):
|
||||
source_id = match.group(1)
|
||||
return f'https://www.youtube.com/watch?v={source_id}'
|
||||
return url
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue