Fixing JS warnings.

Handling all combinations of toggle in light and dark.

Changing CPUUsageProgress colors to be more readable in dark mode.

Moving theme handling into Theme.js.  Importing element wrappers from Theme.js to handle element theming.

Changing video menu to dark mode.

Creating ThemeContext to share Dark Mode setting.

Removing duplicate code for search functions.

Displaying X when downloads are disabled.

Using react context to share settings across components.

Refusing to start workers when WROL Mode is enabled.

Changing warning to debug.

Queuing recurring downloads first.

Adding Video.url index.

Adding special handling for youtube shorts to videos.

Reworking Downloads to have one worker per process.

Testing creating downloads and getting downloaders.

Testing video validation and poster generation.  Testing video statistics.

Increasing inventory test coverage.

Adding pytest.ini to automatically detect async test.

Adding more archive test coverage.

Testing Archive CRUD.

Catching and logging when encountering error during indexing.

Using FileIcon when Archive has no screenshot.

A Video will be assigned to a Channel by its info json first, then to its directory.

Including suffix in File json.

Adding index for File.suffix.

Gathering suffix during indexing.

Passing entire file down to FileIcon.

Moving logging up.

Returning File.associated in file json.

Associated files are last in file search.

Removing duplicate refreshing code by creating new _refresh_directory function.

Changing File.idempotency to a datetime to avoid having to clear idempotency.

Truncating indexes to avoid error with large files.

Don't overwrite video data when info json is empty.

Using already_downloaded for channel videos.

Awaiting refresh in Channel.refresh.

Testing that videos already downloaded aren't downloaded again.

Adding indexes for video/archive referenced paths.

Changing grid width for mobile or computer.

Adding button to refresh specific directories.

Restoring Channel edit link.

Fixing caption path in video_modeler.

Adding ZIP files to mimetype colors.

Using model instead of first file's model.

Fetching File models in batches to avoid issuing a SELECT for every File with a model.

Flushing generated poster file.

Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.

Attaching Video to it's Channel during download.

Using ExternalCardLink for ArchivePage singlefile link.

Validating and indexing videos after download.

Handling glob when file name contains matching characters.

Reworking and testing split_path_stem_and_suffix and glob_shared_stem.

Adding associated video files to DB during upsert.

Adding pending downloads to dashboard.

Hiding status when download is open.

Handling when load is empty.

Adding load statistics to Dashboard.

Limiting concurrent after_refresh functions.

Handling when no onSelect is declared.

Adding ImageRow for FileTable.

Moving image date into Card.Meta.

Using ExternalLink for FileCard.

Using ExternalLink for ImageCard.

Handling when error occurs during refresh of a directory.  Committing during refresh to avoid losing all files if an error occurs.

Removing api health check to avoid spamming logs.

Creating ImageCard to display image posters when they aren't too large.

Removing archive.domain_id not null.

Reworking file refresh to process all files in a single directory.

Bumping version for major file rework.

No longer checking for video corruption.

Adding duration by default to speed up tests.

Using register_indexer for all indexers.

Adding CPU usage progress bar in Welcome.

Modifying TextIndexer to ignore large text files.

Counting all files during refresh.

Adding refresh_logger to clarify logging.  Checking for wrol_mode before refresh functions.

Removing unused idempotency from video/channel tables.

Using magic module to speed up mimetype identification.

All tests pass.

Archive tests pass.

Changing video get endpoint to return 'file' to avoid ambiguity.

Video tests pass.

Simplifying regex filtering.

Using regex to ignore case when filtering.

Converting all channel classes to functions.

Restoring channel refresh and download buttons.

Matching format of Archive and Video pages.

Adding sorting to Archives search.

Reworking video orders for file search.  Using SearchInput for videos search.

Archives can be searched on the Archives page.  Domains can be filtered.

Reworking Archive search to use Files.  Videos, Archives and Global search use the FilesView.

Videos are claimed by their Channel based on the Channel's directory.  Prev and Next video cards are displayed on the VideoPage.

Creating useVideo for reworked VideoWrapper and VideoPage.

Adding Paginator to FilesView.

Moving file refresh button out of manage tab.  Removing Files page tabs.  Limiting refresh to one at a time.

Adding class for card icon to match poster height.

Video and Archives always re-claim their files on refresh.

Archive files are indexed and claimed.

FileView table is compact and striped.

Videos page uses FileView.

Updated navlinks to router v6.

Removing ! character from file names.

Disabling select button when cards are visible.

Adding show/hide button for FileTable selection.

The Archive Page is displayed.

Adding mimetype dropdown.

Upgrading to React Router v6 for useSearchParams.

Validating video during modeling.  Removing screenshot during corruption check because it was very slow.  Updating app to use video files as search results.

Updating video search to return Files.

Video and WROLPi tests pass.

Creating function to refresh files within a channel's directory.

Removing troubleshooting print.  Updating result order in test.

Reworking video_factory to handle a specific video file location.

Returning video model on file search.

Getting session using video_file.

Screenshotting when checking for video corruption.

Reducing ffmpeg calls by gathering duration during poster generation.

Moving video file matching into model.

VideoIndexer is moved into video module.  Video files are validated.

Moving files module into wrolpi in preparation to use it in modules.

Title and contents are indexed for text file.

Files are indexed on refresh by using their indexer.

Zip files are indexed using their contents.

Testing multiple zip files.

Adding unique to file.path.  Checking that the correct indexer is used.

Adding index to file.modification_datetime and file.size.

Handling file info gathering in File.do_index.  Only reading contents of file if file.indexed is False.

Video model is found during file refresh.  A Video record is created that has the associated files.

Videos without associated files are found.

Videos are found using registered video_modeler() rather than in refresh code.

Removing old video columns.

Handling video info json files.

Trying to import each module individually.  Will continue to import modules even if one fails.

Reworking refresh so it performs less DB calls.

Converting file.path to primary key.

file.path is always a pathlib.Path.

Updating video_factory to use new file system, testing it.

Bump sanic from 21.9.1 to 21.12.2 in /docker/archive

Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
This commit is contained in:
dependabot[bot] 2022-08-06 05:22:07 +00:00 • committed by lrnselfreliance
parent 1ede5f3f55
commit d4ed5fa5ce
98 changed files with 7441 additions and 5257 deletions

View file

@ -1,20 +1,28 @@
import json
import pathlib
from abc import ABC
from typing import Tuple, Optional
from typing import Tuple, Dict, List
from sqlalchemy.orm import Session
from wrolpi.common import logger
from wrolpi.db import optional_session
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
from wrolpi.db import optional_session, get_db_session
from wrolpi.downloader import Downloader, Download, DownloadResult
from wrolpi.errors import UnrecoverableDownloadError
from wrolpi.files.indexers import register_indexer, Indexer
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from . import lib
from .api import bp # noqa
from .models import Archive
from .lib import match_archive_files
from .models import Archive, Domain
PRETTY_NAME = 'Archive'
logger = logger.getChild(__name__)
__all__ = ['ArchiveDownloader', 'archive_downloader']
class ArchiveDownloader(Downloader, ABC):
name = 'archive'
@ -38,9 +46,134 @@ class ArchiveDownloader(Downloader, ABC):
return DownloadResult(success=True, location=f'/archive/{archive.id}')
@optional_session
def already_downloaded(self, url: str, session: Session = None) -> bool:
return bool(session.query(Archive).filter_by(url=url).count())
def already_downloaded(self, urls: List[str], session: Session = None) -> List:
archives = list(session.query(Archive).filter(Archive.url.in_(urls)))
return archives
# Archive downloader is the last downloader which should be used.
archive_downloader = ArchiveDownloader(priority=100)
@register_modeler
def archive_modeler(groups: Dict[str, List[File]], session: Session):
archive_groups = {}
for stem, group in groups.items():
if any(lib.is_singlefile_file(i.path) for i in group):
archive_groups[stem] = group
count = 0
for stem, group in archive_groups.items():
readability_file, singlefile_file, readability_json_file, readability_txt_file, screenshot_file = \
match_archive_files(group)
archive: Archive = Archive.find_by_path(singlefile_file.path, session)
if not archive:
archive = Archive(singlefile_file=singlefile_file)
session.add(archive)
archive.readability_file = readability_file
archive.readability_json_file = readability_json_file
archive.readability_txt_file = readability_txt_file
archive.screenshot_file = screenshot_file
if archive.readability_file:
archive.readability_file.associated = True
archive.readability_file.do_index()
if archive.readability_json_file:
archive.readability_json_file.associated = True
archive.readability_json_file.do_index()
if archive.readability_txt_file:
archive.readability_txt_file.associated = True
archive.readability_txt_file.do_index()
if archive.screenshot_file:
archive.screenshot_file.associated = True
archive.screenshot_file.do_index()
archive.singlefile_file.model = Archive.__tablename__
archive.singlefile_file.do_index()
archive.validate()
# Remove this group, it will not be processed by other modelers.
del groups[stem]
count += 1
if not count % 100:
# Commit occasionally.
session.commit()
@register_after_refresh
@limit_concurrent(1)
def archive_cleanup():
# Process all Archives that have not been validated. Delete any that no longer exist, or are not real Archives.
# Validate all other Archives.
logger.info('Searching for invalid Archives')
offset = 0
limit = 100
while True:
with get_db_session(commit=True) as session:
archives: List[Archive] = session.query(Archive).order_by(Archive.id).limit(limit).offset(offset)
processed = 0
deleted = 0
for archive in archives:
processed += 1
if not lib.is_singlefile_file(archive.singlefile_path):
# Archive is not a real archive.
archive.singlefile_file.model = None
session.delete(archive)
deleted += 1
if processed < limit:
# Validated all archives.
break
offset += (limit - deleted)
with get_db_session(commit=True) as session:
# Remove any Domains without any Archives.
domain_ids = [i[0] for i in session.execute('SELECT DISTINCT domain_id FROM archive') if i[0]]
deleted = 0
for domain in session.query(Domain):
if domain.id not in domain_ids:
deleted += 1
session.delete(domain)
@register_indexer('text/html')
class ArchiveIndexer(Indexer, ABC):
@staticmethod
def get_title(file: File):
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
readability_json_path = singlefile_path.with_suffix('.readability.json')
if readability_json_path.is_file():
with readability_json_path.open('rt') as fh:
try:
return json.load(fh)['title']
except Exception:
if not PYTEST:
logger.warning(f'Archive readability json file exists, but cannot get title. {file}')
return None
@staticmethod
def get_article(file: File):
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
readability_txt_path = singlefile_path.with_suffix('.readability.txt')
if readability_txt_path.is_file():
return readability_txt_path.read_text()
@classmethod
def create_index(cls, file: File) -> Tuple:
"""
Index an Archive file and it's associated files. This mimetype matches non-archive (non-singlefile) html files,
so this index will return empty if a readability file is found.
a = title
b = <empty>
c = <empty>
d = readability article
"""
if file.path.name.endswith('.readability.html'):
# Readability files are not the primary Archive file.
return None, None, None, None
a = cls.get_title(file)
d = cls.get_article(file)
return a, None, None, d

View file

@ -1,4 +1,3 @@
import asyncio
from http import HTTPStatus
from sanic import response, Request
@ -22,8 +21,9 @@ logger = logger.getChild(__name__)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
async def get_archive(_: Request, archive_id: int):
archive = lib.get_archive(archive_id=archive_id)
alternatives = archive.alternatives
return json_response({'archive': archive, 'alternatives': alternatives})
archive_file = archive.singlefile_file.__json__()
alternatives = [i.singlefile_file.__json__() for i in archive.alternatives]
return json_response({'file': archive_file, 'alternatives': alternatives})
@bp.delete('/<archive_id:int>')
@ -35,13 +35,6 @@ async def delete_archive(_: Request, archive_id: int):
return response.empty()
@bp.post('/refresh')
@wrol_mode_check
async def refresh_archives(_: Request):
asyncio.ensure_future(lib.refresh_archives())
return response.empty()
@bp.get('/domains')
async def fetch_domains(_: Request):
domains = lib.get_domains()
@ -65,6 +58,6 @@ async def search_archives(_: Request, body: schema.ArchiveSearchRequest):
limit = archive_limit_limiter(body.limit)
offset = body.offset or 0
archives, total = lib.search(search_str, domain, limit, offset)
archives, total = lib.archive_search(search_str, domain, limit, offset, body.order_by)
ret = dict(archives=archives, totals=dict(archives=total))
return json_response(ret)

View file

@ -8,6 +8,7 @@ import pytest
from modules.archive.lib import archive_strftime
from modules.archive.models import Archive, Domain
from wrolpi.dates import local_timezone
from wrolpi.files.models import File
@pytest.fixture
@ -18,7 +19,7 @@ def archive_directory(test_directory) -> pathlib.Path:
@pytest.fixture
def archive_factory(test_session, archive_directory):
def archive_factory(test_session, archive_directory, make_files_structure):
def time_generator():
timestamp = datetime.datetime(2000, 1, 1, 0, 0, 0)
while True:
@ -27,12 +28,12 @@ def archive_factory(test_session, archive_directory):
now = time_generator()
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None) -> Archive:
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None,
singlefile_contents: str = None) -> Archive:
if domain:
assert '/' not in domain
title = title or str(uuid4())
url = url or f'https://example.com/{title}'
domain_dir = domain or title
domain_dir = archive_directory / domain_dir
domain_dir.mkdir(exist_ok=True)
@ -40,19 +41,25 @@ def archive_factory(test_session, archive_directory):
archive_datetime = local_timezone(next(now))
timestamp = archive_strftime(archive_datetime)
singlefile_path = domain_dir / f'{timestamp}_{title}.html'
screenshot_path = domain_dir / f'{timestamp}_{title}.png'
readability_path = domain_dir / f'{timestamp}_{title}.readability.html'
readability_txt_path = domain_dir / f'{timestamp}_{title}.readability.txt'
readability_json_path = domain_dir / f'{timestamp}_{title}.readability.json'
with readability_json_path.open('wt') as fh:
json.dump({'url': url}, fh)
json_contents = {}
if title != 'NA':
json_contents['title'] = title
if url:
json_contents['url'] = url
json_contents = json.dumps(json_contents)
singlefile_path, screenshot_path, readability_path, readability_txt_path, readability_json_path = \
make_files_structure({
str(domain_dir / f'{timestamp}_{title}.html'): singlefile_contents or '<html></html>',
str(domain_dir / f'{timestamp}_{title}.png'): None,
str(domain_dir / f'{timestamp}_{title}.readability.html'): '<html></html>',
str(domain_dir / f'{timestamp}_{title}.readability.txt'): contents,
str(domain_dir / f'{timestamp}_{title}.readability.json'): json_contents,
})
if domain:
domain = test_session.query(Domain).filter_by(domain=domain).one_or_none()
if not domain:
if not domain and domain_dir.name != 'NA':
domain = Domain(
domain=domain_dir.name,
directory=domain_dir,
@ -61,23 +68,18 @@ def archive_factory(test_session, archive_directory):
test_session.flush([domain])
archive = Archive(
readability_json_path=readability_json_path,
readability_path=readability_path,
readability_txt_path=readability_txt_path,
screenshot_path=screenshot_path,
singlefile_path=singlefile_path,
readability_json_file=File(path=readability_json_path) if readability_json_path else None,
readability_file=File(path=readability_path) if readability_path else None,
readability_txt_file=File(path=readability_txt_path) if readability_txt_path else None,
screenshot_file=File(path=screenshot_path) if screenshot_path else None,
singlefile_file=File(path=singlefile_path, model='archive'),
title=title,
url=url,
domain_id=domain.id,
contents=contents,
domain_id=domain.id if domain else None,
archive_datetime=archive_datetime,
)
test_session.add(archive)
test_session.commit()
for path in archive.my_paths():
path.touch()
archive.singlefile_file.do_index()
return archive

View file

@ -7,21 +7,22 @@ import subprocess
import tempfile
from dataclasses import dataclass
from datetime import datetime
from functools import partial
from itertools import groupby
from typing import Iterator, Optional, Tuple, List
import aiohttp
from bs4 import BeautifulSoup
from selenium import webdriver
from sqlalchemy.orm import Session
from modules.archive.models import Domain, Archive
from wrolpi.cmd import which
from wrolpi.common import get_media_directory, logger, chunks, extract_domain, chdir, escape_file_name, walk, \
aiohttp_post
from wrolpi.common import get_media_directory, logger, extract_domain, chdir, escape_file_name, walk, \
aiohttp_post, match_paths_to_suffixes, iterify
from wrolpi.dates import now, Seconds, local_timezone
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models, optional_session
from wrolpi.errors import InvalidDomain, UnknownURL, InvalidArchive
from wrolpi.db import get_db_session, get_db_curs, optional_session
from wrolpi.errors import UnknownArchive, InvalidOrderBy
from wrolpi.files.lib import handle_search_results
from wrolpi.files.models import File
from wrolpi.vars import DOCKERIZED, PYTEST
logger = logger.getChild(__name__)
@ -284,12 +285,11 @@ async def do_archive(url: str) -> Archive:
archive = Archive(
title=title,
archive_datetime=now(),
singlefile_path=archive_files.singlefile,
readability_path=archive_files.readability,
readability_json_path=archive_files.readability_json,
readability_txt_path=archive_files.readability_txt,
screenshot_path=archive_files.screenshot,
contents=readability_txt,
singlefile_file=File(path=archive_files.singlefile, model='archive'),
readability_file=File(path=archive_files.readability) if archive_files.readability else None,
readability_json_file=File(path=archive_files.readability_json) if archive_files.readability_json else None,
readability_txt_file=File(path=archive_files.readability_txt) if archive_files.readability_txt else None,
screenshot_file=File(path=archive_files.screenshot) if archive_files.screenshot else None,
url=url,
domain_id=domain.id,
)
@ -323,19 +323,12 @@ def get_title_from_html(html: str, url: str = None) -> str:
logger.info(f'Unable to extract title {url}')
def get_domain(session, domain: str) -> Domain:
domain_ = session.query(Domain).filter_by(domain=domain).one_or_none()
if not domain_:
raise InvalidDomain(f'Invalid domain: {domain}')
return domain_
@optional_session
def get_archive(session, archive_id: int) -> Archive:
"""Get an Archive."""
archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
if not archive:
raise InvalidArchive(f'Invalid archive id: {archive_id}')
raise UnknownArchive(f'Could not find archive with id: {archive_id}')
return archive
@ -344,7 +337,7 @@ def delete_archive(archive_id: int):
with get_db_session(commit=True) as session:
archive: Archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
if not archive:
raise UnknownURL(f'Unknown Archive with id: {archive_id}')
raise UnknownArchive(f'Unknown Archive with id: {archive_id}')
# Delete any files associated with this URL.
archive.delete()
@ -476,95 +469,13 @@ def is_archive_file(path: pathlib.Path) -> bool:
return path.is_file() and path.suffix.lower() in ARCHIVE_SUFFIXES and bool(ARCHIVE_MATCHER.match(path.name))
def _refresh_archives():
def is_singlefile_file(path: pathlib.Path) -> bool:
"""
Search the Archives directory for archive files, update the database if new files are found. Remove any orphan
URLs or Domains.
Archive singlefile files are expected to start with the following: %Y-%m-%d-%H-%M-%S
they must end with .html, but never with .readability.html.
"""
archive_directory = get_archive_directory()
# TODO remove this later when everyone has migrated their files.
migrate_archive_files()
singlefile_paths = set()
for domain_directory in filter(lambda i: i.is_dir(), archive_directory.iterdir()):
logger.debug(f'Refreshing directory: {domain_directory}')
all_archives_files = filter(is_archive_file, walk(domain_directory))
archive_groups = group_archive_files(all_archives_files)
archive_groups = list(archive_groups)
archive_count = 0
for chunk in chunks(archive_groups, 20):
with get_db_session(commit=True) as session:
for dt, archive_files in chunk:
archive_count += 1
singlefile_paths.add(str(archive_files.singlefile))
upsert_archive(dt, archive_files, session)
if archive_count:
logger.info(f'Inserted/updated {archive_count} archives in {domain_directory}')
singlefile_paths = list(singlefile_paths)
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM archive WHERE singlefile_path != ALL(%s)', (singlefile_paths,))
with get_db_curs(commit=True) as curs:
stmt = '''
DELETE FROM domains WHERE
id NOT IN (select distinct domain_id from archive)
RETURNING domains.id
'''
curs.execute(stmt)
domains = list(map(dict, curs.fetchall()))
logger.info(f'Deleted {len(domains)} Domains')
with get_db_session(commit=True) as session:
# Get all archives that have no text contents, but have txt files.
archives = session.query(Archive).filter(
Archive.contents == None, # noqa
Archive.readability_txt_path != None,
).all()
for archive in archives:
with archive.readability_txt_path.path.open('rt') as fh:
archive.contents = fh.read()
async def refresh_archives():
_refresh_archives()
def upsert_archive(dt: str, archive_files: ArchiveFiles, session: Session):
"""Get or create an Archive, and it's URL/Domain. If it already exists, update it with these new files."""
# Extract the URL from the JSON. Fail if this is not possible.
try:
with archive_files.readability_json.open() as fh:
json_contents = json.load(fh)
url = json_contents['url']
title = json_contents.get('title')
except Exception as e:
raise InvalidArchive() from e
domain = get_or_create_domain(session, url)
# Get the existing Archive, or create a new one.
archive = session.query(Archive).filter_by(singlefile_path=archive_files.singlefile).one_or_none()
if not archive:
# No archive matches this singlefile_path, create a new one.
archive = Archive(url=url, domain_id=domain.id)
session.add(archive)
session.flush()
if not archive.title and title:
archive.title = title
if not archive.title and archive_files.singlefile:
# As a last resort, get the title from the HTML.
archive.title = get_title_from_html(archive_files.singlefile.read_text(), url)
# Update the archive with the files that we have.
archive.archive_datetime = dt
archive.singlefile_path = archive_files.singlefile
archive.readability_path = archive_files.readability
archive.readability_txt_path = archive_files.readability_txt
archive.readability_json_path = archive_files.readability_json
archive.screenshot_path = archive_files.screenshot
return path.is_file() and path.suffix.lower() == '.html' and not path.name.endswith('.readability.html') \
and bool(ARCHIVE_MATCHER.match(path.name))
def get_domains():
@ -581,46 +492,69 @@ def get_domains():
return domains
def search(search_str: str, domain: str, limit: int, offset: int) -> Tuple[List[Archive], int]:
with get_db_curs() as curs:
columns = 'id, COUNT(*) OVER() AS total'
params = dict(offset=offset, limit=limit)
wheres = ''
order_by = '1 DESC'
ARCHIVE_ORDERS = {
'date': 'a.archive_datetime ASC, LOWER(a.singlefile_path) ASC',
'-date': 'a.archive_datetime DESC NULLS LAST, LOWER(a.singlefile_path) DESC',
'rank': '2 DESC, a.archive_datetime DESC',
'-rank': '2 ASC, a.archive_datetime ASC',
}
if search_str:
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
params['search_str'] = search_str
wheres += '\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)'
# highest rank, then most recent
order_by = '2 DESC, archive_datetime DESC'
if domain:
curs.execute('SELECT id FROM domains WHERE domain=%s', (domain,))
try:
domain_id = curs.fetchone()[0]
except TypeError:
# No domains match the provided domain.
return [], 0
params['domain_id'] = domain_id
wheres += '\nAND domain_id = %(domain_id)s'
def archive_search(search_str: str, domain: str, limit: int, offset: int, order_by: str) -> Tuple[List[Archive], int]:
wheres = []
params = dict(offset=offset, limit=limit)
order = '1 DESC'
# TODO handle different Archive search orders.
stmt = f'''
SELECT {columns}
FROM archive
WHERE
singlefile_path IS NOT NULL AND singlefile_path != ''
{wheres}
ORDER BY {order_by}
OFFSET %(offset)s
LIMIT %(limit)s
'''
curs.execute(stmt, params)
results = [dict(i) for i in curs.fetchall()]
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
if search_str:
columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
params['search_str'] = search_str
wheres.append('\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)')
# highest rank, then most recent
if not order_by:
order = ARCHIVE_ORDERS['rank']
else:
# No search_str provided, get path and total only.
columns = 'f.path, COUNT(*) OVER() AS total'
results = get_ranked_models(ranked_ids, Archive)
if order_by:
try:
order = ARCHIVE_ORDERS[order_by]
except KeyError:
raise InvalidOrderBy(f'Invalid order by: {order_by}')
if domain:
params['domain'] = domain
wheres.append('AND domain_id = (select id from domains where domains.domain = %(domain)s)')
wheres = '\n'.join(wheres)
stmt = f'''
SELECT {columns}
FROM archive a
LEFT JOIN file f on f.path = a.singlefile_path
WHERE
a.singlefile_path != ''
{wheres}
ORDER BY {order}
OFFSET %(offset)s
LIMIT %(limit)s
'''
logger.warning(stmt, params)
results, total = handle_search_results(stmt, params)
return results, total
match_archive_paths = partial(match_paths_to_suffixes, suffix_groups=(
('.readability.html',),
('.html',),
('.readability.json',),
('.readability.txt',),
('.jpg', '.jpeg', '.webp', '.png'),
))
@iterify(tuple)
def match_archive_files(files: List[File]) -> Tuple[File, File, File, File, File]:
archive_paths = match_archive_paths([i.path for i in files])
for path in archive_paths:
yield next((i for i in files if i.path == path), None)

View file

@ -1,13 +1,22 @@
import datetime
import json
import pathlib
import re
from typing import Generator
from sqlalchemy import Column, Integer, String, ForeignKey, Computed
from sqlalchemy.orm import relationship, Session
import pytz
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean
from sqlalchemy.orm import relationship, Session, validates
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import ModelHelper, Base, tsvector
from wrolpi.common import ModelHelper, Base, logger
from wrolpi.dates import TZDateTime
from wrolpi.errors import InvalidArchive
from wrolpi.files.models import File
from wrolpi.media_path import MediaPathType
logger = logger.getChild(__name__)
class Archive(Base, ModelHelper):
__tablename__ = 'archive'
@ -16,34 +25,40 @@ class Archive(Base, ModelHelper):
domain_id = Column(Integer, ForeignKey('domains.id'))
domain = relationship('Domain', primaryjoin='Archive.domain_id==Domain.id')
singlefile_path = Column(MediaPathType)
readability_path = Column(MediaPathType)
readability_json_path = Column(MediaPathType)
readability_txt_path = Column(MediaPathType)
screenshot_path = Column(MediaPathType)
url = Column(String)
title = Column(String)
archive_datetime = Column(TZDateTime)
contents = Column(String)
validated = Column(Boolean)
textsearch = Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, contents), 'D')'''))
# Associated Files.
singlefile_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
singlefile_file: File = relationship('File', primaryjoin='Archive.singlefile_path==File.path')
readability_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_file: File = relationship('File', primaryjoin='Archive.readability_path==File.path')
readability_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_json_file: File = relationship('File', primaryjoin='Archive.readability_json_path==File.path')
readability_txt_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_txt_file: File = relationship('File', primaryjoin='Archive.readability_txt_path==File.path')
screenshot_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
screenshot_file: File = relationship('File', primaryjoin='Archive.screenshot_path==File.path')
def __repr__(self):
return f'<Archive id={self.id} url={self.url} singlefile={self.singlefile_path}>'
if self.domain:
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))} ' \
f'domain={self.domain.domain}>'
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))}>'
def my_paths(self) -> Generator:
if self.singlefile_path:
yield self.singlefile_path.path
yield self.singlefile_path
if self.readability_path:
yield self.readability_path.path
yield self.readability_path
if self.readability_json_path:
yield self.readability_json_path.path
yield self.readability_json_path
if self.readability_txt_path:
yield self.readability_txt_path.path
yield self.readability_txt_path
if self.screenshot_path:
yield self.screenshot_path.path
yield self.screenshot_path
def unlink(self):
"""
@ -73,6 +88,7 @@ class Archive(Base, ModelHelper):
readability_txt_path=self.readability_txt_path,
screenshot_path=self.screenshot_path,
singlefile_path=self.singlefile_path,
stem=self.singlefile_path.stem if self.singlefile_path else None,
title=self.title,
url=self.url,
)
@ -96,11 +112,103 @@ class Archive(Base, ModelHelper):
Get a list of Archives that share my URL.
"""
session = Session.object_session(self)
i = list(session.query(Archive).filter(
alternatives = list(session.query(Archive).filter(
Archive.id != self.id,
Archive.url == self.url,
).order_by(Archive.archive_datetime))
return i
return alternatives
def my_files(self) -> Generator[pathlib.Path, None, None]:
"""Return all paths of this Archive. Returns nothing if all paths are None.""" # noqa
if self.singlefile_path:
yield self.singlefile_path
if self.readability_path:
yield self.readability_path
if self.readability_txt_path:
yield self.readability_txt_path
if self.readability_json_path:
yield self.readability_json_path
if self.screenshot_path:
yield self.screenshot_path
@staticmethod
def find_by_path(path, session: Session) -> Base:
archive = session.query(Archive).filter_by(singlefile_path=path).one_or_none()
return archive
def read_singlefile_data(self):
"""
Read the start of the singlefile (if any) and decode the Archive information.
"""
match_url = re.compile(r'^\s+?url:\s+?(http.*)', re.MULTILINE)
match_date = re.compile(r'^\s+?saved date:\s+?(.*)', re.MULTILINE)
if self.singlefile_path:
with self.singlefile_path.open('rt') as fh:
head = fh.read(500)
if 'Page saved with SingleFile' not in head:
return
try:
if match := match_url.findall(head):
self.url = match[0].strip()
except Exception as e:
logger.error(f'Could not get URL from singlefile {self.singlefile_path}', exc_info=e)
try:
if match := match_date.findall(head):
dt = match[0].strip()
dt = ' '.join(dt.split(' ')[:5])
# SingleFile uses GMT.
dt = datetime.datetime.strptime(
dt,
'%a %b %d %Y %H:%M:%S' # Fri Jun 17 2022 19:24:52
).replace(tzinfo=pytz.timezone('GMT'))
self.archive_datetime = dt
except Exception as e:
logger.error(f'Could not get archive date from singlefile {self.singlefile_path}', exc_info=e)
def read_readability_data(self):
"""Read the Readability JSON file, apply its contents to this record."""
from modules.archive.lib import get_or_create_domain, get_title_from_html
readability_json_file = self.readability_json_file
if not readability_json_file or not readability_json_file.path.is_file():
raise InvalidArchive(f'Archive must have a readability json file: {self.singlefile_file.path}')
try:
with readability_json_file.path.open() as fh:
json_contents = json.load(fh)
url = json_contents.get('url')
title = json_contents.get('title')
except Exception as e:
raise InvalidArchive() from e
domain = None
if url:
domain = get_or_create_domain(Session.object_session(self), url)
if not title:
title = get_title_from_html(self.singlefile_path.read_text())
self.url = url
self.title = title
self.domain_id = domain.id if domain else None
def validate(self):
try:
self.read_readability_data()
self.read_singlefile_data()
self.validated = True
except Exception as e:
logger.warning(f'Unable to validate {self}', exc_info=e)
@staticmethod
def find_by_paths(paths, session):
archives = list(session.query(Archive).filter(Archive.singlefile_path.in_(paths)))
return archives
@property
def primary_path(self):
return self.singlefile_path
class Domain(Base, ModelHelper):
@ -119,3 +227,11 @@ class Domain(Base, ModelHelper):
session = Session.object_session(self)
session.execute('DELETE FROM archive WHERE domain_id=:id', dict(id=self.id))
session.query(Domain).filter_by(id=self.id).delete()
@validates('domain')
def validate_domain(self, key, value: str):
if not isinstance(value, str):
raise ValueError('Domain must be a string')
if len(value.split('.')) < 2:
raise ValueError(f'Domain must contain at least one "." domain={repr(value)}')
return value

View file

@ -29,6 +29,7 @@ class ArchiveSearchRequest:
domain: Optional[str] = None
offset: Optional[int] = None
limit: Optional[int] = None
order_by: Optional[str] = None
@dataclass

View file

@ -5,16 +5,22 @@ from http import HTTPStatus
def check_results(test_client, data, ids):
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == ids, \
f'{response.json["archives"][0]["id"]}..{response.json["archives"][19]["id"]}'
if ids:
assert response.json['archives'], f'Expected {len(ids)} archives but did not receive any'
assert [i['archive']['id'] for i in response.json['archives']] == ids, \
f'{response.json["archives"][0]["archive"]["id"]}..{response.json["archives"]["archive"][19]["id"]}'
def test_archives_search(test_session, archive_directory, archive_factory, test_client):
"""Archives can be searched by their title and their contents."""
# Search with no archives.
check_results(test_client, {'search_str': 'foo'}, [])
archive_factory('example.com', 'https://example.com/one', 'my archive', 'foo bar qux')
archive_factory('example.com', 'https://example.com/one', 'other archive', 'foo baz qux qux')
archive_factory('example.org', title='archive third', contents='baz qux qux qux')
archive_factory('example.org') # has no contents
test_session.commit()
# 1 and 2 contain "foo".
data = {'search_str': 'foo'}
@ -44,15 +50,20 @@ def test_archives_search(test_session, archive_directory, archive_factory, test_
data = {'search_str': 'third'}
check_results(test_client, data, [3, ])
# All titles contain "qux", but they contain different amounts. They are ordered by the amount.
# All contents contain "qux", but they contain different amounts. They are ordered by the amount.
data = {'search_str': 'qux'}
check_results(test_client, data, [3, 2, 1])
data = {'search_str': 'qux', 'order_by': 'bad order_by'}
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.BAD_REQUEST
def test_search_offset(archive_factory, test_client):
def test_search_offset(test_session, archive_factory, test_client):
"""Archive search can be offset."""
for i in range(500):
archive_factory('example.com', f'https://example.com/{i}', contents='foo bar')
test_session.commit()
data = {'search_str': None, 'offset': 0}
check_results(test_client, data, list(range(500, 480, -1)))
@ -66,28 +77,87 @@ def test_search_offset(archive_factory, test_client):
check_results(test_client, data, [])
def test_archives_search_no_query(archive_factory, test_client):
def test_archives_search_no_query(test_session, archive_factory, test_client):
"""Archive Search API endpoint does not require data in the body."""
# Add 100 random archives.
for _ in range(100):
archive_factory('example.com')
test_session.commit()
# All archives are returned when no `search_str` is passed.
request, response = test_client.post('/api/archive/search', content='{}')
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert response.json['totals']['archives'] == 100
# All archives are from "example.com".
data = dict(domain='example.com')
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert response.json['totals']['archives'] == 100
# No archives are from "example.org".
data = dict(domain='example.org')
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == []
assert [i['archive']['id'] for i in response.json['archives']] == []
assert response.json['totals']['archives'] == 0
def test_archive_and_domain_crud(test_session, test_client, archive_factory):
"""
Getting an Archive returns it's File. Testing deleting Archives.
"""
# Can get empty results.
request, response = test_client.get(f'/api/archive/1')
assert response.status_code == HTTPStatus.NOT_FOUND
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'] == []
archive1 = archive_factory(domain='example.com')
archive2 = archive_factory(domain='example.com')
test_session.commit()
# Archive1 has Archive2 as alternative.
request, response = test_client.get(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.OK
assert response.json['file']['archive']['id'] == archive1.id
assert response.json['alternatives'][0]['archive']['id'] == archive2.id
# Archive2 has Archive1 as alternative.
request, response = test_client.get(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.OK
assert archive2.id == response.json['file']['archive']['id']
assert response.json['alternatives'][0]['archive']['id'] == archive1.id
# Only one domain.
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'][0]['domain'] == 'example.com'
# Deleting works.
request, response = test_client.delete(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
# Trying to delete again returns NOT_FOUND.
request, response = test_client.delete(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NOT_FOUND
# Can't get deleted Archive.
request, response = test_client.get(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NOT_FOUND
# Archive2 no longer has alternatives.
request, response = test_client.get(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.OK
assert response.json['file']['archive']['id'] == archive2.id
assert response.json['alternatives'] == []
# No Archives, no Domains.
request, response = test_client.delete(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'] == []

View file

@ -1,17 +1,18 @@
import json
import pathlib
from datetime import datetime
from datetime import datetime, timedelta
from http import HTTPStatus
import mock
import pytest
from modules.archive import lib
from modules.archive.lib import get_or_create_domain, _refresh_archives, \
get_new_archive_files, delete_archive, do_archive, get_domains, get_archive_directory, group_archive_files, \
ArchiveFiles
from modules.archive.lib import get_or_create_domain, get_new_archive_files, delete_archive, do_archive, get_domains, \
group_archive_files, ArchiveFiles
from modules.archive.models import Archive, Domain
from wrolpi.dates import local_timezone
from wrolpi.db import get_db_session
from wrolpi.files.models import File
from wrolpi.media_path import MediaPath
from wrolpi.root_api import CustomJSONEncoder
from wrolpi.test.common import skip_circleci
@ -35,8 +36,8 @@ def make_fake_request_archive(readability=True, screenshot=True, title=True):
async def test_no_screenshot(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(screenshot=False)):
archive = await do_archive('https://example.com')
assert isinstance(archive.singlefile_path, MediaPath)
assert isinstance(archive.readability_path, MediaPath)
assert isinstance(archive.singlefile_path, pathlib.Path)
assert isinstance(archive.readability_path, pathlib.Path)
# Screenshot was empty
assert archive.screenshot_path is None
@ -45,8 +46,8 @@ async def test_no_screenshot(test_session):
async def test_no_readability(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(readability=False)):
archive = await do_archive('https://example.com')
assert isinstance(archive.singlefile_path, MediaPath)
assert isinstance(archive.screenshot_path, MediaPath)
assert isinstance(archive.singlefile_path, pathlib.Path)
assert isinstance(archive.screenshot_path, pathlib.Path)
# Readability empty
assert archive.readability_path is None
assert archive.readability_txt_path is None
@ -66,7 +67,7 @@ async def test_relationships(test_session, test_directory):
url = 'https://wrolpi.org:443'
domain = get_or_create_domain(session, url)
archive = Archive(
singlefile_path=f'{test_directory.name}/wrolpi.org:443/foo',
singlefile_file=File(path=test_directory / 'wrolpi.org:443/foo'),
title='bar',
url=url,
domain_id=domain.id,
@ -77,18 +78,7 @@ async def test_relationships(test_session, test_directory):
assert archive.domain == domain
@pytest.mark.asyncio
async def test_validate_paths(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive = await do_archive('https://example.com')
try:
with get_db_session(commit=True):
archive.singlefile_path = 'asdf'
except ValueError as e:
assert 'relative' in str(e), f'Relative path error was not raised'
def test_archive_refresh_deleted_archive(test_session, archive_directory, archive_factory):
def test_archive_refresh_deleted_archive(test_client, test_session, archive_directory, archive_factory):
"""
Archives/Domains should be deleted when archive files are deleted.
"""
@ -97,58 +87,58 @@ def test_archive_refresh_deleted_archive(test_session, archive_directory, archiv
archive3 = archive_factory('example.com')
archive4 = archive_factory('example.org')
archive5 = archive_factory()
test_session.commit()
# Empty directories should be ignored.
empty = archive_directory / 'empty'
empty.mkdir()
(archive_directory / 'empty').mkdir()
def check_counts(archive_count, domain_count):
assert test_session.query(Archive).count() == archive_count
assert test_session.query(Domain).count() == domain_count
assert test_session.query(Archive).count() == archive_count, 'Archive count does not match'
assert test_session.query(Domain).count() == domain_count, 'Domain count does not match'
# All 3 archives are already in the DB.
check_counts(archive_count=5, domain_count=3)
_refresh_archives()
check_counts(archive_count=5, domain_count=3)
# All 5 archives are already in the DB.
check_counts(archive_count=5, domain_count=2)
test_client.post('/api/files/refresh')
check_counts(archive_count=5, domain_count=1)
# Delete archive2's files, it's the latest for 'https://example.com/1'
for path in archive2.my_paths():
for path in archive2.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=4, domain_count=3)
test_client.post('/api/files/refresh')
check_counts(archive_count=4, domain_count=1)
# Delete archive1's files, now the URL is empty.
for path in archive1.my_paths():
for path in archive1.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=3, domain_count=3)
test_client.post('/api/files/refresh')
check_counts(archive_count=3, domain_count=0)
# Delete archive3, now there is now example.com domain
for path in archive3.my_paths():
for path in archive3.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=2, domain_count=2)
test_client.post('/api/files/refresh')
check_counts(archive_count=2, domain_count=0)
# Delete all the rest of the archives
for path in archive4.my_paths():
for path in archive4.my_files():
path.unlink()
for path in archive5.my_paths():
for path in archive5.my_files():
path.unlink()
_refresh_archives()
test_client.post('/api/files/refresh')
check_counts(archive_count=0, domain_count=0)
def test_fills_contents_with_refresh(test_session, archive_factory, test_client):
def test_fills_contents_with_refresh(test_client, test_session, archive_factory):
"""Refreshing archives fills in any missing contents."""
archive1 = archive_factory('example.com', 'https://example.com/one')
archive2 = archive_factory('example.com', 'https://example.com/one')
archive3 = archive_factory('example.org')
# Title can be found in archive3's singlefile
archive3 = archive_factory('example.org',
singlefile_contents='<html> some stuff<title>last title</title> other stuff</html>')
archive4 = archive_factory('example.org') # has no contents
# Clear the title, this will be filled in.
archive1.title = archive2.title = archive3.title = archive4.title = None
# Title can be found in archive3's singlefile
archive3.singlefile_path.path.write_text('<html> some stuff<title>last title</title> other stuff</html>')
contents_title_archive = [
('foo bar qux', 'my archive', archive1),
@ -156,26 +146,26 @@ def test_fills_contents_with_refresh(test_session, archive_factory, test_client)
('baz qux qux qux', None, archive3),
]
for contents, json_title, archive in contents_title_archive:
with archive.readability_txt_path.path.open('wt') as fh:
with archive.readability_txt_path.open('wt') as fh:
fh.write(contents)
if json_title:
archive.readability_json_path.path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
archive.readability_json_path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
test_session.commit()
# Contents are empty. Archives are all missing the title.
assert not archive1.contents
assert not archive2.contents
assert not archive3.contents
assert not archive4.contents
assert not archive1.singlefile_file.d_text
assert not archive2.singlefile_file.d_text
assert not archive3.singlefile_file.d_text
assert not archive4.singlefile_file.d_text
# Fill the contents.
_refresh_archives()
test_client.post('/api/files/refresh')
# The archives will be renamed with their title.
archive1, archive2, archive3, archive4 = test_session.query(Archive).order_by(Archive.id)
assert archive1.contents
assert archive2.contents
assert archive3.contents
assert not archive4.contents
assert archive1.singlefile_file.title
assert archive2.singlefile_file.title
assert archive3.singlefile_file.title
assert not archive4.singlefile_file.d_text
# Missing title is filled.
assert archive1.title == 'my archive'
@ -207,6 +197,7 @@ def test_get_domains(test_session, archive_factory):
archive1 = archive_factory('example.com')
archive2 = archive_factory('example.com')
archive3 = archive_factory('example.org')
test_session.commit()
assert [i['domain'] for i in get_domains()] == ['example.com', 'example.org']
@ -231,22 +222,22 @@ async def test_new_archive(test_session, fake_now):
# Everything is filled out.
assert isinstance(archive1, Archive)
assert archive1.archive_datetime is not None
assert isinstance(archive1.singlefile_path, MediaPath)
assert isinstance(archive1.readability_path, MediaPath)
assert isinstance(archive1.readability_txt_path, MediaPath)
assert isinstance(archive1.screenshot_path, MediaPath)
assert isinstance(archive1.singlefile_path, pathlib.Path)
assert isinstance(archive1.readability_path, pathlib.Path)
assert isinstance(archive1.readability_txt_path, pathlib.Path)
assert isinstance(archive1.screenshot_path, pathlib.Path)
assert archive1.title == 'ジにてこちら'
assert archive1.url is not None
assert archive1.domain is not None
# The actual files were dumped and read correctly.
with open(archive1.singlefile_path.path) as fh:
with open(archive1.singlefile_path) as fh:
assert fh.read() == '<html>\ntest single-file\nジにてこちら\n<title>some title</title></html>'
with open(archive1.readability_path.path) as fh:
with open(archive1.readability_path) as fh:
assert fh.read() == '<html>test readability content</html>'
with open(archive1.readability_txt_path.path) as fh:
with open(archive1.readability_txt_path) as fh:
assert fh.read() == '<html>test readability textContent</html>'
with open(archive1.readability_json_path.path) as fh:
with open(archive1.readability_json_path) as fh:
assert json.load(fh) == {'title': 'ジにてこちら', 'url': 'https://example.com'}
fake_now(datetime(2000, 1, 2))
@ -259,7 +250,7 @@ async def test_new_archive(test_session, fake_now):
async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 1))
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title == 'ジにてこちら'
async def fake_request_archive(_):
@ -273,7 +264,7 @@ async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 2))
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title == 'some title'
async def fake_request_archive(_):
@ -287,7 +278,7 @@ async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 3))
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title is None
@ -321,20 +312,20 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
fake_now(datetime(2000, 1, 1))
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive1 = await do_archive('example.com')
archive1 = await do_archive('https://example.com')
assert archive1.title == 'ジにてこちら'
assert str(archive1.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.html'
assert str(archive1.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.html'
assert str(archive1.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.txt'
assert str(archive1.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.json'
assert str(archive1.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.png'
assert str(archive1.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.html'
assert str(archive1.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.html'
assert str(archive1.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.txt'
assert str(archive1.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.json'
assert str(archive1.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.png'
async def fake_request_archive(_):
singlefile = '<html>\ntest single-file\nジにてこちら\n</html>' # no title in HTML
@ -347,18 +338,18 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
return singlefile, r, s
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive2 = await do_archive('example.com')
archive2 = await do_archive('https://example.com')
assert str(archive2.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.html'
assert str(archive2.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.html'
assert str(archive2.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.txt'
assert str(archive2.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.json'
assert str(archive2.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.png'
assert str(archive2.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.html'
assert str(archive2.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.html'
assert str(archive2.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.txt'
assert str(archive2.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.json'
assert str(archive2.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.png'
async def fake_request_archive(_):
singlefile = '<html>\ntest single-file\nジにてこちら\n<title>dangerous ;\\//_title</html></html>'
@ -371,62 +362,101 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
return singlefile, r, s
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive3 = await do_archive('example.com')
archive3 = await do_archive('https://example.com')
assert str(archive3.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.html'
assert str(archive3.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.html'
assert str(archive3.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
assert str(archive3.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.json'
assert str(archive3.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.png'
assert str(archive3.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.html'
assert str(archive3.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.html'
assert str(archive3.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
assert str(archive3.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.json'
assert str(archive3.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.png'
@skip_circleci
def test_refresh_archives(test_session, test_directory):
"""Archives can be found and put in the database. A single Archive will have multiple files."""
archive_directory = get_archive_directory()
def test_refresh_archives(test_session, test_directory, test_client, make_files_structure):
"""
Archives can be found and put in the database. A single Archive will have multiple files.
"""
# The start of a typical singlefile html file.
singlefile_contents = '''<!DOCTYPE html> <html lang="en"><!--
Page saved with SingleFile
url: https://example.com
saved date: Mon May 16 2022 23:51:35 GMT+0000 (Coordinated Universal Time)
--><head><meta charset="utf-8">'''
# Make some test files to refresh.
example_dir = archive_directory / 'example.com'
example_dir.mkdir()
(example_dir / '2021-10-05 16:20:10.346823.html').touch() # renames to 2021-10-05-16-20-10_NA.html
(example_dir / '2021-10-05 16:20:10.346823.png').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.txt').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.json').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.html').touch()
with (example_dir / '2021-10-05 16:20:10.346823-readability.json').open('wt') as fh:
json.dump({'url': 'foo'}, fh)
# These should log an error because they are missing the singlefile path.
(example_dir / '2021-10-05 16:20:10.readability.html').touch()
(example_dir / '2021-10-05 16:20:11.readability.json').touch()
# Bad files should also be ignored.
(example_dir / 'random file').touch()
(example_dir / '2021-10-05 16:20:10.346827-something.html').mkdir()
(example_dir / '2021-10-05 16:20:10 no extension').touch()
make_files_structure({
# These are all for an individual Archive.
'archive/example.com/2021-10-05 16:20:10.346823.html': singlefile_contents, # renames to 2021-10-05-16-20-10_NA
'archive/example.com/2021-10-05 16:20:10.346823.png': None,
'archive/example.com/2021-10-05 16:20:10.346823-readability.txt': 'article text contents',
'archive/example.com/2021-10-05 16:20:10.346823-readability.json': '{"url": "https://example.com"}',
'archive/example.com/2021-10-05 16:20:10.346823-readability.html': '<html></html>',
# These should log an error because they are missing the singlefile path.
'archive/example.com/2021-10-05 16:20:10.readability.html': '<html></html>',
'archive/example.com/2021-10-05 16:20:10.readability.json': None,
# Bad files should also be ignored.
'archive/example.com/random file': 'hello',
'archive/example.com/2021-10-05 16:20:10 no extension': None,
})
# The single archive is found.
_refresh_archives()
with get_db_session() as session:
assert session.query(Archive).count() == 1
test_client.post('/api/files/refresh')
assert test_session.query(Archive).count() == 1
# Running the refresh does not result in a new archive.
_refresh_archives()
with get_db_session() as session:
assert session.query(Archive).count() == 1
test_client.post('/api/files/refresh')
assert test_session.query(Archive).count() == 1
# Archives file format was changed, lets check the new formats are found.
(example_dir / '2021-10-05-16-20-11_The Title.html').touch()
(example_dir / '2021-10-05-16-20-11_The Title.readability.json').write_text(json.dumps({'url': 'bar'}))
_refresh_archives()
make_files_structure({
'archive/example.com/2021-10-05-16-20-11_The Title.html': '<html></html>',
'archive/example.com/2021-10-05-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
})
test_client.post('/api/files/refresh')
# The old formatted archive above is renamed.
assert (example_dir / '2021-10-05-16-20-10_NA.html').is_file()
with get_db_session() as session:
assert session.query(Archive).count() == 2
assert (test_directory / 'archive/example.com/2021-10-05-16-20-10_NA.html').is_file()
assert test_session.query(Archive).count() == 2
content = json.dumps({'search_str': 'text', 'model': 'archive'})
request, response = test_client.post('/api/files/search', content=content)
assert response.status_code == HTTPStatus.OK
assert response.json['files'], 'No files matched "text"'
assert response.json['files'][0]['model'] == 'archive', 'Returned file was not an archive'
assert response.json['files'][0].get('path') == 'archive/example.com/2021-10-05-16-20-10_NA.html', \
'Could not find readability text file containing "text"'
assert response.json['files'][0]['archive']['archive_datetime'], 'Archive has no datetime'
def test_refresh_archives_invalid_file(test_session, test_directory, test_client, make_files_structure):
"""
Invalid Archives should be removed during refresh.
"""
*_, bogus_path = make_files_structure({
'2022-09-04-16-20-11_The Title.html': '<html></html>',
'2022-09-04-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
'bogus file': None,
})
test_client.post('/api/files/refresh')
assert test_session.query(File).count() == 3 # Three files total.
assert test_session.query(Archive).count() == 1 # Only one real archive.
# Create an invalid Archive, it should be removed during refresh.
test_session.add(Archive(singlefile_path=bogus_path))
# Claim the bogus file as an Archive, this should be removed.
bogus_file: File = test_session.query(File).filter_by(path=bogus_path).one()
bogus_file.model = 'archive'
test_session.commit()
assert test_session.query(File).count() == 3
assert test_session.query(Archive).count() == 2
test_client.post('/api/files/refresh')
assert test_session.query(File).count() == 3, 'More files in DB than we created'
assert test_session.query(Archive).count() == 1, 'The bogus Archive was not removed'
assert bogus_file.model is None, 'Model was not removed'
def test_group_archive_files(test_directory):
@ -488,9 +518,10 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
archive1 = archive_factory('example.com', title='the title')
archive2 = archive_factory('example.com')
archive3 = archive_factory('example.org')
assert archive1.singlefile_path.path.is_file()
assert archive2.singlefile_path.path.is_file()
assert archive3.singlefile_path.path.is_file()
test_session.commit()
assert archive1.singlefile_path.is_file()
assert archive2.singlefile_path.is_file()
assert archive3.singlefile_path.is_file()
example_com_directory = archive_directory / 'example.com'
example_org_directory = archive_directory / 'example.org'
@ -532,6 +563,25 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
'example.org/2000-01-01-00-00-00_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.readability.json')),
]
# New Archive files still exist.
assert all(i.is_file() for i in archive1.my_paths())
assert all(i.is_file() for i in archive2.my_paths())
assert all(i.is_file() for i in archive3.my_paths())
assert all(i.is_file() for i in archive1.my_files())
assert all(i.is_file() for i in archive2.my_files())
assert all(i.is_file() for i in archive3.my_files())
def test_archive_files(test_directory):
af = lib.ArchiveFiles(
singlefile=test_directory / 'archive/singlefile.html',
)
assert repr(af) == \
"<ArchiveFiles singlefile='singlefile.html' readability=None readability_txt=None readability_json=None" \
" screenshot=None>"
def test_archive_order(test_session, test_directory, archive_factory):
archive1 = archive_factory()
archive2 = archive_factory()
archive2.archive_datetime += timedelta(seconds=10)
test_session.commit()
assert archive2 > archive1
assert archive1 < archive2

View file

@ -1 +0,0 @@
from .api import bp

View file

@ -1,91 +0,0 @@
import pathlib
from typing import List
from urllib.request import Request
from sanic import response
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.common import get_media_directory, api_param_limiter
from wrolpi.errors import InvalidFile
from wrolpi.root_api import get_blueprint, json_response
from . import lib, schema
bp = get_blueprint('Files', '/api/files')
def paths_to_files(paths: List[pathlib.Path]):
"""Convert Paths to what the React UI expects."""
media_directory = get_media_directory()
new_files = []
for path in paths:
stat = path.stat()
key = path.relative_to(media_directory)
modified = stat.st_mtime
if path.is_dir():
key = f'{key}/'
new_files.append(dict(
key=key,
modified=modified,
url=key,
name=path.name,
))
else:
# A File should know it's size.
new_files.append(dict(
key=key,
modified=modified,
size=stat.st_size,
url=key,
name=path.name,
))
return new_files
@bp.post('/')
@openapi.definition(
summary='List files in a directory',
body=schema.FilesRequest,
)
@validate(schema.FilesRequest)
async def get_files(_: Request, body: schema.FilesRequest):
directories = body.directories or []
files = lib.list_files(directories)
files = paths_to_files(files)
return json_response({'files': files})
@bp.post('/delete')
@openapi.definition(
summary='Delete a single file. Returns an error if WROL Mode is enabled.',
body=schema.DeleteRequest,
)
@validate(schema.DeleteRequest)
async def delete_file(_: Request, body: schema.DeleteRequest):
if not body.file:
raise InvalidFile('file cannot be empty')
lib.delete_file(body.file)
return response.empty()
files_limit_limiter = api_param_limiter(100)
files_offset_limiter = api_param_limiter(100, 0)
@bp.post('/refresh')
@openapi.description('Find and index all files')
async def refresh(_: Request):
lib.refresh_files()
return response.empty()
@bp.post('/search')
@openapi.definition(
summary='Search Files',
body=schema.FilesSearchRequest,
)
@validate(schema.FilesSearchRequest)
async def search_files(_: Request, body: schema.FilesSearchRequest):
files, total = lib.search(body.search_str, body.limit, body.offset)
return json_response(dict(files=files, totals=dict(files=total)))

View file

@ -1,21 +0,0 @@
import pathlib
from typing import List, Callable
import pytest
@pytest.fixture
def make_files_structure(test_directory) -> Callable[[List[str]], List[pathlib.Path]]:
def create_files(paths: List[str]) -> List[pathlib.Path]:
files = []
for name in paths:
path = test_directory / name
if name.endswith('/'):
path.mkdir()
else:
path.parent.mkdir(parents=True, exist_ok=True)
path.touch()
files.append(path)
return files
return create_files

View file

@ -1,185 +0,0 @@
import asyncio
import re
import subprocess
from functools import wraps
from pathlib import Path
from typing import List
from uuid import uuid4
import psycopg2
from sqlalchemy.orm import Session
from modules.files.models import File
from wrolpi.cmd import which
from wrolpi.common import get_media_directory, wrol_mode_check, walk, chunks, logger
from wrolpi.dates import from_timestamp
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
from wrolpi.errors import InvalidFile
from wrolpi.vars import PYTEST
logger = logger.getChild(__name__)
__all__ = ['list_files', 'delete_file', 'split_file_name', 'upsert_file', 'refresh_files', 'search', 'get_mimetype']
def filter_parent_directories(directories: List[Path]) -> List[Path]:
"""
Remove parent directories if their children are in the list.
>>> filter_parent_directories([Path('foo'), Path('foo/bar'), Path('baz')])
[Path('foo/bar'), Path('baz')]
"""
unique_children = set()
for directory in sorted(directories):
for parent in directory.parents:
# Remove any parent of this child.
if parent in unique_children:
unique_children.remove(parent)
unique_children.add(directory)
# Restore the original order.
new_directories = [i for i in directories if i in unique_children]
return new_directories
def list_files(directories: List[str]) -> List[Path]:
"""
List all files down to the directories provided. This includes all parent directories of the directories.
"""
media_directory = get_media_directory()
# Always display the media_directory files.
paths = list(media_directory.iterdir())
if directories:
directories = [media_directory / i for i in directories if i]
directories = filter_parent_directories(directories)
for directory in directories:
directory = media_directory / directory
for parent in directory.parents:
is_relative_to = str(media_directory).startswith(str(parent))
if parent == Path('.') or is_relative_to:
continue
paths.extend(parent.iterdir())
paths.extend(directory.iterdir())
return paths
@wrol_mode_check
def delete_file(file: str):
"""Delete a file in the media directory."""
file = get_media_directory() / file
if file.is_dir() or not file.is_file():
raise InvalidFile(f'Invalid file {file}')
file.unlink()
FILE_NAME_REGEX = re.compile(r'[_ ]')
def split_file_name(path: Path) -> List[str]:
"""
Split a file name into words.
>>> split_file_name(Path('foo.mp4'))
['foo']
>>> split_file_name(Path('foo bar.mp4'))
['foo', 'bar']
"""
words = [str(i) for i in FILE_NAME_REGEX.split(path.stem)]
return words
FILE_BIN = which('file', '/usr/bin/file')
def get_mimetype(path: Path) -> str:
"""Get the mimetype using the builtin `file` command."""
cmd = (FILE_BIN, '--mime-type', str(path.absolute()))
output = subprocess.check_output(cmd)
output = output.decode()
mimetype = output.split(' ')[-1].strip()
return mimetype
def upsert_file(path: Path, session: Session) -> File:
"""Update/insert a File in the DB. Gather metadata about it."""
file = session.query(File).filter_by(path=path).one_or_none()
if not file:
file = File(path=path)
session.add(file)
if not file.mimetype:
file.mimetype = get_mimetype(path)
if not file.title:
file.title = split_file_name(path)
if not file.size or not file.modification_datetime:
stat = path.stat()
file.size = stat.st_size
file.modification_datetime = file.modification_datetime or from_timestamp(stat.st_mtime)
return file
def _refresh_files():
"""Find and index all files"""
logger.info('Refreshing Files')
# Mark all files as unverified. Any records that are unverified after refresh will be deleted.
with get_db_curs(commit=True) as curs:
curs.execute('UPDATE file SET idempotency=null') # noqa
paths = filter(lambda i: i.is_file(), walk(get_media_directory()))
idempotency = str(uuid4())
for chunk in chunks(paths, 20):
with get_db_session(commit=True) as session:
for path in chunk:
file = upsert_file(path, session)
file.idempotency = idempotency
# Remove any records where the file no longer exists.
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM file WHERE idempotency IS NULL')
logger.info('Done refreshing Files')
@wraps(_refresh_files)
def refresh_files():
"""Schedule a refresh task if not testing. If testing, do a synchronous refresh."""
if PYTEST:
return _refresh_files()
async def _():
return _refresh_files()
asyncio.create_task(_())
def search(search_str: str, limit: int, offset: int):
"""
Search the "title" of each file. The title is the words in the file's stem.
"""
with get_db_curs() as curs:
stmt = '''
SELECT id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total
FROM file
WHERE textsearch @@ websearch_to_tsquery(%(search_str)s)
OFFSET %(offset)s LIMIT %(limit)s
'''
params = dict(search_str=search_str, offset=offset, limit=limit)
curs.execute(stmt, params)
try:
results = list(map(dict, curs.fetchall()))
except psycopg2.ProgrammingError:
# No files.
return [], 0
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
with get_db_session() as session:
results = get_ranked_models(ranked_ids, File, session)
results = [i.__json__() for i in results]
return results, total

View file

@ -1,36 +0,0 @@
from sqlalchemy import Integer, Column, String, Computed
from sqlalchemy.orm import deferred
from wrolpi.common import Base, ModelHelper, get_media_directory, tsvector
from wrolpi.dates import TZDateTime
from wrolpi.media_path import MediaPathType
class File(ModelHelper, Base):
__tablename__ = 'file'
id = Column(Integer, primary_key=True)
path = Column(MediaPathType)
idempotency = Column(String)
mimetype = Column(String)
modification_datetime = Column(TZDateTime)
size = Column(Integer)
title = deferred(Column(String)) # used by textsearch column
textsearch = deferred(
Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A'::"char")''')))
def __repr__(self):
return f'<File id={self.id} path={self.path.relative_to(get_media_directory())} mime={self.mimetype}>'
def __json__(self):
path = self.path.path.relative_to(get_media_directory())
d = dict(
id=self.id,
mimetype=self.mimetype,
modified=self.modification_datetime, # React browser expects this.
path=path,
size=self.size,
key=path,
)
return d

View file

@ -1,19 +0,0 @@
from dataclasses import dataclass, field
from typing import List, Optional
@dataclass
class FilesRequest:
directories: List[str] = field(default_factory=list)
@dataclass
class DeleteRequest:
file: str
@dataclass
class FilesSearchRequest:
search_str: str
limit: Optional[int] = 20
offset: Optional[int] = 0

View file

@ -1,159 +0,0 @@
import json
from http import HTTPStatus
from itertools import zip_longest
from unittest import mock
import pytest
from wrolpi.errors import API_ERRORS, WROLModeEnabled
from wrolpi.test.common import assert_dict_contains
from wrolpi.vars import PROJECT_DIR
def test_list_files_api(test_client, make_files_structure, test_directory):
files = [
'archives/bar.txt',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory/',
'videos/other video.mp4',
'videos/some video.mp4',
]
files = make_files_structure(files)
files[0].write_text('bar contents')
def check_get_files(directories, expected_files):
request, response = test_client.post('/api/files', content=json.dumps({'directories': directories}))
assert not response.json.get('errors')
results = sorted(response.json['files'], key=lambda i: i['key'])
for f1, f2 in zip_longest(results, expected_files):
if f1 is None or f2 is None:
assert f1 == f2
for key in f2.keys():
assert f1[key] == f2[key], f'{f1} != {f2}'
# Requesting no directories results in the top-level results.
expected = [dict(key='archives/'), dict(key='empty directory/'), dict(key='videos/')]
check_get_files([], expected)
# empty directory is empty
check_get_files(['empty directory'], expected)
expected = [
dict(key='archives/'),
dict(key='archives/bar.txt', size=12),
dict(key='archives/baz/'),
dict(key='archives/foo.txt'),
dict(key='empty directory/'),
dict(key='videos/'),
]
check_get_files(['archives'], expected)
# Sub-directories are supported.
expected = [
dict(key='archives/'),
dict(key='archives/bar.txt', size=12),
dict(key='archives/baz/'),
dict(key='archives/baz/bar.txt'),
dict(key='archives/baz/foo.txt'),
dict(key='archives/foo.txt'),
dict(key='empty directory/'),
dict(key='videos/'),
]
check_get_files(['archives/baz'], expected)
expected = [
dict(key='archives/'),
dict(key='archives/bar.txt', size=12),
dict(key='archives/baz/'),
dict(key='archives/foo.txt'),
dict(key='empty directory/'),
dict(key='videos/'),
dict(key='videos/other video.mp4'),
dict(key='videos/some video.mp4'),
]
check_get_files(['archives', 'videos'], expected)
# Order does not matter.
check_get_files(['archives', 'videos', 'empty directory'], expected)
check_get_files(['archives', 'empty directory', 'videos'], expected)
def test_delete_file(test_client, make_files_structure, test_directory):
files = ['bar.txt', 'baz/']
make_files_structure(files)
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'bar.txt'}))
assert response.status_code == HTTPStatus.NO_CONTENT
assert not (test_directory / 'bar.txt').is_file()
assert (test_directory / 'baz').is_dir()
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'baz'}))
assert response.status_code == HTTPStatus.BAD_REQUEST
assert not (test_directory / 'bar.txt').is_file()
assert (test_directory / 'baz').is_dir()
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'bad file'}))
assert response.status_code == HTTPStatus.BAD_REQUEST
@pytest.mark.parametrize(
'file', [
'',
]
)
def test_delete_invalid_file(test_client, file):
with mock.patch('modules.files.api.lib.delete_file') as mock_delete_file:
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': file}))
assert response.status_code == HTTPStatus.BAD_REQUEST
mock_delete_file.assert_not_called()
def test_delete_wrol_mode(test_client):
"""
Can't delete a file when WROL Mode is enabled.
"""
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'foo'}))
assert response.status_code == HTTPStatus.FORBIDDEN
assert response.json['code'] == API_ERRORS[WROLModeEnabled]['code']
def do_search(test_client, search_str, total, expected):
content = json.dumps({'search_str': search_str})
request, response = test_client.post('/api/files/search', content=content)
assert response.json['totals']['files'] == total
for file, expected in zip_longest(response.json['files'], expected):
assert_dict_contains(file, expected)
# FileBrowser in React requires a key. We use the path.
assert file['path'] == file['key']
def test_files_search(test_session, test_client, make_files_structure):
# You can search an empty directory.
do_search(test_client, 'nothing', 0, [])
# Create files in the temporary directory. Add some contents so the mimetype can be tested.
files = [
'foo_is_the_name.txt',
'archives/bar.txt',
'baz.mp4',
'baz baz two.mp4'
]
foo, bar, baz, baz2 = make_files_structure(files)
foo.write_text('foo contents')
bar.write_text('the bar contents')
baz.write_bytes((PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4').read_bytes())
baz2.write_bytes((PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4').read_bytes())
# Refresh so files can be searched.
request, response = test_client.post('/api/files/refresh')
assert response.status_code == HTTPStatus.NO_CONTENT
do_search(test_client, 'foo', 1, [dict(path='foo_is_the_name.txt', mimetype='text/plain', size=12)])
do_search(test_client, 'bar', 1, [dict(path='archives/bar.txt', mimetype='text/plain', size=16)])
do_search(test_client, 'baz', 2, [
dict(path='baz.mp4', mimetype='video/mp4', size=1055736),
dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736),
])
do_search(test_client, 'two', 1, [dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736)])
do_search(test_client, 'nothing', 0, [])

View file

@ -1,207 +0,0 @@
import shutil
from pathlib import Path
from typing import List, Iterable
import pytest
from sqlalchemy.orm import Session
from modules.files import lib
from modules.files.models import File
from wrolpi.errors import InvalidFile
from wrolpi.media_path import MediaPathType
from wrolpi.vars import PROJECT_DIR
def assert_files(session: Session, expected):
files = {str(i.path.relative) for i in session.query(File).all()}
assert files == set(expected)
def test_list_files(make_files_structure, test_directory):
files = [
'archives/bar.txt',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory/',
'videos/other video.mp4',
'videos/some video.mp4',
]
make_files_structure(files)
def check_list_files(path, expected):
result = lib.list_files(path)
result = sorted(result)
assert [str(i.relative_to(test_directory)) for i in result] == expected
check_list_files([], ['archives', 'empty directory', 'videos'])
# Falsey directories are ignored.
check_list_files([None, ''], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
# Trailing slash is ignored.
check_list_files(['empty directory/'], ['archives', 'empty directory', 'videos'])
check_list_files(['archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
# Including duplicate directories does not change results.
check_list_files(['archives/baz', 'archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['videos'], [
'archives',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
# All files and directories are listed.
check_list_files(['videos', 'archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
@pytest.mark.parametrize(
'directories,expected', [
([Path('foo')], [Path('foo')]),
([Path('foo'), Path('bar')], [Path('foo'), Path('bar')]),
([Path('foo'), Path('foo/bar'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('foo/bar'), Path('foo'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('baz'), Path('foo/bar'), Path('foo')], [Path('baz'), Path('foo/bar')]),
([Path('foo/bar'), Path('foo'), Path('foo/bar/baz')], [Path('foo/bar/baz')]),
]
)
def test_filter_parent_directories(directories, expected):
assert lib.filter_parent_directories(directories) == expected
def test_delete_file(make_files_structure, test_directory):
"""
File in the media directory can be deleted.
"""
files = [
'archives/foo.txt',
'bar.txt',
'baz/',
]
make_files_structure(files)
lib.delete_file('bar.txt')
assert (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
assert (test_directory / 'baz').is_dir()
lib.delete_file('archives/foo.txt')
assert not (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
with pytest.raises(InvalidFile):
lib.delete_file('baz')
with pytest.raises(InvalidFile):
lib.delete_file('does not exist')
assert (test_directory / 'baz').is_dir()
@pytest.mark.parametrize(
'path,expected',
[
('foo.mp4', ['foo']),
('foo bar.txt', ['foo', 'bar']),
('foo_bar.mp4', ['foo', 'bar']),
('foo-bar', ['foo-bar']),
('123 foo bar', ['123', 'foo', 'bar']),
('123foo bar', ['123foo', 'bar']),
]
)
def test_split_file_name(path, expected):
assert lib.split_file_name(Path(path)) == expected
def test_refresh_files(test_session, make_files_structure, test_directory):
files = make_files_structure([
'foo.txt',
'bar.txt',
'baz.txt',
])
foo, bar, baz = files
def get_relative_strs(files_: Iterable[MediaPathType]) -> List[str]:
return sorted([str(i.path.path.relative_to(test_directory)) for i in files_])
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == ['bar.txt', 'baz.txt', 'foo.txt']
baz.unlink()
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == ['bar.txt', 'foo.txt']
foo.unlink()
bar.unlink()
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == []
def test_mime_type(test_session, make_files_structure, test_directory):
"""Files module uses the `file` command to get the mimetype of each file."""
from PIL import Image
foo, bar, baz, empty = make_files_structure([
'dir/foo text.txt',
'dir/bar.jpeg',
'dir/baz.mp4',
'dir/empty',
])
foo.write_text('some text')
Image.new('RGB', (25, 25), color='grey').save(bar)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', baz)
lib.refresh_files()
assert_files(test_session, ['dir/foo text.txt', 'dir/bar.jpeg', 'dir/baz.mp4', 'dir/empty'])
foo = test_session.query(File).filter_by(path=f'{test_directory}/dir/foo text.txt').one()
bar = test_session.query(File).filter_by(path=f'{test_directory}/dir/bar.jpeg').one()
baz = test_session.query(File).filter_by(path=f'{test_directory}/dir/baz.mp4').one()
empty = test_session.query(File).filter_by(path=f'{test_directory}/dir/empty').one()
assert foo.mimetype == 'text/plain'
assert bar.mimetype == 'image/jpeg'
assert baz.mimetype == 'video/mp4'
assert empty.mimetype == 'inode/x-empty'

View file

@ -8,7 +8,7 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi.common import run_after, recursive_map
from wrolpi.errors import ValidationError
from wrolpi.root_api import get_blueprint, json_response
from . import common, inventory, schema
from modules.inventory import common, inventory, schema
NAME = 'inventory'
@ -16,25 +16,25 @@ bp = get_blueprint('Inventory', '/api/inventory')
@bp.get('/categories')
def _get_categories(_: Request):
def get_categories(_: Request):
categories = inventory.get_categories()
return json_response(dict(categories=categories))
@bp.get('/brands')
def _get_brands(_: Request):
def get_brands(_: Request):
brands = inventory.get_brands()
return json_response(dict(brands=brands))
@bp.get('/')
def inventories_get(_: Request):
def get_inventories(_: Request):
inventories = inventory.get_inventories()
return json_response(dict(inventories=inventories))
@bp.get('/<inventory_id:int>')
def inventory_get(_: Request, inventory_id: int):
def get_inventory(_: Request, inventory_id: int):
by_category = common.get_inventory_by_category(inventory_id)
by_subcategory = common.get_inventory_by_subcategory(inventory_id)
by_name = common.get_inventory_by_name(inventory_id)

View file

@ -123,7 +123,7 @@ def get_categories() -> List[Tuple[str, str]]:
"""
categories = DEFAULT_CATEGORIES.copy()
with get_db_curs() as curs:
curs.execute('SELECT DISTINCT subcategory, category FROM item ORDER BY 1, 2')
curs.execute("SELECT DISTINCT COALESCE(subcategory, ''), COALESCE(category, '') FROM item ORDER BY 1, 2")
try:
db_categories = curs.fetchall()
db_categories = [tuple(i) for i in db_categories]

View file

@ -84,6 +84,15 @@ def test_inventory_api(test_session, test_client):
inventory = response.json['inventories'][0]
inventory_id = inventory['id']
request, response = test_client.get(f'/api/inventory/{inventory_id}')
assert response.status_code == HTTPStatus.OK
request, response = test_client.get('/api/inventory/categories')
assert response.status_code == HTTPStatus.OK
request, response = test_client.get('/api/inventory/brands')
assert response.status_code == HTTPStatus.OK
inventory = dict(name='new name')
request, response = test_client.put(f'/api/inventory/{inventory_id}', content=json.dumps(inventory))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code

View file

@ -35,5 +35,5 @@ class MapFile(Base, ModelHelper):
'path': self.path,
'imported': self.imported,
'size': self.size,
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path.path, self.size)),
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path, self.size)),
}

View file

@ -0,0 +1,21 @@
from unittest import mock
from modules.map import lib
def test_get_map_paths(test_directory):
pbf_dir = test_directory / 'map/pbf'
pbf_dir.mkdir(parents=True)
pbf = pbf_dir / 'map.osm.pbf'
pbf.touch()
dump = pbf_dir / 'map.dump'
dump.touch()
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
mock_check_output.return_value = b'OpenStreetMap Protocolbuffer Binary Format'
assert lib.get_map_paths() == [pbf, ]
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
mock_check_output.return_value = b'PostgreSQL custom database dump'
assert lib.get_map_paths() == [dump, ]

View file

@ -1 +1,123 @@
from modules.videos import downloader
import json
import pathlib
from abc import ABC
from typing import List, Dict
from sqlalchemy.orm import Session
from modules.videos.models import Video, Channel
from wrolpi.captions import read_captions, extract_captions
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
from wrolpi.db import get_db_curs
from wrolpi.files.indexers import Indexer, register_indexer
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .downloader import video_downloader # Import downloaders so they are registered.
logger = logger.getChild(__name__)
__all__ = ['video_modeler', 'VideoIndexer']
@register_modeler
def video_modeler(groups: Dict[str, List[File]], session: Session):
# Only process groups that contain a video file.
count = 0
local_groups = groups.copy()
for stem, group in local_groups.items():
video_file = next((i for i in group if i.mimetype.split('/')[0] == 'video' and i.path.suffix != '.part'), None)
if not video_file:
# Not a video group.
continue
session.flush(group)
poster_file = next((i for i in group if i.mimetype.split('/')[0] == 'image'), None)
caption_file = next((i for i in group if i.path.name.endswith('.en.vtt') or i.path.name.endswith('.en.srt')),
None)
info_json_file = next((i for i in group if i.path.name.endswith('.info.json')), None)
if poster_file:
poster_file.associated = True
poster_file.do_index()
if caption_file:
caption_file.associated = True
caption_file.do_index()
if info_json_file:
info_json_file.associated = True
info_json_file.do_index()
video: Video = Video.upsert(video_file, session)
video.poster_file = poster_file
video.caption_file = caption_file
video.info_json_file = info_json_file
video.video_file.do_index()
video.validate(session)
# Remove this group, it will not be processed by other modelers.
del groups[stem]
count += 1
if not count % 100:
# Commit occasionally.
session.commit()
@register_after_refresh
@limit_concurrent(1)
def video_cleanup():
# Claim all Videos in a Channel's directory for that Channel.
logger.info('Claiming Videos for their Channels')
with get_db_curs(commit=True) as curs:
curs.execute('''
UPDATE video v
SET channel_id = c.id
FROM channel c
WHERE
v.video_path LIKE c.directory || '/%'::VARCHAR
AND v.channel_id IS NULL
''')
EXTRACT_SUBTITLES = False
@register_indexer('video')
class VideoIndexer(Indexer, ABC):
"""Handles video files like mp4/ogg."""
@staticmethod
def get_description(file):
video_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
info_json_path = video_path.with_suffix('.info.json')
if info_json_path.is_file():
with info_json_path.open('rt') as fh:
try:
return json.load(fh)['description']
except Exception:
if not PYTEST:
logger.warning(f'Video info json file exists, but cannot get description. {file}')
return None
@classmethod
def create_index(cls, file):
"""
Index the video file and it's associated files.
a = title.
b = <empty>
c = description from info json.
d = captions from info json, or extracted from video file.
"""
a = cls.get_title(file)
c = cls.get_description(file)
# Detect the associated caption file.
caption_path = file.path.with_suffix('.en.vtt')
d = None
if caption_path.is_file():
d = read_captions(caption_path)
elif EXTRACT_SUBTITLES:
d = extract_captions(file.path) or ''
return a, None, c, d

View file

@ -1,16 +1,12 @@
import asyncio
from functools import wraps
from http import HTTPStatus
from typing import List
from sanic import Blueprint, response
from sanic.request import Request
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.common import create_websocket_feed, get_sanic_url, \
wrol_mode_check
from wrolpi.common import logger
from wrolpi.common import wrol_mode_check
from wrolpi.root_api import add_blueprint, json_response
from wrolpi.schema import JSONErrorResponse
from . import lib, schema
@ -29,43 +25,6 @@ add_blueprint(bp)
logger = logger.getChild(__name__)
refresh_queue, refresh_event = create_websocket_feed('refresh', '/feeds/refresh', content_bp)
@content_bp.post('/refresh')
@content_bp.post('/refresh/<channel_id:str>')
@openapi.description('Search for videos that have previously been downloaded and stored.')
@openapi.response(HTTPStatus.OK, schema.StreamResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@wrol_mode_check
async def refresh(_, channel_id: int = None):
refresh_logger = logger.getChild('refresh')
stream_url = get_sanic_url(scheme='ws', path='/api/videos/feeds/refresh')
# Only one refresh can run at a time
if refresh_event.is_set():
return response.json({'error': 'Refresh already running', 'stream_url': stream_url}, HTTPStatus.CONFLICT)
refresh_event.set()
async def do_refresh():
try:
refresh_logger.info('refresh started')
channel_ids = [channel_id] if channel_id else None
await refresh_videos(channel_ids)
refresh_logger.info('refresh complete')
except Exception:
raise
finally:
refresh_event.clear()
coro = do_refresh()
asyncio.ensure_future(coro)
refresh_logger.debug('do_refresh scheduled')
return response.json({'code': 'stream-started', 'stream_url': stream_url})
@content_bp.post('/download/<channel_id:int>')
@openapi.description('Update a channel catalog, download any missing videos')
@ -77,11 +36,6 @@ def download(_, channel_id: int = None):
return response.empty()
@wraps(lib.refresh_videos)
async def refresh_videos(channel_ids: List[int] = None):
return lib.refresh_videos(channel_ids=channel_ids)
@content_bp.post('/favorite')
@openapi.definition(
description='Toggle the favorite flag on a video',

View file

@ -1,55 +0,0 @@
#! /usr/bin/env python3
from pathlib import Path
from typing import Generator, Union
import srt
import webvtt
from wrolpi.common import logger
__all__ = ['get_captions']
def get_caption_text(caption_path: Union[str, Path]) -> Generator:
"""Return all text from each caption of a caption file."""
if str(caption_path).endswith('vtt'):
for caption in webvtt.read(caption_path):
text = str(caption.text).strip()
yield text
else:
with open(caption_path, 'rt') as fh:
contents = fh.read()
for subtitle in srt.parse(contents):
yield subtitle.content
def get_unique_caption_lines(caption_path: Union[str, Path]) -> Generator:
"""
Return all unique lines from each caption of a caption file.
"""
last_line = None
for text in get_caption_text(caption_path):
for line in text.split('\n'):
if line and line != last_line:
last_line = line
yield line
def get_captions(caption_path: Path):
"""Parse video captions from the video's captions file."""
try:
lines = get_unique_caption_lines(str(caption_path))
block = '\n'.join(lines)
return block
except UnicodeDecodeError:
# Failed to decode the caption file
# TODO handle this error
pass
except webvtt.errors.MalformedFileError:
# File format is broken
pass
except webvtt.errors.MalformedCaptionError:
# Captions form is broken somehow
pass
logger.debug(f'Failed to parse caption file {caption_path}')

View file

@ -1,4 +1,5 @@
import asyncio
import pathlib
from http import HTTPStatus
from sanic import response, Blueprint
@ -8,11 +9,12 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi.common import logger, wrol_mode_check, make_media_directory, \
get_media_directory, run_after
from wrolpi.errors import UnknownDirectory, UnknownChannel
from wrolpi.root_api import json_response
from wrolpi.schema import JSONErrorResponse
from wrolpi.vars import PYTEST
from . import lib
from .. import schema
from .. import schema, Channel
from ..lib import save_channels_config
channel_bp = Blueprint('Channel', url_prefix='/api/videos/channels')
@ -32,7 +34,7 @@ async def get_channels(_: Request):
@openapi.description('Get a Channel')
@openapi.response(HTTPStatus.OK, schema.ChannelResponse)
def channel_get(_: Request, channel_id: int = None):
channel = lib.get_channel(channel_id=channel_id)
channel = lib.get_channel(channel_id=channel_id, return_dict=True)
channel.pop('info_json')
return json_response({'channel': channel})
@ -55,9 +57,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
# Refresh the videos asynchronously
if not PYTEST:
from ..api import refresh_videos
coro = refresh_videos([channel.id])
asyncio.ensure_future(coro)
asyncio.ensure_future(channel.refresh_files())
return response.json({'success': 'Channel created successfully'}, HTTPStatus.CREATED,
{'Location': f'/api/videos/channels/{channel.id}'})
@ -74,6 +74,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
@run_after(save_channels_config)
@wrol_mode_check
def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
body.directory = pathlib.Path(body.directory) if body.directory else None
channel = lib.update_channel(data=body, channel_id=channel_id)
return response.raw('', HTTPStatus.NO_CONTENT,
headers={'Location': f'/api/videos/channels/{channel.id}'})
@ -86,3 +87,25 @@ def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
def channel_delete(_: Request, channel_id: int):
lib.delete_channel(channel_id=channel_id)
return response.raw('', HTTPStatus.NO_CONTENT)
@channel_bp.post('/refresh/<channel_id:int>')
@openapi.description('Refresh all files in the Channel directory')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_refresh(_: Request, channel_id: int):
channel: Channel = lib.get_channel(channel_id=channel_id, return_dict=False)
if not channel:
raise UnknownChannel()
asyncio.ensure_future(channel.refresh_files())
return response.empty()
@channel_bp.post('/download/<channel_id:int>')
@openapi.description('Download the latest catalog for a Channel, download any missing videos.')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_download(_: Request, channel_id: int):
lib.download_channel(channel_id)
return response.empty()

View file

@ -1,5 +1,6 @@
import functools
from pathlib import Path
from typing import List, Dict, Union
from typing import List, Dict, Union, Optional
from sqlalchemy.orm import Session
from sqlalchemy.orm.exc import NoResultFound
@ -7,7 +8,6 @@ from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import run_after, logger, \
get_media_directory
from wrolpi.db import get_db_curs, optional_session
from wrolpi.downloader import download_manager
from wrolpi.errors import UnknownChannel, UnknownDirectory, APIError, ValidationError, InvalidDownload
from .. import schema
from ..common import check_for_channel_conflicts
@ -125,6 +125,9 @@ def update_channel(session: Session, *, data: schema.ChannelPutRequest, channel_
data.directory.mkdir()
else:
raise UnknownDirectory()
elif data.directory:
# Keep the old directory because the user can't change a Channel's directory.
data.directory = channel.directory
# Verify that the URL/Name/directory aren't taken
check_for_channel_conflicts(
@ -165,7 +168,7 @@ def create_channel(session: Session, data: schema.ChannelPostRequest, return_dic
name=data.name,
url=data.url or None,
match_regex=data.match_regex,
directory=data.directory, # noqa
directory=get_media_directory() / data.directory,
download_frequency=data.download_frequency,
source_id=data.source_id,
)

View file

@ -9,11 +9,10 @@ import mock
import pytest
from modules.videos.channel.lib import delete_channel
from modules.videos.common import get_no_channel_directory
from modules.videos.models import Channel, Video
from modules.videos.models import Channel
from wrolpi.dates import now
from wrolpi.db import get_db_session
from wrolpi.downloader import download_manager, Download
from wrolpi.downloader import download_manager, Download, DownloadResult
from wrolpi.test.common import assert_dict_contains
@ -26,31 +25,8 @@ def test_get_channels(test_directory, channel_factory, test_client):
assert len(response.json['channels']) == 3
@mock.patch('modules.videos.lib.validate_video')
def test_refresh_no_channel(_, test_session, test_client, test_directory, simple_channel):
# A regular user-defined channel
vid1 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title1.mp4'
vid1.touch()
vid2 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title2.mp4'
vid2.touch()
# The special NO CHANNEL directory.
no_channel_path = get_no_channel_directory()
vid3 = no_channel_path / 'some video.mp4'
vid3.touch()
no_channel_subdir = no_channel_path / 'subdir'
no_channel_subdir.mkdir()
vid4 = no_channel_subdir / 'other video.mp4'
vid4.touch()
test_client.post('/api/videos/refresh')
assert test_session.query(Video).count() == 4
def test_get_video(test_client, test_session, simple_channel, video_factory):
"""
Test that you get can information about a video. Test that video file can be gotten.
"""
"""Test that you get can information about a video. Test that video file can be gotten."""
now_ = now()
video1 = video_factory(channel_id=simple_channel.id)
video1.upload_date, video1.title = now_, 'vid1'
@ -66,11 +42,11 @@ def test_get_video(test_client, test_session, simple_channel, video_factory):
# Get the video info we inserted
_, response = test_client.get('/api/videos/video/1')
assert response.status_code == HTTPStatus.OK, response.json
assert_dict_contains(response.json['video'], {'title': 'vid1'})
assert_dict_contains(response.json['file']['video'], {'title': 'vid1'})
# The next video is included.
assert response.json['prev'] is None
assert_dict_contains(response.json['next'], {'title': 'vid2'})
assert_dict_contains(response.json['next']['video'], {'title': 'vid2'})
def test_channel_no_download_frequency(test_client, test_session, test_directory, simple_channel):
@ -118,7 +94,7 @@ def test_channel_frequency_update(download_channel, test_client, test_session, t
assert old_frequency
data = dict(
directory=str(download_channel.directory.path),
directory=str(download_channel.directory),
name=download_channel.name,
url=download_channel.url,
download_frequency=100,
@ -192,73 +168,12 @@ def test_channel_download_crud(test_session, simple_channel):
assert test_session.query(Download).count() == 0
def test_video_search(test_client, test_session):
"""
Test that videos can be searched and that their order is by their textsearch rank.
"""
# These captions have repeated letters, so they will be higher in the ranking.
videos = [
('1', 'b b b b e d d'),
('2', '2 b b b d'),
('3', 'b b'),
('4', 'b e e'),
('5', ''),
]
for title, caption in videos:
test_session.add(Video(title=title, caption=caption, video_path='foo'))
test_session.commit()
def do_search(search_str, limit=20):
d = json.dumps({'search_str': search_str, 'limit': limit})
_, resp = test_client.post('/api/videos/search', content=d)
return resp
def search_is_as_expected(resp, expected):
assert resp.status_code == HTTPStatus.OK
response_ids = [i['id'] for i in resp.json['videos']]
assert response_ids == expected
assert resp.json['totals']['videos'] == len(expected)
# Repeated runs should return the same result
for _ in range(2):
# Only videos with a b are returned, ordered by the amount of b's
response = do_search('b')
search_is_as_expected(response, [1, 2, 3, 4])
# Only two captions have e
response = do_search('e')
search_is_as_expected(response, [4, 1])
# Only two captions have d
response = do_search('d')
search_is_as_expected(response, [1, 2])
# 5 can be gotten by it's title
response = do_search('5')
search_is_as_expected(response, [5])
# only video 1 has e and d
response = do_search('e d')
search_is_as_expected(response, [1])
# video 1 and 4 have b and e, but 1 has more
response = do_search('b e')
search_is_as_expected(response, [1, 4])
# Check totals are correct even with a limit
response = do_search('b', 2)
assert [i['id'] for i in response.json['videos']] == [1, 2]
assert response.json['totals']['videos'] == 4
def test_video_file_name(test_session, simple_video, test_client):
"""
If a Video has no title, the front-end can use the file name as the title.
"""
"""If a Video has no title, the front-end can use the file name as the title."""
_, resp = test_client.get(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.OK
assert resp.json['video']['video_path'] == 'simple_video.mp4'
assert resp.json['video'].get('stem') == 'simple_video'
assert resp.json['file']['video']['video_path'] == 'simple_video.mp4'
assert resp.json['file']['video'].get('stem') == 'simple_video'
def test_channel_conflicts(test_client, test_session, test_directory):
@ -339,26 +254,46 @@ def test_channel_empty_url_doesnt_conflict(test_client, test_session, test_direc
@pytest.mark.asyncio
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager):
"""A Channel cannot be downloaded until it has been refreshed."""
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager, video_factory):
"""A Channel cannot be downloaded until it has been refreshed.
Videos already downloaded are not downloaded again."""
vid = video_factory(channel_id=download_channel.id, with_video_file=True, with_poster_ext='jpg')
vid.url = 'https://example.com/1'
d = download_channel.get_download()
test_session.commit()
assert not d.next_download
def check_refreshed(expected: bool):
def assert_refreshed(expected: bool):
channel = test_session.query(Channel).one()
assert channel.refreshed == expected
check_refreshed(False)
assert_refreshed(False)
test_session.commit()
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info:
mock_extract_info.return_value = {'entries': [], 'url': 'foo', 'uploader': 'some uploader',
downloaded_urls = []
async def do_download(_, download):
# Track all downloaded URLs.
downloaded_urls.append(download.url)
return DownloadResult(success=True)
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.do_download', do_download):
entries = [
dict(id=vid.source_id, view_count=0, webpage_url='https://example.com/1'), # Already downloaded.
dict(id='not downloaded', view_count=0, webpage_url='https://example.com/2'),
]
mock_extract_info.return_value = {'entries': entries, 'url': 'foo', 'uploader': 'some uploader',
'channel_id': 'the id', 'id': 'the id'}
await video_download_manager.do_downloads()
await video_download_manager.wait_for_all_downloads()
check_refreshed(True)
# Channel was refreshed before downloading videos.
assert_refreshed(True)
assert d.next_download
# Only the missing video was downloaded.
assert downloaded_urls == ['https://example.com/2']
def test_channel_post_directory(test_session, test_client, test_directory):
@ -367,7 +302,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
data = dict(name='foo', directory='foo')
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
assert response.status_code == HTTPStatus.CREATED
directory = test_session.query(Channel).filter_by(id=1).one().directory.path
directory = test_session.query(Channel).filter_by(id=1).one().directory
assert (test_directory / 'foo') == directory
assert not directory.is_dir()
assert directory.is_absolute()
@ -376,7 +311,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
data = dict(name='bar', directory='bar', mkdir=True)
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
assert response.status_code == HTTPStatus.CREATED
directory = test_session.query(Channel).filter_by(id=2).one().directory.path
directory = test_session.query(Channel).filter_by(id=2).one().directory
assert (test_directory / 'bar') == directory
assert directory.is_dir()
assert directory.is_absolute()
@ -455,7 +390,7 @@ def test_channel_crud(test_session, test_client, test_directory):
assert response.status_code == HTTPStatus.NOT_FOUND
def test_search_channel_videos(test_client, test_session):
def test_search_videos_channel(test_client, test_session, video_factory):
with get_db_session(commit=True) as session:
channel1 = Channel(name='Foo')
channel2 = Channel(name='Bar')
@ -469,29 +404,29 @@ def test_search_channel_videos(test_client, test_session):
d = dict(channel_id=channel1.id)
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 0
assert len(response.json['files']) == 0
with get_db_session(commit=True) as session:
vid1 = Video(title='vid1', channel_id=channel2.id, video_path='foo')
vid2 = Video(title='vid2', channel_id=channel1.id, video_path='foo')
vid1 = video_factory(title='vid1', channel_id=channel2.id)
vid2 = video_factory(title='vid2', channel_id=channel1.id)
session.add(vid1)
session.add(vid2)
# Videos are gotten by their respective channels
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 1
assert response.json['totals']['videos'] == 1
assert_dict_contains(response.json['videos'][0], dict(id=2, title='vid2', channel_id=channel1.id))
assert len(response.json['files']) == 1
assert response.json['totals']['files'] == 1
assert_dict_contains(response.json['files'][0], dict(path='vid2.mp4', video=dict(channel_id=channel1.id)))
d = dict(channel_id=channel2.id)
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 1
assert_dict_contains(response.json['videos'][0], dict(id=1, title='vid1', channel_id=channel2.id))
assert len(response.json['files']) == 1
assert_dict_contains(response.json['files'][0], dict(path='vid1.mp4', video=dict(channel_id=channel2.id)))
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory):
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory, assert_video_search):
for i in range(50):
video_factory(channel_id=simple_channel.id)
@ -500,7 +435,7 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
test_session.flush()
test_session.refresh(channel2)
video_factory(channel_id=channel2.id)
test_session.add(Video(title='vid2', channel_id=channel2.id, video_path='foo'))
video_factory(title='vid2', channel_id=channel2.id, with_video_file=True)
test_session.commit()
# Get first, second, third, and empty pages of videos.
@ -513,10 +448,8 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
]
last_ids = []
for offset, video_count in tests:
d = dict(channel_id=simple_channel.id, order_by='id', offset=offset)
_, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == video_count
current_ids = [i['id'] for i in response.json['videos']]
_, response = assert_video_search(channel_id=simple_channel.id, order_by='id', offset=offset, limit=20)
assert len(response.json['files']) == video_count, 'Returned videos does not match'
current_ids = [i['video']['id'] for i in response.json['files']]
assert current_ids != last_ids, f'IDs are unchanged current_ids={current_ids}'
last_ids = current_ids

View file

@ -1,9 +1,11 @@
from unittest import mock
import pytest
from modules.videos import schema
from modules.videos.channel import lib
from modules.videos.lib import save_channels_config, import_channels_config
from modules.videos.models import Channel
from modules.videos.models import Channel, Video
from wrolpi.errors import UnknownChannel
@ -89,3 +91,39 @@ def test_channels_no_url(test_session, test_directory):
save_channels_config()
import_channels_config()
def test_refresh_videos_finds_channel(test_session, test_client, channel_factory, video_factory):
"""A Video will be assigned to a Channel by its info json first, then to its directory."""
channel1 = channel_factory(source_id='channel1')
channel2 = channel_factory(source_id='channel2')
# Put the video in the wrong directory. It should be matched to the info json channel first, then to it's directory
channel1.source_id = 'channel source id'
vid1 = video_factory(
with_video_file=(channel2.directory / 'video1.mp4'),
with_info_json={'channel_id': channel1.source_id},
)
test_session.commit()
assert not vid1.channel_id
with mock.patch('wrolpi.common.after_refresh') as mock_after_refresh:
# Channel can be found in `video_cleanup`, prevent that with mock.
mock_after_refresh.side_effect = []
test_client.post('/api/files/refresh')
# Channel in info_json was found during validation.
assert vid1.channel_id == channel1.id
# Refresh using cleanup, info json should be trusted.
test_client.post('/api/files/refresh')
assert vid1.channel_id == channel1.id
# Remove the info json, the Video's Channel will be its directory.
vid1.channel_id = None
vid1.info_json_path.unlink()
test_session.commit()
test_client.post('/api/files/refresh')
test_session.refresh(vid1)
vid1 = test_session.query(Video).one()
assert vid1.channel_id == channel2.id

View file

@ -1,12 +1,15 @@
import json
import json
import os
import pathlib
import re
import subprocess
import tempfile
from datetime import timedelta
from decimal import Decimal
from functools import partial
from pathlib import Path
from typing import Union, Tuple, List, Set, Iterable, Optional
from typing import Union, List, Set, Iterable, Optional, Tuple
import PIL
from PIL import Image
@ -14,13 +17,13 @@ from sqlalchemy.orm import Session
from wrolpi.cmd import which
from wrolpi.common import logger, iterify, get_media_directory, \
minimize_dict, any_extensions
minimize_dict, any_extensions, match_paths_to_suffixes
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.errors import UnknownFile, ChannelNameConflict, ChannelURLConflict, \
from wrolpi.errors import ChannelNameConflict, ChannelURLConflict, \
ChannelDirectoryConflict, ChannelSourceIdConflict
from wrolpi.media_path import MediaPath
from wrolpi.files.models import File
from wrolpi.vars import DEFAULT_FILE_PERMISSIONS
from .models import Channel, Video
from .models import Channel
logger = logger.getChild(__name__)
@ -32,6 +35,7 @@ MINIMUM_VIDEO_KEYS = {'id', 'title', 'upload_date', 'duration', 'channel', 'chan
'view_count'}
# These are the supported video formats. These are in order of their preference.
VIDEO_EXTENSIONS = ('mp4', 'ogg', 'webm', 'flv')
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
FFMPEG_BIN = which('ffmpeg', '/usr/bin/ffmpeg', warn=True)
@ -49,40 +53,13 @@ def get_videos_directory() -> pathlib.Path:
def get_no_channel_directory() -> pathlib.Path:
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist."""
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist.""" # noqa
directory = get_videos_directory() / 'NO CHANNEL'
if not directory.is_dir():
directory.mkdir(parents=True)
return directory
VALID_VIDEO_KINDS = {'video', 'caption', 'poster', 'description', 'info_json'}
def get_absolute_video_path(video: Video, kind: str = 'video') -> MediaPath:
if kind not in VALID_VIDEO_KINDS:
raise Exception(f'Unknown video path kind {kind}')
path = getattr(video, f'{kind}_path')
if path:
return path
raise UnknownFile()
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
def generate_video_paths(directory: Union[str, pathlib.Path]) -> Tuple[str, pathlib.Path]:
"""Generate a list of video paths in the provided directory."""
directory = pathlib.Path(directory)
for child in directory.iterdir():
if child.is_file() and match_video_extensions(child.name):
child = child.absolute()
yield child
elif child.is_dir():
yield from generate_video_paths(child)
@iterify(set)
def remove_duplicate_video_paths(paths: Iterable[Path]) -> Set[Path]:
"""
@ -190,13 +167,46 @@ def get_matching_directories(path: Union[str, Path]) -> List[str]:
return paths
def generate_video_poster(video_path: Path) -> Path:
"""Create a poster (aka thumbnail) next to the provided video_path."""
def duration_to_seconds(duration: str) -> int:
"""Convert ffmpeg duration to seconds"""
hours, minutes, seconds = duration.split(':')
seconds, microseconds = seconds.split('.')
return int(timedelta(
hours=int(hours),
minutes=int(minutes),
seconds=int(seconds),
microseconds=int(microseconds),
).total_seconds())
def seconds_to_duration(seconds: int) -> str:
"""Convert integer seconds to ffmpeg duration"""
hours, seconds = divmod(seconds, 3600)
minutes, seconds = divmod(seconds, 60)
return f'{hours:02}:{minutes:02}:{seconds:02}.000'
DURATION_REGEX = re.compile(r'Duration: ([\d:.]+),')
def get_duration_from_ffmpeg(stderr: str) -> int:
"""Extract the duration from FFPEG's stderr, return this duration as integer seconds."""
duration = DURATION_REGEX.findall(stderr)
duration = duration_to_seconds(duration[0]) if duration else None
return duration
def generate_video_poster(video_path: Path) -> Tuple[Path, Optional[int]]:
"""Create a poster (aka thumbnail) next to the provided video_path. Also returns the video duration."""
poster_path = video_path.with_suffix('.jpg')
cmd = (FFMPEG_BIN, '-n', '-i', str(video_path), '-f', 'mjpeg', '-vframes', '1', '-ss', '00:00:05.000',
str(poster_path))
if poster_path.exists():
# Poster already exists.
return poster_path, None
try:
subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr = proc.stderr.decode()
logger.info(f'Generated poster at {poster_path}')
except subprocess.CalledProcessError as e:
logger.warning(f'FFMPEG poster generation failed with stdout: {e.stdout.decode()}')
@ -205,7 +215,8 @@ def generate_video_poster(video_path: Path) -> Path:
if not poster_path.exists():
raise Exception(f'Failed to find generated poster! {poster_path}')
return poster_path
duration = get_duration_from_ffmpeg(stderr)
return poster_path, duration
def convert_image(existing_path: Path, destination_path: Path, ext: str = 'jpeg'):
@ -237,7 +248,7 @@ def is_valid_poster(poster_path: Path) -> bool:
FFPROBE_BIN = which('ffprobe', '/usr/bin/ffprobe', warn=True)
def get_video_duration(video_path: Path) -> Optional[int]:
def get_video_duration(video_path: Path) -> int:
"""Get the duration of a video in seconds. Do this using ffprobe."""
if not isinstance(video_path, Path):
video_path = Path(video_path)
@ -261,25 +272,29 @@ def get_video_duration(video_path: Path) -> Optional[int]:
def check_for_video_corruption(video_path: Path) -> bool:
"""Uses ffprobe to check for specific ways a video file can be corrupt."""
if not isinstance(video_path, Path):
video_path = Path(video_path)
"""Uses ffprobe to check for specific ways a video file can be corrupt.
Also attempts to screenshot the video near the end of it's duration, this check if the video is corrupted somewhere
in the middle."""
video_path = Path(video_path) if not isinstance(video_path, Path) else video_path
if not video_path.is_file():
raise FileNotFoundError(f'{video_path} does not exist!')
if not FFPROBE_BIN:
raise SystemError('ffprobe is not installed!')
cmd = [FFPROBE_BIN, str(video_path)]
# Just read the video using FFMPEG, this prints out useful information about the video.
cmd = (FFPROBE_BIN, str(video_path))
try:
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
logger.warning(f'FFPROBE failed to check for corruption with stdout: {e.stdout.decode()}')
logger.warning(f'FFPROBE failed to check for corruption with stderr: {e.stderr.decode()}')
return True # video is corrupt.
messages = (
b'Invalid NAL unit size',
b'Error splitting the input into NAL units',
b'Invalid data found when processing input',
)
corrupt = False
for error in messages:
@ -347,3 +362,19 @@ def minimize_video(video: dict) -> dict:
video['info_json'] = minimize_video_info_json(video['info_json'])
return video
match_video_paths = partial(match_paths_to_suffixes, suffix_groups=(
tuple(f'.{i}' for i in VIDEO_EXTENSIONS),
('.jpg', '.jpeg', '.webp', '.png'),
('.description',),
('.en.vtt', '.en.srt'),
('.info.json',),
))
@iterify(tuple)
def match_video_files(files: List[File]) -> Tuple[File, File, File, File, File]:
video_paths = match_video_paths([i.path for i in files])
for path in video_paths:
yield next((i for i in files if i.path == path), None)

View file

@ -1,6 +1,7 @@
import json
import pathlib
import shutil
from http import HTTPStatus
from itertools import zip_longest
from typing import List
from uuid import uuid4
@ -13,6 +14,7 @@ from modules.videos.downloader import VideoDownloader, ChannelDownloader
from modules.videos.lib import set_test_channels_config, set_test_downloader_config
from modules.videos.models import Channel, Video
from wrolpi.downloader import DownloadFrequency, DownloadManager, Download
from wrolpi.files.models import File
from wrolpi.vars import PROJECT_DIR
@ -54,9 +56,7 @@ def channel_factory(test_session, test_directory):
@pytest.fixture
def download_channel(test_session, test_directory, video_download_manager) -> Channel:
"""
Get a test Channel that has a download frequency.
"""
"""Get a test Channel that has a download frequency."""
# Add a frequency to the test channel, then give it a download.
channel = Channel(
directory=test_directory,
@ -75,64 +75,75 @@ def simple_video(test_session, test_directory, simple_channel) -> Video:
"""A Video with an empty video file whose channel is the Simple Channel."""
video_path = test_directory / 'simple_video.mp4'
video_path.touch()
video = Video(video_path=video_path, channel_id=simple_channel.id)
video_file = File(path=video_path, model='video')
video = Video(video_file=video_file, channel_id=simple_channel.id)
test_session.add(video)
test_session.commit()
video = test_session.query(Video).filter_by(id=video.id).one()
return video
@pytest.fixture
def video_file(test_directory) -> pathlib.Path:
destination = test_directory / f'{uuid4()}.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
yield destination
destination.unlink()
@pytest.fixture
def video_factory(test_session, test_directory):
"""Creates Videos for testing."""
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file: bool = False,
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False):
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file=None,
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False,
source_id: str = None):
title = title or str(uuid4())
if channel_id:
if with_video_file and isinstance(with_video_file, (pathlib.Path, str)):
# Put the video exactly where specified for the test.
path = pathlib.Path(with_video_file) if not isinstance(with_video_file, pathlib.Path) else with_video_file
elif channel_id:
# Put the video in its Channel's directory.
channel = test_session.query(Channel).filter_by(id=channel_id).one()
path = (channel.directory or test_directory) / f'{title}.mp4'
else:
(test_directory / 'videos/NO CHANNEL').mkdir(parents=True, exist_ok=True)
# Put any video not in a Channel and without a specified file in the NO CHANNEL directory.
(test_directory / 'videos/NO CHANNEL').mkdir(exist_ok=True, parents=True)
path = test_directory / f'videos/NO CHANNEL/{title}.mp4'
video_file = File(path=path, model='video')
test_session.add(video_file)
if with_video_file:
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', path)
else:
path.touch()
info_json_path = None
if with_info_json is True:
# User requests a info json file, but does not provide one.
info_json_path = path.with_suffix('.info.json')
info_json_path.write_text(json.dumps({}))
elif with_info_json:
if with_info_json:
# Use the provided object as the json.
with_info_json = {'duration': 5} if with_info_json is True else with_info_json
info_json_path = path.with_suffix('.info.json')
info_json_path.write_text(json.dumps(with_info_json))
test_session.add(File(path=info_json_path))
poster_path = None
if with_poster_ext:
poster_path = path.with_suffix(f'.{with_poster_ext}')
Image.new('RGB', (25, 25)).save(poster_path)
test_session.add(File(path=poster_path))
caption_path = None
if with_caption_file:
caption_path = path.with_suffix('.en.vtt')
caption_path.touch()
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', caption_path)
test_session.add(File(path=caption_path))
video = Video(video_path=str(path), title=title, channel_id=channel_id, source_id=title,
info_json_path=info_json_path, poster_path=poster_path, caption_path=caption_path,
upload_date=upload_date)
video = Video(
video_path=path,
title=title,
channel_id=channel_id,
source_id=source_id or title,
info_json_path=info_json_path,
poster_path=poster_path,
caption_path=caption_path,
upload_date=upload_date,
)
test_session.add(video)
video_file.do_index()
return video
return factory
@ -199,3 +210,71 @@ def assert_video_ids(test_session):
assert id_ == expected, f'Video ids were not as expected: {expected_video_ids=} != {video_ids=}'
return checker
@pytest.fixture
def video_with_search_factory(test_session, test_directory):
"""A factory that creates a Video record with an associated video File record."""
def video_with_search(path: str = None, title: str = None, b_text: str = None, c_text: str = None,
d_text: str = None, mimetype: str = 'video/mp4'):
path = path or (test_directory / str(uuid4())).absolute()
video_file = File(path=path, a_text=title, d_text=d_text, mimetype=mimetype, model='video')
video = Video(title=title, video_file=video_file)
test_session.add(video)
return video
return video_with_search
@pytest.fixture
def assert_video_search(test_client):
"""A fixture which performs a video search request against the API.
If assert_* params are passed, the response is checked."""
def search_videos(
assert_total: int = None,
assert_ids: List[int] = None,
assert_paths: List[str] = None,
search_str: str = None,
filters: List[str] = None,
offset: int = None,
limit: int = None,
order_by: str = None,
channel_id: int = None,
):
content = dict()
if search_str is not None:
content['search_str'] = search_str
if filters is not None:
content['filters'] = filters
if offset is not None:
content['offset'] = offset
if limit is not None:
content['limit'] = limit
if order_by is not None:
content['order_by'] = order_by
if channel_id is not None:
content['channel_id'] = channel_id
request, response = test_client.post('/api/videos/search', content=json.dumps(content))
assert response.json
assert response.status_code == HTTPStatus.OK
if assert_ids is not None:
assert 'files' in response.json, 'No video files in response'
assert [i['video']['id'] for i in response.json['files']] == assert_ids, 'Video IDs do not match'
if assert_paths is not None:
assert 'files' in response.json, 'No video files in response'
assert [i['video']['video_path'] for i in response.json['files']] == assert_paths, \
'Video paths do not match'
if assert_total:
assert response.json['totals']['files'] == int(assert_total), 'Total video files does not match'
return request, response
return search_videos

View file

@ -13,19 +13,20 @@ from yt_dlp.extractor import YoutubeTabIE # noqa
from yt_dlp.utils import UnsupportedError, DownloadError
from wrolpi.cmd import which
from wrolpi.common import logger, extract_domain, get_media_directory
from wrolpi.common import logger, get_media_directory, escape_file_name
from wrolpi.dates import now
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.db import optional_session
from wrolpi.downloader import Downloader, Download, DownloadResult
from wrolpi.errors import UnknownChannel, ChannelURLEmpty, UnrecoverableDownloadError
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .channel.lib import create_channel, get_channel
from .common import apply_info_json, get_no_channel_directory, get_videos_directory
from .lib import upsert_video, refresh_channel_videos, get_downloader_config
from .lib import get_downloader_config
from .models import Video, Channel
from .schema import ChannelPostRequest
from .video_url_resolver import video_url_resolver
from .video_url_resolver import normalize_youtube_shorts_url
logger = logger.getChild(__name__)
ydl_logger = logger.getChild('youtube-dl')
@ -84,16 +85,16 @@ class ChannelDownloader(Downloader, ABC):
async def do_download(self, download: Download) -> DownloadResult:
"""Update a Channel's catalog, then schedule downloads of every missing video."""
info = extract_info(download.url, process=False)
# Resolve the entries generator.
# Resolve the "entries" generator.
info['entries'] = list(info['entries'])
download.info_json = info
if session := Session.object_session(download):
# May not have a session during testing.
session.commit()
name = info['uploader']
kind = info['uploader']
channel_source_id = info['channel_id']
channel = get_or_create_channel(channel_source_id, download.url, name)
channel = get_or_create_channel(channel_source_id, download.url, kind)
channel.dict() # get all attributes while we have the session.
location = f'/videos/channel/{channel.id}/video' if channel and channel.id else None
@ -103,33 +104,49 @@ class ChannelDownloader(Downloader, ABC):
if is_a_playlist:
downloads = self.get_playlist_downloads(download)
else:
downloads = self.get_channel_downloads(download, info, channel)
downloads = await self.get_channel_downloads(download, info, channel)
return DownloadResult(success=True, location=location, downloads=downloads)
except Exception:
name = 'playlist' if is_a_playlist else 'channel'
logger.warning(f'Failed to update catalog of {name} {download.url}')
if PYTEST:
raise
kind = 'playlist' if is_a_playlist else 'channel'
logger.warning(f'Failed to update catalog of {kind} {download.url}')
return DownloadResult(success=False, location=location, error=str(traceback.format_exc()))
@staticmethod
def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
async def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
"""Get a list of all videos in a Channel, schedule downloads for any missing videos."""
channel_id = channel.id
update_channel_catalog(channel, info)
domain = extract_domain(download.url)
missing_videos = find_all_missing_videos(channel.id)
downloads = []
for _, _, missing_video in missing_videos:
url = missing_video.get('webpage_url') or video_url_resolver(domain, missing_video)
downloads.append(url)
with get_db_session(commit=True) as session:
channel = get_channel(session, channel_id=channel_id, return_dict=False)
if not channel.videos or not channel.refreshed:
logger.warning(f'Refreshing videos in {channel.directory} because {channel} has no video records!')
await channel.refresh_files()
session.commit() # Commit the refresh.
session.refresh(channel)
# Get all videos in the channel.
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
# YouTube Shorts are handled specially.
downloads = list(map(normalize_youtube_shorts_url, downloads))
# Only download those that have not yet been downloaded.
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
downloads = [i for i in downloads if i not in already_downloaded]
return downloads
@staticmethod
def get_playlist_downloads(download: Download) -> List[str]:
"""Get a list of all videos in a playlist, schedule downloads for any missing videos."""
# Get all videos in the playlist
downloads = [i['url'] for i in download.info_json['entries']]
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
# YouTube Shorts are handled specially.
downloads = [normalize_youtube_shorts_url(i) for i in downloads]
# Only download those that have not yet been downloaded.
downloads = [i for i in downloads if not video_downloader.already_downloaded(i)]
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
downloads = [i for i in downloads if i not in already_downloaded]
return downloads
@ -153,9 +170,13 @@ class VideoDownloader(Downloader, ABC):
return f'<VideoDownloader>'
@optional_session
def already_downloaded(self, url: str, session: Session = None) -> bool:
def already_downloaded(self, *urls: str, session: Session = None) -> List:
# We only consider a video record with a video file as "downloaded".
return bool(session.query(Video).filter(Video.url == url, Video.video_path != None, ).count()) # noqa
videos = list(session.query(Video).filter(
Video.url.in_(urls),
Video.video_path != None, # noqa
))
return videos
@classmethod
def valid_url(cls, url) -> Tuple[bool, Optional[dict]]:
@ -200,7 +221,7 @@ class VideoDownloader(Downloader, ABC):
# Use the default directory if this video has no channel.
out_dir = get_no_channel_directory()
if channel:
out_dir = channel.directory.path
out_dir = channel.directory
out_dir.mkdir(exist_ok=True, parents=True)
logs = None # noqa
@ -244,10 +265,14 @@ class VideoDownloader(Downloader, ABC):
)
with get_db_session(commit=True) as session:
# If the video is from a channel, it will already be in the database.
source_id = entry['id']
existing_video = session.query(Video).filter_by(source_id=source_id).one_or_none()
video = upsert_video(session, video_path, channel, id_=existing_video.id if existing_video else None)
video_file = File.upsert(video_path, session)
video: Video = Video.upsert(video_file, session)
video.source_id = entry['id']
if channel:
video.channel_id = channel.id
video.video_file.do_index()
video.validate(session)
session.flush([video])
video_id = video.id
except UnrecoverableDownloadError:
raise
@ -335,7 +360,7 @@ def get_or_create_channel(source_id: str = None, url: str = None, name: str = No
raise ValueError(f'Cannot create channel without a name')
# Channel does not exist. Create one in the video directory.
channel_directory = get_videos_directory() / name
channel_directory = get_videos_directory() / escape_file_name(name)
if not channel_directory.is_dir():
channel_directory.mkdir(parents=True)
data = ChannelPostRequest(
@ -409,7 +434,7 @@ def update_channel_catalog(channel: Channel, info: dict):
# Write the Channel's info to a JSON file.
if channel.directory:
info_json_path = channel.directory.path / f'{channel.name}.info.json'
info_json_path = channel.directory / f'{channel.name}.info.json'
with info_json_path.open('wt') as fh:
json.dump(info, fh, indent=2)
@ -440,7 +465,7 @@ def _find_all_missing_videos(channel_id: id) -> List[Tuple]:
return missing_videos
def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
async def find_all_missing_videos(channel_id: int = None):
"""
Find all videos that don't have a video file, but are found in the DB (taken from the channel's info_json).
@ -453,7 +478,7 @@ def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
# Check that the channel has some videos. We can't be sure what is missing if we don't know what we have.
if channel.refreshed is False:
# Refresh this channel's videos.
refresh_channel_videos(channel)
await channel.refresh_files()
match_regex = re.compile(channel.match_regex) if channel.match_regex else None

View file

@ -1,24 +1,23 @@
import functools
import html
import pathlib
import re
from collections import defaultdict
from typing import Tuple, Optional, List, Union
from uuid import uuid1
from typing import Tuple, Optional, Union
from sqlalchemy import or_
from sqlalchemy.orm import Session
from yt_dlp import YoutubeDL
from wrolpi import before_startup
from wrolpi.common import chunks, ConfigFile, get_media_directory, sanitize_link
from wrolpi.common import ConfigFile, get_media_directory, sanitize_link
from wrolpi.dates import from_timestamp, Seconds
from wrolpi.db import get_db_curs, get_db_session, optional_session
from wrolpi.media_path import MediaPath
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .captions import get_captions
from .common import generate_video_paths, remove_duplicate_video_paths, apply_info_json, get_video_duration, \
is_valid_poster, convert_image, generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
get_no_channel_directory, check_for_video_corruption
from .common import is_valid_poster, convert_image, \
generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
get_no_channel_directory, get_video_duration
from .models import Channel, Video
logger = logger.getChild(__name__)
@ -26,156 +25,66 @@ logger = logger.getChild(__name__)
DEFAULT_DOWNLOAD_FREQUENCY = Seconds.week
def refresh_channel_videos(channel: Channel):
"""
Find all video files in a channel's directory. Add any videos not in the DB to the DB.
"""
# Set the idempotency key, we can remove any videos not touched during this search.
with get_db_curs(commit=True) as curs:
curs.execute('UPDATE video SET idempotency=NULL WHERE channel_id=%s', (channel.id,))
idempotency = str(uuid1())
directory = channel.directory.path
# A set of absolute paths that exist in the file system
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
# Update all videos that match the current video paths
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE channel_id = %s AND video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, channel.id, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
# Get the paths for any video not yet in the DB
# (paths in DB are relative, but we need to pass an absolute path)
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
for video_path in chunk:
video_path = pathlib.Path(video_path)
upsert_video(session, video_path, channel, idempotency=idempotency)
logger.debug(f'{channel.name}: Added {video_path.name}')
with get_db_curs(commit=True) as curs:
stmt = 'DELETE FROM video WHERE channel_id=%s AND idempotency IS NULL AND video_path IS NOT NULL RETURNING id'
curs.execute(stmt, (channel.id,))
deleted_count = len(curs.fetchall())
if deleted_count:
deleted_status = f'Deleted {deleted_count} video records from channel {channel.name}'
logger.info(deleted_status)
logger.info(f'{channel.name}: {len(new_videos)} new videos, {len(existing_paths)} already existed. ')
with get_db_session(commit=True) as session:
channel = session.query(Channel).filter_by(id=channel.id).one()
channel.refreshed = True
apply_info_json(channel.id)
class VideoInfoJSON(object):
channel_source_id = None
title = None
duration = None
view_count = None
url = None
def refresh_no_channel_videos():
"""
Refresh the Videos in the NO CHANNEL directory.
"""
from modules.videos.downloader import get_no_channel_directory
directory = get_no_channel_directory()
if not directory.is_dir():
return
logger.info('Refreshing NO CHANNEL videos')
idempotency = str(uuid1())
# Get all WROLPi compatible videos, remove any duplicates (different formats).
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
for video_path in chunk:
upsert_video(session, pathlib.Path(video_path), idempotency=idempotency)
logger.debug(f'Added NO CHANNEL video {video_path}')
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM video WHERE channel_id IS NULL AND idempotency IS NULL RETURNING id')
deleted_count = len(curs.fetchall())
if deleted_count:
deleted_status = f'Deleted {deleted_count} video records in NO CHANNEL.'
logger.info(deleted_status)
def process_video_info_json(video: Video):
def process_video_info_json(video: Video) -> VideoInfoJSON:
"""
Parse the Video's info json file, return the relevant data.
"""
title = duration = view_count = url = None
video_info_json = VideoInfoJSON()
if info_json := video.get_info_json():
title = info_json.get('fulltitle') or info_json.get('title')
title = html.unescape(title) if title else None
video_info_json.title = html.unescape(title) if title else None
duration = info_json.get('duration')
view_count = info_json.get('view_count')
url = info_json.get('webpage_url') or info_json.get('url')
video_info_json.duration = info_json.get('duration')
video_info_json.view_count = info_json.get('view_count')
video_info_json.url = info_json.get('webpage_url') or info_json.get('url')
video_info_json.channel_source_id = info_json.get('channel_id')
return title, duration, view_count, url
return video_info_json
def validate_videos():
@optional_session
@functools.lru_cache
def get_channel_id_by_source_id(session: Session, source_id: str) -> Optional[int]:
"""Return the id of the Channel which matches the provided `source_id`.
These are cached results."""
channel = session.query(Channel).filter_by(source_id=source_id).one_or_none()
return channel.id if channel else None
def validate_video(video: Video, channel_generate_poster: bool, session: Session):
"""
Validate all Videos not yet validated. A Video is validated when we have attempted to find its: title, duration,
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
found, it will be generated from the video file.
Validate a single video.
This function marks the Video as validated, even if no data can be found so a Video will not be validated multiple
times.
"""
with get_db_curs() as curs:
curs.execute('SELECT id FROM video WHERE video_path IS NOT NULL AND validated IS FALSE')
video_ids = [i['id'] for i in curs.fetchall()]
curs.execute('SELECT id, generate_posters FROM channel')
channel_generate_posters = {i['id']: i['generate_posters'] for i in curs.fetchall()}
A Video is validated when we have attempted to find its: title, duration, view_count, url, caption, size, and
channel.
logger.info(f'Validating {len(video_ids)} videos.')
for chunk in chunks(video_ids, 20):
with get_db_session(commit=True) as session:
videos = session.query(Video).filter(Video.id.in_(chunk)).all()
for video in videos:
video.validate()
def validate_video(video: Video, channel_generate_poster: bool):
"""
Validate a single video. A Video is validated when we have attempted to find its: title, duration,
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
A Video is also valid when it has a JPEG poster, if any. If no poster can be
found, it will be generated from the video file.
"""
if not video.title or not video.duration or not video.view_count or not video.url:
info_json_path = video.info_json_file.path if video.info_json_file else video.info_json_path
if info_json_path and (not video.title or not video.duration or not video.view_count or not video.url):
# These properties can be found in the info json.
title, duration, view_count, url = process_video_info_json(video)
video.title = title
video.duration = duration
video.url = url
video_info_json = process_video_info_json(video)
video.title = video_info_json.title
video.duration = video_info_json.duration
video.url = video_info_json.url
# View count will probably be overwritten by more recent data when this Video's Channel is
# updated.
video.view_count = video.view_count or view_count
video.view_count = video.view_count or video_info_json.view_count
video_path = video.video_path.path if isinstance(video.video_path, MediaPath) else video.video_path
if video_info_json.channel_source_id:
video.channel_id = get_channel_id_by_source_id(source_id=video_info_json.channel_source_id)
# Check for corruption, but don't throw away the video even if it is corrupt. Errors will be logged in this call.
check_for_video_corruption(video_path)
video_path = video.video_path
if not video.title or not video.upload_date or not video.source_id:
# Video is missing things that can be extracted from the video file name.
@ -184,41 +93,41 @@ def validate_video(video: Video, channel_generate_poster: bool):
video.title = video.title or html.unescape(title)
video.upload_date = video.upload_date or upload_date
video.source_id = video.source_id or source_id
if not video.duration:
# Video duration was not in the info json, use ffprobe.
video.duration = get_video_duration(video_path)
if not video.caption and video.caption_path:
video.caption = get_captions(video.caption_path.path)
if not video.size or not video.modification_datetime:
stat = video_path.stat()
video.size = video.size or stat.st_size
video.modification_datetime = video.modification_datetime or from_timestamp(stat.st_mtime)
if not video.poster_path:
# Video poster is not found, lets check near the video file.
for ext in ('.jpg', '.jpeg', '.webp', '.png'):
if (poster_path := video_path.with_suffix(ext)).is_file():
video.poster_path = poster_path
break
if channel_generate_poster:
# Try to convert/generate, but keep the old poster if those fail.
video.poster_path = convert_or_generate_poster(video) or video.poster_path
new_poster_path, duration = convert_or_generate_poster(video) or video.poster_path
if new_poster_path:
# Poster was created/updated.
poster_file = File.upsert(new_poster_path, session)
session.flush([poster_file])
video.poster_file = poster_file
video.poster_path = new_poster_path
if duration:
video.duration = duration
if not video.duration:
# Duration was not retrieved during poster generation.
video.duration = get_video_duration(video_path)
def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
def convert_or_generate_poster(video: Video) -> Tuple[Optional[pathlib.Path], Optional[int]]:
"""
If a Video has a poster, but the poster is invalid, convert it. If a Video has no poster, generate one from the
video file.
Returns None if the poster was not converted, and not generated.
"""
video_path = video.video_path.path
video_path = video.video_path
# Modification/generation of poster is enabled for this channel.
if video.poster_path:
if video.poster_path or video.poster_file:
# Check that the poster is a more universally supported JPEG.
old: pathlib.Path = video.poster_path.path if \
isinstance(video.poster_path, MediaPath) else video.poster_path
old: pathlib.Path = video.poster_path if video.poster_path else video.poster_file.path
new = old.with_suffix('.jpg')
if old != new and new.exists():
@ -231,63 +140,22 @@ def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
try:
convert_image(old, new)
old.unlink(missing_ok=True)
logger.info(f'Converted invalid poster {old} to {new}')
return new
logger.info(f'Converted invalid poster {repr(str(old))} to {repr(str(new))}')
return new, None
except Exception as e:
logger.error(f'Failed to convert invalid poster {old} to {new}', exc_info=e)
return
return None, None
if not video.poster_path:
if not video.poster_path or not video.poster_file:
# Video poster was not discovered, or converted. Let's generate it.
try:
poster_path = generate_video_poster(video_path)
poster_path, duration = generate_video_poster(video_path)
logger.debug(f'Generated poster for {video}')
return poster_path
return poster_path, duration
except Exception as e:
logger.error(f'Failed to generate poster for {video}', exc_info=e)
def refresh_videos(channel_ids: List[int] = None):
"""
Find any videos in the channel directories and add them to the DB. Delete DB records of any videos not in the
file system.
Yields status updates to be passed to the UI.
:return:
"""
logger.info('Refreshing video files')
with get_db_session() as session:
if channel_ids:
channels = session.query(Channel).filter(Channel.id.in_(channel_ids))
else:
channels = session.query(Channel).all()
channels = list(channels)
if not channels and channel_ids:
raise Exception(f'No channels match id(s): {channel_ids}')
elif not channels:
raise Exception(f'No channels in DB. Have you created any?')
for channel in channels:
try:
refresh_channel_videos(channel)
except Exception as e:
logger.fatal(f'Failed to refresh videos for channel {channel.name}!', exc_info=e)
pass
if not channel_ids:
# Refresh NO CHANNEL videos when not refreshing a specific channel.
refresh_no_channel_videos()
# Fill in any missing data for all videos.
if not PYTEST:
import_channels_config()
validate_videos()
logger.info('Refresh of video files complete')
return None, None
class ChannelsConfig(ConfigFile):
@ -470,9 +338,10 @@ def get_channels_config_from_db(session: Session) -> dict:
favorites = defaultdict(lambda: {})
for video in favorite_videos:
if video.channel:
favorites[str(video.channel.directory.relative)][video.video_path.path.name] = dict(favorite=video.favorite)
directory = video.channel.directory.relative_to(get_media_directory())
favorites[str(directory)][video.video_path.name] = dict(favorite=video.favorite)
else:
favorites['NO CHANNEL'][video.video_path.path.name] = dict(favorite=video.favorite)
favorites['NO CHANNEL'][video.video_path.name] = dict(favorite=video.favorite)
favorites = dict(favorites)
return dict(channels=channels, favorites=favorites)
@ -510,7 +379,6 @@ def import_channels_config():
"""Import channel settings to the DB. Existing channels will be updated."""
logger.info('Importing videos config')
try:
from .downloader import ChannelDownloader
config = get_channels_config()
channels, favorites = config.channels, config.favorites
@ -569,7 +437,7 @@ def import_channels_config():
if not channel:
logger.warning(f'Cannot find channel {directory=} for favorites!')
continue
channel_dir = channel.directory.path
channel_dir = channel.directory
else:
channel_dir = get_no_channel_directory()
@ -698,74 +566,3 @@ def parse_video_file_name(video_path: pathlib.Path) -> \
# Return the stem as a last resort
title = pathlib.Path(video_path).stem.strip()
return None, None, None, title
def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel = None, idempotency: str = None,
id_: str = None) -> Video:
"""
Insert a video into the DB. Also, find any meta-files near the video file and store them on the video row.
If id_ is provided, update that entry.
"""
if not video_path.is_absolute():
raise ValueError(f'Video path is not absolute: {video_path}')
# This function can update or insert a Video.
video = None
if id_:
video = session.query(Video).filter_by(id=id_).one()
if not video:
# The video may already be in the database, match the source_id to any existing video.
_, _, source_id, _ = parse_video_file_name(video_path)
if source_id:
video = session.query(Video).filter_by(source_id=source_id).one_or_none()
if not video:
video = Video()
session.add(video)
# Set the file values, all other things can be found using these files.
poster_path, description_path, caption_path, info_json_path = find_meta_files(video_path)
video.caption_path = caption_path
video.description_path = description_path
video.info_json_path = info_json_path
video.poster_path = poster_path
video.video_path = video_path
if channel and not str(video_path).startswith(str(channel.directory.path)):
raise ValueError(f'Video path is not within its channel {video_path=} not in {channel.directory=}')
if channel:
video.channel_id = channel.id
video.idempotency = idempotency
session.flush()
session.refresh(video)
video.validate()
session.flush()
return video
def find_meta_files(path: pathlib.Path) -> Tuple[pathlib.Path, pathlib.Path, pathlib.Path, pathlib.Path]:
"""
Find all files that share a file's full path, except their extensions. It is assumed that file with the
same name, but different extension is related to that file. A None will be yielded if the meta file doesn't exist.
Example:
>>> foo = pathlib.Path('foo.bar')
>>> find_meta_files(foo)
(pathlib.Path('foo.jpg'), pathlib.Path('foo.description'),
pathlib.Path('foo.en.vtt'), pathlib.Path('foo.info.json'))
"""
suffix = path.suffix
name, suffix, _ = str(path.name).rpartition(suffix)
meta_file_exts = (('.jpg', '.webp', '.png'), ('.description',), ('.en.vtt', '.en.srt'), ('.info.json',))
for meta_exts in meta_file_exts:
for meta_ext in meta_exts:
meta_path = path.with_suffix(meta_ext)
if meta_path.exists():
yield meta_path
break
else:
yield None

View file

@ -1,18 +1,21 @@
import json
import pathlib
from datetime import datetime
from pathlib import Path
from typing import Optional, Generator
from typing import Optional, Generator, Dict, List
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey, Computed
from sqlalchemy.orm import relationship, Session, deferred
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import Base, tsvector, ModelHelper, logger, get_media_directory
from wrolpi.common import Base, ModelHelper, logger, get_media_directory
from wrolpi.dates import now, TZDateTime
from wrolpi.db import get_db_curs
from wrolpi.downloader import Download, download_manager
from wrolpi.errors import UnknownVideo, UnknownFile, UnknownDirectory
from wrolpi.media_path import MediaPathType, MediaPath
from wrolpi.errors import UnknownVideo
from wrolpi.files.lib import refresh_directory_files_recursively, glob_shared_stem
from wrolpi.files.models import File
from wrolpi.media_path import MediaPathType
logger = logger.getChild(__name__)
@ -22,18 +25,8 @@ class Video(ModelHelper, Base):
id = Column(Integer, primary_key=True)
channel_id = Column(Integer, ForeignKey('channel.id'))
channel = relationship('Channel', primaryjoin='Video.channel_id==Channel.id', back_populates='videos')
idempotency = Column(String)
validated = Column(Boolean, default=False)
# File paths
caption_path = Column(MediaPathType)
description_path = Column(MediaPathType)
ext = Column(String)
info_json_path = Column(MediaPathType)
poster_path = Column(MediaPathType)
video_path = Column(MediaPathType)
caption = deferred(Column(String)) # slow to fetch
censored = Column(Boolean, default=False)
duration = Column(Integer)
favorite = Column(TZDateTime)
@ -46,18 +39,42 @@ class Video(ModelHelper, Base):
view_count = Column(Integer)
viewed = Column(TZDateTime)
textsearch = deferred(
Column(tsvector, Computed('''to_tsvector('english'::regconfig,
((COALESCE(title, ''::text) || ' '::text) ||
COALESCE(caption, ''::text)))''')))
# Associated Files
caption_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
caption_file: File = relationship('File', primaryjoin='Video.caption_path==File.path')
info_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
info_json_file: File = relationship('File', primaryjoin='Video.info_json_path==File.path')
poster_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
poster_file: File = relationship('File', primaryjoin='Video.poster_path==File.path')
video_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
video_file: File = relationship('File', primaryjoin='Video.video_path==File.path')
def __repr__(self):
video_path = self.video_path
if self.video_path and isinstance(self.video_path, MediaPath):
video_path = self.video_path.path.relative_to(get_media_directory())
return f'<Video id={self.id} title={repr(self.title)} path={video_path} channel={self.channel_id} ' \
v = self.video_file
if self.video_path:
video_path = self.video_path.path if hasattr(self.video_path, 'path') else self.video_path
v = video_path.relative_to(get_media_directory())
return f'<Video id={self.id} title={repr(self.title)} path={v} channel={self.channel_id} ' \
f'source_id={repr(self.source_id)}>'
@classmethod
def upsert(cls, video_file: File, session: Session):
video = session.query(Video).filter_by(video_file=video_file).one_or_none()
if not video:
video = Video(video_file=video_file)
video: Video
video.video_path = video.video_path or video_file.path
video.video_file = video.video_file or video_file
video.size = video.video_path.stat().st_size
session.add(video)
video.find_my_files()
# Claim the video file for this model.
video.video_file.model = cls.__tablename__
return video
def dict(self) -> dict:
d = super().dict()
if self.channel_id:
@ -72,35 +89,34 @@ class Video(ModelHelper, Base):
from .common import minimize_video
return minimize_video(self.dict())
def _clear_paths(self):
self.caption_path = None
self.description_path = None
self.ext = None
self.info_json_path = None
self.poster_path = None
self.video_path = None
def _clear_files(self):
# Use setattr to avoid confusing PyCharm
setattr(self, 'caption_file', None)
setattr(self, 'description_file', None)
setattr(self, 'ext', None)
setattr(self, 'info_json_file', None)
setattr(self, 'poster_file', None)
setattr(self, 'video_file', None)
def my_paths(self) -> Generator[Path, None, None]:
def my_files(self) -> Generator[Path, None, None]:
"""Return all paths of this Video. Returns nothing if all paths are None.""" # noqa
if self.description_path: # noqa
yield self.description_path.path
if self.poster_path:
yield self.poster_path.path
if self.caption_path:
yield self.caption_path.path
if self.info_json_path:
yield self.info_json_path.path
if self.video_path:
yield self.video_path.path
if self.poster_file:
yield self.poster_file.path
if self.caption_file:
yield self.caption_file.path
if self.info_json_file:
yield self.info_json_file.path
if self.video_file:
yield self.video_file.path
def delete(self):
"""
Remove all files related to this video. Add it to it's Channel's skip list.
"""
for path in self.my_paths():
for path in self.my_files():
path.unlink(missing_ok=True)
self._clear_paths()
self._clear_files()
needs_save = bool(self.favorite)
@ -126,21 +142,15 @@ class Video(ModelHelper, Base):
def set_viewed(self):
self.viewed = now()
def get_info_json(self) -> Optional[JSON]:
def get_info_json(self) -> Optional[Dict]:
"""If this Video has an info_json file, return it's contents. Otherwise, return None."""
if not self.info_json_path:
info_json_path = self.info_json_file.path if self.info_json_file else self.info_json_path
if not info_json_path:
return
try:
info_json_path = self.info_json_path.path if \
isinstance(self.info_json_path, MediaPath) else self.info_json_path
with open(info_json_path, 'rb') as fh:
contents = json.load(fh)
return contents
except UnknownFile:
pass
except UnknownDirectory:
pass
with info_json_path.open('rb') as fh:
return json.load(fh)
except Exception as e:
logger.warning(f'Unable to parse info json {self.info_json_path}', exc_info=e)
return None
@ -230,7 +240,7 @@ class Video(ModelHelper, Base):
def __json__(self):
from modules.videos.common import minimize_video_info_json
info_json = minimize_video_info_json(self.get_info_json()) if self.info_json_path else None
stem = self.video_path.path.stem if self.video_path else None
stem = self.video_path.stem if self.video_path else None
d = dict(
caption_path=self.caption_path,
@ -249,44 +259,78 @@ class Video(ModelHelper, Base):
upload_date=self.upload_date,
url=self.url,
validated=self.validated,
video_path=self.video_path,
view_count=self.view_count,
video_path=self.video_path.relative_to(get_media_directory()),
viewed=self.viewed,
)
return d
def validate(self):
def validate(self, session: Session):
"""Perform a validation of this video and it's files. Mark this video as validated if no errors occur."""
if not self.video_path:
if not self.video_file:
# Can't validate if there is no video file.
return False
from .lib import validate_video
try:
validate_video(self, self.channel.generate_posters if self.channel else False)
validate_video(self, self.channel.generate_posters if self.channel else False, session)
self.validated = True
except Exception as e:
logger.warning(f'Failed to validate video {self}', exc_info=e)
return self.validated
def check_for_corruption(self):
"""Uses ffprobe to check if this video is corrupt."""
def find_my_files(self):
"""Search near the video file for it's associated files. (poster, caption, etc.)"""
from modules.videos.common import match_video_files
if not self.video_path:
return
raise ValueError("Can't find files when I don't have a video path!")
from .lib import check_for_video_corruption
return check_for_video_corruption(self.video_path.path)
# Associated files share the same stem, but different extensions.
associated_paths = glob_shared_stem(self.video_path)
associated_paths.pop(associated_paths.index(self.video_path))
# Get the session from the video first, fallback to the video file.
session: Session = Session.object_session(self) or Session.object_session(self.video_file)
files = [File.upsert(i, session) for i in associated_paths]
_, poster_file, description_file, caption_file, info_json_file = match_video_files(files)
if poster_file:
poster_file.do_index()
if caption_file:
caption_file.do_index()
if info_json_file:
info_json_file.do_index()
session.flush([i for i in files])
self.poster_file = poster_file
self.caption_file = caption_file
self.info_json_file = info_json_file
@staticmethod
def find_by_path(path, session):
video = session.query(Video).filter_by(video_path=path).one_or_none()
return video
@staticmethod
def find_by_paths(paths: List[pathlib.Path], session) -> List:
videos = list(session.query(Video).filter(Video.video_path.in_(paths)))
return videos
@property
def primary_path(self):
return self.video_path
class Channel(ModelHelper, Base):
__tablename__ = 'channel'
id = Column(Integer, primary_key=True)
name = Column(String)
idempotency = Column(String)
url = Column(String, unique=True)
match_regex = Column(String)
directory: MediaPath = Column(MediaPathType)
directory: pathlib.Path = Column(MediaPathType)
skip_download_videos = Column(ARRAY(String))
generate_posters = Column(Boolean, default=False) # generating posters may delete files, and can be slow.
calculate_duration = Column(Boolean, default=True)
@ -337,6 +381,12 @@ class Channel(ModelHelper, Base):
for key, value in data.items():
setattr(self, key, value)
# We need an absolute directory.
if isinstance(self.directory, pathlib.Path) and not self.directory.is_absolute():
self.directory = get_media_directory() / self.directory
elif isinstance(self.directory, str) and not pathlib.Path(self.directory).is_absolute():
self.directory = get_media_directory() / self.directory
# All channels with a URL and download_frequency should have a download.
session = Session.object_session(self)
if download and not self.download_frequency:
@ -360,7 +410,7 @@ class Channel(ModelHelper, Base):
"""
config = dict(
calculate_duration=self.calculate_duration,
directory=str(self.directory.path),
directory=str(self.directory),
download_frequency=self.download_frequency,
favorites={},
generate_posters=self.generate_posters,
@ -400,6 +450,7 @@ class Channel(ModelHelper, Base):
def dict(self, with_statistics: bool = False):
d = super(Channel, self).dict()
d['directory'] = self.directory.relative_to(get_media_directory())
if with_statistics:
d['statistics'] = self.get_statistics()
return d
@ -425,3 +476,8 @@ class Channel(ModelHelper, Base):
largest_video=largest_video,
)
return statistics
async def refresh_files(self):
"""Refresh all files within this Channel's directory. Mark this channel as refreshed."""
await refresh_directory_files_recursively(self.directory)
self.refreshed = True

View file

@ -131,7 +131,9 @@ class ExtendedVideo(Video):
@dataclass
class VideoResponse:
video = ExtendedVideo
file = ExtendedVideo
prev = ExtendedVideo
next = ExtendedVideo
@dataclass

View file

@ -1,9 +1,12 @@
import json
import shutil
from http import HTTPStatus
from pathlib import Path
from modules.videos.models import Video
from modules.videos.models import Video, Channel
from modules.videos.video import lib as video_lib
from wrolpi.downloader import DownloadFrequency
from wrolpi.files.models import File
from wrolpi.vars import PROJECT_DIR
@ -11,18 +14,17 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
subdir = test_directory / 'subdir'
subdir.mkdir()
# video1 is in a subdirectory.
video1 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
# video1 is in a subdirectory, move its files into the subdirectory.
video1 = video_factory(channel_id=simple_channel.id, with_video_file=subdir / 'video1.mp4',
with_info_json=True,
with_poster_ext='jpg')
test_session.commit()
shutil.move(video1.video_path.path, subdir / video1.video_path.path.name)
shutil.move(video1.poster_path.path, subdir / video1.poster_path.path.name)
video1.video_path = subdir / video1.video_path.path.name
test_session.delete(video1.poster_file)
video1.poster_path = None
# video2 is in the test directory.
video2 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
with_poster_ext='jpg')
video2.poster_path = video1.poster_path = None
video2.poster_path = video2.poster_file = video1.poster_path = None
test_session.commit()
assert not video1.size, 'video1 should not have size during creation'
@ -36,27 +38,73 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
orphan_poster.touch()
# Create a bogus video in the channel.
bogus = Video(video_path='foo', channel_id=simple_channel.id)
video_file = File(path=test_directory / 'foo')
video_file.idempotency = None # File was not added in the current refresh.
bogus = Video(video_file=video_file, channel_id=simple_channel.id)
test_session.add(bogus)
test_session.commit()
test_client.post('/api/videos/refresh')
test_client.post('/api/files/refresh')
# Posters were found during refresh.
assert video1.poster_path
assert 'subdir' in str(video1.poster_path.path)
assert 'subdir' in str(video1.poster_path)
assert video2.poster_path
# Missing video3 was found
video3: Video = test_session.query(Video).filter_by(id=4).one()
assert video3.video_path.path == vid3
assert video3.video_path != video1.video_path
assert video3.video_path != video2.video_path
assert video3.video_path == vid3
assert len(set(test_session.query(Video.video_path))) == 3, 'Should have 3 distinct video files'
# Bogus video was removed.
assert not any('foo' in str(i.video_path.path) for i in test_session.query(Video).all())
assert not any('foo' in str(i.video_path) for i in test_session.query(Video).all())
# Orphan file was not deleted.
assert orphan_poster.is_file(), 'Orphan poster was removed!'
assert video1.size, 'video1 size was not found'
# Remove video1's poster, video1 should be updated.
video1_video_path = str(video1.video_path)
video1.poster_path.unlink()
test_client.post('/api/files/refresh')
video1 = test_session.query(Video).filter_by(video_path=video1_video_path).one()
assert not video1.poster_path
def test_channels_with_videos(test_session, test_client, test_directory, channel_factory, video_factory):
channel1 = channel_factory('channel1', name='channel1')
channel2 = channel_factory('channel2', name='channel2')
vid1 = video_factory(channel_id=channel1.id, with_video_file=True)
vid2 = video_factory(channel_id=channel2.id, with_video_file=True)
vid3 = video_factory(with_video_file=True)
test_session.commit()
vid1_path = vid1.video_path
vid2_path = vid2.video_path
vid3_path = vid3.video_path
test_session.delete(vid1)
test_session.delete(vid2)
test_session.delete(vid3)
test_session.commit()
assert test_session.query(Video).count() == 0, 'Videos were not deleted'
assert test_session.query(Channel).count() == 2, 'Channels were deleted!'
assert vid1_path.is_file(), 'Video file was deleted'
assert vid2_path.is_file(), 'Video file was deleted'
assert vid3_path.is_file(), 'Video file was deleted'
test_client.post('/api/files/refresh')
assert test_session.query(Video).count() == 3, 'Did not find correct amount of video files.'
assert {i[0] for i in test_session.query(Channel.name)} == {'channel1', 'channel2'}, 'Channels were changed.'
vid1: Video = test_session.query(Video).filter_by(video_path=vid1_path).one()
vid2: Video = test_session.query(Video).filter_by(video_path=vid2_path).one()
vid3: Video = test_session.query(Video).filter_by(video_path=vid3_path).one()
assert vid1.channel == channel1
assert vid2.channel == channel2
assert vid3.channel is None
def test_api_download_channel(test_session, test_client, simple_channel):
"""A Channel download (a catalog update) can be triggered via the API."""
@ -70,3 +118,75 @@ def test_api_download_channel(test_session, test_client, simple_channel):
test_session.commit()
request, response = test_client.post(f'/api/videos/download/{simple_channel.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
def test_search_videos_file(test_client, test_session, test_directory, video_with_search_factory):
"""Test that videos can be searched and that their order is by their textsearch rank."""
# These captions have repeated letters, so they will be higher in the ranking.
videos = [
('1', 'b b b b e d d'),
('2', '2 b b b d'),
('3', 'b b'),
('4', 'b e e'),
('5', ''),
]
for title, caption in videos:
video_with_search_factory(title=title, d_text=caption)
test_session.commit()
def assert_search(search_str, expected, limit=20, total=None):
d = json.dumps({'search_str': search_str, 'limit': limit})
_, resp = test_client.post('/api/videos/search', content=d)
assert resp.status_code == HTTPStatus.OK
response_ids = [i['video']['id'] for i in resp.json['files']]
assert response_ids == expected
assert resp.json['totals']['files'] == (total or len(expected))
# Repeated runs should return the same result
for _ in range(2):
# Only videos with a b are returned, ordered by the amount of b's
assert_search('b', [1, 2, 3, 4])
# Only two captions have e
assert_search('e', [4, 1])
# Only two captions have d
assert_search('d', [1, 2])
# 5 can be gotten by it's title
assert_search('5', [5])
# only video 1 has e and d
assert_search('e d', [1])
# video 1 and 4 have b and e, but 1 has more
assert_search('b e', [1, 4])
# Check totals are correct even with a limit
assert_search('b', [1, 2], limit=2, total=4)
def test_search_videos(test_client, test_session, video_factory, assert_video_search, simple_channel):
"""Search the Video table. This does not need to use a join with the File table."""
vid1 = video_factory(upload_date='2022-09-16', with_video_file=True)
vid2 = video_factory(upload_date='2022-09-17', with_video_file=True, channel_id=simple_channel.id)
vid3 = video_factory(upload_date='2022-09-18', with_video_file=True)
vid1.set_favorite(True)
test_session.commit()
assert test_session.query(Video).count() == 3
assert test_session.query(File).count() == 3
assert_video_search(assert_total=3, assert_ids=[vid1.id, vid2.id, vid3.id], order_by='upload_date')
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id, vid1.id], order_by='-upload_date')
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id], order_by='-upload_date', limit=2)
assert_video_search(assert_total=3, assert_ids=[vid2.id, vid1.id], order_by='-upload_date', limit=2, offset=1)
assert_video_search(assert_total=1, assert_ids=[vid2.id], order_by='-upload_date', channel_id=simple_channel.id)
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'])
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'], limit=1)
# No results, no total is returned from the DB.
assert_video_search(assert_total=0, assert_ids=[], filters=['favorite'], offset=1)
# Check all order_by.
for order_by in video_lib.VIDEO_ORDERS:
assert_video_search(order_by=order_by)

View file

@ -1,104 +0,0 @@
import shutil
from pathlib import Path
from modules.videos import captions
from modules.videos.models import Video
from wrolpi.db import get_db_curs, get_db_context
from wrolpi.test.common import wrap_test_db, TestAPI
from wrolpi.vars import PROJECT_DIR
class TestCaption(TestAPI):
def setUp(self) -> None:
super().setUp()
self.vtt_path1 = Path(self.tmp_dir.name) / 'example1.en.vtt'
self.vtt_path1.touch()
self.vtt_path1.write_text((PROJECT_DIR / 'test/example1.en.vtt').read_text())
self.vtt_path2 = Path(self.tmp_dir.name) / 'example2.en.vtt'
self.vtt_path2.touch()
self.vtt_path2.write_text((PROJECT_DIR / 'test/example2.en.vtt').read_text())
def test_example_en_vtt(self):
expected = ['okay welcome to this session this is',
'okay welcome to this session this is',
'okay welcome to this session this is\ncalled the kinetic bunny need to meet',
'called the kinetic bunny need to meet',
'called the kinetic bunny need to meet\nthe virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is\njana car who just to give a little bit',
'jana car who just to give a little bit',
'jana car who just to give a little bit\nof background information we come from a',
'of background information we come from a',
'of background information we come from a\ncompany called le code from finland and']
self.assertEqual(list(captions.get_caption_text(self.vtt_path1)), expected)
expected = ['okay welcome to this session this is',
'called the kinetic bunny need to meet',
'the virtual bonnie emilio stop this is',
'jana car who just to give a little bit',
'of background information we come from a',
'company called le code from finland and']
self.assertEqual(list(captions.get_unique_caption_lines(self.vtt_path1)), expected)
@wrap_test_db
def test_process_captions(self):
_, session = get_db_context()
video1 = Video(title='scream', caption_path=str(self.vtt_path1))
session.add(video1)
session.commit()
video1.caption = captions.get_captions(video1.caption_path.path)
video2 = Video(title='bar', caption_path=str(self.vtt_path2))
session.add(video2)
session.commit()
video2.caption = captions.get_captions(video2.caption_path.path)
session.commit()
session.flush()
session.refresh(video1)
session.refresh(video2)
# Get the video from the DB
video1 = session.query(Video).filter_by(id=video1.id).one()
self.assertIsNotNone(video1.caption)
video2 = session.query(Video).filter_by(id=video2.id).one()
self.assertIsNotNone(video2.caption)
# Search using the tsvector, "sessions" never actually appears in the text, but "session" does
with get_db_curs() as curs:
def select_textsearch(*args):
curs.execute('SELECT id FROM video WHERE textsearch @@ to_tsquery(%s) ORDER BY id', args)
select_textsearch('sessions')
self.assertEqual(curs.fetchall(), [[1, ]])
# Matches video1.title and video2.caption
select_textsearch('scream')
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
# Matches video1.title and video2.caption
select_textsearch('scream | sessions')
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
# Only matches video1.title
select_textsearch('scream & sessions')
self.assertEqual(curs.fetchall(), [[1, ]])
# Matches neither
select_textsearch('scream & sess')
self.assertEqual(curs.fetchall(), [])
# Matches video2.caption
select_textsearch('yawn | sess')
self.assertEqual(curs.fetchall(), [[2, ]])
# Matches video2.caption
select_textsearch('yawn')
self.assertEqual(curs.fetchall(), [[2, ]])
# Matches video2.title
select_textsearch('bar')
self.assertEqual(curs.fetchall(), [[2, ]])
def test_bad_caption(test_directory):
"""A caption file that cannot be parsed is ignored."""
bad_caption_file = PROJECT_DIR / 'test/bad_caption.en.vtt'
test_file = test_directory / 'bad_caption.en.vtt'
shutil.copy(bad_caption_file, test_file)
result = captions.get_captions(test_file)
assert result is None

View file

@ -244,7 +244,7 @@ def test_update_censored_videos(test_session, video_factory, simple_channel):
check_censored([(vid1.id, True), (vid2.id, True), (vid3.id, True), (vid4.id, False)])
def test_import_favorites(test_session, simple_channel, video_factory, test_channels_config):
def test_import_favorites(test_session, test_directory, simple_channel, video_factory, test_channels_config):
"""
A favorited Video will be preserved through everything (channel deletion, DB wipe) and can be imported and will be
favorited again.
@ -254,14 +254,13 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
video_factory(channel_id=simple_channel.id) # never favorited
vid2 = video_factory(channel_id=None) # has no channel
vid3 = video_factory(channel_id=simple_channel.id)
favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
vid3.favorite = favorite
favorite = vid3.favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
test_session.commit()
vid2_video_path = vid2.video_path.path
vid2_video_path = vid2.video_path
favorites = {
str(simple_channel.directory.relative): {vid3.video_path.path.name: {'favorite': favorite}},
'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}},
str(simple_channel.directory.relative_to(test_directory)): {vid3.video_path.name: {'favorite': favorite}},
'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}},
}
# Save config, verify that favorite is set.
@ -299,7 +298,7 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
# Deleting the Video in the model really removes the favorite status.
vid3.delete()
config = get_channels_config()
assert config.favorites == {'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}}}
assert config.favorites == {'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}}}
import_and_verify([vid2.id])
@ -369,11 +368,11 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
vid4 = video_factory(title='vid4')
test_session.commit()
channel1_name, channel_1_directory = channel1.name, channel1.directory.path
channel2_name, channel_2_directory = channel2.name, channel2.directory.path
channel1_name, channel_1_directory = channel1.name, channel1.directory
channel2_name, channel_2_directory = channel2.name, channel2.directory
assert channel1.directory.path.exists() and channel1.directory.path.is_absolute()
assert channel2.directory.path.exists() and channel2.directory.path.is_absolute()
assert channel1.directory.exists() and channel1.directory.is_absolute()
assert channel2.directory.exists() and channel2.directory.is_absolute()
# Channel's used to have a relative directory in the DB.
test_session.execute('UPDATE channel SET directory=:directory WHERE id=2',
@ -390,9 +389,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
channel['download_frequency'] = DownloadFrequency.biweekly
# Change the favorites to the old "link" method as well.
channels_config.favorites = {
sanitize_link(channel1.name): {str(vid1.video_path.path): {'favorite': now()}},
sanitize_link(channel2.name): {str(vid2.video_path.path): {'favorite': now()}},
'NO CHANNEL': {str(vid3.video_path.path): {'favorite': now()}},
sanitize_link(channel1.name): {str(vid1.video_path): {'favorite': now()}},
sanitize_link(channel2.name): {str(vid2.video_path): {'favorite': now()}},
'NO CHANNEL': {str(vid3.video_path): {'favorite': now()}},
}
channels_config.save()
@ -407,9 +406,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
# No new channels were created. Existing channels were updated.
assert test_session.query(Channel).count() == 2
channel1, channel2 = test_session.query(Channel).order_by(Channel.id)
assert channel1.name == channel1_name and str(channel1.directory.path) == str(channel_1_directory)
assert channel1.name == channel1_name and str(channel1.directory) == str(channel_1_directory)
# Channel2's directory is now absolute.
assert channel2.name == channel2_name and str(channel2.directory.path) == str(channel_2_directory)
assert channel2.name == channel2_name and str(channel2.directory) == str(channel_2_directory)
assert channel1.download_frequency == DownloadFrequency.biweekly
assert channel2.download_frequency == DownloadFrequency.biweekly
# Videos were favorited by their channel "link".

View file

@ -14,12 +14,14 @@ from modules.videos.models import Channel, Video
from wrolpi.db import get_db_context
from wrolpi.downloader import DownloadManager, Download, DownloadResult
from wrolpi.errors import InvalidDownload
from wrolpi.files.models import File
from wrolpi.test.common import TestAPI
from wrolpi.test.test_downloader import HTTPDownloader
from wrolpi.vars import PROJECT_DIR
def test_find_all_missing_videos(test_session, channel_factory, video_factory):
@pytest.mark.asyncio
async def test_find_all_missing_videos(test_session, channel_factory, video_factory):
channel1 = channel_factory(url='some url')
channel1.info_json = {'entries': [{'id': 'foo', 'view_count': 0}]}
test_session.commit()
@ -29,14 +31,14 @@ def test_find_all_missing_videos(test_session, channel_factory, video_factory):
video_factory()
# No missing videos
assert [] == list(find_all_missing_videos(channel1.id))
assert [] == [i async for i in find_all_missing_videos(channel1.id)]
# Create a video that has no video file.
video = Video(title='needs to be downloaded', channel_id=channel1.id, source_id='foo')
test_session.add(video)
test_session.commit()
missing = list(find_all_missing_videos(channel1.id))
missing = [i async for i in find_all_missing_videos(channel1.id)]
assert len(missing) == 1
# The video returned is the one we faked.
@ -175,7 +177,7 @@ async def test_video_download_no_channel(test_session, video_download_manager, v
mock_video_process_runner.assert_called_once()
video: Video = test_session.query(Video).one()
assert str(video.video_path.path) == f'{channel_dir}/video.mp4'
assert str(video.video_path) == f'{channel_dir}/video.mp4'
@pytest.mark.asyncio
@ -364,19 +366,23 @@ async def test_invalid_download_url(test_session, test_download_manager, mock_vi
@pytest.mark.asyncio
async def test_video_download_1(test_session, test_directory, simple_channel, video_download_manager,
mock_video_process_runner):
mock_video_process_runner, image_file):
"""A video download is performed, files are associated."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_file = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_file)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
# Create a poster file which was downloaded.
poster_path = video_path.with_suffix('.png')
image_file.rename(poster_path)
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename:
mock_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_file, {'id': 'foo'})
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(url)
await video_download_manager.wait_for_all_downloads()
@ -389,7 +395,10 @@ async def test_video_download_1(test_session, test_directory, simple_channel, vi
assert test_session.query(Channel).one()
video: Video = test_session.query(Video).one()
assert video.video_path.path.is_absolute()
assert video.video_path.is_absolute(), 'Video path is not absolute'
assert video.poster_path == poster_path, 'Video poster was not discovered'
assert video.validated, 'Video was not validated'
assert video.video_file and video.video_file.indexed, 'Video was not indexed'
@pytest.mark.asyncio
@ -428,9 +437,9 @@ example_playlist_json = {
'ie_key': 'Youtube',
'title': 'video 2 title',
'uploader': None,
'url': 'https://youtube.com/watch?v=video_2_url',
'url': 'https://www.youtube.com/shorts/video_2_url',
'view_count': 1413,
'webpage_url': 'https://youtube.com/watch?v=video_2_url'},
'webpage_url': 'https://www.youtube.com/shorts/video_2_url'},
{'_type': 'url',
'description': None,
'duration': None,
@ -438,9 +447,9 @@ example_playlist_json = {
'ie_key': 'Youtube',
'title': 'video 1 title',
'uploader': None,
'url': 'https://youtube.com/watch?v=video_1_url',
'url': 'https://www.youtube.com/watch?v=video_1_url',
'view_count': 58504,
'webpage_url': 'https://youtube.com/watch?v=video_1_url'},
'webpage_url': 'https://www.youtube.com/watch?v=video_1_url'},
{'_type': 'url',
'description': None,
'duration': None,
@ -473,15 +482,14 @@ example_playlist_json = {
@pytest.mark.asyncio
async def test_download_playlist(test_session, test_directory, mock_video_extract_info, video_download_manager):
"""
All videos in a playlist can be downloaded for it's Channel.
"""
"""All videos in a playlist can be downloaded for it's Channel."""
download = Download(url='playlist url')
test_session.add(download)
channel = get_or_create_channel(example_playlist_json['channel_id'], download.url, example_channel_json['uploader'])
test_session.add(Video(url='https://youtube.com/watch?v=video_1_url',
source_id='video 1 id', channel_id=channel.id, video_path='video exists',
poster_path='poster exists'))
video_file = File(path=test_directory / 'video file.mp4')
test_session.add(video_file)
test_session.add(Video(url='https://www.youtube.com/watch?v=video_1_url',
source_id='video 1 id', channel_id=channel.id, video_file=video_file))
test_session.commit()
mock_video_extract_info.return_value = example_playlist_json # Playlist info is fetched first.
@ -491,6 +499,10 @@ async def test_download_playlist(test_session, test_directory, mock_video_extrac
result = await channel_downloader.do_download(download)
assert result.success is True, 'Download was not successful'
assert set(result.downloads) == {
'https://youtube.com/watch?v=video_2_url',
'https://www.youtube.com/watch?v=video_2_url', # Shorts is converted to regular URL.
'https://youtube.com/watch?v=video_3_url',
}
def test_channel_download_match_title(test_session, test_directory, simple_channel, video_download_manager):
pass

View file

@ -1,25 +1,20 @@
import json
import pathlib
import shutil
from datetime import datetime
from unittest import mock
import pytest
import sqlalchemy
from modules.videos import lib
from modules.videos.common import apply_info_json
from modules.videos.lib import validate_videos, parse_video_file_name, upsert_video
from modules.videos.lib import parse_video_file_name, validate_video, get_statistics
from modules.videos.models import Video
from modules.videos.video.lib import video_search
from wrolpi.dates import local_timezone
from wrolpi.vars import PROJECT_DIR
from wrolpi.files import lib as files_lib
from wrolpi.files.models import File
def test_search_censored_videos(test_session, simple_channel):
def test_search_censored_videos(test_session, simple_channel, video_factory):
for i in map(str, range(50)):
test_session.add(Video(source_id=i, channel=simple_channel, video_path='foo'))
vid = Video(source_id='51', video_path='bar') # this should never be modified because it has no channel
video_factory(channel_id=simple_channel.id, source_id=i)
vid = video_factory(source_id=51) # this should never be modified because it has no channel
test_session.add(vid)
test_session.commit()
@ -32,135 +27,23 @@ def test_search_censored_videos(test_session, simple_channel):
# All source_ids are in the entries.
set_entries(map(str, range(50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == []
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == []
assert total == 0
# First 5 are censored.
set_entries(map(str, range(5, 50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == [str(i) for i in range(5)]
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(5)]
assert total == 5
# First 25 are censored.
set_entries(map(str, range(25, 50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == [str(i) for i in range(20)]
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(20)]
assert total == 25
def test_validate_videos(test_session, simple_channel, video_factory):
"""
Videos that aren't validated should have their data filled in while being validated.
"""
simple_channel.generate_posters = True
vid1_json = {'title': 'info&apos;s title'}
vid3_json = {'duration': 100}
vid5_json = {'view_count': 42}
vid6_json = {'webpage_url': 'https://example.com/webpage', 'url': 'https://example.com/url'}
vid1 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid1_json)
vid2 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
vid3 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid3_json)
vid4 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
vid5 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid5_json)
vid6 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid6_json)
vid7 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg')
vid8 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='webp')
vid9 = video_factory(simple_channel.id, with_video_file=True)
# These videos are missing this data, and a refresh should fill them in.
vid1.title = None
vid2.caption = None
vid2.title = None
vid3.duration = None
vid4.size = None
vid5.view_count = None
vid6.url = None
vid7.poster_path = None
test_session.commit()
# Write the associated files. This data should be processed.
vid2.caption_path = vid2.video_path.path.with_suffix('.en.vtt')
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', vid2.caption_path)
test_session.commit()
# All Videos are validated, all data is filled out. The info json data is trusted first.
validate_videos()
assert all([i.validated for i in test_session.query(Video)])
assert vid1.title == "info's title"
assert vid2.caption.startswith('okay welcome')
assert 'mp4' not in vid2.title
assert vid3.duration == 100
assert vid4.size == 1055736
assert vid5.view_count == 42
assert vid6.url == 'https://example.com/webpage'
assert str(vid7.poster_path).endswith('.jpg')
assert str(vid8.poster_path).endswith('.jpg')
assert str(vid9.poster_path).endswith('.jpg') # this was generated from the video file.
vid1.title = None
vid3.duration = None
vid5.view_count = None
vid6.url = None
vid9.poster_path = None
test_session.commit()
# Validation does not happen because all are still validated.
validate_videos()
# All videos need to be validated again.
for i in test_session.query(Video):
i.validated = False
# Use backup methods to fetch data.
vid1.info_json_path = None
vid3.info_json_path = None
vid5.info_json_path = None
vid6.info_json_path.path.write_text(json.dumps({'url': 'https://example.com/url'}))
test_session.commit()
validate_videos()
assert all([i.validated for i in test_session.query(Video)])
assert vid1.title != "info's title" # file name is used.
assert vid3.duration == 5 # video file duration is used.
assert vid5.view_count is None
assert vid6.url == 'https://example.com/url' # url is used as backup url.
# Posters are both JPEG format. Webp file is removed.
assert vid7.poster_path.path.is_file()
assert vid8.poster_path.path.is_file()
assert not vid8.poster_path.path.with_suffix('.webp').is_file()
assert str(vid9.poster_path).endswith('.jpg') # the generated file was rediscovered.
def test_validate_video_exception(test_session, simple_channel, video_factory):
"""
Test that even if a Video cannot be validated, the other Videos will still be validated.
"""
vid1 = video_factory(simple_channel.id, with_video_file=True)
vid2 = video_factory(simple_channel.id, with_video_file=True)
# The validator should not even check this Video because it does not have a video file.
vid3 = video_factory(simple_channel.id)
vid3.video_path = None
assert not all([i.validated for i in test_session.query(Video)])
test_session.commit()
with mock.patch('modules.videos.lib.process_video_info_json') as mock_process_video_info_json:
mock_process_video_info_json.side_effect = [
Exception('skip this video'),
(None, None, None, None),
('should not be set to video 3', None, None, None), # This should not be called!
]
validate_videos()
# Only first two videos are validated.
assert mock_process_video_info_json.call_count == 2, mock_process_video_info_json.call_count
test_session.commit()
assert not vid1.validated
assert vid2.validated
assert not vid3.validated
@pytest.mark.parametrize('file_name,expected', [
('channel_20000101_12345678910_ some title.mp4', ('channel', '20000101', '12345678910', 'some title')),
('channel name_NA_12345678910_ some title.mp4', ('channel name', None, '12345678910', 'some title')),
@ -186,67 +69,120 @@ def test_parse_video_file_name(file_name, expected):
assert parse_video_file_name(video_path) == expected, f'{file_name} != {expected}'
def test_validate_does_not_generate_when_disabled(test_session, channel_factory, video_factory):
@pytest.mark.asyncio
async def test_associated_files(test_session, test_directory, video_file, image_file, vtt_file1):
"""A Video file has files related to it. They should be attached to the Video record."""
caption_file = video_file.with_suffix('.en.vtt')
vtt_file1.rename(caption_file)
poster_file = video_file.with_suffix('.jpeg')
image_file.rename(poster_file)
await files_lib._refresh_all_files()
test_session.commit()
# All files were found.
file_paths = {i.path for i in test_session.query(File)}
assert len(file_paths) == 3
video: Video = test_session.query(Video).one()
video_id = video.id
assert video.video_file == video_file
assert video.poster_file == poster_file
assert video.caption_file == caption_file
assert video.video_file.modification_datetime
assert video.video_file.size
assert video.video_file.indexed
await files_lib._refresh_all_files()
test_session.commit()
# No new files were added.
assert {i.path for i in test_session.query(File)} == file_paths
# Same video exists.
video: Video = test_session.query(Video).one()
assert video.id == video_id, 'Video id changed. Was the video recreated?'
@pytest.mark.asyncio
async def test_unassociated_video(test_session, test_directory, video_file):
"""A Video file without associated files is still a Video."""
# Move the video file into a new subdirectory. The video file should still be found.
sub_dir = test_directory / 'subdir'
sub_dir.mkdir()
new_video_file_path = sub_dir / video_file.name
video_file.rename(new_video_file_path)
video_file = new_video_file_path
await files_lib._refresh_all_files()
test_session.commit()
video: Video = test_session.query(Video).one()
assert video.video_file == video_file
@pytest.mark.asyncio
async def test_video_factory(test_session, video_factory, channel_factory):
"""
A poster should not be generated if the Video's Channel forbids it.
The `video_factory` pytest fixture is used in many video tests. Test all it's functionality.
"""
channel1 = channel_factory() # posters should not be generated by default.
channel = channel_factory()
video = video_factory(
channel_id=channel.id,
with_video_file=True,
with_info_json={'description': 'hello'},
with_caption_file=True,
with_poster_ext='png',
source_id='some id',
)
test_session.commit()
assert video.video_path and video.video_file and video.video_path.is_file()
assert video.caption_path and video.caption_file and video.caption_path.is_file()
assert video.poster_path and video.poster_file and video.poster_path.is_file()
assert video.info_json_path and video.info_json_file and video.info_json_path.is_file()
assert video.video_file.a_text # title
assert video.video_file.b_text is None
assert video.video_file.c_text and 'hello' in video.video_file.c_text # description
assert video.video_file.d_text # captions
assert video.source_id == 'some id'
def test_validate_video(test_session, test_directory, video_factory):
"""A video poster will be generated only if the channel permits."""
vid1 = video_factory(with_video_file=True)
assert not vid1.poster_path
validate_video(vid1, True, test_session)
assert vid1.poster_path, 'Poster was not created'
assert vid1.poster_path.is_file(), 'Poster path does not exist'
assert vid1.video_path.stem == vid1.poster_path.stem
assert vid1.poster_path.suffix == '.jpg'
# A PNG is replaced.
vid2 = video_factory(with_video_file=True, with_poster_ext='.png')
vid2.poster_path.with_suffix('.jpg').touch() # New poster exists, replace it.
assert vid2.poster_path and vid2.poster_path.is_file() and vid2.poster_path.suffix == '.png', \
'Poster was not initialized'
validate_video(vid2, True, test_session)
assert vid2.poster_path.is_file(), 'New poster was not generated'
assert vid2.poster_path.suffix == '.jpg'
@pytest.mark.asyncio
async def test_get_statistics(test_session, video_factory, channel_factory):
# Can get statistics in empty DB.
await get_statistics()
channel1 = channel_factory()
channel2 = channel_factory()
channel2.generate_posters = True
vid1 = video_factory(channel1.id, with_video_file=True)
vid2 = video_factory(channel2.id, with_video_file=True)
# webp will not be converted unless generate_posters is enabled.
vid3 = video_factory(channel1.id, with_video_file=True, with_poster_ext='webp')
test_session.commit()
assert not vid1.poster_path
assert not vid2.poster_path
assert str(vid3.poster_path).endswith('webp')
video_factory(channel_id=channel1.id)
video_factory(channel_id=channel1.id)
video_factory(channel_id=channel2.id)
video_factory()
validate_videos()
assert not vid1.poster_path
assert vid2.poster_path
assert str(vid3.poster_path).endswith('webp')
def test_video_modification_datetime(test_session, simple_channel, video_file):
"""
A Video has a modification_datetime which represents the video file's mtime.
"""
expected = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
with mock.patch('modules.videos.lib.from_timestamp') as mock_from_timestamp:
mock_from_timestamp.return_value = expected
video = upsert_video(test_session, video_file, simple_channel)
test_session.commit()
assert video.modification_datetime == expected
def test_refresh_channel_videos(test_session, channel_factory, video_factory, video_file, assert_video_ids):
assert_video_ids([])
# Simulate a channel's catalog being updated.
channel1 = channel_factory(name='channel1')
test_session.add(Video(source_id='source1', channel_id=channel1.id))
test_session.add(Video(source_id='source2', channel_id=channel1.id))
# Add some real files to be found.
destination = channel1.directory / f'channel1_20220510_source1_the title.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
destination = channel1.directory / f'channel1_20220511_source2_the title.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
lib.refresh_channel_videos(channel1)
assert_video_ids([1, 2])
# These video files were deleted.
test_session.add(Video(source_id='this should be removed', channel_id=channel1.id, video_path='foo'))
test_session.add(Video(source_id='this should also be removed', video_path='bar'))
test_session.commit()
assert_video_ids([1, 2, 3, 4])
lib.refresh_channel_videos(channel1)
assert_video_ids([1, 2, 4])
lib.refresh_videos()
assert_video_ids([1, 2])
result = await get_statistics()
assert 'statistics' in result
assert 'videos' in result['statistics']
assert 'channels' in result['statistics']
assert 'historical' in result['statistics']

View file

@ -25,7 +25,7 @@ logger = logger.getChild(__name__)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
def video_get(_: Request, video_id: int):
video, previous_video, next_video = lib.get_video_for_app(video_id)
return json_response({'video': video, 'prev': previous_video, 'next': next_video})
return json_response({'file': video, 'prev': previous_video, 'next': next_video})
@video_bp.post('/search')
@ -40,7 +40,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
if body.order_by not in lib.VIDEO_ORDERS:
raise InvalidOrderBy('Invalid order by')
videos, videos_total = lib.video_search(
files, videos_total = lib.video_search(
body.search_str,
body.offset,
body.limit,
@ -49,7 +49,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
body.filters,
)
ret = {'videos': list(videos), 'totals': {'videos': videos_total}}
ret = {'files': list(files), 'totals': {'files': videos_total}}
return json_response(ret)

View file

@ -1,3 +1,4 @@
import pathlib
from datetime import datetime
from typing import Tuple, Optional, List
@ -8,6 +9,9 @@ from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import run_after, logger
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
from wrolpi.errors import UnknownVideo
from wrolpi.files.lib import assign_file_prefetched_models, handle_search_results
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from ..lib import save_channels_config
from ..models import Video
@ -31,49 +35,49 @@ def get_video_for_app(video_id: int) -> Tuple[dict, Optional[dict], Optional[dic
video.set_viewed()
previous_video, next_video = video.get_surrounding_videos()
caption = video.caption
video = video.__json__()
video['caption'] = caption
previous_video = previous_video.__json__() if previous_video else None
next_video = next_video.__json__() if next_video else None
caption = video.video_file.d_text
video = video.video_file.__json__()
video['video']['caption'] = caption
previous_video = previous_video.video_file.__json__() if previous_video else None
next_video = next_video.video_file.__json__() if next_video else None
return video, previous_video, next_video
VIDEO_ORDERS = {
'upload_date': 'upload_date ASC, LOWER(video_path) ASC',
'-upload_date': 'upload_date DESC NULLS LAST, LOWER(video_path) DESC',
'rank': '2 DESC, LOWER(video_path) DESC',
'-rank': '2 ASC, LOWER(video_path) ASC',
'id': 'id ASC',
'-id': 'id DESC',
'size': 'size ASC, LOWER(video_path) ASC',
'-size': 'size DESC, LOWER(video_path) DESC',
'duration': 'duration ASC, LOWER(video_path) ASC',
'-duration': 'duration DESC, LOWER(video_path) DESC',
'favorite': 'favorite ASC, LOWER(video_path) ASC',
'-favorite': 'favorite DESC, LOWER(video_path) DESC',
'viewed': 'viewed ASC',
'-viewed': 'viewed DESC',
'view_count': 'view_count ASC',
'-view_count': 'view_count DESC',
'modification_datetime': 'modification_datetime ASC',
'-modification_datetime': 'modification_datetime DESC',
'upload_date': 'v.upload_date ASC, LOWER(v.video_path) ASC',
'-upload_date': 'v.upload_date DESC NULLS LAST, LOWER(v.video_path) DESC',
'rank': '2 DESC, LOWER(v.video_path) DESC',
'-rank': '2 ASC, LOWER(v.video_path) ASC',
'id': 'v.id ASC',
'-id': 'v.id DESC',
'size': 'f.size ASC, LOWER(v.video_path) ASC',
'-size': 'f.size DESC, LOWER(v.video_path) DESC',
'duration': 'duration ASC, LOWER(v.video_path) ASC',
'-duration': 'duration DESC, LOWER(v.video_path) DESC',
'favorite': 'favorite ASC, LOWER(v.video_path) ASC',
'-favorite': 'favorite DESC, LOWER(v.video_path) DESC',
'viewed': 'v.viewed ASC',
'-viewed': 'v.viewed DESC',
'view_count': 'v.view_count ASC',
'-view_count': 'v.view_count DESC',
'modification_datetime': 'f.modification_datetime ASC',
'-modification_datetime': 'f.modification_datetime DESC',
}
NO_NULL_ORDERS = {
'viewed': '\nAND viewed IS NOT NULL',
'-viewed': '\nAND viewed IS NOT NULL',
'duration': '\nAND duration IS NOT NULL',
'-duration': '\nAND duration IS NOT NULL',
'size': '\nAND size IS NOT NULL',
'-size': '\nAND size IS NOT NULL',
'view_count': '\nAND view_count IS NOT NULL',
'-view_count': '\nAND view_count IS NOT NULL',
'modification_datetime': '\nAND modification_datetime IS NOT NULL',
'-modification_datetime': '\nAND modification_datetime IS NOT NULL',
'viewed': 'v.viewed IS NOT NULL',
'-viewed': 'v.viewed IS NOT NULL',
'duration': 'v.duration IS NOT NULL',
'-duration': 'v.duration IS NOT NULL',
'size': 'f.size IS NOT NULL',
'-size': 'f.size IS NOT NULL',
'view_count': 'v.view_count IS NOT NULL',
'-view_count': 'v.view_count IS NOT NULL',
'modification_datetime': 'f.modification_datetime IS NOT NULL',
'-modification_datetime': 'f.modification_datetime IS NOT NULL',
}
DEFAULT_VIDEO_ORDER = 'rank'
VIDEO_QUERY_LIMIT = 20
VIDEO_QUERY_LIMIT = 24
def video_search(
@ -84,71 +88,61 @@ def video_search(
order_by: str = None,
filters: List[str] = None,
) -> Tuple[List[dict], int]:
with get_db_curs() as curs:
params = dict(search_str=search_str, offset=offset)
channel_where = ''
if channel_id:
channel_where = 'AND channel_id = %(channel_id)s'
params['channel_id'] = channel_id
wheres = ['v.video_path IS NOT NULL']
# Filter for/against favorites, if it was provided
favorites_where = ''
if isinstance(filters, list) and 'favorite' in filters:
favorites_where = 'AND favorite IS NOT NULL'
params = dict(search_str=search_str, offset=offset)
if channel_id:
wheres.append('v.channel_id = %(channel_id)s')
params['channel_id'] = channel_id
censored_where = ''
if isinstance(filters, list) and 'censored' in filters:
censored_where = 'AND censored = true'
# Filter for/against favorites, if it was provided
if isinstance(filters, list) and 'favorite' in filters:
wheres.append('v.favorite IS NOT NULL')
where = ''
if search_str:
# A search_str was provided by the user, modify the query to filter by it.
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
where = 'AND textsearch @@ websearch_to_tsquery(%(search_str)s)'
params['search_str'] = search_str
else:
# No search_str provided. Get id and total only.
columns = 'id, COUNT(*) OVER() AS total'
if isinstance(filters, list) and 'censored' in filters:
wheres.append('v.censored = true')
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
# whitelist.
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
if order_by:
try:
order = VIDEO_ORDERS[order_by]
except KeyError:
raise
if order_by in NO_NULL_ORDERS:
where += NO_NULL_ORDERS[order_by]
if search_str:
# A search_str was provided by the user, modify the query to filter by it.
select_columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), ' \
'COUNT(*) OVER() AS total'
wheres.append('f.textsearch @@ websearch_to_tsquery(%(search_str)s)')
params['search_str'] = search_str
join = 'LEFT JOIN file f on f.path = v.video_path'
else:
# No search_str provided. Get path and total only.
select_columns = 'v.video_path AS path, COUNT(*) OVER() AS total'
join = ''
query = f'''
SELECT
{columns}
FROM video
WHERE
video_path IS NOT NULL
{where}
{channel_where}
{favorites_where}
{censored_where}
ORDER BY {order}
OFFSET %(offset)s LIMIT {int(limit)}
'''.strip()
logger.debug(query)
if order_by in {'size', '-size', 'modification_datetime', '-modification_datetime'}:
# Size and modification_datetime are from the video file.
join = 'LEFT JOIN file f on f.path = v.video_path'
curs.execute(query, params)
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
# whitelist.
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
if order_by:
try:
results = [dict(i) for i in curs.fetchall()]
except psycopg2.ProgrammingError:
# No videos
return [], 0
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
order = VIDEO_ORDERS[order_by]
except KeyError:
raise
if order_by in NO_NULL_ORDERS:
wheres.append(NO_NULL_ORDERS[order_by])
with get_db_session() as session:
results = get_ranked_models(ranked_ids, Video, session=session)
results = [i.__json__() for i in results]
wheres = '\n AND '.join(wheres)
where = f'WHERE\n{wheres}' if wheres else ''
stmt = f'''
SELECT
{select_columns}
FROM video v
{join}
{where}
ORDER BY {order}
OFFSET %(offset)s LIMIT {int(limit)}
'''.strip()
logger.debug(stmt, params)
results, total = handle_search_results(stmt, params)
return results, total

View file

@ -1,7 +1,6 @@
from datetime import timedelta
from http import HTTPStatus
from json import dumps
from unittest import mock
from modules.videos.models import Video
from modules.videos.video.lib import get_video_for_app
@ -71,7 +70,7 @@ def test_get_video_prev_next(test_session, channel_factory, video_factory):
def test_get_video_for_app(test_session, simple_channel, simple_video):
vid, prev, next_ = get_video_for_app(simple_video.id)
assert vid['id'] == simple_video.id
assert vid['video']['id'] == simple_video.id
def test_video_delete(test_session, test_directory, channel_factory, video_factory):
@ -81,8 +80,8 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
vid2 = video_factory(channel_id=channel2.id, with_video_file=True, with_info_json=True)
test_session.commit()
vid1_video_path, vid1_caption_path = vid1.video_path.path, vid1.caption_path.path
vid2_video_path, vid2_info_json_path = vid2.video_path.path, vid2.info_json_path.path
vid1_video_path, vid1_caption_path = vid1.video_path, vid1.caption_path
vid2_video_path, vid2_info_json_path = vid2.video_path, vid2.info_json_path
# No videos have been deleted yet.
assert vid1_video_path.is_file() and vid1_caption_path.is_file()
@ -106,36 +105,42 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
assert vid2_video_path.is_file() is False and vid2_info_json_path.is_file() is False
def test_wrol_mode(test_directory, simple_channel, simple_video):
def test_wrol_mode(test_directory, simple_channel, simple_video, wrol_mode_fixture, test_download_manager):
"""Many methods are blocked when WROL Mode is enabled."""
channel = dumps(dict(name=simple_channel.name, directory=str(simple_channel.directory)))
favorite = dumps(dict(video_id=simple_video.id, favorite=True))
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
# Can't create, update, or delete a channel.
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
wrol_mode_fixture(True)
# Can't delete a video
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't create, update, or delete a channel.
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't refresh or download
_, resp = api_app.test_client.post('/api/videos/refresh')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't delete a video
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# THE REST OF THESE METHODS ARE ALLOWED
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
assert resp.status_code == HTTPStatus.OK
# Can't refresh or download
_, resp = api_app.test_client.post('/api/files/refresh')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# THE REST OF THESE METHODS ARE ALLOWED
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
assert resp.status_code == HTTPStatus.OK
assert test_download_manager.stopped.is_set()
wrol_mode_fixture(False)
assert not test_download_manager.stopped.is_set()

View file

@ -1,3 +1,9 @@
import re
from wrolpi.common import logger
logger = logger.getChild(__name__)
formatters = {
'youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
'www.youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
@ -22,3 +28,16 @@ def video_url_resolver(domain: str, entry: dict) -> str:
except KeyError:
# Unable to resolve, try the URL as is.
return entry['url']
YOUTUBE_SHORT_REGEX = re.compile(r'https?:\/\/w{3}?\.?youtube.com\/shorts\/(.{5,15})')
def normalize_youtube_shorts_url(url: str) -> str:
"""Convert a YouTube "shorts" URL to a regular video URL.
This is to avoid downloading shorts twice."""
if match := YOUTUBE_SHORT_REGEX.match(url.strip()):
source_id = match.group(1)
return f'https://www.youtube.com/watch?v={source_id}'
return url