wrolpi/modules/videos/test/test_common.py
lrnselfreliance 46796f783a
Arbitrary videos can be downloaded. (#63)
* Squashed commit of the following:

commit c527475fcb4d4906457be5c10f0e9c6dee7fedc1
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:16:36 2021 -0700

    Only saving a favorite video to the config if it has a source_id.

commit ae8b712ebeaef467b8d5a5b87ca0f4f276373a0d
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:14:26 2021 -0700

    Ading description and changing Save button to Download.

commit 03a72be6dfdfa383915950a80e3f1ee9febf6705
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:09:44 2021 -0700

    Removing uploader for now.

commit 5d0231284bf84e459687f6ab903b06bf4e3762bd
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:07:17 2021 -0700

    Removing unique data from example info.

commit 9cc721a32d4715d9d6433e51db3b35e039c78d68
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:02:33 2021 -0700

    Updating downloader tests.

commit 8e193b03349ae4613f105418cd3e8176c7101304
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:53:31 2021 -0700

    Toasting error when WROL Mode is enabled.  Disabling new downloads during WROL Mode.

commit 48c4311c31db8f2d2c8dd0a65f979a5f0a49a911
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:45:06 2021 -0700

    Raising error if download exits without a 0.  Clearing kill event when starting a new download.

commit 54ba3a890d59facc837410d34bf7afb40db3b685
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:16:46 2021 -0700

    Downloads can be restarted and killed.

commit bd60751262247259c3e9ceb5116e017d6314c219
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Dec 13 16:28:18 2021 -0700

    Downloads can be started again.  Downloads are sorted on Downloads page.

commit ea7f39231b92982ae0b263b76c5bd1902cc8169d
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 21:24:16 2021 -0700

    Building Downloads tab.

commit d3321872ac3743ff6df781b07ef0fb537f1d4c98
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 20:41:46 2021 -0700

    Not scheduling channels withtout a URL/frequency.

commit 84f268a0d4f58c85f86b6377900492e0766f6619
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 19:56:08 2021 -0700

    Channel downloads will be spread out when each channel is created.

commit 26fa6beea38ef3f8d6b7bf218f18222a51a87a60
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 13:08:11 2021 -0700

    Recurring downloads are renewed on startup.

commit fc2ab850533f1c3ad073f4692c7dea2be43e85d6
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 17:03:57 2021 -0700

    Fetching download again after do_download.

commit fdadad327fc09c17a49eb63b11e632fb72cc3195
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 16:20:51 2021 -0700

    Fixing error with test.  Mock was not being reset.

commit 070dee238e53377acd22097c63cf3794e68ed423
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 15:36:39 2021 -0700

    Fixed bug in date handling (big surprise).

commit e53a8992268d68620c679455adaec3a590f20602
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 16:48:36 2021 -0700

    Using chunks for regular channel refresh.

commit 3bd60c58d1bef0226325f321978c0434b36b3be5
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:22:02 2021 -0700

    Reformatting.

commit 4fdc325f25a32c311ca6543fe321624b2bd564b2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:13:34 2021 -0700

    Creating function that refreshes NO CHANNEL videos.

commit d67feafb6c2994f5a4a1295920a70db369cc6816
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:35:54 2021 -0700

    Removing unused functions.

commit 6b3918e4e78fbac64920f85c4de0df873ec366a1
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:31:48 2021 -0700

    Removing reporter and events.  This will be replaced in the future with some sort of SSE.

commit 54fa1b0f5e93ca25c80b780dcdc3876b700273b1
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:55:43 2021 -0700

    Better error handling for archive button.

commit c31d124e511b27f626ed2df71486d47302b396f3
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:29:43 2021 -0700

    Displaying View Count in a human readable format.

commit 0ee2bd65588fd93b475cad6144e76f1b45191b39
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 09:29:41 2021 -0700

    Fixing tests.

commit e64aff32bab95c311ea3747e1a21b58b99750d5e
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 08:59:51 2021 -0700

    Handling when a Video has no Channel in UI.

commit b9549ecd5aac03c002adca7b7667f0a7189b662a
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 26 08:35:36 2021 -0700

    Downloaders can raise UnrecoverableDownloadError to signal a downlaod should not be retried.

commit da88798e2584d3e84900a70e34166c8c208c64fb
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 20:40:29 2021 -0700

    Downloaders can refuse a download, which will then be deleted.

commit b865d856121db5ec8a3660c5d3c13b9846fde688
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:35:43 2021 -0700

    Removing relative path handling.

commit 6d7757870b02ba030d78b3ef7e43f28b906eea61
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:28:27 2021 -0700

    Using __json__ for Video and Channel models.

commit 649e4d6a61608272759d1e7ee02b490bb5c20b3e
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 18:23:38 2021 -0700

    Fixing video links in UI.

commit 9d081af12742070b8519bd752812130cb5dbe393
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:42:38 2021 -0700

    Fixing docs.

commit 1202cbe0f9ae744d7801b53a5071cded8ea39924
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:40:36 2021 -0700

    Fixing wrolpi tests.

commit 3f6af8cc1123f2f63a3fc0f21ec4d994c642cf13
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:13:35 2021 -0700

    Fixing tests to use MediaPath.

commit faec04f2a5e4c267a502c03884940a9c5bcfe387
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 19 10:21:20 2021 -0700

    Testing when a video has no channel.

commit cbf22aad7a0b72e61685e0236e843554b57c4af1
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 16:24:03 2021 -0700

    Using MediaPath for Video and Channel models.  Testing VideoDownloader.do_download.

commit 7916ed61e43439e6cd60f23ce3fdf6c1a75bf02d
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 12:43:44 2021 -0700

    Channels/Videos downloaders can get gotten.

commit 808ef796cec01db6e7009065f9e26d25d9500ce2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 21:45:19 2021 -0700

    Upgrading to Sanic 21.9.1.

commit ea32813be28e78393ac456ee782a7245c098188f
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 20:29:01 2021 -0700

    Creating Downloader and DownloaderManager classes which handle downloading URLs.

commit e859a77a77277ddf8b6e221ec9c9383c8ab91d66
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 12:41:39 2021 -0700

    Creating UI segement which allows the download/upload of media.

* Displaying all recurring downloads.

* Not downloading channels with empty url.

* Showing positive row for recurring downloads.

* Removing default last_successful_download.

* Only displaying Last Successful Download when there is one.

* Refreshing videos before finding missing videos.

* Using short video path name for log.

* Removing download history feature in favor of new Downloads page.

* Only showing info date if it exists.

* Making get_or_create_channel more reliable.

* Allowing null for channel download frequency.

* Logging when searching for channel.

* Storing channel's source_id in config.

* Importing using link instead of name.

* Getting channel id when importing channels.

* Removing redunant parentheses from repr.

* Preserving database channel values if there are not values in the config.

* Only highlighting pending downloads.

* Deleting old downloads periodically.

* Sorting recurring by next_download/frequency.

* Creating fixtures for channels, media directory and db session.

* Removing unused function.  Wrapping config for tests.

* Deleting channel download when deleting a channel.

* Testing that a channel's download is deleted when the channel is deleted.

* Updating youtube-dl

* Switching from youtube-dl in favor of yt-dlp.

* Sorting once-downlaods by last_successful_download then id.

* Adding special handling to video_url_resolver for full urls.

* Consolidating downloader methods and functions.  Reversing once-download sort order so the most recently downloaded are at the top.

* Getting session if one is not provided.

* Adding test for create_downloads.

* Not passing format option when getting filename.

* Linting

Signed-off-by: Roland <roland@learningselfreliance.com>
2021-12-20 16:09:11 -07:00

284 lines
11 KiB
Python

import pathlib
import subprocess
import tempfile
from unittest import mock
from unittest.mock import Mock, MagicMock
import pytest
from PIL import Image
from modules.videos.models import Channel, Video
from modules.videos.test.common import create_channel_structure
from wrolpi.common import get_absolute_media_path
from wrolpi.db import get_db_session
from wrolpi.test.common import build_test_directories, wrap_test_db, TestAPI
from wrolpi.vars import PROJECT_DIR
from ..common import get_matching_directories, convert_image, bulk_validate_posters, remove_duplicate_video_paths, \
update_view_count, get_video_duration
class TestCommon(TestAPI):
def test_get_absolute_media_path(self):
path = get_absolute_media_path('videos')
self.assertTrue(str(path).endswith('videos'))
def test_matching_directories(self):
structure = [
'foo/qux/',
'Bar/',
'baz/baz'
'barr',
'bazz',
]
with build_test_directories(structure) as temp_dir:
temp_dir = pathlib.Path(temp_dir)
# No directories have c
matches = get_matching_directories(temp_dir / 'c')
assert matches == []
# Get all directories starting with f
matches = get_matching_directories(temp_dir / 'f')
assert matches == [str(temp_dir / 'foo')]
# Get all directories starting with b, ignore case
matches = get_matching_directories(temp_dir / 'b')
assert matches == [str(temp_dir / 'Bar'), str(temp_dir / 'baz')]
# baz matches, but it has no subdirectories
matches = get_matching_directories(temp_dir / 'baz')
assert matches == [str(temp_dir / 'baz')]
# foo is an exact match, return subdirectories
matches = get_matching_directories(temp_dir / 'foo')
assert matches == [str(temp_dir / 'foo/qux')]
def test_get_video_duration(self):
"""
Video duration can be retrieved from the video file.
"""
video_path = PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4'
self.assertEqual(get_video_duration(video_path), 5)
video_path = PROJECT_DIR / 'test/does not exist.mp4'
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
video_path = str(PROJECT_DIR / 'test/does not exist.mp4')
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
with tempfile.NamedTemporaryFile() as fh:
self.assertRaises(subprocess.CalledProcessError, get_video_duration, fh.name)
@pytest.mark.parametrize(
'paths,expected',
(
(['1.mp4', ], ['1.mp4', ]),
(['1.mp4', '2.mp4'], ['1.mp4', '2.mp4']),
(['1.mp4', '1.ogg'], ['1.mp4']),
(['1.bad_ext', '1.other_ext'], ['1.bad_ext']),
)
)
def test_remove_duplicate_video_paths(paths, expected):
result = remove_duplicate_video_paths(map(pathlib.Path, paths))
assert sorted(result) == sorted(map(pathlib.Path, expected))
def test_convert_image():
"""
An image's format can be changed. This tests that convert_image() converts from a WEBP format, to a JPEG format.
"""
foo = Image.new('RGB', (25, 25), color='grey')
with tempfile.TemporaryDirectory() as tempdir:
tempdir = pathlib.Path(tempdir)
# Save the new image to "foo.webp".
existing_path = tempdir / 'foo.webp'
foo.save(existing_path)
assert Image.open(existing_path).format == 'WEBP'
destination_path = tempdir / 'foo.jpg'
assert not destination_path.is_file()
# Convert the WEBP to a JPEG. The WEBP image should be removed.
convert_image(existing_path, destination_path)
assert not existing_path.is_file()
assert destination_path.is_file()
assert Image.open(destination_path).format == 'JPEG'
@pytest.mark.parametrize(
'_structure,paths',
(
(
{'channel1': ['vid1.mp4']},
[
'channel1/vid1.mp4',
],
),
(
{'channel1': ['vid1.mp4'], 'channel2': ['vid1.mp4']},
[
'channel1/vid1.mp4',
'channel2/vid1.mp4',
],
),
(
{'channel1': ['vid1.mp4'], 'channel2': ['vid1.mp4', 'vid2.mp4', 'vid2.en.vtt']},
[
'channel1/vid1.mp4',
'channel2/vid1.mp4',
'channel2/vid2.mp4',
'channel2/vid2.en.vtt',
],
),
)
)
def test_create_db_structure(_structure, paths):
@create_channel_structure(_structure)
def test_func(_, tempdir):
assert isinstance(tempdir, pathlib.Path)
for path in paths:
path = (tempdir / path)
assert path.exists()
assert path.is_file()
with get_db_session() as session:
for channel_name in _structure:
channel = session.query(Channel).filter_by(name=channel_name).one()
assert (tempdir / channel_name).is_dir()
assert channel
assert channel.directory == tempdir / channel_name
assert len(channel.videos) == len([i for i in _structure[channel_name] if i.endswith('mp4')])
with tempfile.TemporaryDirectory() as tmp_dir, \
mock.patch('wrolpi.media_path.get_media_directory') as mock_get_directory:
mock_get_directory.return_value = pathlib.Path(tmp_dir)
test_ = MagicMock()
test_.tmp_dir.name = tmp_dir
test_func(test_)
@wrap_test_db
@create_channel_structure(
{
'channel1': ['vid1.mp4', 'vid1.jpg'],
'channel2': ['vid2.flv', 'vid2.webp'],
}
)
def test_bulk_replace_invalid_posters(tempdir: pathlib.Path):
"""
Test that when a video has an invalid poster format, we convert it to JPEG.
"""
channel1, channel2 = sorted(tempdir.iterdir())
jpg, mp4 = sorted(channel1.iterdir())
flv, webp = sorted(channel2.iterdir())
Image.new('RGB', (25, 25)).save(jpg)
Image.new('RGB', (25, 25)).save(webp)
with open(jpg, 'rb') as jpg_fh, open(webp, 'rb') as webp_fh:
# Files are different formats.
jpg_fh_contents = jpg_fh.read()
webp_fh_contents = webp_fh.read()
assert jpg_fh_contents != webp_fh_contents
assert Image.open(jpg_fh).format == 'JPEG'
assert Image.open(webp_fh).format == 'WEBP'
with get_db_session() as session:
vid1 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel1/vid1.jpg').one()
assert vid1.validated_poster is False
vid2 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel2/vid2.webp').one()
assert vid2.validated_poster is False
# Convert the WEBP image. convert_image() should only be called once.
mocked_convert_image = Mock(wraps=convert_image)
with mock.patch('modules.videos.common.convert_image', mocked_convert_image):
video_ids = [vid1.id, vid2.id]
bulk_validate_posters(video_ids)
mocked_convert_image.assert_called_once_with(webp, tempdir / 'channel2/vid2.jpg')
with get_db_session() as session:
# Get the video by ID because it's poster is now a JPEG.
vid2 = session.query(Video).filter_by(id=vid2.id).one()
assert str(vid2.poster_path.path).split('/')[-1] == 'vid2.jpg'
assert all('webp' not in str(i.poster_path.path) for i in session.query(Video).all())
assert vid2.validated_poster is True
# Vid1's image was validated, but not converted.
vid1 = session.query(Video).filter_by(id=vid1.id).one()
assert str(vid1.poster_path.path).split('/')[-1] == 'vid1.jpg'
assert vid1.validated_poster is True
# Old webp was removed
assert not webp.is_file()
new_jpg = tempdir / 'channel2/vid2.jpg'
assert new_jpg.is_file()
# chmod 644
assert new_jpg.stat().st_mode == 0o100644
with open(new_jpg, 'rb') as new_jpg_fh:
# The converted image is the same as the other JPEG because both are black 25x25 pixel images.
assert jpg_fh_contents == new_jpg_fh.read()
assert Image.open(new_jpg_fh).format == 'JPEG'
# Calling convert again has no effect.
mocked_convert_image.reset_mock()
with mock.patch('modules.videos.common.convert_image', mocked_convert_image):
video_ids = [vid1.id, vid2.id]
bulk_validate_posters(video_ids)
mocked_convert_image.assert_not_called()
@wrap_test_db
@create_channel_structure(
{
'channel1': ['vid1.mp4', 'vid1.jpg'],
'channel2': ['vid2.mp4'],
'channel3': ['vid3.mp4', 'vid4.mp4'],
}
)
def test_update_view_count(tempdir: pathlib.Path):
def check_view_counts(view_counts):
with get_db_session() as session_:
for source_id, view_count in view_counts.items():
vid = session_.query(Video).filter_by(source_id=source_id).one()
assert vid.view_count == view_count
with get_db_session(commit=True) as session:
channel1, channel2, channel3 = session.query(Channel).order_by(Channel.id).all()
channel1.info_json = {'entries': [{'id': 'vid1.mp4', 'view_count': 10}]}
channel2.info_json = {'entries': [{'id': 'vid2.mp4', 'view_count': 11}, {'id': 'bad_id', 'view_count': 12}]}
channel3.info_json = {'entries': [{'id': 'vid3.mp4', 'view_count': 13}, {'id': 'vid4.mp4', 'view_count': 14}]}
# Use the video file name as a unique source_id
for v in session.query(Video).all():
v.source_id = str(v.video_path.path).split('/')[-1]
# Check all videos are empty.
check_view_counts({'vid1.mp4': None, 'vid2.mp4': None, 'vid3.mp4': None, 'vid4.mp4': None})
# Channel 1 is updated, the other channels are left alone.
update_view_count(channel1.id)
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': None, 'vid3.mp4': None, 'vid4.mp4': None})
# Channel 2 is updated, the other channels are left alone. The 'bad_id' video is ignored.
update_view_count(channel2.id)
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': None, 'vid4.mp4': None})
# All videos are updated.
update_view_count(channel3.id)
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})
# An outdated view count will be overwritten.
with get_db_session(commit=True) as session:
vid = session.query(Video).filter_by(id=1).one()
vid.view_count = 8
check_view_counts({'vid1.mp4': 8, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})
update_view_count(channel1.id)
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})