mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-11-21 15:36:18 -05:00
* Squashed commit of the following:
commit c527475fcb4d4906457be5c10f0e9c6dee7fedc1
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 22:16:36 2021 -0700
Only saving a favorite video to the config if it has a source_id.
commit ae8b712ebeaef467b8d5a5b87ca0f4f276373a0d
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 22:14:26 2021 -0700
Ading description and changing Save button to Download.
commit 03a72be6dfdfa383915950a80e3f1ee9febf6705
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 22:09:44 2021 -0700
Removing uploader for now.
commit 5d0231284bf84e459687f6ab903b06bf4e3762bd
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 22:07:17 2021 -0700
Removing unique data from example info.
commit 9cc721a32d4715d9d6433e51db3b35e039c78d68
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 22:02:33 2021 -0700
Updating downloader tests.
commit 8e193b03349ae4613f105418cd3e8176c7101304
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 21:53:31 2021 -0700
Toasting error when WROL Mode is enabled. Disabling new downloads during WROL Mode.
commit 48c4311c31db8f2d2c8dd0a65f979a5f0a49a911
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 21:45:06 2021 -0700
Raising error if download exits without a 0. Clearing kill event when starting a new download.
commit 54ba3a890d59facc837410d34bf7afb40db3b685
Author: Roland <roland@learningselfreliance.com>
Date: Tue Dec 14 21:16:46 2021 -0700
Downloads can be restarted and killed.
commit bd60751262247259c3e9ceb5116e017d6314c219
Author: Roland <roland@learningselfreliance.com>
Date: Mon Dec 13 16:28:18 2021 -0700
Downloads can be started again. Downloads are sorted on Downloads page.
commit ea7f39231b92982ae0b263b76c5bd1902cc8169d
Author: Roland <roland@learningselfreliance.com>
Date: Sat Dec 11 21:24:16 2021 -0700
Building Downloads tab.
commit d3321872ac3743ff6df781b07ef0fb537f1d4c98
Author: Roland <roland@learningselfreliance.com>
Date: Sat Dec 11 20:41:46 2021 -0700
Not scheduling channels withtout a URL/frequency.
commit 84f268a0d4f58c85f86b6377900492e0766f6619
Author: Roland <roland@learningselfreliance.com>
Date: Sat Dec 11 19:56:08 2021 -0700
Channel downloads will be spread out when each channel is created.
commit 26fa6beea38ef3f8d6b7bf218f18222a51a87a60
Author: Roland <roland@learningselfreliance.com>
Date: Sat Dec 11 13:08:11 2021 -0700
Recurring downloads are renewed on startup.
commit fc2ab850533f1c3ad073f4692c7dea2be43e85d6
Author: Roland <roland@learningselfreliance.com>
Date: Fri Dec 10 17:03:57 2021 -0700
Fetching download again after do_download.
commit fdadad327fc09c17a49eb63b11e632fb72cc3195
Author: Roland <roland@learningselfreliance.com>
Date: Fri Dec 10 16:20:51 2021 -0700
Fixing error with test. Mock was not being reset.
commit 070dee238e53377acd22097c63cf3794e68ed423
Author: Roland <roland@learningselfreliance.com>
Date: Fri Dec 10 15:36:39 2021 -0700
Fixed bug in date handling (big surprise).
commit e53a8992268d68620c679455adaec3a590f20602
Author: Roland <roland@learningselfreliance.com>
Date: Wed Dec 1 16:48:36 2021 -0700
Using chunks for regular channel refresh.
commit 3bd60c58d1bef0226325f321978c0434b36b3be5
Author: Roland <roland@learningselfreliance.com>
Date: Wed Dec 1 15:22:02 2021 -0700
Reformatting.
commit 4fdc325f25a32c311ca6543fe321624b2bd564b2
Author: Roland <roland@learningselfreliance.com>
Date: Wed Dec 1 15:13:34 2021 -0700
Creating function that refreshes NO CHANNEL videos.
commit d67feafb6c2994f5a4a1295920a70db369cc6816
Author: Roland <roland@learningselfreliance.com>
Date: Wed Dec 1 14:35:54 2021 -0700
Removing unused functions.
commit 6b3918e4e78fbac64920f85c4de0df873ec366a1
Author: Roland <roland@learningselfreliance.com>
Date: Wed Dec 1 14:31:48 2021 -0700
Removing reporter and events. This will be replaced in the future with some sort of SSE.
commit 54fa1b0f5e93ca25c80b780dcdc3876b700273b1
Author: Roland <roland@learningselfreliance.com>
Date: Sun Nov 28 17:55:43 2021 -0700
Better error handling for archive button.
commit c31d124e511b27f626ed2df71486d47302b396f3
Author: Roland <roland@learningselfreliance.com>
Date: Sun Nov 28 17:29:43 2021 -0700
Displaying View Count in a human readable format.
commit 0ee2bd65588fd93b475cad6144e76f1b45191b39
Author: Roland <roland@learningselfreliance.com>
Date: Sat Nov 27 09:29:41 2021 -0700
Fixing tests.
commit e64aff32bab95c311ea3747e1a21b58b99750d5e
Author: Roland <roland@learningselfreliance.com>
Date: Sat Nov 27 08:59:51 2021 -0700
Handling when a Video has no Channel in UI.
commit b9549ecd5aac03c002adca7b7667f0a7189b662a
Author: Roland <roland@learningselfreliance.com>
Date: Fri Nov 26 08:35:36 2021 -0700
Downloaders can raise UnrecoverableDownloadError to signal a downlaod should not be retried.
commit da88798e2584d3e84900a70e34166c8c208c64fb
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 20:40:29 2021 -0700
Downloaders can refuse a download, which will then be deleted.
commit b865d856121db5ec8a3660c5d3c13b9846fde688
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 19:35:43 2021 -0700
Removing relative path handling.
commit 6d7757870b02ba030d78b3ef7e43f28b906eea61
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 19:28:27 2021 -0700
Using __json__ for Video and Channel models.
commit 649e4d6a61608272759d1e7ee02b490bb5c20b3e
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 18:23:38 2021 -0700
Fixing video links in UI.
commit 9d081af12742070b8519bd752812130cb5dbe393
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 14:42:38 2021 -0700
Fixing docs.
commit 1202cbe0f9ae744d7801b53a5071cded8ea39924
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 14:40:36 2021 -0700
Fixing wrolpi tests.
commit 3f6af8cc1123f2f63a3fc0f21ec4d994c642cf13
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 24 14:13:35 2021 -0700
Fixing tests to use MediaPath.
commit faec04f2a5e4c267a502c03884940a9c5bcfe387
Author: Roland <roland@learningselfreliance.com>
Date: Fri Nov 19 10:21:20 2021 -0700
Testing when a video has no channel.
commit cbf22aad7a0b72e61685e0236e843554b57c4af1
Author: Roland <roland@learningselfreliance.com>
Date: Thu Nov 18 16:24:03 2021 -0700
Using MediaPath for Video and Channel models. Testing VideoDownloader.do_download.
commit 7916ed61e43439e6cd60f23ce3fdf6c1a75bf02d
Author: Roland <roland@learningselfreliance.com>
Date: Thu Nov 18 12:43:44 2021 -0700
Channels/Videos downloaders can get gotten.
commit 808ef796cec01db6e7009065f9e26d25d9500ce2
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 17 21:45:19 2021 -0700
Upgrading to Sanic 21.9.1.
commit ea32813be28e78393ac456ee782a7245c098188f
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 17 20:29:01 2021 -0700
Creating Downloader and DownloaderManager classes which handle downloading URLs.
commit e859a77a77277ddf8b6e221ec9c9383c8ab91d66
Author: Roland <roland@learningselfreliance.com>
Date: Wed Nov 17 12:41:39 2021 -0700
Creating UI segement which allows the download/upload of media.
* Displaying all recurring downloads.
* Not downloading channels with empty url.
* Showing positive row for recurring downloads.
* Removing default last_successful_download.
* Only displaying Last Successful Download when there is one.
* Refreshing videos before finding missing videos.
* Using short video path name for log.
* Removing download history feature in favor of new Downloads page.
* Only showing info date if it exists.
* Making get_or_create_channel more reliable.
* Allowing null for channel download frequency.
* Logging when searching for channel.
* Storing channel's source_id in config.
* Importing using link instead of name.
* Getting channel id when importing channels.
* Removing redunant parentheses from repr.
* Preserving database channel values if there are not values in the config.
* Only highlighting pending downloads.
* Deleting old downloads periodically.
* Sorting recurring by next_download/frequency.
* Creating fixtures for channels, media directory and db session.
* Removing unused function. Wrapping config for tests.
* Deleting channel download when deleting a channel.
* Testing that a channel's download is deleted when the channel is deleted.
* Updating youtube-dl
* Switching from youtube-dl in favor of yt-dlp.
* Sorting once-downlaods by last_successful_download then id.
* Adding special handling to video_url_resolver for full urls.
* Consolidating downloader methods and functions. Reversing once-download sort order so the most recently downloaded are at the top.
* Getting session if one is not provided.
* Adding test for create_downloads.
* Not passing format option when getting filename.
* Linting
Signed-off-by: Roland <roland@learningselfreliance.com>
284 lines
11 KiB
Python
284 lines
11 KiB
Python
import pathlib
|
|
import subprocess
|
|
import tempfile
|
|
from unittest import mock
|
|
from unittest.mock import Mock, MagicMock
|
|
|
|
import pytest
|
|
from PIL import Image
|
|
|
|
from modules.videos.models import Channel, Video
|
|
from modules.videos.test.common import create_channel_structure
|
|
from wrolpi.common import get_absolute_media_path
|
|
from wrolpi.db import get_db_session
|
|
from wrolpi.test.common import build_test_directories, wrap_test_db, TestAPI
|
|
from wrolpi.vars import PROJECT_DIR
|
|
from ..common import get_matching_directories, convert_image, bulk_validate_posters, remove_duplicate_video_paths, \
|
|
update_view_count, get_video_duration
|
|
|
|
|
|
class TestCommon(TestAPI):
|
|
|
|
def test_get_absolute_media_path(self):
|
|
path = get_absolute_media_path('videos')
|
|
self.assertTrue(str(path).endswith('videos'))
|
|
|
|
def test_matching_directories(self):
|
|
structure = [
|
|
'foo/qux/',
|
|
'Bar/',
|
|
'baz/baz'
|
|
'barr',
|
|
'bazz',
|
|
]
|
|
|
|
with build_test_directories(structure) as temp_dir:
|
|
temp_dir = pathlib.Path(temp_dir)
|
|
|
|
# No directories have c
|
|
matches = get_matching_directories(temp_dir / 'c')
|
|
assert matches == []
|
|
|
|
# Get all directories starting with f
|
|
matches = get_matching_directories(temp_dir / 'f')
|
|
assert matches == [str(temp_dir / 'foo')]
|
|
|
|
# Get all directories starting with b, ignore case
|
|
matches = get_matching_directories(temp_dir / 'b')
|
|
assert matches == [str(temp_dir / 'Bar'), str(temp_dir / 'baz')]
|
|
|
|
# baz matches, but it has no subdirectories
|
|
matches = get_matching_directories(temp_dir / 'baz')
|
|
assert matches == [str(temp_dir / 'baz')]
|
|
|
|
# foo is an exact match, return subdirectories
|
|
matches = get_matching_directories(temp_dir / 'foo')
|
|
assert matches == [str(temp_dir / 'foo/qux')]
|
|
|
|
def test_get_video_duration(self):
|
|
"""
|
|
Video duration can be retrieved from the video file.
|
|
"""
|
|
video_path = PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4'
|
|
self.assertEqual(get_video_duration(video_path), 5)
|
|
|
|
video_path = PROJECT_DIR / 'test/does not exist.mp4'
|
|
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
|
|
|
|
video_path = str(PROJECT_DIR / 'test/does not exist.mp4')
|
|
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
|
|
|
|
with tempfile.NamedTemporaryFile() as fh:
|
|
self.assertRaises(subprocess.CalledProcessError, get_video_duration, fh.name)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
'paths,expected',
|
|
(
|
|
(['1.mp4', ], ['1.mp4', ]),
|
|
(['1.mp4', '2.mp4'], ['1.mp4', '2.mp4']),
|
|
(['1.mp4', '1.ogg'], ['1.mp4']),
|
|
(['1.bad_ext', '1.other_ext'], ['1.bad_ext']),
|
|
)
|
|
)
|
|
def test_remove_duplicate_video_paths(paths, expected):
|
|
result = remove_duplicate_video_paths(map(pathlib.Path, paths))
|
|
assert sorted(result) == sorted(map(pathlib.Path, expected))
|
|
|
|
|
|
def test_convert_image():
|
|
"""
|
|
An image's format can be changed. This tests that convert_image() converts from a WEBP format, to a JPEG format.
|
|
"""
|
|
foo = Image.new('RGB', (25, 25), color='grey')
|
|
|
|
with tempfile.TemporaryDirectory() as tempdir:
|
|
tempdir = pathlib.Path(tempdir)
|
|
|
|
# Save the new image to "foo.webp".
|
|
existing_path = tempdir / 'foo.webp'
|
|
foo.save(existing_path)
|
|
assert Image.open(existing_path).format == 'WEBP'
|
|
|
|
destination_path = tempdir / 'foo.jpg'
|
|
assert not destination_path.is_file()
|
|
|
|
# Convert the WEBP to a JPEG. The WEBP image should be removed.
|
|
convert_image(existing_path, destination_path)
|
|
assert not existing_path.is_file()
|
|
assert destination_path.is_file()
|
|
assert Image.open(destination_path).format == 'JPEG'
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
'_structure,paths',
|
|
(
|
|
(
|
|
{'channel1': ['vid1.mp4']},
|
|
[
|
|
'channel1/vid1.mp4',
|
|
],
|
|
),
|
|
(
|
|
{'channel1': ['vid1.mp4'], 'channel2': ['vid1.mp4']},
|
|
[
|
|
'channel1/vid1.mp4',
|
|
'channel2/vid1.mp4',
|
|
],
|
|
),
|
|
(
|
|
{'channel1': ['vid1.mp4'], 'channel2': ['vid1.mp4', 'vid2.mp4', 'vid2.en.vtt']},
|
|
[
|
|
'channel1/vid1.mp4',
|
|
'channel2/vid1.mp4',
|
|
'channel2/vid2.mp4',
|
|
'channel2/vid2.en.vtt',
|
|
],
|
|
),
|
|
)
|
|
)
|
|
def test_create_db_structure(_structure, paths):
|
|
@create_channel_structure(_structure)
|
|
def test_func(_, tempdir):
|
|
assert isinstance(tempdir, pathlib.Path)
|
|
for path in paths:
|
|
path = (tempdir / path)
|
|
assert path.exists()
|
|
assert path.is_file()
|
|
|
|
with get_db_session() as session:
|
|
for channel_name in _structure:
|
|
channel = session.query(Channel).filter_by(name=channel_name).one()
|
|
assert (tempdir / channel_name).is_dir()
|
|
assert channel
|
|
assert channel.directory == tempdir / channel_name
|
|
assert len(channel.videos) == len([i for i in _structure[channel_name] if i.endswith('mp4')])
|
|
|
|
with tempfile.TemporaryDirectory() as tmp_dir, \
|
|
mock.patch('wrolpi.media_path.get_media_directory') as mock_get_directory:
|
|
mock_get_directory.return_value = pathlib.Path(tmp_dir)
|
|
test_ = MagicMock()
|
|
test_.tmp_dir.name = tmp_dir
|
|
test_func(test_)
|
|
|
|
|
|
@wrap_test_db
|
|
@create_channel_structure(
|
|
{
|
|
'channel1': ['vid1.mp4', 'vid1.jpg'],
|
|
'channel2': ['vid2.flv', 'vid2.webp'],
|
|
}
|
|
)
|
|
def test_bulk_replace_invalid_posters(tempdir: pathlib.Path):
|
|
"""
|
|
Test that when a video has an invalid poster format, we convert it to JPEG.
|
|
"""
|
|
channel1, channel2 = sorted(tempdir.iterdir())
|
|
jpg, mp4 = sorted(channel1.iterdir())
|
|
flv, webp = sorted(channel2.iterdir())
|
|
|
|
Image.new('RGB', (25, 25)).save(jpg)
|
|
Image.new('RGB', (25, 25)).save(webp)
|
|
|
|
with open(jpg, 'rb') as jpg_fh, open(webp, 'rb') as webp_fh:
|
|
# Files are different formats.
|
|
jpg_fh_contents = jpg_fh.read()
|
|
webp_fh_contents = webp_fh.read()
|
|
assert jpg_fh_contents != webp_fh_contents
|
|
assert Image.open(jpg_fh).format == 'JPEG'
|
|
assert Image.open(webp_fh).format == 'WEBP'
|
|
|
|
with get_db_session() as session:
|
|
vid1 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel1/vid1.jpg').one()
|
|
assert vid1.validated_poster is False
|
|
|
|
vid2 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel2/vid2.webp').one()
|
|
assert vid2.validated_poster is False
|
|
|
|
# Convert the WEBP image. convert_image() should only be called once.
|
|
mocked_convert_image = Mock(wraps=convert_image)
|
|
with mock.patch('modules.videos.common.convert_image', mocked_convert_image):
|
|
video_ids = [vid1.id, vid2.id]
|
|
bulk_validate_posters(video_ids)
|
|
|
|
mocked_convert_image.assert_called_once_with(webp, tempdir / 'channel2/vid2.jpg')
|
|
|
|
with get_db_session() as session:
|
|
# Get the video by ID because it's poster is now a JPEG.
|
|
vid2 = session.query(Video).filter_by(id=vid2.id).one()
|
|
assert str(vid2.poster_path.path).split('/')[-1] == 'vid2.jpg'
|
|
assert all('webp' not in str(i.poster_path.path) for i in session.query(Video).all())
|
|
assert vid2.validated_poster is True
|
|
|
|
# Vid1's image was validated, but not converted.
|
|
vid1 = session.query(Video).filter_by(id=vid1.id).one()
|
|
assert str(vid1.poster_path.path).split('/')[-1] == 'vid1.jpg'
|
|
assert vid1.validated_poster is True
|
|
|
|
# Old webp was removed
|
|
assert not webp.is_file()
|
|
new_jpg = tempdir / 'channel2/vid2.jpg'
|
|
assert new_jpg.is_file()
|
|
# chmod 644
|
|
assert new_jpg.stat().st_mode == 0o100644
|
|
with open(new_jpg, 'rb') as new_jpg_fh:
|
|
# The converted image is the same as the other JPEG because both are black 25x25 pixel images.
|
|
assert jpg_fh_contents == new_jpg_fh.read()
|
|
assert Image.open(new_jpg_fh).format == 'JPEG'
|
|
|
|
# Calling convert again has no effect.
|
|
mocked_convert_image.reset_mock()
|
|
with mock.patch('modules.videos.common.convert_image', mocked_convert_image):
|
|
video_ids = [vid1.id, vid2.id]
|
|
bulk_validate_posters(video_ids)
|
|
|
|
mocked_convert_image.assert_not_called()
|
|
|
|
|
|
@wrap_test_db
|
|
@create_channel_structure(
|
|
{
|
|
'channel1': ['vid1.mp4', 'vid1.jpg'],
|
|
'channel2': ['vid2.mp4'],
|
|
'channel3': ['vid3.mp4', 'vid4.mp4'],
|
|
}
|
|
)
|
|
def test_update_view_count(tempdir: pathlib.Path):
|
|
def check_view_counts(view_counts):
|
|
with get_db_session() as session_:
|
|
for source_id, view_count in view_counts.items():
|
|
vid = session_.query(Video).filter_by(source_id=source_id).one()
|
|
assert vid.view_count == view_count
|
|
|
|
with get_db_session(commit=True) as session:
|
|
channel1, channel2, channel3 = session.query(Channel).order_by(Channel.id).all()
|
|
channel1.info_json = {'entries': [{'id': 'vid1.mp4', 'view_count': 10}]}
|
|
channel2.info_json = {'entries': [{'id': 'vid2.mp4', 'view_count': 11}, {'id': 'bad_id', 'view_count': 12}]}
|
|
channel3.info_json = {'entries': [{'id': 'vid3.mp4', 'view_count': 13}, {'id': 'vid4.mp4', 'view_count': 14}]}
|
|
|
|
# Use the video file name as a unique source_id
|
|
for v in session.query(Video).all():
|
|
v.source_id = str(v.video_path.path).split('/')[-1]
|
|
|
|
# Check all videos are empty.
|
|
check_view_counts({'vid1.mp4': None, 'vid2.mp4': None, 'vid3.mp4': None, 'vid4.mp4': None})
|
|
|
|
# Channel 1 is updated, the other channels are left alone.
|
|
update_view_count(channel1.id)
|
|
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': None, 'vid3.mp4': None, 'vid4.mp4': None})
|
|
|
|
# Channel 2 is updated, the other channels are left alone. The 'bad_id' video is ignored.
|
|
update_view_count(channel2.id)
|
|
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': None, 'vid4.mp4': None})
|
|
|
|
# All videos are updated.
|
|
update_view_count(channel3.id)
|
|
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})
|
|
|
|
# An outdated view count will be overwritten.
|
|
with get_db_session(commit=True) as session:
|
|
vid = session.query(Video).filter_by(id=1).one()
|
|
vid.view_count = 8
|
|
check_view_counts({'vid1.mp4': 8, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})
|
|
update_view_count(channel1.id)
|
|
check_view_counts({'vid1.mp4': 10, 'vid2.mp4': 11, 'vid3.mp4': 13, 'vid4.mp4': 14})
|