wrolpi/modules/videos/test/test_downloader.py
lrnselfreliance 5358c75a2b
Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:25:23 -06:00

1864 lines
86 KiB
Python

import asyncio
import json
import shutil
from copy import copy
from http import HTTPStatus
from unittest import mock
import pytest
from modules.videos.downloader import VideoDownloader, \
get_or_create_channel, channel_downloader, video_downloader, preview_filename, \
prepare_filename, convert_wrolpi_filename_format, _bot_blocked, _skip_download, \
parse_ytdlp_progress, is_youtube_rss_feed_url, is_youtube_url, format_selector, VIDEO_RESOLUTION_MAP
from modules.videos.lib import get_videos_downloader_config
from modules.videos.models import Channel, Video
from wrolpi.conftest import test_directory, await_switches
from wrolpi.downloader import Download, DownloadResult, get_download_manager_config
from wrolpi.errors import InvalidDownload, UnrecoverableDownloadError
from wrolpi.test.common import skip_circleci
from wrolpi.vars import PROJECT_DIR
example_video_json = {
'age_limit': 0,
'average_rating': 4.6923075,
'categories': ['Howto & Style'],
'channel': 'channel name',
'channel_id': 'channel id',
'channel_url': 'https://example.com/example_video_json_channel',
'duration': 635,
'extractor': 'youtube',
'extractor_key': 'Youtube',
'id': 'some long id',
'is_live': None,
'like_count': 24,
'subtitles': {},
'title': "The video's title",
'upload_date': '20190707',
'uploader': 'uploader name',
'uploader_id': 'uploader id',
'uploader_url': 'https://example.com/example_video_json_uploader',
'view_count': 406,
'webpage_url': 'webpage url',
'webpage_url_basename': 'watch',
}
example_channel_json = {
'_type': 'playlist',
'channel_id': 'some id',
'entries': [
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 1 id',
'ie_key': 'Youtube',
'title': 'video 1 title',
'uploader': None,
'url': 'video_1_url',
'view_count': 58504,
'upload_date': '20190707',
'webpage_url': 'https://youtube.com/watch?v=video_1_url'},
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 2 id',
'ie_key': 'Youtube',
'title': 'video 2 title',
'uploader': None,
'url': 'video_2_url',
'view_count': 1413,
'upload_date': '20190707',
'webpage_url': 'https://youtube.com/watch?v=video_2_url'},
],
'extractor': 'youtube:tab',
'extractor_key': 'YoutubeTab',
'id': 'some id',
'title': 'channel title',
'uploader': 'the uploader',
'webpage_url': 'channel url',
'webpage_url_basename': 'videos',
}
@pytest.mark.asyncio
async def test_download_no_channel(test_session, video_download_manager, test_directory, await_switches,
mock_video_extract_info, mock_video_process_runner):
"""A video can be downloaded even if it does not have a Channel."""
channel_dir = test_directory / 'NO CHANNEL'
channel_dir.mkdir(parents=True)
video_path = channel_dir / 'video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
channel_downloader, video_downloader = video_download_manager.instances
# Video has no channel
info_json = copy(example_video_json)
del info_json['channel']
del info_json['channel_id']
del info_json['channel_url']
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
mock_video_extract_info.return_value = info_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
video: Video = test_session.query(Video).one()
assert str(video.video_path) == f'{channel_dir}/video.mp4'
@pytest.mark.asyncio
async def test_download_video_tags(test_session, video_download_manager, video_file_factory, test_directory,
mock_video_extract_info, mock_video_process_runner, tag_factory, await_switches):
"""A Video is tagged when Download record requires it."""
video_path = video_file_factory()
tag1, tag2 = await tag_factory(), await tag_factory()
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
mock_video_extract_info.return_value = copy(example_video_json)
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
settings = dict(tag_names=[tag1.name, tag2.name])
video_download_manager.create_download(test_session, url, video_downloader.name, settings=settings)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
video: Video = test_session.query(Video).one()
assert [i.tag.name for i in video.file_group.tag_files] == [tag1.name, tag2.name]
@skip_circleci
@pytest.mark.asyncio
async def test_download_channel(test_session, test_directory, simple_channel, video_download_manager, video_file,
mock_video_extract_info, mock_video_prepare_filename, await_switches,
mock_video_process_runner):
"""Downloading (updating the catalog of) a Channel updates its info_json.
If a Channel has `match_regex` only those videos with matching titles will be downloaded."""
url = 'https://www.youtube.com/c/LearningSelfReliance/videos'
mock_video_prepare_filename.return_value = video_file
mock_video_extract_info.return_value = example_channel_json
with mock.patch('modules.videos.downloader.get_channel') as mock_get_channel:
mock_get_channel.return_value = simple_channel
simple_channel.get_or_create_download(test_session, url, 60)
# Download channel.
await video_download_manager.wait_for_all_downloads()
# Download videos.
await video_download_manager.wait_for_all_downloads()
def reset_downloads():
[i.delete(add_to_skip_list=False) for i in test_session.query(Download)]
# Let background tasks run.
await asyncio.sleep(0)
# Two videos are in the example channel.
downloads = video_download_manager.get_once_downloads(test_session)
downloads = filter(lambda i: 'watch' in i.url, downloads)
assert {i.url for i in downloads} == \
{'https://youtube.com/watch?v=video_2_url', 'https://youtube.com/watch?v=video_1_url'}
assert all(i.status_code == 'complete' for i in downloads)
reset_downloads()
# A channel with `title_include` only returns matching video URLs.
test_session.commit()
with mock.patch('modules.videos.downloader.get_channel') as mock_get_channel:
mock_get_channel.return_value = simple_channel
download = simple_channel.get_or_create_download(test_session, url, 60)
download.frequency = 100
download.settings = {'title_include': '2'}
await video_download_manager.wait_for_all_downloads()
downloads = video_download_manager.get_once_downloads(test_session)
assert {i.url for i in downloads} == {'https://youtube.com/watch?v=video_2_url'}
assert downloads[0].settings == {
'collection_id': simple_channel.collection_id,
'channel_url': 'https://www.youtube.com/c/LearningSelfReliance/videos',
'destination': str(test_directory),
'parent_download_url': url,
}
reset_downloads()
# A channel with `title_exclude` only returns non-matching video URLs.
test_session.commit()
with mock.patch('modules.videos.downloader.get_channel') as mock_get_channel:
mock_get_channel.return_value = simple_channel
download = simple_channel.get_or_create_download(test_session, url, 60)
download.frequency = 100
download.settings = {'title_exclude': '2'}
await video_download_manager.wait_for_all_downloads()
downloads = video_download_manager.get_once_downloads(test_session)
assert {i.url for i in downloads} == {'https://youtube.com/watch?v=video_1_url'}
assert downloads[0].settings == {
'collection_id': simple_channel.collection_id,
'channel_url': 'https://www.youtube.com/c/LearningSelfReliance/videos',
'destination': str(test_directory),
'parent_download_url': url,
}
@pytest.mark.asyncio
async def test_get_or_create_channel(async_client, test_session, test_directory, tag_factory, await_switches):
"""A Channel may need to be created for an arbitrary download.
Attempt to use an existing Channel if we can match it.
"""
from wrolpi.collections import Collection
one, two = await tag_factory(), await tag_factory()
# Create Collections for each Channel
def create_channel_with_collection(name, source_id=None, url=None):
collection = Collection(name=name, kind='channel')
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id=source_id, url=url)
test_session.add(channel)
test_session.flush([channel])
return channel
c1 = create_channel_with_collection('foo', source_id='foo', url='https://example.com')
c2 = create_channel_with_collection('bar', source_id='bar')
c3 = create_channel_with_collection('baz', source_id='baz', url='https://example.net')
c4 = create_channel_with_collection('qux')
test_session.commit()
# All existing channels should be used.
tests = [
(dict(source_id='foo'), c1),
(dict(url='https://example.com'), c1),
(dict(url='https://example.com', source_id='bar'), c2), # source_id has priority.
(dict(name='foo', source_id='bar'), c2),
(dict(source_id='bar'), c2),
(dict(source_id='baz'), c3),
(dict(name='qux'), c4),
]
for kwargs, expected in tests:
channel = get_or_create_channel(test_session, **kwargs)
assert expected.id == channel.id, f'Expected {expected} for {kwargs} but got {channel}'
# A new channel is created. It will not be automatically downloaded.
channel = get_or_create_channel(test_session, source_id='quux', name='quux', url='https://example.org')
assert channel.id == 5
assert channel.source_id == 'quux'
assert channel.name == 'quux'
assert channel.url == 'https://example.org'
# New channel can be retrieved.
assert get_or_create_channel(test_session, source_id='quux') == channel
# A Channel can be created with a tag name.
channel = get_or_create_channel(test_session, name='One Channel', tag_name=one.name)
await await_switches()
assert channel.directory == test_directory / f'videos/one/One Channel'
assert channel.tag_name == one.name and channel.tag == one
# Getting the same Channel will not change the tag.
channel = get_or_create_channel(test_session, name='One Channel', tag_name=two.name)
assert channel.directory == test_directory / f'videos/one/One Channel'
assert channel.tag_name == one.name and channel.tag == one
channel = get_or_create_channel(test_session, name='One Channel')
assert channel.directory == test_directory / f'videos/one/One Channel'
assert channel.directory.is_dir()
assert channel.tag_name == one.name and channel.tag == one
@pytest.mark.asyncio
async def test_get_or_create_channel_conflict_recovery(async_client, test_session, test_directory):
"""get_or_create_channel should find an existing channel even when source_id and url don't match,
rather than raising ChannelNameConflict."""
from wrolpi.collections import Collection
# Create a channel with a specific source_id and url.
collection = Collection(name='My Channel', kind='channel')
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='old_source_id',
url='https://www.youtube.com/channel/UCold')
test_session.add(channel)
test_session.commit()
# Calling with different source_id and url but same name should return the existing channel.
result = get_or_create_channel(test_session, source_id='new_source_id',
url='https://www.youtube.com/@mychannel', name='My Channel')
assert result.id == channel.id
@pytest.mark.asyncio
async def test_get_or_create_channel_youtube_channel_url(async_client, test_session, test_directory):
"""get_or_create_channel should find an existing channel when the URL contains a YouTube /channel/ ID
that matches the channel's source_id."""
from wrolpi.collections import Collection
collection = Collection(name='WROLPi', kind='channel')
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='UC4t8bw1besFTyjW7ZBCOIrw',
url='https://www.youtube.com/@wrolpi')
test_session.add(channel)
test_session.commit()
# A /channel/UCXXX URL should match via source_id extraction.
result = get_or_create_channel(test_session,
url='https://www.youtube.com/channel/UC4t8bw1besFTyjW7ZBCOIrw',
name='WROLPi Channel')
assert result.id == channel.id
@pytest.mark.asyncio
async def test_get_or_create_channel_with_destination(async_client, test_session, test_directory):
"""A new Channel should be created at the destination directory when provided."""
destination = test_directory / 'custom/my_channel'
channel = get_or_create_channel(test_session, source_id='abc', name='My Channel',
url='https://example.com', destination=destination)
assert channel.name == 'My Channel'
assert channel.directory == destination
assert destination.is_dir()
@pytest.mark.asyncio
async def test_get_or_create_channel_destination_conflict(async_client, test_session, test_directory):
"""When destination matches an existing channel's directory, that channel should be returned
instead of trying to create a new one (which would raise ChannelDirectoryConflict)."""
from wrolpi.collections import Collection
# Create an existing channel at the destination directory.
existing_dir = test_directory / 'videos/existing'
existing_dir.mkdir(parents=True)
collection = Collection(name='Existing Channel', kind='channel', directory=existing_dir)
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='existing')
test_session.add(channel)
test_session.commit()
# Destination matches an existing channel, so it should be returned even though source_id/name/url don't match.
result = get_or_create_channel(test_session, source_id='new_id', name='New Channel',
url='https://example.com', destination=existing_dir)
assert result.id == channel.id
@pytest.mark.asyncio
async def test_get_or_create_channel_destination_nested_conflict(async_client, test_session, test_directory):
"""When destination is nested inside an existing channel's directory, the parent channel should be returned."""
from wrolpi.collections import Collection
parent_dir = test_directory / 'videos/parent_channel'
parent_dir.mkdir(parents=True)
collection = Collection(name='Parent Channel', kind='channel', directory=parent_dir)
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='parent')
test_session.add(channel)
test_session.commit()
# Nested directory should find the parent channel.
nested_dir = parent_dir / 'nested'
result = get_or_create_channel(test_session, source_id='nested_id', name='Nested Channel',
url='https://example.net', destination=nested_dir)
assert result.id == channel.id
@pytest.mark.asyncio
async def test_get_or_create_channel_computed_directory_conflict(async_client, test_session, test_directory):
"""When no existing channel matches by source_id/url/name, but the directory that would be created
already belongs to a channel, that channel should be returned instead of raising.
YouTube collaboration videos report the primary uploader's channel, which may not match any local
channel by name (e.g. the local channel was renamed) even though its directory is already taken."""
from wrolpi.collections import Collection
existing_dir = test_directory / 'videos/Uploader'
existing_dir.mkdir(parents=True)
collection = Collection(name='Uploader (old name)', kind='channel', directory=existing_dir)
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='UCold', url='https://www.youtube.com/@olduploader')
test_session.add(channel)
test_session.commit()
result = get_or_create_channel(test_session, source_id='UCnew',
url='https://www.youtube.com/channel/UCnew', name='Uploader')
assert result.id == channel.id
@pytest.mark.asyncio
async def test_get_or_create_channel_existing_ignores_destination(async_client, test_session, test_directory):
"""When a channel already exists (matched by source_id), the destination parameter should be ignored."""
from wrolpi.collections import Collection
existing_dir = test_directory / 'videos/existing'
existing_dir.mkdir(parents=True)
collection = Collection(name='Existing', kind='channel', directory=existing_dir)
test_session.add(collection)
test_session.flush([collection])
channel = Channel(collection_id=collection.id, source_id='match_me', url='https://example.com')
test_session.add(channel)
test_session.commit()
# Even with a different destination, the existing channel is returned.
other_dir = test_directory / 'videos/other'
result = get_or_create_channel(test_session, source_id='match_me', name='Existing',
destination=other_dir)
assert result.id == channel.id
assert result.directory == existing_dir
@pytest.mark.asyncio
async def test_get_channel_prefers_destination_over_ytdlp(async_client, test_session, test_directory, channel_factory):
"""When yt-dlp metadata identifies ChannelA but the download destination is ChannelB's directory,
_get_channel should return ChannelB (the destination's channel)."""
channel_a = channel_factory(source_id='channel_a_id', name='ChannelA',
url='https://example.com/channel_a')
channel_b = channel_factory(source_id='channel_b_id', name='ChannelB',
url='https://example.com/channel_b')
# Download has yt-dlp metadata matching ChannelA, but destination is ChannelB's directory.
download = Download(
url='https://example.com/video1',
destination=channel_b.directory,
info_json={'channel': 'ChannelA', 'channel_id': 'channel_a_id',
'channel_url': 'https://example.com/channel_a'},
)
channel = await VideoDownloader._get_channel(test_session, download)
assert channel.id == channel_b.id, 'Should prefer the destination channel over yt-dlp metadata'
@pytest.mark.asyncio
async def test_get_channel_uses_ytdlp_when_destination_matches(async_client, test_session, test_directory,
channel_factory):
"""When yt-dlp metadata identifies ChannelA and the download destination is also ChannelA's directory,
_get_channel should return ChannelA (no conflict)."""
channel_a = channel_factory(source_id='channel_a_id', name='ChannelA',
url='https://example.com/channel_a')
download = Download(
url='https://example.com/video1',
destination=channel_a.directory,
info_json={'channel': 'ChannelA', 'channel_id': 'channel_a_id',
'channel_url': 'https://example.com/channel_a'},
)
channel = await VideoDownloader._get_channel(test_session, download)
assert channel.id == channel_a.id
def test_channel_downloader_hidden(video_download_manager):
"""
ChannelDownloader should not be presented to the User.
"""
downloaders = video_download_manager.list_downloaders()
assert [i.__json__() for i in downloaders] == [(dict(name='video', pretty_name='Videos')), ]
def test_bad_downloader(test_session, video_download_manager):
"""
Attempting to use an unknown downloader should raise an error.
"""
with pytest.raises(InvalidDownload):
video_download_manager.create_download(test_session, 'https://example.com', downloader_name='bad downloader')
@pytest.mark.parametrize('url,expected', [
('https://www.youtube.com/feeds/videos.xml?channel_id=UC2JGH5ZBtWQJuTYCyfI3M3Q', True),
('http://youtube.com/feeds/videos.xml?channel_id=UC123', True),
('https://www.youtube.com/feeds/videos.xml', True),
('https://www.youtube.com/@SomeChannel/videos', False),
('https://www.youtube.com/watch?v=abc123', False),
('https://example.com/feeds/videos.xml', False),
('', False),
(None, False),
])
def test_is_youtube_rss_feed_url(url, expected):
"""Detect YouTube RSS feed URLs so ChannelDownloader can refuse to process them."""
assert is_youtube_rss_feed_url(url) is expected
@pytest.mark.asyncio
async def test_channel_downloader_rejects_rss_feed_url(test_session, mock_video_extract_info):
"""ChannelDownloader must refuse YouTube RSS feed URLs.
Previously, yt-dlp would return no `uploader`/`channel_id` for an RSS feed, so
`name = info.get('uploader') or info.get('webpage_url_basename')` fell back to
'videos.xml' and `source_id` fell back to 'videos', creating a junk channel named
'videos.xml' in directory 'videos/videos.xml' that never received any videos.
"""
download = Download(
url='https://www.youtube.com/feeds/videos.xml?channel_id=UC2JGH5ZBtWQJuTYCyfI3M3Q',
downloader=channel_downloader.name,
)
test_session.add(download)
test_session.commit()
# If the rejection was missing, ChannelDownloader would call extract_info and create a channel.
mock_video_extract_info.return_value = {
'id': 'videos',
'webpage_url_basename': 'videos.xml',
'entries': [],
}
with pytest.raises(UnrecoverableDownloadError):
# RSS-feed rejection happens in the prepare phase (sync), so we don't even
# need to drive execute_download here.
channel_downloader.prepare_download(test_session, download)
# No Channel should have been created from the RSS URL.
assert test_session.query(Channel).count() == 0
mock_video_extract_info.assert_not_called()
@pytest.mark.asyncio
async def test_video_download(test_session, test_directory, mock_video_extract_info, simple_channel, await_switches,
video_download_manager, mock_video_process_runner, image_file):
"""A video download is performed, files are grouped."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
# Create a poster file which was downloaded.
poster_path = video_path.with_suffix('.png')
image_file.rename(poster_path)
# `formats` should be cleaned from info json.
info_json_path = video_path.with_suffix('.info.json')
info_json_path.write_text(json.dumps({'duration': 5, 'formats': [1, 2, 3]}))
# .part file should be deleted
part_path = video_path.with_suffix('.mp4.part')
part_path.touch()
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
download, cmd, out_dir = mock_video_process_runner.call_args[0]
download: Download = test_session.query(Download).one()
assert download.url in cmd
assert test_session.query(Channel).one()
video: Video = test_session.query(Video).one()
assert video.video_path.is_absolute() and video.video_path == video_path, 'Video path is not absolute'
assert video.video_path.is_absolute() and video.poster_path == poster_path, 'Video poster was not discovered'
assert not any([i for i in video.file_group.my_paths() if i.name.endswith('.part')]), \
'Part file should not be in group'
assert not part_path.exists(), 'Part file should be deleted.'
# Some info json data is deleted during download.
assert video.get_info_json(), 'Download should keep info json file'
assert 'duration' in video.get_info_json(), 'Duration should stay in info json'
assert 'formats' not in video.get_info_json(), 'formats should be deleted from info json'
@pytest.mark.asyncio
async def test_download_info_json_finds_existing_video(test_session, test_directory, video_factory):
"""`download_info_json` looks up the existing Video by its URL.
Regression: `Video.get_by_url` was called with its arguments swapped — `(url, session)` instead of
`(session, url)` — which raised `AttributeError: 'str' object has no attribute 'query'`."""
source_id = 'video-source-id'
url = 'https://example.com/the-video'
video = video_factory(with_info_json={'id': source_id}, source_id=source_id)
video.file_group.url = url
test_session.commit()
download = Download(url=url, info_json={'id': source_id}, settings={})
# The info_json download itself fails here, but the buggy existing-video lookup runs first.
with mock.patch('modules.videos.downloader.download_video_info_json') as mock_download:
mock_download.side_effect = Exception('no network')
result = await VideoDownloader.download_info_json(download, video.video_path)
assert result.success is False, 'Download should fail gracefully once the info json download fails'
mock_download.assert_called_once_with(url)
@pytest.mark.asyncio
async def test_download_result(test_session, test_directory, video_download_manager, mock_video_process_runner,
mock_video_extract_info, image_file, await_switches):
"""VideoDownloader returns a DownloadResult when complete."""
channel_directory = test_directory / 'videos/channel name'
channel_directory.mkdir(parents=True)
video_file = channel_directory / 'the video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_file)
image_file.rename(video_file.with_suffix('.jpg'))
mock_video_extract_info.return_value = example_video_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_prepare_filename.return_value = [video_file, {'id': 'foo'}]
video_download_manager.create_download(test_session, 'https://example.com', video_downloader.name)
await video_download_manager.wait_for_all_downloads()
# Sleep to allow background tasks to finish.
await asyncio.sleep(1)
download: Download = test_session.query(Download).one()
assert download.url == 'https://example.com'
video = test_session.query(Video).one()
assert download.location == f'/videos/{video.file_group_id}'
# Video has its files.
video = test_session.query(Video).one()
assert video.video_path.is_file(), 'Video file was not found.'
assert video.poster_path.is_file(), 'Poster file was not found.'
@pytest.mark.asyncio
async def test_download_destination(test_session, test_directory, video_download_manager, await_switches,
mock_video_process_runner, mock_video_extract_info, mock_video_prepare_filename):
"""A Video can be downloaded to a directory other than it's Channel's directory."""
# yt-dlp would return the video json.
mock_video_extract_info.return_value = example_video_json
# This result will be ignored during the test, we just need some value so the download does not fail.
mock_video_prepare_filename.return_value = str(test_directory / 'test video.mp4')
video_download_manager.create_download(test_session, 'https://example.com/1', downloader_name=VideoDownloader.name)
await video_download_manager.wait_for_all_downloads()
# Output directory matches the channel directory.
assert mock_video_prepare_filename.call_args_list[0].kwargs['ydl'].params['outtmpl']['default'] \
.startswith(f'{test_directory}/videos/channel name/%(uploader)s')
mock_video_prepare_filename.reset_mock()
video_download_manager.create_download(test_session, 'https://example.com/2', downloader_name=VideoDownloader.name,
destination=f'{test_directory}/custom')
await video_download_manager.wait_for_all_downloads()
# Output directory matches the custom directory specified.
assert mock_video_prepare_filename.call_args_list[0].kwargs['ydl'].params['outtmpl']['default'] \
.startswith(f'{test_directory}/custom/%(uploader)s')
example_playlist_json = {
'_type': 'playlist',
'availability': None,
'channel': 'the channel name',
'channel_follower_count': None,
'channel_id': 'the channel id',
'channel_url': 'channel url',
'description': '',
'entries': [
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 2 id',
'ie_key': 'Youtube',
'title': 'video 2 title',
'uploader': None,
'upload_date': '20241108',
'url': 'https://www.youtube.com/shorts/video_2_url',
'view_count': 1413,
'webpage_url': 'https://www.youtube.com/shorts/video_2_url'},
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 1 id',
'ie_key': 'Youtube',
'title': 'video 1 title',
'uploader': None,
'upload_date': '20241108',
'url': 'https://www.youtube.com/watch?v=video_1_url',
'view_count': 58504,
'webpage_url': 'https://www.youtube.com/watch?v=video_1_url'},
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 3 id',
'ie_key': 'Youtube',
'title': 'video 3 title',
'uploader': None,
'upload_date': '20241108',
'url': 'https://youtube.com/watch?v=video_3_url',
'view_count': 58504,
'webpage_url': 'https://youtube.com/watch?v=video_3_url'},
],
'extractor': 'youtube:tab',
'extractor_key': 'YoutubeTab',
'id': 'the playlist id',
'modified_date': '20220426',
'original_url': 'original url',
'playlist_count': 10,
'requested_entries': None,
'tags': [],
'title': 'some title',
'uploader': 'Playlist Uploader',
'uploader_id': 'uploader id',
'uploader_url': 'uploader url',
'view_count': 22298,
'webpage_url': 'webpage url',
'webpage_url_basename': 'playlist',
'webpage_url_domain': 'youtube.com',
}
@pytest.mark.asyncio
async def test_download_playlist(test_session, test_directory, mock_video_extract_info, video_download_manager,
video_file):
"""All videos in a playlist can be downloaded for it's Channel."""
download = Download(url='https://example.com/playlist-url')
test_session.add(download)
channel = get_or_create_channel(test_session, source_id=example_playlist_json['channel_id'], url=download.url,
name=example_channel_json['uploader'])
video = Video.from_paths(test_session, video_file)
video.channel_id = channel.id
video.source_id = 'video 1 id'
video.file_group.url = 'https://www.youtube.com/watch?v=video_1_url'
test_session.commit()
mock_video_extract_info.return_value = example_playlist_json # Playlist info is fetched first.
# Drive the phase-split API directly: prepare → execute → finalize. No dispatch
# loop, so child video downloads are never spawned here — no need to stub them.
from modules.archive.conftest import make_test_ctx
prepared = channel_downloader.prepare_download(test_session, download)
executed = await channel_downloader.execute_download(prepared, make_test_ctx(), download=download)
result = channel_downloader.finalize_download(test_session, download, executed)
assert result.success is True, 'Download was not successful'
assert set(result.downloads) == {
'https://www.youtube.com/watch?v=video_2_url', # Shorts is converted to regular URL.
'https://youtube.com/watch?v=video_3_url',
}
def test_get_missing_videos_views_order(test_session):
"""The 'views' download order sorts videos most-viewed-first. Entries missing a `view_count`
(some listings omit it) sort last rather than raising a KeyError."""
from modules.videos.downloader import channel_downloader
download = Download(
url='https://www.youtube.com/@example/videos',
settings={'download_order': 'views'},
info_json={'entries': [
{'_type': 'url', 'id': 'a', 'title': 'a', 'url': 'https://youtube.com/watch?v=a', 'view_count': 918},
{'_type': 'url', 'id': 'b', 'title': 'b', 'url': 'https://youtube.com/watch?v=b'}, # No view_count.
{'_type': 'url', 'id': 'c', 'title': 'c', 'url': 'https://youtube.com/watch?v=c', 'view_count': 3000},
]},
)
test_session.add(download)
test_session.commit()
missing_videos = channel_downloader.get_missing_videos(test_session, download)
assert missing_videos == [
'https://youtube.com/watch?v=c',
'https://youtube.com/watch?v=a',
'https://youtube.com/watch?v=b',
]
def test_get_missing_videos_views_order_no_view_counts(test_session):
"""When no entry has a `view_count` (old yt-dlp does not extract it from YouTube's lockupViewModel
listings), the 'views' download order fails cleanly rather than downloading in arbitrary order."""
from modules.videos.downloader import channel_downloader
download = Download(
url='https://www.youtube.com/@example/videos',
settings={'download_order': 'views'},
info_json={'entries': [
{'_type': 'url', 'id': 'a', 'title': 'a', 'url': 'https://youtube.com/watch?v=a'},
{'_type': 'url', 'id': 'b', 'title': 'b', 'url': 'https://youtube.com/watch?v=b'},
]},
)
test_session.add(download)
test_session.commit()
with pytest.raises(UnrecoverableDownloadError):
channel_downloader.get_missing_videos(test_session, download)
@pytest.mark.asyncio
async def test_channel_download_crud(test_session, async_client, assert_downloads, tag_factory):
"""Test creating more complex Channel Download."""
one, two = await tag_factory(), await tag_factory()
download = {
'urls': ['https://www.youtube.com/@wrolpi/videos'],
'destination': '',
'downloader': 'video_channel',
'sub_downloader': 'video',
'frequency': 2592000,
'tag_names': [one.name, two.name],
'settings': {
'channel_tag_name': [one.name, ],
'download_order': 'newest',
'maximum_duration': 600,
'minimum_duration': 60,
'title_exclude': 'foo,bar',
'title_include': 'baz,qux',
'video_count_limit': 100,
'video_resolutions': ['1080p', '720p', '480p', '360p', 'maximum'],
'video_format': 'mp4',
}
}
request, response = await async_client.post('/api/download', json=download)
assert response.status == HTTPStatus.CREATED, request.body
download = test_session.query(Download).one()
assert download.url == 'https://www.youtube.com/@wrolpi/videos'
assert download.destination is None
assert download.tag_names == ['one', 'two']
assert download.downloader == 'video_channel'
assert download.sub_downloader == 'video'
assert download.frequency == 2592000
assert download.settings
assert download.settings['channel_tag_name'] == ['one']
assert download.settings['download_order'] == 'newest'
assert download.settings['maximum_duration'] == 600
assert download.settings['minimum_duration'] == 60
assert download.settings['title_exclude'] == 'foo,bar'
assert download.settings['title_include'] == 'baz,qux'
assert download.settings['video_count_limit'] == 100
assert download.settings['video_resolutions'] == ['1080p', '720p', '480p', '360p', 'maximum']
assert download.settings['video_format'] == 'mp4'
def test_preview_filename(test_directory, fake_now):
assert preview_filename('%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s') \
== 'WROLPi_20000101_Qz-FuenRylQ_The title of the video.mp4'
assert preview_filename('%(upload_date)s_%(id)s_%(title)s.%(ext)s') \
== '20000101_Qz-FuenRylQ_The title of the video.mp4'
assert preview_filename('%(upload_date)s_%(title)s.%(ext)s') \
== '20000101_The title of the video.mp4'
assert preview_filename('%(upload_date)s_%(description)s.%(ext)s') \
== '20000101_A description of the video.mp4'
assert preview_filename('%(timestamp)s_%(title)s.%(ext)s') \
== '946684800_The title of the video.mp4'
with pytest.raises(RuntimeError):
preview_filename('%(upload_date)s_%(title)s.asdf')
def test_convert_wrolpi_filename_format():
"""WROLPi filename variables are converted to yt-dlp syntax."""
from modules.videos.downloader import convert_wrolpi_filename_format
# Test upload_year conversion
assert convert_wrolpi_filename_format('%(upload_year)s/%(title)s.%(ext)s') \
== '%(upload_date>%Y)s/%(title)s.%(ext)s'
# Test upload_month conversion
assert convert_wrolpi_filename_format('%(upload_month)s/%(title)s.%(ext)s') \
== '%(upload_date>%m)s/%(title)s.%(ext)s'
# Test both together
assert convert_wrolpi_filename_format('%(upload_year)s/%(upload_month)s/%(title)s.%(ext)s') \
== '%(upload_date>%Y)s/%(upload_date>%m)s/%(title)s.%(ext)s'
# Test with other variables unchanged
format_str = '%(upload_year)s/%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s'
expected = '%(upload_date>%Y)s/%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s'
assert convert_wrolpi_filename_format(format_str) == expected
# Test format without WROLPi variables unchanged
original = '%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s'
assert convert_wrolpi_filename_format(original) == original
def test_preview_filename_with_upload_year(test_directory, fake_now):
"""preview_filename works with upload_year and upload_month variables."""
# Year only
assert preview_filename('%(upload_year)s/%(title)s.%(ext)s') \
== '2000/The title of the video.mp4'
# Month only
assert preview_filename('%(upload_month)s/%(title)s.%(ext)s') \
== '01/The title of the video.mp4'
# Year and month together
assert preview_filename('%(upload_year)s/%(upload_month)s/%(title)s.%(ext)s') \
== '2000/01/The title of the video.mp4'
# Combined with traditional format
assert preview_filename('%(upload_year)s/%(uploader)s_%(upload_date)s_%(title)s.%(ext)s') \
== '2000/WROLPi_20000101_The title of the video.mp4'
def test_preview_filename_matches_youtubedl(test_directory, fake_now):
"""Ensure our lightweight preview_filename() matches YoutubeDL's actual output.
This test validates that our simple string formatting produces identical
results to YoutubeDL's prepare_filename(). If yt-dlp changes behavior,
this test will catch it.
"""
from copy import deepcopy
from yt_dlp import YoutubeDL
from wrolpi.dates import now
test_formats = [
'%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s',
'%(upload_date)s_%(id)s_%(title)s.%(ext)s',
'%(upload_year)s/%(upload_month)s/%(title)s.%(ext)s',
'%(upload_year)s/%(uploader)s_%(upload_date)s_%(title)s.%(ext)s',
'%(channel)s/%(id)s.%(ext)s',
]
# Sample entry used by both methods - must match preview_filename's variables
sample_date = now()
entry = dict(
uploader='WROLPi',
channel='WROLPi',
timestamp=int(sample_date.timestamp()),
upload_date=sample_date.strftime('%Y%m%d'),
id='Qz-FuenRylQ',
title='The title of the video',
ext='mp4',
description='A description of the video',
)
for fmt in test_formats:
# Get result from our lightweight implementation
lightweight_result = preview_filename(fmt)
# Get result from actual YoutubeDL
converted_format = convert_wrolpi_filename_format(fmt)
options = get_videos_downloader_config().yt_dlp_options
options = deepcopy(options)
options['outtmpl'] = converted_format
ydl = YoutubeDL(options)
ytdlp_result = prepare_filename(entry, ydl=ydl).lstrip('/')
assert lightweight_result == ytdlp_result, \
f"Mismatch for format '{fmt}':\n lightweight: {lightweight_result}\n ytdlp: {ytdlp_result}"
def test_normalize_video_file_names(test_directory, video_download_manager):
"""VideoDownloader renames caption files that do not share the video file's stem."""
downloader = VideoDownloader()
mp4 = test_directory / 'video.mp4'
info_json = test_directory / 'video.info.json'
# Minimal video files.
vtt = test_directory / 'video.vtt'
assert set(downloader.normalize_video_file_names(mp4, [mp4, vtt, info_json])) == {mp4, vtt, info_json}
# Normal video files.
vtt = test_directory / 'video.en.vtt'
assert set(downloader.normalize_video_file_names(mp4, [mp4, vtt, info_json])) == {mp4, vtt, info_json}
# Rare -auto file.
vtt = test_directory / 'video.en-auto.vtt'
assert set(downloader.normalize_video_file_names(mp4, [mp4, vtt, info_json])) == {mp4, vtt, info_json}
# Extra characters get removed from the vtt file.
vtt = test_directory / 'video.en-uYU-mmqFLq8.vtt'
vtt.touch() # must exist so it can be renamed.
assert set(downloader.normalize_video_file_names(mp4, [mp4, vtt, info_json])) == {
mp4,
(test_directory / 'video.en.vtt'),
info_json,
}
assert not vtt.exists()
# VTT suffix that does not match expected pattern is left unchanged.
vtt = test_directory / 'video.unknown_format.vtt'
vtt.touch()
assert set(downloader.normalize_video_file_names(mp4, [mp4, vtt, info_json])) == {mp4, vtt, info_json}
assert vtt.exists()
@pytest.mark.asyncio
async def test_channel_downloader_sets_collection_id(test_session, channel_factory, mock_video_extract_info,
video_download_manager, await_switches):
"""Test that ChannelDownloader.do_download sets download.collection_id to the channel's collection.
When a channel download runs, it should link the download to the channel's collection immediately.
"""
channel = channel_factory(url='https://example.com/channel', source_id='UCtest123')
download = Download(
url='https://example.com/channel',
downloader=channel_downloader.name,
frequency=99,
)
test_session.add(download)
test_session.commit()
assert not download.collection_id, 'Download should not be linked yet'
# Mock extract_info to return channel info
mock_info = {
'id': 'UCtest123',
'channel_id': 'UCtest123',
'uploader': channel.name,
'entries': [],
'webpage_url_basename': 'videos',
}
mock_video_extract_info.return_value = mock_info
# Drive the phase-split API directly — no dispatch loop fires, so child video
# downloads are never spawned and don't need stubbing. refresh_files and
# file_worker.wait_for_job are still patched because ChannelDownloader's
# prepare_channel_for_downloads calls them inline.
with mock.patch.object(Channel, 'refresh_files', return_value='test-job-id'), \
mock.patch('modules.videos.downloader.file_worker.wait_for_job', new_callable=mock.AsyncMock):
from modules.archive.conftest import make_test_ctx
prepared = channel_downloader.prepare_download(test_session, download)
executed = await channel_downloader.execute_download(prepared, make_test_ctx(), download=download)
channel_downloader.finalize_download(test_session, download, executed)
await await_switches()
test_session.refresh(download)
assert download.collection_id == channel.collection_id, \
'ChannelDownloader should link download to channel collection'
@pytest.mark.asyncio
async def test_channel_passes_all_inheritable_settings_to_video(test_session, channel_factory,
mock_video_extract_info,
video_download_manager, await_switches):
"""ChannelDownloader passes all CHANNEL_INHERITABLE_SETTINGS to child video downloads."""
from modules.videos.lib import CHANNEL_INHERITABLE_SETTINGS
channel = channel_factory(url='https://example.com/channel', source_id='UCtest456')
channel_settings = {
'video_resolutions': ['720p'],
'video_format': 'mkv',
'writesubtitles': False,
'sleep_requests': 3.0,
'user_agent': 'TestBot/1.0',
'video_codecs': ['h264'],
'audio_codecs': ['aac'],
'transcode': True,
'strict_codecs': True,
}
download = Download(
url='https://example.com/channel',
downloader=channel_downloader.name,
frequency=99,
settings=channel_settings,
)
test_session.add(download)
test_session.commit()
mock_info = {
'id': 'UCtest456',
'channel_id': 'UCtest456',
'uploader': channel.name,
'entries': [{'id': 'vid1', 'url': 'https://youtube.com/watch?v=vid1', 'title': 'Video 1'}],
'webpage_url_basename': 'videos',
}
mock_video_extract_info.return_value = mock_info
# Direct phase-split call — child video downloads are never dispatched here.
with mock.patch.object(Channel, 'refresh_files', return_value='test-job-id'), \
mock.patch('modules.videos.downloader.file_worker.wait_for_job', new_callable=mock.AsyncMock):
from modules.archive.conftest import make_test_ctx
prepared = channel_downloader.prepare_download(test_session, download)
executed = await channel_downloader.execute_download(prepared, make_test_ctx(), download=download)
result = channel_downloader.finalize_download(test_session, download, executed)
# Verify all channel-level inheritable settings are passed through
for key in ('video_resolutions', 'video_format', 'writesubtitles', 'sleep_requests', 'user_agent',
'video_codecs', 'audio_codecs', 'transcode', 'strict_codecs'):
assert key in result.settings, f'{key} missing from DownloadResult.settings'
assert result.settings[key] == channel_settings[key], \
f'{key}: expected {channel_settings[key]}, got {result.settings[key]}'
@pytest.mark.asyncio
async def test_channel_does_not_pass_continue_dl_nooverwrites(test_session, channel_factory,
mock_video_extract_info,
video_download_manager, await_switches):
"""continue_dl and nooverwrites should NOT be passed from channel to video."""
channel = channel_factory(url='https://example.com/channel', source_id='UCtest789')
channel_settings = {
'continue_dl': False,
'nooverwrites': False,
'video_format': 'mkv',
}
download = Download(
url='https://example.com/channel',
downloader=channel_downloader.name,
frequency=99,
settings=channel_settings,
)
test_session.add(download)
test_session.commit()
mock_info = {
'id': 'UCtest789',
'channel_id': 'UCtest789',
'uploader': channel.name,
'entries': [{'id': 'vid2', 'url': 'https://youtube.com/watch?v=vid2', 'title': 'Video 2'}],
'webpage_url_basename': 'videos',
}
mock_video_extract_info.return_value = mock_info
# Direct phase-split call — child video downloads are never dispatched here.
with mock.patch.object(Channel, 'refresh_files', return_value='test-job-id'), \
mock.patch('modules.videos.downloader.file_worker.wait_for_job', new_callable=mock.AsyncMock):
from modules.archive.conftest import make_test_ctx
prepared = channel_downloader.prepare_download(test_session, download)
executed = await channel_downloader.execute_download(prepared, make_test_ctx(), download=download)
result = channel_downloader.finalize_download(test_session, download, executed)
assert 'continue_dl' not in result.settings, 'continue_dl should not be passed to child video downloads'
assert 'nooverwrites' not in result.settings, 'nooverwrites should not be passed to child video downloads'
assert result.settings['video_format'] == 'mkv', 'video_format should be passed through'
@pytest.mark.asyncio
async def test_video_download_uses_effective_settings(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
image_file, simple_channel, test_videos_downloader_config):
"""VideoDownloader uses merged effective settings from global + download overrides."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
image_file.rename(video_path.with_suffix('.jpg'))
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
# Download with writesubtitles=False override (global default is True)
settings = {'writesubtitles': False, 'sleep_requests': 5.0}
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name, settings=settings)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
download, cmd, out_dir = mock_video_process_runner.call_args[0]
# writesubtitles=False means no --write-subs in command
assert '--write-subs' not in cmd, 'writesubtitles=False should prevent --write-subs'
# sleep_requests=5.0 should appear
assert '--sleep-requests' in cmd, 'sleep_requests override should appear in command'
sleep_idx = cmd.index('--sleep-requests')
assert cmd[sleep_idx + 1] == '5.0', f'Expected sleep_requests=5.0, got {cmd[sleep_idx + 1]}'
@pytest.mark.parametrize('video_resolutions', [
['1080p', '720p', '480p', 'maximum'], # The default.
['480p'],
['2160p', 'maximum'],
])
def test_format_selector_contains_no_comma(video_resolutions):
"""`format_selector` joins selectors with "/" so yt-dlp downloads only the first match.
A "," would tell yt-dlp to download *every* matching selector, which downloads and
post-processes the same video once per match.
"""
selector = format_selector(video_resolutions)
assert ',' not in selector, f'A comma makes yt-dlp download every matching format: {selector}'
# Every selector of every requested resolution is present, in order, as a "/" fallback.
expected = [j for i in video_resolutions for j in VIDEO_RESOLUTION_MAP[i]]
assert selector == '/'.join(expected)
@pytest.mark.asyncio
async def test_video_download_requests_one_format(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
image_file, simple_channel, test_videos_downloader_config):
"""The `-f` passed to yt-dlp is a fallback chain, so only one format is downloaded."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
image_file.rename(video_path.with_suffix('.jpg'))
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
download, cmd, out_dir = mock_video_process_runner.call_args[0]
assert '-f' in cmd, 'A video download must request formats'
format_arg = cmd[cmd.index('-f') + 1]
assert ',' not in format_arg, \
f'A comma makes yt-dlp download the video once per matching format: {format_arg}'
# The multiple resolutions the config requests are all still offered, as fallbacks.
assert format_arg == format_selector(get_videos_downloader_config().video_resolutions)
@pytest.mark.asyncio
async def test_live_video_retry_seconds(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
simple_channel):
"""A live video download should return retry_seconds=3600 so it retries in 1 hour."""
from wrolpi.cmd import CommandResult
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
# Simulate yt-dlp output when a video is live.
mock_video_process_runner.return_value = CommandResult(
return_code=0, cancelled=False,
stdout=b'does not pass filter (!is_live), skipping ..', stderr=b'', elapsed=0,
)
url = 'https://www.youtube.com/watch?v=live_video'
mock_video_extract_info.return_value = example_video_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
# Return a path that won't exist (video wasn't downloaded because it's live).
mock_prepare_filename.return_value = (simple_channel.directory / 'live video.mp4', {'id': 'foo'})
download = Download(url=url, downloader='video', attempts=0)
# Drive the phase-split API directly: when execute_download detects a live video
# it short-circuits with a DownloadResult (the dispatch surfaces it verbatim and
# skips finalize_download).
from modules.archive.conftest import make_test_ctx
prepared = video_downloader.prepare_download(test_session, download)
result = await video_downloader.execute_download(prepared, make_test_ctx(), download=download)
assert result.success is False
assert result.retry_seconds == 3600
assert 'live' in result.error.lower()
def test_bot_blocked_detection():
"""Bot detection errors should be identified by _bot_blocked."""
assert _bot_blocked("Sign in to confirm you're not a bot") is True
assert _bot_blocked("Sign in to confirm you\u2019re not a bot") is True
assert _bot_blocked("cookies are no longer valid") is True
assert _bot_blocked("ERROR: Sign in to confirm you're not a bot. Use --cookies") is True
# These should NOT be detected as bot-blocked.
assert _bot_blocked("404: Not Found") is False
assert _bot_blocked("requires payment") is False
assert _bot_blocked("some random error") is False
def test_skip_download_excludes_bot_blocked():
"""Bot-blocked errors should NOT cause a video to be skipped (added to skip list)."""
# Bot-blocked errors should not be skipped (they should stop all downloads instead).
assert _skip_download("Sign in to confirm you're not a bot") is False
assert _skip_download("cookies are no longer valid") is False
# Regular "Sign in" errors that are NOT bot-related should still be skipped.
assert _skip_download("Sign in to view this video") is True
# Other unrecoverable errors should still be skipped.
assert _skip_download("404: Not Found") is True
assert _skip_download("requires payment") is True
@pytest.mark.asyncio
async def test_video_download_playlist_appended_extension(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
simple_channel, image_file):
"""A video download from a playlist (e.g. Twitter/X) where yt-dlp appends the real extension to a bogus .NA
extension should still find the file."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
# Simulate yt-dlp saving a file with .NA.mp4 (appended extension from playlist entry with ext=NA).
video_path_na = simple_channel.directory / 'uploader name_20190707_some long id_The video title.NA.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path_na)
url = 'https://x.com/example/status/123'
mock_video_extract_info.return_value = example_video_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
# prepare_filename returns the path without the real extension (ending in .NA).
bogus_path = video_path_na.parent / video_path_na.stem # Removes .mp4, leaving .NA as "extension"
mock_prepare_filename.return_value = (bogus_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
video: Video = test_session.query(Video).one()
assert video.video_path == video_path_na
def test_parse_ytdlp_progress():
"""yt-dlp progress lines are parsed correctly."""
line = '[download] 50.0% of ~ 45.67MiB at 2.34MiB/s ETA 00:19'
result = parse_ytdlp_progress(line)
assert result is not None
total = int(45.67 * 1024 ** 2)
assert result['total_bytes'] == total
assert result['percent'] == 50
assert result['bytes_downloaded'] == int(0.5 * total)
assert result['speed'] == int(2.34 * 1024 ** 2)
assert result['eta'] == '00:19'
def test_parse_ytdlp_progress_no_estimated():
"""yt-dlp progress lines without ~ estimated marker are parsed."""
line = '[download] 100% of 45.67MiB at 3.45MiB/s ETA 00:00'
result = parse_ytdlp_progress(line)
assert result is not None
assert result['percent'] == 100
assert result['total_bytes'] == int(45.67 * 1024 ** 2)
assert result['speed'] == int(3.45 * 1024 ** 2)
assert result['eta'] == '00:00'
def test_parse_ytdlp_progress_small_percent():
"""yt-dlp progress lines with small percentages are parsed."""
line = '[download] 0.1% of ~ 45.67MiB at 1.23MiB/s ETA 00:37'
result = parse_ytdlp_progress(line)
assert result is not None
assert result['percent'] == 0
assert result['eta'] == '00:37'
def test_parse_ytdlp_progress_completion_line():
"""The final yt-dlp line with 'in' instead of 'at/ETA' returns None (no speed/ETA)."""
line = '[download] 100% of 45.67MiB in 00:13'
result = parse_ytdlp_progress(line)
assert result is None
def test_parse_ytdlp_progress_not_progress_line():
"""Non-progress lines return None."""
assert parse_ytdlp_progress('[info] Downloading video 1 of 1') is None
assert parse_ytdlp_progress('') is None
assert parse_ytdlp_progress('[download] Destination: video.mp4') is None
@pytest.mark.asyncio
async def test_fetch_video_duration_persistent_cache(test_download_cache_config):
"""Video durations are persisted to the download cache config and reused on subsequent calls."""
from modules.videos.downloader import _cache_video_duration
from wrolpi.downloader import get_download_cache_config
url = 'https://www.youtube.com/watch?v=test123'
# Verify cache is initially empty.
assert get_download_cache_config().video_durations == []
# Persist a duration directly.
_cache_video_duration(url, 120)
assert get_download_cache_config().video_durations == [[url, 120]]
# Re-caching the same URL updates the entry (no duplicate).
_cache_video_duration(url, 200)
assert get_download_cache_config().video_durations == [[url, 200]]
@pytest.mark.asyncio
async def test_fetch_video_duration_cache_eviction(test_download_cache_config):
"""Video duration cache evicts oldest entries when over the max size."""
from modules.videos.downloader import _cache_video_duration, VIDEO_DURATION_CACHE_MAX_SIZE
from wrolpi.downloader import get_download_cache_config
# Fill to max size.
for i in range(VIDEO_DURATION_CACHE_MAX_SIZE):
_cache_video_duration(f'https://example.com/video{i}', i)
assert len(get_download_cache_config().video_durations) == VIDEO_DURATION_CACHE_MAX_SIZE
# Adding one more should evict the oldest (video0), keeping it at max size.
_cache_video_duration('https://example.com/new_video', 999)
durations = get_download_cache_config().video_durations
assert len(durations) == VIDEO_DURATION_CACHE_MAX_SIZE
# Newest entry is at the end.
assert durations[-1] == ['https://example.com/new_video', 999]
# Oldest entry (video0) was evicted.
durations_dict = dict(durations)
assert 'https://example.com/video0' not in durations_dict
# Second oldest (video1) is still present.
assert durations_dict['https://example.com/video1'] == 1
@pytest.mark.asyncio
async def test_channel_download_preserves_skip_list(test_session, test_directory, simple_channel,
video_download_manager, video_file,
mock_video_extract_info, mock_video_prepare_filename,
await_switches, mock_video_process_runner,
test_download_manager_config):
"""A skipped video URL should remain in the skip list when its channel downloads again.
Regression: child video downloads created by ChannelDownloader used reset_attempts=True which
caused create_downloads to remove the URL from the skip list."""
skipped_url = 'https://youtube.com/watch?v=video_1_url'
url = 'https://www.youtube.com/c/ExampleChannel/videos'
# Skip video_1_url — simulating user skipping a video.
get_download_manager_config().skip_urls = [skipped_url]
assert get_download_manager_config().skip_urls == [skipped_url]
mock_video_prepare_filename.return_value = video_file
mock_video_extract_info.return_value = example_channel_json
with mock.patch('modules.videos.downloader.get_channel') as mock_get_channel:
mock_get_channel.return_value = simple_channel
simple_channel.get_or_create_download(test_session, url, 60)
# Download channel catalog.
await video_download_manager.wait_for_all_downloads()
# Download the child videos.
await video_download_manager.wait_for_all_downloads()
# The skipped URL must still be in the skip list.
assert skipped_url in get_download_manager_config().skip_urls, \
'Channel download removed a URL from the skip list!'
# Only video_2 should have been downloaded, not the skipped video_1.
once_downloads = video_download_manager.get_once_downloads(test_session)
video_downloads = [i for i in once_downloads if 'watch' in i.url]
assert {i.url for i in video_downloads} == {'https://youtube.com/watch?v=video_2_url'}
# ---------------------------------------------------------------------------
# Phase-split unit tests for VideoDownloader.prepare_download.
#
# These exercise the sync prep invariants in isolation — no async_client, no
# extract_info, no subprocess. Heavier execute/finalize behavior is covered by
# the integration tests above (which drive through video_download_manager).
# ---------------------------------------------------------------------------
from modules.videos.downloader import PreparedVideo # noqa: E402
def test_prepare_max_attempts_raises(test_session):
"""prepare_download enforces the 10-attempt cap (videos retry more than other
downloaders because YouTube transients are common)."""
download = Download(url='https://youtube.com/watch?v=foo', downloader='video', attempts=10)
with pytest.raises(UnrecoverableDownloadError, match='Max download attempts'):
video_downloader.prepare_download(test_session, download)
def test_prepare_normalizes_url(test_session):
"""Shorts URLs become watch URLs in the PreparedVideo (matches DB-stored format)."""
download = Download(
url='https://www.youtube.com/shorts/ABC123',
downloader='video',
settings={},
)
prepared = video_downloader.prepare_download(test_session, download)
assert isinstance(prepared, PreparedVideo)
assert prepared.url == 'https://www.youtube.com/watch?v=ABC123'
def test_prepare_copies_upstream_settings_onto_download(test_session):
"""ChannelDownloader passes destination + tag_names via settings; prep copies them
onto the Download row so downstream code sees them as columns."""
download = Download(
url='https://youtube.com/watch?v=foo',
downloader='video',
destination=None,
tag_names=None,
settings={'destination': '/media/wrolpi/videos/foo', 'tag_names': ['news', 'tech']},
)
prepared = video_downloader.prepare_download(test_session, download)
assert download.destination == '/media/wrolpi/videos/foo'
assert download.tag_names == ['news', 'tech']
assert prepared.tag_names == ['news', 'tech']
def test_prepare_uses_existing_video_location(test_session, video_factory):
"""If a Video already exists at the URL, prep captures its location for error
attribution (so a failed re-download still tells the user where the previous file is)."""
url = 'https://youtube.com/watch?v=existing'
existing = video_factory(with_video_file=True)
existing.file_group.url = url
test_session.commit()
download = Download(url=url, downloader='video', settings={})
prepared = video_downloader.prepare_download(test_session, download)
assert prepared.location == existing.location
assert prepared.location is not None
@pytest.mark.parametrize('url,expected', [
('https://www.youtube.com/watch?v=abc123', True),
('https://youtube.com/watch?v=abc123', True),
('https://m.youtube.com/watch?v=abc123', True),
('https://music.youtube.com/watch?v=abc123', True),
('https://youtu.be/abc123', True),
# Lookalike / malicious hosts must be rejected.
('https://youtube.com.example.org/watch?v=abc123', False),
('https://notyoutube.com/watch?v=abc123', False),
('https://vimeo.com/12345', False),
('https://example.com/youtube.com', False),
('', False),
(None, False),
])
def test_is_youtube_url(url, expected):
"""is_youtube_url matches YouTube hosts (and subdomains) but rejects lookalike hosts."""
assert is_youtube_url(url) is expected
def test_format_selector_with_codecs():
"""Codec preferences generate codec-major selectors: a preferred codec at a lower resolution
beats a non-preferred codec at a higher resolution. The legacy resolution selectors are
appended as a fallback unless strict."""
from modules.videos.downloader import VIDEO_CODEC_YT_DLP_PATTERNS, AUDIO_CODEC_YT_DLP_PATTERNS
h264 = VIDEO_CODEC_YT_DLP_PATTERNS['h264']
aac = AUDIO_CODEC_YT_DLP_PATTERNS['aac']
legacy = format_selector(['1080p', '720p'])
selector = format_selector(['1080p', '720p'], video_codecs=['h264'], audio_codecs=['aac'])
expected = '/'.join([
f"bestvideo[height=1080][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[height=1080][vcodec~='{h264}']+bestaudio",
f"bestvideo[width=1080][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[width=1080][vcodec~='{h264}']+bestaudio",
f"bestvideo[height=720][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[height=720][vcodec~='{h264}']+bestaudio",
f"bestvideo[width=720][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[width=720][vcodec~='{h264}']+bestaudio",
legacy,
])
assert selector == expected
assert ',' not in selector
# Strict drops the any-codec audio fallbacks and the legacy tail.
strict = format_selector(['1080p', '720p'], video_codecs=['h264'], audio_codecs=['aac'],
strict_codecs=True)
expected_strict = '/'.join([
f"bestvideo[height=1080][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[width=1080][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[height=720][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
f"bestvideo[width=720][vcodec~='{h264}']+bestaudio[acodec~='{aac}']",
])
assert strict == expected_strict
assert ',' not in strict
# Codec-major: every h264 selector comes before the first vp9 selector.
vp9 = VIDEO_CODEC_YT_DLP_PATTERNS['vp9']
two = format_selector(['1080p'], video_codecs=['h264', 'vp9'], strict_codecs=True)
assert two.rindex(h264) < two.index(vp9)
assert two.count('bestvideo') == 4 # (height + width) x 2 codecs, one audio selector each.
# No codecs at all: byte-identical to the legacy selector.
assert format_selector(['1080p', '720p']) == legacy
# Only audio codecs: video is unfiltered but audio is preferred.
audio_only_pref = format_selector(['720p'], audio_codecs=['opus'])
assert "bestvideo[height=720]+bestaudio[acodec~='^opus']" in audio_only_pref
assert audio_only_pref.endswith(format_selector(['720p']))
# 'maximum' has no dimension filter.
maximum = format_selector(['maximum'], video_codecs=['h264'], strict_codecs=True)
assert maximum == f"bestvideo[vcodec~='{h264}']+bestaudio"
def test_format_selector_codec_patterns_are_safe():
"""Codec regex patterns are embedded in `-f` bracket filters; a comma would make yt-dlp
download every matching format, and a slash would break the fallback chain."""
from modules.videos.downloader import VIDEO_CODEC_YT_DLP_PATTERNS, AUDIO_CODEC_YT_DLP_PATTERNS
for pattern in (*VIDEO_CODEC_YT_DLP_PATTERNS.values(), *AUDIO_CODEC_YT_DLP_PATTERNS.values()):
assert ',' not in pattern
assert '/' not in pattern
@pytest.mark.asyncio
async def test_video_download_requests_codec_format(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
image_file, simple_channel, test_videos_downloader_config):
"""Codec settings change the `-f` selector; the legacy selectors remain as fallbacks."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
image_file.rename(video_path.with_suffix('.jpg'))
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
settings = {'video_codecs': ['h264'], 'audio_codecs': ['aac']}
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name, settings=settings)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
download, cmd, out_dir = mock_video_process_runner.call_args[0]
format_arg = cmd[cmd.index('-f') + 1]
resolutions = get_videos_downloader_config().video_resolutions
assert format_arg == format_selector(resolutions, video_codecs=['h264'], audio_codecs=['aac'])
assert "vcodec~=" in format_arg
# The legacy selectors are still offered as a fallback (strict is off).
assert format_arg.endswith(format_selector(resolutions))
@pytest.mark.asyncio
async def test_strict_selector_when_transcode_cannot_satisfy(test_session, test_directory,
mock_video_extract_info,
video_download_manager,
mock_video_process_runner, image_file,
simple_channel,
test_videos_downloader_config):
"""Transcode suppresses strict only when it can produce a preferred codec; av1 has no
transcode target, so strict still removes the fallback selectors."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
image_file.rename(video_path.with_suffix('.jpg'))
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
settings = {'video_codecs': ['av1'], 'strict_codecs': True, 'transcode': True}
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename, \
mock.patch('modules.videos.downloader.enforce_codecs', new_callable=mock.AsyncMock) as mock_enforce:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
mock_enforce.side_effect = lambda path, paths, effective: (path, paths)
video_download_manager.create_download(test_session, url, video_downloader.name, settings=settings)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
download, cmd, out_dir = mock_video_process_runner.call_args[0]
format_arg = cmd[cmd.index('-f') + 1]
resolutions = get_videos_downloader_config().video_resolutions
assert format_arg == format_selector(resolutions, video_codecs=['av1'], strict_codecs=True)
# No fallback to the legacy selectors.
assert not format_arg.endswith(format_selector(resolutions))
@pytest.mark.asyncio
async def test_strict_codecs_unavailable_fails_download(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
image_file, simple_channel,
test_videos_downloader_config):
"""With strict_codecs, yt-dlp's "Requested format is not available" fails the download
permanently (the source will never grow new formats)."""
from wrolpi.cmd import CommandResult
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
settings = {'video_codecs': ['h264'], 'strict_codecs': True}
mock_video_process_runner.return_value = CommandResult(
return_code=1, cancelled=False, stdout=b'',
stderr=b'ERROR: [youtube] 31jPEBiAC3c: Requested format is not available.', elapsed=0)
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name, settings=settings)
await video_download_manager.wait_for_all_downloads()
download = test_session.query(Download).one()
assert download.is_failed, f'Download should have failed permanently, got {download.status}'
assert 'transcoding cannot produce them' in download.error
@pytest.mark.asyncio
async def test_codecs_unavailable_defers_without_strict(test_session, test_directory, mock_video_extract_info,
video_download_manager, mock_video_process_runner,
image_file, simple_channel,
test_videos_downloader_config):
"""The same yt-dlp failure without strict_codecs is deferred (retried), as before."""
from wrolpi.cmd import CommandResult
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_path = simple_channel.directory / 'a video.mp4'
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
mock_video_process_runner.return_value = CommandResult(
return_code=1, cancelled=False, stdout=b'',
stderr=b'ERROR: [youtube] 31jPEBiAC3c: Requested format is not available.', elapsed=0)
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name,
settings={'video_codecs': ['h264']})
await video_download_manager.wait_for_all_downloads()
download = test_session.query(Download).one()
assert download.is_deferred, f'Download should defer without strict_codecs, got {download.status}'
@pytest.mark.asyncio
async def test_enforce_codecs():
"""enforce_codecs transcodes, fails, or keeps the file depending on the effective settings."""
import pathlib
from modules.videos.downloader import enforce_codecs
video_path = pathlib.Path('/tmp/video.webm')
poster_path = pathlib.Path('/tmp/video.jpg')
video_paths = [video_path, poster_path]
vp9_ffprobe = {'streams': [
{'codec_type': 'video', 'codec_name': 'vp9'},
{'codec_type': 'audio', 'codec_name': 'opus'},
]}
with mock.patch('modules.videos.downloader.get_or_create_ffprobe_json',
new_callable=mock.AsyncMock) as mock_ffprobe, \
mock.patch('modules.videos.downloader.transcode_video_file',
new_callable=mock.AsyncMock) as mock_transcode:
mock_ffprobe.return_value = (vp9_ffprobe, None)
# Codecs match: nothing happens.
effective = {'video_codecs': ['vp9'], 'audio_codecs': ['opus']}
result = await enforce_codecs(video_path, video_paths, effective)
assert result == (video_path, video_paths)
mock_transcode.assert_not_called()
# Mismatch with neither transcode nor strict: file is kept.
effective = {'video_codecs': ['h264']}
result = await enforce_codecs(video_path, video_paths, effective)
assert result == (video_path, video_paths)
mock_transcode.assert_not_called()
# Mismatch with strict: permanent failure.
effective = {'video_codecs': ['h264'], 'strict_codecs': True}
with pytest.raises(UnrecoverableDownloadError):
await enforce_codecs(video_path, video_paths, effective)
mock_transcode.assert_not_called()
# Mismatch with transcode: the file is transcoded and the paths updated.
new_path = video_path.with_suffix('.mp4')
mock_transcode.return_value = new_path
effective = {'video_codecs': ['h264'], 'audio_codecs': ['aac'], 'transcode': True,
'video_format': 'mp4'}
result = await enforce_codecs(video_path, video_paths, effective)
assert result == (new_path, [new_path, poster_path])
mock_transcode.assert_called_once_with(video_path, 'h264', 'aac', container='mp4')
# Transcode wins over strict when it can produce the preferred codec.
mock_transcode.reset_mock()
mock_transcode.return_value = new_path
effective = {'video_codecs': ['h264'], 'transcode': True, 'strict_codecs': True,
'video_format': 'mp4'}
result = await enforce_codecs(video_path, video_paths, effective)
assert result[0] == new_path
mock_transcode.assert_called_once()
# No supported transcode target: file is kept, no error.
mock_transcode.reset_mock()
effective = {'video_codecs': ['av1'], 'transcode': True}
result = await enforce_codecs(video_path, video_paths, effective)
assert result == (video_path, video_paths)
mock_transcode.assert_not_called()
# Regression: strict must hold when transcode cannot produce the preferred codec.
effective = {'video_codecs': ['av1'], 'transcode': True, 'strict_codecs': True}
with pytest.raises(UnrecoverableDownloadError):
await enforce_codecs(video_path, video_paths, effective)
mock_transcode.assert_not_called()
# No partial transcode: an unfixable video mismatch (av1) blocks the audio-only
# conversion; the file is kept as-is (or fails when strict).
effective = {'video_codecs': ['av1'], 'audio_codecs': ['aac'], 'transcode': True}
result = await enforce_codecs(video_path, video_paths, effective)
assert result == (video_path, video_paths)
mock_transcode.assert_not_called()
effective = {'video_codecs': ['av1'], 'audio_codecs': ['aac'], 'transcode': True,
'strict_codecs': True}
with pytest.raises(UnrecoverableDownloadError):
await enforce_codecs(video_path, video_paths, effective)
mock_transcode.assert_not_called()
@pytest.mark.asyncio
async def test_metadata_refresh_keeps_wrolpi_section(test_session, test_directory, video_factory):
"""Refreshing a Video's metadata replaces yt-dlp's info json; the wrolpi section (the user's
custom title and description) is carried over and still wins."""
source_id = 'video-source-id'
url = 'https://example.com/the-video'
video = video_factory(with_info_json={'id': source_id, 'title': 'Orig',
'wrolpi': {'custom_title': 'Mine', 'custom_description': 'my words'}},
source_id=source_id)
video.file_group.url = url
test_session.commit()
assert video.file_group.title == 'Mine'
download = Download(url=url, info_json={'id': source_id}, settings={})
with mock.patch('modules.videos.downloader.download_video_info_json') as mock_download:
mock_download.return_value = {'id': source_id, 'title': 'Orig v2', 'description': 'theirs', 'duration': 9}
result = await VideoDownloader.download_info_json(download, video.video_path)
assert result.success is True, result
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
info_json = video.get_info_json()
assert info_json['title'] == 'Orig v2', 'yt-dlp data was refreshed'
assert info_json['wrolpi'] == {'custom_title': 'Mine', 'custom_description': 'my words'}
assert video.file_group.title == 'Mine'
assert video.get_description() == 'my words'
@pytest.mark.asyncio
async def test_redownload_keeps_wrolpi_section(test_session, test_directory, mock_video_extract_info, simple_channel,
await_switches, video_download_manager, mock_video_process_runner,
image_file):
"""A full re-download has yt-dlp overwrite the info json; the wrolpi section is carried over."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
info_json_path = video_path.with_suffix('.info.json')
info_json_path.write_text(json.dumps({'title': 'Orig', 'webpage_url': url, 'wrolpi': {'custom_title': 'Mine'}}))
from modules.videos.normalize_video_url import normalize_video_url
existing = Video.from_paths(test_session, video_path, info_json_path)
existing.validate(test_session)
test_session.commit()
assert existing.file_group.title == 'Mine'
assert Video.get_by_url(test_session, normalize_video_url(url)) is not None, 'lookup by normalized url'
assert existing.file_group.get_wrolpi_json() == {'custom_title': 'Mine'}
from wrolpi.cmd import CommandResult
async def yt_dlp_overwrites_info_json(*args, **kwargs):
# yt-dlp rewrites the whole info json; the wrolpi section is gone from disk.
info_json_path.write_text(json.dumps({'title': 'Orig v2', 'webpage_url': url, 'duration': 5}))
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
mock_video_process_runner.side_effect = yt_dlp_overwrites_info_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
test_session.expire_all()
video: Video = test_session.query(Video).one()
info_json = video.get_info_json()
assert info_json['title'] == 'Orig v2'
assert info_json['wrolpi'] == {'custom_title': 'Mine'}
assert video.file_group.title == 'Mine'