wrolpi/modules/videos/test/test_common.py
lrnselfreliance 5358c75a2b
Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:25:23 -06:00

681 lines
28 KiB
Python

import pathlib
import subprocess
import tempfile
from copy import copy
from unittest import mock
import pytest
from PIL import Image
from modules.videos.models import Channel, Video
from wrolpi.collections import Collection
from wrolpi.conftest import probe_write_lock_is_held, production_like_sessions
from wrolpi.common import get_absolute_media_path, get_wrolpi_config, get_relative_to_media_directory
from wrolpi.downloader import Download, DownloadFrequency
from wrolpi.vars import PROJECT_DIR
from .. import common
from ..common import convert_image, update_view_counts_and_censored, extract_video_duration, generate_video_poster, \
is_valid_poster
from ..lib import save_channels_config, get_channels_config, import_channels_config, ChannelsConfig
def test_get_absolute_media_path(test_directory):
path = get_absolute_media_path('videos')
assert str(path) == f'{test_directory}/videos'
# An absolute path is not changed.
path = get_absolute_media_path(path)
assert str(path) == f'{test_directory}/videos'
def test_extract_video_duration(test_directory, corrupted_video_file):
"""
Video duration can be retrieved from the video file.
"""
video_path = PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4'
assert extract_video_duration(video_path) == 5
with pytest.raises(FileNotFoundError):
extract_video_duration(PROJECT_DIR / 'test/does not exist.mp4')
with pytest.raises(subprocess.CalledProcessError):
empty_file = test_directory / 'empty.mp4'
empty_file.touch()
extract_video_duration(empty_file)
with pytest.raises(subprocess.CalledProcessError):
extract_video_duration(corrupted_video_file)
def test_convert_image():
"""
An image's format can be changed. This tests that convert_image() converts from a WEBP format, to a JPEG format.
"""
foo = Image.new('RGB', (25, 25), color='grey')
with tempfile.TemporaryDirectory() as tempdir:
tempdir = pathlib.Path(tempdir)
# Save the new image to "foo.webp".
existing_path = tempdir / 'foo.webp'
foo.save(existing_path)
assert Image.open(existing_path).format == 'WEBP'
destination_path = tempdir / 'foo.jpg'
assert not destination_path.is_file()
# Convert the WEBP to a JPEG. The WEBP image should be removed.
convert_image(existing_path, destination_path)
assert not existing_path.is_file()
assert destination_path.is_file()
assert Image.open(destination_path).format == 'JPEG'
@pytest.mark.parametrize(
'ext,expected',
[
('jpg', True),
('jpeg', True),
('png', False),
('webp', False),
('tif', False),
]
)
def test_is_valid_poster(ext, expected, test_directory):
foo = Image.new('RGB', (25, 25), color='blue')
image_path = test_directory / f'image.{ext}'
foo.save(image_path)
assert is_valid_poster(image_path) == expected, f'is_valid_poster({image_path}) should be {expected}'
@pytest.mark.asyncio
async def test_update_view_count(test_session, channel_factory, video_factory):
def check_view_counts(view_counts):
for source_id, view_count in view_counts.items():
video = test_session.query(Video).filter_by(source_id=source_id).one()
assert video.view_count == view_count
channel1, channel2, channel3 = channel_factory(), channel_factory(), channel_factory()
video_factory(channel_id=channel1.id, with_poster_ext='jpg', title='vid1')
video_factory(channel_id=channel2.id, title='vid2')
video_factory(channel_id=channel3.id, with_poster_ext='jpg', title='vid3')
video_factory(channel_id=channel3.id, title='vid4')
channel1.info_json = {'entries': [{'id': 'vid1', 'view_count': 10}]}
channel2.info_json = {'entries': [{'id': 'vid2', 'view_count': 11}, {'id': 'bad_id', 'view_count': 12}]}
channel3.info_json = {'entries': [{'id': 'vid3', 'view_count': 13}, {'id': 'vid4', 'view_count': 14}]}
test_session.commit()
# Check all videos are empty.
check_view_counts({'vid1': None, 'vid2': None, 'vid3': None, 'vid4': None})
# Channel 1 is updated, the other channels are left alone.
await update_view_counts_and_censored(channel1.id)
check_view_counts({'vid1': 10, 'vid2': None, 'vid3': None, 'vid4': None})
# Channel 2 is updated, the other channels are left alone. The 'bad_id' video is ignored.
await update_view_counts_and_censored(channel2.id)
check_view_counts({'vid1': 10, 'vid2': 11, 'vid3': None, 'vid4': None})
# All videos are updated.
await update_view_counts_and_censored(channel3.id)
check_view_counts({'vid1': 10, 'vid2': 11, 'vid3': 13, 'vid4': 14})
# An outdated view count will be overwritten.
vid = test_session.query(Video).filter_by(id=1).one()
vid.view_count = 8
test_session.commit() # Commit so raw SQL in update_view_counts_and_censored doesn't deadlock
check_view_counts({'vid1': 8, 'vid2': 11, 'vid3': 13, 'vid4': 14})
await update_view_counts_and_censored(channel1.id)
test_session.expire_all() # Expire cached values to see raw SQL changes
check_view_counts({'vid1': 10, 'vid2': 11, 'vid3': 13, 'vid4': 14})
@pytest.mark.asyncio
async def test_update_view_count_missing_view_count(test_session, channel_factory, video_factory):
"""Entries without a view_count key (e.g. private/unavailable videos) are skipped without error."""
channel = channel_factory()
video_factory(channel_id=channel.id, with_poster_ext='jpg', title='vid1')
channel.info_json = {'entries': [
{'id': 'vid1', 'view_count': 42},
{'id': 'vid_private'}, # No view_count key.
{'id': 'vid_null', 'view_count': None},
]}
test_session.commit()
# Should not raise KeyError.
await update_view_counts_and_censored(channel.id)
test_session.expire_all()
video = test_session.query(Video).filter_by(source_id='vid1').one()
assert video.view_count == 42
def test_generate_video_poster(video_file):
"""
A poster can be generated from a video file.
"""
poster_path = video_file.with_suffix('.jpg')
_, duration = generate_video_poster(video_file)
assert poster_path.is_file(), f'{poster_path} was not created!'
assert poster_path.stat().st_size > 0
assert duration == 5
@pytest.mark.asyncio
async def test_import_channel_downloads(await_switches, test_session, channel_factory, test_channels_config,
tag_factory):
"""Importing the Channels' config should create any missing download records"""
channel1 = channel_factory(source_id='foo', url='https://example.com/channel1')
channel2 = channel_factory(source_id='bar', url='https://example.com/channel2')
tag = await tag_factory()
assert len(test_session.query(Channel).all()) == 2
test_session.commit()
def update_channel_config(conf: ChannelsConfig, source_id, d):
# Creat a copy of the old config, replace the data of the provided Channel.
old_config = conf._config.copy()
for c in old_config['channels']:
if c['source_id'] == source_id:
c.update(d)
conf.channels = old_config['channels']
conf.save()
# Two Channels, but neither have Downloads.
save_channels_config()
await await_switches()
# Import after config is saved.
import_channels_config()
assert not channel1.downloads
assert not channel2.downloads
assert len(test_session.query(Channel).all()) == 2
assert test_session.query(Download).count() == 0
# Add a frequency to `channel1`.
channels_config = get_channels_config()
update_channel_config(channels_config, 'foo',
{'downloads': [
{'url': 'https://example.com/channel1', 'frequency': DownloadFrequency.weekly.value}
]})
await await_switches()
# Download record is created on import.
import_channels_config()
assert channel1.downloads
assert len(test_session.query(Channel).all()) == 2, 'Both Channels should still exist'
assert test_session.query(Download).count() == 1, 'One Channel has a Download'
download: Download = test_session.query(Download).one()
assert download.url == channel1.url
assert download.frequency
assert download.downloader == 'video_channel'
# Download.next_download was not deleted.
next_download = str(download.next_download)
import_channels_config()
download: Download = test_session.query(Download).one()
assert next_download == str(download.next_download)
# Creating Download that matches Channel2's URL means they are related. Delete it and it should be re-created.
channel2.get_or_create_download(test_session, channel2.url, 60)
save_channels_config()
Download.find_by_url(test_session, channel2.url).delete(add_to_skip_list=False)
test_session.commit()
await await_switches()
# Missing Download is re-created on import.
import_channels_config()
channel2 = Channel.get_by_id(test_session, channel2.id)
downloads = test_session.query(Download).all()
assert len(downloads) == 1, downloads
assert len(channel1.downloads) == 1
assert len(channel2.downloads) == 0
assert downloads[0].downloader == 'video_channel'
# Add a Download to Channel2 which does not match Channel.url.
channel2.get_or_create_download(test_session, 'https://example.org', 60)
channel2.get_or_create_download(test_session, 'https://example.com/channel2', 60)
test_session.commit()
save_channels_config()
await await_switches()
# Check config is written to match new URLs.
config = get_channels_config()
assert len(config.channels) == 2
for channel_config in config.channels:
if channel_config['source_id'] == channel2.source_id:
assert channel_config['source_id'] == 'bar'
assert len(channel_config['downloads']) == 2
assert {i['url'] for i in channel_config['downloads']} \
== {'https://example.com/channel2', 'https://example.org'}
else:
assert channel_config['source_id'] == 'foo'
# Get Channels with their Downloads.
channels_backup = copy(get_channels_config().channels)
# Reset Downloads. Downloads should be recreated from the channels config file.
[i.delete(add_to_skip_list=False) for i in test_session.query(Download).all()]
# Write config with Channel Downloads now that the delete() has removed them from the config.
get_channels_config().channels = channels_backup
import_channels_config()
channel1, channel2 = test_session.query(Channel).order_by(Channel.url).all()
assert test_session.query(Download).count() == 3
assert channel1.url == 'https://example.com/channel1'
assert channel2.url == 'https://example.com/channel2'
assert {i.url for i in channel1.downloads} == {'https://example.com/channel1'}
assert {i.url for i in channel2.downloads} == {'https://example.com/channel2', 'https://example.org'}
# A Channel with a `tag_name` is tagged.
update_channel_config(get_channels_config(),
channel1.source_id,
{'tag_name': tag.name})
import_channels_config()
channel1, channel2 = test_session.query(Channel).order_by(Channel.url).all()
assert channel1.tag == tag and channel1.tag_id == tag.id
assert channel2.tag is None and channel2.tag_id is None
@pytest.mark.asyncio
async def test_import_channel_delete_missing_channels(await_switches, test_session, channel_factory,
test_channels_config):
"""The Channel import function deletes any Channels that are not in the config."""
# Create a DB and config with two Channels.
channel1 = channel_factory(source_id='foo')
channel2 = channel_factory(source_id='bar')
test_session.commit()
# Capture directory values before deletion (channels will be detached after deletion)
# Config stores relative paths for portability
channel1_dir = str(get_relative_to_media_directory(channel1.directory))
channel2_dir = str(get_relative_to_media_directory(channel2.directory))
# Write Channels to the config file.
save_channels_config()
test_session.delete(channel1)
test_session.delete(channel2)
test_session.commit()
await await_switches()
# Importing the config creates two Channels.
import_channels_config()
assert len(test_session.query(Channel).all()) == 2
assert channel1_dir in test_channels_config.read_text()
assert channel2_dir in test_channels_config.read_text()
# Delete channel2 from the config file.
config = get_channels_config()
config_dict = config.dict()
config_dict['channels'] = [i for i in config.channels if i['directory'] != channel2_dir]
config.update(config_dict)
await await_switches()
assert channel1_dir in test_channels_config.read_text()
assert channel2_dir not in test_channels_config.read_text()
# Importing the config deletes the Channel record.
import_channels_config()
assert len(test_session.query(Channel).all()) == 1
assert channel1_dir in test_channels_config.read_text()
assert channel2_dir not in test_channels_config.read_text()
# Saving and importing does not change anything.
save_channels_config()
import_channels_config()
assert channel1_dir in test_channels_config.read_text()
assert channel2_dir not in test_channels_config.read_text()
@pytest.mark.asyncio
async def test_import_channel_deletes_orphaned_collections(await_switches, test_session, channel_factory,
test_channels_config):
"""When a Channel is deleted from config, its associated Collection should also be deleted.
This prevents orphaned Collections (kind='channel') from appearing in the UI with null channel_id.
"""
# Create two channels (each creates a Collection with kind='channel')
channel1 = channel_factory(source_id='foo')
channel2 = channel_factory(source_id='bar')
test_session.commit()
# Verify both Collections exist
assert test_session.query(Collection).filter_by(kind='channel').count() == 2
# Capture directory values before deletion (config stores relative paths)
channel1_dir = str(get_relative_to_media_directory(channel1.directory))
channel2_dir = str(get_relative_to_media_directory(channel2.directory))
collection2_id = channel2.collection_id
# Write Channels to the config file
save_channels_config()
await await_switches()
# Remove channel2 from the config file
config = get_channels_config()
config_dict = config.dict()
config_dict['channels'] = [i for i in config.channels if i['directory'] != channel2_dir]
config.update(config_dict)
await await_switches()
# Import config - this should delete channel2 AND its Collection
import_channels_config()
# Verify only one Channel remains
assert test_session.query(Channel).count() == 1
# Verify the orphaned Collection was also deleted (this is the bug we're fixing)
assert test_session.query(Collection).filter_by(kind='channel').count() == 1
assert test_session.query(Collection).filter_by(id=collection2_id).count() == 0
@pytest.mark.asyncio
async def test_import_channel_cleans_up_preexisting_orphaned_collections(await_switches, test_session, channel_factory,
test_channels_config):
"""Pre-existing orphaned Collections (kind='channel') with no Channel should be deleted during import.
This handles the case where a Collection was orphaned before the cleanup fix was deployed.
"""
# Create a channel (which creates a Collection)
channel1 = channel_factory(source_id='foo')
test_session.commit()
# Create an orphaned Collection directly (simulating pre-existing orphan)
orphan_collection = Collection(
name='Orphaned Channel',
kind='channel',
directory=None,
)
test_session.add(orphan_collection)
test_session.commit()
orphan_id = orphan_collection.id
# Verify we have 2 Collections with kind='channel'
assert test_session.query(Collection).filter_by(kind='channel').count() == 2
# Write the valid channel to config
save_channels_config()
await await_switches()
# Import config - should clean up the pre-existing orphan
import_channels_config()
# Verify only one Collection remains (the one with a Channel)
assert test_session.query(Collection).filter_by(kind='channel').count() == 1
assert test_session.query(Collection).filter_by(id=orphan_id).count() == 0
@pytest.mark.asyncio
async def test_once_download_excluded_from_channel_config(await_switches, test_session, channel_factory,
test_channels_config):
"""A once-download (no frequency) associated with a Channel's Collection should not be saved to the channels
config. This prevents errors during import when a once-download has no frequency."""
channel = channel_factory(download_frequency=DownloadFrequency.weekly)
# Create a once-download associated with the channel's collection (simulates a video download that got
# associated with the channel via apply_yt_channel).
once_download = Download(url='https://www.youtube.com/watch?v=once_video', downloader='video',
collection_id=channel.collection_id)
test_session.add(once_download)
test_session.commit()
# The once-download is in the Collection, but Channel.downloads filters it out.
assert len(channel.collection.downloads) == 2
assert len(channel.downloads) == 1
save_channels_config()
await await_switches()
# The once-download should not appear in the config.
config_text = test_channels_config.read_text()
assert 'once_video' not in config_text
# The recurring download should still be there.
assert channel.url in config_text
# Importing should succeed without the "Refusing to create Download" error.
test_session.delete(once_download)
test_session.commit()
import_channels_config()
assert len(channel.downloads) == 1
@pytest.mark.asyncio
async def test_import_channel_download_comments(await_switches, test_session, channel_factory,
test_channels_config):
channel = channel_factory()
assert channel.download_missing_data is True
# download_missing_data is saved to the config.
save_channels_config()
await await_switches()
assert 'download_missing_data: true' in test_channels_config.read_text()
# change download_missing_data to False, channel should be updated on import.
contents = test_channels_config.read_text()
contents = contents.replace('download_missing_data: true', 'download_missing_data: false')
test_channels_config.write_text(contents)
import_channels_config()
assert channel.download_missing_data is False
@pytest.mark.asyncio
async def test_save_channels_config_does_not_wipe_config(await_switches, test_session, channel_factory,
test_channels_config):
"""Saving channels config when DB is empty must not overwrite an existing config file that has channels.
Regression test: a fresh DB (e.g. after repair) triggering save_channels_config would write channels: []
to the config file, permanently wiping the channel definitions.
"""
# Create a channel and save it to the config file.
channel = channel_factory()
test_session.commit()
save_channels_config()
await await_switches()
assert test_channels_config.is_file()
config_text = test_channels_config.read_text()
assert 'channels: []' not in config_text, 'Config should have a channel'
# Simulate a fresh DB by deleting all channels.
test_session.query(Channel).delete()
test_session.query(Collection).filter_by(kind='channel').delete()
test_session.commit()
assert test_session.query(Channel).count() == 0
# Saving with an empty DB should NOT wipe the config file.
save_channels_config()
await await_switches()
config_text = test_channels_config.read_text()
assert 'channels: []' not in config_text, 'Config was wiped by saving with empty DB!'
@pytest.mark.asyncio
async def test_ffprobe_json(async_client, video_file, corrupted_video_file):
content = await common.ffprobe_json(video_file)
assert not content['chapters']
assert content['format']['duration'] == '5.312000'
assert content['format']['size'] == '1056318'
assert content['streams']
assert content['streams'][0]['codec_name'] == 'h264'
with pytest.raises(RuntimeError):
await common.ffprobe_json(corrupted_video_file)
def test_read_write_ffprobe_json_file(test_directory):
"""Test reading and writing .ffprobe.json files."""
ffprobe_path = test_directory / 'video.ffprobe.json'
ffprobe_data = {'streams': [{'codec_name': 'h264'}], 'format': {'duration': '5.0'}}
# File doesn't exist, should return None
assert common.read_ffprobe_json_file(ffprobe_path) is None
assert common.read_ffprobe_json_file(None) is None
# Write the file
common.write_ffprobe_json_file(ffprobe_path, ffprobe_data)
assert ffprobe_path.is_file()
# File should be human-readable (indented)
contents = ffprobe_path.read_text()
assert '\n' in contents, 'ffprobe json should be formatted with newlines'
# Read it back
loaded = common.read_ffprobe_json_file(ffprobe_path)
assert loaded == ffprobe_data
# Invalid JSON should return None
ffprobe_path.write_text('not valid json')
assert common.read_ffprobe_json_file(ffprobe_path) is None
@pytest.mark.asyncio
async def test_video_ffprobe_json(async_client, test_session, video_file, refresh_files):
"""ffprobe data is extracted when a video is modeled."""
with mock.patch('modules.videos.lib.extract_video_duration') as mock_extract_video_duration:
mock_extract_video_duration.side_effect = Exception('duration should be from ffprobe json')
await refresh_files()
video = test_session.query(Video).one()
assert video.ffprobe_json
assert video.file_group.length
@pytest.mark.asyncio
async def test_video_ffprobe_json_caches_to_file(async_client, test_session, video_file, refresh_files):
"""ffprobe results are cached to .ffprobe.json file."""
await refresh_files()
video = test_session.query(Video).one()
# .ffprobe.json file should have been created during validation
ffprobe_path = video.ffprobe_json_path
assert ffprobe_path.is_file(), '.ffprobe.json file should have been created'
# File should contain the ffprobe data
cached_data = common.read_ffprobe_json_file(ffprobe_path)
assert cached_data is not None
assert 'streams' in cached_data
assert 'format' in cached_data
@pytest.mark.asyncio
async def test_video_ffprobe_json_reads_from_cache(async_client, test_session, video_file, refresh_files):
"""get_ffprobe_json() uses cached .ffprobe.json file instead of running ffprobe."""
await refresh_files()
video = test_session.query(Video).one()
ffprobe_path = video.ffprobe_json_path
# Clear the DB cache to simulate a fresh reindex
video.ffprobe_json = None
test_session.commit()
# Write a fake ffprobe cache file with a marker
fake_data = {'streams': [{'codec_name': 'cached_marker'}], 'format': {'duration': '999'}}
common.write_ffprobe_json_file(ffprobe_path, fake_data)
# get_ffprobe_json should read from the cached file, not call ffprobe
result = await video.get_ffprobe_json()
assert result['streams'][0]['codec_name'] == 'cached_marker', \
'Should have read cached file instead of running ffprobe'
assert video.ffprobe_json == fake_data, 'DB should be updated with cached data'
@pytest.mark.asyncio
async def test_video_ffprobe_json_returns_db_first(async_client, test_session, video_file, refresh_files):
"""get_ffprobe_json() returns DB data if already populated, without reading file."""
await refresh_files()
video = test_session.query(Video).one()
original_data = video.ffprobe_json.copy()
# Write different data to the file (simulating stale cache)
fake_data = {'streams': [{'codec_name': 'stale_data'}], 'format': {}}
common.write_ffprobe_json_file(video.ffprobe_json_path, fake_data)
# Should return DB data, not file data
result = await video.get_ffprobe_json()
assert result == original_data
assert result['streams'][0]['codec_name'] != 'stale_data'
def test_get_videos_directory(test_directory):
"""Default videos directory path"""
# Directory does not yet exist.
assert not (test_directory / 'videos').exists()
# Directory is created when first gotten.
assert common.get_videos_directory() == (test_directory / 'videos')
assert (test_directory / 'videos').is_dir()
def test_get_custom_videos_directory(test_directory, test_wrolpi_config):
"""Custom directory can be used for videos directory."""
config = get_wrolpi_config()
config.videos_destination = 'custom/directory/videos'
assert common.get_videos_directory() == (test_directory / 'custom/directory/videos')
assert (test_directory / 'custom/directory/videos').is_dir()
assert not (test_directory / 'videos').is_dir()
@pytest.mark.asyncio
async def test_video_modeler_does_not_hold_write_lock_during_ffprobe(async_client, test_session, video_factory):
"""Modeling must not hold the SQLite write lock while ffprobe runs.
`video_modeler` opened one write session per batch and `await`ed `get_ffprobe_json` -- an ffprobe
subprocess -- for up to 20 videos inside it. Every other writer on the box waits on that lock
and gives up after `busy_timeout` (30s). ffprobe belongs outside the transaction: read the
batch, probe the files, then write what was learned.
"""
import modules.videos as videos_module
from modules.videos import video_modeler
video = video_factory(with_video_file=True)
video.file_group.indexed = False
video.file_group.model = None
test_session.delete(video)
test_session.commit()
db_file = test_session.get_bind().url.database
lock_held_during_ffprobe = []
real_get_or_create_ffprobe_json = videos_module.get_or_create_ffprobe_json
async def probing_ffprobe(video_path):
lock_held_during_ffprobe.append(probe_write_lock_is_held(db_file))
return await real_get_or_create_ffprobe_json(video_path)
with production_like_sessions(test_session):
with mock.patch.object(videos_module, 'get_or_create_ffprobe_json', probing_ffprobe):
await video_modeler()
assert lock_held_during_ffprobe, 'ffprobe never ran, so the test proves nothing'
assert not any(lock_held_during_ffprobe), \
'the write lock was held while ffprobe ran; every other writer waits out busy_timeout'
def test_ffmpeg_video_complete_real_files(test_directory, video_file):
"""A whole video passes; a truncated copy fails. Uses the real ffmpeg on the example video."""
from modules.videos.common import ffmpeg_video_complete, tail_decode_command
assert ffmpeg_video_complete(video_file) is True
truncated = test_directory / 'truncated.mp4'
data = video_file.read_bytes()
truncated.write_bytes(data[:int(len(data) * 0.6)])
assert ffmpeg_video_complete(truncated) is False
empty = test_directory / 'empty.mp4'
empty.write_bytes(b'')
with pytest.raises(RuntimeError, match='empty'):
ffmpeg_video_complete(empty)
with pytest.raises(FileNotFoundError):
ffmpeg_video_complete(test_directory / 'missing.mp4')
cmd = tail_decode_command(video_file)
assert cmd[cmd.index('-sseof') + 1] == '-30' and cmd[cmd.index('-f') + 1] == 'null' and cmd[-1] == '-'
@pytest.mark.asyncio
async def test_ffmpeg_video_complete_async_real_files(test_directory, video_file):
"""The async variant reaches the same verdicts and returns ffmpeg's error text."""
from modules.videos.common import ffmpeg_video_complete_async
ok, errors = await ffmpeg_video_complete_async(video_file)
assert ok is True and errors == ''
truncated = test_directory / 'truncated.mp4'
data = video_file.read_bytes()
truncated.write_bytes(data[:int(len(data) * 0.6)])
ok, errors = await ffmpeg_video_complete_async(truncated)
assert ok is False and errors