rt/wrolpi
1
0
Fork 0
mirror of https://github.com/lrnselfreliance/wrolpi.git synced 2026-10-25 18:40:06 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)

Browse source 
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
lrnselfreliance 2026-09-16 19:25:23 -06:00 • committed by GitHub
parent c2ffc9286e
commit 5358c75a2b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
27 changed files with 3770 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

77
modules/videos/common.py
View file

@ -149,31 +149,70 @@ def ffmpeg_poster(video_path: Path, poster_path: Path, seconds: int):
return stderr
def ffmpeg_video_complete(video_path: Path, seconds: int = None) -> bool:
"""Checks if video file is complete by taking screenshot from the end of the video.
# How much of a video's tail is decoded to prove the file is complete. Corruption from an
# interrupted download or write lands at the end of the file; decoding all of it would take as
# long as playing it.
TAIL_DECODE_SECONDS = 30
TAIL_DECODE_TIMEOUT = 10 * 60
@raise FileNotFoundError: raised if the video file does not exist.
"""
def tail_decode_command(video_path: Path, seconds: int = None) -> Tuple[str | Path, ...]:
"""ffmpeg command which decodes the last `seconds` of `video_path` to nothing, printing only
errors. `-sseof` seeks from the end; a file shorter than `seconds` is decoded whole."""
seconds = seconds or TAIL_DECODE_SECONDS
return FFMPEG_BIN, '-v', 'error', '-sseof', f'-{int(seconds)}', '-i', video_path, '-f', 'null', '-'
def _tail_decode_verdict(return_code: int, stderr: bytes | str) -> Tuple[bool, str]:
errors = (stderr.decode(errors='replace') if isinstance(stderr, bytes) else (stderr or '')).strip()
ok = return_code == 0 and not errors
return ok, errors
def _check_video_file(video_path: Path):
if not video_path.is_file():
raise FileNotFoundError(f'Video file not found: {video_path}')
if not video_path.stat().st_size:
raise RuntimeError(f'Video file is empty: {video_path}')
if not FFMPEG_BIN:
raise RuntimeError('ffmpeg was not found')
with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as fh:
path = Path(fh.name)
path.unlink()
try:
seconds = seconds or extract_video_duration(video_path) - 5
ffmpeg_poster(video_path, path, seconds)
return True
except subprocess.CalledProcessError:
return False
except TypeError:
# "N/A" duration.
return False
finally:
if path.is_file():
path.unlink()
def ffmpeg_video_complete(video_path: Path, seconds: int = None) -> bool:
"""True if the last `seconds` of the video decode without error.
Used after a download, and after a transcode before the original is replaced. A truncated or
corrupt file fails here because its damage is at the end; the index (moov) must also be
readable for ffmpeg to seek from the end at all.
@raise FileNotFoundError: raised if the video file does not exist.
@raise RuntimeError: raised if the video file is empty, or ffmpeg is missing.
"""
_check_video_file(video_path)
cmd = tuple(str(i) for i in tail_decode_command(video_path, seconds))
try:
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=TAIL_DECODE_TIMEOUT)
except subprocess.TimeoutExpired:
logger.error(f'Timed out checking video is complete: {video_path}')
return False
ok, errors = _tail_decode_verdict(proc.returncode, proc.stderr)
if not ok:
logger.warning(f'Video is incomplete or corrupt: {video_path} (exit {proc.returncode}) {errors[-500:]}')
return ok
async def ffmpeg_video_complete_async(video_path: Path, seconds: int = None, runner=None) -> Tuple[bool, str]:
"""`ffmpeg_video_complete` for async callers; returns (complete, ffmpeg's error output).
`runner` defaults to `run_command`; a Job passes its own so ffmpeg's output lands in the Job log.
"""
_check_video_file(video_path)
runner = runner or run_command
result = await runner(tail_decode_command(video_path, seconds), cwd=video_path.parent, timeout=TAIL_DECODE_TIMEOUT)
ok, errors = _tail_decode_verdict(result.return_code, result.stderr)
if not ok:
logger.warning(f'Video is incomplete or corrupt: {video_path} (exit {result.return_code}) {errors[-500:]}')
return ok, errors
def generate_video_poster(video_path: Path, seconds: int = 5) -> Tuple[Path, Optional[int]]:

16
modules/videos/downloader.py
View file

@ -887,6 +887,7 @@ class PreparedVideo:
destination: Optional[pathlib.Path] = None
info_json: Optional[dict] = None
location: Optional[str] = None # existing video, for error reports
wrolpi: dict = field(default_factory=dict) # existing video's wrolpi section, to carry over
@dataclass
@ -907,6 +908,7 @@ class ExecutedVideo:
tag_names: List[str]
settings: dict
location: Optional[str] # for error reporting
wrolpi: dict = field(default_factory=dict) # existing video's wrolpi section, to carry over
class VideoDownloader(Downloader, ABC):
@ -954,6 +956,8 @@ class VideoDownloader(Downloader, ABC):
# can still tell the user where the previous attempt landed.
existing = Video.get_by_url(session, url)
location = existing.location if existing else None
# yt-dlp rewrites the whole info json; the user's edits (its wrolpi section) must survive.
wrolpi = existing.file_group.get_wrolpi_json() if existing else dict()
return PreparedVideo(
url=url,
@ -962,6 +966,7 @@ class VideoDownloader(Downloader, ABC):
destination=download.destination,
info_json=download.info_json,
location=location,
wrolpi=wrolpi,
)
async def execute_download(self, prepared: PreparedVideo, ctx: DownloadContext,
@ -1201,6 +1206,7 @@ class VideoDownloader(Downloader, ABC):
tag_names=prepared.tag_names,
settings=settings,
location=location,
wrolpi=prepared.wrolpi,
)
except BotBlockedDownloadError:
@ -1238,6 +1244,11 @@ class VideoDownloader(Downloader, ABC):
video.channel_id = executed.channel_id
if video.info_json_path and (new_info_json := video.clean_info_json()):
video.replace_info_json(new_info_json, clean=False)
if executed.wrolpi and video.info_json_path:
# The re-downloaded info json lost the user's edits; put them back, then derive the
# title/description again so the custom ones win (from_paths validated without them).
video.file_group.update_wrolpi_json(executed.wrolpi)
video.validate(session)
if parent_download_url := executed.settings.get('parent_download_url'):
try:
@ -1368,9 +1379,12 @@ class VideoDownloader(Downloader, ABC):
settings = download.settings or dict()
with get_db_session(commit=True) as session:
# Find the existing video, replace its info json.
# Find the existing video, replace its info json; keep the user's edits (wrolpi section).
video = Video.get_by_url(session, url)
wrolpi = video.file_group.get_wrolpi_json()
video.replace_info_json(info_json)
if wrolpi:
video.file_group.update_wrolpi_json(wrolpi)
if parent_download_url := settings.get('parent_download_url'):
try:

4
modules/videos/lib.py
View file

@ -58,7 +58,9 @@ def extract_video_info_json(video: Video) -> VideoInfoJSON:
"""
video_info_json = VideoInfoJSON()
if info_json := video.get_info_json():
title = info_json.get('fulltitle') or info_json.get('title')
# A title the user set (Edit > Edit) is kept apart from yt-dlp's keys, and wins.
wrolpi_section = info_json.get('wrolpi') or dict()
title = wrolpi_section.get('custom_title') or info_json.get('fulltitle') or info_json.get('title')
video_info_json.title = html.unescape(title) if title else None
upload_date = dates.strpdate(i) if (i := info_json.get('upload_date')) else None

12
modules/videos/models.py
View file

@ -141,10 +141,16 @@ class Video(ModelHelper, Base):
def get_video_description(self) -> Optional[str]:
"""
Get the Video description from the file system.
Get the Video description from the file system. A description the user set (Edit > Edit)
lives in the info json's `wrolpi` section and wins; an empty one means "none".
"""
if (info_json := self.get_info_json()) and (description := info_json.get('description')):
return description
info_json = self.get_info_json()
if not info_json:
return None
wrolpi_section = info_json.get('wrolpi') or dict()
if 'custom_description' in wrolpi_section:
return wrolpi_section['custom_description'] or None
return info_json.get('description') or None
def get_description(self) -> Optional[str]:
"""The indexed description (`FileGroup.c_text`), falling back to the info json on disk.

20
modules/videos/schema.py
View file

@ -186,6 +186,26 @@ class VideoSearchRequest:
deep: bool = False # When True, search all text including captions (d_text)
@dataclass
class VideoUpdateRequest:
# Only the fields given are changed.
title: Optional[str] = None
description: Optional[str] = None
@dataclass
class VideoTranscodeRequest:
video_codec: Optional[str] = None # A key of TRANSCODE_VIDEO_TARGETS, or None to copy the stream.
audio_codec: Optional[str] = None # A key of TRANSCODE_AUDIO_TARGETS, or None to copy the stream.
container: str = 'mp4'
fragmented: bool = False # Fragmented mp4: quick start/seek for very long videos (mp4/m4a only).
@dataclass
class VideoTranscodeResponse:
job_id: str
@dataclass
class VideoSearchResponse:
videos: List[VideoWithChannel]

35
modules/videos/test/test_common.py
View file

@ -644,3 +644,38 @@ async def test_video_modeler_does_not_hold_write_lock_during_ffprobe(async_clien
assert lock_held_during_ffprobe, 'ffprobe never ran, so the test proves nothing'
assert not any(lock_held_during_ffprobe), \
'the write lock was held while ffprobe ran; every other writer waits out busy_timeout'
def test_ffmpeg_video_complete_real_files(test_directory, video_file):
"""A whole video passes; a truncated copy fails. Uses the real ffmpeg on the example video."""
from modules.videos.common import ffmpeg_video_complete, tail_decode_command
assert ffmpeg_video_complete(video_file) is True
truncated = test_directory / 'truncated.mp4'
data = video_file.read_bytes()
truncated.write_bytes(data[:int(len(data) * 0.6)])
assert ffmpeg_video_complete(truncated) is False
empty = test_directory / 'empty.mp4'
empty.write_bytes(b'')
with pytest.raises(RuntimeError, match='empty'):
ffmpeg_video_complete(empty)
with pytest.raises(FileNotFoundError):
ffmpeg_video_complete(test_directory / 'missing.mp4')
cmd = tail_decode_command(video_file)
assert cmd[cmd.index('-sseof') + 1] == '-30' and cmd[cmd.index('-f') + 1] == 'null' and cmd[-1] == '-'
@pytest.mark.asyncio
async def test_ffmpeg_video_complete_async_real_files(test_directory, video_file):
"""The async variant reaches the same verdicts and returns ffmpeg's error text."""
from modules.videos.common import ffmpeg_video_complete_async
ok, errors = await ffmpeg_video_complete_async(video_file)
assert ok is True and errors == ''
truncated = test_directory / 'truncated.mp4'
data = video_file.read_bytes()
truncated.write_bytes(data[:int(len(data) * 0.6)])
ok, errors = await ffmpeg_video_complete_async(truncated)
assert ok is False and errors

73
modules/videos/test/test_downloader.py
View file

@ -1789,3 +1789,76 @@ async def test_enforce_codecs():
with pytest.raises(UnrecoverableDownloadError):
await enforce_codecs(video_path, video_paths, effective)
mock_transcode.assert_not_called()
@pytest.mark.asyncio
async def test_metadata_refresh_keeps_wrolpi_section(test_session, test_directory, video_factory):
"""Refreshing a Video's metadata replaces yt-dlp's info json; the wrolpi section (the user's
custom title and description) is carried over and still wins."""
source_id = 'video-source-id'
url = 'https://example.com/the-video'
video = video_factory(with_info_json={'id': source_id, 'title': 'Orig',
'wrolpi': {'custom_title': 'Mine', 'custom_description': 'my words'}},
source_id=source_id)
video.file_group.url = url
test_session.commit()
assert video.file_group.title == 'Mine'
download = Download(url=url, info_json={'id': source_id}, settings={})
with mock.patch('modules.videos.downloader.download_video_info_json') as mock_download:
mock_download.return_value = {'id': source_id, 'title': 'Orig v2', 'description': 'theirs', 'duration': 9}
result = await VideoDownloader.download_info_json(download, video.video_path)
assert result.success is True, result
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
info_json = video.get_info_json()
assert info_json['title'] == 'Orig v2', 'yt-dlp data was refreshed'
assert info_json['wrolpi'] == {'custom_title': 'Mine', 'custom_description': 'my words'}
assert video.file_group.title == 'Mine'
assert video.get_description() == 'my words'
@pytest.mark.asyncio
async def test_redownload_keeps_wrolpi_section(test_session, test_directory, mock_video_extract_info, simple_channel,
await_switches, video_download_manager, mock_video_process_runner,
image_file):
"""A full re-download has yt-dlp overwrite the info json; the wrolpi section is carried over."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
info_json_path = video_path.with_suffix('.info.json')
info_json_path.write_text(json.dumps({'title': 'Orig', 'webpage_url': url, 'wrolpi': {'custom_title': 'Mine'}}))
from modules.videos.normalize_video_url import normalize_video_url
existing = Video.from_paths(test_session, video_path, info_json_path)
existing.validate(test_session)
test_session.commit()
assert existing.file_group.title == 'Mine'
assert Video.get_by_url(test_session, normalize_video_url(url)) is not None, 'lookup by normalized url'
assert existing.file_group.get_wrolpi_json() == {'custom_title': 'Mine'}
from wrolpi.cmd import CommandResult
async def yt_dlp_overwrites_info_json(*args, **kwargs):
# yt-dlp rewrites the whole info json; the wrolpi section is gone from disk.
info_json_path.write_text(json.dumps({'title': 'Orig v2', 'webpage_url': url, 'duration': 5}))
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
mock_video_process_runner.side_effect = yt_dlp_overwrites_info_json
with mock.patch('modules.videos.downloader.prepare_video_filename') as mock_prepare_filename:
mock_video_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(test_session, url, video_downloader.name)
await video_download_manager.wait_for_all_downloads()
mock_video_process_runner.assert_called_once()
test_session.expire_all()
video: Video = test_session.query(Video).one()
info_json = video.get_info_json()
assert info_json['title'] == 'Orig v2'
assert info_json['wrolpi'] == {'custom_title': 'Mine'}
assert video.file_group.title == 'Mine'

280
modules/videos/test/test_transcode.py
View file

@ -1,3 +1,4 @@
import asyncio
import json
import pathlib
from unittest import mock
@ -5,10 +6,51 @@ from unittest import mock
import pytest
from modules.videos.transcode import codecs_match, get_stream_codec_names, get_transcode_target, \
transcode_can_satisfy_codecs, transcode_video_file, TRANSCODE_VIDEO_TARGETS, TRANSCODE_AUDIO_TARGETS
transcode_can_satisfy_codecs, transcode_video_file, TRANSCODE_VIDEO_TARGETS, TRANSCODE_AUDIO_TARGETS, \
verify_transcode_output, validate_transcode_request, REMOVE_VIDEO, FRAGMENT_DURATION_US
from wrolpi.cmd import CommandResult
def make_probe(vcodec='vp9', acodec='opus', duration=10.0, format_name='matroska,webm'):
"""An ffprobe result with one video and one audio stream."""
streams = []
if vcodec:
streams.append({'codec_type': 'video', 'codec_name': vcodec})
if acodec:
streams.append({'codec_type': 'audio', 'codec_name': acodec})
return {'streams': streams, 'format': {'format_name': format_name, 'duration': str(duration)}}
MP4 = 'mov,mp4,m4a,3gp,3g2,mj2'
class FakeProbes:
"""What ffprobe reports for the source (a vp9/opus webm) and for the transcode output.
The default output is a full h264/aac transcode; a test that copies a stream sets `output`
to what ffmpeg would really produce (a copied stream keeps the source codec)."""
def __init__(self):
self.source = make_probe('vp9', 'opus', 10.0)
self.output = make_probe('h264', 'aac', 10.0, MP4)
def __call__(self, path: pathlib.Path):
# Sync: mock.patch wraps the async target in an AsyncMock, which awaits for us.
if '.transcode.' in path.name:
output = dict(self.output)
if path.suffix != '.mp4' and output.get('format'):
output['format'] = {**output['format'], 'format_name': 'matroska,webm'}
return output
return self.source
@pytest.fixture(autouse=True)
def mock_probe() -> FakeProbes:
probes = FakeProbes()
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes):
yield probes
def test_get_stream_codec_names():
"""Embedded thumbnail streams (mjpeg/png) are not video."""
data = {'streams': [
@ -79,6 +121,9 @@ async def test_transcode_video_file(test_directory, async_client):
async def fake_run_command(cmd, **kwargs):
# ffmpeg writes the temporary output file.
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'transcoded data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
@ -92,7 +137,7 @@ async def test_transcode_video_file(test_directory, async_client):
assert not sidecar.exists(), 'The stale ffprobe sidecar should be deleted'
assert not (test_directory / 'video.transcode.mp4').exists(), 'The temporary file should be renamed'
cmd = mock_run.call_args[0][0]
cmd = mock_run.call_args_list[0][0][0]
assert '-c:v' in cmd and cmd[cmd.index('-c:v') + 1] == 'libx264'
assert '-c:a' in cmd and cmd[cmd.index('-c:a') + 1] == 'aac'
# Attached-picture streams are excluded; a missing audio stream is tolerated.
@ -100,12 +145,16 @@ async def test_transcode_video_file(test_directory, async_client):
@pytest.mark.asyncio
async def test_transcode_video_file_copies_matching_stream(test_directory, async_client):
async def test_transcode_video_file_copies_matching_stream(test_directory, async_client, mock_probe):
"""Only the mismatching stream is re-encoded; the other stream is copied."""
video_path = test_directory / 'video.mp4'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('h264', 'opus', 10.0, MP4) # Audio was copied.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'transcoded data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
@ -113,18 +162,22 @@ async def test_transcode_video_file_copies_matching_stream(test_directory, async
result = await transcode_video_file(video_path, target_vcodec='h264', container='mp4')
assert result == video_path, 'Same container: the file is replaced in place'
cmd = mock_run.call_args[0][0]
cmd = mock_run.call_args_list[0][0][0]
assert cmd[cmd.index('-c:v') + 1] == 'libx264'
assert cmd[cmd.index('-c:a') + 1] == 'copy'
@pytest.mark.asyncio
async def test_transcode_video_file_forces_supported_container(test_directory, async_client):
async def test_transcode_video_file_forces_supported_container(test_directory, async_client, mock_probe):
"""webm cannot contain h264/aac; the container is forced to mp4."""
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('h264', 'opus', 10.0, MP4) # Audio was copied.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'transcoded data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
@ -141,6 +194,9 @@ async def test_transcode_video_file_failure_removes_tmp(test_directory, async_cl
video_path.write_bytes(b'fake video data')
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'partial data')
return CommandResult(return_code=1, cancelled=False, stdout=b'', stderr=b'boom', elapsed=1)
@ -168,7 +224,7 @@ async def test_transcode_video_file_disk_space_guard(test_directory, async_clien
@pytest.mark.asyncio
async def test_transcode_lock_serializes(test_directory, async_client):
async def test_transcode_lock_serializes(test_directory, async_client, mock_probe):
"""A transcode waits while `shared_ctx.transcode_lock` is held and releases it when done.
This exercises the wait/release logic in-process; cross-process sharing of the lock is
@ -178,8 +234,12 @@ async def test_transcode_lock_serializes(test_directory, async_client):
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('h264', 'opus', 10.0, MP4) # Audio was copied.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'transcoded data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
@ -203,7 +263,9 @@ async def test_transcode_lock_serializes(test_directory, async_client):
task.cancel()
assert result == test_directory / 'video.mp4'
mock_run.assert_called_once()
# One encode (plus the tail-decode verification of its output); the second waiter ran nothing yet.
encodes = [c for c in mock_run.call_args_list if '-progress' in c[0][0]]
assert len(encodes) == 1
# The lock was released after the transcode.
assert lock.acquire(block=False), 'transcode_video_file did not release the lock'
@ -211,6 +273,204 @@ async def test_transcode_lock_serializes(test_directory, async_client):
@pytest.mark.asyncio
async def test_transcode_video_file_requires_target():
with pytest.raises(RuntimeError, match='target codec'):
await transcode_video_file(pathlib.Path('/tmp/video.mp4'))
async def test_transcode_video_file_remux(test_directory, async_client, mock_probe):
"""No target codec is a remux: both streams are copied into the requested container."""
video_path = test_directory / 'video.mkv'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('vp9', 'opus', 10.0, MP4) # Both copied.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
pathlib.Path(cmd[-1]).write_bytes(b'remuxed data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command) as mock_run:
result = await transcode_video_file(video_path, container='mp4')
assert result == test_directory / 'video.mp4' and result.is_file()
assert not video_path.exists()
cmd = mock_run.call_args_list[0][0][0]
assert cmd[cmd.index('-c:v') + 1] == 'copy' and cmd[cmd.index('-c:a') + 1] == 'copy'
assert '+faststart' in cmd
# ------------------------------------------------------------------ output verification
def test_verify_transcode_output_ok():
source = make_probe('vp9', 'opus', 100.0)
output = make_probe('h264', 'opus', 99.6, MP4) # Audio copied; a frame trimmed.
verify_transcode_output(source, output, target_vcodec='h264', target_acodec=None, container='mp4')
def test_verify_transcode_output_remux_ok():
source = make_probe('vp9', 'opus', 100.0)
output = make_probe('vp9', 'opus', 100.0, MP4)
verify_transcode_output(source, output, target_vcodec=None, target_acodec=None, container='mp4')
def test_verify_transcode_output_no_audio_ok():
"""A silent source has no audio to demand of the output."""
source = make_probe('vp9', None, 100.0)
output = make_probe('h264', None, 100.0, MP4)
verify_transcode_output(source, output, target_vcodec='h264', target_acodec=None, container='mp4')
@pytest.mark.parametrize('output,message', [
(make_probe('h264', 'aac', 50.0, MP4), 'duration'), # Truncated.
(make_probe('h264', None, 100.0, MP4), 'audio'), # Lost the audio stream.
(make_probe(None, 'aac', 100.0, MP4), 'video'), # Lost the video stream.
(make_probe('vp9', 'aac', 100.0, MP4), 'video codec'), # Not re-encoded.
(make_probe('h264', 'opus', 100.0, MP4), 'audio codec'), # Not re-encoded.
(make_probe('h264', 'aac', 100.0, 'matroska,webm'), 'container'), # Wrong container.
({}, 'probe'), # ffprobe could not read it.
])
def test_verify_transcode_output_rejects(output, message):
source = make_probe('vp9', 'opus', 100.0)
with pytest.raises(RuntimeError, match=message):
verify_transcode_output(source, output, target_vcodec='h264', target_acodec='aac', container='mp4')
def test_verify_transcode_output_copied_stream_must_match_source():
"""A copied stream must come out with the source's codec."""
source = make_probe('vp9', 'opus', 100.0)
output = make_probe('h264', 'aac', 100.0, MP4)
with pytest.raises(RuntimeError, match='audio codec'):
verify_transcode_output(source, output, target_vcodec='h264', target_acodec=None, container='mp4')
def test_verify_transcode_output_duration_tolerance():
"""One second, or one percent, whichever is larger."""
source = make_probe('vp9', 'opus', 10.0)
verify_transcode_output(source, make_probe('h264', 'aac', 9.1, MP4), 'h264', 'aac', 'mp4')
with pytest.raises(RuntimeError, match='duration'):
verify_transcode_output(source, make_probe('h264', 'aac', 8.9, MP4), 'h264', 'aac', 'mp4')
long_source = make_probe('vp9', 'opus', 40000.0)
verify_transcode_output(long_source, make_probe('h264', 'aac', 39700.0, MP4), 'h264', 'aac', 'mp4')
with pytest.raises(RuntimeError, match='duration'):
verify_transcode_output(long_source, make_probe('h264', 'aac', 39500.0, MP4), 'h264', 'aac', 'mp4')
@pytest.mark.asyncio
async def test_transcode_video_file_rejects_truncated_output(test_directory, async_client, mock_probe):
"""ffmpeg exits 0 but the output is short: the original is kept, the output is removed."""
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('h264', 'aac', 4.0, MP4) # Source is 10s.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] != '-':
pathlib.Path(cmd[-1]).write_bytes(b'partial data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command):
with pytest.raises(RuntimeError, match='duration'):
await transcode_video_file(video_path, target_vcodec='h264', target_acodec='aac')
assert video_path.read_bytes() == b'fake video data', 'The original must survive'
assert not (test_directory / 'video.mp4').exists()
assert not (test_directory / 'video.transcode.mp4').exists(), 'The bad output is removed'
@pytest.mark.asyncio
async def test_transcode_video_file_rejects_undecodable_tail(test_directory, async_client):
"""The last seconds of the output must decode cleanly, or the original is kept."""
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
tail_cmds = []
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
tail_cmds.append(cmd)
return CommandResult(return_code=0, cancelled=False, stdout=b'',
stderr=b'[h264 @ 0x1] Invalid NAL unit size', elapsed=1)
pathlib.Path(cmd[-1]).write_bytes(b'transcoded data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command):
with pytest.raises(RuntimeError, match='decode'):
await transcode_video_file(video_path, target_vcodec='h264', target_acodec='aac')
assert video_path.read_bytes() == b'fake video data'
assert not (test_directory / 'video.transcode.mp4').exists()
# The check decodes only the tail of the temporary output, discarding the frames.
assert len(tail_cmds) == 1
cmd = tail_cmds[0]
assert '-sseof' in cmd and cmd[cmd.index('-sseof') + 1].startswith('-')
assert str(cmd[cmd.index('-i') + 1]).endswith('video.transcode.mp4')
assert cmd[cmd.index('-f') + 1] == 'null'
@pytest.mark.asyncio
async def test_transcode_video_file_cancel_removes_tmp(test_directory, async_client):
"""A cancelled encode (a Job's run_command raises CancelledError, a BaseException) must still
remove the temporary output and leave the original."""
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
async def fake_run_command(cmd, **kwargs):
pathlib.Path(cmd[-1]).write_bytes(b'partial data')
raise asyncio.CancelledError('killed')
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command):
with pytest.raises(asyncio.CancelledError):
await transcode_video_file(video_path, target_vcodec='h264')
assert video_path.read_bytes() == b'fake video data'
assert not (test_directory / 'video.transcode.mp4').exists(), 'The partial output must be removed'
@pytest.mark.asyncio
async def test_transcode_video_file_fragmented(test_directory, async_client, mock_probe):
"""`fragmented=True` writes a fragmented mp4: a tiny moov, a global seek index, and fragments
at keyframes instead of fast start. Very long videos then start and seek quickly on phones."""
video_path = test_directory / 'video.mp4'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('vp9', 'opus', 10.0, MP4) # A remux: both streams copied.
async def fake_run_command(cmd, **kwargs):
if cmd[-1] != '-':
pathlib.Path(cmd[-1]).write_bytes(b'fragmented data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command) as mock_run:
result = await transcode_video_file(video_path, container='mp4', fragmented=True)
assert result == video_path
cmd = mock_run.call_args_list[0][0][0]
flags = cmd[cmd.index('-movflags') + 1]
for flag in ('frag_keyframe', 'empty_moov', 'default_base_moof', 'global_sidx'):
assert flag in flags, flags
assert 'faststart' not in flags
assert cmd[cmd.index('-frag_duration') + 1] == str(FRAGMENT_DURATION_US)
@pytest.mark.asyncio
async def test_transcode_video_file_fragmented_only_mp4(test_directory, async_client, mock_probe):
"""Fragments are an mp4 concept; for another container the flag is ignored."""
video_path = test_directory / 'video.mp4'
video_path.write_bytes(b'fake video data')
mock_probe.output = make_probe('vp9', 'opus', 10.0, 'matroska,webm')
async def fake_run_command(cmd, **kwargs):
if cmd[-1] != '-':
pathlib.Path(cmd[-1]).write_bytes(b'data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command) as mock_run:
await transcode_video_file(video_path, container='mkv', fragmented=True)
cmd = mock_run.call_args_list[0][0][0]
assert '-movflags' not in cmd and '-frag_duration' not in cmd
def test_validate_fragmented_needs_mp4():
validate_transcode_request(None, None, 'mp4', fragmented=True)
validate_transcode_request(REMOVE_VIDEO, None, 'm4a', fragmented=True) # m4a is an mp4.
with pytest.raises(ValueError, match='mp4'):
validate_transcode_request(None, None, 'mkv', fragmented=True)
with pytest.raises(ValueError, match='mp4'):
validate_transcode_request(REMOVE_VIDEO, 'mp3', 'mp3', fragmented=True)

265
modules/videos/test/test_transcode_audio.py Normal file
View file

@ -0,0 +1,265 @@
"""Audio-only transcodes: removing the video stream from a video, and transcoding audio files."""
import json
import pathlib
import shutil
from unittest import mock
import pytest
from modules.videos.transcode import transcode_video_file, validate_transcode_request, verify_transcode_output, \
REMOVE_VIDEO, TRANSCODE_AUDIO_CONTAINERS, default_audio_container, transcode_video_job
from wrolpi import jobs
from wrolpi.cmd import CommandResult
from wrolpi.vars import PROJECT_DIR
MP4 = 'mov,mp4,m4a,3gp,3g2,mj2'
def make_probe(vcodec, acodec, duration=10.0, format_name=MP4):
streams = []
if vcodec:
streams.append({'codec_type': 'video', 'codec_name': vcodec})
if acodec:
streams.append({'codec_type': 'audio', 'codec_name': acodec})
return {'streams': streams, 'format': {'format_name': format_name, 'duration': str(duration)}}
class FakeProbes:
def __init__(self, source, output):
self.source, self.output = source, output
def __call__(self, path: pathlib.Path):
return self.output if '.transcode.' in path.name else self.source
def fake_ffmpeg(cmd, **kwargs):
"""Writes the output file; the tail-decode check writes nothing."""
if cmd[-1] != '-':
pathlib.Path(cmd[-1]).write_bytes(b'output data')
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
async def async_fake_ffmpeg(cmd, **kwargs):
return fake_ffmpeg(cmd, **kwargs)
# ------------------------------------------------------------------ validation
def test_validate_remove_video():
"""Removing the video needs an audio container; an audio container needs a compatible codec."""
validate_transcode_request(REMOVE_VIDEO, 'aac', 'm4a')
validate_transcode_request(REMOVE_VIDEO, None, 'ogg') # Copy: checked against the source later.
validate_transcode_request(REMOVE_VIDEO, 'mp3', 'mp3')
with pytest.raises(ValueError, match='audio container'):
validate_transcode_request(REMOVE_VIDEO, 'aac', 'mp4')
with pytest.raises(ValueError, match='ogg'):
validate_transcode_request(REMOVE_VIDEO, 'aac', 'ogg')
with pytest.raises(ValueError, match='mp3'):
validate_transcode_request(REMOVE_VIDEO, 'opus', 'mp3')
with pytest.raises(ValueError, match='video'):
validate_transcode_request('h264', 'aac', 'm4a') # A video stream cannot live in m4a.
assert set(TRANSCODE_AUDIO_CONTAINERS) == {'m4a', 'ogg', 'mp3'}
def test_default_audio_container():
assert default_audio_container('aac') == 'm4a'
assert default_audio_container('opus') == 'ogg'
assert default_audio_container('vorbis') == 'ogg'
assert default_audio_container('mp3') == 'mp3'
assert default_audio_container('flac') == 'ogg'
assert default_audio_container(None) == 'm4a'
# ------------------------------------------------------------------ verification
def test_verify_audio_only_output():
source = make_probe('vp9', 'opus', 100.0, 'matroska,webm')
good = make_probe(None, 'aac', 100.0)
verify_transcode_output(source, good, REMOVE_VIDEO, 'aac', 'm4a')
# The video stream must be gone.
with pytest.raises(RuntimeError, match='video stream'):
verify_transcode_output(source, make_probe('vp9', 'aac', 100.0), REMOVE_VIDEO, 'aac', 'm4a')
# An embedded cover is not a video stream.
verify_transcode_output(source, make_probe('mjpeg', 'aac', 100.0), REMOVE_VIDEO, 'aac', 'm4a')
# Audio is required.
with pytest.raises(RuntimeError, match='audio'):
verify_transcode_output(source, make_probe(None, None, 100.0), REMOVE_VIDEO, 'aac', 'm4a')
def test_verify_container_names():
"""ffprobe names containers differently than the file suffix does."""
source = make_probe('h264', 'aac', 10.0)
verify_transcode_output(source, make_probe('h264', 'aac', 10.0, 'matroska,webm'), None, None, 'mkv')
verify_transcode_output(source, make_probe(None, 'opus', 10.0, 'ogg'), REMOVE_VIDEO, 'opus', 'ogg')
verify_transcode_output(source, make_probe(None, 'mp3', 10.0, 'mp3'), REMOVE_VIDEO, 'mp3', 'mp3')
verify_transcode_output(source, make_probe(None, 'aac', 10.0, MP4), REMOVE_VIDEO, None, 'm4a')
with pytest.raises(RuntimeError, match='container'):
verify_transcode_output(source, make_probe(None, 'mp3', 10.0, 'mp3'), REMOVE_VIDEO, 'mp3', 'ogg')
def test_verify_audio_source():
"""An audio-only source has no video to demand; the output is audio-only too."""
source = make_probe(None, 'mp3', 100.0, 'mp3')
verify_transcode_output(source, make_probe(None, 'aac', 100.0), None, 'aac', 'm4a')
verify_transcode_output(source, make_probe(None, 'mp3', 100.0, 'mp3'), None, None, 'mp3')
# ------------------------------------------------------------------ transcode_video_file
@pytest.mark.asyncio
async def test_remove_video(test_directory, async_client):
"""Removing the video: no video map, `-vn`, the audio is required, the output is an audio file
named for its container, and the original is replaced."""
video_path = test_directory / 'movie.mp4'
video_path.write_bytes(b'fake video data')
probes = FakeProbes(make_probe('h264', 'aac', 10.0), make_probe(None, 'aac', 10.0))
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('modules.videos.transcode.run_command', side_effect=async_fake_ffmpeg) as mock_run:
result = await transcode_video_file(video_path, target_vcodec=REMOVE_VIDEO, target_acodec=None,
container='m4a')
assert result == test_directory / 'movie.m4a' and result.is_file()
assert not video_path.exists()
cmd = mock_run.call_args_list[0][0][0]
assert '-vn' in cmd
assert '0:V:0' not in cmd
assert cmd[cmd.index('-map') + 1] == '0:a:0', 'Audio is required, not optional'
assert '-c:v' not in cmd
assert cmd[cmd.index('-c:a') + 1] == 'copy'
assert '+faststart' in cmd, 'm4a is an mp4; fast start applies'
assert str(cmd[-1]).endswith('movie.transcode.m4a')
@pytest.mark.asyncio
async def test_remove_video_to_ogg_no_faststart(test_directory, async_client):
video_path = test_directory / 'movie.webm'
video_path.write_bytes(b'fake video data')
probes = FakeProbes(make_probe('vp9', 'opus', 10.0, 'matroska,webm'), make_probe(None, 'opus', 10.0, 'ogg'))
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('modules.videos.transcode.run_command', side_effect=async_fake_ffmpeg) as mock_run:
result = await transcode_video_file(video_path, target_vcodec=REMOVE_VIDEO, container='ogg')
assert result == test_directory / 'movie.ogg'
cmd = mock_run.call_args_list[0][0][0]
assert '-movflags' not in cmd, 'movflags is an mp4 muxer option'
@pytest.mark.asyncio
async def test_audio_source_is_audio_output(test_directory, async_client):
"""An audio file has no video to keep or remove: the output is audio-only whatever the video
target says, and an unsuitable container is replaced by the codec's natural one."""
audio_path = test_directory / 'song.webm'
audio_path.write_bytes(b'fake audio data')
probes = FakeProbes(make_probe(None, 'opus', 10.0, 'matroska,webm'), make_probe(None, 'opus', 10.0, 'ogg'))
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('modules.videos.transcode.run_command', side_effect=async_fake_ffmpeg) as mock_run:
result = await transcode_video_file(audio_path, target_vcodec=None, target_acodec=None, container='mp4')
assert result == test_directory / 'song.ogg', 'opus copied: ogg, not the video container asked for'
cmd = mock_run.call_args_list[0][0][0]
assert '-vn' in cmd and '0:V:0' not in cmd
@pytest.mark.asyncio
async def test_copy_audio_into_incompatible_container_is_refused(test_directory, async_client):
"""Copying opus into m4a cannot work; refuse before ffmpeg runs."""
video_path = test_directory / 'movie.webm'
video_path.write_bytes(b'fake video data')
probes = FakeProbes(make_probe('vp9', 'opus', 10.0, 'matroska,webm'), make_probe(None, 'opus', 10.0))
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('modules.videos.transcode.run_command', side_effect=async_fake_ffmpeg) as mock_run:
with pytest.raises(RuntimeError, match='m4a'):
await transcode_video_file(video_path, target_vcodec=REMOVE_VIDEO, target_acodec=None, container='m4a')
mock_run.assert_not_called()
assert video_path.is_file()
@pytest.mark.asyncio
async def test_video_target_on_audio_source_is_refused(test_directory, async_client):
audio_path = test_directory / 'song.mp3'
audio_path.write_bytes(b'fake audio data')
probes = FakeProbes(make_probe(None, 'mp3', 10.0, 'mp3'), make_probe(None, 'mp3', 10.0, 'mp3'))
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('modules.videos.transcode.run_command', side_effect=async_fake_ffmpeg) as mock_run:
with pytest.raises(RuntimeError, match='no video stream'):
await transcode_video_file(audio_path, target_vcodec='h264', container='mp4')
mock_run.assert_not_called()
# ------------------------------------------------------------------ Job + API
@pytest.mark.asyncio
async def test_remove_video_job(test_session, test_directory, async_client, video_factory):
"""The FileGroup follows the video into its audio-only life: same id, new suffix and mimetype."""
from modules.videos.models import Video
from wrolpi.files.models import FileGroup
video = video_factory()
video.file_group.length = 5
test_session.commit()
file_group_id = video.file_group_id
video_path = video.video_path
probes = FakeProbes(make_probe('h264', 'aac', 5.3), make_probe(None, 'mp3', 5.3, 'mp3'))
async def fake_run_command(cmd, stdout_callback=None, **kwargs):
if cmd[-1] != '-':
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny.mp3', cmd[-1])
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probes), \
mock.patch('wrolpi.cmd.run_command', side_effect=fake_run_command):
job_id = transcode_video_job.enqueue(file_group_id=file_group_id, video_codec=REMOVE_VIDEO,
audio_codec='mp3', container='mp3')
record = await jobs.wait_for_job(job_id)
assert record['status'] == jobs.COMPLETE, record
new_path = video_path.with_suffix('.mp3')
assert new_path.is_file() and not video_path.exists()
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, file_group_id)
assert video.video_path == new_path
assert video.file_group.mimetype.startswith('audio/')
assert test_session.query(FileGroup).count() == 1
@pytest.mark.asyncio
async def test_transcode_api_audio_file(async_client, test_session, audio_factory):
"""An audio file can be transcoded (audio codec / audio container); a video target is a 400."""
audio = audio_factory()
test_session.commit()
request, response = await async_client.post(f'/api/videos/{audio.file_group_id}/transcode',
content=json.dumps({'video_codec': 'h264', 'container': 'mp4'}))
assert response.status == 400, response.json
assert 'audio' in response.json['message'].lower() or 'audio' in response.json['error'].lower()
request, response = await async_client.post(f'/api/videos/{audio.file_group_id}/transcode',
content=json.dumps({'audio_codec': 'aac', 'container': 'm4a'}))
assert response.status == 200, response.json
jobs.cancel_job(response.json['job_id'])
# Remove-video on an audio file is a no-op wording-wise but allowed (it is already audio).
request, response = await async_client.post(f'/api/videos/{audio.file_group_id}/transcode',
content=json.dumps({'video_codec': REMOVE_VIDEO, 'container': 'mp3'}))
assert response.status == 200, response.json
jobs.cancel_job(response.json['job_id'])
@pytest.mark.asyncio
async def test_transcode_api_remove_video_needs_audio_container(async_client, test_session, video_factory):
video = video_factory()
test_session.commit()
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'video_codec': REMOVE_VIDEO, 'container': 'mp4'}))
assert response.status == 400, response.json
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'video_codec': REMOVE_VIDEO, 'audio_codec': 'aac',
'container': 'm4a'}))
assert response.status == 200, response.json
jobs.cancel_job(response.json['job_id'])

295
modules/videos/test/test_transcode_job.py Normal file
View file

@ -0,0 +1,295 @@
"""Tests for the transcode Job and its API (see `wrolpi.jobs`)."""
import json
import pathlib
import shutil
from unittest import mock
import pytest
from modules.videos.models import Video
from modules.videos.transcode import make_ffmpeg_progress_parser, transcode_video_job
from wrolpi import jobs
from wrolpi.cmd import CommandResult
from wrolpi.files.models import FileGroup
from wrolpi.vars import PROJECT_DIR
MP4 = 'mov,mp4,m4a,3gp,3g2,mj2'
@pytest.fixture(autouse=True)
def mock_probe():
"""The output probe agrees with what the Job asked for (targets or copied source codecs)."""
async def probe(path: pathlib.Path):
# The test video is h264/aac; a copied stream keeps that, a target replaces it.
fmt = MP4 if path.suffix == '.mp4' else 'matroska,webm'
return {'streams': [{'codec_type': 'video', 'codec_name': 'h264'},
{'codec_type': 'audio', 'codec_name': 'aac'}],
'format': {'format_name': fmt, 'duration': '5.3'}}
with mock.patch('modules.videos.transcode.probe_for_verify', side_effect=probe):
yield
def test_make_ffmpeg_progress_parser():
"""`-progress pipe:1` lines are converted to whole-percent progress, reported only on change."""
reported = []
parse = make_ffmpeg_progress_parser(100.0, reported.append)
parse('frame=12') # Ignored.
parse('out_time_us=25000000')
parse('out_time_us=25400000') # Still 25%, not reported again.
parse('out_time_ms=50000000') # Older ffmpeg key, also microseconds.
parse('out_time_us=garbage')
parse('out_time_us=250000000') # Clamped to 100.
assert reported == [25, 50, 100]
# No duration: nothing can be reported.
reported.clear()
make_ffmpeg_progress_parser(None, reported.append)('out_time_us=25000000')
assert reported == []
@pytest.mark.asyncio
async def test_transcode_video_job(test_session, test_directory, async_client, video_factory):
"""The transcode Job runs ffmpeg inside the Job (log + progress captured) and re-indexes the
Video so its FileGroup points at the new file."""
video_path = test_directory / 'videos/NO CHANNEL/movie.webm'
video_path.parent.mkdir(parents=True, exist_ok=True)
video = video_factory(with_video_file=video_path)
video.file_group.length = 10
test_session.commit()
file_group_id = video.file_group_id
async def fake_run_command(cmd, stdout_callback=None, **kwargs):
# ffmpeg reports progress on stdout, then writes the temporary output file. The output
# must be a real video: the refresh would otherwise drop the Video model (mimetype changed).
stdout_callback('out_time_us=5000000')
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', cmd[-1])
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'ffmpeg version fake', elapsed=1)
# In tests the file refresh is drained inline by the waiter, inside the Job's task, so its
# (many) log lines land in the Job log too and could push the early lines out of the tail.
with mock.patch('wrolpi.cmd.run_command', side_effect=fake_run_command), \
mock.patch.object(jobs, 'JOB_LOG_LINES', 10_000):
job_id = transcode_video_job.enqueue(description='Transcode movie.webm', file_group_id=file_group_id,
video_codec='h264', audio_codec='aac', container='mp4')
record = await jobs.wait_for_job(job_id)
assert record['status'] == jobs.COMPLETE, record
assert record['result'] == {'path': 'videos/NO CHANNEL/movie.mp4'}
assert 'out_time_us=5000000' in record['log'], 'ffmpeg stdout is in the Job log'
assert 'ffmpeg version fake' in record['log'], 'ffmpeg stderr is in the Job log'
assert any('Transcoding' in i for i in record['log']), 'logger output is in the Job log'
assert not video_path.exists()
assert (test_directory / 'videos/NO CHANNEL/movie.mp4').is_file()
test_session.expire_all()
# The same FileGroup (tags, history) now points at the mp4; no new FileGroup was created.
video = Video.find_by_file_group_id(test_session, file_group_id)
assert video.video_path == test_directory / 'videos/NO CHANNEL/movie.mp4'
assert video.file_group.primary_path == test_directory / 'videos/NO CHANNEL/movie.mp4'
assert video.file_group.mimetype == 'video/mp4'
# The modeler probed the new file and wrote a fresh sidecar; the webm is gone.
assert sorted(i['path'] for i in video.file_group.files) == ['movie.ffprobe.json', 'movie.mp4']
assert video.ffprobe_json, 'ffprobe ran again on the new file'
assert video.file_group.indexed is True, 'The refresh re-modeled the Video'
assert test_session.query(FileGroup).count() == 1
@pytest.mark.asyncio
async def test_transcode_video_job_remux(test_session, test_directory, async_client, video_factory):
"""No target codec is a remux: both streams are copied into the requested container."""
video_path = test_directory / 'videos/NO CHANNEL/movie.mkv'
video_path.parent.mkdir(parents=True, exist_ok=True)
video = video_factory(with_video_file=video_path)
test_session.commit()
async def fake_run_command(cmd, stdout_callback=None, **kwargs):
if cmd[-1] == '-':
# The tail-decode check: nothing to write.
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', cmd[-1])
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('wrolpi.cmd.run_command', side_effect=fake_run_command) as mock_run:
job_id = transcode_video_job.enqueue(file_group_id=video.file_group_id, container='mp4')
record = await jobs.wait_for_job(job_id)
assert record['status'] == jobs.COMPLETE, record
cmd = mock_run.call_args_list[0][0][0]
assert cmd[cmd.index('-c:v') + 1] == 'copy' and cmd[cmd.index('-c:a') + 1] == 'copy'
assert '+faststart' in cmd
assert not video_path.exists()
assert (test_directory / 'videos/NO CHANNEL/movie.mp4').is_file()
@pytest.mark.asyncio
async def test_transcode_video_job_failure(test_session, test_directory, async_client, video_factory):
"""A failed ffmpeg fails the Job and leaves the original file in place."""
video = video_factory()
video_path = video.video_path
async def fake_run_command(cmd, stdout_callback=None, **kwargs):
return CommandResult(return_code=1, cancelled=False, stdout=b'', stderr=b'Conversion failed!', elapsed=1)
with mock.patch('wrolpi.cmd.run_command', side_effect=fake_run_command):
job_id = transcode_video_job.enqueue(file_group_id=video.file_group_id, video_codec='h264')
record = await jobs.wait_for_job(job_id)
assert record['status'] == jobs.FAILED
assert 'Conversion failed!' in record['error']
assert video_path.is_file()
@pytest.mark.asyncio
async def test_video_transcode_api(async_client, test_session, video_factory, wrol_mode_fixture):
"""The transcode endpoint validates the request, then queues a Job."""
video = video_factory()
test_session.commit()
# Unsupported target.
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'video_codec': 'av1'}))
assert response.status == 400
# Unknown video.
request, response = await async_client.post('/api/videos/999999/transcode',
content=json.dumps({'video_codec': 'h264'}))
assert response.status == 404
assert jobs.get_jobs() == [], 'Nothing was queued'
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'video_codec': 'h264', 'audio_codec': 'aac'}))
assert response.status == 200, response.json
job_id = response.json['job_id']
record = jobs.get_job(job_id)
assert record['status'] == jobs.PENDING
assert record['name'] == 'transcode_video'
assert record['description'] == f'Transcode {video.video_path.name}'
assert record['kwargs'] == {'file_group_id': video.file_group_id, 'video_codec': 'h264',
'audio_codec': 'aac', 'container': 'mp4', 'fragmented': False}
# Do not run it (ffmpeg); cancel instead.
jobs.cancel_job(job_id)
# No codec at all is allowed: a container-only remux.
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'container': 'mkv'}))
assert response.status == 200, response.json
assert jobs.get_job(response.json['job_id'])['kwargs'] == {
'file_group_id': video.file_group_id, 'video_codec': None, 'audio_codec': None, 'container': 'mkv', 'fragmented': False}
jobs.cancel_job(response.json['job_id'])
# WROL mode forbids modifying files.
await wrol_mode_fixture(True)
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'video_codec': 'h264'}))
assert response.status == 403
await wrol_mode_fixture(False)
@pytest.mark.asyncio
async def test_transcode_video_job_wrol_mode(test_session, test_directory, async_client, video_factory,
wrol_mode_fixture):
"""A transcode queued before WROL Mode was enabled must not run: the Job fails, ffmpeg is never
started, and the file is untouched."""
video = video_factory()
video_path = video.video_path
before = video_path.read_bytes()
job_id = transcode_video_job.enqueue(file_group_id=video.file_group_id, video_codec='h264')
await wrol_mode_fixture(True)
try:
with mock.patch('wrolpi.cmd.run_command') as mock_run:
record = await jobs.wait_for_job(job_id)
finally:
await wrol_mode_fixture(False)
assert record['status'] == jobs.FAILED, record
assert 'WROL Mode' in record['error']
mock_run.assert_not_called()
assert video_path.read_bytes() == before
@pytest.mark.asyncio
async def test_transcode_video_job_retargets_before_deleting(test_session, test_directory, async_client,
video_factory):
"""When the suffix changes, the FileGroup is pointed at the new file (and committed) while the
old file still exists; the old file is deleted only afterwards. A crash in between then leaves
both files with the database pointing at a real one."""
from modules.videos import transcode as transcode_module
video_path = test_directory / 'videos/NO CHANNEL/movie.webm'
video_path.parent.mkdir(parents=True, exist_ok=True)
video = video_factory(with_video_file=video_path)
test_session.commit()
seen = {}
real_retarget = transcode_module.retarget_file_group
def spy_retarget(file_group_id, old_path, new_path):
seen['old_exists'] = old_path.exists()
seen['new_exists'] = new_path.exists()
real_retarget(file_group_id, old_path, new_path)
test_session.expire_all()
seen['db_primary'] = Video.find_by_file_group_id(test_session, file_group_id).file_group.primary_path
async def fake_run_command(cmd, stdout_callback=None, **kwargs):
if cmd[-1] == '-':
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', cmd[-1])
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('wrolpi.cmd.run_command', side_effect=fake_run_command), \
mock.patch.object(transcode_module, 'retarget_file_group', side_effect=spy_retarget):
job_id = transcode_video_job.enqueue(file_group_id=video.file_group_id, video_codec='h264',
audio_codec='aac', container='mp4')
record = await jobs.wait_for_job(job_id)
assert record['status'] == jobs.COMPLETE, record
new_path = test_directory / 'videos/NO CHANNEL/movie.mp4'
assert seen == {'old_exists': True, 'new_exists': True, 'db_primary': new_path}, seen
assert not video_path.exists(), 'The old file is deleted once the database points at the new one'
assert new_path.is_file()
@pytest.mark.asyncio
async def test_transcode_video_file_keep_original(test_directory, async_client):
"""`transcode_video_file(keep_original=True)` renames the output into place but leaves the
original for the caller to delete once its records point at the new file."""
from modules.videos.transcode import transcode_video_file
video_path = test_directory / 'video.webm'
video_path.write_bytes(b'fake video data')
async def fake_run_command(cmd, **kwargs):
if cmd[-1] == '-':
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=0)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', cmd[-1])
return CommandResult(return_code=0, cancelled=False, stdout=b'', stderr=b'', elapsed=1)
with mock.patch('modules.videos.transcode.run_command', side_effect=fake_run_command):
result = await transcode_video_file(video_path, target_vcodec='h264', target_acodec='aac',
container='mp4', keep_original=True)
assert result == test_directory / 'video.mp4' and result.is_file()
assert video_path.read_bytes() == b'fake video data', 'The original is kept'
@pytest.mark.asyncio
async def test_video_transcode_api_fragmented(async_client, test_session, video_factory):
"""`fragmented` reaches the Job; it needs an mp4 container."""
video = video_factory()
test_session.commit()
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'container': 'mp4', 'fragmented': True}))
assert response.status == 200, response.json
record = jobs.get_job(response.json['job_id'])
assert record['kwargs']['fragmented'] is True
assert record['description'].startswith('Remux')
jobs.cancel_job(response.json['job_id'])
request, response = await async_client.post(f'/api/videos/{video.file_group_id}/transcode',
content=json.dumps({'container': 'mkv', 'fragmented': True}))
assert response.status == 400

212
modules/videos/test/test_video_edit.py Normal file
View file

@ -0,0 +1,212 @@
"""Editing a Video's details from the video page.
Edits never touch yt-dlp's own keys: they are written to the `wrolpi` section of the info json
(`custom_title`, `custom_description`), which the modeler prefers when it derives the Video."""
import json
import pytest
from modules.videos.lib import extract_video_info_json
from modules.videos.models import Video
from modules.videos.video.lib import update_video
from wrolpi.errors import ValidationError
def test_update_video_title(test_session, test_directory, video_factory):
"""The new title lands in the wrolpi section; yt-dlp's title/fulltitle stay as they were. The
Video is re-validated, so its title and search text change."""
video = video_factory(with_info_json={'title': 'Old Title', 'fulltitle': 'Old Title', 'duration': 7,
'formats': ['kept, not cleaned']})
test_session.commit()
assert video.file_group.title == 'Old Title'
update_video(video.file_group_id, ' New Title ')
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
assert video.file_group.title == 'New Title'
assert video.file_group.a_text == 'New Title'
assert video.file_group.length == 7, 'Other details from the info json are unchanged'
info_json = json.loads(video.info_json_path.read_text())
assert info_json['title'] == 'Old Title' and info_json['fulltitle'] == 'Old Title', 'The original is kept'
assert info_json['wrolpi']['custom_title'] == 'New Title'
assert info_json['formats'] == ['kept, not cleaned']
def test_update_video_title_creates_info_json(test_session, test_directory, video_factory):
"""A Video without an info json gets one holding only the wrolpi section."""
video = video_factory()
test_session.commit()
assert video.info_json_path is None
update_video(video.file_group_id, 'Brand New')
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
assert video.file_group.title == 'Brand New'
assert video.info_json_path and video.info_json_path.is_file()
info_json = json.loads(video.info_json_path.read_text())
assert info_json == {'wrolpi': {'custom_title': 'Brand New'}}
assert video.info_json_path.name in [i['path'] for i in video.file_group.files], 'The new file is tracked'
def test_update_video_description(test_session, test_directory, video_factory):
"""The description lands in the wrolpi section and is indexed; yt-dlp's description stays.
An empty custom description clears the shown description."""
video = video_factory(with_info_json={'title': 'Kept', 'description': 'old words'})
test_session.commit()
assert video.file_group.c_text == 'old words'
update_video(video.file_group_id, description='new words\nsecond line')
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
assert video.file_group.title == 'Kept'
assert video.file_group.c_text == 'new words\nsecond line'
assert video.get_description() == 'new words\nsecond line'
info_json = json.loads(video.info_json_path.read_text())
assert info_json['description'] == 'old words', 'The original is kept'
assert info_json['wrolpi']['custom_description'] == 'new words\nsecond line'
update_video(video.file_group_id, description='')
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, video.file_group_id)
assert video.get_description() is None
assert json.loads(video.info_json_path.read_text())['description'] == 'old words'
assert video.file_group.title == 'Kept'
def test_update_video_nothing(test_session, video_factory):
video = video_factory(with_info_json={'title': 'Old'})
test_session.commit()
with pytest.raises(ValidationError):
update_video(video.file_group_id)
def test_update_video_title_empty(test_session, video_factory):
video = video_factory(with_info_json={'title': 'Old'})
test_session.commit()
with pytest.raises(ValidationError):
update_video(video.file_group_id, ' ')
test_session.expire_all()
assert Video.find_by_file_group_id(test_session, video.file_group_id).file_group.title == 'Old'
def test_extract_video_info_json_prefers_custom_title(test_session, video_factory):
"""The modeler's reader prefers the wrolpi custom title over fulltitle/title."""
video = video_factory(with_info_json={'title': 'Orig', 'fulltitle': 'Orig Full',
'wrolpi': {'custom_title': 'Mine &amp; yours'}})
assert extract_video_info_json(video).title == 'Mine & yours', 'HTML entities are unescaped as for any title'
video2 = video_factory(with_info_json={'title': 'Orig', 'fulltitle': 'Orig Full', 'wrolpi': {}})
assert extract_video_info_json(video2).title == 'Orig Full'
@pytest.mark.asyncio
async def test_custom_title_survives_refresh(test_session, test_directory, async_client, video_factory):
"""A file refresh re-models the Video from its files; the custom title must win again."""
from wrolpi.files.worker import file_worker
video = video_factory(with_info_json={'title': 'Original'})
test_session.commit()
update_video(video.file_group_id, title='Custom', description='custom words')
video_path = video.video_path
file_group_id = video.file_group_id
# Force a full re-model, as a refresh of a modified file does.
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, file_group_id)
video.file_group.indexed = False
video.file_group.title = None
video.file_group.c_text = None
test_session.commit()
job_id = file_worker.queue_refresh([video_path], send_events=False)
await file_worker.wait_for_job(job_id)
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, file_group_id)
assert video.file_group.title == 'Custom'
assert video.get_description() == 'custom words'
assert json.loads(video.info_json_path.read_text())['title'] == 'Original'
@pytest.mark.asyncio
async def test_video_update_api(async_client, test_session, video_factory, wrol_mode_fixture):
video = video_factory(with_info_json={'title': 'Old'})
test_session.commit()
request, response = await async_client.put(f'/api/videos/{video.file_group_id}',
content=json.dumps({'title': 'Edited'}))
assert response.status == 200, response.json
assert response.json['file_group']['title'] == 'Edited'
assert response.json['file_group']['id'] == video.file_group_id
request, response = await async_client.put(f'/api/videos/{video.file_group_id}', content=json.dumps({'title': ''}))
assert response.status == 400
request, response = await async_client.put(f'/api/videos/{video.file_group_id}', content=json.dumps({}))
assert response.status == 400
request, response = await async_client.put(f'/api/videos/{video.file_group_id}',
content=json.dumps({'description': 'about it'}))
assert response.status == 200, response.json
assert response.json['file_group']['title'] == 'Edited', 'Title untouched'
request, response = await async_client.get(f'/api/videos/{video.file_group_id}/description')
assert response.json['description'] == 'about it'
request, response = await async_client.put('/api/videos/999999', content=json.dumps({'title': 'x'}))
assert response.status == 404
await wrol_mode_fixture(True)
request, response = await async_client.put(f'/api/videos/{video.file_group_id}',
content=json.dumps({'title': 'Nope'}))
assert response.status == 403
await wrol_mode_fixture(False)
@pytest.mark.asyncio
async def test_update_video_wrol_mode(test_session, video_factory, wrol_mode_fixture):
"""The library function refuses in WROL Mode, not only the API route, so no caller can edit."""
from wrolpi.errors import WROLModeEnabled
video = video_factory(with_info_json={'title': 'Old'})
test_session.commit()
await wrol_mode_fixture(True)
try:
with pytest.raises(WROLModeEnabled):
update_video(video.file_group_id, title='Nope')
finally:
await wrol_mode_fixture(False)
test_session.expire_all()
assert Video.find_by_file_group_id(test_session, video.file_group_id).file_group.title == 'Old'
@pytest.mark.asyncio
async def test_edit_creates_info_json_that_survives_refresh(test_session, test_directory, async_client,
video_factory):
"""A Video with no info json is edited (the edit creates one), then its file is refreshed: the
new sidecar must stay in the FileGroup and the custom title/description must still win."""
from wrolpi.files.worker import file_worker
video = video_factory()
test_session.commit()
video_path = video.video_path
file_group_id = video.file_group_id
update_video(file_group_id, title='Custom', description='my words')
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, file_group_id)
assert video.info_json_path and video.info_json_path.is_file()
assert video.get_description() == 'my words'
job_id = file_worker.queue_refresh([video_path], send_events=False)
await file_worker.wait_for_job(job_id)
test_session.expire_all()
video = Video.find_by_file_group_id(test_session, file_group_id)
assert video.info_json_path is not None, 'The refresh forgot the info json'
names = {i['path'] for i in video.file_group.files}
assert video_path.name in names and video.info_json_path.name in names, names
assert video.file_group.title == 'Custom'
assert video.get_description() == 'my words'

345
modules/videos/transcode.py
View file

@ -2,11 +2,14 @@ import asyncio
import os
import pathlib
import shutil
from typing import List, Optional, Tuple
from typing import List, Optional, Tuple, Callable
from modules.videos.common import ffprobe_json, ffmpeg_video_complete_async
from wrolpi.captions import FFMPEG_BIN
from wrolpi.cmd import run_command
from wrolpi.common import logger
from wrolpi.common import logger, get_relative_to_media_directory, wrol_mode_enabled
from wrolpi.db import get_db_session
from wrolpi.jobs import register_job, get_current_job
from wrolpi.vars import DEFAULT_FILE_PERMISSIONS, PYTEST
logger = logger.getChild(__name__)
@ -41,10 +44,45 @@ TRANSCODE_AUDIO_TARGETS = {
}
# Containers that can hold an h264/aac transcode result. webm cannot.
TRANSCODE_CONTAINERS = ('mp4', 'mkv')
TRANSCODE_CONTAINERS = ('mp4', 'mkv') # For output with a video stream.
# For audio-only output: the video stream removed (`REMOVE_VIDEO`), or an audio-only source.
TRANSCODE_AUDIO_CONTAINERS = ('m4a', 'ogg', 'mp3')
# ffprobe codec names each audio container can hold without re-encoding.
# Ordered: `default_audio_container` picks the first container that holds a codec, so the most
# specific container comes first (mp3 fits in m4a too, but an .mp3 is what a user expects).
AUDIO_CONTAINER_CODECS = {
'mp3': {'mp3'},
'ogg': {'opus', 'vorbis', 'flac'},
'm4a': {'aac', 'mp3', 'alac'},
}
# `target_vcodec` value meaning "drop the video stream"; the output is an audio file.
REMOVE_VIDEO = 'none'
# ffprobe's `format_name` token for each container (mkv reports "matroska,webm").
CONTAINER_FORMAT_NAMES = {'mp4': 'mp4', 'mkv': 'matroska', 'm4a': 'm4a', 'ogg': 'ogg', 'mp3': 'mp3'}
# `-movflags +faststart` is an mp4-muxer option; other muxers warn about it.
FASTSTART_CONTAINERS = ('mp4', 'm4a')
# Fragmented mp4 (`fragmented=True`): a tiny moov plus a global seek index, samples described per
# fragment. A very long video's whole-file index can reach tens of MB, which a phone must fetch
# and parse before the first frame; fragments make it start and seek quickly. mp4/m4a only.
FRAGMENTED_MOVFLAGS = '+frag_keyframe+empty_moov+default_base_moof+global_sidx'
FRAGMENT_DURATION_US = 10_000_000
def default_audio_container(codec: Optional[str]) -> str:
"""The natural audio container for a codec: what a player expects from the suffix."""
for container, codecs in AUDIO_CONTAINER_CODECS.items():
if codec in codecs:
return container
return 'm4a'
TRANSCODE_TIMEOUT = 4 * 60 * 60 # A long video on a Raspberry Pi can take hours.
# Output verification (see `verify_transcode_output`): the output's duration may differ from the
# source by this much before it is considered truncated. Muxers trim a frame or two; a truncated
# file is off by minutes.
DURATION_TOLERANCE_SECONDS = 1.0
DURATION_TOLERANCE_RATIO = 0.01
# Transcoding writes a whole new copy of the video next to the original.
MINIMUM_FREE_SPACE_RATIO = 1.5
@ -114,34 +152,178 @@ async def _acquire_transcode_lock():
async def transcode_video_file(video_path: pathlib.Path,
target_vcodec: Optional[str] = None,
target_acodec: Optional[str] = None,
container: str = 'mp4') -> pathlib.Path:
container: str = 'mp4',
duration: Optional[float] = None,
keep_original: bool = False,
fragmented: bool = False) -> pathlib.Path:
"""Transcode a video file in place (same stem, possibly a new container extension).
Only one transcode runs at a time machine-wide (across all Sanic workers); this call waits
for its turn. Only the stream(s) with a target are re-encoded; the other stream is copied.
The output is written to a temporary file in the same directory, then atomically renamed over
the final path. The original file is deleted if the extension changed. Returns the final
path.
the final path. The original file is deleted if the extension changed, unless
`keep_original`: a caller with database records pointing at the original must retarget them
first, then delete it (see `transcode_video_job`), so a crash never leaves the records pointing
at a deleted file. Returns the final path.
When run inside a Job (see `wrolpi.jobs`), ffmpeg's output is captured into the Job log and, if
`duration` (seconds) is known, progress is reported to the Job.
@raise RuntimeError: when ffmpeg fails, or there is not enough free disk space.
"""
if not FFMPEG_BIN:
raise RuntimeError('ffmpeg was not found')
if not target_vcodec and not target_acodec:
raise RuntimeError('Refusing to transcode without a target codec')
if container not in TRANSCODE_CONTAINERS:
logger.info(f'Forcing mp4 container for transcode of {video_path} ({container} is not supported)')
container = 'mp4'
# No target for either stream is a remux: both streams are copied into the (new) container.
# `target_vcodec=REMOVE_VIDEO` (or an audio-only source) makes the output an audio file; see
# `resolve_output`, which also settles an unsuitable container.
lock = await _acquire_transcode_lock()
try:
return await _transcode_video_file(video_path, target_vcodec, target_acodec, container)
return await _transcode_video_file(video_path, target_vcodec, target_acodec, container, duration,
keep_original, fragmented)
finally:
lock.release()
async def probe_for_verify(path: pathlib.Path) -> dict:
"""ffprobe a file for `verify_transcode_output`. Separate so tests can replace it."""
return await ffprobe_json(path)
def _format_names(probe: dict) -> List[str]:
return str((probe.get('format') or {}).get('format_name') or '').split(',')
def _duration(probe: dict) -> Optional[float]:
try:
return float((probe.get('format') or {}).get('duration'))
except (TypeError, ValueError):
return None
def verify_transcode_output(source: dict, output: dict, target_vcodec: Optional[str],
target_acodec: Optional[str], container: str):
"""Compare ffprobe results of the source and the transcoded output; the output must be a
complete rendition of the source before the source is replaced by it.
Checks: the output parses (its index was written), it is in the requested container, it has a
video stream (and an audio stream when the source has one), each stream's codec is the target
or, when copied, the source's, and the duration matches within a tolerance.
@raise RuntimeError: describing the first failed check.
"""
if not output or output.get('streams') is None:
raise RuntimeError('Transcode output could not be probed (no streams); its index may be missing')
expected_format = CONTAINER_FORMAT_NAMES.get(container, container)
if expected_format not in _format_names(output):
raise RuntimeError(f'Transcode output container is {_format_names(output)}, expected {container}')
source_video = get_stream_codec_names(source, 'video')
source_audio = get_stream_codec_names(source, 'audio')
output_video = get_stream_codec_names(output, 'video')
output_audio = get_stream_codec_names(output, 'audio')
audio_only = target_vcodec == REMOVE_VIDEO or not source_video
if audio_only:
if output_video:
raise RuntimeError(f'Transcode output still has a video stream ({output_video[0]})')
if not output_audio:
raise RuntimeError('Transcode output has no audio stream')
else:
if not output_video:
raise RuntimeError('Transcode output has no video stream')
if source_audio and not output_audio:
raise RuntimeError('Transcode output has no audio stream, but the source has one')
expected_video = target_vcodec or (source_video[0] if source_video else None)
if expected_video and output_video[0] != expected_video:
raise RuntimeError(f'Transcode output video codec is {output_video[0]}, expected {expected_video}')
expected_audio = target_acodec or (source_audio[0] if source_audio else None)
if output_audio and expected_audio and output_audio[0] != expected_audio:
raise RuntimeError(f'Transcode output audio codec is {output_audio[0]}, expected {expected_audio}')
source_duration, output_duration = _duration(source), _duration(output)
if source_duration is not None:
if output_duration is None:
raise RuntimeError('Transcode output has no duration')
tolerance = max(DURATION_TOLERANCE_SECONDS, DURATION_TOLERANCE_RATIO * source_duration)
if abs(source_duration - output_duration) > tolerance:
raise RuntimeError(
f'Transcode output duration is {output_duration:.1f}s, source is {source_duration:.1f}s')
async def _verify_tail_decodes(path: pathlib.Path, runner: Callable):
"""The same completeness check a download gets (`ffmpeg_video_complete`), as a hard failure.
@raise RuntimeError: when the tail of `path` does not decode cleanly."""
ok, errors = await ffmpeg_video_complete_async(path, runner=runner)
if not ok:
raise RuntimeError(f'Transcode output failed to decode: {errors[-2000:] or "ffmpeg exited non-zero"}')
def make_ffmpeg_progress_parser(duration: Optional[float], set_progress: Callable[[float], None]) \
-> Callable[[str], None]:
"""Return a callback for `ffmpeg -progress pipe:1` stdout lines which reports percent complete.
ffmpeg emits `out_time_us=<microseconds>` (older builds: `out_time_ms`, also microseconds)."""
last = [-1]
def parse(line: str):
if not duration or duration <= 0:
return
key, sep, value = line.partition('=')
if not sep or key not in ('out_time_us', 'out_time_ms'):
return
try:
seconds = int(value) / 1_000_000
except ValueError:
return
percent = int(min(100.0, max(0.0, 100.0 * seconds / duration)))
if percent != last[0]:
last[0] = percent
set_progress(percent)
return parse
def resolve_output(source_probe: dict, target_vcodec: Optional[str], target_acodec: Optional[str],
container: str) -> Tuple[bool, str]:
"""Decide whether the output is audio-only and which container it gets.
Audio-only when the video is to be removed, or the source has no video stream. An audio
container that cannot hold the (copied or target) audio codec is refused; a video container
asked for an audio-only output is replaced by the codec's natural one.
@raise RuntimeError: when the request cannot be satisfied by this source."""
source_video = get_stream_codec_names(source_probe, 'video')
source_audio = get_stream_codec_names(source_probe, 'audio')
if target_vcodec and target_vcodec != REMOVE_VIDEO and not source_video:
raise RuntimeError(f'Cannot transcode video to {target_vcodec}: the file has no video stream')
audio_only = target_vcodec == REMOVE_VIDEO or not source_video
if not audio_only:
if container not in TRANSCODE_CONTAINERS:
logger.info(f'Forcing mp4 container ({container} cannot hold a video stream)')
container = 'mp4'
return False, container
if not source_audio:
raise RuntimeError('Cannot produce an audio file: the source has no audio stream')
effective_acodec = target_acodec or source_audio[0]
if container not in TRANSCODE_AUDIO_CONTAINERS:
container = default_audio_container(effective_acodec)
logger.info(f'Using {container} container for audio-only output ({effective_acodec})')
elif effective_acodec not in AUDIO_CONTAINER_CODECS[container]:
raise RuntimeError(
f'{effective_acodec} audio cannot be stored in {container}; choose another container or codec')
return True, container
async def _transcode_video_file(video_path: pathlib.Path, target_vcodec: Optional[str],
target_acodec: Optional[str], container: str) -> pathlib.Path:
target_acodec: Optional[str], container: str,
duration: Optional[float] = None, keep_original: bool = False,
fragmented: bool = False) -> pathlib.Path:
"""The ffmpeg work of `transcode_video_file`; the caller holds the transcode lock."""
# Notify here, after the lock: the wait for another transcode can last hours, and the user
# should hear "Transcoding" only when this file's work actually begins.
@ -158,28 +340,63 @@ async def _transcode_video_file(video_path: pathlib.Path, target_vcodec: Optiona
raise RuntimeError(
f'Not enough free space to transcode {video_path} ({free} bytes free, source is {source_size} bytes)')
video_args = TRANSCODE_VIDEO_TARGETS[target_vcodec] if target_vcodec else ('-c:v', 'copy')
# Probed before ffmpeg: the command depends on the source's streams, and verification compares
# against the file as it was.
source_probe = await probe_for_verify(video_path)
audio_only, container = resolve_output(source_probe, target_vcodec, target_acodec, container)
audio_args = TRANSCODE_AUDIO_TARGETS[target_acodec] if target_acodec else ('-c:a', 'copy')
if audio_only:
# `-vn` also drops embedded cover art; the audio stream is required.
map_args = ('-map', '0:a:0', '-vn')
video_args = ()
else:
# 0:V:0 excludes attached-picture streams (embedded thumbnails); 0:a:0? tolerates a video
# with no audio stream.
map_args = ('-map', '0:V:0', '-map', '0:a:0?')
video_args = TRANSCODE_VIDEO_TARGETS[target_vcodec] if target_vcodec else ('-c:v', 'copy')
if container in FASTSTART_CONTAINERS and fragmented:
mux_args = ('-movflags', FRAGMENTED_MOVFLAGS, '-frag_duration', str(FRAGMENT_DURATION_US))
elif container in FASTSTART_CONTAINERS:
mux_args = ('-movflags', '+faststart')
else:
mux_args = ()
final_path = video_path.with_suffix(f'.{container}')
tmp_path = video_path.with_suffix(f'.transcode.{container}')
# 0:V:0 excludes attached-picture streams (embedded thumbnails); 0:a:0? tolerates a video
# with no audio stream.
cmd = (FFMPEG_BIN, '-y',
'-i', video_path,
'-map', '0:V:0', '-map', '0:a:0?',
*map_args,
*video_args,
*audio_args,
'-movflags', '+faststart',
*mux_args,
# Machine-readable progress on stdout (the human progress line is suppressed).
'-nostats', '-progress', 'pipe:1',
str(tmp_path))
logger.warning(f'Transcoding {video_path} to {final_path} (video={target_vcodec}, audio={target_acodec})')
try:
result = await run_command(cmd, cwd=video_path.parent, timeout=TRANSCODE_TIMEOUT)
job = get_current_job()
runner = job.run_command if job else run_command
if job:
# Inside a Job: ffmpeg's output lands in the Job log and progress is reported.
result = await job.run_command(cmd, cwd=video_path.parent, timeout=TRANSCODE_TIMEOUT,
stdout_callback=make_ffmpeg_progress_parser(duration, job.set_progress))
else:
result = await run_command(cmd, cwd=video_path.parent, timeout=TRANSCODE_TIMEOUT)
if result.return_code != 0:
raise RuntimeError(
f'ffmpeg exited with {result.return_code} while transcoding {video_path}:'
f'\n{result.stderr.decode()[-2000:]}')
except Exception:
# The original is only replaced by a complete output: compare probes, then prove the tail
# decodes. A failure here leaves the original untouched.
logger.info(f'Verifying transcode output {tmp_path}')
output_probe = await probe_for_verify(tmp_path)
verify_transcode_output(source_probe, output_probe, target_vcodec, target_acodec, container)
await _verify_tail_decodes(tmp_path, runner)
except BaseException:
# BaseException: a cancelled Job's run_command raises CancelledError, which must not leave
# the partial output behind for a refresh to index.
tmp_path.unlink(missing_ok=True)
raise
@ -194,7 +411,7 @@ async def _transcode_video_file(video_path: pathlib.Path, target_vcodec: Optiona
# The old ffprobe sidecar describes the old streams.
video_path.with_suffix('.ffprobe.json').unlink(missing_ok=True)
tmp_path.rename(final_path)
if final_path != video_path:
if final_path != video_path and not keep_original:
video_path.unlink()
final_path.chmod(DEFAULT_FILE_PERMISSIONS)
@ -207,3 +424,89 @@ async def _transcode_video_file(video_path: pathlib.Path, target_vcodec: Optiona
finally:
os.close(fd)
return final_path
def validate_transcode_request(video_codec: Optional[str], audio_codec: Optional[str], container: str,
fragmented: bool = False):
"""@raise ValueError: when the request cannot be transcoded. No codec at all is a remux;
`video_codec=REMOVE_VIDEO` drops the video stream and needs an audio container; `fragmented`
needs an mp4 container."""
if video_codec and video_codec != REMOVE_VIDEO and video_codec not in TRANSCODE_VIDEO_TARGETS:
raise ValueError(f'Cannot transcode video to {video_codec!r}; supported: {sorted(TRANSCODE_VIDEO_TARGETS)}')
if audio_codec and audio_codec not in TRANSCODE_AUDIO_TARGETS:
raise ValueError(f'Cannot transcode audio to {audio_codec!r}; supported: {sorted(TRANSCODE_AUDIO_TARGETS)}')
containers = TRANSCODE_CONTAINERS + TRANSCODE_AUDIO_CONTAINERS
if container not in containers:
raise ValueError(f'Unsupported container {container!r}; supported: {containers}')
if fragmented and container not in FASTSTART_CONTAINERS:
raise ValueError(f'A fragmented file must be mp4 (or m4a), not {container}')
if video_codec == REMOVE_VIDEO and container not in TRANSCODE_AUDIO_CONTAINERS:
raise ValueError(f'Removing the video needs an audio container ({TRANSCODE_AUDIO_CONTAINERS}), not {container}')
if container in TRANSCODE_AUDIO_CONTAINERS:
if video_codec and video_codec != REMOVE_VIDEO:
raise ValueError(f'A video stream cannot be stored in {container}; remove the video or choose a video container')
if audio_codec and audio_codec not in AUDIO_CONTAINER_CODECS[container]:
raise ValueError(f'{audio_codec} audio cannot be stored in {container}; '
f'it holds {sorted(AUDIO_CONTAINER_CODECS[container])}')
def retarget_file_group(file_group_id: int, old_path: pathlib.Path, new_path: pathlib.Path):
"""Point a Video's FileGroup at the transcoded file and mark it for re-modeling. Committed."""
from modules.videos.models import Video
from wrolpi.files.lib import get_mimetype
with get_db_session(commit=True) as session:
video = Video.find_by_file_group_id(session, file_group_id)
file_group = video.file_group
if new_path != old_path:
mimetype = get_mimetype(new_path)
file_group.files = [
{**i, 'path': new_path.name, 'mimetype': mimetype} if i['path'] == old_path.name else i
for i in file_group.files
]
file_group.primary_path = new_path
file_group.mimetype = mimetype
# The streams changed: the modeler must ffprobe the file again.
video.ffprobe_json = None
file_group.indexed = False
@register_job('transcode_video')
async def transcode_video_job(file_group_id: int, video_codec: Optional[str] = None,
audio_codec: Optional[str] = None, container: str = 'mp4',
fragmented: bool = False) -> dict:
"""Transcode a Video's file, then re-index it so the FileGroup reflects the new file.
Run as a Job (`transcode_video_job.enqueue(...)`) so the user can watch and cancel it."""
from modules.videos.models import Video
from wrolpi.files.worker import file_worker
# The API refuses to queue in WROL Mode; this catches a Job queued before it was enabled.
if wrol_mode_enabled():
raise RuntimeError('Cannot transcode while WROL Mode is enabled')
validate_transcode_request(video_codec, audio_codec, container, fragmented)
with get_db_session() as session:
video = Video.find_by_file_group_id(session, file_group_id)
video_path = video.video_path
duration = video.file_group.length
if not video_path or not video_path.is_file():
raise RuntimeError(f'Video file does not exist: {video_path}')
# The original is kept until the database points at the new file (below).
final_path = await transcode_video_file(video_path, video_codec, audio_codec, container, duration=duration,
keep_original=True, fragmented=fragmented)
# Point the existing FileGroup at the new file *before* deleting the original and *before*
# refreshing. A crash after the delete would leave the database pointing at a missing file;
# a refresh which finds a new filename on disk would delete this FileGroup and create another,
# losing tags and history.
retarget_file_group(file_group_id, video_path, final_path)
if final_path != video_path:
video_path.unlink(missing_ok=True)
job_id = file_worker.queue_refresh([final_path], send_events=False)
await file_worker.wait_for_job(job_id)
return {'path': str(get_relative_to_media_directory(final_path))}

54
modules/videos/video/api.py
View file

@ -7,11 +7,12 @@ from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.api_utils import json_response
from wrolpi.common import logger
from wrolpi.errors import InvalidOrderBy
from wrolpi.common import logger, wrol_mode_check
from wrolpi.errors import InvalidOrderBy, InvalidJob
from wrolpi.schema import JSONErrorResponse
from . import lib
from .. import schema
from ..transcode import transcode_video_job, validate_transcode_request, REMOVE_VIDEO
video_bp = Blueprint('Video', '/api/videos')
@ -28,6 +29,20 @@ def video_get(request: Request, file_group_id: int):
return json_response({'file_group': video, 'prev': previous_video, 'next': next_video})
@video_bp.put('/<file_group_id:int>')
@openapi.description('Edit a Video\'s details. Written to its info json; the Video is validated again.')
@openapi.body({'application/json': schema.VideoUpdateRequest})
@openapi.response(HTTPStatus.OK, schema.VideoResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@validate(schema.VideoUpdateRequest)
@wrol_mode_check
async def video_update(_: Request, file_group_id: int, body: schema.VideoUpdateRequest):
lib.update_video(file_group_id, title=body.title, description=body.description)
video, previous_video, next_video = lib.get_video_for_app(file_group_id)
return json_response({'file_group': video, 'prev': previous_video, 'next': next_video})
@video_bp.get('/<file_group_id:int>/comments')
@openapi.description('Get Video comments')
@openapi.response(HTTPStatus.OK, schema.VideoCommentsResponse)
@ -55,6 +70,41 @@ def video_get_captions(_: Request, file_group_id: int):
return json_response({'captions': video.get_caption_chunks()})
@video_bp.post('/<file_group_id:int>/transcode')
@openapi.description('Queue a Job which transcodes the Video to the requested codecs/container.')
@openapi.body({'application/json': schema.VideoTranscodeRequest})
@openapi.response(HTTPStatus.OK, schema.VideoTranscodeResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@validate(schema.VideoTranscodeRequest)
@wrol_mode_check
async def video_transcode(_: Request, file_group_id: int, body: schema.VideoTranscodeRequest):
try:
validate_transcode_request(body.video_codec, body.audio_codec, body.container, body.fragmented)
except ValueError as e:
raise InvalidJob(str(e))
# Raises UnknownVideo (404) before anything is queued.
video = lib.get_video(file_group_id)
is_audio = (video.file_group.mimetype or '').startswith('audio/')
if is_audio and body.video_codec and body.video_codec != REMOVE_VIDEO:
raise InvalidJob(f'This is an audio file; it has no video stream to transcode to {body.video_codec}')
if body.video_codec == REMOVE_VIDEO and not is_audio:
action = 'Extract audio from'
elif not body.video_codec and not body.audio_codec:
action = 'Remux'
else:
action = 'Transcode'
job_id = transcode_video_job.enqueue(
description=f'{action} {video.video_path.name}',
file_group_id=file_group_id,
video_codec=body.video_codec,
audio_codec=body.audio_codec,
container=body.container,
fragmented=body.fragmented,
)
return json_response({'job_id': job_id})
@video_bp.post('/search')
@openapi.definition(
summary='Search Video titles and captions',

40
modules/videos/video/lib.py
View file

@ -50,6 +50,46 @@ def get_video(file_group_id: int) -> Video:
return video
@wrol_mode_check
def update_video(file_group_id: int, title: str = None, description: str = None) -> Video:
"""Edit a Video's details. The files are the source of truth, so the change is written to the
Video's .info.json (created if missing) and the Video is validated again from it. Only the
fields given (not None) are changed; an empty description clears it.
Edits go into the info json's `wrolpi` section (`custom_title`, `custom_description`), never
over yt-dlp's own keys: the original title and description stay in the file, and a metadata
re-download carries the section over.
@raise UnknownVideo: if the Video can not be found
@raise ValidationError: when nothing is given, or the title is empty
"""
from wrolpi.errors import ValidationError
if title is None and description is None:
raise ValidationError('Nothing to change')
if title is not None:
title = title.strip()
if not title:
raise ValidationError('Video title cannot be empty')
changes = dict()
if title is not None:
changes['custom_title'] = title
if description is not None:
changes['custom_description'] = description
with get_db_session(commit=True) as session:
video = Video.find_by_file_group_id(session, file_group_id)
if video.info_json_path is None or not video.info_json_path.is_file():
# No info json yet: create one holding only the wrolpi section.
video.replace_info_json({'wrolpi': changes}, clean=False)
else:
video.file_group.update_wrolpi_json(changes)
# Re-derive the title (and search text) from the file just written.
video.validate(session)
video.flush()
return video
VIDEO_ORDERS = {
# fg.id (the rowid) is the pagination tiebreaker: it is present in every index, so these
# orders stay index-only. LOWER(fg.primary_path) forced SQLite to read every (large) row.