wrolpi/modules/videos/schema.py
lrnselfreliance 5358c75a2b
Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:25:23 -06:00

305 lines
6.3 KiB
Python

from dataclasses import dataclass, field
from typing import List, Optional
from modules.videos.video.lib import DEFAULT_VIDEO_ORDER, VIDEO_QUERY_LIMIT
from wrolpi.errors import ValidationError
@dataclass
class ChannelPostRequest:
name: str
directory: str
calculate_duration: Optional[bool] = None
download_missing_data: Optional[bool] = True
generate_posters: Optional[bool] = None
source_id: Optional[str] = None
tag_name: Optional[str] = None
url: Optional[str] = None
def __post_init__(self):
self.name = self.name.strip() or None
self.directory = self.directory.strip() or None
self.url = self.url.strip() if self.url else None
self.source_id = self.source_id.strip() if self.source_id else None
@dataclass
class ChannelPutRequest:
calculate_duration: Optional[bool] = None
directory: Optional[str] = None
download_missing_data: Optional[bool] = True
generate_posters: Optional[bool] = None
mkdir: Optional[bool] = None
name: Optional[str] = None
url: Optional[str] = None
def __post_init__(self):
self.name = self.name.strip() or None
self.directory = self.directory.strip() or None
self.url = self.url.strip() if self.url else None
@dataclass
class ChannelTagRequest:
tag_name: Optional[str] = None
directory: Optional[str] = None
def __post_init__(self):
if self.directory and self.directory.startswith('/'):
raise ValidationError('Directory must be relative to media directory')
@dataclass
class ChannelTagInfoRequest:
channel_id: Optional[int] = None
tag_name: Optional[str] = None
@dataclass
class ChannelDownloadRequest:
url: str
frequency: int
settings: dict = field(default_factory=dict)
def __post_init__(self):
self.url = self.url.strip()
if not self.url:
raise ValidationError('url cannot be empty')
# Validate settings contents. Remove empty values.
from wrolpi.schema import DownloadSettings
self.settings = {k: v for k, v in DownloadSettings(**self.settings).__dict__.items() if v not in ([], None)}
@dataclass
class RefreshRequest:
channel_name: str
@dataclass
class ChannelModel:
id: int
url: str
name: str
directory: str
@dataclass
class ChannelResponse:
channel = ChannelModel
@dataclass
class ChannelPostResponse:
success: str
@dataclass
class ChannelsChannelModel:
id: int
name: str
@dataclass
class ChannelsResponse:
channels: List[ChannelsChannelModel]
@dataclass
class StreamResponse:
success: str
stream_url: str
@dataclass
class Video:
id: int
description_path: str
ext: str
poster_path: str
source_id: str
title: str
video_path: str
name: str
caption_path: str
idempotency: str
info_json_path: str
channel_id: int
@dataclass
class VideoWithChannel:
id: int
description_path: str
ext: str
poster_path: str
source_id: str
title: str
video_path: str
name: str
caption_path: str
idempotency: str
info_json_path: str
channel_id: int
channel = ChannelModel
@dataclass
class ExtendedVideo(Video):
info_json: dict
@dataclass
class VideoResponse:
file_group = ExtendedVideo
prev = ExtendedVideo
next = ExtendedVideo
@dataclass
class VideoCommentsResponse:
comments: list[dict]
@dataclass
class VideoCaptionsResponse:
captions: list
@dataclass
class VideoDescriptionResponse:
description: Optional[str]
@dataclass
class VideoSearchRequest:
search_str: Optional[str] = None
tags: List[str] = field(default_factory=list)
offset: Optional[int] = None
limit: Optional[int] = VIDEO_QUERY_LIMIT
order_by: Optional[str] = DEFAULT_VIDEO_ORDER
channel_id: Optional[int] = None
tag_names: List[str] = field(default_factory=list)
headline: bool = False
censored: bool = False
deep: bool = False # When True, search all text including captions (d_text)
@dataclass
class VideoUpdateRequest:
# Only the fields given are changed.
title: Optional[str] = None
description: Optional[str] = None
@dataclass
class VideoTranscodeRequest:
video_codec: Optional[str] = None # A key of TRANSCODE_VIDEO_TARGETS, or None to copy the stream.
audio_codec: Optional[str] = None # A key of TRANSCODE_AUDIO_TARGETS, or None to copy the stream.
container: str = 'mp4'
fragmented: bool = False # Fragmented mp4: quick start/seek for very long videos (mp4/m4a only).
@dataclass
class VideoTranscodeResponse:
job_id: str
@dataclass
class VideoSearchResponse:
videos: List[VideoWithChannel]
tsquery: str
@dataclass
class PaginationQuery:
offset: int
@dataclass
class TagRequest:
video_id: int
tag_name: str
@dataclass
class WROLModeRequest:
enabled: bool
@dataclass
class VideoStatistics:
videos: int
week: int
month: int
year: int
sum_duration: int
sum_size: int
max_size: int
have_comments: int
missing_comments: int
failed_comments: int
@dataclass
class ChannelStatistics:
channels: int
tagged_channels: int
@dataclass
class VideosStatisticsResponse:
videos = VideoStatistics
channels = ChannelStatistics
@dataclass
class CensoredVideoRequest:
limit: int
offset: int
@dataclass
class CensoredVideoResponse:
videos: List[VideoWithChannel]
@dataclass
class ChannelSearchRequest:
tag_names: List[str] = field(default_factory=list)
@dataclass
class VideoFileFormatRequest:
video_file_format: str
@dataclass
class CookiesUploadRequest:
"""Request to upload and encrypt cookies."""
cookies_content: str
password: str
def __post_init__(self):
if not self.cookies_content:
raise ValidationError('cookies_content cannot be empty')
if not self.password or len(self.password) < 8:
raise ValidationError('password must be at least 8 characters')
@dataclass
class CookiesUnlockRequest:
"""Request to unlock cookies with password."""
password: str
def __post_init__(self):
if not self.password:
raise ValidationError('password cannot be empty')
@dataclass
class CookiesStatusResponse:
"""Response with cookies status."""
cookies_exist: bool
cookies_unlocked: bool