wrolpi/modules/videos/schema.py
lrnselfreliance 97a6566e3d
Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678)
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page

Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:

* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
  the blocking `multiprocessing.Event.wait(timeout=1)`.  With no switch pending
  (the normal state) this froze the loop for a second at a time; about a third
  of all requests, including trivial ones, took ~1,000 ms.  It now polls
  `is_set()` and returns immediately.

* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
  ran synchronous SQLite queries on the event loop, so one slow search stalled
  every other request on that worker.  They now run via `asyncio.to_thread`.

* The unfiltered Videos page (`POST /api/videos/search` with only an order)
  used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
  every video's index entry before LIMIT applies: seconds cold on a large
  library on a spinning disk, and ~10 s hot for a deep page.  It now seeks each
  exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
  rows and merges them, with the total from a separate count cached for 60 s.
  Results are identical to the generic query.

Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results).  Host profiles live in a
gitignored JSON file; an example is included.

Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Serve the video description from its own endpoint instead of Video.__json__

`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk.  The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays.  The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.

Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions.  The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored.  The player page fetches it through
`useVideoExtras` alongside the comments.

Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: thread-local zim sessions, fast-path equivalence test, comments

* Zim search, single-zim search, and search estimates run libzim off the event
  loop, each opening its own session inside the worker thread rather than
  passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
  query and asserts identical rows and paging for both date orders.  The one
  intentional difference is documented: past the end, the fast path still
  reports the true total.
* switch_worker returns before its try block so the empty path does not clear
  the event; its comment states the invariant only.  The copied "synchronous
  SQLite" comments are reduced to one line in one place.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:53 -06:00

285 lines
5.7 KiB
Python

from dataclasses import dataclass, field
from typing import List, Optional
from modules.videos.video.lib import DEFAULT_VIDEO_ORDER, VIDEO_QUERY_LIMIT
from wrolpi.errors import ValidationError
@dataclass
class ChannelPostRequest:
name: str
directory: str
calculate_duration: Optional[bool] = None
download_missing_data: Optional[bool] = True
generate_posters: Optional[bool] = None
source_id: Optional[str] = None
tag_name: Optional[str] = None
url: Optional[str] = None
def __post_init__(self):
self.name = self.name.strip() or None
self.directory = self.directory.strip() or None
self.url = self.url.strip() if self.url else None
self.source_id = self.source_id.strip() if self.source_id else None
@dataclass
class ChannelPutRequest:
calculate_duration: Optional[bool] = None
directory: Optional[str] = None
download_missing_data: Optional[bool] = True
generate_posters: Optional[bool] = None
mkdir: Optional[bool] = None
name: Optional[str] = None
url: Optional[str] = None
def __post_init__(self):
self.name = self.name.strip() or None
self.directory = self.directory.strip() or None
self.url = self.url.strip() if self.url else None
@dataclass
class ChannelTagRequest:
tag_name: Optional[str] = None
directory: Optional[str] = None
def __post_init__(self):
if self.directory and self.directory.startswith('/'):
raise ValidationError('Directory must be relative to media directory')
@dataclass
class ChannelTagInfoRequest:
channel_id: Optional[int] = None
tag_name: Optional[str] = None
@dataclass
class ChannelDownloadRequest:
url: str
frequency: int
settings: dict = field(default_factory=dict)
def __post_init__(self):
self.url = self.url.strip()
if not self.url:
raise ValidationError('url cannot be empty')
# Validate settings contents. Remove empty values.
from wrolpi.schema import DownloadSettings
self.settings = {k: v for k, v in DownloadSettings(**self.settings).__dict__.items() if v not in ([], None)}
@dataclass
class RefreshRequest:
channel_name: str
@dataclass
class ChannelModel:
id: int
url: str
name: str
directory: str
@dataclass
class ChannelResponse:
channel = ChannelModel
@dataclass
class ChannelPostResponse:
success: str
@dataclass
class ChannelsChannelModel:
id: int
name: str
@dataclass
class ChannelsResponse:
channels: List[ChannelsChannelModel]
@dataclass
class StreamResponse:
success: str
stream_url: str
@dataclass
class Video:
id: int
description_path: str
ext: str
poster_path: str
source_id: str
title: str
video_path: str
name: str
caption_path: str
idempotency: str
info_json_path: str
channel_id: int
@dataclass
class VideoWithChannel:
id: int
description_path: str
ext: str
poster_path: str
source_id: str
title: str
video_path: str
name: str
caption_path: str
idempotency: str
info_json_path: str
channel_id: int
channel = ChannelModel
@dataclass
class ExtendedVideo(Video):
info_json: dict
@dataclass
class VideoResponse:
file_group = ExtendedVideo
prev = ExtendedVideo
next = ExtendedVideo
@dataclass
class VideoCommentsResponse:
comments: list[dict]
@dataclass
class VideoCaptionsResponse:
captions: list
@dataclass
class VideoDescriptionResponse:
description: Optional[str]
@dataclass
class VideoSearchRequest:
search_str: Optional[str] = None
tags: List[str] = field(default_factory=list)
offset: Optional[int] = None
limit: Optional[int] = VIDEO_QUERY_LIMIT
order_by: Optional[str] = DEFAULT_VIDEO_ORDER
channel_id: Optional[int] = None
tag_names: List[str] = field(default_factory=list)
headline: bool = False
censored: bool = False
deep: bool = False # When True, search all text including captions (d_text)
@dataclass
class VideoSearchResponse:
videos: List[VideoWithChannel]
tsquery: str
@dataclass
class PaginationQuery:
offset: int
@dataclass
class TagRequest:
video_id: int
tag_name: str
@dataclass
class WROLModeRequest:
enabled: bool
@dataclass
class VideoStatistics:
videos: int
week: int
month: int
year: int
sum_duration: int
sum_size: int
max_size: int
have_comments: int
missing_comments: int
failed_comments: int
@dataclass
class ChannelStatistics:
channels: int
tagged_channels: int
@dataclass
class VideosStatisticsResponse:
videos = VideoStatistics
channels = ChannelStatistics
@dataclass
class CensoredVideoRequest:
limit: int
offset: int
@dataclass
class CensoredVideoResponse:
videos: List[VideoWithChannel]
@dataclass
class ChannelSearchRequest:
tag_names: List[str] = field(default_factory=list)
@dataclass
class VideoFileFormatRequest:
video_file_format: str
@dataclass
class CookiesUploadRequest:
"""Request to upload and encrypt cookies."""
cookies_content: str
password: str
def __post_init__(self):
if not self.cookies_content:
raise ValidationError('cookies_content cannot be empty')
if not self.password or len(self.password) < 8:
raise ValidationError('password must be at least 8 characters')
@dataclass
class CookiesUnlockRequest:
"""Request to unlock cookies with password."""
password: str
def __post_init__(self):
if not self.password:
raise ValidationError('password cannot be empty')
@dataclass
class CookiesStatusResponse:
"""Response with cookies status."""
cookies_exist: bool
cookies_unlocked: bool