wrolpi/modules/docs/api.py

65 lines
1.8 KiB
Python
Raw Normal View History

Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678) * Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page Three response-time fixes found by benchmarking the read-only API endpoints hot and cold on real libraries: * `switch_worker` runs on the Sanic worker's event loop every 0.1s and called the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending (the normal state) this froze the loop for a second at a time; about a third of all requests, including trivial ones, took ~1,000 ms. It now polls `is_set()` and returns immediately. * The search handlers (videos, files, archives, docs, zims) and `/api/statistics` ran synchronous SQLite queries on the event loop, so one slow search stalled every other request on that worker. They now run via `asyncio.to_thread`. * The unfiltered Videos page (`POST /api/videos/search` with only an order) used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits every video's index entry before LIMIT applies: seconds cold on a large library on a spinning disk, and ~10 s hot for a deep page. It now seeks each exact mimetype on the (mimetype, effective_datetime) index for `limit + offset` rows and merges them, with the total from a separate count cached for 60 s. Results are identical to the generic query. Also adds `scripts/api_perf.py` (times every read-only endpoint against a host, optionally restarting the API over SSH for cold numbers) and `scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a gitignored JSON file; an example is included. Measured on an 8-core docker host with a 10 GB library: median hot request 1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve the video description from its own endpoint instead of Video.__json__ `Video.__json__` computed `description` as `c_text or get_video_description()`, where the fallback opens and parses the video's info json from disk. The list endpoints serialize every result with `__json__`, so any page containing videos without an indexed description (common: many videos have none) read their info json files on every request, cold, for a field the list never displays. The description is not part of search either: search runs against the FTS copy of `c_text`, so the fallback never fed the index. Drop `description` from `__json__` and serve it from `GET /api/videos/<id>/description`, like comments and captions. The endpoint returns the indexed `c_text`, falling back to the info json only there, for a row indexed before descriptions were stored. The player page fetches it through `useVideoExtras` alongside the comments. Tests assert the list and detail responses never touch the info json, and cover the endpoint's indexed, fallback, and not-found cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: thread-local zim sessions, fast-path equivalence test, comments * Zim search, single-zim search, and search estimates run libzim off the event loop, each opening its own session inside the worker thread rather than passing the request's SQLAlchemy Session across threads. * The Videos fast-path test now runs the same requests through the generic query and asserts identical rows and paging for both date orders. The one intentional difference is documented: past the end, the fast path still reports the true total. * switch_worker returns before its try block so the empty path does not clear the event; its comment states the invariant only. The copied "synchronous SQLite" comments are reduced to one line in one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:41 -06:00
import asyncio
from http import HTTPStatus
2026-05-03 18:50:22 -06:00
from sanic import Request, Blueprint
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.api_utils import json_response
2026-05-03 18:50:22 -06:00
from wrolpi.common import logger, api_param_limiter
from wrolpi.schema import JSONErrorResponse
from . import schema
2026-05-03 18:50:22 -06:00
from .lib import get_statistics, _doc_response, _get_doc, _search_docs
NAME = 'docs'
docs_bp = Blueprint('Docs', '/api/docs')
logger = logger.getChild(__name__)
@docs_bp.get('/statistics')
@openapi.response(HTTPStatus.OK, schema.DocStatisticsResponse)
async def statistics(_: Request):
ret = get_statistics()
return json_response(ret, HTTPStatus.OK)
@docs_bp.get('/<file_group_id:int>')
@openapi.description('Get a doc by FileGroup ID')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
async def get_doc(request: Request, file_group_id: int):
session = request.ctx.session
doc = _get_doc(session, file_group_id)
return json_response(_doc_response(doc))
doc_limit_limiter = api_param_limiter(100)
@docs_bp.post('/search')
@openapi.definition(
summary='Search docs with filtering',
body=schema.DocSearchRequest,
)
@openapi.response(HTTPStatus.OK, schema.DocSearchResponse)
@validate(schema.DocSearchRequest)
async def search_docs(_: Request, body: schema.DocSearchRequest):
limit = doc_limit_limiter(body.limit)
offset = body.offset or 0
Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678) * Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page Three response-time fixes found by benchmarking the read-only API endpoints hot and cold on real libraries: * `switch_worker` runs on the Sanic worker's event loop every 0.1s and called the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending (the normal state) this froze the loop for a second at a time; about a third of all requests, including trivial ones, took ~1,000 ms. It now polls `is_set()` and returns immediately. * The search handlers (videos, files, archives, docs, zims) and `/api/statistics` ran synchronous SQLite queries on the event loop, so one slow search stalled every other request on that worker. They now run via `asyncio.to_thread`. * The unfiltered Videos page (`POST /api/videos/search` with only an order) used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits every video's index entry before LIMIT applies: seconds cold on a large library on a spinning disk, and ~10 s hot for a deep page. It now seeks each exact mimetype on the (mimetype, effective_datetime) index for `limit + offset` rows and merges them, with the total from a separate count cached for 60 s. Results are identical to the generic query. Also adds `scripts/api_perf.py` (times every read-only endpoint against a host, optionally restarting the API over SSH for cold numbers) and `scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a gitignored JSON file; an example is included. Measured on an 8-core docker host with a 10 GB library: median hot request 1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve the video description from its own endpoint instead of Video.__json__ `Video.__json__` computed `description` as `c_text or get_video_description()`, where the fallback opens and parses the video's info json from disk. The list endpoints serialize every result with `__json__`, so any page containing videos without an indexed description (common: many videos have none) read their info json files on every request, cold, for a field the list never displays. The description is not part of search either: search runs against the FTS copy of `c_text`, so the fallback never fed the index. Drop `description` from `__json__` and serve it from `GET /api/videos/<id>/description`, like comments and captions. The endpoint returns the indexed `c_text`, falling back to the info json only there, for a row indexed before descriptions were stored. The player page fetches it through `useVideoExtras` alongside the comments. Tests assert the list and detail responses never touch the info json, and cover the endpoint's indexed, fallback, and not-found cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: thread-local zim sessions, fast-path equivalence test, comments * Zim search, single-zim search, and search estimates run libzim off the event loop, each opening its own session inside the worker thread rather than passing the request's SQLAlchemy Session across threads. * The Videos fast-path test now runs the same requests through the generic query and asserts identical rows and paging for both date orders. The one intentional difference is documented: past the end, the fast path still reports the true total. * switch_worker returns before its try block so the empty path does not clear the event; its comment states the invariant only. The copied "synchronous SQLite" comments are reduced to one line in one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:41 -06:00
file_groups, total = await asyncio.to_thread(
_search_docs,
search_str=body.search_str,
author=body.author,
subject=body.subject,
language=body.language,
mimetype=body.mimetype,
limit=limit,
offset=offset,
order_by=body.order_by,
tag_names=body.tag_names,
Search: fast ABC mode by default, opt-in deep search The file_group.textsearch tsvector indexes all four text columns; d_text (captions, document text, zip contents) is ~90% of the data, making ts_rank searches take seconds on large libraries. Add a textsearch_abc generated column (title/author/description only, ~8x smaller) and default all searches to it, with deep=true opting into the full column. Database: - Migration adds the generated column and a GIN index (rewrites the table; a few minutes on large libraries). Backend: - FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and DocSearchRequest accept `deep`; each search function picks the column. - /api/search_file_estimates returns both counts in one index-bound query (COUNT FILTER on the abc subset): `file_groups` (fast) and `file_groups_deep`. UI (one coherent story across the count surfaces): - The typing dropdown always suggests the deep count (what is findable). - The tab badge shows the active mode's count so it matches the result list. - The gap is reconciled by a hint offering deep search: count-aware on the global search ("No results in fast search. N results available in deep search." / "N more results available..."), generic on the module pages (no deep estimate there). The hint resets pagination. - "Search Depth" toggle in the search filter modals, persisted as ?deep=true. Tests that relied on d_text matches now pass deep=True, with fast-mode assertions alongside. Also skip flaky test_download_channel on CircleCI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 10:09:08 -06:00
deep=body.deep,
)
ret = dict(file_groups=file_groups, totals=dict(file_groups=total))
return json_response(ret)