mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 02:48:38 -04:00
97a6566e3d
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page Three response-time fixes found by benchmarking the read-only API endpoints hot and cold on real libraries: * `switch_worker` runs on the Sanic worker's event loop every 0.1s and called the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending (the normal state) this froze the loop for a second at a time; about a third of all requests, including trivial ones, took ~1,000 ms. It now polls `is_set()` and returns immediately. * The search handlers (videos, files, archives, docs, zims) and `/api/statistics` ran synchronous SQLite queries on the event loop, so one slow search stalled every other request on that worker. They now run via `asyncio.to_thread`. * The unfiltered Videos page (`POST /api/videos/search` with only an order) used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits every video's index entry before LIMIT applies: seconds cold on a large library on a spinning disk, and ~10 s hot for a deep page. It now seeks each exact mimetype on the (mimetype, effective_datetime) index for `limit + offset` rows and merges them, with the total from a separate count cached for 60 s. Results are identical to the generic query. Also adds `scripts/api_perf.py` (times every read-only endpoint against a host, optionally restarting the API over SSH for cold numbers) and `scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a gitignored JSON file; an example is included. Measured on an 8-core docker host with a 10 GB library: median hot request 1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve the video description from its own endpoint instead of Video.__json__ `Video.__json__` computed `description` as `c_text or get_video_description()`, where the fallback opens and parses the video's info json from disk. The list endpoints serialize every result with `__json__`, so any page containing videos without an indexed description (common: many videos have none) read their info json files on every request, cold, for a field the list never displays. The description is not part of search either: search runs against the FTS copy of `c_text`, so the fallback never fed the index. Drop `description` from `__json__` and serve it from `GET /api/videos/<id>/description`, like comments and captions. The endpoint returns the indexed `c_text`, falling back to the info json only there, for a row indexed before descriptions were stored. The player page fetches it through `useVideoExtras` alongside the comments. Tests assert the list and detail responses never touch the info json, and cover the endpoint's indexed, fallback, and not-found cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: thread-local zim sessions, fast-path equivalence test, comments * Zim search, single-zim search, and search estimates run libzim off the event loop, each opening its own session inside the worker thread rather than passing the request's SQLAlchemy Session across threads. * The Videos fast-path test now runs the same requests through the generic query and asserts identical rows and paging for both date orders. The one intentional difference is documented: past the end, the fast path still reports the true total. * switch_worker returns before its try block so the empty path does not clear the event; its comment states the invariant only. The copied "synchronous SQLite" comments are reduced to one line in one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
152 lines
2 KiB
Text
152 lines
2 KiB
Text
# Local blobs directory (planet overview, etc.)
|
|
/blobs/
|
|
|
|
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
db.sqlite3
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/_build/
|
|
|
|
# PyBuilder
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# pyenv
|
|
.python-version
|
|
|
|
# celery beat schedule file
|
|
celerybeat-schedule
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
|
|
# intellij
|
|
.idea/
|
|
|
|
# example files
|
|
test/videos
|
|
test/archive
|
|
test/map
|
|
|
|
# docker configs
|
|
test/config
|
|
|
|
modules/node_modules/
|
|
docker/archive/node_modules/
|
|
docker/archive/package-lock.json
|
|
|
|
# open street map files
|
|
*.pbf
|
|
mapnik.xml
|
|
|
|
# docker override
|
|
docker-compose.override.yml
|
|
|
|
# test directory is used as media directory, we don't want to commit what a user downloads.
|
|
/test
|
|
# Legacy local Postgres data directory from pre-SQLite docker-compose (safe to delete).
|
|
pg_data
|
|
|
|
# Directories used to build images
|
|
/debian-live-config/config/includes.chroot/opt/wrolpi-blobs/map-db-gis.dump
|
|
/debian-live-config/build.log
|
|
/debian-live-config/*.iso
|
|
/pi-gen/*xz
|
|
|
|
.DS_Store
|
|
app/cypress/screenshots
|
|
app/cypress/videos
|
|
|
|
.claude
|
|
*.md
|
|
/.mcp.json
|
|
|
|
# Generated by release/release.sh before upload
|
|
release/latest.json
|
|
perf_results/
|
|
scripts/api_perf_hosts.json
|