db: serializers read only loaded data (#777)

* db: serializers read only loaded data

Serializing a model must not query.  A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.

- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
  an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
  per serializer and lets the query run.  Tests record each trip as well, and the autouse
  `no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
  swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
  Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
  Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
  Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.

Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.

Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* db: serializer loads per query; the guard ignores transaction control

The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer.  Transaction-control statements are now ignored.

Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json.  A second refresh of 100 videos took 0.40 s instead of 0.16 s.

Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it).  Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection.  A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
lrnselfreliance 2026-10-04 18:28:08 -06:00 • committed by GitHub
parent 1f687fa825
commit cfbed9e4d7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
24 changed files with 371 additions and 72 deletions

View file

@ -340,7 +340,7 @@ def _doc_response(doc) -> dict:
def _get_doc(session, file_group_id: int):
from .models import Doc
doc = session.query(Doc).filter_by(file_group_id=file_group_id).one_or_none()
doc = session.query(Doc).options(*Doc.json_options()).filter_by(file_group_id=file_group_id).one_or_none()
if not doc:
raise ValidationError(f'Doc with file_group_id {file_group_id} not found')
return doc
@ -375,7 +375,7 @@ def _search_docs(session: Session, search_str=None, author=None, subject=None, l
# the much smaller a/b/c columns.
match = fts.translate_websearch(search_str, None if deep else fts.ABC_COLUMNS) if search_str else None
query = session.query(FileGroup).join(Doc, Doc.file_group_id == FileGroup.id)
query = session.query(FileGroup).options(*FileGroup.json_options()).join(Doc, Doc.file_group_id == FileGroup.id)
fts_sq = None
if match is not None:
@ -441,7 +441,8 @@ def _search_docs(session: Session, search_str=None, author=None, subject=None, l
browse_sql = _doc_browse_sql(order_by if order_by in DOC_BROWSE_ORDERS else 'id')
if browse_sql:
fg_ids = [i for i, in session.execute(text(browse_sql), dict(limit=limit, offset=offset)).fetchall()]
by_id = {fg.id: fg for fg in session.query(FileGroup).filter(FileGroup.id.in_(fg_ids))} if fg_ids else {}
by_id = {fg.id: fg for fg in session.query(FileGroup).options(*FileGroup.json_options())
.filter(FileGroup.id.in_(fg_ids))} if fg_ids else {}
file_groups = [by_id[i] for i in fg_ids]
else:
file_groups = query.offset(offset).limit(limit).all()

View file

@ -2,10 +2,10 @@ import pathlib
from typing import Optional
from sqlalchemy import Column, Integer, BigInteger, ForeignKey, String, Text, Index, or_
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm import relationship, Session, joinedload
from wrolpi.common import ModelHelper, Base
from wrolpi.db import get_db_session
from wrolpi.db import get_db_session, serializer
from wrolpi.files.models import FileGroup
EPUB_MIMETYPE = 'application/epub'
@ -65,6 +65,12 @@ class Doc(ModelHelper, Base):
path = str(self.file_group.primary_path) if self.file_group else 'None'
return f'<Doc id={self.id} path={repr(path)} file_group_id={self.file_group_id}>'
@staticmethod
def json_options() -> tuple:
"""Loader options for a query whose Docs will be serialized: `__json__` reads the FileGroup and its Tags."""
return joinedload(Doc.file_group).selectinload(FileGroup.tag_files),
@serializer
def __json__(self) -> dict:
d = self.file_group.__json__()
if d.get('data') and d['data'].get('doc_path'):