2024-08-06 18:13:21 -06:00
|
|
|
import pathlib
|
2022-04-08 09:57:04 -06:00
|
|
|
from pathlib import Path
|
2025-10-29 22:16:33 -06:00
|
|
|
from typing import List, Union
|
2021-11-01 16:42:52 -06:00
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
from sqlalchemy import or_, func, desc
|
2025-10-29 22:16:33 -06:00
|
|
|
from sqlalchemy.orm import Session, joinedload
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2024-08-07 08:36:33 -06:00
|
|
|
from wrolpi import flags
|
2025-10-29 22:16:33 -06:00
|
|
|
from wrolpi.collections import Collection
|
2024-09-15 15:33:07 -06:00
|
|
|
from wrolpi.common import logger, \
|
2024-08-15 19:32:01 -06:00
|
|
|
get_media_directory, wrol_mode_check, background_task
|
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
|
|
|
from wrolpi.db import get_db_session, session_curs
|
2024-12-11 11:52:33 -07:00
|
|
|
from wrolpi.downloader import save_downloads_config, download_manager, Download
|
2026-01-09 14:01:25 -07:00
|
|
|
from wrolpi.errors import APIError, ValidationError, FileWorkerConflict
|
2025-12-24 17:42:17 -07:00
|
|
|
from wrolpi.events import Events
|
2025-10-29 22:16:33 -06:00
|
|
|
from wrolpi.tags import Tag
|
2022-04-08 09:57:04 -06:00
|
|
|
from .. import schema
|
2021-11-01 16:42:52 -06:00
|
|
|
from ..common import check_for_channel_conflicts
|
2025-01-26 12:44:16 -07:00
|
|
|
from ..errors import UnknownChannel
|
2021-11-01 16:42:52 -06:00
|
|
|
from ..lib import save_channels_config
|
2024-07-20 15:35:16 -06:00
|
|
|
from ..models import Channel, Video
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2021-12-20 16:08:13 -07:00
|
|
|
logger = logger.getChild(__name__)
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
|
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
|
|
|
async def get_minimal_channels(session: Session) -> List[dict]:
|
2021-11-01 16:42:52 -06:00
|
|
|
"""
|
|
|
|
|
Get the minimum amount of information necessary about all channels.
|
|
|
|
|
"""
|
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
|
|
|
with session_curs(session) as curs:
|
2024-06-25 22:12:13 -06:00
|
|
|
# Get all channels, even if they don't have videos. Also get the minimum frequency download because this is the
|
|
|
|
|
# one that will consume the most resources.
|
2021-11-01 16:42:52 -06:00
|
|
|
stmt = '''
|
2025-08-16 12:03:58 -06:00
|
|
|
SELECT c.id,
|
2025-10-29 22:16:33 -06:00
|
|
|
col.name AS "name",
|
|
|
|
|
col.directory,
|
2025-08-16 12:03:58 -06:00
|
|
|
c.url,
|
2025-10-29 22:16:33 -06:00
|
|
|
t.name AS "tag_name",
|
2025-08-16 12:03:58 -06:00
|
|
|
c.video_count,
|
|
|
|
|
c.total_size,
|
|
|
|
|
c.minimum_frequency
|
|
|
|
|
FROM channel AS c
|
2025-10-29 22:16:33 -06:00
|
|
|
INNER JOIN collection col ON col.id = c.collection_id
|
|
|
|
|
LEFT JOIN tag t ON t.id = col.tag_id
|
2025-08-16 12:03:58 -06:00
|
|
|
'''
|
2021-11-01 16:42:52 -06:00
|
|
|
curs.execute(stmt)
|
2025-08-16 12:03:58 -06:00
|
|
|
logger.debug(stmt)
|
2023-05-10 22:59:16 -06:00
|
|
|
channels = sorted([dict(i) for i in curs.fetchall()], key=lambda i: i['name'].lower())
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
return channels
|
|
|
|
|
|
|
|
|
|
|
2021-12-27 16:43:52 -07:00
|
|
|
COD = Union[dict, Channel]
|
|
|
|
|
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
def get_channel(session: Session, *, channel_id: int = None, source_id: str = None, url: str = None,
|
|
|
|
|
directory: str = None, name: str = None, return_dict: bool = True) -> COD:
|
2021-12-20 16:08:13 -07:00
|
|
|
"""
|
2026-07-07 18:25:34 -06:00
|
|
|
Attempt to find a Channel using the provided params. The params are in order of reliability, except that
|
|
|
|
|
directory beats source_id/url: a Video's files must live in their Channel's directory, so the directory a file
|
|
|
|
|
is in reflects the user's intent and beats yt-dlp metadata.
|
2021-12-20 16:08:13 -07:00
|
|
|
|
|
|
|
|
Raises UnknownChannel if no channel is found.
|
|
|
|
|
"""
|
2022-04-08 09:57:04 -06:00
|
|
|
channel: COD = None # noqa
|
|
|
|
|
# Try to find the channel by the most reliable methods first.
|
2025-10-29 22:16:33 -06:00
|
|
|
# Always eagerly load Collection to ensure name/directory/tag_id are available
|
2022-04-08 09:57:04 -06:00
|
|
|
if channel_id:
|
2025-10-29 22:16:33 -06:00
|
|
|
channel = session.query(Channel).options(joinedload(Channel.collection)).filter_by(id=channel_id).one_or_none()
|
2022-04-08 09:57:04 -06:00
|
|
|
if not channel and directory:
|
|
|
|
|
directory = Path(directory)
|
|
|
|
|
if not directory.is_absolute():
|
|
|
|
|
directory = get_media_directory() / directory
|
2026-02-07 17:16:08 -07:00
|
|
|
directory = directory.absolute()
|
|
|
|
|
|
|
|
|
|
# Walk up the directory tree to find a matching channel
|
|
|
|
|
# This allows finding channels for files in subdirectories (e.g., year subdirs)
|
|
|
|
|
media_dir = get_media_directory()
|
|
|
|
|
current = directory
|
|
|
|
|
while current != media_dir and current != current.parent:
|
|
|
|
|
channel = session.query(Channel).options(joinedload(Channel.collection)).join(Collection).filter(
|
|
|
|
|
Collection.directory == str(current)).one_or_none()
|
|
|
|
|
if channel:
|
|
|
|
|
break
|
|
|
|
|
current = current.parent
|
2026-07-07 18:25:34 -06:00
|
|
|
if not channel and source_id:
|
|
|
|
|
channel = session.query(Channel).options(joinedload(Channel.collection)).filter_by(
|
|
|
|
|
source_id=source_id).one_or_none()
|
|
|
|
|
if not channel and url:
|
|
|
|
|
channel = session.query(Channel).options(joinedload(Channel.collection)).filter_by(url=url).one_or_none()
|
|
|
|
|
if not channel and url:
|
|
|
|
|
# Try to extract YouTube channel ID from the URL and match by source_id.
|
|
|
|
|
from modules.videos.common import get_youtube_channel_id
|
|
|
|
|
yt_channel_id = get_youtube_channel_id(url)
|
|
|
|
|
if yt_channel_id:
|
|
|
|
|
channel = session.query(Channel).options(joinedload(Channel.collection)).filter_by(
|
|
|
|
|
source_id=yt_channel_id).one_or_none()
|
2022-04-08 09:57:04 -06:00
|
|
|
if not channel and name:
|
2025-10-29 22:16:33 -06:00
|
|
|
channel = session.query(Channel).join(Collection).filter(Collection.name == name).one_or_none()
|
2022-04-08 09:57:04 -06:00
|
|
|
|
|
|
|
|
if not channel:
|
|
|
|
|
raise UnknownChannel(f'No channel matches {channel_id=} {source_id=} {url=} {directory=}')
|
|
|
|
|
|
|
|
|
|
logger.debug(f'Found {channel=} using {channel_id=} {source_id=} {url=} {directory=}')
|
2022-05-20 15:15:31 -06:00
|
|
|
session.refresh(channel)
|
2022-04-08 09:57:04 -06:00
|
|
|
if return_dict:
|
|
|
|
|
statistics = channel.get_statistics()
|
|
|
|
|
channel = channel.dict()
|
|
|
|
|
channel['statistics'] = statistics
|
|
|
|
|
return channel
|
2021-12-20 16:08:13 -07:00
|
|
|
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
async def update_channel(session: Session, *, data: schema.ChannelPutRequest, channel_id: int) -> Channel:
|
2022-01-27 20:47:52 -07:00
|
|
|
"""Update a Channel's DB record"""
|
2025-12-10 21:16:10 -07:00
|
|
|
channel = Channel.find_by_id(session, channel_id)
|
2022-04-08 09:57:04 -06:00
|
|
|
|
|
|
|
|
# Verify that the URL/Name/directory aren't taken
|
|
|
|
|
check_for_channel_conflicts(
|
|
|
|
|
session,
|
|
|
|
|
id_=channel.id,
|
|
|
|
|
url=data.url,
|
|
|
|
|
name=data.name,
|
|
|
|
|
directory=data.directory,
|
|
|
|
|
)
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
data = data.__dict__
|
|
|
|
|
old_directory = pathlib.Path(channel.directory)
|
|
|
|
|
new_directory = get_media_directory() / data.pop('directory')
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
# Apply the changes now that we've OK'd them
|
2024-08-15 19:32:01 -06:00
|
|
|
channel.update(data)
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
session.commit()
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
if new_directory != old_directory:
|
|
|
|
|
if not new_directory.is_dir():
|
|
|
|
|
new_directory.mkdir(parents=True)
|
|
|
|
|
|
2025-12-24 17:42:17 -07:00
|
|
|
# Always use background task with its own session to avoid
|
|
|
|
|
# DetachedInstanceError when the original session is closed.
|
|
|
|
|
background_task(_background_move_channel(channel.id, new_directory))
|
2024-08-15 19:32:01 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
save_channels_config.activate_switch()
|
|
|
|
|
|
2021-12-20 16:08:13 -07:00
|
|
|
return channel
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
def create_channel(session: Session, data: schema.ChannelPostRequest, return_dict: bool = True) -> Union[Channel, dict]:
|
2021-11-01 16:42:52 -06:00
|
|
|
"""
|
|
|
|
|
Create a new Channel. Check for conflicts with existing Channels.
|
|
|
|
|
"""
|
2022-04-08 09:57:04 -06:00
|
|
|
try:
|
|
|
|
|
# Verify that the URL/Name/directory aren't taken
|
|
|
|
|
check_for_channel_conflicts(
|
|
|
|
|
session,
|
2022-05-03 10:46:10 -06:00
|
|
|
url=data.url or None,
|
2022-04-08 09:57:04 -06:00
|
|
|
name=data.name,
|
|
|
|
|
directory=data.directory,
|
|
|
|
|
source_id=data.source_id,
|
2021-11-01 16:42:52 -06:00
|
|
|
)
|
2022-04-08 09:57:04 -06:00
|
|
|
except APIError as e:
|
|
|
|
|
raise ValidationError() from e
|
|
|
|
|
|
2024-09-17 18:04:00 -07:00
|
|
|
directory = pathlib.Path(data.directory)
|
|
|
|
|
directory = get_media_directory() / directory if not directory.is_absolute() else directory
|
|
|
|
|
directory.mkdir(parents=True, exist_ok=True)
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
# Create Collection first
|
|
|
|
|
collection = Collection(
|
|
|
|
|
name=data.name,
|
|
|
|
|
kind='channel',
|
|
|
|
|
directory=directory,
|
|
|
|
|
)
|
|
|
|
|
session.add(collection)
|
|
|
|
|
session.flush([collection])
|
|
|
|
|
|
|
|
|
|
# Create Channel linked to Collection
|
|
|
|
|
channel = Channel(collection_id=collection.id)
|
2022-04-08 09:57:04 -06:00
|
|
|
session.add(channel)
|
2023-04-02 11:47:16 -06:00
|
|
|
session.flush([channel])
|
|
|
|
|
# Apply the changes now that we've OK'd them
|
|
|
|
|
channel.update(data.__dict__)
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
session.commit()
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
save_channels_config.activate_switch()
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
return channel.dict() if return_dict else channel
|
2021-12-20 16:08:13 -07:00
|
|
|
|
|
|
|
|
|
2022-04-08 09:57:04 -06:00
|
|
|
def delete_channel(session: Session, *, channel_id: int):
|
2025-12-10 21:16:10 -07:00
|
|
|
channel = Channel.find_by_id(session, channel_id)
|
2021-12-20 16:08:13 -07:00
|
|
|
|
2023-08-17 18:46:42 -06:00
|
|
|
channel_dict = channel.dict()
|
2022-04-08 09:57:04 -06:00
|
|
|
channel.delete_with_videos()
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
save_channels_config.activate_switch()
|
|
|
|
|
|
2023-07-17 17:32:29 -06:00
|
|
|
return channel_dict
|
2021-12-20 16:08:13 -07:00
|
|
|
|
|
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
async def search_channels_by_name(session: Session, name: str, limit: int = 5,
|
2024-04-16 10:04:06 -06:00
|
|
|
order_by_video_count: bool = False) -> List[Channel]:
|
2024-02-19 14:55:53 -07:00
|
|
|
name = name or ''
|
|
|
|
|
name_no_spaces = ''.join(name.split(' '))
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
# SQLite sorts by byte value; emulate the old locale-aware (en_US.UTF-8) ordering: compare
|
|
|
|
|
# names case-insensitively and without spaces, ties broken with lowercase names first.
|
|
|
|
|
name_order = (func.lower(func.replace(Collection.name, ' ', '')).asc(), Collection.name.desc())
|
2024-04-16 10:04:06 -06:00
|
|
|
if order_by_video_count:
|
|
|
|
|
stmt = session.query(Channel, func.count(Video.id).label('video_count')) \
|
db: serializers read only loaded data (#777)
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:28:08 -06:00
|
|
|
.options(*Channel.json_options()) \
|
2025-10-29 22:16:33 -06:00
|
|
|
.join(Collection) \
|
2024-04-16 10:04:06 -06:00
|
|
|
.filter(or_(
|
2025-10-29 22:16:33 -06:00
|
|
|
Collection.name.ilike(f'%{name}%'),
|
|
|
|
|
Collection.name.ilike(f'%{name_no_spaces}%'),
|
2024-04-16 10:04:06 -06:00
|
|
|
)) \
|
2024-04-16 15:48:09 -06:00
|
|
|
.outerjoin(Video, Video.channel_id == Channel.id) \
|
2025-10-29 22:16:33 -06:00
|
|
|
.group_by(Channel.id, Collection.id, Collection.name) \
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
.order_by(desc('video_count'), *name_order) \
|
2024-04-16 10:04:06 -06:00
|
|
|
.limit(limit)
|
|
|
|
|
channels = [i[0] for i in stmt]
|
|
|
|
|
else:
|
|
|
|
|
stmt = session.query(Channel) \
|
db: serializers read only loaded data (#777)
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:28:08 -06:00
|
|
|
.options(*Channel.json_options()) \
|
2025-10-29 22:16:33 -06:00
|
|
|
.join(Collection) \
|
2024-04-16 10:04:06 -06:00
|
|
|
.filter(or_(
|
2025-10-29 22:16:33 -06:00
|
|
|
Collection.name.ilike(f'%{name}%'),
|
|
|
|
|
Collection.name.ilike(f'%{name_no_spaces}%'),
|
2024-04-16 10:04:06 -06:00
|
|
|
)) \
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
.order_by(*name_order) \
|
2024-04-16 10:04:06 -06:00
|
|
|
.limit(limit)
|
|
|
|
|
channels = stmt.all()
|
2023-04-18 22:03:45 -06:00
|
|
|
return channels
|
2024-06-25 22:12:13 -06:00
|
|
|
|
|
|
|
|
|
2024-08-06 19:36:31 -06:00
|
|
|
@wrol_mode_check
|
2024-07-20 15:35:16 -06:00
|
|
|
async def create_channel_download(channel_id: int, url: str, frequency: int, settings: dict):
|
|
|
|
|
"""Create Download for Channel."""
|
2024-06-25 22:12:13 -06:00
|
|
|
with get_db_session(commit=True) as session:
|
2025-12-10 21:16:10 -07:00
|
|
|
channel = Channel.find_by_id(session, channel_id)
|
|
|
|
|
download = channel.get_or_create_download(session, url, frequency, reset_attempts=True)
|
2024-07-20 15:35:16 -06:00
|
|
|
download.settings = settings
|
2024-06-25 22:12:13 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
save_channels_config.activate_switch()
|
|
|
|
|
save_downloads_config.activate_switch()
|
2024-06-25 22:12:13 -06:00
|
|
|
|
2024-07-20 15:35:16 -06:00
|
|
|
return download
|
2024-06-25 22:12:13 -06:00
|
|
|
|
|
|
|
|
|
2024-08-06 19:36:31 -06:00
|
|
|
@wrol_mode_check
|
2024-07-20 15:35:16 -06:00
|
|
|
async def update_channel_download(channel_id: int, download_id: int, url: str, frequency: int, settings: dict):
|
|
|
|
|
"""Fetch Channel's Download, update its properties."""
|
2024-06-25 22:12:13 -06:00
|
|
|
with get_db_session(commit=True) as session:
|
2024-07-20 15:35:16 -06:00
|
|
|
# Ensure that the Channel exists.
|
2025-12-10 21:16:10 -07:00
|
|
|
Channel.find_by_id(session, channel_id)
|
|
|
|
|
download = Download.find_by_id(session, download_id)
|
2024-07-20 15:35:16 -06:00
|
|
|
download.url = url
|
2024-08-06 18:13:21 -06:00
|
|
|
download.frequency = frequency
|
2024-07-20 15:35:16 -06:00
|
|
|
download.settings = settings
|
2024-08-06 18:13:21 -06:00
|
|
|
session.commit()
|
2024-06-25 22:12:13 -06:00
|
|
|
|
2024-07-20 15:35:16 -06:00
|
|
|
download_manager.remove_from_skip_list(url)
|
2024-06-25 22:12:13 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
save_channels_config.activate_switch()
|
|
|
|
|
save_downloads_config.activate_switch()
|
2024-06-25 22:12:13 -06:00
|
|
|
|
2024-07-20 15:35:16 -06:00
|
|
|
return download
|
2024-08-06 18:13:21 -06:00
|
|
|
|
|
|
|
|
|
2025-12-24 17:42:17 -07:00
|
|
|
async def _background_move_channel(channel_id: int, target_directory: pathlib.Path):
|
|
|
|
|
"""Helper to run channel move in background with its own database session.
|
|
|
|
|
|
|
|
|
|
This allows the move to continue even if the original HTTP request is cancelled
|
|
|
|
|
(e.g., user closes browser tab), and avoids DetachedInstanceError by creating
|
|
|
|
|
a fresh session for the background task.
|
|
|
|
|
"""
|
|
|
|
|
try:
|
|
|
|
|
with get_db_session(commit=True) as session:
|
|
|
|
|
channel = Channel.find_by_id(session, channel_id)
|
|
|
|
|
if not channel:
|
|
|
|
|
logger.error(f'_background_move_channel: channel {channel_id} not found')
|
|
|
|
|
Events.send_file_move_failed(f'Channel move failed: channel not found')
|
|
|
|
|
return
|
|
|
|
|
await channel.move_channel(target_directory, session, send_events=True)
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error(f'_background_move_channel: failed for channel_id={channel_id}', exc_info=e)
|
|
|
|
|
Events.send_file_move_failed(f'Channel move failed: {e}')
|
|
|
|
|
|
|
|
|
|
|
2024-08-06 19:36:31 -06:00
|
|
|
@wrol_mode_check
|
2025-12-10 21:16:10 -07:00
|
|
|
async def tag_channel(session: Session, tag_name: str | None, directory: pathlib.Path | None, channel_id: int):
|
2024-08-06 18:13:21 -06:00
|
|
|
"""Add a Tag to a Channel, or remove a Tag from a Channel if no `tag_name` is provided.
|
|
|
|
|
|
|
|
|
|
Move the Channel to the new directory, if provided."""
|
|
|
|
|
|
2026-01-09 14:01:25 -07:00
|
|
|
if directory and flags.file_worker_busy.is_set():
|
|
|
|
|
raise FileWorkerConflict('Refusing to move channel while FileWorker is busy')
|
2024-08-07 08:36:33 -06:00
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
channel = Channel.find_by_id(session, channel_id)
|
2024-08-06 18:13:21 -06:00
|
|
|
|
2024-08-15 21:45:06 -06:00
|
|
|
# May also clear the tag if `tag_name` is None.
|
|
|
|
|
channel.set_tag(tag_name)
|
|
|
|
|
channel.flush()
|
|
|
|
|
session.commit()
|
|
|
|
|
|
2024-08-06 18:13:21 -06:00
|
|
|
# Only move Channel when requested.
|
|
|
|
|
if directory:
|
|
|
|
|
# Move to newly defined directory only if necessary.
|
|
|
|
|
directory.mkdir(parents=True, exist_ok=True)
|
|
|
|
|
if channel.directory != directory:
|
2025-12-24 17:42:17 -07:00
|
|
|
# Always use background task with its own session to avoid
|
|
|
|
|
# DetachedInstanceError when the original session is closed.
|
|
|
|
|
background_task(_background_move_channel(channel.id, directory))
|
2024-09-15 15:33:07 -06:00
|
|
|
else:
|
|
|
|
|
save_channels_config.activate_switch()
|
2024-08-06 18:13:21 -06:00
|
|
|
else:
|
|
|
|
|
logger.info(f'Tagging {channel} with {tag_name}')
|
2025-12-06 09:27:40 -07:00
|
|
|
# Always save config when tag changes, even without directory
|
|
|
|
|
save_channels_config.activate_switch()
|
2024-08-06 18:13:21 -06:00
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
async def search_channels(session: Session, tag_names: List[str]) -> List[Channel]:
|
2024-08-15 19:32:01 -06:00
|
|
|
"""Search Tagged Channels."""
|
db: serializers read only loaded data (#777)
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:28:08 -06:00
|
|
|
channels = session.query(Channel).options(*Channel.json_options()) \
|
|
|
|
|
.join(Collection).join(Tag).filter(Tag.name.in_(tag_names)).all()
|
2024-08-15 19:32:01 -06:00
|
|
|
return channels
|