PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between. They now write inside
request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query. An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async. Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.
Collection, CollectionItem and Channel dict() now run inside the serializer guard. Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
the session.
Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Count Channels with any of the Tags in the Other search estimate
The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Search Channels by name in the Other search tab
The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing. Channels now match the search text by name,
and the Tags when there are both. The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Domains to the Other search tab
Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Playlists to the Other search tab
Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint. The search and the estimate share search_collections_query. The
search now loads each Collection's Downloads up front instead of once per
Collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse playlist changes until playlists.yaml is imported
While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change). Playlist changes now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse domain changes until domains.yaml is imported
A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import. These now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* archive: importing domains.yaml keeps domains which have archives
Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it. A domain with Archives is now kept (and saved to the config with the next change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* videos: refuse Channel changes until channels.yaml is imported
A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: one request-scoped database handle per API request
Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that
Session's connection, so a request needs at most one SQLite connection. Creating it does no I/O; requests
that never touch the database never connect. `request.ctx.session` remains the handle's Session.
A `RequestSession` binds itself to one lazily-opened Connection. An engine-bound Session releases its
connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected.
Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the
write lock is not held for the rest of the handler or by background work it starts. This replaces the
handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder,
channel create). The zim search handlers use the request's Session in their worker thread.
Tests count the connections a request opens with production's per-call sessions
(`count_request_connections`). Endpoints whose library functions still open their own connection are
strict xfails, to be removed as those functions take the request's session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: name the real extra connections in the request connection xfails
Channel detail: get_channel already uses the request session; the extra connection is
Channel.get_statistics opening a raw cursor (get_db_curs). Search: count_file_groups opens a
cursor for the total, then handle_file_group_search_results opens a cursor and a session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
POST /api/videos/channels failed with "database is locked" on
`INSERT INTO collection` on the docker WROLPi while background workers
(config saves, refreshes) were writing. The handler wrote through the
deferred request session: `create_channel` reads (the URL/name/directory
conflict check) and then INSERTs, and SQLite refuses that mid-transaction
lock upgrade instantly when another connection wrote in between --
busy_timeout is skipped for upgrades.
Same shape and same fix as POST /api/download (daf2ba39): the handler
now uses `get_db_session(commit=True)`, which begins IMMEDIATE so the
concurrent writer waits its turn, and which commits before the refresh
is fired so the refresh does not wait on a lock the request still holds.
The regression test commits a write from a second connection between
the handler's conflict check and its INSERT. The existing
`write_lock_held_briefly` helper cannot reproduce this: the Sanic test
client takes over a second to start the app, so its lock is released
before the handler runs. Before the fix the test fails with the
production error verbatim.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.
Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.
channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a Channel refresh, log how many video/audio FileGroups are in the
Channel's directory, and how many are assigned to this Channel, other
Channels, or no Channel. Helps debug Videos missing from their Channel
after files are moved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag_file join in Channel.get_statistics() produced one row per tag,
so a Video with N Tags was counted N times in video_count, size, and
length. Count tagged Videos in a subquery instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Storing video settings in session storage to help user start downloads consistently.
Applying channel tag name to new channels when download videos or channels.
Initializing map tiles when importing and resetting map.
