wrolpi/modules/videos/test
Roland daf2ba39d4 db: write intent is the opt-in, not a second session helper
Three "database is locked" failures on the docker WROLPi, one cause each:

  * `INSERT INTO download` from POST /api/download -- a user creating a
    video download while a refresh ran (2026-08-06 14:06:46);
  * six `INSERT INTO file_group` from `upsert_file`, uploads landing in
    the same three-minute window;
  * `UPDATE download SET status` from `renew_recurring_downloads`.

All three are the deferred read-then-write trap: a transaction that
begins as a reader and upgrades at its first write is refused
*instantly* when another connection wrote in between, because
busy_timeout is skipped for lock upgrades.  `get_immediate_db_session`
existed to avoid it, but an opt-in helper only works if every author
remembers, and read-then-write is the ordinary shape of a write.  So it
is gone: `get_db_session(commit=True)` -- the existing declaration of
write intent -- now begins IMMEDIATE.  Rails 7.1 and Django 5.1 made the
same call, defaulting SQLite transactions to IMMEDIATE rather than
asking application code to opt in.

That default is only safe if write transactions stay short, and one did
not.  `video_cleanup` ran its whole Channel-claiming join --
`UPDATE video ... FROM channel ... LEFT JOIN file_group`, O(channels x
file_groups) -- inside one write transaction: 110 to 130 seconds each of
the eight times it appears in the log, on a library with ~1500 channels.
While it ran, *every* writer on the box failed, including ones that
began IMMEDIATE, because the hold far exceeds the 30s busy_timeout.  The
14:04-14:06 failures above sit inside its 130-second window.  The join
is now a read -- WAL readers never block writers -- and only the
resulting UPDATEs take the lock, 500 rows per transaction.
`claim_videos_for_channels` had the same join and the same fix; channel
import calls it with every channel in channels.yaml.

Also here:

  * `renew_recurring_downloads` was a read session with an inner commit;
    it is a write session now.  Its computed `next_download` values now
    persist even when nothing was renewed -- that is the point of
    computing them.
  * POST /api/download no longer writes through `request.ctx.session`.
    That session is committed by the response middleware, so a writer
    would hold the lock across the whole request; a write session
    commits before the handler returns, ahead of the background dispatch
    it fires.
  * `_immediate_txn` is a ContextVar, not a threading.local.  The
    writers that set it include coroutines, and one asyncio thread runs
    many of them interleaved.
  * `write_lock_held_briefly` moved from the move tests into conftest,
    since two suites now need it.

Both regression tests were verified by mutation: reverting either site
alone fails its own test, and the API one fails with the production
error verbatim.

Sources: fractaledmind.com/2023/12/11/sqlite-on-rails-improving-concurrency,
tenthousandmeters.com/blog/sqlite-concurrent-writes-and-database-is-locked-errors,
berthub.eu/articles/posts/a-brief-post-on-sqlite3-database-locked-despite-timeout

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 20:15:50 -06:00
..
__init__.py A new start 2021-11-01 16:43:09 -06:00
test_api.py SQLite: make the browse pages index-only. 2026-07-14 07:46:29 -06:00
test_channel_directory_scenarios.py db: write intent is the opt-in, not a second session helper 2026-08-07 20:15:50 -06:00
test_channel_refresh_external_files.py Displaying download progress (speed, eta) for file and video downloads. 2026-03-12 19:31:35 -06:00
test_channels_config.py FileGroups in Collections can be reorganized in the Collection's directory. 2026-01-31 18:08:41 -07:00
test_common.py Fixing bug where configs could be overwritten with empty configs on first refresh. Fixing bug where configs were displayed as JSON in admin settings. 2026-03-22 12:56:37 -06:00
test_cookies.py Using shared context to sync cookies between processes. 2026-02-22 14:32:27 -07:00
test_downloader.py fix: a video download requests one format, not every matching format 2026-08-05 17:25:19 -06:00
test_lib.py Videos: a Channel's directory beats source_id when assigning a Video. 2026-07-07 18:33:15 -06:00
test_models.py Handling when skip_urls is a None. Logging a warning (and ignoring) when duplicate json files are in a Video FileGroup. 2026-03-13 15:27:43 -06:00
test_video.py review: flush before the rename, not just after one delete 2026-08-07 15:08:40 -06:00
test_video_modeler_loop.py Fixing bug where not all archives were modeled. 2025-12-18 17:19:57 -07:00
test_video_url_resolver.py Trying to get existing channel using more variations of Youtube URLs. 2026-03-20 17:26:51 -06:00