2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Tests for ChannelsConfig import/upgrade scenarios.
|
|
|
|
|
|
|
|
|
|
Tests cover:
|
|
|
|
|
1. Onboarding (new install) - no config file exists
|
|
|
|
|
2. Upgrade - config file exists with data
|
|
|
|
|
3. Edge cases - empty config, missing config with DB data
|
|
|
|
|
"""
|
Collections: changes wait for their config's import; domains with archives survive the import (#784)
* collections: refuse playlist changes until playlists.yaml is imported
While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change). Playlist changes now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse domain changes until domains.yaml is imported
A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import. These now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* archive: importing domains.yaml keeps domains which have archives
Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it. A domain with Archives is now kept (and saved to the config with the next change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* videos: refuse Channel changes until channels.yaml is imported
A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:33:51 -06:00
|
|
|
import json
|
|
|
|
|
from http import HTTPStatus
|
|
|
|
|
|
2025-12-06 14:47:52 -07:00
|
|
|
import pytest
|
|
|
|
|
import yaml
|
|
|
|
|
|
|
|
|
|
from modules.videos.lib import get_channels_config
|
|
|
|
|
from modules.videos.models import Channel
|
Database configs carry a version the database keeps too (#866)
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
|
|
|
from wrolpi.conftest import config_was_imported, config_versions_lost
|
2025-12-06 14:47:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
class TestChannelsConfigOnboarding:
|
|
|
|
|
"""Tests for new install scenarios where no config file exists."""
|
|
|
|
|
|
|
|
|
|
async def test_new_install_no_db_data(self, test_session, test_directory, async_client, test_channels_config):
|
|
|
|
|
"""
|
|
|
|
|
New install with empty DB and no config file.
|
|
|
|
|
Should succeed with empty config.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Ensure no config file exists
|
|
|
|
|
if config_path.is_file():
|
|
|
|
|
config_path.unlink()
|
|
|
|
|
|
|
|
|
|
# Ensure DB has no channels
|
|
|
|
|
assert test_session.query(Channel).count() == 0
|
|
|
|
|
|
|
|
|
|
# Import should succeed
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
async def test_new_install_with_db_data(self, test_session, test_directory, async_client, test_channels_config,
|
|
|
|
|
channel_factory):
|
|
|
|
|
"""
|
|
|
|
|
Config file missing but DB has data.
|
|
|
|
|
Import should succeed (nothing to import), DB data preserved.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create a channel in DB
|
|
|
|
|
channel = channel_factory(name='Existing Channel')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Remove config file to simulate missing config
|
|
|
|
|
if config_path.is_file():
|
|
|
|
|
config_path.unlink()
|
|
|
|
|
|
|
|
|
|
# Import should succeed (nothing to import)
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# DB data should be preserved
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 1
|
|
|
|
|
assert channels[0].name == 'Existing Channel'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
class TestChannelsConfigUpgrade:
|
|
|
|
|
"""Tests for upgrade scenarios where config file exists."""
|
|
|
|
|
|
|
|
|
|
async def test_upgrade_config_exists_with_data(self, test_session, test_directory, async_client,
|
2025-12-06 14:49:46 -07:00
|
|
|
test_channels_config, channel_factory):
|
2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Upgrade with existing config containing channel data.
|
|
|
|
|
Should import and sync DB with config.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create a channel in DB
|
|
|
|
|
channel = channel_factory(name='Config Channel')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Dump to create config file
|
|
|
|
|
config.dump_config()
|
|
|
|
|
assert config_path.is_file()
|
|
|
|
|
|
|
|
|
|
# Now import should succeed and preserve the channel
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# Channel should still exist
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 1
|
|
|
|
|
assert channels[0].name == 'Config Channel'
|
|
|
|
|
|
|
|
|
|
async def test_upgrade_config_exists_empty_list(self, test_session, test_directory, async_client,
|
2025-12-06 14:49:46 -07:00
|
|
|
test_channels_config, channel_factory):
|
2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Config exists but has empty channels list.
|
|
|
|
|
Should NOT delete existing DB channels (never delete on empty).
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create a channel in DB
|
|
|
|
|
channel = channel_factory(name='Should Not Delete')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Write empty config
|
|
|
|
|
with open(config_path, 'w') as f:
|
|
|
|
|
yaml.dump({'version': 0, 'channels': []}, f)
|
|
|
|
|
|
|
|
|
|
# Reinitialize config to read the empty file
|
|
|
|
|
config.initialize()
|
|
|
|
|
|
|
|
|
|
# Import should succeed but NOT delete the channel
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# Channel should still exist (never delete on empty config)
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 1
|
|
|
|
|
assert channels[0].name == 'Should Not Delete'
|
|
|
|
|
|
|
|
|
|
async def test_upgrade_config_missing_db_has_data(self, test_session, test_directory, async_client,
|
2025-12-06 14:49:46 -07:00
|
|
|
test_channels_config, channel_factory):
|
2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Config file missing but DB has channels.
|
|
|
|
|
Import should succeed (nothing to import), DB data preserved.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create channels in DB
|
|
|
|
|
channel1 = channel_factory(name='Channel One')
|
|
|
|
|
channel2 = channel_factory(name='Channel Two')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Remove config file
|
|
|
|
|
if config_path.is_file():
|
|
|
|
|
config_path.unlink()
|
|
|
|
|
|
|
|
|
|
# Import should succeed (nothing to import)
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# DB data should be preserved
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 2
|
|
|
|
|
channel_names = {c.name for c in channels}
|
|
|
|
|
assert 'Channel One' in channel_names
|
|
|
|
|
assert 'Channel Two' in channel_names
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
class TestChannelsConfigEdgeCases:
|
|
|
|
|
"""Tests for edge cases in config import."""
|
|
|
|
|
|
|
|
|
|
async def test_config_deletes_removed_items(self, test_session, test_directory, async_client, test_channels_config,
|
2025-12-06 14:49:46 -07:00
|
|
|
channel_factory):
|
2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Config removes a channel that exists in DB.
|
|
|
|
|
Should delete the channel from DB.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create two channels
|
|
|
|
|
channel1 = channel_factory(name='Keep This')
|
|
|
|
|
channel2 = channel_factory(name='Delete This')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Dump to config
|
|
|
|
|
config.dump_config()
|
|
|
|
|
|
|
|
|
|
# Modify config to remove one channel
|
|
|
|
|
with open(config_path) as f:
|
|
|
|
|
config_data = yaml.safe_load(f)
|
|
|
|
|
|
|
|
|
|
config_data['channels'] = [c for c in config_data['channels'] if c['name'] == 'Keep This']
|
|
|
|
|
|
|
|
|
|
with open(config_path, 'w') as f:
|
|
|
|
|
yaml.dump(config_data, f)
|
|
|
|
|
|
|
|
|
|
# Reinitialize and import
|
|
|
|
|
config.initialize()
|
Database configs carry a version the database keeps too (#866)
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
|
|
|
config_was_imported(test_session, 'channels.yaml')
|
2025-12-06 14:47:52 -07:00
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# Only one channel should remain
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 1
|
|
|
|
|
assert channels[0].name == 'Keep This'
|
|
|
|
|
|
|
|
|
|
async def test_config_adds_new_items(self, test_session, test_directory, async_client, test_channels_config):
|
|
|
|
|
"""
|
|
|
|
|
Config has a new channel that doesn't exist in DB.
|
|
|
|
|
Should create the channel in DB.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Ensure no channels in DB
|
|
|
|
|
assert test_session.query(Channel).count() == 0
|
|
|
|
|
|
|
|
|
|
# Write config with a new channel
|
|
|
|
|
new_channel_dir = test_directory / 'videos' / 'New Channel'
|
|
|
|
|
new_channel_dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
|
|
|
|
|
|
config_data = {
|
|
|
|
|
'version': 0,
|
|
|
|
|
'channels': [{
|
|
|
|
|
'name': 'New Channel',
|
|
|
|
|
'directory': str(new_channel_dir),
|
|
|
|
|
'url': 'https://example.com/new',
|
|
|
|
|
'download_frequency': 604800,
|
|
|
|
|
'downloads': [],
|
|
|
|
|
}]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
with open(config_path, 'w') as f:
|
|
|
|
|
yaml.dump(config_data, f)
|
|
|
|
|
|
|
|
|
|
# Reinitialize and import
|
|
|
|
|
config.initialize()
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# Channel should be created
|
|
|
|
|
channels = test_session.query(Channel).all()
|
|
|
|
|
assert len(channels) == 1
|
|
|
|
|
assert channels[0].name == 'New Channel'
|
|
|
|
|
assert channels[0].url == 'https://example.com/new'
|
|
|
|
|
|
|
|
|
|
async def test_config_updates_existing_items(self, test_session, test_directory, async_client, test_channels_config,
|
2025-12-06 14:49:46 -07:00
|
|
|
channel_factory):
|
2025-12-06 14:47:52 -07:00
|
|
|
"""
|
|
|
|
|
Config updates an existing channel's URL.
|
|
|
|
|
Should update the channel in DB.
|
|
|
|
|
"""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create a channel
|
|
|
|
|
channel = channel_factory(name='Update Me', url='https://example.com/old')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Dump to config
|
|
|
|
|
config.dump_config()
|
|
|
|
|
|
|
|
|
|
# Modify config to change URL
|
|
|
|
|
with open(config_path) as f:
|
|
|
|
|
config_data = yaml.safe_load(f)
|
|
|
|
|
|
|
|
|
|
for c in config_data['channels']:
|
|
|
|
|
if c['name'] == 'Update Me':
|
|
|
|
|
c['url'] = 'https://example.com/new'
|
|
|
|
|
|
|
|
|
|
with open(config_path, 'w') as f:
|
|
|
|
|
yaml.dump(config_data, f)
|
|
|
|
|
|
|
|
|
|
# Reinitialize and import
|
|
|
|
|
config.initialize()
|
|
|
|
|
config.import_config()
|
|
|
|
|
assert config.successful_import is True
|
|
|
|
|
|
|
|
|
|
# Channel should be updated
|
|
|
|
|
channel = test_session.query(Channel).filter_by(url='https://example.com/new').first()
|
|
|
|
|
assert channel is not None
|
|
|
|
|
assert channel.name == 'Update Me'
|
2026-01-23 15:11:12 -07:00
|
|
|
|
|
|
|
|
async def test_channel_config_view_uses_relative_directory(self, test_session, test_directory, async_client,
|
|
|
|
|
test_channels_config, channel_factory):
|
|
|
|
|
"""Channel.config_view() should export directory as relative path for portability."""
|
|
|
|
|
channel = channel_factory()
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
config = channel.config_view()
|
|
|
|
|
|
|
|
|
|
# Directory should be relative, not absolute
|
|
|
|
|
assert not config['directory'].startswith('/'), \
|
|
|
|
|
f"Expected relative path but got absolute: {config['directory']}"
|
2026-01-23 19:51:02 -07:00
|
|
|
|
|
|
|
|
async def test_channel_config_view_includes_file_format(self, test_session, test_directory, async_client,
|
|
|
|
|
test_channels_config, channel_factory):
|
|
|
|
|
"""Channel.config_view() should include file_format from underlying Collection."""
|
|
|
|
|
channel = channel_factory()
|
|
|
|
|
channel.collection.file_format = '%(uploader)s_%(id)s_%(title)s.%(ext)s'
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
config = channel.config_view()
|
|
|
|
|
assert config['file_format'] == '%(uploader)s_%(id)s_%(title)s.%(ext)s'
|
|
|
|
|
|
|
|
|
|
async def test_channel_file_format_round_trip(self, test_session, test_directory, async_client,
|
|
|
|
|
test_channels_config, channel_factory):
|
|
|
|
|
"""File format should survive config dump and import cycle."""
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config_path = config.get_file()
|
|
|
|
|
|
|
|
|
|
# Create channel with file_format
|
|
|
|
|
channel = channel_factory(name='Format Test')
|
|
|
|
|
channel.collection.file_format = '%(upload_date)s_%(title)s.%(ext)s'
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Dump to config
|
|
|
|
|
config.dump_config()
|
|
|
|
|
|
|
|
|
|
# Verify file_format in YAML
|
|
|
|
|
with open(config_path) as f:
|
|
|
|
|
config_data = yaml.safe_load(f)
|
|
|
|
|
channel_config = next(c for c in config_data['channels'] if c['name'] == 'Format Test')
|
|
|
|
|
assert channel_config['file_format'] == '%(upload_date)s_%(title)s.%(ext)s'
|
|
|
|
|
|
|
|
|
|
# Clear the file_format to simulate factory reset
|
|
|
|
|
channel.collection.file_format = None
|
|
|
|
|
test_session.commit()
|
Database configs carry a version the database keeps too (#866)
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
|
|
|
config_versions_lost(test_session)
|
2026-01-23 19:51:02 -07:00
|
|
|
|
|
|
|
|
# Re-import
|
|
|
|
|
config.initialize()
|
|
|
|
|
config.import_config()
|
|
|
|
|
|
|
|
|
|
# Verify file_format restored
|
|
|
|
|
test_session.refresh(channel)
|
|
|
|
|
assert channel.collection.file_format == '%(upload_date)s_%(title)s.%(ext)s'
|
Collections: changes wait for their config's import; domains with archives survive the import (#784)
* collections: refuse playlist changes until playlists.yaml is imported
While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change). Playlist changes now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse domain changes until domains.yaml is imported
A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import. These now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* archive: importing domains.yaml keeps domains which have archives
Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it. A domain with Archives is now kept (and saved to the config with the next change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* videos: refuse Channel changes until channels.yaml is imported
A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:33:51 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_channel_changes_wait_for_config_import(test_session, test_directory, async_client, channel_factory,
|
|
|
|
|
test_channels_config):
|
|
|
|
|
"""Until channels.yaml is imported (e.g. while WROLPi starts), Channels cannot change; the import would undo the
|
|
|
|
|
change (a new Channel would be deleted, with its Downloads)."""
|
|
|
|
|
channel = channel_factory(name='Kept')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
config = get_channels_config()
|
|
|
|
|
config.dump_config()
|
|
|
|
|
assert config.get_file().is_file()
|
|
|
|
|
config.successful_import = False
|
|
|
|
|
|
|
|
|
|
cid, collection_id = channel.id, channel.collection_id
|
|
|
|
|
requests = [
|
|
|
|
|
('post', '/api/videos/channels', dict(name='New', directory='videos/new', url='https://example.com/new')),
|
|
|
|
|
('put', f'/api/videos/channels/{cid}', dict(name='Changed', directory=str(channel.directory))),
|
|
|
|
|
('post', f'/api/videos/channels/{cid}/tag', dict(tag_name='Tagged')),
|
|
|
|
|
('delete', f'/api/videos/channels/{cid}', None),
|
|
|
|
|
('put', f'/api/collections/{collection_id}', dict(description='changed')),
|
|
|
|
|
('post', f'/api/collections/{collection_id}/tag', dict(tag_name='Tagged')),
|
|
|
|
|
('delete', f'/api/collections/{collection_id}', None),
|
|
|
|
|
]
|
|
|
|
|
for method, url, body in requests:
|
|
|
|
|
kwargs = dict(content=json.dumps(body)) if body else {}
|
|
|
|
|
request, response = await getattr(async_client, method)(url, **kwargs)
|
|
|
|
|
assert response.status_code == HTTPStatus.SERVICE_UNAVAILABLE, (method, url, response.json)
|
|
|
|
|
assert response.json['code'] == 'CONFIG_NOT_IMPORTED'
|
|
|
|
|
|
|
|
|
|
test_session.expire_all()
|
|
|
|
|
channel = test_session.query(Channel).one()
|
|
|
|
|
assert channel.name == 'Kept'
|
|
|
|
|
assert channel.tag_name is None
|
|
|
|
|
assert channel.collection.description is None
|
|
|
|
|
|
|
|
|
|
# Once imported, Channels can change again.
|
|
|
|
|
config.import_config()
|
|
|
|
|
request, response = await async_client.put(f'/api/collections/{collection_id}',
|
|
|
|
|
content=json.dumps(dict(description='x')))
|
|
|
|
|
assert response.status_code == HTTPStatus.OK, response.json
|