wrolpi/modules/videos/test/test_channels_config.py

372 lines
14 KiB
Python
Raw Permalink Normal View History

"""
Tests for ChannelsConfig import/upgrade scenarios.
Tests cover:
1. Onboarding (new install) - no config file exists
2. Upgrade - config file exists with data
3. Edge cases - empty config, missing config with DB data
"""
Collections: changes wait for their config's import; domains with archives survive the import (#784) * collections: refuse playlist changes until playlists.yaml is imported While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from the config, undoing any change). Playlist changes now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * collections: refuse domain changes until domains.yaml is imported A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi starts) was undone by the import. These now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * archive: importing domains.yaml keeps domains which have archives Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives with it. A domain with Archives is now kept (and saved to the config with the next change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * videos: refuse Channel changes until channels.yaml is imported A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved, and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:33:51 -06:00
import json
from http import HTTPStatus
import pytest
import yaml
from modules.videos.lib import get_channels_config
from modules.videos.models import Channel
Database configs carry a version the database keeps too (#866) * Benchmark every database config import (#865) An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk, writes each config with its own dump_config, then times every import in startup order: a steady pass (the database already matches) and a rebuild (config-owned rows deleted, everything restored from the configs). It prints seconds, SQL statements and writes per import, and asserts the rebuild restored every row so no timed import silently skipped its work. WROLPI_CONFIG_BENCH_SCALE multiplies the row counts. The baseline for the config import rework. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Database configs carry a version the database keeps too Step 1 of the config import rework. A database config (tags, downloads, channels, domains, playlists, repos, recently viewed) and the database now each carry an integer version, so an import can tell which side is newer without a clock (a Raspberry Pi's resets at boot): - config_version (new table) holds the database's version of each config. - An after_flush hook adds one to it for every config whose rows a flush changed, in the same transaction. Each config registers what it owns (register_ownership): models, optionally only some columns, optionally only some rows (a Collection's kind; a Download's Collection's kind). Raw SQL writes to owned tables count explicitly. - dump_config writes the database's version, read with the rows it saves; save() no longer adds one for these configs. - An import compares versions first: file == database or newer imports with deletes; file older (the database changed and was not saved before a restart) saves the database instead of undoing that change; no database row (a config this code never imported) imports without deletes. The import records the file's version last. - The Import button and backup imports apply the file whatever its version (import_forced). Importers only gate their existing deletes on the decision here; they are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions: keep unsaved rows after a no-delete import, count every owned write Review fixes: - An import without deletes (no database version yet) no longer records the file's version, which let the next startup delete the rows it had kept. The version stays unknown, and the database is saved into the file instead; a successful save records the version. A save after a partial import is refused, so the version stays unknown until an import applies the whole file. - An import removes the database's version while it runs, so its own writes do not count: one that fails partway no longer looks newer than the file and gets saved over it on the next startup. - Playlists imported without deletes keep their database items in order; the file's other items are added after their neighbor from the file. - A Download leaving a Channel or Repo counts for that Collection's config (collection_id loads its previous value). - Bulk writes count: untagging, retrying Downloads, deleting a duplicate's Download, removing many items from a playlist. Deleting Downloads by id counts channels.yaml/repos.yaml only for their Downloads. - Only a playlist's items count for playlists.yaml (author and subject Collections have items too). Collection kinds are looked up once per flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions count only the columns each config saves A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count, info_json), a Collection's counts, and a Download attempt (attempts, error) counted as config changes, so the next import of that config saved instead of importing: a hand edit to repos.yaml was overwritten after any fetch. Each model now registers only the columns its config saves. domains.yaml saves its domains' Downloads, which now count for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
from wrolpi.conftest import config_was_imported, config_versions_lost
@pytest.mark.asyncio
class TestChannelsConfigOnboarding:
"""Tests for new install scenarios where no config file exists."""
async def test_new_install_no_db_data(self, test_session, test_directory, async_client, test_channels_config):
"""
New install with empty DB and no config file.
Should succeed with empty config.
"""
config = get_channels_config()
config_path = config.get_file()
# Ensure no config file exists
if config_path.is_file():
config_path.unlink()
# Ensure DB has no channels
assert test_session.query(Channel).count() == 0
# Import should succeed
config.import_config()
assert config.successful_import is True
async def test_new_install_with_db_data(self, test_session, test_directory, async_client, test_channels_config,
channel_factory):
"""
Config file missing but DB has data.
Import should succeed (nothing to import), DB data preserved.
"""
config = get_channels_config()
config_path = config.get_file()
# Create a channel in DB
channel = channel_factory(name='Existing Channel')
test_session.commit()
# Remove config file to simulate missing config
if config_path.is_file():
config_path.unlink()
# Import should succeed (nothing to import)
config.import_config()
assert config.successful_import is True
# DB data should be preserved
channels = test_session.query(Channel).all()
assert len(channels) == 1
assert channels[0].name == 'Existing Channel'
@pytest.mark.asyncio
class TestChannelsConfigUpgrade:
"""Tests for upgrade scenarios where config file exists."""
async def test_upgrade_config_exists_with_data(self, test_session, test_directory, async_client,
2025-12-06 14:49:46 -07:00
test_channels_config, channel_factory):
"""
Upgrade with existing config containing channel data.
Should import and sync DB with config.
"""
config = get_channels_config()
config_path = config.get_file()
# Create a channel in DB
channel = channel_factory(name='Config Channel')
test_session.commit()
# Dump to create config file
config.dump_config()
assert config_path.is_file()
# Now import should succeed and preserve the channel
config.import_config()
assert config.successful_import is True
# Channel should still exist
channels = test_session.query(Channel).all()
assert len(channels) == 1
assert channels[0].name == 'Config Channel'
async def test_upgrade_config_exists_empty_list(self, test_session, test_directory, async_client,
2025-12-06 14:49:46 -07:00
test_channels_config, channel_factory):
"""
Config exists but has empty channels list.
Should NOT delete existing DB channels (never delete on empty).
"""
config = get_channels_config()
config_path = config.get_file()
# Create a channel in DB
channel = channel_factory(name='Should Not Delete')
test_session.commit()
# Write empty config
with open(config_path, 'w') as f:
yaml.dump({'version': 0, 'channels': []}, f)
# Reinitialize config to read the empty file
config.initialize()
# Import should succeed but NOT delete the channel
config.import_config()
assert config.successful_import is True
# Channel should still exist (never delete on empty config)
channels = test_session.query(Channel).all()
assert len(channels) == 1
assert channels[0].name == 'Should Not Delete'
async def test_upgrade_config_missing_db_has_data(self, test_session, test_directory, async_client,
2025-12-06 14:49:46 -07:00
test_channels_config, channel_factory):
"""
Config file missing but DB has channels.
Import should succeed (nothing to import), DB data preserved.
"""
config = get_channels_config()
config_path = config.get_file()
# Create channels in DB
channel1 = channel_factory(name='Channel One')
channel2 = channel_factory(name='Channel Two')
test_session.commit()
# Remove config file
if config_path.is_file():
config_path.unlink()
# Import should succeed (nothing to import)
config.import_config()
assert config.successful_import is True
# DB data should be preserved
channels = test_session.query(Channel).all()
assert len(channels) == 2
channel_names = {c.name for c in channels}
assert 'Channel One' in channel_names
assert 'Channel Two' in channel_names
@pytest.mark.asyncio
class TestChannelsConfigEdgeCases:
"""Tests for edge cases in config import."""
async def test_config_deletes_removed_items(self, test_session, test_directory, async_client, test_channels_config,
2025-12-06 14:49:46 -07:00
channel_factory):
"""
Config removes a channel that exists in DB.
Should delete the channel from DB.
"""
config = get_channels_config()
config_path = config.get_file()
# Create two channels
channel1 = channel_factory(name='Keep This')
channel2 = channel_factory(name='Delete This')
test_session.commit()
# Dump to config
config.dump_config()
# Modify config to remove one channel
with open(config_path) as f:
config_data = yaml.safe_load(f)
config_data['channels'] = [c for c in config_data['channels'] if c['name'] == 'Keep This']
with open(config_path, 'w') as f:
yaml.dump(config_data, f)
# Reinitialize and import
config.initialize()
Database configs carry a version the database keeps too (#866) * Benchmark every database config import (#865) An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk, writes each config with its own dump_config, then times every import in startup order: a steady pass (the database already matches) and a rebuild (config-owned rows deleted, everything restored from the configs). It prints seconds, SQL statements and writes per import, and asserts the rebuild restored every row so no timed import silently skipped its work. WROLPI_CONFIG_BENCH_SCALE multiplies the row counts. The baseline for the config import rework. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Database configs carry a version the database keeps too Step 1 of the config import rework. A database config (tags, downloads, channels, domains, playlists, repos, recently viewed) and the database now each carry an integer version, so an import can tell which side is newer without a clock (a Raspberry Pi's resets at boot): - config_version (new table) holds the database's version of each config. - An after_flush hook adds one to it for every config whose rows a flush changed, in the same transaction. Each config registers what it owns (register_ownership): models, optionally only some columns, optionally only some rows (a Collection's kind; a Download's Collection's kind). Raw SQL writes to owned tables count explicitly. - dump_config writes the database's version, read with the rows it saves; save() no longer adds one for these configs. - An import compares versions first: file == database or newer imports with deletes; file older (the database changed and was not saved before a restart) saves the database instead of undoing that change; no database row (a config this code never imported) imports without deletes. The import records the file's version last. - The Import button and backup imports apply the file whatever its version (import_forced). Importers only gate their existing deletes on the decision here; they are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions: keep unsaved rows after a no-delete import, count every owned write Review fixes: - An import without deletes (no database version yet) no longer records the file's version, which let the next startup delete the rows it had kept. The version stays unknown, and the database is saved into the file instead; a successful save records the version. A save after a partial import is refused, so the version stays unknown until an import applies the whole file. - An import removes the database's version while it runs, so its own writes do not count: one that fails partway no longer looks newer than the file and gets saved over it on the next startup. - Playlists imported without deletes keep their database items in order; the file's other items are added after their neighbor from the file. - A Download leaving a Channel or Repo counts for that Collection's config (collection_id loads its previous value). - Bulk writes count: untagging, retrying Downloads, deleting a duplicate's Download, removing many items from a playlist. Deleting Downloads by id counts channels.yaml/repos.yaml only for their Downloads. - Only a playlist's items count for playlists.yaml (author and subject Collections have items too). Collection kinds are looked up once per flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions count only the columns each config saves A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count, info_json), a Collection's counts, and a Download attempt (attempts, error) counted as config changes, so the next import of that config saved instead of importing: a hand edit to repos.yaml was overwritten after any fetch. Each model now registers only the columns its config saves. domains.yaml saves its domains' Downloads, which now count for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
config_was_imported(test_session, 'channels.yaml')
config.import_config()
assert config.successful_import is True
# Only one channel should remain
channels = test_session.query(Channel).all()
assert len(channels) == 1
assert channels[0].name == 'Keep This'
async def test_config_adds_new_items(self, test_session, test_directory, async_client, test_channels_config):
"""
Config has a new channel that doesn't exist in DB.
Should create the channel in DB.
"""
config = get_channels_config()
config_path = config.get_file()
# Ensure no channels in DB
assert test_session.query(Channel).count() == 0
# Write config with a new channel
new_channel_dir = test_directory / 'videos' / 'New Channel'
new_channel_dir.mkdir(parents=True, exist_ok=True)
config_data = {
'version': 0,
'channels': [{
'name': 'New Channel',
'directory': str(new_channel_dir),
'url': 'https://example.com/new',
'download_frequency': 604800,
'downloads': [],
}]
}
with open(config_path, 'w') as f:
yaml.dump(config_data, f)
# Reinitialize and import
config.initialize()
config.import_config()
assert config.successful_import is True
# Channel should be created
channels = test_session.query(Channel).all()
assert len(channels) == 1
assert channels[0].name == 'New Channel'
assert channels[0].url == 'https://example.com/new'
async def test_config_updates_existing_items(self, test_session, test_directory, async_client, test_channels_config,
2025-12-06 14:49:46 -07:00
channel_factory):
"""
Config updates an existing channel's URL.
Should update the channel in DB.
"""
config = get_channels_config()
config_path = config.get_file()
# Create a channel
channel = channel_factory(name='Update Me', url='https://example.com/old')
test_session.commit()
# Dump to config
config.dump_config()
# Modify config to change URL
with open(config_path) as f:
config_data = yaml.safe_load(f)
for c in config_data['channels']:
if c['name'] == 'Update Me':
c['url'] = 'https://example.com/new'
with open(config_path, 'w') as f:
yaml.dump(config_data, f)
# Reinitialize and import
config.initialize()
config.import_config()
assert config.successful_import is True
# Channel should be updated
channel = test_session.query(Channel).filter_by(url='https://example.com/new').first()
assert channel is not None
assert channel.name == 'Update Me'
async def test_channel_config_view_uses_relative_directory(self, test_session, test_directory, async_client,
test_channels_config, channel_factory):
"""Channel.config_view() should export directory as relative path for portability."""
channel = channel_factory()
test_session.commit()
config = channel.config_view()
# Directory should be relative, not absolute
assert not config['directory'].startswith('/'), \
f"Expected relative path but got absolute: {config['directory']}"
async def test_channel_config_view_includes_file_format(self, test_session, test_directory, async_client,
test_channels_config, channel_factory):
"""Channel.config_view() should include file_format from underlying Collection."""
channel = channel_factory()
channel.collection.file_format = '%(uploader)s_%(id)s_%(title)s.%(ext)s'
test_session.commit()
config = channel.config_view()
assert config['file_format'] == '%(uploader)s_%(id)s_%(title)s.%(ext)s'
async def test_channel_file_format_round_trip(self, test_session, test_directory, async_client,
test_channels_config, channel_factory):
"""File format should survive config dump and import cycle."""
config = get_channels_config()
config_path = config.get_file()
# Create channel with file_format
channel = channel_factory(name='Format Test')
channel.collection.file_format = '%(upload_date)s_%(title)s.%(ext)s'
test_session.commit()
# Dump to config
config.dump_config()
# Verify file_format in YAML
with open(config_path) as f:
config_data = yaml.safe_load(f)
channel_config = next(c for c in config_data['channels'] if c['name'] == 'Format Test')
assert channel_config['file_format'] == '%(upload_date)s_%(title)s.%(ext)s'
# Clear the file_format to simulate factory reset
channel.collection.file_format = None
test_session.commit()
Database configs carry a version the database keeps too (#866) * Benchmark every database config import (#865) An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk, writes each config with its own dump_config, then times every import in startup order: a steady pass (the database already matches) and a rebuild (config-owned rows deleted, everything restored from the configs). It prints seconds, SQL statements and writes per import, and asserts the rebuild restored every row so no timed import silently skipped its work. WROLPI_CONFIG_BENCH_SCALE multiplies the row counts. The baseline for the config import rework. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Database configs carry a version the database keeps too Step 1 of the config import rework. A database config (tags, downloads, channels, domains, playlists, repos, recently viewed) and the database now each carry an integer version, so an import can tell which side is newer without a clock (a Raspberry Pi's resets at boot): - config_version (new table) holds the database's version of each config. - An after_flush hook adds one to it for every config whose rows a flush changed, in the same transaction. Each config registers what it owns (register_ownership): models, optionally only some columns, optionally only some rows (a Collection's kind; a Download's Collection's kind). Raw SQL writes to owned tables count explicitly. - dump_config writes the database's version, read with the rows it saves; save() no longer adds one for these configs. - An import compares versions first: file == database or newer imports with deletes; file older (the database changed and was not saved before a restart) saves the database instead of undoing that change; no database row (a config this code never imported) imports without deletes. The import records the file's version last. - The Import button and backup imports apply the file whatever its version (import_forced). Importers only gate their existing deletes on the decision here; they are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions: keep unsaved rows after a no-delete import, count every owned write Review fixes: - An import without deletes (no database version yet) no longer records the file's version, which let the next startup delete the rows it had kept. The version stays unknown, and the database is saved into the file instead; a successful save records the version. A save after a partial import is refused, so the version stays unknown until an import applies the whole file. - An import removes the database's version while it runs, so its own writes do not count: one that fails partway no longer looks newer than the file and gets saved over it on the next startup. - Playlists imported without deletes keep their database items in order; the file's other items are added after their neighbor from the file. - A Download leaving a Channel or Repo counts for that Collection's config (collection_id loads its previous value). - Bulk writes count: untagging, retrying Downloads, deleting a duplicate's Download, removing many items from a playlist. Deleting Downloads by id counts channels.yaml/repos.yaml only for their Downloads. - Only a playlist's items count for playlists.yaml (author and subject Collections have items too). Collection kinds are looked up once per flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions count only the columns each config saves A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count, info_json), a Collection's counts, and a Download attempt (attempts, error) counted as config changes, so the next import of that config saved instead of importing: a hand edit to repos.yaml was overwritten after any fetch. Each model now registers only the columns its config saves. domains.yaml saves its domains' Downloads, which now count for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:07 -06:00
config_versions_lost(test_session)
# Re-import
config.initialize()
config.import_config()
# Verify file_format restored
test_session.refresh(channel)
assert channel.collection.file_format == '%(upload_date)s_%(title)s.%(ext)s'
Collections: changes wait for their config's import; domains with archives survive the import (#784) * collections: refuse playlist changes until playlists.yaml is imported While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from the config, undoing any change). Playlist changes now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * collections: refuse domain changes until domains.yaml is imported A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi starts) was undone by the import. These now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * archive: importing domains.yaml keeps domains which have archives Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives with it. A domain with Archives is now kept (and saved to the config with the next change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * videos: refuse Channel changes until channels.yaml is imported A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved, and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:33:51 -06:00
@pytest.mark.asyncio
async def test_channel_changes_wait_for_config_import(test_session, test_directory, async_client, channel_factory,
test_channels_config):
"""Until channels.yaml is imported (e.g. while WROLPi starts), Channels cannot change; the import would undo the
change (a new Channel would be deleted, with its Downloads)."""
channel = channel_factory(name='Kept')
test_session.commit()
config = get_channels_config()
config.dump_config()
assert config.get_file().is_file()
config.successful_import = False
cid, collection_id = channel.id, channel.collection_id
requests = [
('post', '/api/videos/channels', dict(name='New', directory='videos/new', url='https://example.com/new')),
('put', f'/api/videos/channels/{cid}', dict(name='Changed', directory=str(channel.directory))),
('post', f'/api/videos/channels/{cid}/tag', dict(tag_name='Tagged')),
('delete', f'/api/videos/channels/{cid}', None),
('put', f'/api/collections/{collection_id}', dict(description='changed')),
('post', f'/api/collections/{collection_id}/tag', dict(tag_name='Tagged')),
('delete', f'/api/collections/{collection_id}', None),
]
for method, url, body in requests:
kwargs = dict(content=json.dumps(body)) if body else {}
request, response = await getattr(async_client, method)(url, **kwargs)
assert response.status_code == HTTPStatus.SERVICE_UNAVAILABLE, (method, url, response.json)
assert response.json['code'] == 'CONFIG_NOT_IMPORTED'
test_session.expire_all()
channel = test_session.query(Channel).one()
assert channel.name == 'Kept'
assert channel.tag_name is None
assert channel.collection.description is None
# Once imported, Channels can change again.
config.import_config()
request, response = await async_client.put(f'/api/collections/{collection_id}',
content=json.dumps(dict(description='x')))
assert response.status_code == HTTPStatus.OK, response.json