mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 02:48:38 -04:00
946043450e
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Benchmark every database config import (#865) An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk, writes each config with its own dump_config, then times every import in startup order: a steady pass (the database already matches) and a rebuild (config-owned rows deleted, everything restored from the configs). It prints seconds, SQL statements and writes per import, and asserts the rebuild restored every row so no timed import silently skipped its work. WROLPI_CONFIG_BENCH_SCALE multiplies the row counts. The baseline for the config import rework. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Database configs carry a version the database keeps too Step 1 of the config import rework. A database config (tags, downloads, channels, domains, playlists, repos, recently viewed) and the database now each carry an integer version, so an import can tell which side is newer without a clock (a Raspberry Pi's resets at boot): - config_version (new table) holds the database's version of each config. - An after_flush hook adds one to it for every config whose rows a flush changed, in the same transaction. Each config registers what it owns (register_ownership): models, optionally only some columns, optionally only some rows (a Collection's kind; a Download's Collection's kind). Raw SQL writes to owned tables count explicitly. - dump_config writes the database's version, read with the rows it saves; save() no longer adds one for these configs. - An import compares versions first: file == database or newer imports with deletes; file older (the database changed and was not saved before a restart) saves the database instead of undoing that change; no database row (a config this code never imported) imports without deletes. The import records the file's version last. - The Import button and backup imports apply the file whatever its version (import_forced). Importers only gate their existing deletes on the decision here; they are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions: keep unsaved rows after a no-delete import, count every owned write Review fixes: - An import without deletes (no database version yet) no longer records the file's version, which let the next startup delete the rows it had kept. The version stays unknown, and the database is saved into the file instead; a successful save records the version. A save after a partial import is refused, so the version stays unknown until an import applies the whole file. - An import removes the database's version while it runs, so its own writes do not count: one that fails partway no longer looks newer than the file and gets saved over it on the next startup. - Playlists imported without deletes keep their database items in order; the file's other items are added after their neighbor from the file. - A Download leaving a Channel or Repo counts for that Collection's config (collection_id loads its previous value). - Bulk writes count: untagging, retrying Downloads, deleting a duplicate's Download, removing many items from a playlist. Deleting Downloads by id counts channels.yaml/repos.yaml only for their Downloads. - Only a playlist's items count for playlists.yaml (author and subject Collections have items too). Collection kinds are looked up once per flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config versions count only the columns each config saves A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count, info_json), a Collection's counts, and a Download attempt (attempts, error) counted as config changes, so the next import of that config saved instead of importing: a hand edit to repos.yaml was overwritten after any fetch. Each model now registers only the columns its config saves. domains.yaml saves its domains' Downloads, which now count for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
292 lines
9.5 KiB
Python
292 lines
9.5 KiB
Python
"""
|
|
Tests for DomainsConfig import/upgrade scenarios.
|
|
|
|
Tests cover:
|
|
1. Onboarding (new install) - no config file exists
|
|
2. Upgrade - config file exists with data
|
|
3. Edge cases - empty config, missing config with DB data
|
|
"""
|
|
import pytest
|
|
import yaml
|
|
|
|
from modules.archive.lib import get_domains_config
|
|
from wrolpi.collections import Collection
|
|
from wrolpi.conftest import config_was_imported
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
class TestDomainsConfigOnboarding:
|
|
"""Tests for new install scenarios where no config file exists."""
|
|
|
|
async def test_new_install_no_db_data(self, test_session, test_directory, async_client):
|
|
"""
|
|
New install with empty DB and no config file.
|
|
Should succeed with empty config.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Ensure no config file exists
|
|
if config_path.is_file():
|
|
config_path.unlink()
|
|
|
|
# Ensure DB has no domain collections
|
|
assert test_session.query(Collection).filter_by(kind='domain').count() == 0
|
|
|
|
# Import should succeed
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
async def test_new_install_with_db_data(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config file missing but DB has data.
|
|
Import should succeed (nothing to import), DB data preserved.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create a domain collection in DB
|
|
collection = Collection(
|
|
name='example.com',
|
|
kind='domain',
|
|
directory=None,
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Remove config file to simulate missing config
|
|
if config_path.is_file():
|
|
config_path.unlink()
|
|
|
|
# Import should succeed (nothing to import)
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# DB data should be preserved
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 1
|
|
assert collections[0].name == 'example.com'
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
class TestDomainsConfigUpgrade:
|
|
"""Tests for upgrade scenarios where config file exists."""
|
|
|
|
async def test_upgrade_config_exists_with_data(self, test_session, test_directory, async_client):
|
|
"""
|
|
Upgrade with existing config containing collection data.
|
|
Should import and sync DB with config.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create a collection in DB
|
|
collection = Collection(
|
|
name='test.org',
|
|
kind='domain',
|
|
directory=None,
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Dump to create config file
|
|
config.dump_config()
|
|
assert config_path.is_file()
|
|
|
|
# Now import should succeed and preserve the collection
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# Collection should still exist
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 1
|
|
assert collections[0].name == 'test.org'
|
|
|
|
async def test_upgrade_config_exists_empty_list(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config exists but has empty collections list.
|
|
Should NOT delete existing DB collections (never delete on empty).
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create a collection in DB
|
|
collection = Collection(
|
|
name='preserve.me',
|
|
kind='domain',
|
|
directory=None,
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Write empty config
|
|
config_path.parent.mkdir(parents=True, exist_ok=True)
|
|
with open(config_path, 'w') as f:
|
|
yaml.dump({'version': 0, 'collections': []}, f)
|
|
|
|
# Reinitialize config to read the empty file
|
|
config.initialize()
|
|
|
|
# Import should succeed but NOT delete the collection
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# Collection should still exist (never delete on empty config)
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 1
|
|
assert collections[0].name == 'preserve.me'
|
|
|
|
async def test_upgrade_config_missing_db_has_data(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config file missing but DB has domain collections.
|
|
Import should succeed (nothing to import), DB data preserved.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create collections in DB
|
|
for name in ['domain1.com', 'domain2.org']:
|
|
collection = Collection(
|
|
name=name,
|
|
kind='domain',
|
|
directory=None,
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Remove config file
|
|
if config_path.is_file():
|
|
config_path.unlink()
|
|
|
|
# Import should succeed (nothing to import)
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# DB data should be preserved
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 2
|
|
collection_names = {c.name for c in collections}
|
|
assert 'domain1.com' in collection_names
|
|
assert 'domain2.org' in collection_names
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
class TestDomainsConfigEdgeCases:
|
|
"""Tests for edge cases in config import."""
|
|
|
|
async def test_config_deletes_removed_items(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config removes a collection that exists in DB.
|
|
Should delete the collection from DB.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create two collections
|
|
for name in ['keep.this', 'delete.this']:
|
|
collection = Collection(
|
|
name=name,
|
|
kind='domain',
|
|
directory=None,
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Dump to config
|
|
config.dump_config()
|
|
|
|
# Modify config to remove one collection
|
|
with open(config_path) as f:
|
|
config_data = yaml.safe_load(f)
|
|
|
|
config_data['collections'] = [c for c in config_data['collections'] if c['name'] == 'keep.this']
|
|
|
|
with open(config_path, 'w') as f:
|
|
yaml.dump(config_data, f)
|
|
|
|
# Reinitialize and import
|
|
config.initialize()
|
|
config_was_imported(test_session, 'domains.yaml')
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# Only one collection should remain
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 1
|
|
assert collections[0].name == 'keep.this'
|
|
|
|
async def test_config_adds_new_items(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config has a new collection that doesn't exist in DB.
|
|
Should create the collection in DB.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Ensure no collections in DB
|
|
assert test_session.query(Collection).filter_by(kind='domain').count() == 0
|
|
|
|
# Write config with a new collection
|
|
config_data = {
|
|
'version': 0,
|
|
'collections': [{
|
|
'name': 'new.domain',
|
|
'kind': 'domain',
|
|
}]
|
|
}
|
|
|
|
config_path.parent.mkdir(parents=True, exist_ok=True)
|
|
with open(config_path, 'w') as f:
|
|
yaml.dump(config_data, f)
|
|
|
|
# Reinitialize and import
|
|
config.initialize()
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# Collection should be created
|
|
collections = test_session.query(Collection).filter_by(kind='domain').all()
|
|
assert len(collections) == 1
|
|
assert collections[0].name == 'new.domain'
|
|
|
|
async def test_config_updates_existing_items(self, test_session, test_directory, async_client):
|
|
"""
|
|
Config updates an existing collection's description.
|
|
Should update the collection in DB.
|
|
"""
|
|
config = get_domains_config()
|
|
config_path = config.get_file()
|
|
|
|
# Create a collection
|
|
collection = Collection(
|
|
name='update.me',
|
|
kind='domain',
|
|
directory=None,
|
|
description='Old description',
|
|
)
|
|
test_session.add(collection)
|
|
test_session.commit()
|
|
|
|
# Dump to config
|
|
config.dump_config()
|
|
|
|
# Modify config to change description
|
|
with open(config_path) as f:
|
|
config_data = yaml.safe_load(f)
|
|
|
|
for c in config_data['collections']:
|
|
if c['name'] == 'update.me':
|
|
c['description'] = 'New description'
|
|
|
|
with open(config_path, 'w') as f:
|
|
yaml.dump(config_data, f)
|
|
|
|
# Reinitialize and import
|
|
config.initialize()
|
|
config.import_config()
|
|
assert config.successful_import is True
|
|
|
|
# Collection should be updated
|
|
collection = test_session.query(Collection).filter_by(name='update.me').first()
|
|
assert collection is not None
|
|
assert collection.description == 'New description'
|