wrolpi/wrolpi/schema_ddl.py
Roland d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00

138 lines
5.9 KiB
Python

"""Raw SQLite DDL shared by the Alembic baseline and the test database.
SQLAlchemy models (`Base.metadata`) define the tables; this module holds everything the models
cannot express: triggers and the FTS5 full-text-search tables (see `wrolpi.fts`).
`install_raw_ddl` is executed by BOTH:
* the Alembic baseline migration (production databases), and
* `wrolpi.conftest.test_db` (test databases, after `Base.metadata.create_all`),
so tests exercise exactly the schema production runs.
All statements are idempotent (IF NOT EXISTS).
"""
# Triggers maintaining the summary columns `channel.video_count`, `channel.total_size`,
# `channel.minimum_frequency` and `file_group.effective_datetime`.
#
# These are re-authored from the old PL/pgSQL triggers (see git history of alembic/versions/).
# The old triggers had known bugs which are fixed here:
# * video_count used NEW.channel_id on DELETE (always NULL in Postgres), so counts never
# decremented, and did not fire on UPDATE OF channel_id (re-parenting).
# * total_size used NEW.id (a video/file_group id) as the channel id.
TRIGGER_DDL = [
# channel.video_count + channel.total_size follow the videos in the channel.
'''
CREATE TRIGGER IF NOT EXISTS video_insert_channel_summary
AFTER INSERT ON video WHEN new.channel_id IS NOT NULL
BEGIN
UPDATE channel SET
video_count = (SELECT COUNT(*) FROM video WHERE channel_id = channel.id),
total_size = (SELECT COALESCE(SUM(fg.size), 0) FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = channel.id)
WHERE id = new.channel_id;
END
''',
'''
CREATE TRIGGER IF NOT EXISTS video_delete_channel_summary
AFTER DELETE ON video WHEN old.channel_id IS NOT NULL
BEGIN
UPDATE channel SET
video_count = (SELECT COUNT(*) FROM video WHERE channel_id = channel.id),
total_size = (SELECT COALESCE(SUM(fg.size), 0) FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = channel.id)
WHERE id = old.channel_id;
END
''',
# Re-parenting a video (or changing its FileGroup) updates both affected channels.
'''
CREATE TRIGGER IF NOT EXISTS video_update_channel_summary
AFTER UPDATE OF channel_id, file_group_id ON video
BEGIN
UPDATE channel SET
video_count = (SELECT COUNT(*) FROM video WHERE channel_id = channel.id),
total_size = (SELECT COALESCE(SUM(fg.size), 0) FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = channel.id)
WHERE id IN (old.channel_id, new.channel_id);
END
''',
# A FileGroup's size changed; update the Channel of the Video which owns it (if any).
'''
CREATE TRIGGER IF NOT EXISTS file_group_size_channel_summary
AFTER UPDATE OF size ON file_group WHEN old.size IS NOT new.size
BEGIN
UPDATE channel SET
total_size = (SELECT COALESCE(SUM(fg.size), 0) FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = channel.id)
WHERE id = (SELECT channel_id FROM video WHERE file_group_id = new.id);
END
''',
# channel.minimum_frequency follows the Downloads of the Channel's Collection.
'''
CREATE TRIGGER IF NOT EXISTS download_insert_minimum_frequency
AFTER INSERT ON download WHEN new.collection_id IS NOT NULL
BEGIN
UPDATE channel SET
minimum_frequency = (SELECT MIN(frequency) FROM download
WHERE collection_id = channel.collection_id)
WHERE collection_id = new.collection_id;
END
''',
'''
CREATE TRIGGER IF NOT EXISTS download_update_minimum_frequency
AFTER UPDATE OF frequency, collection_id ON download
BEGIN
UPDATE channel SET
minimum_frequency = (SELECT MIN(frequency) FROM download
WHERE collection_id = channel.collection_id)
WHERE collection_id IN (old.collection_id, new.collection_id);
END
''',
'''
CREATE TRIGGER IF NOT EXISTS download_delete_minimum_frequency
AFTER DELETE ON download WHEN old.collection_id IS NOT NULL
BEGIN
UPDATE channel SET
minimum_frequency = (SELECT MIN(frequency) FROM download
WHERE collection_id = channel.collection_id)
WHERE collection_id = old.collection_id;
END
''',
# file_group.effective_datetime = COALESCE(published_datetime, download_datetime).
# Also maintained by an ORM event (wrolpi.files.models.update_effective_datetime); these
# triggers cover raw-SQL writers. The WHEN clause avoids rewriting rows the ORM already set.
'''
CREATE TRIGGER IF NOT EXISTS file_group_effective_datetime_insert
AFTER INSERT ON file_group
WHEN new.effective_datetime IS NOT COALESCE(new.published_datetime, new.download_datetime)
BEGIN
UPDATE file_group
SET effective_datetime = COALESCE(new.published_datetime, new.download_datetime)
WHERE id = new.id;
END
''',
'''
CREATE TRIGGER IF NOT EXISTS file_group_effective_datetime_update
AFTER UPDATE OF published_datetime, download_datetime ON file_group
WHEN new.effective_datetime IS NOT COALESCE(new.published_datetime, new.download_datetime)
BEGIN
UPDATE file_group
SET effective_datetime = COALESCE(new.published_datetime, new.download_datetime)
WHERE id = new.id;
END
''',
]
def install_raw_ddl(conn):
"""Install all raw DDL (triggers + FTS5) on a SQLite database.
`conn` may be a SQLAlchemy Connection (e.g. `op.get_bind()` in Alembic) or a raw
`sqlite3.Connection`. Idempotent."""
from wrolpi import fts
for statement in [*TRIGGER_DDL, *fts.FTS_DDL]:
conn.execute(statement)