Postgres -> SQLite hard cutover: the database moves into the media directory.

The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Roland 2026-07-12 13:59:20 -06:00
parent 7da89d883e
commit d9303a0ea8
No known key found for this signature in database
GPG key ID: 1BA620CE6C9A4680
150 changed files with 2749 additions and 6833 deletions

View file

@ -8,11 +8,6 @@ jobs:
api-tests-3-11:
docker:
- image: cimg/python:3.11
- image: cimg/postgres:15.6
environment:
POSTGRES_USER: postgres
POSTGRES_DB: wrolpi
POSTGRES_PASSWORD: "wrolpi"
# `large` (4 vCPU, 8 GB) so pytest-xdist's `-nauto` has room to scale.
resource_class: large
steps:
@ -41,11 +36,6 @@ jobs:
api-alembic-migrations:
docker:
- image: cimg/python:3.11
- image: cimg/postgres:15.6
environment:
POSTGRES_USER: wrolpi
POSTGRES_DB: wrolpi
POSTGRES_PASSWORD: "wrolpi"
resource_class: medium
steps:
- checkout
@ -57,24 +47,18 @@ jobs:
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- save_cache:
key: deps-3.11-{{ checksum "requirements.txt" }}-5
paths:
- "venv"
- run:
name: Create Media Directory
command: mkdir -p /tmp/wrolpi-media/config
- run:
name: Run Alembic Migrations
command: ./venv/bin/python ./main.py db upgrade
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db upgrade
- run:
name: Check for Missing Migrations
command: ./venv/bin/python ./main.py db check
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db check
api-tests-3-12:
docker:
- image: cimg/python:3.12
- image: cimg/postgres:15.6
environment:
POSTGRES_USER: postgres
POSTGRES_DB: wrolpi
POSTGRES_PASSWORD: "wrolpi"
resource_class: large
steps:
- checkout

View file

@ -11,7 +11,6 @@ tasks:
docker-compose build --parallel
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
docker-compose up -d db
docker-compose run --rm api db upgrade
# command runs each time a user starts their workspace
command: docker-compose up

View file

@ -79,13 +79,11 @@ You can try out WROLPi by running the docker containers.
6. Create volumes.
* `docker volume create --name=openstreetmap-data`
* `docker volume create --name=openstreetmap-rendered-tiles`
7. Start the database.
* `docker-compose up -d db`
8. Initialize the database.
7. Initialize the database (SQLite in the media directory).
* `docker-compose run --rm api db upgrade`
9. Start the docker containers.
8. Start the docker containers.
* `docker-compose up`
10. Browse to WROLPi.
9. Browse to WROLPi.
* https://0.0.0.0:8443
More Docker documentation is available in [docker/README.md](docker/README.md)

View file

@ -35,11 +35,17 @@ prepend_sys_path = .
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
# Revision ids ARE dates (see process_revision_directives in env.py), so filenames self-sort
# chronologically and the alembic_version table says when a database's schema was created.
file_template = %%(rev)s_%%(slug)s
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
sqlalchemy.url = postgresql://postgres:wrolpi@localhost:5432/wrolpi
# Placeholder: env.py derives the real URL from the media directory (see wrolpi.db.get_db_uri);
# `main.py db ...` overrides it explicitly.
sqlalchemy.url = sqlite:///CHANGEME
[post_write_hooks]

View file

@ -1,9 +1,6 @@
import os
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config
from sqlalchemy import pool
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
@ -26,13 +23,40 @@ from wrolpi.collections.models import Collection, CollectionItem # noqa: F401
from modules.videos.models import Video, Channel # noqa: F401
from modules.archive.models import Archive # noqa: F401
from modules.zim.models import Zim, TagZimEntry, ZimSubscription # noqa: F401
target_metadata = Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def get_url() -> str:
"""The database URL: from the ini/CLI when overridden, otherwise derived from the media directory."""
url = config.get_main_option('sqlalchemy.url')
if url and not url.startswith('sqlite:///CHANGEME'):
return url
from wrolpi.db import get_db_uri
return get_db_uri()
def include_object(object_, name, type_, reflected, compare_to):
"""Exclude tables Alembic must not manage from autogenerate/check.
The FTS5 virtual tables and their shadow tables (created by wrolpi.schema_ddl /
wrolpi.fts) plus SQLite internals would otherwise be proposed for DROP."""
if type_ == 'table':
from wrolpi.fts import FTS_TABLE_PREFIXES
if name == 'sqlite_sequence' or any(name.startswith(prefix) for prefix in FTS_TABLE_PREFIXES):
return False
return True
def process_revision_directives(context_, revision, directives):
"""New revisions get date ids (e.g. 2026_07_10_1350) instead of random hashes.
Filenames therefore self-sort chronologically, and the `alembic_version` value in a
database says when its schema was created. Underscores, not dashes: alembic forbids
`-` in revision identifiers (it is reserved for relative-revision syntax like `head-1`)."""
from datetime import datetime, timezone
script = directives[0]
script.rev_id = datetime.now(timezone.utc).strftime('%Y_%m_%d_%H%M')
def run_migrations_offline():
@ -47,12 +71,14 @@ def run_migrations_offline():
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
url=get_url(),
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
render_as_batch=True,
include_object=include_object,
process_revision_directives=process_revision_directives,
)
with context.begin_transaction():
@ -66,18 +92,18 @@ def run_migrations_online():
and associate a connection with the context.
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
from wrolpi.db import create_wrolpi_engine
connectable = create_wrolpi_engine(get_url())
with connectable.connect() as connection:
if not os.environ.get('DOCKER', '').lower().startswith('t'):
# Set role when not in a docker container
connection.execute("SET ROLE 'wrolpi'")
context.configure(
connection=connection, target_metadata=target_metadata
connection=connection,
target_metadata=target_metadata,
# SQLite's limited ALTER TABLE requires batch mode for most column changes.
render_as_batch=True,
include_object=include_object,
process_revision_directives=process_revision_directives,
)
with context.begin_transaction():

View file

@ -5,9 +5,8 @@ Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
import os
from alembic import op
from sqlalchemy.orm import Session
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
@ -16,18 +15,10 @@ down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
${upgrades if upgrades else "pass"}
if not DOCKERIZED:
session.execute('ALTER TABLE public.TABLE_NAME OWNER TO wrolpi')
def downgrade():
${downgrades if downgrades else "pass"}

View file

@ -1,33 +0,0 @@
"""FileGroup.censored
Revision ID: 00f11f309c53
Revises: 1c041aeee0fc
Create Date: 2024-06-17 14:02:09.881477
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '00f11f309c53'
down_revision = '1c041aeee0fc'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file_group ADD COLUMN censored BOOLEAN DEFAULT FALSE')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS censored')

View file

@ -1,42 +0,0 @@
"""Video textsearch coalesce fix.
Revision ID: 0400005fe2dc
Revises: 53f6780aa599
Create Date: 2022-01-04 11:00:21.776124
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '0400005fe2dc'
down_revision = '53f6780aa599'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Replace the textsearch and index. The old textsearch failed if one of the columns was null.
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS textsearch')
session.execute('''
ALTER TABLE video
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(title, '')), 'A') ||
setweight(to_tsvector('english'::regconfig, COALESCE(caption, '')), 'D')
) STORED
''')
session.execute('CREATE INDEX video_textsearch_idx ON video USING GIN(textsearch)')
def downgrade():
# The upgrade is backwards compatible.
pass

View file

@ -1,70 +0,0 @@
"""Init inventory tables.
Revision ID: 053f17f07c4e
Revises: f0d0086a73d1
Create Date: 2021-09-20 16:21:00.322598
"""
import os
from alembic import op
from sqlalchemy.orm import Session
revision = '053f17f07c4e'
down_revision = 'f0d0086a73d1'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE inventories_version (
version SERIAL PRIMARY KEY NOT NULL
)
''')
session.execute('''
CREATE TABLE public.inventory (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
viewed_at TIMESTAMP WITHOUT TIME ZONE,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
deleted_at TIMESTAMP WITHOUT TIME ZONE
)''')
session.execute('''
CREATE TABLE public.item (
id SERIAL PRIMARY KEY,
inventory_id INTEGER REFERENCES inventory(id) ON DELETE CASCADE,
brand TEXT,
name TEXT,
count DECIMAL,
item_size DECIMAL,
unit TEXT,
serving INTEGER,
category TEXT,
subcategory TEXT,
expiration_date DATE,
purchase_date DATE,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
deleted_at TIMESTAMP WITHOUT TIME ZONE
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')
session.execute('ALTER TABLE public.inventory OWNER TO wrolpi')
session.execute('ALTER TABLE public.item OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS public.inventories_version''')
session.execute('DROP TABLE IF EXISTS public.item''')
session.execute('DROP TABLE IF EXISTS public.inventory''')

View file

@ -1,33 +0,0 @@
"""Archive url index.
Revision ID: 13542a230c78
Revises: c72d305cf11e
Create Date: 2022-11-02 15:19:42.408462
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '13542a230c78'
down_revision = 'c72d305cf11e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('CREATE INDEX IF NOT EXISTS archive_url_idx ON archive(url)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS archive_url_idx')

View file

@ -1,26 +0,0 @@
"""Lower FileGroup.primary_path
Revision ID: 158973fda20f
Revises: 69e9754d1a63
Create Date: 2025-08-16 10:54:58.874094
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '158973fda20f'
down_revision = '69e9754d1a63'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
op.execute('CREATE INDEX file_group_primary_path_lower_idx ON file_group (LOWER(primary_path))')
def downgrade():
op.execute('DROP INDEX IF EXISTS file_group_primary_path_lower_idx')

View file

@ -1,35 +0,0 @@
"""Create Download.destination column.
Revision ID: 170d1be52bc7
Revises: 9ec4c765ef8d
Create Date: 2024-11-02 11:16:15.297850
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '170d1be52bc7'
down_revision = '9ec4c765ef8d'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS destination TEXT')
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS tag_names TEXT[]')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS tag_names')
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS destination')

View file

@ -1,51 +0,0 @@
"""Downloader
Revision ID: 184329ed5137
Revises: 1870bec7c81e
Create Date: 2021-11-17 19:13:58.305140
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '184329ed5137'
down_revision = '1870bec7c81e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE download (
id SERIAL PRIMARY KEY,
url TEXT NOT NULL,
status TEXT DEFAULT 'new',
next_download TIMESTAMPTZ,
last_successful_download TIMESTAMPTZ,
frequency INT,
attempts INT DEFAULT 0
)
''')
session.execute('ALTER TABLE video ADD COLUMN url TEXT')
session.execute('ALTER TABLE channel ADD COLUMN source_id TEXT')
session.execute('CREATE INDEX channel_source_id ON channel (source_id)')
if not DOCKERIZED:
session.execute('ALTER TABLE public.download OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS download')
session.execute('DROP INDEX IF EXISTS channel_source_id')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS source_id')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS url')

View file

@ -1,70 +0,0 @@
"""Initialize archive tables.
Revision ID: 1870bec7c81e
Revises: 053f17f07c4e
Create Date: 2021-09-22 18:31:15.056697
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '1870bec7c81e'
down_revision = '053f17f07c4e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE domains (
id SERIAL PRIMARY KEY,
domain TEXT NOT NULL UNIQUE,
directory TEXT NOT NULL UNIQUE
)''')
session.execute('''
CREATE TABLE url (
id SERIAL PRIMARY KEY,
domain_id INTEGER REFERENCES domains(id) NOT NULL,
url TEXT NOT NULL UNIQUE,
latest_datetime TIMESTAMPTZ
)''')
session.execute('''
CREATE TABLE archive (
id SERIAL PRIMARY KEY,
url_id INTEGER REFERENCES url(id) ON DELETE CASCADE NOT NULL,
domain_id INTEGER REFERENCES domains(id) ON DELETE CASCADE NOT NULL,
singlefile_path TEXT,
readability_path TEXT,
readability_txt_path TEXT,
readability_json_path TEXT,
screenshot_path TEXT,
title TEXT,
archive_datetime TIMESTAMPTZ,
status TEXT
)''')
session.execute('ALTER TABLE url ADD COLUMN latest_id INTEGER REFERENCES archive(id) ON DELETE CASCADE')
if not DOCKERIZED:
session.execute('ALTER TABLE public.domains OWNER TO wrolpi')
session.execute('ALTER TABLE public.url OWNER TO wrolpi')
session.execute('ALTER TABLE public.archive OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS domains CASCADE')
session.execute('DROP TABLE IF EXISTS url CASCADE')
session.execute('DROP TABLE IF EXISTS archive CASCADE')

View file

@ -1,33 +0,0 @@
"""Download location
Revision ID: 1a70196c05bb
Revises: d8fcdb7773c0
Create Date: 2022-05-02 12:01:51.210070
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '1a70196c05bb'
down_revision = 'd8fcdb7773c0'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS location TEXT')
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS error TEXT')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS location')
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS error')

View file

@ -1,36 +0,0 @@
"""Doc indexer.
Revision ID: 1c041aeee0fc
Revises: d96a04eb2785
Create Date: 2024-03-05 16:54:34.478974
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '1c041aeee0fc'
down_revision = 'd96a04eb2785'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
# Re-index Word documents.
bind = op.get_bind()
session = Session(bind=bind)
session.execute(
"""
UPDATE file_group
SET indexed=false
WHERE mimetype in
('application/msword','application/vnd.openxmlformats-officedocument.wordprocessingml.document')
""")
def downgrade():
pass

View file

@ -1,137 +0,0 @@
"""Create `Download.destination` and `Download.tag_names`, migrate from `Download.settings`.
Revision ID: 1f2325523525
Revises: 170d1be52bc7
Create Date: 2024-11-02 11:20:45.028865
"""
import os
from copy import copy
from alembic import op
from sqlalchemy import Column, Integer, Text, String, ARRAY
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
from wrolpi.common import ModelHelper
from wrolpi.dates import TZDateTime
from wrolpi.downloader import get_download_manager_config, DownloadStatus
# revision identifiers, used by Alembic.
revision = '1f2325523525'
down_revision = '170d1be52bc7'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
Base = declarative_base()
class MDownload(ModelHelper, Base):
__tablename__ = 'download'
id = Column(Integer, primary_key=True)
url = Column(String, nullable=False, unique=True)
attempts = Column(Integer, default=0)
destination = Column(Text)
downloader = Column(Text)
sub_downloader = Column(Text)
error = Column(Text)
frequency = Column(Integer)
info_json = Column(JSONB)
last_successful_download = Column(TZDateTime)
location = Column(Text)
next_download = Column(TZDateTime)
settings = Column(JSONB)
status = Column(String, default=DownloadStatus.new)
tag_names = Column(ARRAY(Text))
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
for download in session.query(MDownload):
if not download.settings:
continue
# Move `download.settings['destination']` to `download.destination`, if any.
settings = copy(download.settings) if download.settings else dict()
destination = settings.pop('destination', None)
tag_names = settings.pop('tag_names', None)
if isinstance(excluded_urls := settings.get('excluded_urls'), list):
settings['excluded_urls'] = ','.join(excluded_urls)
download.destination = destination or None
download.tag_names = tag_names or None
download.settings = settings
# This will migrate the config as-is at the time that this migration was written. This code is probably not
# safe to reuse in the future!
# Read config file directly.
try:
config = get_download_manager_config()
config._config.update(config.read_config_file())
except FileNotFoundError:
# Config does not exist, probably testing, or a new WROLPi.
return
new_downloads = []
for download in session.query(MDownload).order_by(MDownload.url):
if download.last_successful_download and not download.frequency:
# This once-download has completed, do not save it.
continue
new_download = dict(
destination=download.destination,
downloader=download.downloader,
frequency=download.frequency,
last_successful_download=download.last_successful_download,
next_download=download.next_download,
settings=download.settings,
status=download.status,
sub_downloader=download.sub_downloader,
url=download.url,
)
new_downloads.append(new_download)
# Write directly to file, bypassing usual checks.
config._config['downloads'] = new_downloads
config.write_config_data(config._config, config.get_file())
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
for download in session.query(MDownload):
if download.destination:
settings = copy(download.settings) if download.settings else dict()
settings['destination'] = download.destination or settings.get('destination')
settings['tag_names'] = download.tag_names or settings.get('tag_names')
if isinstance(excluded_urls := settings.get('excluded_urls'), str):
settings['excluded_urls'] = excluded_urls.split(',')
download.settings = settings
try:
config = get_download_manager_config()
config._config.update(config.read_config_file())
except FileNotFoundError:
# Config does not exist, probably testing, or a new WROLPi.
return
new_downloads = []
for download in session.query(MDownload).order_by(MDownload.url):
new_download = dict(
downloader=download.downloader,
frequency=download.frequency,
last_successful_download=download.last_successful_download,
next_download=download.next_download,
settings=download.settings,
status=download.status,
sub_downloader=download.sub_downloader,
url=download.url,
)
new_downloads.append(new_download)
config._config['downloads'] = new_downloads
config.write_config_data(config._config, config.get_file())

View file

@ -0,0 +1,378 @@
"""The SQLite baseline.
The entire Postgres-era migration history (85 revisions) was squashed into this baseline when
WROLPi moved its database into the media directory (SQLite). See the git history of
alembic/versions/ for the old chain.
Revision ID: 2026_07_10_1350
Revises:
Create Date: 2026-07-10 13:50:59.070646
"""
from alembic import op
import sqlalchemy as sa
import wrolpi.dates
import wrolpi.files.models
import wrolpi.media_path
# revision identifiers, used by Alembic.
revision = '2026_07_10_1350'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('directory',
sa.Column('path', wrolpi.media_path.MediaPathType(), nullable=False),
sa.Column('name', sa.String(), nullable=False),
sa.Column('idempotency', wrolpi.dates.TZDateTime(), nullable=True),
sa.PrimaryKeyConstraint('path')
)
with op.batch_alter_table('directory', schema=None) as batch_op:
batch_op.create_index('directory_name_idx', ['name'], unique=False)
op.create_table('file_group',
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=False),
sa.Column('author', sa.String(), nullable=True),
sa.Column('censored', sa.Boolean(), nullable=True),
sa.Column('data', wrolpi.files.models.FancyJSON(), nullable=True),
sa.Column('download_datetime', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('files', wrolpi.files.models.FancyJSON(), nullable=True),
sa.Column('idempotency', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('indexed', sa.Boolean(), nullable=True),
sa.Column('length', sa.BigInteger(), nullable=True),
sa.Column('mimetype', sa.String(), nullable=True),
sa.Column('model', sa.String(), nullable=True),
sa.Column('modification_datetime', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('primary_path', wrolpi.media_path.MediaPathType(), nullable=False),
sa.Column('published_datetime', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('published_modified_datetime', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('size', sa.BigInteger(), nullable=True),
sa.Column('suffix', sa.String(), nullable=True),
sa.Column('title', sa.String(), nullable=True),
sa.Column('url', sa.String(), nullable=True),
sa.Column('viewed', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('effective_datetime', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('a_text', sa.String(), nullable=True),
sa.Column('b_text', sa.String(), nullable=True),
sa.Column('c_text', sa.String(), nullable=True),
sa.Column('d_text', sa.String(), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('primary_path')
)
with op.batch_alter_table('file_group', schema=None) as batch_op:
batch_op.create_index('file_group_author_idx', ['author'], unique=False)
batch_op.create_index('file_group_directory_idx', ['directory'], unique=False)
batch_op.create_index('file_group_download_datetime_idx', ['download_datetime'], unique=False)
batch_op.create_index('file_group_effective_datetime_idx', ['effective_datetime'], unique=False)
batch_op.create_index('file_group_length_idx', ['length'], unique=False)
batch_op.create_index('file_group_mimetype_idx', ['mimetype'], unique=False)
batch_op.create_index('file_group_model_idx', ['model'], unique=False)
batch_op.create_index('file_group_modification_datetime_idx', ['modification_datetime'], unique=False)
batch_op.create_index('file_group_published_datetime_idx', ['published_datetime'], unique=False)
batch_op.create_index('file_group_published_modified_datetime_idx', ['published_modified_datetime'], unique=False)
batch_op.create_index('file_group_size_ix', ['size'], unique=False)
batch_op.create_index('file_group_url_idx', ['url'], unique=False)
batch_op.create_index('file_group_viewed_idx', ['viewed'], unique=False)
batch_op.create_index(batch_op.f('ix_file_group_suffix'), ['suffix'], unique=False)
op.create_table('tag',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(), nullable=False),
sa.Column('color', sa.String(), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('wrolpi_flag',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('refresh_complete', sa.Boolean(), nullable=True),
sa.Column('outdated_zims', sa.Boolean(), nullable=True),
sa.PrimaryKeyConstraint('id')
)
op.create_table('collection',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('kind', sa.String(), server_default='channel', nullable=False),
sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=True),
sa.Column('tag_id', sa.Integer(), nullable=True),
sa.Column('file_format', sa.String(), nullable=True),
sa.Column('created_date', sa.DateTime(), server_default=sa.text('(CURRENT_TIMESTAMP)'), nullable=False),
sa.Column('item_count', sa.Integer(), nullable=False),
sa.Column('total_size', sa.BigInteger(), nullable=False),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('directory', name='uq_collection_directory'),
sa.UniqueConstraint('name', 'kind', name='uq_collection_name_kind')
)
with op.batch_alter_table('collection', schema=None) as batch_op:
batch_op.create_index('idx_collection_item_count', ['item_count'], unique=False)
batch_op.create_index('idx_collection_kind', ['kind'], unique=False)
batch_op.create_index('idx_collection_total_size', ['total_size'], unique=False)
op.create_table('doc',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('size', sa.BigInteger(), nullable=True),
sa.Column('publisher', sa.String(), nullable=True),
sa.Column('language', sa.String(), nullable=True),
sa.Column('page_count', sa.Integer(), nullable=True),
sa.Column('subject', sa.String(), nullable=True),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('file_group_id')
)
with op.batch_alter_table('doc', schema=None) as batch_op:
batch_op.create_index('doc_size_idx', ['size'], unique=False)
op.create_table('tag_file',
sa.Column('created_at', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
sa.PrimaryKeyConstraint('tag_id', 'file_group_id'),
sa.UniqueConstraint('tag_id', 'file_group_id', name='tag_file_tag_id_file_group_id_key')
)
op.create_table('zim',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('path', wrolpi.media_path.MediaPathType(), nullable=False),
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
sa.Column('auto_search', sa.Boolean(), nullable=True),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('path', name='zim_path_key')
)
op.create_table('archive',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('collection_id', sa.Integer(), nullable=True),
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('file_group_id')
)
with op.batch_alter_table('archive', schema=None) as batch_op:
batch_op.create_index('idx_archive_collection_id', ['collection_id'], unique=False)
op.create_table('channel',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('url', sa.String(), nullable=True),
sa.Column('generate_posters', sa.Boolean(), nullable=True),
sa.Column('calculate_duration', sa.Boolean(), nullable=True),
sa.Column('download_missing_data', sa.Boolean(), nullable=True),
sa.Column('source_id', sa.String(), nullable=True),
sa.Column('refreshed', sa.Boolean(), nullable=True),
sa.Column('video_count', sa.Integer(), nullable=False),
sa.Column('total_size', sa.BigInteger(), nullable=False),
sa.Column('minimum_frequency', sa.Integer(), nullable=True),
sa.Column('info_json', sa.JSON(), nullable=True),
sa.Column('info_date', sa.Date(), nullable=True),
sa.Column('collection_id', sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
with op.batch_alter_table('channel', schema=None) as batch_op:
batch_op.create_index('channel_minimum_frequency_idx', ['minimum_frequency'], unique=False)
batch_op.create_index('channel_source_id', ['source_id'], unique=False)
batch_op.create_index('channel_total_size_idx', ['total_size'], unique=False)
batch_op.create_index('channel_url_key', ['url'], unique=True, postgresql_where=sa.text('url IS NOT NULL'), sqlite_where=sa.text('url IS NOT NULL'))
batch_op.create_index('channel_video_count_idx', ['video_count'], unique=False)
op.create_table('collection_item',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('collection_id', sa.Integer(), nullable=False),
sa.Column('item_kind', sa.String(), server_default='file', nullable=False),
sa.Column('file_group_id', sa.Integer(), nullable=True),
sa.Column('zim_id', sa.Integer(), nullable=True),
sa.Column('zim_entry', sa.Text(), nullable=True),
sa.Column('url', sa.Text(), nullable=True),
sa.Column('title', sa.Text(), nullable=True),
sa.Column('position', sa.Integer(), nullable=False),
sa.Column('added_date', sa.DateTime(), server_default=sa.text('(CURRENT_TIMESTAMP)'), nullable=False),
sa.CheckConstraint("(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL) OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL AND file_group_id IS NULL AND url IS NULL) OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)", name='ck_collection_item_kind'),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['zim_id'], ['zim.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('collection_id', 'file_group_id', name='uq_collection_file_group'),
sa.UniqueConstraint('collection_id', 'url', name='uq_collection_url'),
sa.UniqueConstraint('collection_id', 'zim_id', 'zim_entry', name='uq_collection_zim_entry')
)
with op.batch_alter_table('collection_item', schema=None) as batch_op:
batch_op.create_index('idx_collection_item_collection_id', ['collection_id'], unique=False)
batch_op.create_index('idx_collection_item_collection_position', ['collection_id', 'position'], unique=False)
batch_op.create_index('idx_collection_item_file_group_id', ['file_group_id'], unique=False)
batch_op.create_index('idx_collection_item_position', ['position'], unique=False)
op.create_table('doc_section',
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
sa.Column('doc_id', sa.Integer(), nullable=False),
sa.Column('kind', sa.String(), nullable=False),
sa.Column('ordinal', sa.Integer(), nullable=False),
sa.Column('label', sa.String(), nullable=True),
sa.Column('content', sa.Text(), nullable=True),
sa.ForeignKeyConstraint(['doc_id'], ['doc.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
with op.batch_alter_table('doc_section', schema=None) as batch_op:
batch_op.create_index('doc_section_doc_id_idx', ['doc_id', 'ordinal'], unique=False)
op.create_table('download',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('url', sa.String(), nullable=False),
sa.Column('attempts', sa.Integer(), nullable=True),
sa.Column('destination', wrolpi.media_path.MediaPathType(), nullable=True),
sa.Column('downloader', sa.Text(), nullable=True),
sa.Column('sub_downloader', sa.Text(), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('frequency', sa.Integer(), nullable=True),
sa.Column('info_json', sa.JSON(), nullable=True),
sa.Column('last_download_attempt', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('last_successful_download', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('location', sa.Text(), nullable=True),
sa.Column('next_download', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('settings', sa.JSON(), nullable=True),
sa.Column('status', sa.String(), nullable=True),
sa.Column('tag_names', sa.JSON(), nullable=True),
sa.Column('collection_id', sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('url')
)
with op.batch_alter_table('download', schema=None) as batch_op:
batch_op.create_index('idx_download_collection_id', ['collection_id'], unique=False)
batch_op.create_index('idx_download_last_download_attempt', ['last_download_attempt'], unique=False)
op.create_table('tag_zim',
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('zim_id', sa.Integer(), nullable=False),
sa.Column('zim_entry', sa.Text(), nullable=False),
sa.Column('created_at', wrolpi.dates.TZDateTime(), nullable=True),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
sa.ForeignKeyConstraint(['zim_id'], ['zim.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('tag_id', 'zim_id', 'zim_entry'),
sa.UniqueConstraint('tag_id', 'zim_id', 'zim_entry', name='tag_zim_tag_id_zim_id_zim_entry_key')
)
op.create_table('video',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.String(), nullable=True),
sa.Column('view_count', sa.BigInteger(), nullable=True),
sa.Column('ffprobe_json', sa.JSON(), nullable=True),
sa.Column('have_comments', sa.Boolean(), nullable=True),
sa.Column('comments_failed', sa.Boolean(), nullable=True),
sa.Column('channel_id', sa.Integer(), nullable=True),
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
sa.ForeignKeyConstraint(['channel_id'], ['channel.id'], ),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('file_group_id'),
sqlite_autoincrement=True
)
with op.batch_alter_table('video', schema=None) as batch_op:
batch_op.create_index('video_channel_id_idx', ['channel_id'], unique=False)
batch_op.create_index('video_source_id_idx', ['source_id'], unique=False)
batch_op.create_index('video_view_count_idx', ['view_count'], unique=False)
op.create_table('zim_subscription',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.Text(), nullable=False),
sa.Column('language', sa.Text(), nullable=False),
sa.Column('download_id', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['download_id'], ['download.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
# ### end Alembic commands ###
# Triggers (channel summaries, effective_datetime) and the FTS5 full-text-search tables.
# Shared with the test databases (wrolpi/conftest.py) so tests exercise the same schema.
from wrolpi.schema_ddl import install_raw_ddl
from wrolpi.fts import rebuild_fts
connection = op.get_bind()
install_raw_ddl(connection)
# Index any existing content (no-op on a fresh database).
rebuild_fts(connection)
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('zim_subscription')
with op.batch_alter_table('video', schema=None) as batch_op:
batch_op.drop_index('video_view_count_idx')
batch_op.drop_index('video_source_id_idx')
batch_op.drop_index('video_channel_id_idx')
op.drop_table('video')
op.drop_table('tag_zim')
with op.batch_alter_table('download', schema=None) as batch_op:
batch_op.drop_index('idx_download_last_download_attempt')
batch_op.drop_index('idx_download_collection_id')
op.drop_table('download')
with op.batch_alter_table('doc_section', schema=None) as batch_op:
batch_op.drop_index('doc_section_doc_id_idx')
op.drop_table('doc_section')
with op.batch_alter_table('collection_item', schema=None) as batch_op:
batch_op.drop_index('idx_collection_item_position')
batch_op.drop_index('idx_collection_item_file_group_id')
batch_op.drop_index('idx_collection_item_collection_position')
batch_op.drop_index('idx_collection_item_collection_id')
op.drop_table('collection_item')
with op.batch_alter_table('channel', schema=None) as batch_op:
batch_op.drop_index('channel_video_count_idx')
batch_op.drop_index('channel_url_key', postgresql_where=sa.text('url IS NOT NULL'), sqlite_where=sa.text('url IS NOT NULL'))
batch_op.drop_index('channel_total_size_idx')
batch_op.drop_index('channel_source_id')
batch_op.drop_index('channel_minimum_frequency_idx')
op.drop_table('channel')
with op.batch_alter_table('archive', schema=None) as batch_op:
batch_op.drop_index('idx_archive_collection_id')
op.drop_table('archive')
op.drop_table('zim')
op.drop_table('tag_file')
with op.batch_alter_table('doc', schema=None) as batch_op:
batch_op.drop_index('doc_size_idx')
op.drop_table('doc')
with op.batch_alter_table('collection', schema=None) as batch_op:
batch_op.drop_index('idx_collection_total_size')
batch_op.drop_index('idx_collection_kind')
batch_op.drop_index('idx_collection_item_count')
op.drop_table('collection')
op.drop_table('wrolpi_flag')
op.drop_table('tag')
with op.batch_alter_table('file_group', schema=None) as batch_op:
batch_op.drop_index(batch_op.f('ix_file_group_suffix'))
batch_op.drop_index('file_group_viewed_idx')
batch_op.drop_index('file_group_url_idx')
batch_op.drop_index('file_group_size_ix')
batch_op.drop_index('file_group_published_modified_datetime_idx')
batch_op.drop_index('file_group_published_datetime_idx')
batch_op.drop_index('file_group_modification_datetime_idx')
batch_op.drop_index('file_group_model_idx')
batch_op.drop_index('file_group_mimetype_idx')
batch_op.drop_index('file_group_length_idx')
batch_op.drop_index('file_group_effective_datetime_idx')
batch_op.drop_index('file_group_download_datetime_idx')
batch_op.drop_index('file_group_directory_idx')
batch_op.drop_index('file_group_author_idx')
op.drop_table('file_group')
with op.batch_alter_table('directory', schema=None) as batch_op:
batch_op.drop_index('directory_name_idx')
op.drop_table('directory')
# ### end Alembic commands ###

View file

@ -1,58 +0,0 @@
"""make_filegroup_directory_not_null
Revision ID: 26c5a62ec683
Revises: af4527503bdf
Create Date: 2025-12-23 20:47:11.549264
This migration makes the `directory` column NOT NULL after the previous migration
has populated it for all existing FileGroups.
"""
import os
import sqlalchemy as sa
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '26c5a62ec683'
down_revision = 'af4527503bdf'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Safety check: ensure no NULL directory values remain
result = session.execute(sa.text(
"SELECT COUNT(*) FROM file_group WHERE directory IS NULL"
))
null_count = result.scalar()
if null_count > 0:
# Fill any remaining NULL values (shouldn't happen if previous migration ran)
print(f"Warning: Found {null_count} FileGroups with NULL directory, filling from primary_path...")
session.execute(sa.text("""
UPDATE file_group
SET directory = regexp_replace(primary_path, '/[^/]+$', '')
WHERE directory IS NULL AND primary_path IS NOT NULL
"""))
session.commit()
# Make the column NOT NULL
op.alter_column('file_group', 'directory', nullable=False)
# Create index on directory column for efficient directory lookups
op.create_index('file_group_directory_idx', 'file_group', ['directory'])
if not DOCKERIZED:
session.execute(sa.text('ALTER TABLE public.file_group OWNER TO wrolpi'))
def downgrade():
# Drop the index
op.drop_index('file_group_directory_idx', 'file_group')
# Make the column nullable again
op.alter_column('file_group', 'directory', nullable=True)

View file

@ -1,137 +0,0 @@
"""MTag names migration
Revision ID: 2bfb3b50b178
Revises: 8d0d81bc9c34
Create Date: 2024-07-31 14:58:29.583029
"""
import os
import pathlib
from datetime import datetime
import yaml
from alembic import op
from sqlalchemy import Column, Integer, BigInteger, ForeignKey, Text, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
from sqlalchemy.orm.exc import NoResultFound # noqa
from wrolpi import dates
from wrolpi.common import escape_file_name, get_media_directory, ModelHelper
from wrolpi.common import get_relative_to_media_directory
from wrolpi.dates import TZDateTime
from wrolpi.tags import get_tags_config
# revision identifiers, used by Alembic.
revision = '2bfb3b50b178'
down_revision = '8d0d81bc9c34'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
Base = declarative_base()
class MFileGroup(ModelHelper, Base):
__tablename__ = 'file_group'
id: int = Column(BigInteger, primary_key=True)
primary_path: pathlib.Path = Column(String, unique=True, nullable=False)
class MTagFile(ModelHelper, Base):
__tablename__ = 'tag_file'
tag_id = Column(Integer, ForeignKey('tag.id', ondelete='CASCADE'), primary_key=True)
file_group_id = Column(BigInteger, ForeignKey('file_group.id', ondelete='CASCADE'), primary_key=True)
class MTag(ModelHelper, Base):
__tablename__ = 'tag'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True, nullable=False)
color = Column(String)
class MZim(Base):
__tablename__ = 'zim'
id = Column(Integer, primary_key=True)
path: pathlib.Path = Column(String, unique=True, nullable=False)
class MTagZimEntry(Base):
__tablename__ = 'tag_zim'
tag_id = Column(Integer, ForeignKey('tag.id', ondelete='CASCADE'), primary_key=True)
zim_id = Column(Integer, ForeignKey('zim.id', ondelete='CASCADE'), primary_key=True)
zim_entry: str = Column(Text, nullable=False, primary_key=True)
created_at: datetime = Column(TZDateTime, default=dates.now)
def save_tags(session: Session):
media_directory = get_media_directory()
tags = dict()
tag_rows = session.query(MTag)
for tag in tag_rows:
tags[tag.name] = dict(color=tag.color)
results = session.query(MTag, MTagFile, MFileGroup) \
.filter(MTagFile.tag_id == MTag.id, MTagFile.file_group_id == MFileGroup.id) \
.order_by(MFileGroup.primary_path)
tag_files = []
for tag, tag_file, file_group in results:
value = [
tag.name,
str(file_group.primary_path.relative_to(media_directory)),
# Fallback to current time if not set.
tag_file.created_at.isoformat() if tag_file.created_at else dates.now().isoformat(),
]
tag_files.append(value)
results = session.query(MTag, MZim, MTagZimEntry) \
.filter(MTag.id == MTagZimEntry.tag_id, MZim.id == MTagZimEntry.zim_id) \
.order_by(MTagZimEntry.zim_id, MTagZimEntry.zim_entry)
tag_zims = []
for tag, zim, tag_zim_entry in results:
zim: MZim
tag_zim_entry: MTagZimEntry
value = [
tag.name,
str(get_relative_to_media_directory(zim.path)),
tag_zim_entry.zim_entry,
# Fallback to current time if not set.
tag_zim_entry.created_at.isoformat() if tag_zim_entry.created_at else dates.now().isoformat(),
]
tag_zims.append(value)
# Write to the config.
with get_tags_config().get_file().open('wt') as fh:
data = {
'tag_files': tag_files,
'tag_zims': tag_zims,
'tags': tags,
}
yaml.dump(data, fh)
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
need_commit = False
tags = session.query(MTag).all()
for tag in tags:
escaped_name = escape_file_name(tag.name)
if escaped_name != tag.name:
tag.name = escaped_name
session.flush([tag, ])
need_commit = True
if need_commit:
session.commit()
save_tags(session)
def downgrade():
pass

View file

@ -1,40 +0,0 @@
"""Create MapFile table.
Revision ID: 2fb39c1861c7
Revises: 36cae041ad31
Create Date: 2022-03-03 19:13:39.997580
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '2fb39c1861c7'
down_revision = '36cae041ad31'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''CREATE TABLE map_file (
id SERIAL PRIMARY KEY,
path TEXT NOT NULL,
imported BOOLEAN DEFAULT FALSE,
size BIGINT
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.map_file OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS map_file')

View file

@ -1,42 +0,0 @@
"""Archive contents search.
Revision ID: 31896cc29b71
Revises: bddb5e95a91e
Create Date: 2021-12-31 18:53:25.326674
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '31896cc29b71'
down_revision = 'bddb5e95a91e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE archive ADD COLUMN contents TEXT')
session.execute('''
ALTER TABLE archive
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, contents), 'D')
) STORED
''')
session.execute('CREATE INDEX IF NOT EXISTS archive_textsearch_idx ON archive USING GIN(textsearch)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS archive_textsearch_idx')
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS contents')
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS textsearch')

View file

@ -1,33 +0,0 @@
"""Video censored
Revision ID: 32d96eb9122d
Revises: 184329ed5137
Create Date: 2021-12-23 14:03:53.677547
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '32d96eb9122d'
down_revision = '184329ed5137'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN censored BOOL DEFAULT false')
session.execute('CREATE INDEX video_censored ON video (censored)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS censored')
session.execute('DROP INDEX IF EXISTS video_censored')

View file

@ -1,49 +0,0 @@
"""eBook creator.
Revision ID: 365c985893b8
Revises: 5ad9efbf3570
Create Date: 2023-10-29 16:54:28.319427
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '365c985893b8'
down_revision = '5ad9efbf3570'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
UPDATE file_group
SET author=creator
FROM ebook
WHERE ebook.file_group_id = file_group.id
''')
session.execute('DROP INDEX IF EXISTS ebook_creator_idx')
session.execute('ALTER TABLE ebook DROP COLUMN IF EXISTS creator')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE ebook ADD COLUMN IF NOT EXISTS creator TEXT')
session.execute('''
UPDATE ebook
SET creator=author
FROM file_group
WHERE
ebook.file_group_id = file_group.id
AND model = 'ebook'
''')
session.execute('CREATE INDEX IF NOT EXISTS ebook_creator_idx ON ebook(creator)')

View file

@ -1,53 +0,0 @@
"""Files
Revision ID: 36cae041ad31
Revises: b97c1313f986
Create Date: 2022-02-01 11:05:57.605541
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '36cae041ad31'
down_revision = 'b97c1313f986'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''CREATE TABLE file (
id BIGSERIAL PRIMARY KEY,
path TEXT NOT NULL,
title TEXT,
mimetype TEXT,
size INTEGER,
modification_datetime TIMESTAMPTZ,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A'::"char")
) stored,
idempotency TEXT
)''')
session.execute('CREATE INDEX file_path_idx ON file(path)')
session.execute('CREATE INDEX file_mimetype_idx ON file(textsearch)')
session.execute('CREATE INDEX file_textsearch_idx ON file(textsearch)')
if not DOCKERIZED:
session.execute('ALTER TABLE public.file OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS file_textsearch_idx')
session.execute('DROP INDEX IF EXISTS file_mimetype_idx')
session.execute('DROP INDEX IF EXISTS file_path_idx')
session.execute('DROP TABLE IF EXISTS file')

View file

@ -1,242 +0,0 @@
"""File group.
Revision ID: 3b6918aeca4b
Revises: 930b9b7adb79
Create Date: 2023-02-27 14:52:46.943351
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '3b6918aeca4b'
down_revision = '930b9b7adb79'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DELETE FROM video') # noqa
session.execute('ALTER TABLE video DROP COLUMN caption_path')
session.execute('ALTER TABLE video DROP COLUMN censored')
session.execute('ALTER TABLE video DROP COLUMN favorite')
session.execute('ALTER TABLE video DROP COLUMN info_json_path')
session.execute('ALTER TABLE video DROP COLUMN modification_datetime')
session.execute('ALTER TABLE video DROP COLUMN poster_path')
session.execute('ALTER TABLE video DROP COLUMN size')
session.execute('ALTER TABLE video DROP COLUMN title')
session.execute('ALTER TABLE video DROP COLUMN validated')
session.execute('ALTER TABLE video DROP COLUMN video_path')
session.execute('DELETE FROM archive') # noqa
session.execute('ALTER TABLE archive DROP COLUMN readability_json_path')
session.execute('ALTER TABLE archive DROP COLUMN readability_path')
session.execute('ALTER TABLE archive DROP COLUMN readability_txt_path')
session.execute('ALTER TABLE archive DROP COLUMN screenshot_path')
session.execute('ALTER TABLE archive DROP COLUMN singlefile_path')
session.execute('ALTER TABLE archive DROP COLUMN title')
session.execute('ALTER TABLE archive DROP COLUMN validated')
session.execute('DELETE FROM ebook') # noqa
session.execute('ALTER TABLE ebook DROP COLUMN cover_path')
session.execute('ALTER TABLE ebook DROP COLUMN ebook_path')
session.execute('ALTER TABLE ebook DROP COLUMN title')
session.execute('DROP TABLE IF EXISTS file')
session.execute('''
CREATE TABLE file_group (
id BIGSERIAL PRIMARY KEY,
data JSON,
files JSON,
idempotency TIMESTAMP WITH TIME ZONE,
indexed BOOLEAN DEFAULT FALSE,
mimetype TEXT NOT NULL,
model TEXT,
modification_datetime TIMESTAMP WITH TIME ZONE,
primary_path TEXT NOT NULL UNIQUE,
size BIGINT,
title TEXT,
a_text TEXT,
b_text TEXT,
c_text TEXT,
d_text TEXT,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
) STORED
)''')
session.execute(
'ALTER TABLE video ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
session.execute(
'ALTER TABLE ebook ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
session.execute(
'ALTER TABLE archive ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
session.execute('''
CREATE TABLE tag (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
color VARCHAR(7)
)''')
session.execute('''
CREATE TABLE tag_file (
tag_id INTEGER REFERENCES tag(id),
file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
UNIQUE (tag_id, file_group_id)
)''')
session.execute('''
INSERT INTO tag (name, color) VALUES
('Automotive', '#353535'),
('Clothing', '#9858C5'),
('Comms', '#ECD71C'),
('Cooking', '#00A04A'),
('Electronics', '#1b731f'),
('Entertainment', '#981473'),
('Favorite', '#000000'),
('Fire Safety', '#d16711'),
('Firearms', '#6c6c6c'),
('First Aid', '#C40405'),
('Food Preservation', '#676816'),
('Food', '#2F8055'),
('Fuel', '#c47fb3'),
('Garden', '#3E8B22'),
('Heating', '#ff5400'),
('Husbandry','#B58039'),
('Hygiene', '#adadad'),
('Knots', '#9d6a4c'),
('Light', '#EEEEEE'),
('Mechanical', '#7A7A7A'),
('Medical', '#4a869c'),
('Navigation', '#1ab3ce'),
('Nuclear', '#0dff00'),
('Pests', '#050068'),
('Power', '#D837FF'),
('Primitive', '#A18A62'),
('Radio', '#134cbb'),
('Schooling', '#6d752e'),
('Security', '#333333'),
('Shelter', '#625641'),
('Social', '#490c73'),
('Tools', '#f0d730'),
('WROL', '#4700ab'),
('Water', '#291BC2')
''')
session.execute('UPDATE wrolpi_flag SET refresh_complete = false')
if not DOCKERIZED:
session.execute('ALTER TABLE public.file_group OWNER TO wrolpi')
session.execute('ALTER TABLE public.tag OWNER TO wrolpi')
session.execute('ALTER TABLE public.tag_file OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS tag_file')
session.execute('DROP TABLE IF EXISTS tag')
session.execute('ALTER TABLE video DROP COLUMN file_group_id')
session.execute('ALTER TABLE ebook DROP COLUMN file_group_id')
session.execute('ALTER TABLE archive DROP COLUMN file_group_id')
session.execute('DROP TABLE file_group')
session.execute('''
create table file
(
path text not null primary key,
idempotency timestamp with time zone,
indexed boolean default false,
mimetype text,
suffix text,
model text,
associated boolean default false,
modification_datetime timestamp with time zone,
size bigint,
title text,
a_text text,
b_text text,
c_text text,
d_text text,
textsearch tsvector generated always as ((
((setweight(to_tsvector('english'::regconfig, COALESCE(a_text, ''::text)), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, ''::text)), 'B'::"char")) ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, ''::text)), 'C'::"char")) ||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, ''::text)), 'D'::"char"))) stored,
full_stem text
);
alter table file
owner to postgres;
create index file_mimetype_idx
on file (mimetype);
create index file_textsearch_idx
on file using gin (textsearch);
create index file_modification_datetime_idx
on file (modification_datetime);
create index file_size_idx
on file (size);
create index file_model_idx
on file (model);
create index file_idempotency_idx
on file (idempotency);
create index file_suffix_idx
on file (suffix);
create index file_full_stem_idx
on file (full_stem);
''')
session.execute('DELETE FROM video') # noqa
session.execute('ALTER TABLE video ADD COLUMN caption_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE video ADD COLUMN censored BOOLEAN DEFAULT FALSE')
session.execute('ALTER TABLE video ADD COLUMN favorite TIMESTAMPTZ')
session.execute('ALTER TABLE video ADD COLUMN info_json_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE video ADD COLUMN poster_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE video ADD COLUMN video_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE video ADD COLUMN modification_datetime TIMESTAMPTZ')
session.execute('ALTER TABLE video ADD COLUMN size BIGINT')
session.execute('ALTER TABLE video ADD COLUMN title TEXT')
session.execute('ALTER TABLE video ADD COLUMN validated BOOLEAN DEFAULT FALSE')
session.execute('DELETE FROM archive') # noqa
session.execute('ALTER TABLE archive ADD COLUMN readability_json_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE archive ADD COLUMN readability_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE archive ADD COLUMN readability_txt_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE archive ADD COLUMN screenshot_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE archive ADD COLUMN singlefile_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE archive ADD COLUMN title TEXT')
session.execute('ALTER TABLE archive ADD COLUMN validated BOOLEAN DEFAULT FALSE')
session.execute('DELETE FROM ebook') # noqa
session.execute('ALTER TABLE ebook ADD COLUMN cover_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE ebook ADD COLUMN ebook_path TEXT REFERENCES file(path)')
session.execute('ALTER TABLE ebook ADD COLUMN title TEXT')
session.execute('UPDATE wrolpi_flag SET refresh_complete = false')
if not DOCKERIZED:
session.execute('ALTER TABLE public.file OWNER TO wrolpi')

View file

@ -1,35 +0,0 @@
"""Video validated.
Revision ID: 3c33b1dae069
Revises: f2c6326cad29
Create Date: 2022-01-13 16:14:23.409242
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '3c33b1dae069'
down_revision = 'f2c6326cad29'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN validated BOOLEAN DEFAULT FALSE')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS validated_poster')
# Generating posters is slow, and may delete files.
session.execute('ALTER TABLE channel ALTER COLUMN generate_posters SET DEFAULT false')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS validated')
session.execute('ALTER TABLE video ADD COLUMN validated_poster BOOLEAN DEFAULT FALSE')

View file

@ -1,29 +0,0 @@
"""Download.sub_downloader.
Revision ID: 467235201c70
Revises: 1a70196c05bb
Create Date: 2022-05-20 11:51:58.064020
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '467235201c70'
down_revision = '1a70196c05bb'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
session = Session(bind=op.get_bind())
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS sub_downloader TEXT')
def downgrade():
session = Session(bind=op.get_bind())
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS sub_downloader')

View file

@ -1,32 +0,0 @@
"""Reindex archives.
Revision ID: 4b54ad052e2e
Revises: 6b0b249a4f81
Create Date: 2023-10-20 13:28:55.517086
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '4b54ad052e2e'
down_revision = '6b0b249a4f81'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Archives must be re-indexed because of the previous revision.
session.execute("UPDATE file_group SET indexed=false WHERE model='archive'")
session.execute("DELETE FROM archive") # noqa
def downgrade():
pass

View file

@ -1,31 +0,0 @@
"""Drop Channel.skip_download_videos.
Revision ID: 4c8da5502c8a
Revises: 8bdb48a486b9
Create Date: 2023-09-29 14:45:42.999526
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '4c8da5502c8a'
down_revision = '8bdb48a486b9'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS skip_download_videos')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS skip_download_videos TEXT[]')

View file

@ -1,47 +0,0 @@
"""file_group.effective_date trigger
Revision ID: 4f03b9548f6e
Revises: 54c19aa847da
Create Date: 2025-08-17 10:49:35.754548
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '4f03b9548f6e'
down_revision = '54c19aa847da'
branch_labels = None
depends_on = None
def upgrade():
# Create trigger function
op.execute("""
CREATE OR REPLACE FUNCTION update_effective_datetime()
RETURNS TRIGGER AS
$$
BEGIN
NEW.effective_datetime = COALESCE(NEW.published_datetime, NEW.download_datetime);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""")
# Create trigger
op.execute("""
CREATE TRIGGER file_group_effective_datetime_trigger
BEFORE INSERT OR UPDATE OF published_datetime, download_datetime
ON file_group
FOR EACH ROW
EXECUTE FUNCTION update_effective_datetime();
""")
def downgrade():
# Drop trigger
op.execute("DROP TRIGGER IF EXISTS file_group_effective_datetime_trigger ON file_group")
# Drop trigger function
op.execute("DROP FUNCTION IF EXISTS update_effective_datetime")

View file

@ -1,39 +0,0 @@
"""Drop URLs.
Revision ID: 53f6780aa599
Revises: 31896cc29b71
Create Date: 2022-01-03 12:25:51.575505
"""
import os
from alembic import op
from sqlalchemy import text
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '53f6780aa599'
down_revision = '31896cc29b71'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
urls = {i['id']: i['url'] for i in session.execute(text('SELECT id, url FROM url'))}
archives = {i['id']: i['url_id'] for i in session.execute(text('SELECT id, url_id FROM archive'))}
session.execute(text('ALTER TABLE archive ADD COLUMN url TEXT'))
for archive_id, url_id in archives.items():
session.execute(
text('UPDATE archive SET url = :url WHERE id = :id'),
{'url': urls[url_id], 'id': archive_id}
)
session.execute(text('ALTER TABLE archive DROP COLUMN url_id'))
session.execute(text('DROP TABLE url'))

View file

@ -1,145 +0,0 @@
"""Channel summary columns, triggers and indexes.
Revision ID: 54c19aa847da
Revises: 158973fda20f
Create Date: 2025-08-16 11:45:48.190427
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '54c19aa847da'
down_revision = '158973fda20f'
branch_labels = None
depends_on = None
def upgrade():
# Add columns
op.execute("""
ALTER TABLE channel
ADD COLUMN video_count INTEGER DEFAULT 0 NOT NULL,
ADD COLUMN total_size BIGINT DEFAULT 0 NOT NULL,
ADD COLUMN minimum_frequency INTEGER;
""")
# Populate columns
op.execute("""
UPDATE channel
SET video_count = (SELECT COUNT(*)
FROM video
WHERE channel_id = channel.id),
total_size = (SELECT COALESCE(SUM(fg.size), 0)::BIGINT
FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = channel.id),
minimum_frequency = (SELECT MIN(frequency)
FROM download
WHERE channel_id = channel.id);
""")
# Create trigger functions
op.execute("""
CREATE OR REPLACE FUNCTION update_channel_video_count()
RETURNS TRIGGER AS
$$
BEGIN
UPDATE channel
SET video_count = (SELECT COUNT(*) FROM video WHERE channel_id = NEW.channel_id)
WHERE id = NEW.channel_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""")
op.execute("""
CREATE OR REPLACE FUNCTION update_channel_size()
RETURNS TRIGGER AS
$$
BEGIN
UPDATE channel
SET total_size = (SELECT COALESCE(SUM(fg.size), 0)::BIGINT
FROM video v
LEFT JOIN file_group fg ON v.file_group_id = fg.id
WHERE v.channel_id = NEW.id)
WHERE id = NEW.id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""")
op.execute("""
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
RETURNS TRIGGER AS
$$
BEGIN
UPDATE channel
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE channel_id = NEW.channel_id)
WHERE id = NEW.channel_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""")
# Create triggers
op.execute("""
CREATE TRIGGER video_count_trigger
AFTER INSERT OR DELETE
ON video
FOR EACH ROW
EXECUTE FUNCTION update_channel_video_count();
""")
op.execute("""
CREATE TRIGGER video_size_trigger
AFTER INSERT OR DELETE
ON video
FOR EACH ROW
EXECUTE FUNCTION update_channel_size();
""")
op.execute("""
CREATE TRIGGER file_group_size_trigger
AFTER UPDATE OF size
ON file_group
FOR EACH ROW
WHEN (OLD.size IS DISTINCT FROM NEW.size)
EXECUTE FUNCTION update_channel_size();
""")
op.execute("""
CREATE TRIGGER download_frequency_trigger
AFTER INSERT OR UPDATE OF frequency OR DELETE
ON download
FOR EACH ROW
EXECUTE FUNCTION update_channel_minimum_frequency();
""")
# Create indexes
op.execute("""
CREATE INDEX channel_video_count_idx ON channel (video_count);
CREATE INDEX channel_total_size_idx ON channel (total_size);
CREATE INDEX channel_minimum_frequency_idx ON channel (minimum_frequency);
""")
def downgrade():
# Drop triggers
op.execute("DROP TRIGGER IF EXISTS video_count_trigger ON video;")
op.execute("DROP TRIGGER IF EXISTS video_size_trigger ON video;")
op.execute("DROP TRIGGER IF EXISTS file_group_size_trigger ON file_group;")
op.execute("DROP TRIGGER IF EXISTS download_frequency_trigger ON download;")
# Drop trigger functions
op.execute("DROP FUNCTION IF EXISTS update_channel_video_count;")
op.execute("DROP FUNCTION IF EXISTS update_channel_size;")
op.execute("DROP FUNCTION IF EXISTS update_channel_minimum_frequency;")
# Drop indexes
op.execute("DROP INDEX IF EXISTS channel_minimum_frequency_idx;")
op.execute("DROP INDEX IF EXISTS channel_total_size_idx;")
op.execute("DROP INDEX IF EXISTS channel_video_count_idx;")
# Drop columns
op.execute("""
ALTER TABLE channel
DROP COLUMN IF EXISTS minimum_frequency,
DROP COLUMN IF EXISTS total_size,
DROP COLUMN IF EXISTS video_count;
""")

View file

@ -1,26 +0,0 @@
"""add_archive_collection_id_index
Revision ID: 577e414f1bc4
Revises: f38cfe4b5cb2
Create Date: 2025-12-07 17:25:27.809843
"""
from alembic import op
# revision identifiers, used by Alembic.
revision = '577e414f1bc4'
down_revision = 'f38cfe4b5cb2'
branch_labels = None
depends_on = None
def upgrade():
# Add index on archive.collection_id to speed up GROUP BY queries
# and per-collection lookups. This index was missing and causing
# full table scans on the archive table (115k+ rows).
op.create_index('idx_archive_collection_id', 'archive', ['collection_id'])
def downgrade():
op.drop_index('idx_archive_collection_id', table_name='archive')

View file

@ -1,141 +0,0 @@
"""
Migrates Video.duration/.url/.viewed/Archive.url to FileGroup.length/.url/.viewed.
Revision ID: 5ad9efbf3570
Revises: 4b54ad052e2e
Create Date: 2023-10-24 09:25:33.818373
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '5ad9efbf3570'
down_revision = '4b54ad052e2e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file_group ADD COLUMN length INT DEFAULT 0')
session.execute('''
UPDATE file_group
SET length=video.duration
FROM video
WHERE file_group.id = video.file_group_id
''')
# Migrate Archive.url and Video.url to FileGroup.url.
session.execute('ALTER TABLE file_group ADD COLUMN url TEXT')
session.execute('''
UPDATE file_group
SET url=archive.url
FROM archive
WHERE file_group.id = archive.file_group_id
''')
session.execute('''
UPDATE file_group
SET url=video.url
FROM video
WHERE file_group.id = video.file_group_id
''')
# Migrate Archive.archive_datetime to FileGroup.download_datetime.
session.execute('ALTER TABLE file_group ADD COLUMN download_datetime TIMESTAMP WITH TIME ZONE')
session.execute('''
UPDATE file_group
SET download_datetime=archive.archive_datetime
FROM archive
WHERE file_group.id = archive.file_group_id
''')
# Migrate Video.viewed to FileGroup.viewed.
session.execute('ALTER TABLE file_group ADD COLUMN viewed TIMESTAMP WITH TIME ZONE')
session.execute('''
UPDATE file_group
SET viewed=video.viewed
FROM video
WHERE file_group.id = video.file_group_id
''')
session.execute('DROP INDEX IF EXISTS archive_archive_datetime_idx')
session.execute('DROP INDEX IF EXISTS archive_url_idx')
session.execute('DROP INDEX IF EXISTS video_url_idx')
session.execute('DROP INDEX IF EXISTS video_viewed_idx')
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS archive_datetime')
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS url')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS url')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS viewed')
session.execute('CREATE INDEX file_group_download_datetime_idx ON file_group(download_datetime)')
session.execute('CREATE INDEX file_group_length_idx ON file_group(length)')
session.execute('CREATE INDEX file_group_url_idx ON file_group(url)')
session.execute('CREATE INDEX file_group_viewed_idx ON file_group(viewed)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS duration INTEGER')
session.execute('''
UPDATE video
SET duration=file_group.length
FROM file_group
WHERE file_group.id = video.file_group_id
''')
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS url TEXT')
session.execute('''
UPDATE video
SET url=file_group.url
FROM file_group
WHERE file_group.id = video.file_group_id
''')
# Video has `viewed`, Archive does not.
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS viewed TIMESTAMP WITH TIME ZONE')
session.execute('''
UPDATE video
SET url=file_group.url
FROM file_group
WHERE file_group.id = video.file_group_id
''')
session.execute('ALTER TABLE archive ADD COLUMN IF NOT EXISTS url TEXT')
session.execute('''
UPDATE archive
SET url=file_group.url
FROM file_group
WHERE file_group.id = archive.file_group_id
''')
session.execute('ALTER TABLE archive ADD COLUMN IF NOT EXISTS archive_datetime TIMESTAMP WITH TIME ZONE')
session.execute('''
UPDATE archive
SET archive_datetime=file_group.download_datetime
FROM file_group
WHERE file_group.id = archive.file_group_id
''')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS download_datetime')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS length')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS url')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS viewed')
session.execute('DROP INDEX IF EXISTS file_group_download_datetime_idx')
session.execute('DROP INDEX IF EXISTS file_group_length_idx')
session.execute('DROP INDEX IF EXISTS file_group_url_idx')
session.execute('DROP INDEX IF EXISTS file_group_viewed_idx')
session.execute('CREATE INDEX IF NOT EXISTS archive_archive_datetime_idx ON archive(archive_datetime)')
session.execute('CREATE INDEX IF NOT EXISTS archive_url_idx ON archive(url)')
session.execute('CREATE INDEX IF NOT EXISTS video_duration_idx ON video(duration)')
session.execute('CREATE INDEX IF NOT EXISTS video_url_idx ON video(url)')
session.execute('CREATE INDEX IF NOT EXISTS video_viewed_idx ON video(viewed)')

View file

@ -1,40 +0,0 @@
"""Archive textsearch coalesce fix.
Revision ID: 62acddc55091
Revises: 0400005fe2dc
Create Date: 2022-01-04 11:11:29.100317
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '62acddc55091'
down_revision = '0400005fe2dc'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS archive_textsearch_idx')
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS textsearch')
session.execute('''
ALTER TABLE archive
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(title, '')), 'A') ||
setweight(to_tsvector('english'::regconfig, COALESCE(contents, '')), 'D')
) STORED
''')
session.execute('CREATE INDEX archive_textsearch_idx ON archive USING GIN(textsearch)')
def downgrade():
# The upgrade is backwards compatible.
pass

View file

@ -1,43 +0,0 @@
"""File stem and directory
Revision ID: 6489cd50c889
Revises: 13542a230c78
Create Date: 2022-12-04 15:39:55.485355
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '6489cd50c889'
down_revision = '13542a230c78'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file ADD COLUMN directory TEXT')
session.execute('ALTER TABLE file ADD COLUMN full_stem TEXT')
session.execute('ALTER TABLE file ADD COLUMN stem TEXT')
session.execute('CREATE INDEX file_directory_idx ON file(directory)')
session.execute('CREATE INDEX file_full_stem_idx ON file(full_stem)')
session.execute('CREATE INDEX file_stem_idx ON file(stem)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS file_directory_idx')
session.execute('DROP INDEX IF EXISTS file_full_stem_idx')
session.execute('DROP INDEX IF EXISTS file_stem_idx')
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS directory')
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS full_stem')
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS stem')

View file

@ -1,87 +0,0 @@
"""Create collection table for domain collections
Revision ID: 66407d145b76
Revises: 4f03b9548f6e
Create Date: 2025-10-26 10:57:16.462524
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '66407d145b76'
down_revision = '4f03b9548f6e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Step 1: Create new collection table (keeping channel table separate for now)
op.create_table(
'collection',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('kind', sa.String(), nullable=False),
sa.Column('directory', sa.Text(), nullable=True),
sa.Column('tag_id', sa.Integer(), nullable=True),
sa.Column('created_date', sa.DateTime(), server_default=sa.func.now(), nullable=False),
sa.Column('item_count', sa.Integer(), nullable=False, server_default='0'),
sa.Column('total_size', sa.Integer(), nullable=False, server_default='0'),
sa.PrimaryKeyConstraint('id'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name='collection_tag_id_fkey'),
sa.UniqueConstraint('directory', name='uq_collection_directory')
)
# Create indexes for collection
op.create_index('idx_collection_kind', 'collection', ['kind'], unique=False)
op.create_index('idx_collection_item_count', 'collection', ['item_count'], unique=False)
op.create_index('idx_collection_total_size', 'collection', ['total_size'], unique=False)
# Step 2: Create collection_item junction table
op.create_table(
'collection_item',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('collection_id', sa.Integer(), nullable=False),
sa.Column('file_group_id', sa.Integer(), nullable=False),
sa.Column('position', sa.Integer(), nullable=False, server_default='0'),
sa.Column('added_date', sa.DateTime(), server_default=sa.func.now(), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
sa.UniqueConstraint('collection_id', 'file_group_id', name='uq_collection_file_group')
)
# Create indexes for collection_item
op.create_index('idx_collection_item_collection_id', 'collection_item', ['collection_id'], unique=False)
op.create_index('idx_collection_item_file_group_id', 'collection_item', ['file_group_id'], unique=False)
op.create_index('idx_collection_item_position', 'collection_item', ['position'], unique=False)
op.create_index('idx_collection_item_collection_position', 'collection_item', ['collection_id', 'position'], unique=False)
# Ensure table owner in non-docker environments
if not DOCKERIZED:
session.execute(sa.text('ALTER TABLE public.collection OWNER TO wrolpi'))
session.execute(sa.text('ALTER TABLE public.collection_item OWNER TO wrolpi'))
def downgrade():
# Drop collection_item table and its indexes
op.drop_index('idx_collection_item_collection_position', table_name='collection_item')
op.drop_index('idx_collection_item_position', table_name='collection_item')
op.drop_index('idx_collection_item_file_group_id', table_name='collection_item')
op.drop_index('idx_collection_item_collection_id', table_name='collection_item')
op.drop_table('collection_item')
# Drop collection table and its indexes
op.drop_index('idx_collection_total_size', table_name='collection')
op.drop_index('idx_collection_item_count', table_name='collection')
op.drop_index('idx_collection_kind', table_name='collection')
op.drop_table('collection')

View file

@ -1,37 +0,0 @@
"""FileGroup.effective_datetime
Revision ID: 69e9754d1a63
Revises: f7229a67e333
Create Date: 2025-08-16 10:28:42.661108
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '69e9754d1a63'
down_revision = 'f7229a67e333'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
op.execute('ALTER TABLE file_group ADD COLUMN IF NOT EXISTS effective_datetime TIMESTAMPTZ')
# Populating effective_datetime for existing rows
op.execute("""
UPDATE file_group
SET effective_datetime = COALESCE(published_datetime, download_datetime)
WHERE published_datetime IS NOT NULL
OR download_datetime IS NOT NULL;
""")
op.execute('CREATE INDEX IF NOT EXISTS file_group_effective_datetime_idx ON file_group(effective_datetime)')
def downgrade():
op.execute('DROP INDEX IF EXISTS file_group_effective_datetime_idx')
op.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS effective_datetime')

View file

@ -1,61 +0,0 @@
"""Archive published_datetime
Revision ID: 6b0b249a4f81
Revises: 4c8da5502c8a
Create Date: 2023-10-20 12:54:55.879180
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '6b0b249a4f81'
down_revision = '4c8da5502c8a'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file_group ADD COLUMN author TEXT')
session.execute('ALTER TABLE file_group ADD COLUMN published_datetime TIMESTAMP WITH TIME ZONE')
session.execute('ALTER TABLE file_group ADD COLUMN published_modified_datetime TIMESTAMP WITH TIME ZONE')
# Migrate Video.upload_date to FileGroup.published_datetime.
session.execute('''
UPDATE file_group
SET published_datetime=upload_date
FROM video
WHERE file_group.id = video.file_group_id
''')
session.execute('CREATE INDEX IF NOT EXISTS file_group_author_idx ON file_group(author)')
session.execute('CREATE INDEX IF NOT EXISTS file_group_published_datetime_idx ON file_group(published_datetime)')
session.execute('CREATE INDEX IF NOT EXISTS file_group_published_modified_datetime_idx'
' ON file_group(published_modified_datetime)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Migrate FileGroup.published_datetime to Video.upload_date.
session.execute('''
UPDATE video
SET upload_date=published_datetime
FROM file_group
WHERE file_group.id = video.file_group_id
''')
session.execute('DROP INDEX IF EXISTS file_group_author_idx')
session.execute('DROP INDEX IF EXISTS file_group_published_datetime_idx')
session.execute('DROP INDEX IF EXISTS file_group_published_modified_datetime_idx')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS author')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS published_datetime')
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS published_modified_datetime')

View file

@ -1,218 +0,0 @@
"""Files rework. Unifying files for Videos and Archives.
Revision ID: 70db5ba8763d
Revises: aac864072193
Create Date: 2022-07-27 20:11:30.990396
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '70db5ba8763d'
down_revision = 'aac864072193'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Recreate the file table. Use file.path as the primary key.
session.execute('DROP TABLE IF EXISTS file')
session.execute('''CREATE TABLE file (
path TEXT PRIMARY KEY,
idempotency TIMESTAMP WITH TIME ZONE,
indexed BOOLEAN DEFAULT FALSE,
mimetype TEXT,
suffix TEXT,
model TEXT,
associated BOOLEAN DEFAULT FALSE,
modification_datetime TIMESTAMP WITH TIME ZONE,
size BIGINT,
title TEXT,
a_text TEXT,
b_text TEXT,
c_text TEXT,
d_text TEXT,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
) STORED
)''')
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')
session.execute('CREATE INDEX file_modification_datetime_idx ON file USING BTREE(modification_datetime)')
session.execute('CREATE INDEX file_size_idx ON file USING BTREE(size)')
session.execute('CREATE INDEX file_model_idx ON file USING BTREE(model)')
session.execute('CREATE INDEX file_idempotency_idx ON file USING BTREE(idempotency)')
session.execute('CREATE INDEX file_suffix_idx ON file USING BTREE(suffix)')
# Videos are fundamentally different, replace the video table.
session.execute('DROP TABLE IF EXISTS video')
session.execute('''
CREATE TABLE video (
id BIGSERIAL PRIMARY KEY,
title TEXT,
source_id TEXT,
channel_id INTEGER REFERENCES channel(id),
upload_date TIMESTAMPTZ,
duration INTEGER,
favorite TIMESTAMPTZ,
size BIGINT,
viewed TIMESTAMPTZ,
view_count INTEGER,
url TEXT,
censored BOOLEAN DEFAULT FALSE,
validated BOOLEAN DEFAULT FALSE,
modification_datetime TIMESTAMPTZ,
video_path TEXT REFERENCES file(path),
info_json_path TEXT REFERENCES file(path),
poster_path TEXT REFERENCES file(path),
caption_path TEXT REFERENCES file(path)
)''')
session.execute('''
CREATE INDEX video_caption_path_idx ON video(caption_path);
CREATE INDEX video_info_json_path_idx ON video(info_json_path);
CREATE INDEX video_poster_path_idx ON video(poster_path);
CREATE INDEX video_video_path_idx ON video(video_path);
CREATE INDEX video_url_idx ON video(url);
CREATE INDEX video_channel_id_idx ON video(channel_id);
''')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS idempotency')
session.execute('DELETE FROM archive') # noqa
session.execute('DELETE FROM domains') # noqa
session.execute('DROP TABLE IF EXISTS archive')
session.execute('DROP TABLE IF EXISTS domains')
session.execute('''
CREATE TABLE domains (
id SERIAL PRIMARY KEY,
domain TEXT NOT NULL,
directory TEXT
)''')
session.execute('''
CREATE TABLE archive (
id SERIAL PRIMARY KEY,
domain_id INTEGER REFERENCES domains(id),
title TEXT,
archive_datetime timestamptz,
url TEXT,
validated BOOLEAN DEFAULT FALSE,
singlefile_path TEXT REFERENCES file(path),
readability_path TEXT REFERENCES file(path),
readability_json_path TEXT REFERENCES file(path),
readability_txt_path TEXT REFERENCES file(path),
screenshot_path TEXT REFERENCES file(path)
)''')
session.execute('''
CREATE INDEX archive_singlefile_path ON archive(singlefile_path);
CREATE INDEX archive_readability_path ON archive(readability_path);
CREATE INDEX archive_readability_json_path ON archive(readability_json_path);
CREATE INDEX archive_readability_txt_path ON archive(readability_txt_path);
CREATE INDEX archive_screenshot_path ON archive(screenshot_path);
''')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
DROP INDEX IF EXISTS archive_singlefile_path;
DROP INDEX IF EXISTS archive_readability_path;
DROP INDEX IF EXISTS archive_readability_json_path;
DROP INDEX IF EXISTS archive_readability_txt_path;
DROP INDEX IF EXISTS archive_screenshot_path;
''')
session.execute('DELETE FROM archive') # noqa
session.execute('''
ALTER TABLE archive DROP COLUMN IF EXISTS singlefile_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_json_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_txt_path;
ALTER TABLE archive DROP COLUMN IF EXISTS screenshot_path;
ALTER TABLE archive DROP COLUMN IF EXISTS associated;
''')
session.execute('''
ALTER TABLE archive ADD COLUMN IF NOT EXISTS singlefile_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_json_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_txt_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS screenshot_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS contents TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS status TEXT;
''')
session.execute('ALTER TABLE archive ALTER COLUMN domain_id SET NOT NULL')
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS idempotency TEXT')
session.execute('DELETE FROM video') # noqa
session.execute('''
ALTER TABLE video DROP COLUMN IF EXISTS caption_path;
ALTER TABLE video DROP COLUMN IF EXISTS info_json_path;
ALTER TABLE video DROP COLUMN IF EXISTS poster_path;
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
''')
session.execute('''
ALTER TABLE video ADD COLUMN IF NOT EXISTS idempotency TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS description_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS info_json_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS poster_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS video_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption TEXT;
''')
session.execute('''
DROP INDEX IF EXISTS video_caption_path_idx;
DROP INDEX IF EXISTS video_info_json_path_idx;
DROP INDEX IF EXISTS video_poster_path_idx;
DROP INDEX IF EXISTS video_video_path_idx;
DROP INDEX IF EXISTS video_url_idx;
DROP INDEX IF EXISTS video_channel_id_idx;
''')
session.execute('''
ALTER TABLE video
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, caption), 'D')
) STORED
''')
session.execute('CREATE INDEX IF NOT EXISTS video_textsearch_idx ON video USING GIN(textsearch)')
session.execute('DROP INDEX IF EXISTS file_mimetype_idx')
session.execute('DROP INDEX IF EXISTS file_textsearch_idx')
session.execute('DROP INDEX IF EXISTS file_modification_datetime_idx')
session.execute('DROP INDEX IF EXISTS file_size_idx')
session.execute('DROP INDEX IF EXISTS file_model_idx')
session.execute('DROP INDEX IF EXISTS file_idempotency_idx')
session.execute('DROP INDEX IF EXISTS file_suffix_idx')
# Restore the old file table.
session.execute('DROP TABLE IF EXISTS file')
session.execute('''CREATE TABLE file (
id SERIAL PRIMARY KEY,
path TEXT,
title TEXT,
mimetype TEXT,
size BIGINT,
modification_datetime TIMESTAMP WITH TIME ZONE,
idempotency TEXT DEFAULT NULL,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A'::"char")
) STORED
)''')
session.execute('CREATE UNIQUE INDEX file_path_idx ON file USING BTREE(path)')
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')

View file

@ -1,66 +0,0 @@
"""Ebook
Revision ID: 717f36a4da94
Revises: d0bbe45e56c3
Create Date: 2022-12-09 15:32:32.228539
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '717f36a4da94'
down_revision = 'd0bbe45e56c3'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
DROP INDEX IF EXISTS ebook_title_idx;
DROP INDEX IF EXISTS ebook_size_idx;
DROP INDEX IF EXISTS ebook_creator_idx;
DROP INDEX IF EXISTS ebook_ebook_path_idx;
DROP INDEX IF EXISTS ebook_cover_path_idx;
DROP TABLE IF EXISTS ebook;
''')
session.execute('''
CREATE TABLE ebook (
id SERIAL PRIMARY KEY,
size INTEGER,
title TEXT,
creator TEXT,
ebook_path TEXT REFERENCES file(path) ON DELETE CASCADE,
cover_path TEXT REFERENCES file(path) ON DELETE CASCADE
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.ebook OWNER TO wrolpi')
session.execute('CREATE INDEX ebook_title_idx ON ebook(title)')
session.execute('CREATE INDEX ebook_size_idx ON ebook(size)')
session.execute('CREATE INDEX ebook_creator_idx ON ebook(creator)')
session.execute('CREATE INDEX ebook_ebook_path_idx ON ebook(ebook_path)')
session.execute('CREATE INDEX ebook_cover_path_idx ON ebook(cover_path)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
DROP INDEX IF EXISTS ebook_title_idx;
DROP INDEX IF EXISTS ebook_size_idx;
DROP INDEX IF EXISTS ebook_creator_idx;
DROP INDEX IF EXISTS ebook_ebook_path_idx;
DROP INDEX IF EXISTS ebook_cover_path_idx;
DROP TABLE IF EXISTS ebook;
''')

View file

@ -1,67 +0,0 @@
"""Zim
Revision ID: 751aecf46b88
Revises: 75d34a2b442d
Create Date: 2023-06-04 19:48:41.305111
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '751aecf46b88'
down_revision = '75d34a2b442d'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE zim (
id SERIAL PRIMARY KEY,
path TEXT NOT NULL UNIQUE,
file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL
)''')
session.execute('''
CREATE TABLE tag_zim (
tag_id INTEGER REFERENCES tag(id),
zim_id INTEGER REFERENCES zim(id) ON DELETE CASCADE,
zim_entry TEXT NOT NULL,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
UNIQUE (tag_id, zim_id, zim_entry)
)''')
session.execute('''
CREATE TABLE zim_subscription (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
language TEXT NOT NULL,
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
)
''')
# We want to index all Zim files using `zim_modeler`
session.execute("UPDATE file_group SET indexed=false WHERE primary_path ILIKE '%.zim'")
if not DOCKERIZED:
session.execute('ALTER TABLE public.zim OWNER TO wrolpi')
session.execute('ALTER TABLE public.tag_zim OWNER TO wrolpi')
session.execute('ALTER TABLE public.zim_subscription OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS zim_subscription')
session.execute('DROP TABLE IF EXISTS tag_zim')
session.execute('DROP TABLE IF EXISTS zim')
session.execute("UPDATE file_group SET indexed=false WHERE primary_path ILIKE '%.zim'")

View file

@ -1,33 +0,0 @@
"""Archive datetime index.
Revision ID: 75d34a2b442d
Revises: f9928a3b6bc5
Create Date: 2023-05-15 13:09:47.661940
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '75d34a2b442d'
down_revision = 'f9928a3b6bc5'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('CREATE INDEX archive_datetime_idx ON archive(archive_datetime)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS archive_datetime_idx')

View file

@ -1,50 +0,0 @@
"""json_to_jsonb
Revision ID: 76add63ed225
Revises: 26c5a62ec683
Create Date: 2026-01-06 06:13:14.044016
"""
import os
from alembic import op
from sqlalchemy.orm import Session
from sqlalchemy import text
# revision identifiers, used by Alembic.
revision = '76add63ed225'
down_revision = '26c5a62ec683'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Convert FileGroup columns from JSON to JSONB
session.execute(text('ALTER TABLE file_group ALTER COLUMN data TYPE jsonb USING data::jsonb'))
session.execute(text('ALTER TABLE file_group ALTER COLUMN files TYPE jsonb USING files::jsonb'))
# Convert Video column from JSON to JSONB
session.execute(text('ALTER TABLE video ALTER COLUMN ffprobe_json TYPE jsonb USING ffprobe_json::jsonb'))
# Convert Channel column from JSON to JSONB
session.execute(text('ALTER TABLE channel ALTER COLUMN info_json TYPE jsonb USING info_json::jsonb'))
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Revert FileGroup columns from JSONB to JSON
session.execute(text('ALTER TABLE file_group ALTER COLUMN data TYPE json USING data::json'))
session.execute(text('ALTER TABLE file_group ALTER COLUMN files TYPE json USING files::json'))
# Revert Video column from JSONB to JSON
session.execute(text('ALTER TABLE video ALTER COLUMN ffprobe_json TYPE json USING ffprobe_json::json'))
# Revert Channel column from JSONB to JSON
session.execute(text('ALTER TABLE channel ALTER COLUMN info_json TYPE json USING info_json::json'))

View file

@ -1,31 +0,0 @@
"""Drop channel next_download
Revision ID: 84b6993a6091
Revises: 8df411232c17
Create Date: 2021-12-27 16:42:32.099796
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '84b6993a6091'
down_revision = '8df411232c17'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS next_download')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS next_download DATE')

View file

@ -1,35 +0,0 @@
"""Video ffprobe json.
Revision ID: 8bdb48a486b9
Revises: bf704baa185d
Create Date: 2023-07-24 10:59:17.426100
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '8bdb48a486b9'
down_revision = 'bf704baa185d'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN ffprobe_json JSON')
# Re-index all videos so the ffprobe data is fetched.
session.execute("UPDATE file_group SET indexed=false WHERE mimetype like 'video/%'")
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS ffprobe_json')

View file

@ -1,148 +0,0 @@
"""Channel.downloads / Download.channel
Revision ID: 8d0d81bc9c34
Revises: 00f11f309c53
Create Date: 2024-06-25 22:13:23.885687
"""
import os
import pathlib
from alembic import op
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, Text, ForeignKey
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session, deferred, relationship
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import ModelHelper
from wrolpi.dates import TZDateTime
from wrolpi.downloader import DownloadStatus
from wrolpi.media_path import MediaPathType
# revision identifiers, used by Alembic.
revision = '8d0d81bc9c34'
down_revision = '00f11f309c53'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
Base = declarative_base()
# The `Channel` model at the time of this migration.
class MChannel(ModelHelper, Base):
__tablename__ = 'channel'
id = Column(Integer, primary_key=True)
name = Column(String)
url = Column(String, unique=True)
directory: pathlib.Path = Column(MediaPathType)
generate_posters = Column(Boolean, default=False)
calculate_duration = Column(Boolean, default=True)
source_id = Column(String)
refreshed = Column(Boolean, default=False)
info_json = deferred(Column(JSON))
info_date = Column(Date)
downloads: InstrumentedList = relationship('MDownload', primaryjoin='MDownload.channel_id==MChannel.id')
@staticmethod
def get_by_url(url: str, session: Session = None):
if not url:
raise RuntimeError('Must provide URL to get Channel')
channel = session.query(MChannel).filter_by(url=url).one_or_none()
return channel
def get_rss_url(self) -> str | None:
if self.url and self.source_id and 'youtube.com' in self.url:
return f'https://www.youtube.com/feeds/videos.xml?channel_id={self.source_id}'
# The `Download` model at the time of this migration.
class MDownload(ModelHelper, Base):
__tablename__ = 'download'
id = Column(Integer, primary_key=True)
url = Column(String, nullable=False, unique=True)
attempts = Column(Integer, default=0)
downloader = Column(Text)
sub_downloader = Column(Text)
error = Column(Text)
frequency = Column(Integer)
info_json = Column(JSONB)
last_successful_download = Column(TZDateTime)
location = Column(Text)
next_download = Column(TZDateTime)
settings = Column(JSONB)
status = Column(String, default=DownloadStatus.new)
channel_id = Column(Integer, ForeignKey('channel.id'))
channel = relationship('MChannel', primaryjoin='MDownload.channel_id==MChannel.id', back_populates='downloads')
def _link_channel_and_downloads(session: Session):
"""Associate each Download with its Channel via `Download.channel_id`.
This is a frozen copy of the linking logic as it existed when this migration was written. It deliberately does
NOT import `modules.videos.lib.link_channel_and_downloads`; that function has since been rewritten for the
Collections feature and now relies on `Download.collection_id` / `Download.destination`, neither of which exists at
this point in the migration chain. A migration must remain frozen against the schema it targets.
"""
downloads = list(session.query(MDownload).all())
downloads_by_url = {i.url: i for i in downloads}
downloads_by_destination = {i.settings['destination']: i
for i in downloads if 'destination' in (i.settings or dict())}
channels = session.query(MChannel).all()
need_commit = False
for channel in channels:
directory = str(channel.directory)
download = downloads_by_destination.get(directory) or downloads_by_url.get(channel.url)
if download and not download.channel_id:
download.channel_id = channel.id
need_commit = True
# Get any Downloads for a Channel's RSS feed.
rss_url = channel.get_rss_url()
download = downloads_by_url.get(rss_url) if rss_url else None
if download and not download.channel_id:
download.channel_id = channel.id
need_commit = True
# Associate any Download which shares a Channel's URL.
for download in downloads:
if download.url and not download.channel_id:
channel = MChannel.get_by_url(download.url, session)
if channel:
download.channel_id = channel.id
need_commit = True
if need_commit:
session.commit()
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download ADD CONSTRAINT download_url_unique UNIQUE (url)')
session.execute('ALTER TABLE download ADD COLUMN channel_id INTEGER REFERENCES channel(id)')
_link_channel_and_downloads(session)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS match_regex')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS download_frequency')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS channel_download')
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS channel_id')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS channel_id')
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS match_regex TEXT')
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS download_frequency INTEGER')
session.execute('ALTER TABLE download DROP CONSTRAINT IF EXISTS download_url_unique')

View file

@ -1,37 +0,0 @@
"""Video indexes
Revision ID: 8df411232c17
Revises: 32d96eb9122d
Create Date: 2021-12-24 14:42:02.365100
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '8df411232c17'
down_revision = '32d96eb9122d'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('CREATE INDEX video_favorite ON video (favorite)')
session.execute('CREATE INDEX video_view_count ON video (view_count)')
session.execute('CREATE INDEX video_validated_poster ON video (validated_poster)')
session.execute('CREATE INDEX video_upload_date ON video (upload_date)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS video_favorite')
session.execute('DROP INDEX IF EXISTS video_view_count')
session.execute('DROP INDEX IF EXISTS video_validated_poster')
session.execute('DROP INDEX IF EXISTS video_upload_date')

View file

@ -1,39 +0,0 @@
"""Remove file directory and stem.
Revision ID: 90ac16f9140a
Revises: 717f36a4da94
Create Date: 2022-12-16 09:58:32.238873
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '90ac16f9140a'
down_revision = '717f36a4da94'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS file_directory_idx')
session.execute('DROP INDEX IF EXISTS file_stem_idx')
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS directory')
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS stem')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file ADD COLUMN directory TEXT')
session.execute('ALTER TABLE file ADD COLUMN stem TEXT')
session.execute('CREATE INDEX file_directory_idx ON file(directory)')
session.execute('CREATE INDEX file_stem_idx ON file(stem)')

View file

@ -1,39 +0,0 @@
"""WROLPi flags
Revision ID: 930b9b7adb79
Revises: 90ac16f9140a
Create Date: 2022-12-20 21:02:28.975897
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '930b9b7adb79'
down_revision = '90ac16f9140a'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''CREATE TABLE wrolpi_flag(
id SERIAL PRIMARY KEY,
refresh_complete BOOLEAN DEFAULT FALSE
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.wrolpi_flag OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS wrolpi_flag')

View file

@ -1,40 +0,0 @@
"""Remove inventories version.
Revision ID: 9ec4c765ef8d
Revises: d4e446637f18
Create Date: 2024-09-24 16:01:41.716017
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '9ec4c765ef8d'
down_revision = 'd4e446637f18'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS inventories_version')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE inventories_version (
version SERIAL PRIMARY KEY NOT NULL
)
''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')

View file

@ -1,48 +0,0 @@
"""add_collection_file_format
Revision ID: a1b2c3d4e5f6
Revises: 76add63ed225
Create Date: 2026-01-23
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'a1b2c3d4e5f6'
down_revision = '76add63ed225'
branch_labels = None
depends_on = None
def upgrade():
# Add file_format column to collection table
# This stores the file_name_format used when files were last organized
op.add_column('collection', sa.Column('file_format', sa.String(), nullable=True))
# Set default values from current configs so existing collections don't need reorganization
from modules.archive.lib import get_archive_downloader_config
from modules.videos.lib import get_videos_downloader_config
connection = op.get_bind()
# Update domain collections with archive config's file_name_format
archive_config = get_archive_downloader_config()
if archive_config.file_name_format:
connection.execute(
sa.text("UPDATE collection SET file_format = :format WHERE kind = 'domain'"),
{'format': archive_config.file_name_format}
)
# Update channel collections with video config's file_name_format
video_config = get_videos_downloader_config()
if video_config.file_name_format:
connection.execute(
sa.text("UPDATE collection SET file_format = :format WHERE kind = 'channel'"),
{'format': video_config.file_name_format}
)
def downgrade():
op.drop_column('collection', 'file_format')

View file

@ -1,29 +0,0 @@
"""Drop map_file table.
PMTiles files are now scanned directly from disk; the map_file table is no longer needed.
Revision ID: a2b3c4d5e6f7
Revises: e7f8a9b0c1d2
Create Date: 2026-03-29
"""
from alembic import op
# revision identifiers, used by Alembic.
revision = 'a2b3c4d5e6f7'
down_revision = 'e7f8a9b0c1d2'
branch_labels = None
depends_on = None
def upgrade():
op.drop_table('map_file')
def downgrade():
op.execute('''CREATE TABLE map_file (
id SERIAL PRIMARY KEY,
path TEXT NOT NULL,
imported BOOLEAN DEFAULT FALSE,
size BIGINT
)''')

View file

@ -1,58 +0,0 @@
"""Make collection_item polymorphic: file | zim | url items.
CollectionItem previously only referenced a FileGroup. Playlists (kind='playlist') also need to
order Zim articles (zim_id + zim_entry, mirroring TagZimEntry) and arbitrary URLs (a link the WROLPi
browser can open, e.g. a map location). This adds item_kind plus the zim/url/title columns, relaxes
file_group_id to nullable, and adds per-kind unique constraints and a CHECK that exactly one kind's
columns are populated. Existing rows default to item_kind='file'.
Revision ID: a7b1c2d3e4f5
Revises: e6f7a8b9c0d1
Create Date: 2026-06-08
"""
from alembic import op
import sqlalchemy as sa
revision = 'a7b1c2d3e4f5'
down_revision = 'e6f7a8b9c0d1'
branch_labels = None
depends_on = None
CHECK_SQL = (
"(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL)"
" OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL"
" AND file_group_id IS NULL AND url IS NULL)"
" OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)"
)
def upgrade():
op.add_column('collection_item',
sa.Column('item_kind', sa.String(), nullable=False, server_default='file'))
op.add_column('collection_item', sa.Column('zim_id', sa.Integer(), nullable=True))
op.add_column('collection_item', sa.Column('zim_entry', sa.Text(), nullable=True))
op.add_column('collection_item', sa.Column('url', sa.Text(), nullable=True))
op.add_column('collection_item', sa.Column('title', sa.Text(), nullable=True))
# `file` is no longer the only kind, so a row need not have a FileGroup.
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=True)
op.create_foreign_key('collection_item_zim_id_fkey', 'collection_item', 'zim',
['zim_id'], ['id'], ondelete='CASCADE')
op.create_unique_constraint('uq_collection_zim_entry', 'collection_item',
['collection_id', 'zim_id', 'zim_entry'])
op.create_unique_constraint('uq_collection_url', 'collection_item', ['collection_id', 'url'])
op.create_check_constraint('ck_collection_item_kind', 'collection_item', CHECK_SQL)
def downgrade():
op.drop_constraint('ck_collection_item_kind', 'collection_item', type_='check')
op.drop_constraint('uq_collection_url', 'collection_item', type_='unique')
op.drop_constraint('uq_collection_zim_entry', 'collection_item', type_='unique')
op.drop_constraint('collection_item_zim_id_fkey', 'collection_item', type_='foreignkey')
# Restore NOT NULL (valid only if no non-file rows remain).
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=False)
op.drop_column('collection_item', 'title')
op.drop_column('collection_item', 'url')
op.drop_column('collection_item', 'zim_entry')
op.drop_column('collection_item', 'zim_id')
op.drop_column('collection_item', 'item_kind')

View file

@ -1,30 +0,0 @@
"""Increase file size column.
Revision ID: aac864072193
Revises: 467235201c70
Create Date: 2022-06-22 17:34:33.635528
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'aac864072193'
down_revision = '467235201c70'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE file ALTER COLUMN size TYPE bigint')
def downgrade():
pass

View file

@ -1,95 +0,0 @@
"""Add unique constraint on collection (name, kind)
This migration:
1. Removes duplicate collections (keeping the one with most items/archives)
2. Adds a unique constraint on (name, kind) to prevent future duplicates
Revision ID: add_unique_collection_name_kind
Revises: migrate_download_to_collection
Create Date: 2025-11-27
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'add_unique_collection_name_kind'
down_revision = 'migrate_download_to_collection'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
print("\n" + "=" * 60)
print("Add Unique Constraint on Collection (name, kind)")
print("=" * 60 + "\n")
# Step 1: Find and remove duplicate collections
print("Step 1: Finding duplicate collections...")
# Find duplicates - keep the one with the lowest id (first created)
duplicates = session.execute(text("""
SELECT name, kind, array_agg(id ORDER BY id) as ids
FROM collection
GROUP BY name, kind
HAVING COUNT(*) > 1
""")).fetchall()
if duplicates:
print(f"Found {len(duplicates)} sets of duplicate collections")
for name, kind, ids in duplicates:
keep_id = ids[0] # Keep the first one (lowest id)
remove_ids = ids[1:] # Remove the rest
print(f" - '{name}' ({kind}): keeping id={keep_id}, removing ids={remove_ids}")
# Move any archives from duplicate collections to the one we're keeping
for remove_id in remove_ids:
session.execute(text("""
UPDATE archive SET collection_id = :keep_id
WHERE collection_id = :remove_id
"""), {'keep_id': keep_id, 'remove_id': remove_id})
# Move any downloads from duplicate collections
session.execute(text("""
UPDATE download SET collection_id = :keep_id
WHERE collection_id = :remove_id
"""), {'keep_id': keep_id, 'remove_id': remove_id})
# Move any collection items
session.execute(text("""
UPDATE collection_item SET collection_id = :keep_id
WHERE collection_id = :remove_id
"""), {'keep_id': keep_id, 'remove_id': remove_id})
# Delete the duplicate collection
session.execute(text("""
DELETE FROM collection WHERE id = :remove_id
"""), {'remove_id': remove_id})
session.commit()
print("Duplicates removed\n")
else:
print("No duplicate collections found\n")
# Step 2: Add unique constraint
print("Step 2: Adding unique constraint on (name, kind)...")
op.create_unique_constraint('uq_collection_name_kind', 'collection', ['name', 'kind'])
print("Done\n")
print("=" * 60)
print("Migration Complete")
print("=" * 60 + "\n")
if not DOCKERIZED:
session.execute(text('ALTER TABLE public.collection OWNER TO wrolpi'))
def downgrade():
op.drop_constraint('uq_collection_name_kind', 'collection', type_='unique')

View file

@ -1,192 +0,0 @@
"""add_filegroup_directory_column
Revision ID: af4527503bdf
Revises: 577e414f1bc4
Create Date: 2025-12-23 19:35:18.776018
This migration:
1. Adds `directory` column to file_group (stores parent directory of primary_path)
2. Converts paths in `files` JSON from absolute to relative (filename only)
3. Converts paths in `data` JSON from absolute to relative (filename only)
After this migration:
- directory: /media/wrolpi/videos/channel1 (absolute)
- primary_path: /media/wrolpi/videos/channel1/video.mp4 (absolute, unchanged for now)
- files: [{path: "video.mp4", ...}] (relative filenames)
- data: {poster_path: "video.jpg", ...} (relative filenames)
Processing is done in batches to handle large databases (hundreds of thousands of FileGroups)
on memory-constrained devices like Raspberry Pi.
"""
import os
import sqlalchemy as sa
from alembic import op
from sqlalchemy.orm import Session
import wrolpi.media_path
# revision identifiers, used by Alembic.
revision = 'af4527503bdf'
down_revision = '577e414f1bc4'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
# Batch size for processing - balances memory usage vs. migration speed
BATCH_SIZE = 1000
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Add the directory column (nullable initially)
# Index is created in the next migration (26c5a62ec683) AFTER data is populated for efficiency.
op.add_column('file_group', sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=True))
# Phase 1: Populate directory from existing primary_path (batched)
# Uses regexp_replace to remove the filename (everything after the last /).
print("Phase 1: Populating directory column...")
total_updated = 0
while True:
result = session.execute(sa.text("""
UPDATE file_group
SET directory = regexp_replace(primary_path, '/[^/]+$', '')
WHERE id IN (
SELECT id FROM file_group
WHERE directory IS NULL
AND primary_path IS NOT NULL
LIMIT :batch_size
)
"""), {'batch_size': BATCH_SIZE})
session.commit()
if result.rowcount == 0:
break
total_updated += result.rowcount
print(f" Processed {total_updated} directories...")
print(f"Phase 1 complete: {total_updated} directories populated.")
# Phase 2: Convert files JSON paths from absolute to relative (batched)
# Only processes rows where at least one path is still absolute (starts with /)
print("Phase 2: Converting files JSON to relative paths...")
total_updated = 0
while True:
result = session.execute(sa.text("""
UPDATE file_group
SET files = (
SELECT jsonb_agg(
jsonb_set(
elem,
'{path}',
to_jsonb(regexp_replace(elem->>'path', '.+/', ''))
)
)
FROM jsonb_array_elements(files::jsonb) AS elem
)
WHERE id IN (
SELECT id FROM file_group
WHERE files IS NOT NULL
AND jsonb_array_length(files::jsonb) > 0
AND EXISTS (
SELECT 1 FROM jsonb_array_elements(files::jsonb) AS elem
WHERE elem->>'path' LIKE '/%'
)
LIMIT :batch_size
)
"""), {'batch_size': BATCH_SIZE})
session.commit()
if result.rowcount == 0:
break
total_updated += result.rowcount
print(f" Converted {total_updated} files arrays...")
print(f"Phase 2 complete: {total_updated} files arrays converted.")
# Phase 3: Convert data JSON paths from absolute to relative (batched)
# Handle known path fields: video_path, info_json_path, poster_path, ebook_path, cover_path, screenshot_path
# Only processes rows where data contains at least one absolute path
print("Phase 3: Converting data JSON to relative paths...")
total_updated = 0
while True:
result = session.execute(sa.text("""
UPDATE file_group
SET data = (
SELECT COALESCE(
jsonb_object_agg(
key,
CASE
-- Handle path fields that are strings starting with /
-- Use ::jsonb cast instead of to_jsonb() since value::text already has JSON quotes
WHEN key IN ('video_path', 'info_json_path', 'poster_path', 'ebook_path', 'cover_path', 'screenshot_path',
'readability_json_path', 'readability_path', 'readability_txt_path', 'singlefile_path')
AND jsonb_typeof(value) = 'string'
AND value::text LIKE '"/%%'
THEN regexp_replace(value::text, '^".+/', '"')::jsonb
-- Handle caption_paths which is an array of strings
WHEN key = 'caption_paths' AND jsonb_typeof(value) = 'array'
THEN (
SELECT jsonb_agg(
CASE
WHEN jsonb_typeof(elem) = 'string' AND elem::text LIKE '"/%%'
THEN regexp_replace(elem::text, '^".+/', '"')::jsonb
ELSE elem
END
)
FROM jsonb_array_elements(value) AS elem
)
-- Keep other values unchanged
ELSE value
END
),
'{}'::jsonb
)
FROM jsonb_each(data::jsonb)
)
WHERE id IN (
SELECT id FROM file_group
WHERE data IS NOT NULL
AND data::text != 'null'
AND data::text != '{}'
AND (
-- Check if any known path field contains an absolute path
data->>'video_path' LIKE '/%'
OR data->>'info_json_path' LIKE '/%'
OR data->>'poster_path' LIKE '/%'
OR data->>'ebook_path' LIKE '/%'
OR data->>'cover_path' LIKE '/%'
OR data->>'screenshot_path' LIKE '/%'
OR data->>'readability_json_path' LIKE '/%'
OR data->>'readability_path' LIKE '/%'
OR data->>'readability_txt_path' LIKE '/%'
OR data->>'singlefile_path' LIKE '/%'
OR EXISTS (
SELECT 1 FROM jsonb_array_elements_text(
CASE WHEN data->>'caption_paths' IS NOT NULL THEN (data->'caption_paths')::jsonb ELSE '[]'::jsonb END
) AS elem
WHERE elem LIKE '/%'
)
)
LIMIT :batch_size
)
"""), {'batch_size': BATCH_SIZE})
session.commit()
if result.rowcount == 0:
break
total_updated += result.rowcount
print(f" Converted {total_updated} data objects...")
print(f"Phase 3 complete: {total_updated} data objects converted.")
print("Migration complete!")
if not DOCKERIZED:
session.execute(sa.text('ALTER TABLE public.file_group OWNER TO wrolpi'))
def downgrade():
# Note: Downgrade does not restore absolute paths in files/data columns.
# This would require knowing the original directory for each record.
# Index is dropped in the previous downgrade (26c5a62ec683).
op.drop_column('file_group', 'directory')

View file

@ -1,189 +0,0 @@
"""Fix media paths incorrectly resolved against /opt/wrolpi.
MediaPathType.process_bind_param() was calling .absolute() on relative paths,
which resolves against the current working directory (cwd) instead of the
media directory. In production, cwd is /opt/wrolpi, so paths like
'archive/spiegel.de' became '/opt/wrolpi/archive/spiegel.de' instead of
'/media/wrolpi/archive/spiegel.de'.
This migration fixes any corrupted paths in the database and saves the
corrected paths to config files.
Revision ID: b2b9700f5048
Revises: a1b2c3d4e5f6
Create Date: 2026-01-27
"""
import os
import pathlib
import yaml
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'b2b9700f5048'
down_revision = 'a1b2c3d4e5f6'
branch_labels = None
depends_on = None
def get_relative_path(absolute_path: str, media_directory: str) -> str:
"""Convert absolute path to relative path under media directory."""
if absolute_path and absolute_path.startswith(media_directory):
return absolute_path[len(media_directory):].lstrip('/')
return absolute_path
def save_config_file(config: dict, file_path: pathlib.Path, width: int = 120):
"""Write config dict to YAML file."""
file_path.parent.mkdir(parents=True, exist_ok=True)
with file_path.open('wt') as fh:
yaml.dump(config, fh, width=width, sort_keys=True)
fh.flush()
os.fsync(fh.fileno())
def upgrade():
connection = op.get_bind()
# Fix collection.directory
result = connection.execute(sa.text("""
UPDATE collection
SET directory = REPLACE(directory, '/opt/wrolpi/', '/media/wrolpi/')
WHERE directory LIKE '/opt/wrolpi/%'
"""))
if result.rowcount:
print(f"Fixed {result.rowcount} collection directory paths")
# Fix file_group.directory
result = connection.execute(sa.text("""
UPDATE file_group
SET directory = REPLACE(directory, '/opt/wrolpi/', '/media/wrolpi/')
WHERE directory LIKE '/opt/wrolpi/%'
"""))
if result.rowcount:
print(f"Fixed {result.rowcount} file_group directory paths")
# Fix file_group.primary_path
result = connection.execute(sa.text("""
UPDATE file_group
SET primary_path = REPLACE(primary_path, '/opt/wrolpi/', '/media/wrolpi/')
WHERE primary_path LIKE '/opt/wrolpi/%'
"""))
if result.rowcount:
print(f"Fixed {result.rowcount} file_group primary_path paths")
# Fix download.destination
result = connection.execute(sa.text("""
UPDATE download
SET destination = REPLACE(destination, '/opt/wrolpi/', '/media/wrolpi/')
WHERE destination LIKE '/opt/wrolpi/%'
"""))
if result.rowcount:
print(f"Fixed {result.rowcount} download destination paths")
# Save corrected paths to config files directly (bypass config framework which needs SharedContext)
try:
from wrolpi.common import get_media_directory
media_dir = str(get_media_directory())
config_dir = get_media_directory() / 'config'
# Save domains config
domains_result = connection.execute(sa.text("""
SELECT id, name, kind, description, directory, file_format
FROM collection
WHERE kind = 'domain'
ORDER BY name
"""))
domains_collections = []
for row in domains_result:
collection_config = {'name': row[1], 'kind': row[2]}
if row[3]: # description
collection_config['description'] = row[3]
if row[4]: # directory
collection_config['directory'] = get_relative_path(row[4], media_dir + '/')
if row[5]: # file_format
collection_config['file_format'] = row[5]
# Get tag name if exists
tag_result = connection.execute(sa.text("""
SELECT t.name FROM tag t
JOIN collection c ON c.tag_id = t.id
WHERE c.id = :id
"""), {'id': row[0]}).first()
collection_config['tag_name'] = tag_result[0] if tag_result else None
domains_collections.append(collection_config)
# Read existing config to preserve version
domains_file = config_dir / 'domains.yaml'
version = 0
if domains_file.exists():
with domains_file.open() as f:
existing = yaml.safe_load(f) or {}
version = existing.get('version', 0)
domains_config = {'version': version + 1, 'collections': domains_collections}
save_config_file(domains_config, domains_file)
print(f"Saved domains config with {len(domains_collections)} collections")
# Save channels config
# Columns: 0=id, 1=name, 2=kind, 3=description, 4=directory, 5=file_format,
# 6=url, 7=source_id, 8=generate_posters, 9=calculate_duration, 10=download_missing_data
channels_result = connection.execute(sa.text("""
SELECT c.id, c.name, c.kind, c.description, c.directory, c.file_format,
ch.url, ch.source_id, ch.generate_posters, ch.calculate_duration,
ch.download_missing_data
FROM collection c
JOIN channel ch ON ch.collection_id = c.id
WHERE c.kind = 'channel'
ORDER BY c.name
"""))
channels_list = []
for row in channels_result:
channel_config = {'name': row[1]}
if row[4]: # directory
channel_config['directory'] = get_relative_path(row[4], media_dir + '/')
if row[6]: # url
channel_config['url'] = row[6]
if row[7]: # source_id
channel_config['source_id'] = row[7]
channel_config['generate_posters'] = row[8]
channel_config['calculate_duration'] = row[9]
channel_config['download_missing_data'] = row[10]
if row[5]: # file_format
channel_config['file_format'] = row[5]
# Get tag name if exists
tag_result = connection.execute(sa.text("""
SELECT t.name FROM tag t
JOIN collection c ON c.tag_id = t.id
WHERE c.id = :id
"""), {'id': row[0]}).first()
channel_config['tag_name'] = tag_result[0] if tag_result else None
# Get downloads for this channel's collection
downloads_result = connection.execute(sa.text("""
SELECT url, frequency FROM download
WHERE collection_id = :collection_id
"""), {'collection_id': row[0]})
downloads = [{'url': d[0], 'frequency': d[1]} for d in downloads_result]
if downloads:
channel_config['downloads'] = downloads
channels_list.append(channel_config)
# Read existing config to preserve version
channels_file = config_dir / 'channels.yaml'
version = 0
if channels_file.exists():
with channels_file.open() as f:
existing = yaml.safe_load(f) or {}
version = existing.get('version', 0)
channels_config = {'version': version + 1, 'channels': channels_list}
save_config_file(channels_config, channels_file)
print(f"Saved channels config with {len(channels_list)} channels")
except Exception as e:
print(f"Warning: Could not save configs: {e}")
def downgrade():
# Data fix, no meaningful downgrade
pass

View file

@ -1,74 +0,0 @@
"""Drop inventory tables (inventory is now config-only).
The inventory system no longer uses the database; each inventory is a YAML file at config/inventory/<slug>.yaml.
Legacy data is migrated to those files on startup (see modules/inventory/migrate.py) before these tables are dropped.
Revision ID: b2e4f6a8c1d3
Revises: a7b1c2d3e4f5
Create Date: 2026-06-13 00:00:00.000000
"""
import os
from alembic import op
from sqlalchemy.orm import Session
revision = 'b2e4f6a8c1d3'
down_revision = 'a7b1c2d3e4f5'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# `item` first due to its foreign key to `inventory`.
session.execute('DROP TABLE IF EXISTS public.item')
session.execute('DROP TABLE IF EXISTS public.inventory')
session.execute('DROP TABLE IF EXISTS public.inventories_version')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE inventories_version (
version SERIAL PRIMARY KEY NOT NULL
)
''')
session.execute('''
CREATE TABLE public.inventory (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
viewed_at TIMESTAMP WITHOUT TIME ZONE,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
deleted_at TIMESTAMP WITHOUT TIME ZONE
)''')
session.execute('''
CREATE TABLE public.item (
id SERIAL PRIMARY KEY,
inventory_id INTEGER REFERENCES inventory(id) ON DELETE CASCADE,
brand TEXT,
name TEXT,
count DECIMAL,
item_size DECIMAL,
unit TEXT,
serving INTEGER,
category TEXT,
subcategory TEXT,
expiration_date DATE,
purchase_date DATE,
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
deleted_at TIMESTAMP WITHOUT TIME ZONE
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')
session.execute('ALTER TABLE public.inventory OWNER TO wrolpi')
session.execute('ALTER TABLE public.item OWNER TO wrolpi')

View file

@ -1,26 +0,0 @@
"""Create map_subscription table.
Revision ID: b3c4d5e6f7a8
Revises: a2b3c4d5e6f7
Create Date: 2026-03-29
"""
from alembic import op
revision = 'b3c4d5e6f7a8'
down_revision = 'a2b3c4d5e6f7'
branch_labels = None
depends_on = None
def upgrade():
op.execute('''CREATE TABLE map_subscription (
id SERIAL PRIMARY KEY,
name TEXT UNIQUE NOT NULL,
region TEXT NOT NULL,
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
)''')
def downgrade():
op.execute('DROP TABLE IF EXISTS map_subscription')

View file

@ -1,73 +0,0 @@
"""Remove duplicate Channel fields delegated to Collection
This migration removes fields from Channel that now delegate to Collection:
- name (now property: channel.name → collection.name)
- directory (now property: channel.directory → collection.directory)
- tag_id (now property: channel.tag_id → collection.tag_id)
These fields were already synced to Collection in previous migration.
Revision ID: b43f70f369d0
Revises: ba98bd360b7a
Create Date: 2025-11-19 21:48:58.488850
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = 'b43f70f369d0'
down_revision = 'ba98bd360b7a'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
print("\n" + "="*60)
print("Removing Duplicate Fields from Channel")
print("="*60 + "\n")
# Step 1: Drop foreign key constraint on tag_id
print("Step 1: Dropping tag_id foreign key constraint...")
op.drop_constraint('channel_tag_id_fkey', 'channel', type_='foreignkey')
print("✓ Dropped foreign key constraint\n")
# Step 2: Drop columns (data already in Collection)
print("Step 2: Dropping duplicate columns...")
op.drop_column('channel', 'name')
print(" ✓ Dropped channel.name")
op.drop_column('channel', 'directory')
print(" ✓ Dropped channel.directory")
op.drop_column('channel', 'tag_id')
print(" ✓ Dropped channel.tag_id\n")
print("="*60)
print("✓ Channel Cleanup Complete")
print(" Channels now delegate name/directory/tag to Collection")
print("="*60 + "\n")
def downgrade():
# Re-add the columns
op.add_column('channel', sa.Column('name', sa.String(), nullable=True))
op.add_column('channel', sa.Column('directory', sa.String(), nullable=True))
op.add_column('channel', sa.Column('tag_id', sa.Integer(), nullable=True))
# Re-add foreign key
op.create_foreign_key('channel_tag_id_fkey', 'channel', 'tag', ['tag_id'], ['id'], ondelete='CASCADE')
# Restore data from Collection
bind = op.get_bind()
bind.execute(text("""
UPDATE channel
SET name = collection.name,
directory = collection.directory,
tag_id = collection.tag_id
FROM collection
WHERE channel.collection_id = collection.id
"""))

View file

@ -1,34 +0,0 @@
"""Video modification_datetime
Revision ID: b97c1313f986
Revises: 3c33b1dae069
Create Date: 2022-01-25 08:00:26.944183
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'b97c1313f986'
down_revision = '3c33b1dae069'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN modification_datetime TIMESTAMPTZ')
session.execute('CREATE INDEX video_modification_datetime_idx ON video USING BRIN(modification_datetime)')
session.execute('UPDATE video SET validated=false') # get modification time for every video duration validation.
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS video_modification_datetime_idx')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS modification_datetime')

View file

@ -1,130 +0,0 @@
"""Add collection_id to Channel model
This migration:
1. Adds collection_id column to channel table
2. Creates Collection records for existing Channels
3. Links Channels to their Collections
4. Adds foreign key constraint
Revision ID: ba98bd360b7a
Revises: migrate_domains_to_collections
Create Date: 2025-11-19 21:10:56.472836
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'ba98bd360b7a'
down_revision = 'migrate_domains_to_collections'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
print("\n" + "="*60)
print("Channel → Collection Migration")
print("="*60 + "\n")
# Step 1: Add collection_id column to channel table (nullable initially)
print("Step 1: Adding collection_id column to channel table...")
op.add_column('channel', sa.Column('collection_id', sa.Integer(), nullable=True))
print("✓ Added collection_id column\n")
# Step 2: Create Collection records for each Channel
print("Step 2: Creating Collection records for existing Channels...")
# Fetch all channels
result = session.execute(text("""
SELECT id, name, directory, tag_id
FROM channel
ORDER BY name
"""))
channels = []
for row in result:
channels.append({
'id': row[0],
'name': row[1],
'directory': row[2],
'tag_id': row[3],
})
print(f"Found {len(channels)} Channel records to migrate")
if channels:
for channel in channels:
channel_id = channel['id']
channel_name = channel['name']
print(f" Processing Channel {channel_id}: {channel_name}")
# Check if Collection already exists (using raw SQL to avoid ORM model mismatch)
existing = session.execute(text("""
SELECT id FROM collection
WHERE name = :name AND kind = 'channel'
LIMIT 1
"""), {'name': channel_name}).first()
if existing:
print(f" Collection already exists (id={existing[0]})")
collection_id = existing[0]
else:
# Create Collection using raw SQL
result = session.execute(text("""
INSERT INTO collection (name, kind, directory, tag_id)
VALUES (:name, 'channel', :directory, :tag_id)
RETURNING id
"""), {
'name': channel_name,
'directory': channel['directory'],
'tag_id': channel['tag_id'],
})
collection_id = result.first()[0]
print(f" Created Collection id={collection_id}")
# Link Channel to Collection
session.execute(
text("UPDATE channel SET collection_id = :collection_id WHERE id = :channel_id"),
{'collection_id': collection_id, 'channel_id': channel_id}
)
session.commit()
print(f"✓ Created Collections and linked Channels\n")
else:
print("No channels to migrate\n")
# Step 3: Add foreign key constraint
print("Step 3: Adding foreign key constraint...")
op.create_foreign_key(
'fk_channel_collection_id',
'channel',
'collection',
['collection_id'],
['id'],
ondelete='CASCADE'
)
print("✓ Added foreign key constraint\n")
print("="*60)
print("✓ Channel → Collection Migration Complete")
print("="*60 + "\n")
if not DOCKERIZED:
session.execute(text('ALTER TABLE public.channel OWNER TO wrolpi'))
def downgrade():
# Remove foreign key constraint
op.drop_constraint('fk_channel_collection_id', 'channel', type_='foreignkey')
# Drop collection_id column
op.drop_column('channel', 'collection_id')

View file

@ -1,48 +0,0 @@
"""Video textsearch index
Revision ID: bddb5e95a91e
Revises: db0a1560d923
Create Date: 2021-12-31 18:12:00.407694
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'bddb5e95a91e'
down_revision = 'db0a1560d923'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS textsearch')
session.execute('''
ALTER TABLE video
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, caption), 'D')
) STORED
''')
session.execute('CREATE INDEX IF NOT EXISTS video_textsearch_idx ON video USING GIN(textsearch)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
session.execute("""
ALTER TABLE video
ADD COLUMN IF NOT EXISTS textsearch tsvector
GENERATED ALWAYS AS (to_tsvector('english'::regconfig,
((COALESCE(title, ''::text) || ' '::text) ||
COALESCE(caption, ''::text)))) STORED
""")

View file

@ -1,33 +0,0 @@
"""Zim flag
Revision ID: bf704baa185d
Revises: 751aecf46b88
Create Date: 2023-07-01 20:00:35.601752
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'bf704baa185d'
down_revision = '751aecf46b88'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE wrolpi_flag ADD COLUMN outdated_zims BOOLEAN DEFAULT FALSE')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE wrolpi_flag DROP COLUMN outdated_zims')

View file

@ -1,29 +0,0 @@
"""download_last_download_attempt
Add the `last_download_attempt` column to the `download` table. This records the last time a Download was
dispatched (whether the attempt succeeds or fails) and is used to enforce per-domain and global daily download
limits. An index supports the "attempted since local midnight" count query.
Revision ID: c3d5f7a9b1e2
Revises: b2e4f6a8c1d3
Create Date: 2026-06-28
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'c3d5f7a9b1e2'
down_revision = 'b2e4f6a8c1d3'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('download', sa.Column('last_download_attempt', sa.DateTime(timezone=True), nullable=True))
op.create_index('idx_download_last_download_attempt', 'download', ['last_download_attempt'])
def downgrade():
op.drop_index('idx_download_last_download_attempt', table_name='download')
op.drop_column('download', 'last_download_attempt')

View file

@ -1,28 +0,0 @@
"""Drop map_subscription table.
Map subscriptions are now managed via settings.regions on a single Download record.
Revision ID: c4d5e6f7a8b9
Revises: b3c4d5e6f7a8
Create Date: 2026-03-31
"""
from alembic import op
revision = 'c4d5e6f7a8b9'
down_revision = 'b3c4d5e6f7a8'
branch_labels = None
depends_on = None
def upgrade():
op.execute('DROP TABLE IF EXISTS map_subscription')
def downgrade():
op.execute('''CREATE TABLE map_subscription (
id SERIAL PRIMARY KEY,
name TEXT UNIQUE NOT NULL,
region TEXT NOT NULL,
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
)''')

View file

@ -1,29 +0,0 @@
"""Download settings
Revision ID: c72d305cf11e
Revises: d2eace02f5b9
Create Date: 2022-10-14 13:00:30.333384
"""
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'c72d305cf11e'
down_revision = 'd2eace02f5b9'
branch_labels = None
depends_on = None
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download ADD COLUMN settings JSONB')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS settings')

View file

@ -1,59 +0,0 @@
"""file_group indexes
Revision ID: c9561bda3cf0
Revises: 3b6918aeca4b
Create Date: 2023-03-25 20:45:00.470336
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'c9561bda3cf0'
down_revision = '3b6918aeca4b'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('CREATE INDEX file_group_mimetype_idx ON file_group(mimetype)')
session.execute('CREATE INDEX file_group_model_idx ON file_group(model)')
session.execute('CREATE INDEX file_group_modification_datetime_idx ON file_group(modification_datetime)')
session.execute('CREATE INDEX file_group_size_ix ON file_group(size)')
session.execute('CREATE INDEX file_group_textsearch_idx ON file_group USING GIN(textsearch)')
session.execute('CREATE INDEX archive_domain_id_idx ON archive(domain_id)')
session.execute('CREATE INDEX domains_domain_idx ON domains(domain)')
session.execute('CREATE INDEX video_source_id_idx ON video(source_id)')
session.execute('CREATE INDEX video_upload_date_idx ON video(upload_date)')
session.execute('CREATE INDEX video_view_count_idx ON video(view_count)')
session.execute('CREATE INDEX video_viewed_idx ON video(viewed)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS file_group_full_stem_idx')
session.execute('DROP INDEX IF EXISTS file_group_mimetype_idx')
session.execute('DROP INDEX IF EXISTS file_group_model_idx')
session.execute('DROP INDEX IF EXISTS file_group_modification_datetime_idx')
session.execute('DROP INDEX IF EXISTS file_group_size_ix')
session.execute('DROP INDEX IF EXISTS archive_domain_id_idx')
session.execute('DROP INDEX IF EXISTS domains_domain_idx')
session.execute('DROP INDEX IF EXISTS video_source_id_idx')
session.execute('DROP INDEX IF EXISTS video_upload_date_idx')
session.execute('DROP INDEX IF EXISTS video_view_count_idx')
session.execute('DROP INDEX IF EXISTS video_viewed_idx')

View file

@ -1,30 +0,0 @@
"""Reindex pdfs
Revision ID: d0bbe45e56c3
Revises: dbf05c8e8dcc
Create Date: 2022-12-09 12:27:18.908533
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'd0bbe45e56c3'
down_revision = 'dbf05c8e8dcc'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute("UPDATE file SET indexed=false WHERE mimetype = 'application/pdf'")
def downgrade():
pass

View file

@ -1,58 +0,0 @@
"""file.path on cascade
Revision ID: d2eace02f5b9
Revises: 70db5ba8763d
Create Date: 2022-09-28 15:14:43.306819
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'd2eace02f5b9'
down_revision = '70db5ba8763d'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
ALTER TABLE video DROP CONSTRAINT video_caption_path_fkey;
ALTER TABLE video DROP CONSTRAINT video_info_json_path_fkey;
ALTER TABLE video DROP CONSTRAINT video_poster_path_fkey;
ALTER TABLE video DROP CONSTRAINT video_video_path_fkey;
ALTER TABLE video ADD FOREIGN KEY (caption_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE video ADD FOREIGN KEY (info_json_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE video ADD FOREIGN KEY (poster_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE video ADD FOREIGN KEY (video_path) REFERENCES file ON DELETE CASCADE;
''')
session.execute('''
ALTER TABLE archive DROP CONSTRAINT archive_readability_json_path_fkey;
ALTER TABLE archive DROP CONSTRAINT archive_readability_path_fkey;
ALTER TABLE archive DROP CONSTRAINT archive_readability_txt_path_fkey;
ALTER TABLE archive DROP CONSTRAINT archive_screenshot_path_fkey;
ALTER TABLE archive DROP CONSTRAINT archive_singlefile_path_fkey;
ALTER TABLE archive ADD FOREIGN KEY (readability_json_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE archive ADD FOREIGN KEY (readability_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE archive ADD FOREIGN KEY (readability_txt_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE archive ADD FOREIGN KEY (screenshot_path) REFERENCES file ON DELETE CASCADE;
ALTER TABLE archive ADD FOREIGN KEY (singlefile_path) REFERENCES file ON DELETE CASCADE;
''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.file OWNER TO wrolpi')
session.execute('ALTER TABLE public.video OWNER TO wrolpi')
session.execute('ALTER TABLE public.domains OWNER TO wrolpi')
session.execute('ALTER TABLE public.archive OWNER TO wrolpi')
def downgrade():
pass

View file

@ -1,29 +0,0 @@
"""Video view_count bigint.
Revision ID: d371333159ab
Revises: b2b9700f5048
Create Date: 2026-02-08
This migration changes the view_count column from INTEGER to BIGINT
to accommodate YouTube videos with view counts exceeding 2,147,483,647.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'd371333159ab'
down_revision = 'b2b9700f5048'
branch_labels = None
depends_on = None
def upgrade():
op.alter_column('video', 'view_count',
existing_type=sa.Integer(),
type_=sa.BigInteger())
def downgrade():
op.alter_column('video', 'view_count',
existing_type=sa.BigInteger(),
type_=sa.Integer())

View file

@ -1,33 +0,0 @@
"""Channel tag
Revision ID: d4e446637f18
Revises: 2bfb3b50b178
Create Date: 2024-08-03 17:49:17.133086
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'd4e446637f18'
down_revision = '2bfb3b50b178'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS tag_id INTEGER REFERENCES tag(id)')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS tag_id')

View file

@ -1,69 +0,0 @@
"""file_group_suffix
Add the `suffix` column to the `file_group` table. This stores the lowercased suffix of the primary file
(e.g. ".bin") so searches can filter by file type using an index, rather than relying on full-text tokens.
`suffix` is populated at ingest time alongside `mimetype` (wrolpi.files.lib._upsert_files and
FileGroup.from_paths). Existing rows are backfilled here using the same `split_path_stem_and_suffix` logic so
special multi-part suffixes like ".info.json" are stored correctly (not a naive ".json").
Revision ID: d4e6f8a0c2b5
Revises: c3d5f7a9b1e2
Create Date: 2026-07-01
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'd4e6f8a0c2b5'
down_revision = 'c3d5f7a9b1e2'
branch_labels = None
depends_on = None
# Backfill in batches to bound memory on low-RAM devices (e.g. Raspberry Pi).
BATCH_SIZE = 5000
def upgrade():
op.add_column('file_group', sa.Column('suffix', sa.String(), nullable=True))
op.create_index('ix_file_group_suffix', 'file_group', ['suffix'])
# Backfill existing rows using the same suffix logic used at ingest time so the stored value matches what
# future refreshes would produce (including WROLPi special suffixes like ".info.json").
from wrolpi.files.lib import split_path_stem_and_suffix
conn = op.get_bind()
last_id = 0
while True:
rows = conn.execute(
sa.text('SELECT id, primary_path FROM file_group WHERE id > :last_id ORDER BY id LIMIT :limit'),
{'last_id': last_id, 'limit': BATCH_SIZE},
).fetchall()
if not rows:
break
ids, suffixes = [], []
for row_id, primary_path in rows:
suffix = (split_path_stem_and_suffix(primary_path)[1] or '').lower() or None
if suffix:
ids.append(row_id)
suffixes.append(suffix)
if ids:
conn.execute(
sa.text('''
UPDATE file_group AS fg
SET suffix = v.suffix
FROM (SELECT unnest(cast(:ids AS integer[])) AS id,
unnest(cast(:suffixes AS text[])) AS suffix) AS v
WHERE fg.id = v.id
'''),
{'ids': ids, 'suffixes': suffixes},
)
last_id = rows[-1][0]
def downgrade():
op.drop_index('ix_file_group_suffix', table_name='file_group')
op.drop_column('file_group', 'suffix')

View file

@ -1,61 +0,0 @@
"""Create doc_section table for per-chapter/per-page full-text search.
Enables deep-linking from search results into a specific EPUB chapter or PDF page,
and fixes a latent completeness bug in full-text search where FileGroup.d_text is
truncated to ~100KB (so only the first ~30 pages of a big PDF were searchable).
Existing EPUB and PDF FileGroups are marked `indexed = FALSE` so that the regular
refresh pipeline re-runs their extractors and populates doc_section rows. No one-off
script is required. Note: the next refresh on a populated library will be heavier
than usual while every EPUB and PDF is re-extracted.
Revision ID: d5e6f7a8b9c0
Revises: c4d5e6f7a8b9
Create Date: 2026-04-13
"""
from alembic import op
revision = 'd5e6f7a8b9c0'
down_revision = 'c4d5e6f7a8b9'
branch_labels = None
depends_on = None
def upgrade():
op.execute('''
CREATE TABLE doc_section (
id BIGSERIAL PRIMARY KEY,
doc_id INTEGER NOT NULL REFERENCES doc(id) ON DELETE CASCADE,
kind TEXT NOT NULL,
ordinal INTEGER NOT NULL,
label TEXT,
content TEXT,
tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english'::regconfig, COALESCE(content, ''))) STORED
)
''')
op.execute('CREATE INDEX doc_section_doc_id_idx ON doc_section (doc_id, ordinal)')
op.execute('CREATE INDEX doc_section_tsv_idx ON doc_section USING GIN (tsv)')
# Force a re-index of all EPUB and PDF file groups so the refresh pipeline
# populates the new doc_section table.
op.execute('''
UPDATE file_group
SET indexed = FALSE
WHERE mimetype LIKE 'application/epub%'
OR mimetype = 'application/pdf'
''')
def downgrade():
op.execute('DROP TABLE IF EXISTS doc_section')
# If we drop the table while some file groups still have indexed=FALSE (the
# upgrade sets that to trigger re-extraction), the next refresh would try to
# write DocSection rows to a table that no longer exists. Restore indexed=TRUE
# for those rows so a downgrade leaves the system in a consistent state.
op.execute('''
UPDATE file_group
SET indexed = TRUE
WHERE mimetype LIKE 'application/epub%'
OR mimetype = 'application/pdf'
''')

View file

@ -1,37 +0,0 @@
"""Drop channel.link.
Revision ID: d8fcdb7773c0
Revises: 2fb39c1861c7
Create Date: 2022-04-08 12:52:02.877449
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'd8fcdb7773c0'
down_revision = '2fb39c1861c7'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP CONSTRAINT IF EXISTS channel_link_key')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS link')
session.execute('DROP INDEX IF EXISTS channel_url_key')
session.execute("UPDATE channel SET url=null WHERE url=''")
session.execute('CREATE UNIQUE INDEX channel_url_key ON channel (url) WHERE url IS NOT NULL')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD CONSTRAINT channel_link_key UNIQUE (link)')
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS link TEXT')
# Notice we are not removing the channel_url_key constraint.

View file

@ -1,35 +0,0 @@
"""Video.have_comments
Revision ID: d96a04eb2785
Revises: 365c985893b8
Create Date: 2024-02-27 16:57:29.831210
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'd96a04eb2785'
down_revision = '365c985893b8'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video ADD COLUMN have_comments BOOLEAN DEFAULT FALSE')
session.execute('ALTER TABLE video ADD COLUMN comments_failed BOOLEAN DEFAULT FALSE')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS have_comments')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS comments_failed')

View file

@ -1,33 +0,0 @@
"""Downloader info
Revision ID: db0a1560d923
Revises: 84b6993a6091
Create Date: 2021-12-27 16:46:02.929119
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'db0a1560d923'
down_revision = '84b6993a6091'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS info_json JSONB')
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS downloader TEXT')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS info_json')
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS downloader')

View file

@ -1,30 +0,0 @@
"""Invalidate videos for orphaned video files.
Revision ID: dbf05c8e8dcc
Revises: 6489cd50c889
Create Date: 2022-12-07 18:26:45.017144
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'dbf05c8e8dcc'
down_revision = '6489cd50c889'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('UPDATE video SET validated=false') # noqa
def downgrade():
pass

View file

@ -1,33 +0,0 @@
"""Zim auto search
Revision ID: dc6637ced17b
Revises: 1f2325523525
Create Date: 2025-01-30 19:02:32.692428
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'dc6637ced17b'
down_revision = '1f2325523525'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE zim ADD COLUMN auto_search BOOLEAN DEFAULT TRUE')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE zim DROP COLUMN IF EXISTS auto_search')

View file

@ -1,43 +0,0 @@
"""file_group_textsearch_abc
Add the `textsearch_abc` generated tsvector column to `file_group`. It indexes only a_text (title),
b_text (author), and c_text (description) — excluding d_text (captions/document body), which is ~90% of all
text data. Searches default to this much smaller column; the existing `textsearch` column remains for
opt-in "deep" search.
Adding a STORED generated column rewrites the table; expect a few minutes on large libraries.
Revision ID: e5f7a9c1d3b6
Revises: d4e6f8a0c2b5
Create Date: 2026-07-03
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = 'e5f7a9c1d3b6'
down_revision = 'd4e6f8a0c2b5'
branch_labels = None
depends_on = None
def upgrade():
conn = op.get_bind()
conn.execute(sa.text('''
ALTER TABLE file_group
ADD COLUMN textsearch_abc tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char")
) STORED
'''))
conn.execute(sa.text(
'CREATE INDEX IF NOT EXISTS file_group_textsearch_abc_idx ON file_group USING GIN(textsearch_abc)'))
def downgrade():
conn = op.get_bind()
conn.execute(sa.text('DROP INDEX IF EXISTS file_group_textsearch_abc_idx'))
conn.execute(sa.text('ALTER TABLE file_group DROP COLUMN IF EXISTS textsearch_abc'))

View file

@ -1,43 +0,0 @@
"""Force EPUB re-extraction after spine-ordering fix.
A previous version of `extract_epub` used `book.get_items_of_type(ITEM_DOCUMENT)`
to enumerate sections, which returns manifest order (not reading-spine order) and
can include non-spine documents like EPUB 3 `nav.xhtml`. The viewer resolves
deep-links via `spine.get(n)`, so any `doc_section.ordinal` written by the old
code can land users on the wrong chapter.
The fix iterates `book.spine` directly; this migration flips `indexed = FALSE`
on every EPUB file group so the regular refresh pipeline re-extracts them with
correct spine-indexed ordinals. PDFs are untouched — their per-page ordinals
were always correct.
Revision ID: e6f7a8b9c0d1
Revises: d5e6f7a8b9c0
Create Date: 2026-04-13
"""
from alembic import op
revision = 'e6f7a8b9c0d1'
down_revision = 'd5e6f7a8b9c0'
branch_labels = None
depends_on = None
def upgrade():
op.execute('''
UPDATE file_group
SET indexed = FALSE
WHERE mimetype LIKE 'application/epub%'
''')
def downgrade():
# Symmetric with the parent migration's downgrade safeguard: if we re-apply
# downgrade, leave file groups marked indexed so a refresh doesn't try to
# repopulate doc_section rows that may have been dropped by the parent.
op.execute('''
UPDATE file_group
SET indexed = TRUE
WHERE mimetype LIKE 'application/epub%'
''')

View file

@ -1,172 +0,0 @@
"""Rename ebook table to doc and add new columns.
Revision ID: a1b2c3d4e5f6
Revises: d371333159ab
Create Date: 2026-03-15 15:30:00.000000
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'e7f8a9b0c1d2'
down_revision = 'd371333159ab'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
BATCH_SIZE = 1000
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
if not DOCKERIZED:
bind.execute(sa.text("SET ROLE 'wrolpi'"))
# Rename ebook table to doc.
print("Migrating ebook -> doc...")
op.rename_table('ebook', 'doc')
# Rename the index.
op.execute('ALTER INDEX IF EXISTS ebook_size_idx RENAME TO doc_size_idx')
op.execute('ALTER INDEX IF EXISTS ebook_file_group_id_key RENAME TO doc_file_group_id_key')
op.execute('ALTER INDEX IF EXISTS ebook_pkey RENAME TO doc_pkey')
# Rename the sequence.
op.execute('ALTER SEQUENCE IF EXISTS ebook_id_seq RENAME TO doc_id_seq')
# Add new columns.
op.add_column('doc', sa.Column('publisher', sa.String(), nullable=True))
op.add_column('doc', sa.Column('language', sa.String(), nullable=True))
op.add_column('doc', sa.Column('page_count', sa.Integer(), nullable=True))
op.add_column('doc', sa.Column('subject', sa.String(), nullable=True))
op.add_column('doc', sa.Column('description', sa.Text(), nullable=True))
# Widen size from Integer to BigInteger.
op.alter_column('doc', 'size', type_=sa.BigInteger(), existing_type=sa.Integer())
# Update file_group.model: 'ebook' -> 'doc'.
# Mark all touched file_groups as not indexed so metadata is re-extracted on next refresh.
total = 0
while True:
result = bind.execute(sa.text(
"UPDATE file_group SET model = 'doc', indexed = false"
" WHERE id IN (SELECT id FROM file_group WHERE model = 'ebook' LIMIT :batch_size)"
), {'batch_size': BATCH_SIZE})
if result.rowcount == 0:
break
total += result.rowcount
print(f" Updated {total} ebook file_groups -> doc...")
session.commit()
print(f" Updated {total} ebook file_groups -> doc (done)")
# Update file_group.model: 'pdf' -> 'doc'.
total = 0
while True:
result = bind.execute(sa.text(
"UPDATE file_group SET model = 'doc', indexed = false"
" WHERE id IN (SELECT id FROM file_group WHERE model = 'pdf' LIMIT :batch_size)"
), {'batch_size': BATCH_SIZE})
if result.rowcount == 0:
break
total += result.rowcount
print(f" Updated {total} pdf file_groups -> doc...")
session.commit()
print(f" Updated {total} pdf file_groups -> doc (done)")
# Create Doc records for existing PDFs that had no model table.
# Metadata will be extracted when the doc_modeler re-indexes these files.
total = 0
while True:
result = bind.execute(sa.text("""
INSERT INTO doc (file_group_id)
SELECT fg.id
FROM file_group fg
LEFT JOIN doc d ON d.file_group_id = fg.id
WHERE fg.mimetype = 'application/pdf'
AND d.id IS NULL
AND fg.primary_path IS NOT NULL
LIMIT :batch_size
"""), {'batch_size': BATCH_SIZE})
if result.rowcount == 0:
break
total += result.rowcount
print(f" Created {total} new doc records for existing PDFs...")
session.commit()
print(f" Created {total} new doc records for existing PDFs (done)")
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
if not DOCKERIZED:
bind.execute(sa.text("SET ROLE 'wrolpi'"))
# Delete doc records that were PDFs (have no ebook equivalent).
total = 0
while True:
result = bind.execute(sa.text("""
DELETE FROM doc
WHERE id IN (
SELECT d.id FROM doc d
JOIN file_group fg ON fg.id = d.file_group_id
WHERE fg.mimetype = 'application/pdf'
LIMIT :batch_size
)
"""), {'batch_size': BATCH_SIZE})
if result.rowcount == 0:
break
total += result.rowcount
print(f" Deleted {total} PDF doc records...")
session.commit()
print(f" Deleted {total} PDF doc records (done)")
# Revert file_group.model and re-mark as not indexed.
for old_model, mimetype_clause in [
('ebook', "mimetype LIKE 'application/epub%'"),
('ebook', "mimetype = 'application/x-mobipocket-ebook'"),
('pdf', "mimetype = 'application/pdf'"),
]:
total = 0
while True:
result = bind.execute(sa.text(f"""
UPDATE file_group SET model = :old_model, indexed = false
WHERE id IN (
SELECT id FROM file_group
WHERE model = 'doc' AND {mimetype_clause}
LIMIT :batch_size
)
"""), {'old_model': old_model, 'batch_size': BATCH_SIZE})
if result.rowcount == 0:
break
total += result.rowcount
print(f" Reverted {total} file_groups -> {old_model} ({mimetype_clause})...")
session.commit()
print(f" Reverted {total} file_groups -> {old_model} ({mimetype_clause}) (done)")
# Revert size column type.
op.alter_column('doc', 'size', type_=sa.Integer(), existing_type=sa.BigInteger())
# Drop new columns.
op.drop_column('doc', 'description')
op.drop_column('doc', 'subject')
op.drop_column('doc', 'page_count')
op.drop_column('doc', 'language')
op.drop_column('doc', 'publisher')
# Rename sequences back.
op.execute('ALTER SEQUENCE IF EXISTS doc_id_seq RENAME TO ebook_id_seq')
# Rename indexes back.
op.execute('ALTER INDEX IF EXISTS doc_pkey RENAME TO ebook_pkey')
op.execute('ALTER INDEX IF EXISTS doc_file_group_id_key RENAME TO ebook_file_group_id_key')
op.execute('ALTER INDEX IF EXISTS doc_size_idx RENAME TO ebook_size_idx')
# Rename table back.
op.rename_table('doc', 'ebook')

View file

@ -1,79 +0,0 @@
"""Init videos tables.
Revision ID: f0d0086a73d1
Revises:
Create Date: 2021-09-20 16:19:36.576217
"""
import os
from alembic import op
from sqlalchemy.orm import Session
revision = 'f0d0086a73d1'
down_revision = None
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''CREATE TABLE public.channel
(
id BIGSERIAL PRIMARY KEY,
name TEXT,
link TEXT UNIQUE,
idempotency TEXT,
url TEXT,
match_regex TEXT,
directory TEXT,
info_json JSON,
info_date DATE,
skip_download_videos text[],
generate_posters BOOLEAN DEFAULT true,
calculate_duration BOOLEAN DEFAULT true,
download_frequency INTEGER,
next_download DATE
)''')
session.execute('''CREATE TABLE public.video
(
id BIGSERIAL PRIMARY KEY,
title TEXT,
ext TEXT,
source_id TEXT,
description_path TEXT,
poster_path TEXT,
video_path TEXT,
caption_path TEXT,
info_json_path TEXT,
idempotency TEXT,
channel_id INTEGER REFERENCES channel(id) ON DELETE CASCADE,
caption TEXT,
textsearch tsvector GENERATED ALWAYS AS (to_tsvector('english'::regconfig,
((COALESCE(title, ''::text) || ' '::text) ||
COALESCE(caption, ''::text)))) STORED,
upload_date TIMESTAMP WITHOUT TIME ZONE,
duration INTEGER,
favorite TIMESTAMP WITHOUT TIME ZONE,
size BIGINT,
viewed TIMESTAMP WITHOUT TIME ZONE,
validated_poster BOOLEAN DEFAULT false,
view_count INTEGER
)''')
if not DOCKERIZED:
session.execute('ALTER TABLE public.channel OWNER TO wrolpi')
session.execute('ALTER TABLE public.video OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS channel CASCADE')
session.execute('DROP TABLE IF EXISTS video CASCADE')

View file

@ -1,41 +0,0 @@
"""Video source_id index.
Revision ID: f2c6326cad29
Revises: f8c65e7e6802
Create Date: 2022-01-10 08:27:39.082067
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'f2c6326cad29'
down_revision = 'f8c65e7e6802'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('CREATE INDEX video_source_id_idx ON video(source_id)')
session.execute('ALTER INDEX video_censored RENAME TO video_censored_idx')
session.execute('ALTER INDEX video_favorite RENAME TO video_favorite_idx')
session.execute('ALTER INDEX video_view_count RENAME TO video_view_count_idx')
session.execute('ALTER INDEX video_validated_poster RENAME TO video_validated_poster_idx')
session.execute('ALTER INDEX video_upload_date RENAME TO video_upload_date_idx')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS video_source_id_idx')
session.execute('ALTER INDEX video_censored_idx RENAME TO video_censored')
session.execute('ALTER INDEX video_favorite_idx RENAME TO video_favorite')
session.execute('ALTER INDEX video_view_count_idx RENAME TO video_view_count')
session.execute('ALTER INDEX video_validated_poster_idx RENAME TO video_validated_poster')
session.execute('ALTER INDEX video_upload_date_idx RENAME TO video_upload_date')

View file

@ -1,68 +0,0 @@
"""collections feature
Revision ID: f38cfe4b5cb2
Revises: add_unique_collection_name_kind
Create Date: 2025-11-29 16:51:34.222412
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = 'f38cfe4b5cb2'
down_revision = 'add_unique_collection_name_kind'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.alter_column('archive', 'collection_id',
existing_type=sa.INTEGER(),
nullable=True)
op.alter_column('file_group', 'mimetype',
existing_type=sa.TEXT(),
nullable=True)
op.alter_column('tag_file', 'tag_id',
existing_type=sa.INTEGER(),
nullable=False)
op.alter_column('tag_file', 'file_group_id',
existing_type=sa.BIGINT(),
nullable=False)
op.alter_column('tag_zim', 'tag_id',
existing_type=sa.INTEGER(),
nullable=False)
op.alter_column('tag_zim', 'zim_id',
existing_type=sa.INTEGER(),
nullable=False)
op.drop_index('video_upload_date_idx', table_name='video')
op.drop_column('video', 'duration')
op.drop_column('video', 'upload_date')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('video', sa.Column('upload_date', postgresql.TIMESTAMP(timezone=True), autoincrement=False, nullable=True))
op.add_column('video', sa.Column('duration', sa.INTEGER(), autoincrement=False, nullable=True))
op.create_index('video_upload_date_idx', 'video', ['upload_date'], unique=False)
op.alter_column('tag_zim', 'zim_id',
existing_type=sa.INTEGER(),
nullable=True)
op.alter_column('tag_zim', 'tag_id',
existing_type=sa.INTEGER(),
nullable=True)
op.alter_column('tag_file', 'file_group_id',
existing_type=sa.BIGINT(),
nullable=True)
op.alter_column('tag_file', 'tag_id',
existing_type=sa.INTEGER(),
nullable=True)
op.alter_column('archive', 'collection_id',
existing_type=sa.INTEGER(),
nullable=False)
op.alter_column('file_group', 'mimetype',
existing_type=sa.TEXT(),
nullable=False)
# ### end Alembic commands ###

View file

@ -1,31 +0,0 @@
"""Channel.download_missing_data
Revision ID: f7229a67e333
Revises: dc6637ced17b
Create Date: 2025-02-19 14:02:57.952791
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'f7229a67e333'
down_revision = 'dc6637ced17b'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS download_missing_data BOOLEAN DEFAULT True')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS download_missing_data')

View file

@ -1,31 +0,0 @@
"""Channel.refresh
Revision ID: f8c65e7e6802
Revises: 62acddc55091
Create Date: 2022-01-07 14:28:21.787842
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'f8c65e7e6802'
down_revision = '62acddc55091'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel ADD COLUMN refreshed BOOL DEFAULT false')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS refreshed')

View file

@ -1,45 +0,0 @@
"""File Directories
Revision ID: f9928a3b6bc5
Revises: c9561bda3cf0
Create Date: 2023-04-18 16:58:40.043115
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'f9928a3b6bc5'
down_revision = 'c9561bda3cf0'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE directory (
path TEXT PRIMARY KEY,
name TEXT NOT NULL,
idempotency TIMESTAMP WITH TIME ZONE DEFAULT current_timestamp
)
''')
session.execute('CREATE INDEX directory_name_idx ON directory(name)')
if not DOCKERIZED:
session.execute('ALTER TABLE public.directory OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP INDEX IF EXISTS directory_name_idx')
session.execute('DROP TABLE IF EXISTS directory')

View file

@ -1,576 +0,0 @@
"""Migrate Domain model to Collection model
This migration:
1. Adds collection_id column to archive table
2. Runs the data migration to populate collections
3. Makes collection_id NOT NULL
4. Drops the old domain_id column and domains table
Revision ID: migrate_domains_to_collections
Revises: 66407d145b76
Create Date: 2025-10-29
"""
import os
import pathlib
from typing import Dict, List, Optional
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text, Column, Integer, String, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'migrate_domains_to_collections'
down_revision = '66407d145b76'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
# Local model definitions for migration stability.
# These are minimal copies of the Collection and CollectionItem models
# that contain only the columns needed for this migration.
# This ensures the migration will work regardless of future changes to the actual models.
MBase = declarative_base()
class MCollection(MBase):
"""Migration-stable Collection model with minimum required columns."""
__tablename__ = 'collection'
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
kind = Column(String, nullable=False, server_default='channel', default='channel')
directory = Column(String, nullable=True)
class MCollectionItem(MBase):
"""Migration-stable CollectionItem model with minimum required columns.
Note: ForeignKey constraints are NOT declared here because they reference
tables (file_group) that are not in this migration's declarative base.
The actual FK constraints exist in the database schema.
"""
__tablename__ = 'collection_item'
id = Column(Integer, primary_key=True)
collection_id = Column(Integer, nullable=False)
file_group_id = Column(Integer, nullable=False)
position = Column(Integer, nullable=False, default=0)
class DomainMigrationStats:
"""Track migration statistics."""
def __init__(self):
self.domains_found = 0
self.domains_migrated = 0
self.domains_skipped = 0
self.collections_created = 0
self.collections_with_directory = 0
self.collection_items_created = 0
self.archives_linked = 0
self.errors: List[str] = []
self.warnings: List[str] = []
def add_error(self, msg: str):
self.errors.append(msg)
print(f" ✗ ERROR: {msg}")
def add_warning(self, msg: str):
self.warnings.append(msg)
print(f" ⚠ WARNING: {msg}")
def print_summary(self, dry_run: bool = False):
"""Print migration summary."""
mode = "DRY-RUN" if dry_run else "ACTUAL"
print(f"\n{'='*60}")
print(f"Domain → Collection Migration Summary ({mode})")
print(f"{'='*60}")
print(f"Domains found: {self.domains_found}")
print(f"Domains migrated: {self.domains_migrated}")
print(f"Domains skipped (no archives): {self.domains_skipped}")
print(f"Collections created: {self.collections_created}")
print(f"Collections with directory: {self.collections_with_directory}")
print(f"CollectionItems created: {self.collection_items_created}")
print(f"Archives linked: {self.archives_linked}")
if self.warnings:
print(f"\nWarnings: {len(self.warnings)}")
for warning in self.warnings:
print(f" ⚠ {warning}")
if self.errors:
print(f"\nErrors: {len(self.errors)}")
for error in self.errors:
print(f" ✗ {error}")
if not self.errors:
print(f"\n✓ Migration completed successfully")
else:
print(f"\n✗ Migration completed with {len(self.errors)} errors")
print(f"{'='*60}\n")
def get_domain_data(session: Session) -> List[Dict]:
"""Fetch all Domain records from database."""
result = session.execute(text("SELECT id, domain, directory FROM domains ORDER BY domain"))
domains = []
for row in result:
domains.append({
'id': row[0],
'domain': row[1],
'directory': row[2],
})
return domains
def get_archives_for_domain(session: Session, domain_id: int) -> List[Dict]:
"""Get all Archive IDs for a given domain."""
result = session.execute(
text("SELECT id, file_group_id FROM archive WHERE domain_id = :domain_id"),
{'domain_id': domain_id}
)
archives = []
for row in result:
if row[1]: # file_group_id must not be null
archives.append({
'archive_id': row[0],
'file_group_id': row[1],
})
return archives
def validate_domain_name(domain: str) -> bool:
"""Validate domain name format - must contain at least one dot."""
if not domain or not isinstance(domain, str):
return False
if domain.startswith('.') or domain.endswith('.'):
return False
return '.' in domain
def detect_common_directory_from_archives(session: Session, domain_id: int) -> Optional[str]:
"""
Detect the common directory for all archives belonging to a domain.
Returns the directory path as a string, or None if no common directory exists.
"""
result = session.execute(
text("""
SELECT fg.primary_path
FROM archive a
JOIN file_group fg ON a.file_group_id = fg.id
WHERE a.domain_id = :domain_id AND fg.primary_path IS NOT NULL
"""),
{'domain_id': domain_id}
)
paths = [pathlib.Path(row[0]) for row in result if row[0]]
if not paths:
return None
# Find common ancestor directory
common_dir = paths[0].parent
for path in paths[1:]:
while common_dir != common_dir.parent: # Stop at root
try:
path.relative_to(common_dir)
break
except ValueError:
common_dir = common_dir.parent
if common_dir == common_dir.parent:
return None # Reached root, no common directory
return str(common_dir)
def check_directory_conflict(session: Session, directory: str) -> bool:
"""Check if a directory is already used by an existing Collection."""
result = session.execute(
text("SELECT id FROM collection WHERE directory = :directory"),
{'directory': directory}
)
return result.first() is not None
def perform_domain_migration(session: Session, verbose: bool = True) -> DomainMigrationStats:
"""
Migrate all Domain records to Collection records.
This function is integrated into the alembic migration.
Uses local MCollection model for migration stability, and raw SQL for
CollectionItem operations to avoid FK resolution issues with file_group.
"""
stats = DomainMigrationStats()
# Step 1: Fetch all domains
print("Step 1: Fetching Domain records...")
domains = get_domain_data(session)
stats.domains_found = len(domains)
print(f"Found {len(domains)} Domain records")
if not domains:
print("No domains to migrate")
return stats
# Step 2: Create Collections from Domains
print("\nStep 2: Creating Collection records...")
domain_to_collection: Dict[int, int] = {} # domain_id -> collection_id
for domain in domains:
domain_id = domain['id']
domain_name = domain['domain']
if verbose:
print(f" Processing Domain {domain_id}: {domain_name}")
# Validate domain name
if not validate_domain_name(domain_name):
stats.add_error(f"Invalid domain name: {repr(domain_name)} (skipping)")
continue
# Skip domains with no archives - don't create empty collections
archives = get_archives_for_domain(session, domain_id)
if not archives:
if verbose:
print(f" Skipping domain {repr(domain_name)} - no archives")
stats.domains_skipped += 1
continue
# Check if Collection already exists
existing = session.query(MCollection).filter_by(
name=domain_name,
kind='domain'
).first()
if existing:
stats.add_warning(f"Collection already exists for domain {repr(domain_name)} (id={existing.id})")
domain_to_collection[domain_id] = existing.id
stats.domains_migrated += 1
continue
# Determine directory for the collection
directory = None
# Step 1: Check if Domain.directory is set
domain_directory = domain.get('directory')
if domain_directory:
directory = domain_directory
if verbose:
print(f" Using Domain.directory: {directory}")
else:
# Step 2: Detect common directory from archive file paths
detected = detect_common_directory_from_archives(session, domain_id)
if detected:
directory = detected
if verbose:
print(f" Detected directory from archives: {directory}")
# Step 3: Check for directory conflicts
if directory and check_directory_conflict(session, directory):
stats.add_warning(
f"Directory {repr(directory)} already used by another Collection "
f"(domain {repr(domain_name)} will have no directory)"
)
directory = None
# Create Collection
collection = MCollection(
name=domain_name,
kind='domain',
directory=directory,
)
session.add(collection)
session.flush([collection])
domain_to_collection[domain_id] = collection.id
stats.collections_created += 1
stats.domains_migrated += 1
if directory:
stats.collections_with_directory += 1
if verbose:
if directory:
print(f" Created Collection id={collection.id} with directory={directory}")
else:
print(f" Created Collection id={collection.id} (no directory)")
print(f"Created {stats.collections_created} Collection records")
# Step 3: Update archive.collection_id for all Archives
print("\nStep 3: Updating archive.collection_id...")
for domain in domains:
domain_id = domain['id']
domain_name = domain['domain']
if domain_id not in domain_to_collection:
# Domain was skipped due to error
continue
collection_id = domain_to_collection[domain_id]
# Update all archives for this domain
result = session.execute(
text("UPDATE archive SET collection_id = :collection_id WHERE domain_id = :domain_id"),
{'collection_id': collection_id, 'domain_id': domain_id}
)
updated_count = result.rowcount
if verbose and updated_count > 0:
print(f" Domain {repr(domain_name)}: Updated {updated_count} archives")
stats.archives_linked += updated_count
print(f"Updated {stats.archives_linked} Archives with collection_id")
# Step 3b: Handle orphan archives (those with NULL domain_id) by creating collections from their URLs
print("\nStep 3b: Handling orphan archives (NULL domain_id)...")
result = session.execute(text("""
SELECT a.id, a.file_group_id, fg.url
FROM archive a
LEFT JOIN file_group fg ON a.file_group_id = fg.id
WHERE a.domain_id IS NULL
"""))
orphans = list(result)
if orphans:
print(f" Found {len(orphans)} orphan archives:")
# Group orphans by domain extracted from URL
from urllib.parse import urlparse
domain_orphans = {}
for archive_id, file_group_id, url in orphans:
print(f" Archive {archive_id}: {url}")
try:
parsed = urlparse(url)
domain = parsed.netloc
if domain.startswith('www.'):
domain = domain[4:]
except Exception:
domain = 'unknown.domain'
if domain not in domain_orphans:
domain_orphans[domain] = []
domain_orphans[domain].append((archive_id, file_group_id))
# Create collections for each domain and assign orphans
for domain_name, archives in domain_orphans.items():
print(f" Creating collection for domain: {domain_name} ({len(archives)} archives)")
# Check if collection already exists
existing = session.execute(
text("SELECT id FROM collection WHERE name = :name AND kind = 'domain'"),
{'name': domain_name}
).first()
if existing:
collection_id = existing[0]
print(f" Using existing collection id={collection_id}")
else:
# Create new collection
session.execute(
text("INSERT INTO collection (name, kind) VALUES (:name, 'domain')"),
{'name': domain_name}
)
collection_id = session.execute(text("SELECT lastval()")).scalar()
stats.collections_created += 1
print(f" Created collection id={collection_id}")
# Update archives with collection_id and create CollectionItems
for archive_id, file_group_id in archives:
session.execute(
text("UPDATE archive SET collection_id = :collection_id WHERE id = :archive_id"),
{'collection_id': collection_id, 'archive_id': archive_id}
)
stats.archives_linked += 1
# Create CollectionItem if not exists
existing_item = session.execute(
text("SELECT id FROM collection_item WHERE collection_id = :cid AND file_group_id = :fgid"),
{'cid': collection_id, 'fgid': file_group_id}
).first()
if not existing_item:
session.execute(
text("INSERT INTO collection_item (collection_id, file_group_id, position) VALUES (:cid, :fgid, 0)"),
{'cid': collection_id, 'fgid': file_group_id}
)
stats.collection_items_created += 1
print(f" Processed {len(orphans)} orphan archives into {len(domain_orphans)} collections")
else:
print(" No orphan archives found")
# Step 4: Create CollectionItem records
print("\nStep 4: Creating CollectionItem records...")
for domain in domains:
domain_id = domain['id']
domain_name = domain['domain']
if domain_id not in domain_to_collection:
continue
collection_id = domain_to_collection[domain_id]
# Get all archives for this domain
archives = get_archives_for_domain(session, domain_id)
if not archives:
if verbose:
print(f" Domain {repr(domain_name)}: No archives found")
continue
if verbose:
print(f" Domain {repr(domain_name)}: {len(archives)} archives")
for idx, archive in enumerate(archives, start=1):
file_group_id = archive['file_group_id']
archive_id = archive['archive_id']
# Check if CollectionItem already exists (use raw SQL to avoid FK resolution issues)
result = session.execute(
text("SELECT id FROM collection_item WHERE collection_id = :collection_id AND file_group_id = :file_group_id"),
{'collection_id': collection_id, 'file_group_id': file_group_id}
)
existing_item = result.first()
if existing_item:
if verbose:
print(f" Archive {archive_id} already linked to collection")
continue
# Create CollectionItem (use raw SQL to avoid FK resolution issues)
session.execute(
text("INSERT INTO collection_item (collection_id, file_group_id, position) VALUES (:collection_id, :file_group_id, :position)"),
{'collection_id': collection_id, 'file_group_id': file_group_id, 'position': idx}
)
stats.collection_items_created += 1
print(f"Created {stats.collection_items_created} CollectionItem records")
# Step 5: Skip config export during migration
# The config export requires a separate database session, but we're inside
# an alembic transaction. The application will export on startup instead.
print("\nStep 5: Skipping domains.yaml export during migration...")
print(" (Config will be exported on next application startup)")
return stats
def upgrade():
"""
Migrate Domains to Collections.
This is a one-way migration - there is no downgrade path because:
- Domain data is transformed into Collection data
- Once migrated, the system uses Collections exclusively
- Downgrade would require reimplementing Domain model
"""
bind = op.get_bind()
session = Session(bind=bind)
print("\n" + "="*60)
print("DOMAIN → COLLECTION MIGRATION")
print("="*60 + "\n")
# Step 1: Add collection_id column to archive table (nullable initially)
print("Step 1: Adding collection_id column to archive table...")
op.add_column('archive',
sa.Column('collection_id', sa.Integer(), nullable=True))
# Add foreign key to collection table
op.create_foreign_key(
'archive_collection_id_fkey',
'archive', 'collection',
['collection_id'], ['id'],
ondelete='CASCADE'
)
print("✓ Added collection_id column\n")
# Step 2: Run data migration script
print("Step 2: Running data migration script...")
print("This will:")
print(" - Create Collection records from Domain records")
print(" - Create CollectionItem records for Archives\n")
try:
# Commit the schema changes before running migration
session.commit()
# Run the data migration
stats = perform_domain_migration(session, verbose=True)
# Print summary
stats.print_summary(dry_run=False)
if stats.errors:
raise Exception(f"Migration completed with {len(stats.errors)} errors. See logs above.")
except Exception as e:
print(f"\n✗ Migration failed: {e}")
raise
# Step 3: Make collection_id NOT NULL
print("\nStep 3: Making collection_id NOT NULL...")
# Check if any archives don't have a collection_id
result = session.execute(sa.text(
"SELECT COUNT(*) FROM archive WHERE collection_id IS NULL"
))
null_count = result.scalar()
if null_count > 0:
raise Exception(
f"Cannot make collection_id NOT NULL: {null_count} archives have NULL collection_id. "
f"The migration script should have assigned all archives to collections."
)
op.alter_column('archive', 'collection_id', nullable=False)
print("✓ collection_id is now NOT NULL\n")
# Step 4: Drop old domain_id column and foreign key
print("Step 4: Dropping domain_id column and foreign key...")
op.drop_constraint('archive_domain_id_fkey', 'archive', type_='foreignkey')
op.drop_column('archive', 'domain_id')
print("✓ Dropped domain_id column\n")
# Step 5: Drop domains table
print("Step 5: Dropping domains table...")
op.drop_table('domains')
print("✓ Dropped domains table\n")
# Ensure table ownership in non-docker environments
if not DOCKERIZED:
print("Setting table ownership...")
session.execute(sa.text('ALTER TABLE public.archive OWNER TO wrolpi'))
session.commit()
print("="*60)
print("MIGRATION COMPLETED SUCCESSFULLY")
print("="*60 + "\n")
def downgrade():
"""
No downgrade path for this migration.
This is intentional because:
1. Domain → Collection is a one-way data transformation
2. The Domain model is being removed from the codebase
3. Rolling back would require:
- Recreating Domain model code
- Reversing the data transformation (Collection → Domain)
- Handling edge cases where Collections have features Domains didn't
If you need to rollback, you should:
1. Restore from a database backup taken before migration
2. Re-index all files to rebuild Domain records
"""
raise NotImplementedError(
"This migration has no downgrade path. "
"To rollback, reset database and re-index files."
)

View file

@ -1,163 +0,0 @@
"""Migrate Download.channel_id to Download.collection_id
This migration:
1. Adds collection_id column to download table
2. Migrates existing channel_id references to collection_id via Channel.collection_id
3. Drops channel_id column and foreign key
Revision ID: migrate_download_to_collection
Revises: b43f70f369d0
Create Date: 2025-11-27
"""
import os
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'migrate_download_to_collection'
down_revision = 'b43f70f369d0'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
print("\n" + "=" * 60)
print("Download channel_id -> collection_id Migration")
print("=" * 60 + "\n")
# Step 1: Add collection_id column (nullable initially)
print("Step 1: Adding collection_id column to download table...")
op.add_column('download', sa.Column('collection_id', sa.Integer(), nullable=True))
print("Done\n")
# Step 2: Migrate data - lookup collection_id through channel
print("Step 2: Migrating channel_id to collection_id...")
result = session.execute(text("""
UPDATE download
SET collection_id = channel.collection_id
FROM channel
WHERE download.channel_id = channel.id
AND download.channel_id IS NOT NULL
"""))
print(f"Updated {result.rowcount} download records\n")
# Step 3: Add foreign key constraint
print("Step 3: Adding foreign key constraint...")
op.create_foreign_key(
'fk_download_collection_id',
'download',
'collection',
['collection_id'],
['id'],
ondelete='SET NULL'
)
print("Done\n")
# Step 4: Create index for performance
print("Step 4: Creating index on collection_id...")
op.create_index('idx_download_collection_id', 'download', ['collection_id'])
print("Done\n")
# Step 5: Drop old channel_id foreign key and column
print("Step 5: Dropping channel_id column...")
op.drop_constraint('download_channel_id_fkey', 'download', type_='foreignkey')
op.drop_column('download', 'channel_id')
print("Done\n")
# Step 6: Fix the update_channel_minimum_frequency trigger function
# It was referencing channel_id which no longer exists
print("Step 6: Updating update_channel_minimum_frequency trigger function...")
op.execute("""
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
RETURNS TRIGGER AS
$$
DECLARE
v_channel_id INTEGER;
BEGIN
-- Handle INSERT and UPDATE: use NEW record
IF (TG_OP = 'DELETE') THEN
-- For DELETE, we need to find the channel from OLD record
IF OLD.collection_id IS NOT NULL THEN
SELECT id INTO v_channel_id FROM channel WHERE collection_id = OLD.collection_id;
IF v_channel_id IS NOT NULL THEN
UPDATE channel
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE collection_id = OLD.collection_id)
WHERE id = v_channel_id;
END IF;
END IF;
RETURN OLD;
ELSE
-- For INSERT and UPDATE, use NEW record
IF NEW.collection_id IS NOT NULL THEN
SELECT id INTO v_channel_id FROM channel WHERE collection_id = NEW.collection_id;
IF v_channel_id IS NOT NULL THEN
UPDATE channel
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE collection_id = NEW.collection_id)
WHERE id = v_channel_id;
END IF;
END IF;
RETURN NEW;
END IF;
END;
$$ LANGUAGE plpgsql;
""")
print("Done\n")
print("=" * 60)
print("Migration Complete")
print("=" * 60 + "\n")
if not DOCKERIZED:
session.execute(text('ALTER TABLE public.download OWNER TO wrolpi'))
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Restore the old trigger function that uses channel_id
op.execute("""
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
RETURNS TRIGGER AS
$$
BEGIN
UPDATE channel
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE channel_id = NEW.channel_id)
WHERE id = NEW.channel_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
""")
# Add back channel_id column
op.add_column('download', sa.Column('channel_id', sa.Integer(), nullable=True))
# Migrate collection_id back to channel_id
session.execute(text("""
UPDATE download
SET channel_id = channel.id
FROM channel
WHERE download.collection_id = channel.collection_id
AND download.collection_id IS NOT NULL
"""))
# Add foreign key constraint
op.create_foreign_key(
'download_channel_id_fkey',
'download',
'channel',
['channel_id'],
['id']
)
# Drop collection_id
op.drop_index('idx_download_collection_id', 'download')
op.drop_constraint('fk_download_collection_id', 'download', type_='foreignkey')
op.drop_column('download', 'collection_id')

View file

@ -77,14 +77,6 @@ DEFAULT_CONFIG = {
"description": "Kiwix/Zim server",
},
# System services
{
"name": "postgresql",
"systemd_name": "postgresql",
"port": 5432,
"viewable": False,
"use_https": False,
"description": "PostgreSQL database",
},
{
"name": "caddy",
"systemd_name": "caddy",

View file

@ -57,7 +57,7 @@ class TestManagedServicesConfig:
def test_managed_services_count(self):
"""Should have expected number of managed services."""
services = DEFAULT_CONFIG["managed_services"]
assert len(services) == 12
assert len(services) == 11
@pytest.mark.parametrize("name,expected", [
("wrolpi-api",
@ -67,7 +67,6 @@ class TestManagedServicesConfig:
("wrolpi-app", {"port": 3000, "viewable": False}),
("wrolpi-app-dev", {"port": 3000, "viewable": False, "show_only_when_running": True}),
("caddy", {"port": 443}),
("postgresql", {"port": 5432, "viewable": False}),
("wrolpi-upgrade", {"port": None, "viewable": False, "show_only_when_running": True}),
])
def test_service_config(self, name, expected):
@ -83,7 +82,6 @@ class TestManagedServicesConfig:
("wrolpi-kiwix", True),
("wrolpi-api", False),
("wrolpi-app", False),
("postgresql", False),
])
def test_service_use_https(self, name, use_https):
"""HTTPS services should have use_https=True, others should have False."""

View file

@ -1,21 +1,7 @@
# Development containers, these should only be used if you are developing.
# The database is SQLite inside the media directory (./test/config/wrolpi.db); no db service.
version: '3'
services:
db: # https://github.com/docker-library/docs/tree/master/postgres
image: postgres:12
environment:
- POSTGRES_DB=wrolpi
- POSTGRES_PASSWORD=wrolpi
ports:
- "5432:5432"
healthcheck:
test: [ 'CMD-SHELL', 'pg_isready -U postgres' ]
interval: 10s
start_period: 30s
start_interval: 1s
volumes:
- ./pg_data:/var/lib/postgresql/data
controller: # WROLPi Controller Service
build:
context: .
@ -47,8 +33,6 @@ services:
api:
depends_on:
db:
condition: service_healthy
controller:
condition: service_started
build:

View file

@ -11,7 +11,6 @@ git submodule update --init
docker-compose build --parallel
docker volume create --name=openstreetmap-data
docker volume create --name=openstreetmap-rendered-tiles
docker-compose up -d db
docker-compose run --rm api db upgrade
docker-compose up
```
@ -26,7 +25,6 @@ Browse to https://localhost:8443
| **api** | 8081 | Sanic API server |
| **app** | (via web) | React frontend |
| **controller** | 8080 | System management (FastAPI) |
| **db** | 5432 | PostgreSQL 12 |
| **archive** | 8083 | SingleFile/Readability |
| **zim** | (via web:8085) | Kiwix |
| **help** | (via web:8086) | MkDocs documentation |

View file

@ -1,6 +1,5 @@
[Unit]
Description=WROLPi Python API (Development)
Wants=postgresql.service
PartOf=wrolpi.target
Conflicts=wrolpi-api.service

View file

@ -1,6 +1,5 @@
[Unit]
Description=WROLPi Python API
Wants=postgresql.service
PartOf=wrolpi.target
Conflicts=wrolpi-api-dev.service

View file

@ -2,7 +2,7 @@
Description=WROLPi Controller Service
Documentation=https://github.com/lrnselfreliance/wrolpi
After=network.target
Before=wrolpi-api.service wrolpi-app.service postgresql.service
Before=wrolpi-api.service wrolpi-app.service
[Service]
Type=simple

View file

@ -1,3 +1,3 @@
[Unit]
Description=WROLPi services group
Wants=wrolpi-controller.service wrolpi-api.service wrolpi-app.service caddy.service postgresql.service wrolpi-kiwix.service
Wants=wrolpi-controller.service wrolpi-api.service wrolpi-app.service caddy.service wrolpi-kiwix.service

Some files were not shown because too many files have changed in this diff Show more