mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-11-21 07:26:18 -05:00
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files, configs, AND database) travels with the drive between WROLPi systems. Core: - wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s, foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3 adapters keep raw-SQL datetime formats identical to the ORM's. - wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN. Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines. Stemming contract pinned by dedicated tests (watch/watches/watched/ watching are interchangeable). - wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic baseline and test DBs; fixes the channel video_count DELETE/UPDATE bugs and a latent total_size bug (it used the video id as channel id). - wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor; network-filesystem refusal; never creates/migrates/trusts a database beneath an unmounted production media directory (shadow-DB guard). - Alembic: 85 Postgres-era migrations squashed into one baseline; revision ids are dates (2026_07_10_1350) so filenames self-sort and alembic_version says when a schema was created; render_as_batch=True. - All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> -> HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM, generate_series histograms->strftime GROUP BY, DISTINCT ON-> row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted to SQLAlchemy instead (flag upsert, download retry/summary/tag checks, channel-by-tag estimate, Directory delete). - Config imports count skipped entries; background dumps are blocked after a lossy import so partial imports can never overwrite YAML configs (they are the sole source of truth for rebuilds now). - Search tolerates explicit JSON nulls for limit/offset (Postgres treated OFFSET NULL as 0; SQLite errors). - file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite (BIGINT is not a rowid alias; FTS5 content_rowid requires it). Ops teardown: docker-compose db service, CI postgres sidecars, initialize_api_db.sh, controller postgresql service, psycopg2-binary all removed. upgrade.sh stops/disables postgresql (data is NOT deleted; `systemctl enable --now postgresql` restores it); systemd units no longer Want postgresql.service. Existing-user upgrade: first boot creates+migrates the DB, configs import (tags recreate their FileGroups from disk), a refresh rebuilds the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the Postgres baseline exactly; survived an uncontrolled hard reboot with clean integrity checks. Accepted behavior changes: FTS5 has no stopwords; bm25 length normalization shifts some rank orders; fast-path d_headline is un-highlighted; all-negative queries match nothing. Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked rebuild refresh with progress UX. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
7da89d883e
commit
d9303a0ea8
150 changed files with 2749 additions and 6833 deletions
|
|
@ -8,11 +8,6 @@ jobs:
|
|||
api-tests-3-11:
|
||||
docker:
|
||||
- image: cimg/python:3.11
|
||||
- image: cimg/postgres:15.6
|
||||
environment:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_DB: wrolpi
|
||||
POSTGRES_PASSWORD: "wrolpi"
|
||||
# `large` (4 vCPU, 8 GB) so pytest-xdist's `-nauto` has room to scale.
|
||||
resource_class: large
|
||||
steps:
|
||||
|
|
@ -41,11 +36,6 @@ jobs:
|
|||
api-alembic-migrations:
|
||||
docker:
|
||||
- image: cimg/python:3.11
|
||||
- image: cimg/postgres:15.6
|
||||
environment:
|
||||
POSTGRES_USER: wrolpi
|
||||
POSTGRES_DB: wrolpi
|
||||
POSTGRES_PASSWORD: "wrolpi"
|
||||
resource_class: medium
|
||||
steps:
|
||||
- checkout
|
||||
|
|
@ -57,24 +47,18 @@ jobs:
|
|||
python3 -m venv venv
|
||||
. venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
- save_cache:
|
||||
key: deps-3.11-{{ checksum "requirements.txt" }}-5
|
||||
paths:
|
||||
- "venv"
|
||||
- run:
|
||||
name: Create Media Directory
|
||||
command: mkdir -p /tmp/wrolpi-media/config
|
||||
- run:
|
||||
name: Run Alembic Migrations
|
||||
command: ./venv/bin/python ./main.py db upgrade
|
||||
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db upgrade
|
||||
- run:
|
||||
name: Check for Missing Migrations
|
||||
command: ./venv/bin/python ./main.py db check
|
||||
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db check
|
||||
api-tests-3-12:
|
||||
docker:
|
||||
- image: cimg/python:3.12
|
||||
- image: cimg/postgres:15.6
|
||||
environment:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_DB: wrolpi
|
||||
POSTGRES_PASSWORD: "wrolpi"
|
||||
resource_class: large
|
||||
steps:
|
||||
- checkout
|
||||
|
|
|
|||
|
|
@ -11,7 +11,6 @@ tasks:
|
|||
docker-compose build --parallel
|
||||
docker volume create --name=openstreetmap-data
|
||||
docker volume create --name=openstreetmap-rendered-tiles
|
||||
docker-compose up -d db
|
||||
docker-compose run --rm api db upgrade
|
||||
# command runs each time a user starts their workspace
|
||||
command: docker-compose up
|
||||
|
|
|
|||
|
|
@ -79,13 +79,11 @@ You can try out WROLPi by running the docker containers.
|
|||
6. Create volumes.
|
||||
* `docker volume create --name=openstreetmap-data`
|
||||
* `docker volume create --name=openstreetmap-rendered-tiles`
|
||||
7. Start the database.
|
||||
* `docker-compose up -d db`
|
||||
8. Initialize the database.
|
||||
7. Initialize the database (SQLite in the media directory).
|
||||
* `docker-compose run --rm api db upgrade`
|
||||
9. Start the docker containers.
|
||||
8. Start the docker containers.
|
||||
* `docker-compose up`
|
||||
10. Browse to WROLPi.
|
||||
9. Browse to WROLPi.
|
||||
* https://0.0.0.0:8443
|
||||
|
||||
More Docker documentation is available in [docker/README.md](docker/README.md)
|
||||
|
|
|
|||
|
|
@ -35,11 +35,17 @@ prepend_sys_path = .
|
|||
# directories, initial revisions must be specified with --version-path
|
||||
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
|
||||
|
||||
# Revision ids ARE dates (see process_revision_directives in env.py), so filenames self-sort
|
||||
# chronologically and the alembic_version table says when a database's schema was created.
|
||||
file_template = %%(rev)s_%%(slug)s
|
||||
|
||||
# the output encoding used when revision files
|
||||
# are written from script.py.mako
|
||||
# output_encoding = utf-8
|
||||
|
||||
sqlalchemy.url = postgresql://postgres:wrolpi@localhost:5432/wrolpi
|
||||
# Placeholder: env.py derives the real URL from the media directory (see wrolpi.db.get_db_uri);
|
||||
# `main.py db ...` overrides it explicitly.
|
||||
sqlalchemy.url = sqlite:///CHANGEME
|
||||
|
||||
|
||||
[post_write_hooks]
|
||||
|
|
|
|||
|
|
@ -1,9 +1,6 @@
|
|||
import os
|
||||
from logging.config import fileConfig
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config
|
||||
from sqlalchemy import pool
|
||||
|
||||
# this is the Alembic Config object, which provides
|
||||
# access to the values within the .ini file in use.
|
||||
|
|
@ -26,13 +23,40 @@ from wrolpi.collections.models import Collection, CollectionItem # noqa: F401
|
|||
from modules.videos.models import Video, Channel # noqa: F401
|
||||
from modules.archive.models import Archive # noqa: F401
|
||||
from modules.zim.models import Zim, TagZimEntry, ZimSubscription # noqa: F401
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
# ... etc.
|
||||
def get_url() -> str:
|
||||
"""The database URL: from the ini/CLI when overridden, otherwise derived from the media directory."""
|
||||
url = config.get_main_option('sqlalchemy.url')
|
||||
if url and not url.startswith('sqlite:///CHANGEME'):
|
||||
return url
|
||||
from wrolpi.db import get_db_uri
|
||||
return get_db_uri()
|
||||
|
||||
|
||||
def include_object(object_, name, type_, reflected, compare_to):
|
||||
"""Exclude tables Alembic must not manage from autogenerate/check.
|
||||
|
||||
The FTS5 virtual tables and their shadow tables (created by wrolpi.schema_ddl /
|
||||
wrolpi.fts) plus SQLite internals would otherwise be proposed for DROP."""
|
||||
if type_ == 'table':
|
||||
from wrolpi.fts import FTS_TABLE_PREFIXES
|
||||
if name == 'sqlite_sequence' or any(name.startswith(prefix) for prefix in FTS_TABLE_PREFIXES):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def process_revision_directives(context_, revision, directives):
|
||||
"""New revisions get date ids (e.g. 2026_07_10_1350) instead of random hashes.
|
||||
|
||||
Filenames therefore self-sort chronologically, and the `alembic_version` value in a
|
||||
database says when its schema was created. Underscores, not dashes: alembic forbids
|
||||
`-` in revision identifiers (it is reserved for relative-revision syntax like `head-1`)."""
|
||||
from datetime import datetime, timezone
|
||||
script = directives[0]
|
||||
script.rev_id = datetime.now(timezone.utc).strftime('%Y_%m_%d_%H%M')
|
||||
|
||||
|
||||
def run_migrations_offline():
|
||||
|
|
@ -47,12 +71,14 @@ def run_migrations_offline():
|
|||
script output.
|
||||
|
||||
"""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(
|
||||
url=url,
|
||||
url=get_url(),
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
render_as_batch=True,
|
||||
include_object=include_object,
|
||||
process_revision_directives=process_revision_directives,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
|
|
@ -66,18 +92,18 @@ def run_migrations_online():
|
|||
and associate a connection with the context.
|
||||
|
||||
"""
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
from wrolpi.db import create_wrolpi_engine
|
||||
|
||||
connectable = create_wrolpi_engine(get_url())
|
||||
|
||||
with connectable.connect() as connection:
|
||||
if not os.environ.get('DOCKER', '').lower().startswith('t'):
|
||||
# Set role when not in a docker container
|
||||
connection.execute("SET ROLE 'wrolpi'")
|
||||
context.configure(
|
||||
connection=connection, target_metadata=target_metadata
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
# SQLite's limited ALTER TABLE requires batch mode for most column changes.
|
||||
render_as_batch=True,
|
||||
include_object=include_object,
|
||||
process_revision_directives=process_revision_directives,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
|
|
|
|||
|
|
@ -5,9 +5,8 @@ Revises: ${down_revision | comma,n}
|
|||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
|
|
@ -16,18 +15,10 @@ down_revision = ${repr(down_revision)}
|
|||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.TABLE_NAME OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
${downgrades if downgrades else "pass"}
|
||||
|
|
|
|||
|
|
@ -1,33 +0,0 @@
|
|||
"""FileGroup.censored
|
||||
|
||||
Revision ID: 00f11f309c53
|
||||
Revises: 1c041aeee0fc
|
||||
Create Date: 2024-06-17 14:02:09.881477
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '00f11f309c53'
|
||||
down_revision = '1c041aeee0fc'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN censored BOOLEAN DEFAULT FALSE')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS censored')
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
"""Video textsearch coalesce fix.
|
||||
|
||||
Revision ID: 0400005fe2dc
|
||||
Revises: 53f6780aa599
|
||||
Create Date: 2022-01-04 11:00:21.776124
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '0400005fe2dc'
|
||||
down_revision = '53f6780aa599'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Replace the textsearch and index. The old textsearch failed if one of the columns was null.
|
||||
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS textsearch')
|
||||
session.execute('''
|
||||
ALTER TABLE video
|
||||
ADD COLUMN textsearch tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(title, '')), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(caption, '')), 'D')
|
||||
) STORED
|
||||
''')
|
||||
session.execute('CREATE INDEX video_textsearch_idx ON video USING GIN(textsearch)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
# The upgrade is backwards compatible.
|
||||
pass
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
"""Init inventory tables.
|
||||
|
||||
Revision ID: 053f17f07c4e
|
||||
Revises: f0d0086a73d1
|
||||
Create Date: 2021-09-20 16:21:00.322598
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
revision = '053f17f07c4e'
|
||||
down_revision = 'f0d0086a73d1'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE inventories_version (
|
||||
version SERIAL PRIMARY KEY NOT NULL
|
||||
)
|
||||
''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE public.inventory (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
viewed_at TIMESTAMP WITHOUT TIME ZONE,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
deleted_at TIMESTAMP WITHOUT TIME ZONE
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE public.item (
|
||||
id SERIAL PRIMARY KEY,
|
||||
inventory_id INTEGER REFERENCES inventory(id) ON DELETE CASCADE,
|
||||
brand TEXT,
|
||||
name TEXT,
|
||||
count DECIMAL,
|
||||
item_size DECIMAL,
|
||||
unit TEXT,
|
||||
serving INTEGER,
|
||||
category TEXT,
|
||||
subcategory TEXT,
|
||||
expiration_date DATE,
|
||||
purchase_date DATE,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
deleted_at TIMESTAMP WITHOUT TIME ZONE
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.inventory OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.item OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS public.inventories_version''')
|
||||
session.execute('DROP TABLE IF EXISTS public.item''')
|
||||
session.execute('DROP TABLE IF EXISTS public.inventory''')
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Archive url index.
|
||||
|
||||
Revision ID: 13542a230c78
|
||||
Revises: c72d305cf11e
|
||||
Create Date: 2022-11-02 15:19:42.408462
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '13542a230c78'
|
||||
down_revision = 'c72d305cf11e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('CREATE INDEX IF NOT EXISTS archive_url_idx ON archive(url)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS archive_url_idx')
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
"""Lower FileGroup.primary_path
|
||||
|
||||
Revision ID: 158973fda20f
|
||||
Revises: 69e9754d1a63
|
||||
Create Date: 2025-08-16 10:54:58.874094
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '158973fda20f'
|
||||
down_revision = '69e9754d1a63'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('CREATE INDEX file_group_primary_path_lower_idx ON file_group (LOWER(primary_path))')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('DROP INDEX IF EXISTS file_group_primary_path_lower_idx')
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""Create Download.destination column.
|
||||
|
||||
Revision ID: 170d1be52bc7
|
||||
Revises: 9ec4c765ef8d
|
||||
Create Date: 2024-11-02 11:16:15.297850
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '170d1be52bc7'
|
||||
down_revision = '9ec4c765ef8d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS destination TEXT')
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS tag_names TEXT[]')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS tag_names')
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS destination')
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
"""Downloader
|
||||
|
||||
Revision ID: 184329ed5137
|
||||
Revises: 1870bec7c81e
|
||||
Create Date: 2021-11-17 19:13:58.305140
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '184329ed5137'
|
||||
down_revision = '1870bec7c81e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE download (
|
||||
id SERIAL PRIMARY KEY,
|
||||
url TEXT NOT NULL,
|
||||
status TEXT DEFAULT 'new',
|
||||
next_download TIMESTAMPTZ,
|
||||
last_successful_download TIMESTAMPTZ,
|
||||
frequency INT,
|
||||
attempts INT DEFAULT 0
|
||||
)
|
||||
''')
|
||||
session.execute('ALTER TABLE video ADD COLUMN url TEXT')
|
||||
session.execute('ALTER TABLE channel ADD COLUMN source_id TEXT')
|
||||
session.execute('CREATE INDEX channel_source_id ON channel (source_id)')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.download OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP TABLE IF EXISTS download')
|
||||
session.execute('DROP INDEX IF EXISTS channel_source_id')
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS source_id')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS url')
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
"""Initialize archive tables.
|
||||
|
||||
Revision ID: 1870bec7c81e
|
||||
Revises: 053f17f07c4e
|
||||
Create Date: 2021-09-22 18:31:15.056697
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '1870bec7c81e'
|
||||
down_revision = '053f17f07c4e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE domains (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain TEXT NOT NULL UNIQUE,
|
||||
directory TEXT NOT NULL UNIQUE
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE url (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain_id INTEGER REFERENCES domains(id) NOT NULL,
|
||||
url TEXT NOT NULL UNIQUE,
|
||||
latest_datetime TIMESTAMPTZ
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE archive (
|
||||
id SERIAL PRIMARY KEY,
|
||||
url_id INTEGER REFERENCES url(id) ON DELETE CASCADE NOT NULL,
|
||||
domain_id INTEGER REFERENCES domains(id) ON DELETE CASCADE NOT NULL,
|
||||
singlefile_path TEXT,
|
||||
readability_path TEXT,
|
||||
readability_txt_path TEXT,
|
||||
readability_json_path TEXT,
|
||||
screenshot_path TEXT,
|
||||
title TEXT,
|
||||
archive_datetime TIMESTAMPTZ,
|
||||
status TEXT
|
||||
)''')
|
||||
|
||||
session.execute('ALTER TABLE url ADD COLUMN latest_id INTEGER REFERENCES archive(id) ON DELETE CASCADE')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.domains OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.url OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.archive OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS domains CASCADE')
|
||||
session.execute('DROP TABLE IF EXISTS url CASCADE')
|
||||
session.execute('DROP TABLE IF EXISTS archive CASCADE')
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Download location
|
||||
|
||||
Revision ID: 1a70196c05bb
|
||||
Revises: d8fcdb7773c0
|
||||
Create Date: 2022-05-02 12:01:51.210070
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '1a70196c05bb'
|
||||
down_revision = 'd8fcdb7773c0'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS location TEXT')
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS error TEXT')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS location')
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS error')
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
"""Doc indexer.
|
||||
|
||||
Revision ID: 1c041aeee0fc
|
||||
Revises: d96a04eb2785
|
||||
Create Date: 2024-03-05 16:54:34.478974
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '1c041aeee0fc'
|
||||
down_revision = 'd96a04eb2785'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Re-index Word documents.
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute(
|
||||
"""
|
||||
UPDATE file_group
|
||||
SET indexed=false
|
||||
WHERE mimetype in
|
||||
('application/msword','application/vnd.openxmlformats-officedocument.wordprocessingml.document')
|
||||
""")
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,137 +0,0 @@
|
|||
"""Create `Download.destination` and `Download.tag_names`, migrate from `Download.settings`.
|
||||
|
||||
Revision ID: 1f2325523525
|
||||
Revises: 170d1be52bc7
|
||||
Create Date: 2024-11-02 11:20:45.028865
|
||||
|
||||
"""
|
||||
import os
|
||||
from copy import copy
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import Column, Integer, Text, String, ARRAY
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from wrolpi.common import ModelHelper
|
||||
from wrolpi.dates import TZDateTime
|
||||
from wrolpi.downloader import get_download_manager_config, DownloadStatus
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '1f2325523525'
|
||||
down_revision = '170d1be52bc7'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
class MDownload(ModelHelper, Base):
|
||||
__tablename__ = 'download'
|
||||
id = Column(Integer, primary_key=True)
|
||||
url = Column(String, nullable=False, unique=True)
|
||||
attempts = Column(Integer, default=0)
|
||||
destination = Column(Text)
|
||||
downloader = Column(Text)
|
||||
sub_downloader = Column(Text)
|
||||
error = Column(Text)
|
||||
frequency = Column(Integer)
|
||||
info_json = Column(JSONB)
|
||||
last_successful_download = Column(TZDateTime)
|
||||
location = Column(Text)
|
||||
next_download = Column(TZDateTime)
|
||||
settings = Column(JSONB)
|
||||
status = Column(String, default=DownloadStatus.new)
|
||||
tag_names = Column(ARRAY(Text))
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
for download in session.query(MDownload):
|
||||
if not download.settings:
|
||||
continue
|
||||
# Move `download.settings['destination']` to `download.destination`, if any.
|
||||
settings = copy(download.settings) if download.settings else dict()
|
||||
destination = settings.pop('destination', None)
|
||||
tag_names = settings.pop('tag_names', None)
|
||||
if isinstance(excluded_urls := settings.get('excluded_urls'), list):
|
||||
settings['excluded_urls'] = ','.join(excluded_urls)
|
||||
download.destination = destination or None
|
||||
download.tag_names = tag_names or None
|
||||
download.settings = settings
|
||||
|
||||
# This will migrate the config as-is at the time that this migration was written. This code is probably not
|
||||
# safe to reuse in the future!
|
||||
|
||||
# Read config file directly.
|
||||
try:
|
||||
config = get_download_manager_config()
|
||||
config._config.update(config.read_config_file())
|
||||
except FileNotFoundError:
|
||||
# Config does not exist, probably testing, or a new WROLPi.
|
||||
return
|
||||
|
||||
new_downloads = []
|
||||
for download in session.query(MDownload).order_by(MDownload.url):
|
||||
if download.last_successful_download and not download.frequency:
|
||||
# This once-download has completed, do not save it.
|
||||
continue
|
||||
new_download = dict(
|
||||
destination=download.destination,
|
||||
downloader=download.downloader,
|
||||
frequency=download.frequency,
|
||||
last_successful_download=download.last_successful_download,
|
||||
next_download=download.next_download,
|
||||
settings=download.settings,
|
||||
status=download.status,
|
||||
sub_downloader=download.sub_downloader,
|
||||
url=download.url,
|
||||
)
|
||||
new_downloads.append(new_download)
|
||||
|
||||
# Write directly to file, bypassing usual checks.
|
||||
config._config['downloads'] = new_downloads
|
||||
config.write_config_data(config._config, config.get_file())
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
for download in session.query(MDownload):
|
||||
if download.destination:
|
||||
settings = copy(download.settings) if download.settings else dict()
|
||||
settings['destination'] = download.destination or settings.get('destination')
|
||||
settings['tag_names'] = download.tag_names or settings.get('tag_names')
|
||||
if isinstance(excluded_urls := settings.get('excluded_urls'), str):
|
||||
settings['excluded_urls'] = excluded_urls.split(',')
|
||||
download.settings = settings
|
||||
|
||||
try:
|
||||
config = get_download_manager_config()
|
||||
config._config.update(config.read_config_file())
|
||||
except FileNotFoundError:
|
||||
# Config does not exist, probably testing, or a new WROLPi.
|
||||
return
|
||||
|
||||
new_downloads = []
|
||||
for download in session.query(MDownload).order_by(MDownload.url):
|
||||
new_download = dict(
|
||||
downloader=download.downloader,
|
||||
frequency=download.frequency,
|
||||
last_successful_download=download.last_successful_download,
|
||||
next_download=download.next_download,
|
||||
settings=download.settings,
|
||||
status=download.status,
|
||||
sub_downloader=download.sub_downloader,
|
||||
url=download.url,
|
||||
)
|
||||
new_downloads.append(new_download)
|
||||
|
||||
config._config['downloads'] = new_downloads
|
||||
config.write_config_data(config._config, config.get_file())
|
||||
378
alembic/versions/2026_07_10_1350_sqlite_baseline.py
Normal file
378
alembic/versions/2026_07_10_1350_sqlite_baseline.py
Normal file
|
|
@ -0,0 +1,378 @@
|
|||
"""The SQLite baseline.
|
||||
|
||||
The entire Postgres-era migration history (85 revisions) was squashed into this baseline when
|
||||
WROLPi moved its database into the media directory (SQLite). See the git history of
|
||||
alembic/versions/ for the old chain.
|
||||
|
||||
Revision ID: 2026_07_10_1350
|
||||
Revises:
|
||||
Create Date: 2026-07-10 13:50:59.070646
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
import wrolpi.dates
|
||||
import wrolpi.files.models
|
||||
import wrolpi.media_path
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '2026_07_10_1350'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table('directory',
|
||||
sa.Column('path', wrolpi.media_path.MediaPathType(), nullable=False),
|
||||
sa.Column('name', sa.String(), nullable=False),
|
||||
sa.Column('idempotency', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.PrimaryKeyConstraint('path')
|
||||
)
|
||||
with op.batch_alter_table('directory', schema=None) as batch_op:
|
||||
batch_op.create_index('directory_name_idx', ['name'], unique=False)
|
||||
|
||||
op.create_table('file_group',
|
||||
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
|
||||
sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=False),
|
||||
sa.Column('author', sa.String(), nullable=True),
|
||||
sa.Column('censored', sa.Boolean(), nullable=True),
|
||||
sa.Column('data', wrolpi.files.models.FancyJSON(), nullable=True),
|
||||
sa.Column('download_datetime', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('files', wrolpi.files.models.FancyJSON(), nullable=True),
|
||||
sa.Column('idempotency', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('indexed', sa.Boolean(), nullable=True),
|
||||
sa.Column('length', sa.BigInteger(), nullable=True),
|
||||
sa.Column('mimetype', sa.String(), nullable=True),
|
||||
sa.Column('model', sa.String(), nullable=True),
|
||||
sa.Column('modification_datetime', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('primary_path', wrolpi.media_path.MediaPathType(), nullable=False),
|
||||
sa.Column('published_datetime', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('published_modified_datetime', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('size', sa.BigInteger(), nullable=True),
|
||||
sa.Column('suffix', sa.String(), nullable=True),
|
||||
sa.Column('title', sa.String(), nullable=True),
|
||||
sa.Column('url', sa.String(), nullable=True),
|
||||
sa.Column('viewed', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('effective_datetime', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('a_text', sa.String(), nullable=True),
|
||||
sa.Column('b_text', sa.String(), nullable=True),
|
||||
sa.Column('c_text', sa.String(), nullable=True),
|
||||
sa.Column('d_text', sa.String(), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('primary_path')
|
||||
)
|
||||
with op.batch_alter_table('file_group', schema=None) as batch_op:
|
||||
batch_op.create_index('file_group_author_idx', ['author'], unique=False)
|
||||
batch_op.create_index('file_group_directory_idx', ['directory'], unique=False)
|
||||
batch_op.create_index('file_group_download_datetime_idx', ['download_datetime'], unique=False)
|
||||
batch_op.create_index('file_group_effective_datetime_idx', ['effective_datetime'], unique=False)
|
||||
batch_op.create_index('file_group_length_idx', ['length'], unique=False)
|
||||
batch_op.create_index('file_group_mimetype_idx', ['mimetype'], unique=False)
|
||||
batch_op.create_index('file_group_model_idx', ['model'], unique=False)
|
||||
batch_op.create_index('file_group_modification_datetime_idx', ['modification_datetime'], unique=False)
|
||||
batch_op.create_index('file_group_published_datetime_idx', ['published_datetime'], unique=False)
|
||||
batch_op.create_index('file_group_published_modified_datetime_idx', ['published_modified_datetime'], unique=False)
|
||||
batch_op.create_index('file_group_size_ix', ['size'], unique=False)
|
||||
batch_op.create_index('file_group_url_idx', ['url'], unique=False)
|
||||
batch_op.create_index('file_group_viewed_idx', ['viewed'], unique=False)
|
||||
batch_op.create_index(batch_op.f('ix_file_group_suffix'), ['suffix'], unique=False)
|
||||
|
||||
op.create_table('tag',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('name', sa.String(), nullable=False),
|
||||
sa.Column('color', sa.String(), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('name')
|
||||
)
|
||||
op.create_table('wrolpi_flag',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('refresh_complete', sa.Boolean(), nullable=True),
|
||||
sa.Column('outdated_zims', sa.Boolean(), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_table('collection',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('name', sa.String(), nullable=False),
|
||||
sa.Column('description', sa.Text(), nullable=True),
|
||||
sa.Column('kind', sa.String(), server_default='channel', nullable=False),
|
||||
sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=True),
|
||||
sa.Column('tag_id', sa.Integer(), nullable=True),
|
||||
sa.Column('file_format', sa.String(), nullable=True),
|
||||
sa.Column('created_date', sa.DateTime(), server_default=sa.text('(CURRENT_TIMESTAMP)'), nullable=False),
|
||||
sa.Column('item_count', sa.Integer(), nullable=False),
|
||||
sa.Column('total_size', sa.BigInteger(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('directory', name='uq_collection_directory'),
|
||||
sa.UniqueConstraint('name', 'kind', name='uq_collection_name_kind')
|
||||
)
|
||||
with op.batch_alter_table('collection', schema=None) as batch_op:
|
||||
batch_op.create_index('idx_collection_item_count', ['item_count'], unique=False)
|
||||
batch_op.create_index('idx_collection_kind', ['kind'], unique=False)
|
||||
batch_op.create_index('idx_collection_total_size', ['total_size'], unique=False)
|
||||
|
||||
op.create_table('doc',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('size', sa.BigInteger(), nullable=True),
|
||||
sa.Column('publisher', sa.String(), nullable=True),
|
||||
sa.Column('language', sa.String(), nullable=True),
|
||||
sa.Column('page_count', sa.Integer(), nullable=True),
|
||||
sa.Column('subject', sa.String(), nullable=True),
|
||||
sa.Column('description', sa.Text(), nullable=True),
|
||||
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('file_group_id')
|
||||
)
|
||||
with op.batch_alter_table('doc', schema=None) as batch_op:
|
||||
batch_op.create_index('doc_size_idx', ['size'], unique=False)
|
||||
|
||||
op.create_table('tag_file',
|
||||
sa.Column('created_at', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('tag_id', sa.Integer(), nullable=False),
|
||||
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
|
||||
sa.PrimaryKeyConstraint('tag_id', 'file_group_id'),
|
||||
sa.UniqueConstraint('tag_id', 'file_group_id', name='tag_file_tag_id_file_group_id_key')
|
||||
)
|
||||
op.create_table('zim',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('path', wrolpi.media_path.MediaPathType(), nullable=False),
|
||||
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
|
||||
sa.Column('auto_search', sa.Boolean(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('path', name='zim_path_key')
|
||||
)
|
||||
op.create_table('archive',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('collection_id', sa.Integer(), nullable=True),
|
||||
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('file_group_id')
|
||||
)
|
||||
with op.batch_alter_table('archive', schema=None) as batch_op:
|
||||
batch_op.create_index('idx_archive_collection_id', ['collection_id'], unique=False)
|
||||
|
||||
op.create_table('channel',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('url', sa.String(), nullable=True),
|
||||
sa.Column('generate_posters', sa.Boolean(), nullable=True),
|
||||
sa.Column('calculate_duration', sa.Boolean(), nullable=True),
|
||||
sa.Column('download_missing_data', sa.Boolean(), nullable=True),
|
||||
sa.Column('source_id', sa.String(), nullable=True),
|
||||
sa.Column('refreshed', sa.Boolean(), nullable=True),
|
||||
sa.Column('video_count', sa.Integer(), nullable=False),
|
||||
sa.Column('total_size', sa.BigInteger(), nullable=False),
|
||||
sa.Column('minimum_frequency', sa.Integer(), nullable=True),
|
||||
sa.Column('info_json', sa.JSON(), nullable=True),
|
||||
sa.Column('info_date', sa.Date(), nullable=True),
|
||||
sa.Column('collection_id', sa.Integer(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
with op.batch_alter_table('channel', schema=None) as batch_op:
|
||||
batch_op.create_index('channel_minimum_frequency_idx', ['minimum_frequency'], unique=False)
|
||||
batch_op.create_index('channel_source_id', ['source_id'], unique=False)
|
||||
batch_op.create_index('channel_total_size_idx', ['total_size'], unique=False)
|
||||
batch_op.create_index('channel_url_key', ['url'], unique=True, postgresql_where=sa.text('url IS NOT NULL'), sqlite_where=sa.text('url IS NOT NULL'))
|
||||
batch_op.create_index('channel_video_count_idx', ['video_count'], unique=False)
|
||||
|
||||
op.create_table('collection_item',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('collection_id', sa.Integer(), nullable=False),
|
||||
sa.Column('item_kind', sa.String(), server_default='file', nullable=False),
|
||||
sa.Column('file_group_id', sa.Integer(), nullable=True),
|
||||
sa.Column('zim_id', sa.Integer(), nullable=True),
|
||||
sa.Column('zim_entry', sa.Text(), nullable=True),
|
||||
sa.Column('url', sa.Text(), nullable=True),
|
||||
sa.Column('title', sa.Text(), nullable=True),
|
||||
sa.Column('position', sa.Integer(), nullable=False),
|
||||
sa.Column('added_date', sa.DateTime(), server_default=sa.text('(CURRENT_TIMESTAMP)'), nullable=False),
|
||||
sa.CheckConstraint("(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL) OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL AND file_group_id IS NULL AND url IS NULL) OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)", name='ck_collection_item_kind'),
|
||||
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['zim_id'], ['zim.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('collection_id', 'file_group_id', name='uq_collection_file_group'),
|
||||
sa.UniqueConstraint('collection_id', 'url', name='uq_collection_url'),
|
||||
sa.UniqueConstraint('collection_id', 'zim_id', 'zim_entry', name='uq_collection_zim_entry')
|
||||
)
|
||||
with op.batch_alter_table('collection_item', schema=None) as batch_op:
|
||||
batch_op.create_index('idx_collection_item_collection_id', ['collection_id'], unique=False)
|
||||
batch_op.create_index('idx_collection_item_collection_position', ['collection_id', 'position'], unique=False)
|
||||
batch_op.create_index('idx_collection_item_file_group_id', ['file_group_id'], unique=False)
|
||||
batch_op.create_index('idx_collection_item_position', ['position'], unique=False)
|
||||
|
||||
op.create_table('doc_section',
|
||||
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
|
||||
sa.Column('doc_id', sa.Integer(), nullable=False),
|
||||
sa.Column('kind', sa.String(), nullable=False),
|
||||
sa.Column('ordinal', sa.Integer(), nullable=False),
|
||||
sa.Column('label', sa.String(), nullable=True),
|
||||
sa.Column('content', sa.Text(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['doc_id'], ['doc.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
with op.batch_alter_table('doc_section', schema=None) as batch_op:
|
||||
batch_op.create_index('doc_section_doc_id_idx', ['doc_id', 'ordinal'], unique=False)
|
||||
|
||||
op.create_table('download',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('url', sa.String(), nullable=False),
|
||||
sa.Column('attempts', sa.Integer(), nullable=True),
|
||||
sa.Column('destination', wrolpi.media_path.MediaPathType(), nullable=True),
|
||||
sa.Column('downloader', sa.Text(), nullable=True),
|
||||
sa.Column('sub_downloader', sa.Text(), nullable=True),
|
||||
sa.Column('error', sa.Text(), nullable=True),
|
||||
sa.Column('frequency', sa.Integer(), nullable=True),
|
||||
sa.Column('info_json', sa.JSON(), nullable=True),
|
||||
sa.Column('last_download_attempt', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('last_successful_download', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('location', sa.Text(), nullable=True),
|
||||
sa.Column('next_download', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.Column('settings', sa.JSON(), nullable=True),
|
||||
sa.Column('status', sa.String(), nullable=True),
|
||||
sa.Column('tag_names', sa.JSON(), nullable=True),
|
||||
sa.Column('collection_id', sa.Integer(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('url')
|
||||
)
|
||||
with op.batch_alter_table('download', schema=None) as batch_op:
|
||||
batch_op.create_index('idx_download_collection_id', ['collection_id'], unique=False)
|
||||
batch_op.create_index('idx_download_last_download_attempt', ['last_download_attempt'], unique=False)
|
||||
|
||||
op.create_table('tag_zim',
|
||||
sa.Column('tag_id', sa.Integer(), nullable=False),
|
||||
sa.Column('zim_id', sa.Integer(), nullable=False),
|
||||
sa.Column('zim_entry', sa.Text(), nullable=False),
|
||||
sa.Column('created_at', wrolpi.dates.TZDateTime(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], ),
|
||||
sa.ForeignKeyConstraint(['zim_id'], ['zim.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('tag_id', 'zim_id', 'zim_entry'),
|
||||
sa.UniqueConstraint('tag_id', 'zim_id', 'zim_entry', name='tag_zim_tag_id_zim_id_zim_entry_key')
|
||||
)
|
||||
op.create_table('video',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('source_id', sa.String(), nullable=True),
|
||||
sa.Column('view_count', sa.BigInteger(), nullable=True),
|
||||
sa.Column('ffprobe_json', sa.JSON(), nullable=True),
|
||||
sa.Column('have_comments', sa.Boolean(), nullable=True),
|
||||
sa.Column('comments_failed', sa.Boolean(), nullable=True),
|
||||
sa.Column('channel_id', sa.Integer(), nullable=True),
|
||||
sa.Column('file_group_id', sa.BigInteger(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['channel_id'], ['channel.id'], ),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('file_group_id'),
|
||||
sqlite_autoincrement=True
|
||||
)
|
||||
with op.batch_alter_table('video', schema=None) as batch_op:
|
||||
batch_op.create_index('video_channel_id_idx', ['channel_id'], unique=False)
|
||||
batch_op.create_index('video_source_id_idx', ['source_id'], unique=False)
|
||||
batch_op.create_index('video_view_count_idx', ['view_count'], unique=False)
|
||||
|
||||
op.create_table('zim_subscription',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('name', sa.Text(), nullable=False),
|
||||
sa.Column('language', sa.Text(), nullable=False),
|
||||
sa.Column('download_id', sa.Integer(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['download_id'], ['download.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('name')
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
# Triggers (channel summaries, effective_datetime) and the FTS5 full-text-search tables.
|
||||
# Shared with the test databases (wrolpi/conftest.py) so tests exercise the same schema.
|
||||
from wrolpi.schema_ddl import install_raw_ddl
|
||||
from wrolpi.fts import rebuild_fts
|
||||
|
||||
connection = op.get_bind()
|
||||
install_raw_ddl(connection)
|
||||
# Index any existing content (no-op on a fresh database).
|
||||
rebuild_fts(connection)
|
||||
|
||||
|
||||
def downgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_table('zim_subscription')
|
||||
with op.batch_alter_table('video', schema=None) as batch_op:
|
||||
batch_op.drop_index('video_view_count_idx')
|
||||
batch_op.drop_index('video_source_id_idx')
|
||||
batch_op.drop_index('video_channel_id_idx')
|
||||
|
||||
op.drop_table('video')
|
||||
op.drop_table('tag_zim')
|
||||
with op.batch_alter_table('download', schema=None) as batch_op:
|
||||
batch_op.drop_index('idx_download_last_download_attempt')
|
||||
batch_op.drop_index('idx_download_collection_id')
|
||||
|
||||
op.drop_table('download')
|
||||
with op.batch_alter_table('doc_section', schema=None) as batch_op:
|
||||
batch_op.drop_index('doc_section_doc_id_idx')
|
||||
|
||||
op.drop_table('doc_section')
|
||||
with op.batch_alter_table('collection_item', schema=None) as batch_op:
|
||||
batch_op.drop_index('idx_collection_item_position')
|
||||
batch_op.drop_index('idx_collection_item_file_group_id')
|
||||
batch_op.drop_index('idx_collection_item_collection_position')
|
||||
batch_op.drop_index('idx_collection_item_collection_id')
|
||||
|
||||
op.drop_table('collection_item')
|
||||
with op.batch_alter_table('channel', schema=None) as batch_op:
|
||||
batch_op.drop_index('channel_video_count_idx')
|
||||
batch_op.drop_index('channel_url_key', postgresql_where=sa.text('url IS NOT NULL'), sqlite_where=sa.text('url IS NOT NULL'))
|
||||
batch_op.drop_index('channel_total_size_idx')
|
||||
batch_op.drop_index('channel_source_id')
|
||||
batch_op.drop_index('channel_minimum_frequency_idx')
|
||||
|
||||
op.drop_table('channel')
|
||||
with op.batch_alter_table('archive', schema=None) as batch_op:
|
||||
batch_op.drop_index('idx_archive_collection_id')
|
||||
|
||||
op.drop_table('archive')
|
||||
op.drop_table('zim')
|
||||
op.drop_table('tag_file')
|
||||
with op.batch_alter_table('doc', schema=None) as batch_op:
|
||||
batch_op.drop_index('doc_size_idx')
|
||||
|
||||
op.drop_table('doc')
|
||||
with op.batch_alter_table('collection', schema=None) as batch_op:
|
||||
batch_op.drop_index('idx_collection_total_size')
|
||||
batch_op.drop_index('idx_collection_kind')
|
||||
batch_op.drop_index('idx_collection_item_count')
|
||||
|
||||
op.drop_table('collection')
|
||||
op.drop_table('wrolpi_flag')
|
||||
op.drop_table('tag')
|
||||
with op.batch_alter_table('file_group', schema=None) as batch_op:
|
||||
batch_op.drop_index(batch_op.f('ix_file_group_suffix'))
|
||||
batch_op.drop_index('file_group_viewed_idx')
|
||||
batch_op.drop_index('file_group_url_idx')
|
||||
batch_op.drop_index('file_group_size_ix')
|
||||
batch_op.drop_index('file_group_published_modified_datetime_idx')
|
||||
batch_op.drop_index('file_group_published_datetime_idx')
|
||||
batch_op.drop_index('file_group_modification_datetime_idx')
|
||||
batch_op.drop_index('file_group_model_idx')
|
||||
batch_op.drop_index('file_group_mimetype_idx')
|
||||
batch_op.drop_index('file_group_length_idx')
|
||||
batch_op.drop_index('file_group_effective_datetime_idx')
|
||||
batch_op.drop_index('file_group_download_datetime_idx')
|
||||
batch_op.drop_index('file_group_directory_idx')
|
||||
batch_op.drop_index('file_group_author_idx')
|
||||
|
||||
op.drop_table('file_group')
|
||||
with op.batch_alter_table('directory', schema=None) as batch_op:
|
||||
batch_op.drop_index('directory_name_idx')
|
||||
|
||||
op.drop_table('directory')
|
||||
# ### end Alembic commands ###
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
"""make_filegroup_directory_not_null
|
||||
|
||||
Revision ID: 26c5a62ec683
|
||||
Revises: af4527503bdf
|
||||
Create Date: 2025-12-23 20:47:11.549264
|
||||
|
||||
This migration makes the `directory` column NOT NULL after the previous migration
|
||||
has populated it for all existing FileGroups.
|
||||
"""
|
||||
import os
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '26c5a62ec683'
|
||||
down_revision = 'af4527503bdf'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Safety check: ensure no NULL directory values remain
|
||||
result = session.execute(sa.text(
|
||||
"SELECT COUNT(*) FROM file_group WHERE directory IS NULL"
|
||||
))
|
||||
null_count = result.scalar()
|
||||
if null_count > 0:
|
||||
# Fill any remaining NULL values (shouldn't happen if previous migration ran)
|
||||
print(f"Warning: Found {null_count} FileGroups with NULL directory, filling from primary_path...")
|
||||
session.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET directory = regexp_replace(primary_path, '/[^/]+$', '')
|
||||
WHERE directory IS NULL AND primary_path IS NOT NULL
|
||||
"""))
|
||||
session.commit()
|
||||
|
||||
# Make the column NOT NULL
|
||||
op.alter_column('file_group', 'directory', nullable=False)
|
||||
|
||||
# Create index on directory column for efficient directory lookups
|
||||
op.create_index('file_group_directory_idx', 'file_group', ['directory'])
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute(sa.text('ALTER TABLE public.file_group OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Drop the index
|
||||
op.drop_index('file_group_directory_idx', 'file_group')
|
||||
# Make the column nullable again
|
||||
op.alter_column('file_group', 'directory', nullable=True)
|
||||
|
|
@ -1,137 +0,0 @@
|
|||
"""MTag names migration
|
||||
|
||||
Revision ID: 2bfb3b50b178
|
||||
Revises: 8d0d81bc9c34
|
||||
Create Date: 2024-07-31 14:58:29.583029
|
||||
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
from datetime import datetime
|
||||
|
||||
import yaml
|
||||
from alembic import op
|
||||
from sqlalchemy import Column, Integer, BigInteger, ForeignKey, Text, String
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.orm.exc import NoResultFound # noqa
|
||||
|
||||
from wrolpi import dates
|
||||
from wrolpi.common import escape_file_name, get_media_directory, ModelHelper
|
||||
from wrolpi.common import get_relative_to_media_directory
|
||||
from wrolpi.dates import TZDateTime
|
||||
from wrolpi.tags import get_tags_config
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '2bfb3b50b178'
|
||||
down_revision = '8d0d81bc9c34'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
class MFileGroup(ModelHelper, Base):
|
||||
__tablename__ = 'file_group'
|
||||
id: int = Column(BigInteger, primary_key=True)
|
||||
primary_path: pathlib.Path = Column(String, unique=True, nullable=False)
|
||||
|
||||
|
||||
class MTagFile(ModelHelper, Base):
|
||||
__tablename__ = 'tag_file'
|
||||
tag_id = Column(Integer, ForeignKey('tag.id', ondelete='CASCADE'), primary_key=True)
|
||||
file_group_id = Column(BigInteger, ForeignKey('file_group.id', ondelete='CASCADE'), primary_key=True)
|
||||
|
||||
|
||||
class MTag(ModelHelper, Base):
|
||||
__tablename__ = 'tag'
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
color = Column(String)
|
||||
|
||||
|
||||
class MZim(Base):
|
||||
__tablename__ = 'zim'
|
||||
id = Column(Integer, primary_key=True)
|
||||
path: pathlib.Path = Column(String, unique=True, nullable=False)
|
||||
|
||||
|
||||
class MTagZimEntry(Base):
|
||||
__tablename__ = 'tag_zim'
|
||||
tag_id = Column(Integer, ForeignKey('tag.id', ondelete='CASCADE'), primary_key=True)
|
||||
zim_id = Column(Integer, ForeignKey('zim.id', ondelete='CASCADE'), primary_key=True)
|
||||
zim_entry: str = Column(Text, nullable=False, primary_key=True)
|
||||
created_at: datetime = Column(TZDateTime, default=dates.now)
|
||||
|
||||
|
||||
def save_tags(session: Session):
|
||||
media_directory = get_media_directory()
|
||||
|
||||
tags = dict()
|
||||
tag_rows = session.query(MTag)
|
||||
for tag in tag_rows:
|
||||
tags[tag.name] = dict(color=tag.color)
|
||||
|
||||
results = session.query(MTag, MTagFile, MFileGroup) \
|
||||
.filter(MTagFile.tag_id == MTag.id, MTagFile.file_group_id == MFileGroup.id) \
|
||||
.order_by(MFileGroup.primary_path)
|
||||
|
||||
tag_files = []
|
||||
for tag, tag_file, file_group in results:
|
||||
value = [
|
||||
tag.name,
|
||||
str(file_group.primary_path.relative_to(media_directory)),
|
||||
# Fallback to current time if not set.
|
||||
tag_file.created_at.isoformat() if tag_file.created_at else dates.now().isoformat(),
|
||||
]
|
||||
tag_files.append(value)
|
||||
|
||||
results = session.query(MTag, MZim, MTagZimEntry) \
|
||||
.filter(MTag.id == MTagZimEntry.tag_id, MZim.id == MTagZimEntry.zim_id) \
|
||||
.order_by(MTagZimEntry.zim_id, MTagZimEntry.zim_entry)
|
||||
|
||||
tag_zims = []
|
||||
for tag, zim, tag_zim_entry in results:
|
||||
zim: MZim
|
||||
tag_zim_entry: MTagZimEntry
|
||||
value = [
|
||||
tag.name,
|
||||
str(get_relative_to_media_directory(zim.path)),
|
||||
tag_zim_entry.zim_entry,
|
||||
# Fallback to current time if not set.
|
||||
tag_zim_entry.created_at.isoformat() if tag_zim_entry.created_at else dates.now().isoformat(),
|
||||
]
|
||||
tag_zims.append(value)
|
||||
|
||||
# Write to the config.
|
||||
with get_tags_config().get_file().open('wt') as fh:
|
||||
data = {
|
||||
'tag_files': tag_files,
|
||||
'tag_zims': tag_zims,
|
||||
'tags': tags,
|
||||
}
|
||||
yaml.dump(data, fh)
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
need_commit = False
|
||||
tags = session.query(MTag).all()
|
||||
for tag in tags:
|
||||
escaped_name = escape_file_name(tag.name)
|
||||
if escaped_name != tag.name:
|
||||
tag.name = escaped_name
|
||||
session.flush([tag, ])
|
||||
need_commit = True
|
||||
|
||||
if need_commit:
|
||||
session.commit()
|
||||
save_tags(session)
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
"""Create MapFile table.
|
||||
|
||||
Revision ID: 2fb39c1861c7
|
||||
Revises: 36cae041ad31
|
||||
Create Date: 2022-03-03 19:13:39.997580
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '2fb39c1861c7'
|
||||
down_revision = '36cae041ad31'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''CREATE TABLE map_file (
|
||||
id SERIAL PRIMARY KEY,
|
||||
path TEXT NOT NULL,
|
||||
imported BOOLEAN DEFAULT FALSE,
|
||||
size BIGINT
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.map_file OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP TABLE IF EXISTS map_file')
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
"""Archive contents search.
|
||||
|
||||
Revision ID: 31896cc29b71
|
||||
Revises: bddb5e95a91e
|
||||
Create Date: 2021-12-31 18:53:25.326674
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '31896cc29b71'
|
||||
down_revision = 'bddb5e95a91e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE archive ADD COLUMN contents TEXT')
|
||||
session.execute('''
|
||||
ALTER TABLE archive
|
||||
ADD COLUMN textsearch tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, title), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, contents), 'D')
|
||||
) STORED
|
||||
''')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS archive_textsearch_idx ON archive USING GIN(textsearch)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS archive_textsearch_idx')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS contents')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS textsearch')
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Video censored
|
||||
|
||||
Revision ID: 32d96eb9122d
|
||||
Revises: 184329ed5137
|
||||
Create Date: 2021-12-23 14:03:53.677547
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '32d96eb9122d'
|
||||
down_revision = '184329ed5137'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE video ADD COLUMN censored BOOL DEFAULT false')
|
||||
session.execute('CREATE INDEX video_censored ON video (censored)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS censored')
|
||||
session.execute('DROP INDEX IF EXISTS video_censored')
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
"""eBook creator.
|
||||
|
||||
Revision ID: 365c985893b8
|
||||
Revises: 5ad9efbf3570
|
||||
Create Date: 2023-10-29 16:54:28.319427
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '365c985893b8'
|
||||
down_revision = '5ad9efbf3570'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET author=creator
|
||||
FROM ebook
|
||||
WHERE ebook.file_group_id = file_group.id
|
||||
''')
|
||||
session.execute('DROP INDEX IF EXISTS ebook_creator_idx')
|
||||
session.execute('ALTER TABLE ebook DROP COLUMN IF EXISTS creator')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE ebook ADD COLUMN IF NOT EXISTS creator TEXT')
|
||||
session.execute('''
|
||||
UPDATE ebook
|
||||
SET creator=author
|
||||
FROM file_group
|
||||
WHERE
|
||||
ebook.file_group_id = file_group.id
|
||||
AND model = 'ebook'
|
||||
''')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS ebook_creator_idx ON ebook(creator)')
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
"""Files
|
||||
|
||||
Revision ID: 36cae041ad31
|
||||
Revises: b97c1313f986
|
||||
Create Date: 2022-02-01 11:05:57.605541
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '36cae041ad31'
|
||||
down_revision = 'b97c1313f986'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''CREATE TABLE file (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
path TEXT NOT NULL,
|
||||
title TEXT,
|
||||
mimetype TEXT,
|
||||
size INTEGER,
|
||||
modification_datetime TIMESTAMPTZ,
|
||||
textsearch tsvector GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, title), 'A'::"char")
|
||||
) stored,
|
||||
idempotency TEXT
|
||||
)''')
|
||||
session.execute('CREATE INDEX file_path_idx ON file(path)')
|
||||
session.execute('CREATE INDEX file_mimetype_idx ON file(textsearch)')
|
||||
session.execute('CREATE INDEX file_textsearch_idx ON file(textsearch)')
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.file OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_textsearch_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_mimetype_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_path_idx')
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS file')
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
"""File group.
|
||||
|
||||
Revision ID: 3b6918aeca4b
|
||||
Revises: 930b9b7adb79
|
||||
Create Date: 2023-02-27 14:52:46.943351
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '3b6918aeca4b'
|
||||
down_revision = '930b9b7adb79'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DELETE FROM video') # noqa
|
||||
session.execute('ALTER TABLE video DROP COLUMN caption_path')
|
||||
session.execute('ALTER TABLE video DROP COLUMN censored')
|
||||
session.execute('ALTER TABLE video DROP COLUMN favorite')
|
||||
session.execute('ALTER TABLE video DROP COLUMN info_json_path')
|
||||
session.execute('ALTER TABLE video DROP COLUMN modification_datetime')
|
||||
session.execute('ALTER TABLE video DROP COLUMN poster_path')
|
||||
session.execute('ALTER TABLE video DROP COLUMN size')
|
||||
session.execute('ALTER TABLE video DROP COLUMN title')
|
||||
session.execute('ALTER TABLE video DROP COLUMN validated')
|
||||
session.execute('ALTER TABLE video DROP COLUMN video_path')
|
||||
|
||||
session.execute('DELETE FROM archive') # noqa
|
||||
session.execute('ALTER TABLE archive DROP COLUMN readability_json_path')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN readability_path')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN readability_txt_path')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN screenshot_path')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN singlefile_path')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN title')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN validated')
|
||||
|
||||
session.execute('DELETE FROM ebook') # noqa
|
||||
session.execute('ALTER TABLE ebook DROP COLUMN cover_path')
|
||||
session.execute('ALTER TABLE ebook DROP COLUMN ebook_path')
|
||||
session.execute('ALTER TABLE ebook DROP COLUMN title')
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS file')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE file_group (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
data JSON,
|
||||
files JSON,
|
||||
idempotency TIMESTAMP WITH TIME ZONE,
|
||||
indexed BOOLEAN DEFAULT FALSE,
|
||||
mimetype TEXT NOT NULL,
|
||||
model TEXT,
|
||||
modification_datetime TIMESTAMP WITH TIME ZONE,
|
||||
primary_path TEXT NOT NULL UNIQUE,
|
||||
size BIGINT,
|
||||
title TEXT,
|
||||
a_text TEXT,
|
||||
b_text TEXT,
|
||||
c_text TEXT,
|
||||
d_text TEXT,
|
||||
textsearch tsvector GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
|
||||
) STORED
|
||||
)''')
|
||||
|
||||
session.execute(
|
||||
'ALTER TABLE video ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
|
||||
session.execute(
|
||||
'ALTER TABLE ebook ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
|
||||
session.execute(
|
||||
'ALTER TABLE archive ADD COLUMN file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL UNIQUE')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE tag (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
color VARCHAR(7)
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE tag_file (
|
||||
tag_id INTEGER REFERENCES tag(id),
|
||||
file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE (tag_id, file_group_id)
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
INSERT INTO tag (name, color) VALUES
|
||||
('Automotive', '#353535'),
|
||||
('Clothing', '#9858C5'),
|
||||
('Comms', '#ECD71C'),
|
||||
('Cooking', '#00A04A'),
|
||||
('Electronics', '#1b731f'),
|
||||
('Entertainment', '#981473'),
|
||||
('Favorite', '#000000'),
|
||||
('Fire Safety', '#d16711'),
|
||||
('Firearms', '#6c6c6c'),
|
||||
('First Aid', '#C40405'),
|
||||
('Food Preservation', '#676816'),
|
||||
('Food', '#2F8055'),
|
||||
('Fuel', '#c47fb3'),
|
||||
('Garden', '#3E8B22'),
|
||||
('Heating', '#ff5400'),
|
||||
('Husbandry','#B58039'),
|
||||
('Hygiene', '#adadad'),
|
||||
('Knots', '#9d6a4c'),
|
||||
('Light', '#EEEEEE'),
|
||||
('Mechanical', '#7A7A7A'),
|
||||
('Medical', '#4a869c'),
|
||||
('Navigation', '#1ab3ce'),
|
||||
('Nuclear', '#0dff00'),
|
||||
('Pests', '#050068'),
|
||||
('Power', '#D837FF'),
|
||||
('Primitive', '#A18A62'),
|
||||
('Radio', '#134cbb'),
|
||||
('Schooling', '#6d752e'),
|
||||
('Security', '#333333'),
|
||||
('Shelter', '#625641'),
|
||||
('Social', '#490c73'),
|
||||
('Tools', '#f0d730'),
|
||||
('WROL', '#4700ab'),
|
||||
('Water', '#291BC2')
|
||||
''')
|
||||
|
||||
session.execute('UPDATE wrolpi_flag SET refresh_complete = false')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.file_group OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.tag OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.tag_file OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS tag_file')
|
||||
session.execute('DROP TABLE IF EXISTS tag')
|
||||
|
||||
session.execute('ALTER TABLE video DROP COLUMN file_group_id')
|
||||
session.execute('ALTER TABLE ebook DROP COLUMN file_group_id')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN file_group_id')
|
||||
|
||||
session.execute('DROP TABLE file_group')
|
||||
|
||||
session.execute('''
|
||||
create table file
|
||||
(
|
||||
path text not null primary key,
|
||||
idempotency timestamp with time zone,
|
||||
indexed boolean default false,
|
||||
mimetype text,
|
||||
suffix text,
|
||||
model text,
|
||||
associated boolean default false,
|
||||
modification_datetime timestamp with time zone,
|
||||
size bigint,
|
||||
title text,
|
||||
a_text text,
|
||||
b_text text,
|
||||
c_text text,
|
||||
d_text text,
|
||||
textsearch tsvector generated always as ((
|
||||
((setweight(to_tsvector('english'::regconfig, COALESCE(a_text, ''::text)), 'A'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, ''::text)), 'B'::"char")) ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, ''::text)), 'C'::"char")) ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, ''::text)), 'D'::"char"))) stored,
|
||||
full_stem text
|
||||
);
|
||||
|
||||
alter table file
|
||||
owner to postgres;
|
||||
|
||||
create index file_mimetype_idx
|
||||
on file (mimetype);
|
||||
|
||||
create index file_textsearch_idx
|
||||
on file using gin (textsearch);
|
||||
|
||||
create index file_modification_datetime_idx
|
||||
on file (modification_datetime);
|
||||
|
||||
create index file_size_idx
|
||||
on file (size);
|
||||
|
||||
create index file_model_idx
|
||||
on file (model);
|
||||
|
||||
create index file_idempotency_idx
|
||||
on file (idempotency);
|
||||
|
||||
create index file_suffix_idx
|
||||
on file (suffix);
|
||||
|
||||
create index file_full_stem_idx
|
||||
on file (full_stem);
|
||||
''')
|
||||
|
||||
session.execute('DELETE FROM video') # noqa
|
||||
session.execute('ALTER TABLE video ADD COLUMN caption_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE video ADD COLUMN censored BOOLEAN DEFAULT FALSE')
|
||||
session.execute('ALTER TABLE video ADD COLUMN favorite TIMESTAMPTZ')
|
||||
session.execute('ALTER TABLE video ADD COLUMN info_json_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE video ADD COLUMN poster_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE video ADD COLUMN video_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE video ADD COLUMN modification_datetime TIMESTAMPTZ')
|
||||
session.execute('ALTER TABLE video ADD COLUMN size BIGINT')
|
||||
session.execute('ALTER TABLE video ADD COLUMN title TEXT')
|
||||
session.execute('ALTER TABLE video ADD COLUMN validated BOOLEAN DEFAULT FALSE')
|
||||
|
||||
session.execute('DELETE FROM archive') # noqa
|
||||
session.execute('ALTER TABLE archive ADD COLUMN readability_json_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN readability_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN readability_txt_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN screenshot_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN singlefile_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN title TEXT')
|
||||
session.execute('ALTER TABLE archive ADD COLUMN validated BOOLEAN DEFAULT FALSE')
|
||||
|
||||
session.execute('DELETE FROM ebook') # noqa
|
||||
session.execute('ALTER TABLE ebook ADD COLUMN cover_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE ebook ADD COLUMN ebook_path TEXT REFERENCES file(path)')
|
||||
session.execute('ALTER TABLE ebook ADD COLUMN title TEXT')
|
||||
|
||||
session.execute('UPDATE wrolpi_flag SET refresh_complete = false')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.file OWNER TO wrolpi')
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""Video validated.
|
||||
|
||||
Revision ID: 3c33b1dae069
|
||||
Revises: f2c6326cad29
|
||||
Create Date: 2022-01-13 16:14:23.409242
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '3c33b1dae069'
|
||||
down_revision = 'f2c6326cad29'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE video ADD COLUMN validated BOOLEAN DEFAULT FALSE')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS validated_poster')
|
||||
# Generating posters is slow, and may delete files.
|
||||
session.execute('ALTER TABLE channel ALTER COLUMN generate_posters SET DEFAULT false')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS validated')
|
||||
session.execute('ALTER TABLE video ADD COLUMN validated_poster BOOLEAN DEFAULT FALSE')
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""Download.sub_downloader.
|
||||
|
||||
Revision ID: 467235201c70
|
||||
Revises: 1a70196c05bb
|
||||
Create Date: 2022-05-20 11:51:58.064020
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '467235201c70'
|
||||
down_revision = '1a70196c05bb'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
session = Session(bind=op.get_bind())
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS sub_downloader TEXT')
|
||||
|
||||
|
||||
def downgrade():
|
||||
session = Session(bind=op.get_bind())
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS sub_downloader')
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
"""Reindex archives.
|
||||
|
||||
Revision ID: 4b54ad052e2e
|
||||
Revises: 6b0b249a4f81
|
||||
Create Date: 2023-10-20 13:28:55.517086
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '4b54ad052e2e'
|
||||
down_revision = '6b0b249a4f81'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Archives must be re-indexed because of the previous revision.
|
||||
session.execute("UPDATE file_group SET indexed=false WHERE model='archive'")
|
||||
session.execute("DELETE FROM archive") # noqa
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""Drop Channel.skip_download_videos.
|
||||
|
||||
Revision ID: 4c8da5502c8a
|
||||
Revises: 8bdb48a486b9
|
||||
Create Date: 2023-09-29 14:45:42.999526
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '4c8da5502c8a'
|
||||
down_revision = '8bdb48a486b9'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS skip_download_videos')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS skip_download_videos TEXT[]')
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
"""file_group.effective_date trigger
|
||||
|
||||
Revision ID: 4f03b9548f6e
|
||||
Revises: 54c19aa847da
|
||||
Create Date: 2025-08-17 10:49:35.754548
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '4f03b9548f6e'
|
||||
down_revision = '54c19aa847da'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Create trigger function
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_effective_datetime()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
BEGIN
|
||||
NEW.effective_datetime = COALESCE(NEW.published_datetime, NEW.download_datetime);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
|
||||
# Create trigger
|
||||
op.execute("""
|
||||
CREATE TRIGGER file_group_effective_datetime_trigger
|
||||
BEFORE INSERT OR UPDATE OF published_datetime, download_datetime
|
||||
ON file_group
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_effective_datetime();
|
||||
""")
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Drop trigger
|
||||
op.execute("DROP TRIGGER IF EXISTS file_group_effective_datetime_trigger ON file_group")
|
||||
|
||||
# Drop trigger function
|
||||
op.execute("DROP FUNCTION IF EXISTS update_effective_datetime")
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""Drop URLs.
|
||||
|
||||
Revision ID: 53f6780aa599
|
||||
Revises: 31896cc29b71
|
||||
Create Date: 2022-01-03 12:25:51.575505
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '53f6780aa599'
|
||||
down_revision = '31896cc29b71'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
urls = {i['id']: i['url'] for i in session.execute(text('SELECT id, url FROM url'))}
|
||||
archives = {i['id']: i['url_id'] for i in session.execute(text('SELECT id, url_id FROM archive'))}
|
||||
|
||||
session.execute(text('ALTER TABLE archive ADD COLUMN url TEXT'))
|
||||
|
||||
for archive_id, url_id in archives.items():
|
||||
session.execute(
|
||||
text('UPDATE archive SET url = :url WHERE id = :id'),
|
||||
{'url': urls[url_id], 'id': archive_id}
|
||||
)
|
||||
|
||||
session.execute(text('ALTER TABLE archive DROP COLUMN url_id'))
|
||||
session.execute(text('DROP TABLE url'))
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
"""Channel summary columns, triggers and indexes.
|
||||
|
||||
Revision ID: 54c19aa847da
|
||||
Revises: 158973fda20f
|
||||
Create Date: 2025-08-16 11:45:48.190427
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '54c19aa847da'
|
||||
down_revision = '158973fda20f'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Add columns
|
||||
op.execute("""
|
||||
ALTER TABLE channel
|
||||
ADD COLUMN video_count INTEGER DEFAULT 0 NOT NULL,
|
||||
ADD COLUMN total_size BIGINT DEFAULT 0 NOT NULL,
|
||||
ADD COLUMN minimum_frequency INTEGER;
|
||||
""")
|
||||
|
||||
# Populate columns
|
||||
op.execute("""
|
||||
UPDATE channel
|
||||
SET video_count = (SELECT COUNT(*)
|
||||
FROM video
|
||||
WHERE channel_id = channel.id),
|
||||
total_size = (SELECT COALESCE(SUM(fg.size), 0)::BIGINT
|
||||
FROM video v
|
||||
LEFT JOIN file_group fg ON v.file_group_id = fg.id
|
||||
WHERE v.channel_id = channel.id),
|
||||
minimum_frequency = (SELECT MIN(frequency)
|
||||
FROM download
|
||||
WHERE channel_id = channel.id);
|
||||
""")
|
||||
|
||||
# Create trigger functions
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_channel_video_count()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
BEGIN
|
||||
UPDATE channel
|
||||
SET video_count = (SELECT COUNT(*) FROM video WHERE channel_id = NEW.channel_id)
|
||||
WHERE id = NEW.channel_id;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_channel_size()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
BEGIN
|
||||
UPDATE channel
|
||||
SET total_size = (SELECT COALESCE(SUM(fg.size), 0)::BIGINT
|
||||
FROM video v
|
||||
LEFT JOIN file_group fg ON v.file_group_id = fg.id
|
||||
WHERE v.channel_id = NEW.id)
|
||||
WHERE id = NEW.id;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
BEGIN
|
||||
UPDATE channel
|
||||
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE channel_id = NEW.channel_id)
|
||||
WHERE id = NEW.channel_id;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
|
||||
# Create triggers
|
||||
op.execute("""
|
||||
CREATE TRIGGER video_count_trigger
|
||||
AFTER INSERT OR DELETE
|
||||
ON video
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_channel_video_count();
|
||||
""")
|
||||
op.execute("""
|
||||
CREATE TRIGGER video_size_trigger
|
||||
AFTER INSERT OR DELETE
|
||||
ON video
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_channel_size();
|
||||
""")
|
||||
op.execute("""
|
||||
CREATE TRIGGER file_group_size_trigger
|
||||
AFTER UPDATE OF size
|
||||
ON file_group
|
||||
FOR EACH ROW
|
||||
WHEN (OLD.size IS DISTINCT FROM NEW.size)
|
||||
EXECUTE FUNCTION update_channel_size();
|
||||
""")
|
||||
op.execute("""
|
||||
CREATE TRIGGER download_frequency_trigger
|
||||
AFTER INSERT OR UPDATE OF frequency OR DELETE
|
||||
ON download
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_channel_minimum_frequency();
|
||||
""")
|
||||
|
||||
# Create indexes
|
||||
op.execute("""
|
||||
CREATE INDEX channel_video_count_idx ON channel (video_count);
|
||||
CREATE INDEX channel_total_size_idx ON channel (total_size);
|
||||
CREATE INDEX channel_minimum_frequency_idx ON channel (minimum_frequency);
|
||||
""")
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Drop triggers
|
||||
op.execute("DROP TRIGGER IF EXISTS video_count_trigger ON video;")
|
||||
op.execute("DROP TRIGGER IF EXISTS video_size_trigger ON video;")
|
||||
op.execute("DROP TRIGGER IF EXISTS file_group_size_trigger ON file_group;")
|
||||
op.execute("DROP TRIGGER IF EXISTS download_frequency_trigger ON download;")
|
||||
|
||||
# Drop trigger functions
|
||||
op.execute("DROP FUNCTION IF EXISTS update_channel_video_count;")
|
||||
op.execute("DROP FUNCTION IF EXISTS update_channel_size;")
|
||||
op.execute("DROP FUNCTION IF EXISTS update_channel_minimum_frequency;")
|
||||
|
||||
# Drop indexes
|
||||
op.execute("DROP INDEX IF EXISTS channel_minimum_frequency_idx;")
|
||||
op.execute("DROP INDEX IF EXISTS channel_total_size_idx;")
|
||||
op.execute("DROP INDEX IF EXISTS channel_video_count_idx;")
|
||||
|
||||
# Drop columns
|
||||
op.execute("""
|
||||
ALTER TABLE channel
|
||||
DROP COLUMN IF EXISTS minimum_frequency,
|
||||
DROP COLUMN IF EXISTS total_size,
|
||||
DROP COLUMN IF EXISTS video_count;
|
||||
""")
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
"""add_archive_collection_id_index
|
||||
|
||||
Revision ID: 577e414f1bc4
|
||||
Revises: f38cfe4b5cb2
|
||||
Create Date: 2025-12-07 17:25:27.809843
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '577e414f1bc4'
|
||||
down_revision = 'f38cfe4b5cb2'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Add index on archive.collection_id to speed up GROUP BY queries
|
||||
# and per-collection lookups. This index was missing and causing
|
||||
# full table scans on the archive table (115k+ rows).
|
||||
op.create_index('idx_archive_collection_id', 'archive', ['collection_id'])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index('idx_archive_collection_id', table_name='archive')
|
||||
|
|
@ -1,141 +0,0 @@
|
|||
"""
|
||||
Migrates Video.duration/.url/.viewed/Archive.url to FileGroup.length/.url/.viewed.
|
||||
|
||||
Revision ID: 5ad9efbf3570
|
||||
Revises: 4b54ad052e2e
|
||||
Create Date: 2023-10-24 09:25:33.818373
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '5ad9efbf3570'
|
||||
down_revision = '4b54ad052e2e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN length INT DEFAULT 0')
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET length=video.duration
|
||||
FROM video
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
# Migrate Archive.url and Video.url to FileGroup.url.
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN url TEXT')
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET url=archive.url
|
||||
FROM archive
|
||||
WHERE file_group.id = archive.file_group_id
|
||||
''')
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET url=video.url
|
||||
FROM video
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
# Migrate Archive.archive_datetime to FileGroup.download_datetime.
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN download_datetime TIMESTAMP WITH TIME ZONE')
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET download_datetime=archive.archive_datetime
|
||||
FROM archive
|
||||
WHERE file_group.id = archive.file_group_id
|
||||
''')
|
||||
|
||||
# Migrate Video.viewed to FileGroup.viewed.
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN viewed TIMESTAMP WITH TIME ZONE')
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET viewed=video.viewed
|
||||
FROM video
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS archive_archive_datetime_idx')
|
||||
session.execute('DROP INDEX IF EXISTS archive_url_idx')
|
||||
session.execute('DROP INDEX IF EXISTS video_url_idx')
|
||||
session.execute('DROP INDEX IF EXISTS video_viewed_idx')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS archive_datetime')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS url')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS url')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS viewed')
|
||||
|
||||
session.execute('CREATE INDEX file_group_download_datetime_idx ON file_group(download_datetime)')
|
||||
session.execute('CREATE INDEX file_group_length_idx ON file_group(length)')
|
||||
session.execute('CREATE INDEX file_group_url_idx ON file_group(url)')
|
||||
session.execute('CREATE INDEX file_group_viewed_idx ON file_group(viewed)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS duration INTEGER')
|
||||
session.execute('''
|
||||
UPDATE video
|
||||
SET duration=file_group.length
|
||||
FROM file_group
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS url TEXT')
|
||||
session.execute('''
|
||||
UPDATE video
|
||||
SET url=file_group.url
|
||||
FROM file_group
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
# Video has `viewed`, Archive does not.
|
||||
session.execute('ALTER TABLE video ADD COLUMN IF NOT EXISTS viewed TIMESTAMP WITH TIME ZONE')
|
||||
session.execute('''
|
||||
UPDATE video
|
||||
SET url=file_group.url
|
||||
FROM file_group
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('ALTER TABLE archive ADD COLUMN IF NOT EXISTS url TEXT')
|
||||
session.execute('''
|
||||
UPDATE archive
|
||||
SET url=file_group.url
|
||||
FROM file_group
|
||||
WHERE file_group.id = archive.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('ALTER TABLE archive ADD COLUMN IF NOT EXISTS archive_datetime TIMESTAMP WITH TIME ZONE')
|
||||
session.execute('''
|
||||
UPDATE archive
|
||||
SET archive_datetime=file_group.download_datetime
|
||||
FROM file_group
|
||||
WHERE file_group.id = archive.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS download_datetime')
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS length')
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS url')
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS viewed')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_download_datetime_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_length_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_url_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_viewed_idx')
|
||||
|
||||
session.execute('CREATE INDEX IF NOT EXISTS archive_archive_datetime_idx ON archive(archive_datetime)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS archive_url_idx ON archive(url)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS video_duration_idx ON video(duration)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS video_url_idx ON video(url)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS video_viewed_idx ON video(viewed)')
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
"""Archive textsearch coalesce fix.
|
||||
|
||||
Revision ID: 62acddc55091
|
||||
Revises: 0400005fe2dc
|
||||
Create Date: 2022-01-04 11:11:29.100317
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '62acddc55091'
|
||||
down_revision = '0400005fe2dc'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS archive_textsearch_idx')
|
||||
session.execute('ALTER TABLE archive DROP COLUMN IF EXISTS textsearch')
|
||||
session.execute('''
|
||||
ALTER TABLE archive
|
||||
ADD COLUMN textsearch tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(title, '')), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(contents, '')), 'D')
|
||||
) STORED
|
||||
''')
|
||||
session.execute('CREATE INDEX archive_textsearch_idx ON archive USING GIN(textsearch)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
# The upgrade is backwards compatible.
|
||||
pass
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
"""File stem and directory
|
||||
|
||||
Revision ID: 6489cd50c889
|
||||
Revises: 13542a230c78
|
||||
Create Date: 2022-12-04 15:39:55.485355
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '6489cd50c889'
|
||||
down_revision = '13542a230c78'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file ADD COLUMN directory TEXT')
|
||||
session.execute('ALTER TABLE file ADD COLUMN full_stem TEXT')
|
||||
session.execute('ALTER TABLE file ADD COLUMN stem TEXT')
|
||||
session.execute('CREATE INDEX file_directory_idx ON file(directory)')
|
||||
session.execute('CREATE INDEX file_full_stem_idx ON file(full_stem)')
|
||||
session.execute('CREATE INDEX file_stem_idx ON file(stem)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_directory_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_full_stem_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_stem_idx')
|
||||
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS directory')
|
||||
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS full_stem')
|
||||
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS stem')
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
"""Create collection table for domain collections
|
||||
|
||||
Revision ID: 66407d145b76
|
||||
Revises: 4f03b9548f6e
|
||||
Create Date: 2025-10-26 10:57:16.462524
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '66407d145b76'
|
||||
down_revision = '4f03b9548f6e'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Step 1: Create new collection table (keeping channel table separate for now)
|
||||
op.create_table(
|
||||
'collection',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('name', sa.String(), nullable=False),
|
||||
sa.Column('description', sa.Text(), nullable=True),
|
||||
sa.Column('kind', sa.String(), nullable=False),
|
||||
sa.Column('directory', sa.Text(), nullable=True),
|
||||
sa.Column('tag_id', sa.Integer(), nullable=True),
|
||||
sa.Column('created_date', sa.DateTime(), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column('item_count', sa.Integer(), nullable=False, server_default='0'),
|
||||
sa.Column('total_size', sa.Integer(), nullable=False, server_default='0'),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name='collection_tag_id_fkey'),
|
||||
sa.UniqueConstraint('directory', name='uq_collection_directory')
|
||||
)
|
||||
|
||||
# Create indexes for collection
|
||||
op.create_index('idx_collection_kind', 'collection', ['kind'], unique=False)
|
||||
op.create_index('idx_collection_item_count', 'collection', ['item_count'], unique=False)
|
||||
op.create_index('idx_collection_total_size', 'collection', ['total_size'], unique=False)
|
||||
|
||||
# Step 2: Create collection_item junction table
|
||||
op.create_table(
|
||||
'collection_item',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('collection_id', sa.Integer(), nullable=False),
|
||||
sa.Column('file_group_id', sa.Integer(), nullable=False),
|
||||
sa.Column('position', sa.Integer(), nullable=False, server_default='0'),
|
||||
sa.Column('added_date', sa.DateTime(), server_default=sa.func.now(), nullable=False),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['file_group_id'], ['file_group.id'], ondelete='CASCADE'),
|
||||
sa.UniqueConstraint('collection_id', 'file_group_id', name='uq_collection_file_group')
|
||||
)
|
||||
|
||||
# Create indexes for collection_item
|
||||
op.create_index('idx_collection_item_collection_id', 'collection_item', ['collection_id'], unique=False)
|
||||
op.create_index('idx_collection_item_file_group_id', 'collection_item', ['file_group_id'], unique=False)
|
||||
op.create_index('idx_collection_item_position', 'collection_item', ['position'], unique=False)
|
||||
op.create_index('idx_collection_item_collection_position', 'collection_item', ['collection_id', 'position'], unique=False)
|
||||
|
||||
# Ensure table owner in non-docker environments
|
||||
if not DOCKERIZED:
|
||||
session.execute(sa.text('ALTER TABLE public.collection OWNER TO wrolpi'))
|
||||
session.execute(sa.text('ALTER TABLE public.collection_item OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Drop collection_item table and its indexes
|
||||
op.drop_index('idx_collection_item_collection_position', table_name='collection_item')
|
||||
op.drop_index('idx_collection_item_position', table_name='collection_item')
|
||||
op.drop_index('idx_collection_item_file_group_id', table_name='collection_item')
|
||||
op.drop_index('idx_collection_item_collection_id', table_name='collection_item')
|
||||
op.drop_table('collection_item')
|
||||
|
||||
# Drop collection table and its indexes
|
||||
op.drop_index('idx_collection_total_size', table_name='collection')
|
||||
op.drop_index('idx_collection_item_count', table_name='collection')
|
||||
op.drop_index('idx_collection_kind', table_name='collection')
|
||||
op.drop_table('collection')
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
"""FileGroup.effective_datetime
|
||||
|
||||
Revision ID: 69e9754d1a63
|
||||
Revises: f7229a67e333
|
||||
Create Date: 2025-08-16 10:28:42.661108
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '69e9754d1a63'
|
||||
down_revision = 'f7229a67e333'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('ALTER TABLE file_group ADD COLUMN IF NOT EXISTS effective_datetime TIMESTAMPTZ')
|
||||
|
||||
# Populating effective_datetime for existing rows
|
||||
op.execute("""
|
||||
UPDATE file_group
|
||||
SET effective_datetime = COALESCE(published_datetime, download_datetime)
|
||||
WHERE published_datetime IS NOT NULL
|
||||
OR download_datetime IS NOT NULL;
|
||||
""")
|
||||
|
||||
op.execute('CREATE INDEX IF NOT EXISTS file_group_effective_datetime_idx ON file_group(effective_datetime)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('DROP INDEX IF EXISTS file_group_effective_datetime_idx')
|
||||
op.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS effective_datetime')
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
"""Archive published_datetime
|
||||
|
||||
Revision ID: 6b0b249a4f81
|
||||
Revises: 4c8da5502c8a
|
||||
Create Date: 2023-10-20 12:54:55.879180
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '6b0b249a4f81'
|
||||
down_revision = '4c8da5502c8a'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN author TEXT')
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN published_datetime TIMESTAMP WITH TIME ZONE')
|
||||
session.execute('ALTER TABLE file_group ADD COLUMN published_modified_datetime TIMESTAMP WITH TIME ZONE')
|
||||
|
||||
# Migrate Video.upload_date to FileGroup.published_datetime.
|
||||
session.execute('''
|
||||
UPDATE file_group
|
||||
SET published_datetime=upload_date
|
||||
FROM video
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('CREATE INDEX IF NOT EXISTS file_group_author_idx ON file_group(author)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS file_group_published_datetime_idx ON file_group(published_datetime)')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS file_group_published_modified_datetime_idx'
|
||||
' ON file_group(published_modified_datetime)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Migrate FileGroup.published_datetime to Video.upload_date.
|
||||
session.execute('''
|
||||
UPDATE video
|
||||
SET upload_date=published_datetime
|
||||
FROM file_group
|
||||
WHERE file_group.id = video.file_group_id
|
||||
''')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_group_author_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_published_datetime_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_published_modified_datetime_idx')
|
||||
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS author')
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS published_datetime')
|
||||
session.execute('ALTER TABLE file_group DROP COLUMN IF EXISTS published_modified_datetime')
|
||||
|
|
@ -1,218 +0,0 @@
|
|||
"""Files rework. Unifying files for Videos and Archives.
|
||||
|
||||
Revision ID: 70db5ba8763d
|
||||
Revises: aac864072193
|
||||
Create Date: 2022-07-27 20:11:30.990396
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '70db5ba8763d'
|
||||
down_revision = 'aac864072193'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Recreate the file table. Use file.path as the primary key.
|
||||
session.execute('DROP TABLE IF EXISTS file')
|
||||
session.execute('''CREATE TABLE file (
|
||||
path TEXT PRIMARY KEY,
|
||||
idempotency TIMESTAMP WITH TIME ZONE,
|
||||
indexed BOOLEAN DEFAULT FALSE,
|
||||
mimetype TEXT,
|
||||
suffix TEXT,
|
||||
model TEXT,
|
||||
associated BOOLEAN DEFAULT FALSE,
|
||||
modification_datetime TIMESTAMP WITH TIME ZONE,
|
||||
size BIGINT,
|
||||
title TEXT,
|
||||
a_text TEXT,
|
||||
b_text TEXT,
|
||||
c_text TEXT,
|
||||
d_text TEXT,
|
||||
textsearch tsvector GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
|
||||
) STORED
|
||||
)''')
|
||||
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
|
||||
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')
|
||||
session.execute('CREATE INDEX file_modification_datetime_idx ON file USING BTREE(modification_datetime)')
|
||||
session.execute('CREATE INDEX file_size_idx ON file USING BTREE(size)')
|
||||
session.execute('CREATE INDEX file_model_idx ON file USING BTREE(model)')
|
||||
session.execute('CREATE INDEX file_idempotency_idx ON file USING BTREE(idempotency)')
|
||||
session.execute('CREATE INDEX file_suffix_idx ON file USING BTREE(suffix)')
|
||||
|
||||
# Videos are fundamentally different, replace the video table.
|
||||
session.execute('DROP TABLE IF EXISTS video')
|
||||
session.execute('''
|
||||
CREATE TABLE video (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
title TEXT,
|
||||
source_id TEXT,
|
||||
channel_id INTEGER REFERENCES channel(id),
|
||||
upload_date TIMESTAMPTZ,
|
||||
duration INTEGER,
|
||||
favorite TIMESTAMPTZ,
|
||||
size BIGINT,
|
||||
viewed TIMESTAMPTZ,
|
||||
view_count INTEGER,
|
||||
url TEXT,
|
||||
censored BOOLEAN DEFAULT FALSE,
|
||||
validated BOOLEAN DEFAULT FALSE,
|
||||
modification_datetime TIMESTAMPTZ,
|
||||
video_path TEXT REFERENCES file(path),
|
||||
info_json_path TEXT REFERENCES file(path),
|
||||
poster_path TEXT REFERENCES file(path),
|
||||
caption_path TEXT REFERENCES file(path)
|
||||
)''')
|
||||
session.execute('''
|
||||
CREATE INDEX video_caption_path_idx ON video(caption_path);
|
||||
CREATE INDEX video_info_json_path_idx ON video(info_json_path);
|
||||
CREATE INDEX video_poster_path_idx ON video(poster_path);
|
||||
CREATE INDEX video_video_path_idx ON video(video_path);
|
||||
CREATE INDEX video_url_idx ON video(url);
|
||||
CREATE INDEX video_channel_id_idx ON video(channel_id);
|
||||
''')
|
||||
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS idempotency')
|
||||
|
||||
session.execute('DELETE FROM archive') # noqa
|
||||
session.execute('DELETE FROM domains') # noqa
|
||||
session.execute('DROP TABLE IF EXISTS archive')
|
||||
session.execute('DROP TABLE IF EXISTS domains')
|
||||
session.execute('''
|
||||
CREATE TABLE domains (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain TEXT NOT NULL,
|
||||
directory TEXT
|
||||
)''')
|
||||
session.execute('''
|
||||
CREATE TABLE archive (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain_id INTEGER REFERENCES domains(id),
|
||||
title TEXT,
|
||||
archive_datetime timestamptz,
|
||||
url TEXT,
|
||||
validated BOOLEAN DEFAULT FALSE,
|
||||
singlefile_path TEXT REFERENCES file(path),
|
||||
readability_path TEXT REFERENCES file(path),
|
||||
readability_json_path TEXT REFERENCES file(path),
|
||||
readability_txt_path TEXT REFERENCES file(path),
|
||||
screenshot_path TEXT REFERENCES file(path)
|
||||
)''')
|
||||
session.execute('''
|
||||
CREATE INDEX archive_singlefile_path ON archive(singlefile_path);
|
||||
CREATE INDEX archive_readability_path ON archive(readability_path);
|
||||
CREATE INDEX archive_readability_json_path ON archive(readability_json_path);
|
||||
CREATE INDEX archive_readability_txt_path ON archive(readability_txt_path);
|
||||
CREATE INDEX archive_screenshot_path ON archive(screenshot_path);
|
||||
''')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
DROP INDEX IF EXISTS archive_singlefile_path;
|
||||
DROP INDEX IF EXISTS archive_readability_path;
|
||||
DROP INDEX IF EXISTS archive_readability_json_path;
|
||||
DROP INDEX IF EXISTS archive_readability_txt_path;
|
||||
DROP INDEX IF EXISTS archive_screenshot_path;
|
||||
''')
|
||||
|
||||
session.execute('DELETE FROM archive') # noqa
|
||||
session.execute('''
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS singlefile_path;
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS readability_path;
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS readability_json_path;
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS readability_txt_path;
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS screenshot_path;
|
||||
ALTER TABLE archive DROP COLUMN IF EXISTS associated;
|
||||
''')
|
||||
session.execute('''
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS singlefile_path TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_path TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_json_path TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_txt_path TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS screenshot_path TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS contents TEXT;
|
||||
ALTER TABLE archive ADD COLUMN IF NOT EXISTS status TEXT;
|
||||
''')
|
||||
session.execute('ALTER TABLE archive ALTER COLUMN domain_id SET NOT NULL')
|
||||
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS idempotency TEXT')
|
||||
|
||||
session.execute('DELETE FROM video') # noqa
|
||||
session.execute('''
|
||||
ALTER TABLE video DROP COLUMN IF EXISTS caption_path;
|
||||
ALTER TABLE video DROP COLUMN IF EXISTS info_json_path;
|
||||
ALTER TABLE video DROP COLUMN IF EXISTS poster_path;
|
||||
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
|
||||
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
|
||||
''')
|
||||
session.execute('''
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS idempotency TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption_path TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS description_path TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS info_json_path TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS poster_path TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS video_path TEXT;
|
||||
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption TEXT;
|
||||
''')
|
||||
session.execute('''
|
||||
DROP INDEX IF EXISTS video_caption_path_idx;
|
||||
DROP INDEX IF EXISTS video_info_json_path_idx;
|
||||
DROP INDEX IF EXISTS video_poster_path_idx;
|
||||
DROP INDEX IF EXISTS video_video_path_idx;
|
||||
DROP INDEX IF EXISTS video_url_idx;
|
||||
DROP INDEX IF EXISTS video_channel_id_idx;
|
||||
''')
|
||||
session.execute('''
|
||||
ALTER TABLE video
|
||||
ADD COLUMN textsearch tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, title), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, caption), 'D')
|
||||
) STORED
|
||||
''')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS video_textsearch_idx ON video USING GIN(textsearch)')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_mimetype_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_textsearch_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_modification_datetime_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_size_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_model_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_idempotency_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_suffix_idx')
|
||||
|
||||
# Restore the old file table.
|
||||
session.execute('DROP TABLE IF EXISTS file')
|
||||
session.execute('''CREATE TABLE file (
|
||||
id SERIAL PRIMARY KEY,
|
||||
path TEXT,
|
||||
title TEXT,
|
||||
mimetype TEXT,
|
||||
size BIGINT,
|
||||
modification_datetime TIMESTAMP WITH TIME ZONE,
|
||||
idempotency TEXT DEFAULT NULL,
|
||||
textsearch tsvector GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, title), 'A'::"char")
|
||||
) STORED
|
||||
)''')
|
||||
session.execute('CREATE UNIQUE INDEX file_path_idx ON file USING BTREE(path)')
|
||||
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
|
||||
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
"""Ebook
|
||||
|
||||
Revision ID: 717f36a4da94
|
||||
Revises: d0bbe45e56c3
|
||||
Create Date: 2022-12-09 15:32:32.228539
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '717f36a4da94'
|
||||
down_revision = 'd0bbe45e56c3'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
DROP INDEX IF EXISTS ebook_title_idx;
|
||||
DROP INDEX IF EXISTS ebook_size_idx;
|
||||
DROP INDEX IF EXISTS ebook_creator_idx;
|
||||
DROP INDEX IF EXISTS ebook_ebook_path_idx;
|
||||
DROP INDEX IF EXISTS ebook_cover_path_idx;
|
||||
DROP TABLE IF EXISTS ebook;
|
||||
''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE ebook (
|
||||
id SERIAL PRIMARY KEY,
|
||||
size INTEGER,
|
||||
title TEXT,
|
||||
creator TEXT,
|
||||
ebook_path TEXT REFERENCES file(path) ON DELETE CASCADE,
|
||||
cover_path TEXT REFERENCES file(path) ON DELETE CASCADE
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.ebook OWNER TO wrolpi')
|
||||
|
||||
session.execute('CREATE INDEX ebook_title_idx ON ebook(title)')
|
||||
session.execute('CREATE INDEX ebook_size_idx ON ebook(size)')
|
||||
session.execute('CREATE INDEX ebook_creator_idx ON ebook(creator)')
|
||||
session.execute('CREATE INDEX ebook_ebook_path_idx ON ebook(ebook_path)')
|
||||
session.execute('CREATE INDEX ebook_cover_path_idx ON ebook(cover_path)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
DROP INDEX IF EXISTS ebook_title_idx;
|
||||
DROP INDEX IF EXISTS ebook_size_idx;
|
||||
DROP INDEX IF EXISTS ebook_creator_idx;
|
||||
DROP INDEX IF EXISTS ebook_ebook_path_idx;
|
||||
DROP INDEX IF EXISTS ebook_cover_path_idx;
|
||||
DROP TABLE IF EXISTS ebook;
|
||||
''')
|
||||
|
|
@ -1,67 +0,0 @@
|
|||
"""Zim
|
||||
|
||||
Revision ID: 751aecf46b88
|
||||
Revises: 75d34a2b442d
|
||||
Create Date: 2023-06-04 19:48:41.305111
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '751aecf46b88'
|
||||
down_revision = '75d34a2b442d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE zim (
|
||||
id SERIAL PRIMARY KEY,
|
||||
path TEXT NOT NULL UNIQUE,
|
||||
file_group_id BIGINT REFERENCES file_group(id) ON DELETE CASCADE NOT NULL
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE tag_zim (
|
||||
tag_id INTEGER REFERENCES tag(id),
|
||||
zim_id INTEGER REFERENCES zim(id) ON DELETE CASCADE,
|
||||
zim_entry TEXT NOT NULL,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE (tag_id, zim_id, zim_entry)
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE zim_subscription (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
language TEXT NOT NULL,
|
||||
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
|
||||
)
|
||||
''')
|
||||
|
||||
# We want to index all Zim files using `zim_modeler`
|
||||
session.execute("UPDATE file_group SET indexed=false WHERE primary_path ILIKE '%.zim'")
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.zim OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.tag_zim OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.zim_subscription OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS zim_subscription')
|
||||
session.execute('DROP TABLE IF EXISTS tag_zim')
|
||||
session.execute('DROP TABLE IF EXISTS zim')
|
||||
session.execute("UPDATE file_group SET indexed=false WHERE primary_path ILIKE '%.zim'")
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Archive datetime index.
|
||||
|
||||
Revision ID: 75d34a2b442d
|
||||
Revises: f9928a3b6bc5
|
||||
Create Date: 2023-05-15 13:09:47.661940
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '75d34a2b442d'
|
||||
down_revision = 'f9928a3b6bc5'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('CREATE INDEX archive_datetime_idx ON archive(archive_datetime)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS archive_datetime_idx')
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
"""json_to_jsonb
|
||||
|
||||
Revision ID: 76add63ed225
|
||||
Revises: 26c5a62ec683
|
||||
Create Date: 2026-01-06 06:13:14.044016
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '76add63ed225'
|
||||
down_revision = '26c5a62ec683'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Convert FileGroup columns from JSON to JSONB
|
||||
session.execute(text('ALTER TABLE file_group ALTER COLUMN data TYPE jsonb USING data::jsonb'))
|
||||
session.execute(text('ALTER TABLE file_group ALTER COLUMN files TYPE jsonb USING files::jsonb'))
|
||||
|
||||
# Convert Video column from JSON to JSONB
|
||||
session.execute(text('ALTER TABLE video ALTER COLUMN ffprobe_json TYPE jsonb USING ffprobe_json::jsonb'))
|
||||
|
||||
# Convert Channel column from JSON to JSONB
|
||||
session.execute(text('ALTER TABLE channel ALTER COLUMN info_json TYPE jsonb USING info_json::jsonb'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Revert FileGroup columns from JSONB to JSON
|
||||
session.execute(text('ALTER TABLE file_group ALTER COLUMN data TYPE json USING data::json'))
|
||||
session.execute(text('ALTER TABLE file_group ALTER COLUMN files TYPE json USING files::json'))
|
||||
|
||||
# Revert Video column from JSONB to JSON
|
||||
session.execute(text('ALTER TABLE video ALTER COLUMN ffprobe_json TYPE json USING ffprobe_json::json'))
|
||||
|
||||
# Revert Channel column from JSONB to JSON
|
||||
session.execute(text('ALTER TABLE channel ALTER COLUMN info_json TYPE json USING info_json::json'))
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""Drop channel next_download
|
||||
|
||||
Revision ID: 84b6993a6091
|
||||
Revises: 8df411232c17
|
||||
Create Date: 2021-12-27 16:42:32.099796
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '84b6993a6091'
|
||||
down_revision = '8df411232c17'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS next_download')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS next_download DATE')
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""Video ffprobe json.
|
||||
|
||||
Revision ID: 8bdb48a486b9
|
||||
Revises: bf704baa185d
|
||||
Create Date: 2023-07-24 10:59:17.426100
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '8bdb48a486b9'
|
||||
down_revision = 'bf704baa185d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE video ADD COLUMN ffprobe_json JSON')
|
||||
# Re-index all videos so the ffprobe data is fetched.
|
||||
session.execute("UPDATE file_group SET indexed=false WHERE mimetype like 'video/%'")
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS ffprobe_json')
|
||||
|
|
@ -1,148 +0,0 @@
|
|||
"""Channel.downloads / Download.channel
|
||||
|
||||
Revision ID: 8d0d81bc9c34
|
||||
Revises: 00f11f309c53
|
||||
Create Date: 2024-06-25 22:13:23.885687
|
||||
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, Text, ForeignKey
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import Session, deferred, relationship
|
||||
from sqlalchemy.orm.collections import InstrumentedList
|
||||
|
||||
from wrolpi.common import ModelHelper
|
||||
from wrolpi.dates import TZDateTime
|
||||
from wrolpi.downloader import DownloadStatus
|
||||
from wrolpi.media_path import MediaPathType
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '8d0d81bc9c34'
|
||||
down_revision = '00f11f309c53'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
# The `Channel` model at the time of this migration.
|
||||
class MChannel(ModelHelper, Base):
|
||||
__tablename__ = 'channel'
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String)
|
||||
url = Column(String, unique=True)
|
||||
directory: pathlib.Path = Column(MediaPathType)
|
||||
generate_posters = Column(Boolean, default=False)
|
||||
calculate_duration = Column(Boolean, default=True)
|
||||
source_id = Column(String)
|
||||
refreshed = Column(Boolean, default=False)
|
||||
|
||||
info_json = deferred(Column(JSON))
|
||||
info_date = Column(Date)
|
||||
|
||||
downloads: InstrumentedList = relationship('MDownload', primaryjoin='MDownload.channel_id==MChannel.id')
|
||||
|
||||
@staticmethod
|
||||
def get_by_url(url: str, session: Session = None):
|
||||
if not url:
|
||||
raise RuntimeError('Must provide URL to get Channel')
|
||||
channel = session.query(MChannel).filter_by(url=url).one_or_none()
|
||||
return channel
|
||||
|
||||
def get_rss_url(self) -> str | None:
|
||||
if self.url and self.source_id and 'youtube.com' in self.url:
|
||||
return f'https://www.youtube.com/feeds/videos.xml?channel_id={self.source_id}'
|
||||
|
||||
|
||||
# The `Download` model at the time of this migration.
|
||||
class MDownload(ModelHelper, Base):
|
||||
__tablename__ = 'download'
|
||||
id = Column(Integer, primary_key=True)
|
||||
url = Column(String, nullable=False, unique=True)
|
||||
|
||||
attempts = Column(Integer, default=0)
|
||||
downloader = Column(Text)
|
||||
sub_downloader = Column(Text)
|
||||
error = Column(Text)
|
||||
frequency = Column(Integer)
|
||||
info_json = Column(JSONB)
|
||||
last_successful_download = Column(TZDateTime)
|
||||
location = Column(Text)
|
||||
next_download = Column(TZDateTime)
|
||||
settings = Column(JSONB)
|
||||
status = Column(String, default=DownloadStatus.new)
|
||||
|
||||
channel_id = Column(Integer, ForeignKey('channel.id'))
|
||||
channel = relationship('MChannel', primaryjoin='MDownload.channel_id==MChannel.id', back_populates='downloads')
|
||||
|
||||
|
||||
def _link_channel_and_downloads(session: Session):
|
||||
"""Associate each Download with its Channel via `Download.channel_id`.
|
||||
|
||||
This is a frozen copy of the linking logic as it existed when this migration was written. It deliberately does
|
||||
NOT import `modules.videos.lib.link_channel_and_downloads`; that function has since been rewritten for the
|
||||
Collections feature and now relies on `Download.collection_id` / `Download.destination`, neither of which exists at
|
||||
this point in the migration chain. A migration must remain frozen against the schema it targets.
|
||||
"""
|
||||
downloads = list(session.query(MDownload).all())
|
||||
downloads_by_url = {i.url: i for i in downloads}
|
||||
downloads_by_destination = {i.settings['destination']: i
|
||||
for i in downloads if 'destination' in (i.settings or dict())}
|
||||
channels = session.query(MChannel).all()
|
||||
|
||||
need_commit = False
|
||||
for channel in channels:
|
||||
directory = str(channel.directory)
|
||||
download = downloads_by_destination.get(directory) or downloads_by_url.get(channel.url)
|
||||
if download and not download.channel_id:
|
||||
download.channel_id = channel.id
|
||||
need_commit = True
|
||||
|
||||
# Get any Downloads for a Channel's RSS feed.
|
||||
rss_url = channel.get_rss_url()
|
||||
download = downloads_by_url.get(rss_url) if rss_url else None
|
||||
if download and not download.channel_id:
|
||||
download.channel_id = channel.id
|
||||
need_commit = True
|
||||
|
||||
# Associate any Download which shares a Channel's URL.
|
||||
for download in downloads:
|
||||
if download.url and not download.channel_id:
|
||||
channel = MChannel.get_by_url(download.url, session)
|
||||
if channel:
|
||||
download.channel_id = channel.id
|
||||
need_commit = True
|
||||
|
||||
if need_commit:
|
||||
session.commit()
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE download ADD CONSTRAINT download_url_unique UNIQUE (url)')
|
||||
session.execute('ALTER TABLE download ADD COLUMN channel_id INTEGER REFERENCES channel(id)')
|
||||
|
||||
_link_channel_and_downloads(session)
|
||||
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS match_regex')
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS download_frequency')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS channel_download')
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS channel_id')
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS channel_id')
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS match_regex TEXT')
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS download_frequency INTEGER')
|
||||
session.execute('ALTER TABLE download DROP CONSTRAINT IF EXISTS download_url_unique')
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
"""Video indexes
|
||||
|
||||
Revision ID: 8df411232c17
|
||||
Revises: 32d96eb9122d
|
||||
Create Date: 2021-12-24 14:42:02.365100
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '8df411232c17'
|
||||
down_revision = '32d96eb9122d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('CREATE INDEX video_favorite ON video (favorite)')
|
||||
session.execute('CREATE INDEX video_view_count ON video (view_count)')
|
||||
session.execute('CREATE INDEX video_validated_poster ON video (validated_poster)')
|
||||
session.execute('CREATE INDEX video_upload_date ON video (upload_date)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS video_favorite')
|
||||
session.execute('DROP INDEX IF EXISTS video_view_count')
|
||||
session.execute('DROP INDEX IF EXISTS video_validated_poster')
|
||||
session.execute('DROP INDEX IF EXISTS video_upload_date')
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""Remove file directory and stem.
|
||||
|
||||
Revision ID: 90ac16f9140a
|
||||
Revises: 717f36a4da94
|
||||
Create Date: 2022-12-16 09:58:32.238873
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '90ac16f9140a'
|
||||
down_revision = '717f36a4da94'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_directory_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_stem_idx')
|
||||
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS directory')
|
||||
session.execute('ALTER TABLE file DROP COLUMN IF EXISTS stem')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file ADD COLUMN directory TEXT')
|
||||
session.execute('ALTER TABLE file ADD COLUMN stem TEXT')
|
||||
session.execute('CREATE INDEX file_directory_idx ON file(directory)')
|
||||
session.execute('CREATE INDEX file_stem_idx ON file(stem)')
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""WROLPi flags
|
||||
|
||||
Revision ID: 930b9b7adb79
|
||||
Revises: 90ac16f9140a
|
||||
Create Date: 2022-12-20 21:02:28.975897
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '930b9b7adb79'
|
||||
down_revision = '90ac16f9140a'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''CREATE TABLE wrolpi_flag(
|
||||
id SERIAL PRIMARY KEY,
|
||||
refresh_complete BOOLEAN DEFAULT FALSE
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.wrolpi_flag OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS wrolpi_flag')
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
"""Remove inventories version.
|
||||
|
||||
Revision ID: 9ec4c765ef8d
|
||||
Revises: d4e446637f18
|
||||
Create Date: 2024-09-24 16:01:41.716017
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '9ec4c765ef8d'
|
||||
down_revision = 'd4e446637f18'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS inventories_version')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE inventories_version (
|
||||
version SERIAL PRIMARY KEY NOT NULL
|
||||
)
|
||||
''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
"""add_collection_file_format
|
||||
|
||||
Revision ID: a1b2c3d4e5f6
|
||||
Revises: 76add63ed225
|
||||
Create Date: 2026-01-23
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'a1b2c3d4e5f6'
|
||||
down_revision = '76add63ed225'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Add file_format column to collection table
|
||||
# This stores the file_name_format used when files were last organized
|
||||
op.add_column('collection', sa.Column('file_format', sa.String(), nullable=True))
|
||||
|
||||
# Set default values from current configs so existing collections don't need reorganization
|
||||
from modules.archive.lib import get_archive_downloader_config
|
||||
from modules.videos.lib import get_videos_downloader_config
|
||||
|
||||
connection = op.get_bind()
|
||||
|
||||
# Update domain collections with archive config's file_name_format
|
||||
archive_config = get_archive_downloader_config()
|
||||
if archive_config.file_name_format:
|
||||
connection.execute(
|
||||
sa.text("UPDATE collection SET file_format = :format WHERE kind = 'domain'"),
|
||||
{'format': archive_config.file_name_format}
|
||||
)
|
||||
|
||||
# Update channel collections with video config's file_name_format
|
||||
video_config = get_videos_downloader_config()
|
||||
if video_config.file_name_format:
|
||||
connection.execute(
|
||||
sa.text("UPDATE collection SET file_format = :format WHERE kind = 'channel'"),
|
||||
{'format': video_config.file_name_format}
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column('collection', 'file_format')
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""Drop map_file table.
|
||||
|
||||
PMTiles files are now scanned directly from disk; the map_file table is no longer needed.
|
||||
|
||||
Revision ID: a2b3c4d5e6f7
|
||||
Revises: e7f8a9b0c1d2
|
||||
Create Date: 2026-03-29
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'a2b3c4d5e6f7'
|
||||
down_revision = 'e7f8a9b0c1d2'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.drop_table('map_file')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('''CREATE TABLE map_file (
|
||||
id SERIAL PRIMARY KEY,
|
||||
path TEXT NOT NULL,
|
||||
imported BOOLEAN DEFAULT FALSE,
|
||||
size BIGINT
|
||||
)''')
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
"""Make collection_item polymorphic: file | zim | url items.
|
||||
|
||||
CollectionItem previously only referenced a FileGroup. Playlists (kind='playlist') also need to
|
||||
order Zim articles (zim_id + zim_entry, mirroring TagZimEntry) and arbitrary URLs (a link the WROLPi
|
||||
browser can open, e.g. a map location). This adds item_kind plus the zim/url/title columns, relaxes
|
||||
file_group_id to nullable, and adds per-kind unique constraints and a CHECK that exactly one kind's
|
||||
columns are populated. Existing rows default to item_kind='file'.
|
||||
|
||||
Revision ID: a7b1c2d3e4f5
|
||||
Revises: e6f7a8b9c0d1
|
||||
Create Date: 2026-06-08
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = 'a7b1c2d3e4f5'
|
||||
down_revision = 'e6f7a8b9c0d1'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
CHECK_SQL = (
|
||||
"(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL)"
|
||||
" OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL"
|
||||
" AND file_group_id IS NULL AND url IS NULL)"
|
||||
" OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)"
|
||||
)
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column('collection_item',
|
||||
sa.Column('item_kind', sa.String(), nullable=False, server_default='file'))
|
||||
op.add_column('collection_item', sa.Column('zim_id', sa.Integer(), nullable=True))
|
||||
op.add_column('collection_item', sa.Column('zim_entry', sa.Text(), nullable=True))
|
||||
op.add_column('collection_item', sa.Column('url', sa.Text(), nullable=True))
|
||||
op.add_column('collection_item', sa.Column('title', sa.Text(), nullable=True))
|
||||
# `file` is no longer the only kind, so a row need not have a FileGroup.
|
||||
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=True)
|
||||
op.create_foreign_key('collection_item_zim_id_fkey', 'collection_item', 'zim',
|
||||
['zim_id'], ['id'], ondelete='CASCADE')
|
||||
op.create_unique_constraint('uq_collection_zim_entry', 'collection_item',
|
||||
['collection_id', 'zim_id', 'zim_entry'])
|
||||
op.create_unique_constraint('uq_collection_url', 'collection_item', ['collection_id', 'url'])
|
||||
op.create_check_constraint('ck_collection_item_kind', 'collection_item', CHECK_SQL)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_constraint('ck_collection_item_kind', 'collection_item', type_='check')
|
||||
op.drop_constraint('uq_collection_url', 'collection_item', type_='unique')
|
||||
op.drop_constraint('uq_collection_zim_entry', 'collection_item', type_='unique')
|
||||
op.drop_constraint('collection_item_zim_id_fkey', 'collection_item', type_='foreignkey')
|
||||
# Restore NOT NULL (valid only if no non-file rows remain).
|
||||
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=False)
|
||||
op.drop_column('collection_item', 'title')
|
||||
op.drop_column('collection_item', 'url')
|
||||
op.drop_column('collection_item', 'zim_entry')
|
||||
op.drop_column('collection_item', 'zim_id')
|
||||
op.drop_column('collection_item', 'item_kind')
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
"""Increase file size column.
|
||||
|
||||
Revision ID: aac864072193
|
||||
Revises: 467235201c70
|
||||
Create Date: 2022-06-22 17:34:33.635528
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'aac864072193'
|
||||
down_revision = '467235201c70'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE file ALTER COLUMN size TYPE bigint')
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
"""Add unique constraint on collection (name, kind)
|
||||
|
||||
This migration:
|
||||
1. Removes duplicate collections (keeping the one with most items/archives)
|
||||
2. Adds a unique constraint on (name, kind) to prevent future duplicates
|
||||
|
||||
Revision ID: add_unique_collection_name_kind
|
||||
Revises: migrate_download_to_collection
|
||||
Create Date: 2025-11-27
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'add_unique_collection_name_kind'
|
||||
down_revision = 'migrate_download_to_collection'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("Add Unique Constraint on Collection (name, kind)")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
# Step 1: Find and remove duplicate collections
|
||||
print("Step 1: Finding duplicate collections...")
|
||||
|
||||
# Find duplicates - keep the one with the lowest id (first created)
|
||||
duplicates = session.execute(text("""
|
||||
SELECT name, kind, array_agg(id ORDER BY id) as ids
|
||||
FROM collection
|
||||
GROUP BY name, kind
|
||||
HAVING COUNT(*) > 1
|
||||
""")).fetchall()
|
||||
|
||||
if duplicates:
|
||||
print(f"Found {len(duplicates)} sets of duplicate collections")
|
||||
for name, kind, ids in duplicates:
|
||||
keep_id = ids[0] # Keep the first one (lowest id)
|
||||
remove_ids = ids[1:] # Remove the rest
|
||||
print(f" - '{name}' ({kind}): keeping id={keep_id}, removing ids={remove_ids}")
|
||||
|
||||
# Move any archives from duplicate collections to the one we're keeping
|
||||
for remove_id in remove_ids:
|
||||
session.execute(text("""
|
||||
UPDATE archive SET collection_id = :keep_id
|
||||
WHERE collection_id = :remove_id
|
||||
"""), {'keep_id': keep_id, 'remove_id': remove_id})
|
||||
|
||||
# Move any downloads from duplicate collections
|
||||
session.execute(text("""
|
||||
UPDATE download SET collection_id = :keep_id
|
||||
WHERE collection_id = :remove_id
|
||||
"""), {'keep_id': keep_id, 'remove_id': remove_id})
|
||||
|
||||
# Move any collection items
|
||||
session.execute(text("""
|
||||
UPDATE collection_item SET collection_id = :keep_id
|
||||
WHERE collection_id = :remove_id
|
||||
"""), {'keep_id': keep_id, 'remove_id': remove_id})
|
||||
|
||||
# Delete the duplicate collection
|
||||
session.execute(text("""
|
||||
DELETE FROM collection WHERE id = :remove_id
|
||||
"""), {'remove_id': remove_id})
|
||||
|
||||
session.commit()
|
||||
print("Duplicates removed\n")
|
||||
else:
|
||||
print("No duplicate collections found\n")
|
||||
|
||||
# Step 2: Add unique constraint
|
||||
print("Step 2: Adding unique constraint on (name, kind)...")
|
||||
op.create_unique_constraint('uq_collection_name_kind', 'collection', ['name', 'kind'])
|
||||
print("Done\n")
|
||||
|
||||
print("=" * 60)
|
||||
print("Migration Complete")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute(text('ALTER TABLE public.collection OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_constraint('uq_collection_name_kind', 'collection', type_='unique')
|
||||
|
|
@ -1,192 +0,0 @@
|
|||
"""add_filegroup_directory_column
|
||||
|
||||
Revision ID: af4527503bdf
|
||||
Revises: 577e414f1bc4
|
||||
Create Date: 2025-12-23 19:35:18.776018
|
||||
|
||||
This migration:
|
||||
1. Adds `directory` column to file_group (stores parent directory of primary_path)
|
||||
2. Converts paths in `files` JSON from absolute to relative (filename only)
|
||||
3. Converts paths in `data` JSON from absolute to relative (filename only)
|
||||
|
||||
After this migration:
|
||||
- directory: /media/wrolpi/videos/channel1 (absolute)
|
||||
- primary_path: /media/wrolpi/videos/channel1/video.mp4 (absolute, unchanged for now)
|
||||
- files: [{path: "video.mp4", ...}] (relative filenames)
|
||||
- data: {poster_path: "video.jpg", ...} (relative filenames)
|
||||
|
||||
Processing is done in batches to handle large databases (hundreds of thousands of FileGroups)
|
||||
on memory-constrained devices like Raspberry Pi.
|
||||
"""
|
||||
import os
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import wrolpi.media_path
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'af4527503bdf'
|
||||
down_revision = '577e414f1bc4'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
# Batch size for processing - balances memory usage vs. migration speed
|
||||
BATCH_SIZE = 1000
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Add the directory column (nullable initially)
|
||||
# Index is created in the next migration (26c5a62ec683) AFTER data is populated for efficiency.
|
||||
op.add_column('file_group', sa.Column('directory', wrolpi.media_path.MediaPathType(), nullable=True))
|
||||
|
||||
# Phase 1: Populate directory from existing primary_path (batched)
|
||||
# Uses regexp_replace to remove the filename (everything after the last /).
|
||||
print("Phase 1: Populating directory column...")
|
||||
total_updated = 0
|
||||
while True:
|
||||
result = session.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET directory = regexp_replace(primary_path, '/[^/]+$', '')
|
||||
WHERE id IN (
|
||||
SELECT id FROM file_group
|
||||
WHERE directory IS NULL
|
||||
AND primary_path IS NOT NULL
|
||||
LIMIT :batch_size
|
||||
)
|
||||
"""), {'batch_size': BATCH_SIZE})
|
||||
session.commit()
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total_updated += result.rowcount
|
||||
print(f" Processed {total_updated} directories...")
|
||||
print(f"Phase 1 complete: {total_updated} directories populated.")
|
||||
|
||||
# Phase 2: Convert files JSON paths from absolute to relative (batched)
|
||||
# Only processes rows where at least one path is still absolute (starts with /)
|
||||
print("Phase 2: Converting files JSON to relative paths...")
|
||||
total_updated = 0
|
||||
while True:
|
||||
result = session.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET files = (
|
||||
SELECT jsonb_agg(
|
||||
jsonb_set(
|
||||
elem,
|
||||
'{path}',
|
||||
to_jsonb(regexp_replace(elem->>'path', '.+/', ''))
|
||||
)
|
||||
)
|
||||
FROM jsonb_array_elements(files::jsonb) AS elem
|
||||
)
|
||||
WHERE id IN (
|
||||
SELECT id FROM file_group
|
||||
WHERE files IS NOT NULL
|
||||
AND jsonb_array_length(files::jsonb) > 0
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM jsonb_array_elements(files::jsonb) AS elem
|
||||
WHERE elem->>'path' LIKE '/%'
|
||||
)
|
||||
LIMIT :batch_size
|
||||
)
|
||||
"""), {'batch_size': BATCH_SIZE})
|
||||
session.commit()
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total_updated += result.rowcount
|
||||
print(f" Converted {total_updated} files arrays...")
|
||||
print(f"Phase 2 complete: {total_updated} files arrays converted.")
|
||||
|
||||
# Phase 3: Convert data JSON paths from absolute to relative (batched)
|
||||
# Handle known path fields: video_path, info_json_path, poster_path, ebook_path, cover_path, screenshot_path
|
||||
# Only processes rows where data contains at least one absolute path
|
||||
print("Phase 3: Converting data JSON to relative paths...")
|
||||
total_updated = 0
|
||||
while True:
|
||||
result = session.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET data = (
|
||||
SELECT COALESCE(
|
||||
jsonb_object_agg(
|
||||
key,
|
||||
CASE
|
||||
-- Handle path fields that are strings starting with /
|
||||
-- Use ::jsonb cast instead of to_jsonb() since value::text already has JSON quotes
|
||||
WHEN key IN ('video_path', 'info_json_path', 'poster_path', 'ebook_path', 'cover_path', 'screenshot_path',
|
||||
'readability_json_path', 'readability_path', 'readability_txt_path', 'singlefile_path')
|
||||
AND jsonb_typeof(value) = 'string'
|
||||
AND value::text LIKE '"/%%'
|
||||
THEN regexp_replace(value::text, '^".+/', '"')::jsonb
|
||||
|
||||
-- Handle caption_paths which is an array of strings
|
||||
WHEN key = 'caption_paths' AND jsonb_typeof(value) = 'array'
|
||||
THEN (
|
||||
SELECT jsonb_agg(
|
||||
CASE
|
||||
WHEN jsonb_typeof(elem) = 'string' AND elem::text LIKE '"/%%'
|
||||
THEN regexp_replace(elem::text, '^".+/', '"')::jsonb
|
||||
ELSE elem
|
||||
END
|
||||
)
|
||||
FROM jsonb_array_elements(value) AS elem
|
||||
)
|
||||
|
||||
-- Keep other values unchanged
|
||||
ELSE value
|
||||
END
|
||||
),
|
||||
'{}'::jsonb
|
||||
)
|
||||
FROM jsonb_each(data::jsonb)
|
||||
)
|
||||
WHERE id IN (
|
||||
SELECT id FROM file_group
|
||||
WHERE data IS NOT NULL
|
||||
AND data::text != 'null'
|
||||
AND data::text != '{}'
|
||||
AND (
|
||||
-- Check if any known path field contains an absolute path
|
||||
data->>'video_path' LIKE '/%'
|
||||
OR data->>'info_json_path' LIKE '/%'
|
||||
OR data->>'poster_path' LIKE '/%'
|
||||
OR data->>'ebook_path' LIKE '/%'
|
||||
OR data->>'cover_path' LIKE '/%'
|
||||
OR data->>'screenshot_path' LIKE '/%'
|
||||
OR data->>'readability_json_path' LIKE '/%'
|
||||
OR data->>'readability_path' LIKE '/%'
|
||||
OR data->>'readability_txt_path' LIKE '/%'
|
||||
OR data->>'singlefile_path' LIKE '/%'
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM jsonb_array_elements_text(
|
||||
CASE WHEN data->>'caption_paths' IS NOT NULL THEN (data->'caption_paths')::jsonb ELSE '[]'::jsonb END
|
||||
) AS elem
|
||||
WHERE elem LIKE '/%'
|
||||
)
|
||||
)
|
||||
LIMIT :batch_size
|
||||
)
|
||||
"""), {'batch_size': BATCH_SIZE})
|
||||
session.commit()
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total_updated += result.rowcount
|
||||
print(f" Converted {total_updated} data objects...")
|
||||
print(f"Phase 3 complete: {total_updated} data objects converted.")
|
||||
|
||||
print("Migration complete!")
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute(sa.text('ALTER TABLE public.file_group OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Note: Downgrade does not restore absolute paths in files/data columns.
|
||||
# This would require knowing the original directory for each record.
|
||||
# Index is dropped in the previous downgrade (26c5a62ec683).
|
||||
op.drop_column('file_group', 'directory')
|
||||
|
|
@ -1,189 +0,0 @@
|
|||
"""Fix media paths incorrectly resolved against /opt/wrolpi.
|
||||
|
||||
MediaPathType.process_bind_param() was calling .absolute() on relative paths,
|
||||
which resolves against the current working directory (cwd) instead of the
|
||||
media directory. In production, cwd is /opt/wrolpi, so paths like
|
||||
'archive/spiegel.de' became '/opt/wrolpi/archive/spiegel.de' instead of
|
||||
'/media/wrolpi/archive/spiegel.de'.
|
||||
|
||||
This migration fixes any corrupted paths in the database and saves the
|
||||
corrected paths to config files.
|
||||
|
||||
Revision ID: b2b9700f5048
|
||||
Revises: a1b2c3d4e5f6
|
||||
Create Date: 2026-01-27
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import yaml
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'b2b9700f5048'
|
||||
down_revision = 'a1b2c3d4e5f6'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def get_relative_path(absolute_path: str, media_directory: str) -> str:
|
||||
"""Convert absolute path to relative path under media directory."""
|
||||
if absolute_path and absolute_path.startswith(media_directory):
|
||||
return absolute_path[len(media_directory):].lstrip('/')
|
||||
return absolute_path
|
||||
|
||||
|
||||
def save_config_file(config: dict, file_path: pathlib.Path, width: int = 120):
|
||||
"""Write config dict to YAML file."""
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with file_path.open('wt') as fh:
|
||||
yaml.dump(config, fh, width=width, sort_keys=True)
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
|
||||
|
||||
def upgrade():
|
||||
connection = op.get_bind()
|
||||
|
||||
# Fix collection.directory
|
||||
result = connection.execute(sa.text("""
|
||||
UPDATE collection
|
||||
SET directory = REPLACE(directory, '/opt/wrolpi/', '/media/wrolpi/')
|
||||
WHERE directory LIKE '/opt/wrolpi/%'
|
||||
"""))
|
||||
if result.rowcount:
|
||||
print(f"Fixed {result.rowcount} collection directory paths")
|
||||
|
||||
# Fix file_group.directory
|
||||
result = connection.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET directory = REPLACE(directory, '/opt/wrolpi/', '/media/wrolpi/')
|
||||
WHERE directory LIKE '/opt/wrolpi/%'
|
||||
"""))
|
||||
if result.rowcount:
|
||||
print(f"Fixed {result.rowcount} file_group directory paths")
|
||||
|
||||
# Fix file_group.primary_path
|
||||
result = connection.execute(sa.text("""
|
||||
UPDATE file_group
|
||||
SET primary_path = REPLACE(primary_path, '/opt/wrolpi/', '/media/wrolpi/')
|
||||
WHERE primary_path LIKE '/opt/wrolpi/%'
|
||||
"""))
|
||||
if result.rowcount:
|
||||
print(f"Fixed {result.rowcount} file_group primary_path paths")
|
||||
|
||||
# Fix download.destination
|
||||
result = connection.execute(sa.text("""
|
||||
UPDATE download
|
||||
SET destination = REPLACE(destination, '/opt/wrolpi/', '/media/wrolpi/')
|
||||
WHERE destination LIKE '/opt/wrolpi/%'
|
||||
"""))
|
||||
if result.rowcount:
|
||||
print(f"Fixed {result.rowcount} download destination paths")
|
||||
|
||||
# Save corrected paths to config files directly (bypass config framework which needs SharedContext)
|
||||
try:
|
||||
from wrolpi.common import get_media_directory
|
||||
media_dir = str(get_media_directory())
|
||||
config_dir = get_media_directory() / 'config'
|
||||
|
||||
# Save domains config
|
||||
domains_result = connection.execute(sa.text("""
|
||||
SELECT id, name, kind, description, directory, file_format
|
||||
FROM collection
|
||||
WHERE kind = 'domain'
|
||||
ORDER BY name
|
||||
"""))
|
||||
domains_collections = []
|
||||
for row in domains_result:
|
||||
collection_config = {'name': row[1], 'kind': row[2]}
|
||||
if row[3]: # description
|
||||
collection_config['description'] = row[3]
|
||||
if row[4]: # directory
|
||||
collection_config['directory'] = get_relative_path(row[4], media_dir + '/')
|
||||
if row[5]: # file_format
|
||||
collection_config['file_format'] = row[5]
|
||||
# Get tag name if exists
|
||||
tag_result = connection.execute(sa.text("""
|
||||
SELECT t.name FROM tag t
|
||||
JOIN collection c ON c.tag_id = t.id
|
||||
WHERE c.id = :id
|
||||
"""), {'id': row[0]}).first()
|
||||
collection_config['tag_name'] = tag_result[0] if tag_result else None
|
||||
domains_collections.append(collection_config)
|
||||
|
||||
# Read existing config to preserve version
|
||||
domains_file = config_dir / 'domains.yaml'
|
||||
version = 0
|
||||
if domains_file.exists():
|
||||
with domains_file.open() as f:
|
||||
existing = yaml.safe_load(f) or {}
|
||||
version = existing.get('version', 0)
|
||||
|
||||
domains_config = {'version': version + 1, 'collections': domains_collections}
|
||||
save_config_file(domains_config, domains_file)
|
||||
print(f"Saved domains config with {len(domains_collections)} collections")
|
||||
|
||||
# Save channels config
|
||||
# Columns: 0=id, 1=name, 2=kind, 3=description, 4=directory, 5=file_format,
|
||||
# 6=url, 7=source_id, 8=generate_posters, 9=calculate_duration, 10=download_missing_data
|
||||
channels_result = connection.execute(sa.text("""
|
||||
SELECT c.id, c.name, c.kind, c.description, c.directory, c.file_format,
|
||||
ch.url, ch.source_id, ch.generate_posters, ch.calculate_duration,
|
||||
ch.download_missing_data
|
||||
FROM collection c
|
||||
JOIN channel ch ON ch.collection_id = c.id
|
||||
WHERE c.kind = 'channel'
|
||||
ORDER BY c.name
|
||||
"""))
|
||||
channels_list = []
|
||||
for row in channels_result:
|
||||
channel_config = {'name': row[1]}
|
||||
if row[4]: # directory
|
||||
channel_config['directory'] = get_relative_path(row[4], media_dir + '/')
|
||||
if row[6]: # url
|
||||
channel_config['url'] = row[6]
|
||||
if row[7]: # source_id
|
||||
channel_config['source_id'] = row[7]
|
||||
channel_config['generate_posters'] = row[8]
|
||||
channel_config['calculate_duration'] = row[9]
|
||||
channel_config['download_missing_data'] = row[10]
|
||||
if row[5]: # file_format
|
||||
channel_config['file_format'] = row[5]
|
||||
# Get tag name if exists
|
||||
tag_result = connection.execute(sa.text("""
|
||||
SELECT t.name FROM tag t
|
||||
JOIN collection c ON c.tag_id = t.id
|
||||
WHERE c.id = :id
|
||||
"""), {'id': row[0]}).first()
|
||||
channel_config['tag_name'] = tag_result[0] if tag_result else None
|
||||
# Get downloads for this channel's collection
|
||||
downloads_result = connection.execute(sa.text("""
|
||||
SELECT url, frequency FROM download
|
||||
WHERE collection_id = :collection_id
|
||||
"""), {'collection_id': row[0]})
|
||||
downloads = [{'url': d[0], 'frequency': d[1]} for d in downloads_result]
|
||||
if downloads:
|
||||
channel_config['downloads'] = downloads
|
||||
channels_list.append(channel_config)
|
||||
|
||||
# Read existing config to preserve version
|
||||
channels_file = config_dir / 'channels.yaml'
|
||||
version = 0
|
||||
if channels_file.exists():
|
||||
with channels_file.open() as f:
|
||||
existing = yaml.safe_load(f) or {}
|
||||
version = existing.get('version', 0)
|
||||
|
||||
channels_config = {'version': version + 1, 'channels': channels_list}
|
||||
save_config_file(channels_config, channels_file)
|
||||
print(f"Saved channels config with {len(channels_list)} channels")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not save configs: {e}")
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Data fix, no meaningful downgrade
|
||||
pass
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
"""Drop inventory tables (inventory is now config-only).
|
||||
|
||||
The inventory system no longer uses the database; each inventory is a YAML file at config/inventory/<slug>.yaml.
|
||||
Legacy data is migrated to those files on startup (see modules/inventory/migrate.py) before these tables are dropped.
|
||||
|
||||
Revision ID: b2e4f6a8c1d3
|
||||
Revises: a7b1c2d3e4f5
|
||||
Create Date: 2026-06-13 00:00:00.000000
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
revision = 'b2e4f6a8c1d3'
|
||||
down_revision = 'a7b1c2d3e4f5'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# `item` first due to its foreign key to `inventory`.
|
||||
session.execute('DROP TABLE IF EXISTS public.item')
|
||||
session.execute('DROP TABLE IF EXISTS public.inventory')
|
||||
session.execute('DROP TABLE IF EXISTS public.inventories_version')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE inventories_version (
|
||||
version SERIAL PRIMARY KEY NOT NULL
|
||||
)
|
||||
''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE public.inventory (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
viewed_at TIMESTAMP WITHOUT TIME ZONE,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
deleted_at TIMESTAMP WITHOUT TIME ZONE
|
||||
)''')
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE public.item (
|
||||
id SERIAL PRIMARY KEY,
|
||||
inventory_id INTEGER REFERENCES inventory(id) ON DELETE CASCADE,
|
||||
brand TEXT,
|
||||
name TEXT,
|
||||
count DECIMAL,
|
||||
item_size DECIMAL,
|
||||
unit TEXT,
|
||||
serving INTEGER,
|
||||
category TEXT,
|
||||
subcategory TEXT,
|
||||
expiration_date DATE,
|
||||
purchase_date DATE,
|
||||
created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
deleted_at TIMESTAMP WITHOUT TIME ZONE
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.inventories_version OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.inventory OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.item OWNER TO wrolpi')
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
"""Create map_subscription table.
|
||||
|
||||
Revision ID: b3c4d5e6f7a8
|
||||
Revises: a2b3c4d5e6f7
|
||||
Create Date: 2026-03-29
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = 'b3c4d5e6f7a8'
|
||||
down_revision = 'a2b3c4d5e6f7'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('''CREATE TABLE map_subscription (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT UNIQUE NOT NULL,
|
||||
region TEXT NOT NULL,
|
||||
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
|
||||
)''')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('DROP TABLE IF EXISTS map_subscription')
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
"""Remove duplicate Channel fields delegated to Collection
|
||||
|
||||
This migration removes fields from Channel that now delegate to Collection:
|
||||
- name (now property: channel.name → collection.name)
|
||||
- directory (now property: channel.directory → collection.directory)
|
||||
- tag_id (now property: channel.tag_id → collection.tag_id)
|
||||
|
||||
These fields were already synced to Collection in previous migration.
|
||||
|
||||
Revision ID: b43f70f369d0
|
||||
Revises: ba98bd360b7a
|
||||
Create Date: 2025-11-19 21:48:58.488850
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'b43f70f369d0'
|
||||
down_revision = 'ba98bd360b7a'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
print("\n" + "="*60)
|
||||
print("Removing Duplicate Fields from Channel")
|
||||
print("="*60 + "\n")
|
||||
|
||||
# Step 1: Drop foreign key constraint on tag_id
|
||||
print("Step 1: Dropping tag_id foreign key constraint...")
|
||||
op.drop_constraint('channel_tag_id_fkey', 'channel', type_='foreignkey')
|
||||
print("✓ Dropped foreign key constraint\n")
|
||||
|
||||
# Step 2: Drop columns (data already in Collection)
|
||||
print("Step 2: Dropping duplicate columns...")
|
||||
op.drop_column('channel', 'name')
|
||||
print(" ✓ Dropped channel.name")
|
||||
op.drop_column('channel', 'directory')
|
||||
print(" ✓ Dropped channel.directory")
|
||||
op.drop_column('channel', 'tag_id')
|
||||
print(" ✓ Dropped channel.tag_id\n")
|
||||
|
||||
print("="*60)
|
||||
print("✓ Channel Cleanup Complete")
|
||||
print(" Channels now delegate name/directory/tag to Collection")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Re-add the columns
|
||||
op.add_column('channel', sa.Column('name', sa.String(), nullable=True))
|
||||
op.add_column('channel', sa.Column('directory', sa.String(), nullable=True))
|
||||
op.add_column('channel', sa.Column('tag_id', sa.Integer(), nullable=True))
|
||||
|
||||
# Re-add foreign key
|
||||
op.create_foreign_key('channel_tag_id_fkey', 'channel', 'tag', ['tag_id'], ['id'], ondelete='CASCADE')
|
||||
|
||||
# Restore data from Collection
|
||||
bind = op.get_bind()
|
||||
bind.execute(text("""
|
||||
UPDATE channel
|
||||
SET name = collection.name,
|
||||
directory = collection.directory,
|
||||
tag_id = collection.tag_id
|
||||
FROM collection
|
||||
WHERE channel.collection_id = collection.id
|
||||
"""))
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
"""Video modification_datetime
|
||||
|
||||
Revision ID: b97c1313f986
|
||||
Revises: 3c33b1dae069
|
||||
Create Date: 2022-01-25 08:00:26.944183
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'b97c1313f986'
|
||||
down_revision = '3c33b1dae069'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE video ADD COLUMN modification_datetime TIMESTAMPTZ')
|
||||
session.execute('CREATE INDEX video_modification_datetime_idx ON video USING BRIN(modification_datetime)')
|
||||
session.execute('UPDATE video SET validated=false') # get modification time for every video duration validation.
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS video_modification_datetime_idx')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS modification_datetime')
|
||||
|
|
@ -1,130 +0,0 @@
|
|||
"""Add collection_id to Channel model
|
||||
|
||||
This migration:
|
||||
1. Adds collection_id column to channel table
|
||||
2. Creates Collection records for existing Channels
|
||||
3. Links Channels to their Collections
|
||||
4. Adds foreign key constraint
|
||||
|
||||
Revision ID: ba98bd360b7a
|
||||
Revises: migrate_domains_to_collections
|
||||
Create Date: 2025-11-19 21:10:56.472836
|
||||
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'ba98bd360b7a'
|
||||
down_revision = 'migrate_domains_to_collections'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Channel → Collection Migration")
|
||||
print("="*60 + "\n")
|
||||
|
||||
# Step 1: Add collection_id column to channel table (nullable initially)
|
||||
print("Step 1: Adding collection_id column to channel table...")
|
||||
op.add_column('channel', sa.Column('collection_id', sa.Integer(), nullable=True))
|
||||
print("✓ Added collection_id column\n")
|
||||
|
||||
# Step 2: Create Collection records for each Channel
|
||||
print("Step 2: Creating Collection records for existing Channels...")
|
||||
|
||||
# Fetch all channels
|
||||
result = session.execute(text("""
|
||||
SELECT id, name, directory, tag_id
|
||||
FROM channel
|
||||
ORDER BY name
|
||||
"""))
|
||||
|
||||
channels = []
|
||||
for row in result:
|
||||
channels.append({
|
||||
'id': row[0],
|
||||
'name': row[1],
|
||||
'directory': row[2],
|
||||
'tag_id': row[3],
|
||||
})
|
||||
|
||||
print(f"Found {len(channels)} Channel records to migrate")
|
||||
|
||||
if channels:
|
||||
for channel in channels:
|
||||
channel_id = channel['id']
|
||||
channel_name = channel['name']
|
||||
|
||||
print(f" Processing Channel {channel_id}: {channel_name}")
|
||||
|
||||
# Check if Collection already exists (using raw SQL to avoid ORM model mismatch)
|
||||
existing = session.execute(text("""
|
||||
SELECT id FROM collection
|
||||
WHERE name = :name AND kind = 'channel'
|
||||
LIMIT 1
|
||||
"""), {'name': channel_name}).first()
|
||||
|
||||
if existing:
|
||||
print(f" Collection already exists (id={existing[0]})")
|
||||
collection_id = existing[0]
|
||||
else:
|
||||
# Create Collection using raw SQL
|
||||
result = session.execute(text("""
|
||||
INSERT INTO collection (name, kind, directory, tag_id)
|
||||
VALUES (:name, 'channel', :directory, :tag_id)
|
||||
RETURNING id
|
||||
"""), {
|
||||
'name': channel_name,
|
||||
'directory': channel['directory'],
|
||||
'tag_id': channel['tag_id'],
|
||||
})
|
||||
collection_id = result.first()[0]
|
||||
print(f" Created Collection id={collection_id}")
|
||||
|
||||
# Link Channel to Collection
|
||||
session.execute(
|
||||
text("UPDATE channel SET collection_id = :collection_id WHERE id = :channel_id"),
|
||||
{'collection_id': collection_id, 'channel_id': channel_id}
|
||||
)
|
||||
|
||||
session.commit()
|
||||
print(f"✓ Created Collections and linked Channels\n")
|
||||
else:
|
||||
print("No channels to migrate\n")
|
||||
|
||||
# Step 3: Add foreign key constraint
|
||||
print("Step 3: Adding foreign key constraint...")
|
||||
op.create_foreign_key(
|
||||
'fk_channel_collection_id',
|
||||
'channel',
|
||||
'collection',
|
||||
['collection_id'],
|
||||
['id'],
|
||||
ondelete='CASCADE'
|
||||
)
|
||||
print("✓ Added foreign key constraint\n")
|
||||
|
||||
print("="*60)
|
||||
print("✓ Channel → Collection Migration Complete")
|
||||
print("="*60 + "\n")
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute(text('ALTER TABLE public.channel OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Remove foreign key constraint
|
||||
op.drop_constraint('fk_channel_collection_id', 'channel', type_='foreignkey')
|
||||
|
||||
# Drop collection_id column
|
||||
op.drop_column('channel', 'collection_id')
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
"""Video textsearch index
|
||||
|
||||
Revision ID: bddb5e95a91e
|
||||
Revises: db0a1560d923
|
||||
Create Date: 2021-12-31 18:12:00.407694
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'bddb5e95a91e'
|
||||
down_revision = 'db0a1560d923'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS textsearch')
|
||||
session.execute('''
|
||||
ALTER TABLE video
|
||||
ADD COLUMN textsearch tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, title), 'A') ||
|
||||
setweight(to_tsvector('english'::regconfig, caption), 'D')
|
||||
) STORED
|
||||
''')
|
||||
session.execute('CREATE INDEX IF NOT EXISTS video_textsearch_idx ON video USING GIN(textsearch)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS video_textsearch_idx')
|
||||
session.execute("""
|
||||
ALTER TABLE video
|
||||
ADD COLUMN IF NOT EXISTS textsearch tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english'::regconfig,
|
||||
((COALESCE(title, ''::text) || ' '::text) ||
|
||||
COALESCE(caption, ''::text)))) STORED
|
||||
""")
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Zim flag
|
||||
|
||||
Revision ID: bf704baa185d
|
||||
Revises: 751aecf46b88
|
||||
Create Date: 2023-07-01 20:00:35.601752
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'bf704baa185d'
|
||||
down_revision = '751aecf46b88'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE wrolpi_flag ADD COLUMN outdated_zims BOOLEAN DEFAULT FALSE')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE wrolpi_flag DROP COLUMN outdated_zims')
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""download_last_download_attempt
|
||||
|
||||
Add the `last_download_attempt` column to the `download` table. This records the last time a Download was
|
||||
dispatched (whether the attempt succeeds or fails) and is used to enforce per-domain and global daily download
|
||||
limits. An index supports the "attempted since local midnight" count query.
|
||||
|
||||
Revision ID: c3d5f7a9b1e2
|
||||
Revises: b2e4f6a8c1d3
|
||||
Create Date: 2026-06-28
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'c3d5f7a9b1e2'
|
||||
down_revision = 'b2e4f6a8c1d3'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column('download', sa.Column('last_download_attempt', sa.DateTime(timezone=True), nullable=True))
|
||||
op.create_index('idx_download_last_download_attempt', 'download', ['last_download_attempt'])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index('idx_download_last_download_attempt', table_name='download')
|
||||
op.drop_column('download', 'last_download_attempt')
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
"""Drop map_subscription table.
|
||||
|
||||
Map subscriptions are now managed via settings.regions on a single Download record.
|
||||
|
||||
Revision ID: c4d5e6f7a8b9
|
||||
Revises: b3c4d5e6f7a8
|
||||
Create Date: 2026-03-31
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = 'c4d5e6f7a8b9'
|
||||
down_revision = 'b3c4d5e6f7a8'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('DROP TABLE IF EXISTS map_subscription')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('''CREATE TABLE map_subscription (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name TEXT UNIQUE NOT NULL,
|
||||
region TEXT NOT NULL,
|
||||
download_id INTEGER NOT NULL REFERENCES download(id) ON DELETE CASCADE
|
||||
)''')
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""Download settings
|
||||
|
||||
Revision ID: c72d305cf11e
|
||||
Revises: d2eace02f5b9
|
||||
Create Date: 2022-10-14 13:00:30.333384
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'c72d305cf11e'
|
||||
down_revision = 'd2eace02f5b9'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE download ADD COLUMN settings JSONB')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS settings')
|
||||
|
|
@ -1,59 +0,0 @@
|
|||
"""file_group indexes
|
||||
|
||||
Revision ID: c9561bda3cf0
|
||||
Revises: 3b6918aeca4b
|
||||
Create Date: 2023-03-25 20:45:00.470336
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'c9561bda3cf0'
|
||||
down_revision = '3b6918aeca4b'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('CREATE INDEX file_group_mimetype_idx ON file_group(mimetype)')
|
||||
session.execute('CREATE INDEX file_group_model_idx ON file_group(model)')
|
||||
session.execute('CREATE INDEX file_group_modification_datetime_idx ON file_group(modification_datetime)')
|
||||
session.execute('CREATE INDEX file_group_size_ix ON file_group(size)')
|
||||
session.execute('CREATE INDEX file_group_textsearch_idx ON file_group USING GIN(textsearch)')
|
||||
|
||||
session.execute('CREATE INDEX archive_domain_id_idx ON archive(domain_id)')
|
||||
|
||||
session.execute('CREATE INDEX domains_domain_idx ON domains(domain)')
|
||||
|
||||
session.execute('CREATE INDEX video_source_id_idx ON video(source_id)')
|
||||
session.execute('CREATE INDEX video_upload_date_idx ON video(upload_date)')
|
||||
session.execute('CREATE INDEX video_view_count_idx ON video(view_count)')
|
||||
session.execute('CREATE INDEX video_viewed_idx ON video(viewed)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS file_group_full_stem_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_mimetype_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_model_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_modification_datetime_idx')
|
||||
session.execute('DROP INDEX IF EXISTS file_group_size_ix')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS archive_domain_id_idx')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS domains_domain_idx')
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS video_source_id_idx')
|
||||
session.execute('DROP INDEX IF EXISTS video_upload_date_idx')
|
||||
session.execute('DROP INDEX IF EXISTS video_view_count_idx')
|
||||
session.execute('DROP INDEX IF EXISTS video_viewed_idx')
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
"""Reindex pdfs
|
||||
|
||||
Revision ID: d0bbe45e56c3
|
||||
Revises: dbf05c8e8dcc
|
||||
Create Date: 2022-12-09 12:27:18.908533
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd0bbe45e56c3'
|
||||
down_revision = 'dbf05c8e8dcc'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute("UPDATE file SET indexed=false WHERE mimetype = 'application/pdf'")
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
"""file.path on cascade
|
||||
|
||||
Revision ID: d2eace02f5b9
|
||||
Revises: 70db5ba8763d
|
||||
Create Date: 2022-09-28 15:14:43.306819
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd2eace02f5b9'
|
||||
down_revision = '70db5ba8763d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
ALTER TABLE video DROP CONSTRAINT video_caption_path_fkey;
|
||||
ALTER TABLE video DROP CONSTRAINT video_info_json_path_fkey;
|
||||
ALTER TABLE video DROP CONSTRAINT video_poster_path_fkey;
|
||||
ALTER TABLE video DROP CONSTRAINT video_video_path_fkey;
|
||||
ALTER TABLE video ADD FOREIGN KEY (caption_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE video ADD FOREIGN KEY (info_json_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE video ADD FOREIGN KEY (poster_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE video ADD FOREIGN KEY (video_path) REFERENCES file ON DELETE CASCADE;
|
||||
''')
|
||||
|
||||
session.execute('''
|
||||
ALTER TABLE archive DROP CONSTRAINT archive_readability_json_path_fkey;
|
||||
ALTER TABLE archive DROP CONSTRAINT archive_readability_path_fkey;
|
||||
ALTER TABLE archive DROP CONSTRAINT archive_readability_txt_path_fkey;
|
||||
ALTER TABLE archive DROP CONSTRAINT archive_screenshot_path_fkey;
|
||||
ALTER TABLE archive DROP CONSTRAINT archive_singlefile_path_fkey;
|
||||
ALTER TABLE archive ADD FOREIGN KEY (readability_json_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE archive ADD FOREIGN KEY (readability_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE archive ADD FOREIGN KEY (readability_txt_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE archive ADD FOREIGN KEY (screenshot_path) REFERENCES file ON DELETE CASCADE;
|
||||
ALTER TABLE archive ADD FOREIGN KEY (singlefile_path) REFERENCES file ON DELETE CASCADE;
|
||||
''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.file OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.video OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.domains OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.archive OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""Video view_count bigint.
|
||||
|
||||
Revision ID: d371333159ab
|
||||
Revises: b2b9700f5048
|
||||
Create Date: 2026-02-08
|
||||
|
||||
This migration changes the view_count column from INTEGER to BIGINT
|
||||
to accommodate YouTube videos with view counts exceeding 2,147,483,647.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd371333159ab'
|
||||
down_revision = 'b2b9700f5048'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.alter_column('video', 'view_count',
|
||||
existing_type=sa.Integer(),
|
||||
type_=sa.BigInteger())
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.alter_column('video', 'view_count',
|
||||
existing_type=sa.BigInteger(),
|
||||
type_=sa.Integer())
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Channel tag
|
||||
|
||||
Revision ID: d4e446637f18
|
||||
Revises: 2bfb3b50b178
|
||||
Create Date: 2024-08-03 17:49:17.133086
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd4e446637f18'
|
||||
down_revision = '2bfb3b50b178'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS tag_id INTEGER REFERENCES tag(id)')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS tag_id')
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
"""file_group_suffix
|
||||
|
||||
Add the `suffix` column to the `file_group` table. This stores the lowercased suffix of the primary file
|
||||
(e.g. ".bin") so searches can filter by file type using an index, rather than relying on full-text tokens.
|
||||
|
||||
`suffix` is populated at ingest time alongside `mimetype` (wrolpi.files.lib._upsert_files and
|
||||
FileGroup.from_paths). Existing rows are backfilled here using the same `split_path_stem_and_suffix` logic so
|
||||
special multi-part suffixes like ".info.json" are stored correctly (not a naive ".json").
|
||||
|
||||
Revision ID: d4e6f8a0c2b5
|
||||
Revises: c3d5f7a9b1e2
|
||||
Create Date: 2026-07-01
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd4e6f8a0c2b5'
|
||||
down_revision = 'c3d5f7a9b1e2'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
# Backfill in batches to bound memory on low-RAM devices (e.g. Raspberry Pi).
|
||||
BATCH_SIZE = 5000
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column('file_group', sa.Column('suffix', sa.String(), nullable=True))
|
||||
op.create_index('ix_file_group_suffix', 'file_group', ['suffix'])
|
||||
|
||||
# Backfill existing rows using the same suffix logic used at ingest time so the stored value matches what
|
||||
# future refreshes would produce (including WROLPi special suffixes like ".info.json").
|
||||
from wrolpi.files.lib import split_path_stem_and_suffix
|
||||
conn = op.get_bind()
|
||||
last_id = 0
|
||||
while True:
|
||||
rows = conn.execute(
|
||||
sa.text('SELECT id, primary_path FROM file_group WHERE id > :last_id ORDER BY id LIMIT :limit'),
|
||||
{'last_id': last_id, 'limit': BATCH_SIZE},
|
||||
).fetchall()
|
||||
if not rows:
|
||||
break
|
||||
|
||||
ids, suffixes = [], []
|
||||
for row_id, primary_path in rows:
|
||||
suffix = (split_path_stem_and_suffix(primary_path)[1] or '').lower() or None
|
||||
if suffix:
|
||||
ids.append(row_id)
|
||||
suffixes.append(suffix)
|
||||
|
||||
if ids:
|
||||
conn.execute(
|
||||
sa.text('''
|
||||
UPDATE file_group AS fg
|
||||
SET suffix = v.suffix
|
||||
FROM (SELECT unnest(cast(:ids AS integer[])) AS id,
|
||||
unnest(cast(:suffixes AS text[])) AS suffix) AS v
|
||||
WHERE fg.id = v.id
|
||||
'''),
|
||||
{'ids': ids, 'suffixes': suffixes},
|
||||
)
|
||||
|
||||
last_id = rows[-1][0]
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index('ix_file_group_suffix', table_name='file_group')
|
||||
op.drop_column('file_group', 'suffix')
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
"""Create doc_section table for per-chapter/per-page full-text search.
|
||||
|
||||
Enables deep-linking from search results into a specific EPUB chapter or PDF page,
|
||||
and fixes a latent completeness bug in full-text search where FileGroup.d_text is
|
||||
truncated to ~100KB (so only the first ~30 pages of a big PDF were searchable).
|
||||
|
||||
Existing EPUB and PDF FileGroups are marked `indexed = FALSE` so that the regular
|
||||
refresh pipeline re-runs their extractors and populates doc_section rows. No one-off
|
||||
script is required. Note: the next refresh on a populated library will be heavier
|
||||
than usual while every EPUB and PDF is re-extracted.
|
||||
|
||||
Revision ID: d5e6f7a8b9c0
|
||||
Revises: c4d5e6f7a8b9
|
||||
Create Date: 2026-04-13
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = 'd5e6f7a8b9c0'
|
||||
down_revision = 'c4d5e6f7a8b9'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('''
|
||||
CREATE TABLE doc_section (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
doc_id INTEGER NOT NULL REFERENCES doc(id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL,
|
||||
ordinal INTEGER NOT NULL,
|
||||
label TEXT,
|
||||
content TEXT,
|
||||
tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english'::regconfig, COALESCE(content, ''))) STORED
|
||||
)
|
||||
''')
|
||||
op.execute('CREATE INDEX doc_section_doc_id_idx ON doc_section (doc_id, ordinal)')
|
||||
op.execute('CREATE INDEX doc_section_tsv_idx ON doc_section USING GIN (tsv)')
|
||||
|
||||
# Force a re-index of all EPUB and PDF file groups so the refresh pipeline
|
||||
# populates the new doc_section table.
|
||||
op.execute('''
|
||||
UPDATE file_group
|
||||
SET indexed = FALSE
|
||||
WHERE mimetype LIKE 'application/epub%'
|
||||
OR mimetype = 'application/pdf'
|
||||
''')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('DROP TABLE IF EXISTS doc_section')
|
||||
# If we drop the table while some file groups still have indexed=FALSE (the
|
||||
# upgrade sets that to trigger re-extraction), the next refresh would try to
|
||||
# write DocSection rows to a table that no longer exists. Restore indexed=TRUE
|
||||
# for those rows so a downgrade leaves the system in a consistent state.
|
||||
op.execute('''
|
||||
UPDATE file_group
|
||||
SET indexed = TRUE
|
||||
WHERE mimetype LIKE 'application/epub%'
|
||||
OR mimetype = 'application/pdf'
|
||||
''')
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
"""Drop channel.link.
|
||||
|
||||
Revision ID: d8fcdb7773c0
|
||||
Revises: 2fb39c1861c7
|
||||
Create Date: 2022-04-08 12:52:02.877449
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd8fcdb7773c0'
|
||||
down_revision = '2fb39c1861c7'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel DROP CONSTRAINT IF EXISTS channel_link_key')
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS link')
|
||||
session.execute('DROP INDEX IF EXISTS channel_url_key')
|
||||
session.execute("UPDATE channel SET url=null WHERE url=''")
|
||||
session.execute('CREATE UNIQUE INDEX channel_url_key ON channel (url) WHERE url IS NOT NULL')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel ADD CONSTRAINT channel_link_key UNIQUE (link)')
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS link TEXT')
|
||||
# Notice we are not removing the channel_url_key constraint.
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""Video.have_comments
|
||||
|
||||
Revision ID: d96a04eb2785
|
||||
Revises: 365c985893b8
|
||||
Create Date: 2024-02-27 16:57:29.831210
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'd96a04eb2785'
|
||||
down_revision = '365c985893b8'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE video ADD COLUMN have_comments BOOLEAN DEFAULT FALSE')
|
||||
session.execute('ALTER TABLE video ADD COLUMN comments_failed BOOLEAN DEFAULT FALSE')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS have_comments')
|
||||
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS comments_failed')
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Downloader info
|
||||
|
||||
Revision ID: db0a1560d923
|
||||
Revises: 84b6993a6091
|
||||
Create Date: 2021-12-27 16:46:02.929119
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'db0a1560d923'
|
||||
down_revision = '84b6993a6091'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS info_json JSONB')
|
||||
session.execute('ALTER TABLE download ADD COLUMN IF NOT EXISTS downloader TEXT')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS info_json')
|
||||
session.execute('ALTER TABLE download DROP COLUMN IF EXISTS downloader')
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
"""Invalidate videos for orphaned video files.
|
||||
|
||||
Revision ID: dbf05c8e8dcc
|
||||
Revises: 6489cd50c889
|
||||
Create Date: 2022-12-07 18:26:45.017144
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'dbf05c8e8dcc'
|
||||
down_revision = '6489cd50c889'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('UPDATE video SET validated=false') # noqa
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Zim auto search
|
||||
|
||||
Revision ID: dc6637ced17b
|
||||
Revises: 1f2325523525
|
||||
Create Date: 2025-01-30 19:02:32.692428
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'dc6637ced17b'
|
||||
down_revision = '1f2325523525'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE zim ADD COLUMN auto_search BOOLEAN DEFAULT TRUE')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('ALTER TABLE zim DROP COLUMN IF EXISTS auto_search')
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
"""file_group_textsearch_abc
|
||||
|
||||
Add the `textsearch_abc` generated tsvector column to `file_group`. It indexes only a_text (title),
|
||||
b_text (author), and c_text (description) — excluding d_text (captions/document body), which is ~90% of all
|
||||
text data. Searches default to this much smaller column; the existing `textsearch` column remains for
|
||||
opt-in "deep" search.
|
||||
|
||||
Adding a STORED generated column rewrites the table; expect a few minutes on large libraries.
|
||||
|
||||
Revision ID: e5f7a9c1d3b6
|
||||
Revises: d4e6f8a0c2b5
|
||||
Create Date: 2026-07-03
|
||||
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'e5f7a9c1d3b6'
|
||||
down_revision = 'd4e6f8a0c2b5'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
conn = op.get_bind()
|
||||
conn.execute(sa.text('''
|
||||
ALTER TABLE file_group
|
||||
ADD COLUMN textsearch_abc tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
|
||||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char")
|
||||
) STORED
|
||||
'''))
|
||||
conn.execute(sa.text(
|
||||
'CREATE INDEX IF NOT EXISTS file_group_textsearch_abc_idx ON file_group USING GIN(textsearch_abc)'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
conn = op.get_bind()
|
||||
conn.execute(sa.text('DROP INDEX IF EXISTS file_group_textsearch_abc_idx'))
|
||||
conn.execute(sa.text('ALTER TABLE file_group DROP COLUMN IF EXISTS textsearch_abc'))
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
"""Force EPUB re-extraction after spine-ordering fix.
|
||||
|
||||
A previous version of `extract_epub` used `book.get_items_of_type(ITEM_DOCUMENT)`
|
||||
to enumerate sections, which returns manifest order (not reading-spine order) and
|
||||
can include non-spine documents like EPUB 3 `nav.xhtml`. The viewer resolves
|
||||
deep-links via `spine.get(n)`, so any `doc_section.ordinal` written by the old
|
||||
code can land users on the wrong chapter.
|
||||
|
||||
The fix iterates `book.spine` directly; this migration flips `indexed = FALSE`
|
||||
on every EPUB file group so the regular refresh pipeline re-extracts them with
|
||||
correct spine-indexed ordinals. PDFs are untouched — their per-page ordinals
|
||||
were always correct.
|
||||
|
||||
Revision ID: e6f7a8b9c0d1
|
||||
Revises: d5e6f7a8b9c0
|
||||
Create Date: 2026-04-13
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = 'e6f7a8b9c0d1'
|
||||
down_revision = 'd5e6f7a8b9c0'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('''
|
||||
UPDATE file_group
|
||||
SET indexed = FALSE
|
||||
WHERE mimetype LIKE 'application/epub%'
|
||||
''')
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Symmetric with the parent migration's downgrade safeguard: if we re-apply
|
||||
# downgrade, leave file groups marked indexed so a refresh doesn't try to
|
||||
# repopulate doc_section rows that may have been dropped by the parent.
|
||||
op.execute('''
|
||||
UPDATE file_group
|
||||
SET indexed = TRUE
|
||||
WHERE mimetype LIKE 'application/epub%'
|
||||
''')
|
||||
|
|
@ -1,172 +0,0 @@
|
|||
"""Rename ebook table to doc and add new columns.
|
||||
|
||||
Revision ID: a1b2c3d4e5f6
|
||||
Revises: d371333159ab
|
||||
Create Date: 2026-03-15 15:30:00.000000
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'e7f8a9b0c1d2'
|
||||
down_revision = 'd371333159ab'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
BATCH_SIZE = 1000
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
if not DOCKERIZED:
|
||||
bind.execute(sa.text("SET ROLE 'wrolpi'"))
|
||||
|
||||
# Rename ebook table to doc.
|
||||
print("Migrating ebook -> doc...")
|
||||
op.rename_table('ebook', 'doc')
|
||||
|
||||
# Rename the index.
|
||||
op.execute('ALTER INDEX IF EXISTS ebook_size_idx RENAME TO doc_size_idx')
|
||||
op.execute('ALTER INDEX IF EXISTS ebook_file_group_id_key RENAME TO doc_file_group_id_key')
|
||||
op.execute('ALTER INDEX IF EXISTS ebook_pkey RENAME TO doc_pkey')
|
||||
|
||||
# Rename the sequence.
|
||||
op.execute('ALTER SEQUENCE IF EXISTS ebook_id_seq RENAME TO doc_id_seq')
|
||||
|
||||
# Add new columns.
|
||||
op.add_column('doc', sa.Column('publisher', sa.String(), nullable=True))
|
||||
op.add_column('doc', sa.Column('language', sa.String(), nullable=True))
|
||||
op.add_column('doc', sa.Column('page_count', sa.Integer(), nullable=True))
|
||||
op.add_column('doc', sa.Column('subject', sa.String(), nullable=True))
|
||||
op.add_column('doc', sa.Column('description', sa.Text(), nullable=True))
|
||||
|
||||
# Widen size from Integer to BigInteger.
|
||||
op.alter_column('doc', 'size', type_=sa.BigInteger(), existing_type=sa.Integer())
|
||||
|
||||
# Update file_group.model: 'ebook' -> 'doc'.
|
||||
# Mark all touched file_groups as not indexed so metadata is re-extracted on next refresh.
|
||||
total = 0
|
||||
while True:
|
||||
result = bind.execute(sa.text(
|
||||
"UPDATE file_group SET model = 'doc', indexed = false"
|
||||
" WHERE id IN (SELECT id FROM file_group WHERE model = 'ebook' LIMIT :batch_size)"
|
||||
), {'batch_size': BATCH_SIZE})
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total += result.rowcount
|
||||
print(f" Updated {total} ebook file_groups -> doc...")
|
||||
session.commit()
|
||||
print(f" Updated {total} ebook file_groups -> doc (done)")
|
||||
|
||||
# Update file_group.model: 'pdf' -> 'doc'.
|
||||
total = 0
|
||||
while True:
|
||||
result = bind.execute(sa.text(
|
||||
"UPDATE file_group SET model = 'doc', indexed = false"
|
||||
" WHERE id IN (SELECT id FROM file_group WHERE model = 'pdf' LIMIT :batch_size)"
|
||||
), {'batch_size': BATCH_SIZE})
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total += result.rowcount
|
||||
print(f" Updated {total} pdf file_groups -> doc...")
|
||||
session.commit()
|
||||
print(f" Updated {total} pdf file_groups -> doc (done)")
|
||||
|
||||
# Create Doc records for existing PDFs that had no model table.
|
||||
# Metadata will be extracted when the doc_modeler re-indexes these files.
|
||||
total = 0
|
||||
while True:
|
||||
result = bind.execute(sa.text("""
|
||||
INSERT INTO doc (file_group_id)
|
||||
SELECT fg.id
|
||||
FROM file_group fg
|
||||
LEFT JOIN doc d ON d.file_group_id = fg.id
|
||||
WHERE fg.mimetype = 'application/pdf'
|
||||
AND d.id IS NULL
|
||||
AND fg.primary_path IS NOT NULL
|
||||
LIMIT :batch_size
|
||||
"""), {'batch_size': BATCH_SIZE})
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total += result.rowcount
|
||||
print(f" Created {total} new doc records for existing PDFs...")
|
||||
session.commit()
|
||||
print(f" Created {total} new doc records for existing PDFs (done)")
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
if not DOCKERIZED:
|
||||
bind.execute(sa.text("SET ROLE 'wrolpi'"))
|
||||
|
||||
# Delete doc records that were PDFs (have no ebook equivalent).
|
||||
total = 0
|
||||
while True:
|
||||
result = bind.execute(sa.text("""
|
||||
DELETE FROM doc
|
||||
WHERE id IN (
|
||||
SELECT d.id FROM doc d
|
||||
JOIN file_group fg ON fg.id = d.file_group_id
|
||||
WHERE fg.mimetype = 'application/pdf'
|
||||
LIMIT :batch_size
|
||||
)
|
||||
"""), {'batch_size': BATCH_SIZE})
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total += result.rowcount
|
||||
print(f" Deleted {total} PDF doc records...")
|
||||
session.commit()
|
||||
print(f" Deleted {total} PDF doc records (done)")
|
||||
|
||||
# Revert file_group.model and re-mark as not indexed.
|
||||
for old_model, mimetype_clause in [
|
||||
('ebook', "mimetype LIKE 'application/epub%'"),
|
||||
('ebook', "mimetype = 'application/x-mobipocket-ebook'"),
|
||||
('pdf', "mimetype = 'application/pdf'"),
|
||||
]:
|
||||
total = 0
|
||||
while True:
|
||||
result = bind.execute(sa.text(f"""
|
||||
UPDATE file_group SET model = :old_model, indexed = false
|
||||
WHERE id IN (
|
||||
SELECT id FROM file_group
|
||||
WHERE model = 'doc' AND {mimetype_clause}
|
||||
LIMIT :batch_size
|
||||
)
|
||||
"""), {'old_model': old_model, 'batch_size': BATCH_SIZE})
|
||||
if result.rowcount == 0:
|
||||
break
|
||||
total += result.rowcount
|
||||
print(f" Reverted {total} file_groups -> {old_model} ({mimetype_clause})...")
|
||||
session.commit()
|
||||
print(f" Reverted {total} file_groups -> {old_model} ({mimetype_clause}) (done)")
|
||||
|
||||
# Revert size column type.
|
||||
op.alter_column('doc', 'size', type_=sa.Integer(), existing_type=sa.BigInteger())
|
||||
|
||||
# Drop new columns.
|
||||
op.drop_column('doc', 'description')
|
||||
op.drop_column('doc', 'subject')
|
||||
op.drop_column('doc', 'page_count')
|
||||
op.drop_column('doc', 'language')
|
||||
op.drop_column('doc', 'publisher')
|
||||
|
||||
# Rename sequences back.
|
||||
op.execute('ALTER SEQUENCE IF EXISTS doc_id_seq RENAME TO ebook_id_seq')
|
||||
|
||||
# Rename indexes back.
|
||||
op.execute('ALTER INDEX IF EXISTS doc_pkey RENAME TO ebook_pkey')
|
||||
op.execute('ALTER INDEX IF EXISTS doc_file_group_id_key RENAME TO ebook_file_group_id_key')
|
||||
op.execute('ALTER INDEX IF EXISTS doc_size_idx RENAME TO ebook_size_idx')
|
||||
|
||||
# Rename table back.
|
||||
op.rename_table('doc', 'ebook')
|
||||
|
|
@ -1,79 +0,0 @@
|
|||
"""Init videos tables.
|
||||
|
||||
Revision ID: f0d0086a73d1
|
||||
Revises:
|
||||
Create Date: 2021-09-20 16:19:36.576217
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
revision = 'f0d0086a73d1'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''CREATE TABLE public.channel
|
||||
(
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
name TEXT,
|
||||
link TEXT UNIQUE,
|
||||
idempotency TEXT,
|
||||
url TEXT,
|
||||
match_regex TEXT,
|
||||
directory TEXT,
|
||||
info_json JSON,
|
||||
info_date DATE,
|
||||
skip_download_videos text[],
|
||||
generate_posters BOOLEAN DEFAULT true,
|
||||
calculate_duration BOOLEAN DEFAULT true,
|
||||
download_frequency INTEGER,
|
||||
next_download DATE
|
||||
)''')
|
||||
|
||||
session.execute('''CREATE TABLE public.video
|
||||
(
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
title TEXT,
|
||||
ext TEXT,
|
||||
source_id TEXT,
|
||||
description_path TEXT,
|
||||
poster_path TEXT,
|
||||
video_path TEXT,
|
||||
caption_path TEXT,
|
||||
info_json_path TEXT,
|
||||
idempotency TEXT,
|
||||
channel_id INTEGER REFERENCES channel(id) ON DELETE CASCADE,
|
||||
caption TEXT,
|
||||
textsearch tsvector GENERATED ALWAYS AS (to_tsvector('english'::regconfig,
|
||||
((COALESCE(title, ''::text) || ' '::text) ||
|
||||
COALESCE(caption, ''::text)))) STORED,
|
||||
upload_date TIMESTAMP WITHOUT TIME ZONE,
|
||||
duration INTEGER,
|
||||
favorite TIMESTAMP WITHOUT TIME ZONE,
|
||||
size BIGINT,
|
||||
viewed TIMESTAMP WITHOUT TIME ZONE,
|
||||
validated_poster BOOLEAN DEFAULT false,
|
||||
view_count INTEGER
|
||||
)''')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.channel OWNER TO wrolpi')
|
||||
session.execute('ALTER TABLE public.video OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP TABLE IF EXISTS channel CASCADE')
|
||||
session.execute('DROP TABLE IF EXISTS video CASCADE')
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
"""Video source_id index.
|
||||
|
||||
Revision ID: f2c6326cad29
|
||||
Revises: f8c65e7e6802
|
||||
Create Date: 2022-01-10 08:27:39.082067
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'f2c6326cad29'
|
||||
down_revision = 'f8c65e7e6802'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('CREATE INDEX video_source_id_idx ON video(source_id)')
|
||||
session.execute('ALTER INDEX video_censored RENAME TO video_censored_idx')
|
||||
session.execute('ALTER INDEX video_favorite RENAME TO video_favorite_idx')
|
||||
session.execute('ALTER INDEX video_view_count RENAME TO video_view_count_idx')
|
||||
session.execute('ALTER INDEX video_validated_poster RENAME TO video_validated_poster_idx')
|
||||
session.execute('ALTER INDEX video_upload_date RENAME TO video_upload_date_idx')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('DROP INDEX IF EXISTS video_source_id_idx')
|
||||
session.execute('ALTER INDEX video_censored_idx RENAME TO video_censored')
|
||||
session.execute('ALTER INDEX video_favorite_idx RENAME TO video_favorite')
|
||||
session.execute('ALTER INDEX video_view_count_idx RENAME TO video_view_count')
|
||||
session.execute('ALTER INDEX video_validated_poster_idx RENAME TO video_validated_poster')
|
||||
session.execute('ALTER INDEX video_upload_date_idx RENAME TO video_upload_date')
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
"""collections feature
|
||||
|
||||
Revision ID: f38cfe4b5cb2
|
||||
Revises: add_unique_collection_name_kind
|
||||
Create Date: 2025-11-29 16:51:34.222412
|
||||
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'f38cfe4b5cb2'
|
||||
down_revision = 'add_unique_collection_name_kind'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.alter_column('archive', 'collection_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('file_group', 'mimetype',
|
||||
existing_type=sa.TEXT(),
|
||||
nullable=True)
|
||||
op.alter_column('tag_file', 'tag_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=False)
|
||||
op.alter_column('tag_file', 'file_group_id',
|
||||
existing_type=sa.BIGINT(),
|
||||
nullable=False)
|
||||
op.alter_column('tag_zim', 'tag_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=False)
|
||||
op.alter_column('tag_zim', 'zim_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=False)
|
||||
op.drop_index('video_upload_date_idx', table_name='video')
|
||||
op.drop_column('video', 'duration')
|
||||
op.drop_column('video', 'upload_date')
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.add_column('video', sa.Column('upload_date', postgresql.TIMESTAMP(timezone=True), autoincrement=False, nullable=True))
|
||||
op.add_column('video', sa.Column('duration', sa.INTEGER(), autoincrement=False, nullable=True))
|
||||
op.create_index('video_upload_date_idx', 'video', ['upload_date'], unique=False)
|
||||
op.alter_column('tag_zim', 'zim_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('tag_zim', 'tag_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('tag_file', 'file_group_id',
|
||||
existing_type=sa.BIGINT(),
|
||||
nullable=True)
|
||||
op.alter_column('tag_file', 'tag_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('archive', 'collection_id',
|
||||
existing_type=sa.INTEGER(),
|
||||
nullable=False)
|
||||
op.alter_column('file_group', 'mimetype',
|
||||
existing_type=sa.TEXT(),
|
||||
nullable=False)
|
||||
# ### end Alembic commands ###
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""Channel.download_missing_data
|
||||
|
||||
Revision ID: f7229a67e333
|
||||
Revises: dc6637ced17b
|
||||
Create Date: 2025-02-19 14:02:57.952791
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'f7229a67e333'
|
||||
down_revision = 'dc6637ced17b'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS download_missing_data BOOLEAN DEFAULT True')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS download_missing_data')
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""Channel.refresh
|
||||
|
||||
Revision ID: f8c65e7e6802
|
||||
Revises: 62acddc55091
|
||||
Create Date: 2022-01-07 14:28:21.787842
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'f8c65e7e6802'
|
||||
down_revision = '62acddc55091'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel ADD COLUMN refreshed BOOL DEFAULT false')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS refreshed')
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
"""File Directories
|
||||
|
||||
Revision ID: f9928a3b6bc5
|
||||
Revises: c9561bda3cf0
|
||||
Create Date: 2023-04-18 16:58:40.043115
|
||||
|
||||
"""
|
||||
import os
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'f9928a3b6bc5'
|
||||
down_revision = 'c9561bda3cf0'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('''
|
||||
CREATE TABLE directory (
|
||||
path TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
idempotency TIMESTAMP WITH TIME ZONE DEFAULT current_timestamp
|
||||
)
|
||||
''')
|
||||
|
||||
session.execute('CREATE INDEX directory_name_idx ON directory(name)')
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute('ALTER TABLE public.directory OWNER TO wrolpi')
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
session.execute('DROP INDEX IF EXISTS directory_name_idx')
|
||||
session.execute('DROP TABLE IF EXISTS directory')
|
||||
|
|
@ -1,576 +0,0 @@
|
|||
"""Migrate Domain model to Collection model
|
||||
|
||||
This migration:
|
||||
1. Adds collection_id column to archive table
|
||||
2. Runs the data migration to populate collections
|
||||
3. Makes collection_id NOT NULL
|
||||
4. Drops the old domain_id column and domains table
|
||||
|
||||
Revision ID: migrate_domains_to_collections
|
||||
Revises: 66407d145b76
|
||||
Create Date: 2025-10-29
|
||||
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
from typing import Dict, List, Optional
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text, Column, Integer, String, ForeignKey
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'migrate_domains_to_collections'
|
||||
down_revision = '66407d145b76'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
# Local model definitions for migration stability.
|
||||
# These are minimal copies of the Collection and CollectionItem models
|
||||
# that contain only the columns needed for this migration.
|
||||
# This ensures the migration will work regardless of future changes to the actual models.
|
||||
MBase = declarative_base()
|
||||
|
||||
|
||||
class MCollection(MBase):
|
||||
"""Migration-stable Collection model with minimum required columns."""
|
||||
__tablename__ = 'collection'
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String, nullable=False)
|
||||
kind = Column(String, nullable=False, server_default='channel', default='channel')
|
||||
directory = Column(String, nullable=True)
|
||||
|
||||
|
||||
class MCollectionItem(MBase):
|
||||
"""Migration-stable CollectionItem model with minimum required columns.
|
||||
|
||||
Note: ForeignKey constraints are NOT declared here because they reference
|
||||
tables (file_group) that are not in this migration's declarative base.
|
||||
The actual FK constraints exist in the database schema.
|
||||
"""
|
||||
__tablename__ = 'collection_item'
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
collection_id = Column(Integer, nullable=False)
|
||||
file_group_id = Column(Integer, nullable=False)
|
||||
position = Column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class DomainMigrationStats:
|
||||
"""Track migration statistics."""
|
||||
def __init__(self):
|
||||
self.domains_found = 0
|
||||
self.domains_migrated = 0
|
||||
self.domains_skipped = 0
|
||||
self.collections_created = 0
|
||||
self.collections_with_directory = 0
|
||||
self.collection_items_created = 0
|
||||
self.archives_linked = 0
|
||||
self.errors: List[str] = []
|
||||
self.warnings: List[str] = []
|
||||
|
||||
def add_error(self, msg: str):
|
||||
self.errors.append(msg)
|
||||
print(f" ✗ ERROR: {msg}")
|
||||
|
||||
def add_warning(self, msg: str):
|
||||
self.warnings.append(msg)
|
||||
print(f" ⚠ WARNING: {msg}")
|
||||
|
||||
def print_summary(self, dry_run: bool = False):
|
||||
"""Print migration summary."""
|
||||
mode = "DRY-RUN" if dry_run else "ACTUAL"
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Domain → Collection Migration Summary ({mode})")
|
||||
print(f"{'='*60}")
|
||||
print(f"Domains found: {self.domains_found}")
|
||||
print(f"Domains migrated: {self.domains_migrated}")
|
||||
print(f"Domains skipped (no archives): {self.domains_skipped}")
|
||||
print(f"Collections created: {self.collections_created}")
|
||||
print(f"Collections with directory: {self.collections_with_directory}")
|
||||
print(f"CollectionItems created: {self.collection_items_created}")
|
||||
print(f"Archives linked: {self.archives_linked}")
|
||||
|
||||
if self.warnings:
|
||||
print(f"\nWarnings: {len(self.warnings)}")
|
||||
for warning in self.warnings:
|
||||
print(f" ⚠ {warning}")
|
||||
|
||||
if self.errors:
|
||||
print(f"\nErrors: {len(self.errors)}")
|
||||
for error in self.errors:
|
||||
print(f" ✗ {error}")
|
||||
|
||||
if not self.errors:
|
||||
print(f"\n✓ Migration completed successfully")
|
||||
else:
|
||||
print(f"\n✗ Migration completed with {len(self.errors)} errors")
|
||||
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
|
||||
def get_domain_data(session: Session) -> List[Dict]:
|
||||
"""Fetch all Domain records from database."""
|
||||
result = session.execute(text("SELECT id, domain, directory FROM domains ORDER BY domain"))
|
||||
domains = []
|
||||
for row in result:
|
||||
domains.append({
|
||||
'id': row[0],
|
||||
'domain': row[1],
|
||||
'directory': row[2],
|
||||
})
|
||||
return domains
|
||||
|
||||
|
||||
def get_archives_for_domain(session: Session, domain_id: int) -> List[Dict]:
|
||||
"""Get all Archive IDs for a given domain."""
|
||||
result = session.execute(
|
||||
text("SELECT id, file_group_id FROM archive WHERE domain_id = :domain_id"),
|
||||
{'domain_id': domain_id}
|
||||
)
|
||||
archives = []
|
||||
for row in result:
|
||||
if row[1]: # file_group_id must not be null
|
||||
archives.append({
|
||||
'archive_id': row[0],
|
||||
'file_group_id': row[1],
|
||||
})
|
||||
return archives
|
||||
|
||||
|
||||
def validate_domain_name(domain: str) -> bool:
|
||||
"""Validate domain name format - must contain at least one dot."""
|
||||
if not domain or not isinstance(domain, str):
|
||||
return False
|
||||
if domain.startswith('.') or domain.endswith('.'):
|
||||
return False
|
||||
return '.' in domain
|
||||
|
||||
|
||||
def detect_common_directory_from_archives(session: Session, domain_id: int) -> Optional[str]:
|
||||
"""
|
||||
Detect the common directory for all archives belonging to a domain.
|
||||
|
||||
Returns the directory path as a string, or None if no common directory exists.
|
||||
"""
|
||||
result = session.execute(
|
||||
text("""
|
||||
SELECT fg.primary_path
|
||||
FROM archive a
|
||||
JOIN file_group fg ON a.file_group_id = fg.id
|
||||
WHERE a.domain_id = :domain_id AND fg.primary_path IS NOT NULL
|
||||
"""),
|
||||
{'domain_id': domain_id}
|
||||
)
|
||||
|
||||
paths = [pathlib.Path(row[0]) for row in result if row[0]]
|
||||
|
||||
if not paths:
|
||||
return None
|
||||
|
||||
# Find common ancestor directory
|
||||
common_dir = paths[0].parent
|
||||
|
||||
for path in paths[1:]:
|
||||
while common_dir != common_dir.parent: # Stop at root
|
||||
try:
|
||||
path.relative_to(common_dir)
|
||||
break
|
||||
except ValueError:
|
||||
common_dir = common_dir.parent
|
||||
|
||||
if common_dir == common_dir.parent:
|
||||
return None # Reached root, no common directory
|
||||
|
||||
return str(common_dir)
|
||||
|
||||
|
||||
def check_directory_conflict(session: Session, directory: str) -> bool:
|
||||
"""Check if a directory is already used by an existing Collection."""
|
||||
result = session.execute(
|
||||
text("SELECT id FROM collection WHERE directory = :directory"),
|
||||
{'directory': directory}
|
||||
)
|
||||
return result.first() is not None
|
||||
|
||||
|
||||
def perform_domain_migration(session: Session, verbose: bool = True) -> DomainMigrationStats:
|
||||
"""
|
||||
Migrate all Domain records to Collection records.
|
||||
|
||||
This function is integrated into the alembic migration.
|
||||
Uses local MCollection model for migration stability, and raw SQL for
|
||||
CollectionItem operations to avoid FK resolution issues with file_group.
|
||||
"""
|
||||
stats = DomainMigrationStats()
|
||||
|
||||
# Step 1: Fetch all domains
|
||||
print("Step 1: Fetching Domain records...")
|
||||
domains = get_domain_data(session)
|
||||
stats.domains_found = len(domains)
|
||||
print(f"Found {len(domains)} Domain records")
|
||||
|
||||
if not domains:
|
||||
print("No domains to migrate")
|
||||
return stats
|
||||
|
||||
# Step 2: Create Collections from Domains
|
||||
print("\nStep 2: Creating Collection records...")
|
||||
domain_to_collection: Dict[int, int] = {} # domain_id -> collection_id
|
||||
|
||||
for domain in domains:
|
||||
domain_id = domain['id']
|
||||
domain_name = domain['domain']
|
||||
|
||||
if verbose:
|
||||
print(f" Processing Domain {domain_id}: {domain_name}")
|
||||
|
||||
# Validate domain name
|
||||
if not validate_domain_name(domain_name):
|
||||
stats.add_error(f"Invalid domain name: {repr(domain_name)} (skipping)")
|
||||
continue
|
||||
|
||||
# Skip domains with no archives - don't create empty collections
|
||||
archives = get_archives_for_domain(session, domain_id)
|
||||
if not archives:
|
||||
if verbose:
|
||||
print(f" Skipping domain {repr(domain_name)} - no archives")
|
||||
stats.domains_skipped += 1
|
||||
continue
|
||||
|
||||
# Check if Collection already exists
|
||||
existing = session.query(MCollection).filter_by(
|
||||
name=domain_name,
|
||||
kind='domain'
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
stats.add_warning(f"Collection already exists for domain {repr(domain_name)} (id={existing.id})")
|
||||
domain_to_collection[domain_id] = existing.id
|
||||
stats.domains_migrated += 1
|
||||
continue
|
||||
|
||||
# Determine directory for the collection
|
||||
directory = None
|
||||
|
||||
# Step 1: Check if Domain.directory is set
|
||||
domain_directory = domain.get('directory')
|
||||
if domain_directory:
|
||||
directory = domain_directory
|
||||
if verbose:
|
||||
print(f" Using Domain.directory: {directory}")
|
||||
else:
|
||||
# Step 2: Detect common directory from archive file paths
|
||||
detected = detect_common_directory_from_archives(session, domain_id)
|
||||
if detected:
|
||||
directory = detected
|
||||
if verbose:
|
||||
print(f" Detected directory from archives: {directory}")
|
||||
|
||||
# Step 3: Check for directory conflicts
|
||||
if directory and check_directory_conflict(session, directory):
|
||||
stats.add_warning(
|
||||
f"Directory {repr(directory)} already used by another Collection "
|
||||
f"(domain {repr(domain_name)} will have no directory)"
|
||||
)
|
||||
directory = None
|
||||
|
||||
# Create Collection
|
||||
collection = MCollection(
|
||||
name=domain_name,
|
||||
kind='domain',
|
||||
directory=directory,
|
||||
)
|
||||
session.add(collection)
|
||||
session.flush([collection])
|
||||
|
||||
domain_to_collection[domain_id] = collection.id
|
||||
stats.collections_created += 1
|
||||
stats.domains_migrated += 1
|
||||
if directory:
|
||||
stats.collections_with_directory += 1
|
||||
|
||||
if verbose:
|
||||
if directory:
|
||||
print(f" Created Collection id={collection.id} with directory={directory}")
|
||||
else:
|
||||
print(f" Created Collection id={collection.id} (no directory)")
|
||||
|
||||
print(f"Created {stats.collections_created} Collection records")
|
||||
|
||||
# Step 3: Update archive.collection_id for all Archives
|
||||
print("\nStep 3: Updating archive.collection_id...")
|
||||
|
||||
for domain in domains:
|
||||
domain_id = domain['id']
|
||||
domain_name = domain['domain']
|
||||
|
||||
if domain_id not in domain_to_collection:
|
||||
# Domain was skipped due to error
|
||||
continue
|
||||
|
||||
collection_id = domain_to_collection[domain_id]
|
||||
|
||||
# Update all archives for this domain
|
||||
result = session.execute(
|
||||
text("UPDATE archive SET collection_id = :collection_id WHERE domain_id = :domain_id"),
|
||||
{'collection_id': collection_id, 'domain_id': domain_id}
|
||||
)
|
||||
updated_count = result.rowcount
|
||||
|
||||
if verbose and updated_count > 0:
|
||||
print(f" Domain {repr(domain_name)}: Updated {updated_count} archives")
|
||||
|
||||
stats.archives_linked += updated_count
|
||||
|
||||
print(f"Updated {stats.archives_linked} Archives with collection_id")
|
||||
|
||||
# Step 3b: Handle orphan archives (those with NULL domain_id) by creating collections from their URLs
|
||||
print("\nStep 3b: Handling orphan archives (NULL domain_id)...")
|
||||
result = session.execute(text("""
|
||||
SELECT a.id, a.file_group_id, fg.url
|
||||
FROM archive a
|
||||
LEFT JOIN file_group fg ON a.file_group_id = fg.id
|
||||
WHERE a.domain_id IS NULL
|
||||
"""))
|
||||
orphans = list(result)
|
||||
if orphans:
|
||||
print(f" Found {len(orphans)} orphan archives:")
|
||||
|
||||
# Group orphans by domain extracted from URL
|
||||
from urllib.parse import urlparse
|
||||
domain_orphans = {}
|
||||
for archive_id, file_group_id, url in orphans:
|
||||
print(f" Archive {archive_id}: {url}")
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc
|
||||
if domain.startswith('www.'):
|
||||
domain = domain[4:]
|
||||
except Exception:
|
||||
domain = 'unknown.domain'
|
||||
|
||||
if domain not in domain_orphans:
|
||||
domain_orphans[domain] = []
|
||||
domain_orphans[domain].append((archive_id, file_group_id))
|
||||
|
||||
# Create collections for each domain and assign orphans
|
||||
for domain_name, archives in domain_orphans.items():
|
||||
print(f" Creating collection for domain: {domain_name} ({len(archives)} archives)")
|
||||
|
||||
# Check if collection already exists
|
||||
existing = session.execute(
|
||||
text("SELECT id FROM collection WHERE name = :name AND kind = 'domain'"),
|
||||
{'name': domain_name}
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
collection_id = existing[0]
|
||||
print(f" Using existing collection id={collection_id}")
|
||||
else:
|
||||
# Create new collection
|
||||
session.execute(
|
||||
text("INSERT INTO collection (name, kind) VALUES (:name, 'domain')"),
|
||||
{'name': domain_name}
|
||||
)
|
||||
collection_id = session.execute(text("SELECT lastval()")).scalar()
|
||||
stats.collections_created += 1
|
||||
print(f" Created collection id={collection_id}")
|
||||
|
||||
# Update archives with collection_id and create CollectionItems
|
||||
for archive_id, file_group_id in archives:
|
||||
session.execute(
|
||||
text("UPDATE archive SET collection_id = :collection_id WHERE id = :archive_id"),
|
||||
{'collection_id': collection_id, 'archive_id': archive_id}
|
||||
)
|
||||
stats.archives_linked += 1
|
||||
|
||||
# Create CollectionItem if not exists
|
||||
existing_item = session.execute(
|
||||
text("SELECT id FROM collection_item WHERE collection_id = :cid AND file_group_id = :fgid"),
|
||||
{'cid': collection_id, 'fgid': file_group_id}
|
||||
).first()
|
||||
if not existing_item:
|
||||
session.execute(
|
||||
text("INSERT INTO collection_item (collection_id, file_group_id, position) VALUES (:cid, :fgid, 0)"),
|
||||
{'cid': collection_id, 'fgid': file_group_id}
|
||||
)
|
||||
stats.collection_items_created += 1
|
||||
|
||||
print(f" Processed {len(orphans)} orphan archives into {len(domain_orphans)} collections")
|
||||
else:
|
||||
print(" No orphan archives found")
|
||||
|
||||
# Step 4: Create CollectionItem records
|
||||
print("\nStep 4: Creating CollectionItem records...")
|
||||
|
||||
for domain in domains:
|
||||
domain_id = domain['id']
|
||||
domain_name = domain['domain']
|
||||
|
||||
if domain_id not in domain_to_collection:
|
||||
continue
|
||||
|
||||
collection_id = domain_to_collection[domain_id]
|
||||
|
||||
# Get all archives for this domain
|
||||
archives = get_archives_for_domain(session, domain_id)
|
||||
|
||||
if not archives:
|
||||
if verbose:
|
||||
print(f" Domain {repr(domain_name)}: No archives found")
|
||||
continue
|
||||
|
||||
if verbose:
|
||||
print(f" Domain {repr(domain_name)}: {len(archives)} archives")
|
||||
|
||||
for idx, archive in enumerate(archives, start=1):
|
||||
file_group_id = archive['file_group_id']
|
||||
archive_id = archive['archive_id']
|
||||
|
||||
# Check if CollectionItem already exists (use raw SQL to avoid FK resolution issues)
|
||||
result = session.execute(
|
||||
text("SELECT id FROM collection_item WHERE collection_id = :collection_id AND file_group_id = :file_group_id"),
|
||||
{'collection_id': collection_id, 'file_group_id': file_group_id}
|
||||
)
|
||||
existing_item = result.first()
|
||||
|
||||
if existing_item:
|
||||
if verbose:
|
||||
print(f" Archive {archive_id} already linked to collection")
|
||||
continue
|
||||
|
||||
# Create CollectionItem (use raw SQL to avoid FK resolution issues)
|
||||
session.execute(
|
||||
text("INSERT INTO collection_item (collection_id, file_group_id, position) VALUES (:collection_id, :file_group_id, :position)"),
|
||||
{'collection_id': collection_id, 'file_group_id': file_group_id, 'position': idx}
|
||||
)
|
||||
stats.collection_items_created += 1
|
||||
print(f"Created {stats.collection_items_created} CollectionItem records")
|
||||
|
||||
# Step 5: Skip config export during migration
|
||||
# The config export requires a separate database session, but we're inside
|
||||
# an alembic transaction. The application will export on startup instead.
|
||||
print("\nStep 5: Skipping domains.yaml export during migration...")
|
||||
print(" (Config will be exported on next application startup)")
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def upgrade():
|
||||
"""
|
||||
Migrate Domains to Collections.
|
||||
|
||||
This is a one-way migration - there is no downgrade path because:
|
||||
- Domain data is transformed into Collection data
|
||||
- Once migrated, the system uses Collections exclusively
|
||||
- Downgrade would require reimplementing Domain model
|
||||
"""
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("DOMAIN → COLLECTION MIGRATION")
|
||||
print("="*60 + "\n")
|
||||
|
||||
# Step 1: Add collection_id column to archive table (nullable initially)
|
||||
print("Step 1: Adding collection_id column to archive table...")
|
||||
op.add_column('archive',
|
||||
sa.Column('collection_id', sa.Integer(), nullable=True))
|
||||
|
||||
# Add foreign key to collection table
|
||||
op.create_foreign_key(
|
||||
'archive_collection_id_fkey',
|
||||
'archive', 'collection',
|
||||
['collection_id'], ['id'],
|
||||
ondelete='CASCADE'
|
||||
)
|
||||
print("✓ Added collection_id column\n")
|
||||
|
||||
# Step 2: Run data migration script
|
||||
print("Step 2: Running data migration script...")
|
||||
print("This will:")
|
||||
print(" - Create Collection records from Domain records")
|
||||
print(" - Create CollectionItem records for Archives\n")
|
||||
|
||||
try:
|
||||
# Commit the schema changes before running migration
|
||||
session.commit()
|
||||
|
||||
# Run the data migration
|
||||
stats = perform_domain_migration(session, verbose=True)
|
||||
|
||||
# Print summary
|
||||
stats.print_summary(dry_run=False)
|
||||
|
||||
if stats.errors:
|
||||
raise Exception(f"Migration completed with {len(stats.errors)} errors. See logs above.")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Migration failed: {e}")
|
||||
raise
|
||||
|
||||
# Step 3: Make collection_id NOT NULL
|
||||
print("\nStep 3: Making collection_id NOT NULL...")
|
||||
|
||||
# Check if any archives don't have a collection_id
|
||||
result = session.execute(sa.text(
|
||||
"SELECT COUNT(*) FROM archive WHERE collection_id IS NULL"
|
||||
))
|
||||
null_count = result.scalar()
|
||||
|
||||
if null_count > 0:
|
||||
raise Exception(
|
||||
f"Cannot make collection_id NOT NULL: {null_count} archives have NULL collection_id. "
|
||||
f"The migration script should have assigned all archives to collections."
|
||||
)
|
||||
|
||||
op.alter_column('archive', 'collection_id', nullable=False)
|
||||
print("✓ collection_id is now NOT NULL\n")
|
||||
|
||||
# Step 4: Drop old domain_id column and foreign key
|
||||
print("Step 4: Dropping domain_id column and foreign key...")
|
||||
op.drop_constraint('archive_domain_id_fkey', 'archive', type_='foreignkey')
|
||||
op.drop_column('archive', 'domain_id')
|
||||
print("✓ Dropped domain_id column\n")
|
||||
|
||||
# Step 5: Drop domains table
|
||||
print("Step 5: Dropping domains table...")
|
||||
op.drop_table('domains')
|
||||
print("✓ Dropped domains table\n")
|
||||
|
||||
# Ensure table ownership in non-docker environments
|
||||
if not DOCKERIZED:
|
||||
print("Setting table ownership...")
|
||||
session.execute(sa.text('ALTER TABLE public.archive OWNER TO wrolpi'))
|
||||
session.commit()
|
||||
|
||||
print("="*60)
|
||||
print("MIGRATION COMPLETED SUCCESSFULLY")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
def downgrade():
|
||||
"""
|
||||
No downgrade path for this migration.
|
||||
|
||||
This is intentional because:
|
||||
1. Domain → Collection is a one-way data transformation
|
||||
2. The Domain model is being removed from the codebase
|
||||
3. Rolling back would require:
|
||||
- Recreating Domain model code
|
||||
- Reversing the data transformation (Collection → Domain)
|
||||
- Handling edge cases where Collections have features Domains didn't
|
||||
|
||||
If you need to rollback, you should:
|
||||
1. Restore from a database backup taken before migration
|
||||
2. Re-index all files to rebuild Domain records
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"This migration has no downgrade path. "
|
||||
"To rollback, reset database and re-index files."
|
||||
)
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
"""Migrate Download.channel_id to Download.collection_id
|
||||
|
||||
This migration:
|
||||
1. Adds collection_id column to download table
|
||||
2. Migrates existing channel_id references to collection_id via Channel.collection_id
|
||||
3. Drops channel_id column and foreign key
|
||||
|
||||
Revision ID: migrate_download_to_collection
|
||||
Revises: b43f70f369d0
|
||||
Create Date: 2025-11-27
|
||||
"""
|
||||
import os
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'migrate_download_to_collection'
|
||||
down_revision = 'b43f70f369d0'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("Download channel_id -> collection_id Migration")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
# Step 1: Add collection_id column (nullable initially)
|
||||
print("Step 1: Adding collection_id column to download table...")
|
||||
op.add_column('download', sa.Column('collection_id', sa.Integer(), nullable=True))
|
||||
print("Done\n")
|
||||
|
||||
# Step 2: Migrate data - lookup collection_id through channel
|
||||
print("Step 2: Migrating channel_id to collection_id...")
|
||||
result = session.execute(text("""
|
||||
UPDATE download
|
||||
SET collection_id = channel.collection_id
|
||||
FROM channel
|
||||
WHERE download.channel_id = channel.id
|
||||
AND download.channel_id IS NOT NULL
|
||||
"""))
|
||||
print(f"Updated {result.rowcount} download records\n")
|
||||
|
||||
# Step 3: Add foreign key constraint
|
||||
print("Step 3: Adding foreign key constraint...")
|
||||
op.create_foreign_key(
|
||||
'fk_download_collection_id',
|
||||
'download',
|
||||
'collection',
|
||||
['collection_id'],
|
||||
['id'],
|
||||
ondelete='SET NULL'
|
||||
)
|
||||
print("Done\n")
|
||||
|
||||
# Step 4: Create index for performance
|
||||
print("Step 4: Creating index on collection_id...")
|
||||
op.create_index('idx_download_collection_id', 'download', ['collection_id'])
|
||||
print("Done\n")
|
||||
|
||||
# Step 5: Drop old channel_id foreign key and column
|
||||
print("Step 5: Dropping channel_id column...")
|
||||
op.drop_constraint('download_channel_id_fkey', 'download', type_='foreignkey')
|
||||
op.drop_column('download', 'channel_id')
|
||||
print("Done\n")
|
||||
|
||||
# Step 6: Fix the update_channel_minimum_frequency trigger function
|
||||
# It was referencing channel_id which no longer exists
|
||||
print("Step 6: Updating update_channel_minimum_frequency trigger function...")
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
DECLARE
|
||||
v_channel_id INTEGER;
|
||||
BEGIN
|
||||
-- Handle INSERT and UPDATE: use NEW record
|
||||
IF (TG_OP = 'DELETE') THEN
|
||||
-- For DELETE, we need to find the channel from OLD record
|
||||
IF OLD.collection_id IS NOT NULL THEN
|
||||
SELECT id INTO v_channel_id FROM channel WHERE collection_id = OLD.collection_id;
|
||||
IF v_channel_id IS NOT NULL THEN
|
||||
UPDATE channel
|
||||
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE collection_id = OLD.collection_id)
|
||||
WHERE id = v_channel_id;
|
||||
END IF;
|
||||
END IF;
|
||||
RETURN OLD;
|
||||
ELSE
|
||||
-- For INSERT and UPDATE, use NEW record
|
||||
IF NEW.collection_id IS NOT NULL THEN
|
||||
SELECT id INTO v_channel_id FROM channel WHERE collection_id = NEW.collection_id;
|
||||
IF v_channel_id IS NOT NULL THEN
|
||||
UPDATE channel
|
||||
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE collection_id = NEW.collection_id)
|
||||
WHERE id = v_channel_id;
|
||||
END IF;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END IF;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
print("Done\n")
|
||||
|
||||
print("=" * 60)
|
||||
print("Migration Complete")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
if not DOCKERIZED:
|
||||
session.execute(text('ALTER TABLE public.download OWNER TO wrolpi'))
|
||||
|
||||
|
||||
def downgrade():
|
||||
bind = op.get_bind()
|
||||
session = Session(bind=bind)
|
||||
|
||||
# Restore the old trigger function that uses channel_id
|
||||
op.execute("""
|
||||
CREATE OR REPLACE FUNCTION update_channel_minimum_frequency()
|
||||
RETURNS TRIGGER AS
|
||||
$$
|
||||
BEGIN
|
||||
UPDATE channel
|
||||
SET minimum_frequency = (SELECT MIN(frequency) FROM download WHERE channel_id = NEW.channel_id)
|
||||
WHERE id = NEW.channel_id;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
""")
|
||||
|
||||
# Add back channel_id column
|
||||
op.add_column('download', sa.Column('channel_id', sa.Integer(), nullable=True))
|
||||
|
||||
# Migrate collection_id back to channel_id
|
||||
session.execute(text("""
|
||||
UPDATE download
|
||||
SET channel_id = channel.id
|
||||
FROM channel
|
||||
WHERE download.collection_id = channel.collection_id
|
||||
AND download.collection_id IS NOT NULL
|
||||
"""))
|
||||
|
||||
# Add foreign key constraint
|
||||
op.create_foreign_key(
|
||||
'download_channel_id_fkey',
|
||||
'download',
|
||||
'channel',
|
||||
['channel_id'],
|
||||
['id']
|
||||
)
|
||||
|
||||
# Drop collection_id
|
||||
op.drop_index('idx_download_collection_id', 'download')
|
||||
op.drop_constraint('fk_download_collection_id', 'download', type_='foreignkey')
|
||||
op.drop_column('download', 'collection_id')
|
||||
|
|
@ -77,14 +77,6 @@ DEFAULT_CONFIG = {
|
|||
"description": "Kiwix/Zim server",
|
||||
},
|
||||
# System services
|
||||
{
|
||||
"name": "postgresql",
|
||||
"systemd_name": "postgresql",
|
||||
"port": 5432,
|
||||
"viewable": False,
|
||||
"use_https": False,
|
||||
"description": "PostgreSQL database",
|
||||
},
|
||||
{
|
||||
"name": "caddy",
|
||||
"systemd_name": "caddy",
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ class TestManagedServicesConfig:
|
|||
def test_managed_services_count(self):
|
||||
"""Should have expected number of managed services."""
|
||||
services = DEFAULT_CONFIG["managed_services"]
|
||||
assert len(services) == 12
|
||||
assert len(services) == 11
|
||||
|
||||
@pytest.mark.parametrize("name,expected", [
|
||||
("wrolpi-api",
|
||||
|
|
@ -67,7 +67,6 @@ class TestManagedServicesConfig:
|
|||
("wrolpi-app", {"port": 3000, "viewable": False}),
|
||||
("wrolpi-app-dev", {"port": 3000, "viewable": False, "show_only_when_running": True}),
|
||||
("caddy", {"port": 443}),
|
||||
("postgresql", {"port": 5432, "viewable": False}),
|
||||
("wrolpi-upgrade", {"port": None, "viewable": False, "show_only_when_running": True}),
|
||||
])
|
||||
def test_service_config(self, name, expected):
|
||||
|
|
@ -83,7 +82,6 @@ class TestManagedServicesConfig:
|
|||
("wrolpi-kiwix", True),
|
||||
("wrolpi-api", False),
|
||||
("wrolpi-app", False),
|
||||
("postgresql", False),
|
||||
])
|
||||
def test_service_use_https(self, name, use_https):
|
||||
"""HTTPS services should have use_https=True, others should have False."""
|
||||
|
|
|
|||
|
|
@ -1,21 +1,7 @@
|
|||
# Development containers, these should only be used if you are developing.
|
||||
# The database is SQLite inside the media directory (./test/config/wrolpi.db); no db service.
|
||||
version: '3'
|
||||
services:
|
||||
db: # https://github.com/docker-library/docs/tree/master/postgres
|
||||
image: postgres:12
|
||||
environment:
|
||||
- POSTGRES_DB=wrolpi
|
||||
- POSTGRES_PASSWORD=wrolpi
|
||||
ports:
|
||||
- "5432:5432"
|
||||
healthcheck:
|
||||
test: [ 'CMD-SHELL', 'pg_isready -U postgres' ]
|
||||
interval: 10s
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
volumes:
|
||||
- ./pg_data:/var/lib/postgresql/data
|
||||
|
||||
controller: # WROLPi Controller Service
|
||||
build:
|
||||
context: .
|
||||
|
|
@ -47,8 +33,6 @@ services:
|
|||
|
||||
api:
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
controller:
|
||||
condition: service_started
|
||||
build:
|
||||
|
|
|
|||
|
|
@ -11,7 +11,6 @@ git submodule update --init
|
|||
docker-compose build --parallel
|
||||
docker volume create --name=openstreetmap-data
|
||||
docker volume create --name=openstreetmap-rendered-tiles
|
||||
docker-compose up -d db
|
||||
docker-compose run --rm api db upgrade
|
||||
docker-compose up
|
||||
```
|
||||
|
|
@ -26,7 +25,6 @@ Browse to https://localhost:8443
|
|||
| **api** | 8081 | Sanic API server |
|
||||
| **app** | (via web) | React frontend |
|
||||
| **controller** | 8080 | System management (FastAPI) |
|
||||
| **db** | 5432 | PostgreSQL 12 |
|
||||
| **archive** | 8083 | SingleFile/Readability |
|
||||
| **zim** | (via web:8085) | Kiwix |
|
||||
| **help** | (via web:8086) | MkDocs documentation |
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
[Unit]
|
||||
Description=WROLPi Python API (Development)
|
||||
Wants=postgresql.service
|
||||
PartOf=wrolpi.target
|
||||
Conflicts=wrolpi-api.service
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
[Unit]
|
||||
Description=WROLPi Python API
|
||||
Wants=postgresql.service
|
||||
PartOf=wrolpi.target
|
||||
Conflicts=wrolpi-api-dev.service
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
Description=WROLPi Controller Service
|
||||
Documentation=https://github.com/lrnselfreliance/wrolpi
|
||||
After=network.target
|
||||
Before=wrolpi-api.service wrolpi-app.service postgresql.service
|
||||
Before=wrolpi-api.service wrolpi-app.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
[Unit]
|
||||
Description=WROLPi services group
|
||||
Wants=wrolpi-controller.service wrolpi-api.service wrolpi-app.service caddy.service postgresql.service wrolpi-kiwix.service
|
||||
Wants=wrolpi-controller.service wrolpi-api.service wrolpi-app.service caddy.service wrolpi-kiwix.service
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue