SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2. This is a mechanical port; the code
stays synchronous.
- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
statement committed on its own. 2.0 autobegins, so the unique-stem
migration's read snapshot could not become a writer after its helpers wrote
on another connection ("database is locked"). The begin listener emits no
BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.
- A Repo is a Collection (kind='repo') plus a `repository` row. The new `git` downloader clones it, then updates it
(weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
An existing clone can be imported where it is. repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download. READMEs are searchable
under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
updated.
- Tagging a Repo moves its clone as one directory. Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
destination. A failed tag move restores the Collection's tag. A Collection's directory cannot leave the media
directory.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
WROLPi runs alembic migrations in-process at startup (db_bootstrap).
alembic/env.py called fileConfig unconditionally, which replaced the
root handlers, set the root level to WARN, and disabled every logger
that already existed, so the API's logs went quiet after any startup
migration until the next restart. env.py now configures logging only
when alembic runs from its command line (cmd_opts is set).
This also caused an order-dependent test failure: test_jobs's capture
test failed whenever test_db_bootstrap ran first in the same worker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.
Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
into memory and the single source of truth. New reusable MultiFileConfig
(wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
(text/number/quantity/date/select/location/calories). Unit math/aggregation
moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
inventory save with version-based optimistic concurrency; 404 for missing),
DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
(pound->lb), and an alembic migration drops the tables.
Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
calories column, and embeds a ration estimate (how long the stock lasts).
Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
catalog_defaults.yaml (~60 generic items, total calories per package) with
merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
modal, and entry-row pre-fill of category/size/calories from a name match.
Calculators
- Generic Food Storage planner (like the Water calculator): household +
editable per-person/year amounts -> total to store over a chosen duration,
with women/children weighted by DGA calorie shares and an imperial/metric
toggle.
Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
