* Tag names have leading and trailing whitespace stripped
A padded name now finds the existing Tag instead of creating a
near-duplicate, and a whitespace-only name is rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Migration strips whitespace from Tag names and merges collisions
Tags whose names match once stripped are merged (tagged files, Zim
entries, Collections); the first color by id is kept. Downloads and the
configs that name Tags are rewritten too, so the next import is in step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tags.yaml keeps the database's color for merged Tags
The file was merged in sorted key order, not by Tag id, and the next
import copies the file's color over the database's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2. This is a mechanical port; the code
stays synchronous.
- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
statement committed on its own. 2.0 autobegins, so the unique-stem
migration's read snapshot could not become a writer after its helpers wrote
on another connection ("database is locked"). The begin listener emits no
BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Track how far the user is through a file, kept in recently_viewed.yaml
FileGroup gains `progress` (0.0 to 1.0, 1.0 is finished) and `position` (where
to resume: a time, a page, or an EPUB CFI). Both are returned with the file,
including by POST /api/files/file.
POST /api/files/progress records progress and is also a view. A file 95%
through is finished and keeps no position; one barely started (under 10 s,
the first page, under 1%) is not worth resuming. The write happens before
the response, so the next request sees it. POST /api/files/progress/clear
starts a file over. Both refuse files outside the media directory.
recently_viewed.yaml keeps the 1000 most recently viewed files so viewing
history and positions survive a database rebuild. On import the newer
`viewed` wins, so a file viewed while WROLPi was starting keeps its newer
position. The config is saved when a file is opened, when progress is final
(paused or closed), when it is cleared, and after refresh cleanup (moves,
renames, deletes); heartbeats during playback only write the database.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Progress: write as a writer, confine and merge the import, keep FTS on downgrade
- The progress endpoints begin as writers (request.ctx.db.write()), so a
heartbeat during a refresh waits instead of failing a lock upgrade. The
config switch fires after the commit.
- `..` is collapsed before the ignored-directory check. Clear only resets
an existing FileGroup; it never indexes a file.
- Import confines config paths to the media directory. A newer plain view
keeps its `viewed` but takes the config's position; only progress written
after the config wins. Files are globbed before the write transaction.
- The dump skips a row outside the media directory instead of failing.
- The downgrade drops the columns with ALTER TABLE, which keeps file_group's
FTS triggers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.
- A Repo is a Collection (kind='repo') plus a `repository` row. The new `git` downloader clones it, then updates it
(weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
An existing clone can be imported where it is. repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download. READMEs are searchable
under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
updated.
- Tagging a Repo moves its clone as one directory. Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
destination. A failed tag move restores the Collection's tag. A Collection's directory cannot leave the media
directory.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
WROLPi runs alembic migrations in-process at startup (db_bootstrap).
alembic/env.py called fileConfig unconditionally, which replaced the
root handlers, set the root level to WARN, and disabled every logger
that already existed, so the API's logs went quiet after any startup
migration until the next restart. env.py now configures logging only
when alembic runs from its command line (cmd_opts is set).
This also caused an order-dependent test failure: test_jobs's capture
test failed whenever test_db_bootstrap ran first in the same worker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Preview markdown files formatted, with a raw toggle
Markdown files are typed text/markdown (libmagic reports them as text/plain, or
as a programming language when they hold code) and keep the text indexer, so
they stay deep-searchable. Existing files switch type on their next refresh.
The file preview renders markdown with markdown-it into the page, so it follows
the theme tokens and the night media filter. A Formatted/Raw toggle shows the
source instead, and the choice is remembered. Relative links and images
resolve against the file's directory; URLs that climb out of /media/ are
dropped. Raw HTML in a file is escaped rather than parsed, and the output is
also sanitized with DOMPurify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Do not load images from other sites in the markdown preview
An image from another site (http:, https:, or protocol-relative //) loaded as
soon as the preview opened, telling that site who was reading the file and
when. Such an image now becomes a link labeled with its alt text, so the
reader chooses whether to follow it. Inside a link (a README badge) it becomes
the outer link's text, since links cannot nest. Media and embedded data:
images still load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The listing still read every channel row (for the summary columns) and every download
row of the listed Collections (for the minimum frequency). Both tables store yt-dlp
info_json, so rows are tens of KB and each sits on its own scattered page; selecting
fewer columns did not help because the seek per row is the cost. From a cold disk
cache that was 5 s per table, 10+ s total, on a large library.
Two covering indexes, channel (collection_id, video_count, total_size) and
download (collection_id, frequency), answer both queries without touching a row. The
listing no longer loads Download objects at all; it computes min_download_frequency
with one grouped query and drops the unused `downloads` array from listing rows (the
detail endpoints keep it; the list views only read min_download_frequency).
Adds a models-vs-migrations test (alembic check) so an index or column added to a
model without a migration, or vice versa, fails the suite.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* collections: channel listing reads trigger columns and batch-loads tag/downloads
Listing channel collections summed FileGroup sizes through a video join, which reads a
random page of the file_group table per video, and lazy-loaded every collection's tag
and downloads one query at a time. On a large library with a cold disk cache the
listing took over two minutes, past the browser's 60 s timeout, so the Channels page
showed an error.
channel.video_count and channel.total_size are already maintained by triggers
(wrolpi/schema_ddl.py), so read those. Tag and downloads are selectin-loaded, so the
query count no longer grows with the number of collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* collections: domain listing reads trigger-maintained summary columns
Domain Collections summed file_group.size through an archive join on every listing,
one random page read per archive (about 9 s from a cold disk cache on a large
library). New triggers keep collection.item_count and collection.total_size current
for Domain Collections, as the channel triggers already do for Channels, and the
listing reads those columns. A migration installs the triggers and backfills
existing Domain Collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Discover/compare (find, TEMP table, SQL diff) now lives in
wrolpi.files.refresh so FileWorker is the queue plus apply/move/reorganize.
Existing `from wrolpi.files.worker import compare_file_groups` still works.
file_group.idempotency is unused since SQL compare. DROP COLUMN on SQLite
3.40 does not rebuild the table, so file_group_fts stays valid.
directory.idempotency is unchanged.
* files: pass find -printf \0 as an escape, not a Python NUL (#716)
* files: pass find -printf \\0 as an escape, not a Python NUL
'%h\\0%f\\0%T@\\0' in a Python string is an embedded null byte. subprocess
cannot pass that as an argument, so every GNU-find refresh on Linux CI
raised ValueError. find wants the two-character escape \\0 in the format
string and emits NULs itself.
* files: serialize stem backfill so a cancelled compare cannot lock SQLite
compare_file_groups runs ensure_file_group_stems in a thread. Cancelling
the task does not stop that thread, and a second compare then writes
wrolpi_kv on the same connection (tests share one session). A lock
plus a version re-check makes the second call a no-op.
* files: unique (directory, stem) after collapsing duplicate FileGroups
713 added stem and a non-unique index. This release backfills any NULL
stems, merges leftover duplicates (tags and collection membership move
onto the winner), and creates a UNIQUE index.
Upsert deletes sidecar FileGroups before INSERT so a new primary cannot
collide. Refresh deletes before upsert for the same reason.
from_paths looks up by stem so adding a sidecar does not insert a second
row. Same-stem files without a modeled primary stay one FileGroup.
* files: do not use the test SQLite connection from a worker thread
get_db_curs in pytest shares the test session connection. Running
ensure_file_group_stems and _upsert_files via asyncio.to_thread then
segfaults when the event loop commits that same connection (CI xdist).
Keep to_thread in production, where get_db_curs opens a dedicated
raw connection.
* files: keep tags when a sidecar joins an unmodeled group
choose_primary_file is the single fallback for unmodeled stem groups so
_upsert_files and _upsert_file_groups cannot pick different primaries
and DELETE the tagged row. FileGroup.move merges an existing
(directory, stem) group and flushes the delete before taking that stem.
refresh_files_directly deletes before upsert, matching the other two
callers.
* files: group by (directory, stem), not stem alone
group_files_by_stem mixed marauder/s3.bin and other/s3.bin into one
FileGroup because it ignored directory. Unique (directory, stem) then
kept only one of them, so flasher search missed a firmware file.
run_command_streaming awaits the stdout reader instead of cancelling it
when the process exits, so the last lines are not lost on a fast CI run.
* jobs: write the exception into the Job log on failure
test_jobs_failure only saw 'boom' in the log when JobLogHandler captured
logger.error(..., exc_info=e). Sanic/pytest dictConfig replaces root
handlers (especially under xdist on CI), so that path is empty even
though status and error are set. Log the exception on the Job itself
and store it in the same _update_job as FAILED.
* files: SQL FileGroup compare, stem column, drop unused discovery path
Refresh walked the tree twice (find, then stat every unchanged group),
and a two-directory refresh scanned the whole media directory. Compare
now diffs in SQL on a TEMP table that already has mtime, accepts multiple
roots, and groups by a stored stem that is versioned so a suffix-rule
change cannot delete-and-reinsert tagged files.
Deletes refresh_discover_paths and /api/files/refresh_progress, which
nothing in production called. Video/doc/zim modelers use the same
(no domain row OR indexed != True) gate.
* files: keep stem current on reorganize; NUL-safe find parse
Reorganize wrote the new primary_path but not stem, so the next refresh
treated the FileGroup as deleted+new and dropped tags. GNU find now
emits NUL-delimited records so a tab or newline in a filename cannot
abort a global refresh. Stem backfill skips the table scan once the
algorithm version matches, and runs off the event loop. An empty roots
list is a global refresh, not invalid SQL.
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Browser-based ESP32/ESP8266 flasher (esptool-js + Web Serial); nothing is
uploaded to WROLPi, the flash is written straight to the device over USB.
Firmware sources (themed, tabbed): a saved set, the WROLPi media directory,
or a local file. Local files can be dropped anywhere on the page, which jumps
to the "Add from computer" dropzone. The media picker searches .bin files by
path and can detect the connected chip to filter firmware to matching images.
- Indexed file_group.suffix + path filter for fast .bin search (alembic
migration backfills via split_path_stem_and_suffix)
- modules/flasher: chip detection from ESP image headers, /search endpoint,
and flasher.yaml saved configurations (/configs CRUD)
- Flash progress modal with ETA, connect timeout, boot-mode hint, auto-
scrolling logs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day
Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.
Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
stops dispatching counted downloads for the cycle. daily_download_counts derives the
day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
(base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
validation, 0 = unlimited), and the React Settings page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.
Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
into memory and the single source of truth. New reusable MultiFileConfig
(wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
(text/number/quantity/date/select/location/calories). Unit math/aggregation
moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
inventory save with version-based optimistic concurrency; 404 for missing),
DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
(pound->lb), and an alembic migration drops the tables.
Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
calories column, and embeds a ration estimate (how long the stock lasts).
Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
catalog_defaults.yaml (~60 generic items, total calories per package) with
merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
modal, and entry-row pre-fill of category/size/calories from a name match.
Calculators
- Generic Food Storage planner (like the Water calculator): household +
editable per-person/year amounts -> total to store over a chosen duration,
with women/children weighted by DGA calorie shares and an imperial/metric
toggle.
Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add Playlists: ordered, mixed-type collections
A playlist (Collection kind='playlist') is a manually-curated, ordered list that
mixes item types. Squash of the playlists feature branch (24 commits).
Data model & API:
- CollectionItem is polymorphic via item_kind: 'file' (FileGroup), 'zim'
(zim_id + zim_entry), or 'url' (link to anything the WROLPi browser opens,
e.g. /map?lat=&lon=&z=). CHECK constraint enforces exactly one kind; per-kind
unique constraints; migration a7b1c2d3e4f5.
- Endpoints: create playlist; add/remove/reorder items (idempotent adds: 201 new,
200 existing); GET returns the ordered items. All mutations wrol_mode_check'd.
Config (source of truth):
- playlists.yaml (PlaylistsConfig) stores each playlist's metadata AND ordered
items (file by relative path, zim by path+entry, url inline); round-trips a
DB wipe. Supports backup import (DB_CONFIG_FILE_NAMES).
- The redundant generic CollectionsConfig (collections.yaml) is deleted:
channels/domains own their configs, authors/subjects are re-derived.
On-disk sync (sync_playlists_directory switch):
- Each playlist materializes into the configurable Playlists Directory
(playlists_destination, Settings UI input, excluded from indexing) with
position-prefixed entries: NNNN_<file> hard links (copy fallback across
filesystems) and NNNN_<title>.html stubs for zim/url items.
- Tagged playlists are namespaced like channels: <playlists>/<tag>/<name>;
a tag change moves the directory and prunes the empty old tag dir.
- Deletion safety: managed files are only removed when they are our marked
stubs or still have another hard link (st_nlink > 1) -- never a last copy.
Stray loose files at the playlists root are removed; stub URLs are
scheme-allowlisted (no javascript:/data:).
Frontend:
- /playlists list page (CollectionTable, name filter, New Playlist modal, Tag
column) between Files and Zim in the nav.
- View-only detail page (Name link) and edit page (Edit button): themed table
with model icons/colors + poster thumbnails, items link to their model page
(video/archive/doc) or preview, up/down reorder, remove, add-URL form,
single-tag selector; all editing disabled in WROL Mode.
- Reusable green "Add to Playlist" button on Video/Archive/Doc detail pages,
the FilePreview toolbar, Zim search results, and Map pins (added as a /map
URL item with a map-marker icon).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Playlist edit page parity: rename, custom directory, tag modal move suggestion
Match the ChannelEditPage/DomainEditPage UX on the playlist edit page:
Back/View buttons, an edit form (name, description, directory), and the
shared CollectionTagModal with the violet "Tag" button. The tag modal now
suggests moving the playlist to its managed tag location
(playlists/<tag>/<name>); accepting the suggestion stores directory=None so
the sync keeps auto-managing the location. Tag buttons on the Channel and
Domain edit pages are now violet to match.
Backend: update_collection learns rename (with per-kind validation and
duplicate checks) and playlist custom directories (cleanup of the abandoned
directory, config round-trip, commit-before-switch to avoid the background
sync racing pre-commit state). tag_collection's duplicated untag/tag
branches are merged, and the per-kind config-save dispatch is consolidated
into _activate_config_save().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate Collection error handling and Tag creation
Delete the duplicate UnknownCollection in wrolpi/collections/errors.py and
re-export the canonical one from wrolpi/errors.py, which carries
status_code=NOT_FOUND (the local copy inherited 500). Replace the repeated
query/raise pattern in reorganize.py with Collection.find_by_id.
Add Tag.get_or_create_tag and use it for collection tagging; flush the new
Tag before returning so callers can read .id immediately -- previously
tagging a collection with a brand-new tag name set tag_id=None. Regression
test included.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Storing video settings in session storage to help user start downloads consistently.
Applying channel tag name to new channels when download videos or channels.
Initializing map tiles when importing and resetting map.
