* domains.yaml imports by diffing
domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format. An unchanged
import writes nothing.
A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it. An invalid entry (name
or directory) is skipped, counted, and its domain kept. Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).
Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test_bookmarks_config_import starts the app like its siblings
It imports a config without `async_client`, so it fails (no
shared_ctx.configs_imported) whenever it is the first test on its
worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* domains.yaml: converge without a save; no links while entries are skipped
A save after the import is refused while an entry is skipped, so the
import must converge without one:
- a removed domain with Archives that is already reset is left as it is
(prepare reads `import_db_rows`); before, every import reset it again;
- after prepare, the framework drops changed rows that now match the
database (a dropped unknown Tag), so they are not written again.
Linking Downloads changes domains.yaml after its version is recorded;
while entries are skipped the save is refused, the database stays newer,
and the next startup would save over the skipped entries. Linking now
waits for an import without skipped entries.
An entry that is not a mapping is skipped and counted, instead of failing
the whole file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Readability extraction no longer takes minutes on image-heavy pages
SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.
Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output. Applied in
both the archive container and the native (Raspberry Pi/Debian) path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders cannot be forged by page text
The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times. Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Data URI placeholders survive Readability's lazy-image cleanup
Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder. The ~50 character placeholder triggered this, so those images
lost their inline data. The token is now 160 hex characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.
Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* File downloads: return plain values from upsert_file
Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`. `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone. SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends. Failed downloads retry, re-downloading their files.
`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Archive uploads: return plain values from singlefile_to_archive
Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved. It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query. An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async. Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.
Collection, CollectionItem and Channel dict() now run inside the serializer guard. Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
the session.
Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive. An
Archive with 51 history entries ran 107 statements per request. Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2. This is a mechanical port; the code
stays synchronous.
- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
statement committed on its own. 2.0 autobegins, so the unique-stem
migration's read snapshot could not become a writer after its helpers wrote
on another connection ("database is locked"). The begin listener emits no
BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CircleCI is gone, so the skip and its flag (CIRCLECI → IS_CI) are named for
CI in general, and the flag reads only GITHUB_ACTIONS.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9. Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).
Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The archive modeler claims WROLPi's own singlefiles, but a foreign
compressed (SingleFileZ) HTML file (e.g. saved by the browser extension and
dropped into the media directory) falls through to the generic HTML
indexer, which crashed reading the binary ZIP tail as text. The indexer
now reads bytes and indexes the index.html inside the ZIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.
single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.
The native path and the Archive docker container now run single-file with
"deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.
Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind. The Archive
downloader now has a ten minute timeout. Install scripts install Deno
(previously only upgrade.sh and the image builds did).
* Singlefile writes to a file instead of dumping to stdout.
Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives). single-file now writes its result to a
temporary file which WROLPi reads back.
As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.
* Compressed Archive screenshots and readability images are no longer broken.
The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths. Screenshots are now taken from the fully
extracted ZIP so those references resolve. Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).
* Compressed Archives are created by converting the uncompressed capture.
The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed). When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file. The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages. If the
conversion fails, the uncompressed singlefile is stored instead.
* Consolidating duplicate compressed-singlefile tests.
The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one. The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.
* Addressing review comments: removeprefix, PermissionError when killing.
* Bumping version to 0.24.1-beta.
* Install scripts assert the deno.zip hash to avoid malicious or invalid files.
The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.
* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Archives can be compressed (SingleFileZ self-extracting HTML).
A new compress_singlefile download setting runs single-file with
--compress-content, creating a self-extracting HTML file which is also a
valid ZIP. Only the singlefile .html is compressed; readability files are
always written uncompressed. Readability, screenshots, titles, and URLs
are extracted from the index.html inside the ZIP. A Compress checkbox in
the Archive and RSS download forms remembers the user's choice in local
storage.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:
- wrolpi-chrome.zip Load-Unpacked sideload for Chrome / Brave / Edge
- wrolpi-firefox.xpi Mozilla-signed (unlisted self-distribution),
one-click install in any Firefox
- versions.json Per-browser version metadata for the install page
Backend (modules/archive/api.py, wrolpi/root_api.py):
- POST /api/archive/upload accepts an optional repeated `tagNames`
multipart form field; the upload switch handler applies tags to the
new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
created inside a per-name SAVEPOINT so concurrent inserts on the same
novel name don't abort the transaction (Tag.name has unique=True).
- GET /api/extensions returns metadata (per-file availability + size +
versions.json contents) for the install page.
- GET /api/extensions/<filename> serves the binary, whitelisted by
filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
Content-Disposition: attachment header, so Firefox triggers its
add-on install handler instead of saving the file.
Frontend (app/src/):
- New /admin/extension page (linked from AdminRoute) detects the user's
browser via navigator.userAgent and renders an install card per
Firefox / Chromium / Other. Pulls metadata from /api/extensions to
show file size and version, and gracefully reports 'Not yet
available' when binaries are missing.
- Dashboard's Getters component supports a deep link from the
extension's 'Configure Download' button:
/?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
On mount, the matching downloader form auto-opens with URLs pre-
filled. Query params are cleared on close so a refresh doesn't
re-open the modal.
- DownloadMenu and the per-downloader forms accept an `initialUrls`
seed via the existing `download` prop pattern, joined with newlines
for the `urls` textarea (and first URL for single-URL forms).
Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
