The kiwix naming scheme is {name}_{YYYY-MM}.zim, and a re-release in the same
month carries a letter: 3dprinting.stackexchange.com_en_all_2026-07a.zim.
parse_name rejected the letter, so such a Zim was never checked for newer
releases and zims.yaml keyed it by its full stem instead of its name. The
letter is now accepted and folded into the day of the month-precision date, so
a re-release orders after the plain release wherever dates are compared. The
pattern also anchors the ".zim" extension instead of matching any character.
A Zim that is not kiwix-named (edibleplantdb.zim) has no date to compare and is
simply not checked. find_outdated_zim_files logged that at WARNING for every
such file at every start; it is a DEBUG line now.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
GET /api/zim went through Zims.get_all, which opened every Zim's Archive in the
API process, and then Zim.__json__ opened each one again for its metadata.
python-libzim holds the GIL and a cold open of a large Zim takes seconds, so
loading the Zims page stalled that API worker for every other request (27.8 s
cold against 0.75 s warm for 44 Zims). #928 moved estimates and searches to the
worker process; the list now uses the same worker for metadata, and a Zim the
worker cannot read is left out as before. count_tagged_zim_entries shares the
new readable_zims helper. Zims.get_all had no other callers and is removed.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A rebuilt database imports zims.yaml before the refresh has modeled a single
Zim. That import has no database version to compare with, so it saves the
config afterward, and the save wrote only the Zims in the database: none. The
file's choices were erased and every Zim was then modeled with the default.
The dump now starts from the file's entries and overlays the database's, so a
choice survives until its Zim is modeled again, and a Zim in the database still
decides its own line.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The toggle was only the zim.auto_search column, which defaults to on. The
database is rebuilt from files and configs, and a Zim's newer version is a new
row, so every rebuild and every subscription update turned search back on for
every Zim. On a large library the global search then searched all of them.
zims.yaml records auto_search for each Zim, keyed by the Zim's name without
its date, so a newer version of a Zim is the same Zim. A Zim the file does not
name is searched (the default stays on). The config is imported on startup
with the usual version gating, saved when the toggle changes, and a newly
modeled Zim takes its setting from it.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Deletes functions nothing in the app calls, along with the tests that only
exercised them. Tests that covered live logic through a dead wrapper now call
the live functions: the Download-linking tests use `domain_download_links` /
`channel_download_links` with `_link_downloads`, the archive tests create
Archives with `write_archive_files` + `register_archive`, and the collection
lifecycle test removes an item with `Collection.remove_item`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deleting a Zim cascade-deletes its TagZimEntry rows, and the next tags
save writes them out of the config, so every outdated-Zim delete lost
its entry tags. The config-import migration only runs at startup, too
late to help. Move each outdated Zim's tags to the current Zim of the
same name before deleting, then save the tags config.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Reset file worker status when a synchronous refresh is cancelled
refresh_sync runs inside request handlers. When Sanic cancelled one on
response timeout, the phase flag cleared as its `with` block unwound but
the shared status did not, so the worker reported 'modeling' forever while
nothing ran. Reset to idle on cancellation.
Deleting outdated Zims also ran the global modelers inside the request,
which is what timed out. Skip post-processing there and update the
outdated_zims flag directly, the only cleanup step it relied on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: observe the modeling flag in the cancelled refresh_sync test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
cachetools 6 removed cachetools.func.mru_cache; the Zim search estimate
cache was its only user, so it becomes an lru_cache and the pin moves
to 7.2.0 with the Dependabot ignore removed.
reportWebVitals still destructured the web-vitals 2.x names (getCLS,
getFID, ...) which no longer exist, so it silently did nothing. Use
the on* names, replace FID with INP, return the import promise, and
accept any callable rather than instanceof Function.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* files: shared modeler batch loop for video, doc, zim, archive
Paging, skip-ids, progress, and cancel yields live in
wrolpi.files.modeler.run_modeler_loop. Each module still owns select /
prepare / apply and its transaction style: video probes outside the write
lock, docs use one write per file with lock retries, zim and archive use
one write per page. Archive HTML that is not a SingleFile raises
SkipModeler so apply_indexers can claim it.
* files: quieter batch-modeler errors; drop unused imports
Break out of a batch page on InvalidRequestError after a flush failure
so the rest of the page does not each log the same poisoned session.
Re-raise PYTEST apply failures without a misleading commit-failure
message. Remove asyncio/List/OperationalError/PYTEST/Iterable leftovers
from docs, archive, and zim.
* files: SQL FileGroup compare, stem column, drop unused discovery path
Refresh walked the tree twice (find, then stat every unchanged group),
and a two-directory refresh scanned the whole media directory. Compare
now diffs in SQL on a TEMP table that already has mtime, accepts multiple
roots, and groups by a stored stem that is versioned so a suffix-rule
change cannot delete-and-reinsert tagged files.
Deletes refresh_discover_paths and /api/files/refresh_progress, which
nothing in production called. Video/doc/zim modelers use the same
(no domain row OR indexed != True) gate.
* files: keep stem current on reorganize; NUL-safe find parse
Reorganize wrote the new primary_path but not stem, so the next refresh
treated the FileGroup as deleted+new and dropped tags. GNU find now
emits NUL-delimited records so a tab or newline in a filename cannot
abort a global refresh. Stem backfill skips the table scan once the
algorithm version matches, and runs off the event loop. An empty roots
list is a global refresh, not invalid SQL.
* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`cmd = ('sudo', '/usr/bin/systemctl', 'restart' 'wrolpi-kiwix.service')`
^ missing comma here
Python implicit string-literal concatenation joined the two strings into
'restartwrolpi-kiwix.service', so systemctl saw one argument it didn't
recognise and exited 2 (INVALIDARGUMENT). Net effect: kiwix never
restarted after a Zim download, the user still had to manually
`sudo systemctl restart wrolpi-kiwix` to see the new Zim.
Adds a unit test that pins the argv shape so an adjacent-literal typo
here surfaces in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Displaying star for Uploader's comments, heart for favorited comments.
Adding wrolpi help files desktop shortcut to rpi image.
Using zim file names as FileGroup.title.
Removing accidental output commit.