Both Zim search endpoints handed an empty request to the worker-job builder,
whose RuntimeError surfaced as a 500 "Unexpected error" with a traceback. The
request is the client's mistake: the handlers now raise ValidationError, as the
other endpoints do, before any work is scheduled.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The kiwix naming scheme is {name}_{YYYY-MM}.zim, and a re-release in the same
month carries a letter: 3dprinting.stackexchange.com_en_all_2026-07a.zim.
parse_name rejected the letter, so such a Zim was never checked for newer
releases and zims.yaml keyed it by its full stem instead of its name. The
letter is now accepted and folded into the day of the month-precision date, so
a re-release orders after the plain release wherever dates are compared. The
pattern also anchors the ".zim" extension instead of matching any character.
A Zim that is not kiwix-named (edibleplantdb.zim) has no date to compare and is
simply not checked. find_outdated_zim_files logged that at WARNING for every
such file at every start; it is a DEBUG line now.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
GET /api/zim went through Zims.get_all, which opened every Zim's Archive in the
API process, and then Zim.__json__ opened each one again for its metadata.
python-libzim holds the GIL and a cold open of a large Zim takes seconds, so
loading the Zims page stalled that API worker for every other request (27.8 s
cold against 0.75 s warm for 44 Zims). #928 moved estimates and searches to the
worker process; the list now uses the same worker for metadata, and a Zim the
worker cannot read is left out as before. count_tagged_zim_entries shares the
new readable_zims helper. Zims.get_all had no other callers and is removed.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Tests fail when a request writes outside request.ctx.db.write()
A request's transaction begins deferred. A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent. #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.
Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test. allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.
It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
row's delete when the middleware's commit failed under contention. It
now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
Session, after the request ended. set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
deferred get_db_session(). It now begins as a writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tasks don't inherit a declared write; tags import change backed out
From review of #864:
- background_task's task copied its creator's Context, including
_write_intent when started inside get_db_session(commit=True) or
RequestDB.write(), so the guard ignored the task's own writes for its
lifetime. The task now starts with no declared write. That exposes
fetch_channel_source_id (started by the channels config import): it read
the Channel, asked the remote site for its source_id, then committed in
the same deferred transaction. It now reads, fetches with no
transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
walk, and its exit commit saved changes the need_commit gate used to
drop. Reverted to master; splitting its reads from a short write is its
own change. The two tests that run it through a request are marked
allow_deferred_request_writes until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between. They now write inside request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
* A new zim module has been added, this module allows several things to interact with Zim files.
* Allow searching of Zim files using libzim.
* Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
* Creation of a new zim docker container which allows viewing of the Zim file contents.
* This commit also adds move/rename/mkdir features in the File Browser.
* File tags are preserved when moving/renaming.