Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
|
|
|
import asyncio
|
2023-05-29 18:04:03 -06:00
|
|
|
import json
|
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
|
|
|
import sys
|
|
|
|
|
from unittest import mock
|
2023-05-29 18:04:03 -06:00
|
|
|
from http import HTTPStatus
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from modules.zim import lib
|
|
|
|
|
from modules.zim.models import Zim
|
2026-09-12 15:18:13 -06:00
|
|
|
from wrolpi import flags
|
2026-10-21 16:48:39 -06:00
|
|
|
from wrolpi.conftest import production_like_sessions, write_lock_held_briefly
|
2023-05-29 18:04:03 -06:00
|
|
|
from wrolpi.downloader import Download
|
|
|
|
|
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_zim_search(async_client, test_session, test_zim, tag_factory, assert_zim_search):
|
2023-05-29 18:04:03 -06:00
|
|
|
"""Entries can be tagged and retrieved using their containing Zim."""
|
2024-08-15 19:32:01 -06:00
|
|
|
tag1, tag2 = await tag_factory(), await tag_factory()
|
2025-12-10 21:16:10 -07:00
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'one')
|
|
|
|
|
test_zim.tag_entry(test_session, tag2.name, 'one')
|
|
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'two')
|
2023-05-29 18:04:03 -06:00
|
|
|
test_session.commit()
|
|
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
# Test searching with only `search_str`. (Ranks/headlines come from FTS5 `headline_texts`.)
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('one', 1, {
|
2023-05-29 18:04:03 -06:00
|
|
|
'path': test_zim.path,
|
|
|
|
|
'search': [
|
|
|
|
|
# Entry "One" contains a 'one' in the title, and the text.
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
{'path': 'one', 'headline': 'This is the first item', 'rank': 1e-06},
|
2023-05-29 18:04:03 -06:00
|
|
|
# Home only contains one 'one'.
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
{'path': 'home', 'rank': 8.8e-07, 'headline': '…an example ZIM file for testing.\n<b>One</b>\nTwo', },
|
2023-05-29 18:04:03 -06:00
|
|
|
],
|
|
|
|
|
'estimate': 2,
|
|
|
|
|
})
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('two', 1, {
|
2023-05-29 18:04:03 -06:00
|
|
|
'path': test_zim.path,
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
'search': [{'path': 'two', 'headline': 'This is the second item', 'rank': 1e-06},
|
|
|
|
|
{'path': 'home', 'headline': '…One\n<b>Two</b>', 'rank': 8.8e-07}],
|
2023-05-29 18:04:03 -06:00
|
|
|
'estimate': 2,
|
|
|
|
|
})
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('item', 1, {
|
2023-05-29 18:04:03 -06:00
|
|
|
'path': test_zim.path,
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
'search': [{'path': 'one', 'headline': 'This is the first <b>item</b>', 'rank': 1e-06},
|
|
|
|
|
{'path': 'two', 'headline': 'This is the second <b>item</b>', 'rank': 1e-06}],
|
2023-05-29 18:04:03 -06:00
|
|
|
'estimate': 2,
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
# Test searching with `search_str` and `tag_names`.
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('one', 1, tag_names=[tag2.name, ], expected={
|
2023-05-29 18:04:03 -06:00
|
|
|
'search': [
|
|
|
|
|
# Only entry "One" is tagged with Tag 2.
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
{'path': 'one', 'headline': 'This is the first item', 'rank': 1e-06},
|
2023-05-29 18:04:03 -06:00
|
|
|
],
|
|
|
|
|
'estimate': 1,
|
|
|
|
|
})
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('one', 1, tag_names=[tag1.name, tag2.name], expected={
|
2023-05-29 18:04:03 -06:00
|
|
|
'search': [
|
|
|
|
|
# Only "One" is tagged with Tag 1 and contains "one".
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
{'path': 'one', 'headline': 'This is the first item', 'rank': 1e-06},
|
2023-05-29 18:04:03 -06:00
|
|
|
],
|
|
|
|
|
'estimate': 1,
|
|
|
|
|
})
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('two', 1, tag_names=[tag1.name, ], expected={
|
2023-05-29 18:04:03 -06:00
|
|
|
'search': [
|
|
|
|
|
# Only "Two" was tagged with Tag 1 and contains "two".
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
{'path': 'two', 'headline': 'This is the second item', 'rank': 1e-06},
|
2023-05-29 18:04:03 -06:00
|
|
|
],
|
|
|
|
|
'estimate': 1,
|
|
|
|
|
})
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('two', 1, tag_names=[tag2.name, ], expected={
|
2023-05-29 18:04:03 -06:00
|
|
|
# Two was not tagged with Tag 1.
|
|
|
|
|
'search': [],
|
|
|
|
|
'estimate': 0,
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
# Test search with only `tag_names`.
|
2024-08-15 19:32:01 -06:00
|
|
|
await assert_zim_search('', 1, tag_names=[tag1.name, tag2.name], expected={
|
2023-05-29 18:04:03 -06:00
|
|
|
'search': [
|
|
|
|
|
# Return all entries that are tagged with the Tags.
|
|
|
|
|
{'path': 'one', 'headline': 'This is the first item', 'rank': 0.0, 'tag_names': ['one', 'two']},
|
|
|
|
|
],
|
|
|
|
|
'estimate': 1,
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_entries_by_tag(async_client, test_session, test_zim, tag_factory):
|
2023-05-29 18:04:03 -06:00
|
|
|
"""Entries can be tagged and retrieved using their containing Zim."""
|
2024-08-15 19:32:01 -06:00
|
|
|
tag1, tag2 = await tag_factory(), await tag_factory()
|
2025-12-10 21:16:10 -07:00
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'one')
|
|
|
|
|
test_zim.tag_entry(test_session, tag2.name, 'one')
|
|
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'two')
|
2023-05-29 18:04:03 -06:00
|
|
|
test_session.commit()
|
|
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
result = test_zim.entries_with_tags(test_session, [tag1.name, tag2.name])
|
2023-05-29 18:04:03 -06:00
|
|
|
assert [(i.path, i.title) for i in result] == [('one', 'One'), ]
|
|
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
result = test_zim.entries_with_tags(test_session, [tag1.name, ])
|
2023-05-29 18:04:03 -06:00
|
|
|
assert [(i.path, i.title) for i in result] == [('one', 'One'), ('two', 'Two')]
|
|
|
|
|
|
2025-12-10 21:16:10 -07:00
|
|
|
result = test_zim.entries_with_tags(test_session, [tag2.name, ])
|
2023-05-29 18:04:03 -06:00
|
|
|
assert [(i.path, i.title) for i in result] == [('one', 'One'), ]
|
|
|
|
|
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_zim_crud(async_client, test_session, test_directory, test_zim):
|
|
|
|
|
request, response = await async_client.get('/api/zim')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert len(response.json['zims']) == 1
|
|
|
|
|
assert response.json['zims'][0]['path'] == test_zim.path.name
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.delete('/api/zim/1')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
assert not test_zim.path.exists()
|
|
|
|
|
assert test_session.query(Zim).count() == 0
|
|
|
|
|
|
|
|
|
|
|
2024-08-15 19:32:01 -06:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_zim_tag_and_untag(async_client, test_session, test_zim, tag_factory):
|
2023-05-29 18:04:03 -06:00
|
|
|
"""Zim entries can be Tagged and Untagged."""
|
2024-08-15 19:32:01 -06:00
|
|
|
tag1, tag2 = await tag_factory(), await tag_factory()
|
2023-05-29 18:04:03 -06:00
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
# Entries can be tagged in the API.
|
|
|
|
|
content = dict(tag_name=tag1.name, zim_id=test_zim.id, zim_entry='home')
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.post('/api/zim/tag', content=json.dumps(content))
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
|
|
|
|
|
|
|
|
# Entries can be untagged in the API.
|
|
|
|
|
content = dict(tag_name=tag1.name, zim_id=test_zim.id, zim_entry='home')
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.post('/api/zim/untag', content=json.dumps(content))
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
|
|
|
|
|
# Cannot untag twice.
|
|
|
|
|
content = dict(tag_name=tag1.name, zim_id=test_zim.id, zim_entry='home')
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.post('/api/zim/untag', content=json.dumps(content))
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_zim_subscribe(test_session, async_client):
|
2023-05-29 18:04:03 -06:00
|
|
|
"""A Kiwix subscription can be scheduled in the API. The language can be changed."""
|
|
|
|
|
# Subscribe to English.
|
|
|
|
|
content = dict(name='Wikipedia (mini)', language='en')
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.post('/api/zim/subscribe', content=json.dumps(content))
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
|
|
|
|
|
|
|
|
download: Download = test_session.query(Download).one()
|
|
|
|
|
assert download.url == 'https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_mini_'
|
|
|
|
|
assert download.info_json == {'language': 'en', 'name': 'Wikipedia (mini)'}
|
|
|
|
|
assert download.frequency
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.get('/api/zim/subscribe')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert response.json['subscriptions'] == {
|
|
|
|
|
'Wikipedia (mini)': {'download_id': 1,
|
|
|
|
|
'download_url': 'https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_mini_',
|
|
|
|
|
'id': 1,
|
|
|
|
|
'language': 'en',
|
|
|
|
|
'name': 'Wikipedia (mini)'}}
|
|
|
|
|
|
|
|
|
|
# Change subscription to French.
|
|
|
|
|
content = dict(name='Wikipedia (mini)', language='fr')
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.post('/api/zim/subscribe', content=json.dumps(content))
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
|
|
|
assert test_session.query(Download).count() == 1
|
|
|
|
|
|
|
|
|
|
download: Download = test_session.query(Download).one()
|
|
|
|
|
assert download.url == 'https://download.kiwix.org/zim/wikipedia/wikipedia_fr_all_mini_'
|
|
|
|
|
assert download.info_json == {'language': 'fr', 'name': 'Wikipedia (mini)'}
|
|
|
|
|
assert download.frequency
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.get('/api/zim/subscribe')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert response.json['subscriptions'] == {
|
|
|
|
|
'Wikipedia (mini)': {'download_id': 1,
|
|
|
|
|
'download_url': 'https://download.kiwix.org/zim/wikipedia/wikipedia_fr_all_mini_',
|
|
|
|
|
'id': 1,
|
|
|
|
|
'language': 'fr',
|
|
|
|
|
'name': 'Wikipedia (mini)'}}
|
|
|
|
|
|
|
|
|
|
# Delete subscription.
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.delete('/api/zim/subscribe/1')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.get('/api/zim/subscribe')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert response.json['subscriptions'] == {}
|
|
|
|
|
# No subscriptions or downloads.
|
|
|
|
|
assert test_session.query(lib.ZimSubscription).count() == 0
|
|
|
|
|
assert test_session.query(Download).count() == 0
|
|
|
|
|
|
2024-02-12 13:05:14 -07:00
|
|
|
# English Stack Exchange is special.
|
|
|
|
|
content = dict(name='Stackoverflow (Stack Exchange)', language='en')
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.post('/api/zim/subscribe', content=json.dumps(content))
|
2024-02-12 13:05:14 -07:00
|
|
|
assert response.status_code == HTTPStatus.CREATED
|
|
|
|
|
download: Download = test_session.query(Download).one()
|
|
|
|
|
assert download.url == 'https://download.kiwix.org/zim/stack_exchange/stackoverflow.com_en_all_'
|
|
|
|
|
|
2023-05-29 18:04:03 -06:00
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_get_zim_entry(test_session, async_client, test_zim):
|
|
|
|
|
request, response = await async_client.get('/api/zim/1/entry/home')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
request, response = await async_client.get('/api/zim/1/entry/does not exist')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NOT_FOUND
|
|
|
|
|
|
|
|
|
|
|
2026-03-28 16:04:33 -06:00
|
|
|
def test_parse_name():
|
|
|
|
|
"""Zim file names that match the kiwix naming scheme are parsed. Non-matching names raise ValueError."""
|
|
|
|
|
import pathlib
|
|
|
|
|
from datetime import datetime
|
|
|
|
|
|
|
|
|
|
name, date = lib.parse_name(pathlib.Path('wikipedia_en_all_maxi_2023-01.zim'))
|
|
|
|
|
assert name == 'wikipedia_en_all_maxi'
|
|
|
|
|
assert date == datetime(2023, 1, 1)
|
|
|
|
|
|
|
|
|
|
with pytest.raises(ValueError):
|
|
|
|
|
lib.parse_name(pathlib.Path('edibleplantdb.zim'))
|
|
|
|
|
|
|
|
|
|
with pytest.raises(ValueError):
|
|
|
|
|
lib.parse_name(pathlib.Path('nodate.zim'))
|
|
|
|
|
|
|
|
|
|
|
2023-05-29 18:04:03 -06:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_find_outdated_zim_files(test_session, test_directory, test_zim_bytes, async_client):
|
2023-05-29 18:04:03 -06:00
|
|
|
"""Outdated Zim files are found and reported. They can automatically be deleted."""
|
|
|
|
|
# Can search empty directory.
|
|
|
|
|
outdated, current = lib.find_outdated_zim_files()
|
|
|
|
|
assert not outdated and not current
|
|
|
|
|
|
|
|
|
|
(test_directory / 'zims/books').mkdir(parents=True)
|
|
|
|
|
|
|
|
|
|
wikibooks_one = test_directory / 'zims/books/wikibooks_en_nopic_maxi_2023-01.zim' # unique
|
|
|
|
|
wikipedia_one = test_directory / 'zims/wikipedia_en_all_maxi_2023-01.zim' # outdated
|
|
|
|
|
wikipedia_two = test_directory / 'zims/wikipedia_en_all_maxi_2023-02.zim' # keep this one
|
|
|
|
|
asdf = test_directory / 'zims/asdf.zim' # Name cannot be parsed.
|
|
|
|
|
wikibooks_one.write_bytes(test_zim_bytes)
|
|
|
|
|
wikipedia_one.write_bytes(test_zim_bytes)
|
|
|
|
|
wikipedia_two.write_bytes(test_zim_bytes)
|
|
|
|
|
asdf.write_bytes(test_zim_bytes)
|
|
|
|
|
empty = test_directory / 'zims/wikipedia_en_all_maxi_2023-03.zim' # empty files should be ignored.
|
|
|
|
|
empty.touch()
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.get('/api/zim/outdated')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert response.json['outdated'] == [str(wikipedia_one.relative_to(test_directory)), ]
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.delete('/api/zim/outdated')
|
2023-05-29 18:04:03 -06:00
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
# Only the outdated was deleted.
|
|
|
|
|
assert wikibooks_one.is_file()
|
|
|
|
|
assert not wikipedia_one.exists()
|
|
|
|
|
assert wikipedia_two.is_file()
|
|
|
|
|
assert asdf.is_file()
|
|
|
|
|
assert empty.is_file()
|
2024-01-01 11:31:17 -07:00
|
|
|
|
|
|
|
|
|
2026-09-12 15:18:13 -06:00
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_delete_outdated_zims_clears_stale_flag(test_session, test_directory, test_zim_bytes, async_client,
|
|
|
|
|
flags_lock):
|
|
|
|
|
"""Deleting outdated Zims always leaves the `outdated_zims` flag matching disk, even when nothing was deleted."""
|
|
|
|
|
(test_directory / 'zims').mkdir(parents=True)
|
|
|
|
|
(test_directory / 'zims/wikipedia_en_all_maxi_2023-02.zim').write_bytes(test_zim_bytes)
|
|
|
|
|
flags.outdated_zims.set()
|
|
|
|
|
|
|
|
|
|
request, response = await async_client.get('/api/zim/outdated')
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert response.json['outdated'] == []
|
|
|
|
|
|
|
|
|
|
request, response = await async_client.delete('/api/zim/outdated')
|
|
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
assert not flags.outdated_zims.is_set(), 'Flag should be recomputed even when nothing was deleted'
|
|
|
|
|
|
|
|
|
|
|
2024-01-01 11:31:17 -07:00
|
|
|
@pytest.mark.asyncio
|
2024-09-15 15:33:07 -06:00
|
|
|
async def test_search_estimates(test_session, async_client, zim_factory):
|
2024-01-01 11:31:17 -07:00
|
|
|
zim_factory('test zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
body = dict(search_str='one')
|
2024-09-15 15:33:07 -06:00
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
2024-01-01 11:31:17 -07:00
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert len(response.json['zims_estimates']) == 1
|
|
|
|
|
assert response.json['zims_estimates'][0]['estimate'] == 2
|
2025-02-02 10:16:11 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_zim_auto_search(test_session, async_client, zim_factory):
|
|
|
|
|
"""The auto-searching of any Zim file can be toggled."""
|
|
|
|
|
zim = zim_factory('test_zim_auto_search.zim')
|
|
|
|
|
assert zim.auto_search is True, 'Zim should be searched by default.'
|
|
|
|
|
|
|
|
|
|
# Zim is estimated by default.
|
|
|
|
|
body = dict(search_str='one')
|
|
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert len(response.json['zims_estimates']) == 1
|
|
|
|
|
|
|
|
|
|
# Change auto_search so Zim is not estimated by default.
|
|
|
|
|
body = dict(auto_search=False)
|
|
|
|
|
request, response = await async_client.post(f'/api/zim/{zim.id}/auto_search', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
assert zim.auto_search is False
|
|
|
|
|
|
|
|
|
|
# Estimates are empty because the only Zim was filtered out.
|
|
|
|
|
body = dict(search_str='one')
|
|
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert len(response.json['zims_estimates']) == 0
|
|
|
|
|
|
|
|
|
|
# Change auto_search so Zim is now estimated again.
|
|
|
|
|
body = dict(auto_search=True)
|
|
|
|
|
request, response = await async_client.post(f'/api/zim/{zim.id}/auto_search', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT
|
|
|
|
|
assert zim.auto_search is True
|
|
|
|
|
|
|
|
|
|
# Zim is estimated by default.
|
|
|
|
|
body = dict(search_str='one')
|
|
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert len(response.json['zims_estimates']) == 1
|
2026-10-21 16:48:39 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
@pytest.mark.parametrize('step', ['auto_search', 'tag', 'untag', 'subscribe', 'unsubscribe'])
|
|
|
|
|
async def test_zim_writes_wait_for_a_concurrent_writer(async_client, test_session, test_zim, tag_factory, step):
|
|
|
|
|
"""These requests read before they write, so each must begin as a writer: SQLite fails a deferred
|
|
|
|
|
transaction's lock upgrade instantly ("database is locked") if another connection wrote meanwhile."""
|
|
|
|
|
tag = await tag_factory()
|
|
|
|
|
test_session.commit()
|
|
|
|
|
entry = dict(tag_name=tag.name, zim_id=test_zim.id, zim_entry='home')
|
|
|
|
|
if step == 'untag':
|
|
|
|
|
await lib.add_tag(test_session, tag.name, test_zim.id, 'home')
|
|
|
|
|
if step == 'unsubscribe':
|
|
|
|
|
await lib.subscribe(test_session, 'Wikipedia (mini)', 'en')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
db_file = test_session.get_bind().url.database
|
|
|
|
|
|
|
|
|
|
with production_like_sessions(test_session):
|
|
|
|
|
# Held past any slow work a request does before it writes.
|
|
|
|
|
with write_lock_held_briefly(db_file, seconds=3):
|
|
|
|
|
if step == 'auto_search':
|
|
|
|
|
_, response = await async_client.post(f'/api/zim/{test_zim.id}/auto_search',
|
|
|
|
|
content=json.dumps(dict(auto_search=False)))
|
|
|
|
|
elif step in ('tag', 'untag'):
|
|
|
|
|
_, response = await async_client.post(f'/api/zim/{step}', content=json.dumps(entry))
|
|
|
|
|
elif step == 'subscribe':
|
|
|
|
|
_, response = await async_client.post(
|
|
|
|
|
'/api/zim/subscribe', content=json.dumps(dict(name='Wikipedia (mini)', language='en')))
|
|
|
|
|
else:
|
|
|
|
|
_, response = await async_client.delete('/api/zim/subscribe/1')
|
|
|
|
|
|
|
|
|
|
assert response.status_code in (HTTPStatus.CREATED, HTTPStatus.NO_CONTENT), response.body
|
Tests fail when a request writes outside request.ctx.db.write() (#864)
* Tests fail when a request writes outside request.ctx.db.write()
A request's transaction begins deferred. A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent. #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.
Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test. allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.
It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
row's delete when the middleware's commit failed under contention. It
now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
Session, after the request ended. set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
deferred get_db_session(). It now begins as a writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tasks don't inherit a declared write; tags import change backed out
From review of #864:
- background_task's task copied its creator's Context, including
_write_intent when started inside get_db_session(commit=True) or
RequestDB.write(), so the guard ignored the task's own writes for its
lifetime. The task now starts with no declared write. That exposes
fetch_channel_source_id (started by the channels config import): it read
the Channel, asked the remote site for its source_id, then committed in
the same deferred transaction. It now reads, fetches with no
transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
walk, and its exit commit saved changes the need_commit gate used to
drop. Reverted to master; splitting its reads from a short write is its
own change. The two tests that run it through a request are marked
allow_deferred_request_writes until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 17:09:22 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_delete_zim_waits_for_a_concurrent_writer(async_client, test_session, test_zim):
|
|
|
|
|
"""Deleting a Zim must delete its row as well as its file, even while something else is writing. It wrote
|
|
|
|
|
in the request's deferred transaction, so its commit in the response middleware failed ("database is
|
|
|
|
|
locked") after the file was deleted and the response was sent."""
|
|
|
|
|
zim_id, zim_path = test_zim.id, test_zim.path
|
|
|
|
|
test_session.commit()
|
|
|
|
|
db_file = test_session.get_bind().url.database
|
|
|
|
|
|
|
|
|
|
with production_like_sessions(test_session) as maker:
|
|
|
|
|
with write_lock_held_briefly(db_file, seconds=3):
|
|
|
|
|
_, response = await async_client.delete(f'/api/zim/{zim_id}')
|
|
|
|
|
assert response.status_code == HTTPStatus.NO_CONTENT, response.body
|
|
|
|
|
|
|
|
|
|
assert not zim_path.exists()
|
|
|
|
|
with maker() as session:
|
|
|
|
|
assert session.query(Zim).filter_by(id=zim_id).one_or_none() is None
|
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_estimate_zims_runs_outside_the_api_process(test_session, zim_factory):
|
|
|
|
|
"""libzim holds the GIL, so estimates run in a separate process rather than a thread of the API worker."""
|
|
|
|
|
zim = zim_factory('test zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
with mock.patch('modules.zim.lib.asyncio.create_subprocess_exec', wraps=asyncio.create_subprocess_exec) as m:
|
|
|
|
|
counts = await lib.estimate_zims(test_session, 'one')
|
|
|
|
|
assert counts == {zim: 2}
|
|
|
|
|
assert m.call_args.args[:2] == (sys.executable, '-I')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_estimate_zims_skips_a_zim_it_cannot_read(test_session, zim_factory):
|
2026-11-02 18:19:22 -07:00
|
|
|
"""A Zim the estimate process cannot open is left out of the estimates, as the Zim list leaves it out."""
|
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
|
|
|
good = zim_factory('good zim')
|
|
|
|
|
bad = zim_factory('bad zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
bad.path.write_bytes(b'not a zim file')
|
|
|
|
|
|
|
|
|
|
counts = await lib.estimate_zims(test_session, 'one')
|
|
|
|
|
assert counts == {good: 2}
|
|
|
|
|
|
|
|
|
|
bad.path.unlink()
|
|
|
|
|
counts = await lib.estimate_zims(test_session, 'one')
|
|
|
|
|
assert counts == {good: 2}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_zim_search_runs_outside_the_api_process(async_client, test_session, test_zim):
|
|
|
|
|
"""Zim searches, like estimates, do all their libzim work in the worker script's process."""
|
|
|
|
|
with mock.patch('modules.zim.lib.asyncio.create_subprocess_exec', wraps=asyncio.create_subprocess_exec) as m:
|
|
|
|
|
request, response = await async_client.post(f'/api/zim/search/{test_zim.id}', json=dict(search_str='one'))
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert [i['path'] for i in response.json['zim']['search']] == ['one', 'home']
|
|
|
|
|
assert response.json['zim']['metadata']['title'], 'Metadata should come from the worker'
|
|
|
|
|
|
|
|
|
|
request, response = await async_client.post('/api/zim/search', json=dict(search_str='one'))
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert [i['path'] for i in response.json['zims'][0]['search']] == ['one', 'home']
|
|
|
|
|
assert m.call_count == 2
|
|
|
|
|
assert all(call.args[:2] == (sys.executable, '-I') for call in m.call_args_list)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_search_all_zims_skips_a_zim_it_cannot_read(async_client, test_session, zim_factory):
|
|
|
|
|
"""One unreadable Zim does not fail the search of all Zims."""
|
|
|
|
|
good = zim_factory('good zim')
|
|
|
|
|
bad = zim_factory('bad zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
bad.path.write_bytes(b'not a zim file')
|
|
|
|
|
|
|
|
|
|
request, response = await async_client.post('/api/zim/search', json=dict(search_str='one'))
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
zims = response.json['zims']
|
|
|
|
|
assert len(zims) == 1 and zims[0]['path'].endswith('good zim')
|
|
|
|
|
assert zims[0]['estimate'] == 2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_tagged_search_estimates_do_not_open_zims(async_client, test_session, test_zim, tag_factory):
|
|
|
|
|
"""Estimates with tags count from the database; the API process never opens a Zim, even for its metadata."""
|
|
|
|
|
tag1, tag2 = await tag_factory(), await tag_factory()
|
|
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'one')
|
|
|
|
|
test_zim.tag_entry(test_session, tag1.name, 'two')
|
|
|
|
|
test_zim.tag_entry(test_session, tag2.name, 'one')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
with mock.patch('modules.zim.models.Zim.get_zim', side_effect=AssertionError('opened a Zim in the API')):
|
|
|
|
|
body = dict(search_str='one', tag_names=[tag1.name])
|
|
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert [(i['estimate'], bool(i['metadata']['title'])) for i in response.json['zims_estimates']] \
|
|
|
|
|
== [(2, True)]
|
|
|
|
|
|
|
|
|
|
body = dict(search_str='', tag_names=[tag1.name, tag2.name])
|
|
|
|
|
request, response = await async_client.post('/api/zim/search_estimates', json=body)
|
|
|
|
|
assert [i['estimate'] for i in response.json['zims_estimates']] == [1]
|
|
|
|
|
|
|
|
|
|
|
2026-11-02 18:19:22 -07:00
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_zim_list_does_not_open_zims(async_client, test_session, zim_factory):
|
|
|
|
|
"""Listing Zims reads their metadata in the worker; the API process never opens a Zim. A cold open takes
|
|
|
|
|
seconds per Zim and libzim holds the GIL, so the Zims page used to stall its API worker."""
|
|
|
|
|
zim_factory('one zim')
|
|
|
|
|
zim_factory('two zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
with mock.patch('modules.zim.models.Zim.get_zim', side_effect=AssertionError('opened a Zim in the API')):
|
|
|
|
|
request, response = await async_client.get('/api/zim')
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
zims = response.json['zims']
|
|
|
|
|
assert [i['path'].endswith(('one zim', 'two zim')) for i in zims] == [True, True]
|
|
|
|
|
assert all(i['metadata']['title'] for i in zims), 'Metadata should come from the worker'
|
|
|
|
|
assert all(i['size'] for i in zims)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_zim_list_skips_a_zim_it_cannot_read(async_client, test_session, zim_factory):
|
|
|
|
|
"""A Zim the worker cannot read, or whose file is gone, is left out of the list; the rest are listed."""
|
|
|
|
|
good = zim_factory('good zim')
|
|
|
|
|
bad = zim_factory('bad zim')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
bad.path.write_bytes(b'not a zim file')
|
|
|
|
|
|
|
|
|
|
request, response = await async_client.get('/api/zim')
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert [i['path'].endswith('good zim') for i in response.json['zims']] == [True]
|
|
|
|
|
|
|
|
|
|
bad.path.unlink()
|
|
|
|
|
request, response = await async_client.get('/api/zim')
|
|
|
|
|
assert response.status_code == HTTPStatus.OK
|
|
|
|
|
assert [i['path'].endswith('good zim') for i in response.json['zims']] == [True]
|
|
|
|
|
assert good.path.is_file()
|
|
|
|
|
|
|
|
|
|
|
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_cancelled_libzim_jobs_kill_the_worker(test_directory):
|
|
|
|
|
"""Sanic cancels a handler at its response timeout or on disconnect; the worker process must not outlive it."""
|
|
|
|
|
slow_worker = test_directory / 'slow_worker.py'
|
|
|
|
|
slow_worker.write_text('import time; time.sleep(60)')
|
|
|
|
|
procs = []
|
|
|
|
|
create_subprocess_exec = asyncio.create_subprocess_exec
|
|
|
|
|
|
|
|
|
|
async def spy(*args, **kwargs):
|
|
|
|
|
proc = await create_subprocess_exec(*args, **kwargs)
|
|
|
|
|
procs.append(proc)
|
|
|
|
|
return proc
|
|
|
|
|
|
|
|
|
|
with mock.patch('modules.zim.lib.LIBZIM_WORKER', slow_worker), \
|
|
|
|
|
mock.patch('modules.zim.lib.asyncio.create_subprocess_exec', side_effect=spy):
|
|
|
|
|
task = asyncio.create_task(lib.run_libzim_jobs([]))
|
|
|
|
|
while not procs:
|
|
|
|
|
await asyncio.sleep(0.01)
|
|
|
|
|
task.cancel()
|
|
|
|
|
with pytest.raises(asyncio.CancelledError):
|
|
|
|
|
await task
|
|
|
|
|
|
|
|
|
|
assert procs[0].returncode is not None, 'The worker was killed and reaped'
|