A refresh on a large library aborted the whole doc_modeler run on the
first lock error, so the ~20k stranded docs never got modeled.
doc_modeler held one write transaction across a whole batch of 10 docs
while doing slow per-doc extraction (a PDF/epub can be tens of MB). When
another connection committed mid-batch, the deferred transaction failed
to upgrade to a write with SQLITE_BUSY_SNAPSHOT ("database is locked",
which busy_timeout does not absorb -- the same trap that motivated the
short-transaction rewrite of apply_indexers). The per-item error handler
then did `logger.error(f'... {file_group}')`, which lazy-loaded an
attribute on the now-rolled-back session and raised a *second* exception,
escaping the handler and aborting the entire modeler.
Restructure:
- Pick each batch of candidate ids in a short read-only transaction, then
model each doc in its own short write transaction. A lock/error on one
doc rolls back only that doc and cannot poison the rest of the run.
- Retry a doc a few times on "database is locked" -- a fresh transaction
gets a fresh snapshot and succeeds.
- Log the plain file_group id on failure; never touch the ORM object on a
possibly rolled-back session.
- Track only *failed* ids for the termination guard (successes are
excluded by the Doc.id IS NULL gate), keeping the notin_ set small and
well under SQLite's bound-variable limit.
Adds test_doc_modeler_survives_database_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: the new `Doc.id IS NULL` gate keeps matching a
doc whose _model_doc fails without persisting a Doc row. With
>=DOC_PROCESSING_LIMIT such docs, every batch re-selected the same failing
files, `processed` never dropped below the limit, and the loop hung --
starving later docs. Reproduced with a patched _model_doc that always
raises.
Track attempted FileGroup ids per run and exclude them from the query, so
the candidate pool strictly shrinks and the loop always terminates. Failed
docs are retried on the next refresh (unchanged behavior).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc_modeler gated its query on `FileGroup.indexed == False`, but
`indexed` is a full-text-search flag that `apply_indexers` sets on every
FileGroup during a refresh. Because the refresh pipeline runs
apply_modelers -> apply_indexers, any doc file that reached apply_indexers
without first receiving a Doc row (modeler errored, was interrupted, or
files imported already-indexed) got `indexed=1` and was then excluded by
the `indexed == False` gate forever -- leaving it with model=NULL, no Doc
row, and invisible in /api/docs.
On one library this stranded 20,795 PDFs/epubs/mobis (only the 10
individually-tagged docs, modeled via the per-file do_model path, showed
up). No doc-mimetype files were left with indexed=0, so nothing would
ever recover them.
Gate the modeler on "no Doc row yet OR explicitly flagged for
re-indexing" instead, so already-indexed docs are picked up on the next
refresh while preserving re-index behavior and skipping already-modeled
docs. Self-healing: a global refresh now models the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
