Addresses review feedback: the new `Doc.id IS NULL` gate keeps matching a
doc whose _model_doc fails without persisting a Doc row. With
>=DOC_PROCESSING_LIMIT such docs, every batch re-selected the same failing
files, `processed` never dropped below the limit, and the loop hung --
starving later docs. Reproduced with a patched _model_doc that always
raises.
Track attempted FileGroup ids per run and exclude them from the query, so
the candidate pool strictly shrinks and the loop always terminates. Failed
docs are retried on the next refresh (unchanged behavior).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc_modeler gated its query on `FileGroup.indexed == False`, but
`indexed` is a full-text-search flag that `apply_indexers` sets on every
FileGroup during a refresh. Because the refresh pipeline runs
apply_modelers -> apply_indexers, any doc file that reached apply_indexers
without first receiving a Doc row (modeler errored, was interrupted, or
files imported already-indexed) got `indexed=1` and was then excluded by
the `indexed == False` gate forever -- leaving it with model=NULL, no Doc
row, and invisible in /api/docs.
On one library this stranded 20,795 PDFs/epubs/mobis (only the 10
individually-tagged docs, modeled via the per-file do_model path, showed
up). No doc-mimetype files were left with indexed=0, so nothing would
ever recover them.
Gate the modeler on "no Doc row yet OR explicitly flagged for
re-indexing" instead, so already-indexed docs are picked up on the next
refresh while preserving re-index behavior and skipping already-modeled
docs. Self-healing: a global refresh now models the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
