Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
- strip_youtube_caption_positioning now writes to a temp file in the same
directory and os.replace()s the original, so a failed/partial write
(e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
which parses the hostname and matches youtube.com / youtu.be (and
subdomains), rejecting lookalikes like youtube.com.example.org and
notyoutube.com. Add a parametrized test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (via yt-dlp) stamp every cue timing line
with `align:start position:0%`, pinning the on-screen text to the
bottom-left of the player. This strips that exact cue setting from the
downloaded .vtt files so browsers render the captions centered.
- add strip_youtube_caption_positioning() to wrolpi/captions.py — a
targeted text replacement (not a webvtt round-trip) so the `<c>`
word-timing tags in the cue text are preserved; matches only the exact
auto-caption signature, so deliberately positioned cues (e.g. with a
`line` setting) are left untouched; idempotent.
- call it from VideoDownloader after normalize_video_file_names, gated on
a youtube.com / youtu.be source URL. It runs before the FileGroup is
created, so the first index reads the corrected file — no re-index
plumbing needed.
Complements the client-side fix in #543 (which centers captions in the
player for existing/any-source files); this makes new YouTube downloads
correct at the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.
Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.
channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.
Unlike the old implementation, the sort degrades gracefully:
- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
download fails cleanly with UnrecoverableDownloadError telling the
user to update yt-dlp or switch to Newest/Oldest — no retry loop.
Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a Channel refresh, log how many video/audio FileGroups are in the
Channel's directory, and how many are assigned to this Channel, other
Channels, or no Channel. Helps debug Videos missing from their Channel
after files are moved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag_file join in Channel.get_statistics() produced one row per tag,
so a Video with N Tags was counted N times in video_count, size, and
length. Count tagged Videos in a subquery instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
YouTube's new listing layout means yt-dlp no longer returns `view_count`
in flat channel/playlist entries, so sorting by views raised KeyError and
the download failed and retried forever.
Sorting by views now requires per-video extraction (hundreds of requests
per channel), so the feature is deprecated rather than degraded:
- downloader: the 'views' branch raises UnrecoverableDownloadError so the
download fails cleanly (no retry loop) with a message to switch to
Newest/Oldest. May be revisited via the YouTube Data API.
- schema: download_order validation drops 'views' (newest/oldest/null only).
- UI: removed "Most Views" from the download-order dropdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day
Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.
Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
stops dispatching counted downloads for the cycle. daily_download_counts derives the
day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
(base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
validation, 0 = unlimited), and the React Settings page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore the censored-video search filter
Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.
Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored). When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`download_info_json` called `Video.get_by_url(url, session)`, but the method
signature is `get_by_url(session, url)`. This passed the URL string where the
session was expected, raising `AttributeError: 'str' object has no attribute
'query'` whenever a Video metadata refresh ran.
Swap the arguments at both call sites and add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- In get_effective_video_settings: on unparseable sleep_requests, delete
the key instead of forcing 0.0. This lets the global config default
(already present from _extract_global_inheritable) take effect, as
suggested by Greptile.
- In handleInputEvent: add explicit `value !== ''` guard before the
isNaN check. Prevents silent NaN in form state for empty number
fields (isNaN('') === false in JS).
Both changes are strict improvements with no behavior change for
valid inputs.
`sleep_requests` (and other numeric inheritable video download settings)
could arrive as strings ('' or '0.75') from the frontend NumberInputForm
and DownloadRequest bodies. These were persisted into Download.settings
(JSONB) and later triggered:
if effective['sleep_requests'] > 0:
TypeError: '>' not supported between instances of 'str' and 'int'
in VideoDownloader.execute_download (and similar sites).
Root cause: DownloadSettings.__post_init__ only filtered None/[] and did
no type coercion, even though the dataclass declared Optional[float]/int.
Changes (minimal targeted fix):
- Add _coerce_optional_number helper and apply it to all numeric fields
in DownloadSettings.__post_init__ (the existing sanitization boundary).
- Centralize legacy string coercion for persisted data in the single
read chokepoint: get_effective_video_settings.
- Simplify VideoDownloaderConfig.sleep_requests getter.
- Frontend: use parseFloat for type=number inputs + null defaults for
sleep_requests (prevents '' from ever being submitted).
- Add focused regression test for string coercion path.
- Revert unrelated change to string field filtering behavior.
No NaN/Inf speculation, no duplicated try/float guards at every call site,
no behavior change for non-numeric settings.
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
