Commit graph

176 commits

Author SHA1 Message Date
Roland
f5615440e5
videos: address review — atomic caption write + strict YouTube host gate
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
- strip_youtube_caption_positioning now writes to a temp file in the same
  directory and os.replace()s the original, so a failed/partial write
  (e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
  which parses the hostname and matches youtube.com / youtu.be (and
  subdomains), rejecting lookalikes like youtube.com.example.org and
  notyoutube.com. Add a parametrized test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 18:51:51 -06:00
Roland
3967a49989
SQLite: make the browse pages index-only.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.

- Browse archives with file_group as the outer table (CROSS JOIN) so
  SQLite walks the date index and probes archive's covering index;
  driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
  rowid is present in every index, the path expression forced full-row
  reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
  mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
  sqlite_stat1 at all after the rebuild.

Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 07:46:29 -06:00
Roland
d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00
Roland
cdd28d0a39
Videos: a Channel's directory beats source_id when assigning a Video.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.

Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.

channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 18:33:15 -06:00
Roland
f65f0aca6b
Restore "Most Views" download order
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.

Unlike the old implementation, the sort degrades gracefully:

- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
  download fails cleanly with UnrecoverableDownloadError telling the
  user to update yt-dlp or switch to Newest/Oldest — no retry loop.

Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:34:50 -06:00
Roland
73fafcd645
Search: fast ABC mode by default, opt-in deep search
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries.  Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.

Database:
- Migration adds the generated column and a GIN index (rewrites the
  table; a few minutes on large libraries).

Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
  DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
  query (COUNT FILTER on the abc subset): `file_groups` (fast) and
  `file_groups_deep`.

UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
  list.
- The gap is reconciled by a hint offering deep search: count-aware on
  the global search ("No results in fast search. N results available in
  deep search." / "N more results available..."), generic on the module
  pages (no deep estimate there).  The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
  ?deep=true.

Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside.  Also skip flaky test_download_channel on
CircleCI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 10:32:16 -06:00
Roland
fcb64fed13
Deprecate "Most Views" download order
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
YouTube's new listing layout means yt-dlp no longer returns `view_count`
in flat channel/playlist entries, so sorting by views raised KeyError and
the download failed and retried forever.

Sorting by views now requires per-video extraction (hundreds of requests
per channel), so the feature is deprecated rather than degraded:

- downloader: the 'views' branch raises UnrecoverableDownloadError so the
  download fails cleanly (no retry loop) with a message to switch to
  Newest/Oldest. May be revisited via the YouTube Data API.
- schema: download_order validation drops 'views' (newest/oldest/null only).
- UI: removed "Most Views" from the download-order dropdown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 18:06:34 -06:00
lrnselfreliance
cafc6514cb
Restore the censored-video search filter (#499)
* Restore the censored-video search filter

Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.

Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored).  When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 09:48:36 -06:00
Roland
5427beb5fb Fix swapped arguments in download_info_json Video.get_by_url call
`download_info_json` called `Video.get_by_url(url, session)`, but the method
signature is `get_by_url(session, url)`. This passed the URL string where the
session was expected, raising `AttributeError: 'str' object has no attribute
'query'` whenever a Video metadata refresh ran.

Swap the arguments at both call sites and add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 12:01:42 -06:00
Roland
1951ca548d
Using stubs to simplify tests.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-05-07 20:07:46 -06:00
Roland
8f0bb78722 Converting RSS and Video Downloaders to new testable strategy. 2026-05-07 20:07:31 -06:00
Roland
f80b3ea769 Fixing bug where Channel Downloader would create empty videos.xml channel if RSS feed was sent to it. 2026-05-07 15:54:53 -06:00
Roland
ba63170e55
Consolidating deletion endpoints.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-05-03 19:14:30 -06:00
Roland
09a967535b Models like Archive/Video/Doc can be deleted, even if they are tagged. User is shown a warning and must confirm deletion of tagged files. 2026-05-03 19:14:15 -06:00
Roland
9468448a54
Modifying Video/Channel/RSS Downloaders so that they can download videos, but audio only. These audio files are grouped under their channel directories. 2026-04-14 15:33:01 -06:00
Roland
250575ef87
Delete duplicate video on download, even if it has tags.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-04-04 19:05:16 -06:00
Roland
105ae377f9
Video download destination can download to the incorrect Channel.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-04-04 15:52:47 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
755b133fa4
Fixing bug where configs could be overwritten with empty configs on first refresh. Fixing bug where configs were displayed as JSON in admin settings. 2026-03-22 12:56:37 -06:00
Roland
00cfdc14c6
Trying to get existing channel using more variations of Youtube URLs.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-20 17:26:51 -06:00
Roland
66a891c759
Refactoring videos and archives UI paths for consistent scheme. 2026-03-20 15:38:32 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
e6aaab1016
Creating video duration cache to avoid requesting video duration between restarts. 2026-03-20 13:02:46 -06:00
Roland
b1db71f04a
Handling when skip_urls is a None. Logging a warning (and ignoring) when duplicate json files are in a Video FileGroup.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-13 15:27:43 -06:00
Roland
4579562be5
Displaying download progress (speed, eta) for file and video downloads. 2026-03-12 19:31:35 -06:00
Roland
da6e62ee45 Backup configs can be restored in the UI. User can choose between Merge or Overwrite, the UI displays a comparison. Configs are compared when user opens the restore modal, configs which won't cause changes are marked. 2026-03-10 20:30:46 -06:00
Roland
1cf402821c
Renaming VTT files on video download if there are extra characters in the name.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-09 15:00:24 -06:00
Roland
8c294dad39
Downloading only the first video when the VideoDownloader gets a playlist by accident.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-08 11:55:59 -06:00
Roland
b2185ad7c3
Failing all downloads of a domain if cookies errors are received. 2026-03-08 11:08:08 -06:00
Roland
d2d09da296
Video downloads are retried in an hour, if they are live.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-06 15:22:27 -07:00
Roland
0ffbc05e91
Unifying video download settings so that global/channel/rss/video
download settings have inheritance.  Allowing more granular control of
potentially failed downloads.
2026-03-05 17:40:17 -07:00
Roland
9e52f658ea
Storing parent_download_url in a file group's json file after download has completed.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-03-03 15:54:22 -07:00
Roland
71f20d8c59
Using shared context to sync cookies between processes. 2026-02-22 14:32:27 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
4537c63d89
Adding DEBUS to api systemd services. 2026-02-19 18:01:55 -07:00
Roland
929468821b
Using gnomekeyring for yt-dlp brave cookie parsing. 2026-02-19 17:44:57 -07:00
Roland
d8c51db5a3
Fetching video comments uses video downloader browser profile. 2026-02-19 16:25:35 -07:00
Roland
ae17152e79
Expanding video browser profile selection to match archive feature. Adding toggle to disable downloading missing video comments. 2026-02-19 16:15:18 -07:00
Roland
99a716aa13
Storing parent url for every child download, useful for knowing why WROLPi is downloading something. This is displayed on the Video and Archive download edit page. Updating Archive Download edit modal to use single url input.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-18 15:16:10 -07:00
Roland
3f5470d974
Channels are linked to downloads by directory or download source id when downloading, or when importing config. 2026-02-08 14:27:59 -07:00
Roland
387ef6dfe8
Saving video ffprobe json when video file is uploaded. 2026-02-08 12:29:25 -07:00
Roland
a6d6aaf9b4
Adding .ffprobe.json for video file groups to speed up re-indexing in the future; adding save_ffprobe_json to the WROLPiConfig to disable this feature. 2026-02-07 18:40:56 -07:00
Roland
fa5c116b62
Detecting collection directories using collection formatting rather than relying on parent directory of file groups, because those are now longer always in the collection directory. 2026-02-07 17:16:08 -07:00
Roland
8a4f9201f6
Using Channel name when reorganizing channel's files, fallback to info json uploader name (which may be outdated).
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-02-05 11:48:25 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
31af5120b8
Handling edge case when configs have absolute directories. 2026-01-23 15:29:20 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
2a27d354a5
Creating AutoAwaitTestClient for tests which automatically runs background tasks and expires db objects.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-01-05 18:26:49 -07:00
Roland
150ad6ee78
Deleting existing orphaned collections when no channel exists in the config. 2025-12-26 11:28:06 -07:00
Roland
b408d11ea1
Deleting orphaned collections when no channel is in the channels config. 2025-12-26 10:00:32 -07:00