Commit graph

134 commits

Author SHA1 Message Date
Roland
06ac0d2055 review: ffprobe out of the write transaction, and three tests that were watching nothing
All three bugs confirmed; the two suggestions taken.

[bug] `video_modeler` awaited ffprobe -- a subprocess per video, up to 20
per batch -- inside its `commit=True` session.  With IMMEDIATE as the
default that holds the write lock for the whole batch, which is the same
starvation this branch is curing in `video_cleanup`.  (It was already
partly true before: the batch's first `session.flush([video])` took the
lock ahead of the remaining ffprobes.)  The batch is now read, ffprobed
with no transaction open, then written.  `get_or_create_ffprobe_json` in
modules/videos/common.py is the path-only entry point that makes that
possible; `Video.get_ffprobe_json` delegates to it and keeps appending
the cache file to its FileGroup.  `validate()` -- and the poster
generation it can trigger -- is still inside the write transaction, and
now says so.

[bug] `test_download_dispatched_outside_immediate_transaction` probed
`getattr(_immediate_txn, 'active', False)`, which a ContextVar never has,
so it passed unconditionally.  It now probes the actual write lock from a
competing connection under `production_like_sessions`.  Mutation-checked:
moving the dispatch back inside the claim transaction fails it.

[bug] `asyncio.create_task` snapshots the current Context, so a flag held
for a write session's lifetime was inherited by every task spawned inside
it -- and the parent's `reset()` cannot reach a child's copy, leaving
that task issuing BEGIN IMMEDIATE for its *read* sessions.
`create_downloads` spawns `dispatch_downloads` exactly that way, so every
user-created download hit it.  The flag now lives only across the
session's opening `connection()` call, which also means the write lock is
taken when the session opens rather than at the caller's first statement.

[suggestion] `renew_recurring_downloads` scanned every recurring Download
under the write lock, including the usual cycle where nothing is due.  It
reads first now and takes the lock only for the rows that change,
re-checking status since the dispatcher may have claimed one in between.

[nit] Parenthesized the directory OR in the claim join.

`probe_write_lock_is_held` moved to conftest alongside
`write_lock_held_briefly`; three suites use it now.

Suite: 1712 passed, 5 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 20:15:50 -06:00
Roland
c79d006a06
review: flush before the rename, not just after one delete
Move the flush to immediately before FileGroup.move so it covers every
deletion queued in this function, not only the one found by
get_by_path.  The earlier missing-file deletions were already reaching
the DB -- the get_by_path query autoflushes them -- but that is an
incidental property of the query sitting between them and the rename,
not something the rename should depend on.

Also assert the kept Video's old path is gone, so the test reads a move
rather than a copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:08:40 -06:00
Roland
3a57e28cc6 fix: the deleted duplicate still held the path we renamed onto
`delete_duplicate_videos` deletes the smaller copy sitting at the
destination path, then renames the copy it keeps onto that path.
`FileGroup.delete` unlinks the files right away but only queues the row
deletion on the session, and SQLAlchemy emits UPDATEs before DELETEs
within a flush -- so the rename's UPDATE of `primary_path` collided with
the row that was about to be deleted:

    sqlite3.IntegrityError: UNIQUE constraint failed: file_group.primary_path

The whole download transaction rolled back, so a re-downloaded video
lost its DB record while its files stayed on disk (and the previous
copy's files were already unlinked).

Flush the delete before claiming the path.  The earlier missing-file
deletions in the same function never hit this because the `get_by_path`
query between them autoflushes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:08:31 -06:00
Roland
4cafb9f00e
Remove stale Postgres references after SQLite cutover.
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
2026-07-18 13:28:54 -06:00
Roland
d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00
Roland
68fab29a4b
Channels: log directory video counts after a refresh
After a Channel refresh, log how many video/audio FileGroups are in the
Channel's directory, and how many are assigned to this Channel, other
Channels, or no Channel. Helps debug Videos missing from their Channel
after files are moved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:45:10 -06:00
Roland
64b1ca8029
Channels: fix statistics inflated by tagged videos
The tag_file join in Channel.get_statistics() produced one row per tag,
so a Video with N Tags was counted N times in video_count, size, and
length. Count tagged Videos in a subquery instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:29:24 -06:00
Roland
09a967535b Models like Archive/Video/Doc can be deleted, even if they are tagged. User is shown a warning and must confirm deletion of tagged files. 2026-05-03 19:14:15 -06:00
Roland
9468448a54
Modifying Video/Channel/RSS Downloaders so that they can download videos, but audio only. These audio files are grouped under their channel directories. 2026-04-14 15:33:01 -06:00
Roland
250575ef87
Delete duplicate video on download, even if it has tags.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-04-04 19:05:16 -06:00
Roland
282b0f63e8
Adding links next to video captions to allow scrubbing of video by the captions, adding filter to make finding quick. 2026-03-21 16:50:27 -06:00
Roland
66a891c759
Refactoring videos and archives UI paths for consistent scheme. 2026-03-20 15:38:32 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
4579562be5
Displaying download progress (speed, eta) for file and video downloads. 2026-03-12 19:31:35 -06:00
Roland
9e52f658ea
Storing parent_download_url in a file group's json file after download has completed.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-03-03 15:54:22 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
a5dee64af2
Changing video view_count to bigint for popular videos.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-08 12:33:37 -07:00
Roland
a6d6aaf9b4
Adding .ffprobe.json for video file groups to speed up re-indexing in the future; adding save_ffprobe_json to the WROLPiConfig to disable this feature. 2026-02-07 18:40:56 -07:00
Roland
fa5c116b62
Detecting collection directories using collection formatting rather than relying on parent directory of file groups, because those are now longer always in the collection directory. 2026-02-07 17:16:08 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
31af5120b8
Handling edge case when configs have absolute directories. 2026-01-23 15:29:20 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
1d9b794fca
Converting JSON columns to JSONB for performance.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-01-06 06:23:32 -07:00
Roland
3733eb4efd
Always refreshing channel in background task. 2025-12-26 13:43:51 -07:00
Roland
150ad6ee78
Deleting existing orphaned collections when no channel exists in the config. 2025-12-26 11:28:06 -07:00
Roland
5d50562768
Reworking Cards in the App so that they use the new FileGroup.files and .data. 2025-12-24 12:30:04 -07:00
Roland
298e852484 Creating FileGroup.directory, converting paths to relative to avoid updating all the paths on move. 2025-12-24 11:26:45 -07:00
Roland
344fe9035f
Removing optional_session, passing session down from request middleware. Moving session param to the front of functions.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-11 16:03:18 -07:00
Roland
1747442db3 Migrating move_channel functionality into move_collection so domain files are moved.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-09 16:44:22 -07:00
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
d7a02b5735 Speeding up channels config import. 2025-12-06 15:11:37 -07:00
Roland
1d8738ace2 Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
c635267eda
Fixing bug where Archive screenshots would not be tracked when a user uploads a screenshot.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-10-12 17:19:54 -06:00
Roland
b14c153664
Setting default value for channel trigger columns. 2025-08-22 11:02:47 -06:00
Roland
25f9568154 Adding summary columns to Channel table. Adding triggers to keep columns updated. 2025-08-17 10:50:55 -06:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
e49ffc8a6c
Creating Channel.info_json_path property to avoid redefining channel's info json path. 2024-12-24 14:41:51 -07:00
Roland
7d6b7114a7
Adding TRACE level for logging beyond debug. Reworking async_client for tests, not all tests need the App to handle signals, so any test that needs signals can use the new await_switches fixture which primes the app. 2024-12-11 11:52:33 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
0784f740bb
Reworking Downloader.process_runner to use real stdout/stderr files to avoid issue with pipes being truncated. 2024-10-16 18:57:07 -06:00
Roland
77ad52473a
Fixing upsert_file usage. Adding log details for error. 2024-10-16 18:56:43 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
317f3f3dfb
Top processes are displayed on Status page. Nice of renderd is lowered to avoid clobbering system at startup. 2024-10-16 18:56:27 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
905042b600
Files which do not have a model can still be uploaded, they are indexed instead. 2024-10-16 18:56:15 -06:00
Roland
a98aebfdda
A video that is uploaded is attached to its Channel. Saving Channels config with both download url and frequency in case downloads config is corrupted. 2024-10-16 18:56:11 -06:00
Roland
0e99266379
A directory containing files that have been tagged is maintained automatically by WROLPi. 2024-10-16 18:56:05 -06:00