Commit graph

198 commits

Author SHA1 Message Date
Roland
8f0bb78722 Converting RSS and Video Downloaders to new testable strategy. 2026-05-07 20:07:31 -06:00
Roland
8f49f33ba2 Passing download down archive downloader.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-05-07 15:54:53 -06:00
Roland
a29df682ef Migrating FileDownloader. 2026-05-07 15:54:53 -06:00
Roland
4080b7a641 Refactoring ArchiveDownloader so that it is more testable. 2026-05-07 15:54:53 -06:00
Roland
ba63170e55
Consolidating deletion endpoints.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-05-03 19:14:30 -06:00
Roland
09a967535b Models like Archive/Video/Doc can be deleted, even if they are tagged. User is shown a warning and must confirm deletion of tagged files. 2026-05-03 19:14:15 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
755b133fa4
Fixing bug where configs could be overwritten with empty configs on first refresh. Fixing bug where configs were displayed as JSON in admin settings. 2026-03-22 12:56:37 -06:00
Roland
1c4f57226f
Adding skip_viewed to archive and video fetching endpoints.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-20 20:17:27 -06:00
Roland
66a891c759
Refactoring videos and archives UI paths for consistent scheme. 2026-03-20 15:38:32 -06:00
Roland
4a21499bdd Include archive details in file search. 2026-03-20 14:16:56 -06:00
Roland
ae7ab1981f
Adding toggle to prevent downloads from starting when cookies are locked. 2026-03-14 12:08:23 -06:00
Roland
da6e62ee45 Backup configs can be restored in the UI. User can choose between Merge or Overwrite, the UI displays a comparison. Configs are compared when user opens the restore modal, configs which won't cause changes are marked. 2026-03-10 20:30:46 -06:00
Roland
9e52f658ea
Storing parent_download_url in a file group's json file after download has completed.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-03-03 15:54:22 -07:00
Roland
fa5c116b62
Detecting collection directories using collection formatting rather than relying on parent directory of file groups, because those are now longer always in the collection directory. 2026-02-07 17:16:08 -07:00
Roland
2c8f765f2b
Adding statistics page for Archives. 2026-02-02 15:15:28 -07:00
Roland
9005c0deff
Adding toggle to disable the tags directory feature. 2026-01-31 18:08:42 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
2a27d354a5
Creating AutoAwaitTestClient for tests which automatically runs background tasks and expires db objects.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-01-05 18:26:49 -07:00
Roland
2b0b161dad
Changing archive special directory default to include domain tag. 2025-12-25 15:42:35 -07:00
Roland
298e852484 Creating FileGroup.directory, converting paths to relative to avoid updating all the paths on move. 2025-12-24 11:26:45 -07:00
Roland
4ba5226c68
Adding ext to archive file format to match video file format.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-23 19:11:07 -07:00
Roland
80decbd560
Marking archives_downloader config as imported when it is imported. 2025-12-23 17:42:46 -07:00
Roland
1ebcae0965
Allowing user to configure collection download destination directories, as well as directories in the file structure. This will allow sorting archives by destination year, for example. 2025-12-23 16:46:15 -07:00
Roland
35853f2a59
Enhancing trace logging by switching to dev api service. 2025-12-19 17:55:06 -07:00
Roland
1ce8f499d8
Fixing archive bug where they did not download to the tagged domain's directory. 2025-12-18 17:39:33 -07:00
Roland
3e3694869b
Fixing bug where not all archives were modeled. 2025-12-18 17:19:57 -07:00
Roland
a8ff1a1c38
Updating archive downloader to use download.destination. 2025-12-17 16:56:58 -07:00
Roland
344fe9035f
Removing optional_session, passing session down from request middleware. Moving session param to the front of functions.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-11 16:03:18 -07:00
Roland
1747442db3 Migrating move_channel functionality into move_collection so domain files are moved.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-09 16:44:22 -07:00
Roland
c4267ec105
Batch processing in get_collections to speed up collection pages. 2025-12-07 17:42:04 -07:00
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
5b29144d70 Testing that nearly-empty configs are ignored on import. Adding setting to disable automatic upgrade checking. 2025-12-06 15:11:37 -07:00
Roland
47044cdd1f Ensuring domain config saves when tagging/untagging a domain. 2025-12-06 15:11:37 -07:00
Roland
d7a02b5735 Speeding up channels config import. 2025-12-06 15:11:37 -07:00
Roland
79e0700609 Fixing local imports. 2025-12-06 15:11:37 -07:00
Roland
1d8738ace2 Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
1ed0d63bac Creating Generate Screenshot button for Archives which are missing a screenshot. 2025-12-06 15:11:37 -07:00
Roland
41e1ed6be6
Using template0 when creating map db to avoid utf8 bug.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Fixed test_new_archive to handle platform-specific HTML formatting from BeautifulSoup.
Bumped version to 0.16.1-beta.
2025-11-24 07:50:50 -07:00
Roland
c635267eda
Fixing bug where Archive screenshots would not be tracked when a user uploads a screenshot.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-10-12 17:19:54 -06:00
Roland
8b4121fc2a
Loosening test requirements to avoid formatting difference between systems. 2025-08-22 11:03:10 -06:00
Roland
338d961284 Adding file_group.effective_datetime to speed up video and archive browsing.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
2025-08-16 11:38:22 -06:00
Roland
cff827383e
Marking download as complete when an archive is uploaded by singlefile browser extension. 2025-08-16 10:23:19 -06:00
Roland
76a454ea34
Linking to uploaded file on download after a user uploads a file with the same URL.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-08-11 16:54:10 -06:00
Roland
d7691fba71
Handling when Singlefile extension uploads may be not be the usual text/html.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-08-07 12:41:36 -06:00
Roland
76b9ead174
Handling rare error when parsing articles.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
2025-08-06 15:01:40 -06:00
Roland
e8b8eb84a9
Logging queue size when converting singlefiles to archives. 2025-08-06 15:01:15 -06:00
Roland
57081e327e Converting singlefile_upload_switch_handler to use multiprocessing.Queue to enable uploading of multiple singlefile archives at once. 2025-08-06 14:05:21 -06:00