Commit graph

128 commits

Author SHA1 Message Date
Roland
da340a00b3
Displaying warning when no drive is mounted. 2026-05-14 16:16:18 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
bbc23dbf90
Adding map default view option to the map context menu. 2026-04-04 14:56:25 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
ae7ab1981f
Adding toggle to prevent downloads from starting when cookies are locked. 2026-03-14 12:08:23 -06:00
Roland
ccc6cf78d3
Setting system timezone using Controller.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-13 15:56:21 -06:00
Roland
c2c5d237d7
Adding timezone to WROLPi config to schedule download window. 2026-03-13 15:03:13 -06:00
Roland
35994bd363
Adding download window which can prevent downloads from running at unwanted times.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-12 20:36:57 -06:00
Roland
f59a4198d9
Renaming all co-occurring to overlapping, easier to spell. Filtering by only overlapping tags when searching by a tag, this is because any non-overlapping will not produce results anyways. 2026-03-07 15:55:04 -07:00
Roland
3d74b4bcfc
Always fetching frequent tags. Not showing the most recent tags if there are no results. Handling quoted URL for tag names. 2026-03-07 12:25:41 -07:00
Roland
1913755db5
Adding tags filter input on tags selector, adding frequently used tags to suggest tags that are used together.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-07 10:47:18 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
a6d6aaf9b4
Adding .ffprobe.json for video file groups to speed up re-indexing in the future; adding save_ffprobe_json to the WROLPiConfig to disable this feature. 2026-02-07 18:40:56 -07:00
Roland
9005c0deff
Adding toggle to disable the tags directory feature. 2026-01-31 18:08:42 -07:00
Roland
31af5120b8
Handling edge case when configs have absolute directories. 2026-01-23 15:29:20 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
6a9ffcaadc
Adding selection to the Downloads page, user can delete or retry multiple downloads at once. 2025-12-26 10:35:30 -07:00
Roland
78251b067e
Removing WROLPi admin functionality that is now in Controller.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-19 18:41:27 -07:00
Roland
42f5607281
Reworking and unifying log levels. 2025-12-16 19:20:04 -07:00
Roland
344fe9035f
Removing optional_session, passing session down from request middleware. Moving session param to the front of functions.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-11 16:03:18 -07:00
Roland
5b29144d70 Testing that nearly-empty configs are ignored on import. Adding setting to disable automatic upgrade checking. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
376850ceeb
Moving config api endpoints to their own file.. 2024-12-03 22:33:19 -07:00
Roland
fc084e2421
Updating api's openapi docs. 2024-12-03 22:33:18 -07:00
Roland
fe521c99f3
Adding edit form for Scrape downloader. 2024-12-03 12:44:38 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
a25b8373c9
Retry button resets attempts. 2024-10-16 18:57:16 -06:00
Roland
acf3dee179
Configs are displayed in new ConfigsPage under Admin.
- Configs can be imported and saved.
- Configs are validated before imported.
- API will refuse to save a config if it was not imported successfully.
2024-10-16 18:56:45 -06:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
Roland
317f3f3dfb
Top processes are displayed on Status page. Nice of renderd is lowered to avoid clobbering system at startup. 2024-10-16 18:56:27 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved. 2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
e72d461461
Refusing to start hotspot if Wifi device is in use. 2024-10-16 18:56:08 -06:00
Roland
35ec6d44fe
Creating setting for changing navbar color. 2024-10-16 18:56:04 -06:00
Roland
82af9a0d3c
Fixing bug where download.downloader would be replaced for all Channel Downloads. 2024-10-16 18:55:58 -06:00
Roland
53d2402d32
Only files that have been tagged will be returned when new 'Any' tag button is used to filter file search results. 2024-10-16 18:55:56 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
63381c1d77
Created download retry button. 2024-05-08 17:05:12 -06:00
Roland
f0e4302de8
Restoring main.py as method to start WROLPi API because it is required to upgrade docker database. 2024-05-02 10:56:43 -06:00
Roland
41f3969330
Modifying launch of API to use 'sanic' executable. This was done because startup is much faster. 2024-05-01 18:42:52 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
2c127a03f5
Testing channel suggestions being ordered by video count. 2024-04-16 15:48:09 -06:00
Roland
7669cfe361
Ordering suggested channel names by video count. 2024-04-16 10:04:06 -06:00
Roland
969a065a6c
Splitting FileGroups from search suggestions to speed up channel/domain response time. 2024-04-16 09:46:47 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories. 2024-03-09 16:51:26 -07:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
f865d9fb0f
Videos display comments. User can download comments using Refresh button. 2024-02-20 13:47:12 -07:00
Roland
6d5a56cacc
A user can ignore a directory by selecting it and ignoring it in the FileBrowser. 2024-02-19 14:44:20 -07:00