Commit graph

144 commits

Author SHA1 Message Date
Roland
43dac5d63d
Controller page: group hotspot settings in a segment below the toggles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 09:16:44 -06:00
Roland
2887309aae
Hotspot: address review comments.
* POST /api/hotspot/settings checks is_docker_mode() directly instead of
  string-matching the error message (still 500, matching the sibling
  hotspot/bluetooth/throttle endpoints).
* Escape \ ; , " in the WiFi QR code per the WPA QR format.
* Skip the controller.yaml write when a settings update changes nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 09:04:17 -06:00
Roland
0d2c13e4e7 Controller: move hotspot settings to controller.yaml with WiFi device dropdown.
Hotspot was reported as "not supported" on x86 machines (Ubuntu/Debian PCs)
because the Controller only watched the hardcoded default wlan0 device, while
predictable interface naming produces names like wlp2s0. The Settings page
saved hotspot_device to wrolpi.yaml, which the Controller never read.

The Controller now owns hotspot settings in controller.yaml:

* New endpoints: GET/POST /api/hotspot/settings, GET /api/hotspot/devices
  (WiFi interfaces discovered via nmcli).
* Legacy hotspot_* values in wrolpi.yaml are migrated into controller.yaml
  once on startup, unless controller.yaml already sets them.
* The Hotspot SSID/password/device fields moved from the Settings page to the
  Control page next to the Hotspot toggle; the device is now a dropdown of
  the WiFi devices found on the machine.
* The Controller fallback UI gets the same hotspot settings form for parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 09:03:58 -06:00
Roland
706e22bd8f
Address review: clear daily limits via 0; document soft-cap semantics
- Settings.js: send 0 (not null) when a daily-limit field is cleared, so the
  settings API actually clears a previously-saved limit instead of dropping the
  null and keeping the old value.
- downloader.py: document that the daily limit is a best-effort soft cap; minor
  bounded overshoot is possible when create_downloads' background dispatch overlaps
  the perpetual cycle, and self-corrects the next cycle. A strict cross-process cap
  is intentionally not implemented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 17:18:01 -06:00
Roland
d95249218b Add per-domain and global daily download limits
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day

Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.

Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
  the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
  stops dispatching counted downloads for the cycle. daily_download_counts derives the
  day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
  (base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
  collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
  the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
  validation, 0 = unlimited), and the React Settings page.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 17:17:42 -06:00
lrnselfreliance
dac849f9f6
Surface coarse SMART health for USB-bridge drives (#498)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* Surface coarse SMART health for USB-bridge drives

USB enclosures that reject ATA pass-through (e.g. Seagate Expansion) are
invisible to pySMART/DeviceList, so the drive shows no SMART at all on the
Controller page. Fall back to smartctl over the SCSI transport, which many
such bridges still answer, to report a coarse PASS/FAIL health plus drive
identity (model/serial/capacity).

The health (-d scsi -H) and identity (-d scsi -i) reads are issued as
separate calls: combining them makes smartctl suppress the health line when
the identity read reports "SMART support is: Unavailable" (which these
bridges do). The entry is marked smart_limited (labelled "(limited)" in both
UIs) so a green PASS is not mistaken for full monitoring — these drives
expose no attributes or temperature, and some bridges report "OK" statically.

* Harden SCSI health fallback (Greptile review)

- _scsi_health: treat any unrecognised status (e.g. "UNKNOWN", "N/A") as
  unreadable (None) instead of a false FAIL. Only "OK" -> PASS and "FAIL*"
  -> FAIL; anything else omits the drive rather than showing a red FAIL that
  contradicts the smart_limited disclaimer.
- get_all_smart_status: isolate the SCSI supplement loop in its own
  try/except so an error there cannot discard the pySMART results already
  collected.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 09:05:06 -06:00
Roland
c3110f3940
Warning when SMART reports unreadable sectors. Parse Seagate drive hours correctly. 2026-06-24 15:59:34 -06:00
lrnselfreliance
9a5d1f1630
Add ordered collections (playlists): polymorphic items + item API (#475)
* Add Playlists: ordered, mixed-type collections

A playlist (Collection kind='playlist') is a manually-curated, ordered list that
mixes item types. Squash of the playlists feature branch (24 commits).

Data model & API:
- CollectionItem is polymorphic via item_kind: 'file' (FileGroup), 'zim'
  (zim_id + zim_entry), or 'url' (link to anything the WROLPi browser opens,
  e.g. /map?lat=&lon=&z=). CHECK constraint enforces exactly one kind; per-kind
  unique constraints; migration a7b1c2d3e4f5.
- Endpoints: create playlist; add/remove/reorder items (idempotent adds: 201 new,
  200 existing); GET returns the ordered items. All mutations wrol_mode_check'd.

Config (source of truth):
- playlists.yaml (PlaylistsConfig) stores each playlist's metadata AND ordered
  items (file by relative path, zim by path+entry, url inline); round-trips a
  DB wipe. Supports backup import (DB_CONFIG_FILE_NAMES).
- The redundant generic CollectionsConfig (collections.yaml) is deleted:
  channels/domains own their configs, authors/subjects are re-derived.

On-disk sync (sync_playlists_directory switch):
- Each playlist materializes into the configurable Playlists Directory
  (playlists_destination, Settings UI input, excluded from indexing) with
  position-prefixed entries: NNNN_<file> hard links (copy fallback across
  filesystems) and NNNN_<title>.html stubs for zim/url items.
- Tagged playlists are namespaced like channels: <playlists>/<tag>/<name>;
  a tag change moves the directory and prunes the empty old tag dir.
- Deletion safety: managed files are only removed when they are our marked
  stubs or still have another hard link (st_nlink > 1) -- never a last copy.
  Stray loose files at the playlists root are removed; stub URLs are
  scheme-allowlisted (no javascript:/data:).

Frontend:
- /playlists list page (CollectionTable, name filter, New Playlist modal, Tag
  column) between Files and Zim in the nav.
- View-only detail page (Name link) and edit page (Edit button): themed table
  with model icons/colors + poster thumbnails, items link to their model page
  (video/archive/doc) or preview, up/down reorder, remove, add-URL form,
  single-tag selector; all editing disabled in WROL Mode.
- Reusable green "Add to Playlist" button on Video/Archive/Doc detail pages,
  the FilePreview toolbar, Zim search results, and Map pins (added as a /map
  URL item with a map-marker icon).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Playlist edit page parity: rename, custom directory, tag modal move suggestion

Match the ChannelEditPage/DomainEditPage UX on the playlist edit page:
Back/View buttons, an edit form (name, description, directory), and the
shared CollectionTagModal with the violet "Tag" button.  The tag modal now
suggests moving the playlist to its managed tag location
(playlists/<tag>/<name>); accepting the suggestion stores directory=None so
the sync keeps auto-managing the location.  Tag buttons on the Channel and
Domain edit pages are now violet to match.

Backend: update_collection learns rename (with per-kind validation and
duplicate checks) and playlist custom directories (cleanup of the abandoned
directory, config round-trip, commit-before-switch to avoid the background
sync racing pre-commit state).  tag_collection's duplicated untag/tag
branches are merged, and the per-kind config-save dispatch is consolidated
into _activate_config_save().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deduplicate Collection error handling and Tag creation

Delete the duplicate UnknownCollection in wrolpi/collections/errors.py and
re-export the canonical one from wrolpi/errors.py, which carries
status_code=NOT_FOUND (the local copy inherited 500).  Replace the repeated
query/raise pattern in reorganize.py with Collection.find_by_id.

Add Tag.get_or_create_tag and use it for collection tagging; flush the new
Tag before returning so callers can read .id immediately -- previously
tagging a collection with a brand-new tag name set tag_id=None.  Regression
test included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:34 -06:00
Roland
9d6eb0873e
Controller can unmount foreign mounts; protect the primary drive.
The /api/disks/unmount endpoint only removed entries from fstab.yaml and
asked the reconciler to unmount, but the reconciler deliberately never
unmounts paths it has not claimed.  Foreign mounts (udisks2 desktop
automounts, manual mounts) therefore could not be unmounted from the UI,
failing with a confusing "check journalctl" error.

- Fall back to a direct umount(8) when a reconcile leaves the requested
  mount point live.
- Reject unmounting the primary WROLPi drive (/media/wrolpi) with a
  clear 400, normalized to catch path traversal.
- Require the normalized mount point to be strictly under /media/
  before any direct umount, since the fallback runs as root.
- Show "System" instead of an Unmount button for /media/wrolpi in both
  the React UI and the Controller's simplified UI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:30 -06:00
Roland
cd8792f697
Warning users when files exist in mount location. Users may have accidentally downloaded files into the media directory, we want to warn them.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-05-14 17:02:52 -06:00
Roland
da340a00b3
Displaying warning when no drive is mounted. 2026-05-14 16:16:18 -06:00
Roland
513ae52a6e
Using style on extension page description.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-05-03 13:57:33 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
a720818625
Removing Controller link in favor of the button in services. 2026-04-09 15:03:28 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
ce0cdd3a2e Samba is installed and can be configured in the Controller. 2026-03-29 10:19:28 -06:00
Roland
1c55745884
Adding bluetooth radio toggle to Controller. Writing controller.yaml to media drive on startup, if it doesn't exist. 2026-03-28 10:09:58 -06:00
Roland
755b133fa4
Fixing bug where configs could be overwritten with empty configs on first refresh. Fixing bug where configs were displayed as JSON in admin settings. 2026-03-22 12:56:37 -06:00
Roland
f146b332a8
When opening config preview modal, displaying the contents instead of using preview modal which tries to download it.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-14 13:29:03 -06:00
Roland
ae7ab1981f
Adding toggle to prevent downloads from starting when cookies are locked. 2026-03-14 12:08:23 -06:00
Roland
c2c5d237d7
Adding timezone to WROLPi config to schedule download window. 2026-03-13 15:03:13 -06:00
Roland
c2bab03f42
Checking local time for download window.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-13 07:08:35 -06:00
Roland
35994bd363
Adding download window which can prevent downloads from running at unwanted times.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-12 20:36:57 -06:00
Roland
4579562be5
Displaying download progress (speed, eta) for file and video downloads. 2026-03-12 19:31:35 -06:00
Roland
fa97b25831
Adding HelpHeader to display a modal with the relevant Help document.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-11 15:47:01 -06:00
Roland
da6e62ee45 Backup configs can be restored in the UI. User can choose between Merge or Overwrite, the UI displays a comparison. Configs are compared when user opens the restore modal, configs which won't cause changes are marked. 2026-03-10 20:30:46 -06:00
Roland
fbb71e012d Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
  (RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
  proxying to all services (API, React app, Controller, Kiwix, Help, Map),
  and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
  uses Name Constraints to restrict signing to local addresses only
  (.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
  risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
  certificate installation on iOS, Android, macOS, Windows, Linux, and
  Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
  /download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
  status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 18:48:44 -07:00
Roland
20ae3a9963
Grouping hotspot settings in a segment.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-05 19:00:36 -07:00
Roland
36d5401ebd
Reducing margin on load statistics so that they show on one line on mobile. 2026-03-05 19:00:20 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
99a716aa13
Storing parent url for every child download, useful for knowing why WROLPi is downloading something. This is displayed on the Video and Archive download edit page. Updating Archive Download edit modal to use single url input.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-18 15:16:10 -07:00
Roland
94f13183b1
Adding InfoPopup for video and archive specia directory forms. 2026-02-08 10:40:23 -07:00
Roland
f779369bb8
Updating download edit form to include the tag names alread set for the download. 2026-02-08 09:39:28 -07:00
Roland
a6d6aaf9b4
Adding .ffprobe.json for video file groups to speed up re-indexing in the future; adding save_ffprobe_json to the WROLPiConfig to disable this feature. 2026-02-07 18:40:56 -07:00
Roland
98ea05b689
Reworking FileBrowser selection using context to reduce prop drilling. 2026-02-03 16:36:44 -07:00
Roland
9005c0deff
Adding toggle to disable the tags directory feature. 2026-01-31 18:08:42 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
6a9ffcaadc
Adding selection to the Downloads page, user can delete or retry multiple downloads at once. 2025-12-26 10:35:30 -07:00
Roland
d4f1cfb859
Adding server uptime to status pages. 2025-12-26 10:05:13 -07:00
Roland
c55c8e24c9
Replacing Semantic Confirm with custom confirm to have themeing and mobile support.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-25 16:03:09 -07:00
Roland
5fb49ad001
Putting iostat on dashboards.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-24 15:51:41 -07:00
Roland
35853f2a59
Enhancing trace logging by switching to dev api service. 2025-12-19 17:55:06 -07:00
Roland
421e3d6c39
Combining status column on with service column on controller page.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-17 18:54:48 -07:00
Roland
94347c1300
Reworking control service columns to hide less useful columns on mobile. 2025-12-17 18:31:40 -07:00
Roland
c028cfe705
Hiding Control less-useful tabs on mobile. 2025-12-17 15:11:35 -07:00
Roland
628b9f3096
Adding disk mount handling to React UI.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-16 17:04:46 -07:00
Roland
1a02a4eb5b
Adding Download and Refresh buttons to the Controller log modals.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-15 16:53:24 -07:00
Roland
b9158b60fe Creating Controller service which will control WROLPi services, admin, SMART, disks, and status. nginx will fallback to Controller UI when React UI is unavailable (such as during upgrades). 2025-12-15 15:48:40 -07:00
Roland
a92104728d Converting Theme.js to typescript so we can use shorthands like Card.Icon.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-07 12:41:10 -07:00
Roland
8ea3ea8a67
Updating code to use new version of react-router. 2025-12-07 11:54:20 -07:00