Commit graph

143 commits

Author SHA1 Message Date
Roland
73fafcd645
Search: fast ABC mode by default, opt-in deep search
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries.  Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.

Database:
- Migration adds the generated column and a GIN index (rewrites the
  table; a few minutes on large libraries).

Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
  DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
  query (COUNT FILTER on the abc subset): `file_groups` (fast) and
  `file_groups_deep`.

UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
  list.
- The gap is reconciled by a hint offering deep search: count-aware on
  the global search ("No results in fast search. N results available in
  deep search." / "N more results available..."), generic on the module
  pages (no deep estimate there).  The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
  ?deep=true.

Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside.  Also skip flaky test_download_channel on
CircleCI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 10:32:16 -06:00
Roland
2acdc2e16d
Expanding FileBrowser button widths when page is wide enough.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-07-01 19:52:15 -06:00
lrnselfreliance
cafc6514cb
Restore the censored-video search filter (#499)
* Restore the censored-video search filter

Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.

Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored).  When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 09:48:36 -06:00
Roland
c3110f3940
Warning when SMART reports unreadable sectors. Parse Seagate drive hours correctly. 2026-06-24 15:59:34 -06:00
Roland
16070e98b3
Fixing error where focusing on an empty Directory search would throw an error. 2026-06-24 15:23:32 -06:00
Roland
3d6aead072
Rebuild Inventory as a config-only system with food catalog and calculators
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.

Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
  into memory and the single source of truth. New reusable MultiFileConfig
  (wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
  (text/number/quantity/date/select/location/calories). Unit math/aggregation
  moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
  inventory save with version-based optimistic concurrency; 404 for missing),
  DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
  (pound->lb), and an alembic migration drops the tables.

Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
  Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
  customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
  inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
  calories column, and embeds a ration estimate (how long the stock lasts).

Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
  catalog_defaults.yaml (~60 generic items, total calories per package) with
  merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
  modal, and entry-row pre-fill of category/size/calories from a name match.

Calculators
- Generic Food Storage planner (like the Water calculator): household +
  editable per-person/year amounts -> total to store over a chosen duration,
  with women/children weighted by DGA calorie shares and an imperial/metric
  toggle.

Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 17:14:54 -06:00
lrnselfreliance
e018616daf
Warn in the navbar when a drive's SMART health is failing. (#471)
A failing SMART self-assessment predicts imminent drive failure, so it
deserves a prominent warning.  useDriveHealth() reads the assessment
already present in smart_stats (pySMART reports PASS / FAIL / null) and
flags any drive whose assessment is FAIL.  Nav.js renders a red warning
icon in its own always-on slot — like the power warning, and unlike the
single-slot priority chain where a transient high temp could hide it.

Frontend-only: the assessment field is already in the cached payload.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:33 -06:00
lrnselfreliance
1bd102c768
Warn in the navbar when a hard drive runs hot (#470)
* Warn in the navbar when a hard drive runs hot.

The navbar already warns about high CPU temperature; add the equivalent
for drive temperature, since an overheating disk risks data loss and a
user running an off-grid library won't be watching for it.

- controller/lib/smart.py: build_smart_stats() returns the per-drive list
  plus warning thresholds (55C warn / 65C critical) so the frontend stays
  dumb. Reading SMART wakes a spun-down drive, so drives in standby are
  detected with `smartctl -n standby` (which works through the USB-SATA
  bridges where `hdparm -C` only reports "unknown") and NOT read; their
  last-known reading is reused instead.
- controller/lib/status_worker.py: collect SMART on a slow 60s cadence
  (cached on app.state) rather than the 5s status loop, so the poll does
  not keep USB drives awake. The result flows into the same cached payload
  via a new collect_all_status(smart_stats=...) kwarg.
- Frontend: useDriveTemperature() reports the hottest drive; Nav.js shows
  a yellow/red `hdd` icon with a popup, mirroring the CPU warning.

Verified on hardware: smart_stats reports real temps and SMART runs once
a minute, not every 5s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Address Greptile review on HDD temperature warning.

- _drive_is_asleep also treats SLEEP mode as asleep, not just STANDBY.
  smartctl -n standby exits "Device is in SLEEP mode, exit(2)" for the
  deeper power mode; matching only "STANDBY" meant a SLEEP drive was read
  and woken — the exact failure the check prevents.  (TDD: added a failing
  SLEEP test, then fixed.)
- useDriveTemperature uses ?? instead of || for the temperature fallback
  to express "fall back only on null/undefined".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:33 -06:00
Roland
345b69b7d7
chore: address Greptile P2 review comments on PR 449
- In get_effective_video_settings: on unparseable sleep_requests, delete
  the key instead of forcing 0.0. This lets the global config default
  (already present from _extract_global_inheritable) take effect, as
  suggested by Greptile.

- In handleInputEvent: add explicit `value !== ''` guard before the
  isNaN check. Prevents silent NaN in form state for empty number
  fields (isNaN('') === false in JS).

Both changes are strict improvements with no behavior change for
valid inputs.
2026-05-31 13:02:44 -06:00
Roland
9cc5c15e32 fix: coerce sleep_requests and numeric settings from string form values
`sleep_requests` (and other numeric inheritable video download settings)
could arrive as strings ('' or '0.75') from the frontend NumberInputForm
and DownloadRequest bodies. These were persisted into Download.settings
(JSONB) and later triggered:

    if effective['sleep_requests'] > 0:
    TypeError: '>' not supported between instances of 'str' and 'int'

in VideoDownloader.execute_download (and similar sites).

Root cause: DownloadSettings.__post_init__ only filtered None/[] and did
no type coercion, even though the dataclass declared Optional[float]/int.

Changes (minimal targeted fix):
- Add _coerce_optional_number helper and apply it to all numeric fields
  in DownloadSettings.__post_init__ (the existing sanitization boundary).
- Centralize legacy string coercion for persisted data in the single
  read chokepoint: get_effective_video_settings.
- Simplify VideoDownloaderConfig.sleep_requests getter.
- Frontend: use parseFloat for type=number inputs + null defaults for
  sleep_requests (prevents '' from ever being submitted).
- Add focused regression test for string coercion path.
- Revert unrelated change to string field filtering behavior.

No NaN/Inf speculation, no duplicated try/float guards at every call site,
no behavior change for non-numeric settings.
2026-05-31 13:02:31 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
ae460af8d6
URLs can be dropped onto download textarea.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-05-02 13:35:08 -06:00
Roland
7bf1c80847
Fixing bug where pagination would show an extra link that would load no results.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-04-30 17:40:26 -04:00
Roland
66cecce5e6
Adding Doc filter button so users can search just PDFs/EPUB/etc. 2026-04-14 15:35:35 -06:00
Roland
4b34d37838 Navigation links automatically hide based on screen width, rather than hard breaks. 2026-04-09 16:17:38 -06:00
Roland
53f4e29b64
Updating controller hotspot and bluetooth toggle commands with real paths for RPi. 2026-03-28 10:35:07 -06:00
Roland
1c55745884
Adding bluetooth radio toggle to Controller. Writing controller.yaml to media drive on startup, if it doesn't exist. 2026-03-28 10:09:58 -06:00
Roland
282b0f63e8
Adding links next to video captions to allow scrubbing of video by the captions, adding filter to make finding quick. 2026-03-21 16:50:27 -06:00
Roland
1c79c1c495
Refactoring docs to match new url scheme like videos and archives. 2026-03-20 15:53:21 -06:00
Roland
66a891c759
Refactoring videos and archives UI paths for consistent scheme. 2026-03-20 15:38:32 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
dd270b37dc
Not clearing settings when frontend fails to fetch, this caused the navbar color and such to be forgotten. 2026-03-07 16:24:20 -07:00
Roland
21c96d9501
Displaying cmd+k/ctrl+k global search keyboard shortcut suggestion. 2026-02-23 16:07:22 -07:00
Roland
a9b2ec321e
Channel saving logic was inverted. 2026-02-05 11:12:44 -07:00
Roland
2c8f765f2b
Adding statistics page for Archives. 2026-02-02 15:15:28 -07:00
Roland
74cb5171da
FileGroups in Collections can be reorganized in the Collection's directory.
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
2026-01-31 18:08:41 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
3f411679bd
Adding keyboard shortcuts to UI.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-26 18:59:20 -07:00
Roland
bdaccdeca1
Enhancing file browser with drag-to-select. 2025-12-26 11:28:10 -07:00
Roland
1ebcae0965
Allowing user to configure collection download destination directories, as well as directories in the file structure. This will allow sorting archives by destination year, for example. 2025-12-23 16:46:15 -07:00
Roland
b9158b60fe Creating Controller service which will control WROLPi services, admin, SMART, disks, and status. nginx will fallback to Controller UI when React UI is unavailable (such as during upgrades). 2025-12-15 15:48:40 -07:00
Roland
a92104728d Converting Theme.js to typescript so we can use shorthands like Card.Icon.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-07 12:41:10 -07:00
Roland
8ea3ea8a67
Updating code to use new version of react-router. 2025-12-07 11:54:20 -07:00
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
63a77b87e6 Removing collection metadata. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
3725f0f39d Adding edit button so user can edit downloads, currently supports videos and archives. 2025-08-06 13:03:32 -06:00
Roland
ee434b3fb8
Reworking useFileUpload to upload files synchronously, disabling upload form when upload is in progress. 2025-05-14 18:25:00 -06:00
Roland
41c52b9c1e
Not using browser profile if it is disabled, even if always_user_browser_profile is enabled. This enables the user to selectively disable cookies. 2025-05-05 12:56:24 -06:00
Roland
00c86e1a1a
Searching wrolpi user's home directory for browser profiles. 2025-05-02 17:24:51 -06:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
Roland
15f6e7ed5c
Handling new issue with youtube caption file names. Adding QR Code calculator. 2025-02-07 11:44:17 -07:00
Roland
fab3fd62ed
A user can overwrite existing files when uploading. 2025-02-03 17:16:13 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
01338e4d89
Displaying plug icon when API fails to respond. Using window.apiDown to communicate to other hooks to stop fetching, the status interval will keep checking if the API is back up. 2025-01-24 15:35:26 -07:00
Roland
ee36b90e8e
Renaming HelpPopup to InfoPopup to avoid confusion with future Help features. 2025-01-24 14:28:04 -07:00
Roland
d295f815ea
Adding Calculators to the Dashboard. Adding evaluation input to the dashboard which attempts to use mathjs to evaluate the value. 2025-01-19 11:58:43 -07:00
Roland
cef4335a53
Fixing htmlFor usage. 2025-01-11 12:04:22 -07:00
Roland
d8b3db9d63
Using iostat to detect busy disk issues, reporting them in the UI. Detecting under-voltage/over-current issues using dmesg and reporting them in the UI. 2025-01-10 11:33:21 -07:00
Roland
74a1486f0e
Hiding syntax errors when API is down. 2024-12-18 17:32:00 -07:00