Commit graph

175 commits

Author SHA1 Message Date
lrnselfreliance
d75be8e1a4
Download forms: allow a frequency of "Once" (#804)
* app: a download frequency of "Once" no longer blocks submission

useForm treated any falsy value as a missing required field.  "Once" is
frequency 0, so the Channel/Playlist, RSS, and Zim download forms never
became ready and the Download button stayed disabled.  Only null,
undefined, '', false, and NaN count as missing now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* downloads: editing to "Once" stores a once-download, not frequency 0

Once-downloads have a NULL frequency.  A 0 frequency counted as recurring
(frequency != None) but could never be scheduled (not frequency), so an
edited download stayed on the recurring list and could break renewal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:24 -06:00
lrnselfreliance
3c362799b5
Resume EPUBs and comic books where the reader left off (#798)
* Resume EPUBs and comic books where the reader left off

The EPUB viewer (epub.html) accepts `?cfi=` and posts its location and
progress to the page that embeds it on every page turn.  Progress comes from
the chapter and the page within it, rather than book.locations.generate(),
which stalls a Raspberry Pi on a large book.  A search result's `loc` still
wins over the saved position.

The comic viewer opens at the saved page and saves each page shown.

Both resume on the doc page and in the file preview.  The resume point is
decided once per opened file, so saves made while reading do not change the
EPUB viewer's URL and reload it.  The doc page tracks only the FileGroup's
primary file; another format beside it is not tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* EPUB/comic progress: never save a place the server would discard

The server treats a position at the start (the cover, under 1%) as not
started, and replaces the saved place with progress 0.  Both viewers
reported the place they opened on, so opening a finished book un-finished
it, and peeking at the cover erased the reader's page.

- The comic viewer does not save the page it opens on, nor the cover; a
  peek at the cover keeps the last page saved.
- The EPUB viewer posts nothing at the start of the book.
- Progress before the end is kept between 1% and 94% (resumableProgress), so
  early pages of a long book and the last pages before the end keep their
  position.  Only the last page (comics) or the end (EPUB) is finished.
- Non-finite EPUB progress is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:19:51 -06:00
lrnselfreliance
1ad451cddf
Resume videos and audio where the user left off; show progress on cards (#791)
* Resume videos and audio where the user left off; show progress on cards

Videos and audio resume where the user left off, on the video page and in
the file preview, and keep playing.  `?t=` starts at an explicit time
instead.

useFileProgress saves how far the user is through a file: every 15 seconds
while it changes, and a final save (a beacon, which survives the page
closing) on pause, end, hiding the page, leaving it, or switching files.
Each save is also kept in sessionStorage; on open the newer of that and the
server's position wins, so coming straight back resumes exactly even if the
final save is still in flight.

Cards show a progress bar along the bottom of their poster.  The dashboard's
Recently Viewed Files gains All / Continue; Continue uses the new
`in_progress` file search filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Progress review fixes: in_progress totals, latest dashboard request, preview viewed

- The in_progress search filter skips the uncounted fast path and is part of
  the totals cache key, so Continue never reports All's (or the library's)
  count.
- The dashboard's Recently Viewed hook applies only its latest response, and
  clears the list when switching All/Continue until the new files arrive.
- POST /api/files/file returns `viewed`, so a preview can tell a file
  finished elsewhere from an older position this tab holds.
- The progress hook tests restore Blob, fetch and sendBeacon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:29:22 -06:00
lrnselfreliance
a1292005ce
Other search tab: search Channels by name, add Domains and Playlists, fix the estimate (#785)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Count Channels with any of the Tags in the Other search estimate

The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Search Channels by name in the Other search tab

The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing.  Channels now match the search text by name,
and the Tags when there are both.  The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Domains to the Other search tab

Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Playlists to the Other search tab

Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint.  The search and the estimate share search_collections_query.  The
search now loads each Collection's Downloads up front instead of once per
Collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 18:22:12 -06:00
lrnselfreliance
93d878952c
Add a Repos module that mirrors git repositories (#783)
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.

- A Repo is a Collection (kind='repo') plus a `repository` row.  The new `git` downloader clones it, then updates it
  (weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
  An existing clone can be imported where it is.  repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download.  READMEs are searchable
  under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
  repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
  updated.
- Tagging a Repo moves its clone as one directory.  Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
  destination.  A failed tag move restores the Collection's tag.  A Collection's directory cannot leave the media
  directory.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:58 -06:00
lrnselfreliance
20541d7184
app: number fields no longer throw "e.preventDefault is not a function" (#735)
`useForm`'s input handlers called `e.preventDefault()` unconditionally, but the
Mantine-backed `NumberField` (Sleep between requests, Depth, Maximum Pages)
hands them a bare `{target}` object because NumberInput emits a value, not an
event.  Every keystroke in those fields threw.

`handleInputEvent` now only calls `preventDefault` when the argument actually
has one, and `getInputProps`' wrapper no longer repeats the call.  Audited every
other `preventDefault` call site: all are bound to real DOM/React events or a
guarded no-argument call.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:41:52 -06:00
lrnselfreliance
eead78007d
app: status and settings contexts say whether their first poll landed (#734)
Both contexts start as {} and nothing distinguished that from a
server that reported nothing, so for up to one poll interval after a
page load consumers rendered the negative case as fact: the Settings
form appeared with blank values and an "undefined/config/wrolpi.yaml"
path (the ready check `settings ? true : undefined` was always true),
the Downloads toggle was clickable and read "Downloading Disabled",
the upgrade panel said "up to date, vundefined", and the five hardware
toggles on the Controller page claimed "not supported on this server".

Invariants now enforced and tested:
* StatusContext.loaded is true once a poll returned a real status;
  the API being down does not count.  SettingsContext.loaded is true
  once settings were fetched, and `failed` when a fetch returned
  nothing before that; a failed refresh after a load keeps the last
  known values and `loaded`.
* The Settings page shows its loader until loaded and its error only
  when failed.  The upgrade panel shows a loader until status loaded.
  The Downloads toggle is disabled with a neutral label until then.
* A subsystem toggle is disabled without claiming "unsupported" while
  status is unknown (undefined); "unsupported" (null) is reserved for
  a loaded status that lacks the field.  A call in flight is unknown,
  not unsupported.

The context fixtures declare loaded so existing specs render as
before.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:25:46 -06:00
lrnselfreliance
306386e801
app: Flasher saved configs and Statistics distinguish pending from empty (#733)
The Flasher's saved configurations started as [] with no loading
flag, and the Saved Firmwares tab is the default tab, so every load
painted "No saved configurations yet." until the fetch returned.  The
api helper returned [] on a non-OK response too, so a failed fetch
read as an empty list.

useStatistics used useState() with no initial value, so the
Statistics page's first render saw undefined and flashed "Failed to
fetch statistics" for one frame before the effect set {} and the
spinner appeared.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / object = loaded.  The
  api helper returns undefined (and toasts) on a non-OK response.
* The Saved Firmwares tab shows a placeholder while pending, an error
  on failure, and "No saved configurations yet." only once the server
  confirmed there are none.  Only the newest configs request may write
  state, since save and delete refetch.
* The Statistics page shows its loader while pending and its error
  only for a failed fetch.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:05:02 -06:00
lrnselfreliance
9caa140fb6
app: search Other tab distinguishes pending channels from none (#728)
* app: search Other tab distinguishes pending channels from none

useSearchChannels initialised channels to [] and loading to false, and
only set loading inside its mount effect, so the Other tab's first
render painted "No Channels" with a count of 0 before the search had
run.  A failed search was rendered the same way, since the hook had no
catch and the API helper returns undefined on a non-OK response.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = no channels; loading
  is true from the first render because the search starts on mount.
* Changing tags resets to pending, and only the newest request may
  write state, so a slow response for previous tags is dropped.
* The tab shows a loader while pending, an error on failure, and
  "No Channels" only once the server confirmed there are none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useSearchChannels follows its caller's tags

The hook read its tag argument only as the initial state, so the
Other tab never refetched when the filter modal changed the URL's tags
while the tab stayed mounted; the list, including a stale "No
Channels", was whatever the mount fetched.

The internal tag state now follows the argument's contents, keyed on
the serialized list because the caller builds a new array every
render.  Tests change the hook argument after a loaded result and
expect a second search and a reset to pending, check that a new array
with the same tags does not refetch, and keep the slow-response drop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 12:01:37 -06:00
lrnselfreliance
83362a65c1
app: video comments and description distinguish pending from empty (#727)
* app: video comments and description distinguish pending from empty

useVideoExtras initialised both values to null and the page treated
null the same as "none", so every video open (and every prev/next)
showed "No comments have been downloaded. Refresh the video?" with a
live Refresh button until the comments arrived.  The Description tab
showed "No description available." the same way.  A failed fetch was
also rendered as "none", and the previous video's data lingered until
the next video's responses arrived.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / '' = loaded and empty.
  The API serialises a missing info_json field as null, so the hook
  normalises that to empty; consumers never see null for "none".
* Switching video resets both to pending, and a slow response for the
  previous video is dropped.
* The page renders a placeholder while pending, an error message on
  failure, and offers Refresh only once the server confirmed there are
  no comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useVideoExtras drops any request that is not the newest

Matching on the video id let a stale request land when the user left
and returned to the same video, or saved an edit while the page's
first description request was still in flight.  Each fetch now takes a
generation number and only the newest generation may write state,
per value, so a description refetch never invalidates a comments
request.

The edit dialog adopts a description that arrives while it is open
only if the user has not typed, so a draft written over the
placeholder is not wiped by the empty or failed value that follows.

Tests: 7 -> none -> 8 -> 7 with the first request rejecting last; a
save refetch beating the initial request; the draft surviving; the
pending placeholder asserted; the console spy restored in finally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 11:22:37 -06:00
lrnselfreliance
25e00f75fd
app: useLatestRequest reports loading before the debounce (#725)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
`loading` only became true once the debounce timer fired, so for the
first 300-500ms after every sendRequest the hook reported
data=null, loading=false.  Consumers branch on `loading` to choose
between a spinner and an empty state, so the global search dropdown,
the Zim search tab, and the map search rendered "No results" (or
"No Zims have been indexed") for a search that had not run yet.

Invariants now enforced and tested:
* loading is true from sendRequest until the latest request settles.
* The latest-request id is claimed synchronously in sendRequest, so a
  fetch already in flight is stale the moment a newer request is sent
  and cannot write data or clear loading during the new debounce.
* A stale request that rejects late no longer nulls newer data.
* A pending debounce is cancelled on unmount, and any in-flight fetch
  is treated as stale.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-25 14:05:08 -06:00
lrnselfreliance
5358c75a2b
Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:25:23 -06:00
lrnselfreliance
e0fdb762fc
zim: recompute the outdated-Zims flag after startup and on Delete (#706)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* zim: recompute the outdated-Zims flag after startup and on Delete

The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan.  The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.

- The startup recompute now runs after the DB is up and configs are
  imported, so it overrides the value `init_flags` restored from the DB.
  Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
  was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
  of showing a placeholder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete

- Startup calls flags.init_flags() then flag_outdated_zim_files() right
  after db_up, before the DB config import, so the disk scan always
  overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
  file was deleted.
- Trim narrative comments to the constraint they enforce.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 15:19:36 -06:00
lrnselfreliance
8008cd1fd8
search: don't COUNT(*) OVER() the page query (#702)
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.

Count separately and cache for 60s by filters so indexed orders can
stop after one page.  Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
2026-09-12 14:44:30 -06:00
lrnselfreliance
97a6566e3d
Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678)
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page

Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:

* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
  the blocking `multiprocessing.Event.wait(timeout=1)`.  With no switch pending
  (the normal state) this froze the loop for a second at a time; about a third
  of all requests, including trivial ones, took ~1,000 ms.  It now polls
  `is_set()` and returns immediately.

* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
  ran synchronous SQLite queries on the event loop, so one slow search stalled
  every other request on that worker.  They now run via `asyncio.to_thread`.

* The unfiltered Videos page (`POST /api/videos/search` with only an order)
  used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
  every video's index entry before LIMIT applies: seconds cold on a large
  library on a spinning disk, and ~10 s hot for a deep page.  It now seeks each
  exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
  rows and merges them, with the total from a separate count cached for 60 s.
  Results are identical to the generic query.

Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results).  Host profiles live in a
gitignored JSON file; an example is included.

Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Serve the video description from its own endpoint instead of Video.__json__

`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk.  The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays.  The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.

Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions.  The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored.  The player page fetches it through
`useVideoExtras` alongside the comments.

Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: thread-local zim sessions, fast-path equivalence test, comments

* Zim search, single-zim search, and search estimates run libzim off the event
  loop, each opening its own session inside the worker thread rather than
  passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
  query and asserts identical rows and paging for both date orders.  The one
  intentional difference is documented: past the end, the fast path still
  reports the true total.
* switch_worker returns before its try block so the empty path does not clear
  the event; its comment states the invariant only.  The copied "synchronous
  SQLite" comments are reduced to one line in one place.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:53 -06:00
lrnselfreliance
b46faa005a
ui: a download form that emptied itself after every submit (#665)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* ui: a download form that emptied itself after every submit

A successful submit reset the whole form, so adding a second download meant
re-choosing the tags, the destination and the video settings that the first
one had already been given.

It now empties only the URLs.  Everything else stays for the next download,
and a Clear button beside Download puts the form back to its defaults when
the next one is unrelated.

`useForm` gains `clearPathsOnSuccess`; `clearOnSuccess` is untouched for the
forms that want the whole reset.

* review: the one download form that kept its URLs

Scrape took a `clearOnSuccess` prop that no caller passed, so mapping it to
the URL paths left create-mode scrape clearing nothing.  It follows the same
rule as Videos and Archive now: the edit form supplies `onSuccess`, and only
that keeps the form as it was.

`clearPaths` also wrote every path it was given, so a `urls` form gained a
`url: ''` it never declared -- which the comment beside it and the test named
for it both said it would not do.  It skips a path neither the defaults nor
the form has, and the test asserts the key is absent.
2026-08-13 17:14:30 -06:00
lrnselfreliance
4c71e84696
ui: two upload dialogs that had drifted apart (#664)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* ui: two upload dialogs that had drifted apart

The dashboard and the file browser each had their own upload component: two
dropzones, two overwrite toggles, two copies of the progress-bar rendering,
against the same `useUploadFile` state.  The file browser's dialog was made
fullscreen with a drop target that grows; the dashboard's stayed a `large`
dialog with a strip to drop onto, because it was a different file.

They differ in one thing that is real: the file browser knows the destination
because the user selected it, and the dashboard has to ask.  That is now the
`destination` prop -- `undefined` asks, a string uploads there, and '' is the
media directory rather than a missing destination.

Everything else is one component, so the next change to either entry point
lands on both.

* review: an upload dialog that looped on the media directory

Pressing Upload in the file browser with nothing selected passes `''` -- the
media directory.  Three checks along that path read it as "no destination":

* The clearing effect ran, and `doClear` is a new function every render whose
  own setState scheduled the next one.  React ended it with "Maximum update
  depth exceeded", 16 warnings deep, on the default Upload path.  The effect
  now clears only while asking, and `doClear` is stable.
* The title read `Upload to: /media/wrolpi/`.
* `doUpload` bailed, so a drop on the media root uploaded nothing.

All three predate this branch; moving the dialog is what put them behind one
component.  Tests cover the title, what the hook is told, and that a fixed
destination is never the thing that empties the queue.
2026-08-13 15:15:04 -06:00
Roland
e00f0439da
ui: the last of Semantic UI, in name and in dead CSS
The library has been uninstalled for some time, but the codebase still read as
though it were there: identifiers named after it, rules written against its
class names, and about 180 comments explaining a rule in terms of a thing that
is no longer installed.

Renamed, keeping the vocabulary the app actually uses: semanticUIColorMap ->
navColorHexMap, semanticColorNames -> paletteColorNames, semanticNames ->
iconNameAliases, semanticSizes -> sizeAliases, and SemanticUIColors ->
NavColors in wrolpi/schema.py, which is what validates nav_color on the way in.
ColorToSemanticHexColor had no callers at all and is gone.

The baseline ratchet has done its job -- the list reached zero, so the JSON is
deleted and the guard keeps only the checks with teeth: no package, no import,
no `className="ui "`, no toast library.  Its Theme.tsx check went with the
wrappers it was watching for.  It also asserts its own walk found files, so a
clean result cannot mean it scanned nothing.

Deleted from App.css: every rule requiring `.ui`, `.inverted`, `.segment` or
`.modal`, none of which any component has set since the stylesheet stopped
loading, plus the form skeletons, ui-alerts and progress-text rules that
nothing renders.  Two of them were doing harm rather than nothing: the drag
tints had a dark-theme variant under `.inverted` that stopped applying, leaving
15% of a fixed blue over a dark row, and now mix a token instead.

Two selectors naming markup that no longer exists were live bugs.  The file
browser's filter had `.ui.input.file-browser-filter`, so it rendered at the
browser default width on a bar laid out to give it 34em -- now qualified by its
container, because `.wrolpi-action-input` sets `width: 100%` at the same
specificity and is loaded later.  And SearchBox takes named props rather than a
rest spread, so the `id` DestinationForm hands it was dropped in silence and
the label it renders pointed at nothing; the download forms' Destination field
has had no associated label since the migration.

"Semantic roles" survives in four places, where it means what it says.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:05:54 -06:00
Roland
f30ffe13e0
test: the label test read half the alignment contract
It asserted the label is block but not the 4px gap, so dropping the
margin -- the half that actually lines an InputForm up with the
DestinationForm beside it -- would still have passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:05:53 -06:00
Roland
4d173cbd38
ui: an input goes under its label, not beside it
InputForm returned a fragment, so its label and its field arrived as two
children of whatever contained them.  In a grid that is two cells: on
/videos/settings the "Video File Format" and "Extra yt-dlp Arguments"
inputs sat in the column to the right of their own labels.

The label is now block with the same 4px gap the migrated fields use
(FieldLabel in Download.js), so an InputForm and a DestinationForm side
by side start their inputs at the same height -- the Channel Name and
Directory fields on the Channel edit page were off by the difference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:05:53 -06:00
Roland
3f1e0b386f
ui: the corner triggers, the More button, and the search modal
Four faults, each measured in the running app before and after.

The search glyph carried Semantic's `item` class, which has had no rules at all
since Semantic was removed: no pointer cursor, no hover, and an 18.6px hit area
against a 54.4px bar.  It is the same kind of thing as Help and Admin and now
takes the same class.  The share glyph did compute `cursor: pointer` -- it is an
`<a href>` -- but its target was the 24px box the icon occupies, so the pointer
appeared only if you found it exactly.  Corner anchors now stretch and pad like
a link.

The More button rendered with NO class at all: a bare grey UA button while the
rest of the bar took the user's colour.  Menu.Target clones its child and hands
it a className of its own, and the trigger spread those cloned props after its
own `className` attribute, so the clone replaced ours.  It is merged now.  This
was mine, from extracting NavDropdownTrigger last commit -- and it also made the
previous commit's "measured and drawn are the same component" claim only half
true, because the hidden placeholder is not cloned and so kept its class.

The bar still wrapped, for a reason the padding fix could not reach: the corner
does not keep a constant width.  The Cmd-K hint renders only once
useKeyboardDetected has seen a keypress, so it appears at an arbitrary later
moment and widens the corner by ~30px; the load, memory, temperature and drive
warnings come and go with the machine's state.  None of that resizes the
container, which was the only thing observed -- so nothing recalculated.  That
is the other half of "at certain widths": the same viewport wrapped or not
depending on whether the user had touched a key.  The corner and the home link
are observed too now.

The search modal showed an input and the first line of the first group.  The
suggestion list is absolutely positioned, which is right everywhere else -- a
list that reflows the page while you type is unusable -- but in a modal it fails
twice: an absolute list adds no height, so the panel sized itself to the search
box alone at 114px, and the panel's own `overflow: auto` then clipped the list
at that edge.  In the modal the list sits in flow; the inverse is asserted so
the rule cannot leak to the pages where overlaying is correct.

Two of the new tests were wrong first and reverting the fix left them green:
they asserted hand-written markup carrying the classes I expected the components
to have, rather than the real SearchIconButton and ShareButton.  They use the
real components now.  Every fix was then planted against: 5 red for the More
class, 2 for the search class, 2 for the corner hit area, 1 for the observer,
2 for the modal.

1115 jest / 62 suites, 196 in ui-layout.cy.js, clean npm run build.
Confirmed live: share and search both pointer at 54.4px, More white on the bar
colour with a transparent face, and the modal's list in flow and unclipped.
2026-08-08 18:05:49 -06:00
Roland
c056611912
ui: the navbar corner was nudged rather than laid out, and it showed
Three faults in the bar, all reproduced by measurement before being fixed.

The indicators sat low.  The slot carried `marginTop: 0.8em`, a nudge left over
from Semantic's menu; the corner centres its children, and a top margin is part
of the margin box being centred, so it applied as a ~6px downward shift.  It
read as an error on the hotspot and the theme picker but not on share or search,
which is why it survived: an IconButton is a square ActionIcon that centres its
glyph exactly, while a bare-icon anchor sits high inside a line box that
reserves descender space for text that is not there.  The nudge cancelled that
out on one shape and was pure error on the other.  Both are laid out now --
flex, centred, no nudge -- and the anchors get `display: flex` so they hug their
glyph instead of a text line.

Help and Admin highlighted only a box around their text while Videos and
Statistics highlighted the full height.  Same class, same rules, different
parent: the bar stretches its children, but the corner is a nested flex
container that centres its own, so a link inside it was sized to its content.
The hover background is painted on the link box, so that box IS the affordance.

The bar broke onto a second line at certain widths.  Two measurement errors that
only overlap in a narrow band, which is why it appeared while resizing rather
than always.  useOverflowNav measured the WRAPPER's offsetWidth and subtracted
only the home link and the corner, missing the bar's own `padding: 0 0.25em` --
about 8px it believed it had.  And the hidden "More" placeholder it measured was
a plain button with no dropdown caret, about 21px narrower than the button that
eventually rendered.  The placeholder is now the same component as the real
trigger, so the two cannot drift again.

Tested in ui-layout.cy.js against the real <DesktopNav/>, which takes its
colours, home link and indicators as props and so needs none of NavBar's polling
hooks.  Indicator centres are measured against the bar's centre rather than each
other, because three things can be equally and consistently wrong and were.  The
wrap case sweeps 1180px down to 640px in 6px steps -- a spot check steps over a
band that narrow.  All three failed first: share 3.4px high, Help 24.8px against
a 54.4px bar, and two rows at 1156px.

Confirmed on the running app: every indicator at 0.0px from the bar's centre,
Help and Admin at the full 54.4px, and no wrap anywhere from 1500px to 780px.

1115 jest / 62 suites, 182 in ui-layout.cy.js, clean npm run build.
2026-08-08 18:05:48 -06:00
Roland
8ba73e64cd
build: uninstall Semantic UI
`semantic-ui-react`, `react-semantic-toasts-2` and `semantic-ui-offline` are
gone from package.json, the lockfile and node_modules -- 19 MB of dependencies
that nothing had imported since the migration finished.

Four things still referenced them and would have broken on uninstall.  None was
caught by a build or by jest:

  Four test files still ran `jest.mock('react-semantic-toasts-2', ...)` for
  components that no longer use it.  A jest.mock of a module that does not exist
  throws, so the packages could not have come out without this.

  `src/components/Download.cy.js` and `cypress/support/component.js` both
  imported semantic.min.css.  The support file now loads what the app loads --
  index.css, the font and token stylesheets, and Mantine's.

The ratchet changes job rather than being deleted.  An import of a missing module
fails loudly now, but three quieter mistakes would not: a `className="ui button"`
that styles nothing, a file reaching back through Theme.tsx, and someone
reinstalling a package to "just use one Semantic component".  It asserts all
three, plus that package.json names nothing matching /semantic/i.

Two pre-existing problems in the cypress component harness surfaced while
checking the uninstall had not broken it, both fixed because that file had to be
edited anyway:

  It imported `react-router-dom`, which is not installed -- this app is on
  react-router v7.  Dates to 5621d473, so the component suite has been
  un-runnable since well before this migration.

  Its mount helper supplied no MantineProvider, so every library component threw
  -- the same gap test-utils.js had for jest.

The component specs now load and four pass; eleven assertions are still written
against markup from before they stopped running.  Repairing those is its own
piece of work, deliberately not bundled in here.  The e2e suite is unaffected:
43/43.
2026-08-08 18:04:03 -06:00
Roland
51f9c8a987
ui: the last Semantic UI file, and the stylesheet with it
ControllerPage was the final call site; Theme.tsx then had nothing left to wrap,
so it drops from 861 lines to 338 -- the provider that resolves the theme, stamps
`data-theme` and `data-media-filter`, persists both and hands Mantine our tokens.
Everything below that was Semantic wrappers.

The dark-mode compatibility props went with them.  `i`, `s`, `t` and `inverted`
existed so unmigrated components could borrow Semantic's dark treatment, and
they carried hardcoded greys -- #1B1C1D, #dddddd, #eeeeee -- which is exactly
what the token themes replaced.  Three call sites in Common.js were still
injecting the #eeeeee one into links.

Then `semantic-ui-offline/semantic.min.css` came out of App.js.  That took more
with it than expected, and looking at the running app is the only reason it was
caught:

  Every link lost its colour and gained the browser's underline and visited
  purple.  The dashboard's load averages and section headings were underlined
  and purple.  Links now take `--blue`, which remaps per theme, and underline
  only on hover -- many of them wrap cards, posters and icons.

  `box-sizing: border-box` disappeared, which silently changes the size of every
  padded element in the app.  Semantic's reset had been providing it.

  The navigation bar overflowed a 1000px viewport with `nowrap` and cut "Admin"
  off the right edge -- the one link a user needs to reach the settings that fix
  a broken WROLPi.  Semantic's menu wrapped; ours does now too.

The gzipped CSS bundle is 47.5 kB, and one call site was still leaning on
Semantic's grid classes -- now a flex column, since all three of its callers
pass library components.

The ratchet stays rather than being deleted.  At zero it asserts four things: no
file imports Semantic UI, none calls its toast library, none leans on its
stylesheet, and no file reaches back through Theme.tsx.  It can go when the
packages are uninstalled.

Theme.test.js is deleted, but not before porting the two behaviours it
protected: that Button forwards a ref to the real DOM element (a class-component
ref once crashed the Control page, because portals call `node.contains`), and
that the progress percentage stays readable on every bar colour.
2026-08-08 18:04:03 -06:00
Roland
e8dd804010
ui: thirty-five files off Semantic, leaving fifteen
Phase 2, part 4.  Baseline 50 -> 15.

Migrated: all eight calculators, the rest of inventory and collections, the
Map/Zim/Docs cluster, Videos, Files, Search, Archive, DashboardPage,
DatesSelector, HelpModal, Events, Tags, Flasher's neighbours, admin Settings
and Status, Download, and App.

Also the three shared modules everything imports: api.js and customHooks.js
swap the Semantic toast library for the library's shim (same call shape), and
useForm.js renders the library's Message.  useForm accepts both the old
`{content, header, positive|negative}` shape and the new `{kind, title}`,
because the pages building those objects migrate separately.

Two library gaps closed, both found by call sites needing them:

  ActionInput is a forwardRef now.  Its ref reaches the input rather than the
  wrapper, because callers use it to focus or select the field -- a ref to the
  surrounding div would do neither, which is why one call site had resorted to
  hand-copying the component's markup.

  Header takes a `color` token name.  Three separate call sites had reached for
  an inline style to colour a heading that carries meaning ("3 items to add"
  against "2 to remove"), which is how a hex ends up in markup.

Two files needed repair after being interrupted mid-edit: MapViewer had a
Segment left in its error branch, and VideoPlayer had lost the useContext
import its own code still used.  Neither jest nor tsc noticed; only
`npm run build` did, which is the whole reason that check exists.

Fifteen files remain: Archive, BulkTagModal, Channels, Docs, FileBrowser,
FilePreview, Flasher, Nav, Playlists, Theme.tsx, Upload, VideoPlayer, Zim,
ControllerPage, admin/Downloads.  Theme.tsx is deliberately last -- the
unmigrated files still import its Semantic-backed wrappers, so it can only go
once they have.
2026-08-08 18:04:01 -06:00
Roland
2629891ebc
ui: Common.js and thirteen more files off Semantic
Phase 2, part 3.  Baseline 63 -> 50.

Common.js is the one that mattered: 51 files import it, so its Semantic usage
was propping up the rest of the migration.  Everything in it now comes from the
library -- Paginator, the two search inputs, DirectorySearch, TabLinks, the
toggles, the messages, APIButton, Breadcrumbs, SimpleAccordion, IframeViewer,
CardPoster, FileIcon, LoadStatistic, the theme menu, and the hotspot icons.

Also migrated: Placeholder, Apps, ArchivePreview, CbzViewer, AddToPlaylist,
TaggedDeleteConfirmModal, the collection edit/tag forms, and four inventory
components.

New in the library, all in the gallery and tested:

  SearchBox     Semantic's `<Search category>` with its grouped results shape
                kept, because four call sites already build it.  A real
                combobox rather than a div that listens for clicks.
  Pagination    Semantic's prop names; onPageChange gets the page number.
  TabBar        the strip only -- routing stays with the caller.
  IconStack     Semantic's IconGroup, named once for assistive technology.
  Table.HeaderCell sorted/onSort -- a real button carrying aria-sort.
  Button/IconButton translate Semantic's size names.

That last one is defensive: unmigrated callers still pass size='tiny', and
Mantine drops a size it does not recognise without a word, so the button
quietly renders at the wrong size.  Mapping here keeps the mixed state honest.

Three real bugs found and fixed on the way, none of them cosmetic:

  * `fileSuffixIconName` returned fourteen icon names the map did not have, so
    opening a folder with a PDF, a Word document or an archive in it logged an
    error and drew a fallback glyph.  All fourteen mapped, with a test that
    walks the mapper's own return values, and another asserting every mapped
    name resolves to a Tabler component that exists.
  * `Statistic` relied on Semantic's `Statistic.Group` for its spacing, so on
    the dashboard the load averages ran together into "1 MIN. LOAD5 MIN.
    LOAD15 MIN. LOAD".  It carries its own spacing now.
  * Semantic's inverted stylesheet colors every descendant of an inverted
    segment, so Toggle labels inherited #dddddd -- white pixels in night mode,
    on the Settings page, which is exactly where someone adjusts their WROLPi
    in the dark.  Restated from tokens.

The baseline ratchet gained a third assertion: a migrated file may still lean
on Semantic's CSS classes without importing the library, which the import check
cannot see and which would collapse silently when Phase 4 deletes the
stylesheet.  There is one such bridge today and it is now declared.
2026-08-08 18:04:00 -06:00
Roland
540081203f
controller: restore useConfigs and useDownloads
The toggle-hook de-duplication replaced a range of customHooks.js that also
contained these two unrelated hooks, breaking the production build. Restored
byte-identical from master; exports now match master exactly.

Jest missed this because it only compiles what tests import — `npm run build`
is what catches a missing export.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-30 13:42:34 -06:00
Roland
07af0c9f28 controller: de-duplicate the hardware toggles
The Hotspot, Bluetooth, Desktop, VNC and Throttle toggles had each been written
out longhand at four layers. Collapse the shared parts:

- useSubsystemToggle() replaces five near-identical hooks. This also fixes a
  latent bug: only useDesktop/useVnc restored state when a request failed, so a
  failed Bluetooth/Throttle/Hotspot toggle left  at null and — since the
  status poll sees no change — the toggle stayed disabled and read as
  unsupported until it remounted.
- SubsystemToggle renders all five, with the Desktop's stop confirmation and
  VNC's desktop gating as declared options rather than bespoke copies.
- The fallback UI's four update*/toggle* pairs become one config-driven
  implementation (-100 lines).
- _raise_for_action() centralises the 13 endpoint failure checks, including
  VNC's 409-vs-500 split.

Deliberately not shared: the six get_*_status_dict() builders (payloads
genuinely differ) and the per-subsystem mechanisms (nmcli/rfkill/sysfs/systemctl).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:48:08 -06:00
Roland
2328524ab2 controller: VNC toggle, gated on the Desktop running
Add a VNC toggle to the Controller (both UIs) that starts/stops whichever VNC
backend is installed: wayvnc for Wayland (Raspberry Pi OS), RealVNC for X11.

Runtime-only, like the Desktop toggle: the unit is never enabled, so VNC does
not return after a reboot. Users who want it permanently on can enable it with
raspi-config.

VNC serves the desktop session, so starting it without a desktop gives clients
a blank screen. The gate lives in the backend, not just the UI, so curl callers
get it too (409, distinct from a systemctl failure). Stopping is never gated so
a running VNC server can't be stranded on with no way to turn it off.

Backend selection deliberately avoids raspi-config's live-compositor probe
(pgrep labwc), which reports X11 — and would pick the wrong backend — whenever
the desktop is stopped. wayvnc also requires its config to exist; that is
checked directly rather than via the unit's ConditionResult, which reports 'no'
for any unit that has never been started.

Extracts _unit_properties()/_systemctl_unit() so Desktop and VNC share one
systemd probe and start/stop path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 18:10:05 -06:00
Roland
9e8050bd48
controller: restore Desktop toggle state when a start/stop request fails
Greptile P1: on failure the status poll never re-syncs (the desktop did
not change), so the toggle stayed disabled until remount.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 15:19:47 -06:00
Roland
3e559e1bf8 controller: Desktop start/stop toggle; rename hotspot/bluetooth actions to what they do
Add a Desktop toggle to the Controller (both UIs) that starts/stops
display-manager.service. Fail open: the unit is never enabled/disabled,
so the desktop always returns on reboot; users who want it disabled
across reboots can set the default target themselves.

Rename the hotspot and bluetooth actions to reflect their real behavior:
- enable/disable_hotspot -> start/stop_hotspot (/api/hotspot/start|stop)
- enable/disable_bluetooth -> unblock/block_bluetooth (rfkill semantics,
  /api/bluetooth/unblock|block)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 15:19:36 -06:00
Roland
0850b6a7fe
app: handle form fetch failures without unhandled rejection
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Catch useForm fetcher errors and surface them so DomainEditPage can show a not-found message instead of hanging on the loader (and so the webpack overlay no longer flakes Cypress).
2026-07-16 19:40:28 -06:00
Roland
73fafcd645
Search: fast ABC mode by default, opt-in deep search
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries.  Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.

Database:
- Migration adds the generated column and a GIN index (rewrites the
  table; a few minutes on large libraries).

Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
  DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
  query (COUNT FILTER on the abc subset): `file_groups` (fast) and
  `file_groups_deep`.

UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
  list.
- The gap is reconciled by a hint offering deep search: count-aware on
  the global search ("No results in fast search. N results available in
  deep search." / "N more results available..."), generic on the module
  pages (no deep estimate there).  The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
  ?deep=true.

Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside.  Also skip flaky test_download_channel on
CircleCI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 10:32:16 -06:00
Roland
2acdc2e16d
Expanding FileBrowser button widths when page is wide enough.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-07-01 19:52:15 -06:00
lrnselfreliance
cafc6514cb
Restore the censored-video search filter (#499)
* Restore the censored-video search filter

Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.

Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored).  When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 09:48:36 -06:00
Roland
c3110f3940
Warning when SMART reports unreadable sectors. Parse Seagate drive hours correctly. 2026-06-24 15:59:34 -06:00
Roland
16070e98b3
Fixing error where focusing on an empty Directory search would throw an error. 2026-06-24 15:23:32 -06:00
Roland
3d6aead072
Rebuild Inventory as a config-only system with food catalog and calculators
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.

Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
  into memory and the single source of truth. New reusable MultiFileConfig
  (wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
  (text/number/quantity/date/select/location/calories). Unit math/aggregation
  moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
  inventory save with version-based optimistic concurrency; 404 for missing),
  DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
  (pound->lb), and an alembic migration drops the tables.

Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
  Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
  customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
  inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
  calories column, and embeds a ration estimate (how long the stock lasts).

Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
  catalog_defaults.yaml (~60 generic items, total calories per package) with
  merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
  modal, and entry-row pre-fill of category/size/calories from a name match.

Calculators
- Generic Food Storage planner (like the Water calculator): household +
  editable per-person/year amounts -> total to store over a chosen duration,
  with women/children weighted by DGA calorie shares and an imperial/metric
  toggle.

Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 17:14:54 -06:00
lrnselfreliance
e018616daf
Warn in the navbar when a drive's SMART health is failing. (#471)
A failing SMART self-assessment predicts imminent drive failure, so it
deserves a prominent warning.  useDriveHealth() reads the assessment
already present in smart_stats (pySMART reports PASS / FAIL / null) and
flags any drive whose assessment is FAIL.  Nav.js renders a red warning
icon in its own always-on slot — like the power warning, and unlike the
single-slot priority chain where a transient high temp could hide it.

Frontend-only: the assessment field is already in the cached payload.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:33 -06:00
lrnselfreliance
1bd102c768
Warn in the navbar when a hard drive runs hot (#470)
* Warn in the navbar when a hard drive runs hot.

The navbar already warns about high CPU temperature; add the equivalent
for drive temperature, since an overheating disk risks data loss and a
user running an off-grid library won't be watching for it.

- controller/lib/smart.py: build_smart_stats() returns the per-drive list
  plus warning thresholds (55C warn / 65C critical) so the frontend stays
  dumb. Reading SMART wakes a spun-down drive, so drives in standby are
  detected with `smartctl -n standby` (which works through the USB-SATA
  bridges where `hdparm -C` only reports "unknown") and NOT read; their
  last-known reading is reused instead.
- controller/lib/status_worker.py: collect SMART on a slow 60s cadence
  (cached on app.state) rather than the 5s status loop, so the poll does
  not keep USB drives awake. The result flows into the same cached payload
  via a new collect_all_status(smart_stats=...) kwarg.
- Frontend: useDriveTemperature() reports the hottest drive; Nav.js shows
  a yellow/red `hdd` icon with a popup, mirroring the CPU warning.

Verified on hardware: smart_stats reports real temps and SMART runs once
a minute, not every 5s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Address Greptile review on HDD temperature warning.

- _drive_is_asleep also treats SLEEP mode as asleep, not just STANDBY.
  smartctl -n standby exits "Device is in SLEEP mode, exit(2)" for the
  deeper power mode; matching only "STANDBY" meant a SLEEP drive was read
  and woken — the exact failure the check prevents.  (TDD: added a failing
  SLEEP test, then fixed.)
- useDriveTemperature uses ?? instead of || for the temperature fallback
  to express "fall back only on null/undefined".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:33 -06:00
Roland
345b69b7d7
chore: address Greptile P2 review comments on PR 449
- In get_effective_video_settings: on unparseable sleep_requests, delete
  the key instead of forcing 0.0. This lets the global config default
  (already present from _extract_global_inheritable) take effect, as
  suggested by Greptile.

- In handleInputEvent: add explicit `value !== ''` guard before the
  isNaN check. Prevents silent NaN in form state for empty number
  fields (isNaN('') === false in JS).

Both changes are strict improvements with no behavior change for
valid inputs.
2026-05-31 13:02:44 -06:00
Roland
9cc5c15e32 fix: coerce sleep_requests and numeric settings from string form values
`sleep_requests` (and other numeric inheritable video download settings)
could arrive as strings ('' or '0.75') from the frontend NumberInputForm
and DownloadRequest bodies. These were persisted into Download.settings
(JSONB) and later triggered:

    if effective['sleep_requests'] > 0:
    TypeError: '>' not supported between instances of 'str' and 'int'

in VideoDownloader.execute_download (and similar sites).

Root cause: DownloadSettings.__post_init__ only filtered None/[] and did
no type coercion, even though the dataclass declared Optional[float]/int.

Changes (minimal targeted fix):
- Add _coerce_optional_number helper and apply it to all numeric fields
  in DownloadSettings.__post_init__ (the existing sanitization boundary).
- Centralize legacy string coercion for persisted data in the single
  read chokepoint: get_effective_video_settings.
- Simplify VideoDownloaderConfig.sleep_requests getter.
- Frontend: use parseFloat for type=number inputs + null defaults for
  sleep_requests (prevents '' from ever being submitted).
- Add focused regression test for string coercion path.
- Revert unrelated change to string field filtering behavior.

No NaN/Inf speculation, no duplicated try/float guards at every call site,
no behavior change for non-numeric settings.
2026-05-31 13:02:31 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
ae460af8d6
URLs can be dropped onto download textarea.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-05-02 13:35:08 -06:00
Roland
7bf1c80847
Fixing bug where pagination would show an extra link that would load no results.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-04-30 17:40:26 -04:00
Roland
66cecce5e6
Adding Doc filter button so users can search just PDFs/EPUB/etc. 2026-04-14 15:35:35 -06:00
Roland
4b34d37838 Navigation links automatically hide based on screen width, rather than hard breaks. 2026-04-09 16:17:38 -06:00
Roland
53f4e29b64
Updating controller hotspot and bluetooth toggle commands with real paths for RPi. 2026-03-28 10:35:07 -06:00
Roland
1c55745884
Adding bluetooth radio toggle to Controller. Writing controller.yaml to media drive on startup, if it doesn't exist. 2026-03-28 10:09:58 -06:00
Roland
282b0f63e8
Adding links next to video captions to allow scrubbing of video by the captions, adding filter to make finding quick. 2026-03-21 16:50:27 -06:00