* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page
Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:
* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending
(the normal state) this froze the loop for a second at a time; about a third
of all requests, including trivial ones, took ~1,000 ms. It now polls
`is_set()` and returns immediately.
* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
ran synchronous SQLite queries on the event loop, so one slow search stalled
every other request on that worker. They now run via `asyncio.to_thread`.
* The unfiltered Videos page (`POST /api/videos/search` with only an order)
used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
every video's index entry before LIMIT applies: seconds cold on a large
library on a spinning disk, and ~10 s hot for a deep page. It now seeks each
exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
rows and merges them, with the total from a separate count cached for 60 s.
Results are identical to the generic query.
Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a
gitignored JSON file; an example is included.
Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve the video description from its own endpoint instead of Video.__json__
`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk. The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays. The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.
Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions. The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored. The player page fetches it through
`useVideoExtras` alongside the comments.
Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: thread-local zim sessions, fast-path equivalence test, comments
* Zim search, single-zim search, and search estimates run libzim off the event
loop, each opening its own session inside the worker thread rather than
passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
query and asserts identical rows and paging for both date orders. The one
intentional difference is documented: past the end, the fast path still
reports the true total.
* switch_worker returns before its try block so the empty path does not clear
the event; its comment states the invariant only. The copied "synchronous
SQLite" comments are reduced to one line in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ui: a download form that emptied itself after every submit
A successful submit reset the whole form, so adding a second download meant
re-choosing the tags, the destination and the video settings that the first
one had already been given.
It now empties only the URLs. Everything else stays for the next download,
and a Clear button beside Download puts the form back to its defaults when
the next one is unrelated.
`useForm` gains `clearPathsOnSuccess`; `clearOnSuccess` is untouched for the
forms that want the whole reset.
* review: the one download form that kept its URLs
Scrape took a `clearOnSuccess` prop that no caller passed, so mapping it to
the URL paths left create-mode scrape clearing nothing. It follows the same
rule as Videos and Archive now: the edit form supplies `onSuccess`, and only
that keeps the form as it was.
`clearPaths` also wrote every path it was given, so a `urls` form gained a
`url: ''` it never declared -- which the comment beside it and the test named
for it both said it would not do. It skips a path neither the defaults nor
the form has, and the test asserts the key is absent.
* ui: two upload dialogs that had drifted apart
The dashboard and the file browser each had their own upload component: two
dropzones, two overwrite toggles, two copies of the progress-bar rendering,
against the same `useUploadFile` state. The file browser's dialog was made
fullscreen with a drop target that grows; the dashboard's stayed a `large`
dialog with a strip to drop onto, because it was a different file.
They differ in one thing that is real: the file browser knows the destination
because the user selected it, and the dashboard has to ask. That is now the
`destination` prop -- `undefined` asks, a string uploads there, and '' is the
media directory rather than a missing destination.
Everything else is one component, so the next change to either entry point
lands on both.
* review: an upload dialog that looped on the media directory
Pressing Upload in the file browser with nothing selected passes `''` -- the
media directory. Three checks along that path read it as "no destination":
* The clearing effect ran, and `doClear` is a new function every render whose
own setState scheduled the next one. React ended it with "Maximum update
depth exceeded", 16 warnings deep, on the default Upload path. The effect
now clears only while asking, and `doClear` is stable.
* The title read `Upload to: /media/wrolpi/`.
* `doUpload` bailed, so a drop on the media root uploaded nothing.
All three predate this branch; moving the dialog is what put them behind one
component. Tests cover the title, what the hook is told, and that a fixed
destination is never the thing that empties the queue.
The library has been uninstalled for some time, but the codebase still read as
though it were there: identifiers named after it, rules written against its
class names, and about 180 comments explaining a rule in terms of a thing that
is no longer installed.
Renamed, keeping the vocabulary the app actually uses: semanticUIColorMap ->
navColorHexMap, semanticColorNames -> paletteColorNames, semanticNames ->
iconNameAliases, semanticSizes -> sizeAliases, and SemanticUIColors ->
NavColors in wrolpi/schema.py, which is what validates nav_color on the way in.
ColorToSemanticHexColor had no callers at all and is gone.
The baseline ratchet has done its job -- the list reached zero, so the JSON is
deleted and the guard keeps only the checks with teeth: no package, no import,
no `className="ui "`, no toast library. Its Theme.tsx check went with the
wrappers it was watching for. It also asserts its own walk found files, so a
clean result cannot mean it scanned nothing.
Deleted from App.css: every rule requiring `.ui`, `.inverted`, `.segment` or
`.modal`, none of which any component has set since the stylesheet stopped
loading, plus the form skeletons, ui-alerts and progress-text rules that
nothing renders. Two of them were doing harm rather than nothing: the drag
tints had a dark-theme variant under `.inverted` that stopped applying, leaving
15% of a fixed blue over a dark row, and now mix a token instead.
Two selectors naming markup that no longer exists were live bugs. The file
browser's filter had `.ui.input.file-browser-filter`, so it rendered at the
browser default width on a bar laid out to give it 34em -- now qualified by its
container, because `.wrolpi-action-input` sets `width: 100%` at the same
specificity and is loaded later. And SearchBox takes named props rather than a
rest spread, so the `id` DestinationForm hands it was dropped in silence and
the label it renders pointed at nothing; the download forms' Destination field
has had no associated label since the migration.
"Semantic roles" survives in four places, where it means what it says.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It asserted the label is block but not the 4px gap, so dropping the
margin -- the half that actually lines an InputForm up with the
DestinationForm beside it -- would still have passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
InputForm returned a fragment, so its label and its field arrived as two
children of whatever contained them. In a grid that is two cells: on
/videos/settings the "Video File Format" and "Extra yt-dlp Arguments"
inputs sat in the column to the right of their own labels.
The label is now block with the same 4px gap the migrated fields use
(FieldLabel in Download.js), so an InputForm and a DestinationForm side
by side start their inputs at the same height -- the Channel Name and
Directory fields on the Channel edit page were off by the difference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four faults, each measured in the running app before and after.
The search glyph carried Semantic's `item` class, which has had no rules at all
since Semantic was removed: no pointer cursor, no hover, and an 18.6px hit area
against a 54.4px bar. It is the same kind of thing as Help and Admin and now
takes the same class. The share glyph did compute `cursor: pointer` -- it is an
`<a href>` -- but its target was the 24px box the icon occupies, so the pointer
appeared only if you found it exactly. Corner anchors now stretch and pad like
a link.
The More button rendered with NO class at all: a bare grey UA button while the
rest of the bar took the user's colour. Menu.Target clones its child and hands
it a className of its own, and the trigger spread those cloned props after its
own `className` attribute, so the clone replaced ours. It is merged now. This
was mine, from extracting NavDropdownTrigger last commit -- and it also made the
previous commit's "measured and drawn are the same component" claim only half
true, because the hidden placeholder is not cloned and so kept its class.
The bar still wrapped, for a reason the padding fix could not reach: the corner
does not keep a constant width. The Cmd-K hint renders only once
useKeyboardDetected has seen a keypress, so it appears at an arbitrary later
moment and widens the corner by ~30px; the load, memory, temperature and drive
warnings come and go with the machine's state. None of that resizes the
container, which was the only thing observed -- so nothing recalculated. That
is the other half of "at certain widths": the same viewport wrapped or not
depending on whether the user had touched a key. The corner and the home link
are observed too now.
The search modal showed an input and the first line of the first group. The
suggestion list is absolutely positioned, which is right everywhere else -- a
list that reflows the page while you type is unusable -- but in a modal it fails
twice: an absolute list adds no height, so the panel sized itself to the search
box alone at 114px, and the panel's own `overflow: auto` then clipped the list
at that edge. In the modal the list sits in flow; the inverse is asserted so
the rule cannot leak to the pages where overlaying is correct.
Two of the new tests were wrong first and reverting the fix left them green:
they asserted hand-written markup carrying the classes I expected the components
to have, rather than the real SearchIconButton and ShareButton. They use the
real components now. Every fix was then planted against: 5 red for the More
class, 2 for the search class, 2 for the corner hit area, 1 for the observer,
2 for the modal.
1115 jest / 62 suites, 196 in ui-layout.cy.js, clean npm run build.
Confirmed live: share and search both pointer at 54.4px, More white on the bar
colour with a transparent face, and the modal's list in flow and unclipped.
Three faults in the bar, all reproduced by measurement before being fixed.
The indicators sat low. The slot carried `marginTop: 0.8em`, a nudge left over
from Semantic's menu; the corner centres its children, and a top margin is part
of the margin box being centred, so it applied as a ~6px downward shift. It
read as an error on the hotspot and the theme picker but not on share or search,
which is why it survived: an IconButton is a square ActionIcon that centres its
glyph exactly, while a bare-icon anchor sits high inside a line box that
reserves descender space for text that is not there. The nudge cancelled that
out on one shape and was pure error on the other. Both are laid out now --
flex, centred, no nudge -- and the anchors get `display: flex` so they hug their
glyph instead of a text line.
Help and Admin highlighted only a box around their text while Videos and
Statistics highlighted the full height. Same class, same rules, different
parent: the bar stretches its children, but the corner is a nested flex
container that centres its own, so a link inside it was sized to its content.
The hover background is painted on the link box, so that box IS the affordance.
The bar broke onto a second line at certain widths. Two measurement errors that
only overlap in a narrow band, which is why it appeared while resizing rather
than always. useOverflowNav measured the WRAPPER's offsetWidth and subtracted
only the home link and the corner, missing the bar's own `padding: 0 0.25em` --
about 8px it believed it had. And the hidden "More" placeholder it measured was
a plain button with no dropdown caret, about 21px narrower than the button that
eventually rendered. The placeholder is now the same component as the real
trigger, so the two cannot drift again.
Tested in ui-layout.cy.js against the real <DesktopNav/>, which takes its
colours, home link and indicators as props and so needs none of NavBar's polling
hooks. Indicator centres are measured against the bar's centre rather than each
other, because three things can be equally and consistently wrong and were. The
wrap case sweeps 1180px down to 640px in 6px steps -- a spot check steps over a
band that narrow. All three failed first: share 3.4px high, Help 24.8px against
a 54.4px bar, and two rows at 1156px.
Confirmed on the running app: every indicator at 0.0px from the bar's centre,
Help and Admin at the full 54.4px, and no wrap anywhere from 1500px to 780px.
1115 jest / 62 suites, 182 in ui-layout.cy.js, clean npm run build.
`semantic-ui-react`, `react-semantic-toasts-2` and `semantic-ui-offline` are
gone from package.json, the lockfile and node_modules -- 19 MB of dependencies
that nothing had imported since the migration finished.
Four things still referenced them and would have broken on uninstall. None was
caught by a build or by jest:
Four test files still ran `jest.mock('react-semantic-toasts-2', ...)` for
components that no longer use it. A jest.mock of a module that does not exist
throws, so the packages could not have come out without this.
`src/components/Download.cy.js` and `cypress/support/component.js` both
imported semantic.min.css. The support file now loads what the app loads --
index.css, the font and token stylesheets, and Mantine's.
The ratchet changes job rather than being deleted. An import of a missing module
fails loudly now, but three quieter mistakes would not: a `className="ui button"`
that styles nothing, a file reaching back through Theme.tsx, and someone
reinstalling a package to "just use one Semantic component". It asserts all
three, plus that package.json names nothing matching /semantic/i.
Two pre-existing problems in the cypress component harness surfaced while
checking the uninstall had not broken it, both fixed because that file had to be
edited anyway:
It imported `react-router-dom`, which is not installed -- this app is on
react-router v7. Dates to 5621d473, so the component suite has been
un-runnable since well before this migration.
Its mount helper supplied no MantineProvider, so every library component threw
-- the same gap test-utils.js had for jest.
The component specs now load and four pass; eleven assertions are still written
against markup from before they stopped running. Repairing those is its own
piece of work, deliberately not bundled in here. The e2e suite is unaffected:
43/43.
ControllerPage was the final call site; Theme.tsx then had nothing left to wrap,
so it drops from 861 lines to 338 -- the provider that resolves the theme, stamps
`data-theme` and `data-media-filter`, persists both and hands Mantine our tokens.
Everything below that was Semantic wrappers.
The dark-mode compatibility props went with them. `i`, `s`, `t` and `inverted`
existed so unmigrated components could borrow Semantic's dark treatment, and
they carried hardcoded greys -- #1B1C1D, #dddddd, #eeeeee -- which is exactly
what the token themes replaced. Three call sites in Common.js were still
injecting the #eeeeee one into links.
Then `semantic-ui-offline/semantic.min.css` came out of App.js. That took more
with it than expected, and looking at the running app is the only reason it was
caught:
Every link lost its colour and gained the browser's underline and visited
purple. The dashboard's load averages and section headings were underlined
and purple. Links now take `--blue`, which remaps per theme, and underline
only on hover -- many of them wrap cards, posters and icons.
`box-sizing: border-box` disappeared, which silently changes the size of every
padded element in the app. Semantic's reset had been providing it.
The navigation bar overflowed a 1000px viewport with `nowrap` and cut "Admin"
off the right edge -- the one link a user needs to reach the settings that fix
a broken WROLPi. Semantic's menu wrapped; ours does now too.
The gzipped CSS bundle is 47.5 kB, and one call site was still leaning on
Semantic's grid classes -- now a flex column, since all three of its callers
pass library components.
The ratchet stays rather than being deleted. At zero it asserts four things: no
file imports Semantic UI, none calls its toast library, none leans on its
stylesheet, and no file reaches back through Theme.tsx. It can go when the
packages are uninstalled.
Theme.test.js is deleted, but not before porting the two behaviours it
protected: that Button forwards a ref to the real DOM element (a class-component
ref once crashed the Control page, because portals call `node.contains`), and
that the progress percentage stays readable on every bar colour.
Phase 2, part 4. Baseline 50 -> 15.
Migrated: all eight calculators, the rest of inventory and collections, the
Map/Zim/Docs cluster, Videos, Files, Search, Archive, DashboardPage,
DatesSelector, HelpModal, Events, Tags, Flasher's neighbours, admin Settings
and Status, Download, and App.
Also the three shared modules everything imports: api.js and customHooks.js
swap the Semantic toast library for the library's shim (same call shape), and
useForm.js renders the library's Message. useForm accepts both the old
`{content, header, positive|negative}` shape and the new `{kind, title}`,
because the pages building those objects migrate separately.
Two library gaps closed, both found by call sites needing them:
ActionInput is a forwardRef now. Its ref reaches the input rather than the
wrapper, because callers use it to focus or select the field -- a ref to the
surrounding div would do neither, which is why one call site had resorted to
hand-copying the component's markup.
Header takes a `color` token name. Three separate call sites had reached for
an inline style to colour a heading that carries meaning ("3 items to add"
against "2 to remove"), which is how a hex ends up in markup.
Two files needed repair after being interrupted mid-edit: MapViewer had a
Segment left in its error branch, and VideoPlayer had lost the useContext
import its own code still used. Neither jest nor tsc noticed; only
`npm run build` did, which is the whole reason that check exists.
Fifteen files remain: Archive, BulkTagModal, Channels, Docs, FileBrowser,
FilePreview, Flasher, Nav, Playlists, Theme.tsx, Upload, VideoPlayer, Zim,
ControllerPage, admin/Downloads. Theme.tsx is deliberately last -- the
unmigrated files still import its Semantic-backed wrappers, so it can only go
once they have.
Phase 2, part 3. Baseline 63 -> 50.
Common.js is the one that mattered: 51 files import it, so its Semantic usage
was propping up the rest of the migration. Everything in it now comes from the
library -- Paginator, the two search inputs, DirectorySearch, TabLinks, the
toggles, the messages, APIButton, Breadcrumbs, SimpleAccordion, IframeViewer,
CardPoster, FileIcon, LoadStatistic, the theme menu, and the hotspot icons.
Also migrated: Placeholder, Apps, ArchivePreview, CbzViewer, AddToPlaylist,
TaggedDeleteConfirmModal, the collection edit/tag forms, and four inventory
components.
New in the library, all in the gallery and tested:
SearchBox Semantic's `<Search category>` with its grouped results shape
kept, because four call sites already build it. A real
combobox rather than a div that listens for clicks.
Pagination Semantic's prop names; onPageChange gets the page number.
TabBar the strip only -- routing stays with the caller.
IconStack Semantic's IconGroup, named once for assistive technology.
Table.HeaderCell sorted/onSort -- a real button carrying aria-sort.
Button/IconButton translate Semantic's size names.
That last one is defensive: unmigrated callers still pass size='tiny', and
Mantine drops a size it does not recognise without a word, so the button
quietly renders at the wrong size. Mapping here keeps the mixed state honest.
Three real bugs found and fixed on the way, none of them cosmetic:
* `fileSuffixIconName` returned fourteen icon names the map did not have, so
opening a folder with a PDF, a Word document or an archive in it logged an
error and drew a fallback glyph. All fourteen mapped, with a test that
walks the mapper's own return values, and another asserting every mapped
name resolves to a Tabler component that exists.
* `Statistic` relied on Semantic's `Statistic.Group` for its spacing, so on
the dashboard the load averages ran together into "1 MIN. LOAD5 MIN.
LOAD15 MIN. LOAD". It carries its own spacing now.
* Semantic's inverted stylesheet colors every descendant of an inverted
segment, so Toggle labels inherited #dddddd -- white pixels in night mode,
on the Settings page, which is exactly where someone adjusts their WROLPi
in the dark. Restated from tokens.
The baseline ratchet gained a third assertion: a migrated file may still lean
on Semantic's CSS classes without importing the library, which the import check
cannot see and which would collapse silently when Phase 4 deletes the
stylesheet. There is one such bridge today and it is now declared.
The toggle-hook de-duplication replaced a range of customHooks.js that also
contained these two unrelated hooks, breaking the production build. Restored
byte-identical from master; exports now match master exactly.
Jest missed this because it only compiles what tests import — `npm run build`
is what catches a missing export.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hotspot, Bluetooth, Desktop, VNC and Throttle toggles had each been written
out longhand at four layers. Collapse the shared parts:
- useSubsystemToggle() replaces five near-identical hooks. This also fixes a
latent bug: only useDesktop/useVnc restored state when a request failed, so a
failed Bluetooth/Throttle/Hotspot toggle left at null and — since the
status poll sees no change — the toggle stayed disabled and read as
unsupported until it remounted.
- SubsystemToggle renders all five, with the Desktop's stop confirmation and
VNC's desktop gating as declared options rather than bespoke copies.
- The fallback UI's four update*/toggle* pairs become one config-driven
implementation (-100 lines).
- _raise_for_action() centralises the 13 endpoint failure checks, including
VNC's 409-vs-500 split.
Deliberately not shared: the six get_*_status_dict() builders (payloads
genuinely differ) and the per-subsystem mechanisms (nmcli/rfkill/sysfs/systemctl).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a VNC toggle to the Controller (both UIs) that starts/stops whichever VNC
backend is installed: wayvnc for Wayland (Raspberry Pi OS), RealVNC for X11.
Runtime-only, like the Desktop toggle: the unit is never enabled, so VNC does
not return after a reboot. Users who want it permanently on can enable it with
raspi-config.
VNC serves the desktop session, so starting it without a desktop gives clients
a blank screen. The gate lives in the backend, not just the UI, so curl callers
get it too (409, distinct from a systemctl failure). Stopping is never gated so
a running VNC server can't be stranded on with no way to turn it off.
Backend selection deliberately avoids raspi-config's live-compositor probe
(pgrep labwc), which reports X11 — and would pick the wrong backend — whenever
the desktop is stopped. wayvnc also requires its config to exist; that is
checked directly rather than via the unit's ConditionResult, which reports 'no'
for any unit that has never been started.
Extracts _unit_properties()/_systemctl_unit() so Desktop and VNC share one
systemd probe and start/stop path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Greptile P1: on failure the status poll never re-syncs (the desktop did
not change), so the toggle stayed disabled until remount.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a Desktop toggle to the Controller (both UIs) that starts/stops
display-manager.service. Fail open: the unit is never enabled/disabled,
so the desktop always returns on reboot; users who want it disabled
across reboots can set the default target themselves.
Rename the hotspot and bluetooth actions to reflect their real behavior:
- enable/disable_hotspot -> start/stop_hotspot (/api/hotspot/start|stop)
- enable/disable_bluetooth -> unblock/block_bluetooth (rfkill semantics,
/api/bluetooth/unblock|block)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catch useForm fetcher errors and surface them so DomainEditPage can show a not-found message instead of hanging on the loader (and so the webpack overlay no longer flakes Cypress).
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Restore the censored-video search filter
Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.
Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored). When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.
Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
into memory and the single source of truth. New reusable MultiFileConfig
(wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
(text/number/quantity/date/select/location/calories). Unit math/aggregation
moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
inventory save with version-based optimistic concurrency; 404 for missing),
DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
(pound->lb), and an alembic migration drops the tables.
Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
calories column, and embeds a ration estimate (how long the stock lasts).
Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
catalog_defaults.yaml (~60 generic items, total calories per package) with
merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
modal, and entry-row pre-fill of category/size/calories from a name match.
Calculators
- Generic Food Storage planner (like the Water calculator): household +
editable per-person/year amounts -> total to store over a chosen duration,
with women/children weighted by DGA calorie shares and an imperial/metric
toggle.
Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A failing SMART self-assessment predicts imminent drive failure, so it
deserves a prominent warning. useDriveHealth() reads the assessment
already present in smart_stats (pySMART reports PASS / FAIL / null) and
flags any drive whose assessment is FAIL. Nav.js renders a red warning
icon in its own always-on slot — like the power warning, and unlike the
single-slot priority chain where a transient high temp could hide it.
Frontend-only: the assessment field is already in the cached payload.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Warn in the navbar when a hard drive runs hot.
The navbar already warns about high CPU temperature; add the equivalent
for drive temperature, since an overheating disk risks data loss and a
user running an off-grid library won't be watching for it.
- controller/lib/smart.py: build_smart_stats() returns the per-drive list
plus warning thresholds (55C warn / 65C critical) so the frontend stays
dumb. Reading SMART wakes a spun-down drive, so drives in standby are
detected with `smartctl -n standby` (which works through the USB-SATA
bridges where `hdparm -C` only reports "unknown") and NOT read; their
last-known reading is reused instead.
- controller/lib/status_worker.py: collect SMART on a slow 60s cadence
(cached on app.state) rather than the 5s status loop, so the poll does
not keep USB drives awake. The result flows into the same cached payload
via a new collect_all_status(smart_stats=...) kwarg.
- Frontend: useDriveTemperature() reports the hottest drive; Nav.js shows
a yellow/red `hdd` icon with a popup, mirroring the CPU warning.
Verified on hardware: smart_stats reports real temps and SMART runs once
a minute, not every 5s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Greptile review on HDD temperature warning.
- _drive_is_asleep also treats SLEEP mode as asleep, not just STANDBY.
smartctl -n standby exits "Device is in SLEEP mode, exit(2)" for the
deeper power mode; matching only "STANDBY" meant a SLEEP drive was read
and woken — the exact failure the check prevents. (TDD: added a failing
SLEEP test, then fixed.)
- useDriveTemperature uses ?? instead of || for the temperature fallback
to express "fall back only on null/undefined".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- In get_effective_video_settings: on unparseable sleep_requests, delete
the key instead of forcing 0.0. This lets the global config default
(already present from _extract_global_inheritable) take effect, as
suggested by Greptile.
- In handleInputEvent: add explicit `value !== ''` guard before the
isNaN check. Prevents silent NaN in form state for empty number
fields (isNaN('') === false in JS).
Both changes are strict improvements with no behavior change for
valid inputs.
`sleep_requests` (and other numeric inheritable video download settings)
could arrive as strings ('' or '0.75') from the frontend NumberInputForm
and DownloadRequest bodies. These were persisted into Download.settings
(JSONB) and later triggered:
if effective['sleep_requests'] > 0:
TypeError: '>' not supported between instances of 'str' and 'int'
in VideoDownloader.execute_download (and similar sites).
Root cause: DownloadSettings.__post_init__ only filtered None/[] and did
no type coercion, even though the dataclass declared Optional[float]/int.
Changes (minimal targeted fix):
- Add _coerce_optional_number helper and apply it to all numeric fields
in DownloadSettings.__post_init__ (the existing sanitization boundary).
- Centralize legacy string coercion for persisted data in the single
read chokepoint: get_effective_video_settings.
- Simplify VideoDownloaderConfig.sleep_requests getter.
- Frontend: use parseFloat for type=number inputs + null defaults for
sleep_requests (prevents '' from ever being submitted).
- Add focused regression test for string coercion path.
- Revert unrelated change to string field filtering behavior.
No NaN/Inf speculation, no duplicated try/float guards at every call site,
no behavior change for non-numeric settings.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:
- wrolpi-chrome.zip Load-Unpacked sideload for Chrome / Brave / Edge
- wrolpi-firefox.xpi Mozilla-signed (unlisted self-distribution),
one-click install in any Firefox
- versions.json Per-browser version metadata for the install page
Backend (modules/archive/api.py, wrolpi/root_api.py):
- POST /api/archive/upload accepts an optional repeated `tagNames`
multipart form field; the upload switch handler applies tags to the
new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
created inside a per-name SAVEPOINT so concurrent inserts on the same
novel name don't abort the transaction (Tag.name has unique=True).
- GET /api/extensions returns metadata (per-file availability + size +
versions.json contents) for the install page.
- GET /api/extensions/<filename> serves the binary, whitelisted by
filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
Content-Disposition: attachment header, so Firefox triggers its
add-on install handler instead of saving the file.
Frontend (app/src/):
- New /admin/extension page (linked from AdminRoute) detects the user's
browser via navigator.userAgent and renders an install card per
Firefox / Chromium / Other. Pulls metadata from /api/extensions to
show file size and version, and gracefully reports 'Not yet
available' when binaries are missing.
- Dashboard's Getters component supports a deep link from the
extension's 'Configure Download' button:
/?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
On mount, the matching downloader form auto-opens with URLs pre-
filled. Query params are cleared on close so a refresh doesn't
re-open the modal.
- DownloadMenu and the per-downloader forms accept an `initialUrls`
seed via the existing `download` prop pattern, joined with newlines
for the `urls` textarea (and first URL for single-URL forms).
Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).
Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
* Batch reorganization is supported for both Domain and Channel Collections.
* Batch reorganization happens when any Collection does not match the respective file format. For example, a Channel Collection file format that does not match the `yt_dlp_options.file_name_format` will show `needs_reorganization`.
* Individual Collections can always be reorganized. The reorganization is a little more aggressive. Useful for when some files may have been missed in the batch reorganization.
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
