YouTube's new listing layout means yt-dlp no longer returns `view_count`
in flat channel/playlist entries, so sorting by views raised KeyError and
the download failed and retried forever.
Sorting by views now requires per-video extraction (hundreds of requests
per channel), so the feature is deprecated rather than degraded:
- downloader: the 'views' branch raises UnrecoverableDownloadError so the
download fails cleanly (no retry loop) with a message to switch to
Newest/Oldest. May be revisited via the YouTube Data API.
- schema: download_order validation drops 'views' (newest/oldest/null only).
- UI: removed "Most Views" from the download-order dropdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
getSettings() in api.js remaps the /settings response through a fixed field
whitelist, which dropped download_daily_limit_global/per_domain before they
reached the SettingsContext. The values saved correctly but always rendered
empty. Add both fields to the mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings.js: send 0 (not null) when a daily-limit field is cleared, so the
settings API actually clears a previously-saved limit instead of dropping the
null and keeping the old value.
- downloader.py: document that the daily limit is a best-effort soft cap; minor
bounded overshoot is possible when create_downloads' background dispatch overlaps
the perpetual cycle, and self-corrects the next cycle. A strict cross-process cap
is intentionally not implemented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day
Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.
Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
stops dispatching counted downloads for the cycle. daily_download_counts derives the
day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
(base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
validation, 0 = unlimited), and the React Settings page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dark-theme rule hardcoded the inverted Accordion's title/content
background to #171616, which is the Modal content color. Inside a Modal
this matched, but on the /admin/status page (and any Segment, which
renders at #1B1C1D) the Accordion showed up as a darker band.
Make the inverted Accordion transparent so it inherits whatever
container it sits in, matching how the light theme already renders
accordions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix FileWorker status stranded mid-phase (e.g. stuck on "Cleanup")
The FileWorker stores its status in shared context and relies on an explicit
reset_status() in each handler. Two execution paths set the status (comparing/
upserting/deleting/modeling/indexing/cleanup) but could finish without ever
resetting it, freezing the UI at the last phase:
- refresh_sync() ran _apply_post_processing (final phase sets status='cleanup')
but never called reset_status(). Deleting files in the UI goes through
lib.delete -> refresh_sync, so any delete that changed the DB stranded the
worker at 'cleanup' (observed on 10.0.0.8 after deleting STL files).
- process_queue() wrapped the handler dispatch in try/finally with no except.
An exception in handle_count/handle_refresh (which, unlike handle_move/
handle_reorganize, have no try/except) escaped to the perpetual worker loop
and left the status frozen at whatever phase was running ("any state").
Fixes:
- refresh_sync(): reset_status() in a finally.
- process_queue(): on exception record a terminal 'error' status (so the worker
is not stranded mid-phase) and re-raise so callers and the perpetual loop
still observe the failure. This covers both handle_count and handle_refresh.
Tests: add replication tests for the synchronous-refresh, handler-error, and
count-error stranding paths, plus the previously-untested wait_for_job timeout.
* refresh_sync: leave terminal 'error' status on failure
Address Greptile review: refresh_sync reset to idle in a finally that also ran
on failure, so a synchronous refresh that raised after a phase was set reported
a clean 'idle' to the status endpoint instead of the failure (the exception was
still raised to the caller). Make it symmetric with process_queue: reset to idle
only on success, record status='error' and re-raise on failure.
Add test_refresh_sync_records_error_status_on_failure.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tracked release/ directory collides with the release branch name, so a
bare `git checkout release` is ambiguous ("could be both a local file and a
tracking branch") and aborts a fresh install. Use `checkout -B <branch>
origin/<branch>`, which can never be parsed as a path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore the censored-video search filter
Censored videos (no longer publicly available — detected when a channel
refresh drops them from the channel list, or when comment downloads fail
with "video unavailable/private/removed") are already flagged on
FileGroup.censored, but there was no way to find them since the search
rework dropped the old filter.
Add a "Censored" toggle to the reworked search filter modal, shown only on
the Videos page (showCensored). When enabled the video search returns only
videos with FileGroup.censored = TRUE, tracked via a ?censored=true query
param (useSearchCensored).
* Move Availability filter below the Tags/Results row
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Surface coarse SMART health for USB-bridge drives
USB enclosures that reject ATA pass-through (e.g. Seagate Expansion) are
invisible to pySMART/DeviceList, so the drive shows no SMART at all on the
Controller page. Fall back to smartctl over the SCSI transport, which many
such bridges still answer, to report a coarse PASS/FAIL health plus drive
identity (model/serial/capacity).
The health (-d scsi -H) and identity (-d scsi -i) reads are issued as
separate calls: combining them makes smartctl suppress the health line when
the identity read reports "SMART support is: Unavailable" (which these
bridges do). The entry is marked smart_limited (labelled "(limited)" in both
UIs) so a green PASS is not mistaken for full monitoring — these drives
expose no attributes or temperature, and some bridges report "OK" statically.
* Harden SCSI health fallback (Greptile review)
- _scsi_health: treat any unrecognised status (e.g. "UNKNOWN", "N/A") as
unreadable (None) instead of a false FAIL. Only "OK" -> PASS and "FAIL*"
-> FAIL; anything else omits the drive rather than showing a red FAIL that
contradicts the smart_limited disclaimer.
- get_all_smart_status: isolate the SCSI supplement loop in its own
try/except so an error there cannot discard the pySMART results already
collected.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Consolidate search options into a single Filter modal
Reworks the file-search UI on the Dashboard, Videos, Archives, and Docs
pages. The crowded row of controls (sort, tags, limit, file-type filter,
date selector) is replaced with a single "Filter" button that opens one
comprehensive modal.
- Add SearchFilterModal + SearchFilterButton (violet + count badge when
active). Sections (Sort, File Type, Tags, Results Per Page, Published
Date) are shown per-page via props; layout uses a Semantic Grid.
- Selections are drafted locally and applied in a single updateQuery only
on close/Done, so the search isn't re-run on every field change.
- Results Per Page and Sort use buttons (fewer clicks, no clipped dropdown).
- Extract shared SearchControlBar (view toggle + input + Filter) for the
three results pages; move the global-search view toggle onto the tab row.
- Rework the Dashboard search line into one adaptive flex row; clear button
only shows when there's an active search.
- Remove now-dead FilesView limit/tag returns and SearchLimitDropdown.
* Address review: reset limit on Clear All; use replace over replaceAll
- handleClearAll now resets Results Per Page to the default page size
(DEFAULT_SEARCH_LIMIT = 24) so "Clear All" actually clears every setting.
- Replace start-anchored replaceAll(/^-/g) with replace(/^-/) for the sort key.
- Add a test covering the Clear All limit reset.
* Remove now-unused Media import from Files.js
The FilesSearchView refactor dropped the last <Media> usage in this file,
leaving an unused import (a build warning). Remove it.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #494 re-pointed the submodule at 80648d3 to preserve the previously-pinned
snapshot, but that commit predates the Inventory documentation, so the Docker
help image had no Inventory pages. Bump to 002e747 (current help master tip),
which includes the Inventory docs and the Restore section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The wrolpi-help history was rewritten to correct the author email
(rolobio@rolobio.com -> roland@learningselfreliance.com) and GPG-sign all
commits, which changed every SHA from the affected commits onward and orphaned
the previously-pinned 2c75fbb. Re-point the submodule at 80648d3, the
content-identical (same tree), signed equivalent of the old pin, so
`git submodule update --init` and the Docker help build keep working.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inventories are a config-only MultiFileConfig (config/inventory/<slug>.yaml) and
were not reachable by the admin config Restore UI, so there was no way to re-import
a hand-edited/copied-in file or restore an inventory from its daily backup.
Backend:
- MultiFileConfig gains generic helpers: get_backup_dates(slug)/_get_backup_file
(surface the daily config/backup/<subdir>/<slug>-DATE.yaml files) and reimport()
(rebuild the in-memory store from disk; picks up hand-edits and copied-in files,
drops files deleted on disk). backup_date is constrained to YYYYMMDD so it cannot
traverse out of the backup directory.
- InventoriesConfig.preview_restore/apply_restore: overwrite replaces the whole
inventory; merge unions items by id (keeps current fields and existing items).
The pre-restore state is backed up automatically, so a restore is undoable.
- API: POST /inventory/reimport, GET /inventory/<slug>/backups,
POST /inventory/<slug>/restore/preview, POST /inventory/<slug>/restore.
Frontend:
- New InventoryImportModal (opened from a Restore button in the Inventory toolbar):
Re-import from disk, plus a dated-backup table with per-row Merge/Overwrite that
previews the change before applying.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user can type a calculation into any numeric inventory field (number,
quantity magnitude, calories) and have it evaluated in place — e.g. weigh a
400 g box of nails, enter '400 - 20', and Tab/Enter swaps in 380. The result
feeds dependent logic like count-by-weight automatically.
- evaluateExpression() (units.js) reuses the scoped mathjs instance, guarded by
a strict regex so only bare arithmetic (digits, + - * / ( ) and spaces) is
evaluated; anything with a letter, a plain number, or unparseable input is
left untouched. Float noise is rounded off.
- FieldCell evaluates on blur (Tab/click-away) and on Enter: in a numeric field
the first Enter resolves a pending expression, a second submits the row.
- NUMERIC_TYPES is exported once from units.js and shared by the table and cell
editors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Counting bulk hardware (screws, nails) by hand is slow. This adds a workflow
to estimate the count from weight instead: weigh one item, weigh the lot, and
the count is filled in automatically.
- The Fields modal gains a "Count by Weight" button that adds two quantity
fields, Unit Weight and Total Weight (default grams), and tags the Count
field as computed. An existing `count` field is linked in place rather than
duplicated, so hand-counted items keep their values.
- The link is plain metadata on the Count field
(`compute: {kind: 'count_by_weight', total, unit}`), so it round-trips
through the config/API with no backend change (covered by a backend test).
- During item entry, Count = round(Total Weight / Unit Weight), converting
mismatched units first (g/kg/oz) and refusing to divide when only one side
has a unit. It rounds to a whole number (no 403.7 nails).
- When a weight changes, the count always reflects the new reality: it is set
to the computed value (0 for a sub-unit total) when both weights are present,
or cleared when they can't produce one (a weight blanked, incompatible
units) — so a stale count can never persist against edited weights. A
hand-typed count is only touched when a weight actually changes.
countByWeight lives in units.js; the field preset, config detection, and the
entry-time apply hook live in a new computeFields.js. Covered by unit tests
(countByWeight, computeFields), InventoryTable auto-fill / clear / sub-unit on
draft and inline edit, the FieldSchemaEditor preset, and a backend
field-metadata round-trip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking a cell in the Items table now focuses that field's input and selects
its contents, so the user can type a replacement value immediately. FieldCell
gains an `autoFocus` prop (focus + select on mount via a merged ref that still
supports the draft row's first-field ref); InventoryTable tracks which cell was
clicked and flags the matching FieldCell.
Also add a FieldSchemaEditor test covering the select-options comma fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build on the rebuilt config-only Inventory with three related additions.
Recommended example inventory:
- A fresh install now seeds the "Food Storage" example with the recommended
one-adult / one-year (~3,000 kcal/day) emergency food supply (60 items,
~1.12M kcal), instead of an empty inventory.
- The counts are layered on top of the shipped food catalog (single source of
per-package nutrition) via RECOMMENDED_ONE_YEAR_FOOD_COUNTS, joined at seed
time by recommended_food_storage_items().
- Genericized the two remaining brand names in the catalog defaults
(Velveeta -> Processed Cheese, Spam -> Canned Lunch Meat) so both the catalog
and the seeded inventory stay brand-free.
Ration tab (split out of Summary):
- Moved the Ration Estimate (People / Daily Calories / estimate table) out of
the increasingly long Summary tab into its own "Ration" tab, shown only for
inventories that have a calories field.
Supply planner ("Plan More Supply"):
- A slider picks a target duration longer than the current estimate; the whole
inventory is scaled up proportionally (every item by the same factor, mix
kept balanced) into a sortable shopping list of what to buy.
- Any split package rounds UP to a whole one (ceil), matching the "1.6 buckets
-> buy 1 more" rule.
- The shopping list is exportable as CSV (reusing toCSV) and PDF (reusing the
print-to-PDF path, scoped via body.printing-shopping so only the list prints
-- no inventory or summary).
Search filter:
- Moved the items search out of the top toolbar (which implied it filtered
every tab) into the Items tab, so it clearly narrows Items only;
Summary/Ration/Export/print always use the full inventory.
Aggregation/role helpers (findNameKey, planSupplyPurchase) live in summarize.js,
the single source of truth shared by Summary, export, and the ration views.
All new pure functions and UI flows are covered by tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`download_info_json` called `Video.get_by_url(url, session)`, but the method
signature is `get_by_url(session, url)`. This passed the URL string where the
session was expected, raising `AttributeError: 'str' object has no attribute
'query'` whenever a Video metadata refresh ran.
Swap the arguments at both call sites and add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New Export tab with two client-side formats (no new dependencies):
- CSV: the raw items table (RFC-4180 quoting, UTF-8 BOM for Excel), generated
entirely in the browser. The download anchor is attached to the DOM and the
object URL is revoked on the next tick for cross-browser robustness.
- PDF: a printable full-inventory table followed by a group-by summary table,
via the browser's print dialog ("Save as PDF"). The summary's grouping is
chosen in the Export tab.
Search filter: a box on its own line under the inventory selector filters the
active inventory across every column's displayed value (case-insensitive;
whitespace-separated terms are AND-ed). It narrows the whole view — Items
table, Summary aggregation, and Export/PDF — while edits/adds in the table
still operate on the full inventory, so filtering never drops data.
Summary/export grouping:
- The group-by aggregation is extracted into a shared summarize.js (field-role
detection + count-aware totals via units.js + calories), now the single
source of truth for the Summary tab, the PDF summary, and the ration
calculator. findCaloriesKey/findCountKey moved here from RationCalculator
(re-exported for back-compat).
- The Sum selector now offers number and calories fields in addition to
quantity: number/calories are summed as a plain column total, while quantity
stays count-aware and unit-compacted and the dedicated Calories column stays
count-aware.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Portrait mobile shows a condensed, read-only table; landscape/tablet+ keeps
the full editable spreadsheet. The columns shown on mobile are configurable
per field via a new "Mobile" toggle in the Customize Fields editor.
- defaults.py: seed `mobile` on each type's default columns (food: Name,
Subcategory, Size, Count; fuel/tool: sensible analogs).
- FieldSchemaEditor: per-field Mobile toggle (persists through PUT /<slug>/fields).
- InventoryItemsMobile: read-only condensed view driven by the `mobile` flag,
falling back to a default column set for schemas predating the flag.
- InventoryRoute: split Items tab via <Media> — mobile vs tablet+.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the DB-backed inventory module with a config-only design and build out
food-storage tooling around it.
Inventory (config-only, no database)
- Each inventory is a YAML file at config/inventory/<slug>.yaml, loaded once
into memory and the single source of truth. New reusable MultiFileConfig
(wrolpi/common.py) manages a directory of per-entity files.
- Inventory types (food/fuel/tool) seed a customizable field schema
(text/number/quantity/date/select/location/calories). Unit math/aggregation
moved to the frontend (mathjs); Pint dropped from the backend.
- Minimal API: GET / (all inventories full), POST /, PUT /<slug> (whole-
inventory save with version-based optimistic concurrency; 404 for missing),
DELETE. Legacy inventories.yaml / old tables migrate to per-inventory files
(pound->lb), and an alembic migration drops the tables.
Frontend
- New app/src/components/inventory/ package: spreadsheet-style table with
Tab/Enter keyboard entry, sortable columns, expired-row highlighting,
customizable field editor, and pooled cross-inventory location autocomplete.
- A single useInventories store loads everything once and persists whole
inventories optimistically; list/items/summary/locations derive client-side.
- Summary groups items (default by Category), totals size x count, shows a
calories column, and embeds a ration estimate (how long the stock lasts).
Food catalog
- Shared, editable catalog (config/inventory/catalog.yaml) seeded from
catalog_defaults.yaml (~60 generic items, total calories per package) with
merge-keep-edits + tombstones. GET/PUT /api/inventory/catalog, a management
modal, and entry-row pre-fill of category/size/calories from a name match.
Calculators
- Generic Food Storage planner (like the Water calculator): household +
editable per-person/year amounts -> total to store over a chosen duration,
with women/children weighted by DGA calorie shares and an imperial/metric
toggle.
Also fixes ConfigFile.save() to accept nested file_names. Backend pytest and
frontend jest suites updated/added throughout; alembic check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Move One-Time Pad encryption to the browser; delete backend OTP module
OTP encryption/decryption is now performed entirely in the browser so
messages, keys, and generated pads never leave the originating device.
- Add app/src/components/otp.js: a JS port of the former modules/otp/lib.py
(encrypt/decrypt, message formatting, validation, and secure pad
generation via crypto.getRandomValues with rejection sampling).
- Encrypt/Decrypt now run live as the user types or pastes, with inline
error messages instead of submit buttons.
- Generate New Pad builds the printable HTML locally (blob URL) instead of
hitting /api/otp/html.
- Migrate all backend OTP tests to app/src/components/otp.test.js with
identical data.
- Delete the modules/otp backend module and its blueprint registration.
* Revoke the One-Time Pad blob URL after the new tab loads
Addresses a minor blob-handle leak: handleGenerateNewPad created an object
URL on every click but never released it.
* Let users choose the One-Time Pad character set
Add an "Advanced" accordion under the buttons with three options: A-Z,
A-Z/0-9 (default, unchanged), and a custom character set. The chosen set is
shared by encrypt, decrypt, and pad generation.
- otp.js: parameterize encrypt/decrypt/generate on the alphabet (default
OTP_CHARS for back-compat); add OTP_CHARS_ALPHA and validateCharset; input
is auto-uppercased only when the set is all-uppercase, so custom sets with
lowercase are case-sensitive; the printable pad is labeled with its set.
- Apps.js: lift charset state into OTP(); gate encrypt/decrypt/generate on a
valid set with inline validation errors.
- otp.test.js: add A-Z, custom round-trip, auto-uppercase, validateCharset,
and pad-label tests.
* Scope inverted-accordion dark background to direct children
The rule forcing #171616 on .ui.inverted.accordion .title/.content used a
descendant selector, so it leaked onto any nested element using the `content`
class — e.g. an ErrorMessage's Message.Content rendered inside SimpleAccordion
got a dark band behind its text. Scope to direct children so the accordion
still styles its own title/content but not nested components.
* Add optional error-detection checksum to the One-Time Pad
Adapts the position-weighted checksum from otp-checksum.md to the
frontend-only OTP. checksum = sum(position * value) mod N, parameterized on
the selected character set (N = chars.length), so it follows whichever pad
alphabet is chosen. Position weighting catches transposition errors.
- otp.js: add calculateChecksum/appendChecksum/verifyChecksum.
- Apps.js: opt-in "Append error-detection checksum" toggle in Advanced
(default off). When on, encrypt appends a checksum char; decrypt verifies
it, warns on mismatch, and still shows the decrypted text.
- otp.test.js: checksum unit tests, charset-variation, and end-to-end vectors.
* Show compact checksum status instead of a large warning box
Replace the bulky semantic-ui WarningMessage on decrypt with a small inline
status line: a green check + "Checksum is valid" or a red x + "Checksum is
invalid", shown whenever the checksum option is on and a message is being
decrypted.
* Fix checksum-mode decrypt truncating messages without a checksum
When the "Append error-detection checksum" option was on, Decrypt always
stripped the last ciphertext character before decrypting, so a message that
carried no trailing checksum (wrong sender setting, body pasted alone, etc.)
lost its final character and decrypted to garbage.
Add stripChecksum(received, chars), which removes the trailing character only
when it actually verifies as the checksum of the preceding characters; a
message without a valid checksum is returned intact and flagged invalid. The
decrypt UI now uses it.
Also broaden test coverage:
- stripChecksum tests, including a regression for the truncation bug.
- validation error classes (InvalidOTP/Plaintext/Ciphertext) and the
longer-than-OTP length guards.
- deterministic encrypt/decrypt roundtrip fuzz across several character sets.
- generateMessage character-count and charset-membership checks.
- end-to-end checksummed encrypt -> format -> strip -> decrypt flow.
Use a dedicated InvalidMessage error for the checksum helpers instead of the
misleading InvalidPlaintext.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The archive modeler claims WROLPi's own singlefiles, but a foreign
compressed (SingleFileZ) HTML file (e.g. saved by the browser extension and
dropped into the media directory) falls through to the generic HTML
indexer, which crashed reading the binary ZIP tail as text. The indexer
now reads bytes and indexes the index.html inside the ZIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.
single-file-cli is designed for Deno. Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.
The native path and the Archive docker container now run single-file with
"deno run". Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum). Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.
Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind. The Archive
downloader now has a ten minute timeout. Install scripts install Deno
(previously only upgrade.sh and the image builds did).
* Singlefile writes to a file instead of dumping to stdout.
Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives). single-file now writes its result to a
temporary file which WROLPi reads back.
As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.
* Compressed Archive screenshots and readability images are no longer broken.
The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths. Screenshots are now taken from the fully
extracted ZIP so those references resolve. Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).
* Compressed Archives are created by converting the uncompressed capture.
The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed). When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file. The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages. If the
conversion fails, the uncompressed singlefile is stored instead.
* Consolidating duplicate compressed-singlefile tests.
The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one. The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.
* Addressing review comments: removeprefix, PermissionError when killing.
* Bumping version to 0.24.1-beta.
* Install scripts assert the deno.zip hash to avoid malicious or invalid files.
The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.
* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Old single-file versions (1.x) create compressed Archives without the
SingleFile comment header in the self-extracting prelude, which WROLPi
rejects as invalid. The install scripts only install single-file when it
is missing, so existing machines were never upgraded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Archives can be compressed (SingleFileZ self-extracting HTML).
A new compress_singlefile download setting runs single-file with
--compress-content, creating a self-extracting HTML file which is also a
valid ZIP. Only the singlefile .html is compressed; readability files are
always written uncompressed. Readability, screenshots, titles, and URLs
are extracted from the index.html inside the ZIP. A Compress checkbox in
the Archive and RSS download forms remembers the user's choice in local
storage.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Playlists: ordered, mixed-type collections
A playlist (Collection kind='playlist') is a manually-curated, ordered list that
mixes item types. Squash of the playlists feature branch (24 commits).
Data model & API:
- CollectionItem is polymorphic via item_kind: 'file' (FileGroup), 'zim'
(zim_id + zim_entry), or 'url' (link to anything the WROLPi browser opens,
e.g. /map?lat=&lon=&z=). CHECK constraint enforces exactly one kind; per-kind
unique constraints; migration a7b1c2d3e4f5.
- Endpoints: create playlist; add/remove/reorder items (idempotent adds: 201 new,
200 existing); GET returns the ordered items. All mutations wrol_mode_check'd.
Config (source of truth):
- playlists.yaml (PlaylistsConfig) stores each playlist's metadata AND ordered
items (file by relative path, zim by path+entry, url inline); round-trips a
DB wipe. Supports backup import (DB_CONFIG_FILE_NAMES).
- The redundant generic CollectionsConfig (collections.yaml) is deleted:
channels/domains own their configs, authors/subjects are re-derived.
On-disk sync (sync_playlists_directory switch):
- Each playlist materializes into the configurable Playlists Directory
(playlists_destination, Settings UI input, excluded from indexing) with
position-prefixed entries: NNNN_<file> hard links (copy fallback across
filesystems) and NNNN_<title>.html stubs for zim/url items.
- Tagged playlists are namespaced like channels: <playlists>/<tag>/<name>;
a tag change moves the directory and prunes the empty old tag dir.
- Deletion safety: managed files are only removed when they are our marked
stubs or still have another hard link (st_nlink > 1) -- never a last copy.
Stray loose files at the playlists root are removed; stub URLs are
scheme-allowlisted (no javascript:/data:).
Frontend:
- /playlists list page (CollectionTable, name filter, New Playlist modal, Tag
column) between Files and Zim in the nav.
- View-only detail page (Name link) and edit page (Edit button): themed table
with model icons/colors + poster thumbnails, items link to their model page
(video/archive/doc) or preview, up/down reorder, remove, add-URL form,
single-tag selector; all editing disabled in WROL Mode.
- Reusable green "Add to Playlist" button on Video/Archive/Doc detail pages,
the FilePreview toolbar, Zim search results, and Map pins (added as a /map
URL item with a map-marker icon).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Playlist edit page parity: rename, custom directory, tag modal move suggestion
Match the ChannelEditPage/DomainEditPage UX on the playlist edit page:
Back/View buttons, an edit form (name, description, directory), and the
shared CollectionTagModal with the violet "Tag" button. The tag modal now
suggests moving the playlist to its managed tag location
(playlists/<tag>/<name>); accepting the suggestion stores directory=None so
the sync keeps auto-managing the location. Tag buttons on the Channel and
Domain edit pages are now violet to match.
Backend: update_collection learns rename (with per-kind validation and
duplicate checks) and playlist custom directories (cleanup of the abandoned
directory, config round-trip, commit-before-switch to avoid the background
sync racing pre-commit state). tag_collection's duplicated untag/tag
branches are merged, and the per-kind config-save dispatch is consolidated
into _activate_config_save().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate Collection error handling and Tag creation
Delete the duplicate UnknownCollection in wrolpi/collections/errors.py and
re-export the canonical one from wrolpi/errors.py, which carries
status_code=NOT_FOUND (the local copy inherited 500). Replace the repeated
query/raise pattern in reorganize.py with Collection.find_by_id.
Add Tag.get_or_create_tag and use it for collection tagging; flush the new
Tag before returning so callers can read .id immediately -- previously
tagging a collection with a brand-new tag name set tag_id=None. Regression
test included.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Persist finalize_download mutations; isolate child-download enqueue
A channel/RSS Download could get stuck permanently "pending". Root cause:
the phase-split dispatch in signal_download_download passes the Download
loaded in an already-closed session to finalize_download, so it is
DETACHED. ChannelDownloader.finalize_download sets
`download.sub_downloader = 'video'` (and info_json/collection_id) on that
detached object, so the write never reaches the DB. The status block then
re-queries the Download fresh, reads sub_downloader=None, and calls
create_downloads(downloader_name=None) ->
`InvalidDownload: Cannot find downloader with name None`.
That exception was raised inside the status-commit transaction (outside the
inner try), so it (a) rolled back the parent's complete(), reverting it to
the `pending` set by started(), and (b) was swallowed by the outer bare
`except Exception`, which only logs. Result: the parent never failed, never
completed — it sat `pending` forever, re-crashing the same way on every
restart.
Fixes:
- finalize_download is now given a Download re-loaded in (attached to) its
own session, so its mutations persist.
- Child-download creation moves to its own transaction after the parent's
status is committed. A failure there is logged and cannot roll back or
strand the parent.
The existing test harness reuses one shared DB session for every
get_db_session call, so detached-object and rollback bugs never reproduced
under it (which is why this shipped). Add `production_like_sessions` to
conftest.py — a context manager handing out a fresh Session per call, like
production — and two regression tests that fail without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Re-attach Download in prepare_download too (consistency)
prepare_download receives the same detached Download as finalize_download
did, so VideoDownloader.prepare_download's writes to the row
(destination/tag_names) are not reliably persisted. Re-load the Download
attached to the prepare session, mirroring the finalize fix, so the row's
metadata is accurate.
Unlike sub_downloader (which the status block re-reads and feeds to
create_downloads), nothing downstream re-reads these values, so this is a
consistency/cosmetic improvement, not a functional bug — and it cannot be
made to fail in a test (the loaded Download's session outlives the worker),
so no dedicated test is added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Log (not swallow) session-close failures in production_like_sessions
Addresses Greptile review: surface a failed session.close() at debug level
instead of silently passing. engine.dispose() still does the authoritative
cleanup, so the close failure remains non-fatal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A failing SMART self-assessment predicts imminent drive failure, so it
deserves a prominent warning. useDriveHealth() reads the assessment
already present in smart_stats (pySMART reports PASS / FAIL / null) and
flags any drive whose assessment is FAIL. Nav.js renders a red warning
icon in its own always-on slot — like the power warning, and unlike the
single-slot priority chain where a transient high temp could hide it.
Frontend-only: the assessment field is already in the cached payload.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Warn in the navbar when a hard drive runs hot.
The navbar already warns about high CPU temperature; add the equivalent
for drive temperature, since an overheating disk risks data loss and a
user running an off-grid library won't be watching for it.
- controller/lib/smart.py: build_smart_stats() returns the per-drive list
plus warning thresholds (55C warn / 65C critical) so the frontend stays
dumb. Reading SMART wakes a spun-down drive, so drives in standby are
detected with `smartctl -n standby` (which works through the USB-SATA
bridges where `hdparm -C` only reports "unknown") and NOT read; their
last-known reading is reused instead.
- controller/lib/status_worker.py: collect SMART on a slow 60s cadence
(cached on app.state) rather than the 5s status loop, so the poll does
not keep USB drives awake. The result flows into the same cached payload
via a new collect_all_status(smart_stats=...) kwarg.
- Frontend: useDriveTemperature() reports the hottest drive; Nav.js shows
a yellow/red `hdd` icon with a popup, mirroring the CPU warning.
Verified on hardware: smart_stats reports real temps and SMART runs once
a minute, not every 5s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Greptile review on HDD temperature warning.
- _drive_is_asleep also treats SLEEP mode as asleep, not just STANDBY.
smartctl -n standby exits "Device is in SLEEP mode, exit(2)" for the
deeper power mode; matching only "STANDBY" meant a SLEEP drive was read
and woken — the exact failure the check prevents. (TDD: added a failing
SLEEP test, then fixed.)
- useDriveTemperature uses ?? instead of || for the temperature fallback
to express "fall back only on null/undefined".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applied directly since the dependabot PR could not auto-rebase after the
grouped merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alembic 1.18.4 requires SQLAlchemy>=1.4.23, but the project pins
SQLAlchemy==1.3.22, making the dependency set unresolvable. This slipped
through because the backend pytest suite is not a required CI check
(only controller-tests and jest-tests are).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
