git rev-parse --abbrev-ref HEAD reports "heads/release" when a stale
tag named release exists alongside the release branch (tags outrank
heads in git ref resolution). check_for_update then looked up the
nonexistent origin/heads/release and permanently reported no updates
available, and the Controller UI pre-filled "heads/release" as the
upgrade branch.
Use git branch --show-current everywhere instead; it never
disambiguates. A detached HEAD (empty output) now defaults to the
release branch for upgrades; a genuine git failure still returns None.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add daily_limit_reached to the DownloadManager summary (/api/status):
True when due (new, non-recurring) downloads exist and every one of
them is blocked by the global or per-domain daily download limit.
Show a "Max Daily Downloads Reached" banner on the Downloads page when
set (and the manager is neither disabled nor stopped).
Previously a queue stalled by the daily limit looked broken: the UI
reported 0 pending with no explanation, and the skip reason was only
visible at DEBUG log level.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses two processing_domains leaks raised in review; each could permanently
exclude a domain and eventually stall the scheduler until restart.
1. Claim-transaction failure (Greptile P1): `_add_processing_domain` mutated shared,
non-transactional state before the claim transaction committed. A commit failure
skipped dispatch but left the domain claimed forever. Fix: defer both the
processing_domain add AND the signal dispatch to after the transaction commits;
track claimed domains in a local `claimed` list so the loop's own dedup /
concurrency cap is unchanged.
2. Row deleted before dispatch (Greptile P1): if a claimed download was deleted
between commit and the signal running, `Download.find_by_id` raised before
`download_domain` was set, so the handler's `finally` released nothing. Fix:
derive `download_domain` from the URL (`urlparse(url).netloc`, the same value the
dispatcher claimed) up front, so cleanup always runs.
Regression tests written bug-first (both fail before this change):
- test_signal_releases_processing_domain_when_download_missing
- test_dispatch_releases_claimed_domains_when_transaction_fails
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_dispatch_new_downloads` claimed downloads inside a `get_immediate_db_session()`
(BEGIN IMMEDIATE) transaction and then `await`ed `api_app.dispatch(...)` while that
write transaction was still open. Under NullPool (production, one connection per
session) the dispatched `signal_download_download` opens its OWN connection and —
inheriting the thread-local `_immediate_txn` flag — also issues BEGIN IMMEDIATE,
which blocks on the write lock the outer transaction still holds until busy_timeout
(30s) expires: `(sqlite3.OperationalError) database is locked [SQL: BEGIN IMMEDIATE]`.
Every dispatch cycle died on the first download, so nothing downloaded at all.
Fix: claim the downloads inside the transaction, collect (id, url), and dispatch the
signals AFTER the `with` block commits and releases the write lock.
Diagnosed live on 10.0.0.9 (downloads had silently stopped); with this change the
14 stuck recurring downloads drained immediately and no lock errors recurred.
Why tests missed it: the `test_session` fixture returns a single shared
(engine, session) for every `_get_db_session` call, so all sessions in a test ride
one connection -- the second-connection write-lock contention that causes the
deadlock cannot occur. Added a regression test asserting the invariant instead: the
download signal must be dispatched with no immediate transaction active.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigating downloads stalling on 10.0.0.9 (recurring downloads going overdue
without dispatching). The dispatch path had blind spots that hid the cause, so add
targeted DEBUG logging, change-detected where the cadence would otherwise flood:
- blocked_download_reason(): single source of truth for can_download that returns
*why* downloading is blocked. dispatch_downloads() logs the reason (and the
unblock transition) only when it changes, instead of silently returning every 5s.
- is_within_download_window(): logs the OPEN/CLOSED transition with the computed
local time, window, and timezone — surfaces any clock/timezone mismatch (the box
shows system UTC-5 vs configured America/Denver) at the moment of evaluation.
- _dispatch_new_downloads(): per-cycle snapshot (new count, processing_domains,
daily global_count, download_wait) plus a bounded tally of why due downloads were
NOT dispatched. Prime suspect is a leaked processing_domains slot (a dispatch that
dies without cleanup blocks that domain, and once 4 leak, everything) — that path
previously had no logging.
Behavior-preserving: can_download semantics are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes from PR review:
- `PRAGMA journal_mode=WAL` returns the mode SQLite actually selected and can leave the
prior rollback journal without raising. Confirm the returned mode instead of assuming
the request succeeded, so a silent WAL refusal correctly falls back to TRUNCATE +
synchronous=FULL rather than caching WAL with the weaker synchronous=NORMAL.
- `fuseblk` (block-backed FUSE such as ntfs-3g) has unreliable byte-range locking that
can corrupt the database under the workers' concurrent access. Move it out of the
WAL-incompatible allow-list into an UNSAFE_LOCKING_FS_TYPES refuse-list (renamed from
NETWORK_FS_TYPES). NTFS on a modern kernel mounts via the native `ntfs3` driver, which
stays allowed; exFAT is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media drive formatted for Windows compatibility (exFAT/FAT/NTFS) cannot support
SQLite WAL: it needs mmap'd shared-memory those filesystems don't provide, so
`PRAGMA journal_mode=WAL` raised `(sqlite3.OperationalError) disk I/O error` on
every connection, taking down the whole API (observed on 10.0.0.8; the tell was
`Media directory has the wrong permissions: 0o40777`, since these filesystems can't
store Unix permission bits).
The engine now tries WAL first and, on failure, falls back to a TRUNCATE rollback
journal with synchronous=FULL (crash-safe, since WROLPi is off-grid). The decision
is memoized per-engine so WAL isn't re-attempted or re-logged on every fresh
NullPool connection. The bootstrap environment check no longer refuses these
drives; it warns once that write-concurrency is reduced and recommends ext4.
Network filesystems remain refused: their locking is unsafe regardless of journal mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The download dispatcher (_dispatch_new_downloads) reads the `new` downloads
then claims each by writing last_download_attempt, all in one ORM
transaction. Since ORM transactions begin *deferred*, that write is a lock
upgrade (SHARED->RESERVED). SQLite skips busy_timeout for lock upgrades to
avoid deadlock, so under concurrency the write failed instantly with
"database is locked" instead of waiting out the 30s busy_timeout. This
surfaced repeatedly on the SQLite build after the Postgres->SQLite cutover.
Add get_immediate_db_session(), a committing session that takes the write
lock up front (BEGIN IMMEDIATE) so busy_timeout absorbs the contention --
mirroring what the raw-cursor get_db_curs path already does. Plain
get_db_session stays deferred so ordinary writes and reads keep WAL
read-concurrency; only read-then-write call sites opt in. Convert the
dispatcher to use it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triage of Greptile findings on the default-Zim / configurable-directory PR:
- Stale config on restart (P1): settings update() only queues an async
background_save, and the restart switch has no ordering guarantee (the switch
worker pops LIFO), so start_kiwix_serve.sh could read the old zims_destination.
Save the config synchronously before activating the restart switch.
- YAML null (P1): read_config_value.sh now treats an unquoted null/~ as unset
(returns the default), matching PyYAML/the Python config layer. A quoted
"null" remains the literal string.
- Path traversal (P1/security): the settings API now rejects a zims_destination
containing "..", and start_kiwix_serve.sh defends in depth by falling back to
`zims` for any absolute/traversal/empty value, so Kiwix can never scan or
rewrite library.xml outside the media root.
- Reproducible build (P2): pin the zim-tools image to 3.7.0 instead of :latest.
Tests added for null handling, traversal rejection (API + script).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related gaps in how WROLPi serves Kiwix content:
1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
Kiwix. Every image now ships a small built-in default Zim that Kiwix falls
back to, telling the user to download real Zim files.
2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
`zims_destination` (which the Python side already honors).
Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
non-Python scripts can reliably read top-level wrolpi.yaml scalars. Also a
config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
(ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback. It needs no
config read: it already scans the whole mounted media dir, and
zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
serves the new directory without a reboot.
Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When downloading is enabled at startup, the download worker races a burst
of concurrent SQLite writes on the single DB file: retry_downloads() has
just marked every download 'new', RSS catalog refreshes call
create_downloads() (which fires its own background dispatch_downloads()),
multiple download workers claim downloads, and save_downloads_config
re-saves the config. dispatch_downloads reads-then-commits in a deferred
transaction, so when another writer commits mid-cycle it fails with
SQLITE_BUSY_SNAPSHOT ("database is locked", not absorbed by busy_timeout).
This is self-healing -- the perpetual worker retries on its next cycle and
downloads succeed -- but the raised OperationalError flooded the logs with
full tracebacks and tripped the Sanic traceback renderer (tracerite) into
its "Variable inspector failed" secondary errors.
Split the claim-and-dispatch DB work into _dispatch_new_downloads() and
have dispatch_downloads() swallow a transient lock (debug log, return) so
the next cycle retries cleanly. Non-lock database errors still propagate.
The split keeps the retry decision outside the transaction that also
awaits the download signal dispatches, so a lock is never handled by
re-dispatching (no double-dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refreshing all files failed on a large media directory with
RuntimeError: values_clause: too many parameters (44454 rows x 3).
After the Postgres->SQLite cutover, values_clause guards against
exceeding SQLite's ~32k bound-variable limit, but upsert_directories
still built a single multi-row VALUES clause over every discovered
directory.
Insert in BULK_SQL_CHUNK_SIZE batches, matching the existing bulk
tag-file callers. Adds a regression test that inserts more directories
than a single statement can bind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When importing hundreds of tag_files for files that exist on disk but
have no FileGroup yet (a fresh import before a file refresh), the import
crawled at ~a dozen rows/second. Every tagged file hit a fallback that
globbed its parent directory (O(files²) for large Channel/Domain dirs)
and flushed the DB once per row.
Batch the work instead:
- Cache each directory's listing so it is scanned once, not per file.
- Create new FileGroups under no_autoflush and flush them in a single
batch, then create TagFiles in a second pass (one batched INSERT).
- Cache every member path of a newly-created FileGroup so a file tagged
by multiple tags, or another config entry for a different member of
the same shared-stem group, reuses one FileGroup instead of inserting
a duplicate primary_path that would abort the import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_special_directories runs during every refresh ignore filter. Calling
get_videos_directory/get_no_channel_directory created videos/ and
NO CHANNEL under an empty media tree, which broke empty-refresh and
directory listing tests. Resolve those paths without creating them.
Zim Manage and search depend on FileGroup modeling of files under
zims/. Ignoring that directory left Kiwix serving files on disk while
/api/zim stayed empty. Refuse to ignore zims (alongside other special
dirs), strip it from existing configs on settings load, and share
normalized ignore handling between refresh and upsert.
`git rev-parse --abbrev-ref HEAD` prints the literal "HEAD" on a detached
HEAD, which get_current_branch() returned as if it were a branch. check_for_update
then compared origin/HEAD (the remote's default branch, usually master) and
reported a bogus "N commits behind on HEAD" — a false "update available".
Return None for a detached HEAD so no phantom upgrade is offered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the four test_list_tar_*_contents and four
test_download_single_member_tar* functions into two parametrized tests
covering gz/xz/plain/bz2. All eight format cases still run; no coverage
loss. Identified via pytest-deduplicate (identical line coverage).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
- strip_youtube_caption_positioning now writes to a temp file in the same
directory and os.replace()s the original, so a failed/partial write
(e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
which parses the hostname and matches youtube.com / youtu.be (and
subdomains), rejecting lookalikes like youtube.com.example.org and
notyoutube.com. Add a parametrized test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (via yt-dlp) stamp every cue timing line
with `align:start position:0%`, pinning the on-screen text to the
bottom-left of the player. This strips that exact cue setting from the
downloaded .vtt files so browsers render the captions centered.
- add strip_youtube_caption_positioning() to wrolpi/captions.py — a
targeted text replacement (not a webvtt round-trip) so the `<c>`
word-timing tags in the cue text are preserved; matches only the exact
auto-caption signature, so deliberately positioned cues (e.g. with a
`line` setting) are left untouched; idempotent.
- call it from VideoDownloader after normalize_video_file_names, gated on
a youtube.com / youtu.be source URL. It runs before the FileGroup is
created, so the first index reads the corrected file — no re-index
plumbing needed.
Complements the client-side fix in #543 (which centers captions in the
player for existing/any-source files); this makes new YouTube downloads
correct at the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
go-pmtiles release assets are named ..._Linux_x86_64.tar.gz, not _amd64;
the #538 consolidation used $arch=amd64 in the URL, so every amd64 build
(Debian Live ISO, docker) 404'd installing pmtiles and the wrolpi chroot
hook failed. Use x86_64 for the URL arch on x86_64 hosts; the SHA var and
the aarch64 (arm64) path are unchanged and already correct.
Verified: both v1.30.1 tarballs download and match the pinned SHAs
(x86_64 23a2a222..., arm64 a1f9f42d...).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.
- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
arch-independent SHA). Version-guarded, so an image that already baked in the
pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
call the shared script. This also adds SHA pinning to the on-device pmtiles
path (previously unpinned).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9. Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).
Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply_indexers held one ORM session across a whole batch of indexers --
minutes of readability/subprocess work -- so its final commit had to
upgrade a stale read snapshot to a write. SQLite answers that with
"database is locked" (SQLITE_BUSY_SNAPSHOT, which busy_timeout does not
apply to) whenever any other connection commits a write in the meantime,
which killed the refresh on 10.0.0.9 at 75%.
Read the batch as plain columns, index on transient FileGroups with no
open transaction, then write the results in one short BEGIN IMMEDIATE
transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three build targets move from Debian 12/bookworm to Debian 13/trixie
for fresh installs; existing Debian 12 systems remain supported in place.
Docker: api/help on python:3.13-trixie, controller on 3.13-slim-trixie,
app on node:20-trixie (matches trixie's nodejs 20), samba/zim on
debian:13. Full backend suite passes on Python 3.13 in the api container.
debian-live-config: --distribution trixie; postgres teardown for the
SQLite cutover (pg_dropcluster hook block removed, wrolpi-bootstrap.sh no
longer creates a cluster or calls the deleted initialize_api_db.sh — the
API creates wrolpi.db itself on first start). README documents the
SQLite layout and drops Etcher recommendations.
pi-gen: upstream has no trixie branch — the arm64 branch now defaults to
RELEASE=trixie; KEYBOARD_KEYMAP=en_US is rejected by trixie's
console-setup (ckbcomp validates), use 'us'; stop stamping a Postgres
.pgpass into /etc/skel.
Package lists (both targets): postgresql-15/postgresql-doc/libpq-dev
deleted; libtiff5-dev renamed to libtiff-dev; software-properties-common
and cpufrequtils removed from trixie (Controller CPU throttle falls back
to sysfs, dead cpufreq-set sudoers rule removed); locales and gnupg added
(no longer arrive transitively under --apt-recommends false); sqlite3 CLI
added for reset_tags.sh.
help.sh: replace the Postgres service/port/socket/cluster/psql checks
(all false FAILEDs after the SQLite cutover) with SQLite checks:
wrolpi.db exists, PRAGMA quick_check, file_group table + row count, run
as the wrolpi user so root never creates the -shm sidecar.
Both build scripts take a WROLPI_BUILD_DIR override so development builds
can run beside the release pipeline's build trees; the sudo re-exec
passes it through explicitly (a plain sudo would drop it).
test_common.py: the truncate tests hard-coded sys.getsizeof results from
the 3.11 object layout (str/list shrank in 3.12); assert the contract
(fits budget, prefix, monotonic in budget) so they pass on 3.11 and 3.13.
Verified: trixie ISO (3.7G) and pi-gen desktop image (4.3G) both built
from scratch and passed their sanity checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collection.batch_from_config OR'd a (name AND kind) pair per collection, so
importing more than ~1000 collections (e.g. a 1,670-domain domains.yaml)
failed with "Expression tree is too large (maximum depth 1000)". Query a
flat name IN (...) and match (name, kind) in Python instead.
get_bulk_tag_preview had the same unbounded pattern (an OR'd LIKE per stem);
chunk the query at 500 stems.
Found upgrading 10.0.0.9 (decay).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.
Unlike the old implementation, the sort degrades gracefully:
- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
download fails cleanly with UnrecoverableDownloadError telling the
user to update yt-dlp or switch to Newest/Oldest — no retry loop.
Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries. Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.
Database:
- Migration adds the generated column and a GIN index (rewrites the
table; a few minutes on large libraries).
Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
query (COUNT FILTER on the abc subset): `file_groups` (fast) and
`file_groups_deep`.
UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
list.
- The gap is reconciled by a hint offering deep search: count-aware on
the global search ("No results in fast search. N results available in
deep search." / "N more results available..."), generic on the module
pages (no deep estimate there). The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
?deep=true.
Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside. Also skip flaky test_download_channel on
CircleCI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Browser-based ESP32/ESP8266 flasher (esptool-js + Web Serial); nothing is
uploaded to WROLPi, the flash is written straight to the device over USB.
Firmware sources (themed, tabbed): a saved set, the WROLPi media directory,
or a local file. Local files can be dropped anywhere on the page, which jumps
to the "Add from computer" dropzone. The media picker searches .bin files by
path and can detect the connected chip to filter firmware to matching images.
- Indexed file_group.suffix + path filter for fast .bin search (alembic
migration backfills via split_path_stem_and_suffix)
- modules/flasher: chip detection from ESP image headers, /search endpoint,
and flasher.yaml saved configurations (/configs CRUD)
- Flash progress modal with ETA, connect timeout, boot-mode hint, auto-
scrolling logs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube's new listing layout means yt-dlp no longer returns `view_count`
in flat channel/playlist entries, so sorting by views raised KeyError and
the download failed and retried forever.
Sorting by views now requires per-video extraction (hundreds of requests
per channel), so the feature is deprecated rather than degraded:
- downloader: the 'views' branch raises UnrecoverableDownloadError so the
download fails cleanly (no retry loop) with a message to switch to
Newest/Oldest. May be revisited via the YouTube Data API.
- schema: download_order validation drops 'views' (newest/oldest/null only).
- UI: removed "Most Views" from the download-order dropdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings.js: send 0 (not null) when a daily-limit field is cleared, so the
settings API actually clears a previously-saved limit instead of dropping the
null and keeping the old value.
- downloader.py: document that the daily limit is a best-effort soft cap; minor
bounded overshoot is possible when create_downloads' background dispatch overlaps
the perpetual cycle, and self-corrects the next cycle. A strict cross-process cap
is intentionally not implemented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two optional DownloadManager settings:
- download_daily_limit_per_domain: max non-recurring downloads per domain per day
- download_daily_limit_global: max non-recurring downloads across all domains per day
Recurring Channel/RSS catalog updates are never gated and never counted; only the
videos/archives they spawn and one-off downloads count. Both successful and failed
attempts consume a slot (counted once per download per day). Limits reset at local
midnight (configured timezone). Over-limit downloads stay "new" and resume the next day.
Implementation:
- New Download.last_download_attempt column (+ index, migration) set at dispatch time so
the count survives restarts and is correct across workers.
- dispatch_downloads gates non-recurring downloads: per-domain skips that domain, global
stops dispatching counted downloads for the cycle. daily_download_counts derives the
day's totals from the DB.
- Domain grouping uses normalized URLs via a new Downloader.normalize_domain method
(base strips www/m/mobile + port); VideoDownloader/ChannelDownloader override it to
collapse youtu.be, /shorts/, and tracking params to youtube.com. ArchiveDownloader is
the seam for future archive URL normalization.
- Settings wired through WROLPiConfig, the /settings API (GET/PATCH, non-negative
validation, 0 = unlimited), and the React Settings page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix FileWorker status stranded mid-phase (e.g. stuck on "Cleanup")
The FileWorker stores its status in shared context and relies on an explicit
reset_status() in each handler. Two execution paths set the status (comparing/
upserting/deleting/modeling/indexing/cleanup) but could finish without ever
resetting it, freezing the UI at the last phase:
- refresh_sync() ran _apply_post_processing (final phase sets status='cleanup')
but never called reset_status(). Deleting files in the UI goes through
lib.delete -> refresh_sync, so any delete that changed the DB stranded the
worker at 'cleanup' (observed on 10.0.0.8 after deleting STL files).
- process_queue() wrapped the handler dispatch in try/finally with no except.
An exception in handle_count/handle_refresh (which, unlike handle_move/
handle_reorganize, have no try/except) escaped to the perpetual worker loop
and left the status frozen at whatever phase was running ("any state").
Fixes:
- refresh_sync(): reset_status() in a finally.
- process_queue(): on exception record a terminal 'error' status (so the worker
is not stranded mid-phase) and re-raise so callers and the perpetual loop
still observe the failure. This covers both handle_count and handle_refresh.
Tests: add replication tests for the synchronous-refresh, handler-error, and
count-error stranding paths, plus the previously-untested wait_for_job timeout.
* refresh_sync: leave terminal 'error' status on failure
Address Greptile review: refresh_sync reset to idle in a finally that also ran
on failure, so a synchronous refresh that raised after a phase was set reported
a clean 'idle' to the status endpoint instead of the failure (the exception was
still raised to the caller). Make it symmetric with process_queue: reset to idle
only on success, record status='error' and re-raise on failure.
Add test_refresh_sync_records_error_status_on_failure.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>