* test: the component suite has not run since cypress 15, on any machine
Six component spec files -- 334 tests, including the 4,300-line layout suite -- could not
start. Cypress exited before printing anything, and under its default Electron browser it
did not even do that: it hung at 0% CPU until killed. Chrome prints the error in seconds,
which is the only reason this was findable.
Three things broke, all of them from the 13.17.0 -> 15.20.0 bump.
`baseUrl: "src"`. Cypress 15 loads its config with tsx, and tsx honours tsconfig path
resolution. Cypress requires Node's core `events` module; with that baseUrl the bare
specifier resolves against `src`, and on a case-insensitive filesystem it matches
`src/Events.js`. That file imports the app, so the whole component tree was required in
Node: JSX in `.js` files, which esbuild will not parse.
This is why CI stayed green while every local run died. Linux is case-sensitive, `events`
never matched `Events.js`, and it resolved to the built-in. The baseUrl buys nothing --
there is not one absolute import in the codebase -- so it goes.
`framework: "create-react-app"`. Cypress 14 removed that dev server; the handler is in the
13.17.0 binary and absent from 15. The webpack config is handed over explicitly now, taken
from react-scripts so components compile exactly as `npm start` compiles them. Two things
follow from that: `webpack-dev-server` 5 is a devDependency because Cypress requires 5 and
react-scripts pins 4 (npm nests react-scripts' copy, and the production build is unchanged),
and babel-loader is widened to `cypress/`, which react-scripts does not compile -- the
support file is JSX and was reaching babel with no preset.
The suite ran nowhere. `cypress run` means e2e -- its specPattern is `cypress/e2e/**` --
and these specs sit beside the components in `src`, so no job has ever executed them. A
`cypress-component-tests` job runs them now; without it this breaks again in silence.
One test fails the moment the suite can run, and it is mine, from #660: it asserted the URL
column is the widest in the row. It is not, and never was -- at that width the fixed columns
take ~307px of 400, leaving the URL ~93 against Control's 146. It shipped green because
nothing could run it. It asserts a usable remainder instead.
334 component tests pass, 69 suites / 1214 jest tests pass, `npm run build` is clean.
* review: an assertion with a floor, and comments that told the story instead of the rule
The leftover-width test had a lower bound and nothing else, so it passed on any column
wide enough to have padding -- including one that had grown, which is what losing
table-layout: fixed does. It measures the leftover now: the sized columns are held to
the widths they declare, and the truncating column to exactly what they did not take.
Comments in the config, the CI job and the spec cut back to the rule a reader needs. The
version history and the measurements are in the PR.
Every Python job started failing in "Install Requirements" with
Error: [Errno 2] No such file or directory: '/home/circleci/project/venv/bin/python3'
on branches whose only changes were under app/. The same config passed two days
earlier, so this is the environment moving, not the code.
The cache key names the minor version (`deps-3.13-...`) but not the patch, and the
cached paths include the whole `venv`. A venv's bin/python3 points at the exact
interpreter that built it, so once cimg/python:3.13 takes a patch bump the restored
venv refers to an interpreter the image no longer has, and creating a venv over that
directory fails instead of repairing it.
Two changes, both needed:
- Rebuild the venv when its interpreter cannot run. This heals the restored cache
whatever left it unusable, and costs a reinstall only on the first run after a bump.
- Move the cache key (-5 -> -6). CircleCI keys are immutable and `save_cache` will
not overwrite one, so without this the poisoned entry is restored and discarded on
every run forever and is never replaced.
Not reproduced locally: macOS python recreates the symlink and exits 0, so the exact
crash is CI's Linux python. The guard does not depend on the mechanism -- it removes
any venv whose bin/python3 will not run, which is the state the error names. Verified
that the guard heals a venv with a dangling interpreter and keeps a healthy one, so the
cache still pays off.
Unrelated and still open: api-tests fails on master itself, in
test_format_video_filename_uses_channel_name_over_info_json, which flakes under -nauto
and passes alone.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI has been testing 3.11 while nothing ships it. docker/api/Dockerfile and
docker/help/Dockerfile are python:3.13-trixie, docker/controller is
python:3.13-slim-trixie, Trixie's system interpreter is 3.13, and Debian 12 --
the last place 3.11 was current -- is pinned to the debian12-final tag and takes
no new code. So every Python job here exercised an interpreter no user has.
All four CircleCI Python jobs and the GitHub Actions controller job move to 3.13.
The cache key goes with them: it stores the whole venv, so restoring a
deps-3.11 venv into a 3.13 container would hand it an interpreter that no longer
exists.
The disabled api-tests-3-12 job is deleted rather than repointed. It was
switched off with the note "Sanic does not yet support 3.12", which 3.13 disproves,
and carrying a second version around is what let this drift open in the first
place.
Verified on 3.13.12 / pytest 9.0.3 before changing anything: the backend suite
passes (1709 passed, 5 skipped), alembic upgrade runs and `db check` reports no
new operations, and the controller suite passes bar two docker-mode dashboard
tests that fail identically on 3.11 here -- pre-existing and environment-dependent,
not a 3.13 regression.
`sanic-api-start` is the one job I could not verify locally: wrolpi/api_utils.py
sets `Sanic.start_method = "fork"` and macOS has defaulted to spawn since 3.8, so
that job fails here on any Python version. Linux CI is unaffected -- fork is
still the default there on 3.13 -- and the test suite already runs the Sanic app
in-process, which is the substantive check. Worth knowing that 3.14 changes the
Linux default away from fork, so that line has a deadline on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps, one of them wide.
The contract tests added in c349e15c already ran inside `pytest -nauto`, but
fixture drift is silent by nature -- the frontend suite passes against a body the
API stopped sending -- so it deserves a check whose name says what broke.
`api-contract-tests` restores the same dependency cache as the other Python jobs,
skips the ffmpeg/catdoc install it does not need (verified), and runs the one
file, so it answers in under a minute instead of somewhere inside seventeen
hundred results.
The wide one: jest-tests.yml triggered on `pull_request: branches: [master]`, so
a pull request into anything else ran neither jest nor the webpack build. The
entire Semantic UI to Mantine migration went in that way -- seventy-five files
across nine pull requests into feature/custom-theme, not one of which ran either.
The build step is the one that matters most: jest and tsc both pass on a named
import that does not exist and only webpack rejects it, which is exactly how a
broken Archive.js survived a "clean" run earlier in this migration. The suite
did run on the branch after each merge, which is the wrong side of the merge to
learn about it.
`pull_request` is now unfiltered rather than listing branches, so the next
long-lived branch does not have to remember to add itself. It stays
`pull_request`, not `pull_request_target`: a fork's run gets a read-only token
and no secrets, and this workflow interpolates no event data, so widening the
trigger adds no injection or exfiltration path.
Worth knowing separately: none of this gates anything. master has no branch
protection and the only ruleset covers `release` (deletion, non-fast-forward), so
every check here is advisory. That is a repository setting, and Roland's to make.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(app): depend on @testing-library/dom directly
jest-dom 7 moved @testing-library/dom from a bundled dependency to a peer
dependency. Nothing else pulls the top-level copy in — @testing-library/react
13 carries its own nested v8 — so it was present only as a peer, and CI
installs with --legacy-peer-deps, which skips peers. Every suite then failed
to load setupTests.js with "Cannot find module '@testing-library/dom'".
Declare it as a devDependency so it installs under either resolution mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(app): import mount from cypress/react
Cypress 14 removed the cypress/react18 subpath; cypress/react mounts React
18 itself. The e2e support file loads commands.js, so the stale import broke
bundling for every spec, not just component tests:
Module not found: Package path ./react18 is not exported from package cypress
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: install and cache Cypress the way the bump requires
The Cypress binary lives in ~/.cache/Cypress, outside node_modules, but only
node_modules was cached. A restored cache satisfied npm, so npm skipped the
postinstall download and the 15.x binary was never fetched:
The cypress npm package is installed, but the Cypress binary is missing.
Cache both paths and install the binary explicitly.
Also align the JS toolchain with what actually ships. Debian 13 devices run
nodejs 20.19.2 and install with a plain `npm install`; the Docker app image
does the same. CI pinned older Node and passed --legacy-peer-deps, which
skips peer dependencies -- that is why jest-dom 7's peer requirement passed
review and broke CI. Match the devices instead.
Pin caddy to 2.11.4, the version Debian 13 devices run, rather than latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: test on Python 3.13, the version that ships
Debian 13 devices run Python 3.13.5 and the api/controller/help images are
python:3.13-trixie, but CI ran 3.11 (and one job on 3.12), so the runtime
under test was two minors behind the runtime in production.
Verified on 3.13 before switching: backend 1712 passed, 5 skipped; controller
883 passed. The only new warning is SQLAlchemy 1.3.22 reaching Python's
deprecated datetime.utcfromtimestamp(), which is noise rather than a break.
Drop api-tests-3-12: no workflow referenced it, it duplicated api-tests, and
its "Sanic does not yet support 3.12" note is stale now that devices run 3.13.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:
- wrolpi-chrome.zip Load-Unpacked sideload for Chrome / Brave / Edge
- wrolpi-firefox.xpi Mozilla-signed (unlisted self-distribution),
one-click install in any Firefox
- versions.json Per-browser version metadata for the install page
Backend (modules/archive/api.py, wrolpi/root_api.py):
- POST /api/archive/upload accepts an optional repeated `tagNames`
multipart form field; the upload switch handler applies tags to the
new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
created inside a per-name SAVEPOINT so concurrent inserts on the same
novel name don't abort the transaction (Tag.name has unique=True).
- GET /api/extensions returns metadata (per-file availability + size +
versions.json contents) for the install page.
- GET /api/extensions/<filename> serves the binary, whitelisted by
filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
Content-Disposition: attachment header, so Firefox triggers its
add-on install handler instead of saving the file.
Frontend (app/src/):
- New /admin/extension page (linked from AdminRoute) detects the user's
browser via navigator.userAgent and renders an install card per
Firefox / Chromium / Other. Pulls metadata from /api/extensions to
show file size and version, and gracefully reports 'Not yet
available' when binaries are missing.
- Dashboard's Getters component supports a deep link from the
extension's 'Configure Download' button:
/?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
On mount, the matching downloader form auto-opens with URLs pre-
filled. Query params are cleared on close so a refresh doesn't
re-open the modal.
- DownloadMenu and the per-downloader forms accept an `initialUrls`
seed via the existing `download` prop pattern, joined with newlines
for the `urls` textarea (and first URL for single-URL forms).
Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Storing video settings in session storage to help user start downloads consistently.
Applying channel tag name to new channels when download videos or channels.
Initializing map tiles when importing and resetting map.
Python 3.11 is now required, old versions were causing unpredictability in tests. (Sanic does not yet
support 3.12)
Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
https://github.com/sanic-org/sanic/issues/2921
New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes. The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation. Now all multiprocessing tools are attached to the
app.shared_ctx. This means `api_app` is imported in many, many places.
This forced a change in how the DownloadManager works. Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue. Now, a single worker checks
for new downloads and sends a Sanic signal.
Flags have been reworked to use the `api_app`. I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
Displaying star for Uploader's comments, heart for favorited comments.
Adding wrolpi help files desktop shortcut to rpi image.
Using zim file names as FileGroup.title.
Removing accidental output commit.