Commit graph

54 commits

Author SHA1 Message Date
lrnselfreliance
1867382d7b
test: the component suite has not run since cypress 15, on any machine (#661)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* test: the component suite has not run since cypress 15, on any machine

Six component spec files -- 334 tests, including the 4,300-line layout suite -- could not
start.  Cypress exited before printing anything, and under its default Electron browser it
did not even do that: it hung at 0% CPU until killed.  Chrome prints the error in seconds,
which is the only reason this was findable.

Three things broke, all of them from the 13.17.0 -> 15.20.0 bump.

`baseUrl: "src"`.  Cypress 15 loads its config with tsx, and tsx honours tsconfig path
resolution.  Cypress requires Node's core `events` module; with that baseUrl the bare
specifier resolves against `src`, and on a case-insensitive filesystem it matches
`src/Events.js`.  That file imports the app, so the whole component tree was required in
Node: JSX in `.js` files, which esbuild will not parse.

This is why CI stayed green while every local run died.  Linux is case-sensitive, `events`
never matched `Events.js`, and it resolved to the built-in.  The baseUrl buys nothing --
there is not one absolute import in the codebase -- so it goes.

`framework: "create-react-app"`.  Cypress 14 removed that dev server; the handler is in the
13.17.0 binary and absent from 15.  The webpack config is handed over explicitly now, taken
from react-scripts so components compile exactly as `npm start` compiles them.  Two things
follow from that: `webpack-dev-server` 5 is a devDependency because Cypress requires 5 and
react-scripts pins 4 (npm nests react-scripts' copy, and the production build is unchanged),
and babel-loader is widened to `cypress/`, which react-scripts does not compile -- the
support file is JSX and was reaching babel with no preset.

The suite ran nowhere.  `cypress run` means e2e -- its specPattern is `cypress/e2e/**` --
and these specs sit beside the components in `src`, so no job has ever executed them.  A
`cypress-component-tests` job runs them now; without it this breaks again in silence.

One test fails the moment the suite can run, and it is mine, from #660: it asserted the URL
column is the widest in the row.  It is not, and never was -- at that width the fixed columns
take ~307px of 400, leaving the URL ~93 against Control's 146.  It shipped green because
nothing could run it.  It asserts a usable remainder instead.

334 component tests pass, 69 suites / 1214 jest tests pass, `npm run build` is clean.

* review: an assertion with a floor, and comments that told the story instead of the rule

The leftover-width test had a lower bound and nothing else, so it passed on any column
wide enough to have padding -- including one that had grown, which is what losing
table-layout: fixed does.  It measures the leftover now: the sized columns are held to
the widths they declare, and the truncating column to exactly what they did not take.

Comments in the config, the CI job and the spec cut back to the rule a reader needs.  The
version history and the measurements are in the PR.
2026-08-12 21:57:07 -06:00
lrnselfreliance
8a74140f95
ci: a cached venv outlives the interpreter it was built for (#655)
Every Python job started failing in "Install Requirements" with

    Error: [Errno 2] No such file or directory: '/home/circleci/project/venv/bin/python3'

on branches whose only changes were under app/.  The same config passed two days
earlier, so this is the environment moving, not the code.

The cache key names the minor version (`deps-3.13-...`) but not the patch, and the
cached paths include the whole `venv`.  A venv's bin/python3 points at the exact
interpreter that built it, so once cimg/python:3.13 takes a patch bump the restored
venv refers to an interpreter the image no longer has, and creating a venv over that
directory fails instead of repairing it.

Two changes, both needed:

  - Rebuild the venv when its interpreter cannot run.  This heals the restored cache
    whatever left it unusable, and costs a reinstall only on the first run after a bump.
  - Move the cache key (-5 -> -6).  CircleCI keys are immutable and `save_cache` will
    not overwrite one, so without this the poisoned entry is restored and discarded on
    every run forever and is never replaced.

Not reproduced locally: macOS python recreates the symlink and exits 0, so the exact
crash is CI's Linux python.  The guard does not depend on the mechanism -- it removes
any venv whose bin/python3 will not run, which is the state the error names.  Verified
that the guard heals a venv with a dangling interpreter and keeps a healthy one, so the
cache still pays off.

Unrelated and still open: api-tests fails on master itself, in
test_format_video_filename_uses_channel_name_over_info_json, which flakes under -nauto
and passes alone.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:22:21 -06:00
Roland
a4db2aceac
ci: test on Python 3.13, the version every deployment actually runs
CI has been testing 3.11 while nothing ships it.  docker/api/Dockerfile and
docker/help/Dockerfile are python:3.13-trixie, docker/controller is
python:3.13-slim-trixie, Trixie's system interpreter is 3.13, and Debian 12 --
the last place 3.11 was current -- is pinned to the debian12-final tag and takes
no new code.  So every Python job here exercised an interpreter no user has.

All four CircleCI Python jobs and the GitHub Actions controller job move to 3.13.
The cache key goes with them: it stores the whole venv, so restoring a
deps-3.11 venv into a 3.13 container would hand it an interpreter that no longer
exists.

The disabled api-tests-3-12 job is deleted rather than repointed.  It was
switched off with the note "Sanic does not yet support 3.12", which 3.13 disproves,
and carrying a second version around is what let this drift open in the first
place.

Verified on 3.13.12 / pytest 9.0.3 before changing anything: the backend suite
passes (1709 passed, 5 skipped), alembic upgrade runs and `db check` reports no
new operations, and the controller suite passes bar two docker-mode dashboard
tests that fail identically on 3.11 here -- pre-existing and environment-dependent,
not a 3.13 regression.

`sanic-api-start` is the one job I could not verify locally: wrolpi/api_utils.py
sets `Sanic.start_method = "fork"` and macOS has defaulted to spawn since 3.8, so
that job fails here on any Python version.  Linux CI is unaffected -- fork is
still the default there on 3.13 -- and the test suite already runs the Sanic app
in-process, which is the substantive check.  Worth knowing that 3.14 changes the
Linux default away from fork, so that line has a deadline on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:05:43 -06:00
Roland
fb4677bcd6
ci: a named check for fixture drift, and run jest on pull requests again
Two gaps, one of them wide.

The contract tests added in c349e15c already ran inside `pytest -nauto`, but
fixture drift is silent by nature -- the frontend suite passes against a body the
API stopped sending -- so it deserves a check whose name says what broke.
`api-contract-tests` restores the same dependency cache as the other Python jobs,
skips the ffmpeg/catdoc install it does not need (verified), and runs the one
file, so it answers in under a minute instead of somewhere inside seventeen
hundred results.

The wide one: jest-tests.yml triggered on `pull_request: branches: [master]`, so
a pull request into anything else ran neither jest nor the webpack build.  The
entire Semantic UI to Mantine migration went in that way -- seventy-five files
across nine pull requests into feature/custom-theme, not one of which ran either.
The build step is the one that matters most: jest and tsc both pass on a named
import that does not exist and only webpack rejects it, which is exactly how a
broken Archive.js survived a "clean" run earlier in this migration.  The suite
did run on the branch after each merge, which is the wrong side of the merge to
learn about it.

`pull_request` is now unfiltered rather than listing branches, so the next
long-lived branch does not have to remember to add itself.  It stays
`pull_request`, not `pull_request_target`: a fork's run gets a read-only token
and no secrets, and this workflow interpolates no event data, so widening the
trigger adds no injection or exfiltration path.

Worth knowing separately: none of this gates anything.  master has no branch
protection and the only ruleset covers `release` (deletion, non-fast-forward), so
every check here is advisory.  That is a repository setting, and Roland's to make.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:05:08 -06:00
lrnselfreliance
616424b223
fix: repair CI after the dependency bumps, and align it with the Trixie build (#651)
* fix(app): depend on @testing-library/dom directly

jest-dom 7 moved @testing-library/dom from a bundled dependency to a peer
dependency. Nothing else pulls the top-level copy in — @testing-library/react
13 carries its own nested v8 — so it was present only as a peer, and CI
installs with --legacy-peer-deps, which skips peers. Every suite then failed
to load setupTests.js with "Cannot find module '@testing-library/dom'".

Declare it as a devDependency so it installs under either resolution mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(app): import mount from cypress/react

Cypress 14 removed the cypress/react18 subpath; cypress/react mounts React
18 itself. The e2e support file loads commands.js, so the stale import broke
bundling for every spec, not just component tests:

  Module not found: Package path ./react18 is not exported from package cypress

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: install and cache Cypress the way the bump requires

The Cypress binary lives in ~/.cache/Cypress, outside node_modules, but only
node_modules was cached. A restored cache satisfied npm, so npm skipped the
postinstall download and the 15.x binary was never fetched:

  The cypress npm package is installed, but the Cypress binary is missing.

Cache both paths and install the binary explicitly.

Also align the JS toolchain with what actually ships. Debian 13 devices run
nodejs 20.19.2 and install with a plain `npm install`; the Docker app image
does the same. CI pinned older Node and passed --legacy-peer-deps, which
skips peer dependencies -- that is why jest-dom 7's peer requirement passed
review and broke CI. Match the devices instead.

Pin caddy to 2.11.4, the version Debian 13 devices run, rather than latest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: test on Python 3.13, the version that ships

Debian 13 devices run Python 3.13.5 and the api/controller/help images are
python:3.13-trixie, but CI ran 3.11 (and one job on 3.12), so the runtime
under test was two minors behind the runtime in production.

Verified on 3.13 before switching: backend 1712 passed, 5 skipped; controller
883 passed. The only new warning is SQLAlchemy 1.3.22 reaching Python's
deprecated datetime.utcfromtimestamp(), which is noise rather than a break.

Drop api-tests-3-12: no workflow referenced it, it duplicated api-tests, and
its "Sanic does not yet support 3.12" note is stale now that devices run 3.13.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:01:52 -06:00
Roland
d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00
Roland
a9a460c54c
Creating browser extension to facilitate starting downloads. Chrome/Brave and Firefox supported.
The extension is hosted in a sibling repo and
is distributed off-store. Each WROLPi release ships the prebuilt artifacts
under wrolpi/static/extensions/ and serves them at /api/extensions/<file>:

  - wrolpi-chrome.zip       Load-Unpacked sideload for Chrome / Brave / Edge
  - wrolpi-firefox.xpi      Mozilla-signed (unlisted self-distribution),
                            one-click install in any Firefox
  - versions.json           Per-browser version metadata for the install page

Backend (modules/archive/api.py, wrolpi/root_api.py):

  - POST /api/archive/upload accepts an optional repeated `tagNames`
    multipart form field; the upload switch handler applies tags to the
    new FileGroup via FileGroup.set_tags(). Unknown tag names are auto-
    created inside a per-name SAVEPOINT so concurrent inserts on the same
    novel name don't abort the transaction (Tag.name has unique=True).
  - GET /api/extensions returns metadata (per-file availability + size +
    versions.json contents) for the install page.
  - GET /api/extensions/<filename> serves the binary, whitelisted by
    filename. Firefox .xpi is served as application/x-xpinstall WITHOUT a
    Content-Disposition: attachment header, so Firefox triggers its
    add-on install handler instead of saving the file.

Frontend (app/src/):

  - New /admin/extension page (linked from AdminRoute) detects the user's
    browser via navigator.userAgent and renders an install card per
    Firefox / Chromium / Other. Pulls metadata from /api/extensions to
    show file size and version, and gracefully reports 'Not yet
    available' when binaries are missing.
  - Dashboard's Getters component supports a deep link from the
    extension's 'Configure Download' button:
      /?downloader=<archive|video|...>&download_url=<u1>&download_url=<u2>
    On mount, the matching downloader form auto-opens with URLs pre-
    filled. Query params are cleared on close so a refresh doesn't
    re-open the modal.
  - DownloadMenu and the per-downloader forms accept an `initialUrls`
    seed via the existing `download` prop pattern, joined with newlines
    for the `urls` textarea (and first URL for single-URL forms).

Behavior is fully backwards compatible: pre-existing API callers can
still POST to /api/archive/upload without `tagNames`, and the
DashboardPage modal continues to open via the existing button click
when no deep-link query params are present.
2026-05-03 11:48:50 -06:00
Roland
16d24a588d Running FE jest tests on github. 2025-12-06 15:11:37 -07:00
Roland
1d8738ace2 Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
b9f39d7819
No longer running tests on CircleCI that consistently fail to cleanup. 2025-05-05 11:17:58 -06:00
Roland
c2ca5fb3b3
Disabling cypress firefox testing because of irregular failures. 2024-12-16 16:37:37 -07:00
Roland
7d6b7114a7
Adding TRACE level for logging beyond debug. Reworking async_client for tests, not all tests need the App to handle signals, so any test that needs signals can use the new await_switches fixture which primes the app. 2024-12-11 11:52:33 -07:00
Roland
90a14c3dc0
Increasing App CircleCI sizes which may help with server disconnects. 2024-11-28 09:49:30 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
317f3f3dfb
Top processes are displayed on Status page. Nice of renderd is lowered to avoid clobbering system at startup. 2024-10-16 18:56:27 -06:00
Roland
265c08f25c
Testing that react app can be built and started. 2024-05-08 19:16:55 -06:00
Roland
ee99803586
Testing node 22. 2024-05-08 18:18:52 -06:00
Roland
d50534b0b7
Storing junit test results file. 2024-05-07 10:50:06 -06:00
Roland
8ad653fc40
Testing that api can start. 2024-05-07 10:39:00 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
Displaying star for Uploader's comments, heart for favorited comments.

Adding wrolpi help files desktop shortcut to rpi image.

Using zim file names as FileGroup.title.

Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
1209fa8054
Modifying pytest config to test wrolpi directories. 2024-02-12 11:56:46 -07:00
Roland
a4aceb3cfc
Adding Node 20 test. 2024-02-10 21:58:35 -07:00
Roland
e1621fb57b
Removing React 12 Circle CI test. 2024-02-05 14:27:18 -07:00
Roland
04cb24e579
Adding Node 20 to circle ci tests. 2023-11-29 21:26:58 -07:00
Roland
63e8512fb2
Changing cache name to avoid Circleci issue. 2023-04-10 16:52:32 -06:00
Roland
023e6c771e
Adding three library to help circleci tests pass. 2023-04-05 10:47:56 -06:00
Roland
d0be1ea774
Changing cache name to force stl viewer installation. 2023-04-05 10:44:23 -06:00
Roland
06f99abac4
Testing Python 3.11. 2023-03-01 13:25:58 -07:00
Roland
cedbff9f9c
Using small container for circleci. 2023-02-09 14:21:46 -07:00
Roland
a658c357a6
Caching based on version and requirements checksum only. 2023-01-07 20:23:14 -07:00
Roland
b1d8e33cd8
Modifying CircleCI config to also test React App. Caching Python venv on CircleCI. 2023-01-07 20:05:25 -07:00
Roland
162357554a
Including app-tests in circleci jobs. 2023-01-07 18:55:07 -07:00
Roland
84802eaf66
Updating dependencies for App testing. 2023-01-07 18:53:22 -07:00
Roland
f639e01951
Removing Latex as dependency. Using browser to convert OTP pads into printable documents. 2022-06-26 16:27:56 -06:00
Roland
82b58e40c0
Removing 3.11. 2022-03-09 19:16:50 -07:00
Roland
47964ef845
Adding 3.11 to circleci tests. 2022-03-09 19:16:14 -07:00
Roland
3debc87bf0
Fixing job names. 2022-03-09 19:13:21 -07:00
Roland
3ee9492c37
Testing multiple python versions. 2022-03-09 19:09:09 -07:00
Roland
76aee2620d
Renaming circleci job. 2022-03-09 18:40:25 -07:00
Roland
4697014d97
Opening download urls in new tab. 2022-03-09 18:16:45 -07:00
Roland
52e183c7ce
Installing latex. 2022-02-20 11:33:32 -07:00
Roland
4a85f66889
Calling pytest relatively. 2022-02-20 11:28:53 -07:00
Roland
1ca8746967
Updating before installing. 2022-02-20 11:26:58 -07:00
Roland
5076e95a80
Installing ffmpeg. 2022-02-20 11:26:14 -07:00
Roland
aaf07bdfd6
Installing requirements into venv directory, caching it. 2022-02-20 11:24:43 -07:00
Roland
d40c3a5994
Changing db port. 2022-02-20 11:22:33 -07:00