Commit graph

2559 commits

Author SHA1 Message Date
lrnselfreliance
06e35e1095
Archive container reaps its zombie processes (#837)
The archive service is PID 1 in its container and never reaps orphans.
check_output kills each command's process group, so every archive leaves
chrome, cat, and node grandchildren behind as zombies.  A long-running
container accumulated hundreds of them, counting against its PID limit.

init: true runs tini as PID 1, matching the web container.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:10:17 -06:00
lrnselfreliance
c6613dfee0
Debian ISO: hostname defaults to wrolpi, boot menu times out (#836)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Debian ISO: hostname defaults to wrolpi, boot menu times out

The installer's hostname question defaulted to "debian", so an install that
kept the default was not reachable at wrolpi.local.  A preseed in the
installer initrd now defaults it to wrolpi; the question is still shown (and
a DHCP-provided hostname still wins).

The ISO's GRUB and ISOLINUX menus had no timeout, so a Live USB waited at the
boot menu forever after any reboot or power loss.  The menus now boot the
default entry ("Run WROLPi (live)") after 10 seconds.

check-iso.sh fails the build if either regresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Debian ISO: say what the wrolpi hostname is for

The image does not include avahi-daemon, so the hostname alone does not make
wrolpi.local resolve.  The default still names the TLS certificate, the
SMB/NetBIOS share, and the DHCP hostname a router may publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 06:59:21 -06:00
lrnselfreliance
baa86b4ad1
Upgraded Live USBs keep a writable library (#835)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
ISOs before wrolpi became a system account wrote the Live USB's library on
the persistence partition as UID 1000.  Upgrading such a USB with a newer
ISO (wrolpi-usb.sh upgrade) keeps the persistence partition, so the library
would belong to the live session's "user" and the services could not write
to it.  Bootstrap now re-owns the library once when its owner is not this
image's wrolpi.  Live USBs only: installed systems' drives are left alone.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:23:53 -06:00
lrnselfreliance
f644a80c9a
Debian installs create the user the installer asks for (#834)
The ISO's chroot hook created wrolpi as a login user (UID 1000, password
"wrolpi", full sudo, LightDM autologin).  The Debian installer creates the
user it asks for only when no UID 1000-59999 user exists, so it silently
skipped that user: the only login on an installed system was wrolpi/wrolpi,
with SSH and sudo.

wrolpi is now a system account with no password or login sudo, as on every
other WROLPi deployment.  The installer creates the user it asks for (with
sudo, no autologin).  The live session logs in as live-config's "user".
Desktop shortcuts moved to /etc/skel so every user gets them.

check-iso.sh now fails the build when the image has a UID 1000-59999 user.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:23:40 -06:00
lrnselfreliance
826f1cdc42
The drive wizard never offers an EFI system partition (#833)
On a Debian install the onboarding wizard offered the EFI system partition
(unlabeled vfat at /boot/efi) as the primary WROLPi drive; choosing it would
mount the boot partition over /media/wrolpi.  Candidates were filtered by
the "EFI" label only, and the Debian installer leaves the ESP unlabeled.

/boot/efi is now an excluded mountpoint, and partitions with the EFI system
partition type (GPT GUID or MBR 0xef, from lsblk PARTTYPE) are excluded even
when unmounted, e.g. on a second internal disk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:26:18 -06:00
lrnselfreliance
80678e1f6e
A library on the internal disk gets a database (#832)
"Run repair without external drive" kept the library on the internal disk,
but the API's unmounted-media guard refuses to create or use a database
beneath /media/wrolpi when it is not a mount.  After onboarding without a
drive the API restarted forever with no database, and downloads were
blocked as "media not mounted".

The Controller now records the choice as drives.library_on_internal_disk in
controller.yaml -- the only source of it.  The database guard and the
media_mounted check both read that marker, so require_media_mounted keeps
its meaning and needs no change.  Onboarding a drive later clears the
marker, so it is never copied onto the drive.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:15:33 -06:00
lrnselfreliance
ad56134851
install.sh installs on Debian 13 (#830)
install.sh only accepted Debian 12, which cannot run this version, and
rejected Debian 13.  It now requires Debian 13 or newer on both PCs and
Raspberry Pis.  The install scripts point to the drive wizard instead of
manual mounting.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:51:40 -06:00
lrnselfreliance
559bfebc56
Installed Debian systems keep Caddy (#831)
* Installed Debian systems keep Caddy

The Debian installer purges every package listed in the ISO's
live/filesystem.packages-remove.  live-build lists there every package
installed after the package-lists install pass, which includes anything a
chroot hook installs with apt -- and the hook installed Caddy that way.
An installed system therefore had no Caddy: wrolpi-bootstrap failed copying
the Caddyfile, the API/app/Kiwix were never enabled, and HTTPS never came up.

Caddy now comes from package-lists/, with its repository and signing key in
config/archives/ (installed into both the build chroot and the image).
check-iso.sh fails the build when the installer would remove any package,
and can be run by hand against any ISO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Debian ISOs install Caddy from Caddy's repository

Moving caddy into package-lists/ made apt read config/archives/ before
ca-certificates was installed, so the HTTPS fetch of Caddy's repository
failed certificate verification.  apt ignores a failed repository, and the
image silently got Debian's much older caddy 2.6.2 instead of 2.11.

ca-certificates is now part of the bootstrap, and Debian's caddy is pinned
to -1, so an unreachable Caddy repository fails the build and installed
systems never switch to Debian's caddy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:42:37 -06:00
lrnselfreliance
7aa7481a56
The hotspot starts on boot (#829)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Nothing started the hotspot since hotspot_on_startup moved out of the
API, and NetworkManager does not autoconnect the Hotspot connection.
The Controller now starts it once per boot unless hotspot.on_startup is
false; a legacy wrolpi.yaml hotspot_on_startup: false is migrated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:29:15 -06:00
lrnselfreliance
b83a9cf017
Onboarding a drive keeps a library on the internal disk (#828)
The onboarding wizard deleted /media/wrolpi/config before mounting a
drive, and the shadowed data check ignored config/.  A library kept on
the internal disk keeps wrolpi.db and its configs there, so onboarding
a drive later deleted them without warning.  Only a config/ holding
nothing but the ssl/ CA is now treated as transient.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:19:41 -06:00
lrnselfreliance
1554dfb268
A new database is refreshed on startup (#827)
Download and Upload are disabled until a full refresh completes, and a
fresh install never had one until the user found the refresh link.
Startup now queues it, and the dashboard says why Download and Upload
are disabled meanwhile.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:07:04 -06:00
lrnselfreliance
295b3ccfc8
INSTALL.md points to the Controller drive wizard (#825)
* INSTALL.md points to the Controller drive wizard

Debian installs are Debian 13 from the amd64 ISO.  The Raspberry Pi
steps no longer hand-edit fstab or run repair.sh; a shared First Boot
section covers the drive, certificate trust, and the first refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* INSTALL.md: Debian installs use the drive wizard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:57:10 -06:00
lrnselfreliance
dc2ab22ae0
Debian installs show the Controller drive wizard (#826)
wrolpi-bootstrap.sh created controller.yaml on installed systems, which
hid the onboarding wizard.  Installed systems now wait for the wizard,
whose repair.sh finishes setup.

"Run repair without external drive" now writes controller.yaml before
starting repair, so the wizard does not return after repair.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:50:20 -06:00
Roland
697a8180c7
Bump version to 0.30.0-beta
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:13:56 -06:00
lrnselfreliance
ce923cbff0
Video transcodes run at the least CPU priority (#824)
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:10:44 -06:00
lrnselfreliance
4e53e02519
Notifications without a URL are not links (#823)
Transcode, job, and Kiwix notifications send no URL, but every user notify
toast said "Click here to view the shared page" and opened /null on click.
Only a notification carrying a URL is followable now.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:37:26 -06:00
lrnselfreliance
87e146487b
Database schema lists each column's foreign key (#822)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Without them a model guessed joins: it joined video.channel_id to collection.id
(it is channel.id), and the query silently returned the wrong channel names.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:38:39 -06:00
lrnselfreliance
61e58fac17
Controller script elapsed time is correct outside UTC (#821)
systemd prints the start timestamp in local time, which was subtracted from
utcnow(), so elapsed_seconds was off by the UTC offset (6 hours in MDT).  Ask
systemd for a Unix timestamp instead.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 18:30:08 -06:00
Roland
06ca3d421f
Sign master for device upgrades
upgrade.sh only installs a branch tip signed with the WROLPi key; the tip was a
GitHub squash merge (#820).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:49:54 -06:00
lrnselfreliance
f27720683d
Read-only SQL query endpoint for advanced users (#820)
* Read-only SQL query endpoint for advanced users

POST /api/db/query runs one read-only statement; GET /api/db/schema describes
the tables.  Both are refused unless `allow_database_query: true` is set in
wrolpi.yaml (there is no UI for it).  The connection is opened with mode=ro and
an authorizer that allows only reads; queries are interrupted after 10 seconds,
rows are capped, and long values are truncated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Database queries cannot be enabled through the API, and use bounded memory

The config update API refuses to change `allow_database_query`; only an edit of
wrolpi.yaml on the WROLPi can enable queries.  Query connections refuse values
over 32 MB and rows over 100 columns, rows are read one at a time, and the
response stops at 5 million characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The database query setting can be changed like any other config

WROLPi is meant to be as secure as a bookshelf in your home; a LAN user who can
reach the API is trusted, so gating one key in the config API adds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Database queries can search FTS5 tables on SQLite 3.46

FTS5 reads `PRAGMA data_version` through the authorizer when it opens a table
on SQLite 3.46 (Debian 13), so every MATCH and the schema endpoint failed with
"vtable constructor failed".  A denied VACUUM INTO also explains itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A config status test attaches the shared context it reads

It passed only when an earlier test in the same worker had attached it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:34:00 -06:00
lrnselfreliance
3ea8eb81f6
Model reprs read only loaded state (#819)
A repr runs inside log calls. Reading a relationship or an expired column there runs a query, and
on an object whose session is gone it raises DetachedInstanceError instead of logging. Archive's
repr also stat'ed its SingleFile file on disk.

`wrolpi.common.loaded()` follows attributes through state that is already loaded (primary keys
from the identity key) and never runs SQL. Every model repr uses it; with everything loaded the
output is unchanged, except Video and Archive now show the FileGroup's primary path.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:47:32 -06:00
lrnselfreliance
c717f11821
Duplicate video cleanup keeps a complete copy and its viewing history (#818)
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest.  A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted.  The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.

Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before.  The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:25:25 -06:00
lrnselfreliance
a95b187edc
A yt-dlp format file is not the primary file when merged media is present (#817)
`get_primary_file` returned the first video/audio file in whatever order it
was given, so a group holding both `<stem>.mp4` and a leftover video-only
`<stem>.f313.webm` could make the leftover the primary file.  The Video then
pointed at a file without audio, and a download of it was deferred with
"Download did not contain audio stream" on every retry.

Skip yt-dlp per-format files (`.fNNN` stems) when a merged media file is in
the group.  A group of only format files still gets one as its primary.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:20:44 -06:00
lrnselfreliance
fc7db34864
Video download deletes yt-dlp format leftovers of any extension (#816)
A merge that never finished leaves `<stem>.f313.webm` (video-only) next to
where the merged file belongs.  A later download of the same video globs
every `<stem>.*` file into its FileGroup, but only `.part` files and format
files with the *same* extension as the merged video were deleted.  A `.webm`
leftover beside a new `.mp4` survived, joined the new FileGroup, and could
become the Video's file.

Delete every `<stem>.fNNN.*` file in the group, whatever its extension, which
also covers WROLPi's poster/ffprobe sidecars of an indexed leftover and
yt-dlp's `.ytdl` state files.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:20:25 -06:00
lrnselfreliance
4adc4e6a15
Drop the SQLAlchemy asyncio extra (#815)
Nothing uses sqlalchemy.ext.asyncio; an AsyncSession pilot on the request path was measured on a
Pi and dropped. SQLAlchemy 2.0 still requires greenlet on aarch64 and x86_64, so installs on the
Pi, Debian and Docker are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:18:16 -06:00
lrnselfreliance
8e2ab935af
Archive search counts from the search index (#814)
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:37:27 -06:00
lrnselfreliance
99a0fc275d
Archive and doc pages record a view in its own write transaction (#813)
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.

Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:13:50 -06:00
lrnselfreliance
4202e6a7e5
Downloads and uploads: don't read ORM objects after their session is gone (#812)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* File downloads: return plain values from upsert_file

Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`.  `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone.  SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends.  Failed downloads retry, re-downloading their files.

`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Archive uploads: return plain values from singlefile_to_archive

Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved.  It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:41:04 -06:00
lrnselfreliance
673c23188c
Request sessions keep objects after commit; guard dict() serializers (#811)
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query.  An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async.  Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.

Collection, CollectionItem and Channel dict() now run inside the serializer guard.  Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
  handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
  Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
  reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
  the session.

Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:15:33 -06:00
dependabot[bot]
1f8a887c11
build(deps): bump shell-quote from 1.10.0 to 1.12.0 in /app (#792)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.10.0 to 1.12.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/shell-quote/compare/v1.10.0...v1.12.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.12.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:48 -06:00
dependabot[bot]
d8d28503b5
build(deps): bump compression from 1.8.1 to 1.8.2 in /app (#793)
Bumps [compression](https://github.com/expressjs/compression) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/expressjs/compression/releases)
- [Changelog](https://github.com/expressjs/compression/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/compression/compare/v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: compression
  dependency-version: 1.8.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:43 -06:00
dependabot[bot]
562150631c
build(deps): bump proxy-addr from 2.0.7 to 2.0.8 in /app (#794)
Bumps [proxy-addr](https://github.com/jshttp/proxy-addr) from 2.0.7 to 2.0.8.
- [Release notes](https://github.com/jshttp/proxy-addr/releases)
- [Changelog](https://github.com/jshttp/proxy-addr/blob/master/HISTORY.md)
- [Commits](https://github.com/jshttp/proxy-addr/compare/v2.0.7...v2.0.8)

---
updated-dependencies:
- dependency-name: proxy-addr
  dependency-version: 2.0.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:39 -06:00
dependabot[bot]
d205f7ca72
build(deps): bump source-map-js from 1.2.1 to 1.2.2 in /app (#795)
Bumps [source-map-js](https://github.com/7rulnik/source-map-js) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/7rulnik/source-map-js/releases)
- [Changelog](https://github.com/7rulnik/source-map-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/7rulnik/source-map-js/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: source-map-js
  dependency-version: 1.2.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:34 -06:00
dependabot[bot]
a36e3a3770
build(deps): bump the npm-dependencies group in /app with 3 updates (#807)
Bumps the npm-dependencies group in /app with 3 updates: [@tabler/icons-react](https://github.com/tabler/tabler-icons/tree/HEAD/packages/icons-react), [web-vitals](https://github.com/GoogleChrome/web-vitals) and [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@tabler/icons-react` from 3.48.0 to 3.49.0
- [Release notes](https://github.com/tabler/tabler-icons/releases)
- [Changelog](https://github.com/tabler/tabler-icons/blob/main/packages/icons-react/CHANGELOG.md)
- [Commits](https://github.com/tabler/tabler-icons/commits/v3.49.0/packages/icons-react)

Updates `web-vitals` from 6.2.2 to 6.2.3
- [Changelog](https://github.com/GoogleChrome/web-vitals/blob/main/CHANGELOG.md)
- [Commits](https://github.com/GoogleChrome/web-vitals/compare/v6.2.2...v6.2.3)

Updates `@types/node` from 26.6.3 to 26.6.4
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@tabler/icons-react"
  dependency-version: 3.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: web-vitals
  dependency-version: 6.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.6.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:30 -06:00
dependabot[bot]
d5c02b34ae
build(deps): bump aiohttp (#806)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: controller-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:26 -06:00
dependabot[bot]
226f955dd0
build(deps): bump the python-dependencies group with 6 updates (#808)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: cachetools
  dependency-version: 7.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: libzim
  dependency-version: 3.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: python-dotenv
  dependency-version: 1.2.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytz
  dependency-version: '2026.5'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: selenium
  dependency-version: 4.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:21 -06:00
lrnselfreliance
ab6a40be20
API JSON: encode paths without a lookup per path (#810)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
CustomJSONEncoder looked up the media directory and called Path.relative_to
for every path it encoded.  A response with hundreds of paths (an Archive's
page with 51 history entries holds 416) spent most of its encoding time
there: 18.8 ms of 19 ms per response, measured on a Mac.  The encoder now
looks up the media directory once per response and strips it as a string
prefix: 1.5 ms for the same response.

Paths outside the media directory, relative paths, and siblings that share
the media directory's name prefix are still returned unchanged; the media
directory itself and "." are still "".

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:45:43 -06:00
lrnselfreliance
a8dde44d4f
Archive page: load history and tags eagerly (#809)
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive.  An
Archive with 51 history entries ran 107 statements per request.  Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:45:13 -06:00
lrnselfreliance
dab333124e
Upgrade to SQLAlchemy 2.0 (synchronous) (#803)
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2.  This is a mechanical port; the code
stays synchronous.

- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
  attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
  accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
  works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
  session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
  the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
  statement committed on its own.  2.0 autobegins, so the unique-stem
  migration's read snapshot could not become a writer after its helpers wrote
  on another connection ("database is locked").  The begin listener emits no
  BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
  its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
  no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:08:40 -06:00
lrnselfreliance
bb8945e8ad
Render unknown URLs as a 404 page inside the app layout (#805)
An unknown URL (e.g. /archive instead of /archives) had no matching route,
so react-router rendered the root errorElement in place of Root.  That put
PageNotFound outside ThemeProvider, and its Mantine Box threw
"MantineProvider was not found", leaving a blank page.

- A catch-all route renders PageNotFound inside Root, with the navigation
  bar, footer and theme.
- The root errorElement brings its own ThemeProvider, so it can render when
  Root itself fails.
- App exports its routes so a test can mount them in a memory router.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:36 -06:00
lrnselfreliance
d75be8e1a4
Download forms: allow a frequency of "Once" (#804)
* app: a download frequency of "Once" no longer blocks submission

useForm treated any falsy value as a missing required field.  "Once" is
frequency 0, so the Channel/Playlist, RSS, and Zim download forms never
became ready and the Download button stayed disabled.  Only null,
undefined, '', false, and NaN count as missing now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* downloads: editing to "Once" stores a once-download, not frequency 0

Once-downloads have a NULL frequency.  A 0 frequency counted as recurring
(frequency != None) but could never be scheduled (not frequency), so an
edited download stayed on the recurring list and could break renewal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:24 -06:00
lrnselfreliance
c8d5765ea3
Scrape: fix multiple suffixes and relative link resolution (#801)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Scrape: match any of the comma-separated suffixes

The UI's File Suffixes field accepts a comma-separated list, but the
downloader compared each link against the whole string, so a scrape with
more than one suffix never matched anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Scrape: resolve relative links against the page's directory

resolve_url appended a relative href to the page URL, so a link to
"a.txt" on ".../dir/page.html" became ".../dir/page.html/a.txt" and the
file download 404'd. Use urljoin, and resolve against the final URL after
redirects so a directory URL without a trailing slash still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:05:59 -06:00
lrnselfreliance
22b8b9c878
Scrape: honor <meta> charset when decoding pages (#800)
aiohttp's response.text() ignores <meta> charset declarations and assumes
UTF-8 when the Content-Type header has no charset, so legacy Latin-1 pages
crashed the Scrape downloader with UnicodeDecodeError. Decode with
UnicodeDammit instead: header charset first, then the page's declared
charset, then detection.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:39:39 -06:00
lrnselfreliance
5ec85c3e04
Help: document resuming where you left off (#799)
Bumps the help docs for the new "Resume Where You Left Off" page and the
recently_viewed.yaml config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:25:29 -06:00
lrnselfreliance
3c362799b5
Resume EPUBs and comic books where the reader left off (#798)
* Resume EPUBs and comic books where the reader left off

The EPUB viewer (epub.html) accepts `?cfi=` and posts its location and
progress to the page that embeds it on every page turn.  Progress comes from
the chapter and the page within it, rather than book.locations.generate(),
which stalls a Raspberry Pi on a large book.  A search result's `loc` still
wins over the saved position.

The comic viewer opens at the saved page and saves each page shown.

Both resume on the doc page and in the file preview.  The resume point is
decided once per opened file, so saves made while reading do not change the
EPUB viewer's URL and reload it.  The doc page tracks only the FileGroup's
primary file; another format beside it is not tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* EPUB/comic progress: never save a place the server would discard

The server treats a position at the start (the cover, under 1%) as not
started, and replaces the saved place with progress 0.  Both viewers
reported the place they opened on, so opening a finished book un-finished
it, and peeking at the cover erased the reader's page.

- The comic viewer does not save the page it opens on, nor the cover; a
  peek at the cover keeps the last page saved.
- The EPUB viewer posts nothing at the start of the book.
- Progress before the end is kept between 1% and 94% (resumableProgress), so
  early pages of a long book and the last pages before the end keep their
  position.  Only the last page (comics) or the end (EPUB) is finished.
- Non-finite EPUB progress is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:19:51 -06:00
lrnselfreliance
1ad451cddf
Resume videos and audio where the user left off; show progress on cards (#791)
* Resume videos and audio where the user left off; show progress on cards

Videos and audio resume where the user left off, on the video page and in
the file preview, and keep playing.  `?t=` starts at an explicit time
instead.

useFileProgress saves how far the user is through a file: every 15 seconds
while it changes, and a final save (a beacon, which survives the page
closing) on pause, end, hiding the page, leaving it, or switching files.
Each save is also kept in sessionStorage; on open the newer of that and the
server's position wins, so coming straight back resumes exactly even if the
final save is still in flight.

Cards show a progress bar along the bottom of their poster.  The dashboard's
Recently Viewed Files gains All / Continue; Continue uses the new
`in_progress` file search filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Progress review fixes: in_progress totals, latest dashboard request, preview viewed

- The in_progress search filter skips the uncounted fast path and is part of
  the totals cache key, so Continue never reports All's (or the library's)
  count.
- The dashboard's Recently Viewed hook applies only its latest response, and
  clears the list when switching All/Continue until the new files arrive.
- POST /api/files/file returns `viewed`, so a preview can tell a file
  finished elsewhere from an older position this tab holds.
- The progress hook tests restore Blob, fetch and sendBeacon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:29:22 -06:00
lrnselfreliance
fbdf5bac82
Rename skip_circleci to skip_ci (#797)
CircleCI is gone, so the skip and its flag (CIRCLECI → IS_CI) are named for
CI in general, and the flag reads only GITHUB_ACTIONS.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:28:13 -06:00
lrnselfreliance
bcbc42632d
Remove CircleCI (#796)
GitHub Actions runs every job CircleCI did (#789), so CircleCI only
duplicated them.  The README badge now shows the backend tests on master.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:19:38 -06:00
lrnselfreliance
1e112b678a
Run the CircleCI jobs on GitHub Actions (#789)
* Run the CircleCI jobs on GitHub Actions

Backend tests (pytest, API contract, migrations + API start) and Cypress
(component, e2e) now run as GitHub Actions workflows.  CircleCI stays in
place until these have run green alongside it.

- pip and npm caches via setup-python/setup-node, plus the Cypress binary;
  node_modules is no longer cached, to stay inside the 10 GB cache cap.
- Superseded pull request runs are cancelled to stay under the concurrent
  job limit.
- react-app-start is dropped: the e2e job waits for `npm start`, and
  jest-tests.yml already runs `npm run build`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the CI-unsupported tests on GitHub Actions too

skip_circleci only read CIRCLECI, so on GitHub Actions test_archive_upload
ran without the readability tooling it needs, and failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Run the backend tests on every vCPU

-nauto counts physical cores when psutil is installed, so the four-vCPU
runner got two workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Split the backend tests into four jobs by directory

The groups are balanced by test time, so each runs about a quarter of the
suite.  `rest` ignores the other groups' paths rather than listing its own,
so a new directory is tested before it is assigned to a group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:08:16 -06:00
lrnselfreliance
cb189bacb1
Give the repos ownership tests a fixed git identity (#790)
test_ownership.py's git() helper did not use TEST_GIT_ENV, so its commits
read the machine's git identity.  CI runners have none and cannot guess one,
so seven tests failed there on every run.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:01:59 -06:00