Commit graph

2678 commits

Author SHA1 Message Date
lrnselfreliance
6cd233dce6
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP

A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them.  wrolpi.yaml contributes only its library
settings.  Files deleted on the primary are kept.

A secondary never saves the synced configs and never downloads.  Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy.  It syncs in a daily window or on "Sync now"; WROL Mode stops it.

On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync has its own /admin tab; a sync's reads from the primary are bounded

The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.

A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped.  A primary address may not carry a
user name or password (it is logged and shown).  While a sync runs, the API
answers only the exact requests that show its progress or stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync keeps the secondary's replaced configs in today's config backup

The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Docker's Caddy logs every request

`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync reports its progress as it downloads, and refuses quietly

The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds.  Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time.  The primary's version is stored
without its trailing newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A WROLPi can back up to a second drive; sync settings live in sync.yaml

The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted.  The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start.  A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears.  Not in Docker, which
cannot reach other drives.

Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Sync tab shows the configured backup drive

getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured.  It also dropped
`version`.  A test now fails for any setting the API sends that getSettings
drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup can go to a folder on its drive, and remembers its last success

The Backup Drive panel takes a folder on the chosen drive.  The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused.  A missing folder is created once the drive is checked.

When the last backup succeeded is saved in sync.yaml, so it is known after a
restart.  "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The backup drive has its own window

The sync window was shown in the Sync panel but also scheduled the backup
drive.  The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync and the backup drive can each exclude folders

A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory.  Excluded
folders are never copied; what was copied before is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The folder browser does not offer lost+found

Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync never writes into config or tags, whatever their case

On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup writes the database snapshot only when the database changed

Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove the unused save_sync_config switch

Nothing activated it; SyncConfig saves directly.  The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:45 -07:00
lrnselfreliance
86f168d2c1
A script's result reads the journal when systemd has cleared its timestamps (#956)
The upgrade reloads systemd, which clears the run's exit timestamp, so a
successful upgrade looked like a script that never ran: finished false and
success null. The journal keeps systemd's unit success/failure record of the
run's end, so it now decides whether a run finished, and when.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:25:38 -07:00
lrnselfreliance
19ff59d5e0
A failed upgrade is shown on the upgrade page instead of redirecting home (#955)
* A failed upgrade is shown on the upgrade page instead of redirecting home

A failed upgrade restarts the services it stopped, so the upgrade page saw
them come back and redirected home as if the upgrade had worked.  The
Controller now reports how the last run of a script ended
(GET /api/scripts/{name}/result, from systemd's Result and exit status),
and the upgrade page reads it before redirecting.  A failure turns the
banner red, gives the exit code, opens the output, and offers a link back.

Only a run the page has seen start counts, because the result of an
earlier run is still reported until the new run begins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An upgrade's result survives the daemon-reload it does

The upgrade reinstalls its unit and reloads systemd, which clears the run's
timestamps but keeps Result and the exit status.  A failure is now known
from those alone, and the result reports whether the script is running, so
the upgrade page waits on that instead of on a timestamp that may never come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:11:59 -07:00
lrnselfreliance
d88db90ce9
The Settings upgrade can switch to a chosen branch (#954)
* The Settings upgrade can switch to a chosen branch

The Upgrade panel gains a Branch input: a dropdown listing the current
branch, master, and release that also takes a typed branch name.
/api/upgrade/start accepts an optional {"branch": ...}; without it the
current branch is upgraded as before.  A branch is written into the systemd
env file and passed to `upgrade.sh -b`, so only plain git branch names are
accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Controller upgrade refuses a branch that is not a plain branch name

The Controller's upgrade script wrote its branch straight into the systemd
env file, so a newline in it added a variable, and a leading dash became an
upgrade.sh option.  It now accepts the same branch names as the API, and no
script parameter may span lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:49:26 -07:00
lrnselfreliance
f3f762ef1b
Controller logs: an unparseable since is a 400, not empty logs (#953)
journalctl exits 1 with nothing on stdout when it cannot parse --since
(a bare "1h" is one such value); the controller returned that empty
stdout as the logs, with a 200.  A failed journalctl is now an error:
400 when it could not parse since, 500 otherwise.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:02:48 -07:00
Roland
d25ebb9e63
Bump version to 0.30.6-beta 2026-11-07 17:14:13 -07:00
lrnselfreliance
2e0e50c3aa
3D previews load ASCII STLs, stand upright, and fit the preview modal (#952)
* STL files preview with three's own STLLoader

react-stl-viewer inlines a copy of three-stdlib 2.17.2's STLLoader, which reads
ASCII STL text with LoaderUtils.decodeText.  three removed that in r175, so
every ASCII STL preview failed.  The three-stdlib override never reached that
copy, and react-stl-viewer has had no release since January 2024.

The 3MF preview's three.js viewer now previews STL files too, loading each
format with the loader that ships with three, so the loader always matches the
installed three.  react-stl-viewer and the three-stdlib override are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An e2e spec previews ASCII and binary STL files in a real browser

The STL preview only fails in a browser with WebGL, so the spec opens both
encodings on the live stack and checks the dimensions line appears without a
load error.  fb:reset can now write binary files ({base64: ...}).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* STL previews stand Z-up models upright

STLs are Z-up (the 3D printing convention) and three is Y-up, so the
preview rotates STL geometry to stand the model upright, as react-stl-viewer
did.  Model Z reads as Height again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3MF previews stand Z-up models upright too

3MF is Z-up like STL, so the rotation moves from the STL loader into the
shared viewer and applies to every model.  Model Z reads as Height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* File previews show their buttons without scrolling

.preview-fit reserved a fixed 18.75rem for the rest of the preview modal,
which was short: every framed preview overflowed a little, and a 3D model's
dimensions line pushed the action buttons below the fold.  The preview modal
now fills its height as a flex column, and the .preview-fit takes whatever
height is left.  Previews without a .preview-fit keep their content's height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3D previews frame the model and light it brightly

- The camera fits the model's bounding sphere into the narrower field of view,
  from a raised three-quarter front view, instead of backing off 2.2x along Z.
- A hemisphere light and a headlight that follows the camera (from a little
  above and left) replace lights whose intensities were set before three's
  r155 lighting change and read dim.
- Geometry without normals (3MF) gets creased normals, so flat faces no longer
  shade into their neighbors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:33:58 -07:00
lrnselfreliance
6c24d7d762
A move neither scans every FileGroup nor stalls the downloader's writes (#951)
* A move neither scans every FileGroup nor stalls the downloader's writes

Two stalls made a 109 file move take three minutes on a large library:

- Planning joined the move_sources temp table to file_group.  The temp
  table has no statistics, so with a real library's statistics SQLite
  scanned all of file_group (233k rows, 2.4 GB) instead of looking up 109
  paths in the primary_path index.  CROSS JOIN fixes the join order.

- The move's write transaction yielded to the event loop between chunks.
  The process that moves files also downloads; a download's write then
  waited on the move's lock synchronously, so the event loop could not
  resume the move: each write waited out the 30 s busy_timeout and failed
  ("Unable to renew downloads!").  The transaction now runs in a thread
  (inline in tests, which share one connection), so another writer waits
  only until the move commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A cancelled move waits for its thread however often it is cancelled

Review fix: the move waited for its thread on the first cancellation by
awaiting the thread's future again, unshielded.  Shutdown cancels twice
(run_loops cancels the file worker loop and stops waiting after 15 s, then
asyncio.run cancels every pending task, including the thread's future), so
the second cancellation raised while the thread was still in its
transaction, and handle_move reverted the files the thread then committed.

The move now waits on a threading.Event the thread sets when it leaves its
transaction, absorbing every cancellation meanwhile.  A cancelled future
reads "done" while its thread still runs, so it cannot be that signal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:47:12 -07:00
lrnselfreliance
aff5f1eb1f
A directory rename is finished by the file worker, not the request (#950)
* A directory rename is finished by the file worker, not the request

A directory rename queued a move, waited for it, then removed the old
directory and updated nested Collections itself.  A rename that outlasted
Sanic's RESPONSE_TIMEOUT (60 s) was cancelled while waiting: the move still
finished, but the old directory and its emptied subdirectories were left
behind, nested Collections kept their old directories, and the browser was
told the rename failed.

The move job now finishes the rename (nested Collections, then removing the
old directory, empty directories only), and the request returns once the
move is queued.  A failure is reported by the job's event.  An empty
directory can be renamed (it raised before).  The worker status carries
`rename_from`, so the Files page progress bar labels the rename, and the
file browser fetches again when the file worker is done.

A directory rename no longer queues a refresh of its parent: the file
worker runs the newest task first, so it scanned the directory before the
rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A queued rename returns its job, and the file browser waits for it

Review fixes:

- The cancellation test patched `wait_for_job`, which this path no longer
  calls, so it cancelled nothing.  It now asserts the invariant: the
  request only queues the rename (it never waits, and nothing has moved
  when it returns), and the file worker alone finishes it.

- The file browser inferred the end of a rename from the file worker's
  polled status.  A short job can start and finish between two polls, and
  the first idle poll (before the job started) spent the "fetch once"
  token, so the table could keep the old name.  POST /api/files/rename now
  returns `{"job_id": ...}` (null when the rename is already done), GET
  /api/files/jobs/<job_id> reports pending/complete/failed, and the file
  browser polls that job before fetching.

- useBrowseFiles ignores a fetch that settles after a newer one, so a
  listing requested before a rename cannot replace the one after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:09:33 -07:00
lrnselfreliance
c7bafaa7fb
Starting the app again adds no signal routes (#949)
Sanic's SignalRouter.finalize appends a `sanic.__signal__.__init__` handler
before it checks whether the router is already finalized.  Every start of the
app finalizes it again (the test client, Sanic's startup, and
finalize_signals_after_auto_routes), so each test left two more routes.  After
a few hundred tests in one process the compiled router was too deep for the
compiler (RecursionError in sanic_routing, intermittently failing whichever
tests a worker happened to run late).

api_app's signal router now skips a finalize when it is already finalized, and
registers that handler once.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:00:41 -07:00
lrnselfreliance
e5987fdd7e
ASCII STL files preview again (#948)
react-stl-viewer pins three-stdlib 2.17.2, whose STLLoader decodes ASCII STL
files with three's LoaderUtils.decodeText.  three removed that function, so
every ASCII STL preview failed while binary STLs (which never decode text)
still worked.  Override three-stdlib to 2.36.1, which ships its own decodeText.

Jest can now load three: a small transform converts three/three-stdlib ES
modules to CommonJS without babel-preset-react-app, whose class-properties
plugin rejects three's static class blocks at our pinned Babel version.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:55:55 -07:00
lrnselfreliance
20439437f8
Subtitle language-region suffixes are checked against the codes, not a 336,000-entry set (#947)
Every API process built a set of every language-region and language-auto
.srt/.vtt suffix at import: about 1 s and 73 MiB per process on a Pi 4.  A
suffix is now split into its language and region, and each is checked against
its own code set.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 17:19:14 -07:00
lrnselfreliance
ba9b47edf3
API startup no longer imports pymupdf (#946)
pymupdf 2.0 imports pymupdf.layout and onnxruntime, which every API start
paid for (about 2.5 s on a Pi 4) although only PDF extraction uses it.  It is
now imported when a PDF is extracted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:01:45 -07:00
lrnselfreliance
cafa2a2279
Workers finalize the signal router once at startup, not once per route (#945)
sanic-ext adds the automatic HEAD and OPTIONS routes one at a time in a
before_server_start listener.  Each route re-finalized the already-finalized
signal router (Sanic.amend), and each finalize registered one more
`sanic.__signal__` route, so a worker rebuilt an ever-growing router about 245
times.  The signal router is now left unfinalized while sanic-ext adds those
routes and finalized once afterwards.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:50:10 -07:00
Roland
89b84c9765
Bump version to 0.30.5-beta 2026-11-02 20:12:14 -07:00
lrnselfreliance
479aa2254e
A Zim search without a search string or tag names is a 400, not a 500 (#944)
Both Zim search endpoints handed an empty request to the worker-job builder,
whose RuntimeError surfaced as a 500 "Unexpected error" with a traceback.  The
request is the client's mistake: the handlers now raise ValidationError, as the
other endpoints do, before any work is scheduled.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:55:47 -07:00
lrnselfreliance
87137cc40c
Zim names with a re-release letter are parsed, and an undated Zim no longer warns at every start (#943)
The kiwix naming scheme is {name}_{YYYY-MM}.zim, and a re-release in the same
month carries a letter: 3dprinting.stackexchange.com_en_all_2026-07a.zim.
parse_name rejected the letter, so such a Zim was never checked for newer
releases and zims.yaml keyed it by its full stem instead of its name.  The
letter is now accepted and folded into the day of the month-precision date, so
a re-release orders after the plain release wherever dates are compared.  The
pattern also anchors the ".zim" extension instead of matching any character.

A Zim that is not kiwix-named (edibleplantdb.zim) has no date to compare and is
simply not checked.  find_outdated_zim_files logged that at WARNING for every
such file at every start; it is a DEBUG line now.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:36:07 -07:00
lrnselfreliance
9797af157a
Removing a duplicate video keeps the Download that is finishing (#942)
* A refresh that finishes before tags.yaml is imported no longer reports a failed save (#941)

Every refresh fires save_tags_config at its cleanup.  In the first seconds
after a start, a refresh can finish before the startup import of tags.yaml,
which on a rebuilt database happens while the first refresh runs.  The dump
then asked ConfigFile.save to write a database that held no tags over the
file; the save guard refused, as it must, but it did so with an ERROR traceback
and a "Failed to save config: config/tags.yaml" event for the user, although
nothing was lost: the import that followed read the file and recreated the tags.

save_tags_config now skips the dump while the config has not been imported and
says so at WARNING.  The file stays the source of truth until the import.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* Removing a duplicate video keeps the Download that is finishing

After a video download, Video.delete_duplicate_videos removes the other copies
of the same video, which happens when the file name changed (a renamed file, a
new title).  Each copy was deleted through FileGroup.delete, which also deletes
the Download of the copy's URL and, without one, adds the URL to the skip list.
The duplicates share that URL with the video just downloaded, so the Download
still running was deleted from under its worker: the worker read its row again
to record the result and failed with NoResultFound ("Unexpected error"), and the
download's record was gone.

FileGroup.delete and Video.delete take keep_download; the duplicate cleanup
passes it, so the URL's Download and the skip list are left alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Only a duplicate at the downloaded URL keeps its Download

delete_duplicate_videos also finds copies by source_id, and those can sit under
another URL.  Such a copy is deleted as any video is, with that URL's Download
or a skip-list entry; only a copy that shares the URL being downloaded leaves
the finishing Download alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:18:46 -07:00
lrnselfreliance
0aa4019f86
A refresh that finishes before tags.yaml is imported no longer reports a failed save (#941)
Every refresh fires save_tags_config at its cleanup.  In the first seconds
after a start, a refresh can finish before the startup import of tags.yaml,
which on a rebuilt database happens while the first refresh runs.  The dump
then asked ConfigFile.save to write a database that held no tags over the
file; the save guard refused, as it must, but it did so with an ERROR traceback
and a "Failed to save config: config/tags.yaml" event for the user, although
nothing was lost: the import that followed read the file and recreated the tags.

save_tags_config now skips the dump while the config has not been imported and
says so at WARNING.  The file stays the source of truth until the import.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:12:41 -07:00
lrnselfreliance
871a26324d
Startup no longer renews failed once-downloads (#940)
retry_downloads_on_startup set pending, deferred and failed Downloads back to
"new".  A failed once-download has already given up: its downloader raised
UnrecoverableDownloadError or it reached its attempt limit.  Renewed at every
boot, it only failed again at once and logged a traceback-level warning each
time.  Startup now renews only the pending and deferred Downloads, which were
interrupted or are waiting for their next try.  The user's Retry button still
renews failed Downloads and resets their attempts.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:36:57 -07:00
lrnselfreliance
c78378d4f5
The Zim list reads each Zim's metadata in the libzim worker instead of opening it in the API (#939)
GET /api/zim went through Zims.get_all, which opened every Zim's Archive in the
API process, and then Zim.__json__ opened each one again for its metadata.
python-libzim holds the GIL and a cold open of a large Zim takes seconds, so
loading the Zims page stalled that API worker for every other request (27.8 s
cold against 0.75 s warm for 44 Zims).  #928 moved estimates and searches to the
worker process; the list now uses the same worker for metadata, and a Zim the
worker cannot read is left out as before.  count_tagged_zim_entries shares the
new readable_zims helper.  Zims.get_all had no other callers and is removed.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:19:29 -07:00
lrnselfreliance
b8122a4637
The tail-decode check ignores the h264 decoder's "mmco: unref short failure" note (#938)
After a download, ffmpeg decodes the last 30 seconds of the video and any stderr
line not on the ignore list marks the video incomplete.  Seeking into an h264
stream can make the decoder print "mmco: unref short failure": a note about
reference-picture marking for frames before the seek point.  The frames decode
and ffmpeg exits 0, but the verdict was "Video was incomplete", so the download
was deferred and the same sound video was downloaded again on every retry.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:00:03 -07:00
Roland
6c8ea61eaa
Bump version to 0.30.4-beta 2026-11-02 16:55:04 -07:00
lrnselfreliance
2991f737da
test_get_custom_videos_directory attaches the shared context it needs (#937)
Setting a config field calls ConfigFile.update(), which takes
api_app.shared_ctx.config_update_lock.  The test did not use async_client,
so the lock only existed when another test in the same process had already
attached the shared context; running first in a worker, it failed with
"'SharedContext' object has no attribute 'config_update_lock'".

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 16:31:55 -07:00
dependabot[bot]
662a41afc3
build(deps): update pytest-asyncio requirement in /controller (#926)
Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
- [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases)
- [Commits](https://github.com/pytest-dev/pytest-asyncio/compare/v1.4.0...v1.4.1)

---
updated-dependencies:
- dependency-name: pytest-asyncio
  dependency-version: 1.4.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:05:36 -07:00
lrnselfreliance
31b47dacf8
The inventory legacy-date repair no longer takes the save lock during import (#915)
MultiFileConfig.initialize() imports every file while holding the shared save
lock, which is not reentrant. The repair added in #901 took that lock again from
inside import_all, so any WROLPi whose inventory still held epoch-second dates
hung every API process at startup.

The repair now only rewrites the in-memory entities during import and records
which inventories changed. import_inventories_config() writes those files once
it runs, outside the lock. A regression test runs initialize() against an
epoch-date file with a timeout.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:58:42 -07:00
dependabot[bot]
7f275284a1
build(deps): bump query-string from 9.5.1 to 10.0.0 in /app (#917)
Bumps [query-string](https://github.com/sindresorhus/query-string) from 9.5.1 to 10.0.0.
- [Release notes](https://github.com/sindresorhus/query-string/releases)
- [Commits](https://github.com/sindresorhus/query-string/compare/v9.5.1...v10.0.0)

---
updated-dependencies:
- dependency-name: query-string
  dependency-version: 10.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:53:24 -07:00
dependabot[bot]
f782f347d1
build(deps): bump qs and body-parser in /app (#891)
Bumps [qs](https://github.com/ljharb/qs) and [body-parser](https://github.com/expressjs/body-parser). These dependencies needed to be updated together.

Updates `qs` from 6.15.3 to 6.16.2
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.15.3...v6.16.2)

Updates `qs` from 6.16.0 to 6.16.2
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.15.3...v6.16.2)

Updates `body-parser` from 1.20.6 to 1.20.9
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/1.20.9/HISTORY.md)
- [Commits](https://github.com/expressjs/body-parser/compare/1.20.6...1.20.9)

---
updated-dependencies:
- dependency-name: qs
  dependency-version: 6.16.2
  dependency-type: indirect
- dependency-name: qs
  dependency-version: 6.16.2
  dependency-type: indirect
- dependency-name: body-parser
  dependency-version: 1.20.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:53:19 -07:00
dependabot[bot]
ab6f17fb5b
build(deps): bump postcss-selector-parser from 6.0.11 to 6.1.5 in /app (#892)
Bumps [postcss-selector-parser](https://github.com/postcss/postcss-selector-parser) from 6.0.11 to 6.1.5.
- [Release notes](https://github.com/postcss/postcss-selector-parser/releases)
- [Changelog](https://github.com/postcss/postcss-selector-parser/blob/6.1.5/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss-selector-parser/compare/v6.0.11...6.1.5)

---
updated-dependencies:
- dependency-name: postcss-selector-parser
  dependency-version: 6.1.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:53:14 -07:00
dependabot[bot]
7487cb07a1
build(deps): bump fast-uri from 3.1.8 to 3.1.9 in /app (#893)
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.8 to 3.1.9.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](https://github.com/fastify/fast-uri/compare/v3.1.8...v3.1.9)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:53:09 -07:00
dependabot[bot]
b9aae3f049
build(deps): bump the npm-dependencies group in /app with 6 updates (#916)
Bumps the npm-dependencies group in /app with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [@tabler/icons-react](https://github.com/tabler/tabler-icons/tree/HEAD/packages/icons-react) | `3.49.0` | `3.50.0` |
| [@testing-library/user-event](https://github.com/testing-library/user-event) | `14.6.7` | `14.6.8` |
| [react-router](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router) | `7.18.4` | `7.18.5` |
| [@types/lodash](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/lodash) | `4.17.25` | `4.17.27` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.6.4` | `26.6.6` |
| [webpack-dev-server](https://github.com/webpack/webpack-dev-server) | `5.2.6` | `5.2.7` |


Updates `@tabler/icons-react` from 3.49.0 to 3.50.0
- [Release notes](https://github.com/tabler/tabler-icons/releases)
- [Changelog](https://github.com/tabler/tabler-icons/blob/main/packages/icons-react/CHANGELOG.md)
- [Commits](https://github.com/tabler/tabler-icons/commits/v3.50.0/packages/icons-react)

Updates `@testing-library/user-event` from 14.6.7 to 14.6.8
- [Release notes](https://github.com/testing-library/user-event/releases)
- [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/user-event/compare/v14.6.7...v14.6.8)

Updates `react-router` from 7.18.4 to 7.18.5
- [Release notes](https://github.com/remix-run/react-router/releases)
- [Changelog](https://github.com/remix-run/react-router/blob/react-router@7.18.5/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/remix-run/react-router/commits/react-router@7.18.5/packages/react-router)

Updates `@types/lodash` from 4.17.25 to 4.17.27
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/lodash)

Updates `@types/node` from 26.6.4 to 26.6.6
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `webpack-dev-server` from 5.2.6 to 5.2.7
- [Release notes](https://github.com/webpack/webpack-dev-server/releases)
- [Changelog](https://github.com/webpack/webpack-dev-server/blob/v5.2.7/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-server/compare/v5.2.6...v5.2.7)

---
updated-dependencies:
- dependency-name: "@tabler/icons-react"
  dependency-version: 3.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@testing-library/user-event"
  dependency-version: 14.6.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: react-router
  dependency-version: 7.18.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/lodash"
  dependency-version: 4.17.27
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.6.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: webpack-dev-server
  dependency-version: 5.2.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:53:04 -07:00
dependabot[bot]
9c9b740d55
build(deps): update pytest requirement in /controller (#924)
Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.1.1...9.2.0)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.2.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:52:23 -07:00
dependabot[bot]
9af8ee89f5
build(deps): update certifi requirement (#919)
Updates the requirements on [certifi](https://github.com/certifi/python-certifi) to permit the latest version.
- [Commits](https://github.com/certifi/python-certifi/compare/2026.10.19...2026.10.23)

---
updated-dependencies:
- dependency-name: certifi
  dependency-version: 2026.10.23
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:52:18 -07:00
dependabot[bot]
c6d7084527
build(deps): update pymupdf requirement from >=1.28.2 to >=2.0 (#922)
Updates the requirements on [pymupdf](https://github.com/pymupdf/pymupdf) to permit the latest version.
- [Release notes](https://github.com/pymupdf/pymupdf/releases)
- [Changelog](https://github.com/pymupdf/PyMuPDF/blob/main/changes.txt)
- [Commits](https://github.com/pymupdf/pymupdf/compare/1.28.2...2.0)

---
updated-dependencies:
- dependency-name: pymupdf
  dependency-version: '2.0'
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:52:14 -07:00
dependabot[bot]
03a904d31a
build(deps): update fastapi requirement in /controller (#925)
Updates the requirements on [fastapi](https://github.com/fastapi/fastapi) to permit the latest version.
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](https://github.com/fastapi/fastapi/compare/0.142.2...0.162.0)

---
updated-dependencies:
- dependency-name: fastapi
  dependency-version: 0.162.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:52:10 -07:00
dependabot[bot]
00fdfad8e2
build(deps): update pydantic requirement in /controller (#927)
Updates the requirements on [pydantic](https://github.com/pydantic/pydantic) to permit the latest version.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md)
- [Commits](https://github.com/pydantic/pydantic/compare/v2.13.5...v2.14.1)

---
updated-dependencies:
- dependency-name: pydantic
  dependency-version: 2.14.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:52:05 -07:00
lrnselfreliance
5be899bf1f
Saving zims.yaml keeps the entries of Zims the database does not have yet (#935)
A rebuilt database imports zims.yaml before the refresh has modeled a single
Zim.  That import has no database version to compare with, so it saves the
config afterward, and the save wrote only the Zims in the database: none.  The
file's choices were erased and every Zim was then modeled with the default.

The dump now starts from the file's entries and overlays the database's, so a
choice survives until its Zim is modeled again, and a Zim in the database still
decides its own line.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:11:41 -07:00
lrnselfreliance
c0452e29e6
Hiding a Calibre book's cover and metadata uses the modeler's session (#936)
Doc._hide_calibre_files opened a second write session while the modeler held
the item's transaction.  SQLite has one writer, so the second session waited
out the 30 s busy timeout and failed with "database is locked"; the modeler
retried three times, so every Calibre epub cost about 98 s and stayed
unmodeled, and the cover.jpg / metadata.opf FileGroups it should have deleted
survived.  On a library of 1663 Calibre books the global refresh's modeling
phase would have taken about two days.

The nested session dates from before the SQLite cutover, when Postgres row
locks let it work.  The delete now runs in the caller's session.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:04:28 -07:00
lrnselfreliance
e6b7b9e5dd
Map downloads are checked against the hash in the signed manifest (#934)
* Map downloads are checked against the hash in the signed manifest

A meta4 signs only a bare hash, so any genuinely signed meta4 (an older version of a region, another region) passed
its own check.  The manifest now lists each file's sha256; it is signed as a whole, so the hash is tied to the URL.
The catalog copies each hash into its download's settings, and aria2c checks the file against it.  Downloads from a
manifest without hashes still require the file's signed meta4.

build_map_regions.py writes the hashes, and hash_map_manifest.py adds them to the published manifest from the
verified meta4s without rebuilding any maps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hash_map_manifest.py hashes each file's bytes instead of trusting its meta4's name

Only the bare hash in a meta4 is signed, so an older version's meta4 with its name rewritten (or terrain's, whose name
never changes) passed the name and duplicate-hash checks.  Each file is now hashed from a local copy of the published
files (--files) or downloaded from the CDN, and must match its signed meta4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 20:38:57 -07:00
lrnselfreliance
ee6c384dc6
A refresh keeps a FileGroup's tags and viewing state when its primary file changes (#932)
_upsert_files deleted the FileGroup of every companion file before
inserting, so a FileGroup whose primary file became a companion (foo.jpg,
then foo.mp4 arrives) was replaced by a new, untagged row.  A FileGroup
whose primary file was replaced (foo.txt deleted, foo.mp4 added) still owned
the (directory, stem), and the INSERT failed on the unique index.

Before inserting, the FileGroup owning each incoming (directory, stem) is
renamed to the new primary path, keeping its id, tags, collection
membership, model and viewing state.  When a different FileGroup owns the
primary path, the two collapse into the one dedupe prefers (modeled, then
tagged).  A companion's leftover FileGroup gives its tags, collection
membership and viewing state to its group's FileGroup before it is deleted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 20:21:34 -07:00
lrnselfreliance
851b12b060
A Zim's Search toggle is saved in zims.yaml so a rebuilt database or a newer Zim keeps it (#931)
The toggle was only the zim.auto_search column, which defaults to on.  The
database is rebuilt from files and configs, and a Zim's newer version is a new
row, so every rebuild and every subscription update turned search back on for
every Zim.  On a large library the global search then searched all of them.

zims.yaml records auto_search for each Zim, keyed by the Zim's name without
its date, so a newer version of a Zim is the same Zim.  A Zim the file does not
name is searched (the default stays on).  The config is imported on startup
with the usual version gating, saved when the toggle changes, and a newly
modeled Zim takes its setting from it.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:09:52 -07:00
lrnselfreliance
6fc147ccdf
A refresh re-indexes a FileGroup whose files changed even when their count did not (#933)
_upsert_files kept a FileGroup's files and indexed state when the size,
newest modification time and number of files were unchanged.  A file that
replaced another (foo.png for foo.jpg, same size, older than foo.mp4) left the
old file in `files` and the FileGroup indexed.

The upsert now compares the sorted file names.  FileGroups that still store
absolute paths (the old format) compare by name too, so they are not
re-indexed when nothing changed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:04:32 -07:00
lrnselfreliance
21006eae09
Map files are never downloaded without a signed hash (#930)
* Map files are never downloaded without a signed hash

A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later)
instead of downloading with TLS as its only check.  The search index downloader had no fallback branch at all, so it
downloaded with no hash whenever the .meta4 was missing.

A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when
the meta4's unsigned name differs from the URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* aria2c checks map files only against the signed hash, and a rejected file is never kept

Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4.  A sha-512 of a tampered
file added beside the signed sha-256 made aria2c accept the tampered file.  aria2c now gets a meta4 built from the
signed hash and the download URL alone.

A rejected download was left in the map directory under its final name, where the next attempt took it as already
downloaded.  Map files now download into map/.partial and move into place only once aria2c accepts them; a file that
fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Only the map file aria2c checked is moved into place

download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c
saves and checks the file under the name in the meta4 WROLPi writes.  When they differed, an unchecked file (such as
an interrupted download's leftover in map/.partial) could be moved into place.  The meta4 name now comes from the
URL, must be a plain file name, and any other reported path fails the download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
lrnselfreliance
bfe7b8afbd
Zim estimates and searches run in a libzim worker process so a cold Zim cannot stall the API worker (#928)
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker

The global search estimates every auto-search Zim on each keystroke.  On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it.  Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.

Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them).  It reads the Zim paths and search string on stdin and
writes the counts on stdout.  A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Zim searches run in the libzim worker process too

The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.

The estimate script is now a general libzim worker with estimate, search and
entries jobs.  A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop.  Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker

Review of the earlier commits found two gaps.  Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:02:26 -07:00
lrnselfreliance
9c258b41ee
A refresh groups each stem's files together even when another stem sorts between them (#929)
group_files_by_stem and chunks_by_stem assumed a stem's files are adjacent
once sorted.  They are not: foo.f137.mp4 (stem foo.f137) sorts between
foo.en.vtt and foo.mp4 (stem foo), and foo.tar.gz between foo.jpg and
foo.txt.  _upsert_files then built two FileGroups for one (directory, stem)
and the refresh failed with "UNIQUE constraint failed: file_group.directory,
file_group.stem".  A chunk boundary could split a stem the same way.

Both helpers now group by (directory, stem).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:38:48 -07:00
lrnselfreliance
8edace08bd
Python packages install from hash-locked requirements; yt-dlp still floats (#913)
* Python packages install from hash-locked requirements; yt-dlp still floats

Every pip install trusted PyPI over TLS: requirements.txt pins only top-level versions
(some with >=), the controller's are almost all >=, and dependencies were never pinned, so
a forged certificate (as the eMudhra key compromise allowed) could change code in the API,
the controller, or the archive container.  scripts/lock_requirements.sh generates a
universal, hash-locked .lock beside each requirements.txt, and
install_python_requirements.sh installs it with --require-hashes at every site.  yt-dlp is
left out of the lock and upgraded afterwards so devices still get YouTube fixes; its
dependencies are locked.  The unpinned `pip install --upgrade pip` steps are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The unhashed yt-dlp install cannot replace locked packages, and nothing builds from source

Review of #913 found two holes.  The yt-dlp step ran without --no-deps, so pip's
only-if-needed strategy would still fetch an unhashed dependency whenever yt-dlp's metadata
asked for a version the lock did not install.  And srt is published only as a source
archive, whose build installed setuptools with hash checking off on every install.

The yt-dlp step now uses --no-deps (its dependencies come only from the lock), both steps
are retried as the image builds did before, and installs and locks are wheels-only.  srt
3.5.3 is vendored unmodified as wrolpi/vendor/srt.py (MIT), verified against the sdist
hash the lock pinned; webvtt-py's SRT reader rejects WROLPi's own test file, so it is not
a replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:00:39 -07:00
Joel Shearon
fe85815c35
Format inventory expiration dates instead of epoch timestamps (#901)
Legacy inventory items stored date fields as Unix epoch seconds. The table showed the raw numbers, and sorting and expired highlighting did not treat them as dates.

- Frontend parses date values as YYYY-MM-DD or legacy epoch seconds; display, mobile view, export, search, sort and the expired check all use calendar dates with no timezone shift.
- The legacy migration converts epoch seconds to YYYY-MM-DD, and import repairs already-migrated files once.
- Jest runs with a pinned timezone; the expiry tests pass under any process timezone.

Closes #41

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 12:27:21 -06:00
lrnselfreliance
d69f8317aa
serve, single-file and readability-extractor install from a committed lockfile (#912)
They were installed with `npm i -g`, which pinned only the three top-level versions and
resolved their 156 dependencies fresh from the registry, so a forged TLS certificate (as
the eMudhra key compromise allowed) could change code WROLPi runs on every archive.
npm-tools/package-lock.json pins every package with its sha512, and install_npm_tools.sh
runs `npm ci` and links the commands into /usr/local/bin for every install path, upgrades,
both images and the archive container.  The app container's unpinned, unused global serve
is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:39:08 -06:00
lrnselfreliance
ecdad6c3e0
Caddy's apt repository is trusted through the key committed with WROLPi (#911)
The install scripts, the nginx-to-Caddy upgrade, and the Pi image downloaded Caddy's apt
signing key and source list over TLS and trusted them as-is, so a forged certificate could
serve a key that signs other caddy packages.  add_caddy_apt_repo.sh installs the key the
ISO already uses (fingerprint 65760C51EDEA2017CEA2CA15155B6D79CA56EA34) as an armored
keyring and names it in the source's signed-by.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:22:19 -06:00
lrnselfreliance
bde1209c63
An upgrade refuses to roll back to an older commit of the current branch (#910)
* An upgrade refuses to roll back to an older commit of the current branch

upgrade.sh verifies the fetched commit's signature, but a signature proves a commit is
WROLPi's, not that it is the newest; an intercepted fetch could serve an old signed
release and the forced checkout would roll the device back to it.  refuse_rollback stops
when the target is an older commit of the branch the device is on (a detached HEAD from a
release image counts as that branch).  Branch switches, the same commit, and history the
device does not contain (Debian 12 devices on debian12-final) are allowed, and -d allows
a deliberate downgrade; it reaches the upgrade service through the env file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An upgrade also refuses a signed commit with a lower version from other history

Ancestry alone only caught older commits in the device's own history; any other signed
commit, such as debian12-final or an old experimental branch, passed.  refuse_rollback
now also compares wrolpi/version.txt, which is inside the signed tree, and refuses a lower
or missing version.  A force-pushed branch at the same or a higher version still upgrades.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tests pin the two parts of refuse_rollback that only one rule catches

An earlier commit of the same version is refused only by the ancestry check, and
0.30.10-beta after 0.30.3-beta only orders correctly with sort -V; the fixture's
single-digit versions passed with either part removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 09:33:25 -06:00