Compare commits

..

155 commits

Author SHA1 Message Date
lrnselfreliance
bc58a5cc35
Save download_manager.yaml after new Downloads are committed (#882)
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
create_downloads activated save_downloads_config right after its flush, before the caller committed.  The
switch runs in the perpetual process, which cannot see uncommitted Downloads, so a save that ran first wrote
the file without them and nothing saved them later.  The switch is now activated when the session commits, and
not at all when it rolls back.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 20:32:18 -06:00
lrnselfreliance
eb82489426
Keep playlists.yaml in shared memory so the perpetual process can save it (#881)
playlists_config was the only config not attached to a shared_ctx dict.  Switches save configs in the perpetual
process, which never imports the files, so its playlists_config kept the default version 0.  While the database had
no version row for playlists.yaml, the dump wrote version 1 and `save` refused to overwrite the newer file; the
refused save never recorded the row, so every later save was refused too, and new playlists never reached the
config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:57:29 -06:00
lrnselfreliance
ae97727b98
Save channels.yaml with the database's version (#880)
save_channels_config wrote the Channels with `update()`, which keeps the version the config already had in
memory.  After a Channel changed, the database's version was ahead of the saved file's, so the next import
treated the file as older and saved the database over it: an edit made to channels.yaml in the meantime was
lost.  It now saves with `dump_config`, which writes the database's version, as every other database config
does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:39:45 -06:00
lrnselfreliance
d16bac29e8
Renaming a Tag commits its Download renames with the request (#879)
* Renaming a Tag commits its Download renames with the request

update_tag renamed Downloads in a background task using the request's
session, after RequestDB.write() had committed.  Its flush was never
committed, so the rename was lost, and the open write transaction held
SQLite's write lock: every later write failed with "database is locked"
until the API restarted.  Only the Channel moves stay in the background.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Save the Downloads config after a Tag rename commits; scan Downloads only on a rename

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:34:34 -06:00
lrnselfreliance
fffb2ed981
Link a Download to its domain by its destination column (#878)
`domain_download_links` read the destination only from a Download's settings, but the destination is now a column
(the config import moves an old settings destination into it), so no Download was linked to a domain by its
directory.  Read the column, and the settings for an old Download, as channels already do.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:21:55 -06:00
lrnselfreliance
c577cae9d8
Renaming a Tag renames it in Download settings (#877)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
update_tag never renamed settings.tag_names, and its channel_tag_name
edit mutated the JSON in place, so it was never saved.  The old name
stayed, and a Download that later found a new Channel failed (UnknownTag).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:34:47 -06:00
lrnselfreliance
79526afebe
A malformed config entry is skipped and counted instead of failing the import (#875)
* A malformed config entry is skipped and counted instead of failing the import

An entry that is not a mapping (or whose key is not text, or a list that is not a list) made the whole file fail to
import in playlists.yaml, repos.yaml, download_manager.yaml and tags.yaml (tag_files/tag_zims), and a non-text name
did in domains.yaml.  Each is now skipped and counted in import_skipped, so the config is not saved over it, and the
rest of the file imports.  A Channel whose `downloads` is not a list keeps its Downloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An unreadable entry keeps no row and does not make a section present

The key of an entry too malformed to name a row is now `Unreadable(position)`.  It is still counted in
import_skipped, but `present` ignores it, so a section whose entries are all unreadable removes nothing (as an empty
section), instead of deleting every row.  A playlist whose `items` is not a list keeps its items, and a Channel
Download whose url is not text is counted (its Channel keeps its Downloads) instead of failing the import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:28:34 -06:00
lrnselfreliance
0dd61b8e64
Strip whitespace from Tag names (#876)
* Tag names have leading and trailing whitespace stripped

A padded name now finds the existing Tag instead of creating a
near-duplicate, and a whitespace-only name is rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Migration strips whitespace from Tag names and merges collisions

Tags whose names match once stripped are merged (tagged files, Zim
entries, Collections); the first color by id is kept.  Downloads and the
configs that name Tags are rewritten too, so the next import is in step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tags.yaml keeps the database's color for merged Tags

The file was merged in sorted key order, not by Tag id, and the next
import copies the file's color over the database's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:22:16 -06:00
lrnselfreliance
c3e25b84d6
Configs without database rows read and validate only; imports run off the event loop (#874)
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request).  The
per-config overrides that only set successful_import are removed.

wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.

The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:17:31 -06:00
lrnselfreliance
ee10b34b15
recently_viewed.yaml imports by diffing (#873)
* recently_viewed.yaml imports by diffing

recently_viewed.yaml declares one section, its files keyed by
media-relative path, comparing viewed, progress and position.  The
database's rows are the window the config saves (the newest
RECENTLY_VIEWED_LIMIT views, inside the media directory), so an
unchanged import writes nothing.

A file removed from the config has its viewing history cleared
(decision 6), but only when its view is at least as recent as the
config's oldest entry: an older view fell off the end of the window.  A
malformed entry keeps its file's history and is not counted as skipped
(that would stop every recently viewed save).  Without deletes, progress
written after the config is kept, and a newer plain view keeps its
`viewed` but takes the config's position (as before).  prepare finds the
files of new entries on disk before the write; a deleted file's history
goes with it.

Moving a viewed FileGroup counts as a recently_viewed.yaml change (ORM,
the bulk move, and a refresh that changes the primary file).  The media
directory checks are batched (each resolved the media directory again).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* recently_viewed.yaml: the no-delete merge covers files outside the window

Without a database version, newer history in the database wins (its
progress is kept; a newer plain view keeps its `viewed` and takes the
config's position).  That applied only to files among the newest views;
a file listed in the config that had fallen out of them was "added",
and its stale entry replaced the newer history.  prepare now reads the
database's history of added files and applies the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 14:39:45 -06:00
lrnselfreliance
315b60e9eb
channels.yaml imports by diffing (#872)
* channels.yaml imports by diffing

channels.yaml declares two sections: Channels keyed by directory
(comparing name, URL, Tag, file format, and the source id and flags,
which keep the database's value when missing), and the Downloads listed
under each Channel, keyed by (directory, URL); an existing Download keeps
its own frequency.  An unchanged import writes nothing (it ran about a
thousand queries and committed several times).

Removed Channels are deleted with their Collection (and Downloads);
a Download removed from a Channel is deleted.  A channel Collection
without a Channel is deleted unless listed (then it gets its Channel).
A Channel whose directory changed, with the same name, is moved.  An
invalid entry is skipped and counted, and keeps its Channel (by name when
its directory is invalid) and that Channel's Downloads; so does an entry
without a list of Downloads.  Duplicate directories still refuse the
import.  Unknown Tags are dropped once tags.yaml is imported.

New Downloads are created in the import's one write (commit=False on
Channel.get_or_create_download, Download.delete and delete_with_videos).
After the import, Videos are claimed and source ids fetched only for new
and changed Channels; Downloads are linked without a query per Download,
and not while entries are skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* channels.yaml: moves, invalid Downloads, and saves after the write

- A moved Channel's Download diff is retargeted to its new directory:
  one its new entry lists is the same Download, one it does not is
  deleted in the same import (the old directory was treated as kept).
- An unreadable Download is counted as skipped, and its Channel keeps
  its Downloads (it was dropped, and the Channel's Download deleted).
- A new Download without a frequency is dropped before the write and
  counted (it re-opened a write on every import).
- No source id is fetched while entries are skipped (its save would be
  refused, leaving the database newer than the file); a fetched one is
  saved with dump_config, which records the database's version.
- Download.delete(commit=False) neither changes the skip list nor starts
  config saves, which would read the database before the import commits
  (and write a removed Download back into channels.yaml).  The import
  saves download_manager.yaml after its write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 13:13:42 -06:00
lrnselfreliance
57c2a29022
domains.yaml imports by diffing (#871)
* domains.yaml imports by diffing

domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format.  An unchanged
import writes nothing.

A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it.  An invalid entry (name
or directory) is skipped, counted, and its domain kept.  Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).

Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test_bookmarks_config_import starts the app like its siblings

It imports a config without `async_client`, so it fails (no
shared_ctx.configs_imported) whenever it is the first test on its
worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* domains.yaml: converge without a save; no links while entries are skipped

A save after the import is refused while an entry is skipped, so the
import must converge without one:
- a removed domain with Archives that is already reset is left as it is
  (prepare reads `import_db_rows`); before, every import reset it again;
- after prepare, the framework drops changed rows that now match the
  database (a dropped unknown Tag), so they are not written again.

Linking Downloads changes domains.yaml after its version is recorded;
while entries are skipped the save is refused, the database stays newer,
and the next startup would save over the skipped entries.  Linking now
waits for an import without skipped entries.

An entry that is not a mapping is skipped and counted, instead of failing
the whole file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 11:41:59 -06:00
lrnselfreliance
5cdb4aaac0
repos.yaml imports by diffing (#870)
* repos.yaml imports by diffing

repos.yaml declares one section, its Repos keyed by repo_url_key (another
URL of the same Repo is that Repo), comparing name, description,
directory, Tag, mode, branch, submodules, frequency and clone token (a
missing directory or clone token keeps the database's).  An unchanged
import writes nothing (every import rewrote every Repo and its
Download).

An invalid entry is skipped and counted, and its Repo is kept; before,
a Repo whose entry failed validation was deleted.  New Repos are created
inside the import's one write: recurring_download (via
get_or_create_download and _set_repository_download) takes commit=False,
and the URL is not looked up again for each new Repo (O(N^2)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* repos.yaml: a branch URL is keyed by its Repo's URL

file_rows keyed the raw URL, but stores parse_repo_url's URL, which
turns a forge's branch URL (GitHub /tree/<branch>, GitLab
/-/tree/<branch>) into the repo URL and a branch.  The entry looked like
a new Repo: its insert hit the existing Repo's name (rolling back every
import of that file), and an invalid such entry did not protect its Repo
from deletion.  Rows and invalid keys now use the parsed URL; a URL that
cannot be parsed is skipped by its raw key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 10:36:12 -06:00
lrnselfreliance
adbcf944a5
playlists.yaml imports by diffing (#869)
* playlists.yaml imports by diffing

playlists.yaml declares two sections: playlists, keyed by name, and
their items, keyed by what each points to (a file's or Zim's
media-relative path and entry, or a URL) with position and title
compared.  A playlist has at most one item per reference, so a moved
item changes only its position and keeps its row; an unchanged import
writes nothing (every import deleted and re-created every item).

prepare skips items whose file or Zim is not indexed before the write
(their place is kept), drops a Tag that no longer exists once tags.yaml
is imported (the config is then saved), and, without deletes, keeps the
database's order and adds the file's other items after the item that
precedes them.

Moving a FileGroup in a playlist counts as a playlists.yaml change (ORM,
the bulk move, and a refresh that changes the primary file).

media_relative moves to wrolpi.config_import for both configs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Playlist items: keep database order without deletes; empty lists delete

Without a database version, every playlist with an added or changed item
keeps the database's order (the file's new items go after their
predecessor; only titles come from the file), and a row whose merged
position and title equal the database's is not written.  Before, only
playlists that gained an item were merged, so a file that only reordered
(or lacked an item the database had) gave two items one position.

A section can say it is present without rows (`ConfigSection.present`):
a listed playlist with an empty items list has its items removed; an
empty or missing playlists list still removes nothing.  An item that
cannot be read, or repeats another, is counted in import_skipped, so the
config is not saved over it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 09:42:26 -06:00
lrnselfreliance
b67d51b69a
tags.yaml imports by diffing (#868)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* tags.yaml imports by diffing

tags.yaml declares three sections (Tags, tagged files, tagged Zim
entries), so its import reads the file once, diffs it against a snapshot,
and writes only the differences, with the file's version, in one write.
An unchanged import writes nothing.

A removed Tag is deleted with its tagged files and Zim entries; a
Collection using it loses only the Tag.  A removed tagged file or Zim
entry is untagged.  A tagged file whose Tag the config does not list is
skipped.  prepare finds files without a FileGroup on disk, skips missing
ones before the write, moves Zim entries of a replaced Zim to the newest
one, and maps a file named by a secondary member to its group; the
config is then saved with the database's paths.

Moving a tagged FileGroup counts as a tags.yaml change (ORM, the bulk
move, and a refresh that changes the primary file), so a restart before
the save does not untag it.

Sections can define `same` (a tagged file without a date matches any
date), and prepare can count skipped rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A removed Tag leaves no Download that would apply it

The tags import removes a deleted Tag's name from every Download's
tag_names and from the tag_names and channel_tag_name its settings pass
on (a change to download_manager.yaml, which is then saved).  Applying a
Tag that does not exist fails the download, and it is retried.  Once
tags.yaml is imported, a newer download_manager.yaml drops names no Tag
has, and is saved without them.

A replaced Zim's entry tags move to the newest Zim of the same parsed
name, by the date in its name (as migrate_outdated_zim_tags picks it),
not by name prefix and path order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 20:16:39 -06:00
lrnselfreliance
6434d5ca20
Database configs import by diffing; download_manager.yaml first (#867)
* Database configs import by diffing; download_manager.yaml first

Step 2 of the config import rework.  A database config can now declare a
ConfigSection for each kind of row it saves, and ConfigFile.import_config runs
a fixed sequence for it:

- read and validate the file once;
- decide from the versions (wrolpi.config_version) what the import may do;
- snapshot each section's rows as plain data keyed the way the file names them,
  and diff: added, changed, removed (none from an empty section, or without a
  current version);
- prepare (files, network) with no transaction open;
- write only the differences, and the file's version, in one short write.  A
  failure writes nothing.  If the database's version changed since the
  snapshot, the import decides again (and saves a newer database).

Subclasses override the section methods (file_rows, db_rows, upsert, delete)
and the prepare/after_import hooks.  The other database configs keep their
importers until their own steps.

download_manager.yaml is the first: an unchanged config imports with no write
(was 5,000 statements for 2,500 Downloads), new Downloads are one INSERT, and a
missing ZimSubscription is restored without rewriting the rest.  Settings equal
to the global defaults compare as unset, as they are saved.  A destination
written relative to the media directory is stored absolute.  Completed
once-downloads are never removed (they are not saved; #679).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync Zim subscriptions directly during the downloads import

A row now compares its subscription's (name, language), derived from the
URL, instead of whether one exists.  The import moves a subscription of
that name onto the written Download (a changed language or URL), removes
one the file no longer implies, and leaves the Download's own columns as
the file has them; change_download renamed the old Download's URL into
the one just inserted, and reset attempts and the downloaders.  A Kiwix
URL that names no known subscription is a plain Download instead of
failing the whole write.

prepare runs only when there are differences, and its docstring states
that it must be safe to repeat and to abandon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 18:49:24 -06:00
lrnselfreliance
946043450e
Database configs carry a version the database keeps too (#866)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Benchmark every database config import (#865)

An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs).  It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.

The baseline for the config import rework.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Database configs carry a version the database keeps too

Step 1 of the config import rework.  A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):

- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
  changed, in the same transaction.  Each config registers what it owns
  (register_ownership): models, optionally only some columns, optionally
  only some rows (a Collection's kind; a Download's Collection's kind).
  Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
  save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
  with deletes; file older (the database changed and was not saved before a
  restart) saves the database instead of undoing that change; no database
  row (a config this code never imported) imports without deletes.  The
  import records the file's version last.
- The Import button and backup imports apply the file whatever its version
  (import_forced).

Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config versions: keep unsaved rows after a no-delete import, count every owned write

Review fixes:

- An import without deletes (no database version yet) no longer records the
  file's version, which let the next startup delete the rows it had kept.
  The version stays unknown, and the database is saved into the file instead;
  a successful save records the version.  A save after a partial import is
  refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
  do not count: one that fails partway no longer looks newer than the file
  and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
  file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
  (collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
  Download, removing many items from a playlist.  Deleting Downloads by id
  counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
  Collections have items too).  Collection kinds are looked up once per flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config versions count only the columns each config saves

A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch.  Each
model now registers only the columns its config saves.

domains.yaml saves its domains' Downloads, which now count for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:58:18 -06:00
lrnselfreliance
b5fa59a5c4
Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs).  It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.

The baseline for the config import rework.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:02:34 -06:00
lrnselfreliance
a93630576f
Tests fail when a request writes outside request.ctx.db.write() (#864)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Tests fail when a request writes outside request.ctx.db.write()

A request's transaction begins deferred.  A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent.  #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.

Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test.  allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.

It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
  row's delete when the middleware's commit failed under contention.  It
  now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
  Session, after the request ended.  set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
  deferred get_db_session().  It now begins as a writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tasks don't inherit a declared write; tags import change backed out

From review of #864:

- background_task's task copied its creator's Context, including
  _write_intent when started inside get_db_session(commit=True) or
  RequestDB.write(), so the guard ignored the task's own writes for its
  lifetime.  The task now starts with no declared write.  That exposes
  fetch_channel_source_id (started by the channels config import): it read
  the Channel, asked the remote site for its source_id, then committed in
  the same deferred transaction.  It now reads, fetches with no
  transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
  walk, and its exit commit saved changes the need_commit gate used to
  drop.  Reverted to master; splitting its reads from a short write is its
  own change.  The two tests that run it through a request are marked
  allow_deferred_request_writes until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 17:09:39 -06:00
lrnselfreliance
604d467861
Renaming a directory with Collections waits for a concurrent writer (#863)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Remove a file's Tags Directory links when WROLPi deletes it (#861)

* Remove a file's Tags Directory links when WROLPi deletes it

When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not.  The link became the only copy, so the tags sync
refused to delete it and warned on every sync.  The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove Tags Directory links on path deletes; keep a valid poster's link

The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links.  It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.

A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Renaming a directory with Collections waits for a concurrent writer

POST /api/files/rename on a directory changed Collections on the request's
deferred transaction.  For a plain directory, that transaction was opened
before the FileWorker moved the files and wrote only after, so if anything
else wrote meanwhile the Collection update failed ("database is locked"):
the request returned 409 with the files already moved, and the nested
Collections still pointed at the old directory.  Renaming a Collection's
own directory failed the same way, before moving anything.

rename_directory now changes Collections in a `write` (request.ctx.db.write
from the endpoint), which begins as a writer.  Files are moved outside it;
nested Collections are read inside it, after the move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:42:34 -06:00
lrnselfreliance
9bbef50ca1
Remove a file's Tags Directory links when WROLPi deletes it (#861)
* Remove a file's Tags Directory links when WROLPi deletes it

When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not.  The link became the only copy, so the tags sync
refused to delete it and warned on every sync.  The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove Tags Directory links on path deletes; keep a valid poster's link

The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links.  It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.

A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:40:57 -06:00
lrnselfreliance
da3dadde07
Apply the offset in a SingleFile saved date, and repair misread Archive dates (#862)
* Apply the offset in a SingleFile saved date

SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700").  The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Repair Archive download datetimes misread from a SingleFile saved date

Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value.  A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read the whole SingleFile header; finish the date repair only when every Archive was read

The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT.  The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.

The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:34:29 -06:00
lrnselfreliance
1f3762ff20
Clear the FileWorker private queue between tests (#860)
The test harness resets the shared file worker public queue for each test,
but the FileWorker's private LifoQueue is per-process state that survived
from one test to the next. Tests that process only a count task leave its
chained refresh task queued, and it later ran inside an unrelated test.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 13:03:55 -06:00
dependabot[bot]
58192a7052
build(deps): bump dompurify from 3.4.16 to 3.4.17 in /app (#857)
* Distrust compromised eMudhra intermediates in certifi and Deno (#854)

* Distrust compromised eMudhra intermediates in certifi and Deno

The private keys of eight eMudhra intermediate CAs can be recovered from
their certificates, so anyone can mint a trusted certificate for any host.

* Require certifi>=2026.10.19; yt-dlp, requests and curl-cffi use its bundle,
  and 2026.10.19 removes the emSign ECC Root CA - G3 root they chain to.
* Set DENO_TLS_CA_STORE=system for the API (systemd + Docker) and the archive
  container so Deno follows the apt-patched system store instead of roots
  that only change with a Deno upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep Deno's bundled roots in the archive container

browserless/chrome:1.61.1 is Ubuntu 22.04 whose newest ca-certificates is
from 2023, older than Deno 2.8.3's bundled roots.  The system store only
becomes the better choice once the archive base image is upgraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* build(deps): bump dompurify from 3.4.16 to 3.4.17 in /app

Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.16 to 3.4.17.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.16...3.4.17)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.17
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-23 12:33:35 -06:00
dependabot[bot]
dd2e778fea
build(deps): bump launch-editor from 2.14.1 to 2.14.2 in /app (#855)
Bumps [launch-editor](https://github.com/vitejs/launch-editor) from 2.14.1 to 2.14.2.
- [Commits](https://github.com/vitejs/launch-editor/compare/v2.14.1...v2.14.2)

---
updated-dependencies:
- dependency-name: launch-editor
  dependency-version: 2.14.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-23 12:33:26 -06:00
lrnselfreliance
74e4654a26
Refresh keeps a duplicate FileGroup's tags when collapsing duplicates (#859)
When refresh finds FileGroups sharing (directory, stem) it keeps the highest-id row and deletes the
rest.  Deleting cascades, so the loser's tags and collection membership were silently lost, and the
next tags config save persisted the loss.  Before the duplicate is deleted, its tags and collection
membership now move onto the FileGroup that survives it, reusing the merge the stem migration uses.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:33:11 -06:00
Roland
d992dc99d2
Ignore the null muxer's duplicate-DTS warning in the tail-decode check (#858)
ffmpeg exits 0 but prints "non monotonically increasing dts to muxer"
from the null output when two frames share a timestamp, collapsed into
"Last message repeated N times" when piped.  Any stderr failed the
completeness check, so such videos (common on X) were rejected as
incomplete on every retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:33:23 -06:00
lrnselfreliance
9ccf19b82b
Distrust compromised eMudhra intermediates in certifi and Deno (#854)
* Distrust compromised eMudhra intermediates in certifi and Deno

The private keys of eight eMudhra intermediate CAs can be recovered from
their certificates, so anyone can mint a trusted certificate for any host.

* Require certifi>=2026.10.19; yt-dlp, requests and curl-cffi use its bundle,
  and 2026.10.19 removes the emSign ECC Root CA - G3 root they chain to.
* Set DENO_TLS_CA_STORE=system for the API (systemd + Docker) and the archive
  container so Deno follows the apt-patched system store instead of roots
  that only change with a Deno upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep Deno's bundled roots in the archive container

browserless/chrome:1.61.1 is Ubuntu 22.04 whose newest ca-certificates is
from 2023, older than Deno 2.8.3's bundled roots.  The system store only
becomes the better choice once the archive base image is upgraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 08:54:57 -06:00
lrnselfreliance
cbf69e3013
Repo changes wait for a concurrent writer (#853)
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Adding, importing, changing, deleting and updating a Repo read, then
wrote, in the request's deferred transaction, so they failed with a 500
("database is locked") when another connection wrote in between.  They
now write inside request.ctx.db.write().

Importing reads the clone with git and walks it for its size; those run
before the write transaction, so it is not held for them.  Everything
that can refuse the import still runs before the clone is changed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:50:09 -06:00
lrnselfreliance
bcf977cfae
which() prefers the executable found in $PATH (#852)
which() kept looking after shutil.which found a name in $PATH, so a later
fallback path (e.g. '/usr/bin/git' in which('git', '/usr/bin/git'))
always replaced it.  Its docstring says $PATH comes first; stop at the
first match.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:49:41 -06:00
dependabot[bot]
82cc824fb7
build(deps): bump markdown-it from 15.0.2 to 15.0.3 in /app (#844)
Bumps [markdown-it](https://github.com/markdown-it/markdown-it) from 15.0.2 to 15.0.3.
- [Changelog](https://github.com/markdown-it/markdown-it/blob/master/CHANGELOG.md)
- [Commits](https://github.com/markdown-it/markdown-it/compare/15.0.2...15.0.3)

---
updated-dependencies:
- dependency-name: markdown-it
  dependency-version: 15.0.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-21 16:49:27 -06:00
dependabot[bot]
72bea377ee
build(deps): bump pillow from 12.3.0 to 13.0.0 (#843)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.3.0 to 13.0.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/12.3.0...13.0.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 13.0.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-21 16:49:13 -06:00
lrnselfreliance
3946e315fc
Channel and map subscription changes wait for a concurrent writer (#851)
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between.  They now write inside
request.ctx.db.write().

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:48:52 -06:00
lrnselfreliance
cb830594ae
Zim tagging, auto-search and subscriptions wait for a concurrent writer (#850)
POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between.  They now write inside request.ctx.db.write().

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:48:39 -06:00
lrnselfreliance
212366c2eb
Tagging, untagging and deleting files wait for a concurrent writer (#849)
POST /api/files/tag, /untag and /delete_groups read, then wrote, in the
request's deferred transaction.  SQLite fails that lock upgrade instantly
("database is locked") when another connection wrote in between.  Tag and
untag returned a 500; delete_groups returned 204 after FileGroup.delete had
unlinked the files, then its commit failed in the response middleware, so
the FileGroup row outlived its files.  They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:48:27 -06:00
lrnselfreliance
a40b69cee9
Download and tag requests wait for a concurrent writer (#848)
Deleting, restarting, clearing and batch-editing Downloads, and creating,
renaming and deleting Tags, read then wrote in the request's deferred
transaction.  SQLite fails that lock upgrade instantly ("database is
locked") when another connection wrote in between, so these failed with a
500 while a download or refresh was writing.  They now write inside
request.ctx.db.write(), which begins as a writer and lets busy_timeout
absorb the contention.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:48:16 -06:00
lrnselfreliance
22afb81879
Reorganize conflicts and no-move reorganize wait for a concurrent writer (#847)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Follow-up to #846, from review:

- The reorganize conflicts endpoint read the preview, ran ffprobe, then
  flushed and committed in the request's deferred transaction.  It now
  reads, runs ffprobe through get_or_create_ffprobe_json (no session),
  and stores the results in one short request.ctx.db.write().
- Its ffprobe step never ran: it matched model_type == 'channel', but a
  conflicting video's model_type is its FileGroup's model, 'video'.
- execute_reorganization's no-move path stored the new file_format with
  session.commit() on the caller's deferred session, from the reorganize
  endpoint and from the batch job.  The scan stays a read; the store is a
  short write (request.ctx.db.write() in a request, its own connection
  otherwise).  Called without a session it now opens a read session, so
  that write does not wait on its own caller's lock.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 17:16:17 -06:00
lrnselfreliance
97d7eb260c
Collection create/update/tag/delete wait for a concurrent writer (#846)
These endpoints read, then wrote, in the request's deferred transaction.
SQLite fails a deferred transaction's lock upgrade instantly ("database
is locked") when another connection wrote in between, so a collection
change during a refresh or download could fail with a 500.  They now
write inside request.ctx.db.write(), which begins as a writer and lets
busy_timeout absorb the contention; errors are mapped outside the block
so a 4xx rolls back.

Fixes #674

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:42:14 -06:00
lrnselfreliance
2f5d88ad4e
Live USB: allow password SSH (#845)
live-config's openssh-server component rewrites sshd_config to
"PasswordAuthentication no" on every live boot, so the live session only
accepted SSH keys.  An sshd_config.d drop-in sets it back to yes; sshd uses
the first value it reads and sshd_config includes the drop-ins first, so the
drop-in wins.  The login is user / live (live-config's default), now noted
in INSTALL.md.  check-iso.sh fails the build if the drop-in is missing.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:29:40 -06:00
dependabot[bot]
4becad7d36
build(deps): bump follow-redirects from 1.16.0 to 1.16.1 in /app (#842)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.16.0 to 1.16.1.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-version: 1.16.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:05:21 -06:00
lrnselfreliance
3a9beae119
Debian ISO: WROLPi banner in the graphical installer (#841)
The installer opened with Debian's "debian 13" banner.  Debian's
derivative guidelines say derivatives must not be presented as Debian,
and its trademark policy forbids implying endorsement, so the installer
now shows a WROLPi banner in the boot splash's style (pie logo, WROLPi,
"Off-grid library", purple accent).

The banner replaces rootskel-gtk's logo_debian.png and logo_debian_dark.png
(logo_installer*.png link to them) via config/includes.installer, which
live-build adds to the installer initrd.  It is 800x75 like Debian's; the
installer stretches its edge columns to the screen width, so the edges are
solid color.  check-iso.sh fails the build if the graphical installer's
initrd does not carry it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:00:09 -06:00
lrnselfreliance
1e89cb55f6
Fresh Live USBs don't report an ownership update (#840)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
The library-ownership migration compared the owner of
/mnt/persistence/wrolpi after creating it, so on a new persistence
partition the root-owned directory always looked like an old library and
the first-boot dialog said "Updated the library's ownership for this
WROLPi version".  Only re-own a library that already existed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:52:26 -06:00
lrnselfreliance
22be73a1c9
Debian ISO: advertise wrolpi.local with Avahi (#839)
INSTALL.md and the onboarding wizard send people to wrolpi.local, but the
Debian image had no mDNS responder, so the name never resolved on a Debian
install or Live USB.  Raspberry Pi OS ships Avahi; the ISO now does too.

libnss-mdns lets the WROLPi's own desktop resolve .local names (apt
recommends are off in the build, so it is listed explicitly).  check-iso.sh
fails the build if either package is missing.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:39:33 -06:00
Roland
c6bdd759e4
Bump version to 0.30.1-beta
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 19:57:29 -06:00
lrnselfreliance
1f07b5f944
Readability extraction no longer takes minutes on image-heavy pages (#838)
* Readability extraction no longer takes minutes on image-heavy pages

SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.

Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output.  Applied in
both the archive container and the native (Raspberry Pi/Debian) path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Data URI placeholders cannot be forged by page text

The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times.  Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Data URI placeholders survive Readability's lazy-image cleanup

Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder.  The ~50 character placeholder triggered this, so those images
lost their inline data.  The token is now 160 hex characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:50:21 -06:00
lrnselfreliance
06e35e1095
Archive container reaps its zombie processes (#837)
The archive service is PID 1 in its container and never reaps orphans.
check_output kills each command's process group, so every archive leaves
chrome, cat, and node grandchildren behind as zombies.  A long-running
container accumulated hundreds of them, counting against its PID limit.

init: true runs tini as PID 1, matching the web container.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:10:17 -06:00
lrnselfreliance
c6613dfee0
Debian ISO: hostname defaults to wrolpi, boot menu times out (#836)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Debian ISO: hostname defaults to wrolpi, boot menu times out

The installer's hostname question defaulted to "debian", so an install that
kept the default was not reachable at wrolpi.local.  A preseed in the
installer initrd now defaults it to wrolpi; the question is still shown (and
a DHCP-provided hostname still wins).

The ISO's GRUB and ISOLINUX menus had no timeout, so a Live USB waited at the
boot menu forever after any reboot or power loss.  The menus now boot the
default entry ("Run WROLPi (live)") after 10 seconds.

check-iso.sh fails the build if either regresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Debian ISO: say what the wrolpi hostname is for

The image does not include avahi-daemon, so the hostname alone does not make
wrolpi.local resolve.  The default still names the TLS certificate, the
SMB/NetBIOS share, and the DHCP hostname a router may publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 06:59:21 -06:00
lrnselfreliance
baa86b4ad1
Upgraded Live USBs keep a writable library (#835)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
ISOs before wrolpi became a system account wrote the Live USB's library on
the persistence partition as UID 1000.  Upgrading such a USB with a newer
ISO (wrolpi-usb.sh upgrade) keeps the persistence partition, so the library
would belong to the live session's "user" and the services could not write
to it.  Bootstrap now re-owns the library once when its owner is not this
image's wrolpi.  Live USBs only: installed systems' drives are left alone.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:23:53 -06:00
lrnselfreliance
f644a80c9a
Debian installs create the user the installer asks for (#834)
The ISO's chroot hook created wrolpi as a login user (UID 1000, password
"wrolpi", full sudo, LightDM autologin).  The Debian installer creates the
user it asks for only when no UID 1000-59999 user exists, so it silently
skipped that user: the only login on an installed system was wrolpi/wrolpi,
with SSH and sudo.

wrolpi is now a system account with no password or login sudo, as on every
other WROLPi deployment.  The installer creates the user it asks for (with
sudo, no autologin).  The live session logs in as live-config's "user".
Desktop shortcuts moved to /etc/skel so every user gets them.

check-iso.sh now fails the build when the image has a UID 1000-59999 user.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:23:40 -06:00
lrnselfreliance
826f1cdc42
The drive wizard never offers an EFI system partition (#833)
On a Debian install the onboarding wizard offered the EFI system partition
(unlabeled vfat at /boot/efi) as the primary WROLPi drive; choosing it would
mount the boot partition over /media/wrolpi.  Candidates were filtered by
the "EFI" label only, and the Debian installer leaves the ESP unlabeled.

/boot/efi is now an excluded mountpoint, and partitions with the EFI system
partition type (GPT GUID or MBR 0xef, from lsblk PARTTYPE) are excluded even
when unmounted, e.g. on a second internal disk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:26:18 -06:00
lrnselfreliance
80678e1f6e
A library on the internal disk gets a database (#832)
"Run repair without external drive" kept the library on the internal disk,
but the API's unmounted-media guard refuses to create or use a database
beneath /media/wrolpi when it is not a mount.  After onboarding without a
drive the API restarted forever with no database, and downloads were
blocked as "media not mounted".

The Controller now records the choice as drives.library_on_internal_disk in
controller.yaml -- the only source of it.  The database guard and the
media_mounted check both read that marker, so require_media_mounted keeps
its meaning and needs no change.  Onboarding a drive later clears the
marker, so it is never copied onto the drive.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:15:33 -06:00
lrnselfreliance
ad56134851
install.sh installs on Debian 13 (#830)
install.sh only accepted Debian 12, which cannot run this version, and
rejected Debian 13.  It now requires Debian 13 or newer on both PCs and
Raspberry Pis.  The install scripts point to the drive wizard instead of
manual mounting.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:51:40 -06:00
lrnselfreliance
559bfebc56
Installed Debian systems keep Caddy (#831)
* Installed Debian systems keep Caddy

The Debian installer purges every package listed in the ISO's
live/filesystem.packages-remove.  live-build lists there every package
installed after the package-lists install pass, which includes anything a
chroot hook installs with apt -- and the hook installed Caddy that way.
An installed system therefore had no Caddy: wrolpi-bootstrap failed copying
the Caddyfile, the API/app/Kiwix were never enabled, and HTTPS never came up.

Caddy now comes from package-lists/, with its repository and signing key in
config/archives/ (installed into both the build chroot and the image).
check-iso.sh fails the build when the installer would remove any package,
and can be run by hand against any ISO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Debian ISOs install Caddy from Caddy's repository

Moving caddy into package-lists/ made apt read config/archives/ before
ca-certificates was installed, so the HTTPS fetch of Caddy's repository
failed certificate verification.  apt ignores a failed repository, and the
image silently got Debian's much older caddy 2.6.2 instead of 2.11.

ca-certificates is now part of the bootstrap, and Debian's caddy is pinned
to -1, so an unreachable Caddy repository fails the build and installed
systems never switch to Debian's caddy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:42:37 -06:00
lrnselfreliance
7aa7481a56
The hotspot starts on boot (#829)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Nothing started the hotspot since hotspot_on_startup moved out of the
API, and NetworkManager does not autoconnect the Hotspot connection.
The Controller now starts it once per boot unless hotspot.on_startup is
false; a legacy wrolpi.yaml hotspot_on_startup: false is migrated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:29:15 -06:00
lrnselfreliance
b83a9cf017
Onboarding a drive keeps a library on the internal disk (#828)
The onboarding wizard deleted /media/wrolpi/config before mounting a
drive, and the shadowed data check ignored config/.  A library kept on
the internal disk keeps wrolpi.db and its configs there, so onboarding
a drive later deleted them without warning.  Only a config/ holding
nothing but the ssl/ CA is now treated as transient.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:19:41 -06:00
lrnselfreliance
1554dfb268
A new database is refreshed on startup (#827)
Download and Upload are disabled until a full refresh completes, and a
fresh install never had one until the user found the refresh link.
Startup now queues it, and the dashboard says why Download and Upload
are disabled meanwhile.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:07:04 -06:00
lrnselfreliance
295b3ccfc8
INSTALL.md points to the Controller drive wizard (#825)
* INSTALL.md points to the Controller drive wizard

Debian installs are Debian 13 from the amd64 ISO.  The Raspberry Pi
steps no longer hand-edit fstab or run repair.sh; a shared First Boot
section covers the drive, certificate trust, and the first refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* INSTALL.md: Debian installs use the drive wizard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:57:10 -06:00
lrnselfreliance
dc2ab22ae0
Debian installs show the Controller drive wizard (#826)
wrolpi-bootstrap.sh created controller.yaml on installed systems, which
hid the onboarding wizard.  Installed systems now wait for the wizard,
whose repair.sh finishes setup.

"Run repair without external drive" now writes controller.yaml before
starting repair, so the wizard does not return after repair.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:50:20 -06:00
Roland
697a8180c7
Bump version to 0.30.0-beta
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:13:56 -06:00
lrnselfreliance
ce923cbff0
Video transcodes run at the least CPU priority (#824)
Some checks failed
Backend Tests / api-tests (core) (push) Has been cancelled
Backend Tests / api-tests (files) (push) Has been cancelled
Backend Tests / api-tests (media) (push) Has been cancelled
Backend Tests / api-tests (rest) (push) Has been cancelled
Backend Tests / api-contract-tests (push) Has been cancelled
Backend Tests / api-migrations-and-start (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Cypress Tests / cypress-component-tests (push) Has been cancelled
Cypress Tests / cypress-e2e-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:10:44 -06:00
lrnselfreliance
4e53e02519
Notifications without a URL are not links (#823)
Transcode, job, and Kiwix notifications send no URL, but every user notify
toast said "Click here to view the shared page" and opened /null on click.
Only a notification carrying a URL is followable now.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:37:26 -06:00
lrnselfreliance
87e146487b
Database schema lists each column's foreign key (#822)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Without them a model guessed joins: it joined video.channel_id to collection.id
(it is channel.id), and the query silently returned the wrong channel names.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:38:39 -06:00
lrnselfreliance
61e58fac17
Controller script elapsed time is correct outside UTC (#821)
systemd prints the start timestamp in local time, which was subtracted from
utcnow(), so elapsed_seconds was off by the UTC offset (6 hours in MDT).  Ask
systemd for a Unix timestamp instead.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 18:30:08 -06:00
Roland
06ca3d421f
Sign master for device upgrades
upgrade.sh only installs a branch tip signed with the WROLPi key; the tip was a
GitHub squash merge (#820).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:49:54 -06:00
lrnselfreliance
f27720683d
Read-only SQL query endpoint for advanced users (#820)
* Read-only SQL query endpoint for advanced users

POST /api/db/query runs one read-only statement; GET /api/db/schema describes
the tables.  Both are refused unless `allow_database_query: true` is set in
wrolpi.yaml (there is no UI for it).  The connection is opened with mode=ro and
an authorizer that allows only reads; queries are interrupted after 10 seconds,
rows are capped, and long values are truncated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Database queries cannot be enabled through the API, and use bounded memory

The config update API refuses to change `allow_database_query`; only an edit of
wrolpi.yaml on the WROLPi can enable queries.  Query connections refuse values
over 32 MB and rows over 100 columns, rows are read one at a time, and the
response stops at 5 million characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The database query setting can be changed like any other config

WROLPi is meant to be as secure as a bookshelf in your home; a LAN user who can
reach the API is trusted, so gating one key in the config API adds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Database queries can search FTS5 tables on SQLite 3.46

FTS5 reads `PRAGMA data_version` through the authorizer when it opens a table
on SQLite 3.46 (Debian 13), so every MATCH and the schema endpoint failed with
"vtable constructor failed".  A denied VACUUM INTO also explains itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A config status test attaches the shared context it reads

It passed only when an earlier test in the same worker had attached it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:34:00 -06:00
lrnselfreliance
3ea8eb81f6
Model reprs read only loaded state (#819)
A repr runs inside log calls. Reading a relationship or an expired column there runs a query, and
on an object whose session is gone it raises DetachedInstanceError instead of logging. Archive's
repr also stat'ed its SingleFile file on disk.

`wrolpi.common.loaded()` follows attributes through state that is already loaded (primary keys
from the identity key) and never runs SQL. Every model repr uses it; with everything loaded the
output is unchanged, except Video and Archive now show the FileGroup's primary path.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:47:32 -06:00
lrnselfreliance
c717f11821
Duplicate video cleanup keeps a complete copy and its viewing history (#818)
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest.  A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted.  The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.

Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before.  The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:25:25 -06:00
lrnselfreliance
a95b187edc
A yt-dlp format file is not the primary file when merged media is present (#817)
`get_primary_file` returned the first video/audio file in whatever order it
was given, so a group holding both `<stem>.mp4` and a leftover video-only
`<stem>.f313.webm` could make the leftover the primary file.  The Video then
pointed at a file without audio, and a download of it was deferred with
"Download did not contain audio stream" on every retry.

Skip yt-dlp per-format files (`.fNNN` stems) when a merged media file is in
the group.  A group of only format files still gets one as its primary.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:20:44 -06:00
lrnselfreliance
fc7db34864
Video download deletes yt-dlp format leftovers of any extension (#816)
A merge that never finished leaves `<stem>.f313.webm` (video-only) next to
where the merged file belongs.  A later download of the same video globs
every `<stem>.*` file into its FileGroup, but only `.part` files and format
files with the *same* extension as the merged video were deleted.  A `.webm`
leftover beside a new `.mp4` survived, joined the new FileGroup, and could
become the Video's file.

Delete every `<stem>.fNNN.*` file in the group, whatever its extension, which
also covers WROLPi's poster/ffprobe sidecars of an indexed leftover and
yt-dlp's `.ytdl` state files.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:20:25 -06:00
lrnselfreliance
4adc4e6a15
Drop the SQLAlchemy asyncio extra (#815)
Nothing uses sqlalchemy.ext.asyncio; an AsyncSession pilot on the request path was measured on a
Pi and dropped. SQLAlchemy 2.0 still requires greenlet on aarch64 and x86_64, so installs on the
Pi, Debian and Docker are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:18:16 -06:00
lrnselfreliance
8e2ab935af
Archive search counts from the search index (#814)
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:37:27 -06:00
lrnselfreliance
99a0fc275d
Archive and doc pages record a view in its own write transaction (#813)
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.

Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:13:50 -06:00
lrnselfreliance
4202e6a7e5
Downloads and uploads: don't read ORM objects after their session is gone (#812)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* File downloads: return plain values from upsert_file

Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`.  `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone.  SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends.  Failed downloads retry, re-downloading their files.

`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Archive uploads: return plain values from singlefile_to_archive

Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved.  It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:41:04 -06:00
lrnselfreliance
673c23188c
Request sessions keep objects after commit; guard dict() serializers (#811)
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query.  An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async.  Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.

Collection, CollectionItem and Channel dict() now run inside the serializer guard.  Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
  handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
  Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
  reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
  the session.

Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:15:33 -06:00
dependabot[bot]
1f8a887c11
build(deps): bump shell-quote from 1.10.0 to 1.12.0 in /app (#792)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.10.0 to 1.12.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/shell-quote/compare/v1.10.0...v1.12.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.12.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:48 -06:00
dependabot[bot]
d8d28503b5
build(deps): bump compression from 1.8.1 to 1.8.2 in /app (#793)
Bumps [compression](https://github.com/expressjs/compression) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/expressjs/compression/releases)
- [Changelog](https://github.com/expressjs/compression/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/compression/compare/v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: compression
  dependency-version: 1.8.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:43 -06:00
dependabot[bot]
562150631c
build(deps): bump proxy-addr from 2.0.7 to 2.0.8 in /app (#794)
Bumps [proxy-addr](https://github.com/jshttp/proxy-addr) from 2.0.7 to 2.0.8.
- [Release notes](https://github.com/jshttp/proxy-addr/releases)
- [Changelog](https://github.com/jshttp/proxy-addr/blob/master/HISTORY.md)
- [Commits](https://github.com/jshttp/proxy-addr/compare/v2.0.7...v2.0.8)

---
updated-dependencies:
- dependency-name: proxy-addr
  dependency-version: 2.0.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:39 -06:00
dependabot[bot]
d205f7ca72
build(deps): bump source-map-js from 1.2.1 to 1.2.2 in /app (#795)
Bumps [source-map-js](https://github.com/7rulnik/source-map-js) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/7rulnik/source-map-js/releases)
- [Changelog](https://github.com/7rulnik/source-map-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/7rulnik/source-map-js/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: source-map-js
  dependency-version: 1.2.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:34 -06:00
dependabot[bot]
a36e3a3770
build(deps): bump the npm-dependencies group in /app with 3 updates (#807)
Bumps the npm-dependencies group in /app with 3 updates: [@tabler/icons-react](https://github.com/tabler/tabler-icons/tree/HEAD/packages/icons-react), [web-vitals](https://github.com/GoogleChrome/web-vitals) and [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@tabler/icons-react` from 3.48.0 to 3.49.0
- [Release notes](https://github.com/tabler/tabler-icons/releases)
- [Changelog](https://github.com/tabler/tabler-icons/blob/main/packages/icons-react/CHANGELOG.md)
- [Commits](https://github.com/tabler/tabler-icons/commits/v3.49.0/packages/icons-react)

Updates `web-vitals` from 6.2.2 to 6.2.3
- [Changelog](https://github.com/GoogleChrome/web-vitals/blob/main/CHANGELOG.md)
- [Commits](https://github.com/GoogleChrome/web-vitals/compare/v6.2.2...v6.2.3)

Updates `@types/node` from 26.6.3 to 26.6.4
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@tabler/icons-react"
  dependency-version: 3.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: web-vitals
  dependency-version: 6.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.6.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:30 -06:00
dependabot[bot]
d5c02b34ae
build(deps): bump aiohttp (#806)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: controller-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:26 -06:00
dependabot[bot]
226f955dd0
build(deps): bump the python-dependencies group with 6 updates (#808)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: cachetools
  dependency-version: 7.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: libzim
  dependency-version: 3.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: python-dotenv
  dependency-version: 1.2.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytz
  dependency-version: '2026.5'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: selenium
  dependency-version: 4.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-10 11:00:21 -06:00
lrnselfreliance
ab6a40be20
API JSON: encode paths without a lookup per path (#810)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
CustomJSONEncoder looked up the media directory and called Path.relative_to
for every path it encoded.  A response with hundreds of paths (an Archive's
page with 51 history entries holds 416) spent most of its encoding time
there: 18.8 ms of 19 ms per response, measured on a Mac.  The encoder now
looks up the media directory once per response and strips it as a string
prefix: 1.5 ms for the same response.

Paths outside the media directory, relative paths, and siblings that share
the media directory's name prefix are still returned unchanged; the media
directory itself and "." are still "".

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:45:43 -06:00
lrnselfreliance
a8dde44d4f
Archive page: load history and tags eagerly (#809)
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive.  An
Archive with 51 history entries ran 107 statements per request.  Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:45:13 -06:00
lrnselfreliance
dab333124e
Upgrade to SQLAlchemy 2.0 (synchronous) (#803)
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2.  This is a mechanical port; the code
stays synchronous.

- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
  attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
  accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
  works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
  session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
  the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
  statement committed on its own.  2.0 autobegins, so the unique-stem
  migration's read snapshot could not become a writer after its helpers wrote
  on another connection ("database is locked").  The begin listener emits no
  BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
  its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
  no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:08:40 -06:00
lrnselfreliance
bb8945e8ad
Render unknown URLs as a 404 page inside the app layout (#805)
An unknown URL (e.g. /archive instead of /archives) had no matching route,
so react-router rendered the root errorElement in place of Root.  That put
PageNotFound outside ThemeProvider, and its Mantine Box threw
"MantineProvider was not found", leaving a blank page.

- A catch-all route renders PageNotFound inside Root, with the navigation
  bar, footer and theme.
- The root errorElement brings its own ThemeProvider, so it can render when
  Root itself fails.
- App exports its routes so a test can mount them in a memory router.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:36 -06:00
lrnselfreliance
d75be8e1a4
Download forms: allow a frequency of "Once" (#804)
* app: a download frequency of "Once" no longer blocks submission

useForm treated any falsy value as a missing required field.  "Once" is
frequency 0, so the Channel/Playlist, RSS, and Zim download forms never
became ready and the Download button stayed disabled.  Only null,
undefined, '', false, and NaN count as missing now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* downloads: editing to "Once" stores a once-download, not frequency 0

Once-downloads have a NULL frequency.  A 0 frequency counted as recurring
(frequency != None) but could never be scheduled (not frequency), so an
edited download stayed on the recurring list and could break renewal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:24 -06:00
lrnselfreliance
c8d5765ea3
Scrape: fix multiple suffixes and relative link resolution (#801)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Scrape: match any of the comma-separated suffixes

The UI's File Suffixes field accepts a comma-separated list, but the
downloader compared each link against the whole string, so a scrape with
more than one suffix never matched anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Scrape: resolve relative links against the page's directory

resolve_url appended a relative href to the page URL, so a link to
"a.txt" on ".../dir/page.html" became ".../dir/page.html/a.txt" and the
file download 404'd. Use urljoin, and resolve against the final URL after
redirects so a directory URL without a trailing slash still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:05:59 -06:00
lrnselfreliance
22b8b9c878
Scrape: honor <meta> charset when decoding pages (#800)
aiohttp's response.text() ignores <meta> charset declarations and assumes
UTF-8 when the Content-Type header has no charset, so legacy Latin-1 pages
crashed the Scrape downloader with UnicodeDecodeError. Decode with
UnicodeDammit instead: header charset first, then the page's declared
charset, then detection.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:39:39 -06:00
lrnselfreliance
5ec85c3e04
Help: document resuming where you left off (#799)
Bumps the help docs for the new "Resume Where You Left Off" page and the
recently_viewed.yaml config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:25:29 -06:00
lrnselfreliance
3c362799b5
Resume EPUBs and comic books where the reader left off (#798)
* Resume EPUBs and comic books where the reader left off

The EPUB viewer (epub.html) accepts `?cfi=` and posts its location and
progress to the page that embeds it on every page turn.  Progress comes from
the chapter and the page within it, rather than book.locations.generate(),
which stalls a Raspberry Pi on a large book.  A search result's `loc` still
wins over the saved position.

The comic viewer opens at the saved page and saves each page shown.

Both resume on the doc page and in the file preview.  The resume point is
decided once per opened file, so saves made while reading do not change the
EPUB viewer's URL and reload it.  The doc page tracks only the FileGroup's
primary file; another format beside it is not tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* EPUB/comic progress: never save a place the server would discard

The server treats a position at the start (the cover, under 1%) as not
started, and replaces the saved place with progress 0.  Both viewers
reported the place they opened on, so opening a finished book un-finished
it, and peeking at the cover erased the reader's page.

- The comic viewer does not save the page it opens on, nor the cover; a
  peek at the cover keeps the last page saved.
- The EPUB viewer posts nothing at the start of the book.
- Progress before the end is kept between 1% and 94% (resumableProgress), so
  early pages of a long book and the last pages before the end keep their
  position.  Only the last page (comics) or the end (EPUB) is finished.
- Non-finite EPUB progress is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:19:51 -06:00
lrnselfreliance
1ad451cddf
Resume videos and audio where the user left off; show progress on cards (#791)
* Resume videos and audio where the user left off; show progress on cards

Videos and audio resume where the user left off, on the video page and in
the file preview, and keep playing.  `?t=` starts at an explicit time
instead.

useFileProgress saves how far the user is through a file: every 15 seconds
while it changes, and a final save (a beacon, which survives the page
closing) on pause, end, hiding the page, leaving it, or switching files.
Each save is also kept in sessionStorage; on open the newer of that and the
server's position wins, so coming straight back resumes exactly even if the
final save is still in flight.

Cards show a progress bar along the bottom of their poster.  The dashboard's
Recently Viewed Files gains All / Continue; Continue uses the new
`in_progress` file search filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Progress review fixes: in_progress totals, latest dashboard request, preview viewed

- The in_progress search filter skips the uncounted fast path and is part of
  the totals cache key, so Continue never reports All's (or the library's)
  count.
- The dashboard's Recently Viewed hook applies only its latest response, and
  clears the list when switching All/Continue until the new files arrive.
- POST /api/files/file returns `viewed`, so a preview can tell a file
  finished elsewhere from an older position this tab holds.
- The progress hook tests restore Blob, fetch and sendBeacon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:29:22 -06:00
lrnselfreliance
fbdf5bac82
Rename skip_circleci to skip_ci (#797)
CircleCI is gone, so the skip and its flag (CIRCLECI → IS_CI) are named for
CI in general, and the flag reads only GITHUB_ACTIONS.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:28:13 -06:00
lrnselfreliance
bcbc42632d
Remove CircleCI (#796)
GitHub Actions runs every job CircleCI did (#789), so CircleCI only
duplicated them.  The README badge now shows the backend tests on master.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:19:38 -06:00
lrnselfreliance
1e112b678a
Run the CircleCI jobs on GitHub Actions (#789)
* Run the CircleCI jobs on GitHub Actions

Backend tests (pytest, API contract, migrations + API start) and Cypress
(component, e2e) now run as GitHub Actions workflows.  CircleCI stays in
place until these have run green alongside it.

- pip and npm caches via setup-python/setup-node, plus the Cypress binary;
  node_modules is no longer cached, to stay inside the 10 GB cache cap.
- Superseded pull request runs are cancelled to stay under the concurrent
  job limit.
- react-app-start is dropped: the e2e job waits for `npm start`, and
  jest-tests.yml already runs `npm run build`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the CI-unsupported tests on GitHub Actions too

skip_circleci only read CIRCLECI, so on GitHub Actions test_archive_upload
ran without the readability tooling it needs, and failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Run the backend tests on every vCPU

-nauto counts physical cores when psutil is installed, so the four-vCPU
runner got two workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Split the backend tests into four jobs by directory

The groups are balanced by test time, so each runs about a quarter of the
suite.  `rest` ignores the other groups' paths rather than listing its own,
so a new directory is tested before it is assigned to a group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:08:16 -06:00
lrnselfreliance
cb189bacb1
Give the repos ownership tests a fixed git identity (#790)
test_ownership.py's git() helper did not use TEST_GIT_ENV, so its commits
read the machine's git identity.  CI runners have none and cannot guess one,
so seven tests failed there on every run.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:01:59 -06:00
lrnselfreliance
184c95e250
Ignored directories: match whole directories, and check where a link points (#788)
An ignored directory was matched as a string prefix, so ignoring `private`
also ignored `private-notes`.  A file is now ignored only when it is the
ignored directory or inside it.

Viewing a file, or saving its progress, checked only the path it was given,
so a link outside an ignored directory to a file inside it was tracked.
is_ignored_file checks the link and its target.  Refresh does not follow
links, so the bulk paths keep the cheaper lexical check.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:29:06 -06:00
lrnselfreliance
9588af2601
Track how far the user is through a file, kept in recently_viewed.yaml (#787)
* Track how far the user is through a file, kept in recently_viewed.yaml

FileGroup gains `progress` (0.0 to 1.0, 1.0 is finished) and `position` (where
to resume: a time, a page, or an EPUB CFI).  Both are returned with the file,
including by POST /api/files/file.

POST /api/files/progress records progress and is also a view.  A file 95%
through is finished and keeps no position; one barely started (under 10 s,
the first page, under 1%) is not worth resuming.  The write happens before
the response, so the next request sees it.  POST /api/files/progress/clear
starts a file over.  Both refuse files outside the media directory.

recently_viewed.yaml keeps the 1000 most recently viewed files so viewing
history and positions survive a database rebuild.  On import the newer
`viewed` wins, so a file viewed while WROLPi was starting keeps its newer
position.  The config is saved when a file is opened, when progress is final
(paused or closed), when it is cleared, and after refresh cleanup (moves,
renames, deletes); heartbeats during playback only write the database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Progress: write as a writer, confine and merge the import, keep FTS on downgrade

- The progress endpoints begin as writers (request.ctx.db.write()), so a
  heartbeat during a refresh waits instead of failing a lock upgrade.  The
  config switch fires after the commit.
- `..` is collapsed before the ignored-directory check.  Clear only resets
  an existing FileGroup; it never indexes a file.
- Import confines config paths to the media directory.  A newer plain view
  keeps its `viewed` but takes the config's position; only progress written
  after the config wins.  Files are globbed before the write transaction.
- The dump skips a row outside the media directory instead of failing.
- The downgrade drops the columns with ALTER TABLE, which keeps file_group's
  FTS triggers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:25 -06:00
lrnselfreliance
531721c157
Fix file view tracking: docs mark viewed, previews count one view (#786)
Opening a doc on its page never set FileGroup.viewed, so docs were missing
from Recently Viewed.  GET /api/docs/:id now marks the doc viewed, with the
same skip_viewed parameter as archives and videos.

The file preview marked a file viewed more than once: opening from a
?preview= URL sent two tracked requests, and every tag change re-fetched the
file with tracking on.  The fetches now skip tracking, and the preview marks
each opening viewed once.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:04:13 -06:00
lrnselfreliance
a1292005ce
Other search tab: search Channels by name, add Domains and Playlists, fix the estimate (#785)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Count Channels with any of the Tags in the Other search estimate

The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Search Channels by name in the Other search tab

The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing.  Channels now match the search text by name,
and the Tags when there are both.  The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Domains to the Other search tab

Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Playlists to the Other search tab

Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint.  The search and the estimate share search_collections_query.  The
search now loads each Collection's Downloads up front instead of once per
Collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 18:22:12 -06:00
lrnselfreliance
93d878952c
Add a Repos module that mirrors git repositories (#783)
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.

- A Repo is a Collection (kind='repo') plus a `repository` row.  The new `git` downloader clones it, then updates it
  (weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
  An existing clone can be imported where it is.  repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download.  READMEs are searchable
  under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
  repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
  updated.
- Tagging a Repo moves its clone as one directory.  Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
  destination.  A failed tag move restores the Collection's tag.  A Collection's directory cannot leave the media
  directory.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:58 -06:00
lrnselfreliance
1ded249fc7
Collections: changes wait for their config's import; domains with archives survive the import (#784)
* collections: refuse playlist changes until playlists.yaml is imported

While WROLPi starts, the API accepts requests before every config is imported.  A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change).  Playlist changes now wait for the import (503, "still starting").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* collections: refuse domain changes until domains.yaml is imported

A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import.  These now wait for the import (503, "still starting").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* archive: importing domains.yaml keeps domains which have archives

Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported).  The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it.  A domain with Archives is now kept (and saved to the config with the next change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* videos: refuse Channel changes until channels.yaml is imported

A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config".  Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:33:51 -06:00
lrnselfreliance
b742a4a25f
Keep the API's logging when migrating at startup (#782)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
WROLPi runs alembic migrations in-process at startup (db_bootstrap).
alembic/env.py called fileConfig unconditionally, which replaced the
root handlers, set the root level to WARN, and disabled every logger
that already existed, so the API's logs went quiet after any startup
migration until the next restart.  env.py now configures logging only
when alembic runs from its command line (cmd_opts is set).

This also caused an order-dependent test failure: test_jobs's capture
test failed whenever test_db_bootstrap ran first in the same worker.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:55:53 -06:00
lrnselfreliance
2b2361ed64
Save playlists config when a playlist's file is renamed or moved (#781)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
playlists.yaml references file items by path. Renaming, moving, or
reorganizing a FileGroup updated the database but not the config, so the
next import (startup or global refresh) could not find the file and
silently dropped the item from the playlist.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:03:45 -06:00
lrnselfreliance
04779b8557
Preview markdown files formatted, with a raw toggle (#780)
* Preview markdown files formatted, with a raw toggle

Markdown files are typed text/markdown (libmagic reports them as text/plain, or
as a programming language when they hold code) and keep the text indexer, so
they stay deep-searchable.  Existing files switch type on their next refresh.

The file preview renders markdown with markdown-it into the page, so it follows
the theme tokens and the night media filter.  A Formatted/Raw toggle shows the
source instead, and the choice is remembered.  Relative links and images
resolve against the file's directory; URLs that climb out of /media/ are
dropped.  Raw HTML in a file is escaped rather than parsed, and the output is
also sanitized with DOMPurify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Do not load images from other sites in the markdown preview

An image from another site (http:, https:, or protocol-relative //) loaded as
soon as the preview opened, telling that site who was reading the file and
when.  Such an image now becomes a link labeled with its alt text, so the
reader chooses whether to follow it.  Inside a link (a README badge) it becomes
the outer link's text, since links cannot nest.  Media and embedded data:
images still load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 13:26:42 -06:00
lrnselfreliance
9739dc9e1b
tests: fix the midnight and event-loop timing flakes (#779)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
test_import_config_preserves_completed_once_downloads created a download "5 minutes ago" and expected it to
count toward today's limit, which starts at local midnight; in the first five minutes after local midnight it
was yesterday's.  It failed in CI started at 00:00:35 UTC, and reproduces on master with the clock at 00:02.
The test now pins the clock to local noon, as the download manager reckons local time.

test_import_all_db_configs_does_not_block_event_loop required 90% of the expected 50 ms ticks, so timer
overshoot on a busy runner added up across the run (CI got 28 of the 28.8 required).  It now measures the
longest gap between ticks, including after the last tick: an import on the loop freezes it for the whole 0.2 s
sleep, while jitter delays a tick by milliseconds.  Moving the first or the last import onto the loop still
fails it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:43:00 -06:00
lrnselfreliance
cc8afa5370
Add a Solar calculator (#778)
* Add a Solar calculator with offline NASA POWER data

Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.

The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter.  "Best for the year" and "Best for winter" sweep
the tilt.

Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens.  Users can replace it with their own monthly values.

Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.

Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Open the Solar calculator from the map, pins and current location

The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location.  A button beside Latitude fills in the browser's current
location.  Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.

The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button).  It now only cleans up while the
URL is still a map page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add off-grid sizing to the Solar calculator

A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):

- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
  at 12, 24 or 48 V.  Outdoor lead-acid is enlarged for the cold; outdoor
  lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
  angle, with a button to apply the count.  A PWM controller's loss (battery
  voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
  against the controller's limit, since cold panels exceed their rated Voc.
  The coldest morning defaults to 25 C below the coldest monthly average.

All inputs live in the URL like the rest of the calculator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bump wrolpi-help for the Solar calculator help page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Credit NASA POWER as it asks, and say how the data was changed

- The calculator gains a "Data attribution" link under the location with
  NASA POWER's requested acknowledgment, the data reference (Climatology API
  v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
  modified the data (rounded to one byte per value, temperature averaged
  into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
  beside the data, with the API version taken from NASA's responses and the
  date the tiles were downloaded.  A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: fix PWM sizing, Voc sign, unit switching and loading on review

- PWM usable power now uses the string's voltage (Vmp x panels in series),
  delivers nothing when the charging voltage is at or above the string's Voc,
  and falls off between Vmp and Voc.  The "too little voltage" warning covers
  PWM as well as MPPT.  Previously two panels in series counted double, and a
  panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
  negative, in the math, the URL and the input, so the cold-morning check
  cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
  of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
  than asking for a location, and holds the cold-morning verdict until the
  site's temperatures are known or one has been entered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: move the data attribution to the bottom of the page

The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location.  The source line under the
location points to it.  Bumps wrolpi-help to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: drop the data source line under the location

The attribution at the bottom of the page is enough.  The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values.  Bumps wrolpi-help for the
refreshed screenshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:36:45 -06:00
lrnselfreliance
cfbed9e4d7
db: serializers read only loaded data (#777)
* db: serializers read only loaded data

Serializing a model must not query.  A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.

- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
  an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
  per serializer and lets the query run.  Tests record each trip as well, and the autouse
  `no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
  swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
  Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
  Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
  Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.

Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.

Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* db: serializer loads per query; the guard ignores transaction control

The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer.  Transaction-control statements are now ignored.

Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json.  A second refresh of 100 videos took 0.40 s instead of 0.16 s.

Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it).  Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection.  A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:28:08 -06:00
lrnselfreliance
1f687fa825
Move entry tags to the current Zim when deleting outdated Zims (#776)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Deleting a Zim cascade-deletes its TagZimEntry rows, and the next tags
save writes them out of the config, so every outdated-Zim delete lost
its entry tags.  The config-import migration only runs at startup, too
late to help.  Move each outdated Zim's tags to the current Zim of the
same name before deleting, then save the tags config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:04:11 -06:00
lrnselfreliance
8713a6136b
Reset file worker status when a synchronous refresh is cancelled (#775)
* Reset file worker status when a synchronous refresh is cancelled

refresh_sync runs inside request handlers.  When Sanic cancelled one on
response timeout, the phase flag cleared as its `with` block unwound but
the shared status did not, so the worker reported 'modeling' forever while
nothing ran.  Reset to idle on cancellation.

Deleting outdated Zims also ran the global modelers inside the request,
which is what timed out.  Skip post-processing there and update the
outdated_zims flag directly, the only cleanup step it relied on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tests: observe the modeling flag in the cancelled refresh_sync test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:25:25 -06:00
lrnselfreliance
bed25b429d
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection

Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection.  Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.

- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
  the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode.  `get_video_for_app` and
  `update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
  reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
  download list, retry, kill and the status summary, and docs/videos/archive statistics.

Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.

The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* downloads: startup retries in a write session of its own

start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran.  Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.

Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
lrnselfreliance
a2bef48c6c
db: one request-scoped database handle per API request (#773)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* db: one request-scoped database handle per API request

Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that
Session's connection, so a request needs at most one SQLite connection.  Creating it does no I/O; requests
that never touch the database never connect.  `request.ctx.session` remains the handle's Session.

A `RequestSession` binds itself to one lazily-opened Connection.  An engine-bound Session releases its
connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected.

Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the
write lock is not held for the rest of the handler or by background work it starts.  This replaces the
handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder,
channel create).  The zim search handlers use the request's Session in their worker thread.

Tests count the connections a request opens with production's per-call sessions
(`count_request_connections`).  Endpoints whose library functions still open their own connection are
strict xfails, to be removed as those functions take the request's session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tests: name the real extra connections in the request connection xfails

Channel detail: get_channel already uses the request session; the extra connection is
Channel.get_statistics opening a raw cursor (get_db_curs).  Search: count_file_groups opens a
cursor for the total, then handle_file_group_search_results opens a cursor and a session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 15:42:06 -06:00
Roland
d7deea668d
Bump version to 0.29.0-beta
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 11:53:21 -06:00
lrnselfreliance
1c8bbf1f10
deps: cachetools 7 and the current web-vitals API (#771)
cachetools 6 removed cachetools.func.mru_cache; the Zim search estimate
cache was its only user, so it becomes an lru_cache and the pin moves
to 7.2.0 with the Dependabot ignore removed.

reportWebVitals still destructured the web-vitals 2.x names (getCLS,
getFID, ...) which no longer exist, so it silently did nothing.  Use
the on* names, replace FID with INP, return the import promise, and
accept any callable rather than instanceof Function.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 11:04:16 -06:00
lrnselfreliance
c28a9e0078
Revert "build(deps-dev): bump webpack-dev-server from 5.2.6 to 6.0.0 in /app (#769)" (#772)
Cypress component testing resolves the top-level webpack-dev-server
and only accepts major 5, so the bump crashed cypress.config.js on
every run ("Unexpected major version of webpack-dev-server").  The
react-scripts dev server was unaffected because it uses its own
nested 4.x, which is what the pre-merge check looked at.

Pin webpack-dev-server below 6 in dependabot.yml until Cypress
supports it.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 11:00:27 -06:00
lrnselfreliance
f37098cb81
tests: fix the two suites failing on master (#770)
test_handle_reorganize_marks_job_failed_on_exception patched
wrolpi.files.worker.shutil.move, but #719 moved handle_reorganize and
its shutil.move call into wrolpi.files.reorganize_job.

test_file_worker_refresh_inserts_new_files loaded FileGroups into the
shared pytest session between refreshes.  The refresh upserts by raw
SQL, so the modeler then read the stale identity-map object and wrote
its old file list back over the upsert.  Production opens a fresh
session per block; expire the test session before the second refresh.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 10:50:24 -06:00
dependabot[bot]
8610f9c97e
build(deps): bump react-router in /app in the npm-dependencies group (#768)
Bumps the npm-dependencies group in /app with 1 update: [react-router](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router).


Updates `react-router` from 7.18.2 to 7.18.4
- [Release notes](https://github.com/remix-run/react-router/releases)
- [Changelog](https://github.com/remix-run/react-router/blob/react-router@7.18.4/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/remix-run/react-router/commits/react-router@7.18.4/packages/react-router)

---
updated-dependencies:
- dependency-name: react-router
  dependency-version: 7.18.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:44:22 -06:00
dependabot[bot]
6b79323d2c
build(deps-dev): bump webpack-dev-server from 5.2.6 to 6.0.0 in /app (#769)
Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 5.2.6 to 6.0.0.
- [Release notes](https://github.com/webpack/webpack-dev-server/releases)
- [Changelog](https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-server/compare/v5.2.6...v6.0.0)

---
updated-dependencies:
- dependency-name: webpack-dev-server
  dependency-version: 6.0.0
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:43:52 -06:00
lrnselfreliance
51d9c07784
dependabot: also hold cypress 16, react-router 8 and testing-library/react majors (#767)
Each regenerated after the last config change and each needs its own
migration: cypress 16 needs Node 22, react-router 8 touches ~40 files,
and @testing-library/react 16 fails 15 menu/popover tests.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 10:34:52 -06:00
dependabot[bot]
9fc9929647
build(deps): bump web-vitals from 5.3.0 to 6.2.2 in /app (#764)
Bumps [web-vitals](https://github.com/GoogleChrome/web-vitals) from 5.3.0 to 6.2.2.
- [Changelog](https://github.com/GoogleChrome/web-vitals/blob/main/CHANGELOG.md)
- [Commits](https://github.com/GoogleChrome/web-vitals/compare/v5.3.0...v6.2.2)

---
updated-dependencies:
- dependency-name: web-vitals
  dependency-version: 6.2.2
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:33:23 -06:00
dependabot[bot]
39cef45c59
build(deps): bump the npm-dependencies group in /app with 3 updates (#761)
Bumps the npm-dependencies group in /app with 3 updates: [react](https://github.com/react/react/tree/HEAD/packages/react), [react-dom](https://github.com/react/react/tree/HEAD/packages/react-dom) and [react-dropzone](https://github.com/react-dropzone/react-dropzone).


Updates `react` from 18.2.0 to 18.3.1
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v18.3.1/packages/react)

Updates `react-dom` from 18.2.0 to 18.3.1
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v18.3.1/packages/react-dom)

Updates `react-dropzone` from 14.2.3 to 14.4.1
- [Release notes](https://github.com/react-dropzone/react-dropzone/releases)
- [Commits](https://github.com/react-dropzone/react-dropzone/compare/v14.2.3...v14.4.1)

---
updated-dependencies:
- dependency-name: react
  dependency-version: 18.3.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: react-dom
  dependency-version: 18.3.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: react-dropzone
  dependency-version: 14.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:33:11 -06:00
dependabot[bot]
920b619bcc
build(deps): bump the python-dependencies group with 12 updates (#763)
Bumps the python-dependencies group with 12 updates:

| Package | From | To |
| --- | --- | --- |
| [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) | `1.3.22` | `1.3.24` |
| [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/) | `4.14.3` | `4.15.0` |
| [cachetools](https://github.com/tkem/cachetools) | `5.5.0` | `5.5.2` |
| [cryptography](https://github.com/pyca/cryptography) | `50.0.0` | `50.0.2` |
| [feedparser](https://github.com/kurtmckee/feedparser) | `6.0.12` | `6.0.14` |
| [jc](https://github.com/kellyjonbrazil/jc) | `1.25.6` | `1.26.0` |
| [libzim](https://github.com/openzim/python-libzim) | `3.10.0` | `3.13.0` |
| [pytest](https://github.com/pytest-dev/pytest) | `9.0.3` | `9.1.1` |
| [python-dotenv](https://github.com/theskumar/python-dotenv) | `1.2.2` | `1.2.3` |
| [pytz](https://github.com/stub42/pytz) | `2026.2` | `2026.4` |
| [selenium](https://github.com/SeleniumHQ/Selenium) | `4.44.0` | `4.49.0` |
| [tqdm](https://github.com/tqdm/tqdm) | `4.67.3` | `4.70.1` |


Updates `sqlalchemy` from 1.3.22 to 1.3.24
- [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases)
- [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst)
- [Commits](https://github.com/sqlalchemy/sqlalchemy/commits)

Updates `beautifulsoup4` from 4.14.3 to 4.15.0

Updates `cachetools` from 5.5.0 to 5.5.2
- [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/tkem/cachetools/compare/v5.5.0...v5.5.2)

Updates `cryptography` from 50.0.0 to 50.0.2
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/50.0.0...50.0.2)

Updates `feedparser` from 6.0.12 to 6.0.14
- [Release notes](https://github.com/kurtmckee/feedparser/releases)
- [Changelog](https://github.com/kurtmckee/feedparser/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/kurtmckee/feedparser/compare/v6.0.12...v6.0.14)

Updates `jc` from 1.25.6 to 1.26.0
- [Release notes](https://github.com/kellyjonbrazil/jc/releases)
- [Changelog](https://github.com/kellyjonbrazil/jc/blob/master/CHANGELOG)
- [Commits](https://github.com/kellyjonbrazil/jc/compare/v1.25.6...v1.26.0)

Updates `libzim` from 3.10.0 to 3.13.0
- [Release notes](https://github.com/openzim/python-libzim/releases)
- [Changelog](https://github.com/openzim/python-libzim/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openzim/python-libzim/compare/v3.10.0...v3.13.0)

Updates `pytest` from 9.0.3 to 9.1.1
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.3...9.1.1)

Updates `python-dotenv` from 1.2.2 to 1.2.3
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/theskumar/python-dotenv/compare/v1.2.2...v1.2.3)

Updates `pytz` from 2026.2 to 2026.4
- [Release notes](https://github.com/stub42/pytz/releases)
- [Commits](https://github.com/stub42/pytz/compare/release_2026.2...release_2026.4)

Updates `selenium` from 4.44.0 to 4.49.0
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.44.0...selenium-4.49.0)

Updates `tqdm` from 4.67.3 to 4.70.1
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](https://github.com/tqdm/tqdm/compare/v4.67.3...v4.70.1)

---
updated-dependencies:
- dependency-name: sqlalchemy
  dependency-version: 1.3.24
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: beautifulsoup4
  dependency-version: 4.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: cachetools
  dependency-version: 5.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: cryptography
  dependency-version: 50.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: feedparser
  dependency-version: 6.0.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: jc
  dependency-version: 1.26.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: libzim
  dependency-version: 3.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: pytest
  dependency-version: 9.1.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: python-dotenv
  dependency-version: 1.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytz
  dependency-version: '2026.4'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: selenium
  dependency-version: 4.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: tqdm
  dependency-version: 4.70.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:33:05 -06:00
dependabot[bot]
a7103002b6
build(deps): bump nanoid from 3.3.16 to 3.3.19 in /app (#759)
Bumps [nanoid](https://github.com/ai/nanoid) from 3.3.16 to 3.3.19.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.3.16...3.3.19)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-version: 3.3.19
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:22:20 -06:00
dependabot[bot]
5e76914485
build(deps): bump qs and express in /app (#758)
Bumps [qs](https://github.com/ljharb/qs) and [express](https://github.com/expressjs/express). These dependencies needed to be updated together.

Updates `qs` from 6.15.3 to 6.16.0
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.15.3...v6.16.0)

Updates `express` from 4.22.2 to 4.22.3
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/v4.22.3/History.md)
- [Commits](https://github.com/expressjs/express/compare/v4.22.2...v4.22.3)

---
updated-dependencies:
- dependency-name: qs
  dependency-version: 6.16.0
  dependency-type: indirect
- dependency-name: express
  dependency-version: 4.22.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:22:15 -06:00
lrnselfreliance
0d2431c41a
dependabot: move major-version exclusions into the config (#760)
Comment-based ignores did not survive the last config change, and the
SQLAlchemy rule only blocked majors while 1.4 (the 2.0 transition
release) slipped into the python group. Hold SQLAlchemy <1.4, alembic
<1.11, cachetools majors (mru_cache removed in 6), and the npm majors
that each need a migration branch: maplibre-gl 6, react-dropzone 15+,
React 19, TypeScript 6+, Mantine 9.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 10:21:50 -06:00
dependabot[bot]
3d0eb76337
build(deps): update pymupdf requirement from >=1.27.2.3 to >=1.28.2 (#756)
Updates the requirements on [pymupdf](https://github.com/pymupdf/pymupdf) to permit the latest version.
- [Release notes](https://github.com/pymupdf/pymupdf/releases)
- [Changelog](https://github.com/pymupdf/PyMuPDF/blob/main/changes.txt)
- [Commits](https://github.com/pymupdf/pymupdf/compare/1.27.2.3...1.28.2)

---
updated-dependencies:
- dependency-name: pymupdf
  dependency-version: 1.28.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:21:14 -06:00
dependabot[bot]
3800f4d332
build(deps): update fastapi requirement in /controller (#748)
Updates the requirements on [fastapi](https://github.com/fastapi/fastapi) to permit the latest version.
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](https://github.com/fastapi/fastapi/compare/0.141.1...0.142.2)

---
updated-dependencies:
- dependency-name: fastapi
  dependency-version: 0.142.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:19:52 -06:00
dependabot[bot]
323b7e66df
build(deps): bump fast-uri from 3.1.7 to 3.1.8 in /app (#746)
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.7 to 3.1.8.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](https://github.com/fastify/fast-uri/compare/v3.1.7...v3.1.8)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:19:48 -06:00
dependabot[bot]
9eb56967e4
build(deps): bump brace-expansion in /app (#745)
Bumps  and [brace-expansion](https://github.com/juliangruber/brace-expansion). These dependencies needed to be updated together.

Updates `brace-expansion` from 1.1.18 to 1.1.21
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.18...v1.1.21)

Updates `brace-expansion` from 2.1.4 to 2.1.7
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.18...v1.1.21)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.21
  dependency-type: indirect
- dependency-name: brace-expansion
  dependency-version: 2.1.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:19:43 -06:00
lrnselfreliance
ecd8ce64be
dependabot: hold alembic below 1.11 while SQLAlchemy stays on 1.3 (#747)
alembic 1.20 requires SQLAlchemy>=2.0, so every grouped python
update that included it failed to install (ResolutionImpossible).
Ignore alembic >=1.11 and SQLAlchemy major bumps until the
SQLAlchemy 2 migration is done.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 10:11:08 -06:00
dependabot[bot]
71287c9a23
build(deps): bump the npm-dependencies group in /app with 7 updates (#741)
Bumps the npm-dependencies group in /app with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [@tabler/icons-react](https://github.com/tabler/tabler-icons/tree/HEAD/packages/icons-react) | `3.46.0` | `3.48.0` |
| [@testing-library/user-event](https://github.com/testing-library/user-event) | `14.6.6` | `14.6.7` |
| [esptool-js](https://github.com/espressif/esptool-js) | `0.6.1` | `0.7.0` |
| [react-colorful](https://github.com/omgovich/react-colorful) | `5.8.0` | `5.8.1` |
| [three](https://github.com/mrdoob/three.js) | `0.185.1` | `0.186.1` |
| [@testing-library/dom](https://github.com/testing-library/dom-testing-library) | `10.4.1` | `10.4.2` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.4.0` | `26.6.3` |


Updates `@tabler/icons-react` from 3.46.0 to 3.48.0
- [Release notes](https://github.com/tabler/tabler-icons/releases)
- [Commits](https://github.com/tabler/tabler-icons/commits/v3.48.0/packages/icons-react)

Updates `@testing-library/user-event` from 14.6.6 to 14.6.7
- [Release notes](https://github.com/testing-library/user-event/releases)
- [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/user-event/compare/v14.6.6...v14.6.7)

Updates `esptool-js` from 0.6.1 to 0.7.0
- [Release notes](https://github.com/espressif/esptool-js/releases)
- [Changelog](https://github.com/espressif/esptool-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/espressif/esptool-js/compare/v0.6.1...v0.7.0)

Updates `react-colorful` from 5.8.0 to 5.8.1
- [Release notes](https://github.com/omgovich/react-colorful/releases)
- [Changelog](https://github.com/omgovich/react-colorful/blob/master/CHANGELOG.md)
- [Commits](https://github.com/omgovich/react-colorful/commits)

Updates `three` from 0.185.1 to 0.186.1
- [Release notes](https://github.com/mrdoob/three.js/releases)
- [Commits](https://github.com/mrdoob/three.js/commits)

Updates `@testing-library/dom` from 10.4.1 to 10.4.2
- [Release notes](https://github.com/testing-library/dom-testing-library/releases)
- [Changelog](https://github.com/testing-library/dom-testing-library/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/dom-testing-library/compare/v10.4.1...v10.4.2)

Updates `@types/node` from 26.4.0 to 26.6.3
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@tabler/icons-react"
  dependency-version: 3.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@testing-library/user-event"
  dependency-version: 14.6.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: esptool-js
  dependency-version: 0.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: react-colorful
  dependency-version: 5.8.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: three
  dependency-version: 0.186.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@testing-library/dom"
  dependency-version: 10.4.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.6.3
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:08:32 -06:00
dependabot[bot]
0837330bd7
build(deps): bump colord from 2.9.3 to 2.10.0 in /app (#703)
Bumps [colord](https://github.com/omgovich/colord) from 2.9.3 to 2.10.0.
- [Release notes](https://github.com/omgovich/colord/releases)
- [Changelog](https://github.com/omgovich/colord/blob/master/CHANGELOG.md)
- [Commits](https://github.com/omgovich/colord/commits/v2.10)

---
updated-dependencies:
- dependency-name: colord
  dependency-version: 2.10.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:07:26 -06:00
dependabot[bot]
5439fff871
build(deps): bump js-yaml in /app (#705)
Bumps  and [js-yaml](https://github.com/nodeca/js-yaml). These dependencies needed to be updated together.

Updates `js-yaml` from 3.15.0 to 3.15.2
- [Changelog](https://github.com/nodeca/js-yaml/blob/3.15.2/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/3.15.0...3.15.2)

Updates `js-yaml` from 4.3.0 to 4.3.2
- [Changelog](https://github.com/nodeca/js-yaml/blob/3.15.2/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/3.15.0...3.15.2)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 3.15.2
  dependency-type: indirect
- dependency-name: js-yaml
  dependency-version: 4.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:07:22 -06:00
dependabot[bot]
c973a4106f
build(deps): bump sanic from 24.12.0 to 24.12.1 in /docker/archive (#714)
Bumps [sanic](https://github.com/sanic-org/sanic) from 24.12.0 to 24.12.1.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Commits](https://github.com/sanic-org/sanic/compare/v24.12.0...v24.12.1)

---
updated-dependencies:
- dependency-name: sanic
  dependency-version: 24.12.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:07:17 -06:00
dependabot[bot]
dd7269d1ca
build(deps): update uvicorn requirement in /controller (#742)
Updates the requirements on [uvicorn](https://github.com/Kludex/uvicorn) to permit the latest version.
- [Release notes](https://github.com/Kludex/uvicorn/releases)
- [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md)
- [Commits](https://github.com/Kludex/uvicorn/compare/0.52.4...0.54.0)

---
updated-dependencies:
- dependency-name: uvicorn
  dependency-version: 0.54.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:07:13 -06:00
lrnselfreliance
57e647c5b4
docker: web container reaps zombies; zim and help healthchecks can pass (#744)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Caddy is PID 1 in the web container and reaps nothing.  The healthcheck's
busybox `wget --spider https://...` forks an `ssl_client` helper that is
orphaned to caddy and stays a zombie, one per check.  `init: true` runs
tini as PID 1 so they are reaped.

The zim healthcheck calls curl, which the debian:13 kiwix image does not
install, so every check exited 127.  The help healthcheck probed port
8086, which is Caddy's HTTPS proxy on the web container; mkdocs serves
on port 80 inside its own container.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 15:49:34 -06:00
lrnselfreliance
8d6f5b5cad
api: forked workers die on SIGTERM before Sanic installs their handlers (#743)
A Sanic worker is forked from the manager and carries the manager's
Python SIGTERM/SIGINT handlers until `_setup_system_signals` installs its
own, which happens after the startup listeners (about two seconds with
sixteen workers starting at once).  When auto-reload fires a second time
inside that window the manager terminates the half-started generation,
each child runs `WorkerManager.shutdown_signal` instead of exiting, marks
itself TERMINATED, and then finishes starting.  The manager has already
spawned replacements, so the old generation lives on: untracked, holding
the listening socket, serving stale code, and holding ~220 MB each.

Register an at-fork hook in the main process that resets both signals to
their default action in the child.  The perpetual process already did
this by hand for itself; server workers now get the same treatment.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 15:49:06 -06:00
lrnselfreliance
5c78aa332d
collections: listing answers channel and download queries from covering indexes (#740)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
The listing still read every channel row (for the summary columns) and every download
row of the listed Collections (for the minimum frequency). Both tables store yt-dlp
info_json, so rows are tens of KB and each sits on its own scattered page; selecting
fewer columns did not help because the seek per row is the cost. From a cold disk
cache that was 5 s per table, 10+ s total, on a large library.

Two covering indexes, channel (collection_id, video_count, total_size) and
download (collection_id, frequency), answer both queries without touching a row. The
listing no longer loads Download objects at all; it computes min_download_frequency
with one grouped query and drops the unused `downloads` array from listing rows (the
detail endpoints keep it; the list views only read min_download_frequency).

Adds a models-vs-migrations test (alembic check) so an index or column added to a
model without a migration, or vice versa, fails the suite.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 12:53:40 -06:00
lrnselfreliance
20be5dfc27
collections: channel and domain listings read trigger-maintained summary columns (#739)
* collections: channel listing reads trigger columns and batch-loads tag/downloads

Listing channel collections summed FileGroup sizes through a video join, which reads a
random page of the file_group table per video, and lazy-loaded every collection's tag
and downloads one query at a time. On a large library with a cold disk cache the
listing took over two minutes, past the browser's 60 s timeout, so the Channels page
showed an error.

channel.video_count and channel.total_size are already maintained by triggers
(wrolpi/schema_ddl.py), so read those. Tag and downloads are selectin-loaded, so the
query count no longer grows with the number of collections.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* collections: domain listing reads trigger-maintained summary columns

Domain Collections summed file_group.size through an archive join on every listing,
one random page read per archive (about 9 s from a cold disk cache on a large
library). New triggers keep collection.item_count and collection.total_size current
for Domain Collections, as the channel triggers already do for Channels, and the
listing reads those columns. A migration installs the triggers and backfills
existing Domain Collections.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 11:36:08 -06:00
lrnselfreliance
4fea50943b
map: clicking a POI shows its kind, detail, elevation and coordinates (#738)
The Protomaps basemap keeps a few attributes per POI beyond the name the
map already labels: kind, kind_detail and elevation.  Clicking a POI now
opens a popup with those, plus the feature's coordinates (or the click
location when the feature has no point).  The same POI appears at several
min_zoom levels within one tile, so features are deduped by name and kind.

Map popups (this one and the existing pin popup) now follow the active
theme's tokens instead of MapLibre's fixed white, covering light, dark,
night and amber.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 08:23:03 -06:00
lrnselfreliance
15d3792b0e
videos: transcode can remove the audio stream (silent video) (#737)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
A video's audio stream can be dropped the same way its video stream could
already be: a new REMOVE_AUDIO audio target maps only the video stream with
-an, needs a video container, and is refused for audio files.  Output
verification insists the result has no audio stream.

Removing both streams is refused by the validator, the API, resolve_output,
and the modal: choosing Remove for one stream resets the other to Keep and
withdraws the other's Remove option.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:49:14 -06:00
lrnselfreliance
63ddad69e8
app: api helpers report failure instead of returning an empty value (#736)
Six helpers turned a non-OK response into an empty value, so their
consumers rendered a server error as "nothing here":

* getConfigBackups -> {dates: []}: "No backups available for this
  config."  A thrown request also left "Loading backups..." forever.
* getInventoryBackups -> []: "No backups yet."
* getDownloaders -> {downloaders: []} on a throw, the error body on a
  non-OK response (which then crashed the .map): the RSS form's
  Downloader select sat empty.
* searchMap -> {results: []}: "No map locations found".
* getMapSearchStatus -> {indexed: [], missing: []}: "No maps are
  installed".
* flasherSearch -> [null, null], then `|| []`: "No .bin firmware
  found".

Invariants now enforced and tested:
* On a non-OK response each helper returns undefined; an empty value
  only ever comes from the server.
* Each consumer shows an error for a failed or thrown fetch and keeps
  its "none" message for a confirmed empty result.  The Downloader
  select is disabled and says it is loading or could not fetch.  A
  failed map status is unknown, never "no maps are installed".
* The map search tab loads from the first render, and a query shorter
  than the two characters the request needs now says so instead of
  "No map locations found".

Archive browsers and recent tags are left as they are: a failed fetch
there hides an optional picker or suggestion list without claiming
anything.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 17:47:32 -06:00
lrnselfreliance
20541d7184
app: number fields no longer throw "e.preventDefault is not a function" (#735)
`useForm`'s input handlers called `e.preventDefault()` unconditionally, but the
Mantine-backed `NumberField` (Sleep between requests, Depth, Maximum Pages)
hands them a bare `{target}` object because NumberInput emits a value, not an
event.  Every keystroke in those fields threw.

`handleInputEvent` now only calls `preventDefault` when the argument actually
has one, and `getInputProps`' wrapper no longer repeats the call.  Audited every
other `preventDefault` call site: all are bound to real DOM/React events or a
guarded no-argument call.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:41:52 -06:00
lrnselfreliance
eead78007d
app: status and settings contexts say whether their first poll landed (#734)
Both contexts start as {} and nothing distinguished that from a
server that reported nothing, so for up to one poll interval after a
page load consumers rendered the negative case as fact: the Settings
form appeared with blank values and an "undefined/config/wrolpi.yaml"
path (the ready check `settings ? true : undefined` was always true),
the Downloads toggle was clickable and read "Downloading Disabled",
the upgrade panel said "up to date, vundefined", and the five hardware
toggles on the Controller page claimed "not supported on this server".

Invariants now enforced and tested:
* StatusContext.loaded is true once a poll returned a real status;
  the API being down does not count.  SettingsContext.loaded is true
  once settings were fetched, and `failed` when a fetch returned
  nothing before that; a failed refresh after a load keeps the last
  known values and `loaded`.
* The Settings page shows its loader until loaded and its error only
  when failed.  The upgrade panel shows a loader until status loaded.
  The Downloads toggle is disabled with a neutral label until then.
* A subsystem toggle is disabled without claiming "unsupported" while
  status is unknown (undefined); "unsupported" (null) is reserved for
  a loaded status that lacks the field.  A call in flight is unknown,
  not unsupported.

The context fixtures declare loaded so existing specs render as
before.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:25:46 -06:00
lrnselfreliance
306386e801
app: Flasher saved configs and Statistics distinguish pending from empty (#733)
The Flasher's saved configurations started as [] with no loading
flag, and the Saved Firmwares tab is the default tab, so every load
painted "No saved configurations yet." until the fetch returned.  The
api helper returned [] on a non-OK response too, so a failed fetch
read as an empty list.

useStatistics used useState() with no initial value, so the
Statistics page's first render saw undefined and flashed "Failed to
fetch statistics" for one frame before the effect set {} and the
spinner appeared.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / object = loaded.  The
  api helper returns undefined (and toasts) on a non-OK response.
* The Saved Firmwares tab shows a placeholder while pending, an error
  on failure, and "No saved configurations yet." only once the server
  confirmed there are none.  Only the newest configs request may write
  state, since save and delete refetch.
* The Statistics page shows its loader while pending and its error
  only for a failed fetch.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:05:02 -06:00
lrnselfreliance
6e251a009a
app: map pins and subscriptions tables distinguish pending from empty (#732)
* app: map pins and subscriptions tables distinguish pending from empty

Both lists were initialised to [], which SortableTable renders as
"loaded and empty".  The Pins tab therefore showed "No pins.
Right-click on the map to add one." until the pins arrived, and the
Manage tab's subscriptions table showed "No results" until the catalog
did.  A non-OK response (the api helpers return undefined) or a thrown
request left the same messages up permanently.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = confirmed empty, so
  SortableTable shows its placeholder while pending and each empty
  message appears only once the server confirmed the list is empty.
* A failed fetch renders an error in place of the table.
* Only the newest request for each list may write state, since edits,
  deletes, subscribes and unsubscribes refetch while the mount request
  may still be in flight.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: the regions preview waits for the catalog

"View All Regions" stayed enabled while the catalog was pending or
failed, and the modal received a fresh `catalog || []` on every
render.  Its map effect depends on that array, and ManageMap
re-renders on every status poll, so an open preview tore its WebGL
map down and rebuilt it every three seconds, and a failed catalog was
previewed as zero regions over the error message.

The button is disabled until the catalog is an array, and the modal
is mounted only then, so it receives the state array itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 16:55:58 -06:00
lrnselfreliance
ae745f1a18
app: search dropdown offers "Channels: No results" only after a search (#730)
useSuggestions started with channels: [] and reset to that on every
keystroke, and the normalizer treats an empty list as "the server
found none".  So the global search dropdown showed a "Channels: No
results" row on focus before anything was typed, and on every
keystroke through the debounce and the request.  Because that row
counts as a result, the SearchBox's own "Searching…" state never
appeared.

The channel, domain, author and subject groups now start as null,
meaning not fetched; an empty list only ever comes from the server.
The count summary tolerates the null.  Tests render the hook and
check no Channels group exists before a search or while the request
is pending, and that the "No results" row and real rows still appear
after the response.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 15:07:00 -06:00
lrnselfreliance
5dda94492e
videos: hide the transcoder's temporary output from the file refresh (#731)
ffmpeg wrote `<stem>.transcode.<container>` next to the source.  A refresh
that overlapped the transcode discovered that file, probed the half-written
output (errors in the log), and created a FileGroup and Video for a path the
final rename removed moments later.

The temporary file is now `.<stem>.transcode.<container>`: every discovery
path already skips hidden files, and staying in the same directory keeps
the rename over the final path atomic.  `transcode_tmp_path` is the single
place the name is built.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 14:59:51 -06:00
lrnselfreliance
86d4bbab98
Bookmarks: a nav menu of user links, stored in bookmarks.yaml (#729)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* bookmarks: user bookmarks in the navigation bar, stored in bookmarks.yaml

A "Bookmarks" tab between Calculators and Flasher opens on hover or click and
lists the user's bookmarks, with directories as flyout submenus (Menu.Sub).
The mobile hamburger shows directories as labelled groups instead.  The tab
always shows, with an "Edit bookmarks" item, so the feature can be found
before any bookmark exists.

Bookmarks live in a new config-only bookmarks.yaml: a nested tree where a
node with `children` is a directory and a node with `url` is a bookmark.
Every node carries an integer id so the API can rename, move and delete
without relying on names.  A URL may be a path on this WROLPi, a port on this
host (`:8096/` or `http://:8096/`, resolved against whatever hostname the
browser used, so it works over LAN IP, mDNS and the hotspot alike), or an
absolute URL.  Each bookmark chooses whether to open in a new tab.

The editor at /more/bookmarks follows the playlist items table: numbered rows,
up/down/edit/delete, and inline "Add a bookmark" / "Add a directory" forms
with a Location picker.  Deleting a directory asks first, since its contents
go with it.  The menu and the editor are in the component gallery.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: mark More current when the editor is folded inside it; rem indents in the mobile menu

Found on the QA Pi at a width where Bookmarks overflows into More: the More
button carried no mark on /more/bookmarks, because the bookmarks link has no
`to` for isLinkActive to match.  isLinkActive now knows the bookmarks link owns
its editor page, and the Bookmarks tab uses the same rule.

The mobile menu's directory labels sat a few pixels left of the labels around
them: the indent was in em, and a Menu.Label is set smaller than a Menu.Item.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: parse URLs instead of matching a prefix; validate the whole tree on every write

The URL contract was checked by prefix on both sides, and a prefix is not a
form: `//evil.com` starts like a path and `:8096@evil.com` starts like a port,
and a browser takes both off this host.  A path is now a single leading slash
that resolves to this origin, a port form is an optional scheme, `:`, digits
and then only a path, query or fragment (no `@`, no backslash, no control
characters), and anything else must parse as http(s).  The same rules live in
wrolpi/bookmarks.py and BookmarksContext.js, with the same bypass strings in
both test suites.

The config's dataclass validator only saw that `bookmarks` was a list, so a
hand-edited file or the generic config API could store a `javascript:` URL
that the bookmarks API would refuse, and the nav rendered it.  The config now
validates the tree itself on every write and import: URL forms, exactly one of
url/children, unique integer ids, boolean new_tab, and no cycles (a YAML
anchor can build one).  An invalid file is not imported and, since it was never
successfully imported, is never overwritten.

On the page, a stored URL that does not resolve is rendered as text, never as
an href.  A refused add keeps what was typed instead of clearing the form, and
an edit whose move is refused still refreshes the nav (the rename was
committed) and keeps the modal open.  An IPv6 host is bracketed before a port.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: document Bookmarks

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 13:11:36 -06:00
lrnselfreliance
9caa140fb6
app: search Other tab distinguishes pending channels from none (#728)
* app: search Other tab distinguishes pending channels from none

useSearchChannels initialised channels to [] and loading to false, and
only set loading inside its mount effect, so the Other tab's first
render painted "No Channels" with a count of 0 before the search had
run.  A failed search was rendered the same way, since the hook had no
catch and the API helper returns undefined on a non-OK response.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = no channels; loading
  is true from the first render because the search starts on mount.
* Changing tags resets to pending, and only the newest request may
  write state, so a slow response for previous tags is dropped.
* The tab shows a loader while pending, an error on failure, and
  "No Channels" only once the server confirmed there are none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useSearchChannels follows its caller's tags

The hook read its tag argument only as the initial state, so the
Other tab never refetched when the filter modal changed the URL's tags
while the tab stayed mounted; the list, including a stale "No
Channels", was whatever the mount fetched.

The internal tag state now follows the argument's contents, keyed on
the serialized list because the caller builds a new array every
render.  Tests change the hook argument after a loaded result and
expect a second search and a reset to pending, check that a new array
with the same tags does not refetch, and keep the slow-response drop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 12:01:37 -06:00
lrnselfreliance
811793d767
perpetual: clear in-progress flags after a crash, downloads only in the perpetual process, KillMode=mixed (#726)
* perpetual: clear in-progress flags after a crash; downloads only in the perpetual process

Found by a kill -9 stress test on a device:

A SIGKILL inside `with flags.file_worker_counting` (or any of the file
worker's progress flags) left the shared flag set, so every later refresh
raised "flag is already set" until the whole API was restarted.
reconcile_after_predecessor now clears the perpetual process's in-progress
flags (file_worker_*, global_refresh_active) and reports it.

create_downloads started a dispatch in whatever process called it, so an
API worker executed downloads (and could take transcode_lock) despite the
single-process design.  It now dispatches only inside the perpetual
process (or under pytest); the download loop picks new rows up within
seconds otherwise.

The shared restart cooldown drops from 60 s to 15 s: it only has to
outlast the manager acting on a request, and a replacement that dies
right away was left dead for the rest of the minute.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* systemd: KillMode=mixed for the API so main shuts its own children down

With control-group kill, SIGTERM reaches the multiprocessing Manager at
the same moment as Sanic's main process; main's cleanup then writes
worker state to a dead Manager (broken pipe) and exits 1, so a plain
`systemctl stop` is recorded as a failure whenever a child takes a
moment to stop (a transcode being killed, for instance).  Signal only
main; it stops the workers and the perpetual process itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* perpetual: clear an in-progress flag only when its holder is dead

`with flag:` now records the holder pid in shared_ctx.flag_holders and
drops it on exit.  The reconcile step clears the flags only the perpetual
process ever enters (file_worker_counting, file_worker_discovery,
global_refresh_active) whenever they are set, and the flags API workers
also enter (file_worker_modeling / indexing / cleanup via refresh_sync,
map_search_building via the map rebuild endpoints) only when the
recorded holder is dead.  file_worker_busy is no longer touched: a
collection move in an API worker holds it as a mutex, and
process_queue drops a stale one itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 11:22:58 -06:00
lrnselfreliance
83362a65c1
app: video comments and description distinguish pending from empty (#727)
* app: video comments and description distinguish pending from empty

useVideoExtras initialised both values to null and the page treated
null the same as "none", so every video open (and every prev/next)
showed "No comments have been downloaded. Refresh the video?" with a
live Refresh button until the comments arrived.  The Description tab
showed "No description available." the same way.  A failed fetch was
also rendered as "none", and the previous video's data lingered until
the next video's responses arrived.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / '' = loaded and empty.
  The API serialises a missing info_json field as null, so the hook
  normalises that to empty; consumers never see null for "none".
* Switching video resets both to pending, and a slow response for the
  previous video is dropped.
* The page renders a placeholder while pending, an error message on
  failure, and offers Refresh only once the server confirmed there are
  no comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useVideoExtras drops any request that is not the newest

Matching on the video id let a stale request land when the user left
and returned to the same video, or saved an edit while the page's
first description request was still in flight.  Each fetch now takes a
generation number and only the newest generation may write state,
per value, so a description refetch never invalidates a comments
request.

The edit dialog adopts a description that arrives while it is open
only if the user has not typed, so a draft written over the
placeholder is not wiped by the empty or failed value that follows.

Tests: 7 -> none -> 8 -> 7 with the first request rejecting last; a
save refetch beating the initial request; the draft surviving; the
pending placeholder asserted; the console spy restored in finally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 11:22:37 -06:00
lrnselfreliance
25e00f75fd
app: useLatestRequest reports loading before the debounce (#725)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
`loading` only became true once the debounce timer fired, so for the
first 300-500ms after every sendRequest the hook reported
data=null, loading=false.  Consumers branch on `loading` to choose
between a spinner and an empty state, so the global search dropdown,
the Zim search tab, and the map search rendered "No results" (or
"No Zims have been indexed") for a search that had not run yet.

Invariants now enforced and tested:
* loading is true from sendRequest until the latest request settles.
* The latest-request id is claimed synchronously in sendRequest, so a
  fetch already in flight is stale the moment a newer request is sent
  and cannot write data or clear loading during the new debounce.
* A stale request that rejects late no longer nulls newer data.
* A pending debounce is cancelled on unmount, and any in-flight fetch
  is treated as stale.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-25 14:05:08 -06:00
lrnselfreliance
bcd12c9b96
startup: run database config imports off the worker event loop (#723)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* startup: run database config imports off the worker event loop

import_all_db_configs runs in a Sanic worker's after_server_start
listener, after that worker's socket is already accepting connections.
Every import in it was a synchronous call on the event loop, so a large
tags/channels/domains/downloads config froze that worker for the whole
import and every request the kernel routed to it hung. With the default
worker count equal to CPU count, a one- or two-core host lost all or
half of the API for minutes on startup, and the frontend's 60s request
limit turned that into repeated silent retries.

Each synchronous import now runs in asyncio.to_thread. The SQLite engine
already allows cross-thread connections (check_same_thread off, NullPool)
and other startup work already uses to_thread. Imports still run one at
a time in the documented order; only the loop is no longer blocked.

Test: a ticker task must keep ticking while two deliberately slow
synchronous imports run. It scored zero ticks before this change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* startup: thread the downloads import too; test covers every import

import_downloads_config is async but never awaited anything: its body
called the synchronous DownloadManagerConfig.import_config on the event
loop, so a large download_manager.yaml still froze the worker. The
db_up check stays on the loop and only the import moves to a thread.

The responsiveness test now patches every import the startup path
performs, including that sync downloads import, with the same blocking
sleep, and requires ~90% of the total sleep budget in ticks so any one
import returning to the loop fails it. The ticker is awaited after
cancel so the task does not outlive the loop. Docstring trimmed to the
one constraint not visible at the call site.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-24 18:22:52 -06:00
lrnselfreliance
92fa3f785f
files: shared modeler batch loop for video, doc, zim, archive (#722)
* files: shared modeler batch loop for video, doc, zim, archive

Paging, skip-ids, progress, and cancel yields live in
wrolpi.files.modeler.run_modeler_loop.  Each module still owns select /
prepare / apply and its transaction style: video probes outside the write
lock, docs use one write per file with lock retries, zim and archive use
one write per page.  Archive HTML that is not a SingleFile raises
SkipModeler so apply_indexers can claim it.

* files: quieter batch-modeler errors; drop unused imports

Break out of a batch page on InvalidRequestError after a flush failure
so the rest of the page does not each log the same poisoned session.
Re-raise PYTEST apply failures without a misleading commit-failure
message.  Remove asyncio/List/OperationalError/PYTEST/Iterable leftovers
from docs, archive, and zim.
2026-09-24 16:47:46 -06:00
lrnselfreliance
2d0f433450
files: split FileWorker move and reorganize into mixins (#719)
FileWorker stays the queue, status, and refresh apply path.  Move planning
and execute/revert live in wrolpi.files.move; collection reorganize jobs
live in wrolpi.files.reorganize_job.  Perpetual-process dispatch is
unchanged.

Existing `from wrolpi.files.worker import build_move_plan_bulk` still works.
2026-09-24 14:17:42 -06:00
380 changed files with 32461 additions and 10826 deletions

View file

@ -1,253 +0,0 @@
# Use the latest 2.1 version of CircleCI pipeline process engine.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:
api-tests:
docker:
- image: cimg/python:3.13
# `large` (4 vCPU, 8 GB) so pytest-xdist's `-nauto` has room to scale.
resource_class: large
steps:
- checkout
- run: sudo apt-get update
- run: sudo apt-get install -y ffmpeg catdoc
- restore_cache:
key: deps-3.13-{{ checksum "requirements.txt" }}-6
- run:
name: Install Requirements
command: |
# A restored venv points at the exact interpreter that built it. CircleCI's
# cimg/python:3.13 gets patch bumps, and this cache key does not name the
# patch, so a venv cached before a bump has a dangling bin/python3 -- and
# `python3 -m venv` over it fails with `[Errno 2]` instead of repairing it,
# which broke every Python job. Rebuild when it cannot run: that costs a
# reinstall only on the first run after a bump.
if ! ./venv/bin/python3 --version >/dev/null 2>&1; then rm -rf venv; fi
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- save_cache:
key: deps-3.13-{{ checksum "requirements.txt" }}-6
paths:
- "venv"
# `-nauto` parallelizes across all available cores via pytest-xdist.
# `-svv` was dumping the full test stdout to CircleCI logs on every run,
# which inflated wall time and produced multi-MB log artifacts; dropped.
- run:
command: './venv/bin/pytest -nauto --junitxml=test-results.xml'
- store_test_results:
path: test-results.xml
# Do the API and the frontend's test fixtures still agree?
#
# `api-tests` runs these too, as part of the whole suite. This job exists for the
# signal: fixture drift is silent by nature -- the frontend suite passes against a body the
# API stopped sending -- so it is worth a check whose name says what broke, failing in under a
# minute rather than somewhere inside seventeen hundred results.
#
# It needs no ffmpeg or catdoc (verified), and shares the dependency cache key with the other
# Python jobs, so it costs a venv restore and one test file.
api-contract-tests:
docker:
- image: cimg/python:3.13
resource_class: medium
steps:
- checkout
- restore_cache:
key: deps-3.13-{{ checksum "requirements.txt" }}-6
- run:
name: Install Requirements
command: |
# A restored venv points at the exact interpreter that built it. CircleCI's
# cimg/python:3.13 gets patch bumps, and this cache key does not name the
# patch, so a venv cached before a bump has a dangling bin/python3 -- and
# `python3 -m venv` over it fails with `[Errno 2]` instead of repairing it,
# which broke every Python job. Rebuild when it cannot run: that costs a
# reinstall only on the first run after a bump.
if ! ./venv/bin/python3 --version >/dev/null 2>&1; then rm -rf venv; fi
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- run:
name: Fixtures match the OpenAPI spec, and the spec matches the handlers
command: './venv/bin/pytest wrolpi/test/test_api_fixtures.py -v'
api-alembic-migrations:
docker:
- image: cimg/python:3.13
resource_class: medium
steps:
- checkout
- restore_cache:
key: deps-3.13-{{ checksum "requirements.txt" }}-6
- run:
name: Install Requirements
command: |
# A restored venv points at the exact interpreter that built it. CircleCI's
# cimg/python:3.13 gets patch bumps, and this cache key does not name the
# patch, so a venv cached before a bump has a dangling bin/python3 -- and
# `python3 -m venv` over it fails with `[Errno 2]` instead of repairing it,
# which broke every Python job. Rebuild when it cannot run: that costs a
# reinstall only on the first run after a bump.
if ! ./venv/bin/python3 --version >/dev/null 2>&1; then rm -rf venv; fi
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- run:
name: Create Media Directory
command: mkdir -p /tmp/wrolpi-media/config
- run:
name: Run Alembic Migrations
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db upgrade
- run:
name: Check for Missing Migrations
command: MEDIA_DIRECTORY=/tmp/wrolpi-media ./venv/bin/python ./main.py db check
sanic-api-start:
docker:
- image: cimg/python:3.13
steps:
- checkout
- restore_cache:
key: deps-3.13-{{ checksum "requirements.txt" }}-6
- run:
name: Install dependencies
command: |
# A restored venv points at the exact interpreter that built it. CircleCI's
# cimg/python:3.13 gets patch bumps, and this cache key does not name the
# patch, so a venv cached before a bump has a dangling bin/python3 -- and
# `python3 -m venv` over it fails with `[Errno 2]` instead of repairing it,
# which broke every Python job. Rebuild when it cannot run: that costs a
# reinstall only on the first run after a bump.
if ! ./venv/bin/python3 --version >/dev/null 2>&1; then rm -rf venv; fi
python -m venv venv
. venv/bin/activate
pip install -r requirements.txt
sudo mkdir -p /media/wrolpi
- run:
name: Start Sanic API
command: |
. venv/bin/activate
sanic main:api_app -H 0.0.0.0 -p 8081
background: true
- run:
name: Test Sanic API
command: |
sleep 20
curl http://0.0.0.0:8081/api/echo
- run:
name: Stop Sanic API
command: pkill -f sanic
react-app-start:
docker:
- image: cimg/node:20.19
resource_class: large
steps:
- checkout
- restore_cache:
key: app-{{ checksum "app/package-lock.json" }}
- run: cd app && npm install
- save_cache:
key: app-{{ checksum "app/package-lock.json" }}
paths:
- ./app/node_modules
- run:
name: Build React App
command: cd app && DISABLE_ESLINT_PLUGIN=true npm run build
- run:
name: Start React App
command: cd app && DISABLE_ESLINT_PLUGIN=true npm run start
background: true
- run:
name: Test React App
command: |
sleep 20
curl http://0.0.0.0:3000/manifest.json
- run:
name: Stop React App
command: pkill -f npm
# Jest tests moved to GitHub Actions (.github/workflows/jest-tests.yml)
# Component specs live under `src/`, so they need `cypress run --component`; the e2e job's
# `cypress run` will not match them. Cypress starts its own dev server, so no `npm start`.
cypress-component-tests:
docker:
- image: cimg/node:20.19-browsers
resource_class: large
steps:
- checkout
- restore_cache:
key: app-browsers-{{ checksum "app/package-lock.json" }}
- run:
name: Install Dependencies
command: cd app && npm install
- run:
name: Install Cypress Binary
command: cd app && npx cypress install
- run:
name: Run Cypress Component Tests
command: cd app && CI=true npx cypress run --component
- store_artifacts:
path: app/cypress/screenshots
cypress-e2e-tests:
docker:
- image: cimg/node:20.19-browsers
resource_class: large
steps:
- checkout
- restore_cache:
key: app-browsers-{{ checksum "app/package-lock.json" }}
- run:
name: Install Dependencies
command: cd app && npm install
# The Cypress binary lives outside node_modules, so caching only
# node_modules lets the two drift apart: a restored cache satisfies npm,
# npm skips the postinstall download, and the binary for the new version
# is never fetched. Cache both, and install explicitly (a no-op when the
# binary is already present).
- run:
name: Install Cypress Binary
command: cd app && npx cypress install
- save_cache:
key: app-browsers-{{ checksum "app/package-lock.json" }}
paths:
- ./app/node_modules
- ~/.cache/Cypress
- run:
name: Start React App in Background
command: cd app && DISABLE_ESLINT_PLUGIN=true npm start
background: true
- run:
name: Wait for React App
command: |
for i in {1..60}; do
if curl -s http://localhost:3000 > /dev/null; then
echo "React app is ready!"
exit 0
fi
echo "Waiting for React app... ($i/60)"
sleep 2
done
echo "React app failed to start"
exit 1
- run:
name: Run Cypress Tests
command: cd app && CI=true npx cypress run
- store_artifacts:
path: app/cypress/videos
- store_artifacts:
path: app/cypress/screenshots
workflows:
wrolpi-api-tests:
jobs:
- sanic-api-start
- api-tests
- api-contract-tests
- api-alembic-migrations
wrolpi-app-test:
jobs:
- react-app-start
# jest-tests moved to GitHub Actions
- cypress-e2e-tests
- cypress-component-tests

View file

@ -7,6 +7,11 @@ updates:
interval: "monthly"
target-branch: "master"
open-pull-requests-limit: 5
ignore:
# SQLAlchemy 2.1 is a minor release with its own deprecations; upgrade it
# deliberately rather than by an automatic bump.
- dependency-name: "SQLAlchemy"
versions: [">=2.1"]
groups:
python-dependencies:
patterns:
@ -37,6 +42,42 @@ updates:
interval: "monthly"
target-branch: "master"
open-pull-requests-limit: 5
ignore:
# Major upgrades below need their own migration branch, not a bump PR.
# maplibre-gl 6: /map build depends on pinned babel/terser handling.
- dependency-name: "maplibre-gl"
versions: [">=6"]
# react-dropzone 15+ changed the accept API; 20 requires Node >=22.
- dependency-name: "react-dropzone"
versions: [">=15"]
# React 19: @react-three/fiber 8 pins react <19.
- dependency-name: "react"
update-types: ["version-update:semver-major"]
- dependency-name: "react-dom"
update-types: ["version-update:semver-major"]
- dependency-name: "@types/react"
update-types: ["version-update:semver-major"]
- dependency-name: "@types/react-dom"
update-types: ["version-update:semver-major"]
# TypeScript 6+/7 breaks the react-scripts toolchain.
- dependency-name: "typescript"
versions: [">=6"]
# Mantine 9: all @mantine packages move together with app code changes.
- dependency-name: "@mantine/*"
update-types: ["version-update:semver-major"]
# Cypress 16 requires Node >=22; CI and the app image run Node 20.
- dependency-name: "cypress"
versions: [">=16"]
# react-router 8 is a routing migration across ~40 files.
- dependency-name: "react-router"
update-types: ["version-update:semver-major"]
# @testing-library/react 14+ async act semantics break menu/popover tests.
- dependency-name: "@testing-library/react"
update-types: ["version-update:semver-major"]
# Cypress component testing resolves the top-level webpack-dev-server and only
# accepts major 5; 6 crashes cypress.config.js ("Unexpected major version").
- dependency-name: "webpack-dev-server"
versions: [">=6"]
groups:
npm-dependencies:
patterns:

145
.github/workflows/backend.yml vendored Normal file
View file

@ -0,0 +1,145 @@
name: Backend Tests
# Same triggers as jest-tests.yml: `pull_request` is unfiltered so pull requests into feature
# branches are checked too, not only those into master.
on:
push:
branches: [master, feature/*]
pull_request:
# A new push to a pull request makes the run for the previous push worthless; cancel it so it
# does not hold runners (the free plan allows 20 concurrent jobs). Runs on master always finish.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
# The suite is split by directory so it finishes in a few minutes, and so a check's name says
# which area broke. The groups are balanced by test time; `rest` runs everything the others do
# not, so a new directory is tested before anyone assigns it. A path added to any other group
# must also be added to `rest` as an --ignore, or its tests run twice.
api-tests:
name: api-tests (${{ matrix.name }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- name: core
paths: wrolpi/test
- name: media
paths: modules/videos modules/zim modules/docs modules/map modules/inventory modules/flasher
- name: files
paths: wrolpi/files wrolpi/collections
- name: rest
paths: >-
--ignore=wrolpi/test
--ignore=modules/videos --ignore=modules/zim --ignore=modules/docs --ignore=modules/map
--ignore=modules/inventory --ignore=modules/flasher
--ignore=wrolpi/files --ignore=wrolpi/collections
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install system packages
run: sudo apt-get update && sudo apt-get install -y ffmpeg catdoc
- name: Setup Python
uses: actions/setup-python@v5
with:
# Trixie ships 3.13.
python-version: '3.13'
cache: 'pip'
cache-dependency-path: requirements.txt
- name: Install dependencies
run: pip install -r requirements.txt
# `logical`, not `auto`: with psutil installed, `auto` counts physical cores, which is two on
# the four-vCPU runner. No `-v`: the full test output inflates wall time and logs.
- name: Run pytest
env:
PATHS: ${{ matrix.paths }}
run: pytest -n logical $PATHS --junitxml=test-results.xml
- name: Upload test results
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: api-test-results-${{ matrix.name }}
path: test-results.xml
retention-days: 30
# Do the API and the frontend's test fixtures still agree?
#
# `api-tests` runs these too, as part of the whole suite. This job exists for the signal:
# fixture drift is silent by nature -- the frontend suite passes against a body the API stopped
# sending -- so it is worth a check whose name says what broke, failing in under a minute rather
# than somewhere inside seventeen hundred results. It needs no ffmpeg or catdoc.
api-contract-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.13'
cache: 'pip'
cache-dependency-path: requirements.txt
- name: Install dependencies
run: pip install -r requirements.txt
- name: Fixtures match the OpenAPI spec, and the spec matches the handlers
run: pytest wrolpi/test/test_api_fixtures.py -v
# The migrations apply to an empty database, the models match them, and the API starts.
api-migrations-and-start:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.13'
cache: 'pip'
cache-dependency-path: requirements.txt
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run Alembic migrations
env:
MEDIA_DIRECTORY: ${{ runner.temp }}/wrolpi-media
run: |
mkdir -p "$MEDIA_DIRECTORY/config"
python ./main.py db upgrade
- name: Check for missing migrations
env:
MEDIA_DIRECTORY: ${{ runner.temp }}/wrolpi-media
run: python ./main.py db check
- name: Start Sanic API
run: |
sudo mkdir -p /media/wrolpi
nohup sanic main:api_app -H 0.0.0.0 -p 8081 > sanic.log 2>&1 &
for i in $(seq 1 60); do
if curl -sf http://localhost:8081/api/echo; then
echo
echo "Sanic API is up"
exit 0
fi
sleep 2
done
echo "Sanic API failed to start"
cat sanic.log
exit 1
- name: Stop Sanic API
if: ${{ always() }}
run: pkill -f sanic || true

121
.github/workflows/cypress.yml vendored Normal file
View file

@ -0,0 +1,121 @@
name: Cypress Tests
on:
push:
branches: [master, feature/*]
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
# Component specs live under `src/`, so they need `cypress run --component`; the e2e job's
# `cypress run` will not match them. Cypress starts its own dev server, so no `npm start`.
cypress-component-tests:
runs-on: ubuntu-latest
defaults:
run:
working-directory: app
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.19'
cache: 'npm'
cache-dependency-path: app/package-lock.json
# The Cypress binary lives outside node_modules, and setup-node only caches ~/.npm. Key it
# on the lockfile so a Cypress bump fetches the matching binary.
- name: Cache Cypress binary
uses: actions/cache@v4
with:
path: ~/.cache/Cypress
key: cypress-${{ runner.os }}-${{ hashFiles('app/package-lock.json') }}
- name: Install dependencies
run: npm install
# A no-op when the cached binary matches.
- name: Install Cypress binary
run: npx cypress install
- name: Run Cypress component tests
run: npx cypress run --component
- name: Upload screenshots
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: cypress-component-screenshots
path: app/cypress/screenshots
retention-days: 14
# Waiting for `npm start` to serve the app also checks that the dev server starts.
cypress-e2e-tests:
runs-on: ubuntu-latest
defaults:
run:
working-directory: app
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.19'
cache: 'npm'
cache-dependency-path: app/package-lock.json
- name: Cache Cypress binary
uses: actions/cache@v4
with:
path: ~/.cache/Cypress
key: cypress-${{ runner.os }}-${{ hashFiles('app/package-lock.json') }}
- name: Install dependencies
run: npm install
- name: Install Cypress binary
run: npx cypress install
# GitHub Actions sets `CI=true`, which makes cypress.config.js use http://localhost:3000 as
# the base URL.
- name: Start React app
env:
DISABLE_ESLINT_PLUGIN: 'true'
BROWSER: none
run: |
nohup npm start > react.log 2>&1 &
for i in $(seq 1 90); do
if curl -sf http://localhost:3000/manifest.json > /dev/null; then
echo "React app is ready"
exit 0
fi
sleep 2
done
echo "React app failed to start"
cat react.log
exit 1
- name: Run Cypress e2e tests
run: npx cypress run
- name: Upload videos and screenshots
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: cypress-e2e-media
path: |
app/cypress/videos
app/cypress/screenshots
retention-days: 14
if-no-files-found: ignore

3
.gitignore vendored
View file

@ -150,3 +150,6 @@ app/cypress/videos
release/latest.json
perf_results/
scripts/api_perf_hosts.json
# Tile cache for scripts/build_solar_climatology.py
/scripts/.solar_climatology_cache/

View file

@ -3,8 +3,10 @@
There are several ways to run WROLPi. Pick the one that matches your hardware:
* [WROLPi Portable (Live USB)](#wrolpi-portable-live-usb) — boot a full WROLPi from a USB stick on any x86 PC or laptop, no installation required.
* [Debian 12 Install](#debian-12-install) — install WROLPi onto an x86 PC or laptop.
* [Debian 13 Install](#debian-13-install) — install WROLPi onto an x86 PC or laptop.
* [Raspberry Pi Install](#raspberry-pi-install) — install WROLPi onto a Raspberry Pi with an external drive.
After installing, follow [First Boot](#first-boot) to set up your drive.
* [Docker](docker/README.md) — run the WROLPi containers on an existing Linux host (best for trying it out or development).
See [UPGRADE.md](UPGRADE.md) for upgrading an existing WROLPi.
@ -36,6 +38,7 @@ Steps:
minute or two; progress is shown on screen.
5. Once the desktop loads, browse to https://wrolpi.local or the IP address of
your WROLPi. Wi-Fi hotspot and all other features work as usual.
6. To use a terminal remotely: `ssh user@wrolpi.local`, password `live`.
Notes:
@ -52,30 +55,50 @@ Notes:
* **Upgrading.** See [Upgrading a WROLPi Portable USB](UPGRADE.md#upgrading-a-wrolpi-portable-usb)
to move to a newer ISO without losing your library.
## Debian 12 Install
## Debian 13 Install
Steps necessary to initialize your WROLPi after installing the Debian image from wrolpi.org
Install WROLPi onto the internal disk of an x86 PC or laptop. This uses the same
`WROLPi-v*-amd64.iso` as WROLPi Portable.
1. Download and copy a pre-built Debian image from https://wrolpi.org onto a USB thumb-drive (USB 2 recommended).
2. Insert the thumb-drive into the laptop, boot to the thumb-drive.
1. Select "Start Installer".
2. Install Debian 12 as you would like.
1. It is recommended to use the hostname **wrolpi**.
2. (WROLPi will be installed during the installation without your intervention.)
3. Unplug the thumb-drive after the installation has completed.
4. Login as the user _you_ created during installation.
5. Initialize the WROLPi databases using the repair script: `/opt/wrolpi/repair.sh`
6. Reboot: `reboot`
7. Browse to https://wrolpi.local or the IP address of your WROLPi!
1. Download the latest `WROLPi-v*-amd64.iso` from [wrolpi.org](https://wrolpi.org) and flash it to a USB drive
(see [WROLPi Portable](#wrolpi-portable-live-usb) step 2).
2. Boot the PC from the USB drive. At the boot menu, choose **Install** or **Graphical Install**.
3. Install Debian 13 as you would like.
1. It is recommended to use the hostname **wrolpi**.
2. WROLPi is installed along with Debian; there is nothing extra to select.
4. Unplug the USB drive after the installation has completed, and reboot.
5. Continue with [First Boot](#first-boot).
## Raspberry Pi Install
Steps necessary to initialize your WROLPi after installing the Raspberry Pi image from wrolpi.org
Install WROLPi onto a Raspberry Pi. An external drive is recommended for your library.
1. Download and copy a pre-built image from https://wrolpi.org onto an SD card.
2. Boot the Raspberry Pi, login with username pi and password wrolpi.
3. Modify fstab to mount your external drive to /media/wrolpi (modify this command to match your system).
* `echo '/dev/sda1 /media/wrolpi auto defaults,nofail 0 0' | sudo tee -a /etc/fstab`
4. Initialize the WROLPi databases using the repair script: `sudo /opt/wrolpi/repair.sh`
5. Reboot `sudo reboot`
6. Join the Hotspot or browse to https://wrolpi.local or the IP address of your WROLPi!
1. Download the latest `WROLPi-v*-aarch64-desktop.img.xz` from [wrolpi.org](https://wrolpi.org).
2. Flash it to an SD card using Raspberry Pi Imager.
3. Plug in your external drive, insert the SD card, and boot the Raspberry Pi.
4. On the first boot, Raspberry Pi OS asks you to create your own user.
5. Continue with [First Boot](#first-boot).
## First Boot
These steps are the same for every install.
1. Join the WROLPi hotspot (network **WROLPi**, password **wrolpi hotspot**; it starts on every boot), or connect
your WROLPi to your network, then browse to https://wrolpi.local (or the IP address of your WROLPi).
2. **Choose your drive.**
* **Raspberry Pi and Debian:** until a drive is set up, you will see the **Controller** with a *Welcome to WROLPi*
wizard (also at http://wrolpi.local). Select the drive which will hold your library; WROLPi mounts it
at `/media/wrolpi`, adds it to `/etc/fstab`, and runs the repair script for you. If the drive already holds a
WROLPi library (from another WROLPi, or a backup), its configuration is used. No external drive? Choose **Run
repair without external drive** to keep your library on the internal disk or SD card.
* **WROLPi Portable:** nothing to do; your library lives in the persistence partition on the USB drive.
3. **Trust the certificate.** Your browser will warn about the HTTPS certificate because it was made by your WROLPi,
not by a public authority. Download https://wrolpi.local/ca.crt and trust it on each device you use; see
*Help > Certificates* on your WROLPi for per-device instructions.
4. **Wait for the first refresh.** WROLPi refreshes all your files when it first starts so it can find anything already
on the drive. Your library looks empty, and Download and Upload are disabled, until the refresh completes.
5. Start adding content: download a video channel, archive a web page, or add a Zim (Wikipedia) file. The *Help* page
on your WROLPi explains each module.
You do not need to edit `/etc/fstab` or run `repair.sh` by hand. `sudo /opt/wrolpi/repair.sh` is still available if
your WROLPi ever needs repairing.

View file

@ -1,6 +1,6 @@
# WROLPi
[![Build Status](https://img.shields.io/circleci/build/github/lrnselfreliance/wrolpi/release?label=release%20build)](https://app.circleci.com/pipelines/github/lrnselfreliance/wrolpi?branch=release&filter=all)
[![Backend Tests](https://github.com/lrnselfreliance/wrolpi/actions/workflows/backend.yml/badge.svg?branch=master)](https://github.com/lrnselfreliance/wrolpi/actions/workflows/backend.yml?query=branch%3Amaster)
[![License: GPL v3](https://img.shields.io/github/license/lrnselfreliance/wrolpi?style=flat-square)](https://github.com/lrnselfreliance/wrolpi/blob/master/LICENSE)
[![Discord](https://img.shields.io/discord/900430987987681330?label=Discord&logo=discord)](https://discord.gg/HrwFk7nqA2)
[![Gitpod](https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod)](https://gitpod.io/#https://github.com/lrnselfreliance/wrolpi)
@ -93,7 +93,7 @@ More Docker documentation is available in [docker/README.md](docker/README.md)
Full installation instructions are in [INSTALL.md](INSTALL.md). Pick the option that matches your hardware:
* **[WROLPi Portable (Live USB)](INSTALL.md#wrolpi-portable-live-usb)** — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
* **[Debian 12](INSTALL.md#debian-12-install)** — install WROLPi onto an x86 PC or laptop.
* **[Debian 13](INSTALL.md#debian-13-install)** — install WROLPi onto an x86 PC or laptop.
* **[Raspberry Pi](INSTALL.md#raspberry-pi-install)** — install WROLPi onto a Raspberry Pi with an external drive.
* **[Docker](docker/README.md)** — run the WROLPi containers on an existing Linux host.

View file

@ -3,7 +3,7 @@
# Table of Contents
* [Upgrading Docker containers](#upgrading-docker-containers)
* [Upgrading Raspberry Pi or Debian 12 Installation](#upgrading-raspberry-pi-or-debian-12-installation)
* [Upgrading Raspberry Pi or Debian Installation](#upgrading-raspberry-pi-or-debian-installation)
* [Upgrading a WROLPi Portable USB](#upgrading-a-wrolpi-portable-usb)
## Upgrading Docker containers
@ -24,7 +24,9 @@
7. Start all docker containers
* `docker-compose up -d`
## Upgrading Raspberry Pi or Debian 12 Installation
## Upgrading Raspberry Pi or Debian Installation
A Debian 12 (bookworm) WROLPi stays on the final version for Debian 12; reinstall on Debian 13 to get newer versions.
1. Run the upgrade script
* `sudo /bin/bash /opt/wrolpi/upgrade.sh`

View file

@ -6,9 +6,10 @@ from alembic import context
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
# Configure logging only for the alembic command line. WROLPi also migrates in-process at startup
# (db_bootstrap), where fileConfig would replace the API's handlers and disable its existing loggers.
if config.cmd_opts is not None:
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
@ -23,6 +24,7 @@ from wrolpi.collections.models import Collection, CollectionItem # noqa: F401
from modules.videos.models import Video, Channel # noqa: F401
from modules.archive.models import Archive # noqa: F401
from modules.zim.models import Zim, TagZimEntry, ZimSubscription # noqa: F401
from modules.repos.models import Repository # noqa: F401
target_metadata = Base.metadata
@ -96,7 +98,10 @@ def run_migrations_online():
connectable = create_wrolpi_engine(get_url())
with connectable.connect() as connection:
# Autocommit: each migration statement is its own transaction. A migration may call WROLPi
# functions which write on their own connection; a migration connection holding an open read
# transaction could not then upgrade to a writer ("database is locked").
with connectable.connect().execution_options(isolation_level='AUTOCOMMIT') as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,

View file

@ -0,0 +1,45 @@
"""Maintain collection.item_count/total_size for Domain Collections with triggers.
Listing Domain Collections summed file_group.size through an archive join on every request,
one random page read per archive. The new triggers keep the summary columns current
(as the channel triggers already do for Channels); this migration installs them and
backfills existing Domain Collections.
Revision ID: 2026_09_27_1200
Revises: 2026_09_20_1200
"""
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_09_27_1200'
down_revision = '2026_09_20_1200'
branch_labels = None
depends_on = None
TRIGGERS = (
'archive_insert_collection_summary',
'archive_delete_collection_summary',
'archive_update_collection_summary',
'file_group_size_collection_summary',
)
def upgrade():
from wrolpi.schema_ddl import install_raw_ddl
# Idempotent: only the new triggers are created.
install_raw_ddl(op.get_bind())
op.execute('''
UPDATE collection SET
item_count = (SELECT COUNT(*) FROM archive WHERE collection_id = collection.id),
total_size = (SELECT COALESCE(SUM(fg.size), 0) FROM archive a
LEFT JOIN file_group fg ON a.file_group_id = fg.id
WHERE a.collection_id = collection.id)
WHERE kind = 'domain'
''')
def downgrade():
for name in TRIGGERS:
op.execute(f'DROP TRIGGER IF EXISTS {name}')

View file

@ -0,0 +1,29 @@
"""Covering indexes for the Collection listing.
Listing Collections read every channel row (for video_count/total_size) and every download
row of those Collections (for the minimum frequency). Both tables store yt-dlp info_json,
so their rows are tens of KB and each sits on its own scattered page; from a cold disk cache
that was 10+ seconds. These indexes answer both queries without touching a row.
Revision ID: 2026_09_28_1200
Revises: 2026_09_27_1200
"""
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_09_28_1200'
down_revision = '2026_09_27_1200'
branch_labels = None
depends_on = None
def upgrade():
op.execute('CREATE INDEX IF NOT EXISTS channel_collection_summary_idx'
' ON channel (collection_id, video_count, total_size)')
op.execute('CREATE INDEX IF NOT EXISTS download_collection_frequency_idx'
' ON download (collection_id, frequency)')
def downgrade():
op.execute('DROP INDEX IF EXISTS channel_collection_summary_idx')
op.execute('DROP INDEX IF EXISTS download_collection_frequency_idx')

View file

@ -0,0 +1,31 @@
"""Retag existing markdown file groups as text/markdown.
A refresh skips a group whose files have not changed, so markdown indexed before text/markdown
existed would keep the type libmagic gave it (text/plain, or a programming language when the file
holds code) and the indexer that type chose. Marking the rows unindexed lets apply_indexers
re-index them with TextIndexer. SQLite's LIKE is case-insensitive for ASCII, so `.MD` matches.
Revision ID: 2026_10_05_1200
Revises: 2026_09_28_1200
"""
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_05_1200'
down_revision = '2026_09_28_1200'
branch_labels = None
depends_on = None
def upgrade():
op.execute("""
UPDATE file_group SET mimetype = 'text/markdown', "indexed" = 0
WHERE (primary_path LIKE '%.md' OR primary_path LIKE '%.markdown')
AND mimetype LIKE 'text/%'
AND mimetype != 'text/markdown'
""")
def downgrade():
# The original types are not recorded; text/plain is what libmagic reports for most markdown.
op.execute("""UPDATE file_group SET mimetype = 'text/plain', "indexed" = 0 WHERE mimetype = 'text/markdown'""")

View file

@ -0,0 +1,43 @@
"""Add the repository table for git Repos (Collections with kind='repo').
Revision ID: 2026_10_06_1200
Revises: 2026_10_05_1200
"""
import sqlalchemy as sa
from alembic import op
import wrolpi.dates
# revision identifiers, used by Alembic.
revision = '2026_10_06_1200'
down_revision = '2026_10_05_1200'
branch_labels = None
depends_on = None
def upgrade():
op.create_table(
'repository',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('url', sa.String(), nullable=False),
sa.Column('host', sa.String(), nullable=True),
sa.Column('owner', sa.String(), nullable=True),
sa.Column('mode', sa.String(), nullable=False),
sa.Column('branch', sa.String(), nullable=True),
sa.Column('default_branch', sa.String(), nullable=True),
sa.Column('head_sha', sa.String(), nullable=True),
sa.Column('head_date', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('head_message', sa.Text(), nullable=True),
sa.Column('last_fetch', wrolpi.dates.TZDateTime(), nullable=True),
sa.Column('size', sa.BigInteger(), nullable=True),
sa.Column('readme_path', sa.String(), nullable=True),
sa.Column('collection_id', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['collection_id'], ['collection.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('collection_id'),
sa.UniqueConstraint('url'),
)
def downgrade():
op.drop_table('repository')

View file

@ -0,0 +1,39 @@
"""Search git Repos by name and README: repository.search_name/readme_text and repository_fts.
Revision ID: 2026_10_06_1300
Revises: 2026_10_06_1200
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_06_1300'
down_revision = '2026_10_06_1200'
branch_labels = None
depends_on = None
def upgrade():
from wrolpi.schema_ddl import install_raw_ddl
op.add_column('repository', sa.Column('search_name', sa.String(), nullable=True))
op.add_column('repository', sa.Column('readme_text', sa.Text(), nullable=True))
op.execute('''
UPDATE repository SET search_name = TRIM(
COALESCE((SELECT name FROM collection WHERE collection.id = repository.collection_id), '')
|| ' ' || COALESCE(owner, ''))
''')
# Creates repository_fts and its triggers now that the columns exist. The README text is filled by the next
# update of each Repo.
install_raw_ddl(op.get_bind())
op.execute("INSERT INTO repository_fts(repository_fts) VALUES('rebuild')")
def downgrade():
for name in ('repository_fts_ai', 'repository_fts_ad', 'repository_fts_au'):
op.execute(f'DROP TRIGGER IF EXISTS {name}')
op.execute('DROP TABLE IF EXISTS repository_fts')
with op.batch_alter_table('repository') as batch_op:
batch_op.drop_column('readme_text')
batch_op.drop_column('search_name')

View file

@ -0,0 +1,22 @@
"""A git Repo can also download its submodules: repository.submodules.
Revision ID: 2026_10_06_1400
Revises: 2026_10_06_1300
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_06_1400'
down_revision = '2026_10_06_1300'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('repository', sa.Column('submodules', sa.Boolean(), server_default=sa.false(), nullable=False))
def downgrade():
with op.batch_alter_table('repository') as batch_op:
batch_op.drop_column('submodules')

View file

@ -0,0 +1,22 @@
"""A git Repo's clone holds a random token: repository.clone_token.
Revision ID: 2026_10_06_1500
Revises: 2026_10_06_1400
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_06_1500'
down_revision = '2026_10_06_1400'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('repository', sa.Column('clone_token', sa.String(), nullable=True))
def downgrade():
with op.batch_alter_table('repository') as batch_op:
batch_op.drop_column('clone_token')

View file

@ -0,0 +1,24 @@
"""A FileGroup records how far the user is through it: file_group.progress and file_group.position.
Revision ID: 2026_10_08_1200
Revises: 2026_10_06_1500
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_08_1200'
down_revision = '2026_10_06_1500'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('file_group', sa.Column('progress', sa.Float(), nullable=True))
op.add_column('file_group', sa.Column('position', sa.JSON(), nullable=True))
def downgrade():
# Not batch_alter_table: rebuilding file_group drops its FTS triggers (see 2026_09_20_1200).
op.execute('ALTER TABLE file_group DROP COLUMN position')
op.execute('ALTER TABLE file_group DROP COLUMN progress')

View file

@ -0,0 +1,27 @@
"""Each database config's version, so an import can tell whether the database or the file is newer
(see wrolpi.config_version).
Revision ID: 2026_10_24_1200
Revises: 2026_10_08_1200
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '2026_10_24_1200'
down_revision = '2026_10_08_1200'
branch_labels = None
depends_on = None
def upgrade():
# No rows: a config's next import (without deletes) records its version.
op.create_table(
'config_version',
sa.Column('name', sa.String(), primary_key=True),
sa.Column('version', sa.Integer(), nullable=False),
)
def downgrade():
op.drop_table('config_version')

View file

@ -0,0 +1,175 @@
"""Tag names have no leading or trailing whitespace.
A Tag whose name had whitespace is renamed. Tags whose names are the same once stripped are merged into one: it
keeps every tagged file, Zim entry and Collection, and the first color (by Tag id) of the merged Tags. Downloads
and the config files that name Tags are rewritten the same way, so the next import finds them in step with the
database (an import of the old names would create the padded Tags again, or delete the stripped ones).
Revision ID: 2026_10_25_1200
Revises: 2026_10_24_1200
"""
import json
import logging
import yaml
from alembic import op
from wrolpi.common import get_media_directory, write_config_data
# revision identifiers, used by Alembic.
revision = '2026_10_25_1200'
down_revision = '2026_10_24_1200'
branch_labels = None
depends_on = None
logger = logging.getLogger(__name__)
# The configs which name Tags.
CONFIG_FILES = ('tags.yaml', 'download_manager.yaml', 'channels.yaml', 'domains.yaml', 'playlists.yaml',
'repos.yaml', 'videos_downloader.yaml', 'archives_downloader.yaml')
def _strip(name):
"""A Tag name without its whitespace; a name that would be empty is kept (it cannot be renamed)."""
if isinstance(name, str) and name.strip():
return name.strip()
return name
def _strip_names(names: list) -> list:
stripped = list()
for name in map(_strip, names):
if name not in stripped:
stripped.append(name)
return stripped
def _strip_nested(data):
"""Strip the Tag names in a config's (or a Download's settings') `tag_name`, `tag_names` and
`channel_tag_name` values, at any depth."""
if isinstance(data, list):
return [_strip_nested(i) for i in data]
if not isinstance(data, dict):
return data
result = dict()
for key, value in data.items():
if key == 'tag_name':
value = _strip(value)
elif key in ('tag_names', 'channel_tag_name') and isinstance(value, list):
value = _strip_names(value)
else:
value = _strip_nested(value)
result[key] = value
return result
def _strip_rows(rows: list, key_length: int) -> list:
"""tags.yaml's tag_files or tag_zims, each row keyed by its first `key_length` values (the Tag name first)."""
result, seen = list(), set()
for row in rows:
if isinstance(row, list) and row:
row = [_strip(row[0]), *row[1:]]
key = json.dumps(row[:key_length], default=str)
if key in seen:
continue
seen.add(key)
result.append(row)
return result
def _strip_tags_config(config: dict, colors: dict) -> dict:
"""`colors`: each renamed Tag's color in the database, which the file must match (an import copies the file's
color into the database)."""
config = dict(config)
if isinstance(config.get('tags'), dict):
tags = dict()
for name, attrs in config['tags'].items():
stripped = _strip(name)
if stripped not in tags:
tags[stripped] = attrs
elif isinstance(attrs, dict) and attrs.get('color') and not (tags[stripped] or dict()).get('color'):
# The first color of the merged Tags.
tags[stripped] = {**(tags[stripped] or dict()), 'color': attrs['color']}
for name in tags.keys() & colors.keys():
tags[name] = {**(tags[name] or dict()), 'color': colors[name]}
config['tags'] = tags
if isinstance(config.get('tag_files'), list):
config['tag_files'] = _strip_rows(config['tag_files'], 2)
if isinstance(config.get('tag_zims'), list):
config['tag_zims'] = _strip_rows(config['tag_zims'], 3)
return config
def strip_tag_names_in_configs(colors: dict):
config_directory = get_media_directory() / 'config'
for file_name in CONFIG_FILES:
file = config_directory / file_name
if not file.is_file():
continue
try:
with file.open('rt') as fh:
config = yaml.load(fh, Loader=yaml.CSafeLoader)
except Exception as e:
logger.error(f'Cannot strip Tag names of invalid config: {file}', exc_info=e)
continue
if not isinstance(config, dict):
continue
stripped = _strip_tags_config(config, colors) if file_name == 'tags.yaml' else config
stripped = _strip_nested(stripped)
if stripped != config:
logger.warning(f'Stripping whitespace from Tag names in {file}')
write_config_data(stripped, file, width=500 if file_name == 'tags.yaml' else 90)
def strip_tag_names_in_db(connection) -> dict:
"""Returns the color of each renamed Tag, by its new name."""
groups, colors = dict(), dict()
for id_, name, color in connection.exec_driver_sql('SELECT id, name, color FROM tag ORDER BY id').fetchall():
groups.setdefault(_strip(name), list()).append((id_, name, color))
for stripped, tags in groups.items():
if all(name == stripped for _, name, _ in tags):
continue
# Keep the Tag already named this, if any, otherwise the oldest.
survivor_id = next((id_ for id_, name, _ in tags if name == stripped), tags[0][0])
color = colors[stripped] = next((color for _, _, color in tags if color), None)
for id_, name, _ in tags:
if id_ == survivor_id:
continue
logger.warning(f'Merging Tag {name!r} into {stripped!r}')
connection.exec_driver_sql(
'INSERT OR IGNORE INTO tag_file (tag_id, file_group_id, created_at)'
' SELECT ?, file_group_id, created_at FROM tag_file WHERE tag_id = ?', (survivor_id, id_))
connection.exec_driver_sql(
'INSERT OR IGNORE INTO tag_zim (tag_id, zim_id, zim_entry, created_at)'
' SELECT ?, zim_id, zim_entry, created_at FROM tag_zim WHERE tag_id = ?', (survivor_id, id_))
connection.exec_driver_sql('DELETE FROM tag_file WHERE tag_id = ?', (id_,))
connection.exec_driver_sql('DELETE FROM tag_zim WHERE tag_id = ?', (id_,))
connection.exec_driver_sql('UPDATE collection SET tag_id = ? WHERE tag_id = ?', (survivor_id, id_))
connection.exec_driver_sql('DELETE FROM tag WHERE id = ?', (id_,))
logger.warning(f'Renaming Tag to {stripped!r}')
connection.exec_driver_sql('UPDATE tag SET name = ?, color = ? WHERE id = ?', (stripped, color, survivor_id))
# The Downloads which apply them (download_manager.yaml is stripped the same way).
downloads = connection.exec_driver_sql(
'SELECT id, tag_names, settings FROM download WHERE tag_names IS NOT NULL OR settings IS NOT NULL').fetchall()
for id_, tag_names, settings in downloads:
tag_names, settings = json.loads(tag_names) if tag_names else None, json.loads(settings) if settings else None
new_tag_names = _strip_names(tag_names) if isinstance(tag_names, list) else tag_names
new_settings = _strip_nested(settings)
if new_tag_names != tag_names or new_settings != settings:
connection.exec_driver_sql(
'UPDATE download SET tag_names = ?, settings = ? WHERE id = ?',
(json.dumps(new_tag_names) if new_tag_names is not None else None,
json.dumps(new_settings) if new_settings is not None else None, id_))
return colors
def upgrade():
colors = strip_tag_names_in_db(op.get_bind())
strip_tag_names_in_configs(colors)
def downgrade():
# Whitespace that was stripped is not restored.
pass

View file

@ -1,5 +1,5 @@
/*
* The five Special Directories fields on /admin/settings.
* The six Special Directories fields on /admin/settings.
*
* These are the fields that shipped printing their own prefix over their own value, and the
* ones whose visible labels were never associated with their inputs. Both faults were
@ -19,6 +19,7 @@ describe('Special Directories settings', () => {
['Map Directory', 'map'],
['Zims Directory', 'zims'],
['Playlists Directory', 'playlists'],
['Repos Directory', 'repos/%(repo_tag)s/%(repo_name)s'],
];
beforeEach(() => {
@ -38,8 +39,8 @@ describe('Special Directories settings', () => {
it('names every directory field, so they can be told apart', () => {
/*
* All five fields sit in one grid, hold similar-looking relative paths, and share the
* same prefix. Without a name attached to the input, a screen reader user hears five
* All six fields sit in one grid, hold similar-looking relative paths, and share the
* same prefix. Without a name attached to the input, a screen reader user hears six
* fields described only as "/media/wrolpi/" and cannot tell which one saves videos.
*/
directories.forEach(([label, value]) => {

View file

@ -41,7 +41,7 @@ import React from "react";
Cypress.on('uncaught:exception', (err, runnable) => {
if (err.message.includes('ChunkLoadError')) {
return false; // Ignore this error which happens after CircleCI tests.
return false; // Ignore this error which happens after CI tests.
}
// Otherwise, fail like normal
return true;

654
app/package-lock.json generated

File diff suppressed because it is too large Load diff

View file

@ -9,28 +9,30 @@
"@mantine/core": "^8.3.18",
"@mantine/hooks": "^8.3.18",
"@mantine/notifications": "^8.3.18",
"@tabler/icons-react": "^3.46.0",
"@tabler/icons-react": "^3.49.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^14.6.6",
"@testing-library/user-event": "^14.6.7",
"cypress": "^15.21.1",
"esptool-js": "^0.6.1",
"dompurify": "^3.4.17",
"esptool-js": "^0.7.0",
"lodash": "^4.18.1",
"maplibre-gl": "^5.24.0",
"markdown-it": "^15.0.3",
"mathjs": "^15.2.0",
"pmtiles": "^4.5.0",
"protomaps-themes-base": "^4.5.0",
"query-string": "^9.5.1",
"react": "^18.2.0",
"react-colorful": "^5.8.0",
"react-dom": "^18.2.0",
"react-dropzone": "^14.2.3",
"react": "^18.3.1",
"react-colorful": "^5.8.1",
"react-dom": "^18.3.1",
"react-dropzone": "^14.4.1",
"react-hotkeys-hook": "^5.3.3",
"react-qr-code": "^2.2.0",
"react-router": "^7.18.2",
"react-router": "^7.18.4",
"react-scripts": "5.0.1",
"react-stl-viewer": "^2.5.0",
"three": "^0.185",
"web-vitals": "^5.3.0"
"three": "^0.186",
"web-vitals": "^6.2.3"
},
"scripts": {
"start": "react-scripts start",
@ -58,10 +60,10 @@
]
},
"devDependencies": {
"@testing-library/dom": "^10.4.1",
"@testing-library/dom": "^10.4.2",
"@testing-library/jest-dom": "^7.0.1",
"@types/lodash": "^4.17.25",
"@types/node": "^26.4.0",
"@types/node": "^26.6.4",
"@types/react": "^18.3.17",
"@types/react-dom": "^18.3.5",
"typescript": "^5.9.3",

View file

@ -0,0 +1,11 @@
Solar climatology for the WROLPi Solar calculator
Source: NASA POWER, https://power.larc.nasa.gov/
Data: 2001-2020 monthly climatology of ALLSKY_SFC_SW_DWN, ALLSKY_SFC_SW_DIFF, ALLSKY_SRF_ALB, T2M, on a 1° grid.
The data was obtained from the POWER Project's POWER Climatology API v2.10.0 version on 2026/10/04.
The data was obtained from National Aeronautics and Space Administration (NASA) Langley Research Center's Prediction Of Worldwide Energy Resources (POWER) project funded through the NASA Earth Science Division.
License: Creative Commons Attribution 4.0 International (CC BY 4.0), https://creativecommons.org/licenses/by/4.0/
Changes: WROLPi modified the data: values are rounded to fit one byte each (0.04 kWh/m²/day for irradiance, 0.004 for albedo, 0.5 °C for temperature), temperatures are averaged from NASA's 0.5° x 0.625° grid into 1° cells, and the calculator interpolates between cells.

Binary file not shown.

View file

@ -242,6 +242,8 @@ either expressed or implied, of the FreeBSD Project. -->
// Optional search term passed in from a search-result deep link. If present we'll
// search the spine item at `loc` and jump to + highlight the first match.
let searchQuery = (params && params.get("q")) ? params.get("q") : null;
// Where the reader left off (an EPUB CFI). A search result's `loc` wins over it.
let startCfi = (params && params.get("cfi")) ? params.get("cfi") : null;
let book = ePub(url);
let rendition = book.renderTo("viewer", {
@ -250,7 +252,33 @@ either expressed or implied, of the FreeBSD Project. -->
height: "100%"
});
rendition.display(currentSectionIndex);
rendition.display(currentSectionIndex !== undefined ? currentSectionIndex : (startCfi || undefined));
// How far through the book `location` is, from its chapter and its page within that chapter. Cheaper
// than `book.locations.generate()`, which stalls a Raspberry Pi on a large book.
function readingProgress(location) {
if (location.atEnd) {
return 1;
}
const start = location.start;
const sections = book.spine.length || 1;
const displayed = start.displayed || {page: 1, total: 1};
const withinSection = (displayed.page - 1) / Math.max(displayed.total, 1);
return Math.min((start.index + withinSection) / sections, 1);
}
// Tell the WROLPi page that embeds this viewer where the reader is, so it can save their place. Nothing is
// posted at the start of the book: that is where a finished book opens, and saving it would un-finish it.
function postLocation(location) {
if (window.parent === window || !location || !location.start || location.atStart) {
return;
}
window.parent.postMessage({
type: "wrolpi:epub-location",
cfi: location.start.cfi,
progress: readingProgress(location),
}, window.location.origin);
}
// --- In-book search ---------------------------------------------------
// State is module-level so the toolbar buttons can drive it. `hits` is the
@ -516,6 +544,7 @@ either expressed or implied, of the FreeBSD Project. -->
prev.style.visibility = "visible";
}
postLocation(location);
});
rendition.on("layout", function (layout) {

View file

@ -316,6 +316,90 @@ pre.wrap-text {
height: 100%;
}
/* A markdown file preview, formatted or raw. Scrolls rather than fitting like .preview-fit, because
it is a document to read, not media to frame. Colors come from the theme tokens. */
.markdown-preview {
height: calc(100dvh - 21rem);
min-height: 15rem;
overflow: auto;
margin-top: 0.5em;
padding: 0 0.25em;
}
/* A repo's README is part of its page, not a modal; it grows with its content. */
.repo-readme .markdown-preview {
height: auto;
min-height: 0;
overflow: visible;
}
.markdown-raw {
margin: 0;
white-space: pre-wrap;
overflow-wrap: anywhere;
}
.markdown-body {
max-width: 50rem;
line-height: 1.6;
overflow-wrap: break-word;
}
.markdown-body > :first-child {
margin-top: 0;
}
.markdown-body h1, .markdown-body h2 {
padding-bottom: 0.25em;
border-bottom: 1px solid var(--table-line);
}
.markdown-body img {
max-width: 100%;
}
.markdown-body blockquote {
margin: 0 0 1em;
padding: 0 1em;
color: var(--muted);
border-left: 0.25em solid var(--border);
}
.markdown-body code {
padding: 0.1em 0.3em;
border-radius: 0.25em;
background-color: var(--head);
}
.markdown-body pre {
padding: 0.75em 1em;
overflow-x: auto;
border-radius: 0.25em;
background-color: var(--head);
}
.markdown-body pre code {
padding: 0;
background-color: transparent;
}
.markdown-body table {
display: block;
max-width: 100%;
overflow-x: auto;
border-collapse: collapse;
}
.markdown-body th, .markdown-body td {
padding: 0.3em 0.75em;
border: 1px solid var(--table-line);
}
.markdown-body hr {
border: none;
border-top: 1px solid var(--border);
}
td.file-path {
cursor: pointer;
}
@ -725,3 +809,35 @@ tr.drag-deselecting td {
display: block;
}
}
/* Map popups (POI details and pins) follow the active theme's tokens. */
/* Scoped under .maplibregl-popup to outrank maplibre-gl.css, which loads after this file. */
.maplibregl-popup .maplibregl-popup-content {
background: var(--panel);
color: var(--text);
border: 1px solid var(--border);
border-radius: 0.25rem;
}
.maplibregl-popup .maplibregl-popup-close-button {
color: var(--muted);
}
.maplibregl-popup .maplibregl-popup-close-button:hover {
background: var(--head);
color: var(--text);
}
.maplibregl-popup.maplibregl-popup-anchor-top .maplibregl-popup-tip,
.maplibregl-popup.maplibregl-popup-anchor-top-left .maplibregl-popup-tip,
.maplibregl-popup.maplibregl-popup-anchor-top-right .maplibregl-popup-tip {
border-bottom-color: var(--panel);
}
.maplibregl-popup.maplibregl-popup-anchor-bottom .maplibregl-popup-tip,
.maplibregl-popup.maplibregl-popup-anchor-bottom-left .maplibregl-popup-tip,
.maplibregl-popup.maplibregl-popup-anchor-bottom-right .maplibregl-popup-tip {
border-top-color: var(--panel);
}
.maplibregl-popup.maplibregl-popup-anchor-left .maplibregl-popup-tip {
border-right-color: var(--panel);
}
.maplibregl-popup.maplibregl-popup-anchor-right .maplibregl-popup-tip {
border-left-color: var(--panel);
}

View file

@ -17,6 +17,7 @@ import {ArchiveRoute} from "./components/Archive";
import {FilesRoute} from "./components/Files";
import {QueryProvider, StatusProvider} from "./hooks/customHooks";
import {FileWorkerStatusProvider} from "./contexts/FileWorkerStatusContext";
import {BookmarksProvider} from "./contexts/BookmarksContext";
import {ChannelEditPage, ChannelNewPage, ChannelsPage} from "./components/Channels";
import {MapRoute} from "./components/Map";
import {MediaContextProvider, mediaStyles, StatusContext} from "./contexts/contexts";
@ -29,9 +30,9 @@ import {FilePreviewProvider} from "./components/FilePreview";
import {TagsProvider} from "./Tags";
import {ZimRoute} from "./components/Zim";
import {PlaylistsRoute} from "./components/Playlists";
import {ReposRoute} from "./components/Repos";
import {DocsRoute} from "./components/Docs";
import {FlasherRoute} from "./components/Flasher";
import {AiRoute} from "./components/Ai";
import ErrorBoundary from "./components/ErrorBoundary";
import {KeyboardShortcutsProvider} from "./components/KeyboardShortcutsProvider";
@ -70,13 +71,15 @@ function Root() {
<TagsProvider>
<KeyboardShortcutsProvider>
<FilePreviewProvider>
<header>
<NavBar/>
</header>
<ErrorBoundary>
<Outlet/>
</ErrorBoundary>
<Footer/>
<BookmarksProvider>
<header>
<NavBar/>
</header>
<ErrorBoundary>
<Outlet/>
</ErrorBoundary>
<Footer/>
</BookmarksProvider>
</FilePreviewProvider>
</KeyboardShortcutsProvider>
</TagsProvider>
@ -84,10 +87,12 @@ function Root() {
</QueryProvider>
}
const router = createBrowserRouter(createRoutesFromElements(<Route
export const routes = createRoutesFromElements(<Route
path='/'
element={<Root/>}
errorElement={<PageNotFound/>}
// Root (and its ThemeProvider) is not rendered when Root itself fails, so the error page
// brings its own theme; Mantine components cannot render without one.
errorElement={<ThemeProvider><PageNotFound/></ThemeProvider>}
>
<Route index element={<ErrorBoundary><DashboardPage/></ErrorBoundary>}/>
<Route path='search/*' element={<ErrorBoundary><DashboardPage/></ErrorBoundary>}/>
@ -115,10 +120,14 @@ const router = createBrowserRouter(createRoutesFromElements(<Route
<Route path='map/*' element={<ErrorBoundary><MapRoute/></ErrorBoundary>}/>
<Route path='zim/*' element={<ErrorBoundary><ZimRoute/></ErrorBoundary>}/>
<Route path='playlists/*' element={<ErrorBoundary><PlaylistsRoute/></ErrorBoundary>}/>
<Route path='repos/*' element={<ErrorBoundary><ReposRoute/></ErrorBoundary>}/>
<Route path='files/*' element={<ErrorBoundary><FilesRoute/></ErrorBoundary>}/>
<Route path='flasher/*' element={<ErrorBoundary><FlasherRoute/></ErrorBoundary>}/>
<Route path='ai/*' element={<ErrorBoundary><AiRoute/></ErrorBoundary>}/>
</Route>));
{/* Unknown URLs render inside Root, with the navigation bar and theme. */}
<Route path='*' element={<PageNotFound/>}/>
</Route>);
const router = createBrowserRouter(routes);
export default function App() {
useEventsInterval();

52
app/src/App.test.js Normal file
View file

@ -0,0 +1,52 @@
import React from 'react';
import {render, screen} from '@testing-library/react';
import {createMemoryRouter, RouterProvider} from 'react-router';
import {routes} from './App';
import {StatusProvider} from './hooks/customHooks';
import {FileWorkerStatusProvider} from './contexts/FileWorkerStatusContext';
import {MediaContextProvider} from './contexts/contexts';
// Jest does not transform CSS inside node_modules (see transformIgnorePatterns).
jest.mock('@mantine/core/styles.css', () => ({}));
jest.mock('@mantine/notifications/styles.css', () => ({}));
// Routes this test does not visit, whose libraries jsdom/Jest cannot load (maplibre needs
// browser APIs; esptool-js is ESM-only).
jest.mock('./components/Map', () => ({MapRoute: () => null}));
jest.mock('./components/Flasher', () => ({FlasherRoute: () => null}));
describe('unknown routes', () => {
beforeEach(() => {
// resetMocks clears setupTests' implementation; the theme and media providers read it.
window.matchMedia.mockImplementation(query => ({
matches: false,
media: query,
addListener: jest.fn(),
removeListener: jest.fn(),
addEventListener: jest.fn(),
removeEventListener: jest.fn(),
}));
global.fetch = jest.fn(() => Promise.resolve({
ok: true,
status: 200,
headers: {get: () => 'application/json'},
json: () => Promise.resolve({tags: []}),
text: () => Promise.resolve('{}'),
}));
});
it('renders Page Not Found inside the app layout', async () => {
const router = createMemoryRouter(routes, {initialEntries: ['/no-such-page']});
// The providers App wraps around the router.
render(<StatusProvider>
<FileWorkerStatusProvider>
<MediaContextProvider>
<RouterProvider router={router}/>
</MediaContextProvider>
</FileWorkerStatusProvider>
</StatusProvider>);
expect(await screen.findByText('Page Not Found!')).toBeInTheDocument();
// Rendered inside Root's layout (the footer comes from Root), not as a bare error page.
expect(screen.getByText('Donate')).toBeInTheDocument();
});
});

View file

@ -5,6 +5,7 @@ import {
PageContainer,
RefreshHeader,
SearchResultsInput,
useLocalStorage,
useTitle
} from "./components/Common";
import React, {useContext, useState} from "react";
@ -14,6 +15,7 @@ import {DownloadMenu} from "./components/Download";
import {
ActionInput,
Button,
ButtonGroup,
Divider,
Header,
Icon,
@ -250,6 +252,8 @@ export function Getters() {
</div>
<span className='wrolpi-or-label'>Or</span>
</div>
{status?.flags && gettersDisabled &&
<p>Download and Upload are available after your files are refreshed.</p>}
</Panel>;
let getterModal;
@ -351,7 +355,14 @@ export function DashboardStatus() {
}
function DashboardRecentFiles() {
const {searchFiles, loading, fetchFiles} = useSearchRecentFiles();
// Show only the files the user is part way through.
const [continueOnly, setContinueOnly] = useLocalStorage('dashboard_recent_continue', false);
const {searchFiles, loading, fetchFiles} = useSearchRecentFiles(continueOnly);
let files = <FileCards files={searchFiles} loading={loading}/>;
if (continueOnly && searchFiles && searchFiles.length === 0) {
files = <p>Nothing in progress. A video, audio file, ebook or comic you stop part way through will be here.</p>;
}
return <Panel>
<RefreshHeader
@ -359,7 +370,15 @@ function DashboardRecentFiles() {
popupContents='Fetch the most recent files again'
onRefresh={fetchFiles}
/>
<FileCards files={searchFiles} loading={loading}/>
<ButtonGroup>
<Button size='small' variant={continueOnly ? 'default' : 'filled'} onClick={() => setContinueOnly(false)}>
All
</Button>
<Button size='small' variant={continueOnly ? 'filled' : 'default'} onClick={() => setContinueOnly(true)}>
Continue
</Button>
</ButtonGroup>
<div style={{marginTop: '1em'}}>{files}</div>
</Panel>
}

View file

@ -21,11 +21,11 @@ jest.mock('./hooks/customHooks', () => ({
useWROLMode: () => false,
}));
function renderAtUrl(url) {
function renderAtUrl(url, flags = {refresh_complete: true}) {
// test-utils' render() wraps content in a BrowserRouter (not MemoryRouter),
// so point the jsdom location at the desired URL before rendering.
window.history.pushState({}, '', url);
const status = {flags: {refresh_complete: true}};
const status = {flags};
return render(
<StatusContext.Provider value={{status}}>
<Getters/>
@ -67,6 +67,25 @@ describe('DashboardPage Getters deep-link', () => {
});
});
describe('DashboardPage Getters before the first refresh', () => {
const note = /available after your files are refreshed/i;
it('explains why Download and Upload are disabled', () => {
renderAtUrl('/', {refresh_complete: false});
expect(screen.getByText(note)).toBeInTheDocument();
});
it('says nothing once the refresh has completed', () => {
renderAtUrl('/');
expect(screen.queryByText(note)).toBeNull();
});
it('says nothing before the status has loaded', () => {
renderAtUrl('/', null);
expect(screen.queryByText(note)).toBeNull();
});
});
// The DownloadMenu's seed-object construction (joining initialUrls with
// newlines into the `download` prop's `urls` field) is exercised in the
// browser via manual verification, not unit-tested here — reaching into the

View file

@ -9,7 +9,7 @@ function eventToast(title, description, type = 'info', time = 5000, onClick = nu
toast({type: type, title: title, description: description, time: time, onClick: onClick});
}
function handleEvents(events) {
export function handleEvents(events) {
const newestEvents = {};
events.forEach(e => {
@ -63,13 +63,17 @@ function handleEvents(events) {
}
if (event === 'user_notify_message') {
console.log(message, url);
eventToast(
message,
'Click here to view the shared page',
'info',
10_000,
() => window.open(url, '_self'));
// Only a shared page carries a URL; other notifications (jobs, transcodes) are text.
if (url) {
eventToast(
message,
'Click here to view the shared page',
'info',
10_000,
() => window.open(url, '_self'));
} else {
eventToast(message, null, 'info', 10_000);
}
}
if (event === 'deleted') {

33
app/src/Events.test.js Normal file
View file

@ -0,0 +1,33 @@
import {handleEvents} from './Events';
import {toast} from './components/ui';
jest.mock('./components/ui', () => ({toast: jest.fn()}));
const notify = (url) => ({event: 'user_notify_message', message: 'Transcoding video.mp4', dt: '2026-10-12', url});
afterEach(() => {
toast.mockClear();
});
describe('user_notify_message', () => {
it('a notification without a URL is plain text, not a link', () => {
handleEvents([notify(null)]);
expect(toast).toHaveBeenCalledTimes(1);
const options = toast.mock.calls[0][0];
expect(options.title).toBe('Transcoding video.mp4');
expect(options.description).toBeFalsy();
expect(options.onClick).toBeFalsy();
});
it('a shared page opens its URL when clicked', () => {
const open = jest.spyOn(window, 'open').mockImplementation(() => null);
handleEvents([notify('/archive/1')]);
const options = toast.mock.calls[0][0];
expect(options.description).toBe('Click here to view the shared page');
options.onClick();
expect(open).toHaveBeenCalledWith('/archive/1', '_self');
open.mockRestore();
});
});

View file

@ -0,0 +1,84 @@
import {
flasherSearch, getConfigBackups, getDownloaders, getInventoryBackups, getMapSearchStatus, searchMap,
} from './api';
// A helper that fails must say so. These used to turn a non-OK response into an empty value
// ([], {dates: []}, {results: []}, ...), so every consumer rendered "nothing here" for a server
// error. They now return undefined; an empty value only ever comes from the server.
jest.mock('./components/ui', () => ({
...jest.requireActual('./components/ui'),
toast: jest.fn(),
}));
const response = (ok, body = {}) => {
const r = {
ok,
status: ok ? 200 : 500,
json: async () => body,
text: async () => JSON.stringify(body),
};
r.clone = () => r;
return r;
};
let originalFetch;
beforeEach(() => {
originalFetch = global.fetch;
});
afterEach(() => {
global.fetch = originalFetch;
});
const failWith = () => {
global.fetch = jest.fn(() => Promise.resolve(response(false, {error: 'boom'})));
};
const succeedWith = (body) => {
global.fetch = jest.fn(() => Promise.resolve(response(true, body)));
};
describe('on a non-OK response, the helper returns undefined', () => {
test.each([
['getConfigBackups', () => getConfigBackups('wrolpi.yaml')],
['getInventoryBackups', () => getInventoryBackups('food')],
['getDownloaders', () => getDownloaders()],
['searchMap', () => searchMap('camp')],
['getMapSearchStatus', () => getMapSearchStatus()],
['flasherSearch', () => flasherSearch(null, null)],
])('%s', async (_name, call) => {
failWith();
expect(await call()).toBeUndefined();
});
});
describe('a confirmed empty result is still empty', () => {
test('getConfigBackups', async () => {
succeedWith({dates: []});
expect(await getConfigBackups('wrolpi.yaml')).toEqual({dates: []});
});
test('getInventoryBackups', async () => {
succeedWith({dates: []});
expect(await getInventoryBackups('food')).toEqual([]);
});
test('getDownloaders', async () => {
succeedWith({downloaders: []});
expect(await getDownloaders()).toEqual({downloaders: []});
});
test('searchMap', async () => {
succeedWith({results: [], total: 0});
expect(await searchMap('camp')).toEqual({results: [], total: 0});
});
test('getMapSearchStatus', async () => {
succeedWith({indexed: [], missing: []});
expect(await getMapSearchStatus()).toEqual({indexed: [], missing: []});
});
test('flasherSearch', async () => {
succeedWith({file_groups: [], totals: {file_groups: 0}});
expect(await flasherSearch(null, null)).toEqual([[], 0]);
});
});

View file

@ -7,6 +7,7 @@ import {
COLLECTIONS_API,
DEFAULT_LIMIT,
Downloaders,
REPOS_API,
VIDEOS_API,
ZIM_API
} from "./components/Vars";
@ -406,6 +407,7 @@ export async function getSettings() {
nav_color: content.nav_color,
media_directory: content.media_directory,
playlists_destination: content.playlists_destination,
repos_destination: content.repos_destination,
require_cookies_unlocked: content.require_cookies_unlocked,
require_media_mounted: content.require_media_mounted,
save_ffprobe_json: content.save_ffprobe_json,
@ -522,7 +524,7 @@ export async function getConfigBackups(fileName) {
if (response.ok) {
return await response.json();
}
return {dates: []};
// Undefined, not {dates: []}, so the modal can tell a failed fetch from having no backups.
}
export async function postConfigBackupPreview(fileName, backupDate, mode) {
@ -720,7 +722,7 @@ export async function getInventoryBackups(slug) {
if (response.ok) {
return (await response.json())['dates'];
}
return [];
// Undefined, not [], so the modal can tell a failed fetch from having no backups.
}
export async function postInventoryRestorePreview(slug, backupDate, mode) {
@ -927,6 +929,21 @@ export async function getCollectionTagInfo(collectionId, tagName) {
}
}
// Tag any Collection (and optionally move its directory).
export async function tagCollection(collectionId, tagName, directory) {
const body = {tag_name: tagName};
if (directory) {
body['directory'] = directory;
}
const response = await apiPost(`${COLLECTIONS_API}/${collectionId}/tag`, body);
if (response.ok) {
return await response.json();
}
const message = await getErrorMessage(response, 'Unable to tag. See server logs.');
toast({type: 'error', title: 'Tag Error', description: message, time: 5000});
throw new Error(message);
}
export async function tagDomain(domainId, tagName, directory) {
const body = {
tag_name: tagName,
@ -1259,12 +1276,15 @@ export async function startDownloads() {
}
export async function getDownloaders() {
// Not toasting because this will happen often. Undefined on any failure, not an empty list, so
// the form can say the downloaders could not be fetched.
try {
const response = await apiGet(`${API_URI}/downloaders`);
// Not toasting because this will happen often.
return await response.json();
if (response.ok) {
return await response.json();
}
} catch (e) {
return {downloaders: []};
// The API is down; reported as undefined below.
}
}
@ -1285,7 +1305,7 @@ export async function deleteDownload(downloadId) {
}
}
export async function filesSearch(offset, limit, searchStr, mimetypes, model, tagNames, headline, months, fromYear, toYear, anyTag, order, suffix, path, deep) {
export async function filesSearch(offset, limit, searchStr, mimetypes, model, tagNames, headline, months, fromYear, toYear, anyTag, order, suffix, path, deep, inProgress) {
const body = {search_str: searchStr, offset: parseInt(offset), limit: parseInt(limit), any_tag: anyTag};
if (suffix) {
body['suffix'] = suffix;
@ -1320,6 +1340,9 @@ export async function filesSearch(offset, limit, searchStr, mimetypes, model, ta
if (deep) {
body['deep'] = true;
}
if (inProgress) {
body['in_progress'] = true;
}
console.info('searching files', body);
const response = await apiPost(`${API_URI}/files/search`, body);
@ -1351,7 +1374,7 @@ export async function flasherSearch(chip, path) {
}
const message = await getErrorMessage(response, 'Cannot search firmware. See server logs.');
toast({type: 'error', title: 'Unable to search firmware', description: message, time: 5000});
return [null, null];
// Undefined, not [null, null], so the picker can tell a failed search from finding nothing.
}
export async function getFlasherConfigs() {
@ -1361,7 +1384,9 @@ export async function getFlasherConfigs() {
const data = await response.json();
return data['configurations'] || [];
}
return [];
// Undefined, not [], so the page can tell a failed fetch from an empty list.
const message = await getErrorMessage(response, 'Could not get saved firmware configurations');
toast({type: 'error', title: 'Flasher Error', description: message, time: 5000});
}
export async function saveFlasherConfig(name, files, eraseAll) {
@ -1656,7 +1681,8 @@ export async function searchMap(query, limit = 12, offset = 0, lat = null, lon =
}
const response = await apiGet(url);
if (!response.ok) {
return {results: [], total: 0};
// Undefined, not {results: []}, so a failed search is not shown as finding nothing.
return undefined;
}
return await response.json();
}
@ -1664,7 +1690,8 @@ export async function searchMap(query, limit = 12, offset = 0, lat = null, lon =
export async function getMapSearchStatus() {
const response = await apiGet(`${API_URI}/map/search/status`);
if (!response.ok) {
return {indexed: [], missing: []};
// Undefined, not {indexed: [], missing: []}, which read as "no maps are installed".
return undefined;
}
return await response.json();
}
@ -2044,6 +2071,7 @@ export async function searchSuggestions(search_str) {
domains: content.domains,
authors: content.authors || [],
subjects: content.subjects || [],
repos: content.repos || [],
}
} else {
console.error('Failed to get file search suggestions!');
@ -2082,8 +2110,8 @@ export async function searchEstimateZims(search_str, tagNames) {
}
}
export async function searchEstimateOthers(tagNames) {
const body = {tag_names: tagNames};
export async function searchEstimateOthers(tagNames, searchStr) {
const body = {tag_names: tagNames, search_str: searchStr || null};
const response = await apiPost(`${API_URI}/search_other_estimates`, body);
if (response.ok) {
const content = await response.json();
@ -2093,8 +2121,8 @@ export async function searchEstimateOthers(tagNames) {
}
}
export async function searchChannels(tagNames) {
const body = {tag_names: tagNames};
export async function searchChannels(tagNames, searchStr) {
const body = {tag_names: tagNames, search_str: searchStr};
const response = await apiPost(`${API_URI}/videos/channels/search`, body);
if (response.ok) {
const content = await response.json();
@ -2104,6 +2132,16 @@ export async function searchChannels(tagNames) {
}
}
// Collections of `kind` whose name matches `searchStr` and/or which have any of the tags.
export async function searchCollections(kind, tagNames, searchStr) {
const body = {kind, tag_names: tagNames, search_str: searchStr};
const response = await apiPost(`${COLLECTIONS_API}/search`, body);
if (response.ok) {
const content = await response.json();
return {collections: content.collections};
}
}
export async function getOutdatedZims() {
const response = await apiGet(`${API_URI}/zim/outdated`);
if (response.status === 200) {
@ -2439,32 +2477,154 @@ export async function setPlaylistTag(playlistId, tagName, directory) {
throw Error(message);
}
// --- AI assistant ---
export async function getAiCatalog() {
const response = await apiGet(`${API_URI}/ai/manage/catalog`);
// Bookmarks: a tree of nodes, a node with `children` being a directory.
export async function getBookmarks() {
const response = await apiGet(`${API_URI}/bookmarks`);
if (response.ok) {
return await response.json();
return (await response.json())['bookmarks'];
}
const message = await getErrorMessage(response, 'Failed to fetch AI catalog.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw new Error('Could not fetch bookmarks');
}
export async function postAiSettings(settings) {
const response = await apiPost(`${API_URI}/ai/manage/settings`, settings);
async function bookmarksModify(request, fallbackMessage) {
const response = await request;
if (response.ok) {
return await response.json();
return response.status === 204 ? null : await response.json();
}
const message = await getErrorMessage(response, 'Failed to save AI settings.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
const message = await getErrorMessage(response, fallbackMessage);
toast({type: 'error', title: 'Bookmarks Error', description: message, time: 5000});
throw new Error(message);
}
export async function addBookmark({name, url, new_tab = false, parent_id = null, position = null}) {
const body = {name, url, new_tab, parent_id, position};
return await bookmarksModify(apiPost(`${API_URI}/bookmarks`, body), 'Could not add bookmark');
}
export async function addBookmarkDirectory({name, parent_id = null, position = null}) {
const body = {name, parent_id, position};
return await bookmarksModify(apiPost(`${API_URI}/bookmarks/directory`, body), 'Could not add directory');
}
export async function updateBookmark(nodeId, {name, url, new_tab}) {
const body = {name, url, new_tab};
return await bookmarksModify(apiPut(`${API_URI}/bookmarks/${nodeId}`, body), 'Could not update bookmark');
}
export async function moveBookmark(nodeId, parent_id, position = null) {
const body = {parent_id, position};
return await bookmarksModify(apiPost(`${API_URI}/bookmarks/${nodeId}/move`, body), 'Could not move bookmark');
}
export async function deleteBookmark(nodeId) {
return await bookmarksModify(apiDelete(`${API_URI}/bookmarks/${nodeId}`), 'Could not delete bookmark');
}
async function reposError(response, fallbackMessage) {
const message = await getErrorMessage(response, fallbackMessage);
toast({type: 'error', title: 'Repos Error', description: message, time: 5000});
throw Error(message);
}
export async function getChatModes() {
const response = await apiGet(`${API_URI}/chat/modes`);
export async function fetchRepos() {
const response = await apiGet(REPOS_API);
if (response.ok) {
return (await response.json())['modes'];
return (await response.json())['repos'];
}
const message = await getErrorMessage(response, 'Failed to fetch chat modes.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
await reposError(response, 'Failed to fetch repos.');
}
export async function getRepo(repoId) {
const response = await apiGet(`${REPOS_API}/${repoId}`);
if (response.ok) {
return (await response.json())['repo'];
}
await reposError(response, 'Failed to fetch repo.');
}
// `repo`: {url, directory?, name?, tag_name?, frequency?, mode?, branch?, description?}
export async function createRepo(repo) {
const response = await apiPost(REPOS_API, repo);
if (response.ok) {
return (await response.json())['repo'];
}
await reposError(response, 'Failed to add repo.');
}
// `changes`: {description?, frequency?, mode?, branch?}. An empty branch follows the remote's default branch.
export async function updateRepo(repoId, changes) {
const response = await apiPut(`${REPOS_API}/${repoId}`, changes);
if (response.ok) {
return (await response.json())['repo'];
}
await reposError(response, 'Failed to update repo.');
}
export async function deleteRepo(repoId, deleteFiles = false) {
const response = await apiDelete(`${REPOS_API}/${repoId}?delete_files=${deleteFiles ? 'true' : 'false'}`);
if (!response.ok) {
await reposError(response, 'Failed to delete repo.');
}
}
// Download the repo's updates now.
export async function updateRepoNow(repoId) {
const response = await apiPost(`${REPOS_API}/${repoId}/update`);
if (!response.ok) {
await reposError(response, 'Failed to start the repo update.');
}
}
// One directory of a repo's files: {path, entries: [{name, path, is_dir, size}], readme_path}.
export async function getRepoTree(repoId, path) {
const query = path ? `?path=${encodeURIComponent(path)}` : '';
const response = await apiGet(`${REPOS_API}/${repoId}/tree${query}`);
if (response.ok) {
return await response.json();
}
await reposError(response, 'Failed to list the repo files.');
}
// Repos matching the search (name and README) and/or the tags. Each has an escaped `readme_headline`.
export async function searchRepos(searchStr, tagNames) {
const body = {search_str: searchStr || null, tag_names: tagNames || []};
const response = await apiPost(`${REPOS_API}/search`, body);
if (response.ok) {
const content = await response.json();
return {repos: content.repos, total: content.totals.repos};
}
await reposError(response, 'Failed to search repos.');
}
// The commits of a repo's checked out branch, newest first: {commits: [{sha, author, date, message}], total}.
export async function getRepoLog(repoId, offset = 0, limit = 50) {
const response = await apiGet(`${REPOS_API}/${repoId}/log?limit=${limit}&offset=${offset}`);
if (response.ok) {
return await response.json();
}
await reposError(response, 'Failed to fetch the repo history.');
}
// A ZIP of a repo's checked out files; the browser downloads it.
export function repoArchiveURL(repoId) {
return `${REPOS_API}/${repoId}/archive.zip`;
}
// What importing the git clone in `directory` (relative to the media directory) would do. Changes nothing.
export async function inspectRepoImport(directory, name, tagName, url) {
const body = {directory, name: name || null, tag_name: tagName || null, url: url || null};
const response = await apiPost(`${REPOS_API}/import/inspect`, body);
if (response.ok) {
return {inspection: (await response.json())['inspection']};
}
return {error: await getErrorMessage(response, 'Cannot import this directory.')};
}
// Import an existing git clone as a Repo; it stays where it is. `repo`: {directory, confirm, url?, name?, ...}
export async function importRepo(repo) {
const response = await apiPost(`${REPOS_API}/import`, repo);
if (response.ok) {
return (await response.json())['repo'];
}
await reposError(response, 'Failed to import the repo.');
}

View file

@ -1,216 +0,0 @@
import React, {useContext, useEffect, useState} from 'react';
import {Route, Routes} from 'react-router';
import {getAiCatalog, postAiSettings, postDownload} from '../api';
import {getServiceStatus, startService, stopService} from '../api/controller';
import {AiChatProvider} from '../contexts/AiChatContext';
import {StatusContext} from '../contexts/contexts';
import {AiChat} from './AiChat';
import {APIButton, humanFileSize, InfoMessage, PageContainer, TabLinks, useTitle, WROLModeMessage} from './Common';
import {SortableTable} from './SortableTable';
import {Button, Divider, Group, Header, NumberInput, Select, Table, Text, Toggle, toast} from './ui';
import {Downloaders} from './Vars';
// The Controller's name for llama-server: the compose service in docker, the systemd unit natively.
export const llmServiceName = (dockerized) => dockerized ? 'llm' : 'wrolpi-llm';
export function AiModelRow({model, onDownload}) {
const {name, tier, size, description, downloaded, active} = model;
return <Table.Row>
<Table.Cell>
<Text fw={active ? 700 : undefined}>{name}{active ? ' (active)' : ''}</Text>
{description && <Text size='sm' c='dimmed'>{description}</Text>}
</Table.Cell>
<Table.Cell>{tier}</Table.Cell>
<Table.Cell>{humanFileSize(size)}</Table.Cell>
<Table.Cell>
{downloaded
? <Button size='xs' icon='check' disabled role='save'>
{tier === 'custom' ? 'On disk' : 'Downloaded'}
</Button>
: <APIButton size='xs' icon='download' color='violet' onClick={() => onDownload(model)}>
Download
</APIButton>}
</Table.Cell>
</Table.Row>;
}
export function ManageAi() {
const [catalog, setCatalog] = useState(null);
const [serviceStatus, setServiceStatus] = useState(null);
const [pendingSave, setPendingSave] = useState(false);
const {status} = useContext(StatusContext);
const serviceName = llmServiceName(status?.dockerized);
const fetchCatalog = async () => {
try {
const data = await getAiCatalog();
if (data) {
setCatalog(data);
}
} catch (e) {
console.error(e);
}
};
const fetchService = async () => {
try {
setServiceStatus(await getServiceStatus(serviceName));
} catch (e) {
// The service is unknown until AI is installed; not an error worth toasting.
setServiceStatus(null);
}
};
useEffect(() => {
fetchCatalog();
fetchService();
const interval = setInterval(() => {
fetchCatalog();
fetchService();
}, 30000);
return () => clearInterval(interval);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [serviceName]);
const handleDownload = async (model) => {
await postDownload({
urls: [model.url],
downloader: Downloaders.AiModel,
destination: 'ai/models',
});
toast({
type: 'success',
title: 'Download started',
description: 'Monitor progress on the Downloads page.',
time: 5000,
});
};
const saveSettings = async (settings) => {
setPendingSave(true);
try {
await postAiSettings(settings);
await fetchCatalog();
} finally {
setPendingSave(false);
}
};
const handleServiceButton = async () => {
if (serviceStatus?.status === 'running') {
await stopService(serviceName);
} else {
await startService(serviceName);
}
await fetchService();
};
if (!catalog) {
return <WROLModeMessage content='Cannot modify AI'/>;
}
const {models, recommended_tier, total_ram, slow_hardware, disk_usage, enabled, active_model} = catalog;
const downloadedModels = models.filter(i => i.downloaded);
const modelHeaders = [
{key: 'name', text: 'Model', sortBy: i => i.name.toLowerCase()},
{key: 'tier', text: 'Tier', sortBy: i => i.tier},
{key: 'size', text: 'Size', sortBy: i => i.size || 0},
{key: 'actions', text: '', sortBy: null},
];
return <>
<WROLModeMessage content='Cannot modify AI'/>
{slow_hardware && <InfoMessage storageName='hint_ai_slow_hardware'>
This hardware will run the assistant slowly. Answers may take minutes; the smallest
model is recommended.
</InfoMessage>}
<Header as='h3'>AI Assistant</Header>
<Group gap='lg'>
<Toggle
label='Enabled'
checked={enabled === true}
disabled={pendingSave || !active_model}
onChange={e => saveSettings({enabled: e.target.checked})}
/>
<Select
label='Active Model'
data={downloadedModels.map(i => ({value: i.name, label: i.name}))}
value={active_model || null}
placeholder={downloadedModels.length ? 'Select a model' : 'Download a model below'}
disabled={pendingSave || !downloadedModels.length}
onChange={value => value && saveSettings({active_model: value})}
/>
<NumberInput
label='Idle unload (minutes)'
min={1}
value={catalog.idle_unload_minutes}
disabled={pendingSave}
onChange={value => value && saveSettings({idle_unload_minutes: value})}
/>
</Group>
{enabled && <Group gap='xs' style={{marginTop: '1em'}}>
<Text>Service: {serviceStatus?.status || 'unknown'}</Text>
<APIButton
size='xs'
color={serviceStatus?.status === 'running' ? undefined : 'violet'}
role={serviceStatus?.status === 'running' ? 'danger' : undefined}
onClick={handleServiceButton}
>
{serviceStatus?.status === 'running' ? 'Stop' : 'Start'}
</APIButton>
</Group>}
<Divider/>
<Header as='h3'>Model Catalog</Header>
<Text size='sm'>
This device has {humanFileSize(total_ram)} of RAM;
the <b>{recommended_tier}</b> tier is recommended.
Models use {humanFileSize(disk_usage)} in <b>ai/models/</b>.
</Text>
<SortableTable
tableHeaders={modelHeaders}
data={models}
rowFunc={(model) => <AiModelRow key={model.name} model={model} onDownload={handleDownload}/>}
rowKey='name'
defaultSortColumn='size'
/>
<InfoMessage storageName='hint_ai_models'>
<p>The AI assistant runs entirely on this WROLPi — no internet is needed after a model
is downloaded. Download a model, select it, then enable the assistant. Monitor
download progress on the <b>Downloads</b> page.</p>
<p>You can also copy your own GGUF file into <b>ai/models/</b>; it will appear here as a
<b>custom</b> model the next time this page loads. Use a plain ASCII file name without
<b>#</b>, quotes, or <b>..</b>. The model must support tool calling (Qwen3, Llama 3.x,
and Mistral instruct models do). Selecting a custom model resets the context size to the
default; set <b>context_size</b> in <b>ai.yaml</b> to override it.</p>
</InfoMessage>
</>;
}
export function AiRoute() {
useTitle('AI');
const links = [
{text: 'Chat', to: '/ai', key: 'chat', end: true},
{text: 'Manage', to: '/ai/manage', key: 'manage'},
];
// The provider wraps both tabs so the conversation survives visiting Manage.
return <AiChatProvider>
<PageContainer>
<TabLinks links={links}/>
<Routes>
<Route path='/' exact element={<AiChat/>}/>
<Route path='manage' exact element={<ManageAi/>}/>
</Routes>
</PageContainer>
</AiChatProvider>;
}

View file

@ -1,130 +0,0 @@
import React from 'react';
import {fireEvent, screen, waitFor} from '@testing-library/react';
import {render} from '../test-utils';
import {AiModelRow, ManageAi, llmServiceName} from './Ai';
import {Table} from './ui';
jest.mock('../api', () => ({
getAiCatalog: jest.fn(),
postAiSettings: jest.fn().mockResolvedValue({}),
postDownload: jest.fn().mockResolvedValue({}),
}));
jest.mock('../api/controller', () => ({
getServiceStatus: jest.fn().mockResolvedValue({name: 'wrolpi-llm', status: 'stopped'}),
startService: jest.fn().mockResolvedValue({}),
stopService: jest.fn().mockResolvedValue({}),
}));
const {getAiCatalog, postAiSettings, postDownload} = require('../api');
const catalogFixture = (overrides = {}) => ({
models: [
{
name: 'Qwen3-1.7B-Q4_K_M.gguf', tier: 'small', size: 1_200_000_000,
url: 'https://cdn.example.com/ai/Qwen3-1.7B-Q4_K_M.gguf',
description: 'Small model.', downloaded: true, active: true,
},
{
name: 'Qwen3-4B-Instruct-2507-Q4_K_M.gguf', tier: 'medium', size: 2_600_000_000,
url: 'https://cdn.example.com/ai/Qwen3-4B-Instruct-2507-Q4_K_M.gguf',
description: 'Medium model.', downloaded: false, active: false,
},
],
catalog_source: 'bundled',
total_ram: 8 * 1024 ** 3,
recommended_tier: 'medium',
slow_hardware: false,
models_directory: '/media/wrolpi/ai/models',
disk_usage: 1_200_000_000,
enabled: false,
active_model: 'Qwen3-1.7B-Q4_K_M.gguf',
idle_unload_minutes: 15,
context_size: null,
...overrides,
});
describe('llmServiceName', () => {
it('is the compose service in docker, the systemd unit natively', () => {
expect(llmServiceName(true)).toBe('llm');
expect(llmServiceName(false)).toBe('wrolpi-llm');
});
});
describe('AiModelRow', () => {
const renderRow = (model, onDownload = jest.fn()) => render(
<Table><Table.Body><AiModelRow model={model} onDownload={onDownload}/></Table.Body></Table>);
it('shows a downloaded, active model without a download button', () => {
renderRow(catalogFixture().models[0]);
expect(screen.getByText(/Qwen3-1.7B-Q4_K_M.gguf \(active\)/)).toBeInTheDocument();
expect(screen.getByText('Downloaded')).toBeInTheDocument();
expect(screen.queryByText('Download')).not.toBeInTheDocument();
});
it('offers a download for a model that is not downloaded', async () => {
const onDownload = jest.fn();
renderRow(catalogFixture().models[1], onDownload);
fireEvent.click(screen.getByText('Download'));
await waitFor(() => expect(onDownload).toHaveBeenCalled());
expect(onDownload.mock.calls[0][0].name).toBe('Qwen3-4B-Instruct-2507-Q4_K_M.gguf');
});
});
describe('ManageAi', () => {
beforeEach(() => {
jest.clearAllMocks();
getAiCatalog.mockResolvedValue(catalogFixture());
});
it('renders the catalog with the RAM recommendation', async () => {
render(<ManageAi/>);
await waitFor(() => expect(screen.getByText(/tier is recommended/)).toBeInTheDocument());
// The active model appears in both the Select and its catalog row.
expect(screen.getAllByText(/Qwen3-1.7B-Q4_K_M.gguf/).length).toBeGreaterThanOrEqual(1);
expect(screen.getByText(/of RAM/)).toBeInTheDocument();
});
it('starts a model download with the ai_model downloader', async () => {
render(<ManageAi/>);
await waitFor(() => expect(screen.getByText('Download')).toBeInTheDocument());
fireEvent.click(screen.getByText('Download'));
await waitFor(() => expect(postDownload).toHaveBeenCalled());
expect(postDownload).toHaveBeenCalledWith({
urls: ['https://cdn.example.com/ai/Qwen3-4B-Instruct-2507-Q4_K_M.gguf'],
downloader: 'ai_model',
destination: 'ai/models',
});
});
it('saves the enabled toggle', async () => {
render(<ManageAi/>);
await waitFor(() => expect(screen.getByLabelText('Enabled')).toBeInTheDocument());
fireEvent.click(screen.getByLabelText('Enabled'));
await waitFor(() => expect(postAiSettings).toHaveBeenCalledWith({enabled: true}));
});
it('lists a custom GGUF from ai/models in the Active Model dropdown', async () => {
const custom = {
name: 'My-Custom-Model-Q8_0.gguf', tier: 'custom', size: 3_000_000_000, url: null,
description: 'Custom model found in ai/models/. Not from the WROLPi catalog.',
downloaded: true, active: false,
};
const fixture = catalogFixture();
getAiCatalog.mockResolvedValue({...fixture, models: [...fixture.models, custom]});
render(<ManageAi/>);
await waitFor(() => expect(screen.getByText('On disk')).toBeInTheDocument());
expect(screen.getByRole('cell', {name: 'custom'})).toBeInTheDocument();
fireEvent.click(screen.getByRole('textbox', {name: 'Active Model'}));
const option = await screen.findByRole('option', {name: 'My-Custom-Model-Q8_0.gguf'});
fireEvent.click(option);
await waitFor(() => expect(postAiSettings).toHaveBeenCalledWith({active_model: 'My-Custom-Model-Q8_0.gguf'}));
});
it('shows the slow-hardware warning on a Pi 4', async () => {
getAiCatalog.mockResolvedValue(catalogFixture({slow_hardware: true}));
render(<ManageAi/>);
await waitFor(() => expect(screen.getByText(/run the assistant slowly/)).toBeInTheDocument());
});
});

View file

@ -1,154 +0,0 @@
import React, {useEffect, useRef, useState} from 'react';
import {Link} from 'react-router';
import {getAiCatalog, getChatModes} from '../api';
import {useAiChat} from '../contexts/AiChatContext';
import {InfoMessage} from './Common';
import {Button, Group, Icon, Loader, Text, TextInput} from './ui';
const MODE_LABELS = {help: 'Help', research: 'Research', system: 'System'};
const MODE_DESCRIPTIONS = {
help: 'Learn how to use WROLPi',
research: 'Find and summarize library content',
system: 'Troubleshoot this WROLPi',
};
// Render an assistant message with WROLPi links clickable. The model only ever returns
// relative links (/videos/123), which the server computed — never external URLs.
export function LinkifiedText({content}) {
const parts = String(content || '').split(/(\/[a-zA-Z0-9_\-./?=&%#]+)/g);
return <span style={{whiteSpace: 'pre-wrap', overflowWrap: 'anywhere'}}>
{parts.map((part, i) => part.startsWith('/') && part.length > 1
? <Link key={i} to={part}>{part}</Link>
: part)}
</span>;
}
function ToolActivity({activity}) {
if (!activity || !activity.length) {
return null;
}
return <div>
{activity.map((item, i) => <Text key={i} size='sm' c='dimmed'>
{item.pending
? <Loader size='xs'/>
: <Icon name={item.success ? 'check' : 'warning circle'}/>}
{' '}using {String(item.tool || '').replace(/_/g, ' ')}
</Text>)}
</div>;
}
export function ChatMessage({message}) {
if (message.role === 'user') {
return <div style={{textAlign: 'right', margin: '0.5em 0'}}>
<span style={{
display: 'inline-block', padding: '0.5em 0.8em', borderRadius: 8,
background: 'var(--panel)', maxWidth: '85%', textAlign: 'left',
}}>{message.content}</span>
</div>;
}
return <div style={{margin: '0.5em 0'}}>
{message.status && <Text size='sm' c='dimmed'>{message.status}</Text>}
<ToolActivity activity={message.activity}/>
{message.content && <LinkifiedText content={message.content}/>}
{message.streaming && !message.content && !message.activity?.length && <Loader size='xs'/>}
{message.error && <Text style={{color: 'var(--danger)'}}>{message.error}</Text>}
</div>;
}
export function AiChat() {
const {mode, setMode, messages, sending, send, stop, disabledCode} = useAiChat();
const [modes, setModes] = useState([]);
const [slowHardware, setSlowHardware] = useState(false);
const [aiEnabled, setAiEnabled] = useState(null);
const [input, setInput] = useState('');
const bottomRef = useRef(null);
useEffect(() => {
getChatModes().then(i => i && setModes(i)).catch(console.error);
getAiCatalog().then(catalog => {
if (catalog) {
setSlowHardware(catalog.slow_hardware === true);
setAiEnabled(Boolean(catalog.enabled && catalog.active_model));
}
}).catch(console.error);
}, []);
useEffect(() => {
bottomRef.current?.scrollIntoView?.({behavior: 'smooth'});
}, [messages]);
const handleSend = async (content) => {
setInput('');
await send(content);
};
const notReady = aiEnabled === false || disabledCode === 'AI_DISABLED';
if (notReady) {
return <InfoMessage>
The AI assistant is not ready. Download a model and enable it on
the <Link to='/ai/manage'>Manage</Link> tab.
</InfoMessage>;
}
const suggestions = (modes.find(i => i.name === mode) || {}).suggestions || [];
return <>
{slowHardware && <InfoMessage storageName='hint_ai_chat_slow'>
This hardware runs the assistant slowly — answers can take minutes.
</InfoMessage>}
<Group gap='xs'>
{(modes.length ? modes.map(i => i.name) : Object.keys(MODE_LABELS)).map(name =>
<Button
key={name}
size='sm'
color={mode === name ? 'violet' : undefined}
aria-pressed={mode === name}
onClick={() => setMode(name)}
>
{MODE_LABELS[name] || name}
</Button>)}
{mode && <Text size='sm' c='dimmed'>{MODE_DESCRIPTIONS[mode]}</Text>}
</Group>
{!mode && <Text style={{marginTop: '1em'}}>
Pick a mode to start: <b>Help</b> teaches WROLPi, <b>Research</b> searches your
library, <b>System</b> troubleshoots this device.
</Text>}
{mode && <>
<div style={{minHeight: '40vh', margin: '1em 0'}}>
{messages.length === 0 && <Group gap='xs'>
{suggestions.map(suggestion =>
<Button key={suggestion} size='xs' onClick={() => handleSend(suggestion)}>
{suggestion}
</Button>)}
</Group>}
{messages.map((message, i) => <ChatMessage key={i} message={message}/>)}
<div ref={bottomRef}/>
</div>
<form onSubmit={e => {
e.preventDefault();
if (!sending) {
handleSend(input.trim());
}
}}>
<Group gap='xs' wrap='nowrap'>
<TextInput
style={{flexGrow: 1}}
placeholder='Ask the assistant…'
value={input}
disabled={sending}
onChange={e => setInput(e.target.value)}
aria-label='Chat message'
/>
{sending
? <Button role='cancel' onClick={stop}>Stop</Button>
: <Button type='submit' color='violet' disabled={!input.trim()}>Send</Button>}
</Group>
</form>
</>}
</>;
}

View file

@ -1,121 +0,0 @@
import React from 'react';
import {fireEvent, screen, waitFor} from '@testing-library/react';
import {render} from '../test-utils';
import {AiChatProvider} from '../contexts/AiChatContext';
import {parseSSEBlock} from '../sse';
import {AiChat, LinkifiedText} from './AiChat';
jest.mock('../api', () => ({
getChatModes: jest.fn().mockResolvedValue([
{name: 'help', suggestions: ['How do I download videos?']},
{name: 'research', suggestions: ['Find videos about canning']},
{name: 'system', suggestions: ['Why is my drive full?']},
]),
getAiCatalog: jest.fn().mockResolvedValue({
enabled: true, active_model: 'test.gguf', slow_hardware: false,
}),
}));
jest.mock('../sse', () => ({
...jest.requireActual('../sse'),
streamChat: jest.fn(),
}));
const {getAiCatalog, getChatModes} = require('../api');
const {streamChat} = require('../sse');
const renderChat = () => render(<AiChatProvider><AiChat/></AiChatProvider>);
describe('parseSSEBlock', () => {
it('parses an event and its JSON data', () => {
expect(parseSSEBlock('event: token\ndata: {"content": "hi"}'))
.toEqual({event: 'token', data: {content: 'hi'}});
expect(parseSSEBlock('')).toBeNull();
});
});
describe('LinkifiedText', () => {
it('makes relative WROLPi links clickable', () => {
render(<LinkifiedText content='See /videos/123 for details'/>);
const link = screen.getByRole('link', {name: '/videos/123'});
expect(link).toHaveAttribute('href', '/videos/123');
});
});
describe('AiChat', () => {
beforeEach(() => {
jest.clearAllMocks();
getChatModes.mockResolvedValue([
{name: 'help', suggestions: ['How do I download videos?']},
{name: 'research', suggestions: ['Find videos about canning']},
{name: 'system', suggestions: ['Why is my drive full?']},
]);
getAiCatalog.mockResolvedValue({enabled: true, active_model: 'test.gguf', slow_hardware: false});
});
it('starts with the mode picker; picking a mode shows its suggestion chips', async () => {
renderChat();
await waitFor(() => expect(screen.getByRole('button', {name: 'Research'})).toBeInTheDocument());
expect(screen.getByText(/Pick a mode/)).toBeInTheDocument();
fireEvent.click(screen.getByRole('button', {name: 'Research'}));
await waitFor(() => expect(screen.getByText('Find videos about canning')).toBeInTheDocument());
});
it('streams tokens and tool activity into the conversation', async () => {
streamChat.mockImplementation(async ({mode, messages}, onEvent) => {
expect(mode).toBe('research');
expect(messages[messages.length - 1]).toEqual({role: 'user', content: 'find canning'});
onEvent('tool_call', {tool: 'search_files', args: {search_str: 'canning', kind: 'video'}});
onEvent('tool_result', {tool: 'search_files', success: true});
onEvent('token', {content: 'One video: '});
onEvent('token', {content: '/videos/123'});
onEvent('done', {content: 'One video: /videos/123'});
});
renderChat();
await waitFor(() => expect(screen.getByRole('button', {name: 'Research'})).toBeInTheDocument());
fireEvent.click(screen.getByRole('button', {name: 'Research'}));
fireEvent.change(screen.getByLabelText('Chat message'), {target: {value: 'find canning'}});
fireEvent.click(screen.getByText('Send'));
await waitFor(() => expect(screen.getByText(/using search files/)).toBeInTheDocument());
await waitFor(() => expect(screen.getByRole('link', {name: '/videos/123'})).toBeInTheDocument());
expect(streamChat).toHaveBeenCalledTimes(1);
});
it('switching modes clears the conversation', async () => {
streamChat.mockImplementation(async (_, onEvent) => {
onEvent('done', {content: 'An answer.'});
});
renderChat();
await waitFor(() => expect(screen.getByRole('button', {name: 'Help'})).toBeInTheDocument());
fireEvent.click(screen.getByRole('button', {name: 'Help'}));
fireEvent.change(screen.getByLabelText('Chat message'), {target: {value: 'hello'}});
fireEvent.click(screen.getByText('Send'));
await waitFor(() => expect(screen.getByText('An answer.')).toBeInTheDocument());
fireEvent.click(screen.getByRole('button', {name: 'System'}));
await waitFor(() => expect(screen.queryByText('An answer.')).not.toBeInTheDocument());
});
it('shows an error event without crashing the conversation', async () => {
streamChat.mockImplementation(async (_, onEvent) => {
onEvent('error', {message: 'The AI failed to answer.'});
});
renderChat();
await waitFor(() => expect(screen.getByRole('button', {name: 'Help'})).toBeInTheDocument());
fireEvent.click(screen.getByRole('button', {name: 'Help'}));
fireEvent.change(screen.getByLabelText('Chat message'), {target: {value: 'hello'}});
fireEvent.click(screen.getByText('Send'));
await waitFor(() => expect(screen.getByText('The AI failed to answer.')).toBeInTheDocument());
});
it('points at the Manage tab when AI is not enabled', async () => {
getAiCatalog.mockResolvedValue({enabled: false, active_model: '', slow_hardware: false});
renderChat();
await waitFor(() => expect(screen.getByText(/not ready/)).toBeInTheDocument());
expect(screen.getByRole('link', {name: 'Manage'})).toHaveAttribute('href', '/ai/manage');
});
});

View file

@ -12,8 +12,10 @@ import {
import {Header, Loading, Panel, Statistic, StatisticGroup} from "./ui";
import {useStatistics} from "../hooks/customHooks";
import {CalculatorsPage} from "./Calculators";
import {BookmarksPage} from "./Bookmarks";
function StatisticsPage() {
// `statistics` follows useStatistics: null = pending, undefined = fetch failed.
export function StatisticsPage() {
useTitle('Statistics');
const {statistics} = useStatistics();
@ -25,7 +27,7 @@ function StatisticsPage() {
</>
}
if (statistics['global_statistics']) {
if (statistics && statistics['global_statistics']) {
let {global_statistics, file_statistics} = statistics;
const {
archive_count,
@ -104,6 +106,7 @@ export function MoreRoute(props) {
<Routes>
<Route path='calculators' element={<CalculatorsPage/>}/>
<Route path='statistics' exact element={<StatisticsPage/>}/>
<Route path='bookmarks' element={<BookmarksPage/>}/>
</Routes>
</PageContainer>
}

View file

@ -0,0 +1,453 @@
import React, {useState} from 'react';
import {Link} from 'react-router';
import {IconBookmark} from '@tabler/icons-react';
import {Button, Group, Header, Icon, Menu, Message, Modal, Panel, Select, Table, Text, TextInput, Toggle} from './ui';
import {APIButton, PageContainer, useTitle} from './Common';
import {isBookmarkDirectory, resolveBookmarkUrl, useBookmarks} from '../contexts/BookmarksContext';
import {addBookmark, addBookmarkDirectory, deleteBookmark, moveBookmark, updateBookmark} from '../api';
export const EDIT_BOOKMARKS_PATH = '/more/bookmarks';
/**
* The props that make a Menu.Item open a bookmark: an in-app Link for a path, a plain
* anchor for everything else, and a new tab when the bookmark asks for one. A URL that
* does not resolve (see resolveBookmarkUrl) gets a disabled item, never a raw href.
*/
function bookmarkItemProps(node) {
const {internal, href} = resolveBookmarkUrl(node.url);
if (!href) {
return {disabled: true};
}
const target = node.new_tab ? {target: '_blank', rel: 'noopener noreferrer'} : {};
if (internal) {
return {component: Link, to: href, ...target};
}
return {component: 'a', href, ...target};
}
/**
* The items of a bookmarks menu.
*
* `nested` draws a directory as a flyout (Menu.Sub), which opens beside the menu on hover.
* Without it a directory is a labelled, indented run of items instead: the mobile
* hamburger is the one place bookmarks are shown where a flyout cannot be hovered. The
* indent is in rem: a label is set smaller than an item, so an em indent would put a
* directory's label a few pixels left of the items beside it.
*/
export function BookmarkMenuItems({nodes, nested = true, depth = 0}) {
if (!nodes || nodes.length === 0) {
return null;
}
return nodes.map(node => {
if (isBookmarkDirectory(node)) {
if (nested) {
return <Menu.Sub key={node.id}>
<Menu.Sub.Target>
<Menu.Sub.Item leftSection={<Icon name='folder' size='small'/>}>{node.name}</Menu.Sub.Item>
</Menu.Sub.Target>
<Menu.Sub.Dropdown>
{node.children.length > 0
? <BookmarkMenuItems nodes={node.children} nested={nested}/>
: <Menu.Item disabled>Empty</Menu.Item>}
</Menu.Sub.Dropdown>
</Menu.Sub>;
}
return <React.Fragment key={node.id}>
<Menu.Label style={{paddingLeft: `${0.75 + depth}rem`}}>{node.name}</Menu.Label>
<BookmarkMenuItems nodes={node.children} nested={false} depth={depth + 1}/>
</React.Fragment>;
}
return <Menu.Item
key={node.id}
{...bookmarkItemProps(node)}
style={nested ? undefined : {paddingLeft: `${0.75 + depth}rem`}}
rightSection={node.new_tab ? <Icon name='external' size='small'/> : undefined}
>
{node.name}
</Menu.Item>;
});
}
/**
* Everything inside a bookmarks dropdown: the bookmarks, then the link to edit them. The
* edit link is always there, so a user with no bookmarks yet can find where to add one.
*/
export function BookmarksMenuContents({nested = true}) {
const {bookmarks} = useBookmarks();
const nodes = bookmarks || [];
return <>
<BookmarkMenuItems nodes={nodes} nested={nested}/>
{nodes.length > 0 && <Menu.Divider/>}
<Menu.Item component={Link} to={EDIT_BOOKMARKS_PATH} leftSection={<Icon name='edit' size='small'/>}>
Edit bookmarks
</Menu.Item>
</>;
}
/** Every directory in the tree with its full path, for the "Location" picker. */
export function directoryOptions(nodes, prefix = '') {
const options = [];
for (const node of nodes || []) {
if (isBookmarkDirectory(node)) {
const label = `${prefix}${node.name}`;
options.push({value: String(node.id), label, id: node.id});
options.push(...directoryOptions(node.children, `${label} / `));
}
}
return options;
}
/** The ids of `node` and everything beneath it. */
const subtreeIds = (node) => {
const ids = [node.id];
if (isBookmarkDirectory(node)) {
node.children.forEach(child => ids.push(...subtreeIds(child)));
}
return ids;
};
const ROOT = 'root';
/** The places a node may live: the top level, and every directory not inside `node`. */
export function locationOptions(nodes, node = null) {
// A directory cannot be moved into itself or its own children.
const excluded = node ? new Set(subtreeIds(node)) : new Set();
return [
{value: ROOT, label: 'Top level'},
...directoryOptions(nodes).filter(i => !excluded.has(i.id)).map(({value, label}) => ({value, label})),
];
}
/**
* The form for one node. `node` is null when adding; `directory` says which kind.
* `parentId` is where a new node goes, or where an edited one currently lives.
*/
export function BookmarkNodeModal({open, node, directory, parentId, nodes, onSave, onClose}) {
const [name, setName] = useState(node?.name || '');
const [url, setUrl] = useState(node?.url || '');
const [newTab, setNewTab] = useState(Boolean(node?.new_tab));
const [location, setLocation] = useState(parentId === null || parentId === undefined ? ROOT : String(parentId));
const [saving, setSaving] = useState(false);
const locations = locationOptions(nodes, node);
const valid = name.trim() && (directory || url.trim());
const title = node ? (directory ? 'Edit directory' : 'Edit bookmark') : (directory ? 'New directory' : 'New bookmark');
const handleSave = async () => {
if (!valid) return;
setSaving(true);
try {
await onSave({
name: name.trim(),
url: url.trim(),
new_tab: newTab,
parent_id: location === ROOT ? null : Number(location),
});
} finally {
setSaving(false);
}
};
return <Modal open={open} onClose={onClose} size='tiny'>
<Modal.Header>{title}</Modal.Header>
<Modal.Content>
<form onSubmit={e => {
e.preventDefault();
handleSave();
}}>
<TextInput
autoFocus
label='Name'
value={name}
onChange={e => setName(e.currentTarget.value)}
/>
{!directory && <TextInput
label='URL'
style={{marginTop: '1em'}}
value={url}
onChange={e => setUrl(e.currentTarget.value)}
placeholder='/videos, :8096/, or https://example.com'
description='A path on this WROLPi (/videos), a port on this host (:8096/ or http://:8096/), or a full URL.'
/>}
<Select
label='Location'
style={{marginTop: '1em'}}
data={locations}
value={location}
onChange={value => setLocation(value || ROOT)}
allowDeselect={false}
/>
{!directory && <Toggle
style={{marginTop: '1em'}}
label='Open in a new tab'
checked={newTab}
onChange={e => setNewTab(e.currentTarget.checked)}
/>}
</form>
</Modal.Content>
<Modal.Actions>
<Button role='cancel' onClick={onClose}>Cancel</Button>
<Button role='save' disabled={!valid || saving} onClick={handleSave}>Save</Button>
</Modal.Actions>
</Modal>;
}
/** The rows of the editor, one per node, children beneath their directory and indented. */
function BookmarkTableRows({nodes, parentId, depth, editable, actions}) {
return nodes.map((node, index) => {
const directory = isBookmarkDirectory(node);
const {href} = directory ? {} : resolveBookmarkUrl(node.url);
return <React.Fragment key={node.id}>
<Table.Row>
<Table.Cell>{String(index + 1).padStart(2, '0')}</Table.Cell>
<Table.Cell style={{width: '90px', textAlign: 'center'}}>
{directory
? <Icon name='folder' size='large'/>
: <Icon name={node.new_tab ? 'external' : 'linkify'} size='large'/>}
</Table.Cell>
<Table.Cell style={{paddingLeft: depth ? `calc(0.75em + ${depth * 1.5}em)` : undefined}}>
{directory && <b>{node.name}</b>}
{!directory && (href
? <a href={href} target={node.new_tab ? '_blank' : undefined} rel='noopener noreferrer'>
{node.name}
</a>
: <span>{node.name}</span>)}
{!directory && <Text size='xs' c='dimmed' truncate='end'>{node.url}</Text>}
</Table.Cell>
{editable && <Table.Cell>
<div className='wrolpi-button-row' style={{justifyContent: 'flex-end'}}>
<Button icon='arrow up' size='xs' aria-label={`Move ${node.name} up`} disabled={index === 0}
onClick={() => actions.move(node, parentId, index - 1)}/>
<Button icon='arrow down' size='xs' aria-label={`Move ${node.name} down`}
disabled={index === nodes.length - 1}
onClick={() => actions.move(node, parentId, index + 1)}/>
<Button icon='edit' size='xs' aria-label={`Edit ${node.name}`}
onClick={() => actions.edit(node, parentId)}/>
{directory
? <APIButton
role='danger' icon='trash' size='xs' aria-label={`Delete ${node.name}`}
confirmContent={`Delete the directory "${node.name}" and everything in it?`}
confirmButton='Delete'
onClick={() => actions.remove(node)}/>
: <Button role='danger' icon='trash' size='xs' aria-label={`Delete ${node.name}`}
onClick={() => actions.remove(node)}/>}
</div>
</Table.Cell>}
</Table.Row>
{directory && <BookmarkTableRows nodes={node.children} parentId={node.id} depth={depth + 1}
editable={editable} actions={actions}/>}
</React.Fragment>;
});
}
/**
* The bookmarks table, in the shape of the playlist items table. `actions` carries
* edit/move/remove so the gallery can render it with none of them wired.
*/
export function BookmarkTable({nodes, editable = true, actions}) {
return <Table>
<Table.Header>
<Table.Row>
<Table.HeaderCell>#</Table.HeaderCell>
<Table.HeaderCell>Kind</Table.HeaderCell>
<Table.HeaderCell>Bookmark</Table.HeaderCell>
{editable && <Table.HeaderCell/>}
</Table.Row>
</Table.Header>
<Table.Body>
<BookmarkTableRows nodes={nodes} parentId={null} depth={0} editable={editable} actions={actions}/>
</Table.Body>
</Table>;
}
/** The "Add a bookmark" and "Add a directory" forms beneath the table. */
export function BookmarkAddForms({nodes, onAddBookmark, onAddDirectory}) {
const [url, setUrl] = useState('');
const [name, setName] = useState('');
const [newTab, setNewTab] = useState(false);
const [location, setLocation] = useState(ROOT);
const [directoryName, setDirectoryName] = useState('');
const [directoryLocation, setDirectoryLocation] = useState(ROOT);
const locations = locationOptions(nodes);
const parentOf = (value) => value === ROOT ? null : Number(value);
// The form is cleared only once the server has accepted the node. A refused add has
// already shown its toast; the typed URL stays so the user can correct it.
const handleAddBookmark = async () => {
if (!url.trim() || !name.trim()) return;
try {
await onAddBookmark({name: name.trim(), url: url.trim(), new_tab: newTab, parent_id: parentOf(location)});
} catch (e) {
return;
}
setUrl('');
setName('');
};
const handleAddDirectory = async () => {
if (!directoryName.trim()) return;
try {
await onAddDirectory({name: directoryName.trim(), parent_id: parentOf(directoryLocation)});
} catch (e) {
return;
}
setDirectoryName('');
};
return <>
<Header as='h4'>Add a bookmark</Header>
<form onSubmit={e => {
e.preventDefault();
handleAddBookmark();
}}>
<Group align='flex-end' wrap='wrap'>
<TextInput
style={{flex: '2 1 300px'}}
aria-label='Bookmark URL'
placeholder='URL (e.g. /videos, :8096/, or https://example.com)'
value={url}
onChange={e => setUrl(e.currentTarget.value)}/>
<TextInput
style={{flex: '1 1 200px'}}
aria-label='Bookmark name'
placeholder='Name'
value={name}
onChange={e => setName(e.currentTarget.value)}/>
<Select
style={{flex: '1 1 160px'}}
aria-label='Bookmark location'
data={locations}
value={location}
onChange={value => setLocation(value || ROOT)}
allowDeselect={false}/>
<Toggle
label='New tab'
checked={newTab}
onChange={e => setNewTab(e.currentTarget.checked)}/>
<Button role='primary' type='submit' disabled={!url.trim() || !name.trim()}>Add</Button>
</Group>
</form>
<p style={{marginTop: '0.5em'}}>
<Text size='xs' c='dimmed' component='span'>
A URL is a path on this WROLPi (/videos), a port on this host (:8096/ or http://:8096/) that
works whichever address you reach it by, or a full URL.
</Text>
</p>
<Header as='h4'>Add a directory</Header>
<form onSubmit={e => {
e.preventDefault();
handleAddDirectory();
}}>
<Group align='flex-end' wrap='wrap'>
<TextInput
style={{flex: '2 1 300px'}}
aria-label='Directory name'
placeholder='Directory name'
value={directoryName}
onChange={e => setDirectoryName(e.currentTarget.value)}/>
<Select
style={{flex: '1 1 160px'}}
aria-label='Directory location'
data={locations}
value={directoryLocation}
onChange={value => setDirectoryLocation(value || ROOT)}
allowDeselect={false}/>
<Button role='primary' type='submit' disabled={!directoryName.trim()}>Add</Button>
</Group>
</form>
</>;
}
export function BookmarksPage() {
useTitle('Bookmarks');
const {bookmarks, refresh} = useBookmarks();
// {node, directory, parentId} while the edit modal is open; a `key` remounts the form fresh.
const [modal, setModal] = useState(null);
const [modalKey, setModalKey] = useState(0);
const nodes = bookmarks || [];
const actions = {
edit: (node, parentId) => {
setModalKey(k => k + 1);
setModal({node, directory: isBookmarkDirectory(node), parentId});
},
move: async (node, parentId, position) => {
try {
await moveBookmark(node.id, parentId, position);
await refresh();
} catch (e) {
// Error toast already shown by the API client.
}
},
remove: async (node) => {
try {
await deleteBookmark(node.id);
await refresh();
} catch (e) {
}
},
};
// These reject on failure (the API client has already shown the toast) so the add
// forms know to keep what was typed.
const handleAddBookmark = async (bookmark) => {
await addBookmark(bookmark);
await refresh();
};
const handleAddDirectory = async (directory) => {
await addBookmarkDirectory(directory);
await refresh();
};
const handleEditSave = async ({name, url, new_tab, parent_id}) => {
const {node, directory, parentId} = modal;
try {
await updateBookmark(node.id, directory ? {name} : {name, url, new_tab});
} catch (e) {
return; // Nothing changed; the modal stays open with the toast beside it.
}
// The rename is committed even if the move is refused, so the nav must show it
// either way; only a fully successful edit closes the modal.
let moved = true;
if (parent_id !== parentId) {
try {
await moveBookmark(node.id, parent_id);
} catch (e) {
moved = false;
}
}
await refresh();
if (moved) {
setModal(null);
}
};
return <PageContainer>
<Panel>
<Header as='h1' icon={IconBookmark}>Bookmarks</Header>
<p>
Bookmarks appear in the Bookmarks menu of the navigation bar. They can point at pages on this
WROLPi, at other services running beside it, or anywhere on the web.
</p>
{nodes.length === 0
? <Message title='There are no bookmarks yet'>Add a bookmark below.</Message>
: <BookmarkTable nodes={nodes} actions={actions}/>}
<BookmarkAddForms nodes={nodes} onAddBookmark={handleAddBookmark} onAddDirectory={handleAddDirectory}/>
</Panel>
{modal && <BookmarkNodeModal
key={modalKey}
open
node={modal.node}
directory={modal.directory}
parentId={modal.parentId}
nodes={nodes}
onSave={handleEditSave}
onClose={() => setModal(null)}
/>}
</PageContainer>;
}

View file

@ -0,0 +1,199 @@
import React from 'react';
import {screen} from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import {render} from '../test-utils';
import {BookmarkAddForms, BookmarkNodeModal, BookmarkTable, directoryOptions, locationOptions} from './Bookmarks';
import {resolveBookmarkUrl} from '../contexts/BookmarksContext';
const tree = [
{id: 1, name: 'Channels', url: '/videos/channel', new_tab: false},
{
id: 2, name: 'Services', children: [
{id: 3, name: 'Jellyfin', url: ':8096/', new_tab: true},
{id: 4, name: 'Radio', children: []},
]
},
];
describe('resolveBookmarkUrl', () => {
const location = {protocol: 'https:', hostname: '10.0.0.2', host: '10.0.0.2'};
it.each([
['/videos', {internal: true, href: '/videos'}],
[' /videos/channel/3 ', {internal: true, href: '/videos/channel/3'}],
// A port on this host takes the page's scheme, or its own when it names one.
[':8096/web', {internal: false, href: 'https://10.0.0.2:8096/web'}],
[':8096', {internal: false, href: 'https://10.0.0.2:8096'}],
['http://:8096/web', {internal: false, href: 'http://10.0.0.2:8096/web'}],
['HTTPS://:8085', {internal: false, href: 'https://10.0.0.2:8085'}],
['https://example.com/a?b=c', {internal: false, href: 'https://example.com/a?b=c'}],
])('resolves %s', (url, expected) => {
expect(resolveBookmarkUrl(url, location)).toEqual(expected);
});
it.each([
// Paths a browser takes off this host: network paths, backslashes, and control
// characters a URL parser strips before deciding.
'//evil.com', '///evil.com', '/\\evil.com', '/\\@evil.com', '/\t/evil.com', '/\n/evil.com',
// Port forms where the host would become credentials.
':8096@evil.com', 'http://:8096@evil.com', ':8096\t@evil.com', ':8096/x@evil.com', ':8096\\evil.com',
// Not http(s).
'javascript:alert(1)', 'data:text/html,hi', 'ftp://example.com', 'videos', '',
])('refuses %j rather than guessing', (url) => {
expect(resolveBookmarkUrl(url, location)).toEqual({internal: false, href: null});
});
it('brackets an IPv6 host for a port bookmark', () => {
// The browser reports an IPv6 hostname bracketed already.
const v6 = {protocol: 'https:', hostname: '[::1]', host: '[::1]'};
expect(resolveBookmarkUrl(':8096/web', v6)).toEqual({internal: false, href: 'https://[::1]:8096/web'});
expect(resolveBookmarkUrl(':8096', {protocol: 'http:', hostname: '::1', host: '[::1]'}))
.toEqual({internal: false, href: 'http://[::1]:8096'});
});
});
describe('directoryOptions', () => {
it('lists every directory with its path, and no bookmarks', () => {
expect(directoryOptions(tree)).toEqual([
{value: '2', label: 'Services', id: 2},
{value: '4', label: 'Services / Radio', id: 4},
]);
});
});
describe('BookmarkTable', () => {
const actions = () => ({edit: jest.fn(), move: jest.fn(), remove: jest.fn()});
it('renders the tree with resolved links, numbered within each directory', () => {
render(<BookmarkTable nodes={tree} actions={actions()}/>);
const channels = screen.getByRole('link', {name: 'Channels'});
expect(channels).toHaveAttribute('href', '/videos/channel');
expect(channels).not.toHaveAttribute('target');
const jellyfin = screen.getByRole('link', {name: 'Jellyfin'});
expect(jellyfin).toHaveAttribute('href', `${window.location.protocol}//${window.location.hostname}:8096/`);
expect(jellyfin).toHaveAttribute('target', '_blank');
expect(screen.getByText('Radio')).toBeInTheDocument();
// Channels, Services, then Services' two children.
const numbers = screen.getAllByRole('row').slice(1).map(row => row.firstChild.textContent);
expect(numbers).toEqual(['01', '02', '01', '02']);
});
it('moves a node among its siblings and only within bounds', async () => {
const a = actions();
render(<BookmarkTable nodes={tree} actions={a}/>);
// The first top-level node cannot move up; the last cannot move down.
expect(screen.getByRole('button', {name: 'Move Channels up'})).toBeDisabled();
expect(screen.getByRole('button', {name: 'Move Services down'})).toBeDisabled();
await userEvent.click(screen.getByRole('button', {name: 'Move Channels down'}));
expect(a.move).toHaveBeenCalledWith(tree[0], null, 1);
// A child moves within its directory: parent 2, from index 1 to 0.
await userEvent.click(screen.getByRole('button', {name: 'Move Radio up'}));
expect(a.move).toHaveBeenCalledWith(tree[1].children[1], 2, 0);
});
it('shows a bookmark whose URL does not resolve as text, not a link', () => {
const nodes = [{id: 1, name: 'Bad', url: '//evil.com', new_tab: false}];
render(<BookmarkTable nodes={nodes} actions={actions()}/>);
expect(screen.queryByRole('link', {name: 'Bad'})).not.toBeInTheDocument();
expect(screen.getByText('Bad')).toBeInTheDocument();
});
it('edits with the parent the node lives in, and deletes a bookmark without asking', async () => {
const a = actions();
render(<BookmarkTable nodes={tree} actions={a}/>);
await userEvent.click(screen.getByRole('button', {name: 'Edit Jellyfin'}));
expect(a.edit).toHaveBeenCalledWith(tree[1].children[0], 2);
await userEvent.click(screen.getByRole('button', {name: 'Delete Channels'}));
expect(a.remove).toHaveBeenCalledWith(tree[0]);
});
it('asks before deleting a directory, since its contents go with it', async () => {
const a = actions();
render(<BookmarkTable nodes={tree} actions={a}/>);
await userEvent.click(screen.getByRole('button', {name: 'Delete Services'}));
expect(a.remove).not.toHaveBeenCalled();
await userEvent.click(await screen.findByRole('button', {name: 'Delete'}));
expect(a.remove).toHaveBeenCalledWith(tree[1]);
});
});
describe('BookmarkAddForms', () => {
it('adds a bookmark at the top level and clears the form', async () => {
const onAddBookmark = jest.fn();
render(<BookmarkAddForms nodes={tree} onAddBookmark={onAddBookmark} onAddDirectory={jest.fn()}/>);
const adds = screen.getAllByRole('button', {name: 'Add'});
expect(adds[0]).toBeDisabled();
await userEvent.type(screen.getByLabelText('Bookmark URL'), ':8123/');
expect(adds[0]).toBeDisabled();
await userEvent.type(screen.getByLabelText('Bookmark name'), 'Home Assistant');
await userEvent.click(screen.getByLabelText('New tab'));
await userEvent.click(adds[0]);
expect(onAddBookmark).toHaveBeenCalledWith({name: 'Home Assistant', url: ':8123/', new_tab: true, parent_id: null});
expect(screen.getByLabelText('Bookmark URL')).toHaveValue('');
});
it('keeps what was typed when the server refuses the bookmark', async () => {
const onAddBookmark = jest.fn().mockRejectedValue(new Error('refused'));
render(<BookmarkAddForms nodes={tree} onAddBookmark={onAddBookmark} onAddDirectory={jest.fn()}/>);
await userEvent.type(screen.getByLabelText('Bookmark URL'), '//evil.com');
await userEvent.type(screen.getByLabelText('Bookmark name'), 'Oops');
await userEvent.click(screen.getAllByRole('button', {name: 'Add'})[0]);
expect(onAddBookmark).toHaveBeenCalled();
expect(screen.getByLabelText('Bookmark URL')).toHaveValue('//evil.com');
expect(screen.getByLabelText('Bookmark name')).toHaveValue('Oops');
});
it('adds a directory', async () => {
const onAddDirectory = jest.fn();
render(<BookmarkAddForms nodes={tree} onAddBookmark={jest.fn()} onAddDirectory={onAddDirectory}/>);
await userEvent.type(screen.getByLabelText('Directory name'), 'Tools');
await userEvent.click(screen.getAllByRole('button', {name: 'Add'})[1]);
expect(onAddDirectory).toHaveBeenCalledWith({name: 'Tools', parent_id: null});
});
});
describe('BookmarkNodeModal', () => {
// Mantine's Select scrolls the highlighted option into view; jsdom has no layout.
beforeAll(() => {
Element.prototype.scrollIntoView = jest.fn();
});
it('saves a new bookmark with its location and new-tab choice', async () => {
const onSave = jest.fn();
render(<BookmarkNodeModal open node={null} directory={false} parentId={2} nodes={tree}
onSave={onSave} onClose={jest.fn()}/>);
const save = screen.getByRole('button', {name: 'Save'});
expect(save).toBeDisabled();
await userEvent.type(screen.getByLabelText('Name'), 'Home Assistant');
expect(save).toBeDisabled();
await userEvent.type(screen.getByLabelText('URL'), ':8123/');
await userEvent.click(screen.getByLabelText('Open in a new tab'));
await userEvent.click(save);
expect(onSave).toHaveBeenCalledWith({name: 'Home Assistant', url: ':8123/', new_tab: true, parent_id: 2});
});
it('a directory has no URL or new-tab choice', () => {
render(<BookmarkNodeModal open node={tree[1]} directory parentId={null} nodes={tree}
onSave={jest.fn()} onClose={jest.fn()}/>);
expect(screen.getByLabelText('Name')).toHaveValue('Services');
expect(screen.queryByLabelText('URL')).not.toBeInTheDocument();
expect(screen.queryByLabelText('Open in a new tab')).not.toBeInTheDocument();
});
});
describe('locationOptions', () => {
it('offers the top level and every directory to a new node', () => {
expect(locationOptions(tree).map(o => o.label)).toEqual(['Top level', 'Services', 'Services / Radio']);
});
it('never offers a directory itself, or its children', () => {
expect(locationOptions(tree, tree[1]).map(o => o.label)).toEqual(['Top level']);
expect(locationOptions(tree, tree[1].children[1]).map(o => o.label)).toEqual(['Top level', 'Services']);
});
});

View file

@ -14,6 +14,7 @@ import {FoodStorageCalculator} from "./calculators/FoodStorageCalculator";
import {OneTimePadCalculator} from "./calculators/OneTimePadCalculator";
import {VinDecoderCalculator} from "./calculators/VinDecoderCalculator";
import {SpeedTestCalculator} from "./calculators/SpeedTestCalculator";
import {SolarCalculator} from "./calculators/SolarCalculator";
// Calculators are shown in these titled groups, in this order. Each calculator carries a
// `group` matching one of these names; any calculator whose group is missing/unknown falls
@ -35,6 +36,7 @@ export const useCalculators = () => {
{key: 'ratio', icon: 'th large', button: 'Ratio', group: 'Engineering', contents: <RatioCalculators/>},
{key: 'electrical', icon: 'lightning', button: 'Electrical', group: 'Engineering', contents: <ElectricalCalculators/>},
{key: 'drive', icon: 'cogs', button: 'Drive Ratio', group: 'Engineering', contents: <DriveCalculator/>},
{key: 'solar', icon: 'sun', button: 'Solar', group: 'Engineering', contents: <SolarCalculator/>},
{key: 'water', icon: 'tint', button: 'Water Storage', group: 'Preparedness & Storage', contents: <WaterCalculator/>},
{key: 'ration', icon: 'food', button: 'Food Storage', group: 'Preparedness & Storage', contents: <FoodStorageCalculator/>},
{key: 'antenna', icon: 'signal', button: 'Antenna', group: 'Radio & Communications', contents: <DipoleAntennaCalculator/>},

View file

@ -3,6 +3,7 @@ import {IconArrowLeft, IconArrowRight} from '@tabler/icons-react';
import {getArchiveContents, getArchiveMemberUrl} from '../api';
import {IconButton, Label, Loading, Message} from './ui';
import {useLocalStorage} from './Common';
import {usePageProgress} from '../hooks/useFileProgress';
const IMAGE_EXTENSIONS = /\.(jpe?g|png|gif|webp|bmp|tiff?)$/i;
@ -22,9 +23,20 @@ function flattenEntries(entries, prefix = '') {
return paths;
}
export function CbzViewer({path}) {
/**
* @param path The comic book file.
* @param progressPath The FileGroup's primary path, to resume and save the reader's page; omit to not track it.
* @param fileGroup The FileGroup JSON, which holds the saved page.
*/
export function CbzViewer({path, progressPath = null, fileGroup = null}) {
const [pages, setPages] = useState(null);
const [currentPage, setCurrentPage] = useState(0);
const {initialPage, onPageChange} = usePageProgress(progressPath, fileGroup, {enabled: !!progressPath});
// Read when the pages load; a later save must not move the reader.
const initialPageRef = useRef(initialPage);
initialPageRef.current = initialPage;
// The page shown on opening is where the reader already is (or the cover of a finished comic); it is not saved.
const justOpenedRef = useRef(true);
const [rtl, setRtl] = useLocalStorage('cbzViewerRtl', false);
const [fullscreen, setFullscreen] = useState(false);
const [error, setError] = useState(null);
@ -41,8 +53,9 @@ export function CbzViewer({path}) {
.filter(name => IMAGE_EXTENSIONS.test(name))
.sort();
const urls = images.map(member => getArchiveMemberUrl(path, member));
justOpenedRef.current = true;
setPages(urls);
setCurrentPage(0);
setCurrentPage(Math.min(initialPageRef.current, Math.max(urls.length - 1, 0)));
} catch (e) {
setError(e.message || 'Could not read archive contents.');
}
@ -50,6 +63,17 @@ export function CbzViewer({path}) {
load();
}, [path]);
useEffect(() => {
if (!pages || pages.length === 0) {
return;
}
if (justOpenedRef.current) {
justOpenedRef.current = false;
return;
}
onPageChange(currentPage, pages.length);
}, [currentPage, pages, onPageChange]);
const goNext = useCallback(() => {
setCurrentPage(p => Math.min(p + 1, (pages?.length || 1) - 1));
}, [pages]);

View file

@ -0,0 +1,84 @@
import React from 'react';
import {act, fireEvent, screen} from '@testing-library/react';
import {renderWithProviders} from '../test-utils';
import {CbzViewer} from './CbzViewer';
import {getArchiveContents} from '../api';
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getArchiveContents: jest.fn(),
getArchiveMemberUrl: (path, member) => `/member/${member}`,
}));
const PATH = 'comics/issue.cbz';
const PAGES = ['01.jpg', '02.jpg', '03.jpg', '04.jpg'];
const {fetch: realFetch} = global;
const realSendBeacon = navigator.sendBeacon;
beforeEach(() => {
window.sessionStorage.clear();
getArchiveContents.mockResolvedValue({entries: PAGES.map(name => ({name, path: name, is_dir: false}))});
navigator.sendBeacon = jest.fn(() => false);
global.fetch = jest.fn(() => Promise.resolve({ok: true}));
});
afterEach(() => {
global.fetch = realFetch;
navigator.sendBeacon = realSendBeacon;
});
const saves = () => global.fetch.mock.calls.map(([, init]) => JSON.parse(init.body));
describe('CbzViewer progress', () => {
test('opens at the saved page, and saves the page the reader leaves on', async () => {
const fileGroup = {progress: 0.75, position: {kind: 'page', page: 2, updated_at: 1000}};
const {unmount} = renderWithProviders(
<CbzViewer path={PATH} progressPath={PATH} fileGroup={fileGroup}/>);
expect(await screen.findByText('3 / 4')).toBeInTheDocument();
expect(screen.getByAltText('Page 3')).toHaveAttribute('src', '/member/03.jpg');
fireEvent.click(screen.getByLabelText('Next page'));
expect(screen.getByText('4 / 4')).toBeInTheDocument();
unmount();
expect(saves()).toEqual([expect.objectContaining({
file: PATH, progress: 1, final: true, position: expect.objectContaining({kind: 'page', page: 3}),
})]);
});
test('a saved page past the end opens the last page', async () => {
const fileGroup = {progress: 0.5, position: {kind: 'page', page: 40, updated_at: 1000}};
renderWithProviders(<CbzViewer path={PATH} progressPath={PATH} fileGroup={fileGroup}/>);
expect(await screen.findByText('4 / 4')).toBeInTheDocument();
});
test('without a progress path it opens at the first page and saves nothing', async () => {
const fileGroup = {progress: 0.75, position: {kind: 'page', page: 2, updated_at: 1000}};
const {unmount} = renderWithProviders(<CbzViewer path={PATH} fileGroup={fileGroup}/>);
expect(await screen.findByText('1 / 4')).toBeInTheDocument();
await act(async () => fireEvent.click(screen.getByLabelText('Next page')));
unmount();
expect(saves()).toEqual([]);
});
test('opening a comic saves nothing, so a finished comic stays finished', async () => {
const fileGroup = {progress: 1, position: null, viewed: new Date(9000).toISOString()};
const {unmount} = renderWithProviders(<CbzViewer path={PATH} progressPath={PATH} fileGroup={fileGroup}/>);
expect(await screen.findByText('1 / 4')).toBeInTheDocument();
unmount();
expect(saves()).toEqual([]);
});
test('peeking at the cover keeps the reader\'s place', async () => {
const fileGroup = {progress: 0.75, position: {kind: 'page', page: 2, updated_at: 1000}};
const {unmount} = renderWithProviders(<CbzViewer path={PATH} progressPath={PATH} fileGroup={fileGroup}/>);
expect(await screen.findByText('3 / 4')).toBeInTheDocument();
fireEvent.click(screen.getByLabelText('Previous page'));
fireEvent.click(screen.getByLabelText('Previous page'));
expect(screen.getByText('1 / 4')).toBeInTheDocument();
unmount();
expect(saves()).toEqual([expect.objectContaining({position: expect.objectContaining({page: 1})})]);
});
});

View file

@ -49,7 +49,7 @@ describe('Channels Page', () => {
{initialEntries: ['/videos/channel/1/edit']}
);
cy.wait('@channelFetch');
cy.wait('@searchDirectories'); // Needed for CircleCi.
cy.wait('@searchDirectories'); // Needed in CI.
cy.wait(500);
cy.get('input[name="name"]').should('be.visible');
cy.get('input[name="name"]').should('have.value', 'Editing Channel');

View file

@ -757,6 +757,18 @@ export function findPosterPath(file) {
}
}
/** How far the user is through a file, as a bar along the bottom of its poster. A full bar is finished. */
export function ViewProgress({progress}) {
if (!(progress > 0)) {
return null;
}
const percent = Math.round(Math.min(progress, 1) * 100);
return <div className='wrolpi-view-progress' role='progressbar' aria-label='Viewing progress'
aria-valuemin={0} aria-valuemax={100} aria-valuenow={percent}>
<div className='wrolpi-view-progress-bar' style={{width: `${percent}%`}}/>
</div>
}
export function CardPoster({to, file, overlay}) {
const navigate = useNavigate();
@ -785,6 +797,7 @@ export function CardPoster({to, file, overlay}) {
{imageLabel}
<img alt='poster' src={posterPath}/>
{overlay}
<ViewProgress progress={file.progress}/>
</>;
return <div className='wrolpi-card-poster'>
@ -802,6 +815,7 @@ export function CardPoster({to, file, overlay}) {
<div className='wrolpi-card-icon'>
{imageLabel}
<FileIcon file={file}/>
<ViewProgress progress={file.progress}/>
</div>
</PreviewLink>
} else if (!posterPath && to) {
@ -810,6 +824,7 @@ export function CardPoster({to, file, overlay}) {
<div className='wrolpi-card-icon' onClick={() => navigate(to)}>
{imageLabel}
<FileIcon file={file}/>
<ViewProgress progress={file.progress}/>
</div>
</Link>
}
@ -928,7 +943,10 @@ export function HelpHeader({
// why it cannot be used. `confirmStop` asks before switching off.
function SubsystemToggle({label, on, onChange, unsupportedMessage, info = null, disabled = null, confirmStop = null}) {
const [confirmOpen, setConfirmOpen] = React.useState(false);
// null = the status has loaded without this subsystem; undefined = the status has not loaded
// yet (or a call is in flight). Both disable the toggle; only the first claims unsupported.
const unsupported = on === null;
const unknown = on === undefined;
const handleChange = (checked) => {
if (!checked && confirmStop) {
@ -961,11 +979,11 @@ function SubsystemToggle({label, on, onChange, unsupportedMessage, info = null,
</Confirm>}
<Toggle
label={label}
disabled={disabled === null ? unsupported : disabled}
disabled={disabled === null ? (unsupported || unknown) : disabled}
checked={on === true}
onChange={handleChange}
/>
{popup && <InfoPopup content={popup}/>}
{popup && <span data-testid='subsystem-popup'><InfoPopup content={popup}/></span>}
</div>;
}
@ -1065,9 +1083,10 @@ export function Toggle({label, checked, disabled, onChange, icon, popupContent =
export function DisableDownloadsToggle() {
const [pending, setPending] = React.useState(false);
const {status, fetchStatus} = React.useContext(StatusContext);
const {status, fetchStatus, loaded} = React.useContext(StatusContext);
const {downloads} = status ? status : {downloads: null};
// Until the first status poll lands there is no downloads state to show, only a disabled toggle.
const downloads = loaded ? status?.downloads : null;
const wrolModeEnabled = useWROLMode();
const setDownloads = async (enable) => {
@ -1082,8 +1101,9 @@ export function DisableDownloadsToggle() {
}
const on = downloads && downloads['disabled'] === false && downloads['stopped'] === false;
const label = downloads === null ? 'Downloading' : on === true ? 'Downloading Enabled' : 'Downloading Disabled';
return <Toggle
label={on === true ? 'Downloading Enabled' : 'Downloading Disabled'}
label={label}
disabled={wrolModeEnabled || pending || downloads === null}
checked={on === true}
onChange={setDownloads}
@ -1160,7 +1180,7 @@ export function mimetypeIconName(mimetype, lowerPath = '') {
return 'file code';
} else if (mimetype.startsWith('application/pdf')) {
return 'file pdf';
} else if (mimetype.startsWith('text/plain')) {
} else if (mimetype.startsWith('text/plain') || mimetype.startsWith('text/markdown')) {
return 'file text';
} else if (mimetype.startsWith('image/')) {
return 'image';

View file

@ -1121,7 +1121,7 @@ describe('a row of buttons', () => {
* the two had been inline before.
*
* A row fixes it without touching the stack: the row is the stretched child, and it lays
* its buttons out at their own width. Nine sites, so this scan rather than nine
* its buttons out at their own width. Thirteen sites, so this scan rather than thirteen
* assertions -- and a scan because a stretched button is still a working button.
*/
const offenders = [];
@ -1140,8 +1140,8 @@ describe('a row of buttons', () => {
}
}
// Nine renders. Common.js declares it, and a declaration is not a `<BackButton`.
expect(found).toBe(9);
// Thirteen renders. Common.js declares it, and a declaration is not a `<BackButton`.
expect(found).toBe(13);
expect(offenders).toEqual([]);
});

View file

@ -38,6 +38,7 @@ import {toast} from "./ui";
import {CollectionTable} from "./collections/CollectionTable";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {CbzViewer} from "./CbzViewer";
import {useEpubProgress} from "../hooks/useFileProgress";
import _ from "lodash";
function DocsPage() {
@ -165,6 +166,23 @@ function DocsPage() {
</>;
}
/**
* The embedded viewer (the EPUB viewer, or the browser's PDF viewer). An EPUB opens where the user left off,
* unless a search result says where (`deepLinked`), and saves their place as they read.
*
* @param epubProgressPath The FileGroup's primary path when an EPUB is shown and tracked, otherwise null.
*/
function DocViewerFrame({url, epubProgressPath, fileGroup, deepLinked, title}) {
const {iframeRef, resumeCfi} = useEpubProgress(epubProgressPath, fileGroup, {enabled: !!epubProgressPath});
const src = resumeCfi && !deepLinked ? `${url}&cfi=${encodeURIComponent(resumeCfi)}` : url;
return <iframe
ref={iframeRef}
src={src}
title={title}
style={{width: '100%', height: '80vh', border: '1px solid var(--border)'}}
/>
}
function DocPage() {
const {fileGroupId} = useParams();
const navigate = useNavigate();
@ -245,6 +263,8 @@ function DocPage() {
? `/media/${encodeMediaPath(activePath)}${pageParam ? `#page=${encodeURIComponent(pageParam)}` : ''}`
: null;
const embedUrl = isEpub ? viewerUrl : pdfSrc;
// Progress belongs to the FileGroup; another format of the same book (a PDF beside the EPUB) is not tracked.
const trackedPath = String(activePath) === String(docFile.primary_path) ? docFile.primary_path : null;
const handleDelete = async (force = false) => {
if (doc && doc.id) {
@ -354,7 +374,7 @@ function DocPage() {
return <>
<div className='wrolpi-button-row'><BackButton/></div>
{isCbz && activePath && <CbzViewer path={activePath}/>}
{isCbz && activePath && <CbzViewer path={activePath} progressPath={trackedPath} fileGroup={docFile}/>}
{canEmbed && embedUrl && !isCbz && <div style={{marginBottom: '1em'}}>
{/*
@ -364,11 +384,8 @@ function DocPage() {
* re-gates rather than inheriting the last one's reveal.
*/}
<MediaGate key={activePath} gated={isPdf && mediaFilterEnabled}>
<iframe
src={embedUrl}
title={docFile.title || docFile.name}
style={{width: '100%', height: '80vh', border: '1px solid var(--border)'}}
/>
<DocViewerFrame url={embedUrl} epubProgressPath={isEpub ? trackedPath : null} fileGroup={docFile}
deepLinked={!!locParam} title={docFile.title || docFile.name}/>
</MediaGate>
</div>}

View file

@ -19,10 +19,15 @@ const source = fs.readFileSync(path.join(__dirname, 'Docs.js'), 'utf8');
describe('DocPage embed gating (source contract)', () => {
it('wraps the embedded viewer in a MediaGate', () => {
// The iframe must not be reachable without passing through the gate.
const embed = source.match(/<MediaGate[^>]*>\s*<iframe/);
// The iframe must not be reachable without passing through the gate. The gate's only child is the
// viewer frame, and the frame is just the iframe.
const embed = source.match(/<MediaGate[^>]*>\s*<DocViewerFrame/);
expect(embed).not.toBeNull();
const frame = source.match(/function DocViewerFrame[\s\S]*?\n}\n/);
expect(frame).not.toBeNull();
expect(frame[0]).toMatch(/return <iframe/);
expect(source.match(/<iframe/g)).toHaveLength(1);
});
it('gates on the live filter state, not on the theme name', () => {

View file

@ -127,6 +127,15 @@ describe('<ChannelDownloadForm/>', () => {
shouldBeInvalid('#url_input');
});
it('can be submitted with a frequency of Once', () => {
cy.get('#url_input').type('https://example.com/playlist').wait(500);
cy.get('#frequency_select').click();
cy.contains('[role="option"]:visible', 'Once').should('be.visible').click({force: true});
cy.get('#frequency_select').should('have.value', 'Once');
cy.get('#download_form_download_button').should('not.be.disabled');
});
it('can choose different options', () => {
/*
* `#download_frequency_selector` and `.item` named the old dropdown and its menu rows.

View file

@ -247,28 +247,38 @@ export function DownloadFrequencySelector({
}
export function DownloaderSelector({form, name = 'sub_downloader', path = 'sub_downloader'}) {
const [downloaders, setDownloaders] = React.useState([]);
// null = pending, undefined = could not be fetched, a list = loaded. Until loaded, the select is
// disabled and says why, rather than sitting empty as though there were no downloaders.
const [downloaders, setDownloaders] = React.useState(null);
const fetchDownloaders = async () => {
let {downloaders: downloaders_} = await getDownloaders();
downloaders_ = downloaders_.map((i) => {
const result = await getDownloaders();
if (!result || !Array.isArray(result.downloaders)) {
setDownloaders(undefined);
return;
}
setDownloaders(result.downloaders.map((i) => {
return {key: i.name, text: i.pretty_name || i.name, value: i.name}
})
setDownloaders(downloaders_);
}));
}
React.useEffect(() => {
fetchDownloaders();
}, []);
const placeholder = downloaders === null ? 'Loading downloaders…'
: downloaders === undefined ? 'Could not fetch downloaders'
: 'Select a downloader';
return <SelectField
form={form}
name={name}
path={path}
required
label='Downloader'
placeholder='Select a downloader'
options={downloaders}
placeholder={placeholder}
options={downloaders || []}
disabled={!Array.isArray(downloaders)}
/>
}

View file

@ -1,8 +1,9 @@
import React from 'react';
import {render, screen, waitFor} from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import {DestinationForm} from './Download';
import {createTestForm} from '../test-utils';
import {DepthInputForm, DestinationForm} from './Download';
import {useForm} from '../hooks/useForm';
import {createTestForm, renderWithProviders} from '../test-utils';
// Mock DirectorySearch component - simplified to avoid useState/useEffect warnings
jest.mock('./Common', () => {
@ -251,3 +252,26 @@ describe('DestinationForm', () => {
});
});
});
describe('NumberField', () => {
// Renders a NumberField against a real useForm: the Mantine NumberInput passes a value (not an
// event) to onChange, which used to reach `e.preventDefault()` and throw while typing.
function DepthHarness({onData}) {
const form = useForm({defaultFormData: {settings: {depth: ''}}, submitter: jest.fn()});
onData(form.formData);
return <DepthInputForm form={form}/>;
}
it('typing into a NumberField updates the form without throwing', async () => {
let latest = null;
renderWithProviders(<DepthHarness onData={(d) => latest = d}/>);
const input = screen.getByRole('textbox', {name: /Depth/});
await userEvent.type(input, '3');
await waitFor(() => {
expect(latest.settings.depth).toBe(3);
});
expect(input).toHaveValue('3');
});
});

View file

@ -0,0 +1,43 @@
import React from 'react';
import {act, render, screen} from '../test-utils';
import {createTestForm} from '../test-utils';
import {DownloaderSelector} from './Download';
import {getDownloaders} from '../api';
// The RSS form's Downloader select is filled from the API. Until the list arrives, or if it could
// not be fetched, the select must say so rather than sit empty as though there were no downloaders.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getDownloaders: jest.fn(),
}));
const makeForm = () => createTestForm({}, {
overrides: {getSelectionProps: () => [{value: null, onChange: jest.fn(), disabled: false, error: null}]},
});
beforeEach(() => getDownloaders.mockReset());
test('pending: the select says it is loading and is disabled', () => {
getDownloaders.mockReturnValue(new Promise(() => {
}));
render(<DownloaderSelector form={makeForm()}/>);
const input = screen.getByPlaceholderText('Loading downloaders…');
expect(input).toBeDisabled();
});
test('failed: the select says the downloaders could not be fetched', async () => {
getDownloaders.mockResolvedValue(undefined);
render(<DownloaderSelector form={makeForm()}/>);
await act(async () => {
});
expect(screen.getByPlaceholderText('Could not fetch downloaders')).toBeDisabled();
});
test('loaded: the select offers the downloaders', async () => {
getDownloaders.mockResolvedValue({downloaders: [{name: 'video', pretty_name: 'Videos'}]});
render(<DownloaderSelector form={makeForm()}/>);
await act(async () => {
});
expect(screen.getByPlaceholderText('Select a downloader')).toBeEnabled();
});

View file

@ -8,12 +8,14 @@ import {CbzViewer} from "./CbzViewer";
import {StlViewer} from "react-stl-viewer";
import {Button, ButtonGroup, Header, Icon, IconButton, MediaGate, Menu, Modal, toast} from "./ui";
import {useOneQuery} from "../hooks/customHooks";
import {useEpubProgress, useMediaProgress} from "../hooks/useFileProgress";
import {ShareButton} from "./Share";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {pathDirectory} from "./FileBrowser";
import {InlineErrorBoundary} from "./ErrorBoundary";
import {useLocation} from "react-router";
import {ThemeContext} from "../contexts/contexts";
import {isMarkdownFile, MarkdownPreview} from "./MarkdownPreview";
// Routes where file views should not be tracked
const EXCLUDED_TRACKING_ROUTES = [
@ -100,7 +102,7 @@ function getEpubViewerURL(previewFile) {
* A component rather than part of `getIframePreviewModal` because it reads the theme: the
* modal builders are plain functions called from an event handler, where hooks cannot run.
*/
export function IframePreview({url, gatePdf}) {
export function IframePreview({url, gatePdf, iframeRef}) {
const {isDark, mediaFilterEnabled} = React.useContext(ThemeContext);
/*
@ -116,7 +118,7 @@ export function IframePreview({url, gatePdf}) {
*/
return <MediaGate key={url} gated={!!gatePdf && !!mediaFilterEnabled}>
<div className='preview-fit'>
<iframe title='textModal' src={url}
<iframe title='textModal' src={url} ref={iframeRef}
style={{
height: '100%', width: '100%', border: 'none',
// A same-origin document has no styles of its own; it renders with the
@ -137,6 +139,19 @@ function getIframePreviewModal(previewFile, url, {gatePdf = false} = {}) {
</Modal.Content>
}
/** The EPUB viewer, opened where the user left off. */
function EpubPreview({previewFile}) {
const path = previewFile.primary_path ?? previewFile.path;
const {iframeRef, resumeCfi} = useEpubProgress(path, previewFile);
let url = getEpubViewerURL(previewFile);
if (resumeCfi) {
url = `${url}&cfi=${encodeURIComponent(resumeCfi)}`;
}
return <Modal.Content>
<IframePreview url={url} iframeRef={iframeRef}/>
</Modal.Content>
}
function getImagePreviewModal(previewFile) {
const path = previewFile.primary_path ?? previewFile.path;
const url = getMediaPathURL(previewFile);
@ -156,6 +171,14 @@ function getImagePreviewModal(previewFile) {
</React.Fragment>
}
/** A <video> or <audio> which resumes where the user left off, and saves how far they got. */
function PreviewMediaPlayer({as: Element, previewFile, path, style, children}) {
const progressHandlers = useMediaProgress(path, previewFile);
return <Element controls autoPlay={true} id="player" playsInline={true} style={style} {...progressHandlers}>
{children}
</Element>
}
function getVideoPreviewModal(previewFile) {
const url = getMediaPathURL(previewFile);
const path = previewFile.primary_path ?? previewFile.path;
@ -167,14 +190,10 @@ function getVideoPreviewModal(previewFile) {
<Modal.Content>
<Header as='h5'>{path['path']}</Header>
<div className='preview-fit'>
<video controls
autoPlay={true}
id="player"
playsInline={true}
style={{maxHeight: '100%', maxWidth: '100%'}}
>
<PreviewMediaPlayer as='video' previewFile={previewFile} path={path}
style={{maxHeight: '100%', maxWidth: '100%'}}>
<source src={url}/>
</video>
</PreviewMediaPlayer>
</div>
</Modal.Content>
</React.Fragment>
@ -191,14 +210,10 @@ function getAudioPreviewModal(previewFile) {
{name}
</Modal.Header>
<Modal.Content>
<audio controls
autoPlay={true}
id="player"
playsInline={true}
style={{width: '90%', maxWidth: '95%'}}
>
<PreviewMediaPlayer as='audio' previewFile={previewFile} path={path}
style={{width: '90%', maxWidth: '95%'}}>
<source src={url} type={type}/>
</audio>
</PreviewMediaPlayer>
</Modal.Content>
</React.Fragment>
}
@ -441,6 +456,9 @@ export function FilePreviewProvider({children}) {
const [previewQuery, setPreviewQuery] = useOneQuery('preview');
const [errorModalOpen, setErrorModalOpen] = React.useState(false);
const location = useLocation();
// The path of the open preview that has been marked viewed. One opening is one view, however
// many times previewFile is replaced while it is open (re-fetching after tagging, for example).
const trackedPathRef = React.useRef(null);
// Determine if tracking should be skipped based on current route
const skipTracking = shouldSkipTracking(location.pathname);
@ -470,7 +488,8 @@ export function FilePreviewProvider({children}) {
const initPreviewFile = async () => {
// Get simple information about the file, preview the file.
try {
const file = await getFile(previewQuery, skipTracking);
// Not a view; the preview effect marks the file viewed once it is displayed.
const file = await getFile(previewQuery, true);
setPreviewFile(file);
} catch (e) {
console.error(e);
@ -495,7 +514,7 @@ export function FilePreviewProvider({children}) {
// Get the file again with its Tags.
const {path, primary_path} = previewFile;
try {
const file = await getFile(primary_path ?? path, skipTracking);
const file = await getFile(primary_path ?? path, true);
setPreviewFile(file);
} finally {
await handleCallbacks();
@ -585,6 +604,9 @@ export function FilePreviewProvider({children}) {
React.useEffect(() => {
setPreviewModal(null);
if (_.isEmpty(previewFile)) {
trackedPathRef.current = null;
}
if (previewFile && !_.isObject(previewFile)) {
console.error(`Unknown previewFile type: ${typeof previewFile}`);
toast({
@ -607,6 +629,9 @@ export function FilePreviewProvider({children}) {
if (mimetype.startsWith('text/') && size > MAXIMUM_TEXT_SIZE) {
// Large text files should be downloaded.
window.open(downloadURL);
} else if (isMarkdownFile(mimetype, lowerPath)) {
setModalContent(<Modal.Content><MarkdownPreview url={url}/></Modal.Content>, url, downloadURL, path,
taggable);
} else if (mimetype.startsWith('text/') || mimetype.startsWith('application/json')) {
setModalContent(getIframePreviewModal(previewFile), url, downloadURL, path, taggable);
} else if (mimetype.startsWith('video/')) {
@ -615,7 +640,7 @@ export function FilePreviewProvider({children}) {
setModalContent(getAudioPreviewModal(previewFile), url, downloadURL, path, taggable);
} else if (mimetype.startsWith('application/epub')) {
const viewerURL = getEpubViewerURL(previewFile);
setModalContent(getIframePreviewModal(previewFile, viewerURL), viewerURL, downloadURL, path, taggable);
setModalContent(<EpubPreview previewFile={previewFile}/>, viewerURL, downloadURL, path, taggable);
} else if (mimetype.startsWith('application/pdf')) {
setModalContent(getIframePreviewModal(previewFile, null, {gatePdf: true}), url, downloadURL, path, taggable);
} else if (mimetype.startsWith('image/')) {
@ -637,7 +662,9 @@ export function FilePreviewProvider({children}) {
<OpenInSlicerButton previewFile={previewFile}/>);
} else if (mimetype.includes('cbz') || mimetype.includes('cbr') || mimetype.includes('comicbook+zip') || mimetype.includes('comicbook-rar')
|| lowerPath.endsWith('.cbz') || lowerPath.endsWith('.cbr') || lowerPath.endsWith('.cbt') || lowerPath.endsWith('.cb7')) {
setModalContent(<Modal.Content><CbzViewer path={path}/></Modal.Content>, null, downloadURL, path, taggable);
setModalContent(<Modal.Content>
<CbzViewer path={path} progressPath={path} fileGroup={previewFile}/>
</Modal.Content>, null, downloadURL, path, taggable);
} else if (isSupportedArchive(mimetype, lowerPath)) {
setModalContent(<ArchivePreviewContent previewFile={previewFile}/>, null, downloadURL, path, taggable);
} else {
@ -646,7 +673,8 @@ export function FilePreviewProvider({children}) {
}
// Trigger tracking for the file view (unless on an excluded route)
if (!skipTracking) {
if (!skipTracking && trackedPathRef.current !== path) {
trackedPathRef.current = path;
getFile(path).catch(e => {
console.error(e);
console.error('Failed to get file to set FileGroup.viewed');

View file

@ -101,6 +101,26 @@ describe('preview gate remounting (source contract)', () => {
});
});
describe('markdown preview routing (source contract)', () => {
/*
* Markdown is text/*, so the generic text branch would also accept it and show it raw in an
* iframe. The markdown branch has to be tested first for the formatted view to be reachable.
*/
const fs = require('fs');
const path = require('path');
const source = fs.readFileSync(path.join(__dirname, 'FilePreview.js'), 'utf8');
test('markdown is routed to MarkdownPreview before the generic text preview', () => {
const markdownBranch = source.indexOf('} else if (isMarkdownFile(mimetype, lowerPath)) {');
const textBranch = source.indexOf("} else if (mimetype.startsWith('text/') || mimetype.startsWith('application/json')) {");
expect(markdownBranch).toBeGreaterThan(-1);
expect(textBranch).toBeGreaterThan(-1);
expect(markdownBranch).toBeLessThan(textBranch);
expect(source.slice(markdownBranch, textBranch)).toContain('<MarkdownPreview url={url}/>');
});
});
describe('IframePreview color scheme', () => {
/*
* The previewed file is a bare document with no styles of its own, so it renders with the

View file

@ -0,0 +1,80 @@
import React from 'react';
import {act, waitFor} from '@testing-library/react';
import {renderWithProviders} from '../test-utils';
import {FilePreviewContext, FilePreviewProvider} from './FilePreview';
import {getFile} from '../api';
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getFile: jest.fn(),
tagFileGroup: jest.fn(),
untagFileGroup: jest.fn(),
}));
const imageFile = {primary_path: 'pictures/cat.png', mimetype: 'image/png', size: 100, taggable: true, tags: []};
/** getFile calls that would mark the file viewed (skipTracking not set). */
const trackedCalls = () => getFile.mock.calls.filter(([, skipTracking]) => !skipTracking);
/** Exposes the provider's context so a test can open a preview the way PreviewLink does. */
let context;
const ContextSpy = () => {
context = React.useContext(FilePreviewContext);
return null;
};
const renderProvider = (preview = null) => renderWithProviders(
<FilePreviewProvider><ContextSpy/></FilePreviewProvider>,
{
route: '/files',
query: {searchParams: new URLSearchParams(preview ? {preview} : {}), updateQuery: jest.fn()},
},
);
describe('FilePreview view tracking', () => {
beforeEach(() => {
getFile.mockReset();
getFile.mockResolvedValue(imageFile);
context = null;
});
test('opening a preview from the URL marks the file viewed once', async () => {
renderProvider(imageFile.primary_path);
await waitFor(() => expect(getFile).toHaveBeenCalled());
await waitFor(() => expect(context.previewModal).not.toBeNull());
expect(trackedCalls()).toHaveLength(1);
});
test('opening a preview from a link marks the file viewed once', async () => {
renderProvider();
act(() => context.setPreviewFile(imageFile));
await waitFor(() => expect(trackedCalls()).toHaveLength(1));
expect(getFile).toHaveBeenCalledTimes(1);
});
test('refreshing the open file (after tagging) is not another view', async () => {
renderProvider();
act(() => context.setPreviewFile(imageFile));
await waitFor(() => expect(trackedCalls()).toHaveLength(1));
// localFetchFile replaces previewFile with the re-fetched file after a tag change.
act(() => context.setPreviewFile({...imageFile, tags: ['animals']}));
await waitFor(() => expect(context.previewFile.tags).toEqual(['animals']));
expect(trackedCalls()).toHaveLength(1);
});
test('closing and reopening the same file is another view', async () => {
renderProvider();
act(() => context.setPreviewFile(imageFile));
await waitFor(() => expect(trackedCalls()).toHaveLength(1));
act(() => context.setPreviewFile(null));
act(() => context.setPreviewFile(imageFile));
await waitFor(() => expect(trackedCalls()).toHaveLength(2));
});
});

View file

@ -3,7 +3,7 @@ import {Route, Routes} from "react-router";
import {ESPLoader, Transport} from "esptool-js";
import {useDropzone} from "react-dropzone";
import _ from "lodash";
import {encodeMediaPath, humanFileSize, PageContainer, useTitle} from "./Common";
import {encodeMediaPath, ErrorMessage, humanFileSize, PageContainer, useTitle} from "./Common";
import {
Button,
Checkbox,
@ -17,6 +17,7 @@ import {
Message,
Modal,
Panel,
Placeholder,
Progress,
Select,
Stack,
@ -269,7 +270,8 @@ export function FlasherPage() {
// Media-directory firmware picker.
const [mediaFilter, setMediaFilter] = useState('');
const [mediaResults, setMediaResults] = useState([]);
// null = not searched yet, undefined = the search failed, a list = the server's results.
const [mediaResults, setMediaResults] = useState(null);
const [mediaLoading, setMediaLoading] = useState(false);
// When set (e.g. "ESP32-S2"), the picker only shows firmware for that chip (detected from the device).
const [deviceChip, setDeviceChip] = useState(null);
@ -297,8 +299,12 @@ export function FlasherPage() {
// Which firmware-source tab is active (controlled so a file drop can switch to "Add from computer").
const [activeTab, setActiveTab] = useState(TAB_SAVED);
// Saved firmware configurations (flasher.yaml).
const [savedConfigs, setSavedConfigs] = useState([]);
// Saved firmware configurations (flasher.yaml). null = pending, undefined = fetch failed,
// [] = none saved; the "No saved configurations yet." message is only for the last.
const [savedConfigs, setSavedConfigs] = useState(null);
// Only the newest configs request may write state: a save or delete refetches while the mount
// request may still be in flight.
const savedConfigsGen = useRef(0);
const [saveModalOpen, setSaveModalOpen] = useState(false);
const [saveName, setSaveName] = useState('');
@ -388,14 +394,15 @@ export function FlasherPage() {
const fetchMediaFirmware = React.useCallback(async (pathFilter, chip) => {
setMediaLoading(true);
try {
const [fileGroups] = await flasherSearch(chip || null, pathFilter || null);
setMediaResults(fileGroups || []);
const result = await flasherSearch(chip || null, pathFilter || null);
// The api helper returns undefined (and toasts) on a non-OK response.
setMediaResults(result ? result[0] || [] : undefined);
} catch (e) {
// Surface connectivity errors instead of showing the empty "no firmware found" state, which would
// be indistinguishable from genuinely having no firmware.
setError(e && e.message ? e.message : String(e));
setBootHint(false);
setMediaResults([]);
setMediaResults(undefined);
} finally {
setMediaLoading(false);
}
@ -424,7 +431,19 @@ export function FlasherPage() {
// Saved firmware configurations.
const fetchSavedConfigs = React.useCallback(async () => {
setSavedConfigs(await getFlasherConfigs());
const gen = ++savedConfigsGen.current;
try {
const configs = await getFlasherConfigs();
if (gen === savedConfigsGen.current) {
// The api helper returns undefined on a non-OK response.
setSavedConfigs(configs ?? undefined);
}
} catch (e) {
console.error(e);
if (gen === savedConfigsGen.current) {
setSavedConfigs(undefined);
}
}
}, []);
useEffect(() => {
@ -750,8 +769,10 @@ export function FlasherPage() {
onChange={(e) => handleMediaFilterChange(e.currentTarget.value)}
/>
<div style={{marginTop: '1em', overflowX: 'auto'}}>
{mediaLoading
{mediaLoading || mediaResults === null
? <Loading/>
: mediaResults === undefined
? <ErrorMessage>Could not search firmware.</ErrorMessage>
: (mediaResults.length === 0
? <p style={{opacity: 0.7}}>
No <code>.bin</code> firmware found
@ -812,7 +833,9 @@ export function FlasherPage() {
Configure the firmware files and offsets, then save them as a named set to re-flash later
(e.g. a Meshtastic T-Deck: firmware at <code>0x0</code>, littlefs at <code>0xc90000</code>).
</p>
{savedConfigs.length > 0
{savedConfigs === null && <Placeholder lines={2}/>}
{savedConfigs === undefined && <ErrorMessage>Could not fetch saved configurations.</ErrorMessage>}
{Array.isArray(savedConfigs) && (savedConfigs.length > 0
? <Panel style={{padding: 0}}>
{savedConfigs.map((configuration, index) => <div
key={configuration.name}
@ -846,7 +869,7 @@ export function FlasherPage() {
onClick={() => handleDeleteConfig(configuration.name)}/>
</div>)}
</Panel>
: <p style={{opacity: 0.7}}>No saved configurations yet.</p>}
: <p style={{opacity: 0.7}}>No saved configurations yet.</p>)}
</>;
return <div {...getRootProps()}>

View file

@ -265,6 +265,43 @@ describe('FlasherPage', () => {
}
});
// The Saved Firmwares tab is the default tab. "No saved configurations yet." may only appear once the
// server confirmed there are none; a pending or failed fetch must not read as an empty list.
it('shows a placeholder, not "no saved configurations", while the configs load', () => {
Object.defineProperty(global.navigator, 'serial', {value: {}, configurable: true});
getFlasherConfigs.mockReturnValue(new Promise(() => {
}));
const {container} = render(<FlasherPage/>);
expect(container.querySelector('.wrolpi-placeholder')).toBeInTheDocument();
expect(screen.queryAllByText('No saved configurations yet.')).toHaveLength(0);
});
it('shows an error, not "no saved configurations", when the configs fetch failed', async () => {
Object.defineProperty(global.navigator, 'serial', {value: {}, configurable: true});
getFlasherConfigs.mockResolvedValue(undefined);
render(<FlasherPage/>);
expect((await screen.findAllByText(/Could not fetch saved configurations/)).length).toBeGreaterThan(0);
expect(screen.queryAllByText('No saved configurations yet.')).toHaveLength(0);
});
it('shows "no saved configurations" once the server confirmed there are none', async () => {
Object.defineProperty(global.navigator, 'serial', {value: {}, configurable: true});
render(<FlasherPage/>);
expect((await screen.findAllByText('No saved configurations yet.')).length).toBeGreaterThan(0);
});
// "No .bin firmware found" may only appear once the server searched and found none.
it('shows an error, not "no firmware found", when the firmware search failed', async () => {
Object.defineProperty(global.navigator, 'serial', {value: {}, configurable: true});
// The api helper returns undefined on a non-OK response.
flasherSearch.mockResolvedValue(undefined);
render(<FlasherPage/>);
await waitFor(() => expect(flasherSearch).toHaveBeenCalled());
switchTab('Choose from your WROLPi');
expect((await screen.findAllByText(/Could not search firmware/)).length).toBeGreaterThan(0);
expect(screen.queryAllByText(/firmware found/)).toHaveLength(0);
});
// PageContainer renders both the mobile and desktop breakpoints (fresnel Media), so each label appears twice.
it('warns when Web Serial is unavailable', () => {
delete global.navigator.serial;

View file

@ -1,4 +1,4 @@
import React, {useEffect, useState} from "react";
import React, {useEffect, useRef, useState} from "react";
import {
APIButton,
ErrorMessage,
@ -20,6 +20,7 @@ import {Media, StatusContext} from "../contexts/contexts";
import {MAP_VIEWER_URI} from "./Vars";
import MapViewer from "./MapViewer";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {solarCalculatorPath} from "./calculators/solar";
import maplibregl from "maplibre-gl";
import {Protocol} from "pmtiles";
import layers from "protomaps-themes-base";
@ -284,11 +285,15 @@ function MapCatalogRow({item, subscribedRegions, fetchData}) {
function ManageMap() {
const [files, setFiles] = useState(null);
const [catalog, setCatalog] = useState([]);
// null = pending, undefined = fetch failed, [] = no regions in the catalog.
const [catalog, setCatalog] = useState(null);
const [subscriptions, setSubscriptions] = useState([]);
const [allRegionsOpen, setAllRegionsOpen] = useState(false);
const {status} = React.useContext(StatusContext);
const searchBuilding = status?.flags?.map_search_building;
// Only the newest subscriptions request may write state: a subscribe or unsubscribe refetches
// while the mount request may still be in flight.
const subscriptionsGen = useRef(0);
const fetchFiles = async () => {
try {
@ -303,14 +308,24 @@ function ManageMap() {
};
const fetchSubscriptionData = async () => {
const gen = ++subscriptionsGen.current;
try {
const data = await fetchMapSubscriptions();
if (gen !== subscriptionsGen.current) {
return;
}
if (data) {
setCatalog(data.catalog || []);
setSubscriptions(data.subscriptions || []);
} else {
// The api helper already toasted the non-OK response.
setCatalog(undefined);
}
} catch (e) {
console.error(e);
if (gen === subscriptionsGen.current) {
setCatalog(undefined);
}
}
};
@ -397,6 +412,8 @@ function ManageMap() {
</>;
}
const catalogLoaded = Array.isArray(catalog);
/*
* A fragment, not a `PageContainer`. `MapRoute` already provides the page chrome -- a
* `wrolpi-stack` wrapper with its own top margin -- so a second one put this page a further
@ -419,18 +436,26 @@ function ManageMap() {
<div style={{display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: '0.5em'}}>
<Header as='h3'>Map Subscriptions</Header>
<Button size='xs' icon='globe' onClick={() => setAllRegionsOpen(true)}>View All Regions</Button>
{/* A pending or failed catalog is not "zero regions": the preview waits for one. The
modal is mounted only then, so it receives the state array itself and its map is
not rebuilt by a fresh `[]` on every status tick. */}
<Button size='xs' icon='globe' disabled={!catalogLoaded} onClick={() => setAllRegionsOpen(true)}>
View All Regions
</Button>
</div>
<AllRegionsPreviewModal catalog={catalog} open={allRegionsOpen} onClose={() => setAllRegionsOpen(false)}/>
{catalogLoaded && <AllRegionsPreviewModal catalog={catalog} open={allRegionsOpen}
onClose={() => setAllRegionsOpen(false)}/>}
<SortableTable
tableHeaders={catalogHeaders}
data={catalog}
rowFunc={catalogRowFunc}
rowKey='region'
defaultSortColumn='name'
/>
{catalog === undefined
? <ErrorMessage>Could not fetch map subscriptions</ErrorMessage>
: <SortableTable
tableHeaders={catalogHeaders}
data={catalog}
rowFunc={catalogRowFunc}
rowKey='region'
defaultSortColumn='name'
/>}
<InfoMessage storageName='hint_map_subscribe'>
<div style={{fontWeight: 600, marginBottom: 4}}>How to get map files</div>
@ -497,17 +522,29 @@ function PinEditRow({pin, onSave, onCancel}) {
}
function MapPins() {
const [pins, setPins] = useState([]);
// null = pending, undefined = fetch failed, [] = no pins. The empty row invites the user to
// add one, so it may only appear for a confirmed empty list.
const [pins, setPins] = useState(null);
const [filter, setFilter] = useState('');
const [editingId, setEditingId] = useState(null);
const navigate = useNavigate();
// Only the newest pins request may write state: an edit or delete refetches while the mount
// request may still be in flight.
const pinsGen = useRef(0);
const fetchPins = async () => {
const gen = ++pinsGen.current;
try {
const data = await getMapPins();
if (data) setPins(data.pins || []);
if (gen === pinsGen.current) {
// The api helper returns undefined on a non-OK response.
setPins(data ? data.pins || [] : undefined);
}
} catch (e) {
console.error(e);
if (gen === pinsGen.current) {
setPins(undefined);
}
}
};
@ -531,6 +568,9 @@ function MapPins() {
navigate(`/map?lat=${pin.lat}&lon=${pin.lon}&z=14`);
};
const solarButton = (pin) => <Button size='xs' icon='sun' aria-label={`Solar estimate for ${pin.label}`}
onClick={() => navigate(solarCalculatorPath(pin.lat, pin.lon))}/>;
// --- Desktop headers and row ---
const fullHeaders = [
{key: 'color', text: '', sortBy: i => i.color},
@ -564,10 +604,11 @@ function MapPins() {
</Table.Cell>
<Table.Cell>{pin.created}</Table.Cell>
<Table.Cell>
{/* See PinEditRow: a Group so the three controls line up and are spaced. */}
{/* See PinEditRow: a Group so the controls line up and are spaced. */}
<Group gap='xs' wrap='nowrap'>
<Button size='xs' icon='edit' aria-label={`Edit ${pin.label}`}
onClick={() => setEditingId(pin.id)}/>
{solarButton(pin)}
<AddToPlaylistButton
size='xs'
icon='list'
@ -610,15 +651,18 @@ function MapPins() {
<div style={{fontSize: '0.6875rem', color: 'var(--muted)'}}>{pin.lat.toFixed(4)}, {pin.lon.toFixed(4)}</div>
</Table.Cell>
<Table.Cell>
<APIButton
size='xs'
role='danger'
icon='trash'
aria-label={`Delete ${pin.label}`}
confirmContent='Delete this pin?'
confirmButton='Delete'
onClick={() => handleDelete(pin.id)}
/>
<Group gap='xs' wrap='nowrap'>
{solarButton(pin)}
<APIButton
size='xs'
role='danger'
icon='trash'
aria-label={`Delete ${pin.label}`}
confirmContent='Delete this pin?'
confirmButton='Delete'
onClick={() => handleDelete(pin.id)}
/>
</Group>
</Table.Cell>
</Table.Row>;
};
@ -627,8 +671,13 @@ function MapPins() {
<Table.Cell colSpan={5}>No pins. Right-click on the map to add one.</Table.Cell>
</Table.Row>;
if (pins === undefined) {
return <ErrorMessage>Could not fetch map pins</ErrorMessage>;
}
const lowerFilter = filter.toLowerCase();
const filteredPins = lowerFilter
// Pending (null) passes through so SortableTable shows its placeholder.
const filteredPins = lowerFilter && pins
? pins.filter(p =>
(p.label || '').toLowerCase().includes(lowerFilter) ||
`${p.lat.toFixed(4)}, ${p.lon.toFixed(4)}`.includes(lowerFilter)

View file

@ -0,0 +1,63 @@
import {poiPopupHtml} from './mapPoiPopup';
// Clicking a POI on the map opens a popup built from attributes already in the pmtiles
// file: the name, its category (kind), sub-category (kind_detail), and elevation.
const feature = (properties) => ({properties});
test('name, kind, detail and elevation are all shown, humanized', () => {
const html = poiPopupHtml([feature({
name: 'Flattop Mountain', kind: 'peak', kind_detail: 'volcano', elevation: 1070,
})]);
expect(html).toContain('Flattop Mountain');
expect(html).toContain('Peak');
expect(html).toContain('Volcano');
expect(html).toContain('1070 m');
expect(html).toContain('3510 ft');
});
test('missing attributes are left out rather than shown as undefined', () => {
const html = poiPopupHtml([feature({name: 'Franz', kind: 'bakery'})]);
expect(html).toContain('Franz');
expect(html).toContain('Bakery');
expect(html).not.toContain('undefined');
expect(html).not.toContain('Elevation');
});
test('kind_detail of "none" is not shown', () => {
const html = poiPopupHtml([feature({name: 'West High', kind: 'school', kind_detail: 'none'})]);
expect(html).not.toContain('none');
expect(html).not.toContain('None');
});
test('the same feature at several min_zoom levels appears once', () => {
const html = poiPopupHtml([
feature({name: 'Westchester Lagoon', kind: 'water', min_zoom: 12}),
feature({name: 'Westchester Lagoon', kind: 'water', min_zoom: 14}),
]);
expect(html.match(/Westchester Lagoon/g)).toHaveLength(1);
});
test('feature properties are HTML escaped', () => {
const html = poiPopupHtml([feature({name: '<img src=x onerror=alert(1)>', kind: 'a&b'})]);
expect(html).not.toContain('<img');
expect(html).toContain('&lt;img');
expect(html).toContain('A&amp;b');
});
test('a point feature shows its own coordinates, lat then lon', () => {
const html = poiPopupHtml([{
properties: {name: 'Peak Two', kind: 'peak'},
geometry: {type: 'Point', coordinates: [-149.67301, 61.08571]},
}], {lng: -149.5, lat: 61.0});
expect(html).toContain('61.08571, -149.67301');
});
test('a feature without a point falls back to the clicked location', () => {
const html = poiPopupHtml([feature({name: 'Lagoon', kind: 'water'})], {lng: -149.5, lat: 61.0});
expect(html).toContain('61.00000, -149.50000');
});
test('no features gives no popup', () => {
expect(poiPopupHtml([])).toBeNull();
});

View file

@ -2,7 +2,7 @@ import React, {useEffect, useState} from "react";
import {Link, useNavigate} from "react-router";
import {getMapSearchStatus, searchMap} from "../api";
import {Card, CardGroup, Header, Loading, Panel} from "./ui";
import {Paginator} from "./Common";
import {ErrorMessage, Paginator} from "./Common";
import {useLatestRequest, usePages} from "../hooks/customHooks";
import {QueryContext} from "../contexts/contexts";
import {MAP_VIEWER_URI} from "./Vars";
@ -68,16 +68,24 @@ export function MapSearchView() {
const {offset, limit, activePage, setPage, totalPages, setTotal} = usePages(12);
// useLatestRequest discards stale responses and catches errors, so rapid pagination
// can't show out-of-order results and a network failure can't leave the spinner stuck.
const {data, sendRequest, loading} = useLatestRequest(300);
// Loading from the first render: a query of two or more characters always sends a request (below),
// so this settles, and the shorter-query branches render before the loading check.
const {data, sendRequest, loading} = useLatestRequest(300, true);
// null = unknown (pending, or the status could not be fetched), otherwise whether any map is
// installed. Only a real status may claim that no maps are installed.
const [hasMaps, setHasMaps] = useState(null);
useEffect(() => {
let cancelled = false;
(async () => {
const status = await getMapSearchStatus();
if (!cancelled) {
const installed = (status.indexed || []).length + (status.missing || []).length;
setHasMaps(installed > 0);
try {
const status = await getMapSearchStatus();
if (!cancelled && status) {
const installed = (status.indexed || []).length + (status.missing || []).length;
setHasMaps(installed > 0);
}
} catch (e) {
console.error(e);
}
})();
return () => {
@ -106,12 +114,22 @@ export function MapSearchView() {
</Panel>;
}
if (!searchStr) {
return <Panel><Header as="h4">Enter a search query to find map locations.</Header></Panel>;
}
if (searchStr.length < 2) {
return <Panel><Header as="h4">Enter at least two characters to find map locations.</Header></Panel>;
}
if (loading) {
return <Panel><Loading/></Panel>;
}
if (!searchStr) {
return <Panel><Header as="h4">Enter a search query to find map locations.</Header></Panel>;
// Settled with no data: the request failed (the api helper returns undefined on a non-OK
// response, and useLatestRequest leaves null when it throws).
if (data == null) {
return <Panel><ErrorMessage>Could not search the map.</ErrorMessage></Panel>;
}
if (results.length === 0) {

View file

@ -0,0 +1,61 @@
import React from 'react';
import {render, screen} from '../test-utils';
import {MapSearchView} from './MapSearchView';
import {getMapSearchStatus, searchMap} from '../api';
// The map search tab says "No map locations found" only when the server searched and found none,
// and "No maps are installed" only when the server said so. A failed request is an error.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getMapSearchStatus: jest.fn(),
searchMap: jest.fn(),
}));
const NOT_FOUND = /No map locations found/;
const NO_MAPS = /No maps are installed/;
const FAILED = /Could not search the map/;
const renderAt = (q) => render(<MapSearchView/>, {query: {searchParams: new URLSearchParams({q})}});
beforeEach(() => {
getMapSearchStatus.mockReset();
searchMap.mockReset();
getMapSearchStatus.mockResolvedValue({indexed: ['utah'], missing: []});
});
test('found nothing: says so', async () => {
searchMap.mockResolvedValue({results: [], total: 0});
renderAt('camp');
expect(await screen.findByText(NOT_FOUND)).toBeInTheDocument();
});
test('a failed search shows an error, not "No map locations found"', async () => {
searchMap.mockResolvedValue(undefined);
renderAt('camp');
expect(await screen.findByText(FAILED)).toBeInTheDocument();
expect(screen.queryByText(NOT_FOUND)).not.toBeInTheDocument();
});
test('a failed status check does not claim no maps are installed', async () => {
getMapSearchStatus.mockResolvedValue(undefined);
searchMap.mockResolvedValue({results: [], total: 0});
renderAt('camp');
expect(await screen.findByText(NOT_FOUND)).toBeInTheDocument();
expect(screen.queryByText(NO_MAPS)).not.toBeInTheDocument();
});
test('a thrown status check does not claim no maps are installed', async () => {
getMapSearchStatus.mockRejectedValue(new Error('down'));
searchMap.mockResolvedValue({results: [], total: 0});
renderAt('camp');
expect(await screen.findByText(NOT_FOUND)).toBeInTheDocument();
expect(screen.queryByText(NO_MAPS)).not.toBeInTheDocument();
});
test('the server saying no maps are installed is shown', async () => {
getMapSearchStatus.mockResolvedValue({indexed: [], missing: []});
searchMap.mockResolvedValue({results: [], total: 0});
renderAt('camp');
expect(await screen.findByText(NO_MAPS)).toBeInTheDocument();
});

View file

@ -0,0 +1,158 @@
import React from 'react';
import {act, fireEvent, screen} from '@testing-library/react';
import {Route, Routes} from 'react-router';
import {renderWithProviders as render} from '../test-utils';
import {MapRoute} from './Map';
// What the map's Pins and Manage tabs show while their lists are still loading, when the
// fetch failed, and when the server confirmed the list is empty. The empty messages invite
// the user to act ("Right-click on the map to add one"), so they may only appear for a
// confirmed empty list.
jest.mock('maplibre-gl', () => ({
__esModule: true,
default: {Map: class {
}, addProtocol: jest.fn(), Marker: class {
}},
}));
jest.mock('pmtiles', () => ({
__esModule: true, Protocol: class {
tile = jest.fn();
},
}));
jest.mock('protomaps-themes-base', () => ({__esModule: true, default: () => []}));
jest.mock('./MapViewer', () => ({__esModule: true, default: () => null}));
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getMapFiles: jest.fn(),
fetchMapSubscriptions: jest.fn(),
getMapPins: jest.fn(),
}));
const api = require('../api');
const pending = () => new Promise(() => {
});
const NO_PINS = /No pins\. Right-click/;
const NO_RESULTS = /^No results$/;
// The pins page renders one table per breakpoint through `Media`, so the real media provider is
// needed; the queries below tolerate either variant being in the tree.
const renderAt = (route) => render(
<Routes><Route path='/map/*' element={<MapRoute/>}/></Routes>,
{route, withMedia: true},
);
beforeEach(() => {
jest.clearAllMocks();
// The media provider reads matchMedia; the global stub's implementation is reset between
// tests, so give it one here. Only the computer breakpoint (1024px) matches, so the
// provider reports a desktop viewport and the pins page renders its desktop table.
window.matchMedia = jest.fn().mockImplementation(query => ({
matches: query.includes('1024'), media: query, onchange: null,
addListener: jest.fn(), removeListener: jest.fn(),
addEventListener: jest.fn(), removeEventListener: jest.fn(), dispatchEvent: jest.fn(),
}));
api.getMapFiles.mockResolvedValue({files: []});
api.fetchMapSubscriptions.mockResolvedValue({catalog: [], subscriptions: []});
api.getMapPins.mockResolvedValue({pins: []});
});
describe('the Pins tab', () => {
test('pending: shows a placeholder, not "No pins"', () => {
api.getMapPins.mockReturnValue(pending());
const {container} = renderAt('/map/pins');
expect(container.querySelector('.wrolpi-placeholder')).toBeInTheDocument();
expect(screen.queryAllByText(NO_PINS)).toHaveLength(0);
});
test('failed: shows an error, not "No pins"', async () => {
// The api helper returns undefined on a non-OK response.
api.getMapPins.mockResolvedValue(undefined);
renderAt('/map/pins');
await act(async () => {
});
expect(screen.getByText(/Could not fetch map pins/)).toBeInTheDocument();
expect(screen.queryAllByText(NO_PINS)).toHaveLength(0);
});
test('empty: shows "No pins"', async () => {
renderAt('/map/pins');
await act(async () => {
});
expect(screen.getAllByText(NO_PINS).length).toBeGreaterThan(0);
});
test('loaded: lists the pins', async () => {
api.getMapPins.mockResolvedValue({pins: [{id: 1, lat: 40.1, lon: -111.2, label: 'Camp', color: '#f00'}]});
renderAt('/map/pins');
await act(async () => {
});
expect(screen.getAllByText('Camp').length).toBeGreaterThan(0);
expect(screen.queryAllByText(NO_PINS)).toHaveLength(0);
});
test('a pin opens the Solar calculator at its location', async () => {
api.getMapPins.mockResolvedValue({pins: [{id: 1, lat: 40.12346, lon: -111.2, label: 'Camp', color: '#f00'}]});
renderAt('/map/pins');
await act(async () => {
});
fireEvent.click(screen.getAllByRole('button', {name: 'Solar estimate for Camp'})[0]);
expect(window.location.pathname).toBe('/more/calculators');
expect(window.location.search).toBe('?calc=solar&lat=40.1235&lon=-111.2');
});
});
describe('the Manage tab subscriptions table', () => {
test('pending: shows a placeholder, not "No results"', async () => {
api.fetchMapSubscriptions.mockReturnValue(pending());
const {container} = renderAt('/map/manage');
await act(async () => {
});
// The files table has loaded (empty); the subscriptions table is still pending.
expect(container.querySelector('.wrolpi-placeholder')).toBeInTheDocument();
expect(screen.queryByText(NO_RESULTS)).not.toBeInTheDocument();
});
test('failed: shows an error, not "No results"', async () => {
api.fetchMapSubscriptions.mockResolvedValue(undefined);
renderAt('/map/manage');
await act(async () => {
});
expect(screen.getByText(/Could not fetch map subscriptions/)).toBeInTheDocument();
expect(screen.queryByText(NO_RESULTS)).not.toBeInTheDocument();
});
test('empty: shows "No results"', async () => {
renderAt('/map/manage');
await act(async () => {
});
expect(screen.getByText(NO_RESULTS)).toBeInTheDocument();
});
// The "View All Regions" preview draws the catalog; a pending or failed catalog must not be
// previewed as zero regions.
test('pending: the regions preview is disabled', async () => {
api.fetchMapSubscriptions.mockReturnValue(pending());
renderAt('/map/manage');
await act(async () => {
});
expect(screen.getByRole('button', {name: 'View All Regions'})).toBeDisabled();
});
test('failed: the regions preview is disabled', async () => {
api.fetchMapSubscriptions.mockResolvedValue(undefined);
renderAt('/map/manage');
await act(async () => {
});
expect(screen.getByRole('button', {name: 'View All Regions'})).toBeDisabled();
});
test('loaded: the regions preview is enabled', async () => {
renderAt('/map/manage');
await act(async () => {
});
expect(screen.getByRole('button', {name: 'View All Regions'})).toBeEnabled();
});
});

View file

@ -1,5 +1,5 @@
import React, {useCallback, useContext, useEffect, useRef, useState} from "react";
import {useSearchParams} from "react-router";
import {useNavigate, useSearchParams} from "react-router";
import maplibregl from "maplibre-gl";
import "maplibre-gl/dist/maplibre-gl.css";
import {Protocol} from "pmtiles";
@ -10,6 +10,8 @@ import {MAP_VIEWER_URI} from "./Vars";
import {Button, Checkbox, Header, Icon, Panel, TextInput, Toggle} from "./ui";
import {SettingsContext, ThemeContext} from "../contexts/contexts";
import {mapFlavor, mapSprite} from "../themes/names";
import {poiPopupHtml} from "./mapPoiPopup";
import {solarCalculatorPath} from "./calculators/solar";
// Terrain DEM file prefix for hillshade and contours.
const TERRAIN_PREFIX = "terrain-";
@ -421,10 +423,13 @@ function MapSearch({map}) {
lon = center.lng;
}
// A failed search (undefined) shows no dropdown, same as finding nothing: this is an
// as-you-type suggestion list, not a results page.
searchMap(q, 8, 0, lat, lon).then(data => {
setResults(data.results || []);
setShowResults((data.results || []).length > 0);
});
const found = data?.results || [];
setResults(found);
setShowResults(found.length > 0);
}).catch(e => console.error(e));
}, [map]);
const handleInput = useCallback((e) => {
@ -527,6 +532,7 @@ export default function MapViewer() {
const [mapReady, setMapReady] = useState(false);
const [error, setError] = useState(null);
const [contextMenu, setContextMenu] = useState(null);
const navigate = useNavigate();
const [addingPin, setAddingPin] = useState(null); // {lat, lon} when pin dialog is open
const markersRef = useRef([]);
const pinsVisibleRef = useRef(true);
@ -672,6 +678,29 @@ export default function MapViewer() {
map.on("click", () => setContextMenu(null));
map.on("movestart", () => setContextMenu(null));
// Click a POI to see the details the pmtiles file already has (kind, detail, elevation).
const poiLayerIds = () => (map.getStyle()?.layers || [])
.map(l => l.id)
.filter(id => id === "pois" || id.endsWith(":pois"))
.filter(id => map.getLayer(id));
map.on("click", (e) => {
if (destroyed) return;
const layers = poiLayerIds();
if (!layers.length) return;
const html = poiPopupHtml(map.queryRenderedFeatures(e.point, {layers}), e.lngLat);
if (!html) return;
new maplibregl.Popup({offset: 8, maxWidth: "260px", className: "wrolpi-map-popup"})
.setLngLat(e.lngLat)
.setHTML(html)
.addTo(map);
});
map.on("mousemove", (e) => {
if (destroyed) return;
const layers = poiLayerIds();
const hit = layers.length && map.queryRenderedFeatures(e.point, {layers}).length;
map.getCanvas().style.cursor = hit ? "pointer" : "";
});
// Long-press context menu (mobile).
let longPressTimer = null;
let touchStartPos = null;
@ -745,12 +774,16 @@ export default function MapViewer() {
}
mapRef.current = null;
setMapReady(false);
// Clean up map params from URL so they don't leak to other pages.
// Clean up map params from URL so they don't leak to other pages. Only while the URL is
// still a map page: after navigating away, lat/lon belong to the new page (the Solar
// calculator, for one).
const url = new URL(window.location);
url.searchParams.delete("lat");
url.searchParams.delete("lon");
url.searchParams.delete("z");
window.history.replaceState(null, '', url.pathname + url.search);
if (url.pathname.startsWith("/map")) {
url.searchParams.delete("lat");
url.searchParams.delete("lon");
url.searchParams.delete("z");
window.history.replaceState(null, '', url.pathname + url.search);
}
};
}, []); // eslint-disable-line react-hooks/exhaustive-deps
@ -896,6 +929,12 @@ export default function MapViewer() {
setContextMenu(null);
}, [contextMenu]);
const handleSolar = useCallback(() => {
if (!contextMenu) return;
setContextMenu(null);
navigate(solarCalculatorPath(contextMenu.lat, contextMenu.lon));
}, [contextMenu, navigate]);
const handleSetDefaultLocation = useCallback(async () => {
const map = mapRef.current;
if (!map) return;
@ -972,6 +1011,14 @@ export default function MapViewer() {
>
Add Pin Here
</div>
<div
style={{padding: "0.5rem 0.75rem", cursor: "pointer", userSelect: "none"}}
onMouseEnter={e => e.target.style.background = "#f0f0f0"}
onMouseLeave={e => e.target.style.background = "transparent"}
onClick={handleSolar}
>
Calculate Solar Performance
</div>
<div
style={{padding: "0.5rem 0.75rem", cursor: "pointer", userSelect: "none"}}
onMouseEnter={e => e.target.style.background = "#f0f0f0"}

View file

@ -0,0 +1,45 @@
import React from 'react';
import {renderWithProviders as render} from '../test-utils';
import MapViewer from './MapViewer';
// MapLibre needs WebGL, which jsdom lacks; these tests only cover what the viewer does to the URL.
jest.mock('maplibre-gl', () => ({
__esModule: true,
default: {Map: class {
}, addProtocol: jest.fn(), Marker: class {
}, Popup: class {
}},
}));
jest.mock('maplibre-gl/dist/maplibre-gl.css', () => ({}));
jest.mock('pmtiles', () => ({
__esModule: true, Protocol: class {
tile = jest.fn();
},
}));
jest.mock('protomaps-themes-base', () => ({__esModule: true, default: () => []}));
jest.mock('@acalcutt/maplibre-contour-pmtiles', () => ({__esModule: true, default: {}}));
jest.mock('../api', () => ({
...jest.requireActual('../api'),
// Never resolves: the map never initializes, which these tests do not need.
getMapFiles: jest.fn(() => new Promise(() => {
})),
getMapPins: jest.fn(() => new Promise(() => {
})),
}));
describe('the map position in the URL', () => {
test('is removed when the map closes', () => {
const {unmount} = render(<MapViewer/>, {route: '/map?lat=40&lon=-105&z=8&layer=a'});
unmount();
expect(window.location.pathname).toBe('/map');
expect(window.location.search).toBe('?layer=a');
});
test('is not removed from the page the map navigated to', () => {
// "Calculate Solar Performance" navigates to a page whose own lat/lon must survive the map closing.
const {unmount} = render(<MapViewer/>, {route: '/map?lat=40&lon=-105&z=8'});
window.history.pushState({}, '', '/more/calculators?calc=solar&lat=40.0154&lon=-105.2705');
unmount();
expect(window.location.search).toBe('?calc=solar&lat=40.0154&lon=-105.2705');
});
});

View file

@ -0,0 +1,170 @@
import React from "react";
import MarkdownIt from "markdown-it";
import DOMPurify from "dompurify";
import {useLocalStorage} from "./Common";
import {Button, ButtonGroup, Loading, Message} from "./ui";
/*
* Raw HTML in the file is escaped, not parsed: markdown files arrive with downloads, and the preview
* renders on the API's origin. `validateLink` (on by default) refuses javascript:, vbscript:, file:,
* and non-image data: URLs. DOMPurify runs anyway, so neither guard is the only one.
*/
const markdown = new MarkdownIt({html: false, linkify: true});
// A URL with a scheme (https:, mailto:, ...) points somewhere other than the media directory.
const URL_WITH_SCHEME = /^[a-z][a-z0-9+.-]*:/i;
/**
* Resolve a URL written in a markdown file against the file's own media URL, the way a browser
* would resolve it against a document in that directory.
*
* Returns the URL unchanged if it is not relative, or null if it climbs out of the media directory;
* only media is served relative to a file.
*/
function resolveRelativeURL(url, fileURL) {
if (!url || URL_WITH_SCHEME.test(url) || url.startsWith('/') || url.startsWith('#')) {
return url;
}
const resolved = new URL(url, new URL(fileURL, window.location.origin));
if (!resolved.pathname.startsWith('/media/')) {
return null;
}
return resolved.pathname + resolved.search + resolved.hash;
}
function rewriteURL(token, attr, fileURL) {
const url = token.attrGet(attr);
if (url === null) {
return;
}
const resolved = resolveRelativeURL(url, fileURL);
if (resolved === null) {
token.attrs = token.attrs.filter(([name]) => name !== attr);
} else {
token.attrSet(attr, resolved);
}
}
function openInNewTab(token) {
const href = token.attrGet('href');
// Following a link would close the preview; a fragment stays within the document.
if (href && !href.startsWith('#')) {
token.attrSet('target', '_blank');
token.attrSet('rel', 'noopener noreferrer');
}
}
/** An image the preview may load: one served by this WROLPi, or embedded in the file. */
function isLocalImage(src) {
return !src || src.startsWith('data:') || (src.startsWith('/') && !src.startsWith('//'));
}
/**
* Replace an image from another site with a link to it. Loading it would tell that site who is
* reading the file and when (a tracking pixel), so the reader chooses whether to follow it.
*
* Inside a link (a README badge) the image becomes the outer link's text; links cannot nest.
*/
function replaceRemoteImage(state, image, insideLink) {
const src = image.attrGet('src');
const text = new state.Token('text', '', 0);
text.content = image.content || src;
if (insideLink) {
return [text];
}
const linkOpen = new state.Token('link_open', 'a', 1);
linkOpen.attrSet('href', src);
openInNewTab(linkOpen);
return [linkOpen, text, new state.Token('link_close', 'a', -1)];
}
markdown.core.ruler.push('wrolpi_media_urls', (state) => {
const {fileURL} = state.env;
for (const token of state.tokens) {
if (!token.children) {
continue;
}
let insideLink = false;
token.children = token.children.flatMap(child => {
if (child.type === 'image') {
rewriteURL(child, 'src', fileURL);
return isLocalImage(child.attrGet('src')) ? [child] : replaceRemoteImage(state, child, insideLink);
} else if (child.type === 'link_open') {
insideLink = true;
rewriteURL(child, 'href', fileURL);
openInNewTab(child);
} else if (child.type === 'link_close') {
insideLink = false;
}
return [child];
});
}
});
/** Render markdown to sanitized HTML. Relative URLs resolve against `fileURL`, the file's /media/ URL. */
export function renderMarkdown(text, fileURL) {
const html = markdown.render(text, {fileURL});
return DOMPurify.sanitize(html, {ADD_ATTR: ['target']});
}
export function isMarkdownFile(mimetype, lowerPath) {
// By suffix too, so a markdown file whose stored type is still text/plain (or a language libmagic
// guessed) is not shown raw; the 2026_10_05_1200 migration retags those rows, a refresh does not.
return mimetype.startsWith('text/markdown')
|| (mimetype.startsWith('text/') && (lowerPath.endsWith('.md') || lowerPath.endsWith('.markdown')));
}
/** Preview a markdown file, formatted or as its raw source. */
export function MarkdownPreview({url}) {
const [formatted, setFormatted] = useLocalStorage('markdownPreviewFormatted', true);
const [text, setText] = React.useState(null);
const [error, setError] = React.useState(null);
React.useEffect(() => {
const controller = new AbortController();
setText(null);
setError(null);
fetch(url, {signal: controller.signal})
.then(response => {
if (!response.ok) {
throw new Error(`${response.status} ${response.statusText}`);
}
return response.text();
})
.then(setText)
.catch(e => {
if (e.name !== 'AbortError') {
console.error(e);
setError(e.message);
}
});
return () => controller.abort();
}, [url]);
const html = React.useMemo(() => text === null ? null : renderMarkdown(text, url), [text, url]);
let body;
if (error) {
body = <Message kind='error' title='Could not load file'>{error}</Message>;
} else if (text === null) {
body = <Loading/>;
} else if (formatted) {
body = <div className='markdown-body' dangerouslySetInnerHTML={{__html: html}}/>;
} else {
body = <pre className='markdown-raw' data-testid='markdown-raw'>{text}</pre>;
}
return <>
<ButtonGroup>
<Button size='small' variant={formatted ? 'filled' : 'default'} onClick={() => setFormatted(true)}>
Formatted
</Button>
<Button size='small' variant={formatted ? 'default' : 'filled'} onClick={() => setFormatted(false)}>
Raw
</Button>
</ButtonGroup>
<div className='markdown-preview'>
{body}
</div>
</>
}

View file

@ -0,0 +1,198 @@
import React from 'react';
import {fireEvent, screen, waitFor} from '@testing-library/react';
import {isMarkdownFile, MarkdownPreview, renderMarkdown} from './MarkdownPreview';
import {renderUI} from '../test-utils';
const FILE_URL = '/media/notes/water/storage.md';
/** Render markdown into a detached element so assertions read the DOM, not the HTML string. */
const renderToDOM = (text, fileURL = FILE_URL) => {
const div = document.createElement('div');
div.innerHTML = renderMarkdown(text, fileURL);
return div;
};
describe('renderMarkdown', () => {
test('formats headings, emphasis, lists, code, and tables', () => {
const div = renderToDOM([
'# Water',
'',
'Rotate *every* barrel.',
'',
'- one',
'- two',
'',
'```',
'boil 1 minute',
'```',
'',
'| a | b |',
'|---|---|',
'| 1 | 2 |',
].join('\n'));
expect(div.querySelector('h1').textContent).toBe('Water');
expect(div.querySelector('em').textContent).toBe('every');
expect(div.querySelectorAll('ul li')).toHaveLength(2);
expect(div.querySelector('pre code').textContent).toBe('boil 1 minute\n');
expect(div.querySelectorAll('table td')).toHaveLength(2);
});
test('raw HTML in the file is shown as text, never parsed', () => {
const div = renderToDOM('<script>alert(1)</script>\n\n<img src=x onerror="alert(2)">');
expect(div.querySelector('script')).toBeNull();
expect(div.querySelector('img')).toBeNull();
expect(div.textContent).toContain('<script>alert(1)</script>');
});
test('script URLs are not turned into links', () => {
const div = renderToDOM('[click](javascript:alert(1))');
expect(div.querySelector('a')).toBeNull();
});
test('relative images resolve against the markdown file directory', () => {
const div = renderToDOM('![pump](images/hand%20pump.png)\n\n![tank](../tank.jpg)');
const [pump, tank] = div.querySelectorAll('img');
expect(pump.getAttribute('src')).toBe('/media/notes/water/images/hand%20pump.png');
expect(tank.getAttribute('src')).toBe('/media/notes/tank.jpg');
});
test('relative links resolve against the markdown file directory and open a new tab', () => {
const a = renderToDOM('[filters](filters.md#ceramic)').querySelector('a');
expect(a.getAttribute('href')).toBe('/media/notes/water/filters.md#ceramic');
expect(a.getAttribute('target')).toBe('_blank');
});
test('a relative URL that climbs out of the media directory is dropped', () => {
const div = renderToDOM('[api](../../../api/status)\n\n![x](../../../../x.png)');
expect(div.querySelector('a').hasAttribute('href')).toBe(false);
expect(div.querySelector('img').hasAttribute('src')).toBe(false);
});
test.each([
['https://tracker.example/pixel.gif'],
['http://tracker.example/pixel.gif'],
// Protocol-relative: starts with a slash, but loads from another host.
['//tracker.example/pixel.gif'],
])('an image from another site (%s) is not loaded, it becomes a link', (src) => {
const div = renderToDOM(`![pixel](${src})`);
expect(div.querySelector('img')).toBeNull();
const a = div.querySelector('a');
expect(a.getAttribute('href')).toBe(src);
expect(a.textContent).toBe('pixel');
expect(a.getAttribute('target')).toBe('_blank');
expect(a.getAttribute('rel')).toBe('noopener noreferrer');
});
test('an image from another site with no alt text is labeled by its URL', () => {
const a = renderToDOM('![](https://example.com/a.png)').querySelector('a');
expect(a.textContent).toBe('https://example.com/a.png');
});
test('an image from another site inside a link becomes the link text, not a nested link', () => {
// The badge pattern common in READMEs.
const div = renderToDOM('[![build status](https://ci.example/badge.svg)](https://ci.example/project)');
expect(div.querySelector('img')).toBeNull();
const links = div.querySelectorAll('a');
expect(links).toHaveLength(1);
expect(links[0].getAttribute('href')).toBe('https://ci.example/project');
expect(links[0].textContent).toBe('build status');
});
test('media and embedded data images still load', () => {
const [media, data] = renderToDOM('![a](/media/x/a.png)\n\n![b](data:image/png;base64,AAAA)')
.querySelectorAll('img');
expect(media.getAttribute('src')).toBe('/media/x/a.png');
expect(data.getAttribute('src')).toBe('data:image/png;base64,AAAA');
});
test('external links open in a new tab without an opener', () => {
const a = renderToDOM('<https://example.com/page>').querySelector('a');
expect(a.getAttribute('href')).toBe('https://example.com/page');
expect(a.getAttribute('target')).toBe('_blank');
expect(a.getAttribute('rel')).toBe('noopener noreferrer');
});
test('absolute and fragment URLs are left alone', () => {
const [root, fragment] = renderToDOM('[root](/files)\n\n[top](#top)').querySelectorAll('a');
expect(root.getAttribute('href')).toBe('/files');
expect(fragment.getAttribute('href')).toBe('#top');
expect(fragment.hasAttribute('target')).toBe(false);
});
});
describe('MarkdownPreview', () => {
const SOURCE = '# Water storage\n\nRotate *every* six months.\n';
beforeEach(() => {
window.localStorage.clear();
global.fetch = jest.fn(() => Promise.resolve({
ok: true,
text: () => Promise.resolve(SOURCE),
}));
});
afterEach(() => {
delete global.fetch;
});
test('fetches the file and shows it formatted', async () => {
renderUI(<MarkdownPreview url={FILE_URL}/>);
const heading = await screen.findByRole('heading', {name: 'Water storage'});
expect(heading.tagName).toBe('H1');
expect(global.fetch).toHaveBeenCalledWith(FILE_URL, expect.anything());
});
test('Raw shows the unformatted source, and the choice is remembered', async () => {
const {unmount} = renderUI(<MarkdownPreview url={FILE_URL}/>);
await screen.findByRole('heading', {name: 'Water storage'});
fireEvent.click(screen.getByRole('button', {name: 'Raw'}));
expect(screen.queryByRole('heading')).toBeNull();
expect(screen.getByTestId('markdown-raw').textContent).toBe(SOURCE);
// The next preview opens in the mode the reader chose last.
unmount();
renderUI(<MarkdownPreview url={FILE_URL}/>);
await waitFor(() => expect(screen.getByTestId('markdown-raw').textContent).toBe(SOURCE));
expect(screen.queryByRole('heading')).toBeNull();
fireEvent.click(screen.getByRole('button', {name: 'Formatted'}));
expect(screen.getByRole('heading', {name: 'Water storage'})).toBeTruthy();
});
test('a failed fetch shows an error instead of loading forever', async () => {
global.fetch = jest.fn(() => Promise.resolve({ok: false, status: 404, statusText: 'Not Found'}));
renderUI(<MarkdownPreview url={FILE_URL}/>);
expect(await screen.findByText(/404 Not Found/)).toBeTruthy();
});
});
describe('isMarkdownFile', () => {
test.each([
['text/markdown', 'notes/a.md', true],
// A stored type from before text/markdown existed; refresh does not retag unchanged files.
['text/plain', 'notes/a.md', true],
['text/plain', 'notes/a.markdown', true],
['text/plain', 'notes/a.txt', false],
// A suffix alone is not enough; a binary file named .md is not markdown.
['application/octet-stream', 'notes/a.md', false],
])('%s %s -> %s', (mimetype, lowerPath, expected) => {
expect(isMarkdownFile(mimetype, lowerPath)).toBe(expected);
});
});

View file

@ -21,6 +21,7 @@ import {HELP_VIEWER_URI, NAME, navColorHexMap} from "./Vars";
import {useOverflowNav} from "../hooks/useOverflowNav";
import {defaultNavColor, navBarStyle, useNavColors} from "../themes/navColors";
import _ from "lodash";
import {BookmarksMenuContents, EDIT_BOOKMARKS_PATH} from "./Bookmarks";
function updateFavicon(colorName) {
// Fall back to violet if invalid color
@ -60,17 +61,23 @@ const allLinks = [
{text: 'Files', to: '/files', key: 'files'},
{text: 'Playlists', to: '/playlists', key: 'playlists'},
{text: 'Zim', to: '/zim', key: 'zim'},
{text: 'Repos', to: '/repos', key: 'repos'},
{text: 'Inventory', to: '/inventory', key: 'inventory'},
{text: 'AI', to: '/ai', key: 'ai'},
// No `end` on these two: a tab stays current for the pages beneath it, and `end` would
// drop the mark the moment either grows a sub-page.
{to: '/more/calculators', text: 'Calculators', key: 'calculators'},
// The user's bookmarks, read from BookmarksContext when rendered. No `to`: it is a
// menu, and its items are not sections of the app.
{text: 'Bookmarks', key: 'bookmarks', bookmarks: true},
{text: 'Flasher', to: '/flasher', key: 'flasher'},
{to: '/more/statistics', text: 'Statistics', key: 'statistics'},
];
function MenuLink({link}) {
// A top-level tab rendered directly in the bar (not inside a dropdown).
if (link.bookmarks) {
return <BookmarksDropdown link={link}/>
}
if (link.links) {
return <DropdownLinks link={link}/>
}
@ -86,13 +93,31 @@ function MenuLink({link}) {
</NavLink>
}
function DropdownMenuItem({link}) {
function DropdownMenuItem({link, nested = true}) {
// An item inside a Menu.Dropdown: the mobile hamburger menu, or the desktop
// "More" overflow menu.
// "More" overflow menu. `nested` allows flyout submenus, which the hamburger
// declines: a flyout opens on hover, and a phone has none.
if (link.bookmarks) {
if (nested) {
return <Menu.Sub>
<Menu.Sub.Target>
<Menu.Sub.Item>{link.text}</Menu.Sub.Item>
</Menu.Sub.Target>
<Menu.Sub.Dropdown>
<BookmarksMenuContents nested/>
</Menu.Sub.Dropdown>
</Menu.Sub>
}
return <React.Fragment>
<Menu.Divider/>
<Menu.Label>{link.text}</Menu.Label>
<BookmarksMenuContents nested={false}/>
<Menu.Divider/>
</React.Fragment>
}
if (link.links) {
// Mantine's Menu does not nest submenus without extra plumbing; a labelled,
// indented group stands in for one. No current data uses this path (no
// allLinks entry has its own `links`).
// A labelled, indented group stands in for a submenu here. No current data uses
// this path (no allLinks entry has its own `links`).
return <React.Fragment>
<Menu.Label>{link.text}</Menu.Label>
{link.links.map(l => <DropdownMenuItem key={l.key} link={l}/>)}
@ -142,6 +167,11 @@ NavDropdownTrigger.displayName = 'NavDropdownTrigger';
* the hooks that would answer it cannot be called in a loop.
*/
export const isLinkActive = (pathname, link) => {
// The Bookmarks menu owns only its editor page: a bookmark pointing at /videos does
// not make the menu own the Videos section.
if (link.bookmarks) {
return pathname === EDIT_BOOKMARKS_PATH || pathname.startsWith(`${EDIT_BOOKMARKS_PATH}/`);
}
// An off-site link (Help) is never the current section.
if (!link.to || /^[a-z]+:\/\//i.test(link.to)) return false;
const to = link.to.length > 1 ? link.to.replace(/\/+$/, '') : link.to;
@ -176,6 +206,24 @@ function DropdownLinks({link}) {
</Menu>
}
/**
* The Bookmarks tab: a dropdown that opens on hover as well as on click, holding the
* user's bookmarks and the link to edit them. Marked current on the editor page only
* (see isLinkActive).
*/
function BookmarksDropdown({link}) {
const {pathname} = useLocation();
const holdsCurrent = isLinkActive(pathname, link);
return <Menu position='bottom-start' withinPortal trigger='click-hover' openDelay={100} closeDelay={200}>
<Menu.Target>
<NavDropdownTrigger text={link.text} className={holdsCurrent ? 'active' : undefined}/>
</Menu.Target>
<Menu.Dropdown>
<BookmarksMenuContents nested/>
</Menu.Dropdown>
</Menu>
}
function MobileMenu({links}) {
// The mobile hamburger menu: an icon-only trigger holding every link.
return <Menu position='bottom-end' withinPortal>
@ -183,7 +231,7 @@ function MobileMenu({links}) {
<IconButton icon={IconMenu2} label='Menu' variant='subtle'/>
</Menu.Target>
<Menu.Dropdown>
{links.map(link => <DropdownMenuItem key={link.key} link={link}/>)}
{links.map(link => <DropdownMenuItem key={link.key} link={link} nested={false}/>)}
</Menu.Dropdown>
</Menu>
}

View file

@ -231,3 +231,91 @@ describe('NavBar current tab', () => {
expect(ruleBody('\\.mantine-Menu-item\\.active')).toMatch(/background:|font-weight:/);
});
});
describe('NavBar bookmarks', () => {
const {BookmarksContext} = require('../contexts/BookmarksContext');
const bookmarks = [
{id: 1, name: 'Channels', url: '/videos/channel', new_tab: false},
{id: 2, name: 'Services', children: [{id: 3, name: 'Jellyfin', url: ':8096/', new_tab: true}]},
];
const renderWithBookmarks = (route, nodes) => render(<NavBar/>, {
route,
withMedia: true,
contexts: [
[FileWorkerStatusContext, fileWorkerStatus],
[BookmarksContext, {bookmarks: nodes, error: null, refresh: jest.fn()}],
],
});
it('sits between Calculators and Flasher', () => {
renderNav('/');
const texts = [...document.querySelectorAll('#global_navbar .wrolpi-navbar-link')]
.map(el => el.textContent.trim());
const at = texts.indexOf('Bookmarks');
expect(at).toBeGreaterThan(0);
expect(texts[at - 1]).toBe('Calculators');
expect(texts[at + 1]).toBe('Flasher');
});
it('opens on click with the bookmarks, a directory flyout, and the edit link', async () => {
useRealRects();
renderWithBookmarks('/', bookmarks);
await userEvent.click(screen.getByRole('button', {name: /Bookmarks/}));
const channels = await screen.findByRole('menuitem', {name: 'Channels'});
expect(channels).toHaveAttribute('href', '/videos/channel');
expect(channels).not.toHaveAttribute('target');
// A directory is a submenu; its bookmarks appear once it is hovered.
expect(screen.queryByRole('menuitem', {name: 'Jellyfin'})).not.toBeInTheDocument();
await userEvent.hover(screen.getByRole('menuitem', {name: 'Services'}));
// `hidden`: the flyout is placed by floating-ui, which jsdom cannot lay out, so it
// stays hidden to accessibility queries even though it has rendered.
const jellyfin = await screen.findByRole('menuitem', {name: 'Jellyfin', hidden: true});
expect(jellyfin).toHaveAttribute('href', `${window.location.protocol}//${window.location.hostname}:8096/`);
expect(jellyfin).toHaveAttribute('target', '_blank');
expect(screen.getByRole('menuitem', {name: 'Edit bookmarks', hidden: true})).toHaveAttribute('href', '/more/bookmarks');
});
it('offers the edit link even with no bookmarks, so the feature can be found', async () => {
useRealRects();
renderWithBookmarks('/', []);
await userEvent.click(screen.getByRole('button', {name: /Bookmarks/}));
expect(await screen.findByRole('menuitem', {name: 'Edit bookmarks'})).toBeInTheDocument();
expect(screen.getAllByRole('menuitem')).toHaveLength(1);
});
it('is marked current on the editor page, and never for a bookmark pointing at a section', () => {
renderWithBookmarks('/more/bookmarks', bookmarks);
expect(screen.getByRole('button', {name: /Bookmarks/})).toHaveClass('active');
expect(currentTab()).toEqual([]);
});
it('marks More when the editor page is folded inside it', () => {
useBarWidth(200);
renderWithBookmarks('/more/bookmarks', bookmarks);
expect(screen.getByRole('button', {name: /More/})).toHaveClass('active');
});
it('does not own the section a bookmark points at', () => {
renderWithBookmarks('/videos/channel', bookmarks);
expect(screen.getByRole('button', {name: /Bookmarks/})).not.toHaveClass('active');
expect(currentTab()).toEqual(['Videos']);
});
it('lists bookmarks as a labelled group in the mobile menu', async () => {
useViewport('mobile');
useRealRects();
renderWithBookmarks('/', bookmarks);
await userEvent.click(screen.getByRole('button', {name: 'Menu'}));
expect(await screen.findByRole('menuitem', {name: 'Channels'})).toBeInTheDocument();
// No flyout on a phone: the directory's bookmarks are listed directly.
expect(screen.getByRole('menuitem', {name: 'Jellyfin'})).toBeInTheDocument();
expect(screen.getByText('Services')).toBeInTheDocument();
expect(screen.getByRole('menuitem', {name: 'Edit bookmarks'})).toBeInTheDocument();
});
});

View file

@ -0,0 +1,174 @@
import React from 'react';
import {act, render, screen} from '../test-utils';
import {OtherSearchView} from './Search';
import {searchChannels, searchCollections, searchRepos} from '../api';
// What the search page's Other tab shows for each state of useSearchChannels. "No Channels"
// must only appear once the server has confirmed there are none.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
searchChannels: jest.fn(),
searchCollections: jest.fn(),
searchRepos: jest.fn(),
}));
const NO_CHANNELS = /No Channels/;
beforeEach(() => {
searchChannels.mockReset();
searchRepos.mockReset();
searchRepos.mockResolvedValue({repos: [], total: 0});
searchCollections.mockReset();
searchCollections.mockResolvedValue({collections: []});
});
test('pending: shows a loader, not "No Channels"', () => {
searchChannels.mockReturnValue(new Promise(() => {
}));
const {container} = render(<OtherSearchView loading={false}/>);
expect(container.querySelector('.wrolpi-loading')).toBeInTheDocument();
expect(screen.queryByText(NO_CHANNELS)).not.toBeInTheDocument();
});
test('failed: shows an error, not "No Channels"', async () => {
searchChannels.mockResolvedValue(undefined);
const spy = jest.spyOn(console, 'error').mockImplementation(() => {
});
try {
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText(/Could not fetch the channels/)).toBeInTheDocument();
expect(screen.queryByText(NO_CHANNELS)).not.toBeInTheDocument();
} finally {
spy.mockRestore();
}
});
test('empty: shows "No Channels"', async () => {
searchChannels.mockResolvedValue({channels: []});
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText(NO_CHANNELS)).toBeInTheDocument();
});
test('loaded: lists the channels', async () => {
searchChannels.mockResolvedValue({channels: [{id: 1, name: 'Cooking'}]});
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText('Cooking')).toBeInTheDocument();
expect(screen.queryByText(NO_CHANNELS)).not.toBeInTheDocument();
});
describe('Repos', () => {
test('pending: the Repos section loads while Channels are shown', async () => {
searchChannels.mockResolvedValue({channels: []});
searchRepos.mockReturnValue(new Promise(() => {
}));
const {container} = render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText(NO_CHANNELS)).toBeInTheDocument();
expect(container.querySelector('.wrolpi-loading')).toBeInTheDocument();
expect(screen.queryByText('No Repos')).not.toBeInTheDocument();
});
test('empty: shows "No Repos"', async () => {
searchChannels.mockResolvedValue({channels: []});
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText('No Repos')).toBeInTheDocument();
});
test('failed: shows an error only in the Repos section', async () => {
searchChannels.mockResolvedValue({channels: [{id: 1, name: 'Cooking'}]});
searchRepos.mockRejectedValue(new Error('boom'));
const spy = jest.spyOn(console, 'error').mockImplementation(() => {
});
try {
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText('Cooking')).toBeInTheDocument();
expect(screen.getByText(/Could not fetch the repos/)).toBeInTheDocument();
} finally {
spy.mockRestore();
}
});
test('loaded: lists repos by the search, with their README headline', async () => {
searchChannels.mockResolvedValue({channels: []});
searchRepos.mockResolvedValue({
total: 1, repos: [{
id: 7, name: 'kiwix-tools', tag_name: null,
// The server escapes the README; only the highlight is markup.
readme_headline: 'Serve <b>Zim</b> files &lt;script&gt;',
}],
});
const searchParams = new URLSearchParams('q=zim&tag=Software');
const {container} = render(<OtherSearchView loading={false}/>, {query: {searchParams}});
await act(async () => {
});
expect(searchRepos).toHaveBeenCalledWith('zim', ['Software']);
expect(screen.getByText('kiwix-tools').closest('a')).toHaveAttribute('href', '/repos/7');
expect(container.querySelector('u').textContent).toBe('Zim');
expect(container.querySelector('script')).toBeNull();
expect(screen.getByText(/files <script>/)).toBeInTheDocument();
});
});
// Domains and Playlists are Collections searched by kind.
describe.each([
['domain', 'Domains', 'example.com', '/archives?domain=example.com'],
['playlist', 'Playlists', 'Road Trip', '/playlists/3'],
])('%s Collections', (kind, title, name, href) => {
const collectionsOf = (wanted, collections) => (k) => Promise.resolve({collections: k === wanted ? collections : []});
test(`empty: shows "No ${title}"`, async () => {
searchChannels.mockResolvedValue({channels: []});
render(<OtherSearchView loading={false}/>, {query: {searchParams: new URLSearchParams('q=nothing')}});
await act(async () => {
});
expect(screen.getByText(`No ${title}`)).toBeInTheDocument();
});
test('nothing to search by: shows none without fetching', async () => {
searchChannels.mockResolvedValue({channels: []});
render(<OtherSearchView loading={false}/>);
await act(async () => {
});
expect(screen.getByText(`No ${title}`)).toBeInTheDocument();
expect(searchCollections).not.toHaveBeenCalled();
});
test(`failed: shows an error only in the ${title} section`, async () => {
searchChannels.mockResolvedValue({channels: [{id: 1, name: 'Cooking'}]});
searchCollections.mockImplementation(k => k === kind ? Promise.reject(new Error('boom')) : Promise.resolve({collections: []}));
const spy = jest.spyOn(console, 'error').mockImplementation(() => {
});
try {
render(<OtherSearchView loading={false}/>, {query: {searchParams: new URLSearchParams('q=x')}});
await act(async () => {
});
expect(screen.getByText('Cooking')).toBeInTheDocument();
expect(screen.getByText(new RegExp(`Could not fetch the ${title.toLowerCase()}`))).toBeInTheDocument();
} finally {
spy.mockRestore();
}
});
test('loaded: lists them by the search and tags, linked', async () => {
searchChannels.mockResolvedValue({channels: []});
searchCollections.mockImplementation(collectionsOf(kind, [{id: 3, name, kind, tag_name: null}]));
const searchParams = new URLSearchParams('q=road&tag=News');
render(<OtherSearchView loading={false}/>, {query: {searchParams}});
await act(async () => {
});
expect(searchCollections).toHaveBeenCalledWith(kind, ['News'], 'road');
expect(screen.getByText(name).closest('a')).toHaveAttribute('href', href);
});
});

1062
app/src/components/Repos.js Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,394 @@
import React from 'react';
import {Route, Routes} from 'react-router';
import {act, fireEvent, render, screen, waitFor, within} from '../test-utils';
import {
RepoEditPage,
RepoImportPage,
repoNameFromURL,
RepoNewPage,
ReposPage,
RepoViewPage,
safeRepoHref,
} from './Repos';
import {
createRepo, deleteRepo, fetchRepos, getFile, getRepo, getRepoLog, getRepoTree, importRepo, inspectRepoImport,
updateRepo,
} from '../api';
import {FilePreviewContext} from './FilePreview';
jest.mock('../api', () => ({
...jest.requireActual('../api'),
createRepo: jest.fn(),
deleteRepo: jest.fn(),
fetchRepos: jest.fn(),
getFile: jest.fn(),
getRepo: jest.fn(),
getRepoLog: jest.fn(),
getRepoTree: jest.fn(),
importRepo: jest.fn(),
inspectRepoImport: jest.fn(),
updateRepo: jest.fn(),
}));
// The directory picker searches the server; picking a directory is all these specs need.
jest.mock('./Common', () => ({
...jest.requireActual('./Common'),
DirectorySearch: ({onSelect}) => <button type='button' onClick={() => onSelect('code/kiwix-tools')}>
Pick directory
</button>,
}));
// The README is fetched from the media server; its rendering is MarkdownPreview's own spec.
jest.mock('./MarkdownPreview', () => ({
...jest.requireActual('./MarkdownPreview'),
MarkdownPreview: ({url}) => <div data-testid='markdown-preview'>{url}</div>,
}));
const repoFixture = (overrides = {}) => ({
id: 1, collection_id: 10, name: 'kiwix-tools', tag_name: null, description: null,
url: 'https://github.com/kiwix/kiwix-tools', mode: 'full', branch: null, default_branch: 'main',
head_sha: '3f9c2a1e7b', head_message: 'Fix the mimetype', head_date: '2026-09-30T14:02:00Z',
size: 1000, directory: 'repos/kiwix-tools', frequency: 604800, download_error: null,
...overrides,
});
const treeFixture = {
path: '',
readme_path: 'README.md',
entries: [
{name: 'src', path: 'src', is_dir: true, size: null},
{name: 'README.md', path: 'README.md', is_dir: false, size: 120},
],
};
const flush = () => act(async () => {
});
beforeEach(() => jest.clearAllMocks());
describe('safeRepoHref', () => {
it('only links https URLs', () => {
expect(safeRepoHref('https://github.com/a/b')).toBe('https://github.com/a/b');
expect(safeRepoHref('javascript:alert(1)')).toBeNull();
expect(safeRepoHref('http://github.com/a/b')).toBeNull();
expect(safeRepoHref('file:///etc/passwd')).toBeNull();
expect(safeRepoHref('not a url')).toBeNull();
});
});
describe('repoNameFromURL', () => {
it('is the last path segment, without .git or a branch', () => {
expect(repoNameFromURL('https://github.com/kiwix/kiwix-tools.git')).toBe('kiwix-tools');
expect(repoNameFromURL('https://github.com/kiwix/kiwix-tools/tree/dev')).toBe('kiwix-tools');
expect(repoNameFromURL('https://gitlab.com/group/sub/project/')).toBe('project');
expect(repoNameFromURL('nope')).toBe('');
});
});
describe('ReposPage', () => {
it('lists repos, linking each to its page', async () => {
fetchRepos.mockResolvedValue([
repoFixture(),
repoFixture({id: 2, name: 'gone', download_error: 'git ls-remote failed', head_sha: null}),
]);
render(<ReposPage/>, {route: '/repos'});
await flush();
expect(screen.getAllByText('kiwix-tools')[0].closest('a')).toHaveAttribute('href', '/repos/1');
expect(screen.getAllByText('gone')[0].closest('a')).toHaveAttribute('href', '/repos/2');
expect(screen.getByText('New Repo').closest('a')).toHaveAttribute('href', '/repos/new');
});
it('shows a repo which is never updated', async () => {
fetchRepos.mockResolvedValue([repoFixture({frequency: 0})]);
render(<ReposPage/>, {route: '/repos'});
await flush();
expect(screen.getAllByText('Never').length).toBeGreaterThan(0);
});
it('explains an empty list', async () => {
fetchRepos.mockResolvedValue([]);
render(<ReposPage/>, {route: '/repos'});
await flush();
expect(screen.getByText(/No repos yet/)).toBeInTheDocument();
});
});
function renderViewPage({repo, tree = treeFixture, query = {}, setPreviewFile = jest.fn()} = {}) {
getRepo.mockResolvedValue(repo);
getRepoTree.mockResolvedValue(tree);
return render(<Routes><Route path='/repos/:repoId' element={<RepoViewPage/>}/></Routes>, {
route: '/repos/1',
query,
contexts: [[FilePreviewContext, {setPreviewFile}]],
});
}
describe('RepoViewPage', () => {
it('shows the latest commit, the files, and the README', async () => {
renderViewPage({repo: repoFixture()});
await flush();
expect(getRepoTree).toHaveBeenCalledWith('1', '');
expect(screen.getByText('3f9c2a1')).toBeInTheDocument();
expect(screen.getByText('Fix the mimetype')).toBeInTheDocument();
expect(screen.getByText('main')).toBeInTheDocument();
expect(screen.getByText('(default)')).toBeInTheDocument();
expect(screen.getByText('https://github.com/kiwix/kiwix-tools').closest('a'))
.toHaveAttribute('href', 'https://github.com/kiwix/kiwix-tools');
expect(screen.getByText('src')).toBeInTheDocument();
expect(screen.getByTestId('markdown-preview')).toHaveTextContent('/media/repos/kiwix-tools/README.md');
});
it('opens a directory through the URL, and a file in the preview', async () => {
const updateQuery = jest.fn();
const setPreviewFile = jest.fn();
const file = {path: 'repos/kiwix-tools/README.md', mimetype: 'text/markdown', size: 120};
getFile.mockResolvedValue(file);
renderViewPage({repo: repoFixture(), query: {updateQuery}, setPreviewFile});
await flush();
fireEvent.click(screen.getByText('src'));
expect(updateQuery).toHaveBeenCalledWith({path: 'src'}, false);
fireEvent.click(screen.getAllByText('README.md')[0]);
await flush();
expect(getFile).toHaveBeenCalledWith('repos/kiwix-tools/README.md', true);
expect(setPreviewFile).toHaveBeenCalledWith(file);
});
it('lists the directory in the URL, with a way back up', async () => {
const tree = {path: 'src/lib', readme_path: null, entries: [{name: 'a.c', path: 'src/lib/a.c', is_dir: false}]};
const updateQuery = jest.fn();
renderViewPage({repo: repoFixture(), tree, query: {searchParams: new URLSearchParams('path=src/lib'), updateQuery}});
await flush();
expect(getRepoTree).toHaveBeenCalledWith('1', 'src/lib');
expect(screen.queryByTestId('markdown-preview')).not.toBeInTheDocument();
fireEvent.click(screen.getByText('..'));
expect(updateQuery).toHaveBeenCalledWith({path: 'src'}, false);
// The breadcrumb's root.
fireEvent.click(screen.getAllByText('kiwix-tools').find(i => i.tagName === 'BUTTON'));
expect(updateQuery).toHaveBeenCalledWith({path: ''}, false);
});
it('offers a ZIP of the files', async () => {
renderViewPage({repo: repoFixture()});
await flush();
const link = screen.getByText('Download ZIP').closest('a');
expect(link).toHaveAttribute('href', expect.stringMatching(/\/api\/repos\/1\/archive\.zip$/));
expect(link).toHaveAttribute('download');
});
it('shows the history instead of the files', async () => {
getRepoLog.mockResolvedValue({
total: 51,
commits: [{sha: 'abcdef1234', author: 'Ada', date: '2026-09-30T14:02:00Z', message: 'Fix the mimetype'}],
});
const updateQuery = jest.fn();
renderViewPage({repo: repoFixture(), query: {searchParams: new URLSearchParams('view=history'), updateQuery}});
await flush();
expect(getRepoLog).toHaveBeenCalledWith(1, 0, 50);
expect(getRepoTree).not.toHaveBeenCalled();
expect(screen.getByText('51 commits')).toBeInTheDocument();
expect(screen.getByText('abcdef1')).toBeInTheDocument();
expect(screen.getByText('Ada')).toBeInTheDocument();
// The next page is appended.
getRepoLog.mockResolvedValue({
total: 51,
commits: [{sha: '0123456789', author: 'Ben', date: '2026-09-01T00:00:00Z', message: 'Older'}],
});
fireEvent.click(screen.getByText('Load more'));
await flush();
expect(getRepoLog).toHaveBeenLastCalledWith(1, 1, 50);
expect(screen.getByText('abcdef1')).toBeInTheDocument();
expect(screen.getByText('0123456')).toBeInTheDocument();
fireEvent.click(screen.getByText('Files'));
expect(updateQuery).toHaveBeenCalledWith({view: null}, false);
});
it('explains a repo that is not downloaded yet', async () => {
renderViewPage({repo: repoFixture({head_sha: null, head_message: null, head_date: null})});
await flush();
expect(screen.getByText('Not downloaded yet')).toBeInTheDocument();
expect(getRepoTree).not.toHaveBeenCalled();
expect(screen.queryByText('Download ZIP')).not.toBeInTheDocument();
expect(screen.queryByText('History')).not.toBeInTheDocument();
});
it('keeps showing the files when the last update failed', async () => {
renderViewPage({repo: repoFixture({download_error: 'Traceback...\nDownloadError: git ls-remote failed (128)'})});
await flush();
expect(screen.getByText('The last update failed')).toBeInTheDocument();
expect(screen.getByText(/git ls-remote failed/)).toBeInTheDocument();
expect(screen.getByText('src')).toBeInTheDocument();
});
it('says a repo which is never updated is only updated on request', async () => {
renderViewPage({repo: repoFixture({frequency: 0, download_error: 'git ls-remote failed (128)'})});
await flush();
expect(screen.getByText('The last update failed')).toBeInTheDocument();
expect(screen.getByText(/never updated automatically/)).toBeInTheDocument();
expect(screen.queryByText(/will be tried again/)).not.toBeInTheDocument();
});
it('never links an upstream that is not https', async () => {
renderViewPage({repo: repoFixture({url: 'javascript:alert(1)'})});
await flush();
expect(screen.queryByText('javascript:alert(1)')).not.toBeInTheDocument();
});
});
describe('RepoNewPage', () => {
it('adds a repo, then opens it', async () => {
createRepo.mockResolvedValue(repoFixture({id: 5}));
render(<Routes>
<Route path='/repos/new' element={<RepoNewPage/>}/>
<Route path='/repos/:repoId' element={<div>repo page</div>}/>
</Routes>, {route: '/repos/new'});
const add = screen.getByText('Add Repo').closest('button');
expect(add).toBeDisabled();
fireEvent.change(screen.getByLabelText(/URL/), {target: {value: 'git@github.com:a/b'}});
expect(add).toBeDisabled();
expect(screen.getByText('Must be an https:// URL')).toBeInTheDocument();
fireEvent.change(screen.getByLabelText(/URL/), {target: {value: 'https://github.com/kiwix/kiwix-tools'}});
expect(screen.getByPlaceholderText('kiwix-tools')).toBeInTheDocument();
fireEvent.change(screen.getByLabelText('Branch'), {target: {value: ' dev '}});
fireEvent.click(screen.getByText('Pick directory'));
fireEvent.click(add);
await flush();
expect(createRepo).toHaveBeenCalledWith({
url: 'https://github.com/kiwix/kiwix-tools', name: null, tag_name: null, frequency: 604800,
mode: 'full', branch: 'dev', submodules: false, description: null, directory: 'code/kiwix-tools',
});
expect(screen.getByText('repo page')).toBeInTheDocument();
});
});
describe('RepoEditPage', () => {
function renderEditPage(repo) {
getRepo.mockResolvedValue(repo);
return render(<Routes>
<Route path='/repos/:repoId/edit' element={<RepoEditPage/>}/>
<Route path='/repos' element={<div>repos page</div>}/>
</Routes>, {route: '/repos/1/edit'});
}
it('saves the settings', async () => {
updateRepo.mockResolvedValue(repoFixture());
renderEditPage(repoFixture({branch: 'dev', description: 'Tools'}));
await flush();
expect(screen.getByLabelText('Branch')).toHaveValue('dev');
expect(screen.getByLabelText('Include submodules')).not.toBeChecked();
fireEvent.change(screen.getByLabelText('Branch'), {target: {value: ''}});
fireEvent.click(screen.getByLabelText('Include submodules'));
fireEvent.click(screen.getByText('Save'));
await flush();
expect(updateRepo).toHaveBeenCalledWith('1', {
description: 'Tools', frequency: 604800, mode: 'full', branch: '', submodules: true,
});
});
it('keeps a repo which is never updated', async () => {
updateRepo.mockResolvedValue(repoFixture({frequency: 0}));
renderEditPage(repoFixture({frequency: 0}));
await flush();
expect(screen.getByDisplayValue('Never')).toBeInTheDocument();
fireEvent.click(screen.getByText('Save'));
await flush();
expect(updateRepo).toHaveBeenCalledWith('1', expect.objectContaining({frequency: 0}));
});
it('deletes the repo, and its files only when asked', async () => {
deleteRepo.mockResolvedValue(undefined);
renderEditPage(repoFixture());
await flush();
fireEvent.click(screen.getByText('Delete'));
const dialog = await screen.findByRole('dialog');
fireEvent.click(within(dialog).getByLabelText(/Also delete its files/));
fireEvent.click(within(dialog).getByText('Delete'));
await flush();
expect(deleteRepo).toHaveBeenCalledWith('1', true);
expect(screen.getByText('repos page')).toBeInTheDocument();
});
});
describe('RepoImportPage', () => {
const inspection = (overrides = {}) => ({
directory: 'code/kiwix-tools', origin: 'git@github.com:kiwix/kiwix-tools.git',
url: 'https://github.com/kiwix/kiwix-tools', name: 'kiwix-tools', branch: 'main',
head_sha: 'abcdef1234', head_message: 'Local work', head_date: '2026-09-30T14:02:00Z', local_commits: 2,
shallow: false, ignored: false,
...overrides,
});
function renderImportPage() {
return render(<Routes>
<Route path='/repos/import' element={<RepoImportPage/>}/>
<Route path='/repos/:repoId' element={<div>repo page</div>}/>
</Routes>, {route: '/repos/import'});
}
it('inspects the directory, warns about local commits, and imports once confirmed', async () => {
inspectRepoImport.mockResolvedValue({inspection: inspection()});
importRepo.mockResolvedValue(repoFixture({id: 9}));
renderImportPage();
fireEvent.click(screen.getByText('Pick directory'));
expect(await screen.findByText('2 commits are not on the remote')).toBeInTheDocument();
expect(inspectRepoImport).toHaveBeenCalledWith('code/kiwix-tools', '', null, '');
// Inspected again with the URL the import will use.
await waitFor(() => expect(inspectRepoImport).toHaveBeenCalledWith('code/kiwix-tools', '', null,
'https://github.com/kiwix/kiwix-tools'));
expect(screen.getByText('git@github.com:kiwix/kiwix-tools.git')).toBeInTheDocument();
expect(screen.getByText(/Its files are indexed/)).toBeInTheDocument();
// The URL comes from the origin.
expect(screen.getByLabelText(/^URL/)).toHaveValue('https://github.com/kiwix/kiwix-tools');
const button = screen.getByText('Import').closest('button');
expect(button).toBeDisabled();
fireEvent.click(screen.getByLabelText(/I understand/));
expect(button).toBeEnabled();
fireEvent.click(button);
await flush();
expect(importRepo).toHaveBeenCalledWith({
directory: 'code/kiwix-tools', confirm: true, url: 'https://github.com/kiwix/kiwix-tools', name: null,
tag_name: null, frequency: 604800, mode: 'full', branch: null, submodules: false,
});
expect(screen.getByText('repo page')).toBeInTheDocument();
});
it('asks for the URL when the clone has no https origin', async () => {
inspectRepoImport.mockResolvedValue({inspection: inspection({origin: null, url: null, local_commits: 0})});
renderImportPage();
fireEvent.click(screen.getByText('Pick directory'));
expect(await screen.findByText(/no https:\/\/ origin/)).toBeInTheDocument();
fireEvent.click(screen.getByLabelText(/I understand/));
expect(screen.getByText('Import').closest('button')).toBeDisabled();
fireEvent.change(screen.getByLabelText(/^URL/), {target: {value: 'https://github.com/kiwix/kiwix-tools'}});
expect(screen.getByText('Import').closest('button')).toBeEnabled();
});
it('explains a directory which cannot be imported', async () => {
inspectRepoImport.mockResolvedValue({error: 'code/kiwix-tools is not a git clone'});
renderImportPage();
fireEvent.click(screen.getByText('Pick directory'));
expect(await screen.findByText('code/kiwix-tools is not a git clone')).toBeInTheDocument();
expect(screen.queryByText('Import')).not.toBeInTheDocument();
});
it('says when the files are not indexed', async () => {
inspectRepoImport.mockResolvedValue({inspection: inspection({ignored: true})});
renderImportPage();
fireEvent.click(screen.getByText('Pick directory'));
expect(await screen.findByText(/Its files are not indexed/)).toBeInTheDocument();
});
});

View file

@ -1,17 +1,27 @@
import React, {useState} from "react";
import {Link, Route, Routes, useLocation, useNavigate} from "react-router";
import {FilesSearchView, SearchViewButton} from "./Files";
import {useLatestRequest, usePages, useSearchChannels, useSearchDate, useSearchDeep, useSearchFilter} from "../hooks/customHooks";
import {
useLatestRequest,
usePages,
useSearchChannels,
useSearchDate,
useSearchDeep,
useSearchFilter,
useSearchCollections,
useSearchRepos,
} from "../hooks/customHooks";
import {ShortcutHint} from "./ShortcutHint";
import {ZimSearchView} from "./Zim";
import {MapSearchView} from "./MapSearchView";
import {searchEstimateFiles, searchEstimateMap, searchEstimateOthers, searchEstimateZims, searchSuggestions} from "../api";
import {filterToMimetypes, fuzzyMatch, normalizeEstimate, SearchResultsInput, TabLinks} from "./Common";
import {ErrorMessage, filterToMimetypes, fuzzyMatch, normalizeEstimate, SearchResultsInput, TabLinks} from "./Common";
import _ from "lodash";
import {TagsContext} from "../Tags";
import {Accordion, Header, Icon, Label, Loading, Panel, Stack} from "./ui";
import {QueryContext} from "../contexts/contexts";
import {KeyboardShortcutsContext} from "../contexts/KeyboardShortcutsContext";
import {HeadlineText} from "./Headline";
const SUGGESTED_APPS = [
{location: '/admin', title: 'Downloads', description: 'View and control your downloads'},
@ -131,12 +141,15 @@ export const useSearch = (defaultLimit = 48, totalPages = 0, emptySearch = false
}
export function useSuggestions(searchStr, tagNames, filter, anyTag) {
// The general groups are null until the suggestion request returns: an empty list means the
// server found none, and the dropdown may say "No results" only for that.
const defaultSuggestions = {
fileGroups: [],
channels: [],
domains: [],
authors: [],
subjects: [],
channels: null,
domains: null,
authors: null,
subjects: null,
repos: null,
zimsEstimates: [],
}
const [suggestions, setSuggestions] = React.useState(defaultSuggestions);
@ -161,7 +174,7 @@ export function useSuggestions(searchStr, tagNames, filter, anyTag) {
// The files estimate always uses deep search server-side, so users know results are possible.
sendFilesRequest(async () => await searchEstimateFiles(searchStr, tagNames, mimetypes, months, dateRange, anyTag));
sendZimRequest(async () => await searchEstimateZims(searchStr, tagNames));
sendOtherRequest(async () => await searchEstimateOthers(tagNames));
sendOtherRequest(async () => await searchEstimateOthers(tagNames, searchStr));
}
}, [
searchStr,
@ -183,6 +196,7 @@ export function useSuggestions(searchStr, tagNames, filter, anyTag) {
domains: generalData.domains,
authors: generalData.authors,
subjects: generalData.subjects,
repos: generalData.repos,
}
});
}
@ -310,6 +324,13 @@ export function useSearchSuggestions(defaultSearchStr, defaultTagNames, anyTag)
})
}
}
if (newSuggestions.repos && newSuggestions.repos.length > 0) {
results.repos = {
name: 'Repos', results: newSuggestions.repos.map(i => {
return {type: 'repo', title: i.name, id: i.id, location: `/repos/${i.id}`}
})
}
}
if (newSuggestions.authors && newSuggestions.authors.length > 0) {
results.authors = {
name: 'Authors', results: newSuggestions.authors.map(i => {
@ -401,10 +422,11 @@ export function useSearchSuggestions(defaultSearchStr, defaultTagNames, anyTag)
zims: zimSum,
otherSum: otherSum,
mapPlaces: newSuggestions.mapPlaces || 0,
channels: newSuggestions.channels.length,
domains: newSuggestions.domains.length,
channels: newSuggestions.channels?.length,
domains: newSuggestions.domains?.length,
authors: newSuggestions.authors?.length,
subjects: newSuggestions.subjects?.length,
repos: newSuggestions.repos?.length,
tags: matchingTags?.length,
apps: matchingApps?.length,
});
@ -541,31 +563,99 @@ function SearchChannelPreview({channel}) {
</div>
}
function OtherSearchView({loading}) {
const {searchParams} = React.useContext(QueryContext);
const [activeValue, setActiveValue] = React.useState('channels');
const activeTags = searchParams.getAll('tag');
const {channels, loading: channelsLoading} = useSearchChannels(activeTags);
function SearchCollectionPreview({collection, to}) {
const {SingleTag} = React.useContext(TagsContext);
return <div>
<span style={{display: 'inline-flex', alignItems: 'center', gap: 8, flexWrap: 'wrap'}}>
<Link to={to}>{collection.name}</Link>
{collection.tag_name && <SingleTag name={collection.tag_name}/>}
</span>
</div>
}
if (loading || channelsLoading) {
function SearchRepoPreview({repo}) {
const {SingleTag} = React.useContext(TagsContext);
return <div>
<span style={{display: 'inline-flex', alignItems: 'center', gap: 8, flexWrap: 'wrap'}}>
<Link to={`/repos/${repo.id}`}>{repo.name}</Link>
{repo.tag_name && <SingleTag name={repo.tag_name}/>}
</span>
{/* The server escapes the README; only its <b> highlights are markup. */}
{repo.readme_headline && <div><small><HeadlineText headline={repo.readme_headline}/></small></div>}
</div>
}
function OtherSearchSection({value, title, items, preview, emptyText, errorText}) {
let body;
if (items === null) {
body = <Loading/>;
} else if (items === undefined) {
body = <ErrorMessage>{errorText}</ErrorMessage>;
} else {
body = <Stack gap='xs'>{!_.isEmpty(items) ? items.map(preview) : emptyText}</Stack>;
}
return <Accordion.Item value={value}>
<Accordion.Control>
<span style={{display: 'inline-flex', alignItems: 'center', gap: 8}}>
<Header as='h3' style={{margin: 0}}>{title}</Header>
{Array.isArray(items) && <Label>{normalizeEstimate(items.length)}</Label>}
</span>
</Accordion.Control>
<Accordion.Panel>{body}</Accordion.Panel>
</Accordion.Item>
}
// Each section's items follow useSearchChannels, useSearchCollections, or useSearchRepos: null = pending,
// undefined = fetch failed, [] = none.
export function OtherSearchView({loading}) {
const {searchParams} = React.useContext(QueryContext);
const [activeValues, setActiveValues] = React.useState(['channels', 'domains', 'playlists', 'repos']);
const activeTags = searchParams.getAll('tag');
const searchStr = searchParams.get('q') || '';
const {channels, loading: channelsLoading} = useSearchChannels(activeTags, searchStr);
const {collections: domains} = useSearchCollections('domain', searchStr, activeTags);
const {collections: playlists} = useSearchCollections('playlist', searchStr, activeTags);
const {repos} = useSearchRepos(searchStr, activeTags);
if (loading || channelsLoading || channels === null) {
return <Panel><Loading/></Panel>
} else if (channels === undefined) {
return <Panel><ErrorMessage>Could not fetch the channels.</ErrorMessage></Panel>
}
return <Accordion value={activeValue} onChange={setActiveValue}>
<Accordion.Item value='channels'>
<Accordion.Control>
<span style={{display: 'inline-flex', alignItems: 'center', gap: 8}}>
<Header as='h3' style={{margin: 0}}>Channels</Header>
<Label>{normalizeEstimate(channels.length)}</Label>
</span>
</Accordion.Control>
<Accordion.Panel>
<Stack gap='xs'>
{!_.isEmpty(channels) ?
channels.map(i => <SearchChannelPreview key={i.id} channel={i}/>)
: 'No Channels'}
</Stack>
</Accordion.Panel>
</Accordion.Item>
return <Accordion multiple value={activeValues} onChange={setActiveValues}>
<OtherSearchSection
value='channels'
title='Channels'
items={channels}
preview={i => <SearchChannelPreview key={i.id} channel={i}/>}
emptyText='No Channels'
errorText='Could not fetch the channels.'
/>
<OtherSearchSection
value='domains'
title='Domains'
items={domains}
preview={i => <SearchCollectionPreview key={i.id} collection={i}
to={`/archives?domain=${encodeURIComponent(i.name)}`}/>}
emptyText='No Domains'
errorText='Could not fetch the domains.'
/>
<OtherSearchSection
value='playlists'
title='Playlists'
items={playlists}
preview={i => <SearchCollectionPreview key={i.id} collection={i} to={`/playlists/${i.id}`}/>}
emptyText='No Playlists'
errorText='Could not fetch the playlists.'
/>
<OtherSearchSection
value='repos'
title='Repos'
items={repos}
preview={i => <SearchRepoPreview key={i.id} repo={i}/>}
emptyText='No Repos'
errorText='Could not fetch the repos.'
/>
</Accordion>
}

View file

@ -0,0 +1,139 @@
import React from 'react';
import {act, render, screen} from '../test-utils';
import {useSearchSuggestions} from './Search';
import {TagsContext} from '../Tags';
import {tagsContextFixture} from '../test-fixtures';
import {
searchEstimateFiles, searchEstimateMap, searchEstimateOthers, searchEstimateZims, searchSuggestions,
} from '../api';
// The global search dropdown's groups are built from useSuggestions. A "Channels: No results" row
// may only appear once the suggestion request has come back empty, never before the user has
// typed, and never while the request is pending.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
searchSuggestions: jest.fn(),
searchEstimateFiles: jest.fn(),
searchEstimateMap: jest.fn(),
searchEstimateOthers: jest.fn(),
searchEstimateZims: jest.fn(),
}));
const DEBOUNCE = 500;
const pending = () => new Promise(() => {
});
let hook;
function Probe() {
hook = useSearchSuggestions('', [], false);
return <pre data-testid='results'>{JSON.stringify(hook.suggestionsResults)}</pre>;
}
const results = () => JSON.parse(screen.getByTestId('results').textContent);
const renderProbe = () => render(<Probe/>, {
contexts: [[TagsContext, tagsContextFixture({fuzzyMatchTagsByName: () => []})]],
});
beforeEach(() => {
jest.useFakeTimers();
searchEstimateFiles.mockReturnValue(pending());
searchEstimateMap.mockReturnValue(pending());
searchEstimateOthers.mockReturnValue(pending());
searchEstimateZims.mockReturnValue(pending());
});
afterEach(() => {
jest.useRealTimers();
jest.clearAllMocks();
});
test('no Channels group before anything has been searched', () => {
searchSuggestions.mockReturnValue(pending());
renderProbe();
expect(results().channels).toBeUndefined();
});
test('no Channels group while the suggestion request is pending', () => {
searchSuggestions.mockReturnValue(pending());
renderProbe();
act(() => {
hook.setSearchStr('foo');
});
expect(results().channels).toBeUndefined();
act(() => {
jest.advanceTimersByTime(DEBOUNCE);
});
expect(searchSuggestions).toHaveBeenCalledWith('foo');
expect(results().channels).toBeUndefined();
expect(hook.loading).toBe(true);
});
test('a Channels "No results" row once the server said there are none', async () => {
let resolve;
searchSuggestions.mockReturnValue(new Promise(res => resolve = res));
renderProbe();
act(() => {
hook.setSearchStr('foo');
});
act(() => {
jest.advanceTimersByTime(DEBOUNCE);
});
await act(async () => {
resolve({channels: [], domains: [], authors: [], subjects: []});
});
expect(results().channels.results).toEqual([{title: 'No results'}]);
});
test('Channels rows once the server found some', async () => {
let resolve;
searchSuggestions.mockReturnValue(new Promise(res => resolve = res));
renderProbe();
act(() => {
hook.setSearchStr('foo');
});
act(() => {
jest.advanceTimersByTime(DEBOUNCE);
});
await act(async () => {
resolve({channels: [{id: 3, name: 'Food'}], domains: [], authors: [], subjects: []});
});
expect(results().channels.results.map(i => i.title)).toEqual(['Food']);
});
test('Repos rows link to each repo, and the Other estimate gets the search', async () => {
let resolve;
searchSuggestions.mockReturnValue(new Promise(res => resolve = res));
renderProbe();
act(() => {
hook.setSearchStr('kiwix');
});
act(() => {
jest.advanceTimersByTime(DEBOUNCE);
});
await act(async () => {
resolve({channels: [], domains: [], authors: [], subjects: [], repos: [{id: 4, name: 'kiwix-tools'}]});
});
expect(results().repos.results).toEqual([{type: 'repo', title: 'kiwix-tools', id: 4, location: '/repos/4'}]);
// Repos count toward the Other tab by their name and README, not only by tag.
expect(searchEstimateOthers).toHaveBeenCalledWith([], 'kiwix');
});
test('no Repos group when none match', async () => {
let resolve;
searchSuggestions.mockReturnValue(new Promise(res => resolve = res));
renderProbe();
act(() => {
hook.setSearchStr('foo');
});
act(() => {
jest.advanceTimersByTime(DEBOUNCE);
});
await act(async () => {
resolve({channels: [], domains: [], authors: [], subjects: [], repos: []});
});
expect(results().repos).toBeUndefined();
});

View file

@ -0,0 +1,49 @@
import React from 'react';
import {act, render, screen} from '../test-utils';
import {StatisticsPage} from './Apps';
import {getStatistics} from '../api';
// What the Statistics page shows while the numbers are still loading, when the fetch failed, and
// once they arrive. "Failed to fetch statistics" may only appear for a failed fetch.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getStatistics: jest.fn(),
}));
const FAILED = /Failed to fetch statistics/;
const STATS = {
global_statistics: {db_size: 1024},
file_statistics: {
archive_count: 1, audio_count: 2, ebook_count: 3, image_count: 4, pdf_count: 5, total_count: 21,
video_count: 6, zip_count: 0, total_size: 4096, tagged_files: 7, tagged_zims: 0, tags_count: 2,
},
};
beforeEach(() => getStatistics.mockReset());
test('pending: shows a loader, not the error', () => {
getStatistics.mockReturnValue(new Promise(() => {
}));
const {container} = render(<StatisticsPage/>);
expect(container.querySelector('.wrolpi-loading')).toBeInTheDocument();
expect(screen.queryByText(FAILED)).not.toBeInTheDocument();
});
test('failed: shows the error', async () => {
// The api helper toasts and returns undefined on a non-OK response.
getStatistics.mockResolvedValue(undefined);
render(<StatisticsPage/>);
await act(async () => {
});
expect(screen.getByText(FAILED)).toBeInTheDocument();
});
test('loaded: shows the numbers', async () => {
getStatistics.mockResolvedValue(STATS);
render(<StatisticsPage/>);
await act(async () => {
});
expect(screen.getByText('All Files')).toBeInTheDocument();
expect(screen.queryByText(FAILED)).not.toBeInTheDocument();
});

View file

@ -0,0 +1,69 @@
import React from 'react';
import {render, screen} from '../test-utils';
import {DisableDownloadsToggle, HotspotToggle} from './Common';
import {SettingsPage, UpgradeSegment} from './admin/Settings';
// The status and settings contexts start empty and are filled by the first poll. Until then a
// consumer must not claim a negative ("Downloading Disabled", "not supported", "up to date") or
// render a form over blank values; `loaded` on each context says whether the first poll landed.
jest.mock('../api', () => ({
...jest.requireActual('../api'),
getConfigs: jest.fn(() => new Promise(() => {
})),
}));
const unloadedStatus = {status: {}, loaded: false};
describe('DisableDownloadsToggle before status has loaded', () => {
test('is disabled and does not claim downloading is disabled', () => {
render(<DisableDownloadsToggle/>, {status: unloadedStatus});
expect(screen.getByTestId('toggle')).toBeDisabled();
expect(screen.queryByText('Downloading Disabled')).not.toBeInTheDocument();
});
test('once loaded, reads the downloads state', () => {
render(<DisableDownloadsToggle/>, {status: {status: {downloads: {disabled: true, stopped: false}}, loaded: true}});
expect(screen.getByText('Downloading Disabled')).toBeInTheDocument();
expect(screen.getByTestId('toggle')).toBeEnabled();
});
});
describe('HotspotToggle before status has loaded', () => {
test('is disabled without claiming the hotspot is unsupported', () => {
render(<HotspotToggle/>, {status: unloadedStatus});
expect(screen.getByTestId('toggle')).toBeDisabled();
expect(screen.queryByTestId('subsystem-popup')).not.toBeInTheDocument();
});
test('once loaded without a hotspot field, is unsupported', () => {
render(<HotspotToggle/>, {status: {status: {}, loaded: true}});
expect(screen.getByTestId('toggle')).toBeDisabled();
expect(screen.getByTestId('subsystem-popup')).toBeInTheDocument();
});
});
describe('UpgradeSegment before status has loaded', () => {
test('does not claim the WROLPi is up to date', () => {
render(<UpgradeSegment/>, {status: unloadedStatus});
expect(screen.queryByText(/up to date/)).not.toBeInTheDocument();
});
test('once loaded with no update, says so', () => {
render(<UpgradeSegment/>, {status: {status: {update_available: false, version: '1.0'}, loaded: true}});
expect(screen.getByText(/up to date/)).toBeInTheDocument();
});
});
describe('SettingsPage before settings have loaded', () => {
test('shows a loader, not a form over blank values', () => {
const {container} = render(<SettingsPage/>, {settings: {settings: {}, loaded: false, failed: false}});
expect(container.querySelector('.wrolpi-loading')).toBeInTheDocument();
expect(screen.queryByText(/Any changes will be written/)).not.toBeInTheDocument();
});
test('shows an error when the settings could not be fetched', () => {
render(<SettingsPage/>, {settings: {settings: {}, loaded: false, failed: true}});
expect(screen.getByText('Unable to fetch settings')).toBeInTheDocument();
});
});

View file

@ -45,11 +45,13 @@ import {
toast,
} from './ui';
import {paletteColorNames} from '../themes/mantine';
import {contrastingColor} from './Common';
import {AiModelRow} from './Ai';
import {ChatMessage} from './AiChat';
import {contrastingColor, ViewProgress} from './Common';
import {navBarStyle, navColorNames, useNavColors} from '../themes/navColors';
import {IconMenu2} from '@tabler/icons-react';
import {BookmarkAddForms, BookmarkMenuItems, BookmarkTable} from './Bookmarks';
import {
RepoBreadcrumb, RepoHistoryTable, RepoImportSummary, RepoStatusIcon, RepoSummary, RepoTreeTable,
} from './Repos';
/*
* A gallery of every component in the library, in the current theme.
@ -158,6 +160,41 @@ export const NavBarSample = ({color}) => {
// press adds a toast rather than replacing the first. Module scope, so it survives re-renders.
let toastCounter = 0;
// Sample data for the bookmarks section; the editor's actions do nothing here.
const sampleBookmarks = [
{id: 1, name: 'Channels', url: '/videos/channel', new_tab: false},
{id: 2, name: 'Services', children: [
{id: 3, name: 'Jellyfin', url: ':8096/', new_tab: true},
{id: 4, name: 'Radio', children: [
{id: 5, name: 'OpenWebRX', url: 'http://:8073/', new_tab: true},
]},
]},
{id: 6, name: 'WROLPi.org', url: 'https://wrolpi.org', new_tab: true},
];
const noBookmarkActions = {edit: () => null, move: () => null, remove: () => null};
const sampleRepo = {
id: 1, name: 'kiwix-tools', tag_name: null, url: 'https://github.com/kiwix/kiwix-tools',
description: 'Command line tools for Kiwix and Zim files.', mode: 'full', branch: null, default_branch: 'main',
head_sha: '3f9c2a1e7b', head_message: 'Serve Zim files with the correct mimetype', head_date: '2026-09-30T14:02:00Z',
size: 18_400_000, directory: 'repos/kiwix-tools', download_error: null,
};
const sampleFailedRepo = {
...sampleRepo, name: 'gone-upstream', branch: 'release', mode: 'snapshot',
download_error: "git ls-remote failed (128): remote: Repository not found.\nfatal: repository " +
"'https://github.com/example/gone-upstream/' not found",
};
const sampleRepoCommits = [
{sha: '3f9c2a1e7b', author: 'Ada Example', date: '2026-09-30T14:02:00Z',
message: 'Serve Zim files with the correct mimetype'},
{sha: '91b0d44c02', author: 'Ben Example', date: '2026-09-12T09:15:00Z', message: 'Merge pull request #42'},
];
const sampleRepoEntries = [
{name: 'docs', path: 'src/docs', is_dir: true, size: null},
{name: 'kiwix-serve.cpp', path: 'src/kiwix-serve.cpp', is_dir: false, size: 48_210},
{name: 'README.md', path: 'src/README.md', is_dir: false, size: 2_104},
];
export function ThemeSamplePage() {
const {theme, mediaFilterEnabled} = useContext(ThemeContext);
const [confirmOpen, setConfirmOpen] = useState(false);
@ -727,51 +764,6 @@ export function ThemeSamplePage() {
</Table>
</Section>
<Section label='AI model catalog'>
{/* The real AiModelRow from the AI page's Manage tab, in both of its states. */}
<Table>
<Table.Header>
<Table.Row>
<Table.HeaderCell>Model</Table.HeaderCell>
<Table.HeaderCell>Tier</Table.HeaderCell>
<Table.HeaderCell>Size</Table.HeaderCell>
<Table.HeaderCell/>
</Table.Row>
</Table.Header>
<Table.Body>
<AiModelRow model={{
name: 'Qwen3-1.7B-Q4_K_M.gguf', tier: 'small', size: 1_200_000_000,
description: 'The default assistant for 4GB devices (Pi 4/5).',
downloaded: true, active: true,
}} onDownload={() => {}}/>
<AiModelRow model={{
name: 'Qwen3-4B-Instruct-2507-Q4_K_M.gguf', tier: 'medium', size: 2_600_000_000,
description: 'The default assistant for 8GB+ devices.',
downloaded: false, active: false,
}} onDownload={() => {}}/>
</Table.Body>
</Table>
</Section>
<Section label='AI chat'>
{/* The real ChatMessage from the AI Chat tab, in each of its states. */}
<ChatMessage message={{role: 'user', content: 'Find videos about canning'}}/>
<ChatMessage message={{
role: 'assistant',
content: 'I found one video: Pressure Canning Basics — /videos/123',
activity: [{tool: 'search_files', pending: false, success: true}],
}}/>
<ChatMessage message={{
role: 'assistant', content: '',
activity: [{tool: 'search_files', pending: true}], streaming: true,
}}/>
<ChatMessage message={{
role: 'assistant', content: '',
activity: [{tool: 'read_content', pending: false, success: false}],
error: 'The AI failed to answer. Check the logs.',
}}/>
</Section>
<Section label='Form controls'>
<Panel>
<div style={{display: 'grid', gridTemplateColumns: '2fr 1fr 1fr', gap: 12}}>
@ -1040,6 +1032,31 @@ export function ThemeSamplePage() {
rather than upscaled.
</p>
<Header as='h4'>Viewing progress</Header>
<CardGroup>
{[
['Part way through', 0.4],
['Nearly done', 0.9],
['Finished', 1],
['Not started', null],
].map(([label, progress]) => <Card
key={label}
title={label}
meta={progress ? `${Math.round(progress * 100)}%` : 'no bar'}
color='blue'
media={<div className='wrolpi-card-poster'>
<div className='wrolpi-card-poster-frame'>
<img alt='' src={samplePoster(1280, 720)}/>
<ViewProgress progress={progress}/>
</div>
</div>}
/>)}
</CardGroup>
<p style={{fontSize: '0.75rem', color: 'var(--muted)', marginTop: 10}}>
How far the user is through a video, audio file, ebook or comic, along the bottom
of its poster. A full bar is finished; a file not yet started has none.
</p>
<Header as='h4'>Meta, body and actions</Header>
<CardGroup>
{[
@ -1125,6 +1142,71 @@ export function ThemeSamplePage() {
</Panel>
</Section>
<Section label='Bookmarks'>
<Panel>
<p style={{fontSize: '0.75rem', color: 'var(--muted)', marginTop: 0}}>
The Bookmarks menu in the navigation bar, and the editor under More. A
directory is a flyout that opens beside the menu. In the mobile menu it is a
labelled, indented run of items instead.
</p>
<Row>
<Menu position='bottom-start' withinPortal trigger='click-hover'>
<Menu.Target>
<Button iconAfter='dropdown'>Bookmarks</Button>
</Menu.Target>
<Menu.Dropdown>
<BookmarkMenuItems nodes={sampleBookmarks}/>
<Menu.Divider/>
<Menu.Item leftSection={<Icon name='edit' size='small'/>}>Edit bookmarks</Menu.Item>
</Menu.Dropdown>
</Menu>
<Menu position='bottom-start' withinPortal>
<Menu.Target>
<Button iconAfter='dropdown'>As the mobile menu shows it</Button>
</Menu.Target>
<Menu.Dropdown>
<Menu.Label>Bookmarks</Menu.Label>
<BookmarkMenuItems nodes={sampleBookmarks} nested={false}/>
</Menu.Dropdown>
</Menu>
</Row>
<div style={{marginTop: 12}}>
<BookmarkTable nodes={sampleBookmarks} actions={noBookmarkActions}/>
<BookmarkAddForms nodes={sampleBookmarks} onAddBookmark={() => null} onAddDirectory={() => null}/>
</div>
</Panel>
</Section>
<Section label='Repos'>
<Panel>
<p style={{fontSize: '0.75rem', color: 'var(--muted)', marginTop: 0}}>
A git repo's page: its summary, then its files. A repo whose last update failed keeps
its files and explains why. The status icons appear beside a repo's name in the Repos
table: <RepoStatusIcon repo={sampleFailedRepo}/> failed,{' '}
<RepoStatusIcon repo={{...sampleRepo, head_sha: null}}/> not downloaded yet.
</p>
<RepoSummary repo={sampleRepo}/>
<div style={{marginTop: 12}}>
<RepoBreadcrumb name={sampleRepo.name} path='src' onNavigate={() => null}/>
<RepoTreeTable path='src' entries={sampleRepoEntries} onOpenDirectory={() => null}
onOpenFile={() => null}/>
</div>
<div style={{marginTop: 12}}>
<RepoHistoryTable commits={sampleRepoCommits}/>
</div>
<div style={{marginTop: 12}}>
<RepoImportSummary inspection={{
origin: 'git@github.com:kiwix/kiwix-tools.git', branch: 'main', head_sha: '3f9c2a1e7b',
head_message: 'Local work', head_date: '2026-09-30T14:02:00Z', local_commits: 2,
shallow: true, directory: 'code/kiwix-tools', ignored: false,
}}/>
</div>
<div style={{marginTop: 12}}>
<RepoSummary repo={sampleFailedRepo}/>
</div>
</Panel>
</Section>
<Section label='Danger zone'>
<Panel danger>
<h2 style={{color: 'var(--danger)', fontSize: '0.875rem', margin: '0 0 4px'}}>Danger zone</h2>

View file

@ -3,6 +3,7 @@ export const VIDEOS_API = `${API_URI}/videos`;
export const ARCHIVES_API = `${API_URI}/archive`;
export const COLLECTIONS_API = `${API_URI}/collections`;
export const ZIM_API = `${API_URI}/zim`;
export const REPOS_API = `${API_URI}/repos`;
export const DEFAULT_LIMIT = 20;
export const NAME = process.env && process.env.REACT_APP_NAME ? process.env.REACT_APP_NAME : null;
@ -168,11 +169,14 @@ export const transcodeVideoCodecOptions = [
{value: 'h264', label: 'h264 (avc1)'},
{value: TRANSCODE_REMOVE_VIDEO, label: 'Remove video (audio only)'},
];
// REMOVE_AUDIO in modules/videos/transcode.py: drop the audio stream, the result is a silent video.
export const TRANSCODE_REMOVE_AUDIO = 'none';
export const transcodeAudioCodecOptions = [
{value: TRANSCODE_COPY, label: 'Keep (copy)'},
{value: 'aac', label: 'aac'},
{value: 'opus', label: 'opus'},
{value: 'mp3', label: 'mp3'},
{value: TRANSCODE_REMOVE_AUDIO, label: 'Remove audio (silent video)'},
];
// Containers for output with a video stream ...
export const transcodeContainerOptions = [
@ -194,7 +198,6 @@ export function audioContainerForCodec(codec) {
}
export const Downloaders = {
AiModel: 'ai_model',
Archive: 'archive',
File: 'file',
KiwixCatalog: 'kiwix_catalog',

View file

@ -8,6 +8,7 @@ import {
import {
audioContainerForCodec,
TRANSCODE_COPY,
TRANSCODE_REMOVE_AUDIO,
TRANSCODE_REMOVE_VIDEO,
transcodeAudioCodecOptions,
transcodeAudioContainerOptions,
@ -68,6 +69,41 @@ export function containerFor({audioOutput, audioCodec, video}) {
return transcodeContainerOptions.some(i => i.value === current) ? current : 'mp4';
}
/**
* The video codec choices: Remove video is withdrawn once the audio is being removed (nothing
* would remain).
*/
export function videoCodecOptionsFor({removeAudio}) {
return removeAudio ? transcodeVideoCodecOptions.filter(i => i.value !== TRANSCODE_REMOVE_VIDEO)
: transcodeVideoCodecOptions;
}
/**
* The audio codec choices: Remove audio is only for a video whose video stream is kept; an audio
* file has nothing else, and removing the video already chose the audio as the result.
*/
export function audioCodecOptionsFor({audioOnly, removeVideo}) {
return (audioOnly || removeVideo) ? transcodeAudioCodecOptions.filter(i => i.value !== TRANSCODE_REMOVE_AUDIO)
: transcodeAudioCodecOptions;
}
/**
* The codec pair after one select changes. Both streams can never be removed (nothing would
* remain): choosing Remove for one stream puts the other back to Keep.
*/
export function chooseCodecs({videoCodec, audioCodec}, {video, audio}) {
let nextVideo = video !== undefined ? (video || TRANSCODE_COPY) : videoCodec;
let nextAudio = audio !== undefined ? (audio || TRANSCODE_COPY) : audioCodec;
if (nextVideo === TRANSCODE_REMOVE_VIDEO && nextAudio === TRANSCODE_REMOVE_AUDIO) {
if (video !== undefined) {
nextAudio = TRANSCODE_COPY;
} else {
nextVideo = TRANSCODE_COPY;
}
}
return {videoCodec: nextVideo, audioCodec: nextAudio};
}
const statusKind = (status) => {
if (status === 'complete') return 'complete';
if (status === 'running') return 'active';
@ -214,6 +250,7 @@ export function TranscodeModal({
const current = currentCodecs(video);
const removeVideo = videoCodec === TRANSCODE_REMOVE_VIDEO;
const removeAudio = !audioOnly && !removeVideo && audioCodec === TRANSCODE_REMOVE_AUDIO;
// An audio file has no video stream to keep or remove; removing the video makes an audio file.
const audioOutput = audioOnly || removeVideo;
@ -250,18 +287,20 @@ export function TranscodeModal({
};
}, [open]);
const changeVideoCodec = (value) => {
const next = value || TRANSCODE_COPY;
setVideoCodec(next);
setContainer(containerFor({audioOutput: audioOnly || next === TRANSCODE_REMOVE_VIDEO, audioCodec, video}));
};
const changeAudioCodec = (value) => {
const next = value || TRANSCODE_COPY;
setAudioCodec(next);
setContainer(containerFor({audioOutput, audioCodec: next, video}));
// `chooseCodecs` keeps Remove video and Remove audio from ever being chosen together; the
// other select also withdraws its Remove option (see `videoCodecOptionsFor`/`audioCodecOptionsFor`).
const applyCodecs = (change) => {
const next = chooseCodecs({videoCodec, audioCodec}, change);
setVideoCodec(next.videoCodec);
setAudioCodec(next.audioCodec);
setContainer(containerFor({
audioOutput: audioOnly || next.videoCodec === TRANSCODE_REMOVE_VIDEO, audioCodec: next.audioCodec, video,
}));
};
const changeVideoCodec = (value) => applyCodecs({video: value});
const changeAudioCodec = (value) => applyCodecs({audio: value});
const remuxOnly = !removeVideo && videoCodec === TRANSCODE_COPY && audioCodec === TRANSCODE_COPY;
const remuxOnly = !removeVideo && !removeAudio && videoCodec === TRANSCODE_COPY && audioCodec === TRANSCODE_COPY;
const sameContainer = container === currentContainer(video);
const containerOptions = audioOutput ? transcodeAudioContainerOptions : transcodeContainerOptions;
// Fragments are an mp4 concept (m4a is an mp4).
@ -272,7 +311,8 @@ export function TranscodeModal({
try {
const id = await transcodeVideo(fileGroupId, {
video_codec: videoCodec === TRANSCODE_COPY ? null : videoCodec,
audio_codec: audioCodec === TRANSCODE_COPY ? null : audioCodec,
audio_codec: audioCodec === TRANSCODE_COPY || (removeVideo && audioCodec === TRANSCODE_REMOVE_AUDIO)
? null : audioCodec,
container,
fragmented: canFragment && fragmented,
});
@ -292,6 +332,9 @@ export function TranscodeModal({
if (defaultsLoaded && removeVideo) {
hint = `The video stream is removed; the result is an audio file (.${container}). `
+ (audioCodec === TRANSCODE_COPY ? 'The audio is copied without re-encoding.' : `The audio is converted to ${audioCodec}.`);
} else if (defaultsLoaded && removeAudio) {
hint = `The audio stream is removed; the result is a silent video (.${container}). `
+ (videoCodec === TRANSCODE_COPY ? 'The video is copied without re-encoding.' : `The video is converted to ${videoCodec}.`);
} else if (defaultsLoaded && remuxOnly) {
const kept = audioOnly ? 'The audio is kept' : 'Both streams are kept';
if (sameContainer && canFragment && fragmented) {
@ -303,7 +346,7 @@ export function TranscodeModal({
hint = `${kept}: only the container changes to ${container} (a quick remux, no quality loss).`;
}
}
const buttonLabel = removeVideo ? 'Extract Audio' : remuxOnly ? 'Remux' : 'Transcode';
const buttonLabel = removeVideo ? 'Extract Audio' : removeAudio ? 'Remove Audio' : remuxOnly ? 'Remux' : 'Transcode';
return <Modal open={open} onClose={onClose} size='small'>
<Modal.Header>{audioOnly ? 'Transcode Audio' : 'Transcode Video'}</Modal.Header>
@ -319,7 +362,7 @@ export function TranscodeModal({
{!showJob && <>
{!audioOnly && <Select
label='Video codec'
data={transcodeVideoCodecOptions}
data={videoCodecOptionsFor({removeAudio})}
value={videoCodec}
onChange={changeVideoCodec}
disabled={!defaultsLoaded}
@ -327,7 +370,7 @@ export function TranscodeModal({
/>}
<Select
label='Audio codec'
data={transcodeAudioCodecOptions}
data={audioCodecOptionsFor({audioOnly, removeVideo})}
value={audioCodec}
onChange={changeAudioCodec}
disabled={!defaultsLoaded}
@ -379,14 +422,27 @@ export function EditVideoModal({open, onClose, fileGroupId, videoFile, descripti
const [title, setTitle] = useState('');
const [description, setDescription] = useState('');
const [saving, setSaving] = useState(false);
// The description last copied into the textarea; the user has typed when the field differs.
const syncedDescription = useRef('');
useEffect(() => {
if (open) {
// Start from the current details on every open, discarding an abandoned edit.
setTitle(videoFile?.title || '');
setDescription(currentDescription || '');
syncedDescription.current = currentDescription || '';
}
}, [open, videoFile?.title, currentDescription]);
}, [open, videoFile?.title]);
useEffect(() => {
// The description may still be loading when the dialog opens (null), and it is refetched
// after a save. Adopt what arrives only while the user has not typed, so a draft written
// over the placeholder is never wiped by the empty or failed value that follows it.
if (open && currentDescription != null && description === syncedDescription.current) {
setDescription(currentDescription);
syncedDescription.current = currentDescription;
}
}, [currentDescription]);
const trimmed = title.trim();
const titleChanged = trimmed !== (videoFile?.title || '');

View file

@ -2,11 +2,11 @@ import React from 'react';
import {act, fireEvent, screen, waitFor} from '@testing-library/react';
import {renderWithProviders} from '../test-utils';
import {
containerFor, currentCodecs, currentContainer, EditVideoModal, JobProgress, preferredTarget, TranscodeModal,
useJob, VideoEditMenu,
audioCodecOptionsFor, chooseCodecs, containerFor, currentCodecs, currentContainer, EditVideoModal, JobProgress, preferredTarget,
TranscodeModal, useJob, VideoEditMenu, videoCodecOptionsFor,
} from './VideoEditMenu';
import {
audioContainerForCodec, TRANSCODE_COPY, TRANSCODE_REMOVE_VIDEO, transcodeAudioCodecOptions,
audioContainerForCodec, TRANSCODE_COPY, TRANSCODE_REMOVE_AUDIO, TRANSCODE_REMOVE_VIDEO, transcodeAudioCodecOptions,
transcodeVideoCodecOptions,
} from './Vars';
@ -221,6 +221,35 @@ describe('TranscodeModal', () => {
// Selecting it is exercised through containerFor (Mantine's dropdown is not driveable in jsdom).
});
test('the Remove audio option is offered for videos, not audio files, and never with Remove video', () => {
expect(transcodeAudioCodecOptions.some(i => i.value === TRANSCODE_REMOVE_AUDIO)).toBe(true);
const has = (options, value) => options.some(i => i.value === value);
expect(has(audioCodecOptionsFor({audioOnly: false, removeVideo: false}), TRANSCODE_REMOVE_AUDIO)).toBe(true);
expect(has(audioCodecOptionsFor({audioOnly: true, removeVideo: false}), TRANSCODE_REMOVE_AUDIO)).toBe(false);
expect(has(audioCodecOptionsFor({audioOnly: false, removeVideo: true}), TRANSCODE_REMOVE_AUDIO)).toBe(false);
// The real codecs and Keep are never withdrawn.
expect(audioCodecOptionsFor({audioOnly: true, removeVideo: false}).map(i => i.value))
.toEqual([TRANSCODE_COPY, 'aac', 'opus', 'mp3']);
expect(has(videoCodecOptionsFor({removeAudio: false}), TRANSCODE_REMOVE_VIDEO)).toBe(true);
expect(has(videoCodecOptionsFor({removeAudio: true}), TRANSCODE_REMOVE_VIDEO)).toBe(false);
expect(has(videoCodecOptionsFor({removeAudio: true}), 'h264')).toBe(true);
});
test('Remove video and Remove audio can never be chosen together', () => {
const keepBoth = {videoCodec: TRANSCODE_COPY, audioCodec: TRANSCODE_COPY};
// Removing one stream after the other was chosen puts the other back to Keep.
expect(chooseCodecs({videoCodec: TRANSCODE_REMOVE_VIDEO, audioCodec: 'aac'}, {audio: TRANSCODE_REMOVE_AUDIO}))
.toEqual({videoCodec: TRANSCODE_COPY, audioCodec: TRANSCODE_REMOVE_AUDIO});
expect(chooseCodecs({videoCodec: 'h264', audioCodec: TRANSCODE_REMOVE_AUDIO}, {video: TRANSCODE_REMOVE_VIDEO}))
.toEqual({videoCodec: TRANSCODE_REMOVE_VIDEO, audioCodec: TRANSCODE_COPY});
// Ordinary choices are untouched; a cleared select means Keep.
expect(chooseCodecs(keepBoth, {video: 'h264'})).toEqual({videoCodec: 'h264', audioCodec: TRANSCODE_COPY});
expect(chooseCodecs({videoCodec: 'h264', audioCodec: 'aac'}, {audio: null}))
.toEqual({videoCodec: 'h264', audioCodec: TRANSCODE_COPY});
expect(chooseCodecs(keepBoth, {audio: TRANSCODE_REMOVE_AUDIO}))
.toEqual({videoCodec: TRANSCODE_COPY, audioCodec: TRANSCODE_REMOVE_AUDIO});
});
test('WROL Mode disables starting a transcode', async () => {
renderWithProviders(
<TranscodeModal open={true} onClose={jest.fn()} fileGroupId={7} video={video}/>,
@ -269,6 +298,26 @@ describe('EditVideoModal', () => {
await waitFor(() => expect(updateVideo).toHaveBeenCalledWith(7, {description: ''}));
});
test('a draft typed while the description was still loading survives its arrival', () => {
// The page passes null while the description is pending, then '' when the server has none.
const {rerender} = renderWithProviders(
<EditVideoModal open={true} onClose={jest.fn()} fileGroupId={7} videoFile={videoFile} description={null}/>,
);
fireEvent.change(screen.getByLabelText('Description'), {target: {value: 'my new words'}});
rerender(<EditVideoModal open={true} onClose={jest.fn()} fileGroupId={7} videoFile={videoFile} description=''/>);
expect(screen.getByLabelText('Description')).toHaveValue('my new words');
});
test('a description that arrives while the field is untouched is adopted', () => {
const {rerender} = renderWithProviders(
<EditVideoModal open={true} onClose={jest.fn()} fileGroupId={7} videoFile={videoFile} description={null}/>,
);
expect(screen.getByLabelText('Description')).toHaveValue('');
rerender(<EditVideoModal open={true} onClose={jest.fn()} fileGroupId={7} videoFile={videoFile}
description='From the server'/>);
expect(screen.getByLabelText('Description')).toHaveValue('From the server');
});
test('WROL Mode disables saving', async () => {
renderWithProviders(
<EditVideoModal open={true} onClose={jest.fn()} fileGroupId={7} videoFile={videoFile}/>,

View file

@ -0,0 +1,66 @@
import React from 'react';
import {render, screen} from '../test-utils';
import {Comments, VideoDescription} from './VideoPlayer';
// What the video page shows for each state of useVideoExtras. The states are null (pending),
// undefined (fetch failed), and an empty value (loaded, nothing to show). "No comments have been
// downloaded" carries a Refresh button that re-downloads the video, so it must only appear once
// the server has confirmed there are none.
const video = {id: 7, url: 'https://example.com/v'};
const NO_COMMENTS = /No comments have been/;
const NO_DESCRIPTION = /No description available/;
describe('Comments', () => {
test('pending: shows a placeholder, not "no comments" or an error', () => {
const {container} = render(<Comments comments={null} video={video}/>);
expect(container.querySelector('.wrolpi-placeholder')).toBeInTheDocument();
expect(screen.queryByText(NO_COMMENTS)).not.toBeInTheDocument();
expect(screen.queryByText(/Could not/)).not.toBeInTheDocument();
expect(screen.queryByRole('button', {name: 'Refresh'})).not.toBeInTheDocument();
});
test('failed: shows an error, not "no comments"', () => {
render(<Comments comments={undefined} video={video}/>);
expect(screen.getByText(/Could not fetch the comments/)).toBeInTheDocument();
expect(screen.queryByText(NO_COMMENTS)).not.toBeInTheDocument();
});
test('empty: shows "no comments" with the Refresh button', () => {
render(<Comments comments={[]} video={video}/>);
expect(screen.getByText(NO_COMMENTS)).toBeInTheDocument();
expect(screen.getByRole('button', {name: 'Refresh'})).toBeInTheDocument();
});
test('loaded: renders the comments', () => {
render(<Comments comments={[{id: 'c1', parent: 'root', text: 'first!', author: 'a'}]} video={video}/>);
expect(screen.getByText('first!')).toBeInTheDocument();
expect(screen.queryByText(NO_COMMENTS)).not.toBeInTheDocument();
});
});
describe('VideoDescription', () => {
test('pending: shows a placeholder, not "no description" or an error', () => {
const {container} = render(<VideoDescription description={null} setVideoTime={() => null}/>);
expect(container.querySelector('.wrolpi-placeholder')).toBeInTheDocument();
expect(screen.queryByText(NO_DESCRIPTION)).not.toBeInTheDocument();
expect(screen.queryByText(/Could not/)).not.toBeInTheDocument();
});
test('failed: shows an error, not "no description"', () => {
render(<VideoDescription description={undefined} setVideoTime={() => null}/>);
expect(screen.getByText(/Could not fetch the description/)).toBeInTheDocument();
expect(screen.queryByText(NO_DESCRIPTION)).not.toBeInTheDocument();
});
test('empty: shows "no description"', () => {
render(<VideoDescription description='' setVideoTime={() => null}/>);
expect(screen.getByText(NO_DESCRIPTION)).toBeInTheDocument();
});
test('loaded: renders the description', () => {
render(<VideoDescription description='A fine video.' setVideoTime={() => null}/>);
expect(screen.getByText('A fine video.')).toBeInTheDocument();
expect(screen.queryByText(NO_DESCRIPTION)).not.toBeInTheDocument();
});
});

View file

@ -8,6 +8,7 @@ import {
BackButton,
DirectoryLink,
encodeMediaPath,
ErrorMessage,
humanFileSize,
humanNumber,
isoDatetimeToAgoPopup,
@ -18,8 +19,9 @@ import {
useTitle
} from "./Common";
import {VideoPlaceholder} from "./Placeholder";
import {useChannel, useVideoCaptions, useVideoExtras} from "../hooks/customHooks";
import {Button, Grid, Header, Icon, Label, Loading, Panel, Tabs, TextInput} from "./ui";
import {useChannel, useOneQuery, useVideoCaptions, useVideoExtras} from "../hooks/customHooks";
import {useMediaProgress} from "../hooks/useFileProgress";
import {Button, Grid, Header, Icon, Label, Loading, Panel, Placeholder, Tabs, TextInput} from "./ui";
import {VideoCard} from "./Videos";
import {TagsSelector} from "../Tags";
import {AddToPlaylistButton} from "./AddToPlaylist";
@ -170,8 +172,14 @@ const VideoComment = ({comment, children}) => {
}
const Comments = ({comments, video}) => {
if (!comments || _.isEmpty(comments)) {
// `comments` follows useVideoExtras: null = pending, undefined = fetch failed, [] = none downloaded.
// Only the last of those may offer the Refresh button.
export const Comments = ({comments, video}) => {
if (comments === null) {
return <Placeholder lines={3}/>
} else if (comments === undefined) {
return <ErrorMessage>Could not fetch the comments.</ErrorMessage>
} else if (_.isEmpty(comments)) {
return <NoComments video={video}/>
}
@ -223,6 +231,12 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
video = videoFile.video;
}
// `?t=120` starts at 2:00, rather than where the user left off.
const [startTimeQuery] = useOneQuery('t');
const startSeconds = startTimeQuery && Number.isFinite(parseFloat(startTimeQuery))
? parseFloat(startTimeQuery) : null;
const progressHandlers = useMediaProgress(videoFile?.primary_path, videoFile, {startSeconds});
// Seeks to the `seconds` on video player.
const setVideoTime = (seconds) => {
videoRef.current.currentTime = seconds;
@ -343,12 +357,6 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
caption_files.filter(i => i['mimetype'] === 'text/vtt').map(i => `${MEDIA_PATH}/${encodeMediaPath(i['path'])}`)
: [];
let description = 'No description available.';
if (videoDescription) {
// Only replace empty description if there is one available.
description = formatVideoDescription(videoDescription, setVideoTime)
}
const getFile = (suffix) => videoFile.files.find(i => i.path.toLowerCase().endsWith(suffix));
const fileSize = (file) => file && file.size
? <span style={{marginLeft: '1em', color: 'var(--muted)'}}>({humanFileSize(file.size)})</span>
@ -455,6 +463,7 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
id="player"
style={{width: '100%', marginTop: posterUrl ? '0.5em' : 0}}
ref={videoRef}
{...progressHandlers}
>
{videoSource}
</audio>
@ -466,6 +475,7 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
playsInline={true}
style={{maxWidth: '100%'}}
ref={videoRef}
{...progressHandlers}
>
{videoSource}
{/* Only WebVTT captions can be displayed. */}
@ -550,9 +560,7 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
</Tabs.Panel>
<Tabs.Panel value='description' style={{paddingTop: '1em'}}>
<pre className="wrap-text">
{description}
</pre>
<VideoDescription description={videoDescription} setVideoTime={setVideoTime}/>
</Tabs.Panel>
<Tabs.Panel value='files' style={{paddingTop: '1em'}}>
@ -717,6 +725,18 @@ export default VideoPage;
const chapterRegex = new RegExp('^(\\(?(?:((\\d?\\d):)?(?:(\\d?\\d):(\\d\\d)))\\)?)\.?\\s+(.*)$', 'i');
// `description` follows useVideoExtras: null = pending, undefined = fetch failed, '' = none.
export const VideoDescription = ({description, setVideoTime}) => {
if (description === null) {
return <Placeholder lines={3}/>
} else if (description === undefined) {
return <ErrorMessage>Could not fetch the description.</ErrorMessage>
}
return <pre className="wrap-text">
{description ? formatVideoDescription(description, setVideoTime) : 'No description available.'}
</pre>
}
function formatVideoDescription(description, setVideoTime) {
// Convert timestamps to links which change the video's playback location. Change hashtags to search links.
if (description && description.length > 0) {

View file

@ -0,0 +1,43 @@
import React from 'react';
import {render, screen} from '../../test-utils';
import {BackupsModal} from './Configs';
import {getConfigBackups} from '../../api';
// The config backups modal says "No backups available" only once the server confirmed there are
// none; a failed fetch is an error, and a thrown one must not leave the loader up forever.
jest.mock('../../api', () => ({
...jest.requireActual('../../api'),
getConfigBackups: jest.fn(),
postConfigBackupPreview: jest.fn(() => new Promise(() => {
})),
}));
const NONE = /No backups available/;
const FAILED = /Could not fetch backups/;
const renderModal = () => render(
<BackupsModal open fileName='wrolpi.yaml' onClose={jest.fn()} fetchConfigs={jest.fn()}/>,
);
beforeEach(() => getConfigBackups.mockReset());
test('empty: says there are no backups', async () => {
getConfigBackups.mockResolvedValue({dates: []});
renderModal();
expect(await screen.findByText(NONE)).toBeInTheDocument();
});
test('failed: shows an error, not "No backups available"', async () => {
getConfigBackups.mockResolvedValue(undefined);
renderModal();
expect(await screen.findByText(FAILED)).toBeInTheDocument();
expect(screen.queryByText(NONE)).not.toBeInTheDocument();
});
test('thrown: shows an error, not a loader forever', async () => {
getConfigBackups.mockRejectedValue(new Error('down'));
renderModal();
expect(await screen.findByText(FAILED)).toBeInTheDocument();
expect(screen.queryByText(/Loading backups/)).not.toBeInTheDocument();
});

View file

@ -1,4 +1,4 @@
import {APIButton} from "../Common";
import {APIButton, ErrorMessage} from "../Common";
import {Button, Header, Icon, IconButton, Loader, Loading, Modal, Placeholder, Table, Tooltip} from "../ui";
import React, {useEffect, useState} from "react";
import {getConfigText, getConfigBackups, postConfigBackupImport, postConfigBackupPreview} from "../../api";
@ -117,9 +117,11 @@ function BackupDateRow({date, previews, previewsLoaded, onSelect}) {
</Table.Row>;
}
function BackupsModal({open, onClose, fileName, fetchConfigs}) {
export function BackupsModal({open, onClose, fileName, fetchConfigs}) {
const [dates, setDates] = useState([]);
const [datesLoading, setDatesLoading] = useState(true);
// The backups could not be fetched; distinct from the server confirming there are none.
const [datesFailed, setDatesFailed] = useState(false);
const [previews, setPreviews] = useState({});
const [previewsLoaded, setPreviewsLoaded] = useState({});
const [selectedAction, setSelectedAction] = useState(null); // {date, mode, preview}
@ -130,17 +132,30 @@ function BackupsModal({open, onClose, fileName, fetchConfigs}) {
if (!open) return;
let cancelled = false;
setDatesLoading(true);
setDatesFailed(false);
setDates([]);
setPreviews({});
setPreviewsLoaded({});
setSelectedAction(null);
setApplying(false);
getConfigBackups(fileName).then(result => {
if (cancelled) return;
setDates(result?.dates || []);
setDatesLoading(false);
});
getConfigBackups(fileName)
.then(result => {
if (cancelled) return;
// The api helper returns undefined on a non-OK response.
if (result) {
setDates(result.dates || []);
} else {
setDatesFailed(true);
}
})
.catch(e => {
console.error(e);
if (!cancelled) setDatesFailed(true);
})
.finally(() => {
if (!cancelled) setDatesLoading(false);
});
return () => { cancelled = true; };
}, [open, fileName]);
@ -253,6 +268,8 @@ function BackupsModal({open, onClose, fileName, fetchConfigs}) {
<Modal.Content>
{datesLoading
? <Loading>Loading backups...</Loading>
: datesFailed
? <ErrorMessage>Could not fetch backups for this config.</ErrorMessage>
: dates.length === 0
? <p>No backups available for this config.</p>
: <div style={{maxHeight: '400px', overflowY: 'auto'}}>

View file

@ -101,8 +101,8 @@ export function ShutdownButton() {
</APIButton>
}
function UpgradeSegment() {
const {status, fetchStatus} = React.useContext(StatusContext);
export function UpgradeSegment() {
const {status, fetchStatus, loaded} = React.useContext(StatusContext);
const dockerized = useDockerized();
const [upgrading, setUpgrading] = React.useState(false);
const [checking, setChecking] = React.useState(false);
@ -144,6 +144,14 @@ function UpgradeSegment() {
</Panel>;
}
// Nothing is known until the first status poll lands; "up to date" is a claim about the server.
if (!loaded) {
return <Panel id='upgrade'>
<Header as='h3'>System Upgrade</Header>
<Loading/>
</Panel>;
}
// No update available
if (!status?.update_available) {
return <Panel id='upgrade'>
@ -325,7 +333,9 @@ export function SettingsPage() {
const [isDevMode, setIsDevMode] = React.useState(false);
// Get current settings from the API.
const {settings, saveSettings, fetchSettings} = React.useContext(SettingsContext);
const {
settings, saveSettings, fetchSettings, loaded: settingsLoaded, failed: settingsFailed,
} = React.useContext(SettingsContext);
const {status} = React.useContext(StatusContext);
// Used to track changes to the settings form between saves/loads.
const [state, setState] = React.useState({});
@ -463,7 +473,8 @@ export function SettingsPage() {
React.useEffect(() => {
console.debug('settings changed, replacing state...');
setReady(settings ? true : undefined);
// true = show the form, false = still loading, undefined = could not fetch.
setReady(settingsLoaded ? true : settingsFailed ? undefined : false);
setState({
// Only these settings can be changed on the SettingsPage.
archive_destination: settings.archive_destination,
@ -488,9 +499,10 @@ export function SettingsPage() {
videos_destination: settings.videos_destination,
zims_destination: settings.zims_destination,
playlists_destination: settings.playlists_destination,
repos_destination: settings.repos_destination,
save_ffprobe_json: settings.save_ffprobe_json,
});
}, [JSON.stringify(settings)]);
}, [JSON.stringify(settings), settingsLoaded, settingsFailed]);
const handleInputChange = async (name, value) => {
setState({...state, [name]: value});
@ -838,6 +850,32 @@ export function SettingsPage() {
onChange={e => handleInputChange('playlists_destination', e.currentTarget.value)}
/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<PathInput
label={<>
Repos Directory
<InfoPopup
content={<>
<p>Variables:</p>
<ul>
<li><code>%(repo_name)s</code> - Repo name</li>
<li><code>%(repo_tag)s</code> - Tag name for the repo (empty if no tag)
</li>
<li><code>%(repo_owner)s</code> - Owner from the repo URL</li>
<li><code>%(repo_host)s</code> - Host from the repo URL</li>
</ul>
<p>Must start with a fixed directory (like <code>repos/</code>), which
is never indexed, and contain <code>%(repo_name)s</code>.</p>
</>}
position='top'
/>
</>}
prefix={mediaDirectoryLabel}
value={state.repos_destination}
disabled={!editSpecialDirectories}
onChange={e => handleInputChange('repos_destination', e.currentTarget.value)}
/>
</Grid.Col>
<Grid.Col span={12}>
<Toggle
label='Edit Directories'

View file

@ -0,0 +1,625 @@
import React from "react";
import {useSearchParams} from "react-router";
import {
Accordion,
Button,
Grid,
Group,
Header,
IconButton,
Loader,
Message,
NumberInput,
Panel,
Select,
Statistic,
StatisticGroup,
Table,
Tabs,
TextInput,
Toggle,
} from "../ui";
import {InfoPopup, roundDigits} from "../Common";
import {
DEFAULT_LOSSES,
defaultAzimuth,
estimateProduction,
formatMonthly,
INVERTER_NOMINAL_EFFICIENCY,
LOSS_CATEGORIES,
MODULE_TYPES,
MONTH_NAMES,
MOUNTINGS,
optimalTilt,
parseCoordinate,
parseMonthly,
totalLoss,
TRACKING,
} from "./solar";
import {
CLIMATOLOGY_ACKNOWLEDGMENT,
CLIMATOLOGY_CHANGES,
CLIMATOLOGY_DATA_REFERENCE,
CLIMATOLOGY_LICENSE,
lookupClimatology,
useClimatology,
} from "./solarData";
import {SolarOffGrid} from "./SolarOffGrid";
// Every input lives in the URL query so the Share button (which shares window.location.href) shares
// the whole configuration. Values equal to their default are left out to keep the URL, and its QR
// code, small. Keys are short and must stay stable; shared links depend on them.
const DEFAULT_PANEL_WATTS = 400;
const DEFAULT_PANEL_COUNT = 1;
const COMPASS = [
{value: '0', label: 'N'}, {value: '45', label: 'NE'}, {value: '90', label: 'E'}, {value: '135', label: 'SE'},
{value: '180', label: 'S'}, {value: '225', label: 'SW'}, {value: '270', label: 'W'}, {value: '315', label: 'NW'},
];
const fToC = f => (f - 32) * 5 / 9;
const cToF = c => c * 9 / 5 + 32;
const numberOr = (value, fallback) => {
if (value === null || value === undefined || value === '') {
return fallback;
}
const n = Number(value);
return Number.isFinite(n) ? n : fallback;
};
const fmt = (value, digits = 2) => (Number.isFinite(value) ? `${roundDigits(value, digits)}` : '—');
// Read and write the calculator's query parameters. Writes replace the history entry, so typing does
// not fill the Back button with one entry per keystroke.
export function useSolarParams() {
const [searchParams, setSearchParams] = useSearchParams();
const get = key => searchParams.get(key);
// `updates` maps keys to values; null, undefined and '' remove the key.
const set = updates => setSearchParams(prev => {
const next = new URLSearchParams(prev);
Object.entries(updates).forEach(([key, value]) => {
if (value === null || value === undefined || value === '') {
next.delete(key);
} else {
next.set(key, `${value}`);
}
});
return next;
}, {replace: true});
// Store a number, or remove it when it equals the default.
const setNumber = (key, value, fallback) =>
set({[key]: value === '' || Number(value) === fallback ? null : value});
return {searchParams, get, set, setNumber};
}
// Turn the query parameters into the model's inputs. Invalid values fall back to defaults, so a
// hand-edited or truncated link never crashes the page.
export function systemFromParams(get) {
const latitude = parseCoordinate(get('lat'), 'lat');
const longitude = parseCoordinate(get('lon'), 'lon');
const fahrenheit = get('tu') === 'f';
// Own sunlight data replaces the NASA data when present.
const ownData = get('ghi') !== null;
const ghi = parseMonthly(get('ghi')).map(v => (v !== null && v >= 0 ? v : null));
const temperatureInput = parseMonthly(get('temp'));
const temperature = temperatureInput.map(v => (v === null ? null : (fahrenheit ? fToC(v) : v)));
const panelWatts = Math.max(0, numberOr(get('pw'), DEFAULT_PANEL_WATTS));
const panelCount = Math.max(0, Math.round(numberOr(get('pn'), DEFAULT_PANEL_COUNT)));
const kwp = panelWatts * panelCount / 1000;
const tracking = TRACKING[get('track')] ? get('track') : 'fixed';
const mounting = MOUNTINGS[get('mount')] ? get('mount') : 'open';
const gamma = numberOr(get('gamma'), MODULE_TYPES.standard.gamma);
const defaultTilt = latitude === null ? 0 : Math.round(Math.abs(latitude));
const tilt = Math.min(90, Math.max(0, numberOr(get('tilt'), defaultTilt)));
const azimuth = numberOr(get('az'), defaultAzimuth(latitude ?? 0));
// Null means "use the monthly albedo from the data", which includes seasonal snow.
const albedoInput = numberOr(get('alb'), null);
const albedo = albedoInput === null ? null : Math.min(1, Math.max(0, albedoInput));
const lossValues = parseMonthly(get('losses'));
const losses = LOSS_CATEGORIES.map((c, i) => {
const v = lossValues[i];
return v !== null && v >= 0 && v <= 100 ? v : c.default;
});
const inverterOn = get('inv') === '1';
const acRating = numberOr(get('iac'), roundDigits(kwp / 1.2, 2));
const efficiency = numberOr(get('ieff'), INVERTER_NOMINAL_EFFICIENCY);
return {
latitude, longitude, fahrenheit, ownData, ghi, temperatureInput, temperature, panelWatts, panelCount,
kwp, tracking, mounting, gamma, defaultTilt, tilt, azimuth, albedo, losses, inverterOn, acRating,
efficiency,
};
}
// The sunlight inputs for the model: the user's own values, or the NASA data at the location.
export function sunlightFor(s, grid) {
if (s.ownData) {
return {ghi: s.ghi, diffuse: null, temperature: s.temperature, albedo: null};
}
if (!grid || s.latitude === null || s.longitude === null) {
return null;
}
return lookupClimatology(grid, s.latitude, s.longitude);
}
// What stops an estimate, or an empty list when it can run.
export function missingInputs(s, sunlight) {
const missing = [];
if (s.latitude === null) {
missing.push('a latitude');
}
if (!s.ownData && s.longitude === null) {
missing.push('a longitude');
}
if (s.ownData && s.ghi.some(v => v === null)) {
missing.push('sunlight for all 12 months');
}
if (!(s.kwp > 0)) {
missing.push('panel watts and count');
}
if (missing.length === 0 && (!sunlight || sunlight.ghi.some(v => v === null))) {
missing.push('a location with sunlight data');
}
return missing;
}
const modelInputs = (s, sunlight) => ({
latitude: s.latitude,
ghi: sunlight.ghi,
diffuse: sunlight.diffuse,
temperature: sunlight.temperature,
albedo: s.albedo ?? sunlight.albedo ?? undefined,
kwp: s.kwp,
tilt: s.tilt,
azimuth: s.azimuth,
tracking: s.tracking,
mounting: s.mounting,
gamma: s.gamma,
losses: s.losses,
inverter: s.inverterOn ? {acRating: s.acRating, efficiency: s.efficiency} : null,
});
const compassLabel = azimuth => {
const exact = COMPASS.find(c => Number(c.value) === azimuth);
return exact ? exact.label : `${azimuth}°`;
};
// Small explanatory text at the top of a collapsed section.
const Hint = ({children}) => <p style={{fontSize: '0.9em', opacity: 0.75, marginTop: 0}}>{children}</p>;
// NASA POWER's requested acknowledgment, the license, and how WROLPi changed the data (CC BY 4.0).
function DataAttribution() {
return <section aria-labelledby='solar-data-attribution'
style={{marginTop: '2em', fontSize: '0.85em', opacity: 0.75}}>
<Header as='h4' id='solar-data-attribution'>Data attribution</Header>
<p>{CLIMATOLOGY_ACKNOWLEDGMENT} {CLIMATOLOGY_DATA_REFERENCE}</p>
<p>License: {CLIMATOLOGY_LICENSE}</p>
<p>{CLIMATOLOGY_CHANGES}</p>
</section>;
}
const ESTIMATE_INFO = 'A typical-year estimate from monthly averages. Real years vary; expect roughly ±10–15% '
+ 'over a year and more in any single month. Shading from trees and buildings is not modeled beyond the '
+ 'Shading loss.';
// Browsers only offer geolocation to pages served over HTTPS.
export const geolocationAvailable = () => Boolean(window.isSecureContext && navigator.geolocation);
const GEOLOCATION_ERRORS = {
1: 'Location permission was denied. Allow it in your browser\'s settings for this site.',
2: 'Your device could not find its location. Phones with GPS work offline; most computers need the Internet '
+ 'for this.',
3: 'Finding your location took too long. Phones with GPS work offline; most computers need the Internet for '
+ 'this.',
};
function LocationSection({get, set, s, climatology}) {
const [locating, setLocating] = React.useState(false);
const [geoError, setGeoError] = React.useState(null);
const latError = get('lat') && s.latitude === null ? 'Latitude must be between -90 and 90' : null;
const lonError = get('lon') && s.longitude === null ? 'Longitude must be between -180 and 180' : null;
const locate = () => {
setLocating(true);
setGeoError(null);
navigator.geolocation.getCurrentPosition(
({coords}) => {
setLocating(false);
// 4 decimals (about 11 m) is far finer than the 1° data and keeps the shared URL short.
set({lat: Number(coords.latitude.toFixed(4)), lon: Number(coords.longitude.toFixed(4))});
},
error => {
setLocating(false);
setGeoError(GEOLOCATION_ERRORS[error.code] || error.message);
},
{enableHighAccuracy: false, timeout: 20000, maximumAge: 10 * 60 * 1000},
);
};
// The data's attribution is at the bottom of the page; this line only reports the data's state.
let source = null;
if (s.ownData) {
source = 'Using your own sunlight data.';
} else if (climatology.loading) {
source = <Loader size='xs' label='Loading sunlight data'/>;
} else if (climatology.error) {
source = <Message kind='warning' title='Sunlight data could not be loaded'>
Enter your own monthly sunlight under "Use my own sunlight data" below.
</Message>;
}
return <>
<Header as='h3'>1. Location</Header>
<Grid align='flex-end'>
<Grid.Col span='content'>
<IconButton icon='current location' label='Use my current location' size='input-sm'
loading={locating} disabled={!geolocationAvailable()} onClick={locate}/>
</Grid.Col>
<Grid.Col span='auto'>
<TextInput label='Latitude' placeholder='40.015 or 40°0′54″N' name='lat'
value={get('lat') || ''} error={latError}
onChange={e => set({lat: e.target.value})}/>
</Grid.Col>
<Grid.Col span={{base: 12, xs: 'auto'}}>
<TextInput label='Longitude' placeholder='-105.27 or 105°16′12″W' name='lon'
value={get('lon') || ''} error={lonError}
onChange={e => set({lon: e.target.value})}/>
</Grid.Col>
</Grid>
{geoError && <Message kind='warning' title='Could not get your location'>{geoError}</Message>}
{source && <div style={{fontSize: '0.85em', opacity: 0.75, marginTop: '0.5em'}}>{source}</div>}
</>;
}
function PanelsSection({setNumber, s}) {
return <>
<Header as='h3' style={{marginTop: '1em'}}>2. Your panels</Header>
<Grid>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Watts per panel' name='pw' value={s.panelWatts} min={0} step={10}
onChange={v => setNumber('pw', v, DEFAULT_PANEL_WATTS)}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Panels' name='pn' value={s.panelCount} min={0} step={1}
onChange={v => setNumber('pn', v, DEFAULT_PANEL_COUNT)}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}} style={{display: 'flex', alignItems: 'flex-end'}}>
<span style={{opacity: 0.75, paddingBottom: '0.5em'}}>
{fmt(s.kwp * 1000, 0)} W total ({fmt(s.kwp)} kW)
</span>
</Grid.Col>
</Grid>
</>;
}
function Results({result, s}) {
const unit = s.inverterOn ? 'kWh AC' : 'kWh DC';
const rows = result.months.map(m => <Table.Row key={m.month}>
<Table.Cell>{m.month}</Table.Cell>
<Table.Cell>{fmt(m.energyDaily)}</Table.Cell>
<Table.Cell>{fmt(m.energy, 0)}</Table.Cell>
<Table.Cell>{fmt(m.poa)}</Table.Cell>
<Table.Cell>{fmt(m.dayLength, 1)} h</Table.Cell>
</Table.Row>);
return <Panel style={{marginTop: '1em'}}>
<Header as='h3'>Estimate <InfoPopup content={ESTIMATE_INFO}/></Header>
<StatisticGroup>
<Statistic value={fmt(result.annual, 0)} label={`${unit} per year`}/>
<Statistic value={fmt(result.annual / 365, 1)} label={`${unit} per day, average`}/>
<Statistic value={fmt(result.worst.energyDaily, 1)} label={`${unit} per day in ${result.worst.month}`}/>
<Statistic value={fmt(result.specificYield, 0)} label='kWh per year per kW of panels'/>
</StatisticGroup>
<Table>
<Table.Header>
<Table.Row>
<Table.HeaderCell>Month</Table.HeaderCell>
<Table.HeaderCell>{unit}/day</Table.HeaderCell>
<Table.HeaderCell>{unit}/month</Table.HeaderCell>
<Table.HeaderCell>Sun hours on panel</Table.HeaderCell>
<Table.HeaderCell>Day length</Table.HeaderCell>
</Table.Row>
</Table.Header>
<Table.Body>{rows}</Table.Body>
</Table>
</Panel>;
}
function AngleSection({setNumber, s, sunlight, ready}) {
const compass = COMPASS.find(c => Number(c.value) === s.azimuth)?.value || null;
const applyOptimal = goal => setNumber('tilt', optimalTilt(modelInputs(s, sunlight), goal), s.defaultTilt);
const azimuthDefault = defaultAzimuth(s.latitude ?? 0);
if (s.tracking !== 'fixed') {
return <Hint>
A tracking array turns to follow the sun, so it has no fixed angle. Change Tracking under Panel details
to set one.
</Hint>;
}
return <>
<Hint>
Leave these alone unless your panels are already mounted. The default (tilted to your latitude, facing
the equator) is close to the best for the year. "Best for winter" suits off-grid systems that must get
through the darkest month.
</Hint>
<Grid>
<Grid.Col span={{base: 12, sm: 6}}>
<NumberInput label='Tilt (degrees from flat)' name='tilt' value={s.tilt} min={0} max={90} step={1}
onChange={v => setNumber('tilt', v, s.defaultTilt)}/>
<Group gap='xs' style={{marginTop: '0.25em'}}>
<Button type='button' size='small' disabled={!ready} onClick={() => applyOptimal('annual')}>
Best for the year
</Button>
<Button type='button' size='small' disabled={!ready} onClick={() => applyOptimal('winter')}>
Best for winter
</Button>
</Group>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<Group gap='xs' wrap='nowrap' align='flex-end'>
<NumberInput label='Facing (compass degrees)' name='az' value={s.azimuth} min={0} max={359}
step={5} style={{flex: 1}} onChange={v => setNumber('az', v, azimuthDefault)}/>
<Select aria-label='Compass direction' data={COMPASS} value={compass} placeholder='—'
style={{width: '5.5em'}} allowDeselect={false}
onChange={v => setNumber('az', v, azimuthDefault)}/>
</Group>
</Grid.Col>
</Grid>
</>;
}
function DetailsSection({set, setNumber, s}) {
const moduleType = Object.entries(MODULE_TYPES).find(([, m]) => m.gamma === s.gamma)?.[0] || 'custom';
return <>
<Hint>
The defaults suit a typical ground or pole mounted array of ordinary panels. The temperature coefficient
is on your panel's datasheet; panels lose this much power for each °C they run above 25 °C.
</Hint>
<Grid>
<Grid.Col span={{base: 12, sm: 6}}>
<Select label='Panel type'
data={[
...Object.entries(MODULE_TYPES).map(([value, m]) => ({value, label: m.label})),
{value: 'custom', label: 'Custom (from datasheet)'},
]}
value={moduleType} allowDeselect={false}
onChange={v => v !== 'custom' && setNumber('gamma', MODULE_TYPES[v].gamma,
MODULE_TYPES.standard.gamma)}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<NumberInput label='Temperature coefficient of power (%/°C)' name='gamma' value={s.gamma}
max={0} step={0.01} decimalScale={3}
onChange={v => setNumber('gamma', v, MODULE_TYPES.standard.gamma)}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<Select label='Mounting' data={Object.entries(MOUNTINGS).map(([value, m]) => ({value, label: m.label}))}
value={s.mounting} allowDeselect={false}
onChange={v => set({mount: v === 'open' ? null : v})}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<Select label='Tracking' data={Object.entries(TRACKING).map(([value, t]) => ({value, label: t.label}))}
value={s.tracking} allowDeselect={false}
onChange={v => set({track: v === 'fixed' ? null : v})}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<NumberInput label='Ground reflectance (albedo)' name='alb' value={s.albedo ?? ''}
placeholder='From the data (monthly, includes snow)' min={0} max={1} step={0.05}
decimalScale={2} onChange={v => set({alb: v === '' ? null : v})}/>
</Grid.Col>
</Grid>
</>;
}
function LossesSection({set, s}) {
const setLoss = (index, value) => {
const next = [...s.losses];
next[index] = value === '' ? LOSS_CATEGORIES[index].default : Number(value);
const isDefault = next.every((v, i) => v === DEFAULT_LOSSES[i]);
set({losses: isDefault ? null : formatMonthly(next)});
};
return <>
<Hint>
Real systems lose some power to dirt, wiring, shading and more. These are PVWatts' defaults. Raise Shading
if trees or buildings shade your panels, or Snow if snow sits on them for weeks.
</Hint>
<Grid>
{LOSS_CATEGORIES.map((c, i) => <Grid.Col key={c.key} span={{base: 6, sm: 4}}>
<NumberInput label={`${c.label} (%)`} name={`loss-${c.key}`} value={s.losses[i]}
min={0} max={100} step={0.5} onChange={v => setLoss(i, v)}/>
</Grid.Col>)}
</Grid>
<Button type='button' size='small' style={{marginTop: '0.5em'}} onClick={() => set({losses: null})}>
Reset to PVWatts defaults
</Button>
</>;
}
function InverterSection({set, setNumber, s}) {
return <>
<Hint>
Leave this off if your panels charge batteries through a charge controller. Turn it on to see AC output
from a grid-tie or hybrid inverter.
</Hint>
<Toggle label='Include an inverter (AC output)' checked={s.inverterOn}
onChange={e => set({inv: e.currentTarget.checked ? '1' : null})}/>
{s.inverterOn && <Grid style={{marginTop: '0.5em'}}>
<Grid.Col span={{base: 6}}>
<NumberInput label='AC rating (kW)' name='iac' value={s.acRating} min={0} step={0.1} decimalScale={2}
onChange={v => setNumber('iac', v, roundDigits(s.kwp / 1.2, 2))}/>
</Grid.Col>
<Grid.Col span={{base: 6}}>
<NumberInput label='Efficiency (%)' name='ieff' value={s.efficiency} min={50} max={100} step={0.5}
onChange={v => setNumber('ieff', v, INVERTER_NOMINAL_EFFICIENCY)}/>
</Grid.Col>
</Grid>}
</>;
}
function OwnDataSection({set, s, nasa}) {
const unit = s.fahrenheit ? '°F' : '°C';
// Start from the NASA values, when available, so the user edits rather than types all 24.
const enable = on => {
if (!on) {
set({ghi: null, temp: null});
return;
}
const ghi = nasa ? nasa.ghi.map(v => (v === null ? '' : roundDigits(v, 2))) : Array(12).fill('');
const temp = nasa
? nasa.temperature.map(v => (v === null ? '' : roundDigits(s.fahrenheit ? cToF(v) : v, 1)))
: Array(12).fill('');
// An all-blank list is still written for sunlight, so the section stays on for typing.
set({ghi: formatMonthly(ghi), temp: nasa ? formatMonthly(temp) : null});
};
// Convert the temperatures with the unit, so they stay the same physical temperatures.
const switchUnits = toFahrenheit => {
const temps = s.temperatureInput.map(v => (v === null ? null
: roundDigits(toFahrenheit ? cToF(v) : fToC(v), 1)));
set({tu: toFahrenheit ? 'f' : null, temp: temps.some(v => v !== null) ? formatMonthly(temps) : null});
};
const setMonth = (key, values, index, value) => {
const next = [...values];
next[index] = value === '' ? null : value;
set({[key]: formatMonthly(next)});
};
const rows = MONTH_NAMES.map((name, i) => <Table.Row key={name}>
<Table.Cell>{name}</Table.Cell>
<Table.Cell>
<NumberInput name={`ghi-${i}`} aria-label={`${name} sunlight`} value={s.ghi[i] ?? ''} min={0}
step={0.1} hideControls onChange={v => setMonth('ghi', s.ghi, i, v)}/>
</Table.Cell>
<Table.Cell>
<NumberInput name={`temp-${i}`} aria-label={`${name} temperature`}
placeholder={s.fahrenheit ? '68' : '20'} value={s.temperatureInput[i] ?? ''} step={1}
hideControls onChange={v => setMonth('temp', s.temperatureInput, i, v)}/>
</Table.Cell>
</Table.Row>);
return <>
<Hint>
Only needed if you have better local measurements than the built-in data, for example in mountains, where
a 1° grid cell (about 69 miles across) can average very different terrain. Sunlight is global horizontal
irradiance in kWh/m²/day, the same as "peak sun hours" on flat ground. Blank temperatures use 20 °C.
</Hint>
<Group gap='lg'>
<Toggle label='Use my own sunlight data' checked={s.ownData}
onChange={e => enable(e.currentTarget.checked)}/>
{s.ownData && <Toggle label='Fahrenheit' checked={s.fahrenheit}
onChange={e => switchUnits(e.currentTarget.checked)}/>}
</Group>
{s.ownData && <Table style={{marginTop: '0.5em'}}>
<Table.Header>
<Table.Row>
<Table.HeaderCell>Month</Table.HeaderCell>
<Table.HeaderCell>Sunlight (kWh/m²/day)</Table.HeaderCell>
<Table.HeaderCell>Avg. temp ({unit})</Table.HeaderCell>
</Table.Row>
</Table.Header>
<Table.Body>{rows}</Table.Body>
</Table>}
</>;
}
export function SolarCalculator() {
const {searchParams, get, set, setNumber} = useSolarParams();
const climatology = useClimatology();
const queryString = searchParams.toString();
// eslint-disable-next-line react-hooks/exhaustive-deps
const s = React.useMemo(() => systemFromParams(get), [queryString]);
const nasa = React.useMemo(
() => (climatology.grid && s.latitude !== null && s.longitude !== null
? lookupClimatology(climatology.grid, s.latitude, s.longitude) : null),
[climatology.grid, s.latitude, s.longitude]);
const sunlight = s.ownData ? sunlightFor(s, null) : nasa;
const missing = missingInputs(s, sunlight);
const ready = missing.length === 0;
const result = React.useMemo(
() => (ready ? estimateProduction(modelInputs(s, sunlight)) : null),
// eslint-disable-next-line react-hooks/exhaustive-deps
[s, sunlight, ready]);
// Off-grid sizing works from DC production per kW of panels, whatever the inverter setting.
const worstPerKwp = React.useMemo(
() => (ready ? estimateProduction({...modelInputs(s, sunlight), kwp: 1, inverter: null}).worst : null),
// eslint-disable-next-line react-hooks/exhaustive-deps
[s, sunlight, ready]);
const tab = get('tab') === 'offgrid' ? 'offgrid' : 'estimate';
const angleTitle = s.tracking === 'fixed'
? `Panel angle: ${s.tilt}° tilt, facing ${compassLabel(s.azimuth)}`
: 'Panel angle: follows the sun';
const detailsTitle = `Panel details: ${TRACKING[s.tracking].label.toLowerCase()}, `
+ `${MOUNTINGS[s.mounting].label.toLowerCase()}`;
const waitingForData = climatology.loading && !s.ownData;
return <div>
<Header as='h1'>Solar</Header>
<p>
Estimate what solar panels will produce over a typical year. Enter a location and your panels; everything
else has sensible defaults. Settings are saved in the page address, so you can bookmark or share them.
</p>
<LocationSection get={get} set={set} s={s} climatology={climatology}/>
<PanelsSection setNumber={setNumber} s={s}/>
<Tabs value={tab} onChange={v => set({tab: v === 'offgrid' ? v : null})} keepMounted={false}
style={{marginTop: '1.5em'}}>
<Tabs.List>
<Tabs.Tab value='estimate'>Estimate</Tabs.Tab>
<Tabs.Tab value='offgrid'>Off-grid sizing</Tabs.Tab>
</Tabs.List>
<Tabs.Panel value='estimate'>
{ready && <Results result={result} s={s}/>}
{!ready && !waitingForData && <div style={{marginTop: '1em'}}>
<Message kind='info' title='Almost there'>Enter {missing.join(', ')} to see an estimate.</Message>
</div>}
</Tabs.Panel>
<Tabs.Panel value='offgrid'>
<SolarOffGrid get={get} set={set} setNumber={setNumber} s={s} worst={worstPerKwp}
temperatures={sunlight?.temperature} loading={waitingForData}/>
</Tabs.Panel>
</Tabs>
<Header as='h3' style={{marginTop: '1.5em'}}>Fine-tune (optional)</Header>
<Accordion multiple>
<Accordion.Item value='angle'>
<Accordion.Control>{angleTitle}</Accordion.Control>
<Accordion.Panel>
<AngleSection setNumber={setNumber} s={s} sunlight={sunlight} ready={ready}/>
</Accordion.Panel>
</Accordion.Item>
<Accordion.Item value='details'>
<Accordion.Control>{detailsTitle}</Accordion.Control>
<Accordion.Panel><DetailsSection set={set} setNumber={setNumber} s={s}/></Accordion.Panel>
</Accordion.Item>
<Accordion.Item value='losses'>
<Accordion.Control>System losses: {fmt(totalLoss(s.losses))}%</Accordion.Control>
<Accordion.Panel><LossesSection set={set} s={s}/></Accordion.Panel>
</Accordion.Item>
<Accordion.Item value='inverter'>
<Accordion.Control>Inverter: {s.inverterOn ? 'on (AC output)' : 'off (DC output)'}</Accordion.Control>
<Accordion.Panel><InverterSection set={set} setNumber={setNumber} s={s}/></Accordion.Panel>
</Accordion.Item>
<Accordion.Item value='own'>
<Accordion.Control>Use my own sunlight data: {s.ownData ? 'on' : 'off'}</Accordion.Control>
<Accordion.Panel><OwnDataSection set={set} s={s} nasa={nasa}/></Accordion.Panel>
</Accordion.Item>
</Accordion>
<DataAttribution/>
</div>;
}

View file

@ -0,0 +1,256 @@
import React from "react";
import {fireEvent, screen, within} from "@testing-library/react";
import {renderWithProviders} from "../../test-utils";
import {missingInputs, SolarCalculator, sunlightFor, systemFromParams} from "./SolarCalculator";
import {CalculatorsPage} from "../Calculators";
import {DEFAULT_LOSSES, estimateProduction, MODULE_TYPES} from "./solar";
import {decodeClimatology, forgetClimatology, lookupClimatology} from "./solarData";
import {mockClimatologyFetch, SUNNY_GHI, uniformClimatology} from "./solarData.fixtures";
import {roundDigits} from "../Common";
const BASE = '/more/calculators?calc=solar';
const DENVER = `${BASE}&lat=40&lon=-105`;
const params = query => {
const searchParams = new URLSearchParams(query);
return key => searchParams.get(key);
};
const currentParams = () => new URLSearchParams(window.location.search);
// The estimate appears once the data file has loaded and decoded, which can be slow on a busy machine.
const findEstimate = (unit = 'DC') => screen.findByText(`kWh ${unit} per year`, {}, {timeout: 5000});
// The annual estimate the page should show for a location, from the same uniform test data.
const expectedAnnual = (overrides = {}) => {
const grid = decodeClimatology(uniformClimatology());
const sunlight = lookupClimatology(grid, 40, -105);
return estimateProduction({latitude: 40, kwp: 0.4, tilt: 40, azimuth: 180, ...sunlight, ...overrides}).annual;
};
beforeEach(() => {
forgetClimatology();
mockClimatologyFetch();
});
describe('systemFromParams', () => {
test('defaults when the URL is empty', () => {
const s = systemFromParams(params(''));
expect(s.latitude).toBeNull();
expect(s.kwp).toBeCloseTo(0.4, 6);
expect(s.tracking).toBe('fixed');
expect(s.mounting).toBe('open');
expect(s.gamma).toBe(MODULE_TYPES.standard.gamma);
expect(s.losses).toEqual(DEFAULT_LOSSES);
expect(s.inverterOn).toBe(false);
expect(s.ownData).toBe(false);
expect(s.albedo).toBeNull();
});
test('tilt and azimuth default from the latitude', () => {
expect(systemFromParams(params('lat=40.4')).tilt).toBe(40);
expect(systemFromParams(params('lat=40.4')).azimuth).toBe(180);
expect(systemFromParams(params('lat=-33.9')).tilt).toBe(34);
expect(systemFromParams(params('lat=-33.9')).azimuth).toBe(0);
});
test('invalid values fall back instead of crashing', () => {
const s = systemFromParams(params('lat=999&track=bogus&mount=x&tilt=abc&losses=1,2,abc&alb=7&pn=-3'));
expect(s.latitude).toBeNull();
expect(s.tracking).toBe('fixed');
expect(s.mounting).toBe('open');
expect(s.tilt).toBe(0);
expect(s.losses.slice(0, 3)).toEqual([1, 2, DEFAULT_LOSSES[2]]);
expect(s.albedo).toBe(1);
expect(s.kwp).toBe(0);
});
test('own Fahrenheit temperatures are converted to Celsius', () => {
const s = systemFromParams(params('ghi=5&tu=f&temp=32,212'));
expect(s.ownData).toBe(true);
expect(s.temperature[0]).toBeCloseTo(0, 6);
expect(s.temperature[1]).toBeCloseTo(100, 6);
expect(s.temperatureInput[0]).toBe(32);
});
});
describe('sunlight and missing inputs', () => {
const grid = decodeClimatology(uniformClimatology());
test('the NASA data is used unless the user gave their own', () => {
const nasa = sunlightFor(systemFromParams(params('lat=40&lon=-105')), grid);
expect(nasa.ghi[5]).toBeCloseTo(SUNNY_GHI[5], 1);
expect(nasa.diffuse[5]).toBeCloseTo(SUNNY_GHI[5] * 0.3, 1);
const own = sunlightFor(systemFromParams(params(`lat=40&lon=-105&ghi=${Array(12).fill(1).join(',')}`)), grid);
expect(own.ghi).toEqual(Array(12).fill(1));
expect(own.diffuse).toBeNull();
});
test('a location is enough with the NASA data', () => {
const s = systemFromParams(params('lat=40&lon=-105'));
expect(missingInputs(s, sunlightFor(s, grid))).toEqual([]);
const noLon = systemFromParams(params('lat=40'));
expect(missingInputs(noLon, sunlightFor(noLon, grid))).toEqual(['a longitude']);
});
test('own data needs all 12 months but no longitude', () => {
const s = systemFromParams(params('lat=40&ghi=1,2,3'));
expect(missingInputs(s, sunlightFor(s, grid))).toEqual(['sunlight for all 12 months']);
});
});
describe('SolarCalculator', () => {
test('is linked from the Engineering group of the calculators page', () => {
renderWithProviders(<CalculatorsPage/>, {route: '/more/calculators'});
const engineering = screen.getByRole('heading', {name: 'Engineering'}).closest('.wrolpi-header').parentElement;
const link = within(engineering).getByRole('link', {name: /Solar/});
expect(link).toHaveAttribute('href', '/more/calculators?calc=solar');
});
test('asks for a location', async () => {
renderWithProviders(<SolarCalculator/>, {route: BASE});
expect(await screen.findByText('Enter a latitude, a longitude to see an estimate.', {}, {timeout: 5000}))
.toBeInTheDocument();
});
test('a location alone gives an estimate from the built-in data', async () => {
renderWithProviders(<SolarCalculator/>, {route: DENVER});
expect(await findEstimate()).toBeInTheDocument();
expect(screen.getByText(`${roundDigits(expectedAnnual(), 0)}`)).toBeInTheDocument();
// The attribution is only at the bottom of the page.
expect(screen.queryByText(/Sunlight and temperature/)).not.toBeInTheDocument();
expect(global.fetch).toHaveBeenCalledWith(expect.stringMatching(/\/data\/solar-climatology\.bin$/));
});
test('the data attribution is at the bottom of the page', async () => {
renderWithProviders(<SolarCalculator/>, {route: DENVER});
await findEstimate();
const heading = screen.getByRole('heading', {name: 'Data attribution'});
const section = screen.getByRole('region', {name: 'Data attribution'});
expect(within(section).getByText(/Prediction Of Worldwide Energy Resources/)).toBeInTheDocument();
expect(within(section).getByText(/WROLPi modified the data/)).toBeInTheDocument();
expect(within(section).getByText(/CC BY 4\.0/)).toBeInTheDocument();
// After everything else, including the last Fine-tune section.
const lastControl = screen.getByText('Use my own sunlight data: off');
expect(lastControl.compareDocumentPosition(heading) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
});
test('typing writes the URL and leaves defaults out', async () => {
renderWithProviders(<SolarCalculator/>, {route: BASE});
fireEvent.change(screen.getByLabelText('Latitude'), {target: {value: '40°0\'54"N'}});
expect(currentParams().get('lat')).toBe('40°0\'54"N');
expect(currentParams().get('calc')).toBe('solar');
fireEvent.change(screen.getByLabelText('Panels'), {target: {value: '4'}});
expect(currentParams().get('pn')).toBe('4');
fireEvent.change(screen.getByLabelText('Panels'), {target: {value: '1'}});
expect(currentParams().has('pn')).toBe(false);
});
describe('use my current location', () => {
const withGeolocation = getCurrentPosition => {
Object.defineProperty(window, 'isSecureContext', {value: true, configurable: true});
Object.defineProperty(navigator, 'geolocation', {value: {getCurrentPosition}, configurable: true});
};
afterEach(() => {
delete window.isSecureContext;
delete navigator.geolocation;
});
test('fills in the location, rounded to 4 decimals', async () => {
withGeolocation(success => success({coords: {latitude: 39.7392358, longitude: -104.990251}}));
renderWithProviders(<SolarCalculator/>, {route: BASE});
fireEvent.click(screen.getByRole('button', {name: 'Use my current location'}));
expect(currentParams().get('lat')).toBe('39.7392');
expect(currentParams().get('lon')).toBe('-104.9903');
expect(screen.getByLabelText('Latitude')).toHaveValue('39.7392');
expect(await findEstimate()).toBeInTheDocument();
});
test('explains a denied permission', () => {
withGeolocation((success, failure) => failure({code: 1, message: 'User denied Geolocation'}));
renderWithProviders(<SolarCalculator/>, {route: BASE});
fireEvent.click(screen.getByRole('button', {name: 'Use my current location'}));
expect(screen.getByText('Could not get your location')).toBeInTheDocument();
expect(screen.getByText(/Location permission was denied/)).toBeInTheDocument();
expect(currentParams().has('lat')).toBe(false);
});
test('is disabled without HTTPS', () => {
Object.defineProperty(window, 'isSecureContext', {value: false, configurable: true});
renderWithProviders(<SolarCalculator/>, {route: BASE});
expect(screen.getByRole('button', {name: 'Use my current location'})).toBeDisabled();
});
});
test('an invalid latitude is flagged', () => {
renderWithProviders(<SolarCalculator/>, {route: `${BASE}&lat=95`});
expect(screen.getByText('Latitude must be between -90 and 90')).toBeInTheDocument();
});
test('the inverter switches the results to AC', async () => {
renderWithProviders(<SolarCalculator/>, {route: DENVER});
await findEstimate();
fireEvent.click(screen.getByText('Inverter: off (DC output)'));
fireEvent.click(screen.getByLabelText('Include an inverter (AC output)'));
expect(currentParams().get('inv')).toBe('1');
expect(screen.getByText('kWh AC per year')).toBeInTheDocument();
});
test('best-for-the-year replaces a poor tilt', async () => {
renderWithProviders(<SolarCalculator/>, {route: `${DENVER}&tilt=10`});
await findEstimate();
expect(screen.getByText('Panel angle: 10° tilt, facing S')).toBeInTheDocument();
fireEvent.click(screen.getByText('Panel angle: 10° tilt, facing S'));
fireEvent.click(await screen.findByRole('button', {name: 'Best for the year'}));
// A result of 40 equals the default (the latitude), so it is left out of the URL.
const tilt = Number(currentParams().get('tilt') ?? 40);
expect(tilt).toBeGreaterThanOrEqual(30);
expect(tilt).toBeLessThanOrEqual(44);
});
test('own sunlight data starts from the built-in values and replaces them', async () => {
renderWithProviders(<SolarCalculator/>, {route: DENVER});
await findEstimate();
fireEvent.click(screen.getByText('Use my own sunlight data: off'));
fireEvent.click(screen.getByLabelText('Use my own sunlight data'));
const ghi = currentParams().get('ghi').split(',').map(Number);
ghi.forEach((v, i) => expect(v).toBeCloseTo(SUNNY_GHI[i], 1));
expect(screen.getByText('Using your own sunlight data.')).toBeInTheDocument();
expect(screen.getByLabelText('Jun sunlight')).toHaveValue(`${ghi[5]}`);
// Halving June's sunlight lowers the estimate.
fireEvent.change(screen.getByLabelText('Jun sunlight'), {target: {value: `${ghi[5] / 2}`}});
expect(currentParams().get('ghi').split(',')[5]).toBe(`${ghi[5] / 2}`);
fireEvent.click(screen.getByLabelText('Use my own sunlight data'));
expect(currentParams().has('ghi')).toBe(false);
expect(currentParams().has('temp')).toBe(false);
});
test('switching to Fahrenheit converts your temperatures instead of relabeling them', async () => {
const temps = '0,10,,,,,,,,,,';
renderWithProviders(<SolarCalculator/>,
{route: `${DENVER}&ghi=${SUNNY_GHI.join(',')}&temp=${temps}`});
await findEstimate();
fireEvent.click(screen.getByText('Use my own sunlight data: on'));
fireEvent.click(screen.getByLabelText('Fahrenheit'));
expect(currentParams().get('tu')).toBe('f');
expect(currentParams().get('temp')).toBe('32,50,,,,,,,,,,');
expect(screen.getByLabelText('Jan temperature')).toHaveValue('32');
fireEvent.click(screen.getByLabelText('Fahrenheit'));
expect(currentParams().has('tu')).toBe(false);
expect(currentParams().get('temp')).toBe(temps);
});
test('a failed download explains how to continue', async () => {
forgetClimatology();
global.fetch = jest.fn(() => Promise.resolve({ok: false, status: 404}));
renderWithProviders(<SolarCalculator/>, {route: DENVER});
expect(await screen.findByText('Sunlight data could not be loaded', {}, {timeout: 5000})).toBeInTheDocument();
expect(screen.getByText('Enter a location with sunlight data to see an estimate.')).toBeInTheDocument();
});
});

View file

@ -0,0 +1,350 @@
import React from "react";
import {Button, Grid, Group, Header, IconButton, Loader, Message, NumberInput, Panel, Select, Statistic,
StatisticGroup, Table, TextInput, Toggle} from "../ui";
import {roundDigits} from "../Common";
import {
BATTERIES,
batteryWarnings,
CONTROLLERS,
controllerSizing,
dailyLoadWh,
defaultColdest,
formatLoads,
leadAcidCapacityFactor,
loadWh,
parseLoads,
requiredKwp,
sizeBattery,
SYSTEM_VOLTAGES,
} from "./offgrid";
// Defaults are typical of a 400 W residential panel; the user should copy their panel's label.
const PANEL_DEFAULTS = {voc: 37, vmp: 31, isc: 13.8, bvoc: -0.27};
const DEFAULT_LOAD_INVERTER_EFFICIENCY = 90;
const DEFAULT_DAYS = 2;
const DEFAULT_CONTROLLER_MAX_VOLTAGE = 100;
// Above this, a higher system voltage needs a much smaller controller and thinner wire.
const HIGH_CURRENT = 60;
const numberOr = (value, fallback) => {
if (value === null || value === undefined || value === '') {
return fallback;
}
const n = Number(value);
return Number.isFinite(n) ? n : fallback;
};
const fmt = (value, digits = 0) => (Number.isFinite(value) ? `${roundDigits(value, digits)}` : '—');
const Hint = ({children}) => <p style={{fontSize: '0.9em', opacity: 0.75, marginTop: 0}}>{children}</p>;
// The off-grid inputs, read from the URL like the rest of the calculator. Invalid values fall back.
export function offGridFromParams(get, monthlyTemperatures) {
const battery = BATTERIES[get('bat')] ? get('bat') : 'lfp';
const systemVoltage = SYSTEM_VOLTAGES.includes(Number(get('sv'))) ? Number(get('sv')) : 12;
const controller = CONTROLLERS[get('ctl')] ? get('ctl') : 'mppt';
const coldestMonthly = (monthlyTemperatures || []).filter(Number.isFinite).length
? Math.min(...monthlyTemperatures.filter(Number.isFinite)) : null;
const defaultTmin = defaultColdest(monthlyTemperatures);
return {
loads: parseLoads(get('load')),
acLoads: get('ac') !== '0',
loadInverterEfficiency: numberOr(get('leff'), DEFAULT_LOAD_INVERTER_EFFICIENCY),
battery,
systemVoltage,
dod: numberOr(get('dod'), BATTERIES[battery].dod),
days: Math.max(0, numberOr(get('aut'), DEFAULT_DAYS)),
outdoors: get('bout') === '1',
controller,
voc: numberOr(get('voc'), PANEL_DEFAULTS.voc),
vmp: numberOr(get('vmp'), PANEL_DEFAULTS.vmp),
isc: numberOr(get('isc'), PANEL_DEFAULTS.isc),
// Always negative for silicon panels; a missing minus sign must not hide the cold-weather rise.
betaVoc: -Math.abs(numberOr(get('bvoc'), PANEL_DEFAULTS.bvoc)),
series: Math.max(1, Math.round(numberOr(get('ns'), 1))),
controllerMaxVoltage: numberOr(get('cmax'), DEFAULT_CONTROLLER_MAX_VOLTAGE),
coldestMonthly,
defaultTmin,
coldest: numberOr(get('tmin'), defaultTmin),
};
}
/**
* Everything the off-grid tab shows, from its inputs.
*
* @param {Object} o offGridFromParams()
* @param {Object} worst the worst month for a 1 kW array: {month, energyDaily}
* @param {number} panelWatts
* @param {number} panelCount
*/
export function offGridResults(o, worst, panelWatts, panelCount) {
const devicesWh = dailyLoadWh(o.loads);
// AC devices draw through an inverter, so the batteries supply more than the devices use.
const batteryLoadWh = o.acLoads && o.loadInverterEfficiency > 0
? devicesWh / (o.loadInverterEfficiency / 100) : devicesWh;
const capacityFactor = o.outdoors && o.battery === 'lead' ? leadAcidCapacityFactor(o.coldestMonthly) : 1;
const bank = sizeBattery({
loadWh: batteryLoadWh, days: o.days, dod: o.dod, voltage: o.systemVoltage, capacityFactor,
});
const sizing = controllerSizing({
panelCount, panelWatts, series: o.series, voc: o.voc, vmp: o.vmp, isc: o.isc, betaVoc: o.betaVoc,
coldest: o.coldest, controller: o.controller, controllerMaxVoltage: o.controllerMaxVoltage,
systemVoltage: o.systemVoltage,
});
// A PWM controller wastes part of every panel's output, so more panels are needed.
const worstPerKwp = worst ? worst.energyDaily * sizing.pwmFraction : null;
const kwp = requiredKwp(batteryLoadWh, worstPerKwp, BATTERIES[o.battery].efficiency);
const panelsNeeded = kwp && panelWatts > 0 ? Math.ceil(kwp * 1000 / panelWatts) : null;
const yourArrayWh = worstPerKwp !== null
? worstPerKwp * (panelCount * panelWatts / 1000) * 1000 * BATTERIES[o.battery].efficiency / 100 : null;
return {
devicesWh, batteryLoadWh, bank, kwp, panelsNeeded, sizing,
coverage: batteryLoadWh > 0 && yourArrayWh !== null ? yourArrayWh / batteryLoadWh : null,
warnings: batteryWarnings({battery: o.battery, outdoors: o.outdoors, coldestMonthly: o.coldestMonthly}),
};
}
function LoadsSection({o, r, set, setNumber}) {
const rows = o.loads.length ? o.loads : [{name: '', watts: '', hours: ''}];
const write = next => set({load: next.some(row => row.name || row.watts || row.hours) ? formatLoads(next) : null});
const update = (index, field, value) => write(rows.map((row, i) => (i === index ? {...row, [field]: value} : row)));
const remove = index => write(rows.filter((_, i) => i !== index));
return <>
<Header as='h3'>Your devices</Header>
<Hint>
List what you will run and for how long each day. The watts are on the device's label or power supply;
a refrigerator cycles on and off, so use about a third of its rating for 24 hours.
</Hint>
<Table>
<Table.Header>
<Table.Row>
<Table.HeaderCell>Device</Table.HeaderCell>
<Table.HeaderCell>Watts</Table.HeaderCell>
<Table.HeaderCell>Hours per day</Table.HeaderCell>
<Table.HeaderCell>Wh per day</Table.HeaderCell>
<Table.HeaderCell/>
</Table.Row>
</Table.Header>
<Table.Body>
{rows.map((row, i) => <Table.Row key={i}>
<Table.Cell>
<TextInput aria-label={`Device ${i + 1} name`} placeholder='Refrigerator' value={row.name}
onChange={e => update(i, 'name', e.target.value)}/>
</Table.Cell>
<Table.Cell>
<NumberInput aria-label={`Device ${i + 1} watts`} placeholder='50' value={row.watts} min={0}
hideControls onChange={v => update(i, 'watts', v)}/>
</Table.Cell>
<Table.Cell>
<NumberInput aria-label={`Device ${i + 1} hours`} placeholder='24' value={row.hours} min={0}
max={24} hideControls onChange={v => update(i, 'hours', v)}/>
</Table.Cell>
<Table.Cell>{fmt(loadWh(row))}</Table.Cell>
<Table.Cell>
<IconButton icon='trash' label={`Remove device ${i + 1}`} onClick={() => remove(i)}/>
</Table.Cell>
</Table.Row>)}
</Table.Body>
</Table>
<Group gap='md' style={{marginTop: '0.5em'}}>
<Button type='button' size='small' icon='add'
onClick={() => write([...rows, {name: '', watts: '', hours: ''}])}>
Add device
</Button>
<span><b>{fmt(r.devicesWh)} Wh</b> per day</span>
</Group>
<Group gap='lg' align='flex-end' style={{marginTop: '0.75em'}}>
<Toggle label='Devices run on AC, through an inverter' checked={o.acLoads}
onChange={e => set({ac: e.currentTarget.checked ? null : '0'})}/>
{o.acLoads && <NumberInput label='Inverter efficiency (%)' name='leff' value={o.loadInverterEfficiency}
min={50} max={100} step={1} style={{width: '12em'}}
onChange={v => setNumber('leff', v, DEFAULT_LOAD_INVERTER_EFFICIENCY)}/>}
</Group>
</>;
}
function BatterySection({o, set, setNumber}) {
return <>
<Header as='h3' style={{marginTop: '1.5em'}}>Batteries</Header>
<Hint>
Days without sun is how long the batteries must carry your devices through cloudy weather. Two or three
days is typical. Larger systems (over about 1,500 W of panels) usually use 24 or 48 volts.
</Hint>
<Grid>
<Grid.Col span={{base: 12, sm: 6}}>
<Select label='Battery type'
data={Object.entries(BATTERIES).map(([value, b]) => ({value, label: b.label}))}
value={o.battery} allowDeselect={false}
onChange={v => set({bat: v === 'lfp' ? null : v, dod: null})}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<Select label='System voltage' data={SYSTEM_VOLTAGES.map(v => ({value: `${v}`, label: `${v} V`}))}
value={`${o.systemVoltage}`} allowDeselect={false}
onChange={v => set({sv: v === '12' ? null : v})}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Days without sun' name='aut' value={o.days} min={0} step={0.5}
onChange={v => setNumber('aut', v, DEFAULT_DAYS)}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}}>
<NumberInput label='Usable depth of discharge (%)' name='dod' value={o.dod} min={10} max={100} step={5}
onChange={v => setNumber('dod', v, BATTERIES[o.battery].dod)}/>
</Grid.Col>
<Grid.Col span={{base: 12, sm: 6}} style={{display: 'flex', alignItems: 'flex-end'}}>
<Toggle label='Batteries are outdoors or unheated' checked={o.outdoors}
onChange={e => set({bout: e.currentTarget.checked ? '1' : null})}/>
</Grid.Col>
</Grid>
</>;
}
function NeedsSection({o, r, worst, s, setNumber}) {
if (!(r.devicesWh > 0)) {
return <div style={{marginTop: '1em'}}>
<Message kind='info' title='Add your devices'>List your devices above to size the system.</Message>
</div>;
}
const coverage = r.coverage;
return <Panel style={{marginTop: '1.5em'}}>
<Header as='h3'>What you need</Header>
<StatisticGroup>
<Statistic value={fmt(r.panelsNeeded)} label={`${s.panelWatts} W panels for ${worst.month}`}/>
<Statistic value={fmt(r.bank?.ah)} label={`Ah of ${o.systemVoltage} V battery`}/>
<Statistic value={fmt(r.bank ? r.bank.nominalWh / 1000 : null, 1)} label='kWh of battery'/>
<Statistic value={fmt(r.batteryLoadWh)} label='Wh per day from the batteries'/>
</StatisticGroup>
<p>
Sized for {worst.month}, your darkest month, when each kW of panels makes about{' '}
{fmt(worst.energyDaily, 1)} kWh a day at the angle chosen under Fine-tune.
{o.controller === 'pwm' && r.sizing.pwmFraction > 0 && ` The PWM controller uses only `
+ `${fmt(r.sizing.pwmFraction * 100)}% of the panels' power, so more panels are needed.`}
{o.controller === 'pwm' && r.sizing.pwmFraction === 0 && ' These panels cannot charge this battery '
+ 'through a PWM controller; see the warning below.'}
</p>
{coverage !== null && <p>
Your {s.panelCount} panel{s.panelCount === 1 ? '' : 's'} would supply about{' '}
<b>{fmt(Math.min(coverage, 9.99) * 100)}%</b> of your use in {worst.month}.{' '}
{r.panelsNeeded !== s.panelCount && r.panelsNeeded > 0 &&
<Button type='button' size='small' onClick={() => setNumber('pn', r.panelsNeeded, 1)}>
Use {r.panelsNeeded} panels
</Button>}
</p>}
{r.warnings.map(w => <Message key={w} kind='warning' title='Battery'>{w}</Message>)}
</Panel>;
}
function ControllerSection({o, r, set, setNumber, temperaturesKnown}) {
const z = r.sizing;
const panelField = (key, label, value, fallback, step, max) => <Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label={label} name={key} value={value} step={step} decimalScale={3} max={max}
onChange={v => setNumber(key, v, fallback)}/>
</Grid.Col>;
return <>
<Header as='h3' style={{marginTop: '1.5em'}}>Charge controller and wiring</Header>
<Hint>
Copy these from the label on the back of your panel. Panels make a higher voltage when cold; too many in
series can exceed the controller's limit on a cold morning and destroy it. The coldest morning starts
25 °C below your coldest monthly average, because averages hide cold snaps; enter your area's record
low if you know it.
</Hint>
<Grid>
{panelField('voc', 'Open-circuit voltage, Voc (V)', o.voc, PANEL_DEFAULTS.voc, 0.1)}
{panelField('vmp', 'Max-power voltage, Vmp (V)', o.vmp, PANEL_DEFAULTS.vmp, 0.1)}
{panelField('isc', 'Short-circuit current, Isc (A)', o.isc, PANEL_DEFAULTS.isc, 0.1)}
{panelField('bvoc', 'Voc temperature coefficient (%/°C)', o.betaVoc, PANEL_DEFAULTS.bvoc, 0.01, 0)}
<Grid.Col span={{base: 6, sm: 3}}>
<Select label='Controller type'
data={Object.entries(CONTROLLERS).map(([value, c]) => ({value, label: c.label}))}
value={o.controller} allowDeselect={false}
onChange={v => set({ctl: v === 'mppt' ? null : v})}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Controller max PV voltage (V)' name='cmax' value={o.controllerMaxVoltage} min={0}
step={10} onChange={v => setNumber('cmax', v, DEFAULT_CONTROLLER_MAX_VOLTAGE)}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Panels in series' name='ns' value={o.series} min={1} step={1}
onChange={v => setNumber('ns', v, 1)}/>
</Grid.Col>
<Grid.Col span={{base: 6, sm: 3}}>
<NumberInput label='Coldest morning (°C)' name='tmin' value={o.coldest} step={1}
onChange={v => setNumber('tmin', v, o.defaultTmin)}/>
</Grid.Col>
</Grid>
{/* The cold-morning limit depends on the site's temperatures; do not show a placeholder's verdict. */}
{!temperaturesKnown && <div style={{marginTop: '1em'}}><Loader size='xs' label='Loading temperatures'/></div>}
{temperaturesKnown && <>
<Table style={{marginTop: '1em'}}>
<Table.Body>
<Table.Row>
<Table.Cell>Strings</Table.Cell>
<Table.Cell>{z.strings} of {o.series} panel{o.series === 1 ? '' : 's'} in series</Table.Cell>
</Table.Row>
<Table.Row>
<Table.Cell>String voltage on the coldest morning</Table.Cell>
<Table.Cell>{fmt(z.stringColdVoc, 1)} V (most panels in series: {z.maxSeries})</Table.Cell>
</Table.Row>
<Table.Row>
<Table.Cell>Controller current rating, at least</Table.Cell>
<Table.Cell>{fmt(z.current)} A</Table.Cell>
</Table.Row>
</Table.Body>
</Table>
{z.overVoltage && <Message kind='error' title='Too much voltage for the controller'>
{o.series} panels in series reach {fmt(z.stringColdVoc, 1)} V on a {o.coldest} °C morning,
over the controller's {o.controllerMaxVoltage} V limit. Use at most {z.maxSeries} in series, or a
controller rated for more voltage.
</Message>}
{z.lowVoltage && <Message kind='warning' title='Too little voltage to charge'>
{o.controller === 'pwm' && z.pwmFraction === 0
? `${o.series} panel${o.series === 1 ? '' : 's'} in series cannot reach the ${o.systemVoltage} V `
+ 'battery\'s charging voltage, so a PWM controller cannot charge it at all.'
: `On a hot day ${o.series} panel${o.series === 1 ? '' : 's'} in series may not stay far enough `
+ 'above the battery\'s charging voltage for the controller to charge well.'}
{' '}Put more panels in series, or use a lower system voltage.
</Message>}
{z.unevenStrings && <Message kind='warning' title='Uneven strings'>
Your panel count does not divide evenly into strings of {o.series}. Every string should have the same
number of panels.
</Message>}
{o.controller === 'mppt' && z.current > HIGH_CURRENT && o.systemVoltage < 48 &&
<Message kind='info' title='High current'>
{fmt(z.current)} A needs a large controller and thick wire. A higher system voltage would cut the
current.
</Message>}
</>}
</>;
}
/**
* The off-grid sizing tab.
*
* @param {Object} worst the worst month for a 1 kW DC array, or null when there is no estimate
* @param {number[]} temperatures monthly average temperatures at the site, °C
* @param {boolean} loading the built-in sunlight and temperature data is still downloading
*/
export function SolarOffGrid({get, set, setNumber, s, worst, temperatures, loading}) {
const o = offGridFromParams(get, temperatures);
const r = offGridResults(o, worst, s.panelWatts, s.panelCount);
return <div style={{marginTop: '1em'}}>
<Hint>
Size a battery system to run your devices through the darkest month. Production here is DC; the
Inverter section under Fine-tune is for grid-tie systems and does not apply.
</Hint>
<LoadsSection o={o} r={r} set={set} setNumber={setNumber}/>
<BatterySection o={o} set={set} setNumber={setNumber}/>
{worst && <NeedsSection o={o} r={r} worst={worst} s={s} setNumber={setNumber}/>}
{!worst && loading && <div style={{marginTop: '1em'}}><Loader size='xs' label='Loading sunlight data'/></div>}
{!worst && !loading && <div style={{marginTop: '1em'}}>
<Message kind='info' title='Almost there'>Enter a location above to size the system.</Message>
</div>}
<ControllerSection o={o} r={r} set={set} setNumber={setNumber}
temperaturesKnown={!loading || get('tmin') !== null}/>
</div>;
}

View file

@ -0,0 +1,173 @@
import React from "react";
import {fireEvent, screen} from "@testing-library/react";
import {renderWithProviders} from "../../test-utils";
import {offGridFromParams, offGridResults} from "./SolarOffGrid";
import {SolarCalculator} from "./SolarCalculator";
import {forgetClimatology} from "./solarData";
import {mockClimatologyFetch} from "./solarData.fixtures";
const params = query => {
const searchParams = new URLSearchParams(query);
return key => searchParams.get(key);
};
const currentParams = () => new URLSearchParams(window.location.search);
const TEMPS = [-1, 0, 5, 10, 15, 20, 24, 22, 18, 10, 4, -1];
const DEC = {month: 'Dec', energyDaily: 3};
describe('offGridFromParams', () => {
test('defaults', () => {
const o = offGridFromParams(params(''), TEMPS);
expect(o.loads).toEqual([]);
expect(o.acLoads).toBe(true);
expect(o.battery).toBe('lfp');
expect(o.dod).toBe(80);
expect(o.systemVoltage).toBe(12);
expect(o.days).toBe(2);
expect(o.controller).toBe('mppt');
expect(o.series).toBe(1);
expect(o.coldestMonthly).toBe(-1);
expect(o.coldest).toBe(-26);
});
test('the depth of discharge follows the battery type unless set', () => {
expect(offGridFromParams(params('bat=lead'), TEMPS).dod).toBe(50);
expect(offGridFromParams(params('bat=lead&dod=60'), TEMPS).dod).toBe(60);
});
test('a Voc coefficient without its minus sign is read as negative, even from a shared link', () => {
expect(offGridFromParams(params('bvoc=0.27'), TEMPS).betaVoc).toBe(-0.27);
expect(offGridFromParams(params('bvoc=-0.3'), TEMPS).betaVoc).toBe(-0.3);
});
test('invalid values fall back', () => {
const o = offGridFromParams(params('bat=x&sv=13&ctl=y&ns=0&aut=-1'), null);
expect(o.battery).toBe('lfp');
expect(o.systemVoltage).toBe(12);
expect(o.controller).toBe('mppt');
expect(o.series).toBe(1);
expect(o.days).toBe(0);
expect(o.coldest).toBe(-20);
});
});
describe('offGridResults', () => {
// 1000 Wh/day of AC devices through a 90% inverter, LiFePO4 at 80%, 2 days, 12 V.
const o = offGridFromParams(params('load=Stuff~100~10'), TEMPS);
test('battery and panels for the worst month', () => {
const r = offGridResults(o, DEC, 400, 1);
expect(r.devicesWh).toBe(1000);
expect(r.batteryLoadWh).toBeCloseTo(1111.1, 1);
expect(r.bank.nominalWh).toBeCloseTo(2777.8, 1);
expect(r.bank.ah).toBeCloseTo(231.5, 1);
// 1.111 kWh / (3 kWh/kWp x 95%) = 0.390 kW: one 400 W panel.
expect(r.kwp).toBeCloseTo(0.3899, 4);
expect(r.panelsNeeded).toBe(1);
// One 400 W panel: 3 x 0.4 x 95% = 1.14 kWh for 1.111 kWh of use.
expect(r.coverage).toBeCloseTo(1.026, 3);
});
test('DC devices skip the inverter loss', () => {
const dc = offGridResults(offGridFromParams(params('load=Stuff~100~10&ac=0'), TEMPS), DEC, 400, 1);
expect(dc.batteryLoadWh).toBe(1000);
});
test('a PWM controller needs more panels', () => {
const pwm = offGridResults(offGridFromParams(params('load=Stuff~100~10&ctl=pwm'), TEMPS), DEC, 400, 1);
// Only 14.4 / 31 of each panel is used: 0.390 / 0.4645 = 0.839 kW, three panels.
expect(pwm.panelsNeeded).toBe(3);
});
test('an outdoor lead-acid bank is enlarged for the cold', () => {
const indoor = offGridResults(offGridFromParams(params('load=Stuff~100~10&bat=lead'), TEMPS), DEC, 400, 1);
const outdoor = offGridResults(offGridFromParams(params('load=Stuff~100~10&bat=lead&bout=1'), TEMPS),
DEC, 400, 1);
// Coldest month -1 °C: 79% of rated capacity.
expect(outdoor.bank.nominalWh).toBeCloseTo(indoor.bank.nominalWh / 0.79, 1);
expect(outdoor.warnings).toHaveLength(1);
});
test('nothing to size without devices or an estimate', () => {
const none = offGridResults(offGridFromParams(params(''), TEMPS), DEC, 400, 1);
expect(none.bank).toBeNull();
expect(none.panelsNeeded).toBeNull();
expect(offGridResults(o, null, 400, 1).panelsNeeded).toBeNull();
});
});
describe('the off-grid tab', () => {
const OFFGRID = '/more/calculators?calc=solar&lat=40&lon=-105&tab=offgrid';
beforeEach(() => {
forgetClimatology();
mockClimatologyFetch();
});
const findNeeds = () => screen.findByText('What you need', {}, {timeout: 5000});
test('devices are added to the URL and the system is sized', async () => {
renderWithProviders(<SolarCalculator/>, {route: OFFGRID});
expect(await screen.findByText('Add your devices', {}, {timeout: 5000})).toBeInTheDocument();
fireEvent.change(screen.getByLabelText('Device 1 name'), {target: {value: 'Fridge'}});
fireEvent.change(screen.getByLabelText('Device 1 watts'), {target: {value: '50'}});
fireEvent.change(screen.getByLabelText('Device 1 hours'), {target: {value: '24'}});
expect(currentParams().get('load')).toBe('Fridge~50~24');
expect(await findNeeds()).toBeInTheDocument();
expect(screen.getByText('1200 Wh')).toBeInTheDocument();
fireEvent.click(screen.getByRole('button', {name: 'Add device'}));
expect(screen.getByLabelText('Device 2 name')).toBeInTheDocument();
fireEvent.click(screen.getByRole('button', {name: 'Remove device 2'}));
expect(screen.queryByLabelText('Device 2 name')).not.toBeInTheDocument();
});
test('the suggested panel count can be applied', async () => {
renderWithProviders(<SolarCalculator/>, {route: `${OFFGRID}&load=Heater~500~10`});
await findNeeds();
const button = screen.getByRole('button', {name: /^Use \d+ panels$/});
const suggested = button.textContent.match(/\d+/)[0];
fireEvent.click(button);
expect(currentParams().get('pn')).toBe(suggested);
});
test('too many panels in series warns before it destroys the controller', async () => {
renderWithProviders(<SolarCalculator/>, {route: `${OFFGRID}&load=Fridge~50~24&ns=3&pn=3`});
await findNeeds();
expect(screen.getByText('Too much voltage for the controller')).toBeInTheDocument();
});
test('PWM panels that cannot reach a 48 V battery say so instead of sizing a system', async () => {
renderWithProviders(<SolarCalculator/>, {route: `${OFFGRID}&load=Fridge~50~24&ctl=pwm&sv=48`});
await findNeeds();
expect(screen.getByText('Too little voltage to charge')).toBeInTheDocument();
expect(screen.getByText(/a PWM controller cannot charge it at all/)).toBeInTheDocument();
expect(screen.queryByRole('button', {name: /^Use \d+ panels$/})).not.toBeInTheDocument();
});
test('while the data downloads, it shows a loader and no cold-morning verdict', async () => {
forgetClimatology();
global.fetch = jest.fn(() => new Promise(() => {
}));
renderWithProviders(<SolarCalculator/>, {route: `${OFFGRID}&load=Fridge~50~24`});
// One beside the location, one in place of the sizing.
expect(screen.getAllByLabelText('Loading sunlight data')).toHaveLength(2);
expect(screen.queryByText('Almost there')).not.toBeInTheDocument();
expect(screen.queryByText('String voltage on the coldest morning')).not.toBeInTheDocument();
});
test('a coldest morning the user typed is checked even while the data downloads', async () => {
forgetClimatology();
global.fetch = jest.fn(() => new Promise(() => {
}));
renderWithProviders(<SolarCalculator/>, {route: `${OFFGRID}&load=Fridge~50~24&tmin=-30`});
expect(screen.getByText('String voltage on the coldest morning')).toBeInTheDocument();
});
test('the tab is remembered in the URL', async () => {
renderWithProviders(<SolarCalculator/>, {route: '/more/calculators?calc=solar&lat=40&lon=-105'});
fireEvent.click(screen.getByRole('tab', {name: 'Off-grid sizing'}));
expect(currentParams().get('tab')).toBe('offgrid');
fireEvent.click(screen.getByRole('tab', {name: 'Estimate'}));
expect(currentParams().has('tab')).toBe(false);
});
});

View file

@ -0,0 +1,210 @@
// Pure calculation functions for the Solar calculator's off-grid sizing. No React here.
//
// Off-grid systems are sized for the worst month, not the yearly average: the battery has to carry the
// loads through a run of cloudy days, and the panels have to refill it in the darkest month.
//
// Energy is Wh, power W, voltage V, current A, temperature °C, percentages 0-100.
const clamp = (x, lo, hi) => Math.min(hi, Math.max(lo, x));
// ---------------------------------------------------------------------------
// Loads
// ---------------------------------------------------------------------------
// Devices are stored in the URL as "name~watts~hours" joined by "|".
const ROW_SEPARATOR = '|';
const FIELD_SEPARATOR = '~';
export function parseLoads(text) {
if (!text) {
return [];
}
return text.split(ROW_SEPARATOR).map(row => {
const [name = '', watts = '', hours = ''] = row.split(FIELD_SEPARATOR);
return {name, watts, hours};
});
}
export function formatLoads(rows) {
// The separators cannot appear inside a name.
const clean = value => `${value ?? ''}`.replace(/[|~]/g, ' ');
return rows.map(r => [clean(r.name), clean(r.watts), clean(r.hours)].join(FIELD_SEPARATOR)).join(ROW_SEPARATOR);
}
// Watt-hours a device uses per day; blank or invalid values count as nothing.
export function loadWh({watts, hours}) {
const w = Number(watts), h = Number(hours);
return w > 0 && h > 0 ? w * Math.min(h, 24) : 0;
}
export function dailyLoadWh(rows) {
return rows.reduce((acc, row) => acc + loadWh(row), 0);
}
// ---------------------------------------------------------------------------
// Battery
// ---------------------------------------------------------------------------
export const BATTERIES = {
lfp: {
label: 'Lithium iron phosphate (LiFePO4)',
// Usable depth of discharge, and round-trip efficiency.
dod: 80, efficiency: 95,
},
lead: {
label: 'Lead-acid (flooded or AGM)',
// Discharging below half shortens a lead-acid battery's life sharply.
dod: 50, efficiency: 85,
},
};
export const SYSTEM_VOLTAGES = [12, 24, 48];
// Charging voltage of a battery bank, used to judge PWM controllers and MPPT headroom.
export const chargeVoltage = systemVoltage => systemVoltage * 1.2;
// Share of rated capacity a cold lead-acid battery delivers: about 80% at 0 °C and 60% at -20 °C, as on
// typical manufacturer curves. Lithium batteries are not derated here; they must not be CHARGED below
// 0 °C, which is a warning instead (see batteryWarnings).
export function leadAcidCapacityFactor(temperature) {
if (!Number.isFinite(temperature) || temperature >= 20) {
return 1;
}
return clamp(1 - 0.01 * (20 - temperature), 0.5, 1);
}
/**
* Battery bank for `days` of the daily load with no sun.
*
* @returns {{usableWh: number, nominalWh: number, ah: number}}
*/
export function sizeBattery({loadWh: daily, days, dod, voltage, capacityFactor = 1}) {
if (!(daily > 0) || !(days > 0) || !(dod > 0) || !(voltage > 0)) {
return null;
}
const usableWh = daily * days;
const nominalWh = usableWh / (dod / 100) / capacityFactor;
return {usableWh, nominalWh, ah: nominalWh / voltage};
}
// ---------------------------------------------------------------------------
// Array
// ---------------------------------------------------------------------------
/**
* kW of panels needed to supply the daily load in the worst month.
*
* @param {number} loadWh daily energy the battery bank must supply
* @param {number} worstDailyPerKwp DC kWh per day per kW of panels in the worst month
* @param {number} batteryEfficiency round-trip efficiency, %; conservatively applied to all energy
*/
export function requiredKwp(loadWh, worstDailyPerKwp, batteryEfficiency) {
if (!(loadWh > 0) || !(worstDailyPerKwp > 0) || !(batteryEfficiency > 0)) {
return null;
}
return (loadWh / 1000) / (worstDailyPerKwp * batteryEfficiency / 100);
}
// ---------------------------------------------------------------------------
// Charge controller and strings
// ---------------------------------------------------------------------------
export const CONTROLLERS = {
mppt: {label: 'MPPT'},
pwm: {label: 'PWM'},
};
// A PWM controller connects a string of `series` panels straight to the battery, holding it at the battery's
// charging voltage instead of its best voltage (Vmp), so only this share of its power is used:
// - charging voltage at or above the string's Voc: no current flows, nothing is delivered;
// - between Vmp and Voc: the current falls from its maximum toward zero at Voc (taken as linear);
// - below Vmp: full current at the lower voltage, charging voltage / Vmp of the power.
export function pwmUsableFraction(vmp, voc, series, systemVoltage) {
const stringVmp = vmp * series, stringVoc = voc * series;
if (!(stringVmp > 0) || !(stringVoc > 0)) {
return 1;
}
const charge = chargeVoltage(systemVoltage);
if (charge >= stringVoc) {
return 0;
}
if (charge > stringVmp) {
return (charge / stringVmp) * (stringVoc - charge) / (stringVoc - stringVmp);
}
return charge / stringVmp;
}
// Panel open-circuit voltage rises as it gets colder. `betaVoc` is its temperature coefficient, %/°C, which is
// negative for every silicon panel; a coefficient typed without its minus sign is treated as negative, because
// a positive one would hide the cold-weather rise that destroys controllers.
export function coldVoc(voc, betaVoc, temperature) {
return voc * (1 - (Math.abs(betaVoc) / 100) * (temperature - 25));
}
/**
* Wiring and controller checks for an array of `panelCount` panels, `series` of them per string.
*/
export function controllerSizing({
panelCount, panelWatts, series, voc, vmp, isc, betaVoc, coldest,
controller, controllerMaxVoltage, systemVoltage,
}) {
const strings = series > 0 ? Math.ceil(panelCount / series) : 0;
const arrayWatts = panelCount * panelWatts;
const stringColdVoc = coldVoc(voc, betaVoc, coldest) * series;
const maxSeries = Math.floor(controllerMaxVoltage / coldVoc(voc, betaVoc, coldest));
let current;
if (controller === 'pwm') {
// Panel current passes straight through, plus the usual 25% margin.
current = isc * strings * 1.25;
} else {
current = arrayWatts / systemVoltage * 1.25;
}
return {
strings,
arrayWatts,
stringColdVoc,
maxSeries,
current,
overVoltage: stringColdVoc > controllerMaxVoltage,
// An MPPT controller needs the string's voltage, which sags about 15% in summer heat, to stay above the
// battery's charging voltage with some headroom. A PWM controller loses power as soon as the string's
// Vmp is below the charging voltage, and all of it once its Voc is.
lowVoltage: controller === 'mppt'
? vmp * series * 0.85 < chargeVoltage(systemVoltage) + 5
: vmp * series < chargeVoltage(systemVoltage),
unevenStrings: series > 0 && panelCount % series !== 0,
pwmFraction: controller === 'pwm' ? pwmUsableFraction(vmp, voc, series, systemVoltage) : 1,
};
}
// Default for the coldest morning when the user has not entered one: well below the coldest monthly
// average, because monthly averages hide cold snaps. Errs cold, since underestimating the cold is what
// destroys controllers.
export const COLD_MARGIN = 25;
export function defaultColdest(monthlyTemperatures) {
const known = (monthlyTemperatures || []).filter(Number.isFinite);
if (known.length === 0) {
return -20;
}
return Math.round(Math.min(...known) - COLD_MARGIN);
}
// Advice about the battery for the site's climate. `coldestMonthly` is the coldest monthly average, °C.
export function batteryWarnings({battery, outdoors, coldestMonthly}) {
const warnings = [];
if (!outdoors || !Number.isFinite(coldestMonthly)) {
return warnings;
}
if (battery === 'lfp' && coldestMonthly < 5) {
warnings.push('Lithium (LiFePO4) batteries are damaged by charging below 0 °C. Keep them indoors, or buy '
+ 'batteries with built-in heaters or low-temperature charge protection.');
}
if (battery === 'lead' && coldestMonthly < 20) {
warnings.push(`Cold lead-acid batteries hold less: the bank is enlarged to deliver ` +
`${Math.round(leadAcidCapacityFactor(coldestMonthly) * 100)}% of its rating in the coldest month. ` +
'Discharged lead-acid batteries can also freeze and crack; keep them charged.');
}
return warnings;
}

View file

@ -0,0 +1,174 @@
import {
BATTERIES,
batteryWarnings,
chargeVoltage,
coldVoc,
controllerSizing,
dailyLoadWh,
defaultColdest,
formatLoads,
leadAcidCapacityFactor,
loadWh,
parseLoads,
pwmUsableFraction,
requiredKwp,
sizeBattery,
} from "./offgrid";
describe('loads', () => {
test('round-trip through the URL format', () => {
const rows = [{name: 'Fridge', watts: '60', hours: '24'}, {name: 'Lights', watts: '20', hours: '5'}];
expect(formatLoads(rows)).toBe('Fridge~60~24|Lights~20~5');
expect(parseLoads(formatLoads(rows))).toEqual(rows);
expect(parseLoads('')).toEqual([]);
expect(parseLoads(null)).toEqual([]);
});
test('separators are removed from names', () => {
expect(parseLoads(formatLoads([{name: 'A|B~C', watts: '1', hours: '1'}]))[0].name).toBe('A B C');
});
test('daily energy is watts times hours, ignoring blanks and capping at 24 hours', () => {
expect(loadWh({watts: '60', hours: '24'})).toBe(1440);
expect(loadWh({watts: '60', hours: '30'})).toBe(1440);
expect(loadWh({watts: '', hours: '5'})).toBe(0);
expect(loadWh({watts: '-5', hours: '5'})).toBe(0);
expect(dailyLoadWh(parseLoads('Fridge~60~24|Lights~20~5|~~'))).toBe(1540);
});
});
describe('battery', () => {
test('a bank for 2 days of 1 kWh at 50% depth of discharge holds 4 kWh', () => {
const bank = sizeBattery({loadWh: 1000, days: 2, dod: 50, voltage: 12});
expect(bank.usableWh).toBe(2000);
expect(bank.nominalWh).toBe(4000);
expect(bank.ah).toBeCloseTo(333.3, 1);
});
test('a cold lead-acid bank must be bigger', () => {
expect(leadAcidCapacityFactor(25)).toBe(1);
expect(leadAcidCapacityFactor(0)).toBeCloseTo(0.8, 6);
expect(leadAcidCapacityFactor(-20)).toBeCloseTo(0.6, 6);
expect(leadAcidCapacityFactor(-60)).toBe(0.5);
const warm = sizeBattery({loadWh: 1000, days: 1, dod: 50, voltage: 12});
const cold = sizeBattery({loadWh: 1000, days: 1, dod: 50, voltage: 12, capacityFactor: 0.8});
expect(cold.nominalWh).toBeCloseTo(warm.nominalWh / 0.8, 6);
});
test('lithium needs a smaller bank than lead-acid for the same load', () => {
const lfp = sizeBattery({loadWh: 1000, days: 2, dod: BATTERIES.lfp.dod, voltage: 24});
const lead = sizeBattery({loadWh: 1000, days: 2, dod: BATTERIES.lead.dod, voltage: 24});
expect(lfp.nominalWh).toBeLessThan(lead.nominalWh);
});
test('no bank without a load', () => {
expect(sizeBattery({loadWh: 0, days: 2, dod: 50, voltage: 12})).toBeNull();
});
});
describe('array', () => {
test('panels needed for the worst month', () => {
// 2 kWh/day, 3 kWh/kWp/day in the worst month, 95% battery efficiency: 2 / 2.85 = 0.702 kW.
expect(requiredKwp(2000, 3, 95)).toBeCloseTo(0.7018, 4);
expect(requiredKwp(2000, 0, 95)).toBeNull();
});
});
describe('charge controller', () => {
test('a PWM controller wastes most of a high-voltage panel on a 12 V battery', () => {
expect(chargeVoltage(12)).toBeCloseTo(14.4, 6);
// 14.4 V / 31 V: under half the panel's power.
expect(pwmUsableFraction(31, 37, 1, 12)).toBeCloseTo(0.4645, 3);
// A "12 V" panel (Vmp ~18 V) suits PWM much better.
expect(pwmUsableFraction(18, 22, 1, 12)).toBeCloseTo(0.8, 6);
});
test('PWM uses the string voltage, so panels in series waste even more', () => {
// Two 31 V panels make a 62 V string: 14.4 / 62.
expect(pwmUsableFraction(31, 37, 2, 12)).toBeCloseTo(0.2323, 3);
});
test('PWM delivers nothing when the string cannot reach the charging voltage', () => {
// One 37 Voc panel against a 48 V bank's 57.6 V.
expect(pwmUsableFraction(31, 37, 1, 48)).toBe(0);
expect(pwmUsableFraction(10, 12, 1, 12)).toBe(0);
});
test('PWM between Vmp and Voc delivers part of the power as current falls toward Voc', () => {
// 14.4 V sits just above a 14 V Vmp: (14.4 / 14) x (21 - 14.4) / (21 - 14).
expect(pwmUsableFraction(14, 21, 1, 12)).toBeCloseTo(0.9698, 3);
});
test('open-circuit voltage rises in the cold', () => {
// -0.27 %/°C at -25 °C (50 °C below the 25 °C rating): +13.5%.
expect(coldVoc(37, -0.27, -25)).toBeCloseTo(41.995, 3);
expect(coldVoc(37, -0.27, 25)).toBe(37);
});
test('a Voc coefficient typed without its minus sign still errs safe', () => {
expect(coldVoc(37, 0.27, -25)).toBeCloseTo(coldVoc(37, -0.27, -25), 9);
const sizing = controllerSizing({...typical, series: 3, betaVoc: 0.27});
expect(sizing.overVoltage).toBe(true);
expect(sizing.maxSeries).toBe(2);
});
const typical = {
panelCount: 4, panelWatts: 400, series: 2, voc: 37, vmp: 31, isc: 13.8, betaVoc: -0.27, coldest: -25,
controller: 'mppt', controllerMaxVoltage: 100, systemVoltage: 24,
};
test('two 400 W panels in series on a 100 V MPPT controller are safe', () => {
const result = controllerSizing(typical);
expect(result.strings).toBe(2);
expect(result.stringColdVoc).toBeCloseTo(83.99, 2);
expect(result.overVoltage).toBe(false);
expect(result.maxSeries).toBe(2);
// 1600 W / 24 V x 1.25.
expect(result.current).toBeCloseTo(83.33, 2);
expect(result.lowVoltage).toBe(false);
expect(result.unevenStrings).toBe(false);
});
test('three in series exceed 100 V on a cold morning', () => {
const result = controllerSizing({...typical, series: 3});
expect(result.overVoltage).toBe(true);
expect(result.unevenStrings).toBe(true);
});
test('one panel per string is too little voltage to charge 48 V through MPPT', () => {
expect(controllerSizing({...typical, series: 1, systemVoltage: 48}).lowVoltage).toBe(true);
});
test('PWM current is the panels\' short-circuit current plus 25%', () => {
const result = controllerSizing({...typical, controller: 'pwm', series: 1, systemVoltage: 12});
expect(result.current).toBeCloseTo(13.8 * 4 * 1.25, 6);
expect(result.pwmFraction).toBeCloseTo(0.4645, 3);
expect(result.lowVoltage).toBe(false);
});
test('PWM on a battery above the string\'s voltage warns and delivers nothing', () => {
const result = controllerSizing({...typical, controller: 'pwm', series: 1, systemVoltage: 48});
expect(result.pwmFraction).toBe(0);
expect(result.lowVoltage).toBe(true);
});
});
describe('defaultColdest', () => {
test('is well below the coldest monthly average', () => {
expect(defaultColdest([-1, 0, 5, 10, 15, 20, 24, 22, 18, 10, 4, -1])).toBe(-26);
expect(defaultColdest([null, null])).toBe(-20);
expect(defaultColdest(null)).toBe(-20);
});
});
describe('batteryWarnings', () => {
test('warn about charging lithium in the cold, only when the batteries are outdoors', () => {
expect(batteryWarnings({battery: 'lfp', outdoors: true, coldestMonthly: -5})[0]).toMatch(/below 0 °C/);
expect(batteryWarnings({battery: 'lfp', outdoors: false, coldestMonthly: -5})).toEqual([]);
expect(batteryWarnings({battery: 'lfp', outdoors: true, coldestMonthly: 15})).toEqual([]);
});
test('explain the cold lead-acid enlargement', () => {
expect(batteryWarnings({battery: 'lead', outdoors: true, coldestMonthly: 0})[0]).toMatch(/80% of its rating/);
});
});

View file

@ -0,0 +1,481 @@
// Pure calculation functions for the Solar calculator. No React here; SolarCalculator.js is the UI.
//
// The model estimates a typical year from 12 monthly averages, the standard approach when hourly
// typical-year weather files are not available:
//
// 1. Sun geometry for Klein's representative day of each month (Duffie & Beckman ch. 1).
// 2. Monthly global horizontal irradiation (GHI) is split into diffuse and beam (Erbs monthly
// correlation), then spread across the day (Collares-Pereira & Rabl for total, Liu & Jordan
// for diffuse).
// 3. Each step is transposed onto the panel with the HDKR sky model, for a fixed, single-axis or
// dual-axis array.
// 4. PVWatts' models turn plane-of-array irradiance into energy: ASHRAE angle-of-incidence loss,
// SAPM cell temperature, linear temperature-corrected DC power, multiplicative system losses,
// and an optional inverter efficiency curve with clipping.
//
// Angles are in degrees at the API boundary. Azimuths are compass bearings (0 = north, 90 = east,
// 180 = south, 270 = west); internally the Duffie & Beckman convention is used (0 = south, east
// negative, west positive). Energy is kWh, irradiation kWh/m², irradiance W/m², power kW.
export const SOLAR_CONSTANT = 1361; // W/m²
const RAD = Math.PI / 180;
const sin = deg => Math.sin(deg * RAD);
const cos = deg => Math.cos(deg * RAD);
const tan = deg => Math.tan(deg * RAD);
const acos = x => Math.acos(Math.min(1, Math.max(-1, x))) / RAD;
const clamp = (x, lo, hi) => Math.min(hi, Math.max(lo, x));
export const MONTH_NAMES = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
export const DAYS_IN_MONTH = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31];
// Klein's recommended average day of each month: the day whose extraterrestrial radiation is
// closest to the month's mean. Day of the year, 1-365.
export const KLEIN_DAYS = [17, 47, 75, 105, 135, 162, 198, 228, 258, 288, 318, 344];
// ---------------------------------------------------------------------------
// Sun geometry
// ---------------------------------------------------------------------------
// Solar declination (Cooper), degrees.
export function declination(dayOfYear) {
return 23.45 * sin(360 * (284 + dayOfYear) / 365);
}
// Sunset hour angle on a horizontal surface, degrees. 0 during polar night, 180 during polar day.
export function sunsetHourAngle(latitude, decl) {
return acos(-tan(latitude) * tan(decl));
}
// Orbit eccentricity correction to the solar constant.
export function eccentricity(dayOfYear) {
return 1 + 0.033 * cos(360 * dayOfYear / 365);
}
// Daily extraterrestrial irradiation on a horizontal surface, kWh/m²/day (D&B eq. 1.10.3).
export function extraterrestrialDaily(latitude, dayOfYear) {
const decl = declination(dayOfYear);
const ws = sunsetHourAngle(latitude, decl);
const wh = (24 / Math.PI) * SOLAR_CONSTANT * eccentricity(dayOfYear)
* (cos(latitude) * cos(decl) * sin(ws) + (Math.PI * ws / 180) * sin(latitude) * sin(decl));
return Math.max(0, wh / 1000);
}
// Day length in hours.
export function dayLength(latitude, decl) {
return 2 * sunsetHourAngle(latitude, decl) / 15;
}
// Sun elevation at solar noon, degrees. Negative means the sun does not rise.
export function noonElevation(latitude, decl) {
return 90 - Math.abs(latitude - decl);
}
// Sun position for an hour angle (degrees, 15° per hour from solar noon, afternoon positive).
// Returns cosine of the zenith angle and the azimuth in the D&B convention (0 = south).
export function sunPosition(latitude, decl, hourAngle) {
const cosZenith = cos(latitude) * cos(decl) * cos(hourAngle) + sin(latitude) * sin(decl);
const sinZenith = Math.sqrt(Math.max(0, 1 - cosZenith * cosZenith));
let azimuth = 0;
if (sinZenith > 1e-9 && Math.abs(cos(latitude)) > 1e-9) {
azimuth = acos((cosZenith * sin(latitude) - sin(decl)) / (sinZenith * cos(latitude)));
azimuth = hourAngle < 0 ? -azimuth : azimuth;
}
return {cosZenith, zenith: acos(cosZenith), azimuth};
}
// Cosine of the angle of incidence on a surface of tilt `tilt` and D&B azimuth `surfaceAzimuth`.
export function cosIncidence(zenith, sunAzimuth, tilt, surfaceAzimuth) {
return cos(zenith) * cos(tilt) + sin(zenith) * sin(tilt) * cos(sunAzimuth - surfaceAzimuth);
}
// Compass bearing (0 = north) to the D&B azimuth convention (0 = south, east negative).
export function compassToSolarAzimuth(compass) {
let a = ((compass % 360) + 360) % 360 - 180;
return a === -180 ? 180 : a;
}
// Default panel azimuth: face the equator.
export function defaultAzimuth(latitude) {
return latitude < 0 ? 0 : 180;
}
// ---------------------------------------------------------------------------
// Monthly irradiation to sub-daily steps
// ---------------------------------------------------------------------------
// Monthly average diffuse fraction Hd/H from the clearness index (Erbs, Klein & Duffie 1982).
// The correlation was fit for 0.3 <= KT <= 0.8, so KT is clamped to that range.
export function erbsMonthlyDiffuseFraction(kt, sunsetAngle) {
const k = clamp(kt, 0.3, 0.8);
if (sunsetAngle <= 81.4) {
return 1.391 - 3.560 * k + 4.189 * k * k - 2.137 * k * k * k;
}
return 1.311 - 3.022 * k + 3.427 * k * k - 1.821 * k * k * k;
}
// Ratio of hourly to daily total irradiation at hour angle `w` (Collares-Pereira & Rabl).
export function totalHourlyRatio(w, sunsetAngle) {
if (Math.abs(w) >= sunsetAngle) {
return 0;
}
const a = 0.409 + 0.5016 * sin(sunsetAngle - 60);
const b = 0.6609 - 0.4767 * sin(sunsetAngle - 60);
return diffuseHourlyRatio(w, sunsetAngle) * (a + b * cos(w));
}
// Ratio of hourly to daily diffuse irradiation at hour angle `w` (Liu & Jordan).
export function diffuseHourlyRatio(w, sunsetAngle) {
if (Math.abs(w) >= sunsetAngle) {
return 0;
}
const denominator = sin(sunsetAngle) - (Math.PI * sunsetAngle / 180) * cos(sunsetAngle);
return (Math.PI / 24) * (cos(w) - cos(sunsetAngle)) / denominator;
}
// Number of evenly spaced steps between sunrise and sunset. Steps adapt to the day length, so a
// short polar-edge day is still sampled.
export const STEPS_PER_DAY = 48;
// The average day of a month as a list of steps, each with its hour angle, duration (hours), and
// the horizontal irradiances (W/m²) for global, beam and diffuse. The two hourly ratios are
// normalized over the day so the steps add back up to exactly the monthly GHI and diffuse given.
export function averageDaySteps(latitude, monthIndex, ghi, diffuse = null) {
const n = KLEIN_DAYS[monthIndex];
const decl = declination(n);
const ws = sunsetHourAngle(latitude, decl);
const h0 = extraterrestrialDaily(latitude, n);
if (!(ghi > 0) || !(ws > 0) || !(h0 > 0)) {
return {decl, sunsetAngle: ws, kt: null, diffuseFraction: null, steps: []};
}
const kt = ghi / h0;
const hd = diffuse != null && diffuse >= 0
? Math.min(diffuse, ghi)
: ghi * erbsMonthlyDiffuseFraction(kt, ws);
const dw = 2 * ws / STEPS_PER_DAY;
const dt = dw / 15;
const raw = [];
let totalSum = 0, diffuseSum = 0;
for (let i = 0; i < STEPS_PER_DAY; i++) {
const w = -ws + (i + 0.5) * dw;
const rt = totalHourlyRatio(w, ws);
const rd = diffuseHourlyRatio(w, ws);
raw.push({w, rt, rd});
totalSum += rt * dt;
diffuseSum += rd * dt;
}
const steps = raw.map(({w, rt, rd}) => {
// kWh/m² per hour is numerically kW/m², so x1000 gives the step's mean irradiance in W/m².
const global = totalSum > 0 ? (rt / totalSum) * ghi * 1000 : 0;
const diff = Math.min(global, diffuseSum > 0 ? (rd / diffuseSum) * hd * 1000 : 0);
return {hourAngle: w, hours: dt, global, diffuse: diff, beam: global - diff};
});
return {decl, sunsetAngle: ws, kt, diffuseFraction: hd / ghi, steps};
}
// ---------------------------------------------------------------------------
// Panel orientation and plane-of-array irradiance
// ---------------------------------------------------------------------------
export const TRACKING = {
fixed: {label: 'Fixed'},
single: {label: 'Single-axis (N–S axis)'},
dual: {label: 'Dual-axis'},
};
// Single-axis trackers are limited to this rotation either side of flat.
export const SINGLE_AXIS_LIMIT = 60;
// Panel tilt and D&B azimuth for a step. `tilt`/`azimuth` are only used for fixed arrays.
export function panelOrientation(tracking, sun, tilt, azimuth) {
if (tracking === 'dual') {
return {tilt: sun.zenith, azimuth: sun.azimuth};
}
if (tracking === 'single') {
// Horizontal north-south axis rotating east-west (Marion & Dobos ideal rotation).
const rotation = clamp(
Math.atan(Math.tan(sun.zenith * RAD) * Math.sin(sun.azimuth * RAD)) / RAD,
-SINGLE_AXIS_LIMIT, SINGLE_AXIS_LIMIT);
return {tilt: Math.abs(rotation), azimuth: rotation >= 0 ? 90 : -90};
}
return {tilt, azimuth};
}
// Plane-of-array irradiance for one step with the HDKR (Hay-Davies-Klucher-Reindl) sky model.
// Returns the beam (direct plus circumsolar), sky diffuse and ground-reflected parts in W/m², and
// the angle of incidence.
export function poaIrradiance({global, beam, diffuse}, sun, orientation, albedo, dayOfYear) {
const {tilt, azimuth} = orientation;
const cosTheta = cosIncidence(sun.zenith, sun.azimuth, tilt, azimuth);
// Beam normal irradiance, capped at the extraterrestrial value; near sunrise the division by a
// tiny cos(zenith) would otherwise blow up.
const extraNormal = SOLAR_CONSTANT * eccentricity(dayOfYear);
const cosZ = Math.max(sun.cosZenith, 0.01745);
const dni = Math.min(beam / cosZ, extraNormal);
const rb = Math.max(0, cosTheta) / cosZ;
const anisotropy = clamp(dni / extraNormal, 0, 1);
const horizonFactor = global > 0 ? Math.sqrt(beam / global) : 0;
const beamPoa = dni * Math.max(0, cosTheta) + diffuse * anisotropy * rb;
const skyPoa = diffuse * (1 - anisotropy) * ((1 + cos(tilt)) / 2)
* (1 + horizonFactor * Math.pow(sin(tilt / 2), 3));
const groundPoa = global * albedo * (1 - cos(tilt)) / 2;
return {beam: beamPoa, sky: skyPoa, ground: groundPoa, total: beamPoa + skyPoa + groundPoa, cosTheta};
}
// ASHRAE incidence angle modifier for the glass cover; applied to the beam part.
export function ashraeIam(cosTheta, b0 = 0.05) {
if (cosTheta <= 0) {
return 0;
}
return clamp(1 - b0 * (1 / cosTheta - 1), 0, 1);
}
// ---------------------------------------------------------------------------
// Temperature, DC power, losses, inverter
// ---------------------------------------------------------------------------
// Sandia (SAPM) cell temperature coefficients by mounting.
export const MOUNTINGS = {
open: {label: 'Open rack (ground or pole)', a: -3.56, b: -0.075, deltaT: 3},
roof: {label: 'Close roof mount', a: -2.98, b: -0.0471, deltaT: 1},
flush: {label: 'Insulated back / flush', a: -2.81, b: -0.0455, deltaT: 0},
};
// PVWatts v8 temperature coefficients of power, %/°C.
export const MODULE_TYPES = {
standard: {label: 'Standard (crystalline)', gamma: -0.37},
premium: {label: 'Premium (high efficiency)', gamma: -0.35},
thinFilm: {label: 'Thin film', gamma: -0.32},
};
export const DEFAULT_WIND_SPEED = 1; // m/s; deliberately low, so cell temperatures are not optimistic.
// SAPM cell temperature, °C.
export function cellTemperature(poa, ambient, mounting = 'open', windSpeed = DEFAULT_WIND_SPEED) {
const m = MOUNTINGS[mounting] || MOUNTINGS.open;
const moduleTemp = poa * Math.exp(m.a + m.b * windSpeed) + ambient;
return moduleTemp + (poa / 1000) * m.deltaT;
}
// Daily air temperature swing, used when only a monthly mean is known: the mean +/- this amount.
export const DEFAULT_DAILY_SWING = 5;
// Air temperature at an hour angle, peaking mid-afternoon (3 pm).
export function ambientTemperature(mean, hourAngle, swing = DEFAULT_DAILY_SWING) {
return mean + swing * cos(hourAngle - 45);
}
// DC power, kW, for an array of `kwp` kilowatts (STC rating) and temperature coefficient `gamma` (%/°C).
export function dcPower(effectiveIrradiance, cellTemp, kwp, gamma) {
return Math.max(0, kwp * (effectiveIrradiance / 1000) * (1 + (gamma / 100) * (cellTemp - 25)));
}
// PVWatts system loss categories and their default percentages.
export const LOSS_CATEGORIES = [
{key: 'soiling', label: 'Soiling', default: 2},
{key: 'shading', label: 'Shading', default: 3},
{key: 'snow', label: 'Snow', default: 0},
{key: 'mismatch', label: 'Mismatch', default: 2},
{key: 'wiring', label: 'Wiring', default: 2},
{key: 'connections', label: 'Connections', default: 0.5},
{key: 'lid', label: 'Light-induced degradation', default: 1.5},
{key: 'nameplate', label: 'Nameplate rating', default: 1},
{key: 'age', label: 'Age', default: 0},
{key: 'availability', label: 'Availability', default: 3},
];
export const DEFAULT_LOSSES = LOSS_CATEGORIES.map(c => c.default);
// Combine loss percentages multiplicatively, as PVWatts does. Returns a percentage.
export function totalLoss(percentages) {
const kept = percentages.reduce((acc, p) => acc * (1 - clamp(Number(p) || 0, 0, 100) / 100), 1);
return 100 * (1 - kept);
}
export const INVERTER_NOMINAL_EFFICIENCY = 96;
const INVERTER_REFERENCE_EFFICIENCY = 0.9637;
// AC power, kW, from the PVWatts inverter model: part-load efficiency curve, clipped at the AC rating.
export function inverterOutput(pdc, acRating, nominalEfficiency = INVERTER_NOMINAL_EFFICIENCY) {
if (!(pdc > 0) || !(acRating > 0)) {
return 0;
}
const etaNom = nominalEfficiency / 100;
const pdc0 = acRating / etaNom;
const zeta = pdc / pdc0;
const eta = (etaNom / INVERTER_REFERENCE_EFFICIENCY) * (-0.0162 * zeta - 0.0059 / zeta + 0.9858);
return Math.min(Math.max(0, eta * pdc), acRating);
}
// ---------------------------------------------------------------------------
// The whole model
// ---------------------------------------------------------------------------
export const DEFAULT_ALBEDO = 0.2;
/**
* Estimate monthly and annual production.
*
* @param {Object} system
* @param {number} system.latitude degrees, north positive
* @param {number[]} system.ghi 12 monthly average GHI values, kWh/m²/day
* @param {number[]} [system.diffuse] 12 monthly diffuse values, kWh/m²/day; estimated when absent
* @param {number[]} [system.temperature] 12 monthly mean air temperatures, °C; 20 when absent
* @param {number[]} [system.albedo] ground reflectance, one value or 12 monthly values
* @param {number} system.kwp array STC rating, kW
* @param {number} [system.tilt] degrees from horizontal (fixed arrays)
* @param {number} [system.azimuth] compass degrees (fixed arrays)
* @param {string} [system.tracking] 'fixed' | 'single' | 'dual'
* @param {string} [system.mounting] key of MOUNTINGS
* @param {number} [system.gamma] temperature coefficient of power, %/°C
* @param {number[]} [system.losses] system loss percentages, see LOSS_CATEGORIES
* @param {Object} [system.inverter] {acRating (kW), efficiency (%)}; null for DC output
*/
export function estimateProduction({
latitude,
ghi,
diffuse = null,
temperature = null,
albedo = DEFAULT_ALBEDO,
kwp,
tilt = 0,
azimuth = defaultAzimuth(latitude),
tracking = 'fixed',
mounting = 'open',
gamma = MODULE_TYPES.standard.gamma,
losses = DEFAULT_LOSSES,
inverter = null,
}) {
const lossFactor = 1 - totalLoss(losses) / 100;
const surfaceAzimuth = compassToSolarAzimuth(azimuth);
const months = MONTH_NAMES.map((name, m) => {
const n = KLEIN_DAYS[m];
const givenAlbedo = Array.isArray(albedo) ? albedo[m] : albedo;
const monthAlbedo = Number.isFinite(givenAlbedo) ? givenAlbedo : DEFAULT_ALBEDO;
const meanTemp = temperature && Number.isFinite(temperature[m]) ? temperature[m] : 20;
const day = averageDaySteps(latitude, m, ghi[m], diffuse ? diffuse[m] : null);
let poaDaily = 0, energyDaily = 0;
for (const step of day.steps) {
const sun = sunPosition(latitude, day.decl, step.hourAngle);
if (sun.cosZenith <= 0) {
continue;
}
const orientation = panelOrientation(tracking, sun, tilt, surfaceAzimuth);
const poa = poaIrradiance(step, sun, orientation, monthAlbedo, n);
const effective = poa.beam * ashraeIam(poa.cosTheta) + poa.sky + poa.ground;
const tc = cellTemperature(poa.total, ambientTemperature(meanTemp, step.hourAngle), mounting);
let power = dcPower(effective, tc, kwp, gamma) * lossFactor;
if (inverter) {
power = inverterOutput(power, inverter.acRating, inverter.efficiency);
}
poaDaily += poa.total * step.hours / 1000;
energyDaily += power * step.hours;
}
return {
month: name,
ghi: ghi[m],
kt: day.kt,
diffuseFraction: day.diffuseFraction,
// Plane-of-array kWh/m²/day: the honest "peak sun hours" for this panel.
poa: poaDaily,
energyDaily,
energy: energyDaily * DAYS_IN_MONTH[m],
dayLength: dayLength(latitude, day.decl),
noonElevation: noonElevation(latitude, day.decl),
};
});
const annual = months.reduce((acc, m) => acc + m.energy, 0);
const worst = months.reduce((a, b) => (b.energyDaily < a.energyDaily ? b : a));
const best = months.reduce((a, b) => (b.energyDaily > a.energyDaily ? b : a));
return {
months,
annual,
specificYield: kwp > 0 ? annual / kwp : null,
capacityFactor: kwp > 0 ? annual / (kwp * 8760) : null,
worst,
best,
totalLoss: totalLoss(losses),
};
}
// Sweep fixed tilts 0-90° at the given azimuth and return the best one. `goal` is 'annual' (most
// energy in a year) or 'winter' (most energy in the worst month, which sizes off-grid systems).
export function optimalTilt(system, goal = 'annual') {
let best = {tilt: 0, value: -1};
for (let tilt = 0; tilt <= 90; tilt++) {
const result = estimateProduction({...system, tilt, tracking: 'fixed'});
const value = goal === 'winter' ? result.worst.energyDaily : result.annual;
if (value > best.value + 1e-9) {
best = {tilt, value};
}
}
return best.tilt;
}
// ---------------------------------------------------------------------------
// Input parsing
// ---------------------------------------------------------------------------
// Parse latitude/longitude text: decimal ("40.015", "-105.27") or degrees-minutes-seconds
// ("40°0'54\"N", "105 16 12 W"). Returns null when unparseable or out of range.
export function parseCoordinate(text, kind = 'lat') {
if (text === null || text === undefined) {
return null;
}
const s = `${text}`.trim().toUpperCase();
if (!s) {
return null;
}
const hemisphere = s.match(/[NSEW]/);
const parts = s.replace(/[NSEW]/g, ' ').match(/-?\d+(\.\d+)?/g);
if (!parts || parts.length > 3) {
return null;
}
const [d, m = 0, sec = 0] = parts.map(Number);
if (m < 0 || m >= 60 || sec < 0 || sec >= 60) {
return null;
}
let value = Math.abs(d) + m / 60 + sec / 3600;
if (d < 0 || s.trim().startsWith('-')) {
value = -value;
}
if (hemisphere) {
const h = hemisphere[0];
if ((kind === 'lat' && (h === 'E' || h === 'W')) || (kind === 'lon' && (h === 'N' || h === 'S'))) {
return null;
}
if (h === 'S' || h === 'W') {
value = -Math.abs(value);
}
}
const limit = kind === 'lat' ? 90 : 180;
return Math.abs(value) <= limit ? value : null;
}
// Parse a comma separated list of 12 monthly numbers. Blank entries become null.
export function parseMonthly(text) {
const values = (text || '').split(',').map(v => (v.trim() === '' ? null : Number(v)));
return MONTH_NAMES.map((_, i) => (Number.isFinite(values[i]) ? values[i] : null));
}
export function formatMonthly(values) {
return values.map(v => (v === null || v === undefined || v === '' ? '' : `${v}`)).join(',');
}
// Link to the Solar calculator for a location, e.g. from the map. 4 decimals (about 11 m) is far finer
// than the 1° data and keeps the shared URL short. MapLibre reports longitudes past ±180° once the map
// has been panned around the world, so those are wrapped back.
export function solarCalculatorPath(latitude, longitude) {
const lon = longitude >= -180 && longitude <= 180 ? longitude : ((longitude + 180) % 360 + 360) % 360 - 180;
const round = value => Number(value.toFixed(4));
return `/more/calculators?calc=solar&lat=${round(latitude)}&lon=${round(lon)}`;
}

View file

@ -0,0 +1,324 @@
import {
ambientTemperature,
ashraeIam,
averageDaySteps,
cellTemperature,
compassToSolarAzimuth,
DAYS_IN_MONTH,
dcPower,
declination,
DEFAULT_LOSSES,
defaultAzimuth,
diffuseHourlyRatio,
erbsMonthlyDiffuseFraction,
estimateProduction,
extraterrestrialDaily,
formatMonthly,
inverterOutput,
KLEIN_DAYS,
optimalTilt,
parseCoordinate,
parseMonthly,
poaIrradiance,
solarCalculatorPath,
sunPosition,
sunsetHourAngle,
totalHourlyRatio,
totalLoss,
} from "./solar";
// A sunny 40°N GHI profile, kWh/m²/day, with a clearness index of about 0.6 year-round (Denver-like).
const GHI_40N = [2.6, 3.5, 4.7, 5.8, 6.5, 7.2, 7.0, 6.2, 5.2, 4.0, 2.9, 2.3];
const TEMP_40N = [0, 2, 6, 10, 15, 21, 24, 23, 18, 11, 4, 0];
const system = (overrides = {}) => ({
latitude: 40,
ghi: GHI_40N,
temperature: TEMP_40N,
kwp: 1,
tilt: 40,
azimuth: 180,
...overrides,
});
const integrate = (ratio, ws) => {
// Midpoint rule over the day, ratios are per hour.
const n = 2000, dw = 2 * ws / n;
let sum = 0;
for (let i = 0; i < n; i++) {
sum += ratio(-ws + (i + 0.5) * dw, ws) * dw / 15;
}
return sum;
};
describe('sun geometry', () => {
test('declination matches Cooper at the solstices and equinox', () => {
expect(declination(172)).toBeCloseTo(23.45, 1); // June 21
expect(declination(355)).toBeCloseTo(-23.45, 1); // December 21
expect(Math.abs(declination(81))).toBeLessThan(0.5); // March 22
// Klein's February day (D&B Table 1.6.1 lists -13.0).
expect(declination(KLEIN_DAYS[1])).toBeCloseTo(-12.95, 1);
});
test('sunset hour angle is 90° at the equator and handles polar day and night', () => {
expect(sunsetHourAngle(0, 20)).toBeCloseTo(90, 5);
expect(sunsetHourAngle(40, 0)).toBeCloseTo(90, 5);
expect(sunsetHourAngle(80, 20)).toBe(180);
expect(sunsetHourAngle(80, -20)).toBe(0);
});
test('extraterrestrial daily irradiation matches Duffie & Beckman Example 1.10.1', () => {
// 43°N on April 15: H0 = 33.8 MJ/m² = 9.39 kWh/m².
expect(extraterrestrialDaily(43, 105)).toBeCloseTo(33.8 / 3.6, 1);
expect(extraterrestrialDaily(80, 355)).toBe(0); // polar night
});
test('sun is due south at solar noon and east in the morning (northern hemisphere)', () => {
const noon = sunPosition(40, 0, 0);
expect(noon.azimuth).toBeCloseTo(0, 5);
expect(noon.zenith).toBeCloseTo(40, 5);
expect(sunPosition(40, 0, -45).azimuth).toBeLessThan(0);
expect(sunPosition(40, 0, 45).azimuth).toBeGreaterThan(0);
});
test('compass azimuth converts to the Duffie & Beckman convention', () => {
expect(compassToSolarAzimuth(180)).toBe(0);
expect(compassToSolarAzimuth(90)).toBe(-90);
expect(compassToSolarAzimuth(270)).toBe(90);
expect(compassToSolarAzimuth(0)).toBe(180);
expect(compassToSolarAzimuth(360)).toBe(180);
expect(defaultAzimuth(40)).toBe(180);
expect(defaultAzimuth(-33)).toBe(0);
});
});
describe('monthly irradiation to steps', () => {
test('Erbs monthly diffuse fraction follows both branches and clamps KT', () => {
expect(erbsMonthlyDiffuseFraction(0.5, 80)).toBeCloseTo(0.3911, 4);
expect(erbsMonthlyDiffuseFraction(0.5, 100)).toBeCloseTo(1.311 - 1.511 + 0.85675 - 0.227625, 4);
expect(erbsMonthlyDiffuseFraction(0.95, 80)).toBeCloseTo(erbsMonthlyDiffuseFraction(0.8, 80), 10);
expect(erbsMonthlyDiffuseFraction(0.1, 80)).toBeCloseTo(erbsMonthlyDiffuseFraction(0.3, 80), 10);
});
test('Liu & Jordan diffuse ratio integrates to one day', () => {
for (const ws of [60, 90, 120, 180]) {
expect(integrate(diffuseHourlyRatio, ws)).toBeCloseTo(1, 3);
}
});
test('Collares-Pereira & Rabl total ratio integrates to about one day', () => {
// The correlation is empirical, so it only approximately conserves the daily total; the
// model normalizes it, but a large error here would mean a transcription mistake.
for (const ws of [60, 90, 120]) {
expect(Math.abs(integrate(totalHourlyRatio, ws) - 1)).toBeLessThan(0.03);
}
});
test('average day steps add back up to the monthly GHI and diffuse', () => {
const day = averageDaySteps(40, 5, 7.2);
const sum = key => day.steps.reduce((acc, s) => acc + s[key] * s.hours, 0) / 1000;
expect(sum('global')).toBeCloseTo(7.2, 6);
expect(sum('diffuse')).toBeCloseTo(7.2 * day.diffuseFraction, 6);
expect(day.steps.every(s => s.beam >= 0 && s.diffuse >= 0)).toBe(true);
// Peak at solar noon.
const peak = day.steps.reduce((a, b) => (b.global > a.global ? b : a));
expect(Math.abs(peak.hourAngle)).toBeLessThan(5);
});
test('a supplied diffuse value is used instead of the Erbs estimate', () => {
expect(averageDaySteps(40, 5, 7.2, 1.8).diffuseFraction).toBeCloseTo(0.25, 6);
});
test('polar night and missing GHI produce no steps', () => {
expect(averageDaySteps(80, 11, 0.1).steps).toEqual([]);
expect(averageDaySteps(40, 5, null).steps).toEqual([]);
});
});
describe('plane of array', () => {
test('a flat panel receives exactly the horizontal irradiance', () => {
const day = averageDaySteps(40, 5, 7.2);
const decl = day.decl;
for (const step of day.steps) {
const sun = sunPosition(40, decl, step.hourAngle);
if (sun.zenith > 85) continue; // the DNI cap applies right at sunrise
const poa = poaIrradiance(step, sun, {tilt: 0, azimuth: 0}, 0.2, KLEIN_DAYS[5]);
expect(poa.total).toBeCloseTo(step.global, 6);
}
});
test('a vertical panel sees half the ground', () => {
const step = {global: 500, beam: 0, diffuse: 500};
const sun = sunPosition(40, 0, 0);
const poa = poaIrradiance(step, sun, {tilt: 90, azimuth: 0}, 0.2, 81);
expect(poa.ground).toBeCloseTo(500 * 0.2 / 2, 6);
});
test('ASHRAE incidence angle modifier', () => {
expect(ashraeIam(1)).toBe(1);
expect(ashraeIam(0.5)).toBeCloseTo(0.95, 6); // 60°
expect(ashraeIam(0)).toBe(0);
expect(ashraeIam(0.01)).toBe(0); // never negative
});
});
describe('temperature, power, losses, inverter', () => {
test('SAPM cell temperature, open rack glass/polymer', () => {
// 1000 * exp(-3.56 - 0.075) + 25 = 51.38, plus deltaT 3.
expect(cellTemperature(1000, 25, 'open', 1)).toBeCloseTo(54.38, 2);
expect(cellTemperature(1000, 25, 'flush', 1)).toBeGreaterThan(cellTemperature(1000, 25, 'open', 1));
expect(cellTemperature(0, 10, 'open')).toBe(10);
});
test('air temperature peaks at 3 pm and averages to the mean', () => {
expect(ambientTemperature(20, 45)).toBeCloseTo(25, 6);
expect(ambientTemperature(20, -135)).toBeCloseTo(15, 6);
});
test('DC power scales with irradiance and loses power when hot', () => {
expect(dcPower(1000, 25, 1, -0.37)).toBeCloseTo(1, 6);
expect(dcPower(500, 25, 2, -0.37)).toBeCloseTo(1, 6);
expect(dcPower(1000, 35, 1, -0.37)).toBeCloseTo(0.963, 6);
expect(dcPower(0, 25, 1, -0.37)).toBe(0);
});
test('PVWatts default losses combine to 14.08%', () => {
expect(totalLoss(DEFAULT_LOSSES)).toBeCloseTo(14.08, 2);
expect(totalLoss([])).toBe(0);
expect(totalLoss([50, 50])).toBeCloseTo(75, 6);
});
test('PVWatts inverter is at nominal efficiency at full load and clips', () => {
// At zeta = 1 the curve evaluates to exactly the nominal efficiency.
const ac = 1, pdc0 = ac / 0.96;
expect(inverterOutput(pdc0, ac) / pdc0).toBeCloseTo(0.96, 4);
expect(inverterOutput(5, ac)).toBe(ac);
expect(inverterOutput(0, ac)).toBe(0);
// Part load is less efficient.
expect(inverterOutput(pdc0 * 0.05, ac) / (pdc0 * 0.05)).toBeLessThan(0.96);
});
});
describe('estimateProduction', () => {
test('a 1 kWp south-facing array at 40°N gives a plausible annual yield', () => {
const result = estimateProduction(system());
// A clear 40°N site yields roughly 1500-1900 kWh/kWp DC after 14% losses.
expect(result.annual).toBeGreaterThan(1500);
expect(result.annual).toBeLessThan(1900);
// Performance ratio (energy over plane-of-array sun-hours): losses plus temperature.
const poaAnnual = result.months.reduce((acc, m, i) => acc + m.poa * DAYS_IN_MONTH[i], 0);
expect(result.annual / poaAnnual).toBeGreaterThan(0.75);
expect(result.annual / poaAnnual).toBeLessThan(0.88);
expect(result.specificYield).toBeCloseTo(result.annual, 6);
expect(result.months).toHaveLength(12);
expect(result.totalLoss).toBeCloseTo(14.08, 2);
});
test('tilting toward the equator gains in winter and loses in summer', () => {
const flat = estimateProduction(system({tilt: 0}));
const tilted = estimateProduction(system({tilt: 40}));
expect(tilted.months[11].poa).toBeGreaterThan(flat.months[11].poa * 1.5);
expect(tilted.months[5].poa).toBeLessThan(flat.months[5].poa);
expect(flat.months[5].poa).toBeCloseTo(GHI_40N[5], 1);
});
test('tracking beats fixed, and dual-axis beats single-axis', () => {
const fixed = estimateProduction(system()).annual;
const single = estimateProduction(system({tracking: 'single'})).annual;
const dual = estimateProduction(system({tracking: 'dual'})).annual;
expect(single).toBeGreaterThan(fixed);
expect(dual).toBeGreaterThan(single);
// Dual-axis is commonly 30-45% over a well-tilted fixed array.
expect(dual / fixed).toBeLessThan(1.6);
});
test('facing away from the equator produces less', () => {
const south = estimateProduction(system()).annual;
const north = estimateProduction(system({azimuth: 0})).annual;
expect(north).toBeLessThan(south * 0.7);
});
test('the southern hemisphere mirrors the northern', () => {
// Same GHI shifted 6 months, panel facing north.
const shifted = [...GHI_40N.slice(6), ...GHI_40N.slice(0, 6)];
const north = estimateProduction(system({temperature: null}));
const south = estimateProduction(system({latitude: -40, ghi: shifted, azimuth: 0, temperature: null}));
expect(south.annual / north.annual).toBeGreaterThan(0.97);
expect(south.annual / north.annual).toBeLessThan(1.03);
});
test('hot weather reduces output', () => {
const cool = estimateProduction(system({temperature: Array(12).fill(0)})).annual;
const hot = estimateProduction(system({temperature: Array(12).fill(35)})).annual;
expect(hot).toBeLessThan(cool * 0.95);
});
test('the inverter costs a few percent and clips an oversized array', () => {
const dc = estimateProduction(system()).annual;
const ac = estimateProduction(system({inverter: {acRating: 1, efficiency: 96}})).annual;
expect(ac / dc).toBeGreaterThan(0.93);
expect(ac / dc).toBeLessThan(0.97);
const clipped = estimateProduction(system({inverter: {acRating: 0.3, efficiency: 96}})).annual;
expect(clipped).toBeLessThan(ac * 0.7);
});
test('polar night months produce nothing without errors', () => {
const result = estimateProduction(system({latitude: 75, tilt: 75}));
expect(result.months[11].energy).toBe(0);
expect(Number.isFinite(result.annual)).toBe(true);
});
test('missing months count as zero', () => {
const ghi = [...GHI_40N];
ghi[0] = null;
expect(estimateProduction(system({ghi})).months[0].energy).toBe(0);
});
test('the best annual tilt is near the latitude; the best winter tilt is steeper', () => {
const annual = optimalTilt(system());
const winter = optimalTilt(system(), 'winter');
expect(annual).toBeGreaterThanOrEqual(30);
expect(annual).toBeLessThanOrEqual(44);
expect(winter).toBeGreaterThan(annual + 8);
});
});
describe('input parsing', () => {
test('decimal and DMS coordinates', () => {
expect(parseCoordinate('40.015')).toBeCloseTo(40.015, 6);
expect(parseCoordinate('-105.27', 'lon')).toBeCloseTo(-105.27, 6);
expect(parseCoordinate('40°0\'54"N')).toBeCloseTo(40.015, 6);
expect(parseCoordinate('105 16 12 W', 'lon')).toBeCloseTo(-105.27, 6);
expect(parseCoordinate('33.9 S')).toBeCloseTo(-33.9, 6);
expect(parseCoordinate('-0.5')).toBeCloseTo(-0.5, 6);
});
test('invalid coordinates are null', () => {
expect(parseCoordinate('')).toBeNull();
expect(parseCoordinate(null)).toBeNull();
expect(parseCoordinate('91')).toBeNull();
expect(parseCoordinate('181', 'lon')).toBeNull();
expect(parseCoordinate('40 E')).toBeNull();
expect(parseCoordinate('abc')).toBeNull();
expect(parseCoordinate('40 75 0')).toBeNull();
});
test('monthly lists round-trip and tolerate blanks', () => {
expect(parseMonthly('1,2,,4')).toEqual([1, 2, null, 4, null, null, null, null, null, null, null, null]);
expect(parseMonthly(null)).toHaveLength(12);
expect(parseMonthly(formatMonthly(GHI_40N))).toEqual(GHI_40N);
});
});
describe('solarCalculatorPath', () => {
test('links to the calculator with the location rounded to 4 decimals', () => {
expect(solarCalculatorPath(40.0153812, -105.2705456))
.toBe('/more/calculators?calc=solar&lat=40.0154&lon=-105.2705');
});
test('wraps a longitude from a map that has been panned around the world', () => {
expect(solarCalculatorPath(10, 190)).toBe('/more/calculators?calc=solar&lat=10&lon=-170');
expect(solarCalculatorPath(10, -540.5)).toBe('/more/calculators?calc=solar&lat=10&lon=179.5');
expect(solarCalculatorPath(10, 180)).toBe('/more/calculators?calc=solar&lat=10&lon=180');
});
});

View file

@ -0,0 +1,48 @@
import {cellIndex, COLUMNS, PARAMETERS, ROWS} from "./solarData";
// Build a climatology file for tests, the same format scripts/build_solar_climatology.py writes.
// `valueAt(key, month, latitude, longitude)` returns the value for a cell center, or null for no data.
export function encodeClimatology(valueAt) {
const header = 8;
const bytes = new Uint8Array(header + PARAMETERS.length * 12 * ROWS * COLUMNS);
bytes.set([...'WSOL'].map(c => c.charCodeAt(0)), 0);
bytes[4] = 1;
bytes[5] = PARAMETERS.length;
PARAMETERS.forEach(({key, offset, step}, p) => {
for (let month = 0; month < 12; month++) {
for (let row = 0; row < ROWS; row++) {
for (let column = 0; column < COLUMNS; column++) {
const value = valueAt(key, month, row - 89.5, column - 179.5);
bytes[header + cellIndex(p, month, row, column)] = value === null
? 255
: Math.max(0, Math.min(254, Math.round((value + offset) / step)));
}
}
}
});
return bytes.buffer;
}
// A plain climatology: the same sunny 40°N profile everywhere.
export const SUNNY_GHI = [2.6, 3.5, 4.7, 5.8, 6.5, 7.2, 7.0, 6.2, 5.2, 4.0, 2.9, 2.3];
export const MILD_TEMPERATURE = [0, 2, 6, 10, 15, 21, 24, 23, 18, 11, 4, 0];
// Encoding the 3 MB file is slow, so it is built once and shared; nothing writes to it.
let uniform = null;
export const uniformClimatology = () => {
if (!uniform) {
uniform = encodeClimatology((key, month) => ({
ghi: SUNNY_GHI[month],
diffuse: SUNNY_GHI[month] * 0.3,
albedo: 0.2,
temperature: MILD_TEMPERATURE[month],
})[key]);
}
return uniform;
};
// Make `fetch` return a climatology file.
export function mockClimatologyFetch(buffer = uniformClimatology()) {
global.fetch = jest.fn(() => Promise.resolve({ok: true, arrayBuffer: () => Promise.resolve(buffer)}));
return global.fetch;
}

View file

@ -0,0 +1,146 @@
import React from "react";
// Offline solar climatology for the Solar calculator: NASA POWER's 2001-2020 monthly averages on a
// 1° grid, built by scripts/build_solar_climatology.py. The format is described in that script and
// must match PARAMETERS below. The file is only fetched when the calculator opens.
export const CLIMATOLOGY_URL = `${process.env.PUBLIC_URL || ''}/data/solar-climatology.bin`;
// Attribution, as NASA asks (https://power.larc.nasa.gov/docs/referencing/) and CC BY 4.0 requires. The
// build script writes the same text, with the API version and download date, to
// public/data/solar-climatology.LICENSE.txt; a test keeps the two in step.
export const CLIMATOLOGY_ACKNOWLEDGMENT = 'The data was obtained from National Aeronautics and Space '
+ 'Administration (NASA) Langley Research Center\'s Prediction Of Worldwide Energy Resources (POWER) project '
+ 'funded through the NASA Earth Science Division.';
export const CLIMATOLOGY_DATA_REFERENCE = 'The data was obtained from the POWER Project\'s POWER Climatology API '
+ 'v2.10.0 version on 2026/10/04.';
export const CLIMATOLOGY_LICENSE = 'Creative Commons Attribution 4.0 International (CC BY 4.0), '
+ 'https://creativecommons.org/licenses/by/4.0/';
export const CLIMATOLOGY_CHANGES = 'WROLPi modified the data: values are rounded to fit one byte each '
+ '(0.04 kWh/m²/day for irradiance, 0.004 for albedo, 0.5 °C for temperature), temperatures are averaged from '
+ 'NASA\'s 0.5° x 0.625° grid into 1° cells, and the calculator interpolates between cells.';
const MAGIC = 'WSOL';
const VERSION = 1;
const HEADER_BYTES = 8;
export const ROWS = 180;
export const COLUMNS = 360;
const MISSING = 255;
// Stored as round((value + offset) / step), in this order.
export const PARAMETERS = [
{key: 'ghi', offset: 0, step: 0.04},
{key: 'diffuse', offset: 0, step: 0.04},
{key: 'albedo', offset: 0, step: 0.004},
{key: 'temperature', offset: 70, step: 0.5},
];
const PARAMETER_BYTES = 12 * ROWS * COLUMNS;
// Validate a downloaded file and return its value grid.
export function decodeClimatology(buffer) {
const bytes = new Uint8Array(buffer);
const magic = String.fromCharCode(...bytes.slice(0, 4));
if (magic !== MAGIC) {
throw new Error('Not a solar climatology file');
}
if (bytes[4] !== VERSION || bytes[5] !== PARAMETERS.length) {
throw new Error(`Unsupported solar climatology version ${bytes[4]}`);
}
if (bytes.length !== HEADER_BYTES + PARAMETERS.length * PARAMETER_BYTES) {
throw new Error('Solar climatology file is truncated');
}
return bytes.subarray(HEADER_BYTES);
}
// Index of a stored value.
export const cellIndex = (parameter, month, row, column) =>
parameter * PARAMETER_BYTES + (month * ROWS + row) * COLUMNS + column;
/**
* Monthly averages at a location, bilinearly interpolated between the four surrounding cell centers.
* Cells with no data are skipped. Longitude wraps across the antimeridian; latitude is clamped at the
* outermost cell centers.
*
* @returns {{ghi: number[], diffuse: number[], albedo: number[], temperature: number[]}} 12 values each,
* null for a month with no data
*/
export function lookupClimatology(grid, latitude, longitude) {
const y = Math.min(ROWS - 1, Math.max(0, latitude + 90 - 0.5));
const row0 = Math.floor(y);
const row1 = Math.min(ROWS - 1, row0 + 1);
const fy = y - row0;
const x = longitude + 180 - 0.5;
const xFloor = Math.floor(x);
const fx = x - xFloor;
const column0 = ((xFloor % COLUMNS) + COLUMNS) % COLUMNS;
const column1 = (column0 + 1) % COLUMNS;
const corners = [
[row0, column0, (1 - fy) * (1 - fx)],
[row0, column1, (1 - fy) * fx],
[row1, column0, fy * (1 - fx)],
[row1, column1, fy * fx],
];
const result = {};
PARAMETERS.forEach(({key, offset, step}, p) => {
result[key] = Array.from({length: 12}, (_, month) => {
let sum = 0, weight = 0;
for (const [row, column, w] of corners) {
const raw = grid[cellIndex(p, month, row, column)];
if (raw !== MISSING && w > 0) {
sum += (raw * step - offset) * w;
weight += w;
}
}
return weight > 0 ? sum / weight : null;
});
});
return result;
}
let climatologyPromise = null;
// Fetch and decode the file once per page load. A failed fetch is not cached, so it can be retried.
export function loadClimatology() {
if (!climatologyPromise) {
climatologyPromise = fetch(CLIMATOLOGY_URL)
.then(response => {
if (!response.ok) {
throw new Error(`Could not load solar data (${response.status})`);
}
return response.arrayBuffer();
})
.then(decodeClimatology)
.catch(error => {
climatologyPromise = null;
throw error;
});
}
return climatologyPromise;
}
// Drop the loaded file, so the next load fetches again. Tests use this to simulate a failed download.
export function forgetClimatology() {
climatologyPromise = null;
}
// The decoded grid, or null while loading. `error` is set when the file could not be loaded.
export function useClimatology() {
const [grid, setGrid] = React.useState(null);
const [error, setError] = React.useState(null);
React.useEffect(() => {
let cancelled = false;
loadClimatology()
.then(g => !cancelled && setGrid(g))
.catch(e => !cancelled && setError(e));
return () => {
cancelled = true;
};
}, []);
return {grid, error, loading: !grid && !error};
}

View file

@ -0,0 +1,167 @@
import fs from "fs";
import path from "path";
import {
CLIMATOLOGY_ACKNOWLEDGMENT,
CLIMATOLOGY_CHANGES,
CLIMATOLOGY_DATA_REFERENCE,
CLIMATOLOGY_LICENSE,
decodeClimatology,
lookupClimatology,
} from "./solarData";
import {estimateProduction} from "./solar";
import {encodeClimatology, uniformClimatology} from "./solarData.fixtures";
describe('decodeClimatology', () => {
test('accepts a valid file', () => {
expect(decodeClimatology(uniformClimatology()).length).toBe(4 * 12 * 180 * 360);
});
test('rejects a file that is not ours, a newer version, or truncated', () => {
const good = new Uint8Array(uniformClimatology());
const wrongMagic = good.slice();
wrongMagic[0] = 'X'.charCodeAt(0);
expect(() => decodeClimatology(wrongMagic.buffer)).toThrow('Not a solar climatology file');
const newer = good.slice();
newer[4] = 2;
expect(() => decodeClimatology(newer.buffer)).toThrow('Unsupported');
expect(() => decodeClimatology(good.slice(0, 1000).buffer)).toThrow('truncated');
});
});
describe('lookupClimatology', () => {
// A field that varies with longitude, so interpolation is visible: GHI = 3 + lon/100.
const sloped = decodeClimatology(encodeClimatology((key, month, lat, lon) => ({
ghi: 3 + lon / 100,
diffuse: 1,
albedo: lat > 60 ? 0.8 : 0.2,
temperature: -10 + month,
})[key]));
test('returns a cell center value exactly', () => {
const values = lookupClimatology(sloped, 40.5, -104.5);
expect(values.ghi[0]).toBeCloseTo(3 - 1.045, 1);
expect(values.temperature.map(v => Math.round(v))).toEqual([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1]);
expect(values.diffuse[6]).toBeCloseTo(1, 1);
});
test('interpolates between cell centers', () => {
// Halfway between the cells centered on -105.5 and -104.5.
const mid = lookupClimatology(sloped, 40.5, -105).ghi[0];
const west = lookupClimatology(sloped, 40.5, -105.5).ghi[0];
const east = lookupClimatology(sloped, 40.5, -104.5).ghi[0];
expect(mid).toBeCloseTo((west + east) / 2, 6);
// Interpolates in latitude too: halfway between albedo 0.2 and 0.8 cells.
expect(lookupClimatology(sloped, 60, 0).albedo[0]).toBeCloseTo(0.5, 2);
});
test('wraps across the antimeridian', () => {
// Between the cells centered on 179.5 (ghi 4.795) and -179.5 (ghi 1.205).
expect(lookupClimatology(sloped, 0.5, 180).ghi[0]).toBeCloseTo((4.795 + 1.205) / 2, 1);
expect(lookupClimatology(sloped, 0.5, -180).ghi[0]).toBeCloseTo((4.795 + 1.205) / 2, 1);
});
test('clamps at the poles', () => {
expect(lookupClimatology(sloped, 90, 0.5).ghi[0]).toBeCloseTo(3.005, 1);
expect(lookupClimatology(sloped, -90, 0.5).ghi[0]).toBeCloseTo(3.005, 1);
});
test('skips cells with no data and returns null when all are missing', () => {
const holes = decodeClimatology(encodeClimatology((key, month, lat, lon) =>
(lon > 0 ? null : {ghi: 5, diffuse: 1, albedo: 0.2, temperature: 10}[key])));
// Straddling lon 0: only the western cells have data.
expect(lookupClimatology(holes, 10.5, 0).ghi[0]).toBeCloseTo(5, 1);
expect(lookupClimatology(holes, 10.5, 90).ghi[0]).toBeNull();
});
});
// The real data file, when it has been built. The expected values come from NASA POWER's point API
// (2001-2020 climatology), so these confirm the build script and decoder agree end to end.
const REAL_FILE = path.join(__dirname, '../../../public/data/solar-climatology.bin');
const describeReal = fs.existsSync(REAL_FILE) ? describe : describe.skip;
describeReal('the built climatology file', () => {
let grid;
beforeAll(() => {
grid = decodeClimatology(fs.readFileSync(REAL_FILE).buffer.slice(0));
});
// NASA POWER's monthly GHI for two cells, copied from its API (2001-2020 climatology). A cell
// center is returned exactly, give or take the file's 0.04 kWh/m²/day step.
const expectCell = (actual, expected) => actual.forEach((v, i) => expect(Math.abs(v - expected[i]))
.toBeLessThanOrEqual(0.021));
test('matches NASA POWER at the cell near Denver', () => {
const values = lookupClimatology(grid, 40.5, -104.5);
expectCell(values.ghi,
[2.4089, 3.2969, 4.6241, 5.6225, 6.3463, 7.2588, 7.0428, 6.2326, 5.2351, 3.7486, 2.6717, 2.0921]);
expect(values.diffuse.every((v, i) => v > 0 && v < values.ghi[i])).toBe(true);
expect(values.temperature[6]).toBeGreaterThan(values.temperature[0] + 15);
});
test('matches NASA POWER at the cell near Sydney, with the seasons reversed', () => {
const values = lookupClimatology(grid, -33.5, 151.5);
expectCell(values.ghi,
[6.4908, 5.6611, 4.8137, 3.8191, 2.9803, 2.357, 2.772, 3.6725, 4.8242, 5.6698, 6.2074, 6.5594]);
});
// PVWatts v8 results for a 1 kW standard-module, open-rack, fixed array with the default 14.08%
// losses, DC output in kWh, fetched from developer.nlr.gov on 2026-10-04. PVWatts uses NREL's NSRDB
// typical-year weather, not NASA POWER, so the two agree only within the data sets' differences.
const PVWATTS = {
Phoenix: {
lat: 33.45, lon: -112.07, tilt: 33, azimuth: 180,
monthly: [139.7, 139.1, 166.3, 175.4, 175.8, 165.2, 152.4, 155.2, 157.6, 157.8, 144.8, 140.3],
},
Seattle: {
lat: 47.61, lon: -122.33, tilt: 48, azimuth: 180,
monthly: [51.9, 67.3, 89.6, 115.5, 127.1, 116.7, 130.3, 141.0, 111.9, 82.0, 48.6, 42.8],
},
Anchorage: {
lat: 61.22, lon: -149.90, tilt: 61, azimuth: 180,
monthly: [19.6, 45.0, 119.2, 117.4, 124.2, 118.0, 109.4, 109.4, 87.8, 52.0, 23.3, 12.6],
},
};
test.each(Object.entries(PVWATTS))('agrees with PVWatts in %s', (name, site) => {
const sunlight = lookupClimatology(grid, site.lat, site.lon);
const result = estimateProduction({
latitude: site.lat, kwp: 1, tilt: site.tilt, azimuth: site.azimuth, ...sunlight,
});
const annual = site.monthly.reduce((a, b) => a + b, 0);
expect(result.annual / annual).toBeGreaterThan(0.9);
expect(result.annual / annual).toBeLessThan(1.1);
// Dark months are small and the two data sets disagree most there; check the months that matter.
const meanMonth = annual / 12;
result.months.forEach((m, i) => {
if (site.monthly[i] >= 0.6 * meanMonth) {
expect(m.energy / site.monthly[i]).toBeGreaterThan(0.75);
expect(m.energy / site.monthly[i]).toBeLessThan(1.25);
}
});
});
test('has polar night', () => {
expect(lookupClimatology(grid, 80, 15).ghi[11]).toBeLessThan(0.05);
});
});
describe('attribution', () => {
// The build script writes this notice beside the data; the page shows the same text.
const notice = fs.readFileSync(path.join(__dirname, '../../../public/data/solar-climatology.LICENSE.txt'), 'utf8');
test('the notice and the page credit NASA identically', () => {
expect(notice).toContain(CLIMATOLOGY_ACKNOWLEDGMENT);
expect(notice).toContain(CLIMATOLOGY_DATA_REFERENCE);
expect(notice).toContain(CLIMATOLOGY_LICENSE);
expect(notice).toContain(CLIMATOLOGY_CHANGES);
});
test('the acknowledgment is NASA POWER\'s requested wording', () => {
// https://power.larc.nasa.gov/docs/referencing/
expect(CLIMATOLOGY_ACKNOWLEDGMENT).toBe('The data was obtained from National Aeronautics and Space '
+ 'Administration (NASA) Langley Research Center\'s Prediction Of Worldwide Energy Resources (POWER) '
+ 'project funded through the NASA Earth Science Division.');
});
});

View file

@ -77,7 +77,9 @@ function renderRow(collection, columns, routes, SingleTag, onRowClick) {
}
// Format the value based on column configuration
if (col.format === 'bytes') {
if (col.render) {
value = col.render(collection);
} else if (col.format === 'bytes') {
value = humanFileSize(value);
} else if (col.format === 'frequency') {
value = formatFrequency(value);
@ -134,7 +136,9 @@ function MobileCollectionRow({collection, mobileColumns, routes}) {
.filter(col => col.type !== 'actions' && col.key !== primaryColumn.key && col.key !== 'tag_name')
.map(col => {
let value = collection[col.key];
if (col.format === 'bytes') {
if (col.render) {
value = col.render(collection);
} else if (col.format === 'bytes') {
value = humanFileSize(value);
} else if (col.format === 'frequency') {
value = formatFrequency(value);
@ -157,7 +161,8 @@ function MobileCollectionRow({collection, mobileColumns, routes}) {
* Reusable table component for displaying collections (Domains, Channels, etc).
*
* @param {Array} collections - Array of collection objects
* @param {Array} columns - Column configurations for the table
* @param {Array} columns - Column configurations for the table. A column's optional `render(collection)` displays
* the cell; the column still sorts by `collection[key]`.
* @param {Object} routes - Routes configuration for navigation (edit, search, etc.)
* @param {String} searchStr - Search filter string (managed by parent)
* @param {Function} onRowClick - Optional callback when a row is clicked

View file

@ -68,6 +68,27 @@ const DOMAIN_ROUTES = {
describe('CollectionTable', () => {
const mockCollections = createMockDomains(3);
describe('Column render', () => {
it('displays what a column renders, in the desktop and mobile rows', () => {
const columns = [
{key: 'name', label: 'Name', render: (c) => <em data-testid='rendered-name'>{c.name}!</em>},
{key: 'branch', label: 'Branch', render: (c) => c.branch || c.default_branch},
];
render(<CollectionTable
collections={[{id: 1, name: 'kiwix-tools', branch: null, default_branch: 'main'}]}
columns={columns}
routes={{search: '/repos/:id', id_field: 'id'}}
/>);
// Desktop: the rendered name is still the link to the collection.
expect(screen.getByTestId('rendered-name')).toHaveTextContent('kiwix-tools!');
expect(screen.getByTestId('rendered-name').closest('a')).toHaveAttribute('href', '/repos/1');
expect(screen.getByText('main')).toBeInTheDocument();
// Mobile lists the other columns by label.
expect(screen.getByText('Branch: main')).toBeInTheDocument();
});
});
describe('Loading and Error States', () => {
it('renders loading placeholder when collections is null', () => {
const {container} = render(

View file

@ -25,7 +25,7 @@ function ItemNameList({items}) {
* `onChanged` refreshes the page's inventory state after a successful action.
*/
export function InventoryImportModal({open, onClose, slug, name, onChanged}) {
const [dates, setDates] = useState(null); // null while loading
const [dates, setDates] = useState(null); // null while loading, undefined if the fetch failed
const [reimporting, setReimporting] = useState(false);
const [selected, setSelected] = useState(null); // {date, mode, preview} confirmation view
const [previewing, setPreviewing] = useState(null); // `${date}:${mode}` currently loading a preview
@ -41,11 +41,19 @@ export function InventoryImportModal({open, onClose, slug, name, onChanged}) {
setReimporting(false);
setApplying(false);
setPreviewing(null);
getInventoryBackups(slug).then(result => {
if (!cancelled) {
setDates(result || []);
}
});
getInventoryBackups(slug)
.then(result => {
if (!cancelled) {
// The api helper returns undefined on a non-OK response.
setDates(result === undefined ? undefined : result || []);
}
})
.catch(e => {
console.error(e);
if (!cancelled) {
setDates(undefined);
}
});
return () => {
cancelled = true;
};
@ -153,6 +161,8 @@ export function InventoryImportModal({open, onClose, slug, name, onChanged}) {
</p>
{dates === null
? <Loading>Loading backups…</Loading>
: dates === undefined
? <p style={{color: 'var(--danger)'}}>Could not fetch backups for this inventory.</p>
: dates.length === 0
? <p>No backups yet. A backup is saved automatically whenever this inventory changes.</p>
: <div style={{maxHeight: '320px', overflowY: 'auto'}}>

Some files were not shown because too many files have changed in this diff Show more