* INSTALL.md points to the Controller drive wizard
Debian installs are Debian 13 from the amd64 ISO. The Raspberry Pi
steps no longer hand-edit fstab or run repair.sh; a shared First Boot
section covers the drive, certificate trust, and the first refresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* INSTALL.md: Debian installs use the drive wizard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wrolpi-bootstrap.sh created controller.yaml on installed systems, which
hid the onboarding wizard. Installed systems now wait for the wizard,
whose repair.sh finishes setup.
"Run repair without external drive" now writes controller.yaml before
starting repair, so the wizard does not return after repair.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg is wrapped in `nice -n19` so a multi-hour transcode never competes
with the API, downloads, or other background work.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Transcode, job, and Kiwix notifications send no URL, but every user notify
toast said "Click here to view the shared page" and opened /null on click.
Only a notification carrying a URL is followable now.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Without them a model guessed joins: it joined video.channel_id to collection.id
(it is channel.id), and the query silently returned the wrong channel names.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
systemd prints the start timestamp in local time, which was subtracted from
utcnow(), so elapsed_seconds was off by the UTC offset (6 hours in MDT). Ask
systemd for a Unix timestamp instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
upgrade.sh only installs a branch tip signed with the WROLPi key; the tip was a
GitHub squash merge (#820).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read-only SQL query endpoint for advanced users
POST /api/db/query runs one read-only statement; GET /api/db/schema describes
the tables. Both are refused unless `allow_database_query: true` is set in
wrolpi.yaml (there is no UI for it). The connection is opened with mode=ro and
an authorizer that allows only reads; queries are interrupted after 10 seconds,
rows are capped, and long values are truncated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Database queries cannot be enabled through the API, and use bounded memory
The config update API refuses to change `allow_database_query`; only an edit of
wrolpi.yaml on the WROLPi can enable queries. Query connections refuse values
over 32 MB and rows over 100 columns, rows are read one at a time, and the
response stops at 5 million characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The database query setting can be changed like any other config
WROLPi is meant to be as secure as a bookshelf in your home; a LAN user who can
reach the API is trusted, so gating one key in the config API adds nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Database queries can search FTS5 tables on SQLite 3.46
FTS5 reads `PRAGMA data_version` through the authorizer when it opens a table
on SQLite 3.46 (Debian 13), so every MATCH and the schema endpoint failed with
"vtable constructor failed". A denied VACUUM INTO also explains itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A config status test attaches the shared context it reads
It passed only when an earlier test in the same worker had attached it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A repr runs inside log calls. Reading a relationship or an expired column there runs a query, and
on an object whose session is gone it raises DetachedInstanceError instead of logging. Archive's
repr also stat'ed its SingleFile file on disk.
`wrolpi.common.loaded()` follows attributes through state that is already loaded (primary keys
from the identity key) and never runs SQL. Every model repr uses it; with everything loaded the
output is unchanged, except Video and Archive now show the FileGroup's primary path.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`delete_duplicate_videos` kept whichever duplicate had the largest
FileGroup and deleted the rest. A leftover yt-dlp format file (video only,
or audio only) is often the largest, so it could be kept while every
playable copy was deleted. The deleted copies' `viewed`, `progress` and
`position` were also lost; only tags were carried over.
Rank copies by completeness first (has audio, has video, from the stored
ffprobe data; unknown counts as complete so no ffprobe runs inside the write
transaction), then by size, keeping the oldest of equals as before. The kept
copy takes the most recent view, and the resume position of the most
recently viewed duplicate when it has none of its own.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`get_primary_file` returned the first video/audio file in whatever order it
was given, so a group holding both `<stem>.mp4` and a leftover video-only
`<stem>.f313.webm` could make the leftover the primary file. The Video then
pointed at a file without audio, and a download of it was deferred with
"Download did not contain audio stream" on every retry.
Skip yt-dlp per-format files (`.fNNN` stems) when a merged media file is in
the group. A group of only format files still gets one as its primary.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A merge that never finished leaves `<stem>.f313.webm` (video-only) next to
where the merged file belongs. A later download of the same video globs
every `<stem>.*` file into its FileGroup, but only `.part` files and format
files with the *same* extension as the merged video were deleted. A `.webm`
leftover beside a new `.mp4` survived, joined the new FileGroup, and could
become the Video's file.
Delete every `<stem>.fNNN.*` file in the group, whatever its extension, which
also covers WROLPi's poster/ffprobe sidecars of an indexed leftover and
yt-dlp's `.ytdl` state files.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nothing uses sqlalchemy.ext.asyncio; an AsyncSession pilot on the request path was measured on a
Pi and dropped. SQLAlchemy 2.0 still requires greenlet on aarch64 and x86_64, so installs on the
Pi, Debian and Docker are unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The total for an archive search joined `archive LEFT JOIN file_group` to the FTS table. A LEFT JOIN
fixes `archive` as the outer table, so SQLite scanned every Archive and ran the MATCH once per row:
about 20 seconds of CPU on a Pi with 128k Archives, every time the 60-second totals cache expired.
An inner join lets the planner start from the FTS matches (the same count in milliseconds). Every
Archive has a FileGroup, so the count is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Viewing an archive or a doc marked it viewed on the request's read transaction. SQLite cannot
upgrade a read to a writer once another connection has committed (during a download or refresh),
and it does not wait on busy_timeout for that. The archive page returned a 500; the doc page
returned 200 but its view was lost when the request's commit failed.
Build the response on the read, then record the view in `request.ctx.db.write()`, as the video
page already does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* File downloads: return plain values from upsert_file
Every file download failed after the file was saved, with DetachedInstanceError reading
`fg.location`. `upsert_file` returned a FileGroup from a session it had committed, and the
caller read it after the session was gone. SQLAlchemy 1.3 kept that session alive in a
reference cycle until the next garbage collection, so the read worked by accident; 2.0 frees it
as the `with` block ends. Failed downloads retry, re-downloading their files.
`upsert_file` now returns `UpsertedFile(file_group_id, location)`, read inside its session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Archive uploads: return plain values from singlefile_to_archive
Same failure as upsert_file: `singlefile_to_archive` returned an Archive from a committed session,
and the upload handler read `archive.file_group.title/url` and `archive.location` after the
session was freed, so every SingleFile upload raised DetachedInstanceError after the Archive was
saved. It now returns `UploadedArchive(file_group_id, name, url, location)`, read inside its
session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A RequestSession no longer expires loaded objects at commit, so a handler serializes what it
committed without a reload query. An AsyncSession cannot run that reload at all, so this is
groundwork for moving request handling to async. Background sessions (get_db_session) keep
SQLAlchemy's default: they change data outside the ORM (raw SQL, triggers) and rely on the reload.
Collection, CollectionItem and Channel dict() now run inside the serializer guard. Fixes it found:
- Channel.dict() no longer loads the deferred yt-dlp catalog (info_json), which the channel
handler loaded only to discard.
- get_channel eager-loads what Channel.dict() reads (populate_existing replaces a refresh query).
- create_channel / get_or_create_download link through relationships, so a loaded
Collection.downloads stays current.
- Setting a Repo to Never removes the deleted Download from the loaded collection; the response
reported the old frequency.
- The downloader no longer calls channel.dict() to preload a Channel; its callers read it inside
the session.
Tests run each request with expiry off, as production does, and turn it back on for the background
work that shares the test Session.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CustomJSONEncoder looked up the media directory and called Path.relative_to
for every path it encoded. A response with hundreds of paths (an Archive's
page with 51 history entries holds 416) spent most of its encoding time
there: 18.8 ms of 19 ms per response, measured on a Mac. The encoder now
looks up the media directory once per response and strips it as a string
prefix: 1.5 ms for the same response.
Paths outside the media directory, relative paths, and siblings that share
the media directory's name prefix are still returned unchanged; the media
directory itself and "." are still "".
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/archive/<id> loaded each history Archive's FileGroup and Tags with
their own queries while serializing: two queries per history Archive. An
Archive with 51 history entries ran 107 statements per request. Both the
Archive and its history now use Archive.json_options(), so the count no
longer grows with the history (5 statements for the same Archive).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy 1.3 has been end-of-life since 2021, and alembic was held at 1.10
because newer releases need SQLAlchemy 2. This is a mechanical port; the code
stays synchronous.
- SQLAlchemy[asyncio]==2.0.54, alembic==1.20.0.
- Model annotations use Mapped[...] (2.0 rejects bare annotations on mapped
attributes); declarative_base moves to sqlalchemy.orm.
- Plain SQL strings go through text() or exec_driver_sql; schema_ddl.execute_sql
accepts a SQLAlchemy Connection or a raw sqlite3 connection, so rebuild_fts
works from the baseline migration again.
- Session.get replaces Query.get; session.in_transaction() replaces
session.transaction; cache_ok on MediaPathType and FancyJSON.
- The tag-directory sync query joins through TagFile's relationships instead of
the legacy two-entity outerjoin.
- Section-hint rows use attribute access (2.0 rows are tuples).
- Migrations run on an AUTOCOMMIT connection, matching 1.3, where each
statement committed on its own. 2.0 autobegins, so the unique-stem
migration's read snapshot could not become a writer after its helpers wrote
on another connection ("database is locked"). The begin listener emits no
BEGIN for AUTOCOMMIT connections.
- Newer alembic compares column types: WrolpiKV now describes the TEXT table
its migration created.
- The test database installs its DDL inside engine.begin(); 2.0 connections
no longer autocommit, so the FTS tables were rolled back.
- Dependabot now holds SQLAlchemy at 2.0 instead of 1.3.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
An unknown URL (e.g. /archive instead of /archives) had no matching route,
so react-router rendered the root errorElement in place of Root. That put
PageNotFound outside ThemeProvider, and its Mantine Box threw
"MantineProvider was not found", leaving a blank page.
- A catch-all route renders PageNotFound inside Root, with the navigation
bar, footer and theme.
- The root errorElement brings its own ThemeProvider, so it can render when
Root itself fails.
- App exports its routes so a test can mount them in a memory router.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* app: a download frequency of "Once" no longer blocks submission
useForm treated any falsy value as a missing required field. "Once" is
frequency 0, so the Channel/Playlist, RSS, and Zim download forms never
became ready and the Download button stayed disabled. Only null,
undefined, '', false, and NaN count as missing now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: editing to "Once" stores a once-download, not frequency 0
Once-downloads have a NULL frequency. A 0 frequency counted as recurring
(frequency != None) but could never be scheduled (not frequency), so an
edited download stayed on the recurring list and could break renewal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Scrape: match any of the comma-separated suffixes
The UI's File Suffixes field accepts a comma-separated list, but the
downloader compared each link against the whole string, so a scrape with
more than one suffix never matched anything.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Scrape: resolve relative links against the page's directory
resolve_url appended a relative href to the page URL, so a link to
"a.txt" on ".../dir/page.html" became ".../dir/page.html/a.txt" and the
file download 404'd. Use urljoin, and resolve against the final URL after
redirects so a directory URL without a trailing slash still works.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
aiohttp's response.text() ignores <meta> charset declarations and assumes
UTF-8 when the Content-Type header has no charset, so legacy Latin-1 pages
crashed the Scrape downloader with UnicodeDecodeError. Decode with
UnicodeDammit instead: header charset first, then the page's declared
charset, then detection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bumps the help docs for the new "Resume Where You Left Off" page and the
recently_viewed.yaml config.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Resume EPUBs and comic books where the reader left off
The EPUB viewer (epub.html) accepts `?cfi=` and posts its location and
progress to the page that embeds it on every page turn. Progress comes from
the chapter and the page within it, rather than book.locations.generate(),
which stalls a Raspberry Pi on a large book. A search result's `loc` still
wins over the saved position.
The comic viewer opens at the saved page and saves each page shown.
Both resume on the doc page and in the file preview. The resume point is
decided once per opened file, so saves made while reading do not change the
EPUB viewer's URL and reload it. The doc page tracks only the FileGroup's
primary file; another format beside it is not tracked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* EPUB/comic progress: never save a place the server would discard
The server treats a position at the start (the cover, under 1%) as not
started, and replaces the saved place with progress 0. Both viewers
reported the place they opened on, so opening a finished book un-finished
it, and peeking at the cover erased the reader's page.
- The comic viewer does not save the page it opens on, nor the cover; a
peek at the cover keeps the last page saved.
- The EPUB viewer posts nothing at the start of the book.
- Progress before the end is kept between 1% and 94% (resumableProgress), so
early pages of a long book and the last pages before the end keep their
position. Only the last page (comics) or the end (EPUB) is finished.
- Non-finite EPUB progress is ignored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Resume videos and audio where the user left off; show progress on cards
Videos and audio resume where the user left off, on the video page and in
the file preview, and keep playing. `?t=` starts at an explicit time
instead.
useFileProgress saves how far the user is through a file: every 15 seconds
while it changes, and a final save (a beacon, which survives the page
closing) on pause, end, hiding the page, leaving it, or switching files.
Each save is also kept in sessionStorage; on open the newer of that and the
server's position wins, so coming straight back resumes exactly even if the
final save is still in flight.
Cards show a progress bar along the bottom of their poster. The dashboard's
Recently Viewed Files gains All / Continue; Continue uses the new
`in_progress` file search filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Progress review fixes: in_progress totals, latest dashboard request, preview viewed
- The in_progress search filter skips the uncounted fast path and is part of
the totals cache key, so Continue never reports All's (or the library's)
count.
- The dashboard's Recently Viewed hook applies only its latest response, and
clears the list when switching All/Continue until the new files arrive.
- POST /api/files/file returns `viewed`, so a preview can tell a file
finished elsewhere from an older position this tab holds.
- The progress hook tests restore Blob, fetch and sendBeacon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CircleCI is gone, so the skip and its flag (CIRCLECI → IS_CI) are named for
CI in general, and the flag reads only GITHUB_ACTIONS.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GitHub Actions runs every job CircleCI did (#789), so CircleCI only
duplicated them. The README badge now shows the backend tests on master.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Run the CircleCI jobs on GitHub Actions
Backend tests (pytest, API contract, migrations + API start) and Cypress
(component, e2e) now run as GitHub Actions workflows. CircleCI stays in
place until these have run green alongside it.
- pip and npm caches via setup-python/setup-node, plus the Cypress binary;
node_modules is no longer cached, to stay inside the 10 GB cache cap.
- Superseded pull request runs are cancelled to stay under the concurrent
job limit.
- react-app-start is dropped: the e2e job waits for `npm start`, and
jest-tests.yml already runs `npm run build`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Skip the CI-unsupported tests on GitHub Actions too
skip_circleci only read CIRCLECI, so on GitHub Actions test_archive_upload
ran without the readability tooling it needs, and failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Run the backend tests on every vCPU
-nauto counts physical cores when psutil is installed, so the four-vCPU
runner got two workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Split the backend tests into four jobs by directory
The groups are balanced by test time, so each runs about a quarter of the
suite. `rest` ignores the other groups' paths rather than listing its own,
so a new directory is tested before it is assigned to a group.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test_ownership.py's git() helper did not use TEST_GIT_ENV, so its commits
read the machine's git identity. CI runners have none and cannot guess one,
so seven tests failed there on every run.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
An ignored directory was matched as a string prefix, so ignoring `private`
also ignored `private-notes`. A file is now ignored only when it is the
ignored directory or inside it.
Viewing a file, or saving its progress, checked only the path it was given,
so a link outside an ignored directory to a file inside it was tracked.
is_ignored_file checks the link and its target. Refresh does not follow
links, so the bulk paths keep the cheaper lexical check.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Track how far the user is through a file, kept in recently_viewed.yaml
FileGroup gains `progress` (0.0 to 1.0, 1.0 is finished) and `position` (where
to resume: a time, a page, or an EPUB CFI). Both are returned with the file,
including by POST /api/files/file.
POST /api/files/progress records progress and is also a view. A file 95%
through is finished and keeps no position; one barely started (under 10 s,
the first page, under 1%) is not worth resuming. The write happens before
the response, so the next request sees it. POST /api/files/progress/clear
starts a file over. Both refuse files outside the media directory.
recently_viewed.yaml keeps the 1000 most recently viewed files so viewing
history and positions survive a database rebuild. On import the newer
`viewed` wins, so a file viewed while WROLPi was starting keeps its newer
position. The config is saved when a file is opened, when progress is final
(paused or closed), when it is cleared, and after refresh cleanup (moves,
renames, deletes); heartbeats during playback only write the database.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Progress: write as a writer, confine and merge the import, keep FTS on downgrade
- The progress endpoints begin as writers (request.ctx.db.write()), so a
heartbeat during a refresh waits instead of failing a lock upgrade. The
config switch fires after the commit.
- `..` is collapsed before the ignored-directory check. Clear only resets
an existing FileGroup; it never indexes a file.
- Import confines config paths to the media directory. A newer plain view
keeps its `viewed` but takes the config's position; only progress written
after the config wins. Files are globbed before the write transaction.
- The dump skips a row outside the media directory instead of failing.
- The downgrade drops the columns with ALTER TABLE, which keeps file_group's
FTS triggers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Opening a doc on its page never set FileGroup.viewed, so docs were missing
from Recently Viewed. GET /api/docs/:id now marks the doc viewed, with the
same skip_viewed parameter as archives and videos.
The file preview marked a file viewed more than once: opening from a
?preview= URL sent two tracked requests, and every tag change re-fetched the
file with tracking on. The fetches now skip tracking, and the preview marks
each opening viewed once.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Count Channels with any of the Tags in the Other search estimate
The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Search Channels by name in the Other search tab
The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing. Channels now match the search text by name,
and the Tags when there are both. The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Domains to the Other search tab
Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Playlists to the Other search tab
Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint. The search and the estimate share search_collections_query. The
search now loads each Collection's Downloads up front instead of once per
Collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.
- A Repo is a Collection (kind='repo') plus a `repository` row. The new `git` downloader clones it, then updates it
(weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
An existing clone can be imported where it is. repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download. READMEs are searchable
under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
updated.
- Tagging a Repo moves its clone as one directory. Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
destination. A failed tag move restores the Collection's tag. A Collection's directory cannot leave the media
directory.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse playlist changes until playlists.yaml is imported
While WROLPi starts, the API accepts requests before every config is imported. A playlist created then could not be
saved, and the import which followed deleted it as "no longer in config" (and rebuilt every playlist's items from
the config, undoing any change). Playlist changes now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* collections: refuse domain changes until domains.yaml is imported
A domain created, changed, tagged or deleted through the API before domains.yaml was imported (e.g. while WROLPi
starts) was undone by the import. These now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* archive: importing domains.yaml keeps domains which have archives
Archives create their domain (e.g. one uploaded, or found by a refresh, while WROLPi starts, before domains.yaml is
imported). The import deleted every domain missing from the config, and the database deleted that domain's Archives
with it. A domain with Archives is now kept (and saved to the config with the next change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* videos: refuse Channel changes until channels.yaml is imported
A Channel created through the API before channels.yaml was imported (e.g. while WROLPi starts) could not be saved,
and the import which followed deleted it, and its Downloads, as "not in the config". Creating, changing, tagging and
deleting a Channel (through the Channel or Collection API) now wait for the import (503, "still starting").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
WROLPi runs alembic migrations in-process at startup (db_bootstrap).
alembic/env.py called fileConfig unconditionally, which replaced the
root handlers, set the root level to WARN, and disabled every logger
that already existed, so the API's logs went quiet after any startup
migration until the next restart. env.py now configures logging only
when alembic runs from its command line (cmd_opts is set).
This also caused an order-dependent test failure: test_jobs's capture
test failed whenever test_db_bootstrap ran first in the same worker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
playlists.yaml references file items by path. Renaming, moving, or
reorganizing a FileGroup updated the database but not the config, so the
next import (startup or global refresh) could not find the file and
silently dropped the item from the playlist.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Preview markdown files formatted, with a raw toggle
Markdown files are typed text/markdown (libmagic reports them as text/plain, or
as a programming language when they hold code) and keep the text indexer, so
they stay deep-searchable. Existing files switch type on their next refresh.
The file preview renders markdown with markdown-it into the page, so it follows
the theme tokens and the night media filter. A Formatted/Raw toggle shows the
source instead, and the choice is remembered. Relative links and images
resolve against the file's directory; URLs that climb out of /media/ are
dropped. Raw HTML in a file is escaped rather than parsed, and the output is
also sanitized with DOMPurify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Do not load images from other sites in the markdown preview
An image from another site (http:, https:, or protocol-relative //) loaded as
soon as the preview opened, telling that site who was reading the file and
when. Such an image now becomes a link labeled with its alt text, so the
reader chooses whether to follow it. Inside a link (a README badge) it becomes
the outer link's text, since links cannot nest. Media and embedded data:
images still load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test_import_config_preserves_completed_once_downloads created a download "5 minutes ago" and expected it to
count toward today's limit, which starts at local midnight; in the first five minutes after local midnight it
was yesterday's. It failed in CI started at 00:00:35 UTC, and reproduces on master with the clock at 00:02.
The test now pins the clock to local noon, as the download manager reckons local time.
test_import_all_db_configs_does_not_block_event_loop required 90% of the expected 50 ms ticks, so timer
overshoot on a busy runner added up across the run (CI got 28 of the 28.8 required). It now measures the
longest gap between ticks, including after the last tick: an import on the loop freezes it for the whole 0.2 s
sleep, while jitter delays a tick by milliseconds. Moving the first or the last import onto the loop still
fails it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add a Solar calculator with offline NASA POWER data
Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.
The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter. "Best for the year" and "Best for winter" sweep
the tilt.
Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens. Users can replace it with their own monthly values.
Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.
Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Open the Solar calculator from the map, pins and current location
The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location. A button beside Latitude fills in the browser's current
location. Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.
The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button). It now only cleans up while the
URL is still a map page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add off-grid sizing to the Solar calculator
A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):
- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
at 12, 24 or 48 V. Outdoor lead-acid is enlarged for the cold; outdoor
lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
angle, with a button to apply the count. A PWM controller's loss (battery
voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
against the controller's limit, since cold panels exceed their rated Voc.
The coldest morning defaults to 25 C below the coldest monthly average.
All inputs live in the URL like the rest of the calculator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Bump wrolpi-help for the Solar calculator help page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Credit NASA POWER as it asks, and say how the data was changed
- The calculator gains a "Data attribution" link under the location with
NASA POWER's requested acknowledgment, the data reference (Climatology API
v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
modified the data (rounded to one byte per value, temperature averaged
into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
beside the data, with the API version taken from NASA's responses and the
date the tiles were downloaded. A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: fix PWM sizing, Voc sign, unit switching and loading on review
- PWM usable power now uses the string's voltage (Vmp x panels in series),
delivers nothing when the charging voltage is at or above the string's Voc,
and falls off between Vmp and Voc. The "too little voltage" warning covers
PWM as well as MPPT. Previously two panels in series counted double, and a
panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
negative, in the math, the URL and the input, so the cold-morning check
cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
than asking for a location, and holds the cold-morning verdict until the
site's temperatures are known or one has been entered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: move the data attribution to the bottom of the page
The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location. The source line under the
location points to it. Bumps wrolpi-help to match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: drop the data source line under the location
The attribution at the bottom of the page is enough. The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values. Bumps wrolpi-help for the
refreshed screenshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
