git_env copied the whole process environment, and git consults its
environment ahead of `-c`: GIT_ALLOW_PROTOCOL=file re-enabled file://
despite protocol.allow=never, and GIT_CONFIG_COUNT/KEY_*/VALUE_* could
add an `insteadOf` that rewrites an approved URL to another host. Only
PATH, HOME, TMPDIR, TZ and the SSL certificate paths are inherited now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They read `successful_import`, which lives in the shared context that
`async_client` attaches. They only passed when an earlier test in the
same xdist worker had attached it; run first, all 13 failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A relative submodule URL is resolved by git against the repo's remote,
and can climb to another host: with the remote
https://git.example.com/owner/repo, `../../../10.0.0.5/x` is
https://10.0.0.5/x. Relative URLs were allowed unchecked.
git now resolves the URLs first (`submodule init` + `sync` write them to
the repo's config), and those resolved URLs, the ones git fetches, are
checked. A relative URL is never allowed by the check itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* .gitmodules is read NUL-delimited, and an unreadable record refuses
the submodules. A submodule name with a space (`[submodule "a b"]`)
made `--get-regexp`'s "key value" output misfile its URL as a path, so
the URL was never checked.
* Submodule URLs must have a strict shape every parser reads the same
way (no userinfo, backslash, query, fragment, or whitespace), so a URL
cannot be checked as one host and fetched from another.
* Each checked host is pinned (http.curloptResolve) to the addresses
that were checked, so DNS cannot change between the check and git's
fetch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A repo's .gitmodules is chosen by whoever controls the repo, so a
submodule URL could make WROLPi fetch https://localhost:8443/...,
https://10.0.0.1/..., etc. (SSRF), and show the response's error.
Every submodule URL is now checked before git fetches it: relative URLs,
the repo's own host, and hosts whose addresses are all public are
allowed; anything else defers the download with the reason. Nested
submodules are checked out one level at a time (instead of git's
--recursive) so each level is checked, at most 5 levels deep. Submodule
fetches do not follow HTTP redirects, so a checked host cannot redirect
git into the network.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* GET /api/repos/<id>/log: the checked out branch's commits, newest
first, a page at a time, with the total. The repo page has a
Files | History toggle (the tree is only listed for Files).
* GET /api/repos/<id>/archive.zip: `git archive HEAD` streamed to the
browser; the file name is reduced to safe characters. Everything is
read from the database before the response starts, because the
request's session closes once a streamed response begins.
* repository.submodules (default off): submodules are checked out with
the repo's mode and the same git restrictions, and removed when the
option is turned off. A submodule that cannot be downloaded (e.g. an
ssh URL) does not stop the repo from updating; the download is
retried with the submodule's error.
* The README speed test bound is loose enough for a busy machine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repos table (CollectionTable), New Repo and Edit pages (frequency,
mode, branch, description, tag, delete with an optional "also delete
its files").
* A GitHub-like repo page: upstream, branch, latest commit, a status
message when not yet downloaded or the last update failed, the file
tree (directories through ?path=, files in the file preview), and each
directory's README.
* Global search: a Repos section under Other with escaped README
headlines, Repos in the suggestions, and the search text sent to the
Other estimate.
* Settings: the Repos Directory field.
* CollectionTable columns may `render` their cell.
* Theme gallery: the repo summary, tree, and status icons.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A README comes from an untrusted remote.
* Escape the README headline: snippet() marks matches with control
characters, the text is HTML-escaped, then the markers become <b>.
Headlines are rendered as HTML (HeadlineText).
* Never follow a symlinked README (O_NOFOLLOW, regular files only).
* Convert markdown line by line with linear-time patterns. The old
multiline patterns backtracked polynomially; a README of blank lines
never finished, and runs of `<!--`, `![` or `<` were quadratic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
repository_fts (FTS5, external content) indexes each Repo's search_name
(name and owner) and readme_text; a name match outranks a README
mention. The downloader stores the README as plain text (markdown
syntax removed so snippets read well), writing it only when it changed.
* POST /api/repos/search: by search and/or Tag, with README headlines.
* /api/search_other_estimates takes an optional search_str and returns
repo_count, so Repos count toward the global search's Other tab.
* /api/search_suggestions suggests Repos by name.
The repository FTS DDL is installed only once the repository table
exists: the baseline migration also runs install_raw_ddl, before that
table is created.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branch names are passed to git as arguments and come from the API, the
config, a `/tree/<branch>` URL, or the remote itself (its default
branch). A name starting with `-` would be read as an option, so every
branch must pass git's ref-name rules, including in the downloader just
before use.
A repo URL with a `.` or `..` path segment (or a `..` name) could format
a directory outside its template (e.g. `repos/<tag>/..` is the whole
repos directory). Such URLs and names are rejected, and every repo
directory, including one from repos.yaml, must be strictly inside the
media directory with no `..`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Repo is a Collection (kind='repo') plus a `repository` row, kept up to
date by a recurring Download handled by the new `git` downloader.
* Full history by default (all branches and tags, no --prune; a force
push saves the old HEAD under refs/wrolpi/backup/); snapshot mode
(--depth 1) and a pinned branch are optional. `/tree/<branch>` URLs
select a branch.
* git runs HTTPS-only, never prompts, never runs hooks, and checks out
symlinks as plain files. Upstream disappearing defers the download
with git's error; the local copy is kept.
* Repos are cloned to `repos_destination` (repos/%(repo_tag)s/%(repo_name)s)
and the `repos` directory is always ignored, so repo files are never
indexed.
* repos.yaml is the source of truth (import, dump, backup restore).
* /api/repos: list, create, get, edit, delete (files kept unless
delete_files), update now, and a per-directory file tree. Tagging uses
the collections tag endpoint.
Also:
* run_command/process_runner can merge stderr into stdout (git writes
progress to stderr).
* move_collection deletes an old directory that only holds empty
directories (the move left an emptied .git behind).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WROLPi runs alembic migrations in-process at startup (db_bootstrap).
alembic/env.py called fileConfig unconditionally, which replaced the
root handlers, set the root level to WARN, and disabled every logger
that already existed, so the API's logs went quiet after any startup
migration until the next restart. env.py now configures logging only
when alembic runs from its command line (cmd_opts is set).
This also caused an order-dependent test failure: test_jobs's capture
test failed whenever test_db_bootstrap ran first in the same worker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
playlists.yaml references file items by path. Renaming, moving, or
reorganizing a FileGroup updated the database but not the config, so the
next import (startup or global refresh) could not find the file and
silently dropped the item from the playlist.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Preview markdown files formatted, with a raw toggle
Markdown files are typed text/markdown (libmagic reports them as text/plain, or
as a programming language when they hold code) and keep the text indexer, so
they stay deep-searchable. Existing files switch type on their next refresh.
The file preview renders markdown with markdown-it into the page, so it follows
the theme tokens and the night media filter. A Formatted/Raw toggle shows the
source instead, and the choice is remembered. Relative links and images
resolve against the file's directory; URLs that climb out of /media/ are
dropped. Raw HTML in a file is escaped rather than parsed, and the output is
also sanitized with DOMPurify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Do not load images from other sites in the markdown preview
An image from another site (http:, https:, or protocol-relative //) loaded as
soon as the preview opened, telling that site who was reading the file and
when. Such an image now becomes a link labeled with its alt text, so the
reader chooses whether to follow it. Inside a link (a README badge) it becomes
the outer link's text, since links cannot nest. Media and embedded data:
images still load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test_import_config_preserves_completed_once_downloads created a download "5 minutes ago" and expected it to
count toward today's limit, which starts at local midnight; in the first five minutes after local midnight it
was yesterday's. It failed in CI started at 00:00:35 UTC, and reproduces on master with the clock at 00:02.
The test now pins the clock to local noon, as the download manager reckons local time.
test_import_all_db_configs_does_not_block_event_loop required 90% of the expected 50 ms ticks, so timer
overshoot on a busy runner added up across the run (CI got 28 of the 28.8 required). It now measures the
longest gap between ticks, including after the last tick: an import on the loop freezes it for the whole 0.2 s
sleep, while jitter delays a tick by milliseconds. Moving the first or the last import onto the loop still
fails it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add a Solar calculator with offline NASA POWER data
Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.
The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter. "Best for the year" and "Best for winter" sweep
the tilt.
Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens. Users can replace it with their own monthly values.
Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.
Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Open the Solar calculator from the map, pins and current location
The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location. A button beside Latitude fills in the browser's current
location. Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.
The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button). It now only cleans up while the
URL is still a map page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add off-grid sizing to the Solar calculator
A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):
- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
at 12, 24 or 48 V. Outdoor lead-acid is enlarged for the cold; outdoor
lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
angle, with a button to apply the count. A PWM controller's loss (battery
voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
against the controller's limit, since cold panels exceed their rated Voc.
The coldest morning defaults to 25 C below the coldest monthly average.
All inputs live in the URL like the rest of the calculator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Bump wrolpi-help for the Solar calculator help page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Credit NASA POWER as it asks, and say how the data was changed
- The calculator gains a "Data attribution" link under the location with
NASA POWER's requested acknowledgment, the data reference (Climatology API
v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
modified the data (rounded to one byte per value, temperature averaged
into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
beside the data, with the API version taken from NASA's responses and the
date the tiles were downloaded. A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: fix PWM sizing, Voc sign, unit switching and loading on review
- PWM usable power now uses the string's voltage (Vmp x panels in series),
delivers nothing when the charging voltage is at or above the string's Voc,
and falls off between Vmp and Voc. The "too little voltage" warning covers
PWM as well as MPPT. Previously two panels in series counted double, and a
panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
negative, in the math, the URL and the input, so the cold-morning check
cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
than asking for a location, and holds the cold-morning verdict until the
site's temperatures are known or one has been entered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: move the data attribution to the bottom of the page
The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location. The source line under the
location points to it. Bumps wrolpi-help to match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Solar: drop the data source line under the location
The attribution at the bottom of the page is enough. The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values. Bumps wrolpi-help for the
refreshed screenshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializers read only loaded data
Serializing a model must not query. A lazy load there is a hidden query per row (search serialized 24 rows
with 55 statements), and would be a `MissingGreenlet` error under an async Session.
- `wrolpi.db.no_db_access(reason)` and `@serializer` (on every model `__json__`, and around `json_response`):
an engine `before_cursor_execute` listener raises `UnexpectedQuery` during tests, and in production logs once
per serializer and lets the query run. Tests record each trip as well, and the autouse
`no_unexpected_queries` fixture fails a test even when the code caught the error (Video's serializer was
swallowing its per-row loads of the deferred `ffprobe_json`).
- Relationships serializers read load with their parent: `selectin` for FileGroup.tag_files and
Collection.downloads; `joined` for TagFile.tag, Collection.tag, Channel.collection, Video.channel,
Archive.collection, ZimSubscription.download, CollectionItem.zim, and the file_group of Video, Archive, Doc,
Zim and CollectionItem.
- Search and the Video lookups the video page uses load the deferred `ffprobe_json` (`undefer`).
- `wrolpi/test/test_serializers.py`: every serialized model, freshly loaded, serializes without SQL.
Search: 55 -> 5 statements for 24 videos (31 -> 5 for archives), and a few ms faster locally.
Not in this change: guarding the `dict()` serializers and `expire_on_commit=False`, which go together (about 50
serialize-after-commit sites, and 40 tests that read values changed elsewhere after a commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: serializer loads per query; the guard ignores transaction control
The query guard reported the engine's own BEGIN (emitted through the same event as the first statement on a
Session with no open transaction) instead of the lazy load after it, and in production that BEGIN used up the
one warning per serializer. Transaction-control statements are now ignored.
Mapper-level eager loads reached background code: every Video/Archive load in the modelers (one per file,
inside the write transaction) joined file_group/channel/collection/tag and selectin-loaded the Collection's
Download rows with their yt-dlp info_json. A second refresh of 100 videos took 0.40 s instead of 0.16 s.
Serializer needs are now declared per model (`json_options()`) and added only by the queries whose results
are serialized; Download.info_json is deferred there (no serializer reads it). Mapper defaults keep only
narrow joins nearly every use needs: TagFile.tag, Collection.tag and Channel.collection. A second refresh is
back to master's statement count and time; search still drops from 55 to 3 statements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deleting a Zim cascade-deletes its TagZimEntry rows, and the next tags
save writes them out of the config, so every outdated-Zim delete lost
its entry tags. The config-import migration only runs at startup, too
late to help. Move each outdated Zim's tags to the current Zim of the
same name before deleting, then save the tags config.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Reset file worker status when a synchronous refresh is cancelled
refresh_sync runs inside request handlers. When Sanic cancelled one on
response timeout, the phase flag cleared as its `with` block unwound but
the shared status did not, so the worker reported 'modeling' forever while
nothing ran. Reset to idle on cancellation.
Deleting outdated Zims also ran the global modelers inside the request,
which is what timed out. Skip post-processing there and update the
outdated_zims flag directly, the only cleanup step it relied on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: observe the modeling flag in the cancelled refresh_sync test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: one request-scoped database handle per API request
Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that
Session's connection, so a request needs at most one SQLite connection. Creating it does no I/O; requests
that never touch the database never connect. `request.ctx.session` remains the handle's Session.
A `RequestSession` binds itself to one lazily-opened Connection. An engine-bound Session releases its
connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected.
Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the
write lock is not held for the rest of the handler or by background work it starts. This replaces the
handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder,
channel create). The zim search handlers use the request's Session in their worker thread.
Tests count the connections a request opens with production's per-call sessions
(`count_request_connections`). Endpoints whose library functions still open their own connection are
strict xfails, to be removed as those functions take the request's session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: name the real extra connections in the request connection xfails
Channel detail: get_channel already uses the request session; the extra connection is
Channel.get_statistics opening a raw cursor (get_db_curs). Search: count_file_groups opens a
cursor for the total, then handle_file_group_search_results opens a cursor and a session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
cachetools 6 removed cachetools.func.mru_cache; the Zim search estimate
cache was its only user, so it becomes an lru_cache and the pin moves
to 7.2.0 with the Dependabot ignore removed.
reportWebVitals still destructured the web-vitals 2.x names (getCLS,
getFID, ...) which no longer exist, so it silently did nothing. Use
the on* names, replace FID with INP, return the import promise, and
accept any callable rather than instanceof Function.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Cypress component testing resolves the top-level webpack-dev-server
and only accepts major 5, so the bump crashed cypress.config.js on
every run ("Unexpected major version of webpack-dev-server"). The
react-scripts dev server was unaffected because it uses its own
nested 4.x, which is what the pre-merge check looked at.
Pin webpack-dev-server below 6 in dependabot.yml until Cypress
supports it.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
test_handle_reorganize_marks_job_failed_on_exception patched
wrolpi.files.worker.shutil.move, but #719 moved handle_reorganize and
its shutil.move call into wrolpi.files.reorganize_job.
test_file_worker_refresh_inserts_new_files loaded FileGroups into the
shared pytest session between refreshes. The refresh upserts by raw
SQL, so the modeler then read the stale identity-map object and wrote
its old file list back over the upsert. Production opens a fresh
session per block; expire the test session before the second refresh.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Each regenerated after the last config change and each needs its own
migration: cypress 16 needs Node 22, react-router 8 touches ~40 files,
and @testing-library/react 16 fails 15 menu/popover tests.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Comment-based ignores did not survive the last config change, and the
SQLAlchemy rule only blocked majors while 1.4 (the 2.0 transition
release) slipped into the python group. Hold SQLAlchemy <1.4, alembic
<1.11, cachetools majors (mru_cache removed in 6), and the npm majors
that each need a migration branch: maplibre-gl 6, react-dropzone 15+,
React 19, TypeScript 6+, Mantine 9.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
alembic 1.20 requires SQLAlchemy>=2.0, so every grouped python
update that included it failed to install (ResolutionImpossible).
Ignore alembic >=1.11 and SQLAlchemy major bumps until the
SQLAlchemy 2 migration is done.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Caddy is PID 1 in the web container and reaps nothing. The healthcheck's
busybox `wget --spider https://...` forks an `ssl_client` helper that is
orphaned to caddy and stays a zombie, one per check. `init: true` runs
tini as PID 1 so they are reaped.
The zim healthcheck calls curl, which the debian:13 kiwix image does not
install, so every check exited 127. The help healthcheck probed port
8086, which is Caddy's HTTPS proxy on the web container; mkdocs serves
on port 80 inside its own container.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A Sanic worker is forked from the manager and carries the manager's
Python SIGTERM/SIGINT handlers until `_setup_system_signals` installs its
own, which happens after the startup listeners (about two seconds with
sixteen workers starting at once). When auto-reload fires a second time
inside that window the manager terminates the half-started generation,
each child runs `WorkerManager.shutdown_signal` instead of exiting, marks
itself TERMINATED, and then finishes starting. The manager has already
spawned replacements, so the old generation lives on: untracked, holding
the listening socket, serving stale code, and holding ~220 MB each.
Register an at-fork hook in the main process that resets both signals to
their default action in the child. The perpetual process already did
this by hand for itself; server workers now get the same treatment.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The listing still read every channel row (for the summary columns) and every download
row of the listed Collections (for the minimum frequency). Both tables store yt-dlp
info_json, so rows are tens of KB and each sits on its own scattered page; selecting
fewer columns did not help because the seek per row is the cost. From a cold disk
cache that was 5 s per table, 10+ s total, on a large library.
Two covering indexes, channel (collection_id, video_count, total_size) and
download (collection_id, frequency), answer both queries without touching a row. The
listing no longer loads Download objects at all; it computes min_download_frequency
with one grouped query and drops the unused `downloads` array from listing rows (the
detail endpoints keep it; the list views only read min_download_frequency).
Adds a models-vs-migrations test (alembic check) so an index or column added to a
model without a migration, or vice versa, fails the suite.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* collections: channel listing reads trigger columns and batch-loads tag/downloads
Listing channel collections summed FileGroup sizes through a video join, which reads a
random page of the file_group table per video, and lazy-loaded every collection's tag
and downloads one query at a time. On a large library with a cold disk cache the
listing took over two minutes, past the browser's 60 s timeout, so the Channels page
showed an error.
channel.video_count and channel.total_size are already maintained by triggers
(wrolpi/schema_ddl.py), so read those. Tag and downloads are selectin-loaded, so the
query count no longer grows with the number of collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* collections: domain listing reads trigger-maintained summary columns
Domain Collections summed file_group.size through an archive join on every listing,
one random page read per archive (about 9 s from a cold disk cache on a large
library). New triggers keep collection.item_count and collection.total_size current
for Domain Collections, as the channel triggers already do for Channels, and the
listing reads those columns. A migration installs the triggers and backfills
existing Domain Collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The Protomaps basemap keeps a few attributes per POI beyond the name the
map already labels: kind, kind_detail and elevation. Clicking a POI now
opens a popup with those, plus the feature's coordinates (or the click
location when the feature has no point). The same POI appears at several
min_zoom levels within one tile, so features are deduped by name and kind.
Map popups (this one and the existing pin popup) now follow the active
theme's tokens instead of MapLibre's fixed white, covering light, dark,
night and amber.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>