start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* db: one request-scoped database handle per API request
Every API request now gets `request.ctx.db` (`wrolpi.db.RequestDB`): a Session plus raw cursors on that
Session's connection, so a request needs at most one SQLite connection. Creating it does no I/O; requests
that never touch the database never connect. `request.ctx.session` remains the handle's Session.
A `RequestSession` binds itself to one lazily-opened Connection. An engine-bound Session releases its
connection at every commit, and NullPool closes it, so a request that read and then wrote reconnected.
Handlers write in `request.ctx.db.write()`: BEGIN IMMEDIATE, committed when the block exits, so the
write lock is not held for the rest of the handler or by background work it starts. This replaces the
handler-level `get_db_session(commit=True)` blocks (download create, collection item add/remove/reorder,
channel create). The zim search handlers use the request's Session in their worker thread.
Tests count the connections a request opens with production's per-call sessions
(`count_request_connections`). Endpoints whose library functions still open their own connection are
strict xfails, to be removed as those functions take the request's session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* tests: name the real extra connections in the request connection xfails
Channel detail: get_channel already uses the request session; the extra connection is
Channel.get_statistics opening a raw cursor (get_db_curs). Search: count_file_groups opens a
cursor for the total, then handle_file_group_search_results opens a cursor and a session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
cachetools 6 removed cachetools.func.mru_cache; the Zim search estimate
cache was its only user, so it becomes an lru_cache and the pin moves
to 7.2.0 with the Dependabot ignore removed.
reportWebVitals still destructured the web-vitals 2.x names (getCLS,
getFID, ...) which no longer exist, so it silently did nothing. Use
the on* names, replace FID with INP, return the import promise, and
accept any callable rather than instanceof Function.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Cypress component testing resolves the top-level webpack-dev-server
and only accepts major 5, so the bump crashed cypress.config.js on
every run ("Unexpected major version of webpack-dev-server"). The
react-scripts dev server was unaffected because it uses its own
nested 4.x, which is what the pre-merge check looked at.
Pin webpack-dev-server below 6 in dependabot.yml until Cypress
supports it.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
test_handle_reorganize_marks_job_failed_on_exception patched
wrolpi.files.worker.shutil.move, but #719 moved handle_reorganize and
its shutil.move call into wrolpi.files.reorganize_job.
test_file_worker_refresh_inserts_new_files loaded FileGroups into the
shared pytest session between refreshes. The refresh upserts by raw
SQL, so the modeler then read the stale identity-map object and wrote
its old file list back over the upsert. Production opens a fresh
session per block; expire the test session before the second refresh.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Each regenerated after the last config change and each needs its own
migration: cypress 16 needs Node 22, react-router 8 touches ~40 files,
and @testing-library/react 16 fails 15 menu/popover tests.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Comment-based ignores did not survive the last config change, and the
SQLAlchemy rule only blocked majors while 1.4 (the 2.0 transition
release) slipped into the python group. Hold SQLAlchemy <1.4, alembic
<1.11, cachetools majors (mru_cache removed in 6), and the npm majors
that each need a migration branch: maplibre-gl 6, react-dropzone 15+,
React 19, TypeScript 6+, Mantine 9.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
alembic 1.20 requires SQLAlchemy>=2.0, so every grouped python
update that included it failed to install (ResolutionImpossible).
Ignore alembic >=1.11 and SQLAlchemy major bumps until the
SQLAlchemy 2 migration is done.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Caddy is PID 1 in the web container and reaps nothing. The healthcheck's
busybox `wget --spider https://...` forks an `ssl_client` helper that is
orphaned to caddy and stays a zombie, one per check. `init: true` runs
tini as PID 1 so they are reaped.
The zim healthcheck calls curl, which the debian:13 kiwix image does not
install, so every check exited 127. The help healthcheck probed port
8086, which is Caddy's HTTPS proxy on the web container; mkdocs serves
on port 80 inside its own container.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A Sanic worker is forked from the manager and carries the manager's
Python SIGTERM/SIGINT handlers until `_setup_system_signals` installs its
own, which happens after the startup listeners (about two seconds with
sixteen workers starting at once). When auto-reload fires a second time
inside that window the manager terminates the half-started generation,
each child runs `WorkerManager.shutdown_signal` instead of exiting, marks
itself TERMINATED, and then finishes starting. The manager has already
spawned replacements, so the old generation lives on: untracked, holding
the listening socket, serving stale code, and holding ~220 MB each.
Register an at-fork hook in the main process that resets both signals to
their default action in the child. The perpetual process already did
this by hand for itself; server workers now get the same treatment.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The listing still read every channel row (for the summary columns) and every download
row of the listed Collections (for the minimum frequency). Both tables store yt-dlp
info_json, so rows are tens of KB and each sits on its own scattered page; selecting
fewer columns did not help because the seek per row is the cost. From a cold disk
cache that was 5 s per table, 10+ s total, on a large library.
Two covering indexes, channel (collection_id, video_count, total_size) and
download (collection_id, frequency), answer both queries without touching a row. The
listing no longer loads Download objects at all; it computes min_download_frequency
with one grouped query and drops the unused `downloads` array from listing rows (the
detail endpoints keep it; the list views only read min_download_frequency).
Adds a models-vs-migrations test (alembic check) so an index or column added to a
model without a migration, or vice versa, fails the suite.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* collections: channel listing reads trigger columns and batch-loads tag/downloads
Listing channel collections summed FileGroup sizes through a video join, which reads a
random page of the file_group table per video, and lazy-loaded every collection's tag
and downloads one query at a time. On a large library with a cold disk cache the
listing took over two minutes, past the browser's 60 s timeout, so the Channels page
showed an error.
channel.video_count and channel.total_size are already maintained by triggers
(wrolpi/schema_ddl.py), so read those. Tag and downloads are selectin-loaded, so the
query count no longer grows with the number of collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* collections: domain listing reads trigger-maintained summary columns
Domain Collections summed file_group.size through an archive join on every listing,
one random page read per archive (about 9 s from a cold disk cache on a large
library). New triggers keep collection.item_count and collection.total_size current
for Domain Collections, as the channel triggers already do for Channels, and the
listing reads those columns. A migration installs the triggers and backfills
existing Domain Collections.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The Protomaps basemap keeps a few attributes per POI beyond the name the
map already labels: kind, kind_detail and elevation. Clicking a POI now
opens a popup with those, plus the feature's coordinates (or the click
location when the feature has no point). The same POI appears at several
min_zoom levels within one tile, so features are deduped by name and kind.
Map popups (this one and the existing pin popup) now follow the active
theme's tokens instead of MapLibre's fixed white, covering light, dark,
night and amber.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A video's audio stream can be dropped the same way its video stream could
already be: a new REMOVE_AUDIO audio target maps only the video stream with
-an, needs a video container, and is refused for audio files. Output
verification insists the result has no audio stream.
Removing both streams is refused by the validator, the API, resolve_output,
and the modal: choosing Remove for one stream resets the other to Keep and
withdraws the other's Remove option.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Six helpers turned a non-OK response into an empty value, so their
consumers rendered a server error as "nothing here":
* getConfigBackups -> {dates: []}: "No backups available for this
config." A thrown request also left "Loading backups..." forever.
* getInventoryBackups -> []: "No backups yet."
* getDownloaders -> {downloaders: []} on a throw, the error body on a
non-OK response (which then crashed the .map): the RSS form's
Downloader select sat empty.
* searchMap -> {results: []}: "No map locations found".
* getMapSearchStatus -> {indexed: [], missing: []}: "No maps are
installed".
* flasherSearch -> [null, null], then `|| []`: "No .bin firmware
found".
Invariants now enforced and tested:
* On a non-OK response each helper returns undefined; an empty value
only ever comes from the server.
* Each consumer shows an error for a failed or thrown fetch and keeps
its "none" message for a confirmed empty result. The Downloader
select is disabled and says it is loading or could not fetch. A
failed map status is unknown, never "no maps are installed".
* The map search tab loads from the first render, and a query shorter
than the two characters the request needs now says so instead of
"No map locations found".
Archive browsers and recent tags are left as they are: a failed fetch
there hides an optional picker or suggestion list without claiming
anything.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`useForm`'s input handlers called `e.preventDefault()` unconditionally, but the
Mantine-backed `NumberField` (Sleep between requests, Depth, Maximum Pages)
hands them a bare `{target}` object because NumberInput emits a value, not an
event. Every keystroke in those fields threw.
`handleInputEvent` now only calls `preventDefault` when the argument actually
has one, and `getInputProps`' wrapper no longer repeats the call. Audited every
other `preventDefault` call site: all are bound to real DOM/React events or a
guarded no-argument call.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Both contexts start as {} and nothing distinguished that from a
server that reported nothing, so for up to one poll interval after a
page load consumers rendered the negative case as fact: the Settings
form appeared with blank values and an "undefined/config/wrolpi.yaml"
path (the ready check `settings ? true : undefined` was always true),
the Downloads toggle was clickable and read "Downloading Disabled",
the upgrade panel said "up to date, vundefined", and the five hardware
toggles on the Controller page claimed "not supported on this server".
Invariants now enforced and tested:
* StatusContext.loaded is true once a poll returned a real status;
the API being down does not count. SettingsContext.loaded is true
once settings were fetched, and `failed` when a fetch returned
nothing before that; a failed refresh after a load keeps the last
known values and `loaded`.
* The Settings page shows its loader until loaded and its error only
when failed. The upgrade panel shows a loader until status loaded.
The Downloads toggle is disabled with a neutral label until then.
* A subsystem toggle is disabled without claiming "unsupported" while
status is unknown (undefined); "unsupported" (null) is reserved for
a loaded status that lacks the field. A call in flight is unknown,
not unsupported.
The context fixtures declare loaded so existing specs render as
before.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The Flasher's saved configurations started as [] with no loading
flag, and the Saved Firmwares tab is the default tab, so every load
painted "No saved configurations yet." until the fetch returned. The
api helper returned [] on a non-OK response too, so a failed fetch
read as an empty list.
useStatistics used useState() with no initial value, so the
Statistics page's first render saw undefined and flashed "Failed to
fetch statistics" for one frame before the effect set {} and the
spinner appeared.
Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / object = loaded. The
api helper returns undefined (and toasts) on a non-OK response.
* The Saved Firmwares tab shows a placeholder while pending, an error
on failure, and "No saved configurations yet." only once the server
confirmed there are none. Only the newest configs request may write
state, since save and delete refetch.
* The Statistics page shows its loader while pending and its error
only for a failed fetch.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* app: map pins and subscriptions tables distinguish pending from empty
Both lists were initialised to [], which SortableTable renders as
"loaded and empty". The Pins tab therefore showed "No pins.
Right-click on the map to add one." until the pins arrived, and the
Manage tab's subscriptions table showed "No results" until the catalog
did. A non-OK response (the api helpers return undefined) or a thrown
request left the same messages up permanently.
Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = confirmed empty, so
SortableTable shows its placeholder while pending and each empty
message appears only once the server confirmed the list is empty.
* A failed fetch renders an error in place of the table.
* Only the newest request for each list may write state, since edits,
deletes, subscribes and unsubscribes refetch while the mount request
may still be in flight.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* app: the regions preview waits for the catalog
"View All Regions" stayed enabled while the catalog was pending or
failed, and the modal received a fresh `catalog || []` on every
render. Its map effect depends on that array, and ManageMap
re-renders on every status poll, so an open preview tore its WebGL
map down and rebuilt it every three seconds, and a failed catalog was
previewed as zero regions over the error message.
The button is disabled until the catalog is an array, and the modal
is mounted only then, so it receives the state array itself.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
useSuggestions started with channels: [] and reset to that on every
keystroke, and the normalizer treats an empty list as "the server
found none". So the global search dropdown showed a "Channels: No
results" row on focus before anything was typed, and on every
keystroke through the debounce and the request. Because that row
counts as a result, the SearchBox's own "Searching…" state never
appeared.
The channel, domain, author and subject groups now start as null,
meaning not fetched; an empty list only ever comes from the server.
The count summary tolerates the null. Tests render the hook and
check no Channels group exists before a search or while the request
is pending, and that the "No results" row and real rows still appear
after the response.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
ffmpeg wrote `<stem>.transcode.<container>` next to the source. A refresh
that overlapped the transcode discovered that file, probed the half-written
output (errors in the log), and created a FileGroup and Video for a path the
final rename removed moments later.
The temporary file is now `.<stem>.transcode.<container>`: every discovery
path already skips hidden files, and staying in the same directory keeps
the rename over the final path atomic. `transcode_tmp_path` is the single
place the name is built.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* bookmarks: user bookmarks in the navigation bar, stored in bookmarks.yaml
A "Bookmarks" tab between Calculators and Flasher opens on hover or click and
lists the user's bookmarks, with directories as flyout submenus (Menu.Sub).
The mobile hamburger shows directories as labelled groups instead. The tab
always shows, with an "Edit bookmarks" item, so the feature can be found
before any bookmark exists.
Bookmarks live in a new config-only bookmarks.yaml: a nested tree where a
node with `children` is a directory and a node with `url` is a bookmark.
Every node carries an integer id so the API can rename, move and delete
without relying on names. A URL may be a path on this WROLPi, a port on this
host (`:8096/` or `http://:8096/`, resolved against whatever hostname the
browser used, so it works over LAN IP, mDNS and the hotspot alike), or an
absolute URL. Each bookmark chooses whether to open in a new tab.
The editor at /more/bookmarks follows the playlist items table: numbered rows,
up/down/edit/delete, and inline "Add a bookmark" / "Add a directory" forms
with a Location picker. Deleting a directory asks first, since its contents
go with it. The menu and the editor are in the component gallery.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* bookmarks: mark More current when the editor is folded inside it; rem indents in the mobile menu
Found on the QA Pi at a width where Bookmarks overflows into More: the More
button carried no mark on /more/bookmarks, because the bookmarks link has no
`to` for isLinkActive to match. isLinkActive now knows the bookmarks link owns
its editor page, and the Bookmarks tab uses the same rule.
The mobile menu's directory labels sat a few pixels left of the labels around
them: the indent was in em, and a Menu.Label is set smaller than a Menu.Item.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* bookmarks: parse URLs instead of matching a prefix; validate the whole tree on every write
The URL contract was checked by prefix on both sides, and a prefix is not a
form: `//evil.com` starts like a path and `:8096@evil.com` starts like a port,
and a browser takes both off this host. A path is now a single leading slash
that resolves to this origin, a port form is an optional scheme, `:`, digits
and then only a path, query or fragment (no `@`, no backslash, no control
characters), and anything else must parse as http(s). The same rules live in
wrolpi/bookmarks.py and BookmarksContext.js, with the same bypass strings in
both test suites.
The config's dataclass validator only saw that `bookmarks` was a list, so a
hand-edited file or the generic config API could store a `javascript:` URL
that the bookmarks API would refuse, and the nav rendered it. The config now
validates the tree itself on every write and import: URL forms, exactly one of
url/children, unique integer ids, boolean new_tab, and no cycles (a YAML
anchor can build one). An invalid file is not imported and, since it was never
successfully imported, is never overwritten.
On the page, a stored URL that does not resolve is rendered as text, never as
an href. A refused add keeps what was typed instead of clearing the form, and
an edit whose move is refused still refreshes the nav (the rename was
committed) and keeps the modal open. An IPv6 host is bracketed before a port.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* help: document Bookmarks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* app: search Other tab distinguishes pending channels from none
useSearchChannels initialised channels to [] and loading to false, and
only set loading inside its mount effect, so the Other tab's first
render painted "No Channels" with a count of 0 before the search had
run. A failed search was rendered the same way, since the hook had no
catch and the API helper returns undefined on a non-OK response.
Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = no channels; loading
is true from the first render because the search starts on mount.
* Changing tags resets to pending, and only the newest request may
write state, so a slow response for previous tags is dropped.
* The tab shows a loader while pending, an error on failure, and
"No Channels" only once the server confirmed there are none.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* app: useSearchChannels follows its caller's tags
The hook read its tag argument only as the initial state, so the
Other tab never refetched when the filter modal changed the URL's tags
while the tab stayed mounted; the list, including a stale "No
Channels", was whatever the mount fetched.
The internal tag state now follows the argument's contents, keyed on
the serialized list because the caller builds a new array every
render. Tests change the hook argument after a loaded result and
expect a second search and a reset to pending, check that a new array
with the same tags does not refetch, and keep the slow-response drop.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: clear in-progress flags after a crash; downloads only in the perpetual process
Found by a kill -9 stress test on a device:
A SIGKILL inside `with flags.file_worker_counting` (or any of the file
worker's progress flags) left the shared flag set, so every later refresh
raised "flag is already set" until the whole API was restarted.
reconcile_after_predecessor now clears the perpetual process's in-progress
flags (file_worker_*, global_refresh_active) and reports it.
create_downloads started a dispatch in whatever process called it, so an
API worker executed downloads (and could take transcode_lock) despite the
single-process design. It now dispatches only inside the perpetual
process (or under pytest); the download loop picks new rows up within
seconds otherwise.
The shared restart cooldown drops from 60 s to 15 s: it only has to
outlast the manager acting on a request, and a replacement that dies
right away was left dead for the rest of the minute.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* systemd: KillMode=mixed for the API so main shuts its own children down
With control-group kill, SIGTERM reaches the multiprocessing Manager at
the same moment as Sanic's main process; main's cleanup then writes
worker state to a dead Manager (broken pipe) and exits 1, so a plain
`systemctl stop` is recorded as a failure whenever a child takes a
moment to stop (a transcode being killed, for instance). Signal only
main; it stops the workers and the perpetual process itself.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: clear an in-progress flag only when its holder is dead
`with flag:` now records the holder pid in shared_ctx.flag_holders and
drops it on exit. The reconcile step clears the flags only the perpetual
process ever enters (file_worker_counting, file_worker_discovery,
global_refresh_active) whenever they are set, and the flags API workers
also enter (file_worker_modeling / indexing / cleanup via refresh_sync,
map_search_building via the map rebuild endpoints) only when the
recorded holder is dead. file_worker_busy is no longer touched: a
collection move in an API worker holds it as a mutex, and
process_queue drops a stale one itself.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* app: video comments and description distinguish pending from empty
useVideoExtras initialised both values to null and the page treated
null the same as "none", so every video open (and every prev/next)
showed "No comments have been downloaded. Refresh the video?" with a
live Refresh button until the comments arrived. The Description tab
showed "No description available." the same way. A failed fetch was
also rendered as "none", and the previous video's data lingered until
the next video's responses arrived.
Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / '' = loaded and empty.
The API serialises a missing info_json field as null, so the hook
normalises that to empty; consumers never see null for "none".
* Switching video resets both to pending, and a slow response for the
previous video is dropped.
* The page renders a placeholder while pending, an error message on
failure, and offers Refresh only once the server confirmed there are
no comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* app: useVideoExtras drops any request that is not the newest
Matching on the video id let a stale request land when the user left
and returned to the same video, or saved an edit while the page's
first description request was still in flight. Each fetch now takes a
generation number and only the newest generation may write state,
per value, so a description refetch never invalidates a comments
request.
The edit dialog adopts a description that arrives while it is open
only if the user has not typed, so a draft written over the
placeholder is not wiped by the empty or failed value that follows.
Tests: 7 -> none -> 8 -> 7 with the first request rejecting last; a
save refetch beating the initial request; the draft surviving; the
pending placeholder asserted; the console spy restored in finally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
`loading` only became true once the debounce timer fired, so for the
first 300-500ms after every sendRequest the hook reported
data=null, loading=false. Consumers branch on `loading` to choose
between a spinner and an empty state, so the global search dropdown,
the Zim search tab, and the map search rendered "No results" (or
"No Zims have been indexed") for a search that had not run yet.
Invariants now enforced and tested:
* loading is true from sendRequest until the latest request settles.
* The latest-request id is claimed synchronously in sendRequest, so a
fetch already in flight is stale the moment a newer request is sent
and cannot write data or clear loading during the new debounce.
* A stale request that rejects late no longer nulls newer data.
* A pending debounce is cancelled on unmount, and any in-flight fetch
is treated as stale.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* startup: run database config imports off the worker event loop
import_all_db_configs runs in a Sanic worker's after_server_start
listener, after that worker's socket is already accepting connections.
Every import in it was a synchronous call on the event loop, so a large
tags/channels/domains/downloads config froze that worker for the whole
import and every request the kernel routed to it hung. With the default
worker count equal to CPU count, a one- or two-core host lost all or
half of the API for minutes on startup, and the frontend's 60s request
limit turned that into repeated silent retries.
Each synchronous import now runs in asyncio.to_thread. The SQLite engine
already allows cross-thread connections (check_same_thread off, NullPool)
and other startup work already uses to_thread. Imports still run one at
a time in the documented order; only the loop is no longer blocked.
Test: a ticker task must keep ticking while two deliberately slow
synchronous imports run. It scored zero ticks before this change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* startup: thread the downloads import too; test covers every import
import_downloads_config is async but never awaited anything: its body
called the synchronous DownloadManagerConfig.import_config on the event
loop, so a large download_manager.yaml still froze the worker. The
db_up check stays on the loop and only the import moves to a thread.
The responsiveness test now patches every import the startup path
performs, including that sync downloads import, with the same blocking
sleep, and requires ~90% of the total sleep budget in ticks so any one
import returning to the loop fails it. The ticker is awaited after
cancel so the task does not outlive the loop. Docstring trimmed to the
one constraint not visible at the call site.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* files: shared modeler batch loop for video, doc, zim, archive
Paging, skip-ids, progress, and cancel yields live in
wrolpi.files.modeler.run_modeler_loop. Each module still owns select /
prepare / apply and its transaction style: video probes outside the write
lock, docs use one write per file with lock retries, zim and archive use
one write per page. Archive HTML that is not a SingleFile raises
SkipModeler so apply_indexers can claim it.
* files: quieter batch-modeler errors; drop unused imports
Break out of a batch page on InvalidRequestError after a flush failure
so the rest of the page does not each log the same poisoned session.
Re-raise PYTEST apply failures without a misleading commit-failure
message. Remove asyncio/List/OperationalError/PYTEST/Iterable leftovers
from docs, archive, and zim.
FileWorker stays the queue, status, and refresh apply path. Move planning
and execute/revert live in wrolpi.files.move; collection reorganize jobs
live in wrolpi.files.reorganize_job. Perpetual-process dispatch is
unchanged.
Existing `from wrolpi.files.worker import build_move_plan_bulk` still works.
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Discover/compare (find, TEMP table, SQL diff) now lives in
wrolpi.files.refresh so FileWorker is the queue plus apply/move/reorganize.
Existing `from wrolpi.files.worker import compare_file_groups` still works.
file_group.idempotency is unused since SQL compare. DROP COLUMN on SQLite
3.40 does not rebuild the table, so file_group_fts stays valid.
directory.idempotency is unchanged.
* files: pass find -printf \0 as an escape, not a Python NUL (#716)
* files: pass find -printf \\0 as an escape, not a Python NUL
'%h\\0%f\\0%T@\\0' in a Python string is an embedded null byte. subprocess
cannot pass that as an argument, so every GNU-find refresh on Linux CI
raised ValueError. find wants the two-character escape \\0 in the format
string and emits NULs itself.
* files: serialize stem backfill so a cancelled compare cannot lock SQLite
compare_file_groups runs ensure_file_group_stems in a thread. Cancelling
the task does not stop that thread, and a second compare then writes
wrolpi_kv on the same connection (tests share one session). A lock
plus a version re-check makes the second call a no-op.
* files: unique (directory, stem) after collapsing duplicate FileGroups
713 added stem and a non-unique index. This release backfills any NULL
stems, merges leftover duplicates (tags and collection membership move
onto the winner), and creates a UNIQUE index.
Upsert deletes sidecar FileGroups before INSERT so a new primary cannot
collide. Refresh deletes before upsert for the same reason.
from_paths looks up by stem so adding a sidecar does not insert a second
row. Same-stem files without a modeled primary stay one FileGroup.
* files: do not use the test SQLite connection from a worker thread
get_db_curs in pytest shares the test session connection. Running
ensure_file_group_stems and _upsert_files via asyncio.to_thread then
segfaults when the event loop commits that same connection (CI xdist).
Keep to_thread in production, where get_db_curs opens a dedicated
raw connection.
* files: keep tags when a sidecar joins an unmodeled group
choose_primary_file is the single fallback for unmodeled stem groups so
_upsert_files and _upsert_file_groups cannot pick different primaries
and DELETE the tagged row. FileGroup.move merges an existing
(directory, stem) group and flushes the delete before taking that stem.
refresh_files_directly deletes before upsert, matching the other two
callers.
* files: group by (directory, stem), not stem alone
group_files_by_stem mixed marauder/s3.bin and other/s3.bin into one
FileGroup because it ignored directory. Unique (directory, stem) then
kept only one of them, so flasher search missed a firmware file.
run_command_streaming awaits the stdout reader instead of cancelling it
when the process exits, so the last lines are not lost on a fast CI run.
* jobs: write the exception into the Job log on failure
test_jobs_failure only saw 'boom' in the log when JobLogHandler captured
logger.error(..., exc_info=e). Sanic/pytest dictConfig replaces root
handlers (especially under xdist on CI), so that path is empty even
though status and error are set. Log the exception on the Job itself
and store it in the same _update_job as FAILED.
