A file of features often carries a basemap's layers too (roads, water, buildings, place names).
Drawn over the basemap they double every road and label. Those layers, by the names of the
Protomaps and OpenMapTiles schemas, are now off until they are checked; every other layer is on
as before, and a choice already made is kept.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
An overlay drawn with a style of its own names icons, which needed the sprite sheet that style was made for. The map now ships icons for overlays, the Maki and Temaki sets (both CC0), and always draws those, in both viewers.
The sprite sheet is the translation: its index lists every icon under its own name, under each other name it answers to (modules/map/icons/aliases.json), and in both spellings of each. A name with no icon is drawn as a plain dot. The icons are signed distance fields, so a style's icon color and halo apply.
A pattern is not an icon: what it filled (a fill, a line or an extrusion) is drawn in a translucent color, where removing a pattern used to leave MapLibre's opaque black.
A sprite sheet beside a map file is no longer used, served or reported. wrolpi/scripts/build_map_icons.py builds the sheet from the SVGs; the sheet is committed.
The standalone viewer (port 8084) drew basemaps only: a map file which is
not a basemap was skipped. It now offers each overlay in its layer panel,
hidden until turned on, with a checkbox per layer, draws it with the style
beside it when there is one, and shows a feature's attributes on a click.
The overlay logic is the map viewer's own. It moved to dependency-free
modules (app/src/map-shared) which a browser can load as they are; the
standalone viewer, which has no build step, loads copies of them from
modules/map/static/js. A test fails while a copy differs from its module
(`npm run sync:map-shared` makes the copies). What differs between the
viewers is passed in: where a map file and its style are fetched from, and
whether a layer can be checked before MapLibre is given it.
The standalone viewer cannot check a layer first, so it adds an overlay's
layers to the map one at a time once the style has loaded: MapLibre refuses a
whole style over one layer it cannot draw, but only that layer when it is
added by itself.
The API also names a style by its path within the map directory, which is
what the standalone viewer is served.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
The contour label layer named no font, so MapLibre asked for its default
("Open Sans Regular,Arial Unicode MS Regular"), which the map does not ship.
The request failed and the labels were drawn with glyphs the browser made up
locally. The layer now names Noto Sans Regular, in the map viewer and in the
standalone viewer.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
When a MapLibre style (and optionally a sprite sheet) sits beside a PMTiles file, the overlay is drawn with it instead of the generic colors: `<name>.style.json` and `<name>.sprite.json/.png`, or `style.json` and `sprite.json/.png` shared by the files of a directory.
Only the style's layers which draw that file are used; its sources, sprite and glyph URLs never are. Fonts are replaced by the ones the map has, images are pointed at the overlay's own sprite (as strings, expressions or property functions), and each layer is validated so one the map could not draw is skipped instead of stopping the map.
The sprite is served by the API, which falls back to the single resolution sheet when there is no double resolution one.
Any PMTiles file under the map directory, at any depth, can be shown. The API reads each file's header and metadata and says what it is (basemap, terrain, overlay, unreadable). Overlays are offered in the layer control under the name the file gives itself, hidden until turned on, with a checkbox per vector layer, and are drawn generically.
Map files are named by their path within the map directory everywhere, and unfinished downloads are no longer listed. A file can be ignored from the Manage page; ignored files are kept in a new maps.yaml config.
PMTiles metadata is not decompressed past its size limit, and the standalone viewer encodes map file paths.
The title sort ordered by `a_text` (the indexed words) with binary collation:
uppercase titles sorted before all lowercase ones ("AWLS" before "Alps"), and
Docs with no indexed words sorted last though they show a title.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Search results sort as the filter dialog says
Docs sorted ascending orders backwards and fell back to id order for every
descending order (`-size`, `-published_datetime`); it now takes the same `-`
prefixed orders as the other searches and rejects unknown ones with a 400.
Archives and Docs searches without an order now put the most relevant results
first, as Videos does and as the dialog shows.
Rank has no direction in the dialog: choosing it sends `rank` (most relevant
first) instead of inheriting a descending `-rank` (least relevant first).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* An old -rank link becomes rank when the filter dialog closes
The dialog shows Rank without a direction, but it seeded its draft from the URL
unchanged, so Done left a saved `-rank` (least relevant first) in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The filter modal on these pages offered "Any Tag", but only the Files search
sent it; the module searches had no `any_tag` field. Docs also matched any of
the listed tag_names, while every other search requires all of them.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
pymupdf 2.0 imports pymupdf.layout and onnxruntime, which every API start
paid for (about 2.5 s on a Pi 4) although only PDF extraction uses it. It is
now imported when a PDF is extracted.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Both Zim search endpoints handed an empty request to the worker-job builder,
whose RuntimeError surfaced as a 500 "Unexpected error" with a traceback. The
request is the client's mistake: the handlers now raise ValidationError, as the
other endpoints do, before any work is scheduled.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The kiwix naming scheme is {name}_{YYYY-MM}.zim, and a re-release in the same
month carries a letter: 3dprinting.stackexchange.com_en_all_2026-07a.zim.
parse_name rejected the letter, so such a Zim was never checked for newer
releases and zims.yaml keyed it by its full stem instead of its name. The
letter is now accepted and folded into the day of the month-precision date, so
a re-release orders after the plain release wherever dates are compared. The
pattern also anchors the ".zim" extension instead of matching any character.
A Zim that is not kiwix-named (edibleplantdb.zim) has no date to compare and is
simply not checked. find_outdated_zim_files logged that at WARNING for every
such file at every start; it is a DEBUG line now.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* A refresh that finishes before tags.yaml is imported no longer reports a failed save (#941)
Every refresh fires save_tags_config at its cleanup. In the first seconds
after a start, a refresh can finish before the startup import of tags.yaml,
which on a rebuilt database happens while the first refresh runs. The dump
then asked ConfigFile.save to write a database that held no tags over the
file; the save guard refused, as it must, but it did so with an ERROR traceback
and a "Failed to save config: config/tags.yaml" event for the user, although
nothing was lost: the import that followed read the file and recreated the tags.
save_tags_config now skips the dump while the config has not been imported and
says so at WARNING. The file stays the source of truth until the import.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Removing a duplicate video keeps the Download that is finishing
After a video download, Video.delete_duplicate_videos removes the other copies
of the same video, which happens when the file name changed (a renamed file, a
new title). Each copy was deleted through FileGroup.delete, which also deletes
the Download of the copy's URL and, without one, adds the URL to the skip list.
The duplicates share that URL with the video just downloaded, so the Download
still running was deleted from under its worker: the worker read its row again
to record the result and failed with NoResultFound ("Unexpected error"), and the
download's record was gone.
FileGroup.delete and Video.delete take keep_download; the duplicate cleanup
passes it, so the URL's Download and the skip list are left alone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Only a duplicate at the downloaded URL keeps its Download
delete_duplicate_videos also finds copies by source_id, and those can sit under
another URL. Such a copy is deleted as any video is, with that URL's Download
or a skip-list entry; only a copy that shares the URL being downloaded leaves
the finishing Download alone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
GET /api/zim went through Zims.get_all, which opened every Zim's Archive in the
API process, and then Zim.__json__ opened each one again for its metadata.
python-libzim holds the GIL and a cold open of a large Zim takes seconds, so
loading the Zims page stalled that API worker for every other request (27.8 s
cold against 0.75 s warm for 44 Zims). #928 moved estimates and searches to the
worker process; the list now uses the same worker for metadata, and a Zim the
worker cannot read is left out as before. count_tagged_zim_entries shares the
new readable_zims helper. Zims.get_all had no other callers and is removed.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
After a download, ffmpeg decodes the last 30 seconds of the video and any stderr
line not on the ignore list marks the video incomplete. Seeking into an h264
stream can make the decoder print "mmco: unref short failure": a note about
reference-picture marking for frames before the seek point. The frames decode
and ffmpeg exits 0, but the verdict was "Video was incomplete", so the download
was deferred and the same sound video was downloaded again on every retry.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Setting a config field calls ConfigFile.update(), which takes
api_app.shared_ctx.config_update_lock. The test did not use async_client,
so the lock only existed when another test in the same process had already
attached the shared context; running first in a worker, it failed with
"'SharedContext' object has no attribute 'config_update_lock'".
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
MultiFileConfig.initialize() imports every file while holding the shared save
lock, which is not reentrant. The repair added in #901 took that lock again from
inside import_all, so any WROLPi whose inventory still held epoch-second dates
hung every API process at startup.
The repair now only rewrites the in-memory entities during import and records
which inventories changed. import_inventories_config() writes those files once
it runs, outside the lock. A regression test runs initialize() against an
epoch-date file with a timeout.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A rebuilt database imports zims.yaml before the refresh has modeled a single
Zim. That import has no database version to compare with, so it saves the
config afterward, and the save wrote only the Zims in the database: none. The
file's choices were erased and every Zim was then modeled with the default.
The dump now starts from the file's entries and overlays the database's, so a
choice survives until its Zim is modeled again, and a Zim in the database still
decides its own line.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Doc._hide_calibre_files opened a second write session while the modeler held
the item's transaction. SQLite has one writer, so the second session waited
out the 30 s busy timeout and failed with "database is locked"; the modeler
retried three times, so every Calibre epub cost about 98 s and stayed
unmodeled, and the cover.jpg / metadata.opf FileGroups it should have deleted
survived. On a library of 1663 Calibre books the global refresh's modeling
phase would have taken about two days.
The nested session dates from before the SQLite cutover, when Postgres row
locks let it work. The delete now runs in the caller's session.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Map downloads are checked against the hash in the signed manifest
A meta4 signs only a bare hash, so any genuinely signed meta4 (an older version of a region, another region) passed
its own check. The manifest now lists each file's sha256; it is signed as a whole, so the hash is tied to the URL.
The catalog copies each hash into its download's settings, and aria2c checks the file against it. Downloads from a
manifest without hashes still require the file's signed meta4.
build_map_regions.py writes the hashes, and hash_map_manifest.py adds them to the published manifest from the
verified meta4s without rebuilding any maps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* hash_map_manifest.py hashes each file's bytes instead of trusting its meta4's name
Only the bare hash in a meta4 is signed, so an older version's meta4 with its name rewritten (or terrain's, whose name
never changes) passed the name and duplicate-hash checks. Each file is now hashed from a local copy of the published
files (--files) or downloaded from the CDN, and must match its signed meta4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The toggle was only the zim.auto_search column, which defaults to on. The
database is rebuilt from files and configs, and a Zim's newer version is a new
row, so every rebuild and every subscription update turned search back on for
every Zim. On a large library the global search then searched all of them.
zims.yaml records auto_search for each Zim, keyed by the Zim's name without
its date, so a newer version of a Zim is the same Zim. A Zim the file does not
name is searched (the default stays on). The config is imported on startup
with the usual version gating, saved when the toggle changes, and a newly
modeled Zim takes its setting from it.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Map files are never downloaded without a signed hash
A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later)
instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it
downloaded with no hash whenever the .meta4 was missing.
A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when
the meta4's unsigned name differs from the URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* aria2c checks map files only against the signed hash, and a rejected file is never kept
Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered
file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the
signed hash and the download URL alone.
A rejected download was left in the map directory under its final name, where the next attempt took it as already
downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that
fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Only the map file aria2c checked is moved into place
download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c
saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as
an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the
URL, must be a plain file name, and any other reported path fails the download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker
The global search estimates every auto-search Zim on each keystroke. On a
large library that estimate took tens of seconds when the Zims' pages were not
cached, and every other request routed to the same Sanic worker, such as the
channel suggestions, waited behind it. Two things held the worker: opening a
large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that
on the event loop; and python-libzim never releases the GIL, so the estimate
offloaded with asyncio.to_thread still froze the worker.
Everything libzim does for an estimate now runs in a standalone script started
as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot
be forked from them). It reads the Zim paths and search string on stdin and
writes the counts on stdout. A Zim it cannot read is reported and left out, as
`Zims.get_all` would leave it out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Zim searches run in the libzim worker process too
The /api/zim/search endpoints had the same stall as the estimate: they opened
every Zim through Zims.get_all and searched them with libzim, which holds the
GIL, so a thread offload still froze the Sanic worker.
The estimate script is now a general libzim worker with estimate, search and
entries jobs. A search of all Zims sends every Zim's job to one worker
process; the API worker only headlines the returned HTML and looks up tags.
The worker also returns each Zim's metadata so Zim.__json__ does not open the
Zim again on the event loop. Zim.entries_with_tags is split so the tagged
paths come from the database without opening the Zim.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker
Review of the earlier commits found two gaps. Estimates with tags selected
still went through Zims.entries_with_tags, which opened every Zim in the API
process, and after the thread offload was removed it did so on the event loop;
they now count the tagged paths in SQL and ask the worker only for metadata.
And Sanic cancels a handler at its response timeout or on disconnect, which
left the worker process running until its stdout pipe filled; the runner now
kills and reaps it on cancellation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Legacy inventory items stored date fields as Unix epoch seconds. The table showed the raw numbers, and sorting and expired highlighting did not treat them as dates.
- Frontend parses date values as YYYY-MM-DD or legacy epoch seconds; display, mobile view, export, search, sort and the expired check all use calendar dates with no timezone shift.
- The legacy migration converts epoch seconds to YYYY-MM-DD, and import repairs already-migrated files once.
- Jest runs with a pinned timezone; the expiry tests pass under any process timezone.
Closes#41
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Job replaces the video file (and its ffprobe json), so the sums in the info json's
wrolpi section described files that were gone. Once the refresh has re-modeled the
Video, its files are hashed again, without a session held, and the sums replaced.
A Video without recorded sums gains none.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A download records the sha256 of its files in the wrolpi section
A video download writes `wrolpi.sha256` into its .info.json; an archive download writes it into
its .readability.json. Sums are keyed by suffix, so a rename or Reorganize keeps them valid,
and the metadata JSON holding them is not hashed.
Large files are hashed in execute_download (in a thread, without the DB session). Finalize
reuses those sums while a file's mtime is unchanged and hashes only files finalize created or
changed (a converted poster, the ffprobe json).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A video download hashes only for an info json, and finalize never suspends
Finalize holds the write session; awaiting the hash let the loop run a switch whose
write waited on that session, stalling the loop for the busy timeout. Finalize now
hashes inline: the large files were hashed in execute_download.
A download that writes no info json has nowhere to keep the sums, so it no longer
reads its files to hash them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The lookup compared SQLite's lower(name), which folds only ASCII, with Python's
str.lower(), which folds Unicode. "Édouard Hospitalier" never matched its own
collection, so the doc modeler inserted a duplicate, hit the unique constraint, and
re-extracted and failed on that doc on every modeler run. Both sides now use
SQLite's lower().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
channels.yaml saves only a Channel's recurring Downloads (Channel.downloads), but
its import compared every Download of a Channel's Collection. A once-Download of
a Channel's video was "removed from the file", so an import with deletes deleted
it, even one that had not run yet. The import now compares, and deletes, only
recurring Downloads.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A config import records the version of the file it read
Every Sanic process initializes the configs it shares with the others while one of them imports the
database configs. `ConfigFile.initialize` reset the shared dict to the defaults (version 0) before
loading the file, and the import read the file's version from that shared dict. When the two
overlapped, the import recorded version 0 in `config_version`; every later save of that config was
refused as older than its file, and the next startup's import undid the unsaved changes.
- `ConfigFile.initialize` sets the shared dict in one call, and `MultiFileConfig.initialize` no longer
empties it first, so other processes never see the defaults in place of the file.
- An import reads the version and rows from the file it read (`import_file_config`), never the shared
dict.
Tests run the race for every config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Playlists import steps read the file's config; tests restore switch handlers
- `PlaylistsConfig._keep_unreadable_items` and `_merge_items` read `import_file_config`, not the
shared `_config`.
- `test_import_steps_read_the_file`: no versioned config's `prepare`/`after_import` (or the methods
they call) reads the shared config.
- The new tests create configs, which replaced the global configs' switch handlers for later tests
(repos tests then dumped a test's config). `restore_switch_handlers` puts them back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Only the first process loads a shared config; the others attach
A restarted perpetual process initializes the configs while the other processes serve them.
Re-loading the files then reverted or dropped newer changes in the shared dict (an inventory
created or saved since, a config updated but not saved yet).
- `ConfigFile.initialize` loads the file only into an empty shared dict, under
`config_update_lock`; otherwise it keeps what the dict holds.
- `MultiFileConfig.initialize` imports only into an empty shared store, under `save_lock()`, and
points `_configs` back at the shared store if the import fails.
- `MultiFileConfig.create` checks for the slug and inserts it under `save_lock()`.
`test_initialize_keeps_the_shared_config` now changes the shared config between two processes'
initializes (an inventory renamed and another created), and requires the second to keep it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A malformed entry stops a backup preview or merge with an error naming it
Backup previews and merges read every config entry as well-formed: a malformed one (an entry that is not a
mapping, a missing or non-text identity, a short tag_files line) raised an AttributeError or TypeError, so the
preview was a 500 and a merge failed with a generic message. `ConfigFile.backup_entries` now checks each entry
a preview or merge reads, in the backup and in the current config, and raises InvalidConfig naming the file and
the entry; the merge endpoint passes that message on. A malformed entry may be corrupt, so it is not skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A falsy "tags" that is not a mapping is an error in a backup preview or merge
`tags: []`, `''`, `0` or `false` became no tags through `or dict()` before the mapping check. Only a missing
or null `tags` means no tags now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A configured browser_executable was used without checking it exists. A config on the media drive can outlive the
OS it was written on: Raspberry Pi OS's /usr/bin/chromium-browser is /usr/bin/chromium on Debian. The wrapper
discards the browser's stderr, so its failed exec only showed as single-file's "Connection refused ... Child
process has already terminated", and every archive failed. Now a missing configured browser is logged and one is
detected, as when none is configured.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`Download.settings['region'].astext` is PostgreSQL's JSONB operator. On SQLite it raises AttributeError, so the
map catalog failed before queuing any region Download, and an extract never checked for a CDN search index. Use
`.as_string()`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deletes functions nothing in the app calls, along with the tests that only
exercised them. Tests that covered live logic through a dead wrapper now call
the live functions: the Download-linking tests use `domain_download_links` /
`channel_download_links` with `_link_downloads`, the archive tests create
Archives with `write_archive_files` + `register_archive`, and the collection
lifecycle test removes an item with `Collection.remove_item`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A video URL that names a playlist (watch?v=...&list=...) makes yt-dlp return a
redirect to the playlist instead of the playlist itself. The ChannelDownloader
now follows up to 10 redirects. Info without "entries" is not a channel or
playlist, so it is rejected before a Channel is created from it. Previously
such a Channel was named after the URL ("watch"), and the download then failed
with KeyError: 'entries'.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deletes functions, methods, classes, schema dataclasses, error classes, Events
senders, and module-level constants that nothing references, including tests.
Also drops the unused `doc_cleanup(test_directory)` and
`_wait_for_sub_job_with_progress(collection_idx)` parameters, and the imports
that only served removed code.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
save_channels_config wrote the Channels with `update()`, which keeps the version the config already had in
memory. After a Channel changed, the database's version was ahead of the saved file's, so the next import
treated the file as older and saved the database over it: an edit made to channels.yaml in the meantime was
lost. It now saves with `dump_config`, which writes the database's version, as every other database config
does.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`domain_download_links` read the destination only from a Download's settings, but the destination is now a column
(the config import moves an old settings destination into it), so no Download was linked to a domain by its
directory. Read the column, and the settings for an old Download, as channels already do.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* A malformed config entry is skipped and counted instead of failing the import
An entry that is not a mapping (or whose key is not text, or a list that is not a list) made the whole file fail to
import in playlists.yaml, repos.yaml, download_manager.yaml and tags.yaml (tag_files/tag_zims), and a non-text name
did in domains.yaml. Each is now skipped and counted in import_skipped, so the config is not saved over it, and the
rest of the file imports. A Channel whose `downloads` is not a list keeps its Downloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* An unreadable entry keeps no row and does not make a section present
The key of an entry too malformed to name a row is now `Unreadable(position)`. It is still counted in
import_skipped, but `present` ignores it, so a section whose entries are all unreadable removes nothing (as an empty
section), instead of deleting every row. A playlist whose `items` is not a list keeps its items, and a Channel
Download whose url is not text is counted (its Channel keeps its Downloads) instead of failing the import.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request). The
per-config overrides that only set successful_import are removed.
wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.
The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* channels.yaml imports by diffing
channels.yaml declares two sections: Channels keyed by directory
(comparing name, URL, Tag, file format, and the source id and flags,
which keep the database's value when missing), and the Downloads listed
under each Channel, keyed by (directory, URL); an existing Download keeps
its own frequency. An unchanged import writes nothing (it ran about a
thousand queries and committed several times).
Removed Channels are deleted with their Collection (and Downloads);
a Download removed from a Channel is deleted. A channel Collection
without a Channel is deleted unless listed (then it gets its Channel).
A Channel whose directory changed, with the same name, is moved. An
invalid entry is skipped and counted, and keeps its Channel (by name when
its directory is invalid) and that Channel's Downloads; so does an entry
without a list of Downloads. Duplicate directories still refuse the
import. Unknown Tags are dropped once tags.yaml is imported.
New Downloads are created in the import's one write (commit=False on
Channel.get_or_create_download, Download.delete and delete_with_videos).
After the import, Videos are claimed and source ids fetched only for new
and changed Channels; Downloads are linked without a query per Download,
and not while entries are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* channels.yaml: moves, invalid Downloads, and saves after the write
- A moved Channel's Download diff is retargeted to its new directory:
one its new entry lists is the same Download, one it does not is
deleted in the same import (the old directory was treated as kept).
- An unreadable Download is counted as skipped, and its Channel keeps
its Downloads (it was dropped, and the Channel's Download deleted).
- A new Download without a frequency is dropped before the write and
counted (it re-opened a write on every import).
- No source id is fetched while entries are skipped (its save would be
refused, leaving the database newer than the file); a fetched one is
saved with dump_config, which records the database's version.
- Download.delete(commit=False) neither changes the skip list nor starts
config saves, which would read the database before the import commits
(and write a removed Download back into channels.yaml). The import
saves download_manager.yaml after its write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* domains.yaml imports by diffing
domains.yaml declares one section, its domain Collections keyed by name,
comparing description, directory, Tag and file format. An unchanged
import writes nothing.
A removed domain without Archives is deleted; one with Archives is reset
to its default settings (decision 7c: a refresh of its Archives would
re-create it), and the config is saved with it. An invalid entry (name
or directory) is skipped, counted, and its domain kept. Once tags.yaml
is imported, a domain's Tag that no longer exists is dropped and the
config saved without it (`unknown_tag_names`).
Linking Downloads to domains reads first (the domains once, not one query
per RSS Download) and opens a short write only when there is something to
link; the config is then saved with the links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test_bookmarks_config_import starts the app like its siblings
It imports a config without `async_client`, so it fails (no
shared_ctx.configs_imported) whenever it is the first test on its
worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* domains.yaml: converge without a save; no links while entries are skipped
A save after the import is refused while an entry is skipped, so the
import must converge without one:
- a removed domain with Archives that is already reset is left as it is
(prepare reads `import_db_rows`); before, every import reset it again;
- after prepare, the framework drops changed rows that now match the
database (a dropped unknown Tag), so they are not written again.
Linking Downloads changes domains.yaml after its version is recorded;
while entries are skipped the save is refused, the database stays newer,
and the next startup would save over the skipped entries. Linking now
waits for an import without skipped entries.
An entry that is not a mapping is skipped and counted, instead of failing
the whole file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml imports by diffing
repos.yaml declares one section, its Repos keyed by repo_url_key (another
URL of the same Repo is that Repo), comparing name, description,
directory, Tag, mode, branch, submodules, frequency and clone token (a
missing directory or clone token keeps the database's). An unchanged
import writes nothing (every import rewrote every Repo and its
Download).
An invalid entry is skipped and counted, and its Repo is kept; before,
a Repo whose entry failed validation was deleted. New Repos are created
inside the import's one write: recurring_download (via
get_or_create_download and _set_repository_download) takes commit=False,
and the URL is not looked up again for each new Repo (O(N^2)).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* repos.yaml: a branch URL is keyed by its Repo's URL
file_rows keyed the raw URL, but stores parse_repo_url's URL, which
turns a forge's branch URL (GitHub /tree/<branch>, GitLab
/-/tree/<branch>) into the repo URL and a branch. The entry looked like
a new Repo: its insert hit the existing Repo's name (rolling back every
import of that file), and an invalid such entry did not protect its Repo
from deletion. Rows and invalid keys now use the parsed URL; a URL that
cannot be parsed is skipped by its raw key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Benchmark every database config import (#865)
An opt-in test (WROLPI_CONFIG_BENCH=1) that builds a large library in bulk,
writes each config with its own dump_config, then times every import in
startup order: a steady pass (the database already matches) and a rebuild
(config-owned rows deleted, everything restored from the configs). It
prints seconds, SQL statements and writes per import, and asserts the
rebuild restored every row so no timed import silently skipped its work.
WROLPI_CONFIG_BENCH_SCALE multiplies the row counts.
The baseline for the config import rework.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Database configs carry a version the database keeps too
Step 1 of the config import rework. A database config (tags, downloads,
channels, domains, playlists, repos, recently viewed) and the database now
each carry an integer version, so an import can tell which side is newer
without a clock (a Raspberry Pi's resets at boot):
- config_version (new table) holds the database's version of each config.
- An after_flush hook adds one to it for every config whose rows a flush
changed, in the same transaction. Each config registers what it owns
(register_ownership): models, optionally only some columns, optionally
only some rows (a Collection's kind; a Download's Collection's kind).
Raw SQL writes to owned tables count explicitly.
- dump_config writes the database's version, read with the rows it saves;
save() no longer adds one for these configs.
- An import compares versions first: file == database or newer imports
with deletes; file older (the database changed and was not saved before a
restart) saves the database instead of undoing that change; no database
row (a config this code never imported) imports without deletes. The
import records the file's version last.
- The Import button and backup imports apply the file whatever its version
(import_forced).
Importers only gate their existing deletes on the decision here; they are
otherwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions: keep unsaved rows after a no-delete import, count every owned write
Review fixes:
- An import without deletes (no database version yet) no longer records the
file's version, which let the next startup delete the rows it had kept.
The version stays unknown, and the database is saved into the file instead;
a successful save records the version. A save after a partial import is
refused, so the version stays unknown until an import applies the whole file.
- An import removes the database's version while it runs, so its own writes
do not count: one that fails partway no longer looks newer than the file
and gets saved over it on the next startup.
- Playlists imported without deletes keep their database items in order; the
file's other items are added after their neighbor from the file.
- A Download leaving a Channel or Repo counts for that Collection's config
(collection_id loads its previous value).
- Bulk writes count: untagging, retrying Downloads, deleting a duplicate's
Download, removing many items from a playlist. Deleting Downloads by id
counts channels.yaml/repos.yaml only for their Downloads.
- Only a playlist's items count for playlists.yaml (author and subject
Collections have items too). Collection kinds are looked up once per flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Config versions count only the columns each config saves
A Repo's fetch (head_sha, last_fetch), a Channel's refresh (video_count,
info_json), a Collection's counts, and a Download attempt (attempts, error)
counted as config changes, so the next import of that config saved instead of
importing: a hand edit to repos.yaml was overwritten after any fetch. Each
model now registers only the columns its config saves.
domains.yaml saves its domains' Downloads, which now count for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Tests fail when a request writes outside request.ctx.db.write()
A request's transaction begins deferred. A write in it after a read fails
instantly ("database is locked") when another connection wrote in between,
and a write left for the response middleware's commit can be lost after the
response was sent. #846-#853 and #863 moved handlers' writes into
request.ctx.db.write(); this keeps it that way.
Under pytest, a listener records every INSERT/UPDATE/DELETE/REPLACE made
while a request runs (or in a task it started) outside RequestDB.write()
or get_db_session(commit=True), and the autouse no_deferred_request_writes
fixture fails the test. allow_deferred_request_writes opts a test out.
The test-only response listener now runs switches outside the request,
as production's perpetual process does.
It found three more:
- DELETE /api/zim/<ids> deleted the .zim file, returned 204, then lost the
row's delete when the middleware's commit failed under contention. It
now writes in request.ctx.db.write().
- Viewing a file recorded the view in a background task on the request's
Session, after the request ended. set_file_viewed opens its own write.
- tags.import_config (startup and config import) read, then wrote, in a
deferred get_db_session(). It now begins as a writer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tasks don't inherit a declared write; tags import change backed out
From review of #864:
- background_task's task copied its creator's Context, including
_write_intent when started inside get_db_session(commit=True) or
RequestDB.write(), so the guard ignored the task's own writes for its
lifetime. The task now starts with no declared write. That exposes
fetch_channel_source_id (started by the channels config import): it read
the Channel, asked the remote site for its source_id, then committed in
the same deferred transaction. It now reads, fetches with no
transaction open, then writes in a short get_db_session(commit=True).
- tags.import_config: commit=True held the write lock through its directory
walk, and its exit commit saved changes the need_commit gate used to
drop. Reverted to master; splitting its reads from a short write is its
own change. The two tests that run it through a request are marked
allow_deferred_request_writes until then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Remove a file's Tags Directory links when WROLPi deletes it
When WROLPi deleted a tagged file (a forced delete, duplicate Video cleanup, or a poster conversion), the
source was unlinked but its Tags Directory hardlink was not. The link became the only copy, so the tags sync
refused to delete it and warned on every sync. The links of a file are now removed before WROLPi removes it,
and only if the link is the same file as its source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove Tags Directory links on path deletes; keep a valid poster's link
The file browser deletes by path (`lib.delete`), which unlinks the sources before any FileGroup is
deleted, so its tagged files still left orphaned links. It now removes the links of the files it is about
to delete, or of every FileGroup in a deleted directory.
A poster is no longer unlinked from the Tags Directory unless it is actually removed: a valid JPEG that is
not named .jpg is kept with its link, and a converted poster's links are removed only once the conversion
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg exits 0 but prints "non monotonically increasing dts to muxer"
from the null output when two frames share a timestamp, collapsed into
"Last message repeated N times" when piped. Any stderr failed the
completeness check, so such videos (common on X) were rejected as
incomplete on every retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding, importing, changing, deleting and updating a Repo read, then
wrote, in the request's deferred transaction, so they failed with a 500
("database is locked") when another connection wrote in between. They
now write inside request.ctx.db.write().
Importing reads the clone with git and walks it for its size; those run
before the write transaction, so it is not held for them. Everything
that can refuse the import still runs before the clone is changed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between. They now write inside
request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/zim/<id>/auto_search, /tag, /untag, /subscribe and
DELETE /api/zim/subscribe/<id> read, then wrote, in the request's deferred
transaction, so they failed with a 500 ("database is locked") when another
connection wrote in between. They now write inside request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>