Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the FX-8320 the first token took over a minute, and Sanic's default
60-second RESPONSE_TIMEOUT killed the stream before anything was sent.
Raise the timeout above the chat loop's own 600s hard ceiling, and emit
a status event before every model call so the stream is never silent —
for proxies and for the user staring at a slow machine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live testing surfaced three ways small models went wrong summarizing a
library: they gave up after one empty search (misspelled channel names
found nothing), so zero-result searches now carry a retry hint and the
Research prompt requires list_collections before channel/site questions
and search_videos(channel_id) before claiming a channel is summarized;
they presented the familiar YouTube source URL instead of the WROLPi
link, so listings omit the source URL (archive detail keeps it); and
they invented links for channels, so channel collections now carry
their real /videos/channel page link. Search descriptions state that
search_str is optional for browsing the newest items.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three findings from running Qwen3-1.7B against the docker stack:
the llm image lost libgomp when the build tools were removed
(llama-server needs the OpenMP runtime); the Help prompt let the
small model answer from memory instead of the docs, so it now
requires search_help before any answer; and help search results
called the slug "doc" while get_help_doc's argument is "slug", so
the model invented slugs — the names now match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
publish_ai_models.py produces the CDN artifacts for every AI_MODELS
entry — GPG-signed .meta4 sidecars (reusing the map trust chain) and
the signed ai/models.json manifest — and uploads them with s3cmd.
The bundled catalog now carries the real sha256/size of the two Qwen
GGUFs, which match Hugging Face's LFS records for the Apache 2.0
unsloth quants, and whose signed meta4s verify against the committed
public key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every MCP tool is now one /api/ai call with minimal text rendering;
the formatting, link building, and truncation logic it used to carry
untested lives (tested) in modules/ai. Tool names are unchanged for
external clients; a source-scan test guards the names and that the
proxy only reaches /api/ai. New lean blueprint endpoints back the two
tools that had no equivalent: GET archives/{id} (detail + snapshot
history) and GET videos/{id}/comments (paged author lines). Paged
content is read up to the old 50k-char bound. get_inventory_items now
takes the inventory slug (inventories are config-only; the old
numeric-id endpoint no longer exists). This also brings wrolpi_mcp
into version control (venv/ is already gitignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
install_llama_cpp.sh builds llama-server from the SHA-pinned v0.3.0
source tarball (single source of truth: on-device upgrade, Pi image,
Live ISO, and the new docker/llm container all call it; version-marker
makes re-runs no-ops). start_llama_server.sh refuses to start unless
ai.yaml enables AI and the active model file exists — which also covers
unmounted media. wrolpi-llm.service is installed but deliberately
never enabled at boot: the Controller starts it on demand (registered
in managed_services; the llm compose service follows the samba
pattern), and the API's perpetual idle-unload worker asks the
Controller to stop it after the configured idle minutes. llama-server
is localhost-only natively, internal-network-only in docker, and never
proxied by Caddy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ai.yaml is the source of truth (opt-in: enabled=False by default) with
active model, idle-unload minutes, context override, and tool-call
bound. The model catalog follows the map pattern: a bundled constant
keeps the Manage tab working offline, the CDN manifest (ai/models.json
+ detached GPG signature) wins when reachable. GGUF downloads clone
the map-extract flow — the per-file meta4 is GPG-verified once and the
same bytes handed to aria2c, so the verified hash is the enforced one;
unverified downloads are refused. Models land in <media>/ai/models/
(gated by the existing media_mounted download rules). Manage endpoints
(catalog with RAM-tier recommendation, settings) live under
/api/ai/manage/ which the future agent loop excludes from its tool
catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Search the wrolpi-help markdown in-process: files are parsed on demand
and cached by mtime; title matches outrank headings outrank body. No
database table — the corpus is dozens of small files. Pages are read
as paged markdown. The docker api container gets a read-only bind
mount of the help docs at the native /opt/wrolpi-help location; a
missing help directory degrades to empty results with a message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proxy the Controller under /api/ai so the model sees one API: an
aggregate /status to start every diagnosis, lean service list, recent
service logs (lines clamped), and disks + SMART. One controller_client
helper resolves the Controller base URL (docker service name vs
localhost). Every proxy degrades to an `errors` entry when the
Controller is unreachable — never a 500.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lean, flat, LLM-shaped read-only endpoints over the library: global,
video, archive, and doc search; paged captions, archive text, and zim
entries; collections, inventories, and a media-directory text reader.
Links are server-computed; big content is paged, never truncated.
This ports the formatting layer from wrolpi_mcp (untouched for now)
into tested code. First phase of ai.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* files: SQL FileGroup compare, stem column, drop unused discovery path
Refresh walked the tree twice (find, then stat every unchanged group),
and a two-directory refresh scanned the whole media directory. Compare
now diffs in SQL on a TEMP table that already has mtime, accepts multiple
roots, and groups by a stored stem that is versioned so a suffix-rule
change cannot delete-and-reinsert tagged files.
Deletes refresh_discover_paths and /api/files/refresh_progress, which
nothing in production called. Video/doc/zim modelers use the same
(no domain row OR indexed != True) gate.
* files: keep stem current on reorganize; NUL-safe find parse
Reorganize wrote the new primary_path but not stem, so the next refresh
treated the FileGroup as deleted+new and dropped tags. GNU find now
emits NUL-delimited records so a tab or newline in a filename cannot
abort a global refresh. Stem backfill skips the table scan once the
algorithm version matches, and runs off the event loop. An empty roots
list is a global refresh, not invalid SQL.
POST /api/videos/channels failed with "database is locked" on
`INSERT INTO collection` on the docker WROLPi while background workers
(config saves, refreshes) were writing. The handler wrote through the
deferred request session: `create_channel` reads (the URL/name/directory
conflict check) and then INSERTs, and SQLite refuses that mid-transaction
lock upgrade instantly when another connection wrote in between --
busy_timeout is skipped for upgrades.
Same shape and same fix as POST /api/download (daf2ba39): the handler
now uses `get_db_session(commit=True)`, which begins IMMEDIATE so the
concurrent writer waits its turn, and which commits before the refresh
is fired so the refresh does not wait on a lock the request still holds.
The regression test commits a write from a second connection between
the handler's conflict check and its INSERT. The existing
`write_lock_held_briefly` helper cannot reproduce this: the Sanic test
client takes over a second to start the app, so its lock is released
before the handler runs. Before the fix the test fails with the
production error verbatim.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Jobs queue and video transcode from the video page
Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time. Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.
Add a transcode Job and POST /api/videos/<id>/transcode. ffmpeg reports
progress via -progress pipe:1. The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.
Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode). The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: move Delete into the Edit dropdown
The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist. Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: Edit > Edit to change a video's title
PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow. Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: edit the description in Edit > Edit
PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it. The
modal sends only what changed and refetches the description after saving.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode modal: allow a container-only remux
Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start. The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* WROL Mode: refuse video edits and transcodes past the API
The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live. The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: verify the output before replacing the original
After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.
The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete
Review fixes. The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock. enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs. JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.
The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original). A crash in between leaves
both files with the database pointing at a real one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: the Edit menu owns the transcode Job; polling survives Close and errors
Review fixes. The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed). While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits. A failed poll
request is retried instead of ending the polling. getJobs checks the
response like its siblings. Design narration trimmed from comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: audio files, and Remove video to turn a video into an audio file
video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused. Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one. faststart only for mp4/m4a. Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).
The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video page: yellow Edit button
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost
Review fixes. The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index. Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation). Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Jobs: _claim_job docstring states the invariant, not the old race
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode: fragmented mp4 option for very long videos
A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it. `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly. mp4/m4a only; a remux, so it takes seconds per GB.
Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Transcode modal: the Fragmented label says audio for audio output
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section
Editing a title or description no longer overwrites yt-dlp's keys. The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file. An empty custom description means none.
Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download. FileGroup.get_wrolpi_json reads it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs: browse walks the file_group date index instead of probing every doc
The unfiltered /api/docs/search drove its join from `doc`, so SQLite probed
every doc's file_group row for the sort key before returning the first page.
With 20k docs scattered through a 10 GB database on a spinning disk, the first
request took about a minute cold (0.07s warm).
Use the same CROSS JOIN shape as the archive browse: file_group is the outer
table, SQLite walks file_group_published_datetime_idx and probes doc's covering
unique index on file_group_id. Both sides stay index-only and the scan stops at
the page limit. Measured cold: ~60s -> 0.6s, identical rows.
Filtered searches and the size/title orderings keep the ORM query.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: trim browse comments, stabler plan assertion
Review feedback: drop the benchmark narrative from docstrings, state only the
non-obvious constraint at the call site, and assert the doc covering-index probe
by shape rather than by autoindex name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
* videos: codec download preferences with transcode and strict-fail options
Add video_codecs/audio_codecs ordered preference lists to the video,
channel, and RSS downloaders, alongside the existing resolution
selector. Codec preference is codec-major: a preferred codec at a
lower resolution beats another codec at a higher resolution.
When 'transcode' is enabled, a downloaded file whose streams do not
match the preferences is re-encoded with ffmpeg after download (only
the mismatching stream; targets are extensible maps, currently h264
and aac/opus/mp3). When 'strict_codecs' is enabled and transcode is
not, a download with no matching format fails permanently instead of
falling back.
Empty codec lists produce a byte-identical yt-dlp command to before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: honor strict_codecs when transcode cannot produce the preference
Review fixes:
- strict_codecs is now suppressed only when transcoding can produce a
preferred codec for every non-empty preference list (new
transcode_can_satisfy_codecs); av1-only preferences with transcode
enabled fail under strict instead of silently keeping the file.
- No partial transcode: an unfixable video mismatch blocks an
audio-only conversion (and vice versa); the mismatch falls through
to strict/keep.
- fsync the transcode output before rename, and the directory after,
so power loss on a Pi cannot leave a truncated final file.
- mp3 yt-dlp pattern matches mp4a.6B as well as mp4a.6b.
- ffmpeg maps use 0:V:0 (excludes attached pictures) and 0🅰️0?
(tolerates missing audio).
- Strict toggle is no longer disabled by transcode in the UI; popup
wording corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: serialize transcodes machine-wide via shared_ctx lock
Sanic runs multiple worker processes, so an asyncio primitive cannot
serialize transcodes across download workers. A multiprocessing.Lock
in shared_ctx now guarantees only one ffmpeg transcode runs at a time
on the machine; waiting workers poll the lock asynchronously so the
event loop stays responsive while a transcode holds it for hours.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: transcode-lock review fixes
- Document that a crashed lock holder requires an API restart; this is
accepted rather than adding owner-death recovery.
- Move the 'Transcoding' user notification to after the lock is
acquired, so it fires when work begins rather than before a
potentially hours-long wait.
- Soften the lock test docstring to what it measures (in-process
wait/release); trim the acquire docstring to the polling rationale.
- Give transcode_lock its own section header in contexts.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
test_search_videos_does_not_block_event_loop asserted a wall-clock
threshold (echo < 0.4s), which fails on loaded CI machines from
scheduler noise alone. Replace the timing with ordering: the mocked
search parks its thread on an Event, the echo is sent only once the
handler is provably inside the query, and the test asserts the echo
completes while the query is still held open. Events are only waited
on off the event loop, so a regression fails the ordering assertion
instead of deadlocking.
Verified both directions: passes repeatedly against the real endpoint,
and fails when the endpoint is changed to run the search on the loop.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page
Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:
* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending
(the normal state) this froze the loop for a second at a time; about a third
of all requests, including trivial ones, took ~1,000 ms. It now polls
`is_set()` and returns immediately.
* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
ran synchronous SQLite queries on the event loop, so one slow search stalled
every other request on that worker. They now run via `asyncio.to_thread`.
* The unfiltered Videos page (`POST /api/videos/search` with only an order)
used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
every video's index entry before LIMIT applies: seconds cold on a large
library on a spinning disk, and ~10 s hot for a deep page. It now seeks each
exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
rows and merges them, with the total from a separate count cached for 60 s.
Results are identical to the generic query.
Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a
gitignored JSON file; an example is included.
Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve the video description from its own endpoint instead of Video.__json__
`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk. The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays. The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.
Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions. The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored. The player page fetches it through
`useVideoExtras` alongside the comments.
Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: thread-local zim sessions, fast-path equivalence test, comments
* Zim search, single-zim search, and search estimates run libzim off the event
loop, each opening its own session inside the worker thread rather than
passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
query and asserts identical rows and paging for both date orders. The one
intentional difference is documented: past the end, the fast path still
reports the true total.
* switch_worker returns before its try block so the empty path does not clear
the event; its comment states the invariant only. The copied "synchronous
SQLite" comments are reduced to one line in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* videos: a collaboration video's channel directory conflict killed the download
YouTube collaboration videos appear in every collaborator's feed but their
metadata names only the primary uploader's channel. When that channel's
computed directory already belonged to an existing Channel that no longer
matched by source_id/url/name, channel creation raised
ChannelDirectoryConflict and the download failed unrecoverably. Now the
channel that owns the directory is used instead of creating a conflicting one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: trim comments that restated the code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
NAME_PARSER's suffix whitelist had no audio suffixes (and no mkv/m4v),
so a correctly-named audio file got no published date and sorted to the
very end of its Channel -- which reads as missing. The whitelist now
carries the media suffixes a Video FileGroup can lead with.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* videos: an uploaded audio file never appeared in its Channel
The upload flow asks Video.can_model, which only accepted video/*, so an
uploaded .flac was indexed with no Video row -- and the batch modeler's
`indexed != True` gate then excluded it on every later refresh, leaving
the file permanently invisible in its Channel.
can_model now accepts audio/* like the batch modeler always has, and the
modeler gates on the Video row being missing (mirroring the doc_modeler
fix for the same class of bug) so already-stuck files heal on the next
refresh. Audio playlists (.m3u detects as audio/x-mpegurl) are excluded
from both paths and unmodeled by video_cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: widen the playlist denylist, prove modeler termination
The denylist gains audio/x-scpls and audio/scpls -- .pls detection
varies by libmagic version, and any audio/* playlist type must be
refused wherever it comes back. Modeling one file is extracted as
_model_video so a test can force persistent failure and prove the loop
terminates; failed_ids is recorded before the PYTEST re-raise so no
failure path can re-select an id forever. Playlist mimetypes are bound
through json_each like the rest of video_cleanup, and the comments
state the invariants instead of the history.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: the playlist test must not depend on libmagic's m3u opinion
CI's libmagic detects .m3u as text/plain where a Mac's says
audio/x-mpegurl. The invariant is that a playlist mimetype is never
modeled by either path, so the test forces the mimetype like the other
denylist tests instead of asserting what detection returned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The channel downloader refreshes a Channel's files before downloading,
already passing send_events=False, but the parameter was never plumbed
into the file worker, so every pre-download refresh notified the user.
FileTask now carries send_events and handle_refresh skips the
Refreshing/Refreshed events when it is False. User-requested refreshes
and global refreshes still send Events.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ui: a preview map that waited for a container it could never see
Both map preview modals -- the per-region eye button and "View All Regions"
-- opened to an empty box. Each renders a div into a Modal and hands it to
MapLibre from an effect gated on `open`, reading the node out of a `useRef`:
if (!open || !mapContainer.current) return;
Mantine's Modal mounts its body one commit after `opened` flips true, so on
the render that opens the modal the ref is still null. The effect returns,
nothing it depends on changes again, and it never runs a second time. No
console error, no rejected promise, no canvas -- just an empty div.
Semantic UI's Portal mounted the body in the same commit that opened it, so
the ref was attached by the time effects ran. The effect bodies are
byte-identical across that migration; the modal library is the only variable,
confirmed against a device still running the older build where both previews
render.
A callback ref instead, so the effect re-runs when the node arrives and this
stops depending on any modal's mounting schedule. The async source lookup
also gets a cancelled flag: it could resolve after the cleanup ran, building
a map nothing would ever remove and leaking a WebGL context per open.
The tests drive the real page and assert a map was built against a node that
is in the document -- the outcome, rather than the mechanism that quietly
changed underneath it. Seven of the eight fail on the code before this.
* build: docker had no world underneath its map regions
Caddy serves /blobs/map-overview.pmtiles out of /opt/wrolpi-blobs, and that
blob is the world at zoom 0-6 -- what a regional extract sits on top of.
Without it a user who subscribes to a few regions gets those regions floating
on an empty background everywhere their data does not reach. Only a full
planet file, at 134GB, makes it unnecessary.
Raspberry Pi and Debian installs get the blob from pi-gen, the live-build
config, and upgrade.sh. Docker got nothing: the compose file bind-mounted a
git-ignored host directory over /opt/wrolpi-blobs, and a bind mount replaces
the image's directory rather than merging with it, so the path existed and was
empty on every Docker WROLPi. Verified against a running deployment, where
that URL returned 404 while the same URL on hardware returned 206.
So the web image fetches the blob at build time, as the other three builders
do, and the mount that would hide it is gone. `curl -f` so an HTTP error
fails the layer -- an image that builds without the blob is this same bug,
except silent at build time too.
The tests read the deployment files rather than a built container: building
to check costs a CDN download and several minutes, and both things that broke
-- nothing fetches it, a mount hides it -- are visible in the source.
* ui: cancel the load handler too, and make the cancel test falsifiable
Review follow-ups on the preview modals.
The `cancelled` flag was checked before constructing the map but not inside
the `load` handler it installs, so cleanup landing between the two left that
body drawing into a map `remove()` had already torn down. Guarded, mirroring
the `destroyed` pattern in MapViewer.js. The source promise also gets a
`.catch`: the helper cannot reject today, but nothing says it never will, and
an unhandled rejection here would be as quiet as the bug this all started as.
Two test corrections, both cases of an assertion claiming more than it read:
The region-drawing test spied on `addSource` AFTER construction and then
re-fired `load` by hand, so its count came entirely from that manual call and
said nothing about the production path. FakeMap now records the sources it is
given, and the assertion reads what the component's own `load` drew -- each
region once, in order, and none for the entry with no bbox.
The new cancel test passed with the guard deleted, which is worse than not
having it: Mantine had not mounted the modal body when it closed, so the
effect had never run and there was nothing to cancel. It now waits for the
effect to reach the source lookup before closing, and ships beside a control
that releases the same held promise WITHOUT closing and requires a map. The
control is what makes the zero meaningful; verified that removing the guard
fails the cancel test while the control still passes.
* test: do not leave videos_destination as an invalid template
test_format_videos_destination deliberately sets %(channel)s to prove
unknown placeholders fail, then never restored the global WROLPi config.
On xdist workers that later ran channel_factory without async_client
re-init, the polluted template caused KeyError/FileNotFoundError in
test_format_video_filename_uses_channel_name_over_info_json.
* test: stop search_suggestions from flaking empty domains on CI
A different failure than videos_destination pollution: under xdist load,
search_suggestions sometimes returned no domains for a valid query while
channels still matched. Harden the path by:
- running channel/domain/author/subject searches sequentially on the shared
Session (asyncio.gather is not safe with one SQLAlchemy session)
- giving archive_factory a URL when only a domain is set so validate() does
not clear collection_id via apply_domain()
- rewriting test_search_suggestions without hardcoded Collection ids and with
unique names that will not collide with other tests
All three bugs confirmed; the two suggestions taken.
[bug] `video_modeler` awaited ffprobe -- a subprocess per video, up to 20
per batch -- inside its `commit=True` session. With IMMEDIATE as the
default that holds the write lock for the whole batch, which is the same
starvation this branch is curing in `video_cleanup`. (It was already
partly true before: the batch's first `session.flush([video])` took the
lock ahead of the remaining ffprobes.) The batch is now read, ffprobed
with no transaction open, then written. `get_or_create_ffprobe_json` in
modules/videos/common.py is the path-only entry point that makes that
possible; `Video.get_ffprobe_json` delegates to it and keeps appending
the cache file to its FileGroup. `validate()` -- and the poster
generation it can trigger -- is still inside the write transaction, and
now says so.
[bug] `test_download_dispatched_outside_immediate_transaction` probed
`getattr(_immediate_txn, 'active', False)`, which a ContextVar never has,
so it passed unconditionally. It now probes the actual write lock from a
competing connection under `production_like_sessions`. Mutation-checked:
moving the dispatch back inside the claim transaction fails it.
[bug] `asyncio.create_task` snapshots the current Context, so a flag held
for a write session's lifetime was inherited by every task spawned inside
it -- and the parent's `reset()` cannot reach a child's copy, leaving
that task issuing BEGIN IMMEDIATE for its *read* sessions.
`create_downloads` spawns `dispatch_downloads` exactly that way, so every
user-created download hit it. The flag now lives only across the
session's opening `connection()` call, which also means the write lock is
taken when the session opens rather than at the caller's first statement.
[suggestion] `renew_recurring_downloads` scanned every recurring Download
under the write lock, including the usual cycle where nothing is due. It
reads first now and takes the lock only for the rows that change,
re-checking status since the dispatcher may have claimed one in between.
[nit] Parenthesized the directory OR in the claim join.
`probe_write_lock_is_held` moved to conftest alongside
`write_lock_held_briefly`; three suites use it now.
Suite: 1712 passed, 5 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three "database is locked" failures on the docker WROLPi, one cause each:
* `INSERT INTO download` from POST /api/download -- a user creating a
video download while a refresh ran (2026-08-06 14:06:46);
* six `INSERT INTO file_group` from `upsert_file`, uploads landing in
the same three-minute window;
* `UPDATE download SET status` from `renew_recurring_downloads`.
All three are the deferred read-then-write trap: a transaction that
begins as a reader and upgrades at its first write is refused
*instantly* when another connection wrote in between, because
busy_timeout is skipped for lock upgrades. `get_immediate_db_session`
existed to avoid it, but an opt-in helper only works if every author
remembers, and read-then-write is the ordinary shape of a write. So it
is gone: `get_db_session(commit=True)` -- the existing declaration of
write intent -- now begins IMMEDIATE. Rails 7.1 and Django 5.1 made the
same call, defaulting SQLite transactions to IMMEDIATE rather than
asking application code to opt in.
That default is only safe if write transactions stay short, and one did
not. `video_cleanup` ran its whole Channel-claiming join --
`UPDATE video ... FROM channel ... LEFT JOIN file_group`, O(channels x
file_groups) -- inside one write transaction: 110 to 130 seconds each of
the eight times it appears in the log, on a library with ~1500 channels.
While it ran, *every* writer on the box failed, including ones that
began IMMEDIATE, because the hold far exceeds the 30s busy_timeout. The
14:04-14:06 failures above sit inside its 130-second window. The join
is now a read -- WAL readers never block writers -- and only the
resulting UPDATEs take the lock, 500 rows per transaction.
`claim_videos_for_channels` had the same join and the same fix; channel
import calls it with every channel in channels.yaml.
Also here:
* `renew_recurring_downloads` was a read session with an inner commit;
it is a write session now. Its computed `next_download` values now
persist even when nothing was renewed -- that is the point of
computing them.
* POST /api/download no longer writes through `request.ctx.session`.
That session is committed by the response middleware, so a writer
would hold the lock across the whole request; a write session
commits before the handler returns, ahead of the background dispatch
it fires.
* `_immediate_txn` is a ContextVar, not a threading.local. The
writers that set it include coroutines, and one asyncio thread runs
many of them interleaved.
* `write_lock_held_briefly` moved from the move tests into conftest,
since two suites now need it.
Both regression tests were verified by mutation: reverting either site
alone fails its own test, and the API one fails with the production
error verbatim.
Sources: fractaledmind.com/2023/12/11/sqlite-on-rails-improving-concurrency,
tenthousandmeters.com/blog/sqlite-concurrent-writes-and-database-is-locked-errors,
berthub.eu/articles/posts/a-brief-post-on-sqlite3-database-locked-despite-timeout
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the flush to immediately before FileGroup.move so it covers every
deletion queued in this function, not only the one found by
get_by_path. The earlier missing-file deletions were already reaching
the DB -- the get_by_path query autoflushes them -- but that is an
incidental property of the query sitting between them and the rename,
not something the rename should depend on.
Also assert the kept Video's old path is gone, so the test reads a move
rather than a copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`delete_duplicate_videos` deletes the smaller copy sitting at the
destination path, then renames the copy it keeps onto that path.
`FileGroup.delete` unlinks the files right away but only queues the row
deletion on the session, and SQLAlchemy emits UPDATEs before DELETEs
within a flush -- so the rename's UPDATE of `primary_path` collided with
the row that was about to be deleted:
sqlite3.IntegrityError: UNIQUE constraint failed: file_group.primary_path
The whole download transaction rolled back, so a re-downloaded video
lost its DB record while its files stayed on disk (and the previous
copy's files were already unlinked).
Flush the delete before claiming the path. The earlier missing-file
deletions in the same function never hit this because the `get_by_path`
query between them autoflushes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The yt-dlp `-f` selector was built by joining every selector of every
requested resolution with a comma. In yt-dlp a comma means "download all
of these formats"; `/` is the fallback operator. The resolution map is
ordered best-to-worst and its entries already use `/` internally, so the
comma was never the intent.
With the default resolutions this handed yt-dlp ~45 comma-separated
selectors and asked it to download every one that matched. A 720p video
matched 8 of them (including duplicates, which yt-dlp does not dedupe), so
the whole per-download pipeline ran 8 times against the same output path:
subtitles downloaded, thumbnail downloaded, info.json written, subtitles
converted, thumbnail converted, existing subtitles deleted, repeat. The
duplicate work was invisible because each pass overwrites the last.
Join with `/` so the resolution list behaves as the preference chain it
reads as.
modules/inventory/common.py: _normalize_items now strips leading/trailing whitespace from any string-valued item field before persisting an inventory (name, category, subcategory, location, etc.). Non-string values (numbers, None) are left untouched.
Closes#52
Triage of Greptile findings on the default-Zim / configurable-directory PR:
- Stale config on restart (P1): settings update() only queues an async
background_save, and the restart switch has no ordering guarantee (the switch
worker pops LIFO), so start_kiwix_serve.sh could read the old zims_destination.
Save the config synchronously before activating the restart switch.
- YAML null (P1): read_config_value.sh now treats an unquoted null/~ as unset
(returns the default), matching PyYAML/the Python config layer. A quoted
"null" remains the literal string.
- Path traversal (P1/security): the settings API now rejects a zims_destination
containing "..", and start_kiwix_serve.sh defends in depth by falling back to
`zims` for any absolute/traversal/empty value, so Kiwix can never scan or
rewrite library.xml outside the media root.
- Reproducible build (P2): pin the zim-tools image to 3.7.0 instead of :latest.
Tests added for null handling, traversal rejection (API + script).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related gaps in how WROLPi serves Kiwix content:
1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
Kiwix. Every image now ships a small built-in default Zim that Kiwix falls
back to, telling the user to download real Zim files.
2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
`zims_destination` (which the Python side already honors).
Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
non-Python scripts can reliably read top-level wrolpi.yaml scalars. Also a
config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
(ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback. It needs no
config read: it already scans the whole mounted media dir, and
zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
serves the new directory without a reboot.
Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A refresh on a large library aborted the whole doc_modeler run on the
first lock error, so the ~20k stranded docs never got modeled.
doc_modeler held one write transaction across a whole batch of 10 docs
while doing slow per-doc extraction (a PDF/epub can be tens of MB). When
another connection committed mid-batch, the deferred transaction failed
to upgrade to a write with SQLITE_BUSY_SNAPSHOT ("database is locked",
which busy_timeout does not absorb -- the same trap that motivated the
short-transaction rewrite of apply_indexers). The per-item error handler
then did `logger.error(f'... {file_group}')`, which lazy-loaded an
attribute on the now-rolled-back session and raised a *second* exception,
escaping the handler and aborting the entire modeler.
Restructure:
- Pick each batch of candidate ids in a short read-only transaction, then
model each doc in its own short write transaction. A lock/error on one
doc rolls back only that doc and cannot poison the rest of the run.
- Retry a doc a few times on "database is locked" -- a fresh transaction
gets a fresh snapshot and succeeds.
- Log the plain file_group id on failure; never touch the ORM object on a
possibly rolled-back session.
- Track only *failed* ids for the termination guard (successes are
excluded by the Doc.id IS NULL gate), keeping the notin_ set small and
well under SQLite's bound-variable limit.
Adds test_doc_modeler_survives_database_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: the new `Doc.id IS NULL` gate keeps matching a
doc whose _model_doc fails without persisting a Doc row. With
>=DOC_PROCESSING_LIMIT such docs, every batch re-selected the same failing
files, `processed` never dropped below the limit, and the loop hung --
starving later docs. Reproduced with a patched _model_doc that always
raises.
Track attempted FileGroup ids per run and exclude them from the query, so
the candidate pool strictly shrinks and the loop always terminates. Failed
docs are retried on the next refresh (unchanged behavior).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc_modeler gated its query on `FileGroup.indexed == False`, but
`indexed` is a full-text-search flag that `apply_indexers` sets on every
FileGroup during a refresh. Because the refresh pipeline runs
apply_modelers -> apply_indexers, any doc file that reached apply_indexers
without first receiving a Doc row (modeler errored, was interrupted, or
files imported already-indexed) got `indexed=1` and was then excluded by
the `indexed == False` gate forever -- leaving it with model=NULL, no Doc
row, and invisible in /api/docs.
On one library this stranded 20,795 PDFs/epubs/mobis (only the 10
individually-tagged docs, modeled via the per-file do_model path, showed
up). No doc-mimetype files were left with indexed=0, so nothing would
ever recover them.
Gate the modeler on "no Doc row yet OR explicitly flagged for
re-indexing" instead, so already-indexed docs are picked up on the next
refresh while preserving re-index behavior and skipping already-modeled
docs. Self-healing: a global refresh now models the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A corrupt or partial *.zim (e.g. an interrupted download) is rejected by
kiwix-manage ("Cannot add zim ..."). start_kiwix_serve.sh could then leave
Kiwix down and reported a misleading "Could not find any Zim files" — and
if the bad zim was the only one, exit 2 killed the service with no restart.
- Import each zim independently; log and skip a failed add so the remaining
valid zims are still served.
- Always ensure a library.xml exists (empty if needed) and exec kiwix-serve
unconditionally, so a corrupt/absent zim never stops the server starting.
- Fix the mount-wait loop, which never incremented COUNT (infinite wait
instead of a 30s timeout).
- Add MEDIA_DIRECTORY override so the script is testable.
- Add Restart=on-failure to wrolpi-kiwix.service (matches wrolpi-api.service)
so transient failures self-heal.
- New test harness runs the script with fake kiwix-manage/kiwix-serve shims.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
- strip_youtube_caption_positioning now writes to a temp file in the same
directory and os.replace()s the original, so a failed/partial write
(e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
which parses the hostname and matches youtube.com / youtu.be (and
subdomains), rejecting lookalikes like youtube.com.example.org and
notyoutube.com. Add a parametrized test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (via yt-dlp) stamp every cue timing line
with `align:start position:0%`, pinning the on-screen text to the
bottom-left of the player. This strips that exact cue setting from the
downloaded .vtt files so browsers render the captions centered.
- add strip_youtube_caption_positioning() to wrolpi/captions.py — a
targeted text replacement (not a webvtt round-trip) so the `<c>`
word-timing tags in the cue text are preserved; matches only the exact
auto-caption signature, so deliberately positioned cues (e.g. with a
`line` setting) are left untouched; idempotent.
- call it from VideoDownloader after normalize_video_file_names, gated on
a youtube.com / youtu.be source URL. It runs before the FileGroup is
created, so the first index reads the corrected file — no re-index
plumbing needed.
Complements the client-side fix in #543 (which centers captions in the
player for existing/any-source files); this makes new YouTube downloads
correct at the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9. Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).
Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.
Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.
channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.
Unlike the old implementation, the sort degrades gracefully:
- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
download fails cleanly with UnrecoverableDownloadError telling the
user to update yt-dlp or switch to Newest/Oldest — no retry loop.
Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a Channel refresh, log how many video/audio FileGroups are in the
Channel's directory, and how many are assigned to this Channel, other
Channels, or no Channel. Helps debug Videos missing from their Channel
after files are moved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag_file join in Channel.get_statistics() produced one row per tag,
so a Video with N Tags was counted N times in video_count, size, and
length. Count tagged Videos in a subquery instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
