Editing a title or description no longer overwrites yt-dlp's keys. The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file. An empty custom description means none.
Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download. FileGroup.get_wrolpi_json reads it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it. `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly. mp4/m4a only; a remux, so it takes seconds per GB.
Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review fixes. The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index. Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation). Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused. Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one. faststart only for mp4/m4a. Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).
The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review fixes. The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed). While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits. A failed poll
request is retried instead of ending the polling. getJobs checks the
response like its siblings. Design narration trimmed from comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review fixes. The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock. enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs. JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.
The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original). A crash in between leaves
both files with the database pointing at a real one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.
The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live. The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start. The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it. The
modal sends only what changed and refetches the description after saving.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow. Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist. Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time. Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.
Add a transcode Job and POST /api/videos/<id>/transcode. ffmpeg reports
progress via -progress pipe:1. The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.
Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode). The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Refreshing a specific file (the Files page Refresh button) bypassed the
ignored_directories check that directory and global refreshes apply.
Refreshing a hard link inside the playlists directory therefore created a
second Video for the same file.
The direct-file path now drops existing files that live in an ignored
directory before comparing against the database. Deleted paths are kept
so stale records are still removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Hotspot captive portal: welcome page with the WROLPi's address
Devices that join the hotspot probe a fixed plain-HTTP URL to check for
Internet access. NetworkManager's shared-mode dnsmasq now resolves those
probe hostnames (and `wrolpi`) to the hotspot address via an include the
Controller writes when the hotspot starts, the probes land on the Controller
(port 80), and are redirected to /portal: "This is your WROLPi, its address is
10.42.0.1, open WROLPi / open the Controller".
Only the probe hostnames are hijacked so the hotspot keeps forwarding
everything else when the WROLPi has an uplink. The include is corrected and
the connection re-activated once if NetworkManager picked a non-default
subnet. Writing the include is best-effort; the hotspot starts regardless.
`hotspot.captive_portal` (controller.yaml, default true) toggles the probe
redirects and the include; like the other hotspot settings it applies on the
next hotspot start, so saving while the hotspot is up reports
restart_required and both Controller UIs say so. The status endpoint gains
the hotspot address and portal URL, shown in both UIs. repair.sh only
creates the include directory so a disabled setting survives upgrades.
Native only: Docker has no hotspot.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* captive portal: address review
* React: re-read the hotspot address whenever the HotspotToggle starts or
stops the hotspot, matching the fallback UI's refresh-on-status.
* A WPA3 start that reverts to WPA2 leaves the hotspot up, so it now gets
the captive portal address correction too.
* The corrective `nmcli connection up` logs a warning on a non-zero exit;
dnsmasq would otherwise keep the old address silently.
* HotspotStatusResponse.captive_portal defaults to true like everywhere else.
* Trim design narrative from the module docstring and React comments.
* The include-before-activate test now observes the include at the moment
of the `nmcli ... hotspot` call instead of inferring from call order.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* captive portal: never forward any record type for probe hosts
`address=` only synthesizes A/AAAA answers. iOS also asks for the HTTPS
record type; dnsmasq forwarded that upstream, Apple's answer aliased to
captive.g.aaplimg.com, and the phone fetched the genuine "Success" from
Apple instead of the portal. Each probe host now also gets `local=`, so
dnsmasq answers every type itself, and the CDN alias is a probe host too.
Observed with dnsmasq query logging on a hotspot with an eth0 uplink.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* captive portal: let acknowledged clients finish so the OS offers "Done"
A phone's sign-in sheet only shows "Done" once the OS re-probes and gets
the genuine success reply. The Controller now remembers each client (by
hotspot IP) that has loaded the portal page and answers its later probes
with the reply each OS expects (Apple "Success" HTML, Android 204, Windows
"Microsoft Connect Test", ...). The set is cleared when the hotspot
starts so a new session shows the page again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* captive portal: acknowledge on Continue, not on view
iOS fetches the portal page in the background before opening the sheet,
then re-probes after the sheet's own navigation. Acknowledging on view
made that re-probe return "Success" as the sheet's content. Phones only
re-check the network (and offer "Done") after a full-page navigation, so
the welcome page now has a Continue button leading to /portal/continue,
which is the only place a client is acknowledged. The Continue page says
to tap Done and links to WROLPi and the Controller.
Also hijack clients3.google.com (older Android probe host).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* help: bump wrolpi-help to the captive portal and Speed Test docs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* captive portal: comments describe the Continue design; clear acks on stop
Three comments still described acknowledging on view, which c72a7f7b
removed. Acknowledged clients are now also cleared when the hotspot
stops, so a lease reused in a later session sees the portal page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Speed Test calculator: ping, download, and upload against the API
Users asking whether the hotspot is fast enough, or how many people can
stream video at once, had no objective tool that works offline. This
adds one under Calculators > Tools & Reference.
Backend (wrolpi/speedtest.py, /api/speedtest):
* ping: tiny no-store JSON the browser times for round trip latency.
* download: streams slices of a pre-built random buffer for a clamped
number of seconds (0.1-15). Random so no proxy or compression can
flatter the result; slices so no bytes are generated per request.
* upload: streaming-body handler that reads and discards the body and
reports bytes and elapsed time. Requests over 8 MiB are refused.
* info: the client address Caddy forwarded plus how many tests are in
flight. Sanic now trusts one proxy hop (PROXIES_COUNT = 1) so
request.remote_addr is the browser, not Caddy. Nothing else read
that field before.
There is deliberately no one-test-at-a-time lock. Several people
testing at once is how a user learns how the link shares between
streamers, so the server only counts in-flight tests (a shared_ctx
Value created pre-fork so every worker sees the same number) and the
UI reports how many others ran alongside.
Both Caddyfiles route /api/speedtest/* unbuffered, as the SSE chat
route already is, so throughput is measured live rather than in bursts.
Frontend:
* speedtest.js holds the logic with no React in it (bucketed
throughput, steady-state average that drops the first second, ping
stats, stream-capacity tiers, hotspot address check) so Jest drives
it with fake fetch and streams.
* SpeedTestCalculator runs 20 pings, a 10 s download, and a 10 s upload
with a live figure and rolling trace, then shows Mbps and MB/s, ping
verdict, the client address with a hotspot badge, and a table of how
many 480p..4K streams the download speed supports. Upload is
sequential POSTs of one 4 MiB Blob because Safari cannot stream a
request body. Nothing is saved.
* Sparkline is a new inline-SVG ui component (no chart dependency),
added to /theme-sample and checked in all four themes. A missing
sample is a gap, not a plunge to zero.
The page states that it measures the network path, not the drive that
actually serves video files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* speedtest: bound uploads in time, close the Start race, address review
Review on #708 found three bugs and some polish:
* Start could be clicked again while the first fetchInfo() was still
pending, because the phase only left `idle` after that await. A
second click made a second AbortController and a second concurrent
run, and Cancel only reached the later one. start() now refuses when
a controller already exists and enters a `starting` phase
synchronously, so the button is Cancel before the first await.
* The upload phase could overrun the advertised 10 s by however long
the in-flight 4 MiB POST took: on a slow hotspot, the very link this
tool exists for, that is tens of seconds. runUpload() now aborts the
in-flight request at the phase deadline (counting only completed
requests) and uses 1 MiB bodies so a few-Mbps link still completes
several inside the phase.
* The server upload sink was capped by size only, so a slow or paused
body could hold a worker until 8 MiB eventually arrived. It now also
enforces MAX_UPLOAD_SECONDS (15 s, matching download) and answers
408 when exceeded. Both caps together are what "cannot hold a
worker" actually requires.
Also: a NaN `seconds` now gets a NaN message instead of "must not be
negative"; the module, Sparkline, and gallery docstrings are trimmed
to purpose; and the upload counter is covered for the mid-stream 413,
the 408, and a broken body.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs: browse walks the file_group date index instead of probing every doc
The unfiltered /api/docs/search drove its join from `doc`, so SQLite probed
every doc's file_group row for the sort key before returning the first page.
With 20k docs scattered through a 10 GB database on a spinning disk, the first
request took about a minute cold (0.07s warm).
Use the same CROSS JOIN shape as the archive browse: file_group is the outer
table, SQLite walks file_group_published_datetime_idx and probes doc's covering
unique index on file_group_id. Both sides stay index-only and the scan stops at
the page limit. Measured cold: ~60s -> 0.6s, identical rows.
Filtered searches and the size/title orderings keep the ORM query.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: trim browse comments, stabler plan assertion
Review feedback: drop the benchmark narrative from docstrings, state only the
non-obvious constraint at the call site, and assert the doc covering-index probe
by shape rather than by autoindex name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.
Count separately and cache for 60s by filters so indexed orders can
stop after one page. Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
* videos: codec download preferences with transcode and strict-fail options
Add video_codecs/audio_codecs ordered preference lists to the video,
channel, and RSS downloaders, alongside the existing resolution
selector. Codec preference is codec-major: a preferred codec at a
lower resolution beats another codec at a higher resolution.
When 'transcode' is enabled, a downloaded file whose streams do not
match the preferences is re-encoded with ffmpeg after download (only
the mismatching stream; targets are extensible maps, currently h264
and aac/opus/mp3). When 'strict_codecs' is enabled and transcode is
not, a download with no matching format fails permanently instead of
falling back.
Empty codec lists produce a byte-identical yt-dlp command to before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: honor strict_codecs when transcode cannot produce the preference
Review fixes:
- strict_codecs is now suppressed only when transcoding can produce a
preferred codec for every non-empty preference list (new
transcode_can_satisfy_codecs); av1-only preferences with transcode
enabled fail under strict instead of silently keeping the file.
- No partial transcode: an unfixable video mismatch blocks an
audio-only conversion (and vice versa); the mismatch falls through
to strict/keep.
- fsync the transcode output before rename, and the directory after,
so power loss on a Pi cannot leave a truncated final file.
- mp3 yt-dlp pattern matches mp4a.6B as well as mp4a.6b.
- ffmpeg maps use 0:V:0 (excludes attached pictures) and 0🅰️0?
(tolerates missing audio).
- Strict toggle is no longer disabled by transcode in the UI; popup
wording corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: serialize transcodes machine-wide via shared_ctx lock
Sanic runs multiple worker processes, so an asyncio primitive cannot
serialize transcodes across download workers. A multiprocessing.Lock
in shared_ctx now guarantees only one ffmpeg transcode runs at a time
on the machine; waiting workers poll the lock asynchronously so the
event loop stays responsive while a transcode holds it for hours.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* videos: transcode-lock review fixes
- Document that a crashed lock holder requires an API restart; this is
accepted rather than adding owner-death recovery.
- Move the 'Transcoding' user notification to after the lock is
acquired, so it fires when work begins rather than before a
potentially hours-long wait.
- Soften the lock test docstring to what it measures (in-process
wait/release); trim the acquire docstring to the polling rationale.
- Give transcode_lock its own section header in contexts.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
test_search_videos_does_not_block_event_loop asserted a wall-clock
threshold (echo < 0.4s), which fails on loaded CI machines from
scheduler noise alone. Replace the timing with ordering: the mocked
search parks its thread on an Event, the echo is sent only once the
handler is provably inside the query, and the test asserts the echo
completes while the query is still held open. Events are only waited
on off the event loop, so a regression fails the ordering assertion
instead of deadlocking.
Verified both directions: passes repeatedly against the real endpoint,
and fails when the endpoint is changed to run the search on the loop.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* files: left-align the preview toolbar, Open joins the button row
The modal actions row packs its children to the right, which suits a
confirming button but stranded the preview's toolbar. The toolbar now
fills the row so its buttons read from the left, with Open first in
the same row as Download. On phones the tags selector always takes its
own line instead of squeezing in beside the buttons.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tags: the add-tag button is violet like every other tag button
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* share: a real button everywhere but the navbar corner
The share trigger was a bare anchor-wrapped glyph, which matched the
navbar's other corner icons but looked out of place beside real buttons
in the file preview toolbar. It is an IconButton by default now; the
navbar keeps the bare glyph via asIcon.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* files: readable text previews in dark themes
A same-origin text document inherits the page's color-scheme, so dark
themes rendered the browser's default light text on the preview's
forced white background. Force a light color-scheme on the preview
iframe so text is always black on white.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* files: text previews follow the theme
Instead of pinning the preview iframe to a light scheme, follow the
theme: dark, night, and amber render the document light-on-dark against
the theme's panel color; light stays black-on-white.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Uploaded singlefiles are base64-encoded into JSON before being sent to the
archive container, so an 88MB upload becomes a ~118MB request. Sanic's default
REQUEST_MAX_SIZE of 100MB rejected it with a 413 before the handler ran, and the
API surfaced this as KeyError: 'singlefile'.
Only the Docker deployment is affected; Debian/RPi extract in-process.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Completed once-downloads are never written to the downloads config, and
importing the config (which runs on every startup) deleted any DB Download
the config did not list. That erased the last_download_attempt rows the
per-domain and global daily limits count from, so a restart reset the count.
Keep completed once-downloads on import; pending/failed and recurring
downloads still follow the config as the source of truth.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page
Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:
* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
the blocking `multiprocessing.Event.wait(timeout=1)`. With no switch pending
(the normal state) this froze the loop for a second at a time; about a third
of all requests, including trivial ones, took ~1,000 ms. It now polls
`is_set()` and returns immediately.
* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
ran synchronous SQLite queries on the event loop, so one slow search stalled
every other request on that worker. They now run via `asyncio.to_thread`.
* The unfiltered Videos page (`POST /api/videos/search` with only an order)
used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
every video's index entry before LIMIT applies: seconds cold on a large
library on a spinning disk, and ~10 s hot for a deep page. It now seeks each
exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
rows and merges them, with the total from a separate count cached for 60 s.
Results are identical to the generic query.
Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results). Host profiles live in a
gitignored JSON file; an example is included.
Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve the video description from its own endpoint instead of Video.__json__
`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk. The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays. The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.
Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions. The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored. The player page fetches it through
`useVideoExtras` alongside the comments.
Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: thread-local zim sessions, fast-path equivalence test, comments
* Zim search, single-zim search, and search estimates run libzim off the event
loop, each opening its own session inside the worker thread rather than
passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
query and asserts identical rows and paging for both date orders. The one
intentional difference is documented: past the end, the fast path still
reports the true total.
* switch_worker returns before its try block so the empty path does not clear
the event; its comment states the invariant only. The copied "synchronous
SQLite" comments are reduced to one line in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* videos: a collaboration video's channel directory conflict killed the download
YouTube collaboration videos appear in every collaborator's feed but their
metadata names only the primary uploader's channel. When that channel's
computed directory already belonged to an existing Channel that no longer
matched by source_id/url/name, channel creation raised
ChannelDirectoryConflict and the download failed unrecoverably. Now the
channel that owns the directory is used instead of creating a conflicting one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: trim comments that restated the code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
NAME_PARSER's suffix whitelist had no audio suffixes (and no mkv/m4v),
so a correctly-named audio file got no published date and sorted to the
very end of its Channel -- which reads as missing. The whitelist now
carries the media suffixes a Video FileGroup can lead with.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* videos: an uploaded audio file never appeared in its Channel
The upload flow asks Video.can_model, which only accepted video/*, so an
uploaded .flac was indexed with no Video row -- and the batch modeler's
`indexed != True` gate then excluded it on every later refresh, leaving
the file permanently invisible in its Channel.
can_model now accepts audio/* like the batch modeler always has, and the
modeler gates on the Video row being missing (mirroring the doc_modeler
fix for the same class of bug) so already-stuck files heal on the next
refresh. Audio playlists (.m3u detects as audio/x-mpegurl) are excluded
from both paths and unmodeled by video_cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: widen the playlist denylist, prove modeler termination
The denylist gains audio/x-scpls and audio/scpls -- .pls detection
varies by libmagic version, and any audio/* playlist type must be
refused wherever it comes back. Modeling one file is extracted as
_model_video so a test can force persistent failure and prove the loop
terminates; failed_ids is recorded before the PYTEST re-raise so no
failure path can re-select an id forever. Playlist mimetypes are bound
through json_each like the rest of video_cleanup, and the comments
state the invariants instead of the history.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: the playlist test must not depend on libmagic's m3u opinion
CI's libmagic detects .m3u as text/plain where a Mac's says
audio/x-mpegurl. The invariant is that a playlist mimetype is never
modeled by either path, so the test forces the mimetype like the other
denylist tests instead of asserting what detection returned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* common: config writes must be atomic and version-checked under the save lock
A config file must never be observable in a truncated state: writes go to a
temporary file in the same directory and are renamed into place. The version
check in ConfigFile.save() reads the config file, so it must hold
config_save_lock like the write does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* collections: item writes must begin as SQLite writers
The add/remove/reorder item endpoints read before their first write; a
deferred transaction upgrading at that write fails instantly with "database
is locked" when any other connection wrote in between, bypassing
busy_timeout. Open their sessions with get_db_session(commit=True) so the
transaction takes the write lock at BEGIN, matching post_download.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: error responses must not commit the write session; trim comments
Map exceptions to 4xx outside the get_db_session(commit=True) block so an
error response never commits, and serialize inside the session. Close the
mkstemp descriptor when fdopen never took ownership. Trim comments and test
docstrings to what the code and assertions actually cover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* app: the navbar corner wrapped onto a second row on small phones
Every status-icon slot carried a 1.5em separator margin -- ~80px across
three slots -- which wrapped the mobile bar's whole corner onto a second
row on any screen narrower than about 380px. The margin is now 0.25em on
both bars: the corner's gap and the slots' own padding still space the
icons, and on desktop the freed room just goes back to the links that
useOverflowNav folds into "More".
The mobile bar is extracted as MobileNav (mirroring DesktopNav) so a
Cypress suite can mount it and hold it to one row at phone widths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: prove 320px with a faithful fixture, trim the CSS narration
The suite claimed 360px covered narrower devices because the fixture was
wider than production, but the fixture also lacked the real theme
toggle's 0.8em right margin, so the safety margin was overstated. The
fixture now matches a phone -- 32px logo home link, the toggle's margin,
and a coarse pointer so the search link drops its keyboard hint, with a
test asserting the hint is really absent -- and 320px is measured
directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The channel downloader refreshes a Channel's files before downloading,
already passing send_events=False, but the parameter was never plumbed
into the file worker, so every pre-download refresh notified the user.
FileTask now carries send_events and handle_refresh skips the
Refreshing/Refreshed events when it is False. User-requested refreshes
and global refreshes still send Events.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* e2e: exercise the FileBrowser footer buttons against real files
Sixteen Cypress tests covering Rename, Move, Delete (including the
tagged-file force-delete flow), Ignore/Unignore, Upload, and scoped
Refresh, with no API mocking: fb:* tasks seed a cypress-fb/ sandbox
inside the media directory, the UI mutates it through the real API, and
assertions check the disk as well as the table. Skipped in CI, which
runs only the React dev server.
Background jobs (move, refresh) only queue work and worker_status reads
idle while jobs wait, so cy.refreshFiles / cy.waitForEvent poll the
events feed for completion events newer than the server's clock, and
fail loudly when one never arrives. Every suite cleans up
unconditionally: sandbox files, DB rows via a final refresh, test tags
and their tags/ directories, and any leaked ignored-directory config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* e2e: act on review — drain stragglers, teardown that cannot be skipped
Events carry no job id, so waitForEvent cannot prove WHICH job finished.
The rename tests now drain the parent-directory refresh their rename
queues before ending, so the next test's refresh wait cannot be
satisfied by a straggler while the DB still reflects the old tree; move
waits match the destination in the completion message.
Teardown must always reach the disk wipe: a failed command aborts the
rest of its hook, so the tag cleanup and the DB-purge refresh are now
best-effort (failOnStatusCode: false, no assertions), and the unignore
runs after every test instead of once per spec. History and
forward-looking product notes trimmed from comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
perpetual_shutdown is set by the real before_server_stop listener, so any
earlier async_client teardown in the same pytest-xdist worker leaves it
True on the module-global app and the watchdog returns before reclaiming.
Failed under -nauto, passed alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Starting the file-worker pump in every Sanic worker made 5 processes
race the shared status dict, the busy flag, and the same directories.
Non-owners now run only a reclaim watchdog: they refresh nothing unless
the owner PID is gone or its heartbeat is stale.
Production wait_for_job polls every 0.1s instead of 100 Hz. A failed
rename includes the recorded job error in the 409 body.
A directory rename waits inline on a move job. When that move failed
(stale FileGroup rows colliding on primary_path), handle_move reverted
the files and sent a failure event but left the job pending. wait_for_job
only understands 'complete', so POST /api/files/rename spun until its
300s timeout and the Rename modal never closed.
The same sequence could kill file processing for the life of the process.
perpetual_signal treated any CancelledError as terminal and logged
nothing. Separately, perpetual_tasks_started is a shared Event that
survives the worker that set it: docker auto-reloads and worker crashes
left every later after_server_start as a no-op. Queued refreshes sat
untouched while worker_status reported idle. Restarting the API was the
only recovery.
Mark failed jobs failed and raise immediately from wait_for_job. Catch
unexpected cancellation and reschedule unless the app is actually
stopping. Track the perpetual-loop owner PID and retake the claim when
that process is gone; every Sanic worker still runs the file-worker
pump. Harden transfer_queue against the cross-process Empty race, and
stop request processes stealing from the shared public queue.
