Commit graph

2475 commits

Author SHA1 Message Date
Roland
5441d096ae
ai: page list_collections
A library with a few hundred channels returned 137k characters from one
list_collections call, far past any tool-result budget.  The endpoint
now takes limit/offset (clamped like the other AI listings) and a
search_str name filter, and reports total and next_offset; the MCP tool
passes them through and tells the caller how to continue.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
c6d2ec783e
ai: wrolpi_mcp exposes only the compact tool set
The per-kind MCP wrappers (search/search_videos/search_archives/
search_docs, get_video/get_archive/get_doc, get_video_captions/
get_video_comments/get_archive_text) duplicated search_files, get_file,
and read_content and cost an external client ten redundant definitions
on every request.  They are gone.  search_default_zims folds into
search_zim (zim_id optional), get_inventory_items into get_inventory
(slug optional), and read_file is exposed for the first time.  16 tools,
matching what the built-in assistant sees plus the read-only listings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
10f0a4a46c
ai: research mode runs on the compact tool catalog
Research mode switches to nine kind-generic tools: search_files,
get_file, read_content, search_zims, get_zim_entry, get_inventory,
list_files, read_file, search_places.  Tool definitions drop from 20
tools / 8,947 chars (~2,240 tokens) to 9 / 3,881 (~970) on every
request, a sixth of the 8k context the small tier runs with instead of
a quarter.  The remaining descriptions are trimmed toward 300 chars.

The replaced per-kind endpoints are removed (four searches, three
getters, three readers, search_suggestions, the slug inventory route);
their tests move onto the generic routes.  The research prompt, its
worked example, and the empty-search hint no longer tell the model to
list collections first: channels resolve by name from the library map
that is already in the prompt.  list_collections, list_tags, and
get_map_overview stay as endpoints for the MCP wrappers but leave the
chat allowlist; the MCP search_suggestions tool goes since search_files
returns the same matches inline.

scripts/ai_eval.py expects search_files and treats "what channels do I
have" as answerable from context.  On the 1.7B model against the dev
library both catalogs score 14/18, but the compact one answers with one
tool call where the old one needed up to five, no longer invents the
example's channel, and keeps kinds straight when asked for videos.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
12cd8355ad
ai: kind-generic search_files / get_file / read_content
Small models carry every tool definition on every request, and the
per-kind trios (four searches, three getters, three readers) cost a
quarter of an 8k context while giving the model near-duplicate choices.
Three generic endpoints replace them:

- POST files/search: one search over videos, archives, and docs with an
  optional kind; channel (name or id), domain, author, subject each
  imply their kind and are ignored for other kinds.  When the term also
  names channels/domains/authors, they are returned as `matches` so the
  model can narrow without a second tool.  Channels resolve by name, so
  the library map already in the prompt is enough: no list call first.
- GET files/{id}: detail by id whatever the kind; videos now report
  has_captions/has_comments and the full description, archives keep
  their snapshot history.
- GET files/{id}/content?part=text|comments: captions, page text, or
  comments, paged; a wrong part for the kind says what the file is.

Inventories take an optional slug on GET /inventories (one tool lists
and fetches).  Listings are slimmed: no derivable captions/text links,
no size/mimetype, duration as H:MM:SS, published as a date.  The
library map gains the tag names.

wrolpi_mcp keeps every existing tool name; the wrappers now call the
generic routes, and search_files/get_file/read_content are exposed too.
The old per-kind endpoints remain until the modes switch over.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
0bc843d0ed
ai: tags, downloads, map, and search-preview tools
Five new read-only /api/ai endpoints, each with a matching wrolpi_mcp
proxy tool: list_tags (tags with what they are applied to, plus recent
names), search_suggestions (matching channels/domains/authors/subjects
and file/Zim result estimates), list_downloads (queue summary and lean
recurring/one-time entries with relative destinations; status filter;
errors keep their tail, which holds the exception message),
get_map_overview (map files, search-index state, subscriptions, pins
with map links), and search_places (place search with map links and
optional proximity ranking).

Research mode allowlists the library-facing four.  Nothing here can
start, stop, or retry a download.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
703d933f22
ai: list_files tool — browse the media directory tree
New read-only GET /api/ai/files/list lists one directory of the media
directory (directories first, then files with size and guessed
mimetype), paged at 100 entries.  Every entry carries its relative
path for list_files or read_file.  Same guards as read_file via a
shared resolver: no escaping the media directory, and the config
directory is refused and hidden from the root listing.

The agent loop picks the operation up from the spec; library mode
allowlists it.  wrolpi_mcp gains the matching list_files proxy tool.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:18 -06:00
Roland
fec3b795c3
ai: system mode never answers state from memory
The re-run showed "Is everything running?" parroting the example's
all-clear without checking.  The system prompt now states that state is
never known from memory or examples — every health answer starts with a
fresh get_system_status — and the example's answer cites its own
example data (service names, version) so a verbatim parrot is
self-evidently wrong.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
eee72a966b
ai: no invented failures in the system-mode example
The eval's two remaining System misses were the model investigating a
fictional failed wrolpi-kiwix — planted by the few-shot example itself.
The example now shows a healthy status and an all-clear answer; it
still demonstrates status-first without seeding a fake diagnosis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
37791751d3
ai: eval findings — truthful help example, links in list answers
The first scored eval run (39/45) caught two example defects: a user
asking about the help example's own topic got the example's invented
UI detail parroted back without a search, so the example now carries
true content from the real certificates doc on a topic users rarely
lead with; and list-style answers dropped WROLPi links because the
research example only showed links in prose, so its answer now models
a numbered list with a link per item.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
0be1badc85
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:

- Every chat's system prompt now carries a live library map (today's
  date, counts, and the exact names of channels, domains, zims, and
  inventories, capped and cached 60s).  Exact spellings are in context
  before the first token, and the list_collections round-trip — minutes
  on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
  turns are trimmed first, the current question is never dropped, and
  an over-budget turn gets a clear "start a new conversation" error
  instead of a silently truncated prompt.  cache_prompt reuses the KV
  cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
  default (the 4B was running at 8k instead of 16k), and llama-server
  runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
  call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
  conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
  mechanical checks (finished, called the right tool first, included a
  WROLPi link, no external URLs) so prompt changes stop being validated
  by anecdote.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
1dd33d26e2
ai: keep slow-CPU chat streams alive
On the FX-8320 the first token took over a minute, and Sanic's default
60-second RESPONSE_TIMEOUT killed the stream before anything was sent.
Raise the timeout above the chat loop's own 600s hard ceiling, and emit
a status event before every model call so the stream is never silent —
for proxies and for the user staring at a slow machine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
c925345d1b
ai: build every x86 CPU variant of llama.cpp with runtime dispatch
The portable (GGML_NATIVE=OFF) x86-64 build still assumes AVX2, which
SIGILLs on older CPUs — found deploying to an AMD FX-8320.  x86_64 now
builds all CPU-variant backends with runtime dispatch (GGML_BACKEND_DL
+ GGML_CPU_ALL_VARIANTS), the same scheme as llama.cpp's official
release binaries; arm64 keeps the baseline build, which runs on both
Pi 4 and Pi 5.  The binary and its dlopen'd backends install together
in /opt/llama-cpp (the loader scans the executable's directory) with
$ORIGIN rpath, symlinked into /usr/local/bin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
cb9e3e663b
ai: ground the librarian — collection links, retry hints, no source URLs
Live testing surfaced three ways small models went wrong summarizing a
library: they gave up after one empty search (misspelled channel names
found nothing), so zero-result searches now carry a retry hint and the
Research prompt requires list_collections before channel/site questions
and search_videos(channel_id) before claiming a channel is summarized;
they presented the familiar YouTube source URL instead of the WROLPi
link, so listings omit the source URL (archive detail keeps it); and
they invented links for channels, so channel collections now carry
their real /videos/channel page link.  Search descriptions state that
search_str is optional for browsing the newest items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:17 -06:00
Roland
95f04c897b
ai: live-chat fixes from first real-model run
Three findings from running Qwen3-1.7B against the docker stack:
the llm image lost libgomp when the build tools were removed
(llama-server needs the OpenMP runtime); the Help prompt let the
small model answer from memory instead of the docs, so it now
requires search_help before any answer; and help search results
called the slug "doc" while get_help_doc's argument is "slug", so
the model invented slugs — the names now match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
a91d3ae92e
ai: publish tooling + real model hashes in the bundled catalog
publish_ai_models.py produces the CDN artifacts for every AI_MODELS
entry — GPG-signed .meta4 sidecars (reusing the map trust chain) and
the signed ai/models.json manifest — and uploads them with s3cmd.
The bundled catalog now carries the real sha256/size of the two Qwen
GGUFs, which match Hugging Face's LFS records for the Apache 2.0
unsloth quants, and whose signed meta4s verify against the committed
public key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
6d48e3cdb0
ai: wrolpi_mcp becomes a thin proxy over /api/ai
Every MCP tool is now one /api/ai call with minimal text rendering;
the formatting, link building, and truncation logic it used to carry
untested lives (tested) in modules/ai.  Tool names are unchanged for
external clients; a source-scan test guards the names and that the
proxy only reaches /api/ai.  New lean blueprint endpoints back the two
tools that had no equivalent: GET archives/{id} (detail + snapshot
history) and GET videos/{id}/comments (paged author lines).  Paged
content is read up to the old 50k-char bound.  get_inventory_items now
takes the inventory slug (inventories are config-only; the old
numeric-id endpoint no longer exists).  This also brings wrolpi_mcp
into version control (venv/ is already gitignored).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
7ec1100f2b
ai: the Chat tab — SSE streaming chat with modes
The app's first streaming client: sse.js reads the /api/chat POST body
with fetch + a stream reader (apiCall's 60s JSON path cannot stream),
parsing SSE events with abort support.  AiChatContext holds the
conversation (browser-only, per design) so it survives visiting the
Manage tab; switching modes starts a fresh conversation.  The Chat tab
has the three-button mode picker, per-mode suggestion chips, streamed
answers with clickable relative WROLPi links, tool-activity indicators,
a slow-hardware banner, and a "not ready" state pointing at Manage.
ChatMessage states are in the theme-sample gallery.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
aaa3c7d845
ai: the AI nav page — Manage tab (model catalog, settings, service)
New AI nav item and /ai route with Chat and Manage tabs (Chat is a
placeholder until the SSE chat lands).  The Manage tab mirrors the
Map/Zim download UX: the model catalog with the RAM-based tier
recommendation and a Pi 4 performance hint, downloads through the
download manager (progress on the Downloads page), active-model
select, enable toggle, idle-unload minutes, and llama-server
start/stop through the Controller (compose service name in docker,
systemd unit natively).  AiModelRow is in the theme-sample gallery.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
84883c36f6
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift.  Manage endpoints
never become tools.  A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.

POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout.  Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
318b835faf
ai: package llama-server for all three deployments, with idle unload
install_llama_cpp.sh builds llama-server from the SHA-pinned v0.3.0
source tarball (single source of truth: on-device upgrade, Pi image,
Live ISO, and the new docker/llm container all call it; version-marker
makes re-runs no-ops).  start_llama_server.sh refuses to start unless
ai.yaml enables AI and the active model file exists — which also covers
unmounted media.  wrolpi-llm.service is installed but deliberately
never enabled at boot: the Controller starts it on demand (registered
in managed_services; the llm compose service follows the samba
pattern), and the API's perpetual idle-unload worker asks the
Controller to stop it after the configured idle minutes.  llama-server
is localhost-only natively, internal-network-only in docker, and never
proxied by Caddy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:16 -06:00
Roland
119256e529
ai: register AIConfig in get_all_configs so it imports on startup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:15 -06:00
Roland
2f27ce49a7
ai: ai.yaml config, model catalog, and the GGUF downloader
ai.yaml is the source of truth (opt-in: enabled=False by default) with
active model, idle-unload minutes, context override, and tool-call
bound.  The model catalog follows the map pattern: a bundled constant
keeps the Manage tab working offline, the CDN manifest (ai/models.json
+ detached GPG signature) wins when reachable.  GGUF downloads clone
the map-extract flow — the per-file meta4 is GPG-verified once and the
same bytes handed to aria2c, so the verified hash is the enforced one;
unverified downloads are refused.  Models land in <media>/ai/models/
(gated by the existing media_mounted download rules).  Manage endpoints
(catalog with RAM-tier recommendation, settings) live under
/api/ai/manage/ which the future agent loop excludes from its tool
catalog.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:15 -06:00
Roland
9ba81b6c80
ai: help-docs search — Help mode tools
Search the wrolpi-help markdown in-process: files are parsed on demand
and cached by mtime; title matches outrank headings outrank body.  No
database table — the corpus is dozens of small files.  Pages are read
as paged markdown.  The docker api container gets a read-only bind
mount of the help docs at the native /opt/wrolpi-help location; a
missing help directory degrades to empty results with a message.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:15 -06:00
Roland
5d99677568
ai: system endpoints — aggregate status, services, logs, disks
Proxy the Controller under /api/ai so the model sees one API: an
aggregate /status to start every diagnosis, lean service list, recent
service logs (lines clamped), and disks + SMART.  One controller_client
helper resolves the Controller base URL (docker service name vs
localhost).  Every proxy degrades to an `errors` entry when the
Controller is unreachable — never a 500.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:15 -06:00
Roland
1f72a8b97e
ai: add the /api/ai blueprint, the read-only LLM tool catalog
Lean, flat, LLM-shaped read-only endpoints over the library: global,
video, archive, and doc search; paged captions, archive text, and zim
entries; collections, inventories, and a media-directory text reader.
Links are server-computed; big content is paged, never truncated.
This ports the formatting layer from wrolpi_mcp (untouched for now)
into tested code.  First phase of ai.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 20:10:15 -06:00
lrnselfreliance
d701846254
Hotspot captive portal: welcome page with the WROLPi's address (#709)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Hotspot captive portal: welcome page with the WROLPi's address

Devices that join the hotspot probe a fixed plain-HTTP URL to check for
Internet access.  NetworkManager's shared-mode dnsmasq now resolves those
probe hostnames (and `wrolpi`) to the hotspot address via an include the
Controller writes when the hotspot starts, the probes land on the Controller
(port 80), and are redirected to /portal: "This is your WROLPi, its address is
10.42.0.1, open WROLPi / open the Controller".

Only the probe hostnames are hijacked so the hotspot keeps forwarding
everything else when the WROLPi has an uplink.  The include is corrected and
the connection re-activated once if NetworkManager picked a non-default
subnet.  Writing the include is best-effort; the hotspot starts regardless.

`hotspot.captive_portal` (controller.yaml, default true) toggles the probe
redirects and the include; like the other hotspot settings it applies on the
next hotspot start, so saving while the hotspot is up reports
restart_required and both Controller UIs say so.  The status endpoint gains
the hotspot address and portal URL, shown in both UIs.  repair.sh only
creates the include directory so a disabled setting survives upgrades.
Native only: Docker has no hotspot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: address review

* React: re-read the hotspot address whenever the HotspotToggle starts or
  stops the hotspot, matching the fallback UI's refresh-on-status.
* A WPA3 start that reverts to WPA2 leaves the hotspot up, so it now gets
  the captive portal address correction too.
* The corrective `nmcli connection up` logs a warning on a non-zero exit;
  dnsmasq would otherwise keep the old address silently.
* HotspotStatusResponse.captive_portal defaults to true like everywhere else.
* Trim design narrative from the module docstring and React comments.
* The include-before-activate test now observes the include at the moment
  of the `nmcli ... hotspot` call instead of inferring from call order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: never forward any record type for probe hosts

`address=` only synthesizes A/AAAA answers.  iOS also asks for the HTTPS
record type; dnsmasq forwarded that upstream, Apple's answer aliased to
captive.g.aaplimg.com, and the phone fetched the genuine "Success" from
Apple instead of the portal.  Each probe host now also gets `local=`, so
dnsmasq answers every type itself, and the CDN alias is a probe host too.

Observed with dnsmasq query logging on a hotspot with an eth0 uplink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: let acknowledged clients finish so the OS offers "Done"

A phone's sign-in sheet only shows "Done" once the OS re-probes and gets
the genuine success reply.  The Controller now remembers each client (by
hotspot IP) that has loaded the portal page and answers its later probes
with the reply each OS expects (Apple "Success" HTML, Android 204, Windows
"Microsoft Connect Test", ...).  The set is cleared when the hotspot
starts so a new session shows the page again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: acknowledge on Continue, not on view

iOS fetches the portal page in the background before opening the sheet,
then re-probes after the sheet's own navigation.  Acknowledging on view
made that re-probe return "Success" as the sheet's content.  Phones only
re-check the network (and offer "Done") after a full-page navigation, so
the welcome page now has a Continue button leading to /portal/continue,
which is the only place a client is acknowledged.  The Continue page says
to tap Done and links to WROLPi and the Controller.

Also hijack clients3.google.com (older Android probe host).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: bump wrolpi-help to the captive portal and Speed Test docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: comments describe the Continue design; clear acks on stop

Three comments still described acknowledging on view, which c72a7f7b
removed.  Acknowledged clients are now also cleared when the hotspot
stops, so a lease reused in a later session sees the portal page.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:02 -06:00
lrnselfreliance
91f92683ba
Speed Test calculator: ping, download, and upload against the API (#708)
* Speed Test calculator: ping, download, and upload against the API

Users asking whether the hotspot is fast enough, or how many people can
stream video at once, had no objective tool that works offline.  This
adds one under Calculators > Tools & Reference.

Backend (wrolpi/speedtest.py, /api/speedtest):
* ping: tiny no-store JSON the browser times for round trip latency.
* download: streams slices of a pre-built random buffer for a clamped
  number of seconds (0.1-15).  Random so no proxy or compression can
  flatter the result; slices so no bytes are generated per request.
* upload: streaming-body handler that reads and discards the body and
  reports bytes and elapsed time.  Requests over 8 MiB are refused.
* info: the client address Caddy forwarded plus how many tests are in
  flight.  Sanic now trusts one proxy hop (PROXIES_COUNT = 1) so
  request.remote_addr is the browser, not Caddy.  Nothing else read
  that field before.

There is deliberately no one-test-at-a-time lock.  Several people
testing at once is how a user learns how the link shares between
streamers, so the server only counts in-flight tests (a shared_ctx
Value created pre-fork so every worker sees the same number) and the
UI reports how many others ran alongside.

Both Caddyfiles route /api/speedtest/* unbuffered, as the SSE chat
route already is, so throughput is measured live rather than in bursts.

Frontend:
* speedtest.js holds the logic with no React in it (bucketed
  throughput, steady-state average that drops the first second, ping
  stats, stream-capacity tiers, hotspot address check) so Jest drives
  it with fake fetch and streams.
* SpeedTestCalculator runs 20 pings, a 10 s download, and a 10 s upload
  with a live figure and rolling trace, then shows Mbps and MB/s, ping
  verdict, the client address with a hotspot badge, and a table of how
  many 480p..4K streams the download speed supports.  Upload is
  sequential POSTs of one 4 MiB Blob because Safari cannot stream a
  request body.  Nothing is saved.
* Sparkline is a new inline-SVG ui component (no chart dependency),
  added to /theme-sample and checked in all four themes.  A missing
  sample is a gap, not a plunge to zero.

The page states that it measures the network path, not the drive that
actually serves video files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* speedtest: bound uploads in time, close the Start race, address review

Review on #708 found three bugs and some polish:

* Start could be clicked again while the first fetchInfo() was still
  pending, because the phase only left `idle` after that await.  A
  second click made a second AbortController and a second concurrent
  run, and Cancel only reached the later one.  start() now refuses when
  a controller already exists and enters a `starting` phase
  synchronously, so the button is Cancel before the first await.

* The upload phase could overrun the advertised 10 s by however long
  the in-flight 4 MiB POST took: on a slow hotspot, the very link this
  tool exists for, that is tens of seconds.  runUpload() now aborts the
  in-flight request at the phase deadline (counting only completed
  requests) and uses 1 MiB bodies so a few-Mbps link still completes
  several inside the phase.

* The server upload sink was capped by size only, so a slow or paused
  body could hold a worker until 8 MiB eventually arrived.  It now also
  enforces MAX_UPLOAD_SECONDS (15 s, matching download) and answers
  408 when exceeded.  Both caps together are what "cannot hold a
  worker" actually requires.

Also: a NaN `seconds` now gets a NaN message instead of "must not be
negative"; the module, Sparkline, and gallery docstrings are trimmed
to purpose; and the upload counter is covered for the mid-stream 413,
the 408, and a broken body.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 18:47:18 -06:00
lrnselfreliance
7471b41ae1
docs: browse walks the file_group date index instead of probing every doc (#707)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* docs: browse walks the file_group date index instead of probing every doc

The unfiltered /api/docs/search drove its join from `doc`, so SQLite probed
every doc's file_group row for the sort key before returning the first page.
With 20k docs scattered through a 10 GB database on a spinning disk, the first
request took about a minute cold (0.07s warm).

Use the same CROSS JOIN shape as the archive browse: file_group is the outer
table, SQLite walks file_group_published_datetime_idx and probes doc's covering
unique index on file_group_id. Both sides stay index-only and the scan stops at
the page limit. Measured cold: ~60s -> 0.6s, identical rows.

Filtered searches and the size/title orderings keep the ORM query.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs: trim browse comments, stabler plan assertion

Review feedback: drop the benchmark narrative from docstrings, state only the
non-obvious constraint at the call site, and assert the doc covering-index probe
by shape rather than by autoindex name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 12:39:05 -06:00
lrnselfreliance
e0fdb762fc
zim: recompute the outdated-Zims flag after startup and on Delete (#706)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* zim: recompute the outdated-Zims flag after startup and on Delete

The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan.  The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.

- The startup recompute now runs after the DB is up and configs are
  imported, so it overrides the value `init_flags` restored from the DB.
  Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
  was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
  of showing a placeholder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete

- Startup calls flags.init_flags() then flag_outdated_zim_files() right
  after db_up, before the DB config import, so the disk scan always
  overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
  file was deleted.
- Trim narrative comments to the constraint they enforce.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 15:19:36 -06:00
lrnselfreliance
8008cd1fd8
search: don't COUNT(*) OVER() the page query (#702)
COUNT(*) OVER() forced SQLite to visit every matching file_group
before LIMIT, which is a full table scan of the (large) file_group
table — ~22s cold on a Pi for videos ordered by download date.

Count separately and cache for 60s by filters so indexed orders can
stop after one page.  Past-the-end pages now report the true total.
Videos/archives/docs pagers keep totalPages across page changes.
2026-09-12 14:44:30 -06:00
lrnselfreliance
0b61d3ab08
videos: codec download preferences with transcode and strict-fail options (#701)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* videos: codec download preferences with transcode and strict-fail options

Add video_codecs/audio_codecs ordered preference lists to the video,
channel, and RSS downloaders, alongside the existing resolution
selector.  Codec preference is codec-major: a preferred codec at a
lower resolution beats another codec at a higher resolution.

When 'transcode' is enabled, a downloaded file whose streams do not
match the preferences is re-encoded with ffmpeg after download (only
the mismatching stream; targets are extensible maps, currently h264
and aac/opus/mp3).  When 'strict_codecs' is enabled and transcode is
not, a download with no matching format fails permanently instead of
falling back.

Empty codec lists produce a byte-identical yt-dlp command to before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: honor strict_codecs when transcode cannot produce the preference

Review fixes:
- strict_codecs is now suppressed only when transcoding can produce a
  preferred codec for every non-empty preference list (new
  transcode_can_satisfy_codecs); av1-only preferences with transcode
  enabled fail under strict instead of silently keeping the file.
- No partial transcode: an unfixable video mismatch blocks an
  audio-only conversion (and vice versa); the mismatch falls through
  to strict/keep.
- fsync the transcode output before rename, and the directory after,
  so power loss on a Pi cannot leave a truncated final file.
- mp3 yt-dlp pattern matches mp4a.6B as well as mp4a.6b.
- ffmpeg maps use 0:V:0 (excludes attached pictures) and 0🅰️0?
  (tolerates missing audio).
- Strict toggle is no longer disabled by transcode in the UI; popup
  wording corrected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: serialize transcodes machine-wide via shared_ctx lock

Sanic runs multiple worker processes, so an asyncio primitive cannot
serialize transcodes across download workers.  A multiprocessing.Lock
in shared_ctx now guarantees only one ffmpeg transcode runs at a time
on the machine; waiting workers poll the lock asynchronously so the
event loop stays responsive while a transcode holds it for hours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: transcode-lock review fixes

- Document that a crashed lock holder requires an API restart; this is
  accepted rather than adding owner-death recovery.
- Move the 'Transcoding' user notification to after the lock is
  acquired, so it fires when work begins rather than before a
  potentially hours-long wait.
- Soften the lock test docstring to what it measures (in-process
  wait/release); trim the acquire docstring to the polling rationale.
- Give transcode_lock its own section header in contexts.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-07 09:12:00 -06:00
dependabot[bot]
e0ae71048a
build(deps): bump the npm-dependencies group in /app with 7 updates (#700)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Bumps the npm-dependencies group with 7 updates in the /app directory:

| Package | From | To |
| --- | --- | --- |
| [@testing-library/user-event](https://github.com/testing-library/user-event) | `14.6.1` | `14.6.6` |
| [cypress](https://github.com/cypress-io/cypress) | `15.20.0` | `15.21.1` |
| [esptool-js](https://github.com/espressif/esptool-js) | `0.6.0` | `0.6.1` |
| [pmtiles](https://github.com/protomaps/pmtiles) | `4.4.1` | `4.5.0` |
| [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) | `7.0.0` | `7.0.1` |
| [@types/lodash](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/lodash) | `4.17.24` | `4.17.25` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.1.2` | `26.4.0` |



Updates `@testing-library/user-event` from 14.6.1 to 14.6.6
- [Release notes](https://github.com/testing-library/user-event/releases)
- [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/user-event/compare/v14.6.1...v14.6.6)

Updates `cypress` from 15.20.0 to 15.21.1
- [Release notes](https://github.com/cypress-io/cypress/releases)
- [Changelog](https://github.com/cypress-io/cypress/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/cypress-io/cypress/compare/v15.20.0...v15.21.1)

Updates `esptool-js` from 0.6.0 to 0.6.1
- [Release notes](https://github.com/espressif/esptool-js/releases)
- [Changelog](https://github.com/espressif/esptool-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/espressif/esptool-js/compare/v0.6.0...v0.6.1)

Updates `pmtiles` from 4.4.1 to 4.5.0
- [Commits](https://github.com/protomaps/pmtiles/commits)

Updates `@testing-library/jest-dom` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/testing-library/jest-dom/releases)
- [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/jest-dom/compare/v7.0.0...v7.0.1)

Updates `@types/lodash` from 4.17.24 to 4.17.25
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/lodash)

Updates `@types/node` from 26.1.2 to 26.4.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@testing-library/user-event"
  dependency-version: 14.6.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: cypress
  dependency-version: 15.21.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: esptool-js
  dependency-version: 0.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: pmtiles
  dependency-version: 4.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@testing-library/jest-dom"
  dependency-version: 7.0.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/lodash"
  dependency-version: 4.17.25
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 26.4.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:51:13 -06:00
dependabot[bot]
f39eba81e6
build(deps): update pydantic requirement from >=2.13.4 to >=2.13.5 in /controller (#691)
Updates the requirements on [pydantic](https://github.com/pydantic/pydantic) to permit the latest version.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/v2.13.5/HISTORY.md)
- [Commits](https://github.com/pydantic/pydantic/compare/v2.13.4...v2.13.5)

---
updated-dependencies:
- dependency-name: pydantic
  dependency-version: 2.13.5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-03 16:43:12 -06:00
dependabot[bot]
1cd47e4afd
build(deps): bump browserslist from 4.28.1 to 4.28.8 in /app (#695)
Bumps [browserslist](https://github.com/browserslist/browserslist) from 4.28.1 to 4.28.8.
- [Release notes](https://github.com/browserslist/browserslist/releases)
- [Changelog](https://github.com/browserslist/browserslist/blob/main/CHANGELOG.md)
- [Commits](https://github.com/browserslist/browserslist/compare/4.28.1...4.28.8)

---
updated-dependencies:
- dependency-name: browserslist
  dependency-version: 4.28.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:43:07 -06:00
dependabot[bot]
3842d6436f
build(deps): bump decode-uri-component and query-string in /app (#694)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) to 0.5.0 and updates ancestor dependency [query-string](https://github.com/sindresorhus/query-string). These dependencies need to be updated together.


Updates `decode-uri-component` from 0.4.1 to 0.5.0
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.4.1...v0.5.0)

Updates `query-string` from 9.4.1 to 9.5.1
- [Release notes](https://github.com/sindresorhus/query-string/releases)
- [Commits](https://github.com/sindresorhus/query-string/compare/v9.4.1...v9.5.1)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-version: 0.5.0
  dependency-type: indirect
- dependency-name: query-string
  dependency-version: 9.5.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:34:34 -06:00
dependabot[bot]
61deb0a1ee
build(deps): bump fflate in /app (#696)
Bumps  and [fflate](https://github.com/101arrowz/fflate). These dependencies needed to be updated together.

Updates `fflate` from 0.8.2 to 0.8.3
- [Release notes](https://github.com/101arrowz/fflate/releases)
- [Changelog](https://github.com/101arrowz/fflate/blob/master/CHANGELOG.md)
- [Commits](https://github.com/101arrowz/fflate/compare/v0.8.2...v0.8.3)

Updates `fflate` from 0.6.10 to 0.6.11
- [Release notes](https://github.com/101arrowz/fflate/releases)
- [Changelog](https://github.com/101arrowz/fflate/blob/master/CHANGELOG.md)
- [Commits](https://github.com/101arrowz/fflate/compare/v0.8.2...v0.8.3)

---
updated-dependencies:
- dependency-name: fflate
  dependency-version: 0.8.3
  dependency-type: indirect
- dependency-name: fflate
  dependency-version: 0.6.11
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:34:27 -06:00
dependabot[bot]
65217f1a0c
build(deps): update uvicorn requirement from >=0.52.1 to >=0.52.4 in /controller (#689)
Updates the requirements on [uvicorn](https://github.com/Kludex/uvicorn) to permit the latest version.
- [Release notes](https://github.com/Kludex/uvicorn/releases)
- [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md)
- [Commits](https://github.com/Kludex/uvicorn/compare/0.52.1...0.52.4)

---
updated-dependencies:
- dependency-name: uvicorn
  dependency-version: 0.52.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-03 16:25:58 -06:00
dependabot[bot]
22cf798052
build(deps): bump fast-uri from 3.1.5 to 3.1.7 in /app (#693)
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.5 to 3.1.7.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](https://github.com/fastify/fast-uri/compare/v3.1.5...v3.1.7)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:25:47 -06:00
dependabot[bot]
cc85c1a7e6
build(deps): update rarfile requirement from >=4.2 to >=4.5 (#682)
Updates the requirements on [rarfile](https://github.com/markokr/rarfile) to permit the latest version.
- [Release notes](https://github.com/markokr/rarfile/releases)
- [Changelog](https://github.com/markokr/rarfile/blob/master/doc/news.rst)
- [Commits](https://github.com/markokr/rarfile/compare/v4.2...v4.5)

---
updated-dependencies:
- dependency-name: rarfile
  dependency-version: '4.5'
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: lrnselfreliance <33263708+lrnselfreliance@users.noreply.github.com>
2026-09-03 16:25:24 -06:00
lrnselfreliance
9bc067886f
videos: deflake event-loop-blocking search test (#699)
test_search_videos_does_not_block_event_loop asserted a wall-clock
threshold (echo < 0.4s), which fails on loaded CI machines from
scheduler noise alone. Replace the timing with ordering: the mocked
search parks its thread on an Event, the echo is sent only once the
handler is provably inside the query, and the test asserts the echo
completes while the query is still held open. Events are only waited
on off the event loop, so a regression fails the ordering assertion
instead of deadlocking.

Verified both directions: passes repeatedly against the real endpoint,
and fails when the endpoint is changed to run the search on the loop.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 16:24:09 -06:00
dependabot[bot]
d5b933920a
build(deps): update fastapi requirement from >=0.140.13 to >=0.141.1 in /controller (#690)
Updates the requirements on [fastapi](https://github.com/fastapi/fastapi) to permit the latest version.
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](https://github.com/fastapi/fastapi/compare/0.140.13...0.141.1)

---
updated-dependencies:
- dependency-name: fastapi
  dependency-version: 0.141.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-03 16:15:29 -06:00
lrnselfreliance
88d5d55283
files: left-align the preview toolbar (#698)
* files: left-align the preview toolbar, Open joins the button row

The modal actions row packs its children to the right, which suits a
confirming button but stranded the preview's toolbar. The toolbar now
fills the row so its buttons read from the left, with Open first in
the same row as Download. On phones the tags selector always takes its
own line instead of squeezing in beside the buttons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tags: the add-tag button is violet like every other tag button

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* share: a real button everywhere but the navbar corner

The share trigger was a bare anchor-wrapped glyph, which matched the
navbar's other corner icons but looked out of place beside real buttons
in the file preview toolbar. It is an IconButton by default now; the
navbar keeps the bare glyph via asIcon.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 16:03:17 -06:00
lrnselfreliance
9b8ebdc9b3
files: readable text previews in dark themes (#697)
* files: readable text previews in dark themes

A same-origin text document inherits the page's color-scheme, so dark
themes rendered the browser's default light text on the preview's
forced white background. Force a light color-scheme on the preview
iframe so text is always black on white.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* files: text previews follow the theme

Instead of pinning the preview iframe to a light scheme, follow the
theme: dark, night, and amber render the document light-on-dark against
the theme's panel color; light stays black-on-white.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 15:47:44 -06:00
lrnselfreliance
f1270b4e6c
archive: raise archive service request limit to 500MB (#692)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Uploaded singlefiles are base64-encoded into JSON before being sent to the
archive container, so an 88MB upload becomes a ~118MB request.  Sanic's default
REQUEST_MAX_SIZE of 100MB rejected it with a 413 before the handler ran, and the
API surfaced this as KeyError: 'singlefile'.

Only the Docker deployment is affected; Debian/RPi extract in-process.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-01 20:09:35 -06:00
lrnselfreliance
b44a5f1941
downloads: daily limit count was lost on restart (#679)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Completed once-downloads are never written to the downloads config, and
importing the config (which runs on every startup) deleted any DB Download
the config did not list.  That erased the last_download_attempt rows the
per-domain and global daily limits count from, so a restart reset the count.

Keep completed once-downloads on import; pending/failed and recurring
downloads still follow the config as the source of truth.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:24:52 -06:00
lrnselfreliance
97a6566e3d
Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678)
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page

Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:

* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
  the blocking `multiprocessing.Event.wait(timeout=1)`.  With no switch pending
  (the normal state) this froze the loop for a second at a time; about a third
  of all requests, including trivial ones, took ~1,000 ms.  It now polls
  `is_set()` and returns immediately.

* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
  ran synchronous SQLite queries on the event loop, so one slow search stalled
  every other request on that worker.  They now run via `asyncio.to_thread`.

* The unfiltered Videos page (`POST /api/videos/search` with only an order)
  used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
  every video's index entry before LIMIT applies: seconds cold on a large
  library on a spinning disk, and ~10 s hot for a deep page.  It now seeks each
  exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
  rows and merges them, with the total from a separate count cached for 60 s.
  Results are identical to the generic query.

Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results).  Host profiles live in a
gitignored JSON file; an example is included.

Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Serve the video description from its own endpoint instead of Video.__json__

`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk.  The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays.  The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.

Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions.  The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored.  The player page fetches it through
`useVideoExtras` alongside the comments.

Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: thread-local zim sessions, fast-path equivalence test, comments

* Zim search, single-zim search, and search estimates run libzim off the event
  loop, each opening its own session inside the worker thread rather than
  passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
  query and asserts identical rows and paging for both date orders.  The one
  intentional difference is documented: past the end, the fast path still
  reports the true total.
* switch_worker returns before its try block so the empty path does not clear
  the event; its comment states the invariant only.  The copied "synchronous
  SQLite" comments are reduced to one line in one place.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:53 -06:00
Roland
d79a44c0a1
Bumping yt-dlp version.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-08-25 16:44:10 -06:00
lrnselfreliance
bb42798c69
videos: a collaboration video's channel directory conflict killed the download (#677)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* videos: a collaboration video's channel directory conflict killed the download

YouTube collaboration videos appear in every collaborator's feed but their
metadata names only the primary uploader's channel. When that channel's
computed directory already belonged to an existing Channel that no longer
matched by source_id/url/name, channel creation raised
ChannelDirectoryConflict and the download failed unrecoverably. Now the
channel that owns the directory is used instead of creating a conflicting one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: trim comments that restated the code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 20:37:51 -06:00
lrnselfreliance
340aacb633
videos: parse the published date out of audio file names too (#676)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
NAME_PARSER's suffix whitelist had no audio suffixes (and no mkv/m4v),
so a correctly-named audio file got no published date and sorted to the
very end of its Channel -- which reads as missing. The whitelist now
carries the media suffixes a Video FileGroup can lead with.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 12:14:07 -06:00
lrnselfreliance
fdb7258160
videos: an uploaded audio file never appeared in its Channel (#675)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* videos: an uploaded audio file never appeared in its Channel

The upload flow asks Video.can_model, which only accepted video/*, so an
uploaded .flac was indexed with no Video row -- and the batch modeler's
`indexed != True` gate then excluded it on every later refresh, leaving
the file permanently invisible in its Channel.

can_model now accepts audio/* like the batch modeler always has, and the
modeler gates on the Video row being missing (mirroring the doc_modeler
fix for the same class of bug) so already-stuck files heal on the next
refresh. Audio playlists (.m3u detects as audio/x-mpegurl) are excluded
from both paths and unmodeled by video_cleanup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: widen the playlist denylist, prove modeler termination

The denylist gains audio/x-scpls and audio/scpls -- .pls detection
varies by libmagic version, and any audio/* playlist type must be
refused wherever it comes back. Modeling one file is extracted as
_model_video so a test can force persistent failure and prove the loop
terminates; failed_ids is recorded before the PYTEST re-raise so no
failure path can re-select an id forever. Playlist mimetypes are bound
through json_each like the rest of video_cleanup, and the comments
state the invariants instead of the history.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: the playlist test must not depend on libmagic's m3u opinion

CI's libmagic detects .m3u as text/plain where a Mac's says
audio/x-mpegurl. The invariant is that a playlist mimetype is never
modeled by either path, so the test forces the mimetype like the other
denylist tests instead of asserting what detection returned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 15:18:15 -06:00