* The Controller can make a secondary a copy of its primary with rsync
The second part of replacing the sync. The Controller runs it, because the secondary's API is stopped while its
library is replaced: checks, a database snapshot on the primary, stop the API, rsync every file from the primary's
daemon (ignored directories included), fetch the snapshot and put it in the database's place, start the API, purge.
- Files the primary deleted or changed are moved to a dated folder in .sync-trash, and removed after
delete_after_days (a new sync.yaml setting, 30 by default; none never deletes), or sooner when the drive is nearly
full.
- A primary of another version or database migration, or with its media unmounted, is refused before anything is
touched. The database is replaced last, so a sync that stops earlier leaves one that works.
- On a Raspberry Pi and Debian rsync runs as the wrolpi user; in Docker in a helper container made from the api
container. A Controller that starts after an interrupted sync starts the API.
- When a secondary's API starts it takes the library settings of the primary's wrolpi.yaml, which the sync left.
Reached only through the Controller's /api/sync/run for now; the existing sync and its schedule are unchanged.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* The sync copies no link that leaves the library, and root follows none in the media directory
Security review of the Controller's sync:
- An exclusion that ends in a slash matches only a directory. A file or link named config/ssl, tags or
config/backup on the primary would have replaced this machine's directory of that name. The exclusions now match
whatever has the name.
- Links were copied as they are. One that points outside the primary's library would, on the secondary, point at
the secondary's own files, which the web server then serves. Only links that stay inside the library are copied,
and no device files.
- The Controller is root, in a media directory that belongs to another user. It no longer writes, reads, replaces
or purges through a symbolic link there.
Found on the way: the directory the Controller made for the incoming database belonged to root, so rsync, which
runs as the library's owner, could not have written into it. It now belongs to the media directory's owner.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* A sync ends when its primary stops answering, and makes room before it copies
Review found:
- rsync waits forever by default. A primary that lost power or its link during a copy left the secondary's API
stopped until someone cancelled. rsync now gives up after 30 seconds without a connection and 5 minutes without
data, which ends the run and starts the API.
- The trash was only purged after a run that succeeded. The trash is what fills the drive, and a full drive fails
every run, so it never got its turn. It is now also purged before the copy.
- A value in sync.yaml that could not be used raised after the run lock was taken and before the handlers that
release it, leaving "A sync is already running" until the Controller restarted.
- The playlists directory left out of the copy was this machine's, not the primary's, whose hard links would have
arrived as full copies. The primary's wrolpi.yaml is now fetched before the copy, and both are left out.
- A primary whose media directory is elsewhere is refused: its database names every file by its full path.
- In Docker rsync is its own container, which outlives a Controller that stops. A Controller that starts after an
interrupted sync now removes it before it starts the API.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* A secondary WROLPi copies its primary's library over HTTP
A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them. wrolpi.yaml contributes only its library
settings. Files deleted on the primary are kept.
A secondary never saves the synced configs and never downloads. Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy. It syncs in a daily window or on "Sync now"; WROL Mode stops it.
On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync has its own /admin tab; a sync's reads from the primary are bounded
The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.
A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped. A primary address may not carry a
user name or password (it is logged and shown). While a sync runs, the API
answers only the exact requests that show its progress or stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync keeps the secondary's replaced configs in today's config backup
The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Docker's Caddy logs every request
`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync reports its progress as it downloads, and refuses quietly
The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds. Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time. The primary's version is stored
without its trailing newline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A WROLPi can back up to a second drive; sync settings live in sync.yaml
The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted. The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start. A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears. Not in Docker, which
cannot reach other drives.
Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Sync tab shows the configured backup drive
getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured. It also dropped
`version`. A test now fails for any setting the API sends that getSettings
drops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup can go to a folder on its drive, and remembers its last success
The Backup Drive panel takes a folder on the chosen drive. The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused. A missing folder is created once the drive is checked.
When the last backup succeeded is saved in sync.yaml, so it is known after a
restart. "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The backup drive has its own window
The sync window was shown in the Sync panel but also scheduled the backup
drive. The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync and the backup drive can each exclude folders
A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory. Excluded
folders are never copied; what was copied before is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The folder browser does not offer lost+found
Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync never writes into config or tags, whatever their case
On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup writes the database snapshot only when the database changed
Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove the unused save_sync_config switch
Nothing activated it; SyncConfig saves directly. The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request). The
per-config overrides that only set successful_import are removed.
wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.
The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Download and Upload are disabled until a full refresh completes, and a
fresh install never had one until the user found the refresh link.
Startup now queues it, and the dashboard says why Download and Upload
are disabled meanwhile.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* zim: recompute the outdated-Zims flag after startup and on Delete
The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan. The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.
- The startup recompute now runs after the DB is up and configs are
imported, so it overrides the value `init_flags` restored from the DB.
Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
of showing a placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete
- Startup calls flags.init_flags() then flag_outdated_zim_files() right
after db_up, before the DB config import, so the disk scan always
overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
file was deleted.
- Trim narrative comments to the constraint they enforce.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A directory rename waits inline on a move job. When that move failed
(stale FileGroup rows colliding on primary_path), handle_move reverted
the files and sent a failure event but left the job pending. wait_for_job
only understands 'complete', so POST /api/files/rename spun until its
300s timeout and the Rename modal never closed.
The same sequence could kill file processing for the life of the process.
perpetual_signal treated any CancelledError as terminal and logged
nothing. Separately, perpetual_tasks_started is a shared Event that
survives the worker that set it: docker auto-reloads and worker crashes
left every later after_server_start as a no-op. Queued refreshes sat
untouched while worker_status reported idle. Restarting the API was the
only recovery.
Mark failed jobs failed and raise immediately from wait_for_job. Catch
unexpected cancellation and reschedule unless the app is actually
stopping. Track the perpetual-loop owner PID and retake the claim when
that process is gone; every Sanic worker still runs the file-worker
pump. Harden transfer_queue against the cross-process Empty race, and
stop request processes stealing from the shared public queue.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Storing video settings in session storage to help user start downloads consistently.
Applying channel tag name to new channels when download videos or channels.
Initializing map tiles when importing and resetting map.