mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-11-19 22:46:20 -05:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a5ab01198b
|
The backup drive keeps deleted files for 30 days, then removes them (#999)
* The backup drive keeps deleted files for 30 days, then removes them The fifth part of replacing the sync. A backup drive never deleted anything, so it grew forever and kept every file the media directory had ever held. Now, like a secondary: - A file deleted from the media directory, or the old version of one that changed, is moved to a dated folder in .backup-trash on the backup drive. Folders older than backup_delete_after_days (sync.yaml; 30 by default, also for drives set up before this; none never deletes) are removed, before and after each backup, and the oldest go sooner when the drive is nearly full. - A backup refuses a media directory that is not mounted, or has no config/wrolpi.yaml: copied with deletes, an empty folder would move the whole backup drive to the trash. - The Backup Drive panel has the same "Delete after N days / Never delete them" control as the Sync panel, now one component, and refuses an empty number of days. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A nearly full drive keeps the trash it just filled, and makes room for the database Review found, in the backup drive's trash and in the secondary's sync, which share its shape: - Below 1 GiB free, the purge after a copy removed the newest trash folder too. Moving a file to the trash is a rename, which frees nothing, so on a nearly full drive a deleted file was gone at the end of the same run instead of 30 days later. The newest folder is now only removed by age (0 days still deletes at once). - The purge made room only down to 1 GiB, so a database larger than the free space failed to be copied every time while the trash was full. Before the database is written (the backup's snapshot, the secondary's copy of its primary's), the trash now makes room for the whole database. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8cfe36b3ea
|
A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it (#995)
* A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it The third part of replacing the sync. - The Controller syncs within the daily window (every 15 minutes after the last sync finished, in WROLPi's timezone), as the API did. The API's loop no longer runs the old engine, and its /api/sync/run and /api/sync/cancel ask the Controller. - The Sync page and the overlay that covers the app read the sync's status from the Controller, which is the only one that can answer while the API is stopped. The overlay is also shown when the settings cannot be read, as happens while the API is stopped. Sync Now and Stop go to the Controller. - The Controller's fallback page, where Caddy sends the browser while the API is down, shows the sync's progress and a Stop button. - New on the Sync page: how long files the primary deletes are kept (30 days by default, or never), and a note that an excluded folder's files are still listed on a secondary. The old engine's code is removed in the next part. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove SyncRunRequest, unused since a sync is asked of the Controller Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * In Docker, a sync's progress moves while a large file is copied Docker's logs hold a line until it ends, and rsync's progress for one large file is a single line rewritten with carriage returns, so the page showed 0% until the file was done (hours, for a large map). The rsync helper's output is now read by attaching to it, before it is started so that nothing is missed. Found by a sync between two Docker WROLPis. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A sync is recovered before the schedule can start another, and an empty days value is refused Review found: - The schedule and the recovery of an interrupted sync started side by side when the Controller started. The state an interrupted sync leaves is due, so inside the window a new sync could start while the recovery restarted the API and wrote over the new sync's status. Every tick of the schedule now recovers first, and only a status of "running" with no sync holding the lock counts as interrupted. A sync records its outcome before it frees the lock, so a tick between the two cannot mistake it for an interrupted one. - The Controller may start before the drive is mounted. The empty record it read then was kept and saved over the real one, losing the last successful sync. The state file is now read again until it can be. - A cleared number of days was saved as 0, which empties the trash at the next sync. It is now refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6772762ce9
|
A secondary deletes its own stale write-ahead logs, and the sync leaves no stray files (#994)
* A secondary deletes its own stale write-ahead logs, and the sync leaves no stray files Found by the first sync between two real machines: - The primary's unfinished files (SQLite logs, partial downloads) were excluded from the copy, and an excluded path is never deleted on the receiving side. So a secondary kept its own write-ahead log beside a map search database the primary had replaced, and SQLite would apply that log to the new database. These patterns are now only hidden on the primary's side, so the secondary's stale ones are deleted (to the trash). - Reading the incoming snapshot to check it made a -wal and -shm beside it, which stayed behind when the snapshot was moved into place. It is now opened immutable. - The primary's info, written by the Controller (root), belonged to root. Files the Controller writes in the media directory now belong to its owner, like the directories it makes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * With deletes turned off, a secondary still drops its stale unfinished files Review found that the hide rules of the previous commit only take effect through --delete, which is not given when delete_after_days is null (never delete). A secondary then kept its own write-ahead log beside a database the primary had replaced, and SQLite applied it to the new database. Without a trash, --delete is now given with every path protected except the unfinished patterns (and what is inside an unfinished folder), so only those are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ea7cc4dec2
|
The Controller can make a secondary a copy of its primary with rsync (#992)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Dead Code / vulture (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* The Controller can make a secondary a copy of its primary with rsync The second part of replacing the sync. The Controller runs it, because the secondary's API is stopped while its library is replaced: checks, a database snapshot on the primary, stop the API, rsync every file from the primary's daemon (ignored directories included), fetch the snapshot and put it in the database's place, start the API, purge. - Files the primary deleted or changed are moved to a dated folder in .sync-trash, and removed after delete_after_days (a new sync.yaml setting, 30 by default; none never deletes), or sooner when the drive is nearly full. - A primary of another version or database migration, or with its media unmounted, is refused before anything is touched. The database is replaced last, so a sync that stops earlier leaves one that works. - On a Raspberry Pi and Debian rsync runs as the wrolpi user; in Docker in a helper container made from the api container. A Controller that starts after an interrupted sync starts the API. - When a secondary's API starts it takes the library settings of the primary's wrolpi.yaml, which the sync left. Reached only through the Controller's /api/sync/run for now; the existing sync and its schedule are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The sync copies no link that leaves the library, and root follows none in the media directory Security review of the Controller's sync: - An exclusion that ends in a slash matches only a directory. A file or link named config/ssl, tags or config/backup on the primary would have replaced this machine's directory of that name. The exclusions now match whatever has the name. - Links were copied as they are. One that points outside the primary's library would, on the secondary, point at the secondary's own files, which the web server then serves. Only links that stay inside the library are copied, and no device files. - The Controller is root, in a media directory that belongs to another user. It no longer writes, reads, replaces or purges through a symbolic link there. Found on the way: the directory the Controller made for the incoming database belonged to root, so rsync, which runs as the library's owner, could not have written into it. It now belongs to the media directory's owner. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A sync ends when its primary stops answering, and makes room before it copies Review found: - rsync waits forever by default. A primary that lost power or its link during a copy left the secondary's API stopped until someone cancelled. rsync now gives up after 30 seconds without a connection and 5 minutes without data, which ends the run and starts the API. - The trash was only purged after a run that succeeded. The trash is what fills the drive, and a full drive fails every run, so it never got its turn. It is now also purged before the copy. - A value in sync.yaml that could not be used raised after the run lock was taken and before the handlers that release it, leaving "A sync is already running" until the Controller restarted. - The playlists directory left out of the copy was this machine's, not the primary's, whose hard links would have arrived as full copies. The primary's wrolpi.yaml is now fetched before the copy, and both are left out. - A primary whose media directory is elsewhere is refused: its database names every file by its full path. - In Docker rsync is its own container, which outlives a Controller that stops. A Controller that starts after an interrupted sync now removes it before it starts the API. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |