A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it (#995)

* A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it

The third part of replacing the sync.

- The Controller syncs within the daily window (every 15 minutes after the last sync finished, in WROLPi's
  timezone), as the API did.  The API's loop no longer runs the old engine, and its /api/sync/run and
  /api/sync/cancel ask the Controller.
- The Sync page and the overlay that covers the app read the sync's status from the Controller, which is the only
  one that can answer while the API is stopped.  The overlay is also shown when the settings cannot be read, as
  happens while the API is stopped.  Sync Now and Stop go to the Controller.
- The Controller's fallback page, where Caddy sends the browser while the API is down, shows the sync's progress
  and a Stop button.
- New on the Sync page: how long files the primary deletes are kept (30 days by default, or never), and a note that
  an excluded folder's files are still listed on a secondary.

The old engine's code is removed in the next part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove SyncRunRequest, unused since a sync is asked of the Controller

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* In Docker, a sync's progress moves while a large file is copied

Docker's logs hold a line until it ends, and rsync's progress for one large file is a single line rewritten with
carriage returns, so the page showed 0% until the file was done (hours, for a large map).  The rsync helper's
output is now read by attaching to it, before it is started so that nothing is missed.

Found by a sync between two Docker WROLPis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync is recovered before the schedule can start another, and an empty days value is refused

Review found:

- The schedule and the recovery of an interrupted sync started side by side when the Controller started.  The state
  an interrupted sync leaves is due, so inside the window a new sync could start while the recovery restarted the
  API and wrote over the new sync's status.  Every tick of the schedule now recovers first, and only a status of
  "running" with no sync holding the lock counts as interrupted.  A sync records its outcome before it frees the
  lock, so a tick between the two cannot mistake it for an interrupted one.
- The Controller may start before the drive is mounted.  The empty record it read then was kept and saved over the
  real one, losing the last successful sync.  The state file is now read again until it can be.
- A cleared number of days was saved as 0, which empties the trash at the next sync.  It is now refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
lrnselfreliance 2026-11-19 15:06:25 -07:00 • committed by GitHub
parent 6772762ce9
commit 8cfe36b3ea
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 708 additions and 128 deletions

View file

@ -29,10 +29,13 @@ import subprocess
import threading
import time
import uuid
import zoneinfo
from pathlib import Path
from typing import Callable, Optional
from urllib.parse import urlparse
import asyncio
import httpx
import yaml
@ -58,6 +61,10 @@ STATE_FILE = "config/sync_state.yaml"
DEFAULT_DELETE_AFTER_DAYS = 30
# Never fill the secondary's drive; leave this much free.
MIN_FREE_BYTES = 1024 ** 3
# Within the daily sync window, a sync starts this long after the last one finished.
SYNC_REPEAT_SECONDS = 15 * 60
# How often the schedule is looked at.
SCHEDULE_INTERVAL_SECONDS = 60
SNAPSHOT_TIMEOUT_SECONDS = 60 * 60
API_START_TIMEOUT_SECONDS = 10 * 60
# A sync runs unattended with the API stopped: a primary that loses power or its link must end the run, not hold it.
@ -118,6 +125,9 @@ def default_status() -> dict:
_status = None
# True once the state file was read or written: until then the drive may not be mounted, and what is in memory must
# not be mistaken for (or saved over) what the last sync recorded.
_status_from_file = False
_status_lock = threading.Lock()
_run_lock = threading.Lock()
_cancel = threading.Event()
@ -132,16 +142,19 @@ def _state_file() -> Path:
def get_status() -> dict:
"""The progress and outcome of the sync. Read from the state file the first time, so a restarted Controller
still knows when the last sync succeeded."""
global _status
"""The progress and outcome of the sync. Read from the state file, so a restarted Controller still knows when
the last sync succeeded; read again until that works, as the Controller may start before the drive is mounted."""
global _status, _status_from_file
with _status_lock:
if _status is None:
_status = default_status()
if not _status_from_file:
try:
saved = yaml.safe_load(checked_path(_state_file()).read_text())
if isinstance(saved, dict):
_status = default_status()
_status.update({k: v for k, v in saved.items() if k in _status})
_status_from_file = True
except (OSError, yaml.YAMLError, SyncError):
pass
return dict(_status)
@ -154,8 +167,10 @@ def update_status(save: bool = False, **changes) -> dict:
_status.update(changes)
status = dict(_status)
if save:
global _status_from_file
try:
write_file(_state_file(), yaml.safe_dump(status))
_status_from_file = True
except (OSError, SyncError) as e:
logger.warning("Could not save the sync state: %s", e)
return status
@ -374,11 +389,15 @@ class DockerRunner:
def rsync(self, arguments: list, on_line: Callable[[str], None]) -> int:
client, api = _get_client(), self._api()
self.stop_rsync()
helper = client.containers.run(
api.image.id, arguments, entrypoint="rsync", name=self.HELPER_NAME, detach=True,
helper = client.containers.create(
api.image.id, arguments, entrypoint="rsync", name=self.HELPER_NAME,
user=api.attrs["Config"].get("User") or None, volumes_from=[api.name])
try:
stream = helper.logs(stream=True, follow=True)
# Attached, not followed through the logs: Docker's logs hold a line until it ends, and rsync's progress
# for one large file is a single line, rewritten with carriage returns until the file is done.
# Attached before it starts, so nothing it writes is missed.
stream = helper.attach(stdout=True, stderr=True, stream=True)
helper.start()
def read(_: int = 0) -> bytes:
return next(stream, b"")
@ -672,16 +691,17 @@ def run_sync(runner=None) -> dict:
def _finish(runner, api_stopped: bool, **changes) -> dict:
"""Whatever happened, the API runs again and the lock is free."""
"""Whatever happened, the API runs again and the lock is free. The outcome is recorded before the lock is
freed: a status that says "running" while no sync holds the lock is taken for an interrupted sync."""
try:
if api_stopped:
runner.start_api()
trash = get_media_directory() / TRASH_DIRECTORY
return update_status(save=True, phase=None, finished_at=_now(),
trash_bytes=directory_size(trash) if trash.is_dir() and not trash.is_symlink() else 0,
**changes)
finally:
_run_lock.release()
trash = get_media_directory() / TRASH_DIRECTORY
return update_status(save=True, phase=None, finished_at=_now(),
trash_bytes=directory_size(trash) if trash.is_dir() and not trash.is_symlink() else 0,
**changes)
def _read_yaml(file: Path) -> dict:
@ -698,10 +718,11 @@ def request_cancel():
def recover_interrupted_sync(runner=None):
"""Called when the Controller starts. A sync that was running when the Controller stopped left the API stopped;
start it, and say the sync was interrupted."""
"""A status that says "running" while no sync holds the lock was left by a Controller that stopped during a
sync, which left the API stopped: start it, and say the sync was interrupted. Looked at by every tick of the
schedule, before it starts anything, and so also once the state file can be read after the drive mounts."""
status = get_status()
if status["state"] != "running":
if status["state"] != "running" or _run_lock.locked():
return
logger.warning("A sync was interrupted; starting the API")
runner = runner or get_runner()
@ -712,3 +733,64 @@ def recover_interrupted_sync(runner=None):
runner.start_api()
update_status(save=True, state="error", phase=None, finished_at=_now(),
last_error="The sync was interrupted")
# ---------------------------------------------------------------------------------------------------------------------
# The schedule.
def local_now() -> datetime.datetime:
"""Now, in WROLPi's timezone (wrolpi.yaml), or the system's when none is set."""
timezone = _read_yaml(get_media_directory() / "config/wrolpi.yaml").get("timezone")
if timezone:
with contextlib.suppress(zoneinfo.ZoneInfoNotFoundError, ValueError):
return datetime.datetime.now(zoneinfo.ZoneInfo(timezone))
return datetime.datetime.now().astimezone()
def _parse_time(value) -> Optional[datetime.time]:
with contextlib.suppress(TypeError, ValueError):
return datetime.time.fromisoformat(str(value))
return None
def in_window(start: Optional[str], end: Optional[str], now: datetime.time) -> bool:
"""True if `now` is within the daily window from `start` to `end`, which may run past midnight."""
start, end = _parse_time(start), _parse_time(end)
if start is None or end is None:
return False
if start <= end:
return start <= now < end
return now >= start or now < end
def sync_is_due(config: dict, status: dict, now: datetime.datetime) -> bool:
"""True if the daily sync window is open and the last sync finished long enough ago. No window: never (a sync
only runs when asked)."""
if not in_window(config.get("window_start"), config.get("window_end"), now.time()):
return False
with contextlib.suppress(TypeError, ValueError):
finished_at = datetime.datetime.fromisoformat(status["finished_at"])
return (now - finished_at).total_seconds() >= SYNC_REPEAT_SECONDS
return True
def schedule_tick():
"""Start a sync if one is due. A sync that cannot start (WROL Mode, a primary, no primary) is not due."""
recover_interrupted_sync()
config = read_sync_config()
if skip_reason(config) is not None:
return
if sync_is_due(config, get_status(), local_now()):
logger.info("The sync window is open")
run_sync()
async def sync_schedule_loop():
"""Sync within the daily window for as long as the Controller runs."""
while True:
try:
await asyncio.to_thread(schedule_tick)
except Exception as e:
logger.error("The sync schedule failed", exc_info=e)
await asyncio.sleep(SCHEDULE_INTERVAL_SECONDS)

View file

@ -59,7 +59,7 @@ from controller.lib.status import (
get_uptime_status,
)
from controller.lib.status_worker import status_worker_loop
from controller.lib.sync import recover_interrupted_sync
from controller.lib.sync import sync_schedule_loop
from controller.lib.systemd import get_all_services_status
# Templates directory
@ -126,8 +126,9 @@ async def lifespan(app: FastAPI):
else:
logger.info("Primary drive not mounted - using default configuration")
# A sync that was running when the Controller stopped left the API stopped.
app.state.sync_recovery_task = asyncio.create_task(asyncio.to_thread(recover_interrupted_sync))
# A secondary syncs within its daily window. Each tick first recovers a sync that was interrupted (the API it
# stopped is started again), so nothing races that recovery.
app.state.sync_schedule_task = asyncio.create_task(sync_schedule_loop())
# Docker: the library is served to secondary WROLPis from a helper container, kept matching the api container.
if is_docker_mode():
@ -152,8 +153,9 @@ async def lifespan(app: FastAPI):
await status_task
except asyncio.CancelledError:
pass
if rsync_daemon_task := getattr(app.state, "rsync_daemon_task", None):
rsync_daemon_task.cancel()
for name in ("rsync_daemon_task", "sync_schedule_task"):
if task := getattr(app.state, name, None):
task.cancel()
# Create FastAPI app

View file

@ -169,6 +169,33 @@
display: block;
}
/* A secondary WROLPi copying its primary; its API is stopped meanwhile. */
.sync-card {
border: 2px solid #3498db;
border-radius: 4px;
padding: 0.75rem 1rem;
margin-bottom: 1rem;
display: none;
}
.sync-card.show {
display: block;
}
.sync-card .sync-bar {
height: 0.6rem;
background: rgba(127, 127, 127, 0.3);
border-radius: 3px;
overflow: hidden;
margin: 0.5rem 0;
}
.sync-card .sync-bar div {
height: 100%;
background: #3498db;
width: 0;
}
.fallback-banner .loading {
border-color: #333;
border-top-color: #000;
@ -945,6 +972,15 @@
<a id="view-files-btn" class="btn btn-info" style="{% if not app_ready %}display: none;{% endif %} font-size: 1.1rem; padding: 0.6rem 1.5rem; margin-left: 0.5rem;" href="#">View Files</a>
</div>
<div id="sync-card" class="sync-card" role="status" aria-live="polite">
<strong>Syncing from the primary</strong>
<p id="sync-text" style="margin: 0.25rem 0;"></p>
<div class="sync-bar"><div id="sync-bar"></div></div>
<p style="margin: 0.25rem 0; font-size: 0.9em;">WROLPi is unavailable until the sync finishes, then this page
returns to it.</p>
<button id="sync-stop" class="btn btn-danger" type="button">Stop Sync</button>
</div>
<div id="fallback-banner" class="fallback-banner">
<span class="loading"></span>
Main UI is unavailable. Using Controller fallback. Will redirect when ready...
@ -3077,6 +3113,52 @@
refreshToggleButtons();
}, updateInterval);
}
// A secondary WROLPi's sync (controller/lib/sync.py). Its API is stopped while it runs, which is why this page
// is shown; say what is happening. Relative URLs: /controller/api/... behind Caddy, /api/... when direct.
(function () {
const PHASES = {
checks: 'Checking the primary', snapshot: 'The primary is copying its database',
files: 'Copying files', database: 'Copying the database', starting: 'Starting WROLPi',
purge: 'Emptying the trash',
};
const card = document.getElementById('sync-card');
const text = document.getElementById('sync-text');
const bar = document.getElementById('sync-bar');
const stop = document.getElementById('sync-stop');
async function checkSync() {
try {
const resp = await fetch('api/sync/status', {signal: AbortSignal.timeout(5000)});
if (!resp.ok) {
return;
}
const sync = await resp.json();
const running = sync.state === 'running';
card.classList.toggle('show', running);
if (running) {
const copied = (sync.phase === 'files' || sync.phase === 'database')
? ` (${formatBytes(sync.bytes_transferred || 0)})` : '';
text.textContent = `${PHASES[sync.phase] || 'Syncing'}${copied} from ${sync.primary_url || 'the primary'}`;
bar.style.width = `${Math.min(100, sync.percent || 0)}%`;
}
} catch (e) {
// The Controller is busy or restarting; ask again.
}
}
stop.addEventListener('click', async () => {
stop.disabled = true;
try {
await fetch('api/sync/cancel', {method: 'POST'});
} finally {
setTimeout(() => { stop.disabled = false; }, 3000);
}
});
checkSync();
setInterval(checkSync, 2000);
})();
</script>
</body>
</html>

View file

@ -2,8 +2,10 @@
Unit tests for controller.lib.sync: a secondary WROLPi copying its primary with rsync.
"""
import asyncio
import datetime
import os
import re
import shutil
import socket
import sqlite3
@ -35,6 +37,7 @@ def media(tmp_path, monkeypatch) -> Path:
(directory / "config/sync.yaml").write_text(yaml.safe_dump(dict(role="secondary", primary_url=PRIMARY)))
monkeypatch.setenv("MEDIA_DIRECTORY", str(directory))
monkeypatch.setattr(sync, "_status", None)
monkeypatch.setattr(sync, "_status_from_file", False)
sync._cancel.clear()
yield directory
assert not sync._run_lock.locked(), "the test left a sync running"
@ -352,7 +355,7 @@ class TestRunSync:
assert yaml.safe_load((media / sync.PRIMARY_INFO).read_text())["media_directory"] == "/media/wrolpi"
assert (media / sync.PRIMARY_WROLPI_CONFIG).is_file()
# A restarted Controller reads the state file.
sync._status = None
sync._status, sync._status_from_file = None, False
assert sync.get_status()["last_success_at"] == status["last_success_at"]
def test_leaves_out_the_playlists_of_both_machines(self, media, primary_api):
@ -428,7 +431,7 @@ class TestRunSync:
def test_interrupted_sync_is_recovered(self, media):
"""A Controller that stopped during a sync left the API stopped."""
sync.update_status(save=True, state="running", phase="files")
sync._status = None
sync._status, sync._status_from_file = None, False
runner = FakeRunner(media)
sync.recover_interrupted_sync(runner)
# In Docker rsync is its own container and is still replacing files: it is stopped before the API starts.
@ -671,3 +674,178 @@ def test_never_delete_still_removes_stale_logs(tmp_path):
assert (secondary / "kept folder/a.txt").read_text() == "kept"
assert read_database(secondary / "map/region.search.db") == "the primary's new index"
assert not (secondary / ".sync-trash").exists()
class TestSchedule:
@pytest.mark.parametrize("start, end, now, expected", [
("01:00", "05:00", "03:00", True),
("01:00", "05:00", "05:00", False),
("01:00", "05:00", "00:59", False),
# Past midnight.
("22:00", "02:00", "23:30", True),
("22:00", "02:00", "01:59", True),
("22:00", "02:00", "12:00", False),
(None, "02:00", "01:00", False),
("not a time", "02:00", "01:00", False),
])
def test_in_window(self, start, end, now, expected):
assert sync.in_window(start, end, datetime.time.fromisoformat(now)) is expected
def test_due_every_fifteen_minutes_inside_the_window(self):
tz = datetime.timezone(datetime.timedelta(hours=-7))
now = datetime.datetime(2026, 11, 19, 3, 0, tzinfo=tz)
config = dict(window_start="01:00", window_end="05:00")
assert sync.sync_is_due(config, dict(finished_at=None), now)
recent = (now - datetime.timedelta(minutes=14)).astimezone(datetime.timezone.utc).isoformat()
assert not sync.sync_is_due(config, dict(finished_at=recent), now)
older = (now - datetime.timedelta(minutes=15)).astimezone(datetime.timezone.utc).isoformat()
assert sync.sync_is_due(config, dict(finished_at=older), now)
assert not sync.sync_is_due(dict(window_start=None, window_end=None), dict(finished_at=None), now), \
"without a window a sync only runs when asked"
def test_local_now_uses_wrolpis_timezone(self, media):
(media / "config/wrolpi.yaml").write_text("timezone: Asia/Tokyo\n")
assert sync.local_now().utcoffset() == datetime.timedelta(hours=9)
(media / "config/wrolpi.yaml").write_text("timezone: Not/AZone\n")
assert sync.local_now().tzinfo is not None
def test_tick(self, media, monkeypatch):
runs = []
monkeypatch.setattr(sync, "run_sync", lambda: runs.append(1))
monkeypatch.setattr(sync, "local_now",
lambda: datetime.datetime(2026, 11, 19, 3, 0).astimezone())
sync.schedule_tick()
assert runs == [], "no window"
(media / "config/sync.yaml").write_text(yaml.safe_dump(dict(
role="secondary", primary_url=PRIMARY, window_start="01:00", window_end="05:00")))
sync.schedule_tick()
assert runs == [1]
(media / "config/.wrol_mode").touch()
sync.schedule_tick()
assert runs == [1], "WROL Mode stops the schedule"
@pytest.mark.asyncio
async def test_loop_survives_a_failure(self, monkeypatch):
ticks = []
def tick():
ticks.append(1)
raise RuntimeError("the drive went away")
monkeypatch.setattr(sync, "schedule_tick", tick)
monkeypatch.setattr(sync, "SCHEDULE_INTERVAL_SECONDS", 0.01)
task = asyncio.create_task(sync.sync_schedule_loop())
await asyncio.sleep(0.2)
task.cancel()
assert len(ticks) >= 3
def test_fallback_page_shows_the_sync(media):
"""The browser is sent to the Controller's page while the API is stopped; it says a sync is the reason."""
response = TestClient(app).get("/")
assert response.status_code == 200
assert b'id="sync-card"' in response.content and b"api/sync/status" in response.content
assert b"api/sync/cancel" in response.content
def test_frontend_fixture_matches_the_status():
"""The frontend's tests render the sync from controllerSyncStatusFixture; it must have the keys the Controller
sends, or a renamed field passes those tests and breaks the page."""
source = (Path(__file__).parent.parent.parent / "app/src/test-fixtures.js").read_text()
match = re.search(r"export const controllerSyncStatusFixture = \(overrides = \{}\) => \(\{(.*?)\n}\);",
source, re.S)
assert match, "controllerSyncStatusFixture is not in test-fixtures.js in the expected form"
keys = set(re.findall(r"^ ([a-z0-9_]+):", match.group(1), re.M))
assert keys == set(sync.default_status())
def test_docker_rsync_reads_progress_as_it_comes(media, monkeypatch):
"""The helper's output is read by attaching to it before it starts. Docker's logs would hold rsync's progress
for a large file (one line, rewritten with carriage returns) until the file was done."""
from unittest import mock
events = []
helper = mock.Mock()
helper.attach.side_effect = lambda **kwargs: events.append(("attach", kwargs)) or iter(
[b" 1000 10%\r", b" 5000 50%\r", b"Number of regular files transferred: 1\n"])
helper.start.side_effect = lambda: events.append(("start", None))
helper.wait.return_value = {"StatusCode": 0}
api = mock.Mock()
api.image.id, api.name, api.attrs = "sha256:image", "wrolpi-api-1", {"Config": {"User": "1000:1000"}}
client = mock.Mock()
client.containers.create.return_value = helper
client.containers.get.side_effect = lambda name: api if name == "wrolpi-api-1" else (_ for _ in ()).throw(
RuntimeError("no such container"))
monkeypatch.setattr(sync, "_get_client", lambda: client)
lines = []
assert sync.DockerRunner().rsync(["--version"], lines.append) == 0
assert [i[0] for i in events] == ["attach", "start"], "attached before it starts"
assert events[0][1]["stream"] is True
assert lines == [" 1000 10%", " 5000 50%", "Number of regular files transferred: 1"]
helper.logs.assert_not_called()
helper.remove.assert_called_once_with(force=True)
class TestRecoveryAndSchedule:
"""A Controller that starts after an interrupted sync must recover it before the schedule starts another."""
def interrupted(self, media):
(media / "config/sync.yaml").write_text(yaml.safe_dump(dict(
role="secondary", primary_url=PRIMARY, window_start="00:00", window_end="23:59")))
(media / "config/sync_state.yaml").write_text(yaml.safe_dump(dict(state="running", phase="files",
finished_at=None)))
def test_tick_recovers_first_and_starts_nothing(self, media, monkeypatch):
self.interrupted(media)
runner, runs = FakeRunner(media), []
monkeypatch.setattr(sync, "get_runner", lambda: runner)
monkeypatch.setattr(sync, "run_sync", lambda: runs.append(1))
monkeypatch.setattr(sync, "local_now", lambda: datetime.datetime.now().astimezone())
sync.schedule_tick()
assert runner.events == ["stop_rsync", "start_api"]
assert sync.get_status()["state"] == "error" and sync.get_status()["last_error"] == "The sync was interrupted"
assert runs == [], "the recovered sync just finished; the next one waits its 15 minutes"
def test_a_running_sync_is_not_recovered(self, media, monkeypatch):
"""A status of "running" while this Controller's own sync holds the lock is no interrupted sync."""
self.interrupted(media)
runner = FakeRunner(media)
assert sync._run_lock.acquire(blocking=False)
try:
sync.recover_interrupted_sync(runner)
finally:
sync._run_lock.release()
assert runner.events == [] and sync.get_status()["state"] == "running"
def test_state_file_read_once_the_drive_is_mounted(self, media):
"""The Controller may start before the drive is mounted: what it could not read is read later, not
replaced by an empty record."""
state = media / "config/sync_state.yaml"
assert not state.exists()
assert sync.get_status()["last_success_at"] is None
state.write_text(yaml.safe_dump(dict(state="idle", last_success_at="2026-11-18T03:00:00+00:00")))
assert sync.get_status()["last_success_at"] == "2026-11-18T03:00:00+00:00"
# Once read, what is in memory is the record.
sync.update_status(last_error="x")
state.write_text(yaml.safe_dump(dict(state="idle", last_success_at=None)))
assert sync.get_status()["last_success_at"] == "2026-11-18T03:00:00+00:00"
def test_outcome_is_recorded_before_the_lock_is_freed(self, media, primary_api, monkeypatch):
"""Otherwise a tick between the two would take the sync for an interrupted one."""
locked_when_saved = []
update_status = sync.update_status
def watch(save=False, **changes):
if save and changes.get("finished_at"):
locked_when_saved.append(sync._run_lock.locked())
return update_status(save=save, **changes)
monkeypatch.setattr(sync, "update_status", watch)
sync.run_sync(FakeRunner(media))
assert locked_when_saved == [True]