files: a failed directory rename that hung for five minutes

A directory rename waits inline on a move job. When that move failed
(stale FileGroup rows colliding on primary_path), handle_move reverted
the files and sent a failure event but left the job pending. wait_for_job
only understands 'complete', so POST /api/files/rename spun until its
300s timeout and the Rename modal never closed.

The same sequence could kill file processing for the life of the process.
perpetual_signal treated any CancelledError as terminal and logged
nothing. Separately, perpetual_tasks_started is a shared Event that
survives the worker that set it: docker auto-reloads and worker crashes
left every later after_server_start as a no-op. Queued refreshes sat
untouched while worker_status reported idle. Restarting the API was the
only recovery.

Mark failed jobs failed and raise immediately from wait_for_job. Catch
unexpected cancellation and reschedule unless the app is actually
stopping. Track the perpetual-loop owner PID and retake the claim when
that process is gone; every Sanic worker still runs the file-worker
pump. Harden transfer_queue against the cross-process Empty race, and
stop request processes stealing from the shared public queue.
This commit is contained in:
Roland 2026-08-19 15:25:11 -06:00
parent 2b7bc4533f
commit 85f22b61ca
No known key found for this signature in database
GPG key ID: 1BA620CE6C9A4680
10 changed files with 723 additions and 52 deletions

View file

@ -318,7 +318,8 @@ async def start_single_tasks(app: Sanic):
@perpetual_signal(sleep=1)
async def perpetual_file_worker_queue():
file_worker.transfer_queue()
await file_worker.process_queue()
processed = await file_worker.process_queue()
file_worker.check_queue_stall(processed=processed)
@perpetual_signal(sleep=1)