mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-25 18:40:06 -04:00
files: a failed directory rename that hung for five minutes
A directory rename waits inline on a move job. When that move failed (stale FileGroup rows colliding on primary_path), handle_move reverted the files and sent a failure event but left the job pending. wait_for_job only understands 'complete', so POST /api/files/rename spun until its 300s timeout and the Rename modal never closed. The same sequence could kill file processing for the life of the process. perpetual_signal treated any CancelledError as terminal and logged nothing. Separately, perpetual_tasks_started is a shared Event that survives the worker that set it: docker auto-reloads and worker crashes left every later after_server_start as a no-op. Queued refreshes sat untouched while worker_status reported idle. Restarting the API was the only recovery. Mark failed jobs failed and raise immediately from wait_for_job. Catch unexpected cancellation and reschedule unless the app is actually stopping. Track the perpetual-loop owner PID and retake the claim when that process is gone; every Sanic worker still runs the file-worker pump. Harden transfer_queue against the cross-process Empty race, and stop request processes stealing from the shared public queue.
This commit is contained in:
parent
2b7bc4533f
commit
85f22b61ca
10 changed files with 723 additions and 52 deletions
3
main.py
3
main.py
|
|
@ -318,7 +318,8 @@ async def start_single_tasks(app: Sanic):
|
|||
@perpetual_signal(sleep=1)
|
||||
async def perpetual_file_worker_queue():
|
||||
file_worker.transfer_queue()
|
||||
await file_worker.process_queue()
|
||||
processed = await file_worker.process_queue()
|
||||
file_worker.check_queue_stall(processed=processed)
|
||||
|
||||
|
||||
@perpetual_signal(sleep=1)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue