A directory rename is finished by the file worker, not the request (#950)

* A directory rename is finished by the file worker, not the request

A directory rename queued a move, waited for it, then removed the old
directory and updated nested Collections itself.  A rename that outlasted
Sanic's RESPONSE_TIMEOUT (60 s) was cancelled while waiting: the move still
finished, but the old directory and its emptied subdirectories were left
behind, nested Collections kept their old directories, and the browser was
told the rename failed.

The move job now finishes the rename (nested Collections, then removing the
old directory, empty directories only), and the request returns once the
move is queued.  A failure is reported by the job's event.  An empty
directory can be renamed (it raised before).  The worker status carries
`rename_from`, so the Files page progress bar labels the rename, and the
file browser fetches again when the file worker is done.

A directory rename no longer queues a refresh of its parent: the file
worker runs the newest task first, so it scanned the directory before the
rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A queued rename returns its job, and the file browser waits for it

Review fixes:

- The cancellation test patched `wait_for_job`, which this path no longer
  calls, so it cancelled nothing.  It now asserts the invariant: the
  request only queues the rename (it never waits, and nothing has moved
  when it returns), and the file worker alone finishes it.

- The file browser inferred the end of a rename from the file worker's
  polled status.  A short job can start and finish between two polls, and
  the first idle poll (before the job started) spent the "fetch once"
  token, so the table could keep the old name.  POST /api/files/rename now
  returns `{"job_id": ...}` (null when the rename is already done), GET
  /api/files/jobs/<job_id> reports pending/complete/failed, and the file
  browser polls that job before fetching.

- useBrowseFiles ignores a fetch that settles after a newer one, so a
  listing requested before a rename cannot replace the one after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
lrnselfreliance 2026-11-07 13:09:33 -07:00 • committed by GitHub
parent c7bafaa7fb
commit aff5f1eb1f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 532 additions and 96 deletions

View file

@ -108,22 +108,29 @@ import {
selectRow('fb-alpha');
footerButton('Rename').click();
drainQueuedRefresh(() => {
// A directory rename is a file-worker move job; the request returns once it is queued, and the job
// reports when it is done. No refresh is queued.
cy.serverNow().then((since) => {
dialog().within(() => {
cy.get('input').clear().type('fb-alpha-renamed');
cy.contains('button', 'Rename').click();
});
// A directory rename runs as a file-worker move job; the modal stays open until
// the job finishes, which takes longer than the default timeout.
cy.get('[role="dialog"]', {timeout: 30000}).should('not.exist');
row('fb-alpha-renamed').should('be.visible');
// The folder row's text is just the name, so anchoring excludes 'fb-alpha-renamed'.
cy.contains('tr', /fb-alpha$/).should('not.exist');
cy.task('fb:readdir', 'fb-alpha-renamed')
.should('deep.equal', ['fb-one.txt', 'fb-two.txt']);
cy.task('fb:exists', 'fb-alpha').should('be.false');
dialog().should('not.exist');
cy.waitForEvent({
since,
event: 'file_move_completed',
message: 'Renamed fb-alpha to fb-alpha-renamed',
failureEvent: 'file_move_failed',
});
});
// The table is fetched again once the file worker is idle.
cy.contains('tr', 'fb-alpha-renamed', {timeout: 10000}).should('be.visible');
// The folder row's text is just the name, so anchoring excludes 'fb-alpha-renamed'.
cy.contains('tr', /fb-alpha$/).should('not.exist');
cy.task('fb:readdir', 'fb-alpha-renamed')
.should('deep.equal', ['fb-one.txt', 'fb-two.txt']);
cy.task('fb:exists', 'fb-alpha').should('be.false');
});
});

View file

@ -1927,13 +1927,31 @@ export async function searchDirectories(path) {
}
}
/**
* Returns the file worker job id of a queued rename (a directory), or null when the rename is already done.
*/
export async function renamePath(path, newName) {
const body = {path, new_name: newName};
const response = await apiPost(`${API_URI}/files/rename`, body);
if (!response.ok) {
const message = await getErrorMessage(response, 'Failed to rename!');
toast({type: 'error', title: 'Error', description: message, time: 5000});
return null;
}
const json = await response.json();
return json['job_id'] || null;
}
/**
* Returns a file worker job's status ('pending', 'complete', or 'failed'), or null if it is unknown.
*/
export async function fetchFileWorkerJobStatus(jobId) {
const response = await apiGet(`${API_URI}/files/jobs/${encodeURIComponent(jobId)}`);
if (!response.ok) {
return null;
}
const json = await response.json();
return json['status'];
}
export async function movePaths(destination, paths) {

View file

@ -23,7 +23,15 @@ import {
import React, {useContext, useEffect, useState} from "react";
import {IconArrowsMove, IconChevronsUp, IconCursorText, IconTags} from "@tabler/icons-react";
import {useHotkeys} from "react-hotkeys-hook";
import {deleteFile, ignoreDirectory, makeDirectory, movePaths, renamePath, unignoreDirectory} from "../api";
import {
deleteFile,
fetchFileWorkerJobStatus,
ignoreDirectory,
makeDirectory,
movePaths,
renamePath,
unignoreDirectory
} from "../api";
import _ from 'lodash';
import {SortableTable} from "./SortableTable";
import {useBrowseFiles, useElementWidth, useMediaDirectory, useWROLMode} from "../hooks/customHooks";
@ -34,6 +42,7 @@ import {BulkTagModal} from "./BulkTagModal";
import {InlineErrorBoundary} from "./ErrorBoundary";
import {TaggedDeleteConfirmModal} from "./TaggedDeleteConfirmModal";
import {UploadModal} from "./Upload";
import {useFileWorkerStatus} from "../contexts/FileWorkerStatusContext";
function depthIndentation(path) {
// Repeated spaces for every folder a path is in.
@ -228,6 +237,8 @@ function SkeletonFileRow({depth}) {
// Below this footer width (px) the action buttons are icon-only; at or above it they expand to show labels.
// The eight labeled buttons need ~1020px to fit on a single row; this leaves a small buffer.
const FOOTER_LABELS_MIN_WIDTH = 1050;
// How often a queued rename's job is checked.
const RENAME_JOB_POLL_MS = 500;
export function FileBrowser() {
const [selectedPaths, setSelectedPaths] = React.useState([]);
@ -328,6 +339,46 @@ export function FileBrowser() {
}
}, [browseFiles]);
// A move finishes in the file worker after its request returns (the Files page shows its progress), so fetch
// the files again when the file worker is done.
const {status: fileWorkerStatus, setFastPolling} = useFileWorkerStatus();
const fileWorkerBusy = !!fileWorkerStatus && fileWorkerStatus.status !== 'idle';
const fileWorkerWasBusy = React.useRef(false);
useEffect(() => {
if (fileWorkerWasBusy.current && !fileWorkerBusy) {
fetchFiles();
}
fileWorkerWasBusy.current = fileWorkerBusy;
}, [fileWorkerBusy]);
// A queued rename (a directory) returns its job; wait for that job, rather than inferring it from the file
// worker's status (a short job can start and finish between two polls).
const mountedRef = React.useRef(true);
useEffect(() => () => {
mountedRef.current = false;
}, []);
const afterRename = async (jobId) => {
if (!jobId) {
await reset();
return;
}
setSelectedPaths([]);
setFastPolling(true); // Update the progress bar while the rename runs.
try {
while (mountedRef.current) {
await new Promise(resolve => setTimeout(resolve, RENAME_JOB_POLL_MS));
if (await fetchFileWorkerJobStatus(jobId) !== 'pending') {
break;
}
}
} finally {
setFastPolling(false);
}
if (mountedRef.current) {
await reset();
}
}
// Clear loading states when file data updates
useEffect(() => {
if (loadingFolders.size > 0) {
@ -413,7 +464,7 @@ export function FileBrowser() {
onClose={() => setRenameOpen(false)}
onPending={setPending}
path={selectedPaths[0]}
onSubmit={reset}
onSubmit={afterRename}
/> : null}
<Button icon={IconArrowsMove}
color='teal'
@ -619,15 +670,16 @@ export function RenameModal({open, onClose, path, onSubmit, onPending}) {
if (e) {
e.preventDefault();
}
let jobId = null;
try {
console.log(`Renaming: ${path} to ${value}`);
onPending(true);
await renamePath(path, value);
jobId = await renamePath(path, value);
} finally {
onClose();
onPending(false);
if (onSubmit) {
await onSubmit();
await onSubmit(jobId);
}
}
}

View file

@ -6,10 +6,13 @@ import {FileBrowser, filterBrowseFiles} from './FileBrowser';
import {renderWithProviders} from '../test-utils';
import {SettingsContext} from '../contexts/contexts';
import {FilePreviewContext} from './FilePreview';
import {FileWorkerStatusContext} from '../contexts/FileWorkerStatusContext';
import {fileWorkerStatusFixture} from '../test-fixtures';
// Mock the API functions
jest.mock('../api', () => ({
deleteFile: jest.fn(),
fetchFileWorkerJobStatus: jest.fn(),
ignoreDirectory: jest.fn(),
makeDirectory: jest.fn(),
movePaths: jest.fn(),
@ -59,8 +62,13 @@ jest.mock('react-dropzone', () => ({
// Import mocked hooks for manipulation
import {useBrowseFiles, useElementWidth, useWROLMode} from '../hooks/customHooks';
// Set by the wrapper so a test can change the file worker's status, as a poll would.
let setFileWorkerStatus;
// Create wrapper with all required contexts
const AllContextsWrapper = ({children}) => {
const [fileWorkerStatus, setStatus] = React.useState({status: 'idle'});
setFileWorkerStatus = setStatus;
const settingsValue = {
settings: {ignored_directories: []},
fetchSettings: jest.fn(),
@ -71,11 +79,13 @@ const AllContextsWrapper = ({children}) => {
};
return (
<SettingsContext.Provider value={settingsValue}>
<FilePreviewContext.Provider value={filePreviewValue}>
{children}
</FilePreviewContext.Provider>
</SettingsContext.Provider>
<FileWorkerStatusContext.Provider value={fileWorkerStatusFixture({status: fileWorkerStatus})}>
<SettingsContext.Provider value={settingsValue}>
<FilePreviewContext.Provider value={filePreviewValue}>
{children}
</FilePreviewContext.Provider>
</SettingsContext.Provider>
</FileWorkerStatusContext.Provider>
);
};
@ -399,6 +409,77 @@ describe('FileBrowser', () => {
});
});
describe('File worker changes', () => {
it('fetches the files again when the file worker finishes', async () => {
const mockFetchFiles = jest.fn();
useBrowseFiles.mockReturnValue({
browseFiles: mockBrowseFiles,
openFolders: [],
setOpenFolders: jest.fn(),
fetchFiles: mockFetchFiles,
});
renderFileBrowser(<FileBrowser/>);
await act(async () => setFileWorkerStatus({status: 'moving', rename_from: '/media/wrolpi/testdir'}));
expect(mockFetchFiles).not.toHaveBeenCalled();
await act(async () => setFileWorkerStatus({status: 'idle'}));
expect(mockFetchFiles).toHaveBeenCalledTimes(1);
});
const renameTestdir = async () => {
// Select testdir/ (the second checkbox) and rename it.
await act(async () => fireEvent.click(screen.getAllByRole('checkbox')[1]));
const renameButton = document.querySelector('.tabler-icon-cursor-text').closest('button');
await act(async () => fireEvent.click(renameButton));
const nameInput = await screen.findByDisplayValue(/^testdir/);
await act(async () => fireEvent.change(nameInput, {target: {value: 'newdir'}}));
await act(async () => fireEvent.click(screen.getByRole('button', {name: 'Rename'})));
};
it('fetches the files once a queued rename job is done, even if no poll saw it running', async () => {
const mockFetchFiles = jest.fn();
const api = require('../api');
api.renamePath.mockResolvedValue('move-1234');
// The job is still queued at the first check, and done at the second.
api.fetchFileWorkerJobStatus.mockResolvedValueOnce('pending').mockResolvedValue('complete');
useBrowseFiles.mockReturnValue({
browseFiles: mockBrowseFiles,
openFolders: [],
setOpenFolders: jest.fn(),
fetchFiles: mockFetchFiles,
});
renderFileBrowser(<FileBrowser/>);
await renameTestdir();
expect(api.renamePath).toHaveBeenCalledWith('testdir/', 'newdir');
// Nothing is fetched while the rename is only queued.
expect(mockFetchFiles).not.toHaveBeenCalled();
// The file worker's status never left idle; the job is what says the rename is done.
await waitFor(() => expect(mockFetchFiles).toHaveBeenCalledTimes(1), {timeout: 3000});
expect(api.fetchFileWorkerJobStatus).toHaveBeenCalledWith('move-1234');
expect(api.fetchFileWorkerJobStatus).toHaveBeenCalledTimes(2);
});
it('fetches the files right away after a rename that was not queued', async () => {
const mockFetchFiles = jest.fn();
const api = require('../api');
api.renamePath.mockResolvedValue(null);
useBrowseFiles.mockReturnValue({
browseFiles: mockBrowseFiles,
openFolders: [],
setOpenFolders: jest.fn(),
fetchFiles: mockFetchFiles,
});
renderFileBrowser(<FileBrowser/>);
await renameTestdir();
expect(mockFetchFiles).toHaveBeenCalledTimes(1);
expect(api.fetchFileWorkerJobStatus).not.toHaveBeenCalled();
});
});
describe('Delete functionality', () => {
it('removes deleted directory from openFolders', async () => {
const mockSetOpenFolders = jest.fn();

View file

@ -915,7 +915,12 @@ export function FilesSearchView({
</>
}
function getPhaseLabel(status) {
function getPhaseLabel(status, renameFrom) {
if (renameFrom && (status === 'planning' || status === 'moving')) {
// A directory rename is a move of the directory's contents.
const name = renameFrom.split('/').filter(Boolean).pop();
return `Rename ${name}: ${status === 'planning' ? 'Planning' : 'Moving'}`;
}
switch (status) {
case 'counting':
return 'Counting';
@ -1037,8 +1042,8 @@ export function RefreshSteps({progress}) {
</div>
}
function RefreshProgressBar({status, operation_total, operation_processed, operation_percent, error}) {
const phaseLabel = getPhaseLabel(status);
function RefreshProgressBar({status, operation_total, operation_processed, operation_percent, error, rename_from}) {
const phaseLabel = getPhaseLabel(status, rename_from);
if (!phaseLabel) {
return null;
@ -1064,7 +1069,7 @@ export function FilesRefreshProgress() {
return null;
}
const {status, operation_total, operation_processed, operation_percent, error} = progress;
const {status, operation_total, operation_processed, operation_percent, error, rename_from} = progress;
// Show Steps component AND progress bar for global refresh operations
if (isGlobalRefresh(progress)) {
@ -1080,13 +1085,14 @@ export function FilesRefreshProgress() {
</>;
}
// Show just progress bar for other operations (moves, reorganize, targeted refresh)
// Show just progress bar for other operations (moves, renames, reorganize, targeted refresh)
return <RefreshProgressBar
status={status}
operation_total={operation_total}
operation_processed={operation_processed}
operation_percent={operation_percent}
error={error}
rename_from={rename_from}
/>;
}

View file

@ -0,0 +1,44 @@
import React from 'react';
import {screen} from '@testing-library/react';
import {FilesRefreshProgress} from './Files';
import {renderWithProviders} from '../test-utils';
import {fileWorkerStatusFixture} from '../test-fixtures';
import {FileWorkerStatusContext} from '../contexts/FileWorkerStatusContext';
const renderProgress = (status) => renderWithProviders(
<FileWorkerStatusContext.Provider value={fileWorkerStatusFixture({status})}>
<FilesRefreshProgress/>
</FileWorkerStatusContext.Provider>
);
const moving = {
status: 'moving',
task_type: 'move',
destination: '/media/wrolpi/3d models/Tools',
operation_total: 109,
operation_processed: 40,
operation_percent: 36,
error: null,
};
describe('FilesRefreshProgress', () => {
it('labels a move', () => {
renderProgress({...moving, rename_from: null});
expect(screen.getByText('Move: Moving (40 / 109)')).toBeInTheDocument();
});
it('labels a directory rename with the renamed directory', () => {
renderProgress({...moving, rename_from: '/media/wrolpi/3d models/tools'});
expect(screen.getByText('Rename tools: Moving (40 / 109)')).toBeInTheDocument();
});
it('labels a directory rename that is planning', () => {
renderProgress({...moving, status: 'planning', operation_total: 0, rename_from: '/media/wrolpi/3d models/tools'});
expect(screen.getByText('Rename tools: Planning')).toBeInTheDocument();
});
it('shows nothing while idle', () => {
renderProgress({...moving, status: 'idle'});
expect(screen.queryByText(/Move|Rename/)).not.toBeInTheDocument();
});
});

View file

@ -1083,14 +1083,22 @@ export const useSearchFiles = (defaultLimit = 48, emptySearch = false, model) =>
export const useBrowseFiles = () => {
const [browseFiles, setBrowseFiles] = useState(null);
const [openFolders, setOpenFolders] = useAllQuery('folders');
// Only the latest fetch may set the files; an older one that settles late would show files that have since
// been moved or renamed.
const latestFetchRef = React.useRef(0);
const fetchFiles = async () => {
const fetchId = ++latestFetchRef.current;
try {
const files = await getFiles(openFolders);
setBrowseFiles(files);
if (fetchId === latestFetchRef.current) {
setBrowseFiles(files);
}
} catch (e) {
console.error(e);
setBrowseFiles(undefined); // Display error.
if (fetchId === latestFetchRef.current) {
setBrowseFiles(undefined); // Display error.
}
}
}

View file

@ -2,11 +2,11 @@ import React from 'react';
import {act, renderHook} from '@testing-library/react';
import {render, screen} from '../test-utils';
import {
usePages, useDriveTemperature, useDriveHealth, useSearchChannels, useSearchRecentFiles, useSettings,
useStatistics, useStatus, useVideoExtras,
useBrowseFiles, usePages, useDriveTemperature, useDriveHealth, useSearchChannels, useSearchRecentFiles,
useSettings, useStatistics, useStatus, useVideoExtras,
} from './customHooks';
import {
ApiDownError, filesSearch, getSettings, getStatistics, getStatus, getVideoComments, getVideoDescription,
ApiDownError, filesSearch, getFiles, getSettings, getStatistics, getStatus, getVideoComments, getVideoDescription,
searchChannels,
} from '../api';
import {getControllerStats} from '../api/controller';
@ -50,6 +50,7 @@ jest.mock('../api', () => ({
getStatus: jest.fn(),
getSettings: jest.fn(),
filesSearch: jest.fn(),
getFiles: jest.fn(),
}));
jest.mock('../api/controller', () => ({
@ -736,3 +737,35 @@ describe('useSearchRecentFiles', () => {
expect(result.current.searchFiles).toEqual([]);
});
});
describe('useBrowseFiles', () => {
it('ignores a fetch that settles after a newer one', async () => {
// The mount's fetch, then a fetch from before a rename, then one from after it.
let resolveBeforeRename;
getFiles
.mockResolvedValueOnce([{path: 'mounted/'}])
.mockReturnValueOnce(new Promise(resolve => {
resolveBeforeRename = resolve;
}))
.mockResolvedValueOnce([{path: 'renamed/'}]);
const {result} = renderHook(() => useBrowseFiles(), {wrapper: makeWrapper()});
await act(async () => {});
expect(result.current.browseFiles).toEqual([{path: 'mounted/'}]);
let beforeRename;
await act(async () => {
beforeRename = result.current.fetchFiles();
await result.current.fetchFiles();
});
expect(result.current.browseFiles).toEqual([{path: 'renamed/'}]);
// The older fetch settles last; the newer files stay.
await act(async () => {
resolveBeforeRename([{path: 'old-name/'}]);
await beforeRename;
});
expect(result.current.browseFiles).toEqual([{path: 'renamed/'}]);
});
});

View file

@ -306,7 +306,7 @@ async def test_rename_file_updates_playlists_config(
request, response = await async_client.post(
'/api/files/rename', json=dict(path='guide.pdf', new_name='manual.pdf'))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
await await_switches()
await _assert_playlist_survives_reimport(test_session, playlists_config, 'manual.pdf')

View file

@ -229,6 +229,7 @@ def reset_shared_contexts(app: Sanic):
task_type=None,
paths=[],
destination=None,
rename_from=None,
error=None,
operation_total=0,
operation_processed=0,

View file

@ -360,21 +360,38 @@ async def post_rename(request: Request, body: schema.Rename):
elif new_path.exists():
raise FileConflict(f'File already exists: {new_path}')
is_directory = path.is_dir()
try:
await lib.rename(request.ctx.session, path, body.new_name, send_events=True, write=request.ctx.db.write)
job_id = await lib.rename(request.ctx.session, path, body.new_name, send_events=True,
write=request.ctx.db.write)
except Exception as e:
logger.error(f'Failed to rename {path} to {new_path}', exc_info=e)
cause = str(e).split('\n', 1)[0]
raise FileConflict(f'Failed to rename {path} to {new_path}: {cause}') from e
# Refresh the parent directory instead of individual files.
# This ensures all related files (e.g., subtitles, posters) are discovered together.
# Refreshing individual files causes the FileGroup's files list to be overwritten
# with only the specified file, losing track of associated files.
parent_dir = new_path.parent
file_worker.queue_refresh([parent_dir])
if not is_directory:
# Refresh the parent directory instead of individual files.
# This ensures all related files (e.g., subtitles, posters) are discovered together.
# Refreshing individual files causes the FileGroup's files list to be overwritten
# with only the specified file, losing track of associated files.
# A directory's move keeps its FileGroups up to date, and may still be queued; the file worker runs the
# newest task first, so a refresh queued now would scan the directory before it is renamed.
file_worker.queue_refresh([new_path.parent])
return response.empty(HTTPStatus.NO_CONTENT)
# A queued rename's `job_id` can be polled (`/api/files/jobs/<job_id>`) to know when it is done.
return json_response(dict(job_id=job_id))
@files_bp.get('/jobs/<job_id:str>')
@openapi.definition(
summary='Get the status of a queued file worker job: pending, complete, or failed (with its error).',
)
async def get_file_worker_job(_: Request, job_id: str):
try:
status = file_worker.get_job_status(job_id)
except RuntimeError:
return json_response(dict(error=f'Unknown job: {job_id}'), HTTPStatus.NOT_FOUND)
return json_response(dict(job_id=job_id, status=status, error=file_worker.get_job_error(job_id)))
@files_bp.post('/tag')

View file

@ -2096,9 +2096,12 @@ async def rename_file(path: pathlib.Path, new_name: str) -> pathlib.Path:
async def rename_directory(session: Session, directory: pathlib.Path, new_name: str, send_events: bool = False,
write: Callable[[], ContextManager[Session]] = None) -> pathlib.Path:
write: Callable[[], ContextManager[Session]] = None) -> str | None:
"""Rename a directory. This is done by moving all files into the new directory, and removing the old directory.
A Collection's directory is renamed before this returns (returns None). Any other directory is renamed by a queued
file worker move, which this does not wait for; returns the move's job id.
`session` is used to read. Collections are changed in `write()` (`request.ctx.db.write` in a request), which
begins as a writer; the files are moved outside it. Without `write`, Collections are changed in `session`."""
from wrolpi.files.worker import file_worker
@ -2120,45 +2123,25 @@ async def rename_directory(session: Session, directory: pathlib.Path, new_name:
# for the move.
with write() as session:
await collection.move_collection(new_directory, session, send_events=send_events)
else:
# Regular directory rename - move all paths into the new directory.
paths = list(directory.iterdir())
try:
job_id = file_worker.queue_move(new_directory, paths)
await file_worker.wait_for_job(job_id)
if send_events:
Events.send_file_move_completed(f'Directory has been renamed: {directory}')
except Exception:
if send_events:
Events.send_file_move_failed(f'Directory rename has failed: {directory}')
raise
return None
# Update nested Collections (files already moved by file_worker above). Read them in the write: the move
# took a while, and a transaction begun before it cannot write after another connection has.
directory_str = str(directory)
with write() as session:
nested_collections = session.query(Collection).filter(
Collection.directory.like(f'{directory_str}/%')
).all()
for coll in nested_collections:
relative = coll.directory.relative_to(directory)
new_coll_dir = new_directory / relative
await coll.move_collection(new_coll_dir, session, send_events=send_events, with_files=False)
# Remove the old directory (recursive=True to clean up any empty subdirectories
# that remained after files were moved out).
delete_directory(directory, recursive=True)
return new_directory
# Regular directory rename: the file worker moves the contents, updates nested Collections, and removes the old
# directory. This does not wait; a large rename outlasts a request, and the job reports its own progress and
# result.
return file_worker.queue_move(new_directory, list(directory.iterdir()), rename_from=directory)
async def rename(session: Session, path: pathlib.Path, new_name: str, send_events: bool = False,
write: Callable[[], ContextManager[Session]] = None) -> pathlib.Path:
"""Rename a directory or file. Preserve any tags."""
write: Callable[[], ContextManager[Session]] = None) -> str | None:
"""Rename a directory or file. Preserve any tags.
Returns the file worker job id when the rename was queued (see `rename_directory`), otherwise None; the rename is
done."""
if path.is_dir():
return await rename_directory(session, path, new_name, send_events=send_events, write=write)
return await rename_file(path, new_name)
await rename_file(path, new_name)
return None
def get_special_directories() -> List[pathlib.Path]:

View file

@ -22,7 +22,7 @@ from wrolpi.db import get_db_session, get_db_curs
from wrolpi.events import Events
from wrolpi.files.lib import (
split_path_stem_and_suffix, get_unique_files_by_stem, glob_shared_stem,
group_files_by_stem, delete_directory,
group_files_by_stem, delete_directory, directory_or_descendant,
_move_file_group_files, _bulk_update_file_groups_db, MOVE_CHUNK_SIZE,
get_normalized_ignored_directories, remove_files_in_ignored_directories,
)
@ -446,6 +446,37 @@ class FileMoveMixin:
curs.execute('DELETE FROM directory WHERE path IN (SELECT value FROM json_each(:paths))',
{'paths': json.dumps(deleted_dirs_for_db)})
async def _finish_directory_rename(self, old_directory: pathlib.Path, new_directory: pathlib.Path,
send_events: bool = True) -> None:
"""Finish a directory rename after its contents were moved: point nested Collections at the new directory,
and remove the old directory.
Only empty directories are removed, so anything added to the old directory during the move is kept."""
# Avoid a circular import; collections.models imports the file worker.
from wrolpi.collections.models import Collection
with get_db_session(commit=True) as session:
nested_collections = session.query(Collection).filter(
directory_or_descendant(Collection.directory, old_directory),
Collection.directory != str(old_directory),
).all()
for collection in nested_collections:
new_collection_directory = new_directory / collection.directory.relative_to(old_directory)
# The files were already moved, so this only updates the Collection and its config.
await collection.move_collection(new_collection_directory, session, send_events=send_events,
with_files=False)
if not old_directory.is_dir():
return
# Deepest first, so a directory is empty once its emptied children are removed.
for dirpath, _, _ in os.walk(old_directory, topdown=False):
try:
delete_directory(pathlib.Path(dirpath))
except OSError:
pass # Not empty; something was added during the move.
if old_directory.is_dir():
logger.warning(f'Renamed directory was not empty after its contents were moved: {old_directory}')
def _revert_move(
self,
revert_plan: Dict[pathlib.Path, pathlib.Path],
@ -501,8 +532,10 @@ class FileMoveMixin:
destination = task.destination
sources = task.paths
rename_from = task.rename_from
if not destination or not sources:
# An empty directory can be renamed; there is nothing to move.
if not destination or not (sources or rename_from):
logger.error('Invalid move task: missing destination or sources')
self._fail_job(task.job_id, 'Invalid move task: missing destination or sources')
self.reset_status()
@ -510,7 +543,8 @@ class FileMoveMixin:
media_directory = get_media_directory()
if not self._validate_move_paths(sources, destination, media_directory):
if not self._validate_move_paths(sources + ([rename_from] if rename_from else []), destination,
media_directory):
self._fail_job(task.job_id, 'Move paths are not within the media directory')
return
@ -522,6 +556,7 @@ class FileMoveMixin:
task_type='move',
paths=[str(p) for p in sources],
destination=str(destination),
rename_from=str(rename_from) if rename_from else None,
operation_total=planning_total,
operation_processed=0,
operation_percent=0,
@ -561,12 +596,19 @@ class FileMoveMixin:
await self._execute_move_chunks(
plan, session, created_directories, revert_plan
)
# The move committed; the files and the DB agree, so a later failure must not move the files back.
revert_plan.clear()
self._cleanup_old_directories(sources, old_directories)
if rename_from:
await self._finish_directory_rename(rename_from, destination, send_events=task.send_events)
await self._apply_post_processing()
logger.info(f'Move completed: {len(plan)} items moved to {destination}')
Events.send_file_move_completed(f'Moved {len(sources)} items to {destination}')
if rename_from:
Events.send_file_move_completed(f'Renamed {rename_from.name} to {destination.name}')
else:
Events.send_file_move_completed(f'Moved {len(sources)} items to {destination}')
self._complete_job(task.job_id)
except asyncio.CancelledError:
@ -583,6 +625,9 @@ class FileMoveMixin:
self.update_status(status='reverting', error=str(e))
self._revert_move(revert_plan, created_directories, destination, destination_existed)
self._fail_job(task.job_id, e)
Events.send_file_move_failed(f'Move to {destination} failed: {e}')
if rename_from:
Events.send_file_move_failed(f'Renaming {rename_from.name} to {destination.name} failed: {e}')
else:
Events.send_file_move_failed(f'Move to {destination} failed: {e}')
finally:
self.reset_status()

View file

@ -1085,7 +1085,7 @@ async def test_directory_crud(test_session, async_client, test_directory, assert
# Can rename the directory.
content = dict(path='foo/bar', new_name='baz')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
assert not (test_directory / 'foo/bar').exists()
assert (test_directory / 'foo/baz').exists()
assert (test_directory / 'foo/baz/asdf.txt').exists()
@ -1172,7 +1172,7 @@ async def test_rename_file(test_session, test_directory, make_files_structure, a
# mv foo/bar/baz.txt foo/bar/qux.txt
content = dict(path='foo/bar/baz.txt', new_name='qux.txt')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
assert not foo_file.exists()
assert (test_directory / 'foo/bar/qux.txt').is_file()
assert (test_directory / 'foo/bar/qux.txt').read_text() == 'asdf'
@ -1203,7 +1203,7 @@ async def test_rename_file_with_associated_files_via_api(test_session, test_dire
# First rename via API: "example.mp4" -> "example 2.mp4"
content = dict(path='example.mp4', new_name='example 2.mp4')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
# Verify first rename - both files should be renamed
new_video_1 = test_directory / 'example 2.mp4'
@ -1224,7 +1224,7 @@ async def test_rename_file_with_associated_files_via_api(test_session, test_dire
# Second rename via API: "example 2.mp4" -> "example.mp4"
content = dict(path='example 2.mp4', new_name='example.mp4')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
# Verify second rename - BOTH files should be renamed back
final_video = test_directory / 'example.mp4'
@ -1255,7 +1255,7 @@ async def test_rename_directory(test_session, test_directory, make_files_structu
# mv foo/bar foo/qux
content = dict(path='foo/bar', new_name='qux')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
assert not (test_directory / 'foo/bar').exists()
assert (test_directory / 'foo/qux/baz.txt').is_file()
assert (test_directory / 'foo/qux/baz.txt').read_text() == 'asdf'
@ -1263,11 +1263,138 @@ async def test_rename_directory(test_session, test_directory, make_files_structu
assert foo.name == 'foo' and qux.name == 'qux'
@pytest.mark.asyncio
async def test_rename_directory_does_not_depend_on_the_request(test_session, test_directory, make_files_structure,
async_client, refresh_files, await_background_tasks):
"""A directory rename is finished by the file worker; the request only queues it.
A large rename outlasts Sanic's RESPONSE_TIMEOUT. When the request also removed the old directory after the move,
a timed-out request left the old directory and its emptied subdirectories behind."""
make_files_structure({
'models/tools/trowel.stl': 'trowel',
'models/tools/tweezers/tweezers.stl': 'tweezers',
'models/tools/push block/push block.stl': 'push block',
})
await refresh_files()
content = dict(path='models/tools', new_name='hand tools')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.OK
await await_background_tasks()
assert (test_directory / 'models/hand tools/trowel.stl').read_text() == 'trowel'
assert (test_directory / 'models/hand tools/tweezers/tweezers.stl').read_text() == 'tweezers'
assert (test_directory / 'models/hand tools/push block/push block.stl').read_text() == 'push block'
left = sorted(str(i.relative_to(test_directory)) for i in (test_directory / 'models').iterdir())
assert left == ['models/hand tools'], 'the old directory (and its emptied subdirectories) should be gone'
directories = {d.path for d in test_session.query(Directory)}
assert not {str(test_directory / 'models/tools'), str(test_directory / 'models/tools/tweezers')} & directories
assert {fg.primary_path for fg in test_session.query(FileGroup)} == {
test_directory / 'models/hand tools/trowel.stl',
test_directory / 'models/hand tools/tweezers/tweezers.stl',
test_directory / 'models/hand tools/push block/push block.stl',
}
@pytest.mark.asyncio
async def test_rename_directory_is_finished_by_the_file_worker(test_session, test_directory, make_files_structure,
refresh_files, await_background_tasks):
"""The request only queues a directory rename; the file worker does all of it.
A large rename outlasts Sanic's RESPONSE_TIMEOUT (60 s), which cancels the request. When the request waited for
the move and then removed the old directory itself, a cancelled request left the old directory and its emptied
subdirectories behind. So nothing may happen in the request after the move is queued."""
from wrolpi.files import lib
from wrolpi.files.worker import file_worker
make_files_structure({
'models/tools/trowel.stl': 'trowel',
'models/tools/tweezers/tweezers.stl': 'tweezers',
})
await refresh_files()
with mock.patch.object(file_worker, 'wait_for_job', side_effect=AssertionError('the request must not wait')):
job_id = await lib.rename(test_session, test_directory / 'models/tools', 'hand tools')
# The request is done (or cancelled); nothing has been moved or removed yet.
assert file_worker.get_job_status(job_id) == 'pending'
assert (test_directory / 'models/tools/trowel.stl').read_text() == 'trowel'
assert (test_directory / 'models/tools/tweezers/tweezers.stl').read_text() == 'tweezers'
assert not (test_directory / 'models/hand tools').exists()
# The file worker alone finishes the rename.
await await_background_tasks()
assert file_worker.get_job_status(job_id) == 'complete'
assert (test_directory / 'models/hand tools/tweezers/tweezers.stl').read_text() == 'tweezers'
left = sorted(str(i.relative_to(test_directory)) for i in (test_directory / 'models').iterdir())
assert left == ['models/hand tools'], 'the old directory (and its emptied subdirectories) should be gone'
@pytest.mark.asyncio
async def test_get_file_worker_job(test_session, test_directory, make_files_structure, async_client, refresh_files,
await_background_tasks):
"""A queued rename's job can be polled until it is done. (Its pending status is checked by
`test_rename_directory_is_finished_by_the_file_worker`; the test client runs queued jobs during requests.)"""
make_files_structure({'tools/trowel.stl': 'trowel'})
await refresh_files()
content = dict(path='tools', new_name='hand tools')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.OK
job_id = response.json['job_id']
assert job_id
await await_background_tasks()
request, response = await async_client.get(f'/api/files/jobs/{job_id}')
assert response.status_code == HTTPStatus.OK
assert response.json == dict(job_id=job_id, status='complete', error=None)
assert (test_directory / 'hand tools/trowel.stl').is_file()
# A file rename is done when the request returns; there is no job.
content = dict(path='hand tools/trowel.stl', new_name='spade.stl')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.OK
assert response.json == dict(job_id=None)
request, response = await async_client.get('/api/files/jobs/move-unknown')
assert response.status_code == HTTPStatus.NOT_FOUND
@pytest.mark.asyncio
async def test_rename_directory_keeps_what_was_added_during_the_move(test_session, test_directory, make_files_structure,
refresh_files, await_background_tasks):
"""The old directory is only removed when it is empty; a file added while the rename was queued is kept."""
from wrolpi.files import lib
make_files_structure({'models/tools/trowel.stl': 'trowel'})
await refresh_files()
await lib.rename(test_session, test_directory / 'models/tools', 'hand tools')
(test_directory / 'models/tools/late.stl').write_text('late')
await await_background_tasks()
assert (test_directory / 'models/hand tools/trowel.stl').read_text() == 'trowel'
assert (test_directory / 'models/tools/late.stl').read_text() == 'late'
@pytest.mark.asyncio
async def test_rename_empty_directory(test_session, test_directory, async_client, await_background_tasks):
(test_directory / 'empty').mkdir()
content = dict(path='empty', new_name='still empty')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.OK
await await_background_tasks()
assert not (test_directory / 'empty').exists()
assert (test_directory / 'still empty').is_dir()
@pytest.mark.asyncio
async def test_rename_directory_stale_destination_fails_promptly(
test_session, test_directory, make_files_structure, async_client, refresh_files
test_session, test_directory, make_files_structure, async_client, refresh_files, await_background_tasks,
events_fixture,
):
"""A directory rename that collides with a stale FileGroup must fail promptly.
"""A directory rename that collides with a stale FileGroup must fail promptly, and report the move's error.
Regression: the move's IntegrityError left the job 'pending', so
POST /api/files/rename hung until wait_for_job's 300s timeout.
@ -1280,7 +1407,8 @@ async def test_rename_directory_stale_destination_fails_promptly(
content = dict(path='dir-a', new_name='dir-b')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
await await_background_tasks()
# Remove dest on disk without refreshing; recreate the old source.
shutil.rmtree(test_directory / 'dir-b')
@ -1291,10 +1419,16 @@ async def test_rename_directory_stale_destination_fails_promptly(
async_client.post('/api/files/rename', content=json.dumps(content)),
timeout=5,
)
assert response.status_code == HTTPStatus.CONFLICT, response.json
error = (response.json or {}).get('error') or ''
# The recorded move error, not just "Failed to rename …".
assert 'UNIQUE' in error.upper() or 'constraint' in error.lower(), response.json
assert response.status_code == HTTPStatus.OK
await asyncio.wait_for(await_background_tasks(), timeout=5)
# The rename job reports the recorded move error, and the files are moved back.
failed = events_fixture.get_by_name('file_move_failed')
assert failed, events_fixture.events
error = failed[0]['message']
assert 'Renaming dir-a to dir-b failed' in error
assert 'UNIQUE' in error.upper() or 'constraint' in error.lower(), error
assert (test_directory / 'dir-a/one.txt').read_text() == 'hello'
@pytest.mark.asyncio
@ -1317,7 +1451,7 @@ async def test_rename_collection_directory(test_session, test_directory, make_fi
# Rename the directory
content = dict(path='my_collection', new_name='renamed_collection')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
# Verify Collection.directory was updated
collection = test_session.query(Collection).filter_by(id=collection_id).one()
@ -1361,7 +1495,7 @@ async def test_rename_directory_with_nested_collections(test_session, test_direc
# Rename the parent directory
content = dict(path='videos', new_name='my_videos')
request, response = await async_client.post('/api/files/rename', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert response.status_code == HTTPStatus.OK
# Verify nested Collection directories were updated
coll_a = test_session.query(Collection).filter_by(id=coll_a_id).one()
@ -1694,7 +1828,7 @@ async def test_rename_directory_with_collection_waits_for_a_concurrent_writer(
with write_lock_held_briefly(db_file, seconds=3):
_, response = await async_client.post('/api/files/rename',
content=json.dumps(dict(path=rename, new_name=new_name)))
assert response.status_code == HTTPStatus.NO_CONTENT, response.body
assert response.status_code == HTTPStatus.OK, response.body
assert (expected / 'notes.txt').is_file()
with maker() as session:

View file

@ -1363,7 +1363,7 @@ async def test_file_group_merge(async_client, test_session, test_directory, make
@pytest.mark.asyncio
async def test_move_directory(async_client, test_session, test_directory, make_files_structure,
assert_directories, refresh_files):
assert_directories, refresh_files, await_background_tasks):
"""A Directory record is deleted when it's directory is deleted."""
make_files_structure({
'foo/one.txt': 'one',
@ -1373,6 +1373,7 @@ async def test_move_directory(async_client, test_session, test_directory, make_f
bar = test_directory / 'bar'
await lib.rename(test_session, test_directory / 'foo', 'bar')
await await_background_tasks()
assert (bar / 'one.txt').read_text() == 'one'
assert_directories({'bar'})

View file

@ -84,6 +84,8 @@ class FileTask:
task_type: FileTaskType
paths: list[pathlib.Path]
destination: pathlib.Path = None # For move tasks
# For a directory rename's move: the renamed directory, which the move removes once its contents are moved.
rename_from: pathlib.Path = None
count: int = None
found_directories: set[pathlib.Path] = None # Directories found during count phase
unchanged_files: bool = None
@ -136,6 +138,7 @@ class FileWorker(FileMoveMixin, FileReorganizeMixin):
task_type=None,
paths=[],
destination=None,
rename_from=None,
error=None,
operation_total=0,
operation_processed=0,
@ -268,19 +271,22 @@ class FileWorker(FileMoveMixin, FileReorganizeMixin):
logger.info('Refreshing all files because no refresh has completed')
return self.queue_refresh([get_media_directory()])
def queue_move(self, destination: pathlib.Path, sources: list[pathlib.Path]) -> str:
def queue_move(self, destination: pathlib.Path, sources: list[pathlib.Path],
rename_from: pathlib.Path = None) -> str:
"""Queue a move task for background processing.
Args:
destination: Target directory (absolute path)
sources: List of files/directories to move (absolute paths)
rename_from: The directory being renamed to `destination` (`sources` are its contents). The move
finishes the whole rename, so it does not depend on the request that queued it.
Returns:
Job ID for tracking completion
"""
import uuid
job_id = f'move-{uuid.uuid4().hex[:8]}'
task = FileTask(FileTaskType.move, sources, destination=destination, job_id=job_id)
task = FileTask(FileTaskType.move, sources, destination=destination, job_id=job_id, rename_from=rename_from)
self._set_job_status(job_id, 'pending')
self.public_queue.put_nowait(task)
return job_id