mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-10-26 10:51:52 -04:00
947024914a
std::filesystem::path is wrong for mergerfs's hot path:
- `branch->path / fusepath_` heap-allocates per FUSE op (3 allocs each
on libstdc++; the SSO buffer is 15 bytes and any joined path exceeds it).
- The implicit `operator string_type()` adds another std::string when
passing to `fs::*` wrappers that take `const std::string&`.
- Per-branch loops rebuild fullpath from scratch every iteration.
The new fs::relpath is a fixed 8192-byte (2 * PATH_MAX) inline buffer,
NUL-terminated, never heap-allocates. The rel data is anchored at offset
REL_OFFSET = 4096; the 4096 bytes before the anchor form the "prefix
area" so a long base/prefix can be written there in place via
set_prefix() without disturbing the rel. The remaining ~4096 bytes
after the rel are available for trailing append (filenames in
readdir-style loops, etc.).
set_prefix's cost is O(prefix size) -- only the actually-used bytes
are touched -- so the buffer size is a stack-footprint question, not a
hot-path cost. FUSE worker threads use the pthread default stack
(~8 MB on Linux); a 4-relpath frame in fuse_link/fuse_rename is 32 KB,
~0.4% of the stack budget.
The per-branch hot loop:
fs::relpath fullpath = fusepath_; // one copy of the rel
for (auto branch : branches) {
fullpath.set_prefix(branch->path); // writes only [start, 4096)
fs::syscall(fullpath);
}
set_prefix only rewrites the prefix area; the rel stays anchored. A
cached _rel_size means no recomputation across iterations.
Microbenchmark (8 branches, 30-char fusepath, -O3, 4 M path builds):
variant ns/build allocs/build
-------------------------------------- -------- ------------
origin/master (std::filesystem::path) 158 3.00
fs::relpath assign_concat (single-shot) 9.1 0
fs::relpath set_prefix (loop-optimized) 4.5 0
35x faster, zero allocations in the hot path.
Contract: paths flow through fs::relpath in canonical form -- prefix
without trailing '/', rel without leading '/'. No sanitization;
violations yield double slashes. Mergerfs satisfies this by
construction:
- libfuse's try_get_path() walks leaf-to-root via prepend(), then
calls consolidate_to_rel(strip=true) which drops the leading '/'
that prepend produces.
- branch paths are realpath-canonicalized at config load so they
never have a trailing slash.
Mergerfs has exactly one path-manipulation type: a relative fusepath.
Branch paths and other absolute paths are stored as std::string -- they
are spliced into the prefix area of an fs::relpath at syscall time but
are never themselves mutated as path objects.
Major changes:
- vendored/libfuse/{include/fs_path.hpp,lib/fs_path.cpp}: new
fs::relpath class (8 KB inline, REL anchor at 4096, set_prefix /
clear_prefix / consolidate_to_rel, _rel_size cache, self-aliasing-
safe assign_concat, sticky ENAMETOOLONG via errnum()).
- 30+ FUSE-op call sites converted from `fullpath = branch / fusepath`
to the `set_prefix` loop pattern (fuse_unlink, fuse_chmod,
fuse_chown, fuse_listxattr, fuse_truncate, fuse_utimens,
fuse_setxattr, fuse_rmdir, fuse_removexattr, fuse_link's
preserve-path and create-path loops, fuse_rename's create-path
and preserve-path loops, fuse_statfs FULL mode, fs::findallfiles,
fs::findonfs).
- fs::* syscall wrappers that previously only took
`const std::string&` got `const char*` overloads so fs::relpath's
implicit `operator const char*()` lands cleanly (fs::link,
fs::lutimens, fs::umount_lazy, fs::open_dir_ro, fs::readlink,
fs::lremovexattr, fs::lstatvfs, fs::info, fs::statvfs_cache).
- fs::exists gained a (fs::relpath&, basepath, ...) overload that
calls set_prefix internally so per-branch policy fan-out loops
share a single fullpath, rewriting only the prefix area on each
iteration.
- All policies (epall, epff, eplfs, eplus, epmfs, eppfrd, ff, lup,
msplfs, msplus, mspmfs, msppfrd, newest) refactored to hoist the
fs::relpath fullpath construction out of the per-branch loop and
use the set_prefix-based fs::exists overload.
- fs::inode::ReaddirCalc::_fusepath migrated from std::string to
fs::relpath with the same resize/append API.
- std::filesystem removed entirely from the project: branches.cpp
and option_parser.cpp's exists/is_directory/equivalent calls now
use direct ::stat(); mergerfs_fsck.cpp's recursive_directory_iterator
was replaced with an opendir-based walk.
Tests (tests/tests.cpp): 27 new fs::relpath test cases covering
construction, set_prefix in/out of REL anchor, prefix()/rel()
accessors, clear_prefix, concat canonical form, self-aliasing
(`p /= "x"`), operator/=, resize zero-fill, swap, copy/move,
equality, implicit conversions, lexically_relative, remove_filename,
_rel_size invariant across every mutator, capacity bounds.
Full suite: 213/213 pass.
|
||
|---|---|---|
| .. | ||
| OPTIONS_TEST_PLAN.md | ||
| posix_parity.py | ||
| README.md | ||
| run-tests | ||
| TEST_api_xattr | ||
| TEST_cfg_link_rename_exdev | ||
| TEST_cfg_statfs_ignore | ||
| TEST_cfg_xattr_modes | ||
| TEST_io_passthrough_create_open | ||
| TEST_io_passthrough_open_race | ||
| TEST_mount_lifecycle | ||
| TEST_movefile_enospc | ||
| TEST_no_fuse_hidden | ||
| TEST_o_direct | ||
| TEST_open_after_unlink | ||
| TEST_policy_lup | ||
| TEST_posix_access | ||
| TEST_posix_bmap | ||
| TEST_posix_chmod | ||
| TEST_posix_chown | ||
| TEST_posix_copy_file_range | ||
| TEST_posix_create_mknod | ||
| TEST_posix_fsync_flush | ||
| TEST_posix_getattr_fgetattr | ||
| TEST_posix_ioctl | ||
| TEST_posix_link_symlink | ||
| TEST_posix_locking | ||
| TEST_posix_mkdir_rmdir | ||
| TEST_posix_open_read_write | ||
| TEST_posix_poll | ||
| TEST_posix_readdir | ||
| TEST_posix_readdir_plus | ||
| TEST_posix_release | ||
| TEST_posix_releasedir | ||
| TEST_posix_statfs | ||
| TEST_posix_statx | ||
| TEST_posix_syncfs | ||
| TEST_posix_tmpfile | ||
| TEST_posix_truncate_ftruncate | ||
| TEST_posix_unlink_rename | ||
| TEST_posix_utimens | ||
| TEST_posix_xattr | ||
| TEST_posix_xattr_matrix | ||
| TEST_readlink_semantics | ||
| TEST_rmdir_enotempty_priority | ||
| TEST_unlink_rename | ||
| TEST_use_fallocate_after_unlink | ||
| TEST_use_fchmod_after_unlink | ||
| TEST_use_fchown_after_unlink | ||
| TEST_use_fstat_after_unlink | ||
| TEST_use_ftruncate_after_unlink | ||
| TEST_use_futimens_after_unlink | ||
| tests.cpp | ||
POSIX Soft Conformance Suite
This directory contains integration tests that exercise mergerfs FUSE operations and compare behavior against equivalent native syscalls on a local filesystem.
The suite is "soft POSIX": it focuses on practical parity for return values,
errno, and key side effects, including common error paths.
Each test mounts its own ephemeral mergerfs instance with random temporary branches and cleans everything up automatically.
Run with:
python3 tests/run-tests
Coverage Matrix
access->TEST_posix_accesschmod/fchmod->TEST_posix_chmod,TEST_use_fchmod_after_unlinkchown/fchown->TEST_posix_chown,TEST_use_fchown_after_unlinkcreate/mknod->TEST_posix_create_mknodgetattr/fgetattr->TEST_posix_getattr_fgetattr,TEST_use_fstat_after_unlinkopen/read/write->TEST_posix_open_read_write,TEST_o_directflush/release/fsync/fsyncdir->TEST_posix_fsync_flushreleaseclose semantics ->TEST_posix_releasereleasedirclose semantics ->TEST_posix_releasedirtruncate/ftruncate->TEST_posix_truncate_ftruncate,TEST_use_ftruncate_after_unlinkutimens/futimens->TEST_posix_utimens,TEST_use_futimens_after_unlinkfallocate->TEST_use_fallocate_after_unlinkcopy_file_range->TEST_posix_copy_file_rangeioctl->TEST_posix_ioctlpoll->TEST_posix_polltmpfile(O_TMPFILE) ->TEST_posix_tmpfilebmap(viaFIBMAPioctl parity) ->TEST_posix_bmapreadlink->TEST_readlink_semanticslink/symlink->TEST_posix_link_symlinkmkdir/rmdir->TEST_posix_mkdir_rmdirunlink/rename->TEST_posix_unlink_rename,TEST_unlink_renamestatfs->TEST_posix_statfsstatx->TEST_posix_statxsyncfs->TEST_posix_syncfsflock/lock->TEST_posix_lockingsetxattr/getxattr/listxattr/removexattr->TEST_posix_xattr- xattr object/flag matrix (
file,dir, symlinkl*, create/replace) ->TEST_posix_xattr_matrix - xattr mode runtime toggles (
passthrough/noattr/nosys) ->TEST_cfg_xattr_modes - statfs mode and statfs-ignore runtime toggles ->
TEST_cfg_statfs_ignore - EXDEV fallback runtime toggles for link/rename ->
TEST_cfg_link_rename_exdev readdir(+ seek/tell behavior) ->TEST_posix_readdirreaddir_plussemantic parity (list+stat and readdir policy toggles) ->TEST_posix_readdir_plus- lifecycle (
init/destroy) harness check ->TEST_mount_lifecycle - hidden temp exposure checks ->
TEST_no_fuse_hidden
Error Conditions Emphasized
Across tests, parity checks include combinations of:
ENOENTENOTDIREEXISTENOTEMPTYEISDIREBADF- privilege-sensitive outcomes (for example
EPERM) where applicable
Notes
- Some behavior is runtime/environment dependent (kernel, mount options, uid/gid, capabilities). Tests compare mergerfs to native behavior in the same runtime to keep assertions robust.
- Tests that cannot exercise a behavior in the current environment may exit 77
and are reported as
SKIPbytests/run-tests. - The xattr test uses a
user.*key and assumes user xattrs are enabled. - Tests mount their own ephemeral mergerfs instances using the
mergerfs_mountcontext manager inposix_parity.py. Random directories are created and cleaned up automatically.