mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-11-21 23:46:26 -05:00
std::filesystem::path is wrong for mergerfs's hot path:
- `branch->path / fusepath_` heap-allocates per FUSE op (3 allocs each
on libstdc++; the SSO buffer is 15 bytes and any joined path exceeds it).
- The implicit `operator string_type()` adds another std::string when
passing to `fs::*` wrappers that take `const std::string&`.
- Per-branch loops rebuild fullpath from scratch every iteration.
The new fs::relpath is a fixed 8192-byte (2 * PATH_MAX) inline buffer,
NUL-terminated, never heap-allocates. The rel data is anchored at offset
REL_OFFSET = 4096; the 4096 bytes before the anchor form the "prefix
area" so a long base/prefix can be written there in place via
set_prefix() without disturbing the rel. The remaining ~4096 bytes
after the rel are available for trailing append (filenames in
readdir-style loops, etc.).
set_prefix's cost is O(prefix size) -- only the actually-used bytes
are touched -- so the buffer size is a stack-footprint question, not a
hot-path cost. FUSE worker threads use the pthread default stack
(~8 MB on Linux); a 4-relpath frame in fuse_link/fuse_rename is 32 KB,
~0.4% of the stack budget.
The per-branch hot loop:
fs::relpath fullpath = fusepath_; // one copy of the rel
for (auto branch : branches) {
fullpath.set_prefix(branch->path); // writes only [start, 4096)
fs::syscall(fullpath);
}
set_prefix only rewrites the prefix area; the rel stays anchored. A
cached _rel_size means no recomputation across iterations.
Microbenchmark (8 branches, 30-char fusepath, -O3, 4 M path builds):
variant ns/build allocs/build
-------------------------------------- -------- ------------
origin/master (std::filesystem::path) 158 3.00
fs::relpath assign_concat (single-shot) 9.1 0
fs::relpath set_prefix (loop-optimized) 4.5 0
35x faster, zero allocations in the hot path.
Contract: paths flow through fs::relpath in canonical form -- prefix
without trailing '/', rel without leading '/'. No sanitization;
violations yield double slashes. Mergerfs satisfies this by
construction:
- libfuse's try_get_path() walks leaf-to-root via prepend(), then
calls consolidate_to_rel(strip=true) which drops the leading '/'
that prepend produces.
- branch paths are realpath-canonicalized at config load so they
never have a trailing slash.
Mergerfs has exactly one path-manipulation type: a relative fusepath.
Branch paths and other absolute paths are stored as std::string -- they
are spliced into the prefix area of an fs::relpath at syscall time but
are never themselves mutated as path objects.
Major changes:
- vendored/libfuse/{include/fs_path.hpp,lib/fs_path.cpp}: new
fs::relpath class (8 KB inline, REL anchor at 4096, set_prefix /
clear_prefix / consolidate_to_rel, _rel_size cache, self-aliasing-
safe assign_concat, sticky ENAMETOOLONG via errnum()).
- 30+ FUSE-op call sites converted from `fullpath = branch / fusepath`
to the `set_prefix` loop pattern (fuse_unlink, fuse_chmod,
fuse_chown, fuse_listxattr, fuse_truncate, fuse_utimens,
fuse_setxattr, fuse_rmdir, fuse_removexattr, fuse_link's
preserve-path and create-path loops, fuse_rename's create-path
and preserve-path loops, fuse_statfs FULL mode, fs::findallfiles,
fs::findonfs).
- fs::* syscall wrappers that previously only took
`const std::string&` got `const char*` overloads so fs::relpath's
implicit `operator const char*()` lands cleanly (fs::link,
fs::lutimens, fs::umount_lazy, fs::open_dir_ro, fs::readlink,
fs::lremovexattr, fs::lstatvfs, fs::info, fs::statvfs_cache).
- fs::exists gained a (fs::relpath&, basepath, ...) overload that
calls set_prefix internally so per-branch policy fan-out loops
share a single fullpath, rewriting only the prefix area on each
iteration.
- All policies (epall, epff, eplfs, eplus, epmfs, eppfrd, ff, lup,
msplfs, msplus, mspmfs, msppfrd, newest) refactored to hoist the
fs::relpath fullpath construction out of the per-branch loop and
use the set_prefix-based fs::exists overload.
- fs::inode::ReaddirCalc::_fusepath migrated from std::string to
fs::relpath with the same resize/append API.
- std::filesystem removed entirely from the project: branches.cpp
and option_parser.cpp's exists/is_directory/equivalent calls now
use direct ::stat(); mergerfs_fsck.cpp's recursive_directory_iterator
was replaced with an opendir-based walk.
Tests (tests/tests.cpp): 27 new fs::relpath test cases covering
construction, set_prefix in/out of REL anchor, prefix()/rel()
accessors, clear_prefix, concat canonical form, self-aliasing
(`p /= "x"`), operator/=, resize zero-fill, swap, copy/move,
equality, implicit conversions, lexically_relative, remove_filename,
_rel_size invariant across every mutator, capacity bounds.
Full suite: 213/213 pass.
139 lines
3.3 KiB
Text
139 lines
3.3 KiB
Text
/*
|
|
ISC License
|
|
|
|
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
|
|
|
|
Permission to use, copy, modify, and/or distribute this software for any
|
|
purpose with or without fee is hereby granted, provided that the above
|
|
copyright notice and this permission notice appear in all copies.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
|
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
|
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
|
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
|
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
|
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
|
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
|
*/
|
|
|
|
#pragma message "using getdents"
|
|
|
|
#include "fs_inode.hpp"
|
|
#include "fs_getdents64.hpp"
|
|
#include "hashset.hpp"
|
|
#include "branches.hpp"
|
|
#include "error.hpp"
|
|
#include "fs_close.hpp"
|
|
#include "fs_open.hpp"
|
|
#include "fs_path.hpp"
|
|
#include "ugid.hpp"
|
|
#include "scope_guard/scope_guard.hpp"
|
|
|
|
#include "fuse_msgbuf.hpp"
|
|
#include "fuse_dirents.hpp"
|
|
|
|
#include <dirent.h>
|
|
|
|
|
|
struct DirRV
|
|
{
|
|
const std::string *branch_path;
|
|
int fd;
|
|
};
|
|
|
|
static
|
|
inline
|
|
std::vector<std::future<DirRV>>
|
|
_opendir(ThreadPool &tp_,
|
|
const Branches::Ptr &branches_,
|
|
const fs::relpath &rel_dirpath_)
|
|
{
|
|
std::vector<std::future<DirRV>> futures;
|
|
|
|
futures.reserve(branches_->size());
|
|
|
|
for(const auto &branch : *branches_)
|
|
{
|
|
auto func =
|
|
[&branch,&rel_dirpath_]()
|
|
{
|
|
int fd;
|
|
fs::relpath abs_dirpath;
|
|
|
|
abs_dirpath = branch.path / rel_dirpath_;
|
|
|
|
fd = fs::open_dir_ro(abs_dirpath);
|
|
|
|
return DirRV{&branch.path,fd};
|
|
};
|
|
|
|
auto rv = tp_.enqueue_task(std::move(func));
|
|
|
|
futures.emplace_back(std::move(rv));
|
|
}
|
|
|
|
return futures;
|
|
}
|
|
|
|
static
|
|
inline
|
|
int
|
|
_readdir(std::vector<std::future<DirRV>> &dh_futures_,
|
|
const fs::relpath &rel_dirpath_,
|
|
fuse_dirents_t *dirents_)
|
|
{
|
|
Err err;
|
|
HashSet names;
|
|
fuse_msgbuf_t *buf;
|
|
|
|
buf = msgbuf_alloc();
|
|
if(not buf)
|
|
return -ENOMEM;
|
|
DEFER { msgbuf_free(buf); };
|
|
|
|
for(auto &dh_future : dh_futures_)
|
|
{
|
|
DirRV dirrv;
|
|
|
|
dirrv = dh_future.get();
|
|
|
|
err = dirrv.fd;
|
|
if(dirrv.fd < 0)
|
|
continue;
|
|
DEFER { fs::close(dirrv.fd); };
|
|
|
|
fs::inode::ReaddirCalc inodecalc(*dirrv.branch_path,rel_dirpath_);
|
|
while(true)
|
|
{
|
|
ssize_t nread;
|
|
|
|
nread = fs::getdents64(dirrv.fd,buf->mem,buf->size);
|
|
if(nread <= 0)
|
|
break;
|
|
|
|
for(ssize_t pos = 0; pos < nread;)
|
|
{
|
|
int rv;
|
|
int namelen;
|
|
fs::dirent64 *d = reinterpret_cast<fs::dirent64*>(&buf->mem[pos]);
|
|
|
|
pos += d->reclen;
|
|
|
|
namelen = d->namelen();
|
|
|
|
rv = names.put(d->name,namelen);
|
|
if(rv == 0)
|
|
continue;
|
|
|
|
d->ino = inodecalc.calc(d->name,
|
|
namelen,
|
|
DTTOIF(d->type),
|
|
d->ino);
|
|
|
|
fuse_dirents_add(dirents_,d,namelen);
|
|
}
|
|
}
|
|
}
|
|
|
|
return err;
|
|
}
|