diff --git a/mkdocs/docs/faq/technical_behavior_and_limitations.md b/mkdocs/docs/faq/technical_behavior_and_limitations.md index 871cf826..13c43220 100644 --- a/mkdocs/docs/faq/technical_behavior_and_limitations.md +++ b/mkdocs/docs/faq/technical_behavior_and_limitations.md @@ -215,13 +215,63 @@ If `mergerfs` doesn't work as a type it could be due to how the permissions. -## Why was splice support removed? +## Does mergerfs use splice? -After a lot of testing over the years, splicing always appeared to -at best, provide equivalent performance, and in some cases, worse -performance. Splice is not supported on other platforms forcing a -traditional read/write fallback to be provided. The splice code was -removed to simplify the codebase. +On Linux, for large reads only, automatically. When a `read` request +is 64KiB or larger, mergerfs splices the data from the branch file +through a pipe into `/dev/fuse` instead of reading it into its own +memory and writing it back out. The data never crosses userspace, which +saves one copy and some CPU per request. Smaller reads, all writes and +every other request use the normal copy path, because splicing them +costs more in extra syscalls than it saves: earlier versions that +spliced every request were measurably slower for small I/O and metadata +operations. + +There is nothing to configure. If splicing a reply is not possible the +read is served through the copy path instead: a branch filesystem that +does not support splice is detected on the first attempt and skipped for +that open file, and a read extending past the size the branch reports +for the file (the last part of a file, files smaller than the read, +files reporting a size of 0 such as procfs files, and files reporting a +nominal 4096 bytes such as sysfs attributes) uses the copy path, so a +branch whose reported size lags its data still +returns everything. Files on FUSE, SMB/CIFS, NFS, Ceph, AFS, overlayfs +or 9p branches, and any file opened `O_DIRECT`, always use the copy +path: those filesystems' splice_read can serve a page cache that their +normal reads bypass or first revalidate, so a spliced reply could +return stale data, and for an `O_DIRECT` file the kernel would serve +the splice by direct I/O into newly allocated pages, which saves +nothing over the copy path. +A read that arrives while mergerfs is moving the file to another +branch (`moveonenospc`) also uses the copy path. +Unless mergerfs runs with +`CAP_SYS_RESOURCE` (as root on a host normally does, but root in a +container usually does not unless started with `--cap-add=SYS_RESOURCE` +or `--privileged`; `CAP_SYS_ADMIN` is not enough), the `pipe-max-size` +sysctl limits the pipe a reply can use. A reply needs one page more +than the read (for the reply header), two when the read does not start +on a page boundary, and the kernel rounds that up to a power of two, so +`pipe-max-size` must be larger than `fuse-msg-size` for the largest +reads to be spliced: with the defaults (both 1MiB) 1MiB reads use the +copy path while smaller reads keep splicing, with a single log +message. That limit is read when mergerfs starts, so raising +`pipe-max-size` takes effect after mergerfs is restarted. Only a kernel +without splice support, or a seccomp policy that refuses it, disables it +for the whole process. The old +`splice_read`, `splice_write` and `splice_move` options are still +accepted and do nothing. + +Two behaviors of spliced reads differ from the copy path. The data is +copied into the reply only after the branch filesystem's read has +returned, so a write or truncate of the branch file racing a read can +produce a reply mixing old and new contents, even on filesystems such as +XFS that otherwise serialize reads against writes. And on kernels before +6.4, spliced reads do not generate inotify or fanotify access events +(`IN_ACCESS`, `FAN_ACCESS`) on the branch file. + +Files using [passthrough.io](../config/passthrough.md) never reach +mergerfs for reads, so splice does not apply to them, unless +`direct-io` is also in effect for them, which overrides passthrough. ## How does mergerfs handle credentials? diff --git a/src/fileinfo.hpp b/src/fileinfo.hpp index f3524205..4fbcd30c 100644 --- a/src/fileinfo.hpp +++ b/src/fileinfo.hpp @@ -50,6 +50,8 @@ #include "base_types.h" +#include +#include #include @@ -105,8 +107,86 @@ public: // Common case (plain pwrite on stable fd) only needs a shared lock // since the kernel already serializes fd mutations. The danger is // the fs::dup2() rebinding when moveonenospc activates. Writers - // take shared lock; move takes unique. + // take shared lock; move takes unique (lock_exclusive). Spliced + // reads (FUSE::read_buf) also take it shared, but only if that + // needs no waiting. std::shared_mutex mutex; + // Threads waiting in lock_exclusive. FUSE::read_buf starts no splice + // while one waits: the rwlock prefers readers, so overlapping spliced + // reads could otherwise hold a mover off indefinitely. + std::atomic exclusive_waiters{0}; + // Lower bound on the branch file's size, used only by FUSE::read_buf + // to skip its end-of-file size check for reads ending inside this + // bound. Raised by that check, lowered by any short read (FUSE::read) + // or ftruncate and reset by fallocate through this handle. It may go + // stale when the file shrinks through another handle or outside + // mergerfs; the splice reply then hits EOF and falls back to + // FUSE::read, which returns the correct bytes and lowers the bound + // to where that read found the end (to 0 when it read nothing, as + // the end is then somewhere at or before its offset). 0 = unknown. + std::atomic size_floor{0}; + // Whether FUSE::read_buf may offer fd for splicing: unknown until the + // first large read checks the branch filesystem (some serve splice + // from a page cache their reads bypass or revalidate), no once that + // check said so or the fd refused splice (its filesystem has no + // splice_read), after which reads go straight to the copy path. + // Decided once per fd binding; a filesystem refused by type is also + // remembered by device, so later handles on it are settled without + // repeating that check (see FUSE::read_buf). + enum class Splice : uint8_t { unknown, yes, no }; + std::atomic splice_state{Splice::unknown}; + + // Take mutex exclusively (moveonenospc), announced through + // exclusive_waiters so that no new spliced read starts meanwhile. + std::unique_lock + lock_exclusive() + { + // Counted out again even if locking throws, so a failed attempt + // cannot leave spliced reads on this handle declined for good. + struct Waiting + { + std::atomic &count; + + explicit Waiting(std::atomic &count_) + : count(count_) + { + count.fetch_add(1,std::memory_order_relaxed); + } + + ~Waiting() + { + count.fetch_sub(1,std::memory_order_relaxed); + } + } waiting(exclusive_waiters); + + return std::unique_lock(mutex); + } + + void size_floor_lower(const s64 v_) + { + s64 cur = size_floor.load(std::memory_order_relaxed); + while((v_ < cur) && + !size_floor.compare_exchange_weak(cur,v_,std::memory_order_relaxed)) + ; + } + + // The file's size may have changed in a way size_floor_lower cannot + // express: call after the change. That narrows, but does not close, + // the window in which a size check that started before the change + // (and saw 0, the value stored here) installs the old size + // afterwards; like any stale floor that only costs a fallback. + void size_floor_reset() + { + size_floor.store(0,std::memory_order_relaxed); + } + + // fd now refers to a different branch file (moveonenospc's dup2): a + // splice decision or size observed on the old one no longer applies. + void fd_rebound() + { + splice_state.store(Splice::unknown,std::memory_order_relaxed); + size_floor_reset(); + } }; inline diff --git a/src/fuse_fallocate.cpp b/src/fuse_fallocate.cpp index 4eb964ab..81c7d7e7 100644 --- a/src/fuse_fallocate.cpp +++ b/src/fuse_fallocate.cpp @@ -55,14 +55,23 @@ FUSE::fallocate(const fuse_req_ctx_t *ctx_, off_t offset_, off_t len_) { + int rv; FileInfo *fi; fi = state.get_fi(ctx_,fh_); if(not fi) return -EBADF; - return ::_fallocate(fi->fd, - mode_, - offset_, - len_); + rv = ::_fallocate(fi->fd, + mode_, + offset_, + len_); + + /* Reset read_buf's size floor. Defensive only: of the fallocate + modes only COLLAPSE_RANGE shrinks a file (PUNCH_HOLE requires + KEEP_SIZE), and the kernel's FUSE client forwards only KEEP_SIZE, + PUNCH_HOLE and ZERO_RANGE, so no request reaching here does. */ + fi->size_floor_reset(); + + return rv; } diff --git a/src/fuse_ftruncate.cpp b/src/fuse_ftruncate.cpp index 3f1cfdfd..3b7d6404 100644 --- a/src/fuse_ftruncate.cpp +++ b/src/fuse_ftruncate.cpp @@ -50,11 +50,20 @@ FUSE::ftruncate(const fuse_req_ctx_t *ctx_, cu64 fh_, off_t size_) { + int rv; FileInfo *fi; fi = state.get_fi(ctx_,fh_); if(not fi) return -EBADF; - return ::_ftruncate(fi->fd,size_); + rv = ::_ftruncate(fi->fd,size_); + + /* A shrink through this handle would otherwise leave reads past the + new end on the EOF fallback; after a grow the old floor still + holds. */ + if(rv == 0) + fi->size_floor_lower(size_); + + return rv; } diff --git a/src/fuse_read.cpp b/src/fuse_read.cpp index 57c9daae..cfb18e39 100644 --- a/src/fuse_read.cpp +++ b/src/fuse_read.cpp @@ -72,10 +72,19 @@ FUSE::read(const fuse_req_ctx_t *ctx_, if(not fi) return -EBADF; - if(fi->direct_io) - return ::_read_direct_io(fi->fd,buf_,size_,offset_); + const int rv = (fi->direct_io ? + ::_read_direct_io(fi->fd,buf_,size_,offset_) : + ::_read_cached(fi->fd,buf_,size_,offset_)); - return ::_read_cached(fi->fd,buf_,size_,offset_); + /* A short read means end-of-file at offset+rv: keep read_buf's size + floor from claiming more (it is the splice reply's EOF fallback). + Nothing was read at all when rv is 0, so the file may end anywhere + at or before offset_ (it may have shrunk well below the offset + asked about): only 0 is then a bound that cannot be too high. */ + if((rv >= 0) && ((size_t)rv < size_)) + fi->size_floor_lower((rv > 0) ? ((s64)offset_ + rv) : 0); + + return rv; } int diff --git a/src/fuse_read_buf.cpp b/src/fuse_read_buf.cpp new file mode 100644 index 00000000..69d23498 --- /dev/null +++ b/src/fuse_read_buf.cpp @@ -0,0 +1,313 @@ +/* + ISC License + + Copyright (c) 2026, Antonio SJ Musumeci + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ + +#include "fuse_read_buf.hpp" + +#include "fileinfo.hpp" +#include "fs_fstat.hpp" +#include "fs_getfl.hpp" +#include "fs_statx.hpp" +#include "ioprio.hpp" +#include "state.hpp" + +#include "fuse.h" + +#include "boost/unordered/concurrent_flat_map.hpp" + +#include +#include +#include + +#if defined(__linux__) +#include +#include +#endif + + +/* The branch file's type, size and device. A statx asking only for the + first two lets filesystems skip work a full fstat implies (NFS + flushes the file's dirty pages before reporting c/mtime; the device + is always reported); fstat covers builds and filesystems without + statx. */ +static +int +_type_and_size(const int fd_, + mode_t *mode_, + s64 *size_, + dev_t *dev_) +{ + int rv; + struct stat st; + +#ifdef MERGERFS_SUPPORTED_STATX + struct fuse_statx stx; + const unsigned int want = (STATX_TYPE|STATX_SIZE); + + rv = fs::statx(fd_,"",AT_EMPTY_PATH,want,&stx); + if((rv == 0) && ((stx.mask & want) == want)) + { + *mode_ = stx.mode; + *size_ = (s64)stx.size; + *dev_ = makedev(stx.dev_major,stx.dev_minor); + return 0; + } +#endif + + rv = fs::fstat(fd_,&st); + if(rv < 0) + return rv; + + *mode_ = st.st_mode; + *size_ = st.st_size; + *dev_ = st.st_dev; + + return 0; +} + +/* Devices of branch filesystems _splice_coherent refused by type. The + type is a property of the superblock, not of the open file, and on + the network and FUSE filesystems it refuses fstatfs is a round trip + to their server, which the first large read of every opened file + would otherwise pay. Only refusals are remembered, so a device + number reused by another filesystem after an unmount can cost + splicing but never coherence. The value is unused. */ +static boost::concurrent_flat_map g_refused_devs; + +/* Whether splicing from the branch fd returns what pread would, and is + worth doing. A filesystem that cannot be identified is refused too. + Not on FUSE, SMB/CIFS, NFS, Ceph, AFS, overlayfs or 9p + branches: their splice_read serves the branch inode's page cache + (filemap_splice_read) while their read path bypasses it or first + revalidates it (FUSE direct_io and auto_inval_data, SMB without a + read lease, NFS/Ceph/AFS/9p attribute/data caching, overlayfs's + copy-up), so a spliced reply could return stale data where the copy + path returns current data. Nor on an fd opened O_DIRECT, regardless + of filesystem: the kernel serves that splice by direct I/O into + freshly allocated pipe pages (as pread does, so nothing is stale), + which saves no copy over the copy path while adding syscalls, and a + misaligned request's EINVAL would look like a filesystem without + splice_read. A refusal by filesystem type is remembered for dev_ (the + fd's device, 0 if unknown) so later handles on that filesystem skip + the fstatfs. */ +static +bool +_splice_coherent(const int fd_, + const dev_t dev_) +{ +#if defined(__linux__) + constexpr u32 MAGIC_FUSE = 0x65735546; // fuse, fuseblk, virtiofs + constexpr u32 MAGIC_CIFS = 0xFF534D42; + constexpr u32 MAGIC_SMB2 = 0xFE534D42; + constexpr u32 MAGIC_NFS = 0x00006969; + constexpr u32 MAGIC_CEPH = 0x00C36400; + constexpr u32 MAGIC_AFS = 0x5346414F; // OpenAFS (/afs) + constexpr u32 MAGIC_KAFS = 0x6B414653; // in-kernel AFS client + constexpr u32 MAGIC_OVERLAYFS = 0x794C7630; + constexpr u32 MAGIC_V9FS = 0x01021997; + struct statfs st; + int rv; + + if(::fstatfs(fd_,&st) < 0) + return false; + + switch((u32)st.f_type) + { + case MAGIC_FUSE: + case MAGIC_CIFS: + case MAGIC_SMB2: + case MAGIC_NFS: + case MAGIC_CEPH: + case MAGIC_AFS: + case MAGIC_KAFS: + case MAGIC_OVERLAYFS: + case MAGIC_V9FS: + if(dev_ != 0) + g_refused_devs.emplace(dev_,true); + return false; + default: + break; + } + + rv = fs::getfl(fd_); + if((rv >= 0) && (rv & O_DIRECT)) + return false; + + return true; +#else + (void)fd_; + (void)dev_; + return false; +#endif +} + +/* Whether [offset, offset+size) lies inside [0, bound), computed + without overflow: offset+size itself can exceed INT64_MAX for reads + near the top of a huge sparse file. */ +static +inline +bool +_window_inside(const off_t offset_, + const size_t size_, + const s64 bound_) +{ + return ((offset_ >= 0) && + (offset_ < bound_) && + ((u64)size_ <= (u64)(bound_ - offset_))); +} + +/* Describe a large read as a region of the branch file so libfuse can + splice it into /dev/fuse without the data crossing userspace. + Returning a positive value declines, and libfuse calls FUSE::read. + + The reply header, whose length is the region size, is staged before + any data, so only a read lying entirely within the file's reported + size is offered. A read crossing it declines rather than being + trimmed: some branches report a size that lags the data + (attribute-caching network or FUSE filesystems), and a trimmed reply + would end short where pread returns the rest. This also declines + non-regular files and files reporting size 0 (procfs) or 4096 + (sysfs) while returning live content, which a splice past the + reported size would generate twice or block on. A file that shrinks + between the size check and the splice falls back to FUSE::read. + + The size check is skipped when the window ends inside + fi->size_floor, a size already observed on this handle; a stale floor + only costs a fallback, whose short FUSE::read lowers it. + + fi->mutex is held shared so moveonenospc's dup2 cannot rebind fd + while libfuse stages from it (see FileRegion::lock). It is never + waited for: a move holds it for a whole file copy, so the read + declines instead and the copy path serves it from the old fd. */ +int +FUSE::read_buf(const fuse_req_ctx_t *ctx_, + const fuse_file_info_t *ffi_, + FileRegion *region_, + const size_t size_, + const off_t offset_) +{ + FileInfo *fi; + FileInfo::Splice splice; + s64 floor; + dev_t dev = 0; + + fi = state.get_fi(ctx_,ffi_->fh); + if(not fi) + return -EBADF; + + if(fi->splice_state.load(std::memory_order_relaxed) == FileInfo::Splice::no) + return 1; + + std::shared_lock lock(fi->mutex,std::try_to_lock); + if(not lock.owns_lock()) + return 1; + + /* Checked after the lock is held so a mover that started waiting + before try_to_lock succeeded is not joined by this splice. */ + if(fi->exclusive_waiters.load(std::memory_order_relaxed) > 0) + return 1; + + /* Read again under the lock: a move that completed before try_to_lock + rebound fd and reset this to unknown (fd_rebound), and the + once-per-binding check below must see that. */ + splice = fi->splice_state.load(std::memory_order_relaxed); + if(splice == FileInfo::Splice::no) + return 1; + + floor = fi->size_floor.load(std::memory_order_relaxed); + if(not ::_window_inside(offset_,size_,floor)) + { + mode_t mode; + s64 fsize; + + if((::_type_and_size(fi->fd,&mode,&fsize,&dev) < 0) || + !S_ISREG(mode)) + return 1; + + /* On a filesystem already refused by type, settle this handle + now, even for a read that would decline below anyway (every + read of a file smaller than the request does), so its later + reads skip this statx. */ + if((splice == FileInfo::Splice::unknown) && g_refused_devs.contains(dev)) + { + fi->splice_state.compare_exchange_strong(splice, + FileInfo::Splice::no, + std::memory_order_relaxed); + return 1; + } + + /* Only replace the floor this check started from: a short read + may have lowered it meanwhile, and that is newer than fsize. */ + if(fsize > 0) + fi->size_floor.compare_exchange_strong(floor,fsize, + std::memory_order_relaxed); + if(not ::_window_inside(offset_,size_,fsize)) + return 1; + } + + /* Once per fd binding, for a read that could otherwise splice, and + under the lock so the answer is about the file fd refers to. A + concurrent rejection (read_buf_rejected) wins over this check. */ + if(splice == FileInfo::Splice::unknown) + { + /* A filesystem that cannot be identified is not offered either, + and that is settled too: retrying would repeat the fstatfs, + possibly a server round trip, on every later large read. */ + const FileInfo::Splice want = (::_splice_coherent(fi->fd,dev) ? + FileInfo::Splice::yes : + FileInfo::Splice::no); + + /* On failure splice holds the concurrent decision, which wins. */ + if(fi->splice_state.compare_exchange_strong(splice,want, + std::memory_order_relaxed)) + splice = want; + if(splice != FileInfo::Splice::yes) + return 1; + } + + /* ioprio is thread-persistent; without SetFrom the branch-file I/O + performed downstream of this handler (the kernel-side splice in + the reply path runs on this same worker thread) would inherit + whatever priority a previous request left behind. */ + ioprio::SetFrom iop(ctx_->pid); + + region_->fd = fi->fd; + region_->offset = offset_; + region_->size = size_; + region_->lock = std::move(lock); + + return 0; +} + +/* The branch fd refused splice (EINVAL on its first splice: its + filesystem has no splice_read). That will not change for this fd, so + stop offering it and let every later read go straight to FUSE::read. + O_DIRECT fds never get here: _splice_coherent refuses them. libfuse + calls this while the region's lock is still held, so fd is still the + one that refused. */ +void +FUSE::read_buf_rejected(const fuse_req_ctx_t *ctx_, + const fuse_file_info_t *ffi_) +{ + FileInfo *fi; + + fi = state.get_fi(ctx_,ffi_->fh); + if(not fi) + return; + + fi->splice_state.store(FileInfo::Splice::no,std::memory_order_relaxed); +} diff --git a/src/fuse_read_buf.hpp b/src/fuse_read_buf.hpp new file mode 100644 index 00000000..1d16390f --- /dev/null +++ b/src/fuse_read_buf.hpp @@ -0,0 +1,35 @@ +/* + ISC License + + Copyright (c) 2026, Antonio SJ Musumeci + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ + +#pragma once + +#include "fuse_io.hpp" + +#include "fuse.h" + + +namespace FUSE +{ + int read_buf(const fuse_req_ctx_t *ctx, + const fuse_file_info_t *ffi, + FileRegion *region, + const size_t size, + const off_t offset); + void read_buf_rejected(const fuse_req_ctx_t *ctx, + const fuse_file_info_t *ffi); +} diff --git a/src/fuse_write.cpp b/src/fuse_write.cpp index b91c7ba0..b2260d34 100644 --- a/src/fuse_write.cpp +++ b/src/fuse_write.cpp @@ -28,11 +28,13 @@ #include "fs_pwriten.hpp" #include "ioprio.hpp" #include "state.hpp" +#include "syslog.hpp" #include "scope_guard/scope_guard.hpp" #include "fuse.h" +#include #include #include @@ -45,6 +47,41 @@ _out_of_space(const ssize_t err_) (err_ == -EDQUOT)); } +// moveonenospc: move the file to another branch and rebind fi_->fd to +// the copy. Called with fi_->mutex held exclusively. 0 or -errno. +static +int +_move_file(FileInfo *fi_) +{ + int err; + int fd; + + if(cfg.moveonenospc.enabled == false) + return -ENOSPC; + + fd = fs::movefile_and_open_as_root(cfg.moveonenospc.policy, + cfg.branches, + fi_->branch.path, + fi_->fusepath, + fi_->fd); + if(fd < 0) + return fd; + + err = fs::dup2(fd,fi_->fd); + { + const int close_err = fs::close(fd); + if(close_err < 0) + SysLog::warning("moveonenospc: closing the copy's fd after the move" + " failed: {}",strerror(-close_err)); + } + if(err < 0) + return err; + + fi_->fd_rebound(); + + return 0; +} + static int _move_and_pwrite(const char *buf_, @@ -53,23 +90,7 @@ _move_and_pwrite(const char *buf_, FileInfo *fi_, int err_) { - int err; - ssize_t rv; - - if(cfg.moveonenospc.enabled == false) - return err_; - - rv = fs::movefile_and_open_as_root(cfg.moveonenospc.policy, - cfg.branches, - fi_->branch.path, - fi_->fusepath, - fi_->fd); - if(rv < 0) - return err_; - - err = fs::dup2(rv,fi_->fd); - fs::close(rv); - if(err < 0) + if(::_move_file(fi_) < 0) return err_; return fs::pwrite(fi_->fd,buf_,count_,offset_); @@ -87,20 +108,7 @@ _move_and_pwriten(char const *buf_, int err; ssize_t rv; - if(cfg.moveonenospc.enabled == false) - return err_; - - rv = fs::movefile_and_open_as_root(cfg.moveonenospc.policy, - cfg.branches, - fi_->branch.path, - fi_->fusepath, - fi_->fd); - if(rv < 0) - return err_; - - err = fs::dup2(rv,fi_->fd); - fs::close(rv); - if(err < 0) + if(::_move_file(fi_) < 0) return err_; rv = fs::pwriten(fi_->fd, @@ -147,7 +155,11 @@ _write(const fuse_req_ctx_t *ctx_, ssize_t rv; - if(fi->direct_io) + // A writeback (writepage) request must be written completely or + // fail: the kernel ignores a short count for it (see FUSE::write). + // So even on a direct_io handle it takes the looping path below, + // which also reaches moveonenospc when the branch fills mid-write. + if(fi->direct_io && !ffi_->writepage) { { std::shared_lock slk(fi->mutex); @@ -157,7 +169,7 @@ _write(const fuse_req_ctx_t *ctx_, if(not ::_out_of_space(rv)) return rv; - std::unique_lock ulk(fi->mutex); + std::unique_lock ulk = fi->lock_exclusive(); // Re-check under exclusive lock: another writer may have // already moved the file and retired. If so the pwrite should // now succeed without our own move call. @@ -181,7 +193,7 @@ _write(const fuse_req_ctx_t *ctx_, if(err && not ::_out_of_space(err)) return err; - std::unique_lock ulk(fi->mutex); + std::unique_lock ulk = fi->lock_exclusive(); // Re-check under exclusive lock as another move may have // already run. written = fs::pwriten(fi->fd,buf_,count_,offset_,&err); @@ -202,8 +214,20 @@ FUSE::write(const fuse_req_ctx_t *ctx_, off_t offset_) { ioprio::SetFrom iop(ctx_->pid); + int rv; - return ::_write(ctx_,ffi_,buf_,count_,offset_); + rv = ::_write(ctx_,ffi_,buf_,count_,offset_); + + /* Writeback completion (FUSE_WRITE_CACHE) ignores a short reply's byte + count: Linux fuse_writepage_end consults only the error, so a short + nonnegative result would mark the unwritten tail clean and lose it. + _write already loops for these requests; a short count here means + a pwrite made no progress without an error, so report a failure. + Ordinary writes report their prefix. */ + if(ffi_->writepage && (rv >= 0) && ((size_t)rv < count_)) + return -EIO; + + return rv; } int diff --git a/src/mergerfs.cpp b/src/mergerfs.cpp index c4ad8903..02b13a86 100644 --- a/src/mergerfs.cpp +++ b/src/mergerfs.cpp @@ -63,6 +63,7 @@ #include "fuse_opendir.hpp" #include "fuse_poll.hpp" #include "fuse_read.hpp" +#include "fuse_read_buf.hpp" #include "fuse_readdir.hpp" #include "fuse_readdir_plus.hpp" #include "fuse_readlink.hpp" @@ -126,6 +127,10 @@ _get_fuse_operations(struct fuse_operations &ops_, ops_.opendir = FUSE::opendir; ops_.poll = FUSE::poll; ops_.read = (nullrw_ ? FUSE::read_null : FUSE::read); + /* Large reads are spliced from the branch file into /dev/fuse when + possible; nullrw has no branch data to splice. */ + ops_.read_buf = (nullrw_ ? nullptr : FUSE::read_buf); + ops_.read_buf_rejected = (nullrw_ ? nullptr : FUSE::read_buf_rejected); ops_.readdir = FUSE::readdir; ops_.readdir_plus = FUSE::readdir_plus; ops_.readlink = FUSE::readlink; diff --git a/tests/tests.cpp b/tests/tests.cpp index 89ffee31..c734a2aa 100644 --- a/tests/tests.cpp +++ b/tests/tests.cpp @@ -1,6 +1,13 @@ #include "acutest/acutest.h" #include "config.hpp" +#include "fileinfo.hpp" +#include "fuse_cfg.hpp" +#include "fuse_ftruncate.hpp" +#include "fuse_lowlevel.h" +#include "fuse_read.hpp" +#include "fuse_read_buf.hpp" +#include "fuse_write.hpp" #include "fs_copyfile.hpp" #include "fs_inode.hpp" #include "from_string.hpp" @@ -27,6 +34,18 @@ #include #include #include +#include +#include +#include +#if defined(__linux__) +#include +#include +#include +#include +#include +#include +#include +#endif template bool @@ -50,6 +69,685 @@ test_nop() TEST_CHECK(true); } +#if defined(__linux__) +/* Drive fuse_reply_data_splice_fd with a regular file standing in for + /dev/fuse and return every byte the "kernel" received. Unlike a pipe, + the file takes the whole message whatever pipe-max-size or the + per-user pipe quota allow, so a short result is the reply path's. */ +static +std::string +_splice_reply_bytes(const int fd_, + const off_t offset_, + const size_t size_, + FuseReplyResult *result_, + const uint64_t unique_ = 7) +{ + FileRegion region{fd_,offset_,size_,{}}; + + *result_ = FuseReplyResult::failed; + FILE *dev = tmpfile(); + if(dev == nullptr) + return "tmpfile failed"; + + fuse_req_t *req = fuse_req_alloc(); + req->fd = fileno(dev); + req->ctx.unique = unique_; + *result_ = fuse_reply_data_splice_fd(req,region); + fuse_req_free(req); + + std::string got; + char buf[65536]; + ssize_t n; + while((n = pread(fileno(dev),buf,sizeof(buf),(off_t)got.size())) > 0) + got.append(buf,n); + fclose(dev); + return got; +} + +/* Whether this process may grow a pipe to bytes_, as the reply path + must for the tests below: unprivileged, pipe-max-size or the per-user + pipe quota can refuse it, and the reply then correctly falls back. */ +static +bool +_pipe_can_grow(const size_t bytes_) +{ + int fds[2]; + if(pipe2(fds,O_CLOEXEC) != 0) + return false; + const int rv = fcntl(fds[0],F_SETPIPE_SZ,(int)bytes_); + close(fds[0]); + close(fds[1]); + return (rv >= (int)bytes_); +} + +/* A tmpfile() holding size_ bytes of data_, or nullptr. */ +static +FILE* +_tmpfile_with(const void *data_, + const size_t size_) +{ + FILE *file = tmpfile(); + if((file != nullptr) && + (pwrite(fileno(file),data_,size_,0) != (ssize_t)size_)) + { + fclose(file); + return nullptr; + } + return file; +} + +/* Whether got_ is exactly one successful FUSE message for unique_ + carrying the len_ bytes at expected_. */ +static +bool +_is_reply(const std::string &got_, + const uint64_t unique_, + const char *expected_, + const size_t len_) +{ + fuse_out_header hdr; + + if(got_.size() != sizeof(hdr) + len_) + return false; + memcpy(&hdr,got_.data(),sizeof(hdr)); + return ((hdr.len == sizeof(hdr) + len_) && + (hdr.unique == unique_) && + (hdr.error == 0) && + (memcmp(got_.data() + sizeof(hdr),expected_,len_) == 0)); +} + +/* An open handle on fd_ as FUSE::read_buf and friends see it. */ +struct TestHandle +{ + Branch branch; + FileInfo fi; + fuse_req_ctx_t ctx = {}; + fuse_file_info_t ffi = {}; + + TestHandle(const int fd_, + const char *fusepath_) + : fi(fd_,branch,fs::path(fusepath_),false) + { + ffi.fh = fi.to_fh(); + } +}; + +/* A spliced reply must be exactly one FUSE message: a header whose len + covers the payload, the right unique, zero error, then the file bytes + of the requested window, including an unaligned start. */ +void +test_reply_splice_sends_exact_message() +{ + if(!::_pipe_can_grow(262144)) + { + TEST_SKIP("cannot grow a pipe to 256KiB here"); + return; + } + + static char data[300000]; + for(size_t i = 0; i < sizeof(data); i++) + data[i] = (char)(i * 31 + 7); + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + + const off_t off = 4097; + const size_t len = 131072; + FuseReplyResult r; + const std::string got = _splice_reply_bytes(fileno(file),off,len,&r,42); + + TEST_CHECK(r == FuseReplyResult::sent); + TEST_CHECK(::_is_reply(got,42,data + off,len)); + fclose(file); +} + +/* A region that runs past end-of-file must not send a short message: + nothing may reach the kernel and the caller gets fallback, so it can + reply through read(). */ +void +test_reply_splice_eof_falls_back_without_sending() +{ + if(!::_pipe_can_grow(262144)) + { + TEST_SKIP("cannot grow a pipe to 256KiB here"); + return; + } + + static char data[70000]; + for(size_t i = 0; i < sizeof(data); i++) + data[i] = (char)(i * 13 + 5); + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + + FuseReplyResult r; + const std::string got = _splice_reply_bytes(fileno(file),0,131072,&r); + TEST_CHECK(r == FuseReplyResult::fallback); + TEST_CHECK(got.empty()); + + /* The abandoned staging must leave this thread's reply pipe empty: + the next reply is exactly its own message, nothing left over. */ + const std::string got2 = _splice_reply_bytes(fileno(file),4096,65536,&r,9); + TEST_CHECK(r == FuseReplyResult::sent); + TEST_CHECK(::_is_reply(got2,9,data + 4096,65536)); + fclose(file); +} + +/* A reply below the splice threshold is cheaper through read + writev, + so the splice path declines it before staging anything. */ +void +test_reply_splice_small_region_falls_back() +{ + char data[8192]; + memset(data,0x44,sizeof(data)); + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + + FuseReplyResult r; + std::string got = _splice_reply_bytes(fileno(file),0,8192,&r); + TEST_CHECK(r == FuseReplyResult::fallback); + TEST_CHECK(got.empty()); + + got = _splice_reply_bytes(fileno(file),8192,0,&r); + TEST_CHECK(r == FuseReplyResult::fallback); + TEST_CHECK(got.empty()); + fclose(file); +} + +/* A descriptor splice rejects with EINVAL is reported as unsupported, + again with nothing sent. An eventfd has no pread, so splicing from + it at an offset is EINVAL on every kernel (a directory is EINVAL only + since Linux 5.10; older kernels fail it with EISDIR). */ +void +test_reply_splice_unspliceable_fd_is_unsupported() +{ + if(!::_pipe_can_grow(131072)) + { + TEST_SKIP("cannot grow a pipe to 128KiB here"); + return; + } + + const int efd = eventfd(0,EFD_CLOEXEC); + TEST_ASSERT(efd >= 0); + FuseReplyResult r; + const std::string got = _splice_reply_bytes(efd,0,65536,&r); + TEST_CHECK(r == FuseReplyResult::unsupported); + TEST_CHECK(got.empty()); + close(efd); +} + +/* The region's lock is released as FileRegion::lock documents: once the + data is staged for a sent reply, but kept by a fallback during + staging (`tail` runs past end-of-file). */ +void +test_reply_splice_releases_lock_before_submit() +{ + if(!::_pipe_can_grow(262144)) + { + TEST_SKIP("cannot grow a pipe to 256KiB here"); + return; + } + + static char data[200000]; + + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + FILE *dev = tmpfile(); + TEST_ASSERT(dev != nullptr); + std::shared_mutex mutex; + fuse_req_t *req = fuse_req_alloc(); + req->fd = fileno(dev); + req->ctx.unique = 11; + + FileRegion sent{fileno(file),0,131072,std::shared_lock(mutex)}; + TEST_CHECK(fuse_reply_data_splice_fd(req,sent) == FuseReplyResult::sent); + TEST_CHECK(!sent.lock.owns_lock()); + TEST_CHECK(mutex.try_lock()); + mutex.unlock(); + + FileRegion tail{fileno(file),131072,131072,std::shared_lock(mutex)}; + TEST_CHECK(fuse_reply_data_splice_fd(req,tail) == FuseReplyResult::fallback); + TEST_CHECK(tail.lock.owns_lock()); + + fuse_req_free(req); + fclose(dev); + fclose(file); +} + +/* A debug log write that throws once the reply is fully staged (fmt on + a full or broken log stream) must not leave that message in this + thread's reply pipe: the worker survives the exception, and its next + reply would otherwise be sent behind, or instead of, the stale one. */ +void +test_reply_splice_debug_throw_leaves_pipe_empty() +{ + if(!::_pipe_can_grow(262144)) + { + TEST_SKIP("cannot grow a pipe to 256KiB here"); + return; + } + + FILE *full = fopen("/dev/full","w"); + if(full == nullptr) + { + TEST_SKIP("no /dev/full here"); + return; + } + setvbuf(full,nullptr,_IONBF,0); + + static char data[200000]; + for(size_t i = 0; i < sizeof(data); i++) + data[i] = (char)(i * 7 + 3); + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + FILE *dev = tmpfile(); + TEST_ASSERT(dev != nullptr); + + const std::shared_ptr log_file = fuse_cfg.log_file(); + const bool debug = fuse_cfg.debug; + fuse_cfg.log_file(std::shared_ptr(full,fclose)); + fuse_cfg.debug = true; + + fuse_req_t *req = fuse_req_alloc(); + req->fd = fileno(dev); + req->ctx.unique = 5; + FileRegion region{fileno(file),0,131072,{}}; + bool threw = false; + try + { + fuse_reply_data_splice_fd(req,region); + } + catch(...) + { + threw = true; + } + fuse_req_free(req); + + fuse_cfg.debug = debug; + fuse_cfg.log_file(log_file); + + struct stat st; + TEST_CHECK(threw); + TEST_CHECK((fstat(fileno(dev),&st) == 0) && (st.st_size == 0)); + + FuseReplyResult r; + const std::string got = _splice_reply_bytes(fileno(file),4096,65536,&r,9); + TEST_CHECK(r == FuseReplyResult::sent); + TEST_CHECK(::_is_reply(got,9,data + 4096,65536)); + + fclose(dev); + fclose(file); +} + +/* Whether fd_ is on a local filesystem that read_buf must always offer + for splicing (ext2/3/4, XFS, tmpfs, btrfs, f2fs). */ +static +bool +_splice_local_fs(const int fd_) +{ + struct statfs st; + + if(::fstatfs(fd_,&st) != 0) + return false; + + switch((unsigned long)st.f_type) + { + case 0xEF53: // ext2/3/4 + case 0x58465342: // xfs + case 0x01021994: // tmpfs + case 0x9123683E: // btrfs + case 0xF2F52010: // f2fs + return true; + default: + return false; + } +} + +/* Whether read_buf refuses the filesystem tmpfile() uses outright + (overlayfs, NFS, FUSE, ...: see _splice_coherent). tmpfile() always + uses /tmp, which is such a filesystem inside most containers, so the + read_buf tests skip there instead of failing. Never on a local + filesystem read_buf must offer: a refusal there is the regression + the tests exist to catch, so they run (and fail) instead of every + one of them skipping. Probes its own file through a throwaway + handle, before the caller performs any check (acutest cannot skip + after one). */ +static +bool +_read_buf_refuses_tmp() +{ + static char zeros[65536]; + FILE *file = ::_tmpfile_with(zeros,sizeof(zeros)); + + if(file == nullptr) + return false; + + TestHandle h(fileno(file),"/read-buf-probe"); + FileRegion region; + FUSE::read_buf(&h.ctx,&h.ffi,®ion,sizeof(zeros),0); + const bool refused = (h.fi.splice_state.load() == FileInfo::Splice::no); + const bool local = ::_splice_local_fs(fileno(file)); + region = {}; + fclose(file); + + return (refused && not local); +} + +/* read_buf hands out the branch fd and the whole request window, and + only for a window lying entirely within the file's reported size: + the header carrying the region size is staged before any data, and + a reported size may lag the data, so a window crossing it (a tail, + an EOF probe, any read of a size-0 file) declines instead of being + trimmed. A rejected handle declines too; a bad handle fails; a window + below the splice threshold is never asked for. The returned region + holds the handle's lock shared so the fd cannot be rebound + mid-splice. */ +void +test_read_buf_region_contract() +{ + if(::_read_buf_refuses_tmp()) + { + TEST_SKIP("read_buf refuses this /tmp filesystem"); + return; + } + + static char data[200000]; + + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + TestHandle h(fileno(file),"/read-buf"); + FileInfo &fi = h.fi; + const fuse_req_ctx_t &ctx = h.ctx; + const fuse_file_info_t &ffi = h.ffi; + + FileRegion region; + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,65536,1234) == 0); + TEST_CHECK(region.fd == fileno(file)); + TEST_CHECK(region.offset == 1234 && region.size == 65536); + TEST_CHECK(region.lock.owns_lock() && region.lock.mutex() == &fi.mutex); + TEST_CHECK(!fi.mutex.try_lock()); + + region = {}; + TEST_CHECK(fi.mutex.try_lock()); + fi.mutex.unlock(); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,68928) == 0); + TEST_CHECK(region.offset == 68928 && region.size == 131072); + + region = {}; + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,68929) > 0); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,150000) > 0); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,200000) > 0); + + TEST_CHECK(!region.lock.owns_lock()); + TEST_CHECK(fi.mutex.try_lock()); + fi.mutex.unlock(); + + FILE *empty = tmpfile(); + TEST_ASSERT(empty != nullptr); + TestHandle eh(fileno(empty),"/read-buf-empty"); + TEST_CHECK(FUSE::read_buf(&eh.ctx,&eh.ffi,®ion,131072,0) > 0); + fclose(empty); + + region = {}; + FUSE::read_buf_rejected(&ctx,&ffi); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,0) > 0); + + fuse_file_info_t bad = {}; + TEST_CHECK(FUSE::read_buf(&ctx,&bad,®ion,65536,0) == -EBADF); + fclose(file); +} + +/* read_buf never waits for the handle's lock: while moveonenospc holds + it, or waits for it, a read declines to the copy path (which reads + the old fd, as it always has) rather than pinning a worker thread for + the whole move, or holding the mover off. */ +void +test_read_buf_never_waits_for_mover() +{ + if(::_read_buf_refuses_tmp()) + { + TEST_SKIP("read_buf refuses this /tmp filesystem"); + return; + } + + static char data[200000]; + + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + TestHandle h(fileno(file),"/read-buf-mover"); + FileInfo &fi = h.fi; + const fuse_req_ctx_t &ctx = h.ctx; + const fuse_file_info_t &ffi = h.ffi; + FileRegion region; + + fi.mutex.lock(); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,65536,0) > 0); + fi.mutex.unlock(); + + fi.exclusive_waiters.store(1); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,65536,0) > 0); + fi.exclusive_waiters.store(0); + + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,65536,0) == 0); + TEST_CHECK(region.lock.owns_lock()); + region = {}; + fclose(file); +} + +/* An O_DIRECT branch fd is never offered, whatever its filesystem: the + kernel would serve that splice by direct I/O into fresh pages, saving + no copy over the copy path, and a misaligned request's EINVAL would + look like a filesystem without splice_read. */ +void +test_read_buf_declines_odirect() +{ + if(::_read_buf_refuses_tmp()) + { + TEST_SKIP("read_buf refuses this /tmp filesystem"); + return; + } + + FILE *file = tmpfile(); + char path[64]; + int dfd = -1; + if(file != nullptr) + { + snprintf(path,sizeof(path),"/proc/self/fd/%d",fileno(file)); + dfd = open(path,O_RDONLY|O_DIRECT); + } + if(dfd < 0) + { + if(file != nullptr) + fclose(file); + TEST_SKIP("no O_DIRECT tmpfile here"); + return; + } + + static char data[200000]; + + TEST_ASSERT(pwrite(fileno(file),data,sizeof(data),0) == sizeof(data)); + TestHandle h(dfd,"/read-buf-odirect"); + FileRegion region; + + TEST_CHECK(FUSE::read_buf(&h.ctx,&h.ffi,®ion,65536,0) > 0); + TEST_CHECK(h.fi.splice_state.load() == FileInfo::Splice::no); + + close(dfd); + fclose(file); +} + +/* The size floor lets read_buf skip its size check. A shrink outside + this handle leaves it stale, so read_buf still offers the whole + window (the reply then hits the end and falls back, see + reply_splice_eof_falls_back_without_sending); a short FUSE::read + lowers it to where that read found the end, and an ftruncate + through the handle resets it. */ +void +test_read_buf_size_floor_follows_shrink() +{ + if(::_read_buf_refuses_tmp()) + { + TEST_SKIP("read_buf refuses this /tmp filesystem"); + return; + } + + static char data[400000]; + + FILE *file = ::_tmpfile_with(data,sizeof(data)); + TEST_ASSERT(file != nullptr); + TestHandle h(fileno(file),"/read-buf-floor"); + FileInfo &fi = h.fi; + const fuse_req_ctx_t &ctx = h.ctx; + const fuse_file_info_t &ffi = h.ffi; + FileRegion region; + + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,0) == 0); + region = {}; + TEST_ASSERT(ftruncate(fileno(file),100000) == 0); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,131072) == 0); + TEST_CHECK(region.size == 131072); + region = {}; + + static char buf[131072]; + TEST_CHECK(FUSE::read(&ctx,&ffi,buf,sizeof(buf),0) == 100000); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,0) > 0); + + TEST_ASSERT(pwrite(fileno(file),data,sizeof(data),0) == sizeof(data)); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,131072) == 0); + TEST_CHECK(region.size == 131072); + region = {}; + + TEST_CHECK(FUSE::ftruncate(&ctx,fi.to_fh(),200000) == 0); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,131072) > 0); + TEST_CHECK(FUSE::read_buf(&ctx,&ffi,®ion,131072,65536) == 0); + TEST_CHECK(region.size == 131072); + fclose(file); +} + +/* A process whose seccomp policy refuses splice (answering EPERM, as + libseccomp, Docker and systemd policies commonly do, rather than + ENOSYS) gets that error from every branch splice. It must turn the + reply splice path off, as a kernel without splice does, instead of + paying for a staging attempt on every large read. Runs in a child: + the filter and the off switch are both process-wide. */ +void +test_reply_splice_seccomp_refusal_disables() +{ + /* Where the reply pipe cannot grow, the child's staging fails with + EPERM before the filter is involved; that is recorded as a + pipe-max-size bound, which also turns the splice path off, so the + test would pass without testing the seccomp refusal at all. */ + if(!::_pipe_can_grow(262144)) + { + TEST_SKIP("cannot grow a pipe to 256KiB here"); + return; + } + + pid_t child = fork(); + if(child == 0) + { + alarm(5); + FILE *file = tmpfile(); + FILE *dev = tmpfile(); + if((file == nullptr) || (dev == nullptr)) + _exit(77); + static char data[200000]; + + if(pwrite(fileno(file),data,sizeof(data),0) != (ssize_t)sizeof(data)) + _exit(77); + if(!fuse_reply_splice_fits(0,131072)) + _exit(77); + + struct sock_filter filter[] = + { + BPF_STMT(BPF_LD|BPF_W|BPF_ABS,offsetof(struct seccomp_data,nr)), + BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K,__NR_splice,0,1), + BPF_STMT(BPF_RET|BPF_K,SECCOMP_RET_ERRNO|(EPERM & SECCOMP_RET_DATA)), + BPF_STMT(BPF_RET|BPF_K,SECCOMP_RET_ALLOW), + }; + struct sock_fprog prog = {(unsigned short)(sizeof(filter) / sizeof(filter[0])),filter}; + if((prctl(PR_SET_NO_NEW_PRIVS,1,0,0,0) != 0) || + (syscall(SYS_seccomp,SECCOMP_SET_MODE_FILTER,0,&prog) != 0)) + _exit(77); + + fuse_req_t *req = fuse_req_alloc(); + req->fd = fileno(dev); + req->ctx.unique = 3; + FileRegion region{fileno(file),0,131072,{}}; + const FuseReplyResult r = fuse_reply_data_splice_fd(req,region); + fuse_req_free(req); + + struct stat st; + const bool correct = ((r == FuseReplyResult::fallback) && + (fstat(fileno(dev),&st) == 0) && + (st.st_size == 0) && + !fuse_reply_splice_fits(0,131072)); + _exit(correct ? 0 : 1); + } + + /* No check before this point: acutest cannot skip after one. */ + int status = 0; + pid_t waited = -1; + if(child > 0) + do { waited = waitpid(child,&status,0); } while(waited < 0 && errno == EINTR); + if((child > 0) && (waited == child) && WIFEXITED(status) && (WEXITSTATUS(status) == 77)) + { + TEST_SKIP("cannot install a seccomp filter or splice here"); + return; + } + TEST_CHECK((child > 0) && (waited == child) && + WIFEXITED(status) && (WEXITSTATUS(status) == 0)); +} + +/* A writeback request must never end in a short success: Linux's + fuse_writepage_end consults only the error, so a short success would + mark the unwritten tail clean and lose it. RLIMIT_FSIZE=4096 makes an + 8KiB direct_io write complete 4096 bytes with no error. An ordinary + request reports that prefix. A writeback request keeps writing, as on + a cached handle (which is also how moveonenospc gets to run when a + branch fills mid-write), and reports the error the remainder gets + (EFBIG). The file holds exactly the 4096-byte prefix either way. */ +void +test_write_short_success_writeback() +{ + for(int writepage = 0; writepage <= 1; ++writepage) + { + pid_t child = fork(); + TEST_ASSERT(child >= 0); + if(child == 0) + { + alarm(5); + signal(SIGXFSZ,SIG_IGN); + struct rlimit limit = {4096,4096}; + if(setrlimit(RLIMIT_FSIZE,&limit) != 0) + _exit(1); + FILE *file = tmpfile(); + if(file == nullptr) + _exit(2); + char payload[8192]; + memset(payload,0x5a,sizeof(payload)); + Branch branch; + FileInfo fi(fileno(file),branch,fs::path("/write-short-test"),true); + fuse_req_ctx_t ctx = {}; + fuse_file_info_t ffi = {}; + ffi.fh = fi.to_fh(); + ffi.writepage = writepage; + const int rv = FUSE::write(&ctx,&ffi,payload,sizeof(payload),0); + struct stat st; + char stored[4096]; + bool correct = (rv == (writepage ? -EFBIG : 4096)) && + fstat(fileno(file),&st) == 0 && st.st_size == 4096 && + pread(fileno(file),stored,sizeof(stored),0) == sizeof(stored) && + memcmp(stored,payload,sizeof(stored)) == 0; + fclose(file); + _exit(correct ? 0 : 3); + } + int status; + pid_t waited; + do { waited = waitpid(child,&status,0); } while(waited < 0 && errno == EINTR); + TEST_CHECK(waited == child && WIFEXITED(status) && WEXITSTATUS(status) == 0); + } +} +#endif + void test_config_bool() { @@ -3650,6 +4348,20 @@ test_rapidhash_withSeed_preserves_default_output() TEST_LIST = { {"nop",test_nop}, +#if defined(__linux__) + {"reply_splice_sends_exact_message",test_reply_splice_sends_exact_message}, + {"reply_splice_eof_falls_back_without_sending",test_reply_splice_eof_falls_back_without_sending}, + {"reply_splice_small_region_falls_back",test_reply_splice_small_region_falls_back}, + {"reply_splice_unspliceable_fd_is_unsupported",test_reply_splice_unspliceable_fd_is_unsupported}, + {"reply_splice_releases_lock_before_submit",test_reply_splice_releases_lock_before_submit}, + {"reply_splice_debug_throw_leaves_pipe_empty",test_reply_splice_debug_throw_leaves_pipe_empty}, + {"read_buf_region_contract",test_read_buf_region_contract}, + {"read_buf_never_waits_for_mover",test_read_buf_never_waits_for_mover}, + {"read_buf_declines_odirect",test_read_buf_declines_odirect}, + {"read_buf_size_floor_follows_shrink",test_read_buf_size_floor_follows_shrink}, + {"reply_splice_seccomp_refusal_disables",test_reply_splice_seccomp_refusal_disables}, + {"write_short_success_writeback",test_write_short_success_writeback}, +#endif {"config_bool",test_config_bool}, {"config_uint64",test_config_uint64}, {"config_int",test_config_int}, diff --git a/vendored/libfuse/include/fuse.h b/vendored/libfuse/include/fuse.h index c97d2062..eb823911 100644 --- a/vendored/libfuse/include/fuse.h +++ b/vendored/libfuse/include/fuse.h @@ -22,6 +22,9 @@ #include #include +/* Transport type from fuse_io.hpp (C++); opaque to C consumers. */ +struct FileRegion; + EXTERN_C_BEGIN /* ----------------------------------------------------------- * @@ -206,6 +209,26 @@ struct fuse_operations char *buf, size_t size, off_t off); + /* Optional. For large reads on Linux: describe the reply as a region + of a branch fd instead of filling a buffer, so libfuse can splice + it into /dev/fuse. Called only for requests whose reply could be + spliced (fuse_reply_splice_fits). Return 0 with region filled to + splice it, a positive value to decline (libfuse then calls read()), + or -errno to fail the request. When returning 0, region->size MUST + equal size: the reply's length is taken from region->size alone, + so a smaller, non-EOF region would be sent as a short reply + indistinguishable from a real short read. To offer less than the + whole request, decline instead and let read() serve it. libfuse + also falls back to read() whenever the splice cannot be staged, + and calls read_buf_rejected when the region's fd refuses splice so + the filesystem can stop offering that open file. */ + int (*read_buf)(const fuse_req_ctx_t *, + const fuse_file_info_t *ffi, + struct FileRegion *region, + size_t size, + off_t off); + void (*read_buf_rejected)(const fuse_req_ctx_t *, + const fuse_file_info_t *ffi); int (*fallocate)(const fuse_req_ctx_t *, const uint64_t, int, diff --git a/vendored/libfuse/include/fuse_io.hpp b/vendored/libfuse/include/fuse_io.hpp new file mode 100644 index 00000000..38483256 --- /dev/null +++ b/vendored/libfuse/include/fuse_io.hpp @@ -0,0 +1,45 @@ +/* + ISC License + + Copyright (c) 2026, Antonio SJ Musumeci + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ + +#pragma once + +#include +#include +#include + +/* A borrowed, descriptor-referenced region of a branch file that a read + reply can splice into /dev/fuse without the data crossing userspace. + The filesystem fills it; libfuse never owns or closes the fd. + + The reply splices from fd several times, so the filesystem may park + a shared lock in lock to keep fd bound to the same file until every + read from fd is done (staged into the reply). The reply path + releases it as soon as that staging finishes, whether or not the + reply is then successfully delivered: fd is not read again either + way, and once the READ is answered the kernel may send RELEASE, so + the filesystem may free what the lock lives in. A fallback reached + instead DURING staging (fd itself declined or ran out of data) + leaves the lock held; it is released when the region goes out of + scope, before any fallback read. */ +struct FileRegion +{ + int fd = -1; + off_t offset = 0; + size_t size = 0; + std::shared_lock lock; +}; diff --git a/vendored/libfuse/include/fuse_lowlevel.h b/vendored/libfuse/include/fuse_lowlevel.h index 3288d9d5..73b57ebd 100644 --- a/vendored/libfuse/include/fuse_lowlevel.h +++ b/vendored/libfuse/include/fuse_lowlevel.h @@ -5,6 +5,17 @@ #include "fuse_kernel.h" #include "fuse_req.hpp" +#ifdef __cplusplus +#include "fuse_io.hpp" + +/* Outcome of an fd-backed reply. fallback: nothing reached the kernel, + reply another way. unsupported: same, and this descriptor rejects + splice (EINVAL), so do not offer it again. sent: done. failed: a + submission error was already reported to the kernel; never reply + again. */ +enum class FuseReplyResult { fallback, unsupported, sent, failed }; +#endif + #include #include #include @@ -288,6 +299,31 @@ int fuse_reply_data(fuse_req_t *req, char *buf, size_t bufsize); +#ifdef __cplusplus +/* READ replies at or above this size are offered to the splice path. + Measured on Linux 5.14 (tmpfs branch, release build) the splice + path uses less daemon CPU than read+writev from roughly 48KiB per + request; 64KiB keeps a margin. */ +constexpr size_t FUSE_SPLICE_REPLY_MIN = 65536; + +/* Whether a READ reply for size bytes at offset would currently be + attempted by fuse_reply_data_splice_fd: false below + FUSE_SPLICE_REPLY_MIN, once splice is unavailable, or when the reply + needs a pipe larger than pipe-max-size allows. The high-level READ + handler asks read_buf for a region only when this holds, so replies + that could only fall back cost no work in the filesystem. */ +bool fuse_reply_splice_fits(off_t offset, size_t size); + +/* Reply to a READ with a region of a file, spliced into /dev/fuse + without the data crossing userspace (Linux only; elsewhere, and + whenever the splice path is unavailable, returns fallback). Borrows + req: the caller frees it after a sent/failed outcome and must reply + by other means after fallback. Releases region.lock once the data + is staged, before the reply can reach the kernel. */ +FuseReplyResult fuse_reply_data_splice_fd(fuse_req_t *req, + FileRegion ®ion); +#endif + /** * Reply with data vector * diff --git a/vendored/libfuse/lib/fuse.cpp b/vendored/libfuse/lib/fuse.cpp index ea875882..296f016a 100644 --- a/vendored/libfuse/lib/fuse.cpp +++ b/vendored/libfuse/lib/fuse.cpp @@ -2328,22 +2328,14 @@ fuse_lib_open(fuse_req_t *req_, static void -fuse_lib_read(fuse_req_t *req_, - struct fuse_in_header *hdr_) +fuse_lib_read_mem(fuse_req_t *req_, + const fuse_file_info_t *ffi_, + const size_t size_, + const off_t offset_) { int res; - fuse_file_info_t ffi = {}; - struct fuse_read_in *arg; fuse_msgbuf_t *msgbuf; - arg = (fuse_read_in*)fuse_hdr_arg(hdr_); - ffi.fh = arg->fh; - if(req_->conn.proto_minor >= 9) - { - ffi.flags = arg->flags; - ffi.lock_owner = arg->lock_owner; - } - msgbuf = msgbuf_alloc_page_aligned(); if(msgbuf == NULL) { @@ -2352,10 +2344,10 @@ fuse_lib_read(fuse_req_t *req_, } res = f.ops.read(&req_->ctx, - &ffi, + ffi_, msgbuf->mem, - arg->size, - arg->offset); + size_, + offset_); if(res >= 0) fuse_reply_data(req_,msgbuf->mem,res); @@ -2365,6 +2357,84 @@ fuse_lib_read(fuse_req_t *req_, msgbuf_free(msgbuf); } +/* Large reads: ask the filesystem for a branch-file region and splice + it into /dev/fuse. Every staging failure falls back to the memory + path above, which is always correct. A failure of the final + submission splice is answered with an error-only reply and never + replayed: the kernel may already have consumed the payload. The + region's lock is released by the reply path once the data is staged; + after sent/failed only libfuse's own req is touched here, since the + handle may already be released. */ +static +bool +fuse_lib_read_splice(fuse_req_t *req_, + const fuse_file_info_t *ffi_, + const size_t size_, + const off_t offset_) +{ + int res; + FileRegion region; + + res = f.ops.read_buf(&req_->ctx,ffi_,®ion,size_,offset_); + if(res < 0) + { + fuse_reply_err(req_,res); + return true; + } + if(res > 0) + return false; + + /* The reply length comes from region.size alone (see read_buf in + fuse.h): a region smaller than the request would be sent as a short + reply, which the kernel treats as end-of-file. Decline such a region + instead; its destructor releases the lock before the copy path. */ + if(region.size != size_) + return false; + + switch(fuse_reply_data_splice_fd(req_,region)) + { + case FuseReplyResult::sent: + case FuseReplyResult::failed: + fuse_req_free(req_); + return true; + case FuseReplyResult::unsupported: + if(f.ops.read_buf_rejected) + f.ops.read_buf_rejected(&req_->ctx,ffi_); + return false; + case FuseReplyResult::fallback: + break; + } + + return false; +} + +static +void +fuse_lib_read(fuse_req_t *req_, + struct fuse_in_header *hdr_) +{ + fuse_file_info_t ffi = {}; + struct fuse_read_in *arg; + + arg = (fuse_read_in*)fuse_hdr_arg(hdr_); + ffi.fh = arg->fh; + if(req_->conn.proto_minor >= 9) + { + ffi.flags = arg->flags; + ffi.lock_owner = arg->lock_owner; + } + + /* A region is always the whole request, so whether its reply can be + spliced at all (the threshold, the pipe limit) is known here, + before the filesystem does any work for it. */ + if((f.ops.read_buf != nullptr) && + fuse_reply_splice_fits(arg->offset,arg->size) && + fuse_lib_read_splice(req_,&ffi,arg->size,arg->offset)) + return; + + fuse_lib_read_mem(req_,&ffi,arg->size,arg->offset); +} + static void fuse_lib_write(fuse_req_t *req_, diff --git a/vendored/libfuse/lib/fuse_lowlevel.cpp b/vendored/libfuse/lib/fuse_lowlevel.cpp index d754fe85..7fa3a66d 100644 --- a/vendored/libfuse/lib/fuse_lowlevel.cpp +++ b/vendored/libfuse/lib/fuse_lowlevel.cpp @@ -22,6 +22,14 @@ #include "fuse_opt.h" #include "fuse_pollhandle.h" #include "stat_utils.h" +#include "syslog.hpp" + +#include + +#if defined(__linux__) +#include +#include +#endif #include #include @@ -420,6 +428,493 @@ fuse_reply_buf(fuse_req_t *req, return send_reply_ok(req, buf, size); } +#if defined(__linux__) +/* Automatic splice of large READ replies (branch fd -> pipe -> /dev/fuse). + The copy path costs read + pread + writev and copies the payload + twice. The splice path costs read + write(header) + splice(fd -> + pipe) + splice(pipe -> /dev/fuse) and never copies the payload into + userspace; the kernel copies it once, into the request's pages. The + caller gates on fuse_reply_splice_fits (fuse_lowlevel.h). + + Every staging failure is recoverable: nothing has reached the kernel, + so the pipe is drained and the caller replies through the copy path. + Only a kernel without splice (ENOSYS), or a seccomp policy refusing + it, disables the path for the whole process. A pipe-max-size too + small for a reply only caps the reply size that is spliced + (g_reply_pipe_limit), and a branch that rejects splice is reported + per open file (FuseReplyResult::unsupported). */ +static std::atomic g_reply_splice_off{false}; + +/* Smallest reply pipe, in bytes, found to exceed the pipe-max-size + sysctl, which caps unprivileged F_SETPIPE_SZ; a 1MiB reply plus its + header page needs one page more than the default 1MiB. Replies that + need at least this much stay on the copy path unless this thread's + pipe already holds them; smaller replies keep splicing. A refusal + from the per-user pipe quota is not recorded: it lifts as pipes are + closed. 0 = no refusal seen. */ +static std::atomic g_reply_pipe_limit{0}; + +/* The pipe-max-size sysctl, read once at session setup. Read lazily, on + the first refused resize, it could fail for the same shortage of + descriptors that made that resize or pipe fail and then read as 0 for + good. 0 when it cannot be read, in which case every EPERM is treated + as that limit. */ +static std::atomic g_pipe_max_size{0}; + +static +void +_read_pipe_max_size(void) +{ + unsigned long n = 0; + FILE *f = ::fopen("/proc/sys/fs/pipe-max-size","re"); + + if(f == NULL) + return; + if(::fscanf(f,"%lu",&n) == 1) + g_pipe_max_size.store((size_t)n,std::memory_order_relaxed); + ::fclose(f); +} + +static +void +_reply_splice_disable(const char *what_, + const int err_) +{ + if(g_reply_splice_off.exchange(true)) + return; + SysLog::notice("read reply splice disabled: {} failed: {};" + " using copy path", + what_,strerror(err_)); +} + +/* One latch per distinct failure reason: these are independent, + recurring conditions (a pipe quota hiccup, a header-staging error, a + branch-splice error, a submission error, ...), so silencing all of + them after whichever happens to fire first would hide later, + unrelated failures from the log. A submission the kernel refused + before it looked at the request (the copy path then answers it) is + a different incident from one that fails the request, so they do + not share a latch. */ +enum class SpliceWarnReason +{ + pipe_quota, + pipe_create, + header_stage, + branch_splice, + submit_fallback, + submit_failed, + _count +}; + +static std::atomic g_reply_splice_warned[(size_t)SpliceWarnReason::_count] = {}; + +static +void +_reply_splice_warn(const char *what_, + const int err_, + const SpliceWarnReason reason_, + const char *outcome_ = + "falling back to copy path for this reply") +{ + if(g_reply_splice_warned[(size_t)reason_].exchange(true)) + return; + SysLog::warning("read reply splice: {} failed: {}; {}", + what_,strerror(err_),outcome_); +} + +/* pipe_quota (the per-user pipe count/memory limit) is the one reason + that is genuinely transient -- it lifts as pipes are closed, unlike + the others, which are properties of the kernel or a branch fd that + do not change at runtime. Clear its latch once a resize succeeds, so + a later recurrence (a distinct incident) is logged again instead of + being silenced forever by whichever incident fired first. */ +static +void +_reply_splice_warn_clear(const SpliceWarnReason reason_) +{ + g_reply_splice_warned[(size_t)reason_].store(false,std::memory_order_relaxed); +} + +/* Whether splice itself is unusable in this process: missing from the + kernel (ENOSYS) or refused by a seccomp policy, which commonly + answers EPERM but may use any errno. A zero-length splice returns 0 + before looking at its descriptors, so only then does it fail. An + error from splicing a branch file normally comes from that file's + own read path instead (ENOSYS from a FUSE branch whose server does + not implement READ, EIO, ...), which this tells apart. */ +static +bool +_splice_syscall_unusable() +{ + return (::splice(-1,nullptr,-1,nullptr,0,0) == -1); +} + +static +void +fuse_send_errno(const int fd_, + const int errno_, + const uint64_t unique_id_); + +namespace +{ + /* One pipe per worker thread, created on first use and reused for + every large reply that thread sends. Empty between replies: every + path that leaves bytes behind drains or closes it. */ + struct ReplyPipe + { + int fd[2] = {-1,-1}; + size_t cap = 0; + + ~ReplyPipe() { reset(); } + + void + reset() + { + if(fd[0] != -1) ::close(fd[0]); + if(fd[1] != -1) ::close(fd[1]); + fd[0] = fd[1] = -1; + cap = 0; + } + + /* 0 when a pipe of at least bytes_ capacity is held, else -errno. */ + int + ready(const size_t bytes_) + { + if(fd[0] == -1) + { + /* The initial capacity is not queried: cap 0 makes the + first use resize to exactly what it needs, which with 4KiB + pages it would have to do anyway (a reply of 64KiB plus + its header needs more than the default 16 slots). */ + if(::pipe2(fd,O_CLOEXEC|O_NONBLOCK) == -1) + return -errno; + } + if(bytes_ <= cap) + return 0; + if(bytes_ > (size_t)INT_MAX) + return -EINVAL; + const int c = ::fcntl(fd[0],F_SETPIPE_SZ,(int)bytes_); + if(c == -1) + return -errno; + cap = (size_t)c; + return ((cap >= bytes_) ? 0 : -ENOSPC); + } + + /* /dev/null, opened on first use and kept. A failed open (EMFILE at + that moment, say) is retried by the next drain rather than + cached, which would leave every later drain copying. */ + static + int + devnull() + { + static std::atomic g_fd{-1}; + int fd = g_fd.load(std::memory_order_acquire); + + if(fd != -1) + return fd; + + fd = ::open("/dev/null",O_WRONLY|O_CLOEXEC); + if(fd == -1) + return -1; + + int expected = -1; + if(g_fd.compare_exchange_strong(expected,fd,std::memory_order_acq_rel)) + return fd; + + ::close(fd); + + return expected; + } + + /* Discard anything staged. Splicing it into /dev/null drops the + page references without copying the data into userspace; reading + it out is the fallback when that is not available (no /dev/null, + a buffer error), so the pipe/capacity survive to be reused. Both + ends are non-blocking, so each stops at empty (EAGAIN); a pipe + that cannot be emptied by either is closed instead, and ready() + makes a new one on next use. */ + void + drain() + { + /* true once the pipe reads empty (EAGAIN), false on any other + outcome. */ + const auto pump = [](auto op_) + { + for(;;) + { + const ssize_t rv = op_(); + if((rv > 0) || ((rv == -1) && (errno == EINTR))) + continue; + return ((rv == -1) && (errno == EAGAIN)); + } + }; + + const int devnull = ReplyPipe::devnull(); + if((devnull != -1) && + pump([&]{ return ::splice(fd[0],nullptr,devnull,nullptr, + (size_t)1 << 30,SPLICE_F_NONBLOCK); })) + return; + + char buf[16384]; + if(pump([&]{ return ::read(fd[0],buf,sizeof(buf)); })) + return; + + reset(); + } + }; +} + +static thread_local ReplyPipe t_reply_pipe; + +/* Pipe capacity a reply needs: one slot for the header plus the file + range, which may start mid-page. The kernel fills pipes one page per + slot, so capacity is counted in pages, not bytes. */ +static +size_t +_reply_pipe_need(const off_t offset_, + const size_t size_) +{ + const size_t slots = + (((uintptr_t)offset_ % pagesize) + size_ + pagesize - 1) / pagesize; + + return (1 + slots) * pagesize; +} + +/* Whether a pipe of need_ bytes may be used: always below any recorded + pipe-max-size refusal, and above it only when this thread's pipe + already has that capacity (so no resize is attempted). */ +static +bool +_reply_pipe_allowed(const size_t need_) +{ + const size_t limit = g_reply_pipe_limit.load(std::memory_order_relaxed); + return ((limit == 0) || (need_ < limit) || (need_ <= t_reply_pipe.cap)); +} + +/* The pipe capacity a reply of size_ bytes at offset_ needs, or 0 when + it is not spliced: splice is off, the reply is below the threshold + (cheaper through read + writev), or it needs a pipe over a recorded + pipe-max-size refusal. The one predicate behind both + fuse_reply_splice_fits and fuse_reply_data_splice_fd. */ +static +size_t +_reply_splice_need(const off_t offset_, + const size_t size_) +{ + if(g_reply_splice_off.load(std::memory_order_relaxed) || + (size_ < FUSE_SPLICE_REPLY_MIN)) + return 0; + + const size_t need = _reply_pipe_need(offset_,size_); + + return (_reply_pipe_allowed(need) ? need : 0); +} + +bool +fuse_reply_splice_fits(const off_t offset_, + const size_t size_) +{ + return (_reply_splice_need(offset_,size_) != 0); +} + +FuseReplyResult +fuse_reply_data_splice_fd(fuse_req_t *req_, + FileRegion ®ion_) +{ + const size_t need = _reply_splice_need(region_.offset,region_.size); + if(need == 0) + return FuseReplyResult::fallback; + + ReplyPipe &p = t_reply_pipe; + struct fuse_out_header out = {}; + out.unique = req_->ctx.unique; + out.len = sizeof(out) + region_.size; + + const size_t prev_cap = p.cap; + int err = p.ready(need); + if(err != 0) + { + /* EPERM means either larger than pipe-max-size without + CAP_SYS_RESOURCE, a fixed bound worth remembering so replies + this large skip straight to the copy path, or over the + per-user pipe quota, which is transient: only this reply falls + back. An unreadable sysctl is treated as the fixed bound. */ + const size_t max = g_pipe_max_size.load(std::memory_order_relaxed); + if((err == -EPERM) && ((max == 0) || (need > max))) + { + size_t cur = g_reply_pipe_limit.load(std::memory_order_relaxed); + while(((cur == 0) || (need < cur)) && + !g_reply_pipe_limit.compare_exchange_weak(cur,need)) + ; + /* cur is still 0 only in the thread that set the first limit. */ + if(cur == 0) + SysLog::notice("read reply splice: a {} byte pipe exceeds" + " pipe-max-size; reads this large use the" + " copy path, smaller reads are still spliced", + need); + } + else if(err == -EPERM) + _reply_splice_warn("growing the reply pipe (per-user pipe quota)",-err, + SpliceWarnReason::pipe_quota); + else if(err == -ENOSYS) + _reply_splice_disable("creating the reply pipe",-err); + else + _reply_splice_warn("creating the reply pipe",-err, + SpliceWarnReason::pipe_create); + return FuseReplyResult::fallback; + } + + /* A resize just succeeded, so any earlier per-user pipe quota + shortage (the one transient SpliceWarnReason) has lifted; let a + later recurrence be logged again instead of staying silenced by an + earlier one. A pipe that was already big enough says nothing about + the quota: it must not re-arm the warning while other threads still + fail to grow theirs, nor write the shared latch on every reply. */ + if(p.cap != prev_cap) + _reply_splice_warn_clear(SpliceWarnReason::pipe_quota); + + /* write(), not vmsplice(): vmsplice without SPLICE_F_GIFT queues a + reference to this stack page rather than a copy, and the pipe + outlives the frame whenever it is consumed later (or not at all). + A 16-byte write into an empty pipe copies and is atomic. */ + ssize_t n; + do + n = ::write(p.fd[1],&out,sizeof(out)); + while((n == -1) && (errno == EINTR)); + if(n != (ssize_t)sizeof(out)) + { + err = ((n == -1) ? errno : EIO); + p.drain(); + _reply_splice_warn("staging the reply header",err, + SpliceWarnReason::header_stage); + return FuseReplyResult::fallback; + } + + size_t done = 0; + while(done < region_.size) + { + loff_t pos = region_.offset + (off_t)done; + n = ::splice(region_.fd,&pos,p.fd[1],nullptr, + region_.size - done,SPLICE_F_NONBLOCK); + if((n == -1) && (errno == EINTR)) + continue; + if(n > 0) + { + done += (size_t)n; + continue; + } + + /* n == 0: the file ended before the region did (it shrank after + the caller sized the region). n == -1: this descriptor rejects + splice (EINVAL: its filesystem has no splice_read) or failed. + EAGAIN is also end-of-file when the file ended inside the + region's last page of an exactly full pipe (no room is left + for the splice that would return 0). Nothing reached the + kernel. A rejecting descriptor is a property of that branch + file, not of the transport, so the path stays on and the + caller is told to stop offering it; only a first splice can + show that, since one that already moved data proves the + descriptor splices. An error disables the path only when + splice itself is unusable (missing, or refused by seccomp), + not when it is the branch file's read error; otherwise every + large read would pay for a staging attempt that cannot + succeed. */ + err = ((n == -1) ? errno : 0); + p.drain(); + if((err != 0) && (err != EAGAIN)) + { + if(_splice_syscall_unusable()) + _reply_splice_disable("splicing from the branch file",err); + else if(err != EINVAL) + _reply_splice_warn("splicing from the branch file",err, + SpliceWarnReason::branch_splice); + } + return (((err == EINVAL) && (done == 0)) ? + FuseReplyResult::unsupported : + FuseReplyResult::fallback); + } + + /* fd is not read again on any outcome below (see FileRegion::lock). */ + if(region_.lock.owns_lock()) + region_.lock.unlock(); + + /* The whole message is staged now, so a log write that throws (fmt on + a full or broken log file) must not leave it in this thread's pipe: + the worker survives the exception and would splice it in front of + its next reply. */ + if(fuse_cfg.debug) + { + try + { + fuse_debug_data_out(req_->ctx.unique,region_.size); + } + catch(...) + { + p.drain(); + throw; + } + } + + /* Pages spliced in from the branch page cache are never offered for + SPLICE_F_MOVE: stealing them would evict the branch's cache pages. + The kernel copies them into the request's pages instead. */ + const size_t total = out.len; + do + n = ::splice(p.fd[0],nullptr,req_->fd,nullptr,total,0); + while((n == -1) && (errno == EINTR)); + if(n == (ssize_t)total) + return FuseReplyResult::sent; + + err = ((n == -1) ? errno : EIO); + + /* The kernel takes the whole message or none of it. If the pipe still + holds all of it, the submission failed before the kernel looked at + the request (e.g. ENOMEM allocating its buffer list), which is + still pending, so the copy path can answer it. */ + int queued = 0; + if((::ioctl(p.fd[0],FIONREAD,&queued) == 0) && ((size_t)queued == total)) + { + p.drain(); + _reply_splice_warn("submitting the reply",err, + SpliceWarnReason::submit_fallback); + return FuseReplyResult::fallback; + } + + /* Otherwise the kernel may have completed the request before failing + (it validates after copying; a copy error fails it with EIO), so + the payload is never replayed. ENOENT means the request was already + retired or the connection aborted: nothing is left to answer, as + when the copy path's writev fails the same way, so it is neither + logged nor answered. Any other error-only reply completes a request + that is still pending. */ + p.drain(); + if(err != ENOENT) + { + _reply_splice_warn("submitting the reply",err, + SpliceWarnReason::submit_failed, + "failing the request"); + + fuse_send_errno(req_->fd,err,req_->ctx.unique); + } + + return FuseReplyResult::failed; +} +#else +bool +fuse_reply_splice_fits(const off_t, + const size_t) +{ + return false; +} + +FuseReplyResult +fuse_reply_data_splice_fd(fuse_req_t *, + FileRegion &) +{ + return FuseReplyResult::fallback; +} +#endif + int fuse_reply_data(fuse_req_t *req, char *buf_, @@ -1726,6 +2221,10 @@ fuse_lowlevel_new_common(struct fuse_args *args, memcpy(&f.op,op,op_size); f.owner = getuid(); +#if defined(__linux__) + _read_pipe_max_size(); +#endif + se = fuse_session_new(&f, (void*)fuse_ll_buf_receive_read, (void*)fuse_ll_buf_process_read_init,