mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-11-19 22:46:34 -05:00
Add support for FUSE passthrough
This commit is contained in:
parent
92312c8507
commit
e310d24615
412 changed files with 43707 additions and 17432 deletions
|
|
@ -14,117 +14,117 @@ config file.
|
|||
|
||||
## types
|
||||
|
||||
- BOOL = 'true' | 'false'
|
||||
- INT = [MIN_INT,MAX_INT]
|
||||
- UINT = [0,MAX_INT]
|
||||
- SIZE = 'NNM'; NN = INT, M = 'B' | 'K' | 'M' | 'G' | 'T'
|
||||
- PAGESIZE = UINT (representing number of pages) | SIZE (in bytes
|
||||
* BOOL = 'true' | 'false'
|
||||
* INT = [MIN_INT,MAX_INT]
|
||||
* UINT = [0,MAX_INT]
|
||||
* SIZE = 'NNM'; NN = INT, M = 'B' | 'K' | 'M' | 'G' | 'T'
|
||||
* PAGESIZE = UINT (representing number of pages) | SIZE (in bytes
|
||||
which will be converted to pages)
|
||||
- STR = string (may refer to an enumerated value, see details of
|
||||
* STR = string (may refer to an enumerated value, see details of
|
||||
argument)
|
||||
- FUNC = filesystem [function](functions_categories_policies.md)
|
||||
- CATEGORY = function [category](functions_categories_policies.md)
|
||||
- POLICY = mergerfs function [policy](functions_categories_policies.md)
|
||||
* FUNC = filesystem [function](functions_categories_policies.md)
|
||||
* CATEGORY = function [category](functions_categories_policies.md)
|
||||
* POLICY = mergerfs function [policy](functions_categories_policies.md)
|
||||
|
||||
|
||||
## mount options
|
||||
|
||||
- **config**: Path to a config file. Same arguments as below in
|
||||
* **config**: Path to a config file. Same arguments as below in
|
||||
key=val / ini style format.
|
||||
- **[branches](branches.md)**: Colon delimited list of branches. Used
|
||||
* **[branches](branches.md)**: Colon delimited list of branches. Used
|
||||
primarily in config file.
|
||||
- **[minfreespace](minfreespace.md)=SIZE**: The minimum available
|
||||
* **[minfreespace](minfreespace.md)=SIZE**: The minimum available
|
||||
space of a branch necessary to be considered for a create
|
||||
[policy](functions_categories_policies.md). This is a default value
|
||||
applied to all branches and can be overwritten when configuring
|
||||
[branches](branches.md). Understands 'K', 'M', and 'G' to represent
|
||||
kilobyte, megabyte, and gigabyte respectively. (default: 4G)
|
||||
- **[moveonenospc](moveonenospc.md)=BOOL|POLICY**: When enabled if a
|
||||
* **[moveonenospc](moveonenospc.md)=BOOL|POLICY**: When enabled if a
|
||||
**write** fails with **ENOSPC** (no space left on device) or
|
||||
**EDQUOT** (disk quota exceeded) the policy selected will run to
|
||||
find a new location for the file. An attempt to move the file to
|
||||
that branch will occur (keeping all metadata possible) and if
|
||||
successful the original is unlinked and the write retried. (default:
|
||||
pfrd)
|
||||
- **[inodecalc](inodecalc.md)=passthrough|path-hash|devino-hash|hybrid-hash**:
|
||||
* **[inodecalc](inodecalc.md)=passthrough|path-hash|devino-hash|hybrid-hash**:
|
||||
Selects the inode calculation algorithm. (default: hybrid-hash)
|
||||
- **dropcacheonclose=BOOL**: When a file is requested to be closed
|
||||
* **dropcacheonclose=BOOL**: When a file is requested to be closed
|
||||
call `posix_fadvise` on it first to instruct the kernel that we no
|
||||
longer need the data and it can drop its cache. Recommended when
|
||||
**cache.files=partial|full|auto-full|per-process** to limit double
|
||||
caching. (default: false)
|
||||
- **direct-io-allow-mmap=BOOL**: On newer kernels (>= 6.6) it is
|
||||
* **direct-io-allow-mmap=BOOL**: On newer kernels (>= 6.6) it is
|
||||
possible to disable file page caching while still allowing for
|
||||
shared mmap support. mergerfs will enable this feature if available
|
||||
but an option is provided to turn it off for testing and debugging
|
||||
purposes. (default: true)
|
||||
- **[symlinkify](symlinkify.md)=BOOL**: When enabled and a file is not
|
||||
* **[symlinkify](symlinkify.md)=BOOL**: When enabled and a file is not
|
||||
writable and its mtime or ctime is older than **symlinkify_timeout**
|
||||
files will be reported as symlinks to the original files. Please
|
||||
read more below before using. (default: false)
|
||||
- **[symlinkify_timeout](symlinkify.md)=UINT**: Time to wait, in
|
||||
* **[symlinkify_timeout](symlinkify.md)=UINT**: Time to wait, in
|
||||
seconds, to activate the **symlinkify** behavior. (default: 3600)
|
||||
- **nullrw=BOOL**: Turns reads and writes into no-ops. The request
|
||||
* **nullrw=BOOL**: Turns reads and writes into no-ops. The request
|
||||
will succeed but do nothing. Useful for benchmarking
|
||||
mergerfs. (default: false)
|
||||
- **lazy-umount-mountpoint=BOOL**: mergerfs will attempt to "lazy
|
||||
* **lazy-umount-mountpoint=BOOL**: mergerfs will attempt to "lazy
|
||||
umount" the mountpoint before mounting itself. Useful when
|
||||
performing live upgrades of mergerfs. May not work on
|
||||
FreeBSD. (default: false)
|
||||
- **ignorepponrename=BOOL**: Ignore path preserving on
|
||||
* **ignorepponrename=BOOL**: Ignore path preserving on
|
||||
rename. Typically rename and link act differently depending on the
|
||||
policy of `create` (read below). Enabling this will cause rename and
|
||||
link to always use the non-path preserving behavior. This means
|
||||
files, when renamed or linked, will stay on the same
|
||||
filesystem. (default: false)
|
||||
- **[export-support](export-support.md)=BOOL**: Sets a low-level FUSE
|
||||
* **[export-support](export-support.md)=BOOL**: Sets a low-level FUSE
|
||||
feature intended to indicate the filesystem can support being
|
||||
exported via NFS. (default: true)
|
||||
- **security_capability=BOOL**: If false return ENOATTR when xattr
|
||||
* **security_capability=BOOL**: If false return ENOATTR when xattr
|
||||
security.capability is queried. (default: true)
|
||||
- **[xattr](xattr.md)=passthrough|noattr|nosys**: Runtime control of
|
||||
* **[xattr](xattr.md)=passthrough|noattr|nosys**: Runtime control of
|
||||
xattrs. Default is to passthrough xattr requests. 'noattr' will
|
||||
short circuit as if nothing exists. 'nosys' will respond with ENOSYS
|
||||
as if xattrs are not supported or disabled. (default: passthrough)
|
||||
- **[link_cow](link_cow.md)=BOOL**: When enabled if a regular file is
|
||||
* **[link_cow](link_cow.md)=BOOL**: When enabled if a regular file is
|
||||
opened which has a link count > 1 it will copy the file to a
|
||||
temporary file and rename over the original. Breaking the link and
|
||||
providing a basic copy-on-write function similar to
|
||||
cow-shell. (default: false)
|
||||
- **[statfs](statfs.md)=base|full**: Controls how statfs works. 'base'
|
||||
* **[statfs](statfs.md)=base|full**: Controls how statfs works. 'base'
|
||||
means it will always use all branches in statfs calculations. 'full'
|
||||
is in effect path preserving and only includes branches where the
|
||||
path exists. (default: base)
|
||||
- **statfs_ignore=none|ro|nc**: 'ro' will cause statfs calculations to
|
||||
* **statfs_ignore=none|ro|nc**: 'ro' will cause statfs calculations to
|
||||
ignore available space for branches mounted or tagged as 'read-only'
|
||||
or 'no create'. 'nc' will ignore available space for branches tagged
|
||||
as 'no create'. (default: none)
|
||||
- **nfsopenhack=off|git|all**: A workaround for exporting mergerfs
|
||||
* **nfsopenhack=off|git|all**: A workaround for exporting mergerfs
|
||||
over NFS where there are issues with creating files for write while
|
||||
setting the mode to read-only. (default: off)
|
||||
- **branches-mount-timeout=UINT**: Number of seconds to wait at
|
||||
* **branches-mount-timeout=UINT**: Number of seconds to wait at
|
||||
startup for branches to be a mount other than the mountpoint's
|
||||
filesystem. (default: 0)
|
||||
- **[follow-symlinks](follow-symlinks.md)=never|directory|regular|all**:
|
||||
* **[follow-symlinks](follow-symlinks.md)=never|directory|regular|all**:
|
||||
Turns symlinks into what they point to. (default: never)
|
||||
- **[link-exdev](link-exdev.md)=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink**:
|
||||
* **[link-exdev](link-exdev.md)=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink**:
|
||||
When a link fails with EXDEV optionally create a symlink to the file
|
||||
instead.
|
||||
- **[rename-exdev](rename-exdev.md)=passthrough|rel-symlink|abs-symlink**:
|
||||
* **[rename-exdev](rename-exdev.md)=passthrough|rel-symlink|abs-symlink**:
|
||||
When a rename fails with EXDEV optionally move the file to a special
|
||||
directory and symlink to it.
|
||||
- **[readahead](readahead.md)=UINT**: Set readahead (in kilobytes) for
|
||||
* **[readahead](readahead.md)=UINT**: Set readahead (in kilobytes) for
|
||||
mergerfs and branches if greater than 0. (default: 0)
|
||||
- **posix_acl=BOOL**: Enable POSIX ACL support (if supported by kernel
|
||||
* **posix_acl=BOOL**: Enable POSIX ACL support (if supported by kernel
|
||||
and underlying filesystem). (default: false)
|
||||
- **async_read=BOOL**: Perform reads asynchronously. If disabled or
|
||||
* **async_read=BOOL**: Perform reads asynchronously. If disabled or
|
||||
unavailable the kernel will ensure there is at most one pending read
|
||||
request per file handle and will attempt to order requests by
|
||||
offset. (default: true)
|
||||
- **[fuse_msg_size](fuse_msg_size.md)=PAGESIZE**: Set the max number of
|
||||
* **[fuse_msg_size](fuse_msg_size.md)=PAGESIZE**: Set the max number of
|
||||
pages per FUSE message. Only available on Linux >= 4.20 and ignored
|
||||
otherwise. (min: 1; max: 65535; default: "1M")
|
||||
- **[threads](threads.md)=INT**: Number of threads to use. When used
|
||||
* **[threads](threads.md)=INT**: Number of threads to use. When used
|
||||
alone (`process-thread-count=-1`) it sets the number of threads
|
||||
reading and processing FUSE messages. When used together it sets the
|
||||
number of threads reading from FUSE. When set to zero it will
|
||||
|
|
@ -136,68 +136,71 @@ config file.
|
|||
`process-thread-count` then it will try to pick reasonable values
|
||||
based on CPU thread count. NOTE: higher number of threads increases
|
||||
parallelism but usually decreases throughput. (default: 0)
|
||||
- **[read-thread-count](threads.md)=INT**: Alias for `threads`.
|
||||
- **[process-thread-count](threads.md)=INT**: Enables separate thread
|
||||
* **[read-thread-count](threads.md)=INT**: Alias for `threads`.
|
||||
* **[process-thread-count](threads.md)=INT**: Enables separate thread
|
||||
pool to asynchronously process FUSE requests. In this mode
|
||||
`read-thread-count` refers to the number of threads reading FUSE
|
||||
messages which are dispatched to process threads. -1 means disabled
|
||||
otherwise acts like `read-thread-count`. (default: -1)
|
||||
- **[process-thread-queue-depth](threads.md)=UINT**: Sets the number
|
||||
* **[process-thread-queue-depth](threads.md)=UINT**: Sets the number
|
||||
of requests any single process thread can have queued up at one
|
||||
time. Meaning the total memory usage of the queues is queue depth
|
||||
multiplied by the number of process threads plus read thread
|
||||
count. 0 sets the depth to the same as the process thread
|
||||
count. (default: 0)
|
||||
- **[pin-threads](pin-threads.md)=STR**: Selects a strategy to pin
|
||||
* **[pin-threads](pin-threads.md)=STR**: Selects a strategy to pin
|
||||
threads to CPUs (default: unset)
|
||||
- **[flush-on-close](flush-on-close.md)=never|always|opened-for-write**:
|
||||
* **[flush-on-close](flush-on-close.md)=never|always|opened-for-write**:
|
||||
Flush data cache on file close. Mostly for when writeback is enabled
|
||||
or merging network filesystems. (default: opened-for-write)
|
||||
- **scheduling-priority=INT**: Set mergerfs' scheduling
|
||||
* **scheduling-priority=INT**: Set mergerfs' scheduling
|
||||
priority. Valid values range from -20 to 19. See `setpriority` man
|
||||
page for more details. (default: -10)
|
||||
- **fsname=STR**: Sets the name of the filesystem as seen in
|
||||
* **fsname=STR**: Sets the name of the filesystem as seen in
|
||||
**mount**, **df**, etc. Defaults to a list of the source paths
|
||||
concatenated together with the longest common prefix removed.
|
||||
- **[func.FUNC](functions_categories_policies.md)=POLICY**: Sets the
|
||||
* **[func.FUNC](functions_categories_policies.md)=POLICY**: Sets the
|
||||
specific FUSE function's policy. See below for the list of value
|
||||
types. Example: **func.getattr=newest**
|
||||
- **[func.readdir](func_readdir.md)=seq|cosr|cor|cosr:INT|cor:INT**: Sets `readdir`
|
||||
* **[func.readdir](func_readdir.md)=seq|cosr|cor|cosr:INT|cor:INT**: Sets `readdir`
|
||||
policy. INT value sets the number of threads to use for
|
||||
concurrency. (default: seq)
|
||||
- **[category.action](functions_categories_policies.md)=POLICY**: Sets
|
||||
* **[category.action](functions_categories_policies.md)=POLICY**: Sets
|
||||
policy of all FUSE functions in the action category. (default:
|
||||
epall)
|
||||
- **[category.create](functions_categories_policies.md)=POLICY**: Sets
|
||||
* **[category.create](functions_categories_policies.md)=POLICY**: Sets
|
||||
policy of all FUSE functions in the create category. (default:
|
||||
pfrd)
|
||||
- **[category.search](functions_categories_policies.md)=POLICY**: Sets
|
||||
* **[category.search](functions_categories_policies.md)=POLICY**: Sets
|
||||
policy of all FUSE functions in the search category. (default: ff)
|
||||
- **[cache.statfs](cache.md#cachestatfs)=UINT**: 'statfs' cache
|
||||
* **[cache.statfs](cache.md#cachestatfs)=UINT**: 'statfs' cache
|
||||
timeout in seconds. (default: 0)
|
||||
- **[cache.attr](cache.md#cacheattr)=UINT**: File attribute cache
|
||||
* **[cache.attr](cache.md#cacheattr)=UINT**: File attribute cache
|
||||
timeout in seconds. (default: 1)
|
||||
- **[cache.entry](cache.md#cacheentry)=UINT**: File name lookup cache
|
||||
* **[cache.entry](cache.md#cacheentry)=UINT**: File name lookup cache
|
||||
timeout in seconds. (default: 1)
|
||||
- **[cache.negative_entry](cache.md#cachenegative_entry)=UINT**:
|
||||
* **[cache.negative_entry](cache.md#cachenegative_entry)=UINT**:
|
||||
Negative file name lookup cache timeout in seconds. (default: 0)
|
||||
- **[cache.files](cache.md#cachefiles)=off|partial|full|auto-full|per-process**:
|
||||
* **[cache.files](cache.md#cachefiles)=off|partial|full|auto-full|per-process**:
|
||||
File page caching mode (default: off)
|
||||
- **cache.files.process-names=LIST**: A pipe | delimited list of
|
||||
* **cache.files.process-names=LIST**: A pipe | delimited list of
|
||||
process [comm](https://man7.org/linux/man-pages/man5/proc.5.html)
|
||||
names to enable page caching for when
|
||||
`cache.files=per-process`. (default: "rtorrent|qbittorrent-nox")
|
||||
- **[cache.writeback](cache.md#cachewriteback)=BOOL**: Enable kernel
|
||||
* **[cache.writeback](cache.md#cachewriteback)=BOOL**: Enable kernel
|
||||
writeback caching (default: false)
|
||||
- **[cache.symlinks](cache.md#cachesymlinks)=BOOL**: Cache symlinks (if
|
||||
* **[cache.symlinks](cache.md#cachesymlinks)=BOOL**: Cache symlinks (if
|
||||
supported by kernel) (default: false)
|
||||
- **[cache.readdir](cache.md#cachereaddir)=BOOL**: Cache readdir (if
|
||||
* **[cache.readdir](cache.md#cachereaddir)=BOOL**: Cache readdir (if
|
||||
supported by kernel) (default: false)
|
||||
- **parallel-direct-writes=BOOL**: Allow the kernel to dispatch
|
||||
* **parallel-direct-writes=BOOL**: Allow the kernel to dispatch
|
||||
multiple, parallel (non-extending) write requests for files opened
|
||||
with `cache.files=per-process` (if the process is not in
|
||||
`process-names`) or `cache.files=off`. (Is a kernel feature added in
|
||||
v6.2) (default: true)
|
||||
* **[passthrough](passthrough.md)**: Enable [FUSE IO
|
||||
passthrough](https://kernelnewbies.org/Linux_6.9#Faster_FUSE_I.2FO)
|
||||
if available. (default: off)
|
||||
|
||||
**NOTE:** Options are evaluated in the order listed so if the options
|
||||
are **func.rmdir=rand,category.action=ff** the **action** category
|
||||
|
|
|
|||
102
mkdocs/docs/config/passthrough.md
Normal file
102
mkdocs/docs/config/passthrough.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# passthrough
|
||||
|
||||
* default: `off`
|
||||
* arguments:
|
||||
* `off`: Passthrough is never enabled.
|
||||
* `ro`: Only enable passthrough when file opened for reading only.
|
||||
* `wo`: Only enable passthrough when file opened for writing only.
|
||||
* `rw`: Enable passthrough when file opened for reading, writing,
|
||||
or both.
|
||||
|
||||
In [Linux 6.9](https://kernelnewbies.org/Linux_6.9#Faster_FUSE_I.2FO)
|
||||
a IO passthrough feature was added to FUSE. Typically `mergerfs` has
|
||||
to act as an active proxy for all read and write requests. This
|
||||
results in, at times, significant overhead compared to direct
|
||||
interaction with the underlying filesystems. Not because `mergerfs` is
|
||||
doing anything particularly slow but because the additional
|
||||
communication and data transfers required. With the `passthrough`
|
||||
feature `mergerfs` is able to instruct the kernel to perform reads and
|
||||
writes directly on the underlying file. Bypassing `mergerfs` entirely
|
||||
(specifically and only for reads and writes) and thereby providing
|
||||
near native performance.
|
||||
|
||||
This performance does come at the cost of some functionality as
|
||||
`mergerfs` no longer has control over reads and writes which means all
|
||||
features related to read/write are affected.
|
||||
|
||||
* [moveonenospc](moveonenospc.md): Does not work because errors are
|
||||
not reported back to `mergerfs`.
|
||||
* nullrw: Since `mergerfs` no longer receives read/write requests
|
||||
there is no read or write to ignore.
|
||||
* [readahead](readahead.md): Still affects the `readahead` values of
|
||||
underlying filesystems and `mergerfs` itself but no longer relevant
|
||||
to `mergerfs` in that it is not servicing IO requests.
|
||||
* [fuse_msg_size](fuse_msg_size.md): The primary reason to increase
|
||||
the FUSE message size is to allow transferring more data per request
|
||||
which helps improve read and write performance. Without read/write
|
||||
requests being sent to `mergerfs` there is little reason to have
|
||||
larger message sizes since the only other message larger than 1
|
||||
page, directory reading, currently has a small fixed buffer size.
|
||||
* parallel-direct-writes: Not only is `direct-io` and `passthrough`
|
||||
mutually exclusive but since `mergerfs` no longer receives write
|
||||
requests in passthrough mode then there is no parallel writes
|
||||
possible.
|
||||
* [cache.writeback](cache.md): FUSE's writeback caching is
|
||||
incompatible with `passthrough`. If `cache.writeback=true` when
|
||||
enabling `passthrough` it will be reset to `false`.
|
||||
* [cache.files](cache.md): Must be enabled for `passthrough` to
|
||||
work. When `cache.files=off` FUSE's `direct-io` mode is enabled
|
||||
which overrides `passthrough.` Meaning `cache.files` should be set
|
||||
to `partial`, `full` or `auto-full` to use `passthrough`. If
|
||||
`cache.files` is set to `off` when using `passthrough` it will
|
||||
reset it to `auto-full`.
|
||||
|
||||
Technically `mergerfs` has the ability to choose options like
|
||||
`passthrough`, `direct-io`, and page caching independently for every
|
||||
file opened. However, at the moment there is no use case for picking
|
||||
and choosing which to enable outside `cache.files=per-process` (which
|
||||
is largely unnecessary on Linux v6.6 and above. See
|
||||
[direct-io-allow-mmap](options.md)) If such a use case arises please
|
||||
reach out to the author to discuss.
|
||||
|
||||
Unlike [preload.so](../tooling.md#preloadso), `passthrough` will work for
|
||||
any software interacting with `mergerfs`. However, `passthrough`
|
||||
requires Linux v6.9 or above to work.
|
||||
|
||||
**NOTE:** This feature will **ONLY** work if `mergerfs` is running as
|
||||
`root` as currently only `root` is allowed to leverage the kernel
|
||||
feature.
|
||||
|
||||
**NOTE:** If a file has been opened and passthrough enabled, while that
|
||||
file is open, if another open request is made `mergerfs` must also
|
||||
enable `passthrough` for the second open request. This is a limitation
|
||||
of how the passthrough feature works.
|
||||
|
||||
|
||||
## Alternatives
|
||||
|
||||
* [preload.so](../tooling.md#preloadso): Leverages the ability to
|
||||
intercept certain filesystem calls to bypass `mergerfs` for
|
||||
IO. See page for details and limitations.
|
||||
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Simple `dd` based benchmark on a Hades Canyon (Intel i7-8809G, 32GB
|
||||
DDR4-2400 RAM).
|
||||
|
||||
```
|
||||
# /mnt/test is mergerfs over a single tmpfs mount
|
||||
dd if=/dev/zero of=/mnt/test/tmpfile bs=1M count=2000 oflag=nocache
|
||||
conv=fdatasync status=progress
|
||||
```
|
||||
|
||||
Summary: passthrough is ~95% native speed. 200% faster than
|
||||
`direct-io` (cache.files=off).
|
||||
|
||||
* straight to tmpfs: 1.7 GB/s
|
||||
* nullrw=true: 2.1 GB/s
|
||||
* cache.files=off; passthrough=off: 800 MB/s
|
||||
* cache.files=auto-full; passthrough=rw: 1.6 GB/s
|
||||
* cache.files=auto-full; cache.writeback=false: 518 MB/s
|
||||
* cache.files=auto-full; cache.writeback=true: 518 MB/s
|
||||
Loading…
Add table
Add a link
Reference in a new issue