mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-11-21 23:46:26 -05:00
Rework thread pool for increased stability + config and doc updates (#1453)
Also updates fmt to 11.2.0 and replaced syslog usage with a wrapper leveraging fmt.
This commit is contained in:
parent
455a81290b
commit
344014aa81
58 changed files with 9097 additions and 24020 deletions
|
|
@ -1,14 +1,20 @@
|
|||
# func.readdir
|
||||
|
||||
examples: `func.readdir=seq`, `func.readdir=cor:4`
|
||||
Defaults to `seq`
|
||||
|
||||
Examples: `func.readdir=seq`, `func.readdir=cor:4`
|
||||
|
||||
`readdir` has policies to control how it reads directory content.
|
||||
|
||||
| Policy | Description |
|
||||
| ------ | ----------- |
|
||||
| seq | "sequential" : Iterate sequentially over branches in the order defined in `branches`. This is the default and traditional behavior found prior to the readdir policy introduction. This will be increasingly slower as more branches are added to the pool. Especially if needing to wait for drives to spin up or network filesystems to respond. |
|
||||
| cosr | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in the order defined in `branches`. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cosr:N` where `N` is the number of threads. |
|
||||
| cor | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. This should not be a problem since the order of files listed in not guaranteed. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cor:N` where `N` is the number of threads. |
|
||||
| seq | "sequential" : Iterate sequentially over branches in the order defined in `branches`. This is the default and traditional behavior found prior to the readdir policy introduction. This will be increasingly slower as more branches are added to the pool. Especially if needing to wait for drives to spin up or network filesystems to respond. |
|
||||
| cosr:N:M | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in the order defined in `branches`. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. `N` is the number of threads. If negative it will be the core count divided by `abs(N)`. `M` is the queue depth. If either value is `0` it will be decided based on system configuration. |
|
||||
| cosr:N | cosr:N:M with M = 0 |
|
||||
| cosr | cosr:N:M with N = 0 and M = 0 |
|
||||
| cor:N:M | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. This should not be a problem since the order of files listed in not guaranteed. `N` is the number of threads. If negative it will be the core count divided by `abs(N)`. `M` is the queue depth. If either value is `0` it will be decided based on system configuration. |
|
||||
| cor:N | cosr:N:M with M = 0 |
|
||||
| cor | cosr:N:M with N = 0 and M = 0 |
|
||||
|
||||
Keep in mind that `readdir` mostly just provides a list of file names
|
||||
in a directory and possibly some basic metadata about said files. To
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue