Rework thread pool for increased stability + config and doc updates (#1453)

Also updates fmt to 11.2.0 and replaced syslog usage with a wrapper
leveraging fmt.
This commit is contained in:
trapexit 2025-05-05 18:28:49 -05:00 • committed by GitHub
parent 455a81290b
commit 344014aa81
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
58 changed files with 9097 additions and 24020 deletions

View file

@ -1,14 +1,20 @@
# func.readdir
examples: `func.readdir=seq`, `func.readdir=cor:4`
Defaults to `seq`
Examples: `func.readdir=seq`, `func.readdir=cor:4`
`readdir` has policies to control how it reads directory content.
| Policy | Description |
| ------ | ----------- |
| seq | "sequential" : Iterate sequentially over branches in the order defined in `branches`. This is the default and traditional behavior found prior to the readdir policy introduction. This will be increasingly slower as more branches are added to the pool. Especially if needing to wait for drives to spin up or network filesystems to respond. |
| cosr | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in the order defined in `branches`. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cosr:N` where `N` is the number of threads. |
| cor | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. This should not be a problem since the order of files listed in not guaranteed. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cor:N` where `N` is the number of threads. |
| seq | "sequential" : Iterate sequentially over branches in the order defined in `branches`. This is the default and traditional behavior found prior to the readdir policy introduction. This will be increasingly slower as more branches are added to the pool. Especially if needing to wait for drives to spin up or network filesystems to respond. |
| cosr:N:M | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in the order defined in `branches`. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. `N` is the number of threads. If negative it will be the core count divided by `abs(N)`. `M` is the queue depth. If either value is `0` it will be decided based on system configuration. |
| cosr:N | cosr:N:M with M = 0 |
| cosr | cosr:N:M with N = 0 and M = 0 |
| cor:N:M | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. This should not be a problem since the order of files listed in not guaranteed. `N` is the number of threads. If negative it will be the core count divided by `abs(N)`. `M` is the queue depth. If either value is `0` it will be decided based on system configuration. |
| cor:N | cosr:N:M with M = 0 |
| cor | cosr:N:M with N = 0 and M = 0 |
Keep in mind that `readdir` mostly just provides a list of file names
in a directory and possibly some basic metadata about said files. To

View file

@ -6,48 +6,56 @@ parallel behaviors.
## read-thread-count
The number of threads used to read (and possibly process) messages
from the kernel.
Defaults to `0`
* `read-thread-count=0`: Create a thread pool sized to the number of
logical CPUs.
* `read-thread-count=N` where `N>0`: Create a thread pool of `N` threads.
* `read-thread-count=N` where `N<0`: Create a thread pool of `CPUCount /
-N` threads.
* `read-thread-count=-1` where `process-thread-count=-1`: Creates `2`
read threads and `max(2,CPUCount-2)` process threads.
* Defaults to `0`.
The number of threads used to read messages from the kernel. If used
alone processing will be done on the same thread. If
`process-thread-count` is enabled then this thread pool will read
messages and the process thread pool will do the work. This can
increase how much mergerfs can process at one time but may reduce
throughput.
When `process-thread-count=-1` (the default) this option sets the
number of threads which read and then process requests from the
kernel.
When `process-thread-count` is set to anything else mergerfs will
create two thread pools. A "read" thread pool which just reads from
the kernel and hands off requests to the "process" thread pool.
Generally, only 1 or 2 "read" threads are necessary.
* `read-thread-count=0` and `process-thread-count=-1`: Create `1`
read+process thread per logical CPU core upto `8`.
* `read-thread-count=N` where `N>0` and `process-thread-count=-1`:
Create a thread pool of `N` read+process threads.
* `read-thread-count=N` where `N<0` and `process-thread-count=-1`:
Create a read+process thread pool of `CPUCount / -N`
threads. Minimum of `1`.
* `read-thread-count=0` and `process-thread-count=0`: Create `2` read
threads and a process thread per logical CPU core upto `8`.
* `read-thread-count=0` and `process-thread-count!=-1`: Creates `2`
read threads and number of process threads as defined below.
## process-thread-count
Defaults to `-1`
When enabled this sets the number of threads in the message processing pool.
* `process-thread-count=-1`: Process thread pool is disabled.
* `process-thread-count=0`: Create a thread pool sized to the number
of logical CPUs.
* `process-thread-count=0`: Create `1` thread process thread per
logical CPU core upto `8`.
* `process-thread-count=N` where `N>0`: Create a thread pool of `N` threads.
* `process-thread-count=N` where `N<-1`: Create a thread pool of `CPUCount /
-N` threads.
* Defaults to `-1`.
-N` threads. Minimum of 1.
## process-thread-queue-depth
* `process-thread-queue-depth=N` where `N>0`: Sets the number of outstanding
requests that a process thread can have to N. If requests come in
faster than can be processed and the max queue depth hit then
queuing the request will block in order to limit memory growth.
* `process-thread-queue-depth=0`: Sets the queue depth to the thread
pool count.
* Defaults to `0`.
Defaults to `2`
Sets the depth queue for the processing thread queue per
thread. Meaning if the read threads are getting requests faster than
can be processed they will be queued up upto the queue depth. Despite
the calculation being per thread the queue depth is shared across all
in the pool.
* `process-thread-queue-depth=N` where `N>0`: Sets the number of
outstanding requests that the process thread pool can have to `N *
process-thread-count`. If requests come in faster than can be
processed and the max queue depth hit then queuing the request will
block in order to limit memory growth.
* `process-thread-queue-depth<=0`: Sets the queue depth to 2. May be
used in the future to set dynamically.