8f41b40692
_remove_thread_scaling_locked previously enqueued a heap-allocated lambda (capturing a shared_ptr<promise>) to signal a specific worker to exit. This had two problems: 1. Bug: if the 5-second future::wait_for timed out, the function returned -EBUSY but the lambda remained in the queue. When a worker eventually executed it, it would unconditionally set _tl_should_exit and call _remove_self, silently killing an arbitrary thread with no compensating add. 2. Held _scaling_mutex for up to 5 seconds while blocking on the future, serializing all concurrent scaling operations (add_thread, remove_thread, _maybe_grow_on_pressure, monitor hill-climb). Replace the entire mechanism with an atomic _remove_count counter. _remove_thread_scaling_locked simply increments the counter and returns immediately. After each task, _process_func does a CAS-decrement; the winning worker detaches itself via _remove_self and exits. This eliminates: - All heap allocations (shared_ptr, promise, lambda captures) - The thread-local _tl_should_exit flag - The 5-second blocking window under _scaling_mutex - The stale-lambda bug entirely (no lambda to go stale) - The <memory> include (no longer needed) During shutdown, _remove_self is a no-op (_stop is set), so a worker that wins the CAS stays in _threads and is joined normally by the destructor. Any outstanding _remove_count credits are harmlessly abandoned. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .github | ||
| buildtools | ||
| debian | ||
| man | ||
| mkdocs | ||
| src | ||
| tests | ||
| tools | ||
| vendored | ||
| .gitignore | ||
| DEPENDENCIES | ||
| LICENSE | ||
| Makefile | ||
| mergerfs.spec | ||
| README.md | ||
mergerfs - a featureful union filesystem
mergerfs is a FUSE based union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, aufs, DrivePool, etc.
Features
- Logically combine numerous filesystems/paths into a single mount point (JBOFS: Just a Bunch of FileSystems)
- Combine paths of the same or different filesystems
- Ability to add or remove filesystems/paths without impacting the rest of the data
- Unaffected by individual filesystem failure
- Configurable file selection and creation placement
- File IO passthrough for near native IO performance (where supported)
- Works with filesystems of any size
- Works with filesystems of almost any type
- Ignore read-only filesystems when creating files
- Hard links
- Hard link copy-on-write / CoW
- Runtime configurable
- Support for extended attributes (xattrs)
- Support for file attributes (chattr)
- Support for POSIX ACLs
Non-features
- Read/write overlay on top of read-only filesystem like OverlayFS
- File whiteout
- RAID like parity calculation (see SnapRAID)
- Redundancy
- Splitting of files across branches
How it works
mergerfs logically merges multiple filesystem paths together. Not block devices, not filesystem mounts, just paths. It acts as a proxy to the underlying filesystem paths. Combining the behaviors of some functions and being a selector for others.
When the contents of a directory are requested mergerfs combines the list of files from each directory, deduplicating entries, and returns that list.
When a file or directory is created a policy is first run to determine which branch will be selected for the creation.
For functions which change attributes or remove the file the behavior may be applied to all instances found.
The way in which mergerfs behaves is controlled by the config/options/settings. More specifically by policies.
Visualization
A + B = C
/disk1 /disk2 /merged
| | |
+-- /dir1 +-- /dir1 +-- /dir1
| | | | | |
| +-- file1 | | | +-- file1
| | +-- file2 | +-- file2
| | +-- file3 | +-- file3
| | |
+-- /dir2 | +-- /dir2
| | | | |
| +-- file4 | | +-- file4
| | |
| +-- /dir3 +-- /dir3
| | | | |
| | +-- file5 | +-- file5
| | |
+-- file6 | +-- file6
+-- file7 +-- file7 +-- file7
QuickStart
https://trapexit.github.io/mergerfs/latest/quickstart/
Documentation
https://trapexit.github.io/mergerfs
Support
https://trapexit.github.io/mergerfs/latest/support/
Sponsorship and Donations
https://github.com/trapexit/support
Development and support of a project like mergerfs requires a significant amount of time and effort. The software is released under the very liberal ISC license and is therefore free to use for personal or commercial uses.
If you are a non-commercial user and find mergerfs and its support valuable and would like to support the project financially it would be very much appreciated.
If you are using mergerfs commercially please consider sponsoring the project to ensure it continues to be maintained and receive updates. If custom features are needed feel free to contact me directly.