mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-11-19 22:46:34 -05:00
Doc and mover script updates
This commit is contained in:
parent
978d067921
commit
19509e33d2
6 changed files with 161 additions and 65 deletions
|
|
@ -29,45 +29,40 @@ across filesystems (see the mergerfs.dup tool) and setting
|
|||
`func.open=rand`, using `symlinkify`, or using dm-cache or a similar
|
||||
technology to add tiered cache to the underlying device itself.
|
||||
|
||||
With #2 one could use dm-cache as well but there is another solution
|
||||
which requires only mergerfs and a cronjob.
|
||||
With #2 one could use a block cache solution as available via LVM and
|
||||
dm-cache but there is another solution which requires only mergerfs, a
|
||||
script to move files around, and a cron job to run said script.
|
||||
|
||||
1. Create 2 mergerfs pools. One which includes just the slow branches
|
||||
and one which has both the fast branches (SSD,NVME,etc.) and slow
|
||||
branches. The 'base' pool and the 'cache' pool.
|
||||
2. The 'cache' pool should have the cache branches listed first in
|
||||
the branch list.
|
||||
3. The best `create` policies to use for the 'cache' pool would
|
||||
probably be `ff`, `epff`, `lfs`, `msplfs`, or `eplfs`. The latter
|
||||
three under the assumption that the cache filesystem(s) are far
|
||||
smaller than the backing filesystems. If using path preserving
|
||||
policies remember that you'll need to manually create the core
|
||||
directories of those paths you wish to be cached. Be sure the
|
||||
permissions are in sync. Use `mergerfs.fsck` to check / correct
|
||||
them. You could also set the slow filesystems mode to `NC` though
|
||||
that'd mean if the cache filesystems fill you'd get "out of space"
|
||||
errors.
|
||||
4. Enable `moveonenospc` and set `minfreespace` appropriately. To
|
||||
make sure there is enough room on the "slow" pool you might want
|
||||
to set `minfreespace` to at least as large as the size of the
|
||||
largest cache filesystem if not larger. This way in the worst case
|
||||
the whole of the cache filesystem(s) can be moved to the other
|
||||
drives.
|
||||
5. Set your programs to use the 'cache' pool.
|
||||
6. Save one of the below scripts or create you're own. The script's
|
||||
responsibility is to move files from the cache filesystems (not
|
||||
pool) to the 'base' pool.
|
||||
7. Use `cron` (as root) to schedule the command at whatever frequency
|
||||
is appropriate for your workflow.
|
||||
* Create two mergerfs pools. One which includes just the **slow**
|
||||
branches and one which has both the **fast** branches
|
||||
(SSD,NVME,etc.) and **slow** branches. The **base** pool and the
|
||||
**cache** pool.
|
||||
* The **cache** pool should have the cache branches listed first in
|
||||
the branch list in order to to make it easier to prioritize them.
|
||||
* The best `create` policies to use for the **cache** pool would
|
||||
probably be `ff`, `lus`, or `lfs`. The latter two under the
|
||||
assumption that the cache filesystem(s) are far smaller than the
|
||||
backing filesystems.
|
||||
* You can also set the **slow** filesystems mode to `NC` which would
|
||||
give you the ability to use other `create` policies though that'd
|
||||
mean if the cache filesystems fill you'd get "out of space"
|
||||
errors. This however may be good as it would indicate the script
|
||||
moving files around is not configured properly.
|
||||
* Set your programs to use the **cache** pool.
|
||||
* Configure the **base** pool with the `create` policy you would like
|
||||
to lay out files as you like.
|
||||
* Save one of the below scripts or create your own. The script's
|
||||
responsibility is to move files from the **cache** branches (not
|
||||
pool) to the **base** pool.
|
||||
* Use `cron` (as root) to schedule the command at whatever frequency
|
||||
is appropriate for your workflow.
|
||||
|
||||
|
||||
### time based expiring
|
||||
|
||||
Move files from cache to base pool based only on the last time the
|
||||
file was accessed. Replace `-atime` with `-amin` if you want minutes
|
||||
rather than days. May want to use the `fadvise` / `--drop-cache`
|
||||
version of rsync or run rsync with the tool
|
||||
[nocache](https://github.com/Feh/nocache).
|
||||
Move files from cache filesystem to base pool which have an access
|
||||
time older than the supplied number of days. Replace `-atime` with
|
||||
`-amin` in the script if you want minutes rather than days.
|
||||
|
||||
**NOTE:** The arguments to these scripts include the cache
|
||||
**filesystem** itself. Not the pool with the cache filesystem. You
|
||||
|
|
@ -75,14 +70,50 @@ could have data loss if the source is the cache pool.
|
|||
|
||||
[mergerfs.time-based-mover](https://github.com/trapexit/mergerfs/blob/latest-release/tools/mergerfs.time-based-mover?raw=1)
|
||||
|
||||
Download:
|
||||
```
|
||||
curl -o /usr/local/bin/mergerfs.time-based-mover https://raw.githubusercontent.com/trapexit/mergerfs/refs/heads/latest-release/tools/mergerfs.time-based-mover
|
||||
```
|
||||
|
||||
crontab entry:
|
||||
```
|
||||
# m h dom mon dow command
|
||||
0 * * * * /usr/local/bin/mergerfs.time-based-mover /mnt/ssd/cache00 /mnt/base-pool 1
|
||||
```
|
||||
|
||||
If you have more than one cache filesystem then simply add a cron
|
||||
entry for each.
|
||||
|
||||
If you want to only move files from a subdirectory then use the
|
||||
subdirectories. `/mnt/ssd/cache00/foo` and `/mnt/base-pool/foo`
|
||||
respectively.
|
||||
|
||||
|
||||
### percentage full expiring
|
||||
|
||||
Move the oldest file from the cache to the backing pool. Continue till
|
||||
below percentage threshold.
|
||||
While the cache filesystem's percentage full is above the provided
|
||||
value move the oldest file from the cache filesystem to the base pool.
|
||||
|
||||
**NOTE:** The arguments to these scripts include the cache
|
||||
**filesystem** itself. Not the pool with the cache filesystem. You
|
||||
could have data loss if the source is the cache pool.
|
||||
|
||||
[mergerfs.percent-full-mover](https://github.com/trapexit/mergerfs/blob/latest-release/tools/mergerfs.percent-full-mover?raw=1)
|
||||
|
||||
Download:
|
||||
```
|
||||
curl -o /usr/local/bin/mergerfs.percent-full-mover https://raw.githubusercontent.com/trapexit/mergerfs/refs/heads/latest-release/tools/mergerfs.percent-full-mover
|
||||
```
|
||||
|
||||
crontab entry:
|
||||
```
|
||||
# m h dom mon dow command
|
||||
0 * * * * /usr/local/bin/mergerfs.percent-full-mover /mnt/ssd/cache00 /mnt/base-pool 80
|
||||
```
|
||||
|
||||
If you have more than one cache filesystem then simply add a cron
|
||||
entry for each.
|
||||
|
||||
If you want to only move files from a subdirectory then use the
|
||||
subdirectories. `/mnt/ssd/cache00/foo` and `/mnt/base-pool/foo`
|
||||
respectively.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue