Doc and mover script updates

This commit is contained in:
Antonio SJ Musumeci 2025-01-19 21:33:05 -06:00
parent 978d067921
commit 19509e33d2
6 changed files with 161 additions and 65 deletions

View file

@ -29,45 +29,40 @@ across filesystems (see the mergerfs.dup tool) and setting
`func.open=rand`, using `symlinkify`, or using dm-cache or a similar
technology to add tiered cache to the underlying device itself.
With #2 one could use dm-cache as well but there is another solution
which requires only mergerfs and a cronjob.
With #2 one could use a block cache solution as available via LVM and
dm-cache but there is another solution which requires only mergerfs, a
script to move files around, and a cron job to run said script.
1. Create 2 mergerfs pools. One which includes just the slow branches
and one which has both the fast branches (SSD,NVME,etc.) and slow
branches. The 'base' pool and the 'cache' pool.
2. The 'cache' pool should have the cache branches listed first in
the branch list.
3. The best `create` policies to use for the 'cache' pool would
probably be `ff`, `epff`, `lfs`, `msplfs`, or `eplfs`. The latter
three under the assumption that the cache filesystem(s) are far
smaller than the backing filesystems. If using path preserving
policies remember that you'll need to manually create the core
directories of those paths you wish to be cached. Be sure the
permissions are in sync. Use `mergerfs.fsck` to check / correct
them. You could also set the slow filesystems mode to `NC` though
that'd mean if the cache filesystems fill you'd get "out of space"
errors.
4. Enable `moveonenospc` and set `minfreespace` appropriately. To
make sure there is enough room on the "slow" pool you might want
to set `minfreespace` to at least as large as the size of the
largest cache filesystem if not larger. This way in the worst case
the whole of the cache filesystem(s) can be moved to the other
drives.
5. Set your programs to use the 'cache' pool.
6. Save one of the below scripts or create you're own. The script's
responsibility is to move files from the cache filesystems (not
pool) to the 'base' pool.
7. Use `cron` (as root) to schedule the command at whatever frequency
is appropriate for your workflow.
* Create two mergerfs pools. One which includes just the **slow**
branches and one which has both the **fast** branches
(SSD,NVME,etc.) and **slow** branches. The **base** pool and the
**cache** pool.
* The **cache** pool should have the cache branches listed first in
the branch list in order to to make it easier to prioritize them.
* The best `create` policies to use for the **cache** pool would
probably be `ff`, `lus`, or `lfs`. The latter two under the
assumption that the cache filesystem(s) are far smaller than the
backing filesystems.
* You can also set the **slow** filesystems mode to `NC` which would
give you the ability to use other `create` policies though that'd
mean if the cache filesystems fill you'd get "out of space"
errors. This however may be good as it would indicate the script
moving files around is not configured properly.
* Set your programs to use the **cache** pool.
* Configure the **base** pool with the `create` policy you would like
to lay out files as you like.
* Save one of the below scripts or create your own. The script's
responsibility is to move files from the **cache** branches (not
pool) to the **base** pool.
* Use `cron` (as root) to schedule the command at whatever frequency
is appropriate for your workflow.
### time based expiring
Move files from cache to base pool based only on the last time the
file was accessed. Replace `-atime` with `-amin` if you want minutes
rather than days. May want to use the `fadvise` / `--drop-cache`
version of rsync or run rsync with the tool
[nocache](https://github.com/Feh/nocache).
Move files from cache filesystem to base pool which have an access
time older than the supplied number of days. Replace `-atime` with
`-amin` in the script if you want minutes rather than days.
**NOTE:** The arguments to these scripts include the cache
**filesystem** itself. Not the pool with the cache filesystem. You
@ -75,14 +70,50 @@ could have data loss if the source is the cache pool.
[mergerfs.time-based-mover](https://github.com/trapexit/mergerfs/blob/latest-release/tools/mergerfs.time-based-mover?raw=1)
Download:
```
curl -o /usr/local/bin/mergerfs.time-based-mover https://raw.githubusercontent.com/trapexit/mergerfs/refs/heads/latest-release/tools/mergerfs.time-based-mover
```
crontab entry:
```
# m h dom mon dow command
0 * * * * /usr/local/bin/mergerfs.time-based-mover /mnt/ssd/cache00 /mnt/base-pool 1
```
If you have more than one cache filesystem then simply add a cron
entry for each.
If you want to only move files from a subdirectory then use the
subdirectories. `/mnt/ssd/cache00/foo` and `/mnt/base-pool/foo`
respectively.
### percentage full expiring
Move the oldest file from the cache to the backing pool. Continue till
below percentage threshold.
While the cache filesystem's percentage full is above the provided
value move the oldest file from the cache filesystem to the base pool.
**NOTE:** The arguments to these scripts include the cache
**filesystem** itself. Not the pool with the cache filesystem. You
could have data loss if the source is the cache pool.
[mergerfs.percent-full-mover](https://github.com/trapexit/mergerfs/blob/latest-release/tools/mergerfs.percent-full-mover?raw=1)
Download:
```
curl -o /usr/local/bin/mergerfs.percent-full-mover https://raw.githubusercontent.com/trapexit/mergerfs/refs/heads/latest-release/tools/mergerfs.percent-full-mover
```
crontab entry:
```
# m h dom mon dow command
0 * * * * /usr/local/bin/mergerfs.percent-full-mover /mnt/ssd/cache00 /mnt/base-pool 80
```
If you have more than one cache filesystem then simply add a cron
entry for each.
If you want to only move files from a subdirectory then use the
subdirectories. `/mnt/ssd/cache00/foo` and `/mnt/base-pool/foo`
respectively.