rt/snapraid
1
0
Fork 0
mirror of https://github.com/amadvance/snapraid.git synced 2026-10-26 19:01:57 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Various manual updates.

Browse source
This commit is contained in:
Andrea Mazzoleni 2014-12-30 18:30:11 +01:00
parent 0c4c1f78a4
commit 4885a38c8a
3 changed files with 615 additions and 621 deletions
Download patch file Download diff file Expand all files Collapse all files

453
snapraid.1
View file

@ -271,49 +271,6 @@ When this command completes, your data is SAFE.
.PP
Now you can start using your array as you like, and periodically
update the parity information running the \[dq]sync\[dq] command.
.SS Restoring and Undeleting
In recovering, SnapRAID is more like a backup program than a RAID system,
and it can be used to restore or undelete a file or directory to its previous
state using the \-f, \-\-filter option :
.PP
.RS 4
snapraid fix \-f FILE
.PD 0
.PP
.PD
.RE
.PP
or for a directory:
.PP
.RS 4
snapraid fix \-f DIR/
.PD 0
.PP
.PD
.RE
.PP
You can also use it to recover only accidentally deleted files inside
a directory using the \-m, \-\-filter\-missing option, that restores
only missing files, leaving untouched all the others.
.PP
.RS 4
snapraid fix \-m \-f DIR/
.PD 0
.PP
.PD
.RE
.PP
Or to recover all the deleted files in all the drives with:
.PP
.RS 4
snapraid fix \-m
.PD 0
.PP
.PD
.RE
.PP
If instead you have lost an entire disk and you want to recover it,
please see the \'Recovering\' chapter.
.SS Scrubbing
To periodically check the data and parity for errors, you can
run the \[dq]scrub\[dq] command.
@ -325,9 +282,11 @@ snapraid scrub
.PD
.RE
.PP
This command verifies only the oldest data in your array.
Every run of the command checks about 12% of the array,
but not newer than 10 days.
This command verifies the data in your array comparing it with
the hash computed in the \[dq]sync\[dq] command.
.PP
Every run of the command checks about 12% of the array, but not data newer
than 10 days.
You can use the \-p, \-\-percentage option to specify a different amount,
and the \-o, \-\-older\-than option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
@ -339,8 +298,9 @@ snapraid \-p 5 \-o 20 scrub
.PD
.RE
.PP
If a silent or input error is found, the corresponding block is marked
as bad in the \[dq]content\[dq] file, and listed in the \[dq]status\[dq] command.
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad in the \[dq]content\[dq] file,
and listed in the \[dq]status\[dq] command.
.PP
.RS 4
snapraid status
@ -349,8 +309,8 @@ snapraid status
.PD
.RE
.PP
To fix them, you can use the \[dq]fix\[dq] command filtering for files containing
bad blocks:
To fix them, you can use the \[dq]fix\[dq] command filtering for bad blocks with
the \-e, \-\-filter\-error options:
.PP
.RS 4
snapraid \-e fix
@ -370,7 +330,7 @@ snapraid \-p 0 scrub
.PD
.RE
.PP
Take care that running scrub on a not synced array may result in
Take care that running \[dq]scrub\[dq] on a not synced array may result in
errors caused by removed or modified files. These errors are reported
in the \[dq]scrub\[dq] result, but related blocks are not marked as bad.
.SS Pooling
@ -523,6 +483,145 @@ fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
.PP
.PD
.RE
.SS Undeleting
SnapRAID is more like a backup program than a RAID system, and it
can be used to restore or undelete files to their previous state using
the \-f, \-\-filter option :
.PP
.RS 4
snapraid fix \-f FILE
.PD 0
.PP
.PD
.RE
.PP
or for a directory:
.PP
.RS 4
snapraid fix \-f DIR/
.PD 0
.PP
.PD
.RE
.PP
You can also use it to recover only accidentally deleted files inside
a directory using the \-m, \-\-filter\-missing option, that restores
only missing files, leaving untouched all the others.
.PP
.RS 4
snapraid fix \-m \-f DIR/
.PD 0
.PP
.PD
.RE
.PP
Or to recover all the deleted files in all the drives with:
.PP
.RS 4
snapraid fix \-m
.PD 0
.PP
.PD
.RE
.SS Recovering
The worst happened, and you lost a disk!
.PP
DO NOT PANIC! You will be able to recover it!
.PP
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync or scrub.
.PP
Then proceed with the following steps.
.SS STEP 1 \-> Reconfigure
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
.PP
Change the SnapRAID configuration file to make the \[dq]disk\[dq] option
of the failed disk to point to the place where you have enough empty
space to recover the files.
.PP
For example, if you have that disk \[dq]d1\[dq] failed, you can change from:
.PP
.RS 4
disk d1 /mnt/disk1/
.PD 0
.PP
.PD
.RE
.PP
to:
.PP
.RS 4
disk d1 /mnt/new_spare_disk/
.PD 0
.PP
.PD
.RE
.SS STEP 2 \-> Fix
Run the fix command, storing the log in an external file with:
.PP
.RS 4
snapraid \-d NAME \-l fix.log fix
.PD 0
.PP
.PD
.RE
.PP
Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example.
.PP
This command will take a long time.
.PP
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
.PP
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the \[dq].unrecoverable\[dq] extension.
.PP
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with \[dq]unrecoverable:\[dq]
.PP
If you are not satisfied of the recovering, you can retry it as many
time you wish.
.PP
For example, if you have removed files from the array after the last
\[dq]sync\[dq], this may result in some other files not recovered.
In this case, you can retry the \[dq]fix\[dq] using the \-i, \-\-import option,
specifing where these files are now, to include them again in the
recovering process.
.PP
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you cannot retry the \[dq]fix\[dq] command
anymore!
.SS STEP 3 \-> Check
As paranoid check, you can now run a \[dq]check\[dq] command to ensure that
everything is OK on the recovered disk.
.PP
.RS 4
snapraid \-d NAME \-a check
.PD 0
.PP
.PD
.RE
.PP
Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example.
.PP
The options \-d and \-a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
.PP
This command will take a long time, but if you are not paranoid,
you can skip it.
.SS STEP 4 \-> Sync
Run the \[dq]sync\[dq] command to resynchronize the array with the new disk.
.PP
.RS 4
snapraid sync
.PD 0
.PP
.PD
.RE
.PP
If everything is recovered, this command is immediate.
.SH COMMANDS
SnapRAID provides a few simple commands that allow to:
.PD 0
@ -550,25 +649,31 @@ Later modifications are not taken into account.
Nothing is modified.
.SS sync
Updates the parity information. All the modified files
in the disk array are read, and the parity data is
recomputed.
.PP
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the parity data is
recomputed for the whole file.
With inode you can move them on the disk without triggering
any parity recomputation.
in the disk array are read, and the corresponding parity
data is updated.
.PP
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
At the next run the \[dq]sync\[dq] process will start where
interrupted.
.PP
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad.
.PP
Files are identified by path and/or inode and checked by
size and timestamp.
If the file size or timestamp are different, the parity data
is recomputed for the whole file.
If the file is moved or renamed in the same disk, keeping the
same inode, the parity is no recomputed.
If the file is moved to another disk, the parity is recomputed,
but the previously compute hash information is kept.
.PP
The \[dq]content\[dq] and \[dq]parity\[dq] files are modified if necessary.
The files in the array are NOT modified.
.SS scrub
Scrubs the array, checking for silent errors in data or parity disks.
Scrubs the array, checking for silent or input/output errors in data
and parity disks.
.PP
For each command invocation, the 12% of the array is checked, but
nothing that it\'s more recent than 10 days.
@ -577,15 +682,15 @@ at least one time every two months.
.PP
You can use the \-p, \-\-percentage option to specify a different amount,
and the \-o, \-\-older\-than option to specify a different age in days.
You can have a full scrub with \[dq]\-p100 \-o0\[dq].
.PP
Any silent error identified is recorded in the content file,
and it\'s listed in the \[dq]status\[dq] command until it\'s fixed calling
\[dq]fix\[dq] and then \[dq]scrub\[dq].
You can have a full scrub with \[dq]\-p 100 \-o 0\[dq].
.PP
The oldest blocks are scrubbed first ensuring an optimal check.
Blocks already marked as bad are always checked, and if found
correct, they are automatically unmarked.
.PP
For any silent or input/output error found the corresponding blocks
are marked as bad in the \[dq]content\[dq] file.
These bad blocks are listed in \[dq]status\[dq], and can be fixed with \[dq]fix \-e\[dq].
After the fix, at the next scrub they will be rechecked, and if found
corrected, the bad mark will be removed.
.PP
It\'s recommended to run \[dq]scrub\[dq] on a synced array, to avoid to have
reported error caused by unsynced data. These errors are recognized
@ -602,18 +707,19 @@ The files in the array are NOT modified.
Fix all the files and the parity data.
.PP
All the files and the parity data are compared with the snapshot
state saved in the last \[dq]sync\[dq] command.
If a difference is found, it\'s reverted at the stored snapshot.
state saved in the last \[dq]sync\[dq].
If a difference is found, it\'s reverted to the stored snapshot.
.PP
Note that \[dq]fix\[dq] doesn\'t differentiate from errors or intentional
changes at files. It just reverts the state at the last \[dq]sync\[dq] command.
Note that \[dq]fix\[dq] doesn\'t differentiate between errors and intentional
modifications. It inconditionally reverts the file state at the last \[dq]sync\[dq].
.PP
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the \-v, \-\-verbose option.
If no other option is specified the full array is processed.
Use the filter options to select a subset of files or disks to operate on.
.PP
To only fix errors found with \[dq]scrub\[dq] and reported in \[dq]status\[dq],
you can use the \-e, \-\-filter\-error option.
To only fix the blocks marked bad during \[dq]sync\[dq] and \[dq]scrub\[dq],
use the \-e, \-\-filter\-error option.
As difference from other filter options, with this one fixes are
applied only to files that are not modified from the the latest \[dq]sync\[dq].
.PP
All the files that cannot be fixed are renamed adding
the \[dq].unrecoverable\[dq] extension.
@ -626,32 +732,28 @@ The files in the array are modified if necessary.
.SS check
Verify all the files and the parity data.
.PP
It works just like \[dq]fix\[dq], but it only simulates a recovery
and no change is written in the array.
It works like \[dq]fix\[dq], but it only simulates a recovery and no change
is written in the array.
.PP
This command is mostly intended for manual verifications,
like after a recovery process or in other special conditions.
For periodic and scheduled checks uses the \[dq]scrub\[dq] command.
For periodic and scheduled checks uses \[dq]scrub\[dq].
.PP
If you use the \-a, \-\-audit\-only option, only the file
data is checked, and the parity data is ignored for a
faster run.
.PP
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the \-v, \-\-verbose option.
.PP
Files are identified only by path, and not by inode.
.PP
Nothing is modified.
.SS list
Lists all the files contained in the array at the time of the
last \[dq]sync\[dq] command.
last \[dq]sync\[dq].
.PP
Nothing is modified.
.SS diff
Lists all the files modified from the last \[dq]sync\[dq] command that
need to have their parity data recomputed.
Lists all the files modified from the last \[dq]sync\[dq] that need to have
their parity data recomputed.
.PP
This command doesn\'t check the file data, but only the file timestamp
size and inode.
@ -686,13 +788,13 @@ If you are already using the optimal hash, this command
do nothing and tells you that nothing has to be done.
.PP
The rehash isn\'t done immediately, but it takes place
progressively during the \[dq]sync\[dq] and \[dq]scrub\[dq] commands.
progressively during \[dq]sync\[dq] and \[dq]scrub\[dq].
.PP
You can get the rehash state using the \[dq]status\[dq] command.
You can get the rehash state using \[dq]status\[dq].
.PP
During the rehash, SnapRAID maintains full functionality,
with the only exception of the \[dq]dup\[dq] command not able to detect
duplicated files using a different hash.
with the only exception of \[dq]dup\[dq] not able to detect duplicated
files using a different hash.
.SH OPTIONS
SnapRAID provides the following options:
.TP
@ -702,20 +804,18 @@ the file \[dq]/etc/snapraid.conf\[dq] in Unix, and \[dq]snapraid.conf\[dq] in
the current directory in Windows.
.TP
.B \-f, \-\-filter PATTERN
Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq]
commands.
Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq].
Only the files matching the entered pattern are processed.
This option can be used many times.
See the PATTERN section for more details in the
pattern specifications.
In Unix, ensure to quote globbing chars if used.
This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands.
This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq].
Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always
process the whole array.
.TP
.B \-d, \-\-filter\-disk NAME
Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq]
commands.
Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq].
Only the files present in the specified disk are processed.
You must specify a disk name as named in the configuration
file.
@ -724,52 +824,48 @@ option, to avoid to access other disks to check parity data.
If you combine more \-\-filter, \-\-filter\-disk and \-\-filter\-missing options,
only files matching all the set of filters are selected.
This option can be used many times.
This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands.
This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq].
Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always
process the whole array.
.TP
.B \-m, \-\-filter\-missing
Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq]
commands.
Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq].
Only the files missing/deleted from the array are processed.
When used with \[dq]fix\[dq], this is a kind of \[dq]undelete\[dq] command.
If you combine more \-\-filter, \-\-filter\-disk and \-\-filter\-missing options,
only files matching all the set of filters are selected.
This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands.
This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq].
Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always
process the whole array.
.TP
.B \-e, \-\-filter\-error
Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq]
commands.
It processes only the files containing blocks marked with silent
errors during the \[dq]sync\[dq] and \[dq]scrub\[dq] command, and listed in the
\[dq]status\[dq] command.
This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands.
Filters the blocks to process in \[dq]check\[dq] and \[dq]fix\[dq].
It processes only the blocks marked with silent or input/output
errors during \[dq]sync\[dq] and \[dq]scrub\[dq], and listed in \[dq]status\[dq].
This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq].
.TP
.B \-p, \-\-percentage PERC
Selects the part of the array to process in the \[dq]scrub\[dq] command.
Selects the part of the array to process in \[dq]scrub\[dq].
PERC is a numeric value from 0 to 100, default is 12.
When specifying 0, only the blocks marked as bad are scrubbed.
This option can be used only with the \[dq]scrub\[dq] command.
This option can be used only with \[dq]scrub\[dq].
.TP
.B \-o, \-\-older\-than DAYS
Selects the older the part of the array to process in the
\[dq]scrub\[dq] command.
Selects the older the part of the array to process in \[dq]scrub\[dq].
DAYS is the minimum age in days for a block to be scrubbed,
default is 10.
Blocks marked as bad are always scrubbed despite this option.
This option can be used only with the \[dq]scrub\[dq] command.
This option can be used only with \[dq]scrub\[dq].
.TP
.B \-a, \-\-audit\-only
In the \[dq]check\[dq] command verifies the hash of the files without
In \[dq]check\[dq] verifies the hash of the files without
doing any kind of check on the parity data.
If you are interested in checking only the file data this
option can speedup a lot the checking process.
This option can be used only with the \[dq]check\[dq] command.
This option can be used only with \[dq]check\[dq].
.TP
.B \-h, \-\-pre\-hash
In the \[dq]sync\[dq] command run a preliminary hashing phase of all
In \[dq]sync\[dq] runs a preliminary hashing phase of all
the new data to verify the data used in the parity computation.
Usually in \[dq]sync\[dq] no preliminary hashing is done, and the new
data is hashed just before the parity computation when it\'s read
@ -781,16 +877,16 @@ latent hardware problem, it\'s possible to encounter silent errors
what cannot be detected because the data is not yet hashed.
To avoid this risk, you can enable the \[dq]pre\-hash\[dq] mode and have
all the data hashed two times to ensure its integrity.
This option can be used only with the \[dq]sync\[dq] command.
This option can be used only with \[dq]sync\[dq].
.TP
.B \-i, \-\-import DIR
Imports from the specified directory any file that you deleted
from the array after the last \[dq]sync\[dq] command.
If you still have such files, they could be used by the \[dq]check\[dq]
and \[dq]fix\[dq] commands to improve the recover process.
from the array after the last \[dq]sync\[dq].
If you still have such files, they could be used by \[dq]check\[dq]
and \[dq]fix\[dq] to improve the recover process.
The files are read also in subdirectories and they are
identified regardless of their name.
This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands.
This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq].
.TP
.B \-Z, \-\-force\-zero
Forces the insecure operation of syncing a file with zero
@ -801,7 +897,7 @@ This allows to easily detect when after a system crash,
some accessed files were truncated.
This is a possible condition in Linux with the ext3/ext4
filesystems.
This option can be used only with the \[dq]sync\[dq] command.
This option can be used only with \[dq]sync\[dq].
.TP
.B \-E, \-\-force\-empty
Forces the insecure operation of syncing a disk with all
@ -811,7 +907,7 @@ in the disk are missing or rewritten, it stops proceeding
unless you specify this option.
This allows to easily detect when a data file\-system is not
mounted.
This option can be used only with the \[dq]sync\[dq] command.
This option can be used only with \[dq]sync\[dq].
.TP
.B \-U, \-\-force\-uuid
Forces the insecure operation of syncing, checking and fixing
@ -823,8 +919,8 @@ wrong mount points.
It\'s anyway allowed to have a single UUID change with
single parity, and more with multiple parity, because it\'s
the normal case of replacing disks after a recovery.
This option can be used only with the \[dq]sync\[dq], \[dq]check\[dq] or
\[dq]fix\[dq] command.
This option can be used only with \[dq]sync\[dq], \[dq]check\[dq] or
\[dq]fix\[dq].
.TP
.B \-D, \-\-force\-device
Forces the insecure operation of fixing with disks on the same
@ -834,11 +930,10 @@ it stops proceeding, because it\'s not a supported configuration.
But it could happen that you want to temporarily restore a lost
disk in the free space left in an already used disk. and this
option allows you to continue anyway.
This option can be used only with the \[dq]fix\[dq] command.
This option can be used only with \[dq]fix\[dq].
.TP
.B \-N, \-\-force\-nocopy
In the \[dq]sync\[dq], \[dq]check and \[dq]fix\[dq] commands, disables the copy
detection heuristic.
In \[dq]sync\[dq], \[dq]check and \[dq]fix\[dq], disables the copy detection heuristic.
Without this option SnapRAID assumes that files with same
attributes, like name, size and timestamp are copies with the
same data.
@ -848,18 +943,17 @@ to detect silent errors or to recover missing files.
This behavior, in some rare cases, may result in false positives,
or in a slow process due the many hash verifications, and this
option allows to resolve them.
This option can be used only with the \[dq]sync\[dq], \[dq]check\[dq] and \[dq]fix\[dq]
commands.
This option can be used only with \[dq]sync\[dq], \[dq]check\[dq] and \[dq]fix\[dq].
.TP
.B \-F, \-\-force\-full
In the \[dq]sync\[dq] command, forces a full rebuild of the parity.
In \[dq]sync\[dq] forces a full rebuild of the parity.
This option can be used when you reverted back to an old content
file, and you still have the parity data.
file, but using a more recent parity data.
Instead of recomputing the parity from scratch, this allows
to reuse the hashes present in the content file to validate data,
and to maintain data protection during the \[dq]sync\[dq] process using
the old content file and the parity data you have.
This option can be used only with the \[dq]sync\[dq] command.
This option can be used only with \[dq]sync\[dq].
.TP
.B \-l, \-\-log FILE
Write a detailed log of errors found in check and fix.
@ -1056,8 +1150,8 @@ wasted for each file. For example, with 100000 files and a 256 KiB
block size, you are going to waste 12 GiB of parity, that may result
in 12 GiB less space available in the data disk.
.PP
You can get the amount of wasted space in each disk using the \[dq]status\[dq]
command. This is the amount of space that you must leave free in the data
You can get the amount of wasted space in each disk using \[dq]status\[dq].
This is the amount of space that you must leave free in the data
disks, or use for files not included in the array.
.PP
To avoid to problem, you can use a bigger partition for parity.
@ -1333,99 +1427,6 @@ snapraid \-f \[dq]*.mp3\[dq] check
.PP
In Unix, when using globbing chars in the command line, you have to
quote them. Otherwise the shell will try to expand them.
.SH RECOVERING
The worst happened, and you lost a disk!
.PP
DO NOT PANIC! You will be able to recover it!
.PP
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync.
.PP
Then proceed with the following steps.
.SS STEP 1 \-> Reconfigure
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
.PP
Change the SnapRAID configuration file and make the \[dq]disk\[dq] option
of the failed disk to point to the place where you have enough empty
space to recover the files.
.PP
For example, if you have that disk \[dq]d1\[dq] failed, you can change:
.PP
.RS 4
disk d1 /mnt/disk1/
.PD 0
.PP
.PD
.RE
.PP
to:
.PP
.RS 4
disk d1 /mnt/new_spare_disk/
.PD 0
.PP
.PD
.RE
.SS STEP 2 \-> Fix
Run the fix command, storing the log in an external file with:
.PP
.RS 4
snapraid \-d NAME \-l fix.log fix
.PD 0
.PP
.PD
.RE
.PP
Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example.
.PP
This command will take a long time.
.PP
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
.PP
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the \[dq].unrecoverable\[dq] extension.
.PP
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with \[dq]unrecoverable:\[dq]
.PP
If you are not satisfied of the recovering, you can retry it as many time you wish.
For example, if you have remove some files after the last \[dq]sync\[dq], you can retry
to put them inplace, and retry the \[dq]fix\[dq].
.PP
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you will be no more able to retry the
\[dq]fix\[dq] command!
.SS STEP 3 \-> Check
As paranoid check, you can now run a \[dq]check\[dq] command to ensure that everything
is OK on the recovered disk.
.PP
.RS 4
snapraid \-d NAME \-a check
.PD 0
.PP
.PD
.RE
.PP
Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example.
.PP
The options \-d and \-a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
.PP
This command will take a long time.
.SS STEP 4 \-> Sync
Run the \[dq]sync\[dq] command to resynchronize the array with the new disk.
.PP
.RS 4
snapraid sync
.PD 0
.PP
.PD
.RE
.PP
If everything is recovered, this command is immediate.
.SH CONTENT
SnapRAID stores the list and checksums of your files in the content file.
.PP

375
snapraid.d
View file

@ -149,52 +149,31 @@ Getting Started
Now you can start using your array as you like, and periodically
update the parity information running the "sync" command.
Restoring and Undeleting
In recovering, SnapRAID is more like a backup program than a RAID system,
and it can be used to restore or undelete a file or directory to its previous
state using the -f, --filter option :
:snapraid fix -f FILE
or for a directory:
:snapraid fix -f DIR/
You can also use it to recover only accidentally deleted files inside
a directory using the -m, --filter-missing option, that restores
only missing files, leaving untouched all the others.
:snapraid fix -m -f DIR/
Or to recover all the deleted files in all the drives with:
:snapraid fix -m
If instead you have lost an entire disk and you want to recover it,
please see the 'Recovering' chapter.
Scrubbing
To periodically check the data and parity for errors, you can
run the "scrub" command.
:snapraid scrub
This command verifies only the oldest data in your array.
Every run of the command checks about 12% of the array,
but not newer than 10 days.
This command verifies the data in your array comparing it with
the hash computed in the "sync" command.
Every run of the command checks about 12% of the array, but not data newer
than 10 days.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
:snapraid -p 5 -o 20 scrub
If a silent or input error is found, the corresponding block is marked
as bad in the "content" file, and listed in the "status" command.
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad in the "content" file,
and listed in the "status" command.
:snapraid status
To fix them, you can use the "fix" command filtering for files containing
bad blocks:
To fix them, you can use the "fix" command filtering for bad blocks with
the -e, --filter-error options:
:snapraid -e fix
@ -204,7 +183,7 @@ Getting Started
:snapraid -p 0 scrub
Take care that running scrub on a not synced array may result in
Take care that running "scrub" on a not synced array may result in
errors caused by removed or modified files. These errors are reported
in the "scrub" result, but related blocks are not marked as bad.
@ -274,6 +253,106 @@ Getting Started
:fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
Undeleting
SnapRAID is more like a backup program than a RAID system, and it
can be used to restore or undelete files to their previous state using
the -f, --filter option :
:snapraid fix -f FILE
or for a directory:
:snapraid fix -f DIR/
You can also use it to recover only accidentally deleted files inside
a directory using the -m, --filter-missing option, that restores
only missing files, leaving untouched all the others.
:snapraid fix -m -f DIR/
Or to recover all the deleted files in all the drives with:
:snapraid fix -m
Recovering
The worst happened, and you lost a disk!
DO NOT PANIC! You will be able to recover it!
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync or scrub.
Then proceed with the following steps.
STEP 1 -> Reconfigure
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
Change the SnapRAID configuration file to make the "disk" option
of the failed disk to point to the place where you have enough empty
space to recover the files.
For example, if you have that disk "d1" failed, you can change from:
:disk d1 /mnt/disk1/
to:
:disk d1 /mnt/new_spare_disk/
STEP 2 -> Fix
Run the fix command, storing the log in an external file with:
:snapraid -d NAME -l fix.log fix
Where NAME is the name of the disk, like "d1" as in our previous example.
This command will take a long time.
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the ".unrecoverable" extension.
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with "unrecoverable:"
If you are not satisfied of the recovering, you can retry it as many
time you wish.
For example, if you have removed files from the array after the last
"sync", this may result in some other files not recovered.
In this case, you can retry the "fix" using the -i, --import option,
specifing where these files are now, to include them again in the
recovering process.
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you cannot retry the "fix" command
anymore!
STEP 3 -> Check
As paranoid check, you can now run a "check" command to ensure that
everything is OK on the recovered disk.
:snapraid -d NAME -a check
Where NAME is the name of the disk, like "d1" as in our previous example.
The options -d and -a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
This command will take a long time, but if you are not paranoid,
you can skip it.
STEP 4 -> Sync
Run the "sync" command to resynchronize the array with the new disk.
:snapraid sync
If everything is recovered, this command is immediate.
Commands
SnapRAID provides a few simple commands that allow to:
@ -298,26 +377,32 @@ Commands
sync
Updates the parity information. All the modified files
in the disk array are read, and the parity data is
recomputed.
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the parity data is
recomputed for the whole file.
With inode you can move them on the disk without triggering
any parity recomputation.
in the disk array are read, and the corresponding parity
data is updated.
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
At the next run the "sync" process will start where
interrupted.
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad.
Files are identified by path and/or inode and checked by
size and timestamp.
If the file size or timestamp are different, the parity data
is recomputed for the whole file.
If the file is moved or renamed in the same disk, keeping the
same inode, the parity is no recomputed.
If the file is moved to another disk, the parity is recomputed,
but the previously compute hash information is kept.
The "content" and "parity" files are modified if necessary.
The files in the array are NOT modified.
scrub
Scrubs the array, checking for silent errors in data or parity disks.
Scrubs the array, checking for silent or input/output errors in data
and parity disks.
For each command invocation, the 12% of the array is checked, but
nothing that it's more recent than 10 days.
@ -326,15 +411,15 @@ Commands
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days.
You can have a full scrub with "-p100 -o0".
Any silent error identified is recorded in the content file,
and it's listed in the "status" command until it's fixed calling
"fix" and then "scrub".
You can have a full scrub with "-p 100 -o 0".
The oldest blocks are scrubbed first ensuring an optimal check.
Blocks already marked as bad are always checked, and if found
correct, they are automatically unmarked.
For any silent or input/output error found the corresponding blocks
are marked as bad in the "content" file.
These bad blocks are listed in "status", and can be fixed with "fix -e".
After the fix, at the next scrub they will be rechecked, and if found
corrected, the bad mark will be removed.
It's recommended to run "scrub" on a synced array, to avoid to have
reported error caused by unsynced data. These errors are recognized
@ -352,18 +437,19 @@ Commands
Fix all the files and the parity data.
All the files and the parity data are compared with the snapshot
state saved in the last "sync" command.
If a difference is found, it's reverted at the stored snapshot.
state saved in the last "sync".
If a difference is found, it's reverted to the stored snapshot.
Note that "fix" doesn't differentiate from errors or intentional
changes at files. It just reverts the state at the last "sync" command.
Note that "fix" doesn't differentiate between errors and intentional
modifications. It inconditionally reverts the file state at the last "sync".
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the -v, --verbose option.
If no other option is specified the full array is processed.
Use the filter options to select a subset of files or disks to operate on.
To only fix errors found with "scrub" and reported in "status",
you can use the -e, --filter-error option.
To only fix the blocks marked bad during "sync" and "scrub",
use the -e, --filter-error option.
As difference from other filter options, with this one fixes are
applied only to files that are not modified from the the latest "sync".
All the files that cannot be fixed are renamed adding
the ".unrecoverable" extension.
@ -377,34 +463,30 @@ Commands
check
Verify all the files and the parity data.
It works just like "fix", but it only simulates a recovery
and no change is written in the array.
It works like "fix", but it only simulates a recovery and no change
is written in the array.
This command is mostly intended for manual verifications,
like after a recovery process or in other special conditions.
For periodic and scheduled checks uses the "scrub" command.
For periodic and scheduled checks uses "scrub".
If you use the -a, --audit-only option, only the file
data is checked, and the parity data is ignored for a
faster run.
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the -v, --verbose option.
Files are identified only by path, and not by inode.
Nothing is modified.
list
Lists all the files contained in the array at the time of the
last "sync" command.
last "sync".
Nothing is modified.
diff
Lists all the files modified from the last "sync" command that
need to have their parity data recomputed.
Lists all the files modified from the last "sync" that need to have
their parity data recomputed.
This command doesn't check the file data, but only the file timestamp
size and inode.
@ -442,13 +524,13 @@ Commands
do nothing and tells you that nothing has to be done.
The rehash isn't done immediately, but it takes place
progressively during the "sync" and "scrub" commands.
progressively during "sync" and "scrub".
You can get the rehash state using the "status" command.
You can get the rehash state using "status".
During the rehash, SnapRAID maintains full functionality,
with the only exception of the "dup" command not able to detect
duplicated files using a different hash.
with the only exception of "dup" not able to detect duplicated
files using a different hash.
Options
SnapRAID provides the following options:
@ -459,20 +541,18 @@ Options
the current directory in Windows.
-f, --filter PATTERN
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files matching the entered pattern are processed.
This option can be used many times.
See the PATTERN section for more details in the
pattern specifications.
In Unix, ensure to quote globbing chars if used.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-d, --filter-disk NAME
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files present in the specified disk are processed.
You must specify a disk name as named in the configuration
file.
@ -481,52 +561,48 @@ Options
If you combine more --filter, --filter-disk and --filter-missing options,
only files matching all the set of filters are selected.
This option can be used many times.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-m, --filter-missing
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files missing/deleted from the array are processed.
When used with "fix", this is a kind of "undelete" command.
If you combine more --filter, --filter-disk and --filter-missing options,
only files matching all the set of filters are selected.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-e, --filter-error
Filters the files to process in the "check" and "fix"
commands.
It processes only the files containing blocks marked with silent
errors during the "sync" and "scrub" command, and listed in the
"status" command.
This option can be used only with the "check" and "fix" commands.
Filters the blocks to process in "check" and "fix".
It processes only the blocks marked with silent or input/output
errors during "sync" and "scrub", and listed in "status".
This option can be used only with "check" and "fix".
-p, --percentage PERC
Selects the part of the array to process in the "scrub" command.
Selects the part of the array to process in "scrub".
PERC is a numeric value from 0 to 100, default is 12.
When specifying 0, only the blocks marked as bad are scrubbed.
This option can be used only with the "scrub" command.
This option can be used only with "scrub".
-o, --older-than DAYS
Selects the older the part of the array to process in the
"scrub" command.
Selects the older the part of the array to process in "scrub".
DAYS is the minimum age in days for a block to be scrubbed,
default is 10.
Blocks marked as bad are always scrubbed despite this option.
This option can be used only with the "scrub" command.
This option can be used only with "scrub".
-a, --audit-only
In the "check" command verifies the hash of the files without
In "check" verifies the hash of the files without
doing any kind of check on the parity data.
If you are interested in checking only the file data this
option can speedup a lot the checking process.
This option can be used only with the "check" command.
This option can be used only with "check".
-h, --pre-hash
In the "sync" command run a preliminary hashing phase of all
In "sync" runs a preliminary hashing phase of all
the new data to verify the data used in the parity computation.
Usually in "sync" no preliminary hashing is done, and the new
data is hashed just before the parity computation when it's read
@ -538,16 +614,16 @@ Options
what cannot be detected because the data is not yet hashed.
To avoid this risk, you can enable the "pre-hash" mode and have
all the data hashed two times to ensure its integrity.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-i, --import DIR
Imports from the specified directory any file that you deleted
from the array after the last "sync" command.
If you still have such files, they could be used by the "check"
and "fix" commands to improve the recover process.
from the array after the last "sync".
If you still have such files, they could be used by "check"
and "fix" to improve the recover process.
The files are read also in subdirectories and they are
identified regardless of their name.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
-Z, --force-zero
Forces the insecure operation of syncing a file with zero
@ -558,7 +634,7 @@ Options
some accessed files were truncated.
This is a possible condition in Linux with the ext3/ext4
filesystems.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-E, --force-empty
Forces the insecure operation of syncing a disk with all
@ -568,7 +644,7 @@ Options
unless you specify this option.
This allows to easily detect when a data file-system is not
mounted.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-U, --force-uuid
Forces the insecure operation of syncing, checking and fixing
@ -580,8 +656,8 @@ Options
It's anyway allowed to have a single UUID change with
single parity, and more with multiple parity, because it's
the normal case of replacing disks after a recovery.
This option can be used only with the "sync", "check" or
"fix" command.
This option can be used only with "sync", "check" or
"fix".
-D, --force-device
Forces the insecure operation of fixing with disks on the same
@ -591,11 +667,10 @@ Options
But it could happen that you want to temporarily restore a lost
disk in the free space left in an already used disk. and this
option allows you to continue anyway.
This option can be used only with the "fix" command.
This option can be used only with "fix".
-N, --force-nocopy
In the "sync", "check and "fix" commands, disables the copy
detection heuristic.
In "sync", "check and "fix", disables the copy detection heuristic.
Without this option SnapRAID assumes that files with same
attributes, like name, size and timestamp are copies with the
same data.
@ -605,18 +680,17 @@ Options
This behavior, in some rare cases, may result in false positives,
or in a slow process due the many hash verifications, and this
option allows to resolve them.
This option can be used only with the "sync", "check" and "fix"
commands.
This option can be used only with "sync", "check" and "fix".
-F, --force-full
In the "sync" command, forces a full rebuild of the parity.
In "sync" forces a full rebuild of the parity.
This option can be used when you reverted back to an old content
file, and you still have the parity data.
file, but using a more recent parity data.
Instead of recomputing the parity from scratch, this allows
to reuse the hashes present in the content file to validate data,
and to maintain data protection during the "sync" process using
the old content file and the parity data you have.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-l, --log FILE
Write a detailed log of errors found in check and fix.
@ -813,8 +887,8 @@ Configuration
block size, you are going to waste 12 GiB of parity, that may result
in 12 GiB less space available in the data disk.
You can get the amount of wasted space in each disk using the "status"
command. This is the amount of space that you must leave free in the data
You can get the amount of wasted space in each disk using "status".
This is the amount of space that you must leave free in the data
disks, or use for files not included in the array.
To avoid to problem, you can use a bigger partition for parity.
@ -965,79 +1039,6 @@ Pattern
In Unix, when using globbing chars in the command line, you have to
quote them. Otherwise the shell will try to expand them.
Recovering
The worst happened, and you lost a disk!
DO NOT PANIC! You will be able to recover it!
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync.
Then proceed with the following steps.
STEP 1 -> Reconfigure
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
Change the SnapRAID configuration file and make the "disk" option
of the failed disk to point to the place where you have enough empty
space to recover the files.
For example, if you have that disk "d1" failed, you can change:
:disk d1 /mnt/disk1/
to:
:disk d1 /mnt/new_spare_disk/
STEP 2 -> Fix
Run the fix command, storing the log in an external file with:
:snapraid -d NAME -l fix.log fix
Where NAME is the name of the disk, like "d1" as in our previous example.
This command will take a long time.
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the ".unrecoverable" extension.
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with "unrecoverable:"
If you are not satisfied of the recovering, you can retry it as many time you wish.
For example, if you have remove some files after the last "sync", you can retry
to put them inplace, and retry the "fix".
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you will be no more able to retry the
"fix" command!
STEP 3 -> Check
As paranoid check, you can now run a "check" command to ensure that everything
is OK on the recovered disk.
:snapraid -d NAME -a check
Where NAME is the name of the disk, like "d1" as in our previous example.
The options -d and -a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
This command will take a long time.
STEP 4 -> Sync
Run the "sync" command to resynchronize the array with the new disk.
:snapraid sync
If everything is recovered, this command is immediate.
Content
SnapRAID stores the list and checksums of your files in the content file.

408
snapraid.txt
View file

@ -162,33 +162,7 @@ When this command completes, your data is SAFE.
Now you can start using your array as you like, and periodically
update the parity information running the "sync" command.
4.1 Restoring and Undeleting
----------------------------
In recovering, SnapRAID is more like a backup program than a RAID system,
and it can be used to restore or undelete a file or directory to its previous
state using the -f, --filter option :
snapraid fix -f FILE
or for a directory:
snapraid fix -f DIR/
You can also use it to recover only accidentally deleted files inside
a directory using the -m, --filter-missing option, that restores
only missing files, leaving untouched all the others.
snapraid fix -m -f DIR/
Or to recover all the deleted files in all the drives with:
snapraid fix -m
If instead you have lost an entire disk and you want to recover it,
please see the 'Recovering' chapter.
4.2 Scrubbing
4.1 Scrubbing
-------------
To periodically check the data and parity for errors, you can
@ -196,22 +170,25 @@ run the "scrub" command.
snapraid scrub
This command verifies only the oldest data in your array.
Every run of the command checks about 12% of the array,
but not newer than 10 days.
This command verifies the data in your array comparing it with
the hash computed in the "sync" command.
Every run of the command checks about 12% of the array, but not data newer
than 10 days.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
snapraid -p 5 -o 20 scrub
If a silent or input error is found, the corresponding block is marked
as bad in the "content" file, and listed in the "status" command.
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad in the "content" file,
and listed in the "status" command.
snapraid status
To fix them, you can use the "fix" command filtering for files containing
bad blocks:
To fix them, you can use the "fix" command filtering for bad blocks with
the -e, --filter-error options:
snapraid -e fix
@ -221,11 +198,11 @@ marked as bad.
snapraid -p 0 scrub
Take care that running scrub on a not synced array may result in
Take care that running "scrub" on a not synced array may result in
errors caused by removed or modified files. These errors are reported
in the "scrub" result, but related blocks are not marked as bad.
4.3 Pooling
4.2 Pooling
-----------
To have all the files in your array shown in the same directory tree,
@ -293,6 +270,110 @@ remote symlinks with the command:
fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
4.3 Undeleting
--------------
SnapRAID is more like a backup program than a RAID system, and it
can be used to restore or undelete files to their previous state using
the -f, --filter option :
snapraid fix -f FILE
or for a directory:
snapraid fix -f DIR/
You can also use it to recover only accidentally deleted files inside
a directory using the -m, --filter-missing option, that restores
only missing files, leaving untouched all the others.
snapraid fix -m -f DIR/
Or to recover all the deleted files in all the drives with:
snapraid fix -m
4.4 Recovering
--------------
The worst happened, and you lost a disk!
DO NOT PANIC! You will be able to recover it!
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync or scrub.
Then proceed with the following steps.
---- 4.4.1 STEP 1 -> Reconfigure ----
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
Change the SnapRAID configuration file to make the "disk" option
of the failed disk to point to the place where you have enough empty
space to recover the files.
For example, if you have that disk "d1" failed, you can change from:
disk d1 /mnt/disk1/
to:
disk d1 /mnt/new_spare_disk/
---- 4.4.2 STEP 2 -> Fix ----
Run the fix command, storing the log in an external file with:
snapraid -d NAME -l fix.log fix
Where NAME is the name of the disk, like "d1" as in our previous example.
This command will take a long time.
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the ".unrecoverable" extension.
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with "unrecoverable:"
If you are not satisfied of the recovering, you can retry it as many
time you wish.
For example, if you have removed files from the array after the last
"sync", this may result in some other files not recovered.
In this case, you can retry the "fix" using the -i, --import option,
specifing where these files are now, to include them again in the
recovering process.
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you cannot retry the "fix" command
anymore!
---- 4.4.3 STEP 3 -> Check ----
As paranoid check, you can now run a "check" command to ensure that
everything is OK on the recovered disk.
snapraid -d NAME -a check
Where NAME is the name of the disk, like "d1" as in our previous example.
The options -d and -a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
This command will take a long time, but if you are not paranoid,
you can skip it.
---- 4.4.4 STEP 4 -> Sync ----
Run the "sync" command to resynchronize the array with the new disk.
snapraid sync
If everything is recovered, this command is immediate.
5 COMMANDS
==========
@ -324,28 +405,34 @@ Nothing is modified.
--------
Updates the parity information. All the modified files
in the disk array are read, and the parity data is
recomputed.
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the parity data is
recomputed for the whole file.
With inode you can move them on the disk without triggering
any parity recomputation.
in the disk array are read, and the corresponding parity
data is updated.
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
At the next run the "sync" process will start where
interrupted.
If during the process, silent or input/output errors are found,
the corresponding blocks are marked as bad.
Files are identified by path and/or inode and checked by
size and timestamp.
If the file size or timestamp are different, the parity data
is recomputed for the whole file.
If the file is moved or renamed in the same disk, keeping the
same inode, the parity is no recomputed.
If the file is moved to another disk, the parity is recomputed,
but the previously compute hash information is kept.
The "content" and "parity" files are modified if necessary.
The files in the array are NOT modified.
5.3 scrub
---------
Scrubs the array, checking for silent errors in data or parity disks.
Scrubs the array, checking for silent or input/output errors in data
and parity disks.
For each command invocation, the 12% of the array is checked, but
nothing that it's more recent than 10 days.
@ -354,15 +441,15 @@ at least one time every two months.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days.
You can have a full scrub with "-p100 -o0".
Any silent error identified is recorded in the content file,
and it's listed in the "status" command until it's fixed calling
"fix" and then "scrub".
You can have a full scrub with "-p 100 -o 0".
The oldest blocks are scrubbed first ensuring an optimal check.
Blocks already marked as bad are always checked, and if found
correct, they are automatically unmarked.
For any silent or input/output error found the corresponding blocks
are marked as bad in the "content" file.
These bad blocks are listed in "status", and can be fixed with "fix -e".
After the fix, at the next scrub they will be rechecked, and if found
corrected, the bad mark will be removed.
It's recommended to run "scrub" on a synced array, to avoid to have
reported error caused by unsynced data. These errors are recognized
@ -382,18 +469,19 @@ The files in the array are NOT modified.
Fix all the files and the parity data.
All the files and the parity data are compared with the snapshot
state saved in the last "sync" command.
If a difference is found, it's reverted at the stored snapshot.
state saved in the last "sync".
If a difference is found, it's reverted to the stored snapshot.
Note that "fix" doesn't differentiate from errors or intentional
changes at files. It just reverts the state at the last "sync" command.
Note that "fix" doesn't differentiate between errors and intentional
modifications. It inconditionally reverts the file state at the last "sync".
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the -v, --verbose option.
If no other option is specified the full array is processed.
Use the filter options to select a subset of files or disks to operate on.
To only fix errors found with "scrub" and reported in "status",
you can use the -e, --filter-error option.
To only fix the blocks marked bad during "sync" and "scrub",
use the -e, --filter-error option.
As difference from other filter options, with this one fixes are
applied only to files that are not modified from the the latest "sync".
All the files that cannot be fixed are renamed adding
the ".unrecoverable" extension.
@ -409,21 +497,17 @@ The files in the array are modified if necessary.
Verify all the files and the parity data.
It works just like "fix", but it only simulates a recovery
and no change is written in the array.
It works like "fix", but it only simulates a recovery and no change
is written in the array.
This command is mostly intended for manual verifications,
like after a recovery process or in other special conditions.
For periodic and scheduled checks uses the "scrub" command.
For periodic and scheduled checks uses "scrub".
If you use the -a, --audit-only option, only the file
data is checked, and the parity data is ignored for a
faster run.
By default the full array is processed. Use the filter options
to select a subset of files or disks to operate on.
To know that file is checked, you can use the -v, --verbose option.
Files are identified only by path, and not by inode.
Nothing is modified.
@ -432,15 +516,15 @@ Nothing is modified.
--------
Lists all the files contained in the array at the time of the
last "sync" command.
last "sync".
Nothing is modified.
5.7 diff
--------
Lists all the files modified from the last "sync" command that
need to have their parity data recomputed.
Lists all the files modified from the last "sync" that need to have
their parity data recomputed.
This command doesn't check the file data, but only the file timestamp
size and inode.
@ -484,13 +568,13 @@ If you are already using the optimal hash, this command
do nothing and tells you that nothing has to be done.
The rehash isn't done immediately, but it takes place
progressively during the "sync" and "scrub" commands.
progressively during "sync" and "scrub".
You can get the rehash state using the "status" command.
You can get the rehash state using "status".
During the rehash, SnapRAID maintains full functionality,
with the only exception of the "dup" command not able to detect
duplicated files using a different hash.
with the only exception of "dup" not able to detect duplicated
files using a different hash.
6 OPTIONS
@ -504,20 +588,18 @@ SnapRAID provides the following options:
the current directory in Windows.
-f, --filter PATTERN
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files matching the entered pattern are processed.
This option can be used many times.
See the PATTERN section for more details in the
pattern specifications.
In Unix, ensure to quote globbing chars if used.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-d, --filter-disk NAME
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files present in the specified disk are processed.
You must specify a disk name as named in the configuration
file.
@ -526,52 +608,48 @@ SnapRAID provides the following options:
If you combine more --filter, --filter-disk and --filter-missing options,
only files matching all the set of filters are selected.
This option can be used many times.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-m, --filter-missing
Filters the files to process in the "check" and "fix"
commands.
Filters the files to process in "check" and "fix".
Only the files missing/deleted from the array are processed.
When used with "fix", this is a kind of "undelete" command.
If you combine more --filter, --filter-disk and --filter-missing options,
only files matching all the set of filters are selected.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
Note that it cannot be used with "sync" and "scrub", because they always
process the whole array.
-e, --filter-error
Filters the files to process in the "check" and "fix"
commands.
It processes only the files containing blocks marked with silent
errors during the "sync" and "scrub" command, and listed in the
"status" command.
This option can be used only with the "check" and "fix" commands.
Filters the blocks to process in "check" and "fix".
It processes only the blocks marked with silent or input/output
errors during "sync" and "scrub", and listed in "status".
This option can be used only with "check" and "fix".
-p, --percentage PERC
Selects the part of the array to process in the "scrub" command.
Selects the part of the array to process in "scrub".
PERC is a numeric value from 0 to 100, default is 12.
When specifying 0, only the blocks marked as bad are scrubbed.
This option can be used only with the "scrub" command.
This option can be used only with "scrub".
-o, --older-than DAYS
Selects the older the part of the array to process in the
"scrub" command.
Selects the older the part of the array to process in "scrub".
DAYS is the minimum age in days for a block to be scrubbed,
default is 10.
Blocks marked as bad are always scrubbed despite this option.
This option can be used only with the "scrub" command.
This option can be used only with "scrub".
-a, --audit-only
In the "check" command verifies the hash of the files without
In "check" verifies the hash of the files without
doing any kind of check on the parity data.
If you are interested in checking only the file data this
option can speedup a lot the checking process.
This option can be used only with the "check" command.
This option can be used only with "check".
-h, --pre-hash
In the "sync" command run a preliminary hashing phase of all
In "sync" runs a preliminary hashing phase of all
the new data to verify the data used in the parity computation.
Usually in "sync" no preliminary hashing is done, and the new
data is hashed just before the parity computation when it's read
@ -583,16 +661,16 @@ SnapRAID provides the following options:
what cannot be detected because the data is not yet hashed.
To avoid this risk, you can enable the "pre-hash" mode and have
all the data hashed two times to ensure its integrity.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-i, --import DIR
Imports from the specified directory any file that you deleted
from the array after the last "sync" command.
If you still have such files, they could be used by the "check"
and "fix" commands to improve the recover process.
from the array after the last "sync".
If you still have such files, they could be used by "check"
and "fix" to improve the recover process.
The files are read also in subdirectories and they are
identified regardless of their name.
This option can be used only with the "check" and "fix" commands.
This option can be used only with "check" and "fix".
-Z, --force-zero
Forces the insecure operation of syncing a file with zero
@ -603,7 +681,7 @@ SnapRAID provides the following options:
some accessed files were truncated.
This is a possible condition in Linux with the ext3/ext4
filesystems.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-E, --force-empty
Forces the insecure operation of syncing a disk with all
@ -613,7 +691,7 @@ SnapRAID provides the following options:
unless you specify this option.
This allows to easily detect when a data file-system is not
mounted.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-U, --force-uuid
Forces the insecure operation of syncing, checking and fixing
@ -625,8 +703,8 @@ SnapRAID provides the following options:
It's anyway allowed to have a single UUID change with
single parity, and more with multiple parity, because it's
the normal case of replacing disks after a recovery.
This option can be used only with the "sync", "check" or
"fix" command.
This option can be used only with "sync", "check" or
"fix".
-D, --force-device
Forces the insecure operation of fixing with disks on the same
@ -636,11 +714,10 @@ SnapRAID provides the following options:
But it could happen that you want to temporarily restore a lost
disk in the free space left in an already used disk. and this
option allows you to continue anyway.
This option can be used only with the "fix" command.
This option can be used only with "fix".
-N, --force-nocopy
In the "sync", "check and "fix" commands, disables the copy
detection heuristic.
In "sync", "check and "fix", disables the copy detection heuristic.
Without this option SnapRAID assumes that files with same
attributes, like name, size and timestamp are copies with the
same data.
@ -650,18 +727,17 @@ SnapRAID provides the following options:
This behavior, in some rare cases, may result in false positives,
or in a slow process due the many hash verifications, and this
option allows to resolve them.
This option can be used only with the "sync", "check" and "fix"
commands.
This option can be used only with "sync", "check" and "fix".
-F, --force-full
In the "sync" command, forces a full rebuild of the parity.
In "sync" forces a full rebuild of the parity.
This option can be used when you reverted back to an old content
file, and you still have the parity data.
file, but using a more recent parity data.
Instead of recomputing the parity from scratch, this allows
to reuse the hashes present in the content file to validate data,
and to maintain data protection during the "sync" process using
the old content file and the parity data you have.
This option can be used only with the "sync" command.
This option can be used only with "sync".
-l, --log FILE
Write a detailed log of errors found in check and fix.
@ -877,8 +953,8 @@ wasted for each file. For example, with 100000 files and a 256 KiB
block size, you are going to waste 12 GiB of parity, that may result
in 12 GiB less space available in the data disk.
You can get the amount of wasted space in each disk using the "status"
command. This is the amount of space that you must leave free in the data
You can get the amount of wasted space in each disk using "status".
This is the amount of space that you must leave free in the data
disks, or use for files not included in the array.
To avoid to problem, you can use a bigger partition for parity.
@ -1041,92 +1117,8 @@ In Unix, when using globbing chars in the command line, you have to
quote them. Otherwise the shell will try to expand them.
9 RECOVERING
============
The worst happened, and you lost a disk!
DO NOT PANIC! You will be able to recover it!
The first thing you have to do is to avoid further changes at you disk array.
Disable any remote connection to it, any scheduled process, including any
scheduled SnapRAID nightly sync.
Then proceed with the following steps.
9.1 STEP 1 -> Reconfigure
-------------------------
You need some space to recover, even better if you already have an additional
disk, but in case, also an external USB or remote disk is enough.
Change the SnapRAID configuration file and make the "disk" option
of the failed disk to point to the place where you have enough empty
space to recover the files.
For example, if you have that disk "d1" failed, you can change:
disk d1 /mnt/disk1/
to:
disk d1 /mnt/new_spare_disk/
9.2 STEP 2 -> Fix
-----------------
Run the fix command, storing the log in an external file with:
snapraid -d NAME -l fix.log fix
Where NAME is the name of the disk, like "d1" as in our previous example.
This command will take a long time.
Take care that you need also few gigabytes free to store the fix.log file.
Run it from a disk with some free space.
Now you have recovered all the recoverable. If some file is partially or totally
unrecoverable, it will be renamed adding the ".unrecoverable" extension.
You can get a detailed list of all the unrecoverable blocks in the fix.log file
checking all the lines starting with "unrecoverable:"
If you are not satisfied of the recovering, you can retry it as many time you wish.
For example, if you have remove some files after the last "sync", you can retry
to put them inplace, and retry the "fix".
If you are satisfied of the recovering, you can now proceed further,
but take care that after syncing you will be no more able to retry the
"fix" command!
9.3 STEP 3 -> Check
-------------------
As paranoid check, you can now run a "check" command to ensure that everything
is OK on the recovered disk.
snapraid -d NAME -a check
Where NAME is the name of the disk, like "d1" as in our previous example.
The options -d and -a tell SnapRAID to check only the specified disk,
and ignore all the parity data.
This command will take a long time.
9.4 STEP 4 -> Sync
------------------
Run the "sync" command to resynchronize the array with the new disk.
snapraid sync
If everything is recovered, this command is immediate.
10 CONTENT
==========
9 CONTENT
=========
SnapRAID stores the list and checksums of your files in the content file.
@ -1137,7 +1129,7 @@ This file is read and written by the "sync" and "scrub" commands, and
read by "fix", "check" and "status".
11 PARITY
10 PARITY
=========
SnapRAID stores the parity information of your array in the parity
@ -1150,7 +1142,7 @@ These files are read and written by the "sync" and "fix" commands, and
only read by "scrub" and "check".
12 ENCODING
11 ENCODING
===========
SnapRAID in Unix ignores any encoding. It reads and stores the
@ -1167,13 +1159,13 @@ redirect the console output to a file, the resulting file is always
in the UTF-8 format.
13 COPYRIGHT
12 COPYRIGHT
============
This file is Copyright (C) 2011 Andrea Mazzoleni
14 SEE ALSO
13 SEE ALSO
===========
rsync(1)