From 4885a38c8af77d522f88f7a841b5f9a12470f9e5 Mon Sep 17 00:00:00 2001 From: Andrea Mazzoleni Date: Tue, 30 Dec 2014 18:30:11 +0100 Subject: [PATCH] Various manual updates. --- snapraid.1 | 453 ++++++++++++++++++++++++++------------------------- snapraid.d | 375 +++++++++++++++++++++--------------------- snapraid.txt | 408 +++++++++++++++++++++++----------------------- 3 files changed, 615 insertions(+), 621 deletions(-) diff --git a/snapraid.1 b/snapraid.1 index 4f973196..da304019 100644 --- a/snapraid.1 +++ b/snapraid.1 @@ -271,49 +271,6 @@ When this command completes, your data is SAFE. .PP Now you can start using your array as you like, and periodically update the parity information running the \[dq]sync\[dq] command. -.SS Restoring and Undeleting -In recovering, SnapRAID is more like a backup program than a RAID system, -and it can be used to restore or undelete a file or directory to its previous -state using the \-f, \-\-filter option : -.PP -.RS 4 -snapraid fix \-f FILE -.PD 0 -.PP -.PD -.RE -.PP -or for a directory: -.PP -.RS 4 -snapraid fix \-f DIR/ -.PD 0 -.PP -.PD -.RE -.PP -You can also use it to recover only accidentally deleted files inside -a directory using the \-m, \-\-filter\-missing option, that restores -only missing files, leaving untouched all the others. -.PP -.RS 4 -snapraid fix \-m \-f DIR/ -.PD 0 -.PP -.PD -.RE -.PP -Or to recover all the deleted files in all the drives with: -.PP -.RS 4 -snapraid fix \-m -.PD 0 -.PP -.PD -.RE -.PP -If instead you have lost an entire disk and you want to recover it, -please see the \'Recovering\' chapter. .SS Scrubbing To periodically check the data and parity for errors, you can run the \[dq]scrub\[dq] command. @@ -325,9 +282,11 @@ snapraid scrub .PD .RE .PP -This command verifies only the oldest data in your array. -Every run of the command checks about 12% of the array, -but not newer than 10 days. +This command verifies the data in your array comparing it with +the hash computed in the \[dq]sync\[dq] command. +.PP +Every run of the command checks about 12% of the array, but not data newer +than 10 days. You can use the \-p, \-\-percentage option to specify a different amount, and the \-o, \-\-older\-than option to specify a different age in days. For example, to check 5% of the array older than 20 days use: @@ -339,8 +298,9 @@ snapraid \-p 5 \-o 20 scrub .PD .RE .PP -If a silent or input error is found, the corresponding block is marked -as bad in the \[dq]content\[dq] file, and listed in the \[dq]status\[dq] command. +If during the process, silent or input/output errors are found, +the corresponding blocks are marked as bad in the \[dq]content\[dq] file, +and listed in the \[dq]status\[dq] command. .PP .RS 4 snapraid status @@ -349,8 +309,8 @@ snapraid status .PD .RE .PP -To fix them, you can use the \[dq]fix\[dq] command filtering for files containing -bad blocks: +To fix them, you can use the \[dq]fix\[dq] command filtering for bad blocks with +the \-e, \-\-filter\-error options: .PP .RS 4 snapraid \-e fix @@ -370,7 +330,7 @@ snapraid \-p 0 scrub .PD .RE .PP -Take care that running scrub on a not synced array may result in +Take care that running \[dq]scrub\[dq] on a not synced array may result in errors caused by removed or modified files. These errors are reported in the \[dq]scrub\[dq] result, but related blocks are not marked as bad. .SS Pooling @@ -523,6 +483,145 @@ fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1 .PP .PD .RE +.SS Undeleting +SnapRAID is more like a backup program than a RAID system, and it +can be used to restore or undelete files to their previous state using +the \-f, \-\-filter option : +.PP +.RS 4 +snapraid fix \-f FILE +.PD 0 +.PP +.PD +.RE +.PP +or for a directory: +.PP +.RS 4 +snapraid fix \-f DIR/ +.PD 0 +.PP +.PD +.RE +.PP +You can also use it to recover only accidentally deleted files inside +a directory using the \-m, \-\-filter\-missing option, that restores +only missing files, leaving untouched all the others. +.PP +.RS 4 +snapraid fix \-m \-f DIR/ +.PD 0 +.PP +.PD +.RE +.PP +Or to recover all the deleted files in all the drives with: +.PP +.RS 4 +snapraid fix \-m +.PD 0 +.PP +.PD +.RE +.SS Recovering +The worst happened, and you lost a disk! +.PP +DO NOT PANIC! You will be able to recover it! +.PP +The first thing you have to do is to avoid further changes at you disk array. +Disable any remote connection to it, any scheduled process, including any +scheduled SnapRAID nightly sync or scrub. +.PP +Then proceed with the following steps. +.SS STEP 1 \-> Reconfigure +You need some space to recover, even better if you already have an additional +disk, but in case, also an external USB or remote disk is enough. +.PP +Change the SnapRAID configuration file to make the \[dq]disk\[dq] option +of the failed disk to point to the place where you have enough empty +space to recover the files. +.PP +For example, if you have that disk \[dq]d1\[dq] failed, you can change from: +.PP +.RS 4 +disk d1 /mnt/disk1/ +.PD 0 +.PP +.PD +.RE +.PP +to: +.PP +.RS 4 +disk d1 /mnt/new_spare_disk/ +.PD 0 +.PP +.PD +.RE +.SS STEP 2 \-> Fix +Run the fix command, storing the log in an external file with: +.PP +.RS 4 +snapraid \-d NAME \-l fix.log fix +.PD 0 +.PP +.PD +.RE +.PP +Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example. +.PP +This command will take a long time. +.PP +Take care that you need also few gigabytes free to store the fix.log file. +Run it from a disk with some free space. +.PP +Now you have recovered all the recoverable. If some file is partially or totally +unrecoverable, it will be renamed adding the \[dq].unrecoverable\[dq] extension. +.PP +You can get a detailed list of all the unrecoverable blocks in the fix.log file +checking all the lines starting with \[dq]unrecoverable:\[dq] +.PP +If you are not satisfied of the recovering, you can retry it as many +time you wish. +.PP +For example, if you have removed files from the array after the last +\[dq]sync\[dq], this may result in some other files not recovered. +In this case, you can retry the \[dq]fix\[dq] using the \-i, \-\-import option, +specifing where these files are now, to include them again in the +recovering process. +.PP +If you are satisfied of the recovering, you can now proceed further, +but take care that after syncing you cannot retry the \[dq]fix\[dq] command +anymore! +.SS STEP 3 \-> Check +As paranoid check, you can now run a \[dq]check\[dq] command to ensure that +everything is OK on the recovered disk. +.PP +.RS 4 +snapraid \-d NAME \-a check +.PD 0 +.PP +.PD +.RE +.PP +Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example. +.PP +The options \-d and \-a tell SnapRAID to check only the specified disk, +and ignore all the parity data. +.PP +This command will take a long time, but if you are not paranoid, +you can skip it. +.SS STEP 4 \-> Sync +Run the \[dq]sync\[dq] command to resynchronize the array with the new disk. +.PP +.RS 4 +snapraid sync +.PD 0 +.PP +.PD +.RE +.PP +If everything is recovered, this command is immediate. .SH COMMANDS SnapRAID provides a few simple commands that allow to: .PD 0 @@ -550,25 +649,31 @@ Later modifications are not taken into account. Nothing is modified. .SS sync Updates the parity information. All the modified files -in the disk array are read, and the parity data is -recomputed. -.PP -Files are identified by path and/or inode and checked by -size and timestamp. -If the size or timestamp are different, the parity data is -recomputed for the whole file. -With inode you can move them on the disk without triggering -any parity recomputation. +in the disk array are read, and the corresponding parity +data is updated. .PP You can stop this process at any time pressing Ctrl+C, without losing the work already done. At the next run the \[dq]sync\[dq] process will start where interrupted. .PP +If during the process, silent or input/output errors are found, +the corresponding blocks are marked as bad. +.PP +Files are identified by path and/or inode and checked by +size and timestamp. +If the file size or timestamp are different, the parity data +is recomputed for the whole file. +If the file is moved or renamed in the same disk, keeping the +same inode, the parity is no recomputed. +If the file is moved to another disk, the parity is recomputed, +but the previously compute hash information is kept. +.PP The \[dq]content\[dq] and \[dq]parity\[dq] files are modified if necessary. The files in the array are NOT modified. .SS scrub -Scrubs the array, checking for silent errors in data or parity disks. +Scrubs the array, checking for silent or input/output errors in data +and parity disks. .PP For each command invocation, the 12% of the array is checked, but nothing that it\'s more recent than 10 days. @@ -577,15 +682,15 @@ at least one time every two months. .PP You can use the \-p, \-\-percentage option to specify a different amount, and the \-o, \-\-older\-than option to specify a different age in days. -You can have a full scrub with \[dq]\-p100 \-o0\[dq]. -.PP -Any silent error identified is recorded in the content file, -and it\'s listed in the \[dq]status\[dq] command until it\'s fixed calling -\[dq]fix\[dq] and then \[dq]scrub\[dq]. +You can have a full scrub with \[dq]\-p 100 \-o 0\[dq]. .PP The oldest blocks are scrubbed first ensuring an optimal check. -Blocks already marked as bad are always checked, and if found -correct, they are automatically unmarked. +.PP +For any silent or input/output error found the corresponding blocks +are marked as bad in the \[dq]content\[dq] file. +These bad blocks are listed in \[dq]status\[dq], and can be fixed with \[dq]fix \-e\[dq]. +After the fix, at the next scrub they will be rechecked, and if found +corrected, the bad mark will be removed. .PP It\'s recommended to run \[dq]scrub\[dq] on a synced array, to avoid to have reported error caused by unsynced data. These errors are recognized @@ -602,18 +707,19 @@ The files in the array are NOT modified. Fix all the files and the parity data. .PP All the files and the parity data are compared with the snapshot -state saved in the last \[dq]sync\[dq] command. -If a difference is found, it\'s reverted at the stored snapshot. +state saved in the last \[dq]sync\[dq]. +If a difference is found, it\'s reverted to the stored snapshot. .PP -Note that \[dq]fix\[dq] doesn\'t differentiate from errors or intentional -changes at files. It just reverts the state at the last \[dq]sync\[dq] command. +Note that \[dq]fix\[dq] doesn\'t differentiate between errors and intentional +modifications. It inconditionally reverts the file state at the last \[dq]sync\[dq]. .PP -By default the full array is processed. Use the filter options -to select a subset of files or disks to operate on. -To know that file is checked, you can use the \-v, \-\-verbose option. +If no other option is specified the full array is processed. +Use the filter options to select a subset of files or disks to operate on. .PP -To only fix errors found with \[dq]scrub\[dq] and reported in \[dq]status\[dq], -you can use the \-e, \-\-filter\-error option. +To only fix the blocks marked bad during \[dq]sync\[dq] and \[dq]scrub\[dq], +use the \-e, \-\-filter\-error option. +As difference from other filter options, with this one fixes are +applied only to files that are not modified from the the latest \[dq]sync\[dq]. .PP All the files that cannot be fixed are renamed adding the \[dq].unrecoverable\[dq] extension. @@ -626,32 +732,28 @@ The files in the array are modified if necessary. .SS check Verify all the files and the parity data. .PP -It works just like \[dq]fix\[dq], but it only simulates a recovery -and no change is written in the array. +It works like \[dq]fix\[dq], but it only simulates a recovery and no change +is written in the array. .PP This command is mostly intended for manual verifications, like after a recovery process or in other special conditions. -For periodic and scheduled checks uses the \[dq]scrub\[dq] command. +For periodic and scheduled checks uses \[dq]scrub\[dq]. .PP If you use the \-a, \-\-audit\-only option, only the file data is checked, and the parity data is ignored for a faster run. .PP -By default the full array is processed. Use the filter options -to select a subset of files or disks to operate on. -To know that file is checked, you can use the \-v, \-\-verbose option. -.PP Files are identified only by path, and not by inode. .PP Nothing is modified. .SS list Lists all the files contained in the array at the time of the -last \[dq]sync\[dq] command. +last \[dq]sync\[dq]. .PP Nothing is modified. .SS diff -Lists all the files modified from the last \[dq]sync\[dq] command that -need to have their parity data recomputed. +Lists all the files modified from the last \[dq]sync\[dq] that need to have +their parity data recomputed. .PP This command doesn\'t check the file data, but only the file timestamp size and inode. @@ -686,13 +788,13 @@ If you are already using the optimal hash, this command do nothing and tells you that nothing has to be done. .PP The rehash isn\'t done immediately, but it takes place -progressively during the \[dq]sync\[dq] and \[dq]scrub\[dq] commands. +progressively during \[dq]sync\[dq] and \[dq]scrub\[dq]. .PP -You can get the rehash state using the \[dq]status\[dq] command. +You can get the rehash state using \[dq]status\[dq]. .PP During the rehash, SnapRAID maintains full functionality, -with the only exception of the \[dq]dup\[dq] command not able to detect -duplicated files using a different hash. +with the only exception of \[dq]dup\[dq] not able to detect duplicated +files using a different hash. .SH OPTIONS SnapRAID provides the following options: .TP @@ -702,20 +804,18 @@ the file \[dq]/etc/snapraid.conf\[dq] in Unix, and \[dq]snapraid.conf\[dq] in the current directory in Windows. .TP .B \-f, \-\-filter PATTERN -Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq] -commands. +Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq]. Only the files matching the entered pattern are processed. This option can be used many times. See the PATTERN section for more details in the pattern specifications. In Unix, ensure to quote globbing chars if used. -This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands. +This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq]. Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always process the whole array. .TP .B \-d, \-\-filter\-disk NAME -Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq] -commands. +Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq]. Only the files present in the specified disk are processed. You must specify a disk name as named in the configuration file. @@ -724,52 +824,48 @@ option, to avoid to access other disks to check parity data. If you combine more \-\-filter, \-\-filter\-disk and \-\-filter\-missing options, only files matching all the set of filters are selected. This option can be used many times. -This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands. +This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq]. Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always process the whole array. .TP .B \-m, \-\-filter\-missing -Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq] -commands. +Filters the files to process in \[dq]check\[dq] and \[dq]fix\[dq]. Only the files missing/deleted from the array are processed. When used with \[dq]fix\[dq], this is a kind of \[dq]undelete\[dq] command. If you combine more \-\-filter, \-\-filter\-disk and \-\-filter\-missing options, only files matching all the set of filters are selected. -This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands. +This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq]. Note that it cannot be used with \[dq]sync\[dq] and \[dq]scrub\[dq], because they always process the whole array. .TP .B \-e, \-\-filter\-error -Filters the files to process in the \[dq]check\[dq] and \[dq]fix\[dq] -commands. -It processes only the files containing blocks marked with silent -errors during the \[dq]sync\[dq] and \[dq]scrub\[dq] command, and listed in the -\[dq]status\[dq] command. -This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands. +Filters the blocks to process in \[dq]check\[dq] and \[dq]fix\[dq]. +It processes only the blocks marked with silent or input/output +errors during \[dq]sync\[dq] and \[dq]scrub\[dq], and listed in \[dq]status\[dq]. +This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq]. .TP .B \-p, \-\-percentage PERC -Selects the part of the array to process in the \[dq]scrub\[dq] command. +Selects the part of the array to process in \[dq]scrub\[dq]. PERC is a numeric value from 0 to 100, default is 12. When specifying 0, only the blocks marked as bad are scrubbed. -This option can be used only with the \[dq]scrub\[dq] command. +This option can be used only with \[dq]scrub\[dq]. .TP .B \-o, \-\-older\-than DAYS -Selects the older the part of the array to process in the -\[dq]scrub\[dq] command. +Selects the older the part of the array to process in \[dq]scrub\[dq]. DAYS is the minimum age in days for a block to be scrubbed, default is 10. Blocks marked as bad are always scrubbed despite this option. -This option can be used only with the \[dq]scrub\[dq] command. +This option can be used only with \[dq]scrub\[dq]. .TP .B \-a, \-\-audit\-only -In the \[dq]check\[dq] command verifies the hash of the files without +In \[dq]check\[dq] verifies the hash of the files without doing any kind of check on the parity data. If you are interested in checking only the file data this option can speedup a lot the checking process. -This option can be used only with the \[dq]check\[dq] command. +This option can be used only with \[dq]check\[dq]. .TP .B \-h, \-\-pre\-hash -In the \[dq]sync\[dq] command run a preliminary hashing phase of all +In \[dq]sync\[dq] runs a preliminary hashing phase of all the new data to verify the data used in the parity computation. Usually in \[dq]sync\[dq] no preliminary hashing is done, and the new data is hashed just before the parity computation when it\'s read @@ -781,16 +877,16 @@ latent hardware problem, it\'s possible to encounter silent errors what cannot be detected because the data is not yet hashed. To avoid this risk, you can enable the \[dq]pre\-hash\[dq] mode and have all the data hashed two times to ensure its integrity. -This option can be used only with the \[dq]sync\[dq] command. +This option can be used only with \[dq]sync\[dq]. .TP .B \-i, \-\-import DIR Imports from the specified directory any file that you deleted -from the array after the last \[dq]sync\[dq] command. -If you still have such files, they could be used by the \[dq]check\[dq] -and \[dq]fix\[dq] commands to improve the recover process. +from the array after the last \[dq]sync\[dq]. +If you still have such files, they could be used by \[dq]check\[dq] +and \[dq]fix\[dq] to improve the recover process. The files are read also in subdirectories and they are identified regardless of their name. -This option can be used only with the \[dq]check\[dq] and \[dq]fix\[dq] commands. +This option can be used only with \[dq]check\[dq] and \[dq]fix\[dq]. .TP .B \-Z, \-\-force\-zero Forces the insecure operation of syncing a file with zero @@ -801,7 +897,7 @@ This allows to easily detect when after a system crash, some accessed files were truncated. This is a possible condition in Linux with the ext3/ext4 filesystems. -This option can be used only with the \[dq]sync\[dq] command. +This option can be used only with \[dq]sync\[dq]. .TP .B \-E, \-\-force\-empty Forces the insecure operation of syncing a disk with all @@ -811,7 +907,7 @@ in the disk are missing or rewritten, it stops proceeding unless you specify this option. This allows to easily detect when a data file\-system is not mounted. -This option can be used only with the \[dq]sync\[dq] command. +This option can be used only with \[dq]sync\[dq]. .TP .B \-U, \-\-force\-uuid Forces the insecure operation of syncing, checking and fixing @@ -823,8 +919,8 @@ wrong mount points. It\'s anyway allowed to have a single UUID change with single parity, and more with multiple parity, because it\'s the normal case of replacing disks after a recovery. -This option can be used only with the \[dq]sync\[dq], \[dq]check\[dq] or -\[dq]fix\[dq] command. +This option can be used only with \[dq]sync\[dq], \[dq]check\[dq] or +\[dq]fix\[dq]. .TP .B \-D, \-\-force\-device Forces the insecure operation of fixing with disks on the same @@ -834,11 +930,10 @@ it stops proceeding, because it\'s not a supported configuration. But it could happen that you want to temporarily restore a lost disk in the free space left in an already used disk. and this option allows you to continue anyway. -This option can be used only with the \[dq]fix\[dq] command. +This option can be used only with \[dq]fix\[dq]. .TP .B \-N, \-\-force\-nocopy -In the \[dq]sync\[dq], \[dq]check and \[dq]fix\[dq] commands, disables the copy -detection heuristic. +In \[dq]sync\[dq], \[dq]check and \[dq]fix\[dq], disables the copy detection heuristic. Without this option SnapRAID assumes that files with same attributes, like name, size and timestamp are copies with the same data. @@ -848,18 +943,17 @@ to detect silent errors or to recover missing files. This behavior, in some rare cases, may result in false positives, or in a slow process due the many hash verifications, and this option allows to resolve them. -This option can be used only with the \[dq]sync\[dq], \[dq]check\[dq] and \[dq]fix\[dq] -commands. +This option can be used only with \[dq]sync\[dq], \[dq]check\[dq] and \[dq]fix\[dq]. .TP .B \-F, \-\-force\-full -In the \[dq]sync\[dq] command, forces a full rebuild of the parity. +In \[dq]sync\[dq] forces a full rebuild of the parity. This option can be used when you reverted back to an old content -file, and you still have the parity data. +file, but using a more recent parity data. Instead of recomputing the parity from scratch, this allows to reuse the hashes present in the content file to validate data, and to maintain data protection during the \[dq]sync\[dq] process using the old content file and the parity data you have. -This option can be used only with the \[dq]sync\[dq] command. +This option can be used only with \[dq]sync\[dq]. .TP .B \-l, \-\-log FILE Write a detailed log of errors found in check and fix. @@ -1056,8 +1150,8 @@ wasted for each file. For example, with 100000 files and a 256 KiB block size, you are going to waste 12 GiB of parity, that may result in 12 GiB less space available in the data disk. .PP -You can get the amount of wasted space in each disk using the \[dq]status\[dq] -command. This is the amount of space that you must leave free in the data +You can get the amount of wasted space in each disk using \[dq]status\[dq]. +This is the amount of space that you must leave free in the data disks, or use for files not included in the array. .PP To avoid to problem, you can use a bigger partition for parity. @@ -1333,99 +1427,6 @@ snapraid \-f \[dq]*.mp3\[dq] check .PP In Unix, when using globbing chars in the command line, you have to quote them. Otherwise the shell will try to expand them. -.SH RECOVERING -The worst happened, and you lost a disk! -.PP -DO NOT PANIC! You will be able to recover it! -.PP -The first thing you have to do is to avoid further changes at you disk array. -Disable any remote connection to it, any scheduled process, including any -scheduled SnapRAID nightly sync. -.PP -Then proceed with the following steps. -.SS STEP 1 \-> Reconfigure -You need some space to recover, even better if you already have an additional -disk, but in case, also an external USB or remote disk is enough. -.PP -Change the SnapRAID configuration file and make the \[dq]disk\[dq] option -of the failed disk to point to the place where you have enough empty -space to recover the files. -.PP -For example, if you have that disk \[dq]d1\[dq] failed, you can change: -.PP -.RS 4 -disk d1 /mnt/disk1/ -.PD 0 -.PP -.PD -.RE -.PP -to: -.PP -.RS 4 -disk d1 /mnt/new_spare_disk/ -.PD 0 -.PP -.PD -.RE -.SS STEP 2 \-> Fix -Run the fix command, storing the log in an external file with: -.PP -.RS 4 -snapraid \-d NAME \-l fix.log fix -.PD 0 -.PP -.PD -.RE -.PP -Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example. -.PP -This command will take a long time. -.PP -Take care that you need also few gigabytes free to store the fix.log file. -Run it from a disk with some free space. -.PP -Now you have recovered all the recoverable. If some file is partially or totally -unrecoverable, it will be renamed adding the \[dq].unrecoverable\[dq] extension. -.PP -You can get a detailed list of all the unrecoverable blocks in the fix.log file -checking all the lines starting with \[dq]unrecoverable:\[dq] -.PP -If you are not satisfied of the recovering, you can retry it as many time you wish. -For example, if you have remove some files after the last \[dq]sync\[dq], you can retry -to put them inplace, and retry the \[dq]fix\[dq]. -.PP -If you are satisfied of the recovering, you can now proceed further, -but take care that after syncing you will be no more able to retry the -\[dq]fix\[dq] command! -.SS STEP 3 \-> Check -As paranoid check, you can now run a \[dq]check\[dq] command to ensure that everything -is OK on the recovered disk. -.PP -.RS 4 -snapraid \-d NAME \-a check -.PD 0 -.PP -.PD -.RE -.PP -Where NAME is the name of the disk, like \[dq]d1\[dq] as in our previous example. -.PP -The options \-d and \-a tell SnapRAID to check only the specified disk, -and ignore all the parity data. -.PP -This command will take a long time. -.SS STEP 4 \-> Sync -Run the \[dq]sync\[dq] command to resynchronize the array with the new disk. -.PP -.RS 4 -snapraid sync -.PD 0 -.PP -.PD -.RE -.PP -If everything is recovered, this command is immediate. .SH CONTENT SnapRAID stores the list and checksums of your files in the content file. .PP diff --git a/snapraid.d b/snapraid.d index 59995b2c..36224aab 100644 --- a/snapraid.d +++ b/snapraid.d @@ -149,52 +149,31 @@ Getting Started Now you can start using your array as you like, and periodically update the parity information running the "sync" command. - Restoring and Undeleting - In recovering, SnapRAID is more like a backup program than a RAID system, - and it can be used to restore or undelete a file or directory to its previous - state using the -f, --filter option : - - :snapraid fix -f FILE - - or for a directory: - - :snapraid fix -f DIR/ - - You can also use it to recover only accidentally deleted files inside - a directory using the -m, --filter-missing option, that restores - only missing files, leaving untouched all the others. - - :snapraid fix -m -f DIR/ - - Or to recover all the deleted files in all the drives with: - - :snapraid fix -m - - If instead you have lost an entire disk and you want to recover it, - please see the 'Recovering' chapter. - Scrubbing To periodically check the data and parity for errors, you can run the "scrub" command. :snapraid scrub - This command verifies only the oldest data in your array. - Every run of the command checks about 12% of the array, - but not newer than 10 days. + This command verifies the data in your array comparing it with + the hash computed in the "sync" command. + + Every run of the command checks about 12% of the array, but not data newer + than 10 days. You can use the -p, --percentage option to specify a different amount, and the -o, --older-than option to specify a different age in days. For example, to check 5% of the array older than 20 days use: :snapraid -p 5 -o 20 scrub - If a silent or input error is found, the corresponding block is marked - as bad in the "content" file, and listed in the "status" command. + If during the process, silent or input/output errors are found, + the corresponding blocks are marked as bad in the "content" file, + and listed in the "status" command. :snapraid status - To fix them, you can use the "fix" command filtering for files containing - bad blocks: + To fix them, you can use the "fix" command filtering for bad blocks with + the -e, --filter-error options: :snapraid -e fix @@ -204,7 +183,7 @@ Getting Started :snapraid -p 0 scrub - Take care that running scrub on a not synced array may result in + Take care that running "scrub" on a not synced array may result in errors caused by removed or modified files. These errors are reported in the "scrub" result, but related blocks are not marked as bad. @@ -274,6 +253,106 @@ Getting Started :fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1 + Undeleting + SnapRAID is more like a backup program than a RAID system, and it + can be used to restore or undelete files to their previous state using + the -f, --filter option : + + :snapraid fix -f FILE + + or for a directory: + + :snapraid fix -f DIR/ + + You can also use it to recover only accidentally deleted files inside + a directory using the -m, --filter-missing option, that restores + only missing files, leaving untouched all the others. + + :snapraid fix -m -f DIR/ + + Or to recover all the deleted files in all the drives with: + + :snapraid fix -m + + Recovering + The worst happened, and you lost a disk! + + DO NOT PANIC! You will be able to recover it! + + The first thing you have to do is to avoid further changes at you disk array. + Disable any remote connection to it, any scheduled process, including any + scheduled SnapRAID nightly sync or scrub. + + Then proceed with the following steps. + + STEP 1 -> Reconfigure + You need some space to recover, even better if you already have an additional + disk, but in case, also an external USB or remote disk is enough. + + Change the SnapRAID configuration file to make the "disk" option + of the failed disk to point to the place where you have enough empty + space to recover the files. + + For example, if you have that disk "d1" failed, you can change from: + + :disk d1 /mnt/disk1/ + + to: + + :disk d1 /mnt/new_spare_disk/ + + STEP 2 -> Fix + Run the fix command, storing the log in an external file with: + + :snapraid -d NAME -l fix.log fix + + Where NAME is the name of the disk, like "d1" as in our previous example. + + This command will take a long time. + + Take care that you need also few gigabytes free to store the fix.log file. + Run it from a disk with some free space. + + Now you have recovered all the recoverable. If some file is partially or totally + unrecoverable, it will be renamed adding the ".unrecoverable" extension. + + You can get a detailed list of all the unrecoverable blocks in the fix.log file + checking all the lines starting with "unrecoverable:" + + If you are not satisfied of the recovering, you can retry it as many + time you wish. + + For example, if you have removed files from the array after the last + "sync", this may result in some other files not recovered. + In this case, you can retry the "fix" using the -i, --import option, + specifing where these files are now, to include them again in the + recovering process. + + If you are satisfied of the recovering, you can now proceed further, + but take care that after syncing you cannot retry the "fix" command + anymore! + + STEP 3 -> Check + As paranoid check, you can now run a "check" command to ensure that + everything is OK on the recovered disk. + + :snapraid -d NAME -a check + + Where NAME is the name of the disk, like "d1" as in our previous example. + + The options -d and -a tell SnapRAID to check only the specified disk, + and ignore all the parity data. + + This command will take a long time, but if you are not paranoid, + you can skip it. + + STEP 4 -> Sync + Run the "sync" command to resynchronize the array with the new disk. + + :snapraid sync + + If everything is recovered, this command is immediate. + Commands SnapRAID provides a few simple commands that allow to: @@ -298,26 +377,32 @@ Commands sync Updates the parity information. All the modified files - in the disk array are read, and the parity data is - recomputed. - - Files are identified by path and/or inode and checked by - size and timestamp. - If the size or timestamp are different, the parity data is - recomputed for the whole file. - With inode you can move them on the disk without triggering - any parity recomputation. + in the disk array are read, and the corresponding parity + data is updated. You can stop this process at any time pressing Ctrl+C, without losing the work already done. At the next run the "sync" process will start where interrupted. + If during the process, silent or input/output errors are found, + the corresponding blocks are marked as bad. + + Files are identified by path and/or inode and checked by + size and timestamp. + If the file size or timestamp are different, the parity data + is recomputed for the whole file. + If the file is moved or renamed in the same disk, keeping the + same inode, the parity is no recomputed. + If the file is moved to another disk, the parity is recomputed, + but the previously compute hash information is kept. + The "content" and "parity" files are modified if necessary. The files in the array are NOT modified. scrub - Scrubs the array, checking for silent errors in data or parity disks. + Scrubs the array, checking for silent or input/output errors in data + and parity disks. For each command invocation, the 12% of the array is checked, but nothing that it's more recent than 10 days. @@ -326,15 +411,15 @@ Commands You can use the -p, --percentage option to specify a different amount, and the -o, --older-than option to specify a different age in days. - You can have a full scrub with "-p100 -o0". - - Any silent error identified is recorded in the content file, - and it's listed in the "status" command until it's fixed calling - "fix" and then "scrub". + You can have a full scrub with "-p 100 -o 0". The oldest blocks are scrubbed first ensuring an optimal check. - Blocks already marked as bad are always checked, and if found - correct, they are automatically unmarked. + + For any silent or input/output error found the corresponding blocks + are marked as bad in the "content" file. + These bad blocks are listed in "status", and can be fixed with "fix -e". + After the fix, at the next scrub they will be rechecked, and if found + corrected, the bad mark will be removed. It's recommended to run "scrub" on a synced array, to avoid to have reported error caused by unsynced data. These errors are recognized @@ -352,18 +437,19 @@ Commands Fix all the files and the parity data. All the files and the parity data are compared with the snapshot - state saved in the last "sync" command. - If a difference is found, it's reverted at the stored snapshot. + state saved in the last "sync". + If a difference is found, it's reverted to the stored snapshot. - Note that "fix" doesn't differentiate from errors or intentional - changes at files. It just reverts the state at the last "sync" command. + Note that "fix" doesn't differentiate between errors and intentional + modifications. It inconditionally reverts the file state at the last "sync". - By default the full array is processed. Use the filter options - to select a subset of files or disks to operate on. - To know that file is checked, you can use the -v, --verbose option. + If no other option is specified the full array is processed. + Use the filter options to select a subset of files or disks to operate on. - To only fix errors found with "scrub" and reported in "status", - you can use the -e, --filter-error option. + To only fix the blocks marked bad during "sync" and "scrub", + use the -e, --filter-error option. + As difference from other filter options, with this one fixes are + applied only to files that are not modified from the the latest "sync". All the files that cannot be fixed are renamed adding the ".unrecoverable" extension. @@ -377,34 +463,30 @@ Commands check Verify all the files and the parity data. - It works just like "fix", but it only simulates a recovery - and no change is written in the array. + It works like "fix", but it only simulates a recovery and no change + is written in the array. This command is mostly intended for manual verifications, like after a recovery process or in other special conditions. - For periodic and scheduled checks uses the "scrub" command. + For periodic and scheduled checks uses "scrub". If you use the -a, --audit-only option, only the file data is checked, and the parity data is ignored for a faster run. - By default the full array is processed. Use the filter options - to select a subset of files or disks to operate on. - To know that file is checked, you can use the -v, --verbose option. - Files are identified only by path, and not by inode. Nothing is modified. list Lists all the files contained in the array at the time of the - last "sync" command. + last "sync". Nothing is modified. diff - Lists all the files modified from the last "sync" command that - need to have their parity data recomputed. + Lists all the files modified from the last "sync" that need to have + their parity data recomputed. This command doesn't check the file data, but only the file timestamp size and inode. @@ -442,13 +524,13 @@ Commands do nothing and tells you that nothing has to be done. The rehash isn't done immediately, but it takes place - progressively during the "sync" and "scrub" commands. + progressively during "sync" and "scrub". - You can get the rehash state using the "status" command. + You can get the rehash state using "status". During the rehash, SnapRAID maintains full functionality, - with the only exception of the "dup" command not able to detect - duplicated files using a different hash. + with the only exception of "dup" not able to detect duplicated + files using a different hash. Options SnapRAID provides the following options: @@ -459,20 +541,18 @@ Options the current directory in Windows. -f, --filter PATTERN - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files matching the entered pattern are processed. This option can be used many times. See the PATTERN section for more details in the pattern specifications. In Unix, ensure to quote globbing chars if used. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -d, --filter-disk NAME - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files present in the specified disk are processed. You must specify a disk name as named in the configuration file. @@ -481,52 +561,48 @@ Options If you combine more --filter, --filter-disk and --filter-missing options, only files matching all the set of filters are selected. This option can be used many times. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -m, --filter-missing - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files missing/deleted from the array are processed. When used with "fix", this is a kind of "undelete" command. If you combine more --filter, --filter-disk and --filter-missing options, only files matching all the set of filters are selected. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -e, --filter-error - Filters the files to process in the "check" and "fix" - commands. - It processes only the files containing blocks marked with silent - errors during the "sync" and "scrub" command, and listed in the - "status" command. - This option can be used only with the "check" and "fix" commands. + Filters the blocks to process in "check" and "fix". + It processes only the blocks marked with silent or input/output + errors during "sync" and "scrub", and listed in "status". + This option can be used only with "check" and "fix". -p, --percentage PERC - Selects the part of the array to process in the "scrub" command. + Selects the part of the array to process in "scrub". PERC is a numeric value from 0 to 100, default is 12. When specifying 0, only the blocks marked as bad are scrubbed. - This option can be used only with the "scrub" command. + This option can be used only with "scrub". -o, --older-than DAYS - Selects the older the part of the array to process in the - "scrub" command. + Selects the older the part of the array to process in "scrub". DAYS is the minimum age in days for a block to be scrubbed, default is 10. Blocks marked as bad are always scrubbed despite this option. - This option can be used only with the "scrub" command. + This option can be used only with "scrub". -a, --audit-only - In the "check" command verifies the hash of the files without + In "check" verifies the hash of the files without doing any kind of check on the parity data. If you are interested in checking only the file data this option can speedup a lot the checking process. - This option can be used only with the "check" command. + This option can be used only with "check". -h, --pre-hash - In the "sync" command run a preliminary hashing phase of all + In "sync" runs a preliminary hashing phase of all the new data to verify the data used in the parity computation. Usually in "sync" no preliminary hashing is done, and the new data is hashed just before the parity computation when it's read @@ -538,16 +614,16 @@ Options what cannot be detected because the data is not yet hashed. To avoid this risk, you can enable the "pre-hash" mode and have all the data hashed two times to ensure its integrity. - This option can be used only with the "sync" command. + This option can be used only with "sync". -i, --import DIR Imports from the specified directory any file that you deleted - from the array after the last "sync" command. - If you still have such files, they could be used by the "check" - and "fix" commands to improve the recover process. + from the array after the last "sync". + If you still have such files, they could be used by "check" + and "fix" to improve the recover process. The files are read also in subdirectories and they are identified regardless of their name. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". -Z, --force-zero Forces the insecure operation of syncing a file with zero @@ -558,7 +634,7 @@ Options some accessed files were truncated. This is a possible condition in Linux with the ext3/ext4 filesystems. - This option can be used only with the "sync" command. + This option can be used only with "sync". -E, --force-empty Forces the insecure operation of syncing a disk with all @@ -568,7 +644,7 @@ Options unless you specify this option. This allows to easily detect when a data file-system is not mounted. - This option can be used only with the "sync" command. + This option can be used only with "sync". -U, --force-uuid Forces the insecure operation of syncing, checking and fixing @@ -580,8 +656,8 @@ Options It's anyway allowed to have a single UUID change with single parity, and more with multiple parity, because it's the normal case of replacing disks after a recovery. - This option can be used only with the "sync", "check" or - "fix" command. + This option can be used only with "sync", "check" or + "fix". -D, --force-device Forces the insecure operation of fixing with disks on the same @@ -591,11 +667,10 @@ Options But it could happen that you want to temporarily restore a lost disk in the free space left in an already used disk. and this option allows you to continue anyway. - This option can be used only with the "fix" command. + This option can be used only with "fix". -N, --force-nocopy - In the "sync", "check and "fix" commands, disables the copy - detection heuristic. + In "sync", "check and "fix", disables the copy detection heuristic. Without this option SnapRAID assumes that files with same attributes, like name, size and timestamp are copies with the same data. @@ -605,18 +680,17 @@ Options This behavior, in some rare cases, may result in false positives, or in a slow process due the many hash verifications, and this option allows to resolve them. - This option can be used only with the "sync", "check" and "fix" - commands. + This option can be used only with "sync", "check" and "fix". -F, --force-full - In the "sync" command, forces a full rebuild of the parity. + In "sync" forces a full rebuild of the parity. This option can be used when you reverted back to an old content - file, and you still have the parity data. + file, but using a more recent parity data. Instead of recomputing the parity from scratch, this allows to reuse the hashes present in the content file to validate data, and to maintain data protection during the "sync" process using the old content file and the parity data you have. - This option can be used only with the "sync" command. + This option can be used only with "sync". -l, --log FILE Write a detailed log of errors found in check and fix. @@ -813,8 +887,8 @@ Configuration block size, you are going to waste 12 GiB of parity, that may result in 12 GiB less space available in the data disk. - You can get the amount of wasted space in each disk using the "status" - command. This is the amount of space that you must leave free in the data + You can get the amount of wasted space in each disk using "status". + This is the amount of space that you must leave free in the data disks, or use for files not included in the array. To avoid to problem, you can use a bigger partition for parity. @@ -965,79 +1039,6 @@ Pattern In Unix, when using globbing chars in the command line, you have to quote them. Otherwise the shell will try to expand them. -Recovering - The worst happened, and you lost a disk! - - DO NOT PANIC! You will be able to recover it! - - The first thing you have to do is to avoid further changes at you disk array. - Disable any remote connection to it, any scheduled process, including any - scheduled SnapRAID nightly sync. - - Then proceed with the following steps. - - STEP 1 -> Reconfigure - You need some space to recover, even better if you already have an additional - disk, but in case, also an external USB or remote disk is enough. - - Change the SnapRAID configuration file and make the "disk" option - of the failed disk to point to the place where you have enough empty - space to recover the files. - - For example, if you have that disk "d1" failed, you can change: - - :disk d1 /mnt/disk1/ - - to: - - :disk d1 /mnt/new_spare_disk/ - - STEP 2 -> Fix - Run the fix command, storing the log in an external file with: - - :snapraid -d NAME -l fix.log fix - - Where NAME is the name of the disk, like "d1" as in our previous example. - - This command will take a long time. - - Take care that you need also few gigabytes free to store the fix.log file. - Run it from a disk with some free space. - - Now you have recovered all the recoverable. If some file is partially or totally - unrecoverable, it will be renamed adding the ".unrecoverable" extension. - - You can get a detailed list of all the unrecoverable blocks in the fix.log file - checking all the lines starting with "unrecoverable:" - - If you are not satisfied of the recovering, you can retry it as many time you wish. - For example, if you have remove some files after the last "sync", you can retry - to put them inplace, and retry the "fix". - - If you are satisfied of the recovering, you can now proceed further, - but take care that after syncing you will be no more able to retry the - "fix" command! - - STEP 3 -> Check - As paranoid check, you can now run a "check" command to ensure that everything - is OK on the recovered disk. - - :snapraid -d NAME -a check - - Where NAME is the name of the disk, like "d1" as in our previous example. - - The options -d and -a tell SnapRAID to check only the specified disk, - and ignore all the parity data. - - This command will take a long time. - - STEP 4 -> Sync - Run the "sync" command to resynchronize the array with the new disk. - - :snapraid sync - - If everything is recovered, this command is immediate. - Content SnapRAID stores the list and checksums of your files in the content file. diff --git a/snapraid.txt b/snapraid.txt index f4153b92..dead1891 100644 --- a/snapraid.txt +++ b/snapraid.txt @@ -162,33 +162,7 @@ When this command completes, your data is SAFE. Now you can start using your array as you like, and periodically update the parity information running the "sync" command. -4.1 Restoring and Undeleting ----------------------------- - -In recovering, SnapRAID is more like a backup program than a RAID system, -and it can be used to restore or undelete a file or directory to its previous -state using the -f, --filter option : - - snapraid fix -f FILE - -or for a directory: - - snapraid fix -f DIR/ - -You can also use it to recover only accidentally deleted files inside -a directory using the -m, --filter-missing option, that restores -only missing files, leaving untouched all the others. - - snapraid fix -m -f DIR/ - -Or to recover all the deleted files in all the drives with: - - snapraid fix -m - -If instead you have lost an entire disk and you want to recover it, -please see the 'Recovering' chapter. - -4.2 Scrubbing +4.1 Scrubbing ------------- To periodically check the data and parity for errors, you can @@ -196,22 +170,25 @@ run the "scrub" command. snapraid scrub -This command verifies only the oldest data in your array. -Every run of the command checks about 12% of the array, -but not newer than 10 days. +This command verifies the data in your array comparing it with +the hash computed in the "sync" command. + +Every run of the command checks about 12% of the array, but not data newer +than 10 days. You can use the -p, --percentage option to specify a different amount, and the -o, --older-than option to specify a different age in days. For example, to check 5% of the array older than 20 days use: snapraid -p 5 -o 20 scrub -If a silent or input error is found, the corresponding block is marked -as bad in the "content" file, and listed in the "status" command. +If during the process, silent or input/output errors are found, +the corresponding blocks are marked as bad in the "content" file, +and listed in the "status" command. snapraid status -To fix them, you can use the "fix" command filtering for files containing -bad blocks: +To fix them, you can use the "fix" command filtering for bad blocks with +the -e, --filter-error options: snapraid -e fix @@ -221,11 +198,11 @@ marked as bad. snapraid -p 0 scrub -Take care that running scrub on a not synced array may result in +Take care that running "scrub" on a not synced array may result in errors caused by removed or modified files. These errors are reported in the "scrub" result, but related blocks are not marked as bad. -4.3 Pooling +4.2 Pooling ----------- To have all the files in your array shown in the same directory tree, @@ -293,6 +270,110 @@ remote symlinks with the command: fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1 +4.3 Undeleting +-------------- + +SnapRAID is more like a backup program than a RAID system, and it +can be used to restore or undelete files to their previous state using +the -f, --filter option : + + snapraid fix -f FILE + +or for a directory: + + snapraid fix -f DIR/ + +You can also use it to recover only accidentally deleted files inside +a directory using the -m, --filter-missing option, that restores +only missing files, leaving untouched all the others. + + snapraid fix -m -f DIR/ + +Or to recover all the deleted files in all the drives with: + + snapraid fix -m + +4.4 Recovering +-------------- + +The worst happened, and you lost a disk! + +DO NOT PANIC! You will be able to recover it! + +The first thing you have to do is to avoid further changes at you disk array. +Disable any remote connection to it, any scheduled process, including any +scheduled SnapRAID nightly sync or scrub. + +Then proceed with the following steps. + +---- 4.4.1 STEP 1 -> Reconfigure ---- +You need some space to recover, even better if you already have an additional +disk, but in case, also an external USB or remote disk is enough. + +Change the SnapRAID configuration file to make the "disk" option +of the failed disk to point to the place where you have enough empty +space to recover the files. + +For example, if you have that disk "d1" failed, you can change from: + + disk d1 /mnt/disk1/ + +to: + + disk d1 /mnt/new_spare_disk/ + +---- 4.4.2 STEP 2 -> Fix ---- +Run the fix command, storing the log in an external file with: + + snapraid -d NAME -l fix.log fix + +Where NAME is the name of the disk, like "d1" as in our previous example. + +This command will take a long time. + +Take care that you need also few gigabytes free to store the fix.log file. +Run it from a disk with some free space. + +Now you have recovered all the recoverable. If some file is partially or totally +unrecoverable, it will be renamed adding the ".unrecoverable" extension. + +You can get a detailed list of all the unrecoverable blocks in the fix.log file +checking all the lines starting with "unrecoverable:" + +If you are not satisfied of the recovering, you can retry it as many +time you wish. + +For example, if you have removed files from the array after the last +"sync", this may result in some other files not recovered. +In this case, you can retry the "fix" using the -i, --import option, +specifing where these files are now, to include them again in the +recovering process. + +If you are satisfied of the recovering, you can now proceed further, +but take care that after syncing you cannot retry the "fix" command +anymore! + +---- 4.4.3 STEP 3 -> Check ---- +As paranoid check, you can now run a "check" command to ensure that +everything is OK on the recovered disk. + + snapraid -d NAME -a check + +Where NAME is the name of the disk, like "d1" as in our previous example. + +The options -d and -a tell SnapRAID to check only the specified disk, +and ignore all the parity data. + +This command will take a long time, but if you are not paranoid, +you can skip it. + +---- 4.4.4 STEP 4 -> Sync ---- +Run the "sync" command to resynchronize the array with the new disk. + + snapraid sync + +If everything is recovered, this command is immediate. + 5 COMMANDS ========== @@ -324,28 +405,34 @@ Nothing is modified. -------- Updates the parity information. All the modified files -in the disk array are read, and the parity data is -recomputed. - -Files are identified by path and/or inode and checked by -size and timestamp. -If the size or timestamp are different, the parity data is -recomputed for the whole file. -With inode you can move them on the disk without triggering -any parity recomputation. +in the disk array are read, and the corresponding parity +data is updated. You can stop this process at any time pressing Ctrl+C, without losing the work already done. At the next run the "sync" process will start where interrupted. +If during the process, silent or input/output errors are found, +the corresponding blocks are marked as bad. + +Files are identified by path and/or inode and checked by +size and timestamp. +If the file size or timestamp are different, the parity data +is recomputed for the whole file. +If the file is moved or renamed in the same disk, keeping the +same inode, the parity is no recomputed. +If the file is moved to another disk, the parity is recomputed, +but the previously compute hash information is kept. + The "content" and "parity" files are modified if necessary. The files in the array are NOT modified. 5.3 scrub --------- -Scrubs the array, checking for silent errors in data or parity disks. +Scrubs the array, checking for silent or input/output errors in data +and parity disks. For each command invocation, the 12% of the array is checked, but nothing that it's more recent than 10 days. @@ -354,15 +441,15 @@ at least one time every two months. You can use the -p, --percentage option to specify a different amount, and the -o, --older-than option to specify a different age in days. -You can have a full scrub with "-p100 -o0". - -Any silent error identified is recorded in the content file, -and it's listed in the "status" command until it's fixed calling -"fix" and then "scrub". +You can have a full scrub with "-p 100 -o 0". The oldest blocks are scrubbed first ensuring an optimal check. -Blocks already marked as bad are always checked, and if found -correct, they are automatically unmarked. + +For any silent or input/output error found the corresponding blocks +are marked as bad in the "content" file. +These bad blocks are listed in "status", and can be fixed with "fix -e". +After the fix, at the next scrub they will be rechecked, and if found +corrected, the bad mark will be removed. It's recommended to run "scrub" on a synced array, to avoid to have reported error caused by unsynced data. These errors are recognized @@ -382,18 +469,19 @@ The files in the array are NOT modified. Fix all the files and the parity data. All the files and the parity data are compared with the snapshot -state saved in the last "sync" command. -If a difference is found, it's reverted at the stored snapshot. +state saved in the last "sync". +If a difference is found, it's reverted to the stored snapshot. -Note that "fix" doesn't differentiate from errors or intentional -changes at files. It just reverts the state at the last "sync" command. +Note that "fix" doesn't differentiate between errors and intentional +modifications. It inconditionally reverts the file state at the last "sync". -By default the full array is processed. Use the filter options -to select a subset of files or disks to operate on. -To know that file is checked, you can use the -v, --verbose option. +If no other option is specified the full array is processed. +Use the filter options to select a subset of files or disks to operate on. -To only fix errors found with "scrub" and reported in "status", -you can use the -e, --filter-error option. +To only fix the blocks marked bad during "sync" and "scrub", +use the -e, --filter-error option. +As difference from other filter options, with this one fixes are +applied only to files that are not modified from the the latest "sync". All the files that cannot be fixed are renamed adding the ".unrecoverable" extension. @@ -409,21 +497,17 @@ The files in the array are modified if necessary. Verify all the files and the parity data. -It works just like "fix", but it only simulates a recovery -and no change is written in the array. +It works like "fix", but it only simulates a recovery and no change +is written in the array. This command is mostly intended for manual verifications, like after a recovery process or in other special conditions. -For periodic and scheduled checks uses the "scrub" command. +For periodic and scheduled checks uses "scrub". If you use the -a, --audit-only option, only the file data is checked, and the parity data is ignored for a faster run. -By default the full array is processed. Use the filter options -to select a subset of files or disks to operate on. -To know that file is checked, you can use the -v, --verbose option. - Files are identified only by path, and not by inode. Nothing is modified. @@ -432,15 +516,15 @@ Nothing is modified. -------- Lists all the files contained in the array at the time of the -last "sync" command. +last "sync". Nothing is modified. 5.7 diff -------- -Lists all the files modified from the last "sync" command that -need to have their parity data recomputed. +Lists all the files modified from the last "sync" that need to have +their parity data recomputed. This command doesn't check the file data, but only the file timestamp size and inode. @@ -484,13 +568,13 @@ If you are already using the optimal hash, this command do nothing and tells you that nothing has to be done. The rehash isn't done immediately, but it takes place -progressively during the "sync" and "scrub" commands. +progressively during "sync" and "scrub". -You can get the rehash state using the "status" command. +You can get the rehash state using "status". During the rehash, SnapRAID maintains full functionality, -with the only exception of the "dup" command not able to detect -duplicated files using a different hash. +with the only exception of "dup" not able to detect duplicated +files using a different hash. 6 OPTIONS @@ -504,20 +588,18 @@ SnapRAID provides the following options: the current directory in Windows. -f, --filter PATTERN - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files matching the entered pattern are processed. This option can be used many times. See the PATTERN section for more details in the pattern specifications. In Unix, ensure to quote globbing chars if used. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -d, --filter-disk NAME - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files present in the specified disk are processed. You must specify a disk name as named in the configuration file. @@ -526,52 +608,48 @@ SnapRAID provides the following options: If you combine more --filter, --filter-disk and --filter-missing options, only files matching all the set of filters are selected. This option can be used many times. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -m, --filter-missing - Filters the files to process in the "check" and "fix" - commands. + Filters the files to process in "check" and "fix". Only the files missing/deleted from the array are processed. When used with "fix", this is a kind of "undelete" command. If you combine more --filter, --filter-disk and --filter-missing options, only files matching all the set of filters are selected. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". Note that it cannot be used with "sync" and "scrub", because they always process the whole array. -e, --filter-error - Filters the files to process in the "check" and "fix" - commands. - It processes only the files containing blocks marked with silent - errors during the "sync" and "scrub" command, and listed in the - "status" command. - This option can be used only with the "check" and "fix" commands. + Filters the blocks to process in "check" and "fix". + It processes only the blocks marked with silent or input/output + errors during "sync" and "scrub", and listed in "status". + This option can be used only with "check" and "fix". -p, --percentage PERC - Selects the part of the array to process in the "scrub" command. + Selects the part of the array to process in "scrub". PERC is a numeric value from 0 to 100, default is 12. When specifying 0, only the blocks marked as bad are scrubbed. - This option can be used only with the "scrub" command. + This option can be used only with "scrub". -o, --older-than DAYS - Selects the older the part of the array to process in the - "scrub" command. + Selects the older the part of the array to process in "scrub". DAYS is the minimum age in days for a block to be scrubbed, default is 10. Blocks marked as bad are always scrubbed despite this option. - This option can be used only with the "scrub" command. + This option can be used only with "scrub". -a, --audit-only - In the "check" command verifies the hash of the files without + In "check" verifies the hash of the files without doing any kind of check on the parity data. If you are interested in checking only the file data this option can speedup a lot the checking process. - This option can be used only with the "check" command. + This option can be used only with "check". -h, --pre-hash - In the "sync" command run a preliminary hashing phase of all + In "sync" runs a preliminary hashing phase of all the new data to verify the data used in the parity computation. Usually in "sync" no preliminary hashing is done, and the new data is hashed just before the parity computation when it's read @@ -583,16 +661,16 @@ SnapRAID provides the following options: what cannot be detected because the data is not yet hashed. To avoid this risk, you can enable the "pre-hash" mode and have all the data hashed two times to ensure its integrity. - This option can be used only with the "sync" command. + This option can be used only with "sync". -i, --import DIR Imports from the specified directory any file that you deleted - from the array after the last "sync" command. - If you still have such files, they could be used by the "check" - and "fix" commands to improve the recover process. + from the array after the last "sync". + If you still have such files, they could be used by "check" + and "fix" to improve the recover process. The files are read also in subdirectories and they are identified regardless of their name. - This option can be used only with the "check" and "fix" commands. + This option can be used only with "check" and "fix". -Z, --force-zero Forces the insecure operation of syncing a file with zero @@ -603,7 +681,7 @@ SnapRAID provides the following options: some accessed files were truncated. This is a possible condition in Linux with the ext3/ext4 filesystems. - This option can be used only with the "sync" command. + This option can be used only with "sync". -E, --force-empty Forces the insecure operation of syncing a disk with all @@ -613,7 +691,7 @@ SnapRAID provides the following options: unless you specify this option. This allows to easily detect when a data file-system is not mounted. - This option can be used only with the "sync" command. + This option can be used only with "sync". -U, --force-uuid Forces the insecure operation of syncing, checking and fixing @@ -625,8 +703,8 @@ SnapRAID provides the following options: It's anyway allowed to have a single UUID change with single parity, and more with multiple parity, because it's the normal case of replacing disks after a recovery. - This option can be used only with the "sync", "check" or - "fix" command. + This option can be used only with "sync", "check" or + "fix". -D, --force-device Forces the insecure operation of fixing with disks on the same @@ -636,11 +714,10 @@ SnapRAID provides the following options: But it could happen that you want to temporarily restore a lost disk in the free space left in an already used disk. and this option allows you to continue anyway. - This option can be used only with the "fix" command. + This option can be used only with "fix". -N, --force-nocopy - In the "sync", "check and "fix" commands, disables the copy - detection heuristic. + In "sync", "check and "fix", disables the copy detection heuristic. Without this option SnapRAID assumes that files with same attributes, like name, size and timestamp are copies with the same data. @@ -650,18 +727,17 @@ SnapRAID provides the following options: This behavior, in some rare cases, may result in false positives, or in a slow process due the many hash verifications, and this option allows to resolve them. - This option can be used only with the "sync", "check" and "fix" - commands. + This option can be used only with "sync", "check" and "fix". -F, --force-full - In the "sync" command, forces a full rebuild of the parity. + In "sync" forces a full rebuild of the parity. This option can be used when you reverted back to an old content - file, and you still have the parity data. + file, but using a more recent parity data. Instead of recomputing the parity from scratch, this allows to reuse the hashes present in the content file to validate data, and to maintain data protection during the "sync" process using the old content file and the parity data you have. - This option can be used only with the "sync" command. + This option can be used only with "sync". -l, --log FILE Write a detailed log of errors found in check and fix. @@ -877,8 +953,8 @@ wasted for each file. For example, with 100000 files and a 256 KiB block size, you are going to waste 12 GiB of parity, that may result in 12 GiB less space available in the data disk. -You can get the amount of wasted space in each disk using the "status" -command. This is the amount of space that you must leave free in the data +You can get the amount of wasted space in each disk using "status". +This is the amount of space that you must leave free in the data disks, or use for files not included in the array. To avoid to problem, you can use a bigger partition for parity. @@ -1041,92 +1117,8 @@ In Unix, when using globbing chars in the command line, you have to quote them. Otherwise the shell will try to expand them. -9 RECOVERING -============ - -The worst happened, and you lost a disk! - -DO NOT PANIC! You will be able to recover it! - -The first thing you have to do is to avoid further changes at you disk array. -Disable any remote connection to it, any scheduled process, including any -scheduled SnapRAID nightly sync. - -Then proceed with the following steps. - -9.1 STEP 1 -> Reconfigure -------------------------- - -You need some space to recover, even better if you already have an additional -disk, but in case, also an external USB or remote disk is enough. - -Change the SnapRAID configuration file and make the "disk" option -of the failed disk to point to the place where you have enough empty -space to recover the files. - -For example, if you have that disk "d1" failed, you can change: - - disk d1 /mnt/disk1/ - -to: - - disk d1 /mnt/new_spare_disk/ - -9.2 STEP 2 -> Fix ------------------ - -Run the fix command, storing the log in an external file with: - - snapraid -d NAME -l fix.log fix - -Where NAME is the name of the disk, like "d1" as in our previous example. - -This command will take a long time. - -Take care that you need also few gigabytes free to store the fix.log file. -Run it from a disk with some free space. - -Now you have recovered all the recoverable. If some file is partially or totally -unrecoverable, it will be renamed adding the ".unrecoverable" extension. - -You can get a detailed list of all the unrecoverable blocks in the fix.log file -checking all the lines starting with "unrecoverable:" - -If you are not satisfied of the recovering, you can retry it as many time you wish. -For example, if you have remove some files after the last "sync", you can retry -to put them inplace, and retry the "fix". - -If you are satisfied of the recovering, you can now proceed further, -but take care that after syncing you will be no more able to retry the -"fix" command! - -9.3 STEP 3 -> Check -------------------- - -As paranoid check, you can now run a "check" command to ensure that everything -is OK on the recovered disk. - - snapraid -d NAME -a check - -Where NAME is the name of the disk, like "d1" as in our previous example. - -The options -d and -a tell SnapRAID to check only the specified disk, -and ignore all the parity data. - -This command will take a long time. - -9.4 STEP 4 -> Sync ------------------- - -Run the "sync" command to resynchronize the array with the new disk. - - snapraid sync - -If everything is recovered, this command is immediate. - - -10 CONTENT -========== +9 CONTENT +========= SnapRAID stores the list and checksums of your files in the content file. @@ -1137,7 +1129,7 @@ This file is read and written by the "sync" and "scrub" commands, and read by "fix", "check" and "status". -11 PARITY +10 PARITY ========= SnapRAID stores the parity information of your array in the parity @@ -1150,7 +1142,7 @@ These files are read and written by the "sync" and "fix" commands, and only read by "scrub" and "check". -12 ENCODING +11 ENCODING =========== SnapRAID in Unix ignores any encoding. It reads and stores the @@ -1167,13 +1159,13 @@ redirect the console output to a file, the resulting file is always in the UTF-8 format. -13 COPYRIGHT +12 COPYRIGHT ============ This file is Copyright (C) 2011 Andrea Mazzoleni -14 SEE ALSO +13 SEE ALSO =========== rsync(1)