From 147b905495918217c600ff79464d0ea0ba34bc62 Mon Sep 17 00:00:00 2001 From: Andrea Mazzoleni Date: Sun, 18 Aug 2013 17:21:39 +0200 Subject: [PATCH] Updates comments and documentation. --- cpu.h | 2 +- rehash.c | 6 +- snapraid.1 | 146 +++++++++++-------------------------- snapraid.d | 162 +++++++++++------------------------------ snapraid.txt | 202 ++++++++++++--------------------------------------- 5 files changed, 132 insertions(+), 386 deletions(-) diff --git a/cpu.h b/cpu.h index 80246150..0fc18761 100644 --- a/cpu.h +++ b/cpu.h @@ -202,7 +202,7 @@ static inline int cpu_has_slowmult(void) /* ARM Feroceon 88FR131 rev 1 (v5l) * Compiler gcc 4.7.2 - * CPU is not and x86/x64 + * CPU is not a x86/x64 * Memory is little-endian 32-bit * memset 763 [MB/s] * Murmur3 237 [MB/s] diff --git a/rehash.c b/rehash.c index 5bab7b19..c557b507 100644 --- a/rehash.c +++ b/rehash.c @@ -81,8 +81,8 @@ void state_rehash(struct snapraid_state* state) state->need_write = 1; printf( - "A rehash is now scheduled. It will take place in the next 'sync' and 'scrub'\n" - "commands in a transparent way. You can check the rehash progress status using\n" - "the 'status' commands.\n"); + "A rehash is now scheduled. It will take place progressively in the next\n" + "'sync' and 'scrub' commands. You can check the rehash progress using the\n" + "'status' command.\n"); } diff --git a/snapraid.1 b/snapraid.1 index f322d91f..6fb6c138 100644 --- a/snapraid.1 +++ b/snapraid.1 @@ -250,6 +250,12 @@ When this command completes, your data is SAFE. .PP At this point you can start using your array as you like, and periodically update the redundancy information running the \(a"sync\(a" command. +.PP +After a \(a"sync\(a" command, it\(cqs recommended to run a \(a"scrub\(a" command +to progressively check the old synched data. +.PP +You can verify the status of your array using the \(a"status\(a" command, +that lists all the errors found while scrubbing. .SS Pooling To have all the files in your array shown in the same directory tree, you can enable \(a"pooling\(a", that consists in creating a virtual view of all @@ -342,7 +348,9 @@ Make a backup/snapshot \(hy> \(a"sync\(a" .IP \(bu See the files changed from the previous sync \(hy> \(a"diff\(a" .IP \(bu -Check for integrity \(hy> \(a"check\(a" +Periodically checks old data \(hy> \(a"scrub\(a" +.IP \(bu +Check for integrity the full array \(hy> \(a"check\(a" .IP \(bu Restore the last backup/snapshot \(hy> \(a"fix\(a". .PD @@ -367,7 +375,7 @@ Prints a summary of the state of the disk array. .PP It includes information about the parity fragmentation, how old are the blocks without checking, and all the recorded silent -errors encoutered when scrubbing. +errors encoutered while scrubbing. .PP Nothing is modified. .SS scrub @@ -375,11 +383,18 @@ Scrubs the array, checking for silent errors. .PP For each command invocation, the 12% of the array is checked, but nothing that it\(cqs more recent than 10 days. +This means that scrubbing once a week, every bit of data is checked +at least one time every two months. .PP The oldest blocks are scrubbed first ensuring an optimal check. .PP +It\(cqs recommended to run \(a"scrub\(a" on a synched array, to avoid to have +reported error caused by unsynched data. These errors are recognized +as not being silent errors, but are reported anyway. +.PP Any silent error identified is recorded in the content file, -and can be listed with the \(cqstatus\(cq command. +and it\(cqs listed in the \(a"status\(a" command until it\(cqs fixed calling +\(a"fix\(a" and then \(a"sync\(a". .PP The \(a"content\(a" file is modified to update the time of the last check of each block. @@ -397,18 +412,6 @@ hashes are matching. The file data is not read, but only the precomputed hashes are used. .PP Nothing is modified. -.SS pool -Creates or updates in the \(a"pooling\(a" directory a virtual view of all -the files of your disk array. -.PP -The files are not really copied here, but just linked using -symbolic links. -.PP -When updating, all the present symbolic links and empty -subdirectories are deleted and replaced with the new -view of the array. Any othe regular file is left in place. -.PP -Nothing is modified outside the pool directory. .SS check Checks all the files and the redundancy data. All the files are hashed and compared with the snapshot saved @@ -418,6 +421,7 @@ If you use the \(hya, \(hy\(hyaudit\(hyonly option, only the file data is checked, and the redundandy data is ignored. .PP Files are identified by path, and checked by content. +.PP Nothing is modified. .SS fix Checks and fix all the files. It\(cqs like \(a"check\(a" but it @@ -438,16 +442,28 @@ Schedules a rehash of the whole array. .PP This option can be used to change the hash kind used, typically when upgrading from a 32 bits system to a 64 -bits one to switch from the murmur3 to the faster spooky2. +bits one to switch from murmur3 to the faster spooky2. .PP -The rehash isn\(cqt done immediately, but it will take place -incrementally during the \(a"sync\(a" and \(a"scrub\(a" commands. +The rehash isn\(cqt done immediately, but it takes place +progressively during the \(a"sync\(a" and \(a"scrub\(a" commands. .PP -You can get the rehash status using the \(a"status\(a" command. +You can get the rehash state using the \(a"status\(a" command. .PP During the rehash, SnapRAID maintains full functionality, -with the only expection of the \(cqdup\(cq command not able to detect +with the only expection of the \(a"dup\(a" command not able to detect duplicated files using a different hash. +.SS pool +Creates or updates in the \(a"pooling\(a" directory a virtual view of all +the files of your disk array. +.PP +The files are not really copied here, but just linked using +symbolic links. +.PP +When updating, all the present symbolic links and empty +subdirectories are deleted and replaced with the new +view of the array. Any othe regular file is left in place. +.PP +Nothing is modified outside the pool directory. .SH OPTIONS SnapRAID provides the following options: .TP @@ -992,7 +1008,7 @@ Where NAME is the name of the disk, like \(a"d1\(a" as in our previous example. .PP This command will take a long time. .PP -You can also add the \(cq\(hyv\(cq option to see on the console the fixed files. +You can also add the \(a"\(hyv\(a" option to see on the console the fixed files. .PP Take care that you need also few gigabytes free to store the fix.log file. Run it from a disk with some free space. @@ -1045,93 +1061,13 @@ If everything was recovered, this command is immediate. .SH CONTENT SnapRAID stores the list and checksums of your files in the content file. .PP -It\(cqs a text file, listing all the files present in your disk array, +It\(cqs a binary file, listing all the files present in your disk array, with all the checksums to verify their integrity. .PP -You do not need to understand its format, but it\(cqs described here -for documentation. +You do not need to understand its format to use SnapRAID. .PP -This file is read and written by the \(a"sync\(a" command, and only read by -\(a"fix\(a" and \(a"check\(a". -.SS blk_size SIZE -Defines the size of the block in bytes. It must match the size -defined in the configuration file. -.SS checksum CHECKSUM SEED -Defines the checksum kind used. It can be \(a"murmur3\(a" or \(a"spooky2\(a". -SEED is a 128 bits seed used in all the hash operations. -From SnapRAID 2.0 the old \(a"md5\(a" checksum is not supported anymore. -.SS map NAME INDEX [UUID] -Defines the position INDEX of the disk NAME in the parity computation. -The disk UUID is also present if supported for the platform. -.SS sign SIGN -Signature checksum of the content file to ensure that it doesn\(cqt get -corrupted. If you want to modify the content file manually, you have -to remove this line to avoid this check. -.SS file DISK SIZE TIME_SEC[.TIME_NSEC] INODE PATH -Defines a file in the specified DISK. -.PP -The INODE number is used to identify the file in the \(a"sync\(a" -command, allowing to rename or move the file in disk without -the need to recompute the parity for it. -.PP -The SIZE and TIME information are used to identify if the file -changed from the last \(a"sync\(a" command, and if there is the need -to recompute the parity. -.PP -The [.TIME_NSEC] information is optional, and only present if -it\(cqs possible to read a such precise time information. -It\(cqs present only from SnapRAID 2.0. -.PP -The PATH information is used in the \(a"check\(a" and \(a"fix\(a" commands -to identify the file. -.SS blk BLOCK HASH -Defines an ordered parity block, part of the last defined file. -.PP -BLOCK is the block position in the \(a"parity\(a" file. -0 for the first block, 1 for the second one and so on. -.PP -HASH is the hash of the block. In the last block of the file, -the HASH is the hash of only the used part of the block. -.SS new BLOCK -Like \(a"blk\(a", but for new allocated blocks for which the hash is not -yet computed, and the stored parity doesn\(cqt take into account this -new block. -.PP -This field is used only when you interrupt manually the \(a"sync\(a" -command. -.SS chg BLOCK -Like \(a"blk\(a", but for reallocated blocks for which the hash is not -yet computed, and the parity is computed using the previous value -of the block. -.PP -This field is used only when you interrupt manually the \(a"sync\(a" -command. -.SS hole DISK -Defines the list of blocks that are deleted from a disk. -.PP -This field is used only when you interrupt manually the \(a"sync\(a" -command. -.SS off BLOCK -Defines a block deleted from a disk, part of the last defined hole, -for which the parity is computed using the previous value. -.PP -This field is used only when you interrupt manually the \(a"sync\(a" -command. -.SS inf BLOCK TIME -Information shared by all the block at the specified offset. -TIME is the last time of the hash/parity verification done at this -block. -.SS symlink DISK PATH -Defines a symbolic link in the specified DISK. -.SS hardlink DISK PATH -Defines a hard link in the specified DISK. -.SS to PATH -Defines where the previous symlink or hardlink links to. -.SS dir DISK PATH -Defines an empty directory in the specified DISK. -.PP -Directory that contains saved files, are not included, as they -are implicitely defined in the file specifications. +This file is read and written by the \(a"sync\(a" and \(a"scrub\(a" commands, and +only read by \(a"fix\(a" and \(a"check\(a". .SH PARITY SnapRAID stores the redundancy information of your array in the parity and q\(hyparity files. diff --git a/snapraid.d b/snapraid.d index 2b923b9e..1d296b6c 100644 --- a/snapraid.d +++ b/snapraid.d @@ -143,6 +143,12 @@ Getting Started At this point you can start using your array as you like, and periodically update the redundancy information running the "sync" command. + After a "sync" command, it's recommended to run a "scrub" command + to progressively check the old synched data. + + You can verify the status of your array using the "status" command, + that lists all the errors found while scrubbing. + Pooling To have all the files in your array shown in the same directory tree, you can enable "pooling", that consists in creating a virtual view of all @@ -194,7 +200,8 @@ Commands * Make a backup/snapshot -> "sync" * See the files changed from the previous sync -> "diff" - * Check for integrity -> "check" + * Periodically checks old data -> "scrub" + * Check for integrity the full array -> "check" * Restore the last backup/snapshot -> "fix". Take care that the commands have to be written in lower case. @@ -219,7 +226,7 @@ Commands It includes information about the parity fragmentation, how old are the blocks without checking, and all the recorded silent - errors encoutered when scrubbing. + errors encoutered while scrubbing. Nothing is modified. @@ -228,11 +235,18 @@ Commands For each command invocation, the 12% of the array is checked, but nothing that it's more recent than 10 days. + This means that scrubbing once a week, every bit of data is checked + at least one time every two months. The oldest blocks are scrubbed first ensuring an optimal check. + It's recommended to run "scrub" on a synched array, to avoid to have + reported error caused by unsynched data. These errors are recognized + as not being silent errors, but are reported anyway. + Any silent error identified is recorded in the content file, - and can be listed with the 'status' command. + and it's listed in the "status" command until it's fixed calling + "fix" and then "sync". The "content" file is modified to update the time of the last check of each block. @@ -253,19 +267,6 @@ Commands Nothing is modified. - pool - Creates or updates in the "pooling" directory a virtual view of all - the files of your disk array. - - The files are not really copied here, but just linked using - symbolic links. - - When updating, all the present symbolic links and empty - subdirectories are deleted and replaced with the new - view of the array. Any othe regular file is left in place. - - Nothing is modified outside the pool directory. - check Checks all the files and the redundancy data. All the files are hashed and compared with the snapshot saved @@ -275,6 +276,7 @@ Commands data is checked, and the redundandy data is ignored. Files are identified by path, and checked by content. + Nothing is modified. fix @@ -297,17 +299,30 @@ Commands This option can be used to change the hash kind used, typically when upgrading from a 32 bits system to a 64 - bits one to switch from the murmur3 to the faster spooky2. + bits one to switch from murmur3 to the faster spooky2. - The rehash isn't done immediately, but it will take place - incrementally during the "sync" and "scrub" commands. + The rehash isn't done immediately, but it takes place + progressively during the "sync" and "scrub" commands. - You can get the rehash status using the "status" command. + You can get the rehash state using the "status" command. During the rehash, SnapRAID maintains full functionality, - with the only expection of the 'dup' command not able to detect + with the only expection of the "dup" command not able to detect duplicated files using a different hash. + pool + Creates or updates in the "pooling" directory a virtual view of all + the files of your disk array. + + The files are not really copied here, but just linked using + symbolic links. + + When updating, all the present symbolic links and empty + subdirectories are deleted and replaced with the new + view of the array. Any othe regular file is left in place. + + Nothing is modified outside the pool directory. + Options SnapRAID provides the following options: @@ -717,7 +732,7 @@ Recovering This command will take a long time. - You can also add the '-v' option to see on the console the fixed files. + You can also add the "-v" option to see on the console the fixed files. Take care that you need also few gigabytes free to store the fix.log file. Run it from a disk with some free space. @@ -763,108 +778,13 @@ Recovering Content SnapRAID stores the list and checksums of your files in the content file. - It's a text file, listing all the files present in your disk array, + It's a binary file, listing all the files present in your disk array, with all the checksums to verify their integrity. - You do not need to understand its format, but it's described here - for documentation. + You do not need to understand its format to use SnapRAID. - This file is read and written by the "sync" command, and only read by - "fix" and "check". - - blk_size SIZE - Defines the size of the block in bytes. It must match the size - defined in the configuration file. - - checksum CHECKSUM SEED - Defines the checksum kind used. It can be "murmur3" or "spooky2". - SEED is a 128 bits seed used in all the hash operations. - From SnapRAID 2.0 the old "md5" checksum is not supported anymore. - - map NAME INDEX [UUID] - Defines the position INDEX of the disk NAME in the parity computation. - The disk UUID is also present if supported for the platform. - - sign SIGN - Signature checksum of the content file to ensure that it doesn't get - corrupted. If you want to modify the content file manually, you have - to remove this line to avoid this check. - - file DISK SIZE TIME_SEC[.TIME_NSEC] INODE PATH - Defines a file in the specified DISK. - - The INODE number is used to identify the file in the "sync" - command, allowing to rename or move the file in disk without - the need to recompute the parity for it. - - The SIZE and TIME information are used to identify if the file - changed from the last "sync" command, and if there is the need - to recompute the parity. - - The [.TIME_NSEC] information is optional, and only present if - it's possible to read a such precise time information. - It's present only from SnapRAID 2.0. - - The PATH information is used in the "check" and "fix" commands - to identify the file. - - blk BLOCK HASH - Defines an ordered parity block, part of the last defined file. - - BLOCK is the block position in the "parity" file. - 0 for the first block, 1 for the second one and so on. - - HASH is the hash of the block. In the last block of the file, - the HASH is the hash of only the used part of the block. - - new BLOCK - Like "blk", but for new allocated blocks for which the hash is not - yet computed, and the stored parity doesn't take into account this - new block. - - This field is used only when you interrupt manually the "sync" - command. - - chg BLOCK - Like "blk", but for reallocated blocks for which the hash is not - yet computed, and the parity is computed using the previous value - of the block. - - This field is used only when you interrupt manually the "sync" - command. - - hole DISK - Defines the list of blocks that are deleted from a disk. - - This field is used only when you interrupt manually the "sync" - command. - - off BLOCK - Defines a block deleted from a disk, part of the last defined hole, - for which the parity is computed using the previous value. - - This field is used only when you interrupt manually the "sync" - command. - - inf BLOCK TIME - Information shared by all the block at the specified offset. - TIME is the last time of the hash/parity verification done at this - block. - - symlink DISK PATH - Defines a symbolic link in the specified DISK. - - hardlink DISK PATH - Defines a hard link in the specified DISK. - - to PATH - Defines where the previous symlink or hardlink links to. - - dir DISK PATH - Defines an empty directory in the specified DISK. - - Directory that contains saved files, are not included, as they - are implicitely defined in the file specifications. + This file is read and written by the "sync" and "scrub" commands, and + only read by "fix" and "check". Parity SnapRAID stores the redundancy information of your array in the parity diff --git a/snapraid.txt b/snapraid.txt index a9e30bb9..7858f656 100644 --- a/snapraid.txt +++ b/snapraid.txt @@ -156,6 +156,12 @@ When this command completes, your data is SAFE. At this point you can start using your array as you like, and periodically update the redundancy information running the "sync" command. +After a "sync" command, it's recommended to run a "scrub" command +to progressively check the old synched data. + +You can verify the status of your array using the "status" command, +that lists all the errors found while scrubbing. + 4.1 Pooling ----------- @@ -214,7 +220,8 @@ SnapRAID provides four simple commands that allow to: * Make a backup/snapshot -> "sync" * See the files changed from the previous sync -> "diff" -* Check for integrity -> "check" +* Periodically checks old data -> "scrub" +* Check for integrity the full array -> "check" * Restore the last backup/snapshot -> "fix". Take care that the commands have to be written in lower case. @@ -243,7 +250,7 @@ Prints a summary of the state of the disk array. It includes information about the parity fragmentation, how old are the blocks without checking, and all the recorded silent -errors encoutered when scrubbing. +errors encoutered while scrubbing. Nothing is modified. @@ -254,11 +261,18 @@ Scrubs the array, checking for silent errors. For each command invocation, the 12% of the array is checked, but nothing that it's more recent than 10 days. +This means that scrubbing once a week, every bit of data is checked +at least one time every two months. The oldest blocks are scrubbed first ensuring an optimal check. +It's recommended to run "scrub" on a synched array, to avoid to have +reported error caused by unsynched data. These errors are recognized +as not being silent errors, but are reported anyway. + Any silent error identified is recorded in the content file, -and can be listed with the 'status' command. +and it's listed in the "status" command until it's fixed calling +"fix" and then "sync". The "content" file is modified to update the time of the last check of each block. @@ -283,22 +297,7 @@ precomputed hashes are used. Nothing is modified. -5.6 pool --------- - -Creates or updates in the "pooling" directory a virtual view of all -the files of your disk array. - -The files are not really copied here, but just linked using -symbolic links. - -When updating, all the present symbolic links and empty -subdirectories are deleted and replaced with the new -view of the array. Any othe regular file is left in place. - -Nothing is modified outside the pool directory. - -5.7 check +5.6 check --------- Checks all the files and the redundancy data. @@ -309,9 +308,10 @@ If you use the -a, --audit-only option, only the file data is checked, and the redundandy data is ignored. Files are identified by path, and checked by content. + Nothing is modified. -5.8 fix +5.7 fix ------- Checks and fix all the files. It's like "check" but it @@ -328,24 +328,39 @@ The "content" file is NOT modified. The "parity" and "q-parity" files are modified if necessary. The files in the array are modified if necessary. -5.9 rehash +5.8 rehash ---------- Schedules a rehash of the whole array. This option can be used to change the hash kind used, typically when upgrading from a 32 bits system to a 64 -bits one to switch from the murmur3 to the faster spooky2. +bits one to switch from murmur3 to the faster spooky2. -The rehash isn't done immediately, but it will take place -incrementally during the "sync" and "scrub" commands. +The rehash isn't done immediately, but it takes place +progressively during the "sync" and "scrub" commands. -You can get the rehash status using the "status" command. +You can get the rehash state using the "status" command. During the rehash, SnapRAID maintains full functionality, -with the only expection of the 'dup' command not able to detect +with the only expection of the "dup" command not able to detect duplicated files using a different hash. +5.9 pool +-------- + +Creates or updates in the "pooling" directory a virtual view of all +the files of your disk array. + +The files are not really copied here, but just linked using +symbolic links. + +When updating, all the present symbolic links and empty +subdirectories are deleted and replaced with the new +view of the array. Any othe regular file is left in place. + +Nothing is modified outside the pool directory. + 6 OPTIONS ========= @@ -787,7 +802,7 @@ Where NAME is the name of the disk, like "d1" as in our previous example. This command will take a long time. -You can also add the '-v' option to see on the console the fixed files. +You can also add the "-v" option to see on the console the fixed files. Take care that you need also few gigabytes free to store the fix.log file. Run it from a disk with some free space. @@ -840,138 +855,13 @@ If everything was recovered, this command is immediate. SnapRAID stores the list and checksums of your files in the content file. -It's a text file, listing all the files present in your disk array, +It's a binary file, listing all the files present in your disk array, with all the checksums to verify their integrity. -You do not need to understand its format, but it's described here -for documentation. +You do not need to understand its format to use SnapRAID. -This file is read and written by the "sync" command, and only read by -"fix" and "check". - -10.1 blk_size SIZE ------------------- - -Defines the size of the block in bytes. It must match the size -defined in the configuration file. - -10.2 checksum CHECKSUM SEED ---------------------------- - -Defines the checksum kind used. It can be "murmur3" or "spooky2". -SEED is a 128 bits seed used in all the hash operations. -From SnapRAID 2.0 the old "md5" checksum is not supported anymore. - -10.3 map NAME INDEX [UUID] --------------------------- - -Defines the position INDEX of the disk NAME in the parity computation. -The disk UUID is also present if supported for the platform. - -10.4 sign SIGN --------------- - -Signature checksum of the content file to ensure that it doesn't get -corrupted. If you want to modify the content file manually, you have -to remove this line to avoid this check. - -10.5 file DISK SIZE TIME_SEC[.TIME_NSEC] INODE PATH ---------------------------------------------------- - -Defines a file in the specified DISK. - -The INODE number is used to identify the file in the "sync" -command, allowing to rename or move the file in disk without -the need to recompute the parity for it. - -The SIZE and TIME information are used to identify if the file -changed from the last "sync" command, and if there is the need -to recompute the parity. - -The [.TIME_NSEC] information is optional, and only present if -it's possible to read a such precise time information. -It's present only from SnapRAID 2.0. - -The PATH information is used in the "check" and "fix" commands -to identify the file. - -10.6 blk BLOCK HASH -------------------- - -Defines an ordered parity block, part of the last defined file. - -BLOCK is the block position in the "parity" file. -0 for the first block, 1 for the second one and so on. - -HASH is the hash of the block. In the last block of the file, -the HASH is the hash of only the used part of the block. - -10.7 new BLOCK --------------- - -Like "blk", but for new allocated blocks for which the hash is not -yet computed, and the stored parity doesn't take into account this -new block. - -This field is used only when you interrupt manually the "sync" -command. - -10.8 chg BLOCK --------------- - -Like "blk", but for reallocated blocks for which the hash is not -yet computed, and the parity is computed using the previous value -of the block. - -This field is used only when you interrupt manually the "sync" -command. - -10.9 hole DISK --------------- - -Defines the list of blocks that are deleted from a disk. - -This field is used only when you interrupt manually the "sync" -command. - -10.10 off BLOCK ---------------- - -Defines a block deleted from a disk, part of the last defined hole, -for which the parity is computed using the previous value. - -This field is used only when you interrupt manually the "sync" -command. - -10.11 inf BLOCK TIME --------------------- - -Information shared by all the block at the specified offset. -TIME is the last time of the hash/parity verification done at this -block. - -10.12 symlink DISK PATH ------------------------ - -Defines a symbolic link in the specified DISK. - -10.13 hardlink DISK PATH ------------------------- - -Defines a hard link in the specified DISK. - -10.14 to PATH -------------- - -Defines where the previous symlink or hardlink links to. - -10.15 dir DISK PATH -------------------- - -Defines an empty directory in the specified DISK. - -Directory that contains saved files, are not included, as they -are implicitely defined in the file specifications. +This file is read and written by the "sync" and "scrub" commands, and +only read by "fix" and "check". 11 PARITY