diff --git a/HISTORY b/HISTORY index 193c7f7b..75225e61 100644 --- a/HISTORY +++ b/HISTORY @@ -3,10 +3,10 @@ SnapRAID HISTORY 0.2 2011/3 ========== -* Exclusion list of files and directories -* More automatic tests -* Better error management -* Better documentation +* Files are identified by inode and not anymore by name. +* Exclusion list of files and directories. +* Precise error management. +* More regression tests. 0.1 2011/3 diff --git a/snapraid.d b/snapraid.d index 9eba0c0e..86acbf78 100644 --- a/snapraid.d +++ b/snapraid.d @@ -119,6 +119,65 @@ Getting Started last "sync" command executed. It works like a snapshot was taken in "sync". +Commands + SnapRAID provides three simple commands that allow to: + + * Make a backup/snapshot -> "sync" + * Check for integrity -> "check" + * Restore the last backup/snapshot -> "fix". + + =sync + Updates the redundancy information. All the modified files + in the disk array are read, and the redundancy data is + updated. + You can stop this process at any time pressing Ctrl+C, + without losing the work already done. + + =check + Checks all the files the redundancy data. All the files + are hashed and compared with the snapshot saved in the + previous "sync" command. + + =fix + Checks and fix all the files. It's like "check" but it + also tries to fix problems reverting the state of the + disk array at the previous "sync" command. + +Options + -c, --conf CONFIG + Selects the configuration file. If not specified is assumed + the file `/etc/snapraid.conf'. + + -s, --start BLOCK + Starts the processing from the specified + block number. It could be useful to easy retry to check or + fix some specific block, in case of a damaged disk. + + -Z, --force-zero + Forces the insecure operation of synching a file with zero + size that before was not empty. + If SnapRAID detects such condition, it stops proceeding + unless you specify this option. + This allows to easily detect when after a system crash, + some accessed files were zeroed. + + -E, --force-empty + Forces the insecure operation of synching an empty disk + that before was not empty. + If SnapRAID detects such condition, it stops proceeding + unless you specify this option. + This allows to easily detect when a data filesystem is not + mounted. + + -v, --verbose + Prints more information in the processing. + + -h, --help + Prints a short help screen. + + -V, --version + Prints the program version. + Configuration SnapRAID requires a configuration file to know where your disk array is located, and where storing the redundancy information. @@ -149,7 +208,7 @@ Configuration keep the NAME fixed. You should use one option for each disk of the array. - =exclude GLOB_PATTERN + =exclude PATTERN Defines the file or directory patterns to exclude from the sync process. See the PATTERN section for more details in the pattern specifications. @@ -182,30 +241,6 @@ Configuration :exclude tmp/ :block_size 256 -Commands - SnapRAID provides three simple commands that allow to: - - * Make a backup/snapshot -> "sync" - * Check for integrity -> "check" - * Restore the last backup/snapshot -> "fix". - - =sync - Updates the redundancy information. All the modified files - in the disk array are read, and the redundancy data is - updated. - You can stop this process at any time pressing Ctrl+C, - without losing the work already done. - - =check - Checks all the files the redundancy data. All the files - are hashed and compared with the snapshot saved in the - previous "sync" command. - - =fix - Checks and fix all the files. It's like "check" but it - also tries to fix problems reverting the state of the - disk array at the previous "sync" command. - Pattern Patterns are used to define the files and directories to exclude from the redundancy process. @@ -245,40 +280,31 @@ Pattern :# Excludes any directory named "tmp" :exclude tmp/ -Options - -c, --conf CONFIG - Selects the configuration file. If not specified is assumed - the file `/etc/snapraid.conf'. +Content + SnapRAID creates a content file describing the content of your disk array. + You do not have to to modify it, altought the format of this file is + described here. - -s, --start BLOCK - Starts the processing from the specified - block number. It could be useful to easy retry to check or - fix some specific block, in case of a damaged disk. + =blk_size SIZE + Defines the size of the block in bytes. It must match the size + defined in the configuration file. - -Z, --force-zero - Forces the insecure operation of synching a file with zero - size that before was not empty. - If SnapRAID detects such condition, it stops proceeding - unless you specify this option. - This allows to easily detect when after a system crash, - some accessed files were zeroed. + =file DISK SIZE TIME INODE PATH + Defines a file in the specified DISK. + The INODE information is used to identify the file in the 'sync' + command, allowing to move the file anywhere in disk without + the need to recompute the parity for it. + The SIZE and TIME information are used to identify if the file + changed from the last 'sync' command, and if there is the need + to recompute the parity. + The PATH information is used in the 'check' and 'fix' commands + to identify the file. - -E, --force-empty - Forces the insecure operation of synching an empty disk - that before was not empty. - If SnapRAID detects such condition, it stops proceeding - unless you specify this option. - This allows to easily detect when a data filesystem is not - mounted. - - -v, --verbose - Prints more information in the processing. - - -h, --help - Prints a short help screen. - - -V, --version - Prints the program version. + =blk BLOCK HASH + Defines the ordered blocks used by the last defined file. + BLOCK is the block position in the parity file. + HASH is the md5 of the block. In the last block of the file, + the HASH is the hash only of the used part of the block. Copyright This file is Copyright (C) 2011 Andrea Mazzoleni diff --git a/snapraid.html b/snapraid.html index 2b6fb4ea..a23bcfac 100644 --- a/snapraid.html +++ b/snapraid.html @@ -178,100 +178,6 @@ Note that the fix command will revert your data at the state of the last "sync" command executed. It works like a snapshot was taken in "sync".

5 -Configuration -

-SnapRAID requires a configuration file to know where your disk array -is located, and where storing the redundancy information. -

-This configuration file is located in /etc/snapraid.conf and -it should contains the following options: - - -
-parity FILE -
-Defines the file to use to store the redundancy information. -It must be placed in a disk dedicated for this porpose with -as much free space as the biggest disk in the array. -Leaving the parity disk reserved for only this file, ensures that -it doesn't get fragmented, improving the performance. -This option can be used only one time. -
- - -
-content FILE -
-Defines the file to use to store the content of the redundancy -organization. -It can be placed in the same disk of the parity file, or better -in another disk, but NOT in a disk of the array. -This option can be used only one time. -
- - -
-disk NAME DIR -
-Defines the name and the mount point of the disks of the array. -NAME is used to identify the disk, and it must be unique. -DIR is the mount point of the disk in the filesystem. -You can change the mount point as you like, as far you -keep the NAME fixed. -You should use one option for each disk of the array. -
- - -
-exclude GLOB_PATTERN -
-Defines the file or directory patterns to exclude from the sync -process. See the PATTERN section for more details in the -pattern specifications. -This option can be used many times. -
- - -
-block_size SIZE_IN_KILOBYTES -
-Defines the basic block size in kilo bytes of -the redundancy blocks. The default is 256 and it should -work for most conditions. -You should use this option only if you do not have enough -memory to run SnapRAID. -It requires to run TS*24/BS bytes, where TS is the total -size in bytes of your disk array, and BS is the block size -in bytes. -
-

-
-For example with 6 disk of 2 TiB and a block size of 256 KiB -(1 KiB = 1024 Bytes) you have: -
-

-
-
-memory = (6 * 2 * 2^40) * 24 / (256 * 2^10) = 1.1 GiB
-
-
-

-An example of a typical configuration is: -

-
-
-parity /mnt/diskpar/parity
-content /mnt/diskpar/content
-disk d1 /mnt/disk1/
-disk d2 /mnt/disk2/
-disk d3 /mnt/disk3/
-exclude *.bak
-exclude /lost+found/
-exclude tmp/
-block_size 256
-
-
-

6 Commands

SnapRAID provides three simple commands that allow to: @@ -315,67 +221,7 @@ also tries to fix problems reverting the state of the disk array at the previous "sync" command. -

7 -Pattern -

-Patterns are used to define the files and directories to exclude -from the redundancy process. -

-It makes sense to exclude any file not worth to be saved. -

-There are four different types of patterns: - - -
-FILE -
-Excludes any file named as FILE. You can use any globbing -character like * and ?. -This pattern is applied only to files and not to directories. -
- - -
-DIR/ -
-Excludes any directory named DIR. You can use any globbing -character like * and ?. -This pattern is applied only to directories and not to files. -
- - -
-/PATH/FILE -
-Excludes the exact specified file path. You can use any -globbing character like * and ? but they never matches a -directory slash. -This pattern is applied only to files and not to directories. -
- - -
-/PATH/DIR/ -
-Excludes the exact specified directory path. You can use any -globbing character like * and ? but they never matches a -directory slash. -This pattern is applied only to directories and not to files. -
-

-For example: -

-
-
-# Excludes any file named "*.bak"
-exclude *.bak
-# Excludes the root directory "lost+found"
-exclude /lost+found/
-# Excludes any directory named "tmp"
-exclude tmp/
-
-
-

8 +

6 Options

@@ -440,11 +286,204 @@ Prints a short help screen. Prints the program version.
+

7 +Configuration +

+SnapRAID requires a configuration file to know where your disk array +is located, and where storing the redundancy information. +

+This configuration file is located in /etc/snapraid.conf and +it should contains the following options: + + +
+parity FILE +
+Defines the file to use to store the redundancy information. +It must be placed in a disk dedicated for this porpose with +as much free space as the biggest disk in the array. +Leaving the parity disk reserved for only this file, ensures that +it doesn't get fragmented, improving the performance. +This option can be used only one time. +
+ + +
+content FILE +
+Defines the file to use to store the content of the redundancy +organization. +It can be placed in the same disk of the parity file, or better +in another disk, but NOT in a disk of the array. +This option can be used only one time. +
+ + +
+disk NAME DIR +
+Defines the name and the mount point of the disks of the array. +NAME is used to identify the disk, and it must be unique. +DIR is the mount point of the disk in the filesystem. +You can change the mount point as you like, as far you +keep the NAME fixed. +You should use one option for each disk of the array. +
+ + +
+exclude PATTERN +
+Defines the file or directory patterns to exclude from the sync +process. See the PATTERN section for more details in the +pattern specifications. +This option can be used many times. +
+ + +
+block_size SIZE_IN_KILOBYTES +
+Defines the basic block size in kilo bytes of +the redundancy blocks. The default is 256 and it should +work for most conditions. +You should use this option only if you do not have enough +memory to run SnapRAID. +It requires to run TS*24/BS bytes, where TS is the total +size in bytes of your disk array, and BS is the block size +in bytes. +
+

+
+For example with 6 disk of 2 TiB and a block size of 256 KiB +(1 KiB = 1024 Bytes) you have: +
+

+
+
+memory = (6 * 2 * 2^40) * 24 / (256 * 2^10) = 1.1 GiB
+
+
+

+An example of a typical configuration is: +

+
+
+parity /mnt/diskpar/parity
+content /mnt/diskpar/content
+disk d1 /mnt/disk1/
+disk d2 /mnt/disk2/
+disk d3 /mnt/disk3/
+exclude *.bak
+exclude /lost+found/
+exclude tmp/
+block_size 256
+
+
+

8 +Pattern +

+Patterns are used to define the files and directories to exclude +from the redundancy process. +

+It makes sense to exclude any file not worth to be saved. +

+There are four different types of patterns: + + +
+FILE +
+Excludes any file named as FILE. You can use any globbing +character like * and ?. +This pattern is applied only to files and not to directories. +
+ + +
+DIR/ +
+Excludes any directory named DIR. You can use any globbing +character like * and ?. +This pattern is applied only to directories and not to files. +
+ + +
+/PATH/FILE +
+Excludes the exact specified file path. You can use any +globbing character like * and ? but they never matches a +directory slash. +This pattern is applied only to files and not to directories. +
+ + +
+/PATH/DIR/ +
+Excludes the exact specified directory path. You can use any +globbing character like * and ? but they never matches a +directory slash. +This pattern is applied only to directories and not to files. +
+

+For example: +

+
+
+# Excludes any file named "*.bak"
+exclude *.bak
+# Excludes the root directory "lost+found"
+exclude /lost+found/
+# Excludes any directory named "tmp"
+exclude tmp/
+
+

9 +Content +

+SnapRAID creates a content file describing the content of your disk array. +You do not have to to modify it, altought the format of this file is +described here. + + +
+blk_size SIZE +
+Defines the size of the block in bytes. It must match the size +defined in the configuration file. +
+ + +
+file DISK SIZE TIME INODE PATH +
+Defines a file in the specified DISK. +The INODE information is used to identify the file in the 'sync' +command, allowing to move the file anywhere in disk without +the need to recompute the parity for it. +The SIZE and TIME information are used to identify if the file +changed from the last 'sync' command, and if there is the need +to recompute the parity. +The PATH information is used in the 'check' and 'fix' commands +to identify the file. +
+ + +
+blk BLOCK HASH +
+Defines the ordered blocks used by the last defined file. +BLOCK is the block position in the parity file. +HASH is the md5 of the block. In the last block of the file, +the HASH is the hash only of the used part of the block. +
+

10 Copyright

This file is Copyright (C) 2011 Andrea Mazzoleni -

10 +

11 See Also

rsync(1) diff --git a/snapraid.txt b/snapraid.txt index 4e40b4f0..13996f3e 100644 --- a/snapraid.txt +++ b/snapraid.txt @@ -133,7 +133,72 @@ last "sync" command executed. It works like a snapshot was taken in "sync". -5 CONFIGURATION +5 COMMANDS +========== + +SnapRAID provides three simple commands that allow to: + +* Make a backup/snapshot -> "sync" +* Check for integrity -> "check" +* Restore the last backup/snapshot -> "fix". + + sync + Updates the redundancy information. All the modified files + in the disk array are read, and the redundancy data is + updated. + You can stop this process at any time pressing Ctrl+C, + without losing the work already done. + + check + Checks all the files the redundancy data. All the files + are hashed and compared with the snapshot saved in the + previous "sync" command. + + fix + Checks and fix all the files. It's like "check" but it + also tries to fix problems reverting the state of the + disk array at the previous "sync" command. + + +6 OPTIONS +========= + + -c, --conf CONFIG + Selects the configuration file. If not specified is assumed + the file `/etc/snapraid.conf'. + + -s, --start BLOCK + Starts the processing from the specified + block number. It could be useful to easy retry to check or + fix some specific block, in case of a damaged disk. + + -Z, --force-zero + Forces the insecure operation of synching a file with zero + size that before was not empty. + If SnapRAID detects such condition, it stops proceeding + unless you specify this option. + This allows to easily detect when after a system crash, + some accessed files were zeroed. + + -E, --force-empty + Forces the insecure operation of synching an empty disk + that before was not empty. + If SnapRAID detects such condition, it stops proceeding + unless you specify this option. + This allows to easily detect when a data filesystem is not + mounted. + + -v, --verbose + Prints more information in the processing. + + -h, --help + Prints a short help screen. + + -V, --version + Prints the program version. + + +7 CONFIGURATION =============== SnapRAID requires a configuration file to know where your disk array @@ -165,7 +230,7 @@ it should contains the following options: keep the NAME fixed. You should use one option for each disk of the array. - exclude GLOB_PATTERN + exclude PATTERN Defines the file or directory patterns to exclude from the sync process. See the PATTERN section for more details in the pattern specifications. @@ -199,34 +264,7 @@ An example of a typical configuration is: block_size 256 -6 COMMANDS -========== - -SnapRAID provides three simple commands that allow to: - -* Make a backup/snapshot -> "sync" -* Check for integrity -> "check" -* Restore the last backup/snapshot -> "fix". - - sync - Updates the redundancy information. All the modified files - in the disk array are read, and the redundancy data is - updated. - You can stop this process at any time pressing Ctrl+C, - without losing the work already done. - - check - Checks all the files the redundancy data. All the files - are hashed and compared with the snapshot saved in the - previous "sync" command. - - fix - Checks and fix all the files. It's like "check" but it - also tries to fix problems reverting the state of the - disk array at the previous "sync" command. - - -7 PATTERN +8 PATTERN ========= Patterns are used to define the files and directories to exclude @@ -268,51 +306,42 @@ For example: exclude tmp/ -8 OPTIONS +9 CONTENT ========= - -c, --conf CONFIG - Selects the configuration file. If not specified is assumed - the file `/etc/snapraid.conf'. +SnapRAID creates a content file describing the content of your disk array. +You do not have to to modify it, altought the format of this file is +described here. - -s, --start BLOCK - Starts the processing from the specified - block number. It could be useful to easy retry to check or - fix some specific block, in case of a damaged disk. + blk_size SIZE + Defines the size of the block in bytes. It must match the size + defined in the configuration file. - -Z, --force-zero - Forces the insecure operation of synching a file with zero - size that before was not empty. - If SnapRAID detects such condition, it stops proceeding - unless you specify this option. - This allows to easily detect when after a system crash, - some accessed files were zeroed. + file DISK SIZE TIME INODE PATH + Defines a file in the specified DISK. + The INODE information is used to identify the file in the 'sync' + command, allowing to move the file anywhere in disk without + the need to recompute the parity for it. + The SIZE and TIME information are used to identify if the file + changed from the last 'sync' command, and if there is the need + to recompute the parity. + The PATH information is used in the 'check' and 'fix' commands + to identify the file. - -E, --force-empty - Forces the insecure operation of synching an empty disk - that before was not empty. - If SnapRAID detects such condition, it stops proceeding - unless you specify this option. - This allows to easily detect when a data filesystem is not - mounted. - - -v, --verbose - Prints more information in the processing. - - -h, --help - Prints a short help screen. - - -V, --version - Prints the program version. + blk BLOCK HASH + Defines the ordered blocks used by the last defined file. + BLOCK is the block position in the parity file. + HASH is the md5 of the block. In the last block of the file, + the HASH is the hash only of the used part of the block. -9 COPYRIGHT -=========== +10 COPYRIGHT +============ This file is Copyright (C) 2011 Andrea Mazzoleni -10 SEE ALSO +11 SEE ALSO =========== rsync(1)