Merge branch 'master' into tripleparity

Conflicts:
	Makefile.am
	state.c
This commit is contained in:
Andrea Mazzoleni 2013-09-15 21:23:15 +02:00
commit f61fcfd850
9 changed files with 167 additions and 139 deletions

View file

@ -1,8 +1,8 @@
SnapRAID HISTORY
================
4.0 2013/9 (RELEASE CANDIDATE)
==============================
4.0 2013/9
==========
* New 'scrub' command to periodically check the oldest blocks for silent
errors without the need to scan the whole array.
* New 'status' command to check the fragmentation, the last check time
@ -14,12 +14,13 @@ SnapRAID HISTORY
by name if a file is not found searching by inode, automatically
reassigning inodes in restored files without needing to sync
again the file.
This happens only if the file has the same name, size and timestamp
This happens only if the file has the same path, size and timestamp
at nanosecond precision.
* Added a hash seed to make harder intentional collision attacks.
* When insering files for the first time, sort them by their physical
address to improve read performance.
* Optimized the cache use for the all the RAID computations.
This improves a lot the RAID performance.
* Better selection of the RAID6 implementation for different CPUs.
* Added RAID5/RAID6 mmx and sse2 implementations with unrolling by 4.
They are a little faster than the previous unroll by 2.

View file

@ -253,7 +253,7 @@ endif
$(TESTENV) ./snapraid$(EXEEXT) $(CHECKFLAGS) -c $(RAIDTP) fix -l test.log
$(TESTENV) ./snapraid$(EXEEXT) $(CHECKFLAGS) -c $(CONF) check
$(TESTENV) ./snapraid$(EXEEXT) $(CHECKFLAGS) -c $(CONF) sync
echo --- Delete some files and create some new, sync and check in multiple steps
echo --- Delete some files and create some new, sync and check in multiple steps using text content
rm test/disk1/4*
rm test/disk2/5*
rm test/disk3/6*

View file

@ -30,11 +30,11 @@ snapraid [\(hyc, \(hy\(hyconf CONFIG]
.PD 0
.PP
.PD
[\(hys, \(hy\(hystart BLKSTART] [\(hyt, \(hy\(hycount BLKCOUNT]
[\(hyv, \(hy\(hyverbose] [\(hyl, \(hy\(hylog FILE]
.PD 0
.PP
.PD
[\(hyv, \(hy\(hyverbose] [\(hyl, \(hy\(hylog FILE]
[\(hys, \(hy\(hystart BLKSTART] [\(hyt, \(hy\(hycount BLKCOUNT]
.PD 0
.PP
.PD
@ -331,11 +331,12 @@ snapraid scrub
.PD
.RE
.PP
As difference than \(a"check\(a" this command verifies only the oldest data
This command is similar at \(a"check\(a" but verifies only the oldest data
in your array. Every run of the command checks about 12% of the data,
but nothing newer than 10 days.
You can use the \(hyp, \(hy\(hypercentage option to specify a different amount,
and the \(hyo, \(hy\(hyolder\(hythan option to specify a different age in days:
and the \(hyo, \(hy\(hyolder\(hythan option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
.PP
.RS 4
snapraid \(hyp 5 \(hyo 20 scrub
@ -345,7 +346,7 @@ snapraid \(hyp 5 \(hyo 20 scrub
.RE
.PP
If silent errors are found, the corresponding blocks are marked as bad
in the \(a"concent\(a" file, and listed in the \(a"status\(a" command.
in the \(a"content\(a" file, and listed in the \(a"status\(a" command.
.PP
.RS 4
snapraid status
@ -364,8 +365,8 @@ snapraid \(hye fix
.PD
.RE
.PP
and at the next \(a"scrub\(a" the errors will disappear from the \(a"status\(a"
report. You can use \(hyp 0 to scrub only them.
At the next \(a"scrub\(a" the errors will disappear from the \(a"status\(a" report
if really fixed. You can use \(hyp 0 to scrub only blocks marked as bad.
.PP
.RS 4
snapraid \(hyp 0 scrub
@ -424,9 +425,12 @@ Updates the redundancy information. All the modified files
in the disk array are read, and the redundancy data is
recomputed.
.PP
Files are identified by inode and checked by time and size,
meaning that you can move them on the disk without triggering
any redundancy recomputation.
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the redundancy data is
recomputed for the whole file.
Using inode allow you can move them on the disk
without triggering any redundancy recomputation.
.PP
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
@ -470,7 +474,7 @@ This means that scrubbing once a week, every bit of data is checked
at least one time every two months.
.PP
You can use the \(hyp, \(hy\(hypercentage option to specify a different amount,
and the \(hyo, \(hy\(hyolder\(hythan option to specify a different age.
and the \(hyo, \(hy\(hyolder\(hythan option to specify a different age in days.
Note that if only one of \(hyp and \(hyo is specified the default value of
the other option is not used.
.PP
@ -528,7 +532,10 @@ Schedules a rehash of the whole array.
.PP
This option can be used to change the hash kind used,
typically when upgrading from a 32 bits system to a 64
bits one to switch from murmur3 to the faster spooky2.
bits one to switch from MurmurHash3 to the faster SpookyHash.
.PP
If you are already using the optimal hash, this command
do nothing and just inform you that nothing has to be done.
.PP
The rehash isn\(cqt done immediately, but it takes place
progressively during the \(a"sync\(a" and \(a"scrub\(a" commands.
@ -645,7 +652,7 @@ This option can be used only with the \(a"sync\(a" command.
.TP
.B \(hyU, \(hy\(hyforce\(hyuuid
Forces the insecure operation of syncing, checking and fixing
with disks that have changed the UUID.
with disks that have changed their UUID.
If SnapRAID detects that some disks have changed UUID,
it stops proceeding unless you specify this option.
This allows to detect when your disks are mounted in the
@ -661,18 +668,9 @@ Forces the insecure operation of fixing with disks on the same
physical device.
If SnapRAID detects that two disks have the same device ID,
it stops proceeding, because it\(cqs not a supported configuration.
But it could happen what you want to temporarely restore a lost
But it could happen that you want to temporarely restore a lost
disk in the free space left in an already used disk. and this
option will allow you to continue anyway.
.TP
.B \(hys, \(hy\(hystart BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
.TP
.B \(hyt, \(hy\(hycount BLKCOUNT
Process only the specified number of blocks.
It\(cqs present mainly for advanced manual recovering.
option allows you to continue anyway.
.TP
.B \(hyl, \(hy\(hylog FILE
Outputs a detailed log to the specified file.
@ -681,6 +679,16 @@ any file is not recoverable and why.
If this option is not specified, the log is printed in
the console using the standard error stream.
.TP
.B \(hys, \(hy\(hystart BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
It\(cqs present mainly for advanced manual recovering.
.TP
.B \(hyt, \(hy\(hycount BLKCOUNT
Processes only the specified number of blocks.
It\(cqs present mainly for advanced manual recovering.
.TP
.B \(hyv, \(hy\(hyverbose
Prints more information in the processing.
.TP
@ -1144,25 +1152,8 @@ and q\(hyparity files.
They are binary files, containing the computed redundancy of all the
blocks defined in the \(a"content\(a" file.
.PP
You do not need to understand its format, but it\(cqs described here
for documentation.
.PP
These files are read and written by the \(a"sync\(a" and \(a"fix\(a" commands, and
only read by \(a"check\(a".
.PP
For all the blocks at a given position, the parity and the q\(hyparity
are computed as specified in:
.PP
.RS 4
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
.PD 0
.PP
.PD
.RE
.PP
When a file block is shorter than the default block size, for example
because it\(cqs the last block of a file, it\(cqs assumed as filled with 0
at the end.
.SH ENCODING
SnapRAID in Unix ignores any encoding. It simply reads and stores the
file names with the same encoding used by the filesystem.

View file

@ -54,7 +54,7 @@ exclude /lost+found/
# The SIZE argument is specified in gibi bytes -> 1073741824 bytes
# Default value is 0, meaning disabled.
# Format: "autosave SIZE_IN_GiB"
#autosize 500
#autosave 500
# Defines the pooling directory where the virtual view of the disk
# array is created using the "pool" command (uncomment to enable).

View file

@ -55,7 +55,7 @@ exclude \System Volume Information
# The SIZE argument is specified in gibi bytes -> 1073741824 bytes
# Default value is 0, meaning disabled.
# Format: "autosave SIZE_IN_GiB"
#autosize 500
#autosave 500
# Defines the pooling directory where the virtual view of the disk
# array is created using the "pool" command (uncomment to enable).

View file

@ -9,8 +9,8 @@ Synopsis
: [-p, --percentage PERC] [-o, --older-than DAYS]
: [-Z, --force-zero] [-E, --force-empty]
: [-U, --force-uuid] [-D, --force-device]
: [-s, --start BLKSTART] [-t, --count BLKCOUNT]
: [-v, --verbose] [-l, --log FILE]
: [-s, --start BLKSTART] [-t, --count BLKCOUNT]
: sync|status|scrub|diff|dup|pool|check|fix|rehash
:snapraid [-V, --version] [-h, --help]
@ -184,16 +184,17 @@ Getting Started
:snapraid scrub
As difference than "check" this command verifies only the oldest data
This command is similar at "check" but verifies only the oldest data
in your array. Every run of the command checks about 12% of the data,
but nothing newer than 10 days.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days:
and the -o, --older-than option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
:snapraid -p 5 -o 20 scrub
If silent errors are found, the corresponding blocks are marked as bad
in the "concent" file, and listed in the "status" command.
in the "content" file, and listed in the "status" command.
:snapraid status
@ -202,8 +203,8 @@ Getting Started
:snapraid -e fix
and at the next "scrub" the errors will disappear from the "status"
report. You can use -p 0 to scrub only them.
At the next "scrub" the errors will disappear from the "status" report
if really fixed. You can use -p 0 to scrub only blocks marked as bad.
:snapraid -p 0 scrub
@ -239,9 +240,12 @@ Commands
in the disk array are read, and the redundancy data is
recomputed.
Files are identified by inode and checked by time and size,
meaning that you can move them on the disk without triggering
any redundancy recomputation.
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the redundancy data is
recomputed for the whole file.
Using inode allow you can move them on the disk
without triggering any redundancy recomputation.
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
@ -288,7 +292,7 @@ Commands
at least one time every two months.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age.
and the -o, --older-than option to specify a different age in days.
Note that if only one of -p and -o is specified the default value of
the other option is not used.
@ -351,7 +355,10 @@ Commands
This option can be used to change the hash kind used,
typically when upgrading from a 32 bits system to a 64
bits one to switch from murmur3 to the faster spooky2.
bits one to switch from MurmurHash3 to the faster SpookyHash.
If you are already using the optimal hash, this command
do nothing and just inform you that nothing has to be done.
The rehash isn't done immediately, but it takes place
progressively during the "sync" and "scrub" commands.
@ -469,7 +476,7 @@ Options
-U, --force-uuid
Forces the insecure operation of syncing, checking and fixing
with disks that have changed the UUID.
with disks that have changed their UUID.
If SnapRAID detects that some disks have changed UUID,
it stops proceeding unless you specify this option.
This allows to detect when your disks are mounted in the
@ -485,18 +492,9 @@ Options
physical device.
If SnapRAID detects that two disks have the same device ID,
it stops proceeding, because it's not a supported configuration.
But it could happen what you want to temporarely restore a lost
But it could happen that you want to temporarely restore a lost
disk in the free space left in an already used disk. and this
option will allow you to continue anyway.
-s, --start BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
-t, --count BLKCOUNT
Process only the specified number of blocks.
It's present mainly for advanced manual recovering.
option allows you to continue anyway.
-l, --log FILE
Outputs a detailed log to the specified file.
@ -505,6 +503,16 @@ Options
If this option is not specified, the log is printed in
the console using the standard error stream.
-s, --start BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
It's present mainly for advanced manual recovering.
-t, --count BLKCOUNT
Processes only the specified number of blocks.
It's present mainly for advanced manual recovering.
-v, --verbose
Prints more information in the processing.
@ -833,21 +841,9 @@ Parity
They are binary files, containing the computed redundancy of all the
blocks defined in the "content" file.
You do not need to understand its format, but it's described here
for documentation.
These files are read and written by the "sync" and "fix" commands, and
only read by "check".
For all the blocks at a given position, the parity and the q-parity
are computed as specified in:
:http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
When a file block is shorter than the default block size, for example
because it's the last block of a file, it's assumed as filled with 0
at the end.
Encoding
SnapRAID in Unix ignores any encoding. It simply reads and stores the
file names with the same encoding used by the filesystem.

View file

@ -13,8 +13,8 @@ snapraid [-c, --conf CONFIG]
[-p, --percentage PERC] [-o, --older-than DAYS]
[-Z, --force-zero] [-E, --force-empty]
[-U, --force-uuid] [-D, --force-device]
[-s, --start BLKSTART] [-t, --count BLKCOUNT]
[-v, --verbose] [-l, --log FILE]
[-s, --start BLKSTART] [-t, --count BLKCOUNT]
sync|status|scrub|diff|dup|pool|check|fix|rehash
snapraid [-V, --version] [-h, --help]
@ -201,16 +201,17 @@ command.
snapraid scrub
As difference than "check" this command verifies only the oldest data
This command is similar at "check" but verifies only the oldest data
in your array. Every run of the command checks about 12% of the data,
but nothing newer than 10 days.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age in days:
and the -o, --older-than option to specify a different age in days.
For example, to check 5% of the array older than 20 days use:
snapraid -p 5 -o 20 scrub
If silent errors are found, the corresponding blocks are marked as bad
in the "concent" file, and listed in the "status" command.
in the "content" file, and listed in the "status" command.
snapraid status
@ -219,8 +220,8 @@ containing bad blocks:
snapraid -e fix
and at the next "scrub" the errors will disappear from the "status"
report. You can use -p 0 to scrub only them.
At the next "scrub" the errors will disappear from the "status" report
if really fixed. You can use -p 0 to scrub only blocks marked as bad.
snapraid -p 0 scrub
@ -263,9 +264,12 @@ Updates the redundancy information. All the modified files
in the disk array are read, and the redundancy data is
recomputed.
Files are identified by inode and checked by time and size,
meaning that you can move them on the disk without triggering
any redundancy recomputation.
Files are identified by path and/or inode and checked by
size and timestamp.
If the size or timestamp are different, the redundancy data is
recomputed for the whole file.
Using inode allow you can move them on the disk
without triggering any redundancy recomputation.
You can stop this process at any time pressing Ctrl+C,
without losing the work already done.
@ -318,7 +322,7 @@ This means that scrubbing once a week, every bit of data is checked
at least one time every two months.
You can use the -p, --percentage option to specify a different amount,
and the -o, --older-than option to specify a different age.
and the -o, --older-than option to specify a different age in days.
Note that if only one of -p and -o is specified the default value of
the other option is not used.
@ -391,7 +395,10 @@ Schedules a rehash of the whole array.
This option can be used to change the hash kind used,
typically when upgrading from a 32 bits system to a 64
bits one to switch from murmur3 to the faster spooky2.
bits one to switch from MurmurHash3 to the faster SpookyHash.
If you are already using the optimal hash, this command
do nothing and just inform you that nothing has to be done.
The rehash isn't done immediately, but it takes place
progressively during the "sync" and "scrub" commands.
@ -512,7 +519,7 @@ SnapRAID provides the following options:
-U, --force-uuid
Forces the insecure operation of syncing, checking and fixing
with disks that have changed the UUID.
with disks that have changed their UUID.
If SnapRAID detects that some disks have changed UUID,
it stops proceeding unless you specify this option.
This allows to detect when your disks are mounted in the
@ -528,18 +535,9 @@ SnapRAID provides the following options:
physical device.
If SnapRAID detects that two disks have the same device ID,
it stops proceeding, because it's not a supported configuration.
But it could happen what you want to temporarely restore a lost
But it could happen that you want to temporarely restore a lost
disk in the free space left in an already used disk. and this
option will allow you to continue anyway.
-s, --start BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
-t, --count BLKCOUNT
Process only the specified number of blocks.
It's present mainly for advanced manual recovering.
option allows you to continue anyway.
-l, --log FILE
Outputs a detailed log to the specified file.
@ -548,6 +546,16 @@ SnapRAID provides the following options:
If this option is not specified, the log is printed in
the console using the standard error stream.
-s, --start BLKSTART
Starts the processing from the specified
block number. It could be useful to retry to check
or fix some specific block, in case of a damaged disk.
It's present mainly for advanced manual recovering.
-t, --count BLKCOUNT
Processes only the specified number of blocks.
It's present mainly for advanced manual recovering.
-v, --verbose
Prints more information in the processing.
@ -915,21 +923,9 @@ and q-parity files.
They are binary files, containing the computed redundancy of all the
blocks defined in the "content" file.
You do not need to understand its format, but it's described here
for documentation.
These files are read and written by the "sync" and "fix" commands, and
only read by "check".
For all the blocks at a given position, the parity and the q-parity
are computed as specified in:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
When a file block is shorter than the default block size, for example
because it's the last block of a file, it's assumed as filled with 0
at the end.
12 ENCODING
===========

13
state.c
View file

@ -153,9 +153,22 @@ static void state_config_check(struct snapraid_state* state, const char* path)
}
}
<<<<<<< HEAD
for(l=0;l<state->level;++l) {
if (disk->device == state->parity_device[l]) {
fprintf(stderr, "Disk '%s' and %s '%s' are on the same device.\n", disk->dir, lev_name(l), state->parity_path[l]);
=======
if (disk->device == state->parity_device) {
fprintf(stderr, "Disk '%s' and parity '%s' are on the same device.\n", disk->dir, state->parity);
#ifdef _WIN32
fprintf(stderr, "Both have the serial number '%"PRIx64"'.\n", disk->device);
#endif
exit(EXIT_FAILURE);
}
if (state->qarity[0] != 0 && disk->device == state->qarity_device) {
fprintf(stderr, "Disk '%s' and q-parity '%s' are on the same device.\n", disk->dir, state->qarity);
>>>>>>> master
#ifdef _WIN32
fprintf(stderr, "Both have the serial number '%"PRIx64"'.\n", disk->device);
#endif

View file

@ -55,13 +55,14 @@ int state_status(struct snapraid_state* state)
unsigned file_fragmented;
unsigned extra_fragment;
uint64_t file_size;
unsigned unsynched_blocks;
/* get the present time */
now = time(0);
blockmax = parity_size(state);
/* for each disk, recover empty files, symlinks and empty dirs */
/* count fragments */
file_count = 0;
file_size = 0;
file_fragmented = 0;
@ -114,10 +115,33 @@ int state_status(struct snapraid_state* state)
printf("\n");
/* allocate a temp vector for all the blocks */
infomap = malloc_nofail(blockmax * sizeof(snapraid_info));
/* count unsynched blocks */
unsynched_blocks = 0;
for(i=0;i<blockmax;++i) {
int one_invalid;
int one_valid;
/* copy the info in the temp vector */
/* for each disk */
one_invalid = 0;
one_valid = 0;
for(node_disk=state->disklist;node_disk!=0;node_disk=node_disk->next) {
struct snapraid_disk* disk = node_disk->data;
struct snapraid_block* block = disk_block_get(disk, i);
if (block_has_file(block))
one_valid = 1;
if (block_has_invalid_parity(block))
one_invalid = 1;
}
/* if both valid and invalid, we need to update */
if (one_invalid && one_valid) {
++unsynched_blocks;
}
}
/* copy the info a temp vector, and count bad/rehash blocks */
infomap = malloc_nofail(blockmax * sizeof(snapraid_info));
bad = 0;
count = 0;
rehash = 0;
@ -138,23 +162,21 @@ int state_status(struct snapraid_state* state)
}
if (!count) {
fprintf(stderr, "The array appears to be empty.\n");
fprintf(stderr, "The array is empty.\n");
free(infomap);
return 0;
}
if (rehash) {
printf("You have a rehash in progress at %u%%.\n", (count - rehash) * 100 / count);
printf("\n");
}
/* sort it */
/* sort the info to get the time info */
qsort(infomap, count, sizeof(snapraid_info), info_time_compare);
oldest = info_get_time(infomap[0]);
median = info_get_time(infomap[count / 2]);
newest = info_get_time(infomap[count - 1]);
dayoldest = day_ago(oldest, now);
daymedian = day_ago(median, now);
daynewest = day_ago(newest, now);
/* compute graph limits */
barpos = 0;
barmax = 0;
for(i=0;i<GRAPH_COLUMN;++i) {
@ -175,14 +197,6 @@ int state_status(struct snapraid_state* state)
bar[i] = step;
}
dayoldest = day_ago(oldest, now);
daymedian = day_ago(median, now);
daynewest = day_ago(newest, now);
printf("The oldest block was checked %u days ago, the median %u, the newest %u.\n", dayoldest, daymedian, daynewest);
printf("\n");
/* print the graph */
for(y=0;y<GRAPH_ROW;++y) {
if (y == 0)
@ -221,8 +235,25 @@ int state_status(struct snapraid_state* state)
printf("\n");
printf("The oldest block was checked %u days ago, the median %u, the newest %u.\n", dayoldest, daymedian, daynewest);
printf("\n");
if (unsynched_blocks) {
printf("WARNING! The array is NOT fully synched.\n");
printf("You have a sync in progress at %u%%.\n", (blockmax - unsynched_blocks) * 100 / blockmax);
} else {
printf("No sync is in progress.\n");
}
if (rehash) {
printf("You have a rehash in progress at %u%%.\n", (count - rehash) * 100 / count);
} else {
printf("No rehash is in progress.\n");
}
if (bad) {
printf("DANGER! In the array there are %u failing blocks!\n\n", bad);
printf("DANGER! In the array there are %u silent errors!\n\n", bad);
printf("They are:");
@ -243,7 +274,7 @@ int state_status(struct snapraid_state* state)
printf("To fix them use the command 'snapraid --filter-error fix'.\n");
printf("The errors will disapper from the 'status' at the next 'scrub' command.\n");
} else {
printf("No silent error detected!\n");
printf("No silent error detected.\n");
}
/* free the temp vector */