rt/snapraid
1
0
Fork 0
mirror of https://github.com/amadvance/snapraid.git synced 2026-10-26 10:51:54 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

More info about blocksize in the manual.

Browse source
This commit is contained in:
Andrea Mazzoleni 2014-02-19 19:44:15 +01:00
parent 824148dd56
commit e78c34abe2
3 changed files with 104 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

48
snapraid.1
View file

@ -900,11 +900,11 @@ Defines an alternate file and format to store the triple parity.
.PP
This option is an alternative at \(cq3\-parity\(cq mainly intended for
low\-end CPUs like ARM or AMD Phenom, Athlon and Opteron that don\(cqt
support the SSSE3 instructions set, and in such case it may provide
support the SSSE3 instructions set, and in such case it provides
a better performance.
.PP
This format is similar at the one used by the Linux Kernel RAID6 and
ZFS RAIDZ3, but it doesn\(cqt work beyond triple parity.
This format is similar, but faster, at the one used by the ZFS RAIDZ3,
but it doesn\(cqt work beyond triple parity.
.PP
When using \(cq3\-parity\(cq you will be warned if it\(cqs recommended to use
the \(cqz\-parity\(cq format for a performance improvement.
@ -958,6 +958,7 @@ This option can be used many times.
Defines the basic block size in kibi bytes of the parity
blocks. Where one kibi bytes is 1024 bytes.
The default is 256 and it should work for most conditions.
.PP
You could increase this value if you do not have enough RAM
memory to run SnapRAID.
.PP
@ -969,7 +970,7 @@ of RAM memory to run. Where TS is the total size in bytes of
your disk array, and BS is the block size in bytes.
.PP
For example with 4 disk of 3 TiB and a block size of 256 KiB
(1 KiB = 1024 Bytes) you have:
(1 KiB = 1024 bytes) you have:
.PP
RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
.PD 0
@ -978,13 +979,40 @@ RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
.PP
You could instead decrease this value if you have a lot of
small files in the disk array. For each file, even if of few
bytes, a whole block is always allocated, so you may have a lot
of unused space.
As approximation, you can assume that half of the block size is
wasted for each file.
bytes, a whole block of parity is always allocated, so you may
have a lot of unused parity space.
.PP
For example, with 10000 files and a 256 KiB block size, you are
going to waste 1.2 GiB.
The problem of wasted parity, is that when the parity disk is full,
you are not allowed to add more data in the data disks.
Note that wasted parity doesn\(cqt sum between data disk. Wasted space
resulting from a high number of files in a data disk, may limit only
the amount of data in such data disk and not in others.
.PP
As approximation, you can assume that half of the block size is
wasted for each file. For example, with 100000 files and a 256 KiB
block size, you are going to waste 12 GiB of parity.
.PP
To avoid to problem, you can use a bigger partition for parity.
For example, if you have the parity partition bigger than 12 GiB
than data disks, you have enough extra space to handle up to 100000
files in each data disk.
.PP
A trick to get a bigger parity partition in Linux, is to format it
with the command:
.PP
.RS 4
mkfs.ext4 \-m 0 \-T largefile4 DEVICE
.PD 0
.PP
.PD
.RE
.PP
This results in about 1.5% of extra space. Meaning about 48 GiB for
a 3 TiB disk, that allows about 400000 files in each data disk without
any wasted space.
.PP
Note also, that any file in data disks that is excluded from the parity
computation, counts for its size as extra parity space for such data disk.
.SS autosave SIZE_IN_GIBIBYTES
Automatically save the state when synching after the specified amount
of GiB processed.

43
snapraid.d
View file

@ -664,11 +664,11 @@ Configuration
This option is an alternative at '3-parity' mainly intended for
low-end CPUs like ARM or AMD Phenom, Athlon and Opteron that don't
support the SSSE3 instructions set, and in such case it may provide
support the SSSE3 instructions set, and in such case it provides
a better performance.
This format is similar at the one used by the Linux Kernel RAID6 and
ZFS RAIDZ3, but it doesn't work beyond triple parity.
This format is similar, but faster, at the one used by the ZFS RAIDZ3,
but it doesn't work beyond triple parity.
When using '3-parity' you will be warned if it's recommended to use
the 'z-parity' format for a performance improvement.
@ -727,6 +727,7 @@ Configuration
Defines the basic block size in kibi bytes of the parity
blocks. Where one kibi bytes is 1024 bytes.
The default is 256 and it should work for most conditions.
You could increase this value if you do not have enough RAM
memory to run SnapRAID.
@ -738,19 +739,41 @@ Configuration
your disk array, and BS is the block size in bytes.
For example with 4 disk of 3 TiB and a block size of 256 KiB
(1 KiB = 1024 Bytes) you have:
(1 KiB = 1024 bytes) you have:
:RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
You could instead decrease this value if you have a lot of
small files in the disk array. For each file, even if of few
bytes, a whole block is always allocated, so you may have a lot
of unused space.
As approximation, you can assume that half of the block size is
wasted for each file.
bytes, a whole block of parity is always allocated, so you may
have a lot of unused parity space.
For example, with 10000 files and a 256 KiB block size, you are
going to waste 1.2 GiB.
The problem of wasted parity, is that when the parity disk is full,
you are not allowed to add more data in the data disks.
Note that wasted parity doesn't sum between data disk. Wasted space
resulting from a high number of files in a data disk, may limit only
the amount of data in such data disk and not in others.
As approximation, you can assume that half of the block size is
wasted for each file. For example, with 100000 files and a 256 KiB
block size, you are going to waste 12 GiB of parity.
To avoid to problem, you can use a bigger partition for parity.
For example, if you have the parity partition bigger than 12 GiB
than data disks, you have enough extra space to handle up to 100000
files in each data disk.
A trick to get a bigger parity partition in Linux, is to format it
with the command:
:mkfs.ext4 -m 0 -T largefile4 DEVICE
This results in about 1.5% of extra space. Meaning about 48 GiB for
a 3 TiB disk, that allows about 400000 files in each data disk without
any wasted space.
Note also, that any file in data disks that is excluded from the parity
computation, counts for its size as extra parity space for such data disk.
autosave SIZE_IN_GIBIBYTES
Automatically save the state when synching after the specified amount

43
snapraid.txt
View file

@ -718,11 +718,11 @@ Defines an alternate file and format to store the triple parity.
This option is an alternative at '3-parity' mainly intended for
low-end CPUs like ARM or AMD Phenom, Athlon and Opteron that don't
support the SSSE3 instructions set, and in such case it may provide
support the SSSE3 instructions set, and in such case it provides
a better performance.
This format is similar at the one used by the Linux Kernel RAID6 and
ZFS RAIDZ3, but it doesn't work beyond triple parity.
This format is similar, but faster, at the one used by the ZFS RAIDZ3,
but it doesn't work beyond triple parity.
When using '3-parity' you will be warned if it's recommended to use
the 'z-parity' format for a performance improvement.
@ -791,6 +791,7 @@ This option can be used many times.
Defines the basic block size in kibi bytes of the parity
blocks. Where one kibi bytes is 1024 bytes.
The default is 256 and it should work for most conditions.
You could increase this value if you do not have enough RAM
memory to run SnapRAID.
@ -802,19 +803,41 @@ of RAM memory to run. Where TS is the total size in bytes of
your disk array, and BS is the block size in bytes.
For example with 4 disk of 3 TiB and a block size of 256 KiB
(1 KiB = 1024 Bytes) you have:
(1 KiB = 1024 bytes) you have:
RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
You could instead decrease this value if you have a lot of
small files in the disk array. For each file, even if of few
bytes, a whole block is always allocated, so you may have a lot
of unused space.
As approximation, you can assume that half of the block size is
wasted for each file.
bytes, a whole block of parity is always allocated, so you may
have a lot of unused parity space.
For example, with 10000 files and a 256 KiB block size, you are
going to waste 1.2 GiB.
The problem of wasted parity, is that when the parity disk is full,
you are not allowed to add more data in the data disks.
Note that wasted parity doesn't sum between data disk. Wasted space
resulting from a high number of files in a data disk, may limit only
the amount of data in such data disk and not in others.
As approximation, you can assume that half of the block size is
wasted for each file. For example, with 100000 files and a 256 KiB
block size, you are going to waste 12 GiB of parity.
To avoid to problem, you can use a bigger partition for parity.
For example, if you have the parity partition bigger than 12 GiB
than data disks, you have enough extra space to handle up to 100000
files in each data disk.
A trick to get a bigger parity partition in Linux, is to format it
with the command:
mkfs.ext4 -m 0 -T largefile4 DEVICE
This results in about 1.5% of extra space. Meaning about 48 GiB for
a 3 TiB disk, that allows about 400000 files in each data disk without
any wasted space.
Note also, that any file in data disks that is excluded from the parity
computation, counts for its size as extra parity space for such data disk.
---- 7.8.1 autosave SIZE_IN_GIBIBYTES ----
Automatically save the state when synching after the specified amount