diff --git a/TODO b/TODO index 9fdf1f9f..88eb4b30 100644 --- a/TODO +++ b/TODO @@ -5,6 +5,25 @@ This is the list of TODO items for SnapRAID. - Minor +* Check how to reduce memory occupation using smaller memory block. +We can theoreticallty reduce the memory occupation from 28/36 to +17 bytes. 16 for the hash and 1 for the type. +We can store the parity_position<->file,pos relation using some form of binary tree, +and then drop completely blockarr, and the extra data in memory_block. +A simpler simplification is to convert, blockarr to a parity_position->file,pos +relation, removing the file pointer inside the block. + +struct node { + uint32 parity_pos /* parity position */ + file* file; /* pointer at the file */ + uint32 offset; /* position in the file */ + uint32 count; /* number of sequential blocks */ + void* left_by_pp; /* tree by parity_pos */ + void* right_by_pp; + void* left_by_fp; /* tree by file,offset */ + void* right_by_fp; +}; + * Restore ownership and permissions, at least in Unix. * Restore directory timestamps. diff --git a/cmdline/snapraid.c b/cmdline/snapraid.c index f58223bb..441fea6e 100644 --- a/cmdline/snapraid.c +++ b/cmdline/snapraid.c @@ -77,7 +77,15 @@ void usage(void) void memory(void) { msg_tag("memory:used:%" PRIu64 "\n", (uint64_t)malloc_counter()); + + /* size of the block + the size of the blockarr pointer to it */ + msg_tag("memory:block:%" PRIu64 "\n", (uint64_t)(sizeof(struct snapraid_block) + sizeof(void*))); + + /* size of the file + the size of the 3 hashtables pointers to it, * 2 for the hashtable grow factor */ + msg_tag("memory:file:%" PRIu64 "\n", (uint64_t)(sizeof(struct snapraid_file) + 3*2*sizeof(void*))); + msg_progress("Using %u MiB of memory.\n", (unsigned)(malloc_counter() / 1024 / 1024)); + } /****************************************************************************/ diff --git a/snapraid.1 b/snapraid.1 index 8d1a90f4..48e8e564 100644 --- a/snapraid.1 +++ b/snapraid.1 @@ -1169,7 +1169,8 @@ than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. .PP In more details SnapRAID requires about TS*28/BS bytes -of RAM memory to run. Where TS is the total size in bytes of +of RAM memory to run in the 32 bits version, and TS*36/BS +in the 64 bits one. Where TS is the total size in bytes of your disk array, and BS is the block size in bytes. .PP For example with 4 disk of 3 TiB and a block size of 256 KiB @@ -1183,11 +1184,11 @@ RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB Another reason to use a different blocksize is if you have a lot of small files. In the order of many millions. .PP -In details, for each file, even of few bytes, a whole block of parity is -always allocated, and with many files this may result in a lot of unused -parity space. And when you completely fill the parity disk, you are not +For each file, even of few bytes, a whole block of parity is allocated, +and with many files this may result in a lot of unused parity space. +And when you completely fill the parity disk, you are not allowed to add more files in the data disks. -Note that wasted parity doesn\'t sum between data disk. Wasted space +Anywa, the wasted parity doesn\'t sum between data disk. Wasted space resulting from a high number of files in a data disk, limits only the amount of data in such data disk and not in others. .PP @@ -1200,9 +1201,9 @@ You can get the amount of wasted space in each disk using \[dq]status\[dq]. This is the amount of space that you must leave free in the data disks, or use for files not included in the array. If this value is negative, it means that your are near to fill -the parity, and it represent the space you can still waste. +the parity, and it represents the space you can still waste. .PP -To avoid to problem, you can use a bigger partition for parity. +To avoid the problem, you can use a bigger partition for parity. For example, if you have the parity partition bigger than 12 GiB than data disks, you have enough extra space to handle up to 100000 files in each data disk. diff --git a/snapraid.d b/snapraid.d index 6f9ed018..2dbbffde 100644 --- a/snapraid.d +++ b/snapraid.d @@ -913,7 +913,8 @@ Configuration use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. In more details SnapRAID requires about TS*28/BS bytes - of RAM memory to run. Where TS is the total size in bytes of + of RAM memory to run in the 32 bits version, and TS*36/BS + in the 64 bits one. Where TS is the total size in bytes of your disk array, and BS is the block size in bytes. For example with 4 disk of 3 TiB and a block size of 256 KiB @@ -924,11 +925,11 @@ Configuration Another reason to use a different blocksize is if you have a lot of small files. In the order of many millions. - In details, for each file, even of few bytes, a whole block of parity is - always allocated, and with many files this may result in a lot of unused - parity space. And when you completely fill the parity disk, you are not + For each file, even of few bytes, a whole block of parity is allocated, + and with many files this may result in a lot of unused parity space. + And when you completely fill the parity disk, you are not allowed to add more files in the data disks. - Note that wasted parity doesn't sum between data disk. Wasted space + Anywa, the wasted parity doesn't sum between data disk. Wasted space resulting from a high number of files in a data disk, limits only the amount of data in such data disk and not in others. @@ -941,9 +942,9 @@ Configuration This is the amount of space that you must leave free in the data disks, or use for files not included in the array. If this value is negative, it means that your are near to fill - the parity, and it represent the space you can still waste. + the parity, and it represents the space you can still waste. - To avoid to problem, you can use a bigger partition for parity. + To avoid the problem, you can use a bigger partition for parity. For example, if you have the parity partition bigger than 12 GiB than data disks, you have enough extra space to handle up to 100000 files in each data disk. diff --git a/snapraid.txt b/snapraid.txt index d56ca38e..403225a7 100644 --- a/snapraid.txt +++ b/snapraid.txt @@ -989,7 +989,8 @@ than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. In more details SnapRAID requires about TS*28/BS bytes -of RAM memory to run. Where TS is the total size in bytes of +of RAM memory to run in the 32 bits version, and TS*36/BS +in the 64 bits one. Where TS is the total size in bytes of your disk array, and BS is the block size in bytes. For example with 4 disk of 3 TiB and a block size of 256 KiB @@ -1000,11 +1001,11 @@ RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB Another reason to use a different blocksize is if you have a lot of small files. In the order of many millions. -In details, for each file, even of few bytes, a whole block of parity is -always allocated, and with many files this may result in a lot of unused -parity space. And when you completely fill the parity disk, you are not +For each file, even of few bytes, a whole block of parity is allocated, +and with many files this may result in a lot of unused parity space. +And when you completely fill the parity disk, you are not allowed to add more files in the data disks. -Note that wasted parity doesn't sum between data disk. Wasted space +Anywa, the wasted parity doesn't sum between data disk. Wasted space resulting from a high number of files in a data disk, limits only the amount of data in such data disk and not in others. @@ -1017,9 +1018,9 @@ You can get the amount of wasted space in each disk using "status". This is the amount of space that you must leave free in the data disks, or use for files not included in the array. If this value is negative, it means that your are near to fill -the parity, and it represent the space you can still waste. +the parity, and it represents the space you can still waste. -To avoid to problem, you can use a bigger partition for parity. +To avoid the problem, you can use a bigger partition for parity. For example, if you have the parity partition bigger than 12 GiB than data disks, you have enough extra space to handle up to 100000 files in each data disk.