From 20043778e47c11251c654bbacefdda1a94726f25 Mon Sep 17 00:00:00 2001 From: Andrea Mazzoleni Date: Sun, 4 Oct 2015 23:11:05 +0200 Subject: [PATCH] Update manual and history for the new memory usage --- HISTORY | 3 +++ TODO | 52 ---------------------------------------------------- snapraid.1 | 11 +++++------ snapraid.d | 11 +++++------ snapraid.txt | 11 +++++------ 5 files changed, 18 insertions(+), 70 deletions(-) diff --git a/HISTORY b/HISTORY index faf61de4..1773139d 100644 --- a/HISTORY +++ b/HISTORY @@ -14,6 +14,9 @@ SnapRAID HISTORY parity during sync is really correct. The "status" command reports the amount of scrub necessary to ensure to scrub all the synced blocks. + * Reduced the RAM memory usage, expecially with big files. + For each block, it now uses 17 bytes, instead of 28 bytes + (for 32 bit) and 36 bytes (for 64 bit). * The -p, --percentage option supports the new the arguments: "bad", "sync" and "full" to define the kind of scrub to do. * Extends the -h, --pre-hash option to also verify moved and diff --git a/TODO b/TODO index 07fdef35..4964e1c9 100644 --- a/TODO +++ b/TODO @@ -16,58 +16,6 @@ The content file is instead missing in all platforms. - Minor -* Check how to reduce memory occupation using smaller memory block. -We can reduce the memory occupation from 28 (32bit)/36 (64bit) to 17 bytes. -16 for the hash and 1 for the type. - -+ We can store the parity_position<->file,pos relation using some form of binary tree, -and then drop completely blockarr, and the extra data in memory_block. - -struct node { - uint32 parity_pos /* parity position */ - file* file_ptr; /* pointer at the file */ - uint32 file_pos; /* position in the file */ - uint32 count; /* number of sequential blocks */ - void* left_by_pp; /* tree by parity_pos */ - void* right_by_pp; - void* left_by_fp; /* tree by file,offset */ - void* right_by_fp; -}; - -4*8*number_of_fragments = 32*30000 = 960 KB -but we have to add the extra allocation for each small block, so it could be -(4*8+32)**number_of_fragments = 64*30000 = 1920 KB = 1.9 MB -Low memory occupation. -We can likely make a fast insertion at the end when loading, putting nodes in a list, -and later insert them in the tree. - -+ Or instead of a tree we can use two vectors with binary search. - -parity_pos -> file,pos -[parity_pos,file_ptr,file_pos] 4+4+4 - -file,pos -> parity_pos -[file_ptr,file_pos,parity_pos] 4+4+4 - -(3*4*2)*number_of_fragments=24*30000 = 720 KB = 0.7 MB -Less memory occupation, but slow to change in the middle, like removing and adding files. -Very fast to load from disk as we insert only at the end. - -+ We can also get rid of deleted blocks, as the hash will be stored anyway. -+ We can put the state into the hash, dropping 3 bits of the hash of the 128 ones. - -- A problem is that we are going to move hash from file to parity. -How to keep the hash of the file when the file is not in the parity ? -We'll have to allocate hash also in the file, and move data. -We can keep the hash ALWAYS on the file! -But how to handle DELETED blocks ? At now we handle them one-by-one, -We have to change to handle them like files, to allow to have DELETED files, inside the tree. -Here the problem is how to handle when DELETED blocks are overwritten. -Note that we need the UNDELETE state to allow to recover in FIX. -The HASH is also used to search for DELETED blocks. - -TODO CONVERT DELETED BLOCKS TO DELETE FILES - * Restore ownership and permissions, at least in Unix. * Restore directory timestamps. diff --git a/snapraid.1 b/snapraid.1 index fb44e22b..21159db0 100644 --- a/snapraid.1 +++ b/snapraid.1 @@ -1268,15 +1268,14 @@ A reason to use a different blocksize is if your system has less than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. .PP -In more details SnapRAID requires about TS*28/BS bytes -of RAM memory to run in the 32 bits version, and TS*36/BS -in the 64 bits one. Where TS is the total size in bytes of -your disk array, and BS is the block size in bytes. +In more details SnapRAID requires about TS*17/BS bytes of RAM memory. +Where TS is the total size in bytes of your disk array, and BS is the +block size in bytes. .PP -For example with 8 disk of 4 TB and a block size of 256 KiB +For example with 8 disks of 4 TB and a block size of 256 KiB (1 KiB = 1024 bytes) you have: .PP -RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB +RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB .PD 0 .PP .PD diff --git a/snapraid.d b/snapraid.d index c656d086..98381241 100644 --- a/snapraid.d +++ b/snapraid.d @@ -985,15 +985,14 @@ Configuration than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. - In more details SnapRAID requires about TS*28/BS bytes - of RAM memory to run in the 32 bits version, and TS*36/BS - in the 64 bits one. Where TS is the total size in bytes of - your disk array, and BS is the block size in bytes. + In more details SnapRAID requires about TS*17/BS bytes of RAM memory. + Where TS is the total size in bytes of your disk array, and BS is the + block size in bytes. - For example with 8 disk of 4 TB and a block size of 256 KiB + For example with 8 disks of 4 TB and a block size of 256 KiB (1 KiB = 1024 bytes) you have: - :RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB + :RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB Another reason to use a different blocksize is if you have a lot of small files. In the order of many millions. diff --git a/snapraid.txt b/snapraid.txt index be6c8544..785cefc3 100644 --- a/snapraid.txt +++ b/snapraid.txt @@ -1063,15 +1063,14 @@ A reason to use a different blocksize is if your system has less than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory use the default 256, with 2 GiB use 512, and with 1 GiB use 1024. -In more details SnapRAID requires about TS*28/BS bytes -of RAM memory to run in the 32 bits version, and TS*36/BS -in the 64 bits one. Where TS is the total size in bytes of -your disk array, and BS is the block size in bytes. +In more details SnapRAID requires about TS*17/BS bytes of RAM memory. +Where TS is the total size in bytes of your disk array, and BS is the +block size in bytes. -For example with 8 disk of 4 TB and a block size of 256 KiB +For example with 8 disks of 4 TB and a block size of 256 KiB (1 KiB = 1024 bytes) you have: -RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB +RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB Another reason to use a different blocksize is if you have a lot of small files. In the order of many millions.