Update manual and history for the new memory usage

HISTORY

@@ -14,6 +14,9 @@ SnapRAID HISTORY
    parity during sync is really correct.
    The "status" command reports the amount of scrub necessary to
    ensure to scrub all the synced blocks.
+ * Reduced the RAM memory usage, expecially with big files.
+   For each block, it now uses 17 bytes, instead of 28 bytes
+   (for 32 bit) and 36 bytes (for 64 bit).
  * The -p, --percentage option supports the new the arguments:
    "bad", "sync" and "full" to define the kind of scrub to do.
  * Extends the -h, --pre-hash option to also verify moved and

TODO

@@ -16,58 +16,6 @@ The content file is instead missing in all platforms.
 
 - Minor
 
-* Check how to reduce memory occupation using smaller memory block.
-We can reduce the memory occupation from 28 (32bit)/36 (64bit) to 17 bytes.
-16 for the hash and 1 for the type.
-
-+ We can store the parity_position<->file,pos relation using some form of binary tree,
-and then drop completely blockarr, and the extra data in memory_block.
-
-struct node {
-	uint32 parity_pos /* parity position */
-	file* file_ptr; /* pointer at the file */
-	uint32 file_pos; /* position in the file */
-	uint32 count; /* number of sequential blocks */
-	void* left_by_pp; /* tree by parity_pos */
-	void* right_by_pp;
-	void* left_by_fp; /* tree by file,offset */
-	void* right_by_fp;
-};
-
-4*8*number_of_fragments = 32*30000 = 960 KB
-but we have to add the extra allocation for each small block, so it could be
-(4*8+32)**number_of_fragments = 64*30000 = 1920 KB = 1.9 MB
-Low memory occupation.
-We can likely make a fast insertion at the end when loading, putting nodes in a list,
-and later insert them in the tree.
-
-+ Or instead of a tree we can use two vectors with binary search.
-
-parity_pos -> file,pos
-[parity_pos,file_ptr,file_pos] 4+4+4
-
-file,pos -> parity_pos
-[file_ptr,file_pos,parity_pos] 4+4+4
-
-(3*4*2)*number_of_fragments=24*30000 = 720 KB = 0.7 MB
-Less memory occupation, but slow to change in the middle, like removing and adding files.
-Very fast to load from disk as we insert only at the end.
-
-+ We can also get rid of deleted blocks, as the hash will be stored anyway.
-+ We can put the state into the hash, dropping 3 bits of the hash of the 128 ones.
-
-- A problem is that we are going to move hash from file to parity.
-How to keep the hash of the file when the file is not in the parity ?
-We'll have to allocate hash also in the file, and move data.
-We can keep the hash ALWAYS on the file!
-But how to handle DELETED blocks ? At now we handle them one-by-one,
-We have to change to handle them like files, to allow to have DELETED files, inside the tree.
-Here the problem is how to handle when DELETED blocks are overwritten.
-Note that we need the UNDELETE state to allow to recover in FIX.
-The HASH is also used to search for DELETED blocks.
-
-TODO CONVERT DELETED BLOCKS TO DELETE FILES
-
 * Restore ownership and permissions, at least in Unix.
 
 * Restore directory timestamps.

snapraid.1

@@ -1268,15 +1268,14 @@ A reason to use a different blocksize is if your system has less
 than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory
 use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 .PP
-In more details SnapRAID requires about TS*28/BS bytes
-of RAM memory to run in the 32 bits version, and TS*36/BS
-in the 64 bits one. Where TS is the total size in bytes of
-your disk array, and BS is the block size in bytes.
+In more details SnapRAID requires about TS*17/BS bytes of RAM memory.
+Where TS is the total size in bytes of your disk array, and BS is the
+block size in bytes.
 .PP
-For example with 8 disk of 4 TB and a block size of 256 KiB
+For example with 8 disks of 4 TB and a block size of 256 KiB
 (1 KiB = 1024 bytes) you have:
 .PP
-RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB
+RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB
 .PD 0
 .PP
 .PD

snapraid.d

@@ -985,15 +985,14 @@ Configuration
 	than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory
 	use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 
-	In more details SnapRAID requires about TS*28/BS bytes
-	of RAM memory to run in the 32 bits version, and TS*36/BS
-	in the 64 bits one. Where TS is the total size in bytes of
-	your disk array, and BS is the block size in bytes.
+	In more details SnapRAID requires about TS*17/BS bytes of RAM memory.
+	Where TS is the total size in bytes of your disk array, and BS is the
+	block size in bytes.
 
-	For example with 8 disk of 4 TB and a block size of 256 KiB
+	For example with 8 disks of 4 TB and a block size of 256 KiB
 	(1 KiB = 1024 bytes) you have:
 
-	:RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB
+	:RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB
 
 	Another reason to use a different blocksize is if you have a lot of
 	small files. In the order of many millions.

snapraid.txt

@@ -1063,15 +1063,14 @@ A reason to use a different blocksize is if your system has less
 than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory
 use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 
-In more details SnapRAID requires about TS*28/BS bytes
-of RAM memory to run in the 32 bits version, and TS*36/BS
-in the 64 bits one. Where TS is the total size in bytes of
-your disk array, and BS is the block size in bytes.
+In more details SnapRAID requires about TS*17/BS bytes of RAM memory.
+Where TS is the total size in bytes of your disk array, and BS is the
+block size in bytes.
 
-For example with 8 disk of 4 TB and a block size of 256 KiB
+For example with 8 disks of 4 TB and a block size of 256 KiB
 (1 KiB = 1024 bytes) you have:
 
-RAM = (8 * 4 * 10^12) * 28 / (256 * 2^10) = 3.2 GiB
+RAM = (8 * 4 * 10^12) * 17 / (256 * 2^10) = 1.9 GiB
 
 Another reason to use a different blocksize is if you have a lot of
 small files. In the order of many millions.