Review manpage about memory size

TODO

@@ -5,6 +5,25 @@ This is the list of TODO items for SnapRAID.
 
 - Minor
 
+* Check how to reduce memory occupation using smaller memory block.
+We can theoreticallty reduce the memory occupation from 28/36 to
+17 bytes. 16 for the hash and 1 for the type.
+We can store the parity_position<->file,pos relation using some form of binary tree,
+and then drop completely blockarr, and the extra data in memory_block.
+A simpler simplification is to convert, blockarr to a parity_position->file,pos
+relation, removing the file pointer inside the block.
+
+struct node {
+	uint32 parity_pos /* parity position */
+	file* file; /* pointer at the file */
+	uint32 offset; /* position in the file */
+	uint32 count; /* number of sequential blocks */
+	void* left_by_pp; /* tree by parity_pos */
+	void* right_by_pp;
+	void* left_by_fp; /* tree by file,offset */
+	void* right_by_fp;
+};
+
 * Restore ownership and permissions, at least in Unix.
 
 * Restore directory timestamps.

cmdline/snapraid.c

@@ -77,7 +77,15 @@ void usage(void)
 void memory(void)
 {
 	msg_tag("memory:used:%" PRIu64 "\n", (uint64_t)malloc_counter());
+
+	/* size of the block + the size of the blockarr pointer to it */
+	msg_tag("memory:block:%" PRIu64 "\n", (uint64_t)(sizeof(struct snapraid_block) + sizeof(void*)));
+
+	/* size of the file + the size of the 3 hashtables pointers to it, * 2 for the hashtable grow factor */
+	msg_tag("memory:file:%" PRIu64 "\n", (uint64_t)(sizeof(struct snapraid_file) + 3*2*sizeof(void*)));
+
 	msg_progress("Using %u MiB of memory.\n", (unsigned)(malloc_counter() / 1024 / 1024));
+
 }
 
 /****************************************************************************/

snapraid.1

@@ -1169,7 +1169,8 @@ than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory
 use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 .PP
 In more details SnapRAID requires about TS*28/BS bytes
-of RAM memory to run. Where TS is the total size in bytes of
+of RAM memory to run in the 32 bits version, and TS*36/BS
+in the 64 bits one. Where TS is the total size in bytes of
 your disk array, and BS is the block size in bytes.
 .PP
 For example with 4 disk of 3 TiB and a block size of 256 KiB
@@ -1183,11 +1184,11 @@ RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
 Another reason to use a different blocksize is if you have a lot of small
 files. In the order of many millions.
 .PP
-In details, for each file, even of few bytes, a whole block of parity is
-always allocated, and with many files this may result in a lot of unused
-parity space. And when you completely fill the parity disk, you are not
+For each file, even of few bytes, a whole block of parity is allocated,
+and with many files this may result in a lot of unused parity space.
+And when you completely fill the parity disk, you are not
 allowed to add more files in the data disks.
-Note that wasted parity doesn\'t sum between data disk. Wasted space
+Anywa, the wasted parity doesn\'t sum between data disk. Wasted space
 resulting from a high number of files in a data disk, limits only
 the amount of data in such data disk and not in others.
 .PP
@@ -1200,9 +1201,9 @@ You can get the amount of wasted space in each disk using \[dq]status\[dq].
 This is the amount of space that you must leave free in the data
 disks, or use for files not included in the array.
 If this value is negative, it means that your are near to fill
-the parity, and it represent the space you can still waste.
+the parity, and it represents the space you can still waste.
 .PP
-To avoid to problem, you can use a bigger partition for parity.
+To avoid the problem, you can use a bigger partition for parity.
 For example, if you have the parity partition bigger than 12 GiB
 than data disks, you have enough extra space to handle up to 100000
 files in each data disk.

snapraid.d

@@ -913,7 +913,8 @@ Configuration
 	use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 
 	In more details SnapRAID requires about TS*28/BS bytes
-	of RAM memory to run. Where TS is the total size in bytes of
+	of RAM memory to run in the 32 bits version, and TS*36/BS
+	in the 64 bits one. Where TS is the total size in bytes of
 	your disk array, and BS is the block size in bytes.
 
 	For example with 4 disk of 3 TiB and a block size of 256 KiB
@@ -924,11 +925,11 @@ Configuration
 	Another reason to use a different blocksize is if you have a lot of small
 	files. In the order of many millions.
 
-	In details, for each file, even of few bytes, a whole block of parity is
-	always allocated, and with many files this may result in a lot of unused
-	parity space. And when you completely fill the parity disk, you are not
+	For each file, even of few bytes, a whole block of parity is allocated,
+	and with many files this may result in a lot of unused parity space.
+	And when you completely fill the parity disk, you are not
 	allowed to add more files in the data disks.
-	Note that wasted parity doesn't sum between data disk. Wasted space
+	Anywa, the wasted parity doesn't sum between data disk. Wasted space
 	resulting from a high number of files in a data disk, limits only
 	the amount of data in such data disk and not in others.
 
@@ -941,9 +942,9 @@ Configuration
 	This is the amount of space that you must leave free in the data
 	disks, or use for files not included in the array.
 	If this value is negative, it means that your are near to fill
-	the parity, and it represent the space you can still waste.
+	the parity, and it represents the space you can still waste.
 
-	To avoid to problem, you can use a bigger partition for parity.
+	To avoid the problem, you can use a bigger partition for parity.
 	For example, if you have the parity partition bigger than 12 GiB
 	than data disks, you have enough extra space to handle up to 100000
 	files in each data disk.

snapraid.txt

@@ -989,7 +989,8 @@ than 4 GiB of memory. As a rule of thumb, with 4 GiB or more memory
 use the default 256, with 2 GiB use 512, and with 1 GiB use 1024.
 
 In more details SnapRAID requires about TS*28/BS bytes
-of RAM memory to run. Where TS is the total size in bytes of
+of RAM memory to run in the 32 bits version, and TS*36/BS
+in the 64 bits one. Where TS is the total size in bytes of
 your disk array, and BS is the block size in bytes.
 
 For example with 4 disk of 3 TiB and a block size of 256 KiB
@@ -1000,11 +1001,11 @@ RAM = (4 * 3 * 2^40) * 28 / (256 * 2^10) = 1.4 GiB
 Another reason to use a different blocksize is if you have a lot of small
 files. In the order of many millions.
 
-In details, for each file, even of few bytes, a whole block of parity is
-always allocated, and with many files this may result in a lot of unused
-parity space. And when you completely fill the parity disk, you are not
+For each file, even of few bytes, a whole block of parity is allocated,
+and with many files this may result in a lot of unused parity space.
+And when you completely fill the parity disk, you are not
 allowed to add more files in the data disks.
-Note that wasted parity doesn't sum between data disk. Wasted space
+Anywa, the wasted parity doesn't sum between data disk. Wasted space
 resulting from a high number of files in a data disk, limits only
 the amount of data in such data disk and not in others.
 
@@ -1017,9 +1018,9 @@ You can get the amount of wasted space in each disk using "status".
 This is the amount of space that you must leave free in the data
 disks, or use for files not included in the array.
 If this value is negative, it means that your are near to fill
-the parity, and it represent the space you can still waste.
+the parity, and it represents the space you can still waste.
 
-To avoid to problem, you can use a bigger partition for parity.
+To avoid the problem, you can use a bigger partition for parity.
 For example, if you have the parity partition bigger than 12 GiB
 than data disks, you have enough extra space to handle up to 100000
 files in each data disk.