diff --git a/TODO b/TODO index 375c7edb..6961e525 100644 --- a/TODO +++ b/TODO @@ -17,17 +17,16 @@ The content file is instead missing in all platforms. - Minor * Check how to reduce memory occupation using smaller memory block. -We can theoreticallty reduce the memory occupation from 28/36 to -17 bytes. 16 for the hash and 1 for the type. -We can store the parity_position<->file,pos relation using some form of binary tree, +We can reduce the memory occupation from 28 (32bit)/36 (64bit) to 17 bytes. +16 for the hash and 1 for the type. + ++ We can store the parity_position<->file,pos relation using some form of binary tree, and then drop completely blockarr, and the extra data in memory_block. -A simpler simplification is to convert, blockarr to a parity_position->file,pos -relation, removing the file pointer inside the block. struct node { uint32 parity_pos /* parity position */ - file* file; /* pointer at the file */ - uint32 offset; /* position in the file */ + file* file_ptr; /* pointer at the file */ + uint32 file_pos; /* position in the file */ uint32 count; /* number of sequential blocks */ void* left_by_pp; /* tree by parity_pos */ void* right_by_pp; @@ -35,6 +34,28 @@ struct node { void* right_by_fp; }; +4*8*number_of_fragments = 32*30000 = 960 KB +but we have to add the extra allocation for each small block, so it could be +(4*8+32)**number_of_fragments = 64*30000 = 1920 KB = 1.9 MB +Low memory occupation. +We can likely make a fast insertion at the end when loading, putting nodes in a list, +and later insert them in the tree. + ++ Or instead of a tree we can use two vectors with binary search. + +parity_pos -> file,pos +[parity_pos,file_ptr,file_pos] 4+4+4 + +file,pos -> parity_pos +[file_ptr,file_pos,parity_pos] 4+4+4 + +(3*4*2)*number_of_fragments=24*30000 = 720 KB = 0.7 MB +Less memory occupation, but slow to change in the middle, like removing and adding files. +Very fast to load from disk as we insert only at the end. + ++ We can also get rid of deleted blocks, as the hash will be stored anyway. ++ We can put the state into the hash, dropping 3 bits of the hash of the 128 ones. + * Restore ownership and permissions, at least in Unix. * Restore directory timestamps. @@ -96,6 +117,17 @@ rest of the block. The problem is that we don't have any hash to ensure that the block is partially recovered, or completely garbage. But it makes sense to anyway write the "most likely" correct one. +* How to handle corrupt copied/moved files ? At now pre-hash checks them, +but if a corruption is found, it stops the sync process. +This is not optimal, as that blocks sync until a manual intervention. +This is the reason why the check of REP (copied) blocks is not enabled by default, +but only inside pre-hash. +A possible solution could be: +If a copied block is corrupt, we can recover the original one from parity (if moved), +or read it directly from disk (if copied), and we can use this one to build the parity, +working around the corruption, that can be later fixed with "fix". +This requires to process first copied blocks, with a possible complex dependency chain. + - Pooling * Add a new "commit" command to move changes from the pool to the array.