mirror of
https://github.com/amadvance/snapraid.git
synced 2026-10-26 19:01:57 -04:00
Some more notes about the planned changes
This commit is contained in:
parent
49bcad5448
commit
2801634aa4
1 changed files with 39 additions and 7 deletions
46
TODO
46
TODO
|
|
@ -17,17 +17,16 @@ The content file is instead missing in all platforms.
|
|||
- Minor
|
||||
|
||||
* Check how to reduce memory occupation using smaller memory block.
|
||||
We can theoreticallty reduce the memory occupation from 28/36 to
|
||||
17 bytes. 16 for the hash and 1 for the type.
|
||||
We can store the parity_position<->file,pos relation using some form of binary tree,
|
||||
We can reduce the memory occupation from 28 (32bit)/36 (64bit) to 17 bytes.
|
||||
16 for the hash and 1 for the type.
|
||||
|
||||
+ We can store the parity_position<->file,pos relation using some form of binary tree,
|
||||
and then drop completely blockarr, and the extra data in memory_block.
|
||||
A simpler simplification is to convert, blockarr to a parity_position->file,pos
|
||||
relation, removing the file pointer inside the block.
|
||||
|
||||
struct node {
|
||||
uint32 parity_pos /* parity position */
|
||||
file* file; /* pointer at the file */
|
||||
uint32 offset; /* position in the file */
|
||||
file* file_ptr; /* pointer at the file */
|
||||
uint32 file_pos; /* position in the file */
|
||||
uint32 count; /* number of sequential blocks */
|
||||
void* left_by_pp; /* tree by parity_pos */
|
||||
void* right_by_pp;
|
||||
|
|
@ -35,6 +34,28 @@ struct node {
|
|||
void* right_by_fp;
|
||||
};
|
||||
|
||||
4*8*number_of_fragments = 32*30000 = 960 KB
|
||||
but we have to add the extra allocation for each small block, so it could be
|
||||
(4*8+32)**number_of_fragments = 64*30000 = 1920 KB = 1.9 MB
|
||||
Low memory occupation.
|
||||
We can likely make a fast insertion at the end when loading, putting nodes in a list,
|
||||
and later insert them in the tree.
|
||||
|
||||
+ Or instead of a tree we can use two vectors with binary search.
|
||||
|
||||
parity_pos -> file,pos
|
||||
[parity_pos,file_ptr,file_pos] 4+4+4
|
||||
|
||||
file,pos -> parity_pos
|
||||
[file_ptr,file_pos,parity_pos] 4+4+4
|
||||
|
||||
(3*4*2)*number_of_fragments=24*30000 = 720 KB = 0.7 MB
|
||||
Less memory occupation, but slow to change in the middle, like removing and adding files.
|
||||
Very fast to load from disk as we insert only at the end.
|
||||
|
||||
+ We can also get rid of deleted blocks, as the hash will be stored anyway.
|
||||
+ We can put the state into the hash, dropping 3 bits of the hash of the 128 ones.
|
||||
|
||||
* Restore ownership and permissions, at least in Unix.
|
||||
|
||||
* Restore directory timestamps.
|
||||
|
|
@ -96,6 +117,17 @@ rest of the block.
|
|||
The problem is that we don't have any hash to ensure that the block is partially recovered,
|
||||
or completely garbage. But it makes sense to anyway write the "most likely" correct one.
|
||||
|
||||
* How to handle corrupt copied/moved files ? At now pre-hash checks them,
|
||||
but if a corruption is found, it stops the sync process.
|
||||
This is not optimal, as that blocks sync until a manual intervention.
|
||||
This is the reason why the check of REP (copied) blocks is not enabled by default,
|
||||
but only inside pre-hash.
|
||||
A possible solution could be:
|
||||
If a copied block is corrupt, we can recover the original one from parity (if moved),
|
||||
or read it directly from disk (if copied), and we can use this one to build the parity,
|
||||
working around the corruption, that can be later fixed with "fix".
|
||||
This requires to process first copied blocks, with a possible complex dependency chain.
|
||||
|
||||
- Pooling
|
||||
|
||||
* Add a new "commit" command to move changes from the pool to the array.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue