mirror of
https://github.com/amadvance/snapraid.git
synced 2026-10-26 02:49:02 -04:00
163 lines
6.9 KiB
Text
163 lines
6.9 KiB
Text
SnapRAID TODO
|
|
=============
|
|
|
|
This is the list of TODO items for SnapRAID.
|
|
|
|
- Next
|
|
|
|
* In the log file and gui mode, output kind of final report. Like:
|
|
status:ok / status:fail
|
|
|
|
* In sync if a data error is found, we can mark the block as bad, like in scrub
|
|
and allow sync to continue.
|
|
|
|
* Detect files moved from one disk to another. We won't be able to reuse
|
|
parity, but we can reuse the hash to check that the file was moved
|
|
correctly.
|
|
|
|
* Add a way to list the content of the array. A new "list" command ?
|
|
Something similar at "diff" and "dup".
|
|
|
|
- Minor
|
|
|
|
* Enable storing of creation time NTFS, crtime/birth time EXT4.
|
|
But see: http://unix.stackexchange.com/questions/50177/birth-is-empty-on-ext4
|
|
coreutils stat has an example, but it doesn't work in Linux (see lib/stat-time.h)
|
|
|
|
* A new 'init' command to differentiate the first 'sync' operation.
|
|
This 'init' will work also without a content file, and parity files.
|
|
Instead 'sync' will require all of them.
|
|
This will also help when running with the parity filesystem unmounted.
|
|
|
|
* Adds a log configuration option to save log files with the date in the name.
|
|
This could help to keep track of what happens during automated operations.
|
|
|
|
* In fix an existing symlink with the same name of a file to be recovered may stop
|
|
the process making the create() operation to fail.
|
|
The same for directories, when recreating the directory tree.
|
|
|
|
* If a directory exists with the same name of a parity/content file be more explicative
|
|
on the error message. See: https://sourceforge.net/projects/snapraid/forums/forum/1677233/topic/4861034
|
|
|
|
* Support interrupted operations with EINTR. For remote filesystems it could be relevant.
|
|
|
|
* Allow to specify more than one disk directories to cover the case of multi partitions.
|
|
Different partitions have duplicate inode. The only way to support this is to
|
|
add also a kind of device_id, increasing the memory required.
|
|
But it should be only few bits for each file. So, it should be manageable.
|
|
|
|
* Rename sync->backup and fix->restore. It seems to me a naming expressing
|
|
better the meaning of the commands. But not yet sure.
|
|
|
|
* We don't try to do partial block recovering. A block is correct or not.
|
|
But if only some bytes, or a sector, is wrong, it should be possible to recover all the
|
|
rest of the block.
|
|
The problem is that we don't have any hash to ensure that the block is partially recovered,
|
|
or completely garbage. But it makes sense to anyway write the "most likely" correct one.
|
|
|
|
* In the repair() function the euristic to detect if we recovered after the sync, can be extended
|
|
to all the previous blocks, because we always proceeed in block order during a sync.
|
|
So, if for a block we can detect that we recovered using updated parity data,
|
|
also for all the previous blocks this is true.
|
|
Anyway, the case where this information could be useful should be present
|
|
only if changes are committed after an aborted sync.
|
|
|
|
- Pooling
|
|
|
|
* Allow to specify the shared path to use to access remote symlink.
|
|
|
|
* Add a new "commit" command to move changes from the pool to the array.
|
|
It should:
|
|
- Move files copied into the pool (that are no links) to the array.
|
|
The files should be moved to the disk that contains most of the files
|
|
in the directory. If no space, try with the disk with less files
|
|
in the directory, and eventually the disk in the array with more free space.
|
|
- Detect renames, and apply them in the array.
|
|
The file will be renamed and moved to the new directory, if changed,
|
|
but kept in the same disk of the array.
|
|
- Detect deletes, and move file in the array to a "/trash/" directory
|
|
of the same disk. For safety no real deletion is done.
|
|
File with the same name will get an extra extension like ".1", ".2".
|
|
|
|
* In the content file save the timestap of the parity files.
|
|
If they do not match, stop the processing.
|
|
This can be done to avoid to use not syncronized parity and content files,
|
|
resulting in wrong data.
|
|
But if the sync process is killed we need a way to resyncronize them.
|
|
Or maybe we should allow parity never than content, but not viceversa.
|
|
|
|
- Major
|
|
|
|
* Use Async IO for Linux (libaio).
|
|
See thread: https://sourceforge.net/p/snapraid/discussion/1677233/thread/a300a10b/
|
|
|
|
* Split the parity in multiple files and allow it to coexist with data
|
|
in the same disk.
|
|
This is possible if the data in the same disk uses parity addresses not
|
|
contained in the parity file present in the same disk.
|
|
|
|
* The data could be compressed before processing, resulting in parity block of fixed size,
|
|
but matching different data block sizes.
|
|
The complexity is that a file blocks will have to be allocated at runtime,
|
|
and you may run out of them in the middle of the processing.
|
|
We need also a way to compress a stream until the compressed data reach the
|
|
block size, but no more, and then start a new block.
|
|
For each block, we'll have also to store. "size_uncompressed", "size_compressed",
|
|
"hash".
|
|
|
|
Rejected TODO
|
|
=============
|
|
|
|
This is a list of rejected TODO items.
|
|
|
|
* When fixing, before overwriting the present file, make a copy of it just in
|
|
case that the original file cannot be completely recovered.
|
|
We can always open files in read-only mode, if a write is required, we close it,
|
|
rename it to with a .bak extension, and rewrite it up to the required size.
|
|
The same for symlink if a file with the same name exist or viceversa.
|
|
- The idea behind this is to avoid to leave untouched a file if we cannot
|
|
restore it completely. But it's debatable what's better in this case.
|
|
Anyway, considering the typical use, it's not really relevant.
|
|
|
|
* In the import list, uses also all the blocks in the array.
|
|
But we must cover the case of bad blocks. Likely we can just check the
|
|
hash after reading, and in case, skip it, and retry with another copy.
|
|
- It will work only for duplicate files. Not really worth do to it.
|
|
|
|
* Save the content file in compressed .gz format to save space.
|
|
- Compression is too slow. Even using the very fast lzo.
|
|
$ time lzop -1 < content > content.lzo
|
|
real 1m23.014s
|
|
user 0m40.822s
|
|
sys 0m3.389s
|
|
|
|
$ time ./gzip -1 < content > content.gz
|
|
real 1m47.463s
|
|
user 1m23.732s
|
|
sys 0m3.290s
|
|
|
|
$ time ./gzip --huffonly < content > contentH.gz
|
|
real 1m51.607s
|
|
user 1m30.032s
|
|
sys 0m3.245s
|
|
|
|
Similar command done with snapraid without compression, and involving also decoding
|
|
and encoding takes less time.
|
|
|
|
$ time ./snapraid --test-skip-device --test-skip-self -v -c ./test.conf test-rewrite
|
|
real 0m59.087s
|
|
user 0m14.164s
|
|
sys 0m4.398s
|
|
|
|
* Recognizes that a file is moved from one disk to another, and if the parity
|
|
data doesn't overlap, do not recompute it.
|
|
- It's going to work only in RAID5 mode and only in special cases.
|
|
|
|
* Implements a multithread sync command.
|
|
- At now it's questionable if it will result in a performance improvment.
|
|
The murmur3 hash, and the RAID5/6 computations are so fast that even a single
|
|
thread should be able to do them.
|
|
Use the "snapraid -T" comment to see the speed.
|
|
Also, all the file operations are already done in background by the OS,
|
|
so no improvement is expect from this side.
|
|
|