snapraid/TODO

SnapRAID TODO
=============

This is the list of TODO items for SnapRAID.

- v2.2

* Implement an "undelete" command to recover missing files, without the
integrity check of the existing ones.
It should restore only missing files, without recovering modified ones
or check validity of data of existing ones.

* If a content file is missing, sync should rewrite it, even if no change was done at the content.

* A new 'init' command to differentiate the first 'sync' operation.
This 'init' will work also without a content file, and parity files.
Instead 'sync' will require all of them.
This will also help when running with the parity filesystem unmounted.

* Filtered out files should be opened in read-only mode.

* Adds a log option, to save log files with the date in the name. This could
help to keep track of what happens during automated operations.
We must keep the log to stdout/err, and only save a copy to the log file.

* In fix an existing symlink with the same name of a file to be recovered may stop
the process making the create() operation to fail.
The same for directories, when recreating the directory tree.

* If a directory exists with the same name of a parity/content file be more explicative
on the error message. See: https://sourceforge.net/projects/snapraid/forums/forum/1677233/topic/4861034

* Support interrupted operations with EINTR. For remote filesystems it could be relevant.

* In the repair() function the euristic to detect if we recovered after the sync, can be extended
to all the subsequent blocks, because we always proceeed in block order during a sync.
So, if for a block we can detect that we recovered after a sync, also for all the subsequent blocks
this is true.

* In Windows to use FILE_FLAG_BACKUP_SEMANTICS we should also get privileges.
See http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista.

* Support Symlinks also in Windows.
See: http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista

* In Windows we must be case insensitive, at least for ASCII<128 comparison. For other UTF-8 chars
is too complex.

- v3.x

* In the content file, for each block store the last time the hash was checked
with both "check" and also "sync".
It will allow to define a new "check" command, checking only aged blocks
and skipping recently checked ones.
A good name could be "scrub".
This could be problematic for the increased memory occupation.
The check command could be maintaned independent, and not saving the content file.

* Save the content file in compressed .gz format to save space.

* Rename sync->backup and fix->restore. It seems to me a naming expressing
better the meaning of the commands. But not yet sure.

* In the "interim" content file during sync, keep the full list of deleted
files, to be able to restore in case they are put back in their original position.
We use now the "hole/off" fields to represent this information, but we can define
a new "deleted (just like file)/blk,inv,chg,new" keeping the hash.
We must have all the "blk,inv,chg,new" options to be able to address delete file
starting from an interim content.
When recovering, the first try assuming updated parity, will ignore all the "deleted"
entries. The second one, will use them.
Possible problem is when a new file, overwrite part of the deleted ones.
We have to keep in memory different block indexes?
Should collided blocks be allocated in a different way ?
In case, we can lose the hash of the deleted ones, because we need only the
position in the disk. We use them only for recovering, so the block to be
recovered will be checked with its own hash.

* When fixing, before overwriting the present file, make a copy of it just in
case that the original file cannot be completely recovered.
We can always open files in read-only mode, if a write is required, we close it,
rename it to with a .bak extension, and rewrite it up to the required size.
The same for symlink if a file with the same name exist or viceversa.

- v4.x

* Triple parity also called RAID-TP. We can implement it in a compatible way
with the existing RAID-5 and RAID-6 definining a new r-parity file.
See http://blogs.oracle.com/ahl/entry/triple_parity_raid_z and
http://thread.gmane.org/gmane.linux.raid/34195/focus=34236

* In the content file save the timestap of the parity files.
If they do not match, stop the processing.
This can be done to avoid to use not syncronized parity and content files,
resulting in wrong data.
But if the sync process is killed we need a way to resyncronize them.
Or maybe we should allow parity never than content, but not viceversa.


Rejected TODO
=============

This is a list of rejected TODO items.

* Allow to specify more than one disk directories to cover the case of multi partitions.
- Different partitions have duplicate inode. The only way to support this is to
add also a kind of device_id, increasing the memory required.

* Recognizes that a file is moved from one disk to another, and if the parity
data doesn't overlap, do not recompute it.
- It's going to work only in RAID5 mode and only in special cases.

* Implements a multithread sync command.
- At now it's questionable if it will result in a performance improvment.
The murmur3 hash, and the RAID5/6 computations are so fast that even a single
thread should be able to do them.
Use the "snapraid -T" comment to see the speed.
Also, all the file operations are already done in background by the OS,
so no improvement is expect from this side.

* The SSE2 RAID6 computation may be slower than the MMX version.
See for example this case on a AMD Zacate APU E-350 1.6GHz:
memset 1812 [MB/s]
MD5 252 [MB/s]
Murmur3 1116 [MB/s]
RAID5 int32x2 1692 [MB/s]
RAID5 mmxx2 1745 [MB/s]
RAID5 sse2x2 2481 [MB/s]
RAID6 int32x2 807 [MB/s]
RAID6 mmxx2 1304 [MB/s]
RAID6 sse2x2 787 [MB/s]
- It happens in some systems. At now not an high priority issue.