snapraid/TODO

SnapRAID TODO
=============

This is the list of TODO items for SnapRAID.

- Next

* Measures the time waited in blocking syscall for each disk,
and prints stats of the disk speeds when the command finish,
or when it's running.

* In status prints the allocated parity in each data disk.
In case, also takes into account the extra available space
in the parity disks.

* Avoid to do optimized move operation if more files with
same timestamp/size are present.

* Estimate the ETA using the latest minute speed. This likely
requires to keep 60 measures, one for each seconds, to be able
to estimate the speed of the last minute, for each second.

- Minor

* Provides a way to rename disks in content file.

* Checks if splitting hash/parity computation in 4K pages
can improve speed in sync. That should increase cache locality,
because we read the data two times for hash and and parity,
and if we manage to keep it in the cache, we should save time.

* In the content file save the timestamp of the parity files.
If they do not match, stop the processing.
This can be done to avoid to use not synchronized parity and content files,
resulting in wrong data.
But if the sync process is killed we need a way to resyncronize them.
Or maybe we should allow parity never than content, but not viceversa.

* Add an option for dup to list only duplicates with different name.
This supposing that if a file has the same name, it's intentionally duplicated.

* Restore ownership and permission, at least in Unix.

* Restore directory timestamps.

* Enable storing of creation time NTFS, crtime/birth time EXT4.
But see: http://unix.stackexchange.com/questions/50177/birth-is-empty-on-ext4
coreutils stat has an example, but it doesn't work in Linux (see lib/stat-time.h)

* A new 'init' command to differentiate the first 'sync' operation.
This 'init' will work also without a content file, and parity files.
Instead 'sync' will require all of them.
This will also help when running with the parity filesystem unmounted.

* Adds a log configuration option to save log files with the date in the name.
This could help to keep track of what happens during automated operations.

* In fix an existing symlink with the same name of a file to be recovered may stop
the process making the create() operation to fail.
The same for directories, when recreating the directory tree.

* If a directory exists with the same name of a parity/content file be more explicative
on the error message. See: https://sourceforge.net/projects/snapraid/forums/forum/1677233/topic/4861034

* Support interrupted operations with EINTR. For remote filesystems it could be relevant.

* Allow to specify more than one disk directories to cover the case of multi partitions.
Different partitions have duplicate inode. The only way to support this is to
add also a kind of device_id, increasing the memory required.
But it should be only few bits for each file. So, it should be manageable.
A lot of discussions about this feature :)
https://sourceforge.net/p/snapraid/discussion/1677233/thread/b2cd9385/

* Rename sync->backup and fix->restore. It seems to me a naming expressing
better the meaning of the commands. But not yet sure.

* We don't try to do partial block recovering. A block is correct or not.
But if only some bytes, or a sector, is wrong, it should be possible to recover all the
rest of the block.
The problem is that we don't have any hash to ensure that the block is partially recovered,
or completely garbage. But it makes sense to anyway write the "most likely" correct one.

* In the repair() function the heuristic to detect if we recovered after the sync, can be extended
to all the previous blocks, because we always proceed in block order during a sync.
So, if for a block we can detect that we recovered using updated parity data,
also for all the previous blocks this is true.
Anyway, the case where this information could be useful should be present
only if changes are committed after an aborted sync.

- Pooling

* Add a new "commit" command to move changes from the pool to the array.
It should:
- Move files copied into the pool (that are no links) to the array.
The files should be moved to the disk that contains most of the files
in the directory. If no space, try with the disk with less files
in the directory, and eventually the disk in the array with more free space.
- Detect renames, and apply them in the array.
The file will be renamed and moved to the new directory, if changed,
but kept in the same disk of the array.
- Detect deletes, and move file in the array to a "/trash/" directory
of the same disk. For safety no real deletion is done.
File with the same name will get an extra extension like ".1", ".2".

- Major

* Use threads to scan all the disks at the same time.

* Use Async IO for Linux (libaio).
See thread: https://sourceforge.net/p/snapraid/discussion/1677233/thread/a300a10b/

* Create a filesystem snapshot at every sync, and use it in all the other commands
automatically.
At the next sync, drop the old snapshot and create a new one.
This should help recovering, because we'll have the exact copy used by sync.
This feature can be enabled with a specific option, and available
in Windows using Shadow Copy, and in Linux using Btrfs, and in a generic
Unix using ZFS.
See Jens's Windows script at: http://sourceforge.net/p/snapraid/discussion/1677233/thread/a1707211/
Note that a different between Windows and Unix is that in Windows old snapshots
are automatically deleted.

* Use smaller hashes to reduce the memory occupation. We can truncate the 128 bits hash
to 32 or 64 bits and reduce the memory usage by a factor of 4 or 2.
ZFS, and Btrfs supports the 32 bit CRCs. To identify bad blocks it should be enough.
But this likely means to drop the dup and --import feature. We cannot trust a 32 bits hash
as a 128 bits one to search for blocks.
It should be optional, only for architectures with very low memory.

Rejected TODO
=============

This is a list of rejected TODO items.

* Allow to put parity directly into the underline block device without the need
of any filesystem.
That would allow to increase a lot the free space for parity.
We can implement some kind of filesystem detection to avoid to overwrite an already
existing filesystem.
- Still risky if stuffs go wrong.
- In Linux with largefile4 there is only a very small amount of space wasted.
In the order of 0.01%. Not really worth to do it.
- With NTFS the saving is also limited, because the "reserved" MFT of 12.5% is
not really exclusively reserved, but can be used also for normal files,
when all the remaining space is filled.

* Split the parity in multiple files and allow it to coexist with data
in the same disk.
This is possible if the data in the same disk uses parity addresses not
contained in the parity file present in the same disk.
- It could be more complex to setup for the user.
+ It's a nice trick. The disk containing parity, will have less space available,
and then it will need less parity, resolving the problem of the parity disk being too small.
+ We can also think at an automated parity files naming and creation, removing
the need of the user to specify that. We can also completely remove the
concept of parity drive, automatically allocating parity in the most free data drives.
- The problem is that it would be not nice to have the program to automatically
choose where to create the parity, because the choice cannot be optimal.
With a dedicated disk manually chosen, it's instead optimal.

* The data could be compressed before processing, resulting in parity block of
fixed size, but matching different data block sizes.
The complexity is that a file blocks will have to be allocated at runtime,
and you may run out of them in the middle of the processing.
We need also a way to compress a stream until the compressed data reach the
block size, but no more, and then start a new block.
For each block, we'll have also to store. "size_uncompressed", "size_compressed",
"hash".
- It will be too slow.
- Not addressing the problem of a lot of small files, as still one block will be
allocated for each file.

* Uses different block size for parity and file allocation.
We can use a big size, like 1MB for parity allocation and hash computation,
and at the same time using a 4K blocks for files allocation.
This means that a parity blocks may contain more than one file.
- We'll have to drop the dup and import feature,
because it won't be possible anymore to compare the hash of files,
as it would depend also on the start address inside the parity block.
- When a hash fail, it won't be possible to tell which file is really
broken, because more file may share the same parity block.

* Have special parity blocks containing the last blocks of more files
until it's filled up.
For such parity blocks, we'll have more than one hash. One for each
file contained.
- Parity will be too fragmented, because we'll have parity blocks
containing the last blocks of many files.

* In "pool" for Windows, and for unique directories a junction to the
directory could be used, avoiding to use symlinks to files.
This allows to overcome the problem of sharing symlinks.
- It would work, in fact it would work to well. The problem is that
Windows will treat the junction as the real directory, like *really*
deleting its content from Explorer pressing "del" and also from the
command line with "rd". Too dangerous.

* When fixing, before overwriting the present file, make a copy of it just in
case that the original file cannot be completely recovered.
We can always open files in read-only mode, if a write is required, we close it,
rename it to with a .bak extension, and rewrite it up to the required size.
The same for symlink if a file with the same name exist or viceversa.
- The idea behind this is to avoid to leave untouched a file if we cannot
restore it completely. But it's debatable what's better in this case.
Anyway, considering the typical use, it's not really relevant.

* In the import list, uses also all the blocks in the array.
But we must cover the case of bad blocks. Likely we can just check the
hash after reading, and in case, skip it, and retry with another copy.
- It will work only for duplicate files. Not really worth do to it.

* Save the content file in compressed .gz format to save space.
- Compression is too slow. Even using the very fast lzo.
$ time lzop  -1 < content > content.lzo
real    1m23.014s
user    0m40.822s
sys     0m3.389s

$ time ./gzip -1 < content > content.gz
real    1m47.463s
user    1m23.732s
sys     0m3.290s

$ time ./gzip --huffonly < content > contentH.gz
real    1m51.607s
user    1m30.032s
sys     0m3.245s

Similar command done with snapraid without compression, and involving also decoding
and encoding takes less time.

$ time ./snapraid --test-skip-device --test-skip-self -v -c ./test.conf test-rewrite
real    0m59.087s
user    0m14.164s
sys     0m4.398s

* Recognizes that a file is moved from one disk to another, and if the parity
data doesn't overlap, do not recompute it.
- It's going to work only in RAID5 mode and only in special cases.

* Implements a multithread sync command.
- At now it's questionable if it will result in a performance improvement.
The murmur3 hash, and the RAID5/6 computations are so fast that even a single
thread should be able to do them.
Use the "snapraid -T" comment to see the speed.
Also, all the file operations are already done in background by the OS,
so no improvement is expect from this side.