snapraid/raid/README

RAID Library
============

This directory contains the Galois Field GF(2^8) erasure coding engine used by
SnapRAID. It provides multi-parity generation, missing data and parity block
recovery, error candidate validation, and brute-force block failure scanning.

The library supports up to 251 data blocks and up to 6 parity blocks using an
Extended Cauchy Matrix construction that guarantees MDS (Maximum Distance
Separable) properties for all supported combinations of disk failures.


Public API
----------

Header: #include "raid.h"

Before invoking any generation or recovery functions, call raid_init() once to
detect CPU capabilities and initialize mode dispatch.

  void raid_init(void)
    Initializes algorithm tables and detects hardware SIMD instruction sets
    (SSE2, SSSE3, AVX2, AVX-512, GFNI, NEON). Sets default operating mode to
    RAID_MODE_CAUCHY_RAID. Must be called before any other API function.

  int raid_mode(int mode)
    Sets or queries the active operating mode.
    - Pass RAID_MODE_GET (-1) to query current active mode.
    - Pass RAID_MODE_CAUCHY_RAID, RAID_MODE_VANDERMONDE_RAID, or
      RAID_MODE_CAUCHY_AES to set mode.
    Returns the mode active prior to this call.

  void raid_zero(void *zero)
    Registers a read-only buffer filled with zeros of block size 'size'.
    Required prior to calling raid_rec() or raid_data() for delta parity
    computation.

  void raid_gen(int nd, int np, size_t size, void **v)
    Computes 'np' parity blocks from 'nd' data blocks. Vector 'v' contains
    'nd + np' block pointers (data blocks v[0..nd-1], parity blocks
    v[nd..nd+np-1]). Data blocks are read-only; parity blocks are written.

  void raid_rec(int nr, int *ir, int nd, int np, size_t size, void **v)
    Recovers 'nr' failed blocks (data or parity) listed in sorted index array
    'ir'. Indexes start at 0 (data 0..nd-1, parity nd..nd+np-1). Pointers in 'v'
    for good blocks are read; pointers for bad blocks are target destinations.
    Requires nr <= np.

  void raid_data(int nr, int *id, int *ip, int nd, size_t size, void **v)
    Recovers 'nr' failed data blocks listed in sorted array 'id', using
    parities listed in sorted array 'ip'. Parity blocks in 'v' are unmodified.
    Requires nr <= nd and nr <= RAID_PARITY_MAX.

  int raid_check(int nr, int *ir, int nd, int np, size_t size, void **v)
    Validates whether candidate failed block set 'ir' matches available parity
    redundancy. Returns 0 if candidate set matches, or -1 if invalid. Requires
    nr < np. Does not modify data or parity blocks.

  int raid_scan(int *ir, int nd, int np, size_t size, void **v)
    Searches for up to 'np - 1' failed blocks using available redundancy.
    Fills 'ir' with failed block indexes and returns the number of failures
    found (0 if no error), or -1 if failed blocks cannot be uniquely located.

  int raid_selftest(void)
    Runs startup verification tests for parity generation, recovery, and
    scanning across parity levels 1..6. Returns 0 on success, < 0 on failure.


Memory and Alignment Requirements
---------------------------------

  - Vector Layout: In vector 'v', data blocks MUST precede parity blocks:
      v[0 ... nd-1]       -> Data blocks
      v[nd ... nd+np-1]   -> Parity blocks (PAR1, PAR2, ..., PAR6)

  - Block Size: Parameter 'size' MUST be a multiple of 64 bytes
    (size % 64 == 0).

  - Memory Alignment: Blocks should ideally be aligned on 16/32/64-byte
    boundaries to optimize SIMD vector loads and stores.

  - Sorted Index Arrays: Index arrays ('ir', 'id', 'ip') MUST be strictly
    sorted in ascending order (e.g. ir[0] < ir[1] < ... < ir[nr-1]).


Operating Modes and Polynomials
-------------------------------

  RAID_MODE_CAUCHY_RAID (Default mode = 0)
    Uses Cauchy matrix generator with standard RAID polynomial 0x1d
    (x^8 + x^4 + x^3 + x^2 + 1, decimal 285, 0x11d) and primitive generator
    g = 2. Supports parities 1 to 6. Matches standard Linux RAID-6 (Anvin
    method) for 2-parity arrays and ensures on-disk compatibility. On CPUs
    supporting Intel GFNI, generation and recovery are accelerated using
    VGF2P8AFFINEQB with precomputed affine matrices for the RAID field.

  RAID_MODE_VANDERMONDE_RAID (Mode = 1)
    Legacy mode supporting up to 3 parities using standard polynomial 0x1d.
    Uses Cauchy for PAR1/PAR2 and Vandermonde matrix coefficients for PAR3
    (Z parity). Optimized for low-end CPUs without vector instructions.

  RAID_MODE_CAUCHY_AES (Mode = 2)
    Uses Cauchy matrix with AES polynomial 0x1b (x^8 + x^4 + x^3 + x + 1,
    0x11b) and primitive generator g = 3. On CPUs supporting Intel GFNI,
    generation and recovery are accelerated directly using VGF2P8MULB.
    This native GFNI multiplication is typically faster than the affine GFNI
    implementation required by the standard RAID polynomial. Parity data is
    incompatible with standard 0x11d polynomial arrays.

  Both Cauchy modes use the same Extended Cauchy construction with coefficients
  based on powers of their respective primitive generator. GFNI acceleration
  is available for both fields: the RAID field uses affine transformations,
  while the AES field uses the native GFNI multiplication instruction.


Implementation Structure
------------------------

  raid.h        Public API declarations, macros, and mode constants.
  raid.c        Cauchy matrix solver, Gauss-Jordan inversion, delta generation.
  module.c      Algorithm registration, CPU detection, and dynamic dispatch.
  int.c         Portable C fallback implementation (8/32/64-bit word loops).
  tables.c      Precomputed GF(2^8) log/exp, multiplication, Cauchy tables.
  mktables.c    Generator for GF(2^8), Cauchy, and GFNI affine tables.
  internal.h    Private header, assertion macros (BUG_ON), alignment helpers.

  Platform-Specific Acceleration:
    sse2.c      x86 SSE2 (128-bit SIMD vectorization)
    ssse3.c     x86 SSSE3 (128-bit pshufb nibble-split multiplication)
    avx2.c      x86 AVX2 (256-bit vectorization)
    avx512.c    x86 AVX-512BW (512-bit vectorization)
    gfni.c      x86 GFNI generation/recovery (RAID affine and AES multiply)
    neon.c      ARM NEON (128-bit SIMD vectorization)


Testing and Benchmarks
----------------------

Standalone test applications located in raid/test/ provide verification and
performance evaluation:

  raid/test/selftest   Quick startup self-test verifying basic correctness.
  raid/test/fulltest   Exhaustive test suite testing all combinations of data
                       disks (1..251) and single/multi-disk failure recovery.
  raid/test/invtest    Matrix inversion test verifying non-singularity for all
                       square submatrices of the Cauchy matrix.
  raid/test/speedtest  Benchmark measuring single-thread throughput (MiB/s)
                       for parity generation and recovery across all SIMD
                       implementations.