SnapRAID RAID library performance notes ======================================= Scope ----- These notes summarize performance experiments, implementation choices, rejected ideas, recovery optimizations, and non-obvious conclusions reached while reviewing and tuning the SnapRAID RAID library. The focus is the 1..6 parity implementation, especially x86 AVX2, AVX512BW, GFNI, SSSE3, and AArch64/AArch32 NEON backends. This is intentionally a record of what has already been tried and what was learned from it, so that the same ideas are not repeatedly re-investigated. Detailed project-by-project comparisons with other RAID and erasure-coding engines are kept in OTHERS. This file keeps only conclusions that affect SnapRAID implementation or optimization policy. The main rule is to optimize the real SnapRAID workload, not only isolated inner loops. Implementation overview ----------------------- Generation: * For the CPUs, compilers, and SnapRAID workloads tested so far, the 1..6 parity generation paths have no known major performance bottleneck after coefficient-residency optimization. * Relevant scalar, SSE/SSSE3, AVX2, AVX512BW, AArch64/AArch32 NEON, RAID GFNI affine, and AES GFNI paths are implemented. * Specialized GEN2 and GEN3 paths keep useful multiplication tables or coefficient representations in vector registers when register pressure permits it. * This residency optimization preserves the preferred chunk-major, multi-output traversal. It does not add another pass over source blocks. * RAID GFNI generation uses direct Cauchy coefficients. Their 8-byte affine representations are precomputed in a disk-major table, avoiding a dependent Cauchy-coefficient -> affine-matrix lookup in the hot generation loop. Similarly, scalar int8 generation precomputes per-disk 2D multiplication table pointers T[d][p] before the hot loop to eliminate two-level dependent table lookups. * AVX512 128-byte processing, non-GFNI AVX512BW Horner, standalone VPTERNLOGQ transformations, VPBLENDVB, cached parity stores, and large pre-expanded GFNI constants have been investigated. Some are rejected, while VPTERNLOGQ is now used where it is part of a larger profitable GEN2/GEN3 restructuring. Recovery: * The normal one-data-disk recovery with P available is handled via raid_rec1of1() across all backends (Scalar, NEON, SSSE3, AVX2, AVX512BW, GFNI), invoking raid_gen(nd, 1, size, v, 0) with temporal stores. * SIMD recovery has parity-count-specific REC1..REC6 entry points. Fused paths compute exactly the selected syndromes in one scan of the surviving data and reconstruct the missing blocks without full-size temporary syndrome buffers. * AVX512BW and GFNI (AVX2 and AVX512) use raid_rec1of1() for P and fused syndrome generation for non-P REC1 and all REC2..REC6 cases, including P+Q (rec2of2). AVX2 and AVX2 extended use raid_rec1of1() for P, use specialized 1-pass Horner kernels for Q, and use fused selected-syndrome recovery for the other non-P REC1 cases and fused recovery for all REC2..REC6 cases. SSSE3 and SSSE3 extended likewise use raid_rec1of1() for P, use specialized 1-pass Horner kernels for Q-only REC1 and P+Q REC2, and use fused selected-syndrome recovery for the remaining REC1..REC6 cases. AArch64 NEON and AArch32 NEON use raid_rec1of1() for P and fused syndrome generation for non-P REC1 and all REC2..REC6 cases. * Scalar recovery uses raid_rec1of1() for P and fused syndrome recovery for non-P REC1 and all REC2..REC6 cases. raid_delta_gen() is completely eliminated. * Non-GFNI fused recovery splits every completed syndrome into low/high nibbles once and reuses the split values across all reconstructed outputs. * Non-GFNI fused recovery precomputes survivor coefficient and inverse-matrix table pointers in C before entering the hot data loop. * RAID GFNI fused recovery precomputes direct pointers to the 8-byte affine representation of every syndrome and inverse-matrix coefficient. * A recurring RECX local cost is repeated coefficient table/matrix loading or broadcasting inside the data loop. * Specialized P+Q rec2of2 recovery reduces 4 GF multiplications to 2 GF multiplications and 1 XOR; AVX512BW, GFNI, AVX2, and AArch64 NEON handle this directly inside the 1-pass fused kernel. SSSE3 and SSSE3ext use dedicated 1-pass Horner rec2of2 helpers, AArch32 NEON uses a dedicated delta-based helper, and scalar int8 handles it via has_p in raid_rec2_int8. * When P is among the selected parity blocks, optimized SIMD and scalar RECX paths reconstruct only nr - 1 missing blocks through the inverse matrix and derive the final missing block from the raw P delta by XORing the blocks already reconstructed. Architecture and validation: * AArch32 has a separate NEON backend for generation and recovery. Cross-build, QEMU user-mode testing, and CI coverage are available for the ARM32 path. * Matrix inversion has dedicated multithreaded stress testing. With at most six parity levels, inversion setup is not a significant performance target. Real workload assumptions ------------------------- * SnapRAID processes large blocks: typically 256 KiB per data/parity block. * A normal stripe therefore touches a large amount of memory. With 12 data disks, one 256 KiB stripe already means about 3 MiB of data input before considering parity output. * This large-block behavior is very important. Optimizations that reread a source buffer, increase the number of passes, or enlarge lookup tables can be worse even if the inner instruction sequence looks better. * This is also the first filter for ideas taken from other RAID or erasure-coding engines. A design optimized for page-sized recovery, cache-resident tiles, or generic small shards may make a different memory tradeoff and should not be copied mechanically. * Prefer reducing work inside the existing single-pass traversal before considering a traversal change. Saving a broadcast or table load is useful; saving registers by rereading a complete 256 KiB source block usually is not. * SnapRAID uses a queue before parity data is written to disk. In the measured SnapRAID disk-write workloads used during this tuning, very small queues were slower. Throughput improved clearly up to about queue depth 32 and continued to improve beyond that with diminishing returns. These queue-depth results are workload- and storage-dependent and are not an architectural property of the RAID kernels. * With 256 KiB blocks, queue depth 32 means this much parity output can be in flight: 1 parity = 8 MiB 2 parity = 16 MiB 3 parity = 24 MiB 6 parity = 48 MiB Therefore a parity block produced by the RAID kernel is normally not consumed immediately. * A memory read of a 256 KiB block that is no longer cached is still very cheap compared with real storage I/O. Trying to keep many queued parity blocks hot in cache is therefore not necessarily useful. MOVNT / non-temporal stores vs cached generation -------------------------------------------------- * raid_gen(..., streaming) allows the caller to select between non-temporal streaming stores (streaming != 0, using MOVNTDQ/VMOVNTDQ + sfence on x86) and normal temporal/cached stores (streaming == 0, using MOVDQA/VMOVDQA without sfence). * Generation backends select MOVNTDQ / VMOVNTDQ for parity output when streaming != 0 is requested. * The synthetic speedtest naturally favors non-temporal stores because generated parity is not consumed afterwards. This benchmark therefore measures producer throughput, not the complete producer -> queue -> writer pipeline. * In the real SnapRAID sync pipeline, parity blocks are queued to disk writers, which makes the workload non-temporal (streaming = 1). At typical queue depths, cached parity output would occupy many MiB of cache and evict useful input data; MOVNT avoids normal cache allocation/write-allocate traffic and reduces cache pressure. * Conversely, in-memory parity verification in scrub and check, and single-parity recovery in raid_rec1of1() use cached generation (streaming = 0) so that computed parity or recovered data remains hot in cache for immediate CPU consumption. Recovery temporal stores ------------------------ * Recovery output has a different lifetime from generated parity written to disk. A reconstructed data block is an intermediate result that is consumed again immediately. * After raid_data() reconstructs a candidate, repair_step() immediately validates it. With a trustworthy hash, blockcmp()/memhash_block() reads the recovered data. Without such a hash, parity validation regenerates parity from the reconstructed data. On successful hash validation, all configured parity is regenerated as well. * During fix, the same recovered data buffer is then passed to handle_write(), so keeping recently produced cache lines available can also help the following write path. * For this reason, recovery uses ordinary temporal MOVDQA/VMOVDQA stores for recovered data rather than MOVNTDQ/VMOVNTDQ. Single-disk recovery with P (raid_rec1of1()) invokes raid_gen(nd, 1, size, v, 0) with streaming = 0, ensuring temporal stores and omitting sfence across all backends. * With typical 256 KiB blocks and roughly 8..12 data disks, the full recovery working set may fit in L2 on some CPUs and exceed it on others. Do not switch store policy at an architecture-specific disk-count/cache-size threshold: the recovered block itself has strong immediate reuse on all of these machines, while generated parity written during sync is normally queued and consumed later. Temporal recovery stores are therefore the general compromise. * Store policy is deliberately selectable: generation for disk writing (sync) uses non-temporal stores (streaming = 1), while recovery and in-memory verification use temporal stores (streaming = 0). Typical recovery case --------------------- * The common recovery case is one failed data disk while parity P is still available. * raid_rec() selects the first available parity. Therefore, for a single data failure with P available, recovery uses P. * All backends (Scalar, NEON, SSSE3, AVX2, AVX512BW, GFNI) route the P case through raid_rec1of1(). * raid_rec1of1() remaps pointers so that P is used as an XOR input and the failed data buffer is used as the GEN1 output, then calls raid_gen(nd, 1, ..., 0). All backends use this path with temporal stores for optimal P single-disk recovery. * Therefore the common single-disk recovery path executes with no syndrome or GF overhead. * The data traffic is already close to the minimum: read P read all surviving data blocks XOR them write the recovered data block once * There are no temporary syndrome buffers, no GF multiplication, and no matrix inversion in this common path. * This means that large architectural recovery changes mostly optimize rare failure cases, not the usual single-disk recovery. * Two simultaneous data disk failures are already an exceptional case in normal use. Three or more simultaneous data failures are even less important as optimization targets. Recovery speedtest interpretation --------------------------------- The recovery speed tests contain separate rows for the important fast paths and for the generic recovery paths. REC1OF1: * The row uses ip[0] == 0. * All backends use raid_rec1of1(), calling the GEN1 backend with streaming = 0 to reconstruct the missing block directly into cache using temporal stores. * Because the row measures only the recovery producer and does not include the immediate validation/parity-regeneration consumers, temporal-store x86 implementations can benchmark lower than a non-temporal GEN1-based path even when the temporal policy is preferable for the complete check/fix workload. REC1OF2 and REC1OF3: * The rows use Q and R respectively, avoiding the PAR1 fast path. * All backends use single-pass recovery for these non-P cases. AVX2, AVX2 extended, SSSE3, and SSSE3 extended use specialized Horner kernels for Q and fused selected-syndrome kernels for the other parity rows. AVX512BW, GFNI, NEON, AArch32 NEON, and Scalar use their fused recovery paths. REC2OF2: * The row uses ip = { 0, 1 }, meaning P and Q. * On AVX2, AVX512BW, GFNI (AVX2 and AVX512), AArch64 NEON, AArch32 NEON, and Scalar int8, REC2 executes directly in the 1-pass fused kernel, which computes P with XOR, Q with GF multiplication, and derives the second missing block via XOR. * SSSE3 and SSSE3ext call dedicated 1-pass Horner raid_rec2of2_*() helpers that compute P and Q directly from the surviving data in a single pass. * This makes the row a direct benchmark for the specialized P+Q recovery implementation. REC2OF3 and REC2OF4: * The rows use { Q, R } and { R, S } respectively, avoiding the P+Q fast case. * REC2OF3 and REC2OF4 remain useful control rows because they do not match ip = { 0, 1 } and therefore continue to exercise the generic REC2 paths. * They measure generic 2x2 recovery with different selected-parity positions, verifying exact selected-syndrome fusion. REC3OF3 through REC6OF6: * These rows measure multi-failure recovery for each supported failure count using the first N parity levels. SIMD backends and scalar int8 use fused syndrome recovery. Delta generation: * A benchmark forces a matching generation backend only when the measured wrapper can call raid_rec1of1() (specifically in REC1OF1). * Fused and specialized single-pass recovery rows do not depend on raid_gen() because they generate the required syndromes directly inside the recovery kernel. Single-pass fused recovery -------------------------- * Fused SIMD recovery accumulates exactly the parity rows selected by ip[] directly from the stored parity and surviving data. * This avoids the prefix constraint of the old delta-based approach, which computed np = ip[nr - 1] + 1 via raid_gen(). * Example: nr = 3 ip = { 0, 2, 5 } makes an old delta-based path generate through GEN6, while a fused path calculates only P, R and U. * For each SIMD chunk, the fused path starts the selected syndrome accumulators from stored parity, scans every surviving data source once, and then applies the inverse matrix to reconstruct all missing blocks. * AVX512BW and GFNI keep all one to six syndromes in registers. Non-GFNI AVX2, SSSE3, and NEON paths use registers or small aligned per-chunk scratch space according to register pressure; they do not allocate full-size syndrome buffers. * The common rec1of1/P case executes via raid_rec1of1() across all backends, calling the GEN1 generator with streaming = 0 to write the recovered data block using temporal stores with no syndrome or GF recovery setup. * raid_delta_gen() has been completely eliminated from the codebase. Generation: AVX512 width and unrolling -------------------------------------- * Wider processing is not automatically faster. * AVX512 128-byte processing was tested on the tuning systems represented by these notes and did not consistently outperform the existing 64-byte path; in some tested cases it was slower. * Therefore do not assume that x2/x4 spatial unrolling used by some RAID engines is beneficial for SnapRAID. * The block size is already large, memory traffic is substantial, and wider unrolling can increase register pressure, front-end work, and AVX512 power or frequency effects without reducing total memory traffic. * Standalone VPTERNLOGQ transformations were tested on Zen 5 and did not improve performance. * This is not a general rejection of VPTERNLOGQ. AVX512BW GEN2/GEN3 kernels use VPTERNLOGQ where it is part of a larger table-residency restructuring that reduces other work. Judge the complete kernel, not an instruction in isolation. * Conclusion: keep the conservative AVX512 processing width unless a specific benchmark proves otherwise. Generation: GFNI direct coefficients ------------------------------------ * Classic RAID6 P+Q generation has a natural Horner recurrence for Q with the standard RAID polynomial/generator. * Horner implementations were investigated for GFNI generation, but the final implementation uses direct Cauchy coefficients for both RAID and AES GFNI generation. * RAID GFNI represents each fixed Cauchy coefficient with an 8-byte affine matrix consumed by VGF2P8AFFINEQB. * The generation-specific RAID Cauchy affine matrices are precomputed in: raid_gfcauchyaffine_raid[DISK][PARITY - 1][8] with disk-major layout: [disk][Q/R/S/T/U][8-byte affine matrix] * This avoids the previous dependent lookup sequence: load raid_gfcauchy_raid[p][d] use the coefficient to index raid_gfaffine_raid[] load/broadcast the affine matrix and replaces it with one direct address: raid_gfcauchyaffine_raid[d][p - 1] * The generic raid_gfaffine_raid[256][8] table remains necessary for runtime-computed multipliers, especially recovery. * AES GFNI generation uses the corresponding direct Cauchy coefficient byte with VGF2P8MULB. * Keeping both RAID and AES generation in direct-coefficient form also makes their GFNI kernels structurally comparable: the main field-specific difference is VGF2P8AFFINEQB with an 8-byte RAID matrix versus VGF2P8MULB with an AES coefficient byte. * Scalar int8 generation (raid_gen2_int8 .. raid_gen6_int8) similarly precomputes per-disk multiplication table pointers T[d][p] in a 2D array before entering the hot loop, replacing the double-indexed raid_gfmul[d0][raid_gfgen[j][d]] lookup with a single direct table access T[d][p][d0]. Generation: coefficient residency --------------------------------- Recent GEN2/GEN3 work established coefficient residency as a practical generation optimization, not only a theoretical one. * Non-GFNI SIMD GEN2/GEN3 kernels (such as AVX512BW) preload useful low/high multiplication tables into vector registers when the register file has enough space. AVX512 GFNI kernels similarly preload Cauchy affine matrices (for RAID) or constant multiplier bytes (for AES). * The resident set can be partial. When there are more data disks than can be covered by spare registers, later disks fall back to loading their coefficient tables normally. * GEN3 has higher register pressure because both Q and R coefficients are needed. Residency is therefore deliberately bounded by the available register file rather than forcing spills or another source pass. * AVX512BW can use VPTERNLOGQ as part of these resident-table kernels to combine the accumulator with low/high products. The useful optimization is the whole scheduling and residency change, not VPTERNLOGQ by itself. * The source chunk is still loaded once and contributes to all parity outputs before moving on. The optimization therefore preserves the most important large-block memory property. General conclusion: * First use spare registers to remove repeated coefficient setup while preserving one-pass multi-output generation. * Do not trade coefficient residency for extra 256 KiB source passes. * Generation success does not imply that RECX can use the same register map. RECX must also keep multiple syndromes and reconstruction state live, so its coefficient-residency opportunity remains narrower. Overall generation conclusion: * For the CPUs, compilers, and workloads represented by the measurements in these notes, no major generation optimization is currently known with a strong reason to be investigated. * Further generation changes should be driven by new hardware evidence, a changed workload, or measurements that identify a concrete bottleneck rather than by an assumption that an optimization from another implementation will also improve SnapRAID. GFNI field implementation ------------------------- * AES GFNI multiplication maps naturally to VGF2P8MULB because the instruction uses the AES polynomial. * Standard RAID polynomial 0x11d cannot use VGF2P8MULB directly. * RAID GFNI therefore uses VGF2P8AFFINEQB with precomputed affine matrices. * Multiplication by a fixed coefficient in the RAID 0x11d field does not require converting the operand to the AES field. Multiplication by a constant is GF(2)-linear, so it is represented directly by an 8x8 GF(2) matrix and executed with one VGF2P8AFFINEQB instruction. * Native AES VGF2P8MULB is therefore not assumed to be inherently faster than RAID VGF2P8AFFINEQB for the SnapRAID workload. Relative performance depends on the CPU, backend, parity count, and workload. * The global affine matrix table stores 8 bytes per coefficient. * RAID GFNI generation additionally stores the affine representation of each Cauchy coefficient directly in a disk-major raid_gfcauchyaffine_raid[251][5][8] table. This removes the dependent Cauchy-coefficient -> global-affine-table lookup from generation while the generic affine table remains available for runtime multipliers. * A generic exhaustive software test validates all coefficient/input pairs against the normal RAID GF multiplication table. * Dedicated hardware affine-matrix validators were intentionally not added. Normal generation/recovery tests exercise the hardware path. * RAID and AES GFNI recovery/generation have polynomial-specific names: *_gfni_raid *_gfni_aes with backend tags: gfni gfni512 * This naming and registration split is intentional and should be preserved. GFNI RECX coefficient precomputation ------------------------------------ RAID GFNI fused recovery precomputes direct pointers to the complete 8-byte affine representation for: every surviving-data / selected-syndrome coefficient every inverse-matrix reconstruction coefficient The hot loop therefore uses: direct load/broadcast through the precomputed pointer VGF2P8AFFINEQB This keeps the coefficient-byte -> affine-table lookup outside the hot loop while retaining the compact 8-byte representation. Important details: * All 8 bytes form one affine matrix coefficient; it cannot be treated as a single-byte coefficient. * AVX2 and AVX512 RAID GFNI recovery both use this precomputation. * AES GFNI does not need this conversion because VGF2P8MULB consumes the coefficient byte directly. * The remaining repeated operation is the per-source or per-reconstruction-term broadcast of the 8-byte affine matrix for each data chunk. GFNI pre-expanded affine tables ------------------------------- * Some high-performance generic erasure-code engines use pre-expanded GFNI constants, for example a 32-byte representation ready for vector use. * For SnapRAID this would increase the RAID affine table from about 2 KiB to about 8 KiB. * In the tested 256 KiB SnapRAID workloads, the compact 8-byte entries plus broadcast provided the preferred cache-footprint tradeoff. * No measured result has justified trading four times the table footprint for fewer broadcasts. * Decision for the currently tested workloads: keep compact 8-byte affine matrices. External implementation comparisons ----------------------------------- Detailed comparisons are in OTHERS. The implementations studied include specialized RAID engines, filesystem RAID, and general Reed-Solomon libraries. The important conclusions are general rather than project-specific. Classify an external advantage before trying to copy it: algorithm advantage the method performs genuinely less mathematical work implementation advantage the same work is scheduled, vectorized, or prepared better workload-specific advantage the method wins because the external workload has different block, cache, parity, update, or latency assumptions This distinction prevents a good optimization for another engine from becoming an automatic SnapRAID TODO. Memory traversal: * Several modern high-throughput engines confirm that loading one source chunk and updating multiple outputs from it is a strong general strategy. * Other engines deliberately process cache-resident regions and repeatedly update cached parity. That can be correct when parity is consumed again immediately, but it is a different memory model from SnapRAID generation. * Whole-region operations that keep one coefficient resident can be efficient for generic erasure coding. For SnapRAID, the same loop orientation can require rereading large source blocks and is normally a poor trade. * The preferred order remains: preserve the one-pass source traversal, then use spare registers, hoisting, and precomputation to reduce local coefficient cost. Specialization: * SnapRAID can specialize aggressively because parity is limited to 1..6 and the matrix structure is known. * Generic engines may use large generated N-input/M-output kernel families, general decoding matrices, structured solvers, or broader runtime machinery. These are valuable for their scale but do not automatically reduce work in the N <= 6 SnapRAID domain. * The most useful external ideas are usually local: fuse multiple outputs, precompute coefficient representations, reuse source splits, special-case coefficient 0/1 where simple, and exploit algebraic fast paths. Cache and stores: * Store policy should be evaluated against output lifetime. Cached stores can be advantageous when an algorithm immediately reloads parity; non-temporal stores can be advantageous when final parity is queued and consumed later. * Therefore cached stores in another fast implementation are not evidence that SnapRAID should change its MOVNT policy. Recovery: * General matrix inversion or structured Cauchy solvers become important at larger scale. For at most six missing blocks, moving 256 KiB regions is far more important than reducing the tiny matrix-setup cost. * Algorithmic failure-case specializations are different: if a fixed recovery case provably needs fewer GF multiplications, it remains worth measuring. The specialized P+Q rec2of2 case is an example. Runtime and architecture: * Some engines benchmark implementations at startup, while others dispatch only from CPU capabilities. Runtime calibration remains low priority for SnapRAID because the kernel set is small and the workload is stable. * Other engines support additional SIMD families such as SVE, RVV, or VSX. These are useful porting references, not hot-path optimization tasks without a target platform requirement. Memory layout, displacement and stride noise -------------------------------------------- * The memory layout is not a neutral detail. * Fixed displacement between buffers was observed to help. * STRIDE_NOISE behavior differs by CPU. * On Intel, stride noise had little effect in the observed tests. * On Zen 5, disabling stride noise caused a large throughput reduction. * This indicates that hardware prefetch/cache behavior is already strongly affected by the chosen buffer layout. * Therefore new software-prefetch or traversal changes should not be assumed to help. They must be benchmarked against the existing displacement/stride behavior. * This is another reason to avoid adding extra passes over 256 KiB blocks. REC1 optimizations ------------------ The normal REC1 case with P is handled via raid_rec1of1() across all backends (scalar, SSSE3, AVX2, AVX512BW, GFNI, AArch64 NEON, and AArch32 NEON), using streaming = 0 for temporal stores. For generic REC1, used when P is unavailable: * AVX2, AVX512BW, GFNI, SSSE3, SSSE3 extended, AArch64 NEON, and AArch32 NEON use single-pass recovery. AVX2/AVX2ext and SSSE3/SSSE3ext use specialized Horner kernels for Q and fused selected-syndrome kernels for the other parity rows. * Scalar uses its delta-based REC1 kernel. REC1 status: * For the current large-block SnapRAID workload and implemented backends, raid_rec1of1() achieves optimal memory saturation for rec1of1/P. * Generic non-P REC1 is selected per parity count and backend, so single-pass SIMD and delta-based scalar implementations are benchmarked independently. * No major REC1 algorithmic optimization is currently expected. REC2 optimizations ------------------ REC2 has both fused and delta-based implementations. * AVX2, AVX512BW, GFNI, and AArch64 NEON use fused syndrome generation for every selected parity pair, including P+Q. SSSE3 and SSSE3 extended use fused selected-syndrome recovery for generic REC2 and dedicated 1-pass Horner kernels for P+Q. AArch32 NEON uses fused recovery for every pair except P+Q. * Scalar int8 uses delta-based REC2. AArch32 NEON retains a delta-based specialized P+Q helper. Important portability result: * x86-32 has only the low eight XMM/YMM registers available to these old ISA paths. * Optimizations that depend on xmm8..xmm15 or ymm8..ymm15 must remain restricted to CONFIG_X86_64. REC2 status: * Generic REC2 uses the fused path across AVX2, AVX512BW, GFNI, SSSE3, SSSE3ext, and NEON. * AVX2, AVX512BW, GFNI, and AArch64 NEON handle P+Q recovery directly inside the 1-pass fused kernel. SSSE3 and SSSE3ext use dedicated 1-pass Horner rec2of2 helpers, AArch32 NEON uses its dedicated delta-based rec2of2 helper, and scalar int8 handles it via has_p. Specialized rec2of2 P+Q ----------------------- For failures of two data disks recovered specifically with P and Q, the system can be reduced to: Dy = C0 * Pd ^ C1 * Qd Dx = Pd ^ Dy This needs only two GF multiplications for Dy, followed by one XOR for Dx, instead of the generic 2x2 reconstruction with four GF multiplications. Backends implement this specialization as follows: * AVX2, AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON integrate this derivation directly into the unified 1-pass fused kernel (P is accumulated via XOR, Q via GF multiplication, Dy is solved with inverse coefficients, and Dx derived by XOR with Pdelta). * Scalar int8 (has_p branch in raid_rec2_int8) * SSSE3 and SSSE3 ext use dedicated 1-pass Horner kernels (raid_rec2of2_ssse3, raid_rec2of2_ssse3ext) * AArch32 NEON uses the delta-based raid_rec2of2_neon32 helper The rec2of2 benchmark row directly measures this specialized P+Q fast path. REC2OF3 and REC2OF4 remain useful control rows because they do not match ip = { 0, 1 } and therefore continue to exercise the generic REC2 paths. RECX delta handling ------------------- RECX computes the selected syndromes directly from stored parity and one scan of the surviving data for each SIMD chunk. For non-GFNI paths: * each completed syndrome is split into low/high nibbles once * the split values stay in registers or bounded aligned per-chunk scratch space * the same split values are reused across all reconstructed output rows * no full-size temporary syndrome block is generated or reloaded For GFNI paths: * the byte syndromes remain directly in SIMD registers * no nibble split is needed When P is among the selected parity blocks, optimized RECX paths can reconstruct only nr - 1 missing blocks through the inverse matrix and derive the final block from the raw P delta by XORing the blocks already reconstructed. This removes one complete inverse-matrix reconstruction row without adding another source-data pass. On the tested x86 implementations, this improved the affected multi-disk recovery paths by roughly 8% to 12%, depending on the backend and failure count. RECX shuffle-table pointer precomputation ---------------------------------------- Non-GFNI fused recovery precomputes in C: S[s][j] = &raid_gfmulpshufb[A(ip[j], d)][0][0]; R[j][k] = &raid_gfmulpshufb[V[j * N + k]][0][0]; The hot assembly then only performs: load S[s][j] or R[j][k] load or broadcast the low table at offset 0 load or broadcast the high table at offset 16 This has two advantages: 1. It removes coefficient-index address arithmetic from the hot assembly. 2. It keeps C object layout and address calculation in C, leaving the assembly focused on SIMD operations. Users include: * AVX2 and AVX2 extended selected-syndrome REC1/REC2 and REC3..REC6 paths * AVX512BW RECX * SSSE3 and SSSE3 extended selected-syndrome REC1/REC2 and REC3..REC6 paths * AArch64 and AArch32 NEON fused recovery paths * Scalar int8 RECX (raid_recX_int8), which precomputes 256-byte raid_gfmul pointers S[s][j] and R[j][k], and structures inner loops with explicit if (nr >= N) branches so the compiler fully unrolls syndrome accumulation and inverse-matrix reconstruction when nr is a compile-time constant. The remaining cost is that the low/high tables are still loaded or broadcast for every coefficient of every data chunk. RECX register state ------------------- The fused RECX structure uses registers aggressively and falls back to small per-chunk scratch space where the register file is constrained. AVX512BW: * up to six syndrome values are kept as low/high pairs in zmm0..zmm11 * zmm12..zmm15 are reconstruction temporaries/accumulators * zmm31 keeps the low-nibble mask * coefficient shuffle tables are still broadcast through R[][] during reconstruction AVX512 RAID GFNI: * up to six byte syndromes are kept in zmm0..zmm5 * temporary/accumulator registers are used for reconstruction * coefficient affine matrices come directly through precomputed R[][] pointers * one 8-byte affine matrix is still broadcast per coefficient per chunk AVX2 RAID GFNI: * the two 32-byte halves of up to six syndromes occupy most of the available YMM register file * S[][] and R[][] remove indexed affine-table lookup * register pressure leaves less room for keeping many coefficient matrices resident AVX2/SSSE3 non-GFNI: * up to four raw syndromes are accumulated in registers on the specialized paths * larger failure counts use aligned per-chunk syndrome scratch space * register pressure is significantly higher than AVX512 * coefficient table residency is therefore much more constrained NEON: * selected syndromes are accumulated in one surviving-data scan * S[][] and R[][] provide direct pointers to low/high multiplication tables * reconstruction loads each coefficient table pair as needed * AArch64 NEON unrolls non-P REC1 and REC2 to 64-byte chunks across four 16-byte vector lanes, keeping all syndrome accumulators and Pdelta resident in registers without temporary memory buffers * AArch64 NEON REC3..REC6 use the 32-byte generic kernel across two 16-byte lanes RECX coefficient residency -------------------------- The main RECX local optimization question is: can repeated coefficient loads/broadcasts be reduced by keeping useful coefficient tables or affine matrices resident in spare vector registers? This is most plausible on AVX512 because of the larger register file. Possible experiments: * keep one output row of coefficients resident across data chunks * keep a useful prefix of coefficient tables resident when N permits it * on AVX512 RAID GFNI, keep several 8-byte affine matrices broadcast in ZMM registers instead of broadcasting them again for every chunk * on AVX512BW, keep some low/high shuffle-table pairs resident Constraints: * Do not add a second pass over the 256 KiB block. * Do not spill syndrome values or accumulators merely to save coefficient broadcasts. * Do not add large N-specific kernel families unless measurements justify the complexity. * AVX2 and SSSE3 register pressure is already high, so the likely opportunity is much smaller there. * NEON has more architectural registers than old x86 SIMD but RECX already uses a significant part of them; any residency experiment should be measured carefully. This is a narrow local problem. Fused syndrome generation, nibble reuse, coefficient address precomputation, and GFNI affine lookup precomputation are already implemented. Recovery optimization policy ---------------------------- For recovery, avoid new architecture-level changes unless a very strong case appears. Preferred optimizations are basic/local: * hoist loop-invariant values * keep masks/tables/matrices resident when register pressure allows it * avoid repeated loads or broadcasts * initialize accumulators from the first useful term * keep address calculations in C when doing so simplifies inline assembly * exploit coefficient 0 or 1 only when it stays simple and does not add harmful branching or complexity * use limited spatial unrolling only when it exposes ILP without excessive register pressure; SSSE3ext REC1 was faster with a two-way 32-byte unroll than with the tested four-way 64-byte version Avoid unless new measurements provide a concrete reason: * extra full passes over source blocks * aggressive spatial unrolling * software prefetch without clear evidence * changing MOVNT policy * major loop-orientation changes * large pre-expanded lookup tables * broad runtime calibration machinery Recovery matrix setup and inversion ----------------------------------- * Recovery matrices are at most 6x6 for the 1..6 parity implementation. * The arithmetic cost of constructing and inverting such a matrix is tiny compared with reading and writing 256 KiB data regions. * More elaborate structured Cauchy solvers, inversion caches, or asymptotically faster coding methods are therefore not useful performance targets for the 1..6 parity limit. * A dedicated multithreaded inversion stress test exists to validate the matrix code independently of the data-path SIMD kernels. * At parity counts far beyond six, matrix construction and structured solvers should be re-evaluated together with the whole memory traversal. They should not be optimized in isolation. Inline assembly and portability constraints ------------------------------------------- The SIMD code intentionally uses multiple small or large asm volatile blocks and keeps SIMD state live across them. raid_selftest() is intended to run at every library startup after the RAID mode has been selected. It verifies only the currently selected mode, so changing the mode requires running the self test again before that mode is used. Besides checking the RAID algorithms, it verifies the SIMD implementations actually selected for that mode on the current CPU, compiler, and ABI, providing runtime protection against implementation or compiler-related miscompilation. Important project constraints: * Assembly blocks that dereference pointers internally must model memory effects appropriately. RECX pointer-based blocks use a "memory" clobber. * The x86 SSE/AVX end helpers declare SIMD clobbers needed by the project ABI strategy, including the Windows x64 callee-saved XMM registers. * The NEON end helper declares v0..v31 clobbered so that GCC/Clang preserve AAPCS64 callee-saved v8..v15 when required. * The end-of-function dummy-clobber mechanism is a documented project technique. It is not a fully explicit extended-asm dataflow model, so future compiler portability changes should be reviewed carefully. * This is primarily a compiler and ABI portability consideration rather than a known correctness issue in the current implementation. raid_selftest() verifies the selected mode and implementation at startup so that failures affecting actual RAID results are detected before normal use. * Do not assume x86-32 has xmm8..xmm15 or ymm8..ymm15. * AArch32 NEON is a separate backend. Do not assume the AArch64 register or ABI model when changing ARM32 assembly. * RAID thread-safety requirements are documented. Avoid introducing mutable global calibration or dispatch state without preserving those requirements. * RAID block sizes passed to the SIMD code are multiples of 64, so the 32-byte and 64-byte unrolled loops do not need tail handling. These are established implementation constraints, not correctness findings. Scaling beyond 6 parity levels ------------------------------ The implementation deliberately targets at most six parity levels. * At this scale, fixed-parity specialization keeps matrix setup small and enables tight kernels with explicit register allocation. * General-purpose erasure-code engines demonstrate other designs, including generic matrix kernels, generated multi-input/multi-output kernels, and algorithms intended for much larger shard counts. * Those designs solve a different scaling problem. They should not be adopted merely to generalize a domain where the fixed 1..6 structure is useful. * If the parity limit is increased substantially, generic intrinsics or generated kernels should be compared with handwritten specialization. Memory traversal must remain a first-class design constraint. * Scaling beyond six parity levels is primarily a functional and architectural change, not an unresolved performance problem of the 1..6 implementation. Short list of ideas already tried or rejected --------------------------------------------- * AVX512 128-byte generation processing: tried, can be slower. * Non-GFNI AVX512BW Horner: tried on Zen 5, no improvement. * Standalone VPTERNLOGQ substitutions: tried on Zen 5, no improvement. This does not reject integrated use as part of the profitable GEN2/GEN3 resident-table kernels. * VPBLENDVB x2: tried, no benefit / worse. * MOVNT -> cached stores: not appropriate for the normal queued disk-write workload; keep MOVNT. * GFNI 32-byte pre-expanded affine constants: rejected due to larger cache footprint. * Split outputs into multiple full source passes: rejected because 256 KiB source blocks would be reread. * Whole-region/output-major rewrites intended mainly to keep one coefficient resident: poor fit when they add source rereads. Use register residency inside the existing traversal instead. * Two-pass delta generation (raid_delta_gen()): completely eliminated in favor of generic single-pass fused recovery and dedicated 1-pass Horner kernels. All backends use raid_rec1of1() for rec1of1, and fused/Horner recovery for all other combinations. raid_delta_gen() has been removed. * Software prefetch: no strong reason to add it given the existing long streams and proven sensitivity to displacement/stride tuning. * Runtime backend microbenchmark selection: possible, but low value relative to complexity. * Large generic N-input/M-output kernel families: useful for general erasure coding, but not justified for the fixed 1..6 parity domain without a concrete benchmark need. * Structured or asymptotically faster matrix solvers: unnecessary at N <= 6 because matrix setup is tiny compared with data movement.