SnapRAID RAID library performance notes ======================================= Scope ----- These notes summarize performance experiments, implementation choices, rejected ideas, and non-obvious conclusions reached while reviewing and tuning the SnapRAID RAID library. The focus is the current 1..6 parity implementation, especially x86 AVX2, AVX512BW and GFNI backends. This is intentionally a record of what has already been tried and what was learned from it, so that the same ideas are not repeatedly re-investigated. Real workload assumptions ------------------------- * SnapRAID processes large blocks: typically 256 KiB per data/parity block. * A normal stripe therefore touches a large amount of memory. With 12 data disks, one 256 KiB stripe already means about 3 MiB of data input before considering parity output. * This large-block behavior is very important. Optimizations that reread a source buffer, increase the number of passes, or enlarge lookup tables can be worse even if the inner instruction sequence looks better. * SnapRAID uses a queue before parity data is written to disk. Real measurements with actual disk writes show that very small queues are worse. Performance improves clearly up to about queue depth 32, and still improves after that, but with diminishing returns. * With 256 KiB blocks, queue depth 32 means this much parity output can be in flight: 1 parity = 8 MiB 2 parity = 16 MiB 3 parity = 24 MiB 6 parity = 48 MiB Therefore the parity block produced by the RAID kernel is normally not consumed immediately. * A memory read of a 256 KiB block that is no longer cached is still very cheap compared with real storage I/O. Trying to keep many queued parity blocks hot in cache is therefore not necessarily useful. MOVNT / non-temporal stores --------------------------- * Generation backends use MOVNTDQ / VMOVNTDQ for parity output. * The synthetic speedtest naturally favors non-temporal stores because the generated parity is not consumed afterwards. This benchmark therefore measures producer throughput, not the complete producer -> queue -> writer pipeline. * However, the real SnapRAID queue makes the workload much closer to the non-temporal case than to an immediate-consumer case. * Cached stores could potentially help only with very short queues, where the writer consumes the parity while it is still hot. Queue depths around 1..4 are the most plausible area where this could happen. * Those very short queues are not representative of the real fastest disk pipeline. Real measurements already show that queues up to about 32 provide substantial improvements. * At realistic queue depths, cached parity output would occupy many MiB of cache and could evict useful input data. MOVNT avoids this cache pollution and avoids normal cache allocation/write-allocate behavior for the parity output. * Conclusion: keep MOVNT for the normal generation path. We do not consider replacing MOVNT with cached stores an open optimization anymore. Typical recovery case --------------------- * The common recovery case is one failed data disk, while parity P is still available. * raid_rec() selects the first available parity. Therefore, for a single data failure with P available, recovery uses P. * raid_rec1_avx2()/raid_rec1_avx512bw() special-case ip[0] == 0 and call raid_rec1of1(). * raid_rec1of1() is already effectively fused. It remaps pointers so that P is used as one of the XOR inputs and the failed data buffer is used as GEN1 output, then calls raid_gen(nd, 1, ...). * Therefore the common single-disk recovery path does NOT call raid_delta_gen(). * The data traffic is already close to the minimum: read P read all surviving data blocks XOR them write the recovered data block once * There are no temporary syndrome buffers, no GF multiplication, and no matrix inversion in this common path. * This means that large architectural recovery changes mostly optimize rare failure cases, not the usual single-disk recovery. * Two simultaneous data disk failures are already an exceptional case in normal use. Three or more simultaneous data failures are even less important as optimization targets. raid_delta_gen() fusion ----------------------- * A fused recovery design was studied, where the selected syndromes would be accumulated directly instead of: raid_delta_gen() temporary parity output reload temporary parity solve recovery matrix * The fused design could also compute only the parity rows actually selected by ip[]. * This matters because raid_delta_gen() currently computes: np = ip[nr - 1] + 1 and raid_gen() can generate only a prefix of parity rows. * Example: nr = 3 ip = { 0, 2, 5 } requires GEN6 today even though only P, R and U are needed. * A fused RECX could calculate exactly those three syndromes. * The idea is technically valid and could reduce temporary memory traffic and unnecessary parity work. * However, the common rec1of1/P case already bypasses raid_delta_gen() entirely. * Because the main benefit would apply to less common multiple-failure or missing-P cases, this direction was intentionally deprioritized. * Decision: do not pursue raid_delta_gen() + REC fusion for now. Generation: AVX512 width and unrolling -------------------------------------- * Wider processing is not automatically faster. * AVX512 128-byte processing was tested and can be slower than the existing 64-byte path. * Therefore do not assume that x2/x4 spatial unrolling, like some Linux RAID6 kernels use, is beneficial for SnapRAID. * The block size is already large, memory traffic is substantial, and wider unrolling can increase register pressure, front-end work, and AVX512 power/ frequency effects without reducing total memory traffic. * VPTERNLOGQ based alternatives were also tested on Zen 5 and did not improve performance. * Conclusion: keep the current conservative spatial width unless a specific benchmark proves otherwise. Generation: Horner ------------------ * Linux RAID6 uses the classic Horner recurrence for Q with the standard RAID polynomial/generator. * A Horner-style AVX512BW non-GFNI implementation was tested on Zen 5 and did not improve performance. * Therefore Horner is NOT considered an open optimization for the AVX512BW non-GFNI generation path. * GFNI is different. * For RAID GFNI generation, Horner Q is useful because multiplying by the fixed generator can use one fixed affine matrix instead of a different matrix for every data coefficient. * RAID GFNI Horner Q was implemented as follows: AVX2 RAID GFNI GEN2..GEN5: Horner Q with fixed multiply-by-2 affine matrix resident in a register. AVX2 RAID GFNI GEN6: keep the older direct-coefficient implementation because register pressure is too high to keep the extra fixed matrix efficiently. AVX512 RAID GFNI GEN2..GEN6: Horner Q with the fixed matrix resident in a register. AES GFNI: unchanged, direct VGF2P8MULB coefficient multiplication. * D0/final coefficient 1 is handled directly where appropriate. * Conclusion: GFNI Horner Q is implemented where useful. Non-GFNI AVX512BW Horner was tested and rejected. GFNI field implementation ------------------------- * AES GFNI multiplication maps naturally to VGF2P8MULB because the instruction uses the AES polynomial. * Standard RAID polynomial 0x11d cannot use VGF2P8MULB directly. * RAID GFNI therefore uses VGF2P8AFFINEQB with precomputed affine matrices. * The affine matrix table stores 8 bytes per coefficient. * A generic exhaustive software test validates all coefficient/input pairs against the normal RAID GF multiplication table. * Dedicated hardware affine-matrix validators were intentionally not added. Normal generation/recovery tests exercise the hardware path. * RAID and AES GFNI recovery/generation have polynomial-specific names: *_gfni_raid *_gfni_aes with backend tags: gfni gfni512 * This naming and registration split is intentional and should be preserved. GFNI pre-expanded affine tables ------------------------------- * Intel ISA-L can use pre-expanded GFNI constants, for example a 32-byte representation that is ready for vector use. * For SnapRAID this would increase the RAID affine table from about 2 KiB to about 8 KiB. * With large 256 KiB processing blocks, the current small 8-byte entries plus broadcast have a better cache-footprint tradeoff. * No convincing reason was found to trade four times the table footprint for fewer broadcasts. * Decision: keep compact 8-byte affine matrices. Intel ISA-L comparison ---------------------- The following ISA-L ideas were examined. 1. Horner Q for GFNI RAID generation Useful idea. Implemented for RAID GFNI where register pressure allows it. 2. AVX512 GFNI 128-byte processing Tested. It can be slower. Do not use wider processing simply because more ZMM registers are available. 3. Source-major generic recovery loops Initially interesting because one source could feed several outputs. For the current RECX structure, the delta/syndrome working set is tiny and hot. Reordering loops does not obviously save important memory traffic and can increase register pressure. This is considered low priority / not useful enough. 4. Split output processing, for example 3+3 rows Bad fit for SnapRAID 256 KiB blocks because it requires rereading the source range in another pass. Avoid optimizations that save registers by adding full source passes. 5. Pre-expanded 32-byte GFNI constants Rejected because of the larger table/cache footprint described above. 6. VPBLENDVB based x2 techniques Tested. No benefit / worse performance. 7. Cached parity stores ISA-L style normal stores are not automatically suitable for SnapRAID. SnapRAID has large output blocks and a real queued disk-write pipeline. MOVNT remains the better design choice for this workload. 8. Multiple-row GFNI macros Interesting in isolation, but not enough reason to reread 256 KiB source blocks or enlarge table footprint. 9. Broad runtime dispatch/calibration ISA-L has broad dispatch machinery. SnapRAID already dispatches based on CPU features. A runtime microbenchmark selector is possible, but is not currently considered worth the extra complexity for the known kernels. 10. Incremental parity APIs Useful in other applications, but not a missing optimization in the current full-block SnapRAID parity-generation hot path. Overall ISA-L conclusion: * The important transferable idea was GFNI Horner Q, and that is already implemented. * The other ISA-L techniques examined either lose on the current benchmark, increase source rereads, increase cache footprint, or do not match the real SnapRAID workload. Linux kernel RAID6 comparison ----------------------------- The Linux RAID6 implementation was studied as another optimization reference. Relevant Linux characteristics include: * x1/x2/x4 spatial kernels * wider AVX512 processing * explicit prefetch * independent chains for ILP * classic Horner Q * runtime generator selection * xor_syndrome/incremental interfaces * specialized P+Q two-disk recovery Conclusions for SnapRAID: * Wider x2/x4 AVX512 processing is not automatically useful. 128-byte variants were tested and can be slower. * Horner AVX512BW non-GFNI was tested on Zen 5 and did not improve. * VPTERNLOGQ alternatives were tested on Zen 5 and did not improve. * Explicit software prefetch is not currently considered a strong candidate. The workload already consists of long regular streams, and the existing memory-layout tuning is important. * Runtime microbenchmark-based backend selection would add complexity and is low priority. * xor_syndrome/incremental parity is a different API/use case and does not directly improve the current full-block generation path. * The one Linux-style algorithmic recovery optimization still worth keeping as a TODO is specialized P+Q two-data recovery (rec2of2), but its practical priority is low because two simultaneous data failures are rare. Memory layout, displacement and stride noise -------------------------------------------- * The memory layout is not a neutral detail. * Fixed displacement between buffers was observed to help. * STRIDE_NOISE behavior differs by CPU. * On Intel, stride noise had little effect in the observed tests. * On Zen 5, disabling stride noise caused a large throughput reduction. * This indicates that hardware prefetch/cache behavior is already strongly affected by the chosen buffer layout. * Therefore new software-prefetch or traversal changes should not be assumed to help. They must be benchmarked against the existing displacement/stride behavior. * This is another reason to avoid adding extra passes over 256 KiB blocks. Non-GFNI REC2 table hoisting ---------------------------- A concrete successful recovery micro-optimization was implemented. Before: * multiplication shuffle tables could be broadcast repeatedly inside the 32/64-byte loop * delta low/high nibbles could be recomputed for different outputs After: AVX2 x86-64: * keep the 8 low/high shuffle tables for V0..V3 resident in ymm8..ymm15 * keep the low-nibble mask resident * split d0/d1 once per chunk * reuse the split values for both recovered outputs AVX512BW: * keep the 8 low/high shuffle tables resident in zmm16..zmm23 * keep the low-nibble mask resident * split d0/d1 once per chunk * reuse them for both outputs Important portability result: * The first AVX2 version used ymm8..ymm15 unconditionally. * That is invalid for 32-bit x86, where only ymm0..ymm7 are available. * The final implementation keeps the optimized resident-table path only under CONFIG_X86_64 and preserves the old REC2 body for CONFIG_X86_32. * AVX512BW is already x86-64-only, so zmm16..zmm23 are safe there. This is the preferred class of future recovery optimization: same algorithm, same stride, same memory traffic, fewer repeated loads/broadcasts and better register reuse. Recovery optimization policy from now on ----------------------------------------- For recovery, avoid new architecture-level changes unless a very strong case appears. Preferred optimizations are basic/local: * hoist loop-invariant values * preload coefficient tables into registers * keep masks/matrices resident when register pressure allows it * reuse low/high nibble splits * avoid repeated loads of the same hot temporary value * avoid repeated broadcasts * initialize accumulators from the first useful term when possible instead of zeroing and XORing unnecessarily * exploit coefficient 0 or 1 only when it is simple and does not add harmful branching/complexity Avoid for now: * raid_delta_gen() fusion * extra full passes over source blocks * aggressive spatial unrolling * software prefetch without clear evidence * changing MOVNT policy * major loop-orientation changes * large pre-expanded lookup tables REC1 status ----------- * The important normal case is already raid_rec1of1() with P. * That path is essentially XOR-only and close to the minimum possible memory traffic. * There is little algorithmic work left to remove. * More elaborate REC1 work is mainly relevant when P itself is unavailable and Q/R/S/T/U must be used. * Those cases are much less common than the normal single-disk + P recovery. REC2 status ----------- * Generic non-GFNI REC2 table/broadcast hoisting is implemented. * A specialized rec2of2 P+Q formula remains a TODO: Dy = C0 * Pd ^ C1 * Qd Dx = Pd ^ Dy instead of the generic 2x2 solution with four GF multiplications. * This should be implemented, if desired, for AVX2, AVX512 and GFNI. * Practical priority is low because simultaneous loss of two data disks is an exceptional case. RECX status ----------- * RECX is the main remaining place to look for basic register-reuse optimizations. * The interesting questions are local: can pd[k] stay in a register longer? can low/high nibble splits be reused across outputs? can some multiplication tables remain resident? can repeated broadcasts be removed? * AVX512 has enough registers to make some of these experiments plausible. * AVX2 is much more constrained, so any attempt must be careful not to trade a few broadcasts for spills or a second source pass. * Source-major loop inversion is not currently considered a priority because the pd working set is tiny/hot and the register-pressure tradeoff is not obviously favorable. Generation status ----------------- For current 1..6 parity generation, performance work is considered largely closed. Already covered: * scalar/int paths * SSE/SSSE3/AVX2/AVX512 paths * GFNI RAID affine generation * GFNI AES direct multiplication * RAID GFNI Horner Q * AVX512 width experiments * VPTERNLOGQ experiments * VPBLENDVB experiments * MOVNT policy * memory displacement/stride behavior * ISA-L comparisons * Linux RAID6 comparisons No major generation optimization remains that currently has a strong reason to be tested. More than 6 parity levels ------------------------- * Supporting more than 6 parity levels remains a future feature. * A generic implementation using SSSE3/AVX intrinsics could be compared with the current handwritten assembler style. * This is primarily a functional extension, not an unresolved performance issue for the current 1..6 parity library. Current practical priority -------------------------- 1. Recovery only: continue looking for small invariant-hoisting/register-reuse optimizations. 2. RECX AVX512BW is the most natural place to inspect for repeated load/split/broadcast work. 3. Keep specialized rec2of2 P+Q as a correctness/performance TODO, but with low real-world priority. 4. Do not reopen generation-width, MOVNT, Horner AVX512BW, VPTERNLOGQ, VPBLENDVB, large GFNI-table, source-reread, or raid_delta_gen-fusion ideas unless new hardware or new measurements provide a concrete reason. Short list of ideas already tried or rejected --------------------------------------------- * AVX512 128-byte processing: tried, can be slower. * AVX512BW Horner: tried on Zen 5, no improvement. * VPTERNLOGQ optimization: tried on Zen 5, no improvement. * VPBLENDVB x2: tried, no benefit / worse. * MOVNT -> cached stores: not appropriate for the normal queued disk-write workload; keep MOVNT. * GFNI 32-byte pre-expanded affine constants: rejected due to larger cache footprint. * split GFNI outputs into multiple passes: rejected because 256 KiB source blocks would be reread. * RECX source-major rewrite: low priority; hot tiny pd working set and register-pressure tradeoff. * raid_delta_gen() + REC fusion: technically valid but intentionally deprioritized because the normal rec1of1/P case bypasses raid_delta_gen() already. * software prefetch: no strong reason to add it given the existing long streams and proven sensitivity to displacement/stride tuning. * runtime backend microbenchmark selection: possible, but low value relative to complexity. * Linux x2/x4 style wide spatial kernels: not assumed useful; wider AVX512 already tested negatively. * ISA-L cached-store behavior: not a good match for the real SnapRAID parity queue. * ISA-L source reread / split-output strategies: poor fit for 256 KiB blocks. * specialized P+Q rec2of2: still open, but low practical priority. End state --------- For the current 1..6 parity RAID library: Generation: effectively closed GFNI generation: effectively closed Store policy: closed, keep MOVNT Memory layout: tuned; do not casually change Recovery: only basic/local micro-optimizations remain rec2of2 P+Q: open TODO, low real-world priority >6 parity: future feature The most important workload fact to remember is that the normal recovery is a single failed data disk with P available, which goes through raid_rec1of1() and is already a direct XOR-style recovery with minimal extra work.