diff --git a/raid/NOTES b/raid/NOTES new file mode 100644 index 00000000..9e0d2d92 --- /dev/null +++ b/raid/NOTES @@ -0,0 +1,652 @@ +SnapRAID RAID library performance notes +======================================= + +Scope +----- + +These notes summarize performance experiments, implementation choices, rejected +ideas, and non-obvious conclusions reached while reviewing and tuning the +SnapRAID RAID library. + +The focus is the current 1..6 parity implementation, especially x86 AVX2, +AVX512BW and GFNI backends. + +This is intentionally a record of what has already been tried and what was +learned from it, so that the same ideas are not repeatedly re-investigated. + + +Real workload assumptions +------------------------- + +* SnapRAID processes large blocks: typically 256 KiB per data/parity block. + +* A normal stripe therefore touches a large amount of memory. With 12 data + disks, one 256 KiB stripe already means about 3 MiB of data input before + considering parity output. + +* This large-block behavior is very important. Optimizations that reread a + source buffer, increase the number of passes, or enlarge lookup tables can be + worse even if the inner instruction sequence looks better. + +* SnapRAID uses a queue before parity data is written to disk. Real measurements + with actual disk writes show that very small queues are worse. Performance + improves clearly up to about queue depth 32, and still improves after that, + but with diminishing returns. + +* With 256 KiB blocks, queue depth 32 means this much parity output can be in + flight: + + 1 parity = 8 MiB + 2 parity = 16 MiB + 3 parity = 24 MiB + 6 parity = 48 MiB + + Therefore the parity block produced by the RAID kernel is normally not + consumed immediately. + +* A memory read of a 256 KiB block that is no longer cached is still very cheap + compared with real storage I/O. Trying to keep many queued parity blocks hot + in cache is therefore not necessarily useful. + + +MOVNT / non-temporal stores +--------------------------- + +* Generation backends use MOVNTDQ / VMOVNTDQ for parity output. + +* The synthetic speedtest naturally favors non-temporal stores because the + generated parity is not consumed afterwards. This benchmark therefore + measures producer throughput, not the complete producer -> queue -> writer + pipeline. + +* However, the real SnapRAID queue makes the workload much closer to the + non-temporal case than to an immediate-consumer case. + +* Cached stores could potentially help only with very short queues, where the + writer consumes the parity while it is still hot. Queue depths around 1..4 + are the most plausible area where this could happen. + +* Those very short queues are not representative of the real fastest disk + pipeline. Real measurements already show that queues up to about 32 provide + substantial improvements. + +* At realistic queue depths, cached parity output would occupy many MiB of + cache and could evict useful input data. MOVNT avoids this cache pollution and + avoids normal cache allocation/write-allocate behavior for the parity output. + +* Conclusion: keep MOVNT for the normal generation path. We do not consider + replacing MOVNT with cached stores an open optimization anymore. + + +Typical recovery case +--------------------- + +* The common recovery case is one failed data disk, while parity P is still + available. + +* raid_rec() selects the first available parity. Therefore, for a single data + failure with P available, recovery uses P. + +* raid_rec1_avx2()/raid_rec1_avx512bw() special-case ip[0] == 0 and call + raid_rec1of1(). + +* raid_rec1of1() is already effectively fused. It remaps pointers so that P is + used as one of the XOR inputs and the failed data buffer is used as GEN1 + output, then calls raid_gen(nd, 1, ...). + +* Therefore the common single-disk recovery path does NOT call + raid_delta_gen(). + +* The data traffic is already close to the minimum: + + read P + read all surviving data blocks + XOR them + write the recovered data block once + +* There are no temporary syndrome buffers, no GF multiplication, and no + matrix inversion in this common path. + +* This means that large architectural recovery changes mostly optimize rare + failure cases, not the usual single-disk recovery. + +* Two simultaneous data disk failures are already an exceptional case in + normal use. Three or more simultaneous data failures are even less important + as optimization targets. + + +raid_delta_gen() fusion +----------------------- + +* A fused recovery design was studied, where the selected syndromes would be + accumulated directly instead of: + + raid_delta_gen() + temporary parity output + reload temporary parity + solve recovery matrix + +* The fused design could also compute only the parity rows actually selected by + ip[]. + +* This matters because raid_delta_gen() currently computes: + + np = ip[nr - 1] + 1 + + and raid_gen() can generate only a prefix of parity rows. + +* Example: + + nr = 3 + ip = { 0, 2, 5 } + + requires GEN6 today even though only P, R and U are needed. + +* A fused RECX could calculate exactly those three syndromes. + +* The idea is technically valid and could reduce temporary memory traffic and + unnecessary parity work. + +* However, the common rec1of1/P case already bypasses raid_delta_gen() entirely. + +* Because the main benefit would apply to less common multiple-failure or + missing-P cases, this direction was intentionally deprioritized. + +* Decision: do not pursue raid_delta_gen() + REC fusion for now. + + +Generation: AVX512 width and unrolling +-------------------------------------- + +* Wider processing is not automatically faster. + +* AVX512 128-byte processing was tested and can be slower than the existing + 64-byte path. + +* Therefore do not assume that x2/x4 spatial unrolling, like some Linux RAID6 + kernels use, is beneficial for SnapRAID. + +* The block size is already large, memory traffic is substantial, and wider + unrolling can increase register pressure, front-end work, and AVX512 power/ + frequency effects without reducing total memory traffic. + +* VPTERNLOGQ based alternatives were also tested on Zen 5 and did not improve + performance. + +* Conclusion: keep the current conservative spatial width unless a specific + benchmark proves otherwise. + + +Generation: Horner +------------------ + +* Linux RAID6 uses the classic Horner recurrence for Q with the standard RAID + polynomial/generator. + +* A Horner-style AVX512BW non-GFNI implementation was tested on Zen 5 and did + not improve performance. + +* Therefore Horner is NOT considered an open optimization for the AVX512BW + non-GFNI generation path. + +* GFNI is different. + +* For RAID GFNI generation, Horner Q is useful because multiplying by the fixed + generator can use one fixed affine matrix instead of a different matrix for + every data coefficient. + +* RAID GFNI Horner Q was implemented as follows: + + AVX2 RAID GFNI GEN2..GEN5: + Horner Q with fixed multiply-by-2 affine matrix resident in a register. + + AVX2 RAID GFNI GEN6: + keep the older direct-coefficient implementation because register + pressure is too high to keep the extra fixed matrix efficiently. + + AVX512 RAID GFNI GEN2..GEN6: + Horner Q with the fixed matrix resident in a register. + + AES GFNI: + unchanged, direct VGF2P8MULB coefficient multiplication. + +* D0/final coefficient 1 is handled directly where appropriate. + +* Conclusion: GFNI Horner Q is implemented where useful. Non-GFNI AVX512BW + Horner was tested and rejected. + + +GFNI field implementation +------------------------- + +* AES GFNI multiplication maps naturally to VGF2P8MULB because the instruction + uses the AES polynomial. + +* Standard RAID polynomial 0x11d cannot use VGF2P8MULB directly. + +* RAID GFNI therefore uses VGF2P8AFFINEQB with precomputed affine matrices. + +* The affine matrix table stores 8 bytes per coefficient. + +* A generic exhaustive software test validates all coefficient/input pairs + against the normal RAID GF multiplication table. + +* Dedicated hardware affine-matrix validators were intentionally not added. + Normal generation/recovery tests exercise the hardware path. + +* RAID and AES GFNI recovery/generation have polynomial-specific names: + + *_gfni_raid + *_gfni_aes + + with backend tags: + + gfni + gfni512 + +* This naming and registration split is intentional and should be preserved. + + +GFNI pre-expanded affine tables +------------------------------- + +* Intel ISA-L can use pre-expanded GFNI constants, for example a 32-byte + representation that is ready for vector use. + +* For SnapRAID this would increase the RAID affine table from about 2 KiB to + about 8 KiB. + +* With large 256 KiB processing blocks, the current small 8-byte entries plus + broadcast have a better cache-footprint tradeoff. + +* No convincing reason was found to trade four times the table footprint for + fewer broadcasts. + +* Decision: keep compact 8-byte affine matrices. + + +Intel ISA-L comparison +---------------------- + +The following ISA-L ideas were examined. + +1. Horner Q for GFNI RAID generation + + Useful idea. Implemented for RAID GFNI where register pressure allows it. + +2. AVX512 GFNI 128-byte processing + + Tested. It can be slower. Do not use wider processing simply because more + ZMM registers are available. + +3. Source-major generic recovery loops + + Initially interesting because one source could feed several outputs. + + For the current RECX structure, the delta/syndrome working set is tiny and + hot. Reordering loops does not obviously save important memory traffic and + can increase register pressure. + + This is considered low priority / not useful enough. + +4. Split output processing, for example 3+3 rows + + Bad fit for SnapRAID 256 KiB blocks because it requires rereading the source + range in another pass. + + Avoid optimizations that save registers by adding full source passes. + +5. Pre-expanded 32-byte GFNI constants + + Rejected because of the larger table/cache footprint described above. + +6. VPBLENDVB based x2 techniques + + Tested. No benefit / worse performance. + +7. Cached parity stores + + ISA-L style normal stores are not automatically suitable for SnapRAID. + SnapRAID has large output blocks and a real queued disk-write pipeline. + MOVNT remains the better design choice for this workload. + +8. Multiple-row GFNI macros + + Interesting in isolation, but not enough reason to reread 256 KiB source + blocks or enlarge table footprint. + +9. Broad runtime dispatch/calibration + + ISA-L has broad dispatch machinery. SnapRAID already dispatches based on CPU + features. A runtime microbenchmark selector is possible, but is not currently + considered worth the extra complexity for the known kernels. + +10. Incremental parity APIs + + Useful in other applications, but not a missing optimization in the current + full-block SnapRAID parity-generation hot path. + +Overall ISA-L conclusion: + +* The important transferable idea was GFNI Horner Q, and that is already + implemented. + +* The other ISA-L techniques examined either lose on the current benchmark, + increase source rereads, increase cache footprint, or do not match the real + SnapRAID workload. + + +Linux kernel RAID6 comparison +----------------------------- + +The Linux RAID6 implementation was studied as another optimization reference. + +Relevant Linux characteristics include: + +* x1/x2/x4 spatial kernels +* wider AVX512 processing +* explicit prefetch +* independent chains for ILP +* classic Horner Q +* runtime generator selection +* xor_syndrome/incremental interfaces +* specialized P+Q two-disk recovery + +Conclusions for SnapRAID: + +* Wider x2/x4 AVX512 processing is not automatically useful. 128-byte variants + were tested and can be slower. + +* Horner AVX512BW non-GFNI was tested on Zen 5 and did not improve. + +* VPTERNLOGQ alternatives were tested on Zen 5 and did not improve. + +* Explicit software prefetch is not currently considered a strong candidate. + The workload already consists of long regular streams, and the existing + memory-layout tuning is important. + +* Runtime microbenchmark-based backend selection would add complexity and is + low priority. + +* xor_syndrome/incremental parity is a different API/use case and does not + directly improve the current full-block generation path. + +* The one Linux-style algorithmic recovery optimization still worth keeping as + a TODO is specialized P+Q two-data recovery (rec2of2), but its practical + priority is low because two simultaneous data failures are rare. + + +Memory layout, displacement and stride noise +-------------------------------------------- + +* The memory layout is not a neutral detail. + +* Fixed displacement between buffers was observed to help. + +* STRIDE_NOISE behavior differs by CPU. + +* On Intel, stride noise had little effect in the observed tests. + +* On Zen 5, disabling stride noise caused a large throughput reduction. + +* This indicates that hardware prefetch/cache behavior is already strongly + affected by the chosen buffer layout. + +* Therefore new software-prefetch or traversal changes should not be assumed to + help. They must be benchmarked against the existing displacement/stride + behavior. + +* This is another reason to avoid adding extra passes over 256 KiB blocks. + + +Non-GFNI REC2 table hoisting +---------------------------- + +A concrete successful recovery micro-optimization was implemented. + +Before: + +* multiplication shuffle tables could be broadcast repeatedly inside the + 32/64-byte loop +* delta low/high nibbles could be recomputed for different outputs + +After: + +AVX2 x86-64: + +* keep the 8 low/high shuffle tables for V0..V3 resident in ymm8..ymm15 +* keep the low-nibble mask resident +* split d0/d1 once per chunk +* reuse the split values for both recovered outputs + +AVX512BW: + +* keep the 8 low/high shuffle tables resident in zmm16..zmm23 +* keep the low-nibble mask resident +* split d0/d1 once per chunk +* reuse them for both outputs + +Important portability result: + +* The first AVX2 version used ymm8..ymm15 unconditionally. + +* That is invalid for 32-bit x86, where only ymm0..ymm7 are available. + +* The final implementation keeps the optimized resident-table path only under + CONFIG_X86_64 and preserves the old REC2 body for CONFIG_X86_32. + +* AVX512BW is already x86-64-only, so zmm16..zmm23 are safe there. + +This is the preferred class of future recovery optimization: +same algorithm, same stride, same memory traffic, fewer repeated loads/broadcasts +and better register reuse. + + +Recovery optimization policy from now on +----------------------------------------- + +For recovery, avoid new architecture-level changes unless a very strong case +appears. + +Preferred optimizations are basic/local: + +* hoist loop-invariant values +* preload coefficient tables into registers +* keep masks/matrices resident when register pressure allows it +* reuse low/high nibble splits +* avoid repeated loads of the same hot temporary value +* avoid repeated broadcasts +* initialize accumulators from the first useful term when possible instead of + zeroing and XORing unnecessarily +* exploit coefficient 0 or 1 only when it is simple and does not add harmful + branching/complexity + +Avoid for now: + +* raid_delta_gen() fusion +* extra full passes over source blocks +* aggressive spatial unrolling +* software prefetch without clear evidence +* changing MOVNT policy +* major loop-orientation changes +* large pre-expanded lookup tables + + +REC1 status +----------- + +* The important normal case is already raid_rec1of1() with P. + +* That path is essentially XOR-only and close to the minimum possible memory + traffic. + +* There is little algorithmic work left to remove. + +* More elaborate REC1 work is mainly relevant when P itself is unavailable and + Q/R/S/T/U must be used. + +* Those cases are much less common than the normal single-disk + P recovery. + + +REC2 status +----------- + +* Generic non-GFNI REC2 table/broadcast hoisting is implemented. + +* A specialized rec2of2 P+Q formula remains a TODO: + + Dy = C0 * Pd ^ C1 * Qd + Dx = Pd ^ Dy + + instead of the generic 2x2 solution with four GF multiplications. + +* This should be implemented, if desired, for AVX2, AVX512 and GFNI. + +* Practical priority is low because simultaneous loss of two data disks is an + exceptional case. + + +RECX status +----------- + +* RECX is the main remaining place to look for basic register-reuse + optimizations. + +* The interesting questions are local: + + can pd[k] stay in a register longer? + can low/high nibble splits be reused across outputs? + can some multiplication tables remain resident? + can repeated broadcasts be removed? + +* AVX512 has enough registers to make some of these experiments plausible. + +* AVX2 is much more constrained, so any attempt must be careful not to trade + a few broadcasts for spills or a second source pass. + +* Source-major loop inversion is not currently considered a priority because + the pd working set is tiny/hot and the register-pressure tradeoff is not + obviously favorable. + + +Generation status +----------------- + +For current 1..6 parity generation, performance work is considered largely +closed. + +Already covered: + +* scalar/int paths +* SSE/SSSE3/AVX2/AVX512 paths +* GFNI RAID affine generation +* GFNI AES direct multiplication +* RAID GFNI Horner Q +* AVX512 width experiments +* VPTERNLOGQ experiments +* VPBLENDVB experiments +* MOVNT policy +* memory displacement/stride behavior +* ISA-L comparisons +* Linux RAID6 comparisons + +No major generation optimization remains that currently has a strong reason to +be tested. + + +More than 6 parity levels +------------------------- + +* Supporting more than 6 parity levels remains a future feature. + +* A generic implementation using SSSE3/AVX intrinsics could be compared with + the current handwritten assembler style. + +* This is primarily a functional extension, not an unresolved performance issue + for the current 1..6 parity library. + + +Current practical priority +-------------------------- + +1. Recovery only: continue looking for small invariant-hoisting/register-reuse + optimizations. + +2. RECX AVX512BW is the most natural place to inspect for repeated + load/split/broadcast work. + +3. Keep specialized rec2of2 P+Q as a correctness/performance TODO, but with low + real-world priority. + +4. Do not reopen generation-width, MOVNT, Horner AVX512BW, VPTERNLOGQ, + VPBLENDVB, large GFNI-table, source-reread, or raid_delta_gen-fusion ideas + unless new hardware or new measurements provide a concrete reason. + + +Short list of ideas already tried or rejected +--------------------------------------------- + +* AVX512 128-byte processing: + tried, can be slower. + +* AVX512BW Horner: + tried on Zen 5, no improvement. + +* VPTERNLOGQ optimization: + tried on Zen 5, no improvement. + +* VPBLENDVB x2: + tried, no benefit / worse. + +* MOVNT -> cached stores: + not appropriate for the normal queued disk-write workload; keep MOVNT. + +* GFNI 32-byte pre-expanded affine constants: + rejected due to larger cache footprint. + +* split GFNI outputs into multiple passes: + rejected because 256 KiB source blocks would be reread. + +* RECX source-major rewrite: + low priority; hot tiny pd working set and register-pressure tradeoff. + +* raid_delta_gen() + REC fusion: + technically valid but intentionally deprioritized because the normal + rec1of1/P case bypasses raid_delta_gen() already. + +* software prefetch: + no strong reason to add it given the existing long streams and proven + sensitivity to displacement/stride tuning. + +* runtime backend microbenchmark selection: + possible, but low value relative to complexity. + +* Linux x2/x4 style wide spatial kernels: + not assumed useful; wider AVX512 already tested negatively. + +* ISA-L cached-store behavior: + not a good match for the real SnapRAID parity queue. + +* ISA-L source reread / split-output strategies: + poor fit for 256 KiB blocks. + +* specialized P+Q rec2of2: + still open, but low practical priority. + + +End state +--------- + +For the current 1..6 parity RAID library: + + Generation: effectively closed + GFNI generation: effectively closed + Store policy: closed, keep MOVNT + Memory layout: tuned; do not casually change + Recovery: only basic/local micro-optimizations remain + rec2of2 P+Q: open TODO, low real-world priority + >6 parity: future feature + +The most important workload fact to remember is that the normal recovery is a +single failed data disk with P available, which goes through raid_rec1of1() and +is already a direct XOR-style recovery with minimal extra work.