Optimize scalar 8-bit RAID parity generation and recovery in int.c

This commit is contained in:
Andrea Mazzoleni 2026-08-27 19:19:31 +02:00
parent 69a1ed1125
commit e0c0ea3dcc
7 changed files with 777 additions and 228 deletions

View file

@ -45,6 +45,9 @@ Generation:
* RAID GFNI generation uses direct Cauchy coefficients. Their 8-byte affine
representations are precomputed in a disk-major table, avoiding a dependent
Cauchy-coefficient -> affine-matrix lookup in the hot generation loop.
Similarly, scalar int8 generation precomputes per-disk 2D multiplication table
pointers T[d][p] before the hot loop to eliminate two-level dependent table
lookups.
* AVX512 128-byte processing, non-GFNI AVX512BW Horner, standalone
VPTERNLOGQ transformations, VPBLENDVB, cached parity stores, and large
@ -69,8 +72,8 @@ Recovery:
AArch32 NEON uses fused syndrome generation for all non-P REC1 and REC2 cases
and for REC3..REC6. SSSE3 and SSSE3 extended use it for REC3..REC6.
* Scalar recovery, SSSE3 REC1/REC2, and AArch32 NEON P+Q continue to use
raid_delta_gen().
* Scalar REC1/REC2, SSSE3 REC1/REC2, and AArch32 NEON P+Q continue to use
raid_delta_gen(), while scalar REC3..REC6 uses fused syndrome recovery.
* Non-GFNI fused recovery splits every completed syndrome into low/high nibbles
once and reuses the split values across all reconstructed outputs.
@ -86,13 +89,14 @@ Recovery:
* Specialized P+Q rec2of2 recovery reduces 4 GF multiplications to 2 GF
multiplications and 1 XOR; AVX512BW, GFNI, AVX2, and AArch64 NEON handle this
directly inside the 1-pass fused kernel, while scalar int8, SSSE3, SSSE3ext,
and AArch32 NEON backends implement dedicated rec2of2 helpers.
directly inside the 1-pass fused kernel, while SSSE3, SSSE3ext,
and AArch32 NEON backends implement dedicated rec2of2 helpers, and scalar int8
handles it via has_p in raid_rec2_int8.
* When P is among the selected parity blocks, optimized SIMD RECX paths
reconstruct only nr - 1 missing blocks through the inverse matrix and derive
the final missing block from the raw P delta by XORing the blocks already
reconstructed.
* When P is among the selected parity blocks, optimized SIMD and scalar RECX
paths reconstruct only nr - 1 missing blocks through the inverse matrix and
derive the final missing block from the raw P delta by XORing the blocks
already reconstructed.
Architecture and validation:
@ -251,8 +255,8 @@ REC2OF2:
directly in the 1-pass fused kernel, which computes P with XOR, Q with GF
multiplication, and derives the second missing block via XOR.
* Scalar int8, SSSE3, SSSE3ext, and AArch32 NEON call their specialized
raid_rec2of2_*() delta helpers.
* SSSE3, SSSE3ext, and AArch32 NEON call their specialized
raid_rec2of2_*() delta helpers, while scalar int8 leverages has_p in raid_rec2_int8.
* This makes the row a direct benchmark for the specialized P+Q recovery
implementation.
@ -271,8 +275,8 @@ REC2OF3 and REC2OF4:
REC3OF3 through REC6OF6:
* These rows measure multi-failure recovery for each supported failure count
using the first N parity levels. SIMD columns use fused recovery; the scalar
column retains raid_delta_gen().
using the first N parity levels. SIMD backends and scalar int8 use fused
syndrome recovery.
Delta generation:
@ -316,7 +320,7 @@ raid_delta_gen() fusion
XOR streaming within the fused kernel (no GF matrix multiplication), while on
scalar, SSSE3, and AVX2 it uses the raid_rec1of1() pointer remapping path.
* raid_delta_gen() remains in the scalar implementation, SSSE3 REC1/REC2, and
* raid_delta_gen() remains in scalar REC1/REC2, SSSE3 REC1/REC2, and
AArch32 NEON P+Q.
@ -390,6 +394,11 @@ Generation: GFNI direct coefficients
difference is VGF2P8AFFINEQB with an 8-byte RAID matrix versus
VGF2P8MULB with an AES coefficient byte.
* Scalar int8 generation (raid_gen2_int8 .. raid_gen6_int8) similarly precomputes
per-disk multiplication table pointers T[d][p] in a 2D array before entering
the hot loop, replacing the double-indexed raid_gfmul[d0][raid_gfgen[j][d]]
lookup with a single direct table access T[d][p][d0].
Generation: coefficient residency
---------------------------------
@ -695,8 +704,8 @@ REC2 status:
* Generic REC2 uses the fused path across AVX2, AVX512BW, GFNI, and NEON.
* AVX2, AVX512BW, GFNI, and AArch64 NEON handle P+Q recovery directly inside the
1-pass fused kernel, while scalar int8, SSSE3, SSSE3ext, and AArch32 NEON
backends implement dedicated rec2of2 delta helpers.
1-pass fused kernel, while SSSE3, SSSE3ext, and AArch32 NEON backends implement
dedicated rec2of2 delta helpers, and scalar int8 handles it via has_p.
Specialized rec2of2 P+Q
@ -717,7 +726,7 @@ Backends implement this specialization as follows:
derivation directly into the unified 1-pass fused kernel (P is accumulated via
XOR, Q via GF multiplication, Dy is solved with inverse coefficients, and Dx
derived by XOR with Pdelta).
* Scalar int8 (raid_rec2of2_int8)
* Scalar int8 (has_p branch in raid_rec2_int8)
* SSSE3 and SSSE3 ext (raid_rec2of2_ssse3, raid_rec2of2_ssse3ext)
* AArch32 NEON (raid_rec2of2_neon32)
@ -782,6 +791,10 @@ Users include:
* AVX512BW RECX
* SSSE3 REC3..REC6
* AArch64 and AArch32 NEON REC1..REC6
* Scalar int8 RECX (raid_recX_int8), which precomputes 256-byte raid_gfmul
pointers S[s][j] and R[j][k], and structures inner loops with explicit
if (nr >= N) branches so the compiler fully unrolls syndrome accumulation
and inverse-matrix reconstruction when nr is a compile-time constant.
The remaining cost is that the low/high tables are still loaded or broadcast for
every coefficient of every data chunk.
@ -1032,9 +1045,10 @@ Short list of ideas already tried or rejected
existing traversal instead.
* raid_delta_gen() + REC fusion:
implemented for all SIMD REC1..REC6 on AVX512BW, GFNI, and AArch64 NEON, and for
all AVX2 REC except rec1of1; SSSE3 retains delta for REC1/REC2; only scalar
paths, SSSE3 REC1/REC2, and AArch32 NEON P+Q retain raid_delta_gen().
implemented for all SIMD REC1..REC6 on AVX512BW, GFNI, and AArch64 NEON, for
all AVX2 REC except rec1of1, and for scalar int8 REC3..REC6; SSSE3 retains delta
for REC1/REC2; only scalar REC1/REC2, SSSE3 REC1/REC2, and AArch32 NEON P+Q retain
raid_delta_gen().
* Software prefetch:
no strong reason to add it given the existing long streams and proven