diff --git a/raid/NOTES b/raid/NOTES index dccc9ad7..c32c482c 100644 --- a/raid/NOTES +++ b/raid/NOTES @@ -56,7 +56,7 @@ Recovery: * The normal one-data-disk recovery with P available is handled directly by the fused 1-pass SIMD kernel on AVX512BW, GFNI, and AArch64 NEON (pure XOR - streaming), or by raid_rec1of1() on scalar and older SIMD paths. + streaming), or by raid_rec1of1() on scalar, SSSE3, and AVX2. * SIMD recovery has parity-count-specific REC1..REC6 entry points. Fused paths compute exactly the selected syndromes in one scan of the surviving data and @@ -64,11 +64,12 @@ Recovery: * AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON use fused syndrome generation for all REC1..REC6 cases, including P (rec1of1) and P+Q (rec2of2). + AVX2 uses fused syndrome generation for all non-P REC1 cases and all + REC2..REC6 cases (including P+Q in the fused kernel). AArch32 NEON uses fused syndrome generation for all non-P REC1 and REC2 cases - and for REC3..REC6. SSSE3 and AVX2 use it for REC3..REC6, with additional - fused cases in the extended AVX2 path. + and for REC3..REC6. SSSE3 and SSSE3 extended use it for REC3..REC6. -* Scalar recovery and selected SSSE3/AVX2 REC1/REC2 paths continue to use +* Scalar recovery, SSSE3 REC1/REC2, and AArch32 NEON P+Q continue to use raid_delta_gen(). * Non-GFNI fused recovery splits every completed syndrome into low/high nibbles @@ -84,9 +85,9 @@ Recovery: or broadcasting inside the data loop. * Specialized P+Q rec2of2 recovery reduces 4 GF multiplications to 2 GF - multiplications and 1 XOR; AVX512BW, GFNI, and AArch64 NEON handle this - directly inside the 1-pass fused kernel, while scalar int8, SSSE3, AVX2, and - AArch32 NEON backends implement dedicated rec2of2 helpers. + multiplications and 1 XOR; AVX512BW, GFNI, AVX2, and AArch64 NEON handle this + directly inside the 1-pass fused kernel, while scalar int8, SSSE3, SSSE3ext, + and AArch32 NEON backends implement dedicated rec2of2 helpers. * When P is among the selected parity blocks, optimized SIMD RECX paths reconstruct only nr - 1 missing blocks through the inverse matrix and derive @@ -190,7 +191,7 @@ Typical recovery case * REC1 implementations on AVX512BW, GFNI, and AArch64 NEON execute directly inside the fused streaming kernel (reducing to a single-pass pure XOR stream - without memory swapping), while scalar and older SIMD paths call + without memory swapping), while scalar, SSSE3, and AVX2 paths call raid_rec1of1(). * raid_rec1of1() remaps pointers so that P is used as an XOR input and the @@ -232,7 +233,7 @@ REC1OF1: inside the fused 1-pass kernel, streaming surviving data blocks and P in a single pass. -* For scalar and older SIMD wrappers, REC1 enters raid_rec1of1() to recompute P +* For scalar, SSSE3, and AVX2 wrappers, REC1 enters raid_rec1of1() to recompute P using the generation backend. REC1OF2 and REC1OF3: @@ -246,11 +247,11 @@ REC2OF2: * The row uses ip = { 0, 1 }, meaning P and Q. -* On AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON, REC2 executes +* On AVX2, AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON, REC2 executes directly in the 1-pass fused kernel, which computes P with XOR, Q with GF multiplication, and derives the second missing block via XOR. -* Scalar int8, SSSE3, AVX2, and AArch32 NEON call their specialized +* Scalar int8, SSSE3, SSSE3ext, and AArch32 NEON call their specialized raid_rec2of2_*() delta helpers. * This makes the row a direct benchmark for the specialized P+Q recovery @@ -311,12 +312,12 @@ raid_delta_gen() fusion according to register pressure; they do not allocate full-size syndrome buffers. -* The common rec1of1/P case on AVX512BW and GFNI executes as pure XOR streaming - within the fused kernel (no GF matrix multiplication), while on older backends - it uses the raid_rec1of1() pointer remapping path. +* The common rec1of1/P case on AVX512BW, GFNI, and AArch64 NEON executes as pure + XOR streaming within the fused kernel (no GF matrix multiplication), while on + scalar, SSSE3, and AVX2 it uses the raid_rec1of1() pointer remapping path. -* raid_delta_gen() remains in the scalar implementation and selected - SSSE3/AVX2 REC1/REC2 paths. +* raid_delta_gen() remains in the scalar implementation, SSSE3 REC1/REC2, and + AArch32 NEON P+Q. Generation: AVX512 width and unrolling @@ -646,17 +647,14 @@ REC1 optimizations The normal REC1 case with P is handled either directly in the fused 1-pass SIMD kernel (AVX512BW, GFNI, AArch64 NEON) as pure XOR streaming, or via -raid_rec1of1() (scalar, legacy SIMD). +raid_rec1of1() (scalar, SSSE3, AVX2). For generic REC1, used when P is unavailable: -* AVX512BW, GFNI, AArch64 NEON, and AArch32 NEON use the fused recovery kernel. +* AVX2, AVX512BW, GFNI, AArch64 NEON, and AArch32 NEON use the fused recovery + kernel. -* AVX2 extended uses its delta kernel for Q and fused recovery for later parity - rows. - -* Scalar, SSSE3, base AVX2, and SSSE3 extended use their delta-based REC1 - kernels. +* Scalar, SSSE3, and SSSE3 extended use their delta-based REC1 kernels. REC1 status: @@ -674,14 +672,12 @@ REC2 optimizations REC2 has both fused and delta-based implementations. -* AVX512BW, GFNI, and AArch64 NEON use fused syndrome generation for every - selected parity pair (including P+Q). AArch32 NEON uses fused generation for - non-P pairs and REC3..REC6. +* AVX2, AVX512BW, GFNI, and AArch64 NEON use fused syndrome generation for every + selected parity pair (including P+Q in the fused kernel). AArch32 NEON uses + fused generation for non-P pairs and REC3..REC6. -* AVX2 extended uses its delta kernel for P+Q and fused recovery for other - parity pairs. - -* Scalar, SSSE3, base AVX2, and SSSE3 extended use delta-based REC2 kernels. +* Scalar int8, SSSE3, SSSE3 extended, and AArch32 NEON use delta-based REC2 + kernels. * Delta-based non-GFNI kernels split both deltas once per chunk and reuse the split values for both recovered outputs. @@ -696,11 +692,10 @@ Important portability result: REC2 status: -* Generic REC2 uses the fused path where available and retains local split reuse - in the delta-based paths. +* Generic REC2 uses the fused path across AVX2, AVX512BW, GFNI, and NEON. -* AVX512BW, GFNI, and AArch64 NEON handle P+Q recovery directly inside the - 1-pass fused kernel, while scalar int8, SSSE3, AVX2, and AArch32 NEON +* AVX2, AVX512BW, GFNI, and AArch64 NEON handle P+Q recovery directly inside the + 1-pass fused kernel, while scalar int8, SSSE3, SSSE3ext, and AArch32 NEON backends implement dedicated rec2of2 delta helpers. @@ -718,13 +713,12 @@ instead of the generic 2x2 reconstruction with four GF multiplications. Backends implement this specialization as follows: -* AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON integrate this derivation - directly into the unified 1-pass fused kernel (P is accumulated via XOR, Q via - GF multiplication, Dy is solved with inverse coefficients, and Dx derived by - XOR with Pdelta). +* AVX2, AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON integrate this + derivation directly into the unified 1-pass fused kernel (P is accumulated via + XOR, Q via GF multiplication, Dy is solved with inverse coefficients, and Dx + derived by XOR with Pdelta). * Scalar int8 (raid_rec2of2_int8) * SSSE3 and SSSE3 ext (raid_rec2of2_ssse3, raid_rec2of2_ssse3ext) -* AVX2 and AVX2 ext (raid_rec2of2_avx2, raid_rec2of2_avx2ext) * AArch32 NEON (raid_rec2of2_neon32) The rec2of2 benchmark row directly measures this specialized P+Q fast path. @@ -1034,8 +1028,9 @@ Short list of ideas already tried or rejected existing traversal instead. * raid_delta_gen() + REC fusion: - implemented for SIMD REC3..REC6 and for REC1/REC2 on AVX512BW, GFNI, and NEON; - selected SSSE3/AVX2 REC1/REC2 and scalar paths retain raid_delta_gen(). + implemented for all SIMD REC1..REC6 on AVX512BW, GFNI, and AArch64 NEON, and for + all AVX2 REC except rec1of1; SSSE3 retains delta for REC1/REC2; only scalar + paths, SSSE3 REC1/REC2, and AArch32 NEON P+Q retain raid_delta_gen(). * Software prefetch: no strong reason to add it given the existing long streams and proven diff --git a/raid/avx2.c b/raid/avx2.c index 7c890271..6d0614ad 100644 --- a/raid/avx2.c +++ b/raid/avx2.c @@ -892,62 +892,105 @@ static __always_inline void raid_genX_avx2ext(int nd, size_t size, void **vv, in #endif + + /* - * Recover failure of one data block using AVX2 delta implementation. + * Recover one data failure using selected parity with AVX2. * - * Uses raid_delta_gen() and multiplies by the inverted coefficient table. + * Process 64 bytes per iteration (two 32-byte lanes) using only 8 YMM registers (ymm0..ymm7), + * sharing table broadcasts across both lanes. */ -static __always_inline void raid_rec1_avx2_delta(int *id, int *ip, int nd, size_t size, void **vv) +static __always_inline void raid_rec1_avx2_1(int *id, int *ip, int nd, size_t size, void **vv) { uint8_t **v = (uint8_t **)vv; uint8_t *p; uint8_t *pa; - uint8_t G; - uint8_t V; + uint8_t *src[RAID_DATA_MAX]; + const uint8_t *S[RAID_DATA_MAX]; + const uint8_t *R[2]; + uint8_t G, V; size_t i; + int d, s, ns, k; - /* setup the coefficients matrix */ + /* setup the coefficient and invert it */ G = A(ip[0], id[0]); - - /* invert it to solve the system of linear equations */ V = inv(G); - /* compute delta parity */ - raid_delta_gen(1, id, ip, nd, size, vv); - p = v[nd + ip[0]]; pa = v[id[0]]; + /* build the compact surviving-data list and its syndrome tables */ + ns = 0; + k = 0; + for (d = 0; d < nd; ++d) { + if (k < 1 && d == id[0]) { + ++k; + continue; + } + src[ns] = v[d]; + S[ns] = &raid_gfmulpshufb[A(ip[0], d)][0][0]; + ++ns; + } + BUG_ON(k != 1); + BUG_ON(ns != nd - 1); + + R[0] = &raid_gfmulpshufb[V][0][0]; + R[1] = &raid_gfmulpshufb[V][1][0]; + raid_avx_begin(); asm volatile ("vpbroadcastb %0,%%ymm7" : : "m" (gfconst16.low4[0])); - asm volatile ("vbroadcasti128 %0,%%ymm4" : : "m" (raid_gfmulpshufb[V][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm5" : : "m" (raid_gfmulpshufb[V][1][0])); for (i = 0; i < size; i += 64) { + /* start syndrome with stored parity for both 32-byte lanes */ asm volatile ("vmovdqa %0,%%ymm0" : : "m" (p[i])); asm volatile ("vmovdqa %0,%%ymm1" : : "m" (p[i + 32])); - asm volatile ("vmovdqa %0,%%ymm2" : : "m" (pa[i])); - asm volatile ("vmovdqa %0,%%ymm3" : : "m" (pa[i + 32])); - asm volatile ("vpxor %ymm2,%ymm0,%ymm0"); - asm volatile ("vpxor %ymm3,%ymm1,%ymm1"); + for (s = 0; s < ns; ++s) { + const uint8_t *t = S[s]; - asm volatile ("vpsrlw $4,%ymm0,%ymm2"); - asm volatile ("vpsrlw $4,%ymm1,%ymm3"); + asm volatile ("vmovdqa %0,%%ymm2" : : "m" (src[s][i])); + asm volatile ("vmovdqa %0,%%ymm3" : : "m" (src[s][i + 32])); - asm volatile ("vpand %ymm7,%ymm0,%ymm0"); - asm volatile ("vpand %ymm7,%ymm1,%ymm1"); - asm volatile ("vpand %ymm7,%ymm2,%ymm2"); - asm volatile ("vpand %ymm7,%ymm3,%ymm3"); + asm volatile ("vpsrlw $4,%ymm2,%ymm4"); + asm volatile ("vpsrlw $4,%ymm3,%ymm5"); + asm volatile ("vpand %ymm7,%ymm2,%ymm2"); + asm volatile ("vpand %ymm7,%ymm3,%ymm3"); + asm volatile ("vpand %ymm7,%ymm4,%ymm4"); + asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm0,%ymm4,%ymm0"); - asm volatile ("vpshufb %ymm1,%ymm4,%ymm1"); - asm volatile ("vpshufb %ymm2,%ymm5,%ymm2"); - asm volatile ("vpshufb %ymm3,%ymm5,%ymm3"); + asm volatile ("vbroadcasti128 %0,%%ymm6" : : "m" (t[0])); + asm volatile ("vpshufb %ymm2,%ymm6,%ymm2"); + asm volatile ("vpshufb %ymm3,%ymm6,%ymm3"); + asm volatile ("vpxor %ymm2,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm3,%ymm1,%ymm1"); - asm volatile ("vpxor %ymm2,%ymm0,%ymm0"); - asm volatile ("vpxor %ymm3,%ymm1,%ymm1"); + asm volatile ("vbroadcasti128 %0,%%ymm6" : : "m" (t[16])); + asm volatile ("vpshufb %ymm4,%ymm6,%ymm4"); + asm volatile ("vpshufb %ymm5,%ymm6,%ymm5"); + asm volatile ("vpxor %ymm4,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm5,%ymm1,%ymm1"); + } + + /* split completed syndrome in ymm0 and ymm1 */ + asm volatile ("vpsrlw $4,%ymm0,%ymm4"); + asm volatile ("vpsrlw $4,%ymm1,%ymm5"); + asm volatile ("vpand %ymm7,%ymm0,%ymm0"); + asm volatile ("vpand %ymm7,%ymm1,%ymm1"); + asm volatile ("vpand %ymm7,%ymm4,%ymm4"); + asm volatile ("vpand %ymm7,%ymm5,%ymm5"); + + /* multiply by inverse matrix V (table R) */ + asm volatile ("vbroadcasti128 %0,%%ymm6" : : "m" (R[0][0])); + asm volatile ("vpshufb %ymm0,%ymm6,%ymm0"); + asm volatile ("vpshufb %ymm1,%ymm6,%ymm1"); + + asm volatile ("vbroadcasti128 %0,%%ymm6" : : "m" (R[1][0])); + asm volatile ("vpshufb %ymm4,%ymm6,%ymm4"); + asm volatile ("vpshufb %ymm5,%ymm6,%ymm5"); + + asm volatile ("vpxor %ymm4,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm5,%ymm1,%ymm1"); asm volatile ("vmovdqa %%ymm0,%0" : "=m" (pa[i])); asm volatile ("vmovdqa %%ymm1,%0" : "=m" (pa[i + 32])); @@ -956,176 +999,6 @@ static __always_inline void raid_rec1_avx2_delta(int *id, int *ip, int nd, size_ raid_avx_end(); } -/* - * Recover failure of two data blocks using AVX2 delta implementation. - * - * Uses raid_delta_gen() and solves the 2x2 system directly on delta blocks. - */ -static __always_inline void raid_rec2_avx2_delta(int *id, int *ip, int nd, size_t size, void **vv) -{ - uint8_t **v = (uint8_t **)vv; - uint8_t *p[2]; - uint8_t *pa[2]; - uint8_t G[2 * 2]; - uint8_t V[2 * 2]; - size_t i; - int j, k; - - /* setup the coefficients matrix */ - for (j = 0; j < 2; ++j) - for (k = 0; k < 2; ++k) - G[j * 2 + k] = A(ip[j], id[k]); - - /* invert it to solve the system of linear equations */ - raid_invert(G, V, 2); - - /* compute delta parity */ - raid_delta_gen(2, id, ip, nd, size, vv); - - for (j = 0; j < 2; ++j) { - p[j] = v[nd + ip[j]]; - pa[j] = v[id[j]]; - } - - raid_avx_begin(); - - asm volatile ("vpbroadcastb %0,%%ymm7" : : "m" (gfconst16.low4[0])); - - for (i = 0; i < size; i += 32) { - asm volatile ("vmovdqa %0,%%ymm0" : : "m" (p[0][i])); - asm volatile ("vmovdqa %0,%%ymm2" : : "m" (pa[0][i])); - asm volatile ("vmovdqa %0,%%ymm1" : : "m" (p[1][i])); - asm volatile ("vmovdqa %0,%%ymm3" : : "m" (pa[1][i])); - asm volatile ("vpxor %ymm2,%ymm0,%ymm0"); - asm volatile ("vpxor %ymm3,%ymm1,%ymm1"); - - asm volatile ("vpxor %ymm6,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm2" : : "m" (raid_gfmulpshufb[V[0]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm3" : : "m" (raid_gfmulpshufb[V[0]][1][0])); - asm volatile ("vpsrlw $4,%ymm0,%ymm5"); - asm volatile ("vpand %ymm7,%ymm0,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm2,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm3,%ymm3"); - asm volatile ("vpxor %ymm2,%ymm6,%ymm6"); - asm volatile ("vpxor %ymm3,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm2" : : "m" (raid_gfmulpshufb[V[1]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm3" : : "m" (raid_gfmulpshufb[V[1]][1][0])); - asm volatile ("vpsrlw $4,%ymm1,%ymm5"); - asm volatile ("vpand %ymm7,%ymm1,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm2,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm3,%ymm3"); - asm volatile ("vpxor %ymm2,%ymm6,%ymm6"); - asm volatile ("vpxor %ymm3,%ymm6,%ymm6"); - - asm volatile ("vmovdqa %%ymm6,%0" : "=m" (pa[0][i])); - - asm volatile ("vpxor %ymm6,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm2" : : "m" (raid_gfmulpshufb[V[2]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm3" : : "m" (raid_gfmulpshufb[V[2]][1][0])); - asm volatile ("vpsrlw $4,%ymm0,%ymm5"); - asm volatile ("vpand %ymm7,%ymm0,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm2,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm3,%ymm3"); - asm volatile ("vpxor %ymm2,%ymm6,%ymm6"); - asm volatile ("vpxor %ymm3,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm2" : : "m" (raid_gfmulpshufb[V[3]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm3" : : "m" (raid_gfmulpshufb[V[3]][1][0])); - asm volatile ("vpsrlw $4,%ymm1,%ymm5"); - asm volatile ("vpand %ymm7,%ymm1,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm2,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm3,%ymm3"); - asm volatile ("vpxor %ymm2,%ymm6,%ymm6"); - asm volatile ("vpxor %ymm3,%ymm6,%ymm6"); - - asm volatile ("vmovdqa %%ymm6,%0" : "=m" (pa[1][i])); - } - - raid_avx_end(); -} - -/* - * Recover failure of two data blocks using P and Q AVX2 implementation. - * - * Uses raid_delta_gen() and computes the analytical solution. - */ -static __always_inline void raid_rec2of2_avx2(int *id, int *ip, int nd, size_t size, void **vv) -{ - uint8_t **v = (uint8_t **)vv; - uint8_t *p; - uint8_t *pa; - uint8_t *q; - uint8_t *qa; - uint8_t C[2]; - size_t i; - - /* get multiplication coefficients */ - C[0] = inv(powgen(id[1] - id[0]) ^ 1); - C[1] = inv(powgen(id[0]) ^ powgen(id[1])); - - /* compute delta parity */ - raid_delta_gen(2, id, ip, nd, size, vv); - - p = v[nd]; - q = v[nd + 1]; - pa = v[id[0]]; - qa = v[id[1]]; - - raid_avx_begin(); - - asm volatile ("vpbroadcastb %0,%%ymm7" : : "m" (gfconst16.low4[0])); - - for (i = 0; i < size; i += 32) { - /* Pd */ - asm volatile ("vmovdqa %0,%%ymm0" : : "m" (p[i])); - asm volatile ("vpxor %0,%%ymm0,%%ymm0" : : "m" (pa[i])); - - /* Qd */ - asm volatile ("vmovdqa %0,%%ymm1" : : "m" (q[i])); - asm volatile ("vpxor %0,%%ymm1,%%ymm1" : : "m" (qa[i])); - - /* split Pd and Qd */ - asm volatile ("vpsrlw $4,%ymm0,%ymm2"); - asm volatile ("vpsrlw $4,%ymm1,%ymm3"); - asm volatile ("vpand %ymm7,%ymm0,%ymm4"); - asm volatile ("vpand %ymm7,%ymm1,%ymm5"); - asm volatile ("vpand %ymm7,%ymm2,%ymm2"); - asm volatile ("vpand %ymm7,%ymm3,%ymm3"); - - /* C0 * Pd */ - asm volatile ("vbroadcasti128 %0,%%ymm6" : : "m" (raid_gfmulpshufb[C[0]][0][0])); - asm volatile ("vpshufb %ymm4,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm4" : : "m" (raid_gfmulpshufb[C[0]][1][0])); - asm volatile ("vpshufb %ymm2,%ymm4,%ymm4"); - asm volatile ("vpxor %ymm4,%ymm6,%ymm6"); - - /* C1 * Qd */ - asm volatile ("vbroadcasti128 %0,%%ymm4" : : "m" (raid_gfmulpshufb[C[1]][0][0])); - asm volatile ("vpshufb %ymm5,%ymm4,%ymm4"); - asm volatile ("vpxor %ymm4,%ymm6,%ymm6"); - - asm volatile ("vbroadcasti128 %0,%%ymm4" : : "m" (raid_gfmulpshufb[C[1]][1][0])); - asm volatile ("vpshufb %ymm3,%ymm4,%ymm4"); - asm volatile ("vpxor %ymm4,%ymm6,%ymm6"); - - /* Dy = ymm6, Dx = Pd ^ Dy */ - asm volatile ("vpxor %ymm6,%ymm0,%ymm0"); - - asm volatile ("vmovdqa %%ymm0,%0" : "=m" (pa[i])); - asm volatile ("vmovdqa %%ymm6,%0" : "=m" (qa[i])); - } - - raid_avx_end(); -} - /* * Recover multiple data failures using selected parity blocks with AVX2 optimized for up to four failures. * @@ -1559,76 +1432,277 @@ static __always_inline void raid_recX_avx2(int nr, int *id, int *ip, int nd, siz } #ifdef CONFIG_X86_64 + + /* - * Recover failure of two data blocks using P and Q AVX2 extended implementation. + * Recover multiple data failures using selected parity blocks with AVX2 extended optimized for up to two failures. * - * Uses raid_delta_gen() and keeps all four multiplication tables resident - * in YMM registers (ymm8..ymm11). + * Process 64 bytes at a time as two independent 32-byte lanes. + * Multiplication tables are broadcast once and shared between both lanes. + * + * If P is available, keep the complete P delta syndrome in ymm6. + * Reconstruct only nr - 1 missing blocks through the inverse matrix and + * obtain the last block by XORing the reconstructed blocks out of Pdelta. + * + * Note that it uses 16 registers, meaning that x64 is required. */ -static __always_inline void raid_rec2of2_avx2ext(int *id, int *ip, int nd, size_t size, void **vv) +static __always_inline void raid_recX_avx2ext_12(int nr, int *id, int *ip, int nd, size_t size, void **vv) { uint8_t **v = (uint8_t **)vv; - uint8_t *p[2]; - uint8_t *pa[2]; - uint8_t C[2]; + uint8_t *p[RAID_PARITY_MAX]; + uint8_t *pa[RAID_PARITY_MAX]; + uint8_t *src[RAID_DATA_MAX]; + uint8_t G[RAID_PARITY_MAX * RAID_PARITY_MAX]; + uint8_t V[RAID_PARITY_MAX * RAID_PARITY_MAX]; + const uint8_t *S[RAID_DATA_MAX][RAID_PARITY_MAX]; + const uint8_t *R[RAID_PARITY_MAX][RAID_PARITY_MAX]; size_t i; + int d, j, k, s; + int ns; + int has_p; - /* get multiplication coefficients */ - C[0] = inv(powgen(id[1] - id[0]) ^ 1); - C[1] = inv(powgen(id[0]) ^ powgen(id[1])); + BUG_ON(nr < 1 || nr > 2); - /* compute delta parity */ - raid_delta_gen(2, id, ip, nd, size, vv); + /* setup the coefficients matrix */ + for (j = 0; j < nr; ++j) + for (k = 0; k < nr; ++k) + G[j * nr + k] = A(ip[j], id[k]); - p[0] = v[nd]; - p[1] = v[nd + 1]; - pa[0] = v[id[0]]; - pa[1] = v[id[1]]; + /* invert it to solve the system of linear equations */ + raid_invert(G, V, nr); + + /* setup selected parity and destination pointers */ + for (j = 0; j < nr; ++j) { + p[j] = v[nd + ip[j]]; + pa[j] = v[id[j]]; + } + + /* ip[] is ordered. If P is available, it is always ip[0] */ + has_p = ip[0] == 0; + + /* build the compact surviving-data list and its syndrome tables */ + ns = 0; + k = 0; + + for (d = 0; d < nd; ++d) { + if (k < nr && d == id[k]) { + ++k; + continue; + } + + src[ns] = v[d]; + + for (j = 0; j < nr; ++j) + S[ns][j] = &raid_gfmulpshufb[A(ip[j], d)][0][0]; + + ++ns; + } + + BUG_ON(k != nr); + BUG_ON(ns != nd - nr); + + /* precompute inverse-matrix multiplication table pointers for required outputs */ + for (j = 0; j < nr - has_p; ++j) + for (k = 0; k < nr; ++k) + R[j][k] = &raid_gfmulpshufb[V[j * nr + k]][0][0]; raid_avx_begin(); - asm volatile ("vpbroadcastb %0,%%ymm7" : : "m" (gfconst16.low4[0])); + for (i = 0; i < size; i += 64) { + asm volatile ("vpbroadcastb %0,%%ymm15" : : "m" (gfconst16.low4[0])); - /* the multiplication tables are constant for the whole recovery */ - asm volatile ("vbroadcasti128 %0,%%ymm8" : : "m" (raid_gfmulpshufb[C[0]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm9" : : "m" (raid_gfmulpshufb[C[0]][1][0])); - asm volatile ("vbroadcasti128 %0,%%ymm10" : : "m" (raid_gfmulpshufb[C[1]][0][0])); - asm volatile ("vbroadcasti128 %0,%%ymm11" : : "m" (raid_gfmulpshufb[C[1]][1][0])); - - for (i = 0; i < size; i += 32) { - /* d0 = p[0] ^ pa[0] */ + /* + * Start each syndrome with the corresponding stored parity. + * + * During syndrome computation: + * + * ymm0/ymm1 syndrome 0, first/second 32 bytes + * ymm2/ymm3 syndrome 1, first/second 32 bytes + * + * ymm6/ymm7 source low, first/second 32 bytes + * ymm8/ymm9 source high, first/second 32 bytes + * ymm12/ymm13 multiplication table / temporary + * ymm15 low-nibble mask + */ asm volatile ("vmovdqa %0,%%ymm0" : : "m" (p[0][i])); - asm volatile ("vmovdqa %0,%%ymm2" : : "m" (pa[0][i])); - asm volatile ("vpxor %ymm2,%ymm0,%ymm0"); + asm volatile ("vmovdqa %0,%%ymm1" : : "m" (p[0][i + 32])); - /* d1 = p[1] ^ pa[1] */ - asm volatile ("vmovdqa %0,%%ymm1" : : "m" (p[1][i])); - asm volatile ("vmovdqa %0,%%ymm3" : : "m" (pa[1][i])); - asm volatile ("vpxor %ymm3,%ymm1,%ymm1"); + if (nr >= 2) { + asm volatile ("vmovdqa %0,%%ymm2" : : "m" (p[1][i])); + asm volatile ("vmovdqa %0,%%ymm3" : : "m" (p[1][i + 32])); + } - /* C[0] * d0 */ - asm volatile ("vpsrlw $4,%ymm0,%ymm5"); - asm volatile ("vpand %ymm7,%ymm0,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm8,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm9,%ymm3"); - asm volatile ("vpxor %ymm3,%ymm2,%ymm6"); + /* add all surviving data contributions */ + for (s = 0; s < ns; ++s) { + const uint8_t **t = S[s]; - /* C[1] * d1 */ - asm volatile ("vpsrlw $4,%ymm1,%ymm5"); - asm volatile ("vpand %ymm7,%ymm1,%ymm4"); - asm volatile ("vpand %ymm7,%ymm5,%ymm5"); - asm volatile ("vpshufb %ymm4,%ymm10,%ymm2"); - asm volatile ("vpshufb %ymm5,%ymm11,%ymm3"); - asm volatile ("vpxor %ymm2,%ymm6,%ymm6"); - asm volatile ("vpxor %ymm3,%ymm6,%ymm6"); + asm volatile ("vmovdqa %0,%%ymm6" : : "m" (src[s][i])); + asm volatile ("vmovdqa %0,%%ymm7" : : "m" (src[s][i + 32])); - /* pa[1] = C[0] * d0 ^ C[1] * d1 */ - asm volatile ("vmovdqa %%ymm6,%0" : "=m" (pa[1][i])); + /* + * P has coefficient 1 for every data disk. + * Do the XOR before destroying the original source + * while splitting it into low/high nibbles. + */ + if (has_p) { + asm volatile ("vpxor %ymm6,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm7,%ymm1,%ymm1"); - /* pa[0] = d0 ^ pa[1] */ - asm volatile ("vpxor %ymm6,%ymm0,%ymm0"); - asm volatile ("vmovdqa %%ymm0,%0" : "=m" (pa[0][i])); + /* split both 32-byte source lanes */ + asm volatile ("vpsrlw $4,%ymm6,%ymm8"); + asm volatile ("vpsrlw $4,%ymm7,%ymm9"); + asm volatile ("vpand %ymm15,%ymm6,%ymm6"); + asm volatile ("vpand %ymm15,%ymm7,%ymm7"); + asm volatile ("vpand %ymm15,%ymm8,%ymm8"); + asm volatile ("vpand %ymm15,%ymm9,%ymm9"); + } else { + /* split both 32-byte source lanes */ + asm volatile ("vpsrlw $4,%ymm6,%ymm8"); + asm volatile ("vpsrlw $4,%ymm7,%ymm9"); + asm volatile ("vpand %ymm15,%ymm6,%ymm6"); + asm volatile ("vpand %ymm15,%ymm7,%ymm7"); + asm volatile ("vpand %ymm15,%ymm8,%ymm8"); + asm volatile ("vpand %ymm15,%ymm9,%ymm9"); + + /* + * Syndrome 0. + * + * Share each table broadcast between the two + * 32-byte lanes. + */ + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[0][0])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm6,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm7,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm13,%ymm1,%ymm1"); + + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[0][16])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm8,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm9,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm0,%ymm0"); + asm volatile ("vpxor %ymm13,%ymm1,%ymm1"); + } + + /* syndrome 1 */ + if (nr >= 2) { + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[1][0])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm6,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm7,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm2,%ymm2"); + asm volatile ("vpxor %ymm13,%ymm3,%ymm3"); + + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[1][16])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm8,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm9,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm2,%ymm2"); + asm volatile ("vpxor %ymm13,%ymm3,%ymm3"); + } + } + + /* preserve raw P delta in ymm4/ymm5 before splitting syndrome 0 */ + if (has_p) { + asm volatile ("vmovdqa %ymm0,%ymm4"); + asm volatile ("vmovdqa %ymm1,%ymm5"); + } + + /* + * Split all completed syndromes. + * + * After this: + * + * syndrome 0: + * ymm0 / ymm6 first 32 bytes low/high + * ymm1 / ymm7 second 32 bytes low/high + * + * syndrome 1: + * ymm2 / ymm8 first 32 bytes low/high + * ymm3 / ymm9 second 32 bytes low/high + */ + asm volatile ("vpsrlw $4,%ymm0,%ymm6"); + asm volatile ("vpsrlw $4,%ymm1,%ymm7"); + asm volatile ("vpand %ymm15,%ymm0,%ymm0"); + asm volatile ("vpand %ymm15,%ymm1,%ymm1"); + asm volatile ("vpand %ymm15,%ymm6,%ymm6"); + asm volatile ("vpand %ymm15,%ymm7,%ymm7"); + + if (nr >= 2) { + asm volatile ("vpsrlw $4,%ymm2,%ymm8"); + asm volatile ("vpsrlw $4,%ymm3,%ymm9"); + asm volatile ("vpand %ymm15,%ymm2,%ymm2"); + asm volatile ("vpand %ymm15,%ymm3,%ymm3"); + asm volatile ("vpand %ymm15,%ymm8,%ymm8"); + asm volatile ("vpand %ymm15,%ymm9,%ymm9"); + } + + /* + * Reconstruct missing data blocks through the inverse matrix. + * If P is available, reconstruct only nr - 1 blocks. + * + * ymm12/ymm13 are a shared multiplication table pair. + * ymm14/ymm15 accumulate the two 32-byte output lanes. + * + * At this point ymm15 is no longer needed as the nibble mask, + * so all 16 YMM registers are available. + */ + for (j = 0; j < nr - has_p; ++j) { + const uint8_t **t = R[j]; + + /* + * First coefficient. + * Initialize both output accumulators from the low + * nibble products. + */ + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[0][0])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm0,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm1,%ymm13,%ymm13"); + asm volatile ("vmovdqa %ymm12,%ymm14"); + asm volatile ("vmovdqa %ymm13,%ymm15"); + + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[0][16])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm6,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm7,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm14,%ymm14"); + asm volatile ("vpxor %ymm13,%ymm15,%ymm15"); + + /* second coefficient */ + if (nr >= 2) { + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[1][0])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm2,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm3,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm14,%ymm14"); + asm volatile ("vpxor %ymm13,%ymm15,%ymm15"); + + asm volatile ("vbroadcasti128 %0,%%ymm12" : : "m" (t[1][16])); + asm volatile ("vmovdqa %ymm12,%ymm13"); + asm volatile ("vpshufb %ymm8,%ymm12,%ymm12"); + asm volatile ("vpshufb %ymm9,%ymm13,%ymm13"); + asm volatile ("vpxor %ymm12,%ymm14,%ymm14"); + asm volatile ("vpxor %ymm13,%ymm15,%ymm15"); + } + + asm volatile ("vmovdqa %%ymm14,%0" : "=m" (pa[j][i])); + asm volatile ("vmovdqa %%ymm15,%0" : "=m" (pa[j][i + 32])); + } + + /* derive the final missing block from raw Pdelta when P is available */ + if (has_p) { + if (nr >= 2) { + asm volatile ("vpxor %0,%%ymm4,%%ymm4" : : "m" (pa[0][i])); + asm volatile ("vpxor %0,%%ymm5,%%ymm5" : : "m" (pa[0][i + 32])); + asm volatile ("vmovdqa %%ymm4,%0" : "=m" (pa[1][i])); + asm volatile ("vmovdqa %%ymm5,%0" : "=m" (pa[1][i + 32])); + } else { + asm volatile ("vmovdqa %%ymm4,%0" : "=m" (pa[0][i])); + asm volatile ("vmovdqa %%ymm5,%0" : "=m" (pa[0][i + 32])); + } + } } raid_avx_end(); @@ -2351,20 +2425,13 @@ void raid_rec1_avx2(int nr, int *id, int *ip, int nd, size_t size, void **vv) return; } - raid_rec1_avx2_delta(id, ip, nd, size, vv); + raid_rec1_avx2_1(id, ip, nd, size, vv); } void raid_rec2_avx2(int nr, int *id, int *ip, int nd, size_t size, void **vv) { BUG_ON(nr != 2); - - /* if recovering with P,Q use the specialized function */ - if (ip[0] == 0 && ip[1] == 1) { - raid_rec2of2_avx2(id, ip, nd, size, vv); - return; - } - - raid_rec2_avx2_delta(id, ip, nd, size, vv); + raid_recX_avx2_1234(2, id, ip, nd, size, vv); } void raid_rec3_avx2(int nr, int *id, int *ip, int nd, size_t size, void **vv) @@ -2402,26 +2469,13 @@ void raid_rec1_avx2ext(int nr, int *id, int *ip, int nd, size_t size, void **vv) return; } - /* if recovering with Q uses the delta function */ - if (ip[0] == 1) { - raid_rec1_avx2_delta(id, ip, nd, size, vv); - return; - } - - raid_recX_avx2ext_123(1, id, ip, nd, size, vv); + raid_rec1_avx2_1(id, ip, nd, size, vv); } void raid_rec2_avx2ext(int nr, int *id, int *ip, int nd, size_t size, void **vv) { BUG_ON(nr != 2); - - /* if recovering with P,Q uses the delta function */ - if (ip[0] == 0 && ip[1] == 1) { - raid_rec2of2_avx2ext(id, ip, nd, size, vv); - return; - } - - raid_recX_avx2ext_123(2, id, ip, nd, size, vv); + raid_recX_avx2ext_12(2, id, ip, nd, size, vv); } void raid_rec3_avx2ext(int nr, int *id, int *ip, int nd, size_t size, void **vv)