snapraid/raid/NOTES

SnapRAID RAID library performance notes
=======================================


Scope
-----

These notes summarize performance experiments, implementation choices,
rejected ideas, recovery optimizations, and non-obvious conclusions
reached while reviewing and tuning the SnapRAID RAID library.

The focus is the 1..6 parity implementation, especially x86 AVX2,
AVX512BW, GFNI, SSSE3, and AArch64/AArch32 NEON backends.

This is intentionally a record of what has already been tried and what was
learned from it, so that the same ideas are not repeatedly re-investigated.

Detailed project-by-project comparisons with other RAID and erasure-coding
engines are kept in OTHERS. This file keeps only conclusions that affect
SnapRAID implementation or optimization policy.

The main rule is to optimize the real SnapRAID workload, not only isolated
inner loops.


Implementation overview
-----------------------

Generation:

* For the CPUs, compilers, and SnapRAID workloads tested so far, the 1..6
  parity generation paths have no known major performance bottleneck after
  coefficient-residency optimization.

* Relevant scalar, SSE/SSSE3, AVX2, AVX512BW, AArch64/AArch32 NEON, RAID GFNI
  affine, and AES GFNI paths are implemented.

* Specialized GEN2 and GEN3 paths keep useful multiplication tables or
  coefficient representations in vector registers when register pressure
  permits it.

* This residency optimization preserves the preferred chunk-major,
  multi-output traversal. It does not add another pass over source blocks.

* RAID GFNI generation uses direct Cauchy coefficients. Their 8-byte affine
  representations are precomputed in a disk-major table, avoiding a dependent
  Cauchy-coefficient -> affine-matrix lookup in the hot generation loop.
  Similarly, scalar int8 generation precomputes per-disk 2D multiplication table
  pointers T[d][p] before the hot loop to eliminate two-level dependent table
  lookups.

* AVX512 128-byte processing, non-GFNI AVX512BW Horner, standalone
  VPTERNLOGQ transformations, VPBLENDVB, cached parity stores, and large
  pre-expanded GFNI constants have been investigated. Some are rejected, while
  VPTERNLOGQ is now used where it is part of a larger profitable GEN2/GEN3
  restructuring.

Recovery:

* The normal one-data-disk recovery with P available is handled via raid_rec1of1()
  across all backends (Scalar, NEON, SSSE3, AVX2, AVX512BW, GFNI), invoking
  raid_gen(nd, 1, size, v, 0) with temporal stores.

* SIMD recovery has parity-count-specific REC1..REC6 entry points. Fused paths
  compute exactly the selected syndromes in one scan of the surviving data and
  reconstruct the missing blocks without full-size temporary syndrome buffers.

* AVX512BW and GFNI (AVX2 and AVX512) use raid_rec1of1() for P and fused syndrome
  generation for non-P REC1 and all REC2..REC6 cases, including P+Q (rec2of2).
  AVX2 and AVX2 extended use raid_rec1of1() for P, use specialized 1-pass
  Horner kernels for Q, and use fused selected-syndrome recovery for the other
  non-P REC1 cases and fused recovery for all REC2..REC6 cases.
  SSSE3 and SSSE3 extended likewise use raid_rec1of1() for P, use specialized
  1-pass Horner kernels for Q-only REC1 and P+Q REC2, and use fused
  selected-syndrome recovery for the remaining REC1..REC6 cases. AArch64 NEON
  and AArch32 NEON use raid_rec1of1() for P and fused syndrome generation for
  non-P REC1 and all REC2..REC6 cases.

* Scalar recovery uses raid_rec1of1() for P and fused syndrome recovery for
  non-P REC1 and all REC2..REC6 cases. raid_delta_gen() is completely eliminated.

* Non-GFNI fused recovery splits every completed syndrome into low/high nibbles
  once and reuses the split values across all reconstructed outputs.

* Non-GFNI fused recovery precomputes survivor coefficient and inverse-matrix
  table pointers in C before entering the hot data loop.

* RAID GFNI fused recovery precomputes direct pointers to the 8-byte affine
  representation of every syndrome and inverse-matrix coefficient.

* A recurring RECX local cost is repeated coefficient table/matrix loading
  or broadcasting inside the data loop.

* Specialized P+Q rec2of2 recovery reduces 4 GF multiplications to 2 GF
  multiplications and 1 XOR; AVX512BW, GFNI, AVX2, and AArch64 NEON handle this
  directly inside the 1-pass fused kernel. SSSE3 and SSSE3ext use dedicated
  1-pass Horner rec2of2 helpers, AArch32 NEON uses a dedicated delta-based
  helper, and scalar int8 handles it via has_p in raid_rec2_int8.

* When P is among the selected parity blocks, optimized SIMD and scalar RECX
  paths reconstruct only nr - 1 missing blocks through the inverse matrix and
  derive the final missing block from the raw P delta by XORing the blocks
  already reconstructed.

Architecture and validation:

* AArch32 has a separate NEON backend for generation and recovery. Cross-build,
  QEMU user-mode testing, and CI coverage are available for the ARM32 path.

* Matrix inversion has dedicated multithreaded stress testing. With at most six
  parity levels, inversion setup is not a significant performance target.


Real workload assumptions
-------------------------

* SnapRAID processes large blocks: typically 256 KiB per data/parity block.

* A normal stripe therefore touches a large amount of memory. With 12 data
  disks, one 256 KiB stripe already means about 3 MiB of data input before
  considering parity output.

* This large-block behavior is very important. Optimizations that reread a
  source buffer, increase the number of passes, or enlarge lookup tables can be
  worse even if the inner instruction sequence looks better.

* This is also the first filter for ideas taken from other RAID or
  erasure-coding engines. A design optimized for page-sized recovery,
  cache-resident tiles, or generic small shards may make a different memory
  tradeoff and should not be copied mechanically.

* Prefer reducing work inside the existing single-pass traversal before
  considering a traversal change. Saving a broadcast or table load is useful;
  saving registers by rereading a complete 256 KiB source block usually is not.

* SnapRAID uses a queue before parity data is written to disk. In the measured
  SnapRAID disk-write workloads used during this tuning, very small queues were
  slower. Throughput improved clearly up to about queue depth 32 and continued
  to improve beyond that with diminishing returns. These queue-depth results
  are workload- and storage-dependent and are not an architectural property of
  the RAID kernels.

* With 256 KiB blocks, queue depth 32 means this much parity output can be in
  flight:

      1 parity  =  8 MiB
      2 parity  = 16 MiB
      3 parity  = 24 MiB
      6 parity  = 48 MiB

  Therefore a parity block produced by the RAID kernel is normally not consumed
  immediately.

* A memory read of a 256 KiB block that is no longer cached is still very cheap
  compared with real storage I/O. Trying to keep many queued parity blocks hot
  in cache is therefore not necessarily useful.


MOVNT / non-temporal stores vs cached generation
--------------------------------------------------

* raid_gen(..., streaming) allows the caller to select between non-temporal
  streaming stores (streaming != 0, using MOVNTDQ/VMOVNTDQ + sfence on x86) and
  normal temporal/cached stores (streaming == 0, using MOVDQA/VMOVDQA without sfence).

* Generation backends select MOVNTDQ / VMOVNTDQ for parity output when
  streaming != 0 is requested.

* The synthetic speedtest naturally favors non-temporal stores because generated
  parity is not consumed afterwards. This benchmark therefore measures producer
  throughput, not the complete producer -> queue -> writer pipeline.

* In the real SnapRAID sync pipeline, parity blocks are queued to disk writers,
  which makes the workload non-temporal (streaming = 1). At typical queue depths,
  cached parity output would occupy many MiB of cache and evict useful input data;
  MOVNT avoids normal cache allocation/write-allocate traffic and reduces cache
  pressure.

* Conversely, in-memory parity verification in scrub and check, and
  single-parity recovery in raid_rec1of1() use cached generation (streaming = 0)
  so that computed parity or recovered data remains hot in cache for immediate
  CPU consumption.


Recovery temporal stores
------------------------

* Recovery output has a different lifetime from generated parity written to disk.
  A reconstructed data block is an intermediate result that is consumed again
  immediately.

* After raid_data() reconstructs a candidate, repair_step() immediately validates
  it. With a trustworthy hash, block_data_cmp()/memhash_block() reads the recovered
  data. Without such a hash, parity validation regenerates parity from the
  reconstructed data. On successful hash validation, all configured parity is
  regenerated as well.

* During fix, the same recovered data buffer is then passed to handle_write(),
  so keeping recently produced cache lines available can also help the following
  write path.

* For this reason, recovery uses ordinary temporal MOVDQA/VMOVDQA stores for
  recovered data rather than MOVNTDQ/VMOVNTDQ. Single-disk recovery with P
  (raid_rec1of1()) invokes raid_gen(nd, 1, size, v, 0) with streaming = 0,
  ensuring temporal stores and omitting sfence across all backends.

* With typical 256 KiB blocks and roughly 8..12 data disks, the full recovery
  working set may fit in L2 on some CPUs and exceed it on others. Do not switch
  store policy at an architecture-specific disk-count/cache-size threshold: the
  recovered block itself has strong immediate reuse on all of these machines,
  while generated parity written during sync is normally queued and consumed later.
  Temporal recovery stores are therefore the general compromise.

* Store policy is deliberately selectable: generation for disk writing (sync)
  uses non-temporal stores (streaming = 1), while recovery and in-memory
  verification use temporal stores (streaming = 0).


Typical recovery case
---------------------

* The common recovery case is one failed data disk while parity P is still
  available.

* raid_rec() selects the first available parity. Therefore, for a single data
  failure with P available, recovery uses P.

* All backends (Scalar, NEON, SSSE3, AVX2, AVX512BW, GFNI) route the P case
  through raid_rec1of1().

* raid_rec1of1() remaps pointers so that P is used as an XOR input and the
  failed data buffer is used as the GEN1 output, then calls raid_gen(nd, 1, ..., 0).
  All backends use this path with temporal stores for optimal P single-disk recovery.

* Therefore the common single-disk recovery path executes with no syndrome or GF
  overhead.

* The data traffic is already close to the minimum:

      read P
      read all surviving data blocks
      XOR them
      write the recovered data block once

* There are no temporary syndrome buffers, no GF multiplication, and no matrix
  inversion in this common path.

* This means that large architectural recovery changes mostly optimize rare
  failure cases, not the usual single-disk recovery.

* Two simultaneous data disk failures are already an exceptional case in normal
  use. Three or more simultaneous data failures are even less important as
  optimization targets.


Recovery speedtest interpretation
---------------------------------

The recovery speed tests contain separate rows for the important fast paths and
for the generic recovery paths.

REC1OF1:

* The row uses ip[0] == 0.

* All backends use raid_rec1of1(), calling the GEN1 backend with streaming = 0
  to reconstruct the missing block directly into cache using temporal stores.

* Because the row measures only the recovery producer and does not include the
  immediate validation/parity-regeneration consumers, temporal-store x86
  implementations can benchmark lower than a non-temporal GEN1-based path even
  when the temporal policy is preferable for the complete check/fix workload.

REC1OF2 and REC1OF3:

* The rows use Q and R respectively, avoiding the PAR1 fast path.

* All backends use single-pass recovery for these non-P cases. AVX2,
  AVX2 extended, SSSE3, and SSSE3 extended use specialized Horner kernels for
  Q and fused selected-syndrome kernels for the other parity rows. AVX512BW,
  GFNI, NEON, AArch32 NEON, and Scalar use their fused recovery paths.

REC2OF2:

* The row uses ip = { 0, 1 }, meaning P and Q.

* On AVX2, AVX512BW, GFNI (AVX2 and AVX512), AArch64 NEON, AArch32 NEON, and
  Scalar int8, REC2 executes directly in the 1-pass fused kernel, which computes
  P with XOR, Q with GF multiplication, and derives the second missing block
  via XOR.

* SSSE3 and SSSE3ext call dedicated 1-pass Horner raid_rec2of2_*() helpers that
  compute P and Q directly from the surviving data in a single pass.

* This makes the row a direct benchmark for the specialized P+Q recovery
  implementation.

REC2OF3 and REC2OF4:

* The rows use { Q, R } and { R, S } respectively, avoiding the P+Q fast case.

* REC2OF3 and REC2OF4 remain useful control rows because they do not match
  ip = { 0, 1 } and therefore continue to exercise the generic REC2 paths.

* They measure generic 2x2 recovery with different selected-parity positions,
  verifying exact selected-syndrome fusion.

REC3OF3 through REC6OF6:

* These rows measure multi-failure recovery for each supported failure count
  using the first N parity levels. SIMD backends and scalar int8 use fused
  syndrome recovery.

Delta generation:

* A benchmark forces a matching generation backend only when the measured
  wrapper can call raid_rec1of1() (specifically in REC1OF1).

* Fused and specialized single-pass recovery rows do not depend on raid_gen()
  because they generate the required syndromes directly inside the recovery
  kernel.


Single-pass fused recovery
--------------------------

* Fused SIMD recovery accumulates exactly the parity rows selected by ip[]
  directly from the stored parity and surviving data.

* This avoids the prefix constraint of the old delta-based approach, which
  computed np = ip[nr - 1] + 1 via raid_gen().

* Example:

      nr = 3
      ip = { 0, 2, 5 }

  makes an old delta-based path generate through GEN6, while a fused path
  calculates only P, R and U.

* For each SIMD chunk, the fused path starts the selected syndrome accumulators
  from stored parity, scans every surviving data source once, and then applies
  the inverse matrix to reconstruct all missing blocks.

* AVX512BW and GFNI keep all one to six syndromes in registers. Non-GFNI AVX2,
  SSSE3, and NEON paths use registers or small aligned per-chunk scratch space
  according to register pressure; they do not allocate full-size syndrome
  buffers.

* The common rec1of1/P case executes via raid_rec1of1() across all backends,
  calling the GEN1 generator with streaming = 0 to write the recovered data
  block using temporal stores with no syndrome or GF recovery setup.

* raid_delta_gen() has been completely eliminated from the codebase.


Generation: AVX512 width and unrolling
--------------------------------------

* Wider processing is not automatically faster.

* AVX512 128-byte processing was tested on the tuning systems represented by
  these notes and did not consistently outperform the existing 64-byte path;
  in some tested cases it was slower.

* Therefore do not assume that x2/x4 spatial unrolling used by some RAID
  engines is beneficial for SnapRAID.

* The block size is already large, memory traffic is substantial, and wider
  unrolling can increase register pressure, front-end work, and AVX512 power or
  frequency effects without reducing total memory traffic.

* Standalone VPTERNLOGQ transformations were tested on Zen 5 and did not improve
  performance.

* This is not a general rejection of VPTERNLOGQ. AVX512BW GEN2/GEN3
  kernels use VPTERNLOGQ where it is part of a larger table-residency
  restructuring that reduces other work. Judge the complete kernel, not an
  instruction in isolation.

* Conclusion: keep the conservative AVX512 processing width unless a
  specific benchmark proves otherwise.


Generation: GFNI direct coefficients
------------------------------------

* Classic RAID6 P+Q generation has a natural Horner recurrence for Q with the
  standard RAID polynomial/generator.

* Horner implementations were investigated for GFNI generation, but the final
  implementation uses direct Cauchy coefficients for both RAID and AES GFNI
  generation.

* RAID GFNI represents each fixed Cauchy coefficient with an 8-byte affine
  matrix consumed by VGF2P8AFFINEQB.

* The generation-specific RAID Cauchy affine matrices are precomputed in:

      raid_gfcauchyaffine_raid[DISK][PARITY - 1][8]

  with disk-major layout:

      [disk][Q/R/S/T/U][8-byte affine matrix]

* This avoids the previous dependent lookup sequence:

      load raid_gfcauchy_raid[p][d]
      use the coefficient to index raid_gfaffine_raid[]
      load/broadcast the affine matrix

  and replaces it with one direct address:

      raid_gfcauchyaffine_raid[d][p - 1]

* The generic raid_gfaffine_raid[256][8] table remains necessary for
  runtime-computed multipliers, especially recovery.

* AES GFNI generation uses the corresponding direct Cauchy coefficient byte
  with VGF2P8MULB.

* Keeping both RAID and AES generation in direct-coefficient form also makes
  their GFNI kernels structurally comparable: the main field-specific
  difference is VGF2P8AFFINEQB with an 8-byte RAID matrix versus
  VGF2P8MULB with an AES coefficient byte.

* Scalar int8 generation (raid_gen2_int8 .. raid_gen6_int8) similarly precomputes
  per-disk multiplication table pointers T[d][p] in a 2D array before entering
  the hot loop, replacing the double-indexed raid_gfmul[d0][raid_gfgen[j][d]]
  lookup with a single direct table access T[d][p][d0].


Generation: coefficient residency
---------------------------------

Recent GEN2/GEN3 work established coefficient residency as a practical
generation optimization, not only a theoretical one.

* Non-GFNI SIMD GEN2/GEN3 kernels (such as AVX512BW) preload useful low/high
  multiplication tables into vector registers when the register file has enough
  space. AVX512 GFNI kernels similarly preload Cauchy affine matrices (for RAID)
  or constant multiplier bytes (for AES).

* The resident set can be partial. When there are more data disks than can be
  covered by spare registers, later disks fall back to loading their
  coefficient tables normally.

* GEN3 has higher register pressure because both Q and R coefficients are
  needed. Residency is therefore deliberately bounded by the available
  register file rather than forcing spills or another source pass.

* AVX512BW can use VPTERNLOGQ as part of these resident-table kernels to combine
  the accumulator with low/high products. The useful optimization is the whole
  scheduling and residency change, not VPTERNLOGQ by itself.

* The source chunk is still loaded once and contributes to all parity outputs
  before moving on. The optimization therefore preserves the most important
  large-block memory property.

General conclusion:

* First use spare registers to remove repeated coefficient setup while
  preserving one-pass multi-output generation.

* Do not trade coefficient residency for extra 256 KiB source passes.

* Generation success does not imply that RECX can use the same register map.
  RECX must also keep multiple syndromes and reconstruction state live, so its
  coefficient-residency opportunity remains narrower.

Overall generation conclusion:

* For the CPUs, compilers, and workloads represented by the measurements in
  these notes, no major generation optimization is currently known with a
  strong reason to be investigated.

* Further generation changes should be driven by new hardware evidence, a
  changed workload, or measurements that identify a concrete bottleneck rather
  than by an assumption that an optimization from another implementation will
  also improve SnapRAID.


GFNI field implementation
-------------------------

* AES GFNI multiplication maps naturally to VGF2P8MULB because the instruction
  uses the AES polynomial.

* Standard RAID polynomial 0x11d cannot use VGF2P8MULB directly.

* RAID GFNI therefore uses VGF2P8AFFINEQB with precomputed affine matrices.

* Multiplication by a fixed coefficient in the RAID 0x11d field does not
  require converting the operand to the AES field. Multiplication by a
  constant is GF(2)-linear, so it is represented directly by an 8x8 GF(2)
  matrix and executed with one VGF2P8AFFINEQB instruction.

* Native AES VGF2P8MULB is therefore not assumed to be inherently faster
  than RAID VGF2P8AFFINEQB for the SnapRAID workload. Relative performance
  depends on the CPU, backend, parity count, and workload.

* The global affine matrix table stores 8 bytes per coefficient.

* RAID GFNI generation additionally stores the affine representation of each
  Cauchy coefficient directly in a disk-major
  raid_gfcauchyaffine_raid[251][5][8] table. This removes the dependent
  Cauchy-coefficient -> global-affine-table lookup from generation while the
  generic affine table remains available for runtime multipliers.

* A generic exhaustive software test validates all coefficient/input pairs
  against the normal RAID GF multiplication table.

* Dedicated hardware affine-matrix validators were intentionally not added.
  Normal generation/recovery tests exercise the hardware path.

* RAID and AES GFNI recovery/generation have polynomial-specific names:

      *_gfni_raid
      *_gfni_aes

  with backend tags:

      gfni
      gfni512

* This naming and registration split is intentional and should be preserved.


GFNI RECX coefficient precomputation
------------------------------------

RAID GFNI fused recovery precomputes direct pointers to the complete 8-byte
affine representation for:

      every surviving-data / selected-syndrome coefficient
      every inverse-matrix reconstruction coefficient

The hot loop therefore uses:

      direct load/broadcast through the precomputed pointer
      VGF2P8AFFINEQB

This keeps the coefficient-byte -> affine-table lookup outside the hot loop
while retaining the compact 8-byte representation.

Important details:

* All 8 bytes form one affine matrix coefficient; it cannot be treated as a
  single-byte coefficient.

* AVX2 and AVX512 RAID GFNI recovery both use this precomputation.

* AES GFNI does not need this conversion because VGF2P8MULB consumes the
  coefficient byte directly.

* The remaining repeated operation is the per-source or per-reconstruction-term
  broadcast of the 8-byte affine matrix for each data chunk.


GFNI pre-expanded affine tables
-------------------------------

* Some high-performance generic erasure-code engines use pre-expanded GFNI
  constants, for example a 32-byte representation ready for vector use.

* For SnapRAID this would increase the RAID affine table from about 2 KiB to
  about 8 KiB.

* In the tested 256 KiB SnapRAID workloads, the compact 8-byte entries plus
  broadcast provided the preferred cache-footprint tradeoff.

* No measured result has justified trading four times the table footprint for
  fewer broadcasts.

* Decision for the currently tested workloads: keep compact 8-byte affine
  matrices.


External implementation comparisons
-----------------------------------

Detailed comparisons are in OTHERS. The implementations studied include
specialized RAID engines, filesystem RAID, and general Reed-Solomon libraries.

The important conclusions are general rather than project-specific.

Classify an external advantage before trying to copy it:

    algorithm advantage
        the method performs genuinely less mathematical work

    implementation advantage
        the same work is scheduled, vectorized, or prepared better

    workload-specific advantage
        the method wins because the external workload has different block,
        cache, parity, update, or latency assumptions

This distinction prevents a good optimization for another engine from becoming
an automatic SnapRAID TODO.

Memory traversal:

* Several modern high-throughput engines confirm that loading one source chunk
  and updating multiple outputs from it is a strong general strategy.

* Other engines deliberately process cache-resident regions and repeatedly
  update cached parity. That can be correct when parity is consumed again
  immediately, but it is a different memory model from SnapRAID generation.

* Whole-region operations that keep one coefficient resident can be efficient
  for generic erasure coding. For SnapRAID, the same loop orientation can
  require rereading large source blocks and is normally a poor trade.

* The preferred order remains: preserve the one-pass source traversal, then use
  spare registers, hoisting, and precomputation to reduce local coefficient
  cost.

Specialization:

* SnapRAID can specialize aggressively because parity is limited to 1..6 and
  the matrix structure is known.

* Generic engines may use large generated N-input/M-output kernel families,
  general decoding matrices, structured solvers, or broader runtime machinery.
  These are valuable for their scale but do not automatically reduce work in
  the N <= 6 SnapRAID domain.

* The most useful external ideas are usually local: fuse multiple outputs,
  precompute coefficient representations, reuse source splits, special-case
  coefficient 0/1 where simple, and exploit algebraic fast paths.

Cache and stores:

* Store policy should be evaluated against output lifetime. Cached stores can
  be advantageous when an algorithm immediately reloads parity; non-temporal
  stores can be advantageous when final parity is queued and consumed later.

* Therefore cached stores in another fast implementation are not evidence that
  SnapRAID should change its MOVNT policy.

Recovery:

* General matrix inversion or structured Cauchy solvers become important at
  larger scale. For at most six missing blocks, moving 256 KiB regions is far
  more important than reducing the tiny matrix-setup cost.

* Algorithmic failure-case specializations are different: if a fixed recovery
  case provably needs fewer GF multiplications, it remains worth measuring.
  The specialized P+Q rec2of2 case is an example.

Runtime and architecture:

* Some engines benchmark implementations at startup, while others dispatch only
  from CPU capabilities. Runtime calibration remains low priority for SnapRAID
  because the kernel set is small and the workload is stable.

* Other engines support additional SIMD families such as SVE, RVV, or VSX.
  These are useful porting references, not hot-path optimization tasks
  without a target platform requirement.


Memory layout, displacement and stride noise
--------------------------------------------

* The memory layout is not a neutral detail.

* Fixed displacement between buffers was observed to help.

* STRIDE_NOISE behavior differs by CPU.

* On Intel, stride noise had little effect in the observed tests.

* On Zen 5, disabling stride noise caused a large throughput reduction.

* This indicates that hardware prefetch/cache behavior is already strongly
  affected by the chosen buffer layout.

* Therefore new software-prefetch or traversal changes should not be assumed to
  help. They must be benchmarked against the existing displacement/stride
  behavior.

* This is another reason to avoid adding extra passes over 256 KiB blocks.


REC1 optimizations
------------------

The normal REC1 case with P is handled via raid_rec1of1() across all backends
(scalar, SSSE3, AVX2, AVX512BW, GFNI, AArch64 NEON, and AArch32 NEON), using
streaming = 0 for temporal stores.

For generic REC1, used when P is unavailable:

* AVX2, AVX512BW, GFNI, SSSE3, SSSE3 extended, AArch64 NEON, and AArch32 NEON
  use single-pass recovery. AVX2/AVX2ext and SSSE3/SSSE3ext use specialized
  Horner kernels for Q and fused selected-syndrome kernels for the other
  parity rows.

* Scalar uses its delta-based REC1 kernel.

REC1 status:

* For the current large-block SnapRAID workload and implemented backends,
  raid_rec1of1() achieves optimal memory saturation for rec1of1/P.

* Generic non-P REC1 is selected per parity count and backend, so single-pass
  SIMD and delta-based scalar implementations are benchmarked independently.

* No major REC1 algorithmic optimization is currently expected.


REC2 optimizations
------------------

REC2 has both fused and delta-based implementations.

* AVX2, AVX512BW, GFNI, and AArch64 NEON use fused syndrome generation for every
  selected parity pair, including P+Q. SSSE3 and SSSE3 extended use fused
  selected-syndrome recovery for generic REC2 and dedicated 1-pass Horner
  kernels for P+Q. AArch32 NEON uses fused recovery for every pair except P+Q.

* Scalar int8 uses delta-based REC2. AArch32 NEON retains a delta-based
  specialized P+Q helper.

Important portability result:

* x86-32 has only the low eight XMM/YMM registers available to these old ISA
  paths.

* Optimizations that depend on xmm8..xmm15 or ymm8..ymm15 must remain restricted
  to CONFIG_X86_64.

REC2 status:

* Generic REC2 uses the fused path across AVX2, AVX512BW, GFNI, SSSE3,
  SSSE3ext, and NEON.

* AVX2, AVX512BW, GFNI, and AArch64 NEON handle P+Q recovery directly inside the
  1-pass fused kernel. SSSE3 and SSSE3ext use dedicated 1-pass Horner rec2of2
  helpers, AArch32 NEON uses its dedicated delta-based rec2of2 helper, and
  scalar int8 handles it via has_p.


Specialized rec2of2 P+Q
-----------------------

For failures of two data disks recovered specifically with P and Q, the system
can be reduced to:

      Dy = C0 * Pd ^ C1 * Qd
      Dx = Pd ^ Dy

This needs only two GF multiplications for Dy, followed by one XOR for Dx,
instead of the generic 2x2 reconstruction with four GF multiplications.

Backends implement this specialization as follows:

* AVX2, AVX512BW, GFNI (AVX2 and AVX512), and AArch64 NEON integrate this
  derivation directly into the unified 1-pass fused kernel (P is accumulated via
  XOR, Q via GF multiplication, Dy is solved with inverse coefficients, and Dx
  derived by XOR with Pdelta).
* Scalar int8 (has_p branch in raid_rec2_int8)
* SSSE3 and SSSE3 ext use dedicated 1-pass Horner kernels
  (raid_rec2of2_ssse3, raid_rec2of2_ssse3ext)
* AArch32 NEON uses the delta-based raid_rec2of2_neon32 helper

The rec2of2 benchmark row directly measures this specialized P+Q fast path.

REC2OF3 and REC2OF4 remain useful control rows because they do not match
ip = { 0, 1 } and therefore continue to exercise the generic REC2 paths.


RECX delta handling
-------------------

RECX computes the selected syndromes directly from stored parity and one scan of
the surviving data for each SIMD chunk.

For non-GFNI paths:

* each completed syndrome is split into low/high nibbles once
* the split values stay in registers or bounded aligned per-chunk scratch space
* the same split values are reused across all reconstructed output rows
* no full-size temporary syndrome block is generated or reloaded

For GFNI paths:

* the byte syndromes remain directly in SIMD registers
* no nibble split is needed

When P is among the selected parity blocks, optimized RECX paths can reconstruct
only nr - 1 missing blocks through the inverse matrix and derive the final block
from the raw P delta by XORing the blocks already reconstructed. This removes
one complete inverse-matrix reconstruction row without adding another
source-data pass.

On the tested x86 implementations, this improved the affected multi-disk
recovery paths by roughly 8% to 12%, depending on the backend and failure count.


RECX shuffle-table pointer precomputation
----------------------------------------

Non-GFNI fused recovery precomputes in C:

      S[s][j] = &raid_gfmulpshufb[A(ip[j], d)][0][0];
      R[j][k] = &raid_gfmulpshufb[V[j * N + k]][0][0];

The hot assembly then only performs:

      load S[s][j] or R[j][k]
      load or broadcast the low table at offset 0
      load or broadcast the high table at offset 16

This has two advantages:

1. It removes coefficient-index address arithmetic from the hot assembly.

2. It keeps C object layout and address calculation in C, leaving the assembly
   focused on SIMD operations.

Users include:

* AVX2 and AVX2 extended selected-syndrome REC1/REC2 and REC3..REC6 paths
* AVX512BW RECX
* SSSE3 and SSSE3 extended selected-syndrome REC1/REC2 and REC3..REC6 paths
* AArch64 and AArch32 NEON fused recovery paths
* Scalar int8 RECX (raid_recX_int8), which precomputes 256-byte raid_gfmul
  pointers S[s][j] and R[j][k], and structures inner loops with explicit
  if (nr >= N) branches so the compiler fully unrolls syndrome accumulation
  and inverse-matrix reconstruction when nr is a compile-time constant.

The remaining cost is that the low/high tables are still loaded or broadcast for
every coefficient of every data chunk.


RECX register state
-------------------

The fused RECX structure uses registers aggressively and falls back to small
per-chunk scratch space where the register file is constrained.

AVX512BW:

* up to six syndrome values are kept as low/high pairs in zmm0..zmm11
* zmm12..zmm15 are reconstruction temporaries/accumulators
* zmm31 keeps the low-nibble mask
* coefficient shuffle tables are still broadcast through R[][] during
  reconstruction

AVX512 RAID GFNI:

* up to six byte syndromes are kept in zmm0..zmm5
* temporary/accumulator registers are used for reconstruction
* coefficient affine matrices come directly through precomputed R[][] pointers
* one 8-byte affine matrix is still broadcast per coefficient per chunk

AVX2 RAID GFNI:

* the two 32-byte halves of up to six syndromes occupy most of the available
  YMM register file
* S[][] and R[][] remove indexed affine-table lookup
* register pressure leaves less room for keeping many coefficient matrices
  resident

AVX2/SSSE3 non-GFNI:

* up to four raw syndromes are accumulated in registers on the specialized
  paths
* larger failure counts use aligned per-chunk syndrome scratch space
* register pressure is significantly higher than AVX512
* coefficient table residency is therefore much more constrained

NEON:

* selected syndromes are accumulated in one surviving-data scan
* S[][] and R[][] provide direct pointers to low/high multiplication tables
* reconstruction loads each coefficient table pair as needed
* AArch64 NEON unrolls non-P REC1 and REC2 to 64-byte chunks across four
  16-byte vector lanes, keeping all syndrome accumulators and Pdelta resident
  in registers without temporary memory buffers
* AArch64 NEON REC3..REC6 use the 32-byte generic kernel across two 16-byte lanes


RECX coefficient residency
--------------------------

The main RECX local optimization question is:

      can repeated coefficient loads/broadcasts be reduced by keeping useful
      coefficient tables or affine matrices resident in spare vector registers?

This is most plausible on AVX512 because of the larger register file.

Possible experiments:

* keep one output row of coefficients resident across data chunks

* keep a useful prefix of coefficient tables resident when N permits it

* on AVX512 RAID GFNI, keep several 8-byte affine matrices broadcast in ZMM
  registers instead of broadcasting them again for every chunk

* on AVX512BW, keep some low/high shuffle-table pairs resident

Constraints:

* Do not add a second pass over the 256 KiB block.

* Do not spill syndrome values or accumulators merely to save coefficient
  broadcasts.

* Do not add large N-specific kernel families unless measurements justify the
  complexity.

* AVX2 and SSSE3 register pressure is already high, so the likely opportunity is
  much smaller there.

* NEON has more architectural registers than old x86 SIMD but RECX already
  uses a significant part of them; any residency experiment should be
  measured carefully.

This is a narrow local problem. Fused syndrome generation, nibble reuse,
coefficient address precomputation, and GFNI affine lookup precomputation are
already implemented.


Recovery optimization policy
----------------------------

For recovery, avoid new architecture-level changes unless a very strong case
appears.

Preferred optimizations are basic/local:

* hoist loop-invariant values
* keep masks/tables/matrices resident when register pressure allows it
* avoid repeated loads or broadcasts
* initialize accumulators from the first useful term
* keep address calculations in C when doing so simplifies inline assembly
* exploit coefficient 0 or 1 only when it stays simple and does not add harmful
  branching or complexity
* use limited spatial unrolling only when it exposes ILP without excessive
  register pressure; SSSE3ext REC1 was faster with a two-way 32-byte unroll
  than with the tested four-way 64-byte version

Avoid unless new measurements provide a concrete reason:

* extra full passes over source blocks
* aggressive spatial unrolling
* software prefetch without clear evidence
* changing MOVNT policy
* major loop-orientation changes
* large pre-expanded lookup tables
* broad runtime calibration machinery


Recovery matrix setup and inversion
-----------------------------------

* Recovery matrices are at most 6x6 for the 1..6 parity implementation.

* The arithmetic cost of constructing and inverting such a matrix is tiny
  compared with reading and writing 256 KiB data regions.

* More elaborate structured Cauchy solvers, inversion caches, or asymptotically
  faster coding methods are therefore not useful performance targets for the
  1..6 parity limit.

* A dedicated multithreaded inversion stress test exists to validate the matrix
  code independently of the data-path SIMD kernels.

* At parity counts far beyond six, matrix construction and structured
  solvers should be re-evaluated together with the whole memory traversal.
  They should not be optimized in isolation.


Inline assembly and portability constraints
-------------------------------------------

The SIMD code intentionally uses multiple small or large asm volatile blocks and
keeps SIMD state live across them.

raid_selftest() is intended to run at every library startup after the RAID mode
has been selected. It verifies only the currently selected mode, so changing
the mode requires running the self test again before that mode is used.

Besides checking the RAID algorithms, it verifies the SIMD implementations
actually selected for that mode on the current CPU, compiler, and ABI,
providing runtime protection against implementation or compiler-related
miscompilation.

Important project constraints:

* Assembly blocks that dereference pointers internally must model memory effects
  appropriately. RECX pointer-based blocks use a "memory" clobber.

* The x86 SSE/AVX end helpers declare SIMD clobbers needed by the project ABI
  strategy, including the Windows x64 callee-saved XMM registers.

* The NEON end helper declares v0..v31 clobbered so that GCC/Clang preserve
  AAPCS64 callee-saved v8..v15 when required.

* The end-of-function dummy-clobber mechanism is a documented project technique.
  It is not a fully explicit extended-asm dataflow model, so future compiler
  portability changes should be reviewed carefully.

* This is primarily a compiler and ABI portability consideration rather than a
  known correctness issue in the current implementation. raid_selftest()
  verifies the selected mode and implementation at startup so that failures
  affecting actual RAID results are detected before normal use.

* Do not assume x86-32 has xmm8..xmm15 or ymm8..ymm15.

* AArch32 NEON is a separate backend. Do not assume the AArch64 register or ABI
  model when changing ARM32 assembly.

* RAID thread-safety requirements are documented. Avoid introducing mutable
  global calibration or dispatch state without preserving those requirements.

* RAID block sizes passed to the SIMD code are multiples of 64, so the
  32-byte and 64-byte unrolled loops do not need tail handling.

These are established implementation constraints, not correctness findings.


Scaling beyond 6 parity levels
------------------------------

The implementation deliberately targets at most six parity levels.

* At this scale, fixed-parity specialization keeps matrix setup small and
  enables tight kernels with explicit register allocation.

* General-purpose erasure-code engines demonstrate other designs, including
  generic matrix kernels, generated multi-input/multi-output kernels, and
  algorithms intended for much larger shard counts.

* Those designs solve a different scaling problem. They should not be adopted
  merely to generalize a domain where the fixed 1..6 structure is useful.

* If the parity limit is increased substantially, generic intrinsics or
  generated kernels should be compared with handwritten specialization.
  Memory traversal
  must remain a first-class design constraint.

* Scaling beyond six parity levels is primarily a functional and architectural
  change, not an unresolved performance problem of the 1..6 implementation.


Short list of ideas already tried or rejected
---------------------------------------------

* AVX512 128-byte generation processing:
  tried, can be slower.

* Non-GFNI AVX512BW Horner:
  tried on Zen 5, no improvement.

* Standalone VPTERNLOGQ substitutions:
  tried on Zen 5, no improvement. This does not reject integrated use as part
  of the profitable GEN2/GEN3 resident-table kernels.

* VPBLENDVB x2:
  tried, no benefit / worse.

* MOVNT -> cached stores:
  not appropriate for the normal queued disk-write workload; keep MOVNT.

* GFNI 32-byte pre-expanded affine constants:
  rejected due to larger cache footprint.

* Split outputs into multiple full source passes:
  rejected because 256 KiB source blocks would be reread.

* Whole-region/output-major rewrites intended mainly to keep one coefficient
  resident:
  poor fit when they add source rereads. Use register residency inside the
  existing traversal instead.

* Two-pass delta generation (raid_delta_gen()):
  completely eliminated in favor of generic single-pass fused recovery and
  dedicated 1-pass Horner kernels. All backends use raid_rec1of1() for rec1of1,
  and fused/Horner recovery for all other combinations. raid_delta_gen() has
  been removed.

* Software prefetch:
  no strong reason to add it given the existing long streams and proven
  sensitivity to displacement/stride tuning.

* Runtime backend microbenchmark selection:
  possible, but low value relative to complexity.

* Large generic N-input/M-output kernel families:
  useful for general erasure coding, but not justified for the fixed
  1..6 parity domain without a concrete benchmark need.

* Structured or asymptotically faster matrix solvers:
  unnecessary at N <= 6 because matrix setup is tiny compared with data
  movement.