Skip to content

Chapter 11: SIMD Fundamentals: Vector Registers on ARM NEON vs. x86 AVX

"Scalar loops treat modern superscalar pipelines like a single-lane country road. SIMD turns that road into an eight-lane autobahn, but only if you align your memory and keep the execution ports fed."

Modern microprocessors are fundamentally vector engines that happen to execute scalar instructions as a compatibility layer. An Apple M-series core or an AMD Zen 5 core contains multiple dedicated 128-bit or 256-bit vector execution pipelines capable of completing multiple Fused Multiply-Add (FMA) operations per clock cycle.

If your software processes data through traditional scalar for loops, you are utilizing less than \(12.5\%\) of the silicon's theoretical floating-point capacity. In this chapter, we bridge the architectural divide between ARM NEON (128-bit fixed) and x86 AVX/AVX2/AVX-512 (256/512-bit variable), analyzing register files, execution ports, alignment hazards, and constructing a unified hardware abstraction layer.


11.1 Silicon Architecture: Register Files & Execution Pipelines

ARM NEON Execution Engine

ARM Advanced SIMD (NEON) provides thirty-two 128-bit vector registers (v0 through v31). Each 128-bit register can be viewed as: - \(16 \times 8\)-bit integers (uint8x16_t) - \(8 \times 16\)-bit integers or half-precision floats (float16x8_t) - \(4 \times 32\)-bit single-precision IEEE 754 floats (float32x4_t) - \(2 \times 64\)-bit double-precision floats (float64x2_t)

+---------------------------------------------------------------+
|                      ARM NEON v0 (128 bits)                   |
+---------------+---------------+---------------+---------------+
|  Lane 3 (f32) |  Lane 2 (f32) |  Lane 1 (f32) |  Lane 0 (f32) |
+---------------+---------------+---------------+---------------+
 127          96 95           64 63           32 31            0

On high-performance ARM microarchitectures (such as Apple Firestorm/Avalanche), the CPU contains four independent symmetric 128-bit NEON execution pipelines. In a single clock cycle at \(3.5\text{ GHz}\), the core can dispatch and retire four 128-bit FMAs: $\(\text{Peak Single-Core FLOPs} = 4\text{ pipelines} \times 4\text{ lanes} \times 2\text{ FLOPs/FMA} \times 3.5\times 10^9\text{ Hz} = 112\text{ GFLOPS}\)$

x86 AVX2 Execution Engine

Intel and AMD x86-64 architectures utilize the Advanced Vector Extensions (AVX2), providing sixteen 256-bit vector registers (ymm0 through ymm15). Each register hosts: - \(8 \times 32\)-bit floats (__m256) - \(4 \times 64\)-bit floats (__m256d) - \(32 \times 8\)-bit integers (__m256i)

Intel Skylake through Raptor Lake cores provide two dedicated FMA execution ports (Port 0 and Port 1). In a single clock cycle at \(4.0\text{ GHz}\): $\(\text{Peak Single-Core FLOPs} = 2\text{ ports} \times 8\text{ lanes} \times 2\text{ FLOPs/FMA} \times 4.0\times 10^9\text{ Hz} = 128\text{ GFLOPS}\)$


11.2 Memory Alignment & Cache-Line Crossing Hazards

SIMD instructions require rapid streaming of data between L1 data cache and vector registers.

Aligned vs. Unaligned Loads

  1. Aligned Load (_mm256_load_ps / vld1q_f32): The memory address \(\mathbf{p}\) satisfies: $\(\mathbf{p} \equiv 0 \pmod{32} \quad (\text{AVX2}) \quad \text{or} \quad \mathbf{p} \equiv 0 \pmod{16} \quad (\text{NEON})\)$
  2. Unaligned Load (_mm256_loadu_ps): Any arbitrary address is accepted.

The 64-Byte Cache Line Split Penalty

If an unaligned 256-bit (32-byte) load spans across a 64-byte cache line boundary:

Cache Line 0 [Bytes 0..63]            Cache Line 1 [Bytes 64..127]
+-----------------------------------+-----------------------------------+
|                     ... [Byte 56  |  Byte 87] ...                     |
+-----------------------------------+-----------------------------------+
                          \___________/
                         32-byte vector
                       crosses boundary!
The load cannot be serviced by a single L1 cache hit. The Load-Store Unit (LSU) must split the operation into two micro-operations (\(\mu\text{ops}\)), lock the line fill buffers, query two cache tags, and stitch the data in the register. This introduces a \(4\times\) to \(8\times\) latency penalty and stalls dependent execution ports.


11.3 Instruction Pipeline Latency & Accumulator Unrolling

A common rookie mistake is writing vector loops with a single accumulator:

// Anti-pattern: Single vector accumulator creates pipeline stalls
float32x4_t sum = vdupq_n_f32(0.0f);
for (size_t i = 0; i < N; i += 4) {
    sum = vfmaq_f32(sum, vld1q_f32(&a[i]), vld1q_f32(&b[i])); // Dependency on previous sum!
}

The Read-After-Write (RAW) Hazard

An FMA instruction on modern cores has an execution latency of 3 to 4 clock cycles, even though its throughput is 1 cycle (pipelined). When each iteration depends on sum from the previous iteration, the processor cannot dispatch the next FMA until the previous FMA retires:

\[\text{Throughput} = \frac{1}{\text{Latency}} = \frac{1}{4} = 0.25\text{ FMAs/cycle (75\% silicon idle!)}\]

4-Way Independent Accumulator Unrolling

To saturate dual or quad FMA execution pipelines, software must unroll loops across 4 independent vector accumulators, breaking the dependency chain:

float32x4_t acc0 = vdupq_n_f32(0.0f);
float32x4_t acc1 = vdupq_n_f32(0.0f);
float32x4_t acc2 = vdupq_n_f32(0.0f);
float32x4_t acc3 = vdupq_n_f32(0.0f);

for (size_t i = 0; i < N; i += 16) {
    acc0 = vfmaq_f32(acc0, vld1q_f32(&a[i + 0]),  vld1q_f32(&b[i + 0]));
    acc1 = vfmaq_f32(acc1, vld1q_f32(&a[i + 4]),  vld1q_f32(&b[i + 4]));
    acc2 = vfmaq_f32(acc2, vld1q_f32(&a[i + 8]),  vld1q_f32(&b[i + 8]));
    acc3 = vfmaq_f32(acc3, vld1q_f32(&a[i + 12]), vld1q_f32(&b[i + 12]));
}
// Sum accumulators horizontally at the end
float32x4_t total = vaddq_f32(vaddq_f32(acc0, acc1), vaddq_f32(acc2, acc3));
With 4 independent accumulators, the CPU pipelines all four operations concurrently, achieving 100% execution port saturation.


11.4 Designing a Cross-Platform SIMD Micro-Kernel

To prevent littering our codebase with #ifdef __ARM_NEON__, we define a portable, zero-overhead C23 vector micro-kernel wrapper.

#if defined(__ARM_NEON)
  #include <arm_neon.h>
  typedef float32x4_t rcre_v128_f32;
  #define rcre_v128_load(ptr)            vld1q_f32(ptr)
  #define rcre_v128_store(ptr, vec)      vst1q_f32(ptr, vec)
  #define rcre_v128_fma(acc, a, b)       vfmaq_f32(acc, a, b)
  #define rcre_v128_zero()               vdupq_n_f32(0.0f)
  #define rcre_v128_hsum(vec)            vaddvq_f32(vec)
#elif defined(__AVX2__)
  #include <immintrin.h>
  typedef __m256 rcre_v256_f32;
  #define rcre_v256_load(ptr)            _mm256_loadu_ps(ptr)
  #define rcre_v256_store(ptr, vec)      _mm256_storeu_ps(ptr, vec)
  #define rcre_v256_fma(acc, a, b)       _mm256_fmadd_ps(a, b, acc)
  #define rcre_v256_zero()               _mm256_setzero_ps()
#endif

11.5 Chapter Project: The 4-Way Pipelined Vector Dot Product

Project Objective

Implement an optimized dot-product engine in rcre_simd_dot.h comparing: 1. Pure scalar loop. 2. 1-accumulator vector loop (RAW stalled). 3. 4-accumulator unrolled vector loop (pipeline saturated).

Applied Tasks

  1. Verify arithmetic equivalence across scalar and vector implementations within numerical tolerance \(\epsilon \le 10^{-5}\).
  2. Measure and report execution time, GFLOPS, and speedup factor.
  3. Validate memory access with AddressSanitizer and UndefinedBehaviorSanitizer.