Chapter 11: SIMD Fundamentals: Vector Registers on ARM NEON vs. x86 AVX
"Scalar loops treat modern superscalar pipelines like a single-lane country road. SIMD turns that road into an eight-lane autobahn, but only if you align your memory and keep the execution ports fed."
Modern microprocessors are fundamentally vector engines that happen to execute scalar instructions as a compatibility layer. An Apple M-series core or an AMD Zen 5 core contains multiple dedicated 128-bit or 256-bit vector execution pipelines capable of completing multiple Fused Multiply-Add (FMA) operations per clock cycle.
If your software processes data through traditional scalar for loops, you are utilizing less than \(12.5\%\) of the silicon's theoretical floating-point capacity. In this chapter, we bridge the architectural divide between ARM NEON (128-bit fixed) and x86 AVX/AVX2/AVX-512 (256/512-bit variable), analyzing register files, execution ports, alignment hazards, and constructing a unified hardware abstraction layer.
11.1 Silicon Architecture: Register Files & Execution Pipelines
ARM NEON Execution Engine
ARM Advanced SIMD (NEON) provides thirty-two 128-bit vector registers (v0 through v31). Each 128-bit register can be viewed as:
- \(16 \times 8\)-bit integers (uint8x16_t)
- \(8 \times 16\)-bit integers or half-precision floats (float16x8_t)
- \(4 \times 32\)-bit single-precision IEEE 754 floats (float32x4_t)
- \(2 \times 64\)-bit double-precision floats (float64x2_t)
+---------------------------------------------------------------+
| ARM NEON v0 (128 bits) |
+---------------+---------------+---------------+---------------+
| Lane 3 (f32) | Lane 2 (f32) | Lane 1 (f32) | Lane 0 (f32) |
+---------------+---------------+---------------+---------------+
127 96 95 64 63 32 31 0
On high-performance ARM microarchitectures (such as Apple Firestorm/Avalanche), the CPU contains four independent symmetric 128-bit NEON execution pipelines. In a single clock cycle at \(3.5\text{ GHz}\), the core can dispatch and retire four 128-bit FMAs: $\(\text{Peak Single-Core FLOPs} = 4\text{ pipelines} \times 4\text{ lanes} \times 2\text{ FLOPs/FMA} \times 3.5\times 10^9\text{ Hz} = 112\text{ GFLOPS}\)$
x86 AVX2 Execution Engine
Intel and AMD x86-64 architectures utilize the Advanced Vector Extensions (AVX2), providing sixteen 256-bit vector registers (ymm0 through ymm15). Each register hosts:
- \(8 \times 32\)-bit floats (__m256)
- \(4 \times 64\)-bit floats (__m256d)
- \(32 \times 8\)-bit integers (__m256i)
Intel Skylake through Raptor Lake cores provide two dedicated FMA execution ports (Port 0 and Port 1). In a single clock cycle at \(4.0\text{ GHz}\): $\(\text{Peak Single-Core FLOPs} = 2\text{ ports} \times 8\text{ lanes} \times 2\text{ FLOPs/FMA} \times 4.0\times 10^9\text{ Hz} = 128\text{ GFLOPS}\)$
11.2 Memory Alignment & Cache-Line Crossing Hazards
SIMD instructions require rapid streaming of data between L1 data cache and vector registers.
Aligned vs. Unaligned Loads
- Aligned Load (
_mm256_load_ps/vld1q_f32): The memory address \(\mathbf{p}\) satisfies: $\(\mathbf{p} \equiv 0 \pmod{32} \quad (\text{AVX2}) \quad \text{or} \quad \mathbf{p} \equiv 0 \pmod{16} \quad (\text{NEON})\)$ - Unaligned Load (
_mm256_loadu_ps): Any arbitrary address is accepted.
The 64-Byte Cache Line Split Penalty
If an unaligned 256-bit (32-byte) load spans across a 64-byte cache line boundary:
Cache Line 0 [Bytes 0..63] Cache Line 1 [Bytes 64..127]
+-----------------------------------+-----------------------------------+
| ... [Byte 56 | Byte 87] ... |
+-----------------------------------+-----------------------------------+
\___________/
32-byte vector
crosses boundary!
11.3 Instruction Pipeline Latency & Accumulator Unrolling
A common rookie mistake is writing vector loops with a single accumulator:
// Anti-pattern: Single vector accumulator creates pipeline stalls
float32x4_t sum = vdupq_n_f32(0.0f);
for (size_t i = 0; i < N; i += 4) {
sum = vfmaq_f32(sum, vld1q_f32(&a[i]), vld1q_f32(&b[i])); // Dependency on previous sum!
}
The Read-After-Write (RAW) Hazard
An FMA instruction on modern cores has an execution latency of 3 to 4 clock cycles, even though its throughput is 1 cycle (pipelined). When each iteration depends on sum from the previous iteration, the processor cannot dispatch the next FMA until the previous FMA retires:
4-Way Independent Accumulator Unrolling
To saturate dual or quad FMA execution pipelines, software must unroll loops across 4 independent vector accumulators, breaking the dependency chain:
float32x4_t acc0 = vdupq_n_f32(0.0f);
float32x4_t acc1 = vdupq_n_f32(0.0f);
float32x4_t acc2 = vdupq_n_f32(0.0f);
float32x4_t acc3 = vdupq_n_f32(0.0f);
for (size_t i = 0; i < N; i += 16) {
acc0 = vfmaq_f32(acc0, vld1q_f32(&a[i + 0]), vld1q_f32(&b[i + 0]));
acc1 = vfmaq_f32(acc1, vld1q_f32(&a[i + 4]), vld1q_f32(&b[i + 4]));
acc2 = vfmaq_f32(acc2, vld1q_f32(&a[i + 8]), vld1q_f32(&b[i + 8]));
acc3 = vfmaq_f32(acc3, vld1q_f32(&a[i + 12]), vld1q_f32(&b[i + 12]));
}
// Sum accumulators horizontally at the end
float32x4_t total = vaddq_f32(vaddq_f32(acc0, acc1), vaddq_f32(acc2, acc3));
11.4 Designing a Cross-Platform SIMD Micro-Kernel
To prevent littering our codebase with #ifdef __ARM_NEON__, we define a portable, zero-overhead C23 vector micro-kernel wrapper.
#if defined(__ARM_NEON)
#include <arm_neon.h>
typedef float32x4_t rcre_v128_f32;
#define rcre_v128_load(ptr) vld1q_f32(ptr)
#define rcre_v128_store(ptr, vec) vst1q_f32(ptr, vec)
#define rcre_v128_fma(acc, a, b) vfmaq_f32(acc, a, b)
#define rcre_v128_zero() vdupq_n_f32(0.0f)
#define rcre_v128_hsum(vec) vaddvq_f32(vec)
#elif defined(__AVX2__)
#include <immintrin.h>
typedef __m256 rcre_v256_f32;
#define rcre_v256_load(ptr) _mm256_loadu_ps(ptr)
#define rcre_v256_store(ptr, vec) _mm256_storeu_ps(ptr, vec)
#define rcre_v256_fma(acc, a, b) _mm256_fmadd_ps(a, b, acc)
#define rcre_v256_zero() _mm256_setzero_ps()
#endif
11.5 Chapter Project: The 4-Way Pipelined Vector Dot Product
Project Objective
Implement an optimized dot-product engine in rcre_simd_dot.h comparing:
1. Pure scalar loop.
2. 1-accumulator vector loop (RAW stalled).
3. 4-accumulator unrolled vector loop (pipeline saturated).
Applied Tasks
- Verify arithmetic equivalence across scalar and vector implementations within numerical tolerance \(\epsilon \le 10^{-5}\).
- Measure and report execution time, GFLOPS, and speedup factor.
- Validate memory access with AddressSanitizer and UndefinedBehaviorSanitizer.