C++ and SIMD (Phase 1 §4 — Sub-Track 1)¶
Deep Dive · Digital Foundations
Make the host preprocessing reference trustworthy.
Parent: C++ and Parallel Computing
Modern C++ gives you high-level syntax with low-level performance. Master the language features first — they are the building blocks of every parallel framework that follows.
Prerequisites: Basic C programming (from Phase 1 §3 OS work). Familiarity with pointers, arrays, functions.
First pass (60–90 minutes): read RAII, move semantics, lambdas, and the scalar/vector example. Then run the native SIMD lab: it tests real vector instructions, awkward array lengths, lane ordering and floating-point rounding. Treat the remaining sections as a reference, not a list to memorize. The first useful deliverable is a preprocessing function that agrees with its scalar reference and handles every input length.
Why Modern C++ First¶
Every parallel computing technology in this curriculum — SIMD intrinsics, OpenMP, oneTBB, CUDA, HIP, SYCL — is built on C++. But not the C++ of the 1990s. Modern C++ (C++11 through C++17) introduced features specifically designed for performance, safety, and parallel programming:
- Lambdas → used in oneTBB, SYCL, modern CUDA, parallel STL
- Move semantics → transfer ownership of a resource; moving a handle does not move bytes between CPU and GPU
- Templates → generic CUDA kernels, CUTLASS, CuTe
- Parallel STL → execution policies that permit parallel or unsequenced work when the implementation can use it
constexpr→ compile-time computation for HPC optimization
Learn these features now. You'll use every single one in Sub-Tracks 2–5.
Section 1: Modern C++17 for Parallel Computing¶
1.1 Type Deduction (auto, decltype)¶
auto — let the compiler figure out the type:
auto x = 10; // int
auto y = 3.14; // double
auto z = vec.begin(); // std::vector<int>::iterator — much cleaner
auto reduces verbosity and prevents type mismatch bugs. Use it for local variables, especially with complex types (iterators, template results).
The problem decltype solves¶
In a generic multiply function, what is the return type?
template<typename T, typename U>
??? multiply(T a, U b) {
return a * b;
}
// int * int → int
// int * double → double
// float * double → double
Pre-C++11 workarounds — both inadequate:
// Option 1: force a type — loses precision, not generic
template<typename T, typename U>
double multiply(T a, U b) { return a * b; }
// Option 2: std::common_type — verbose, breaks for custom operators
template<typename T, typename U>
typename std::common_type<T, U>::type multiply(T a, U b) { return a * b; }
C++11 solution — trailing return type with decltype:
The compiler deduces the exact type of the expression a * b — works for built-in types, user-defined operators, and arbitrarily complex template expressions.
C++14 simplification — decltype inferred from the return statement:
template<typename T, typename U>
auto multiply(T a, U b) { // return type deduced automatically
return a * b;
}
No trailing -> decltype(...) needed. The compiler infers from the return expression.
auto vs decltype — the key difference¶
auto strips references and const. decltype preserves exact type semantics.
int x = 5;
int& ref = x;
auto a = ref; // a is int — reference stripped
decltype(ref) b = x; // b is int& — reference preserved
const int c = 10;
auto d = c; // d is int — const stripped
decltype(c) e = c; // e is const int — const preserved
Rule of thumb:
- Use auto → 90% of the time (local variables, loop iterators, lambda captures)
- Use decltype → when you need exact type semantics (template return types, forwarding, trait-style code)
Comparison: old vs modern¶
| Pre-C++11 | C++11 decltype |
C++14 auto return |
|
|---|---|---|---|
| Return type deduction | Manual, error-prone | Exact expression type | Inferred from return |
| Generic correctness | Limited | Exact | Exact |
| Custom operator support | Hard | Yes | Yes |
| Readability | Verbose | Clean | Cleanest |
Why it matters for parallel computing: CUTLASS, CuTe, oneAPI DPC++, and ROCm all use auto + decltype extensively because:
- Kernels are templated on precision (float, half, int8_t)
- Types of intermediate expressions depend on template parameters
- decltype lets the library express "whatever type a*b produces" without hardcoding it
You will see patterns like decltype(auto), std::declval<T>(), and trailing return types throughout CUDA template libraries — recognizing them now saves significant confusion later.
1.2 Smart Pointers (Memory Safety via RAII)¶
The problem with raw pointers:
int* p = new int(5);
// ... 200 lines of code ...
// Did you remember to delete p? Memory leak.
// Did you delete it twice? Undefined behavior.
The solution — RAII (Resource Acquisition Is Initialization):
#include <memory>
// Exclusive ownership — automatically freed when scope ends
auto p = std::make_unique<int>(5);
// No delete needed. Ever.
// Shared ownership — freed when last reference dies
auto q = std::make_shared<int>(10);
auto r = q; // Reference count = 2
// Freed when both q and r go out of scope
Three smart pointer types:
| Type | Ownership | Overhead | Use when |
|---|---|---|---|
std::unique_ptr |
Exclusive (one owner) | Zero (same as raw pointer) | Default choice. Single owner. |
std::shared_ptr |
Shared (reference counted) | Atomic ref count (control block overhead, impl-dependent) | Multiple owners needed |
std::weak_ptr |
Non-owning observer | Minimal | Break circular references |
Rule: Never use new/delete in modern C++. Use make_unique and make_shared.
Breaking Cycles with std::weak_ptr¶
shared_ptr uses reference counting. The count only reaches zero — and the memory only frees — when no shared_ptr points to the object. If two objects hold shared_ptr to each other, neither count ever reaches zero. This is a reference cycle and it silently leaks memory forever.
The cycle problem:
struct Node {
std::shared_ptr<Node> next; // strong reference
int value;
};
auto a = std::make_shared<Node>(); // a: refcount = 1
auto b = std::make_shared<Node>(); // b: refcount = 1
a->next = b; // b refcount = 2
b->next = a; // a refcount = 2
// a and b go out of scope:
// a's refcount drops to 1 (b->next still holds it)
// b's refcount drops to 1 (a->next still holds it)
// Neither reaches 0 → MEMORY LEAK
The fix — weak_ptr for the back-edge:
weak_ptr observes a shared_ptr without incrementing the reference count. The observed object can still be destroyed normally. Before using a weak_ptr, you must lock() it — this returns a shared_ptr that is either valid (object alive) or empty (object was destroyed).
struct Node {
std::shared_ptr<Node> next; // strong: owns the next node
std::weak_ptr<Node> prev; // weak: observes without owning
int value;
};
auto a = std::make_shared<Node>(); // a: refcount = 1
auto b = std::make_shared<Node>(); // b: refcount = 1
a->next = b; // b refcount = 2 (strong)
b->prev = a; // a refcount still = 1 (weak — no increment)
// a and b go out of scope:
// a's refcount drops to 0 → destroyed → a->next released
// b's refcount drops to 0 → destroyed
// No leak.
// Accessing through weak_ptr — always check if still alive:
if (auto owner = b->prev.lock()) { // lock() returns shared_ptr or nullptr
std::cout << "prev value: " << owner->value << "\n";
} else {
std::cout << "prev was destroyed\n";
}
When cycles appear in practice:
| Pattern | Cycle edge | Fix |
|---|---|---|
| Doubly-linked list | prev pointer |
prev as weak_ptr |
| Tree with parent pointer | parent pointer |
parent as weak_ptr |
| Observer / event system | Subscriber holds reference to publisher | Subscriber stores weak_ptr to publisher |
| Computation graph (ML) | Back-edge from output node to input | Back-edges as weak_ptr |
weak_ptr API:
auto sp = std::make_shared<int>(42);
std::weak_ptr<int> wp = sp; // observe, no refcount increment
wp.expired(); // true if the object was destroyed
wp.use_count(); // same as sp.use_count() (0 if expired)
auto sp2 = wp.lock(); // returns shared_ptr<int> or nullptr — ALWAYS use this
if (sp2) { /* safe to use */ }
Do not dereference a
weak_ptrdirectly. Always call.lock()and check the result. The object could be destroyed betweenexpired()returningfalseand your next line.
Why it matters for parallel computing:
- GPU memory (cudaMalloc/cudaFree) follows the same RAII pattern — wrap in a smart pointer or custom RAII class
- Thread-safe reference counting (shared_ptr) is essential for multi-threaded data sharing
- Computation graphs (PyTorch autograd, JAX jaxpr) are graphs with cycles — real implementations use weak back-edges to avoid memory leaks
- Memory leaks in HPC = your 80 GB GPU runs out of memory mid-training
1.3 Range-Based Loops¶
std::vector<float> data = {1.0, 2.0, 3.0, 4.0};
// Modern (clean, less error-prone)
for (auto& x : data) {
x *= 2.0f;
}
// Old style (verbose, index bugs)
for (int i = 0; i < data.size(); i++) {
data[i] *= 2.0f;
}
Why it matters: Range-based loops work naturally with STL algorithms and parallel STL (std::execution::par). They're also easier for the compiler to auto-vectorize.
1.4 Structured Bindings (C++17)¶
Unpack tuples, pairs, and structs directly:
// Pair unpacking
std::pair<int, float> p = {1, 2.5f};
auto [id, value] = p; // id = 1, value = 2.5f
// Map iteration (clean)
std::map<std::string, int> scores = {{"alice", 95}, {"bob", 87}};
for (auto& [name, score] : scores) {
std::cout << name << ": " << score << "\n";
}
// Function returning multiple values
auto [min_val, max_val] = std::minmax_element(data.begin(), data.end());
Why it matters: Cleaner data handling in performance-critical code. Used in profiling results, benchmark data, multi-return-value functions.
1.5 Lambda Functions (Critical for Parallel Computing)¶
Lambdas are the single most important C++ feature for parallel programming. Every parallel framework uses them.
Syntax:
-captures → which variables from the outer scope to bring in
- parameters → like a normal function
- return_type → optional, usually inferred
- body → code block
Basic lambda:
Capture Modes¶
| Mode | Meaning | Thread-safe? | Use when |
|---|---|---|---|
[=] |
All outer vars by value (copy) | Yes | Default for parallel callbacks |
[&] |
All outer vars by reference | No | Single-threaded only |
[var] |
Specific var by value | Yes | Explicit, recommended |
[&var] |
Specific var by reference | No | Read-only, single-threaded |
[=, &var] |
Mixed: most by value, one by ref | Partial | Fine-grained control |
[var = std::move(obj)] |
Move object into lambda | Yes | Transfer ownership (C++14) |
int multiplier = 10;
auto scale_copy = [=](int x) { return x * multiplier; }; // copy — safe in threads
auto scale_ref = [&](int x) { return x * multiplier; }; // ref — dangerous in parallel
auto scale_specific = [multiplier](int x) { return x * multiplier; }; // best practice
Mutable Lambdas¶
By default, values captured by value are const inside the lambda. Use mutable to modify the internal copy without touching the original:
int a = 10;
auto f = [a]() mutable {
a += 5;
std::cout << a; // prints 15
};
f();
std::cout << a; // still 10 — original unchanged
mutable modifies the lambda's own copy, not the outer variable.
Generic Lambdas (C++14)¶
Use auto parameters to make a lambda templated automatically — no explicit template<> needed:
auto add = [](auto a, auto b) { return a + b; };
add(3, 4); // int + int = 7
add(3.14, 2.71); // double + double = 5.85
add(1.0f, 2.0f); // float + float
This is used heavily in parallel STL, oneTBB, and SYCL to write type-generic kernels.
Thread Safety Rules¶
Capture by value [x] — each thread gets its own copy. Safe.
Capture by reference [&x] — all threads share the same variable. Race condition if any thread writes to it.
std::vector<int> data = {1, 2, 3, 4};
int multiplier = 2;
// Safe: each element read is independent, multiplier captured by value
std::for_each(std::execution::par, data.begin(), data.end(),
[multiplier](int& x) { x *= multiplier; });
// Unsafe: if multiplier were captured by ref and modified by any thread
std::for_each(std::execution::par, data.begin(), data.end(),
[&](int& x) { x *= multiplier; }); // race condition if multiplier changes
Rule: prefer
[=]or named value captures in parallel loops. Use[&]only in single-threaded code or when the captured variables are read-only.
Lambda Lifetime — Dangling Reference Trap¶
If a lambda outlives the scope of its captured references, you get a dangling reference:
std::function<int()> make_lambda() {
int local = 42;
return [&]() { return local; }; // DANGLING — local is destroyed when function returns
}
auto f = make_lambda();
f(); // undefined behavior
Fix 1 — capture by value:
Fix 2 — move ownership into the lambda (C++14):
auto buf = std::make_unique<int>(42);
std::thread t([buf = std::move(buf)]() {
std::cout << *buf; // safe: ownership moved into lambda
});
t.join();
[buf = std::move(buf)] transfers ownership of the unique_ptr into the lambda. The lambda now owns the resource — it cannot outlive it.
Move-Into-Lambda — Deep Dive¶
This is worth understanding completely because it appears constantly in parallel and async code.
Why unique_ptr cannot be captured by value:
std::unique_ptr<int> ptr = std::make_unique<int>(42);
auto f = [ptr]() { std::cout << *ptr; }; // DOES NOT COMPILE
// unique_ptr has no copy constructor — copying would violate unique ownership
unique_ptr expresses "exactly one owner". Capture-by-value would require copying the pointer, creating two owners. The compiler forbids it.
Solution — capture by move (C++14):
auto f = [p = std::move(ptr)]() {
std::cout << *p << "\n"; // lambda owns it, safe to use
};
// What happened to ptr?
std::cout << (ptr ? "not empty" : "empty") << "\n"; // "empty" — ptr is now nullptr
f(); // prints 42
Ownership transfer visualized:
Before move:
ptr ──────────────► [ 42 ] (heap)
After [p = std::move(ptr)]:
ptr ──► nullptr
lambda.p ─────────► [ 42 ] (same heap allocation, new owner)
No copy. No new allocation. The lambda took the backpack — the original holder is now empty.
Thread ownership — each thread gets its own resource:
#include <thread>
#include <memory>
#include <iostream>
int main() {
auto buf = std::make_unique<int[]>(1000); // 4 KB buffer
std::thread t([b = std::move(buf)]() {
b[0] = 42;
std::cout << b[0] << "\n"; // lambda owns the buffer exclusively
});
t.join();
// buf is empty here — the thread owned and (after join) released it
}
In this example only the worker accesses the allocation, so its buffer writes need no mutex. Moving a unique_ptr transfers ownership; it does not invalidate raw pointers or references previously made to that allocation. Such aliases can still cause a data race. Joining also ensures the worker has finished before main continues.
Moving other moveable types:
Any type with a move constructor works — not just unique_ptr:
std::vector<float> weights(1000000); // 4 MB
std::string config = load_config();
auto handle = open_gpu_context(); // hypothetical RAII GPU handle
// Move all three into a thread lambda — zero copying
std::thread worker([
w = std::move(weights),
cfg = std::move(config),
ctx = std::move(handle)
]() {
// worker owns all three resources
run_inference(w, cfg, ctx);
});
worker.join();
| What you move | Why |
|---|---|
unique_ptr<T> |
Exclusive GPU/CPU resource handle |
vector<float> |
Large weight/activation buffer |
string |
Config or serialized model data |
std::thread |
Transfer thread ownership to lambda |
| Custom RAII handle | GPU context, file, socket |
Rule: If a type is non-copyable (or copying is expensive), and the lambda needs to own it or outlive the current scope, use [x = std::move(obj)].
Device Lambdas (CUDA / SYCL)¶
Modern CUDA and SYCL support lambdas on GPU device code:
// CUDA — device lambda inside kernel
__global__ void kernel(float* a, float* b, int N) {
int i = threadIdx.x + blockIdx.x * blockDim.x;
if (i < N) {
auto square = [=](float x) { return x * x; }; // Captures nothing here; x is a parameter
a[i] = square(b[i]);
}
}
// SYCL — lambda IS the kernel
q.parallel_for(sycl::range{N}, [=](sycl::id<1> i) {
C[i] = A[i] + B[i];
});
Capture and address space are separate questions. SYCL kernel lambdas capture by value; copied USM pointers must still address memory accessible to the device. CUDA lambdas created inside device code can refer to device-local objects under CUDA rules. Capturing a host pointer by value does not make its pointee device-accessible. See the SYCL kernel rules and CUDA programming guide.
Lambda Quick Reference¶
| Feature | Syntax | Notes |
|---|---|---|
| Capture all by value | [=] |
Safe in threads |
| Capture all by ref | [&] |
Dangerous in parallel |
| Specific var by value | [var] |
Best practice |
| Specific var by ref | [&var] |
Single-threaded only |
| Generic (auto params) | [](auto x, auto y){} |
C++14, type-generic |
| Mutable | [x]() mutable {} |
Modify internal copy |
| Move into lambda | [x = std::move(obj)] |
Transfer ownership, C++14 |
| Immediately invoked | [&]() { ... }() |
IIFE — run in-place |
Where lambdas are used in this curriculum:
| Framework | Lambda usage |
|---|---|
| Parallel STL | std::sort(std::execution::par, v.begin(), v.end(), [](auto a, auto b) { return a > b; }); |
| oneTBB | tbb::parallel_for(0, N, [&](int i) { C[i] = A[i] + B[i]; }); |
| OpenMP (C++17) | #pragma omp parallel for + lambda tasks: omp_set_num_threads(N); #pragma omp task |
| CUDA (modern) | Device lambdas in __device__ context — [=] only |
| SYCL | q.parallel_for(range, [=](id<1> i) { C[i] = A[i] + B[i]; }); |
Key message: If you don't understand lambdas and captures, you cannot write parallel code in modern C++.
1.6 std::optional, std::variant, std::any (C++17)¶
std::optional — a value that may or may not exist:
#include <optional>
std::optional<int> find_index(const std::vector<int>& v, int target) {
for (int i = 0; i < v.size(); i++) {
if (v[i] == target) return i;
}
return std::nullopt; // "not found" — no magic numbers like -1
}
auto result = find_index(data, 42);
if (result) {
std::cout << "Found at index " << *result << "\n";
}
std::variant — type-safe union:
union vs std::variant — why the old way is dangerous¶
The old C union stores multiple types in the same memory, but has no idea which type is currently active:
union Data {
int i;
float f;
};
Data d;
d.i = 42;
std::cout << d.f << "\n"; // undefined behavior — reading int bits as float
Memory layout of a union — 4 bytes, no type information:
+------------------+
| 42 (int bits) | <- what's stored
+------------------+
^ no type tag <- compiler has no idea which member is active
Reading the wrong member is undefined behavior. No compiler error, no runtime check — just wrong results or crashes, often appearing only in release builds.
std::variant adds a type tag alongside the stored value:
+------------------+
| 3.14f (float) | <- stored data
+------------------+
type tag: float <- runtime always knows which type is active
#include <variant>
std::variant<int, float, std::string> v;
v = 42; // holds int — type tag = int
v = 3.14f; // holds float — type tag = float
v = "hello"; // holds string — type tag = string
// std::visit dispatches to the right lambda branch based on the type tag
std::visit([](auto&& val) { std::cout << val << "\n"; }, v);
union vs std::variant comparison¶
union |
std::variant |
|
|---|---|---|
| Type tracking | None — programmer's responsibility | Runtime type tag stored alongside value |
| Wrong-type read | Undefined behavior, silent | std::bad_variant_access exception |
| Size overhead | Zero (just the max member size) | Max member size + small tag (usually 1–8 bytes) |
| Parallel safety | Dangerous — type confusion across threads | Safe — each element carries its own type info |
| Use in HPC | Avoid in new code | Heterogeneous arrays, command dispatch, AST nodes |
std::visit — type-dispatching with a lambda¶
std::visit calls your lambda with the actually stored type, not a generic variant:
std::variant<int, float> v = 3.14f;
// Generic lambda — works for any type in the variant
std::visit([](auto&& x) {
std::cout << x << "\n"; // prints 3.14
std::cout << typeid(x).name() << "\n"; // prints "f" (float)
}, v);
// Typed overloads using overload pattern (C++17)
std::visit(overloaded{
[](int x) { std::cout << "int: " << x << "\n"; },
[](float x) { std::cout << "float: " << x << "\n"; },
}, v); // prints: float: 3.14
Parallel example — mixed-type array processed safely¶
#include <variant>
#include <vector>
#include <algorithm>
#include <execution>
std::vector<std::variant<int, float>> data = {1, 2.5f, 3, 4.5f};
// Double every element in parallel — type-safe, no UB
std::for_each(std::execution::par, data.begin(), data.end(),
[](auto& val) {
std::visit([](auto&& x) { x *= 2; }, val);
});
// Output: 2 5 6 9
for (auto& v : data)
std::visit([](auto&& x) { std::cout << x << " "; }, v);
Each element carries its own type tag, so parallel threads never confuse int and float even when accessing adjacent elements.
Why it matters for HPC: Compiler IR nodes, kernel dispatch tables, and mixed-precision computation graphs need to represent different types. std::variant + std::visit makes the active alternative explicit and avoids manual unchecked casts. Your visitor still needs valid lifetimes, bounds, and synchronization.
1.7 Move Semantics (Performance Core)¶
The problem — expensive copies:
std::vector<float> create_large_buffer() {
std::vector<float> buf(10000000); // 40 MB
// ... fill buffer ...
return buf; // Does this copy 40 MB? NO — move semantics!
}
Copy vs move:
std::vector<int> a = {1, 2, 3, 4, 5};
// Copy: duplicates all data (expensive)
std::vector<int> b = a; // b is a copy, a unchanged
// Move: transfers ownership (cheap — just pointer swap)
std::vector<int> c = std::move(a); // c owns the data, a is now empty
How move works internally:
Before move:
a → [heap: 1, 2, 3, 4, 5] (owns the buffer)
After std::move(a) → c:
a → nullptr (empty, moved-from)
c → [heap: 1, 2, 3, 4, 5] (now owns the buffer)
No data was copied. Just three pointer/size assignments. O(1) instead of O(n).
When to use std::move:
- Returning large objects from functions (compiler often does this automatically — NRVO)
- Transferring ownership of buffers to another thread
- Moving data into containers: vec.push_back(std::move(large_object));
Why it matters for parallel computing:
- GPU memory transfers are expensive. Move semantics = zero-copy mindset.
- Thread pools move work items between threads without copying.
- CUDA's cudaMemcpyAsync + pinned memory is the GPU equivalent of move semantics.
1.8 Multithreading Basics (std::thread, std::mutex)¶
Launching a thread:
#include <thread>
void compute(int id) {
std::cout << "Thread " << id << " running\n";
}
int main() {
std::thread t1(compute, 1);
std::thread t2(compute, 2);
t1.join(); // Wait for t1 to finish
t2.join(); // Wait for t2 to finish
}
Protecting shared data with mutex:
#include <mutex>
std::mutex mtx;
int shared_counter = 0;
void increment(int n) {
for (int i = 0; i < n; i++) {
std::lock_guard<std::mutex> lock(mtx); // RAII lock
shared_counter++;
} // lock released here automatically
}
Better: std::scoped_lock (C++17) for multiple mutexes:
std::mutex m1, m2;
{
std::scoped_lock lock(m1, m2); // Locks both, prevents deadlock
// ... critical section ...
}
std::atomic for simple shared variables:
#include <atomic>
std::atomic<int> counter{0};
void increment(int n) {
for (int i = 0; i < n; i++) {
counter++; // Thread-safe, no mutex needed
}
}
Why it matters: std::thread and std::mutex are the primitives that OpenMP and oneTBB abstract over. Understanding them helps you debug race conditions and deadlocks in any parallel framework.
1.9 STL Algorithms and Parallel STL (C++17)¶
STL algorithms are pre-built, optimized functions that operate on any container. They eliminate manual loops, prevent index bugs, and — with C++17 execution policies — run in parallel with a single extra argument.
Core STL Algorithms¶
| Algorithm | What it does | Example |
|---|---|---|
std::sort |
Sort elements | sort(v.begin(), v.end()) |
std::for_each |
Apply function to every element | for_each(v.begin(), v.end(), f) |
std::transform |
Apply function, write to output | transform(a, a+n, b, out, f) |
std::reduce |
Parallel-safe sum/product/etc. | reduce(v.begin(), v.end()) |
std::find_if |
Search by predicate | find_if(v.begin(), v.end(), pred) |
std::minmax_element |
Min and max in one pass | auto [lo, hi] = minmax_element(...) |
std::count_if |
Count matching elements | count_if(v.begin(), v.end(), pred) |
std::copy_if |
Filtered copy | copy_if(src, src+n, dst, pred) |
Sequential example — sort and iterate¶
#include <vector>
#include <algorithm>
#include <iostream>
std::vector<int> v = {4, 2, 7, 1, 5};
std::sort(v.begin(), v.end());
for (auto x : v) std::cout << x << " "; // 1 2 4 5 7
Lambda + algorithm — custom behavior injected¶
std::vector<int> v = {1, 2, 3, 4, 5};
// Multiply every element by 2 — no loop index, no off-by-one
std::for_each(v.begin(), v.end(), [](int& x) { x *= 2; });
// v = {2, 4, 6, 8, 10}
// Transform: square each element into a new vector
std::vector<int> sq(v.size());
std::transform(v.begin(), v.end(), sq.begin(), [](int x) { return x * x; });
// Find first element > 5
auto it = std::find_if(v.begin(), v.end(), [](int x) { return x > 5; });
Parallel STL — one argument changes everything¶
C++17 execution policies parallelize any STL algorithm:
#include <algorithm>
#include <execution>
#include <vector>
std::vector<float> data(10'000'000);
// Sequential (default, single core)
std::sort(std::execution::seq, data.begin(), data.end());
// Parallel (multi-threaded across all cores)
std::sort(std::execution::par, data.begin(), data.end());
// Parallel + SIMD (threads + vectorization)
std::sort(std::execution::par_unseq, data.begin(), data.end());
Execution policies:
| Policy | Behavior | When to use |
|---|---|---|
seq |
Sequential | Small data, debugging |
par |
Permits parallel calls; may execute sequentially | Independent work, sufficient work per call |
par_unseq |
Permits parallel and unsequenced calls; does not force SIMD | Independent, vectorization-safe work |
unseq (C++20) |
Permits unsequenced calls in the calling thread | Vectorization-safe work |
Parallel transform, reduce, for_each¶
// Parallel transform: c[i] = a[i] + b[i]
std::transform(std::execution::par,
a.begin(), a.end(), b.begin(), c.begin(),
[](float x, float y) { return x + y; });
// Parallel reduce: sum all elements
float total = std::reduce(std::execution::par, data.begin(), data.end());
// Parallel reduce with custom op: product
float product = std::reduce(std::execution::par,
data.begin(), data.end(), 1.0f, std::multiplies<float>{});
// Parallel for_each + SIMD: sqrt every element
std::for_each(std::execution::par_unseq, data.begin(), data.end(),
[](float& x) { x = std::sqrt(x); });
Manual loop vs STL vs Parallel STL¶
| Manual loop | STL + lambda | Parallel STL | |
|---|---|---|---|
| Code length | Long | Short | Short |
| Bounds responsibility | Validate indices | Validate iterator ranges and destination size | Same range requirements plus concurrent access rules |
| Parallelism | Explicitly arranged | Usually sequential | Permitted by execution policy and implementation |
| SIMD | Compiler may vectorize | Compiler may vectorize | Permitted by policy, never guaranteed |
| Debugging | Hard | Easy | Harder (non-deterministic) |
par_unseqrequirements: Avoid data races, order dependencies, and vectorization-unsafe operations. In particular, a mutex inside a callback can deadlock when callbacks interleave on one thread; protecting a shared update with a lock does not make it suitable forpar_unseq. A read from another element is not itself forbidden if that element is immutable; conflicting reads/writes and ordering assumptions are the problem. See the C++ execution-policy rules.Measure the crossover: Scheduling and synchronization have costs. There is no universal element-count threshold: operation cost, data size, runtime backend, and machine all matter. Compare sequential,
par, andpar_unseqacross the sizes your application actually uses.
Toolchain check: implementation depends on the compiler and standard library/backend, not the language version alone. Check the selected library's documentation and linkage requirements. Compilation does not demonstrate that multiple threads or SIMD instructions were used. Floating-point std::reduce can regroup additions, including with seq; compare within a justified tolerance when order matters.
1.10 Templates & Generic Programming¶
Function templates:
template<typename T>
T add(T a, T b) {
return a + b;
}
add(3, 4); // T = int
add(3.14, 2.71); // T = double
Class templates:
template<typename T, int N>
struct Vector {
T data[N];
T& operator[](int i) { return data[i]; }
};
Vector<float, 4> v; // 4-element float vector
Why templates matter for parallel computing:
- CUDA kernels are often templated on data type and tile size: matmul<float, 32><<<grid, block>>>(...)
- CUTLASS is entirely template-based — configurable GEMM dimensions, precisions, and tiling
- CuTe (CUDA Template Engine) uses templates for layout and copy abstractions
- Zero-cost abstraction: templates are resolved at compile time, no runtime overhead
1.11 constexpr (Compile-Time Computation)¶
Move computation from runtime to compile time:
constexpr int square(int x) {
return x * x;
}
constexpr int tile_size = square(16); // Computed at compile time = 256
// Use in array declarations, template parameters, etc.
float buffer[tile_size]; // float buffer[256] — no runtime cost
constexpr + if (C++17):
template<typename T>
void process(T value) {
if constexpr (std::is_integral_v<T>) {
// Integer-specific code — compiled away for floats
} else {
// Float-specific code — compiled away for integers
}
}
Why it matters for HPC:
- Tile sizes, block dimensions, and buffer sizes in GPU kernels are often compile-time constants
- constexpr enables the compiler to optimize aggressively (unroll loops, eliminate branches)
- Used extensively in CUTLASS for configurable kernel parameters
1.12 How C++ Features Map to Parallel Frameworks¶
| C++ Feature | Used in | How |
|---|---|---|
| Lambdas | oneTBB, SYCL, Parallel STL, modern CUDA | Kernel functions, task bodies, parallel_for callbacks |
| Move semantics | GPU memory, thread pools | Zero-copy buffer transfer, work item passing |
| Templates | CUDA, CUTLASS, CuTe, CK | Configurable kernels, generic algorithms |
constexpr |
CUDA, HLS, HPC libraries | Compile-time tile sizes, buffer dimensions |
auto |
Everywhere | Complex return types, template deduction |
| Smart pointers | Resource management | RAII wrappers for GPU memory, file handles |
| Parallel STL | CPU parallelism | Express permitted execution; measure whether the implementation parallelizes |
std::thread/mutex |
Manual threading | Foundation for OpenMP, oneTBB internals |
std::atomic |
Lock-free algorithms | Counters, flags in multi-threaded code |
| Structured bindings | Data processing | Clean iteration over results, tuples |
Section 2: SIMD on CPUs¶
The gateway to GPU thinking — same operation, multiple data.
Now that you have the C++ foundation, apply it to the first level of parallelism: SIMD vector instructions.
What SIMD Is¶
CPU vector instructions that process 4, 8, 16, or 32 data elements in one instruction:
| Instruction Set | Width | Data per instruction | Platform |
|---|---|---|---|
| SSE (1999) | 128-bit | 4x float32 | x86 |
| AVX (2011) | 256-bit | 8x float32 | x86 |
| AVX-512 (2017) | 512-bit | 16x float32 | x86 (server) |
| ARM NEON | 128-bit | 4x float32 | ARM (Jetson, mobile) |
From SIMD to GPU — One Idea, Three Expressions¶
Every level of the parallel stack applies the same operation to multiple data elements at once. What changes is the width, the programming model, and who manages the indices.
Take the simplest possible operation: c[i] = a[i] + b[i]. Here is how you express it at each level:
Scalar (1 element at a time):
c[0] = a[0] + b[0]
c[1] = a[1] + b[1] ← source expresses one element per iteration
c[2] = a[2] + b[2]
...
CPU SIMD / AVX (8 elements at a time):
c[0..7] = a[0..7] + b[0..7] ← you write i += 8, CPU runs 8 adds in one instruction
c[8..15] = ...
GPU warp / CUDA (32 elements at a time):
each thread handles one i ← you write kernel for one element,
threads are grouped in warps scheduling and active lanes depend on execution
c[0..31] = a[0..31] + b[0..31]
GPU grid (thousands at a time):
blocks are scheduled over time ← residency is limited by hardware resources
// ── Scalar ──────────────────────────────────────────────────────
for (int i = 0; i < n; i++)
c[i] = a[i] + b[i]; // compiler may also vectorize this loop
// ── CPU SIMD (AVX) ───────────────────────────────────────────────
int i = 0;
for (; i <= n - 8; i += 8) { // n >= 0; process only complete vectors
__m256 va = _mm256_loadu_ps(&a[i]);
__m256 vb = _mm256_loadu_ps(&b[i]);
_mm256_storeu_ps(&c[i], _mm256_add_ps(va, vb)); // 8 elements/instruction
}
for (; i < n; ++i) c[i] = a[i] + b[i]; // short arrays and final partial vector
// ── GPU (CUDA) ───────────────────────────────────────────────────
__global__ void add(float* a, float* b, float* c, int n) {
int i = threadIdx.x + blockIdx.x * blockDim.x; // each thread owns one i
if (i < n) c[i] = a[i] + b[i]; // 32 threads run this simultaneously
}
The key differences — not a new idea, just new mechanics:
| Scalar | CPU SIMD (AVX) | GPU (CUDA warp) | |
|---|---|---|---|
| Width | 1 element | 8 floats | 32 threads |
Who picks i |
Your loop counter | Your stride (+= 8) |
Hardware (threadIdx + blockIdx) |
| Nearby storage | Registers and caches | Registers and caches | Registers, shared memory, caches; capacity is device-dependent |
| Slow memory | RAM | RAM | Global memory (VRAM) |
| Hiding latency | Out-of-order execution | ILP across loop iterations | Warp switching |
| Conditional work | Branches or predication | Masks/selects or branches | Different active-lane paths can reduce throughput |
The scaling ladder:
AVX2: 8 floats × ~4 GHz × 2 (FMA) = ~64 GFLOPS / core
GPU warp: 32 threads × thousands of warps → tens of TFLOPS
SIMD is the mental foundation. Once you understand why i += 8 processes 8 elements at once, the GPU model — "just make every thread a lane, and launch millions" — follows naturally.
Auto-Vectorization (Compiler Does It)¶
The easiest path — write a simple loop, let the compiler vectorize:
// This loop is auto-vectorizable
void add_vectors(float* a, float* b, float* c, int n) {
for (int i = 0; i < n; i++) {
c[i] = a[i] + b[i];
}
}
# Check if compiler vectorized your loop
g++ -O2 -march=native -fopt-info-vec-optimized my_code.cpp
# Output: "loop vectorized using 32 byte vectors" = success
Rules for auto-vectorization:
- Independence is the simplest case. Dependencies require analysis: compilers can transform some reductions, recurrences and scans; an arbitrary dependency cannot simply run in independent lanes.
- Simple data types (float, int — not complex objects)
- Aligned memory helps (alignas(32) float data[1024];)
- Use -O2 or -O3 with -march=native
Manual Intrinsics (Full Control)¶
When auto-vectorization fails or you need specific instructions:
#include <immintrin.h>
// Contract: n >= 0; a, b, c each contain n floats; c does not overlap inputs.
// Build for an AVX-capable target. Unaligned loads still require 8 valid floats.
void add_vectors_avx(const float* a, const float* b, float* c, int n) {
int i = 0;
for (; i <= n - 8; i += 8) {
__m256 va = _mm256_loadu_ps(&a[i]); // No 32-byte alignment requirement
__m256 vb = _mm256_loadu_ps(&b[i]);
__m256 vc = _mm256_add_ps(va, vb); // Add 8 pairs at once
_mm256_storeu_ps(&c[i], vc);
}
for (; i < n; ++i) c[i] = a[i] + b[i];
}
Key intrinsics patterns:
| Operation | Intrinsic | What it does |
|---|---|---|
| Load aligned | _mm256_load_ps(ptr) |
Requires 32-byte alignment and 8 readable floats; violating the contract can fault |
| Load unaligned | _mm256_loadu_ps(ptr) |
Load 8 floats (any alignment) — safer default |
| Store | _mm256_store_ps(ptr, v) |
Store 8 floats (aligned) |
| Stream store | _mm256_stream_ps(ptr, v) |
Non-temporal store hint; obey alignment and ordering requirements and measure reuse |
| Add | _mm256_add_ps(a, b) |
8 parallel additions |
| Multiply | _mm256_mul_ps(a, b) |
8 parallel multiplications |
| FMA | _mm256_fmadd_ps(a, b, c) |
8 a*b+c results, one rounding each; may differ from separately rounded operations |
| Broadcast | _mm256_set1_ps(x) |
Fill all 8 lanes with same value |
| Horizontal add | _mm256_hadd_ps(a, b) |
Add adjacent pairs within each 128-bit half |
| Compare | _mm256_cmp_ps(a, b, _CMP_GT_OS) |
Per-lane compare → all-0s or all-1s mask |
| Blend | _mm256_blendv_ps(a, b, mask) |
Select b[i] if the mask lane's sign bit is set; otherwise a[i] |
| Permute (cross-lane) | _mm256_permutevar8x32_ps(v, idx) |
Full cross-lane reorder by index vector |
| Gather | _mm256_i32gather_ps(base, idx, 4) |
Load from non-contiguous addresses |
Alignment contract:
_mm256_load_psrequires 32-byte alignment; violating that requirement can fault, and compiler transformations can change how the failure appears. Prefer_mm256_loadu_pswhen alignment is not established. Both forms require eight readable floats. A pointer aligned to four bytes can still be unsuitable for the aligned intrinsic.FMA accounting:
_mm256_fmadd_ps(a, b, c)computes eighta*b+cresults with one rounding per lane. That is conventionally counted as 16 floating-point operations per vector instruction. An assumed issue rate of one such instruction per cycle at 1 GHz gives 16 GFLOP/s; actual issue rate and sustained frequency depend on the CPU. Neural-network kernels also perform loads, stores, indexing, reductions, and non-FMA operations.
Aligned memory:
// Aligned allocation (required for _mm256_load_ps)
alignas(32) float data[1024];
// Or dynamic allocation on implementations providing std::aligned_alloc:
// #include <cstdlib>; byte count must be a multiple of the alignment.
float* dynamic_data = static_cast<float*>(std::aligned_alloc(32, 1024 * sizeof(float)));
if (!dynamic_data) { /* handle allocation failure before accessing it */ }
// ... use only when allocation succeeded ...
std::free(dynamic_data);
Ten Useful Intrinsics to Learn First¶
These examples cover arithmetic, data movement, selection, and rearrangement. Their order is a learning sequence, not a measured popularity ranking. Look up operand constraints and target features before using an intrinsic.
| # | Intrinsic | Required feature | What it does | Example use |
|---|---|---|---|---|
| 1 | _mm256_add_ps(a, b) |
AVX | Add 8 float32 | Core of every FP loop |
| 2 | _mm256_loadu_ps(ptr) |
AVX | Load 8 floats, unaligned | Default load — no alignment constraint |
| 3 | _mm256_storeu_ps(ptr, v) |
AVX | Store 8 floats, unaligned | Default store |
| 4 | _mm256_fmadd_ps(a, b, c) |
FMA + AVX state support | a*b + c with one rounding |
GEMM, convolution, attention |
| 5 | _mm_add_ps(a, b) |
SSE | Add 4 float32 (128-bit) | SSE compat code, horizontal ops |
| 6 | _mm_set1_epi32(x) |
SSE2 | Broadcast int to all 4 lanes | Loading integer constants |
| 7 | _mm256_cmp_ps(a, b, pred) |
AVX | Compare → bitmask | Conditional selection, NaN handling |
| 8 | _mm_loadu_si128(ptr) |
SSE2 | Load 128-bit integer vector | String/byte processing |
| 9 | _mm_shuffle_epi32(v, imm) |
SSE2 | Reorder 32-bit int lanes | Data rearrangement, AoS→SoA |
| 10 | _mm_popcnt_u32(x) |
POPCNT (its own CPU feature) | Count set bits in 32-bit int | Database filters, Hamming distance |
Patterns to recognize:
_psmeans packed single precision — use these examples to learn FP32 vectors, then compare integer or reduced-precision operations for your application.- Unaligned loads/stores simplify alignment requirements (
_loadu_,_storeu_). They still require valid memory for every accessed lane. Cache-line and page crossings can affect cost; measure on the target CPU. - SSE2 128-bit still common — many codebases use SSE2 for compatibility or for 128-bit sub-operations (horizontal reduction, remainder loops, byte processing).
_mm_shuffle_epi32rearranges integer lanes — useful when data layout differs from the following operation's required lane order._mm_popcnt_u32/u64operates on scalar registers — POPCNT has an independent feature bit; SSE4.2 support alone is not the feature contract to check.
// Inner-loop sketch: n is a multiple of 8; A/B contain n floats;
// acc was initialized, and out has space for 8 floats.
// For arbitrary n, handle the remainder as in add_vectors_avx.
for (int i = 0; i < n; i += 8) {
__m256 a = _mm256_loadu_ps(&A[i]); // rank 2
__m256 b = _mm256_loadu_ps(&B[i]); // rank 2
acc = _mm256_fmadd_ps(a, b, acc); // rank 4
}
_mm256_storeu_ps(out, acc); // rank 3
Reference: Intel Intrinsics Guide. Instruction-set names, lane width, and instruction throughput describe different properties.
Intrinsics by Functional Category¶
A different lens — grouped by what you're trying to do, with both SSE (128-bit) and AVX (256-bit) versions side by side. This is how you look them up when solving a problem.
1. Data Movement & Initialization¶
// Unaligned loads accept any valid float address. Cost depends on cache/page splits and CPU.
__m128 a = _mm_loadu_ps(ptr); // SSE: 4 floats
__m256 b = _mm256_loadu_ps(ptr); // AVX: 8 floats
// Store — write register back to memory
_mm_storeu_ps(ptr, a); // SSE
_mm256_storeu_ps(ptr, b); // AVX
// Broadcast — fill all lanes with one scalar (scaling, bias addition)
__m128 scale = _mm_set1_ps(2.0f); // SSE: [2, 2, 2, 2]
__m256 scale = _mm256_set1_ps(2.0f); // AVX: [2, 2, 2, 2, 2, 2, 2, 2]
// Set individual lanes (note: highest lane first)
__m256 v = _mm256_set_ps(7,6,5,4,3,2,1,0); // reverse order
__m256 v = _mm256_setr_ps(0,1,2,3,4,5,6,7); // natural order (r = reversed param order)
2. Arithmetic¶
// Add / Subtract / Multiply / Divide
__m256 r = _mm256_add_ps(a, b); // a[i] + b[i]
__m256 r = _mm256_sub_ps(a, b); // a[i] - b[i]
__m256 r = _mm256_mul_ps(a, b); // a[i] * b[i]
__m256 r = _mm256_div_ps(a, b); // a[i] / b[i]; consult the target CPU's latency/throughput
// FMA — the most important arithmetic intrinsic
__m256 r = _mm256_fmadd_ps(a, b, c); // a[i]*b[i] + c[i], single instruction
// Integer arithmetic (add 8 × int32)
__m256i r = _mm256_add_epi32(a, b);
__m256i r = _mm256_mullo_epi32(a, b); // low 32 bits of each 32×32 product
_ps= packed single (float32)._pd= packed double._epi32= packed int32. Learn the suffix system once and every intrinsic name becomes self-describing.
3. Comparison & Selection (Branchless Conditionals)¶
Vector code can branch. For independent per-element conditions, a useful pattern is compare → mask → blend. This computes a selection per lane; it does not automatically suppress earlier invalid loads or arithmetic in an unselected expression.
// Step 1: Compare → produces per-lane all-1s (true) or all-0s (false)
__m256 mask = _mm256_cmp_ps(a, b, _CMP_GT_OS); // mask[i] = a[i] > b[i] ? 0xFFFFFFFF : 0
// SSE equivalent with explicit predicate:
__m128 mask = _mm_cmplt_ps(a, b); // mask[i] = a[i] < b[i]
// Step 2: Blend — select elements using mask
__m256 result = _mm256_blendv_ps(if_false, if_true, mask);
// result[i] = mask[i] ? if_true[i] : if_false[i]
// Practical example: clamp to [lo, hi] without any branch
__m256 clamped = _mm256_min_ps(_mm256_max_ps(v, lo), hi);
// Practical example: absolute value without branch
__m256 sign_mask = _mm256_set1_ps(-0.0f); // sign bit only
__m256 abs_v = _mm256_andnot_ps(sign_mask, v); // clear sign bit = |v|
Comparison predicates for
_mm256_cmp_ps: ordered comparisons return false for unordered (NaN) operands. Signaling predicates can set the floating-point invalid flag for a quiet NaN; quiet predicates still signal invalid for a signaling NaN. The flag causes a trap only if the relevant exception is unmasked. Choose the predicate and NaN policy explicitly. Comparison results contain all-zero or all-one bits, which work as blend masks; an arbitrary nonzero integer such as1does not have the required sign bit.
4. Data Shuffling & Reduction¶
// Shuffle — rearrange within 128-bit half (compile-time imm8 control)
__m128 s = _mm_shuffle_ps(a, b, _MM_SHUFFLE(3,2,1,0)); // SSE: pick 4 elements from a and b
__m256 s = _mm256_shuffle_ps(a, b, imm8); // AVX: operates per 128-bit half!
// Horizontal add — adds adjacent pairs (use sparingly, ~3 cycles each)
__m128 h = _mm_hadd_ps(a, b); // [a0+a1, a2+a3, b0+b1, b2+b3]
// Better horizontal sum pattern (see "Horizontal sum" section above)
// hadd is slow — only use it 1-2 times at the very end of a reduction loop
Reduction cost: Accumulate vertically and reduce at the end when the algorithm permits it. Horizontal operations inside a reduction loop can add dependencies and shuffle work. Compare instruction latency with latency and reciprocal throughput with throughput on the same microarchitecture; these are different quantities. Use the Intel Intrinsics Guide and measure the complete loop.
SSE vs AVX Side-by-Side Reference¶
| Category | SSE (128-bit, 4× float) | AVX (256-bit, 8× float) | Purpose |
|---|---|---|---|
| Load | _mm_loadu_ps(ptr) |
_mm256_loadu_ps(ptr) |
Read from memory |
| Store | _mm_storeu_ps(ptr, v) |
_mm256_storeu_ps(ptr, v) |
Write to memory |
| Broadcast | _mm_set1_ps(x) |
_mm256_set1_ps(x) |
Fill all lanes with scalar |
| Add | _mm_add_ps(a, b) |
_mm256_add_ps(a, b) |
Element-wise addition |
| Multiply | _mm_mul_ps(a, b) |
_mm256_mul_ps(a, b) |
Element-wise multiply |
| FMA | _mm_fmadd_ps(a, b, c) |
_mm256_fmadd_ps(a, b, c) |
a*b + c |
| Compare | _mm_cmplt_ps(a, b) |
_mm256_cmp_ps(a, b, pred) |
Vectorized compare → mask |
| Blend | _mm_blendv_ps(a, b, m) |
_mm256_blendv_ps(a, b, m) |
Vectorized if/else |
| Shuffle | _mm_shuffle_ps(a, b, i) |
_mm256_shuffle_ps(a, b, i) |
Rearrange elements |
| Hadd | _mm_hadd_ps(a, b) |
_mm256_hadd_ps(a, b) |
Adjacent-pair add |
When to use SSE vs AVX:
- AVX for wider operations — twice as many FP32 elements per instruction as 128-bit SSE; speedup and latency depend on the CPU and workload
- SSE for compatibility — SSE2 is available on every x86-64 CPU since ~2003
- SSE for remainder loops — after the main AVX loop processes n/8*8 elements, handle the last n%8 with SSE or scalar
Minimal Examples — Each Intrinsic in Isolation¶
Before combining intrinsics, see each one do exactly one thing:
#include <immintrin.h>
// ── Data Movement ──────────────────────────────────────────────────────────
// loadu: load 8 floats from any address
float data[8] = {1, 2, 3, 4, 5, 6, 7, 8};
__m256 v = _mm256_loadu_ps(data); // v = [1, 2, 3, 4, 5, 6, 7, 8]
// set1: broadcast scalar to every lane
__m256 v_pi = _mm256_set1_ps(3.14f); // v_pi = [3.14, 3.14, 3.14, 3.14, 3.14, 3.14, 3.14, 3.14]
// storeu: write register back to array
float out[8];
_mm256_storeu_ps(out, v); // out[] = {1, 2, 3, 4, 5, 6, 7, 8}
// ── Arithmetic ─────────────────────────────────────────────────────────────
__m256 v1 = _mm256_set1_ps(2.0f); // [2, 2, 2, 2, 2, 2, 2, 2]
__m256 v2 = _mm256_set1_ps(3.0f); // [3, 3, 3, 3, 3, 3, 3, 3]
__m256 sum = _mm256_add_ps(v1, v2); // [5, 5, 5, 5, 5, 5, 5, 5]
__m256 product = _mm256_mul_ps(v1, v2); // [6, 6, 6, 6, 6, 6, 6, 6]
__m256 fma_res = _mm256_fmadd_ps(v1, v2, v1); // v1*v2 + v1 = [8, 8, 8, 8, 8, 8, 8, 8]
// ── Comparison & Selection (vectorized if/else) ────────────────────────────
__m256 a = _mm256_setr_ps(1,5,3,7,2,6,4,8);
__m256 b = _mm256_set1_ps(4.0f);
__m256 mask = _mm256_cmp_ps(a, b, _CMP_GT_OS); // mask[i] = a[i] > 4 ? 0xFFFFFFFF : 0
// [0, 1, 0, 1, 0, 1, 0, 1]
__m256 result = _mm256_blendv_ps(b, a, mask); // result[i] = mask[i] ? a[i] : b[i]
// [4, 5, 4, 7, 4, 6, 4, 8]
// ── Shuffling ──────────────────────────────────────────────────────────────
__m256 sv = _mm256_setr_ps(0,1,2,3,4,5,6,7);
// _MM_SHUFFLE(d,c,b,a) → lane 0 = src[a], lane 1 = src[b], lane 2 = src[c], lane 3 = src[d]
__m256 shuffled = _mm256_shuffle_ps(sv, sv, _MM_SHUFFLE(0,1,2,3));
// Result: [3,2,1,0, 7,6,5,4] — reversed within each 128-bit half
// ── Horizontal add (slow — only at loop end) ───────────────────────────────
__m256 hv = _mm256_setr_ps(1,2,3,4,5,6,7,8);
__m256 hsum = _mm256_hadd_ps(hv, hv);
// [1+2, 3+4, 1+2, 3+4 | 5+6, 7+8, 5+6, 7+8] = [3, 7, 3, 7, 11, 15, 11, 15]
Complete Example: Scaled Vector Addition¶
All five core intrinsics working together in a realistic loop — the pattern used in neural network activations, image processing, and signal scaling:
#include <immintrin.h>
// Computes: c[i] = (a[i] + b[i]) * scale for all i
// Processes 8 elements per iteration using AVX
void scaled_add(const float* a, const float* b, float* c, float scale, int n) {
// 1. set1: broadcast the scalar constant once, outside the loop
__m256 v_scale = _mm256_set1_ps(scale);
int i = 0;
for (; i <= n - 8; i += 8) {
// 2. loadu: load 8 elements from each input array
__m256 va = _mm256_loadu_ps(&a[i]);
__m256 vb = _mm256_loadu_ps(&b[i]);
// 3. add_ps: element-wise addition
__m256 vsum = _mm256_add_ps(va, vb);
// 4. mul_ps: scale the result
// Keep this operation order to match the stated scalar reference.
__m256 vres = _mm256_mul_ps(vsum, v_scale);
// 5. storeu: write 8 results back to memory
_mm256_storeu_ps(&c[i], vres);
}
// Scalar remainder for elements that don't fill a full AVX register
for (; i < n; i++) {
c[i] = (a[i] + b[i]) * scale;
}
}
What this demonstrates:
- _mm256_set1_ps outside the loop — compute constants once, reuse every iteration
- _mm256_loadu_ps — no alignment constraint needed
- _mm256_add_ps → _mm256_mul_ps — two arithmetic ops, 8 elements each, back-to-back
- _mm256_storeu_ps — write results
- Remainder loop — handle n % 8 trailing elements cleanly
A different operation, suitable for FMA: a*scale+b is a multiply-add. It is not the same expression as (a+b)*scale; rewriting expressions can change both mathematics and rounding.
ARM NEON Intrinsics (Jetson, Apple Silicon, Mobile)¶
Arm NEON (Advanced SIMD) provides 128-bit vector operations. The examples here target AArch64, including Jetson Orin's Cortex-A78AE CPU. Verify the CPU's optional extensions before using FP16 arithmetic or dot-product instructions. CPU preprocessing and CUDA kernels execute on different processors.
NEON vs x86 SIMD¶
| x86 SSE | x86 AVX2 | ARM NEON | ARM SVE/SVE2 | |
|---|---|---|---|---|
| Width | 128-bit | 256-bit | 128-bit | 128–2048-bit (scalable) |
| FP32 lanes | 4 | 8 | 4 | 4–64 |
| Registers | 16 (XMM) | 16 (YMM) | 32 (V0–V31) | 32 (Z0–Z31) |
| FMA | Separate FMA extension | Separate FMA extension | Yes (AArch64 vfmaq) | Yes |
| Header | <immintrin.h> |
<immintrin.h> |
<arm_neon.h> |
<arm_sve.h> |
| Compiler | Select only required SSE extensions | -mavx2; add -mfma for FMA |
AArch64 -march=armv8-a |
-march=armv8.2-a+sve |
These register counts describe AArch64 and ordinary x86-64 SSE/AVX code; AArch32 differs. Register count alone does not predict performance. The native lab compiles its dispatcher for the baseline CPU and isolates AVX2/FMA functions, so unsupported machines can run the scalar path.
NEON Data Types¶
#include <arm_neon.h>
// Type naming: <base><bits>x<lanes>_t
float32x4_t // 4× FP32 in a 128-bit register
float16x8_t // 8× FP16 in a 128-bit register
int32x4_t // 4× INT32
int8x16_t // 16× INT8
uint8x16_t // 16× UINT8
// "×2" variants — register pairs (used for interleaved load/store)
float32x4x2_t // pair of float32x4_t (8 floats across 2 registers)
NEON Intrinsics — Top 15 Most Used¶
| Intrinsic | Operation | x86 equivalent |
|---|---|---|
vld1q_f32(ptr) |
Load 4 floats | _mm_loadu_ps |
vst1q_f32(ptr, v) |
Store 4 floats | _mm_storeu_ps |
vaddq_f32(a, b) |
a + b (4-wide) | _mm_add_ps |
vsubq_f32(a, b) |
a − b | _mm_sub_ps |
vmulq_f32(a, b) |
a × b | _mm_mul_ps |
vfmaq_f32(acc, a, b) |
acc + a×b (FMA) | _mm_fmadd_ps |
vmaxq_f32(a, b) |
max(a, b) | _mm_max_ps |
vminq_f32(a, b) |
min(a, b) | _mm_min_ps |
vdupq_n_f32(s) |
Broadcast scalar to all 4 lanes | _mm_set1_ps |
vmovq_n_f32(0) |
Zero vector | _mm_setzero_ps |
vcvtq_f32_s32(v) |
INT32→FP32 convert | _mm_cvtepi32_ps |
vcvtq_s32_f32(v) |
FP32→INT32, truncate toward zero | _mm_cvttps_epi32 for finite in-range values |
vaddvq_f32(v) |
Horizontal sum (all 4 lanes) | Compare the generated reduction sequence |
vmaxvq_f32(v) |
Horizontal max | No single instruction! |
vld2q_f32(ptr) |
Interleaved load (AoS→SoA) | No equivalent |
One intrinsic does not imply one instruction. The Arm intrinsic reference maps four-lane
vaddvq_f32to twoFADDPoperations. The lab checks real AArch64 compiler output. Interleaved loads such asvld3qcan help RGB processing, but benchmark the complete loop and verify its tail and row stride.
Example 1: Vector Addition (NEON)¶
#include <arm_neon.h>
void add_vectors_neon(const float* a, const float* b, float* c, int n) {
int i = 0;
for (; i + 4 <= n; i += 4) {
float32x4_t va = vld1q_f32(&a[i]);
float32x4_t vb = vld1q_f32(&b[i]);
float32x4_t vc = vaddq_f32(va, vb);
vst1q_f32(&c[i], vc);
}
// Scalar remainder
for (; i < n; i++) c[i] = a[i] + b[i];
}
Example 2: Dot Product with FMA¶
float dot_neon(const float* a, const float* b, int n) {
float32x4_t acc = vmovq_n_f32(0.0f);
int i = 0;
for (; i + 4 <= n; i += 4) {
float32x4_t va = vld1q_f32(&a[i]);
float32x4_t vb = vld1q_f32(&b[i]);
acc = vfmaq_f32(acc, va, vb); // acc += va * vb (fused multiply-add)
}
float result = vaddvq_f32(acc); // horizontal sum; normally two FADDP instructions
// Scalar remainder
for (; i < n; i++) result += a[i] * b[i];
return result;
}
The reduction is outside the vector loop. Its cost depends on the generated instructions and target CPU, and its addition order can differ from a scalar left-to-right sum.
Example 3: ReLU Activation (INT8, 16-wide)¶
INT8 quantized inference is critical on Jetson — NEON processes 16 INT8 elements per instruction:
void relu_int8_neon(const int8_t* input, int8_t* output, int n) {
int8x16_t zero = vmovq_n_s8(0);
int i = 0;
for (; i + 16 <= n; i += 16) {
int8x16_t v = vld1q_s8(&input[i]);
int8x16_t result = vmaxq_s8(v, zero); // max(v, 0) = ReLU, 16 elements
vst1q_s8(&output[i], result);
}
for (; i < n; i++) output[i] = input[i] > 0 ? input[i] : 0;
}
16 ReLU operations per instruction — this is why quantized INT8 inference on ARM is so fast.
Example 4: RGB Deinterleave (AoS→SoA with vld3q)¶
Image preprocessing on Jetson often starts with RGBRGBRGB... (AoS). NEON's vld3q deinterleaves automatically:
void rgb_to_planar(const uint8_t* rgb, uint8_t* r, uint8_t* g, uint8_t* b, int npixels) {
int i = 0;
for (; i + 16 <= npixels; i += 16) {
// Load 48 bytes (16 pixels × 3 channels) and deinterleave
uint8x16x3_t pixel = vld3q_u8(&rgb[i * 3]);
// pixel.val[0] = R R R R R R R R R R R R R R R R (16 red values)
// pixel.val[1] = G G G G G G G G G G G G G G G G (16 green values)
// pixel.val[2] = B B B B B B B B B B B B B B B B (16 blue values)
vst1q_u8(&r[i], pixel.val[0]);
vst1q_u8(&g[i], pixel.val[1]);
vst1q_u8(&b[i], pixel.val[2]);
}
// Scalar remainder
for (; i < npixels; i++) {
r[i] = rgb[i*3]; g[i] = rgb[i*3+1]; b[i] = rgb[i*3+2];
}
}
On x86, this deinterleave requires multiple shuffle/permute/blend instructions. NEON does it in one load.
Example 5: FP16 Vector Addition (Jetson + Apple Silicon)¶
ARMv8.2-a added native FP16 arithmetic — critical for AI inference:
#include <arm_neon.h>
void add_fp16(const __fp16* a, const __fp16* b, __fp16* c, int n) {
int i = 0;
for (; i + 8 <= n; i += 8) {
float16x8_t va = vld1q_f16(&a[i]); // Load 8× FP16
float16x8_t vb = vld1q_f16(&b[i]);
float16x8_t vc = vaddq_f16(va, vb); // 8-wide FP16 add
vst1q_f16(&c[i], vc);
}
for (; i < n; i++) c[i] = a[i] + b[i];
}
Eight FP16 values fit in a 128-bit register, compared with four FP32 values. Storage is halved for the same element count; arithmetic throughput and accuracy depend on the target and algorithm.
Example 6: Tiled 4×4 Matrix Multiply (NEON)¶
Small matrix multiply is common in on-device transformer attention:
void matmul_4x4_neon(const float* A, const float* B, float* C) {
// Load all 4 columns of B
float32x4_t b0 = vld1q_f32(&B[0]);
float32x4_t b1 = vld1q_f32(&B[4]);
float32x4_t b2 = vld1q_f32(&B[8]);
float32x4_t b3 = vld1q_f32(&B[12]);
for (int i = 0; i < 4; i++) {
float32x4_t a_row = vld1q_f32(&A[i * 4]);
// C[i] = A[i,0]*B[0] + A[i,1]*B[1] + A[i,2]*B[2] + A[i,3]*B[3]
float32x4_t c = vmulq_laneq_f32(b0, a_row, 0); // broadcast A[i,0] × B col 0
c = vfmaq_laneq_f32(c, b1, a_row, 1); // += A[i,1] × B col 1
c = vfmaq_laneq_f32(c, b2, a_row, 2); // += A[i,2] × B col 2
c = vfmaq_laneq_f32(c, b3, a_row, 3); // += A[i,3] × B col 3
vst1q_f32(&C[i * 4], c);
}
}
vmulq_laneq_f32(vec, src, lane) broadcasts one lane of src and multiplies with all 4 lanes of vec. vfmaq_laneq_f32 does the same with fused multiply-add. This pattern is the building block of larger tiled GEMM on ARM.
NEON vs AVX2 Summary¶
| AVX2 (x86) | NEON (ARM) | |
|---|---|---|
| Register width | 256-bit | 128-bit |
| FP32 lanes | 8 | 4 |
| Registers | 16 | 32 |
| Horizontal sum | Inspect generated reduction | Four-lane vaddvq_f32 maps to two FADDPs |
| AoS deinterleave | Loads and shuffles or gathers | Structured loads such as LD3/LD4; costs vary |
| FP16 native | No (convert to FP32) | Yes (ARMv8.2-a) |
| Lane rules | Some shuffles stay within 128-bit halves | Read each operation's lane-selection rules |
| Typical platform | Server, workstation | Jetson, phone, Apple Silicon, Graviton |
When to use NEON vs let the compiler vectorize: Same rule as x86 — try auto-vectorization first (-O2 -march=armv8.2-a). Use manual NEON intrinsics when the compiler fails (complex access patterns, interleaved data, FP16 arithmetic) or when measurements justify tighter control over instruction selection. Intrinsics alone do not guarantee latency; memory contention, scheduling, and clocks still matter.
Arm SVE/SVE2 — a separate optional extension¶
SVE supports vector lengths from 128 to 2048 bits in 128-bit increments. A program written in a vector-length-agnostic style can adapt to the available length without recompiling, provided the CPU and operating system support the required instructions. Not every Arm CPU supports SVE; this is not an automatic upgrade of NEON. See Arm's SVE learning path.
#include <arm_sve.h>
void add_sve(const float* a, const float* b, float* c, int n) {
for (int i = 0; i < n; i += svcntw()) { // svcntw() = FP32 lanes at runtime
svbool_t pred = svwhilelt_b32(i, n); // predicate mask for bounds
svfloat32_t va = svld1(pred, &a[i]);
svfloat32_t vb = svld1(pred, &b[i]);
svst1(pred, &c[i], svadd_f32_m(pred, va, vb));
}
}
// The loop obtains the lane count at runtime instead of assuming a particular width.
Keep the example as an SVE-specific function behind feature dispatch. Jetson Orin's Cortex-A78AE uses NEON, not SVE. Do not infer SVE or SVE2 support from an Arm product name, release date, or compiler accepting -march.
Cross-Lane Limitations (The AVX2 Gotcha)¶
Some 256-bit shuffle and horizontal-add instructions operate independently within 128-bit halves. This is the defined lane mapping of those instructions, not a general claim about how every AVX2 instruction executes inside a CPU. Ordinary packed addition adds each of all eight corresponding lanes.
// shuffle_ps looks like it works on 256-bit, but it only shuffles within each 128-bit half:
__m256 v = {7, 6, 5, 4, 3, 2, 1, 0}; // lanes 0-7
__m256 s = _mm256_shuffle_ps(v, v, 0b00011011);
// Result: {4,5,6,7, 0,1,2,3} — reversed within each half, NOT across the full register
// To truly cross lanes, you need explicit cross-lane moves:
// Swap the two 128-bit halves:
__m256 swapped = _mm256_permute2f128_ps(v, v, 0x01);
// Full cross-lane element reorder (AVX2) — the most flexible option:
__m256i idx = _mm256_set_epi32(0, 1, 2, 3, 4, 5, 6, 7); // reverse order
__m256 reversed = _mm256_permutevar8x32_ps(v, idx); // true 8-element permute
| Operation | Cross-lane? | Notes |
|---|---|---|
_mm256_shuffle_ps |
No | Within each 128-bit half only |
_mm256_hadd_ps |
No | Adjacent pairs within each half |
_mm256_permute2f128_ps |
Yes | Swaps/copies 128-bit halves |
_mm256_permutevar8x32_ps |
Yes | Full 8-element permute (AVX2 only) |
Read the operation's pseudocode in the Intel Intrinsics Guide. AVX already includes cross-half operations such as
permute2f128; the extension's name is not enough to determine lane behavior.
Inspecting SIMD Registers (Debugging)¶
Store the vector into an ordinary array before inspecting lanes. Reading a different active union member is not a portable C++ type-punning technique.
float lanes[8];
__m256 result = _mm256_fmadd_ps(va, vb, vc);
_mm256_storeu_ps(lanes, result);
printf("lane 0: %f, lane 3: %f\n", lanes[0], lanes[3]);
// Or with structured binding in C++17:
auto [l0,l1,l2,l3,l4,l5,l6,l7] = lanes;
Horizontal sum — reducing a vector to a scalar:
This pattern appears everywhere: dot products, attention scores, reductions.
// Sum all 8 floats in a __m256 to a scalar
float hsum(__m256 v) {
// Step 1: add corresponding values from the low and high halves → [a+e,b+f,c+g,d+h]
__m128 lo = _mm256_castps256_ps128(v); // lower 128 bits
__m128 hi = _mm256_extractf128_ps(v, 1); // upper 128 bits
__m128 sum = _mm_add_ps(lo, hi); // add the two halves
// Step 2: horizontal adds until scalar
sum = _mm_hadd_ps(sum, sum); // [a+b+e+f, c+d+g+h, ...]
sum = _mm_hadd_ps(sum, sum); // [total, total, ...]
return _mm_cvtss_f32(sum); // extract lane 0
}
Stream stores for large write-once buffers:
// Non-temporal store: requests a streaming/non-temporal cache policy.
// Sketch preconditions: out is 32-byte aligned, n is a multiple of 8,
// and all n output elements are allocated. Handle a remainder separately.
// Consider it for large output buffers you will not read back soon.
// (e.g., preprocessing output, frame buffer writes)
for (int i = 0; i < n; i += 8) {
__m256 result = /* compute */;
_mm256_stream_ps(&out[i], result); // actual cache/coherence behavior is platform-dependent
}
_mm_sfence(); // Order preceding streaming stores before subsequent stores.
// Cross-thread publication still needs a correct C++ synchronization protocol.
// When to use: write-once buffers > L3 cache size
// When NOT: if you'll read the data back soon (defeats the purpose)
Gather — loading from non-contiguous addresses:
// Gather: load from base_ptr + indices[i] * scale
float table[1024] = { /* lookup table */ };
__m256i indices = _mm256_set_epi32(7, 3, 15, 0, 42, 8, 1, 100); // arbitrary indices
__m256 gathered = _mm256_i32gather_ps(table, indices, 4); // scale=4 (sizeof float)
// Gather cost depends on CPU, cache residency, and address distribution.
// The value: the *computation* after gather runs vectorized (8 FMAs instead of 8 scalar ops).
// Good for: embedding lookups, sparse vector ops, non-sequential access patterns.
Cache Lines and Memory Alignment¶
Caches transfer and track blocks called cache lines. A 64-byte line is common on the x86 CPUs used in this lab, but line size is target-dependent. Instruction alignment requirements are a separate contract: _mm256_load_ps needs 32-byte alignment even if all its bytes are already cached.
Cache line = 64 bytes = 16 floats = 8 doubles = 2 AVX registers
[float 0][float 1]...[float 15] ← one cache line fetch
└─ AVX load ──┘└─ AVX load ──┘ ← if misaligned, spans two lines
Why it matters:
| Scenario | Cache line impact |
|---|---|
_mm256_load_ps aligned |
A 32-byte load stays within one 64-byte line |
_mm256_loadu_ps misaligned |
May cross a line or page; this does not automatically double DRAM traffic |
| False sharing (threads) | Independent writes share a coherence block and cause ownership traffic |
alignas(64) struct |
Guarantees 64-byte alignment; useful when it matches the target's coherence granularity |
// False sharing — avoid this
struct Workers {
int counter_a; // same 64-byte line as counter_b
int counter_b;
};
// Fix: pad to cache line boundary
struct alignas(64) WorkerA { int counter; };
struct alignas(64) WorkerB { int counter; };
Rule: Use unaligned operations unless alignment is established, and never access beyond the allocation. Separate frequently written thread-owned data using the target's coherence granularity; measure before and after padding. alignas(64) alone is not a portable guarantee against false sharing on every CPU.
Data Layout: AoS vs SoA¶
How you arrange data in memory is often the single highest-leverage optimization in SIMD and GPU code — more impactful than instruction selection.
Array of Structs (AoS) — natural, object-oriented, SIMD-unfriendly¶
Each particle is one contiguous object. Easy to pass around, easy to reason about.
struct Particle {
float x, y, z; // position
float vx, vy, vz; // velocity
};
Particle particles[4];
Memory layout — fields interleaved across all particles:
particle 0 particle 1 particle 2 particle 3
[x][y][z][vx][vy][vz][x][y][z][vx][vy][vz][x][y][z][vx][vy][vz][x][y][z][vx][vy][vz]
When you want to add dx to all x-positions:
// AoS: stride between consecutive x values = sizeof(Particle) = 24 bytes
for (int i = 0; i < N; i++)
particles[i].x += dx;
// An AVX load at &particles[0].x picks up: x0,y0,z0,vx0,vy0,vz0,x1,y1
// ^^^ garbage fields in the register
// You wanted: x0,x1,x2,x3,x4,x5,x6,x7 — but they're 24 bytes apart, not contiguous
The x-values are strided in memory. A compiler may use scalar loads, gathers, or wider loads followed by shuffles. Their relative cost depends on the CPU, cache residency, and other fields used by the loop.
Struct of Arrays (SoA) — SIMD-friendly¶
Each field gets its own contiguous array. All x values together, all y values together.
struct Particles {
float x[N], y[N], z[N];
float vx[N], vy[N], vz[N];
};
Particles p;
p.x[0]=1; p.x[1]=4; p.x[2]=7; // all x values contiguous
p.y[0]=2; p.y[1]=5; p.y[2]=8;
Memory layout — each field is a flat contiguous array:
x: [x0][x1][x2][x3][x4][x5][x6][x7]... ← one AVX load = 8 x-values
y: [y0][y1][y2][y3][y4][y5][y6][y7]...
z: [z0][z1][z2][z3]...
vx: [vx0][vx1][vx2]...
Now the SIMD loop is clean — one load, one add, one store:
// SoA: x values are contiguous → direct sequential AVX load
__m256 vdx = _mm256_set1_ps(dx);
int i = 0;
for (; i <= N - 8; i += 8) {
__m256 vx = _mm256_loadu_ps(&p.x[i]); // load x0..x7 — perfectly contiguous
_mm256_storeu_ps(&p.x[i], _mm256_add_ps(vx, vdx)); // add dx to all 8 at once
}
for (; i < N; ++i) p.x[i] += dx;
Side-by-side loop comparison¶
// AoS — compiler may use strided loads/gathers or decline vectorization
for (int i = 0; i < N; i++)
particles[i].x += dx; // stride = 24 bytes between x values
// SoA — compiler auto-vectorizes, or you write AVX explicitly
for (int i = 0; i < N; i++)
p.x[i] += dx; // stride = 4 bytes — perfectly sequential
Layout comparison¶
| AoS | SoA | AoSoA | |
|---|---|---|---|
| Memory pattern | xyzxyzxyz... |
xxx...yyy...zzz... |
[xxxx yyyy][xxxx yyyy]... |
| Per-field SIMD access | Often needs extra loads/shuffles | Contiguous values | Contiguous values within each tile |
| GPU coalescing | Depends on fields and lane access | Often convenient for per-field work | Depends on tile and lane mapping |
| Code readability | High — p[i].x |
Medium — p.x[i] |
Low — tiled indexing |
| Best for | Small structs, OOP | Physics, ML kernels, SIMD | Intel oneDNN, CUTLASS tiling |
When to choose each: - AoS — small datasets, all fields used together, readability matters more than throughput - SoA — large arrays, per-field operations, SIMD loops, GPU kernels - AoSoA — groups of records with contiguous fields within each group; choose tile size from the access pattern and measurements
Deep learning connection: NCHW and NHWC also change which values are adjacent, but neither is universally faster. Tensor Core fragment layouts specify a particular distribution of elements across threads and registers; do not infer that mapping from the label AoSoA. Include any required layout conversion in end-to-end timing.
Experimental SIMD and the C++26 Library¶
The following example uses libstdc++'s Parallelism TS API in <experimental/simd>. It is distinct from the C++26 <simd> API; changing only the header is not a migration plan. Check your library's supported interface. See the GCC TS header documentation and the current C++ SIMD draft.
#include <experimental/simd>
namespace stdx = std::experimental;
// Same non-overlap and n >= 0 contract as add_vectors_avx above.
void add_vectors_ts(const float* a, const float* b, float* c, int n) {
using V = stdx::native_simd<float>; // Auto-selects best width
const int width = static_cast<int>(V::size());
int i = 0;
for (; i <= n - width; i += width) {
V va(&a[i], stdx::element_aligned);
V vb(&b[i], stdx::element_aligned);
V vc = va + vb;
vc.copy_to(&c[i], stdx::element_aligned);
}
for (; i < n; ++i) c[i] = a[i] + b[i];
}
Executed course check (2026-10-18): the AVX and TS functions were compiled with GCC 13.3 using -std=c++17 -O2 -mavx2 -fsanitize=address,undefined and checked against scalar addition for lengths 0, 1, 7, 8, 9, 17, and 101, including pointers offset by one float and output guards. No extra library flag was required for this TS example. Recheck compilation and generated code on another standard library or architecture. -march=native/-mavx2 binaries require a compatible execution target.
Measurement First¶
"Measure, don't guess. The bottleneck is never where you think it is."
Before writing a single intrinsic, profile to confirm what kind of bottleneck you actually have. Optimizing the wrong thing wastes time.
# perf stat — hardware counter overview
perf stat -e instructions,cycles,cache-misses,fp_arith_inst_retired.256b_packed_single \
./my_program
# Key ratios to check:
# IPC alone cannot identify a memory bottleneck.
# Cache misses need an access count, hierarchy level, and latency context.
# FP event names/counting rules are CPU-specific; use perf list first.
# Confirm vectorization with a compiler report and disassembly as well.
# Confirm vectorization happened
g++ -O2 -march=native -fopt-info-vec-optimized my_code.cpp
# "loop vectorized using 32 byte vectors" = success
# VTune (Intel) — visual hotspots and vectorization analysis
vtune -collect hotspots ./my_program
Roofline Model¶
The Roofline model tells you whether your kernel is compute-bound or memory-bound — and therefore whether SIMD helps.
Peak FLOPS = freq × cores × SIMD_width × FMA_factor
Peak BW = DRAM bandwidth (GB/s)
AI = FLOPs / bytes_accessed (arithmetic intensity)
If AI > Peak FLOPS / Peak BW → compute-bound (SIMD / FMA helps)
If AI < Peak FLOPS / Peak BW → memory-bound (better layout / caching helps)
Example for a modern desktop CPU:
Peak AVX2 FP32 = 3.6 GHz × 8 lanes × 2 (FMA) = ~58 GFLOPS/core
Peak DRAM BW = ~50 GB/s
Ridge point AI = 58 / 50 ≈ 1.2 FLOP/byte
Vector add: AI = N FLOP / (3N × 4 bytes) = 1/12 FLOP/byte
(two FP32 reads + one write; excludes write-allocate traffic)
Dot product: AI = 2N FLOP / 2N×4 bytes = 0.25 FLOP/byte → memory-bound
Matrix mult: AI = O(N³) FLOP / O(N²) bytes → compute-bound for large N ✓
Interpretation: These are upper bounds, not measured classifications. The example assumes one 8-lane FP32 FMA issued per cycle on one core; actual issue width and sustained frequency vary. Match a single-core compute roof to bandwidth available to that core. SIMD can improve a memory-heavy loop until it saturates that bandwidth; it does not promise an 8× speedup.
SIMD → GPU: Mental Model Bridge¶
The jump from SIMD to CUDA looks large but the underlying idea is the same. The key difference is who picks the element index:
// ── SIMD mindset: one thread, one instruction covers 8 FP32 elements ──
// Contract: nonnegative n, n elements in each array, disjoint output.
int i = 0;
for (; i <= n - 8; i += 8) {
__m256 va = _mm256_loadu_ps(&a[i]);
__m256 vb = _mm256_loadu_ps(&b[i]);
_mm256_storeu_ps(&c[i], _mm256_add_ps(va, vb));
}
for (; i < n; ++i) c[i] = a[i] + b[i];
// You write the stride (i += 8). The hardware runs 8 lanes silently.
// ── GPU / SIMT mindset: one thread per element, hardware spawns thousands ──
__global__ void add(float* a, float* b, float* c, int n) {
int i = threadIdx.x + blockIdx.x * blockDim.x;
if (i < n) c[i] = a[i] + b[i];
}
// You write one element. The hardware runs 32 (warp) or thousands in parallel.
| Aspect | AVX (CPU SIMD) | CUDA warp (GPU SIMT) |
|---|---|---|
| Parallel width | 8× float32 | 32 threads |
| Register view | A __m256 value holds 8 FP32 lanes |
Per-thread registers; allocation depends on compiled kernel and target |
| Memory | L1/L2 cache | Shared memory / global memory |
| Conditional work | May use masks/selects, branches, or inhibit vectorization | Divergent paths change active lanes and can reduce throughput |
| Programmer model | Explicit lanes (intrinsics) | Implicit (thread index) |
Connection to the Stack¶
- L1 (Application): cuDNN and CUTLASS use vectorized memory loads internally
- L2 (Compiler): MLIR's
vectordialect (Phase 4C) targets exactly this abstraction level - L5 (Architecture): When you design an accelerator's vector unit, you're designing custom SIMD hardware
- Sub-Track 3 (CUDA): GPU "SIMT" is SIMD scaled to thousands of threads — same mental model
Performance Traps and Common Mistakes¶
These are the issues that consume weeks of debugging in real HPC and ML systems. Learn them here.
| Trap | What happens | Fix |
|---|---|---|
shared_ptr in hot loops |
Atomic ref-count inc/dec every iteration → cache line contention | Use raw pointer or unique_ptr inside hot paths; bump shared_ptr count once outside the loop |
| False sharing | Independent writes contend for ownership of the same coherence block | Separate hot writes using the target's cache granularity and measure |
Unaligned _mm256_load_ps |
Segfault (#GP) at runtime, not compile time |
Use _mm256_loadu_ps unless you've verified 32-byte alignment with alignas(32) |
| Branching inside vectorized loops | May use masks, branches, or inhibit vectorization | Inspect the vectorization report and preserve bounds/NaN semantics |
| Memory-bound SIMD | Speedup eventually stops at available bandwidth | Measure the complete loop; consider data reuse and layout |
par_unseq with conflicting accesses |
Data races, invalid ordering, or forbidden synchronization | Read-only neighbors can be valid; overlapping writes and callback synchronization need careful analysis |
| Ignoring NUMA | Remote accesses can raise latency and compete for interconnect bandwidth | Measure placement and first-touch behavior on the actual machine |
Projects¶
- Modern C++ warmup — Write a matrix class using
std::vector, move semantics, templates, and operator overloading. Verify that returning a large matrix from a function uses move (not copy) by checking with-fno-elide-constructors. - Lambda benchmark — Implement the same computation three ways: raw loop,
std::for_eachwith lambda, andstd::transformwith lambda. Benchmark all three with-O2. Verify the compiler generates identical assembly (-S). - Parallel STL sort — Sort 10M floats with
std::execution::seq,par, andpar_unseq. Measure speedup. Plot time vs array size. - Auto-vectorization — Write a dot product loop. Compile with
-O2 -march=native -fopt-info-vec. Check if the compiler vectorized it. If not, restructure the loop until it does. - AVX intrinsics dot product — Implement dot product using
_mm256_fmadd_ps. Benchmark against scalar and auto-vectorized versions. Measure GFLOPS. - Aligned vs unaligned — Benchmark
_mm256_load_ps(aligned) vs_mm256_loadu_ps(unaligned) for a large array. Measure the performance difference. - AoS → SoA benchmark — Implement vector dot product for an array of
Vec3structs (AoS) and compare to the SoA layout version. Observe the speedup gap (expect 10–40× from layout alone). - Horizontal sum — Implement a dot product that uses
_mm256_fmadd_psto accumulate 8 at a time, then uses the horizontal sum idiom to reduce to a scalar. Verify against scalar result. - Tiny MHA block — Implement a batched multi-head attention forward pass for seq=8, head_dim=16, heads=4 using raw AVX2 intrinsics. Pre-transpose weight matrices. Benchmark against scalar version. Observe that the speedup is ~2–3×, not 8× — explain why (memory-bound).
- NEON dot product — Implement dot product on ARM using
vfmaq_f32andvaddvq_f32. Benchmark on Jetson or Apple Silicon. Compare GFLOPS with the AVX2 version. - NEON RGB deinterleave — Use
vld3q_u8to convert RGBRGB... to planar R,G,B arrays. Benchmark against scalar loop. Measure on Jetson with a real camera frame (1920×1080). - NEON INT8 ReLU — Implement
max(x, 0)on 16-wide INT8 vectors. Process 1M elements. Measure throughput (GB/s) and compare to theoretical LPDDR5 bandwidth on Jetson. - NEON FP16 GEMM — Implement a tiled 64×64 FP16 matrix multiply using
vfmaq_f16. Compare throughput with FP32 version. Verify output correctness by upconverting to FP32. - NEON vs auto-vec — Write the same 5 kernels (add, dot, relu, scale, reduce) both as manual NEON and as plain loops compiled with
-O2 -march=armv8.2-a. Compare assembly output on Godbolt. Note where the compiler matches your intrinsics and where it doesn't.
Resources¶
Learning¶
| Resource | What it covers |
|---|---|
| A Tour of C++ (Stroustrup) | Modern C++ overview, concise |
| cppreference.com | Definitive C++ reference |
| C++ Concurrency in Action (Williams) | Threads, atomics, memory model |
| SIMD for C++ Developers (const.me) | Best practitioner tutorial: loads, arithmetic, shuffles, masking, cross-lane gotchas. 23 pages. Read this first. |
| hands-on-simd-programming | Progressive labs from basic intrinsics → image processing → full MHA block → quantized GPT decoder. Run ./runme.sh to see benchmarks. |
| awesome-simd | Curated index of real-world SIMD libraries, tools, blogs, and references |
| Agner Fog Optimization Guides | Definitive reference on CPU micro-architecture, instruction latencies, calling conventions |
ARM / NEON¶
| Resource | What it covers |
|---|---|
| ARM NEON Intrinsics Reference | Official searchable intrinsics guide (like Intel's, but for ARM) |
| ARM Neon Programmer's Guide | Official tutorial: data types, operations, optimization |
| ncnn NEON optimization | Real-world NEON optimization for neural network inference |
| Arm Performance Libraries | Optimized BLAS/LAPACK/FFT for ARM (like MKL for x86) |
| SIMDe | Emulates x86 intrinsics on ARM — run AVX code on Jetson |
Reference and Tooling¶
| Resource | What it covers |
|---|---|
| Intel Intrinsics Guide | Searchable reference for every x86 SIMD intrinsic |
| uops.info | Instruction latency, throughput, and port usage for every CPU micro-architecture |
| Compiler Explorer (godbolt.org) | See assembly output for any C++ code — verify vectorization happened |
| SIMD-Visualiser | Browser-based visual diagram of what each intrinsic does to register contents |
| Felix Cloutier x86 reference | HTML x86/x64 instruction reference |
Portable SIMD Libraries (skip raw intrinsics)¶
| Library | Language | What it is |
|---|---|---|
| Google Highway | C++ | Best portable SIMD: length-agnostic, runtime dispatch, SSE/AVX/AVX-512/NEON/SVE |
| xsimd | C++ | Header-only wrappers for SSE/AVX/AVX-512/NEON |
| SIMDe | C/C++ | Emulates x86 intrinsics on ARM and other targets |
| Intel ISPC | ISPC | C-like language that compiles to optimal SIMD for SSE/AVX/AVX-512/NEON/PS5/Xbox |
Real-World SIMD Libraries to Study¶
| Library | What it does |
|---|---|
| simdjson | JSON parsing at >2 GB/s using SIMD — canonical example of real-world vectorized parsing |
| SimSIMD | SIMD-accelerated similarity measures (cosine, L2) for embedding vectors |
| ncnn | On-device neural network inference with hand-tuned SIMD for ARM and x86 |
| StringZilla | SIMD substring search, fuzzy matching, sorting |
Blogs (practitioners writing production SIMD code)¶
| Blog | Author |
|---|---|
| lemire.me/blog | Daniel Lemire — simdjson, SIMDComp, practical vectorization |
| branchfree.org | Geoff Langdale — branchless and SIMD techniques |
| 0x80.pl/notesen | Wojciech Muła — low-level SIMD algorithms |
Next¶
→ Sub-Track 2 — OpenMP and oneTBB — scale from SIMD to multi-core CPU parallelism.
Make the host preprocessing reference trustworthy¶
Implement RGB-to-planar normalization or an INT8 dot product with a scalar reference. State channel order, stride, alignment, tail handling, signedness, and rounding. Test lengths just below and above the vector width, deliberately unaligned inputs, and saturation boundaries. Inspect the compiler’s vectorization report and generated instructions before crediting a SIMD speedup.
Warm and cold cache cases answer different questions. Use an allocation larger than the last-level cache when investigating DRAM, and distinguish logical bytes from write-allocate traffic. The same preprocessing reference will validate a GPU kernel and eventually the compiler’s layout transformation.
Carry forward: Scalar/SIMD parity corpus, vectorization report, and byte-counted timing. Continue with the next course and reuse that evidence.