Computer Architecture for AI Hardware Engineers¶
Module 2 · Computer Architecture
Understand how CPUs, GPUs, caches, memory, and accelerators actually execute work.
Phase 1 · Section 2 — How processors and memory systems work, and why that shapes every AI accelerator ever built.
Goal: By the end of this section you can read a GPU or NPU architecture paper, identify when an AI workload is limited by computation, data movement, or scheduling, and reason about the trade-offs in any custom accelerator design.
Layers: L5 architecture, with L3 runtime connections. Prerequisites: arrays, multiplication, basic digital logic, and the first numerical lab. Roles: AI systems, accelerator, embedded, and robotics engineers. Next: operating systems and parallel computing.
Learning route: first read §§1–4 and complete the numerical lab; then study §§5–9 and one profiling exercise; finish with a workload case study in §10. Assembly snippets are explanatory unless a test status is stated. The important artifact is a correct workload estimate plus measurements that can confirm or reject it, not a memorized product specification table.
1. The Two Computing Paradigms¶
Processors balance latency, throughput, power, area, and cost. CPU and GPU designs emphasize different kinds of work; neither is universally faster.
| Question | Typical high-performance CPU | Typical GPU |
|---|---|---|
| How is work organized? | Independently scheduled cores and software threads | Many threads grouped for SIMT execution |
| How are stalls hidden? | Caches, speculation, out-of-order execution, and sometimes hardware threads | Caches and scheduling other ready warps |
| What work fits well? | Irregular control flow, small tasks, OS services | Large regular workloads with enough parallel work |
| What can dominate a small robot pipeline? | Sensor handling, synchronization, control, preprocessing | Kernel dispatch, transfer, occupancy, and numerical work |
A CPU core, GPU arithmetic lane, and Tensor Core are different resources. Comparing their counts as if each were an interchangeable processor is misleading. Compare the same workload, precision, correctness criterion, and timing boundary. The CUDA programming model explains the host/device and thread hierarchy.
The transistor budget question: more execution units, larger caches, wider interfaces, and richer control logic all consume silicon and power. A useful architecture sketch labels these resources; it does not assign unsupported area percentages to named products.
AI connection: dense attention over a sequence of length n contains quadratic pairwise work, while projection layers contain matrix multiplications. Small-batch decoding, large-batch prefill, sparse operators, and tiny robot control networks can have different bottlenecks. Predict bytes and operations for the actual shape before deciding where to run it.
First runnable exercise: Numbers, shapes, and bytes walks through a tiny quantized operator and a batch-dependent memory budget, with a CPU reference and answer guidance.
2. Instruction Set Architecture (ISA)¶
The ISA is the contract between software and hardware. Hardware can be redesigned completely (new microarchitecture) as long as the ISA stays compatible — software keeps running.
RISC vs CISC¶
| Property | AArch64 / RISC-V | x86-64 |
|---|---|---|
| Instruction length | AArch64: 32 bits; RISC-V base: 32 bits, with optional compressed encodings | Variable length, up to 15 bytes |
| Memory operands | Arithmetic generally operates on registers; loads/stores move data | Many instructions allow a memory operand; not every instruction does |
| Implementation | May be in-order or out-of-order, narrow or wide | May be in-order or out-of-order, narrow or wide |
| Code density and efficiency | Depend on extensions, compiler, implementation, and workload | Depend on extensions, compiler, implementation, and workload |
The ISA is a software-visible contract, not a power-efficiency score. Modern implementations may translate architectural instructions into internal operations. RISC-V's compressed extension permits 16-bit instructions mixed with 32-bit instructions; “RISC means every instruction is 32 bits” is incorrect.
The Three ISAs You Will Encounter¶
ARM64 (AArch64)
31 general-purpose 64-bit registers (X0–X30) + zero register (XZR)
128-bit SIMD/FP registers V0–V31 (also addressable as S/D/Q)
ADD X0, X1, X2 // X0 = X1 + X2
LDR X0, [X1, #8] // X0 = mem[X1 + 8]
FMLA V0.4S, V1.4S, V2.4S // 4-wide FP32 fused multiply-add (NEON)
Used in: Apple Silicon (M-series), NVIDIA Jetson, Qualcomm Snapdragon, AWS Graviton.
x86-64
16 general-purpose 64-bit registers (RAX–R15)
SIMD: XMM (128b), YMM (256b/AVX2), ZMM (512b/AVX-512)
mov rax, [rbx + 8] // load from memory
vmulps ymm0, ymm1, ymm2 // 8-wide FP32 multiply (AVX2)
vfmadd231ps zmm0, zmm1, zmm2 // 16-wide FP32 FMA (AVX-512)
Used in: Intel/AMD data-centre servers — the dominant platform for model training.
RISC-V
Modular open ISA: RV64I base + extensions (M, F, D, A, V, ...)
32 architectural integer registers in RV64I; x0 always reads as zero
add x1, x2, x3 // x1 = x2 + x3
vsetvli t0, a0, e32,m4 // V extension: set vector length, FP32, 4x LMUL
vfmacc.vv v0, v4, v8 // vector FMA (scalable width)
Used in: Emerging edge AI chips (Tenstorrent, SiFive), custom accelerator control cores.
ISA comparison: check the target, not just its family¶
| Feature | AArch64 | x86-64 baseline | RV64I with optional extensions |
|---|---|---|---|
| Writable integer registers | 31 general-purpose registers | 16 general-purpose registers | 31; x0 is fixed zero |
| Vector facilities | NEON; SVE/SVE2 on supporting targets | SSE; AVX/AVX2/AVX-512 on supporting targets | V extension on supporting targets |
| What to record | CPU model, extensions, ABI, compiler target | CPU model, extensions, ABI, compiler target | Exact ISA string, ABI, compiler target |
Do not compile for an extension just because it exists in the architecture family. Query the actual device and toolchain; save the target options with the benchmark.
3. CPU Microarchitecture — The Pipeline¶
Understanding the CPU pipeline gives you the vocabulary to reason about any processor's performance: GPUs, NPUs, and custom accelerators all face the same fundamental constraints.
3.1 The Classic 5-Stage Pipeline¶
Cycle: 1 2 3 4 5 6 7 8
Instr 1: IF ID EX MEM WB
Instr 2: IF ID EX MEM WB
Instr 3: IF ID EX MEM WB
Instr 4: IF ID EX MEM WB
IF = Instruction Fetch (read from instruction cache)
ID = Instruction Decode + register file read
EX = Execute (ALU, FP unit, address calculation)
MEM = Memory access (load/store)
WB = Write-Back (result → register file)
One instruction finishes per cycle at steady state — ideal throughput is 1 IPC (instructions per cycle).
3.2 Hazards — When the Pipeline Stalls¶
Data hazard: instruction needs a result not yet written back.
ADD X1, X2, X3 // produces X1 in WB (cycle 5)
SUB X4, X1, X5 // needs X1 in ID (cycle 3) — too early!
Solution: forwarding (bypass) routes the EX result directly to the next instruction's input, eliminating the stall in most cases.
Load-use hazard — forwarding can't help when the data isn't ready yet:
LDR X1, [X2] // data available after MEM (cycle 4)
ADD X3, X1, X4 // EX needs X1 in cycle 3 — too early even with forwarding!
→ 1-cycle stall (bubble), then forward from MEM/WB
Control hazard: a branch changes the PC; the instructions already in the pipeline may be wrong.
BEQ X1, X2, Label // branch resolved in EX (cycle 3)
→ already fetched 2 wrong instructions behind it
→ must flush them: 2-cycle penalty for a 5-stage pipe
AI connection: when lanes in a warp follow different paths, some lanes are masked while a path executes. This can reduce useful work per instruction. A branch taken uniformly is different from divergence; removing a branch by doing expensive unnecessary work can be slower. Preserve synchronization correctness and measure the actual kernel.
3.3 Superscalar Execution¶
Fetch and issue multiple instructions per cycle — wider pipeline, more parallelism:
4-wide superscalar (modern x86/ARM):
Cycle 1: Fetch 4 instructions
Cycle 2: Decode 4 → 4 micro-ops
Cycle 3: Issue up to 4 to execution units (if no dependencies)
Cycle 4: Multiple ALUs, FP units, load/store units execute in parallel
Ideal: 4 IPC (limited by dependencies, memory stalls, branch mispredictions)
Observed IPC: measure for the named workload and processor; it need not approach 4
Measure rather than equate widths: fetch, decode, dispatch, issue, and retirement widths describe different pipeline boundaries. Decode width alone is not peak IPC, and IPC depends on the instruction mix, dependencies, cache behavior, and measurement interval. Use a named processor's optimization manual when filling in these values.
3.4 Out-of-Order Execution¶
Modern high-performance CPUs don't wait for slow instructions — they look ahead and execute whatever is ready.
Program order: Execution order:
1. ADD X1, X2, X3 1. SUB X6, X7, X8 (no dependencies, runs first)
2. MUL X4, X1, X5 2. ADD X1, X2, X3 (data ready)
3. SUB X6, X7, X8 3. MUL X4, X1, X5 (waits for ADD result)
4. DIV X9, X4, X10 4. DIV X9, X4, X10 (waits for MUL)
Register renaming eliminates false dependencies:
Source code: After renaming:
ADD R1, R2, R3 ADD P47, P2, P3 ← writes physical reg P47
MUL R4, R1, R5 MUL P48, P47, P5 ← reads P47 (true dependency)
ADD R1, R6, R7 ADD P49, P6, P7 ← writes P49 (new physical reg!)
SUB R8, R1, R9 SUB P50, P49, P9 ← reads P49
Without renaming: 3rd ADD writes R1, creating a WAW hazard with 1st ADD.
With renaming: P47 and P49 are independent — 3rd ADD can execute in parallel with 2nd MUL.
Key hardware structures:
| Structure | Role | Resource limit to investigate |
|---|---|---|
| Reorder Buffer (ROB) | Tracks in-flight instructions and in-order retirement | Number of entries and allocation rules |
| Issue queues / reservation stations | Hold operations until operands and execution resources are ready | Queue capacity and issue ports |
| Register renaming | Separates architectural names from physical storage | Physical registers and rename bandwidth |
| Result forwarding | Makes produced values available to dependents | Bypass paths, latency, and fan-out |
| Load/store queues | Track memory operations and ordering constraints | Queue capacity and memory dependencies |
These structures vary by implementation; a textbook common-data-bus diagram is not a literal floorplan of every modern CPU.
OoO execution pipeline flow:
Fetch → Decode → Rename → Dispatch → Issue → Execute → Complete → Retire
│ │ │ │ │
├─────────┤ │ │ │
in-order │ Reservation Out-of-order │
(front-end) │ Stations → (back-end) │
│ Execution │
│ Units (ALU, in-order │
│ FPU, LSU) (commit) │
│ │
└── ROB tracks program order ──────────────────┘
Commit/Retire: results are computed out-of-order but committed to architectural state in program order. This enables precise exceptions — if an instruction faults, all prior instructions have committed and all later instructions are discarded cleanly.
3.5 Branch Prediction¶
A branch predictor guesses the next control-flow path before the branch resolves. A wrong guess discards speculative work and redirects execution. Accuracy and recovery cost depend on the workload and microarchitecture; measure them for a named processor.
2-bit saturating counter — simplest dynamic predictor:
State: 00 01 10 11
Prediction: not taken not taken taken taken
On a taken outcome: increment, saturating at 11.
On a not-taken outcome: decrement, saturating at 00.
Two opposite outcomes are needed to change prediction from a STRONG state;
one is enough from a weak state. Initialization and branch history matter.
Two-level correlating predictor — uses global branch history to predict:
Global History Register (GHR): last N branch outcomes as a bit string
e.g., GHR = 10110 (last 5 branches: T N T T N)
Index into Pattern History Table (PHT):
index = hash(branch_PC, GHR)
PHT[index] = 2-bit counter
Captures correlations: "this branch is taken if the previous two were also taken"
Longer-history predictors: several tables can track different lengths of prior branch history. Tagged entries help distinguish contexts that share an index. The useful lesson is that prediction depends on patterns, capacity, and aliasing; do not infer a product's exact predictor design or accuracy from this sketch.
Additional predictor hardware:
| Structure | Purpose |
|---|---|
| Branch Target Buffer (BTB) | Caches branch PC → target address (no need to decode to know target) |
| Return Stack Buffer (RSB) | Tracks likely return addresses; nesting and context changes affect prediction |
| Indirect Branch Predictor | Predicts target of jmp [reg] (virtual function calls) |
| Loop Predictor | Counts iterations — predicts loop exit after N iterations |
Illustrative cost model (assumed values, not a processor measurement):
4-wide superscalar × 15 cycles = 60 wasted instruction slots per mispredict
If 5% of instructions are branches with 96% accuracy:
Mispredict rate = 5% × 4% = 0.2% of all instructions
CPI penalty = 0.2% × 15 cycles = 0.03 CPI (seems small, but adds up)
4. Memory Hierarchy — Account for Data Movement¶
Fast arithmetic units are useful only when operands arrive in time. Data movement can limit an AI workload even when peak compute is high; the limiting resource depends on shape, reuse, and implementation.
4.1 The Memory Mountain¶
| Resource | What it holds | Constraint to investigate |
|---|---|---|
| Registers | Thread-local live values | Finite capacity, operand ports, instruction dependencies |
| L1 cache / shared memory | Data close to an execution unit | Locality, banking, capacity, allocation |
| L2 / last-level cache | Data reused across a larger region | Working set, sharing, contention |
| DRAM / HBM | Main working memory | Sustained bandwidth, access latency, capacity |
| Storage | Persistent models and datasets | I/O latency, throughput, filesystem behavior |
Registers do not provide zero-latency computation or unlimited bandwidth. Cache and DRAM timings vary by processor, load, access pattern, and clock. A single CPU/GPU timing table cannot supply constants for every platform. Name the boundary and measure: dependent loads reveal latency; independent streams can reveal throughput. Cache reuse can reduce traffic reaching the slower levels.
4.2 Cache Organisation¶
A cache is divided into sets of ways. An address maps to exactly one set; within the set, it can go into any way.
Address bits: [ Tag ] [ Index ] [ Offset ]
↓ ↓
Compare Select set Byte within line
Illustrative 8-way set-associative cache (64-byte lines, 8 MiB):
Sets = 8 MiB / (8 ways × 64 bytes) = 16,384 sets
Index bits = log2(16,384) = 14
Offset bits = log2(64) = 6
Tag bits = address bits - index bits - offset bits
(use the relevant physical/virtual address width, not pointer width blindly)
Associativity trade-off:
| Type | Miss rate | Hardware cost | Used where |
|---|---|---|---|
| Direct-mapped (1-way) | High (conflict misses) | Minimal | Not common today |
| 4–8 way set-associative | Low | Moderate | L1/L2/L3 caches |
| Fully associative | Lowest | Very high | TLB, small caches |
Three kinds of misses:
- Compulsory (cold): first touch — unavoidable
- Capacity: working set larger than cache — solution: tile/block your algorithm
- Conflict: same index but different tags evict each other — solution: increase associativity or change data layout
4.3 Cache Coherence (Why Multi-Core Is Hard)¶
When multiple cores have private caches, they can hold stale copies of the same memory location. Coherence protocols enforce a consistent view.
MESI protocol — each cache line has one of four states:
M (Modified) — this core wrote it; dirty; no other core has it
E (Exclusive) — clean; only this core has it
S (Shared) — multiple cores have valid read-only copies
I (Invalid) — stale or not present
Transitions:
Core 0 reads X: I → E (if only core) or I → S (if others have it)
Core 0 writes X: any → M; all other cores' copies → I (invalidate)
Core 1 reads X while Core 0 has M: Core 0 flushes, both → S
False sharing — two cores write to different variables that share the same cache line:
struct Counter {
int core0_count; // byte 0–3
int core1_count; // byte 4–7 ← same 64-byte cache line!
};
Core 0 writes core0_count → invalidates Core 1's copy
Core 1 writes core1_count → invalidates Core 0's copy
→ cache line ping-pongs between cores (100+ cycle penalty each time)
Fix: pad to separate cache lines
struct Counter {
alignas(64) int core0_count;
alignas(64) int core1_count;
};
Why this matters for AI: Cache coherence and distributed collectives are different mechanisms. GPU interconnects can enable peer memory access, but their presence does not imply transparent cache coherence across every device and allocation. An all-reduce explicitly combines values and communicates results; its cost depends on bytes, topology, synchronization, and algorithm. Verify the platform memory model rather than extending CPU MESI assumptions to GPU fabrics.
4.4 Virtual Memory and TLB¶
Virtual memory gives each process a private address space, mapped to physical memory by the page table.
Virtual Address → TLB lookup → Physical Address
TLB hit: 1 cycle (translation cached)
TLB miss: page table walk → 50–500 cycles!
TLB hierarchy:
L1 dTLB: 64 entries, 1 cycle
L2 TLB: 512–2048 entries, 5–10 cycles
Page walk: traverse 4-level table in memory (multiple cache accesses)
Huge pages reduce TLB pressure:
4 KiB pages: 4 GiB working set = 1,048,576 pages → 64 entries cover 256 KiB
2 MiB pages: 4 GiB working set = 2,048 pages → 64 entries cover 128 MiB
Analytical coverage examples: 4 GiB, 4 KiB / 2 MiB pages, 64-entry TLB.
Actual page size and translation support depend on the allocation and device.
4.5 DRAM and HBM¶
DDR example: a channel transferring 6,400 million transfers/s over 64 data bits has a theoretical payload rate of 6,400×10⁶ × 8 = 51.2 GB/s. Eight such channels give 409.6 GB/s before protocol losses and workload effects. Installed DIMMs, supported rates, channel population, and access pattern determine the useful rate. Capacity and latency are separate properties.
HBM: stacked DRAM connects through many short, wide interfaces within an advanced package. The number of stacks, interface rate, enabled capacity, and packaging differ by product. Compare an exact SKU's bandwidth with measured sustained traffic rather than assuming a fixed HBM-to-GDDR multiplier.
AI connection: low-batch autoregressive decode often has little reuse of model weights per token. Batching, prefill, attention length, cache behavior, quantization overhead, and communication can change the limiting resource. “All LLM inference is memory-bound” is not a useful universal rule.
5. SIMD — One Instruction, Many Data¶
SIMD applies one instruction to several packed data elements. SIMT exposes threads that may follow different control-flow paths, and matrix units expose matrix operations. These models share data parallelism, but their programming and synchronization rules differ.
5.1 How SIMD Works¶
Scalar (no SIMD): SIMD (4-wide FP32):
a[0] = b[0] * c[0] 4 scalar multiplies [a0 a1 a2 a3] = [b0 b1 b2 b3]
a[1] = b[1] * c[1] × [c0 c1 c2 c3]
a[2] = b[2] * c[2] 1 instruction, 4 results
a[3] = b[3] * c[3]
SIMD widths across ISAs:
| ISA | Extension | Width | FP32 lanes | Fused-multiply-add? |
|---|---|---|---|---|
| ARM64 | NEON | 128 bits | 4 | Yes (FMLA) |
| ARM64 | SVE/SVE2 | 128–2048 bits (scalable) | 4–64 | Yes |
| x86-64 | SSE4.2 | 128 bits | 4 | No |
| x86-64 | AVX2 | 256 bits | 8 | FMA is a separate feature to check |
| x86-64 | AVX-512 | 512 bits | 16 | Yes |
| RISC-V | RVV | 128–65,536 bits (scalable) | variable | Yes |
5.2 Example — Vectorised Dot Product (AVX2)¶
// Scalar: N multiplications + N additions
float dot_scalar(const float* a, const float* b, int N) {
float sum = 0.0f;
for (int i = 0; i < N; i++) sum += a[i] * b[i];
return sum;
}
// AVX2: processes 8 FP32 per iteration
#include <immintrin.h>
float dot_avx2(const float* a, const float* b, int N) {
__m256 acc = _mm256_setzero_ps();
int i = 0;
for (; i + 8 <= N; i += 8) {
__m256 va = _mm256_loadu_ps(&a[i]);
__m256 vb = _mm256_loadu_ps(&b[i]);
acc = _mm256_fmadd_ps(va, vb, acc); // acc += va * vb (8-wide FMA)
}
// horizontal reduction of 8 lanes → scalar
__m128 lo = _mm256_castps256_ps128(acc);
__m128 hi = _mm256_extractf128_ps(acc, 1);
lo = _mm_add_ps(lo, hi);
lo = _mm_hadd_ps(lo, lo);
lo = _mm_hadd_ps(lo, lo);
float result = _mm_cvtss_f32(lo);
for (; i < N; ++i) result += a[i] * b[i]; // scalar tail: no out-of-bounds read
return result;
}
// Requires AVX2 and FMA support; compile with matching flags and check the target.
// Different reduction order and FMA rounding require a stated floating-point tolerance.
Tested here: this example was compiled with GCC 13.3.0, -O2 -mavx2 -mfma -fsanitize=address,undefined, and compared with the scalar reference for N = 0, 1, 7, 8, 9, 17, 101 on an AVX2/FMA-capable x86 host. All comparisons passed with no sanitizer report. This validates the tail handling for those cases, not a speedup claim.
5.3 SIMD → GPU SIMT¶
A GPU warp (32 threads) executing the same instruction at the same time is SIMD taken to 32-wide, with the additional twist that each "lane" is an independent thread with its own register state.
CPU SIMD (AVX-512): GPU SIMT (warp of 32):
1 instruction 1 instruction
16 FP32 lanes 32 "lanes" = 32 threads
all same operation all same operation (PC)
same data register each thread has own registers
mask/select per vector lane per-thread paths can diverge
Tensor Cores perform matrix multiply-accumulate work. A supported wmma::mma_sync call describes a cooperative warp operation on matrix fragments; it is not 4,096 independent scalar lanes or a promise of one machine instruction. Fragment shapes and instruction forms depend on type and architecture.
6. GPU Architecture (Conceptual)¶
The CUDA programming model is covered in Section 4. Here we look at the hardware — what the silicon actually does.
6.1 SM: The Streaming Multiprocessor¶
The GPU is a collection of SMs. Every kernel launch distributes thread blocks across available SMs.
A concrete memory/occupancy example is A100 (compute capability 8.0): an SM supports up to 64 resident warps, a 64K-entry 32-bit register file, and a 192 KB combined L1/shared-memory resource, with shared-memory carveout up to 164 KB. These are hardware ceilings, not guarantees that any one kernel reaches them. Register use, shared memory, and block size can reduce occupancy. Compute capability 8.6 has different limits; do not mix its 48-warp ceiling into an A100 diagram. NVIDIA Ampere tuning guide.
6.2 SIMT Execution and Warp Scheduling¶
A warp scheduler selects an eligible warp to issue an instruction. Resident threads retain their state in the SM register file, so selecting another ready warp does not require an OS-style context save. Issue opportunities still depend on dependencies, execution pipelines, scheduling rules, and available work; not every warp executes every cycle.
Cycle 100: Warp 0 issues LOAD (takes 200+ cycles to return)
Cycle 101: Warp scheduler picks Warp 1 (compute-bound, no stall)
Cycle 102: Warp 2
Cycle 103: Warp 3
...
Cycle 300: Warp 0's data arrives; it becomes eligible again
This timeline is illustrative, not a measured latency. Other ready work can hide some latency, but extra occupancy helps only while it provides useful independent work. Both CPUs and GPUs also depend on caches and data reuse.
6.3 GPU Memory Hierarchy¶
| CUDA resource | Scope and purpose | Common mistake |
|---|---|---|
| Registers | Per-thread values backed by a finite SM register file | Assuming every local variable stays in a register |
| Local memory | Per-thread address space backed by device memory and caches | Assuming “local” means physically nearby |
| Shared memory | Explicitly managed block storage; some features extend access across a cluster | Forgetting barriers, bank conflicts, or capacity limits |
| L1 / L2 caches | Hardware-managed reuse within the architecture's hierarchy | Treating every global load as an HBM transaction |
| Global memory | Device-wide allocation space | Ignoring transfers, access patterns, and lifetime |
| Constant / texture access paths | Specialized read behavior and caching | Assuming their performance is identical for every access pattern |
A tiled matrix multiply stages values for reuse. Whether a value is fetched once from HBM depends on the tiling, cache behavior, and full execution, not merely on using shared memory. Query limits for the exact GPU and kernel.
6.4 Tensor Cores¶
Tensor Cores accelerate matrix multiply-accumulate (MMA). The following 16×16×16 operation is a mathematical example; the compiler may lower it to multiple architecture-specific instructions:
D = A × B + C
A: 16×16 matrix (FP16 or BF16)
B: 16×16 matrix (FP16 or BF16)
C: 16×16 accumulator (FP32)
D: 16×16 result (FP32)
Peak throughput depends on data type, accumulation mode, sparsity assumptions, device SKU, and clock. A matrix-unit peak and scalar FP32 peak describe different operations; they are not an application speedup measurement. Frameworks may dispatch to cuBLAS, cuBLASLt, cuDNN, compiler-generated kernels, or other paths. Inspect the actual call and generated kernel rather than assuming every matrix operation becomes a particular wmma instruction.
7. AI Accelerator Design Patterns¶
7.1 Systolic Arrays¶
A systolic array is a grid of processing elements (PEs) where data moves between neighbours in a scheduled pattern. Array shape, data type, buffering, and dataflow are architecture choices; they are not fixed across every TPU or accelerator generation.
Input matrix A rows flow →→→→→→→→
Weight matrix B cols flow ↓↓↓↓↓↓↓
PE(0,0) → PE(0,1) → PE(0,2) → ...
↓ ↓ ↓
PE(1,0) → PE(1,1) → PE(1,2) → ...
↓ ↓ ↓
PE(2,0) → PE(2,1) → PE(2,2) → ...
Each PE: accumulator += A_val × B_val
Result location and movement depend on the chosen dataflow.
Potential benefit: nearby reuse reduces repeated transfers.
Account for input skew, pipeline fill/drain, partial sums, and edge tiles.
A matrix instruction specifies an operation, not a public transistor-level layout. Do not infer that a Tensor Core is a particular systolic array merely from the fragment shape.
7.2 Dataflow Architectures¶
In a dataflow view, operations become eligible when their inputs are available. An accelerator can keep intermediate values near the operations that consume them. CPUs and GPUs also retain intermediates in registers and caches; the comparison below deliberately illustrates a poor unfused memory path, not an unavoidable CPU/GPU behavior.
Von Neumann: Dataflow:
LOAD A from HBM A ──► multiply ──► add ──► result
LOAD B from HBM B ──► ↑
MUL C = A × B C ──────►┘
STORE C to HBM
LOAD C from HBM
LOAD D from HBM
ADD E = C + D
STORE E to HBM ← no HBM round-trips for intermediate values
SambaNova SN40L, Cerebras WSE-3, and Graphcore IPU are commercial dataflow architectures for AI.
7.3 NPUs — Neural Processing Units¶
An NPU dedicates hardware and software support to selected neural-network operations. Support can vary by operator, tensor shape, precision, layout, and runtime version. Unsupported operations may fall back to a CPU or GPU and introduce copies or synchronization.
| Question before deployment | Evidence |
|---|---|
| Can the model run correctly? | Supported-operator report and reference comparison |
| Does it run on the NPU throughout? | Partition/fallback report and runtime trace |
| Does it help this robot? | End-to-end age, power, thermal, and quality measurements |
| Can the result be reproduced? | Exact SoC, runtime/compiler, model, and configuration revisions |
A vendor TOPS figure alone does not establish model throughput or energy efficiency. Count supported precision and sparsity assumptions, then benchmark the workload within its latency and accuracy constraints.
8. The Roofline Model — Your Most Important Analysis Tool¶
The roofline model compares a compute ceiling with a data-movement ceiling. It helps form a bottleneck hypothesis; dispatch overhead, dependencies, synchronization, and poor utilization can keep an implementation below both roofs.
8.1 Arithmetic Intensity¶
Arithmetic Intensity (AI) = FLOPs executed / Bytes transferred from memory
Examples:
Vector addition y = a + b: 1 FLOP / 12 bytes = 0.08 FLOP/byte (memory-bound)
Matrix multiply C = A × B (N=1000): 2N³ FLOPs / (3N² × 4 bytes)
= 2×10⁹ / 12×10⁶ ≈ 167 FLOP/byte (ideal one-read/one-write traffic)
Compare intensity with the chosen device/precision roof; 167 is not automatically compute-bound.
FP32 matrix-vector multiply: about 2 FLOP / 4 weight bytes = 0.5
FP16 matrix-vector multiply: about 2 FLOP / 2 weight bytes = 1.0
(large matrix, no weight reuse, input/output traffic omitted)
Attention: derive from batch, heads, lengths, dtype, and actual tiling
8.2 The Roofline¶
TFLOPS (log)
│ ───────────────── Assumed compute roof (989 TFLOPS)
│ ╱
│ ╱ ╲── compute-bound region
│ ╱ (more compute = more speed)
│ ╱
│ ╱ ← memory-bandwidth roof slope = HBM BW × AI
│ ╱ (memory-bound region: more bandwidth = more speed)
└──────────────────────────────────────────── Arithmetic Intensity (FLOP/byte)
↑
Illustrative ridge ≈ 989 TFLOPS / 3.35 TB/s ≈ 295 FLOP/byte
Replace both assumptions with the named device and precision mode.
Below this ridge, the bandwidth roof is lower than the selected compute roof.
This predicts a ceiling; latency, dependencies, and low utilization may dominate first.
8.3 Where AI Workloads Live¶
Follow the changing shape instead of assigning an operator one permanent intensity:
| Workload change | Predicted effect to test | What may invalidate it |
|---|---|---|
| Reuse weights across a batch | More operations per weight byte | Queue delay, memory capacity, insufficient kernel support |
| Fuse two elementwise operators | Fewer intermediate reads/writes | Register pressure, repeated work, changed numerical behavior |
| Tile a matrix multiply | More reuse near execution units | Small or irregular shapes, capacity limits, edge handling |
| Quantize FP16 weights to INT8 | Half the ideal payload bytes | Scales, metadata, padding, conversion, accuracy loss |
| Increase sequence length | More attention work and cache traffic | Attention variant, caching, tiling, and sparsity |
A robot's latency budget may rule out batching even when batching improves throughput. Use the freshness lab to see how a queue changes capture-to-output age.
9. Performance Analysis and Profiling¶
9.1 Amdahl's Law¶
Not all code parallelises. Amdahl's Law predicts the maximum speedup from parallel execution.
Speedup = 1 / ((1 − P) + P/N)
P = fraction of baseline execution TIME that benefits from ideal parallelism
N = number of cores/processors
Example (2% serial, 98% parallel):
| Cores | Speedup | Efficiency |
|---|---|---|
| 1 | 1.0× | 100% |
| 4 | 3.77× | 94% |
| 16 | 12.31× | 77% |
| 64 | 28.32× | 44% |
| ∞ | 50.0× | → 0% |
The serial 2% becomes the bottleneck. No amount of cores can exceed 50× speedup.
For AI workloads: communication, imbalance, pipeline bubbles, input loading, and changed batch size can all violate ideal scaling assumptions. Report useful work per second and quality at a fixed workload, then separate strong scaling (fixed total work) from weak scaling (work grows with device count). A fixed “typical efficiency” is not a prediction for your system.
9.2 Profiling Tools¶
Command status: the following are upstream-style exercise templates for your own ./program; they were not run here. Install compatible tools, query available counters, and account for profiler overhead. The Nsight Compute profiling guide explains metric selection, replay, and collection overhead.
CPU profiling (Linux):
# Hardware counter sampling
perf stat -e cache-references,cache-misses,L1-dcache-misses,branches,\
branch-misses,instructions,cycles ./program
# Record + flamegraph (the two Perl scripts require a separate FlameGraph checkout)
perf record -F 99 -g ./program
perf script | stackcollapse-perf.pl | flamegraph.pl > profile.svg
GPU profiling (NVIDIA):
# Timeline of kernels, memory transfers, CPU-GPU sync
nsys profile -o trace.nsys-rep ./program
nsys-ui trace.nsys-rep
# Per-kernel detailed metrics (roofline, occupancy, memory throughput)
ncu --set full -o report.ncu-rep ./program
ncu-ui report.ncu-rep
Cache simulation:
Benchmarking checklist:
- Record CPU/GPU, clock/power policy, temperature, OS, compiler, and command line. Decide whether the question concerns normal deployment behavior or a controlled clock experiment.
- Record thread placement and background load; pin threads only when it matches the experiment.
- Separate initialization/cold starts from steady state and choose sufficient warmup.
- Save raw repeated samples, sample count, median, and tail observations. Include mean or other statistics when they answer the stated question.
- Check correctness before timing and repeat the comparison on an unfamiliar shape.
9.3 Bottleneck Diagnosis¶
| Observation | Competing explanations | Next measurement |
|---|---|---|
| Low compute utilization | Memory stalls, dependencies, tiny grid, dispatch gaps | Timeline, bytes moved, eligible warps / CPU stalls |
| Many last-level misses | Large streaming workload or poor locality | Requested bytes, cache reuse, tiling comparison |
| Low IPC | Dependencies, cache misses, branch recovery, instruction mix | Cycles and relevant stall/branch counters |
| Low occupancy | Resource limits or small launch | Register/shared-memory allocation and launch size |
| Low memory throughput | Too little parallel work, uncoalesced access, cached data | Traffic at each cache/memory boundary and transaction efficiency |
No single percentage identifies a root cause. For example, low occupancy can still accompany an efficient kernel with enough instruction-level parallelism.
Worked hypothesis: a row-major naive matrix multiply repeatedly accesses a column of B. Tiling may improve locality, but the size of the gain depends on cache, compiler, shape, and implementation. Start with zero-initialized C, compare with a reference, test edge tiles, and report actual results. Do not attach invented GFLOP/s values to an unmeasured snippet.
10. Workload Case Studies for AI and Robotics¶
These are design exercises with stated assumptions, not measurements from named products. They connect the architecture vocabulary to a decision you can defend.
10.1 A small robot policy¶
Assume a dense layer with 64 FP32 inputs and 32 outputs, used once per observation. Weights occupy 64×32×4 = 8192 bytes, and the layer performs about 4096 FLOPs. A high GPU peak says little about this tiny task: launch, synchronization, and transfers may dominate. A CPU may already hold the weights in cache.
Experiment: compare one warmed CPU call with a GPU call, including the transfers actually required by the application. Keep a second GPU-only timing if useful, but label its smaller boundary. Add the result to the sensor-to-action budget. Do not infer that every small model belongs on a CPU: a fused pipeline with data already on a GPU is a different case.
10.2 Camera perception¶
Assume one uncompressed 640×480 RGB8 frame at 30 frames/s. Payload alone is 640×480×3 = 921,600 bytes/frame, or 27.648 MB/s. Three full copies add traffic even before resizing and inference. Metadata, stride padding, and transport framing are excluded from this estimate.
Experiment: draw where capture, resize, inference, and publication run. Count read/write bytes for each buffer copy and state which memory boundary they cross. Record capture time and the final publication time. An optimization that reduces kernel time while adding another frame of queue delay can make the robot's observation older.
10.3 A batched language-model projection¶
Assume an FP16 1024×1024 projection. With one input vector, weights occupy 2 MiB and the operation has roughly 1 FLOP per weight byte. Processing eight vectors while reusing those weights increases that weight-only ratio toward 8 FLOP/byte; include input/output traffic before using it as a roofline point.
Experiment: sweep batch size and report time per batch, useful outputs/s, peak memory, and per-request age. Hold the input/output quality contract fixed. Throughput, individual latency, and queue delay are separate outcomes.
10.4 Compare devices with an evidence table¶
| Field | Device A | Device B |
|---|---|---|
| Exact SKU, software, power mode | Record | Record |
| Same input shapes and numerical criterion | Record | Record |
| Timing boundary and warmup | Record | Record |
| Raw latency distribution and useful throughput | Measure | Measure |
| Sustained temperature/power and memory | Measure if available | Measure if available |
| Missing capability or unsupported operator | Record | Record |
Use vendor documentation to describe capabilities and your measurements to describe workload behavior. An ISA name, advertised TOPS, or a benchmark score from another workload cannot fill the result rows.
11. Speculative Execution Security¶
Speculative execution can improve performance but can expose side channels. Spectre and Meltdown, publicly disclosed in 2018, showed that speculative loads can leave traces in cache timing that leak secret data. See the original Project Zero disclosure.
Spectre attack (simplified):
1. Attacker trains branch predictor to predict "taken"
2. Victim executes:
if (x < array_size) { // bounds check
y = array2[array1[x] * 256]; // speculated if x is out-of-bounds
}
3. CPU speculatively loads array1[x] (secret byte), uses it as index into array2
4. Speculation is rolled back — but array2 cache state persists
5. Attacker times access to array2 → deduces the secret byte
Mitigation behavior and cost depend on the processor, kernel, enabled controls, and workload. Use platform security advisories and the current OS documentation; a fixed percentage beside a mitigation name is not a transferable performance estimate.
AI hardware impact: attack surfaces differ across CPU, GPU, driver, and shared-service boundaries. SIMT is not a blanket side-channel immunity claim. Consult the exact platform advisories and measure mitigation overhead on the deployed workload.
12. Labs¶
Lab 0 — A reference before profiling¶
Complete Numbers, shapes, and bytes to establish a numerical reference, negative cases, and an analytical traffic boundary. It runs on a CPU with no third-party packages.
Lab 1 — Cache Miss Profiling¶
Goal: measure how access pattern affects cache performance. The following commands are exercise templates, not a supplied matrix_transpose binary or a tested installation recipe. Package names and available performance events depend on distribution, kernel, and CPU; check perf list and permissions.
# Install perf and valgrind
sudo apt install linux-perf valgrind
# L1/L2/L3 miss rates for matrix transpose (row-major vs column-major)
perf stat -e cache-references,cache-misses,L1-dcache-misses \
./matrix_transpose 1024
# Cache simulation
valgrind --tool=cachegrind --cache-sim=yes ./matrix_transpose 1024
cg_annotate cachegrind.out.*
Write a 1024×1024 float matrix. Implement: - Row-major transpose (good spatial locality for reads, bad for writes) - Column-major transpose (bad for reads) - Tiled transpose (start with 32×32 blocks) — predict and measure whether it improves locality
Deliverable: table of L1/L2/L3 miss rates for each version; explain the difference.
Lab 2 — SIMD Vectorisation¶
Goal: measure scalar vs SIMD vs compiler-auto-vectorised throughput on a dot product kernel.
// Compile and compare:
// g++ -O3 -march=native -o dot dot.cpp
// g++ -O3 -fno-tree-vectorize -o dot_scalar dot.cpp
// GCC exercise templates; inspect optimization reports and generated instructions.
// Manual intrinsics version (see section 5.2)
Benchmark a size sweep, including sizes not divisible by eight, and verify outputs against the scalar reference. Use compiler vectorization reports and disassembly to inspect instructions; generic perf stat counters do not prove vectorization. Record warmup, repetitions, cache assumptions, and timing boundaries. Compare optimized builds so the claimed SIMD effect is not simply an -O0 versus -O3 effect.
Deliverable: bandwidth (GB/s) for each version; compare to DRAM bandwidth ceiling.
Lab 3 — Roofline on a Real GPU¶
Goal: compute arithmetic intensity of three kernels and place them on the roofline.
# Use NVIDIA Nsight Compute CLI
ncu --metrics sm__throughput.avg.pct_of_peak_sustained_elapsed,\
l1tex__t_bytes.sum,\
sm__sass_thread_inst_executed_op_fadd_pred_on.sum \
python train_step.py
# Kernels to measure:
# 1. Element-wise ReLU (low AI — memory-bound)
# 2. Batch matrix multiply (high AI — compute-bound)
# 3. Softmax (medium AI)
Deliverable: roofline plot with three kernels marked; conclude which optimisations would help each.
Lab 4 — Architecture ISA Comparison¶
Goal: compare code generation and performance across ISAs.
# Cross-compile the same matrix multiply kernel for three targets:
gcc -O3 -mavx2 -mfma -o matmul_x86 matmul.c # x86-64 with AVX2
aarch64-linux-gnu-gcc -O3 -o matmul_arm matmul.c # ARM64 with NEON
riscv64-linux-gnu-gcc -O3 -march=rv64gcv -o matmul_rv matmul.c # RISC-V
# Inspect generated assembly:
objdump -d matmul_x86 | grep -E "vmulps|vfmadd"
objdump -d matmul_arm | grep -E "fmla|fmul"
Deliverable: code size, selected instructions, and reference-checked outputs for each target. Use each target toolchain's objdump (or a compatible multi-target disassembler). QEMU can support functional testing; emulated timing is not native processor performance. Record native GFLOP/s only on the named hardware and mark unavailable measurements “not run.”
Lab 5 — Branch Predictor Simulator¶
Goal: implement and compare branch prediction strategies.
class TwoBitPredictor:
def __init__(self):
self.table = {} # branch_addr → 2-bit counter (0–3)
def predict(self, addr):
return self.table.get(addr, 1) >= 2 # True = predict taken
def update(self, addr, taken):
c = self.table.get(addr, 1)
if taken:
self.table[addr] = min(c + 1, 3)
else:
self.table[addr] = max(c - 1, 0)
Feed branch traces from real programs (gcc, matrix multiply, sort). Compare: - Always taken, always not-taken - 1-bit predictor - 2-bit saturating counter - Correlating predictor (GHR + PHT)
Deliverable: accuracy table for each predictor on a stated trace and initialization; explain which pattern helps or hurts each predictor. Correlating prediction is not guaranteed to win on every nested loop.
Lab 6 — Access Pattern and Requested-Byte Rate¶
Goal: distinguish a program's requested bytes from physical cache-line and DRAM traffic.
For an array with N elements, positive stride s measured in elements, and R repetitions, the number of reads is R × ceil(N/s). Compute requested bytes as that count times the element size, then divide by elapsed seconds. Counting the full array size for every strided pass overstates the rate when elements are skipped.
Exercise / pseudocode:
initialize the array outside the timed region
choose N, stride_in_elements, repetitions, and an unsigned checksum type
start a monotonic timer
repeat R times:
for i = 0; i < N; i += stride_in_elements:
checksum += data[i]
stop the timer and preserve checksum so the computation remains observable
requested_GB_per_second = reads × bytes_per_element / elapsed_seconds / 1e9
A serial checksum dependency may limit this exercise before memory bandwidth does; inspect generated code and use independent accumulators or an established streaming benchmark when estimating sustained bandwidth. Signed integer overflow is not an acceptable way to prevent optimization.
Sweep working-set sizes relative to the actual processor's cache capacities, and sweep stride separately. A stride of 16 four-byte elements is 64 bytes; it is not 16 cache lines. Report requested-byte rate, checksum, elapsed time, and compiler options. Hardware counters are needed to estimate traffic at a particular cache or DRAM boundary. Cache transitions may be visible, but clean plateaus are not guaranteed.
Lab 7 — OoO CPU Simulator (Capstone)¶
Goal: build a simplified out-of-order CPU simulator; measure IPC and compare to in-order.
Implement: - In-order baseline (5-stage pipeline with stalls) - OoO with ROB (32 entries), reservation stations (16 entries), 2 ALUs + 1 MUL + 1 LSU - Register renaming (64 physical registers, 32 architectural) - Forwarding via CDB
Test on instruction traces: matrix multiply, Fibonacci, quicksort.
Deliverable: - IPC comparison: in-order vs OoO across benchmarks - Execution timeline visualisation - Identify which benchmark benefits most from OoO and why
13. ISA and Architecture Across the AI Stack¶
| Layer in your roadmap | Relevant ISA/arch |
|---|---|
| PyTorch model training | x86-64 (AVX-512) + CUDA (H100/A100) |
| ML compiler backends | ARM64, x86-64, RISC-V, GPU PTX |
| FPGA accelerator control | ARM Cortex-M/A (Zynq PS) |
| Jetson edge deployment | ARM64 (Cortex-A78AE) + NVIDIA Ampere |
| Custom AI chip design | RISC-V (control core) + custom dataflow |
| Mobile inference | ARM64 (Neural Engine, Hexagon) |
Key takeaway: Memory hierarchy and arithmetic intensity bound performance; the compiler and implementation determine how closely you approach the bound. With an assumed 3.35 TB/s roof, 10 FLOP/byte gives an upper bound of 33.5 TFLOP/s, not a guaranteed result. Python may invoke an efficient compiled kernel or introduce substantial overhead; the execution path matters.
14. Where This Takes You¶
This section
│
├── Phase 1 §3 — Operating Systems
│ Memory management, virtual memory, MMU (ties to TLB section above)
│ Device drivers (how OS talks to your GPU/FPGA)
│
├── Phase 1 §4 — C++ and Parallel Computing
│ SIMD intrinsics in practice, OpenMP, CUDA kernels
│ The CUDA memory model directly mirrors the GPU hierarchy in §6
│
├── Phase 4A — Xilinx FPGA
│ Implement your own systolic array in SystemVerilog
│ Understand timing and pipeline stages at the RTL level
│
├── Phase 4B — NVIDIA Jetson
│ ARM64 host CPU + Ampere GPU — both covered here
│
└── Phase 5F — AI Chip Design
Design a custom accelerator using systolic array / dataflow patterns
Roofline model guides your PE count and memory bandwidth decisions
Resources¶
Check your understanding before moving on¶
- Does 167 FLOP/byte guarantee that a matrix multiply is compute-bound? No. Compare with the selected compute/bandwidth ridge. With the illustrative 295 FLOP/byte ridge it lies below the compute roof, and other constraints may reduce performance further.
- A workload spends 2% of baseline time in serial work. What is its ideal speedup on four cores?
1 / (0.02 + 0.98/4) ≈ 3.77. Use time fractions, not the percentage of source-code lines. - Does a 32-thread warp mean every thread issues an instruction every clock? No. Dependencies, scheduling, masking, and execution resources determine issue and completion.
- Why test a vector dot product at N=9? It exercises a full eight-element vector plus a scalar tail. Testing only multiples of eight hides an out-of-bounds bug.
- A strided loop touches every sixteenth four-byte element. Can you count every array byte as read? No. Count requested elements separately from transferred cache lines; use hardware evidence for a physical memory boundary.
If one answer is uncertain, repeat that small example before adding a more complex accelerator. For the phase artifact, keep the raw measurements and explain one hypothesis that the data rejected.
Further reading¶
| Resource | What for |
|---|---|
| Patterson & Hennessy — Computer Organization and Design (ARM edition) | Pipeline, cache, memory — the textbook standard |
| Hennessy & Patterson — Computer Architecture: A Quantitative Approach | Deep OoO, superscalar, memory system design |
| NVIDIA H100 Architecture Whitepaper | SM internals, Tensor Core specs, NVLink |
| "What Every Programmer Should Know About Memory" — Ulrich Drepper | Cache hierarchy, NUMA, prefetching (free PDF) |
| "Roofline: An Insightful Visual Performance Model" — Williams et al. (2009) | Original roofline paper — 10 pages, essential |
| Chips and Cheese (blog) | Reverse-engineered microarchitecture analysis (AMD, Intel, Apple) |
| Wikichip | Die shots, cache sizes, core counts, process nodes |
| NVIDIA Nsight Compute | GPU roofline and memory hierarchy profiling |
| "Computer Systems: A Programmer's Perspective" — Bryant & O'Hallaron | Linking, virtual memory, caching from the programmer's view |
Turn the operator trace into a memory budget¶
For an FP16 1024×1024 matrix-vector product, weights alone occupy 2 MiB and the operation performs about 2.10 million FLOPs. With no weight reuse, that is about 1 FLOP per weight byte; add input/output and metadata traffic before comparing with measurements. Reusing weights across a batch changes this model; increasing peak MAC count alone does not.
Make a worksheet for weights, activations, scratchpad tiles, DMA bursts, and partial sums. Give separate estimates for cycles, external bytes, and on-chip capacity. Then measure the software reference and explain the gap. Hardware-product tables elsewhere in the guide are examples, not procurement advice or a substitute for a SKU-specific datasheet.
Carry forward: A shape-specific traffic/capacity worksheet and predicted limit. Continue with the next course and reuse that evidence.