Skip to content

FlashAttention: a systems and kernel course

ATTN

Phase 4C · Kernel Engineering

Preserve attention's mathematical and API contract while changing its memory and execution schedule.

Core artifact: tested CPU reference and SDPA comparisons · Extension: a pinned GPU kernel experiment

Reviewed 18 October 2026. Ten lectures connect the attention equations to memory traffic, work partitioning, gradients and cached decoding. Begin with the actual CPU attention lab: seven portable tests plus fourteen NumPy/PyTorch tests execute without a GPU or model download. A separate CUDA extension requires compatible hardware and software; source review is not represented as GPU execution.

Parent: Kernel Engineering. Prerequisites: Python arrays, matrix multiplication, softmax and basic memory units. CUDA basics are needed for the hardware extension, not the CPU path. Plan four to six study sessions, then a capstone; divide framework setup from the mathematical exercises.

The question this course answers

Attention computes weighted combinations of value vectors. Standard implementations can store a large score/probability matrix. Tiled implementations keep small score tiles and running normalization while changing the order of computation. The result can be mathematically equivalent with floating-point differences, but only if masks, positions, scaling and layout preserve the same contract.

For robotic vision or VLA workloads, those conditions matter as much as raw speed: a padding mask, camera-token layout or cached position error can produce a fast wrong answer. This course teaches the operator boundary; it does not claim whole-model task quality from a kernel test.

Start with a five-minute diagnostic

  • Q is [2,8], K is [5,8], V is [5,3]. What is the output shape? [2,3].
  • For one newest query and five live keys, must causal attention keep only key 0? No. That is upper-left alignment; a newest-token query normally aligns to the end of the live history.
  • If a tiled implementation avoids the full score matrix, are all bytes read exactly once? No. Tiles can reread inputs; capacity and traffic are different.

If these are unfamiliar, read Lectures 01–02 before installing optional frameworks.

Learning sequence

Lecture What you learn Observable artifact
01 — Attention and roofline Shapes, FLOPs, capacity and traffic Executed budget with assumptions
02 — Online softmax Running numerator/denominator and empty rows Derivation and NumPy comparison
03 — FA1 algorithm Historical loop order and SRAM units Source annotation and traffic model
04 — GPU fundamentals Ownership, synchronization, coalescing and occupancy Buffer-lifetime diagram; optional device profile
05 — API and source Pinned FA2/FA3/FA4 families and layouts Actual source call map
06 — FA2 Query-row partitioning and deferred normalization Ownership sketch and recurrence counterexample
07 — Backward Analytic, finite-difference and autograd gradients Executed gradient/precision report
08 — Decode and KV Rectangular alignment, GQA, pages and RoPE Cache/mask tests and byte budget
09 — FA3 and FA4 Hopper/Blackwell pipeline and source boundaries Dated source map; optional hardware comparison
10 — Capstone One tested implementation change Patch, failure case, raw output and report

Current implementation boundary

The course inspected Dao-AILab source revision 965c9212e6be33c821ee9f2b461564c5c91a2503, dated 16 October 2026. FA2 CUDA, FA3 Hopper C++ and FA4 Python CuTeDSL are separate paths. FA4 has an installable prerelease; it is not merely a proposed future kernel. Import names and device support must be checked against the selected family. Pinned upstream source.

The executed environment uses NumPy 2.3.5 and PyTorch 2.14.0+cpu, explicitly selecting SDPA's math backend. It validates small seeded arrays, masks, gradients, GQA and toy RoPE positions. It does not run a Dao-AILab CUDA kernel. The lab retains hashes, source metadata, test output and actual numerical errors.

Completion and handoff

Finish the CPU path when you can reproduce the tests, explain the upper-left/lower-right difference, derive the online normalization, count both K and V, and add one independently tested mask or cache case. For the GPU extension, add exact hardware/build information, selected-kernel evidence, correctness and measured timing; leave unavailable outcomes marked not run.

Pass the resulting mask/layout/precision/position contract to the compiler stack. A compiler or accelerator optimization must preserve that contract before its latency result becomes useful.