Lecture 5 — Trace a pinned API to the right kernel family¶
Course: FlashAttention · Reviewed: 18 October 2026
Goal: distinguish the distribution, import surface and implementation before building. Artifact: a source map at revision 965c9212e6be33c821ee9f2b461564c5c91a2503, inspected on 18 October 2026. No FlashAttention GPU distribution was installed or executed in the course CPU environment.
1. Separate current package families¶
| Family | Source location in this revision | Public entry example | Evidence boundary |
|---|---|---|---|
| FA2 CUDA path | flash_attn/flash_attn_interface.py, csrc/flash_attn/ |
from flash_attn import flash_attn_func |
Source reviewed |
| FA3 Hopper path | hopper/ |
from flash_attn_3 import flash_attn_interface |
Source reviewed; separate build/package |
| FA4 CuTeDSL | flash_attn/cute/ |
from flash_attn.cute import flash_attn_func |
Installable prerelease, source reviewed |
| PyTorch SDPA | PyTorch's selected backend | torch.nn.functional.scaled_dot_product_attention |
CPU math actually executed |
PyPI metadata at review time lists flash-attn 2.8.3.post2 and flash-attn-4 4.0.0b35. A current source commit and a released distribution are distinct snapshots. The FA3 PyPI project reports 0.0.0; do not infer its source revision or compatibility from that placeholder version alone. Use the selected source/build record. Pinned upstream README · FA4 package metadata.
2. Read the actual FA2 call chain¶
At the pinned revision, the Python interface wraps an autograd function and a Python-registered custom operation. Its forward wrapper calls flash_attn_gpu.fwd, with flash_attn_gpu bound to the flash_attn_2_cuda extension for the selected NVIDIA path. C++ bindings and dispatch then select the launch/kernel. It is incorrect to search only C++ for the Python custom-op registration or assume a nonexistent _C.pyi proves the call chain.
Read these exact files and record the symbols you find:
- Python interface:
flash_attn_func, autograd wrapper,_flash_attn_forwardandflash_attn_gpu.fwd. - C++ API: bindings, validation and forward dispatch.
- Forward launch template: tile traits and launch choices.
The standard interface expects [batch, sequence, heads, dim]; PyTorch SDPA uses [batch, heads, sequence, dim]. A transpose is a layout operation, not permission to ignore stride and contiguity requirements. Varlen packed inputs have different sequence metadata; cache-update APIs mutate state and have separate contracts.
3. Audit support before installing¶
The pinned FA2 NVIDIA README specifies CUDA 12+, supported Ampere/Ada/Hopper devices, FP16/BF16 and stated head-dimension constraints. That is not a guarantee for every Jetson wheel, CUDA/PyTorch ABI or feature combination. FA3 and FA4 have their own hardware/build requirements.
For a hardware extension, create an isolated environment, record Python/PyTorch/CUDA/compiler/driver/GPU, then select a revision or wheel before installation. If building, choose a concurrency limit from available RAM; no fixed build time is promised. Read the actual setup file before inventing an environment flag. Run forward/backward correctness after import; import success alone only establishes that loading succeeded.
The source benchmark script uses in-file shape lists in this snapshot. It does not implement the old lesson's --seqlen/--headdim argument parser. The course instead supplies sdpa_experiment.py, whose declared arguments are exercised on CPU. Its optional GPU path selects one backend and does not silently substitute a benchmark label.
4. Practice and answers¶
Task: choose one family and build a table of input shape, dtype, mask alignment, output, mutation and backward support from its actual interface. Then compare it with the CPU fixture contract.
Answer check: an API name shared by FA2, FA3 and FA4 does not establish identical optional arguments or return values. A PyTorch backend name does not establish that the separately installed Dao-AILab distribution was called. Capture versions and inspect/profiling evidence for the path actually used.