Part 1 — Fundamentals of AI inference¶
Learn to trace a token, count its memory, and define a useful service measurement. The complete beginner route runs on a CPU and requires no model weights. Large-model GPU serving is an optional later experiment with separate evidence.
| Lesson | What you will be able to explain |
|---|---|
| 01 — workload and service metrics | TTFT, average TPOT versus individual ITL, useful throughput, tails, and cost |
| 02 — transformer execution | Prefill, incremental decoding, GQA, cache growth, and absolute-position masks |
| 03 — roofline bounds | FLOPs/bytes, compatible ceilings, units, and why a bound is not a benchmark |
| 04 — precision contracts | Weights/activations/KV, scales and tails, calibration and task quality |
| 05 — runtime selection | Source-backed compatibility and a fair matched comparison |
| CPU lab | Run attention/cache, expert-routing, storage, and speculation checks |
Read one lesson, predict its worked example, then run the relevant lab case. The lab has 22 standard-library tests and two optional PyTorch CPU comparisons, all executed in the recorded validation environment. Three pinned public model configurations support the memory exercises; no pretrained checkpoint is loaded.
Your first artifact is a benchmark contract and a small tested reference, not an invented cluster throughput result. Explain the 40 GiB dense-cache example, the non-square cached-mask counterexample, and Qwen3's explicit head dimension. Keep precision, cache layout, and allocator overhead separate.
Before progressing, distinguish storage capacity from transfer traffic; a per-token active parameter count from batch work; and a kernel timing from client-observed latency. State which conditions make a stage compute-, memory-, communication-, or scheduling-limited instead of memorizing that all decode is bandwidth-bound.
Continue to dense-model deployment reasoning, then MoE routing. Carry the benchmark contract into the Physical AI Design Studio with the hardware measurements clearly separated from CPU reference evidence.