AI Inference Engineer 2026 — Special Course
Transformer execution, dense and MoE serving, and DeepSeek V4.1 Flash NVFP4 checkpoint engineering across the inference stack.
Part 1 — Fundamentals of AI inference
Learn to trace a token, count its memory, and define a useful service measurement. The complete beginner route runs on a CPU and requires no model weights. Large-model GPU serving is an optional later experiment with separate evidence.
Part 2 — Dense models on Hopper: shapes, memory, and serving
Llama 3.3 70B and Qwen2.5 72B are concrete historical architecture case studies. Their similar widths make it easier to see how MLP width, vocabulary, bias, position configuration, and normalization affect a deployment. They are not presented as the newest available models.
Part 3 — MoE inference: route the work and account for the state
Use DeepSeek-V3.1 and Qwen3-235B-A22B as pinned model-configuration case studies. The key distinction is between total resident parameters, per-token active work, and the union of experts touched by a batch. Attention layout, routing policy, topology, and state ownership then determine what the runtime must do.
Part 4 — Optimizing a Real Engine
Parts 1–3 taught the stack. This part is one engine, one model, one node, and a public pull-request history. Published measurements are author reports, with varying levels of supporting evidence; this course independently runs the CPU checks described below.
- 4.01 The workload, the baseline, and the ladder
What does 2.8T on one node actually cost, and what did 60× look like rung by rung?
- 4.02 The scoreboard — a benchmark designed to resist gaming
How do you build the measurement before the optimization, so the numbers survive contact with incentives?
- 4.03 Diagnosis — launch-bound, bandwidth-bound, or comm-bound?
How do per-rank bytes, launch gaps, grid coverage, and collectives distinguish competing bottleneck hypotheses?
- 4.04 Launch geometry — grids, occupancy, and 327 norms per token
Why were the cheapest wins on the board about grid shape, not arithmetic?
- 4.05 Fusion and the activation-quantization discipline
When does quantizing an activation pay, and how do you fuse without breaking bit-identity?
- 4.06 Attention at 128k — split over context, split over heads
Why did decode fall 90% with depth while llama.cpp stayed flat?
- 4.07 Sharding 896 experts — and the Amdahl trap that followed
Where does the collective go, and why did a 3× MoE win destroy TP scaling?
- 4.08 Graph-resident decode — killing the launch bill for good
What is left when ~30 kernel launches per layer × 93 layers stop being launches?
- 4.09 The phase you forgot — batched prefill
How do you spend the recorded development period on decode while prompt ingestion runs one forward per token?
- 4.10 Silently wrong — the failure mode unique to inference engines
What do you do about bugs that produce fluent text, and "speedups" that are corruption?