Skip to content

03 — Compiler Stack for Inference (IR, Scheduling, Codegen)

CSFI

Deep Dive · Compiler Track

Specialized course identity for 03 — Compiler Stack for Inference (IR, Scheduling, Codegen).

Artifact: compiler/inference optimization · Measure: op count, memory, latency

Order: Third. After graph/ops (01) and kernel authoring (02), you see how compilers generate and schedule kernels.

Role target: DL Inference Optimization Engineer · MTS Kernels (Member of Technical Staff, Kernels — roles focused on code generation, compiler–hardware mapping, and owning kernel/backend implementation at scale).


Why this comes third

Kernels (02) are either hand-written or compiler-generated. This unit covers how compilers represent the model (IR), decide fusion and placement (scheduling), and emit code (codegen). You need this to co-design with frameworks and to add or tune backends.

CPU entry: complete the first compiler experiment to inspect actual generated code without a GPU. Its optional tinygrad runner is pinned separately from its standard-library correctness tests.


1. Intermediate representation (IR)

  • Graph IR vs linearized IR — Graph: nodes = ops, edges = tensors. Linearized: list of ops in execution order (e.g. tinygrad's linearized ops).
  • SSA form — Single assignment; each value defined once. Enables clear memory and alias analysis.
  • Memory and alias analysis — Which buffers can overlap; when fusion or in-place is safe.

Takeaway: The IR is the contract between "model graph" and "kernel backend." Your kernels are targets for lowering from this IR.


2. Scheduling and lowering

  • tinygrad — The scheduler groups graph operations into kernels; BEAM searches implementation choices for an already formed kernel. Keep graph fusion decisions separate from kernel optimization search. See tinygrad speed documentation.
  • TVM — Separate Relax graph-level work from TensorIR scheduling and code generation. Pin a TVM release: older AutoTVM/AutoScheduler tutorials and newer tuning APIs are not interchangeable. Start with tile, vectorize, and parallel transformations and inspect the generated loops.
  • MLIR — linalg/tensor dialects; progressive lowering (linalg → loops → vector → gpu). How high-level ops become loops and then GPU kernels.

Takeaway: Scheduling decides which kernels run (fused or not) and how they're tiled/parallelized; codegen then emits CUDA/LLVM/etc.


3. Code generation

  • Backend codegen — From IR/schedule to CUDA, OpenCL, LLVM, or custom target. Role of codegen in Triton, TVM, tinygrad.
  • Kernel fusion and tile selection — How the compiler chooses tile sizes and fusion sets for GPUs and accelerators.

Resources

  • tinygrad — IR, scheduler, BEAM, backends.
  • TVM Documentation — Relax, TensorIR, and external integrations; MetaSchedule for the documented tuning path. Pin the release before copying namespaces; AutoTVM/Relay examples are historical workflows.
  • MLIR Tutorial — Dialects and lowering.

Projects

  1. BEAM in tinygrad — Run tinygrad with BEAM on a small model. Compare scheduled kernel count and runtime vs default scheduler; distinguish scheduler grouping from BEAM kernel choices.
  2. Fusion pass — Implement a simple fusion pass (e.g. Conv+ReLU) in a graph you control (ONNX or tinygrad). Measure impact on kernel count and latency.
  3. Trace lowering — Pick one op (e.g. matmul) in TVM or tinygrad and trace from high-level op to generated code. Document the lowering steps.

Next

→ 04 — Quantization — Low-precision inference (PTQ, QAT) and how it affects kernels and deployment.

Integration lab: What proof obligation connects a rewrite to a machine instruction?

Trace one matrix operator from graph IR through buffer planning, loop/kernel scheduling, and generated code. Add a legal rewrite with an explicit shape/dtype guard and a counterexample rejected by that guard.

Acceptance evidence: Save before/after IR and correctness results. Separate graph fusion from BEAM kernel search, and pass the lowered operator plus memory-layout contract to quantization/runtime work.

Continue: take this artifact to 04 - Quantization. Explain which assumption from this course the next engineer must preserve.

Custom-accelerator handoff: take the lowered operator, buffer-lifetime plan, and reference vectors to Compiler, Runtime ABI, and SoC Integration. Specify command fields, alignment, DMA ownership, completion/error reporting, and fallback so the same contract can drive a functional simulator, RTL implementation, and Linux driver.