Skip to content

Part 2 — Dense models on Hopper: shapes, memory, and serving

Llama 3.3 70B and Qwen2.5 72B are concrete historical architecture case studies. Their similar widths make it easier to see how MLP width, vocabulary, bias, position configuration, and normalization affect a deployment. They are not presented as the newest available models.

Start with the Part 1 CPU lab. It includes a pinned public Qwen2.5 configuration and tested KV arithmetic; the gated Llama configuration was not downloaded. Qwen's saved limit is 32768 positions, with a separate documented YaRN recipe for extension. Do not describe both checkpoints as identically configured for 128K.

Lesson Worked result
01 — derive model shapes Projection/MLP dimensions, RMSNorm, parameter and cache accounting
02 — Hopper hardware SKU capacities, cluster limits, TMA/WGMMA synchronization contracts
03 — quantization experiment A supported format/runtime recipe with paired quality evaluation
04 — tensor parallelism Column/row partition algebra and collective traffic assumptions
05 — serving and scheduling Chunked prefill, attention backends, cache identity, capture boundaries
06 — long context Position recipe, 40 GiB cache example, allocation and quality limits
07 — collective implementation Current NCCL APIs, trace interpretation, versioned capability checks

The CPU route produces a configuration audit, tensor-shape table, memory budget, partition proof, and benchmark plan. These are complete learning artifacts without a GPU cluster. A full deployment milestone additionally requires a supported licensed checkpoint, appropriate hardware, real quality tests, per-rank memory, traces, and raw service measurements. The course does not claim those full 70B hardware runs occurred.

For a hardware extension, compare supported cells in a model × format × runtime × topology table. Keep quality, arrival process, prompt/output lengths, cache state, and total reserved resources aligned. Record unsupported cells instead of forcing every combination to run. Include startup and failure behavior, not only a favorable steady-state number.

You are ready for the next part when you can explain a resource budget and a collective's semantics, identify which measurements would test your bottleneck hypothesis, and distinguish current TensorRT-LLM's documented PyTorch path from the archived engine-build workflow.

Continue to MoE at Blackwell, or carry the tested contracts into the Physical AI Design Studio.