Chapter 4: Multi-GPU and GB200 NVL72 Placement¶
Previous: Single-GPU deployment · Next: Kernel engineering
Place tensors before counting GPUs¶
Multi-GPU execution solves a concrete constraint: model capacity, request throughput, or latency. Begin with Chapter 3's largest-rank memory budget and measured single-group baseline. NVLink adds a fast communication path; it does not merge allocations or eliminate collectives.
A GB200 Superchip combines one Grace CPU and two Blackwell GPUs. The NVL72 configuration combines 36 Grace CPUs and 72 GPUs in a scale-up domain. CPU–GPU coherency over NVLink-C2C is distinct from GPU-to-GPU NVLink connectivity. Remote HBM, local HBM, and Grace memory have different paths and costs. NVIDIA GB200 system overview.
Choose a parallelism strategy with an explicit communication model¶
| Strategy | Memory/compute benefit | Communication to measure | Common bad assumption |
|---|---|---|---|
| Replicas | More independent request capacity | Request routing; optional cache sharing | Replication makes an oversized model fit |
| Tensor parallelism | Splits selected matrices across ranks | Per-layer reductions/gathers and synchronization | More ranks always reduce batch-one latency |
| Pipeline parallelism | Splits layer stages | Activation transfer and pipeline bubbles | Every stage has equal compute/memory |
| Expert parallelism | Places MoE experts across ranks | Token dispatch/combine, routing skew | Active parameters determine total resident expert memory |
| CPU memory/offload | Additional capacity | Movement and synchronization on the actual path | Coherency makes CPU memory as fast as HBM |
The legal TP degree depends on the model and engine's partition rules. Check attention-head, KV-head, hidden-dimension, and kernel constraints. A rack with 72 GPUs does not imply TP=72 is supported or appropriate. Several smaller TP groups plus data-parallel replicas may produce better goodput.
Worked break-even model¶
Let a measured single-group decode step take T_compute. A candidate TP split changes it to approximately:
T_step = T_partitioned_compute + T_collectives + T_launch/sync + T_other
If compute falls from an illustrative 20 ms to 12 ms but newly exposed communication costs 10 ms, the candidate is slower. These numbers are a teaching example, not B200 measurements. Measure overlap: simply summing all profiler event durations can double-count simultaneous compute and communication.
For an 8-rank AllReduce of payload S, nccl-tests' normalized bus bandwidth is (S/t) × 2(7/8). Link specifications often aggregate send and receive directions, while a collective's algorithm bandwidth describes useful payload divided by time. Keep the definitions beside the plot. NCCL bandwidth definitions.
Worked Qwen placement: why TP=72 is not the answer¶
For the pinned dense Qwen2.5 model, 64 query heads and 8 KV heads divide evenly across TP=8: eight query heads and one KV head per rank, under that partitioning scheme. TP=72 cannot evenly divide 64 query heads. The engine may impose further matrix, packing and process-group requirements; divisibility is a necessary check for this simple partition, not sufficient compatibility evidence.
On a 72-GPU domain, nine TP=8 groups or eighteen TP=4 groups are arithmetic candidates. Compare them under the same offered load and total GPU count. TP above the KV-head count may replicate KV heads in supported runtimes, so do not assume cache bytes keep dividing by TP. This model is dense: enabling expert parallelism does not create MoE experts.
Collective calculation: if an eight-rank 1 GB AllReduce takes a hypothetical
2 ms, algorithm bandwidth is 500 GB/s and normalized bus bandwidth is
500×2×7/8=875 GB/s. This does not measure an individual NVLink port. The
CPU collectives lab runs actual
two-rank Gloo operations and exposes wrong averaging with unequal local batches.
Its semantics transfer; its localhost timings do not estimate NCCL/NVLink speed.
Self-check answer: equal shard count is not equal work. Replicated heads, embedding/output tensors, workspace and unequal pipeline stages make the largest rank's budget decisive. A larger TP group may fit while slowing each decode step because its collectives dominate saved computation.
Grace memory is a separate tier¶
A coherent CPU/GPU address model removes some programming complexity but does not remove transfer cost, placement effects, page behavior, or bandwidth contention. Do not use the bidirectional NVLink-C2C link rating as guaranteed LPDDR read bandwidth. A KV offload experiment must record which pages live where, the migration policy, and observed GPU stalls.
A useful experiment compares no offload, a fixed offload fraction, and a lower-concurrency HBM-only configuration. Report capacity gained and latency lost. If the working set is repeatedly fetched across the slower link, a model that fits may still violate its deadline.
Lab: grow a serving group, then grow replicas¶
- Save device/rank/NIC/CPU affinity mapping and peer-access results.
- Run correctness-checked collective tests across representative message sizes. Distinguish a healthy physical fabric from a healthy serving configuration.
- Test one supported TP group at fixed prompts, context, output length, and concurrency. Record per-rank memory and collective time.
- Increase TP only where legal; then compare the same total GPU count arranged as more replicas of the smaller TP group.
- Add load until the latency objective is missed. Count failures, queueing delay, and imbalance.
- On an isolated test job, stop a worker and document failure detection and recovery. A restarted worker is not necessarily a resumed stateful request.
Deliverable: a placement matrix showing GPUs per group, replica count, per-rank peak memory, useful goodput, p99 latency, and failure behavior. Carry it into Chapter 6 for admission and recovery policy.
Translate the lesson to a custom SoC¶
A wide matrix array can be underused when DMA or SRAM banking serializes its input. A large memory sum can hide an inaccessible or slow bank. Reuse the same placement-and-communication reasoning for accelerator tiles, CPU fallbacks, sensor buffers, and wireless contention. Draw the path before assuming a bandwidth improvement helps the control loop.