8x H200 GPU — Training & Inference Deep Dive¶
The NVIDIA H200 SXM5 is a Hopper-generation GPU used here as a reference platform, featuring 141 GB HBM3e memory at 4.8 TB/s bandwidth. An 8-GPU SXM node with NVLink 4.0/NVSwitch is a useful reference configuration for large-model training and inference; actual feasibility depends on the full server and workload.
System Snapshot¶
| Property | Value |
|---|---|
| GPU | NVIDIA H200 SXM5 |
| Count | 8 |
| Memory per GPU | 141 GB HBM3e |
| Total GPU Memory | 1,128 GB (1.1 TB) |
| Memory Bandwidth | 4.8 TB/s per GPU |
| FP8 Tensor Core TFLOPS | ~1,979 dense / ~3,958 with supported structured sparsity |
| BF16 Tensor Core TFLOPS | ~989.5 dense / ~1,979 with supported structured sparsity |
| GPU Interconnect | NVLink 4.0 (900 GB/s bidirectional) |
| NVSwitch | Switched NVLink fabric; record the server topology |
| Host Interconnect | PCIe 5.0; do not assume CXL memory support |
| Form Factor | SXM5 baseboard |
Topic Index¶
| # | Topic | Description |
|---|---|---|
| 01 | Hardware Architecture | Chip design, HBM3e, NVLink 4.0, NVSwitch topology |
| 02 | Training Setup | Distributed training, 3D parallelism, FSDP, DeepSpeed |
| 03 | Inference Setup | Tensor parallel inference, vLLM, TensorRT-LLM |
| 04 | Memory Management | KV cache, paged attention, memory pooling |
| 05 | Performance Optimization | Profiling, roofline, kernel tuning, CUDA Graphs |
| 06 | Benchmarks & Validation | Correctness, MFU assumptions, bounded device tests, service goodput |
Quick Navigation¶
- Training a 70B model? → Start with 02-Training-Setup
- Inference serving? → Start with 03-Inference-Setup
- GPU memory OOM? → See 04-Memory-Management
- Low GPU utilization? → See 05-Performance-Optimization
Capacity is per rank¶
Eight 141 GB devices provide 1,128 GB of aggregate memory, not a single allocation pool. For a teaching 70-billion-parameter BF16 model, raw weights are about 140 GB decimal before KV cache, activations, runtime allocations, and uneven sharding. A single 141 GB GPU therefore is not a practical fit just because the weights approximately match its label. Under tensor parallelism, inspect the largest per-rank allocation and replicated buffers rather than dividing the total by eight.
Carry the per-rank memory worksheet from Chapter 4 into the serving configuration in Chapter 3. In Chapter 6, use dense peak for dense model FLOP utilization; sparse advertised peak is not the denominator for an unpruned model. NVIDIA H200 specifications.
Practice and assessment¶
Entry diagnostic (prerequisite check): Explain process groups, tensor/data parallelism, and per-rank memory. If this is unfamiliar, use the prerequisite material above or the roadmap diagnostics before starting the lab.
- Baseline lab: Establish a small validated workload before scaling to the available H200 group.
- Guided variation: Sweep rank count and context/batch while keeping quality and timing boundaries fixed.
- Independent transfer: repeat the method on a new input or configuration and explain where your original assumptions stop applying.
Evidence to submit: Per-rank memory, collective time, throughput, latency, and topology. Keep raw results and reproduction steps; distinguish measured results from estimates or work not run.
Self-check: Which observation would disprove your explanation of the result? Identify a competing explanation and the smallest experiment that separates them. A passing answer names a concrete predicted difference, not only another tool to try.
Exit criteria: reproduce the baseline, demonstrate the negative/variation check, and meet the shared completion rubric. Use the lesson cycle for retrieval practice and spaced review.