Skip to content

HPC with Nvidia GPU

HWNG

Deep Dive · Specialization

Specialized course identity for HPC with Nvidia GPU.

Artifact: specialization case study · Measure: performance, reliability, role fit

Parent: High Performance Computing

Timeline: 12–24 months (fundamentals and deep dives); 24–48 months for advanced phase.


Basic concepts: what "HPC with Nvidia GPU" means

HPC = high-performance computing: solving problems that need massive compute and memory by using many machines (or many GPUs) working together. In AI, "HPC" usually means large-scale training and high-throughput inference on GPU clusters, not single workstations.

Why this stack? CUDA, cuDNN, CUTLASS, TensorRT and NCCL provide a widely used application-to-hardware path. NVIDIA's platform-specific features make useful case studies in kernel execution, memory placement and distributed communication. Compare other platforms with the same workload and numerical contract rather than assuming a universal performance ranking.

CPU starting point: the two-process Gloo/DDP lab runs without GPUs and shows why correct gradient normalization precedes cluster tuning. GPU-specific recipes are labeled by execution evidence and version.

What this track covers:

  • Single GPU → many GPUs — From one GPU (e.g. Jetson, which you saw in Phase 4 Track B) to multi-GPU nodes and multi-node clusters. You need to understand how jobs are placed, how data and gradients move, and how to avoid communication becoming the bottleneck.
  • Two main workloads:
    • Training — One big model, one big dataset; you split work across GPUs (data parallelism, model parallelism, pipeline parallelism). Performance is about throughput (samples/sec) and scaling to hundreds or thousands of GPUs.
    • Inference — Many requests, one (or many) deployed model; you care about latency and throughput under load. At scale this means batching, KV-cache, and often multi-GPU or multi-node serving (e.g. TensorRT-LLM, vLLM).
  • The stack you must understand:
    • Hardware: GPUs (A100, H100/H200, L40S, etc.), NVLink/NVSwitch inside a node, InfiniBand or Ethernet across nodes.
    • Software: CUDA, drivers, containers (NGC), orchestrators (Slurm, Kubernetes), and collective libraries (NCCL) for multi-GPU communication.
    • Storage and I/O: Getting data to GPUs fast (dataloaders, GPUDirect Storage, high-throughput disks) so the GPU is not waiting.

Key terms (used in this track)

Order: basic ops → attention → distributed.

Term Meaning
Matrix multiply Compute A·B for matrices A, B (often plus bias or activation). The core operation in linear layers and most heavy compute. "GEMM" is the standard name in libraries.
GEMM GEneral Matrix Multiplication: C = α(A·B) + βC. The BLAS/cuBLAS/CUTLASS interface for matrix multiply. GPU GEMM kernels (tiling, tensor cores) dominate training and inference time.
Epilogue fusion In a GEMM kernel, the epilogue is what you do with the result (add bias, ReLU/GELU, write to memory). Fusion = doing that in the same kernel as the multiply. Saves memory bandwidth and launch overhead; CUTLASS and similar libraries support it.
Attention In transformers: each token has Q, K, V (from linear layers); attention = softmax(Q·K^T/√d)·V. Lets the model focus on relevant tokens. The matmuls are heavy for long sequences; optimized attention kernels and KV-cache are critical.
FlashAttention A family of attention kernels that reduce memory traffic by tiling and keeping Q,K,V in SRAM, and avoid materializing the full attention-score matrix in HBM; only tiles need fit on chip. Faster and more memory-efficient than naive attention; standard in LLM training and inference (e.g. FlashAttention-2, -3).
KV-cache In transformer attention: keys and values for previous tokens are cached so you don't recompute them. KV-cache = that cache. Long context → huge cache → memory and bandwidth become the bottleneck; paging/sharding and efficient kernels matter.
Data / model / pipeline parallelism Data: same model on every GPU, different data; sync gradients. Model: split the model across GPUs. Pipeline: different layers on different GPUs, pass activations in a pipeline.
Collectives Multi-GPU operations: AllReduce (everyone gets the same sum), AllGather (everyone gets all pieces), ReduceScatter. Used to sync gradients (data parallel) or exchange activations (model/pipeline parallel).
NCCL Nvidia Collective Communications Library. Implements collectives (AllReduce, AllGather, etc.) for multi-GPU training and inference. Often the bottleneck at scale.
NVLink / NVSwitch NVLink: high-bandwidth GPU↔GPU (and GPU↔CPU) link inside a node. NVSwitch: switch connecting many GPUs in one node. Much faster than PCIe for GPU↔GPU traffic.

What this track covers (after reorganization)

This track now focuses on HPC infrastructure and multi-GPU operations. The DL Inference Optimization content (graph optimization, kernel engineering, compiler stack, quantization, inference runtimes, tinygrad deep dive) has moved to Phase 4 Track C — ML Compiler & Graph Optimization as Part 2, since those skills are foundational for all hardware tracks, not just HPC.

Part Description Guide
HPC Setup Fundamentals, virtualization, interconnects, advanced CUDA/distributed training/performance — plus hardware-specific deep dives (8x H200, L40S, NCCL, CUDA Advanced, GPUDirect Storage) HPC Setup →
~~DL Inference Optimization~~ Moved to Phase 4 Track C Part 2 Track C →

How to use this track

  1. Complete Phase 4 Track C first — covers compiler fundamentals and DL inference optimization (graph ops, kernels, compiler stack, quantization, runtimes).
  2. Then HPC Setup — Covers Nvidia GPU HPC fundamentals, virtualization (vGPU, KVM), interconnects and storage (InfiniBand, GDS, Slurm, Kubernetes), and Phase 2 advanced topics (advanced CUDA, distributed training, performance modeling). Use the deep dives (8x H200, L40S, NCCL, CUDA Advanced, GDS) for your target hardware and stack.

Prerequisite: Phase 4 Track B (Jetson, TensorRT, CUDA) and Phase 4 Track C (compiler + inference optimization).

A two-measurement diagnostic

Measure the same inference step once with device timing and once from input-ready to output-ready on the host. A large gap suggests dispatch, transfer, synchronization, or preprocessing work; it is not evidence that the GEMM kernel is slow. Then sweep batch size while keeping precision and shapes documented.

Carry both traces into HPC Setup. This decides whether the next study should be kernel efficiency, host scheduling, memory capacity, or multi-GPU communication. Hardware peaks describe ceilings; the pipeline determines which ceiling matters.

Practice and assessment

Entry diagnostic (prerequisite check): Draw GPU/CPU/NIC topology and explain bandwidth versus message latency. If this is unfamiliar, use the prerequisite material above or the roadmap diagnostics before starting the lab.

  1. Baseline lab: Establish compute, memory-transfer, and collective baselines on available hardware.
  2. Guided variation: Change placement or message size and test a recoverable worker failure.
  3. Independent transfer: repeat the method on a new input or configuration and explain where your original assumptions stop applying.

Evidence to submit: Topology, achieved bandwidth/latency, scaling efficiency, error logs, and configuration. Keep raw results and reproduction steps; distinguish measured results from estimates or work not run.

Self-check: Which observation would disprove your explanation of the result? Identify a competing explanation and the smallest experiment that separates them. A passing answer names a concrete predicted difference, not only another tool to try.

Exit criteria: reproduce the baseline, demonstrate the negative/variation check, and meet the shared completion rubric. Use the lesson cycle for retrieval practice and spaced review.