Module 4B — ML Engineering & MLOps¶
Deep Dive · AI Workloads
Specialized course identity for Module 4B — ML Engineering & MLOps.
Parent: Phase 3 — Artificial Intelligence · Track B
Build and operate ML training/inference pipelines — model lifecycle, data pipelines, serving infrastructure.
Prerequisites: Module 2 (Frameworks — PyTorch fluency) and one trained model. Agentic AI is a companion when the workload uses tools; it is not a prerequisite for MLOps.
Role targets: Machine Learning Engineer · MLOps Engineer · AI/ML Engineer · ML Platform Engineer
★ Featured deep-dive course: Practical Machine Learning (CS329P) — the full ML lifecycle a practitioner actually meets, adapted from Stanford CS329P (Mu Li & Alex Smola) and refreshed for 2026: data acquisition/labeling, validation that doesn't lie, distribution-shift detection, tuning, and a hardware-facing model-compression lecture. 11 lectures. Start with the Overview.
Why This Matters for AI Hardware¶
ML engineers define the training and serving workloads that hardware must support: - Distributed training (data/model/pipeline parallelism) → L3: NCCL, multi-GPU runtime - Model serving (latency SLAs, throughput targets) → L1a: TensorRT, Triton, vLLM - Experiment tracking and model versioning → infrastructure that uses GPU clusters (Phase 5A) - Data pipelines → I/O bandwidth requirements that drive GPUDirect Storage (Phase 5B)
First complete artifact on a CPU¶
Complete the PyTorch robot-stop experiment before distributed training. It produces a model, export, data/split hashes, errors under sensor shift, and an exact environment. Add those artifacts to an experiment tracker, then reconstruct one prediction from the saved feature order and normalization. No GPU cluster is needed to learn the lifecycle.
Diagnostic: does saving weights alone let you resume AdamW training exactly? Answer: no; also preserve optimizer/scheduler state, random-generator states, epoch/batch or sampler position, preprocessing, and data revisions. Inference restoration and exact training continuation are different contracts. Seeds alone do not guarantee equality across devices or library versions.
1. Training Pipelines¶
- Data pipelines: PyTorch DataLoader, NVIDIA DALI, streaming datasets
- Distributed training: DDP (DistributedDataParallel), FSDP (Fully Sharded), DeepSpeed
- Mixed precision:
torch.ampwith supported device/dtype choices; BF16 and separately validated FP8 training recipes - Checkpointing: model saving, resume from checkpoint, elastic training
- Hyperparameter tuning: Optuna, Ray Tune, grid/random/Bayesian search
Projects: 1. Train a model with DDP across 2+ GPUs. Measure scaling efficiency. 2. Add mixed precision training. Compare FP32 vs BF16 training speed and accuracy.
2. Experiment Tracking & Model Registry¶
- Experiment tracking: MLflow, Weights & Biases (W&B), Neptune
- Model registry: immutable versions, tags, aliases, release gates, and rollback
- Dataset versioning: DVC, LakeFS
- Reproducibility: environment tracking, seed management, deterministic training
Projects:
1. Set up MLflow tracking for a training run. Log hyperparameters, metrics, and artifacts.
2. Register a model in MLflow. Validate a candidate, assign a champion alias, and rehearse restoring the prior version. MLflow model stages are deprecated; use the current aliases/tags workflow (checked 2026-10-18), while retaining a concrete version ID in each release manifest.
3. Model Serving & Inference Infrastructure¶
- Serving frameworks: Triton Inference Server, vLLM, TensorRT-LLM. Treat TorchServe as a historical case study: its repository declares limited maintenance
- Batching: static batching, dynamic batching, continuous batching (for LLMs)
- Scaling: horizontal (replicas), vertical (larger GPU), model parallelism for large models
- Monitoring: latency percentiles (p50/p99), throughput (QPS), error rates, GPU utilization
- Policy gateways: request filtering, moderation sidecars, auth, rate limits, safety traces
- A/B testing: canary deployments, shadow mode, traffic splitting
Projects: 1. Deploy a model on Triton with dynamic batching. Load test and measure p50/p99 latency. 2. Deploy an LLM on vLLM. Compare throughput with different batch sizes and quantization levels. 3. Insert a moderation or policy gateway in front of a serving stack and measure the latency tradeoff.
4. MLOps & CI/CD for Models¶
- CI/CD: GitHub Actions / GitLab CI for model training, testing, deployment
- Container orchestration: Docker, Kubernetes, NVIDIA GPU Operator
- Model testing: unit tests for preprocessing, integration tests for inference, data drift detection
- Safety regression testing: jailbreak suites, blocked-topic tests, prompt-attack replay, policy drift checks
- Feature stores: Feast, Tecton — manage reusable feature definitions; still test point-in-time joins, freshness, and training/serving parity
- Orchestration: Kubeflow Pipelines, Apache Airflow, Prefect
Projects: 1. Build a CI/CD pipeline: on push → train → evaluate quality, latency, and compatibility → stage a candidate → test rollback → promote through an explicit release gate. A better aggregate accuracy score alone is insufficient. 2. Set up GPU-enabled Kubernetes with NVIDIA GPU Operator. Deploy a model serving workload.
Resources¶
| Resource | What it covers |
|---|---|
| MLflow Documentation | Experiment tracking, model registry |
| Triton Inference Server | Production model serving |
| vLLM | LLM serving engine |
| DeepSpeed | Distributed training |
| Designing Machine Learning Systems (Huyen) | ML systems design |
Next¶
→ Module 5B — LLM Application Development
Study method: use the shared lesson cycle and completion rubric with this course's existing labs. Predict first, test one variation, retain a failure case, and revisit the self-check after two days and one week.
Carry this course into the physical-AI system¶
Start from: A trained model with dataset and preprocessing revisions.
Build and challenge: Build a release candidate with quality, timing, memory, and compatibility gates. Rehearse rollback and prove that restoring weights also restores normalization and runtime configuration.
Pass forward: A reproducible release manifest and rollback result for the robot policy. Continue with the connected course and keep the artifact in the Physical AI Design Studio.