Skip to content

Isaac Sim and Isaac Lab — Special Course

SIM

Phase 5 · Robotics · Special Course

Build, simulate, sense, and train robots in NVIDIA Isaac Sim and Isaac Lab — and know exactly where every millisecond and megabyte of GPU goes.

Artifact: custom Isaac Lab task + trained policy + ROS 2 bridge + performance report · Measure: physics step time, render time, env-steps/s, VRAM per env, real-time factor

A simulator is a GPU workload. Treat it like one.

Isaac Sim is NVIDIA's robotics simulator: an Omniverse Kit application that composes a USD scene, steps it with PhysX (or the new Newton engine), renders sensors with RTX, and talks to ROS 2. Isaac Lab is the robot-learning framework on top of it: thousands of cloned environments on one GPU, managers for observations, rewards and randomization, and adapters for RL and imitation-learning libraries.

This course teaches both from the bottom up — USD prims to trained policies — and treats the simulator as what it is on your desk: a GPU program competing for a fixed VRAM and compute budget. The labs connect mathematical and interface checks to measurements of physics, rendering, training and memory. Each result identifies the workload and execution boundary.

Comprehensive VLA companion: VLA Foundations to Frontier 2026 provides 32 lectures spanning history, architectures, data, imitation/flow/RL training, evaluation, deployment, and the research frontier with a source update reviewed on October 18, 2026. Use its CPU labs and capstones alongside this specialization.

Teaching release family (rechecked 2026-10-18): Resolve an exact commit and lockfile using the lab manifest; a moving release branch is not an immutable pin. The compatibility record links the official evidence and distinguishes the full simulator, kit-less Lab workflows, and the locally executed ROS-only lab.

Latest release update: Isaac Sim 6.2.0 GA shipped October 14, 2026. The lessons below retain their named 6.1 examples; use the 6.2 upgrade checks for changed sensor ownership, Newton views/substeps, actuator contracts, and current product compatibility. The moving Lab 3.0 branch now has different dependencies from its EA tag.

Component Version Notes
Isaac Sim 6.1.0 (GA, Sept 2026) Kit 110.3; Python 3.12; open-source extensions (Apache-2.0) on a closed Kit SDK, EULA required
Isaac Lab 3.0.0-EA (tag commit ae37b028ea415c91ea2bc32609efcd759ed2b974, Sept 2026) Full Isaac Sim workflows require Sim 6.1; kit-less workflows can omit it. GA remains an upstream target for late October 2026, not a released version. Breaking changes are in Lecture 09.
OS / ROS 2 Ubuntu 24.04 + ROS 2 Jazzy Humble on 22.04 also supported
Physics PhysX 5 · Newton 1.5 PhysX is Isaac Sim's default; Newton is experimental there. In Isaac Lab 3.0 the backend is selectable, and the core tasks (Cartpole, Lift) default to Newton (newton_mjwarp).

The Core API changed under everyone's feet: Isaac Sim 5.0 introduced the Core Experimental API, and 6.0 deprecated the classic isaacsim.core.api (including World and SimulationContext). This course teaches the new API first and shows the old one only so you can read existing code.

Reference hardware: an 8 GB RTX 4060. That is below NVIDIA's official minimum for Isaac Sim 6.1 (RTX 4080, 16 GB). Operation on it is not guaranteed; use it only for reduced experiments after a compatibility check. The supported full-simulator route uses hardware meeting the documented requirements. Labs run headless where they can, use state observations by default, and size camera counts and resolutions to a VRAM budget. Each lecture has an 8 GB budget note. Where a lab genuinely needs more (many RTX cameras, VLA policies in the loop), it says so and points to a cloud L40S / RTX PRO 6000 as the fallback. Data-center GPUs without RT cores (A100, H100) are not supported for Isaac Sim rendering.

Layer mapping: L3-L8. Scene description and physics (simulation runtime), GPU rendering and sensor pipelines, ROS 2 integration, and the RL training loop.

Role targets: Robotics Simulation Engineer · Robot Learning Engineer · Synthetic Data Engineer · Sim-to-Real Engineer · Robotics Infrastructure / Platform Engineer.

Prerequisites:

  • Python and PyTorch fluency; comfort on the Linux command line.
  • Phase 5 — Robotics — Advanced Robot Operating System (ROS 2 nodes, topics, TF) before Lecture 08.
  • Phase 5 — Robotics — Deep RL for Robot Learning, at least Lectures 01, 04-06, before Lectures 09-10. This course is the simulator side of that one.

What comes after: use Isaac Lab as the rung-2 simulator in Deep RL for Robot Learning, and the evaluation setup for VLA Optimization and Action-Parity Harness.


Lecture map

# Title Focus
01 The Isaac Stack and Your First Simulation Kit, USD, PhysX/Newton, RTX · application vs simulation vs stage · full-simulator requirements and reduced CPU entry path · GUI / extension / standalone workflows · a first measured simulation
02 USD for Roboticists Prims, attributes, stages, layers · references, payloads, variants, instancing · physics schemas · raw USD vs wrappers · USD vs Fabric vs tensors · hosted assets
03 The Core Experimental API Batched wrappers and regex paths · Warp arrays · data backends · SimulationManager / RenderingManager · callbacks · migrating from World
04 PhysX for Robots Rigid bodies, colliders, contact offsets, friction · TGS vs PGS, iterations, timestep · CCD and GPU dynamics · articulations and joint drives · stable grasping
05 Newton and Multi-Physics Newton and MuJoCo-Warp · switching engines · solvers (MJWarp, XPBD, VBD, Featherstone) · deformables · what transfers between engines
06 Robots: Import, Tune, and Move URDF / MJCF importers · collision and inertia validation · Robot Assembler, gain tuner, sysid · cuMotion motion generation · pick-and-place
07 Sensors, Rendering, and Synthetic Data RTX cameras, annotators, tiled cameras · RTX lidar · IMU, contact, raycast · Replicator domain randomization · Cosmos writer · the VRAM cost of pixels
08 ROS 2 and OmniGraph isaacsim.ros2.bridge · action graphs · clock, TF, sensors · ros2_control · Nav2 / MoveIt in the loop · latency and real-time factor
09 Isaac Lab 3.0 Architecture Packages · manager-based vs direct workflows · InteractiveScene and cloning · physics backends · ProxyArray, xyzw · events and randomization · kit-less mode · 2.x → 3.0 migration
10 Training Policies in Isaac Lab isaaclab train · RSL-RL PPO on Cartpole, Lift, and locomotion · num_envs and backend sweeps on 8 GB · camera-based tasks · writing your own direct task
10b Worked Example: Spot Waypoint Navigation Hierarchical control: waypoint follower → 50 Hz learned locomotion policy → 500 Hz PD actuators → physics · the policy contract (48-dim observation, command ranges) · RobotPolicyRunner on PhysX and Newton · train your own Spot policy and deploy it · ROS 2 /cmd_vel bridge
11 Imitation, Teleop, and Policy Evaluation Teleop with Isaac Capture (formerly Isaac Teleop) · Mimic data generation · robomimic · Isaac Lab-Arena for VLA evaluation · RLinf · client-server policies
12 Performance, Sim-to-Real, and Scale-Out Profiling physics vs render vs Python · headless and streaming · containers and cloud · multi-GPU · determinism · actuator models and domain randomization
13 Capstone: A Task, a Policy, and a Bridge Build a custom manipulation task, size training to measured capacity, evaluate with seeds and CIs, and drive it over ROS 2, with a full performance report
14 Reproducible Simulation Setup Version/asset manifest, compatibility, zero/random-action smoke tests
15 Frames, Units, and Asset Validation Known-answer transforms, quaternion conventions, joint/asset contracts
16 Task Correctness and Partial Resets Reward probes, done/timeout semantics, reset isolation, action history
17 Training and Held-Out Evaluation Baselines, seed splits, sample/time efficiency, uncertainty
18 Physics, Sensors, and Performance Stage profiles, rate definitions, sensor/rollout memory, scaling
19 ROS 2 Timing and Policy Export Normalizers, observation age, clocks, deadlines, stale commands
20 System Identification and Transfer Actuator response, delay, randomization ranges, held-out shifts
21 Language-Guided Robotics Capstone Asynchronous planner, bounded skills, inference contention, closed-loop evaluation

The lectures use the Curriculum Authoring Guide: explain the concept, work a small example, run an experiment, and assess the evidence. Lectures 14–21 add self-check guidance and staged engineering exercises; the lab guide identifies which checks run on a CPU and which require Isaac.


Executed CPU entry path

Start with Simulation before the GPU: 27 standard-library tests, actual Newton 1.5.2/Warp 1.16 CPU physics, actual OpenUSD 26.8 composition, measured ROS 2 message rejection, and a separate-process known-weight policy adapter. It also supplies delay-identification and paired-regression exercises. These results are recorded in the validation manifest.

The full Isaac Sim, Isaac Lab training, rendering and trained-policy examples remain optional and were not executed here. Their named source versions matter: 6.1 examples do not automatically become 6.2 examples when a link is updated. Follow the compatibility record and rerun the simulator-specific checks on the selected stack.

Applied engineering labs

Follow the engineering lab sequence for Lectures 14–21 and runnable CPU contract exercises. Together with Lecture 10b, the course now has 22 lectures. The sequence separates simulator, policy, and deployment milestones so learners can show exactly what they have validated.

Learning path: from "what is Isaac Sim" to physical AI

If you are coming from a five-level syllabus (mental model → build a robot → robotics AI → learning → advanced), this table maps each topic to where the course teaches it. The end goal is the same in both: Isaac Sim → Isaac Lab → RL / VLA training → deployment on a real robot.

Level Topics Where
1. Mental model What Isaac Sim is; Isaac Sim vs Isaac Lab 01, 09
USD, stages and prims 02
PhysX, and the articulated-body equation \(M(q)\ddot q + C\dot q + g = \tau + J^\top F\) 04
Simulation time vs wall-clock time; real-time factor 01 §3, 08
Articulations, rigid bodies, joints 03, 04
Extensions and Python scripting 01 §5, 03
2. Build a robot Import a robot; build from USD components 06, 02
Joints, drives, controllers 04, 06
Cameras, lidar, IMU, contact sensors 07
Programmatic control 03, 10b
3. Robotics AI ROS 2 integration 08
Perception pipelines, synthetic data, domain randomization 07, 12 §6
Navigation 10b (waypoints, learned locomotion), 08 (Nav2)
Manipulation and motion planning 06 (cuMotion), 04 (grasping)
4. Learning Isaac Lab and RL environments 09, 10
PPO and policy learning 10, Deep RL 05-06b
Imitation learning 11, Deep RL 02-03
Vision-language-action systems 11 (Arena evaluation), Deep RL 11, 14 (RL post-training of π0.5)
Sim-to-real 12 §6, 10b (Spot RL Researcher Kit)
5. Advanced GPU simulation and parallel environments 05, 09, 12
Digital twins 12 §6.4
Large-scale RL 10, 12 §4
Deploying policies on real robots and Jetson 10b, 13, VLA Optimization, VLA on Jetson

A vision-language robot agent ("put the red cup in the box") is the natural project at the end of this path. It combines this course's simulation and evaluation stack (Lectures 07, 11, 13) with policy training from Deep RL for Robot Learning (Lecture 14 post-trains π0.5) and real-time inference from VLA Optimization.


Why this is a hardware-first course

On an RTX 4060, every simulation decision is a resource decision:

  • Physics and rendering share one GPU. PhysX GPU dynamics, Newton's Warp kernels, RTX ray tracing, your policy network, and the desktop compositor all draw from the same 8 GB and the same SMs. A camera at 640×480 in 64 environments can cost more than the physics of 4,096 state-only environments.
  • Step time has three parts: physics, rendering, and Python/data movement. They respond to different knobs (solver iterations and substeps; render products, resolution and RTX settings; data backend and batch size). Optimizing the wrong one does nothing.
  • The data path is a design choice. USD is for authoring, Fabric is the GPU runtime store for transforms and rendering, and the physics tensor API is the zero-copy path for training. Reading poses through USD in a training loop is the most common silent slowdown.
  • Real-time factor is a contract. A controller or operator tied to wall-clock time may require the simulator to keep up. An all-simulation-time ROS pipeline can run slower or faster if every component handles that clock consistently. For RL, throughput matters only after task correctness is established. Record which timing contract your scene must meet.

Every Measure it section asks for physics ms/step, render ms/frame, env-steps/s, peak VRAM, and VRAM per environment, so by the capstone you have a performance model of your own GPU.


What you ship

By the end of the course you should have, in one repo:

  • isaac_bench/ — a reusable harness that measures physics step time, render time, env-steps/s and VRAM for a scene, with results for every lab on your GPU.
  • USD assets — a robot and a work cell you imported, validated (collision, inertia, drives) and composed with references and variants.
  • A custom Isaac Lab task (direct workflow), with randomization, a trained policy, and learning curves over ≥ 3 seeds.
  • A ROS 2 bridge demo — the trained or scripted robot driven over ROS 2 Jazzy, with measured end-to-end latency and real-time factor.
  • PERFORMANCE.md — the VRAM and step-time budget for your capstone on an 8 GB card, and what you would change on 16 GB and 48 GB cards.

Exit criteria

You are done with this special course when you can:

  • explain what Kit, USD, Fabric, PhysX/Newton, and RTX each do in one frame of simulation, and which one a given symptom points to
  • write a standalone Isaac Sim script with the Core Experimental API, and port a classic World-based script to it
  • tune a gripper until a grasp is stable, and justify every physics parameter you changed
  • predict, before running it, roughly how many environments and cameras fit in 8 GB for a given task, then measure it
  • build, train, and evaluate a custom Isaac Lab task, and explain each of its managers or step functions
  • drive the simulation over ROS 2 and report its latency and real-time factor

References for the whole course


Next

Study method: use the shared lesson cycle and completion rubric with this course's existing labs. Predict first, test one variation, retain a failure case, and revisit the self-check after two days and one week.

Carry this course into the physical-AI system

Start from: A version/hardware manifest and a small scene that resets correctly.

Build and challenge: Vary physics step, sensor rate, and policy decimation independently. Compare simulated time with wall time, then inject delayed observations while measuring task outcome.

Pass forward: An environment contract, reset test, and observation-to-action traces. Continue with the connected course and keep the artifact in the Physical AI Design Studio.