Lecture 01: The Isaac Stack and Your First Simulation¶
Overview¶
Isaac Sim looks like a 3D editor with a play button. Underneath it is a stack of four GPU-hungry systems glued together by a scene description format, and nearly every problem you will hit later — a robot that explodes, a camera that eats gigabytes of VRAM, a training loop that runs at 10% of the speed it should — comes from not knowing which layer you are talking to.
This lecture builds that map, installs the named Isaac Sim 6.1 baseline on a compatible RTX system, and provides a first standalone simulation to measure: physics time per step, frame time with rendering, and VRAM. Those three numbers are the baseline every later lecture compares against.
By the end you should be able to:
- name the layers of Isaac Sim (Kit, USD, Fabric, physics, RTX) and say what each does in one frame
- explain application vs simulation vs stage vs scene, and how Isaac Sim 6.x replaced
World - install Isaac Sim 6.1 with pip, run the compatibility checker, and get assets loading
- choose between the GUI, extension, and standalone workflows for a given job
- write a standalone script with the Core Experimental API and time its physics and rendering separately
- account for where the 8 GB on your card goes
1. Why it matters: three jobs, one simulator¶
People use Isaac Sim for three different jobs, and they stress the GPU differently:
| Job | What runs | Bottleneck | Lectures |
|---|---|---|---|
| Sim-in-the-loop for a robot software stack (Nav2, MoveIt, perception) | One robot, several sensors, ROS 2 bridge, roughly real time | Rendering (RTX sensors) and real-time factor | 07, 08 |
| Synthetic data generation (SDG) for perception models | Randomized scenes, many rendered frames, annotations written to disk | Rendering throughput and disk I/O | 07 |
| Robot learning (RL, imitation) | Thousands of cloned environments, mostly physics, sometimes cameras | Physics throughput, VRAM per environment | 09-11 |
The same scene can be fine for one job and hopeless for another. A warehouse with four RTX lidars might hold real time for Nav2 testing and still be useless for RL, where you want 1,000× real time. Knowing which job you are doing tells you which numbers to watch.
2. Mental model: the stack¶
┌──────────────────────────────────────────────────────────────────────────┐
│ Omniverse Kit application (isaacsim.exp.full, or your SimulationApp) │
│ extensions: UI, importers, sensors, ROS 2 bridge, your code │
├──────────────────────────────────────────────────────────────────────────┤
│ USD stage — the scene description: prims + attributes + composition │ authoring
├───────────────┬───────────────────────────────┬──────────────────────────┤
│ Fabric / │ Physics: PhysX 5 (default) │ RTX renderer │
│ USDRT │ or Newton (experimental) │ (ray-traced, Hydra) │ runtime
│ GPU runtime │ + tensor API for batched │ cameras, lidar, radar │
│ store │ state (zero-copy to torch) │ via render products │
└───────────────┴───────────────────────────────┴──────────────────────────┘
▲ ▲ ▲
transforms for Python reads/writes annotators → RGB,
rendering joint states, poses depth, segmentation
- Kit is NVIDIA's application framework. Isaac Sim is a set of Kit extensions (the Isaac-specific ones have been open source under Apache-2.0 since 5.0; Kit itself stays closed and needs the Omniverse EULA). An experience (
.kitfile) is a list of extensions plus settings;isaacsim.exp.fullis the standard app. - USD (OpenUSD) is the scene description. Everything — robot, table, light, camera, physics scene — is a prim with attributes. USD is optimized for authoring and composition, not for being read 1,000 times a second.
- Fabric (with its Python API, USDRT) is the GPU-friendly runtime copy of the scene data that changes every frame, chiefly transforms for rendering.
- Physics steps the dynamic prims. PhysX is the default. Newton (a Warp-based, open-source engine using MuJoCo-Warp) is experimental in 6.x and selectable at launch (Lecture 05). The physics tensor API exposes batched state (
ArticulationView,RigidBodyView) after the simulation starts, which is the fast path for training. - RTX renders the viewport and every sensor. A camera is a render product; annotators turn it into RGB, depth, segmentation, and so on (Lecture 07).
2.1 Application, simulation, stage, scene, world¶
NVIDIA's docs explain these words with a theater: the application is the theater (your gateway: GUI, rendering, input); the simulation is the play, moving prims forward through time by changing their attributes; the stage is the USD stage where everything exists and where relationships between prims (a mug on a table) are expressed; a scene is the set of props and actors in the current act; and the world is the stage crew that runs the curtain, resets the props between scenes, and keeps the play on schedule.
That last role — the stage crew — is where Isaac Sim changed. In the classic Core API (5.x and earlier) it was one object, World. Isaac Sim 6.0 deprecated World and SimulationContext along with the rest of isaacsim.core.api. There is no single replacement; its jobs are split:
| Concept | Classic Core API (5.x, deprecated) | Isaac Sim 6.1 |
|---|---|---|
| Create / open a stage | World() creates one implicitly |
isaacsim.core.experimental.utils.stage (create_new_stage, add_reference_to_stage) |
| Add objects | world.scene.add(DynamicCuboid(...)) |
Create prims directly: Cube(...), GeomPrim(..., apply_collision_apis=True), RigidPrim(...) |
| Play / step physics | world.reset(), world.step(render=True) |
Timeline play(); SimulationManager.step() for physics; simulation_app.update() steps physics and renders |
| Render | inside world.step(render=True) |
RenderingManager.render() |
| Per-step callbacks | world.add_physics_callback(...) |
SimulationManager callbacks (e.g. post-physics-step events) |
| Read state | cube.get_world_pose() (one prim, NumPy) |
cube.get_world_poses() (batched, Warp arrays; .numpy() to convert) |
Most tutorials, forum answers, and GitHub code you find still use the left column. Lecture 03 covers the migration properly. For now: read the left, write the right.
2.2 The Core API is a wrapper¶
The Core API (classic or experimental) is a robotics-shaped wrapper over raw USD and physics schemas. Adding one falling cube with raw USD means defining a physics scene, a ground plane, a UsdGeom.Cube, a transform op, and applying UsdPhysics.RigidBodyAPI and UsdPhysics.CollisionAPI:
from pxr import Gf, UsdGeom, UsdPhysics
import omni.usd
stage = omni.usd.get_context().get_stage()
UsdPhysics.Scene.Define(stage, "/World/physics") # gravity defaults, solver settings via PhysxSchema
cube = UsdGeom.Cube.Define(stage, "/World/Cube")
cube.CreateSizeAttr(0.5)
cube.AddTranslateOp().Set(Gf.Vec3f(0.5, 0.2, 1.0))
UsdPhysics.RigidBodyAPI.Apply(cube.GetPrim()) # "this prim is a dynamic body"
UsdPhysics.CollisionAPI.Apply(cube.GetPrim()) # "this prim collides"
The experimental Core API says the same thing in robot terms, and works on one prim or a thousand:
from isaacsim.core.experimental.objects import Cube
from isaacsim.core.experimental.prims import GeomPrim, RigidPrim
shape = Cube(paths="/World/Cube", positions=np.array([[0.5, 0.2, 1.0]]), sizes=[0.5])
GeomPrim(paths=shape.paths, apply_collision_apis=True) # collision
cube = RigidPrim(paths=shape.paths) # dynamics
Both produce the same USD. Knowing the raw form matters for two reasons: imported robots arrive as raw USD you will need to inspect (Lecture 02), and when a wrapper hides a physics parameter, you set it on the schema directly (Lecture 04).
3. The hardware view: one frame, one GPU¶
A standalone simulation loop does this every simulation_app.update():
Python callbacks ──► physics step(s) ──► sync to Fabric ──► RTX render ──► annotators / UI
(CPU) (GPU or CPU) (GPU) (GPU) (GPU → CPU)
Physics and rendering run at different rates. Physics typically steps at a fixed small timestep (and may take several substeps per rendered frame); rendering happens per app update or per sensor rate. In a training loop you often step physics many times without rendering at all. That is why this lecture times them separately.
Where the 8 GB goes. On a 4060, VRAM is the constraint you will hit first. These are the consumers:
| Consumer | Scales with | Lever |
|---|---|---|
| Display / desktop compositor | Monitors, other apps (browser tabs use GPU too) | Close them; run headless on a second machine if you have one |
| Kit + RTX renderer baseline | Renderer mode and settings; loaded once | Headless; lower-cost render settings |
| Scene assets (meshes, textures, materials) | Number of unique assets | USD instancing (Lecture 02); simpler materials |
| Physics GPU buffers | Bodies, shapes, contacts, articulations | Fewer collision shapes per body; contact buffer sizes (Lecture 04) |
| Render products (each camera) | Count × resolution × annotators | The biggest lever in Lectures 07 and 10 |
| PyTorch caching allocator | Policy, batch, replay buffers | torch.cuda.memory_summary(); free what you do not need |
8 GB budget. On a 4060, close the browser and other GPU apps before measuring, run labs headless unless you need to look, and treat the GUI as a debugging tool rather than the default. Physics-only scene capacity must be measured; the first things to overflow are many RTX cameras and large textured environments (Lecture 07). If your fixed cost from Lab 1a leaves only a few GB, plan to run camera-heavy labs on a cloud L40S or RTX PRO 6000 and keep the 4060 for physics and policy work.
Rule for this course: measure the baseline before adding anything. If the empty app already takes a large part of your card, every later VRAM-per-env number has to fit in what is left.
4. Install Isaac Sim 6.1 on an RTX 4060¶
NVIDIA's published minimum for 6.1 is an RTX 4080 with 16 GB. An 8 GB 4060 is below that minimum; startup and useful operation are not guaranteed. Complete the CPU simulation lab first if suitable hardware is unavailable. This repository did not execute the full Isaac Sim examples.
Prerequisites. Ubuntu 22.04 or 24.04 (x86_64), GLIBC 2.35+, a recent NVIDIA driver (the 6.1 docs list 595.58.03 as the tested Linux driver), 32 GB RAM recommended, 50 GB+ free SSD.
# 1. Python 3.12 environment (Isaac Sim 6.x requires 3.12)
python3.12 -m venv ~/env_isaacsim && source ~/env_isaacsim/bin/activate
pip install --upgrade pip
# 2. PyTorch first: preserve this baseline's documented CUDA wheel family
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128
# 3. Isaac Sim 6.1 from NVIDIA's index (large download)
pip install "isaacsim[all,extscache]==6.1.0.0" --extra-index-url https://pypi.nvidia.com
# 4. Read the linked NVIDIA license terms and accept them only if you agree.
# For an installation whose terms you have accepted:
export OMNI_KIT_ACCEPT_EULA=YES
# 5. Check your system against the requirements
isaacsim isaacsim.exp.compatibility_check
# 6. Launch the full app (the first launch compiles shaders and can take many minutes)
isaacsim isaacsim.exp.full
Alternatives: the workstation zip (./isaac-sim.sh, scripts run with ./python.sh), the container nvcr.io/nvidia/isaac-sim:6.1.0 (Linux, rootless, -e ACCEPT_EULA=Y), or building from source from the GitHub repo. The pip path is the easiest to combine with Isaac Lab 3.0 later.
Assets. Since October 2025 there is no Omniverse Launcher or local Nucleus. Isaac Sim 6.1 loads its sample assets over HTTPS from a hosted service, and isaacsim.storage.native.get_assets_root_path() returns the root. Behind a firewall, set ISAACSIM_ASSET_ROOT to a mirror or download the offline asset packs. In 6.1 the robot assets moved from Isaac/Robots/ to Isaac/Robots_Multiphysics/, so older tutorials' paths may 404.
Run VS Code against it: python -m isaacsim --generate-vscode-settings writes settings so imports resolve.
5. Three workflows¶
| Workflow | How it runs | Use it for | Avoid it for |
|---|---|---|---|
| GUI | isaacsim isaacsim.exp.full; build and inspect interactively |
Inspecting imported robots, tuning physics visually, debugging | Anything you need to reproduce |
| Extension | Your code is a Kit extension; runs asynchronously inside the app; hot-reloads | Tools and UI panels, interactive demos | Precise control of step timing |
| Standalone Python | Your script creates a SimulationApp and owns the loop |
Benchmarks, data generation, RL, CI, headless runs | Quick visual poking |
This course uses standalone Python for every lab, because it is the only workflow where you control exactly when physics steps and when rendering happens. Two rules:
- Create
SimulationAppfirst.from isaacsim import SimulationAppand construct it before importing anything fromomni.*orisaacsim.*, because those modules only exist once Kit has started. - Close it.
simulation_app.close()at the end, or the process can hang holding GPU memory.
6. Build it¶
Everything goes in isaac_bench/, the harness you will extend all course.
Lab 1a — Baseline VRAM¶
Measure what the system costs before your scene adds anything. In one terminal:
Record memory.used in four states, about 30 s each:
- Desktop idle (nothing running)
isaacsim isaacsim.exp.fullopen on an empty stage- The same, after loading
Isaac/Environments/Grid/default_environment.usd - A headless
SimulationApp({"headless": True})with the same environment (Lab 1b script, rendering on)
The difference between 1 and 4 is your fixed cost. What remains of 8 GB is your budget for everything else in the course. Write both down.
Lab 1b — First standalone simulation, timed¶
# isaac_bench/falling_cubes.py
import argparse, time
parser = argparse.ArgumentParser()
parser.add_argument("--n", type=int, default=64) # number of cubes
parser.add_argument("--headless", action="store_true")
parser.add_argument("--steps", type=int, default=600)
parser.add_argument("--device", default="cuda:0") # physics device: "cuda:0" or "cpu"
args = parser.parse_args()
from isaacsim import SimulationApp # must come before omni/isaacsim imports
simulation_app = SimulationApp({"headless": args.headless})
import numpy as np, omni.timeline
import isaacsim.core.experimental.utils.stage as stage_utils
from isaacsim.core.experimental.objects import Cube
from isaacsim.core.experimental.prims import GeomPrim, RigidPrim
from isaacsim.core.simulation_manager import SimulationManager
from isaacsim.storage.native import get_assets_root_path
stage_utils.add_reference_to_stage(
usd_path=get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd",
path="/World/ground",
)
# A grid of n cubes, 25 cm apart, dropped from 1 m (side**2 >= n, so it is one layer: each cube lands on the ground).
side = int(np.ceil(np.sqrt(args.n)))
idx = np.arange(args.n)
pos = np.stack([(idx % side) * 0.25, (idx // side % side) * 0.25, 1.0 + (idx // side**2) * 0.3], axis=1)
paths = [f"/World/Cube_{i}" for i in range(args.n)]
t0 = time.perf_counter()
shape = Cube(paths=paths, positions=pos, sizes=[0.1] * args.n) # one batched wrapper for all cubes
GeomPrim(paths=shape.paths, apply_collision_apis=True)
cubes = RigidPrim(paths="/World/Cube_.*") # regex: the same n prims
t_build = time.perf_counter() - t0
# Set the physics device explicitly: the default depends on Kit settings, and GPU vs CPU changes everything.
# On "cuda:0" this enables GPU dynamics, the GPU broadphase and Fabric (USD writeback off).
SimulationManager.setup_simulation(device=args.device)
omni.timeline.get_timeline_interface().play()
simulation_app.update() # first update starts physics
def timed(fn, steps):
n0, t = SimulationManager.get_num_physics_steps(), time.perf_counter()
for _ in range(steps):
fn()
cubes.get_world_poses()[0].numpy() # readback: wait for async GPU work
ms = (time.perf_counter() - t) / steps * 1e3 # ms per call
return ms, (SimulationManager.get_num_physics_steps() - n0) / steps # physics steps per call
ms_physics, sp_physics = timed(SimulationManager.step, args.steps) # physics only, no render
ms_update, sp_update = timed(simulation_app.update, args.steps) # physics + render (+ UI if not headless)
positions, _ = cubes.get_world_poses() # Warp array, shape (n, 3)
z = positions.numpy()[:, 2]
print(f"n={args.n} device={SimulationManager.get_device()} headless={args.headless} build={t_build:.2f}s "
f"physics={ms_physics:.3f} ms/step ({sp_physics:.2f} steps/call) "
f"update={ms_update:.3f} ms/frame ({sp_update:.2f} steps/call) "
f"throughput={args.n * 1e3 / ms_physics:,.0f} body-steps/s z: min={z.min():.3f} max={z.max():.3f}")
simulation_app.close()
python isaac_bench/falling_cubes.py --n 64 # with the GUI viewport
python isaac_bench/falling_cubes.py --n 64 --headless
python isaac_bench/falling_cubes.py --n 64 --headless --device cpu # same scene, CPU pipeline
Check two things before trusting the timings. First, the steps/call columns. If either is not 1.00, the two loops are not doing the same physics work, and their times cannot be compared directly. If SimulationManager.get_num_physics_steps() does not count manual step() calls on your build, the physics column will read 0.00; then rely on the device and the readback. Second, the physics itself: after 1,200 steps every cube should be resting near the ground (z close to half the cube size, 0.05 m). If cubes are still at 1 m, the simulation is not playing. If any z is far below 0, something fell through the floor.
If your build of the wrapper rejects a list of paths, create the cubes in a loop and keep the single regex RigidPrim. Building is a one-time cost; the regex wrapper is what matters for stepping.
Lab 1c — Scaling sweep¶
Run Lab 1b for n ∈ {1, 16, 64, 256, 1024, 4096}, headless and windowed, while logging VRAM with nvidia-smi. Plot:
- physics ms/step vs
n(log-log) - app-update ms/frame vs
n - peak VRAM minus your Lab 1a fixed cost, vs
n
What to expect, qualitatively. At small n, physics time is almost flat, because fixed per-step overhead dominates. At large n it grows with the number of bodies and contacts. Here every cube rests on the ground rather than on another cube, so contacts grow roughly linearly with n; dense stacks (Lecture 04) grow them faster. Update time includes rendering, so the gap between the two curves is the price of drawing a frame. Note where your 4060 stops being comfortable. That number feeds the environment-count estimates in Lecture 10.
Reference results: RTX 4060 Laptop GPU. One learner's run of the earlier version of Lab 1b (headless, 300 steps per timed loop, Isaac Sim 6.1, physics device not set explicitly, and no steps/call diagnostics). The throughput column is computed from the physics column:
| n cubes | Build | Physics ms/step | Update ms/frame | Physics steps/s | Throughput (body-steps/s) |
|---|---|---|---|---|---|
| 16 | 0.14 s | 3.66 | 70.6 (first-launch warm-up) | ~273 | ~4,400 |
| 64 | 0.59 s | 5.03 | 13.8 | ~199 | ~12,700 |
| 256 | 2.20 s | 9.51 | 14.4 | ~105 | ~26,900 |
| 1024 | 8.84 s | 23.72 | 14.1 | ~42 | ~43,200 |
| 4096 | 35.7 s | 81.76 | 108.0 | ~12 | ~50,100 |
How to read it, including two traps:
- Build is linear, about 9 ms per cube. It is a one-time authoring cost, but it is why reset-by-rebuilding is a bad idea at scale.
- Physics is affine, not sublinear: a fixed floor of about 3.5 ms, then about 19 µs per cube. The marginal cost is roughly constant from 256 to 4,096 cubes (18.5 µs, then 18.9 µs per cube per step). Going from 64 to 4,096 cubes (64×) costs only 16× because of the floor, not because the solver gets more efficient per body.
- Fewer steps per second is not slower training. Physics steps/s falls from ~199 to ~12, but body-steps/s rises from ~12,700 to ~50,100, about 4×. For RL, the number that matters is Lecture 01's env-steps/s, which is steps/s × number of parallel copies. These are single rigid cubes, not environments: a robot environment costs much more per step, so do not read these columns as environment counts.
- Trap 1: update is faster than physics at 256 and 1,024 cubes. Each
update()should include a physics step plus a render, so it cannot honestly beat a bare physics step (14.1 ms against 23.7 ms at n = 1,024). That is a measurement red flag, not "physics hidden under the frame budget". Either the two loops did different physics work per call, orSimulationManager.step()paid extra costs per call that the app's update path does not, such as writing results back to USD when Fabric is off. The current script prints steps/call and the device for exactly this reason. - Trap 2: the device was never set. The default physics device depends on Kit settings. A constant ~19 µs per body is in line with CPU stepping plus per-step USD writeback, which
setup_simulation(device="cuda:0")turns off. Rerun the sweep with--device cuda:0and--device cpuand compare the slopes before drawing conclusions about the GPU. - No cube-cube contacts. At 25 cm spacing, the cubes never touch each other. Each one rests on the ground with a handful of contact points, so contacts grow linearly with
nand the 4,096 case has no self-collision overhead.
7. Use it in the real stack¶
- The canonical starting points in the docs are the Core API tutorials (Hello World, then Hello Robot), now written against the experimental API. Read Hello World side by side with Lab 1b.
- Standalone examples ship with Isaac Sim; in the workstation install they live under
standalone_examples/. Many still show the classic API, so treat them as reading material. - Isaac Lab (Lectures 09-13) is itself a standalone-Python application on top of everything in this lecture. Its
--viz noneflag is the Isaac Lab 3.0 version ofheadless=True.
8. Measure it¶
| Metric | How | Why it matters |
|---|---|---|
| Fixed VRAM cost | Lab 1a, nvidia-smi |
Your budget is 8 GB minus this |
| Physics ms/step | SimulationManager.step() timed |
Upper bound on sim speed without rendering |
| Frame ms | simulation_app.update() timed |
Cost when rendering is on: sim-in-the-loop and cameras |
| Real-time factor | physics_dt / wall-clock per step |
Must be ≥ 1 for ROS 2 sim-in-the-loop; as high as possible for RL |
| VRAM per body | Slope of the Lab 1c VRAM curve | First data point for "how many envs fit" |
| Build time | t_build |
Scene construction cost, which matters for reset-heavy workflows |
For the real-time factor you need the physics timestep. Read it from the physics scene in your stage, and record it with your results.
9. Ship it¶
Commit isaac_bench/ with:
falling_cubes.py,run_sweep.sh, andvram_log.csvfilesresults.csv— one row per (n, headless) with build s, physics ms/step, frame ms, peak VRAMscaling.png— the three curves from Lab 1cBASELINE.md— your GPU, driver, Isaac Sim version, fixed VRAM cost, remaining budget, and one paragraph: at whatndoes rendering dominate frame time, and at whatndoes physics?
Exit criteria¶
You can move on when you can:
- draw the stack from memory and say which layer owns transforms, contacts, and pixels
- explain why
Worldis gone in 6.x, and what replaces each of its jobs - install Isaac Sim 6.1 from scratch, and get assets loading with or without internet access
- write a standalone script that creates batched objects, steps physics, and reads poses as Warp arrays
- state your card's fixed VRAM cost and the physics vs render split for a 1,024-body scene
Self-check¶
- A teammate's script imports
isaacsim.core.experimental.primson line 1 and constructsSimulationAppon line 3. It fails with aModuleNotFoundError. Why, and what is the fix? - Lab 1c shows physics at 2 ms/step and frame time at 25 ms for 1,024 cubes, windowed. Which layer is responsible for most of the frame, and name two changes that would shrink it without touching physics.
- Your sim-in-the-loop test needs a real-time factor ≥ 1 with a 1/60 s physics timestep and rendering on. Lab 1b measures 21 ms per
update(). Does it meet the contract? What are your options? - An old tutorial calls
world.scene.add(DynamicCuboid(...))andworld.step(render=True). Write the Isaac Sim 6.1 equivalent in three or four lines, and name what changed about the return type of pose queries. - Your empty Isaac Sim GUI already uses a large share of your 8 GB card. List three things you can do to recover memory before a training run, in order of how much each is likely to save.
References¶
- Isaac Sim 6.1 — requirements, pip install, container, accessing assets
- Development workflows — docs
- Standalone Python and
SimulationApp— docs - Core API overview (wrappers, application/simulation/world/scene/stage) — docs
- Core API Hello World tutorial — docs
- Core API → Core Experimental migration guide — docs
- Physics data pathways (USD, Fabric, tensors) — docs
- Isaac Sim on GitHub — repo