Lecture 01: The RL Problem and the Rollout Engine¶
Overview¶
Reinforcement learning means learning by trying things and getting a score, instead of being told the right answer. It sounds like a small change from supervised learning, but two things change at once:
- Your actions change your next input. In supervised learning, getting example 3 wrong does not change example 4. In RL, if the gripper misses the bowl, the next camera frame shows a missed bowl. The data distribution is produced by the policy you are training.
- Nobody tells you the right action. You get a reward — often just
1if the task finished and0if it did not, at the very end of a 300-step episode.
This lecture sets up the vocabulary used for the rest of the course, and then asks the question every robot-learning team eventually hits: when you run an RL algorithm, where does the wall-clock time actually go? You will measure that for three very different setups — a CPU toy, a GPU-parallel manipulation simulator, and a billion-parameter VLA policy — because the answer decides which algorithms are practical.
Running example used throughout the course: a 7-DoF robot arm on a simulated kitchen table, told in words to "put the bowl on the plate."
| What it is for our arm | |
|---|---|
| Observation | 1-3 RGB camera images (224×224), the language instruction, and proprioceptive state (joint angles, gripper width) |
| Action | A 7-D vector: 3 translation + 3 rotation deltas for the end effector, 1 gripper command. VLAs emit a chunk of ~10-50 of these at once. |
| Reward | Sparse and binary: 1 when the task predicate is true (bowl on plate), else 0 |
| Episode | Up to a few hundred control steps at 10-20 Hz |
By the end you should be able to:
- write down the MDP objective and define V, Q, and the advantage in one line each
- explain why a binary success signal still gives you something smooth to optimize
- classify any RL algorithm by how it does try, judge, and improve
- measure env-steps/s, policy-inferences/s, and learner-updates/s for a training setup, and say which one is the bottleneck
- explain why sample efficiency and wall-clock efficiency rank algorithms differently
1. Why it matters: two ways to teach a computer¶
| Supervised learning | Reinforcement learning | |
|---|---|---|
| Signal | The correct answer for every input | A scalar score, often delayed and sparse |
| Data | Fixed, i.i.d. dataset | Generated by the current policy; changes as it learns |
| Failure mode | Overfitting | Compounding errors, exploitation of the simulator, no signal at all if it never succeeds |
| Ceiling | As good as the labels | Can exceed the demonstrator |
The last row is why RL matters for robotics. Copying expert demonstrations (Lectures 02-03) can only make a policy as good as the demonstrations — and in practice somewhat worse, because errors compound. Practicing against a score can find behaviors that are better than the examples: faster grasps, recoveries the teleoperator never showed, robustness to layouts nobody recorded.
The modern recipe uses both, in this order:
large, messy data small, clean demos practice with a score
(pre-training) ──► (supervised fine-tune) ──► (RL post-training)
general knowledge "what we want" "better than the demos"
This is the same three-stage pipeline used for chat LLMs (Lecture 11). π0.5 arrives at stage 2 already done; the capstone (Lecture 14) is stage 3.
2. Mental model: the rules of the game¶
Every RL problem is a Markov decision process (MDP):
| Symbol | Name | Meaning |
|---|---|---|
| \(s_t\) | state | Everything about the world at time \(t\) (all object poses, velocities, joint states) |
| \(o_t\) | observation | What the policy actually receives (pixels + instruction + proprio). Usually less than \(s_t\). |
| \(a_t\) | action | What the policy does |
| \(p(s_{t+1} \mid s_t, a_t)\) | dynamics | How the world responds — the simulator, or physics |
| \(r(s_t, a_t)\) | reward | The score for this step |
| \(\pi_\theta(a_t \mid o_t)\) | policy | The network with parameters \(\theta\) — the thing we train |
| \(\gamma \in [0,1)\) | discount | How much later rewards count relative to sooner ones |
| \(T\) | horizon | Episode length |
Markov property: if \(s_t\) is complete, the future depends only on \(s_t\) and \(a_t\), not on the history. When the policy sees only \(o_t\) (a camera can't see inside the drawer), the problem is a POMDP, and the policy may need memory — which is why VLAs often take a short history of frames.
2.1 The objective¶
A trajectory \(\tau = (s_0, a_0, s_1, a_1, \dots)\) is sampled by running the policy in the world. The goal is the highest expected total reward:
Two words in that equation carry most of the weight:
- Total. We optimize the whole episode, not the next step. Braking before the crash, not at it. Reaching around the bowl to approach it from the side even though that temporarily moves the gripper away.
- Expected. Each episode is random (initial layout, policy sampling noise, physics). For our arm, every episode scores 0 or 1 — a step function. But the success rate \(J(\theta) = \Pr_\theta[\text{success}]\) changes smoothly as \(\theta\) changes, because the probability of each trajectory changes smoothly. That smoothness is what lets gradient methods work on a reward that is itself not differentiable.
2.2 Value, Q, and advantage¶
Three quantities show up in every later lecture:
With a sparse binary reward and \(\gamma \approx 1\), \(V^\pi(s)\) is simply the probability of eventually succeeding from state \(s\) under the current policy. That is a very intuitive thing for a critic to learn, and Lecture 05 will exploit it.
2.3 Try → Judge → Improve¶
Every RL algorithm in this course is a loop of three steps:
┌──────────────────────────────────────────────────────────┐
▼ │
┌─────────┐ ┌──────────────────┐ ┌─────────────┐ │
│ TRY │ ───► │ JUDGE │ ───► │ IMPROVE │ ─┘
│ collect │ │ estimate return, │ │ update the │
│ rollouts│ │ V, Q, or a model │ │ policy │
└─────────┘ └──────────────────┘ └─────────────┘
env + policy fwd critic / returns policy backward
(inference) (cheap or a 2nd net) (training)
The algorithms differ in how each box works:
| Algorithm family | Try | Judge | Improve | Can reuse old data? | Lecture |
|---|---|---|---|---|---|
| Behavioral cloning | expert does it | (none — expert is right) | supervised regression | yes (only expert data) | 02-03 |
| Policy gradient (REINFORCE, GRPO) | run current policy | sum of rewards, minus a baseline | raise log-prob of good actions | no (on-policy) | 04, 11 |
| Actor-critic / PPO | run current policy | learned \(V(s)\), GAE | clipped policy-gradient step | a few epochs | 05-06 |
| Q-learning / SAC | any policy, stored in replay | learned \(Q(s,a)\) via bootstrapping | argmax or actor step on Q | yes (off-policy) | 07 |
| Model-based | run policy, fit dynamics | imagined rollouts in the model | plan, or train on imagined data | yes | 09 |
| Offline RL | nobody — fixed dataset | conservative Q/V | constrained policy step | only old data | 10 |
On-policy means the "try" data must come from the current policy, so it is thrown away after each update. Off-policy means old data can be reused. That single property moves the bottleneck around between the boxes — and that is the subject of the rest of this lecture.
3. The hardware view: where does wall-clock go?¶
Map the loop onto a machine:
TRY (rollout) IMPROVE (learner)
┌──────────────────────────────────┐ ┌───────────────────────────┐
│ for each step: │ │ for each minibatch: │
│ obs ──► policy forward ──► a │ data │ forward + backward │
│ a ──► simulator step ──► obs│ ────► │ optimizer step │
└──────────────────────────────────┘ └───────────────────────────┘
t_env per step + t_policy per step t_learn per update
For one iteration of an on-policy algorithm with \(N\) parallel environments and \(K\) steps each:
where \(E\) is the number of passes over the data and \(B\) the minibatch size. Which term dominates depends entirely on the setup:
| Regime | Example | \(t_{\text{env}}\) | \(t_{\text{policy}}\) | \(t_{\text{learn}}\) | Bottleneck |
|---|---|---|---|---|---|
| A. CPU toy | Gymnasium CartPole-v1, MLP, 8 envs |
~µs of physics, ~10-50 µs Python overhead per env step | ~0.1 ms per batched forward (launch-bound) | tiny | Python + kernel-launch overhead. The GPU sits idle. |
| B. GPU-parallel sim | ManiSkill3 / Isaac Lab / MJX, 1-4k envs, MLP | one batched GPU step for all envs, ~ms | one batched forward, ~ms | grows with batch | Learner + GPU memory. Throughput depends on scene, rendering, batch, and hardware; data generation still consumes compute. |
| C. VLA in sim | LIBERO + π0.5 (multi-billion params) | sim step on CPU, ~ms | tens of ms per action chunk at batch 1; better per sample when batched | backward through a VLA: large | Policy forward pass. The simulator is cheap; generating actions is the cost. |
| D. Agent in a sandbox (contrast case) | LLM coding/browsing agent, one container per rollout | a shell command or test run: seconds to minutes, plus sandbox boot | LLM generation, seconds per turn | LLM backward | Rollout latency and its long tail. Hundreds of thousands of concurrent sandboxes at the largest labs. |
(Order-of-magnitude figures. The point of the lab is to replace them with your own measurements. Regime D is not used in this course; it is here because LLM-agent RL is where much of today's RL infrastructure is being built, and Lecture 11 maps its four layers back onto the robot pipeline.)
Three consequences follow, and they recur through the whole course:
- Sample efficiency ≠ wall-clock efficiency. In regime B, cheaper rollouts can favor a simple update, but sample requirements and simulation/rendering costs still matter. In regime C, every sample costs a VLA forward pass, so reusing samples (off-policy, offline, more epochs per batch) becomes valuable.
- Rollouts are inference. In regime C, RL training is an inference-serving problem in disguise: batch actions across many parallel environments, keep the policy on the GPU, use CUDA Graphs, cut flow-matching steps where accuracy allows. Everything in the VLA Optimization course speeds up training too, not only deployment.
- Action chunks are a throughput lever. If the policy emits a 50-step chunk and you execute \(H\) of those steps before re-planning, the policy runs \(T/H\) times per episode instead of \(T\). For a 300-step episode with \(H = 10\), that is 30 VLA calls instead of 300. (openpi's LIBERO config for π0.5 predicts 10-action chunks and executes 5, so a 300-step LIBERO episode costs 60 calls.) Lecture 13 shows that chunking also shortens the effective horizon that errors compound over.
3.1 A back-of-envelope for regime C¶
For an episode of \(T\) control steps, executing \(H\) actions per chunk, with per-chunk inference time \(t_{\text{chunk}}(N)\) at batch size \(N\):
Plug in your measured \(t_{\text{chunk}}\) for batch 1, 8, and 32. You will usually find that batching across environments is the single biggest win. This is also why RL frameworks for VLAs separate "rollout workers" (batched inference servers) from "env workers" (CPU simulators).
4. Build it: the rollout benchmark¶
You will build rollout_bench/, a small harness that measures the three terms above for each rung of the course's environment ladder.
Lab 1a — CPU vector envs (regime A)¶
# rollout_bench/cpu_vec.py
import time, numpy as np, gymnasium as gym
def bench(env_id: str, num_envs: int, mode: str, steps: int = 2000) -> float:
envs = gym.make_vec(env_id, num_envs=num_envs, vectorization_mode=mode)
envs.reset(seed=0)
t0 = time.perf_counter()
for _ in range(steps):
envs.step(envs.action_space.sample()) # random policy: isolates env cost
dt = time.perf_counter() - t0
envs.close()
return num_envs * steps / dt # env steps per second
for mode in ("sync", "async"):
for n in (1, 2, 4, 8, 16, 32, 64):
print(f"{mode:5s} n={n:3d} {bench('CartPole-v1', n, mode):>12,.0f} steps/s")
Expected shape: sync scales sub-linearly and flattens. async (one process per env) is slower for tiny envs like CartPole, because inter-process communication costs more than the physics. It only wins when each step is expensive (MuJoCo, rendering). Write down the crossover point.
Lab 1b — GPU-parallel simulation (regime B)¶
# rollout_bench/gpu_sim.py
import time, torch, gymnasium as gym
import mani_skill.envs # registers ManiSkill3 tasks
def bench(num_envs: int, steps: int = 200) -> tuple[float, float]:
env = gym.make("PickCube-v1", num_envs=num_envs, obs_mode="state") # GPU sim when num_envs > 1
env.reset(seed=0)
torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()
t0 = time.perf_counter()
for _ in range(steps):
env.step(env.action_space.sample())
torch.cuda.synchronize()
dt = time.perf_counter() - t0
mem = torch.cuda.max_memory_allocated() / 2**30
env.close()
return num_envs * steps / dt, mem
for n in (1, 16, 64, 256, 1024, 4096):
sps, mem = bench(n)
print(f"n={n:5d} {sps:>12,.0f} steps/s peak {mem:5.2f} GiB")
Then switch obs_mode to "rgb" and rerun. Rendering cameras is often the real cost in pixel-based robot RL; measure how much it drops steps/s and adds memory per environment.
Lab 1c — Policy forward cost (the regime B → C transition)¶
Hold the environment fixed and replace the random policy with policies of increasing size, timing only the forward pass at batch sizes matching your env counts:
| Policy | Params | How to get it |
|---|---|---|
| MLP 2×256 | ~0.1 M | torch.nn.Sequential |
| Small CNN encoder + MLP | ~1-5 M | NatureCNN-style |
| ViT-B/16 image encoder | ~86 M | torchvision.models.vit_b_16 |
| π0.5 (optional, needs ~16+ GB) | multi-B | openpi inference example; time one action-chunk call |
@torch.inference_mode()
def time_forward(policy, obs, iters=50):
for _ in range(5): policy(obs) # warmup / autotune
torch.cuda.synchronize(); t0 = time.perf_counter()
for _ in range(iters): policy(obs)
torch.cuda.synchronize()
return (time.perf_counter() - t0) / iters # seconds per batched call
For π0.5, also vary the number of flow-matching integration steps if the config exposes it (the default is around 10). You will reuse this number in Lecture 03, and again in the capstone, when deciding how many denoising steps rollouts can afford.
5. Use it in the real stack¶
The production versions of what you just built:
- GPU-parallel simulators: Isaac Lab (PhysX on GPU, the standard for legged/humanoid RL), ManiSkill3 (GPU-parallel manipulation with rendering), MuJoCo Playground (MJX/JAX), Genesis.
- CPU-simulated VLA benchmarks: LIBERO (robosuite/MuJoCo), SimplerEnv, RoboCasa. These are usually run as many CPU worker processes feeding a batched GPU policy server.
- RL frameworks that separate rollout from learning: CleanRL (single-file, read these), rsl_rl (Isaac Lab's PPO), Stable-Baselines3, and VLA-scale frameworks such as RLinf that schedule rollout, inference, and training across GPUs.
5.1 The environment contract outlives every environment¶
Notice that every tool above, from CartPole to LIBERO, is driven through the same two calls: reset() starts an episode, and step(action) returns an observation, a reward, and whether the episode ended. That contract has a history worth knowing, because it repeats:
| Year | Milestone | What it added |
|---|---|---|
| 2012 | Arcade Learning Environment (ALE) | Atari games as a shared research platform |
| 2013-15 | DQN | The breakthrough, learning Atari from pixels, built on top of ALE |
| 2016 | OpenAI Gym | The reset / step contract; CartPole as its hello world |
| 2016 | OpenAI Universe | Games, websites and apps through screen pixels, keyboard and mouse, exposed through Gym. Archived without becoming the general training platform it aimed to be. |
| 2022 | Gymnasium | Gym's maintained successor ("a drop-in replacement"). The Gym repo is archived; the API survived. |
| 2025 | OpenEnv (Meta-PyTorch + Hugging Face) | The same Gymnasium-style reset / step / state, served over WebSockets, so environments for LLM agents ship as isolated, remote artifacts |
Two lessons carry over to robot learning. First, the environment comes before the breakthrough: ALE before DQN, ManiSkill/Isaac/LIBERO before today's robot RL results. Second, Universe's failure was not technical: it had real programs, Docker, and a working starter agent. It was missing a pretrained model worth starting from, tasks within reach of that model (so that some attempts succeed and RL gets signal), an interface that is easy to explore, and a practical recipe for sparse rewards. Every one of those maps onto this course: π0.5 is the pretrained starting point, perturbed layouts are the tasks within reach (Lecture 12), chunked actions are the easier interface, and group-relative RL is the recipe (Lectures 04 and 11). (This framing follows S. Paniego, "Why RL environments work now (and could not in 2016)", Hugging Face community blog, Sep 2026.)
Read cleanrl/ppo_continuous_action.py end to end now, before Lecture 04. It is about 300 lines, and you will be able to point to the try/judge/improve boxes in it.
6. Measure it¶
| Metric | Definition | Why it matters |
|---|---|---|
| env-steps/s | Simulator transitions per second, random policy | Upper bound on data generation |
| policy-inferences/s | Batched forward calls per second × batch | The real data rate once the policy is non-trivial |
| effective steps/s | Steps/s with the actual policy in the loop | What your training run will actually see |
| learner-updates/s | Optimizer steps per second at your minibatch size | Bounds how much you can reuse each sample |
| peak GPU memory | torch.cuda.max_memory_allocated |
Sets max num_envs and whether a critic/reference copy fits |
| samples-to-success | Env steps to reach X% success | Sample efficiency (algorithm property) |
| wall-clock-to-success | Seconds to reach X% success | What you actually pay (algorithm × hardware) |
Report the first five for each ladder rung now. The last two start appearing from Lecture 04.
7. Ship it¶
Commit rollout_bench/ containing:
cpu_vec.py,gpu_sim.py,policy_fwd.pyand arun_all.shresults.csvwith one row per (environment, num_envs, obs_mode, policy, batch) and the metrics abovethroughput.png— steps/s vs num_envs on a log-log plot, one line per policy sizeWHERE_TIME_GOES.md— one paragraph per regime naming the bottleneck, with your numbers, and one sentence predicting which algorithm family (on-policy vs off-policy) that regime favors
That note is the first entry in your course cost ledger. Later lectures will compare against it.
Exit criteria¶
You can move on when you can:
- write \(J(\theta)\), \(V^\pi\), \(Q^\pi\), and \(A^\pi\) from memory and explain each in one plain sentence
- explain why a 0/1 reward still yields a smooth objective
- place any algorithm in the try/judge/improve table and say whether it is on- or off-policy
- show, with your own measurements, which term dominates \(t_{\text{iter}}\) in each of the three regimes
- compute how many VLA forward passes a LIBERO episode needs at a given chunk-execution horizon \(H\)
Self-check¶
- Your arm's camera can't see whether the drawer is already open. Is this an MDP or a POMDP, and what are two ways to give the policy what it needs?
- Two algorithms reach 80% success on PickCube. A needs 5 M env steps and 200 learner updates. B needs 0.5 M env steps and 50,000 learner updates. In regime B (GPU sim, MLP policy), which is likely faster in wall-clock? In regime C (VLA policy)? Show your reasoning with the \(t_{\text{iter}}\) formula.
- With \(\gamma = 1\) and a reward of 1 only on success, what is \(V^\pi(s)\), in words? Why does that make it a convenient thing for a critic to predict?
- Lab 1a shows
asyncvector envs are slower thansyncfor CartPole. Why? For what kind of environment does that flip? - A π0.5 action chunk costs 60 ms at batch 1 and 120 ms at batch 16. A LIBERO episode is 300 steps and you execute 10 actions per chunk. Estimate episodes per second for batch 1 vs batch 16, ignoring simulator cost.
References¶
- CS 285 Lecture 4 "Introduction to RL" — slides and video
- Sutton & Barto, Reinforcement Learning: An Introduction, ch. 3 (finite MDPs) — online
- Gymnasium vector environments — docs
- ManiSkill3: GPU-parallelized robotics simulation and rendering — paper, docs
- Isaac Lab — docs
- Bellemare et al., "The Arcade Learning Environment: An Evaluation Platform for General Agents", 2012 — arXiv:1207.4708
- Brockman et al., "OpenAI Gym", 2016 — arXiv:1606.01540
- OpenEnv — repo
- S. Paniego, "Why RL environments work now (and could not in 2016)", Hugging Face community blog, Sep 2026
- Huang et al., "The 37 Implementation Details of PPO" — ICLR blog track (good preview of how much engineering hides in an RL loop)