Skip to content

Lecture 01: The RL Problem and the Rollout Engine

Overview

Reinforcement learning means learning by trying things and getting a score, instead of being told the right answer. It sounds like a small change from supervised learning, but two things change at once:

  • Your actions change your next input. In supervised learning, getting example 3 wrong does not change example 4. In RL, if the gripper misses the bowl, the next camera frame shows a missed bowl. The data distribution is produced by the policy you are training.
  • Nobody tells you the right action. You get a reward — often just 1 if the task finished and 0 if it did not, at the very end of a 300-step episode.

This lecture sets up the vocabulary used for the rest of the course, and then asks the question every robot-learning team eventually hits: when you run an RL algorithm, where does the wall-clock time actually go? You will measure that for three very different setups — a CPU toy, a GPU-parallel manipulation simulator, and a billion-parameter VLA policy — because the answer decides which algorithms are practical.

Running example used throughout the course: a 7-DoF robot arm on a simulated kitchen table, told in words to "put the bowl on the plate."

What it is for our arm
Observation 1-3 RGB camera images (224×224), the language instruction, and proprioceptive state (joint angles, gripper width)
Action A 7-D vector: 3 translation + 3 rotation deltas for the end effector, 1 gripper command. VLAs emit a chunk of ~10-50 of these at once.
Reward Sparse and binary: 1 when the task predicate is true (bowl on plate), else 0
Episode Up to a few hundred control steps at 10-20 Hz

By the end you should be able to:

  • write down the MDP objective and define V, Q, and the advantage in one line each
  • explain why a binary success signal still gives you something smooth to optimize
  • classify any RL algorithm by how it does try, judge, and improve
  • measure env-steps/s, policy-inferences/s, and learner-updates/s for a training setup, and say which one is the bottleneck
  • explain why sample efficiency and wall-clock efficiency rank algorithms differently

1. Why it matters: two ways to teach a computer

Supervised learning Reinforcement learning
Signal The correct answer for every input A scalar score, often delayed and sparse
Data Fixed, i.i.d. dataset Generated by the current policy; changes as it learns
Failure mode Overfitting Compounding errors, exploitation of the simulator, no signal at all if it never succeeds
Ceiling As good as the labels Can exceed the demonstrator

The last row is why RL matters for robotics. Copying expert demonstrations (Lectures 02-03) can only make a policy as good as the demonstrations — and in practice somewhat worse, because errors compound. Practicing against a score can find behaviors that are better than the examples: faster grasps, recoveries the teleoperator never showed, robustness to layouts nobody recorded.

The modern recipe uses both, in this order:

  large, messy data          small, clean demos            practice with a score
  (pre-training)      ──►    (supervised fine-tune)  ──►   (RL post-training)
  general knowledge          "what we want"                "better than the demos"

This is the same three-stage pipeline used for chat LLMs (Lecture 11). π0.5 arrives at stage 2 already done; the capstone (Lecture 14) is stage 3.


2. Mental model: the rules of the game

Every RL problem is a Markov decision process (MDP):

Symbol Name Meaning
\(s_t\) state Everything about the world at time \(t\) (all object poses, velocities, joint states)
\(o_t\) observation What the policy actually receives (pixels + instruction + proprio). Usually less than \(s_t\).
\(a_t\) action What the policy does
\(p(s_{t+1} \mid s_t, a_t)\) dynamics How the world responds — the simulator, or physics
\(r(s_t, a_t)\) reward The score for this step
\(\pi_\theta(a_t \mid o_t)\) policy The network with parameters \(\theta\) — the thing we train
\(\gamma \in [0,1)\) discount How much later rewards count relative to sooner ones
\(T\) horizon Episode length

Markov property: if \(s_t\) is complete, the future depends only on \(s_t\) and \(a_t\), not on the history. When the policy sees only \(o_t\) (a camera can't see inside the drawer), the problem is a POMDP, and the policy may need memory — which is why VLAs often take a short history of frames.

2.1 The objective

A trajectory \(\tau = (s_0, a_0, s_1, a_1, \dots)\) is sampled by running the policy in the world. The goal is the highest expected total reward:

\[ J(\theta) = \mathbb{E}_{\tau \sim p_\theta(\tau)} \left[ \sum_{t=0}^{T} \gamma^t \, r(s_t, a_t) \right] \]

Two words in that equation carry most of the weight:

  • Total. We optimize the whole episode, not the next step. Braking before the crash, not at it. Reaching around the bowl to approach it from the side even though that temporarily moves the gripper away.
  • Expected. Each episode is random (initial layout, policy sampling noise, physics). For our arm, every episode scores 0 or 1 — a step function. But the success rate \(J(\theta) = \Pr_\theta[\text{success}]\) changes smoothly as \(\theta\) changes, because the probability of each trajectory changes smoothly. That smoothness is what lets gradient methods work on a reward that is itself not differentiable.

2.2 Value, Q, and advantage

Three quantities show up in every later lecture:

\[ V^\pi(s) = \mathbb{E}_\pi\!\left[\textstyle\sum_{t' \ge t} \gamma^{t'-t} r_{t'} \;\middle|\; s_t = s\right] \quad\text{"how good is this place?"} \]
\[ Q^\pi(s, a) = \mathbb{E}_\pi\!\left[\textstyle\sum_{t' \ge t} \gamma^{t'-t} r_{t'} \;\middle|\; s_t = s, a_t = a\right] \quad\text{"how good is this move, here?"} \]
\[ A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s) \quad\text{"how much better than usual is this move?"} \]

With a sparse binary reward and \(\gamma \approx 1\), \(V^\pi(s)\) is simply the probability of eventually succeeding from state \(s\) under the current policy. That is a very intuitive thing for a critic to learn, and Lecture 05 will exploit it.

2.3 Try → Judge → Improve

Every RL algorithm in this course is a loop of three steps:

        ┌──────────────────────────────────────────────────────────┐
        ▼                                                          │
   ┌─────────┐        ┌──────────────────┐        ┌─────────────┐  │
   │  TRY    │  ───►  │  JUDGE           │  ───►  │  IMPROVE    │ ─┘
   │ collect │        │ estimate return, │        │ update the  │
   │ rollouts│        │ V, Q, or a model │        │ policy      │
   └─────────┘        └──────────────────┘        └─────────────┘
   env + policy fwd    critic / returns           policy backward
   (inference)         (cheap or a 2nd net)       (training)

The algorithms differ in how each box works:

Algorithm family Try Judge Improve Can reuse old data? Lecture
Behavioral cloning expert does it (none — expert is right) supervised regression yes (only expert data) 02-03
Policy gradient (REINFORCE, GRPO) run current policy sum of rewards, minus a baseline raise log-prob of good actions no (on-policy) 04, 11
Actor-critic / PPO run current policy learned \(V(s)\), GAE clipped policy-gradient step a few epochs 05-06
Q-learning / SAC any policy, stored in replay learned \(Q(s,a)\) via bootstrapping argmax or actor step on Q yes (off-policy) 07
Model-based run policy, fit dynamics imagined rollouts in the model plan, or train on imagined data yes 09
Offline RL nobody — fixed dataset conservative Q/V constrained policy step only old data 10

On-policy means the "try" data must come from the current policy, so it is thrown away after each update. Off-policy means old data can be reused. That single property moves the bottleneck around between the boxes — and that is the subject of the rest of this lecture.


3. The hardware view: where does wall-clock go?

Map the loop onto a machine:

   TRY  (rollout)                               IMPROVE  (learner)
   ┌──────────────────────────────────┐        ┌───────────────────────────┐
   │  for each step:                  │        │  for each minibatch:      │
   │    obs  ──► policy forward ──► a │  data  │    forward + backward     │
   │    a    ──► simulator step  ──► obs│ ────► │    optimizer step         │
   └──────────────────────────────────┘        └───────────────────────────┘
       t_env per step  +  t_policy per step         t_learn per update

For one iteration of an on-policy algorithm with \(N\) parallel environments and \(K\) steps each:

\[ t_{\text{iter}} \approx K \cdot \big( t_{\text{env}}(N) + t_{\text{policy}}(N) \big) \;+\; E \cdot \frac{N K}{B} \cdot t_{\text{learn}}(B) \]

where \(E\) is the number of passes over the data and \(B\) the minibatch size. Which term dominates depends entirely on the setup:

Regime Example \(t_{\text{env}}\) \(t_{\text{policy}}\) \(t_{\text{learn}}\) Bottleneck
A. CPU toy Gymnasium CartPole-v1, MLP, 8 envs ~µs of physics, ~10-50 µs Python overhead per env step ~0.1 ms per batched forward (launch-bound) tiny Python + kernel-launch overhead. The GPU sits idle.
B. GPU-parallel sim ManiSkill3 / Isaac Lab / MJX, 1-4k envs, MLP one batched GPU step for all envs, ~ms one batched forward, ~ms grows with batch Learner + GPU memory. Throughput depends on scene, rendering, batch, and hardware; data generation still consumes compute.
C. VLA in sim LIBERO + π0.5 (multi-billion params) sim step on CPU, ~ms tens of ms per action chunk at batch 1; better per sample when batched backward through a VLA: large Policy forward pass. The simulator is cheap; generating actions is the cost.
D. Agent in a sandbox (contrast case) LLM coding/browsing agent, one container per rollout a shell command or test run: seconds to minutes, plus sandbox boot LLM generation, seconds per turn LLM backward Rollout latency and its long tail. Hundreds of thousands of concurrent sandboxes at the largest labs.

(Order-of-magnitude figures. The point of the lab is to replace them with your own measurements. Regime D is not used in this course; it is here because LLM-agent RL is where much of today's RL infrastructure is being built, and Lecture 11 maps its four layers back onto the robot pipeline.)

Three consequences follow, and they recur through the whole course:

  1. Sample efficiency ≠ wall-clock efficiency. In regime B, cheaper rollouts can favor a simple update, but sample requirements and simulation/rendering costs still matter. In regime C, every sample costs a VLA forward pass, so reusing samples (off-policy, offline, more epochs per batch) becomes valuable.
  2. Rollouts are inference. In regime C, RL training is an inference-serving problem in disguise: batch actions across many parallel environments, keep the policy on the GPU, use CUDA Graphs, cut flow-matching steps where accuracy allows. Everything in the VLA Optimization course speeds up training too, not only deployment.
  3. Action chunks are a throughput lever. If the policy emits a 50-step chunk and you execute \(H\) of those steps before re-planning, the policy runs \(T/H\) times per episode instead of \(T\). For a 300-step episode with \(H = 10\), that is 30 VLA calls instead of 300. (openpi's LIBERO config for π0.5 predicts 10-action chunks and executes 5, so a 300-step LIBERO episode costs 60 calls.) Lecture 13 shows that chunking also shortens the effective horizon that errors compound over.

3.1 A back-of-envelope for regime C

For an episode of \(T\) control steps, executing \(H\) actions per chunk, with per-chunk inference time \(t_{\text{chunk}}(N)\) at batch size \(N\):

\[ \text{episodes/s} \approx \frac{N}{(T/H) \cdot t_{\text{chunk}}(N) + T \cdot t_{\text{env}}} \]

Plug in your measured \(t_{\text{chunk}}\) for batch 1, 8, and 32. You will usually find that batching across environments is the single biggest win. This is also why RL frameworks for VLAs separate "rollout workers" (batched inference servers) from "env workers" (CPU simulators).


4. Build it: the rollout benchmark

You will build rollout_bench/, a small harness that measures the three terms above for each rung of the course's environment ladder.

Lab 1a — CPU vector envs (regime A)

# rollout_bench/cpu_vec.py
import time, numpy as np, gymnasium as gym

def bench(env_id: str, num_envs: int, mode: str, steps: int = 2000) -> float:
    envs = gym.make_vec(env_id, num_envs=num_envs, vectorization_mode=mode)
    envs.reset(seed=0)
    t0 = time.perf_counter()
    for _ in range(steps):
        envs.step(envs.action_space.sample())   # random policy: isolates env cost
    dt = time.perf_counter() - t0
    envs.close()
    return num_envs * steps / dt                 # env steps per second

for mode in ("sync", "async"):
    for n in (1, 2, 4, 8, 16, 32, 64):
        print(f"{mode:5s} n={n:3d}  {bench('CartPole-v1', n, mode):>12,.0f} steps/s")

Expected shape: sync scales sub-linearly and flattens. async (one process per env) is slower for tiny envs like CartPole, because inter-process communication costs more than the physics. It only wins when each step is expensive (MuJoCo, rendering). Write down the crossover point.

Lab 1b — GPU-parallel simulation (regime B)

# rollout_bench/gpu_sim.py
import time, torch, gymnasium as gym
import mani_skill.envs  # registers ManiSkill3 tasks

def bench(num_envs: int, steps: int = 200) -> tuple[float, float]:
    env = gym.make("PickCube-v1", num_envs=num_envs, obs_mode="state")  # GPU sim when num_envs > 1
    env.reset(seed=0)
    torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()
    t0 = time.perf_counter()
    for _ in range(steps):
        env.step(env.action_space.sample())
    torch.cuda.synchronize()
    dt = time.perf_counter() - t0
    mem = torch.cuda.max_memory_allocated() / 2**30
    env.close()
    return num_envs * steps / dt, mem

for n in (1, 16, 64, 256, 1024, 4096):
    sps, mem = bench(n)
    print(f"n={n:5d}  {sps:>12,.0f} steps/s  peak {mem:5.2f} GiB")

Then switch obs_mode to "rgb" and rerun. Rendering cameras is often the real cost in pixel-based robot RL; measure how much it drops steps/s and adds memory per environment.

Lab 1c — Policy forward cost (the regime B → C transition)

Hold the environment fixed and replace the random policy with policies of increasing size, timing only the forward pass at batch sizes matching your env counts:

Policy Params How to get it
MLP 2×256 ~0.1 M torch.nn.Sequential
Small CNN encoder + MLP ~1-5 M NatureCNN-style
ViT-B/16 image encoder ~86 M torchvision.models.vit_b_16
π0.5 (optional, needs ~16+ GB) multi-B openpi inference example; time one action-chunk call
@torch.inference_mode()
def time_forward(policy, obs, iters=50):
    for _ in range(5): policy(obs)              # warmup / autotune
    torch.cuda.synchronize(); t0 = time.perf_counter()
    for _ in range(iters): policy(obs)
    torch.cuda.synchronize()
    return (time.perf_counter() - t0) / iters  # seconds per batched call

For π0.5, also vary the number of flow-matching integration steps if the config exposes it (the default is around 10). You will reuse this number in Lecture 03, and again in the capstone, when deciding how many denoising steps rollouts can afford.


5. Use it in the real stack

The production versions of what you just built:

  • GPU-parallel simulators: Isaac Lab (PhysX on GPU, the standard for legged/humanoid RL), ManiSkill3 (GPU-parallel manipulation with rendering), MuJoCo Playground (MJX/JAX), Genesis.
  • CPU-simulated VLA benchmarks: LIBERO (robosuite/MuJoCo), SimplerEnv, RoboCasa. These are usually run as many CPU worker processes feeding a batched GPU policy server.
  • RL frameworks that separate rollout from learning: CleanRL (single-file, read these), rsl_rl (Isaac Lab's PPO), Stable-Baselines3, and VLA-scale frameworks such as RLinf that schedule rollout, inference, and training across GPUs.

5.1 The environment contract outlives every environment

Notice that every tool above, from CartPole to LIBERO, is driven through the same two calls: reset() starts an episode, and step(action) returns an observation, a reward, and whether the episode ended. That contract has a history worth knowing, because it repeats:

Year Milestone What it added
2012 Arcade Learning Environment (ALE) Atari games as a shared research platform
2013-15 DQN The breakthrough, learning Atari from pixels, built on top of ALE
2016 OpenAI Gym The reset / step contract; CartPole as its hello world
2016 OpenAI Universe Games, websites and apps through screen pixels, keyboard and mouse, exposed through Gym. Archived without becoming the general training platform it aimed to be.
2022 Gymnasium Gym's maintained successor ("a drop-in replacement"). The Gym repo is archived; the API survived.
2025 OpenEnv (Meta-PyTorch + Hugging Face) The same Gymnasium-style reset / step / state, served over WebSockets, so environments for LLM agents ship as isolated, remote artifacts

Two lessons carry over to robot learning. First, the environment comes before the breakthrough: ALE before DQN, ManiSkill/Isaac/LIBERO before today's robot RL results. Second, Universe's failure was not technical: it had real programs, Docker, and a working starter agent. It was missing a pretrained model worth starting from, tasks within reach of that model (so that some attempts succeed and RL gets signal), an interface that is easy to explore, and a practical recipe for sparse rewards. Every one of those maps onto this course: π0.5 is the pretrained starting point, perturbed layouts are the tasks within reach (Lecture 12), chunked actions are the easier interface, and group-relative RL is the recipe (Lectures 04 and 11). (This framing follows S. Paniego, "Why RL environments work now (and could not in 2016)", Hugging Face community blog, Sep 2026.)

Read cleanrl/ppo_continuous_action.py end to end now, before Lecture 04. It is about 300 lines, and you will be able to point to the try/judge/improve boxes in it.


6. Measure it

Metric Definition Why it matters
env-steps/s Simulator transitions per second, random policy Upper bound on data generation
policy-inferences/s Batched forward calls per second × batch The real data rate once the policy is non-trivial
effective steps/s Steps/s with the actual policy in the loop What your training run will actually see
learner-updates/s Optimizer steps per second at your minibatch size Bounds how much you can reuse each sample
peak GPU memory torch.cuda.max_memory_allocated Sets max num_envs and whether a critic/reference copy fits
samples-to-success Env steps to reach X% success Sample efficiency (algorithm property)
wall-clock-to-success Seconds to reach X% success What you actually pay (algorithm × hardware)

Report the first five for each ladder rung now. The last two start appearing from Lecture 04.


7. Ship it

Commit rollout_bench/ containing:

  • cpu_vec.py, gpu_sim.py, policy_fwd.py and a run_all.sh
  • results.csv with one row per (environment, num_envs, obs_mode, policy, batch) and the metrics above
  • throughput.png — steps/s vs num_envs on a log-log plot, one line per policy size
  • WHERE_TIME_GOES.md — one paragraph per regime naming the bottleneck, with your numbers, and one sentence predicting which algorithm family (on-policy vs off-policy) that regime favors

That note is the first entry in your course cost ledger. Later lectures will compare against it.


Exit criteria

You can move on when you can:

  • write \(J(\theta)\), \(V^\pi\), \(Q^\pi\), and \(A^\pi\) from memory and explain each in one plain sentence
  • explain why a 0/1 reward still yields a smooth objective
  • place any algorithm in the try/judge/improve table and say whether it is on- or off-policy
  • show, with your own measurements, which term dominates \(t_{\text{iter}}\) in each of the three regimes
  • compute how many VLA forward passes a LIBERO episode needs at a given chunk-execution horizon \(H\)

Self-check

  1. Your arm's camera can't see whether the drawer is already open. Is this an MDP or a POMDP, and what are two ways to give the policy what it needs?
  2. Two algorithms reach 80% success on PickCube. A needs 5 M env steps and 200 learner updates. B needs 0.5 M env steps and 50,000 learner updates. In regime B (GPU sim, MLP policy), which is likely faster in wall-clock? In regime C (VLA policy)? Show your reasoning with the \(t_{\text{iter}}\) formula.
  3. With \(\gamma = 1\) and a reward of 1 only on success, what is \(V^\pi(s)\), in words? Why does that make it a convenient thing for a critic to predict?
  4. Lab 1a shows async vector envs are slower than sync for CartPole. Why? For what kind of environment does that flip?
  5. A π0.5 action chunk costs 60 ms at batch 1 and 120 ms at batch 16. A LIBERO episode is 300 steps and you execute 10 actions per chunk. Estimate episodes per second for batch 1 vs batch 16, ignoring simulator cost.

References

  • CS 285 Lecture 4 "Introduction to RL" — slides and video
  • Sutton & Barto, Reinforcement Learning: An Introduction, ch. 3 (finite MDPs) — online
  • Gymnasium vector environments — docs
  • ManiSkill3: GPU-parallelized robotics simulation and rendering — paper, docs
  • Isaac Lab — docs
  • Bellemare et al., "The Arcade Learning Environment: An Evaluation Platform for General Agents", 2012 — arXiv:1207.4708
  • Brockman et al., "OpenAI Gym", 2016 — arXiv:1606.01540
  • OpenEnv — repo
  • S. Paniego, "Why RL environments work now (and could not in 2016)", Hugging Face community blog, Sep 2026
  • Huang et al., "The 37 Implementation Details of PPO" — ICLR blog track (good preview of how much engineering hides in an RL loop)

Next in this special course