Lecture 06b: Actor-Critic PPO in Practice: Inside RSL-RL¶
Overview¶
Lecture 05 introduces actor-critic and GAE; Lecture 06 adds PPO. This lecture reads RSL-RL, a supported robot-learning backend used with Isaac Lab and other simulator integrations. It is one implementation with specific choices; a paper reporting PPO need not use this library or its defaults.
The pinned algorithm and runner are compact enough to inspect alongside the equations. Read the exact tag named below; source organization, APIs, and line counts can change. The custom-environment training exercise is an integration task, not evidence that this checkout has executed an RSL-RL training run.
Version pin for this standalone lecture: rsl-rl-lib==5.5.1, tag v5.5.1. Line numbers refer to that tag. This differs from Isaac Lab's September v3.0.0-EA tag, which pins 5.4.1; a later reviewed 3.0 branch commit uses 5.5.1. See the exact compatibility record. Older tutorials use a single ActorCritic module and a policy= config; Section 9 explains the changed organization.
By the end you should be able to:
- draw RSL-RL's runner → algorithm → models → storage structure and say what each piece owns
- point to the exact lines that implement GAE, the clipped surrogate, the clipped value loss, and the adaptive learning rate
- explain the time-out bootstrapping trick, what approximation it makes, and when Isaac Lab turns it on
- configure an asymmetric actor-critic with observation groups and per-model normalization
- contrast RSL-RL's defaults with a CleanRL-style PPO and predict which differences matter for a given task
- train RSL-RL on a custom vectorized environment and read its training logs
1. Why it matters: inspect a concrete robot PPO implementation¶
The same 500 lines of PPO train ANYmal and Unitree quadrupeds, humanoids, and Franka manipulation tasks across several simulators. Three reasons to know it well:
- Reproducing papers. For a result that uses RSL-RL, inspect its actual configuration rather than assuming defaults: adaptive learning rate, clipped value loss, separate actor and critic networks, a short rollout over thousands of environments. If you reimplement PPO "from the paper", you will not match their curves until you match these choices.
- Debugging your own training. When an Isaac Lab run plateaus, the fix is usually in this file: a value target off by a time-out, a reward scale fighting the value clip, a learning rate the KL rule drove to its floor. You need to know which line is responsible.
- Extending it. The library's purpose is to be modified. Asymmetric critics, symmetry augmentation, curiosity, and student-teacher distillation are all small, readable additions to the same loop.
2. Mental model: four pieces and a loop¶
OnPolicyRunner.learn() rsl_rl/runners/on_policy_runner.py
└─ for it in iterations:
├─ with torch.inference_mode(): ── ROLLOUT (collect_time)
│ for t in range(num_steps_per_env):
│ a = alg.act(obs) actor samples a; critic predicts V(s); log π(a|s) stored
│ obs, r, done, extras = env.step(a)
│ alg.process_env_step(obs, r, done, extras) time-out bootstrap; store transition
│ alg.compute_returns(obs) GAE, backward over the T stored steps
└─ alg.update() ── LEARN (learn_time)
for epoch × minibatch: recompute log π, V, H → adapt lr from KL → loss → clip grads → Adam step
then update observation normalizers; clear storage
PPO rsl_rl/algorithms/ppo.py the algorithm
MLPModel ×2 rsl_rl/models/mlp_model.py actor (+ GaussianDistribution), critic
RolloutStorage rsl_rl/storage/rollout_storage.py T × N tensors, on the training device
VecEnv rsl_rl/env/vec_env.py get_observations() → TensorDict; step()
This is Lecture 01's try → judge → improve loop, with the boxes named. Four structural choices define the library:
- The actor and critic are separate networks (two
MLPModels,ppo.py:92-93) trained by one optimizer over both parameter sets (ppo.py:100-102). There is no shared trunk by default. This lets the critic take different inputs from the actor (Section 3). The exception is image encoders, which can be shared withshare_cnn_encoders. - Everything lives on the training device. The environment returns GPU tensors, the storage is preallocated on the GPU, and the rollout runs under
torch.inference_mode(). Nothing crosses to the CPU in the hot loop. - Short rollouts over many environments. A typical Isaac Lab config collects 24 steps from each of thousands of environments per iteration, then runs 5 epochs × 4 minibatches. Lecture 06 §8.1 explained why that batch shape suits GPU simulation.
- Observations are dictionaries. The environment returns a
TensorDictof named observation groups, and a config maps groups to models.
3. Observation groups: the asymmetric critic in one dictionary¶
Lecture 05 §6 argued that in simulation the critic should see privileged state the actor cannot: exact object poses, friction coefficients, contact forces. RSL-RL makes that a configuration choice. The environment returns several groups:
and the runner config maps observation sets (what each model needs) to groups (what the environment provides):
Each model concatenates its groups in order (mlp_model.py:113-118). resolve_obs_groups (utils/utils.py:192) fills in a missing set: first with a group of the same name, then with the "policy" group, and otherwise it raises an error. The sets RSL-RL knows are actor, critic, student, teacher, and rnd_state.
Normalization is per model. With obs_normalization=True, each model owns an EmpiricalNormalization module (modules/normalization.py), a running mean and variance over the whole batch (not per environment). It normalizes as \((x - \mu) / (\sigma + 10^{-2})\). Two details matter:
- The statistics are updated once per iteration, after
update(), from that iteration's rollout (ppo.py:327-330), and only in training mode. During one PPO update the normalizer is frozen, so the old and new policies see identically normalized inputs. That keeps the importance ratio meaningful. - The normalizer is copied into the exported JIT and ONNX policies (
mlp_model.py:196-242). A deployed policy takes raw observations, so there is no separate normalization step to forget on the robot.
4. The rollout: act() and process_env_step()¶
act() (ppo.py:124-135) does four things per step: sample an action from the actor's distribution, compute \(V(s_t)\) with the critic, compute \(\log \pi_{\theta_\text{old}}(a_t \mid s_t)\), and store the distribution parameters (mean and std for a Gaussian). The parameters are stored so that the update can later compute an exact KL divergence between the old and new policies, not just a sample estimate.
The default action distribution is GaussianDistribution: the MLP outputs the mean, and the standard deviation is a state-independent learned parameter, stored as a raw scale by default (std_type="scalar") and clamped to a range. Log-probabilities and entropies are summed over action dimensions. Unbounded Gaussian samples go straight to the environment. Isaac Lab's wrapper can clamp them with clip_actions, but there is no tanh squashing and no Jacobian correction.
4.1 Time-out bootstrapping¶
process_env_step() (ppo.py:137-164) contains the most important line in the file for robot tasks:
# Bootstrapping on time outs ppo.py:153-158
if "time_outs" in extras:
self.transition.rewards += self.gamma * torch.squeeze(
self.transition.values * extras["time_outs"].unsqueeze(1).to(self.device), 1)
Robot episodes are cut by a time limit (say 20 s of walking), and the time limit is not a real terminal state. Lecture 05 §5.1 showed that treating a truncation as termination teaches the critic that the world ends at 20 s. RSL-RL's fix is to fold the missing future into the last reward:
and then let the done flag cut the GAE recursion as usual. Notice that this pinned implementation uses \(V(s_t)\), the value before the step, rather than \(V(s_{t+1})\). It is an approximation in this collection path, where auto-reset can replace the final observation. An adapter can instead preserve the true final observation and evaluate it before reset; this is not a mathematical requirement of GPU simulation. The bootstrap error from substituting the old value is \(\gamma[V(s_t)-V(s_{t+1})]\). A 50 Hz rate alone does not bound that error near contact, failure, or success transitions. Test the assumption against preserved final-state values if it matters for your task.
Isaac Lab's RslRlVecEnvWrapper sets dones = terminated | truncated, and it reports extras["time_outs"] = truncated only for infinite-horizon tasks (cfg.is_finite_horizon == False). For a task that really ends at the time limit (the robot is scored at the final step), the time-out is a real terminal state and must not be bootstrapped. Lab 6b-c measures what happens when bootstrapping is missing.
Test the simultaneous-flags case too. The EA wrapper source forwards truncated without excluding terminated. If your task can set both on the same transition, the reward correction above can bootstrap a genuine terminal state. Preserve both flags and enforce terminal precedence in your task/adapter, for example a time-out correction mask of truncated & ~terminated, under the continuing-task interpretation. The CPU test includes this boundary case.
5. Returns: GAE in sixteen lines¶
compute_returns() (ppo.py:166-191) is Lecture 05's GAE written as a backward loop:
for step in reversed(range(T)):
next_values = last_values if step == T - 1 else values[step + 1]
next_is_not_terminal = 1.0 - dones[step].float()
delta = rewards[step] + next_is_not_terminal * gamma * next_values - values[step] # TD error
advantage = delta + next_is_not_terminal * gamma * lam * advantage # GAE recursion
returns[step] = advantage + values[step] # value target
advantages = returns - values
advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8) # whole batch
That is exactly
with the critic's value of the final observation, last_values, bootstrapping the end of the rollout. That final bootstrap is what makes 24-step rollouts work: GAE never needs an episode to finish. One detail differs from many CleanRL-style implementations. Advantages are normalized once over the whole batch by default. Set normalize_advantage_per_mini_batch=True to normalize each minibatch instead.
6. The update: one loss, line by line¶
update() (ppo.py:193-358) loops over minibatches from mini_batch_generator (rollout_storage.py:225-259). That generator draws one random permutation per update and reuses the same minibatch partition in every epoch. The minibatch size is \(N \cdot T / \texttt{num\_mini\_batches}\). For each minibatch it recomputes \(\log \pi_\theta\), \(V_\phi\), and the entropy under the current parameters, then builds four pieces.
6.1 The adaptive learning rate (ppo.py:240-266)¶
CleanRL-style PPO uses a fixed or annealed learning rate and optionally stops an epoch early when the KL gets too large. RSL-RL instead steers the learning rate with the exact Gaussian KL between the stored old distribution and the current one, measured on every minibatch before its gradient step:
with desired_kl=0.01 by default (schedule="adaptive"). This is a soft trust region in the spirit of Lecture 06 §5. The step size grows while updates are timid and shrinks when the policy moves too far. It also explains a log pattern you will see: the learning rate rises quickly at the start of training, then oscillates. If it sits at the \(10^{-5}\) floor, something is pushing the policy too hard (a reward spike, a bad normalizer); if it is pinned at \(10^{-2}\), the policy has stopped changing. Set schedule="fixed" to disable it.
6.2 The clipped surrogate (ppo.py:268-274)¶
ratio = torch.exp(actions_log_prob - old_actions_log_prob)
surrogate = -advantages * ratio
surrogate_clipped = -advantages * torch.clamp(ratio, 1.0 - clip_param, 1.0 + clip_param)
surrogate_loss = torch.max(surrogate, surrogate_clipped).mean()
The maximum of the negated terms is the minimum of Lecture 06's clipped objective, so this is exactly \(-L^{\text{CLIP}}\).
6.3 The clipped value loss (ppo.py:276-283)¶
RSL-RL turns this on by default and uses the same \(\varepsilon\) (0.2) as the policy clip. Lecture 06 §8 noted that large studies did not find value clipping reliably helpful. In RSL-RL it has a specific side effect you should know: \(\varepsilon\) is in reward units, and once a sample's prediction has moved more than 0.2 from its rollout-time value toward the target, the clipped branch has the larger error, the max selects it, and that sample stops contributing gradient. The critic can move each prediction by at most about 0.2 per update. With small, well-scaled rewards the cap rarely binds. With returns in the hundreds it binds on many samples, and the critic needs more updates to catch up. Whether that slows learning depends on the task. In the Pendulum runs of Section 12, unscaled rewards made no difference beyond seed noise, with or without value clipping, because advantage normalization and \(\lambda = 0.95\) make the policy update fairly tolerant of a lagging critic. So treat reward scale as a hyperparameter to check on your own task, using Loss/value and the critic's explained variance, not as a known failure.
6.4 Total loss, gradients, step (ppo.py:285-311)¶
with value_loss_coef \(c_V = 1.0\) and entropy_coef \(c_H\) (0.01 class default; 0.005 in Isaac Lab's ANYmal-D config). One backward pass covers both networks. Then each network's gradient is clipped separately to max_grad_norm (1.0), and a single Adam step updates both. Separate clipping keeps a large critic gradient, common early in training, from shrinking the actor's step.
7. The defaults, and how they differ¶
| Choice | RSL-RL 5.5.1 (class default · Isaac Lab ANYmal-D config) | CleanRL-style (ManiSkill ppo.py, from Lecture 06 §8) |
|---|---|---|
| Networks | Separate actor and critic, ELU · [512, 256, 128] | Separate actor and critic MLPs |
| Rollout shape | num_steps_per_env 24, thousands of envs |
50 steps × 512 envs |
| Epochs × minibatches | 5 × 4 | 4 × 32 |
| Learning rate | 1e-3, adaptive from KL (target 0.01) | 3e-4 fixed, early stop at target_kl 0.1 |
| Advantage normalization | Whole batch (default) | Per minibatch |
| Value loss | Clipped, coefficient 1.0 | Unclipped, coefficient 0.5 |
| Entropy coefficient | 0.01 · 0.005 | 0.0 |
| Gradient clipping | 1.0 per network | 0.5 global |
| Policy std | State-independent, raw scale, init 1.0 | State-independent log-std, init −0.5 |
| \(\gamma\), \(\lambda\) | 0.99, 0.95 | 0.8, 0.9 (dense reward, 50-step tasks) |
| Time-outs | Bootstrapped through the reward | Bootstrapped from the stored final observation |
Neither column is "correct". The RSL-RL column was tuned on long-horizon locomotion with thousands of environments; the CleanRL column on short manipulation episodes. When porting a task between libraries, change one row at a time and keep your seeds (Lecture 13).
8. Beyond vanilla PPO: what else is in the box¶
Each of these is a small, optional addition to the same loop:
| Feature | Where | What it does | Lecture |
|---|---|---|---|
| Recurrent models | models/rnn_model.py (LSTM/GRU) |
Memory for partial observability. The recurrent minibatch generator splits by environment, not by time, so each minibatch holds whole trajectories (size num_envs // num_mini_batches) |
03 |
| CNN encoders | models/cnn_model.py, share_cnn_encoders |
Pixel observations, optionally one encoder shared by actor and critic | 05 |
| Other distributions | HeteroscedasticGaussianDistribution, BetaDistribution |
State-dependent std; bounded actions | 04 |
| Symmetry | extensions/symmetry.py |
Mirror-augmented minibatches and/or a mirror loss (not with recurrent models) | — |
| Curiosity (RND) | extensions/rnd.py |
Intrinsic reward added in process_env_step, its own optimizer |
12 |
| Student-teacher distillation | algorithms/distillation.py, DistillationRunner |
Train a deployable student on onboard observations to imitate a teacher trained with privileged ones | 02, 05 |
| Multi-GPU | torchrun; NCCL |
One process per GPU: gradients averaged in buckets, KL all-reduced, learning rate broadcast from rank 0, normalizer statistics combined | 12 |
| Speed | torch_compile_mode, use_mixed_precision (bf16 autocast) |
Faster update for larger networks | — |
| Export | export_policy_to_jit, export_policy_to_onnx |
Deterministic (mean) policy with the normalizer included | — |
9. The hardware view¶
Where an iteration's time goes. The logger writes Perf/collection_time (rollout plus GAE), Perf/learning_time (the update), and Perf/total_fps = \(N T / (t_\text{collect} + t_\text{learn})\). In Lecture 01's regime B (GPU simulator, small MLP), collection is mostly simulator time. Learning is num_learning_epochs × num_mini_batches = 20 optimizer steps per iteration, each on a minibatch of \(NT/4\) samples. With 4,096 environments × 24 steps, a minibatch is 24,576 samples, large enough to keep a GPU busy even with a small network. Watch the ratio of the two times. If learning dominates, larger networks, more epochs, or more minibatches cost wall-clock directly, and torch_compile_mode or bf16 are the levers. If collection dominates, the simulator is the bottleneck, and the PPO settings barely matter for speed.
Storage memory is preallocated:
for float32 observations (all groups, stored once), actions, two distribution-parameter tensors, and the scalars (reward, done, value, log-prob, return, advantage). With \(T = 24\), \(N = 4096\), \(d_\text{obs} = 256\), \(d_a = 12\), that is about 117 MB. That is small next to the simulator's footprint. Rollout storage only becomes a problem with image observations, where Isaac Lab Lecture 10 shows the camera Cartpole buffer reaching gigabytes.
On an 8 GB card (the reference GPU for the Isaac Sim and Isaac Lab course), state-based RSL-RL training is limited by the simulator, not by PPO. The actor, critic, Adam state, and storage for a locomotion task fit in a few hundred MB. Reduce --num_envs for the simulator's sake, and expect the KL rule to compensate partly for the smaller batch. Smaller batches give noisier gradients, which tend to produce larger KL per step and push the learning rate down.
10. Build it¶
Save the supplied source block as rslrl_lab/pendulum_rsl.py, creating the directory first. Use an isolated environment with compatible PyTorch and rsl-rl-lib==5.5.1; the latter installs its declared dependencies, including TensorDict. This is an upstream-based implementation exercise, not an executed RSL-RL training result in the course evidence. Begin with --num_envs 8 --steps 8 --iters 1 --device cpu and record actual runtime before choosing a larger run. A GPU speedup depends on workload size and is not guaranteed.
Lab 6b-a — Map the equations to the code¶
Clone the tag and read ppo.py, on_policy_runner.py, rollout_storage.py, and mlp_model.py in full:
Fill in RSLRL_MAP.md, one row per equation from Lectures 05-06: the equation, the file and lines, and one sentence on any difference from the lecture's version. Starter rows, which you should verify:
| Equation | Where in v5.5.1 |
|---|---|
| Sample \(a \sim \pi_\theta\), store \(V\), \(\log\pi\), distribution params | ppo.py:124-135 |
| Time-out bootstrap \(\tilde r_t = r_t + \gamma V(s_t)\) | ppo.py:153-158 |
| GAE and value targets | ppo.py:166-191 |
| KL-adaptive learning rate | ppo.py:240-266 |
| Clipped surrogate | ppo.py:268-274 |
| Clipped value loss | ppo.py:276-283 |
| Total loss, per-network gradient clipping | ppo.py:285-311 |
| Normalizer update after the epoch loop | ppo.py:327-330 |
Lab 6b-b — RSL-RL on a vectorized environment you wrote¶
RSL-RL trains anything that implements VecEnv: get_observations() returning a TensorDict, and step() returning (obs, rewards, dones, extras). This environment is Gymnasium's Pendulum-v1, vectorized in torch. Episodes only end by time-out after 200 steps, which makes it a clean test of Section 4.1.
# rslrl_lab/pendulum_rsl.py — train RSL-RL PPO on a batched, torch-native Pendulum.
import argparse, math, time, torch
from tensordict import TensorDict
from rsl_rl.env import VecEnv
from rsl_rl.runners import OnPolicyRunner
class TorchPendulum(VecEnv):
"""Gymnasium Pendulum-v1 dynamics, vectorized in torch. Episodes only end by time-out (200 steps)."""
def __init__(self, num_envs, device="cpu", reward_scale=1.0, report_timeouts=True):
self.num_envs, self.num_actions, self.device = num_envs, 1, device
self.max_episode_length, self.cfg = 200, {}
self.reward_scale, self.report_timeouts = reward_scale, report_timeouts
self.episode_length_buf = torch.zeros(num_envs, dtype=torch.long, device=device)
self.th = torch.zeros(num_envs, device=device)
self.thdot = torch.zeros(num_envs, device=device)
self._reset(torch.arange(num_envs, device=device))
def _reset(self, ids):
self.th[ids] = (torch.rand(len(ids), device=self.device) * 2 - 1) * math.pi
self.thdot[ids] = torch.rand(len(ids), device=self.device) * 2 - 1
self.episode_length_buf[ids] = 0
def get_observations(self):
obs = torch.stack([torch.cos(self.th), torch.sin(self.th), self.thdot], dim=-1)
return TensorDict({"policy": obs}, batch_size=[self.num_envs], device=self.device)
def step(self, actions):
u = actions.squeeze(-1).clamp(-2.0, 2.0)
ang = (self.th + math.pi) % (2 * math.pi) - math.pi # angle_normalize
cost = ang**2 + 0.1 * self.thdot**2 + 0.001 * u**2
self.thdot = (self.thdot + (3 * 10.0 / 2 * torch.sin(self.th) + 3.0 * u) * 0.05).clamp(-8.0, 8.0)
self.th = self.th + self.thdot * 0.05
self.episode_length_buf += 1
time_outs = self.episode_length_buf >= self.max_episode_length
self._reset(time_outs.nonzero().squeeze(-1)) # auto-reset: obs below is post-reset
extras = {"time_outs": time_outs.float()} if self.report_timeouts else {}
return self.get_observations(), -cost * self.reward_scale, time_outs.long(), extras
def train_cfg(args):
return {
"num_steps_per_env": args.steps, "save_interval": 10_000,
"obs_groups": {"actor": ["policy"], "critic": ["policy"]},
"actor": {"class_name": "MLPModel", "hidden_dims": [64, 64], "activation": "elu",
"obs_normalization": True,
"distribution_cfg": {"class_name": "GaussianDistribution", "init_std": 1.0}},
"critic": {"class_name": "MLPModel", "hidden_dims": [64, 64], "activation": "elu",
"obs_normalization": True},
"algorithm": {"class_name": "PPO", "num_learning_epochs": 5, "num_mini_batches": 4,
"clip_param": 0.2, "gamma": 0.99, "lam": 0.95, "learning_rate": 1e-3,
"schedule": args.schedule, "desired_kl": 0.01, "entropy_coef": 0.0,
"value_loss_coef": 1.0, "use_clipped_value_loss": bool(args.clip_value),
"max_grad_norm": 1.0},
}
@torch.inference_mode()
def evaluate(policy, device, episodes=256):
env = TorchPendulum(episodes, device) # unscaled reward
obs, ret = env.get_observations(), torch.zeros(episodes, device=device)
for _ in range(env.max_episode_length): # exactly one episode per env
obs, r, _, _ = env.step(policy(obs)) # deterministic: mean action
ret += r
return ret.mean().item()
if __name__ == "__main__":
p = argparse.ArgumentParser()
p.add_argument("--num_envs", type=int, default=512)
p.add_argument("--steps", type=int, default=32)
p.add_argument("--iters", type=int, default=150)
p.add_argument("--seed", type=int, default=0)
p.add_argument("--reward_scale", type=float, default=0.1)
p.add_argument("--no_timeouts", action="store_true") # ablation: treat time-outs as terminal
p.add_argument("--clip_value", type=int, default=1)
p.add_argument("--schedule", default="adaptive") # or "fixed"
p.add_argument("--device", default="cuda:0" if torch.cuda.is_available() else "cpu")
args = p.parse_args()
torch.manual_seed(args.seed)
env = TorchPendulum(args.num_envs, args.device, args.reward_scale, report_timeouts=not args.no_timeouts)
runner = OnPolicyRunner(env, train_cfg(args), log_dir=None, device=args.device)
t0 = time.perf_counter()
runner.learn(num_learning_iterations=args.iters, init_at_random_ep_len=True)
wall = time.perf_counter() - t0
score = evaluate(runner.get_inference_policy(args.device), args.device)
print(f"RESULT timeouts={not args.no_timeouts} clip_value={bool(args.clip_value)} "
f"reward_scale={args.reward_scale} schedule={args.schedule} iters={args.iters} seed={args.seed} eval_return={score:.1f} wall={wall:.1f}s")
pip install rsl-rl-lib==5.5.1
python rslrl_lab/pendulum_rsl.py --num_envs 8 --steps 8 --iters 1 --device cpu # unexecuted smoke recipe
The evaluation uses the unscaled Pendulum reward, so scores compare directly with Gymnasium's. A random policy scores around −1,200 per episode, and a policy that swings up and balances scores around −150. Things to notice:
init_at_random_ep_len=Truestaggers the environments' episode clocks, so time-outs spread evenly across iterations instead of all landing on the same step.- The rollout (32 steps) is much shorter than an episode (200 steps). The final-value bootstrap and the time-out bootstrap together make that safe.
- Pass a real
log_dirtoOnPolicyRunnerto get TensorBoard logs (Loss/surrogate,Loss/value,Loss/entropy,Loss/learning_rate,Policy/mean_std,Perf/*,Train/mean_reward). Plot the learning rate. You should see the adaptive rule at work.
Lab 6b-c — Ablations: which choices matter?¶
Run 3 seeds of each configuration at two budgets, --iters 30 (learning speed) and --iters 150 (final performance), and report the mean and spread of the evaluation return:
| Run | Flags | Tests |
|---|---|---|
| Baseline | (defaults) | — |
| No time-out bootstrapping | --no_timeouts |
Section 4.1: time-outs treated as terminal states |
| Unscaled reward, clipped value | --reward_scale 1.0 |
Section 6.3: the value clip in reward units |
| Unscaled reward, unclipped value | --reward_scale 1.0 --clip_value 0 |
Isolates the value clip from the reward scale itself |
| Fixed learning rate | --schedule fixed |
Section 6.1: what the KL rule buys |
The runs are independent, so run them in parallel (xargs -P with OMP_NUM_THREADS=1 on CPU). Results from one CPU test run of all five rows are in Section 12. Treat them as a reference point and measure your own.
Lab 6b-d — The same PPO in Isaac Lab¶
If you have an RTX GPU with Isaac Lab 3.0 installed (Isaac Lab Lecture 10), train a task whose agent config is the RSL-RL config from Section 7:
uv run isaaclab train --rl_library rsl_rl --task Isaac-Velocity-Flat-AnymalD --viz none --num_envs 1024
Open source/isaaclab_tasks/isaaclab_tasks/core/velocity/config/anymal_d/agents/rsl_rl_ppo_cfg.py and match each field to the PPO.__init__ arguments you read in Lab 6b-a. Then open the TensorBoard logs and check that the learning-rate curve shows the same adaptive pattern as in your Pendulum runs. On an 8 GB card, lower --num_envs until it fits and note how Perf/total_fps and the learning-rate curve change.
11. Use it in the real stack¶
- Isaac Lab 3.0 wraps RSL-RL in
isaaclab_rl.rsl_rl. A task's agent config is anRslRlOnPolicyRunnerCfgwithactorandcriticset toRslRlMLPModelCfg(orRslRlRNNModelCfg/RslRlCNNModelCfg). It carries the action distribution asdistribution_cfg=RslRlMLPModelCfg.GaussianDistributionCfg(init_std=...)and anRslRlPpoAlgorithmCfgwhose fields are thePPO.__init__arguments.RslRlVecEnvWrapperadapts the environment and appliesclip_actions. - You will see this in older code. Before RSL-RL 4.0, configs had a single
policy=RslRlPpoActorCriticCfg(init_noise_std=..., actor_hidden_dims=..., critic_hidden_dims=...)and a runner-levelempirical_normalizationflag. In 5.x the equivalents are separateactorandcriticconfigs,distribution_cfg.init_std, and per-modelobs_normalization. Isaac Lab 3.0 still accepts the old fields with deprecation warnings. - Other users: legged_gym (Isaac Gym, the original RSL-RL client, on an older API), mjlab (MuJoCo-Warp), and MuJoCo Playground. Your Pendulum
VecEnvfrom Lab 6b-b is the template for plugging in any other simulator, including ManiSkill.
12. Measure it¶
| Metric | Source | What it tells you |
|---|---|---|
Train/mean_reward, Train/mean_episode_length |
Logger, from completed episodes | Learning progress (noisy; compare with seeds) |
Loss/learning_rate |
Adaptive schedule | Floor means updates are too aggressive; ceiling means the policy has stopped moving |
Loss/value vs Loss/surrogate |
update() |
A value loss that stays high usually points to reward scale or missing time-outs |
Policy/mean_std |
Gaussian std | Exploration collapsing too fast or never shrinking |
Perf/collection_time vs Perf/learning_time |
Runner | Whether the simulator or PPO bounds wall-clock time |
Perf/total_fps |
\(NT / (t_c + t_l)\) | Throughput to compare across hardware and num_envs |
| Evaluation return, ≥ 3 seeds | Lab script | The number to report, with spread (Lecture 13) |
Previously reported results, not reproduced in the 2026-10-18 review. The original course records Lab 6b-c with 512 environments, 32 steps, 150 iterations, RSL-RL 5.5.1, PyTorch 2.14 CPU, and three seeds. Raw logs, an environment lock, and the executed script revision are not present alongside this table. Preserve it as an unverified historical report; do not use these numbers as a learner pass threshold or a current performance claim. A new run must save those missing artifacts.
| Configuration | Eval return after 30 iterations | Eval return after 150 iterations |
|---|---|---|
| Baseline | −296 ± 159 (−183, −478, −228) | −149 ± 3 (−147, −153, −148) |
| No time-out bootstrapping | −688 ± 81 (−743, −726, −595) | −149 ± 4 (−146, −153, −148) |
| Fixed learning rate | −760 ± 97 (−680, −733, −868) | −152 ± 4 (−155, −153, −147) |
| Unscaled reward, clipped value | −361 ± 260 (−204, −217, −662) | −152 ± 1 (−151, −152, −154) |
| Unscaled reward, unclipped value | −406 ± 227 (−158, −457, −602) | −148 ± 4 (−144, −152, −147) |
Mean ± sample std over seeds 0, 1, 2, with the individual seeds in parentheses. Evaluation is the deterministic policy over 256 episodes, in Gymnasium Pendulum units (random ≈ −1,200). Each 150-iteration run took about 1 minute with four runs sharing a 4-core CPU.
Conditional interpretation of the reported table, pending reproduction:
- Every configuration solves Pendulum by 150 iterations. On an easy task, none of these choices changes the final result. They change how fast you get there, which is what you pay for in GPU-hours on a hard task.
- Time-out bootstrapping and the adaptive learning rate both speed up early learning by a wide margin. Their 30-iteration returns are more than 2× worse than the baseline, and the seed ranges do not overlap. Without time-outs, the critic is taught that the world ends every 200 steps, and its values carry a time dependence it cannot observe. The fixed-rate gap is the adaptive rule at work. In baseline seed 0, it pushed the learning rate from 1e-3 to its 1e-2 ceiling within the first iteration and held it there for 27 iterations, 10× the fixed rate, before backing off to 2e-3 as the KL grew.
- Reward scale and value clipping showed no effect beyond seed noise on this task (Section 6.3).
- Three seeds is a small sample. The baseline's 30-iteration spread (−183 to −478) shows how much early learning varies. Use more seeds before claiming a difference smaller than these (Lecture 13).
13. Ship it¶
Commit rslrl_lab/ with:
RSLRL_MAP.md— the completed equation-to-code table, with one paragraph on every place where RSL-RL differs from Lectures 05-06pendulum_rsl.pyandrun_ablations.shablations.csv— one row per (configuration, seed): evaluation return, wall-clock, final learning rateablations.png— evaluation return per configuration with seed spread, plus the learning-rate curves of one baseline seedNOTES.md— half a page: which RSL-RL choice mattered most on Pendulum, why, and whether you expect the same ranking on a locomotion task
Exit criteria¶
You can move on when you can:
- name the four classes in the RSL-RL loop and the file each lives in
- point to the lines implementing GAE, the clipped surrogate, the clipped value loss, and the adaptive learning rate, and write each as an equation
- explain the time-out bootstrap, why it uses \(V(s_t)\), and why Isaac Lab only reports time-outs for infinite-horizon tasks
- configure an asymmetric actor-critic with
obs_groups, and say where its normalizers are updated and saved - read an RSL-RL training log and diagnose a stuck learning rate, a high value loss, or collapsing policy std
- show your ablation results with seed spread and explain the ranking
Self-check¶
- Your Isaac Lab locomotion run's
Loss/learning_ratedrops to \(10^{-5}\) within 50 iterations and stays there, while the reward stops improving. Which lines ofppo.pyare involved, what does this tell you about the per-minibatch KL, and name two things you would check first. - A teammate ports a task to RSL-RL and multiplies every reward by 100 "to make the signal stronger". Using Section 6.3, explain how this could slow the critic, which logged metrics would confirm or rule that out, and why the Pendulum runs in Section 12 showed no effect. Give two fixes that don't involve changing the reward back.
- A manipulation task ends after a fixed 5 s, and success is scored only at the last step. Should
time_outsbe reported to RSL-RL? What would bootstrapping do to the value targets at the end of each episode? - You add privileged friction and mass observations to the critic only. Write the
obs_groupsentry and the environment's observation dictionary. Then explain why the exported ONNX policy still runs on the real robot, and what would break if you had put the privileged group in the actor's set. - Lab 6b-b uses 32-step rollouts for 200-step episodes. Walk through what happens to the GAE recursion at step 31 of a rollout when the episode is still running, and at a step where a time-out occurs.
- Your 8 GB GPU forces you from 4,096 to 1,024 environments. Predict how
Perf/total_fps, the minibatch size, and the adaptive learning-rate trajectory change. Which RSL-RL setting would you adjust to keep the update statistics similar?
References¶
- Schwarke, Mittal, Rudin, Hoeller, Hutter, "RSL-RL: A Learning Library for Robotics Research," 2025 — arXiv:2509.10771
- RSL-RL source, tag v5.5.1 — GitHub · PyPI
rsl-rl-lib - Schulman et al., "Proximal Policy Optimization Algorithms," 2017 — arXiv:1707.06347
- Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation," 2015 — arXiv:1506.02438
- Rudin, Hoeller, Reist, Hutter, "Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning," 2021 — arXiv:2109.11978
- Pinto et al., "Asymmetric Actor Critic for Image-Based Robot Learning," 2017 — arXiv:1710.06542
- Andrychowicz et al., "What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study," 2020 — arXiv:2006.05990
- Huang et al., "The 37 Implementation Details of Proximal Policy Optimization" — ICLR blog track
- Isaac Lab 3.0, reinforcement learning concepts — docs