Lecture 14: Capstone: RL Post-Training π0.5 on LIBERO¶
Overview¶
Picture a basketball player who never misses in their home gym. Every shot goes in, but they have practiced from exactly the same spots ten thousand times. Move the hoop half a meter, change the lighting, or put a chair where they usually stand, and their percentage collapses. They did not learn to shoot. They learned those shots. Coaching this player is not about more reps from the same spots. It is about practicing from new spots, scoring every attempt honestly, and not breaking the form they already have.
That is the state of π0.5 on LIBERO. openpi publishes a π0.5 checkpoint fine-tuned on LIBERO, and its LIBERO example reports success in the high 90s on the Spatial, Object, and Goal suites and the low 90s on LIBERO-10. The standard suites leave little headroom. Robustness studies tell a different story: LIBERO-PRO and LIBERO-Plus perturb objects, initial states, layouts, cameras, and instructions, and report that VLAs scoring above 90% on standard LIBERO degrade sharply under such changes. Their authors conclude that much of the standard score reflects memorized position-to-motion mappings. The headroom is in robustness, and this capstone targets it.
This kind of work is also scored in public settings — for example, Bittensor subnet SN80 scores improved π0.5 checkpoints in simulation with hidden tests that move objects. The protocol below is what you need to make a claim like that credible whether or not you enter one.
Project statement. Design and execute a pinned RL-post-training study for a supported pi05_libero checkpoint. Predefine a paired improvement target on held-out perturbed layouts and explicit noninferiority margins for nominal suites. Report confidence intervals for paired differences, regressions, costs, and inconclusive results. This is a hardware-dependent capstone plan: no pretrained π0.5 RL run is supplied or claimed by this checkout. Start with the executable CPU return checks and trained action-parity bridge.
The project follows an eight-step game plan, and every step uses an earlier lecture:
| Step | What you do | Lectures |
|---|---|---|
| 1. Understand the failure | Measure the nominal → perturbed drop and classify every failure | 02, 13 |
| 2. Better practice data | Privileged scripted teacher / DAgger, recovery examples, unusual layouts | 02, 03, 12 |
| 3. Start simple | Copy good tries more strongly; compare each layout to its own average | 04, 10, 11 |
| 4. Add a smart critic | A value function that sees exact object poses | 05 |
| 5. Pick a flow-policy RL method | Noise-injected PPO/GRPO, weighted regression, or critic-guided selection — then bake into the weights | 06, 10, 11 |
| 6. Protect general skills | KL leash, small steps, a little randomness, balanced suites | 06, 08, 12 |
| 7. Keep tasks short | Chunk length and replan horizon | 03, 13 |
| 8. Test like a scientist | Seeds, per-suite CIs, held-out perturbations, gated claims | 13 |
By the end you should be able to:
- reproduce a pinned π0.5 LIBERO protocol, explain deviations from reported results, and measure a held-out perturbed-layout gap
- build a failure taxonomy that tells you which training signal to add
- run at least two RL post-training methods on a flow-matching VLA and compare them at an equal rollout budget
- keep the policy close to its starting point and prove with measurements that general skills survived
- plan and account for the GPU-hours of the whole project, split into rollout, learner, and evaluation
- defend the final number to a skeptical reviewer
1. The system you are post-training¶
From the π0.5 paper and the openpi configs: a PaliGemma-class vision-language model (gemma_2b backbone with a SigLIP image encoder) plus a ~300M-parameter flow-matching action expert (gemma_300m). Together that is a ~3B-class model. Each call takes two 224×224 camera images (agent view and wrist), the proprioceptive state, and the instruction, and integrates a flow-matching ODE from Gaussian noise to an action chunk. openpi's sampler defaults to 10 Euler steps.
Three details of the LIBERO setup matter for everything below, and are easy to get wrong:
- Chunk and replan horizon. The base π0.5 config uses 50-step chunks, but
pi05_liberousesaction_horizon=10, and openpi'sexamples/libero/main.pyexecutes the firstreplan_steps=5actions before querying again. - Fixed initial states. The eval script runs
num_trials_per_task=50episodes per task, each starting from one of LIBERO's 50 stored initial states per task (task_suite.get_task_init_states), with a fixed environment seed (default7). These 500 episodes per suite are the test set. Never train on them. - Horizons. Maximum steps per episode are 220 (Spatial), 280 (Object), 300 (Goal), and 520 (LIBERO-10), after 10 no-op steps that let objects settle. At 5 actions per call, a LIBERO-10 episode costs up to ~104 VLA calls.
2. Hardware view: the compute plan¶
This is regime C. The simulator is a CPU process. The policy forward pass dominates rollouts, and a backward pass through a 3B model dominates the learner.
2.1 Memory budget (estimates — measure yours)¶
| Item | Full-parameter | LoRA |
|---|---|---|
| Policy weights (bf16, ~3.3B params) | ~6.6 GB trainable | ~6.6 GB frozen + adapters (MBs) |
| Optimizer state (AdamW, fp32 master + 2 moments) | ~40 GB | adapters only — small |
| Gradients | ~7-13 GB | adapters only |
| Reference policy for the KL leash | +~6.6 GB frozen copy | No duplicate base only if adapters-off exactly reconstructs the intended reference; otherwise preserve its frozen adapter/state |
| Critic | Small MLP on privileged state: negligible. A head on the VLM: shares the trunk, adds activations | same |
| Activations | Grow with batch × (prefix tokens + chunk × denoising steps) | same, minus frozen-weight grads |
| Rollout storage | Host RAM / disk: 2 uint8 images ≈ 0.3 MB per call, ~18 MB per 60-call episode | same |
openpi's README lists >22.5 GB for LoRA fine-tuning and >70 GB for full fine-tuning (supervised). RL adds the reference policy, rollout inference, and stored per-call data, so plan for headroom.
2.2 Two paths¶
| LoRA path | Full-parameter path | |
|---|---|---|
| Hardware | 1 × 80 GB (or 2: one rollout, one learner) | 4-8 × 80 GB |
| Layout | Colocated: rollout and learner share the GPU, time-sliced | Disaggregated: rollout GPUs serve batched inference; learner GPUs train with FSDP (openpi's TrainConfig has an fsdp_devices field) |
| Weight sync | Adapters only (MBs) | Full weights (~6.6 GB bf16) every policy update |
| Expressiveness | Limited by rank; less forgetting by construction | Highest; forgetting must be controlled explicitly |
| Notes | openpi ships a LoRA config for π0 (pi0_libero_low_mem_finetune) that shows the pattern (gemma_2b_lora / gemma_300m_lora variants plus a freeze filter); write the π0.5 equivalent. At the time of writing, openpi's PyTorch port lists LoRA as unsupported — use the JAX path or an RL framework |
Use an RL framework that already schedules rollout / inference / training placement |
Per iteration, with \(N\) parallel environments, \(T\) steps, \(H\) actions per call, and \(t_{\text{chunk}}(N)\) per batched call:
The usual lessons apply: batch across environments until rollout throughput flattens, keep CPU simulator workers ahead of the GPU, and measure the rollout/learner split before optimizing either.
2.3 The cost ledger¶
Keep COST_LEDGER.csv from day one, one row per run:
| run_id | phase | GPUs | rollout GPU-h | learner GPU-h | eval GPU-h | episodes collected | learner steps | outcome |
|---|---|---|---|---|---|---|---|---|
| p0-baseline | 0 | 1×H100 | — | — | measured | 2,000 | — | reproduced? |
| p3-awr-s0 | 3 | 1×H100 | measured | measured | measured | … | … | gate passed? |
Derive GPU-hours from timestamps logged by each process, not from estimates. The final report quotes the totals and the cost of the winning recipe alone, so another team can price a rerun.
3. Phase 0 — Baseline protocol¶
Goal: reproduce the published pi05_libero numbers and wire the evaluation into eval_harness/ from Lecture 13.
- Clone openpi with submodules (
git submodule update --init --recursive) and followexamples/libero/README.md. The Dockerized path is recommended. By default it serves thepi05_liberocheckpoint (gs://openpi-assets/checkpoints/pi05_libero) and runs the client:
Select a suite with CLIENT_ARGS="--args.task-suite-name libero_10", and serve your own checkpoints with SERVER_ARGS="--env LIBERO policy:checkpoint --policy.config pi05_libero --policy.dir <dir>". The non-Docker path runs scripts/serve_policy.py --env LIBERO in one terminal and examples/libero/main.py (Python 3.8 venv) in another.
2. Patch the client to log one row per episode. The stock script logs running totals and treats a simulator exception as a failed episode. Separate the two:
# Integration sketch: these lines belong at different places in eval_libero(), not one runnable script.
rows = [] # before the task loop
calls, env_steps, done, crashed = 0, 0, False, False # at the start of each episode
t_start = time.perf_counter()
# ... increment calls immediately before each client.infer request
# ... increment env_steps after each completed env.step (including settling no-ops)
# ... in the except branch: set crashed = True
rows.append(dict(policy=POLICY_TAG, suite=args.task_suite_name, task_id=task_id,
init_id=episode_idx, eval_seed=args.seed, success=bool(done) and not crashed,
crashed=crashed, env_steps=env_steps, policy_calls=calls,
wall_s=time.perf_counter() - t_start)) # after each episode
pd.DataFrame(rows).to_csv(out_csv, index=False) # after the task loop
Also include the episode index in video filenames. The stock names overwrite each other per task and outcome. 3. Run the selected suites with pinned checkpoints, adapters, preprocessing, initial states, and scorer. Compare raw outcomes and configuration against the published protocol. A published point estimate is not a known ground-truth probability, so requiring your Wilson interval to contain it is not a valid reproduction gate. Investigate discrepancies and report both sampling uncertainty and protocol differences before training. 4. Measure throughput: per-call latency at batch 1 and at your planned batch size, calls per episode, and episodes per GPU-hour. These go into the cost model.
3.1 The perturbed baseline¶
Define three layout sources and keep their roles separate:
| Source | Role | Notes |
|---|---|---|
| Your own perturbation generator | Training distribution for RL | Sample new object placements. Calling env.reset() without set_init_state gives a fresh placement from the task's BDDL regions — verify by rendering a few. Or perturb object poses in the stored states, letting them settle during the no-op steps |
| Held-out development draws from your generator (validation-range seeds) | Validation for checkpoint selection | Never used for gradient updates |
| LIBERO-PRO and/or LIBERO-Plus variants | Test: a perturbation generator you did not write | Guards against overfitting to your own generator |
Run the base checkpoint on all of them. Gate: the nominal-to-perturbed gap is large enough to measure. With the episode counts from your episodes_needed.md, the CIs of the nominal and perturbed rates must not overlap. If the gap is within noise, choose harder perturbations — no improvement you make will be detectable otherwise.
4. Phase 1 — Failure taxonomy (step 1)¶
Watch videos and use privileged simulator state to label every failed episode of the base policy on the validation layouts. Start from this template and extend it:
| Category | Definition | Auto-detect from sim state | Base nominal | Base perturbed | Final nominal | Final perturbed |
|---|---|---|---|---|---|---|
| Memorized location | Reaches to where the object usually is | Gripper's closest approach is near the nominal pose, far from the actual one | ||||
| Wrong object | Manipulates a distractor | First contact with a non-target object | ||||
| Missed grasp | Reaches the target, closes on air | Gripper closed near target, object never lifted | ||||
| Drop / slip | Lifted, then lost in transport | Object height rises then falls away from goal | ||||
| Near-miss placement | Released close to but outside the goal region | Final object-goal distance under a threshold | ||||
| Collision / knock-over | Disturbs the scene irrecoverably | Large non-target object displacement | ||||
| Stall / oscillation | No progress for many calls | Low end-effector displacement over a window | ||||
| Instruction ignored | Performs a different task from the same scene | Predicate of another task satisfied | ||||
| Timeout near success | Correct behavior, too slow | Success predicate within a few steps of being true | ||||
| Sim exception | Crash, not a policy failure | Logged crashed=True |
The taxonomy decides the training signal. "Memorized location" and "wrong object" call for layout diversity and RL on perturbed layouts. "Drop / slip" and "missed grasp" call for recovery data. "Instruction ignored" calls for multi-task balance and language checks. Report counts with CIs before and after.
5. Phase 2 — Better practice data (step 2)¶
The cheapest improvement is often supervised. Before any RL:
- Privileged scripted teacher. In simulation a teacher can read every object pose. Writing waypoint controllers (approach, grasp, lift, transport, place) for 40 tasks is real engineering. Start with the task families that fail most in your taxonomy. Use the teacher to (a) generate demonstrations on perturbed layouts and (b) label states the student visits — DAgger without a human (Lecture 02).
- Recovery data. Inject perturbations mid-episode (nudge the object, open the gripper, add DART-style action noise) and record the teacher's recovery.
- Self-generated successes. Run π0.5 itself on perturbed layouts and keep successful episodes. This needs no teacher, and it is the first step of Phase 3.
Fine-tune with openpi's standard flow-matching recipe on the base LIBERO data mixed with the new data, converted to LeRobot format the same way examples/libero/convert_libero_data_to_lerobot.py converts the original. Keep normalization statistics consistent with the checkpoint unless you deliberately recompute them and retrain. This SFT-augmented checkpoint is the baseline every RL method below must beat at equal GPU-hours.
6. Phase 3 — Start simple: group baselines and weighted copying (step 3)¶
For each training layout (a sampled seed), run a group of \(G\) rollouts (e.g., \(G = 8\)) from the identical initial state. Flow sampling starts from fresh Gaussian noise on every call, so rollouts differ. Verify that two calls with identical observations return different chunks; if they do not, control the sampler's noise explicitly. With binary success \(R_i \in \{0, 1\}\):
which is an inclusive per-initial-state group baseline from Lecture 04. Under independent rollouts it scales an unnormalized score-function gradient by (G−1)/G; leave-one-out baselines remove that particular bias. Here the advantages instead define weights for an explicitly different regression objective. All-success/all-failure groups have zero centered advantages, but still provide useful outcome/value evidence, and successful examples can support imitation. Log their fraction. Adaptive layout sampling changes the training distribution; retain the original evaluation distribution to assess the resulting policy.
# capstone/groups.py — advantages and the dataset for weighted flow-matching regression
import numpy as np
def group_advantages(success: np.ndarray) -> tuple[np.ndarray, float]:
"""success: (num_layouts, G) in {0,1}. Returns per-rollout advantages and the zero-signal fraction."""
success = np.asarray(success, dtype=np.float64)
if success.ndim != 2 or success.shape[0] == 0 or success.shape[1] < 2:
raise ValueError("nonempty layout-by-group array with G >= 2 required")
if not np.isfinite(success).all() or not np.isin(success, (0.0, 1.0)).all():
raise ValueError("binary finite episode outcomes required")
adv = success - success.mean(axis=1, keepdims=True)
zero_signal = float(np.mean(success.std(axis=1) == 0))
return adv, zero_signal
def regression_weights(adv: np.ndarray, beta: float = 0.5, mode: str = "awr") -> np.ndarray:
adv = np.asarray(adv, dtype=np.float64)
if adv.size == 0 or not np.isfinite(adv).all():
raise ValueError("nonempty finite advantages required")
if mode == "filtered": # positive-group-advantage filter; drops all-success groups too
return (adv > 0).astype(np.float32)
if mode != "awr" or not np.isfinite(beta) or beta <= 0:
raise ValueError("mode awr/filtered and positive finite AWR temperature required")
# Zero-signal groups (all success / all fail) have adv = 0 -> weight 1: plain self-distillation.
# Pass a mask (or drop those episodes) if you want only informative groups to move the policy.
# Cap the exponent before exp, so a small positive beta cannot overflow it.
with np.errstate(over="ignore", under="ignore"):
w = np.exp(np.minimum(adv / beta, np.log(20.0)))
return w.astype(np.float32)
The filtered branch keeps positive centered advantages, so it also drops successes from all-success groups. That is a deliberate group-relative filter. A success-only BC baseline should instead use the original success labels and retain all successes. Reject or skip a zero-total-weight training batch under an explicit update rule.
Train with the ordinary flow-matching loss, weighting each logged chunk by its episode's weight, mixed with a fixed fraction of the original LIBERO demonstrations. No log-probs are needed. This is advantage-weighted regression, not PPO or an unbiased policy-gradient estimator. RECAP (π*0.6) instead conditions a policy on an improvement indicator derived from learned values; sharing the use of advantages does not make the two algorithms identical.
Gate: use a prespecified paired development comparison and nominal noninferiority margins; do not infer improvement or non-regression from overlap of separate confidence intervals. Select the next iteration on development evidence, then reserve untouched test cases for the final claim.
7. Phase 4 — A critic that sees everything (step 4)¶
The group mean is a baseline for \(s_0\) only. A privileged critic \(V_\phi(s_t)\) trained on exact object poses, gripper state, and task ID (Lecture 05's asymmetric actor-critic) gives per-call advantages along the episode. It provides temporal credit estimates for many decisions in one rollout; a critic change alone does not establish that a particular chunk caused the outcome. With only one terminal 0/1 reward and γ=1, the exact policy value equals success probability. With discounting it weights success by its delay, and shaping rewards or function-approximation error further change that interpretation. Calibrate any progress-monitor use separately.
- Train \(V_\phi\) on Monte-Carlo returns from Phase 3 rollouts first, then TD targets. Track explained variance on held-out episodes.
- The critic sees state, never pixels, so it costs almost nothing. The actor still sees only cameras, instruction, and proprio. Nothing privileged may reach the policy at evaluation.
- Use GAE over policy calls (one decision = one chunk) with \(\lambda \approx 0.95\).
8. Phase 5 — Pick a flow-policy RL method, then bake it in (step 5)¶
π0.5 has no cheap exact action log-likelihood (Lecture 11). Choose one primary method and compare it with Phase 3 at equal rollout GPU-hours:
| Method | How it gets a training signal | Learner cost | Main risk |
|---|---|---|---|
| Noise-injected PPO / GRPO (πRL's Flow-Noise / Flow-SDE, ReinFlow) | Make each denoising step stochastic so it is Gaussian with a tractable log-prob, then use clipped-ratio updates with group or critic advantages | Store the denoising trajectory per call; backprop through \(K\) expert steps | Training sampler (noisy) differs from deployment sampler (ODE) — evaluate with the deployment sampler |
| Advantage-weighted / conditioned regression (Phase 3, AWR, RECAP-style) | Flow-matching loss on self-generated data, reweighted or conditioned by advantage | Same as SFT | Improves slowly; bounded by behaviors the policy already samples |
| Noise-space steering (DSRL) | Train a small policy that chooses the input noise of the frozen flow model, using a critic | Tiny actor + critic; base frozen | The result lives outside the weights until distilled |
| Critic-guided sample-and-rank | Sample \(N\) chunks, execute the one the critic scores highest, then distill the selected behavior | \(N\)× rollout inference | Exploiting critic errors (Lecture 08 reward hacking) |
Bake it into the weights. Steering and sample-and-rank improve behavior at inference time with extra machinery. If the deliverable is a checkpoint — as it is here, and in any competition that accepts only the model — collect the improved behavior on training layouts and distill it into π0.5 with the flow-matching loss. Then evaluate the distilled checkpoint alone.
For the full-parameter path, frameworks such as RLinf (π0 / π0.5 support, LIBERO plus its PRO and Plus variants, PPO / GRPO and other algorithms; its README links the πRL report as its π0 / π0.5 RL fine-tuning report) handle rollout/inference/training placement. Read their current configs rather than relying on remembered flags.
9. Phase 6 — Protect general skills (step 6)¶
RL on perturbed layouts can destroy what made π0.5 work: language following, nominal tasks, smooth motion.
- KL leash to the base policy. Flow policies have no closed-form KL between action distributions. With noise injection, each denoising step is Gaussian, so the per-step KL to the reference is closed-form. When both share the step variance \(\sigma^2\) it reduces to \(\mathrm{KL} = \lVert \mu_\theta - \mu_{\text{ref}} \rVert^2 / (2\sigma^2)\); with learned noise (as in ReinFlow) use the general diagonal-Gaussian formula. Otherwise, penalize the squared difference between current and reference velocity predictions on the same noisy inputs, as a proxy. On the LoRA path, adapters-off is the intended reference only when the frozen base and reference adapter state match exactly. A per-step Gaussian penalty on stored chains is a conditional replay-state regularizer, not automatically the action-marginal KL.
- Small steps. A low learning rate, clipped ratios, few epochs per batch, and gradient-norm clipping.
- A little randomness. Keep sampling stochastic during collection. At identical initial conditions, identical policy/environment randomness can remove between-rollout reward variation. Deterministic deployment policies can still be robust; measure robustness rather than equating it with sampling entropy.
- Balanced practice. Sample training layouts across all four suites with a floor per suite, and mix original demonstrations into every update. Easy suites must not crowd out hard ones, or the reverse.
- Regression tests every N iterations. A small fixed development set per suite, separate from the official final-evaluation initial states, plus an instruction-swap probe (same scene, a different valid instruction), since robustness studies report that VLAs often ignore language.
10. Phase 7 — Keep tasks short (step 7)¶
Chunking sets both the effective horizon (Lecture 13) and the rollout cost. Sweep replan_steps ∈ {2, 5, 10} on validation layouts and measure success, calls per episode, and failures by category. Shorter replan horizons react better to slips but cost proportionally more inference (5 → 2 is 2.5× the calls) and lengthen the credit-assignment chain. You may also cut denoising steps during rollouts to save compute. If you do, check that the training-time sampler still matches the deployment sampler on a parity set of observations (Lecture 03; the action-parity harness), and evaluate the final checkpoint with exactly the settings you ship.
11. Phase 8 — Test like a scientist (step 8)¶
The final evaluation is frozen before the last training run:
- Test sets: the four standard nominal suites (500 episodes each, fixed initial states, eval seed recorded), your held-out perturbed layouts, and at least one external robustness suite (LIBERO-PRO or LIBERO-Plus).
- ≥ 3 training seeds of the final recipe. Report per-seed results, mean, IQM, and stratified-bootstrap CIs from
eval_harness/. - Per-suite and per-perturbation-type success with Wilson CIs. A paired McNemar test against the base checkpoint on identical episodes.
- Checkpoints selected on validation layouts only; test sets touched once.
- Claim rule: use a design-appropriate interval for each prespecified paired difference. The perturbed improvement lower bound must exceed the chosen target; each nominal noninferiority lower bound must exceed its negative allowed regression margin. Account for multiple suite/variant claims. Otherwise report the measured effect and uncertainty as inconclusive or regressed, as applicable; nonsignificance and overlapping individual intervals do not prove non-regression.
12. Risk register¶
| Risk | Symptom | Detection | Mitigation |
|---|---|---|---|
| Reward hacking via sim bugs | Success rises; videos show objects tunneling, launched, or the predicate triggered by knocking things over | Review random successes every round, not only failures; physics sanity checks on object velocities | Fix the predicate or the scene; add termination on implausible states |
| Forgetting | Nominal suites or instruction probes decline | Regression tests every N iterations | KL leash, demo mixing, lower LR, LoRA |
| Eval leakage | Large gains on standard states only | Audit which seeds and init states each run sampled | Disjoint seed ranges; standard init states eval-only |
| Overfitting to your perturbation generator | Gains on your generator, none on LIBERO-PRO / LIBERO-Plus | Always report the external suite | Diversify the generator; select checkpoints on validation, not test |
| Selection bias | Best-of-many checkpoint looks great, does not replicate | Re-evaluate on fresh test seeds | Fix the selection rule in advance |
| Sampler mismatch | Training metrics improve, deployment-sampler eval does not | Eval with the deployment sampler each round | Distill; match NFE and noise settings |
| Zero-signal collapse | Most groups all-0 or all-1 | zero_signal fraction in logs |
Adaptive layout sampling, curriculum, larger \(G\) for hard layouts |
| Budget overrun | GPU-hours per gate far above plan | Ledger vs plan after each phase | Go/no-go gates; cut scope before cutting evaluation |
13. Milestones and go/no-go gates¶
| Milestone | Deliverable | Go criterion | If no-go |
|---|---|---|---|
| M0 Baseline | Per-episode CSVs, throughput numbers | Every nominal suite's CI contains the published number | Debug preprocessing / versions; do not train |
| M1 Headroom | Perturbed baseline + taxonomy | Nominal and perturbed CIs separated | Harder or different perturbations |
| M2 Data | SFT-augmented checkpoint | Perturbed validation improves; no nominal regression | Revisit teacher coverage / data mix |
| M3 Simple RL | Group-baseline weighted regression | Beats M2 at equal GPU-hours | Check zero-signal fraction, \(\beta\), mixing ratio |
| M4 Critic | Privileged critic, explained variance | Explained variance clearly above zero on held-out data; per-call advantages beat M3 | Use M3 with group baselines only |
| M5 Flow RL | One Phase 5 method, distilled into weights | Beats M3 at equal GPU-hours, or documented negative result | Ship M3 |
| M6 Final eval | Frozen protocol, ≥ 3 seeds | Claim rule satisfied | Report "no significant difference" honestly |
| M7 Ship | Repo, report, ledger | Another engineer reproduces within CI | Fix reproducibility gaps |
A documented negative result at M5 is a valid outcome. Shipping M3 with honest numbers beats shipping M5 without them.
14. Measure it¶
| Metric | Definition | Why it matters |
|---|---|---|
| Per-suite success, nominal | Wilson CI, 500 episodes per suite | Regression guard |
| Per-suite / per-perturbation success | Held-out and external suites, Wilson CI | The headline |
| Paired improvement vs base | McNemar on identical episodes | Whether the claim holds |
| Seed spread | IQM + stratified bootstrap over ≥ 3 seeds | Recipe, not luck |
| Zero-signal group fraction | All-0 or all-1 groups / total groups | Wasted rollout compute |
| KL / action drift to base | Per-step Gaussian KL or velocity-MSE proxy | Leash health |
| Rollout throughput | Episodes/GPU-h, calls/episode, latency at batch | Regime-C cost driver |
| GPU-hour split | Rollout / learner / eval from the ledger | Cost to reproduce |
| Failure taxonomy shift | Category counts before → after | What actually changed |
15. Ship it¶
Matching the course's "What you ship":
capstone/— data-generation scripts (teacher, perturbation generator, group collector), training configs for every phase, and aREPRODUCE.mdwith exact commands, commit hashes, and seeds.- The checkpoint — the final post-trained π0.5, loadable through openpi's
serve_policy.pywith your config, plus its normalization assets. EVAL_REPORT.md— per-suite nominal and perturbed results with CIs for base, SFT-augmented, and final checkpoints over ≥ 3 seeds; the paired test; the external robustness suite; and the seed-list hash.FAILURE_TAXONOMY.md— the Phase 1 table filled in before and after, with representative videos.COST_LEDGER.csv+COST_SUMMARY.md— GPU-hours split into rollout, learner, and evaluation, per phase and for the winning recipe alone.- Links to
rollout_bench/,eval_harness/, and the per-lecture implementations this capstone used.
Exit criteria¶
You are done when:
- the baseline reproduction passes its gate, and the perturbed gap is measured with CIs
- at least two post-training methods have been compared at equal rollout GPU-hours, with the comparison in the report
- the final checkpoint satisfies the claim rule — or the report says clearly that it does not, and why
- nominal regression tests and instruction probes show general skills survived
- another engineer, given the repo and the ledger, can reproduce your headline number within its CI and predict the GPU-hours it will take
Self-check¶
- After two rounds of RL, perturbed validation success is up 12 points, but LIBERO-Goal dropped 4 points and its CI no longer overlaps the base. Which Phase 6 mechanisms do you check first, and what does the failure taxonomy tell you that the aggregate does not?
- Your logs show 70% of groups are all-success or all-failure. What does this cost in GPU-hours per useful gradient, and name three changes (from Lectures 04, 12, and this lecture) that would reduce it.
- Success on training layouts jumps from 55% to 90% in one round. Videos of randomly chosen successes show the bowl occasionally clipping through the plate and registering as placed. What is happening, why did the failure review miss it, and what do you change in the protocol?
- You have one 80 GB GPU. Justify LoRA over full-parameter training using the memory table, and explain why the KL leash costs almost nothing on that path. What do you give up?
- Your checkpoint gains 15 points on your own perturbation generator and 1 point (CI includes zero) on LIBERO-Plus layout perturbations. What do you claim, and what would you change in training before trying again?
- DSRL steering gives your best validation numbers, but the deliverable must be a single checkpoint loaded by
serve_policy.py. Describe how you convert it, which metric you re-measure after conversion, and which risk-register entry applies.
References¶
- Black et al., "π0.5: a Vision-Language-Action Model with Open-World Generalization," 2025 — arXiv:2504.16054
- Black et al., "π0: A Vision-Language-Action Flow Model for General Robot Control," 2024 — arXiv:2410.24164
- openpi — repo, LIBERO example
- Liu et al., "LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning," 2023 — arXiv:2306.03310
- "LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization," 2025 — arXiv:2510.03827, code
- Fei et al., "LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models," 2025 — arXiv:2510.13626, code
- Chen et al., "πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models," 2025 — arXiv:2510.25889
- "ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning," 2025 — arXiv:2505.22094
- "Steering Your Diffusion Policy with Latent Space Reinforcement Learning" (DSRL), 2025 — arXiv:2506.15799
- Physical Intelligence, "π*0.6: a VLA That Learns From Experience" (RECAP), 2025 — arXiv:2511.14759, blog
- "RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models," 2025 — arXiv:2510.06710, RLinf repo
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," 2021 — arXiv:2106.09685
Next in this special course¶
- What comes after: VLA Optimization and Action-Parity Harness — make the checkpoint you just trained meet a real control-loop deadline without losing the success rate you earned
- Previous: Lecture 13 — Theory, Evaluation Rigor, and Sim-to-Real
- Back: Deep RL for Robot Learning — Overview