CPU lab: get the return target right before training¶
Compute a tiny actor-critic rollout by hand, compare it with a runnable reference, and expose the two boundary bugs that can invalidate a much larger training run. This lab accompanies Lecture 05 and Lecture 06. It needs only Python 3.10+ and finishes in seconds.
Tested here: Python 3.12.3, 2026-10-18; all 16 standard-library tests pass. The result is synthetic arithmetic, not a learned policy or a simulator benchmark. Budget 15 minutes for the worked calculation, 20 minutes for the variation, and 15 minutes for the explanation.
One rollout, two different endings¶
Use discount gamma=0.9 and GAE decay lambda=0.8. Rows 0–1 belong to one episode; row 1 is cut by an external time limit. Row 2 starts another episode and reaches a true terminal state.
| Row | Reward | V(current) | V(true post-action state) | Terminated | Truncated |
|---|---|---|---|---|---|
| 0 | 1 | 0.5 | 1 | false | false |
| 1 | 2 | 1 | 4 | false | true |
| 2 | 3 | 9 | 0 | true | false |
The bootstrap mask is zero only on true termination. The trace mask is zero at either termination or truncation, because the next stored row then belongs to another episode:
delta[t] = reward[t] + gamma * (not terminated[t]) * next_value[t] - value[t]
adv[t] = delta[t] + gamma * lambda * (not (terminated[t] or truncated[t])) * adv[t+1]
target[t] = adv[t] + value[t]
At the end of the collected buffer, set the future advantage carry to zero. This is a rollout cut, not necessarily an MDP terminal state: its last delta still uses the true next-state value if the episode continues.
Work backward:
- Row 2 terminates:
delta = 3 - 9 = -6,adv = -6, target3. - Row 1 truncates:
delta = 2 + 0.9*4 - 1 = 4.6. Stop the trace so row 2 cannot leak backward. Target is5.6. - Row 0 continues:
delta = 1 + 0.9*1 - 0.5 = 1.4;adv = 1.4 + 0.9*0.8*4.6 = 4.712; target is5.212.
From the repository root, enter this course and run:
cd "Phase 5 - Advanced Topics and Specialization/Track D - Robotics/Deep RL for Robot Learning"
python3 labs/returns.py
python3 -m unittest discover -s labs -p 'test_*.py' -v
The expected advantages are [4.712, 4.6, -6.0], and value targets [5.212, 5.6, 3.0].
Make two wrong implementations disagree¶
If row 1 is incorrectly treated as terminal, its target becomes 2, losing 3.6 of bootstrap value. If a same-step auto-reset adapter incorrectly supplies the new episode's value 9 rather than the true final-state value 4, the target becomes 2 + 0.9*9 = 10.1. These are different bugs with opposite effects in this fixture.
Use the environment's documented reset mode. In a same-step mode, preserve the terminal observation from the information returned by the environment. In a next-step mode, the reset-only event must not enter the rollout as an action-caused transition. Names of final-observation fields differ by library/version; inspect the adapter rather than guessing. Farama autoreset modes
A task with a true finite horizon needs that horizon represented in the state when it changes optimal decisions. An external training time limit on a continuing task is different. Label the task semantics first, then select masks. Gymnasium termination/truncation explanation
Check PPO's sign before optimizing¶
PPO maximizes min(ratio*A, clip(ratio,1-eps,1+eps)*A). It does not simply clip every ratio and use the result. For eps=0.2:
| Advantage | Ratio | Unclipped contribution | Clipped contribution | Chosen objective |
|---|---|---|---|---|
| +2 | 1.5 | 3.0 | 2.4 | 2.4 |
| −2 | 0.5 | −1.0 | −1.6 | −1.6 |
| −2 | 1.5 | −3.0 | −2.4 | −3.0 |
The last row retains a penalty for increasing the probability of a bad action. A loss minimized by gradient descent negates this objective. Clipping removes incentives for some excessive changes; it does not enforce a hard probability-ratio constraint.
Practice and answer guidance¶
Guided variation: change row 2's reward from 3 to 900. Rows 0–1 must not change, because row 1 ends the trace. If they do, your implementation leaks information across resets.
Independent task: build a complete three-step terminal episode with your own rewards and values. Set lambda=1, and compare every target against a direct discounted sum of rewards. Then cut the rollout before termination and supply an intentionally wrong bootstrap value: explain why the target now depends on that error. lambda=1 does not remove bootstrap bias when the collected trajectory stops before the true end.
Self-check: Should both terminated=True and truncated=True bootstrap? No: true termination takes precedence. Does a successful GAE test prove training will improve? No: policy gradients, reward semantics, data collection, and optimization still need independent checks. Can a negative explained variance identify this bug uniquely? No; it flags poor predictions but does not by itself distinguish masks, observations, optimization, or an almost-constant target distribution.
Artifact: save the input table, manual calculation, program output, one detected bug, and a new complete-episode comparison. Carry the verified two-mask implementation into the GPU lab, where you must also validate the actual final-observation adapter.
Preserve the objective when executing action chunks¶
A policy may choose two controls before observing again. With per-control discount gamma=0.9, rewards [1,2], and next-state value 10, the correct chunk target is:
1 + 0.9×2 + 0.9²×10 = 10.9.
If that chunk truly terminates the episode, the target is 2.8. If it is cut externally after only the first control, the target uses the actual one-step duration: 1 + 0.9×10 = 10. The new discounted_chunk_target function and four tests check these boundaries. Shortening the number of decisions does not let you keep the old reward/discount units unchanged; Lecture 13 explains the theoretical consequence.
Optional: check the exact lecture functions with real CPU frameworks¶
The optional tests extract named function definitions directly from Lectures 06, 11, 13, and 14. They do not create a second implementation of the lesson. Independent checks use scalar log-density/gradient calculations, torch.distributions.Normal, exact enumeration of four Bernoulli outcomes, a known Wilson endpoint, a deliberately correlated bootstrap fixture, and hand-computed group weights that distinguish positive-advantage filtering from success-only imitation. Small temperatures must not overflow exponentiation.
Fifteen tests passed on 2026-10-18 using Python 3.12.3, PyTorch 2.14.0+cpu, NumPy 2.3.5, and SciPy 1.18.1. The review reused an existing isolated environment with those versions; no simulator, pretrained weights, GPU execution, or complete RL trainer was involved. The validation record records source hashes and boundaries; upstream source evidence pins the inspected policy/framework code.
If you already have the shared course PyTorch CPU environment, reuse it and add the small SciPy requirements file. To create the documented Linux x86-64 / CPython3.12 environment from this course directory:
python3.12 -m venv /tmp/course-rl-cpu-venv
/tmp/course-rl-cpu-venv/bin/python -m pip install --require-hashes -r "../../../Phase 3 - Artificial Intelligence/2. Deep Learning Frameworks/PyTorch/CPU-Robot-Stop-Lab/labs/requirements-linux-cpu-py312.txt"
/tmp/course-rl-cpu-venv/bin/python -m pip install --require-hashes --no-deps -r labs/requirements-scipy-linux-py312.txt
/tmp/course-rl-cpu-venv/bin/python -m unittest discover -s labs/optional -v
The SciPy file is an addition to the complete shared CPU lock: its NumPy dependency is already installed by that lock. Other operating systems/Python versions need a corresponding resolved dependency set. Optional tests explicitly skip unavailable frameworks; skipped tests are not passed numerical validation.
Three results to explain:
- With two independent Bernoulli samples at success probability 0.3, the logit return gradient is 0.21. The inclusive group-mean estimator has expectation 0.105; leave-one-out recovers 0.21. Std normalization and PPO clipping introduce further changes.
- A zero-intermediate-noise flow sampler can still draw a random initial latent. Sampling with zero sigma is possible; the Gaussian transition log-density at zero sigma is undefined and rejected.
- Two training runs score
[0,1]and[1,0]on two tasks. Resampling whole shared runs preserves their mean 0.5; resampling task columns independently invents combinations and produces a different uncertainty interval. Choose the sampling unit from how the policies were trained.
Carry these numerical checks into a real integration only after validating episode IDs, final observations, masks, action units, sampler noise, and reference-policy state. A passing snippet test establishes its tested mathematics; it does not establish that PPO or VLA RL improves a robot task.