Lecture 16 — Task correctness, rewards, and partial resets¶
Prerequisites: Lectures 09–10 and 15. Time: 3 hours. Outcome: show that a vectorized environment implements the intended task before training.
Treat the task as a program with invariants¶
Training cannot repair an incorrect action mapping, reward sign, or reset. Write the MDP contract: observations, actions, transition timestep, reward terms, success condition, termination, timeout, and randomization.
Distinguish task termination from a time-limit truncation. The learning wrapper's bootstrapping behavior must match that distinction. Consult the manager-based environment tutorial and the actual wrapper in your pinned release.
Partial reset is the central vectorization test¶
For N environments, reset a strict subset such as {1, 3}. Check that only those environments' state, episode counters, command histories, recurrent state, and delayed-action buffers change. Setting every environment's last action to zero is a subtle cross-environment leak.
At reset boundaries, identify whether returned observations belong to the final state or the next episode. Do not pair a final reward with an initial observation and silently treat it as one uninterrupted trajectory. In the reviewed EA DirectRLEnv, compute_final_obs=True captures extras["final_obs"] before resetting completed rows. Consume it only under the current done mask: a retained extras entry is not a fresh terminal state for every row. Confirm how the chosen RL wrapper uses truncation and final observations.
The corrected reach task exposes previous action because its action-change reward depends on it, clamps targets to joint limits, clears only reset rows and requests a kinematics refresh. Its actual CPU tensor check tests those methods with a fake scene; the first real post-reset body/sensor pose remains a required simulator check.
Work a reward example¶
Suppose a task rewards progress toward a goal and penalizes joint velocity:
Specify whether terms are instantaneous rates integrated over policy dt or per-transition values. Changing decimation without checking this convention changes the learning problem. Success must be measured directly from task state; higher reward alone need not imply success.
Construct states where the robot is at goal, far away, violating a limit, and stationary. Predict reward and done flags before calling the implementation.
Lab¶
- Start at 4–16 environments with zero or scripted actions.
- Check finite observations, action bounds, dimensions, and named joint mapping.
- Unit-test reward components on known states.
- Reset two environments and verify unchanged environments remain unchanged.
- Force success, failure, and timeout independently.
- Test observation history and delay buffers immediately after reset.
- Repeat with a new seed and a second environment count.
Run the CPU reset-isolation fixture first, then implement the same invariant against real task tensors. The fixture does not test Isaac APIs.
Independent extension: inject an off-by-one done flag or fail to clear action history. Show the resulting test failure, fix it, and rerun the task smoke test.
Self-check¶
- Why can reward improve while the task is broken? The policy can exploit reward or reset bugs.
- Why test an untouched environment? A full-reset test cannot detect accidental broadcast resets.
- Is a timeout automatically a physical failure? It depends on the task definition; preserve that distinction in evaluation.
Exit artifact: a task contract and test table with pass/fail evidence for every invariant above. Start expensive training only after these checks pass.