Lab: train and challenge a small robot policy on a CPU¶
Purpose: complete the train → select → evaluate → roll out cycle with code you can read. Prerequisites: math and evaluation. Artifact: a reproducible JSON report with split identities, numerical checks, baseline comparisons, and episode outcomes.
Access: Python 3.10 or later and a CPU. Standard library only; after installing Python, no additional package, dataset download, simulator, GPU, API key, or robot is needed. Effort: setup 5–15 minutes; guided experiment 45–60 minutes; analysis 30 minutes. The default calculation takes seconds or less on a typical laptop. Stop and inspect the environment if this tiny job takes longer than a minute.
1. Predict before running¶
The synthetic teacher uses acceleration = 0.8 × error - 0.6 × velocity. The dataset adds Gaussian action-label noise with standard deviation 0.02 m/s². Training estimates three parameters: two weights and a bias. The cart dynamics are defined in the task contract.
Write predictions for these three comparisons:
- Will the learned policy beat a constant zero action on held-out demonstration frames?
- Will its RMSE reach exactly zero even if the model family can represent the teacher?
- What happens if the policy always observes goal error
0.6 mlarger than the true error?
The third change is an observation fault. Labels and the physical goal remain unchanged.
2. Run the reference¶
Run from the repository root: the folder containing mkdocs.yml, rather than
this lesson's folder. If your terminal is still inside Robotics Foundations,
open another terminal at the repository root. These commands were tested
here on Python 3.12.3, Linux x86-64, on 2026-10-18. No third-party Python
dependencies are used.
python3 "Phase 3 - Artificial Intelligence/0. AI and Robot Learning Foundations/labs/policy_lab.py" --output /tmp/ai-policy-report.json
python3 -m unittest discover -s "Phase 3 - Artificial Intelligence/0. AI and Robot Learning Foundations/labs" -p "test_policy_lab.py" -v
The first command prints a JSON report: nested named fields, lists, and numbers.
It also saves the report to the requested path. /tmp is temporary storage;
copy reports you want to keep into your own experiment folder. Use a different
output path for each variation. The second command runs this lab's ten checks,
including independent hand calculations, numerical gradients, split leakage,
aggregation weighting, and changed-sensor behavior. The later timed-policy lab
adds its own integration checks.
Read the report in this order:
| Report field | What to look for first |
|---|---|
split_episode_ids |
Three non-overlapping lists of demonstration identities |
max_gradient_absolute_error, leakage_fault_caught |
Numerical agreement and a deliberately detected data fault |
candidates, selected_learning_rate |
Selection by validation error before test reporting |
test, test_zero_baseline, test_training_mean_baseline |
Held-out action RMSE compared with two simple prediction rules |
rollout, rollout_error_bias_0_6_m, rollout_zero_baseline |
Final task outcomes under paired initial states, not just action agreement |
The constant training-mean baseline uses only training labels to choose its output. The zero-action baseline is also useful for the motion experiment: zero acceleration preserves an existing velocity, so it does not mean “stop the cart.”
The local reference run produced these synthetic-fixture results:
| Check | Local result | Interpretation |
|---|---|---|
| Split | 24 / 6 / 6 episodes | Training / validation / test remain separate |
| Deliberate leakage | Caught | A copied training episode was rejected |
| Gradient maximum absolute error | 3.8e-12 |
Analytical derivatives match finite differences at the checked point |
| Learned test action RMSE | 0.02193 m/s² |
Close to the noisy synthetic teacher |
| Zero baseline action RMSE | 0.23942 m/s² |
Learning improves this prediction task |
| Biased-observation action RMSE | 0.47887 m/s² |
Wrong observation semantics overwhelm prediction quality |
| Clean / biased / zero-action rollouts | 12/12, 0/12, 0/12 |
Different behaviors under the stated toy dynamics |
These are not robot benchmarks or universal pass thresholds. Different seeds can change numerical values. Keep exact raw outputs rather than copying this table as your own measurement.
3. Read the code in data-flow order¶
Open policy_lab.py. Start with these functions:
| Function | Explain before moving on |
|---|---|
make_dataset |
How neighboring observations share a demonstration |
split_episodes / validate_splits |
Why no episode identifier can appear in two partitions |
fit_scaler / features |
Which rows determine normalization and where the bias feature comes from |
loss_and_gradient |
How each sample contributes to average loss and derivatives |
gradient_error |
Why both a positive and negative nudge are used |
train_model |
Why gradients are subtracted and why the epoch limit is bounded |
evaluate |
How pooled frame RMSE differs from mean episode RMSE |
rollout |
How the policy's action changes its next input |
run |
How validation selects a candidate before test evaluation |
The three candidate learning rates are 0.01, 0.05, and 0.2. Each starts from the same zero parameters. Candidate selection uses validation action RMSE. Only the selected candidate is reported on the test split.
The learned weights operate on normalized inputs. To compare them with the physical teacher coefficients, convert them:
physical_weight[j] = normalized_weight[j] / training_std[j]
physical_bias = normalized_bias - sum(physical_weight[j] × training_mean[j])
Expect physical weights near 0.8 and −0.6, not necessarily normalized weights with those values. This is a common reason correct models appear to have “wrong” parameters.
Use the saved model without training again¶
Training changes the weights. Inference applies the frozen weights to a new
observation. After the reference command has created /tmp/ai-policy-report.json,
run this self-contained block from any folder:
python3 - <<'PY'
import json
from pathlib import Path
report = json.loads(Path('/tmp/ai-policy-report.json').read_text())
weights = report['weights_normalized_inputs']
means = report['scaler_training_mean']
scales = report['scaler_training_std']
goal_m, position_m, velocity_m_s = 0.0, 0.5, 0.2
error_m = goal_m - position_m
features = [(error_m - means[0]) / scales[0],
(velocity_m_s - means[1]) / scales[1], 1.0]
raw_acceleration = sum(w * x for w, x in zip(weights, features))
limit = report['rollout_contract']['acceleration_limit_m_s2']
acceleration = max(-limit, min(limit, raw_acceleration))
print(f'error={error_m:.3f} m, action={acceleration:.6f} m/s^2')
PY
Expect an action close to the teacher's −0.52 m/s², with a small difference
because the parameters were fitted from noisy labels. This command performs
one prediction; it does not train or move anything. The trailing feature 1.0
supplies the learned bias term. Removing normalization or exchanging the two
features changes the model even when the weights are identical.
The next layer must apply that acceleration using the documented timestep and state update. The timed-loop lab adds capture age, queues and command expiry around the same learned prediction.
4. Make a bounded fault and catch it¶
Work in a copy of the script or use version control to review and restore your edit.
- Gradient fault: negate the derivative contribution in
loss_and_gradient. The finite-difference check should fail before training. Explain why this is more informative than waiting for a bad final score. - Data fault: the reference already tries to put one training sample into validation. Inspect the resulting overlap check. Removing an identifier from the report does not repair the actual data leak.
- Sensor fault: inspect
error_bias=0.6in the rollout. The policy settles near the wrong position because its observation says that the wrong position is the goal. A correct loss implementation cannot repair a wrong sensor contract.
Choose one fault, save evidence of its detection, restore the reference, and rerun the relevant check.
5. Run a guided variation¶
First predict what 20 training epochs will change. Then run:
python3 "Phase 3 - Artificial Intelligence/0. AI and Robot Learning Foundations/labs/policy_lab.py" --epochs 20 --output /tmp/ai-policy-20-epochs.json
Compare validation scores, selected learning rate, test RMSE, and paired rollout outcomes. The command limits epochs to 1–2,000. Do not increase training merely to chase a lower test score. An additional seed is a reproducibility variation, not a substitute for a large independent evaluation dataset.
6. Record what the result means¶
Your report contains Python/platform versions, script and generated-data SHA-256 values, seeds, split episode IDs, normalization, candidate selection, and per-episode results. Save the command and repository revision alongside it. Wall time measures the whole CPU lab; it is not a policy inference latency benchmark.
Answer in your own words:
- Why is the noisy teacher's achievable test RMSE not exactly zero?
- Why does the observation bias harm a model that passed the gradient check?
- What can this exercise establish about robot learning, and what still needs a richer simulator or physical robot?
- Which values must travel with the model to reproduce its predictions on another machine?
Answer guidance
1. Unpredictable label noise remains even when the deterministic teacher rule is learned. A finite test sample also fluctuates. 2. The gradient check verifies differentiation at sampled values. It does not establish correct observations, task semantics, distribution coverage, or closed-loop success. 3. It establishes the implemented train/evaluate procedure and outcomes for this mathematical fixture. It cannot establish camera perception, contact behavior, real actuator response, hardware latency, or physical safety. 4. Weights, feature order, units, sign conventions, normalization means/scales, action clipping, script revision, runtime version, and evaluation configuration. Dataset and split identities are needed to reproduce the training/evaluation result.Ship it¶
Submit the reference JSON, one variation JSON, numerical prediction, caught-fault evidence, and a one-page interpretation. Include at least one failure even if the clean reference succeeds. Then complete one independent transfer task and continue to Neural Networks.
Next simulator step: run this same saved policy in MuJoCo. Compare the original equations with actual CPU-engine dynamics, then change mass, force limits and physics timestep. Its package environment is optional; the training exercise above remains dependency-free.