Skip to content

Phase 5 — Track D: Robotics

ROS

Track 5D · Robotics

Integrate perception, planning, ROS 2, real-time constraints, and embodied AI systems.

Artifact: robot subsystem demo · Measure: loop rate, latency, safety, robustness

Contribution to the physical-AI SoC: robots and drones define the workloads and timing constraints the chip must serve. Use the VLA and Isaac courses to measure model quality, observation-to-action latency, memory, energy, and controller requirements. Carry that evidence into the custom accelerator, compiler, Linux, and wireless capstone.

New to robotics? Start with Robotics Foundations. Six lessons teach frames, kinematics, feedback, camera geometry, uncertainty, planning, and the ROS interface through worked examples and CPU-only labs. You do not need a GPU or simulator to learn these prerequisites. Use its ten-minute diagnostic to skip material you can already demonstrate.

Next, Robot Dynamics and Task Execution connects those basics to torque and current limits, LQR/MPC, swept collision checks, trajectory timing, and grasp/object-level success. Its four worked lessons and CPU labs explain why a reachable pose or accurate policy prediction can still fail during execution.

After the foundations, this phase has four application lecture tracks (each has its own folder and Lecture-01.md) and deeper robot-learning courses. Continue with Advanced Robot Operating System, then choose the application that matches your task.

# Track Folder / lecture
0 Robotics Foundations (six beginner lessons; standard-library labs; geometry, feedback, estimation, planning, and ROS contracts) Robotics Foundations/README.md
0+ Robot Dynamics and Task Execution (gravity/actuation, optimal control, timed collision-aware motion, and grasp outcomes) Robot Dynamics and Task Execution/README.md
1 Advanced Robot Operating System (ROS / ROS 2, Nav2, SLAM, MoveIt 2) Advanced Robot Operating System/Lecture-01.md
2 Industrial and Embedded Robotics (ROS-I, embedded, Docker, Gazebo / Isaac Sim) Industrial and Embedded Robotics/Lecture-01.md
3 Advanced Perception and AI for Robotics (applied ROS 2 perception loop, deep learning perception, robot learning) Advanced Perception and AI for Robotics/Lecture-01.md
4 Multi-Robot Systems and Swarm Robotics Multi-Robot Systems and Swarm Robotics/Lecture-01.md
★ Special course — Isaac Sim and Isaac Lab (USD, PhysX/Newton, RTX sensors, ROS 2 bridge, Isaac Lab 3.0 training; named 6.1 teaching baseline plus 6.2 GA update and compatibility checks) Isaac Sim and Isaac Lab/README.md
★ Special course — Deep RL for Robot Learning (imitation, policy gradients, PPO, SAC, model-based and offline RL, RL post-training of a flow-matching VLA; follows UC Berkeley CS 185/285) Deep RL for Robot Learning/README.md
★ Professional course — VLA Foundations to Frontier 2026 (32 detailed lectures: history, architectures, data, training, RL, evaluation, Isaac/ROS deployment, October 13 frontier, CPU labs, two capstones) VLA Foundations to Frontier 2026/README.md
★ Special course — Egocentric Data and VLA Fine-Tuning (egocentric datasets and licenses, Aria geometry and time, auditing, hand pose, retargeting to robot actions, latent actions, LeRobot/openpi/GR00T fine-tuning on 8–80 GB, paired evaluation; 13 lectures, CPU labs) Egocentric Data and VLA Fine-Tuning/README.md
★ Special course — VLA Optimization and Action-Parity Harness (deploy and validate Vision-Language-Action policies on real robots) VLA Optimization and Action-Parity Harness/README.md

Each Lecture-01.md is a full lecture: learning objectives, worked conceptual sections, tables/diagrams where helpful, self-check questions, and links to the next/previous track.

Optional depth after the working loop: Lectures 3–4 include the expanded topics that were previously labeled “Phase 2 (Significantly Expanded)” in older versions of this file (deep perception, robot learning, multi-robot / swarm). All of that material now lives in Lecture-01.md for tracks 3 and 4.


Recommended courses (online)

Use these structured courses alongside each lecture’s Recommended courses section. Official ROS 2 and vendor docs stay the source of truth; courses help when you want guided projects, browser-based simulators (e.g. The Construct), or university-style theory. Full catalog (The Construct): Robotics & ROS courses.

Start with the CPU foundation experiments, then a frame- and timestamp-correct ROS 2 loop, a validated Isaac scene, and a tested Isaac Lab task. Train only after action/reset/reward checks pass. Evaluate held-out episodes before exporting a policy. Finish with the Isaac engineering labs and the language-guided robotics capstone; the language service and local controller have separate timing contracts.

Carry this course into the physical-AI system

Start from: One manipulation task and the required observation/action interfaces.

Build and challenge: Learn ROS 2 integration, establish an Isaac baseline, then study the matching VLA family. Add RL or multi-robot work only when it answers a demonstrated task limitation.

Pass forward: One working robot loop with explicit model/controller timing boundaries. Continue with the connected course and keep the artifact in the Physical AI Design Studio.