Phase 5 — Track D: Robotics¶
Track 5D · Robotics
Integrate perception, planning, ROS 2, real-time constraints, and embodied AI systems.
Contribution to the physical-AI SoC: robots and drones define the workloads and timing constraints the chip must serve. Use the VLA and Isaac courses to measure model quality, observation-to-action latency, memory, energy, and controller requirements. Carry that evidence into the custom accelerator, compiler, Linux, and wireless capstone.
New to robotics? Start with Robotics Foundations. Six lessons teach frames, kinematics, feedback, camera geometry, uncertainty, planning, and the ROS interface through worked examples and CPU-only labs. You do not need a GPU or simulator to learn these prerequisites. Use its ten-minute diagnostic to skip material you can already demonstrate.
Next, Robot Dynamics and Task Execution connects those basics to torque and current limits, LQR/MPC, swept collision checks, trajectory timing, and grasp/object-level success. Its four worked lessons and CPU labs explain why a reachable pose or accurate policy prediction can still fail during execution.
After the foundations, this phase has four application lecture tracks (each has its own folder and Lecture-01.md) and deeper robot-learning courses. Continue with Advanced Robot Operating System, then choose the application that matches your task.
| # | Track | Folder / lecture |
|---|---|---|
| 0 | Robotics Foundations (six beginner lessons; standard-library labs; geometry, feedback, estimation, planning, and ROS contracts) | Robotics Foundations/README.md |
| 0+ | Robot Dynamics and Task Execution (gravity/actuation, optimal control, timed collision-aware motion, and grasp outcomes) | Robot Dynamics and Task Execution/README.md |
| 1 | Advanced Robot Operating System (ROS / ROS 2, Nav2, SLAM, MoveIt 2) | Advanced Robot Operating System/Lecture-01.md |
| 2 | Industrial and Embedded Robotics (ROS-I, embedded, Docker, Gazebo / Isaac Sim) | Industrial and Embedded Robotics/Lecture-01.md |
| 3 | Advanced Perception and AI for Robotics (applied ROS 2 perception loop, deep learning perception, robot learning) | Advanced Perception and AI for Robotics/Lecture-01.md |
| 4 | Multi-Robot Systems and Swarm Robotics | Multi-Robot Systems and Swarm Robotics/Lecture-01.md |
| ★ | Special course — Isaac Sim and Isaac Lab (USD, PhysX/Newton, RTX sensors, ROS 2 bridge, Isaac Lab 3.0 training; named 6.1 teaching baseline plus 6.2 GA update and compatibility checks) | Isaac Sim and Isaac Lab/README.md |
| ★ | Special course — Deep RL for Robot Learning (imitation, policy gradients, PPO, SAC, model-based and offline RL, RL post-training of a flow-matching VLA; follows UC Berkeley CS 185/285) | Deep RL for Robot Learning/README.md |
| ★ | Professional course — VLA Foundations to Frontier 2026 (32 detailed lectures: history, architectures, data, training, RL, evaluation, Isaac/ROS deployment, October 13 frontier, CPU labs, two capstones) | VLA Foundations to Frontier 2026/README.md |
| ★ | Special course — Egocentric Data and VLA Fine-Tuning (egocentric datasets and licenses, Aria geometry and time, auditing, hand pose, retargeting to robot actions, latent actions, LeRobot/openpi/GR00T fine-tuning on 8–80 GB, paired evaluation; 13 lectures, CPU labs) | Egocentric Data and VLA Fine-Tuning/README.md |
| ★ | Special course — VLA Optimization and Action-Parity Harness (deploy and validate Vision-Language-Action policies on real robots) | VLA Optimization and Action-Parity Harness/README.md |
Each Lecture-01.md is a full lecture: learning objectives, worked conceptual sections, tables/diagrams where helpful, self-check questions, and links to the next/previous track.
Optional depth after the working loop: Lectures 3–4 include the expanded topics that were previously labeled “Phase 2 (Significantly Expanded)” in older versions of this file (deep perception, robot learning, multi-robot / swarm). All of that material now lives in Lecture-01.md for tracks 3 and 4.
Recommended courses (online)
Use these structured courses alongside each lecture’s Recommended courses section. Official ROS 2 and vendor docs stay the source of truth; courses help when you want guided projects, browser-based simulators (e.g. The Construct), or university-style theory. Full catalog (The Construct): Robotics & ROS courses.
-
§1 — ROS 2 core, Nav2, MoveIt (maps to Lecture 1)
- ROS 2 Basics in Python or ROS 2 Basics in C++ — pick one language first, then add the other.
- TF ROS 2 — frames and
robot_state_publisherpatterns. - Intermediate ROS 2 — launch, parameters, QoS, lifecycle.
- ROS 2 Navigation and Advanced ROS 2 Navigation — Nav2-style stacks.
- ROS 2 Manipulation Basics or ROS 2 Manipulation & Perception — MoveIt 2–style manipulation.
- Robotics Specialization (University of Pennsylvania) — aerial robotics, planning, perception, mobility, capstone; strong on math and control (often MATLAB in assignments; not ROS-version-specific).
-
§2 — Simulation, embedded Linux, Docker, industrial context (maps to Lecture 2)
- Introduction to Gazebo Sim with ROS 2 and Mastering Gazebo Simulator — Gazebo Sim + worlds/models.
- Docker for Robotics — containerized dev and deploy.
- Linux for Robotics — shell, permissions, workflows (ROS 1 in the title; skills transfer directly).
- C++ for Robotics / Python 3 for Robotics — language prep before ROS 2 nodes.
- Nvidia Isaac Sim documentation — tutorials; see ROS 2 tutorials when bridging to ROS 2 work.
-
§3 — Applied perception, tracking, control, drones (maps to Lecture 3, Part A)
- Kalman Filters — estimation intuition tied to mobile robots.
- ROS 2 Perception — sensors and perception stack in ROS 2.
- Behavior Trees for ROS 2 — ties to Nav2 / high-level behaviors.
- ROS 2 Control Framework —
ros2_controland hardware interfaces. - Programming Drones with ROS — complements PX4/MAVLink self-study from docs.
- State Estimation and Localization for Self-Driving Cars (Coursera) — Kalman, EKF, UKF, particle filters (transferable to tracking and fusion).
- ETH Zürich Autonomous Mobile Robots (edX) — kinematics, localization, mapping, planning (theory; pair with ROS 2 projects).
- ETH RSL Programming for Robotics (ROS) — official ROS 2 lecture materials (free outline/readings; on-campus course; use as a syllabus alongside your own labs).
Recommended project progression¶
Start with the CPU foundation experiments, then a frame- and timestamp-correct ROS 2 loop, a validated Isaac scene, and a tested Isaac Lab task. Train only after action/reset/reward checks pass. Evaluate held-out episodes before exporting a policy. Finish with the Isaac engineering labs and the language-guided robotics capstone; the language service and local controller have separate timing contracts.
Carry this course into the physical-AI system¶
Start from: One manipulation task and the required observation/action interfaces.
Build and challenge: Learn ROS 2 integration, establish an Isaac baseline, then study the matching VLA family. Add RL or multi-robot work only when it answers a demonstrated task limitation.
Pass forward: One working robot loop with explicit model/controller timing boundaries. Continue with the connected course and keep the artifact in the Physical AI Design Studio.