Lecture 3: Advanced Perception and AI for Robotics¶
Before this survey: work through camera geometry and uncertainty and its CPU projection/filter lab. A detector's pixel output is not yet a robot-frame pose; calibration, depth, timestamp, and uncertainty assumptions connect the two. Artifact: a perception-to-estimation trace with known frame/unit checks, held-out quality metrics, and capture-to-use age.
Worked practice: the capture-time tracking lab derives a camera-to-world measurement, propagates uncertainty, runs Kalman prediction and gating, and exposes an identity ambiguity. Its optional TF2 experiment was executed with the actual Jazzy library; no detector checkpoint is required.
Overview¶
This lecture has three parts:
- Part A — Applied perception, tracking, and control (ROS 2): Closing the loop from pixels to actuation with standard messages, estimators, and middleware—what many applied research / perception engineer roles require.
- Part B — Advanced perception: Deep learning 2D/3D perception, semantics, VIO, and multi-modal sensing for manipulation and navigation.
- Part C — Robot learning: RL, imitation, foundation models, and whole-body / legged control—how learning stacks sit above or beside classical ROS 2 stacks.
By the end of this lecture you should be able to:
- Sketch a ROS 2 graph for detector → tracker → planner/controller, with
vision_msgsand TF2. - Explain Kalman prediction + association for multi-object tracking at a systems level.
- Name when learned depth fails and geometry (stereo, structure-from-motion) still matters.
- Describe sim-to-real levers (randomization, latency, calibration) for RL policies.
- Place LLM / VLM planners as high-level supervisors over ROS 2 primitives.
Part A — Applied perception, tracking, and control (ROS 2 focus)¶
This block maps detection → tracking → estimation → actuation to ROS 2 primitives: nodes, topics, QoS, launch, ros2 bag, and bridges to non-ROS services.
A.1 Detection in the ROS graph¶
Goal: Turn a camera or LiDAR stream into stable, typed messages for downstream nodes.
| Piece | Role |
|---|---|
sensor_msgs/Image |
Raw image; image_transport can select transports with different wire representations |
cv_bridge |
Convert ROS images ↔ OpenCV; inspect encoding, channel order, row stride, ownership, and copies |
vision_msgs/Detection2D / Detection3D |
Typed detections; retain the source frame and acquisition timestamp |
cv_bridge is not a promise of zero-copy conversion. message_filters can pair
messages according to their timestamps and policy; it does not synchronize
hardware clocks or repair a receive-time stamp mislabeled as capture time.
Inspect the relevant image bridge
and message filter implementation for
your release.
Deployment path: Train in PyTorch → export ONNX → TensorRT on Jetson → thin ROS 2 node that only runs inference and publishes.
A.2 Multi-object tracking (MOT)¶
Core loop: predict each track using a motion model, such as constant velocity inside a Kalman filter; gate unlikely measurements; solve an assignment problem; update matched tracks; and manage unmatched detections/tracks. The Kalman filter and the constant-velocity model are different layers of the same example.
ByteTrack is a tracking-by-detection method that associates high-confidence detections and then uses lower-confidence detections to recover additional matches; its association is not inherently a learned appearance model. Track identity can remain ambiguous when objects overlap or leave view. In ROS 2, publish track IDs and markers for RViz2 to expose these failures. ByteTrack paper
Roadmap deep dive: Phase 3 — Multi-Object Tracking guide (Kalman + assignment + ROS 2 patterns).
A.3 State estimation and robot_localization¶
Use the sensor/frame contracts in the fusion section
before combining wheel odometry, IMU, visual odometry, or geographically transformed
GPS. A wrong odom → base_link or an incorrect transform timestamp corrupts
both tracking and Nav2. Two outputs derived from the same image are not
automatically independent evidence.
A.4 Semantic layer and behavior trees¶
Nav2 already uses BehaviorTree.CPP. For semantic goals (“only drive on labeled floor”), feed segmentation into costmap plugins or BT conditions that branch on class labels.
A.5 Control and visual servoing¶
Visual servoing: Regulate image features (points, lines) to desired positions; control law outputs twist or joint velocities. Always transform setpoints through TF2 (geometry_msgs) so the controller operates in the correct frame.
Aerial / PX4: PX4's uXRCE-DDS client/agent bridge connects selected PX4 messages with DDS/ROS 2. This is not another name for the micro-ROS client stack. Pin compatible PX4/message definitions and check frame conventions: PX4's NED/FRD conventions and ROS's common ENU/FLU conventions need explicit conversion, including orientation. Begin with SITL and documented offboard timing/failsafe behavior before interpreting a published setpoint as an accepted or executed command.
A.6 Monocular geometry¶
Pure geometric reconstruction from monocular images has a scale ambiguity. Additional information such as inertial measurements under suitable motion and calibration, known dimensions, or learned priors can constrain scale; it does not make every monocular estimate metrically correct. For a VIO package, verify the actual sensor configuration, ROS interface/version, initialization requirements, output frames, and covariance before fusion.
A.7 Simulation and sim-to-real¶
- Gazebo +
ros_gz: ground robots, sensors. - PX4 SITL: aerial stacks before flight.
Sim-to-real: Inject latency, noise, and miscalibration in sim before claiming hardware readiness.
A.8 Service integration (FastAPI, NATS)¶
Pattern: keep the sensing/control path separate from dashboard HTTP and analytics events. A ROS 2 graph is not inherently real-time. Bound bridge queues, carry acquisition timestamps, and avoid blocking the executor callback with an HTTP request. Measure the resulting age and missed deadlines under load.
QoS: work through the executed compatibility and backlog lab for sensor streams versus commands.
A.9 GPU, Docker, GStreamer¶
NVIDIA Container Toolkit for GPU nodes in Docker. gscam or pipeline shims when v4l2 is not enough for camera ingest.
Recommended courses (Part A)¶
- Kalman Filters
- ROS 2 Perception
- Behavior Trees for ROS 2
- ROS 2 Control Framework
- Programming Drones with ROS
- State Estimation and Localization for Self-Driving Cars (Coursera)
- ETH Autonomous Mobile Robots (edX)
- ETH RSL Programming for Robotics (ROS)
Canonical links (Part A)¶
- ROS 2 Jazzy tutorial sources · QoS · distribution choices
- Nav2 · Behavior trees in Nav2
- ros2_control
vision_msgs· cv_bridge source- Barfoot, State Estimation for Robotics — resource page
- Gazebo · ros_gz
- PX4 ROS 2 · MAVSDK
- Docker + ROS 2 — choose an image for the same distribution and target architecture
- FastAPI · NATS
How Part A connects to other lectures¶
| Topic | Where |
|---|---|
Nav2, SLAM, robot_localization |
Lecture 1 — Advanced ROS, Lecture 2 — Industrial |
| Multi-robot | Lecture 4 — Multi-Robot |
Projects (Part A)¶
- Detector → tracker → RViz2:
vision_msgs+ track markers +ros2 bag. - PX4 SITL or Gazebo + Nav2: Compare command latency sim vs hardware.
- Bridge: ROS 2 state → FastAPI + NATS events; measure end-to-end delay.
Part B — Advanced perception and AI (expanded track)¶
B.1 Deep learning–based perception¶
- 2D detection: YOLO-family, RT-DETR, etc.—optimize for latency on Jetson (TensorRT).
- 6D pose: FoundationPose, DenseFusion-style methods for grasp—outputs must feed MoveIt 2 via TF and collision-aware planning.
- 3D point clouds: Open3D / PCL for classical geometry; PointNet++ / voxel nets for learned segmentation in structured scenes.
B.2 VIO / SLAM¶
VIO fuses IMU (high rate, biased) with camera (lower rate, rich). Failure modes: motion blur, rolling shutter, textureless regions. Always compare against wheel odometry or mocap when possible.
B.3 Semantics and scene graphs¶
Semantic segmentation (e.g. drivable vs obstacle) feeds Nav2 costmaps. 3D scene graphs attach objects and relations for task planning and HRI (“the cup on the left table”).
Open-vocabulary detectors (Grounding DINO, CLIP-based) reduce retraining but require latency and grounding validation on your robot.
B.4 Tactile and multi-modal sensing¶
Tactile arrays estimate slip and contact; fusion with vision helps in-hand manipulation. Audio can flag collision or motor anomalies—treat as asynchronous cues to supervisors, not hard real-time control unless validated.
Resources (Part B)¶
- Open3D
- Siciliano et al., Robotics: Modelling, Planning and Control
- Berkeley Robot Sensing (BRS) line of work (papers)
Projects (Part B)¶
- 6D pose + grasp: Jetson Orin + table-top objects + MoveIt 2.
- VIO benchmark: ORB-SLAM3 vs ground truth.
- Open-vocabulary pick-and-place: language → detection → grasp.
Part C — Robot learning and autonomous behaviors¶
Go deeper: this part is a survey. The full treatment — behavioral cloning and DAgger, flow-matching policies, policy gradients, PPO, SAC, model-based and offline RL, and RL post-training of a VLA — is the Deep RL for Robot Learning special course.
C.1 Reinforcement learning¶
Sim-to-real: Randomize dynamics, friction, sensor noise, latency; domain randomization reduces overfitting to one simulator build.
Algorithms: PPO and SAC are common for continuous control; TD-MPC and model-based variants sample-efficiently in some setups.
Frameworks: Stable-Baselines3, RLlib, Isaac Lab for GPU-heavy training.
C.2 Imitation and offline RL¶
Behavior cloning can fail out of distribution; DAgger collects expert labels on states visited by the learner to address this distribution shift. Diffusion Policy models multimodal action sequences; its denoising procedure does not impose hard smoothness, velocity, acceleration, or collision constraints. Validate the executed trajectory and object-level outcome separately. See the Diffusion Policy paper and the dynamics and task-execution course.
C.3 Foundation models for robotics¶
Vision-language-action (VLA) models aim to map images + language to actions. In deployment, LLMs often act as high-level planners that call ROS 2 skills (navigate, pick, place)—verify each step with executable checks.
C.4 Whole-body and legged control¶
WBC coordinates many DoF under constraints (contacts, COM). Legged systems blend model-based (MPC, WBC) with RL policies. Contact-rich manipulation uses hybrid force/position control.
Resources (Part C)¶
- Isaac Lab compatibility and source pins — full treatment in the Isaac Sim and Isaac Lab special course
- Sutton & Barto, Reinforcement Learning: An Introduction
- Lerobot (Hugging Face)
Projects (Part C)¶
- Sim-to-real locomotion: Isaac Sim → real quadruped (document transfer).
- Diffusion policy: Teleop demos → train → evaluate on arm.
- LLM task planner: LLM outputs skill sequence executed via ROS 2 actions/services.
Self-check (whole lecture)¶
Part A: (1) What does vision_msgs buy you vs a custom float array topic? (2) Name one reason to keep FastAPI out of the critical DDS callback path.
Part B: When does monocular depth fail outdoors at high speed?
Part C: What is one failure mode of behavior cloning without DAgger?
Answer guidance: Part A: typed detections provide explicit structure that tools and downstream consumers can inspect; they still need a shared semantic contract. Blocking an HTTP request inside the critical callback can delay sensor processing or control. Part B: blur, changing exposure, weak texture, and distribution shift can degrade depth; fast motion also makes an old estimate spatially wrong. Part C: small imitation errors can move the policy into states absent from its demonstrations, where errors accumulate; DAgger addresses coverage by gathering expert labels on policy-visited states, under its assumptions.