Skip to content

Module 4 — Advanced Perception and Prediction

MAPA

Deep Dive · Specialization

Specialized course identity for Module 4 — Advanced Perception and Prediction.

Artifact: specialization case study · Measure: performance, reliability, role fit

Parent: Phase 5 — Autonomous Driving

Time: 6–12 months

Prerequisites: Modules 1–3, Phase 3 (Sensor Fusion — Kalman filtering, multi-sensor math).


Why this is advanced

Modules 1–3 establish a geometric/replay baseline, source ownership in openpilot, and checked CPU inference. This module adds multi-sensor perception, BEV representations, trajectory prediction and maps. The named research systems are study examples; using one does not establish an automation level or make a system ready for deployment.


1. Advanced Sensor Hardware

  • LiDAR:

    • Working principles: pulsed time-of-flight versus FMCW ranging, separately from mechanical, MEMS, or other scanning mechanisms. A vendor name alone does not identify the ranging method.
    • Trade-offs: range, resolution, FoV, latency, cost.
    • Point cloud representations: raw points, voxels, range images, pillars.
  • Radar:

    • Automotive radar: 77 GHz FMCW, range-Doppler processing, angular resolution.
    • 4D imaging radar: elevation + azimuth + range + velocity.
    • Radar advantages: relative robustness in some adverse conditions and direct radial-velocity measurement; multipath, interference, occlusion, and weather still affect performance.
  • Camera systems:

    • Sensor types: CCD vs CMOS, global vs rolling shutter.
    • Optics: FoV, focal length, aperture, HDR techniques.
    • ISP pipelines for automotive (connection to Module 2 camerad).

2. Multi-Sensor Calibration

  • Intrinsic calibration:

    • Camera intrinsics (focal length, principal point, distortion) using checkerboard or ArUco patterns with OpenCV or Kalibr.
  • Extrinsic calibration:

    • Spatial transforms between sensors: camera-LiDAR, camera-radar, LiDAR-IMU.
    • Target-based (checkerboard, ArUco) and targetless methods (LI-Calib, ACSC).
  • Temporal calibration:

    • Aligning sensor timestamps across hardware clocks.
    • PPS signals, IEEE 1588 PTP, cross-correlation of observed events.
    • Impact of time misalignment on fusion accuracy.

Projects: * Use Kalibr for its documented camera/IMU calibration problems. For camera–LiDAR calibration, select a tool that explicitly supports those sensors and validate projection on held-out scenes; Kalibr is not a generic LiDAR calibrator. Kalibr capabilities. * Implement a radar-camera fusion pipeline: associate radar detections with camera bounding boxes for velocity-augmented detection.


3. BEV (Bird's-Eye View) Perception

  • BEV transformers:

    • BEVFormer: attention-based cross-view feature lifting from perspective to BEV.
    • BEVFusion: multi-modal (camera + LiDAR) fusion in BEV space.
    • Tesla's Occupancy Network: dense 3D voxel prediction from cameras.
  • 3D occupancy prediction:

    • Voxel-based occupancy (Occ3D, SurroundOcc) as alternative to explicit object detection.
    • Represent occupancy in a 3D grid, retaining unknown or unobserved space separately from observed free space. Occlusion is not evidence that a voxel is free.
  • Temporal fusion:

    • Incorporate temporal information using recurrent architectures or long-range temporal attention.
    • Track implicit object states across frames without explicit tracking.

Projects: * Train a BEVFusion-style model on nuScenes for camera+LiDAR 3D object detection. Compare with LiDAR-only baseline.


4. Trajectory Prediction

  • Multi-modal prediction:

    • TNT, MTR, Wayformer: predict multiple plausible future trajectories with probability scores.
    • Why multi-modal: a vehicle at an intersection might go straight, turn left, or turn right.
  • Interaction modeling:

    • Social force models, graph neural networks (GRIP, HYPER).
    • Transformer-based interaction encoders for agent-agent reasoning.
  • Map-conditioned prediction:

    • Incorporate lane geometry and traffic rules.
    • VectorNet, MapTR-style map encoding.
    • Constraining predictions to physically plausible, lane-following trajectories.

Projects: * Implement TNT or MTR on the Waymo Open Motion Dataset. Evaluate minADE/minFDE. Visualize multi-modal predictions.


5. HD Maps and Online Mapping

  • HD map components:

    • Layers: lane geometry, road markings, traffic signs, traffic lights, 3D landmarks.
    • Formats: OpenDRIVE, Lanelet2.
  • Online map building:

    • MapTR, BeMapNet: predict vectorized local map elements from sensor data; measure runtime and coverage on the chosen implementation rather than assuming a complete HD map or a deadline guarantee.
    • Reduces dependence on pre-built maps, supports uncharted areas.
  • Map-based localization:

    • Point cloud matching: NDT, ICP.
    • Camera-map matching for precise positioning within HD map.

Projects: * Deploy a MapTR-based online map system in CARLA. Compare online map quality against ground-truth HD map.


6. Sensor Simulation and Synthetic Data

  • Synthetic data generation:

    • CARLA and other sensor simulators can produce images and annotations; SUMO contributes traffic simulation in a coupled setup rather than photorealistic sensor rendering itself. Verify which physical sensor effects the selected simulator actually models.
    • Labels: bounding boxes, semantic masks, depth, radar detections.
  • Sensor model simulation:

    • LiDAR ray casting, radar multipath, camera noise models.
    • Reducing sim-to-real gap in training data.
  • Scenario generation:

    • Corner cases: fog, rain, night, occlusions, near-miss events.
    • Rare or unsafe scenarios that cannot be collected in real driving.

Projects: * Generate a 10,000-frame synthetic dataset in CARLA with varying weather, lighting, and traffic density. Train a 3D detector on synthetic vs. real data and compare.


Resources

Resource Why
nuScenes Multi-modal AV dataset with 3D annotations and HD maps
Waymo Open Dataset Large-scale dataset with motion prediction benchmarks
BEVFormer / BEVFusion papers Foundational BEV perception architectures
Kalibr Multi-sensor calibration toolkit
CARLA Sensor simulation and synthetic data

Next

→ Module 5 — Safety Standards and Deployment — ISO 26262, SOTIF, V2X, HIL testing, and production deployment.

Distinguish model failure from timestamp failure

Hold a camera–LiDAR model fixed and inject a known timestamp offset in recorded or simulated data. Evaluate detection quality against offset, then repeat with a perturbed extrinsic transform. Keep those experiments separate: retraining a model is not the first remedy for a broken frame/time contract.

For trajectory prediction, report horizon, sampling interval, number of predicted modes, miss rate, and probability calibration beside minADE/minFDE. Increasing the number of candidates can improve an oracle best-of-K metric without improving the trajectory a planner actually selects. Deliver the calibrated sensor contract, resource profile, and worst scenarios to Module 5. These become the SoC's sensor synchronization and deadline requirements.

Practice and assessment

Entry diagnostic (prerequisite check): Trace perception, prediction, planning, and control with frames and timestamps. If this is unfamiliar, use the prerequisite material above or the roadmap diagnostics before starting the lab.

  1. Baseline lab: Evaluate a small recorded or simulated driving scenario against a simple baseline.
  2. Guided variation: Perturb timing or sensor availability and inspect failure handling in replay/simulation.
  3. Independent transfer: repeat the method on a new input or configuration and explain where your original assumptions stop applying.

Evidence to submit: Task/trajectory error, latency, collision/constraint outcomes in simulation, and scenario coverage. Keep raw results and reproduction steps; distinguish measured results from estimates or work not run.

Self-check: Which observation would disprove your explanation of the result? Identify a competing explanation and the smallest experiment that separates them. A passing answer names a concrete predicted difference, not only another tool to try.

Exit criteria: reproduce the baseline, demonstrate the negative/variation check, and meet the shared completion rubric. Use the lesson cycle for retrieval practice and spaced review.