Kalibr — Multi-Sensor Calibration for Autonomous Vehicles¶
Deep Dive · AI Workloads
Specialized course identity for Kalibr — Multi-Sensor Calibration for Autonomous Vehicles.
Source: ethz-asl/kalibr — ETH Zurich Autonomous Systems Lab What it solves: Camera intrinsics, multi-camera extrinsics, camera-IMU spatial+temporal calibration, rolling-shutter calibration — all through batch continuous-time trajectory optimization. Why it matters: Every downstream system (BEVFusion, SLAM, openpilot, VIO) depends on correct calibration. Extrinsic and timing errors can bias downstream estimation; their effect depends on geometry, motion, and the estimator.
1. Why Calibration is Critical¶
What Calibration Errors Actually Cost¶
All multi-sensor fusion depends on two things being known precisely:
Intrinsic calibration: how does this camera map 3D rays to 2D pixels?
errors here: warped images, wrong depth estimates
Extrinsic calibration: what is the rigid transform between sensor A and sensor B?
errors here: LiDAR points project to wrong pixels,
IMU integration drifts, fusion degrades or diverges
Concrete Error Impact¶
Camera intrinsic error (1 pixel reprojection):
At 20m depth and focal length 500px: ~4cm lateral error per pixel
In BEVFusion: camera features misaligned with LiDAR → fusion hurts accuracy
Camera-LiDAR extrinsic error (1° rotation):
Point at 30m range: 30 × tan(1°) = 52cm lateral shift
BEVFusion camera BEV features miss LiDAR BEV features by ~52cm
Camera-IMU temporal error (1ms time offset):
At 1 m/s IMU excitation: 1mm position error per ms
This constant-velocity displacement is not a universal accumulated VIO error.
Test rotational motion and clock offset separately.
Camera-IMU spatial error (1cm translation):
Gyroscope correction for camera motion uses wrong lever arm
In VIO: incorrect rotational acceleration compensation can bias estimation
The Calibration Chain for an Autonomous Vehicle¶
Step 1: Calibrate each camera intrinsically (separate step)
Step 2: Calibrate multi-camera rig extrinsics (camera 0 ← → camera 1 ... N)
Step 3: Calibrate camera-IMU (spatial + temporal)
Step 4: Calibrate LiDAR-camera (target-based or targetless)
Step 5: Calibrate radar-camera (if applicable)
Step 6: Validate full system — project LiDAR onto camera image
Kalibr handles steps 1, 2, 3 with a single unified framework.
Steps 4–5 need separate tools (see Section 12).
2. Kalibr Architecture¶
Continuous-Time Trajectory Optimization¶
Unlike OpenCV's single-frame intrinsic calibration, Kalibr uses continuous-time batch optimization:
Traditional (OpenCV):
N static images → extract corners → minimize reprojection error per image
Works well for intrinsics, not suitable for camera-IMU (no IMU model)
Kalibr:
Video sequence → B-spline trajectory fitting → unified cost function
Cost = reprojection error + IMU pre-integration error + temporal alignment error
Why B-spline?
IMU measures continuous motion: ω(t), a(t)
B-spline provides continuous position p(t), orientation q(t)
Can compute p'(t), p''(t) analytically for any timestamp t
→ Compare B-spline acceleration with IMU accelerometer reading directly
What Kalibr Estimates Jointly¶
Camera intrinsics: [fu, fv, cu, cv, distortion params]
Camera extrinsics: T_cam1_cam0 (4×4 transform from cam0 to cam1 frame)
Camera-IMU: T_imu_cam (4×4 transform) + time_offset (seconds)
IMU intrinsics: scale, misalignment, bias (if --imu-models is specified)
3. Calibration Targets¶
A. AprilGrid (Strongly Recommended)¶
AprilGrid uses fiducial markers (each with a unique ID), enabling: - Partial visibility — the board doesn't need to be fully in frame - No pose ambiguity — unique ID resolves the 180° checkerboard ambiguity - Robust detection under motion blur
# aprilgrid_6x6.yaml
target_type: 'aprilgrid'
tagCols: 6 # tags across
tagRows: 6 # tags down
tagSize: 0.088 # tag outer edge length in meters — MEASURE AFTER PRINTING
tagSpacing: 0.3 # ratio: gap_between_tags / tagSize
# Generate the PDF target
kalibr_create_target_pdf \
--type apriltag \
--nx 6 \
--ny 6 \
--tsize 0.088 \
--tspace 0.3 \
--output aprilgrid_6x6.pdf
# Print at 100% scale (no scaling/fit to page)
# Glue to a flat, rigid board — aluminum composite or 10mm forex
# ALWAYS re-measure tagSize after printing with calipers
# Printer scale error of 2% → 2% error in metric scale → scale error in depth
B. Checkerboard (Simpler, Less Robust)¶
Standard chessboard. Cheap and easy to make but: - Requires full visibility in every frame - Has 180° pose ambiguity (mitigated by Kalibr's init, but less reliable) - Only works for global-shutter cameras reliably
# checkerboard_7x6.yaml
target_type: 'checkerboard'
targetCols: 7 # internal corner count, horizontal
targetRows: 6 # internal corner count, vertical
rowSpacingMeters: 0.03 # square size in meters
colSpacingMeters: 0.03
C. Circlegrid (Rarely Used)¶
Physical Target Construction¶
For AprilGrid:
Paper: Print → laminate → glue to 5mm aluminum composite
Foam: Print → glue to 10mm foam board (lighter, ok for indoor)
Size: At least 40×40cm for cameras at 0.5–2m distance
Critical measurements after printing:
1. Measure tagSize with digital calipers at 4+ locations → average
2. Measure tagSpacing ratio: (gap between tags) / tagSize
3. Update your YAML before calibration — wrong measurements = wrong scale
White border:
Leave at least one tag-width of white space around the entire grid
Without it, corner detection fails at board edges
4. Camera Models and Distortion Models¶
Kalibr specifies the model as projection-distortion. Example: pinhole-radtan.
Projection Models¶
| Model | CLI name | Parameters | Use Case |
|---|---|---|---|
| Pinhole | pinhole |
fu, fv, cu, cv | Standard cameras, FoV < ~120° |
| Omnidirectional | omni |
xi, fu, fv, cu, cv | Fisheye / catadioptric (FoV ~180°) |
| Double Sphere | ds |
xi, alpha, fu, fv, cu, cv | Wide-angle fisheye (FoV ~195°) |
| Extended Unified | eucm |
alpha, beta, fu, fv, cu, cv | Alternative fisheye model |
Distortion Models¶
| Model | CLI name | Parameters | Use Case |
|---|---|---|---|
| Radial-Tangential | radtan |
k1, k2, r1, r2 | Standard lens, FoV < 120° |
| Equidistant | equi |
k1, k2, k3, k4 | Fisheye lens (preferred for wide) |
| Field-of-View | fov |
w | Simple single-param fisheye model |
| None | none |
— | Ideal/simulated cameras |
Choosing Your Model¶
USB webcam (60–90° FoV): pinhole-radtan
DSLR / action cam (90–120°): pinhole-radtan or pinhole-equi
Wide-angle / GoPro (120–180°): pinhole-equi
Fisheye surround cam (>180°): omni-radtan or ds-none
Jetson CSI camera (IMX219): pinhole-radtan
Intel RealSense D435i:
RGB: pinhole-radtan
IR stereo: pinhole-radtan
5. Installation and environment boundary¶
The upstream installation guide documents ROS 1/catkin environments and Dockerfiles. Installing ROS 2 Humble packages is not a substitute for those dependencies. Use a pinned upstream revision and its matching build instructions; keep calibration tooling separate from the robot's ROS 2 runtime.
For this course, perform calibration on a workstation and deploy the resulting YAML plus provenance to the embedded target. Save the Git revision, container image digest, command, bag checksum, target dimensions, and output report. Verify the image architecture before attempting to build or run it on Jetson.
Inside the documented and sourced Kalibr environment, first run:
Then use the upstream sample data before a new sensor recording. For ROS 2 recordings, follow the upstream conversion workflow and compare timestamps, topic counts, encodings, and units before and after conversion. The examples below describe calibration arguments; launch them through the selected environment's supported entry point.
6. IMU Noise Model — Allan Variance¶
Before camera-IMU calibration, you need the IMU noise parameters: noise density and bias random walk for both gyroscope and accelerometer. These characterize how noisy your specific IMU is.
What These Parameters Mean¶
Gyroscope noise density (gyr_n): [rad/s/√Hz]
White noise on angular rate measurement
Higher → less trust in gyro in short-term
Gyroscope bias random walk (gyr_w): [rad/s²/√Hz]
How fast the gyro bias drifts over time
Higher → bias must be re-estimated more often
Accelerometer noise density (acc_n): [m/s²/√Hz]
White noise on acceleration measurement
Accelerometer bias random walk (acc_w): [m/s³/√Hz]
How fast accelerometer bias drifts
These directly scale the IMU terms in Kalibr's cost function.
Wrong values → poor calibration convergence or wrong result.
Allan deviation: fit the noise terms you actually model¶
# Step 1: Record IMU at rest for 2+ hours (the longer the better)
# Mount IMU on a vibration-isolated surface, do NOT touch it
# ROS2: record IMU topic
ros2 bag record -o imu_static_2h /imu/data
# Convert to ROS1 bag (allan_variance_ros uses ROS1)
# ori-drs/allan_variance_ros is a ROS 1/catkin tool; it is not a ROS 2 fork.
# Step 2: Install allan_variance_ros
cd ~/kalibr_ws/src
git clone https://github.com/ori-drs/allan_variance_ros.git
catkin build allan_variance_ros
# Step 3: Run Allan variance computation
rosrun allan_variance_ros allan_variance \
/path/to/folder_containing_bag \
/path/to/sensor_config.yaml
# Step 4: Plot Allan deviation to find noise floor and bias instability
rosrun allan_variance_ros analysis.py --data allan_variance.csv
Reading the Allan Deviation Plot¶
Allan Deviation σ(τ)
│
│ Angle Random Walk (ARW)
│ slope = -0.5
10⁻³│ \
│ \____________________ ← Bias Instability (minimum point)
│ ← read gyr_w here (slope = +0.5)
10⁻⁵│
└─────────────────────────────→ τ (seconds, log scale)
1 10 100 1000
gyr_n = σ at τ = 1s (read from slope -0.5 line)
gyr_w = σ at minimum × √(2*ln(2)/π) (conversion factor)
For accelerometer: same plot, different units (m/s² vs rad/s)
Example IMU YAML Files¶
# imu_mpu6050.yaml — cheap IMU (MPU6050 on Jetson GPIO)
rostopic: /imu/data
update_rate: 100 # Hz
accelerometer_noise_density: 0.0087 # [m/s^2/sqrt(Hz)]
accelerometer_random_walk: 0.00043 # [m/s^3/sqrt(Hz)]
gyroscope_noise_density: 0.0015 # [rad/s/sqrt(Hz)]
gyroscope_random_walk: 0.000019 # [rad/s^2/sqrt(Hz)]
# imu_realsense_d435i.yaml — Intel RealSense D435i built-in IMU
rostopic: /camera/imu
update_rate: 400 # Hz
accelerometer_noise_density: 0.0028 # factory spec
accelerometer_random_walk: 0.0003
gyroscope_noise_density: 0.00016
gyroscope_random_walk: 0.0000022
# imu_xsens_mti30.yaml — high-end tactical IMU
rostopic: /imu/data
update_rate: 400
accelerometer_noise_density: 0.002
accelerometer_random_walk: 0.00003
gyroscope_noise_density: 0.00003
gyroscope_random_walk: 0.000001
Datasheet values and their limits¶
If you cannot run 2-hour static test, use manufacturer datasheets:
# Convert datasheet specs to Kalibr units
# Datasheet: gyro noise = 0.005 °/s/√Hz
import math
gyro_noise_deg = 0.005 # °/s/√Hz from datasheet
gyr_n = gyro_noise_deg * math.pi / 180 # → rad/s/√Hz = 8.7e-5
# A bias-instability specification is NOT a bias random-walk density.
# Do not divide it by an arbitrary sqrt(time) to invent gyr_w.
# Fit a declared bias process from Allan-deviation data or obtain that parameter
# from the manufacturer's noise model, then validate on a held-out recording.
print(f"gyr_n: {gyr_n:.2e}")
7. Camera Intrinsic Calibration¶
Data Collection¶
# Method A: OpenCV (simplest for intrinsics alone)
# Record a ROS bag while moving the target in front of the fixed camera
# Recommended motion pattern:
# 1. Fill the frame at different distances (close, medium, far)
# 2. Tilt the target left/right/up/down ~30° from camera axis
# 3. Move target to all corners of the frame
# 4. Include frames where target is partially visible (AprilGrid only)
# Duration: 2–3 minutes at slow, deliberate motion
# Frame rate: ~4 Hz (Kalibr --bag-freq flag sub-samples to this)
# Record with ROS2
ros2 bag record -o camera_calib /camera/image_raw
# Convert ROS2 bag to ROS1 format (Kalibr requires ROS1 bag)
# Install ros2bag_to_rosbag converter:
pip3 install rosbags
rosbags-convert camera_calib/ --dst camera_calib.bag
Run Intrinsic Calibration¶
# Single camera, pinhole-radtan model
kalibr_calibrate_cameras \
--bag camera_calib.bag \
--topics /camera/image_raw \
--models pinhole-radtan \
--target aprilgrid_6x6.yaml \
--bag-freq 4.0 \
--show-extraction
# --show-extraction: opens a window showing detected corners — check it!
# --bag-freq 4.0: sub-sample bag to 4 Hz (reduces redundant frames)
# Two cameras (stereo rig)
kalibr_calibrate_cameras \
--bag stereo_calib.bag \
--topics /cam0/image_raw /cam1/image_raw \
--models pinhole-radtan pinhole-radtan \
--target aprilgrid_6x6.yaml \
--bag-freq 4.0
# Fisheye camera
kalibr_calibrate_cameras \
--bag fisheye_calib.bag \
--topics /camera/image_raw \
--models pinhole-equi \
--target aprilgrid_6x6.yaml \
--bag-freq 4.0
Output Files¶
results-cam-camera_calib.txt ← human-readable summary
report-cam-camera_calib.pdf ← plots: corners, reprojection errors
camchain-camera_calib.yaml ← machine-readable, input to next steps
Reading camchain.yaml (Intrinsics Only)¶
# camchain-camera_calib.yaml
cam0:
camera_model: pinhole
intrinsics: [461.629, 460.152, 362.680, 246.049]
# fu fv cu cv (pixels)
distortion_model: radtan
distortion_coeffs: [-0.27695, 0.06712, 0.00100, 0.00020]
# k1 k2 r1 r2
resolution: [752, 480]
# width height
T_cn_cnm1:
# Only present for cam1, cam2... (extrinsic to previous camera)
# cam0 is the reference (identity transform)
Reprojection Error Quality Criteria¶
Reprojection error = distance between detected corner and projected corner
Good: mean < 0.5 pixels, max < 1.5 pixels
OK: mean < 1.0 pixel
Poor: mean > 1.0 pixel → re-collect data or check target measurements
Common causes of high reprojection error:
1. Wrong tagSize in YAML (re-measure with calipers)
2. Motion blur (slow down, use faster shutter)
3. Non-flat target (re-mount on rigid board)
4. Too few poses / all poses at same distance
5. Defocus / poor lighting
8. Multi-Camera Extrinsic Calibration¶
Requirements for Multi-Camera Calibration¶
- All cameras must see the target simultaneously in enough frames
- Cameras do not need overlapping FoV IF you use intermediate cameras as bridges:
Data Collection for Multi-Camera¶
# Record all cameras synchronized
ros2 bag record -o multicam_calib \
/cam0/image_raw \
/cam1/image_raw \
/cam2/image_raw \
/cam3/image_raw # surround-view setup
# Motion pattern for multi-camera:
# Move the TARGET (not the camera rig) in the shared FoV region
# Ensure each pair of adjacent cameras sees the target together
# Duration: 3–5 minutes
# Convert to ROS1 bag
rosbags-convert multicam_calib/ --dst multicam_calib.bag
Run Multi-Camera Calibration¶
# 4-camera surround view (front, left, rear, right)
kalibr_calibrate_cameras \
--bag multicam_calib.bag \
--topics /cam0/image_raw \
/cam1/image_raw \
/cam2/image_raw \
/cam3/image_raw \
--models pinhole-equi \
pinhole-equi \
pinhole-equi \
pinhole-equi \
--target aprilgrid_6x6.yaml \
--bag-freq 4.0 \
--dont-show-report
# Output: camchain-multicam_calib.yaml with T_cn_cnm1 for each camera pair
camchain.yaml for Multi-Camera Rig¶
# camchain-multicam_calib.yaml
cam0:
camera_model: pinhole
intrinsics: [fu0, fv0, cu0, cv0]
distortion_model: equi
distortion_coeffs: [k1, k2, k3, k4]
resolution: [1280, 720]
T_cn_cnm1: # identity (cam0 is reference frame)
- [1, 0, 0, 0]
- [0, 1, 0, 0]
- [0, 0, 1, 0]
- [0, 0, 0, 1]
cam1: # mounted to the left of cam0
camera_model: pinhole
intrinsics: [fu1, fv1, cu1, cv1]
distortion_model: equi
distortion_coeffs: [k1, k2, k3, k4]
resolution: [1280, 720]
T_cn_cnm1: # T_cam1_cam0: transform from cam0 frame to cam1 frame
- [ 0.9998, -0.0098, 0.0181, 0.1205] # rotation + translation
- [ 0.0102, 0.9999, -0.0076, -0.0032]
- [-0.0180, 0.0078, 0.9998, -0.0015]
- [ 0.0000, 0.0000, 0.0000, 1.0000]
cam2: # T_cam2_cam1
...
cam3: # T_cam3_cam2
...
Converting T_cn_cnm1 to the Transform You Need¶
# calibration_utils.py
import numpy as np
def load_camchain(yaml_path):
import yaml
with open(yaml_path) as f:
chain = yaml.safe_load(f)
return chain
def get_T_cam_cam0(chain, cam_idx):
"""
Get T_camN_cam0: the transform that maps a point in cam0 frame
to camN frame. (= T_camN_camN-1 @ ... @ T_cam1_cam0)
"""
T = np.eye(4)
for i in range(1, cam_idx + 1):
T_step = np.array(chain[f'cam{i}']['T_cn_cnm1'])
T = T_step @ T
return T
# Example: get transform from cam0 to cam3
chain = load_camchain('camchain-multicam.yaml')
T_cam3_cam0 = get_T_cam_cam0(chain, 3)
# Transform a 3D point from cam0 to cam3 frame
point_cam0 = np.array([1.0, 0.5, 3.0, 1.0]) # homogeneous
point_cam3 = T_cam3_cam0 @ point_cam0
print(f"Point in cam3 frame: {point_cam3[:3]}")
9. Camera-IMU Calibration¶
This is the most important calibration for autonomous vehicles using Visual-Inertial Odometry (VIO), SLAM, or any system that combines camera and IMU data.
Kalibr estimates: - T_imu_cam — rigid body transform: position and orientation of camera in IMU frame - time_offset — time delay between camera exposure and IMU timestamp (can be ±10ms)
Data Collection — Critical Details¶
# Setup:
# 1. Mount camera + IMU rigidly together — no flex, no vibration
# 2. Fix the AprilGrid target to a WALL or rigid stand
# 3. Move the camera-IMU assembly in front of the fixed target
# Motion requirements (ALL must be excited):
# Translation: X, Y, Z axis independently
# Rotation: Roll, Pitch, Yaw independently
# Fast enough to excite IMU, slow enough to avoid blur
# Recommended motion script (2–3 minutes):
# Phase 1: slow rotations (roll 45°, pitch 45°, yaw 90°) — 30 seconds
# Phase 2: slow translations (±20cm in X, Y, Z) — 30 seconds
# Phase 3: figure-8 pattern combining rotation + translation — 60 seconds
# Phase 4: random vigorous movement — 30 seconds
# Camera rate: 20 Hz
# IMU rate: 200 Hz minimum (400 Hz preferred)
# Important: LOW MOTION BLUR
# Use short exposure time: < 1/500s
# Good lighting: LED panel or natural light
# Avoid fluorescent lights (50/60 Hz flicker)
# Record all topics
ros2 bag record -o cam_imu_calib \
/camera/image_raw \
/imu/data
# Convert to ROS1 bag
rosbags-convert cam_imu_calib/ --dst cam_imu_calib.bag
# Verify bag contents
rosbag info cam_imu_calib.bag
# Check: camera at ~20 Hz, IMU at ~200 Hz, no gaps in IMU stream
Verify IMU Frequency¶
# Check IMU topic rate in ROS1 bag
rosbag play cam_imu_calib.bag &
rostopic hz /imu/data
# Must be stable at your declared update_rate (200 or 400 Hz)
# Irregular IMU timestamps are a common cause of calibration failure
# Check timestamp alignment (rough — within a few seconds is fine)
python3 -c "
import rosbag
bag = rosbag.Bag('cam_imu_calib.bag')
cam_times = [t.to_sec() for _, _, t in bag.read_messages('/camera/image_raw')]
imu_times = [t.to_sec() for _, _, t in bag.read_messages('/imu/data')]
print(f'Camera: {len(cam_times)} frames, {cam_times[-1]-cam_times[0]:.1f}s')
print(f'IMU: {len(imu_times)} samples, {imu_times[-1]-imu_times[0]:.1f}s')
print(f'IMU rate: {len(imu_times)/(imu_times[-1]-imu_times[0]):.1f} Hz')
bag.close()
"
Run Camera-IMU Calibration¶
# Step 1: camera intrinsics must exist first
# Use camchain from Section 7 as input
# Run calibration
kalibr_calibrate_imu_camera \
--bag cam_imu_calib.bag \
--cam camchain-camera_calib.yaml \
--imu imu_mpu6050.yaml \
--target aprilgrid_6x6.yaml \
--bag-freq 20.0
# Temporal calibration is enabled by default in the checked upstream CLI.
# --show-extraction: visualize corner detection (add for debugging)
# --no-time-calibration disables it; first verify synchronization/exposure timestamps.
# For multiple cameras + IMU
kalibr_calibrate_imu_camera \
--bag cam_imu_calib.bag \
--cam camchain-multicam.yaml \
--imu imu.yaml \
--target aprilgrid_6x6.yaml \
--bag-freq 20.0
Camera-IMU Calibration Output¶
# camchain-imucam-cam_imu_calib.yaml
cam0:
camera_model: pinhole
intrinsics: [461.629, 460.152, 362.680, 246.049]
distortion_model: radtan
distortion_coeffs: [-0.277, 0.067, 0.001, 0.000]
resolution: [752, 480]
T_cam_imu: # ← THE KEY RESULT
- [ 0.01479, -0.99977, 0.01553, -0.00288] # rotation matrix | translation
- [ 0.99985, 0.01502, -0.00891, -0.07294] # | (meters)
- [ 0.00914, 0.01540, 0.99984, -0.00745]
- [ 0.00000, 0.00000, 0.00000, 1.00000]
timeshift_cam_imu: -0.0040 # seconds, defined by t_imu = t_cam + shift
# Example: camera time 10.000 corresponds to IMU time 9.996.
# Use this equation instead of ambiguous "leads/lags" wording.
Understanding T_cam_imu¶
T_cam_imu transforms a point expressed in IMU frame to camera frame:
p_cam = T_cam_imu × p_imu
To go the other direction (point in camera frame → IMU frame):
T_imu_cam = inv(T_cam_imu)
p_imu = T_imu_cam × p_cam
In practice, you need:
- VIO: inspect the specific configuration; packages differ in transform direction.
- BEVFusion: needs T_cam_lidar AND T_imu_lidar (from LiDAR-IMU calib)
- ROS2 TF: for parent imu and child camera optical frame, publish T_imu_cam.
T_cam_imu instead maps IMU coordinates into camera coordinates.
# Use the calibration result
import numpy as np
import yaml
def load_cam_imu_calib(yaml_path, cam_name='cam0'):
with open(yaml_path) as f:
data = yaml.safe_load(f)
cam = data[cam_name]
T_cam_imu = np.array(cam['T_cam_imu'])
timeshift = cam.get('timeshift_cam_imu', 0.0)
intrinsics = cam['intrinsics'] # [fu, fv, cu, cv]
dist_coeffs = cam['distortion_coeffs']
return T_cam_imu, timeshift, intrinsics, dist_coeffs
T_cam_imu, dt, K_params, dist = load_cam_imu_calib(
'camchain-imucam-cam_imu_calib.yaml'
)
T_imu_cam = np.linalg.inv(T_cam_imu)
print(f"Camera-IMU rotation:\n{T_cam_imu[:3,:3]}")
print("Camera-IMU translation (cm):", np.round(T_cam_imu[:3, 3] * 100, 2))
print(f"Time offset: {dt*1000:.2f} ms (camera relative to IMU)")
10. Rolling Shutter Calibration¶
Many cameras (USB webcams, phone cameras, some automotive cameras) use a rolling shutter — each row is exposed at a slightly different time. This causes the "jello effect" during motion.
Kalibr can estimate the readout time (time between first and last row exposure).
# Rolling shutter calibration requires a MOVING TARGET approach
# (opposite to global shutter: you move the camera, not the target)
# Actually: both methods work — just need sufficient motion
kalibr_calibrate_cameras \
--bag rolling_calib.bag \
--topics /camera/image_raw \
--models pinhole-radtan \
--target aprilgrid_6x6.yaml \
--approx-synced \
--bag-freq 4.0
# For rolling shutter, add:
# --camera-models rs-pinhole-radtan (rs = rolling shutter)
# (not all Kalibr versions support this — check your version)
11. Reading and Validating Results¶
Reprojection Error Analysis¶
# parse_kalibr_report.py
# Parse the results text file for quick quality check
import re
def parse_results(results_txt):
with open(results_txt) as f:
text = f.read()
# Extract reprojection errors per camera
errors = {}
for match in re.finditer(
r'cam(\d+).*?Reprojection error.*?mean:\s*([\d.]+).*?max:\s*([\d.]+)',
text, re.DOTALL
):
cam_id = int(match.group(1))
errors[f'cam{cam_id}'] = {
'mean': float(match.group(2)),
'max': float(match.group(3))
}
return errors
errors = parse_results('results-cam-calib.txt')
for cam, e in errors.items():
status = '✓' if e['mean'] < 0.5 else ('⚠' if e['mean'] < 1.0 else '✗')
print(f"{status} {cam}: mean={e['mean']:.3f}px max={e['max']:.3f}px")
Visual Validation: Project LiDAR onto Camera¶
# validate_cam_lidar.py
import numpy as np
import cv2
def project_lidar_to_camera(points_lidar, T_cam_lidar, K, dist_coeffs, img_shape):
"""
Project LiDAR points onto camera image using calibration results.
Visual check: LiDAR points should land on the correct objects in the image.
"""
N = points_lidar.shape[0]
pts_h = np.hstack([points_lidar[:, :3], np.ones((N, 1))]) # [N, 4]
pts_cam = (T_cam_lidar @ pts_h.T).T # [N, 4] cam frame
# Keep only points in front of camera
mask = pts_cam[:, 2] > 0.1
pts_cam = pts_cam[mask]
depths = pts_cam[:, 2]
# Project using camera intrinsics
fu, fv, cu, cv = K
u = (pts_cam[:, 0] / depths) * fu + cu
v = (pts_cam[:, 1] / depths) * fv + cv
# Keep only points within image bounds
H, W = img_shape[:2]
in_frame = (u >= 0) & (u < W) & (v >= 0) & (v < H)
u, v, depths = u[in_frame], v[in_frame], depths[in_frame]
return u.astype(int), v.astype(int), depths
# Load calibration
K_params = [461.629, 460.152, 362.680, 246.049] # fu, fv, cu, cv
T_cam_lidar = np.array([...]) # from your LiDAR-camera extrinsic calibration
# Load image and point cloud
img = cv2.imread('test_frame.png')
points = np.fromfile('test_scan.bin', dtype=np.float32).reshape(-1, 4)
u, v, depths = project_lidar_to_camera(points, T_cam_lidar, K_params, None, img.shape)
# Color points by depth (near=red, far=blue)
depth_norm = np.clip(depths / 50.0, 0, 1)
for i in range(len(u)):
color = (int(255*(1-depth_norm[i])), 0, int(255*depth_norm[i]))
cv2.circle(img, (u[i], v[i]), 2, color, -1)
cv2.imshow('LiDAR on Camera — validate calibration', img)
cv2.waitKey(0)
# GOOD: LiDAR points align with object edges in image
# BAD: LiDAR points float above/below objects → wrong extrinsics
Publish Calibration as ROS2 Static Transforms¶
# publish_calibration_tf.py
import rclpy
from rclpy.node import Node
from tf2_ros import StaticTransformBroadcaster
from geometry_msgs.msg import TransformStamped
import numpy as np
import yaml
from scipy.spatial.transform import Rotation
class CalibrationPublisher(Node):
def __init__(self, camchain_path):
super().__init__('calibration_publisher')
self.tf_broadcaster = StaticTransformBroadcaster(self)
with open(camchain_path) as f:
chain = yaml.safe_load(f)
transforms = []
# Publish camera-IMU transform
for cam_name, cam_data in chain.items():
if 'T_cam_imu' not in cam_data:
continue
T = np.array(cam_data['T_cam_imu'])
T_imu_cam = np.linalg.inv(T)
ts = TransformStamped()
ts.header.stamp = self.get_clock().now().to_msg()
ts.header.frame_id = 'imu'
ts.child_frame_id = cam_name
# Translation
ts.transform.translation.x = T_imu_cam[0, 3]
ts.transform.translation.y = T_imu_cam[1, 3]
ts.transform.translation.z = T_imu_cam[2, 3]
# Rotation → quaternion
R = Rotation.from_matrix(T_imu_cam[:3, :3])
q = R.as_quat() # [x, y, z, w]
ts.transform.rotation.x = q[0]
ts.transform.rotation.y = q[1]
ts.transform.rotation.z = q[2]
ts.transform.rotation.w = q[3]
transforms.append(ts)
self.tf_broadcaster.sendTransform(transforms)
self.get_logger().info(f'Published {len(transforms)} static calibration transforms')
def main():
rclpy.init()
node = CalibrationPublisher('camchain-imucam-calib.yaml')
rclpy.spin(node)
rclpy.shutdown()
12. Using Calibration in Downstream Systems¶
BEVFusion (Camera-LiDAR Fusion)¶
BEVFusion needs camera intrinsics and the camera-to-LiDAR extrinsic. LiDAR-camera extrinsic is NOT from Kalibr directly (Kalibr doesn't calibrate LiDAR), but you combine Kalibr camera-IMU + LiDAR-IMU results:
# T_cam_lidar = T_cam_imu @ T_imu_lidar
T_cam_imu = load_kalibr_cam_imu('camchain-imucam.yaml')
T_lidar_imu = load_lidar_imu_calib('lidar_imu_calib.yaml') # from LI-Calib or similar
T_imu_lidar = np.linalg.inv(T_lidar_imu)
T_cam_lidar = T_cam_imu @ T_imu_lidar
The following is a schematic calibration manifest, not a drop-in config for every BEVFusion implementation. Map it into the chosen loader and verify the transform direction.
data:
cameras:
CAM_FRONT:
camera_model: pinhole
intrinsics:
fx: 461.629
fy: 460.152
cx: 362.680
cy: 246.049
distortion: [-0.277, 0.067, 0.001, 0.000]
extrinsic: # T_cam_lidar as 4×4
- [R00, R01, R02, tx]
- [R10, R11, R12, ty]
- [R20, R21, R22, tz]
- [0, 0, 0, 1 ]
OpenVINS / VINS-Mono (Visual-Inertial Odometry)¶
Kalibr output maps directly to VIO config:
# openvins config.yaml
camera_intrinsics:
- [461.629, 460.152, 362.680, 246.049] # fu, fv, cu, cv
camera_distortion_coeffs:
- [-0.277, 0.067, 0.001, 0.000]
T_imu_cam0: # use T_imu_cam (inverse of Kalibr's T_cam_imu)
rows: 4
cols: 4
data: [R00, R01, R02, tx,
R10, R11, R12, ty,
R20, R21, R22, tz,
0, 0, 0, 1]
timeshift_cam_imu: -0.004 # copy from Kalibr output
ORB-SLAM3 (Monocular/Stereo/IMU)¶
# ORB_SLAM3/config/EuRoC.yaml
Camera.type: "PinHole"
Camera.fx: 461.629
Camera.fy: 460.152
Camera.cx: 362.680
Camera.cy: 246.049
Camera.k1: -0.277
Camera.k2: 0.067
Camera.p1: 0.001
Camera.p2: 0.000
# IMU/body-from-camera extrinsic: T_bc = inverse(T_cam_imu) when b is the IMU frame
Tbc: !!opencv-matrix
rows: 4
cols: 4
dt: f
data: [R00, R01, R02, tx, R10, R11, R12, ty, R20, R21, R22, tz, 0, 0, 0, 1]
IMU.NoiseGyro: 0.00016 # gyr_n from imu.yaml
IMU.NoiseAcc: 0.0028 # acc_n
IMU.GyroWalk: 0.0000022 # gyr_w
IMU.AccWalk: 0.0003 # acc_w
IMU.Frequency: 400
Openpilot / comma.ai¶
Openpilot's calibration and model preprocessing are application-specific. CalibrationParams is not a generic JSON store for camera intrinsics, and a Kalibr YAML cannot be installed by inventing parameter keys. A custom camera requires the selected openpilot revision's sensor configuration, intrinsic/warp conventions, calibration serialization, and model input contract. Inspect the official source at a pinned revision and validate recorded-image preprocessing against that revision before attempting a hardware integration. This course does not claim a custom-camera integration was executed.
13. Common Pitfalls and Debugging¶
Pitfall 1: Wrong tagSize in YAML¶
Symptom: Calibration succeeds but depth estimates are systematically off
LiDAR points project onto wrong locations in image
Cause: tagSize in YAML does not match the PRINTED target size
Fix: Re-measure with digital calipers AFTER printing
Printers scale to fit page → actual size ≠ intended size
Pitfall 2: Blurry Images¶
Symptom: "Could not find enough corners" or high reprojection error
Cause: Motion blur, out-of-focus, or dark corners
Fix: - Use shorter exposure (< 1/500s)
- Add more lighting (LED panel)
- Move target more slowly
- Check focus: target must be sharp at the calibration distances
Pitfall 3: IMU Timestamp Jitter¶
Symptom: Camera-IMU calibration fails to converge
"IMU integration error is very large"
Cause: Irregular IMU timestamps (USB latency, poor driver)
Fix: - Check with: rostopic hz /imu/data (should be stable ±1%)
- Use hardware-timestamped IMU (SPI/I2C directly, not USB)
- Temporal calibration is enabled by default; inspect --no-time-calibration and --timeoffset-padding in your pinned CLI
- If jitter > 1ms: fix the IMU driver or hardware first
Pitfall 4: Insufficient Motion for Camera-IMU¶
Symptom: Large uncertainty on T_imu_cam rotation
"Warning: observability is poor"
Cause: Motion did not excite all 6 DOF
Fix: - Record a new bag with deliberate roll/pitch/yaw rotations
- Each axis must be rotated at least 45°
- Add fast, sharp jerks (excites accelerometer)
- Motion must be fast enough to generate non-trivial IMU signal
Pitfall 5: Non-Overlapping Camera FoVs¶
Symptom: "No common observations between cam0 and cam2"
Cause: Cameras don't share enough target visibility
Fix: - Use intermediate cameras as bridges
- Record while overlapping FoV region of adjacent camera pairs
- Reduce --bag-freq to include more diverse frames
Pitfall 6: Inconsistent Results Between Runs¶
Symptom: T_cam_imu differs by >5° between two calibration runs
Cause: Insufficient data, or target/sensor moved between recording
Fix: - Collect longer bags (3+ minutes for camera-IMU)
- Do not change camera focus or IMU mounting between sessions
- Use --show-extraction to verify corners are detected consistently
- Average results of 3 calibration runs (take median quaternion)
Debugging Checklist¶
Before recording:
□ tagSize measured with calipers and updated in YAML
□ Target mounted flat on rigid surface
□ Camera in focus at the working distance
□ IMU frequency verified (rostopic hz)
□ Good lighting, no motion blur at planned speeds
During recording:
□ All cameras seeing target in every frame (for extrinsic)
□ IMU and camera recording simultaneously
□ Sufficient motion diversity (all 6 DOF for camera-IMU)
□ No gaps or drops in IMU stream
After calibration:
□ Reprojection error < 0.5px (intrinsic)
□ Report PDF reviewed — corner detections look correct
□ T_cam_imu makes physical sense (matches physical measurement roughly)
□ Timeshift within expected range (|dt| < 50ms for software-triggered systems)
□ Validate by projecting LiDAR on camera image
14. Projects¶
Project 1: Full Calibration Chain¶
Starting from a Jetson Orin Nano with a CSI camera (IMX219) and an MPU6050 IMU:
1. Record 2-hour IMU static bag → compute Allan variance → populate imu.yaml
2. Print and mount a 6×6 AprilGrid on foam board, measure tagSize
3. Record camera intrinsic bag → run kalibr_calibrate_cameras
4. Record camera-IMU bag → run kalibr_calibrate_imu_camera
5. Validate: reprojection error < 0.5px, time shift stable across held-out motions and inside the configured optimization range
Project 2: Stereo Camera Calibration¶
Calibrate a stereo pair (two CSI cameras or USB webcams). Compare results with OpenCV's stereoCalibrate. Measure the baseline (distance between cameras) with calipers and verify Kalibr's estimated translation matches within 2mm.
Project 3: RealSense D435i Calibration¶
The D435i has a built-in IMU, RGB camera, and IR stereo cameras. Calibrate the RGB-IMU pair with Kalibr. Compare with Intel's factory calibration values (stored on device). Quantify the difference.
Project 4: Calibration Quality vs Data Duration¶
Record three bags: 1 minute, 3 minutes, 5 minutes. Run camera-IMU calibration on each. Plot: estimated T_cam_imu uncertainty vs recording duration. Find the minimum data needed for your target uncertainty.
Project 5: Integrate into BEVFusion¶
Use Kalibr results to configure BEVFusion for your custom camera+LiDAR rig. Run the visual validation (project LiDAR onto camera). Compare BEVFusion detection accuracy with: - Default/identity extrinsic (no calibration) - Rough manual measurement - Kalibr calibrated result
Project 6: Calibration Validation Tooling¶
Write a Python script that:
1. Loads a camchain-imucam.yaml
2. Loads a short validation ROS bag (different from calibration bag)
3. Detects AprilGrid corners in camera frames
4. Projects the detected corner 3D positions through T_cam_imu
5. Computes reprojection error on held-out data
6. Reports pass/fail with a configurable threshold
15. Resources¶
Official¶
- Kalibr GitHub — github.com/ethz-asl/kalibr: source code, Docker, issues
- Kalibr Wiki — github.com/ethz-asl/kalibr/wiki: all documentation pages
- Kalibr Paper: "A Toolbox for Easily Calibrating Onboard Cameras" (Furgale et al., IROS 2013)
- Camera-IMU Paper: "Unified Temporal and Spatial Calibration for Multi-Sensor Systems" (Furgale et al., IROS 2013)
Allan Variance¶
- allan_variance_ros — github.com/ori-drs/allan_variance_ros: ROS package for IMU noise characterization
- IEEE Std 952-1997: formal definition of Allan Variance for IMU characterization
Related Calibration Tools¶
- LI-Calib — github.com/APRIL-ZJU/LI-Calib: LiDAR-IMU calibration (use results with Kalibr camera-IMU to get full camera-LiDAR-IMU chain)
- ACSC — github.com/HViktorTsoi/ACSC: automatic camera-LiDAR spatial calibration
- targetless_calibration — github.com/OpenCalib/JointCalib: targetless camera-LiDAR calibration using natural scene features
- OpenCV calibrateCamera — docs.opencv.org: simpler single-camera intrinsic calibration (use for quick checks; Kalibr for production)
Visual-Inertial Systems (where Kalibr calibration is used)¶
- OpenVINS — github.com/rpng/open_vins: production-quality VIO, excellent Kalibr integration docs
- VINS-Mono — github.com/HKUST-Aerial-Robotics/VINS-Mono: monocular VIO
- ORB-SLAM3 — github.com/UZ-SLAMLab/ORB_SLAM3: stereo/mono/IMU SLAM
Up: Sensor Fusion Guide See also: BEVFusion — uses Kalibr camera intrinsics and extrinsics See also: Multi-Object Tracking
Carry this course into the physical-AI system¶
Start from: A measured calibration target, sensor models, and an offline recording.
Build and challenge: Fit on one trajectory and validate on another. Verify transform direction, units, clock convention, and timestamp preservation through any bag conversion.
Pass forward: Calibration YAML, held-out residuals, and provenance for the fusion pipeline. Continue with the connected course and keep the artifact in the Physical AI Design Studio.
Numerical checks before using calibration¶
Kalibr's continuous white measurement noise density scales to per-sample standard deviation as sigma_sample = sigma_density / sqrt(dt) under its sampling/filtering assumptions. Bias random-walk increments instead scale as sigma_bias_step = sigma_bias_density * sqrt(dt). At 200 Hz (dt=0.005), a gyro density of 8.73e-5 rad/s/√Hz corresponds to approximately 0.00123 rad/s per sample. These two square-root scalings go in opposite directions. Downsampling/filtering can change the assumptions; do not copy a datasheet number without its bandwidth convention.
A board dimension error changes metric scale without necessarily increasing reprojection error. Run the known-scale CPU experiment before interpreting a subpixel residual as physical accuracy. Noise YAMLs above are illustrative values, not calibrated specifications for every device bearing that sensor name.
Sources checked 2026-10-18: Kalibr IMU noise model, transform/time YAML definitions, actual calibration CLI, Allan-variance tool usage. This review checked source arguments and conventions; it did not run a physical Kalibr calibration or validate the example device noise values.