Skip to content

Egocentric Data and VLA Fine-Tuning — Source Register

Course overview · Lab guide

Checked: October 26, 2026. Every version, commit, license and hardware figure in this course is pinned to the state of its primary source on that date. Sources disagree in places; where they do, the conflict is listed instead of resolved silently. License readings in this course are an engineer's reading of the license text, not legal advice. Have counsel review any commercial use.

Lecture text, worked arithmetic, synthetic fixtures and lab code are original teaching material. A number reported by a paper or a README is not a measurement made here, and completing a CPU lab does not reproduce any published result.

Egocentric and human-demonstration datasets

Dataset Scale as officially stated Data license Access and format Lectures Conflicts and caveats
Ego4D (arXiv 2110.07058) 3,670 h, 74 locations, 9 countries; 3.85M narration sentences Signed Ego4D License Agreement (public draft PDF) Request form, approval about 48 h, AWS credentials valid 14 days; pip install ego4d CLI (PyPI 1.7.3, default v2_1); MP4 + JSON; full-scale video "~7 TB" 02, 04, 05, 06, 08 Participants 931 (paper) / 923 (homepage) / 926 (docs). Draft allows commercial product development for listed purposes and forbids redistribution; robot control is not a named purpose; the agreement signed today may differ from the 2022 draft
Ego-Exo4D (arXiv 2311.18259) V2: 5,035 takes, 740 participants, 123 sites, 1,286.3 ego+exo hours (221.26 ego-hours) Separate signed agreement; docs say it "covers … research purposes and commercial use" Aria VRS (ego) + GoPro MP4 (exo); egoexo CLI in the ego4d package; "default" download about 12.1 TB 02, 04, 06, 08 Docs intro still says 1,422 h and >800 participants (arXiv v1). Agreement text not publicly readable: unverified
EPIC-KITCHENS-100 (arXiv 2006.13256) 100 h, 45 kitchens, 37 participants, 89,977 action segments, 97 verbs, 300 nouns CC BY-NC 4.0 (license.txt); also VISOR, HD-EPIC No form; data.bris, Academic Torrents or epic_downloader.py; annotations repo ea8b404 (v1.1.0) 02, 05 Non-commercial; commercial licensing by email only
HOI4D (arXiv 2203.01577) 2.4M RGB-D frames, 4,000 sequences, 800 object instances, 16 categories CC BY-NC 4.0 Direct links; RGB-D video, MANO hand pose, object pose, segmentation 02 Participants 9 (CVPR paper) vs 4 (arXiv v3/v4 abstract); resolution and fps unverified; MANO labels
HoloAssist (arXiv 2309.17024) 166 h (abstract), 350 instructor–performer pairs; HoloLens 2 RGB, depth, head pose, 26-joint hands, gaze CDLA-Permissive-2.0 Direct Azure tar downloads; hand pose 219 GB, compressed video 145 GB 02, 04, 05, 06, 08 Website says 169 h
Project Aria datasets: ADT, AEA, ASE, AEO ADT "200 sequences (~400 mins)" Meta Dataset License Agreement: non-commercial research in listed fields (robotics not listed) Email sign-up, URL file valid 14 days, aria_dataset_downloader; VRS + CSV/JSON 02, 04 ADT 200 (site) vs 236 (docs) sequences; AEA 7.3 vs 7.5 h
Nymeria (arXiv 2406.09905) 300 h, 264 participants, full-body mocap + Aria CC BY-NC 4.0 Dataset Explorer; VRS, body motion files 02 Sequence count 1,200 / 1,100 / 1,083 across sources
HOT3D (arXiv 2411.19167) 833 min, 19 subjects, 33 objects, Aria + Quest 3 Sequences CC BY-SA; hand annotations CC BY-NC-SA (license) VRS + hand trajectories in UmeTrack and MANO formats; HOT3D-Clips on HF 02, 06 Commercial use of the hand labels is not permitted
Aria Gen 2 Pilot Dataset (arXiv 2510.16134) 12 sequences of about 5 min, 4 participants CC BY-NC 4.0 Includes on-device and MPS hand tracking; projectaria-gen2-pilot-dataset 1.1.0 02 Only Gen 2 dataset listed on the check date
EgoDex (arXiv 2505.11709) 829 h of 30 Hz 1080p video, 338,000 demonstrations, 194 tasks, about 2.0 TB "CC-by-NC-ND terms" Ungated zip downloads; HDF5 with SE(3) transforms for head, upper body and 25 joints per hand, plus MP4 01, 02, 03, 04, 06, 07 No derivatives; CC version and participant count unverified; poses are in a per-episode ARKit origin frame
EgoMimic data (arXiv 2410.24221) Paper Table II: e.g. Object-in-Bowl 1,400 human demos / 60 min None stated (code MIT) Robomimic-style HDF5, 23–46 GB per file 01, 02, 08, 12 Unclear license: ask the authors before use
PH2D (arXiv 2503.13441) "~27k demonstrations" HF card license: mit Per-task HDF5, about 15.8 GB on HF 02, 08 Whether the HF upload holds all demos, and total hours, unverified
UMI (arXiv 2402.10329) 1,400 cup-arrangement demos, 30 locations (paper) No data license stated; code MIT cup_in_the_wild.zarr.zip (18 GB) at real.stanford.edu; mirror lerobot/umi_cup_in_the_wild (v3.0, tagged apache-2.0 by the uploader) 01, 02, 03, 04, 05, 07 Mirror has 1,447 episodes vs 1,400 in the paper
DexCap-Data (arXiv 2403.07788) 30 min wiping + 60 min in-the-wild packaging mocap CC BY 4.0 Raw zip + robomimic HDF5 02, 07 Chest-mounted cameras and gloves, not head-mounted video
EgoScale (arXiv 2602.16710) 20,854 h of action-labeled egocentric video used for training Not released No download 01, 02, 06, 07, 08, 09 Source datasets for the in-the-wild portion unverified
Extras: Egocentric-10K, DexWild, HiFi-UMI-2K, VITRA-1M, EgoVerse Egocentric-10K: 10,000 h factory video, no hand labels Apache-2.0 (gated); MIT; CC BY 4.0; MIT metadata only; not stated HF datasets 02, 08, 12 VITRA-1M does not redistribute the source videos, which keep their own licenses; EgoVerse hours 1,362 (paper) vs 4,003 (site)

Capture devices, geometry and hand pose

Source Pin checked Lectures Supported claim Limits
projectaria_tools Release 2.3.0 (2026-08-27, commit 36607a4), Apache-2.0; PyPI projectaria-tools==2.3.0 03, 13 Supports Gen 1 and Gen 2; camera models Linear, Spherical, KannalaBrandtK3, Fisheye624 (15 parameters: f, cu, cv, k0–k5, p0–p1, s0–s3), Fisheye62; time domains DeviceTime (recommended for single-device data), HostTime, TimeCode, TicSync, SubGhz The lab's Kannala–Brandt model is a simpler relative of Fisheye624
Aria MPS output formats MpsFileNames.h, Trajectory.h, HandTracking.h, EyeGaze.h at 2.3.0 02, 03, 04, 06, 07 slam/closed_loop_trajectory.csv gives T_world_device per graphUid; open-loop poses are in an odometry frame; hand tracking gives 21 landmarks per hand in the device frame and T_Device_Wrist; gaze is yaw/pitch/depth in the central pupil frame Poses with different graphUid values are in different world frames
HaMeR (arXiv 2312.05251) HEAD 3a01849, code MIT 02, 06, 08 Single-image 3D hand reconstruction returning MANO parameters, 3D keypoints and camera translation Requires the MANO model files under the MANO license
WiLoR (arXiv 2409.12259) HEAD fcb9113 02, 06 End-to-end hand detection and reconstruction; --fast half precision Models CC BY-NC-ND 4.0; depends on MANO and Ultralytics licenses
MANO license Checked 2026-10-26 02, 06, 08, 13 Non-commercial research only; prohibits training networks for commercial use; no redistribution Any label produced with MANO inherits these terms

Methods that learn from human or egocentric data

Method arXiv (first version) Code pin Lectures Supported claim
EgoMimic 2410.24221 (2024-10-31) SimarKareer/EgoMimic 6d63e9a, MIT 01, 02, 08, 12 Co-training on Aria human video and robot data; "1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data" for its tasks
GR00T N1 2503.14734 (2025-03-18) NVIDIA/Isaac-GR00T 01, 08, 09 Data pyramid; VQ-VAE latent actions as a "LAPA" embodiment; IDM pseudo-actions; 88 h teleoperation → about 827 h of generated video
LAPA 2410.11758 (2024-10-15) LatentActionPretraining/LAPA 46aca51, MIT 01, 08, 09 Latent actions learned between frames; pretraining without action labels; fine-tuning maps latents to robot actions
Emergence of Human to Robot Transfer in Vision-Language-Action Models 2512.22414 (2025-12-27) — 01, 08, 09, 10, 12 Transfer emerges with broad pretraining; human video as an embodiment with 3D hand-position actions; about 2× on four human-only scenarios (blog)
EgoScale 2602.16710 (2026-02-18) Not released 01, 02, 06, 07, 08, 09 20,854 h of action-labeled egocentric video; log-linear scaling; human pretraining then aligned human–robot mid-training
Phantom 2503.00779 (2025-03-02) MarionLepert/phantom a8bb81c, MIT 01, 08 Arm inpainting plus rendered robot overlay; zero robot data; uses a HaMeR fork and MANO
EgoZero 2505.20290 (2025-05-26) vliu15/egozero 51dc1c6, MIT 08 Aria demos only, 70% zero-shot success over 7 tasks with about 20 min per task
UMI 2402.10329 (2024-02-15) real-stanford/universal_manipulation_interface d095ba9, MIT 01, 02, 03, 04, 05, 07 Hand-held gripper with GoPro and SLAM; relative-trajectory actions; latency matching
Humanoid Policy ~ Human Policy 2503.13441 (2025-03-17) RogerQi/human-policy 2d9d73c, MIT 01, 02, 07, 08 PH2D and a unified human/humanoid state–action space
EgoVLA 2507.12440 (2025-07-16) RchalYang/EgoVLA_Release 09645b9 07 Predicts human wrist and hand actions; IK and retargeting to robot actions; few robot demos
VITRA 2510.21571 (2025-10-24) microsoft/VITRA b355172, MIT 01, 02, 08 Automatic conversion of unannotated egocentric video into 1M hand-action episodes
MimicPlay 2302.12422 (2023-02-24) j96w/MimicPlay 334ad94, MIT 08 Latent plans from human play video guide a low-level robot controller
EgoBridge 2509.19626 (2025-09-23) — 08, 12 Optimal-transport alignment of human and robot policy latents; reports +44% absolute over human-augmented baselines
DexWild 2505.07813 (2025-05-12) — 02, 12 Wearable human capture co-trained with robot data; reports 68.5% success in unseen environments
DexCap 2403.07788 (2024-03-12) j96w/DexCap 4b0bed0, MIT 02, 07 Portable hand mocap with IK and point-cloud imitation
PhysBrain 2512.16793 (2025-12-18) — 01, 08 Egocentric-video VLM initialization for VLA fine-tuning

These are reported results under each paper's own tasks, hardware and evaluation protocol. None was reproduced in this course.

VLA fine-tuning stacks

Stack Pin checked Lectures Documented facts used Conflicts and caveats
LeRobot Release v0.6.1 (2026-08-03), Apache-2.0; main 45cca1a 01, 02, 04, 05, 09, 10, 11, 13 Dataset format v3.0 (meta/info.json, meta/stats.json, meta/tasks.parquet, many episodes per Parquet/MP4 file); per-feature stats min/max/mean/std/count/q01–q99 from 5,000-bin histograms; NormalizationMode MIN_MAX, MEAN_STD, IDENTITY, QUANTILES, QUANTILE10; lerobot-edit-dataset … recompute_stats; load-time timestamp tolerance 1e-4 s; finalize() before push_to_hub(); example schema lerobot/svla_so101_pickplace (action float32 [6]); groot policy type runs GR00T N1.7, N1.5 support removed (pin lerobot==0.5.1 for N1.5) (groot.mdx) Main is ahead of v0.6.1 (its pyproject.toml says the unreleased 0.6.2); cite v0.6.1 for reproducibility. The v3 doc lists only mean/std/min/max, but the code also writes count and quantiles
SmolVLA (arXiv 2506.01844) lerobot/smolvla_base, 450,046,176 parameters, apache-2.0 01, 04, 09, 10, 11, 13 Defaults freeze_vision_encoder=True, train_expert_only=True, chunk_size=50, MEAN_STD for state and action; about 50 episodes recommended; 20k steps "~4 hrs on a single A100"; inference "requires only ~2GB" (async.mdx); main's doc adds --rename_map —
LeRobot PEFT v0.6.1 09, 11 lerobot[peft]; --peft.method_type=LORA --peft.r=64 --peft.lora_alpha=64; learning rate about 10× full fine-tuning (1e-3); SmolVLA default targets are q_proj/v_proj of the LM expert plus the state and action projections Built-in default targets for π0 / π0.5 appear only in main's doc, after v0.6.1
LeRobot π0 / π0.5 ports lerobot/pi0_base (3.50B), lerobot/pi05_base (3.62B) 09, 10, 11 π0.5 uses QUANTILES and fails with ValueError: QUANTILES normalization mode requires q01 and q99 stats when they are missing; LIBERO full fine-tune "Sized for a single 80 GB GPU" (--policy.gradient_checkpointing=true --policy.dtype=bfloat16 --batch_size=64); train_expert_only / freeze_vision_encoder give "less memory, at some cost in success rate"; needs the gated google/paligemma-3b-pt-224 tokenizer; "PI0 occupies 14GB of memory at inference time" (async.mdx) Docs say Apache 2.0; HF cards are tagged license: gemma (PaliGemma base). Treat weights as Gemma-terms derivatives
LeRobot hardware guide v0.6.1 09, 11, 13 Indicative peak VRAM at batch 8: ACT ~2–6 GB; Diffusion ~8–14 GB; SmolVLA fine-tuning ~10–16 GB. Large VLAs (π0, π0-FAST, π0.5, …) ~24–40 GB, "24 GB tight at BS 1"; memory roughly linear in batch size; training-time anchors (5 epochs, about 45k frames); --job.target=<flavor> for HF Jobs. ACT doc (act.mdx): "Start with batch size 8" The guide calls its figures "indicative"
openpi HEAD 215abfb (2026-08-24; no tags), Apache-2.0 plus LICENSE_GEMMA.txt 01, 04, 09, 10, 11, 13 Memory: inference "> 8 GB", LoRA "> 22.5 GB", full "> 70 GB" (single GPU; FSDP via --fsdp-devices); scripts/compute_norm_stats.py stores mean, std, q01, q99 for state and actions in assets/<config_name>/<repo_id>/norm_stats.json; π0 uses mean/std, π0-FAST and π0.5 use quantiles; LoRA configs exist for π0 and π0-FAST, not π0.5; pi05_libero is full fine-tuning (batch 256, peak lr 5e-5, 30k steps, EMA 0.999); pins transformers==4.53.2, torch==2.7.1, Python ≥3.11 Pins a LeRobot commit with dataset format v2.1; the PyTorch port does not support LoRA, FSDP or mixed precision (full bf16 or fp32 only)
Isaac-GR00T Release n1.7-release (2026-04-18); HEAD 944fe70; nvidia/GR00T-N1.7-3B 01, 04, 07, 08, 09, 10, 11, 13 Relative end-effector actions shared by robot and human data; "pretrained on 20K hours of EgoScale human video"; action_horizon 40; gated Cosmos-Reason2-2B backbone (NVIDIA Open Model License); transformers 4.57.3; data is LeRobot v2 plus meta/modality.json; v3→v2 converter; meta/stats.json plus meta/relative_stats.json, min-max by default; inference "16 GB+"; fine-tuning "Minimum: 1 GPU with 40 GB+ VRAM", default tuning peaks "under ~35 GB", --tune-llm / --tune-visual "80 GB+"; no LoRA option documented; NEW_EMBODIMENT tag. Hardware figures and commands read at HEAD 944fe70 README body says weights are Apache 2.0; README License section and HF card say NVIDIA Open Model License. Use: code Apache-2.0, weights NVIDIA Open Model License
OpenVLA-OFT (arXiv 2502.19645) HEAD e4287e9, MIT 04, 09, 10, 11 LoRA rank 32 on OpenVLA-7B; RLDS data; FAQ memory: minimum 25.6 GB at batch 1 (LIBERO, 1 image), 62.5 GB at batch 8 (LIBERO, 2 images + proprio), recommended up to 73.5 GB (ALOHA, batch 4), inference 15.9–18.0 GB; BOUNDS_Q99 normalization for LIBERO and Bridge, BOUNDS for ALOHA; q01/q99 saved as dataset_statistics.json in the checkpoint directory A new dataset must be registered in three config files
OpenVLA HEAD c8f03f4, MIT 09 Base checkpoint openvla/openvla-7b (7.54B parameters, MIT); LoRA needs "at least ~27 GB" —
NVIDIA GeForce RTX 4060 specifications Checked 2026-10-26 04, 11, 13 Desktop RTX 4060 "Standard Memory Config": 8 GB GDDR6, the course's reference consumer GPU —

Evaluation statistics

Source Lectures Supported claim
Wilson, "Probable Inference, the Law of Succession, and Statistical Inference", JASA 22(158), 1927 12, 13 Score interval for a binomial proportion, used for success rates
McNemar, "Note on the sampling error of the difference between correlated proportions or percentages", Psychometrika 12(2), 1947 12, 13 Test on discordant pairs for paired binary outcomes; the course uses the exact binomial form
Deep RL for Robot Learning, Lecture 13 12 Course background on evaluation rigor and intervals

Still unverified on the check date

  • The text of the Ego-Exo4D license agreement (the request page blocks automated access).
  • Whether the Ego4D agreement signed today matches the public 2022 draft.
  • EgoDex participant count and Creative Commons version.
  • Data licenses for EgoMimic, UMI cup_in_the_wild and EgoVerse.
  • PH2D total hours and whether the Hugging Face upload is complete.
  • Which in-the-wild datasets EgoScale trained on.
  • HOI4D frame resolution and frame rate.

Lectures do not state any of these as fact. Re-check them before relying on them.