Egocentric Data and VLA Fine-Tuning — Source Register¶
Checked: October 26, 2026. Every version, commit, license and hardware figure in this course is pinned to the state of its primary source on that date. Sources disagree in places; where they do, the conflict is listed instead of resolved silently. License readings in this course are an engineer's reading of the license text, not legal advice. Have counsel review any commercial use.
Lecture text, worked arithmetic, synthetic fixtures and lab code are original teaching material. A number reported by a paper or a README is not a measurement made here, and completing a CPU lab does not reproduce any published result.
Egocentric and human-demonstration datasets¶
| Dataset | Scale as officially stated | Data license | Access and format | Lectures | Conflicts and caveats |
|---|---|---|---|---|---|
| Ego4D (arXiv 2110.07058) | 3,670 h, 74 locations, 9 countries; 3.85M narration sentences | Signed Ego4D License Agreement (public draft PDF) | Request form, approval about 48 h, AWS credentials valid 14 days; pip install ego4d CLI (PyPI 1.7.3, default v2_1); MP4 + JSON; full-scale video "~7 TB" |
02, 04, 05, 06, 08 | Participants 931 (paper) / 923 (homepage) / 926 (docs). Draft allows commercial product development for listed purposes and forbids redistribution; robot control is not a named purpose; the agreement signed today may differ from the 2022 draft |
| Ego-Exo4D (arXiv 2311.18259) | V2: 5,035 takes, 740 participants, 123 sites, 1,286.3 ego+exo hours (221.26 ego-hours) | Separate signed agreement; docs say it "covers … research purposes and commercial use" | Aria VRS (ego) + GoPro MP4 (exo); egoexo CLI in the ego4d package; "default" download about 12.1 TB |
02, 04, 06, 08 | Docs intro still says 1,422 h and >800 participants (arXiv v1). Agreement text not publicly readable: unverified |
| EPIC-KITCHENS-100 (arXiv 2006.13256) | 100 h, 45 kitchens, 37 participants, 89,977 action segments, 97 verbs, 300 nouns | CC BY-NC 4.0 (license.txt); also VISOR, HD-EPIC | No form; data.bris, Academic Torrents or epic_downloader.py; annotations repo ea8b404 (v1.1.0) |
02, 05 | Non-commercial; commercial licensing by email only |
| HOI4D (arXiv 2203.01577) | 2.4M RGB-D frames, 4,000 sequences, 800 object instances, 16 categories | CC BY-NC 4.0 | Direct links; RGB-D video, MANO hand pose, object pose, segmentation | 02 | Participants 9 (CVPR paper) vs 4 (arXiv v3/v4 abstract); resolution and fps unverified; MANO labels |
| HoloAssist (arXiv 2309.17024) | 166 h (abstract), 350 instructor–performer pairs; HoloLens 2 RGB, depth, head pose, 26-joint hands, gaze | CDLA-Permissive-2.0 | Direct Azure tar downloads; hand pose 219 GB, compressed video 145 GB | 02, 04, 05, 06, 08 | Website says 169 h |
| Project Aria datasets: ADT, AEA, ASE, AEO | ADT "200 sequences (~400 mins)" | Meta Dataset License Agreement: non-commercial research in listed fields (robotics not listed) | Email sign-up, URL file valid 14 days, aria_dataset_downloader; VRS + CSV/JSON |
02, 04 | ADT 200 (site) vs 236 (docs) sequences; AEA 7.3 vs 7.5 h |
| Nymeria (arXiv 2406.09905) | 300 h, 264 participants, full-body mocap + Aria | CC BY-NC 4.0 | Dataset Explorer; VRS, body motion files | 02 | Sequence count 1,200 / 1,100 / 1,083 across sources |
| HOT3D (arXiv 2411.19167) | 833 min, 19 subjects, 33 objects, Aria + Quest 3 | Sequences CC BY-SA; hand annotations CC BY-NC-SA (license) | VRS + hand trajectories in UmeTrack and MANO formats; HOT3D-Clips on HF | 02, 06 | Commercial use of the hand labels is not permitted |
| Aria Gen 2 Pilot Dataset (arXiv 2510.16134) | 12 sequences of about 5 min, 4 participants | CC BY-NC 4.0 | Includes on-device and MPS hand tracking; projectaria-gen2-pilot-dataset 1.1.0 |
02 | Only Gen 2 dataset listed on the check date |
| EgoDex (arXiv 2505.11709) | 829 h of 30 Hz 1080p video, 338,000 demonstrations, 194 tasks, about 2.0 TB | "CC-by-NC-ND terms" | Ungated zip downloads; HDF5 with SE(3) transforms for head, upper body and 25 joints per hand, plus MP4 | 01, 02, 03, 04, 06, 07 | No derivatives; CC version and participant count unverified; poses are in a per-episode ARKit origin frame |
| EgoMimic data (arXiv 2410.24221) | Paper Table II: e.g. Object-in-Bowl 1,400 human demos / 60 min | None stated (code MIT) | Robomimic-style HDF5, 23–46 GB per file | 01, 02, 08, 12 | Unclear license: ask the authors before use |
| PH2D (arXiv 2503.13441) | "~27k demonstrations" | HF card license: mit |
Per-task HDF5, about 15.8 GB on HF | 02, 08 | Whether the HF upload holds all demos, and total hours, unverified |
| UMI (arXiv 2402.10329) | 1,400 cup-arrangement demos, 30 locations (paper) | No data license stated; code MIT | cup_in_the_wild.zarr.zip (18 GB) at real.stanford.edu; mirror lerobot/umi_cup_in_the_wild (v3.0, tagged apache-2.0 by the uploader) |
01, 02, 03, 04, 05, 07 | Mirror has 1,447 episodes vs 1,400 in the paper |
| DexCap-Data (arXiv 2403.07788) | 30 min wiping + 60 min in-the-wild packaging mocap | CC BY 4.0 | Raw zip + robomimic HDF5 | 02, 07 | Chest-mounted cameras and gloves, not head-mounted video |
| EgoScale (arXiv 2602.16710) | 20,854 h of action-labeled egocentric video used for training | Not released | No download | 01, 02, 06, 07, 08, 09 | Source datasets for the in-the-wild portion unverified |
| Extras: Egocentric-10K, DexWild, HiFi-UMI-2K, VITRA-1M, EgoVerse | Egocentric-10K: 10,000 h factory video, no hand labels | Apache-2.0 (gated); MIT; CC BY 4.0; MIT metadata only; not stated | HF datasets | 02, 08, 12 | VITRA-1M does not redistribute the source videos, which keep their own licenses; EgoVerse hours 1,362 (paper) vs 4,003 (site) |
Capture devices, geometry and hand pose¶
| Source | Pin checked | Lectures | Supported claim | Limits |
|---|---|---|---|---|
| projectaria_tools | Release 2.3.0 (2026-08-27, commit 36607a4), Apache-2.0; PyPI projectaria-tools==2.3.0 |
03, 13 | Supports Gen 1 and Gen 2; camera models Linear, Spherical, KannalaBrandtK3, Fisheye624 (15 parameters: f, cu, cv, k0–k5, p0–p1, s0–s3), Fisheye62; time domains DeviceTime (recommended for single-device data), HostTime, TimeCode, TicSync, SubGhz |
The lab's Kannala–Brandt model is a simpler relative of Fisheye624 |
| Aria MPS output formats | MpsFileNames.h, Trajectory.h, HandTracking.h, EyeGaze.h at 2.3.0 |
02, 03, 04, 06, 07 | slam/closed_loop_trajectory.csv gives T_world_device per graphUid; open-loop poses are in an odometry frame; hand tracking gives 21 landmarks per hand in the device frame and T_Device_Wrist; gaze is yaw/pitch/depth in the central pupil frame |
Poses with different graphUid values are in different world frames |
| HaMeR (arXiv 2312.05251) | HEAD 3a01849, code MIT |
02, 06, 08 | Single-image 3D hand reconstruction returning MANO parameters, 3D keypoints and camera translation | Requires the MANO model files under the MANO license |
| WiLoR (arXiv 2409.12259) | HEAD fcb9113 |
02, 06 | End-to-end hand detection and reconstruction; --fast half precision |
Models CC BY-NC-ND 4.0; depends on MANO and Ultralytics licenses |
| MANO license | Checked 2026-10-26 | 02, 06, 08, 13 | Non-commercial research only; prohibits training networks for commercial use; no redistribution | Any label produced with MANO inherits these terms |
Methods that learn from human or egocentric data¶
| Method | arXiv (first version) | Code pin | Lectures | Supported claim |
|---|---|---|---|---|
| EgoMimic | 2410.24221 (2024-10-31) | SimarKareer/EgoMimic 6d63e9a, MIT |
01, 02, 08, 12 | Co-training on Aria human video and robot data; "1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data" for its tasks |
| GR00T N1 | 2503.14734 (2025-03-18) | NVIDIA/Isaac-GR00T | 01, 08, 09 | Data pyramid; VQ-VAE latent actions as a "LAPA" embodiment; IDM pseudo-actions; 88 h teleoperation → about 827 h of generated video |
| LAPA | 2410.11758 (2024-10-15) | LatentActionPretraining/LAPA 46aca51, MIT |
01, 08, 09 | Latent actions learned between frames; pretraining without action labels; fine-tuning maps latents to robot actions |
| Emergence of Human to Robot Transfer in Vision-Language-Action Models | 2512.22414 (2025-12-27) | — | 01, 08, 09, 10, 12 | Transfer emerges with broad pretraining; human video as an embodiment with 3D hand-position actions; about 2× on four human-only scenarios (blog) |
| EgoScale | 2602.16710 (2026-02-18) | Not released | 01, 02, 06, 07, 08, 09 | 20,854 h of action-labeled egocentric video; log-linear scaling; human pretraining then aligned human–robot mid-training |
| Phantom | 2503.00779 (2025-03-02) | MarionLepert/phantom a8bb81c, MIT |
01, 08 | Arm inpainting plus rendered robot overlay; zero robot data; uses a HaMeR fork and MANO |
| EgoZero | 2505.20290 (2025-05-26) | vliu15/egozero 51dc1c6, MIT |
08 | Aria demos only, 70% zero-shot success over 7 tasks with about 20 min per task |
| UMI | 2402.10329 (2024-02-15) | real-stanford/universal_manipulation_interface d095ba9, MIT |
01, 02, 03, 04, 05, 07 | Hand-held gripper with GoPro and SLAM; relative-trajectory actions; latency matching |
| Humanoid Policy ~ Human Policy | 2503.13441 (2025-03-17) | RogerQi/human-policy 2d9d73c, MIT |
01, 02, 07, 08 | PH2D and a unified human/humanoid state–action space |
| EgoVLA | 2507.12440 (2025-07-16) | RchalYang/EgoVLA_Release 09645b9 |
07 | Predicts human wrist and hand actions; IK and retargeting to robot actions; few robot demos |
| VITRA | 2510.21571 (2025-10-24) | microsoft/VITRA b355172, MIT |
01, 02, 08 | Automatic conversion of unannotated egocentric video into 1M hand-action episodes |
| MimicPlay | 2302.12422 (2023-02-24) | j96w/MimicPlay 334ad94, MIT |
08 | Latent plans from human play video guide a low-level robot controller |
| EgoBridge | 2509.19626 (2025-09-23) | — | 08, 12 | Optimal-transport alignment of human and robot policy latents; reports +44% absolute over human-augmented baselines |
| DexWild | 2505.07813 (2025-05-12) | — | 02, 12 | Wearable human capture co-trained with robot data; reports 68.5% success in unseen environments |
| DexCap | 2403.07788 (2024-03-12) | j96w/DexCap 4b0bed0, MIT |
02, 07 | Portable hand mocap with IK and point-cloud imitation |
| PhysBrain | 2512.16793 (2025-12-18) | — | 01, 08 | Egocentric-video VLM initialization for VLA fine-tuning |
These are reported results under each paper's own tasks, hardware and evaluation protocol. None was reproduced in this course.
VLA fine-tuning stacks¶
| Stack | Pin checked | Lectures | Documented facts used | Conflicts and caveats |
|---|---|---|---|---|
| LeRobot | Release v0.6.1 (2026-08-03), Apache-2.0; main 45cca1a |
01, 02, 04, 05, 09, 10, 11, 13 | Dataset format v3.0 (meta/info.json, meta/stats.json, meta/tasks.parquet, many episodes per Parquet/MP4 file); per-feature stats min/max/mean/std/count/q01–q99 from 5,000-bin histograms; NormalizationMode MIN_MAX, MEAN_STD, IDENTITY, QUANTILES, QUANTILE10; lerobot-edit-dataset … recompute_stats; load-time timestamp tolerance 1e-4 s; finalize() before push_to_hub(); example schema lerobot/svla_so101_pickplace (action float32 [6]); groot policy type runs GR00T N1.7, N1.5 support removed (pin lerobot==0.5.1 for N1.5) (groot.mdx) |
Main is ahead of v0.6.1 (its pyproject.toml says the unreleased 0.6.2); cite v0.6.1 for reproducibility. The v3 doc lists only mean/std/min/max, but the code also writes count and quantiles |
| SmolVLA (arXiv 2506.01844) | lerobot/smolvla_base, 450,046,176 parameters, apache-2.0 |
01, 04, 09, 10, 11, 13 | Defaults freeze_vision_encoder=True, train_expert_only=True, chunk_size=50, MEAN_STD for state and action; about 50 episodes recommended; 20k steps "~4 hrs on a single A100"; inference "requires only ~2GB" (async.mdx); main's doc adds --rename_map |
— |
| LeRobot PEFT | v0.6.1 | 09, 11 | lerobot[peft]; --peft.method_type=LORA --peft.r=64 --peft.lora_alpha=64; learning rate about 10× full fine-tuning (1e-3); SmolVLA default targets are q_proj/v_proj of the LM expert plus the state and action projections |
Built-in default targets for π0 / π0.5 appear only in main's doc, after v0.6.1 |
| LeRobot π0 / π0.5 ports | lerobot/pi0_base (3.50B), lerobot/pi05_base (3.62B) |
09, 10, 11 | π0.5 uses QUANTILES and fails with ValueError: QUANTILES normalization mode requires q01 and q99 stats when they are missing; LIBERO full fine-tune "Sized for a single 80 GB GPU" (--policy.gradient_checkpointing=true --policy.dtype=bfloat16 --batch_size=64); train_expert_only / freeze_vision_encoder give "less memory, at some cost in success rate"; needs the gated google/paligemma-3b-pt-224 tokenizer; "PI0 occupies 14GB of memory at inference time" (async.mdx) |
Docs say Apache 2.0; HF cards are tagged license: gemma (PaliGemma base). Treat weights as Gemma-terms derivatives |
| LeRobot hardware guide | v0.6.1 | 09, 11, 13 | Indicative peak VRAM at batch 8: ACT ~2–6 GB; Diffusion ~8–14 GB; SmolVLA fine-tuning ~10–16 GB. Large VLAs (π0, π0-FAST, π0.5, …) ~24–40 GB, "24 GB tight at BS 1"; memory roughly linear in batch size; training-time anchors (5 epochs, about 45k frames); --job.target=<flavor> for HF Jobs. ACT doc (act.mdx): "Start with batch size 8" |
The guide calls its figures "indicative" |
| openpi | HEAD 215abfb (2026-08-24; no tags), Apache-2.0 plus LICENSE_GEMMA.txt |
01, 04, 09, 10, 11, 13 | Memory: inference "> 8 GB", LoRA "> 22.5 GB", full "> 70 GB" (single GPU; FSDP via --fsdp-devices); scripts/compute_norm_stats.py stores mean, std, q01, q99 for state and actions in assets/<config_name>/<repo_id>/norm_stats.json; π0 uses mean/std, π0-FAST and π0.5 use quantiles; LoRA configs exist for π0 and π0-FAST, not π0.5; pi05_libero is full fine-tuning (batch 256, peak lr 5e-5, 30k steps, EMA 0.999); pins transformers==4.53.2, torch==2.7.1, Python ≥3.11 |
Pins a LeRobot commit with dataset format v2.1; the PyTorch port does not support LoRA, FSDP or mixed precision (full bf16 or fp32 only) |
| Isaac-GR00T | Release n1.7-release (2026-04-18); HEAD 944fe70; nvidia/GR00T-N1.7-3B |
01, 04, 07, 08, 09, 10, 11, 13 | Relative end-effector actions shared by robot and human data; "pretrained on 20K hours of EgoScale human video"; action_horizon 40; gated Cosmos-Reason2-2B backbone (NVIDIA Open Model License); transformers 4.57.3; data is LeRobot v2 plus meta/modality.json; v3→v2 converter; meta/stats.json plus meta/relative_stats.json, min-max by default; inference "16 GB+"; fine-tuning "Minimum: 1 GPU with 40 GB+ VRAM", default tuning peaks "under ~35 GB", --tune-llm / --tune-visual "80 GB+"; no LoRA option documented; NEW_EMBODIMENT tag. Hardware figures and commands read at HEAD 944fe70 |
README body says weights are Apache 2.0; README License section and HF card say NVIDIA Open Model License. Use: code Apache-2.0, weights NVIDIA Open Model License |
| OpenVLA-OFT (arXiv 2502.19645) | HEAD e4287e9, MIT |
04, 09, 10, 11 | LoRA rank 32 on OpenVLA-7B; RLDS data; FAQ memory: minimum 25.6 GB at batch 1 (LIBERO, 1 image), 62.5 GB at batch 8 (LIBERO, 2 images + proprio), recommended up to 73.5 GB (ALOHA, batch 4), inference 15.9–18.0 GB; BOUNDS_Q99 normalization for LIBERO and Bridge, BOUNDS for ALOHA; q01/q99 saved as dataset_statistics.json in the checkpoint directory |
A new dataset must be registered in three config files |
| OpenVLA | HEAD c8f03f4, MIT |
09 | Base checkpoint openvla/openvla-7b (7.54B parameters, MIT); LoRA needs "at least ~27 GB" |
— |
| NVIDIA GeForce RTX 4060 specifications | Checked 2026-10-26 | 04, 11, 13 | Desktop RTX 4060 "Standard Memory Config": 8 GB GDDR6, the course's reference consumer GPU | — |
Evaluation statistics¶
| Source | Lectures | Supported claim |
|---|---|---|
| Wilson, "Probable Inference, the Law of Succession, and Statistical Inference", JASA 22(158), 1927 | 12, 13 | Score interval for a binomial proportion, used for success rates |
| McNemar, "Note on the sampling error of the difference between correlated proportions or percentages", Psychometrika 12(2), 1947 | 12, 13 | Test on discordant pairs for paired binary outcomes; the course uses the exact binomial form |
| Deep RL for Robot Learning, Lecture 13 | 12 | Course background on evaluation rigor and intervals |
Still unverified on the check date¶
- The text of the Ego-Exo4D license agreement (the request page blocks automated access).
- Whether the Ego4D agreement signed today matches the public 2022 draft.
- EgoDex participant count and Creative Commons version.
- Data licenses for EgoMimic, UMI
cup_in_the_wildand EgoVerse. - PH2D total hours and whether the Hugging Face upload is complete.
- Which in-the-wild datasets EgoScale trained on.
- HOI4D frame resolution and frame rate.
Lectures do not state any of these as fact. Re-check them before relying on them.