VLA Foundations to Frontier 2026 — Professional Course¶
Phase 5 · Robotics · Professional Course
Understand, train, evaluate, and deploy vision-language-action policies—from their historical foundations to the October 2026 research frontier.
Research snapshot: October 13, 2026, with a targeted source and execution update checked October 18. This is a substantial engineering course, not a claim to cover every paper or developments later in October. Current-model coverage distinguishes announced results, papers, code, downloadable weights, API access, and partner access. Start with Lecture 01; use the frontier register when selecting an actual model artifact.
A VLA maps visual observations, language, and often robot state/history to actions. Professional work requires understanding the entire chain: data provenance, representation, learning objective, model architecture, action semantics, controller integration, inference timing, and closed-loop evidence. This course teaches that chain through worked examples, guided experiments, negative controls, and independent transfer tasks.
Role targets: Robot Learning Engineer · Robotics Foundation-Model Engineer · Embodied AI Research Engineer · VLA Training Engineer · Robotics Inference and Evaluation Engineer. Roadmap layers: L3–L8, from model/training choices through GPU execution, runtime integration, and robot behavior.
The chip this course helps you design¶
This is the model and control foundation for the roadmap's larger goal: a custom physical-AI inference SoC built for advanced VLA workloads, with a dedicated accelerator, an AI compiler, embedded Linux, and integrated wireless for robots, drones, and autonomous machines. Learn how models work so their actual computation, state, numerical sensitivity, and timing requirements can guide the hardware.
Carry a design brief through the course: selected checkpoint and tasks; vision/language/action operator coverage; supported shapes and precision; memory and bandwidth; sensor-to-action deadlines; compiler/runtime interfaces; and closed-loop quality gates. Use Capstone II's hardware co-design route to turn this into an experiment, then take Programmable VLA Inference Accelerator — 10 lectures before the physical-AI SoC capstone. The accelerator course converts your workload evidence into memory, compiler, scheduling, and prototype decisions; the Linux and wireless tracks complete the system integration.
Outcomes and prerequisites¶
By the end, you should be able to derive the central imitation/diffusion/flow objectives, read a VLA paper critically, build a causal robot dataset pipeline, adapt an accessible policy, define its physical action contract, evaluate it in closed loop, and defend one measured training or deployment result. The CPU route establishes foundations and tooling; the GPU/simulator route adds model adaptation and robot-learning integration. Physical deployment requires additional device-specific experience and supervised validation.
Required background is Python, linear algebra, probability, gradient descent, basic neural networks, and elementary feedback control. Familiarity with PyTorch helps with advanced exercises. You do not need to complete every robotics specialization first. For gaps, use the robotics guide, Deep RL for Robot Learning, and the Isaac course selectively.
Entry diagnostic: explain a train/test split, compute an affine normalization, differentiate a squared-error loss, distinguish pose from velocity, and draw a sensor-to-controller loop. If any answer is uncertain, work through Lectures 01, 06, 09, and 14 before attempting model adaptation. The notation and glossary is a companion, not a substitute for those lessons.
Course map¶
Each lecture teaches a concrete mechanism, works through an example, specifies a lab or exercise, identifies failure modes, and provides self-check answer guidance. The historical sequence establishes why architectures changed; later modules teach how to reproduce and evaluate them.
| # | Lecture | Inspectable learning artifact |
|---|---|---|
| 01 | What a VLA policy is | Policy/control contract and POMDP diagram |
| 02 | Historical foundations: control, imitation, and deep visuomotor learning | Historical mechanism timeline and distribution-shift experiment |
| 03 | Language grounding: CLIP, CLIPort, and SayCan | Grounding and skill-affordance comparison |
| 04 | RT-1, RT-2, and PaLM-E: three distinct architectural steps | RT-1/RT-2/PaLM-E architecture and evidence table |
| 05 | Open X-Embodiment, RT-X, Octo, and OpenVLA | Cross-embodiment schema and baseline-selection record |
| 06 | Geometry, robot state, and control contracts | Frame, unit, state, and action validation |
| 07 | Visual and language encoders, fusion, and attention masks | Token shapes, attention masks, and compute budget |
| 08 | Action representations, quantization, and FAST | Action/token codec and reconstruction analysis |
| 09 | Behavioral Cloning, Sequence Losses, and Covariate Shift | Masked BC objective and covariate-shift experiment |
| 10 | Action Chunking, ACT, and Temporal Ensembling | Chunk scheduling and temporal-ensemble comparison |
| 11 | Diffusion Action Policies from Noise to Control | Diffusion objective, sampler, and action-mode analysis |
| 12 | Flow Matching and the π₀ Architecture | Flow-matching derivation and π₀ component map |
| 13 | Robot Data Collection, Teleoperation, and Annotation | Collection protocol and annotation audit |
| 14 | Dataset Schemas, Transformations, Splits, and Mixtures | Causal dataset transform and leakage report |
| 15 | Pretraining and Cross-Embodiment Generalization | Cross-embodiment objective and mixture experiment |
| 16 | Fine-Tuning, Adapters, Normalization, and Calibration | Adaptation plan with normalization parity |
| 17 | Offline and Online RL Post-Training for VLAs | RL objective audit and duration-aware credit assignment |
| 18 | Training Infrastructure, Distributed Execution, and Reproducibility | Reproducible training/resume and resource record |
| 19 | Closed-Loop Evaluation, Benchmarks, and Statistical Evidence | Paired evaluation with uncertainty and exclusions |
| 20 | Integrating a VLA with Isaac Sim and Isaac Lab | Isaac task-to-policy adapter contract |
| 21 | VLA Inference Optimization and Edge Deployment | Inference trace, candidate optimization, and parity plan |
| 22 | Asynchronous Control, ROS 2, and Real-Time Action Chunking | Timestamped scheduler and fault fixtures |
| 23 | Robustness, System Identification, and Sim-to-Real Transfer | Identification experiment and robustness matrix |
| 24 | Deployment Boundaries, Recovery, and Human Intervention | Supervisor state machine and intervention analysis |
| 25 | Physical Intelligence: from π0 to steerable generalists | PI lineage, conditioning, and artifact comparison |
| 26 | NVIDIA GR00T and the Isaac engineering boundary | GR00T/Isaac compatibility and timing audit |
| 27 | Gemini Robotics: reasoning, action, and access boundaries | Reasoning/action boundary and access-aware evaluation |
| 28 | Open VLA systems: choose an artifact, not a headline | Executable open-model selection dossier |
| 29 | Reasoning, memory, hierarchy, and world-action models | Memory/reasoning/world-model ablation design |
| 30 | The October 2026 frontier: comparison and critical reading | Dated critical comparison of frontier evidence |
| 31 | Capstone I: Train, Diagnose, and Integrate a Grounded Policy | Trained CPU grounding model and optional simulator adaptation |
| 32 | Capstone II: A Professional VLA Research and Deployment Review | Reproducible professional project and oral defense |
Study sequence and workload¶
Use a 16-week schedule with two lectures per week, followed by the associated lab. The full engineering route is approximately 180–240 hours, depending on background and chosen capstones, excluding large-scale training and difficult hardware setup. Reading alone takes much less time but does not satisfy the practical assessment. Per-lecture estimates distinguish teaching, setup, and execution.
| Weeks | Module | Milestone |
|---|---|---|
| 1–2 | Lectures 01–04: foundations and historical turning points | Explain why language grounding, visuomotor control, and generalist pretraining are distinct problems |
| 3–4 | Lectures 05–08: datasets and representations | Validate geometry, schemas, visual tokens, and action representations |
| 5–6 | Lectures 09–12: imitation and generative action learning | Derive and diagnose BC, ACT, diffusion, and flow objectives |
| 7–9 | Lectures 13–18: data, adaptation, RL, and training systems | Produce a reproducible data/training design with held-out evaluation |
| 10–12 | Lectures 19–24: evaluation and deployment | Build contracts, timing fixtures, and an Isaac integration/robustness plan |
| 13–15 | Lectures 25–30: model families and current research | Audit a 2026 artifact and defend a protocol-specific comparison |
| 16 and project time | Lectures 31–32: capstones | Submit reproducible evidence and defend its scope |
For a faster role-specific route, retain the shared foundations and adjust the project emphasis:
- Training/research: 01–18, 19, selected 25–30, and both capstones; prioritize data, losses, adaptation, and ablations.
- Inference/deployment: 01, 04–08, 10–12, 14, 16, 19–24, 26, 28, 30–32; prioritize timing, memory, action parity, and controllers.
- Robotics integration: 01–06, 10, 13–14, 19–24, 26–27, 29, 31–32; pair with the Isaac engineering labs.
These routes reduce breadth; complete all 32 lectures for the full course. Do not skip a prerequisite solely because its topic is outside your intended job title.
Learn through prediction, intervention, and explanation¶
Before each worked example, predict the result. Reproduce the reasoning or code, change one assumption, and retain a failure case. Answer the self-check without notes, then compare with the answer guidance. Revisit the key mechanism after two days and one week. Follow the shared learning guide for experiment records and spaced retrieval.
The first CPU capstone trains a tiny nine-parameter grounding model with a fixed visual encoder. It makes gradients and language interventions inspectable; it is explicitly not a miniature claim of general robot intelligence. Later projects use a real released VLA only when its model, data, runtime, and controller contracts can be satisfied.
Labs, hardware, and assessment¶
Start with the laboratory guide. The core supplied CPU exercises use the Python standard library and require no model downloads. The action-generation lab adds seven framework-free numerical checks and optional real PyTorch training for diffusion and flow; that training route needs the documented framework installation. They include a trainable synthetic policy, episode-safe chunks, action normalization, delayed-action eligibility, paired task-level statistics, and a supplied recorded-data conversion and action-dispatch exercise. Tests run in curriculum CI.
GPU exercises depend on the selected model and environment. There is no single VRAM minimum for all VLAs. Measure simultaneous model, camera, simulator, workspace, and training-state memory, and use the selected upstream support matrix. The Isaac lab guide separates supported simulation from reduced exercises. CPU success does not validate RTX simulation, checkpoint fine-tuning, or a physical controller.
Assessment combines eight module artifacts (40%), Capstone I (25%), Capstone II (25%), and an oral/code review (10%). Use the professional project rubric for correctness, reasoning, measurement, reproduction, and transfer. Missing action-contract validation or unreported test leakage must be repaired regardless of aggregate score. This is a self-study professional curriculum, not an accredited credential.
Eight assessed module artifacts¶
Each artifact contributes 5% of the course score. Use the same correctness/reasoning/measurement/reproduction/transfer rubric as the capstones, scaled to the assignment.
| Artifact | Lectures | Required evidence |
|---|---|---|
| 1. Historical and architectural explanation | 01–04 | Mechanism timeline, three architecture diagrams, one grounding counterexample |
| 2. Representation contract | 05–08 | Embodiment schema, frame/unit checks, action-codec reconstruction error |
| 3. Learning objectives | 09–12 | Derived losses, one checked gradient, multimodal-action failure analysis |
| 4. Data and adaptation plan | 13–16 | Collection protocol, causal transforms, split audit, normalizer/checkpoint manifest |
| 5. RL and training systems | 17–18 | Objective/likelihood audit, resume design, resource and duration-aware return calculation |
| 6. Evaluation and simulation boundary | 19–20 | Paired-statistics report and validated adapter specification; label unexecuted simulator work |
| 7. Deployment and robustness | 21–24 | Timing budget, fault fixtures, perturbation matrix, recovery state machine |
| 8. Frontier research review | 25–30 | Primary-source comparison with event dates, access, protocol, and reproduction limits |
Research reading and current coverage¶
Read sources with a question: what is the problem, what changes in the mechanism, what is controlled, what evidence supports the claim, and what can actually be reproduced? The source dossiers provide reading order and limitations:
- Foundations and history
- Training, data, and RL
- Evaluation, simulation, and deployment
- 2026 model families and research
- Dated frontier register
The 2026 coverage includes verified work on π0.7, GR00T N1.7, Gemini Robotics 2, MolmoAct 2, LingBot-VLA 2.0, Helix 2.5, world-action models, and papers revised in October before the cutoff. An inclusion records evidence, not endorsement or a universal ranking. Access, model-card terms, and benchmark configurations can differ between versions.
Connections to the roadmap¶
Continue with Egocentric Data and VLA Fine-Tuning to train on first-person human video and fine-tune an open VLA end to end, VLA Optimization and Action-Parity Harness for specialist inference work, Deep RL for deeper control learning, and Isaac Sim and Isaac Lab for simulator engineering. The DeepSeek NVFP4 course contributes checkpoint and runtime methods; language-model deployment and robot action deployment still require different output and evaluation contracts.
Carry this course into the physical-AI system¶
Start from: The task/observation contract and a released, usable policy artifact.
Build and challenge: Follow history and architecture into action representation, data, training, evaluation, and deployment. Trace a complete action chunk through normalization, frame conversion, expiry, and controller consumption.
Pass forward: A workload dossier and closed-loop reference that constrain the future accelerator. Continue with the connected course and keep the artifact in the Physical AI Design Studio.