Core concept

A vision-language-action (VLA) policy is a policy whose inputs include camera images and a natural-language instruction, usually together with the robot’s own state, and whose output is a robot action:

with $I_t$ the images at time $t$, $\ell$ the instruction and $q_t$ the proprioceptive state. The model is trained on robot data, almost always demonstrations, so that the mapping from these inputs to actions is learned rather than written by hand.

That definition is functional: it says what goes in and what comes out. Three other properties often travel with the word in papers and press releases, and this course keeps them separate:

• Scale. OpenVLA has 7 billion parameters; the MiniVLA built in this level has 110 050. Both map images and words to actions.
• Pretraining. Many VLAs start from a vision-language model trained on web data. A VLA can also be trained from scratch on robot data, as this level’s is.
• Generality. A policy trained on many robots and tasks, and meant to be adapted, is a robot foundation model. A VLA trained on one task family is not.

When one of these matters to a claim, state it explicitly, for example “a 7B VLA pretrained on web data”, rather than letting “VLA” imply it.

[!established] Where the term comes from RT-2 (Brohan et al., arXiv:2307.15818) introduced the name. Its authors co-fine-tune vision-language models on robot trajectories and web vision-language tasks, express actions as text tokens, and write: “We refer to such category of models as vision-language-action models (VLA).” Their category is VLMs that output actions. Later work, including OpenVLA (arXiv:2406.09246) and π0 (arXiv:2410.24164), keeps the pretrained-VLM backbone. The functional definition above is broader: it is this course’s convention, so that a small policy can be studied with the same vocabulary.

Visual intuition

Output (1) below is the whole interface of a VLA in the course’s tabletop scene. It shows one 64 × 64 image from the top camera, one instruction turned into twelve token ids, and four numbers of robot state, mapped to a two-dimensional action. The twelve numbers the policy is not given, where the pucks are and how fast they move, are what its vision has to supply. The next lesson’s lab traces one such observation through every layer of the model.

Mathematics

Six kinds of system, distinguished by their interfaces

The rows are not exclusive, because they classify along different axes. The interface (what goes in, what comes out) makes a VLA, a VLM or a multimodal policy. The training method makes an imitation-learning policy. The data and purpose make a foundation model. The decision procedure makes an agentic policy. The course’s MiniVLA is a VLA (by interface), an imitation-learning policy (by training), a multimodal policy, and not a foundation model. Lesson 23.8 builds an agentic policy that contains it.

What the representative systems are

Each of the following facts is taken from the cited paper’s text; none is a measurement of this course.

Read the “Action output” column against Lesson 22.3. Three different answers to the question “what does the network output?” appear in it: tokens over discretized bins (RT-1, RT-2, OpenVLA), continuous chunks from a generative head (Octo, π0), and, in this level, continuous regression.

[!recommendation] Describing a system as a VLA Name the interface (cameras, instruction, state in; action space and rate out), then scale, pretraining and training data separately. “A VLA” alone tells the reader only the interface.

The course’s VLA, and what it is not

The MiniVLA of Lessons 22.2 and 22.4 maps a 64 × 64 image, an instruction from a 24-word vocabulary and the pusher’s position and velocity to a pusher velocity. Its 110 050 parameters split as follows (output 3): vision encoder 40 032, word embeddings 1 536, state encoder 320, fusion transformer 66 944, action head 130. It has no pretrained component, so it knows no word that its training instructions did not contain: a synonym such as “crimson” is an unknown token to it (Lesson 22.5 measures what that costs). It is a pedagogical VLA-style architecture, and it reproduces none of the systems in the table. It exists so that every stage of the pipeline those systems share (encode, fuse, decode, act, evaluate) can be run, inspected and broken on a laptop.

Implementation

INPUT: tabletop.xml; the course's MiniVLA, untrained
PROCESS: build one observation; pass it through the model; count parameters; list the six interfaces
OUTPUT: printed shapes, parameter counts and interfaces

```python file=examples/l22_1_vla_interface.py import numpy as np import torch

from mjcourse import tabletop as tt from mjcourse.vla import language from mjcourse.vla.model import MiniVLA, VLAConfig from mjcourse.vla.train import STATE_SCALE

def part1_2() -> None: rng = np.random.default_rng(0) text, task = language.instructions(“train”)[0] scene = tt.Tabletop() scene.reset(tt.sample_layout(rng, task)) image, state = scene.render(“top”, 64), scene.state() tokenizer = language.default_tokenizer() words = tokenizer(text) print(“(1) one observation”) print(f” image {image.shape} {image.dtype}, values {image.min()} to {image.max()} (top camera, RGB)”) print(f” instruction {text!r} -> token ids {words.tolist()} (vocabulary of {len(tokenizer.vocab)} words)”) print(f” robot state {np.round(state[:4], 3).tolist()} (pusher x, y in m; vx, vy in m/s)”) print(f” not observed {state.size - 4} numbers: the pucks’ positions and velocities, which the image must supply”) torch.manual_seed(0) model = MiniVLA(VLAConfig(vocab=len(tokenizer.vocab), chunk=1)).eval() with torch.no_grad(): action = model(torch.as_tensor(image).permute(2, 0, 1).float().unsqueeze(0) / 255.0, torch.as_tensor(words).unsqueeze(0), torch.as_tensor((state[:4] * STATE_SCALE)[None], dtype=torch.float32)) print(“(2) the policy’s output”) print(f” action {tuple(action.shape)} = (batch, chunk, 2): a pusher velocity in [-1, 1] x {tt.MAX_SPEED} m/s” f” (untrained: {[round(float(v), 3) for v in action[0, 0]]})”) scene.close()

def part3() -> None: model = MiniVLA(VLAConfig(vocab=len(language.default_tokenizer().vocab), chunk=1)) parts = {“vision encoder”: model.vision, “word embeddings”: model.words, “state encoder”: model.state, “fusion transformer”: model.fusion, “action head”: model.head} print(“(3) parameters”) for name, module in parts.items(): print(f” {name:<20} {sum(p.numel() for p in module.parameters()):>8,d}”) print(f” {‘total’:<20} {sum(p.numel() for p in model.parameters()):>8,d}”)

def part4() -> None: rows = [(“vision-language model”, “image(s) + text”, “text”), (“vision-language-action policy”, “image(s) + instruction (+ robot state)”, “robot action (or a chunk of them)”), (“imitation-learning policy”, “any observation”, “robot action, learned from demonstrations”), (“multimodal policy”, “two or more modalities, e.g. image + joint angles”, “robot action”), (“robot foundation model”, “varies; trained on large, diverse robot data”, “robot action; meant to be adapted”), (“agentic policy”, “observation + goal; may call models and keep memory”, “robot action chosen by a loop of proposals, predictions and selection”)] print(“(4) interfaces”) for name, inputs, outputs in rows: print(f” {name:<31} in: {inputs:<53} out: {outputs}”)

if name == “main”: part1_2() part3() part4()

Output (MuJoCo 3.14.0, PyTorch 2.14 CPU; the untrained model is seeded, so its action is reproducible and meaningless):

```text
(1) one observation
  image        (64, 64, 3) uint8, values 35 to 255  (top camera, RGB)
  instruction  'push the red puck to the upper zone' -> token ids [14, 19, 16, 13, 20, 19, 22, 23, 0, 0, 0, 0]  (vocabulary of 24 words)
  robot state  [0.053, 0.115, 0.0, 0.0]  (pusher x, y in m; vx, vy in m/s)
  not observed 12 numbers: the pucks' positions and velocities, which the image must supply
(2) the policy's output
  action       (1, 1, 2) = (batch, chunk, 2): a pusher velocity in [-1, 1] x 0.2 m/s (untrained: [-0.342, 0.671])
(3) parameters
  vision encoder         40,032
  word embeddings         1,536
  state encoder             320
  fusion transformer     66,944
  action head               130
  total                 110,050
(4) interfaces
  vision-language model           in: image(s) + text                                       out: text

Debugging

A system is called a VLA but outputs a plan in words. That is a VLM used as a planner, followed by a controller. The distinction matters for evaluation: the controller’s failures and the planner’s are separable, as in Lesson 16.1’s parser and executor.

A “language-conditioned” policy receives a task index. If the instruction reaches the network as a one-hot task identifier, the policy has no language input, and paraphrase and synonym evaluations (Lesson 22.5) are meaningless for it.

A comparison between “VLAs” mixes scales and pretraining. Report parameters, pretraining and training data for every system compared; otherwise a difference may come from any of them.

Exercise

For each of the six systems in the table, write its interface as a function signature, with the shapes and units of every input and output that the paper states, and mark the ones it does not state. Then classify the following as VLM, VLA, multimodal policy or none of these: the state policy of Lesson 14.1, the image policy of Lesson 15.1, Lesson 16.1’s parser plus scripted pick-and-place, and the MiniVLA.

Challenge

Read the action-tokenization sections of RT-1 and OpenVLA. Compute, for the MiniVLA’s pusher velocity in [−0.2, 0.2] m/s, the resolution of a 256-bin uniform tokenization and of OpenVLA’s percentile-based variant on the course’s demonstrations (use the actions saved by Lesson 22.4’s data generator). Which actions in those demonstrations fall outside the 1st to 99th percentile, and what does the policy lose by clipping them?

Research connection

[!research] The word and the evidence Because “VLA” names an interface, claims about VLAs as a class are claims about very different systems: a 35M-parameter transformer trained on one fleet’s data (RT-1), a 55B VLM co-fine-tuned on web and robot data (RT-2), a 3.3B model with a flow-matching action expert (π0). A result measured on one does not transfer to another without an argument about which property caused it (inference). Lesson 22.5 builds an evaluation that separates the generalization axes; Lesson 22.6 separates the failure modes.

{"id": "22.1-check", "title": "Knowledge check", "questions": [
  {"kind": "mcq", "q": "Which property makes a policy a vision-language-action policy in this course's definition?",
   "options": ["It has more than a billion parameters", "It is pretrained on web data", "Its inputs include images and a language instruction and its output is a robot action", "It was trained on many robots"],
   "answer": 2,
   "explain": "<p>The definition is functional. Scale, pretraining and generality are separate properties to state explicitly.</p>"},
  {"kind": "mcq", "q": "A policy maps camera images and joint angles to joint targets and was trained from demonstrations. It has no language input. Which labels apply?",
   "options": ["VLA", "multimodal policy and imitation-learning policy", "robot foundation model", "VLM"],
   "answer": 1,
   "explain": "<p>Two modalities in, actions out, trained from demonstrations; no language, so not a VLA.</p>"},
  {"kind": "mcq", "q": "RT-1, RT-2 and OpenVLA all output actions as what?",
   "options": ["Continuous values from a diffusion head", "Each action dimension discretized into 256 bins, predicted as classes or tokens", "Joint torques", "Text plans"],
   "answer": 1,
   "explain": "<p>RT-1 uses 256 uniform bins per dimension; RT-2 writes them as 8 integers; OpenVLA spans the bins over the 1st to 99th percentile of the training actions.</p>"},
  {"kind": "mcq", "q": "The MiniVLA receives the instruction 'push the crimson puck to the upper zone'. What does it know about 'crimson'?",
   "options": ["That it means red, from pretraining", "Nothing: it is an unknown token, because the model has no pretrained language component", "That it is a zone", "It refuses the instruction"],
   "answer": 1,
   "explain": "<p>Its vocabulary comes from its training instructions only.</p>"}
]}

Next

Lesson 22.2 opens the MiniVLA: how an image becomes 64 tokens, an instruction 12, and how attention lets an action query read both.