A language-conditioned task adds one input to everything before it: an instruction that says which of several possible goals this episode is about. In the pick-and-place scene of Level 10 there are three cubes and three destinations, so “put the blue cube in the tray” selects one of nine (object, destination) goals, and a system that follows instructions has two jobs: map the words to a goal, and achieve that goal. Separating them makes evaluation precise. If the mapping is wrong, no controller can succeed; if the mapping is right, the rest is the manipulation problem of Level 10.
Instructions can be new in two different ways. A paraphrase says a known goal in new words (“deposit the crimson block in the bin”). A new combination pairs known words in a way never seen in training (“blue” and “tray” both appeared, but never together). A system can handle one and fail the other, and an evaluation that mixes them cannot say which. This lesson builds both splits, measures three instruction-to-goal baselines on each, and then says what a vision-language-action model changes.
[!established] Vision-language-action models A vision-language-action (VLA) model is a policy whose backbone is a vision-language model pretrained on web data, fine-tuned to output robot actions. RT-2 (arXiv:2307.15818) co-fine-tunes such a model on robot trajectories and web vision-language tasks and expresses actions as text tokens. OpenVLA (arXiv:2406.09246) is a 7-billion-parameter open model built on a Llama 2 language model with a visual encoder that fuses DINOv2 and SigLIP features, trained on 970 000 real-robot demonstrations. Octo (arXiv:2405.12213) is a transformer policy trained on 800 000 trajectories from Open X-Embodiment that accepts language or goal images. π0 (arXiv:2410.24164) puts a flow-matching action model on top of a pretrained vision-language model. All four are in the reading list.
This lesson has no lab; it is about splits, and its picture is the table of output (2). One row fails only on new words, one only on new combinations, and one partly on both.
30 instruction templates are written over the three objects (each with two training names, such as “red cube” and “red block”) and three destinations (“tray”; “left zone” or “zone on the left”; the same on the right). 18 templates are for training; 12 are held-out paraphrases with new verbs and phrasings, and some use synonyms never seen in training (“crimson block”, “azure block”, “emerald cube”, “bin”, “left pad”). Of the 9 (object, destination) pairs, 3 are held out: (blue cube, tray), (red cube, left zone), (green cube, right zone). The three splits:
| Split | Templates | Pairs | Instructions |
|---|---|---|---|
| training | 18 training | 6 training | 108 |
| held-out paraphrases | 12 held out | 6 training | 72 |
| held-out combinations | 18 training | 3 held out | 54 |
Output (2) measures the accuracy of the (object, destination) each baseline returns:
| Baseline | Training | Paraphrases | Combinations |
|---|---|---|---|
| keyword parser (scripted) | 1.00 | 0.50 | 1.00 |
| bag of words, joint labels | 1.00 | 0.71 | 0.00 |
| bag of words, object and destination heads | 1.00 | 0.83 | 0.80 |
The keyword parser looks for colour words and destination words. It is compositional by construction, so it handles every new combination, and it fails on exactly the paraphrases that use a word it does not contain (half of them). The joint classifier (multinomial naive Bayes over word counts, one class per training pair) is the opposite: it tolerates some new wording, because enough known words remain to vote, and it cannot answer a held-out combination at all, because no such class exists. Factorizing the same classifier into an object head and a destination head recovers 80% of the held-out combinations and does best on paraphrases. Its remaining errors are co-occurrence leaking into a head: 10 of its 11 errors are (blue cube, tray) instructions read as the red cube, and the destination head is never wrong. In training the word “tray” appeared only with the red and green cubes, so the object head counts “tray” as evidence against blue.
Output (3) runs the keyword parser in front of the scripted pick-and-place of Lesson 10.2 (with all three cubes on the table, any one as the target, and the tray or a zone as the destination) on 15 instructions per split. Every correctly parsed instruction was carried out (15, 9 and 15 of 15), and no incorrectly parsed one could be. With a reliable controller, the system’s success is its parse accuracy, so the language splits measure the language part.
[!recommendation] Report language generalization by axis Build held-out splits for new wording and for new combinations separately, and add held-out objects, scenes and embodiments as separate axes when they apply. Report each split’s success with an interval, and the parse or grounding accuracy alongside task success where the system has a separable language stage.
A VLA replaces all three baselines and the scripted controller with one network from pixels and words to actions. Pretraining on web text gives it the synonyms that defeated the keyword parser, which is the paraphrase axis (inference from the RT-2 and OpenVLA framing; their own evaluations are the evidence). Nothing in that architecture guarantees the compositional axis: an end-to-end policy trained on robot data in which some (object, destination) pairs never occur can behave like the joint classifier on those pairs, and whether it does is an empirical question for each model and dataset. Because the two stages are fused, the evaluation can no longer attribute a failure to grounding or to control. Splits like the ones above, and scenes in which ground truth is known, as in MuJoCo, are how to recover that attribution (Lesson 21.5).
INPUT: pick_place.xml; Lesson 10.2's pipeline
PROCESS: generate 234 instructions in three splits; evaluate a keyword parser and two bag-of-words classifiers; execute parsed instructions with the scripted pick-and-place
OUTPUT: printed tables
```python file=examples/l16_1_language.py “"”Lesson 16.1: language-conditioned pick-and-place, and the two ways instructions generalize.
INPUT pick_place.xml (three cubes, a tray, a left and a right zone); Lesson 10.2’s pipeline PROCESS (1) 30 instruction templates over 3 objects and 3 destinations: 18 for training, 12 held-out paraphrases (new wording and synonyms); 3 of the 9 object-destination combinations held out. Splits: training (training templates, training combinations), held-out paraphrases, held-out combinations; (2) three instruction-to-goal baselines evaluated on each split: a scripted keyword parser, a bag-of-words classifier over joint (object, destination) labels, and the same classifier factorized into an object head and a destination head; (3) the keyword parser driving the scripted pick-and-place on 15 instructions per split, with all three cubes on the table: parse and task success OUTPUT printed tables
Run: python examples/l16_1_language.py “””
import itertools import math import os import sys from pathlib import Path
import mujoco import numpy as np
sys.path.insert(0, str(Path(file).resolve().parent)) import l10_2_pick_place as pp # noqa: E402 (the pipeline of Lesson 10.2)
FAST = os.environ.get(“MJC_FAST”) == “1” OBJECTS = {“red_cube”: (“red cube”, “red block”), “green_cube”: (“green cube”, “green block”), “blue_cube”: (“blue cube”, “blue block”)} SYNONYMS = {“red_cube”: “crimson block”, “green_cube”: “emerald cube”, “blue_cube”: “azure block”} GOALS = {“tray”: (“tray”, “tray”), “left_zone”: (“left zone”, “zone on the left”), “right_zone”: (“right zone”, “zone on the right”)} GOAL_SYNONYMS = {“tray”: “bin”, “left_zone”: “left pad”, “right_zone”: “right pad”} TRAIN_TEMPLATES = [ “put the {o} in the {g}”, “place the {o} in the {g}”, “move the {o} to the {g}”, “put the {o} on the {g}”, “pick up the {o} and put it in the {g}”, “pick up the {o} and place it on the {g}”, “take the {o} to the {g}”, “bring the {o} to the {g}”, “the {o} goes in the {g}”, “carry the {o} to the {g}”, “set the {o} on the {g}”, “drop the {o} in the {g}”, “move the {o} onto the {g}”, “place the {o} into the {g}”, “please put the {o} in the {g}”, “grab the {o} and move it to the {g}”, “the {g} is where the {o} goes”, “lift the {o} and put it on the {g}”] PARAPHRASE_TEMPLATES = [ “could you transfer the {o} over to the {g}”, “relocate the {o} to the {g}”, “i would like the {o} in the {g}”, “deposit the {o} into the {g}”, “the {o} belongs on the {g}”, “shift the {o} across to the {g}”, “put the {O} in the {G}”, “move the {O} to the {G}”, “place the {O} on the {G}”, “the {O} goes in the {g}”, “take the {o} to the {G}”, “kindly deposit the {O} in the {G}”] HELD_OUT_PAIRS = {(“blue_cube”, “tray”), (“red_cube”, “left_zone”), (“green_cube”, “right_zone”)} PAIRS = list(itertools.product(OBJECTS, GOALS)) TRAIN_PAIRS = [p for p in PAIRS if p not in HELD_OUT_PAIRS]
def instructions(templates, pairs, rng) -> list[tuple[str, tuple[str, str]]]: out = [] for t in templates: for obj, goal in pairs: text = t.format(o=OBJECTS[obj][rng.integers(2)], g=GOALS[goal][rng.integers(2)], O=SYNONYMS[obj], G=GOAL_SYNONYMS[goal]) out.append((text, (obj, goal))) return out
def splits() -> dict: rng = np.random.default_rng(0) return {“training”: instructions(TRAIN_TEMPLATES, TRAIN_PAIRS, rng), “held-out paraphrases”: instructions(PARAPHRASE_TEMPLATES, TRAIN_PAIRS, rng), “held-out combinations”: instructions(TRAIN_TEMPLATES, sorted(HELD_OUT_PAIRS), rng)}
def keyword_parser(text: str): “"”Scripted: colour words name the object, destination words the goal; None if either is missing.””” words = text.split() obj = next((o for o, c in ((“red_cube”, “red”), (“green_cube”, “green”), (“blue_cube”, “blue”)) if c in words), None) if “tray” in words: goal = “tray” elif “zone” in words and (“left” in words or “right” in words): goal = “left_zone” if “left” in words else “right_zone” else: goal = None return (obj, goal) if obj and goal else None
class NaiveBayes: “"”Multinomial naive Bayes over word counts with add-one smoothing.”””
def fit(self, texts, labels):
self.vocab = sorted({w for t in texts for w in t.split()})
self.index = {w: i for i, w in enumerate(self.vocab)}
self.classes = sorted(set(labels))
counts = np.ones((len(self.classes), len(self.vocab)))
prior = np.zeros(len(self.classes))
for t, y in zip(texts, labels):
c = self.classes.index(y)
prior[c] += 1
for w in t.split():
counts[c, self.index[w]] += 1
self.log_prior = np.log(prior / prior.sum())
self.log_like = np.log(counts / counts.sum(axis=1, keepdims=True))
return self
def predict(self, text):
known = [self.index[w] for w in text.split() if w in self.index]
return self.classes[int(np.argmax(self.log_prior + self.log_like[:, known].sum(axis=1)))]
def part1_2(data: dict) -> None: print(f” templates: {len(TRAIN_TEMPLATES)} training, {len(PARAPHRASE_TEMPLATES)} held-out paraphrases; “ f”combinations: {len(TRAIN_PAIRS)} training, {len(HELD_OUT_PAIRS)} held out {sorted(HELD_OUT_PAIRS)}”) for name, items in data.items(): print(f” {name:<22} {len(items):>4} instructions, e.g. "{items[1][0]}"”) train_texts, train_labels = zip(*data[“training”]) joint = NaiveBayes().fit(train_texts, train_labels) obj_head = NaiveBayes().fit(train_texts, [y[0] for y in train_labels]) goal_head = NaiveBayes().fit(train_texts, [y[1] for y in train_labels]) baselines = {“keyword parser (scripted)”: keyword_parser, “bag of words, joint labels”: joint.predict, “bag of words, object and goal heads”: lambda t: (obj_head.predict(t), goal_head.predict(t))} print(f”\n(2) accuracy of the (object, destination) each baseline returns”) print(f” {‘baseline’:<38}” + ““.join(f”{n:>24}” for n in data)) for name, fn in baselines.items(): accuracies = [np.mean([fn(t) == y for t, y in items]) for items in data.values()] print(f” {name:<38}” + ““.join(f”{a:>24.2f}” for a in accuracies)) vocab = set(joint.vocab) unseen = sorted({w for t, _ in data[“held-out paraphrases”] for w in t.split() if w not in vocab}) print(f” words in held-out paraphrases never seen in training: {‘, ‘.join(unseen)}”)
class SceneEpisode(pp.Episode): “"”Lesson 10.2’s episode with all three cubes placed on the table and any one as the target.”””
def __init__(self, layout: dict[str, tuple[float, float, float]], target: str):
red = layout["red_cube"]
super().__init__(np.array(red[:2]), red[2])
m, d = self.model, self.data
for name in ("green_cube", "blue_cube"):
x, y, yaw = layout[name]
adr = m.jnt_qposadr[m.joint(name).id]
d.qpos[adr:adr + 7] = [x, y, 0.02, math.cos(yaw / 2), 0, 0, math.sin(yaw / 2)]
mujoco.mj_forward(m, d)
self.cube, self.cube_geom = m.body(target).id, m.geom(target).id
def layout(rng) -> dict: “"”Three cubes at least 8 cm apart in the 10.2 area.””” while True: xy = np.c_[rng.uniform(pp.AREA[0], 3), rng.uniform(pp.AREA[1], 3)] if min(np.linalg.norm(a - b) for a, b in itertools.combinations(xy, 2)) > 0.08: return {n: (*p, rng.uniform(-math.pi / 4, math.pi / 4)) for n, p in zip(OBJECTS, xy)}
def execute(scene: dict, obj: str, goal: str) -> bool: ep = SceneEpisode(scene, obj) dest = ep.data.site(“tray_center” if goal == “tray” else goal).xpos[:2].copy() half, release = (0.07, 0.07) if goal == “tray” else (0.06, 0.05) yaw = (scene[obj][2] + math.pi / 4) % (math.pi / 2) - math.pi / 4 cube = ep.cube_pos()[:2] steps = [(np.r_[cube, 0.15], yaw, 1.5), (np.r_[cube, 0.022], yaw, 1.0)] if not all(ep.move_to(p, y, t) for p, y, t in steps): return False ep.grip(pp.CLOSED) for p, y, t in ((np.r_[cube, 0.15], yaw, 1.0), (np.r_[dest, 0.15], 0.0, 1.5), (np.r_[dest, release], 0.0, 0.8)): if not ep.move_to(p, y, t): return False ep.grip(pp.OPEN, 0.5) ep.move_to(np.r_[dest, 0.15], 0.0, 0.8) ep.step(round(0.5 / pp.H)) p = ep.cube_pos() return bool(abs(p[0] - dest[0]) < half and abs(p[1] - dest[1]) < half and p[2] < 0.06)
def part3(data: dict) -> None: n = 4 if FAST else 15 rng = np.random.default_rng(16) print(f” {‘split’:<24}{‘parsed correctly’:>18}{‘task success’:>15} (keyword parser, {n} instructions each)”) for name, items in data.items(): picks = [items[i] for i in rng.choice(len(items), n, replace=False)] parsed = executed = 0 for text, truth in picks: guess = keyword_parser(text) if guess == truth: parsed += 1 executed += execute(layout(rng), *guess) print(f” {name:<24}{parsed:>15d}/{n}{executed:>12d}/{n}”) print(“ task success counts the target cube inside its destination after release”)
if name == “main”: data = splits() print(f”MuJoCo {mujoco.version}”) print(“(1) instruction splits”) part1_2(data) print(“(3) instructions executed by the scripted pick-and-place”) part3(data)
Output:
```text
MuJoCo 3.14.0
(1) instruction splits
templates: 18 training, 12 held-out paraphrases; combinations: 6 training, 3 held out [('blue_cube', 'tray'), ('green_cube', 'right_zone'), ('red_cube', 'left_zone')]
training 108 instructions, e.g. "put the red block in the right zone"
held-out paraphrases 72 instructions, e.g. "could you transfer the red block over to the zone on the right"
held-out combinations 54 instructions, e.g. "put the green block in the right zone"
(2) accuracy of the (object, destination) each baseline returns
baseline training held-out paraphrases held-out combinations
keyword parser (scripted) 1.00 0.50 1.00
bag of words, joint labels 1.00 0.71 0.00
bag of words, object and goal heads 1.00 0.83 0.80
words in held-out paraphrases never seen in training: across, azure, belongs, bin, could, crimson, deposit, emerald, i, kindly, like, over, pad, relocate, shift, transfer, would, you
(3) instructions executed by the scripted pick-and-place
split parsed correctly task success (keyword parser, 15 instructions each)
training 15/15 15/15
held-out paraphrases 9/15 9/15
held-out combinations 15/15 15/15
task success counts the target cube inside its destination after release
A language-conditioned policy does the right thing to the wrong object. A grounding error. If the system has a separable language stage, check its output directly on a held-out split; if not, build a probe set where each instruction differs from another in one word.
Accuracy on a held-out set is high but the system fails on new requests. The held-out set shares wording or combinations with training. Split by axis.
A classifier never outputs some goals. They were not training labels. Factorize the output, or train on all combinations.
Errors depend on which other words appear. Co-occurrence statistics leaked into the decision. Balance the training combinations, or check the model’s behaviour on minimal pairs.
Add two held-out paraphrase templates of your own that the keyword parser handles and two that it does not, predict the change in output (2), and verify it. Then add “orange cube” as a fourth object to the scene and say which baselines can handle instructions about it without retraining.
Do the checkpoint of this level properly: 30 templates of your own design over pick_place.xml, the three splits of this lesson plus a fourth with a held-out object colour, and a baseline that uses a pretrained sentence-embedding model (nearest training instruction by cosine similarity) against the keyword parser. Report accuracy per split with Wilson intervals, and execute 20 instructions per split with the scripted pick-and-place.
[!research] Measuring the language axis of a VLA Published VLA evaluations mix generalization axes (new objects, positions, instructions, scenes) and often report a single success rate per suite. Perturbation suites such as LIBERO-PRO and LIBERO-Plus (in the reading list) were designed to separate some of them. A simulator with known object identities lets a researcher go further: generate instructions from templates with controlled vocabulary and combinations, as here, and measure separately whether a policy grounds the right object and whether it then succeeds (inference: this design is a proposal, not a published protocol).
{"id": "16.1-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "A classifier with one output class per (object, destination) pair is trained on 6 of 9 pairs. What is its accuracy on instructions for the other 3, and why?",
"options": ["About 0.33, by chance", "0, because those classes do not exist in the model", "1.0, because the words are known", "It depends on the template"],
"answer": 1,
"explain": "<p>Output (2): 0.00 on held-out combinations.</p>"},
{"kind": "mcq", "q": "The keyword parser scores 1.00 on held-out combinations and 0.50 on held-out paraphrases. What does it lack?",
"options": ["Compositionality", "Knowledge of words it was not given, such as synonyms", "A controller", "Training data"],
"answer": 1,
"explain": "<p>It fails exactly on paraphrases with unseen words (crimson, azure, bin, pad).</p>"},
{"kind": "mcq", "q": "Why did every correctly parsed instruction in output (3) succeed, and no incorrectly parsed one?",
"options": ["Luck", "The scripted pick-and-place is reliable on this scene, so task success equals parse accuracy", "The parser also controls the arm", "The splits are too easy"],
"answer": 1,
"explain": "<p>With a reliable controller, failures are attributable to the language stage.</p>"},
{"kind": "mcq", "q": "Which generalization axis does web-scale language pretraining most directly address?",
"options": ["New combinations of known words", "New wording of known goals (paraphrases, synonyms)", "New robot embodiments", "New physics"],
"answer": 1,
"explain": "<p>Pretrained language models know synonyms; compositional generalization over robot data is a separate question.</p>"}
]}
Level 17 turns from language back to physics: randomizing what the simulator gets wrong so that a controller tuned in simulation survives parameters it has not seen.