A policy that observes the simulator’s state is given, for free, the answer to the hardest part of a real robot’s problem: where things are. Replace that state with pixels and the policy has to learn perception and control together, from the same demonstrations. This lesson measures what that costs, with every other factor held fixed: the same task, the same expert, the same demonstrations rendered three ways, the same training procedure. Then it changes what the camera sees without changing the task (dimmer, moved lights; a camera moved 5 cm and tilted 3°) and plants a shortcut in the training data to see whether the image policy finds it.
The task is the point mass of Level 14 with one change: the goal moves to a new random place in every episode. A state policy receives the goal’s coordinates. An image policy sees a 64 × 64 view from a camera above the arena, where the goal is a green disc, and must find it. Proprioception (the mass’s own position and velocity) is given to every policy, as joint encoders are on a real robot.
[!established] What the comparison controls The demonstrations (100 expert episodes, 4 653 frames) are recorded once and rendered as state, RGB and RGB-D; every policy is trained on the same frames with the same loss and number of steps (
mjcourse.il.bc.fitfor state,mjcourse.il.vision.fit_imagesfor images: three strided convolutions, then a two-layer head that also receives proprioception); every policy is tested on the same 50 held-out tasks, drawn from a fixed generator. Only the observation differs.
This lesson has no lab; training image policies takes minutes of CPU time per seed. The camera of Lesson 11.1’s lab is the right picture to keep in mind: what the policy receives is a small top view of about 1.04 m of floor, roughly 62 pixels per metre, in which the goal disc (7 cm across) is about 4 pixels wide and the agent about 3.
Output (1), success on the 50 held-out tasks, three seeds:
| Observation | Nominal | Dim light | Camera shifted |
|---|---|---|---|
| state + goal coordinates | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 |
| RGB + proprioception | 0.34, 0.24, 0.44 | 0.24, 0.06, 0.20 | 0.08, 0.10, 0.00 |
| RGB-D + proprioception | 0.38, 0.42, 0.26 | 0.10, 0.00, 0.12 | 0.00, 0.00, 0.00 |
With the same 4 653 frames, the state policy solves every task and the image policies about a third. The state policy is unaffected by light and camera, which it does not observe. The image policies lose half their success in dim light and almost all of it with the camera moved 5 cm and tilted 3°: a goal that projects a few pixels away from where the training images put it is, to this network, a different goal. Depth did not help: the goal is a disc 1 mm thick, so the depth channel barely shows it, and a moved camera changes the depth image as much as the colour image. With 50 test tasks, each success rate has a 95% interval of about ±0.13, so differences within a row of less than about 0.15 are not resolved.
Output (3) clones RGB policies from 300 demonstrations (13 964 frames): nominal success rises to 0.62, 0.66 and 0.80, dim light to 0.18 to 0.32, and the shifted camera stays at 0.00 to 0.04. Three times the data doubled in-distribution success and bought almost no robustness: the new demonstrations come from the same camera under the same light, so they teach the network more about those conditions and nothing about others. The remedies for the conditions are the subject of Level 17 (randomize light and camera pose during training) and of better visual representations (pretrained encoders), not of more of the same data.
In output (2) the demonstrations are recorded with the blue distractor disc always 12 cm from the goal, as an accident of a data-collection procedure might arrange. Tested with the distractor in that same relation, the RGB policy succeeds in 0.60, 0.44 and 0.70 of the tasks, better than the policies of output (1), trained with the distractor placed at random (0.34 on average): the pair of discs is a bigger, easier target. Tested with the distractor placed at random, success falls to 0.26, 0.16 and 0.32. The network had learned to use the distractor to find the goal. The state policy, which never sees the distractor, is at 1.00 in both. A shortcut is invisible in the training data and in a test set drawn the same way; it appears only when the correlation is broken on purpose.
[!recommendation] Test vision policies against the conditions they will meet Hold out at least one change of lighting and one of camera pose, and break every correlation you know of in the training data (object colours, positions, distractors, backgrounds) in a separate test split. Report each split separately, with intervals. A vision policy evaluated only on images like its training images has been tested on the easiest condition it will ever see.
INPUT: the point-mass task with a random goal and a top camera
PROCESS: record 100 demonstrations rendered as state, RGB and RGB-D; clone each with 3 seeds; test under three conditions; repeat with a planted shortcut and with 300 demonstrations
OUTPUT: printed tables
```python file=src/mjcourse/il/goalreach.py “"”Reaching a goal the policy must see (Level 15).
The point mass of Level 14, with the goal moved to a random place right of the obstacle in every episode and a camera above the arena. A policy that observes state is given the goal’s coordinates; a policy that observes pixels must find the goal marker in the image. Optionally a distractor (a blue disc) is drawn; during training it can be placed at a fixed offset from the goal, which makes it a shortcut a vision policy may learn instead.
Conditions change what the camera sees without changing the task: “nominal”, “dim light” (the scene’s lights at a third of their intensity and moved), “camera shifted” (the camera moved 5 cm and tilted 3 degrees). The policy acts every 50 ms (5 physics steps); episodes last at most 6 s. “””
from future import annotations
import math
import mujoco import numpy as np
from mjcourse.paths import model_path
SKIP, MAX_ACTIONS = 5, 120 GOAL_LOW, GOAL_HIGH = np.array([0.20, -0.30]), np.array([0.40, 0.30]) DISTRACTOR_OFFSET = np.array([0.0, 0.12]) # training-time shortcut: always this far from the goal CONDITIONS = (“nominal”, “dim light”, “camera shifted”)
def build(condition: str = “nominal”) -> mujoco.MjModel: spec = mujoco.MjSpec.from_file(str(model_path(“point_mass”))) spec.visual.headlight.diffuse = [0.25, 0.25, 0.25] spec.visual.headlight.ambient = [0.15, 0.15, 0.15] spec.worldbody.add_light(name=”key”, pos=[0.3, -0.3, 1.5], dir=[-0.2, 0.2, -1], diffuse=[0.25, 0.25, 0.25]) spec.worldbody.add_geom(name=”distractor”, type=mujoco.mjtGeom.mjGEOM_CYLINDER, size=[0.035, 0.001, 0], pos=[0.0, 0.4, 0.002], rgba=[0.1, 0.3, 0.9, 1.0], contype=0, conaffinity=0) tilt = math.radians(3) if condition == “camera shifted” else 0.0 shift = 0.05 if condition == “camera shifted” else 0.0 spec.worldbody.add_camera(name=”top”, pos=[shift, shift, 1.25], quat=[math.cos(tilt / 2), math.sin(tilt / 2), 0, 0], fovy=45) model = spec.compile() if condition == “dim light”: model.light_diffuse[:] *= 0.33 model.light_pos[:, :2] *= -1.0 goal = model.site(“goal”).id model.site_rgba[goal] = [0.1, 0.75, 0.3, 1.0] # opaque, so it is easy to see model.site_size[goal][0] = 0.035 return model
def expert(state: np.ndarray, goal: np.ndarray) -> np.ndarray: p, v = state[:2], state[2:4] side = 1.0 if p[1] >= 0 else -1.0 target = np.array([0.0, 0.2 * side]) if p[0] < -0.02 else goal return np.clip(4.0 * (target - p) - 1.5 * v, -1.0, 1.0)
class GoalReach: def init(self, condition: str = “nominal”, size: int = 64): self.model = build(condition) self.data = mujoco.MjData(self.model) self.renderer = mujoco.Renderer(self.model, size, size) self.renderer.scene.flags[mujoco.mjtRndFlag.mjRND_SHADOW] = False # shadows cost most of a small frame self.goal_site, self.distractor = self.model.site(“goal”).id, self.model.geom(“distractor”).id
def reset(self, start: np.ndarray, goal: np.ndarray, distractor: np.ndarray | None) -> None:
m, d = self.model, self.data
mujoco.mj_resetData(m, d)
d.qpos[:] = start
m.site_pos[self.goal_site][:2] = goal
m.geom_pos[self.distractor][:2] = distractor if distractor is not None else [0.0, 0.6] # off camera
mujoco.mj_forward(m, d)
self.goal = np.asarray(goal, float)
def state(self) -> np.ndarray:
return np.r_[self.data.qpos, self.data.qvel]
def image(self, depth: bool = False) -> np.ndarray:
"""(H, W, 3) uint8 RGB, or (H, W, 4) float32 RGB in [0, 1] plus depth in metres."""
r = self.renderer
r.update_scene(self.data, "top")
rgb = r.render().copy()
if not depth:
return rgb
r.enable_depth_rendering()
r.update_scene(self.data, "top")
z = r.render().copy()
r.disable_depth_rendering()
return np.concatenate([rgb / 255.0, z[..., None]], axis=-1).astype(np.float32)
def step(self, action: np.ndarray) -> bool:
d = self.data
d.ctrl[:] = np.clip(action, -1, 1)
for _ in range(SKIP):
mujoco.mj_step(self.model, d)
return bool(np.linalg.norm(d.qpos - self.goal) < 0.04 and np.linalg.norm(d.qvel) < 0.05)
def close(self) -> None:
self.renderer.close()
def sample_task(rng: np.random.Generator) -> tuple[np.ndarray, np.ndarray]: start = rng.uniform([-0.40, -0.25], [-0.30, 0.25]) goal = rng.uniform(GOAL_LOW, GOAL_HIGH) return start, goal
def distractor_position(goal: np.ndarray, shortcut: bool, rng: np.random.Generator) -> np.ndarray: “"”At a fixed offset from the goal (the shortcut) or anywhere right of the obstacle.””” if shortcut: return goal + DISTRACTOR_OFFSET * (1 if goal[1] < 0.15 else -1) return rng.uniform(GOAL_LOW, GOAL_HIGH)
```python file=src/mjcourse/il/vision.py
"""Behaviour cloning from images (Level 15).
A small convolutional network reads a 64 x 64 image (RGB, or RGB plus depth), and its
features are concatenated with proprioception (the mass's position and velocity) before a
two-layer head predicts the action. Trained by mean-squared error like the state policy of
mjcourse.il.bc, with the same data.
"""
from __future__ import annotations
import numpy as np
import torch
from torch import nn
class ConvPolicy(nn.Module):
def __init__(self, channels: int, proprio: int, actions: int = 2):
super().__init__()
self.conv = nn.Sequential(
nn.Conv2d(channels, 16, 5, stride=2, padding=2), nn.ReLU(), # 32 x 32
nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(), # 16 x 16
nn.Conv2d(32, 32, 3, stride=2, padding=1), nn.ReLU(), # 8 x 8
nn.Flatten(), nn.Linear(32 * 8 * 8, 128), nn.ReLU())
self.head = nn.Sequential(nn.Linear(128 + proprio, 128), nn.ReLU(), nn.Linear(128, actions))
def forward(self, image: torch.Tensor, proprio: torch.Tensor) -> torch.Tensor:
return self.head(torch.cat([self.conv(image), proprio], dim=-1))
class ImagePolicy:
"""Callable on (image, proprioception) for one step; images (H, W, C), uint8 or float."""
def __init__(self, net, p_mean, p_std, a_mean, a_std, image_scale):
self.net, self.p_mean, self.p_std, self.a_mean, self.a_std, self.image_scale = net, p_mean, p_std, a_mean, a_std, image_scale
def prepare(self, images: np.ndarray) -> torch.Tensor:
x = torch.as_tensor(np.asarray(images, np.float32) / self.image_scale)
return x.permute(0, 3, 1, 2)
@torch.no_grad()
def __call__(self, image: np.ndarray, proprio: np.ndarray) -> np.ndarray:
p = torch.as_tensor(((np.asarray(proprio) - self.p_mean) / self.p_std)[None], dtype=torch.float32)
return self.net(self.prepare(image[None]), p)[0].numpy() * self.a_std + self.a_mean
def fit_images(images: np.ndarray, proprio: np.ndarray, actions: np.ndarray, steps: int = 3000, batch: int = 128,
lr: float = 1e-3, seed: int = 0) -> ImagePolicy:
"""images (N, H, W, C): uint8 RGB, or float32 RGB in [0, 1] with depth (m) as a fourth channel."""
torch.manual_seed(seed)
image_scale = 255.0 if images.dtype == np.uint8 else np.r_[np.ones(3), np.full(images.shape[-1] - 3, 1.25)].astype(np.float32)
p_mean, p_std = proprio.mean(axis=0), proprio.std(axis=0) + 1e-6
a_mean, a_std = actions.mean(axis=0), actions.std(axis=0) + 1e-6
net = ConvPolicy(images.shape[-1], proprio.shape[1], actions.shape[1])
policy = ImagePolicy(net, p_mean, p_std, a_mean, a_std, image_scale)
p_all = torch.as_tensor((proprio - p_mean) / p_std, dtype=torch.float32)
a_all = torch.as_tensor((actions - a_mean) / a_std, dtype=torch.float32)
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
gen = np.random.default_rng(seed)
for _ in range(steps):
idx = gen.integers(0, len(images), min(batch, len(images)))
loss = (net(policy.prepare(images[idx]), p_all[idx]) - a_all[idx]).pow(2).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
return policy
```python file=examples/l15_1_pixels.py “"”Lesson 15.1: from state to pixels, with equal demonstrations, held-out conditions and a shortcut.
INPUT mjcourse.il.goalreach (point mass, random goal, top camera), mjcourse.il.bc and .vision PROCESS (1) 100 expert demonstrations with the distractor placed at random; three policies cloned from them: state (position, velocity, goal coordinates), RGB 64 x 64 plus proprioception, RGB-D plus proprioception; 3 seeds each; success on 50 held-out tasks under nominal conditions, dim light, and a shifted camera; (2) the same demonstrations recorded with the distractor always 12 cm from the goal (a shortcut); RGB policies cloned from them, tested with the distractor in the same place and at random; (3) RGB policies cloned from three times as many demonstrations OUTPUT printed tables
Run: MUJOCO_GL=egl python examples/l15_1_pixels.py (or osmesa without a GPU; about 6 minutes on 18 cores) “””
import os from multiprocessing import Pool
import numpy as np import torch
from mjcourse import stats from mjcourse.il import bc, vision from mjcourse.il import goalreach as gr
FAST = os.environ.get(“MJC_FAST”) == “1” DEMOS, TESTS, SEEDS = (12, 8, 1) if FAST else (100, 50, 3)
def record(shortcut: bool, seed: int = 0, demos: int = DEMOS) -> dict: “"”Demonstrations at the policy’s rate: state, goal, RGB, RGB-D and the expert’s action.””” env, rng, other = gr.GoalReach(“nominal”), np.random.default_rng([seed, 11]), np.random.default_rng([seed, 12]) out = {“state”: [], “goal”: [], “rgb”: [], “rgbd”: [], “action”: []} for _ in range(demos): start, goal = gr.sample_task(rng) env.reset(start, goal, gr.distractor_position(goal, shortcut, other)) for _ in range(gr.MAX_ACTIONS): s = env.state() a = gr.expert(s, goal) for k, v in ((“state”, s), (“goal”, goal), (“rgb”, env.image()), (“rgbd”, env.image(depth=True)), (“action”, a)): out[k].append(v) if env.step(a): break env.close() return {k: np.array(v) for k, v in out.items()}
def evaluate(kind: str, policy, condition: str, shortcut_at_test: bool) -> float: env, rng = gr.GoalReach(condition), np.random.default_rng(5150) # the same tasks for every policy other = np.random.default_rng(5151) # distractors drawn separately wins = 0 for _ in range(TESTS): start, goal = gr.sample_task(rng) env.reset(start, goal, gr.distractor_position(goal, shortcut_at_test, other)) for _ in range(gr.MAX_ACTIONS): s = env.state() if kind == “state”: a = policy(np.r_[s, goal]) else: a = policy(env.image(depth=kind == “rgbd”), s) if env.step(a): wins += 1 break env.close() return wins / TESTS
def train_and_test(job) -> dict: kind, seed, shortcut, demos = job torch.set_num_threads(2) data = record(shortcut, demos=demos) if kind == “state”: policy, _ = bc.fit(np.c_[data[“state”], data[“goal”]], data[“action”], seed=seed) else: policy = vision.fit_images(data[kind], data[“state”], data[“action”], steps=300 if FAST else 3000, seed=seed) if shortcut: return {“kind”: kind, “seed”: seed, “shortcut kept”: evaluate(kind, policy, “nominal”, True), “shortcut broken”: evaluate(kind, policy, “nominal”, False)} return {“kind”: kind if demos == DEMOS else f”{kind} x3”, “seed”: seed, **{c: evaluate(kind, policy, c, False) for c in gr.CONDITIONS}, “frames”: len(data[“action”])}
def show(rows, columns) -> None: for kind in dict.fromkeys(r[“kind”] for r in rows): sel = [r for r in rows if r[“kind”] == kind] cells = [] for c in columns: values = [r[c] for r in sel] mean, lo, hi = stats.bootstrap_ci(values) if len(values) > 1 else (values[0], values[0], values[0]) cells.append(f”{c}: {‘, ‘.join(f’{v:.2f}’ for v in values)} (mean {mean:.2f})”) print(f” {kind:<6} “ + “; “.join(cells))
if name == “main”: print(f”{DEMOS} demonstrations, {TESTS} held-out tasks, {SEEDS} seeds; images 64 x 64 from a camera 1.25 m above”) jobs = ([(k, s, False, DEMOS) for k in (“state”, “rgb”, “rgbd”) for s in range(SEEDS)] + [(k, s, True, DEMOS) for k in (“state”, “rgb”) for s in range(SEEDS)] + [(“rgb”, s, False, 3 * DEMOS) for s in range(SEEDS)]) if FAST: results = [train_and_test(j) for j in jobs] else: with Pool(len(jobs)) as pool: results = pool.map(train_and_test, jobs) plain = [r for r in results if “nominal” in r and not r[“kind”].endswith(“x3”)] print(f” demonstration frames: {plain[0][‘frames’]} (identical data for every policy)”) print(“(1) success on held-out tasks by condition, per seed”) show(plain, gr.CONDITIONS) print(“(2) trained with the distractor always 12 cm from the goal; tested with it there and at random”) show([r for r in results if “shortcut kept” in r], (“shortcut kept”, “shortcut broken”)) more = [r for r in results if r[“kind”].endswith(“x3”)] print(f”(3) RGB with {3 * DEMOS} demonstrations ({more[0][‘frames’]} frames)”) show(more, gr.CONDITIONS)
Output:
```text
100 demonstrations, 50 held-out tasks, 3 seeds; images 64 x 64 from a camera 1.25 m above
demonstration frames: 4653 (identical data for every policy)
(1) success on held-out tasks by condition, per seed
state nominal: 1.00, 1.00, 1.00 (mean 1.00); dim light: 1.00, 1.00, 1.00 (mean 1.00); camera shifted: 1.00, 1.00, 1.00 (mean 1.00)
rgb nominal: 0.34, 0.24, 0.44 (mean 0.34); dim light: 0.24, 0.06, 0.20 (mean 0.17); camera shifted: 0.08, 0.10, 0.00 (mean 0.06)
rgbd nominal: 0.38, 0.42, 0.26 (mean 0.35); dim light: 0.10, 0.00, 0.12 (mean 0.07); camera shifted: 0.00, 0.00, 0.00 (mean 0.00)
(2) trained with the distractor always 12 cm from the goal; tested with it there and at random
state shortcut kept: 1.00, 1.00, 1.00 (mean 1.00); shortcut broken: 1.00, 1.00, 1.00 (mean 1.00)
rgb shortcut kept: 0.60, 0.44, 0.70 (mean 0.58); shortcut broken: 0.26, 0.16, 0.32 (mean 0.25)
(3) RGB with 300 demonstrations (13964 frames)
rgb x3 nominal: 0.62, 0.66, 0.80 (mean 0.69); dim light: 0.18, 0.32, 0.20 (mean 0.23); camera shifted: 0.02, 0.04, 0.00 (mean 0.02)
An image policy is good in training conditions and fails after a small camera change. It learned image coordinates, not the scene. Randomize camera pose in training, or give the policy an input that does not depend on it (a tracked object position, depth back-projected to the world frame).
Success drops when the background, lighting or an unrelated object changes. Shortcut features. Find correlations in the data and break them in a test split.
More demonstrations help less than expected. They come from the same conditions. Spend them on variation (lighting, camera, distractors) instead of repetition.
Training images render differently from evaluation images. Shadows, lights, resolution or the renderer itself changed between recording and testing. Record the rendering settings with the data (Lesson 11.3).
Add a fourth condition, “goal recoloured” (the goal disc drawn in yellow), and predict the success of the state and RGB policies under it before measuring. Then train the RGB policy with the goal colour randomized among four colours and repeat.
Train the RGB policy with camera pose randomized during demonstration recording (±5 cm, ±3°), with the same number of frames as output (1), and report nominal and shifted-camera success over 3 seeds. Then do the checkpoint of this level on the pick task of Lesson 10.2: state against RGB-D with equal demonstrations, a held-out lighting and camera-pose condition, and intervals.
[!research] Visual generalization axes for robot policies Vision-based manipulation policies are usually reported on in-distribution scenes, and the generalization axes they are tested on (new objects, new positions, new instructions, new scenes, new embodiments) differ between papers. This lesson’s three conditions are the visual axes in their simplest form, and its result (more in-distribution data raises in-distribution success and not robustness) is a pattern to test for, not a law: with pretrained visual encoders or randomized rendering, the slope with data can differ (inference). Evaluations such as LIBERO-PRO and LIBERO-Plus (in the reading list) perturb initial states, instructions and visual conditions of an existing benchmark for exactly this reason.
{"id": "15.1-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "With the same demonstrations, the state policy succeeds on every task and the RGB policy on about a third. What does the RGB policy have to learn that the state policy is given?",
"options": ["The dynamics", "Where the goal is, from the image", "The reward", "The expert's gains"],
"answer": 1,
"explain": "<p>Perception: the state policy receives the goal's coordinates.</p>"},
{"kind": "mcq", "q": "Tripling the demonstrations doubled nominal success but left success with a shifted camera near zero. Why?",
"options": ["The network is too small", "The new data came from the same camera and light, so it taught nothing about other conditions", "The test set is too small", "Depth was missing"],
"answer": 1,
"explain": "<p>Output (3): 0.69 nominal on average, 0.02 with the camera shifted.</p>"},
{"kind": "mcq", "q": "A policy trained with a distractor always near the goal does better than one trained without it, as long as the distractor stays there. What did it learn?",
"options": ["To ignore the distractor", "To use the distractor as a cue for the goal, a shortcut that fails when the correlation is broken", "Better dynamics", "Depth"],
"answer": 1,
"explain": "<p>Output (2): 0.58 with the correlation kept, 0.25 with it broken.</p>"},
{"kind": "numeric", "q": "A success rate is measured on 50 tasks and comes out at 0.50. Using the normal approximation, what is the half-width of its 95% interval?",
"answer": 0.139, "tol": 0.003,
"explain": "<p>$1.96\\sqrt{0.5 \\times 0.5 / 50}$ = 0.139.</p>"}
]}
Level 16 conditions tasks on language: what an instruction has to be mapped to, and the two different ways an instruction-following system can fail to generalize.