Evaluating embodied AI in simulation means deciding three things that a reported success rate silently depends on:
This lesson does all three on the push task of Level 12:
[!established] Ground truth from the simulator MuJoCo knows every body’s pose (
xpos,xmat) and every camera’s pose and field of view (cam_xpos,cam_xmat,cam_fovy). It also renders depth along the optical axis, exact to float32 (Lesson 11.2), and per-pixel segmentation of geoms (Lesson 11.2). A 6D pose label is the object’s pose expressed in the camera frame. MuJoCo’s camera frame has x right, y up, and looks along −z. OpenCV and most pose datasets use x right, y down, looking along +z. Converting between them flips the y and z axes: $R_\text{cv} = \text{diag}(1, -1, -1)\, R_\text{mj}$, and likewise for the translation.
There is no lab for this lesson; it needs offscreen rendering in Python. The Lesson 11.2 lab shows the depth and segmentation images this lesson uses.
For 20 random puck poses, the puck’s pose in the overview camera’s frame (320 by 240 pixels, field of view 45°) gives the 3D position of its top face’s centre. Projecting it with the intrinsics $f = \tfrac{H}{2}/\tan(\text{fovy}/2)$ and the image centre gives a pixel. Two checks follow. That pixel must lie in the puck’s segmentation mask. The rendered depth at that pixel must equal the depth, along that pixel’s ray, of the top face as the label places it.
| Camera axes in the label | Projected centre on the puck’s mask | Depth difference |
|---|---|---|
| converted to OpenCV | 20 of 20 poses | at most 0.31 µm |
| MuJoCo’s, used as if OpenCV (conversion forgotten) | 0 of 20 | (no point lands on the puck) |
With the conversion, labels and renders agree to float32 rounding. Without it, every object lies behind the camera ($z < 0$ in the OpenCV convention). That is the most common error in synthetic pose datasets (inference), and a two-line check catches it.
PushEnv declares its puck position as coming from a “camera tracker” (Lesson 12.2), but it reads it from the simulator. Here the tracker is real. Every control step renders depth and segmentation, back-projects the puck’s pixels with the camera’s believed pose, and averages their horizontal positions. The pushing controller of Lesson 17.1 then uses that estimate. Over 50 tasks:
| Puck position from | Success | Mean position error |
|---|---|---|
| the simulator (privileged) | 0.98 [0.90, 1.00] | |
| the camera, pose known | 0.94 [0.84, 0.98] | 12.8 mm |
| the camera, believed pose off by a 1° tilt | 0.76 [0.63, 0.86] | 4.5 mm |
| the camera, believed pose off by a 2° tilt | 0.00 [0.00, 0.07] | 16.4 mm |
With a correct camera, perception costs a few points of success. The 12.8 mm error is plausibly mostly a bias: the camera sees the puck’s near side and top, not its centre (inference). A calibration error of 2° takes success to zero. The mean error is a poor guide: the 1° tilt has the smallest mean error and still loses 18 points. The error’s direction and how it changes as the puck moves matter more than its average size (inference). A success rate measured with privileged state is not a vision-based result. A vision-based result needs its sensing axis, here camera calibration, evaluated explicitly.
The camera-based controller (pose known), on 50 tasks per row, changing one factor each time:
| Axis | Change | Success |
|---|---|---|
| nominal | 0.94 [0.84, 0.98] | |
| object | puck 4 times heavier (0.8 kg) | 0.92 [0.81, 0.97] |
| spatial | goals at x 0.68 to 0.74 m (usual 0.58 to 0.66) | 1.00 [0.93, 1.00] |
| embodiment | arm’s force cap halved | 0.94 [0.84, 0.98] |
| sensing | believed camera pose off by 2° (above) | 0.00 [0.00, 0.07] |
Along the object, spatial and embodiment axes this controller holds; along the sensing axis it fails completely. An evaluation that varied only the physical axes would miss the one failure that matters here. One that varied several at once could not say which one broke it. A fifth axis, instruction, applies to language-conditioned policies: Lesson 16.1 holds out paraphrases and compositions for it.
[!recommendation] An embodied-AI evaluation in MuJoCo
- The policy’s inputs: list every one with its source, check it (
mjcourse.capstoneruns the leak checks), and keep privileged state out of evaluation.- Axes: vary one at a time, and include the sensing axis (camera pose, intrinsics, lighting) for any vision-based policy.
- Labels: check generated perception labels against the renders, including the camera-axis convention, before training on them.
- Reporting: give per-axis results with intervals next to the nominal one, and the privileged-state result as a reference, not as the result.
INPUT: PushEnv, its overview camera at 320 x 240, the Lesson 17.1 controller
PROCESS: 6D pose labels checked against depth and segmentation; puck position from the simulator or from back-projected depth, with camera calibration errors; one-axis-at-a-time evaluation
OUTPUT: printed tables
```python file=examples/l21_5_embodied_eval.py “"”Lesson 21.5: ground truth for perception, privileged state, and generalization axes (Level 21).
INPUT PushEnv (Level 12): arm7 pushes a puck to a goal; its “overview” camera (fovy 45 deg) rendered at 320 x 240; the pushing controller tuned in Lesson 17.1 PROCESS (1) 6D pose labels: for 20 random puck poses, the puck’s pose in the camera frame, converted from MuJoCo’s camera convention (x right, y up, looking along -z) to OpenCV’s (x right, y down, looking along +z); checked against the rendered depth and segmentation: the projected top-face centre’s pixel must lie in the puck’s mask, and the rendered depth there must equal the depth, from the label, of the top face along that pixel’s ray; the same with the conversion forgotten; (2) the puck position the controller uses, from the simulator (privileged), or estimated every step from the camera (depth of the puck’s segmented pixels, back-projected), with the camera’s believed pose correct or off by a 1 or 2 degree tilt: estimation error and success on 50 tasks; (3) generalization axes for the camera-based controller: a 4x heavier puck (object), goals beyond the usual region (spatial), half the arm’s force cap (embodiment), against nominal OUTPUT printed tables
Run: MUJOCO_GL=egl (or osmesa) python examples/l21_5_embodied_eval.py (about 6 minutes) “””
import math import os
import mujoco import numpy as np
from mjcourse.envs import PushEnv from mjcourse.stats import wilson_interval
FAST = os.environ.get(“MJC_FAST”) == “1” W, H = 320, 240 TASKS = 8 if FAST else 50 P = {“speed”: 0.107, “slow_gain”: 2.168, “depth”: 0.004, “lateral”: 0.011} # tuned in Lesson 17.1 R_PUCK, HALF, DT, PUCK_HALF_HEIGHT = 0.035, 0.008, 0.05, 0.02 CV = np.diag([1.0, -1.0, -1.0]) # MuJoCo camera axes -> OpenCV camera axes
def pusher(tcp: np.ndarray, puck: np.ndarray, goal: np.ndarray) -> np.ndarray: “"”Lesson 17.1’s controller: get behind the puck on its line to the goal, then push, slowing near the goal.””” to_goal = goal - puck d = np.linalg.norm(to_goal) if d < 0.01: return np.zeros(2, np.float32) u = to_goal / d n = np.array([-u[1], u[0]]) rel = tcp - puck along, lateral = rel @ u, rel @ n contact = -(R_PUCK + HALF) if along > contact + 0.015 or abs(lateral) > P[“lateral”]: side = n * math.copysign(R_PUCK + HALF + 0.03, lateral if lateral != 0 else 1.0) target = puck + side if along > contact else puck - (R_PUCK + HALF + 0.03) * u speed = 0.3 else: target = puck - (R_PUCK + HALF - P[“depth”]) * u speed = min(P[“speed”], P[“slow_gain”] * d) step = target - tcp dist = np.linalg.norm(step) return np.clip(step / max(dist, 1e-9) * min(speed * DT, dist) / 0.02, -1, 1).astype(np.float32)
class Camera: “"”Renders depth and segmentation from the overview camera; estimates the puck’s position in the plane.”””
def __init__(self, env: PushEnv, tilt_deg: float = 0.0):
self.env, self.renderer = env, mujoco.Renderer(env.model, H, W)
self.cam = env.model.camera("overview").id
self.puck_geom = env.model.geom("puck").id
self.f = 0.5 * H / math.tan(math.radians(env.model.cam_fovy[self.cam]) / 2)
c, s = math.cos(math.radians(tilt_deg)), math.sin(math.radians(tilt_deg))
self.believed_error = np.array([[1, 0, 0], [0, c, -s], [0, s, c]]) # about the camera's own x axis
def frames(self) -> tuple[np.ndarray, np.ndarray]:
r, d = self.renderer, self.env.data
r.update_scene(d, camera=self.cam)
r.enable_depth_rendering()
depth = r.render().copy()
r.disable_depth_rendering()
r.enable_segmentation_rendering()
seg = r.render().copy()
r.disable_segmentation_rendering()
return depth, seg
def pose(self) -> tuple[np.ndarray, np.ndarray]:
d = self.env.data
return d.cam_xpos[self.cam].copy(), d.cam_xmat[self.cam].reshape(3, 3).copy()
def estimate(self) -> np.ndarray | None:
"""Mean x, y of the back-projected puck pixels, using the believed camera pose."""
depth, seg = self.frames()
v, u = np.nonzero((seg[..., 0] == self.puck_geom) & (seg[..., 1] == int(mujoco.mjtObj.mjOBJ_GEOM)))
if len(u) == 0:
return None
z = depth[v, u]
cam = np.c_[(u + 0.5 - W / 2) * z / self.f, -(v + 0.5 - H / 2) * z / self.f, -z] # MuJoCo camera frame
pos, rot = self.pose()
world = pos + cam @ (rot @ self.believed_error).T
return world[:, :2].mean(axis=0)
def run(task_seed: int, source: str = “privileged”, tilt: float = 0.0, options: dict | None = None, max_lead: float = 0.02) -> tuple[bool, float]: env = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_lead=max_lead) obs, _ = env.reset(seed=task_seed, options=options or {}) camera = Camera(env, tilt) if source == “camera” else None errors, done, info = [], False, {} while not done: puck = obs[17:19] if camera is not None: seen = camera.estimate() if seen is not None: errors.append(np.linalg.norm(seen - env.puck_xy())) puck = seen obs, _, terminated, truncated, info = env.step(pusher(obs[14:16], puck, obs[19:21])) done = terminated or truncated if camera is not None: camera.renderer.close() env.close() return bool(info[“distance”] < env.success_radius), float(np.mean(errors)) if errors else 0.0
def summary(results: list[tuple[bool, float]], with_error: bool) -> str: ok = [r[0] for r in results] lo, hi = wilson_interval(int(sum(ok)), len(ok)) text = f”success {np.mean(ok):.2f} [{lo:.2f}, {hi:.2f}]” return text + (f”, position error {1000 * np.mean([r[1] for r in results]):.1f} mm” if with_error else “”)
if name == “main”: print(f”MuJoCo {mujoco.version}; overview camera rendered at {W} x {H}; {TASKS} tasks per row”) print(“(1) 6D pose labels of the puck in the camera frame, checked against the renders (20 poses)”) env = PushEnv(action_mode=”ee_delta”) camera = Camera(env) rng = np.random.default_rng(0) rows = {“OpenCV axes (converted)”: [], “MuJoCo axes used as OpenCV (forgotten)”: []} for k in range(20): env.reset(seed=k, options={“puck”: rng.uniform([0.42, -0.1], [0.55, 0.1])}) dof = env.model.jnt_qposadr[env.model.joint(“puck”).id] yaw = rng.uniform(-np.pi, np.pi) env.data.qpos[dof + 3:dof + 7] = [math.cos(yaw / 2), 0, 0, math.sin(yaw / 2)] mujoco.mj_forward(env.model, env.data) depth, seg = camera.frames() pos, rot = camera.pose() top = env.data.xpos[env.puck] + env.data.xmat[env.puck].reshape(3, 3) @ [0, 0, PUCK_HALF_HEIGHT] t_mj = rot.T @ (top - pos) # top-face centre in MuJoCo camera axes for label, t in ((“OpenCV axes (converted)”, CV @ t_mj), (“MuJoCo axes used as OpenCV (forgotten)”, t_mj)): if t[2] <= 0: rows[label].append((False, float(“nan”))) continue u, v = camera.f * t[0] / t[2] + W / 2 - 0.5, camera.f * t[1] / t[2] + H / 2 - 0.5 ui, vi = int(round(u)), int(round(v)) inside = 0 <= ui < W and 0 <= vi < H hit = inside and seg[vi, ui, 0] == camera.puck_geom and seg[vi, ui, 1] == int(mujoco.mjtObj.mjOBJ_GEOM) if hit: # the depth along the ray through that pixel’s centre, to the top face’s plane ray = rot @ np.array([(ui + 0.5 - W / 2) / camera.f, -(vi + 0.5 - H / 2) / camera.f, -1.0]) normal = env.data.xmat[env.puck].reshape(3, 3)[:, 2] expected = (top - pos) @ normal / (ray @ normal) rows[label].append((bool(hit), abs(depth[vi, ui] - expected) if hit else float(“nan”))) camera.renderer.close() for label, r in rows.items(): hits = [h for h, _ in r] errs = [e for h, e in r if h] text = f”rendered depth against the label’s, largest difference {1e6 * max(errs):.2f} um” if errs else “no projected point lands on the puck” print(f” {label:<40} projected top centre on the puck’s mask in {sum(hits)}/20 poses; {text}”) print(f”(2) where the controller’s puck position comes from, {TASKS} tasks”) tasks = np.random.default_rng(1717).integers(0, 10**6, TASKS) for label, kw in ((“simulator state (privileged)”, {“source”: “privileged”}), (“camera, pose known”, {“source”: “camera”}), (“camera, believed pose off by a 1 deg tilt”, {“source”: “camera”, “tilt”: 1.0}), (“camera, believed pose off by a 2 deg tilt”, {“source”: “camera”, “tilt”: 2.0})): print(f” {label:<44} “ + summary([run(int(t), **kw) for t in tasks], kw[“source”] == “camera”)) print(f”(3) generalization axes for the camera-based controller, {TASKS} tasks”) goal_rng = np.random.default_rng(5) far = [{“goal”: goal_rng.uniform([0.68, -0.2], [0.74, 0.2])} for _ in tasks] for label, maker in ((“nominal”, lambda i: {}), (“object: puck 4x heavier (0.8 kg)”, lambda i: {“options”: {“puck_mass”: 0.8}}), (“spatial: goals at x 0.68-0.74 m (usual 0.58-0.66)”, lambda i: {“options”: far[i]}), (“embodiment: arm force cap halved”, lambda i: {“max_lead”: 0.01})): print(f” {label:<52} “ + summary([run(int(t), source=”camera”, **maker(i)) for i, t in enumerate(tasks)], True))
Output:
```text
MuJoCo 3.14.0; overview camera rendered at 320 x 240; 50 tasks per row
(1) 6D pose labels of the puck in the camera frame, checked against the renders (20 poses)
OpenCV axes (converted) projected top centre on the puck's mask in 20/20 poses; rendered depth against the label's, largest difference 0.31 um
MuJoCo axes used as OpenCV (forgotten) projected top centre on the puck's mask in 0/20 poses; no projected point lands on the puck
(2) where the controller's puck position comes from, 50 tasks
simulator state (privileged) success 0.98 [0.90, 1.00]
camera, pose known success 0.94 [0.84, 0.98], position error 12.8 mm
camera, believed pose off by a 1 deg tilt success 0.76 [0.63, 0.86], position error 4.5 mm
camera, believed pose off by a 2 deg tilt success 0.00 [0.00, 0.07], position error 16.4 mm
(3) generalization axes for the camera-based controller, 50 tasks
nominal success 0.94 [0.84, 0.98], position error 12.8 mm
object: puck 4x heavier (0.8 kg) success 0.92 [0.81, 0.97], position error 12.8 mm
spatial: goals at x 0.68-0.74 m (usual 0.58-0.66) success 1.00 [0.93, 1.00], position error 12.7 mm
embodiment: arm force cap halved success 0.94 [0.84, 0.98], position error 12.9 mm
Pose labels look right in a plot and fail in training. Check them against the render: project a known point and compare its pixel and depth, in the convention your loader uses.
A vision policy does as well as the state policy. Check that it does not receive state: run the leak checks on its observation.
Success collapses on hardware with a small calibration error. Evaluate the sensing axis in simulation first: a 2° tilt took this controller from 0.94 to 0.00.
An axis shows no effect. It may be too weak, or the policy may be insensitive to it. Widen the range until it fails, then report where.
Estimate the puck’s centre instead of the mean of its visible surface, by fitting a cylinder of the known radius to the back-projected points. Measure the position error and success with the pose known and with a 1° tilt.
Build a 6D pose dataset generator for two objects of the course’s scenes: RGB, depth, segmentation and OpenCV-convention labels at random poses and lighting, each label checked as in part (1). Train a pose estimator on it and report its error separately for each of the generalization axes above.
[!research] Evaluation in the VLA era LIBERO-PRO and LIBERO-Plus (in the reading list) perturb a benchmark’s objects, positions, instructions and sensing to test what the original protocol could not, and SimplerEnv measures how simulated success predicts real success. Read them with this lesson’s two questions. What does the policy see? Which single factor does each axis change? 6D pose estimators trained on synthetic data meet the same label-convention and sensing questions (inference).
{"id": "21.5-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "MuJoCo's camera looks along -z with y up. What converts a camera-frame point to OpenCV's convention?",
"options": ["Swap x and y", "Negate y and z", "Negate x", "Nothing"],
"answer": 1,
"explain": "<p>diag(1, -1, -1); forgetting it puts every point behind the camera (0 of 20 on the mask).</p>"},
{"kind": "mcq", "q": "With a 1 degree calibration error the mean position error was smaller than with a correct camera, yet success fell from 0.94 to 0.76. What does that show?",
"options": ["The evaluation is wrong", "Mean error is a poor guide; how the error is oriented and changes along the push matters", "Calibration does not matter", "More tasks are needed"],
"answer": 1,
"explain": "<p>At 2 degrees success is 0.00.</p>"},
{"kind": "mcq", "q": "Which axis broke the camera-based controller?",
"options": ["object (heavier puck)", "spatial (far goals)", "embodiment (half force)", "sensing (camera calibration)"],
"answer": 3,
"explain": "<p>The physical axes stayed at 0.92 to 1.00.</p>"},
{"kind": "mcq", "q": "How should the privileged-state result (0.98) be reported for a vision-based policy?",
"options": ["As the result", "As a reference next to the camera-based results", "Not at all", "Averaged with the camera result"],
"answer": 1,
"explain": "<p>It answers a different question: what the controller could do with perfect perception.</p>"}
]}
Levels 22 and 23 take two threads of this level to research depth: Level 22 builds, evaluates and dissects a small vision-language-action policy, and Level 23 does the same for learned world models and the planners that use them. Level 24 then puts the course to work: seven capstones and a final research capstone.